hope-voice-api/AGENTS.md

# Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.

## Architecture

One Docker container runs four processes via supervisord:

1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
4. **Media server** — FastAPI music library/player coordinator (port 8093 internal, proxied at /media/api/)

```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → LLM (OpenWebUI) → Azure TTS

Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
   └────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘
```

## Quick Start

```bash
# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178
```

LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
(WebRTC media), `7881/tcp` (media TCP fallback).

## Ports

| Port  | Protocol | Service              | Access       |
|-------|----------|----------------------|--------------|
| 7880  | TCP      | LiveKit HTTP/WS      | internal only (proxied via nginx at /livekit/) |
| 7881  | TCP      | LiveKit RTC media (TCP fallback) | LAN |
| 7882  | UDP      | LiveKit RTC media (muxed) | LAN   |
| 8093  | TCP      | Media server (library, player state, stream proxy) | internal only (proxied via nginx at /media/api/) |
| 9178  | TCP      | Web frontend (HTTPS) | LAN          |

## Configuration

All config lives in `.env` (git-ignored). See `.env.example` for the full list.

Key variables:
- `AZURE_SPEECH_KEY` — Azure Speech resource key
- `AZURE_SPEECH_REGION` — default `eastus`
- `AZURE_TTS_VOICE` — default voice (`en-US-Ava:DragonHDOmniLatestNeural`)
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
- `YTDLP_COOKIES` — optional path to a cookies.txt inside the container (e.g. `/data/media/cookies.txt`) for reliable YouTube downloads

## Media Player

Hope can play music on the web UI's media player: YouTube Music search →
yt-dlp raw best-audio download (webm/m4a, no transcode) → progressive
streaming from the media server while the file grows → finished file served
with native HTTP Range for seeking. Songs start playing within seconds of the
download starting and remain in the library for instant offline replay.
Live audio stream URLs (internet radio) are added to the library and proxied
over HTTPS by the media server (mixed-content safe). Search results put ready
library matches first so Hope prefers a downloaded copy over a fresh lookup.

- The permanent library lives in `./media` (bind-mounted to `/data/media`):
  `files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
- `web/media_server.py` owns the library, search, download jobs (one at a
  time), and per-room player desired state. The browser is the actual player:
  it polls `GET /media/api/state?room=…`, reconciles, and reports real
  playback back via `POST /media/api/report`.
- Hope's tools live in `agent/media_mcp.py` (MCP, id "media"):
  `search_music`, `play_song`, `offer_song_choices`, `pause_music`,
  `resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
  `list_library`, `add_stream`. The MCP process is spawned per session with
  `MEDIA_ROOM` set to the LiveKit room name, scoping player state per room.
- **Music and conversation are mutually exclusive.** When a track plays, the
  UI reports `{type: "media_state", state: "playing"}` on data-channel topic
  "media-control" and the agent disables its mic (`_apply_listening` in
  agent.py). Engaging Hope (Start conversation / text message) pauses the
  music; ending the conversation resumes what was playing. Voice commands
  (pause/skip/…) work whenever the conversation is active.

## Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

## SSML / Expressive Speech

The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as `[appreciative]`, `[whispering]`, and `[Neutral]`; the agent prompt allows these markers while keeping responses short and conversational.

## Agent Prompt

The LLM is instructed to:
- Speak as if talking to someone, not writing for them to read
- Keep responses to 1-3 sentences
- Use contractions, natural fillers sparingly
- Never use markdown, lists, or formatting
- Spell out numbers and abbreviations
- **Call tools, don't narrate them** — if a tool matches the request, the
  tool call must appear in the reply; saying "I'll pause the music" without
  the call is a failure (nothing happens)

## Tool Calling on Small Local Models

The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session
conditions: with the full system prompt (~6.3KB) and the full tool surface
(~26 tools), they narrate actions in first person ("Sure, I'll pause the
music for you") **without emitting the tool call**. Measured before the fix:
qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool
probes passed 12/12. Three layers address this:

1. **Intent router** (`agent/intent_router.py`) — deterministic forced tool
   calls for high-frequency, unambiguous intents (media controls,
   now_playing/list_library, "remember that …" → memory_save, explicit
   YouTube Music searches, personal-fact recall → memory_recall /
   search_notes / skill_recall). It runs in `HopeLLM.chat()` before the LLM
   (same mechanism as forced dispatch), only fires when the tool is
   available, and guards against "how do I …" questions. New high-frequency
   voice intents belong here — with unit tests — not in prompt hopes.
   Combined forced-or-emitted coverage after the fix: 8/8 on both models.
2. **Prompt rule** — the "call them, don't narrate them" section in
   SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
3. **Schema sanitizer** (`_sanitize_tool_schema` in agent.py) — llama.cpp
   derives a GBNF grammar from tool schemas for qwen's Hermes-style template
   and **aborts on length-style constraints ≥ 2000** (the memory server's
   `maxLength: 200000` broke all qwen tool calling; gemma's text-based tool
   path tolerated it). The sanitizer caps minLength/maxLength/minItems/
   maxItems at 1000 per chat call. Verified working post-fix: `anyOf`
   nullable fields, `list[dict]` with open items, enums, and regex patterns.
   Also `_strict_tool_schema=False`: llama.cpp's Gemma template ignores
   named tool_choice under strict schemas.

**Diagnostic method** (learned the hard way): a failing tool call with a
small model is almost never the plumbing. Probe with the REAL conditions —
full tool set, real system prompt, and a prompt that matches the tool
(mismatched prompts make small models legitimately decline, which looks
identical to a failure). Isolated single-tool probes will pass and mislead.

## Testing

**Tests run against the Unraid deployment by default** (`https://192.168.86.2:9178`,
via `VOICE_WEB_URL` in tests/conftest.py) — that is the instance users hit
and the one CI (rebuild.yml) deploys on every push to master. The local dev
compose instance is for development only; set `VOICE_WEB_URL=https://localhost:9178`
to target it instead. Tests that can only pass on the deployment host
(repo `./media` bind mount) or that probe the local container via
`docker exec voice` skip automatically on remote targets.

```bash
# Verify the container is running
docker compose ps

# Run the suite against the Unraid deployment (default)
python -m pytest tests/ -v -m "not slow"

# Real end-to-end YouTube Music download (network-heavy, gated)
HOPE_RUN_YTDLP_DOWNLOAD=1 python -m pytest tests/test_media.py -m slow

# Check all logs (single service: "voice")
docker compose logs -f voice

# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3
```

## File Layout

```
~/dev/voice/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container
├── Dockerfile           ← multi-stage build
├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml         ← LiveKit server config
├── nginx.conf           ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API)
├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server, media-server)
├── certs/               ← persisted self-signed cert (volume mount)
├── media/               ← permanent music library (bind-mounted to /data/media)
│   ├── files/           ← raw audio tracks (webm/m4a; media-server streamed, Range when complete)
│   ├── thumbs/          ← cover art
│   ├── jobs/            ← download job status snapshots
│   └── library.json     ← library metadata (permanent)
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline
│   ├── media_mcp.py     ← Hope's music tools (YouTube Music via media server)
│   ├── pyproject.toml   ← Python deps (uv)
│   └── uv.lock          ← locked dependency versions
└── web/
    ├── index.html       ← single-page voice UI (+ music panel)
    ├── app.js           ← LiveKit client logic (+ player reconciliation)
    ├── media_server.py  ← FastAPI library/search/download/player state
    ├── token_server.py  ← LiveKit token endpoint (mints room name)
    ├── manifest.json    ← PWA manifest
    ├── favicon.svg      ← site icon
    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
    └── style.css        ← minimal dark theme
```

## Conventions

- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen2.5-vl-3b`, `qwen3.8-27b`) work.
- **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Music downloads are raw audio, streamed as they grow.** yt-dlp fetches
  bestaudio directly (`-f bestaudio/best`, no `-x`/transcode) into
  `/data/media/files/{video_id}.{ext}`; the stream endpoint
  (`GET /media/api/stream/{id}`) serves the growing `.part` file chunked
  (no Content-Length) while the download runs, then serves the finished file
  with native Range for seeking. `audio_url` is that endpoint for the whole
  lifecycle — a stable URL means the browser never reloads mid-playback. The
  mp3 transcode was dropped: it delayed playback by the whole download+encode,
  and modern browsers play webm/opus and m4a everywhere. Never point the
  player at YouTube directly.
- **Search prefers the library.** `/media/api/search` merges ready library
  matches (title token/substring match) in front of the yt-dlp results, and
  the search_music tool docstring + system prompt tell Hope to always pick
  `in_library=true`. Deterministic preference lives in the server merge, not
  in prompt hopes.
- **Progress lines come from yt-dlp stderr.** `[download] N%` goes to stderr;
  only `--print-json` metadata goes to stdout. The download worker drains
  stderr in a thread (deadlock-safe) and reads stdout for the JSON — parsing
  progress from stdout silently never matched.
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
- **Never hard-require container env in supervisord.** `%(ENV_X)s` expansion aborts the whole supervisord config when the var is missing, taking every program down (this silently killed the run-151 deploy). Programs with safe in-code defaults (media server) must inherit the container env instead of listing it.
- **Healthchecks must use `curl -f`.** Without it, any HTTP response — including a 503 from a squatter — counts as healthy. Run #15 deployed "healthy" against a broken media port this way.
- **Port 8092 is taken on xNAS.** An Apache/YOURLS stack occupies 8092 on the Unraid host; the media server uses 8093. Before picking a new internal port, probe the Unraid deployment host, not just the dev box.

## Git

Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).