hope-voice-api/AGENTS.md

Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.

Architecture

One Docker container runs four processes via supervisord:

1. LiveKit server — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. Voice agent — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. Web frontend — static HTML served by nginx (port 9178 by default)
4. Media server — FastAPI music library/player coordinator (port 8093 internal, proxied at /media/api/)

Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → LLM (OpenWebUI) → Azure TTS

Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
   └────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘

Quick Start

# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178

LAN access requires firewall rules: 9178/tcp (UI + signaling), 7882/udp (WebRTC media), 7881/tcp (media TCP fallback).

Ports

Port	Protocol	Service	Access
7880	TCP	LiveKit HTTP/WS	internal only (proxied via nginx at /livekit/)
7881	TCP	LiveKit RTC media (TCP fallback)	LAN
7882	UDP	LiveKit RTC media (muxed)	LAN
8093	TCP	Media server (library, player state, stream proxy)	internal only (proxied via nginx at /media/api/)
9178	TCP	Web frontend (HTTPS)	LAN

Configuration

All config lives in .env (git-ignored). See .env.example for the full list.

Key variables:

• AZURE_SPEECH_KEY — Azure Speech resource key
• AZURE_SPEECH_REGION — default eastus
• AZURE_TTS_VOICE — default voice (en-US-Ava:DragonHDOmniLatestNeural)
• LLM_BASE_URL — OpenAI-compatible endpoint (default http://192.168.86.2:3005/api, OpenWebUI)
• LLM_MODEL — default model (default qwen2.5-vl-3b; switchable live from the web UI)
• LLM_API_KEY — OpenWebUI API key (bears sk-…)
• YTDLP_COOKIES — optional path to a cookies.txt inside the container (e.g. /data/media/cookies.txt) for reliable YouTube downloads

Media Player

Hope can play music on the web UI's media player: YouTube Music search → yt-dlp download → ffmpeg transcode to mp3 → nginx static streaming (HTTP Range) back to the browser's <audio> element. Live audio stream URLs (internet radio) are added to the library and proxied over HTTPS by the media server (mixed-content safe).

• The permanent library lives in ./media (bind-mounted to /data/media): files/ mp3s, thumbs/ cover art, jobs/ download status, library.json.
• web/media_server.py owns the library, search, download jobs (one at a time), and per-room player desired state. The browser is the actual player: it polls GET /media/api/state?room=…, reconciles, and reports real playback back via POST /media/api/report.
• Hope's tools live in agent/media_mcp.py (MCP, id "media"): search_music, play_song, offer_song_choices, pause_music, resume_music, stop_music, skip_song, seek_music, now_playing, list_library, add_stream. The MCP process is spawned per session with MEDIA_ROOM set to the LiveKit room name, scoping player state per room.
• Music and conversation are mutually exclusive. When a track plays, the UI reports {type: "media_state", state: "playing"} on data-channel topic "media-control" and the agent disables its mic (_apply_listening in agent.py). Engaging Hope (Start conversation / text message) pauses the music; ending the conversation resumes what was playing. Voice commands (pause/skip/…) work whenever the conversation is active.

Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

SSML / Expressive Speech

The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as [appreciative], [whispering], and [Neutral]; the agent prompt allows these markers while keeping responses short and conversational.

Agent Prompt

The LLM is instructed to:

• Speak as if talking to someone, not writing for them to read
• Keep responses to 1-3 sentences
• Use contractions, natural fillers sparingly
• Never use markdown, lists, or formatting
• Spell out numbers and abbreviations

Testing

Tests run against the Unraid deployment by default (https://192.168.86.2:9178, via VOICE_WEB_URL in tests/conftest.py) — that is the instance users hit and the one CI (rebuild.yml) deploys on every push to master. The local dev compose instance is for development only; set VOICE_WEB_URL=https://localhost:9178 to target it instead.

# Verify the container is running
docker compose ps

# Run the suite against the Unraid deployment (default)
python -m pytest tests/ -v -m "not slow"

# Real end-to-end YouTube Music download (network-heavy, gated)
HOPE_RUN_YTDLP_DOWNLOAD=1 python -m pytest tests/test_media.py -m slow

Check all logs (single service: "voice")

docker compose logs -f voice

Tail individual process logs via supervisord stdout

docker exec voice tail -f /dev/stdout

Test TTS directly (outside the agent)

curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1"
-H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY"
-H "Content-Type: application/ssml+xml"
-H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3"
--data-binary '[appreciative] Test complete.' > /tmp/test.mp3

## File Layout

~/dev/voice/ ├── AGENTS.md ← you are here ├── .env.example ← config template (copy to .env) ├── .gitignore ├── docker-compose.yml ← single container ├── Dockerfile ← multi-stage build ├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start ├── livekit.yaml ← LiveKit server config ├── nginx.conf ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API) ├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server) ├── certs/ ← persisted self-signed cert (volume mount) ├── media/ ← permanent music library (bind-mounted to /data/media) │ ├── files/ ← transcoded mp3 tracks (nginx-served, HTTP Range) │ ├── thumbs/ ← cover art │ ├── jobs/ ← download job status snapshots │ └── library.json ← library metadata (permanent) ├── agent/ │ ├── agent.py ← LiveKit Agents voice pipeline │ ├── media_mcp.py ← Hope's music tools (YouTube Music via media server) │ ├── pyproject.toml ← Python deps (uv) │ └── uv.lock ← locked dependency versions └── web/ ├── index.html ← single-page voice UI (+ music panel) ├── app.js ← LiveKit client logic (+ player reconciliation) ├── media_server.py ← FastAPI library/search/download/player state ├── token_server.py ← LiveKit token endpoint (mints room name) ├── manifest.json ← PWA manifest ├── favicon.svg ← site icon ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN) └── style.css ← minimal dark theme

## Conventions

- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen2.5-vl-3b`, `qwen3.8-27b`) work.
- **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Music downloads are mp3, not YouTube streams.** The media server transcodes with yt-dlp (`-x --audio-format mp3 --audio-quality 192K --embed-thumbnail`) into `/data/media/files/{video_id}.mp3`; nginx serves those files as statics with native HTTP Range, which is what makes seeking work in the `<audio>` element. Never point the player at YouTube directly.
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
- **Never hard-require container env in supervisord.** `%(ENV_X)s` expansion aborts the whole supervisord config when the var is missing, taking every program down (this silently killed the run-151 deploy). Programs with safe in-code defaults (media server) must inherit the container env instead of listing it.
- **Healthchecks must use `curl -f`.** Without it, any HTTP response — including a 503 from a squatter — counts as healthy. Run #15 deployed "healthy" against a broken media port this way.
- **Port 8092 is taken on xNAS.** An Apache/YOURLS stack occupies 8092 on the Unraid host; the media server uses 8093. Before picking a new internal port, probe the Unraid deployment host, not just the dev box.

## Git

Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).