hope-voice-api/AGENTS.md

Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.

Architecture

One Docker container runs four processes via supervisord:

1. LiveKit server — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. Voice agent — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. Web frontend — static HTML served by nginx (port 9178 by default)
4. Media server — FastAPI music library/player coordinator (port 8093 internal, proxied at /media/api/)

Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → LLM (OpenWebUI) → Azure TTS

Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
   └────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘

Quick Start

# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178

LAN access requires firewall rules: 9178/tcp (UI + signaling), 7882/udp (WebRTC media), 7881/tcp (media TCP fallback).

Ports

Port	Protocol	Service	Access
7880	TCP	LiveKit HTTP/WS	internal only (proxied via nginx at /livekit/)
7881	TCP	LiveKit RTC media (TCP fallback)	LAN
7882	UDP	LiveKit RTC media (muxed)	LAN
8093	TCP	Media server (library, player state, stream proxy)	internal only (proxied via nginx at /media/api/)
9178	TCP	Web frontend (HTTPS)	LAN

Configuration

All config lives in .env (git-ignored). See .env.example for the full list.

Key variables:

• AZURE_SPEECH_KEY — Azure Speech resource key
• AZURE_SPEECH_REGION — default eastus
• AZURE_TTS_VOICE — default voice (en-US-Ava:DragonHDOmniLatestNeural)
• LLM_BASE_URL — OpenAI-compatible endpoint (default http://192.168.86.2:3005/api, OpenWebUI)
• LLM_MODEL — default model (default qwen2.5-vl-3b; switchable live from the web UI)
• LLM_API_KEY — OpenWebUI API key (bears sk-…)
• YTDLP_COOKIES — optional path to a cookies.txt inside the container (e.g. /data/media/cookies.txt) for reliable YouTube downloads

Media Player

Hope can play music on the web UI's media player: YouTube Music search → yt-dlp raw best-audio download (webm/m4a, no transcode) → progressive streaming from the media server while the file grows → finished file served with native HTTP Range for seeking. Songs start playing within seconds of the download starting and remain in the library for instant offline replay. Live audio stream URLs (internet radio) are added to the library and proxied over HTTPS by the media server (mixed-content safe). Search results put ready library matches first so Hope prefers a downloaded copy over a fresh lookup.

• The permanent library lives in ./media (bind-mounted to /data/media): files/ mp3s, thumbs/ cover art, jobs/ download status, library.json.
• web/media_server.py owns the library, search, download jobs (one at a time), and per-room player desired state. The browser is the actual player: it polls GET /media/api/state?room=…, reconciles, and reports real playback back via POST /media/api/report.
• Hope's tools live in agent/media_mcp.py (MCP, id "media"): search_music, play_song, offer_song_choices, pause_music, resume_music, stop_music, skip_song, seek_music, now_playing, list_library, add_stream. The MCP process is spawned per session with MEDIA_ROOM set to the LiveKit room name, scoping player state per room.
• Music and conversation are mutually exclusive. When a track plays, the UI reports {type: "media_state", state: "playing"} on data-channel topic "media-control" and the agent disables its mic (_apply_listening in agent.py). Engaging Hope (Start conversation / text message) pauses the music; ending the conversation resumes what was playing. Voice commands (pause/skip/…) work whenever the conversation is active.

Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

SSML / Expressive Speech

The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as [appreciative], [whispering], and [Neutral]; the agent prompt allows these markers while keeping responses short and conversational.

Agent Prompt

The LLM is instructed to:

• Speak as if talking to someone, not writing for them to read
• Keep responses to 1-3 sentences
• Use contractions, natural fillers sparingly
• Never use markdown, lists, or formatting
• Spell out numbers and abbreviations
• Call tools, don't narrate them — if a tool matches the request, the tool call must appear in the reply; saying "I'll pause the music" without the call is a failure (nothing happens)

Tool Calling on Small Local Models

The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session conditions: with the full system prompt (~6.3KB) and the full tool surface (~26 tools), they narrate actions in first person ("Sure, I'll pause the music for you") without emitting the tool call. Measured before the fix: qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool probes passed 12/12. Three layers address this:

1. Intent router (agent/intent_router.py) — deterministic forced tool calls for high-frequency, unambiguous intents (media controls, now_playing/list_library, "remember that …" → memory_save, explicit YouTube Music searches, personal-fact recall → memory_recall / search_notes / skill_recall). It runs in HopeLLM.chat() before the LLM (same mechanism as forced dispatch), only fires when the tool is available, and guards against "how do I …" questions. New high-frequency voice intents belong here — with unit tests — not in prompt hopes. Combined forced-or-emitted coverage after the fix: 8/8 on both models.
2. Prompt rule — the "call them, don't narrate them" section in SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
3. Schema sanitizer (_sanitize_tool_schema in agent.py) — llama.cpp derives a GBNF grammar from tool schemas for qwen's Hermes-style template and aborts on length-style constraints ≥ 2000 (the memory server's maxLength: 200000 broke all qwen tool calling; gemma's text-based tool path tolerated it). The sanitizer caps minLength/maxLength/minItems/ maxItems at 1000 per chat call. Verified working post-fix: anyOf nullable fields, list[dict] with open items, enums, and regex patterns. Also _strict_tool_schema=False: llama.cpp's Gemma template ignores named tool_choice under strict schemas.

Diagnostic method (learned the hard way): a failing tool call with a small model is almost never the plumbing. Probe with the REAL conditions — full tool set, real system prompt, and a prompt that matches the tool (mismatched prompts make small models legitimately decline, which looks identical to a failure). Isolated single-tool probes will pass and mislead.

Testing

Tests run against the Unraid deployment by default (https://192.168.86.2:9178, via VOICE_WEB_URL in tests/conftest.py) — that is the instance users hit and the one CI (rebuild.yml) deploys on every push to master. The local dev compose instance is for development only; set VOICE_WEB_URL=https://localhost:9178 to target it instead. Tests that can only pass on the deployment host (repo ./media bind mount) or that probe the local container via docker exec voice skip automatically on remote targets.

# Verify the container is running
docker compose ps

# Run the suite against the Unraid deployment (default)
python -m pytest tests/ -v -m "not slow"

# Real end-to-end YouTube Music download (network-heavy, gated)
HOPE_RUN_YTDLP_DOWNLOAD=1 python -m pytest tests/test_media.py -m slow

# Check all logs (single service: "voice")
docker compose logs -f voice

# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3

File Layout

~/dev/voice/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container
├── Dockerfile           ← multi-stage build
├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml         ← LiveKit server config
├── nginx.conf           ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API)
├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server, media-server)
├── certs/               ← persisted self-signed cert (volume mount)
├── media/               ← permanent music library (bind-mounted to /data/media)
│   ├── files/           ← raw audio tracks (webm/m4a; media-server streamed, Range when complete)
│   ├── thumbs/          ← cover art
│   ├── jobs/            ← download job status snapshots
│   └── library.json     ← library metadata (permanent)
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline
│   ├── media_mcp.py     ← Hope's music tools (YouTube Music via media server)
│   ├── pyproject.toml   ← Python deps (uv)
│   └── uv.lock          ← locked dependency versions
└── web/
    ├── index.html       ← single-page voice UI (+ music panel)
    ├── app.js           ← LiveKit client logic (+ player reconciliation)
    ├── media_server.py  ← FastAPI library/search/download/player state
    ├── token_server.py  ← LiveKit token endpoint (mints room name)
    ├── manifest.json    ← PWA manifest
    ├── favicon.svg      ← site icon
    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
    └── style.css        ← minimal dark theme

Conventions

• Single container. All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
• No published UDP ports in compose. LiveKit binds its media ports directly on the host network (network_mode: host). This avoids the docker-proxy process explosion that hit hope-webui.
• Agent dispatch via roomConfig token claim. LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds roomConfig.agents in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
• Interruption mode must be "vad". interruption={"mode": "adaptive"} requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
• Mic requires HTTPS. Browsers block getUserMedia outside a secure context. nginx serves the UI on WEB_PORT (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in ./certs/ and the LiveKit WS is proxied through nginx at /livekit/.
• No CDN dependencies. livekit-client UMD bundle is vendored into web/; LAN devices may have no internet access.
• Transcripts flow over the data channel. The agent publishes {type: "transcript", role, text} JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as {type: "set_voice", voice} on topic "voice-control".
• Models come from OpenWebUI. The LLM endpoint is OpenWebUI's OpenAI-compatible API at LLM_BASE_URL (default http://192.168.86.2:3005/api); chat goes to /api/chat/completions and the model list comes from GET {LLM_BASE_URL}/models. Plain /v1/* routes are NOT routed on that instance (they serve the SPA). Pipe models (opencode_go_messages.*, go-chat.*, go-responses.*) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. gemma-4-e4b, qwen2.5-vl-3b, qwen3.8-27b) work.
• Reasoning models may hide their answer. Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with max_completion_tokens=1000 and a HopeLLM subclass that wraps the LLM stream, falling back to reasoning_content if content is empty.
• Azure TTS uses SSML, not JSON. The REST endpoint requires Content-Type: application/ssml+xml. The LiveKit Azure plugin handles this internally.
• Voice changes are live. The web UI sends a data message to the agent; the agent calls tts.update_options(voice=...) without restarting. The agent filters data messages by topic ("voice-control") before processing.
• Model changes are live too. The agent fetches the chat-capable model list (HopeLLM filters out embedding/reranker ids) and publishes {type: "models", models, selected}; the UI's settings sheet renders a picker and sends {type: "set_model", model}, which calls HopeLLM.update_options(model=...) (mutates self._opts.model) and acks with {type: "model_changed", model}. The choice persists in the browser's localStorage.
• All supervisord programs log to /dev/stdout so docker compose logs -f voice shows everything. Individual process logs are no longer written to files.
• Music downloads are raw audio, streamed as they grow. yt-dlp fetches bestaudio directly (-f bestaudio/best, no -x/transcode) into /data/media/files/{video_id}.{ext}; the stream endpoint (GET /media/api/stream/{id}) serves the growing .part file chunked (no Content-Length) while the download runs, then serves the finished file with native Range for seeking. audio_url is that endpoint for the whole lifecycle — a stable URL means the browser never reloads mid-playback. The mp3 transcode was dropped: it delayed playback by the whole download+encode, and modern browsers play webm/opus and m4a everywhere. Never point the player at YouTube directly.
• Search prefers the library. /media/api/search merges ready library matches (title token/substring match) in front of the yt-dlp results, and the search_music tool docstring + system prompt tell Hope to always pick in_library=true. Deterministic preference lives in the server merge, not in prompt hopes.
• Progress lines come from yt-dlp stderr. [download] N% goes to stderr; only --print-json metadata goes to stdout. The download worker drains stderr in a thread (deadlock-safe) and reads stdout for the JSON — parsing progress from stdout silently never matched.
• External stream URLs are proxied, never played directly. An http:// radio URL would be blocked as mixed content on the HTTPS UI, so GET /media/api/stream/{id} re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to /media/files/.
• Player state is per-room desired state, reconciled by the browser. The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get s<uuid>.
• Anonymous yt-dlp first, cookies when flaky. Search (ytsearch) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into ./media/cookies.txt and set YTDLP_COOKIES=/data/media/cookies.txt.
• Never hard-require container env in supervisord. %(ENV_X)s expansion aborts the whole supervisord config when the var is missing, taking every program down (this silently killed the run-151 deploy). Programs with safe in-code defaults (media server) must inherit the container env instead of listing it.
• Healthchecks must use curl -f. Without it, any HTTP response — including a 503 from a squatter — counts as healthy. Run #15 deployed "healthy" against a broken media port this way.
• Port 8092 is taken on xNAS. An Apache/YOURLS stack occupies 8092 on the Unraid host; the media server uses 8093. Before picking a new internal port, probe the Unraid deployment host, not just the dev box.

Git

Commit frequently. Conventional commits (feat:, fix:, chore:).