hope-voice-api/AGENTS.md

Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.

Architecture

One Docker container runs four processes via supervisord:

1. LiveKit server — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. Voice agent — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. Web frontend — static HTML served by nginx (port 9178 by default)
4. Token server — authenticated LiveKit bootstrap endpoint (port 8091 internal, proxied at /token)

Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → LLM (OpenWebUI) → Azure TTS

Quick Start

# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178

LAN access requires firewall rules: 9178/tcp (UI + signaling), 7882/udp (WebRTC media), 7881/tcp (media TCP fallback).

Ports

Port	Protocol	Service	Access
8091	TCP	Token server	internal only (proxied via nginx at /token)
7880	TCP	LiveKit HTTP/WS	internal only (proxied via nginx at /livekit/)
7881	TCP	LiveKit RTC media (TCP fallback)	LAN
7882	UDP	LiveKit RTC media (muxed)	LAN
9178	TCP	Web frontend (HTTPS)	LAN

Configuration

All config lives in .env (git-ignored). See .env.example for the full list.

Key variables:

• AZURE_SPEECH_KEY — Azure Speech resource key
• AZURE_SPEECH_REGION — default eastus
• AZURE_TTS_VOICE — default voice (en-US-Ava:DragonHDOmniLatestNeural)
• LLM_BASE_URL — OpenAI-compatible endpoint (default http://192.168.86.2:3005/api, OpenWebUI)
• LLM_MODEL — default model (default qwen2.5-vl-3b; switchable live from the web UI)
• LLM_API_KEY — OpenWebUI API key (bears sk-…)
• MUSIC_BASE_URL — standalone Hope Music LAN API (default http://192.168.86.2:8094)
• MUSIC_ROOM — stable player scope shared with Hope Music (default voice-room)

Hope Music Integration

Music is a separate product and deployment: snowspeeder/hope-media-music. It owns the library, YouTube Music search/download pipeline, stream proxy, and the standalone browser player at https://192.168.86.2:9179. This repo keeps only the agent-side control client.

• Hope's tools live in agent/media_mcp.py (MCP, id media): search_music, play_song, offer_song_choices, pause_music, resume_music, stop_music, skip_song, seek_music, now_playing, list_library, add_stream. They call MUSIC_BASE_URL over HTTP.
• Music uses the stable MUSIC_ROOM scope (voice-room by default), not the ephemeral LiveKit room minted for each conversation. The standalone player defaults to that same scope, so voice commands control the open music page and playback survives across voice sessions.
• Music and conversation remain mutually exclusive. The agent polls Hope Music's fresh browser-reported state. Actual playback disables the agent's mic; joining a conversation pauses music, and the last participant leaving resumes only music that the agent paused.
• A stale playing report (default: thirty seconds) is treated as stopped so closing the music page cannot leave Hope's mic disabled.
• Hope Music is a peer service: an outage logs a warning but never takes down the voice conversation. Its health and E2E tests belong in its own repo.

Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

SSML / Expressive Speech

The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as [appreciative], [whispering], and [Neutral]; the agent prompt allows these markers while keeping responses short and conversational.

Agent Prompt

The LLM is instructed to:

• Speak as if talking to someone, not writing for them to read
• Keep responses to 1-3 sentences
• Use contractions, natural fillers sparingly
• Never use markdown, lists, or formatting
• Spell out numbers and abbreviations
• Call tools, don't narrate them — if a tool matches the request, the tool call must appear in the reply; saying "I'll pause the music" without the call is a failure (nothing happens)

Tool Calling on Small Local Models

The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session conditions: with the full system prompt (~6.3KB) and the full tool surface (~26 tools), they narrate actions in first person ("Sure, I'll pause the music for you") without emitting the tool call. Measured before the fix: qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool probes passed 12/12. Three layers address this:

1. Intent router (agent/intent_router.py) — deterministic forced tool calls for high-frequency, unambiguous intents (media controls, now_playing/list_library, "remember that …" → memory_save, explicit YouTube Music searches, personal-fact recall → memory_recall / search_notes / skill_recall). It runs in HopeLLM.chat() before the LLM (same mechanism as forced dispatch), only fires when the tool is available, and guards against "how do I …" questions. New high-frequency voice intents belong here — with unit tests — not in prompt hopes. Combined forced-or-emitted coverage after the fix: 8/8 on both models.
2. Prompt rule — the "call them, don't narrate them" section in SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
3. Schema sanitizer (_sanitize_tool_schema in agent.py) — llama.cpp derives a GBNF grammar from tool schemas for qwen's Hermes-style template and aborts on length-style constraints ≥ 2000 (the memory server's maxLength: 200000 broke all qwen tool calling; gemma's text-based tool path tolerated it). The sanitizer caps minLength/maxLength/minItems/ maxItems at 1000 per chat call. Verified working post-fix: anyOf nullable fields, list[dict] with open items, enums, and regex patterns. Also _strict_tool_schema=False: llama.cpp's Gemma template ignores named tool_choice under strict schemas.

Diagnostic method (learned the hard way): a failing tool call with a small model is almost never the plumbing. Probe with the REAL conditions — full tool set, real system prompt, and a prompt that matches the tool (mismatched prompts make small models legitimately decline, which looks identical to a failure). Isolated single-tool probes will pass and mislead.

Testing

Tests run against the Unraid deployment by default (https://192.168.86.2:9178, via VOICE_WEB_URL in tests/conftest.py) — that is the instance users hit and the one CI (rebuild.yml) deploys on every push to master. The local dev compose instance is for development only; set VOICE_WEB_URL=https://localhost:9178 to target it instead. Tests that probe the local container through docker exec voice skip automatically on remote targets.

# Verify the container is running
docker compose ps

# Run the suite against the Unraid deployment (default)
python -m pytest tests/ -v -m "not slow"

# Verify the external Hope Music boundary + client configuration
python -m pytest tests/test_media.py -v

# Check all logs (single service: "voice")
docker compose logs -f voice

# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3

File Layout

~/dev/hope-voice-api/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container
├── Dockerfile           ← multi-stage build
├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml         ← LiveKit server config
├── nginx.conf           ← HTTPS UI + /livekit/ WS proxy
├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server)
├── certs/               ← persisted self-signed cert (volume mount)
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline + Hope Music handshake
│   ├── media_mcp.py     ← thin HTTP client for the separate Hope Music service
│   ├── pyproject.toml   ← Python deps (uv)
│   └── uv.lock          ← locked dependency versions
└── web/
    ├── index.html       ← voice-only single-page UI
    ├── app.js           ← LiveKit client logic (no music player)
    ├── token_server.py  ← LiveKit token endpoint (mints ephemeral room names)
    ├── manifest.json    ← PWA manifest
    ├── favicon.svg      ← site icon
    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
    └── style.css        ← minimal dark theme

Conventions

• Single voice container. LiveKit, the agent, nginx, and token endpoint run in one Docker container via supervisord. Hope Music is deliberately a separate product/container.
• No published UDP ports in compose. LiveKit binds its media ports directly on the host network (network_mode: host). This avoids the docker-proxy process explosion that hit hope-webui.
• Agent dispatch via roomConfig token claim. LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds roomConfig.agents in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server ignores browser-supplied room names and mints voice-{user}-{uuid8} server-side.
• Interruption mode must be "vad". interruption={"mode": "adaptive"} requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
• Mic requires HTTPS. Browsers block getUserMedia outside a secure context. nginx serves the UI on WEB_PORT (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in ./certs/ and the LiveKit WS is proxied through nginx at /livekit/.
• No CDN dependencies. livekit-client UMD bundle is vendored into web/; LAN devices may have no internet access.
• Transcripts flow over the data channel. The agent publishes {type: "transcript", role, text} JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as {type: "set_voice", voice} on topic "voice-control".
• Models come from OpenWebUI. The LLM endpoint is OpenWebUI's OpenAI-compatible API at LLM_BASE_URL (default http://192.168.86.2:3005/api); chat goes to /api/chat/completions and the model list comes from GET {LLM_BASE_URL}/models. Plain /v1/* routes are NOT routed on that instance (they serve the SPA). Pipe models (opencode_go_messages.*, go-chat.*, go-responses.*) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. gemma-4-e4b, qwen2.5-vl-3b, qwen3.8-27b) work.
• Reasoning models may hide their answer. Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with max_completion_tokens=1000 and a HopeLLM subclass that wraps the LLM stream, falling back to reasoning_content if content is empty.
• Azure TTS uses SSML, not JSON. The REST endpoint requires Content-Type: application/ssml+xml. The LiveKit Azure plugin handles this internally.
• Voice changes are live. The web UI sends a data message to the agent; the agent calls tts.update_options(voice=...) without restarting. The agent filters data messages by topic ("voice-control") before processing.
• Model changes are live too. The agent fetches the chat-capable model list (HopeLLM filters out embedding/reranker ids) and publishes {type: "models", models, selected}; the UI's settings sheet renders a picker and sends {type: "set_model", model}, which calls HopeLLM.update_options(model=...) (mutates self._opts.model) and acks with {type: "model_changed", model}. The choice persists in the browser's localStorage.
• Background tasks follow the live model. The dispatch MCP subprocess inherits its env at spawn, so it cannot see later set_model switches. The agent writes the selected model to <task_dir>/model.txt at session start and on every switch; TaskWorker reads that file (falling back to LLM_MODEL env) when starting a task. Without this, a switched session (e.g. gemma-4-e4b) dispatches tasks against the stale env model (qwen2.5-vl-3b), which llama-swap on local-vl-models (--models-max 1) cannot load without killing the conversation model mid-swap.
• All supervisord programs log to /dev/stdout so docker compose logs -f voice shows everything. Individual process logs are no longer written to files.
• Hope Music stays external. hope-media-music owns yt-dlp, ffmpeg, the library volume, streaming endpoints, and the player UI. Do not add those dependencies or files back to this image. This repo owns only the MCP/HTTP client and the conversation/music handshake.
• Music scope is stable. LiveKit room names are intentionally ephemeral; MUSIC_ROOM defaults to voice-room, which is also the standalone player's default. Never derive the music scope from ctx.room.name.
• Playback truth comes from fresh reports. The standalone player reports real state every few seconds. The agent polls it and accepts playing only while the report timestamp is fresh (MUSIC_REPORT_TTL, default thirty seconds), preventing a closed player from silencing the mic forever.
• Peer failure is non-fatal. Hope Music being unavailable must not stop the voice agent or fail the voice container healthcheck. Music calls log and degrade gracefully.
• Never hard-require optional container env in supervisord. %(ENV_X)s expansion aborts the whole config when a variable is missing. Settings with safe in-code defaults (including Hope Music) should inherit the container environment instead of being listed there.
• Healthchecks must use curl -f. Without it, any HTTP response — including a 503 from a squatter — counts as healthy. Voice health covers only LiveKit and its HTTPS UI; Hope Music has its own independent healthcheck.

Git

Commit frequently. Conventional commits (feat:, fix:, chore:).