# Voice — Real-time Voice Assistant A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning. ## Architecture One Docker container runs three processes via supervisord: 1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media) 2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS 3. **Web frontend** — static HTML served by nginx (port 9178 by default) ``` Browser ──WebRTC──► LiveKit Server ──audio──► Agent ▲ │ └────────── audio ◄────────────────────────────┘ Azure STT → LLM (OpenWebUI) → Azure TTS ``` ## Quick Start ```bash # 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH) export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2) # 2. Build and start docker compose up --build -d # 3. Open the web UI (self-signed cert: accept the browser warning once) # https://:9178 ``` LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp` (WebRTC media), `7881/tcp` (media TCP fallback). ## Ports | Port | Protocol | Service | Access | |-------|----------|----------------------|--------------| | 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) | | 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN | | 7882 | UDP | LiveKit RTC media (muxed) | LAN | | 9178 | TCP | Web frontend (HTTPS) | LAN | ## Configuration All config lives in `.env` (git-ignored). See `.env.example` for the full list. Key variables: - `AZURE_SPEECH_KEY` — Azure Speech resource key - `AZURE_SPEECH_REGION` — default `eastus` - `AZURE_TTS_VOICE` — default voice (`en-US-Ava:DragonHDOmniLatestNeural`) - `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI) - `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI) - `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`) ## Voice Selection The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting. Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices ## SSML / Expressive Speech The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as `[appreciative]`, `[whispering]`, and `[Neutral]`; the agent prompt allows these markers while keeping responses short and conversational. ## Agent Prompt The LLM is instructed to: - Speak as if talking to someone, not writing for them to read - Keep responses to 1-3 sentences - Use contractions, natural fillers sparingly - Never use markdown, lists, or formatting - Spell out numbers and abbreviations ## Testing ```bash # Verify the container is running docker compose ps # Check all logs (single service: "voice") docker compose logs -f voice # Tail individual process logs via supervisord stdout docker exec voice tail -f /dev/stdout # Test TTS directly (outside the agent) curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \ -H "Content-Type: application/ssml+xml" \ -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \ --data-binary '[appreciative] Test complete.' > /tmp/test.mp3 ``` ## File Layout ``` ~/dev/voice/ ├── AGENTS.md ← you are here ├── .env.example ← config template (copy to .env) ├── .gitignore ├── docker-compose.yml ← single container ├── Dockerfile ← multi-stage build ├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start ├── livekit.yaml ← LiveKit server config ├── nginx.conf ← HTTPS UI + /livekit/ WS proxy ├── supervisord.conf ← process manager (livekit, agent, nginx, token-server) ├── certs/ ← persisted self-signed cert (volume mount) ├── agent/ │ ├── agent.py ← LiveKit Agents voice pipeline │ ├── pyproject.toml ← Python deps (uv) │ └── uv.lock ← locked dependency versions └── web/ ├── index.html ← single-page voice UI ├── app.js ← LiveKit client logic ├── manifest.json ← PWA manifest ├── favicon.svg ← site icon ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN) └── style.css ← minimal dark theme ``` ## Conventions - **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose. - **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui. - **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names. - **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups. - **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`. - **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access. - **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control". - **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen2.5-vl-3b`, `qwen3.8-27b`) work. - **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty. - **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally. - **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing. - **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage. - **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files. ## Git Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).