hope-voice-api/AGENTS.md

Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.

Architecture

One Docker container runs three processes via supervisord:

1. LiveKit server — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. Voice agent — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. Web frontend — static HTML served by nginx (port 9178 by default)

Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → LLM (OpenWebUI) → Azure TTS

Quick Start

# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178

LAN access requires firewall rules: 9178/tcp (UI + signaling), 7882/udp (WebRTC media), 7881/tcp (media TCP fallback).

Ports

Port	Protocol	Service	Access
7880	TCP	LiveKit HTTP/WS	internal only (proxied via nginx at /livekit/)
7881	TCP	LiveKit RTC media (TCP fallback)	LAN
7882	UDP	LiveKit RTC media (muxed)	LAN
9178	TCP	Web frontend (HTTPS)	LAN

Configuration

All config lives in .env (git-ignored). See .env.example for the full list.

Key variables:

• AZURE_SPEECH_KEY — Azure Speech resource key
• AZURE_SPEECH_REGION — default eastus
• AZURE_TTS_VOICE — default voice (en-US-Ava:DragonHDOmniLatestNeural)
• LLM_BASE_URL — OpenAI-compatible endpoint (default http://192.168.86.2:3005/api, OpenWebUI)
• LLM_MODEL — default model (default qwen2.5-vl-3b; switchable live from the web UI)
• LLM_API_KEY — OpenWebUI API key (bears sk-…)

Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

SSML / Expressive Speech

The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as [appreciative], [whispering], and [Neutral]; the agent prompt allows these markers while keeping responses short and conversational.

Agent Prompt

The LLM is instructed to:

• Speak as if talking to someone, not writing for them to read
• Keep responses to 1-3 sentences
• Use contractions, natural fillers sparingly
• Never use markdown, lists, or formatting
• Spell out numbers and abbreviations

Testing

# Verify the container is running
docker compose ps

# Check all logs (single service: "voice")
docker compose logs -f voice

# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3

File Layout

~/dev/voice/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container
├── Dockerfile           ← multi-stage build
├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml         ← LiveKit server config
├── nginx.conf           ← HTTPS UI + /livekit/ WS proxy
├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server)
├── certs/               ← persisted self-signed cert (volume mount)
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline
│   ├── pyproject.toml   ← Python deps (uv)
│   └── uv.lock          ← locked dependency versions
└── web/
    ├── index.html       ← single-page voice UI
    ├── app.js           ← LiveKit client logic
    ├── manifest.json    ← PWA manifest
    ├── favicon.svg      ← site icon
    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
    └── style.css        ← minimal dark theme

Conventions

• Single container. All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
• No published UDP ports in compose. LiveKit binds its media ports directly on the host network (network_mode: host). This avoids the docker-proxy process explosion that hit hope-webui.
• Agent dispatch via roomConfig token claim. LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds roomConfig.agents in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
• Interruption mode must be "vad". interruption={"mode": "adaptive"} requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
• Mic requires HTTPS. Browsers block getUserMedia outside a secure context. nginx serves the UI on WEB_PORT (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in ./certs/ and the LiveKit WS is proxied through nginx at /livekit/.
• No CDN dependencies. livekit-client UMD bundle is vendored into web/; LAN devices may have no internet access.
• Transcripts flow over the data channel. The agent publishes {type: "transcript", role, text} JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as {type: "set_voice", voice} on topic "voice-control".
• Models come from OpenWebUI. The LLM endpoint is OpenWebUI's OpenAI-compatible API at LLM_BASE_URL (default http://192.168.86.2:3005/api); chat goes to /api/chat/completions and the model list comes from GET {LLM_BASE_URL}/models. Plain /v1/* routes are NOT routed on that instance (they serve the SPA). Pipe models (opencode_go_messages.*, go-chat.*, go-responses.*) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. gemma-4-e4b, qwen2.5-vl-3b, qwen3.8-27b) work.
• Reasoning models may hide their answer. Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with max_completion_tokens=1000 and a HopeLLM subclass that wraps the LLM stream, falling back to reasoning_content if content is empty.
• Azure TTS uses SSML, not JSON. The REST endpoint requires Content-Type: application/ssml+xml. The LiveKit Azure plugin handles this internally.
• Voice changes are live. The web UI sends a data message to the agent; the agent calls tts.update_options(voice=...) without restarting. The agent filters data messages by topic ("voice-control") before processing.
• Model changes are live too. The agent fetches the chat-capable model list (HopeLLM filters out embedding/reranker ids) and publishes {type: "models", models, selected}; the UI's settings sheet renders a picker and sends {type: "set_model", model}, which calls HopeLLM.update_options(model=...) (mutates self._opts.model) and acks with {type: "model_changed", model}. The choice persists in the browser's localStorage.
• All supervisord programs log to /dev/stdout so docker compose logs -f voice shows everything. Individual process logs are no longer written to files.

Git

Commit frequently. Conventional commits (feat:, fix:, chore:).