snowspeeder
a4ae96f01b
Rebuild & Deploy / rebuild (push) Successful in 54s
- Default LLM_MODEL to qwen2.5-vl-3b across agent, task worker, tests, compose, workflow, and docs - Cap min/max length-style constraints in tool schemas to 1000 before LLM requests; llama.cpp rejects grammar repetitions >= 2000 and the memory MCP's update_note body maxLength (200000) broke qwen2.5-vl-3b with 'Failed to initialize samplers: failed to parse grammar'
142 lines
8.6 KiB
Markdown
142 lines
8.6 KiB
Markdown
# Voice — Real-time Voice Assistant
|
|
|
|
A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.
|
|
|
|
## Architecture
|
|
|
|
One Docker container runs three processes via supervisord:
|
|
|
|
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
|
|
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
|
|
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
|
|
|
|
```
|
|
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
|
|
▲ │
|
|
└────────── audio ◄────────────────────────────┘
|
|
Azure STT → LLM (OpenWebUI) → Azure TTS
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
|
|
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)
|
|
|
|
# 2. Build and start
|
|
docker compose up --build -d
|
|
|
|
# 3. Open the web UI (self-signed cert: accept the browser warning once)
|
|
# https://<host-ip>:9178
|
|
```
|
|
|
|
LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
|
|
(WebRTC media), `7881/tcp` (media TCP fallback).
|
|
|
|
## Ports
|
|
|
|
| Port | Protocol | Service | Access |
|
|
|-------|----------|----------------------|--------------|
|
|
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
|
|
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
|
|
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
|
|
| 9178 | TCP | Web frontend (HTTPS) | LAN |
|
|
|
|
## Configuration
|
|
|
|
All config lives in `.env` (git-ignored). See `.env.example` for the full list.
|
|
|
|
Key variables:
|
|
- `AZURE_SPEECH_KEY` — Azure Speech resource key
|
|
- `AZURE_SPEECH_REGION` — default `eastus`
|
|
- `AZURE_TTS_VOICE` — default voice (`en-US-Ava:DragonHDOmniLatestNeural`)
|
|
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
|
|
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
|
|
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
|
|
|
|
## Voice Selection
|
|
|
|
The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.
|
|
|
|
Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices
|
|
|
|
## SSML / Expressive Speech
|
|
|
|
The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as `[appreciative]`, `[whispering]`, and `[Neutral]`; the agent prompt allows these markers while keeping responses short and conversational.
|
|
|
|
## Agent Prompt
|
|
|
|
The LLM is instructed to:
|
|
- Speak as if talking to someone, not writing for them to read
|
|
- Keep responses to 1-3 sentences
|
|
- Use contractions, natural fillers sparingly
|
|
- Never use markdown, lists, or formatting
|
|
- Spell out numbers and abbreviations
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
# Verify the container is running
|
|
docker compose ps
|
|
|
|
# Check all logs (single service: "voice")
|
|
docker compose logs -f voice
|
|
|
|
# Tail individual process logs via supervisord stdout
|
|
docker exec voice tail -f /dev/stdout
|
|
|
|
# Test TTS directly (outside the agent)
|
|
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
|
-H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
|
|
-H "Content-Type: application/ssml+xml" \
|
|
-H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
|
|
--data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3
|
|
```
|
|
|
|
## File Layout
|
|
|
|
```
|
|
~/dev/voice/
|
|
├── AGENTS.md ← you are here
|
|
├── .env.example ← config template (copy to .env)
|
|
├── .gitignore
|
|
├── docker-compose.yml ← single container
|
|
├── Dockerfile ← multi-stage build
|
|
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
|
|
├── livekit.yaml ← LiveKit server config
|
|
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
|
|
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
|
|
├── certs/ ← persisted self-signed cert (volume mount)
|
|
├── agent/
|
|
│ ├── agent.py ← LiveKit Agents voice pipeline
|
|
│ ├── pyproject.toml ← Python deps (uv)
|
|
│ └── uv.lock ← locked dependency versions
|
|
└── web/
|
|
├── index.html ← single-page voice UI
|
|
├── app.js ← LiveKit client logic
|
|
├── manifest.json ← PWA manifest
|
|
├── favicon.svg ← site icon
|
|
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
|
|
└── style.css ← minimal dark theme
|
|
```
|
|
|
|
## Conventions
|
|
|
|
- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
|
|
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
|
|
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
|
|
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
|
|
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
|
|
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
|
|
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
|
|
- **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen2.5-vl-3b`, `qwen3.8-27b`) work.
|
|
- **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
|
|
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
|
|
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
|
|
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
|
|
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
|
|
|
|
## Git
|
|
|
|
Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).
|