feat: switch LLM to OpenWebUI with live model selection
Rebuild & Deploy / rebuild (push) Successful in 3m18s

- Point the agent at OpenWebUI's OpenAI-compatible API (LLM_BASE_URL,
  LLM_API_KEY, LLM_MODEL) with GEMMA_* env fallbacks
- Add list_available_models() fetching {base_url}/models, filtering
  non-chat ids (embedding/reranker)
- Rename GemmaLLM -> HopeLLM with update_options(model=...) for runtime
  model switching (mutates _opts.model) and instance-based compaction
- Data channel: agent publishes {type: models} after connect and on
  get_models; set_model switches live and acks model_changed
- Web UI: Brain section in the settings sheet with a model picker that
  persists the choice in localStorage
- Update tests, CI workflow env, and docs for the new endpoint
This commit is contained in:
snowspeeder committed 2026-08-29 12:09:03 -04:00
1 parent 024c42ffb1
commit 360e3276b0
14 files changed
+334 -76

No files matched your search

+9 -6
View File
@@ -1,20 +1,20 @@
# Voice — Real-time Voice Assistant
A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with a local Gemma LLM for reasoning.
A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.
## Architecture
One Docker container runs three processes via supervisord:
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
▲ │
└────────── audio ◄────────────────────────────┘
Azure STT → Gemma LLM → Azure TTS
Azure STT → LLM (OpenWebUI) → Azure TTS
```
## Quick Start
@@ -50,8 +50,9 @@ Key variables:
- `AZURE_SPEECH_KEY` — Azure Speech resource key
- `AZURE_SPEECH_REGION` — default `eastus`
- `AZURE_TTS_VOICE` — default voice (`en-US-Ava:DragonHDOmniLatestNeural`)
- `GEMMA_BASE_URL` — LLM endpoint (default `http://192.168.86.2:8023/v1`)
- `GEMMA_MODEL` — model name (default `gemma-4-e4b`)
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
- `LLM_MODEL` — default model (default `gemma-4-e4b`; switchable live from the web UI)
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
## Voice Selection
@@ -128,9 +129,11 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen3.8-27b`) work.
- **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
## Git