snowspeeder/hope-voice-api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement full UPDATE.md review — critical fixes, UI upgrade, infra hardening

Browse Source 
Critical frontend bugs:
- Add TrackSubscribed/attach() for agent audio playback
- Fix decodeToString TypeError with TextDecoder
- XSS fix: innerHTML -> textContent in addMessage
- Fresh token on reconnect retry

Agent fixes:
- GemmaLLM subclass with reasoning_content fallback wrapper
- Disable Gemma 4 thinking mode via chat_template_kwargs (6.8s -> 0.5s)
- Remove duplicate session-level LLM
- Replace global _active_session with closure-based handler
- asyncio.create_task instead of deprecated get_event_loop
- Explicit silero VAD, topic filter on voice-control

Infra:
- supervisord: all programs log to /dev/stdout
- Dockerfile: uv sync --frozen with committed uv.lock
- nginx config moved to real file, token_server.py no longer served
- entrypoint.sh: cert persisted, only regenerated on IP change
- compose: healthcheck + cert volume
- token_server: CORS removed, room pinned to voice-room

UI upgrade:
- Orb UI with state machine (idle/connecting/listening/thinking/speaking)
- Streaming transcripts via lk.transcription text streams
- Barge-in hint, thinking chip, audio visualizer
- Glassmorphism, chat bubbles, settings sheet, light mode
- PWA manifest, favicon, wake-lock, safe-area insets
- localStorage conversation history

Docs: AGENTS.md drift fixed
This commit is contained in:
Shane
2026-08-22 15:21:59 -04:00
parent d1eeb01f3d
commit d3f9f2c4ed
17 changed files with 4441 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+17 -9
View File
@@ -37,7 +37,7 @@ LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
| Port | Protocol | Service | Access |
|-------|----------|----------------------|--------------|
| 7880 | TCP | LiveKit HTTP/WS | container (proxied via 8090/livekit) |
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
| 8090 | TCP | Web frontend (HTTPS) | LAN |
@@ -78,8 +78,11 @@ The LLM is instructed to:
# Verify the container is running
docker compose ps
# Check agent logs
docker compose logs -f agent
# Check all logs (single service: "voice")
docker compose logs -f voice
# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout
# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
@@ -100,14 +103,18 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
├── Dockerfile ← multi-stage build
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml ← LiveKit server config
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
├── certs/ ← persisted self-signed cert (volume mount)
├── agent/
│ ├── agent.py ← LiveKit Agents voice pipeline
│ └── pyproject.toml ← Python deps (uv)
│ ├── pyproject.toml ← Python deps (uv)
│ └── uv.lock ← locked dependency versions
└── web/
├── index.html ← single-page voice UI
├── app.js ← LiveKit client logic
├── token_server.py ← signs JWTs + roomConfig claim (agent dispatch)
├── manifest.json ← PWA manifest
├── favicon.svg ← site icon
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
└── style.css ← minimal dark theme
```
@@ -116,14 +123,15 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The cert is persisted in `./certs/` (mounted as a volume) and only regenerated when the LAN IP changes, not on every container start. The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty.
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
## Git