From d3f9f2c4edb3d289f118109b1165a71d1d8b4833 Mon Sep 17 00:00:00 2001 From: Shane Date: Sat, 22 Aug 2026 15:21:59 -0400 Subject: [PATCH] =?UTF-8?q?feat:=20implement=20full=20UPDATE.md=20review?= =?UTF-8?q?=20=E2=80=94=20critical=20fixes,=20UI=20upgrade,=20infra=20hard?= =?UTF-8?q?ening?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical frontend bugs: - Add TrackSubscribed/attach() for agent audio playback - Fix decodeToString TypeError with TextDecoder - XSS fix: innerHTML -> textContent in addMessage - Fresh token on reconnect retry Agent fixes: - GemmaLLM subclass with reasoning_content fallback wrapper - Disable Gemma 4 thinking mode via chat_template_kwargs (6.8s -> 0.5s) - Remove duplicate session-level LLM - Replace global _active_session with closure-based handler - asyncio.create_task instead of deprecated get_event_loop - Explicit silero VAD, topic filter on voice-control Infra: - supervisord: all programs log to /dev/stdout - Dockerfile: uv sync --frozen with committed uv.lock - nginx config moved to real file, token_server.py no longer served - entrypoint.sh: cert persisted, only regenerated on IP change - compose: healthcheck + cert volume - token_server: CORS removed, room pinned to voice-room UI upgrade: - Orb UI with state machine (idle/connecting/listening/thinking/speaking) - Streaming transcripts via lk.transcription text streams - Barge-in hint, thinking chip, audio visualizer - Glassmorphism, chat bubbles, settings sheet, light mode - PWA manifest, favicon, wake-lock, safe-area insets - localStorage conversation history Docs: AGENTS.md drift fixed --- .gitignore | 2 +- AGENTS.md | 26 +- Dockerfile | 23 +- UPDATE.md | 252 ++++ agent/agent.py | 153 ++- agent/pyproject.toml | 1 - agent/uv.lock | 3040 ++++++++++++++++++++++++++++++++++++++++++ docker-compose.yml | 7 + entrypoint.sh | 27 +- nginx.conf | 29 + supervisord.conf | 22 +- web/app.js | 539 +++++++- web/favicon.svg | 9 + web/index.html | 58 +- web/manifest.json | 17 + web/style.css | 468 ++++++- web/token_server.py | 10 +- 17 files changed, 4441 insertions(+), 242 deletions(-) create mode 100644 UPDATE.md create mode 100644 agent/uv.lock create mode 100644 nginx.conf create mode 100644 web/favicon.svg create mode 100644 web/manifest.json diff --git a/.gitignore b/.gitignore index 18e30b5..c2043b1 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,6 @@ __pycache__/ *.pyc .venv/ -uv.lock node_modules/ *.log +certs/ diff --git a/AGENTS.md b/AGENTS.md index d896425..7b2b33b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,7 +37,7 @@ LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp` | Port | Protocol | Service | Access | |-------|----------|----------------------|--------------| -| 7880 | TCP | LiveKit HTTP/WS | container (proxied via 8090/livekit) | +| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) | | 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN | | 7882 | UDP | LiveKit RTC media (muxed) | LAN | | 8090 | TCP | Web frontend (HTTPS) | LAN | @@ -78,8 +78,11 @@ The LLM is instructed to: # Verify the container is running docker compose ps -# Check agent logs -docker compose logs -f agent +# Check all logs (single service: "voice") +docker compose logs -f voice + +# Tail individual process logs via supervisord stdout +docker exec voice tail -f /dev/stdout # Test TTS directly (outside the agent) curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ @@ -100,14 +103,18 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ ├── Dockerfile ← multi-stage build ├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start ├── livekit.yaml ← LiveKit server config +├── nginx.conf ← HTTPS UI + /livekit/ WS proxy ├── supervisord.conf ← process manager (livekit, agent, nginx, token-server) +├── certs/ ← persisted self-signed cert (volume mount) ├── agent/ │ ├── agent.py ← LiveKit Agents voice pipeline -│ └── pyproject.toml ← Python deps (uv) +│ ├── pyproject.toml ← Python deps (uv) +│ └── uv.lock ← locked dependency versions └── web/ ├── index.html ← single-page voice UI ├── app.js ← LiveKit client logic - ├── token_server.py ← signs JWTs + roomConfig claim (agent dispatch) + ├── manifest.json ← PWA manifest + ├── favicon.svg ← site icon ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN) └── style.css ← minimal dark theme ``` @@ -116,14 +123,15 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ - **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose. - **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui. -- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. +- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names. - **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups. -- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content). +- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The cert is persisted in `./certs/` (mounted as a volume) and only regenerated when the LAN IP changes, not on every container start. The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content). - **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access. - **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control". -- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty. +- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty. - **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally. -- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. +- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing. +- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files. ## Git diff --git a/Dockerfile b/Dockerfile index d999fbc..96baf56 100644 --- a/Dockerfile +++ b/Dockerfile @@ -14,13 +14,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && rm -rf /var/lib/apt/lists/* COPY agent/pyproject.toml ./agent/ -RUN cd /app/agent && \ - uv venv .venv && \ - uv pip install --python .venv/bin/python \ - "livekit-agents[mcp]~=1.7" \ - "livekit-plugins-azure~=1.7" \ - "livekit-plugins-openai~=1.7" \ - "python-dotenv" +COPY agent/uv.lock ./agent/ +RUN cd /app/agent && uv sync --frozen # ── Stage 2: Runtime (use same base for Python compat) ───────────────────── FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim AS runtime @@ -48,7 +43,12 @@ COPY agent/agent.py /opt/voice-agent/agent.py COPY agent/web_mcp.py /opt/voice-agent/web_mcp.py # Copy web frontend + token endpoint -COPY web/ /var/www/voice/ +COPY web/index.html /var/www/voice/ +COPY web/app.js /var/www/voice/ +COPY web/style.css /var/www/voice/ +COPY web/livekit-client.umd.js /var/www/voice/ +COPY web/manifest.json /var/www/voice/ +COPY web/favicon.svg /var/www/voice/ COPY web/token_server.py /opt/voice/token_server.py # Config files @@ -59,11 +59,10 @@ COPY supervisord.conf /etc/supervisor/conf.d/voice.conf # Browsers require a secure context (HTTPS or localhost) for microphone access. # The self-signed cert (with the LAN IP in the SAN) is generated at container # start by entrypoint.sh. +COPY nginx.conf /etc/nginx/sites-available/voice RUN rm -f /etc/nginx/sites-enabled/default \ - && mkdir -p /etc/voice/certs \ - && printf 'server {\n listen 8090 ssl;\n root /var/www/voice;\n index index.html;\n ssl_certificate /etc/voice/certs/cert.pem;\n ssl_certificate_key /etc/voice/certs/key.pem;\n location /token {\n proxy_pass http://127.0.0.1:8091/token;\n proxy_set_header Content-Type application/json;\n }\n location = /livekit {\n return 301 /livekit/;\n }\n location /livekit/ {\n proxy_pass http://127.0.0.1:7880/;\n proxy_http_version 1.1;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection "upgrade";\n proxy_set_header Host $host;\n proxy_read_timeout 3600s;\n proxy_send_timeout 3600s;\n }\n location / {\n try_files $uri $uri/ =404;\n }\n}\n' \ - > /etc/nginx/sites-available/voice \ - && ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice + && ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice \ + && mkdir -p /etc/voice/certs # Create non-root user for agent RUN useradd -m -s /bin/bash voiceuser || true diff --git a/UPDATE.md b/UPDATE.md new file mode 100644 index 0000000..edd4ecc --- /dev/null +++ b/UPDATE.md @@ -0,0 +1,252 @@ +# UPDATE — Deep Review: Why the Voice Assistant Doesn't Talk Back + +Date: 2026-08-22 +Scope: full code review of the LiveKit + Azure Speech + Gemma voice stack, root-cause analysis of the "mic moves but no transcript / no speech" failure, plus hardening and product/UX recommendations. + +This is a spec only. No code changes have been made. + +--- + +## 1. Executive Summary + +The backend is essentially fine. I verified inside the built image (`voice-voice:latest`) that `agent.py` imports cleanly against livekit-agents 1.7.0, the full `AgentSession` (Azure STT + Azure TTS + local silero VAD + local turn detector) constructs without error, the Gemma endpoint at `192.168.86.2:8023` answers chat and tool calls, and the local EOT/VAD models are bundled in the wheel (no runtime downloads needed). + +**The product is broken by two front-end bugs in `web/app.js`. Both are confirmed, both are fatal, and together they produce exactly the reported symptoms** (connects, mic meter moves, but no transcript and no audio reply): + +1. **The agent's audio is never played.** `app.js` never handles `RoomEvent.TrackSubscribed` and never calls `track.attach()`. With `autoSubscribe: true` the browser *receives* the agent's TTS track, but the livekit-client SDK does not auto-play audio — you must attach it to a media element. The agent may be speaking perfectly; the user will never hear a single sample. + +2. **Transcripts are silently discarded.** `app.js:83` calls `payload.decodeToString()`. In the vendored livekit-client 2.13.0 UMD bundle, the `DataReceived` payload is a plain `Uint8Array` — there is no `decodeToString` method anywhere in the bundle (verified: zero grep hits). The call throws `TypeError` on every data packet, and the surrounding `try/catch` swallows it, so nothing ever renders. + +Fix those two lines-of-code-level issues and the product should work end to end. + +--- + +## 2. Root Causes — Confirmed Bugs (fix these first) + +### 2.1 No audio playback path (web/app.js) — CRITICAL + +**Symptom:** agent never "speaks back." + +**Evidence:** no `TrackSubscribed` handler and no `attach()` call anywhere in `app.js`; no `