diff --git a/.gitignore b/.gitignore index 18e30b5..c2043b1 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,6 @@ __pycache__/ *.pyc .venv/ -uv.lock node_modules/ *.log +certs/ diff --git a/AGENTS.md b/AGENTS.md index d896425..7b2b33b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,7 +37,7 @@ LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp` | Port | Protocol | Service | Access | |-------|----------|----------------------|--------------| -| 7880 | TCP | LiveKit HTTP/WS | container (proxied via 8090/livekit) | +| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) | | 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN | | 7882 | UDP | LiveKit RTC media (muxed) | LAN | | 8090 | TCP | Web frontend (HTTPS) | LAN | @@ -78,8 +78,11 @@ The LLM is instructed to: # Verify the container is running docker compose ps -# Check agent logs -docker compose logs -f agent +# Check all logs (single service: "voice") +docker compose logs -f voice + +# Tail individual process logs via supervisord stdout +docker exec voice tail -f /dev/stdout # Test TTS directly (outside the agent) curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ @@ -100,14 +103,18 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ ├── Dockerfile ← multi-stage build ├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start ├── livekit.yaml ← LiveKit server config +├── nginx.conf ← HTTPS UI + /livekit/ WS proxy ├── supervisord.conf ← process manager (livekit, agent, nginx, token-server) +├── certs/ ← persisted self-signed cert (volume mount) ├── agent/ │ ├── agent.py ← LiveKit Agents voice pipeline -│ └── pyproject.toml ← Python deps (uv) +│ ├── pyproject.toml ← Python deps (uv) +│ └── uv.lock ← locked dependency versions └── web/ ├── index.html ← single-page voice UI ├── app.js ← LiveKit client logic - ├── token_server.py ← signs JWTs + roomConfig claim (agent dispatch) + ├── manifest.json ← PWA manifest + ├── favicon.svg ← site icon ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN) └── style.css ← minimal dark theme ``` @@ -116,14 +123,15 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ - **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose. - **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui. -- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. +- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names. - **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups. -- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content). +- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The cert is persisted in `./certs/` (mounted as a volume) and only regenerated when the LAN IP changes, not on every container start. The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content). - **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access. - **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control". -- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty. +- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty. - **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally. -- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. +- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing. +- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files. ## Git diff --git a/Dockerfile b/Dockerfile index d999fbc..96baf56 100644 --- a/Dockerfile +++ b/Dockerfile @@ -14,13 +14,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && rm -rf /var/lib/apt/lists/* COPY agent/pyproject.toml ./agent/ -RUN cd /app/agent && \ - uv venv .venv && \ - uv pip install --python .venv/bin/python \ - "livekit-agents[mcp]~=1.7" \ - "livekit-plugins-azure~=1.7" \ - "livekit-plugins-openai~=1.7" \ - "python-dotenv" +COPY agent/uv.lock ./agent/ +RUN cd /app/agent && uv sync --frozen # ── Stage 2: Runtime (use same base for Python compat) ───────────────────── FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim AS runtime @@ -48,7 +43,12 @@ COPY agent/agent.py /opt/voice-agent/agent.py COPY agent/web_mcp.py /opt/voice-agent/web_mcp.py # Copy web frontend + token endpoint -COPY web/ /var/www/voice/ +COPY web/index.html /var/www/voice/ +COPY web/app.js /var/www/voice/ +COPY web/style.css /var/www/voice/ +COPY web/livekit-client.umd.js /var/www/voice/ +COPY web/manifest.json /var/www/voice/ +COPY web/favicon.svg /var/www/voice/ COPY web/token_server.py /opt/voice/token_server.py # Config files @@ -59,11 +59,10 @@ COPY supervisord.conf /etc/supervisor/conf.d/voice.conf # Browsers require a secure context (HTTPS or localhost) for microphone access. # The self-signed cert (with the LAN IP in the SAN) is generated at container # start by entrypoint.sh. +COPY nginx.conf /etc/nginx/sites-available/voice RUN rm -f /etc/nginx/sites-enabled/default \ - && mkdir -p /etc/voice/certs \ - && printf 'server {\n listen 8090 ssl;\n root /var/www/voice;\n index index.html;\n ssl_certificate /etc/voice/certs/cert.pem;\n ssl_certificate_key /etc/voice/certs/key.pem;\n location /token {\n proxy_pass http://127.0.0.1:8091/token;\n proxy_set_header Content-Type application/json;\n }\n location = /livekit {\n return 301 /livekit/;\n }\n location /livekit/ {\n proxy_pass http://127.0.0.1:7880/;\n proxy_http_version 1.1;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection "upgrade";\n proxy_set_header Host $host;\n proxy_read_timeout 3600s;\n proxy_send_timeout 3600s;\n }\n location / {\n try_files $uri $uri/ =404;\n }\n}\n' \ - > /etc/nginx/sites-available/voice \ - && ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice + && ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice \ + && mkdir -p /etc/voice/certs # Create non-root user for agent RUN useradd -m -s /bin/bash voiceuser || true diff --git a/UPDATE.md b/UPDATE.md new file mode 100644 index 0000000..edd4ecc --- /dev/null +++ b/UPDATE.md @@ -0,0 +1,252 @@ +# UPDATE — Deep Review: Why the Voice Assistant Doesn't Talk Back + +Date: 2026-08-22 +Scope: full code review of the LiveKit + Azure Speech + Gemma voice stack, root-cause analysis of the "mic moves but no transcript / no speech" failure, plus hardening and product/UX recommendations. + +This is a spec only. No code changes have been made. + +--- + +## 1. Executive Summary + +The backend is essentially fine. I verified inside the built image (`voice-voice:latest`) that `agent.py` imports cleanly against livekit-agents 1.7.0, the full `AgentSession` (Azure STT + Azure TTS + local silero VAD + local turn detector) constructs without error, the Gemma endpoint at `192.168.86.2:8023` answers chat and tool calls, and the local EOT/VAD models are bundled in the wheel (no runtime downloads needed). + +**The product is broken by two front-end bugs in `web/app.js`. Both are confirmed, both are fatal, and together they produce exactly the reported symptoms** (connects, mic meter moves, but no transcript and no audio reply): + +1. **The agent's audio is never played.** `app.js` never handles `RoomEvent.TrackSubscribed` and never calls `track.attach()`. With `autoSubscribe: true` the browser *receives* the agent's TTS track, but the livekit-client SDK does not auto-play audio — you must attach it to a media element. The agent may be speaking perfectly; the user will never hear a single sample. + +2. **Transcripts are silently discarded.** `app.js:83` calls `payload.decodeToString()`. In the vendored livekit-client 2.13.0 UMD bundle, the `DataReceived` payload is a plain `Uint8Array` — there is no `decodeToString` method anywhere in the bundle (verified: zero grep hits). The call throws `TypeError` on every data packet, and the surrounding `try/catch` swallows it, so nothing ever renders. + +Fix those two lines-of-code-level issues and the product should work end to end. + +--- + +## 2. Root Causes — Confirmed Bugs (fix these first) + +### 2.1 No audio playback path (web/app.js) — CRITICAL + +**Symptom:** agent never "speaks back." + +**Evidence:** no `TrackSubscribed` handler and no `attach()` call anywhere in `app.js`; no `