hope-voice-api/UPDATE.md

UPDATE — Deep Review: Why the Voice Assistant Doesn't Talk Back

Date: 2026-08-22 Scope: full code review of the LiveKit + Azure Speech + Gemma voice stack, root-cause analysis of the "mic moves but no transcript / no speech" failure, plus hardening and product/UX recommendations.

This is a spec only. No code changes have been made.

---

1. Executive Summary

The backend is essentially fine. I verified inside the built image (voice-voice:latest) that agent.py imports cleanly against livekit-agents 1.7.0, the full AgentSession (Azure STT + Azure TTS + local silero VAD + local turn detector) constructs without error, the Gemma endpoint at 192.168.86.2:8023 answers chat and tool calls, and the local EOT/VAD models are bundled in the wheel (no runtime downloads needed).

The product is broken by two front-end bugs in web/app.js. Both are confirmed, both are fatal, and together they produce exactly the reported symptoms (connects, mic meter moves, but no transcript and no audio reply):

1. The agent's audio is never played. app.js never handles RoomEvent.TrackSubscribed and never calls track.attach(). With autoSubscribe: true the browser receives the agent's TTS track, but the livekit-client SDK does not auto-play audio — you must attach it to a media element. The agent may be speaking perfectly; the user will never hear a single sample.

2. Transcripts are silently discarded. app.js:83 calls payload.decodeToString(). In the vendored livekit-client 2.13.0 UMD bundle, the DataReceived payload is a plain Uint8Array — there is no decodeToString method anywhere in the bundle (verified: zero grep hits). The call throws TypeError on every data packet, and the surrounding try/catch swallows it, so nothing ever renders.

Fix those two lines-of-code-level issues and the product should work end to end.

---

2. Root Causes — Confirmed Bugs (fix these first)

2.1 No audio playback path (web/app.js) — CRITICAL

Symptom: agent never "speaks back."

Evidence: no TrackSubscribed handler and no attach() call anywhere in app.js; no <audio> element in index.html.

Fix (in createRoom):

newRoom.on(RoomEvent.TrackSubscribed, (track, publication, participant) => {
    if (track.kind === LivekitClient.Track.Kind.Audio) {
        const el = track.attach();      // creates an <audio> element, autoplays
        el.id = `audio-${participant.identity}`;
        document.body.appendChild(el);
    }
});

newRoom.on(RoomEvent.TrackUnsubscribed, (track) => {
    track.detach().forEach((el) => el.remove());
});

Also required — browser autoplay policy: even attached audio can be blocked if the context isn't "unlocked." The Start button click is a user gesture, so call await room.startAudio() right after connect() resolves, and handle the recovery case:

newRoom.on(RoomEvent.AudioPlaybackStatusChanged, () => {
    if (!newRoom.canPlaybackAudio) {
        // show a "tap to enable audio" button that calls room.startAudio()
    }
});

This matters most on iOS Safari, which the mic-meter comment in the code says is a target device.

2.2 payload.decodeToString() does not exist (web/app.js:83) — CRITICAL

Symptom: no transcripts appear for either side of the conversation.

Evidence: livekit-client 2.13.0 delivers DataReceived payloads as Uint8Array; decodeToString has zero occurrences in the vendored bundle. The TypeError is swallowed by the catch (e) { /* ignore */ } block.

Fix:

newRoom.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
    try {
        const msg = JSON.parse(new TextDecoder().decode(payload));
        if (msg.type === "transcript") addMessage(msg.role || "agent", msg.text);
    } catch (e) {
        console.warn("bad data packet", e);   // don't swallow silently
    }
});

Better long-term fix: drop the custom transcript data channel entirely. livekit-agents 1.x already publishes live transcriptions (including streaming partials) over text streams on the lk.transcription topic. The UI can consume them with:

room.registerTextStreamHandler("lk.transcription", async (reader, participantInfo) => {
    const text = await reader.readAll();
    // reader.info.attributes["lk.transcribed_track_id"] distinguishes user vs agent
});

That removes ~40 lines of custom code on each side and gives you word-by-word streaming transcripts for free (much better UX than the current end-of-utterance-only rendering).

---

3. Secondary Issues (likely to bite next, in priority order)

3.1 reasoning_content handling is documented but not implemented (agent/agent.py)

AGENTS.md claims: "The agent handles this by using max_tokens=1000 and falling back to reasoning_content if content is empty." Neither exists in the current code. This is a doc/code regression.

I verified against the live Gemma endpoint: with tools attached, the model returns "content": "" plus a populated "reasoning_content" field. If Gemma ever finishes a turn with all its tokens in reasoning_content and an empty content (which reasoning models do, especially with small max_tokens), the agent will speak nothing — a second, intermittent "silent agent" failure mode that will look identical to bug 2.1 from the user's chair.

Fix: restore the documented behavior — either pass max_tokens / a reasoning_effort-suppressing option through openai.LLM(...), or wrap the LLM node to fall back to reasoning_content when content is empty. Then update AGENTS.md to match whichever is implemented.

3.2 Duplicate, conflicting LLM instances (agent/agent.py)

VoiceAssistant.__init__ builds one openai.LLM(...) and handle_job builds a second identical one for the session. Agent-level components override session-level ones, so the session's llm= is dead weight. Keep exactly one (recommend: agent-level, delete the session one) so future config edits can't silently apply to the ignored copy.

3.3 Global _active_session (agent/agent.py)

• It's never cleared when a session ends, so a voice-change message after a job finishes pokes a dead session.
• If two rooms are ever dispatched, the second clobbers the first and voice changes route to the wrong room.

Fix: make the data handler a closure inside handle_job capturing its own session, and register it per-room. Delete the global.

3.4 asyncio.get_event_loop() is deprecated (agent/agent.py:210-212)

Inside the sync conversation_item_added callback, use asyncio.create_task(...) (a running loop is guaranteed there). get_event_loop() in Python 3.12 emits deprecation warnings and will eventually raise.

3.5 Agent logs are invisible to docker compose logs (supervisord.conf)

All four programs write to files under /var/log/supervisor/ inside the container. docker compose logs -f (which AGENTS.md tells you to use — and even names a nonexistent agent service) shows nothing. This makes every future debugging session harder than it needs to be.

Fix: in each [program:x] block:

stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true

And fix AGENTS.md: it's docker compose logs -f voice (single service), or docker exec voice tail -f /var/log/supervisor/agent.log today.

3.6 Dockerfile ignores agent/pyproject.toml

The build stage copies pyproject.toml and then hand-installs a separate list with uv pip install "livekit-agents[mcp]~=1.7" .... Two sources of truth, no lockfile — dependency drift is guaranteed and rebuilds aren't reproducible. Fix: uv sync against the pyproject with a committed uv.lock, or delete the pyproject dependency list. ~=1.7 also permits silent minor-version jumps on rebuild; pin exact versions in the lock.

3.7 Self-signed cert regenerated on every container start (entrypoint.sh)

Every restart mints a new key/cert, so every device re-prompts the scary TLS warning after each deploy. Fix: persist /etc/voice/certs in a volume and only generate when missing (or when the LAN IP changed).

3.8 Retry path reuses a stale token assumption (web/app.js)

Minor: the reconnect path reuses the same JWT (fine — 1 h expiry) but if a user leaves the tab open past expiry, Start fails opaquely. Fetch a fresh token on every attempt; they're free.

---

4. General Code Review

agent/agent.py

• Good: clean separation of config, prompt quality is genuinely good for spoken output, MCP toolset wiring is correct for 1.7 (MCPToolset in tools= constructs fine — verified in-container), close_on_disconnect=False rationale is sound and well-commented.
• session.start(...) before ctx.connect() matches the 1.x documented pattern — correct, leave it.
• The user_state_changed / user_input_transcribed diagnostic logging is great; keep it, but see 3.5 or you'll never see it.
• No explicit vad= on the session: 1.7 silently defaults to the bundled local silero VAD (verified: livekit.local_inference native lib present, constructs offline). Works, but make it explicit — vad=inference.VAD(model="silero") — so a future SDK default change can't surprise you.
• interruption={"mode": "vad"} is correct for self-hosted (the SDK source confirms adaptive requires the cloud gateway and is disabled outside dev/hosted mode anyway).
• Voice-change handler doesn't filter on the voice-control topic; it parses every packet. Harmless today, sloppy tomorrow — check packet.topic.

web/app.js

• Besides the two critical bugs: the agent-detection logic (isAgent, poll loop in waitForAgent) is reasonable, though RoomEvent.ParticipantConnected + initial scan makes the 250 ms poll redundant — a promise resolved from the event handler would be tidier.
• The mic meter builds its own AudioContext + analyser; livekit-client already exposes LocalAudioTrack events and createAudioAnalyser() helper. Not wrong, just duplicative.
• stopBtn handler doesn't clear attached audio elements (moot until 2.1 is fixed — then it leaks one <audio> element per session).
• addMessage interpolates transcript text with innerHTML — XSS via speech: whatever the STT/LLM emits lands in the DOM unescaped. Use textContent for the text node.

web/token_server.py

• JWT construction is correct (HS256, roomConfig.agents[].agentName in protojson camelCase — the server accepts this).
• Access-Control-Allow-Origin: * on an endpoint that mints publish-capable tokens: any page a LAN user visits can silently get a token and join the room. Same-origin only — drop the CORS headers entirely (the UI is same-origin through nginx).
• No room-name validation: the client can request any room. Pin it server-side to voice-room or a validated allowlist.
• Single-threaded HTTPServer is fine at this scale; note it blocks during slow clients.

Infra (Dockerfile / nginx / compose / livekit.yaml)

• nginx: WS proxy for /livekit/ is correct (prefix strip + Upgrade headers). The embedded one-line printf config in the Dockerfile is unreadable and un-diffable — move it to a real nginx.conf file that gets COPYed.
• nginx location /token sets proxy_set_header Content-Type ... — that's a no-op/incorrect use (Content-Type is a client request header); remove.
• COPY web/ /var/www/voice/ publishes token_server.py source at https://host:8090/token_server.py. No secrets in it, but don't serve server code; exclude it (.dockerignore-style or copy files explicitly).
• devkey: devsecret in livekit.yaml and compose defaults: anyone on the LAN who reads this repo can mint admin tokens. For a LAN toy it's acceptable; generate a random secret at first boot if this ever leaves the bench.
• compose has no healthcheck; a supervisord child crash-looping (the exact failure class you just debugged) leaves the container "Up" and green. Add a healthcheck that curls https://localhost:8090/ and hits LiveKit's / on 7880.
• supervisord.conf has nodaemon=false while CMD runs supervisord -n — the flag wins, but make the file say what it does.
• AGENTS.md file-layout section is accurate; the Testing section's docker compose logs -f agent is wrong (see 3.5).

Doc drift (AGENTS.md)

• "max_tokens=1000 / reasoning_content fallback" — documented, not implemented (3.1).
• "Interruption mode must be vad" — implemented, consistent. Good.
• Ports table says 8090 is "UI + signaling" — correct via the nginx proxy; 7880 needn't be exposed at all with host networking (it is, via EXPOSE — cosmetic).

---

5. Verification Performed (evidence)

All checks run against the built image voice-voice:latest and the live LAN services:

Check	Result
import agent in-container	OK — no API mismatches against livekit-agents 1.7.0
VoiceAssistant() + full AgentSession(...) construction	OK — MCP toolsets, TurnHandlingOptions accepted
Default VAD	Local silero via livekit.local_inference native lib (34 MB, bundled) — constructs with --network none
Turn detector	Falls back to local v1-mini when not on LiveKit Cloud; degrades to endpointing delay if unavailable — no cloud dependency
Azure STT capabilities	streaming: True — no VAD strictly required for transcription
azure.TTS.update_options	Exists — live voice switching is valid
Gemma /v1/chat/completions	Responds; tool calls work; reasoning_content confirmed present with empty content
decodeToString in vendored livekit-client 2.13.0	0 occurrences — confirmed frontend bug
TrackSubscribed/attach() in app.js	Absent — confirmed frontend bug
Container state	Not currently running (no runtime logs available)

---

6. Recommended Fix Order

1. app.js: add TrackSubscribed → track.attach() + startAudio() + AudioPlaybackStatusChanged fallback (2.1).
2. app.js: TextDecoder fix for DataReceived (2.2) — or migrate both sides to lk.transcription text streams.
3. agent.py: reasoning_content/max_tokens handling (3.1).
4. supervisord → stdout logging + AGENTS.md log command fix (3.5).
5. app.js: innerHTML → textContent (XSS).
6. Remaining review items as time allows (3.2-3.8, section 4).

Test plan after 1+2:

• docker compose up --build -d, wait ~15 s for worker registration.
• Browser → Start → say "hello". Expect: your words render as a user message (streamed if using text streams), agent audio plays within ~1-2 s, agent message renders.
• Watch docker exec voice tail -f /var/log/supervisor/agent.log: confirm user state -> speaking, STT (final): ..., then TTS synthesis.
• Change voice mid-session; next reply uses the new voice.
• Kill/reload the tab, rejoin: agent should still respond (close_on_disconnect=False path).
• iPhone Safari: verify the "tap to enable audio" fallback fires and works.

---

7. Product / Theming & Visual Upgrade Proposals

The current UI is a functional dev console. To make it feel like a product:

Core interaction model

• Replace the transcript-first layout with a voice-first "orb" UI. A large centered circle that breathes in idle, pulses with your mic level while listening (you already compute the level — drive the orb instead of a "mic: 42%" debug string), and ripples with the agent's output level while speaking. This one element communicates listening/thinking/speaking state instantly and is the signature visual of every good voice product.
• Explicit state machine surfaced in the UI: idle → connecting → listening → thinking → speaking, driven by the agent's lk.agent.state participant attribute (the SDK publishes it — free to consume). Replace the freeform status strings.
• Streaming transcripts with partials (via lk.transcription): words appear as you say them, greyed until final. Massive perceived-latency win.
• Tool-use indicator: when Gemma calls web_search, show a subtle "searching the web..." chip. Right now web lookups are silent 5-10 s gaps that feel like crashes.
• Barge-in affordance: show a small "tap or speak to interrupt" hint while the agent is speaking.

Visual design

• Keep the dark theme, but commit to it: one accent hue (the blue→violet gradient is good), true-black #09090b base, glassmorphism panels (backdrop-filter: blur) for the transcript drawer, Inter/Geist variable font instead of system stack.
• Chat bubbles: user right-aligned, agent left-aligned with a small avatar dot in the accent gradient; timestamps on hover. Drop the border-left style.
• Motion: orb idle animation (4 s ease breathing), message enter transitions (~120 ms fade/slide), respect prefers-reduced-motion.
• The voice picker becomes a proper settings sheet: voice cards with a "preview" play button (one cached Azure TTS sample per voice), plus speaking-rate and style (mstts:express-as) controls — the SSML plumbing already exists.
• Light-mode variant via prefers-color-scheme.

Mobile / platform

• PWA manifest + icons so it installs to the home screen (the HTTPS requirement is already met); theme-color meta; safe-area insets.
• Wake-lock (navigator.wakeLock) during a session so the phone doesn't sleep mid-conversation.
• Favicon (currently a 404 on every load).

Nice-to-haves that differentiate

• Conversation history persisted to localStorage with a "clear" button.
• Latency HUD (dev toggle): STT-final→first-audio time per turn — you'll want it while tuning Gemma.
• Push-to-talk mode toggle for noisy rooms (disable VAD interruption, hold spacebar/tap-and-hold the orb).
• Audio visualizer bars on the agent bubble while it speaks (a createAudioAnalyser() on the remote track — the same trick the mic meter already does locally).

---

End of review.