hope-voice-api/UPDATE.md

# UPDATE — Deep Review: Why the Voice Assistant Doesn't Talk Back

Date: 2026-08-22
Scope: full code review of the LiveKit + Azure Speech + Gemma voice stack, root-cause analysis of the "mic moves but no transcript / no speech" failure, plus hardening and product/UX recommendations.

This is a spec only. No code changes have been made.

---

## 1. Executive Summary

The backend is essentially fine. I verified inside the built image (`voice-voice:latest`) that `agent.py` imports cleanly against livekit-agents 1.7.0, the full `AgentSession` (Azure STT + Azure TTS + local silero VAD + local turn detector) constructs without error, the Gemma endpoint at `192.168.86.2:8023` answers chat and tool calls, and the local EOT/VAD models are bundled in the wheel (no runtime downloads needed).

**The product is broken by two front-end bugs in `web/app.js`. Both are confirmed, both are fatal, and together they produce exactly the reported symptoms** (connects, mic meter moves, but no transcript and no audio reply):

1. **The agent's audio is never played.** `app.js` never handles `RoomEvent.TrackSubscribed` and never calls `track.attach()`. With `autoSubscribe: true` the browser *receives* the agent's TTS track, but the livekit-client SDK does not auto-play audio — you must attach it to a media element. The agent may be speaking perfectly; the user will never hear a single sample.

2. **Transcripts are silently discarded.** `app.js:83` calls `payload.decodeToString()`. In the vendored livekit-client 2.13.0 UMD bundle, the `DataReceived` payload is a plain `Uint8Array` — there is no `decodeToString` method anywhere in the bundle (verified: zero grep hits). The call throws `TypeError` on every data packet, and the surrounding `try/catch` swallows it, so nothing ever renders.

Fix those two lines-of-code-level issues and the product should work end to end.

---

## 2. Root Causes — Confirmed Bugs (fix these first)

### 2.1 No audio playback path (web/app.js) — CRITICAL

**Symptom:** agent never "speaks back."

**Evidence:** no `TrackSubscribed` handler and no `attach()` call anywhere in `app.js`; no `<audio>` element in `index.html`.

**Fix (in `createRoom`):**

```js
newRoom.on(RoomEvent.TrackSubscribed, (track, publication, participant) => {
    if (track.kind === LivekitClient.Track.Kind.Audio) {
        const el = track.attach();      // creates an <audio> element, autoplays
        el.id = `audio-${participant.identity}`;
        document.body.appendChild(el);
    }
});

newRoom.on(RoomEvent.TrackUnsubscribed, (track) => {
    track.detach().forEach((el) => el.remove());
});
```

**Also required — browser autoplay policy:** even attached audio can be blocked if the context isn't "unlocked." The Start button click is a user gesture, so call `await room.startAudio()` right after `connect()` resolves, and handle the recovery case:

```js
newRoom.on(RoomEvent.AudioPlaybackStatusChanged, () => {
    if (!newRoom.canPlaybackAudio) {
        // show a "tap to enable audio" button that calls room.startAudio()
    }
});
```

This matters most on iOS Safari, which the mic-meter comment in the code says is a target device.

### 2.2 `payload.decodeToString()` does not exist (web/app.js:83) — CRITICAL

**Symptom:** no transcripts appear for either side of the conversation.

**Evidence:** livekit-client 2.13.0 delivers `DataReceived` payloads as `Uint8Array`; `decodeToString` has zero occurrences in the vendored bundle. The `TypeError` is swallowed by the `catch (e) { /* ignore */ }` block.

**Fix:**

```js
newRoom.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
    try {
        const msg = JSON.parse(new TextDecoder().decode(payload));
        if (msg.type === "transcript") addMessage(msg.role || "agent", msg.text);
    } catch (e) {
        console.warn("bad data packet", e);   // don't swallow silently
    }
});
```

**Better long-term fix:** drop the custom transcript data channel entirely. livekit-agents 1.x already publishes live transcriptions (including *streaming partials*) over text streams on the `lk.transcription` topic. The UI can consume them with:

```js
room.registerTextStreamHandler("lk.transcription", async (reader, participantInfo) => {
    const text = await reader.readAll();
    // reader.info.attributes["lk.transcribed_track_id"] distinguishes user vs agent
});
```

That removes ~40 lines of custom code on each side and gives you word-by-word streaming transcripts for free (much better UX than the current end-of-utterance-only rendering).

---

## 3. Secondary Issues (likely to bite next, in priority order)

### 3.1 `reasoning_content` handling is documented but not implemented (agent/agent.py)

AGENTS.md claims: *"The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty."* **Neither exists in the current code.** This is a doc/code regression.

I verified against the live Gemma endpoint: with tools attached, the model returns `"content": ""` plus a populated `"reasoning_content"` field. If Gemma ever finishes a turn with all its tokens in `reasoning_content` and an empty `content` (which reasoning models do, especially with small `max_tokens`), the agent will speak nothing — a second, intermittent "silent agent" failure mode that will look identical to bug 2.1 from the user's chair.

**Fix:** restore the documented behavior — either pass `max_tokens` / a `reasoning_effort`-suppressing option through `openai.LLM(...)`, or wrap the LLM node to fall back to `reasoning_content` when `content` is empty. Then update AGENTS.md to match whichever is implemented.

### 3.2 Duplicate, conflicting LLM instances (agent/agent.py)

`VoiceAssistant.__init__` builds one `openai.LLM(...)` and `handle_job` builds a *second identical* one for the session. Agent-level components override session-level ones, so the session's `llm=` is dead weight. Keep exactly one (recommend: agent-level, delete the session one) so future config edits can't silently apply to the ignored copy.

### 3.3 Global `_active_session` (agent/agent.py)

- It's never cleared when a session ends, so a voice-change message after a job finishes pokes a dead session.
- If two rooms are ever dispatched, the second clobbers the first and voice changes route to the wrong room.

**Fix:** make the data handler a closure inside `handle_job` capturing its own `session`, and register it per-room. Delete the global.

### 3.4 `asyncio.get_event_loop()` is deprecated (agent/agent.py:210-212)

Inside the sync `conversation_item_added` callback, use `asyncio.create_task(...)` (a running loop is guaranteed there). `get_event_loop()` in Python 3.12 emits deprecation warnings and will eventually raise.

### 3.5 Agent logs are invisible to `docker compose logs` (supervisord.conf)

All four programs write to files under `/var/log/supervisor/` inside the container. `docker compose logs -f` (which AGENTS.md tells you to use — and even names a nonexistent `agent` service) shows nothing. This makes every future debugging session harder than it needs to be.

**Fix:** in each `[program:x]` block:

```ini
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true
```

And fix AGENTS.md: it's `docker compose logs -f voice` (single service), or `docker exec voice tail -f /var/log/supervisor/agent.log` today.

### 3.6 Dockerfile ignores agent/pyproject.toml

The build stage copies `pyproject.toml` and then hand-installs a separate list with `uv pip install "livekit-agents[mcp]~=1.7" ...`. Two sources of truth, no lockfile — dependency drift is guaranteed and rebuilds aren't reproducible. **Fix:** `uv sync` against the pyproject with a committed `uv.lock`, or delete the pyproject dependency list. `~=1.7` also permits silent minor-version jumps on rebuild; pin exact versions in the lock.

### 3.7 Self-signed cert regenerated on every container start (entrypoint.sh)

Every restart mints a new key/cert, so every device re-prompts the scary TLS warning after each deploy. **Fix:** persist `/etc/voice/certs` in a volume and only generate when missing (or when the LAN IP changed).

### 3.8 Retry path reuses a stale token assumption (web/app.js)

Minor: the reconnect path reuses the same JWT (fine — 1 h expiry) but if a user leaves the tab open past expiry, Start fails opaquely. Fetch a fresh token on every attempt; they're free.

---

## 4. General Code Review

### agent/agent.py
- **Good:** clean separation of config, prompt quality is genuinely good for spoken output, MCP toolset wiring is correct for 1.7 (`MCPToolset` in `tools=` constructs fine — verified in-container), `close_on_disconnect=False` rationale is sound and well-commented.
- `session.start(...)` before `ctx.connect()` matches the 1.x documented pattern — correct, leave it.
- The `user_state_changed` / `user_input_transcribed` diagnostic logging is great; keep it, but see 3.5 or you'll never see it.
- No explicit `vad=` on the session: 1.7 silently defaults to the bundled local silero VAD (verified: `livekit.local_inference` native lib present, constructs offline). Works, but make it explicit — `vad=inference.VAD(model="silero")` — so a future SDK default change can't surprise you.
- `interruption={"mode": "vad"}` is correct for self-hosted (the SDK source confirms `adaptive` requires the cloud gateway and is disabled outside dev/hosted mode anyway).
- Voice-change handler doesn't filter on the `voice-control` topic; it parses every packet. Harmless today, sloppy tomorrow — check `packet.topic`.

### web/app.js
- Besides the two critical bugs: the agent-detection logic (`isAgent`, poll loop in `waitForAgent`) is reasonable, though `RoomEvent.ParticipantConnected` + initial scan makes the 250 ms poll redundant — a promise resolved from the event handler would be tidier.
- The mic meter builds its own `AudioContext` + analyser; livekit-client already exposes `LocalAudioTrack` events and `createAudioAnalyser()` helper. Not wrong, just duplicative.
- `stopBtn` handler doesn't clear attached audio elements (moot until 2.1 is fixed — then it leaks one `<audio>` element per session).
- `addMessage` interpolates transcript text with `innerHTML` — **XSS via speech**: whatever the STT/LLM emits lands in the DOM unescaped. Use `textContent` for the text node.

### web/token_server.py
- JWT construction is correct (HS256, `roomConfig.agents[].agentName` in protojson camelCase — the server accepts this).
- `Access-Control-Allow-Origin: *` on an endpoint that mints publish-capable tokens: any page a LAN user visits can silently get a token and join the room. Same-origin only — drop the CORS headers entirely (the UI is same-origin through nginx).
- No room-name validation: the client can request any room. Pin it server-side to `voice-room` or a validated allowlist.
- Single-threaded `HTTPServer` is fine at this scale; note it blocks during slow clients.

### Infra (Dockerfile / nginx / compose / livekit.yaml)
- nginx: WS proxy for `/livekit/` is correct (prefix strip + Upgrade headers). The embedded one-line `printf` config in the Dockerfile is unreadable and un-diffable — move it to a real `nginx.conf` file that gets COPYed.
- nginx `location /token` sets `proxy_set_header Content-Type ...` — that's a no-op/incorrect use (Content-Type is a client request header); remove.
- `COPY web/ /var/www/voice/` publishes `token_server.py` source at `https://host:8090/token_server.py`. No secrets in it, but don't serve server code; exclude it (`.dockerignore`-style or copy files explicitly).
- `devkey: devsecret` in livekit.yaml and compose defaults: anyone on the LAN who reads this repo can mint admin tokens. For a LAN toy it's acceptable; generate a random secret at first boot if this ever leaves the bench.
- compose has no `healthcheck`; a supervisord child crash-looping (the exact failure class you just debugged) leaves the container "Up" and green. Add a healthcheck that curls `https://localhost:8090/` and hits LiveKit's `/` on 7880.
- `supervisord.conf` has `nodaemon=false` while CMD runs `supervisord -n` — the flag wins, but make the file say what it does.
- AGENTS.md file-layout section is accurate; the Testing section's `docker compose logs -f agent` is wrong (see 3.5).

### Doc drift (AGENTS.md)
- "max_tokens=1000 / reasoning_content fallback" — documented, not implemented (3.1).
- "Interruption mode must be vad" — implemented, consistent. Good.
- Ports table says 8090 is "UI + signaling" — correct via the nginx proxy; 7880 needn't be exposed at all with host networking (it is, via EXPOSE — cosmetic).

---

## 5. Verification Performed (evidence)

All checks run against the built image `voice-voice:latest` and the live LAN services:

| Check | Result |
|---|---|
| `import agent` in-container | OK — no API mismatches against livekit-agents 1.7.0 |
| `VoiceAssistant()` + full `AgentSession(...)` construction | OK — MCP toolsets, TurnHandlingOptions accepted |
| Default VAD | Local silero via `livekit.local_inference` native lib (34 MB, bundled) — constructs with `--network none` |
| Turn detector | Falls back to local `v1-mini` when not on LiveKit Cloud; degrades to endpointing delay if unavailable — no cloud dependency |
| Azure STT capabilities | `streaming: True` — no VAD strictly required for transcription |
| `azure.TTS.update_options` | Exists — live voice switching is valid |
| Gemma `/v1/chat/completions` | Responds; tool calls work; **`reasoning_content` confirmed present with empty `content`** |
| `decodeToString` in vendored livekit-client 2.13.0 | **0 occurrences** — confirmed frontend bug |
| `TrackSubscribed`/`attach()` in app.js | **Absent** — confirmed frontend bug |
| Container state | Not currently running (no runtime logs available) |

---

## 6. Recommended Fix Order

1. `app.js`: add `TrackSubscribed` → `track.attach()` + `startAudio()` + `AudioPlaybackStatusChanged` fallback (2.1).
2. `app.js`: `TextDecoder` fix for `DataReceived` (2.2) — or migrate both sides to `lk.transcription` text streams.
3. `agent.py`: reasoning_content/max_tokens handling (3.1).
4. supervisord → stdout logging + AGENTS.md log command fix (3.5).
5. `app.js`: `innerHTML` → `textContent` (XSS).
6. Remaining review items as time allows (3.2-3.8, section 4).

**Test plan after 1+2:**
- `docker compose up --build -d`, wait ~15 s for worker registration.
- Browser → Start → say "hello". Expect: your words render as a user message (streamed if using text streams), agent audio plays within ~1-2 s, agent message renders.
- Watch `docker exec voice tail -f /var/log/supervisor/agent.log`: confirm `user state -> speaking`, `STT (final): ...`, then TTS synthesis.
- Change voice mid-session; next reply uses the new voice.
- Kill/reload the tab, rejoin: agent should still respond (`close_on_disconnect=False` path).
- iPhone Safari: verify the "tap to enable audio" fallback fires and works.

---

## 7. Product / Theming & Visual Upgrade Proposals

The current UI is a functional dev console. To make it feel like a product:

### Core interaction model
- **Replace the transcript-first layout with a voice-first "orb" UI.** A large centered circle that breathes in idle, pulses with *your* mic level while listening (you already compute the level — drive the orb instead of a "mic: 42%" debug string), and ripples with the *agent's* output level while speaking. This one element communicates listening/thinking/speaking state instantly and is the signature visual of every good voice product.
- **Explicit state machine surfaced in the UI:** `idle → connecting → listening → thinking → speaking`, driven by the agent's `lk.agent.state` participant attribute (the SDK publishes it — free to consume). Replace the freeform status strings.
- **Streaming transcripts with partials** (via `lk.transcription`): words appear as you say them, greyed until final. Massive perceived-latency win.
- **Tool-use indicator:** when Gemma calls `web_search`, show a subtle "searching the web..." chip. Right now web lookups are silent 5-10 s gaps that feel like crashes.
- **Barge-in affordance:** show a small "tap or speak to interrupt" hint while the agent is speaking.

### Visual design
- Keep the dark theme, but commit to it: one accent hue (the blue→violet gradient is good), true-black `#09090b` base, glassmorphism panels (`backdrop-filter: blur`) for the transcript drawer, `Inter`/`Geist` variable font instead of system stack.
- Chat bubbles: user right-aligned, agent left-aligned with a small avatar dot in the accent gradient; timestamps on hover. Drop the border-left style.
- Motion: orb idle animation (4 s ease breathing), message enter transitions (~120 ms fade/slide), respect `prefers-reduced-motion`.
- The voice picker becomes a proper settings sheet: voice cards with a **"preview" play button** (one cached Azure TTS sample per voice), plus speaking-rate and style (`mstts:express-as`) controls — the SSML plumbing already exists.
- Light-mode variant via `prefers-color-scheme`.

### Mobile / platform
- PWA manifest + icons so it installs to the home screen (the HTTPS requirement is already met); `theme-color` meta; safe-area insets.
- Wake-lock (`navigator.wakeLock`) during a session so the phone doesn't sleep mid-conversation.
- Favicon (currently a 404 on every load).

### Nice-to-haves that differentiate
- Conversation history persisted to `localStorage` with a "clear" button.
- Latency HUD (dev toggle): STT-final→first-audio time per turn — you'll want it while tuning Gemma.
- Push-to-talk mode toggle for noisy rooms (disable VAD interruption, hold spacebar/tap-and-hold the orb).
- Audio visualizer bars on the agent bubble while it speaks (a `createAudioAnalyser()` on the remote track — the same trick the mic meter already does locally).

---

*End of review.*