hope-voice-api/AGENTS.md

# Voice — Real-time Voice Assistant

> **Architecture migration:** The Hermes multi-user implementation supersedes the
> OpenWebUI/MCP architecture below. Read `IMPLEMENTATION_PLAN.md`,
> `control/README.md`, `QUALIFICATION.md`, and `tests/README.md` for current behavior.
> Speech is Azure → dedicated authenticated Hermes → Azure. Account control is a
> separate trusted container; every account owns its private Hermes container.
> No anonymous bootstrap, model picker, local reasoning fallback, MCP dispatch,
> shared memory or skills engine is packaged. Hope Music stays untouched/external.
> The older notes below are historical, not instructions to restore removed code.
> Offline regression tests must not use the legacy live-default suite.

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.

## Architecture

One Docker container runs four processes via supervisord:

1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
4. **Token server** — authenticated LiveKit bootstrap endpoint (port 8091 internal, proxied at `/token`)

```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → LLM (OpenWebUI) → Azure TTS
```

## Quick Start

```bash
# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178
```

LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
(WebRTC media), `7881/tcp` (media TCP fallback).

## Ports

| Port  | Protocol | Service              | Access       |
|-------|----------|----------------------|--------------|
| 8091  | TCP      | Token server          | internal only (proxied via nginx at /token) |
| 7880  | TCP      | LiveKit HTTP/WS      | internal only (proxied via nginx at /livekit/) |
| 7881  | TCP      | LiveKit RTC media (TCP fallback) | LAN |
| 7882  | UDP      | LiveKit RTC media (muxed) | LAN   |
| 9178  | TCP      | Web frontend (HTTPS) | LAN          |

## Configuration

All config lives in `.env` (git-ignored). See `.env.example` for the full list.

Key variables:
- `AZURE_SPEECH_KEY` — Azure Speech resource key
- `AZURE_SPEECH_REGION` — default `eastus`
- `AZURE_TTS_VOICE` — default voice (`en-US-Ava:DragonHDOmniLatestNeural`)
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
- `MUSIC_BASE_URL` — standalone Hope Music LAN API (default `http://192.168.86.2:8094`)
- `MUSIC_ROOM` — stable player scope shared with Hope Music (default `voice-room`)

## Hope Music Integration

Music is a separate product and deployment: `snowspeeder/hope-media-music`. It
owns the library, YouTube Music search/download pipeline, stream proxy, and
the standalone browser player at `https://192.168.86.2:9179`. This repo keeps
only the agent-side control client.

- Hope's tools live in `agent/media_mcp.py` (MCP, id `media`):
  `search_music`, `play_song`, `offer_song_choices`, `pause_music`,
  `resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
  `list_library`, `add_stream`. They call `MUSIC_BASE_URL` over HTTP.
- Music uses the stable `MUSIC_ROOM` scope (`voice-room` by default), not the
  ephemeral LiveKit room minted for each conversation. The standalone player
  defaults to that same scope, so voice commands control the open music page
  and playback survives across voice sessions.
- **Music and conversation remain mutually exclusive.** The agent polls Hope
  Music's fresh browser-reported state. Actual playback disables the agent's
  mic; joining a conversation pauses music, and the last participant leaving
  resumes only music that the agent paused.
- A stale `playing` report (default: thirty seconds) is treated as stopped so
  closing the music page cannot leave Hope's mic disabled.
- Hope Music is a peer service: an outage logs a warning but never takes down
  the voice conversation. Its health and E2E tests belong in its own repo.

## Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

## SSML / Expressive Speech

The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as `[appreciative]`, `[whispering]`, and `[Neutral]`; the agent prompt allows these markers while keeping responses short and conversational.

## Agent Prompt

The LLM is instructed to:
- Speak as if talking to someone, not writing for them to read
- Keep responses to 1-3 sentences
- Use contractions, natural fillers sparingly
- Never use markdown, lists, or formatting
- Spell out numbers and abbreviations
- **Call tools, don't narrate them** — if a tool matches the request, the
  tool call must appear in the reply; saying "I'll pause the music" without
  the call is a failure (nothing happens)

## Tool Calling on Small Local Models

The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session
conditions: with the full system prompt (~6.3KB) and the full tool surface
(~26 tools), they narrate actions in first person ("Sure, I'll pause the
music for you") **without emitting the tool call**. Measured before the fix:
qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool
probes passed 12/12. Three layers address this:

1. **Intent router** (`agent/intent_router.py`) — deterministic forced tool
   calls for high-frequency, unambiguous intents (media controls,
   now_playing/list_library, "remember that …" → memory_save, explicit
   YouTube Music searches, personal-fact recall → memory_recall /
   search_notes / skill_recall). It runs in `HopeLLM.chat()` before the LLM
   (same mechanism as forced dispatch), only fires when the tool is
   available, and guards against "how do I …" questions. New high-frequency
   voice intents belong here — with unit tests — not in prompt hopes.
   Combined forced-or-emitted coverage after the fix: 8/8 on both models.
2. **Prompt rule** — the "call them, don't narrate them" section in
   SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
3. **Schema sanitizer** (`_sanitize_tool_schema` in agent.py) — llama.cpp
   derives a GBNF grammar from tool schemas for qwen's Hermes-style template
   and **aborts on length-style constraints ≥ 2000** (the memory server's
   `maxLength: 200000` broke all qwen tool calling; gemma's text-based tool
   path tolerated it). The sanitizer caps minLength/maxLength/minItems/
   maxItems at 1000 per chat call. Verified working post-fix: `anyOf`
   nullable fields, `list[dict]` with open items, enums, and regex patterns.
   Also `_strict_tool_schema=False`: llama.cpp's Gemma template ignores
   named tool_choice under strict schemas.

**Diagnostic method** (learned the hard way): a failing tool call with a
small model is almost never the plumbing. Probe with the REAL conditions —
full tool set, real system prompt, and a prompt that matches the tool
(mismatched prompts make small models legitimately decline, which looks
identical to a failure). Isolated single-tool probes will pass and mislead.

## Testing

**Tests run against the Unraid deployment by default** (`https://192.168.86.2:9178`,
via `VOICE_WEB_URL` in tests/conftest.py) — that is the instance users hit
and the one CI (rebuild.yml) deploys on every push to master. The local dev
compose instance is for development only; set `VOICE_WEB_URL=https://localhost:9178`
to target it instead. Tests that probe the local container through
`docker exec voice` skip automatically on remote targets.

```bash
# Verify the container is running
docker compose ps

# Run the suite against the Unraid deployment (default)
python -m pytest tests/ -v -m "not slow"

# Verify the external Hope Music boundary + client configuration
python -m pytest tests/test_media.py -v

# Check all logs (single service: "voice")
docker compose logs -f voice

# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3
```

## File Layout

```
~/dev/hope-voice-api/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container
├── Dockerfile           ← multi-stage build
├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml         ← LiveKit server config
├── nginx.conf           ← HTTPS UI + /livekit/ WS proxy
├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server)
├── certs/               ← persisted self-signed cert (volume mount)
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline + Hope Music handshake
│   ├── media_mcp.py     ← thin HTTP client for the separate Hope Music service
│   ├── pyproject.toml   ← Python deps (uv)
│   └── uv.lock          ← locked dependency versions
└── web/
    ├── index.html       ← voice-only single-page UI
    ├── app.js           ← LiveKit client logic (no music player)
    ├── token_server.py  ← LiveKit token endpoint (mints ephemeral room names)
    ├── manifest.json    ← PWA manifest
    ├── favicon.svg      ← site icon
    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
    └── style.css        ← minimal dark theme
```

## Conventions

- **Single voice container.** LiveKit, the agent, nginx, and token endpoint run in one Docker container via supervisord. Hope Music is deliberately a separate product/container.
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server ignores browser-supplied room names and mints `voice-{user}-{uuid8}` server-side.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen2.5-vl-3b`, `qwen3.8-27b`) work.
- **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **Background tasks follow the live model.** The dispatch MCP subprocess inherits its env at spawn, so it cannot see later `set_model` switches. The agent writes the selected model to `<task_dir>/model.txt` at session start and on every switch; `TaskWorker` reads that file (falling back to `LLM_MODEL` env) when starting a task. Without this, a switched session (e.g. gemma-4-e4b) dispatches tasks against the stale env model (qwen2.5-vl-3b), which llama-swap on `local-vl-models` (`--models-max 1`) cannot load without killing the conversation model mid-swap.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Hope Music stays external.** `hope-media-music` owns yt-dlp, ffmpeg, the
  library volume, streaming endpoints, and the player UI. Do not add those
  dependencies or files back to this image. This repo owns only the MCP/HTTP
  client and the conversation/music handshake.
- **Music scope is stable.** LiveKit room names are intentionally ephemeral;
  `MUSIC_ROOM` defaults to `voice-room`, which is also the standalone player's
  default. Never derive the music scope from `ctx.room.name`.
- **Playback truth comes from fresh reports.** The standalone player reports
  real state every few seconds. The agent polls it and accepts `playing` only
  while the report timestamp is fresh (`MUSIC_REPORT_TTL`, default thirty
  seconds), preventing a closed player from silencing the mic forever.
- **Peer failure is non-fatal.** Hope Music being unavailable must not stop the
  voice agent or fail the voice container healthcheck. Music calls log and
  degrade gracefully.
- **Never hard-require optional container env in supervisord.** `%(ENV_X)s`
  expansion aborts the whole config when a variable is missing. Settings with
  safe in-code defaults (including Hope Music) should inherit the container
  environment instead of being listed there.
- **Healthchecks must use `curl -f`.** Without it, any HTTP response — including
  a 503 from a squatter — counts as healthy. Voice health covers only LiveKit
  and its HTTPS UI; Hope Music has its own independent healthcheck.

## Git

Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).