hope-voice-api/AGENTS.md

# Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with a local Gemma LLM for reasoning.

## Architecture

One Docker container runs three processes via supervisord:

1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
3. **Web frontend** — static HTML served by a tiny HTTP server (port 8090)

```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → Gemma LLM → Azure TTS
```

## Quick Start

```bash
# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:8090
```

LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
(WebRTC media), `7881/tcp` (media TCP fallback).

## Ports

| Port  | Protocol | Service              | Access       |
|-------|----------|----------------------|--------------|
| 7880  | TCP      | LiveKit HTTP/WS      | container (proxied via 8090/livekit) |
| 7881  | TCP      | LiveKit RTC media (TCP fallback) | LAN |
| 7882  | UDP      | LiveKit RTC media (muxed) | LAN   |
| 8090  | TCP      | Web frontend (HTTPS) | LAN          |

## Configuration

All config lives in `.env` (git-ignored). See `.env.example` for the full list.

Key variables:
- `AZURE_SPEECH_KEY` — Azure Speech resource key
- `AZURE_SPEECH_REGION` — default `eastus`
- `AZURE_TTS_VOICE` — default voice (e.g. `en-US-AvaNeural`)
- `GEMMA_BASE_URL` — LLM endpoint (default `http://192.168.86.2:8023/v1`)
- `GEMMA_MODEL` — model name (default `gemma-4-e4b`)

## Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

## SSML / Expressive Speech

The TTS layer uses full SSML with `<mstts:express-as>` for style and `<prosody>` for rate/volume/pitch. The agent's system prompt instructs the LLM to write in a conversational, spoken style (short sentences, natural phrasing) so the output sounds like speech, not text.

## Agent Prompt

The LLM is instructed to:
- Speak as if talking to someone, not writing for them to read
- Keep responses to 1-3 sentences
- Use contractions, natural fillers sparingly
- Never use markdown, lists, or formatting
- Spell out numbers and abbreviations

## Testing

```bash
# Verify the container is running
docker compose ps

# Check agent logs
docker compose logs -f agent

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-AvaNeural">Test</voice></speak>' > /tmp/test.mp3
```

## File Layout

```
~/dev/voice/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container
├── Dockerfile           ← multi-stage build
├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml         ← LiveKit server config
├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server)
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline
│   └── pyproject.toml   ← Python deps (uv)
└── web/
    ├── index.html       ← single-page voice UI
    ├── app.js           ← LiveKit client logic
    ├── token_server.py  ← signs JWTs + roomConfig claim (agent dispatch)
    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
    └── style.css        ← minimal dark theme
```

## Conventions

- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting.

## Git

Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).