snowspeeder/hope-voice-api
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: consolidate iOS with API and web preserving history
Rebuild & Deploy / rebuild (push) Failing after 23s
Details

Browse Source
This commit is contained in:
snowspeeder committed 2026-10-04 07:23:49 -04:00
2 parents 91a0d8820f 7bd893ae5c
commit d735ee72fa
102 files changed
+13077 -258
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+31 -245
View File
@@ -1,255 +1,41 @@
# Voice — Real-time Voice Assistant
# Hope — API, web, and native iOS monorepo
> **Architecture migration:** The Hermes multi-user implementation supersedes the
> OpenWebUI/MCP architecture below. Read `IMPLEMENTATION_PLAN.md`,
> `control/README.md`, `QUALIFICATION.md`, and `tests/README.md` for current behavior.
> Speech is Azure → dedicated authenticated Hermes → Azure. Account control is a
> separate trusted container; every account owns its private Hermes container.
> No anonymous bootstrap, model picker, local reasoning fallback, MCP dispatch,
> shared memory or skills engine is packaged. Hope Music stays untouched/external.
> The older notes below are historical, not instructions to restore removed code.
> Offline regression tests must not use the legacy live-default suite.
## Read first
A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with models hosted by OpenWebUI (xNAS) for reasoning.
- Current architecture and deployment: `IMPLEMENTATION_PLAN.md`, `control/README.md`, `QUALIFICATION.md`, and `tests/README.md`.
- Shared chat behavior: `docs/CHAT_CONTRACT.md` and `docs/CHAT_FEATURES_CONTRACT.md`. These current contracts supersede historical no-model-picker notes.
- Native work: `ios/AGENTS.md`, `ios/docs/PROJECT_STATUS.md`, and `ios/docs/ENGINEERING_LEARNINGS.md` before changing native code, tests, signing, or lifecycle.
- Migration/history: `docs/IOS_MONOREPO_MIGRATION.md`.
## Architecture
## Layout and boundaries
One Docker container runs four processes via supervisord:
- `control/`: trusted multi-user account service and private per-account Hermes provisioning.
- `agent/` and root container configuration: Azure STT → authenticated dedicated Hermes → Azure TTS over LiveKit.
- `web/`: browser client, HTTPS proxy bootstrap, and web assets.
- `ios/`: SwiftUI app, XcodeGen source `project.yml`, generated shared Xcode project, native tests and device guidance.
- `docs/`, `tests/`, `scripts/`: shared contracts, offline qualification, and protected deployment tooling.
- Hope Music is external. Do not embed its player or restore its dependencies into this app/container.
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
4. **Token server** — authenticated LiveKit bootstrap endpoint (port 8091 internal, proxied at `/token`)
## Invariants
```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
▲ │
└────────── audio ◄────────────────────────────┘
Azure STT → LLM (OpenWebUI) → Azure TTS
```
- No anonymous bootstrap, direct upstream provider credentials, shared account memory, trust-all TLS, plaintext fallback, or fabricated integrations.
- Text chat uses authenticated HTTP/SSE and does not start a microphone/LiveKit session. Voice retains its separate explicit session lifecycle.
- Model/thinking preferences are server-owned via the current settings contract; do not restore removed legacy OpenWebUI/MCP dispatch paths.
- Preserve account scoping, logout/server-change teardown, TLS pinning, revision checks, durable conversations and SSE replay semantics across both clients.
- Never expose credentials or private payloads in logs, tests, artifacts, or commits. Keep runtime data and certificates outside CI checkouts.
- Use current contracts as the authority; historical iOS plans/deployment receipts are evidence, not instructions to redeploy obsolete images.
## Quick Start
## Verification
```bash
# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)
- Use the offline commands in `tests/README.md` or the explicit offline selection in `.gitea/workflows/rebuild.yml`; never invoke the legacy live-default suite by accident.
- Run theme generator checks after shared theme changes. Native generated theme lives at `ios/Sources/Chat/HopeTheme.generated.swift`.
- From `ios/`, use XcodeGen when adding/removing sources/resources, and run simulator tests with scheme `Voice_iOS`. Opt-in live tests stay off unless explicitly authorized.
- Date qualification records and distinguish unit/simulator/build evidence from production API, installation, and physical microphone/TTS evidence.
# 2. Build and start
docker compose up --build -d
## Git and deployment
# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:9178
```
LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
(WebRTC media), `7881/tcp` (media TCP fallback).
## Ports
| Port | Protocol | Service | Access |
|-------|----------|----------------------|--------------|
| 8091 | TCP | Token server | internal only (proxied via nginx at /token) |
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
| 9178 | TCP | Web frontend (HTTPS) | LAN |
## Configuration
All config lives in `.env` (git-ignored). See `.env.example` for the full list.
Key variables:
- `AZURE_SPEECH_KEY` — Azure Speech resource key
- `AZURE_SPEECH_REGION` — default `eastus`
- `AZURE_TTS_VOICE` — default voice (`en-US-Ava:DragonHDOmniLatestNeural`)
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
- `MUSIC_BASE_URL` — standalone Hope Music LAN API (default `http://192.168.86.2:8094`)
- `MUSIC_ROOM` — stable player scope shared with Hope Music (default `voice-room`)
## Hope Music Integration
Music is a separate product and deployment: `snowspeeder/hope-media-music`. It
owns the library, YouTube Music search/download pipeline, stream proxy, and
the standalone browser player at `https://192.168.86.2:9179`. This repo keeps
only the agent-side control client.
- Hope's tools live in `agent/media_mcp.py` (MCP, id `media`):
`search_music`, `play_song`, `offer_song_choices`, `pause_music`,
`resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
`list_library`, `add_stream`. They call `MUSIC_BASE_URL` over HTTP.
- Music uses the stable `MUSIC_ROOM` scope (`voice-room` by default), not the
ephemeral LiveKit room minted for each conversation. The standalone player
defaults to that same scope, so voice commands control the open music page
and playback survives across voice sessions.
- **Music and conversation remain mutually exclusive.** The agent polls Hope
Music's fresh browser-reported state. Actual playback disables the agent's
mic; joining a conversation pauses music, and the last participant leaving
resumes only music that the agent paused.
- A stale `playing` report (default: thirty seconds) is treated as stopped so
closing the music page cannot leave Hope's mic disabled.
- Hope Music is a peer service: an outage logs a warning but never takes down
the voice conversation. Its health and E2E tests belong in its own repo.
## Voice Selection
The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.
Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices
## SSML / Expressive Speech
The TTS layer wraps output in SSML. The default Azure Dragon HD Omni voice recognizes inline style markers such as `[appreciative]`, `[whispering]`, and `[Neutral]`; the agent prompt allows these markers while keeping responses short and conversational.
## Agent Prompt
The LLM is instructed to:
- Speak as if talking to someone, not writing for them to read
- Keep responses to 1-3 sentences
- Use contractions, natural fillers sparingly
- Never use markdown, lists, or formatting
- Spell out numbers and abbreviations
- **Call tools, don't narrate them** — if a tool matches the request, the
tool call must appear in the reply; saying "I'll pause the music" without
the call is a failure (nothing happens)
## Tool Calling on Small Local Models
The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session
conditions: with the full system prompt (~6.3KB) and the full tool surface
(~26 tools), they narrate actions in first person ("Sure, I'll pause the
music for you") **without emitting the tool call**. Measured before the fix:
qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool
probes passed 12/12. Three layers address this:
1. **Intent router** (`agent/intent_router.py`) — deterministic forced tool
calls for high-frequency, unambiguous intents (media controls,
now_playing/list_library, "remember that …" → memory_save, explicit
YouTube Music searches, personal-fact recall → memory_recall /
search_notes / skill_recall). It runs in `HopeLLM.chat()` before the LLM
(same mechanism as forced dispatch), only fires when the tool is
available, and guards against "how do I …" questions. New high-frequency
voice intents belong here — with unit tests — not in prompt hopes.
Combined forced-or-emitted coverage after the fix: 8/8 on both models.
2. **Prompt rule** — the "call them, don't narrate them" section in
SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
3. **Schema sanitizer** (`_sanitize_tool_schema` in agent.py) — llama.cpp
derives a GBNF grammar from tool schemas for qwen's Hermes-style template
and **aborts on length-style constraints ≥ 2000** (the memory server's
`maxLength: 200000` broke all qwen tool calling; gemma's text-based tool
path tolerated it). The sanitizer caps minLength/maxLength/minItems/
maxItems at 1000 per chat call. Verified working post-fix: `anyOf`
nullable fields, `list[dict]` with open items, enums, and regex patterns.
Also `_strict_tool_schema=False`: llama.cpp's Gemma template ignores
named tool_choice under strict schemas.
**Diagnostic method** (learned the hard way): a failing tool call with a
small model is almost never the plumbing. Probe with the REAL conditions —
full tool set, real system prompt, and a prompt that matches the tool
(mismatched prompts make small models legitimately decline, which looks
identical to a failure). Isolated single-tool probes will pass and mislead.
## Testing
**Tests run against the Unraid deployment by default** (`https://192.168.86.2:9178`,
via `VOICE_WEB_URL` in tests/conftest.py) — that is the instance users hit
and the one CI (rebuild.yml) deploys on every push to master. The local dev
compose instance is for development only; set `VOICE_WEB_URL=https://localhost:9178`
to target it instead. Tests that probe the local container through
`docker exec voice` skip automatically on remote targets.
```bash
# Verify the container is running
docker compose ps
# Run the suite against the Unraid deployment (default)
python -m pytest tests/ -v -m "not slow"
# Verify the external Hope Music boundary + client configuration
python -m pytest tests/test_media.py -v
# Check all logs (single service: "voice")
docker compose logs -f voice
# Tail individual process logs via supervisord stdout
docker exec voice tail -f /dev/stdout
# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
-H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
-H "Content-Type: application/ssml+xml" \
-H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
--data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-Ava:DragonHDOmniLatestNeural">[appreciative] Test complete.</voice></speak>' > /tmp/test.mp3
```
## File Layout
```
~/dev/hope-voice-api/
├── AGENTS.md ← you are here
├── .env.example ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml ← single container
├── Dockerfile ← multi-stage build
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml ← LiveKit server config
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
├── certs/ ← persisted self-signed cert (volume mount)
├── agent/
│ ├── agent.py ← LiveKit Agents voice pipeline + Hope Music handshake
│ ├── media_mcp.py ← thin HTTP client for the separate Hope Music service
│ ├── pyproject.toml ← Python deps (uv)
│ └── uv.lock ← locked dependency versions
└── web/
├── index.html ← voice-only single-page UI
├── app.js ← LiveKit client logic (no music player)
├── token_server.py ← LiveKit token endpoint (mints ephemeral room names)
├── manifest.json ← PWA manifest
├── favicon.svg ← site icon
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
└── style.css ← minimal dark theme
```
## Conventions
- **Single voice container.** LiveKit, the agent, nginx, and token endpoint run in one Docker container via supervisord. Hope Music is deliberately a separate product/container.
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server ignores browser-supplied room names and mints `voice-{user}-{uuid8}` server-side.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Models come from OpenWebUI.** The LLM endpoint is OpenWebUI's OpenAI-compatible API at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`); chat goes to `/api/chat/completions` and the model list comes from `GET {LLM_BASE_URL}/models`. Plain `/v1/*` routes are NOT routed on that instance (they serve the SPA). Pipe models (`opencode_go_messages.*`, `go-chat.*`, `go-responses.*`) currently fail upstream with auth errors; the local llama.cpp/vLLM models (e.g. `gemma-4-e4b`, `qwen2.5-vl-3b`, `qwen3.8-27b`) work.
- **Reasoning models may hide their answer.** Gemma (and similar) sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `HopeLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **Background tasks follow the live model.** The dispatch MCP subprocess inherits its env at spawn, so it cannot see later `set_model` switches. The agent writes the selected model to `<task_dir>/model.txt` at session start and on every switch; `TaskWorker` reads that file (falling back to `LLM_MODEL` env) when starting a task. Without this, a switched session (e.g. gemma-4-e4b) dispatches tasks against the stale env model (qwen2.5-vl-3b), which llama-swap on `local-vl-models` (`--models-max 1`) cannot load without killing the conversation model mid-swap.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Hope Music stays external.** `hope-media-music` owns yt-dlp, ffmpeg, the
library volume, streaming endpoints, and the player UI. Do not add those
dependencies or files back to this image. This repo owns only the MCP/HTTP
client and the conversation/music handshake.
- **Music scope is stable.** LiveKit room names are intentionally ephemeral;
`MUSIC_ROOM` defaults to `voice-room`, which is also the standalone player's
default. Never derive the music scope from `ctx.room.name`.
- **Playback truth comes from fresh reports.** The standalone player reports
real state every few seconds. The agent polls it and accepts `playing` only
while the report timestamp is fresh (`MUSIC_REPORT_TTL`, default thirty
seconds), preventing a closed player from silencing the mic forever.
- **Peer failure is non-fatal.** Hope Music being unavailable must not stop the
voice agent or fail the voice container healthcheck. Music calls log and
degrade gracefully.
- **Never hard-require optional container env in supervisord.** `%(ENV_X)s`
expansion aborts the whole config when a variable is missing. Settings with
safe in-code defaults (including Hope Music) should inherit the container
environment instead of being listed there.
- **Healthchecks must use `curl -f`.** Without it, any HTTP response — including
a 503 from a squatter — counts as healthy. Voice health covers only LiveKit
and its HTTPS UI; Hope Music has its own independent healthcheck.
## Git
Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).
- `master` of this repository is the single integration branch for API/web/iOS. The old `hope-voice-ios` repository is a read-only archive after migration.
- Preserve original iOS ancestry; do not squash away the imported history. See the migration record for original tips and lookup commands.
- Use conventional commits and explicitly stage intended files; leave unrelated local design artifacts untouched.
- Inspect `.gitea/workflows/rebuild.yml` before pushing: root backend changes can deploy to Unraid; iOS-only and documentation changes are excluded. Do not disable backend qualification or mutate protected runtime configuration.
- Maintain `ios/docs/PROJECT_STATUS.md` for native changes and `QUALIFICATION.md` for backend/web qualification. A Git push is not proof of iPhone installation or physical audio success.