docs: media player subsystem architecture and conventions

Document the media server process, /media/ port map, library layout on
the ./media bind mount, Hope's media MCP toolset, the music/mic
mutual-exclusion rule, and yt-dlp download/proxy conventions.
This commit is contained in:
snowspeeder committed 2026-08-29 13:22:01 -04:00
1 parent 04c4a53594
commit a53c090882
9 files changed
+479 -13

No files matched your search

+49 -5
View File
@@ -4,17 +4,21 @@ A single-container voice assistant built on LiveKit Agents. Speaks and listens i
## Architecture
One Docker container runs three processes via supervisord:
One Docker container runs four processes via supervisord:
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
4. **Media server** — FastAPI music library/player coordinator (port 8092 internal, proxied at /media/api/)
```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
▲ │
└────────── audio ◄────────────────────────────┘
Azure STT → LLM (OpenWebUI) → Azure TTS
Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
└────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘
```
## Quick Start
@@ -40,6 +44,7 @@ LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
| 8092 | TCP | Media server (library, player state, stream proxy) | internal only (proxied via nginx at /media/api/) |
| 9178 | TCP | Web frontend (HTTPS) | LAN |
## Configuration
@@ -53,6 +58,33 @@ Key variables:
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
- `YTDLP_COOKIES` — optional path to a cookies.txt inside the container (e.g. `/data/media/cookies.txt`) for reliable YouTube downloads
## Media Player
Hope can play music on the web UI's media player: YouTube Music search →
yt-dlp download → ffmpeg transcode to mp3 → nginx static streaming (HTTP
Range) back to the browser's `<audio>` element. Live audio stream URLs
(internet radio) are added to the library and proxied over HTTPS by the
media server (mixed-content safe).
- The permanent library lives in `./media` (bind-mounted to `/data/media`):
`files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
- `web/media_server.py` owns the library, search, download jobs (one at a
time), and per-room player desired state. The browser is the actual player:
it polls `GET /media/api/state?room=…`, reconciles, and reports real
playback back via `POST /media/api/report`.
- Hope's tools live in `agent/media_mcp.py` (MCP, id "media"):
`search_music`, `play_song`, `offer_song_choices`, `pause_music`,
`resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
`list_library`, `add_stream`. The MCP process is spawned per session with
`MEDIA_ROOM` set to the LiveKit room name, scoping player state per room.
- **Music and conversation are mutually exclusive.** When a track plays, the
UI reports `{type: "media_state", state: "playing"}` on data-channel topic
"media-control" and the agent disables its mic (`_apply_listening` in
agent.py). Engaging Hope (Start conversation / text message) pauses the
music; ending the conversation resumes what was playing. Voice commands
(pause/skip/…) work whenever the conversation is active.
## Voice Selection
@@ -104,16 +136,24 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
├── Dockerfile ← multi-stage build
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml ← LiveKit server config
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API)
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server)
├── certs/ ← persisted self-signed cert (volume mount)
├── media/ ← permanent music library (bind-mounted to /data/media)
│ ├── files/ ← transcoded mp3 tracks (nginx-served, HTTP Range)
│ ├── thumbs/ ← cover art
│ ├── jobs/ ← download job status snapshots
│ └── library.json ← library metadata (permanent)
├── agent/
│ ├── agent.py ← LiveKit Agents voice pipeline
│ ├── media_mcp.py ← Hope's music tools (YouTube Music via media server)
│ ├── pyproject.toml ← Python deps (uv)
│ └── uv.lock ← locked dependency versions
└── web/
├── index.html ← single-page voice UI
├── app.js ← LiveKit client logic
├── index.html ← single-page voice UI (+ music panel)
├── app.js ← LiveKit client logic (+ player reconciliation)
├── media_server.py ← FastAPI library/search/download/player state
├── token_server.py ← LiveKit token endpoint (mints room name)
├── manifest.json ← PWA manifest
├── favicon.svg ← site icon
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
@@ -135,6 +175,10 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Music downloads are mp3, not YouTube streams.** The media server transcodes with yt-dlp (`-x --audio-format mp3 --audio-quality 192K --embed-thumbnail`) into `/data/media/files/{video_id}.mp3`; nginx serves those files as statics with native HTTP Range, which is what makes seeking work in the `<audio>` element. Never point the player at YouTube directly.
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
## Git