docs: media player subsystem architecture and conventions
Document the media server process, /media/ port map, library layout on the ./media bind mount, Hope's media MCP toolset, the music/mic mutual-exclusion rule, and yt-dlp download/proxy conventions.
This commit is contained in:
1 parent
04c4a53594
commit
a53c090882
9 files changed
+479
-13
No files matched your search
@@ -4,17 +4,21 @@ A single-container voice assistant built on LiveKit Agents. Speaks and listens i
|
||||
|
||||
## Architecture
|
||||
|
||||
One Docker container runs three processes via supervisord:
|
||||
One Docker container runs four processes via supervisord:
|
||||
|
||||
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
|
||||
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
|
||||
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
|
||||
4. **Media server** — FastAPI music library/player coordinator (port 8092 internal, proxied at /media/api/)
|
||||
|
||||
```
|
||||
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
|
||||
▲ │
|
||||
└────────── audio ◄────────────────────────────┘
|
||||
Azure STT → LLM (OpenWebUI) → Azure TTS
|
||||
|
||||
Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
|
||||
└────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
@@ -40,6 +44,7 @@ LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
|
||||
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
|
||||
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
|
||||
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
|
||||
| 8092 | TCP | Media server (library, player state, stream proxy) | internal only (proxied via nginx at /media/api/) |
|
||||
| 9178 | TCP | Web frontend (HTTPS) | LAN |
|
||||
|
||||
## Configuration
|
||||
@@ -53,6 +58,33 @@ Key variables:
|
||||
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
|
||||
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
|
||||
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
|
||||
- `YTDLP_COOKIES` — optional path to a cookies.txt inside the container (e.g. `/data/media/cookies.txt`) for reliable YouTube downloads
|
||||
|
||||
## Media Player
|
||||
|
||||
Hope can play music on the web UI's media player: YouTube Music search →
|
||||
yt-dlp download → ffmpeg transcode to mp3 → nginx static streaming (HTTP
|
||||
Range) back to the browser's `<audio>` element. Live audio stream URLs
|
||||
(internet radio) are added to the library and proxied over HTTPS by the
|
||||
media server (mixed-content safe).
|
||||
|
||||
- The permanent library lives in `./media` (bind-mounted to `/data/media`):
|
||||
`files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
|
||||
- `web/media_server.py` owns the library, search, download jobs (one at a
|
||||
time), and per-room player desired state. The browser is the actual player:
|
||||
it polls `GET /media/api/state?room=…`, reconciles, and reports real
|
||||
playback back via `POST /media/api/report`.
|
||||
- Hope's tools live in `agent/media_mcp.py` (MCP, id "media"):
|
||||
`search_music`, `play_song`, `offer_song_choices`, `pause_music`,
|
||||
`resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
|
||||
`list_library`, `add_stream`. The MCP process is spawned per session with
|
||||
`MEDIA_ROOM` set to the LiveKit room name, scoping player state per room.
|
||||
- **Music and conversation are mutually exclusive.** When a track plays, the
|
||||
UI reports `{type: "media_state", state: "playing"}` on data-channel topic
|
||||
"media-control" and the agent disables its mic (`_apply_listening` in
|
||||
agent.py). Engaging Hope (Start conversation / text message) pauses the
|
||||
music; ending the conversation resumes what was playing. Voice commands
|
||||
(pause/skip/…) work whenever the conversation is active.
|
||||
|
||||
## Voice Selection
|
||||
|
||||
@@ -104,16 +136,24 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
├── Dockerfile ← multi-stage build
|
||||
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
|
||||
├── livekit.yaml ← LiveKit server config
|
||||
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
|
||||
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API)
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server)
|
||||
├── certs/ ← persisted self-signed cert (volume mount)
|
||||
├── media/ ← permanent music library (bind-mounted to /data/media)
|
||||
│ ├── files/ ← transcoded mp3 tracks (nginx-served, HTTP Range)
|
||||
│ ├── thumbs/ ← cover art
|
||||
│ ├── jobs/ ← download job status snapshots
|
||||
│ └── library.json ← library metadata (permanent)
|
||||
├── agent/
|
||||
│ ├── agent.py ← LiveKit Agents voice pipeline
|
||||
│ ├── media_mcp.py ← Hope's music tools (YouTube Music via media server)
|
||||
│ ├── pyproject.toml ← Python deps (uv)
|
||||
│ └── uv.lock ← locked dependency versions
|
||||
└── web/
|
||||
├── index.html ← single-page voice UI
|
||||
├── app.js ← LiveKit client logic
|
||||
├── index.html ← single-page voice UI (+ music panel)
|
||||
├── app.js ← LiveKit client logic (+ player reconciliation)
|
||||
├── media_server.py ← FastAPI library/search/download/player state
|
||||
├── token_server.py ← LiveKit token endpoint (mints room name)
|
||||
├── manifest.json ← PWA manifest
|
||||
├── favicon.svg ← site icon
|
||||
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
|
||||
@@ -135,6 +175,10 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
|
||||
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
|
||||
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
|
||||
- **Music downloads are mp3, not YouTube streams.** The media server transcodes with yt-dlp (`-x --audio-format mp3 --audio-quality 192K --embed-thumbnail`) into `/data/media/files/{video_id}.mp3`; nginx serves those files as statics with native HTTP Range, which is what makes seeking work in the `<audio>` element. Never point the player at YouTube directly.
|
||||
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
|
||||
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
|
||||
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
|
||||
|
||||
## Git
|
||||
|
||||
|
||||
Reference in new issue
Block a user