refactor: split music streaming into Hope Music
Rebuild & Deploy / rebuild (push) Successful in 2m7s
Rebuild & Deploy / rebuild (push) Successful in 2m7s
Move the media server, downloader, library volume, and browser player to the standalone hope-media-music product. Keep Hope's music tools as a thin HTTP client of the LAN service on port 8094. Use a stable voice-room player scope instead of ephemeral LiveKit room names. The agent now owns the conversation handshake: fresh reported playback mutes its input, joining pauses music, and the last participant leaving resumes only music it paused. Peer failures remain non-fatal. Also move the voice deploy workflow onto the active Unraid Gitea runner and remove yt-dlp/ffmpeg/media health and volume dependencies from the image.
This commit is contained in:
snowspeeder authored and 
Shane Maynard committed
1 parent
d4ea4fc984
commit
436b3fc8b1
23 files changed
+356
-2518
No files matched your search
@@ -9,16 +9,13 @@ One Docker container runs four processes via supervisord:
|
||||
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
|
||||
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
|
||||
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
|
||||
4. **Media server** — FastAPI music library/player coordinator (port 8093 internal, proxied at /media/api/)
|
||||
4. **Token server** — authenticated LiveKit bootstrap endpoint (port 8091 internal, proxied at `/token`)
|
||||
|
||||
```
|
||||
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
|
||||
▲ │
|
||||
└────────── audio ◄────────────────────────────┘
|
||||
Azure STT → LLM (OpenWebUI) → Azure TTS
|
||||
|
||||
Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
|
||||
└────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
@@ -41,10 +38,10 @@ LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
|
||||
|
||||
| Port | Protocol | Service | Access |
|
||||
|-------|----------|----------------------|--------------|
|
||||
| 8091 | TCP | Token server | internal only (proxied via nginx at /token) |
|
||||
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
|
||||
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
|
||||
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
|
||||
| 8093 | TCP | Media server (library, player state, stream proxy) | internal only (proxied via nginx at /media/api/) |
|
||||
| 9178 | TCP | Web frontend (HTTPS) | LAN |
|
||||
|
||||
## Configuration
|
||||
@@ -58,36 +55,32 @@ Key variables:
|
||||
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
|
||||
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
|
||||
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
|
||||
- `YTDLP_COOKIES` — optional path to a cookies.txt inside the container (e.g. `/data/media/cookies.txt`) for reliable YouTube downloads
|
||||
- `MUSIC_BASE_URL` — standalone Hope Music LAN API (default `http://192.168.86.2:8094`)
|
||||
- `MUSIC_ROOM` — stable player scope shared with Hope Music (default `voice-room`)
|
||||
|
||||
## Media Player
|
||||
## Hope Music Integration
|
||||
|
||||
Hope can play music on the web UI's media player: YouTube Music search →
|
||||
yt-dlp raw best-audio download (webm/m4a, no transcode) → progressive
|
||||
streaming from the media server while the file grows → finished file served
|
||||
with native HTTP Range for seeking. Songs start playing within seconds of the
|
||||
download starting and remain in the library for instant offline replay.
|
||||
Live audio stream URLs (internet radio) are added to the library and proxied
|
||||
over HTTPS by the media server (mixed-content safe). Search results put ready
|
||||
library matches first so Hope prefers a downloaded copy over a fresh lookup.
|
||||
Music is a separate product and deployment: `snowspeeder/hope-media-music`. It
|
||||
owns the library, YouTube Music search/download pipeline, stream proxy, and
|
||||
the standalone browser player at `https://192.168.86.2:9179`. This repo keeps
|
||||
only the agent-side control client.
|
||||
|
||||
- The permanent library lives in `./media` (bind-mounted to `/data/media`):
|
||||
`files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
|
||||
- `web/media_server.py` owns the library, search, download jobs (one at a
|
||||
time), and per-room player desired state. The browser is the actual player:
|
||||
it polls `GET /media/api/state?room=…`, reconciles, and reports real
|
||||
playback back via `POST /media/api/report`.
|
||||
- Hope's tools live in `agent/media_mcp.py` (MCP, id "media"):
|
||||
- Hope's tools live in `agent/media_mcp.py` (MCP, id `media`):
|
||||
`search_music`, `play_song`, `offer_song_choices`, `pause_music`,
|
||||
`resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
|
||||
`list_library`, `add_stream`. The MCP process is spawned per session with
|
||||
`MEDIA_ROOM` set to the LiveKit room name, scoping player state per room.
|
||||
- **Music and conversation are mutually exclusive.** When a track plays, the
|
||||
UI reports `{type: "media_state", state: "playing"}` on data-channel topic
|
||||
"media-control" and the agent disables its mic (`_apply_listening` in
|
||||
agent.py). Engaging Hope (Start conversation / text message) pauses the
|
||||
music; ending the conversation resumes what was playing. Voice commands
|
||||
(pause/skip/…) work whenever the conversation is active.
|
||||
`list_library`, `add_stream`. They call `MUSIC_BASE_URL` over HTTP.
|
||||
- Music uses the stable `MUSIC_ROOM` scope (`voice-room` by default), not the
|
||||
ephemeral LiveKit room minted for each conversation. The standalone player
|
||||
defaults to that same scope, so voice commands control the open music page
|
||||
and playback survives across voice sessions.
|
||||
- **Music and conversation remain mutually exclusive.** The agent polls Hope
|
||||
Music's fresh browser-reported state. Actual playback disables the agent's
|
||||
mic; joining a conversation pauses music, and the last participant leaving
|
||||
resumes only music that the agent paused.
|
||||
- A stale `playing` report (default: thirty seconds) is treated as stopped so
|
||||
closing the music page cannot leave Hope's mic disabled.
|
||||
- Hope Music is a peer service: an outage logs a warning but never takes down
|
||||
the voice conversation. Its health and E2E tests belong in its own repo.
|
||||
|
||||
## Voice Selection
|
||||
|
||||
@@ -153,8 +146,7 @@ identical to a failure). Isolated single-tool probes will pass and mislead.
|
||||
via `VOICE_WEB_URL` in tests/conftest.py) — that is the instance users hit
|
||||
and the one CI (rebuild.yml) deploys on every push to master. The local dev
|
||||
compose instance is for development only; set `VOICE_WEB_URL=https://localhost:9178`
|
||||
to target it instead. Tests that can only pass on the deployment host
|
||||
(repo `./media` bind mount) or that probe the local container via
|
||||
to target it instead. Tests that probe the local container through
|
||||
`docker exec voice` skip automatically on remote targets.
|
||||
|
||||
```bash
|
||||
@@ -164,8 +156,8 @@ docker compose ps
|
||||
# Run the suite against the Unraid deployment (default)
|
||||
python -m pytest tests/ -v -m "not slow"
|
||||
|
||||
# Real end-to-end YouTube Music download (network-heavy, gated)
|
||||
HOPE_RUN_YTDLP_DOWNLOAD=1 python -m pytest tests/test_media.py -m slow
|
||||
# Verify the external Hope Music boundary + client configuration
|
||||
python -m pytest tests/test_media.py -v
|
||||
|
||||
# Check all logs (single service: "voice")
|
||||
docker compose logs -f voice
|
||||
@@ -184,7 +176,7 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
## File Layout
|
||||
|
||||
```
|
||||
~/dev/voice/
|
||||
~/dev/hope-voice-api/
|
||||
├── AGENTS.md ← you are here
|
||||
├── .env.example ← config template (copy to .env)
|
||||
├── .gitignore
|
||||
@@ -192,24 +184,18 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
├── Dockerfile ← multi-stage build
|
||||
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
|
||||
├── livekit.yaml ← LiveKit server config
|
||||
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API)
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server)
|
||||
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
|
||||
├── certs/ ← persisted self-signed cert (volume mount)
|
||||
├── media/ ← permanent music library (bind-mounted to /data/media)
|
||||
│ ├── files/ ← raw audio tracks (webm/m4a; media-server streamed, Range when complete)
|
||||
│ ├── thumbs/ ← cover art
|
||||
│ ├── jobs/ ← download job status snapshots
|
||||
│ └── library.json ← library metadata (permanent)
|
||||
├── agent/
|
||||
│ ├── agent.py ← LiveKit Agents voice pipeline
|
||||
│ ├── media_mcp.py ← Hope's music tools (YouTube Music via media server)
|
||||
│ ├── agent.py ← LiveKit Agents voice pipeline + Hope Music handshake
|
||||
│ ├── media_mcp.py ← thin HTTP client for the separate Hope Music service
|
||||
│ ├── pyproject.toml ← Python deps (uv)
|
||||
│ └── uv.lock ← locked dependency versions
|
||||
└── web/
|
||||
├── index.html ← single-page voice UI (+ music panel)
|
||||
├── app.js ← LiveKit client logic (+ player reconciliation)
|
||||
├── media_server.py ← FastAPI library/search/download/player state
|
||||
├── token_server.py ← LiveKit token endpoint (mints room name)
|
||||
├── index.html ← voice-only single-page UI
|
||||
├── app.js ← LiveKit client logic (no music player)
|
||||
├── token_server.py ← LiveKit token endpoint (mints ephemeral room names)
|
||||
├── manifest.json ← PWA manifest
|
||||
├── favicon.svg ← site icon
|
||||
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
|
||||
@@ -218,9 +204,9 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
|
||||
- **Single voice container.** LiveKit, the agent, nginx, and token endpoint run in one Docker container via supervisord. Hope Music is deliberately a separate product/container.
|
||||
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
|
||||
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
|
||||
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server ignores browser-supplied room names and mints `voice-{user}-{uuid8}` server-side.
|
||||
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
|
||||
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
|
||||
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
|
||||
@@ -231,31 +217,27 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
|
||||
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
|
||||
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
|
||||
- **Music downloads are raw audio, streamed as they grow.** yt-dlp fetches
|
||||
bestaudio directly (`-f bestaudio/best`, no `-x`/transcode) into
|
||||
`/data/media/files/{video_id}.{ext}`; the stream endpoint
|
||||
(`GET /media/api/stream/{id}`) serves the growing `.part` file chunked
|
||||
(no Content-Length) while the download runs, then serves the finished file
|
||||
with native Range for seeking. `audio_url` is that endpoint for the whole
|
||||
lifecycle — a stable URL means the browser never reloads mid-playback. The
|
||||
mp3 transcode was dropped: it delayed playback by the whole download+encode,
|
||||
and modern browsers play webm/opus and m4a everywhere. Never point the
|
||||
player at YouTube directly.
|
||||
- **Search prefers the library.** `/media/api/search` merges ready library
|
||||
matches (title token/substring match) in front of the yt-dlp results, and
|
||||
the search_music tool docstring + system prompt tell Hope to always pick
|
||||
`in_library=true`. Deterministic preference lives in the server merge, not
|
||||
in prompt hopes.
|
||||
- **Progress lines come from yt-dlp stderr.** `[download] N%` goes to stderr;
|
||||
only `--print-json` metadata goes to stdout. The download worker drains
|
||||
stderr in a thread (deadlock-safe) and reads stdout for the JSON — parsing
|
||||
progress from stdout silently never matched.
|
||||
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
|
||||
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
|
||||
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
|
||||
- **Never hard-require container env in supervisord.** `%(ENV_X)s` expansion aborts the whole supervisord config when the var is missing, taking every program down (this silently killed the run-151 deploy). Programs with safe in-code defaults (media server) must inherit the container env instead of listing it.
|
||||
- **Healthchecks must use `curl -f`.** Without it, any HTTP response — including a 503 from a squatter — counts as healthy. Run #15 deployed "healthy" against a broken media port this way.
|
||||
- **Port 8092 is taken on xNAS.** An Apache/YOURLS stack occupies 8092 on the Unraid host; the media server uses 8093. Before picking a new internal port, probe the Unraid deployment host, not just the dev box.
|
||||
- **Hope Music stays external.** `hope-media-music` owns yt-dlp, ffmpeg, the
|
||||
library volume, streaming endpoints, and the player UI. Do not add those
|
||||
dependencies or files back to this image. This repo owns only the MCP/HTTP
|
||||
client and the conversation/music handshake.
|
||||
- **Music scope is stable.** LiveKit room names are intentionally ephemeral;
|
||||
`MUSIC_ROOM` defaults to `voice-room`, which is also the standalone player's
|
||||
default. Never derive the music scope from `ctx.room.name`.
|
||||
- **Playback truth comes from fresh reports.** The standalone player reports
|
||||
real state every few seconds. The agent polls it and accepts `playing` only
|
||||
while the report timestamp is fresh (`MUSIC_REPORT_TTL`, default thirty
|
||||
seconds), preventing a closed player from silencing the mic forever.
|
||||
- **Peer failure is non-fatal.** Hope Music being unavailable must not stop the
|
||||
voice agent or fail the voice container healthcheck. Music calls log and
|
||||
degrade gracefully.
|
||||
- **Never hard-require optional container env in supervisord.** `%(ENV_X)s`
|
||||
expansion aborts the whole config when a variable is missing. Settings with
|
||||
safe in-code defaults (including Hope Music) should inherit the container
|
||||
environment instead of being listed there.
|
||||
- **Healthchecks must use `curl -f`.** Without it, any HTTP response — including
|
||||
a 503 from a squatter — counts as healthy. Voice health covers only LiveKit
|
||||
and its HTTPS UI; Hope Music has its own independent healthcheck.
|
||||
|
||||
## Git
|
||||
|
||||
|
||||
Reference in new issue
Block a user