feat: progressive music streaming, library-first search, player fixes
Rebuild & Deploy / rebuild (push) Successful in 59s

- Stream downloads as they grow: raw best-audio (no mp3 transcode) served
  chunked from the stream endpoint mid-download, then with native Range
  once complete. Songs play within seconds of hitting Play.
- Fix searched songs never starting: reconcileMedia now wires the stream
  URL when a track that was preparing gains one; audio_url is stable for
  the whole lifecycle so the player never reloads mid-playback.
- Progress bars now parse yt-dlp stderr (stdout never carried [download]).
- Prefer downloaded songs: /media/api/search merges ready library matches
  ahead of YouTube results; search_music/play_song tooling and the agent
  prompt tell Hope to always pick in_library=true.
- Mute button turns solid red while the mic is muted.
This commit is contained in:
snowspeeder committed 2026-08-29 16:41:38 -04:00
1 parent d89918fd2a
commit 4f8eb00c9f
7 files changed
+440 -107

No files matched your search

+69 -8
View File
@@ -63,10 +63,13 @@ Key variables:
## Media Player
Hope can play music on the web UI's media player: YouTube Music search →
yt-dlp download → ffmpeg transcode to mp3 → nginx static streaming (HTTP
Range) back to the browser's `<audio>` element. Live audio stream URLs
(internet radio) are added to the library and proxied over HTTPS by the
media server (mixed-content safe).
yt-dlp raw best-audio download (webm/m4a, no transcode) → progressive
streaming from the media server while the file grows → finished file served
with native HTTP Range for seeking. Songs start playing within seconds of the
download starting and remain in the library for instant offline replay.
Live audio stream URLs (internet radio) are added to the library and proxied
over HTTPS by the media server (mixed-content safe). Search results put ready
library matches first so Hope prefers a downloaded copy over a fresh lookup.
- The permanent library lives in `./media` (bind-mounted to `/data/media`):
`files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
@@ -104,6 +107,45 @@ The LLM is instructed to:
- Use contractions, natural fillers sparingly
- Never use markdown, lists, or formatting
- Spell out numbers and abbreviations
- **Call tools, don't narrate them** — if a tool matches the request, the
tool call must appear in the reply; saying "I'll pause the music" without
the call is a failure (nothing happens)
## Tool Calling on Small Local Models
The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session
conditions: with the full system prompt (~6.3KB) and the full tool surface
(~26 tools), they narrate actions in first person ("Sure, I'll pause the
music for you") **without emitting the tool call**. Measured before the fix:
qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool
probes passed 12/12. Three layers address this:
1. **Intent router** (`agent/intent_router.py`) — deterministic forced tool
calls for high-frequency, unambiguous intents (media controls,
now_playing/list_library, "remember that …" → memory_save, explicit
YouTube Music searches, personal-fact recall → memory_recall /
search_notes / skill_recall). It runs in `HopeLLM.chat()` before the LLM
(same mechanism as forced dispatch), only fires when the tool is
available, and guards against "how do I …" questions. New high-frequency
voice intents belong here — with unit tests — not in prompt hopes.
Combined forced-or-emitted coverage after the fix: 8/8 on both models.
2. **Prompt rule** — the "call them, don't narrate them" section in
SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
3. **Schema sanitizer** (`_sanitize_tool_schema` in agent.py) — llama.cpp
derives a GBNF grammar from tool schemas for qwen's Hermes-style template
and **aborts on length-style constraints ≥ 2000** (the memory server's
`maxLength: 200000` broke all qwen tool calling; gemma's text-based tool
path tolerated it). The sanitizer caps minLength/maxLength/minItems/
maxItems at 1000 per chat call. Verified working post-fix: `anyOf`
nullable fields, `list[dict]` with open items, enums, and regex patterns.
Also `_strict_tool_schema=False`: llama.cpp's Gemma template ignores
named tool_choice under strict schemas.
**Diagnostic method** (learned the hard way): a failing tool call with a
small model is almost never the plumbing. Probe with the REAL conditions —
full tool set, real system prompt, and a prompt that matches the tool
(mismatched prompts make small models legitimately decline, which looks
identical to a failure). Isolated single-tool probes will pass and mislead.
## Testing
@@ -111,7 +153,9 @@ The LLM is instructed to:
via `VOICE_WEB_URL` in tests/conftest.py) — that is the instance users hit
and the one CI (rebuild.yml) deploys on every push to master. The local dev
compose instance is for development only; set `VOICE_WEB_URL=https://localhost:9178`
to target it instead.
to target it instead. Tests that can only pass on the deployment host
(repo `./media` bind mount) or that probe the local container via
`docker exec voice` skip automatically on remote targets.
```bash
# Verify the container is running
@@ -122,7 +166,6 @@ python -m pytest tests/ -v -m "not slow"
# Real end-to-end YouTube Music download (network-heavy, gated)
HOPE_RUN_YTDLP_DOWNLOAD=1 python -m pytest tests/test_media.py -m slow
```
# Check all logs (single service: "voice")
docker compose logs -f voice
@@ -153,7 +196,7 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server)
├── certs/ ← persisted self-signed cert (volume mount)
├── media/ ← permanent music library (bind-mounted to /data/media)
│ ├── files/ ← transcoded mp3 tracks (nginx-served, HTTP Range)
│ ├── files/ ← raw audio tracks (webm/m4a; media-server streamed, Range when complete)
│ ├── thumbs/ ← cover art
│ ├── jobs/ ← download job status snapshots
│ └── library.json ← library metadata (permanent)
@@ -188,7 +231,25 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Music downloads are mp3, not YouTube streams.** The media server transcodes with yt-dlp (`-x --audio-format mp3 --audio-quality 192K --embed-thumbnail`) into `/data/media/files/{video_id}.mp3`; nginx serves those files as statics with native HTTP Range, which is what makes seeking work in the `<audio>` element. Never point the player at YouTube directly.
- **Music downloads are raw audio, streamed as they grow.** yt-dlp fetches
bestaudio directly (`-f bestaudio/best`, no `-x`/transcode) into
`/data/media/files/{video_id}.{ext}`; the stream endpoint
(`GET /media/api/stream/{id}`) serves the growing `.part` file chunked
(no Content-Length) while the download runs, then serves the finished file
with native Range for seeking. `audio_url` is that endpoint for the whole
lifecycle — a stable URL means the browser never reloads mid-playback. The
mp3 transcode was dropped: it delayed playback by the whole download+encode,
and modern browsers play webm/opus and m4a everywhere. Never point the
player at YouTube directly.
- **Search prefers the library.** `/media/api/search` merges ready library
matches (title token/substring match) in front of the yt-dlp results, and
the search_music tool docstring + system prompt tell Hope to always pick
`in_library=true`. Deterministic preference lives in the server merge, not
in prompt hopes.
- **Progress lines come from yt-dlp stderr.** `[download] N%` goes to stderr;
only `--print-json` metadata goes to stdout. The download worker drains
stderr in a thread (deadlock-safe) and reads stdout for the JSON — parsing
progress from stdout silently never matched.
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.