feat: progressive music streaming, library-first search, player fixes
Rebuild & Deploy / rebuild (push) Successful in 59s
Rebuild & Deploy / rebuild (push) Successful in 59s
- Stream downloads as they grow: raw best-audio (no mp3 transcode) served chunked from the stream endpoint mid-download, then with native Range once complete. Songs play within seconds of hitting Play. - Fix searched songs never starting: reconcileMedia now wires the stream URL when a track that was preparing gains one; audio_url is stable for the whole lifecycle so the player never reloads mid-playback. - Progress bars now parse yt-dlp stderr (stdout never carried [download]). - Prefer downloaded songs: /media/api/search merges ready library matches ahead of YouTube results; search_music/play_song tooling and the agent prompt tell Hope to always pick in_library=true. - Mute button turns solid red while the mic is muted.
This commit is contained in:
1 parent
d89918fd2a
commit
4f8eb00c9f
7 files changed
+440
-107
No files matched your search
@@ -63,10 +63,13 @@ Key variables:
|
||||
## Media Player
|
||||
|
||||
Hope can play music on the web UI's media player: YouTube Music search →
|
||||
yt-dlp download → ffmpeg transcode to mp3 → nginx static streaming (HTTP
|
||||
Range) back to the browser's `<audio>` element. Live audio stream URLs
|
||||
(internet radio) are added to the library and proxied over HTTPS by the
|
||||
media server (mixed-content safe).
|
||||
yt-dlp raw best-audio download (webm/m4a, no transcode) → progressive
|
||||
streaming from the media server while the file grows → finished file served
|
||||
with native HTTP Range for seeking. Songs start playing within seconds of the
|
||||
download starting and remain in the library for instant offline replay.
|
||||
Live audio stream URLs (internet radio) are added to the library and proxied
|
||||
over HTTPS by the media server (mixed-content safe). Search results put ready
|
||||
library matches first so Hope prefers a downloaded copy over a fresh lookup.
|
||||
|
||||
- The permanent library lives in `./media` (bind-mounted to `/data/media`):
|
||||
`files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
|
||||
@@ -104,6 +107,45 @@ The LLM is instructed to:
|
||||
- Use contractions, natural fillers sparingly
|
||||
- Never use markdown, lists, or formatting
|
||||
- Spell out numbers and abbreviations
|
||||
- **Call tools, don't narrate them** — if a tool matches the request, the
|
||||
tool call must appear in the reply; saying "I'll pause the music" without
|
||||
the call is a failure (nothing happens)
|
||||
|
||||
## Tool Calling on Small Local Models
|
||||
|
||||
The local 3-4B models (qwen2.5-vl-3b, gemma-4-e4b) drift under real session
|
||||
conditions: with the full system prompt (~6.3KB) and the full tool surface
|
||||
(~26 tools), they narrate actions in first person ("Sure, I'll pause the
|
||||
music for you") **without emitting the tool call**. Measured before the fix:
|
||||
qwen emitted tools on 2/8 realistic intents, gemma 5/8 — while single-tool
|
||||
probes passed 12/12. Three layers address this:
|
||||
|
||||
1. **Intent router** (`agent/intent_router.py`) — deterministic forced tool
|
||||
calls for high-frequency, unambiguous intents (media controls,
|
||||
now_playing/list_library, "remember that …" → memory_save, explicit
|
||||
YouTube Music searches, personal-fact recall → memory_recall /
|
||||
search_notes / skill_recall). It runs in `HopeLLM.chat()` before the LLM
|
||||
(same mechanism as forced dispatch), only fires when the tool is
|
||||
available, and guards against "how do I …" questions. New high-frequency
|
||||
voice intents belong here — with unit tests — not in prompt hopes.
|
||||
Combined forced-or-emitted coverage after the fix: 8/8 on both models.
|
||||
2. **Prompt rule** — the "call them, don't narrate them" section in
|
||||
SYSTEM_PROMPT. Alone it lifted gemma to 8/8 and qwen to 4/8 LLM emission.
|
||||
3. **Schema sanitizer** (`_sanitize_tool_schema` in agent.py) — llama.cpp
|
||||
derives a GBNF grammar from tool schemas for qwen's Hermes-style template
|
||||
and **aborts on length-style constraints ≥ 2000** (the memory server's
|
||||
`maxLength: 200000` broke all qwen tool calling; gemma's text-based tool
|
||||
path tolerated it). The sanitizer caps minLength/maxLength/minItems/
|
||||
maxItems at 1000 per chat call. Verified working post-fix: `anyOf`
|
||||
nullable fields, `list[dict]` with open items, enums, and regex patterns.
|
||||
Also `_strict_tool_schema=False`: llama.cpp's Gemma template ignores
|
||||
named tool_choice under strict schemas.
|
||||
|
||||
**Diagnostic method** (learned the hard way): a failing tool call with a
|
||||
small model is almost never the plumbing. Probe with the REAL conditions —
|
||||
full tool set, real system prompt, and a prompt that matches the tool
|
||||
(mismatched prompts make small models legitimately decline, which looks
|
||||
identical to a failure). Isolated single-tool probes will pass and mislead.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -111,7 +153,9 @@ The LLM is instructed to:
|
||||
via `VOICE_WEB_URL` in tests/conftest.py) — that is the instance users hit
|
||||
and the one CI (rebuild.yml) deploys on every push to master. The local dev
|
||||
compose instance is for development only; set `VOICE_WEB_URL=https://localhost:9178`
|
||||
to target it instead.
|
||||
to target it instead. Tests that can only pass on the deployment host
|
||||
(repo `./media` bind mount) or that probe the local container via
|
||||
`docker exec voice` skip automatically on remote targets.
|
||||
|
||||
```bash
|
||||
# Verify the container is running
|
||||
@@ -122,7 +166,6 @@ python -m pytest tests/ -v -m "not slow"
|
||||
|
||||
# Real end-to-end YouTube Music download (network-heavy, gated)
|
||||
HOPE_RUN_YTDLP_DOWNLOAD=1 python -m pytest tests/test_media.py -m slow
|
||||
```
|
||||
|
||||
# Check all logs (single service: "voice")
|
||||
docker compose logs -f voice
|
||||
@@ -153,7 +196,7 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server)
|
||||
├── certs/ ← persisted self-signed cert (volume mount)
|
||||
├── media/ ← permanent music library (bind-mounted to /data/media)
|
||||
│ ├── files/ ← transcoded mp3 tracks (nginx-served, HTTP Range)
|
||||
│ ├── files/ ← raw audio tracks (webm/m4a; media-server streamed, Range when complete)
|
||||
│ ├── thumbs/ ← cover art
|
||||
│ ├── jobs/ ← download job status snapshots
|
||||
│ └── library.json ← library metadata (permanent)
|
||||
@@ -188,7 +231,25 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
|
||||
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
|
||||
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
|
||||
- **Music downloads are mp3, not YouTube streams.** The media server transcodes with yt-dlp (`-x --audio-format mp3 --audio-quality 192K --embed-thumbnail`) into `/data/media/files/{video_id}.mp3`; nginx serves those files as statics with native HTTP Range, which is what makes seeking work in the `<audio>` element. Never point the player at YouTube directly.
|
||||
- **Music downloads are raw audio, streamed as they grow.** yt-dlp fetches
|
||||
bestaudio directly (`-f bestaudio/best`, no `-x`/transcode) into
|
||||
`/data/media/files/{video_id}.{ext}`; the stream endpoint
|
||||
(`GET /media/api/stream/{id}`) serves the growing `.part` file chunked
|
||||
(no Content-Length) while the download runs, then serves the finished file
|
||||
with native Range for seeking. `audio_url` is that endpoint for the whole
|
||||
lifecycle — a stable URL means the browser never reloads mid-playback. The
|
||||
mp3 transcode was dropped: it delayed playback by the whole download+encode,
|
||||
and modern browsers play webm/opus and m4a everywhere. Never point the
|
||||
player at YouTube directly.
|
||||
- **Search prefers the library.** `/media/api/search` merges ready library
|
||||
matches (title token/substring match) in front of the yt-dlp results, and
|
||||
the search_music tool docstring + system prompt tell Hope to always pick
|
||||
`in_library=true`. Deterministic preference lives in the server merge, not
|
||||
in prompt hopes.
|
||||
- **Progress lines come from yt-dlp stderr.** `[download] N%` goes to stderr;
|
||||
only `--print-json` metadata goes to stdout. The download worker drains
|
||||
stderr in a thread (deadlock-safe) and reads stdout for the JSON — parsing
|
||||
progress from stdout silently never matched.
|
||||
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
|
||||
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
|
||||
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
|
||||
|
||||
Reference in new issue
Block a user