Move the media server, downloader, library volume, and browser player to the
standalone hope-media-music product. Keep Hope's music tools as a thin HTTP
client of the LAN service on port 8094.
Use a stable voice-room player scope instead of ephemeral LiveKit room names.
The agent now owns the conversation handshake: fresh reported playback mutes
its input, joining pauses music, and the last participant leaving resumes only
music it paused. Peer failures remain non-fatal.
Also move the voice deploy workflow onto the active Unraid Gitea runner and
remove yt-dlp/ffmpeg/media health and volume dependencies from the image.
Background research tasks were producing garbage or failing because the
self-hosted Firecrawl /v1/scrape defaults to a ~30s timeout that's too short
for heavy news/research sites, and the small LLM won't reliably chain a
second scrape call after a search that only returns snippets.
task_worker.py:
- web_search now auto-fetches the top results directly from the container
(which has working outbound internet) and returns their real text, so the
model gets actual page content in one call instead of snippet echoes.
- web_scrape requests timeout: 90000 (Firecrawl renders heavy sites but needs
>30s), and falls back to a direct container fetch with tag-stripping if
Firecrawl times out or returns nothing.
task_registry.py / dispatch_mcp.py:
- Add task_status(id) and list_tasks() tools so Hope can check the live state
of any dispatched task (pending / running / completed / failed / STALLED).
- Track last_activity per task and surface a stall heuristic.
- Move the status summary into task_registry (no mcp dependency) so it's
unit-testable.
agent.py: prompt tells Hope about task_status / list_tasks.
tests: add task-status summary and last-activity regression tests.
Hope often starts an already-downloaded song from a voice command, with no
click on the player. Two things could leave it paused at 0:
- The <audio> element sits outside the LiveKit room's audio-unlock flow, so
autoplay policy rejected the poll-driven play(). Capture the first page
gesture (pointerdown/keydown/touchstart) and play the current src inside it
to whitelist the element for autoplay.
- Player state is scoped per LiveKit room, but mediaScope() could return a
stale shell room after a reconnect. Derive it from the connected room.name
so the browser polls exactly the room the agent joined, and a "play X"
request can never land in a scope the browser isn't watching.
Add an in-container regression test (test_media_play_resolves_cached_ready_track)
pinning the server-side resolve: an already-ready track plays without a
re-download and serves a stable audio_url.
The LED wall square switches to the track's cover art whenever a song is
loaded outside a conversation, and returns to the pulsing LED wall while
speaking to the agent (music and conversation are mutually exclusive, so
starting a talk hides the art and ending it restores it).
- Swap the CSS orb for a 21x21 LED dot matrix on a glass panel, matching
the livekit.com LED-array hero visual.
- Pulse driven by the agent's rendered audio: an expanding radial bloom
brightens and widens with remote-audio level, dimmer when listening on
the mic, faint static matrix when idle, and a subtle warm breathing
state while thinking.
- State-tinted palettes: cool azure when speaking, blue when listening,
amber when thinking, dim slate when idle.
- Drop the unused orb CSS-variable pulse plumbing and the presence-breathe
/ orbit-turn keyframes; the wall renders per-frame from the analyser.
- The play/pause button never swapped icons: app.js set the .hidden
property on <svg> elements, which does not reflect to the hidden
attribute (that IDL attribute is on HTMLElement, not SVGElement), so the
CSS [hidden] rule never applied and both glyphs overlapped. Toggle a
.playing class on the button and show/hide the icons via CSS instead.
- The Start conversation and mute buttons were pushed off-screen as the
transcript grew: the workspace grid had no template-rows, so row 1
auto-sized to the tall transcript and stretched the voice-console
beyond the clipped 700px box. Pin the row with grid-template-rows:
minmax(0,1fr) and give both grid columns min-height:0 so the transcript
(and the inner #messages) scroll instead of expanding the layout.
- Stream downloads as they grow: raw best-audio (no mp3 transcode) served
chunked from the stream endpoint mid-download, then with native Range
once complete. Songs play within seconds of hitting Play.
- Fix searched songs never starting: reconcileMedia now wires the stream
URL when a track that was preparing gains one; audio_url is stable for
the whole lifecycle so the player never reloads mid-playback.
- Progress bars now parse yt-dlp stderr (stdout never carried [download]).
- Prefer downloaded songs: /media/api/search merges ready library matches
ahead of YouTube results; search_music/play_song tooling and the agent
prompt tell Hope to always pick in_library=true.
- Mute button turns solid red while the mic is muted.
The fake audio upstream bound the test host's loopback, which the Unraid
media server cannot reach (and its LAN route is firewalled). Local runs
keep the controlled upstream; remote runs use a public reference mp3 and
assert Range semantics without byte equality. docker-exec MCP probes
skip on remote targets (they necessarily exercise the local container).
The Unraid deployment's /media/api requests were answered by an
Apache/YOURLS stack squatting on 8092, surfacing as UI 503s; the CI
healthcheck (curl without -f) counted that 503 as healthy, so run #15
deployed green against a broken media server.
- media server port 8092 -> 8093 everywhere (server, MCP, agent, nginx,
compose, CI, .env.example, docs, tests).
- healthchecks use curl -f in compose and rebuild.yml so HTTP error
statuses fail deployment.
- tests default to the Unraid deployment (VOICE_WEB_URL=
https://192.168.86.2:9178); token goes through nginx (loopback :8091
is unreachable off-host); bind-mount-coupled tests skip on remote
targets. AGENTS.md documents the convention and both failure modes.
A missing env var aborts supervisord's %(ENV_x)s config expansion and
takes every program down, which is why the media-player merge failed
CI (run 151): rebuild.yml's docker run did not pass MEDIA_DIR /
MEDIA_SERVER_PORT / YTDLP_COOKIES.
- supervisord.conf: media-server inherits those vars from the container
environment instead of hard-requiring them (media_server.py has safe
defaults for all three).
- rebuild.yml: mount the media library volume, pass the media env, and
extend the healthcheck to the media server healthz.
- compose: YTDLP_COOKIES defaults to the conventional /data/media/
cookies.txt path (ignored unless the file exists).
Parity verified: container started with CI's exact env (no media vars)
— all five programs RUNNING.
Small local models (qwen2.5-vl-3b, gemma-4-e4b) narrate actions ('Sure,
I'll pause the music') without emitting the tool call once the full
system prompt and 26-tool set are in context. Measured: qwen emitted
tools in 2/8 realistic intents, gemma 5/8; single-tool tests pass 100%,
so this is context-induced drift, not plumbing.
- agent/intent_router.py: deterministic forced-call router for
high-frequency intents — media controls, now_playing/list_library,
'remember that ...' memory saves, explicit YouTube Music searches,
and personal-fact recall (memory_recall/search_notes/skill_recall,
excluded for weather/time questions). Fires only when the tool is
available; guards against how-do-I questions.
- agent.py: generalize _ForcedToolCallStream to any tool; route through
intent_router after the dispatch check. System prompt adds a hard
'call, don't narrate' rule and reconditions 'answer directly'.
- tests/test_intent_router.py: 30 unit tests pinning router behavior.
Verified against the live LLM with the full fidelity harness: combined
forced-or-emitted coverage 8/8 on qwen2.5-vl-3b (was 2/8) and 8/8 on
gemma-4-e4b (was 5/8). Full suite: 79 passed.
- yt-dlp stderr now goes to a temp file so a chatty stderr can never
deadlock the stdout progress loop.
- report/search/seek numeric parsing returns 400 instead of 500.
- now-playing card reflects failed/preparing download states.
- media runtime data (library.json, jobs/) is gitignored; only
.gitkeep is tracked.
Document the media server process, /media/ port map, library layout on
the ./media bind mount, Hope's media MCP toolset, the music/mic
mutual-exclusion rule, and yt-dlp download/proxy conventions.
- web/media_server.py: FastAPI service owning the permanent library
(library.json + ./media volume), yt-dlp YouTube Music search, queued
download jobs with mp3 transcode (ffmpeg), per-room player desired
state, and an HTTPS proxy for external live-stream URLs.
- agent/media_mcp.py: Hope tools — search_music, play_song,
offer_song_choices, pause/resume/stop/skip/seek, now_playing,
list_library, add_stream.
- agent/agent.py: media toolset wiring + conversation rule (music
playing disables the agent mic; re-engaging pauses music).
- web UI: slide-out Music panel (search, suggestions, library, live
stream add, play/pause, ±10s, seek, volume) with a reconciling poller
and media_state data-channel reports.
- token server returns the minted room name; nginx serves /media/files
as static (native Range) and proxies /media/api to the media server.
- supervisord runs media-server; compose mounts ./media and adds
ffmpeg to the image.
- Default LLM_MODEL to qwen2.5-vl-3b across agent, task worker, tests,
compose, workflow, and docs
- Cap min/max length-style constraints in tool schemas to 1000 before
LLM requests; llama.cpp rejects grammar repetitions >= 2000 and the
memory MCP's update_note body maxLength (200000) broke qwen2.5-vl-3b
with 'Failed to initialize samplers: failed to parse grammar'
- Move task_dir assignment before VoiceAssistant construction in
handle_job; UnboundLocalError was crashing every agent job before it
joined the room, leaving the UI stuck on 'Opening a secure room'
- Make settings sheet full-window height (100dvh) to match tasks panel
- Point the agent at OpenWebUI's OpenAI-compatible API (LLM_BASE_URL,
LLM_API_KEY, LLM_MODEL) with GEMMA_* env fallbacks
- Add list_available_models() fetching {base_url}/models, filtering
non-chat ids (embedding/reranker)
- Rename GemmaLLM -> HopeLLM with update_options(model=...) for runtime
model switching (mutates _opts.model) and instance-based compaction
- Data channel: agent publishes {type: models} after connect and on
get_models; set_model switches live and acks model_changed
- Web UI: Brain section in the settings sheet with a model picker that
persists the choice in localStorage
- Update tests, CI workflow env, and docs for the new endpoint
- Updated Memory section to always use session_id='hope' for remember/recall
- This makes memory operations instant (0.3s) instead of 20+ seconds
- Fixed tool_call logging bug (FunctionToolCall has .name, not .function.name)
Reduces tool count from 10 to 8 for the LLM, removing the conflict
between 'use dispatch for everything' and having web_search available.
The task_worker has its own web tools for background tasks.
- Remove memory_mcp.py (stdio markdown-file memory)
- Add Cognee MCP as remote Streamable HTTP toolset at 192.168.86.2:8003/mcp
- Filter to only remember/recall/forget tools via allowed_tools
- Update system prompt: Memory section moved to top priority with
explicit 'call recall FIRST' instructions and examples
- Add COGNEE_MCP_URL env var to docker-compose
- Remove /memory volume mount (no longer needed)
- Rewrite memory tests to use Cognee HTTP client fixture
- 28/28 tests passing
- token_server: generate voice-{uuid8} room name per request so each
browser session creates a fresh room and triggers agent dispatch
- token claim: add jobType JT_ROOM to roomConfig.agents
- livekit.yaml: revert empty_timeout to default (300s)
- agent.py: system prompt now mandates dispatch_task for all non-trivial
tasks (not just research); removed inline web_search instructions;
added tool_call logging in _ReasoningFallbackWrapper
The FastMCP server never started listening on stdio because the file was
missing 'if __name__ == "__main__": mcp.run(transport="stdio")'. The
LiveKit agent's MCP client got 'Connection closed' during initialize,
which killed the entire toolset setup — ALL tools (weather, memory,
skills, dispatch) were unavailable. This is why Hope could hear you but
never called any tools.
Set room.empty_timeout to 86400 (24h) in livekit.yaml. Previously the
default 300s timeout destroyed the room when all participants left,
which killed the agent job and left no worker registered for the next
join — causing 'assistant not ready retry' on tab reopen.
In the container, agent.py is at /opt/voice-agent/agent.py (flat, not a
package). The 'from agent.task_registry import ...' raised ModuleNotFoundError
which killed the job before session.start(), causing 'assistant not ready
retry' in the browser. Added try/except ImportError fallback to bare import.
- agent/task_registry.py: file-based JSONL event registry (cross-process)
- agent/task_worker.py: autonomous LLM loop with weather/time/memory/web tools
- agent/dispatch_mcp.py: MCP tool exposing dispatch_task to the main agent
- agent/agent.py: registers dispatch toolset, polls task events → room data
- web: slide-out task panel (FAB button + badge), live step streaming via
data channel topic 'tasks', status dots (running/completed/failed)
- Dockerfile: copies new task_*.py and dispatch_mcp.py files
The dispatch MCP runs in its own process; events flow through
/tmp/tasks/events.jsonl which the main agent tails every second and
forwards to the browser. Tasks run up to 10 LLM iterations with tool calls.
The 4B model was responding conversationally instead of calling memory_save/
memory_recall. Added imperative language (MUST call), concrete examples of
trigger phrases, and explicit instructions to never skip the tool call.
Verified: model now reliably generates tool_calls for save/recall/list.
When conversation exceeds 24 items (~12 turns), Gemma summarizes the older
messages into a 2-4 sentence recap that replaces them, keeping the last 10
items verbatim. Also triggers after 5 minutes of idle time so returning
users get a compacted context rather than a bloated one.
Falls back to hard truncation if the summary call fails.
Compaction:
- GemmaLLM.chat() truncates ChatContext to last 30 items (~15 turns)
before sending to LLM, preventing context window overflow on long
conversations. Preserves system prompt and removes orphaned tool calls.
Skills:
- agent/skills_mcp.py: MCP server with skill_save, skill_recall,
skill_list, skill_update tools backed by .md files in /skills
- skills/ dir bind-mounted into container, git-trackable
- System prompt instructs Hope to save repeatable procedures as skills
and recall them before performing tasks she's done before
- Distinct from memory (facts) — skills are learned *procedures*
- agent/memory_mcp.py: MCP server with memory_recall, memory_save, memory_list
tools backed by .md files in /memory (simple keyword matching for v1)
- memory/ dir bind-mounted into container, persists across rebuilds,
easily backed up via git
- System prompt instructs Hope to recall on past references and save
personal info/preferences naturally without announcing it
- Dockerfile: copy memory_mcp.py; compose: ./memory:/memory volume
snowspeeder
Shane