Compare commits

...
4 Commits
Author SHA1 Message Date
snowspeeder 48b92585c6 fix: harden media server — temp-file yt-dlp stderr, parse guards, UI failure surfacing
Rebuild & Deploy / rebuild (push) Failing after 4m27s
- yt-dlp stderr now goes to a temp file so a chatty stderr can never
  deadlock the stdout progress loop.
- report/search/seek numeric parsing returns 400 instead of 500.
- now-playing card reflects failed/preparing download states.
- media runtime data (library.json, jobs/) is gitignored; only
  .gitkeep is tracked.
2026-08-29 13:27:58 -04:00
snowspeeder f534b8ccb5 chore: ignore media library runtime data, keep directory via .gitkeep 2026-08-29 13:23:21 -04:00
snowspeeder a53c090882 docs: media player subsystem architecture and conventions
Document the media server process, /media/ port map, library layout on
the ./media bind mount, Hope's media MCP toolset, the music/mic
mutual-exclusion rule, and yt-dlp download/proxy conventions.
2026-08-29 13:22:01 -04:00
snowspeeder 04c4a53594 feat: media player API, web player panel, and Hope music MCP
- web/media_server.py: FastAPI service owning the permanent library
  (library.json + ./media volume), yt-dlp YouTube Music search, queued
  download jobs with mp3 transcode (ffmpeg), per-room player desired
  state, and an HTTPS proxy for external live-stream URLs.
- agent/media_mcp.py: Hope tools — search_music, play_song,
  offer_song_choices, pause/resume/stop/skip/seek, now_playing,
  list_library, add_stream.
- agent/agent.py: media toolset wiring + conversation rule (music
  playing disables the agent mic; re-engaging pauses music).
- web UI: slide-out Music panel (search, suggestions, library, live
  stream add, play/pause, ±10s, seek, volume) with a reconciling poller
  and media_state data-channel reports.
- token server returns the minted room name; nginx serves /media/files
  as static (native Range) and proxies /media/api to the media server.
- supervisord runs media-server; compose mounts ./media and adds
  ffmpeg to the image.
2026-08-29 13:03:12 -04:00
21 changed files with 2356 additions and 17 deletions

No files matched your search

+13
View File
@@ -29,6 +29,19 @@ FIRECRAWL_BASE=http://192.168.86.2:3002
# Per-user memory profiles and Bearer tokens live in config/users.json. # Per-user memory profiles and Bearer tokens live in config/users.json.
VOICE_DEFAULT_USER=shane VOICE_DEFAULT_USER=shane
# ── Media player (music library + YouTube Music downloads) ─────────────────
# The permanent library lives in ./media (bind-mounted to /data/media):
# media/files/ transcoded mp3 tracks
# media/thumbs/ cover art
# media/jobs/ download job status
# MEDIA_SERVER_PORT=8092
#
# Optional: export cookies.txt from a browser logged into YouTube and drop it
# in ./media/cookies.txt to avoid "confirm you're not a bot" download blocks.
# Anonymous access works from most residential IPs; cookies are the reliable
# path.
# YTDLP_COOKIES=/data/media/cookies.txt
# Extra MCP servers (optional). JSON list of {url, transport} objects. # Extra MCP servers (optional). JSON list of {url, transport} objects.
# transport: "sse" | "streamable_http" | omit for auto-detect # transport: "sse" | "streamable_http" | omit for auto-detect
# EXTRA_MCP_SERVERS=[{"url":"http://192.168.86.2:3001/mcp","transport":"streamable_http"}] # EXTRA_MCP_SERVERS=[{"url":"http://192.168.86.2:3001/mcp","transport":"streamable_http"}]
+4
View File
@@ -8,3 +8,7 @@ certs/
.venv-tests/ .venv-tests/
__pycache__/ __pycache__/
.pytest_cache/ .pytest_cache/
# Permanent music library (media server runtime data)
/media/*
!/media/.gitkeep
+49 -5
View File
@@ -4,17 +4,21 @@ A single-container voice assistant built on LiveKit Agents. Speaks and listens i
## Architecture ## Architecture
One Docker container runs three processes via supervisord: One Docker container runs four processes via supervisord:
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media) 1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS 2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → LLM (OpenWebUI) → Azure TTS
3. **Web frontend** — static HTML served by nginx (port 9178 by default) 3. **Web frontend** — static HTML served by nginx (port 9178 by default)
4. **Media server** — FastAPI music library/player coordinator (port 8092 internal, proxied at /media/api/)
``` ```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent Browser ──WebRTC──► LiveKit Server ──audio──► Agent
▲ │ ▲ │
└────────── audio ◄────────────────────────────┘ └────────── audio ◄────────────────────────────┘
Azure STT → LLM (OpenWebUI) → Azure TTS Azure STT → LLM (OpenWebUI) → Azure TTS
Browser ──HTTP──► nginx /media/api/ ──► Media server ──► yt-dlp/ffmpeg ──► ./media
└────── audio element ◄── /media/files/ (nginx static, Range) ──────────┘
``` ```
## Quick Start ## Quick Start
@@ -40,6 +44,7 @@ LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) | | 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN | | 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
| 7882 | UDP | LiveKit RTC media (muxed) | LAN | | 7882 | UDP | LiveKit RTC media (muxed) | LAN |
| 8092 | TCP | Media server (library, player state, stream proxy) | internal only (proxied via nginx at /media/api/) |
| 9178 | TCP | Web frontend (HTTPS) | LAN | | 9178 | TCP | Web frontend (HTTPS) | LAN |
## Configuration ## Configuration
@@ -53,6 +58,33 @@ Key variables:
- `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI) - `LLM_BASE_URL` — OpenAI-compatible endpoint (default `http://192.168.86.2:3005/api`, OpenWebUI)
- `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI) - `LLM_MODEL` — default model (default `qwen2.5-vl-3b`; switchable live from the web UI)
- `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`) - `LLM_API_KEY` — OpenWebUI API key (bears `sk-…`)
- `YTDLP_COOKIES` — optional path to a cookies.txt inside the container (e.g. `/data/media/cookies.txt`) for reliable YouTube downloads
## Media Player
Hope can play music on the web UI's media player: YouTube Music search →
yt-dlp download → ffmpeg transcode to mp3 → nginx static streaming (HTTP
Range) back to the browser's `<audio>` element. Live audio stream URLs
(internet radio) are added to the library and proxied over HTTPS by the
media server (mixed-content safe).
- The permanent library lives in `./media` (bind-mounted to `/data/media`):
`files/` mp3s, `thumbs/` cover art, `jobs/` download status, `library.json`.
- `web/media_server.py` owns the library, search, download jobs (one at a
time), and per-room player desired state. The browser is the actual player:
it polls `GET /media/api/state?room=…`, reconciles, and reports real
playback back via `POST /media/api/report`.
- Hope's tools live in `agent/media_mcp.py` (MCP, id "media"):
`search_music`, `play_song`, `offer_song_choices`, `pause_music`,
`resume_music`, `stop_music`, `skip_song`, `seek_music`, `now_playing`,
`list_library`, `add_stream`. The MCP process is spawned per session with
`MEDIA_ROOM` set to the LiveKit room name, scoping player state per room.
- **Music and conversation are mutually exclusive.** When a track plays, the
UI reports `{type: "media_state", state: "playing"}` on data-channel topic
"media-control" and the agent disables its mic (`_apply_listening` in
agent.py). Engaging Hope (Start conversation / text message) pauses the
music; ending the conversation resumes what was playing. Voice commands
(pause/skip/…) work whenever the conversation is active.
## Voice Selection ## Voice Selection
@@ -104,16 +136,24 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
├── Dockerfile ← multi-stage build ├── Dockerfile ← multi-stage build
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start ├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
├── livekit.yaml ← LiveKit server config ├── livekit.yaml ← LiveKit server config
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy ├── nginx.conf ← HTTPS UI + /livekit/ WS proxy + /media/ (static + API)
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server) ├── supervisord.conf ← process manager (livekit, agent, nginx, token-server, media-server)
├── certs/ ← persisted self-signed cert (volume mount) ├── certs/ ← persisted self-signed cert (volume mount)
├── media/ ← permanent music library (bind-mounted to /data/media)
│ ├── files/ ← transcoded mp3 tracks (nginx-served, HTTP Range)
│ ├── thumbs/ ← cover art
│ ├── jobs/ ← download job status snapshots
│ └── library.json ← library metadata (permanent)
├── agent/ ├── agent/
│ ├── agent.py ← LiveKit Agents voice pipeline │ ├── agent.py ← LiveKit Agents voice pipeline
│ ├── media_mcp.py ← Hope's music tools (YouTube Music via media server)
│ ├── pyproject.toml ← Python deps (uv) │ ├── pyproject.toml ← Python deps (uv)
│ └── uv.lock ← locked dependency versions │ └── uv.lock ← locked dependency versions
└── web/ └── web/
├── index.html ← single-page voice UI ├── index.html ← single-page voice UI (+ music panel)
├── app.js ← LiveKit client logic ├── app.js ← LiveKit client logic (+ player reconciliation)
├── media_server.py ← FastAPI library/search/download/player state
├── token_server.py ← LiveKit token endpoint (mints room name)
├── manifest.json ← PWA manifest ├── manifest.json ← PWA manifest
├── favicon.svg ← site icon ├── favicon.svg ← site icon
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN) ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
@@ -135,6 +175,10 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing. - **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
- **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage. - **Model changes are live too.** The agent fetches the chat-capable model list (`HopeLLM` filters out embedding/reranker ids) and publishes `{type: "models", models, selected}`; the UI's settings sheet renders a picker and sends `{type: "set_model", model}`, which calls `HopeLLM.update_options(model=...)` (mutates `self._opts.model`) and acks with `{type: "model_changed", model}`. The choice persists in the browser's localStorage.
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files. - **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
- **Music downloads are mp3, not YouTube streams.** The media server transcodes with yt-dlp (`-x --audio-format mp3 --audio-quality 192K --embed-thumbnail`) into `/data/media/files/{video_id}.mp3`; nginx serves those files as statics with native HTTP Range, which is what makes seeking work in the `<audio>` element. Never point the player at YouTube directly.
- **External stream URLs are proxied, never played directly.** An `http://` radio URL would be blocked as mixed content on the HTTPS UI, so `GET /media/api/stream/{id}` re-serves the upstream over HTTPS with Range passthrough. Downloaded tracks on that endpoint just redirect to `/media/files/`.
- **Player state is per-room desired state, reconciled by the browser.** The media server never streams to anything; it records what SHOULD play per LiveKit room and the browser polls and reports back. Track ids for YouTube tracks are the video ids (dedupe by video_id); stream tracks get `s<uuid>`.
- **Anonymous yt-dlp first, cookies when flaky.** Search (`ytsearch`) and downloads work anonymously from a residential IP; if YouTube starts bot-blocking, drop a cookies.txt into `./media/cookies.txt` and set `YTDLP_COOKIES=/data/media/cookies.txt`.
## Git ## Git
+9 -3
View File
@@ -24,9 +24,10 @@ ENV PYTHONUNBUFFERED=1
# Fix SSL cert path for the slim base image (ca-certificates installs to /etc/ssl/certs) # Fix SSL cert path for the slim base image (ca-certificates installs to /etc/ssl/certs)
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
# Install LiveKit server binary + supervisord + nginx # Install LiveKit server binary + supervisord + nginx + ffmpeg (yt-dlp audio
# extraction/transcoding for the media player)
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
curl ca-certificates supervisor nginx libasound2 iproute2 \ curl ca-certificates supervisor nginx libasound2 iproute2 ffmpeg \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# Download LiveKit server (latest stable) # Download LiveKit server (latest stable)
@@ -48,8 +49,9 @@ COPY agent/task_registry.py /opt/voice-agent/task_registry.py
COPY agent/task_router.py /opt/voice-agent/task_router.py COPY agent/task_router.py /opt/voice-agent/task_router.py
COPY agent/task_worker.py /opt/voice-agent/task_worker.py COPY agent/task_worker.py /opt/voice-agent/task_worker.py
COPY agent/dispatch_mcp.py /opt/voice-agent/dispatch_mcp.py COPY agent/dispatch_mcp.py /opt/voice-agent/dispatch_mcp.py
COPY agent/media_mcp.py /opt/voice-agent/media_mcp.py
# Copy web frontend + token endpoint # Copy web frontend + token endpoint + media server
COPY web/index.html /var/www/voice/ COPY web/index.html /var/www/voice/
COPY web/app.js /var/www/voice/ COPY web/app.js /var/www/voice/
COPY web/style.css /var/www/voice/ COPY web/style.css /var/www/voice/
@@ -57,6 +59,10 @@ COPY web/livekit-client.umd.js /var/www/voice/
COPY web/manifest.json /var/www/voice/ COPY web/manifest.json /var/www/voice/
COPY web/favicon.svg /var/www/voice/ COPY web/favicon.svg /var/www/voice/
COPY web/token_server.py /opt/voice/token_server.py COPY web/token_server.py /opt/voice/token_server.py
COPY web/media_server.py /opt/voice/media_server.py
# Permanent media library mount point (bind-mounted from ./media)
RUN mkdir -p /data/media/files /data/media/thumbs /data/media/jobs
# Config files # Config files
COPY livekit.yaml /etc/livekit.yaml COPY livekit.yaml /etc/livekit.yaml
+64 -2
View File
@@ -196,6 +196,23 @@ SYSTEM_PROMPT = textwrap.dedent("""\
you receive it, share the findings naturally in your conversational style. you receive it, share the findings naturally in your conversational style.
You can have multiple tasks running at once. You can have multiple tasks running at once.
# Music (media player in the web UI)
You can play music on the user's web player. For any music request, call
search_music first, then:
- One clearly-right result → play_song with its video_id. Announce it in
one short sentence, then stop talking — playback pauses the
conversation until the user comes back to you.
- Ambiguous results → offer_song_choices with the top 2-3 options, name
them briefly, and let the user pick by voice or in the player panel.
Never start playback on a guess.
The library is permanent: list_library shows what is already downloaded,
and those tracks play instantly without searching. If the user names a
library track exactly, just play it. Control playback with pause_music,
resume_music, skip_song and seek_music; check what's on with now_playing.
add_stream registers a live radio or direct audio URL as a library track.
While music is playing you cannot hear the user — they will come back to
you when they want to talk again, and the music pauses automatically.
# Skills # Skills
You have a skill library of learned procedures. Use skill_recall when you need You have a skill library of learned procedures. Use skill_recall when you need
to perform a task you've done before — it will give you the steps. When you to perform a task you've done before — it will give you the steps. When you
@@ -624,6 +641,27 @@ def build_mcp_toolsets(user_profile: dict, task_dir: str) -> list[mcp.MCPToolset
) )
logger.info("Dispatch MCP toolset enabled (%s)", task_dir) logger.info("Dispatch MCP toolset enabled (%s)", task_dir)
# Media tools talk to the local media server; the room name scopes player
# state so concurrent sessions don't fight over one virtual player.
media_server_url = os.environ.get("MEDIA_SERVER_URL", "http://127.0.0.1:8092")
media_mcp_script = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media_mcp.py")
toolsets.append(
mcp.MCPToolset(
id="media",
mcp_server=mcp.MCPServerStdio(
command=python_bin,
args=[media_mcp_script],
env={
**os.environ,
"MEDIA_SERVER_URL": media_server_url,
"MEDIA_ROOM": os.path.basename(task_dir.rstrip("/")) or "default",
},
client_session_timeout_seconds=60,
),
)
)
logger.info("Media MCP toolset enabled (%s)", media_server_url)
extra = os.environ.get("EXTRA_MCP_SERVERS", "") extra = os.environ.get("EXTRA_MCP_SERVERS", "")
if extra: if extra:
try: try:
@@ -740,9 +778,22 @@ async def handle_job(ctx: JobContext) -> None:
# Listen for conversation controls (voice selection and voice/text mode). # Listen for conversation controls (voice selection and voice/text mode).
# Defined here so it captures this job's session via closure. # Defined here so it captures this job's session via closure.
# Music and conversation are mutually exclusive by design: while a track
# plays the agent's ears are off; re-engaging the conversation pauses the
# music, and ending the conversation lets the user resume it.
session_mode = {"value": "voice"} # UI voice/text switch
media_playback = {"value": "stopped"} # UI-reported player state
def _apply_listening() -> None:
listening = (
session_mode["value"] == "voice"
and media_playback["value"] != "playing"
)
session.input.set_audio_enabled(listening)
def _on_room_data(packet) -> None: def _on_room_data(packet) -> None:
topic = getattr(packet, "topic", None) topic = getattr(packet, "topic", None)
if topic and topic not in ("voice-control", "conversation-control"): if topic and topic not in ("voice-control", "conversation-control", "media-control"):
return return
try: try:
msg = json.loads(packet.data.decode("utf-8")) msg = json.loads(packet.data.decode("utf-8"))
@@ -755,11 +806,22 @@ async def handle_job(ctx: JobContext) -> None:
return return
voice_enabled = mode == "voice" voice_enabled = mode == "voice"
logger.info("Switching conversation mode to %s", mode) logger.info("Switching conversation mode to %s", mode)
session_mode["value"] = mode
if not voice_enabled: if not voice_enabled:
# interrupt() returns an already-scheduled Future, not a coroutine. # interrupt() returns an already-scheduled Future, not a coroutine.
session.interrupt() session.interrupt()
session.input.set_audio_enabled(voice_enabled)
session.output.set_audio_enabled(voice_enabled) session.output.set_audio_enabled(voice_enabled)
_apply_listening()
return
if msg.get("type") == "media_state":
state = msg.get("state")
if state not in ("playing", "paused", "stopped"):
return
logger.info("Media playback -> %s (agent listening: %s)",
state, session_mode["value"] == "voice" and state != "playing")
media_playback["value"] = state
_apply_listening()
return return
if msg.get("type") == "set_voice": if msg.get("type") == "set_voice":
+229
View File
@@ -0,0 +1,229 @@
"""Media MCP server — lets Hope find and control music by voice.
Tools talk to the local media server (web/media_server.py) over HTTP, which
owns the library, YouTube Music search/download jobs and per-room player
state. The browser is the actual player: it reconciles against the media
server and reports playback back, so voice controls and the web panel stay
in sync.
Per-session environment (set by the agent when spawning this MCP):
- MEDIA_SERVER_URL e.g. http://127.0.0.1:8092
- MEDIA_ROOM LiveKit room name this session controls
The "stop listening while music plays" rule lives in the agent itself: the
UI reports media_state over the data channel and the agent disables its mic
while a track is playing.
"""
from __future__ import annotations
import json
import os
import httpx
from mcp.server.fastmcp import FastMCP
MEDIA_SERVER_URL = os.environ.get("MEDIA_SERVER_URL", "http://127.0.0.1:8092").rstrip("/")
MEDIA_ROOM = os.environ.get("MEDIA_ROOM", "default")
mcp = FastMCP("media")
def _media_get(path: str, params: dict | None = None) -> dict:
resp = httpx.get(f"{MEDIA_SERVER_URL}{path}", params=params, timeout=10)
resp.raise_for_status()
return resp.json()
def _media_post(path: str, body: dict) -> dict:
resp = httpx.post(f"{MEDIA_SERVER_URL}{path}", json=body, timeout=30)
if resp.status_code >= 400:
try:
detail = resp.json().get("error", resp.text)
except Exception: # noqa: BLE001
detail = resp.text
return {"error": f"{resp.status_code}: {detail}"}
return resp.json()
def _control(action: str, **extra) -> str:
data = _media_post("/media/api/control", {"room": MEDIA_ROOM, "action": action, **extra})
if "error" in data:
return f"Could not {action} the music: {data['error']}"
track = data.get("track") or {}
desired = data.get("desired", "")
name = track.get("title") or "nothing"
return f"OK — {action} applied. Loaded track: {name} (player state: {desired})."
def _fmt_duration(seconds) -> str:
try:
seconds = int(seconds or 0)
except (TypeError, ValueError):
return ""
if seconds <= 0:
return ""
minutes, secs = divmod(seconds, 60)
hours, minutes = divmod(minutes, 60)
return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}"
@mcp.tool()
def search_music(query: str, limit: int = 5) -> str:
"""Search YouTube Music for songs. Call this before playing anything the
user describes by name, artist or lyrics. Returns JSON results with
video_id, title, channel, duration and in_library flags. Pick the result
that clearly matches what the user asked for; if two or more could be
right, use offer_song_choices instead of guessing."""
data = _media_post("/media/api/search", {"query": query, "limit": max(1, min(limit, 10))})
if "error" in data:
return f"Search failed: {data['error']}"
results = data.get("results", [])
if not results:
return f"No YouTube Music results for '{query}'."
return json.dumps(results, ensure_ascii=False)
@mcp.tool()
def play_song(video_id: str, title: str = "", channel: str = "") -> str:
"""Play a song in the user's web player. Pass the video_id from
search_music (or an in_library track_id). Downloads and transcodes first
if needed; playback starts automatically once ready. After calling this,
tell the user what is playing in one short sentence and stop talking —
the player takes over and the conversation pauses until they return."""
data = _media_post("/media/api/play", {
"room": MEDIA_ROOM, "video_id": video_id, "title": title, "channel": channel,
})
if "error" in data:
return f"Could not start playback: {data['error']}"
track = data.get("track") or {}
status = track.get("status")
name = track.get("title") or title or video_id
if status == "downloading":
return (
f"'{name}' is downloading and will start playing automatically when "
"ready (usually under a minute). Say it's on the way, then hand over "
"to the player."
)
return f"Playing '{name}' now. Hand over to the player with a short sign-off."
@mcp.tool()
def offer_song_choices(choices: list[dict]) -> str:
"""Offer the user 2-4 song options when a search is ambiguous. Pass a list
like [{"video_id": "...", "title": "...", "channel": "...", "duration": 213}].
The choices appear as clickable cards in the music panel; the user can
also just say which one they want. Briefly name the options aloud, then
wait for their pick."""
if not choices:
return "No choices provided."
data = _media_post("/media/api/choices", {"room": MEDIA_ROOM, "choices": choices})
if "error" in data:
return f"Could not present choices: {data['error']}"
names = [
f"{c.get('title')}" + (f" by {c['channel']}" if c.get("channel") else "")
for c in choices
]
return "Choices shown in the player: " + "; ".join(names) + ". Ask the user to pick one."
@mcp.tool()
def pause_music() -> str:
"""Pause the currently playing music. The user stays in the conversation
afterwards."""
return _control("pause")
@mcp.tool()
def resume_music() -> str:
"""Resume the loaded music. Warn the user you'll stop listening while it
plays — they'll need to come back to you to talk again."""
return _control("resume")
@mcp.tool()
def stop_music() -> str:
"""Stop playback completely and clear the loaded track."""
return _control("stop")
@mcp.tool()
def skip_song() -> str:
"""Skip to the next track in the library (most recently added first).
If the library has nothing else, playback stops."""
return _control("skip")
@mcp.tool()
def seek_music(delta_seconds: int) -> str:
"""Skip forward or backward in the current track. Positive delta_seconds
jumps ahead, negative goes back (e.g. -30 rewinds 30 seconds, 20 skips
20 seconds forward)."""
try:
reported = (_media_get("/media/api/state", {"room": MEDIA_ROOM}).get("reported") or {})
position = float(reported.get("position") or 0.0)
except Exception: # noqa: BLE001
position = 0.0
target = max(0.0, position + float(delta_seconds))
return _control("seek", position=target)
@mcp.tool()
def now_playing() -> str:
"""Return what is currently loaded/playing in the user's web player and
its playback position. Use when the user asks what's playing or what this
song is."""
try:
st = _media_get("/media/api/state", {"room": MEDIA_ROOM})
except Exception as exc: # noqa: BLE001
return f"Media server unreachable: {exc}"
track = st.get("track") or {}
if not track:
return "Nothing is loaded in the player right now."
reported = st.get("reported") or {}
state = reported.get("state") or st.get("desired") or "unknown"
position = _fmt_duration(reported.get("position"))
duration = _fmt_duration(track.get("duration"))
bits = [f"'{track.get('title')}'"]
if track.get("channel"):
bits.append(f"by {track['channel']}")
if state == "playing" and position:
bits.append(f"at {position}" + (f" of {duration}" if duration else ""))
else:
bits.append(f"(playback state: {state})")
return "Currently loaded: " + " ".join(bits) + "."
@mcp.tool()
def list_library() -> str:
"""List the permanent music library: downloaded YouTube Music tracks and
added live streams, ready to play instantly. Use when the user asks what
music is available or wants something from their collection."""
data = _media_get("/media/api/library")
tracks = data.get("tracks", [])
if not tracks:
return "The library is empty — search for something with search_music first."
lines = []
for t in tracks[:25]:
status = "" if t.get("status") == "ready" else f" ({t.get('status')})"
dur = _fmt_duration(t.get("duration"))
dur = f" [{dur}]" if dur else ""
channel = f" — {t['channel']}" if t.get("channel") else ""
lines.append(f"{t['title']}{channel}{dur}{status}")
return f"{len(tracks)} tracks in the library:\n" + "\n".join(lines)
@mcp.tool()
def add_stream(name: str, url: str) -> str:
"""Add a live audio stream URL (internet radio station, direct mp3/aac
stream) to the permanent library. It then shows up in the web player's
library and can be played by name. Pass a short friendly name."""
if not url.lower().startswith(("http://", "https://")):
return "That does not look like an http(s) stream URL."
data = _media_post("/media/api/streams", {"name": name, "url": url})
if "error" in data:
return f"Could not add the stream: {data['error']}"
return f"Added '{name}' to the library as a live stream. You can play it on request."
if __name__ == "__main__":
mcp.run(transport="stdio")
+4
View File
@@ -9,10 +9,14 @@ description = "Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit
requires-python = ">=3.10,<3.15" requires-python = ">=3.10,<3.15"
dependencies = [ dependencies = [
"fastapi>=0.141.1",
"httpx>=0.28.1",
"livekit-agents[mcp]~=1.7", "livekit-agents[mcp]~=1.7",
"livekit-plugins-azure~=1.7", "livekit-plugins-azure~=1.7",
"livekit-plugins-openai~=1.7", "livekit-plugins-openai~=1.7",
"python-dotenv", "python-dotenv",
"uvicorn>=0.52.4",
"yt-dlp>=2026.8.19",
] ]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
Generated
+33
View File
@@ -671,6 +671,22 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" }, { url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" },
] ]
[[package]]
name = "fastapi"
version = "0.141.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "annotated-doc" },
{ name = "pydantic" },
{ name = "starlette" },
{ name = "typing-extensions" },
{ name = "typing-inspection" },
]
sdist = { url = "https://files.pythonhosted.org/packages/8a/02/91e3416a8fdd715abb903a952a6bec7cdd8d14eed55d415fc8595524c319/fastapi-0.141.1.tar.gz", hash = "sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1", size = 425799, upload-time = "2026-07-29T17:18:05.568Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/cb/03/10388a42375ee7e4ac9b94eb2c5c569c8b5795e377e701c9ac3ad63de890/fastapi-0.141.1-py3-none-any.whl", hash = "sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3", size = 131954, upload-time = "2026-07-29T17:18:04.364Z" },
]
[[package]] [[package]]
name = "frozenlist" name = "frozenlist"
version = "1.8.0" version = "1.8.0"
@@ -2744,18 +2760,26 @@ name = "voice-agent"
version = "0.1.0" version = "0.1.0"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "fastapi" },
{ name = "httpx" },
{ name = "livekit-agents", extra = ["mcp"] }, { name = "livekit-agents", extra = ["mcp"] },
{ name = "livekit-plugins-azure" }, { name = "livekit-plugins-azure" },
{ name = "livekit-plugins-openai" }, { name = "livekit-plugins-openai" },
{ name = "python-dotenv" }, { name = "python-dotenv" },
{ name = "uvicorn" },
{ name = "yt-dlp" },
] ]
[package.metadata] [package.metadata]
requires-dist = [ requires-dist = [
{ name = "fastapi", specifier = ">=0.141.1" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "livekit-agents", extras = ["mcp"], specifier = "~=1.7" }, { name = "livekit-agents", extras = ["mcp"], specifier = "~=1.7" },
{ name = "livekit-plugins-azure", specifier = "~=1.7" }, { name = "livekit-plugins-azure", specifier = "~=1.7" },
{ name = "livekit-plugins-openai", specifier = "~=1.7" }, { name = "livekit-plugins-openai", specifier = "~=1.7" },
{ name = "python-dotenv" }, { name = "python-dotenv" },
{ name = "uvicorn", specifier = ">=0.52.4" },
{ name = "yt-dlp", specifier = ">=2026.8.19" },
] ]
[[package]] [[package]]
@@ -3038,3 +3062,12 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/bf/f4/ed5c402ac8fde4403ed3366c2716bfddc8a6677ebd59f3d62772cc7fe468/yarl-1.24.5-cp314-cp314t-win_arm64.whl", hash = "sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688", size = 97222, upload-time = "2026-07-20T02:07:41.55Z" }, { url = "https://files.pythonhosted.org/packages/bf/f4/ed5c402ac8fde4403ed3366c2716bfddc8a6677ebd59f3d62772cc7fe468/yarl-1.24.5-cp314-cp314t-win_arm64.whl", hash = "sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688", size = 97222, upload-time = "2026-07-20T02:07:41.55Z" },
{ url = "https://files.pythonhosted.org/packages/61/02/962c1cbfc401a30c1d034dc67ff395f64b52302c6d62de556c1fca99acc0/yarl-1.24.5-py3-none-any.whl", hash = "sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7", size = 58612, upload-time = "2026-07-20T02:07:43.461Z" }, { url = "https://files.pythonhosted.org/packages/61/02/962c1cbfc401a30c1d034dc67ff395f64b52302c6d62de556c1fca99acc0/yarl-1.24.5-py3-none-any.whl", hash = "sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7", size = 58612, upload-time = "2026-07-20T02:07:43.461Z" },
] ]
[[package]]
name = "yt-dlp"
version = "2026.8.19"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/1e/e0/832fa4ca334b766a06933a196066edc3dba37cdb6f14cd98d59bcc69a4b4/yt_dlp-2026.8.19.tar.gz", hash = "sha256:9e213e48cea35c66b378e4447903f118f6392a5fa380a2b6d7070ec86f4e0af1", size = 3052025, upload-time = "2026-08-19T23:48:59.291Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/69/b2/8cd1613f56eed7ceb64fbd4df3f1c01246bfb098e6f398228bafda22b80b/yt_dlp-2026.8.19-py3-none-any.whl", hash = "sha256:1d57897e94c6665a0a6f9bc54b34e584284e32c034ffab3a7df25d8f7b24eedf", size = 3185533, upload-time = "2026-08-19T23:48:56.925Z" },
]
+4
View File
@@ -3,7 +3,11 @@ Name: voice-agent
Version: 0.1.0 Version: 0.1.0
Summary: Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit Agents Summary: Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit Agents
Requires-Python: <3.15,>=3.10 Requires-Python: <3.15,>=3.10
Requires-Dist: fastapi>=0.141.1
Requires-Dist: httpx>=0.28.1
Requires-Dist: livekit-agents[mcp]~=1.7 Requires-Dist: livekit-agents[mcp]~=1.7
Requires-Dist: livekit-plugins-azure~=1.7 Requires-Dist: livekit-plugins-azure~=1.7
Requires-Dist: livekit-plugins-openai~=1.7 Requires-Dist: livekit-plugins-openai~=1.7
Requires-Dist: python-dotenv Requires-Dist: python-dotenv
Requires-Dist: uvicorn>=0.52.4
Requires-Dist: yt-dlp>=2026.8.19
+4
View File
@@ -1,4 +1,8 @@
fastapi>=0.141.1
httpx>=0.28.1
livekit-agents[mcp]~=1.7 livekit-agents[mcp]~=1.7
livekit-plugins-azure~=1.7 livekit-plugins-azure~=1.7
livekit-plugins-openai~=1.7 livekit-plugins-openai~=1.7
python-dotenv python-dotenv
uvicorn>=0.52.4
yt-dlp>=2026.8.19
+5 -1
View File
@@ -18,12 +18,16 @@ services:
WEB_PORT: "${WEB_PORT:-9178}" WEB_PORT: "${WEB_PORT:-9178}"
VOICE_USERS_FILE: "/etc/hope/users.json" VOICE_USERS_FILE: "/etc/hope/users.json"
VOICE_DEFAULT_USER: "${VOICE_DEFAULT_USER:-shane}" VOICE_DEFAULT_USER: "${VOICE_DEFAULT_USER:-shane}"
MEDIA_DIR: "/data/media"
MEDIA_SERVER_PORT: "${MEDIA_SERVER_PORT:-8092}"
YTDLP_COOKIES: "${YTDLP_COOKIES:-}"
volumes: volumes:
- ./livekit.yaml:/etc/livekit.yaml:ro - ./livekit.yaml:/etc/livekit.yaml:ro
- ./certs:/etc/voice/certs - ./certs:/etc/voice/certs
- ./skills:/skills - ./skills:/skills
- ./media:/data/media
healthcheck: healthcheck:
test: ["CMD-SHELL", "curl -sk https://localhost:$${WEB_PORT:-9178}/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null"] test: ["CMD-SHELL", "curl -sk https://localhost:$${WEB_PORT:-9178}/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null && curl -s http://localhost:$${MEDIA_SERVER_PORT:-8092}/media/api/healthz -o /dev/null"]
interval: 15s interval: 15s
timeout: 5s timeout: 5s
retries: 3 retries: 3
View File
Whitespace-only changes.
+20
View File
@@ -23,6 +23,26 @@ server {
proxy_send_timeout 3600s; proxy_send_timeout 3600s;
} }
# Media server API (library, search, player state) and the proxy for
# external live-stream URLs (http:// sources would be blocked as mixed
# content in the browser, so they are re-served over HTTPS from here).
location /media/api/ {
proxy_pass http://127.0.0.1:8092/media/api/;
proxy_buffering off;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
# Downloaded tracks are plain static files — nginx serves them with
# native HTTP Range support, which gives the web player seeking for free.
location /media/files/ {
alias /data/media/files/;
}
location /media/thumbs/ {
alias /data/media/thumbs/;
}
location / { location / {
try_files $uri $uri/ =404; try_files $uri $uri/ =404;
} }
+15
View File
@@ -56,3 +56,18 @@ environment=
LIVEKIT_API_SECRET="%(ENV_LIVEKIT_API_SECRET)s", LIVEKIT_API_SECRET="%(ENV_LIVEKIT_API_SECRET)s",
VOICE_USERS_FILE="%(ENV_VOICE_USERS_FILE)s", VOICE_USERS_FILE="%(ENV_VOICE_USERS_FILE)s",
VOICE_DEFAULT_USER="%(ENV_VOICE_DEFAULT_USER)s" VOICE_DEFAULT_USER="%(ENV_VOICE_DEFAULT_USER)s"
[program:media-server]
; Runs as root because the /data/media bind mount is a host (Unraid) share
; whose uid does not match the container's voiceuser.
command=/opt/voice-agent/.venv/bin/python /opt/voice/media_server.py
user=root
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true
environment=
MEDIA_DIR="%(ENV_MEDIA_DIR)s",
MEDIA_SERVER_PORT="%(ENV_MEDIA_SERVER_PORT)s",
YTDLP_COOKIES="%(ENV_YTDLP_COOKIES)s"
+1
View File
@@ -27,6 +27,7 @@ python -m pytest tests/ -v -m "not slow" # skip LLM/network-heavy tests
| `test_web_ui.py` | nginx HTTPS UI (:9178) + token endpoint (:8091) | static assets, `/token` (direct + via nginx), `/livekit/` proxy | | `test_web_ui.py` | nginx HTTPS UI (:9178) + token endpoint (:8091) | static assets, `/token` (direct + via nginx), `/livekit/` proxy |
| `test_llm_api.py` | LLM API directly | completion, tool calling, `enable_thinking=false` latency, streaming | | `test_llm_api.py` | LLM API directly | completion, tool calling, `enable_thinking=false` latency, streaming |
| `test_mcp_tools.py` | MCP servers via `docker exec` | weather/time, authenticated remote memory save/recall/delete, and local skills — probes are cleaned up | | `test_mcp_tools.py` | MCP servers via `docker exec` | weather/time, authenticated remote memory save/recall/delete, and local skills — probes are cleaned up |
| `test_media.py` | media server API + player UI + media MCP | library/player state flow, Range proxy, static mp3 Range, live YT Music search (`-m slow`), gated full download (`HOPE_RUN_YTDLP_DOWNLOAD=1`) |
| `test_agent_integration.py` | running container | agent process alive, worker registered with LiveKit, all 4 supervised processes up, container health, LiveKit HTTP API | | `test_agent_integration.py` | running container | agent process alive, worker registered with LiveKit, all 4 supervised processes up, container health, LiveKit HTTP API |
| `test_compaction.py` | `HopeLLM._compact_context` in-container | compaction triggers >24 items, system prompt preserved, short context untouched | | `test_compaction.py` | `HopeLLM._compact_context` in-container | compaction triggers >24 items, system prompt preserved, short context untouched |
+398
View File
@@ -0,0 +1,398 @@
"""Tests for the media player subsystem.
Covers the media server API (via nginx HTTPS and direct), the stream proxy
with HTTP Range passthrough, static file serving of the ./media volume, the
web player markup, and Hope's media MCP toolset. Download tests are gated
behind HOPE_RUN_YTDLP_DOWNLOAD=1 (network-heavy, occasionally flaky).
"""
import json
import os
import subprocess
import threading
import time
import uuid
from http.server import BaseHTTPRequestHandler, HTTPServer
import httpx
import pytest
from conftest import TOKEN_URL, docker_exec
MEDIA_API = "/media/api"
TEST_ROOM = f"media-test-{uuid.uuid4().hex[:8]}"
# ── local upstream for stream-proxy tests ───────────────────────────────────
def _serve_fake_audio(payload: bytes):
"""Tiny Range-capable HTTP server on an ephemeral host port.
The container uses host networking, so the media server inside the
container reaches it on 127.0.0.1.
"""
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
rng = self.headers.get("Range")
if rng and rng.startswith("bytes="):
start_s, _, end_s = rng[6:].partition("-")
start = int(start_s or 0)
end = int(end_s) if end_s else len(payload) - 1
end = min(end, len(payload) - 1)
body = payload[start : end + 1]
self.send_response(206)
self.send_header(
"Content-Range", f"bytes {start}-{end}/{len(payload)}"
)
else:
body = payload
self.send_response(200)
self.send_header("Content-Type", "audio/mpeg")
self.send_header("Content-Length", str(len(body)))
self.send_header("Accept-Ranges", "bytes")
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
server = HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
return server, f"http://127.0.0.1:{port}/audio.mp3"
# ── health + basic API shape ────────────────────────────────────────────────
def test_media_server_health_via_nginx(client):
resp = client.get(f"{MEDIA_API}/healthz")
assert resp.status_code == 200
assert resp.json()["ok"] is True
def test_media_library_endpoint(client):
resp = client.get(f"{MEDIA_API}/library")
assert resp.status_code == 200
data = resp.json()
assert isinstance(data["tracks"], list)
assert data["version"] >= 1
for track in data["tracks"]:
assert {"id", "kind", "title", "status"} <= set(track)
def test_media_search_requires_query(client):
resp = client.post(f"{MEDIA_API}/search", json={"query": ""})
assert resp.status_code == 400
def test_media_play_requires_target(client):
resp = client.post(f"{MEDIA_API}/play", json={"room": TEST_ROOM})
assert resp.status_code == 400
def test_media_unknown_stream_404(client):
resp = client.get(f"{MEDIA_API}/stream/does-not-exist")
assert resp.status_code == 404
# ── player state flow (uses a live stream track as the fixture) ─────────────
def test_media_player_state_flow(client):
"""add stream -> play -> state -> report -> pause -> seek hint -> stop."""
upstream, url = _serve_fake_audio(b"\x00" * 64)
try:
added = client.post(
f"{MEDIA_API}/streams", json={"name": "Test FM", "url": url}
)
assert added.status_code == 200
track = added.json()["track"]
track_id = track["id"]
assert track["kind"] == "stream"
assert track["status"] == "ready"
assert track["audio_url"].startswith(f"{MEDIA_API}/stream/")
played = client.post(
f"{MEDIA_API}/play", json={"room": TEST_ROOM, "track_id": track_id}
).json()
assert played["desired"] == "playing"
assert played["track"]["id"] == track_id
state = client.get(f"{MEDIA_API}/state", params={"room": TEST_ROOM}).json()
assert state["track"]["id"] == track_id
client.post(
f"{MEDIA_API}/report",
json={"room": TEST_ROOM, "track_id": track_id,
"state": "playing", "position": 12.5},
)
state = client.get(f"{MEDIA_API}/state", params={"room": TEST_ROOM}).json()
assert abs(state["reported"]["position"] - 12.5) < 0.01
client.post(f"{MEDIA_API}/control", json={"room": TEST_ROOM, "action": "pause"})
assert client.get(f"{MEDIA_API}/state", params={"room": TEST_ROOM}).json()["desired"] == "paused"
client.post(
f"{MEDIA_API}/control",
json={"room": TEST_ROOM, "action": "seek", "position": 42},
)
hint = client.get(f"{MEDIA_API}/state", params={"room": TEST_ROOM}).json()["seek_hint"]
assert hint["position"] == 42
# start_paused variant: scope transfer without starting playback
transfer = client.post(
f"{MEDIA_API}/play",
json={"room": TEST_ROOM + "-b", "track_id": track_id, "start_paused": True},
).json()
assert transfer["desired"] == "paused"
# playing clears any open suggestion choices
client.post(
f"{MEDIA_API}/choices",
json={"room": TEST_ROOM, "choices": [{"video_id": "x", "title": "X"}]},
)
client.post(
f"{MEDIA_API}/play", json={"room": TEST_ROOM, "track_id": track_id}
)
assert client.get(f"{MEDIA_API}/state", params={"room": TEST_ROOM}).json()["choices"] == []
stopped = client.post(
f"{MEDIA_API}/control", json={"room": TEST_ROOM, "action": "stop"}
).json()
assert stopped["desired"] == "stopped"
finally:
upstream.shutdown()
# Remove the test stream from the permanent library.
try:
client.delete(f"{MEDIA_API}/tracks/{track_id}")
except (httpx.HTTPError, UnboundLocalError):
pass
def test_media_stream_proxy_and_range(client):
"""External audio URLs must be reachable same-origin (HTTPS) and support
Range passthrough so the browser can seek progressive streams."""
payload = b"0123456789" * 100
upstream, url = _serve_fake_audio(payload)
track_id = None
try:
track_id = client.post(
f"{MEDIA_API}/streams", json={"name": "Range FM", "url": url}
).json()["track"]["id"]
full = client.get(f"{MEDIA_API}/stream/{track_id}")
assert full.status_code == 200
assert full.content == payload
ranged = client.get(
f"{MEDIA_API}/stream/{track_id}", headers={"Range": "bytes=100-199"}
)
assert ranged.status_code == 206
assert ranged.content == payload[100:200]
assert ranged.headers["content-range"] == f"bytes 100-199/{len(payload)}"
finally:
upstream.shutdown()
if track_id:
try:
client.delete(f"{MEDIA_API}/tracks/{track_id}")
except httpx.HTTPError:
pass
def test_media_files_static_range_support(client):
"""Downloaded tracks are nginx static files with native Range support."""
import pathlib
marker = uuid.uuid4().hex[:8]
host_media = pathlib.Path(__file__).parents[1] / "media" / "files"
host_media.mkdir(parents=True, exist_ok=True)
fname = f"test-{marker}.mp3"
payload = b"IMHEADDATA" + b"\x01" * 2048
(host_media / fname).write_bytes(payload)
try:
full = client.get(f"/media/files/{fname}")
assert full.status_code == 200
assert full.content == payload
ranged = client.get(f"/media/files/{fname}", headers={"Range": "bytes=0-9"})
assert ranged.status_code == 206
assert ranged.content == b"IMHEADDATA"
finally:
(host_media / fname).unlink(missing_ok=True)
def test_media_library_persists_in_volume(client):
"""The library JSON is written to the ./media bind mount (permanent)."""
import pathlib
upstream, url = _serve_fake_audio(b"\x00" * 16)
track_id = None
try:
track_id = client.post(
f"{MEDIA_API}/streams", json={"name": "Persist FM", "url": url}
).json()["track"]["id"]
library_file = pathlib.Path(__file__).parents[1] / "media" / "library.json"
data = json.loads(library_file.read_text())
assert track_id in data["tracks"]
finally:
upstream.shutdown()
if track_id:
try:
client.delete(f"{MEDIA_API}/tracks/{track_id}")
except httpx.HTTPError:
pass
# ── web player UI ───────────────────────────────────────────────────────────
def test_ui_media_panel_serves(client):
resp = client.get("/")
body = resp.text
assert 'id="mediaBtn"' in body
assert 'id="mediaPanel"' in body
assert 'id="playPauseBtn"' in body
assert 'id="seekBar"' in body
assert 'id="mediaAudio"' in body
def test_ui_media_logic_serves(client):
resp = client.get("/app.js")
body = resp.text
assert "reconcileMedia" in body
assert "media-control" in body # data-channel topic for the agent
assert "media_state" in body # playback state reports
assert "/media/api/state" in body # reconciliation polling
def test_ui_media_styles_serve(client):
resp = client.get("/style.css")
assert ".media-panel" in resp.text
assert ".media-row" in resp.text
def test_token_response_includes_room():
resp = httpx.post(TOKEN_URL, json={}, timeout=10)
assert resp.status_code == 200
assert resp.json()["room"].startswith("voice-")
# ── Hope's media MCP toolset ────────────────────────────────────────────────
MEDIA_MCP_ENV = {
"MEDIA_SERVER_URL": "http://127.0.0.1:8092",
"MEDIA_ROOM": "mcp-test-room",
}
def test_mcp_media_list_library(mcp_client_factory):
text, is_error = mcp_client_factory(
"media_mcp.py", "list_library", {}, env=MEDIA_MCP_ENV
)
assert not is_error
assert "library" in text.lower()
def test_mcp_media_add_stream_and_now_playing(mcp_client_factory, client):
upstream, url = _serve_fake_audio(b"\x00" * 16)
try:
text, is_error = mcp_client_factory(
"media_mcp.py", "add_stream",
{"name": "MCP Test Radio", "url": url}, env=MEDIA_MCP_ENV,
)
assert not is_error, text
assert "Added" in text
library = client.get(f"{MEDIA_API}/library").json()["tracks"]
probe = next(t for t in library if t["title"] == "MCP Test Radio")
try:
text, is_error = mcp_client_factory(
"media_mcp.py", "now_playing", {}, env=MEDIA_MCP_ENV
)
assert not is_error
# Nothing loaded in mcp-test-room: the tool must say so plainly.
assert "Nothing is loaded" in text
client.post(
f"{MEDIA_API}/play", json={"room": "mcp-test-room", "track_id": probe["id"]}
)
text, is_error = mcp_client_factory(
"media_mcp.py", "now_playing", {}, env=MEDIA_MCP_ENV
)
assert not is_error
assert "MCP Test Radio" in text
finally:
client.delete(f"{MEDIA_API}/tracks/{probe['id']}")
finally:
upstream.shutdown()
def test_mcp_media_resume_without_load(mcp_client_factory, client):
"""Errors from the media server reach the LLM as tool text (not MCP
errors) so Hope can explain them naturally."""
client.post(
f"{MEDIA_API}/control", json={"room": "mcp-empty-room", "action": "stop"}
)
text, is_error = mcp_client_factory(
"media_mcp.py", "resume_music", {}, env={
"MEDIA_SERVER_URL": "http://127.0.0.1:8092",
"MEDIA_ROOM": "mcp-empty-room",
}
)
assert not is_error
assert "nothing loaded" in text.lower()
# ── live YouTube Music (network-dependent) ──────────────────────────────────
@pytest.mark.slow
def test_media_search_live(client):
resp = client.post(
f"{MEDIA_API}/search",
json={"query": "blind guardian timeless miracle", "limit": 3},
)
assert resp.status_code == 200
results = resp.json()["results"]
assert len(results) >= 1
for r in results:
assert r["video_id"]
assert r["title"]
@pytest.mark.slow
def test_media_download_and_stream_end_to_end(client):
"""Full pipeline: search -> play -> yt-dlp download + ffmpeg transcode ->
static mp3 with Range support in the player. Gated: run with
HOPE_RUN_YTDLP_DOWNLOAD=1 (takes 1-3 minutes)."""
if os.environ.get("HOPE_RUN_YTDLP_DOWNLOAD") != "1":
pytest.skip("set HOPE_RUN_YTDLP_DOWNLOAD=1 to run the real download")
search = client.post(
f"{MEDIA_API}/search", json={"query": "kingdom hearts dearly beloved", "limit": 3}
)
results = search.json()["results"]
assert results, "search returned nothing"
video = results[0]
played = client.post(
f"{MEDIA_API}/play",
json={"room": TEST_ROOM, "video_id": video["video_id"],
"title": video["title"], "channel": video["channel"]},
).json()
track_id = played["track"]["id"]
try:
deadline = time.time() + 240
track = played["track"]
while time.time() < deadline:
library = client.get(f"{MEDIA_API}/library").json()["tracks"]
track = next(t for t in library if t["id"] == track_id)
if track["status"] in ("ready", "failed"):
break
time.sleep(3)
assert track["status"] == "ready", f"download failed: {track.get('error')}"
stream = client.get(track["audio_url"], headers={"Range": "bytes=0-1023"})
assert stream.status_code in (200, 206)
assert len(stream.content) >= 1
finally:
client.delete(f"{MEDIA_API}/tracks/{track_id}")
+519 -5
View File
@@ -693,6 +693,7 @@ document.addEventListener("keydown", (event) => {
if (event.key !== "Escape") return; if (event.key !== "Escape") return;
if (settingsSheet.classList.contains("open")) closeSheet(); if (settingsSheet.classList.contains("open")) closeSheet();
if (tasksPanel.classList.contains("open")) setTasksOpen(false); if (tasksPanel.classList.contains("open")) setTasksOpen(false);
if (mediaPanel.classList.contains("open")) setMediaOpen(false);
}); });
// ── Wake lock ─────────────────────────────────────────────────────────────── // ── Wake lock ───────────────────────────────────────────────────────────────
async function acquireWakeLock() { async function acquireWakeLock() {
@@ -816,6 +817,8 @@ function createRoom(token) {
stopBtn.disabled = true; stopBtn.disabled = true;
agentParticipant = null; agentParticipant = null;
agentReady = false; agentReady = false;
// Leaving the conversation hands the room back to the music player.
resumePlaybackAfterTalk();
}); });
return newRoom.connect(LIVEKIT_URL, token, { return newRoom.connect(LIVEKIT_URL, token, {
@@ -873,8 +876,9 @@ async function connectConversation() {
await acquireWakeLock(); await acquireWakeLock();
// Get a signed access token from the token endpoint. // Get a signed access token from the token endpoint.
const token = await fetchToken(ROOM_NAME); const tokenData = await fetchToken(ROOM_NAME);
room = await createRoom(token); await enterMediaScope(tokenData.room);
room = await createRoom(tokenData.token);
// The agent is dispatched when we join. If it misses the dispatch // The agent is dispatched when we join. If it misses the dispatch
// (for example during a server restart), reconnect once. // (for example during a server restart), reconnect once.
@@ -886,8 +890,9 @@ async function connectConversation() {
agentParticipant = null; agentParticipant = null;
agentReady = false; agentReady = false;
await stale.disconnect(); await stale.disconnect();
const freshToken = await fetchToken(ROOM_NAME); const freshTokenData = await fetchToken(ROOM_NAME);
room = await createRoom(freshToken); await enterMediaScope(freshTokenData.room);
room = await createRoom(freshTokenData.token);
if (!(await waitForAgent(15000))) { if (!(await waitForAgent(15000))) {
throw new Error("Assistant did not join — reload the page and try again"); throw new Error("Assistant did not join — reload the page and try again");
} }
@@ -999,6 +1004,8 @@ stopBtn.addEventListener("click", () => {
muteBtn.disabled = true; muteBtn.disabled = true;
micMuted = false; micMuted = false;
updateMuteIcon(); updateMuteIcon();
// Ending the conversation resumes music that was paused for the talk.
resumePlaybackAfterTalk();
}); });
// ── Mic mute toggle ───────────────────────────────────────────────────────── // ── Mic mute toggle ─────────────────────────────────────────────────────────
@@ -1030,7 +1037,7 @@ async function fetchToken(roomName) {
}); });
if (!resp.ok) throw new Error(`Token endpoint returned ${resp.status}`); if (!resp.ok) throw new Error(`Token endpoint returned ${resp.status}`);
const data = await resp.json(); const data = await resp.json();
return data.token; return { token: data.token, room: data.room || null };
} }
// ── Helper: send voice selection to the agent via data channel ───────────── // ── Helper: send voice selection to the agent via data channel ─────────────
@@ -1064,3 +1071,510 @@ function sendModelMessage(model, targetRoom) {
reliable: true, reliable: true,
}); });
} }
// ── Media player ────────────────────────────────────────────────────────────
// The browser <audio> element is the player. The media server owns library +
// per-room desired state; this UI reconciles against GET /media/api/state
// (polling) and reports real playback back. Hope drives the same state via
// her media tools, so voice commands and panel clicks stay in sync.
//
// Conversation rule (mirrored in agent.py): while a track plays the agent's
// mic is off. Connecting to Hope pauses music; ending the conversation
// resumes what was playing.
const mediaBtn = document.getElementById("mediaBtn");
const mediaPanel = document.getElementById("mediaPanel");
const closeMediaBtn = document.getElementById("closeMediaBtn");
const mediaBadge = document.getElementById("mediaBadge");
const mediaArt = document.getElementById("mediaArt");
const mediaArtPlaceholder = document.getElementById("mediaArtPlaceholder");
const mediaTitleEl = document.getElementById("mediaTitle");
const mediaChannelEl = document.getElementById("mediaChannel");
const mediaProgressWrap = document.getElementById("mediaProgressWrap");
const mediaProgressFill = document.getElementById("mediaProgressFill");
const mediaProgressLabel = document.getElementById("mediaProgressLabel");
const rewBtn = document.getElementById("rewBtn");
const fwdBtn = document.getElementById("fwdBtn");
const playPauseBtn = document.getElementById("playPauseBtn");
const playIcon = document.getElementById("playIcon");
const pauseIcon = document.getElementById("pauseIcon");
const stopMediaBtn = document.getElementById("stopMediaBtn");
const seekBar = document.getElementById("seekBar");
const mediaCurEl = document.getElementById("mediaCur");
const mediaDurEl = document.getElementById("mediaDur");
const volumeBar = document.getElementById("volumeBar");
const mediaSearchForm = document.getElementById("mediaSearchForm");
const mediaSearchInput = document.getElementById("mediaSearchInput");
const mediaResultsEl = document.getElementById("mediaResults");
const mediaChoicesWrap = document.getElementById("mediaChoicesWrap");
const mediaChoicesEl = document.getElementById("mediaChoices");
const mediaLibraryEl = document.getElementById("mediaLibrary");
const streamForm = document.getElementById("streamForm");
const streamNameInput = document.getElementById("streamName");
const streamUrlInput = document.getElementById("streamUrl");
const mediaAudio = document.getElementById("mediaAudio");
let mediaRoom = null; // LiveKit room name (player state scope)
let mediaCurrentTrackId = null; // track id loaded into the audio element
let mediaAppliedSeekTs = 0; // last applied seek_hint timestamp
let mediaWasPlayingBeforeTalk = false;
let mediaPollTimer = null;
let mediaLastReport = 0;
function mediaScope() {
return mediaRoom || "default";
}
async function mediaPost(path, body) {
const resp = await fetch(`/media/api${path}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ room: mediaScope(), ...body }),
});
if (!resp.ok) {
const detail = await resp.json().catch(() => ({}));
throw new Error(detail.error || `media API ${path} -> ${resp.status}`);
}
return resp.json();
}
// Called just before joining the LiveKit room: carry whatever the audio
// element holds into the room-scoped player state, paused (talking pauses
// music). resumePlaybackAfterTalk() reverses this when the talk ends.
async function enterMediaScope(newRoom) {
if (!newRoom || newRoom === mediaRoom) return;
const wasPlaying = !!mediaCurrentTrackId && !mediaAudio.paused;
mediaWasPlayingBeforeTalk = wasPlaying;
const carriedTrack = mediaCurrentTrackId;
mediaRoom = newRoom;
if (carriedTrack) {
try {
await mediaPost("/play", { room: mediaRoom, track_id: carriedTrack, start_paused: true });
} catch (e) { console.warn("media scope transfer failed:", e); }
mediaAudio.pause();
}
}
async function resumePlaybackAfterTalk() {
if (!mediaWasPlayingBeforeTalk) return;
mediaWasPlayingBeforeTalk = false;
try { await mediaPost("/control", { action: "resume" }); } catch (e) { /* poll retries */ }
}
function publishMediaState(state) {
if (!room || !room.localParticipant) return;
const payload = new TextEncoder().encode(
JSON.stringify({ type: "media_state", state, track_id: mediaCurrentTrackId })
);
room.localParticipant.publishData(payload, { topic: "media-control", reliable: true });
}
function reportMediaPlayback(state) {
const body = {
state,
track_id: mediaCurrentTrackId,
position: isFinite(mediaAudio.currentTime) ? mediaAudio.currentTime : 0,
};
if (isFinite(mediaAudio.duration)) body.duration = mediaAudio.duration;
mediaPost("/report", body).catch(() => {});
}
// ── Polling + reconciliation ────────────────────────────────────────────────
function startMediaPolling() {
if (mediaPollTimer) return;
mediaPollTimer = setInterval(() => {
pollMediaState();
}, 1500);
}
async function pollMediaState() {
let s;
try {
const resp = await fetch(`/media/api/state?room=${encodeURIComponent(mediaScope())}`);
if (!resp.ok) return;
s = await resp.json();
} catch (e) {
return; // media server briefly unavailable; try again next tick
}
reconcileMedia(s);
}
function reconcileMedia(s) {
renderMediaChoices(s.choices || []);
updateMediaBadge(s);
const track = s.track;
const pendingJob = (s.pending || []).find(j => !track || j.track_id === track.id);
updateMediaProgress(track, pendingJob);
if (track && track.id !== mediaCurrentTrackId) loadMediaTrack(track);
if (!track && mediaCurrentTrackId && s.desired === "stopped") clearMediaAudio();
applyDesiredMediaState(s, track);
if (s.seek_hint && s.seek_hint.ts !== mediaAppliedSeekTs && isFinite(mediaAudio.duration)) {
mediaAppliedSeekTs = s.seek_hint.ts;
const target = Math.min(s.seek_hint.position, mediaAudio.duration);
if (Number.isFinite(target)) {
mediaAudio.currentTime = Math.max(0, target);
updateSeekUI();
}
}
}
function applyDesiredMediaState(s, track) {
if (s.desired === "playing") {
if (track && track.status === "ready" && mediaAudio.src && mediaAudio.paused) {
mediaAudio.play().catch((e) => console.warn("playback blocked:", e));
}
} else if (!mediaAudio.paused) {
mediaAudio.pause();
}
}
function loadMediaTrack(track) {
mediaCurrentTrackId = track.id;
updateMediaNowCard(track);
if (track.audio_url && (track.kind === "stream" || track.status === "ready")) {
if (mediaAudio.getAttribute("src") !== track.audio_url) {
mediaAudio.src = track.audio_url;
mediaAudio.load();
}
} else {
// Downloading or failed: no stream yet. The poll flips this to the
// real URL once status turns "ready".
mediaAudio.removeAttribute("src");
mediaAudio.load();
updateMediaControls();
}
}
function clearMediaAudio() {
mediaCurrentTrackId = null;
mediaAudio.pause();
mediaAudio.removeAttribute("src");
mediaAudio.load();
mediaTitleEl.textContent = "Nothing loaded";
mediaChannelEl.textContent = "";
mediaArt.hidden = true;
mediaArtPlaceholder.hidden = false;
updateMediaControls();
}
function updateMediaNowCard(track) {
mediaTitleEl.textContent = track.title || track.id;
if (track.status === "failed") {
mediaChannelEl.textContent = "Download failed — pick another result or try again.";
} else if (track.status === "downloading") {
mediaChannelEl.textContent = "Preparing…";
} else {
mediaChannelEl.textContent = track.channel || (track.kind === "stream" ? "Live stream" : "");
}
if (track.art_url) {
mediaArt.src = track.art_url;
mediaArt.hidden = false;
mediaArtPlaceholder.hidden = true;
} else {
mediaArt.hidden = true;
mediaArtPlaceholder.hidden = false;
}
}
function updateMediaProgress(track, job) {
if (!job) {
mediaProgressWrap.hidden = true;
return;
}
mediaProgressWrap.hidden = false;
const pct = Math.round(job.progress || 0);
mediaProgressFill.style.width = `${pct}%`;
const stage = job.stage === "transcoding"
? "Re-encoding…"
: job.stage === "queued" ? "Queued…" : "Downloading…";
mediaProgressLabel.textContent = `${stage} ${pct}%`;
if (track) updateMediaNowCard(track);
}
function updateMediaBadge(s) {
const pending = (s.pending || []).length;
const playing = !mediaAudio.paused && !!mediaCurrentTrackId;
if (pending > 0) {
mediaBadge.hidden = false;
mediaBadge.textContent = String(pending);
} else if (playing) {
mediaBadge.hidden = false;
mediaBadge.innerHTML = "&#9835;";
} else {
mediaBadge.hidden = true;
}
}
// ── Playback controls ───────────────────────────────────────────────────────
function updateMediaControls() {
const playing = !mediaAudio.paused && !!mediaCurrentTrackId;
playIcon.hidden = playing;
pauseIcon.hidden = !playing;
playPauseBtn.setAttribute("aria-label", playing ? "Pause" : "Play");
updateMediaBadge({ pending: [] });
}
function fmtTime(sec) {
if (!isFinite(sec) || sec < 0) return "0:00";
const s = Math.floor(sec % 60);
const m = Math.floor((sec / 60) % 60);
const h = Math.floor(sec / 3600);
return h ? `${h}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}` : `${m}:${String(s).padStart(2, "0")}`;
}
function updateSeekUI() {
if (!isFinite(mediaAudio.duration) || mediaAudio.duration <= 0) {
seekBar.value = 0;
mediaDurEl.textContent = "0:00";
mediaCurEl.textContent = fmtTime(mediaAudio.currentTime);
return;
}
seekBar.max = Math.floor(mediaAudio.duration);
seekBar.value = Math.floor(mediaAudio.currentTime);
mediaCurEl.textContent = fmtTime(mediaAudio.currentTime);
mediaDurEl.textContent = fmtTime(mediaAudio.duration);
}
playPauseBtn.addEventListener("click", () => {
if (!mediaCurrentTrackId || !mediaAudio.src) return;
if (mediaAudio.paused) {
mediaAudio.play().catch((e) => console.warn("play failed:", e));
mediaPost("/control", { action: "resume" }).catch(() => {});
} else {
mediaAudio.pause();
mediaPost("/control", { action: "pause" }).catch(() => {});
}
});
rewBtn.addEventListener("click", () => {
if (!mediaAudio.src) return;
mediaAudio.currentTime = Math.max(0, mediaAudio.currentTime - 10);
reportMediaPlayback(!mediaAudio.paused ? "playing" : "paused");
});
fwdBtn.addEventListener("click", () => {
if (!mediaAudio.src || !isFinite(mediaAudio.duration)) return;
mediaAudio.currentTime = Math.min(mediaAudio.duration - 0.5, mediaAudio.currentTime + 10);
reportMediaPlayback(!mediaAudio.paused ? "playing" : "paused");
});
stopMediaBtn.addEventListener("click", () => {
clearMediaAudio();
mediaPost("/control", { action: "stop" }).catch(() => {});
publishMediaState("stopped");
});
seekBar.addEventListener("change", () => {
if (!mediaAudio.src || !isFinite(mediaAudio.duration)) return;
mediaAudio.currentTime = Number(seekBar.value);
reportMediaPlayback(!mediaAudio.paused ? "playing" : "paused");
});
volumeBar.addEventListener("input", () => {
mediaAudio.volume = Number(volumeBar.value) / 100;
try { localStorage.setItem("media_volume", volumeBar.value); } catch (e) { /* ok */ }
});
mediaAudio.volume = Number(localStorage.getItem("media_volume") ?? 90) / 100;
volumeBar.value = Math.round(mediaAudio.volume * 100);
mediaAudio.addEventListener("play", () => {
updateMediaControls();
reportMediaPlayback("playing");
publishMediaState("playing");
});
mediaAudio.addEventListener("pause", () => {
updateMediaControls();
if (mediaCurrentTrackId) {
reportMediaPlayback("paused");
publishMediaState("paused");
}
});
mediaAudio.addEventListener("ended", () => {
updateMediaControls();
reportMediaPlayback("stopped");
publishMediaState("stopped");
mediaPost("/control", { action: "stop" }).catch(() => {});
});
mediaAudio.addEventListener("timeupdate", () => {
updateSeekUI();
const now = Date.now();
if (!mediaAudio.paused && now - mediaLastReport > 5000) {
mediaLastReport = now;
reportMediaPlayback("playing");
}
});
mediaAudio.addEventListener("durationchange", updateSeekUI);
mediaAudio.addEventListener("error", () => {
// An empty src (track still downloading) also fires this — only surface
// real playback failures.
if (mediaCurrentTrackId && mediaAudio.getAttribute("src")) {
mediaChannelEl.textContent = "Playback error — the track may still be preparing.";
}
});
// ── Search / choices / library ──────────────────────────────────────────────
function fmtListDuration(sec) {
const s = Number(sec);
return isFinite(s) && s > 0 ? fmtTime(s) : "";
}
function mediaRowActions() {
const play = document.createElement("button");
play.className = "btn btn-secondary media-row-play";
play.type = "button";
play.textContent = "Play";
return play;
}
function renderMediaResults(results) {
mediaResultsEl.innerHTML = "";
if (!results || results.length === 0) {
mediaResultsEl.innerHTML = '<div class="media-empty">No results.</div>';
return;
}
for (const r of results) {
const row = document.createElement("div");
row.className = "media-row";
const info = document.createElement("div");
info.className = "media-row-info";
const dur = fmtListDuration(r.duration);
info.innerHTML = `<strong>${escapeHtml(r.title)}</strong>` +
`<span>${escapeHtml(r.channel || "")}${dur ? " · " + dur : ""}` +
`${r.in_library ? " · in library" : ""}</span>`;
const play = mediaRowActions();
play.addEventListener("click", () => playVideoId(r));
row.appendChild(info);
row.appendChild(play);
mediaResultsEl.appendChild(row);
}
}
async function playVideoId(r) {
try {
await mediaPost("/play", {
video_id: r.video_id,
title: r.title,
channel: r.channel || "",
});
} catch (e) {
mediaChannelEl.textContent = `Could not start: ${e.message}`;
}
}
mediaSearchForm.addEventListener("submit", async (event) => {
event.preventDefault();
const query = mediaSearchInput.value.trim();
if (!query) return;
mediaResultsEl.innerHTML = '<div class="media-empty">Searching YouTube Music…</div>';
try {
const data = await mediaPost("/search", { query, limit: 6 });
renderMediaResults(data.results);
} catch (e) {
mediaResultsEl.innerHTML = `<div class="media-empty">Search failed: ${escapeHtml(e.message)}</div>`;
}
});
function renderMediaChoices(choices) {
mediaChoicesWrap.hidden = choices.length === 0;
mediaChoicesEl.innerHTML = "";
for (const c of choices) {
const row = document.createElement("div");
row.className = "media-row media-choice";
const info = document.createElement("div");
info.className = "media-row-info";
const dur = fmtListDuration(c.duration);
info.innerHTML = `<strong>${escapeHtml(c.title)}</strong>` +
`<span>${escapeHtml(c.channel || "")}${dur ? " · " + dur : ""}</span>`;
const play = mediaRowActions();
play.addEventListener("click", () => playVideoId(c));
row.appendChild(info);
row.appendChild(play);
mediaChoicesEl.appendChild(row);
}
}
async function refreshLibrary() {
let data;
try {
data = await fetch("/media/api/library").then((r) => r.json());
} catch (e) {
return;
}
mediaLibraryEl.innerHTML = "";
const tracks = data.tracks || [];
if (tracks.length === 0) {
mediaLibraryEl.innerHTML = '<div class="media-empty">Downloaded songs and streams live here.</div>';
return;
}
for (const t of tracks) {
const row = document.createElement("div");
row.className = "media-row";
const info = document.createElement("div");
info.className = "media-row-info";
const dur = fmtListDuration(t.duration);
const statusTag = t.status === "ready" ? "" : ` · ${t.status}`;
info.innerHTML = `<strong>${escapeHtml(t.title)}</strong>` +
`<span>${escapeHtml(t.channel || (t.kind === "stream" ? "Live stream" : ""))}` +
`${dur ? " · " + dur : ""}${statusTag}</span>`;
const play = mediaRowActions();
play.disabled = t.status !== "ready";
play.addEventListener("click", () => {
mediaPost("/play", { track_id: t.id }).catch((e) => {
mediaChannelEl.textContent = `Could not play: ${e.message}`;
});
});
const del = document.createElement("button");
del.className = "btn btn-icon media-row-delete";
del.type = "button";
del.setAttribute("aria-label", `Remove ${t.title}`);
del.title = "Remove from library";
del.innerHTML = '<svg viewBox="0 0 24 24" width="14" height="14" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" aria-hidden="true"><path d="M18 6 6 18M6 6l12 12"></path></svg>';
del.addEventListener("click", async () => {
try {
await fetch(`/media/api/tracks/${encodeURIComponent(t.id)}`, { method: "DELETE" });
if (t.id === mediaCurrentTrackId) clearMediaAudio();
refreshLibrary();
} catch (e) { /* leave the row; next refresh decides */ }
});
row.appendChild(info);
row.appendChild(play);
row.appendChild(del);
mediaLibraryEl.appendChild(row);
}
}
streamForm.addEventListener("submit", async (event) => {
event.preventDefault();
const name = streamNameInput.value.trim() || streamUrlInput.value.trim();
const url = streamUrlInput.value.trim();
if (!url) return;
try {
await mediaPost("/streams", { name, url });
streamNameInput.value = "";
streamUrlInput.value = "";
refreshLibrary();
} catch (e) {
mediaChannelEl.textContent = `Could not add stream: ${e.message}`;
}
});
// ── Panel open/close ────────────────────────────────────────────────────────
function setMediaOpen(open) {
mediaPanel.classList.toggle("open", open);
mediaBtn.setAttribute("aria-expanded", String(open));
if (open) {
refreshLibrary();
pollMediaState();
}
}
mediaBtn.setAttribute("aria-expanded", "false");
mediaBtn.addEventListener("click", () => setMediaOpen(!mediaPanel.classList.contains("open")));
closeMediaBtn.addEventListener("click", () => setMediaOpen(false));
startMediaPolling();
updateMediaControls();
+84
View File
@@ -42,6 +42,11 @@
<span>Tasks</span> <span>Tasks</span>
<span class="tasks-badge" id="tasksBadge" hidden>0</span> <span class="tasks-badge" id="tasksBadge" hidden>0</span>
</button> </button>
<button id="mediaBtn" class="btn btn-secondary tasks-trigger" aria-label="Music player" title="Music player">
<svg viewBox="0 0 24 24" width="17" height="17" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M9 18V5l12-2v13"></path><circle cx="6" cy="18" r="3"></circle><circle cx="18" cy="16" r="3"></circle></svg>
<span>Music</span>
<span class="tasks-badge" id="mediaBadge" hidden>&#9835;</span>
</button>
</div> </div>
</header> </header>
@@ -149,6 +154,85 @@
</div> </div>
</div> </div>
<div class="tasks-panel media-panel" id="mediaPanel" aria-label="Music player">
<div class="tasks-header">
<div>
<h2>Music</h2>
<p>YouTube Music downloads and live streams, streamed to you.</p>
</div>
<button id="closeMediaBtn" class="btn btn-icon" aria-label="Close music player">
<svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" aria-hidden="true"><path d="M18 6 6 18M6 6l12 12"></path></svg>
</button>
</div>
<div class="media-now">
<img id="mediaArt" class="media-art" alt="" hidden>
<div class="media-art-placeholder" id="mediaArtPlaceholder" aria-hidden="true">
<svg viewBox="0 0 24 24" width="22" height="22" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" aria-hidden="true"><path d="M9 18V5l12-2v13"></path><circle cx="6" cy="18" r="3"></circle><circle cx="18" cy="16" r="3"></circle></svg>
</div>
<div class="media-now-info">
<strong id="mediaTitle">Nothing loaded</strong>
<span id="mediaChannel"></span>
<div class="media-progress" id="mediaProgressWrap" hidden>
<div class="media-progress-fill" id="mediaProgressFill"></div>
<span class="media-progress-label" id="mediaProgressLabel">Fetching&hellip;</span>
</div>
</div>
</div>
<div class="media-controls" role="group" aria-label="Playback controls">
<button id="rewBtn" class="btn btn-icon" aria-label="Back 10 seconds" title="Back 10s">
<svg viewBox="0 0 24 24" width="19" height="19" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M11 17a5 5 0 1 0-5-5"></path><path d="M11 17H6v-5"></path><path d="m19 5-4 4h-3"></path><path d="m15 9 4-4"></path></svg>
</button>
<button id="playPauseBtn" class="btn btn-icon media-play" aria-label="Play or pause" title="Play / pause">
<svg id="playIcon" viewBox="0 0 24 24" width="22" height="22" fill="currentColor" aria-hidden="true"><path d="M8 5v14l11-7Z"></path></svg>
<svg id="pauseIcon" viewBox="0 0 24 24" width="22" height="22" fill="currentColor" aria-hidden="true" hidden><path d="M7 5h4v14H7zM13 5h4v14h-4z"></path></svg>
</button>
<button id="fwdBtn" class="btn btn-icon" aria-label="Forward 10 seconds" title="Forward 10s">
<svg viewBox="0 0 24 24" width="19" height="19" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M13 17a5 5 0 1 1 5-5"></path><path d="M13 17h5v-5"></path><path d="m5 5 4 4h3"></path><path d="m9 9 4-4"></path></svg>
</button>
<button id="stopMediaBtn" class="btn btn-icon" aria-label="Stop" title="Stop">
<svg viewBox="0 0 24 24" width="18" height="18" fill="currentColor" aria-hidden="true"><rect x="6" y="6" width="12" height="12" rx="2"></rect></svg>
</button>
</div>
<div class="media-seek">
<span class="media-time" id="mediaCur">0:00</span>
<input type="range" id="seekBar" min="0" max="1000" value="0" step="1" aria-label="Seek position">
<span class="media-time" id="mediaDur">0:00</span>
</div>
<div class="media-volume">
<svg viewBox="0 0 24 24" width="15" height="15" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M11 5 6 9H2v6h4l5 4V5Z"></path><path d="M15.5 8.5a5 5 0 0 1 0 7"></path></svg>
<input type="range" id="volumeBar" min="0" max="100" value="90" aria-label="Volume">
</div>
<h3 class="sheet-section-title">Search YouTube Music</h3>
<form class="media-search" id="mediaSearchForm">
<input id="mediaSearchInput" type="search" placeholder="Song, artist, lyrics&hellip;" autocomplete="off">
<button class="btn btn-secondary" type="submit">Find</button>
</form>
<div class="media-list" id="mediaResults"></div>
<div id="mediaChoicesWrap" hidden>
<h3 class="sheet-section-title">Hope&rsquo;s suggestions</h3>
<div class="media-list" id="mediaChoices"></div>
</div>
<h3 class="sheet-section-title">Library</h3>
<div class="media-list" id="mediaLibrary">
<div class="media-empty">Downloaded songs and streams live here.</div>
</div>
<h3 class="sheet-section-title">Add live stream</h3>
<form class="media-search media-stream-form" id="streamForm">
<input id="streamName" type="text" placeholder="Station name" autocomplete="off">
<input id="streamUrl" type="url" placeholder="https://&hellip; audio stream URL" required>
<button class="btn btn-secondary" type="submit">Add</button>
</form>
</div>
<audio id="mediaAudio" preload="metadata"></audio>
<script src="livekit-client.umd.js"></script> <script src="livekit-client.umd.js"></script>
<script src="app.js"></script> <script src="app.js"></script>
</body> </body>
+639
View File
@@ -0,0 +1,639 @@
#!/usr/bin/env python3
"""Media server for the Hope voice assistant.
A small FastAPI service that owns the permanent music library and player
coordination:
- Library: downloaded YouTube Music tracks (mp3, transcoded via yt-dlp +
ffmpeg) and user-added live audio stream URLs, persisted in library.json.
- Search: yt-dlp ``ytsearch`` (YouTube Music pages share video ids), used by
Hope's media MCP toolset and by the web UI search box.
- Download jobs: one-at-a-time worker thread; progress is parsed from
yt-dlp stdout and surfaced through GET /media/api/state.
- Player state: per-room desired state (loaded track, playing/paused/stopped,
seek hints). The browser is the actual player and reconciles against this;
it reports real playback state back so Hope's voice tools can answer
"what's playing".
Security posture matches the token server: bind 127.0.0.1 only, reachability
exclusively through the nginx HTTPS proxy (/media/api/). File bytes for
downloaded tracks are served as nginx static files (/media/files/, native
HTTP Range support for seeking); this service only proxies external stream
URLs (mixed-content: an http:// radio URL would be blocked by the browser,
so it is fetched server-side and re-served over HTTPS).
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import tempfile
import threading
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Any
import httpx
import uvicorn
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, RedirectResponse, StreamingResponse
MEDIA_DIR = Path(os.environ.get("MEDIA_DIR", "/data/media"))
FILES_DIR = MEDIA_DIR / "files"
THUMBS_DIR = MEDIA_DIR / "thumbs"
JOBS_DIR = MEDIA_DIR / "jobs"
LIBRARY_FILE = MEDIA_DIR / "library.json"
COOKIES_FILE = os.environ.get("YTDLP_COOKIES", "")
MEDIA_HOST = os.environ.get("MEDIA_SERVER_HOST", "127.0.0.1")
MEDIA_PORT = int(os.environ.get("MEDIA_SERVER_PORT", "8092"))
DOWNLOAD_TIMEOUT = int(os.environ.get("MEDIA_DOWNLOAD_TIMEOUT", "600"))
SEARCH_TIMEOUT = int(os.environ.get("MEDIA_SEARCH_TIMEOUT", "45"))
app = FastAPI(title="hope-media", docs_url=None, redoc_url=None, openapi_url=None)
_lock = threading.Lock() # guards library + rooms below
_library: dict[str, Any] = {"tracks": {}, "version": 1}
_rooms: dict[str, dict[str, Any]] = {}
_jobs: dict[str, dict[str, Any]] = {}
_executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="media-dl")
for _d in (FILES_DIR, THUMBS_DIR, JOBS_DIR):
_d.mkdir(parents=True, exist_ok=True)
# The bind mount may be inspected/cleaned from the host by a non-root
# user (Unraid share convention) — keep the dirs world-traversable.
try:
os.chmod(_d, 0o777)
except OSError:
pass
# ── Library store ───────────────────────────────────────────────────────────
def _load_library() -> None:
if LIBRARY_FILE.exists():
try:
data = json.loads(LIBRARY_FILE.read_text(encoding="utf-8"))
tracks = data.get("tracks", {})
if isinstance(tracks, dict):
_library["tracks"] = tracks
_library["version"] = int(data.get("version", 1))
except (json.JSONDecodeError, OSError):
# Corrupt library: keep the broken file aside, start fresh.
try:
LIBRARY_FILE.replace(LIBRARY_FILE.with_suffix(".json.broken"))
except OSError:
pass
def _save_library() -> None:
tmp = LIBRARY_FILE.with_suffix(".json.tmp")
tmp.write_text(json.dumps(_library), encoding="utf-8")
tmp.replace(LIBRARY_FILE)
def _bump_version() -> int:
_library["version"] = int(_library.get("version", 1)) + 1
return _library["version"]
def _track_dict(track: dict) -> dict:
out = dict(track)
if track["kind"] == "youtube":
if track.get("status") == "ready":
out["audio_url"] = f"/media/files/{track['id']}.mp3"
out["art_url"] = (
f"/media/thumbs/{track['id']}.jpg"
if (THUMBS_DIR / f"{track['id']}.jpg").exists()
else None
)
else:
out["audio_url"] = f"/media/api/stream/{track['id']}"
return out
def _get_track(track_id: str) -> dict | None:
return _library["tracks"].get(track_id)
def _upsert_track(track: dict) -> None:
_library["tracks"][track["id"]] = track
_bump_version()
_save_library()
_load_library()
# ── Player state (per room) ─────────────────────────────────────────────────
def _room_state(room: str) -> dict:
return _rooms.setdefault(
room,
{"track_id": None, "desired": "stopped", "reported": None,
"seek_hint": None, "choices": []},
)
def _room_snapshot(room: str) -> dict:
st = _room_state(room)
track = _get_track(st["track_id"]) if st["track_id"] else None
pending = [
{
"job_id": j["id"],
"track_id": j["track_id"],
"stage": j["stage"],
"progress": j["progress"],
"error": j.get("error"),
}
for j in _jobs.values()
if j["stage"] in ("queued", "downloading", "transcoding")
]
return {
"room": room,
"track": _track_dict(track) if track else None,
"desired": st["desired"],
"reported": st["reported"],
"seek_hint": st["seek_hint"],
"choices": st["choices"],
"pending": pending,
"version": _library["version"],
}
# ── yt-dlp helpers ──────────────────────────────────────────────────────────
def _ytdlp_base_args() -> list[str]:
args = [sys.executable, "-m", "yt_dlp", "--no-playlist", "--no-warnings"]
if COOKIES_FILE and Path(COOKIES_FILE).exists():
args += ["--cookies", COOKIES_FILE]
return args
_PROGRESS_RE = re.compile(r"\[download\]\s+([\d.]+)%")
def _run_search(query: str, limit: int) -> list[dict]:
"""Flat ytsearch lookup — cheap, no download, works anonymously."""
cmd = _ytdlp_base_args() + [
"--flat-playlist", "--dump-json",
f"ytsearch{limit}:{query}",
]
try:
proc = subprocess.run(
cmd, capture_output=True, text=True, timeout=SEARCH_TIMEOUT, check=False
)
except subprocess.TimeoutExpired:
raise HTTPException(504, "YouTube Music search timed out")
if proc.returncode != 0 and not proc.stdout.strip():
detail = (proc.stderr or "").strip().splitlines()[-1:] or ["unknown error"]
raise HTTPException(502, f"Search failed: {detail[0][:300]}")
results = []
for line in proc.stdout.splitlines():
line = line.strip()
if not line.startswith("{"):
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
video_id = entry.get("id")
if not video_id:
continue
existing = _get_track(video_id)
results.append({
"video_id": video_id,
"title": entry.get("title") or video_id,
"channel": entry.get("uploader") or entry.get("channel") or "",
"duration": entry.get("duration"),
"view_count": entry.get("view_count"),
"in_library": bool(existing),
"track_id": existing["id"] if existing else None,
"library_status": existing["status"] if existing else None,
})
return results
def _job_dict(job: dict) -> dict:
return {k: job.get(k) for k in
("id", "track_id", "stage", "progress", "error", "started_at")}
def _persist_job(job: dict) -> None:
try:
(JOBS_DIR / f"{job['id']}.json").write_text(
json.dumps(_job_dict(job)), encoding="utf-8"
)
except OSError:
pass
def _fail_job(job: dict, track: dict, message: str) -> None:
job["stage"] = "error"
job["error"] = message[:500]
_persist_job(job)
with _lock:
track["status"] = "failed"
track["error"] = message[:500]
_bump_version()
_save_library()
def _download_worker(job: dict, track: dict, video_id: str) -> None:
"""Runs in the single-worker executor: yt-dlp download + mp3 transcode."""
target = FILES_DIR / f"{track['id']}.mp3"
cmd = _ytdlp_base_args() + [
"-f", "bestaudio/best",
"-x", "--audio-format", "mp3", "--audio-quality", "192K",
"--embed-thumbnail", "--add-metadata",
"--write-thumbnail", "--convert-thumbnails", "jpg",
"--newline", "--progress",
"--print-json",
"--retries", "3", "--fragment-retries", "3",
"-P", str(FILES_DIR),
"-o", f"{track['id']}.%(ext)s",
f"https://www.youtube.com/watch?v={video_id}",
]
job["stage"] = "downloading"
info: dict = {}
# stderr goes to a temp file: reading it from a PIPE while the stdout
# loop blocks could deadlock if yt-dlp ever fills the stderr buffer.
with tempfile.TemporaryFile(mode="w+", encoding="utf-8", errors="replace") as errfile:
try:
proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=errfile,
text=True, errors="replace",
)
deadline = time.monotonic() + DOWNLOAD_TIMEOUT
assert proc.stdout is not None
for line in proc.stdout:
if time.monotonic() > deadline:
proc.kill()
proc.wait(timeout=30)
raise RuntimeError(f"Download exceeded {DOWNLOAD_TIMEOUT}s")
line = line.strip()
m = _PROGRESS_RE.search(line)
if m:
job["progress"] = min(85.0, float(m.group(1)) * 0.85)
_persist_job(job)
elif line.startswith(("[ExtractAudio]", "[Metadata]", "[EmbedThumbnail]")):
job["stage"] = "transcoding"
job["progress"] = max(job["progress"], 86.0)
_persist_job(job)
elif line.startswith("{"):
try:
info = json.loads(line)
except json.JSONDecodeError:
pass
proc.wait(timeout=30)
if proc.returncode != 0:
errfile.seek(0)
stderr_tail = (errfile.read() or "").strip().splitlines()
detail = stderr_tail[-1] if stderr_tail else f"exit code {proc.returncode}"
raise RuntimeError(detail)
if not target.exists():
raise RuntimeError("yt-dlp finished but no mp3 was produced")
except Exception as exc: # noqa: BLE001 — any failure marks the track failed
_fail_job(job, track, str(exc))
return
thumb_src = FILES_DIR / f"{track['id']}.jpg"
try:
if thumb_src.exists():
thumb_src.replace(THUMBS_DIR / f"{track['id']}.jpg")
except OSError:
pass
job["stage"] = "ready"
job["progress"] = 100.0
_persist_job(job)
with _lock:
track["status"] = "ready"
track["error"] = None
if info:
if info.get("title"):
track["title"] = info["title"]
if info.get("uploader") or info.get("channel"):
track["channel"] = info.get("uploader") or info.get("channel")
if info.get("duration"):
track["duration"] = round(float(info["duration"]))
_bump_version()
_save_library()
def _start_download(track: dict, video_id: str) -> dict:
job = {
"id": uuid.uuid4().hex[:12],
"track_id": track["id"],
"stage": "queued",
"progress": 0.0,
"error": None,
"started_at": time.time(),
}
_jobs[job["id"]] = job
_persist_job(job)
_executor.submit(_download_worker, job, track, video_id)
return job
# ── API: health, library ────────────────────────────────────────────────────
@app.get("/media/api/healthz")
def healthz() -> dict:
return {"ok": True, "version": _library["version"]}
@app.get("/media/api/library")
def get_library() -> dict:
with _lock:
tracks = sorted(
(_track_dict(t) for t in _library["tracks"].values()),
key=lambda t: t.get("added_at", 0),
reverse=True,
)
return {"tracks": tracks, "version": _library["version"]}
def _delete_track_files(track: dict) -> None:
if track["kind"] != "youtube":
return
for path in (
FILES_DIR / f"{track['id']}.mp3",
FILES_DIR / f"{track['id']}.jpg",
THUMBS_DIR / f"{track['id']}.jpg",
):
try:
path.unlink(missing_ok=True)
except OSError:
pass
@app.delete("/media/api/tracks/{track_id}")
def delete_track(track_id: str) -> dict:
with _lock:
track = _get_track(track_id)
if not track:
raise HTTPException(404, "unknown track")
del _library["tracks"][track_id]
_bump_version()
_save_library()
for room, st in _rooms.items():
if st["track_id"] == track_id:
st["track_id"] = None
st["desired"] = "stopped"
_delete_track_files(track)
return {"ok": True}
# ── API: search + play + control ────────────────────────────────────────────
@app.post("/media/api/search")
async def search(request: Request) -> dict:
body = await request.json()
query = str(body.get("query", "")).strip()
if not query:
raise HTTPException(400, "query is required")
try:
limit = max(1, min(int(body.get("limit", 6)), 15))
except (TypeError, ValueError):
raise HTTPException(400, "limit must be an integer")
return {"query": query, "results": _run_search(query, limit)}
def _resolve_play_target(body: dict) -> dict:
"""Find or create the track a play request refers to.
Accepts video_id (YouTube), track_id (existing library entry) or a raw
url+name (live stream). Returns the library track dict.
"""
video_id = str(body.get("video_id") or "").strip()
track_id = str(body.get("track_id") or "").strip()
url = str(body.get("url") or "").strip()
if video_id:
existing = _get_track(video_id)
if existing:
return existing
track = {
"id": video_id,
"kind": "youtube",
"video_id": video_id,
"title": str(body.get("title") or video_id),
"channel": str(body.get("channel") or ""),
"duration": body.get("duration"),
"status": "downloading",
"error": None,
"added_at": time.time(),
}
_upsert_track(track)
_start_download(track, video_id)
return track
if track_id:
track = _get_track(track_id)
if not track:
raise HTTPException(404, "unknown track")
return track
if url:
if not re.match(r"^https?://", url):
raise HTTPException(400, "url must be http(s)")
track_id = "s" + uuid.uuid4().hex[:10]
track = {
"id": track_id,
"kind": "stream",
"video_id": None,
"title": str(body.get("name") or url),
"channel": str(body.get("channel") or ""),
"duration": None,
"status": "ready",
"error": None,
"url": url,
"added_at": time.time(),
}
_upsert_track(track)
return track
raise HTTPException(400, "one of video_id, track_id or url is required")
@app.post("/media/api/play")
async def play(request: Request) -> dict:
body = await request.json()
room = str(body.get("room") or "default")
track = _resolve_play_target(body)
with _lock:
st = _room_state(room)
st["track_id"] = track["id"]
# start_paused carries a track into a new scope without starting it —
# the UI uses this when a conversation begins (talking pauses music).
st["desired"] = "paused" if body.get("start_paused") else "playing"
st["choices"] = [] # a pick resolves any open suggestion list
st["seek_hint"] = None
snapshot = _room_snapshot(room)
return {"ok": True, **snapshot}
@app.post("/media/api/control")
async def control(request: Request) -> dict:
body = await request.json()
room = str(body.get("room") or "default")
action = str(body.get("action") or "")
with _lock:
st = _room_state(room)
if action == "pause":
st["desired"] = "paused"
elif action == "resume":
if not st["track_id"]:
raise HTTPException(409, "nothing loaded")
st["desired"] = "playing"
elif action == "stop":
st["desired"] = "stopped"
elif action == "seek":
try:
position = max(0.0, float(body.get("position", 0)))
except (TypeError, ValueError):
raise HTTPException(400, "position must be a number")
st["seek_hint"] = {"position": position, "ts": time.time()}
elif action == "skip":
current = st["track_id"]
candidates = sorted(
(
t for t in _library["tracks"].values()
if t.get("status") == "ready" and t["id"] != current
),
key=lambda t: t.get("added_at", 0),
reverse=True,
)
if candidates:
st["track_id"] = candidates[0]["id"]
st["desired"] = "playing"
st["seek_hint"] = None
else:
st["desired"] = "stopped"
else:
raise HTTPException(400, f"unknown action: {action}")
snapshot = _room_snapshot(room)
return {"ok": True, **snapshot}
@app.post("/media/api/choices")
async def choices(request: Request) -> dict:
"""Publish Hope's song suggestions so the UI can render clickable picks."""
body = await request.json()
room = str(body.get("room") or "default")
raw = body.get("choices") or []
cleaned = []
for c in raw[:5]:
if not isinstance(c, dict) or not c.get("video_id"):
continue
cleaned.append({
"video_id": str(c["video_id"]),
"title": str(c.get("title") or c["video_id"]),
"channel": str(c.get("channel") or ""),
"duration": c.get("duration"),
})
with _lock:
_room_state(room)["choices"] = cleaned
return {"ok": True, "choices": cleaned}
@app.get("/media/api/state")
def state(room: str = "default") -> dict:
with _lock:
return _room_snapshot(room)
@app.post("/media/api/report")
async def report(request: Request) -> dict:
"""The browser reports real playback state (source of truth for the player)."""
body = await request.json()
room = str(body.get("room") or "default")
try:
position = float(body.get("position") or 0.0)
except (TypeError, ValueError):
position = 0.0
with _lock:
st = _room_state(room)
st["reported"] = {
"track_id": body.get("track_id"),
"state": str(body.get("state") or "stopped"),
"position": position,
"duration": body.get("duration"),
"ts": time.time(),
}
return {"ok": True}
@app.post("/media/api/streams")
async def add_stream(request: Request) -> dict:
body = await request.json()
track = _resolve_play_target({"url": body.get("url"), "name": body.get("name")})
return {"ok": True, "track": _track_dict(track)}
# ── API: stream proxy (external http(s) audio over same-origin HTTPS) ───────
_HOP_HEADERS = ("content-type", "content-range", "accept-ranges", "content-length")
@app.get("/media/api/stream/{track_id}")
async def stream_proxy(track_id: str, request: Request):
with _lock:
track = _get_track(track_id)
if not track:
raise HTTPException(404, "unknown track")
if track["kind"] == "youtube":
# Downloaded tracks are nginx static files (native Range support).
if track.get("status") == "ready":
return RedirectResponse(f"/media/files/{track_id}.mp3")
raise HTTPException(409, f"track is {track.get('status')}, not ready")
range_header = request.headers.get("range")
headers = {"Range": range_header} if range_header else {}
client = httpx.AsyncClient(timeout=httpx.Timeout(30, read=600), follow_redirects=True)
try:
upstream = await client.send(client.build_request("GET", track["url"], headers=headers), stream=True)
except httpx.HTTPError as exc:
await client.aclose()
raise HTTPException(502, f"upstream stream failed: {exc}")
if upstream.status_code >= 400:
await upstream.aclose()
await client.aclose()
raise HTTPException(502, f"upstream returned {upstream.status_code}")
out_headers = {
name: value
for name, value in upstream.headers.items()
if name.lower() in _HOP_HEADERS
}
media_type = upstream.headers.get("content-type", "audio/mpeg")
async def relay():
try:
async for chunk in upstream.aiter_bytes(64 * 1024):
yield chunk
finally:
await upstream.aclose()
await client.aclose()
return StreamingResponse(
relay(),
status_code=upstream.status_code,
media_type=media_type,
headers=out_headers,
)
@app.exception_handler(HTTPException)
async def http_error(_request: Request, exc: HTTPException):
return JSONResponse({"error": exc.detail}, status_code=exc.status_code)
if __name__ == "__main__":
uvicorn.run(app, host=MEDIA_HOST, port=MEDIA_PORT, log_level="info")
+259 -1
View File
@@ -1245,6 +1245,263 @@ body[data-mode="text"] .mic-meter {
.task-result { margin-top: 0.5rem; color: #bcebd8; background: rgba(118, 217, 178, 0.1); } .task-result { margin-top: 0.5rem; color: #bcebd8; background: rgba(118, 217, 178, 0.1); }
.task-error { margin-top: 0.5rem; color: #ffc2ba; background: rgba(255, 128, 111, 0.1); } .task-error { margin-top: 0.5rem; color: #ffc2ba; background: rgba(255, 128, 111, 0.1); }
/* ── Media player ───────────────────────────────────────────────────────── */
/* Reuses .tasks-panel positioning (fixed right, slides in via .open); the
content scrolls under a sticky header. */
.media-panel {
overflow-y: auto;
padding-bottom: 1.2rem;
}
.media-panel .tasks-header {
position: sticky;
top: 0;
z-index: 1;
background: #0d1929;
}
.media-now {
display: flex;
align-items: center;
gap: 0.75rem;
margin: 1rem 1.1rem 0;
padding: 0.75rem;
border-radius: 12px;
background: var(--surface-raised);
}
.media-art,
.media-art-placeholder {
width: 56px;
height: 56px;
flex: 0 0 56px;
border-radius: 8px;
object-fit: cover;
background: rgba(23, 41, 65, 0.7);
}
.media-art-placeholder {
display: flex;
align-items: center;
justify-content: center;
color: var(--ink-muted);
}
.media-now-info {
flex: 1;
min-width: 0;
display: flex;
flex-direction: column;
gap: 0.15rem;
}
.media-now-info strong {
color: var(--ink);
font-size: 0.88rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-now-info span {
color: var(--ink-muted);
font-size: 0.72rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-progress {
position: relative;
margin-top: 0.35rem;
display: flex;
align-items: center;
gap: 0.5rem;
}
/* Track bar behind the fill (fill width is driven inline by progress). */
.media-progress::before {
content: "";
position: absolute;
left: 0;
top: 50%;
transform: translateY(-50%);
width: 90px;
height: 4px;
border-radius: 2px;
background: rgba(139, 167, 255, 0.18);
}
.media-progress-fill {
height: 4px;
flex: 0 0 auto;
width: 0;
border-radius: 2px;
background: var(--iris);
transition: width 400ms ease;
}
.media-progress-label {
color: var(--ink-muted);
font-size: 0.66rem;
white-space: nowrap;
}
.media-controls {
display: flex;
align-items: center;
justify-content: center;
gap: 0.9rem;
margin: 0.9rem 1.1rem 0;
}
.media-controls .btn-icon {
color: var(--ink-soft);
}
.media-play {
width: 52px;
height: 52px;
border-radius: 50%;
display: inline-flex;
align-items: center;
justify-content: center;
color: #0b1523 !important;
background: var(--iris) !important;
border: none;
}
.media-play:hover:not(:disabled) {
filter: brightness(1.1);
}
.media-seek {
display: flex;
align-items: center;
gap: 0.55rem;
margin: 0.75rem 1.1rem 0;
}
.media-time {
color: var(--ink-muted);
font-size: 0.68rem;
font-variant-numeric: tabular-nums;
min-width: 2.6rem;
text-align: center;
}
.media-seek input[type="range"],
.media-volume input[type="range"] {
flex: 1;
accent-color: var(--iris);
cursor: pointer;
}
.media-volume {
display: flex;
align-items: center;
gap: 0.55rem;
margin: 0.55rem 1.1rem 0;
color: var(--ink-muted);
}
.media-search {
display: flex;
gap: 0.45rem;
margin: 0 1.1rem;
}
.media-search input {
flex: 1;
min-width: 0;
padding: 0.5rem 0.7rem;
border: 1px solid var(--line-strong);
border-radius: 9px;
color: var(--ink);
background: rgba(5, 11, 20, 0.45);
font-size: 0.8rem;
}
.media-search input::placeholder {
color: var(--ink-muted);
}
.media-stream-form input:first-child {
flex: 0 0 38%;
}
.media-list {
display: flex;
flex-direction: column;
gap: 0.4rem;
margin: 0.65rem 1.1rem 1rem;
max-height: 240px;
overflow-y: auto;
}
.media-row {
display: flex;
align-items: center;
gap: 0.6rem;
padding: 0.55rem 0.65rem;
border-radius: 9px;
background: var(--surface-raised);
}
.media-row-info {
flex: 1;
min-width: 0;
display: flex;
flex-direction: column;
gap: 0.1rem;
}
.media-row-info strong {
color: var(--ink);
font-size: 0.78rem;
font-weight: 600;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-row-info span {
color: var(--ink-muted);
font-size: 0.68rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-row-play {
flex: 0 0 auto;
padding: 0.3rem 0.7rem;
font-size: 0.7rem;
}
.media-row-delete {
flex: 0 0 auto;
color: var(--ink-muted);
}
.media-row-delete:hover {
color: #ffc2ba;
}
.media-empty {
padding: 0.8rem;
border-radius: 9px;
color: var(--ink-muted);
background: var(--surface-raised);
font-size: 0.74rem;
text-align: center;
}
.media-choice {
border: 1px solid rgba(139, 167, 255, 0.35);
background: rgba(60, 95, 201, 0.12);
}
/* ── Responsive ─────────────────────────────────────────────────────────── */ /* ── Responsive ─────────────────────────────────────────────────────────── */
@media (max-height: 760px) and (min-width: 881px) { @media (max-height: 760px) and (min-width: 881px) {
.workspace { .workspace {
@@ -1386,7 +1643,8 @@ body[data-mode="text"] .mic-meter {
padding: 1.25rem 1rem; padding: 1.25rem 1rem;
} }
.tasks-panel { .tasks-panel,
.media-panel {
width: 100vw; width: 100vw;
} }
+3
View File
@@ -97,6 +97,9 @@ class Handler(BaseHTTPRequestHandler):
resp = json.dumps( resp = json.dumps(
{ {
"token": token, "token": token,
# The room name scopes media-server player state; the UI sends
# it with every /media/api/ call.
"room": room_name,
"user": { "user": {
"id": user_id, "id": user_id,
"display_name": profile.get("display_name") or user_id, "display_name": profile.get("display_name") or user_id,