feat: media player API, web player panel, and Hope music MCP

- web/media_server.py: FastAPI service owning the permanent library
  (library.json + ./media volume), yt-dlp YouTube Music search, queued
  download jobs with mp3 transcode (ffmpeg), per-room player desired
  state, and an HTTPS proxy for external live-stream URLs.
- agent/media_mcp.py: Hope tools — search_music, play_song,
  offer_song_choices, pause/resume/stop/skip/seek, now_playing,
  list_library, add_stream.
- agent/agent.py: media toolset wiring + conversation rule (music
  playing disables the agent mic; re-engaging pauses music).
- web UI: slide-out Music panel (search, suggestions, library, live
  stream add, play/pause, ±10s, seek, volume) with a reconciling poller
  and media_state data-channel reports.
- token server returns the minted room name; nginx serves /media/files
  as static (native Range) and proxies /media/api to the media server.
- supervisord runs media-server; compose mounts ./media and adds
  ffmpeg to the image.
This commit is contained in:
snowspeeder committed 2026-08-29 13:03:12 -04:00
1 parent a4ae96f01b
commit 04c4a53594
16 files changed
+1876 -12

No files matched your search

+13
View File
@@ -29,6 +29,19 @@ FIRECRAWL_BASE=http://192.168.86.2:3002
# Per-user memory profiles and Bearer tokens live in config/users.json. # Per-user memory profiles and Bearer tokens live in config/users.json.
VOICE_DEFAULT_USER=shane VOICE_DEFAULT_USER=shane
# ── Media player (music library + YouTube Music downloads) ─────────────────
# The permanent library lives in ./media (bind-mounted to /data/media):
# media/files/ transcoded mp3 tracks
# media/thumbs/ cover art
# media/jobs/ download job status
# MEDIA_SERVER_PORT=8092
#
# Optional: export cookies.txt from a browser logged into YouTube and drop it
# in ./media/cookies.txt to avoid "confirm you're not a bot" download blocks.
# Anonymous access works from most residential IPs; cookies are the reliable
# path.
# YTDLP_COOKIES=/data/media/cookies.txt
# Extra MCP servers (optional). JSON list of {url, transport} objects. # Extra MCP servers (optional). JSON list of {url, transport} objects.
# transport: "sse" | "streamable_http" | omit for auto-detect # transport: "sse" | "streamable_http" | omit for auto-detect
# EXTRA_MCP_SERVERS=[{"url":"http://192.168.86.2:3001/mcp","transport":"streamable_http"}] # EXTRA_MCP_SERVERS=[{"url":"http://192.168.86.2:3001/mcp","transport":"streamable_http"}]
+9 -3
View File
@@ -24,9 +24,10 @@ ENV PYTHONUNBUFFERED=1
# Fix SSL cert path for the slim base image (ca-certificates installs to /etc/ssl/certs) # Fix SSL cert path for the slim base image (ca-certificates installs to /etc/ssl/certs)
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
# Install LiveKit server binary + supervisord + nginx # Install LiveKit server binary + supervisord + nginx + ffmpeg (yt-dlp audio
# extraction/transcoding for the media player)
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
curl ca-certificates supervisor nginx libasound2 iproute2 \ curl ca-certificates supervisor nginx libasound2 iproute2 ffmpeg \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# Download LiveKit server (latest stable) # Download LiveKit server (latest stable)
@@ -48,8 +49,9 @@ COPY agent/task_registry.py /opt/voice-agent/task_registry.py
COPY agent/task_router.py /opt/voice-agent/task_router.py COPY agent/task_router.py /opt/voice-agent/task_router.py
COPY agent/task_worker.py /opt/voice-agent/task_worker.py COPY agent/task_worker.py /opt/voice-agent/task_worker.py
COPY agent/dispatch_mcp.py /opt/voice-agent/dispatch_mcp.py COPY agent/dispatch_mcp.py /opt/voice-agent/dispatch_mcp.py
COPY agent/media_mcp.py /opt/voice-agent/media_mcp.py
# Copy web frontend + token endpoint # Copy web frontend + token endpoint + media server
COPY web/index.html /var/www/voice/ COPY web/index.html /var/www/voice/
COPY web/app.js /var/www/voice/ COPY web/app.js /var/www/voice/
COPY web/style.css /var/www/voice/ COPY web/style.css /var/www/voice/
@@ -57,6 +59,10 @@ COPY web/livekit-client.umd.js /var/www/voice/
COPY web/manifest.json /var/www/voice/ COPY web/manifest.json /var/www/voice/
COPY web/favicon.svg /var/www/voice/ COPY web/favicon.svg /var/www/voice/
COPY web/token_server.py /opt/voice/token_server.py COPY web/token_server.py /opt/voice/token_server.py
COPY web/media_server.py /opt/voice/media_server.py
# Permanent media library mount point (bind-mounted from ./media)
RUN mkdir -p /data/media/files /data/media/thumbs /data/media/jobs
# Config files # Config files
COPY livekit.yaml /etc/livekit.yaml COPY livekit.yaml /etc/livekit.yaml
+64 -2
View File
@@ -196,6 +196,23 @@ SYSTEM_PROMPT = textwrap.dedent("""\
you receive it, share the findings naturally in your conversational style. you receive it, share the findings naturally in your conversational style.
You can have multiple tasks running at once. You can have multiple tasks running at once.
# Music (media player in the web UI)
You can play music on the user's web player. For any music request, call
search_music first, then:
- One clearly-right result → play_song with its video_id. Announce it in
one short sentence, then stop talking — playback pauses the
conversation until the user comes back to you.
- Ambiguous results → offer_song_choices with the top 2-3 options, name
them briefly, and let the user pick by voice or in the player panel.
Never start playback on a guess.
The library is permanent: list_library shows what is already downloaded,
and those tracks play instantly without searching. If the user names a
library track exactly, just play it. Control playback with pause_music,
resume_music, skip_song and seek_music; check what's on with now_playing.
add_stream registers a live radio or direct audio URL as a library track.
While music is playing you cannot hear the user — they will come back to
you when they want to talk again, and the music pauses automatically.
# Skills # Skills
You have a skill library of learned procedures. Use skill_recall when you need You have a skill library of learned procedures. Use skill_recall when you need
to perform a task you've done before — it will give you the steps. When you to perform a task you've done before — it will give you the steps. When you
@@ -624,6 +641,27 @@ def build_mcp_toolsets(user_profile: dict, task_dir: str) -> list[mcp.MCPToolset
) )
logger.info("Dispatch MCP toolset enabled (%s)", task_dir) logger.info("Dispatch MCP toolset enabled (%s)", task_dir)
# Media tools talk to the local media server; the room name scopes player
# state so concurrent sessions don't fight over one virtual player.
media_server_url = os.environ.get("MEDIA_SERVER_URL", "http://127.0.0.1:8092")
media_mcp_script = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media_mcp.py")
toolsets.append(
mcp.MCPToolset(
id="media",
mcp_server=mcp.MCPServerStdio(
command=python_bin,
args=[media_mcp_script],
env={
**os.environ,
"MEDIA_SERVER_URL": media_server_url,
"MEDIA_ROOM": os.path.basename(task_dir.rstrip("/")) or "default",
},
client_session_timeout_seconds=60,
),
)
)
logger.info("Media MCP toolset enabled (%s)", media_server_url)
extra = os.environ.get("EXTRA_MCP_SERVERS", "") extra = os.environ.get("EXTRA_MCP_SERVERS", "")
if extra: if extra:
try: try:
@@ -740,9 +778,22 @@ async def handle_job(ctx: JobContext) -> None:
# Listen for conversation controls (voice selection and voice/text mode). # Listen for conversation controls (voice selection and voice/text mode).
# Defined here so it captures this job's session via closure. # Defined here so it captures this job's session via closure.
# Music and conversation are mutually exclusive by design: while a track
# plays the agent's ears are off; re-engaging the conversation pauses the
# music, and ending the conversation lets the user resume it.
session_mode = {"value": "voice"} # UI voice/text switch
media_playback = {"value": "stopped"} # UI-reported player state
def _apply_listening() -> None:
listening = (
session_mode["value"] == "voice"
and media_playback["value"] != "playing"
)
session.input.set_audio_enabled(listening)
def _on_room_data(packet) -> None: def _on_room_data(packet) -> None:
topic = getattr(packet, "topic", None) topic = getattr(packet, "topic", None)
if topic and topic not in ("voice-control", "conversation-control"): if topic and topic not in ("voice-control", "conversation-control", "media-control"):
return return
try: try:
msg = json.loads(packet.data.decode("utf-8")) msg = json.loads(packet.data.decode("utf-8"))
@@ -755,11 +806,22 @@ async def handle_job(ctx: JobContext) -> None:
return return
voice_enabled = mode == "voice" voice_enabled = mode == "voice"
logger.info("Switching conversation mode to %s", mode) logger.info("Switching conversation mode to %s", mode)
session_mode["value"] = mode
if not voice_enabled: if not voice_enabled:
# interrupt() returns an already-scheduled Future, not a coroutine. # interrupt() returns an already-scheduled Future, not a coroutine.
session.interrupt() session.interrupt()
session.input.set_audio_enabled(voice_enabled)
session.output.set_audio_enabled(voice_enabled) session.output.set_audio_enabled(voice_enabled)
_apply_listening()
return
if msg.get("type") == "media_state":
state = msg.get("state")
if state not in ("playing", "paused", "stopped"):
return
logger.info("Media playback -> %s (agent listening: %s)",
state, session_mode["value"] == "voice" and state != "playing")
media_playback["value"] = state
_apply_listening()
return return
if msg.get("type") == "set_voice": if msg.get("type") == "set_voice":
+229
View File
@@ -0,0 +1,229 @@
"""Media MCP server — lets Hope find and control music by voice.
Tools talk to the local media server (web/media_server.py) over HTTP, which
owns the library, YouTube Music search/download jobs and per-room player
state. The browser is the actual player: it reconciles against the media
server and reports playback back, so voice controls and the web panel stay
in sync.
Per-session environment (set by the agent when spawning this MCP):
- MEDIA_SERVER_URL e.g. http://127.0.0.1:8092
- MEDIA_ROOM LiveKit room name this session controls
The "stop listening while music plays" rule lives in the agent itself: the
UI reports media_state over the data channel and the agent disables its mic
while a track is playing.
"""
from __future__ import annotations
import json
import os
import httpx
from mcp.server.fastmcp import FastMCP
MEDIA_SERVER_URL = os.environ.get("MEDIA_SERVER_URL", "http://127.0.0.1:8092").rstrip("/")
MEDIA_ROOM = os.environ.get("MEDIA_ROOM", "default")
mcp = FastMCP("media")
def _media_get(path: str, params: dict | None = None) -> dict:
resp = httpx.get(f"{MEDIA_SERVER_URL}{path}", params=params, timeout=10)
resp.raise_for_status()
return resp.json()
def _media_post(path: str, body: dict) -> dict:
resp = httpx.post(f"{MEDIA_SERVER_URL}{path}", json=body, timeout=30)
if resp.status_code >= 400:
try:
detail = resp.json().get("error", resp.text)
except Exception: # noqa: BLE001
detail = resp.text
return {"error": f"{resp.status_code}: {detail}"}
return resp.json()
def _control(action: str, **extra) -> str:
data = _media_post("/media/api/control", {"room": MEDIA_ROOM, "action": action, **extra})
if "error" in data:
return f"Could not {action} the music: {data['error']}"
track = data.get("track") or {}
desired = data.get("desired", "")
name = track.get("title") or "nothing"
return f"OK — {action} applied. Loaded track: {name} (player state: {desired})."
def _fmt_duration(seconds) -> str:
try:
seconds = int(seconds or 0)
except (TypeError, ValueError):
return ""
if seconds <= 0:
return ""
minutes, secs = divmod(seconds, 60)
hours, minutes = divmod(minutes, 60)
return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}"
@mcp.tool()
def search_music(query: str, limit: int = 5) -> str:
"""Search YouTube Music for songs. Call this before playing anything the
user describes by name, artist or lyrics. Returns JSON results with
video_id, title, channel, duration and in_library flags. Pick the result
that clearly matches what the user asked for; if two or more could be
right, use offer_song_choices instead of guessing."""
data = _media_post("/media/api/search", {"query": query, "limit": max(1, min(limit, 10))})
if "error" in data:
return f"Search failed: {data['error']}"
results = data.get("results", [])
if not results:
return f"No YouTube Music results for '{query}'."
return json.dumps(results, ensure_ascii=False)
@mcp.tool()
def play_song(video_id: str, title: str = "", channel: str = "") -> str:
"""Play a song in the user's web player. Pass the video_id from
search_music (or an in_library track_id). Downloads and transcodes first
if needed; playback starts automatically once ready. After calling this,
tell the user what is playing in one short sentence and stop talking —
the player takes over and the conversation pauses until they return."""
data = _media_post("/media/api/play", {
"room": MEDIA_ROOM, "video_id": video_id, "title": title, "channel": channel,
})
if "error" in data:
return f"Could not start playback: {data['error']}"
track = data.get("track") or {}
status = track.get("status")
name = track.get("title") or title or video_id
if status == "downloading":
return (
f"'{name}' is downloading and will start playing automatically when "
"ready (usually under a minute). Say it's on the way, then hand over "
"to the player."
)
return f"Playing '{name}' now. Hand over to the player with a short sign-off."
@mcp.tool()
def offer_song_choices(choices: list[dict]) -> str:
"""Offer the user 2-4 song options when a search is ambiguous. Pass a list
like [{"video_id": "...", "title": "...", "channel": "...", "duration": 213}].
The choices appear as clickable cards in the music panel; the user can
also just say which one they want. Briefly name the options aloud, then
wait for their pick."""
if not choices:
return "No choices provided."
data = _media_post("/media/api/choices", {"room": MEDIA_ROOM, "choices": choices})
if "error" in data:
return f"Could not present choices: {data['error']}"
names = [
f"{c.get('title')}" + (f" by {c['channel']}" if c.get("channel") else "")
for c in choices
]
return "Choices shown in the player: " + "; ".join(names) + ". Ask the user to pick one."
@mcp.tool()
def pause_music() -> str:
"""Pause the currently playing music. The user stays in the conversation
afterwards."""
return _control("pause")
@mcp.tool()
def resume_music() -> str:
"""Resume the loaded music. Warn the user you'll stop listening while it
plays — they'll need to come back to you to talk again."""
return _control("resume")
@mcp.tool()
def stop_music() -> str:
"""Stop playback completely and clear the loaded track."""
return _control("stop")
@mcp.tool()
def skip_song() -> str:
"""Skip to the next track in the library (most recently added first).
If the library has nothing else, playback stops."""
return _control("skip")
@mcp.tool()
def seek_music(delta_seconds: int) -> str:
"""Skip forward or backward in the current track. Positive delta_seconds
jumps ahead, negative goes back (e.g. -30 rewinds 30 seconds, 20 skips
20 seconds forward)."""
try:
reported = (_media_get("/media/api/state", {"room": MEDIA_ROOM}).get("reported") or {})
position = float(reported.get("position") or 0.0)
except Exception: # noqa: BLE001
position = 0.0
target = max(0.0, position + float(delta_seconds))
return _control("seek", position=target)
@mcp.tool()
def now_playing() -> str:
"""Return what is currently loaded/playing in the user's web player and
its playback position. Use when the user asks what's playing or what this
song is."""
try:
st = _media_get("/media/api/state", {"room": MEDIA_ROOM})
except Exception as exc: # noqa: BLE001
return f"Media server unreachable: {exc}"
track = st.get("track") or {}
if not track:
return "Nothing is loaded in the player right now."
reported = st.get("reported") or {}
state = reported.get("state") or st.get("desired") or "unknown"
position = _fmt_duration(reported.get("position"))
duration = _fmt_duration(track.get("duration"))
bits = [f"'{track.get('title')}'"]
if track.get("channel"):
bits.append(f"by {track['channel']}")
if state == "playing" and position:
bits.append(f"at {position}" + (f" of {duration}" if duration else ""))
else:
bits.append(f"(playback state: {state})")
return "Currently loaded: " + " ".join(bits) + "."
@mcp.tool()
def list_library() -> str:
"""List the permanent music library: downloaded YouTube Music tracks and
added live streams, ready to play instantly. Use when the user asks what
music is available or wants something from their collection."""
data = _media_get("/media/api/library")
tracks = data.get("tracks", [])
if not tracks:
return "The library is empty — search for something with search_music first."
lines = []
for t in tracks[:25]:
status = "" if t.get("status") == "ready" else f" ({t.get('status')})"
dur = _fmt_duration(t.get("duration"))
dur = f" [{dur}]" if dur else ""
channel = f" — {t['channel']}" if t.get("channel") else ""
lines.append(f"{t['title']}{channel}{dur}{status}")
return f"{len(tracks)} tracks in the library:\n" + "\n".join(lines)
@mcp.tool()
def add_stream(name: str, url: str) -> str:
"""Add a live audio stream URL (internet radio station, direct mp3/aac
stream) to the permanent library. It then shows up in the web player's
library and can be played by name. Pass a short friendly name."""
if not url.lower().startswith(("http://", "https://")):
return "That does not look like an http(s) stream URL."
data = _media_post("/media/api/streams", {"name": name, "url": url})
if "error" in data:
return f"Could not add the stream: {data['error']}"
return f"Added '{name}' to the library as a live stream. You can play it on request."
if __name__ == "__main__":
mcp.run(transport="stdio")
+4
View File
@@ -9,10 +9,14 @@ description = "Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit
requires-python = ">=3.10,<3.15" requires-python = ">=3.10,<3.15"
dependencies = [ dependencies = [
"fastapi>=0.141.1",
"httpx>=0.28.1",
"livekit-agents[mcp]~=1.7", "livekit-agents[mcp]~=1.7",
"livekit-plugins-azure~=1.7", "livekit-plugins-azure~=1.7",
"livekit-plugins-openai~=1.7", "livekit-plugins-openai~=1.7",
"python-dotenv", "python-dotenv",
"uvicorn>=0.52.4",
"yt-dlp>=2026.8.19",
] ]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
Generated
+33
View File
@@ -671,6 +671,22 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" }, { url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" },
] ]
[[package]]
name = "fastapi"
version = "0.141.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "annotated-doc" },
{ name = "pydantic" },
{ name = "starlette" },
{ name = "typing-extensions" },
{ name = "typing-inspection" },
]
sdist = { url = "https://files.pythonhosted.org/packages/8a/02/91e3416a8fdd715abb903a952a6bec7cdd8d14eed55d415fc8595524c319/fastapi-0.141.1.tar.gz", hash = "sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1", size = 425799, upload-time = "2026-07-29T17:18:05.568Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/cb/03/10388a42375ee7e4ac9b94eb2c5c569c8b5795e377e701c9ac3ad63de890/fastapi-0.141.1-py3-none-any.whl", hash = "sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3", size = 131954, upload-time = "2026-07-29T17:18:04.364Z" },
]
[[package]] [[package]]
name = "frozenlist" name = "frozenlist"
version = "1.8.0" version = "1.8.0"
@@ -2744,18 +2760,26 @@ name = "voice-agent"
version = "0.1.0" version = "0.1.0"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "fastapi" },
{ name = "httpx" },
{ name = "livekit-agents", extra = ["mcp"] }, { name = "livekit-agents", extra = ["mcp"] },
{ name = "livekit-plugins-azure" }, { name = "livekit-plugins-azure" },
{ name = "livekit-plugins-openai" }, { name = "livekit-plugins-openai" },
{ name = "python-dotenv" }, { name = "python-dotenv" },
{ name = "uvicorn" },
{ name = "yt-dlp" },
] ]
[package.metadata] [package.metadata]
requires-dist = [ requires-dist = [
{ name = "fastapi", specifier = ">=0.141.1" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "livekit-agents", extras = ["mcp"], specifier = "~=1.7" }, { name = "livekit-agents", extras = ["mcp"], specifier = "~=1.7" },
{ name = "livekit-plugins-azure", specifier = "~=1.7" }, { name = "livekit-plugins-azure", specifier = "~=1.7" },
{ name = "livekit-plugins-openai", specifier = "~=1.7" }, { name = "livekit-plugins-openai", specifier = "~=1.7" },
{ name = "python-dotenv" }, { name = "python-dotenv" },
{ name = "uvicorn", specifier = ">=0.52.4" },
{ name = "yt-dlp", specifier = ">=2026.8.19" },
] ]
[[package]] [[package]]
@@ -3038,3 +3062,12 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/bf/f4/ed5c402ac8fde4403ed3366c2716bfddc8a6677ebd59f3d62772cc7fe468/yarl-1.24.5-cp314-cp314t-win_arm64.whl", hash = "sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688", size = 97222, upload-time = "2026-07-20T02:07:41.55Z" }, { url = "https://files.pythonhosted.org/packages/bf/f4/ed5c402ac8fde4403ed3366c2716bfddc8a6677ebd59f3d62772cc7fe468/yarl-1.24.5-cp314-cp314t-win_arm64.whl", hash = "sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688", size = 97222, upload-time = "2026-07-20T02:07:41.55Z" },
{ url = "https://files.pythonhosted.org/packages/61/02/962c1cbfc401a30c1d034dc67ff395f64b52302c6d62de556c1fca99acc0/yarl-1.24.5-py3-none-any.whl", hash = "sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7", size = 58612, upload-time = "2026-07-20T02:07:43.461Z" }, { url = "https://files.pythonhosted.org/packages/61/02/962c1cbfc401a30c1d034dc67ff395f64b52302c6d62de556c1fca99acc0/yarl-1.24.5-py3-none-any.whl", hash = "sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7", size = 58612, upload-time = "2026-07-20T02:07:43.461Z" },
] ]
[[package]]
name = "yt-dlp"
version = "2026.8.19"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/1e/e0/832fa4ca334b766a06933a196066edc3dba37cdb6f14cd98d59bcc69a4b4/yt_dlp-2026.8.19.tar.gz", hash = "sha256:9e213e48cea35c66b378e4447903f118f6392a5fa380a2b6d7070ec86f4e0af1", size = 3052025, upload-time = "2026-08-19T23:48:59.291Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/69/b2/8cd1613f56eed7ceb64fbd4df3f1c01246bfb098e6f398228bafda22b80b/yt_dlp-2026.8.19-py3-none-any.whl", hash = "sha256:1d57897e94c6665a0a6f9bc54b34e584284e32c034ffab3a7df25d8f7b24eedf", size = 3185533, upload-time = "2026-08-19T23:48:56.925Z" },
]
+4
View File
@@ -3,7 +3,11 @@ Name: voice-agent
Version: 0.1.0 Version: 0.1.0
Summary: Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit Agents Summary: Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit Agents
Requires-Python: <3.15,>=3.10 Requires-Python: <3.15,>=3.10
Requires-Dist: fastapi>=0.141.1
Requires-Dist: httpx>=0.28.1
Requires-Dist: livekit-agents[mcp]~=1.7 Requires-Dist: livekit-agents[mcp]~=1.7
Requires-Dist: livekit-plugins-azure~=1.7 Requires-Dist: livekit-plugins-azure~=1.7
Requires-Dist: livekit-plugins-openai~=1.7 Requires-Dist: livekit-plugins-openai~=1.7
Requires-Dist: python-dotenv Requires-Dist: python-dotenv
Requires-Dist: uvicorn>=0.52.4
Requires-Dist: yt-dlp>=2026.8.19
+4
View File
@@ -1,4 +1,8 @@
fastapi>=0.141.1
httpx>=0.28.1
livekit-agents[mcp]~=1.7 livekit-agents[mcp]~=1.7
livekit-plugins-azure~=1.7 livekit-plugins-azure~=1.7
livekit-plugins-openai~=1.7 livekit-plugins-openai~=1.7
python-dotenv python-dotenv
uvicorn>=0.52.4
yt-dlp>=2026.8.19
+5 -1
View File
@@ -18,12 +18,16 @@ services:
WEB_PORT: "${WEB_PORT:-9178}" WEB_PORT: "${WEB_PORT:-9178}"
VOICE_USERS_FILE: "/etc/hope/users.json" VOICE_USERS_FILE: "/etc/hope/users.json"
VOICE_DEFAULT_USER: "${VOICE_DEFAULT_USER:-shane}" VOICE_DEFAULT_USER: "${VOICE_DEFAULT_USER:-shane}"
MEDIA_DIR: "/data/media"
MEDIA_SERVER_PORT: "${MEDIA_SERVER_PORT:-8092}"
YTDLP_COOKIES: "${YTDLP_COOKIES:-}"
volumes: volumes:
- ./livekit.yaml:/etc/livekit.yaml:ro - ./livekit.yaml:/etc/livekit.yaml:ro
- ./certs:/etc/voice/certs - ./certs:/etc/voice/certs
- ./skills:/skills - ./skills:/skills
- ./media:/data/media
healthcheck: healthcheck:
test: ["CMD-SHELL", "curl -sk https://localhost:$${WEB_PORT:-9178}/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null"] test: ["CMD-SHELL", "curl -sk https://localhost:$${WEB_PORT:-9178}/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null && curl -s http://localhost:$${MEDIA_SERVER_PORT:-8092}/media/api/healthz -o /dev/null"]
interval: 15s interval: 15s
timeout: 5s timeout: 5s
retries: 3 retries: 3
+20
View File
@@ -23,6 +23,26 @@ server {
proxy_send_timeout 3600s; proxy_send_timeout 3600s;
} }
# Media server API (library, search, player state) and the proxy for
# external live-stream URLs (http:// sources would be blocked as mixed
# content in the browser, so they are re-served over HTTPS from here).
location /media/api/ {
proxy_pass http://127.0.0.1:8092/media/api/;
proxy_buffering off;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
# Downloaded tracks are plain static files — nginx serves them with
# native HTTP Range support, which gives the web player seeking for free.
location /media/files/ {
alias /data/media/files/;
}
location /media/thumbs/ {
alias /data/media/thumbs/;
}
location / { location / {
try_files $uri $uri/ =404; try_files $uri $uri/ =404;
} }
+15
View File
@@ -56,3 +56,18 @@ environment=
LIVEKIT_API_SECRET="%(ENV_LIVEKIT_API_SECRET)s", LIVEKIT_API_SECRET="%(ENV_LIVEKIT_API_SECRET)s",
VOICE_USERS_FILE="%(ENV_VOICE_USERS_FILE)s", VOICE_USERS_FILE="%(ENV_VOICE_USERS_FILE)s",
VOICE_DEFAULT_USER="%(ENV_VOICE_DEFAULT_USER)s" VOICE_DEFAULT_USER="%(ENV_VOICE_DEFAULT_USER)s"
[program:media-server]
; Runs as root because the /data/media bind mount is a host (Unraid) share
; whose uid does not match the container's voiceuser.
command=/opt/voice-agent/.venv/bin/python /opt/voice/media_server.py
user=root
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true
environment=
MEDIA_DIR="%(ENV_MEDIA_DIR)s",
MEDIA_SERVER_PORT="%(ENV_MEDIA_SERVER_PORT)s",
YTDLP_COOKIES="%(ENV_YTDLP_COOKIES)s"
+511 -5
View File
@@ -693,6 +693,7 @@ document.addEventListener("keydown", (event) => {
if (event.key !== "Escape") return; if (event.key !== "Escape") return;
if (settingsSheet.classList.contains("open")) closeSheet(); if (settingsSheet.classList.contains("open")) closeSheet();
if (tasksPanel.classList.contains("open")) setTasksOpen(false); if (tasksPanel.classList.contains("open")) setTasksOpen(false);
if (mediaPanel.classList.contains("open")) setMediaOpen(false);
}); });
// ── Wake lock ─────────────────────────────────────────────────────────────── // ── Wake lock ───────────────────────────────────────────────────────────────
async function acquireWakeLock() { async function acquireWakeLock() {
@@ -816,6 +817,8 @@ function createRoom(token) {
stopBtn.disabled = true; stopBtn.disabled = true;
agentParticipant = null; agentParticipant = null;
agentReady = false; agentReady = false;
// Leaving the conversation hands the room back to the music player.
resumePlaybackAfterTalk();
}); });
return newRoom.connect(LIVEKIT_URL, token, { return newRoom.connect(LIVEKIT_URL, token, {
@@ -873,8 +876,9 @@ async function connectConversation() {
await acquireWakeLock(); await acquireWakeLock();
// Get a signed access token from the token endpoint. // Get a signed access token from the token endpoint.
const token = await fetchToken(ROOM_NAME); const tokenData = await fetchToken(ROOM_NAME);
room = await createRoom(token); await enterMediaScope(tokenData.room);
room = await createRoom(tokenData.token);
// The agent is dispatched when we join. If it misses the dispatch // The agent is dispatched when we join. If it misses the dispatch
// (for example during a server restart), reconnect once. // (for example during a server restart), reconnect once.
@@ -886,8 +890,9 @@ async function connectConversation() {
agentParticipant = null; agentParticipant = null;
agentReady = false; agentReady = false;
await stale.disconnect(); await stale.disconnect();
const freshToken = await fetchToken(ROOM_NAME); const freshTokenData = await fetchToken(ROOM_NAME);
room = await createRoom(freshToken); await enterMediaScope(freshTokenData.room);
room = await createRoom(freshTokenData.token);
if (!(await waitForAgent(15000))) { if (!(await waitForAgent(15000))) {
throw new Error("Assistant did not join — reload the page and try again"); throw new Error("Assistant did not join — reload the page and try again");
} }
@@ -999,6 +1004,8 @@ stopBtn.addEventListener("click", () => {
muteBtn.disabled = true; muteBtn.disabled = true;
micMuted = false; micMuted = false;
updateMuteIcon(); updateMuteIcon();
// Ending the conversation resumes music that was paused for the talk.
resumePlaybackAfterTalk();
}); });
// ── Mic mute toggle ───────────────────────────────────────────────────────── // ── Mic mute toggle ─────────────────────────────────────────────────────────
@@ -1030,7 +1037,7 @@ async function fetchToken(roomName) {
}); });
if (!resp.ok) throw new Error(`Token endpoint returned ${resp.status}`); if (!resp.ok) throw new Error(`Token endpoint returned ${resp.status}`);
const data = await resp.json(); const data = await resp.json();
return data.token; return { token: data.token, room: data.room || null };
} }
// ── Helper: send voice selection to the agent via data channel ───────────── // ── Helper: send voice selection to the agent via data channel ─────────────
@@ -1064,3 +1071,502 @@ function sendModelMessage(model, targetRoom) {
reliable: true, reliable: true,
}); });
} }
// ── Media player ────────────────────────────────────────────────────────────
// The browser <audio> element is the player. The media server owns library +
// per-room desired state; this UI reconciles against GET /media/api/state
// (polling) and reports real playback back. Hope drives the same state via
// her media tools, so voice commands and panel clicks stay in sync.
//
// Conversation rule (mirrored in agent.py): while a track plays the agent's
// mic is off. Connecting to Hope pauses music; ending the conversation
// resumes what was playing.
const mediaBtn = document.getElementById("mediaBtn");
const mediaPanel = document.getElementById("mediaPanel");
const closeMediaBtn = document.getElementById("closeMediaBtn");
const mediaBadge = document.getElementById("mediaBadge");
const mediaArt = document.getElementById("mediaArt");
const mediaArtPlaceholder = document.getElementById("mediaArtPlaceholder");
const mediaTitleEl = document.getElementById("mediaTitle");
const mediaChannelEl = document.getElementById("mediaChannel");
const mediaProgressWrap = document.getElementById("mediaProgressWrap");
const mediaProgressFill = document.getElementById("mediaProgressFill");
const mediaProgressLabel = document.getElementById("mediaProgressLabel");
const rewBtn = document.getElementById("rewBtn");
const fwdBtn = document.getElementById("fwdBtn");
const playPauseBtn = document.getElementById("playPauseBtn");
const playIcon = document.getElementById("playIcon");
const pauseIcon = document.getElementById("pauseIcon");
const stopMediaBtn = document.getElementById("stopMediaBtn");
const seekBar = document.getElementById("seekBar");
const mediaCurEl = document.getElementById("mediaCur");
const mediaDurEl = document.getElementById("mediaDur");
const volumeBar = document.getElementById("volumeBar");
const mediaSearchForm = document.getElementById("mediaSearchForm");
const mediaSearchInput = document.getElementById("mediaSearchInput");
const mediaResultsEl = document.getElementById("mediaResults");
const mediaChoicesWrap = document.getElementById("mediaChoicesWrap");
const mediaChoicesEl = document.getElementById("mediaChoices");
const mediaLibraryEl = document.getElementById("mediaLibrary");
const streamForm = document.getElementById("streamForm");
const streamNameInput = document.getElementById("streamName");
const streamUrlInput = document.getElementById("streamUrl");
const mediaAudio = document.getElementById("mediaAudio");
let mediaRoom = null; // LiveKit room name (player state scope)
let mediaCurrentTrackId = null; // track id loaded into the audio element
let mediaAppliedSeekTs = 0; // last applied seek_hint timestamp
let mediaWasPlayingBeforeTalk = false;
let mediaPollTimer = null;
let mediaLastReport = 0;
function mediaScope() {
return mediaRoom || "default";
}
async function mediaPost(path, body) {
const resp = await fetch(`/media/api${path}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ room: mediaScope(), ...body }),
});
if (!resp.ok) {
const detail = await resp.json().catch(() => ({}));
throw new Error(detail.error || `media API ${path} -> ${resp.status}`);
}
return resp.json();
}
// Called just before joining the LiveKit room: carry whatever the audio
// element holds into the room-scoped player state, paused (talking pauses
// music). resumePlaybackAfterTalk() reverses this when the talk ends.
async function enterMediaScope(newRoom) {
if (!newRoom || newRoom === mediaRoom) return;
const wasPlaying = !!mediaCurrentTrackId && !mediaAudio.paused;
mediaWasPlayingBeforeTalk = wasPlaying;
const carriedTrack = mediaCurrentTrackId;
mediaRoom = newRoom;
if (carriedTrack) {
try {
await mediaPost("/play", { room: mediaRoom, track_id: carriedTrack, start_paused: true });
} catch (e) { console.warn("media scope transfer failed:", e); }
mediaAudio.pause();
}
}
async function resumePlaybackAfterTalk() {
if (!mediaWasPlayingBeforeTalk) return;
mediaWasPlayingBeforeTalk = false;
try { await mediaPost("/control", { action: "resume" }); } catch (e) { /* poll retries */ }
}
function publishMediaState(state) {
if (!room || !room.localParticipant) return;
const payload = new TextEncoder().encode(
JSON.stringify({ type: "media_state", state, track_id: mediaCurrentTrackId })
);
room.localParticipant.publishData(payload, { topic: "media-control", reliable: true });
}
function reportMediaPlayback(state) {
const body = {
state,
track_id: mediaCurrentTrackId,
position: isFinite(mediaAudio.currentTime) ? mediaAudio.currentTime : 0,
};
if (isFinite(mediaAudio.duration)) body.duration = mediaAudio.duration;
mediaPost("/report", body).catch(() => {});
}
// ── Polling + reconciliation ────────────────────────────────────────────────
function startMediaPolling() {
if (mediaPollTimer) return;
mediaPollTimer = setInterval(() => {
pollMediaState();
}, 1500);
}
async function pollMediaState() {
let s;
try {
const resp = await fetch(`/media/api/state?room=${encodeURIComponent(mediaScope())}`);
if (!resp.ok) return;
s = await resp.json();
} catch (e) {
return; // media server briefly unavailable; try again next tick
}
reconcileMedia(s);
}
function reconcileMedia(s) {
renderMediaChoices(s.choices || []);
updateMediaBadge(s);
const track = s.track;
const pendingJob = (s.pending || []).find(j => !track || j.track_id === track.id);
updateMediaProgress(track, pendingJob);
if (track && track.id !== mediaCurrentTrackId) loadMediaTrack(track);
if (!track && mediaCurrentTrackId && s.desired === "stopped") clearMediaAudio();
applyDesiredMediaState(s, track);
if (s.seek_hint && s.seek_hint.ts !== mediaAppliedSeekTs && isFinite(mediaAudio.duration)) {
mediaAppliedSeekTs = s.seek_hint.ts;
const target = Math.min(s.seek_hint.position, mediaAudio.duration);
if (Number.isFinite(target)) {
mediaAudio.currentTime = Math.max(0, target);
updateSeekUI();
}
}
}
function applyDesiredMediaState(s, track) {
if (s.desired === "playing") {
if (track && track.status === "ready" && mediaAudio.src && mediaAudio.paused) {
mediaAudio.play().catch((e) => console.warn("playback blocked:", e));
}
} else if (!mediaAudio.paused) {
mediaAudio.pause();
}
}
function loadMediaTrack(track) {
mediaCurrentTrackId = track.id;
updateMediaNowCard(track);
if (track.audio_url && (track.kind === "stream" || track.status === "ready")) {
if (mediaAudio.getAttribute("src") !== track.audio_url) {
mediaAudio.src = track.audio_url;
mediaAudio.load();
}
} else {
// Downloading or failed: no stream yet. The poll flips this to the
// real URL once status turns "ready".
mediaAudio.removeAttribute("src");
mediaAudio.load();
updateMediaControls();
}
}
function clearMediaAudio() {
mediaCurrentTrackId = null;
mediaAudio.pause();
mediaAudio.removeAttribute("src");
mediaAudio.load();
mediaTitleEl.textContent = "Nothing loaded";
mediaChannelEl.textContent = "";
mediaArt.hidden = true;
mediaArtPlaceholder.hidden = false;
updateMediaControls();
}
function updateMediaNowCard(track) {
mediaTitleEl.textContent = track.title || track.id;
mediaChannelEl.textContent = track.channel || (track.kind === "stream" ? "Live stream" : "");
if (track.art_url) {
mediaArt.src = track.art_url;
mediaArt.hidden = false;
mediaArtPlaceholder.hidden = true;
} else {
mediaArt.hidden = true;
mediaArtPlaceholder.hidden = false;
}
}
function updateMediaProgress(track, job) {
if (!job) {
mediaProgressWrap.hidden = true;
return;
}
mediaProgressWrap.hidden = false;
const pct = Math.round(job.progress || 0);
mediaProgressFill.style.width = `${pct}%`;
const stage = job.stage === "transcoding"
? "Re-encoding…"
: job.stage === "queued" ? "Queued…" : "Downloading…";
mediaProgressLabel.textContent = `${stage} ${pct}%`;
if (track) updateMediaNowCard(track);
}
function updateMediaBadge(s) {
const pending = (s.pending || []).length;
const playing = !mediaAudio.paused && !!mediaCurrentTrackId;
if (pending > 0) {
mediaBadge.hidden = false;
mediaBadge.textContent = String(pending);
} else if (playing) {
mediaBadge.hidden = false;
mediaBadge.innerHTML = "&#9835;";
} else {
mediaBadge.hidden = true;
}
}
// ── Playback controls ───────────────────────────────────────────────────────
function updateMediaControls() {
const playing = !mediaAudio.paused && !!mediaCurrentTrackId;
playIcon.hidden = playing;
pauseIcon.hidden = !playing;
playPauseBtn.setAttribute("aria-label", playing ? "Pause" : "Play");
updateMediaBadge({ pending: [] });
}
function fmtTime(sec) {
if (!isFinite(sec) || sec < 0) return "0:00";
const s = Math.floor(sec % 60);
const m = Math.floor((sec / 60) % 60);
const h = Math.floor(sec / 3600);
return h ? `${h}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}` : `${m}:${String(s).padStart(2, "0")}`;
}
function updateSeekUI() {
if (!isFinite(mediaAudio.duration) || mediaAudio.duration <= 0) {
seekBar.value = 0;
mediaDurEl.textContent = "0:00";
mediaCurEl.textContent = fmtTime(mediaAudio.currentTime);
return;
}
seekBar.max = Math.floor(mediaAudio.duration);
seekBar.value = Math.floor(mediaAudio.currentTime);
mediaCurEl.textContent = fmtTime(mediaAudio.currentTime);
mediaDurEl.textContent = fmtTime(mediaAudio.duration);
}
playPauseBtn.addEventListener("click", () => {
if (!mediaCurrentTrackId || !mediaAudio.src) return;
if (mediaAudio.paused) {
mediaAudio.play().catch((e) => console.warn("play failed:", e));
mediaPost("/control", { action: "resume" }).catch(() => {});
} else {
mediaAudio.pause();
mediaPost("/control", { action: "pause" }).catch(() => {});
}
});
rewBtn.addEventListener("click", () => {
if (!mediaAudio.src) return;
mediaAudio.currentTime = Math.max(0, mediaAudio.currentTime - 10);
reportMediaPlayback(!mediaAudio.paused ? "playing" : "paused");
});
fwdBtn.addEventListener("click", () => {
if (!mediaAudio.src || !isFinite(mediaAudio.duration)) return;
mediaAudio.currentTime = Math.min(mediaAudio.duration - 0.5, mediaAudio.currentTime + 10);
reportMediaPlayback(!mediaAudio.paused ? "playing" : "paused");
});
stopMediaBtn.addEventListener("click", () => {
clearMediaAudio();
mediaPost("/control", { action: "stop" }).catch(() => {});
publishMediaState("stopped");
});
seekBar.addEventListener("change", () => {
if (!mediaAudio.src || !isFinite(mediaAudio.duration)) return;
mediaAudio.currentTime = Number(seekBar.value);
reportMediaPlayback(!mediaAudio.paused ? "playing" : "paused");
});
volumeBar.addEventListener("input", () => {
mediaAudio.volume = Number(volumeBar.value) / 100;
try { localStorage.setItem("media_volume", volumeBar.value); } catch (e) { /* ok */ }
});
mediaAudio.volume = Number(localStorage.getItem("media_volume") ?? 90) / 100;
volumeBar.value = Math.round(mediaAudio.volume * 100);
mediaAudio.addEventListener("play", () => {
updateMediaControls();
reportMediaPlayback("playing");
publishMediaState("playing");
});
mediaAudio.addEventListener("pause", () => {
updateMediaControls();
if (mediaCurrentTrackId) {
reportMediaPlayback("paused");
publishMediaState("paused");
}
});
mediaAudio.addEventListener("ended", () => {
updateMediaControls();
reportMediaPlayback("stopped");
publishMediaState("stopped");
mediaPost("/control", { action: "stop" }).catch(() => {});
});
mediaAudio.addEventListener("timeupdate", () => {
updateSeekUI();
const now = Date.now();
if (!mediaAudio.paused && now - mediaLastReport > 5000) {
mediaLastReport = now;
reportMediaPlayback("playing");
}
});
mediaAudio.addEventListener("durationchange", updateSeekUI);
mediaAudio.addEventListener("error", () => {
if (mediaCurrentTrackId) {
mediaChannelEl.textContent = "Playback error — the track may still be preparing.";
}
});
// ── Search / choices / library ──────────────────────────────────────────────
function fmtListDuration(sec) {
const s = Number(sec);
return isFinite(s) && s > 0 ? fmtTime(s) : "";
}
function mediaRowActions() {
const play = document.createElement("button");
play.className = "btn btn-secondary media-row-play";
play.type = "button";
play.textContent = "Play";
return play;
}
function renderMediaResults(results) {
mediaResultsEl.innerHTML = "";
if (!results || results.length === 0) {
mediaResultsEl.innerHTML = '<div class="media-empty">No results.</div>';
return;
}
for (const r of results) {
const row = document.createElement("div");
row.className = "media-row";
const info = document.createElement("div");
info.className = "media-row-info";
const dur = fmtListDuration(r.duration);
info.innerHTML = `<strong>${escapeHtml(r.title)}</strong>` +
`<span>${escapeHtml(r.channel || "")}${dur ? " · " + dur : ""}` +
`${r.in_library ? " · in library" : ""}</span>`;
const play = mediaRowActions();
play.addEventListener("click", () => playVideoId(r));
row.appendChild(info);
row.appendChild(play);
mediaResultsEl.appendChild(row);
}
}
async function playVideoId(r) {
try {
await mediaPost("/play", {
video_id: r.video_id,
title: r.title,
channel: r.channel || "",
});
} catch (e) {
mediaChannelEl.textContent = `Could not start: ${e.message}`;
}
}
mediaSearchForm.addEventListener("submit", async (event) => {
event.preventDefault();
const query = mediaSearchInput.value.trim();
if (!query) return;
mediaResultsEl.innerHTML = '<div class="media-empty">Searching YouTube Music…</div>';
try {
const data = await mediaPost("/search", { query, limit: 6 });
renderMediaResults(data.results);
} catch (e) {
mediaResultsEl.innerHTML = `<div class="media-empty">Search failed: ${escapeHtml(e.message)}</div>`;
}
});
function renderMediaChoices(choices) {
mediaChoicesWrap.hidden = choices.length === 0;
mediaChoicesEl.innerHTML = "";
for (const c of choices) {
const row = document.createElement("div");
row.className = "media-row media-choice";
const info = document.createElement("div");
info.className = "media-row-info";
const dur = fmtListDuration(c.duration);
info.innerHTML = `<strong>${escapeHtml(c.title)}</strong>` +
`<span>${escapeHtml(c.channel || "")}${dur ? " · " + dur : ""}</span>`;
const play = mediaRowActions();
play.addEventListener("click", () => playVideoId(c));
row.appendChild(info);
row.appendChild(play);
mediaChoicesEl.appendChild(row);
}
}
async function refreshLibrary() {
let data;
try {
data = await fetch("/media/api/library").then((r) => r.json());
} catch (e) {
return;
}
mediaLibraryEl.innerHTML = "";
const tracks = data.tracks || [];
if (tracks.length === 0) {
mediaLibraryEl.innerHTML = '<div class="media-empty">Downloaded songs and streams live here.</div>';
return;
}
for (const t of tracks) {
const row = document.createElement("div");
row.className = "media-row";
const info = document.createElement("div");
info.className = "media-row-info";
const dur = fmtListDuration(t.duration);
const statusTag = t.status === "ready" ? "" : ` · ${t.status}`;
info.innerHTML = `<strong>${escapeHtml(t.title)}</strong>` +
`<span>${escapeHtml(t.channel || (t.kind === "stream" ? "Live stream" : ""))}` +
`${dur ? " · " + dur : ""}${statusTag}</span>`;
const play = mediaRowActions();
play.disabled = t.status !== "ready";
play.addEventListener("click", () => {
mediaPost("/play", { track_id: t.id }).catch((e) => {
mediaChannelEl.textContent = `Could not play: ${e.message}`;
});
});
const del = document.createElement("button");
del.className = "btn btn-icon media-row-delete";
del.type = "button";
del.setAttribute("aria-label", `Remove ${t.title}`);
del.title = "Remove from library";
del.innerHTML = '<svg viewBox="0 0 24 24" width="14" height="14" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" aria-hidden="true"><path d="M18 6 6 18M6 6l12 12"></path></svg>';
del.addEventListener("click", async () => {
try {
await fetch(`/media/api/tracks/${encodeURIComponent(t.id)}`, { method: "DELETE" });
if (t.id === mediaCurrentTrackId) clearMediaAudio();
refreshLibrary();
} catch (e) { /* leave the row; next refresh decides */ }
});
row.appendChild(info);
row.appendChild(play);
row.appendChild(del);
mediaLibraryEl.appendChild(row);
}
}
streamForm.addEventListener("submit", async (event) => {
event.preventDefault();
const name = streamNameInput.value.trim() || streamUrlInput.value.trim();
const url = streamUrlInput.value.trim();
if (!url) return;
try {
await mediaPost("/streams", { name, url });
streamNameInput.value = "";
streamUrlInput.value = "";
refreshLibrary();
} catch (e) {
mediaChannelEl.textContent = `Could not add stream: ${e.message}`;
}
});
// ── Panel open/close ────────────────────────────────────────────────────────
function setMediaOpen(open) {
mediaPanel.classList.toggle("open", open);
mediaBtn.setAttribute("aria-expanded", String(open));
if (open) {
refreshLibrary();
pollMediaState();
}
}
mediaBtn.setAttribute("aria-expanded", "false");
mediaBtn.addEventListener("click", () => setMediaOpen(!mediaPanel.classList.contains("open")));
closeMediaBtn.addEventListener("click", () => setMediaOpen(false));
startMediaPolling();
updateMediaControls();
+84
View File
@@ -42,6 +42,11 @@
<span>Tasks</span> <span>Tasks</span>
<span class="tasks-badge" id="tasksBadge" hidden>0</span> <span class="tasks-badge" id="tasksBadge" hidden>0</span>
</button> </button>
<button id="mediaBtn" class="btn btn-secondary tasks-trigger" aria-label="Music player" title="Music player">
<svg viewBox="0 0 24 24" width="17" height="17" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M9 18V5l12-2v13"></path><circle cx="6" cy="18" r="3"></circle><circle cx="18" cy="16" r="3"></circle></svg>
<span>Music</span>
<span class="tasks-badge" id="mediaBadge" hidden>&#9835;</span>
</button>
</div> </div>
</header> </header>
@@ -149,6 +154,85 @@
</div> </div>
</div> </div>
<div class="tasks-panel media-panel" id="mediaPanel" aria-label="Music player">
<div class="tasks-header">
<div>
<h2>Music</h2>
<p>YouTube Music downloads and live streams, streamed to you.</p>
</div>
<button id="closeMediaBtn" class="btn btn-icon" aria-label="Close music player">
<svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" aria-hidden="true"><path d="M18 6 6 18M6 6l12 12"></path></svg>
</button>
</div>
<div class="media-now">
<img id="mediaArt" class="media-art" alt="" hidden>
<div class="media-art-placeholder" id="mediaArtPlaceholder" aria-hidden="true">
<svg viewBox="0 0 24 24" width="22" height="22" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" aria-hidden="true"><path d="M9 18V5l12-2v13"></path><circle cx="6" cy="18" r="3"></circle><circle cx="18" cy="16" r="3"></circle></svg>
</div>
<div class="media-now-info">
<strong id="mediaTitle">Nothing loaded</strong>
<span id="mediaChannel"></span>
<div class="media-progress" id="mediaProgressWrap" hidden>
<div class="media-progress-fill" id="mediaProgressFill"></div>
<span class="media-progress-label" id="mediaProgressLabel">Fetching&hellip;</span>
</div>
</div>
</div>
<div class="media-controls" role="group" aria-label="Playback controls">
<button id="rewBtn" class="btn btn-icon" aria-label="Back 10 seconds" title="Back 10s">
<svg viewBox="0 0 24 24" width="19" height="19" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M11 17a5 5 0 1 0-5-5"></path><path d="M11 17H6v-5"></path><path d="m19 5-4 4h-3"></path><path d="m15 9 4-4"></path></svg>
</button>
<button id="playPauseBtn" class="btn btn-icon media-play" aria-label="Play or pause" title="Play / pause">
<svg id="playIcon" viewBox="0 0 24 24" width="22" height="22" fill="currentColor" aria-hidden="true"><path d="M8 5v14l11-7Z"></path></svg>
<svg id="pauseIcon" viewBox="0 0 24 24" width="22" height="22" fill="currentColor" aria-hidden="true" hidden><path d="M7 5h4v14H7zM13 5h4v14h-4z"></path></svg>
</button>
<button id="fwdBtn" class="btn btn-icon" aria-label="Forward 10 seconds" title="Forward 10s">
<svg viewBox="0 0 24 24" width="19" height="19" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M13 17a5 5 0 1 1 5-5"></path><path d="M13 17h5v-5"></path><path d="m5 5 4 4h3"></path><path d="m9 9 4-4"></path></svg>
</button>
<button id="stopMediaBtn" class="btn btn-icon" aria-label="Stop" title="Stop">
<svg viewBox="0 0 24 24" width="18" height="18" fill="currentColor" aria-hidden="true"><rect x="6" y="6" width="12" height="12" rx="2"></rect></svg>
</button>
</div>
<div class="media-seek">
<span class="media-time" id="mediaCur">0:00</span>
<input type="range" id="seekBar" min="0" max="1000" value="0" step="1" aria-label="Seek position">
<span class="media-time" id="mediaDur">0:00</span>
</div>
<div class="media-volume">
<svg viewBox="0 0 24 24" width="15" height="15" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M11 5 6 9H2v6h4l5 4V5Z"></path><path d="M15.5 8.5a5 5 0 0 1 0 7"></path></svg>
<input type="range" id="volumeBar" min="0" max="100" value="90" aria-label="Volume">
</div>
<h3 class="sheet-section-title">Search YouTube Music</h3>
<form class="media-search" id="mediaSearchForm">
<input id="mediaSearchInput" type="search" placeholder="Song, artist, lyrics&hellip;" autocomplete="off">
<button class="btn btn-secondary" type="submit">Find</button>
</form>
<div class="media-list" id="mediaResults"></div>
<div id="mediaChoicesWrap" hidden>
<h3 class="sheet-section-title">Hope&rsquo;s suggestions</h3>
<div class="media-list" id="mediaChoices"></div>
</div>
<h3 class="sheet-section-title">Library</h3>
<div class="media-list" id="mediaLibrary">
<div class="media-empty">Downloaded songs and streams live here.</div>
</div>
<h3 class="sheet-section-title">Add live stream</h3>
<form class="media-search media-stream-form" id="streamForm">
<input id="streamName" type="text" placeholder="Station name" autocomplete="off">
<input id="streamUrl" type="url" placeholder="https://&hellip; audio stream URL" required>
<button class="btn btn-secondary" type="submit">Add</button>
</form>
</div>
<audio id="mediaAudio" preload="metadata"></audio>
<script src="livekit-client.umd.js"></script> <script src="livekit-client.umd.js"></script>
<script src="app.js"></script> <script src="app.js"></script>
</body> </body>
+619
View File
@@ -0,0 +1,619 @@
#!/usr/bin/env python3
"""Media server for the Hope voice assistant.
A small FastAPI service that owns the permanent music library and player
coordination:
- Library: downloaded YouTube Music tracks (mp3, transcoded via yt-dlp +
ffmpeg) and user-added live audio stream URLs, persisted in library.json.
- Search: yt-dlp ``ytsearch`` (YouTube Music pages share video ids), used by
Hope's media MCP toolset and by the web UI search box.
- Download jobs: one-at-a-time worker thread; progress is parsed from
yt-dlp stdout and surfaced through GET /media/api/state.
- Player state: per-room desired state (loaded track, playing/paused/stopped,
seek hints). The browser is the actual player and reconciles against this;
it reports real playback state back so Hope's voice tools can answer
"what's playing".
Security posture matches the token server: bind 127.0.0.1 only, reachability
exclusively through the nginx HTTPS proxy (/media/api/). File bytes for
downloaded tracks are served as nginx static files (/media/files/, native
HTTP Range support for seeking); this service only proxies external stream
URLs (mixed-content: an http:// radio URL would be blocked by the browser,
so it is fetched server-side and re-served over HTTPS).
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import threading
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Any
import httpx
import uvicorn
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, RedirectResponse, StreamingResponse
MEDIA_DIR = Path(os.environ.get("MEDIA_DIR", "/data/media"))
FILES_DIR = MEDIA_DIR / "files"
THUMBS_DIR = MEDIA_DIR / "thumbs"
JOBS_DIR = MEDIA_DIR / "jobs"
LIBRARY_FILE = MEDIA_DIR / "library.json"
COOKIES_FILE = os.environ.get("YTDLP_COOKIES", "")
MEDIA_HOST = os.environ.get("MEDIA_SERVER_HOST", "127.0.0.1")
MEDIA_PORT = int(os.environ.get("MEDIA_SERVER_PORT", "8092"))
DOWNLOAD_TIMEOUT = int(os.environ.get("MEDIA_DOWNLOAD_TIMEOUT", "600"))
SEARCH_TIMEOUT = int(os.environ.get("MEDIA_SEARCH_TIMEOUT", "45"))
app = FastAPI(title="hope-media", docs_url=None, redoc_url=None, openapi_url=None)
_lock = threading.Lock() # guards library + rooms below
_library: dict[str, Any] = {"tracks": {}, "version": 1}
_rooms: dict[str, dict[str, Any]] = {}
_jobs: dict[str, dict[str, Any]] = {}
_executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="media-dl")
for _d in (FILES_DIR, THUMBS_DIR, JOBS_DIR):
_d.mkdir(parents=True, exist_ok=True)
# ── Library store ───────────────────────────────────────────────────────────
def _load_library() -> None:
if LIBRARY_FILE.exists():
try:
data = json.loads(LIBRARY_FILE.read_text(encoding="utf-8"))
tracks = data.get("tracks", {})
if isinstance(tracks, dict):
_library["tracks"] = tracks
_library["version"] = int(data.get("version", 1))
except (json.JSONDecodeError, OSError):
# Corrupt library: keep the broken file aside, start fresh.
try:
LIBRARY_FILE.replace(LIBRARY_FILE.with_suffix(".json.broken"))
except OSError:
pass
def _save_library() -> None:
tmp = LIBRARY_FILE.with_suffix(".json.tmp")
tmp.write_text(json.dumps(_library), encoding="utf-8")
tmp.replace(LIBRARY_FILE)
def _bump_version() -> int:
_library["version"] = int(_library.get("version", 1)) + 1
return _library["version"]
def _track_dict(track: dict) -> dict:
out = dict(track)
if track["kind"] == "youtube":
if track.get("status") == "ready":
out["audio_url"] = f"/media/files/{track['id']}.mp3"
out["art_url"] = (
f"/media/thumbs/{track['id']}.jpg"
if (THUMBS_DIR / f"{track['id']}.jpg").exists()
else None
)
else:
out["audio_url"] = f"/media/api/stream/{track['id']}"
return out
def _get_track(track_id: str) -> dict | None:
return _library["tracks"].get(track_id)
def _upsert_track(track: dict) -> None:
_library["tracks"][track["id"]] = track
_bump_version()
_save_library()
_load_library()
# ── Player state (per room) ─────────────────────────────────────────────────
def _room_state(room: str) -> dict:
return _rooms.setdefault(
room,
{"track_id": None, "desired": "stopped", "reported": None,
"seek_hint": None, "choices": []},
)
def _room_snapshot(room: str) -> dict:
st = _room_state(room)
track = _get_track(st["track_id"]) if st["track_id"] else None
pending = [
{
"job_id": j["id"],
"track_id": j["track_id"],
"stage": j["stage"],
"progress": j["progress"],
"error": j.get("error"),
}
for j in _jobs.values()
if j["stage"] in ("queued", "downloading", "transcoding")
]
return {
"room": room,
"track": _track_dict(track) if track else None,
"desired": st["desired"],
"reported": st["reported"],
"seek_hint": st["seek_hint"],
"choices": st["choices"],
"pending": pending,
"version": _library["version"],
}
# ── yt-dlp helpers ──────────────────────────────────────────────────────────
def _ytdlp_base_args() -> list[str]:
args = [sys.executable, "-m", "yt_dlp", "--no-playlist", "--no-warnings"]
if COOKIES_FILE and Path(COOKIES_FILE).exists():
args += ["--cookies", COOKIES_FILE]
return args
_PROGRESS_RE = re.compile(r"\[download\]\s+([\d.]+)%")
def _run_search(query: str, limit: int) -> list[dict]:
"""Flat ytsearch lookup — cheap, no download, works anonymously."""
cmd = _ytdlp_base_args() + [
"--flat-playlist", "--dump-json",
f"ytsearch{limit}:{query}",
]
try:
proc = subprocess.run(
cmd, capture_output=True, text=True, timeout=SEARCH_TIMEOUT, check=False
)
except subprocess.TimeoutExpired:
raise HTTPException(504, "YouTube Music search timed out")
if proc.returncode != 0 and not proc.stdout.strip():
detail = (proc.stderr or "").strip().splitlines()[-1:] or ["unknown error"]
raise HTTPException(502, f"Search failed: {detail[0][:300]}")
results = []
for line in proc.stdout.splitlines():
line = line.strip()
if not line.startswith("{"):
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
video_id = entry.get("id")
if not video_id:
continue
existing = _get_track(video_id)
results.append({
"video_id": video_id,
"title": entry.get("title") or video_id,
"channel": entry.get("uploader") or entry.get("channel") or "",
"duration": entry.get("duration"),
"view_count": entry.get("view_count"),
"in_library": bool(existing),
"track_id": existing["id"] if existing else None,
"library_status": existing["status"] if existing else None,
})
return results
def _job_dict(job: dict) -> dict:
return {k: job.get(k) for k in
("id", "track_id", "stage", "progress", "error", "started_at")}
def _persist_job(job: dict) -> None:
try:
(JOBS_DIR / f"{job['id']}.json").write_text(
json.dumps(_job_dict(job)), encoding="utf-8"
)
except OSError:
pass
def _fail_job(job: dict, track: dict, message: str) -> None:
job["stage"] = "error"
job["error"] = message[:500]
_persist_job(job)
with _lock:
track["status"] = "failed"
track["error"] = message[:500]
_bump_version()
_save_library()
def _download_worker(job: dict, track: dict, video_id: str) -> None:
"""Runs in the single-worker executor: yt-dlp download + mp3 transcode."""
target = FILES_DIR / f"{track['id']}.mp3"
cmd = _ytdlp_base_args() + [
"-f", "bestaudio/best",
"-x", "--audio-format", "mp3", "--audio-quality", "192K",
"--embed-thumbnail", "--add-metadata",
"--write-thumbnail", "--convert-thumbnails", "jpg",
"--newline", "--progress",
"--print-json",
"--retries", "3", "--fragment-retries", "3",
"-P", str(FILES_DIR),
"-o", f"{track['id']}.%(ext)s",
f"https://www.youtube.com/watch?v={video_id}",
]
job["stage"] = "downloading"
info: dict = {}
try:
proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, errors="replace",
)
deadline = time.monotonic() + DOWNLOAD_TIMEOUT
assert proc.stdout is not None and proc.stderr is not None
for line in proc.stdout:
if time.monotonic() > deadline:
proc.kill()
raise RuntimeError(f"Download exceeded {DOWNLOAD_TIMEOUT}s")
line = line.strip()
m = _PROGRESS_RE.search(line)
if m:
job["progress"] = min(85.0, float(m.group(1)) * 0.85)
_persist_job(job)
elif line.startswith(("[ExtractAudio]", "[Metadata]", "[EmbedThumbnail]")):
job["stage"] = "transcoding"
job["progress"] = max(job["progress"], 86.0)
_persist_job(job)
elif line.startswith("{"):
try:
info = json.loads(line)
except json.JSONDecodeError:
pass
proc.wait(timeout=30)
if proc.returncode != 0:
stderr_tail = (proc.stderr.read() or "").strip().splitlines()
detail = stderr_tail[-1] if stderr_tail else f"exit code {proc.returncode}"
raise RuntimeError(detail)
if not target.exists():
raise RuntimeError("yt-dlp finished but no mp3 was produced")
except Exception as exc: # noqa: BLE001 — any failure marks the track failed
_fail_job(job, track, str(exc))
return
thumb_src = FILES_DIR / f"{track['id']}.jpg"
try:
if thumb_src.exists():
thumb_src.replace(THUMBS_DIR / f"{track['id']}.jpg")
except OSError:
pass
job["stage"] = "ready"
job["progress"] = 100.0
_persist_job(job)
with _lock:
track["status"] = "ready"
track["error"] = None
if info:
if info.get("title"):
track["title"] = info["title"]
if info.get("uploader") or info.get("channel"):
track["channel"] = info.get("uploader") or info.get("channel")
if info.get("duration"):
track["duration"] = round(float(info["duration"]))
_bump_version()
_save_library()
def _start_download(track: dict, video_id: str) -> dict:
job = {
"id": uuid.uuid4().hex[:12],
"track_id": track["id"],
"stage": "queued",
"progress": 0.0,
"error": None,
"started_at": time.time(),
}
_jobs[job["id"]] = job
_persist_job(job)
_executor.submit(_download_worker, job, track, video_id)
return job
# ── API: health, library ────────────────────────────────────────────────────
@app.get("/media/api/healthz")
def healthz() -> dict:
return {"ok": True, "version": _library["version"]}
@app.get("/media/api/library")
def get_library() -> dict:
with _lock:
tracks = sorted(
(_track_dict(t) for t in _library["tracks"].values()),
key=lambda t: t.get("added_at", 0),
reverse=True,
)
return {"tracks": tracks, "version": _library["version"]}
def _delete_track_files(track: dict) -> None:
if track["kind"] != "youtube":
return
for path in (
FILES_DIR / f"{track['id']}.mp3",
FILES_DIR / f"{track['id']}.jpg",
THUMBS_DIR / f"{track['id']}.jpg",
):
try:
path.unlink(missing_ok=True)
except OSError:
pass
@app.delete("/media/api/tracks/{track_id}")
def delete_track(track_id: str) -> dict:
with _lock:
track = _get_track(track_id)
if not track:
raise HTTPException(404, "unknown track")
del _library["tracks"][track_id]
_bump_version()
_save_library()
for room, st in _rooms.items():
if st["track_id"] == track_id:
st["track_id"] = None
st["desired"] = "stopped"
_delete_track_files(track)
return {"ok": True}
# ── API: search + play + control ────────────────────────────────────────────
@app.post("/media/api/search")
async def search(request: Request) -> dict:
body = await request.json()
query = str(body.get("query", "")).strip()
if not query:
raise HTTPException(400, "query is required")
limit = max(1, min(int(body.get("limit", 6)), 15))
return {"query": query, "results": _run_search(query, limit)}
def _resolve_play_target(body: dict) -> dict:
"""Find or create the track a play request refers to.
Accepts video_id (YouTube), track_id (existing library entry) or a raw
url+name (live stream). Returns the library track dict.
"""
video_id = str(body.get("video_id") or "").strip()
track_id = str(body.get("track_id") or "").strip()
url = str(body.get("url") or "").strip()
if video_id:
existing = _get_track(video_id)
if existing:
return existing
track = {
"id": video_id,
"kind": "youtube",
"video_id": video_id,
"title": str(body.get("title") or video_id),
"channel": str(body.get("channel") or ""),
"duration": body.get("duration"),
"status": "downloading",
"error": None,
"added_at": time.time(),
}
_upsert_track(track)
_start_download(track, video_id)
return track
if track_id:
track = _get_track(track_id)
if not track:
raise HTTPException(404, "unknown track")
return track
if url:
if not re.match(r"^https?://", url):
raise HTTPException(400, "url must be http(s)")
track_id = "s" + uuid.uuid4().hex[:10]
track = {
"id": track_id,
"kind": "stream",
"video_id": None,
"title": str(body.get("name") or url),
"channel": str(body.get("channel") or ""),
"duration": None,
"status": "ready",
"error": None,
"url": url,
"added_at": time.time(),
}
_upsert_track(track)
return track
raise HTTPException(400, "one of video_id, track_id or url is required")
@app.post("/media/api/play")
async def play(request: Request) -> dict:
body = await request.json()
room = str(body.get("room") or "default")
track = _resolve_play_target(body)
with _lock:
st = _room_state(room)
st["track_id"] = track["id"]
# start_paused carries a track into a new scope without starting it —
# the UI uses this when a conversation begins (talking pauses music).
st["desired"] = "paused" if body.get("start_paused") else "playing"
st["choices"] = [] # a pick resolves any open suggestion list
st["seek_hint"] = None
snapshot = _room_snapshot(room)
return {"ok": True, **snapshot}
@app.post("/media/api/control")
async def control(request: Request) -> dict:
body = await request.json()
room = str(body.get("room") or "default")
action = str(body.get("action") or "")
with _lock:
st = _room_state(room)
if action == "pause":
st["desired"] = "paused"
elif action == "resume":
if not st["track_id"]:
raise HTTPException(409, "nothing loaded")
st["desired"] = "playing"
elif action == "stop":
st["desired"] = "stopped"
elif action == "seek":
st["seek_hint"] = {
"position": max(0.0, float(body.get("position", 0))),
"ts": time.time(),
}
elif action == "skip":
current = st["track_id"]
candidates = sorted(
(
t for t in _library["tracks"].values()
if t.get("status") == "ready" and t["id"] != current
),
key=lambda t: t.get("added_at", 0),
reverse=True,
)
if candidates:
st["track_id"] = candidates[0]["id"]
st["desired"] = "playing"
st["seek_hint"] = None
else:
st["desired"] = "stopped"
else:
raise HTTPException(400, f"unknown action: {action}")
snapshot = _room_snapshot(room)
return {"ok": True, **snapshot}
@app.post("/media/api/choices")
async def choices(request: Request) -> dict:
"""Publish Hope's song suggestions so the UI can render clickable picks."""
body = await request.json()
room = str(body.get("room") or "default")
raw = body.get("choices") or []
cleaned = []
for c in raw[:5]:
if not isinstance(c, dict) or not c.get("video_id"):
continue
cleaned.append({
"video_id": str(c["video_id"]),
"title": str(c.get("title") or c["video_id"]),
"channel": str(c.get("channel") or ""),
"duration": c.get("duration"),
})
with _lock:
_room_state(room)["choices"] = cleaned
return {"ok": True, "choices": cleaned}
@app.get("/media/api/state")
def state(room: str = "default") -> dict:
with _lock:
return _room_snapshot(room)
@app.post("/media/api/report")
async def report(request: Request) -> dict:
"""The browser reports real playback state (source of truth for the player)."""
body = await request.json()
room = str(body.get("room") or "default")
with _lock:
st = _room_state(room)
st["reported"] = {
"track_id": body.get("track_id"),
"state": str(body.get("state") or "stopped"),
"position": float(body.get("position") or 0.0),
"duration": body.get("duration"),
"ts": time.time(),
}
return {"ok": True}
@app.post("/media/api/streams")
async def add_stream(request: Request) -> dict:
body = await request.json()
track = _resolve_play_target({"url": body.get("url"), "name": body.get("name")})
return {"ok": True, "track": _track_dict(track)}
# ── API: stream proxy (external http(s) audio over same-origin HTTPS) ───────
_HOP_HEADERS = ("content-type", "content-range", "accept-ranges", "content-length")
@app.get("/media/api/stream/{track_id}")
async def stream_proxy(track_id: str, request: Request):
with _lock:
track = _get_track(track_id)
if not track:
raise HTTPException(404, "unknown track")
if track["kind"] == "youtube":
# Downloaded tracks are nginx static files (native Range support).
if track.get("status") == "ready":
return RedirectResponse(f"/media/files/{track_id}.mp3")
raise HTTPException(409, f"track is {track.get('status')}, not ready")
range_header = request.headers.get("range")
headers = {"Range": range_header} if range_header else {"Range": "bytes=0-"}
client = httpx.AsyncClient(timeout=httpx.Timeout(30, read=600), follow_redirects=True)
try:
upstream = await client.send(client.build_request("GET", track["url"], headers=headers), stream=True)
except httpx.HTTPError as exc:
await client.aclose()
raise HTTPException(502, f"upstream stream failed: {exc}")
if upstream.status_code >= 400:
await upstream.aclose()
await client.aclose()
raise HTTPException(502, f"upstream returned {upstream.status_code}")
out_headers = {
name: value
for name, value in upstream.headers.items()
if name.lower() in _HOP_HEADERS
}
media_type = upstream.headers.get("content-type", "audio/mpeg")
async def relay():
try:
async for chunk in upstream.aiter_bytes(64 * 1024):
yield chunk
finally:
await upstream.aclose()
await client.aclose()
return StreamingResponse(
relay(),
status_code=upstream.status_code,
media_type=media_type,
headers=out_headers,
)
@app.exception_handler(HTTPException)
async def http_error(_request: Request, exc: HTTPException):
return JSONResponse({"error": exc.detail}, status_code=exc.status_code)
if __name__ == "__main__":
uvicorn.run(app, host=MEDIA_HOST, port=MEDIA_PORT, log_level="info")
+259 -1
View File
@@ -1245,6 +1245,263 @@ body[data-mode="text"] .mic-meter {
.task-result { margin-top: 0.5rem; color: #bcebd8; background: rgba(118, 217, 178, 0.1); } .task-result { margin-top: 0.5rem; color: #bcebd8; background: rgba(118, 217, 178, 0.1); }
.task-error { margin-top: 0.5rem; color: #ffc2ba; background: rgba(255, 128, 111, 0.1); } .task-error { margin-top: 0.5rem; color: #ffc2ba; background: rgba(255, 128, 111, 0.1); }
/* ── Media player ───────────────────────────────────────────────────────── */
/* Reuses .tasks-panel positioning (fixed right, slides in via .open); the
content scrolls under a sticky header. */
.media-panel {
overflow-y: auto;
padding-bottom: 1.2rem;
}
.media-panel .tasks-header {
position: sticky;
top: 0;
z-index: 1;
background: #0d1929;
}
.media-now {
display: flex;
align-items: center;
gap: 0.75rem;
margin: 1rem 1.1rem 0;
padding: 0.75rem;
border-radius: 12px;
background: var(--surface-raised);
}
.media-art,
.media-art-placeholder {
width: 56px;
height: 56px;
flex: 0 0 56px;
border-radius: 8px;
object-fit: cover;
background: rgba(23, 41, 65, 0.7);
}
.media-art-placeholder {
display: flex;
align-items: center;
justify-content: center;
color: var(--ink-muted);
}
.media-now-info {
flex: 1;
min-width: 0;
display: flex;
flex-direction: column;
gap: 0.15rem;
}
.media-now-info strong {
color: var(--ink);
font-size: 0.88rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-now-info span {
color: var(--ink-muted);
font-size: 0.72rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-progress {
position: relative;
margin-top: 0.35rem;
display: flex;
align-items: center;
gap: 0.5rem;
}
/* Track bar behind the fill (fill width is driven inline by progress). */
.media-progress::before {
content: "";
position: absolute;
left: 0;
top: 50%;
transform: translateY(-50%);
width: 90px;
height: 4px;
border-radius: 2px;
background: rgba(139, 167, 255, 0.18);
}
.media-progress-fill {
height: 4px;
flex: 0 0 auto;
width: 0;
border-radius: 2px;
background: var(--iris);
transition: width 400ms ease;
}
.media-progress-label {
color: var(--ink-muted);
font-size: 0.66rem;
white-space: nowrap;
}
.media-controls {
display: flex;
align-items: center;
justify-content: center;
gap: 0.9rem;
margin: 0.9rem 1.1rem 0;
}
.media-controls .btn-icon {
color: var(--ink-soft);
}
.media-play {
width: 52px;
height: 52px;
border-radius: 50%;
display: inline-flex;
align-items: center;
justify-content: center;
color: #0b1523 !important;
background: var(--iris) !important;
border: none;
}
.media-play:hover:not(:disabled) {
filter: brightness(1.1);
}
.media-seek {
display: flex;
align-items: center;
gap: 0.55rem;
margin: 0.75rem 1.1rem 0;
}
.media-time {
color: var(--ink-muted);
font-size: 0.68rem;
font-variant-numeric: tabular-nums;
min-width: 2.6rem;
text-align: center;
}
.media-seek input[type="range"],
.media-volume input[type="range"] {
flex: 1;
accent-color: var(--iris);
cursor: pointer;
}
.media-volume {
display: flex;
align-items: center;
gap: 0.55rem;
margin: 0.55rem 1.1rem 0;
color: var(--ink-muted);
}
.media-search {
display: flex;
gap: 0.45rem;
margin: 0 1.1rem;
}
.media-search input {
flex: 1;
min-width: 0;
padding: 0.5rem 0.7rem;
border: 1px solid var(--line-strong);
border-radius: 9px;
color: var(--ink);
background: rgba(5, 11, 20, 0.45);
font-size: 0.8rem;
}
.media-search input::placeholder {
color: var(--ink-muted);
}
.media-stream-form input:first-child {
flex: 0 0 38%;
}
.media-list {
display: flex;
flex-direction: column;
gap: 0.4rem;
margin: 0.65rem 1.1rem 1rem;
max-height: 240px;
overflow-y: auto;
}
.media-row {
display: flex;
align-items: center;
gap: 0.6rem;
padding: 0.55rem 0.65rem;
border-radius: 9px;
background: var(--surface-raised);
}
.media-row-info {
flex: 1;
min-width: 0;
display: flex;
flex-direction: column;
gap: 0.1rem;
}
.media-row-info strong {
color: var(--ink);
font-size: 0.78rem;
font-weight: 600;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-row-info span {
color: var(--ink-muted);
font-size: 0.68rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.media-row-play {
flex: 0 0 auto;
padding: 0.3rem 0.7rem;
font-size: 0.7rem;
}
.media-row-delete {
flex: 0 0 auto;
color: var(--ink-muted);
}
.media-row-delete:hover {
color: #ffc2ba;
}
.media-empty {
padding: 0.8rem;
border-radius: 9px;
color: var(--ink-muted);
background: var(--surface-raised);
font-size: 0.74rem;
text-align: center;
}
.media-choice {
border: 1px solid rgba(139, 167, 255, 0.35);
background: rgba(60, 95, 201, 0.12);
}
/* ── Responsive ─────────────────────────────────────────────────────────── */ /* ── Responsive ─────────────────────────────────────────────────────────── */
@media (max-height: 760px) and (min-width: 881px) { @media (max-height: 760px) and (min-width: 881px) {
.workspace { .workspace {
@@ -1386,7 +1643,8 @@ body[data-mode="text"] .mic-meter {
padding: 1.25rem 1rem; padding: 1.25rem 1rem;
} }
.tasks-panel { .tasks-panel,
.media-panel {
width: 100vw; width: 100vw;
} }
+3
View File
@@ -97,6 +97,9 @@ class Handler(BaseHTTPRequestHandler):
resp = json.dumps( resp = json.dumps(
{ {
"token": token, "token": token,
# The room name scopes media-server player state; the UI sends
# it with every /media/api/ call.
"room": room_name,
"user": { "user": {
"id": user_id, "id": user_id,
"display_name": profile.get("display_name") or user_id, "display_name": profile.get("display_name") or user_id,