Compare commits

...
2 Commits
Author SHA1 Message Date
snowspeeder d7c385301e fix: CI deploy parity with compose — media env, volume, healthcheck
Rebuild & Deploy / rebuild (push) Successful in 59s
A missing env var aborts supervisord's %(ENV_x)s config expansion and
takes every program down, which is why the media-player merge failed
CI (run 151): rebuild.yml's docker run did not pass MEDIA_DIR /
MEDIA_SERVER_PORT / YTDLP_COOKIES.

- supervisord.conf: media-server inherits those vars from the container
  environment instead of hard-requiring them (media_server.py has safe
  defaults for all three).
- rebuild.yml: mount the media library volume, pass the media env, and
  extend the healthcheck to the media server healthz.
- compose: YTDLP_COOKIES defaults to the conventional /data/media/
  cookies.txt path (ignored unless the file exists).

Parity verified: container started with CI's exact env (no media vars)
— all five programs RUNNING.
2026-08-29 15:45:54 -04:00
snowspeeder 8a7cca456f fix: force tool calls for drifted intents and harden tool-use prompt
Small local models (qwen2.5-vl-3b, gemma-4-e4b) narrate actions ('Sure,
I'll pause the music') without emitting the tool call once the full
system prompt and 26-tool set are in context. Measured: qwen emitted
tools in 2/8 realistic intents, gemma 5/8; single-tool tests pass 100%,
so this is context-induced drift, not plumbing.

- agent/intent_router.py: deterministic forced-call router for
  high-frequency intents — media controls, now_playing/list_library,
  'remember that ...' memory saves, explicit YouTube Music searches,
  and personal-fact recall (memory_recall/search_notes/skill_recall,
  excluded for weather/time questions). Fires only when the tool is
  available; guards against how-do-I questions.
- agent.py: generalize _ForcedToolCallStream to any tool; route through
  intent_router after the dispatch check. System prompt adds a hard
  'call, don't narrate' rule and reconditions 'answer directly'.
- tests/test_intent_router.py: 30 unit tests pinning router behavior.

Verified against the live LLM with the full fidelity harness: combined
forced-or-emitted coverage 8/8 on qwen2.5-vl-3b (was 2/8) and 8/8 on
gemma-4-e4b (was 5/8). Full suite: 79 passed.
2026-08-29 15:38:35 -04:00
7 changed files with 474 additions and 20 deletions

No files matched your search

+5 -1
View File
@@ -26,7 +26,7 @@ jobs:
--name voice \
--network host \
--restart unless-stopped \
--health-cmd='curl -sk https://localhost:${WEB_PORT:-9178}/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null' \
--health-cmd='curl -sk https://localhost:${WEB_PORT:-9178}/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null && curl -s http://localhost:${MEDIA_SERVER_PORT:-8092}/media/api/healthz -o /dev/null' \
--health-interval=15s \
--health-timeout=5s \
--health-retries=3 \
@@ -34,6 +34,7 @@ jobs:
-v /mnt/user/dev/hope-voice-api/certs:/etc/voice/certs \
-v /mnt/user/dev/hope-voice-api/skills:/skills \
-v /mnt/user/dev/hope-voice-api/livekit.yaml:/etc/livekit.yaml:ro \
-v /mnt/user/dev/hope-voice-api/media:/data/media \
-e LIVEKIT_URL=http://localhost:7880 \
-e LIVEKIT_API_KEY=devkey \
-e LIVEKIT_API_SECRET=hopevoice-devsecret-2026-lan-only \
@@ -48,6 +49,9 @@ jobs:
-e WEB_PORT=9178 \
-e VOICE_USERS_FILE=/etc/hope/users.json \
-e VOICE_DEFAULT_USER=shane \
-e MEDIA_DIR=/data/media \
-e MEDIA_SERVER_PORT=8092 \
-e YTDLP_COOKIES=/data/media/cookies.txt \
hope-voice-api-voice:latest
- name: Verify deployment
+1
View File
@@ -47,6 +47,7 @@ COPY agent/weather_mcp.py /opt/voice-agent/weather_mcp.py
COPY agent/skills_mcp.py /opt/voice-agent/skills_mcp.py
COPY agent/task_registry.py /opt/voice-agent/task_registry.py
COPY agent/task_router.py /opt/voice-agent/task_router.py
COPY agent/intent_router.py /opt/voice-agent/intent_router.py
COPY agent/task_worker.py /opt/voice-agent/task_worker.py
COPY agent/dispatch_mcp.py /opt/voice-agent/dispatch_mcp.py
COPY agent/media_mcp.py /opt/voice-agent/media_mcp.py
+59 -14
View File
@@ -127,11 +127,19 @@ SYSTEM_PROMPT = textwrap.dedent("""\
no markdown, and no formatting except the Azure delivery markers below.
- Spell out numbers when it sounds more natural ("twenty twenty-six"
instead of "2026").
- Make reasonable assumptions and answer directly. Ask a follow-up only
when it is genuinely impossible to proceed without one.
- Make reasonable assumptions and answer directly once any relevant tool
call has been made. Ask a follow-up only when it is genuinely
impossible to proceed without one.
- Be warm and direct. Don't be sycophantic or overly formal.
- If you don't know something, say so briefly and move on.
# Tools: call them, don't narrate them
When a request matches one of your tools, you MUST emit that tool call
in this reply. Never only say "I'll pause the music", "let me check", or
"I'm saving that now" without the tool call — narrating an action is a
failure because nothing actually happens. After the tool result comes
back, confirm briefly in your normal voice.
# Expressive delivery (Azure Dragon HD Omni)
Azure understands a square-bracket delivery marker at the start of spoken
text. Use zero to two markers per response, only when they genuinely improve
@@ -223,12 +231,14 @@ SYSTEM_PROMPT = textwrap.dedent("""\
""")
# ── Background-task routing ─────────────────────────────────────────────────
# ── Intent routing ──────────────────────────────────────────────────────────
try:
from agent.task_router import should_force_background_task
from agent.intent_router import detect_forced_call
except ImportError:
from task_router import should_force_background_task
from intent_router import detect_forced_call
# ── OpenAI-compatible LLM with reasoning_content fallback ───────────────────
@@ -320,23 +330,39 @@ class _ReasoningFallbackWrapper:
class _ForcedToolCallStream(lk_llm.LLMStream):
"""Emit a dispatch_task call without asking the small LLM to route it."""
"""Emit a tool call without asking the small LLM to produce it.
def __init__(self, llm, *, chat_ctx, tools, conn_options, description: str):
self._description = description
Two callers: the task router (dispatch_task for background work) and the
intent router (deterministic intents like "pause the music" or "remember
that ..."), where small models otherwise narrate the action without ever
emitting the call.
"""
def __init__(
self,
llm,
*,
chat_ctx,
tools,
conn_options,
tool_name: str = "dispatch_task",
arguments: dict | None = None,
):
self._tool_name = tool_name
self._arguments = arguments if arguments is not None else {}
super().__init__(llm, chat_ctx=chat_ctx, tools=tools, conn_options=conn_options)
async def _run(self) -> None:
self._event_ch.send_nowait(
lk_llm.ChatChunk(
id=f"forced-dispatch-{uuid.uuid4().hex}",
id=f"forced-{uuid.uuid4().hex}",
delta=lk_llm.ChoiceDelta(
role="assistant",
tool_calls=[
lk_llm.FunctionToolCall(
name="dispatch_task",
arguments=json.dumps({"description": self._description}),
call_id=f"dispatch-{uuid.uuid4().hex}",
name=self._tool_name,
arguments=json.dumps(self._arguments),
call_id=f"forced-{uuid.uuid4().hex}",
)
],
),
@@ -415,9 +441,11 @@ class HopeLLM(openai.LLM):
)
chat_ctx = self._compact_context(chat_ctx, conn_options)
# Force dispatch for clear background-work requests. Without this,
# small local models often say "I'll get on that" but emit no tool
# call at all.
# Force calls for intents small models drift on. Without this they
# say "I'll pause the music" or "I'll get on that" but emit no tool
# call at all. Task router handles background work; the intent router
# handles deterministic in-conversation intents (media controls,
# memory saves, music searches).
items = list(chat_ctx.items)
latest = items[-1] if items else None
latest_role = getattr(latest, "role", None)
@@ -432,7 +460,24 @@ class HopeLLM(openai.LLM):
chat_ctx=chat_ctx,
tools=tools or [],
conn_options=conn_options,
description=latest_text,
tool_name="dispatch_task",
arguments={"description": latest_text},
)
forced = detect_forced_call(latest_text, tools or []) if latest_text else None
if forced:
tool_name, arguments = forced
logger.info(
"Routing request to forced %s(%s): %.120s",
tool_name, arguments, latest_text,
)
return _ForcedToolCallStream(
self,
chat_ctx=chat_ctx,
tools=tools or [],
conn_options=conn_options,
tool_name=tool_name,
arguments=arguments,
)
stream = super().chat(
+207
View File
@@ -0,0 +1,207 @@
"""Deterministic intent router — forces tool calls for high-frequency intents.
Small local models drift under long context: they narrate an action ("Sure,
I'll pause the music") without emitting the tool call, which is exactly the
failure mode _ForcedToolCallStream was built to bypass for dispatch_task.
This module extends the same idea to the other unambiguous, high-frequency
voice intents: music playback controls, now-playing/library questions,
"remember that ..." memory saves, and explicit YouTube Music searches.
Conservatism rules:
- Never fires when the matching tool is not available in the current
session's toolset (MCP toolsets differ per user config).
- Never fires on explanation/questions ("how do I pause the music?") for
action intents — those deserve a spoken answer, not a side effect.
- "play <something>" only forces a search for what looks like a song name;
generic objects ("play some music", "play a game") fall through to the
LLM.
Returns (tool_name, arguments) or None. The agent emits the call directly;
the tool result then flows back through the normal loop, so Hope still
speaks a natural confirmation afterwards.
"""
from __future__ import annotations
import re
# Question/explanation guard for *action* intents (not for question intents
# like now_playing, which are questions by design).
_QUESTION_GUARD = re.compile(
r"\b(how (?:do|can) i|how to|what happens if|what does|can you explain|"
r"tell me about|why (?:do|does|is)|is there a way to)\b",
re.I,
)
_MEDIA_CONTROLS: list[tuple[re.Pattern[str], str]] = [
(re.compile(r"\bpause\b|\bput .* on hold\b", re.I), "pause_music"),
(re.compile(
r"\bstop\b(?:\s+(?:the|this|that))?\s*(?:music|song|playback|player|tune)s?\b",
re.I,
), "stop_music"),
(re.compile(r"\b(?:resume|unpause)\b", re.I), "resume_music"),
(re.compile(
r"\b(?:skip|next)\b.*\b(?:song|track|one)\b|\bskip\s+(?:this|it|that)\b",
re.I,
), "skip_song"),
(re.compile(r"\bplay\b.*\bagain\b|\bfrom the (?:top|beginning)\b", re.I), "resume_music"),
]
_NOW_PLAYING = re.compile(
r"\bwhat(?:'s| is|s)\s+(?:currently\s+)?playing\b"
r"|\bwhat song is (?:this|playing)\b"
r"|\bwhich song is (?:this|playing)\b"
r"|\bname this song\b"
r"|\bwhat(?:'s| is) this (?:track|tune)\b",
re.I,
)
_LIBRARY = re.compile(
r"\bwhat(?:'s| is)?\s*(?:in|of)\s+my\s+(?:music\s+)?library\b"
r"|\b(?:list|show)(?:\s+me)?\s+my\s+(?:music|library|songs|tracks)\b",
re.I,
)
_MEMORY_SAVE = re.compile(
r"^\s*(?:please\s+)?remember\s+(?:that\s+|this\s*[::]\s*)?(?P<body>.+?)\s*[.!]?\s*$",
re.I,
)
# Personal-fact recall. Excluded when the utterance is really a
# weather/time/date question — those tools already work lexically on small
# models and must win.
_MEMORY_RECALL = re.compile(
r"\bwhat(?:'s| is|s)\s+my\b"
r"|\bwhat(?:'s| is)\s+(?:the\s+)?(?:name|birthday|anniversary)\b.*\bmy\b"
r"|\bdo you remember\b"
r"|\bwhat did i (?:say|tell you|mention)\b"
r"|\bwho am i\b"
r"|\bwhere (?:do|did) i (?:live|work|leave)\b",
re.I,
)
_RECALL_EXCLUDE = re.compile(
r"\b(?:weather|temperature|forecast|time|date|day is it|clock)\b", re.I
)
_SEARCH_NOTES = re.compile(
r"\bdo i have (?:any |some )?notes\b"
r"|\b(?:check|search|look (?:in|through|at))\s+my\s+notes\b"
r"|\bsearch\s+notes\b",
re.I,
)
_SKILL_RECALL = re.compile(
r"\bdo i have (?:a|any) (?:skill|procedure)\b"
r"|\b(?:check|search)\s+(?:your|my)\s+skills\b"
r"|\bany skills (?:for|about)\b",
re.I,
)
_SEARCH_EXPLICIT = re.compile(
r"\b(?:find|search)\s+(?:for\s+)?(?P<q>.+?)\s+on\s+(?:youtube|yt)\s*music\b"
r"|\bsearch\s+(?:youtube\s+music\s+)?for\s+(?P<q2>.+?)\s*(?:on\s+youtube\s*music)?\s*$",
re.I,
)
_PLAY_IMPERATIVE = re.compile(r"^\s*(?:could you\s+|please\s+)?play\s+(?P<q>.+?)\s*[.!]?\s*$", re.I)
# "play <q>" where q is not a song request — fall through to the LLM.
_GENERIC_PLAY = re.compile(
r"^(?:some |any )?(?:music|something|anything|tunes?|songs?|a game|games?|the radio|it|that|one)$",
re.I,
)
def _tool_names(tool) -> list[str]:
names = []
for attr in ("name", "id"):
value = getattr(tool, attr, None)
if isinstance(value, str) and value:
names.append(value)
return names
def tool_available(tools, name: str) -> bool:
"""True if a tool with this name is in the session's toolset.
Handles MCP name mangling (server prefix) by comparing the last
__-separated segment.
"""
for tool in tools or []:
for candidate in _tool_names(tool):
if candidate == name or candidate.split("__")[-1] == name:
return True
return False
def _title_from(body: str) -> str:
words = body.split()
title = " ".join(words[:6]).strip(" .,!?;:")
return (title[:60] or body[:60]).strip()
def detect_forced_call(text: str, tools) -> tuple[str, dict] | None:
"""Match a user utterance to a forced tool call.
Returns (tool_name, arguments) when an unambiguous high-frequency intent
matches an available tool, else None. Media controls win over memory and
search so "pause" is never eaten by a wider pattern.
"""
if not text or not text.strip():
return None
utterance = text.strip()
lowered = utterance.lower()
# ── question intents (no action guard — they ARE questions) ────────────
if tool_available(tools, "now_playing") and _NOW_PLAYING.search(utterance):
return ("now_playing", {})
if tool_available(tools, "list_library") and _LIBRARY.search(utterance):
return ("list_library", {})
guarded = bool(_QUESTION_GUARD.search(utterance))
# ── media playback controls ─────────────────────────────────────────────
if not guarded:
for pattern, tool_name in _MEDIA_CONTROLS:
if pattern.search(utterance) and tool_available(tools, tool_name):
return (tool_name, {})
# ── memory save ─────────────────────────────────────────────────────────
if not guarded and tool_available(tools, "memory_save"):
m = _MEMORY_SAVE.match(utterance)
if m:
body = m.group("body").strip()
if len(body) >= 3:
return ("memory_save", {
"title": _title_from(body),
"body": body,
"tags": ["voice"],
})
# ── YouTube Music search ────────────────────────────────────────────────
if not guarded and tool_available(tools, "search_music"):
m = _SEARCH_EXPLICIT.search(utterance)
query = (m.group("q") or m.group("q2") or "").strip() if m else None
if not query:
m = _PLAY_IMPERATIVE.match(utterance)
if m:
candidate = m.group("q").strip()
if not _GENERIC_PLAY.match(candidate):
query = candidate
if query:
return ("search_music", {"query": query})
# ── recall (memory / notes / skills) ────────────────────────────────────
# The whole utterance is passed as the query — the memory and skills
# backends do their own semantic/keyword matching.
if not guarded:
if (tool_available(tools, "memory_recall")
and _MEMORY_RECALL.search(utterance)
and not _RECALL_EXCLUDE.search(utterance)):
return ("memory_recall", {"query": utterance})
if tool_available(tools, "search_notes") and _SEARCH_NOTES.search(utterance):
return ("search_notes", {"query": utterance})
if tool_available(tools, "skill_recall") and _SKILL_RECALL.search(utterance):
return ("skill_recall", {"query": utterance})
return None
+1 -1
View File
@@ -20,7 +20,7 @@ services:
VOICE_DEFAULT_USER: "${VOICE_DEFAULT_USER:-shane}"
MEDIA_DIR: "/data/media"
MEDIA_SERVER_PORT: "${MEDIA_SERVER_PORT:-8092}"
YTDLP_COOKIES: "${YTDLP_COOKIES:-}"
YTDLP_COOKIES: "${YTDLP_COOKIES:-/data/media/cookies.txt}"
volumes:
- ./livekit.yaml:/etc/livekit.yaml:ro
- ./certs:/etc/voice/certs
+4 -4
View File
@@ -60,6 +60,10 @@ environment=
[program:media-server]
; Runs as root because the /data/media bind mount is a host (Unraid) share
; whose uid does not match the container's voiceuser.
; MEDIA_DIR / MEDIA_SERVER_PORT / YTDLP_COOKIES are inherited from the
; container environment (media_server.py has safe defaults for all three) —
; do NOT list them via %(ENV_x)s here: a missing var aborts supervisord's
; config expansion and takes every program down with it.
command=/opt/voice-agent/.venv/bin/python /opt/voice/media_server.py
user=root
autostart=true
@@ -67,7 +71,3 @@ autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true
environment=
MEDIA_DIR="%(ENV_MEDIA_DIR)s",
MEDIA_SERVER_PORT="%(ENV_MEDIA_SERVER_PORT)s",
YTDLP_COOKIES="%(ENV_YTDLP_COOKIES)s"
+197
View File
@@ -0,0 +1,197 @@
"""Unit tests for the deterministic intent router (host-side, pure functions).
The router forces tool calls for intents small local models drift on —
these tests pin the matching behavior that the agent relies on.
"""
import importlib.util
import sys
from pathlib import Path
import pytest
_SPEC = importlib.util.spec_from_file_location(
"intent_router", Path(__file__).parents[1] / "agent" / "intent_router.py"
)
intent_router = importlib.util.module_from_spec(_SPEC)
sys.modules["intent_router"] = intent_router
_SPEC.loader.exec_module(intent_router)
detect = intent_router.detect_forced_call
available = intent_router.tool_available
FULL_TOOLSET = [
"get_weather", "get_time",
"memory_save", "memory_recall", "search_notes", "read_note",
"skill_recall", "skill_save",
"dispatch_task",
"search_music", "play_song", "offer_song_choices", "pause_music",
"resume_music", "stop_music", "skip_song", "now_playing", "list_library",
]
class FakeTool:
def __init__(self, name, id=None):
self.name = name
self.id = id
def tools(names):
return [FakeTool(n) for n in names]
# ── tool availability ───────────────────────────────────────────────────────
def test_tool_available_exact_and_prefixed():
assert available(tools(["pause_music"]), "pause_music")
assert available(tools(["media__pause_music"]), "pause_music")
assert not available(tools(["play_song"]), "pause_music")
assert not available([], "pause_music")
# id-only attribute (dispatch check path used by agent.py)
assert available([FakeTool(None, id="dispatch_task")], "dispatch_task")
# ── media controls ──────────────────────────────────────────────────────────
@pytest.mark.parametrize("utterance,expected", [
("pause the music", "pause_music"),
("pause", "pause_music"),
("could you pause it", "pause_music"),
("stop the music", "stop_music"),
("stop this song", "stop_music"),
("resume the music", "resume_music"),
("unpause", "resume_music"),
("play it again", "resume_music"),
("skip this song", "skip_song"),
("next track", "skip_song"),
])
def test_media_controls_forced(utterance, expected):
assert detect(utterance, tools(FULL_TOOLSET)) == (expected, {})
def test_stop_without_music_context_not_forced():
# "stop" alone is ambiguous (could mean ending the conversation) — the
# router leaves it to the LLM.
assert detect("stop", tools(FULL_TOOLSET)) is None
def test_media_controls_guard_against_questions():
assert detect("how do I pause the music?", tools(FULL_TOOLSET)) is None
assert detect("what happens if I stop the music", tools(FULL_TOOLSET)) is None
def test_media_control_missing_tool_not_forced():
no_media = [n for n in FULL_TOOLSET if not n.endswith("_music") or n == "play_song"]
assert detect("pause the music", tools(no_media)) is None
# ── memory saves ────────────────────────────────────────────────────────────
def test_memory_save_forced_with_extracted_body():
result = detect("Remember that my coffee order is an oat milk latte",
tools(FULL_TOOLSET))
assert result is not None
tool_name, args = result
assert tool_name == "memory_save"
assert args["body"] == "my coffee order is an oat milk latte"
assert args["title"]
assert isinstance(args["tags"], list)
def test_memory_save_variants():
for utterance in (
"Remember my wifi password is falcon9",
"please remember that the garage code is 1234",
):
result = detect(utterance, tools(FULL_TOOLSET))
assert result and result[0] == "memory_save", utterance
def test_memory_save_guard_and_short_body():
assert detect("how do I make you remember things?", tools(FULL_TOOLSET)) is None
assert detect("remember", tools(FULL_TOOLSET)) is None
# ── music search ────────────────────────────────────────────────────────────
def test_search_forced_for_explicit_and_play_imperative():
cases = [
("find Bohemian Rhapsody on YouTube Music", "Bohemian Rhapsody"),
("search for timeless miracle", "timeless miracle"),
("play Timeless Miracle by Blind Guardian", "Timeless Miracle by Blind Guardian"),
]
for utterance, query in cases:
result = detect(utterance, tools(FULL_TOOLSET))
assert result and result[0] == "search_music", utterance
assert result[1]["query"] == query
@pytest.mark.parametrize("utterance", [
"play some music",
"play something",
"play a game",
"how do I find a song on YouTube Music?",
])
def test_search_not_forced_for_generic_or_questions(utterance):
assert detect(utterance, tools(FULL_TOOLSET)) is None
# ── question intents ────────────────────────────────────────────────────────
@pytest.mark.parametrize("utterance,expected", [
("what's playing right now?", "now_playing"),
("what song is this", "now_playing"),
("what's in my music library?", "list_library"),
("list my songs", "list_library"),
])
def test_question_intents_forced_even_though_questions(utterance, expected):
assert detect(utterance, tools(FULL_TOOLSET)) == (expected, {})
# ── recall (memory / notes / skills) ────────────────────────────────────────
def test_memory_recall_forced_for_personal_facts():
cases = [
"what's my coffee order?",
"What is my wifi password",
"do you remember my garage code?",
"who am I?",
]
for utterance in cases:
result = detect(utterance, tools(FULL_TOOLSET))
assert result and result[0] == "memory_recall", utterance
assert result[1]["query"] == utterance
def test_memory_recall_not_stolen_from_weather_time():
# These must reach the weather/time tools, which work lexically.
for utterance in (
"what's the weather in Raleigh?",
"what time is it?",
"what's my weather like today?",
"what's the date today?",
):
assert detect(utterance, tools(FULL_TOOLSET)) != "memory_recall"
def test_search_notes_and_skill_recall_forced():
assert detect("do I have any notes about project deadlines?",
tools(FULL_TOOLSET))[0] == "search_notes"
assert detect("check my notes for the garage code",
tools(FULL_TOOLSET))[0] == "search_notes"
assert detect("do I have a skill for brewing coffee?",
tools(FULL_TOOLSET))[0] == "skill_recall"
assert detect("check your skills for baking bread",
tools(FULL_TOOLSET))[0] == "skill_recall"
# ── non-intents fall through to the LLM ─────────────────────────────────────
def test_plain_conversation_not_forced():
for utterance in (
"hello",
"what's the weather in Raleigh?",
"tell me about the Roman empire",
"",
):
assert detect(utterance, tools(FULL_TOOLSET)) is None