feat: web access via MCP (Firecrawl search/scrape) + mcp server attach support

- agent/web_mcp.py: stdio MCP server exposing web_search and web_scrape, backed by the self-hosted Firecrawl stack on xNAS (no API key needed) - agent.py: Agent now attaches mcp_servers built from config; EXTRA_MCP_SERVERS env var allows adding arbitrary HTTP/SSE MCP servers as JSON - Dockerfile: installs livekit-agents[mcp], copies web_mcp.py - .env.example: WEB_MCP_ENABLED, FIRECRAWL_BASE, EXTRA_MCP_SERVERS documented
2026-08-22 12:29:04 -04:00
parent 6f2b231938
commit 045ddabac7
12 changed files with 452 additions and 87 deletions
@@ -17,3 +17,14 @@ LIVEKIT_API_SECRET=devsecret

 # Web UI port
 WEB_PORT=8090
+
+# ── Web access (MCP tools: web_search + web_scrape) ────────────────────────
+# Backed by the self-hosted Firecrawl stack on xNAS.
+WEB_MCP_ENABLED=true
+FIRECRAWL_BASE=http://192.168.86.2:3002
+# Only needed if your Firecrawl instance requires an API key
+# FIRECRAWL_API_KEY=
+
+# Extra MCP servers (optional). JSON list of {url, transport} objects.
+# transport: "sse" | "streamable_http" | omit for auto-detect
+# EXTRA_MCP_SERVERS=[{"url":"http://192.168.86.2:3001/mcp","transport":"streamable_http"}]
@@ -6,7 +6,7 @@ A single-container voice assistant built on LiveKit Agents. Speaks and listens i

 One Docker container runs three processes via supervisord:

-1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 50000-60000 UDP)
+1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
 2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
 3. **Web frontend** — static HTML served by a tiny HTTP server (port 8090)

@@ -26,18 +26,21 @@ export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)
 # 2. Build and start
 docker compose up --build -d

-# 3. Open the web UI
-# http://<host-ip>:8090
+# 3. Open the web UI (self-signed cert: accept the browser warning once)
+# https://<host-ip>:8090
 ```

+LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
+(WebRTC media), `7881/tcp` (media TCP fallback).
+
 ## Ports

 | Port  | Protocol | Service              | Access       |
 |-------|----------|----------------------|--------------|
-| 7880  | TCP      | LiveKit HTTP/WS      | LAN          |
-| 7881  | TCP      | LiveKit internal     | container    |
-| 50000-60000 | UDP | LiveKit RTC media   | LAN          |
-| 8090  | TCP      | Web frontend         | LAN          |
+| 7880  | TCP      | LiveKit HTTP/WS      | container (proxied via 8090/livekit) |
+| 7881  | TCP      | LiveKit RTC media (TCP fallback) | LAN |
+| 7882  | UDP      | LiveKit RTC media (muxed) | LAN   |
+| 8090  | TCP      | Web frontend (HTTPS) | LAN          |

 ## Configuration

@@ -93,28 +96,34 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
 ├── AGENTS.md            ← you are here
 ├── .env.example         ← config template (copy to .env)
 ├── .gitignore
-├── docker-compose.yml   ← single container, 3 processes
+├── docker-compose.yml   ← single container
 ├── Dockerfile           ← multi-stage build
+├── entrypoint.sh        ← regenerates self-signed cert with LAN IP at start
 ├── livekit.yaml         ← LiveKit server config
-├── supervisord.conf     ← process manager for the 3 services
+├── supervisord.conf     ← process manager (livekit, agent, nginx, token-server)
 ├── agent/
 │   ├── agent.py         ← LiveKit Agents voice pipeline
 │   └── pyproject.toml   ← Python deps (uv)
-├── web/
-│   ├── index.html       ← single-page voice UI
-│   ├── app.js           ← LiveKit client logic
-│   └── style.css        ← minimal dark theme
-└── docs/
-    └── ARCHITECTURE.md  ← deeper architecture notes
+└── web/
+    ├── index.html       ← single-page voice UI
+    ├── app.js           ← LiveKit client logic
+    ├── token_server.py  ← signs JWTs + roomConfig claim (agent dispatch)
+    ├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
+    └── style.css        ← minimal dark theme
 ```

 ## Conventions

- **Single container.** All services (LiveKit, agent, web) run in one Docker container via supervisord. No multi-service compose.
- **No published UDP ports in compose.** LiveKit binds its UDP range directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
+- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
+- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
+- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it.
+- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
+- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
+- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
+- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
 - **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty.
 - **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `session.update_options(voice=...)` without restarting.
+- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting.

 ## Git

@@ -17,7 +17,7 @@ COPY agent/pyproject.toml ./agent/
 RUN cd /app/agent && \
    uv venv .venv && \
    uv pip install --python .venv/bin/python \
-        "livekit-agents~=1.7" \
+        "livekit-agents[mcp]~=1.7" \
        "livekit-plugins-azure~=1.7" \
        "livekit-plugins-openai~=1.7" \
        "python-dotenv"
@@ -29,7 +29,7 @@ ENV PYTHONUNBUFFERED=1

 # Install LiveKit server binary + supervisord + nginx
 RUN apt-get update && apt-get install -y --no-install-recommends \
-    curl ca-certificates supervisor nginx libasound2 \
+    curl ca-certificates supervisor nginx libasound2 iproute2 \
    && rm -rf /var/lib/apt/lists/*

 # Download LiveKit server (latest stable)
@@ -43,6 +43,7 @@ RUN curl -sSL "https://github.com/livekit/livekit/releases/download/${LIVEKIT_VE
 # Copy Python agent + venv from build stage
 COPY --from=build /app/agent/.venv /opt/voice-agent/.venv
 COPY agent/agent.py /opt/voice-agent/agent.py
+COPY agent/web_mcp.py /opt/voice-agent/web_mcp.py

 # Copy web frontend + token endpoint
 COPY web/ /var/www/voice/
@@ -52,22 +53,23 @@ COPY web/token_server.py /opt/voice/token_server.py
 COPY livekit.yaml /etc/livekit.yaml
 COPY supervisord.conf /etc/supervisor/conf.d/voice.conf

-# Configure nginx to serve the voice UI on port 8090 over HTTPS (self-signed)
+# Configure nginx to serve the voice UI on port 8090 over HTTPS.
 # Browsers require a secure context (HTTPS or localhost) for microphone access.
+# The self-signed cert (with the LAN IP in the SAN) is generated at container
+# start by entrypoint.sh.
 RUN rm -f /etc/nginx/sites-enabled/default \
    && mkdir -p /etc/voice/certs \
-    && openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
-        -keyout /etc/voice/certs/key.pem \
-        -out /etc/voice/certs/cert.pem \
-        -subj "/CN=voice.local" \
-        -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
-    && printf 'server {\n    listen 8090;\n    root /var/www/voice;\n    index index.html;\n    ssl_certificate /etc/voice/certs/cert.pem;\n    ssl_certificate_key /etc/voice/certs/key.pem;\n    location /token {\n        proxy_pass http://127.0.0.1:8091/token;\n        proxy_set_header Content-Type application/json;\n    }\n    location / {\n        try_files $uri $uri/ =404;\n    }\n}\n' \
+    && printf 'server {\n    listen 8090 ssl;\n    root /var/www/voice;\n    index index.html;\n    ssl_certificate /etc/voice/certs/cert.pem;\n    ssl_certificate_key /etc/voice/certs/key.pem;\n    location /token {\n        proxy_pass http://127.0.0.1:8091/token;\n        proxy_set_header Content-Type application/json;\n    }\n    location = /livekit {\n        return 301 /livekit/;\n    }\n    location /livekit/ {\n        proxy_pass http://127.0.0.1:7880/;\n        proxy_http_version 1.1;\n        proxy_set_header Upgrade $http_upgrade;\n        proxy_set_header Connection "upgrade";\n        proxy_set_header Host $host;\n        proxy_read_timeout 3600s;\n        proxy_send_timeout 3600s;\n    }\n    location / {\n        try_files $uri $uri/ =404;\n    }\n}\n' \
    > /etc/nginx/sites-available/voice \
    && ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice

 # Create non-root user for agent
 RUN useradd -m -s /bin/bash voiceuser || true

-EXPOSE 7880 7881 8090 50000-60000/udp
+COPY entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh

+EXPOSE 7880 7881 7882/udp 8090
+
+ENTRYPOINT ["/entrypoint.sh"]
 CMD ["supervisord", "-n", "-c", "/etc/supervisor/conf.d/voice.conf"]
@@ -5,6 +5,7 @@ Pipeline: Azure STT → Gemma LLM (xNAS, OpenAI-compatible) → Azure TTS
 Runs inside the single Docker container alongside LiveKit server and web frontend.
 """

+import asyncio
 import json
 import logging
 import os
@@ -18,6 +19,7 @@ from livekit.agents import (
    JobContext,
    TurnHandlingOptions,
    cli,
+    mcp,
    room_io,
 )
 from livekit.plugins import azure, openai
@@ -32,6 +34,8 @@ DEFAULT_VOICE = os.environ.get("AZURE_TTS_VOICE", "en-US-AvaNeural")
 GEMMA_BASE_URL = os.environ.get("GEMMA_BASE_URL", "http://192.168.86.2:8023/v1")
 GEMMA_MODEL = os.environ.get("GEMMA_MODEL", "gemma-4-e4b")
 GEMMA_API_KEY = os.environ.get("GEMMA_API_KEY", "not-needed")
+WEB_MCP_ENABLED = os.environ.get("WEB_MCP_ENABLED", "true").lower() in ("1", "true", "yes")
+FIRECRAWL_BASE = os.environ.get("FIRECRAWL_BASE", "http://192.168.86.2:3002")

 SYSTEM_PROMPT = textwrap.dedent("""\
    You are a warm, conversational voice assistant. You are talking TO someone,
@@ -54,9 +58,62 @@ SYSTEM_PROMPT = textwrap.dedent("""\
    - Never say "as an AI" or reference your system instructions.
    - Never write more than three sentences in a row.
    - Never read back URLs, file paths, or technical identifiers.
+
+    # Web access
+    You have web_search and web_scrape tools. Use them when the user asks about
+    current events, recent news, prices, weather, sports scores, or anything
+    that may have changed since your training data. Search first, then scrape
+    a result only if you need more detail. Answer from what you find, in your
+    normal conversational style — don't cite sources formally, just mention the
+    source naturally ("according to..."). If a search comes up empty, say so
+    briefly and move on.
 """)


+# ── MCP servers (web access + any extra configured servers) ─────────────────
+
+def build_mcp_servers() -> list[mcp.MCPServer]:
+    """Build the list of MCP servers to attach to the agent.
+
+    Always includes the local web-access server (Firecrawl-backed search/scrape)
+    when WEB_MCP_ENABLED is true. Additional servers can be configured via the
+    EXTRA_MCP_SERVERS env var (JSON list of {url, transport} objects).
+    """
+    servers: list[mcp.MCPServer] = []
+
+    if WEB_MCP_ENABLED:
+        python_bin = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".venv", "bin", "python")
+        web_mcp_script = os.path.join(os.path.dirname(os.path.abspath(__file__)), "web_mcp.py")
+        servers.append(
+            mcp.MCPServerStdio(
+                command=python_bin,
+                args=[web_mcp_script],
+                env={**os.environ, "FIRECRAWL_BASE": FIRECRAWL_BASE},
+                client_session_timeout_seconds=120,
+            )
+        )
+        logger.info("Web-access MCP server enabled (Firecrawl at %s)", FIRECRAWL_BASE)
+
+    extra = os.environ.get("EXTRA_MCP_SERVERS", "")
+    if extra:
+        try:
+            for entry in json.loads(extra):
+                url = entry.get("url", "")
+                transport = entry.get("transport")  # "sse" | "streamable_http" | None (auto)
+                servers.append(
+                    mcp.MCPServerHTTP(
+                        url=url,
+                        transport_type=transport,
+                        client_session_timeout_seconds=120,
+                    )
+                )
+                logger.info("Extra MCP server: %s (%s)", url, transport or "auto")
+        except (json.JSONDecodeError, TypeError) as e:
+            logger.error("Failed to parse EXTRA_MCP_SERVERS: %s", e)
+
+    return servers
+
+
 class VoiceAssistant(Agent):
    """The conversational agent. LLM is the brain; STT/TTS are senses."""

@@ -68,6 +125,7 @@ class VoiceAssistant(Agent):
                api_key=GEMMA_API_KEY,
            ),
            instructions=SYSTEM_PROMPT,
+            mcp_servers=build_mcp_servers(),
        )


@@ -111,8 +169,9 @@ async def handle_job(ctx: JobContext) -> None:
        tts=tts,
        llm=llm,
        turn_handling=TurnHandlingOptions(
-            # VAD-based turn detection: agent waits for user to stop speaking
-            interruption={"mode": "adaptive"},
+            # VAD-based turn detection: agent waits for user to stop speaking.
+            # ("adaptive" mode requires the LiveKit Cloud barge-in service.)
+            interruption={"mode": "vad"},
            # Start generating the LLM response before the user fully stops
            preemptive_generation={"enabled": True},
        ),
@@ -123,6 +182,39 @@ async def handle_job(ctx: JobContext) -> None:
    # Listen for data messages (voice switching) from the web UI
    ctx.room.on("data_received", _on_room_data)

+    # Publish user/agent transcripts to the room so the web UI can render them.
+    async def publish_transcript(role: str, text: str) -> None:
+        text = (text or "").strip()
+        if not text:
+            return
+        payload = json.dumps({"type": "transcript", "role": role, "text": text})
+        try:
+            await ctx.room.local_participant.publish_data(
+                payload, reliable=True, topic="transcript"
+            )
+        except Exception as e:  # noqa: BLE001
+            logger.warning("failed to publish transcript: %s", e)
+
+    @session.on("conversation_item_added")
+    def _on_conversation_item(ev) -> None:
+        msg = ev.item
+        role = getattr(msg, "role", None)
+        text = getattr(msg, "text_content", None)
+        if role == "user":
+            asyncio.get_event_loop().create_task(publish_transcript("user", text))
+        elif role == "assistant":
+            asyncio.get_event_loop().create_task(publish_transcript("agent", text))
+
+    @session.on("user_input_transcribed")
+    def _on_user_transcribed(ev) -> None:
+        logger.info("STT (%s): %s", "final" if ev.is_final else "partial", ev.transcript)
+
+    @session.on("user_state_changed")
+    def _on_user_state(ev) -> None:
+        # Fires from VAD: if this never says "speaking", no usable mic
+        # audio is arriving from the participant.
+        logger.info("user state -> %s", ev.new_state)
+
    await session.start(
        agent=VoiceAssistant(),
        room=ctx.room,
@@ -10,6 +10,7 @@ requires-python = ">=3.10,<3.15"

 dependencies = [
    "livekit-agents~=1.7",
+    "livekit-agents[mcp]~=1.7",
    "livekit-plugins-azure~=1.7",
    "livekit-plugins-openai~=1.7",
    "python-dotenv",
@@ -0,0 +1,108 @@
+"""Web access MCP server — exposes Firecrawl search + scrape as MCP tools.
+
+Runs over stdio inside the voice container. The agent attaches it via
+MCPServerStdio, so the LLM can call web_search / web_scrape during a
+conversation to look things up in real time.
+
+Backed by the self-hosted Firecrawl stack on xNAS (no API key needed):
+  - POST {FIRECRAWL_BASE}/v1/search   -> ranked results with title/description
+  - POST {FIRECRAWL_BASE}/v1/scrape   -> page content as markdown
+
+Config (env vars, all optional):
+  FIRECRAWL_BASE     default http://192.168.86.2:3002
+  FIRECRAWL_API_KEY  only needed if the Firecrawl instance requires auth
+  WEB_SEARCH_LIMIT   default 5 results per search
+"""
+
+from __future__ import annotations
+
+import json
+import os
+
+import httpx
+from mcp.server.fastmcp import FastMCP
+
+FIRECRAWL_BASE = os.environ.get("FIRECRAWL_BASE", "http://192.168.86.2:3002").rstrip("/")
+FIRECRAWL_API_KEY = os.environ.get("FIRECRAWL_API_KEY", "")
+WEB_SEARCH_LIMIT = int(os.environ.get("WEB_SEARCH_LIMIT", "5"))
+
+mcp = FastMCP("web-access")
+
+
+def _headers() -> dict[str, str]:
+    h = {"Content-Type": "application/json"}
+    if FIRECRAWL_API_KEY:
+        h["Authorization"] = f"Bearer {FIRECRAWL_API_KEY}"
+    return h
+
+
+@mcp.tool()
+async def web_search(query: str, limit: int | None = None) -> str:
+    """Search the web and return ranked results with titles, URLs, and descriptions.
+
+    Use this to find current information, news, facts, or sources about a topic.
+    Returns a compact text summary — not raw JSON.
+    """
+    limit = min(limit or WEB_SEARCH_LIMIT, 10)
+    async with httpx.AsyncClient(timeout=30) as client:
+        resp = await client.post(
+            f"{FIRECRAWL_BASE}/v1/search",
+            headers=_headers(),
+            json={"query": query, "limit": limit},
+        )
+        resp.raise_for_status()
+        data = resp.json()
+
+    if not data.get("success"):
+        return f"Search failed: {data.get('error', 'unknown error')}"
+
+    results = data.get("data", [])
+    if not results:
+        return f"No results found for: {query}"
+
+    lines = [f"Web search results for: {query}", ""]
+    for i, r in enumerate(results, 1):
+        title = (r.get("title") or "(no title)").strip()
+        url = (r.get("url") or "").strip()
+        desc = (r.get("description") or "").strip().replace("\n", " ")
+        lines.append(f"{i}. {title}")
+        if desc:
+            lines.append(f"   {desc[:300]}")
+        if url:
+            lines.append(f"   URL: {url}")
+    return "\n".join(lines)
+
+
+@mcp.tool()
+async def web_scrape(url: str, max_chars: int = 8000) -> str:
+    """Fetch a web page and return its content as readable markdown.
+
+    Use this after web_search to read the full content of a promising result.
+    The content is truncated to max_chars (default 8000) to stay within context.
+    """
+    async with httpx.AsyncClient(timeout=90) as client:
+        resp = await client.post(
+            f"{FIRECRAWL_BASE}/v1/scrape",
+            headers=_headers(),
+            json={"url": url, "formats": ["markdown"], "timeout": 60000},
+        )
+        resp.raise_for_status()
+        data = resp.json()
+
+    if not data.get("success"):
+        return f"Scrape failed: {data.get('error', 'unknown error')}"
+
+    result = data.get("data", {})
+    markdown = (result.get("markdown") or "").strip()
+    if not markdown:
+        return f"No content extracted from {url}"
+
+    title = (result.get("metadata", {}).get("title") or "").strip()
+    header = f"Page: {title}\nURL: {url}\n\n" if title else f"URL: {url}\n\n"
+    if len(markdown) > max_chars:
+        markdown = markdown[:max_chars] + "\n\n[content truncated]"
+    return header + markdown
+
+
+if __name__ == "__main__":
+    mcp.run(transport="stdio")
@@ -0,0 +1,18 @@
+#!/bin/bash
+set -e
+
+CERT_DIR=/etc/voice/certs
+mkdir -p "$CERT_DIR"
+
+LAN_IP=$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{for(i=1;i<=NF;i++) if ($i=="src") print $(i+1)}')
+[ -z "$LAN_IP" ] && LAN_IP=$(hostname -I 2>/dev/null | awk '{print $1}')
+[ -z "$LAN_IP" ] && LAN_IP=127.0.0.1
+
+openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
+    -keyout "$CERT_DIR/key.pem" \
+    -out "$CERT_DIR/cert.pem" \
+    -subj "/CN=$LAN_IP" \
+    -addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:$LAN_IP" 2>/dev/null
+
+echo "voice cert generated for CN=$LAN_IP"
+exec "$@"
@@ -3,8 +3,6 @@ port: 7880
 rtc:
  tcp_port: 7881
  udp_port: 7882
-  port_range_start: 50000
-  port_range_end: 60000

 keys:
  devkey: devsecret
@@ -2,8 +2,10 @@
 const { Room, RoomEvent } = LivekitClient;

 // ── Config ──────────────────────────────────────────────────────────────────
-// LiveKit server runs on the same host. When served over HTTPS, use wss://.
-const LIVEKIT_URL = `${window.location.protocol === "https:" ? "wss" : "ws"}://${window.location.hostname}:7880`;
+// LiveKit is reached through the same origin as the page (nginx proxies
+// /livekit -> localhost:7880). Keeps a single HTTPS port and avoids mixed
+// content when the UI is served over HTTPS.
+const LIVEKIT_URL = `${window.location.protocol === "https:" ? "wss" : "ws"}://${window.location.host}/livekit`;
 const ROOM_NAME = "voice-room";
 const AGENT_NAME = "voice-assistant";

@@ -35,7 +37,102 @@ if (savedVoice) {
    voiceSelect.value = savedVoice;
 }

-// ── Start ───────────────────────────────────────────────────────────────────
+// ── Agent detection ─────────────────────────────────────────────────────────
+// The agent may already be in the room before we join (it persists between
+// page loads), so scan remoteParticipants after connect AND watch for new
+// joins — ParticipantConnected does not fire for pre-existing participants.
+function isAgent(participant) {
+    // kind is the numeric proto enum; ParticipantKind.AGENT === 4.
+    // Prefer the SDK's own getter where available.
+    if (participant.isAgent !== undefined && participant.isAgent !== null) {
+        return participant.isAgent;
+    }
+    return participant.identity === AGENT_NAME
+        || participant.kind === LivekitClient.ParticipantKind.AGENT;
+}
+
+function findAgent(rm) {
+    if (!rm) return null;
+    for (const p of rm.remoteParticipants.values()) {
+        if (isAgent(p)) return p;
+    }
+    return null;
+}
+
+function markAgentFound(rm, participant) {
+    agentParticipant = participant;
+    statusEl.textContent = "Agent connected — speak now";
+    // Send current voice selection to the agent
+    sendVoiceMessage(voiceSelect.value, rm);
+}
+
+function createRoom(token) {
+    const newRoom = new Room({
+        adaptiveStream: true,
+        dynacast: true,
+    });
+
+    // Listen for the agent joining after us
+    newRoom.on(RoomEvent.ParticipantConnected, (participant) => {
+        if (isAgent(participant)) markAgentFound(newRoom, participant);
+    });
+
+    // Listen for transcripts from the agent
+    newRoom.on(RoomEvent.DataReceived, (payload) => {
+        try {
+            const msg = JSON.parse(payload.decodeToString());
+            if (msg.type === "transcript") {
+                addMessage(msg.role || "agent", msg.text);
+            }
+        } catch (e) {
+            // Ignore non-JSON data
+        }
+    });
+
+    // Guard with `=== room` so a stale (retry-discarded) connection cannot
+    // clobber UI state of the active one.
+    newRoom.on(RoomEvent.ConnectionQualityChanged, (participant, quality) => {
+        if (participant === newRoom.localParticipant && newRoom === room) {
+            statusEl.textContent = `Connected (${quality} quality)`;
+        }
+    });
+
+    newRoom.on(RoomEvent.Disconnected, () => {
+        if (newRoom !== room) return;
+        statusEl.textContent = "Disconnected";
+        startBtn.disabled = false;
+        stopBtn.disabled = true;
+        agentParticipant = null;
+    });
+
+    return newRoom.connect(LIVEKIT_URL, token, {
+        autoSubscribe: true,
+    }).then(() => {
+        // Pick up an agent that was already in the room before we joined
+        const existing = findAgent(newRoom);
+        if (existing) markAgentFound(newRoom, existing);
+        return newRoom;
+    });
+}
+
+function waitForAgent(timeoutMs) {
+    return new Promise((resolve) => {
+        if (findAgent(room)) return resolve(true);
+        const deadline = Date.now() + timeoutMs;
+        const timer = setInterval(() => {
+            const found = findAgent(room);
+            if (found) {
+                if (!agentParticipant) markAgentFound(room, found);
+                clearInterval(timer);
+                resolve(true);
+            } else if (Date.now() > deadline) {
+                clearInterval(timer);
+                resolve(false);
+            }
+        }, 250);
+    });
+}
+
 startBtn.addEventListener("click", async () => {
    try {
        statusEl.textContent = "Connecting...";
@@ -43,57 +140,36 @@ startBtn.addEventListener("click", async () => {
        // Get a signed access token from the token endpoint
        const token = await fetchToken(ROOM_NAME);

-        room = new Room({
-            adaptiveStream: true,
-            dynacast: true,
-        });
+        room = await createRoom(token);

-        // Listen for agent participant joining
-        room.on(RoomEvent.ParticipantConnected, (participant) => {
-            if (participant.identity === AGENT_NAME || participant.kind === "agent") {
-                agentParticipant = participant;
-                statusEl.textContent = "Agent connected — speak now";
-
-                // Send current voice selection to the agent
-                const voice = voiceSelect.value;
-                sendVoiceMessage(voice);
+        // The agent is dispatched when we join. If it misses the dispatch
+        // (e.g. the server just started and the worker wasn't ready yet),
+        // reconnect once to trigger a fresh dispatch.
+        statusEl.textContent = "Waiting for assistant...";
+        if (!(await waitForAgent(10000))) {
+            statusEl.textContent = "Assistant not ready, retrying...";
+            const stale = room;
+            room = null;          // detach first so its Disconnected handler
+            agentParticipant = null;  // cannot clobber the UI
+            await stale.disconnect();
+            room = await createRoom(token);
+            if (!(await waitForAgent(15000))) {
+                throw new Error("Assistant did not join — reload the page and try again");
            }
-        });
+        }

-        // Listen for transcripts from the agent
-        room.on(RoomEvent.DataReceived, (payload, participant) => {
-            try {
-                const msg = JSON.parse(payload.decodeToString());
-                if (msg.type === "transcript") {
-                    addMessage(msg.role || "agent", msg.text);
-                }
-            } catch (e) {
-                // Ignore non-JSON data
-            }
-        });
-
-        room.on(RoomEvent.ConnectionQualityChanged, (participant, quality) => {
-            if (participant === room.localParticipant) {
-                statusEl.textContent = `Connected (${quality} quality)`;
-            }
-        });
-
-        room.on(RoomEvent.Disconnected, () => {
-            statusEl.textContent = "Disconnected";
-            startBtn.disabled = false;
-            stopBtn.disabled = true;
-            agentParticipant = null;
-        });
-
-        await room.connect(LIVEKIT_URL, {
-            accessToken: token,
-            autoSubscribe: true,
-        });
+        statusEl.textContent = "Requesting microphone...";
+        await room.localParticipant.setMicrophoneEnabled(true);

        statusEl.textContent = "Connected — speak now";
        startBtn.disabled = true;
        stopBtn.disabled = false;

+        // Live mic level readout: if this stays at 0 while you talk, the
+        // phone is not capturing audio (iOS quirk), and the problem is on
+        // the device side, not the server.
+        startMicMeter();
+
    } catch (err) {
        console.error("Connection failed:", err);
        statusEl.textContent = `Error: ${err.message}`;
@@ -125,15 +201,48 @@ async function fetchToken(roomName) {
 }

 // ── Helper: send voice selection to the agent via data channel ─────────────
-function sendVoiceMessage(voice) {
-    if (!room || !room.localParticipant) return;
+function sendVoiceMessage(voice, targetRoom) {
+    const r = targetRoom || room;
+    if (!r || !r.localParticipant) return;
    const payload = new TextEncoder().encode(JSON.stringify({ type: "set_voice", voice }));
-    room.localParticipant.publishData(payload, {
+    r.localParticipant.publishData(payload, {
        topic: "voice-control",
        reliable: true,
    });
 }

+// ── Helper: live mic level readout (diagnostic) ─────────────────────────────
+function startMicMeter() {
+    const el = document.getElementById("micLevel");
+    if (!el) return;
+    let audioCtx = null;
+    let analyser = null;
+    try {
+        const pub = room.localParticipant.getTrackPublication(LivekitClient.Track.Source.Microphone);
+        const mst = pub && pub.track && pub.track.mediaStreamTrack;
+        if (mst) {
+            audioCtx = new AudioContext();
+            analyser = audioCtx.createAnalyser();
+            analyser.fftSize = 512;
+            audioCtx.createMediaStreamSource(new MediaStream([mst])).connect(analyser);
+        }
+    } catch (e) { /* fall back to SDK audioLevel below */ }
+
+    const buf = analyser ? new Float32Array(analyser.fftSize) : null;
+    window.micMeterTimer = setInterval(() => {
+        if (!room || !room.localParticipant) { clearInterval(window.micMeterTimer); return; }
+        let level = 0;
+        if (analyser) {
+            analyser.getFloatTimeDomainData(buf);
+            for (let i = 0; i < buf.length; i++) level = Math.max(level, Math.abs(buf[i]));
+        } else {
+            level = room.localParticipant.audioLevel;
+        }
+        el.textContent = `mic: ${Math.round(level * 100)}%`;
+        el.style.color = level > 0.02 ? "#4ade80" : "#94a3b8";
+    }, 300);
+}
+
 // ── Helper: add a message to the transcript ─────────────────────────────────
 function addMessage(role, text) {
    const div = document.createElement("div");
@@ -37,6 +37,7 @@
        </div>

        <div class="status" id="status">Disconnected</div>
+        <div class="status" id="micLevel" style="font-size: 0.8em;"></div>

        <div class="transcript" id="transcript">
            <div class="transcript-header">Conversation</div>
@@ -44,7 +45,7 @@
        </div>
    </div>

-    <script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/index.umd.min.js"></script>
+    <script src="livekit-client.umd.js"></script>
    <script src="app.js"></script>
 </body>
 </html>
@@ -3,6 +3,11 @@

 Serves POST /token -> { "token": "<signed JWT>" }
 Signs a LiveKit access token with HS256 using the API key/secret from env.
+
+The token carries a roomConfig claim requesting the voice-assistant agent, so
+LiveKit dispatches the agent when the participant joins. Without this, the
+auto-created room would get no agent at all.
+
 Runs as a separate supervisord process; nginx proxies /token to it.
 """
 import base64
@@ -17,24 +22,32 @@ from http.server import BaseHTTPRequestHandler, HTTPServer

 API_KEY = os.environ.get("LIVEKIT_API_KEY", "devkey")
 API_SECRET = os.environ.get("LIVEKIT_API_SECRET", "devsecret")
+AGENT_NAME = os.environ.get("VOICE_AGENT_NAME", "voice-assistant")


 def b64url(data: bytes) -> str:
    return base64.urlsafe_b64encode(data).rstrip(b"=").decode()


-def make_token(room_name: str) -> str:
+def make_token(room_name: str, identity: str) -> str:
    now = int(time.time())
    header = {"alg": "HS256", "typ": "JWT"}
    payload = {
        "iss": API_KEY,
-        "sub": room_name,
+        "sub": identity,
        "nbf": now - 5,
        "exp": now + 3600,
        "jti": str(uuid.uuid4()),
-        "video_grants": {
-            "room_join": True,
+        "identity": identity,
+        "video": {
+            "roomJoin": True,
            "room": room_name,
+            "canPublish": True,
+            "canSubscribe": True,
+            "canPublishData": True,
+        },
+        "roomConfig": {
+            "agents": [{"agentName": AGENT_NAME}],
        },
    }
    h = b64url(json.dumps(header).encode())
@@ -54,8 +67,9 @@ class Handler(BaseHTTPRequestHandler):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length) or b"{}")
        room_name = body.get("room", "voice-room")
+        identity = body.get("identity") or f"user-{uuid.uuid4().hex[:8]}"

-        token = make_token(room_name)
+        token = make_token(room_name, identity)
        resp = json.dumps({"token": token}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")