feat: web access via MCP (Firecrawl search/scrape) + mcp server attach support
- agent/web_mcp.py: stdio MCP server exposing web_search and web_scrape, backed by the self-hosted Firecrawl stack on xNAS (no API key needed) - agent.py: Agent now attaches mcp_servers built from config; EXTRA_MCP_SERVERS env var allows adding arbitrary HTTP/SSE MCP servers as JSON - Dockerfile: installs livekit-agents[mcp], copies web_mcp.py - .env.example: WEB_MCP_ENABLED, FIRECRAWL_BASE, EXTRA_MCP_SERVERS documented
This commit is contained in:
@@ -6,7 +6,7 @@ A single-container voice assistant built on LiveKit Agents. Speaks and listens i
|
||||
|
||||
One Docker container runs three processes via supervisord:
|
||||
|
||||
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 50000-60000 UDP)
|
||||
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
|
||||
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
|
||||
3. **Web frontend** — static HTML served by a tiny HTTP server (port 8090)
|
||||
|
||||
@@ -26,18 +26,21 @@ export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)
|
||||
# 2. Build and start
|
||||
docker compose up --build -d
|
||||
|
||||
# 3. Open the web UI
|
||||
# http://<host-ip>:8090
|
||||
# 3. Open the web UI (self-signed cert: accept the browser warning once)
|
||||
# https://<host-ip>:8090
|
||||
```
|
||||
|
||||
LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
|
||||
(WebRTC media), `7881/tcp` (media TCP fallback).
|
||||
|
||||
## Ports
|
||||
|
||||
| Port | Protocol | Service | Access |
|
||||
|-------|----------|----------------------|--------------|
|
||||
| 7880 | TCP | LiveKit HTTP/WS | LAN |
|
||||
| 7881 | TCP | LiveKit internal | container |
|
||||
| 50000-60000 | UDP | LiveKit RTC media | LAN |
|
||||
| 8090 | TCP | Web frontend | LAN |
|
||||
| 7880 | TCP | LiveKit HTTP/WS | container (proxied via 8090/livekit) |
|
||||
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
|
||||
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
|
||||
| 8090 | TCP | Web frontend (HTTPS) | LAN |
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -93,28 +96,34 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
├── AGENTS.md ← you are here
|
||||
├── .env.example ← config template (copy to .env)
|
||||
├── .gitignore
|
||||
├── docker-compose.yml ← single container, 3 processes
|
||||
├── docker-compose.yml ← single container
|
||||
├── Dockerfile ← multi-stage build
|
||||
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
|
||||
├── livekit.yaml ← LiveKit server config
|
||||
├── supervisord.conf ← process manager for the 3 services
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
|
||||
├── agent/
|
||||
│ ├── agent.py ← LiveKit Agents voice pipeline
|
||||
│ └── pyproject.toml ← Python deps (uv)
|
||||
├── web/
|
||||
│ ├── index.html ← single-page voice UI
|
||||
│ ├── app.js ← LiveKit client logic
|
||||
│ └── style.css ← minimal dark theme
|
||||
└── docs/
|
||||
└── ARCHITECTURE.md ← deeper architecture notes
|
||||
└── web/
|
||||
├── index.html ← single-page voice UI
|
||||
├── app.js ← LiveKit client logic
|
||||
├── token_server.py ← signs JWTs + roomConfig claim (agent dispatch)
|
||||
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
|
||||
└── style.css ← minimal dark theme
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Single container.** All services (LiveKit, agent, web) run in one Docker container via supervisord. No multi-service compose.
|
||||
- **No published UDP ports in compose.** LiveKit binds its UDP range directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
|
||||
- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
|
||||
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
|
||||
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it.
|
||||
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
|
||||
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
|
||||
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
|
||||
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
|
||||
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty.
|
||||
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `session.update_options(voice=...)` without restarting.
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting.
|
||||
|
||||
## Git
|
||||
|
||||
|
||||
Reference in New Issue
Block a user