feat: connect Shane voice sessions to memory

This commit is contained in:
snowspeeder committed 2026-08-28 16:22:18 -04:00
1 parent 2a2a2ff768
commit c940923539
15 files changed
+201 -97

No files matched your search

+5 -5
View File
@@ -8,7 +8,7 @@ One Docker container runs three processes via supervisord:
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
3. **Web frontend** — static HTML served by a tiny HTTP server (port 8090)
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
```
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
@@ -27,10 +27,10 @@ export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)
docker compose up --build -d
# 3. Open the web UI (self-signed cert: accept the browser warning once)
# https://<host-ip>:8090
# https://<host-ip>:9178
```
LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
(WebRTC media), `7881/tcp` (media TCP fallback).
## Ports
@@ -40,7 +40,7 @@ LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
| 8090 | TCP | Web frontend (HTTPS) | LAN |
| 9178 | TCP | Web frontend (HTTPS) | LAN |
## Configuration
@@ -125,7 +125,7 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The cert is persisted in `./certs/` (mounted as a volume) and only regenerated when the LAN IP changes, not on every container start. The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.