feat: connect Shane voice sessions to memory
This commit is contained in:
1 parent
2a2a2ff768
commit
c940923539
15 files changed
+201
-97
No files matched your search
@@ -8,7 +8,7 @@ One Docker container runs three processes via supervisord:
|
||||
|
||||
1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 7882 UDP muxed media)
|
||||
2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
|
||||
3. **Web frontend** — static HTML served by a tiny HTTP server (port 8090)
|
||||
3. **Web frontend** — static HTML served by nginx (port 9178 by default)
|
||||
|
||||
```
|
||||
Browser ──WebRTC──► LiveKit Server ──audio──► Agent
|
||||
@@ -27,10 +27,10 @@ export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)
|
||||
docker compose up --build -d
|
||||
|
||||
# 3. Open the web UI (self-signed cert: accept the browser warning once)
|
||||
# https://<host-ip>:8090
|
||||
# https://<host-ip>:9178
|
||||
```
|
||||
|
||||
LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
|
||||
LAN access requires firewall rules: `9178/tcp` (UI + signaling), `7882/udp`
|
||||
(WebRTC media), `7881/tcp` (media TCP fallback).
|
||||
|
||||
## Ports
|
||||
@@ -40,7 +40,7 @@ LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
|
||||
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
|
||||
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
|
||||
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
|
||||
| 8090 | TCP | Web frontend (HTTPS) | LAN |
|
||||
| 9178 | TCP | Web frontend (HTTPS) | LAN |
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -125,7 +125,7 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
|
||||
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
|
||||
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
|
||||
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The cert is persisted in `./certs/` (mounted as a volume) and only regenerated when the LAN IP changes, not on every container start. The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
|
||||
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on `WEB_PORT` (9178 by default) over HTTPS with a self-signed cert whose SAN includes the detected LAN IP. The cert is persisted in `./certs/` and the LiveKit WS is proxied through nginx at `/livekit/`.
|
||||
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
|
||||
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
|
||||
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
|
||||
|
||||
Reference in new issue
Block a user