feat: implement full UPDATE.md review — critical fixes, UI upgrade, infra hardening
Critical frontend bugs: - Add TrackSubscribed/attach() for agent audio playback - Fix decodeToString TypeError with TextDecoder - XSS fix: innerHTML -> textContent in addMessage - Fresh token on reconnect retry Agent fixes: - GemmaLLM subclass with reasoning_content fallback wrapper - Disable Gemma 4 thinking mode via chat_template_kwargs (6.8s -> 0.5s) - Remove duplicate session-level LLM - Replace global _active_session with closure-based handler - asyncio.create_task instead of deprecated get_event_loop - Explicit silero VAD, topic filter on voice-control Infra: - supervisord: all programs log to /dev/stdout - Dockerfile: uv sync --frozen with committed uv.lock - nginx config moved to real file, token_server.py no longer served - entrypoint.sh: cert persisted, only regenerated on IP change - compose: healthcheck + cert volume - token_server: CORS removed, room pinned to voice-room UI upgrade: - Orb UI with state machine (idle/connecting/listening/thinking/speaking) - Streaming transcripts via lk.transcription text streams - Barge-in hint, thinking chip, audio visualizer - Glassmorphism, chat bubbles, settings sheet, light mode - PWA manifest, favicon, wake-lock, safe-area insets - localStorage conversation history Docs: AGENTS.md drift fixed
This commit is contained in:
Shane
+1
-1
@@ -2,6 +2,6 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
uv.lock
|
||||
node_modules/
|
||||
*.log
|
||||
certs/
|
||||
|
||||
@@ -37,7 +37,7 @@ LAN access requires ufw rules: `8090/tcp` (UI + signaling), `7882/udp`
|
||||
|
||||
| Port | Protocol | Service | Access |
|
||||
|-------|----------|----------------------|--------------|
|
||||
| 7880 | TCP | LiveKit HTTP/WS | container (proxied via 8090/livekit) |
|
||||
| 7880 | TCP | LiveKit HTTP/WS | internal only (proxied via nginx at /livekit/) |
|
||||
| 7881 | TCP | LiveKit RTC media (TCP fallback) | LAN |
|
||||
| 7882 | UDP | LiveKit RTC media (muxed) | LAN |
|
||||
| 8090 | TCP | Web frontend (HTTPS) | LAN |
|
||||
@@ -78,8 +78,11 @@ The LLM is instructed to:
|
||||
# Verify the container is running
|
||||
docker compose ps
|
||||
|
||||
# Check agent logs
|
||||
docker compose logs -f agent
|
||||
# Check all logs (single service: "voice")
|
||||
docker compose logs -f voice
|
||||
|
||||
# Tail individual process logs via supervisord stdout
|
||||
docker exec voice tail -f /dev/stdout
|
||||
|
||||
# Test TTS directly (outside the agent)
|
||||
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
@@ -100,14 +103,18 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
├── Dockerfile ← multi-stage build
|
||||
├── entrypoint.sh ← regenerates self-signed cert with LAN IP at start
|
||||
├── livekit.yaml ← LiveKit server config
|
||||
├── nginx.conf ← HTTPS UI + /livekit/ WS proxy
|
||||
├── supervisord.conf ← process manager (livekit, agent, nginx, token-server)
|
||||
├── certs/ ← persisted self-signed cert (volume mount)
|
||||
├── agent/
|
||||
│ ├── agent.py ← LiveKit Agents voice pipeline
|
||||
│ └── pyproject.toml ← Python deps (uv)
|
||||
│ ├── pyproject.toml ← Python deps (uv)
|
||||
│ └── uv.lock ← locked dependency versions
|
||||
└── web/
|
||||
├── index.html ← single-page voice UI
|
||||
├── app.js ← LiveKit client logic
|
||||
├── token_server.py ← signs JWTs + roomConfig claim (agent dispatch)
|
||||
├── manifest.json ← PWA manifest
|
||||
├── favicon.svg ← site icon
|
||||
├── livekit-client.umd.js ← vendored LiveKit JS SDK (no CDN)
|
||||
└── style.css ← minimal dark theme
|
||||
```
|
||||
@@ -116,14 +123,15 @@ curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
|
||||
|
||||
- **Single container.** All services (LiveKit, agent, web, token endpoint) run in one Docker container via supervisord. No multi-service compose.
|
||||
- **No published UDP ports in compose.** LiveKit binds its media ports directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui.
|
||||
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it.
|
||||
- **Agent dispatch via roomConfig token claim.** LiveKit only dispatches agents to rooms that request them; a room auto-created by a participant join gets none. The token endpoint embeds `roomConfig.agents` in every JWT so the agent is dispatched when the browser joins. Do not pre-create rooms instead — if the agent worker isn't registered yet (first ~15s after container start), the dispatch silently fails and never retries; joining later re-fires it. The token server now pins the room name to "voice-room" server-side and no longer accepts arbitrary room names.
|
||||
- **Interruption mode must be "vad".** `interruption={"mode": "adaptive"}` requires the LiveKit Cloud barge-in service (agent-gateway.livekit.cloud) and spams 401 retries on self-hosted setups.
|
||||
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
|
||||
- **Mic requires HTTPS.** Browsers block getUserMedia outside a secure context. nginx serves the UI on 8090 over HTTPS with a self-signed cert whose SAN includes the detected LAN IP (generated by entrypoint.sh at container start). The cert is persisted in `./certs/` (mounted as a volume) and only regenerated when the LAN IP changes, not on every container start. The LiveKit WS is proxied through nginx at `/livekit/` so everything stays on one origin (no mixed content).
|
||||
- **No CDN dependencies.** livekit-client UMD bundle is vendored into `web/`; LAN devices may have no internet access.
|
||||
- **Transcripts flow over the data channel.** The agent publishes `{type: "transcript", role, text}` JSON on topic "transcript"; the UI renders them. Voice changes flow the other way as `{type: "set_voice", voice}` on topic "voice-control".
|
||||
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty.
|
||||
- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent implements this with `max_completion_tokens=1000` and a `GemmaLLM` subclass that wraps the LLM stream, falling back to `reasoning_content` if `content` is empty.
|
||||
- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally.
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting.
|
||||
- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `tts.update_options(voice=...)` without restarting. The agent filters data messages by topic ("voice-control") before processing.
|
||||
- **All supervisord programs log to /dev/stdout** so `docker compose logs -f voice` shows everything. Individual process logs are no longer written to files.
|
||||
|
||||
## Git
|
||||
|
||||
|
||||
+11
-12
@@ -14,13 +14,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY agent/pyproject.toml ./agent/
|
||||
RUN cd /app/agent && \
|
||||
uv venv .venv && \
|
||||
uv pip install --python .venv/bin/python \
|
||||
"livekit-agents[mcp]~=1.7" \
|
||||
"livekit-plugins-azure~=1.7" \
|
||||
"livekit-plugins-openai~=1.7" \
|
||||
"python-dotenv"
|
||||
COPY agent/uv.lock ./agent/
|
||||
RUN cd /app/agent && uv sync --frozen
|
||||
|
||||
# ── Stage 2: Runtime (use same base for Python compat) ─────────────────────
|
||||
FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim AS runtime
|
||||
@@ -48,7 +43,12 @@ COPY agent/agent.py /opt/voice-agent/agent.py
|
||||
COPY agent/web_mcp.py /opt/voice-agent/web_mcp.py
|
||||
|
||||
# Copy web frontend + token endpoint
|
||||
COPY web/ /var/www/voice/
|
||||
COPY web/index.html /var/www/voice/
|
||||
COPY web/app.js /var/www/voice/
|
||||
COPY web/style.css /var/www/voice/
|
||||
COPY web/livekit-client.umd.js /var/www/voice/
|
||||
COPY web/manifest.json /var/www/voice/
|
||||
COPY web/favicon.svg /var/www/voice/
|
||||
COPY web/token_server.py /opt/voice/token_server.py
|
||||
|
||||
# Config files
|
||||
@@ -59,11 +59,10 @@ COPY supervisord.conf /etc/supervisor/conf.d/voice.conf
|
||||
# Browsers require a secure context (HTTPS or localhost) for microphone access.
|
||||
# The self-signed cert (with the LAN IP in the SAN) is generated at container
|
||||
# start by entrypoint.sh.
|
||||
COPY nginx.conf /etc/nginx/sites-available/voice
|
||||
RUN rm -f /etc/nginx/sites-enabled/default \
|
||||
&& mkdir -p /etc/voice/certs \
|
||||
&& printf 'server {\n listen 8090 ssl;\n root /var/www/voice;\n index index.html;\n ssl_certificate /etc/voice/certs/cert.pem;\n ssl_certificate_key /etc/voice/certs/key.pem;\n location /token {\n proxy_pass http://127.0.0.1:8091/token;\n proxy_set_header Content-Type application/json;\n }\n location = /livekit {\n return 301 /livekit/;\n }\n location /livekit/ {\n proxy_pass http://127.0.0.1:7880/;\n proxy_http_version 1.1;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection "upgrade";\n proxy_set_header Host $host;\n proxy_read_timeout 3600s;\n proxy_send_timeout 3600s;\n }\n location / {\n try_files $uri $uri/ =404;\n }\n}\n' \
|
||||
> /etc/nginx/sites-available/voice \
|
||||
&& ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice
|
||||
&& ln -sf /etc/nginx/sites-available/voice /etc/nginx/sites-enabled/voice \
|
||||
&& mkdir -p /etc/voice/certs
|
||||
|
||||
# Create non-root user for agent
|
||||
RUN useradd -m -s /bin/bash voiceuser || true
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
# UPDATE — Deep Review: Why the Voice Assistant Doesn't Talk Back
|
||||
|
||||
Date: 2026-08-22
|
||||
Scope: full code review of the LiveKit + Azure Speech + Gemma voice stack, root-cause analysis of the "mic moves but no transcript / no speech" failure, plus hardening and product/UX recommendations.
|
||||
|
||||
This is a spec only. No code changes have been made.
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
The backend is essentially fine. I verified inside the built image (`voice-voice:latest`) that `agent.py` imports cleanly against livekit-agents 1.7.0, the full `AgentSession` (Azure STT + Azure TTS + local silero VAD + local turn detector) constructs without error, the Gemma endpoint at `192.168.86.2:8023` answers chat and tool calls, and the local EOT/VAD models are bundled in the wheel (no runtime downloads needed).
|
||||
|
||||
**The product is broken by two front-end bugs in `web/app.js`. Both are confirmed, both are fatal, and together they produce exactly the reported symptoms** (connects, mic meter moves, but no transcript and no audio reply):
|
||||
|
||||
1. **The agent's audio is never played.** `app.js` never handles `RoomEvent.TrackSubscribed` and never calls `track.attach()`. With `autoSubscribe: true` the browser *receives* the agent's TTS track, but the livekit-client SDK does not auto-play audio — you must attach it to a media element. The agent may be speaking perfectly; the user will never hear a single sample.
|
||||
|
||||
2. **Transcripts are silently discarded.** `app.js:83` calls `payload.decodeToString()`. In the vendored livekit-client 2.13.0 UMD bundle, the `DataReceived` payload is a plain `Uint8Array` — there is no `decodeToString` method anywhere in the bundle (verified: zero grep hits). The call throws `TypeError` on every data packet, and the surrounding `try/catch` swallows it, so nothing ever renders.
|
||||
|
||||
Fix those two lines-of-code-level issues and the product should work end to end.
|
||||
|
||||
---
|
||||
|
||||
## 2. Root Causes — Confirmed Bugs (fix these first)
|
||||
|
||||
### 2.1 No audio playback path (web/app.js) — CRITICAL
|
||||
|
||||
**Symptom:** agent never "speaks back."
|
||||
|
||||
**Evidence:** no `TrackSubscribed` handler and no `attach()` call anywhere in `app.js`; no `<audio>` element in `index.html`.
|
||||
|
||||
**Fix (in `createRoom`):**
|
||||
|
||||
```js
|
||||
newRoom.on(RoomEvent.TrackSubscribed, (track, publication, participant) => {
|
||||
if (track.kind === LivekitClient.Track.Kind.Audio) {
|
||||
const el = track.attach(); // creates an <audio> element, autoplays
|
||||
el.id = `audio-${participant.identity}`;
|
||||
document.body.appendChild(el);
|
||||
}
|
||||
});
|
||||
|
||||
newRoom.on(RoomEvent.TrackUnsubscribed, (track) => {
|
||||
track.detach().forEach((el) => el.remove());
|
||||
});
|
||||
```
|
||||
|
||||
**Also required — browser autoplay policy:** even attached audio can be blocked if the context isn't "unlocked." The Start button click is a user gesture, so call `await room.startAudio()` right after `connect()` resolves, and handle the recovery case:
|
||||
|
||||
```js
|
||||
newRoom.on(RoomEvent.AudioPlaybackStatusChanged, () => {
|
||||
if (!newRoom.canPlaybackAudio) {
|
||||
// show a "tap to enable audio" button that calls room.startAudio()
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
This matters most on iOS Safari, which the mic-meter comment in the code says is a target device.
|
||||
|
||||
### 2.2 `payload.decodeToString()` does not exist (web/app.js:83) — CRITICAL
|
||||
|
||||
**Symptom:** no transcripts appear for either side of the conversation.
|
||||
|
||||
**Evidence:** livekit-client 2.13.0 delivers `DataReceived` payloads as `Uint8Array`; `decodeToString` has zero occurrences in the vendored bundle. The `TypeError` is swallowed by the `catch (e) { /* ignore */ }` block.
|
||||
|
||||
**Fix:**
|
||||
|
||||
```js
|
||||
newRoom.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
|
||||
try {
|
||||
const msg = JSON.parse(new TextDecoder().decode(payload));
|
||||
if (msg.type === "transcript") addMessage(msg.role || "agent", msg.text);
|
||||
} catch (e) {
|
||||
console.warn("bad data packet", e); // don't swallow silently
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Better long-term fix:** drop the custom transcript data channel entirely. livekit-agents 1.x already publishes live transcriptions (including *streaming partials*) over text streams on the `lk.transcription` topic. The UI can consume them with:
|
||||
|
||||
```js
|
||||
room.registerTextStreamHandler("lk.transcription", async (reader, participantInfo) => {
|
||||
const text = await reader.readAll();
|
||||
// reader.info.attributes["lk.transcribed_track_id"] distinguishes user vs agent
|
||||
});
|
||||
```
|
||||
|
||||
That removes ~40 lines of custom code on each side and gives you word-by-word streaming transcripts for free (much better UX than the current end-of-utterance-only rendering).
|
||||
|
||||
---
|
||||
|
||||
## 3. Secondary Issues (likely to bite next, in priority order)
|
||||
|
||||
### 3.1 `reasoning_content` handling is documented but not implemented (agent/agent.py)
|
||||
|
||||
AGENTS.md claims: *"The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty."* **Neither exists in the current code.** This is a doc/code regression.
|
||||
|
||||
I verified against the live Gemma endpoint: with tools attached, the model returns `"content": ""` plus a populated `"reasoning_content"` field. If Gemma ever finishes a turn with all its tokens in `reasoning_content` and an empty `content` (which reasoning models do, especially with small `max_tokens`), the agent will speak nothing — a second, intermittent "silent agent" failure mode that will look identical to bug 2.1 from the user's chair.
|
||||
|
||||
**Fix:** restore the documented behavior — either pass `max_tokens` / a `reasoning_effort`-suppressing option through `openai.LLM(...)`, or wrap the LLM node to fall back to `reasoning_content` when `content` is empty. Then update AGENTS.md to match whichever is implemented.
|
||||
|
||||
### 3.2 Duplicate, conflicting LLM instances (agent/agent.py)
|
||||
|
||||
`VoiceAssistant.__init__` builds one `openai.LLM(...)` and `handle_job` builds a *second identical* one for the session. Agent-level components override session-level ones, so the session's `llm=` is dead weight. Keep exactly one (recommend: agent-level, delete the session one) so future config edits can't silently apply to the ignored copy.
|
||||
|
||||
### 3.3 Global `_active_session` (agent/agent.py)
|
||||
|
||||
- It's never cleared when a session ends, so a voice-change message after a job finishes pokes a dead session.
|
||||
- If two rooms are ever dispatched, the second clobbers the first and voice changes route to the wrong room.
|
||||
|
||||
**Fix:** make the data handler a closure inside `handle_job` capturing its own `session`, and register it per-room. Delete the global.
|
||||
|
||||
### 3.4 `asyncio.get_event_loop()` is deprecated (agent/agent.py:210-212)
|
||||
|
||||
Inside the sync `conversation_item_added` callback, use `asyncio.create_task(...)` (a running loop is guaranteed there). `get_event_loop()` in Python 3.12 emits deprecation warnings and will eventually raise.
|
||||
|
||||
### 3.5 Agent logs are invisible to `docker compose logs` (supervisord.conf)
|
||||
|
||||
All four programs write to files under `/var/log/supervisor/` inside the container. `docker compose logs -f` (which AGENTS.md tells you to use — and even names a nonexistent `agent` service) shows nothing. This makes every future debugging session harder than it needs to be.
|
||||
|
||||
**Fix:** in each `[program:x]` block:
|
||||
|
||||
```ini
|
||||
stdout_logfile=/dev/stdout
|
||||
stdout_logfile_maxbytes=0
|
||||
redirect_stderr=true
|
||||
```
|
||||
|
||||
And fix AGENTS.md: it's `docker compose logs -f voice` (single service), or `docker exec voice tail -f /var/log/supervisor/agent.log` today.
|
||||
|
||||
### 3.6 Dockerfile ignores agent/pyproject.toml
|
||||
|
||||
The build stage copies `pyproject.toml` and then hand-installs a separate list with `uv pip install "livekit-agents[mcp]~=1.7" ...`. Two sources of truth, no lockfile — dependency drift is guaranteed and rebuilds aren't reproducible. **Fix:** `uv sync` against the pyproject with a committed `uv.lock`, or delete the pyproject dependency list. `~=1.7` also permits silent minor-version jumps on rebuild; pin exact versions in the lock.
|
||||
|
||||
### 3.7 Self-signed cert regenerated on every container start (entrypoint.sh)
|
||||
|
||||
Every restart mints a new key/cert, so every device re-prompts the scary TLS warning after each deploy. **Fix:** persist `/etc/voice/certs` in a volume and only generate when missing (or when the LAN IP changed).
|
||||
|
||||
### 3.8 Retry path reuses a stale token assumption (web/app.js)
|
||||
|
||||
Minor: the reconnect path reuses the same JWT (fine — 1 h expiry) but if a user leaves the tab open past expiry, Start fails opaquely. Fetch a fresh token on every attempt; they're free.
|
||||
|
||||
---
|
||||
|
||||
## 4. General Code Review
|
||||
|
||||
### agent/agent.py
|
||||
- **Good:** clean separation of config, prompt quality is genuinely good for spoken output, MCP toolset wiring is correct for 1.7 (`MCPToolset` in `tools=` constructs fine — verified in-container), `close_on_disconnect=False` rationale is sound and well-commented.
|
||||
- `session.start(...)` before `ctx.connect()` matches the 1.x documented pattern — correct, leave it.
|
||||
- The `user_state_changed` / `user_input_transcribed` diagnostic logging is great; keep it, but see 3.5 or you'll never see it.
|
||||
- No explicit `vad=` on the session: 1.7 silently defaults to the bundled local silero VAD (verified: `livekit.local_inference` native lib present, constructs offline). Works, but make it explicit — `vad=inference.VAD(model="silero")` — so a future SDK default change can't surprise you.
|
||||
- `interruption={"mode": "vad"}` is correct for self-hosted (the SDK source confirms `adaptive` requires the cloud gateway and is disabled outside dev/hosted mode anyway).
|
||||
- Voice-change handler doesn't filter on the `voice-control` topic; it parses every packet. Harmless today, sloppy tomorrow — check `packet.topic`.
|
||||
|
||||
### web/app.js
|
||||
- Besides the two critical bugs: the agent-detection logic (`isAgent`, poll loop in `waitForAgent`) is reasonable, though `RoomEvent.ParticipantConnected` + initial scan makes the 250 ms poll redundant — a promise resolved from the event handler would be tidier.
|
||||
- The mic meter builds its own `AudioContext` + analyser; livekit-client already exposes `LocalAudioTrack` events and `createAudioAnalyser()` helper. Not wrong, just duplicative.
|
||||
- `stopBtn` handler doesn't clear attached audio elements (moot until 2.1 is fixed — then it leaks one `<audio>` element per session).
|
||||
- `addMessage` interpolates transcript text with `innerHTML` — **XSS via speech**: whatever the STT/LLM emits lands in the DOM unescaped. Use `textContent` for the text node.
|
||||
|
||||
### web/token_server.py
|
||||
- JWT construction is correct (HS256, `roomConfig.agents[].agentName` in protojson camelCase — the server accepts this).
|
||||
- `Access-Control-Allow-Origin: *` on an endpoint that mints publish-capable tokens: any page a LAN user visits can silently get a token and join the room. Same-origin only — drop the CORS headers entirely (the UI is same-origin through nginx).
|
||||
- No room-name validation: the client can request any room. Pin it server-side to `voice-room` or a validated allowlist.
|
||||
- Single-threaded `HTTPServer` is fine at this scale; note it blocks during slow clients.
|
||||
|
||||
### Infra (Dockerfile / nginx / compose / livekit.yaml)
|
||||
- nginx: WS proxy for `/livekit/` is correct (prefix strip + Upgrade headers). The embedded one-line `printf` config in the Dockerfile is unreadable and un-diffable — move it to a real `nginx.conf` file that gets COPYed.
|
||||
- nginx `location /token` sets `proxy_set_header Content-Type ...` — that's a no-op/incorrect use (Content-Type is a client request header); remove.
|
||||
- `COPY web/ /var/www/voice/` publishes `token_server.py` source at `https://host:8090/token_server.py`. No secrets in it, but don't serve server code; exclude it (`.dockerignore`-style or copy files explicitly).
|
||||
- `devkey: devsecret` in livekit.yaml and compose defaults: anyone on the LAN who reads this repo can mint admin tokens. For a LAN toy it's acceptable; generate a random secret at first boot if this ever leaves the bench.
|
||||
- compose has no `healthcheck`; a supervisord child crash-looping (the exact failure class you just debugged) leaves the container "Up" and green. Add a healthcheck that curls `https://localhost:8090/` and hits LiveKit's `/` on 7880.
|
||||
- `supervisord.conf` has `nodaemon=false` while CMD runs `supervisord -n` — the flag wins, but make the file say what it does.
|
||||
- AGENTS.md file-layout section is accurate; the Testing section's `docker compose logs -f agent` is wrong (see 3.5).
|
||||
|
||||
### Doc drift (AGENTS.md)
|
||||
- "max_tokens=1000 / reasoning_content fallback" — documented, not implemented (3.1).
|
||||
- "Interruption mode must be vad" — implemented, consistent. Good.
|
||||
- Ports table says 8090 is "UI + signaling" — correct via the nginx proxy; 7880 needn't be exposed at all with host networking (it is, via EXPOSE — cosmetic).
|
||||
|
||||
---
|
||||
|
||||
## 5. Verification Performed (evidence)
|
||||
|
||||
All checks run against the built image `voice-voice:latest` and the live LAN services:
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `import agent` in-container | OK — no API mismatches against livekit-agents 1.7.0 |
|
||||
| `VoiceAssistant()` + full `AgentSession(...)` construction | OK — MCP toolsets, TurnHandlingOptions accepted |
|
||||
| Default VAD | Local silero via `livekit.local_inference` native lib (34 MB, bundled) — constructs with `--network none` |
|
||||
| Turn detector | Falls back to local `v1-mini` when not on LiveKit Cloud; degrades to endpointing delay if unavailable — no cloud dependency |
|
||||
| Azure STT capabilities | `streaming: True` — no VAD strictly required for transcription |
|
||||
| `azure.TTS.update_options` | Exists — live voice switching is valid |
|
||||
| Gemma `/v1/chat/completions` | Responds; tool calls work; **`reasoning_content` confirmed present with empty `content`** |
|
||||
| `decodeToString` in vendored livekit-client 2.13.0 | **0 occurrences** — confirmed frontend bug |
|
||||
| `TrackSubscribed`/`attach()` in app.js | **Absent** — confirmed frontend bug |
|
||||
| Container state | Not currently running (no runtime logs available) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Recommended Fix Order
|
||||
|
||||
1. `app.js`: add `TrackSubscribed` → `track.attach()` + `startAudio()` + `AudioPlaybackStatusChanged` fallback (2.1).
|
||||
2. `app.js`: `TextDecoder` fix for `DataReceived` (2.2) — or migrate both sides to `lk.transcription` text streams.
|
||||
3. `agent.py`: reasoning_content/max_tokens handling (3.1).
|
||||
4. supervisord → stdout logging + AGENTS.md log command fix (3.5).
|
||||
5. `app.js`: `innerHTML` → `textContent` (XSS).
|
||||
6. Remaining review items as time allows (3.2-3.8, section 4).
|
||||
|
||||
**Test plan after 1+2:**
|
||||
- `docker compose up --build -d`, wait ~15 s for worker registration.
|
||||
- Browser → Start → say "hello". Expect: your words render as a user message (streamed if using text streams), agent audio plays within ~1-2 s, agent message renders.
|
||||
- Watch `docker exec voice tail -f /var/log/supervisor/agent.log`: confirm `user state -> speaking`, `STT (final): ...`, then TTS synthesis.
|
||||
- Change voice mid-session; next reply uses the new voice.
|
||||
- Kill/reload the tab, rejoin: agent should still respond (`close_on_disconnect=False` path).
|
||||
- iPhone Safari: verify the "tap to enable audio" fallback fires and works.
|
||||
|
||||
---
|
||||
|
||||
## 7. Product / Theming & Visual Upgrade Proposals
|
||||
|
||||
The current UI is a functional dev console. To make it feel like a product:
|
||||
|
||||
### Core interaction model
|
||||
- **Replace the transcript-first layout with a voice-first "orb" UI.** A large centered circle that breathes in idle, pulses with *your* mic level while listening (you already compute the level — drive the orb instead of a "mic: 42%" debug string), and ripples with the *agent's* output level while speaking. This one element communicates listening/thinking/speaking state instantly and is the signature visual of every good voice product.
|
||||
- **Explicit state machine surfaced in the UI:** `idle → connecting → listening → thinking → speaking`, driven by the agent's `lk.agent.state` participant attribute (the SDK publishes it — free to consume). Replace the freeform status strings.
|
||||
- **Streaming transcripts with partials** (via `lk.transcription`): words appear as you say them, greyed until final. Massive perceived-latency win.
|
||||
- **Tool-use indicator:** when Gemma calls `web_search`, show a subtle "searching the web..." chip. Right now web lookups are silent 5-10 s gaps that feel like crashes.
|
||||
- **Barge-in affordance:** show a small "tap or speak to interrupt" hint while the agent is speaking.
|
||||
|
||||
### Visual design
|
||||
- Keep the dark theme, but commit to it: one accent hue (the blue→violet gradient is good), true-black `#09090b` base, glassmorphism panels (`backdrop-filter: blur`) for the transcript drawer, `Inter`/`Geist` variable font instead of system stack.
|
||||
- Chat bubbles: user right-aligned, agent left-aligned with a small avatar dot in the accent gradient; timestamps on hover. Drop the border-left style.
|
||||
- Motion: orb idle animation (4 s ease breathing), message enter transitions (~120 ms fade/slide), respect `prefers-reduced-motion`.
|
||||
- The voice picker becomes a proper settings sheet: voice cards with a **"preview" play button** (one cached Azure TTS sample per voice), plus speaking-rate and style (`mstts:express-as`) controls — the SSML plumbing already exists.
|
||||
- Light-mode variant via `prefers-color-scheme`.
|
||||
|
||||
### Mobile / platform
|
||||
- PWA manifest + icons so it installs to the home screen (the HTTPS requirement is already met); `theme-color` meta; safe-area insets.
|
||||
- Wake-lock (`navigator.wakeLock`) during a session so the phone doesn't sleep mid-conversation.
|
||||
- Favicon (currently a 404 on every load).
|
||||
|
||||
### Nice-to-haves that differentiate
|
||||
- Conversation history persisted to `localStorage` with a "clear" button.
|
||||
- Latency HUD (dev toggle): STT-final→first-audio time per turn — you'll want it while tuning Gemma.
|
||||
- Push-to-talk mode toggle for noisy rooms (disable VAD interruption, hold spacebar/tap-and-hold the orb).
|
||||
- Audio visualizer bars on the agent bubble while it speaks (a `createAudioAnalyser()` on the remote track — the same trick the mic meter already does locally).
|
||||
|
||||
---
|
||||
|
||||
*End of review.*
|
||||
+110
-43
@@ -19,6 +19,8 @@ from livekit.agents import (
|
||||
JobContext,
|
||||
TurnHandlingOptions,
|
||||
cli,
|
||||
inference,
|
||||
llm as lk_llm,
|
||||
mcp,
|
||||
room_io,
|
||||
)
|
||||
@@ -70,6 +72,85 @@ SYSTEM_PROMPT = textwrap.dedent("""\
|
||||
""")
|
||||
|
||||
|
||||
# ── Gemma LLM with reasoning_content fallback ───────────────────────────────
|
||||
|
||||
|
||||
class _ReasoningFallbackWrapper:
|
||||
"""Wraps an LLMStream to fall back to ``reasoning_content`` when the model
|
||||
finishes a turn with empty visible content (Gemma spends its whole budget
|
||||
on hidden reasoning). Delegates all iteration to the underlying stream and
|
||||
injects a final content chunk if no real content was produced."""
|
||||
|
||||
def __init__(self, inner: lk_llm.LLMStream) -> None:
|
||||
self._inner = inner
|
||||
self._last_reasoning: str | None = None
|
||||
self._has_content = False
|
||||
|
||||
@property
|
||||
def chat_ctx(self) -> lk_llm.ChatContext:
|
||||
return self._inner.chat_ctx
|
||||
|
||||
@property
|
||||
def tools(self) -> list:
|
||||
return self._inner.tools
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._inner.aclose()
|
||||
|
||||
async def __aenter__(self) -> "_ReasoningFallbackWrapper":
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc) -> None:
|
||||
await self.aclose()
|
||||
|
||||
def __aiter__(self) -> "_ReasoningFallbackWrapper":
|
||||
return self
|
||||
|
||||
async def __anext__(self) -> lk_llm.ChatChunk:
|
||||
try:
|
||||
chunk = await self._inner.__anext__()
|
||||
except StopAsyncIteration:
|
||||
# Stream exhausted — inject reasoning fallback if no content was produced
|
||||
if not self._has_content and self._last_reasoning:
|
||||
logger.warning(
|
||||
"LLM returned empty content; falling back to reasoning_content"
|
||||
)
|
||||
return lk_llm.ChatChunk(
|
||||
id="reasoning-fallback",
|
||||
delta=lk_llm.ChoiceDelta(role="assistant", content=self._last_reasoning),
|
||||
)
|
||||
raise
|
||||
# Track reasoning_content from the chunk's delta if present
|
||||
delta = getattr(chunk, "delta", None)
|
||||
if delta is not None:
|
||||
reasoning = getattr(delta, "reasoning_content", None)
|
||||
if reasoning:
|
||||
self._last_reasoning = reasoning
|
||||
if chunk.has_response():
|
||||
self._has_content = True
|
||||
return chunk
|
||||
|
||||
async def collect(self):
|
||||
return await self._inner.collect()
|
||||
|
||||
|
||||
class GemmaLLM(openai.LLM):
|
||||
"""OpenAI-compatible LLM (llama.cpp Gemma 4) with thinking disabled."""
|
||||
|
||||
def chat(self, *, chat_ctx, tools=None, conn_options=None, **kwargs):
|
||||
if conn_options is None:
|
||||
from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS
|
||||
|
||||
conn_options = DEFAULT_API_CONNECT_OPTIONS
|
||||
stream = super().chat(
|
||||
chat_ctx=chat_ctx,
|
||||
tools=tools,
|
||||
conn_options=conn_options,
|
||||
**kwargs,
|
||||
)
|
||||
return _ReasoningFallbackWrapper(stream)
|
||||
|
||||
|
||||
# ── MCP toolsets (web access + any extra configured servers) ────────────────
|
||||
|
||||
def build_mcp_toolsets() -> list[mcp.MCPToolset]:
|
||||
@@ -125,10 +206,12 @@ class VoiceAssistant(Agent):
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(
|
||||
llm=openai.LLM(
|
||||
llm=GemmaLLM(
|
||||
model=GEMMA_MODEL,
|
||||
base_url=GEMMA_BASE_URL,
|
||||
api_key=GEMMA_API_KEY,
|
||||
max_completion_tokens=1000,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
),
|
||||
instructions=SYSTEM_PROMPT,
|
||||
tools=build_mcp_toolsets(),
|
||||
@@ -138,14 +221,9 @@ class VoiceAssistant(Agent):
|
||||
# ── Agent server ────────────────────────────────────────────────────────────
|
||||
server = AgentServer()
|
||||
|
||||
# Track the active session so we can update its voice on data messages.
|
||||
_active_session: AgentSession | None = None
|
||||
|
||||
|
||||
@server.rtc_session(agent_name="voice-assistant")
|
||||
async def handle_job(ctx: JobContext) -> None:
|
||||
global _active_session
|
||||
|
||||
logger.info("Job started for room %s", ctx.room.name)
|
||||
|
||||
# Azure STT — streaming, reads AZURE_SPEECH_KEY / AZURE_SPEECH_REGION from env
|
||||
@@ -163,17 +241,10 @@ async def handle_job(ctx: JobContext) -> None:
|
||||
speech_region=AZURE_REGION,
|
||||
)
|
||||
|
||||
# Gemma LLM via OpenAI-compatible endpoint (llama.cpp on xNAS)
|
||||
llm = openai.LLM(
|
||||
model=GEMMA_MODEL,
|
||||
base_url=GEMMA_BASE_URL,
|
||||
api_key=GEMMA_API_KEY,
|
||||
)
|
||||
|
||||
session = AgentSession(
|
||||
stt=stt,
|
||||
tts=tts,
|
||||
llm=llm,
|
||||
vad=inference.VAD(model="silero"),
|
||||
turn_handling=TurnHandlingOptions(
|
||||
# VAD-based turn detection: agent waits for user to stop speaking.
|
||||
# ("adaptive" mode requires the LiveKit Cloud barge-in service.)
|
||||
@@ -183,9 +254,30 @@ async def handle_job(ctx: JobContext) -> None:
|
||||
),
|
||||
)
|
||||
|
||||
_active_session = session
|
||||
# Listen for data messages (voice switching) from the web UI.
|
||||
# Defined here so it captures this job's session via closure.
|
||||
def _on_room_data(packet) -> None:
|
||||
topic = getattr(packet, "topic", None)
|
||||
if topic and topic != "voice-control":
|
||||
return
|
||||
try:
|
||||
msg = json.loads(packet.data.decode("utf-8"))
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
return
|
||||
if msg.get("type") != "set_voice":
|
||||
return
|
||||
voice = msg.get("voice", "")
|
||||
if not voice:
|
||||
return
|
||||
logger.info("Switching TTS voice to %s", voice)
|
||||
try:
|
||||
tts = session.tts
|
||||
if hasattr(tts, "update_options"):
|
||||
tts.update_options(voice=voice)
|
||||
logger.info("Voice updated to %s", voice)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("Failed to update voice: %s", e)
|
||||
|
||||
# Listen for data messages (voice switching) from the web UI
|
||||
ctx.room.on("data_received", _on_room_data)
|
||||
|
||||
# Publish user/agent transcripts to the room so the web UI can render them.
|
||||
@@ -207,9 +299,9 @@ async def handle_job(ctx: JobContext) -> None:
|
||||
role = getattr(msg, "role", None)
|
||||
text = getattr(msg, "text_content", None)
|
||||
if role == "user":
|
||||
asyncio.get_event_loop().create_task(publish_transcript("user", text))
|
||||
asyncio.create_task(publish_transcript("user", text))
|
||||
elif role == "assistant":
|
||||
asyncio.get_event_loop().create_task(publish_transcript("agent", text))
|
||||
asyncio.create_task(publish_transcript("agent", text))
|
||||
|
||||
@session.on("user_input_transcribed")
|
||||
def _on_user_transcribed(ev) -> None:
|
||||
@@ -240,30 +332,5 @@ async def handle_job(ctx: JobContext) -> None:
|
||||
logger.info("Agent connected to room %s", ctx.room.name)
|
||||
|
||||
|
||||
# ── Voice switching via data channel ────────────────────────────────────────
|
||||
# The web UI sends a JSON data message: {"type": "set_voice", "voice": "en-US-AvaNeural"}
|
||||
# We listen on the room's data channel and update the TTS voice live.
|
||||
|
||||
|
||||
def _on_room_data(packet) -> None:
|
||||
"""Handle data messages from the web UI (voice selection)."""
|
||||
try:
|
||||
msg = json.loads(packet.data.decode("utf-8"))
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
return
|
||||
|
||||
if msg.get("type") == "set_voice":
|
||||
voice = msg.get("voice", "")
|
||||
if voice and _active_session:
|
||||
logger.info("Switching TTS voice to %s", voice)
|
||||
try:
|
||||
tts = _active_session.tts
|
||||
if hasattr(tts, "update_options"):
|
||||
tts.update_options(voice=voice)
|
||||
logger.info("Voice updated to %s", voice)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to update voice: %s", e)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
cli.run_app(server)
|
||||
|
||||
@@ -9,7 +9,6 @@ description = "Real-time voice assistant: Azure STT/TTS + Gemma LLM via LiveKit
|
||||
requires-python = ">=3.10,<3.15"
|
||||
|
||||
dependencies = [
|
||||
"livekit-agents~=1.7",
|
||||
"livekit-agents[mcp]~=1.7",
|
||||
"livekit-plugins-azure~=1.7",
|
||||
"livekit-plugins-openai~=1.7",
|
||||
|
||||
Generated
+3040
File diff suppressed because it is too large
Load Diff
@@ -17,4 +17,11 @@ services:
|
||||
FIRECRAWL_BASE: "${FIRECRAWL_BASE:-http://192.168.86.2:3002}"
|
||||
volumes:
|
||||
- ./livekit.yaml:/etc/livekit.yaml:ro
|
||||
- ./certs:/etc/voice/certs
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -sk https://localhost:8090/ -o /dev/null && curl -s http://localhost:7880/ -o /dev/null"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
restart: unless-stopped
|
||||
|
||||
+21
-6
@@ -8,11 +8,26 @@ LAN_IP=$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{for(i=1;i<=NF;i++) if ($i==
|
||||
[ -z "$LAN_IP" ] && LAN_IP=$(hostname -I 2>/dev/null | awk '{print $1}')
|
||||
[ -z "$LAN_IP" ] && LAN_IP=127.0.0.1
|
||||
|
||||
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout "$CERT_DIR/key.pem" \
|
||||
-out "$CERT_DIR/cert.pem" \
|
||||
-subj "/CN=$LAN_IP" \
|
||||
-addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:$LAN_IP" 2>/dev/null
|
||||
# Only regenerate if cert is missing or was issued for a different IP
|
||||
if [ ! -f "$CERT_DIR/cert.pem" ]; then
|
||||
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout "$CERT_DIR/key.pem" \
|
||||
-out "$CERT_DIR/cert.pem" \
|
||||
-subj "/CN=$LAN_IP" \
|
||||
-addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:$LAN_IP" 2>/dev/null
|
||||
echo "voice cert generated for CN=$LAN_IP"
|
||||
else
|
||||
existing_cn=$(openssl x509 -in "$CERT_DIR/cert.pem" -noout -subject 2>/dev/null | grep -o 'CN = [^,]*' | cut -d' ' -f3)
|
||||
if [ "$existing_cn" != "$LAN_IP" ]; then
|
||||
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout "$CERT_DIR/key.pem" \
|
||||
-out "$CERT_DIR/cert.pem" \
|
||||
-subj "/CN=$LAN_IP" \
|
||||
-addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:$LAN_IP" 2>/dev/null
|
||||
echo "voice cert regenerated for CN=$LAN_IP (was $existing_cn)"
|
||||
else
|
||||
echo "voice cert already valid for CN=$LAN_IP"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "voice cert generated for CN=$LAN_IP"
|
||||
exec "$@"
|
||||
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
server {
|
||||
listen 8090 ssl;
|
||||
root /var/www/voice;
|
||||
index index.html;
|
||||
ssl_certificate /etc/voice/certs/cert.pem;
|
||||
ssl_certificate_key /etc/voice/certs/key.pem;
|
||||
|
||||
location /token {
|
||||
proxy_pass http://127.0.0.1:8091/token;
|
||||
}
|
||||
|
||||
location = /livekit {
|
||||
return 301 /livekit/;
|
||||
}
|
||||
|
||||
location /livekit/ {
|
||||
proxy_pass http://127.0.0.1:7880/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ =404;
|
||||
}
|
||||
}
|
||||
+13
-9
@@ -1,5 +1,5 @@
|
||||
[supervisord]
|
||||
nodaemon=false
|
||||
nodaemon=true
|
||||
logfile=/var/log/supervisor/supervisord.log
|
||||
pidfile=/run/supervisord.pid
|
||||
|
||||
@@ -8,16 +8,18 @@ command=/usr/local/bin/livekit --config /etc/livekit.yaml
|
||||
user=root
|
||||
autostart=true
|
||||
autorestart=true
|
||||
stdout_logfile=/var/log/supervisor/livekit.log
|
||||
stderr_logfile=/var/log/supervisor/livekit_err.log
|
||||
stdout_logfile=/dev/stdout
|
||||
stdout_logfile_maxbytes=0
|
||||
redirect_stderr=true
|
||||
|
||||
[program:agent]
|
||||
command=/opt/voice-agent/.venv/bin/python /opt/voice-agent/agent.py start
|
||||
user=voiceuser
|
||||
autostart=true
|
||||
autorestart=true
|
||||
stdout_logfile=/var/log/supervisor/agent.log
|
||||
stderr_logfile=/var/log/supervisor/agent_err.log
|
||||
stdout_logfile=/dev/stdout
|
||||
stdout_logfile_maxbytes=0
|
||||
redirect_stderr=true
|
||||
environment=
|
||||
LIVEKIT_URL="%(ENV_LIVEKIT_URL)s",
|
||||
LIVEKIT_API_KEY="%(ENV_LIVEKIT_API_KEY)s",
|
||||
@@ -36,16 +38,18 @@ command=/usr/sbin/nginx -g "daemon off;"
|
||||
user=root
|
||||
autostart=true
|
||||
autorestart=true
|
||||
stdout_logfile=/var/log/supervisor/web.log
|
||||
stderr_logfile=/var/log/supervisor/web_err.log
|
||||
stdout_logfile=/dev/stdout
|
||||
stdout_logfile_maxbytes=0
|
||||
redirect_stderr=true
|
||||
|
||||
[program:token-server]
|
||||
command=/opt/voice-agent/.venv/bin/python /opt/voice/token_server.py
|
||||
user=voiceuser
|
||||
autostart=true
|
||||
autorestart=true
|
||||
stdout_logfile=/var/log/supervisor/token.log
|
||||
stderr_logfile=/var/log/supervisor/token_err.log
|
||||
stdout_logfile=/dev/stdout
|
||||
stdout_logfile_maxbytes=0
|
||||
redirect_stderr=true
|
||||
environment=
|
||||
LIVEKIT_API_KEY="%(ENV_LIVEKIT_API_KEY)s",
|
||||
LIVEKIT_API_SECRET="%(ENV_LIVEKIT_API_SECRET)s"
|
||||
|
||||
+469
-70
@@ -9,32 +9,429 @@ const LIVEKIT_URL = `${window.location.protocol === "https:" ? "wss" : "ws"}://$
|
||||
const ROOM_NAME = "voice-room";
|
||||
const AGENT_NAME = "voice-assistant";
|
||||
|
||||
const VOICES = [
|
||||
{ id: "en-US-AvaNeural", name: "Ava", tag: "DragonHD" },
|
||||
{ id: "en-US-JennyNeural", name: "Jenny" },
|
||||
{ id: "en-US-GuyNeural", name: "Guy" },
|
||||
{ id: "en-US-AndrewNeural", name: "Andrew" },
|
||||
{ id: "en-US-AriaNeural", name: "Aria" },
|
||||
{ id: "en-US-EmmaNeural", name: "Emma" },
|
||||
{ id: "en-US-EricNeural", name: "Eric" },
|
||||
{ id: "en-US-BrianNeural", name: "Brian" },
|
||||
{ id: "en-US-AshleyNeural", name: "Ashley" },
|
||||
{ id: "en-US-RichardNeural", name: "Richard" },
|
||||
{ id: "en-US-TinaNeural", name: "Tina" },
|
||||
{ id: "en-US-SteffanNeural", name: "Steffan" },
|
||||
];
|
||||
|
||||
const HISTORY_KEY = "voice_history";
|
||||
const HISTORY_MAX = 50;
|
||||
|
||||
// ── State ───────────────────────────────────────────────────────────────────
|
||||
let room = null;
|
||||
let agentParticipant = null;
|
||||
let uiState = "idle"; // idle | connecting | listening | thinking | speaking
|
||||
let micLevel = 0; // 0..1, smoothed
|
||||
let agentAudioLevel = 0; // 0..1, smoothed
|
||||
let userSpeaking = false; // local VAD-ish flag from mic level
|
||||
let wakeLock = null;
|
||||
|
||||
// ── DOM ─────────────────────────────────────────────────────────────────────
|
||||
const startBtn = document.getElementById("startBtn");
|
||||
const stopBtn = document.getElementById("stopBtn");
|
||||
const statusEl = document.getElementById("status");
|
||||
const messagesEl = document.getElementById("messages");
|
||||
const voiceSelect = document.getElementById("voiceSelect");
|
||||
const orbEl = document.getElementById("orb");
|
||||
const stateLabelEl = document.getElementById("stateLabel");
|
||||
const thinkingChipEl = document.getElementById("thinkingChip");
|
||||
const bargeHintEl = document.getElementById("bargeHint");
|
||||
const micMeterFillEl = document.getElementById("micMeterFill");
|
||||
const settingsBtn = document.getElementById("settingsBtn");
|
||||
const settingsSheet = document.getElementById("settingsSheet");
|
||||
const sheetBackdrop = document.getElementById("sheetBackdrop");
|
||||
const closeSheetBtn = document.getElementById("closeSheetBtn");
|
||||
const voiceListEl = document.getElementById("voiceList");
|
||||
const clearBtn = document.getElementById("clearBtn");
|
||||
|
||||
// ── Voice switching ─────────────────────────────────────────────────────────
|
||||
voiceSelect.addEventListener("change", () => {
|
||||
const voice = voiceSelect.value;
|
||||
if (room && agentParticipant) {
|
||||
sendVoiceMessage(voice);
|
||||
addMessage("system", `Voice changed to ${voice}`);
|
||||
} else {
|
||||
localStorage.setItem("preferred_voice", voice);
|
||||
// ── Audio analysers (mic + remote agent audio) ─────────────────────────────
|
||||
let micCtx = null;
|
||||
let micAnalyser = null;
|
||||
let micBuf = null;
|
||||
let remoteCtx = null;
|
||||
let remoteAnalyser = null;
|
||||
let remoteFreqBuf = null;
|
||||
|
||||
function setupMicAnalyser() {
|
||||
try {
|
||||
const pub = room.localParticipant.getTrackPublication(LivekitClient.Track.Source.Microphone);
|
||||
const mst = pub && pub.track && pub.track.mediaStreamTrack;
|
||||
if (mst) {
|
||||
micCtx = new AudioContext();
|
||||
micAnalyser = micCtx.createAnalyser();
|
||||
micAnalyser.fftSize = 512;
|
||||
micBuf = new Float32Array(micAnalyser.fftSize);
|
||||
micCtx.createMediaStreamSource(new MediaStream([mst])).connect(micAnalyser);
|
||||
}
|
||||
} catch (e) { /* fall back to SDK audioLevel in the meter loop */ }
|
||||
}
|
||||
|
||||
function setupRemoteAudioAnalyser(audioEl) {
|
||||
try {
|
||||
if (!audioEl.srcObject) return;
|
||||
remoteCtx = new AudioContext();
|
||||
remoteAnalyser = remoteCtx.createAnalyser();
|
||||
remoteAnalyser.fftSize = 256;
|
||||
remoteFreqBuf = new Uint8Array(remoteAnalyser.frequencyBinCount);
|
||||
remoteCtx.createMediaStreamSource(audioEl.srcObject).connect(remoteAnalyser);
|
||||
} catch (e) { /* analyser optional */ }
|
||||
}
|
||||
|
||||
function teardownAudio() {
|
||||
for (const ctx of [micCtx, remoteCtx]) {
|
||||
if (ctx && ctx.state !== "closed") ctx.close().catch(() => {});
|
||||
}
|
||||
micCtx = micAnalyser = micBuf = null;
|
||||
remoteCtx = remoteAnalyser = remoteFreqBuf = null;
|
||||
}
|
||||
|
||||
function readMicLevel() {
|
||||
let level = 0;
|
||||
if (micAnalyser && micBuf) {
|
||||
micAnalyser.getFloatTimeDomainData(micBuf);
|
||||
for (let i = 0; i < micBuf.length; i++) level = Math.max(level, Math.abs(micBuf[i]));
|
||||
} else if (room && room.localParticipant) {
|
||||
level = room.localParticipant.audioLevel || 0;
|
||||
}
|
||||
// Smooth so the orb pulses rather than flickers
|
||||
micLevel = micLevel * 0.6 + level * 0.4;
|
||||
return micLevel;
|
||||
}
|
||||
|
||||
function readAgentAudioLevel() {
|
||||
let level = 0;
|
||||
if (remoteAnalyser && remoteFreqBuf) {
|
||||
remoteAnalyser.getByteFrequencyData(remoteFreqBuf);
|
||||
for (let i = 0; i < remoteFreqBuf.length; i++) level = Math.max(level, remoteFreqBuf[i]);
|
||||
level /= 255;
|
||||
} else if (agentParticipant && agentParticipant.audioLevel !== undefined) {
|
||||
level = agentParticipant.audioLevel || 0;
|
||||
}
|
||||
agentAudioLevel = agentAudioLevel * 0.6 + level * 0.4;
|
||||
return agentAudioLevel;
|
||||
}
|
||||
|
||||
// ── State machine ───────────────────────────────────────────────────────────
|
||||
const STATE_LABELS = {
|
||||
idle: "Idle",
|
||||
connecting: "Connecting",
|
||||
listening: "Listening",
|
||||
thinking: "Thinking",
|
||||
speaking: "Speaking",
|
||||
};
|
||||
|
||||
function setState(next) {
|
||||
if (next === uiState) return;
|
||||
uiState = next;
|
||||
orbEl.dataset.state = next;
|
||||
stateLabelEl.textContent = STATE_LABELS[next] || next;
|
||||
|
||||
// Thinking timer: show the "thinking..." chip if it lingers past 2s
|
||||
if (next === "thinking") {
|
||||
thinkingChipEl.hidden = true;
|
||||
setTimeout(() => {
|
||||
if (uiState === "thinking") thinkingChipEl.hidden = false;
|
||||
}, 2000);
|
||||
} else {
|
||||
thinkingChipEl.hidden = true;
|
||||
}
|
||||
|
||||
// Barge-in hint only while the agent is speaking
|
||||
bargeHintEl.hidden = next !== "speaking";
|
||||
|
||||
if (next === "idle") {
|
||||
micLevel = 0;
|
||||
agentAudioLevel = 0;
|
||||
userSpeaking = false;
|
||||
}
|
||||
}
|
||||
|
||||
function inferState() {
|
||||
if (!room || !room.localParticipant) return "idle";
|
||||
if (!agentParticipant) return "connecting";
|
||||
readMicLevel();
|
||||
readAgentAudioLevel();
|
||||
userSpeaking = micLevel > 0.035;
|
||||
if (agentAudioLevel > 0.02) return "speaking";
|
||||
if (userSpeaking) return "listening";
|
||||
// Not speaking, agent silent: stay in thinking until audio arrives,
|
||||
// otherwise settle back to listening.
|
||||
if (uiState === "thinking" || uiState === "speaking") return "thinking";
|
||||
return "listening";
|
||||
}
|
||||
|
||||
// ── Visual loop: drives orb + meter + state inference ───────────────────────
|
||||
let visualTimer = null;
|
||||
function startVisualLoop() {
|
||||
if (visualTimer) clearInterval(visualTimer);
|
||||
visualTimer = setInterval(() => {
|
||||
if (!room || !room.localParticipant) return;
|
||||
setState(inferState());
|
||||
|
||||
// Mic meter (thin bar under the status line)
|
||||
micMeterFillEl.style.width = `${Math.min(100, Math.round(micLevel * 250))}%`;
|
||||
|
||||
// Orb pulse: scale/opacity from whichever stream is active
|
||||
if (uiState === "speaking") {
|
||||
orbEl.style.setProperty("--pulse-scale", (1 + agentAudioLevel * 0.35).toFixed(3));
|
||||
orbEl.style.setProperty("--pulse-opacity", Math.min(1, 0.55 + agentAudioLevel).toFixed(3));
|
||||
} else if (uiState === "listening" && userSpeaking) {
|
||||
orbEl.style.setProperty("--pulse-scale", (1 + micLevel * 0.4).toFixed(3));
|
||||
orbEl.style.setProperty("--pulse-opacity", Math.min(1, 0.5 + micLevel).toFixed(3));
|
||||
} else {
|
||||
orbEl.style.setProperty("--pulse-scale", "1");
|
||||
orbEl.style.setProperty("--pulse-opacity", "");
|
||||
}
|
||||
|
||||
// Agent audio visualizer bars on the latest agent bubble
|
||||
const viz = messagesEl.querySelector(".message-row.agent:last-of-type .audio-viz");
|
||||
if (viz) {
|
||||
let level = 0;
|
||||
if (remoteAnalyser && remoteFreqBuf) {
|
||||
for (let i = 0; i < remoteFreqBuf.length; i++) level = Math.max(level, remoteFreqBuf[i]);
|
||||
level /= 255;
|
||||
}
|
||||
const bars = viz.children;
|
||||
for (let i = 0; i < bars.length; i++) {
|
||||
// Per-bar phase offset so the bars dance rather than move in lockstep
|
||||
const h = Math.max(0.15, level * (0.6 + 0.4 * Math.sin(Date.now() / 120 + i * 1.3)));
|
||||
bars[i].style.height = `${h * 100}%`;
|
||||
}
|
||||
}
|
||||
}, 80);
|
||||
}
|
||||
|
||||
function stopVisualLoop() {
|
||||
if (visualTimer) clearInterval(visualTimer);
|
||||
visualTimer = null;
|
||||
micMeterFillEl.style.width = "0%";
|
||||
orbEl.style.setProperty("--pulse-scale", "1");
|
||||
orbEl.style.setProperty("--pulse-opacity", "");
|
||||
}
|
||||
|
||||
// ── Conversation history (localStorage) ─────────────────────────────────────
|
||||
function loadHistory() {
|
||||
try {
|
||||
const raw = localStorage.getItem(HISTORY_KEY);
|
||||
if (!raw) return [];
|
||||
const arr = JSON.parse(raw);
|
||||
return Array.isArray(arr) ? arr.slice(-HISTORY_MAX) : [];
|
||||
} catch (e) { return []; }
|
||||
}
|
||||
|
||||
function persistHistory() {
|
||||
try {
|
||||
localStorage.setItem(HISTORY_KEY, JSON.stringify(history.slice(-HISTORY_MAX)));
|
||||
} catch (e) { /* storage full or unavailable */ }
|
||||
}
|
||||
|
||||
let history = loadHistory();
|
||||
|
||||
function renderMessage(msg) {
|
||||
const wrap = document.createElement("div");
|
||||
wrap.className = `message-row ${msg.role}`;
|
||||
wrap.dataset.ts = msg.ts || "";
|
||||
|
||||
if (msg.role === "agent") {
|
||||
const dot = document.createElement("span");
|
||||
dot.className = "avatar-dot";
|
||||
wrap.appendChild(dot);
|
||||
}
|
||||
|
||||
const bubble = document.createElement("div");
|
||||
bubble.className = "bubble";
|
||||
bubble.textContent = msg.text || "";
|
||||
if (msg.role === "agent") {
|
||||
const viz = document.createElement("span");
|
||||
viz.className = "audio-viz";
|
||||
for (let i = 0; i < 5; i++) viz.appendChild(document.createElement("i"));
|
||||
bubble.appendChild(viz);
|
||||
}
|
||||
wrap.appendChild(bubble);
|
||||
messagesEl.appendChild(wrap);
|
||||
messagesEl.scrollTop = messagesEl.scrollHeight;
|
||||
}
|
||||
|
||||
function addMessage(role, text) {
|
||||
const msg = { role, text, ts: Date.now() };
|
||||
history.push(msg);
|
||||
persistHistory();
|
||||
renderMessage(msg);
|
||||
return msg;
|
||||
}
|
||||
|
||||
// Partial transcripts render greyed and update in place until a final arrives.
|
||||
function addPartial(role, text) {
|
||||
const rows = messagesEl.querySelectorAll(".message-row.partial");
|
||||
let last = null;
|
||||
for (const r of rows) last = r;
|
||||
if (last && last.dataset.role === role) {
|
||||
last.querySelector(".bubble").textContent = text;
|
||||
messagesEl.scrollTop = messagesEl.scrollHeight;
|
||||
return;
|
||||
}
|
||||
const wrap = document.createElement("div");
|
||||
wrap.className = `message-row ${role} partial`;
|
||||
wrap.dataset.role = role;
|
||||
if (role === "agent") {
|
||||
const dot = document.createElement("span");
|
||||
dot.className = "avatar-dot";
|
||||
wrap.appendChild(dot);
|
||||
}
|
||||
const bubble = document.createElement("div");
|
||||
bubble.className = "bubble partial";
|
||||
bubble.textContent = text;
|
||||
wrap.appendChild(bubble);
|
||||
messagesEl.appendChild(wrap);
|
||||
messagesEl.scrollTop = messagesEl.scrollHeight;
|
||||
}
|
||||
|
||||
function clearPartials() {
|
||||
messagesEl.querySelectorAll(".message-row.partial").forEach((el) => el.remove());
|
||||
}
|
||||
|
||||
function renderHistory() {
|
||||
messagesEl.innerHTML = "";
|
||||
for (const msg of history) renderMessage(msg);
|
||||
messagesEl.scrollTop = messagesEl.scrollHeight;
|
||||
}
|
||||
|
||||
clearBtn.addEventListener("click", () => {
|
||||
history = [];
|
||||
persistHistory();
|
||||
clearPartials();
|
||||
renderHistory();
|
||||
});
|
||||
|
||||
// Restore saved voice preference
|
||||
const savedVoice = localStorage.getItem("preferred_voice");
|
||||
if (savedVoice) {
|
||||
voiceSelect.value = savedVoice;
|
||||
renderHistory();
|
||||
|
||||
// ── Transcripts ─────────────────────────────────────────────────────────────
|
||||
// Primary: livekit-agents built-in text stream (word-by-word when available).
|
||||
function registerTranscriptionStream(rm) {
|
||||
try {
|
||||
rm.registerTextStreamHandler("lk.transcription", async (reader, participantInfo) => {
|
||||
const text = await reader.readAll();
|
||||
if (!text || !text.trim()) return;
|
||||
const attrs = (reader.info && reader.info.attributes) || {};
|
||||
const trackId = attrs["lk.transcribed_track_id"] || "";
|
||||
// Distinguish user vs agent by the transcribed track's owner.
|
||||
let role = "agent";
|
||||
if (participantInfo && participantInfo.identity === AGENT_NAME) {
|
||||
role = "agent";
|
||||
} else if (trackId) {
|
||||
for (const p of rm.remoteParticipants.values()) {
|
||||
const pub = p.getTrackPublication(trackId);
|
||||
if (pub) { role = isAgent(p) ? "agent" : "user"; break; }
|
||||
}
|
||||
}
|
||||
addMessage(role, text.trim());
|
||||
});
|
||||
} catch (e) {
|
||||
// Older SDKs may not support registerTextStreamHandler — the data
|
||||
// channel handler below still covers transcripts.
|
||||
console.warn("text stream registration failed:", e);
|
||||
}
|
||||
}
|
||||
|
||||
// Secondary: agent publishes {type:"transcript"} on topic "transcript".
|
||||
function handleDataPacket(payload, participant, topic) {
|
||||
try {
|
||||
const msg = JSON.parse(new TextDecoder().decode(payload));
|
||||
if (msg.type === "transcript") {
|
||||
addMessage(msg.role || "agent", msg.text);
|
||||
}
|
||||
// set_voice messages flow the other way; nothing to do client-side.
|
||||
} catch (e) {
|
||||
console.warn("bad data packet", e);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Voice settings sheet ────────────────────────────────────────────────────
|
||||
function currentVoiceId() {
|
||||
const saved = localStorage.getItem("preferred_voice");
|
||||
return VOICES.some((v) => v.id === saved) ? saved : VOICES[0].id;
|
||||
}
|
||||
|
||||
function renderVoiceList() {
|
||||
voiceListEl.innerHTML = "";
|
||||
const active = currentVoiceId();
|
||||
for (const v of VOICES) {
|
||||
const card = document.createElement("div");
|
||||
card.className = "voice-card" + (v.id === active ? " active" : "");
|
||||
card.dataset.voice = v.id;
|
||||
|
||||
const name = document.createElement("span");
|
||||
name.className = "voice-name";
|
||||
name.textContent = v.name + (v.tag ? ` (${v.tag})` : "");
|
||||
|
||||
const previewBtn = document.createElement("button");
|
||||
previewBtn.className = "btn btn-ghost voice-preview";
|
||||
previewBtn.setAttribute("aria-label", `Preview ${v.name}`);
|
||||
previewBtn.title = "Preview (coming soon)";
|
||||
// TODO: play a short Azure TTS sample for this voice.
|
||||
previewBtn.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
addMessage("system", `Preview for ${v.name} is not wired up yet.`);
|
||||
});
|
||||
|
||||
card.appendChild(name);
|
||||
card.appendChild(previewBtn);
|
||||
card.addEventListener("click", () => selectVoice(v.id));
|
||||
voiceListEl.appendChild(card);
|
||||
}
|
||||
}
|
||||
|
||||
function selectVoice(voiceId) {
|
||||
localStorage.setItem("preferred_voice", voiceId);
|
||||
renderVoiceList();
|
||||
if (room && agentParticipant) {
|
||||
sendVoiceMessage(voiceId);
|
||||
const v = VOICES.find((x) => x.id === voiceId);
|
||||
addMessage("system", `Voice changed to ${v ? v.name : voiceId}`);
|
||||
}
|
||||
}
|
||||
|
||||
function openSheet() {
|
||||
renderVoiceList();
|
||||
sheetBackdrop.hidden = false;
|
||||
settingsSheet.classList.add("open");
|
||||
}
|
||||
|
||||
function closeSheet() {
|
||||
settingsSheet.classList.remove("open");
|
||||
sheetBackdrop.hidden = true;
|
||||
}
|
||||
|
||||
settingsBtn.addEventListener("click", openSheet);
|
||||
closeSheetBtn.addEventListener("click", closeSheet);
|
||||
sheetBackdrop.addEventListener("click", closeSheet);
|
||||
document.addEventListener("keydown", (e) => {
|
||||
if (e.key === "Escape") closeSheet();
|
||||
});
|
||||
|
||||
// ── Wake lock ───────────────────────────────────────────────────────────────
|
||||
async function acquireWakeLock() {
|
||||
try {
|
||||
if ("wakeLock" in navigator) {
|
||||
wakeLock = await navigator.wakeLock.request("screen");
|
||||
}
|
||||
} catch (e) { /* not critical */ }
|
||||
}
|
||||
|
||||
function releaseWakeLock() {
|
||||
if (wakeLock) {
|
||||
wakeLock.release().catch(() => {});
|
||||
wakeLock = null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Agent detection ─────────────────────────────────────────────────────────
|
||||
@@ -63,7 +460,7 @@ function markAgentFound(rm, participant) {
|
||||
agentParticipant = participant;
|
||||
statusEl.textContent = "Agent connected — speak now";
|
||||
// Send current voice selection to the agent
|
||||
sendVoiceMessage(voiceSelect.value, rm);
|
||||
sendVoiceMessage(currentVoiceId(), rm);
|
||||
}
|
||||
|
||||
function createRoom(token) {
|
||||
@@ -77,18 +474,49 @@ function createRoom(token) {
|
||||
if (isAgent(participant)) markAgentFound(newRoom, participant);
|
||||
});
|
||||
|
||||
// Listen for transcripts from the agent
|
||||
newRoom.on(RoomEvent.DataReceived, (payload) => {
|
||||
try {
|
||||
const msg = JSON.parse(payload.decodeToString());
|
||||
if (msg.type === "transcript") {
|
||||
addMessage(msg.role || "agent", msg.text);
|
||||
}
|
||||
} catch (e) {
|
||||
// Ignore non-JSON data
|
||||
// Attach and play incoming audio tracks (agent TTS)
|
||||
newRoom.on(RoomEvent.TrackSubscribed, (track, publication, participant) => {
|
||||
if (track.kind === LivekitClient.Track.Kind.Audio) {
|
||||
const el = track.attach();
|
||||
el.id = `audio-${participant.identity}`;
|
||||
document.body.appendChild(el);
|
||||
// Remote audio analyser drives the orb + visualizer while speaking
|
||||
setupRemoteAudioAnalyser(el);
|
||||
}
|
||||
});
|
||||
|
||||
newRoom.on(RoomEvent.TrackUnsubscribed, (track) => {
|
||||
track.detach().forEach((el) => el.remove());
|
||||
});
|
||||
|
||||
// Browsers block audio playback until a user gesture; show an unlock
|
||||
// button if the page loads without one.
|
||||
newRoom.on(RoomEvent.AudioPlaybackStatusChanged, () => {
|
||||
if (!newRoom.canPlaybackAudio) {
|
||||
const existing = document.getElementById("audioUnlockBtn");
|
||||
if (!existing) {
|
||||
const btn = document.createElement("button");
|
||||
btn.id = "audioUnlockBtn";
|
||||
btn.className = "btn btn-primary";
|
||||
btn.textContent = "Tap to enable audio";
|
||||
btn.style.cssText = "position:fixed;bottom:2rem;left:50%;transform:translateX(-50%);z-index:999;padding:1rem 2rem;font-size:1.1rem;border-radius:999px;box-shadow:0 4px 24px rgba(0,0,0,.5)";
|
||||
btn.onclick = () => { newRoom.startAudio(); btn.remove(); };
|
||||
document.body.appendChild(btn);
|
||||
}
|
||||
} else {
|
||||
const existing = document.getElementById("audioUnlockBtn");
|
||||
if (existing) existing.remove();
|
||||
}
|
||||
});
|
||||
|
||||
// Live text stream (word-by-word transcripts)
|
||||
registerTranscriptionStream(newRoom);
|
||||
|
||||
// Data channel: transcripts (secondary source) + voice control
|
||||
newRoom.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
|
||||
handleDataPacket(payload, participant, topic);
|
||||
});
|
||||
|
||||
// Guard with `=== room` so a stale (retry-discarded) connection cannot
|
||||
// clobber UI state of the active one.
|
||||
newRoom.on(RoomEvent.ConnectionQualityChanged, (participant, quality) => {
|
||||
@@ -99,7 +527,11 @@ function createRoom(token) {
|
||||
|
||||
newRoom.on(RoomEvent.Disconnected, () => {
|
||||
if (newRoom !== room) return;
|
||||
teardownAudio();
|
||||
stopVisualLoop();
|
||||
releaseWakeLock();
|
||||
statusEl.textContent = "Disconnected";
|
||||
setState("idle");
|
||||
startBtn.disabled = false;
|
||||
stopBtn.disabled = true;
|
||||
agentParticipant = null;
|
||||
@@ -135,12 +567,15 @@ function waitForAgent(timeoutMs) {
|
||||
|
||||
startBtn.addEventListener("click", async () => {
|
||||
try {
|
||||
setState("connecting");
|
||||
statusEl.textContent = "Connecting...";
|
||||
await acquireWakeLock();
|
||||
|
||||
// Get a signed access token from the token endpoint
|
||||
const token = await fetchToken(ROOM_NAME);
|
||||
|
||||
room = await createRoom(token);
|
||||
await room.startAudio();
|
||||
|
||||
// The agent is dispatched when we join. If it misses the dispatch
|
||||
// (e.g. the server just started and the worker wasn't ready yet),
|
||||
@@ -152,7 +587,8 @@ startBtn.addEventListener("click", async () => {
|
||||
room = null; // detach first so its Disconnected handler
|
||||
agentParticipant = null; // cannot clobber the UI
|
||||
await stale.disconnect();
|
||||
room = await createRoom(token);
|
||||
const freshToken = await fetchToken(ROOM_NAME);
|
||||
room = await createRoom(freshToken);
|
||||
if (!(await waitForAgent(15000))) {
|
||||
throw new Error("Assistant did not join — reload the page and try again");
|
||||
}
|
||||
@@ -165,14 +601,14 @@ startBtn.addEventListener("click", async () => {
|
||||
startBtn.disabled = true;
|
||||
stopBtn.disabled = false;
|
||||
|
||||
// Live mic level readout: if this stays at 0 while you talk, the
|
||||
// phone is not capturing audio (iOS quirk), and the problem is on
|
||||
// the device side, not the server.
|
||||
startMicMeter();
|
||||
setupMicAnalyser();
|
||||
startVisualLoop();
|
||||
|
||||
} catch (err) {
|
||||
console.error("Connection failed:", err);
|
||||
statusEl.textContent = `Error: ${err.message}`;
|
||||
setState("idle");
|
||||
releaseWakeLock();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -183,7 +619,12 @@ stopBtn.addEventListener("click", () => {
|
||||
room = null;
|
||||
agentParticipant = null;
|
||||
}
|
||||
teardownAudio();
|
||||
stopVisualLoop();
|
||||
releaseWakeLock();
|
||||
document.querySelectorAll('[id^="audio-"]').forEach(el => el.remove());
|
||||
statusEl.textContent = "Disconnected";
|
||||
setState("idle");
|
||||
startBtn.disabled = false;
|
||||
stopBtn.disabled = true;
|
||||
});
|
||||
@@ -210,45 +651,3 @@ function sendVoiceMessage(voice, targetRoom) {
|
||||
reliable: true,
|
||||
});
|
||||
}
|
||||
|
||||
// ── Helper: live mic level readout (diagnostic) ─────────────────────────────
|
||||
function startMicMeter() {
|
||||
const el = document.getElementById("micLevel");
|
||||
if (!el) return;
|
||||
let audioCtx = null;
|
||||
let analyser = null;
|
||||
try {
|
||||
const pub = room.localParticipant.getTrackPublication(LivekitClient.Track.Source.Microphone);
|
||||
const mst = pub && pub.track && pub.track.mediaStreamTrack;
|
||||
if (mst) {
|
||||
audioCtx = new AudioContext();
|
||||
analyser = audioCtx.createAnalyser();
|
||||
analyser.fftSize = 512;
|
||||
audioCtx.createMediaStreamSource(new MediaStream([mst])).connect(analyser);
|
||||
}
|
||||
} catch (e) { /* fall back to SDK audioLevel below */ }
|
||||
|
||||
const buf = analyser ? new Float32Array(analyser.fftSize) : null;
|
||||
window.micMeterTimer = setInterval(() => {
|
||||
if (!room || !room.localParticipant) { clearInterval(window.micMeterTimer); return; }
|
||||
let level = 0;
|
||||
if (analyser) {
|
||||
analyser.getFloatTimeDomainData(buf);
|
||||
for (let i = 0; i < buf.length; i++) level = Math.max(level, Math.abs(buf[i]));
|
||||
} else {
|
||||
level = room.localParticipant.audioLevel;
|
||||
}
|
||||
el.textContent = `mic: ${Math.round(level * 100)}%`;
|
||||
el.style.color = level > 0.02 ? "#4ade80" : "#94a3b8";
|
||||
}, 300);
|
||||
}
|
||||
|
||||
// ── Helper: add a message to the transcript ─────────────────────────────────
|
||||
function addMessage(role, text) {
|
||||
const div = document.createElement("div");
|
||||
div.className = `message ${role}`;
|
||||
const label = role === "user" ? "You" : role === "agent" ? "Assistant" : "System";
|
||||
div.innerHTML = `<span class="role">${label}</span> ${text}`;
|
||||
messagesEl.appendChild(div);
|
||||
messagesEl.scrollTop = messagesEl.scrollHeight;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">
|
||||
<defs>
|
||||
<radialGradient id="g" cx="35%" cy="30%" r="75%">
|
||||
<stop offset="0%" stop-color="#60a5fa"/>
|
||||
<stop offset="100%" stop-color="#a78bfa"/>
|
||||
</radialGradient>
|
||||
</defs>
|
||||
<circle cx="32" cy="32" r="28" fill="url(#g)"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 309 B |
+37
-21
@@ -2,8 +2,11 @@
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
|
||||
<meta name="theme-color" content="#09090b">
|
||||
<title>Voice Assistant</title>
|
||||
<link rel="manifest" href="manifest.json">
|
||||
<link rel="icon" type="image/svg+xml" href="favicon.svg">
|
||||
<link rel="stylesheet" href="style.css">
|
||||
</head>
|
||||
<body>
|
||||
@@ -13,38 +16,51 @@
|
||||
<p class="subtitle">Azure Speech + Gemma LLM, real-time conversation</p>
|
||||
</header>
|
||||
|
||||
<div class="orb-stage">
|
||||
<div class="orb" id="orb" data-state="idle">
|
||||
<div class="orb-core"></div>
|
||||
<div class="orb-ring"></div>
|
||||
</div>
|
||||
<div class="orb-labels">
|
||||
<span class="state-label" id="stateLabel">Idle</span>
|
||||
<span class="thinking-chip" id="thinkingChip" hidden>thinking…</span>
|
||||
<span class="barge-hint" id="bargeHint" hidden>tap or speak to interrupt</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="controls">
|
||||
<button id="startBtn" class="btn btn-primary">Start Conversation</button>
|
||||
<button id="stopBtn" class="btn btn-danger" disabled>Stop</button>
|
||||
</div>
|
||||
|
||||
<div class="voice-panel">
|
||||
<label for="voiceSelect">Voice:</label>
|
||||
<select id="voiceSelect">
|
||||
<option value="en-US-AvaNeural" selected>Ava (DragonHD)</option>
|
||||
<option value="en-US-JennyNeural">Jenny</option>
|
||||
<option value="en-US-GuyNeural">Guy</option>
|
||||
<option value="en-US-AndrewNeural">Andrew</option>
|
||||
<option value="en-US-AriaNeural">Aria</option>
|
||||
<option value="en-US-EmmaNeural">Emma</option>
|
||||
<option value="en-US-EricNeural">Eric</option>
|
||||
<option value="en-US-BrianNeural">Brian</option>
|
||||
<option value="en-US-AshleyNeural">Ashley</option>
|
||||
<option value="en-US-RichardNeural">Richard</option>
|
||||
<option value="en-US-TinaNeural">Tina</option>
|
||||
<option value="en-US-SteffanNeural">Steffan</option>
|
||||
</select>
|
||||
<button id="settingsBtn" class="btn btn-icon" aria-label="Voice settings" title="Voice settings">
|
||||
<svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"></circle><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 1 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 1 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 1 1-2.83-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 1 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 1 1 2.83-2.83l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 1 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 1 1 2.83 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 1 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"></path></svg>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="status" id="status">Disconnected</div>
|
||||
<div class="status" id="micLevel" style="font-size: 0.8em;"></div>
|
||||
<div class="mic-meter" id="micLevel" aria-hidden="true"><span class="mic-meter-fill" id="micMeterFill"></span></div>
|
||||
|
||||
<div class="transcript" id="transcript">
|
||||
<div class="transcript-header">Conversation</div>
|
||||
<div class="transcript-header">
|
||||
<span>Conversation</span>
|
||||
<button id="clearBtn" class="btn btn-ghost" title="Clear conversation">Clear</button>
|
||||
</div>
|
||||
<div id="messages"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Settings sheet (voice picker) -->
|
||||
<div class="sheet-backdrop" id="sheetBackdrop" hidden></div>
|
||||
<div class="settings-sheet" id="settingsSheet" role="dialog" aria-modal="true" aria-label="Voice settings">
|
||||
<div class="sheet-handle"></div>
|
||||
<div class="sheet-header">
|
||||
<h2>Voice</h2>
|
||||
<button id="closeSheetBtn" class="btn btn-ghost" aria-label="Close">
|
||||
<svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M18 6 6 18M6 6l12 12"></path></svg>
|
||||
</button>
|
||||
</div>
|
||||
<div class="voice-list" id="voiceList"></div>
|
||||
</div>
|
||||
|
||||
<script src="livekit-client.umd.js"></script>
|
||||
<script src="app.js"></script>
|
||||
</body>
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "Voice Assistant",
|
||||
"short_name": "Voice",
|
||||
"description": "Real-time voice assistant — Azure Speech + Gemma LLM",
|
||||
"start_url": "/",
|
||||
"display": "standalone",
|
||||
"background_color": "#09090b",
|
||||
"theme_color": "#09090b",
|
||||
"icons": [
|
||||
{
|
||||
"src": "favicon.svg",
|
||||
"sizes": "any",
|
||||
"type": "image/svg+xml",
|
||||
"purpose": "any"
|
||||
}
|
||||
]
|
||||
}
|
||||
+407
-61
@@ -1,14 +1,48 @@
|
||||
:root {
|
||||
--bg: #09090b;
|
||||
--panel: rgba(255, 255, 255, 0.05);
|
||||
--panel-strong: rgba(255, 255, 255, 0.08);
|
||||
--border: rgba(255, 255, 255, 0.09);
|
||||
--text: #e4e4e7;
|
||||
--text-dim: #a1a1aa;
|
||||
--text-faint: #71717a;
|
||||
--accent-1: #60a5fa;
|
||||
--accent-2: #a78bfa;
|
||||
--user-bubble: rgba(96, 165, 250, 0.14);
|
||||
--agent-bubble: rgba(167, 139, 250, 0.14);
|
||||
--danger: #ef4444;
|
||||
color-scheme: dark;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: light) {
|
||||
:root {
|
||||
--bg: #fafafa;
|
||||
--panel: rgba(0, 0, 0, 0.04);
|
||||
--panel-strong: rgba(0, 0, 0, 0.07);
|
||||
--border: rgba(0, 0, 0, 0.1);
|
||||
--text: #18181b;
|
||||
--text-dim: #52525b;
|
||||
--text-faint: #a1a1aa;
|
||||
--user-bubble: rgba(37, 99, 235, 0.1);
|
||||
--agent-bubble: rgba(124, 58, 237, 0.1);
|
||||
color-scheme: light;
|
||||
}
|
||||
}
|
||||
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
html, body { height: 100%; }
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: #0f1117;
|
||||
color: #e4e4e7;
|
||||
font-family: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
min-height: 100vh;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: flex-start;
|
||||
padding: 2rem 1rem;
|
||||
padding: calc(2rem + env(safe-area-inset-top)) calc(1rem + env(safe-area-inset-right))
|
||||
calc(2rem + env(safe-area-inset-bottom)) calc(1rem + env(safe-area-inset-left));
|
||||
}
|
||||
|
||||
.container {
|
||||
@@ -18,28 +52,143 @@ body {
|
||||
|
||||
header {
|
||||
text-align: center;
|
||||
margin-bottom: 2rem;
|
||||
margin-bottom: 1.5rem;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 2rem;
|
||||
font-weight: 700;
|
||||
background: linear-gradient(135deg, #60a5fa, #a78bfa);
|
||||
background: linear-gradient(135deg, var(--accent-1), var(--accent-2));
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
background-clip: text;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
color: #71717a;
|
||||
color: var(--text-faint);
|
||||
margin-top: 0.5rem;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
/* ── Orb ─────────────────────────────────────────────────────────────────── */
|
||||
.orb-stage {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
margin-bottom: 1.75rem;
|
||||
}
|
||||
|
||||
.orb {
|
||||
--pulse-scale: 1;
|
||||
position: relative;
|
||||
width: 200px;
|
||||
height: 200px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
}
|
||||
|
||||
.orb-core {
|
||||
width: 130px;
|
||||
height: 130px;
|
||||
border-radius: 50%;
|
||||
background: radial-gradient(circle at 35% 30%, var(--accent-1), var(--accent-2) 70%);
|
||||
box-shadow:
|
||||
0 0 60px 8px color-mix(in srgb, var(--accent-2) 45%, transparent),
|
||||
inset 0 0 30px rgba(255, 255, 255, 0.15);
|
||||
transform: scale(var(--pulse-scale));
|
||||
transition: box-shadow 0.3s ease;
|
||||
animation: breathe 4s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.orb-ring {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
border-radius: 50%;
|
||||
border: 1px solid color-mix(in srgb, var(--accent-2) 35%, transparent);
|
||||
opacity: var(--pulse-opacity, 0.35);
|
||||
transform: scale(var(--pulse-scale));
|
||||
transition: opacity 0.2s ease;
|
||||
}
|
||||
|
||||
/* Per-state glow */
|
||||
.orb[data-state="idle"] .orb-core {
|
||||
filter: saturate(0.5) brightness(0.75);
|
||||
box-shadow: 0 0 30px 4px color-mix(in srgb, var(--accent-2) 25%, transparent);
|
||||
}
|
||||
|
||||
.orb[data-state="connecting"] .orb-core {
|
||||
animation-duration: 1.6s;
|
||||
filter: saturate(0.8) brightness(0.9);
|
||||
}
|
||||
|
||||
.orb[data-state="listening"] .orb-core {
|
||||
box-shadow: 0 0 70px 12px color-mix(in srgb, var(--accent-1) 55%, transparent),
|
||||
inset 0 0 30px rgba(255, 255, 255, 0.18);
|
||||
}
|
||||
|
||||
.orb[data-state="thinking"] .orb-core {
|
||||
animation-duration: 2s;
|
||||
filter: saturate(1.1) brightness(1.05);
|
||||
box-shadow: 0 0 75px 14px color-mix(in srgb, var(--accent-2) 60%, transparent),
|
||||
inset 0 0 30px rgba(255, 255, 255, 0.2);
|
||||
}
|
||||
|
||||
.orb[data-state="speaking"] .orb-core {
|
||||
animation: none;
|
||||
box-shadow: 0 0 90px 18px color-mix(in srgb, var(--accent-1) 65%, transparent),
|
||||
inset 0 0 34px rgba(255, 255, 255, 0.22);
|
||||
}
|
||||
|
||||
@keyframes breathe {
|
||||
0%, 100% { transform: scale(calc(var(--pulse-scale) * 1)); }
|
||||
50% { transform: scale(calc(var(--pulse-scale) * 1.05)); }
|
||||
}
|
||||
|
||||
.orb-labels {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
gap: 0.4rem;
|
||||
margin-top: 1rem;
|
||||
min-height: 3.6rem;
|
||||
}
|
||||
|
||||
.state-label {
|
||||
font-size: 0.85rem;
|
||||
font-weight: 600;
|
||||
letter-spacing: 0.08em;
|
||||
text-transform: uppercase;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.thinking-chip {
|
||||
font-size: 0.75rem;
|
||||
padding: 0.25rem 0.75rem;
|
||||
border-radius: 999px;
|
||||
background: var(--panel-strong);
|
||||
border: 1px solid var(--border);
|
||||
color: var(--text-dim);
|
||||
animation: chipPulse 1.4s ease-in-out infinite;
|
||||
}
|
||||
|
||||
@keyframes chipPulse {
|
||||
0%, 100% { opacity: 0.55; }
|
||||
50% { opacity: 1; }
|
||||
}
|
||||
|
||||
.barge-hint {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-faint);
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
/* ── Controls ────────────────────────────────────────────────────────────── */
|
||||
.controls {
|
||||
display: flex;
|
||||
gap: 1rem;
|
||||
gap: 0.75rem;
|
||||
justify-content: center;
|
||||
margin-bottom: 1.5rem;
|
||||
align-items: center;
|
||||
margin-bottom: 1.25rem;
|
||||
}
|
||||
|
||||
.btn {
|
||||
@@ -50,6 +199,7 @@ h1 {
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s;
|
||||
font-family: inherit;
|
||||
}
|
||||
|
||||
.btn:disabled {
|
||||
@@ -68,7 +218,7 @@ h1 {
|
||||
}
|
||||
|
||||
.btn-danger {
|
||||
background: #ef4444;
|
||||
background: var(--danger);
|
||||
color: white;
|
||||
}
|
||||
|
||||
@@ -76,98 +226,294 @@ h1 {
|
||||
background: #dc2626;
|
||||
}
|
||||
|
||||
.voice-panel {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.75rem;
|
||||
justify-content: center;
|
||||
margin-bottom: 1.5rem;
|
||||
padding: 1rem;
|
||||
background: #1c1e26;
|
||||
border-radius: 8px;
|
||||
.btn-icon {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 44px;
|
||||
height: 44px;
|
||||
padding: 0;
|
||||
border-radius: 50%;
|
||||
background: var(--panel);
|
||||
color: var(--text-dim);
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.voice-panel label {
|
||||
font-weight: 600;
|
||||
color: #a1a1aa;
|
||||
.btn-icon:hover {
|
||||
color: var(--text);
|
||||
background: var(--panel-strong);
|
||||
}
|
||||
|
||||
.voice-panel select {
|
||||
padding: 0.5rem 1rem;
|
||||
background: #27272a;
|
||||
color: #e4e4e7;
|
||||
border: 1px solid #3f3f46;
|
||||
.btn-ghost {
|
||||
background: transparent;
|
||||
color: var(--text-faint);
|
||||
padding: 0.4rem 0.75rem;
|
||||
font-size: 0.8rem;
|
||||
border-radius: 6px;
|
||||
font-size: 0.9rem;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.voice-panel select:focus {
|
||||
outline: none;
|
||||
border-color: #3b82f6;
|
||||
.btn-ghost:hover {
|
||||
color: var(--text);
|
||||
background: var(--panel-strong);
|
||||
}
|
||||
|
||||
/* ── Status + mic meter ──────────────────────────────────────────────────── */
|
||||
.status {
|
||||
text-align: center;
|
||||
padding: 0.75rem;
|
||||
background: #1c1e26;
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
margin-bottom: 1.5rem;
|
||||
margin-bottom: 0.6rem;
|
||||
font-size: 0.9rem;
|
||||
color: #a1a1aa;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.mic-meter {
|
||||
height: 4px;
|
||||
background: var(--panel-strong);
|
||||
border-radius: 2px;
|
||||
margin-bottom: 1.5rem;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.mic-meter-fill {
|
||||
display: block;
|
||||
height: 100%;
|
||||
width: 0%;
|
||||
background: linear-gradient(90deg, var(--accent-1), var(--accent-2));
|
||||
border-radius: 2px;
|
||||
transition: width 0.1s linear;
|
||||
}
|
||||
|
||||
/* ── Transcript ──────────────────────────────────────────────────────────── */
|
||||
.transcript {
|
||||
background: #1c1e26;
|
||||
border-radius: 8px;
|
||||
background: var(--panel);
|
||||
backdrop-filter: blur(12px);
|
||||
-webkit-backdrop-filter: blur(12px);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 12px;
|
||||
padding: 1rem;
|
||||
max-height: 50vh;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.transcript-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
font-weight: 700;
|
||||
color: #a1a1aa;
|
||||
color: var(--text-dim);
|
||||
margin-bottom: 1rem;
|
||||
font-size: 0.85rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
}
|
||||
|
||||
.message {
|
||||
padding: 0.6rem 0.8rem;
|
||||
margin-bottom: 0.5rem;
|
||||
border-radius: 6px;
|
||||
.message-row {
|
||||
display: flex;
|
||||
align-items: flex-end;
|
||||
gap: 0.5rem;
|
||||
margin-bottom: 0.6rem;
|
||||
position: relative;
|
||||
animation: msgIn 120ms ease-out;
|
||||
}
|
||||
|
||||
@keyframes msgIn {
|
||||
from { opacity: 0; transform: translateY(6px); }
|
||||
to { opacity: 1; transform: translateY(0); }
|
||||
}
|
||||
|
||||
.message-row.user {
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.avatar-dot {
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
border-radius: 50%;
|
||||
background: linear-gradient(135deg, var(--accent-1), var(--accent-2));
|
||||
flex-shrink: 0;
|
||||
margin-bottom: 2px;
|
||||
}
|
||||
|
||||
.bubble {
|
||||
max-width: 78%;
|
||||
padding: 0.6rem 0.9rem;
|
||||
border-radius: 14px;
|
||||
font-size: 0.95rem;
|
||||
line-height: 1.5;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.message.user {
|
||||
background: #1e3a5f;
|
||||
border-left: 3px solid #3b82f6;
|
||||
.message-row.user .bubble {
|
||||
background: var(--user-bubble);
|
||||
border-bottom-right-radius: 4px;
|
||||
}
|
||||
|
||||
.message.agent {
|
||||
background: #2d1f4e;
|
||||
border-left: 3px solid #a78bfa;
|
||||
.message-row.agent .bubble {
|
||||
background: var(--agent-bubble);
|
||||
border-bottom-left-radius: 4px;
|
||||
}
|
||||
|
||||
.message.system {
|
||||
background: #27272a;
|
||||
color: #71717a;
|
||||
font-style: italic;
|
||||
font-size: 0.85rem;
|
||||
.bubble.partial {
|
||||
opacity: 0.55;
|
||||
}
|
||||
|
||||
.role {
|
||||
/* Timestamps on hover (CSS only, from data-ts attribute) */
|
||||
.message-row::after {
|
||||
content: attr(data-ts);
|
||||
position: absolute;
|
||||
bottom: -1.1rem;
|
||||
font-size: 0.68rem;
|
||||
color: var(--text-faint);
|
||||
opacity: 0;
|
||||
transition: opacity 0.15s ease;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
.message-row:hover::after {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
.message-row.user::after {
|
||||
right: 0;
|
||||
}
|
||||
|
||||
.message-row.agent::after {
|
||||
left: 2rem;
|
||||
}
|
||||
|
||||
/* ── Agent audio visualizer (bars on latest agent bubble) ─────────────────── */
|
||||
.audio-viz {
|
||||
display: inline-flex;
|
||||
align-items: flex-end;
|
||||
gap: 2px;
|
||||
height: 14px;
|
||||
margin-left: 0.6rem;
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.audio-viz i {
|
||||
width: 3px;
|
||||
height: 15%;
|
||||
border-radius: 2px;
|
||||
background: linear-gradient(180deg, var(--accent-2), var(--accent-1));
|
||||
opacity: 0.8;
|
||||
transition: height 0.08s linear;
|
||||
}
|
||||
|
||||
/* ── Settings sheet (voice picker) ───────────────────────────────────────── */
|
||||
.sheet-backdrop {
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
background: rgba(0, 0, 0, 0.5);
|
||||
z-index: 90;
|
||||
opacity: 0;
|
||||
transition: opacity 0.2s ease;
|
||||
}
|
||||
|
||||
.sheet-backdrop:not([hidden]) {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
.settings-sheet {
|
||||
position: fixed;
|
||||
left: 50%;
|
||||
bottom: 0;
|
||||
transform: translate(-50%, 100%);
|
||||
width: 100%;
|
||||
max-width: 680px;
|
||||
max-height: 75vh;
|
||||
background: var(--panel-strong);
|
||||
backdrop-filter: blur(12px);
|
||||
-webkit-backdrop-filter: blur(12px);
|
||||
border: 1px solid var(--border);
|
||||
border-bottom: none;
|
||||
border-radius: 16px 16px 0 0;
|
||||
padding: 0.75rem 1.25rem calc(1.25rem + env(safe-area-inset-bottom));
|
||||
z-index: 100;
|
||||
transition: transform 0.25s cubic-bezier(0.32, 0.72, 0, 1);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.settings-sheet.open {
|
||||
transform: translate(-50%, 0);
|
||||
}
|
||||
|
||||
.sheet-handle {
|
||||
width: 36px;
|
||||
height: 4px;
|
||||
border-radius: 2px;
|
||||
background: var(--border);
|
||||
margin: 0 auto 0.75rem;
|
||||
}
|
||||
|
||||
.sheet-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
.sheet-header h2 {
|
||||
font-size: 1.1rem;
|
||||
font-weight: 700;
|
||||
margin-right: 0.5rem;
|
||||
}
|
||||
|
||||
.message.user .role { color: #60a5fa; }
|
||||
.message.agent .role { color: #a78bfa; }
|
||||
.voice-list {
|
||||
overflow-y: auto;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.5rem;
|
||||
padding-right: 0.25rem;
|
||||
}
|
||||
|
||||
/* Scrollbar */
|
||||
.transcript::-webkit-scrollbar { width: 6px; }
|
||||
.transcript::-webkit-scrollbar-track { background: transparent; }
|
||||
.transcript::-webkit-scrollbar-thumb { background: #3f3f46; border-radius: 3px; }
|
||||
.voice-card {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 0.75rem;
|
||||
padding: 0.8rem 1rem;
|
||||
border-radius: 10px;
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
cursor: pointer;
|
||||
transition: all 0.15s ease;
|
||||
}
|
||||
|
||||
.voice-card:hover {
|
||||
background: var(--panel-strong);
|
||||
}
|
||||
|
||||
.voice-card.active {
|
||||
border-color: color-mix(in srgb, var(--accent-2) 60%, transparent);
|
||||
background: var(--agent-bubble);
|
||||
}
|
||||
|
||||
.voice-name {
|
||||
font-size: 0.95rem;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
/* ── Scrollbar ───────────────────────────────────────────────────────────── */
|
||||
.transcript::-webkit-scrollbar,
|
||||
.voice-list::-webkit-scrollbar { width: 6px; }
|
||||
.transcript::-webkit-scrollbar-track,
|
||||
.voice-list::-webkit-scrollbar-track { background: transparent; }
|
||||
.transcript::-webkit-scrollbar-thumb,
|
||||
.voice-list::-webkit-scrollbar-thumb { background: var(--border); border-radius: 3px; }
|
||||
|
||||
/* ── Reduced motion ──────────────────────────────────────────────────────── */
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.orb-core,
|
||||
.thinking-chip,
|
||||
.message-row,
|
||||
.settings-sheet,
|
||||
.sheet-backdrop,
|
||||
.audio-viz i,
|
||||
.mic-meter-fill {
|
||||
animation: none !important;
|
||||
transition: none !important;
|
||||
}
|
||||
}
|
||||
|
||||
+1
-9
@@ -66,25 +66,17 @@ class Handler(BaseHTTPRequestHandler):
|
||||
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
body = json.loads(self.rfile.read(length) or b"{}")
|
||||
room_name = body.get("room", "voice-room")
|
||||
room_name = "voice-room"
|
||||
identity = body.get("identity") or f"user-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
token = make_token(room_name, identity)
|
||||
resp = json.dumps({"token": token}).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Access-Control-Allow-Origin", "*")
|
||||
self.send_header("Content-Length", str(len(resp)))
|
||||
self.end_headers()
|
||||
self.wfile.write(resp)
|
||||
|
||||
def do_OPTIONS(self):
|
||||
self.send_response(204)
|
||||
self.send_header("Access-Control-Allow-Origin", "*")
|
||||
self.send_header("Access-Control-Allow-Methods", "POST, OPTIONS")
|
||||
self.send_header("Access-Control-Allow-Headers", "Content-Type")
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass # silence request logging
|
||||
|
||||
|
||||
Reference in New Issue
Block a user