From f7ff1c61ac3e7a74efd10db88b15099326755aaa Mon Sep 17 00:00:00 2001 From: Shane Date: Sat, 22 Aug 2026 07:39:54 -0400 Subject: [PATCH] chore: init project with AGENTS.md, .env.example, .gitignore --- .env.example | 19 ++++++++ .gitignore | 7 +++ AGENTS.md | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+) create mode 100644 .env.example create mode 100644 .gitignore create mode 100644 AGENTS.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..dcdb412 --- /dev/null +++ b/.env.example @@ -0,0 +1,19 @@ +# Azure Speech (required) +AZURE_SPEECH_KEY=your-azure-speech-key +AZURE_SPEECH_REGION=eastus + +# Default TTS voice (any Azure Neural voice) +AZURE_TTS_VOICE=en-US-AvaNeural + +# LLM (Gemma on xNAS via llama.cpp, OpenAI-compatible) +GEMMA_BASE_URL=http://192.168.86.2:8023/v1 +GEMMA_MODEL=gemma-4-e4b +GEMMA_API_KEY=not-needed + +# LiveKit (generated in livekit.yaml, referenced here for the agent) +LIVEKIT_URL=http://localhost:7880 +LIVEKIT_API_KEY=devkey +LIVEKIT_API_SECRET=devsecret + +# Web UI port +WEB_PORT=8090 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..18e30b5 --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +.env +__pycache__/ +*.pyc +.venv/ +uv.lock +node_modules/ +*.log diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..b3a3fac --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,121 @@ +# Voice — Real-time Voice Assistant + +A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with a local Gemma LLM for reasoning. + +## Architecture + +One Docker container runs three processes via supervisord: + +1. **LiveKit server** — open-source WebRTC media transport (port 7880 TCP, 50000-60000 UDP) +2. **Voice agent** — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS +3. **Web frontend** — static HTML served by a tiny HTTP server (port 8090) + +``` +Browser ──WebRTC──► LiveKit Server ──audio──► Agent + ▲ │ + └────────── audio ◄────────────────────────────┘ + Azure STT → Gemma LLM → Azure TTS +``` + +## Quick Start + +```bash +# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH) +export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2) + +# 2. Build and start +docker compose up --build -d + +# 3. Open the web UI +# http://:8090 +``` + +## Ports + +| Port | Protocol | Service | Access | +|-------|----------|----------------------|--------------| +| 7880 | TCP | LiveKit HTTP/WS | LAN | +| 7881 | TCP | LiveKit internal | container | +| 50000-60000 | UDP | LiveKit RTC media | LAN | +| 8090 | TCP | Web frontend | LAN | + +## Configuration + +All config lives in `.env` (git-ignored). See `.env.example` for the full list. + +Key variables: +- `AZURE_SPEECH_KEY` — Azure Speech resource key +- `AZURE_SPEECH_REGION` — default `eastus` +- `AZURE_TTS_VOICE` — default voice (e.g. `en-US-AvaNeural`) +- `GEMMA_BASE_URL` — LLM endpoint (default `http://192.168.86.2:8023/v1`) +- `GEMMA_MODEL` — model name (default `gemma-4-e4b`) + +## Voice Selection + +The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting. + +Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices + +## SSML / Expressive Speech + +The TTS layer uses full SSML with `` for style and `` for rate/volume/pitch. The agent's system prompt instructs the LLM to write in a conversational, spoken style (short sentences, natural phrasing) so the output sounds like speech, not text. + +## Agent Prompt + +The LLM is instructed to: +- Speak as if talking to someone, not writing for them to read +- Keep responses to 1-3 sentences +- Use contractions, natural fillers sparingly +- Never use markdown, lists, or formatting +- Spell out numbers and abbreviations + +## Testing + +```bash +# Verify the container is running +docker compose ps + +# Check agent logs +docker compose logs -f agent + +# Test TTS directly (outside the agent) +curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \ + -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \ + -H "Content-Type: application/ssml+xml" \ + -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \ + --data-binary 'Test' > /tmp/test.mp3 +``` + +## File Layout + +``` +~/dev/voice/ +├── AGENTS.md ← you are here +├── .env.example ← config template (copy to .env) +├── .gitignore +├── docker-compose.yml ← single container, 3 processes +├── Dockerfile ← multi-stage build +├── livekit.yaml ← LiveKit server config +├── supervisord.conf ← process manager for the 3 services +├── agent/ +│ ├── agent.py ← LiveKit Agents voice pipeline +│ └── pyproject.toml ← Python deps (uv) +├── web/ +│ ├── index.html ← single-page voice UI +│ ├── app.js ← LiveKit client logic +│ └── style.css ← minimal dark theme +└── docs/ + └── ARCHITECTURE.md ← deeper architecture notes +``` + +## Conventions + +- **Single container.** All services (LiveKit, agent, web) run in one Docker container via supervisord. No multi-service compose. +- **No published UDP ports in compose.** LiveKit binds its UDP range directly on the host network (`network_mode: host`). This avoids the docker-proxy process explosion that hit hope-webui. +- **Gemma is a reasoning model.** It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using `max_tokens=1000` and falling back to `reasoning_content` if `content` is empty. +- **Azure TTS uses SSML, not JSON.** The REST endpoint requires `Content-Type: application/ssml+xml`. The LiveKit Azure plugin handles this internally. +- **Voice changes are live.** The web UI sends a data message to the agent; the agent calls `session.update_options(voice=...)` without restarting. + +## Git + +Commit frequently. Conventional commits (`feat:`, `fix:`, `chore:`).