Voice — Real-time Voice Assistant

A single-container voice assistant built on LiveKit Agents. Speaks and listens in real time using Azure Speech (STT + TTS) with a local Gemma LLM for reasoning.

Architecture

One Docker container runs three processes via supervisord:

LiveKit server — open-source WebRTC media transport (port 7880 TCP, 50000-60000 UDP)
Voice agent — Python LiveKit Agents pipeline: Azure STT → Gemma LLM → Azure TTS
Web frontend — static HTML served by a tiny HTTP server (port 8090)

Browser ──WebRTC──► LiveKit Server ──audio──► Agent
   ▲                                              │
   └────────── audio ◄────────────────────────────┘
                        Azure STT → Gemma LLM → Azure TTS

Quick Start

# 1. Set your Azure Speech key (already in ~/.hermes/.env as AZURE_SPEECH)
export AZURE_SPEECH_KEY=$(grep '^AZURE_SPEECH=' ~/.hermes/.env | cut -d= -f2)

# 2. Build and start
docker compose up --build -d

# 3. Open the web UI
# http://<host-ip>:8090

Ports

Port	Protocol	Service	Access
7880	TCP	LiveKit HTTP/WS	LAN
7881	TCP	LiveKit internal	container
50000-60000	UDP	LiveKit RTC media	LAN
8090	TCP	Web frontend	LAN

Configuration

All config lives in .env (git-ignored). See .env.example for the full list.

Key variables:

AZURE_SPEECH_KEY — Azure Speech resource key
AZURE_SPEECH_REGION — default eastus
AZURE_TTS_VOICE — default voice (e.g. en-US-AvaNeural)
GEMMA_BASE_URL — LLM endpoint (default http://192.168.86.2:8023/v1)
GEMMA_MODEL — model name (default gemma-4-e4b)

Voice Selection

The web UI includes a voice picker that sends the selected voice to the agent via LiveKit data channel. The agent updates its TTS voice in real time without restarting.

Supported voices: any Azure Neural voice. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=en-us#neural-voices

SSML / Expressive Speech

The TTS layer uses full SSML with <mstts:express-as> for style and <prosody> for rate/volume/pitch. The agent's system prompt instructs the LLM to write in a conversational, spoken style (short sentences, natural phrasing) so the output sounds like speech, not text.

Agent Prompt

The LLM is instructed to:

Speak as if talking to someone, not writing for them to read
Keep responses to 1-3 sentences
Use contractions, natural fillers sparingly
Never use markdown, lists, or formatting
Spell out numbers and abbreviations

Testing

# Verify the container is running
docker compose ps

# Check agent logs
docker compose logs -f agent

# Test TTS directly (outside the agent)
curl -s "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1" \
  -H "Ocp-Apim-Subscription-Key: $AZURE_SPEECH_KEY" \
  -H "Content-Type: application/ssml+xml" \
  -H "X-Microsoft-OutputFormat: audio-24khz-48kbitrate-mono-mp3" \
  --data-binary '<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US"><voice name="en-US-AvaNeural">Test</voice></speak>' > /tmp/test.mp3

File Layout

~/dev/voice/
├── AGENTS.md            ← you are here
├── .env.example         ← config template (copy to .env)
├── .gitignore
├── docker-compose.yml   ← single container, 3 processes
├── Dockerfile           ← multi-stage build
├── livekit.yaml         ← LiveKit server config
├── supervisord.conf     ← process manager for the 3 services
├── agent/
│   ├── agent.py         ← LiveKit Agents voice pipeline
│   └── pyproject.toml   ← Python deps (uv)
├── web/
│   ├── index.html       ← single-page voice UI
│   ├── app.js           ← LiveKit client logic
│   └── style.css        ← minimal dark theme
└── docs/
    └── ARCHITECTURE.md  ← deeper architecture notes

Conventions

Single container. All services (LiveKit, agent, web) run in one Docker container via supervisord. No multi-service compose.
No published UDP ports in compose. LiveKit binds its UDP range directly on the host network (network_mode: host). This avoids the docker-proxy process explosion that hit hope-webui.
Gemma is a reasoning model. It sometimes spends tokens on hidden reasoning before producing content. The agent handles this by using max_tokens=1000 and falling back to reasoning_content if content is empty.
Azure TTS uses SSML, not JSON. The REST endpoint requires Content-Type: application/ssml+xml. The LiveKit Azure plugin handles this internally.
Voice changes are live. The web UI sends a data message to the agent; the agent calls session.update_options(voice=...) without restarting.

Git

Commit frequently. Conventional commits (feat:, fix:, chore:).

5.0 KiB Raw Blame History