pi-web/docs/voice-api.md

Voice Conversation API (v1)

/api/v1/voice is a server-side device API for native iOS and Echo-style clients. It is separate from the browser voice feature. Azure credentials remain on the PI WEB host. Every voice turn automatically includes server-side instructions to produce concise, natural DragonHD-friendly speech without Markdown or visual-only formatting.

Security and provisioning

Create a device token on the PI WEB host:

pi-web voice-token create \
  --project project-id --workspace workspace-id \
  --model openai-codex/gpt-5.6-terra --thinking high

Each restriction flag may be repeated. Omitting a category grants * for that category. A restricted model or thinking scope requires the client to explicitly select an allowed value; it cannot inherit a server default. Project and workspace scopes are both enforced. Token records contain a SHA-256 token hash only and are persisted at mode 0600 in ~/.config/pi-web/voice-api.json; the plaintext (pwv1_...) is printed only once.

pi-web voice-token list
pi-web voice-token revoke <token-id>

Use Authorization: Bearer pwv1_... on every HTTP and WebSocket connection. The API requires HTTPS, except loopback for local development/testing. A conversation is owned by the creating token: other valid device tokens receive 404 when they try to inspect, delete, or attach it. A client may select only a registered workspace id, never a filesystem path.

HTTP

GET /api/v1/voice/targets returns only the caller's authorized registered workspaces plus effective scopes. Use its workspaces[].id to choose a workspace; paths are returned for display but are never accepted as an API input.

GET /api/v1/voice/conversations?workspaceId=<workspace-id> lists prior Pi sessions rooted at that authorized workspace. This lets a device show a conversation picker without learning about sessions outside its workspace scope.

POST /api/v1/voice/conversations

{
  "workspaceId": "registered-workspace-id",
  "model": "openai-codex/gpt-5.6-terra",
  "thinking": "high",
  "context": "You are helping with the home-automation project."
}

For an existing directory beneath the device-safe root /home/hope/workspaces that is not yet registered as a PI WEB workspace, use POST /api/v1/voice/conversations/path with path instead of workspaceId. The path must already exist, resolve beneath that root, and is validated before a session starts.

The response is 201 and includes the opaque conversation id, session id, selected workspace, and input-ready status. GET /api/v1/voice/conversations/:id/models lists the models currently available to that conversation's Pi session, filtered by the device token's model scope. GET and DELETE /api/v1/voice/conversations/:id inspect and close the API conversation handle.

To resume a listed session, create a new token-owned voice handle:

POST /api/v1/voice/conversations/resume
{
  "workspaceId": "registered-workspace-id",
  "sessionId": "previous-pi-session-id"
}

The session must be in the selected workspace and within the caller's token scope.

WebSocket wire protocol

Connect to wss://host/api/v1/voice/stream with the Bearer header. The server first sends:

{
  "type":"hello",
  "protocol":"pi-web.voice.v1",
  "pcm":{"input":"s16le/16000/mono","output":"s16le/24000/mono"},
  "binary":{"version":1,"headerBytes":8,"kind":"audio"}
}

Attach an owned conversation and wait for input.ready before sending a turn:

{"type":"attach","conversationId":"..."}

Text fallback

Send exactly one JSON input while input-ready:

{"type":"input.text","text":"Turn on the kitchen lights."}

Audio input

For speech input, send binary WebSocket frames containing raw signed little-endian 16-bit, 16 kHz, mono PCM (no WAV header). Frames must be nonempty, <=64 KiB, aligned to two bytes, and an utterance is limited to two minutes. Finish with:

{"type":"input.end"}

Azure push-stream recognition emits transcript.partial and transcript.final JSON events. The final transcript is submitted to Pi. This Voice API never returns an Azure token or key to a device.

Turn output and lifecycle

For either input form, the server emits agent.working, then agent.accepted after Pi accepted the prompt. If work is still in progress after two seconds, it sends one agent.progress event with a short spoken status clip (for example, “Hang on while I work on that.”). It never repeats this spoken update within the same turn; ordinary screen-only agent.working heartbeats continue every 15 seconds. While it waits for Pi's native agent.settled event (not merely agent.end), it repeats the existing agent.working frame every 15 seconds; those progress frames stop before assistant.final and never overlap synthesized audio. The wait has a two-minute turn timeout. assistant.delta is streamed while Pi answers; assistant.final contains only assistant text, never STT transcript.

Azure synthesis output is signed little-endian 16-bit, 24 kHz, mono PCM. Every binary server frame has this eight-byte header followed by PCM:

Bytes	Meaning
0	protocol version: 1
1	kind: 1 (audio)
2–3	reserved: 0 (big endian)
4–7	monotonically increasing frame sequence (uint32 big endian)

The server then emits {"type":"audio.end"} and a fresh input.ready. Send {"type":"close"} or close the WebSocket to end; disconnecting closes STT/event resources and clears the handle's working state.