Archived
snowspeederandClaude Fable 5
28a2629c43
Segment the final answer as it streams and synthesize each completed sentence immediately, so a listener hears the reply begin while the model is still writing it. Sentences share the progress clips' FIFO chain and output-sequence counter; only text left unspoken (a hit budget) is synthesized after settle, so audio is never sent twice. A message that grows a tool call stops answer-streaming without re-speaking what already played. Reasoning models inline their chain of thought as <think> blocks in the same delta stream. Strip it while streaming -- carrying a partial tag across delta boundaries -- so it is neither spoken nor shown as answer text. The fixed "Hang on while I work on that." is now a rotation of short natural phrases that never repeats twice running, and a long silent think gets a spoken "still on it" roughly every 45 seconds until real speech begins. Turn timeout rises to five minutes to match the client. Co-Authored-By: Claude Fable 5 <[email protected]>
114 lines
9.2 KiB
Markdown
114 lines
9.2 KiB
Markdown
# Voice Conversation API (v1)
|
||
|
||
`/api/v1/voice` is a server-side device API for native iOS and Echo-style clients. It is separate from the browser voice feature. Azure credentials remain on the PI WEB host. Every voice turn automatically includes server-side instructions to produce concise, natural DragonHD-friendly speech without Markdown or visual-only formatting.
|
||
|
||
## Security and provisioning
|
||
|
||
Create a device token on the PI WEB host:
|
||
|
||
```sh
|
||
pi-web voice-token create \
|
||
--project project-id --workspace workspace-id \
|
||
--model openai-codex/gpt-5.6-terra --thinking high
|
||
```
|
||
|
||
Each restriction flag may be repeated. Omitting a category grants `*` for that category. A restricted model or thinking scope requires the client to explicitly select an allowed value; it cannot inherit a server default. Project and workspace scopes are both enforced. Token records contain a SHA-256 token hash only and are persisted at mode `0600` in `~/.config/pi-web/voice-api.json`; the plaintext (`pwv1_...`) is printed only once.
|
||
|
||
```sh
|
||
pi-web voice-token list
|
||
pi-web voice-token revoke <token-id>
|
||
```
|
||
|
||
Use `Authorization: Bearer pwv1_...` on every HTTP and WebSocket connection. The API requires HTTPS, except loopback for local development/testing. A conversation is owned by the creating token: other valid device tokens receive `404` when they try to inspect, delete, or attach it. A client may select only a registered workspace id, never a filesystem path.
|
||
|
||
## HTTP
|
||
|
||
`GET /api/v1/voice/targets` returns only the caller's authorized registered workspaces plus effective scopes. Use its `workspaces[].id` to choose a workspace; paths are returned for display but are never accepted as an API input.
|
||
|
||
`GET /api/v1/voice/conversations?workspaceId=<workspace-id>` lists prior Pi sessions rooted at that authorized workspace. This lets a device show a conversation picker without learning about sessions outside its workspace scope.
|
||
|
||
`POST /api/v1/voice/conversations`
|
||
|
||
```json
|
||
{
|
||
"workspaceId": "registered-workspace-id",
|
||
"model": "openai-codex/gpt-5.6-terra",
|
||
"thinking": "high",
|
||
"context": "You are helping with the home-automation project."
|
||
}
|
||
```
|
||
|
||
For an existing directory beneath the device-safe root `/home/hope/workspaces` that is not yet registered as a PI WEB workspace, use `POST /api/v1/voice/conversations/path` with `path` instead of `workspaceId`. The path must already exist, resolve beneath that root, and is validated before a session starts.
|
||
|
||
The response is `201` and includes the opaque conversation id, session id, selected workspace, and `input-ready` status. `GET /api/v1/voice/conversations/:id/models` lists the models currently available to that conversation's Pi session, filtered by the device token's model scope. `GET` and `DELETE /api/v1/voice/conversations/:id` inspect and close the API conversation handle.
|
||
|
||
To resume a listed session, create a new token-owned voice handle:
|
||
|
||
```json
|
||
POST /api/v1/voice/conversations/resume
|
||
{
|
||
"workspaceId": "registered-workspace-id",
|
||
"sessionId": "previous-pi-session-id"
|
||
}
|
||
```
|
||
|
||
The session must be in the selected workspace and within the caller's token scope.
|
||
|
||
## WebSocket wire protocol
|
||
|
||
Connect to `wss://host/api/v1/voice/stream` with the Bearer header. The server first sends:
|
||
|
||
```json
|
||
{
|
||
"type":"hello",
|
||
"protocol":"pi-web.voice.v1",
|
||
"pcm":{"input":"s16le/16000/mono","output":"s16le/24000/mono"},
|
||
"binary":{"version":1,"headerBytes":8,"kind":"audio"}
|
||
}
|
||
```
|
||
|
||
Attach an owned conversation and wait for `input.ready` before sending a turn:
|
||
|
||
```json
|
||
{"type":"attach","conversationId":"..."}
|
||
```
|
||
|
||
### Text fallback
|
||
|
||
Send exactly one JSON input while `input-ready`:
|
||
|
||
```json
|
||
{"type":"input.text","text":"Turn on the kitchen lights."}
|
||
```
|
||
|
||
### Audio input
|
||
|
||
For speech input, send binary WebSocket frames containing raw signed little-endian 16-bit, 16 kHz, mono PCM (no WAV header). Frames must be nonempty, <=64 KiB, aligned to two bytes, and an utterance is limited to two minutes. Finish with:
|
||
|
||
```json
|
||
{"type":"input.end"}
|
||
```
|
||
|
||
Azure push-stream recognition emits `transcript.partial` and `transcript.final` JSON events. The final transcript is submitted to Pi. This Voice API never returns an Azure token or key to a device.
|
||
|
||
### Turn output and lifecycle
|
||
|
||
For either input form, the server emits `agent.working`, then `agent.accepted` after Pi accepted the prompt. If work is still in progress after two seconds and nothing has been spoken yet (see below), it sends one `agent.progress` event with a short spoken status clip, picked pseudo-randomly from a small set of natural filler phrases (“One sec.”, “Let me take a look.”, “Hmm, checking now.”, “Working on it.”, “Give me a moment.”, “Let me look into that.”). The server never repeats the same phrase on two consecutive turns. Ordinary screen-only `agent.working` heartbeats continue every 15 seconds regardless.
|
||
|
||
During a long turn, Pi may emit several assistant messages between tool calls before its final answer — each one ends with Pi's `message.end` event. Whenever a completed message includes a tool call (i.e., it is not the turn's final answer), the server also speaks it: it sends `{"type":"agent.progress","text":"<message text>","audio":true}` — `text` is always the message's full, untruncated text — immediately followed by that clip's binary audio frames, unless every word of that text was already spoken live as answer sentences (see below), in which case the clip is skipped entirely, or only its unspoken remainder is synthesized if part of it was spoken live and part was not. `agent.progress` may therefore occur multiple times per turn, always in the order the messages completed, and each is optionally followed by its own clip's binary frames. For speech only, a message longer than about 300 characters is truncated at a sentence boundary where possible (the JSON `text` field is never truncated). Progress clips are sent one at a time — a clip's audio always finishes sending before the next `agent.progress` frame is sent — and a turn's total spoken progress narration is capped at roughly 60 seconds of audio or 8 clips, whichever comes first; the fallback phrase counts toward this same cap. Once the turn's answer has audibly begun (the first `answer.sentence` has been sent — see below), a brand-new message that had no live narration of its own is left quiet rather than starting fresh progress narration that would talk over the answer; a message that was already mid-narration when a tool call appeared in it still gets its own unspoken leftover finished so it isn't cut off abruptly. `agent.progress` frames, spoken or not, always stop before `assistant.final`.
|
||
|
||
While the final answer streams in as Pi generates it, the server does not wait for it to fully finish before speaking it: as each complete sentence of the in-progress message accumulates (split on `.`/`!`/`?` — plus a closing quote or bracket — followed by whitespace; a decimal like “3.5” never triggers a false split, since the period is not followed by whitespace), the server sends `{"type":"answer.sentence","text":"<sentence>"}` immediately followed by that sentence's own binary audio frames, serialized through the same ordered chain as `agent.progress` clips so nothing interleaves or reorders on the single audio output. Markdown-decoration-only fragments (e.g. `---`, `` ``` ``, `**`) are skipped silently. Because a message might still turn out to include a tool call (making it narration, not the final answer), every message is optimistically streamed this way until either a tool call is detected in it or it ends; once a tool call appears, sentence-streaming for that message stops (already-spoken sentences remain spoken — that's fine acoustically) and any unspoken remainder is handled by the `agent.progress` mechanism above instead. A message's trailing fragment with no terminal punctuation yet is held back until the message ends, then flushed as one final `answer.sentence`. Spoken answer audio is capped at roughly 120 seconds or 40 sentences per turn, whichever comes first; once agent.settled fires, `assistant.final` (the full, untruncated answer text) is always sent next, and any part of the answer that was never spoken live because a cap was hit is synthesized once, as ordinary audio frames, between `assistant.final` and `audio.end` — a turn where every sentence streamed live sends zero audio frames in that window. `audio.end` always ends the turn's audio either way.
|
||
|
||
While it waits for Pi's native `agent.settled` event (not merely `agent.end`), it repeats the existing `agent.working` frame every 15 seconds; those progress frames stop before `assistant.final` and never overlap synthesized audio. The wait has a two-minute turn timeout. `assistant.delta` is streamed while Pi answers; `assistant.final` contains only assistant text, never STT transcript.
|
||
|
||
Azure synthesis output is signed little-endian 16-bit, 24 kHz, mono PCM. Every binary server frame has this eight-byte header followed by PCM:
|
||
|
||
| Bytes | Meaning |
|
||
|---|---|
|
||
| 0 | protocol version: `1` |
|
||
| 1 | kind: `1` (audio) |
|
||
| 2–3 | reserved: `0` (big endian) |
|
||
| 4–7 | monotonically increasing frame sequence (uint32 big endian) |
|
||
|
||
The server then emits `{"type":"audio.end"}` and a fresh `input.ready`. Send `{"type":"close"}` or close the WebSocket to end; disconnecting closes STT/event resources and clears the handle's working state.
|