Archived
snowspeederandClaude Fable 5
a7033f9076
After two seconds of agent work, send one agent.progress frame with a synthesized "Hang on while I work on that." clip over the same binary audio channel (shared output-sequence counter), at most once per turn; screen-only agent.working heartbeats continue every 15 seconds. This is the change the 15:11 rebuild already shipped to the running server — committing it pins the protocol addition to the client that now tolerates and plays it. Co-Authored-By: Claude Fable 5 <[email protected]>
108 lines
5.5 KiB
Markdown
108 lines
5.5 KiB
Markdown
# Voice Conversation API (v1)
|
||
|
||
`/api/v1/voice` is a server-side device API for native iOS and Echo-style clients. It is separate from the browser voice feature. Azure credentials remain on the PI WEB host. Every voice turn automatically includes server-side instructions to produce concise, natural DragonHD-friendly speech without Markdown or visual-only formatting.
|
||
|
||
## Security and provisioning
|
||
|
||
Create a device token on the PI WEB host:
|
||
|
||
```sh
|
||
pi-web voice-token create \
|
||
--project project-id --workspace workspace-id \
|
||
--model openai-codex/gpt-5.6-terra --thinking high
|
||
```
|
||
|
||
Each restriction flag may be repeated. Omitting a category grants `*` for that category. A restricted model or thinking scope requires the client to explicitly select an allowed value; it cannot inherit a server default. Project and workspace scopes are both enforced. Token records contain a SHA-256 token hash only and are persisted at mode `0600` in `~/.config/pi-web/voice-api.json`; the plaintext (`pwv1_...`) is printed only once.
|
||
|
||
```sh
|
||
pi-web voice-token list
|
||
pi-web voice-token revoke <token-id>
|
||
```
|
||
|
||
Use `Authorization: Bearer pwv1_...` on every HTTP and WebSocket connection. The API requires HTTPS, except loopback for local development/testing. A conversation is owned by the creating token: other valid device tokens receive `404` when they try to inspect, delete, or attach it. A client may select only a registered workspace id, never a filesystem path.
|
||
|
||
## HTTP
|
||
|
||
`GET /api/v1/voice/targets` returns only the caller's authorized registered workspaces plus effective scopes. Use its `workspaces[].id` to choose a workspace; paths are returned for display but are never accepted as an API input.
|
||
|
||
`GET /api/v1/voice/conversations?workspaceId=<workspace-id>` lists prior Pi sessions rooted at that authorized workspace. This lets a device show a conversation picker without learning about sessions outside its workspace scope.
|
||
|
||
`POST /api/v1/voice/conversations`
|
||
|
||
```json
|
||
{
|
||
"workspaceId": "registered-workspace-id",
|
||
"model": "openai-codex/gpt-5.6-terra",
|
||
"thinking": "high",
|
||
"context": "You are helping with the home-automation project."
|
||
}
|
||
```
|
||
|
||
For an existing directory beneath the device-safe root `/home/hope/workspaces` that is not yet registered as a PI WEB workspace, use `POST /api/v1/voice/conversations/path` with `path` instead of `workspaceId`. The path must already exist, resolve beneath that root, and is validated before a session starts.
|
||
|
||
The response is `201` and includes the opaque conversation id, session id, selected workspace, and `input-ready` status. `GET /api/v1/voice/conversations/:id/models` lists the models currently available to that conversation's Pi session, filtered by the device token's model scope. `GET` and `DELETE /api/v1/voice/conversations/:id` inspect and close the API conversation handle.
|
||
|
||
To resume a listed session, create a new token-owned voice handle:
|
||
|
||
```json
|
||
POST /api/v1/voice/conversations/resume
|
||
{
|
||
"workspaceId": "registered-workspace-id",
|
||
"sessionId": "previous-pi-session-id"
|
||
}
|
||
```
|
||
|
||
The session must be in the selected workspace and within the caller's token scope.
|
||
|
||
## WebSocket wire protocol
|
||
|
||
Connect to `wss://host/api/v1/voice/stream` with the Bearer header. The server first sends:
|
||
|
||
```json
|
||
{
|
||
"type":"hello",
|
||
"protocol":"pi-web.voice.v1",
|
||
"pcm":{"input":"s16le/16000/mono","output":"s16le/24000/mono"},
|
||
"binary":{"version":1,"headerBytes":8,"kind":"audio"}
|
||
}
|
||
```
|
||
|
||
Attach an owned conversation and wait for `input.ready` before sending a turn:
|
||
|
||
```json
|
||
{"type":"attach","conversationId":"..."}
|
||
```
|
||
|
||
### Text fallback
|
||
|
||
Send exactly one JSON input while `input-ready`:
|
||
|
||
```json
|
||
{"type":"input.text","text":"Turn on the kitchen lights."}
|
||
```
|
||
|
||
### Audio input
|
||
|
||
For speech input, send binary WebSocket frames containing raw signed little-endian 16-bit, 16 kHz, mono PCM (no WAV header). Frames must be nonempty, <=64 KiB, aligned to two bytes, and an utterance is limited to two minutes. Finish with:
|
||
|
||
```json
|
||
{"type":"input.end"}
|
||
```
|
||
|
||
Azure push-stream recognition emits `transcript.partial` and `transcript.final` JSON events. The final transcript is submitted to Pi. This Voice API never returns an Azure token or key to a device.
|
||
|
||
### Turn output and lifecycle
|
||
|
||
For either input form, the server emits `agent.working`, then `agent.accepted` after Pi accepted the prompt. If work is still in progress after two seconds, it sends one `agent.progress` event with a short spoken status clip (for example, “Hang on while I work on that.”). It never repeats this spoken update within the same turn; ordinary screen-only `agent.working` heartbeats continue every 15 seconds. While it waits for Pi's native `agent.settled` event (not merely `agent.end`), it repeats the existing `agent.working` frame every 15 seconds; those progress frames stop before `assistant.final` and never overlap synthesized audio. The wait has a two-minute turn timeout. `assistant.delta` is streamed while Pi answers; `assistant.final` contains only assistant text, never STT transcript.
|
||
|
||
Azure synthesis output is signed little-endian 16-bit, 24 kHz, mono PCM. Every binary server frame has this eight-byte header followed by PCM:
|
||
|
||
| Bytes | Meaning |
|
||
|---|---|
|
||
| 0 | protocol version: `1` |
|
||
| 1 | kind: `1` (audio) |
|
||
| 2–3 | reserved: `0` (big endian) |
|
||
| 4–7 | monotonically increasing frame sequence (uint32 big endian) |
|
||
|
||
The server then emits `{"type":"audio.end"}` and a fresh `input.ready`. Send `{"type":"close"}` or close the WebSocket to end; disconnecting closes STT/event resources and clears the handle's working state.
|