snowspeeder/pi-web
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: speak a progress update during long voice turns
CI / Verify and package (ubuntu-latest) (push) Canceled after 0s
Details
CI / Verify and package (windows-latest) (push) Canceled after 0s
Details

Browse Source 
After two seconds of agent work, send one agent.progress frame with a
synthesized "Hang on while I work on that." clip over the same binary
audio channel (shared output-sequence counter), at most once per turn;
screen-only agent.working heartbeats continue every 15 seconds. This is
the change the 15:11 rebuild already shipped to the running server —
committing it pins the protocol addition to the client that now
tolerates and plays it.

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
snowspeeder
2026-08-06 23:10:07 -04:00
co-authored by Claude Fable 5
parent a71cb8334a
commit a7033f9076
3 changed files with 45 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/voice-api.md
+1 -1
View File
@@ -93,7 +93,7 @@ Azure push-stream recognition emits `transcript.partial` and `transcript.final`
### Turn output and lifecycle
For either input form, the server emits `agent.working`, then `agent.accepted` after Pi accepted the prompt. While it waits for Pi's native `agent.settled` event (not merely `agent.end`), it repeats the existing `agent.working` frame every 15 seconds; those progress frames stop before `assistant.final` and never overlap synthesized audio. The wait has a two-minute turn timeout. `assistant.delta` is streamed while Pi answers; `assistant.final` contains only assistant text, never STT transcript.
For either input form, the server emits `agent.working`, then `agent.accepted` after Pi accepted the prompt. If work is still in progress after two seconds, it sends one `agent.progress` event with a short spoken status clip (for example, “Hang on while I work on that.”). It never repeats this spoken update within the same turn; ordinary screen-only `agent.working` heartbeats continue every 15 seconds. While it waits for Pi's native `agent.settled` event (not merely `agent.end`), it repeats the existing `agent.working` frame every 15 seconds; those progress frames stop before `assistant.final` and never overlap synthesized audio. The wait has a two-minute turn timeout. `assistant.delta` is streamed while Pi answers; `assistant.final` contains only assistant text, never STT transcript.
Azure synthesis output is signed little-endian 16-bit, 24 kHz, mono PCM. Every binary server frame has this eight-byte header followed by PCM: