feat: voice and model pickers with sync-on-join (Phase 4)

- Voices.swift: 13 Azure voices, default en-US-Ava
- SettingsStore: persist voiceID/modelID, fallback to default voice
- VoiceSession: sync set_voice/set_model on agent_ready, selectVoice/selectModel, track models + model_changed
- SettingsView: voice and model picker sheets
- ControlChannel: publish via publish(data:) reliable data channel instead of sendText DataStream (agent listens on legacy data_received; DataStream text was dropped)
This commit is contained in:
snowspeeder committed 2026-08-30 07:50:45 -04:00
1 parent 5483e6ed48
commit ca2d26f171
8 files changed
+256 -14

No files matched your search

+21 -4
View File
@@ -22,10 +22,10 @@ Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via X
- [x] Phase 0: scaffold (commit 297fbaa)
- [x] Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
- [x] Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
- [x] Phase 3: state machine, audio levels, wake lock, history (commit 70aef02)
- [ ] Phase 4: media playback reconciliation
- [ ] Phase 5: AVPlayer + local reverse proxy for self-signed https media
- [ ] Phase 6: settings parity (voice/model pickers, text mode)
- [x] Phase 3: state machine, audio levels, wake lock, history (commit 5483e6e)
- [x] Phase 4: voice/model pickers with sync-on-join (commit pending)
- [ ] Phase 5: media playback reconciliation + AVPlayer + local reverse proxy for self-signed https media
- [ ] Phase 6: settings parity (text mode done in Phase 3; remaining polish)
- [ ] Phase 7: polish/edge cases
- [ ] Phase 8: real-device qualification checklist
- [ ] Phase 9: docs/final
@@ -75,6 +75,23 @@ Needs device verification:
- Live voice-driven states (listening while user speaks, thinking gap >2s chip, barge-in) — sim mic is silent (-4010), so local audio level never rises in sim; only remote (agent) audio drove the machine here.
- Mic meter movement with real speech; wake lock behavior on physical screen-off.
## Phase 4 status (2026-08-30 ~11:50)
Done:
- `Voices.swift`: 13 Azure voices from web app.js; default en-US-Ava:DragonHDOmniLatestNeural.
- `SettingsStore`: voiceID/modelID persisted to UserDefaults; invalid stored voice falls back to default.
- `VoiceSession`: sync-on-join on agent_ready sends set_voice + set_model (if known); selectVoice/selectModel send immediately and persist; models list + model_changed acks tracked (`@Published models`, `selectedModel`).
- `SettingsView`: VoicePickerView sheet (13 voices), ModelPickerView sheet (models from session, "Start a conversation..." when not connected).
- `ConversationView`: shows current voice/model in status header when ready.
### SENDTEXT BUG — FIXED (2026-08-30 ~11:47)
Root cause: ControlChannel used `localParticipant.sendText(for:)` (LiveKit DataStream protocol, v2.16). Agent listens on legacy data channel (`ctx.room.on("data_received")`), which never receives DataStream text → "ignoring text stream with topic 'voice-control', no callback attached" in agent logs; set_voice/set_model silently dropped, no model_changed ack.
Fix: `ControlChannel.publish` now uses `localParticipant.publish(data:options:)` with `DataPublishOptions(topic:, reliable: true)` — same path as web UI's `publishData`.
Verified (sim + agent docker logs, room voice-shane-7bf126cf):
- iOS: sync-on-join set_voice sent: en-US-JennyNeural; set_model sent: gemma-4-e4b → model_changed ack received and applied; models list count=30.
- Agent: "Switching TTS voice to en-US-JennyNeural" / "Voice updated" / "LLM model change: qwen2.5-vl-3b -> gemma-4-e4b".
- Persistence: relaunch → sync-on-join sent previously selected Jenny + gemma-4-e4b (survived UserDefaults).
Note: `sendText lk.chat` in VoiceSession is also DataStream-based and the agent has no lk.chat handler — text chat path is unverified server-side, out of scope here.
## Subagent notes
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
- Watch for loops (repeated identical commands) — kill and redirect if seen