feat: conversation state machine, mic meter, history, wake lock (Phase 3)

This commit is contained in:
snowspeeder committed 2026-08-30 07:36:13 -04:00
1 parent 4986b6d7dc
commit 5483e6ed48
8 files changed
+348 -29

No files matched your search

+23 -3
View File
@@ -21,8 +21,8 @@ Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via X
## Phases
- [x] Phase 0: scaffold (commit 297fbaa)
- [x] Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
- [ ] Phase 2: LiveKit session spike — **ACTIVE**
- [ ] Phase 3: state machine, audio levels, wake lock, history
- [x] Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
- [x] Phase 3: state machine, audio levels, wake lock, history (commit 70aef02)
- [ ] Phase 4: media playback reconciliation
- [ ] Phase 5: AVPlayer + local reverse proxy for self-signed https media
- [ ] Phase 6: settings parity (voice/model pickers, text mode)
@@ -51,10 +51,30 @@ Conclusion: simulator microphone hardware/source unavailable here. App code path
### After connect bug fixed
- [x] Verify ready state reached (state=ready logged; UI transitions to conversation view)
- [ ] Verify transcript arrival + disconnect cleanup (blocked on mic/audio in sim; do on device)
- [x] Verify transcript arrival + disconnect cleanup (verified in sim via text mode: user+agent finals arrived, hang-up cleaned up state)
- [x] Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
- [ ] Commit Phase 2, mark done
## Phase 3 status
Done:
- `ConversationStateModel` (Sources/LiveKit/ConversationState.swift): ports app.js inferState — EMA smoothing `prev*0.6 + raw*0.4`, user threshold 0.035, agent threshold 0.02, speaking→thinking when both silent, thinking chip after 2s in thinking phase. Driven by `RoomDelegate.didUpdateSpeakingParticipants` (participant.audioLevel), ticked only while session ready.
- Mic failure is now NON-FATAL: after 3 retries the error is surfaced via `VoiceSession.micError` (orange banner) and state stays `.ready` — text mode still works (also matches Phase 8 mic-denied fallback).
- Wake lock: `isIdleTimerDisabled=true` on room connect, false on disconnect.
- Text mode toggle: enables → setMuted(true) + `set_mode:text`; disables → `set_mode:voice` + re-enable mic. Persisted via SettingsStore.textMode.
- Transcript list: per-row timestamps, delivery-marker strip (web DELIVERY_MARKERS regex), Clear button, history of last 50 finals persisted to UserDefaults JSON (`TranscriptHistory`, key "voice_history").
- Barge-in hint line shown while in text mode; mic meter bar under status line in voice mode.
Verified in simulator (log evidence):
- Start → token→connect(retry)→room connected→agent_ready→state=ready.
- Mic -4010 x3 retries → error logged, session stayed ready, banner shown (non-fatal).
- Text mode toggle → log `set_mode sent: text`; mic muted.
- Composer send → log `sendText lk.chat chars=24` → user transcript final arrived → agent TTS audio drove state machine `connecting -> speaking` (log `[dev.shane.voice:state]`) → agent transcript final "Hello! How can I assist you today?" arrived in list.
- Hang up → `speaking -> idle`, UI back to Start button.
Needs device verification:
- Live voice-driven states (listening while user speaks, thinking gap >2s chip, barge-in) — sim mic is silent (-4010), so local audio level never rises in sim; only remote (agent) audio drove the machine here.
- Mic meter movement with real speech; wake lock behavior on physical screen-off.
## Subagent notes
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
- Watch for loops (repeated identical commands) — kill and redirect if seen