Archived
99 lines
9.2 KiB
Markdown
99 lines
9.2 KiB
Markdown
# Voice iOS — Plan & Progress
|
|
|
|
Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via XcodeBuildMCP.
|
|
|
|
## Setup (reference)
|
|
- Repo /Users/shane/dev/voice-ios, branch feat/ios-app
|
|
- xcodegen at /opt/homebrew/bin/xcodegen; regenerate after file changes: `xcodegen generate`
|
|
- Build: `xcodebuild -project Voice.xcodeproj -scheme Voice_iOS -destination 'generic/platform=iOS Simulator' build` (scheme is **Voice_iOS**, not Voice)
|
|
- Run/relaunch: xcodebuildmcp_build_run_sim or launch_app_sim (bundle dev.shane.voice, sim iPhone 17 Pro AFDA137E-1657-428E-BF01-E5CD7989772C)
|
|
- UI: snapshot_ui → tap elementRef. Read logs: `xcrun simctl spawn booted log show --last Nm --predicate 'process == "Voice"' --style compact | grep -E "io.livekit|dev.shane.voice"`
|
|
- Mic permission (if simulator wiped): `xcrun simctl privacy booted grant microphone dev.shane.voice`
|
|
|
|
## Server facts (verified)
|
|
- Base https://192.168.86.2:9178 (nginx, self-signed CN=192.168.86.2; cert regenerates daily-ish → pins break on rotation)
|
|
- POST /token → {token, room: voice-shane-<hex8>, user{id, display_name}}
|
|
- LiveKit host-network container `voice` on Unraid (ssh root@192.168.86.2): signaling 7880/tcp, RTC tcp 7881, udp 7882 — all reachable from Mac (nc verified)
|
|
- SDK cannot trust self-signed cert on its WebSocket → use plain ws://192.168.86.2:7880 (no /livekit path; SDK derives /rtc). Token+media only via nginx :9178 TLS (AVPlayer media needs local proxy in Phase 5)
|
|
- agent_ready on topic conversation-control; models on voice-control (plain string list); agent identity agent-*, kind .agent; lk.agent.session stream unhandled (harmless)
|
|
- TLSPinning.session() = custom URLSessionDelegate DER pin for our HTTP calls
|
|
|
|
## Phases
|
|
- [x] Phase 0: scaffold (commit 297fbaa)
|
|
- [x] Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
|
|
- [x] Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
|
|
- [x] Phase 3: state machine, audio levels, wake lock, history (commit 5483e6e)
|
|
- [x] Phase 4: voice/model pickers with sync-on-join (commit ca2d26f)
|
|
- [ ] Phase 5: media playback reconciliation + AVPlayer + local reverse proxy for self-signed https media
|
|
- [ ] Phase 6: settings parity (text mode done in Phase 3; remaining polish)
|
|
- [ ] Phase 7: polish/edge cases
|
|
- [ ] Phase 8: real-device qualification checklist
|
|
- [ ] Phase 9: docs/final
|
|
|
|
## Phase 2 status
|
|
Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received.
|
|
UNCOMMITTED: all Phase 2 files + ServerConfig ws:// derivation + Info.plist NSAllowsLocalNetworking.
|
|
|
|
### CONNECT TIMEOUT BUG — FIXED (2026-08-30 ~07:15)
|
|
Root cause: stale `Room` instance after a timed-out connect left signaling/transport in a broken state; subsequent connects hung. Fix in VoiceSession.swift:
|
|
- `room` is now `private(set) var`, recreated fresh via `makeRoom()` (delegates re-attached) on each retry.
|
|
- `attemptConnect()` retries up to 2 extra times (3 total) on `.timedOut`: tears down the room (`teardownRoom()`) and reconnects with a new Room + fresh token.
|
|
- `makeConnectOptions()` sets explicit timeouts (socket 15s, primary/publisher transport 20s).
|
|
- Removed manual `AVAudioSession` config/deactivation — the SDK's `AudioSessionEngineObserver` (isAutomaticConfigurationEnabled=true by default) owns the session; our manual setCategory/setActive raced it.
|
|
Verified: token→room connected in <1s (retry fired once on a slow first attempt, then succeeded), agent_ready received, state=ready. 3/3 connect attempts now reach room-connected.
|
|
|
|
### REMAINING: mic enable -4010 (simulator env limitation, NOT app bug)
|
|
After state=ready, `setMicrophone(enabled:true)` fails with CoreAudio `-4010` (kAudioUnitErr_InvalidElement) on all 3 retries. Investigated thoroughly:
|
|
- TCC: `xcrun simctl privacy booted grant microphone dev.shane.voice` applied; still -4010 → not a permission issue.
|
|
- CoreAudio logs: input node reports **0 channel descriptions** and the engine associates with audio session `(0x0)` (null) → the simulator is not providing a usable mic source to this app in this environment.
|
|
- NSMicrophoneUsageDescription present in Info.plist; SDK auto-configures the session correctly.
|
|
Conclusion: simulator microphone hardware/source unavailable here. App code path is correct (mic enable retries, proper error surfacing). Needs verification on a real device or a simulator with a working mic source. Do NOT treat -4010 as an app regression.
|
|
|
|
### After connect bug fixed
|
|
- [x] Verify ready state reached (state=ready logged; UI transitions to conversation view)
|
|
- [x] Verify transcript arrival + disconnect cleanup (verified in sim via text mode: user+agent finals arrived, hang-up cleaned up state)
|
|
- [x] Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
|
|
- [ ] Commit Phase 2, mark done
|
|
|
|
## Phase 3 status
|
|
Done:
|
|
- `ConversationStateModel` (Sources/LiveKit/ConversationState.swift): ports app.js inferState — EMA smoothing `prev*0.6 + raw*0.4`, user threshold 0.035, agent threshold 0.02, speaking→thinking when both silent, thinking chip after 2s in thinking phase. Driven by `RoomDelegate.didUpdateSpeakingParticipants` (participant.audioLevel), ticked only while session ready.
|
|
- Mic failure is now NON-FATAL: after 3 retries the error is surfaced via `VoiceSession.micError` (orange banner) and state stays `.ready` — text mode still works (also matches Phase 8 mic-denied fallback).
|
|
- Wake lock: `isIdleTimerDisabled=true` on room connect, false on disconnect.
|
|
- Text mode toggle: enables → setMuted(true) + `set_mode:text`; disables → `set_mode:voice` + re-enable mic. Persisted via SettingsStore.textMode.
|
|
- Transcript list: per-row timestamps, delivery-marker strip (web DELIVERY_MARKERS regex), Clear button, history of last 50 finals persisted to UserDefaults JSON (`TranscriptHistory`, key "voice_history").
|
|
- Barge-in hint line shown while in text mode; mic meter bar under status line in voice mode.
|
|
|
|
Verified in simulator (log evidence):
|
|
- Start → token→connect(retry)→room connected→agent_ready→state=ready.
|
|
- Mic -4010 x3 retries → error logged, session stayed ready, banner shown (non-fatal).
|
|
- Text mode toggle → log `set_mode sent: text`; mic muted.
|
|
- Composer send → log `sendText lk.chat chars=24` → user transcript final arrived → agent TTS audio drove state machine `connecting -> speaking` (log `[dev.shane.voice:state]`) → agent transcript final "Hello! How can I assist you today?" arrived in list.
|
|
- Hang up → `speaking -> idle`, UI back to Start button.
|
|
|
|
Needs device verification:
|
|
- Live voice-driven states (listening while user speaks, thinking gap >2s chip, barge-in) — sim mic is silent (-4010), so local audio level never rises in sim; only remote (agent) audio drove the machine here.
|
|
- Mic meter movement with real speech; wake lock behavior on physical screen-off.
|
|
|
|
## Phase 4 status (2026-08-30 ~11:50)
|
|
Done:
|
|
- `Voices.swift`: 13 Azure voices from web app.js; default en-US-Ava:DragonHDOmniLatestNeural.
|
|
- `SettingsStore`: voiceID/modelID persisted to UserDefaults; invalid stored voice falls back to default.
|
|
- `VoiceSession`: sync-on-join on agent_ready sends set_voice + set_model (if known); selectVoice/selectModel send immediately and persist; models list + model_changed acks tracked (`@Published models`, `selectedModel`).
|
|
- `SettingsView`: VoicePickerView sheet (13 voices), ModelPickerView sheet (models from session, "Start a conversation..." when not connected).
|
|
- `ConversationView`: shows current voice/model in status header when ready.
|
|
|
|
### SENDTEXT BUG — FIXED (2026-08-30 ~11:47)
|
|
Root cause: ControlChannel used `localParticipant.sendText(for:)` (LiveKit DataStream protocol, v2.16). Agent listens on legacy data channel (`ctx.room.on("data_received")`), which never receives DataStream text → "ignoring text stream with topic 'voice-control', no callback attached" in agent logs; set_voice/set_model silently dropped, no model_changed ack.
|
|
Fix: `ControlChannel.publish` now uses `localParticipant.publish(data:options:)` with `DataPublishOptions(topic:, reliable: true)` — same path as web UI's `publishData`.
|
|
Verified (sim + agent docker logs, room voice-shane-7bf126cf):
|
|
- iOS: sync-on-join set_voice sent: en-US-JennyNeural; set_model sent: gemma-4-e4b → model_changed ack received and applied; models list count=30.
|
|
- Agent: "Switching TTS voice to en-US-JennyNeural" / "Voice updated" / "LLM model change: qwen2.5-vl-3b -> gemma-4-e4b".
|
|
- Persistence: relaunch → sync-on-join sent previously selected Jenny + gemma-4-e4b (survived UserDefaults).
|
|
Note: `sendText lk.chat` in VoiceSession is also DataStream-based and the agent has no lk.chat handler — text chat path is unverified server-side, out of scope here.
|
|
|
|
## Subagent notes
|
|
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
|
|
- Watch for loops (repeated identical commands) — kill and redirect if seen
|
|
- Don't let subagents loop reading SDK source; compile instead. SDK checkout: ~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/DerivedData/Voice-38a9ac34e5c8/SourcePackages/checkouts/client-sdk-swift
|