hope-voice-api/PLAN.md

Voice iOS — Plan & Progress

Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via XcodeBuildMCP.

Setup (reference)

• Repo /Users/shane/dev/voice-ios, branch feat/ios-app
• xcodegen at /opt/homebrew/bin/xcodegen; regenerate after file changes: xcodegen generate
• Build: xcodebuild -project Voice.xcodeproj -scheme Voice_iOS -destination 'generic/platform=iOS Simulator' build (scheme is Voice_iOS, not Voice)
• Run/relaunch: xcodebuildmcp_build_run_sim or launch_app_sim (bundle dev.shane.voice, sim iPhone 17 Pro AFDA137E-1657-428E-BF01-E5CD7989772C)
• UI: snapshot_ui → tap elementRef. Read logs: xcrun simctl spawn booted log show --last Nm --predicate 'process == "Voice"' --style compact | grep -E "io.livekit|dev.shane.voice"
• Mic permission (if simulator wiped): xcrun simctl privacy booted grant microphone dev.shane.voice

Server facts (verified)

• Base https://192.168.86.2:9178 (nginx, self-signed CN=192.168.86.2; cert regenerates daily-ish → pins break on rotation)
• POST /token → {token, room: voice-shane-, user{id, display_name}}
• LiveKit host-network container voice on Unraid (ssh [email protected].2): signaling 7880/tcp, RTC tcp 7881, udp 7882 — all reachable from Mac (nc verified)
• SDK cannot trust self-signed cert on its WebSocket → use plain ws://192.168.86.2:7880 (no /livekit path; SDK derives /rtc). Token+media only via nginx :9178 TLS (AVPlayer media needs local proxy in Phase 5)
• agent_ready on topic conversation-control; models on voice-control (plain string list); agent identity agent-*, kind .agent; lk.agent.session stream unhandled (harmless)
• TLSPinning.session() = custom URLSessionDelegate DER pin for our HTTP calls

Phases

• Phase 0: scaffold (commit 297fbaa)
• Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
• Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
• Phase 3: state machine, audio levels, wake lock, history (commit 5483e6e)
• Phase 4: voice/model pickers with sync-on-join (commit pending)
• Phase 5: media playback reconciliation + AVPlayer + local reverse proxy for self-signed https media
• Phase 6: settings parity (text mode done in Phase 3; remaining polish)
• Phase 7: polish/edge cases
• Phase 8: real-device qualification checklist
• Phase 9: docs/final

Phase 2 status

Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received. UNCOMMITTED: all Phase 2 files + ServerConfig ws:// derivation + Info.plist NSAllowsLocalNetworking.

CONNECT TIMEOUT BUG — FIXED (2026-08-30 ~07:15)

Root cause: stale Room instance after a timed-out connect left signaling/transport in a broken state; subsequent connects hung. Fix in VoiceSession.swift:

• room is now private(set) var, recreated fresh via makeRoom() (delegates re-attached) on each retry.
• attemptConnect() retries up to 2 extra times (3 total) on .timedOut: tears down the room (teardownRoom()) and reconnects with a new Room + fresh token.
• makeConnectOptions() sets explicit timeouts (socket 15s, primary/publisher transport 20s).
• Removed manual AVAudioSession config/deactivation — the SDK's AudioSessionEngineObserver (isAutomaticConfigurationEnabled=true by default) owns the session; our manual setCategory/setActive raced it. Verified: token→room connected in <1s (retry fired once on a slow first attempt, then succeeded), agent_ready received, state=ready. 3/3 connect attempts now reach room-connected.

REMAINING: mic enable -4010 (simulator env limitation, NOT app bug)

After state=ready, setMicrophone(enabled:true) fails with CoreAudio -4010 (kAudioUnitErr_InvalidElement) on all 3 retries. Investigated thoroughly:

• TCC: xcrun simctl privacy booted grant microphone dev.shane.voice applied; still -4010 → not a permission issue.
• CoreAudio logs: input node reports 0 channel descriptions and the engine associates with audio session (0x0) (null) → the simulator is not providing a usable mic source to this app in this environment.
• NSMicrophoneUsageDescription present in Info.plist; SDK auto-configures the session correctly. Conclusion: simulator microphone hardware/source unavailable here. App code path is correct (mic enable retries, proper error surfacing). Needs verification on a real device or a simulator with a working mic source. Do NOT treat -4010 as an app regression.

After connect bug fixed

• Verify ready state reached (state=ready logged; UI transitions to conversation view)
• Verify transcript arrival + disconnect cleanup (verified in sim via text mode: user+agent finals arrived, hang-up cleaned up state)
• Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
• Commit Phase 2, mark done

Phase 3 status

Done:

• ConversationStateModel (Sources/LiveKit/ConversationState.swift): ports app.js inferState — EMA smoothing prev*0.6 + raw*0.4, user threshold 0.035, agent threshold 0.02, speaking→thinking when both silent, thinking chip after 2s in thinking phase. Driven by RoomDelegate.didUpdateSpeakingParticipants (participant.audioLevel), ticked only while session ready.
• Mic failure is now NON-FATAL: after 3 retries the error is surfaced via VoiceSession.micError (orange banner) and state stays .ready — text mode still works (also matches Phase 8 mic-denied fallback).
• Wake lock: isIdleTimerDisabled=true on room connect, false on disconnect.
• Text mode toggle: enables → setMuted(true) + set_mode:text; disables → set_mode:voice + re-enable mic. Persisted via SettingsStore.textMode.
• Transcript list: per-row timestamps, delivery-marker strip (web DELIVERY_MARKERS regex), Clear button, history of last 50 finals persisted to UserDefaults JSON (TranscriptHistory, key "voice_history").
• Barge-in hint line shown while in text mode; mic meter bar under status line in voice mode.

Verified in simulator (log evidence):

• Start → token→connect(retry)→room connected→agent_ready→state=ready.
• Mic -4010 x3 retries → error logged, session stayed ready, banner shown (non-fatal).
• Text mode toggle → log set_mode sent: text; mic muted.
• Composer send → log sendText lk.chat chars=24 → user transcript final arrived → agent TTS audio drove state machine connecting -> speaking (log [dev.shane.voice:state]) → agent transcript final "Hello! How can I assist you today?" arrived in list.
• Hang up → speaking -> idle, UI back to Start button.

Needs device verification:

• Live voice-driven states (listening while user speaks, thinking gap >2s chip, barge-in) — sim mic is silent (-4010), so local audio level never rises in sim; only remote (agent) audio drove the machine here.
• Mic meter movement with real speech; wake lock behavior on physical screen-off.

Phase 4 status (2026-08-30 ~11:50)

Done:

• Voices.swift: 13 Azure voices from web app.js; default en-US-Ava:DragonHDOmniLatestNeural.
• SettingsStore: voiceID/modelID persisted to UserDefaults; invalid stored voice falls back to default.
• VoiceSession: sync-on-join on agent_ready sends set_voice + set_model (if known); selectVoice/selectModel send immediately and persist; models list + model_changed acks tracked (@Published models, selectedModel).
• SettingsView: VoicePickerView sheet (13 voices), ModelPickerView sheet (models from session, "Start a conversation..." when not connected).
• ConversationView: shows current voice/model in status header when ready.

SENDTEXT BUG — FIXED (2026-08-30 ~11:47)

Root cause: ControlChannel used localParticipant.sendText(for:) (LiveKit DataStream protocol, v2.16). Agent listens on legacy data channel (ctx.room.on("data_received")), which never receives DataStream text → "ignoring text stream with topic 'voice-control', no callback attached" in agent logs; set_voice/set_model silently dropped, no model_changed ack. Fix: ControlChannel.publish now uses localParticipant.publish(data:options:) with DataPublishOptions(topic:, reliable: true) — same path as web UI's publishData. Verified (sim + agent docker logs, room voice-shane-7bf126cf):

• iOS: sync-on-join set_voice sent: en-US-JennyNeural; set_model sent: gemma-4-e4b → model_changed ack received and applied; models list count=30.
• Agent: "Switching TTS voice to en-US-JennyNeural" / "Voice updated" / "LLM model change: qwen2.5-vl-3b -> gemma-4-e4b".
• Persistence: relaunch → sync-on-join sent previously selected Jenny + gemma-4-e4b (survived UserDefaults). Note: sendText lk.chat in VoiceSession is also DataStream-based and the agent has no lk.chat handler — text chat path is unverified server-side, out of scope here.

Subagent notes

• Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
• Watch for loops (repeated identical commands) — kill and redirect if seen
• Don't let subagents loop reading SDK source; compile instead. SDK checkout: ~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/DerivedData/Voice-38a9ac34e5c8/SourcePackages/checkouts/client-sdk-swift