snowspeeder
4986b6d7dc
- VoiceSession: token->connect with fresh-room retry on timeout, agent discovery, agent_ready wait, mic enable/mute, text send over lk.chat - ControlChannel: outbound set_voice/set_model/get_models/set_mode/ media_state; inbound agent_ready/models/model_changed/task_event - TranscriptHandler: transcription segments + transcript topic finals, partial/final dedupe per role - ConversationView/OrbView: status orb, mic/text-mode toggles, composer - ServerConfig: plain ws:// signaling for private IPs (self-signed TLS not injectable into SDK WebSocket); Info.plist local networking
5.2 KiB
5.2 KiB
Voice iOS — Plan & Progress
Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via XcodeBuildMCP.
Setup (reference)
- Repo /Users/shane/dev/voice-ios, branch feat/ios-app
- xcodegen at /opt/homebrew/bin/xcodegen; regenerate after file changes:
xcodegen generate - Build:
xcodebuild -project Voice.xcodeproj -scheme Voice_iOS -destination 'generic/platform=iOS Simulator' build(scheme is Voice_iOS, not Voice) - Run/relaunch: xcodebuildmcp_build_run_sim or launch_app_sim (bundle dev.shane.voice, sim iPhone 17 Pro AFDA137E-1657-428E-BF01-E5CD7989772C)
- UI: snapshot_ui → tap elementRef. Read logs:
xcrun simctl spawn booted log show --last Nm --predicate 'process == "Voice"' --style compact | grep -E "io.livekit|dev.shane.voice" - Mic permission (if simulator wiped):
xcrun simctl privacy booted grant microphone dev.shane.voice
Server facts (verified)
- Base https://192.168.86.2:9178 (nginx, self-signed CN=192.168.86.2; cert regenerates daily-ish → pins break on rotation)
- POST /token → {token, room: voice-shane-, user{id, display_name}}
- LiveKit host-network container
voiceon Unraid (ssh [email protected].2): signaling 7880/tcp, RTC tcp 7881, udp 7882 — all reachable from Mac (nc verified) - SDK cannot trust self-signed cert on its WebSocket → use plain ws://192.168.86.2:7880 (no /livekit path; SDK derives /rtc). Token+media only via nginx :9178 TLS (AVPlayer media needs local proxy in Phase 5)
- agent_ready on topic conversation-control; models on voice-control (plain string list); agent identity agent-*, kind .agent; lk.agent.session stream unhandled (harmless)
- TLSPinning.session() = custom URLSessionDelegate DER pin for our HTTP calls
Phases
- Phase 0: scaffold (commit
297fbaa) - Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit
378b8cb) - Phase 2: LiveKit session spike — ACTIVE
- Phase 3: state machine, audio levels, wake lock, history
- Phase 4: media playback reconciliation
- Phase 5: AVPlayer + local reverse proxy for self-signed https media
- Phase 6: settings parity (voice/model pickers, text mode)
- Phase 7: polish/edge cases
- Phase 8: real-device qualification checklist
- Phase 9: docs/final
Phase 2 status
Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received. UNCOMMITTED: all Phase 2 files + ServerConfig ws:// derivation + Info.plist NSAllowsLocalNetworking.
CONNECT TIMEOUT BUG — FIXED (2026-08-30 ~07:15)
Root cause: stale Room instance after a timed-out connect left signaling/transport in a broken state; subsequent connects hung. Fix in VoiceSession.swift:
roomis nowprivate(set) var, recreated fresh viamakeRoom()(delegates re-attached) on each retry.attemptConnect()retries up to 2 extra times (3 total) on.timedOut: tears down the room (teardownRoom()) and reconnects with a new Room + fresh token.makeConnectOptions()sets explicit timeouts (socket 15s, primary/publisher transport 20s).- Removed manual
AVAudioSessionconfig/deactivation — the SDK'sAudioSessionEngineObserver(isAutomaticConfigurationEnabled=true by default) owns the session; our manual setCategory/setActive raced it. Verified: token→room connected in <1s (retry fired once on a slow first attempt, then succeeded), agent_ready received, state=ready. 3/3 connect attempts now reach room-connected.
REMAINING: mic enable -4010 (simulator env limitation, NOT app bug)
After state=ready, setMicrophone(enabled:true) fails with CoreAudio -4010 (kAudioUnitErr_InvalidElement) on all 3 retries. Investigated thoroughly:
- TCC:
xcrun simctl privacy booted grant microphone dev.shane.voiceapplied; still -4010 → not a permission issue. - CoreAudio logs: input node reports 0 channel descriptions and the engine associates with audio session
(0x0)(null) → the simulator is not providing a usable mic source to this app in this environment. - NSMicrophoneUsageDescription present in Info.plist; SDK auto-configures the session correctly. Conclusion: simulator microphone hardware/source unavailable here. App code path is correct (mic enable retries, proper error surfacing). Needs verification on a real device or a simulator with a working mic source. Do NOT treat -4010 as an app regression.
After connect bug fixed
- Verify ready state reached (state=ready logged; UI transitions to conversation view)
- Verify transcript arrival + disconnect cleanup (blocked on mic/audio in sim; do on device)
- Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
- Commit Phase 2, mark done
Subagent notes
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
- Watch for loops (repeated identical commands) — kill and redirect if seen
- Don't let subagents loop reading SDK source; compile instead. SDK checkout: ~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/DerivedData/Voice-38a9ac34e5c8/SourcePackages/checkouts/client-sdk-swift