Archived
snowspeeder
4986b6d7dc
- VoiceSession: token->connect with fresh-room retry on timeout, agent discovery, agent_ready wait, mic enable/mute, text send over lk.chat - ControlChannel: outbound set_voice/set_model/get_models/set_mode/ media_state; inbound agent_ready/models/model_changed/task_event - TranscriptHandler: transcription segments + transcript topic finals, partial/final dedupe per role - ConversationView/OrbView: status orb, mic/text-mode toggles, composer - ServerConfig: plain ws:// signaling for private IPs (self-signed TLS not injectable into SDK WebSocket); Info.plist local networking
62 lines
5.2 KiB
Markdown
62 lines
5.2 KiB
Markdown
# Voice iOS — Plan & Progress
|
|
|
|
Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via XcodeBuildMCP.
|
|
|
|
## Setup (reference)
|
|
- Repo /Users/shane/dev/voice-ios, branch feat/ios-app
|
|
- xcodegen at /opt/homebrew/bin/xcodegen; regenerate after file changes: `xcodegen generate`
|
|
- Build: `xcodebuild -project Voice.xcodeproj -scheme Voice_iOS -destination 'generic/platform=iOS Simulator' build` (scheme is **Voice_iOS**, not Voice)
|
|
- Run/relaunch: xcodebuildmcp_build_run_sim or launch_app_sim (bundle dev.shane.voice, sim iPhone 17 Pro AFDA137E-1657-428E-BF01-E5CD7989772C)
|
|
- UI: snapshot_ui → tap elementRef. Read logs: `xcrun simctl spawn booted log show --last Nm --predicate 'process == "Voice"' --style compact | grep -E "io.livekit|dev.shane.voice"`
|
|
- Mic permission (if simulator wiped): `xcrun simctl privacy booted grant microphone dev.shane.voice`
|
|
|
|
## Server facts (verified)
|
|
- Base https://192.168.86.2:9178 (nginx, self-signed CN=192.168.86.2; cert regenerates daily-ish → pins break on rotation)
|
|
- POST /token → {token, room: voice-shane-<hex8>, user{id, display_name}}
|
|
- LiveKit host-network container `voice` on Unraid (ssh root@192.168.86.2): signaling 7880/tcp, RTC tcp 7881, udp 7882 — all reachable from Mac (nc verified)
|
|
- SDK cannot trust self-signed cert on its WebSocket → use plain ws://192.168.86.2:7880 (no /livekit path; SDK derives /rtc). Token+media only via nginx :9178 TLS (AVPlayer media needs local proxy in Phase 5)
|
|
- agent_ready on topic conversation-control; models on voice-control (plain string list); agent identity agent-*, kind .agent; lk.agent.session stream unhandled (harmless)
|
|
- TLSPinning.session() = custom URLSessionDelegate DER pin for our HTTP calls
|
|
|
|
## Phases
|
|
- [x] Phase 0: scaffold (commit 297fbaa)
|
|
- [x] Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
|
|
- [ ] Phase 2: LiveKit session spike — **ACTIVE**
|
|
- [ ] Phase 3: state machine, audio levels, wake lock, history
|
|
- [ ] Phase 4: media playback reconciliation
|
|
- [ ] Phase 5: AVPlayer + local reverse proxy for self-signed https media
|
|
- [ ] Phase 6: settings parity (voice/model pickers, text mode)
|
|
- [ ] Phase 7: polish/edge cases
|
|
- [ ] Phase 8: real-device qualification checklist
|
|
- [ ] Phase 9: docs/final
|
|
|
|
## Phase 2 status
|
|
Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received.
|
|
UNCOMMITTED: all Phase 2 files + ServerConfig ws:// derivation + Info.plist NSAllowsLocalNetworking.
|
|
|
|
### CONNECT TIMEOUT BUG — FIXED (2026-08-30 ~07:15)
|
|
Root cause: stale `Room` instance after a timed-out connect left signaling/transport in a broken state; subsequent connects hung. Fix in VoiceSession.swift:
|
|
- `room` is now `private(set) var`, recreated fresh via `makeRoom()` (delegates re-attached) on each retry.
|
|
- `attemptConnect()` retries up to 2 extra times (3 total) on `.timedOut`: tears down the room (`teardownRoom()`) and reconnects with a new Room + fresh token.
|
|
- `makeConnectOptions()` sets explicit timeouts (socket 15s, primary/publisher transport 20s).
|
|
- Removed manual `AVAudioSession` config/deactivation — the SDK's `AudioSessionEngineObserver` (isAutomaticConfigurationEnabled=true by default) owns the session; our manual setCategory/setActive raced it.
|
|
Verified: token→room connected in <1s (retry fired once on a slow first attempt, then succeeded), agent_ready received, state=ready. 3/3 connect attempts now reach room-connected.
|
|
|
|
### REMAINING: mic enable -4010 (simulator env limitation, NOT app bug)
|
|
After state=ready, `setMicrophone(enabled:true)` fails with CoreAudio `-4010` (kAudioUnitErr_InvalidElement) on all 3 retries. Investigated thoroughly:
|
|
- TCC: `xcrun simctl privacy booted grant microphone dev.shane.voice` applied; still -4010 → not a permission issue.
|
|
- CoreAudio logs: input node reports **0 channel descriptions** and the engine associates with audio session `(0x0)` (null) → the simulator is not providing a usable mic source to this app in this environment.
|
|
- NSMicrophoneUsageDescription present in Info.plist; SDK auto-configures the session correctly.
|
|
Conclusion: simulator microphone hardware/source unavailable here. App code path is correct (mic enable retries, proper error surfacing). Needs verification on a real device or a simulator with a working mic source. Do NOT treat -4010 as an app regression.
|
|
|
|
### After connect bug fixed
|
|
- [x] Verify ready state reached (state=ready logged; UI transitions to conversation view)
|
|
- [ ] Verify transcript arrival + disconnect cleanup (blocked on mic/audio in sim; do on device)
|
|
- [x] Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
|
|
- [ ] Commit Phase 2, mark done
|
|
|
|
## Subagent notes
|
|
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
|
|
- Watch for loops (repeated identical commands) — kill and redirect if seen
|
|
- Don't let subagents loop reading SDK source; compile instead. SDK checkout: ~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/DerivedData/Voice-38a9ac34e5c8/SourcePackages/checkouts/client-sdk-swift
|