hope-voice-ios/PLAN.md

Voice iOS — Plan & Progress

Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via XcodeBuildMCP.

Setup (reference)

• Repo /Users/shane/dev/voice-ios, branch feat/ios-app
• xcodegen at /opt/homebrew/bin/xcodegen; regenerate after file changes: xcodegen generate
• Build: xcodebuild -project Voice.xcodeproj -scheme Voice_iOS -destination 'generic/platform=iOS Simulator' build (scheme is Voice_iOS, not Voice)
• Run/relaunch: xcodebuildmcp_build_run_sim or launch_app_sim (bundle dev.shane.voice, sim iPhone 17 Pro AFDA137E-1657-428E-BF01-E5CD7989772C)
• UI: snapshot_ui → tap elementRef. Read logs: xcrun simctl spawn booted log show --last Nm --predicate 'process == "Voice"' --style compact | grep -E "io.livekit|dev.shane.voice"
• Mic permission (if simulator wiped): xcrun simctl privacy booted grant microphone dev.shane.voice

Server facts (verified)

• Base https://192.168.86.2:9178 (nginx, self-signed CN=192.168.86.2; cert regenerates daily-ish → pins break on rotation)
• POST /token → {token, room: voice-shane-, user{id, display_name}}
• LiveKit host-network container voice on Unraid (ssh [email protected].2): signaling 7880/tcp, RTC tcp 7881, udp 7882 — all reachable from Mac (nc verified)
• SDK cannot trust self-signed cert on its WebSocket → use plain ws://192.168.86.2:7880 (no /livekit path; SDK derives /rtc). Token+media only via nginx :9178 TLS (AVPlayer media needs local proxy in Phase 5)
• agent_ready on topic conversation-control; models on voice-control (plain string list); agent identity agent-*, kind .agent; lk.agent.session stream unhandled (harmless)
• TLSPinning.session() = custom URLSessionDelegate DER pin for our HTTP calls

Phases

• Phase 0: scaffold (commit 297fbaa)
• Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
• Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
• Phase 3: state machine, audio levels, wake lock, history (commit 5483e6e)
• Phase 4: voice/model pickers with sync-on-join (commit ca2d26f)
• Phase 5: media player — commit 0af944e (webm playback blocked by iOS AVPlayer; see status below)
• Phase 6: media search, choices, library panel (commit cbf1830)
• Phase 7: background tasks panel (commit af296da)
• Phase 8: polish/hardening/parity pass (commit pending)
• Phase 9: docs & handoff (commit e63e53e)

Phase 2 status

Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received. UNCOMMITTED: all Phase 2 files + ServerConfig ws:// derivation + Info.plist NSAllowsLocalNetworking.

CONNECT TIMEOUT BUG — FIXED (2026-08-30 ~07:15)

Root cause: stale Room instance after a timed-out connect left signaling/transport in a broken state; subsequent connects hung. Fix in VoiceSession.swift:

• room is now private(set) var, recreated fresh via makeRoom() (delegates re-attached) on each retry.
• attemptConnect() retries up to 2 extra times (3 total) on .timedOut: tears down the room (teardownRoom()) and reconnects with a new Room + fresh token.
• makeConnectOptions() sets explicit timeouts (socket 15s, primary/publisher transport 20s).
• Removed manual AVAudioSession config/deactivation — the SDK's AudioSessionEngineObserver (isAutomaticConfigurationEnabled=true by default) owns the session; our manual setCategory/setActive raced it. Verified: token→room connected in <1s (retry fired once on a slow first attempt, then succeeded), agent_ready received, state=ready. 3/3 connect attempts now reach room-connected.

REMAINING: mic enable -4010 (simulator env limitation, NOT app bug)

After state=ready, setMicrophone(enabled:true) fails with CoreAudio -4010 (kAudioUnitErr_InvalidElement) on all 3 retries. Investigated thoroughly:

• TCC: xcrun simctl privacy booted grant microphone dev.shane.voice applied; still -4010 → not a permission issue.
• CoreAudio logs: input node reports 0 channel descriptions and the engine associates with audio session (0x0) (null) → the simulator is not providing a usable mic source to this app in this environment.
• NSMicrophoneUsageDescription present in Info.plist; SDK auto-configures the session correctly. Conclusion: simulator microphone hardware/source unavailable here. App code path is correct (mic enable retries, proper error surfacing). Needs verification on a real device or a simulator with a working mic source. Do NOT treat -4010 as an app regression.

After connect bug fixed

• Verify ready state reached (state=ready logged; UI transitions to conversation view)
• Verify transcript arrival + disconnect cleanup (verified in sim via text mode: user+agent finals arrived, hang-up cleaned up state)
• Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
• Commit Phase 2, mark done

Phase 3 status

Done:

• ConversationStateModel (Sources/LiveKit/ConversationState.swift): ports app.js inferState — EMA smoothing prev*0.6 + raw*0.4, user threshold 0.035, agent threshold 0.02, speaking→thinking when both silent, thinking chip after 2s in thinking phase. Driven by RoomDelegate.didUpdateSpeakingParticipants (participant.audioLevel), ticked only while session ready.
• Mic failure is now NON-FATAL: after 3 retries the error is surfaced via VoiceSession.micError (orange banner) and state stays .ready — text mode still works (also matches Phase 8 mic-denied fallback).
• Wake lock: isIdleTimerDisabled=true on room connect, false on disconnect.
• Text mode toggle: enables → setMuted(true) + set_mode:text; disables → set_mode:voice + re-enable mic. Persisted via SettingsStore.textMode.
• Transcript list: per-row timestamps, delivery-marker strip (web DELIVERY_MARKERS regex), Clear button, history of last 50 finals persisted to UserDefaults JSON (TranscriptHistory, key "voice_history").
• Barge-in hint line shown while in text mode; mic meter bar under status line in voice mode.

Verified in simulator (log evidence):

• Start → token→connect(retry)→room connected→agent_ready→state=ready.
• Mic -4010 x3 retries → error logged, session stayed ready, banner shown (non-fatal).
• Text mode toggle → log set_mode sent: text; mic muted.
• Composer send → log sendText lk.chat chars=24 → user transcript final arrived → agent TTS audio drove state machine connecting -> speaking (log [dev.shane.voice:state]) → agent transcript final "Hello! How can I assist you today?" arrived in list.
• Hang up → speaking -> idle, UI back to Start button.

Needs device verification:

• Live voice-driven states (listening while user speaks, thinking gap >2s chip, barge-in) — sim mic is silent (-4010), so local audio level never rises in sim; only remote (agent) audio drove the machine here.
• Mic meter movement with real speech; wake lock behavior on physical screen-off.

Phase 4 status (2026-08-30 ~11:50)

Done:

• Voices.swift: 13 Azure voices from web app.js; default en-US-Ava:DragonHDOmniLatestNeural.
• SettingsStore: voiceID/modelID persisted to UserDefaults; invalid stored voice falls back to default.
• VoiceSession: sync-on-join on agent_ready sends set_voice + set_model (if known); selectVoice/selectModel send immediately and persist; models list + model_changed acks tracked (@Published models, selectedModel).
• SettingsView: VoicePickerView sheet (13 voices), ModelPickerView sheet (models from session, "Start a conversation..." when not connected).
• ConversationView: shows current voice/model in status header when ready.

SENDTEXT BUG — FIXED (2026-08-30 ~11:47)

Root cause: ControlChannel used localParticipant.sendText(for:) (LiveKit DataStream protocol, v2.16). Agent listens on legacy data channel (ctx.room.on("data_received")), which never receives DataStream text → "ignoring text stream with topic 'voice-control', no callback attached" in agent logs; set_voice/set_model silently dropped, no model_changed ack. Fix: ControlChannel.publish now uses localParticipant.publish(data:options:) with DataPublishOptions(topic:, reliable: true) — same path as web UI's publishData. Verified (sim + agent docker logs, room voice-shane-7bf126cf):

• iOS: sync-on-join set_voice sent: en-US-JennyNeural; set_model sent: gemma-4-e4b → model_changed ack received and applied; models list count=30.
• Agent: "Switching TTS voice to en-US-JennyNeural" / "Voice updated" / "LLM model change: qwen2.5-vl-3b -> gemma-4-e4b".
• Persistence: relaunch → sync-on-join sent previously selected Jenny + gemma-4-e4b (survived UserDefaults). Note: sendText lk.chat in VoiceSession is also DataStream-based and the agent has no lk.chat handler — text chat path is unverified server-side, out of scope here.

Phase 5 status (2026-08-30 ~12:20)

Done:

• MediaAPI (Sources/Media/MediaAPI.swift): client for all 10 endpoints — /state, /report, /play, /control, /search, /library, /tracks/{id} DELETE, /streams, /stream/{track_id}, /healthz. Codable models (MediaTrack, MediaState, MediaPendingJob, MediaSeekHint, MediaReported, MediaChoice, MediaLibrary, MediaSearchResult/Response). Uses TLSPinning.session().
• PinnedMediaResourceLoader (Sources/Media/PinnedMediaResourceLoader.swift): AVAssetResourceLoaderDelegate. AVPlayer loads https natively (ignores registered URLProtocol subclasses) and can't trust the self-signed cert, so the asset uses a custom pinned-media:// scheme; the delegate answers every data request by fetching the real https URL through TLSPinning.session(), forwarding Range headers and setting contentInformationRequest (contentType/contentLength/byteRangeAccessSupported).
• MediaPlayerController (Sources/Media/MediaPlayerController.swift): @MainActor ObservableObject, AVPlayer-backed. 1.5s poll of /state while a room is scoped; reconciliation ported from web app.js — track-id swap → new item; late audio_url → wire source without reload; desired=="playing"+paused → play else pause; seek_hint applied once per ts; /report on play/pause/ended + ≥5s periodic; publishMediaState via ControlChannel; enter-media-scope carry-over (start_paused, wasPlayingBeforeTalk); end-conversation resume. Transport controls (togglePlayPause ±10s seek, stop), volume from SettingsStore.
• MediaPanelView (Sources/Features/Media/MediaPanelView.swift): now-playing card (AsyncImage art, title, channel, status), transport buttons, seek slider, volume slider, pending download jobs.
• Wiring: VoiceSession.mediaController + enterMediaScope before room connect + endConversation on disconnect; ConversationView music.note toolbar button → MediaPanelView.

Verified in simulator (log + UI evidence):

• /healthz and /library reachable via curl through pinned TLS (200).
• Media panel reachable from toolbar; empty state "Nothing loaded" shown when no track.
• Reconciliation: POST /play into the app's room → poll picks up track → log wiring source pinned-media://... → delegate intercepts (log fetching https://... range=bytes=0-1 then full range) → panel shows "Toes (Greatest Hits Version)" with transport controls. Track-id swap, desired-state apply, and reporting all exercised.
• Range proxy works: AVPlayer's bytes=0-1 probe + full-range fetch both answered through the pinned session (no cert error).

BLOCKED — webm playback (platform limitation, NOT app bug):

• The stream proxy delivers valid webm bytes (EBML header + "webm" string verified), but iOS AVPlayer cannot decode webm: item fails AVFoundationErrorDomain -11828 Cannot Open / This media format is not supported. Confirmed via UTType: org.webmproject.webm conformsToAudio=false (iOS supports m4a/aac/mp3/ogg/opus/flac/wav, not webm).
• Both library tracks are webm (server-side yt-dlp output). So end-to-end audible playback cannot be verified on iOS until the server offers an iOS-decodable format (e.g. add -f "bestaudio[ext=m4a]/bestaudio" --remux-video or serve .m4a/.aac alongside webm).
• Everything up to decode is verified working: TLS pinning, Range proxy, reconciliation, polling, reporting, UI. The app will play correctly the moment a supported-format track is available — no client change needed for that.

Needs server change to verify audible playback:

• Server: provide an iOS-decodable audio format (m4a/aac/mp3) for library tracks, then re-test playback in sim.

Phase 6 status (2026-08-30 ~13:00)

Done:

• MediaPanelView expanded: search section (query + Find button, results with "in library" tag), choices section (agent-offered suggestions from state), library section (list with Play/Remove per track, refresh button), add-stream section (name + URL form).
• MediaPlayerController.playVideo / playTrack: send POST /play to the scoped room or "default" when no session is active.
• startDefaultPolling(): Media Panel starts 1.5s state polling on appear so Now Playing card reflects server state without a voice session.
• pollState now logs failures instead of silently swallowing them.
• absolute() helper prefixes server base URL for relative art paths.
• Stream URL field: .textInputAutocapitalization(.never) to prevent "Https://" capitalization.

Verified in simulator (log + UI + server evidence):

• Search: typed "toes", tapped Find → POST /media/api/search 200 → results rendered with "in library" tag on matching track.
• Play (library): tapped Play on "Toes" → POST /media/api/play 200 → state desired: playing confirmed via curl. AVPlayer attempted to decode webm and failed (-11828, expected platform limitation from Phase 5).
• Delete: tapped Remove on "Toes" → confirmation alert "Remove from library?" → tapped Remove → DELETE /media/api/tracks/XMh9vU4g0jc 200 → library refreshed to show only Sheryl Crow.
• Add Stream: entered name + URL, tapped Add → POST /media/api/streams 400 (server rejected test URL) → error message "Could not add stream: Media API request failed with HTTP 400. url must be http(s)" displayed in red.
• Polling without session: confirmed via AVPlayer webm decode error in logs (state was fetched and reconciled even though no voice session was active).

Phase 7 status (2026-08-30 ~12:50)

Done:

• TaskStore (Sources/LiveKit/TaskStore.swift): @MainActor ObservableObject. Folds task_event payloads into TaskItem entries keyed by taskId — created (from nested task object or description), step (appends {role, content}, pending→running), completed (result + completedAt), failed (error). sortedTasks by createdAt desc; activeCount = pending+running (badge); clear().
• ControlChannel.handleTaskEvent fixed to match the real wire format: agent publishes {type:"task_event", task_id, event: created|step|completed|failed, ...} — discriminator is event, not type (old code never matched). Parses nested task.description/task.created_at. Added JSONValue.doubleValue.
• VoiceSession: owns taskStore; .taskEvent in handleControlEvent → taskStore.apply.
• TaskPanelView (Sources/Features/Tasks/TaskPanelView.swift): cards sorted desc, status dot (gray/blue/green/red), description header, steps list with role labels + web-mirrored colors (thinking=blue, tool_call=indigo, tool_result=muted, text=primary), green result block, red error block. Empty state "Nothing running / Ask Hope to research or compare something."
• Toolbar: tray.full button in ConversationView with red count badge when activeCount > 0; pushes TaskPanelView (NavigationLink, matching the media panel pattern).
• DEBUG-only "Sample" toolbar button in the panel injects two synthetic tasks (one completed, one failed) for UI verification.

Verified in simulator — REAL events (primary):

• Start → ready (text mode) → composer: "Research the history of jazz music in a background task…" → send.
• Agent published 8 real task_events on topic tasks (created + 6 steps incl. thinking/tool_call/tool_result/text + completed). Logs: [dev.shane.voice:tasks] task created id=738e422c steps=0 … task completed id=738e422c steps=6.
• Panel opened from toolbar shows the real card: description, all 6 role-styled steps (web_search tool_call visible), green result block with full jazz-history summary. Badge showed "1" while running (activeCount).

Verified — SYNTHETIC events (secondary, via DEBUG Sample button):

• Injected sample-a (created→2 steps→completed) and sample-b (created→step→failed); both cards rendered with correct status dots, error block ("Search backend timed out after 30s"), and badge count.

Notes:

• Screenshot captured at /tmp/taskpanel.png (model can't view images; verified via UI snapshot text + logs instead).
• No server/agent changes needed — the task dispatch pipeline already worked; only the iOS client was missing.

Phase 8 status (2026-08-30 ~13:40)

Done:

• Stale-room guards: sessionGeneration in VoiceSession, pollGeneration in MediaPlayerController. All async callbacks check generation before mutating state. RoomDelegate methods check self.room === room. Prevents post-disconnect mutations.
• Audio interruption: handleAudioInterruption in VoiceSession (mute mic on begin, resume on end if shouldResume). MediaPlayerController pauses/resumes AVPlayer on interruption.
• Error states: isReconnecting published property + reconnect banner in ConversationView. stateError published property + error banner in MediaPanelView.
• Accessibility: OrbView accessibilityLabel/Value (maps state to "Listening", "Thinking", etc.). ConversationView: Send, End, Tasks, Media, Settings labels. MediaPanelView: transport controls + refresh library labels.
• Server URL change fix: Changed session from @ObservedObject to @State in ConversationView so it can be reassigned inside async closures. Added .onReceive(session.objectWillChange) for observation. .onChange(of: settings.serverURL) disconnects old session and creates a new one when idle.

Verified in simulator:

• Build: zero errors, zero warnings.
• App launches cleanly.
• Server URL change (invalid → valid) triggers session recreation without crash.
• Connect/Disconnect/Reconnect matrix: Start → Ready → Hang up → Start again (3x). No crashes, state resets cleanly.
• Logs confirm no post-disconnect mutations (stale callbacks guarded).

Web parity checklist (vs /Users/shane/dev/hope-voice-api/web/app.js):

• Token auth + LiveKit connect
• Agent wait + agent_ready
• Voice/model pickers + sync-on-join
• Text mode toggle + barge-in hint
• Transcript list + history
• Media player (poll, reconcile, transport, seek)
• Media search + choices + library
• Background tasks panel
• Stale-room guards (=== room checks)
• Audio interruption handling
• Error states (reconnect, mic error, media error)
• Accessibility labels
• Web-only: drag-to-seek on waveform (iOS uses slider — acceptable)
• Web-only: keyboard shortcuts (N/A on iOS)

Project status (updated 2026-08-31)

All 10 phases implemented. Simulator verification covers the core paths listed below, but full web parity is not yet complete — several areas remain pending device qualification, server-side changes, or a separate UI redesign task.

Verified in simulator (historical + current): token auth + LiveKit connect (with retry), agent_ready, state machine driven by remote audio, text-mode transcripts (user + agent finals), voice/model pickers with sync-on-join + persistence, media reconciliation/UI/Range proxy/polling/reporting, media search/library/delete/add-stream, background tasks panel with real task_events, stale-room guards, error states, accessibility labels.

Pending device qualification (see docs/DEVICE_TESTING.md): mic capture with real speech (sim returns -4010), live listening/barge-in states from local audio, wake lock on physical screen-off, audible media playback, audio interruption behavior, mic-denied fallback UX.

Pending / blocked:

• UI redesign (Hope visual identity, agent visualizer bars, app icon, reduced-motion, transcript VoiceOver semantics) — specified in ui-ux-spec.md, not yet implemented. This is a separate scoped task, not part of the current correctness pass.
• Supported-format media playback — library tracks are webm; AVPlayer can't decode it (-11828). Server must serve m4a/aac/mp3 for audible playback on iOS.
• Cert pin rotation — the self-signed cert regenerates at container start, breaking the SHA-256 pin in TLSPinning.swift. Re-export the hash after each server restart (README).

Correctness corrections (iteration 01–04): transcript dedupe symmetry + time-bounding, terminal failure cleanup, text-mode mic invariant, media interruption pause/resume, HTTP 206 total length, resource loader hardening, TLS pin scoping, conversation state inference alignment, stale media publish guards, server config semantics, visible history/clear, model picker copy. See .artifacts/ for detailed reports.

Subagent notes

• Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
• Watch for loops (repeated identical commands) — kill and redirect if seen
• Don't let subagents loop reading SDK source; compile instead. SDK checkout: ~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/DerivedData/Voice-38a9ac34e5c8/SourcePackages/checkouts/client-sdk-swift