hope-voice-api/PLAN.md

# Voice iOS — Plan & Progress

Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via XcodeBuildMCP.

## Setup (reference)
- Repo /Users/shane/dev/voice-ios, branch feat/ios-app
- xcodegen at /opt/homebrew/bin/xcodegen; regenerate after file changes: `xcodegen generate`
- Build: `xcodebuild -project Voice.xcodeproj -scheme Voice_iOS -destination 'generic/platform=iOS Simulator' build` (scheme is **Voice_iOS**, not Voice)
- Run/relaunch: xcodebuildmcp_build_run_sim or launch_app_sim (bundle dev.shane.voice, sim iPhone 17 Pro AFDA137E-1657-428E-BF01-E5CD7989772C)
- UI: snapshot_ui → tap elementRef. Read logs: `xcrun simctl spawn booted log show --last Nm --predicate 'process == "Voice"' --style compact | grep -E "io.livekit|dev.shane.voice"`
- Mic permission (if simulator wiped): `xcrun simctl privacy booted grant microphone dev.shane.voice`

## Server facts (verified)
- Base https://192.168.86.2:9178 (nginx, self-signed CN=192.168.86.2; cert regenerates daily-ish → pins break on rotation)
- POST /token → {token, room: voice-shane-<hex8>, user{id, display_name}}
- LiveKit host-network container `voice` on Unraid (ssh root@192.168.86.2): signaling 7880/tcp, RTC tcp 7881, udp 7882 — all reachable from Mac (nc verified)
- SDK cannot trust self-signed cert on its WebSocket → use plain ws://192.168.86.2:7880 (no /livekit path; SDK derives /rtc). Token+media only via nginx :9178 TLS (AVPlayer media needs local proxy in Phase 5)
- agent_ready on topic conversation-control; models on voice-control (plain string list); agent identity agent-*, kind .agent; lk.agent.session stream unhandled (harmless)
- TLSPinning.session() = custom URLSessionDelegate DER pin for our HTTP calls

## Phases
- [x] Phase 0: scaffold (commit 297fbaa)
- [x] Phase 1: ServerConfig/TokenClient/TLSPinning/ConnectionTest (commit 378b8cb)
- [x] Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
- [x] Phase 3: state machine, audio levels, wake lock, history (commit 5483e6e)
- [x] Phase 4: voice/model pickers with sync-on-join (commit ca2d26f)
- [x] Phase 5: media player — commit 0af944e (webm playback blocked by iOS AVPlayer; see status below)
- [x] Phase 6: media search, choices, library panel (commit cbf1830)
- [x] Phase 7: background tasks panel (commit 63f2634)
- [ ] Phase 8: polish/hardening/parity pass
- [ ] Phase 9: docs & handoff

## Phase 2 status
Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received.
UNCOMMITTED: all Phase 2 files + ServerConfig ws:// derivation + Info.plist NSAllowsLocalNetworking.

### CONNECT TIMEOUT BUG — FIXED (2026-08-30 ~07:15)
Root cause: stale `Room` instance after a timed-out connect left signaling/transport in a broken state; subsequent connects hung. Fix in VoiceSession.swift:
- `room` is now `private(set) var`, recreated fresh via `makeRoom()` (delegates re-attached) on each retry.
- `attemptConnect()` retries up to 2 extra times (3 total) on `.timedOut`: tears down the room (`teardownRoom()`) and reconnects with a new Room + fresh token.
- `makeConnectOptions()` sets explicit timeouts (socket 15s, primary/publisher transport 20s).
- Removed manual `AVAudioSession` config/deactivation — the SDK's `AudioSessionEngineObserver` (isAutomaticConfigurationEnabled=true by default) owns the session; our manual setCategory/setActive raced it.
Verified: token→room connected in <1s (retry fired once on a slow first attempt, then succeeded), agent_ready received, state=ready. 3/3 connect attempts now reach room-connected.

### REMAINING: mic enable -4010 (simulator env limitation, NOT app bug)
After state=ready, `setMicrophone(enabled:true)` fails with CoreAudio `-4010` (kAudioUnitErr_InvalidElement) on all 3 retries. Investigated thoroughly:
- TCC: `xcrun simctl privacy booted grant microphone dev.shane.voice` applied; still -4010 → not a permission issue.
- CoreAudio logs: input node reports **0 channel descriptions** and the engine associates with audio session `(0x0)` (null) → the simulator is not providing a usable mic source to this app in this environment.
- NSMicrophoneUsageDescription present in Info.plist; SDK auto-configures the session correctly.
Conclusion: simulator microphone hardware/source unavailable here. App code path is correct (mic enable retries, proper error surfacing). Needs verification on a real device or a simulator with a working mic source. Do NOT treat -4010 as an app regression.

### After connect bug fixed
- [x] Verify ready state reached (state=ready logged; UI transitions to conversation view)
- [x] Verify transcript arrival + disconnect cleanup (verified in sim via text mode: user+agent finals arrived, hang-up cleaned up state)
- [x] Fix Swift 6 warnings: ControlChannel @MainActor, VoiceSession closure, duplicate TranscriptHandler removed from ConversationView (build now warning-free)
- [ ] Commit Phase 2, mark done

## Phase 3 status
Done:
- `ConversationStateModel` (Sources/LiveKit/ConversationState.swift): ports app.js inferState — EMA smoothing `prev*0.6 + raw*0.4`, user threshold 0.035, agent threshold 0.02, speaking→thinking when both silent, thinking chip after 2s in thinking phase. Driven by `RoomDelegate.didUpdateSpeakingParticipants` (participant.audioLevel), ticked only while session ready.
- Mic failure is now NON-FATAL: after 3 retries the error is surfaced via `VoiceSession.micError` (orange banner) and state stays `.ready` — text mode still works (also matches Phase 8 mic-denied fallback).
- Wake lock: `isIdleTimerDisabled=true` on room connect, false on disconnect.
- Text mode toggle: enables → setMuted(true) + `set_mode:text`; disables → `set_mode:voice` + re-enable mic. Persisted via SettingsStore.textMode.
- Transcript list: per-row timestamps, delivery-marker strip (web DELIVERY_MARKERS regex), Clear button, history of last 50 finals persisted to UserDefaults JSON (`TranscriptHistory`, key "voice_history").
- Barge-in hint line shown while in text mode; mic meter bar under status line in voice mode.

Verified in simulator (log evidence):
- Start → token→connect(retry)→room connected→agent_ready→state=ready.
- Mic -4010 x3 retries → error logged, session stayed ready, banner shown (non-fatal).
- Text mode toggle → log `set_mode sent: text`; mic muted.
- Composer send → log `sendText lk.chat chars=24` → user transcript final arrived → agent TTS audio drove state machine `connecting -> speaking` (log `[dev.shane.voice:state]`) → agent transcript final "Hello! How can I assist you today?" arrived in list.
- Hang up → `speaking -> idle`, UI back to Start button.

Needs device verification:
- Live voice-driven states (listening while user speaks, thinking gap >2s chip, barge-in) — sim mic is silent (-4010), so local audio level never rises in sim; only remote (agent) audio drove the machine here.
- Mic meter movement with real speech; wake lock behavior on physical screen-off.

## Phase 4 status (2026-08-30 ~11:50)
Done:
- `Voices.swift`: 13 Azure voices from web app.js; default en-US-Ava:DragonHDOmniLatestNeural.
- `SettingsStore`: voiceID/modelID persisted to UserDefaults; invalid stored voice falls back to default.
- `VoiceSession`: sync-on-join on agent_ready sends set_voice + set_model (if known); selectVoice/selectModel send immediately and persist; models list + model_changed acks tracked (`@Published models`, `selectedModel`).
- `SettingsView`: VoicePickerView sheet (13 voices), ModelPickerView sheet (models from session, "Start a conversation..." when not connected).
- `ConversationView`: shows current voice/model in status header when ready.

### SENDTEXT BUG — FIXED (2026-08-30 ~11:47)
Root cause: ControlChannel used `localParticipant.sendText(for:)` (LiveKit DataStream protocol, v2.16). Agent listens on legacy data channel (`ctx.room.on("data_received")`), which never receives DataStream text → "ignoring text stream with topic 'voice-control', no callback attached" in agent logs; set_voice/set_model silently dropped, no model_changed ack.
Fix: `ControlChannel.publish` now uses `localParticipant.publish(data:options:)` with `DataPublishOptions(topic:, reliable: true)` — same path as web UI's `publishData`.
Verified (sim + agent docker logs, room voice-shane-7bf126cf):
- iOS: sync-on-join set_voice sent: en-US-JennyNeural; set_model sent: gemma-4-e4b → model_changed ack received and applied; models list count=30.
- Agent: "Switching TTS voice to en-US-JennyNeural" / "Voice updated" / "LLM model change: qwen2.5-vl-3b -> gemma-4-e4b".
- Persistence: relaunch → sync-on-join sent previously selected Jenny + gemma-4-e4b (survived UserDefaults).
Note: `sendText lk.chat` in VoiceSession is also DataStream-based and the agent has no lk.chat handler — text chat path is unverified server-side, out of scope here.

## Phase 5 status (2026-08-30 ~12:20)
Done:
- `MediaAPI` (Sources/Media/MediaAPI.swift): client for all 10 endpoints — /state, /report, /play, /control, /search, /library, /tracks/{id} DELETE, /streams, /stream/{track_id}, /healthz. Codable models (MediaTrack, MediaState, MediaPendingJob, MediaSeekHint, MediaReported, MediaChoice, MediaLibrary, MediaSearchResult/Response). Uses TLSPinning.session().
- `PinnedMediaResourceLoader` (Sources/Media/PinnedMediaResourceLoader.swift): AVAssetResourceLoaderDelegate. AVPlayer loads https natively (ignores registered URLProtocol subclasses) and can't trust the self-signed cert, so the asset uses a custom `pinned-media://` scheme; the delegate answers every data request by fetching the real https URL through TLSPinning.session(), forwarding Range headers and setting contentInformationRequest (contentType/contentLength/byteRangeAccessSupported).
- `MediaPlayerController` (Sources/Media/MediaPlayerController.swift): @MainActor ObservableObject, AVPlayer-backed. 1.5s poll of /state while a room is scoped; reconciliation ported from web app.js — track-id swap → new item; late audio_url → wire source without reload; desired=="playing"+paused → play else pause; seek_hint applied once per ts; /report on play/pause/ended + ≥5s periodic; publishMediaState via ControlChannel; enter-media-scope carry-over (start_paused, wasPlayingBeforeTalk); end-conversation resume. Transport controls (togglePlayPause ±10s seek, stop), volume from SettingsStore.
- `MediaPanelView` (Sources/Features/Media/MediaPanelView.swift): now-playing card (AsyncImage art, title, channel, status), transport buttons, seek slider, volume slider, pending download jobs.
- Wiring: VoiceSession.mediaController + enterMediaScope before room connect + endConversation on disconnect; ConversationView music.note toolbar button → MediaPanelView.

Verified in simulator (log + UI evidence):
- /healthz and /library reachable via curl through pinned TLS (200).
- Media panel reachable from toolbar; empty state "Nothing loaded" shown when no track.
- Reconciliation: POST /play into the app's room → poll picks up track → log `wiring source pinned-media://...` → delegate intercepts (log `fetching https://... range=bytes=0-1` then full range) → panel shows "Toes (Greatest Hits Version)" with transport controls. Track-id swap, desired-state apply, and reporting all exercised.
- Range proxy works: AVPlayer's bytes=0-1 probe + full-range fetch both answered through the pinned session (no cert error).

BLOCKED — webm playback (platform limitation, NOT app bug):
- The stream proxy delivers valid webm bytes (EBML header + "webm" string verified), but iOS AVPlayer cannot decode webm: item fails `AVFoundationErrorDomain -11828 Cannot Open / This media format is not supported`. Confirmed via UTType: org.webmproject.webm conformsToAudio=false (iOS supports m4a/aac/mp3/ogg/opus/flac/wav, not webm).
- Both library tracks are webm (server-side yt-dlp output). So end-to-end audible playback cannot be verified on iOS until the server offers an iOS-decodable format (e.g. add `-f "bestaudio[ext=m4a]/bestaudio" --remux-video` or serve .m4a/.aac alongside webm).
- Everything up to decode is verified working: TLS pinning, Range proxy, reconciliation, polling, reporting, UI. The app will play correctly the moment a supported-format track is available — no client change needed for that.

Needs server change to verify audible playback:
- [ ] Server: provide an iOS-decodable audio format (m4a/aac/mp3) for library tracks, then re-test playback in sim.

## Phase 6 status (2026-08-30 ~13:00)
Done:
- `MediaPanelView` expanded: search section (query + Find button, results with "in library" tag), choices section (agent-offered suggestions from state), library section (list with Play/Remove per track, refresh button), add-stream section (name + URL form).
- `MediaPlayerController.playVideo` / `playTrack`: send POST /play to the scoped room or "default" when no session is active.
- `startDefaultPolling()`: Media Panel starts 1.5s state polling on appear so Now Playing card reflects server state without a voice session.
- `pollState` now logs failures instead of silently swallowing them.
- `absolute()` helper prefixes server base URL for relative art paths.
- Stream URL field: `.textInputAutocapitalization(.never)` to prevent "Https://" capitalization.

Verified in simulator (log + UI + server evidence):
- Search: typed "toes", tapped Find → POST /media/api/search 200 → results rendered with "in library" tag on matching track.
- Play (library): tapped Play on "Toes" → POST /media/api/play 200 → state `desired: playing` confirmed via curl. AVPlayer attempted to decode webm and failed (-11828, expected platform limitation from Phase 5).
- Delete: tapped Remove on "Toes" → confirmation alert "Remove from library?" → tapped Remove → DELETE /media/api/tracks/XMh9vU4g0jc 200 → library refreshed to show only Sheryl Crow.
- Add Stream: entered name + URL, tapped Add → POST /media/api/streams 400 (server rejected test URL) → error message "Could not add stream: Media API request failed with HTTP 400. url must be http(s)" displayed in red.
- Polling without session: confirmed via AVPlayer webm decode error in logs (state was fetched and reconciled even though no voice session was active).

## Phase 7 status (2026-08-30 ~12:50)
Done:
- `TaskStore` (Sources/LiveKit/TaskStore.swift): @MainActor ObservableObject. Folds task_event payloads into TaskItem entries keyed by taskId — created (from nested `task` object or description), step (appends {role, content}, pending→running), completed (result + completedAt), failed (error). `sortedTasks` by createdAt desc; `activeCount` = pending+running (badge); `clear()`.
- `ControlChannel.handleTaskEvent` fixed to match the real wire format: agent publishes `{type:"task_event", task_id, event: created|step|completed|failed, ...}` — discriminator is `event`, not `type` (old code never matched). Parses nested `task.description`/`task.created_at`. Added `JSONValue.doubleValue`.
- `VoiceSession`: owns `taskStore`; `.taskEvent` in handleControlEvent → `taskStore.apply`.
- `TaskPanelView` (Sources/Features/Tasks/TaskPanelView.swift): cards sorted desc, status dot (gray/blue/green/red), description header, steps list with role labels + web-mirrored colors (thinking=blue, tool_call=indigo, tool_result=muted, text=primary), green result block, red error block. Empty state "Nothing running / Ask Hope to research or compare something."
- Toolbar: `tray.full` button in ConversationView with red count badge when activeCount > 0; pushes TaskPanelView (NavigationLink, matching the media panel pattern).
- DEBUG-only "Sample" toolbar button in the panel injects two synthetic tasks (one completed, one failed) for UI verification.

Verified in simulator — REAL events (primary):
- Start → ready (text mode) → composer: "Research the history of jazz music in a background task…" → send.
- Agent published 8 real task_events on topic `tasks` (created + 6 steps incl. thinking/tool_call/tool_result/text + completed). Logs: `[dev.shane.voice:tasks] task created id=738e422c steps=0 … task completed id=738e422c steps=6`.
- Panel opened from toolbar shows the real card: description, all 6 role-styled steps (web_search tool_call visible), green result block with full jazz-history summary. Badge showed "1" while running (activeCount).

Verified — SYNTHETIC events (secondary, via DEBUG Sample button):
- Injected sample-a (created→2 steps→completed) and sample-b (created→step→failed); both cards rendered with correct status dots, error block ("Search backend timed out after 30s"), and badge count.

Notes:
- Screenshot captured at /tmp/taskpanel.png (model can't view images; verified via UI snapshot text + logs instead).
- No server/agent changes needed — the task dispatch pipeline already worked; only the iOS client was missing.

## Subagent notes
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
- Watch for loops (repeated identical commands) — kill and redirect if seen
- Don't let subagents loop reading SDK source; compile instead. SDK checkout: ~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/DerivedData/Voice-38a9ac34e5c8/SourcePackages/checkouts/client-sdk-swift