chore: scaffold Voice iOS app (Phase 0)

XcodeGen project, SwiftUI iOS 17 target dev.shane.voice, SPM livekit-client-swift 2.16.0, mic usage + ATS LAN exception in Info.plist, placeholder conversation screen and Settings with server URL field.
2026-08-29 17:15:27 -04:00
commit 297fbaa19c
@@ -0,0 +1,344 @@
+# UPDATE_PATH — iOS App for Hope Voice
+
+Branch: `feat/ios-app` (created from empty repo state)
+
+Goal: a native iOS app that is **full parity with the web UI** of
+`/Users/shane/dev/hope-voice-api`, connecting to the Unraid deployment at
+`https://192.168.86.2:9178`. The server side is NOT modified in this project —
+the app reuses the existing `token_server.py` endpoint and media API exactly as
+the web UI does.
+
+## Decisions (confirmed)
+
+| Decision | Choice |
+|---|---|
+| Scope | Full parity with web UI: voice chat, transcripts, text mode, voice picker, model picker, background tasks panel, media player + library |
+| LiveKit client | `livekit-client-swift` (official Apple SDK via SPM) |
+| Auth | Reuse `POST /token` on the existing token server (no new endpoint, no secrets in the app) |
+| Server URL | Configurable in Settings, default `https://192.168.86.2:9178` |
+| TLS | ATS exception for the LAN host + pinned self-signed cert (`NSPinnedDomainKeys`) |
+
+## What we are replacing / building against
+
+The "voice api" being replaced is whatever a stock LiveKit demo would use
+(devkey/devsecret minted on-device, hardcoded URLs). It is replaced by:
+
+1. **Token endpoint** — `POST {base}/token` (nginx → token_server.py :8091).
+   - Request body is ignored server-side; response:
+     `{ "token": "<JWT>", "room": "voice-shane-<hex8>", "user": { "id", "display_name" } }`
+   - The JWT carries a `roomConfig.agents` claim, so the agent is dispatched
+     when we join. **Joining with a self-minted token would get no agent** —
+     this endpoint is mandatory, not optional.
+   - Room name is unique per call (uuid suffix). It scopes media-server player
+     state; send it as `room` in every `/media/api/` call.
+
+2. **LiveKit signaling** — `{base}/livekit/` (wss), proxied to :7880.
+   Web UI connects to the same origin: `wss://192.168.86.2:9178/livekit`.
+
+3. **Media API** — `{base}/media/api/*` (FastAPI, :8093) + static
+   `/media/files/`, `/media/thumbs/`.
+
+## Protocol reference (from web/app.js + agent/agent.py)
+
+### Data channel messages (LiveKit `publishData`, all reliable)
+
+| Direction | Topic | Payload | Meaning |
+|---|---|---|---|
+| app → agent | `voice-control` | `{"type":"set_voice","voice":<azure voice id>}` | live TTS voice change |
+| app → agent | `voice-control` | `{"type":"set_model","model":<id>}` | live LLM model change; agent acks with `model_changed` |
+| app → agent | `voice-control` | `{"type":"get_models"}` | request model list (agent replies `models`) |
+| app → agent | `conversation-control` | `{"type":"set_mode","mode":"voice"\|"text"}` | interaction mode; text mode mutes mic server-side |
+| app → agent | `media-control` | `{"type":"media_state","state":"playing"\|"paused"\|"stopped","track_id":...}` | tells agent to disable/enable its mic (music ↔ conversation exclusivity) |
+| agent → app | `voice-control` | `{"type":"models","models":[...],"selected":...}` / `{"type":"model_changed","model":...}` | model list + applied-model ack |
+| agent → app | `transcript` | `{"type":"transcript","role":"user"\|"agent","text":...}` | final transcripts (secondary source) |
+| agent → app | any | `{"type":"task_event", ...}` | background task lifecycle: `created`/`step`/`completed`/`failed` with `task_id`, `description`, `role`, `content`, `result`, `error` |
+| agent → app | — | `{"type":"agent_ready"}` | agent pipeline ready (wait for this before enabling mic) |
+
+Agent identity: participant kind `.agent` or identity `"voice-assistant"`.
+Scan `remoteParticipants` after connect AND watch `participantConnected`
+(agent may pre-exist in the room).
+
+Text chat input: `localParticipant.sendText(text, topic: "lk.chat")` (the web
+UI's text-mode path; the agent consumes it as a user turn).
+
+### Transcripts — two sources
+
+1. **Primary:** LiveKit built-in transcription stream on track source
+   `.transcription` (`lk.transcription`). In livekit-client-swift this arrives
+   as `AudioStreamTrack` with source `transcription`; register a text-stream
+   handler / read the track's `TextStream`. Word-by-word partials update the
+   same row in place until final. Role is derived from which participant owns
+   the transcribed track (attribute `lk.transcribed_track_id`).
+2. **Secondary:** `transcript` data messages above.
+
+UI must dedupe: when a final transcript arrives, replace any partial row for
+the same role.
+
+### Media API (all under `{base}/media/api/`, JSON)
+
+| Endpoint | Method | Body / Query | Notes |
+|---|---|---|---|
+| `/state` | GET | `?room=<livekit room>` | Returns `{track, desired: "playing"\|"paused"\|"stopped", pending: [jobs], choices: [], seek_hint: {ts, position}}`. Poll every 1.5 s. |
+| `/report` | POST | `{room, state, track_id, position, duration?}` | Browser/app is the player; report real playback back. |
+| `/play` | POST | `{room, video_id?, title?, channel?, track_id?, start_paused?}` | Start a track (search result or library id). |
+| `/control` | POST | `{room, action: "pause"\|"resume"\|"stop"}` | Desired-state control. |
+| `/search` | POST | `{room, query, limit}` | → `{results: [{video_id, title, channel, duration, in_library}]}`; library matches first. |
+| `/library` | GET | — | → `{tracks: [{id, title, channel, duration, status, kind, art_url, audio_url}]}` |
+| `/tracks/{id}` | DELETE | — | Remove from library. |
+| `/streams` | POST | `{room, name, url}` | Add live stream. |
+| `/stream/{track_id}` | GET | — | Progressive/Range stream URL (audio element source). |
+| `/healthz` | GET | — | Liveness. |
+
+Static: `GET {base}/media/files/<file>` and `{base}/media/thumbs/<img>` (Range
+supported; use as direct AVPlayer URLs).
+
+### Media reconciliation rules (port verbatim from app.js)
+
+- App is the player (AVPlayer). Media server holds per-room **desired** state.
+- On join: if a track was loaded before, carry it into the new room with
+  `start_paused: true` and remember `wasPlayingBeforeTalk`.
+- Poll `/state`; when `track.id` changes → load new AVPlayer item; when a
+  previously-null `audio_url` appears (download finished growing) → wire src.
+- `desired == "playing"` + paused → play; otherwise pause.
+- Apply `seek_hint` once per `ts`.
+- On play/pause/ended/timeupdate(≥5 s): `/report`; publish `media_state` on
+  `media-control` so the agent mics off/on.
+- Ending conversation: if `wasPlayingBeforeTalk` → `/control {action:"resume"}`.
+
+### Web UI feature inventory (parity checklist)
+
+- [ ] Connect flow: token → join → wait for agent (10 s, retry once with fresh
+      token after 15 s) → wait `agent_ready` (10 s) → enable mic / text mode
+- [ ] State machine: idle / connecting / listening / thinking / speaking,
+      inferred from local + remote audio levels; orb pulse animation
+- [ ] Mic meter + agent audio visualizer bars
+- [ ] Transcript list with partials, delivery-marker stripping
+      (`[appreciative]` etc.), history persistence (last 50), clear button
+- [ ] Voice / text mode toggle (persisted); text composer sends `lk.chat`
+- [ ] Mute toggle
+- [ ] Settings sheet: voice picker (13 Azure voices, persisted) + model picker
+      (from `models` message, persisted; ack via `model_changed`)
+- [ ] Background tasks panel: badge count, cards with steps/result/error
+- [ ] Media panel: now-playing card (art, title, channel, status), play/pause,
+      ±10 s, stop, seek bar, volume (persisted), search results, agent-offered
+      choices, library list with delete, add-stream form, download progress
+- [ ] Screen wake lock while connected (`NSProcessInfo.beginActivity`)
+- [ ] Reconnect-on-disconnect handling; stale-room guards
+
+## App architecture
+
+```
+voice-ios/
+├── UPDATE_PATH.md            ← this file
+├── AGENTS.md
+├── opencode.json             (existing)
+├── VoiceApp.xcodeproj        (generated via XcodeGen or SPM-based project)
+├── Package.swift             (SPM: livekit-client-swift)
+└── Sources/
+    ├── App/
+    │   ├── VoiceApp.swift            @main, scene setup
+    │   ├── AppRouter.swift           tab/root navigation
+    │   └── SettingsStore.swift       UserDefaults-backed: serverURL, voice, model, mode, volume
+    ├── Networking/
+    │   ├── ServerConfig.swift        base URL parsing (https only), LiveKit wss derivation
+    │   ├── TokenClient.swift         POST /token → TokenResponse(token, room, user)
+    │   ├── MediaAPI.swift            typed client for all /media/api endpoints
+    │   └── TLSPinning.swift          URLSessionDelegate pinning the self-signed cert (fallback when NSPinnedDomainKeys is insufficient for WebRTC)
+    ├── LiveKit/
+    │   ├── VoiceSession.swift        Room lifecycle: connect, agent discovery, agent_ready wait, retry-once reconnect, disconnect teardown
+    │   ├── AgentMonitor.swift        agent participant detection (kind .agent / identity), audio level observation
+    │   ├── TranscriptHandler.swift   transcription stream + transcript data messages → TranscriptEvent stream (partial/final, role)
+    │   └── ControlChannel.swift      publishData helpers: set_voice, set_model, get_models, set_mode, media_state; inbound models/model_changed/task_event/agent_ready parsing
+    ├── Features/
+    │   ├── Conversation/
+    │   │   ├── ConversationViewModel.swift   state machine (idle/connecting/listening/thinking/speaking), audio-level inference, mic toggle, mode switch
+    │   │   ├── ConversationView.swift        orb + status + transcript list + composer
+    │   │   ├── OrbView.swift                 pulse animation driven by levels
+    │   │   └── TranscriptListView.swift      partials, delivery-marker stripping, history
+    │   ├── Settings/
+    │   │   ├── SettingsView.swift            server URL, voice picker, model picker, clear history
+    │   │   └── ModelPickerView.swift         from agent-published list
+    │   ├── Tasks/
+    │   │   ├── TaskStore.swift               task_event → [TaskItem]
+    │   │   └── TasksPanelView.swift          badge + cards
+    │   └── Media/
+    │       ├── MediaPlayerController.swift   AVPlayer + /state polling (1.5 s) + reconciliation + reporting
+    │       ├── MediaAPIModels.swift          Track, Job, StateResponse, SearchResult…
+    │       └── MediaPanelView.swift          now-playing card, transport, seek, search, library, add-stream
+    └── Support/
+        ├── DeliveryMarkers.swift             regex strip of [style] markers
+        ├── AudioLevels.swift                 smoothing (0.6/0.4 EMA), thresholds (user 0.035, agent 0.02)
+        └── WakeLock.swift                    beginActivity while session active
+└── Resources/
+    ├── Info.plist / entitlements             NSMicrophoneUsageDescription, ATS exceptions
+    └── Assets.xcassets                       app icon (port favicon.svg)
+```
+
+UI framework: **SwiftUI** (single-window conversation UI + sheets for settings,
+tasks, media — mirrors the web's panel model). Audio rendering of remote TTS is
+handled by livekit-client-swift's `Room` audio engine; local mic via its
+capture. No manual WebRTC.
+
+### Key technical notes / risks
+
+- **ATS + self-signed cert.** `NSAppTransportSecurity`:
+  `NSExceptionDomains["192.168.86.2"] = { NSExceptionAllowsInsecureHTTPLoads: false, NSIncludesSubdomains: true }` plus pinning the cert via
+  `NSPinnedDomainKeys` (bundle the `.cer` from `https://192.168.86.2:9178`,
+  extract with `openssl s_client`). If LiveKit's WebRTC sockets bypass the
+  pinned URLSession, fall back to a custom `URLSessionDelegate` in
+  `TLSPinning.swift` that validates the peer cert against the bundled one.
+  Cert is regenerated at container start (entrypoint.sh) — document re-export
+  steps; pinning will break on cert rotation until the app ships an update or
+  we move to a real cert. **Mitigation option:** keep a "trust this server"
+  first-run flow that caches the SPKI hash in Keychain instead of hard-pinning.
+- **Audio session.** `AVAudioSession` category `.playAndRecord`, mode
+  `.spokenAudio`, options `[.defaultToSpeaker, .allowBluetoothA2DP]`; activate
+  on connect, deactivate on disconnect. livekit-client-swift manages the
+  engine but we own the session config.
+- **Agent dispatch timing.** If the agent worker isn't up (~15 s after server
+  start), join gets no agent and there is no retry — hence the web UI's
+  reconnect-once flow; port it exactly (fresh token, fresh room name).
+- **Text stream in livekit-client-swift.** The Swift SDK exposes
+  `AudioStreamTrack.textStream` / data messages; verify partial-word behavior
+  during Phase 2 spike. If transcription track source isn't delivered on this
+  LiveKit version, the secondary `transcript` topic still gives final text —
+  degrade gracefully (no word-by-word).
+- **AVPlayer vs progressive webm/opus.** iOS plays m4a/AAC and mp3 natively;
+  **webm/opus may not play in AVPlayer**. Downloaded tracks are raw bestaudio
+  (often webm). Spike in Phase 5: if AVPlayer rejects webm, options are
+  (a) accept the limitation + surface "not playable on iOS" for those tracks,
+  (b) add an optional server-side transcode later (out of scope here), or
+  (c) use a lightweight opus decoder. Decide at spike; default to (a).
+- **Background.** Out of scope: no background audio continuation beyond the
+  standard AVPlayer behavior while foregrounded; wake lock keeps screen on
+  during conversation only.
+
+## Phases
+
+### Phase 0 — Project scaffold (0.5 day)
+- XcodeGen spec (`project.yml`) or hand-rolled `.xcodeproj`; iOS 17+ target,
+  SwiftUI lifecycle, single app "Voice" (bundle id `dev.shane.voice`).
+- SPM dependency: `livekit-client-swift` (pin latest stable; verify the
+  transcription-track API surface against the pinned version).
+- Info.plist: `NSMicrophoneUsageDescription`, ATS exceptions + pinned cert
+  resource, URL types none.
+- Empty app boots to a placeholder screen; builds & runs in simulator and on
+  device (mic needs real device for full testing — simulator mic works for
+  capture tests).
+- **Exit:** app installs on an iPhone on the LAN; Settings screen shows
+  server URL field defaulting to `https://192.168.86.2:9178`.
+
+### Phase 1 — Connectivity & token (0.5 day)
+- `ServerConfig` (parse base URL, derive `wss://host/livekit`, validate https).
+- `TokenClient`: POST /token with URLSession using the pinned-cert delegate;
+  decode `TokenResponse`; surface 503 user-file errors clearly.
+- Connection test screen: "Get token" button → shows room name + user display
+  name (proves TLS pinning + endpoint work end-to-end).
+- **Exit:** on device, token fetch succeeds against the Unraid deployment;
+  wrong URL / offline produce clean error states.
+
+### Phase 2 — LiveKit session spike (1 day) ← de-risk first
+- `VoiceSession`: create `Room`, connect with token, observe events.
+- Agent discovery: scan + watch for kind `.agent` / identity
+  "voice-assistant"; wait for `agent_ready` data message; implement
+  reconnect-once flow (fresh token on agent timeout).
+- Mic: enable local mic after `agent_ready`; mute toggle.
+- Remote audio: confirm TTS plays through the SDK's audio engine on device.
+- **Transcript spike:** wire transcription track source + `transcript` topic;
+  log partials vs finals. Decide degradation path if word-by-word unavailable.
+- **Exit:** on a real iPhone, start → agent joins → speak → hear TTS back →
+  see at least final transcripts; disconnect cleans up audio session.
+
+### Phase 3 — Conversation UI (1–2 days)
+- `ConversationViewModel` state machine ported from app.js (`inferState`,
+  EMA smoothing, thresholds), driven by local/remote audio levels from the SDK.
+- Orb + status line + mic meter; thinking chip (>2 s); barge-in hint while
+  speaking.
+- Transcript list: partials in place, final replace, delivery-marker strip,
+  history (UserDefaults JSON, last 50), clear button, timestamps.
+- Voice/text mode toggle (persisted): text mode disables mic + sends
+  `set_mode`; composer sends via `sendText(topic:"lk.chat")`.
+- Screen wake lock while connected; disconnect teardown path.
+- **Exit:** full voice conversation on device with live UI states matching the
+  web UI's behavior; text mode round-trips.
+
+### Phase 4 — Settings: voices + models (0.5 day)
+- Voice picker sheet: 13 Azure voices from app.js `VOICES`; persist selection;
+  send `set_voice` on change and on agent join (sync-on-join like the web UI).
+- Model picker: listen for `models` message, render list, persist preference,
+  send `set_model`, apply ack via `model_changed`; note state when not
+  connected ("Start a conversation to load the model list").
+- **Exit:** changing voice/model in the sheet takes effect live on the agent;
+  choices survive relaunch.
+
+### Phase 5 — Media player (1–2 days) ← second de-risk spike inside
+- `MediaAPI` client + models for all endpoints above.
+- `MediaPlayerController`: AVPlayer item from `audio_url`; poll `/state`
+  every 1.5 s; reconciliation rules ported verbatim (track swap, late
+  audio_url wiring, desired-state apply, seek_hint once-per-ts); `/report`
+  on play/pause/ended + ≥5 s while playing; publish `media_state` on
+  `media-control`; enter-media-scope carry-over with `start_paused: true`;
+  resume-after-talk.
+- **webm spike:** load a downloaded webm track in AVPlayer on device; record
+  result and apply decision (a/b/c from risks).
+- Media panel UI: now-playing card (AsyncImage art, title, channel, status),
+  transport (play/pause, ±10 s, stop), seek bar, volume slider (persisted),
+  download progress row.
+- **Exit:** play a library track; pause/skip/seek work and are visible to the
+  agent (mic disables while playing); ending a conversation resumes playback.
+
+### Phase 6 — Media search, choices, library (1 day)
+- Search form → results list with `in_library` tag; Play buttons → `/play`.
+- Agent-offered choices rendered from `state.choices`.
+- Library list: refresh on open, play (ready only), delete with confirm.
+- Add-stream form (name + url) → `/streams`.
+- **Exit:** full media panel parity with the web UI, including library
+  management and stream addition.
+
+### Phase 7 — Background tasks panel (0.5 day)
+- `TaskStore`: fold `task_event` messages into task items (created/step/
+  completed/failed), badge = active count.
+- Panel: cards sorted by created_at desc, steps list with role styling,
+  result/error blocks; open from toolbar like the web UI.
+- **Exit:** ask Hope to research something; panel shows live steps and final
+  result identical to the web UI.
+
+### Phase 8 — Polish, hardening, parity pass (1 day)
+- Error states: token 503, agent never joins, media server down, network loss
+  mid-call (reconnect banner), mic permission denied fallback.
+- Stale-room guards on every async callback (port the `=== room` checks).
+- Audio session interruption handling (phone call, Siri) — resume or pause
+  conversation cleanly.
+- Accessibility: labels/traits matching the web UI's ARIA roles; Dynamic Type.
+- Side-by-side parity test against the web UI on the same deployment for every
+  checklist item above.
+- **Exit:** parity checklist fully ticked; no console-level crash paths in the
+  connect/disconnect/reconnect matrix.
+
+### Phase 9 — Docs & handoff (0.5 day)
+- README.md: build, run, device requirements, cert re-pinning steps after
+  server restart/rotation, how to point at another deployment.
+- AGENTS.md updates (build/test commands, conventions).
+- Commit history cleaned up per conventional commits.
+
+**Total estimate: ~7–8 focused days.** Phases 2 and 5 contain the two real
+technical risks (transcription API surface; webm playback) and are sequenced
+early for that reason.
+
+## Verification strategy
+
+- Every phase ends with a manual pass on a **real iPhone** on the LAN against
+  `https://192.168.86.2:9178` (mic + WebRTC don't work in simulator).
+- XcodeBuildMCP for build/run/screenshot loops where useful; device testing
+  via Xcode directly.
+- No server-side changes are made to hope-voice-api; if a gap is found there
+  (e.g., token endpoint needs an iOS-friendly response), it is documented as a
+  follow-up ticket, not silently patched in this repo.
+
+## Out of scope (explicit)
+
+- Login/user selection (server pins `VOICE_DEFAULT_USER` until the planned
+  login flow; app just displays the returned user).
+- Background audio / lock-screen controls beyond AVPlayer defaults.
+- Server-side changes (transcode, new endpoints, real TLS cert).
+- iPad/macCatalyst/Apple Watch targets.