hope-voice-api/ios/UPDATE_PATH.md

UPDATE_PATH — iOS App for Hope Voice

Branch: feat/ios-app (created from empty repo state)

Goal: a native iOS app that is full parity with the web UI of /Users/shane/dev/hope-voice-api, connecting to the Unraid deployment at https://192.168.86.2:9178. The server side is NOT modified in this project — the app reuses the existing token_server.py endpoint and media API exactly as the web UI does.

Decisions (confirmed)

Decision	Choice
Scope	Full parity with web UI: voice chat, transcripts, text mode, voice picker, model picker, background tasks panel, media player + library
LiveKit client	livekit-client-swift (official Apple SDK via SPM)
Auth	Reuse POST /token on the existing token server (no new endpoint, no secrets in the app)
Server URL	Configurable in Settings, default https://192.168.86.2:9178
TLS	ATS exception for the LAN host + pinned self-signed cert (NSPinnedDomainKeys)

What we are replacing / building against

The "voice api" being replaced is whatever a stock LiveKit demo would use (devkey/devsecret minted on-device, hardcoded URLs). It is replaced by:

1. Token endpoint — POST {base}/token (nginx → token_server.py :8091).

   • Request body is ignored server-side; response: { "token": "<JWT>", "room": "voice-shane-<hex8>", "user": { "id", "display_name" } }
   • The JWT carries a roomConfig.agents claim, so the agent is dispatched when we join. Joining with a self-minted token would get no agent — this endpoint is mandatory, not optional.
   • Room name is unique per call (uuid suffix). It scopes media-server player state; send it as room in every /media/api/ call.

2. LiveKit signaling — {base}/livekit/ (wss), proxied to :7880. Web UI connects to the same origin: wss://192.168.86.2:9178/livekit.

3. Media API — {base}/media/api/* (FastAPI, :8093) + static /media/files/, /media/thumbs/.

Protocol reference (from web/app.js + agent/agent.py)

Data channel messages (LiveKit publishData, all reliable)

Direction	Topic	Payload	Meaning
app → agent	voice-control	{"type":"set_voice","voice":<azure voice id>}	live TTS voice change
app → agent	voice-control	{"type":"set_model","model":<id>}	live LLM model change; agent acks with model_changed
app → agent	voice-control	{"type":"get_models"}	request model list (agent replies models)
app → agent	conversation-control	{"type":"set_mode","mode":"voice"|"text"}	interaction mode; text mode mutes mic server-side
app → agent	media-control	{"type":"media_state","state":"playing"|"paused"|"stopped","track_id":...}	tells agent to disable/enable its mic (music ↔ conversation exclusivity)
agent → app	voice-control	{"type":"models","models":[...],"selected":...} / {"type":"model_changed","model":...}	model list + applied-model ack
agent → app	transcript	{"type":"transcript","role":"user"|"agent","text":...}	final transcripts (secondary source)
agent → app	any	{"type":"task_event", ...}	background task lifecycle: created/step/completed/failed with task_id, description, role, content, result, error
agent → app	—	{"type":"agent_ready"}	agent pipeline ready (wait for this before enabling mic)

Agent identity: participant kind .agent or identity "voice-assistant". Scan remoteParticipants after connect AND watch participantConnected (agent may pre-exist in the room).

Text chat input: localParticipant.sendText(text, topic: "lk.chat") (the web UI's text-mode path; the agent consumes it as a user turn).

Transcripts — two sources

1. Primary: LiveKit built-in transcription stream on track source .transcription (lk.transcription). In livekit-client-swift this arrives as AudioStreamTrack with source transcription; register a text-stream handler / read the track's TextStream. Word-by-word partials update the same row in place until final. Role is derived from which participant owns the transcribed track (attribute lk.transcribed_track_id).
2. Secondary: transcript data messages above.

UI must dedupe: when a final transcript arrives, replace any partial row for the same role.

Media API (all under {base}/media/api/, JSON)

Endpoint	Method	Body / Query	Notes
/state	GET	?room=<livekit room>	Returns {track, desired: "playing"|"paused"|"stopped", pending: [jobs], choices: [], seek_hint: {ts, position}}. Poll every 1.5 s.
/report	POST	{room, state, track_id, position, duration?}	Browser/app is the player; report real playback back.
/play	POST	{room, video_id?, title?, channel?, track_id?, start_paused?}	Start a track (search result or library id).
/control	POST	{room, action: "pause"|"resume"|"stop"}	Desired-state control.
/search	POST	{room, query, limit}	→ {results: [{video_id, title, channel, duration, in_library}]}; library matches first.
/library	GET	—	→ {tracks: [{id, title, channel, duration, status, kind, art_url, audio_url}]}
/tracks/{id}	DELETE	—	Remove from library.
/streams	POST	{room, name, url}	Add live stream.
/stream/{track_id}	GET	—	Progressive/Range stream URL (audio element source).
/healthz	GET	—	Liveness.

Static: GET {base}/media/files/<file> and {base}/media/thumbs/<img> (Range supported; use as direct AVPlayer URLs).

Media reconciliation rules (port verbatim from app.js)

• App is the player (AVPlayer). Media server holds per-room desired state.
• On join: if a track was loaded before, carry it into the new room with start_paused: true and remember wasPlayingBeforeTalk.
• Poll /state; when track.id changes → load new AVPlayer item; when a previously-null audio_url appears (download finished growing) → wire src.
• desired == "playing" + paused → play; otherwise pause.
• Apply seek_hint once per ts.
• On play/pause/ended/timeupdate(≥5 s): /report; publish media_state on media-control so the agent mics off/on.
• Ending conversation: if wasPlayingBeforeTalk → /control {action:"resume"}.

Web UI feature inventory (parity checklist)

• Connect flow: token → join → wait for agent (10 s, retry once with fresh token after 15 s) → wait agent_ready (10 s) → enable mic / text mode
• State machine: idle / connecting / listening / thinking / speaking, inferred from local + remote audio levels; orb pulse animation
• Mic meter + agent audio visualizer bars
• Transcript list with partials, delivery-marker stripping ([appreciative] etc.), history persistence (last 50), clear button
• Voice / text mode toggle (persisted); text composer sends lk.chat
• Mute toggle
• Settings sheet: voice picker (13 Azure voices, persisted) + model picker (from models message, persisted; ack via model_changed)
• Background tasks panel: badge count, cards with steps/result/error
• Media panel: now-playing card (art, title, channel, status), play/pause, ±10 s, stop, seek bar, volume (persisted), search results, agent-offered choices, library list with delete, add-stream form, download progress
• Screen wake lock while connected (NSProcessInfo.beginActivity)
• Reconnect-on-disconnect handling; stale-room guards

App architecture

voice-ios/
├── UPDATE_PATH.md            ← this file
├── AGENTS.md
├── opencode.json             (existing)
├── VoiceApp.xcodeproj        (generated via XcodeGen or SPM-based project)
├── Package.swift             (SPM: livekit-client-swift)
└── Sources/
    ├── App/
    │   ├── VoiceApp.swift            @main, scene setup
    │   ├── AppRouter.swift           tab/root navigation
    │   └── SettingsStore.swift       UserDefaults-backed: serverURL, voice, model, mode, volume
    ├── Networking/
    │   ├── ServerConfig.swift        base URL parsing (https only), LiveKit wss derivation
    │   ├── TokenClient.swift         POST /token → TokenResponse(token, room, user)
    │   ├── MediaAPI.swift            typed client for all /media/api endpoints
    │   └── TLSPinning.swift          URLSessionDelegate pinning the self-signed cert (fallback when NSPinnedDomainKeys is insufficient for WebRTC)
    ├── LiveKit/
    │   ├── VoiceSession.swift        Room lifecycle: connect, agent discovery, agent_ready wait, retry-once reconnect, disconnect teardown
    │   ├── AgentMonitor.swift        agent participant detection (kind .agent / identity), audio level observation
    │   ├── TranscriptHandler.swift   transcription stream + transcript data messages → TranscriptEvent stream (partial/final, role)
    │   └── ControlChannel.swift      publishData helpers: set_voice, set_model, get_models, set_mode, media_state; inbound models/model_changed/task_event/agent_ready parsing
    ├── Features/
    │   ├── Conversation/
    │   │   ├── ConversationViewModel.swift   state machine (idle/connecting/listening/thinking/speaking), audio-level inference, mic toggle, mode switch
    │   │   ├── ConversationView.swift        orb + status + transcript list + composer
    │   │   ├── OrbView.swift                 pulse animation driven by levels
    │   │   └── TranscriptListView.swift      partials, delivery-marker stripping, history
    │   ├── Settings/
    │   │   ├── SettingsView.swift            server URL, voice picker, model picker, clear history
    │   │   └── ModelPickerView.swift         from agent-published list
    │   ├── Tasks/
    │   │   ├── TaskStore.swift               task_event → [TaskItem]
    │   │   └── TasksPanelView.swift          badge + cards
    │   └── Media/
    │       ├── MediaPlayerController.swift   AVPlayer + /state polling (1.5 s) + reconciliation + reporting
    │       ├── MediaAPIModels.swift          Track, Job, StateResponse, SearchResult…
    │       └── MediaPanelView.swift          now-playing card, transport, seek, search, library, add-stream
    └── Support/
        ├── DeliveryMarkers.swift             regex strip of [style] markers
        ├── AudioLevels.swift                 smoothing (0.6/0.4 EMA), thresholds (user 0.035, agent 0.02)
        └── WakeLock.swift                    beginActivity while session active
└── Resources/
    ├── Info.plist / entitlements             NSMicrophoneUsageDescription, ATS exceptions
    └── Assets.xcassets                       app icon (port favicon.svg)

UI framework: SwiftUI (single-window conversation UI + sheets for settings, tasks, media — mirrors the web's panel model). Audio rendering of remote TTS is handled by livekit-client-swift's Room audio engine; local mic via its capture. No manual WebRTC.

Key technical notes / risks

• ATS + self-signed cert. NSAppTransportSecurity: NSExceptionDomains["192.168.86.2"] = { NSExceptionAllowsInsecureHTTPLoads: false, NSIncludesSubdomains: true } plus pinning the cert via NSPinnedDomainKeys (bundle the .cer from https://192.168.86.2:9178, extract with openssl s_client). If LiveKit's WebRTC sockets bypass the pinned URLSession, fall back to a custom URLSessionDelegate in TLSPinning.swift that validates the peer cert against the bundled one. Cert is regenerated at container start (entrypoint.sh) — document re-export steps; pinning will break on cert rotation until the app ships an update or we move to a real cert. Mitigation option: keep a "trust this server" first-run flow that caches the SPKI hash in Keychain instead of hard-pinning.
• Audio session. AVAudioSession category .playAndRecord, mode .spokenAudio, options [.defaultToSpeaker, .allowBluetoothA2DP]; activate on connect, deactivate on disconnect. livekit-client-swift manages the engine but we own the session config.
• Agent dispatch timing. If the agent worker isn't up (~15 s after server start), join gets no agent and there is no retry — hence the web UI's reconnect-once flow; port it exactly (fresh token, fresh room name).
• Text stream in livekit-client-swift. The Swift SDK exposes AudioStreamTrack.textStream / data messages; verify partial-word behavior during Phase 2 spike. If transcription track source isn't delivered on this LiveKit version, the secondary transcript topic still gives final text — degrade gracefully (no word-by-word).
• AVPlayer vs progressive webm/opus. iOS plays m4a/AAC and mp3 natively; webm/opus may not play in AVPlayer. Downloaded tracks are raw bestaudio (often webm). Spike in Phase 5: if AVPlayer rejects webm, options are (a) accept the limitation + surface "not playable on iOS" for those tracks, (b) add an optional server-side transcode later (out of scope here), or (c) use a lightweight opus decoder. Decide at spike; default to (a).
• Background. Out of scope: no background audio continuation beyond the standard AVPlayer behavior while foregrounded; wake lock keeps screen on during conversation only.

Phases

Phase 0 — Project scaffold (0.5 day)

• XcodeGen spec (project.yml) or hand-rolled .xcodeproj; iOS 17+ target, SwiftUI lifecycle, single app "Voice" (bundle id dev.shane.voice).
• SPM dependency: livekit-client-swift (pin latest stable; verify the transcription-track API surface against the pinned version).
• Info.plist: NSMicrophoneUsageDescription, ATS exceptions + pinned cert resource, URL types none.
• Empty app boots to a placeholder screen; builds & runs in simulator and on device (mic needs real device for full testing — simulator mic works for capture tests).
• Exit: app installs on an iPhone on the LAN; Settings screen shows server URL field defaulting to https://192.168.86.2:9178.

Phase 1 — Connectivity & token (0.5 day)

• ServerConfig (parse base URL, derive wss://host/livekit, validate https).
• TokenClient: POST /token with URLSession using the pinned-cert delegate; decode TokenResponse; surface 503 user-file errors clearly.
• Connection test screen: "Get token" button → shows room name + user display name (proves TLS pinning + endpoint work end-to-end).
• Exit: on device, token fetch succeeds against the Unraid deployment; wrong URL / offline produce clean error states.

Phase 2 — LiveKit session spike (1 day) ← de-risk first

• VoiceSession: create Room, connect with token, observe events.
• Agent discovery: scan + watch for kind .agent / identity "voice-assistant"; wait for agent_ready data message; implement reconnect-once flow (fresh token on agent timeout).
• Mic: enable local mic after agent_ready; mute toggle.
• Remote audio: confirm TTS plays through the SDK's audio engine on device.
• Transcript spike: wire transcription track source + transcript topic; log partials vs finals. Decide degradation path if word-by-word unavailable.
• Exit: on a real iPhone, start → agent joins → speak → hear TTS back → see at least final transcripts; disconnect cleans up audio session.

Phase 3 — Conversation UI (1–2 days)

• ConversationViewModel state machine ported from app.js (inferState, EMA smoothing, thresholds), driven by local/remote audio levels from the SDK.
• Orb + status line + mic meter; thinking chip (>2 s); barge-in hint while speaking.
• Transcript list: partials in place, final replace, delivery-marker strip, history (UserDefaults JSON, last 50), clear button, timestamps.
• Voice/text mode toggle (persisted): text mode disables mic + sends set_mode; composer sends via sendText(topic:"lk.chat").
• Screen wake lock while connected; disconnect teardown path.
• Exit: full voice conversation on device with live UI states matching the web UI's behavior; text mode round-trips.

Phase 4 — Settings: voices + models (0.5 day)

• Voice picker sheet: 13 Azure voices from app.js VOICES; persist selection; send set_voice on change and on agent join (sync-on-join like the web UI).
• Model picker: listen for models message, render list, persist preference, send set_model, apply ack via model_changed; note state when not connected ("Start a conversation to load the model list").
• Exit: changing voice/model in the sheet takes effect live on the agent; choices survive relaunch.

Phase 5 — Media player (1–2 days) ← second de-risk spike inside

• MediaAPI client + models for all endpoints above.
• MediaPlayerController: AVPlayer item from audio_url; poll /state every 1.5 s; reconciliation rules ported verbatim (track swap, late audio_url wiring, desired-state apply, seek_hint once-per-ts); /report on play/pause/ended + ≥5 s while playing; publish media_state on media-control; enter-media-scope carry-over with start_paused: true; resume-after-talk.
• webm spike: load a downloaded webm track in AVPlayer on device; record result and apply decision (a/b/c from risks).
• Media panel UI: now-playing card (AsyncImage art, title, channel, status), transport (play/pause, ±10 s, stop), seek bar, volume slider (persisted), download progress row.
• Exit: play a library track; pause/skip/seek work and are visible to the agent (mic disables while playing); ending a conversation resumes playback.

Phase 6 — Media search, choices, library (1 day)

• Search form → results list with in_library tag; Play buttons → /play.
• Agent-offered choices rendered from state.choices.
• Library list: refresh on open, play (ready only), delete with confirm.
• Add-stream form (name + url) → /streams.
• Exit: full media panel parity with the web UI, including library management and stream addition.

Phase 7 — Background tasks panel (0.5 day)

• TaskStore: fold task_event messages into task items (created/step/ completed/failed), badge = active count.
• Panel: cards sorted by created_at desc, steps list with role styling, result/error blocks; open from toolbar like the web UI.
• Exit: ask Hope to research something; panel shows live steps and final result identical to the web UI.

Phase 8 — Polish, hardening, parity pass (1 day)

• Error states: token 503, agent never joins, media server down, network loss mid-call (reconnect banner), mic permission denied fallback.
• Stale-room guards on every async callback (port the === room checks).
• Audio session interruption handling (phone call, Siri) — resume or pause conversation cleanly.
• Accessibility: labels/traits matching the web UI's ARIA roles; Dynamic Type.
• Side-by-side parity test against the web UI on the same deployment for every checklist item above.
• Exit: parity checklist fully ticked; no console-level crash paths in the connect/disconnect/reconnect matrix.

Phase 9 — Docs & handoff (0.5 day)

• README.md: build, run, device requirements, cert re-pinning steps after server restart/rotation, how to point at another deployment.
• AGENTS.md updates (build/test commands, conventions).
• Commit history cleaned up per conventional commits.

Total estimate: ~7–8 focused days. Phases 2 and 5 contain the two real technical risks (transcription API surface; webm playback) and are sequenced early for that reason.

Verification strategy

• Every phase ends with a manual pass on a real iPhone on the LAN against https://192.168.86.2:9178 (mic + WebRTC don't work in simulator).
• XcodeBuildMCP for build/run/screenshot loops where useful; device testing via Xcode directly.
• No server-side changes are made to hope-voice-api; if a gap is found there (e.g., token endpoint needs an iOS-friendly response), it is documented as a follow-up ticket, not silently patched in this repo.

Out of scope (explicit)

• Login/user selection (server pins VOICE_DEFAULT_USER until the planned login flow; app just displays the returned user).
• Background audio / lock-screen controls beyond AVPlayer defaults.
• Server-side changes (transcode, new endpoints, real TLS cert).
• iPad/macCatalyst/Apple Watch targets.