From e63e53ee747f1c9b5ddcfee47bd67e9bac1c9e22 Mon Sep 17 00:00:00 2001 From: Shane Maynard Date: Sun, 30 Aug 2026 09:41:04 -0400 Subject: [PATCH] docs: README + device testing checklist (Phase 9) --- PLAN.md | 23 +++++++++++- README.md | 79 ++++++++++++++++++++++++++++++++++++++++++ docs/DEVICE_TESTING.md | 59 +++++++++++++++++++++++++++++++ 3 files changed, 160 insertions(+), 1 deletion(-) create mode 100644 README.md create mode 100644 docs/DEVICE_TESTING.md diff --git a/PLAN.md b/PLAN.md index 60befbd..5e39fe3 100644 --- a/PLAN.md +++ b/PLAN.md @@ -28,7 +28,7 @@ Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via X - [x] Phase 6: media search, choices, library panel (commit cbf1830) - [x] Phase 7: background tasks panel (commit af296da) - [x] Phase 8: polish/hardening/parity pass (commit pending) -- [ ] Phase 9: docs & handoff +- [x] Phase 9: docs & handoff (commit pending) ## Phase 2 status Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received. @@ -182,6 +182,27 @@ Web parity checklist (vs /Users/shane/dev/hope-voice-api/web/app.js): - [ ] Web-only: drag-to-seek on waveform (iOS uses slider — acceptable) - [ ] Web-only: keyboard shortcuts (N/A on iOS) +## Project status (final, 2026-08-30) + +All 10 phases complete. Full web parity achieved and verified in simulator to the extent +the environment allows. + +**Verified in simulator:** token auth + LiveKit connect (with retry), agent_ready, state +machine driven by remote audio, text-mode transcripts (user + agent finals), voice/model +pickers with sync-on-join + persistence, media reconciliation/UI/Range proxy/polling/ +reporting, media search/library/delete/add-stream, background tasks panel with real +task_events, stale-room guards, error states, accessibility labels. Zero build warnings. + +**Device-only (see docs/DEVICE_TESTING.md):** mic capture with real speech (sim returns +-4010), live listening/thinking/barge-in states from local audio, wake lock on physical +screen-off, audible media playback, audio interruption behavior, mic-denied fallback UX. + +**Open server-side items (not app bugs):** +1. iOS-decodable media format — library tracks are webm; AVPlayer can't decode it (-11828). + Server must serve m4a/aac/mp3 for audible playback on iOS. No client change needed. +2. Cert pin rotation — the self-signed cert regenerates at container start, breaking the + SHA-256 pin in TLSPinning.swift. Re-export the hash after each server restart (README). + ## Subagent notes - Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md - Watch for loops (repeated identical commands) — kill and redirect if seen diff --git a/README.md b/README.md new file mode 100644 index 0000000..c8eab9f --- /dev/null +++ b/README.md @@ -0,0 +1,79 @@ +# Voice — iOS client for Hope Voice + +Native SwiftUI iOS app (iOS 17+) that talks to the [Hope Voice](https://github.com/) agent: +full-parity voice/text conversation, transcript history, voice & model pickers, media player, +and a background-tasks panel. Mirrors the web UI of `hope-voice-api`; no server-side changes. + +## Prerequisites + +- Xcode 16+ (Swift 6), iOS 17 simulator or a real iPhone on the LAN. +- [xcodegen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`). +- The `voice` container running on Unraid (LiveKit + token server + media API). +- Mic: the simulator mic is unreliable in this environment (CoreAudio `-4010`, 0 input + channels) — voice mode needs a real device. If the simulator was wiped, grant permission + first: `xcrun simctl privacy booted grant microphone dev.shane.voice`. + +## Build & run + +```sh +xcodegen generate # regenerate Voice.xcodeproj after file changes +xcodebuild -project Voice.xcodeproj -scheme Voice_iOS \ + -destination 'generic/platform=iOS Simulator' build +``` + +The scheme is **`Voice_iOS`** (not `Voice`). Bundle id: `dev.shane.voice`. +To run in the simulator, use Xcode or `xcrun simctl` with the built app. + +## Server configuration + +Default base URL is `https://192.168.86.2:9178` (see `Sources/App/SettingsStore.swift`). +Change it at runtime in **Settings → Server URL** (the app disconnects and recreates the +session when idle). The app derives LiveKit signaling and media endpoints from this base URL. + +### TLS cert pinning + +The deployment uses a self-signed cert that is **regenerated at container start**, so the +pinned SHA-256 in `Sources/Networking/TLSPinning.swift` breaks on every cert rotation. +Re-export after a server restart: + +```sh +openssl s_client -connect 192.168.86.2:9178 /dev/null \ + | openssl x509 -outform der | openssl dgst -sha256 +``` + +Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token/media calls +fail with a pin mismatch (not a network error). + +## Features + +- **Conversation** — LiveKit room with the Hope agent; voice mode (mic + TTS) and text mode + (composer), toggleable and persisted. State machine (idle/connecting/listening/thinking/ + speaking) driven by audio levels, orb animation, mic meter, barge-in hint. +- **Transcripts** — partials updated in place, finals with timestamps, delivery-marker + stripping, last-50 history persisted to UserDefaults, clear button. +- **Voice & model pickers** — 13 Azure TTS voices; LLM models from the agent's `models` + message. Both persist and sync on join (`set_voice`/`set_model`). +- **Media player** — now-playing card (art/title/channel/status), transport (play/pause, + ±10 s, stop), seek + volume sliders, search with "in library" tags, agent-offered choices, + library list (play/remove), add-stream form. AVPlayer-backed with a pinned-TLS Range proxy; + 1.5 s `/state` polling and full web-parity reconciliation/reporting. +- **Background tasks panel** — live `task_event` cards (steps, result, error) with an active + count badge in the toolbar. + +## Known limitations + +- **Simulator mic**: CoreAudio returns `-4010` (no usable input source) in this environment; + voice capture must be verified on a real device. Text mode works in the simulator. +- **webm media**: iOS AVPlayer cannot decode webm/opus (`-11828`). Library tracks from yt-dlp + are webm, so audible playback requires the server to serve m4a/aac/mp3. Everything up to + decode (pinning, Range proxy, reconciliation, UI) is verified; no client change needed once + a supported format exists. +- **Cert pin rotation**: see above — re-export the pin after each server cert regeneration. +- **LiveKit signaling** uses plain `ws://192.168.86.2:7880` (not wss) because the Swift SDK + cannot trust the self-signed cert on its WebSocket; token + media still go over pinned TLS. + +## Docs + +- [PLAN.md](PLAN.md) — phase-by-phase progress, verified facts, open items. +- [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md) — real-device qualification checklist. +- [UPDATE_PATH.md](UPDATE_PATH.md) — original plan & protocol reference (read-only). diff --git a/docs/DEVICE_TESTING.md b/docs/DEVICE_TESTING.md new file mode 100644 index 0000000..cb07fd6 --- /dev/null +++ b/docs/DEVICE_TESTING.md @@ -0,0 +1,59 @@ +# Device testing checklist + +Simulator verification covered connect/disconnect, transcripts (text mode), media +reconciliation/UI, tasks panel, and the hardening pass. The items below need a **real +iPhone on the LAN** (`https://192.168.86.2:9178`) because mic input, WebRTC capture, +and audio session behavior don't work in this simulator environment (CoreAudio `-4010`, +0 input channels). + +Setup: build with Xcode for a physical device (or `xcodebuild ... -destination`), install, +ensure the server cert pin is current (see README — re-export after any server restart). + +## 1. Mic input with real speech + +- [ ] Start → agent joins → mic enables **without** `-4010`. +- [ ] Speak: orb/status transitions to **listening**; mic meter moves with voice level. +- [ ] Stop speaking, wait >2 s while agent is silent → **thinking** chip appears. +- [ ] Agent replies: state goes to **speaking** (remote audio drives it), TTS audible. +- [ ] User transcript finals arrive in the list with correct role/timestamps. + +## 2. Barge-in + +- [ ] While the agent is speaking, start talking → conversation continues without a + "barge-in" glitch; state returns to listening/thinking as expected. +- [ ] Text mode: barge-in hint line visible while in text mode. + +## 3. Wake lock (screen off) + +- [ ] During an active conversation, press the side button (screen off). +- [ ] Screen stays on / auto-lock is suppressed for the duration of the session + (`isIdleTimerDisabled`); after hang-up, auto-lock returns. + +## 4. Audible media playback + +- [ ] Requires a track in an iOS-decodable format (m4a/aac/mp3) — current library tracks + are webm and will fail with `-11828` until the server serves a supported format. +- [ ] Play a library track: audio is audible, now-playing card shows art/title/status. +- [ ] Pause / ±10 s / stop / seek all work; agent mics off while playing + (`media_state` published), mics back on when stopped. +- [ ] End conversation while music was playing before → playback resumes. + +## 5. Audio interruption (phone call / Siri) + +- [ ] Mid-conversation, trigger an interruption (incoming call or "Hey Siri"). +- [ ] Mic is muted on interruption begin; conversation/media pause cleanly. +- [ ] On end (with `shouldResume`), mic and media resume automatically. +- [ ] Media panel: AVPlayer pauses on interruption and resumes after. + +## 6. Mic permission denied fallback + +- [ ] Deny microphone permission in iOS Settings for the app, relaunch. +- [ ] Start → session reaches **ready** despite mic failure (non-fatal); orange error + banner shows the mic error; text mode remains fully usable. + +## 7. General device sanity + +- [ ] Connect / hang-up / reconnect matrix (3×) — no crashes, state resets cleanly. +- [ ] Change server URL in Settings while idle → session recreates without crash. +- [ ] Reconnect banner appears on network loss mid-call; recovers on return. +- [ ] Voice/model picker changes take effect live on the agent and survive relaunch.