hope-voice-api/docs/PROJECT_STATUS.md

# Project status and handoff

Last updated: 2026-09-19 (documentation review; no new runtime qualification).

Reusable implementation, testing, device and deployment lessons are in
[ENGINEERING_LEARNINGS.md](ENGINEERING_LEARNINGS.md). September 12 test/install
results below are historical evidence, not a September 19 health check.

## Authorized Hermes migration — 2026-10-01

The direct user request supersedes the historical Qwen/client-model direction below.
This repository now implements server-account login (`/api/auth/login`), bearer
verification (`/api/auth/me`), logout (`/api/auth/logout`), and authorized `/token`
bootstrap. Hermes is server-managed: model settings, pickers and control messages
were removed. No backend source, deployment, live endpoint or secrets were touched
in this iOS implementation.

- Bearers live in device-only, unlocked Keychain items scoped to canonical HTTPS
  origin; passwords are never persisted. Login accepts numeric Unix expiry and
  ISO8601 expiry, and ready/provisioning accounts.
- Logout and origin replacement invalidate auth work, cancel bootstrap and
  disconnect transport before removing the account UI. Each account/root gets
  fresh drafts/tasks and history scoped to origin + immutable user id; the global
  legacy `voice_history` is never imported.
- Signaling is same-origin WSS `/livekit` for all hosts, including IPv6/private
  addresses. Userinfo/query/fragment origins are rejected. There is no plaintext
  fallback and no trust-all delegate. HTTP retains the existing exact LAN pin;
  **the LiveKit SDK's separate WSS trust may require an installed trusted CA**.
  Self-signed WSS, actual login, multi-user server isolation, microphone capture,
  audible speech and interruptions require subsequent user hardware qualification.
- `assistant-progress` working/completed/failed events display request activity
  without polluting eventual normal transcripts. Payload logging is removed; SDK
  log level is error rather than release debug.
- Offline UI uses fake auth and memory token storage plus isolated preferences;
  bootstrap fixtures cannot contact a remote server. Live suite is excluded from
  this validation. `project.yml` generates sources and an explicit test scheme.

Verification: simulator build passed. Initial offline unit run: **50 passed, 0
failed** (5 new focused authentication tests), `/tmp/hermes-ios-unit2.xcresult`.
A scheme-less first test attempt failed before running tests; adding the generated
scheme fixed it. Combined offline run: **51 unit + 5 UI passed, 0 failures** at
`/tmp/hermes-ios-offline-final.xcresult`. Expanded run added the logout/account-switch
privacy UI test, which passed; one existing URL normalization assertion raced the
asynchronous save render and failed. It now waits for the saved confirmation before
asserting the normalized value. A subsequent accessibility-size Activity empty-state
assertion timed out after 5 seconds; its wait now allows 15 seconds. Both failed-run
bundles are retained (`/tmp/hermes-ios-offline-qualified.xcresult` and
`/tmp/hermes-ios-offline-release.xcresult`). Final focused re-verification:
**51 unit + 3 UI passed, 0 failures**, including accessibility-size navigation,
settings persistence/normalization and logout/account-switch transcript/task privacy,
`/tmp/hermes-ios-focused-final.xcresult`. Remaining three offline UI cases passed in
both expanded runs. Parent security review identified transient `/me` outages clearing
Keychain prematurely; fixed to clear only 401/403 and added a regression test.
Final source: **52 unit passed, 0 failures**, `/tmp/hermes-ios-unit-final.xcresult`.
All six offline UI cases have passed across the runs described above; the final
three focused cases passed together. No claim of one fully green expanded six-UI
run is made. These are client-only simulator claims, not live/deployment/hardware
qualification.

## Physical iPhone deployment — 2026-10-01

At the user's request, compiled the current Hermes account app with Xcode 26.4
using automatic development signing (`Voice_iOS`, Debug, generic iOS destination).
Build succeeded; artifact `/tmp/hope-voice-phone-build/Build/Products/Debug-iphoneos/Voice.app`,
build log `/tmp/hope-voice-phone-build.log`. Installed bundle `dev.shane.voice`
on the connected, paired iPhone 14 Pro Max (iOS 26.7.1, Developer Mode enabled).
Installation reported success. The first automatic launch was denied because the
iPhone was locked (`FBSOpenApplicationErrorDomain: Locked`). After the user unlocked
it, `devicectl device process launch` succeeded and launched `dev.shane.voice`.
No credentials/preferences were reset, no backend changes, and no physical speech
or authenticated WSS qualification is implied by successful installation.

## Square presence and certificate onboarding — 2026-10-01

User reported invalid server certificate at sign-in and requested the webapp square
presence as the primary native avatar. Replaced Lightwell rendering with an efficient
21×21 Canvas LED matrix, centered 220pt on welcome and 136pt above active transcript;
smaller accessibility sizes preserve controls/readability. Mic/listening and agent
speech levels drive separate patterns; Reduce Motion freezes motion/audio geometry.
Existing semantic theme and single accessible status are preserved.

Actual deployed leaf and SPKI pins match the app. Explicit live native HTTPS probe
passed on both simulator and physical iPhone against `/api/auth/me` (expected401,
no credentials). Physical native password POST reached server and returned401 for
John Doe; user confirmed deleting this test account, so no account was recreated.
No TLS bypass, plaintext fallback, or pin change was made. The reported original
login failure was not reproduced by this test; installed-version or transient cause
is not conclusively established. Admin full-auth test was interrupted by lock/lost
testmanager connection, not claimed successful.

LiveKit's SDK uses an independent system-trust WebSocket session. User approved a
public certificate-only installation profile, exact leaf SHA256
`07907abde3bb9c685265909927627d9ea4705d7e7b27056eefe06e10b192af46`.
Profile is served at `https://192.168.86.2:9178/hope-server.mobileconfig`, requires
manual installation and full trust in iPhone Settings. App login links it only
for the exact configured LAN host/port, and certificate errors include safe recovery.
An initial container-copy download disappeared after another backend deployment;
fixed by persistent cert mount and exact nginx alias; verified200/apple profile MIME,
private-key path404. Nothing installs trust automatically or weakens TLS checks.

Validation: 19 conversation tests passed (3 new LED tests), simulator screenshots
`/tmp/hope-led-welcome-ready.png` and `/tmp/hope-led-history-dark.png`. Complete
unit suite:58 passed, including two explicit-opt-in live TLS cases skipped in offline
run. Actual opted-in physical HTTPS test passed separately. New live tests use
environment-only credentials, never fixtures/source/argv. Updated signed phone
build passed (`/tmp/hope-square-phone-build.log`) and app installed/launch succeeded
on the iPhone14ProMax (`dev.shane.voice`, 18:00 local). Physical speech remains user qualification.

Follow-up: user installed profile/full trust but SDK still reported validation cert
invalid. Reproduced physically: exact pinned HTTPS passed; system URLSession HTTPS
failed -1202/-9807. Controlled A/B removed redundant `NSPinnedCAIdentities` in
XcodeGen (normal iOS trust, HTTPS/WSS and exact HTTP leaf pin remain unchanged).
System trust then passed on same phone, establishing ATS CA-only rule conflict,
not proven certificate validity-length rejection. Final physical native live test
**3 passed**: pinned HTTPS, system-trusted HTTPS, and admin login/me/bootstrap →
actual LiveKit SDK authenticated WSS connect/disconnect → logout; no microphone
enabled. Evidence `/tmp/hope-phone-wss-qualified.log`. John Doe remains deleted.
Signed corrected build passed `/tmp/hope-square-tls-fixed-build.log`.


## Current direction

Keep **Qwen 2.5**. The user explicitly canceled the earlier request to switch to
a ChatGPT subscription/device-token login. Do not revive that migration because
an old saved goal mentions it. Current work is the native iOS UI/UX redesign
and end-to-end qualification.

**2026-09-12: the embedded music player was removed** by user decision — the
feature is gone from the app (Media tab, `Sources/Media/`, `MediaPanelView`,
`MediaAPI`, `MediaPlayerController`, `PinnedMediaResourceLoader`, media tests,
and the `media_state` control publish). The backend media API still exists and
is untouched. The backend AAC compatibility patch (below) is now irrelevant to
the iOS app but remains deployed.

The **Nearlight** native redesign is implemented across Talk, Activity, Settings,
pickers, task details, and connection diagnostics. Current simulator qualification
is green, and GPT-6 Astra's final blocker re-verification returned **PASS**. The
final source also produced a signed generic-iOS build, and once the phone
reconnected the final build was installed and launched on the paired iPhone.
Hardware audio and interruption qualification remains open.

## Completed work

### Native interface and usability

- The app now has native **Talk** and **Activity** tabs sharing one session;
  changing tabs never ends a conversation.
- Implemented Astra's **Nearlight** direction: an adaptive plum/rosewood semantic
  palette, restrained rounded typography, shared spacing/radius/motion tokens, and
  the compact Lightwell aperture as the conversation state signature.
- Rebuilt Talk around a clear pre-connection welcome, Voice/Type choice, explicit
  start/end controls, compact active status, microphone meter, persistent Type
  draft, and a dock that keeps primary actions reachable.
- Reworked the transcript as a readable shared record with speaker/time metadata,
  live partials, copy/selection, explicit destructive confirmation, and measured
  follow-latest behavior. Restored history stays at the top; manual reading,
  selection, and VoiceOver row focus suspend following until **Latest**.
- Made connection and ready-session microphone failures visible without discarding
  the real error. Permission denial exposes an iPhone Settings recovery action;
  Type mode remains available.
- Reworked Settings around voice, mode, model, and connection. Server edits are
  validated and explicitly saved; normalization confirmation survives session
  replacement and clears on the next user edit.
- Improved Activity hierarchy, active-count badge, empty state, state-specific
  rows, and task detail presentation.
- Removed forced dark appearance; all redesigned surfaces support light/dark mode,
  Dynamic Type, Reduce Motion, increased contrast branches, and 44-point-or-larger
  interaction targets where applicable.

### Functional fixes

- Observed nested transcript, conversation-state, and task objects so asynchronous
  updates render.
- Preserved latest-wins server replacement while keeping an open Settings
  navigation stack and its success feedback intact.
- Preserved Type-mode microphone privacy, pending-message send gating, clean session
  teardown, transcript persistence/deduplication, and interruption ordering.
- All media-related client behavior was removed with the music player on 2026-09-12.

### Tests and tooling

- Follow-up commits added the Lightwell app icon (`7ac8a1c`) and regenerated the
  project to include its asset-catalog resource phase (`85169d6`). The qualification
  results below predate those follow-ups; do not describe them as a fresh full run
  of the current HEAD.
- Installed XcodeBuildMCP and its skills; used it for simulator build/run/test work.
- The current combined simulator run passed **50 tests, 0 failures**, with the
  opt-in live suite skipped once. Five offline UI scenarios cover recovery,
  settings validation/persistence and save feedback, transcript/task flows,
  restored-history reading position, and accessibility-size navigation.
- A post-focus-change targeted restored-history UI regression passed independently.
- DEBUG UI fixtures use isolated preferences; ordinary application preferences and
  the NAS are not modified by offline UI runs.
- GPT-6 Astra completed the design specification and four verification rounds. Its
  final scoped review found B1–B3 resolved, N1/N2 preserved, and no concrete
  remaining release blocker. Accepted visual/adaptive/accessibility polish remains
  documented as debt, not as exhaustive WCAG or VoiceOver qualification.

## Backend deployment and cross-repository dependency

SSH host: `[email protected]` (Unraid). App URL: `https://192.168.86.2:9178`.

- `voice` image: `hope-voice-api-voice:68f4c46-ios-audio`, host networking,
  restart policy `unless-stopped`. Includes LiveKit, voice agent, token service,
  media API, and nginx.
- LLM: `qwen2.5-vl-3b` through existing OpenWebUI at
  `http://192.168.86.2:3005/api`. Existing Azure speech configuration retained.
- `voice`, `open-webui`, and `local-vl-models` were present and healthy when the
  user suspected deletion. Do not recreate healthy services unnecessarily.
- Original library, certificates, and skills were retained under
  `/mnt/user/dev/hope-voice-api`; the dirty remote checkout was not overwritten.
- Separate deployment source: `/mnt/user/dev/hope-voice-deploy-68f4c46`.
- Stopped rollback container: `voice-before-ios-audio`.
- Backend AAC conversion preserves original WebM files and serves cached AAC with
  byte-range support. Browser/raw requests retain their original behavior.

**Important:** the backend patch and its test are in a different local repository:
`/Users/shane/dev/hope-voice-api/web/media_server.py` and
`/Users/shane/dev/hope-voice-api/tests/test_ios_audio.py`. They were still uncommitted
at the earlier deployment handoff; their current sibling-repository status has not
been rechecked in this documentation update. Committing/pushing the iOS repository
does **not** publish those files.
The deployed image contains the patch, but a future backend CI rebuild can lose it
unless it is committed and deployed through that repository's workflow. Inspect
that workflow before pushing: it may replace the live container.

Launch configuration and rollback details: [DEPLOYMENT.md](DEPLOYMENT.md).
Do not copy credentials or `.env` contents into documentation or commits.

## Verification performed

Project `Voice.xcodeproj`, scheme `Voice_iOS`, iPhone 17 Pro simulator
`AFDA137E-1657-428E-BF01-E5CD7989772C`.

- Current combined run: **50 passed, 0 failed, 1 live skip** in 104.9 seconds.
  Result bundle:
  `test_sim_2026-09-12T14-03-09-058Z_pid95846_2ca9da7e.xcresult`.
- Post-focus-change restored-history regression: **1 passed, 0 failed**.
  Result bundle:
  `test_sim_2026-09-12T14-05-21-070Z_pid98187_bd3e531a.xcresult`.
- A fresh isolated simulator launch showed restored history top-anchored with
  **Latest** visible; the UI regression then tapped Latest and verified the final
  event became visible.
- Astra's targeted isolated runtime check verified a padded changed-address save
  stays in Settings, normalizes the field, shows its checkmark and confirmation,
  and clears both only after the next user edit. Final verdict: **PASS**.
- iPhone 14 Pro Max `BF93D35E-0853-5470-A125-6EB0DB3F17AB` was paired, signed,
  installed, trusted, and launched successfully. The final Nearlight build
  (`/tmp/voice-ios-device-derived/Build/Products/Debug-iphoneos/Voice.app`,
  `dev.shane.voice`) was reinstalled and launched via `devicectl` after the phone
  reconnected. Free provisioning expires after seven days; reinstall to re-sign.
- Historical live NAS runs verified Qwen replies, background task completion, and
  reconnect. Backend integration remains 16 passed, 7 skipped, 2 deselected.

Exact historical qualifications: [VERIFICATION.md](VERIFICATION.md). Artifacts
live on this Mac under the XcodeBuildMCP workspace, not in Git.

## What remains, in order

1. **Qualify real audio:** microphone capture, TTS audibility, speaker/headphones,
   barge-in, Siri/phone interruptions, lock/background behavior,
   microphone-permission-denied fallback, and network-loss recovery. These require
   the user's speech/listening or a real interruption; simulator state tests do not
   prove them. Follow [DEVICE_TESTING.md](DEVICE_TESTING.md).
2. **Broader live regression:** exercise slow-network cancellation, network-loss
   recovery, repeated server replacement, and voice/model selection against the
   live agent.
3. **Manual accessibility/visual qualification:** inspect populated/error screens
   in light and dark mode, accessibility text sizes, narrow/landscape layouts, and
   a real VoiceOver focus session. Astra accepted the remaining presentation debt
   for this personal-app release; that is not proof of exhaustive conformance.
4. **Publish the backend compatibility source safely** in the sibling repository
   and ensure its normal deployment preserves mounts, certificates, and Qwen
   settings. This is lower priority because the iOS app no longer consumes the
   media API, but the deployed container still serves it for the web UI.

## Repository handoff

The iOS work and this report belong to branch `feat/ios-app`, remote `origin`
(`https://git.srmr.xyz/snowspeeder/hope-voice-ios.git`). Keep the Nearlight
implementation, tests, physical-device notes, and this handoff together when
committing. Check Git history and remote status for the final publish result.