Compare commits

...
16 Commits
Author SHA1 Message Date
snowspeeder 9bdaea2408 Document engineering learnings and refresh handoff guidance 2026-09-19 14:40:24 -04:00
snowspeeder 85169d651c project: regenerate to include asset catalog resource phase 2026-09-12 17:27:00 -04:00
snowspeeder 7ac8a1c2fa brand: add Hope app icon (Lightwell aperture)
- 1024px icon from web/brand/hope-icon.svg (forced-dark, grid-tuned)
- fix xcodegen: xcassets was never in the generated project (no
  resources phase) — move it into sources with buildPhase: resources
- fix iconset manifest: idiom universal (universal-platform rejected
  by actool), single 1024 universal/ios entry
2026-09-12 17:26:57 -04:00
snowspeeder 59d8182b6c docs: record final device install 2026-09-12 17:13:27 -04:00
snowspeeder b7cec80a99 fix: close Nearlight release blockers 2026-09-12 10:13:14 -04:00
snowspeeder 5577162594 Reapply N2 fix: disable Send until transport ready (lost in prior commit) 2026-09-12 08:38:08 -04:00
snowspeeder f6722a223f Fix regressions N1-N3 from re-verification
- Transcript: selection suspension clears when press ends; Latest
  re-enables follow fully (clears stale selection state)
- Send disabled until transport ready in pending Type composer
- Server URL: programmatic success-trim no longer erases its own
  saved-confirmation; only user edits clear it
2026-09-12 08:36:01 -04:00
snowspeeder afe37ad5f4 Fix reading-safety and accessibility majors from review
- Transcript: drag suspends auto-follow; long-press selection suspends
  following; Latest is a trailing outlined capsule that re-enables follow
- Lightwell: pulse via isolated TimelineView (2.4s triangle wave), pauses
  when offscreen; Reduce Motion uses static geometry (no level resizing)
- Increase Contrast: solid Rosewood lightwell fill, 2pt Rosewood selected
  outlines on mode buttons
- Haptics: success only on entering ready, error only on entering failure
2026-09-12 08:27:48 -04:00
snowspeeder f043d6b078 Fix major Nearlight deviations from design review
- Expose End conversation + readable draft during connecting/waiting
- Stop masking assistant speaking/thinking state in Type/muted modes
- Restrict failure diagnosis to real causes (no invented network text)
- Replace old blue tint with Rosewood everywhere
- Fix transcript short-content bottom-anchor (top-align short histories)
- Settings: visible Server URL label, focus on rejected save, edit clears
  saved confirmation, trim-aware unchanged detection
- Task detail: live store lookup with missing-task state
- Task rows: status text label restored
2026-09-12 08:20:43 -04:00
snowspeeder fa21c81787 wip: Nearlight redesign implementation 2026-09-12 08:14:06 -04:00
snowspeeder c76afbd11e docs: record verified device pairing, signing, install 2026-09-12 07:17:55 -04:00
snowspeeder ea9ec75631 Configure development team for device signing 2026-09-12 07:10:41 -04:00
snowspeeder c5db94658d Restore same-URL guard in replaceServer
Accidentally dropped during the Music tab removal edit; without it,
reapplying the current URL recreates the session and loses history.
2026-09-12 06:57:32 -04:00
snowspeeder ed329213de Remove embedded music player
The music feature is retired: delete the Media tab, MediaAPI,
MediaPlayerController, PinnedMediaResourceLoader, MediaPanelView, the
media_state control publish, media scope handling in VoiceSession, the
volume setting, and all media tests. App is now Talk + Activity tabs.

Backend media API in hope-voice-api is untouched (still serves the web UI).
2026-09-12 06:26:58 -04:00
snowspeeder 3620a2387b docs: update repo URL after rename to hope-voice-ios 2026-09-12 06:11:39 -04:00
snowspeeder 24f95639dc Redesign Hope iOS UI, harden playback, and document project handoff 2026-09-09 20:23:44 -04:00
34 changed files with 2211 additions and 2641 deletions

No files matched your search

+8
View File
@@ -1,3 +1,11 @@
# AGENTS.md
- On startup, read `docs/PROJECT_STATUS.md` before planning or changing this project. It is the current handoff report and records decisions, completed work, verification, deployment dependencies, and remaining tasks.
- Keep `docs/PROJECT_STATUS.md` updated as work progresses, especially after implementation, testing, deployment, or changes to blockers and user decisions. Distinguish verified results from assumptions, and update it before handing off or committing material changes.
- If using XcodeBuildMCP, use the installed XcodeBuildMCP skill before calling XcodeBuildMCP tools.
- Read `docs/ENGINEERING_LEARNINGS.md` before changing session lifecycle, transcripts, Settings, tests, signing, or deployment. Preserve its behavioral invariants unless the user explicitly changes the requirement.
- Follow the latest explicit user decisions: keep Qwen 2.5; the embedded music player was removed. Do not restore canceled features from historical goals, plans, or test reports.
- Treat `project.yml` as the project-generation source. Update it and regenerate with XcodeGen when adding/removing sources, tests, or assets; verify resources are in the generated build phase.
- Keep offline UI tests isolated from normal preferences and the NAS. Run live tests only deliberately, and distinguish unit/simulator evidence from physical audio qualification.
- Date verification claims and record their scope/artifacts. A healthy container, successful token request, review PASS, or installed app does not prove a working voice conversation.
- Before deploying or pushing backend changes, inspect the sibling repository and its automation. An iOS commit does not publish backend patches; preserve existing NAS data, certificates, dirty checkouts, and rollback options. Never put credentials in logs, reports, or commits.
+18 -22
View File
@@ -1,19 +1,19 @@
# Voice — iOS client for Hope Voice
Native SwiftUI iOS app (iOS 17+) that talks to the [Hope Voice](https://github.com/) agent:
voice/text conversation, transcript history, voice & model pickers, media player, and a
background-tasks panel. Mirrors the core web UI of `hope-voice-api`; no server-side changes.
voice/text conversation, transcript history, voice & model pickers, and a background-tasks
panel. Uses `hope-voice-api`.
> **Status:** Core functionality is implemented and verified in simulator to the extent the
> environment allows. Full device qualification (mic capture, audible media playback,
> interruption behavior) and the Hope visual redesign (`ui-ux-spec.md`) remain pending —
> environment allows. Full device qualification (mic capture, interruption behavior)
> remains pending. The native Talk/Activity redesign is implemented —
> see [PLAN.md](PLAN.md) and [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md).
## Prerequisites
- Xcode 16+ (Swift 6), iOS 17 simulator or a real iPhone on the LAN.
- [xcodegen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`).
- The `voice` container running on Unraid (LiveKit + token server + media API).
- The `voice` container running on Unraid (LiveKit + token server).
- Mic: the simulator mic is unreliable in this environment (CoreAudio `-4010`, 0 input
channels) — voice mode needs a real device. If the simulator was wiped, grant permission
first: `xcrun simctl privacy booted grant microphone dev.shane.voice`.
@@ -33,52 +33,48 @@ To run in the simulator, use Xcode or `xcrun simctl` with the built app.
Default base URL is `https://192.168.86.2:9178` (see `Sources/App/SettingsStore.swift`).
Change it at runtime in **Settings → Server URL** (the app disconnects and recreates the
session when idle). The app derives LiveKit signaling and media endpoints from this base URL.
session when idle). The app derives the LiveKit signaling endpoint from this base URL.
### TLS cert pinning
The deployment uses a self-signed cert that is **regenerated at container start**, so the
pinned SHA-256 in `Sources/Networking/TLSPinning.swift` breaks on every cert rotation.
Re-export after a server restart:
The deployment uses a persistent self-signed cert. The entrypoint regenerates it if
missing or issued for a different IP. The pinned SHA-256 in
`Sources/Networking/TLSPinning.swift` must match after any actual cert rotation:
```sh
openssl s_client -connect 192.168.86.2:9178 </dev/null 2>/dev/null \
| openssl x509 -outform der | openssl dgst -sha256
```
Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token/media calls
Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token calls
fail with a pin mismatch (not a network error).
## Features
- **Conversation** — LiveKit room with the Hope agent; voice mode (mic + TTS) and text mode
(composer), toggleable and persisted. State machine (idle/connecting/listening/thinking/
speaking) driven by audio levels, orb animation, mic meter, barge-in hint.
speaking) driven by audio levels, with a compact status indicator and mic meter.
- **Transcripts** — partials updated in place, finals with timestamps, delivery-marker
stripping, last-50 history persisted to UserDefaults, clear button.
- **Voice & model pickers** — 13 Azure TTS voices; LLM models from the agent's `models`
message. Both persist and sync on join (`set_voice`/`set_model`).
- **Media player** — now-playing card (art/title/channel/status), transport (play/pause,
±10 s, stop), seek + volume sliders, search with "in library" tags, agent-offered choices,
library list (play/remove), add-stream form. AVPlayer-backed with a pinned-TLS Range proxy;
1.5 s `/state` polling and full web-parity reconciliation/reporting.
- **Background tasks panel** — live `task_event` cards (steps, result, error) with an active
count badge in the toolbar.
- **Activity tab** — live `task_event` updates (steps, result, error) with an active
count badge on the tab.
## Known limitations
- **Simulator mic**: CoreAudio returns `-4010` (no usable input source) in this environment;
voice capture must be verified on a real device. Text mode works in the simulator.
- **webm media**: iOS AVPlayer cannot decode webm/opus (`-11828`). Library tracks from yt-dlp
are webm, so audible playback requires the server to serve m4a/aac/mp3. Everything up to
decode (pinning, Range proxy, reconciliation, UI) is verified; no client change needed once
a supported format exists.
- **Cert pin rotation**: see above — re-export the pin after each server cert regeneration.
- **LiveKit signaling** uses plain `ws://192.168.86.2:7880` (not wss) because the Swift SDK
cannot trust the self-signed cert on its WebSocket; token + media still go over pinned TLS.
cannot trust the self-signed cert on its WebSocket; token calls still go over pinned TLS.
## Docs
- [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) — current handoff, decisions, and remaining work; read first.
- [docs/ENGINEERING_LEARNINGS.md](docs/ENGINEERING_LEARNINGS.md) — regression invariants, diagnostic lessons, and deployment safeguards.
- [docs/VERIFICATION.md](docs/VERIFICATION.md) — test evidence and remaining qualification.
- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — live NAS configuration and rollback.
- [PLAN.md](PLAN.md) — phase-by-phase progress, verified facts, open items.
- [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md) — real-device qualification checklist.
- [UPDATE_PATH.md](UPDATE_PATH.md) — original plan & protocol reference (read-only).
Binary file not shown.

After

Width:  |  Height:  |  Size: 545 KiB

@@ -1,7 +1,8 @@
{
"images" : [
{
"idiom" : "universal-platform",
"filename" : "AppIcon-1024.png",
"idiom" : "universal",
"platform" : "ios",
"size" : "1024x1024"
}
+119
View File
@@ -0,0 +1,119 @@
import SwiftUI
import UIKit
/// Nearlight design tokens (design spec: local://ui-design-spec.md).
///
/// Views reference semantic roles; the primitive hex values live only here.
/// All custom surfaces are opaque; native system materials are untouched.
enum Nearlight {
// MARK: Palette (light / dark variants)
/// Screen background.
static let ground = Color(light: UIColor(red: 0.957, green: 0.949, blue: 0.965, alpha: 1),
dark: UIColor(red: 0.110, green: 0.098, blue: 0.125, alpha: 1))
/// Controls, grouped rows, dock.
static let surface = Color(light: .white,
dark: UIColor(red: 0.157, green: 0.137, blue: 0.180, alpha: 1))
/// Primary text.
static let ink = Color(light: UIColor(red: 0.161, green: 0.137, blue: 0.184, alpha: 1),
dark: UIColor(red: 0.965, green: 0.937, blue: 0.969, alpha: 1))
/// Supporting text, inactive icons.
static let quiet = Color(light: UIColor(red: 0.396, green: 0.365, blue: 0.424, alpha: 1),
dark: UIColor(red: 0.749, green: 0.698, blue: 0.776, alpha: 1))
/// Action tint, selection, Lightwell.
static let rosewood = Color(light: UIColor(red: 0.529, green: 0.278, blue: 0.380, alpha: 1),
dark: UIColor(red: 0.890, green: 0.635, blue: 0.737, alpha: 1))
/// Errors and destructive actions.
static let alert = Color(light: UIColor(red: 0.675, green: 0.208, blue: 0.282, alpha: 1),
dark: UIColor(red: 1.0, green: 0.604, blue: 0.675, alpha: 1))
// MARK: Derived roles
/// Decorative separator: Ink at 14% over its parent.
static var separator: Color { ink.opacity(0.14) }
/// Selected wash: Rosewood at 10% over Surface.
static var selectedWash: Color { rosewood.opacity(0.10) }
/// Error wash: Alert at 8% over Surface.
static var errorWash: Color { alert.opacity(0.08) }
/// Primary-action label: Surface in light, Ground in dark.
static var onRosewood: Color {
UITraitCollection.current.userInterfaceStyle == .dark ? ground : surface
}
// MARK: Spacing scale
static let s4: CGFloat = 4
static let s8: CGFloat = 8
static let s12: CGFloat = 12
static let s16: CGFloat = 16
static let s20: CGFloat = 20
static let s24: CGFloat = 24
static let s32: CGFloat = 32
static let s48: CGFloat = 48
// MARK: Radius scale
static let radiusSegment: CGFloat = 8
static let radiusControl: CGFloat = 12
static let radiusDock: CGFloat = 16
// MARK: Motion tokens
/// quick = 0.12s easeOut.
static let quick: Animation = .easeOut(duration: 0.12)
/// standard = 0.22s timingCurve(0.22, 1, 0.36, 1).
static let standard: Animation = .timingCurve(0.22, 1, 0.36, 1, duration: 0.22)
/// settle = 0.32s timingCurve(0.22, 1, 0.36, 1).
static let settle: Animation = .timingCurve(0.22, 1, 0.36, 1, duration: 0.32)
/// cycle = 2.4s easeInOut (Lightwell pulses).
static let cycle: Animation = .easeInOut(duration: 2.4)
}
extension Color {
/// Builds an adaptive color from explicit light/dark UIColors without
/// requiring an asset catalog.
init(light: UIColor, dark: UIColor) {
self.init(uiColor: UIColor { traits in
traits.userInterfaceStyle == .dark ? dark : light
})
}
}
/// Shared typography roles from the spec. SF Pro default design except the
/// rounded welcome heading and wordmark.
enum NearlightFont {
/// Welcome heading: 32 medium, rounded, largeTitle metrics.
static func welcome(_ sizeCategory: DynamicTypeSize) -> Font {
scaled(.largeTitle, base: 32, weight: .medium, design: .rounded, sizeCategory: sizeCategory)
}
/// "Hope" wordmark: 20 semibold, rounded, title3 metrics.
static func wordmark(_ sizeCategory: DynamicTypeSize) -> Font {
scaled(.title3, base: 20, weight: .semibold, design: .rounded, sizeCategory: sizeCategory)
}
/// Section/detail heading: 22 semibold, title2 metrics.
static func heading(_ sizeCategory: DynamicTypeSize) -> Font {
scaled(.title2, base: 22, weight: .semibold, sizeCategory: sizeCategory)
}
/// Control/row label: 17 semibold, body metrics.
static func control(_ sizeCategory: DynamicTypeSize) -> Font {
scaled(.body, base: 17, weight: .semibold, sizeCategory: sizeCategory)
}
/// Status/support: 15 regular, subheadline metrics.
static func support(_ sizeCategory: DynamicTypeSize) -> Font {
scaled(.subheadline, base: 15, weight: .regular, sizeCategory: sizeCategory)
}
/// Speaker/timestamp: 13 medium, footnote metrics.
static func metadata(_ sizeCategory: DynamicTypeSize) -> Font {
scaled(.footnote, base: 13, weight: .medium, sizeCategory: sizeCategory)
}
/// Scales a base size with the given text style via UIFontMetrics and
/// returns a SwiftUI font with the requested weight/design.
private static func scaled(_ textStyle: UIFont.TextStyle, base: CGFloat, weight: Font.Weight,
design: Font.Design = .default, sizeCategory: DynamicTypeSize) -> Font {
// UIFontMetrics scales the base size against the CURRENT Dynamic Type
// setting at render time (no explicit traits: inherits environment).
let scaledSize = UIFontMetrics(forTextStyle: textStyle).scaledValue(for: base)
return .system(size: scaledSize, weight: weight, design: design)
}
}
+28 -10
View File
@@ -3,19 +3,16 @@ import Combine
final class SettingsStore: ObservableObject {
@Published var serverURL: String {
didSet { UserDefaults.standard.set(serverURL, forKey: Keys.serverURL) }
didSet { AppPreferences.store.set(serverURL, forKey: Keys.serverURL) }
}
@Published var voiceID: String {
didSet { UserDefaults.standard.set(voiceID, forKey: Keys.voiceID) }
didSet { AppPreferences.store.set(voiceID, forKey: Keys.voiceID) }
}
@Published var modelID: String? {
didSet { UserDefaults.standard.set(modelID, forKey: Keys.modelID) }
didSet { AppPreferences.store.set(modelID, forKey: Keys.modelID) }
}
@Published var textMode: Bool {
didSet { UserDefaults.standard.set(textMode, forKey: Keys.textMode) }
}
@Published var volume: Double {
didSet { UserDefaults.standard.set(volume, forKey: Keys.volume) }
didSet { AppPreferences.store.set(textMode, forKey: Keys.textMode) }
}
private enum Keys {
@@ -23,13 +20,12 @@ final class SettingsStore: ObservableObject {
static let voiceID = "voiceID"
static let modelID = "modelID"
static let textMode = "textMode"
static let volume = "volume"
}
static let defaultServerURL = "https://192.168.86.2:9178"
init() {
let d = UserDefaults.standard
let d = AppPreferences.store
// Sanitize legacy values: earlier versions wrote the field to storage on
// every keystroke, so an invalid half-typed URL could be persisted and
// shipped to the network layer. Fall back to the documented default and
@@ -47,10 +43,32 @@ final class SettingsStore: ObservableObject {
voiceID = Voices.all.contains { $0.id == storedVoice } ? (storedVoice ?? "") : Voices.defaultID
modelID = d.string(forKey: Keys.modelID)
textMode = d.bool(forKey: Keys.textMode)
volume = d.object(forKey: Keys.volume) as? Double ?? 0.8
}
var trimmedServerURL: String {
serverURL.trimmingCharacters(in: .whitespacesAndNewlines)
}
}
enum AppPreferences {
static let store: UserDefaults = {
#if DEBUG
if ProcessInfo.processInfo.arguments.contains("--ui-testing") {
let suite = "dev.shane.voice.ui-testing"
let defaults = UserDefaults(suiteName: suite)!
if ProcessInfo.processInfo.arguments.contains("--reset-ui-testing") {
defaults.removePersistentDomain(forName: suite)
// Deterministic offline errors; UI tests must not contact the NAS.
defaults.set("https://127.0.0.1:1", forKey: "serverURL")
}
if ProcessInfo.processInfo.arguments.contains("--live-testing") {
defaults.set(SettingsStore.defaultServerURL, forKey: "serverURL")
defaults.set(true, forKey: "textMode")
defaults.set("qwen2.5-vl-3b", forKey: "modelID")
}
return defaults
}
#endif
return .standard
}()
}
+1 -1
View File
@@ -12,7 +12,7 @@ struct VoiceApp: App {
var body: some Scene {
WindowGroup {
ConversationView(settings: settings)
.preferredColorScheme(.dark)
.tint(Nearlight.rosewood)
}
}
}
@@ -1,343 +1,613 @@
import AVFoundation
import SwiftUI
import Combine
/// Owns one session across all tabs. Changing tabs never ends a conversation.
struct ConversationView: View {
@ObservedObject var settings: SettingsStore
@State private var sessionID = UUID()
@State private var session: VoiceSession
/// The exact server URL the current session was constructed with. Used to
/// avoid redundant session replacement when the applied URL hasn't actually
/// changed (e.g. re-applying the same value from Settings).
@State private var activeServerURL: String
/// Identity guard: prevents a second `.onChange` from starting a concurrent
/// session replacement while the first is still disconnecting the old one.
@State private var isReplacingSession = false
/// Latest accepted server URL that has not yet been applied to a session.
/// Coalesced: rapid Applies during an active replacement retain the newest
/// value rather than dropping it. A single retained worker loop drains
/// arbitrary pending URLs until none remain (Iteration 08 Item 2).
@State private var pendingServerURL: String?
@State private var composerText = ""
@State private var connectError: String?
@State private var tick = 0
@State private var isReplacingSession = false
@State private var selectedTab = 0
init(settings: SettingsStore) {
self.settings = settings
let initialSession = Self.makeSession(settings: settings)
_session = State(initialValue: initialSession)
_activeServerURL = State(initialValue: settings.trimmedServerURL)
let url = settings.trimmedServerURL
_activeServerURL = State(initialValue: url)
_session = State(initialValue: Self.makeSession(url, settings: settings))
}
/// Creates a session from the configured server URL. `SettingsStore.init`
/// guarantees the stored value is always valid (legacy invalid values are
/// sanitized to the documented default), so the failure branch is a safety
/// net only.
private static func makeSession(settings: SettingsStore) -> VoiceSession {
let config = ServerConfig.parse(settings.trimmedServerURL)
switch config {
case .success(let value):
return VoiceSession(config: value, tokenClient: TokenClient(config: value), settings: settings)
case .failure:
// Defensive: should be unreachable because SettingsStore sanitizes.
let fallback = ServerConfig.parse(SettingsStore.defaultServerURL)
if case .success(let value) = fallback {
settings.serverURL = SettingsStore.defaultServerURL
return VoiceSession(config: value, tokenClient: TokenClient(config: value), settings: settings)
private static func makeSession(_ url: String, settings: SettingsStore) -> VoiceSession {
guard case .success(let config) = ServerConfig.parse(url) else {
preconditionFailure("Stored server URL failed validation; init sanitizes to a valid value.")
}
fatalError("Unreachable: default server URL is valid")
let session = VoiceSession(config: config, tokenClient: TokenClient(config: config), settings: settings)
#if DEBUG
if ProcessInfo.processInfo.arguments.contains("--ui-fixtures") {
session.transcriptHandler.receiveDataTranscript(role: .user, text: "Help me plan a quiet weekend.")
session.transcriptHandler.receiveDataTranscript(role: .agent, text: "Let’s leave some room to breathe. A walk, a good book, and time for your favorite music would be a lovely start.")
session.taskStore.injectSampleEvents()
}
if ProcessInfo.processInfo.arguments.contains("--ui-long-transcript") {
for index in 1...30 {
let role: TranscriptEvent.Role = index.isMultiple(of: 2) ? .agent : .user
session.transcriptHandler.receiveDataTranscript(
role: role,
text: "Earlier message \(index): a deliberately long restored transcript row for reading-position verification."
)
}
}
#endif
return session
}
var body: some View {
TabView(selection: $selectedTab) {
NavigationStack {
VStack(spacing: 0) {
statusHeader
TranscriptListView(
events: session.transcriptHandler.events,
onClear: {
// Atomically clear the canonical visible collection,
// dedupe state, and persistence so Clear is truthful.
session.transcriptHandler.clearAll()
ConversationContent(settings: settings, session: session)
.toolbar { settingsLink }
}
)
composer
}
.navigationTitle("Hope Voice")
.toolbar { toolbarContent }
.id(sessionID)
.onAppear {
if case .failure(let error) = ServerConfig.parse(settings.trimmedServerURL) {
connectError = error.localizedDescription
}
}
.onChange(of: settings.textMode) { _, newValue in
guard session.isReady else { return }
Task { await session.applyTextMode(newValue) }
}
}
.onReceive(session.objectWillChange.receive(on: RunLoop.main)) { _ in
tick &+= 1
}
.onChange(of: settings.serverURL) { _, newValue in
// The URL field now uses explicit Apply semantics (SettingsView),
// so this fires only when the user has applied a value that passed
// validation. Coalesce rapid accepted values: every accepted Apply
// writes `pendingServerURL` (latest wins). A single retained worker
// loop drains arbitrary pending URLs until none remain, covering
// A→B→C semantics where C arrives while B is disconnecting
// (Iteration 08 Item 2).
let trimmed = newValue.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty, case .success = ServerConfig.parse(trimmed) else { return }
.tabItem { Label("Talk", systemImage: "bubble.left.and.bubble.right") }
.tag(0)
pendingServerURL = trimmed
// Start the worker if not already active. The loop consumes
// pending values until empty, so a single Task handles any number
// of rapid Applies without nested tasks.
guard !isReplacingSession else { return }
isReplacingSession = true
Task { @MainActor [settings] in
while let url = self.pendingServerURL {
self.pendingServerURL = nil
guard url != self.activeServerURL else { continue }
let oldSession = self.session
await oldSession.disconnect()
self.activeServerURL = url
self.sessionID = UUID()
// Construct from the captured validated config, not by
// rereading settings.serverURL (which may have changed
// during disconnect).
switch ServerConfig.parse(url) {
case .success(let config):
self.session = VoiceSession(config: config, tokenClient: TokenClient(config: config), settings: settings)
case .failure:
let fallback = ServerConfig.parse(SettingsStore.defaultServerURL)
if case .success(let value) = fallback {
self.session = VoiceSession(config: value, tokenClient: TokenClient(config: value), settings: settings)
} else {
fatalError("Unreachable: default server URL is valid")
}
}
}
self.isReplacingSession = false
ActivityTab(taskStore: session.taskStore, settings: settings, session: session)
.tag(1)
}
.tint(Nearlight.rosewood)
.onChange(of: settings.textMode) { _, enabled in
Task { await session.applyTextMode(enabled) }
}
.onChange(of: settings.serverURL) { _, value in replaceServer(value) }
}
private var statusHeader: some View {
VStack(spacing: 12) {
OrbView(state: orbState)
Text(statusText)
.font(.headline)
if session.state == .ready, session.stateModel.isThinkingChipVisible {
Label("Thinking…", systemImage: "ellipsis")
.font(.caption)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(.orange.opacity(0.2), in: Capsule())
}
if session.state == .ready, !settings.textMode {
micMeter
}
if session.state == .ready, settings.textMode {
Text("Barge in anytime — type to reply")
.font(.caption)
.foregroundStyle(.secondary)
}
if let agent = session.agent {
Text("Agent: \(agent.name ?? agent.identity)")
.font(.caption)
.foregroundStyle(.secondary)
}
if session.state == .ready {
Text("\(Voices.display(for: settings.voiceID) ?? settings.voiceID) · \(session.selectedModel ?? settings.modelID ?? "—")")
.font(.caption)
.foregroundStyle(.secondary)
}
if session.state == .ready {
HStack(spacing: 16) {
// The mic toggle is disabled in text mode to enforce the
// privacy invariant at the UI level as well (Finding 4).
Toggle(isOn: Binding(
get: { !session.isMuted },
set: { newValue in Task { await session.setMuted(!newValue) } }
)) {
Label(session.isMuted ? "Muted" : "Mic on", systemImage: session.isMuted ? "mic.slash" : "mic")
}
.toggleStyle(.button)
.disabled(settings.textMode)
Toggle("Text mode", isOn: textModeBinding)
.toggleStyle(.button)
}
}
if session.state == .ready, session.isReconnecting {
Label("Connection lost — reconnecting…", systemImage: "arrow.triangle.2.circlepath")
.font(.footnote)
.foregroundStyle(.orange)
.frame(maxWidth: .infinity, alignment: .leading)
}
if let micError = session.micError {
Label(micError, systemImage: "exclamationmark.triangle")
.font(.footnote)
.foregroundStyle(.orange)
.frame(maxWidth: .infinity, alignment: .leading)
}
if let connectError {
Label(connectError, systemImage: "exclamationmark.triangle")
.font(.footnote)
.foregroundStyle(.red)
}
}
.padding()
}
private var micMeter: some View {
GeometryReader { geo in
ZStack(alignment: .leading) {
Capsule()
.fill(Color(.systemFill))
Capsule()
.fill(Color.green.gradient)
.frame(width: min(geo.size.width, geo.size.width * session.stateModel.micLevel * 2.5))
}
}
.frame(height: 4)
}
private var textModeBinding: Binding<Bool> {
$settings.textMode
}
private var composer: some View {
HStack(spacing: 8) {
switch session.state {
case .idle, .failed:
Button(action: startSession) {
Label("Start", systemImage: "phone.fill")
}
.buttonStyle(.borderedProminent)
case .connecting, .waitingAgent, .tearingDown:
ProgressView()
case .ready:
TextField(settings.textMode ? "Type a message…" : "Voice mode active",
text: $composerText,
axis: .vertical)
.textFieldStyle(.roundedBorder)
.lineLimit(1...4)
.disabled(!settings.textMode)
Button(action: sendComposer) {
Image(systemName: "arrow.up.circle.fill")
.font(.title2)
}
.accessibilityLabel("Send message")
.disabled(!settings.textMode || composerText.isEmpty)
Button(role: .destructive, action: endSession) {
Image(systemName: "phone.down.fill")
}
.accessibilityLabel("End conversation")
}
}
.padding()
}
@ToolbarContentBuilder
private var toolbarContent: some ToolbarContent {
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
TaskPanelView(taskStore: session.taskStore)
} label: {
Image(systemName: "tray.full")
.overlay(alignment: .topTrailing) {
if session.taskStore.activeCount > 0 {
Text("\(session.taskStore.activeCount)")
.font(.caption2.weight(.bold))
.foregroundStyle(.white)
.padding(3)
.background(Color.red, in: Circle())
.offset(x: 6, y: -6)
}
}
}
.accessibilityLabel("Background tasks")
}
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
MediaPanelView(playerController: session.mediaController, settings: settings)
} label: {
Image(systemName: "music.note")
}
.accessibilityLabel("Media player")
}
@ToolbarContentBuilder private var settingsLink: some ToolbarContent {
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
SettingsView(settings: settings, session: session)
} label: {
Image(systemName: "gearshape")
Image(systemName: "gearshape").frame(minWidth: 44, minHeight: 44)
}
.accessibilityLabel("Settings")
}
}
private var orbState: OrbView.OrbState {
private func replaceServer(_ value: String) {
let url = value.trimmingCharacters(in: .whitespacesAndNewlines)
guard case .success = ServerConfig.parse(url) else { return }
pendingServerURL = url
guard !isReplacingSession else { return }
isReplacingSession = true
Task { @MainActor in
// Coalesce rapid Apply actions; construct from the captured URL.
while let next = pendingServerURL {
pendingServerURL = nil
guard next != activeServerURL else { continue }
await session.disconnect()
activeServerURL = next
session = Self.makeSession(next, settings: settings)
}
isReplacingSession = false
}
}
}
private struct ActivityTab: View {
@ObservedObject var taskStore: TaskStore
@ObservedObject var settings: SettingsStore
@ObservedObject var session: VoiceSession
var body: some View {
NavigationStack {
TaskPanelView(taskStore: taskStore)
.toolbar {
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
SettingsView(settings: settings, session: session)
} label: {
Image(systemName: "gearshape").frame(minWidth: 44, minHeight: 44)
}
.accessibilityLabel("Settings")
}
}
}
.tabItem { Label("Activity", systemImage: "checklist") }
.badge(taskStore.activeCount)
}
}
/// Maps session + conversation models onto Lightwell presentation phases.
@MainActor
private func lightwellPhase(session: VoiceSession, conversation: ConversationStateModel,
textMode: Bool) -> LightwellView.Phase {
if session.state.isFailed { return .failed }
switch session.state {
case .idle: return .idle
case .connecting, .waitingAgent, .tearingDown: return .connecting
case .ready:
guard !settings.textMode else { return .idle }
switch session.stateModel.phase {
case .idle: return .idle
case .connecting: return .connecting
case .listening: return .listening
case .thinking: return .thinking
case .speaking: return .speaking
}
case .waitingAgent: return .waiting
case .tearingDown: return .tearingDown
case .failed: return .failed
}
}
private var statusText: String {
switch session.state {
case .idle: return "Ready to connect"
case .connecting: return "Connecting…"
case .waitingAgent: return "Waiting for agent…"
case .tearingDown: return "Cleaning up…"
case .ready:
if settings.textMode { return "Text mode" }
switch session.stateModel.phase {
case .listening: return "Listening"
case .thinking: return "Thinking it through"
case .speaking: return "Hope is speaking"
default: return "Ready when you are"
// Only substitute idle input geometry when otherwise LISTENING.
// Assistant output (speaking/thinking) must stay represented in
// Type and muted modes per spec.
switch conversation.phase {
case .speaking: return .speaking(level: conversation.agentLevel)
case .thinking: return .thinking
default:
if textMode || session.isMuted { return .readyNoInput }
return .listening(level: conversation.micLevel)
}
case .failed(let message): return "Failed: \(message)"
}
}
private func startSession() {
connectError = nil
// Starting a conversation preserves the restored last-50 history; only
// the explicit Clear button erases it.
private struct ConversationContent: View {
@ObservedObject var settings: SettingsStore
@ObservedObject var session: VoiceSession
@ObservedObject private var transcripts: TranscriptHandler
@ObservedObject private var conversation: ConversationStateModel
@State private var composerText = ""
@State private var sendError: String?
@State private var isSending = false
@FocusState private var composerFocused: Bool
@Environment(\.dynamicTypeSize) private var typeSize
@Environment(\.accessibilityReduceMotion) private var reduceMotion
@Environment(\.colorSchemeContrast) private var contrast
init(settings: SettingsStore, session: VoiceSession) {
self.settings = settings
self.session = session
transcripts = session.transcriptHandler
conversation = session.stateModel
}
var body: some View {
VStack(spacing: 0) {
if transcripts.events.isEmpty && !session.isReady {
welcome
if failureNotice != nil { failureBanner }
} else {
statusStrip
if session.isReconnecting { reconnectBanner }
if failureNotice != nil { failureBanner }
TranscriptListView(events: transcripts.events, onClear: transcripts.clearAll)
}
}
.background(Nearlight.ground)
.navigationTitle("Hope")
.navigationBarTitleDisplayMode(.inline)
.safeAreaInset(edge: .bottom, spacing: 0) { dock }
.onChange(of: session.state) { _, state in
if state == .ready && settings.textMode { composerFocused = true }
}
.sensoryFeedback(.success, trigger: session.state == .ready) { old, new in
// Success only on the transition INTO readiness, not out.
!old && new
}
.sensoryFeedback(.error, trigger: session.state.isFailed) { old, new in
// Error only on entering failure, not leaving it.
!old && new
}
}
// MARK: Welcome (pre-connection, no history)
private var welcome: some View {
ScrollView {
VStack(alignment: .leading, spacing: Nearlight.s24) {
LightwellView(phase: lightwellPhase(session: session, conversation: conversation, textMode: settings.textMode))
.padding(.top, Nearlight.s32)
VStack(alignment: .leading, spacing: Nearlight.s12) {
Text("A moment with Hope.")
.font(NearlightFont.welcome(typeSize))
.fixedSize(horizontal: false, vertical: true)
Text("Speak or type to your own assistant.")
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.quiet)
}
statusLine
}
.frame(maxWidth: .infinity, alignment: .leading)
.padding(.horizontal, Nearlight.s20)
.padding(.bottom, Nearlight.s24)
}
}
/// Status text 24 below the supporting copy on welcome.
private var statusLine: some View {
Text(statusTitle)
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.quiet)
.padding(.top, Nearlight.s24)
.accessibilityIdentifier("talk.status")
}
// MARK: Status strip (active conversation)
private var statusStrip: some View {
HStack(spacing: Nearlight.s12) {
LightwellView(phase: lightwellPhase(session: session, conversation: conversation, textMode: settings.textMode), compact: true)
VStack(alignment: .leading, spacing: 2) {
Text(statusTitle)
.font(NearlightFont.control(typeSize))
.foregroundStyle(Nearlight.ink)
.accessibilityIdentifier("talk.status")
if showStatusDetail {
Text(statusDetail)
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.quiet)
}
}
Spacer(minLength: 0)
}
.padding(.horizontal, Nearlight.s20)
.padding(.vertical, Nearlight.s12)
.frame(minHeight: 60, alignment: .leading)
.background(Nearlight.ground)
}
/// Subtitle omitted for ordinary listening/thinking/speaking per spec.
private var showStatusDetail: Bool {
guard session.isReady else { return true }
if settings.textMode || session.isMuted { return true }
return false
}
private var reconnectBanner: some View {
VStack(alignment: .leading, spacing: 2) {
Label("Reconnecting…", systemImage: "wifi.exclamationmark")
.font(NearlightFont.control(typeSize))
Text("Your conversation is still here.")
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.quiet)
}
.padding(Nearlight.s12)
.frame(maxWidth: .infinity, alignment: .leading)
.background(Nearlight.surface)
.clipShape(RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.padding(.horizontal, Nearlight.s20)
.accessibilityIdentifier("talk.connection.notice")
}
/// Classified failure notice with actionable recovery.
private var failureBanner: some View {
VStack(alignment: .leading, spacing: Nearlight.s8) {
Label(failureTitle, systemImage: "exclamationmark.circle")
.font(NearlightFont.control(typeSize))
.foregroundStyle(Nearlight.alert)
Text(failureBody)
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.ink)
if micDenied {
Button("Open iPhone Settings") {
if let url = URL(string: UIApplication.openSettingsURLString) {
UIApplication.shared.open(url)
}
}
.font(NearlightFont.support(typeSize))
.frame(minHeight: 44)
.accessibilityIdentifier("talk.connection.openSystemSettings")
}
}
.padding(Nearlight.s16)
.frame(maxWidth: .infinity, alignment: .leading)
.background(Nearlight.errorWash)
.clipShape(RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.padding(.horizontal, Nearlight.s20)
.accessibilityIdentifier("talk.connection.notice")
}
private var micDenied: Bool {
guard session.state == .ready, let micError = session.micError else {
return false
}
return AVAudioApplication.shared.recordPermission == .denied
|| micError.localizedCaseInsensitiveContains("denied")
}
private var failureTitle: String {
if session.micError != nil { return "Microphone unavailable" }
return session.everConnected ? "Connection interrupted" : "Couldn’t connect"
}
private var failureBody: String {
if let micError = session.micError {
return micError
}
let raw = failureMessage ?? ""
// Only the agent-timeout path is a known classified cause; every
// other failure keeps its actual readable error per spec.
if raw.contains("Agent never joined") {
return "Check that Hope is running on your server."
}
return raw.isEmpty ? "The connection failed. Try again." : raw
}
// MARK: Dock
private var dock: some View {
VStack(spacing: Nearlight.s12) {
if let sendError {
Label(sendError, systemImage: "exclamationmark.circle")
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.alert)
.frame(maxWidth: .infinity, alignment: .leading)
.accessibilityIdentifier("talk.composer.error")
}
modeSelector
if session.isReady {
if settings.textMode { typeControls } else { voiceControls }
} else if isBusy && !session.state.isFailed {
// Connecting/waiting: keep the draft readable and expose the
// existing cancellation path per spec.
if settings.textMode { typeControls }
else { HStack(spacing: Nearlight.s12) { muteButton; micMeter } }
endButton
} else {
startControls
}
}
.padding(.horizontal, Nearlight.s20)
.padding(.top, Nearlight.s12)
.padding(.bottom, Nearlight.s12)
.background(
// Dock: Surface, 16-point top corners, decorative top separator.
VStack(spacing: 0) {
Rectangle().fill(Nearlight.separator).frame(height: 1 / UIScreen.main.scale)
Nearlight.surface
}
.ignoresSafeArea(.container, edges: .bottom)
.clipShape(UnevenRoundedRectangle(topLeadingRadius: Nearlight.radiusDock, topTrailingRadius: Nearlight.radiusDock))
)
}
@ViewBuilder private var modeSelector: some View {
if typeSize.isAccessibilitySize {
// Two stacked 52-point selection buttons at accessibility sizes.
VStack(spacing: Nearlight.s8) {
modeButton("Voice", systemImage: "mic", selected: !settings.textMode, id: "talk.mode.voice") {
settings.textMode = false
}
modeButton("Type", systemImage: "keyboard", selected: settings.textMode, id: "talk.mode.type") {
settings.textMode = true
}
}
.disabled(isBusy)
} else {
Picker("Conversation mode", selection: $settings.textMode) {
Label("Voice", systemImage: "mic").tag(false)
Label("Type", systemImage: "keyboard").tag(true)
}
.pickerStyle(.segmented)
.disabled(isBusy)
}
}
private func modeButton(_ title: String, systemImage: String, selected: Bool,
id: String, action: @escaping () -> Void) -> some View {
Button(action: action) {
HStack(spacing: Nearlight.s8) {
Image(systemName: systemImage)
Text(title)
if selected { Spacer(); Image(systemName: "checkmark") }
}
.font(NearlightFont.control(typeSize))
.foregroundStyle(selected ? Nearlight.rosewood : Nearlight.ink)
.frame(maxWidth: .infinity, minHeight: 52)
.background(selected && contrast == .increased ? Nearlight.surface : selected ? Nearlight.selectedWash : Nearlight.surface,
in: RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.overlay(
RoundedRectangle(cornerRadius: Nearlight.radiusControl)
.strokeBorder(selected && contrast == .increased ? Nearlight.rosewood : selected ? Nearlight.rosewood.opacity(0.4) : Nearlight.quiet,
lineWidth: selected && contrast == .increased ? 2 : 1)
)
}
.accessibilityIdentifier(id)
.accessibilityAddTraits(selected ? .isSelected : [])
}
private var startControls: some View {
VStack(spacing: Nearlight.s8) {
Text(settings.textMode ? "Your microphone stays off." : "Your microphone starts after you connect.")
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.quiet)
Button {
sendError = nil
Task { await session.connect() }
} label: {
HStack(spacing: Nearlight.s8) {
if isBusy {
ProgressView().tint(Nearlight.onRosewood)
} else {
Image(systemName: settings.textMode ? "keyboard" : "mic")
}
Text(isBusy ? statusTitle : session.state.isFailed ? "Try again" : "Start chatting")
.font(NearlightFont.control(typeSize))
}
.frame(maxWidth: .infinity, minHeight: 52)
.background(isBusy ? Nearlight.rosewood.opacity(0.6) : Nearlight.rosewood,
in: RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.foregroundStyle(Nearlight.onRosewood)
}
.disabled(isBusy)
.accessibilityIdentifier("start.chatting")
}
}
private func endSession() {
private var voiceControls: some View {
Group {
if typeSize.isAccessibilitySize {
VStack(spacing: Nearlight.s12) {
HStack(spacing: Nearlight.s12) {
muteButton
micMeter
}
endButton
}
} else {
HStack(spacing: Nearlight.s12) {
muteButton
micMeter
endButton
}
}
}
}
private var muteButton: some View {
Button {
Task { await session.setMuted(!session.isMuted) }
} label: {
Image(systemName: session.isMuted ? "mic.slash" : "mic")
.font(.system(size: 20, weight: .medium))
.frame(width: 52, height: 52)
.background(session.isMuted ? Nearlight.selectedWash : Nearlight.surface,
in: RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.overlay(
RoundedRectangle(cornerRadius: Nearlight.radiusControl)
.strokeBorder(Nearlight.quiet, lineWidth: 1)
)
.foregroundStyle(session.isMuted ? Nearlight.rosewood : Nearlight.ink)
}
.accessibilityLabel(session.isMuted ? "Unmute microphone" : "Mute microphone")
.accessibilityIdentifier("talk.microphone.toggle")
.sensoryFeedback(.selection, trigger: session.isMuted)
}
/// Five-bar capsule meter, 64×24, leading aligned.
private var micMeter: some View {
HStack(spacing: 6) {
let level = session.isMuted || settings.textMode ? 0.0 : conversation.micLevel
let filled = Int(ceil(level * 5))
ForEach(0..<5, id: \.self) { index in
Capsule()
.fill(index < filled ? Nearlight.rosewood : Color.clear)
.overlay(index < filled ? nil : Capsule().strokeBorder(Nearlight.quiet, lineWidth: 1))
.frame(width: 6, height: [6.0, 12.0, 24.0, 12.0, 6.0][index])
}
}
.frame(width: 64, height: 24, alignment: .leading)
.accessibilityElement(children: .ignore)
.accessibilityLabel("Microphone level")
.accessibilityValue(session.isMuted || settings.textMode ? "Off" : "\(Int(round(conversation.micLevel * 100)))%")
}
private var typeControls: some View {
VStack(spacing: Nearlight.s12) {
HStack(alignment: .bottom, spacing: Nearlight.s8) {
TextField("Message Hope", text: $composerText, axis: .vertical)
.lineLimit(1...5)
.padding(Nearlight.s12)
.background(Nearlight.surface, in: RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.overlay(
RoundedRectangle(cornerRadius: Nearlight.radiusControl)
.strokeBorder(Nearlight.quiet, lineWidth: 1)
)
.focused($composerFocused)
.disabled(isSending)
.accessibilityLabel("Message Hope")
Button(action: sendComposer) {
if isSending {
ProgressView().frame(width: 52, height: 52)
} else {
Image(systemName: "arrow.up")
.font(.system(size: 20, weight: .medium))
.frame(width: 52, height: 52)
.background(Nearlight.rosewood, in: RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.foregroundStyle(Nearlight.onRosewood)
}
}
.disabled(composerText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || isSending || session.isReconnecting || !session.isReady)
.accessibilityLabel("Send message")
}
endButton
}
}
private var endButton: some View {
Button(role: .destructive) {
composerFocused = false
Task { await session.disconnect() }
} label: {
Label("End conversation", systemImage: "stop.fill")
.font(NearlightFont.control(typeSize))
.lineLimit(2...3)
.frame(maxWidth: .infinity, minHeight: 52)
.background(Nearlight.surface, in: RoundedRectangle(cornerRadius: Nearlight.radiusControl))
.overlay(
RoundedRectangle(cornerRadius: Nearlight.radiusControl)
.strokeBorder(Nearlight.alert, lineWidth: 1)
)
.foregroundStyle(Nearlight.alert)
}
.accessibilityIdentifier("end.conversation")
}
// MARK: Status copy
private var isBusy: Bool {
session.state == .connecting || session.state == .waitingAgent || session.state == .tearingDown
}
private var failureMessage: String? {
if case .failed(let message) = session.state { return message }
return nil
}
private var failureNotice: String? { session.micError ?? failureMessage }
private var statusTitle: String {
switch session.state {
case .idle: return "Ready when you are"
case .connecting: return "Connecting to your server…"
case .waitingAgent: return "Waiting for Hope…"
case .tearingDown: return "Ending conversation…"
case .failed: return "Connection interrupted"
case .ready:
// Assistant output outranks input-mode substitution.
switch conversation.phase {
case .speaking: return "Hope is speaking"
case .thinking: return "Thinking…"
default:
if settings.textMode { return "Ready for your message" }
if session.isMuted { return "Microphone off" }
return "Listening"
}
}
}
private var statusDetail: String {
if session.state == .waitingAgent { return "The server is connected. Hope has not joined yet." }
if session.state == .connecting { return "Reaching your server." }
if !session.isReady { return "" }
if settings.textMode { return "Microphone off · Type whenever you’re ready" }
if session.isMuted { return "You can still hear Hope." }
return ""
}
private func sendComposer() {
let text = composerText.trimmingCharacters(in: .whitespacesAndNewlines)
guard !text.isEmpty else { return }
composerText = ""
guard !text.isEmpty, !isSending else { return }
isSending = true
sendError = nil
Task {
do {
try await session.sendText(text)
composerText = ""
} catch {
connectError = error.localizedDescription
sendError = "Your message wasn’t sent. Try again when connected."
}
isSending = false
}
}
}
#Preview {
ConversationView(settings: SettingsStore())
extension VoiceSession {
/// Whether any connection reached ready during this session lifetime.
/// Used to title the failure notice per spec.
var everConnected: Bool {
// Sessions are recreated on replacement; reaching ready is recorded
// indirectly through transcript presence. Conservative: treat any
// prior transcript activity as evidence of a completed connection.
!transcriptHandler.events.isEmpty
}
}
#Preview { ConversationView(settings: SettingsStore()) }
@@ -0,0 +1,151 @@
import SwiftUI
import UIKit
/// The Nearlight signature element: an upright, softly lit aperture.
///
/// Two capsules in a ZStack — no assets, shaders, blur, or particles. The
/// inner capsule carries a vertical Rosewood gradient; its geometry encodes
/// conversation state and audio level. Noninteractive and accessibility
/// hidden: the companion status text carries meaning.
struct LightwellView: View {
/// Lightwell presentation state, mapped from session/conversation models.
enum Phase: Equatable {
case idle
case connecting
case waiting
case listening(level: Double)
case thinking
case speaking(level: Double)
case tearingDown
case failed
/// Input never drives the light (muted or Type mode).
case readyNoInput
}
let phase: Phase
/// Compact (24×36) in the conversation status strip; full (64×96) on welcome.
var compact: Bool = false
@Environment(\.accessibilityReduceMotion) private var reduceMotion
/// Frame dimensions never animate; inner dimensions multiply by 0.375
/// in the compact frame.
private var frameSize: CGSize {
compact ? CGSize(width: 24, height: 36) : CGSize(width: 64, height: 96)
}
private var baseInner: CGSize {
let base = CGSize(width: 16, height: 40)
return compact
? CGSize(width: base.width * 0.375, height: base.height * 0.375)
: base
}
@Environment(\.colorSchemeContrast) private var contrast
var body: some View {
Group {
if pulseWanted {
// Isolated TimelineView: ticks only while pending states are
// showing; pauses when offscreen/inactive automatically.
TimelineView(.periodic(from: .now, by: 0.05)) { context in
lightwell(pulse: pulseValue(at: context.date))
}
} else {
lightwell(pulse: 0)
}
}
}
private func lightwell(pulse: Double) -> some View {
let inner = innerSize
return ZStack {
Capsule()
.fill(Nearlight.surface)
Capsule()
.fill(innerFill)
.frame(width: inner.width, height: inner.height)
.opacity(overallOpacity(pulse: pulse))
}
.frame(width: frameSize.width, height: frameSize.height)
.overlay(
Capsule().strokeBorder(
contrast == .increased ? Nearlight.rosewood : Nearlight.quiet,
lineWidth: 1
)
)
.accessibilityHidden(true)
.animation(reduceMotion ? nil : Nearlight.settle, value: innerSize)
}
/// Increase Contrast promotes the gradient to solid Rosewood.
private var innerFill: AnyShapeStyle {
if contrast == .increased {
return AnyShapeStyle(Nearlight.rosewood)
}
return AnyShapeStyle(
LinearGradient(
stops: [
.init(color: Nearlight.rosewood.opacity(0.35), location: 0),
.init(color: Nearlight.rosewood.opacity(1), location: 0.5),
.init(color: Nearlight.rosewood.opacity(0.55), location: 1),
],
startPoint: .top, endPoint: .bottom
)
)
}
/// Whether a pulse TimelineView should run for this phase.
private var pulseWanted: Bool {
switch phase {
case .connecting, .waiting, .thinking: return !reduceMotion
default: return false
}
}
/// Triangle wave 0→1→0 over the 2.4s cycle from the timeline date.
private func pulseValue(at date: Date) -> Double {
let t = date.timeIntervalSinceReferenceDate.truncatingRemainder(dividingBy: 2.4) / 2.4
return t < 0.5 ? t * 2 : (1 - t) * 2
}
/// Inner capsule geometry per spec table. `a` is the smoothed level.
/// Reduce Motion: static 16×40 (idle/listening) and 16×48 (pending/
/// thinking/speaking) with no level-driven resizing.
private var innerSize: CGSize {
let base = baseInner
if reduceMotion {
switch phase {
case .waiting, .thinking, .speaking:
return CGSize(width: base.width, height: base.height + (compact ? 3 : 8))
default:
return base
}
}
switch phase {
case .listening(let a):
let level = min(max(a, 0), 1)
return CGSize(width: base.width + (compact ? 3 : 8) * level,
height: base.height + (compact ? 6 : 16) * level)
case .speaking(let a):
let level = min(max(a, 0), 1)
return CGSize(width: base.width + (compact ? 4.5 : 12) * level,
height: base.height + (compact ? 9 : 24) * level)
case .waiting, .thinking:
// Pulse drives opacity; height steps up one band.
return CGSize(width: base.width, height: base.height + (compact ? 3 : 8))
case .connecting, .idle, .failed, .tearingDown, .readyNoInput:
return base
}
}
private func overallOpacity(pulse: Double) -> Double {
switch phase {
case .connecting, .waiting:
return 0.45 + 0.45 * pulse
case .thinking:
return 0.65 + 0.35 * pulse
case .idle, .failed, .tearingDown:
return 0.45
case .listening, .speaking, .readyNoInput:
return 1
}
}
}
-150
View File
@@ -1,150 +0,0 @@
import SwiftUI
struct OrbView: View {
enum OrbState {
case idle
case connecting
case listening
case speaking
case thinking
case failed
}
let state: OrbState
@State private var animating = false
private var baseColor: Color {
switch state {
case .idle, .connecting: .blue
case .listening: .green
case .speaking, .thinking: .orange
case .failed: .red
}
}
private var scale: CGFloat {
switch state {
case .connecting, .listening, .speaking, .thinking: 1.1
default: 1.0
}
}
private var label: String {
switch state {
case .idle: "Idle"
case .connecting: "Connecting"
case .listening: "Listening"
case .speaking: "Speaking"
case .thinking: "Thinking"
case .failed: "Connection failed"
}
}
var body: some View {
let shape = Circle()
.fill(baseColor.gradient)
.frame(width: 120, height: 120)
let animated = shape
.scaleEffect(animating ? scale : 0.95)
.opacity(state == .idle || state == .failed ? 0.7 : 1.0)
let anim: Animation? = state == .idle || state == .failed
? nil
: .easeInOut(duration: 0.8).repeatForever(autoreverses: true)
animated
.animation(anim, value: animating)
.onAppear { animating = true }
.onChange(of: state) { _, _ in animating = true }
.accessibilityElement()
.accessibilityLabel("Conversation status")
.accessibilityValue(label)
}
}
struct TranscriptListView: View {
/// The single canonical visible transcript (restored persisted finals plus
/// current live events). The view renders exactly this collection — there is
/// no second "restored" store, so the UI and persistence can never diverge.
let events: [TranscriptEvent]
var onClear: () -> Void
private static let deliveryMarkers = #"\[(?:appreciative|calm|concerned|curious|encouraging|excited|happy|joking|quiet|reassuring|reflective|sad|serious|surprised|sympathetic|urgent|whispering|laughter|sighing|breathing|neutral)\]\s*"#
private static let timestampFormatter: DateFormatter = {
let formatter = DateFormatter()
formatter.dateStyle = .none
formatter.timeStyle = .short
return formatter
}()
private var visibleRows: [TranscriptEvent] { events }
var body: some View {
ScrollViewReader { proxy in
VStack(alignment: .leading, spacing: 0) {
HStack {
Text("Transcript")
.font(.caption.bold())
.foregroundStyle(.secondary)
Spacer()
Button(role: .destructive, action: onClear) {
Label("Clear", systemImage: "trash")
.font(.caption)
}
}
.padding(.horizontal)
.padding(.bottom, 4)
ScrollView {
LazyVStack(alignment: .leading, spacing: 8) {
ForEach(visibleRows) { event in
transcriptRow(event)
}
}
.padding(.horizontal)
}
.onChange(of: visibleRows.count) { _, _ in
if let last = visibleRows.last {
withAnimation { proxy.scrollTo(last.id, anchor: .bottom) }
}
}
}
}
}
private func transcriptRow(_ event: TranscriptEvent) -> some View {
let roleName = event.role == .user ? "You" : "Hope"
let nameColor: Color = event.role == .user ? .secondary : .blue
let rowBackground: Color = event.role == .user
? Color(.secondarySystemBackground)
: Color.blue.opacity(0.08)
let textOpacity: Double = event.isFinal ? 1.0 : 0.55
return VStack(alignment: .leading, spacing: 2) {
HStack {
Text(roleName)
.font(.caption2.bold())
.foregroundStyle(nameColor)
Spacer()
Text(Self.timestampFormatter.string(from: event.timestamp))
.font(.caption2)
.foregroundStyle(.tertiary)
}
Text(stripped(event.text))
.font(.body)
.opacity(textOpacity)
}
.frame(maxWidth: .infinity, alignment: .leading)
.padding(8)
.background(rowBackground)
.cornerRadius(10)
.id(event.id)
}
private func stripped(_ text: String) -> String {
text.replacingOccurrences(
of: Self.deliveryMarkers,
with: "",
options: .regularExpression
).trimmingCharacters(in: .whitespaces)
}
}
@@ -0,0 +1,236 @@
import SwiftUI
/// Nearlight transcript: a legible shared record. Full-width rows, no
/// bubbles; metadata row per event; "Live" placeholder on partials.
struct TranscriptListView: View {
let events: [TranscriptEvent]
var onClear: () -> Void
@State private var confirmClear = false
@State private var followingLatest = true
@State private var selectionActive = false
@State private var viewportHeight: CGFloat = 0
@State private var latestBottom: CGFloat = 0
@AccessibilityFocusState private var focusedEventID: String?
@Environment(\.accessibilityReduceMotion) private var reduceMotion
@Environment(\.dynamicTypeSize) private var typeSize
init(events: [TranscriptEvent], onClear: @escaping () -> Void) {
self.events = events
self.onClear = onClear
_followingLatest = State(initialValue: events.isEmpty)
}
/// Auto-follow eligibility: never scroll while the user is selecting
/// text or reading history.
private var isFollowingEligible: Bool {
followingLatest && !selectionActive && focusedEventID == nil
}
private static let deliveryMarkers = #"\[(?:appreciative|calm|concerned|curious|encouraging|excited|happy|joking|quiet|reassuring|reflective|sad|serious|surprised|sympathetic|urgent|whispering|laughter|sighing|breathing|neutral)\]\s*"#
var body: some View {
ScrollViewReader { proxy in
VStack(spacing: 0) {
HStack {
Text("Conversation")
.font(NearlightFont.support(typeSize).weight(.semibold))
.foregroundStyle(Nearlight.quiet)
Spacer()
Menu {
Button("Clear conversation", systemImage: "trash", role: .destructive) { confirmClear = true }
} label: {
Image(systemName: "ellipsis").frame(width: 44, height: 44)
}
.accessibilityLabel("Conversation options")
.disabled(events.isEmpty)
}
.padding(.horizontal, Nearlight.s20)
ScrollView {
LazyVStack(alignment: .leading, spacing: Nearlight.s24) {
if events.isEmpty {
Text("Your conversation will appear here.")
.font(.body)
.foregroundStyle(Nearlight.quiet)
.padding(.top, Nearlight.s24)
}
ForEach(events) { event in transcriptRow(event) }
GeometryReader { geometry in
Color.clear.preference(
key: TranscriptLatestBottomKey.self,
value: geometry.frame(in: .named("transcriptScroll")).maxY
)
}
.frame(height: 1)
.id("latest")
}
.padding(.horizontal, Nearlight.s20)
.padding(.top, Nearlight.s16)
.padding(.bottom, Nearlight.s20)
.frame(minHeight: 200, alignment: .topLeading)
}
.coordinateSpace(name: "transcriptScroll")
.background {
GeometryReader { geometry in
Color.clear.preference(
key: TranscriptViewportHeightKey.self,
value: geometry.size.height
)
}
}
.scrollDismissesKeyboard(.interactively)
.defaultScrollAnchor(.top)
.simultaneousGesture(DragGesture().onChanged { _ in
// Any manual drag suspends automatic following.
followingLatest = false
})
.onChange(of: events.last?.text) { _, _ in
guard isFollowingEligible else { return }
proxy.scrollTo("latest", anchor: .bottom)
}
.onChange(of: events.count) { _, count in
if count == 0 {
selectionActive = false
followingLatest = true
} else if isFollowingEligible {
proxy.scrollTo("latest", anchor: .bottom)
}
}
.onChange(of: selectionActive) { _, active in
// Beginning text selection suspends auto-follow so the
// user's reading position is never disturbed.
if active { followingLatest = false }
}
.onChange(of: focusedEventID) { _, eventID in
// VoiceOver focus on any transcript row means the user is
// reading history. Only Latest resumes automatic following.
if eventID != nil { followingLatest = false }
}
.onPreferenceChange(TranscriptViewportHeightKey.self) { height in
viewportHeight = height
updateFollowingEligibility(bottom: latestBottom, viewport: height)
}
.onPreferenceChange(TranscriptLatestBottomKey.self) { bottom in
latestBottom = bottom
updateFollowingEligibility(bottom: bottom, viewport: viewportHeight)
}
if !followingLatest && !events.isEmpty {
HStack {
Spacer()
Button {
// Explicit Latest restores full eligibility,
// clearing any stale selection suspension.
selectionActive = false
focusedEventID = nil
followingLatest = true
withAnimation(reduceMotion ? nil : Nearlight.standard) {
proxy.scrollTo("latest", anchor: .bottom)
}
} label: {
Label("Latest", systemImage: "arrow.down")
.font(NearlightFont.support(typeSize))
.padding(.horizontal, Nearlight.s12)
.frame(minHeight: 44)
.background(Nearlight.surface, in: Capsule())
.overlay(Capsule().strokeBorder(Nearlight.quiet, lineWidth: 1))
}
.accessibilityIdentifier("talk.latest")
}
.padding(.horizontal, Nearlight.s12)
.padding(.bottom, Nearlight.s12)
}
}
}
.alert("Clear this conversation?", isPresented: $confirmClear) {
Button("Clear conversation", role: .destructive, action: onClear)
Button("Cancel", role: .cancel) {}
} message: {
Text("This removes the transcript saved on this iPhone. It doesn’t reset Hope’s memory or end your current conversation.")
}
}
private func updateFollowingEligibility(bottom: CGFloat, viewport: CGFloat) {
guard bottom > 0,
viewport > 0,
!selectionActive,
focusedEventID == nil else { return }
let shouldFollow = bottom <= viewport + 48
if followingLatest != shouldFollow {
followingLatest = shouldFollow
}
}
private func transcriptRow(_ event: TranscriptEvent) -> some View {
let isUser = event.role == .user
let text = event.text.replacingOccurrences(of: Self.deliveryMarkers, with: "", options: .regularExpression)
.trimmingCharacters(in: .whitespaces)
return VStack(alignment: .leading, spacing: Nearlight.s8) {
// Metadata row: speaker, timestamp trailing. Stacks at
// accessibility sizes per spec.
Group {
if typeSize.isAccessibilitySize {
VStack(alignment: .leading, spacing: Nearlight.s4) {
speakerLabel(isUser)
timeLabel(event)
}
} else {
HStack(spacing: Nearlight.s8) {
speakerLabel(isUser)
timeLabel(event)
}
}
}
Text(text)
.font(.body)
.foregroundStyle(Nearlight.ink)
.textSelection(.enabled)
.fixedSize(horizontal: false, vertical: true)
}
.frame(maxWidth: .infinity, alignment: .leading)
.onLongPressGesture(minimumDuration: 0.05, pressing: { pressing in
// Text-selection interaction begins with long-press on the row;
// suspend auto-follow while pressing/held and restore when the
// press ends so following can resume.
selectionActive = pressing
}, perform: {})
.contextMenu {
Button("Copy message", systemImage: "doc.on.doc") { UIPasteboard.general.string = text }
}
.accessibilityElement(children: .combine)
.accessibilityFocused($focusedEventID, equals: event.id)
.id(event.id)
}
private func speakerLabel(_ isUser: Bool) -> some View {
Text(isUser ? "You" : "Hope")
.font(NearlightFont.metadata(typeSize).weight(.semibold))
.foregroundStyle(Nearlight.ink)
}
private func timeLabel(_ event: TranscriptEvent) -> some View {
Group {
if event.isFinal {
Text(event.timestamp, style: .time)
} else {
Text("Live")
}
}
.font(NearlightFont.metadata(typeSize))
.foregroundStyle(Nearlight.quiet)
.monospacedDigit()
}
}
private struct TranscriptViewportHeightKey: PreferenceKey {
static var defaultValue: CGFloat = 0
static func reduce(value: inout CGFloat, nextValue: () -> CGFloat) {
value = nextValue()
}
}
private struct TranscriptLatestBottomKey: PreferenceKey {
static var defaultValue: CGFloat = 0
static func reduce(value: inout CGFloat, nextValue: () -> CGFloat) {
value = nextValue()
}
}
-516
View File
@@ -1,516 +0,0 @@
import SwiftUI
struct MediaPanelView: View {
@ObservedObject var playerController: MediaPlayerController
@ObservedObject var settings: SettingsStore
@State private var searchQuery = ""
@State private var searchResults: [MediaSearchResult]?
@State private var isSearching = false
@State private var searchError: String?
@State private var libraryTracks: [MediaTrack]?
@State private var libraryError: String?
@State private var streamName = ""
@State private var streamURL = ""
@State private var streamMessage: String?
@State private var trackToDelete: MediaTrack?
var body: some View {
ScrollView {
VStack(spacing: 20) {
if let stateError = playerController.stateError {
Label(stateError, systemImage: "wifi.exclamationmark")
.font(.subheadline)
.foregroundStyle(.red)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(12)
.background(Color.red.opacity(0.1), in: RoundedRectangle(cornerRadius: 14))
}
nowPlayingCard
transportControls
seekSection
volumeSection
pendingSection
choicesSection
searchSection
librarySection
addStreamSection
}
.padding()
}
.navigationTitle("Media")
.onAppear {
playerController.startDefaultPolling()
refreshLibrary()
}
.alert("Remove from library?", isPresented: Binding(
get: { trackToDelete != nil },
set: { if !$0 { trackToDelete = nil } }
)) {
Button("Remove", role: .destructive) {
guard let track = trackToDelete else { return }
Task { await deleteTrack(track) }
}
Button("Cancel", role: .cancel) { trackToDelete = nil }
} message: {
Text(trackToDelete?.title ?? "")
}
}
private var nowPlayingCard: some View {
HStack(spacing: 12) {
if let artURL = playerController.track?.artURL, let url = URL(string: absolute(artURL)) {
AsyncImage(url: url) { phase in
switch phase {
case .success(let image):
image.resizable().scaledToFill()
default:
Image(systemName: "music.note")
.font(.title2)
.foregroundStyle(.secondary)
.frame(maxWidth: .infinity, maxHeight: .infinity)
.background(Color(.systemGray6))
}
}
.frame(width: 72, height: 72)
.clipShape(RoundedRectangle(cornerRadius: 10))
} else {
RoundedRectangle(cornerRadius: 10)
.fill(Color(.systemGray6))
.frame(width: 72, height: 72)
.overlay(
Image(systemName: "music.note")
.font(.title2)
.foregroundStyle(.secondary)
)
}
VStack(alignment: .leading, spacing: 4) {
Text(title)
.font(.headline)
.lineLimit(2)
Text(subtitle)
.font(.subheadline)
.foregroundStyle(.secondary)
.lineLimit(1)
if playerController.track != nil {
Text(statusText)
.font(.caption)
.foregroundStyle(playerController.desired == "playing" ? Color.green : Color.secondary)
}
}
Spacer()
}
.padding(12)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14))
}
private var transportControls: some View {
HStack(spacing: 24) {
Button(action: seekBackward) {
Image(systemName: "gobackward.10")
.font(.title3)
}
.accessibilityLabel("Seek back 10 seconds")
.disabled(playerController.track == nil)
Button(action: togglePlayPause) {
Image(systemName: playerController.desired == "playing" && playerController.track != nil ? "pause.circle.fill" : "play.circle.fill")
.font(.system(size: 44))
.foregroundStyle(playerController.track == nil ? Color.secondary : Color.accentColor)
}
.accessibilityLabel(playerController.desired == "playing" && playerController.track != nil ? "Pause" : "Play")
.disabled(playerController.track == nil)
Button(action: seekForward) {
Image(systemName: "goforward.10")
.font(.title3)
}
.accessibilityLabel("Seek forward 10 seconds")
.disabled(playerController.track == nil)
Button(role: .destructive, action: stopPlayback) {
Image(systemName: "stop.circle")
.font(.title2)
}
.accessibilityLabel("Stop playback")
.disabled(playerController.track == nil)
}
}
private var seekSection: some View {
VStack(spacing: 4) {
if playerController.track != nil {
Slider(
value: Binding(
get: { playerController.currentTime },
set: { newValue in Task { playerController.seek(to: newValue) } }
),
in: 0...max(playerController.duration, 1)
)
HStack {
Text(fmtTime(playerController.currentTime))
.font(.caption.monospacedDigit())
.foregroundStyle(.secondary)
Spacer()
Text(fmtTime(playerController.duration))
.font(.caption.monospacedDigit())
.foregroundStyle(.secondary)
}
} else {
Text("Nothing loaded")
.font(.subheadline)
.foregroundStyle(.secondary)
.frame(maxWidth: .infinity, alignment: .center)
.padding(.vertical, 8)
}
}
}
private var volumeSection: some View {
HStack(spacing: 12) {
Image(systemName: "speaker.fill")
.foregroundStyle(.secondary)
Slider(value: $settings.volume, in: 0...1)
.onChange(of: settings.volume) { _, newValue in
playerController.setVolume(newValue)
}
Image(systemName: "speaker.wave.3.fill")
.foregroundStyle(.secondary)
}
}
@ViewBuilder
private var pendingSection: some View {
if !playerController.pendingJobs.isEmpty {
VStack(alignment: .leading, spacing: 8) {
ForEach(playerController.pendingJobs, id: \.jobID) { job in
HStack {
Text(job.stage == "queued" ? "Queued…" : "Downloading…")
.font(.subheadline)
Spacer()
Text("\(Int((job.progress ?? 0)))%")
.font(.caption.monospacedDigit())
.foregroundStyle(.secondary)
}
}
}
.padding(12)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14))
}
}
@ViewBuilder
private var choicesSection: some View {
if !playerController.choices.isEmpty {
VStack(alignment: .leading, spacing: 8) {
Text("Hope's suggestions")
.font(.headline)
ForEach(playerController.choices, id: \.videoID) { choice in
mediaRow(
title: choice.title ?? choice.videoID ?? "Unknown",
detail: rowDetail(channel: choice.channel, duration: choice.duration),
playEnabled: true
) {
Task {
await playerController.playVideo(
videoID: choice.videoID,
title: choice.title,
channel: choice.channel
)
}
}
}
}
}
}
private var searchSection: some View {
VStack(alignment: .leading, spacing: 8) {
Text("Search YouTube Music")
.font(.headline)
HStack(spacing: 8) {
TextField("Song, artist, lyrics…", text: $searchQuery)
.textFieldStyle(.roundedBorder)
.submitLabel(.search)
.onSubmit(performSearch)
Button("Find", action: performSearch)
.buttonStyle(.bordered)
.disabled(searchQuery.trimmingCharacters(in: .whitespaces).isEmpty || isSearching)
}
if isSearching {
HStack(spacing: 8) {
ProgressView()
Text("Searching YouTube Music…")
.font(.subheadline)
.foregroundStyle(.secondary)
}
} else if let searchError {
Text(searchError)
.font(.subheadline)
.foregroundStyle(.red)
} else if let results = searchResults {
if results.isEmpty {
Text("No results.")
.font(.subheadline)
.foregroundStyle(.secondary)
} else {
ForEach(results, id: \.videoID) { result in
mediaRow(
title: result.title ?? result.videoID ?? "Unknown",
detail: rowDetail(channel: result.channel, duration: result.duration, inLibrary: result.inLibrary),
playEnabled: true
) {
Task {
if let trackID = result.trackID {
await playerController.playTrack(id: trackID)
} else {
await playerController.playVideo(
videoID: result.videoID,
title: result.title,
channel: result.channel
)
}
}
}
}
}
}
}
}
private var librarySection: some View {
VStack(alignment: .leading, spacing: 8) {
HStack {
Text("Library")
.font(.headline)
Spacer()
Button(action: refreshLibrary) {
Image(systemName: "arrow.clockwise")
}
.accessibilityLabel("Refresh library")
}
if let libraryError {
Text(libraryError)
.font(.subheadline)
.foregroundStyle(.red)
} else if let tracks = libraryTracks {
if tracks.isEmpty {
Text("Downloaded songs and streams live here.")
.font(.subheadline)
.foregroundStyle(.secondary)
} else {
ForEach(tracks, id: \.id) { track in
HStack(spacing: 8) {
mediaRow(
title: track.title ?? track.id,
detail: rowDetail(
channel: track.channel ?? (track.kind == "stream" ? "Live stream" : ""),
duration: track.duration,
status: track.status
),
playEnabled: track.status == "ready"
) {
Task { await playerController.playTrack(id: track.id) }
}
Button(role: .destructive) {
trackToDelete = track
} label: {
Image(systemName: "xmark")
.font(.subheadline.weight(.semibold))
.foregroundStyle(.red)
}
.accessibilityLabel("Remove \(track.title ?? track.id)")
}
}
}
}
}
}
private var addStreamSection: some View {
VStack(alignment: .leading, spacing: 8) {
Text("Add live stream")
.font(.headline)
TextField("Station name", text: $streamName)
.textFieldStyle(.roundedBorder)
TextField("https://… audio stream URL", text: $streamURL)
.textFieldStyle(.roundedBorder)
.keyboardType(.URL)
.autocorrectionDisabled()
.textContentType(.URL)
.textInputAutocapitalization(.never)
if let streamMessage {
Text(streamMessage)
.font(.subheadline)
.foregroundStyle(streamMessage.hasPrefix("Added") ? Color.green : Color.red)
}
Button("Add", action: addStream)
.buttonStyle(.bordered)
.disabled(streamURL.trimmingCharacters(in: .whitespaces).isEmpty)
}
}
private func mediaRow(
title: String,
detail: String,
playEnabled: Bool,
onPlay: @escaping () -> Void
) -> some View {
HStack(spacing: 10) {
VStack(alignment: .leading, spacing: 2) {
Text(title)
.font(.subheadline.weight(.semibold))
.lineLimit(1)
if !detail.isEmpty {
Text(detail)
.font(.caption)
.foregroundStyle(.secondary)
.lineLimit(1)
}
}
Spacer()
Button("Play", action: onPlay)
.buttonStyle(.bordered)
.controlSize(.small)
.disabled(!playEnabled)
}
.padding(10)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 12))
}
private func rowDetail(channel: String?, duration: Double?, status: String? = nil, inLibrary: Bool? = nil) -> String {
var parts: [String] = []
if let channel, !channel.isEmpty { parts.append(channel) }
if let duration, duration > 0 { parts.append(fmtTime(duration)) }
if let status, status != "ready" { parts.append(status) }
if inLibrary == true { parts.append("in library") }
return parts.joined(separator: " · ")
}
private func performSearch() {
let query = searchQuery.trimmingCharacters(in: .whitespaces)
guard !query.isEmpty else { return }
isSearching = true
searchError = nil
Task {
do {
let response = try await MediaAPI(config: serverConfig).search(room: "default", query: query, limit: 6)
searchResults = response.results
} catch {
searchError = error.localizedDescription
}
isSearching = false
}
}
private func refreshLibrary() {
Task {
do {
let library = try await MediaAPI(config: serverConfig).library()
libraryTracks = library.tracks
libraryError = nil
} catch {
libraryError = error.localizedDescription
}
}
}
private func addStream() {
let url = streamURL.trimmingCharacters(in: .whitespaces)
guard !url.isEmpty else { return }
let name = streamName.trimmingCharacters(in: .whitespaces).isEmpty ? url : streamName.trimmingCharacters(in: .whitespaces)
Task {
do {
try await MediaAPI(config: serverConfig).addStream(room: "default", name: name, url: url)
streamMessage = "Added \(name)"
streamName = ""
streamURL = ""
refreshLibrary()
} catch {
streamMessage = "Could not add stream: \(error.localizedDescription)"
}
}
}
private func deleteTrack(_ track: MediaTrack) async {
do {
try await MediaAPI(config: serverConfig).deleteTrack(id: track.id)
if playerController.track?.id == track.id {
Task { await playerController.stop() }
}
refreshLibrary()
} catch {
libraryError = error.localizedDescription
}
}
private var serverConfig: ServerConfig {
// Use the configured URL directly. If it's invalid, fall back to the
// documented default (media operations are less critical than the main
// session, and the Settings view will show a validation error).
if case .success(let config) = ServerConfig.parse(settings.trimmedServerURL) {
return config
}
let fallback = ServerConfig.parse("https://192.168.86.2:9178")
if case .success(let config) = fallback {
return config
}
fatalError("Unreachable: default server URL is valid")
}
private var title: String {
playerController.track?.title ?? "Nothing loaded"
}
private var subtitle: String {
guard let track = playerController.track else { return "" }
if track.status == "failed" {
return "Download failed — pick another result or try again."
}
if track.status == "downloading" {
return track.channel.map { "\($0) · downloading" } ?? "Downloading…"
}
return track.channel ?? (track.kind == "stream" ? "Live stream" : "")
}
private var statusText: String {
switch playerController.desired {
case "playing": return "Playing"
case "paused": return "Paused"
default: return "Stopped"
}
}
private func absolute(_ path: String) -> String {
if path.hasPrefix("http") { return path }
let base = serverConfig.baseURL.absoluteString
let suffix = path.hasPrefix("/") ? path : "/" + path
return base + suffix
}
private func togglePlayPause() {
Task { await playerController.togglePlayPause() }
}
private func seekBackward() {
Task { await playerController.seek(by: -10) }
}
private func seekForward() {
Task { await playerController.seek(by: 10) }
}
private func stopPlayback() {
Task { await playerController.stop() }
}
private func fmtTime(_ sec: Double) -> String {
guard sec.isFinite, sec >= 0 else { return "0:00" }
let total = Int(sec)
let s = total % 60
let m = (total / 60) % 60
let h = total / 3600
return h > 0 ? "\(h):\(String(format: "%02d", m)):\(String(format: "%02d", s))" : "\(m):\(String(format: "%02d", s))"
}
}
@@ -26,7 +26,7 @@ struct ConnectionTestView: View {
if isBusy {
ProgressView()
} else {
Text("Get token")
Text("Test connection")
}
Spacer()
}
@@ -44,7 +44,7 @@ struct ConnectionTestView: View {
LabeledContent("User") {
Text(response.user.displayName ?? response.user.id)
}
Label("Token endpoint and TLS are working.", systemImage: "checkmark.circle")
Label("Connected to your server. Start a conversation to check Hope’s voice and replies.", systemImage: "checkmark.circle")
.foregroundStyle(.green)
}
case .failure(let error):
@@ -57,6 +57,7 @@ struct ConnectionTestView: View {
}
}
.navigationTitle("Connection test")
.navigationBarTitleDisplayMode(.inline)
}
private func runTest() {
+81 -38
View File
@@ -12,8 +12,9 @@ struct SettingsView: View {
@State private var urlText: String = ""
@State private var urlError: String?
@State private var showVoicePicker = false
@State private var savedURLConfirmation: String?
@State private var showModelPicker = false
@FocusState private var urlFocused: Bool
init(settings: SettingsStore, session: VoiceSession) {
self.settings = settings
self.session = session
@@ -30,56 +31,92 @@ struct SettingsView: View {
var body: some View {
Form {
Section("Server") {
Section {
Button {
showVoicePicker = true
} label: {
LabeledContent("Speaking voice", value: Voices.display(for: settings.voiceID) ?? settings.voiceID)
}
Picker("Conversation mode", selection: $settings.textMode) {
Text("Voice").tag(false)
Text("Type · Microphone off").tag(true)
}
} header: {
Text("Make it yours")
} footer: {
Text("Voice and typing preferences are saved on this iPhone. You can switch modes from the Talk tab.")
}
Section {
if session.isReady, !session.models.isEmpty {
Button {
showModelPicker = true
} label: {
LabeledContent("Model", value: session.selectedModel ?? settings.modelID ?? "Server default")
}
} else {
LabeledContent("Model", value: settings.modelID ?? "Server default")
Text("Connect from Talk to see the models available on your server.")
.font(.footnote).foregroundStyle(Nearlight.quiet)
}
} header: { Text("Intelligence") }
Section {
Text("Server URL")
.font(.footnote.weight(.semibold))
.foregroundStyle(Nearlight.quiet)
TextField(SettingsStore.defaultServerURL, text: $urlText)
.keyboardType(.URL)
.autocorrectionDisabled()
.textInputAutocapitalization(.never)
.accessibilityLabel("Server URL")
.focused($urlFocused)
.onAppear { urlText = settings.serverURL }
if let urlError {
Text(urlError)
.font(.footnote)
.foregroundStyle(.red)
.onChange(of: urlText) { _, newValue in
// Keep confirmation across the asynchronous onChange
// caused by successful whitespace normalization.
guard newValue != savedURLConfirmation else { return }
savedURLConfirmation = nil
urlError = nil
}
Button("Apply server URL") { applyServerURL() }
.disabled(!canApplyServerURL || urlText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty)
.overlay(alignment: .trailing) {
if savedURLConfirmation != nil {
Image(systemName: "checkmark.circle")
.foregroundStyle(Nearlight.quiet)
.padding(.trailing, 8)
.accessibilityHidden(true)
}
}
if let urlError {
Label(urlError, systemImage: "exclamationmark.circle")
.font(.footnote)
.foregroundStyle(Nearlight.alert)
.accessibilityIdentifier("settings.server.error")
} else if savedURLConfirmation != nil {
Text("Server address saved.")
.font(.footnote)
.foregroundStyle(Nearlight.quiet)
.accessibilityIdentifier("settings.server.saved")
}
Button("Save server address") { applyServerURL() }
.disabled(!canApplyServerURL || urlText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || urlText == settings.serverURL)
if !canApplyServerURL {
Text(session.state == .tearingDown
? "Waiting for failure cleanup to finish…"
: "End the conversation before changing servers.")
.font(.footnote)
.foregroundStyle(.secondary)
.foregroundStyle(Nearlight.quiet)
}
}
Section("Voice") {
Button {
showVoicePicker = true
} label: {
LabeledContent("Voice", value: Voices.display(for: settings.voiceID) ?? settings.voiceID)
}
}
Section("Model") {
if session.isReady, !session.models.isEmpty {
Button {
showModelPicker = true
} label: {
LabeledContent("Model", value: session.selectedModel ?? settings.modelID ?? "—")
}
} else {
Text("Start a conversation to load the model list.")
.font(.footnote)
.foregroundStyle(.secondary)
}
}
Section("Interaction") {
Toggle("Text mode", isOn: $settings.textMode)
}
Section {
NavigationLink("Connection test") {
ConnectionTestView(settings: settings)
}
} header: {
Text("Connection")
} footer: {
Text("Hope runs on your own server. Your iPhone needs to be on the same network or a network that can reach it.")
}
}
.navigationTitle("Settings")
.navigationBarTitleDisplayMode(.inline)
.tint(Nearlight.quiet)
.sheet(isPresented: $showVoicePicker) {
VoicePickerView(settings: settings, session: session)
}
@@ -93,6 +130,7 @@ struct SettingsView: View {
/// left untouched so the active session keeps using its original URL.
/// Guarded: only applies when the session is idle (no active call).
private func applyServerURL() {
savedURLConfirmation = nil
guard canApplyServerURL else {
urlError = "End the conversation before changing servers."
return
@@ -107,15 +145,18 @@ struct SettingsView: View {
settings.serverURL = trimmed
urlText = trimmed
urlError = nil
savedURLConfirmation = trimmed
urlFocused = false
case .failure(let error):
urlError = error.localizedDescription
urlFocused = true
}
}
}
struct VoicePickerView: View {
@ObservedObject var settings: SettingsStore
var session: VoiceSession
@ObservedObject var session: VoiceSession
@Environment(\.dismiss) private var dismiss
var body: some View {
@@ -132,10 +173,11 @@ struct VoicePickerView: View {
if settings.voiceID == voice.id {
Spacer()
Image(systemName: "checkmark")
.foregroundStyle(.blue)
.foregroundStyle(Nearlight.rosewood)
}
}
}
.accessibilityAddTraits(settings.voiceID == voice.id ? .isSelected : [])
}
}
.navigationTitle("Voice")
@@ -150,7 +192,7 @@ struct VoicePickerView: View {
struct ModelPickerView: View {
@ObservedObject var settings: SettingsStore
var session: VoiceSession
@ObservedObject var session: VoiceSession
@Environment(\.dismiss) private var dismiss
private var activeModelID: String? {
@@ -176,10 +218,11 @@ struct ModelPickerView: View {
if activeModelID == model.id {
Spacer()
Image(systemName: "checkmark")
.foregroundStyle(.blue)
.foregroundStyle(Nearlight.rosewood)
}
}
}
.accessibilityAddTraits(activeModelID == model.id ? .isSelected : [])
}
}
.navigationTitle("Model")
+211 -83
View File
@@ -1,137 +1,265 @@
import SwiftUI
/// Nearlight Activity: a working log. Plain rows on Ground, no cards;
/// live updates land in place without reordering while reading.
struct TaskPanelView: View {
@ObservedObject var taskStore: TaskStore
@Environment(\.dynamicTypeSize) private var typeSize
var body: some View {
ScrollView {
VStack(spacing: 12) {
Group {
if taskStore.sortedTasks.isEmpty {
emptyState
} else {
taskList
}
}
.navigationTitle("Activity")
.background(Nearlight.ground)
}
private var taskList: some View {
ScrollView {
LazyVStack(alignment: .leading, spacing: 0) {
ForEach(taskStore.sortedTasks) { task in
TaskCardView(task: task)
NavigationLink {
TaskDetailView(taskStore: taskStore, taskID: task.id)
} label: {
TaskRowView(task: task)
}
.buttonStyle(.plain)
Divider()
.overlay(Nearlight.separator)
}
}
.padding(.horizontal, Nearlight.s20)
}
.padding()
}
.navigationTitle("Background tasks")
#if DEBUG
.toolbar {
ToolbarItem(placement: .topBarTrailing) {
Button("Sample", action: { taskStore.injectSampleEvents() })
.font(.caption)
}
}
#endif
}
private var emptyState: some View {
VStack(spacing: 8) {
Text("Nothing running")
.font(.headline)
Text("Ask Hope to research or compare something.")
.font(.subheadline)
.foregroundStyle(.secondary)
.multilineTextAlignment(.center)
VStack(alignment: .leading, spacing: Nearlight.s8) {
Image(systemName: "checklist")
.font(.system(size: 28))
.foregroundStyle(Nearlight.quiet)
Text("No tasks yet")
.font(NearlightFont.heading(typeSize))
Text("Tasks from your conversations will appear here.")
.font(.body)
.foregroundStyle(Nearlight.quiet)
}
.frame(maxWidth: .infinity)
.padding(.vertical, 48)
.padding(.top, Nearlight.s32)
.padding(.horizontal, Nearlight.s20)
.frame(maxWidth: .infinity, alignment: .leading)
.accessibilityIdentifier("activity.empty")
}
}
struct TaskCardView: View {
/// One activity row: status symbol, title, current step/status, chevron.
private struct TaskRowView: View {
let task: TaskItem
@Environment(\.dynamicTypeSize) private var typeSize
var body: some View {
VStack(alignment: .leading, spacing: 8) {
HStack(spacing: 8) {
statusDot
HStack(spacing: Nearlight.s12) {
Image(systemName: statusSymbol)
.font(.system(size: 20))
.foregroundStyle(statusColor)
.accessibilityHidden(true)
VStack(alignment: .leading, spacing: Nearlight.s4) {
Text(statusTitle)
.font(NearlightFont.support(typeSize))
.foregroundStyle(statusColor)
.fixedSize(horizontal: false, vertical: true)
Text(task.description)
.font(.subheadline.weight(.semibold))
.lineLimit(2)
Spacer()
.font(NearlightFont.control(typeSize))
.foregroundStyle(Nearlight.ink)
.fixedSize(horizontal: false, vertical: true)
if let latest = latestStep {
Text(latest)
.font(NearlightFont.support(typeSize))
.foregroundStyle(Nearlight.quiet)
.fixedSize(horizontal: false, vertical: true)
}
if !task.steps.isEmpty {
VStack(alignment: .leading, spacing: 4) {
Text(task.createdAt, style: .time)
.font(NearlightFont.metadata(typeSize))
.foregroundStyle(Nearlight.quiet)
.monospacedDigit()
}
Spacer(minLength: 0)
Image(systemName: "chevron.right")
.font(.system(size: 13, weight: .medium))
.foregroundStyle(Nearlight.quiet)
.accessibilityHidden(true)
}
.padding(.vertical, Nearlight.s16)
.contentShape(Rectangle())
}
private var statusTitle: String {
switch task.status {
case .pending: return "Queued"
case .running: return "In progress"
case .completed: return "Complete"
case .failed: return "Needs attention"
}
}
private var latestStep: String? {
if let error = task.error { return error }
if let result = task.result { return result }
return task.steps.last?.content
}
private var statusSymbol: String {
switch task.status {
case .pending: return "circle.dotted"
case .running: return "arrow.triangle.2.circlepath"
case .completed: return "checkmark.circle"
case .failed: return "exclamationmark.circle"
}
}
private var statusColor: Color {
switch task.status {
case .pending: return Nearlight.quiet
case .running: return Nearlight.rosewood
case .completed: return Nearlight.quiet
case .failed: return Nearlight.alert
}
}
}
/// Native push detail for one task: title, status, steps, result/error.
/// Looks the task up live in the store so removal shows the missing state.
struct TaskDetailView: View {
@ObservedObject var taskStore: TaskStore
let taskID: String
@Environment(\.dynamicTypeSize) private var typeSize
private var task: TaskItem? {
taskStore.tasks[taskID]
}
var body: some View {
Group {
if let task {
TaskDetailContent(task: task)
} else {
Text("This task is no longer available.")
.font(.body)
.foregroundStyle(Nearlight.quiet)
.padding(Nearlight.s20)
.frame(maxWidth: .infinity, alignment: .leading)
}
}
.background(Nearlight.ground)
.navigationTitle("Task")
.navigationBarTitleDisplayMode(.inline)
}
}
private struct TaskDetailContent: View {
let task: TaskItem
@Environment(\.dynamicTypeSize) private var typeSize
var body: some View {
ScrollView {
VStack(alignment: .leading, spacing: Nearlight.s24) {
VStack(alignment: .leading, spacing: Nearlight.s12) {
Text(task.description)
.font(NearlightFont.heading(typeSize))
.fixedSize(horizontal: false, vertical: true)
Label(statusTitle, systemImage: statusSymbol)
.font(NearlightFont.support(typeSize))
.foregroundStyle(statusColor)
Text(task.createdAt, style: .time)
.font(NearlightFont.metadata(typeSize))
.foregroundStyle(Nearlight.quiet)
.monospacedDigit()
}
VStack(alignment: .leading, spacing: 0) {
Text("Steps")
.font(NearlightFont.heading(typeSize))
.padding(.bottom, Nearlight.s8)
if task.steps.isEmpty {
Text("No steps reported yet.")
.font(.body)
.foregroundStyle(Nearlight.quiet)
} else {
ForEach(task.steps) { step in
HStack(alignment: .top, spacing: 6) {
Text(stepLabel(step.role))
.font(.caption2.weight(.semibold))
.foregroundStyle(stepColor(step.role))
Text(step.content ?? "")
.font(.caption)
.foregroundStyle(stepColor(step.role))
.lineLimit(3)
VStack(alignment: .leading, spacing: Nearlight.s4) {
if let role = step.role {
Text(roleLabel(role))
.font(NearlightFont.support(typeSize).weight(.semibold))
.foregroundStyle(Nearlight.quiet)
}
Text(step.content ?? "")
.font(.body)
.foregroundStyle(Nearlight.ink)
.textSelection(.enabled)
.fixedSize(horizontal: false, vertical: true)
}
.padding(.vertical, Nearlight.s16)
}
}
}
if let result = task.result {
block(result, color: .green, background: Color.green.opacity(0.12))
Text(result)
.font(.body)
.foregroundStyle(Nearlight.ink)
.textSelection(.enabled)
}
if let error = task.error {
block(error, color: .red, background: Color.red.opacity(0.12))
Text(error)
.font(.body)
.foregroundStyle(Nearlight.alert)
.textSelection(.enabled)
}
}
.padding(12)
.frame(maxWidth: .infinity, alignment: .leading)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14))
.padding(.horizontal, Nearlight.s20)
.padding(.top, Nearlight.s24)
.padding(.bottom, Nearlight.s24)
}
}
private var statusDot: some View {
Circle()
.fill(dotColor)
.frame(width: 9, height: 9)
}
private var dotColor: Color {
private var statusTitle: String {
switch task.status {
case .pending: return .gray
case .running: return .blue
case .completed: return .green
case .failed: return .red
case .pending: return "Queued"
case .running: return "In progress"
case .completed: return "Complete"
case .failed: return "Needs attention"
}
}
private func block(_ text: String, color: Color, background: Color) -> some View {
Text(text)
.font(.subheadline)
.foregroundStyle(color)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(10)
.background(background, in: RoundedRectangle(cornerRadius: 10))
private var statusSymbol: String {
switch task.status {
case .pending: return "circle.dotted"
case .running: return "arrow.triangle.2.circlepath"
case .completed: return "checkmark.circle"
case .failed: return "exclamationmark.circle"
}
}
private func stepLabel(_ role: String?) -> String {
private var statusColor: Color {
switch task.status {
case .pending: return Nearlight.quiet
case .running: return Nearlight.rosewood
case .completed: return Nearlight.quiet
case .failed: return Nearlight.alert
}
}
private func roleLabel(_ role: String) -> String {
switch role {
case "thinking": return "Thinking"
case "tool_call": return "Tool"
case "tool_result": return "Result"
case "text": return "Note"
default: return role ?? "Step"
}
}
private func stepColor(_ role: String?) -> Color {
switch role {
case "thinking": return .blue
case "tool_call": return .indigo
case "tool_result": return .secondary
default: return .primary
default: return role
}
}
}
#Preview {
NavigationStack {
TaskPanelView(taskStore: {
let store = TaskStore()
#if DEBUG
store.injectSampleEvents()
#endif
return store
}())
TaskPanelView(taskStore: TaskStore())
}
}
+1 -6
View File
@@ -2,7 +2,7 @@ import Foundation
import LiveKit
/// Outbound helpers for agent control topics and inbound parsing of
/// voice-control / conversation-control / media-control data messages.
/// voice-control / conversation-control data messages.
@MainActor
final class ControlChannel: NSObject, RoomDelegate {
@@ -109,11 +109,6 @@ final class ControlChannel: NSObject, RoomDelegate {
try await publish(["type": "set_mode", "mode": mode], topic: "conversation-control", room: room)
}
func sendMediaState(_ state: String, trackID: String?, room: Room) async throws {
var object: [String: Any] = ["type": "media_state", "state": state]
if let trackID { object["track_id"] = trackID }
try await publish(object, topic: "media-control", room: room)
}
}
// MARK: - Minimal JSON value decoder for flexible agent payloads
+2 -2
View File
@@ -20,7 +20,7 @@ final class TranscriptHistory: ObservableObject {
static let maxEntries = 50
init() {
if let raw = UserDefaults.standard.string(forKey: Self.key),
if let raw = AppPreferences.store.string(forKey: Self.key),
let data = raw.data(using: .utf8),
let decoded = try? JSONDecoder().decode([HistoryEntry].self, from: data) {
entries = Array(decoded.suffix(Self.maxEntries))
@@ -45,6 +45,6 @@ final class TranscriptHistory: ObservableObject {
private func persist() {
guard let data = try? JSONEncoder().encode(entries) else { return }
UserDefaults.standard.set(String(decoding: data, as: UTF8.self), forKey: Self.key)
AppPreferences.store.set(String(decoding: data, as: UTF8.self), forKey: Self.key)
}
}
+4 -14
View File
@@ -48,7 +48,6 @@ final class VoiceSession: ObservableObject {
let transcriptHandler = TranscriptHandler()
let stateModel = ConversationStateModel()
let taskStore = TaskStore()
let mediaController: MediaPlayerController
private let config: ServerConfig
private let tokenClient: TokenClient
private let settings: SettingsStore
@@ -90,9 +89,6 @@ final class VoiceSession: ObservableObject {
self.config = config
self.tokenClient = tokenClient
self.settings = settings
let mediaAPI = MediaAPI(config: config)
self.mediaController = MediaPlayerController(api: mediaAPI, settings: settings)
mediaController.attach(session: self)
room.add(delegate: self)
room.add(delegate: controlChannel)
room.add(delegate: transcriptHandler)
@@ -211,7 +207,6 @@ final class VoiceSession: ObservableObject {
do {
let token = try await tokenClient.fetchToken()
logger.log("token ok room=\(token.room, privacy: .public)")
await mediaController.enterMediaScope(roomName: token.room)
try await room.connect(url: config.livekitURL.absoluteString, token: token.token, connectOptions: makeConnectOptions())
result = .success(())
} catch {
@@ -263,8 +258,7 @@ final class VoiceSession: ObservableObject {
private func handleAgentTimeout(generation: Int) async {
guard generation == sessionGeneration, state == .waitingAgent else { return }
guard !reconnectAttempted else {
// Final failure: tear down the live room, restore the idle timer,
// and end the media scope so no stale room or wake lock survives.
// Final failure: tear down the live room and restore the idle timer.
logger.log("agent timeout final, tearing down")
await teardownForFailure(message: "Agent never joined. Is the agent worker running?")
return
@@ -281,8 +275,7 @@ final class VoiceSession: ObservableObject {
/// current generation (so stale callbacks become no-ops), clear all
/// per-room/session ephemeral published fields (agent, mute state, model
/// list, error state, interruption ownership), disconnect and recreate the
/// room, stop conversation state, restore the idle timer, and end the media
/// scope. Only after all cleanup completes does the session publish
/// room, stop conversation state, restore the idle timer, and publish
/// `.failed(message)`, making retry safe: the old room is fully torn down
/// before Start becomes available again.
///
@@ -308,7 +301,6 @@ final class VoiceSession: ObservableObject {
interruptionSequence += 1
stateModel.stop()
UIApplication.shared.isIdleTimerDisabled = false
await mediaController.endConversation()
await teardownRoom()
// Final reset immediately before publishing .failed: any old-room
@@ -540,15 +532,13 @@ final class VoiceSession: ObservableObject {
interruptionSequence += 1
stateModel.stop()
UIApplication.shared.isIdleTimerDisabled = false
await mediaController.endConversation()
await teardownRoom()
// Final reset before publishing .idle: any old-room callback delivered
// during the teardownRoom() await may have repopulated ephemeral fields.
// This ensures the idle state is clean for server replacement (Iteration 09).
resetEphemeralFields()
// Publish .idle only after media and room teardown have completed.
// This makes idle a truthful readiness signal for server replacement:
// the old session is fully torn down before the UI shows "ready to start".
// Publish .idle only after room teardown has completed: the old session
// is fully torn down before the UI shows "ready to start".
state = .idle
}
-249
View File
@@ -1,249 +0,0 @@
import Foundation
struct MediaTrack: Decodable, Equatable {
let id: String
let kind: String?
let videoID: String?
let title: String?
let channel: String?
let duration: Double?
let status: String?
let error: String?
let audioURL: String?
let artURL: String?
enum CodingKeys: String, CodingKey {
case id, kind, title, channel, duration, status, error
case videoID = "video_id"
case audioURL = "audio_url"
case artURL = "art_url"
}
}
struct MediaPendingJob: Decodable, Equatable {
let jobID: String?
let trackID: String?
let stage: String?
let progress: Double?
let error: String?
enum CodingKeys: String, CodingKey {
case stage, progress, error
case jobID = "job_id"
case trackID = "track_id"
}
}
struct MediaSeekHint: Decodable, Equatable {
let ts: Double
let position: Double
}
struct MediaState: Decodable, Equatable {
let room: String?
let track: MediaTrack?
let desired: String?
let reported: MediaReported?
let seekHint: MediaSeekHint?
let choices: [MediaChoice]
let pending: [MediaPendingJob]
let version: Int?
enum CodingKeys: String, CodingKey {
case room, track, desired, reported, choices, pending, version
case seekHint = "seek_hint"
}
}
struct MediaReported: Decodable, Equatable {
let trackID: String?
let state: String?
let position: Double?
let duration: Double?
let ts: Double?
enum CodingKeys: String, CodingKey {
case state, position, duration, ts
case trackID = "track_id"
}
}
struct MediaChoice: Decodable, Equatable {
let videoID: String?
let title: String?
let channel: String?
let duration: Double?
enum CodingKeys: String, CodingKey {
case title, channel, duration
case videoID = "video_id"
}
}
struct MediaLibrary: Decodable, Equatable {
let tracks: [MediaTrack]
let version: Int?
}
struct MediaSearchResult: Decodable, Equatable {
let videoID: String?
let trackID: String?
let title: String?
let channel: String?
let duration: Double?
let inLibrary: Bool?
enum CodingKeys: String, CodingKey {
case title, channel, duration
case videoID = "video_id"
case trackID = "track_id"
case inLibrary = "in_library"
}
}
struct MediaSearchResponse: Decodable, Equatable {
let results: [MediaSearchResult]
}
enum MediaAPIError: LocalizedError {
case invalidServerURL(String)
case httpStatus(Int, String)
case decodeFailed
var errorDescription: String? {
switch self {
case .invalidServerURL(let raw):
return "Invalid server URL: \(raw)"
case .httpStatus(let code, let detail):
return "Media API request failed with HTTP \(code). \(detail)"
case .decodeFailed:
return "Unexpected response from the media endpoint."
}
}
}
final class MediaAPI {
let serverBase: URL
private let baseURL: URL
private let session: URLSession
init(config: ServerConfig) {
self.serverBase = config.baseURL
self.baseURL = config.baseURL.appendingPathComponent("media/api")
self.session = TLSPinning.session()
}
func state(room: String) async throws -> MediaState {
var components = URLComponents(url: baseURL.appendingPathComponent("state"), resolvingAgainstBaseURL: false)!
components.queryItems = [URLQueryItem(name: "room", value: room)]
return try await get(MediaState.self, url: components.url!)
}
func report(room: String, state: String, trackID: String?, position: Double, duration: Double?) async throws {
var body: [String: Any] = ["room": room, "state": state, "position": position]
if let trackID { body["track_id"] = trackID }
if let duration { body["duration"] = duration }
try await post("report", body: body)
}
func play(room: String, videoID: String? = nil, title: String? = nil, channel: String? = nil, trackID: String? = nil, startPaused: Bool? = nil) async throws {
var body: [String: Any] = ["room": room]
if let videoID { body["video_id"] = videoID }
if let title { body["title"] = title }
if let channel { body["channel"] = channel }
if let trackID { body["track_id"] = trackID }
if let startPaused { body["start_paused"] = startPaused }
try await post("play", body: body)
}
func control(room: String, action: String) async throws {
try await post("control", body: ["room": room, "action": action])
}
func search(room: String, query: String, limit: Int = 6) async throws -> MediaSearchResponse {
try await postDecoded("search", body: ["room": room, "query": query, "limit": limit], as: MediaSearchResponse.self)
}
func library() async throws -> MediaLibrary {
try await get(MediaLibrary.self, url: baseURL.appendingPathComponent("library"))
}
func deleteTrack(id: String) async throws {
var request = URLRequest(url: baseURL.appendingPathComponent("tracks").appending(path: id))
request.httpMethod = "DELETE"
let (data, response) = try await session.data(for: request)
guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed }
guard (200..<300).contains(http.statusCode) else {
throw MediaAPIError.httpStatus(http.statusCode, String(data: data, encoding: .utf8) ?? "")
}
}
func addStream(room: String, name: String, url: String) async throws {
try await post("streams", body: ["room": room, "name": name, "url": url])
}
/// Returns the progressive-media endpoint URL for a track. The `/stream/{id}`
/// path serves raw media bytes (not a text URL), so this constructs the
/// endpoint without downloading or decoding the body.
func streamURL(trackID: String) -> URL {
baseURL.appendingPathComponent("stream").appending(path: trackID)
}
func healthz() async throws -> Bool {
let data = try await getRaw(url: baseURL.appendingPathComponent("healthz"))
guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { return false }
return json["ok"] as? Bool ?? false
}
private func get<T: Decodable>(_ type: T.Type, url: URL) async throws -> T {
let data = try await getRaw(url: url)
do {
return try JSONDecoder().decode(T.self, from: data)
} catch {
throw MediaAPIError.decodeFailed
}
}
private func getRaw(url: URL) async throws -> Data {
let (data, response) = try await session.data(from: url)
guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed }
guard (200..<300).contains(http.statusCode) else {
throw MediaAPIError.httpStatus(http.statusCode, String(data: data, encoding: .utf8) ?? "")
}
return data
}
/// Fire-and-forget POST that only validates HTTP success. The media server
/// returns `{"ok": true, ...snapshot}` for /play, /control, /report and
/// /streams — callers don't need the body, so decoding it (as Data or any
/// typed model) would be wasted work at best and a spurious failure mode.
private func post(_ path: String, body: [String: Any]) async throws {
_ = try await postRaw(path, body: body)
}
private func postDecoded<T: Decodable>(_ path: String, body: [String: Any], as type: T.Type) async throws -> T {
let data = try await postRaw(path, body: body)
do {
return try JSONDecoder().decode(T.self, from: data)
} catch {
throw MediaAPIError.decodeFailed
}
}
private func postRaw(_ path: String, body: [String: Any]) async throws -> Data {
var request = URLRequest(url: baseURL.appendingPathComponent(path))
request.httpMethod = "POST"
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
request.httpBody = try JSONSerialization.data(withJSONObject: body)
let (data, response) = try await session.data(for: request)
guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed }
guard (200..<300).contains(http.statusCode) else {
var detail = String(data: data, encoding: .utf8) ?? ""
if let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], let error = json["error"] as? String {
detail = error
}
throw MediaAPIError.httpStatus(http.statusCode, detail)
}
return data
}
}
-595
View File
@@ -1,595 +0,0 @@
import AVFoundation
import Combine
import Foundation
import os
private let logger = Logger(subsystem: "dev.shane.voice", category: "media")
/// AVPlayer-backed media player. The media server holds per-room desired state;
/// this controller polls /state and reconciles the local player against it,
/// porting the web UI's reconciliation rules (app.js reconcileMedia).
@MainActor
final class MediaPlayerController: ObservableObject {
@Published private(set) var track: MediaTrack?
@Published private(set) var desired: String = "stopped"
@Published private(set) var currentTime: Double = 0
@Published private(set) var duration: Double = 0
@Published private(set) var pendingJobs: [MediaPendingJob] = []
@Published private(set) var choices: [MediaChoice] = []
@Published private(set) var stateError: String?
let player = AVPlayer()
private let api: MediaAPI
private let settings: SettingsStore
private let mediaLoader: PinnedMediaResourceLoader
private weak var session: VoiceSession?
private var room: String?
private var currentTrackID: String?
private var loadedAudioPath: String?
private var currentAsset: AVURLAsset?
private var appliedSeekTs: Double = 0
private var wasPlayingBeforeTalk = false
private var lastReportDate = Date.distantPast
private var pollTask: Task<Void, Never>?
private var timeObserver: Any?
private var statusCancellable: AnyCancellable?
private var rateObservation: NSKeyValueObservation?
/// Bumped on every room-scope change so in-flight /state polls from the
/// previous scope cannot reconcile into the new one (port of the web UI's
/// `=== room` checks).
private var pollGeneration = 0
init(api: MediaAPI, settings: SettingsStore) {
self.api = api
self.settings = settings
self.mediaLoader = PinnedMediaResourceLoader(base: api.serverBase)
player.volume = Float(settings.volume)
NotificationCenter.default.addObserver(
self, selector: #selector(handleInterruption),
name: AVAudioSession.interruptionNotification, object: nil
)
}
deinit {
if let timeObserver {
player.removeTimeObserver(timeObserver)
}
rateObservation?.invalidate()
NotificationCenter.default.removeObserver(self)
}
/// Whether an audio session interruption is currently active. While true,
/// reconciliation must not call player.play() (the next poll would restart
/// audio during a phone call or Siri interaction).
private var isInterruptionActive = false
/// Separate resume-suppression flag: set when an interruption ends without
/// `.shouldResume` and the pause control to the server has not yet been
/// confirmed. While true, reconciliation must not call player.play() even
/// after `isInterruptionActive` is cleared. Cleared only by an explicit user
/// play/resume action or a confirmed coherent paused state from the server.
private var resumeSuppressed = false
/// Whether playback was actually active (rate > 0) when the interruption
/// began. Used to decide whether a `.shouldResume` end should actually
/// resume (a track that was already paused before the interruption must not
/// be force-resumed).
private var wasPlayingBeforeInterruption = false
/// Sequence number for media interruption operations. Incremented on every
/// `.began`; a resume task captures the sequence and rechecks it immediately
/// before `player.play()`. A newer `.began` makes an old resume task
/// incapable of playing (Iteration 07 Item 5).
private var mediaInterruptionSequence = 0
/// The retained operation chain for media interruption control work. Both
/// pause and resume operations are assigned here so a newer begin cancels
/// and supersedes prior control work.
private var mediaInterruptionTask: Task<Void, Never>?
@objc private func handleInterruption(_ notification: Notification) {
guard let info = notification.userInfo else { return }
guard let typeValue = info[AVAudioSessionInterruptionTypeKey] as? UInt,
let type = AVAudioSession.InterruptionType(rawValue: typeValue) else { return }
switch type {
case .began:
// Increment the sequence so any in-flight resume task from a prior
// interruption sees a stale sequence and cannot play.
mediaInterruptionSequence += 1
let seq = mediaInterruptionSequence
// Cancel/supersede prior control work (resume or pause) so it cannot
// complete out of order.
mediaInterruptionTask?.cancel()
isInterruptionActive = true
wasPlayingBeforeInterruption = !player.rate.isZero
let wasPlaying = wasPlayingBeforeInterruption
player.pause()
logger.log("audio interruption began, paused media (wasPlaying=\(wasPlaying), seq=\(seq))")
case .ended:
defer { isInterruptionActive = false }
applyVolume()
let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
let shouldResume = AVAudioSession.InterruptionOptions(rawValue: optionsValue).contains(.shouldResume)
if shouldResume, wasPlayingBeforeInterruption, currentTrackID != nil, desired == "playing", loadedAudioPath != nil {
// Synchronize the server before resuming local playback so the
// two never diverge. Capture generation/room/sequence before the
// async hop. The task is retained in mediaInterruptionTask so a
// newer .began can cancel it.
let gen = pollGeneration
let targetRoom = room
let seq = mediaInterruptionSequence
mediaInterruptionTask = Task { [weak self] in
guard let self else { return }
if let targetRoom {
do {
try await self.api.control(room: targetRoom, action: "resume")
logger.log("interruption resume: server synchronized (seq=\(seq))")
} catch {
logger.log("interruption resume: server sync failed (poll will retry): \(error.localizedDescription)")
}
}
// Recheck the sequence immediately before play: a newer .began
// increments mediaInterruptionSequence, making this stale task
// incapable of restarting audio during the newer interruption.
guard !Task.isCancelled,
self.mediaInterruptionSequence == seq,
!self.isInterruptionActive,
self.pollGeneration == gen,
self.room == targetRoom else { return }
self.player.play()
logger.log("audio interruption ended, resumed media (seq=\(seq))")
}
} else {
// No-resume path: keep suppression even if the pause control
// fails. The flag is cleared only by an explicit user play/resume
// action or a confirmed coherent paused state from the server.
resumeSuppressed = true
let seq = mediaInterruptionSequence
if let room {
mediaInterruptionTask = Task { [weak self] in
guard let self else { return }
do {
try await self.api.control(room: room, action: "pause")
logger.log("audio interruption ended without resume, sent pause to server (seq=\(seq))")
} catch {
// Keep suppression even on failure — the next poll
// will see desired=="paused" and clear it.
logger.log("audio interruption ended without resume, pause control failed (suppression retained): \(error.localizedDescription)")
}
}
}
}
@unknown default:
break
}
}
// MARK: Lifecycle
func attach(session: VoiceSession) {
self.session = session
observePlayer()
observePlaybackState()
applyVolume()
}
/// Called just before joining the LiveKit room. Carries whatever is loaded
/// into the new room-scoped player state, paused (talking pauses music).
func enterMediaScope(roomName: String) async {
pollGeneration += 1
let wasPlaying = currentTrackID != nil && !player.rate.isZero
wasPlayingBeforeTalk = wasPlaying
let carriedTrack = currentTrackID
room = roomName
if let carriedTrack {
do {
try await api.play(room: roomName, trackID: carriedTrack, startPaused: true)
logger.log("media scope carry-over: \(carriedTrack, privacy: .public) start_paused")
} catch {
logger.log("media scope transfer failed: \(String(describing: error), privacy: .public)")
}
player.pause()
}
startPolling()
}
/// Called when the conversation ends. If music was playing before the talk,
/// hand the room back to the player. The previous scope (room name) is kept
/// so the panel can resume polling it via `startDefaultPolling()`; a fresh
/// conversation re-scopes via `enterMediaScope`. Polling restarts here so
/// the resumed state stays visible without the panel being open.
func endConversation() async {
stopPolling()
pollGeneration += 1
if wasPlayingBeforeTalk {
wasPlayingBeforeTalk = false
guard let room else { return }
do {
try await api.control(room: room, action: "resume")
logger.log("resumed playback after talk ended")
} catch {
logger.log("post-talk resume failed (poll will retry): \(String(describing: error), privacy: .public)")
}
}
startPolling()
}
/// Drop the room scope entirely (app-level teardown). After this, polling
/// targets the default room again.
func resetScope() {
stopPolling()
pollGeneration += 1
room = nil
wasPlayingBeforeTalk = false
}
private func startPolling() {
guard pollTask == nil else { return }
pollTask = Task { [weak self] in
while !Task.isCancelled {
await self?.pollState()
try? await Task.sleep(for: .milliseconds(1500))
}
}
}
private func stopPolling() {
pollTask?.cancel()
pollTask = nil
}
/// Resume polling whenever the panel becomes visible and nothing is being
/// polled yet — whether that's the default room (no session) or a previous
/// conversation's room scope (kept after `endConversation`).
func startDefaultPolling() {
guard pollTask == nil else { return }
startPolling()
}
// MARK: Poll + reconcile (ported from app.js pollMediaState/reconcileMedia)
private func pollState() async {
let generation = pollGeneration
let scope = room ?? "default"
do {
let state = try await api.state(room: scope)
guard generation == pollGeneration else { return }
stateError = nil
reconcile(state)
} catch {
guard generation == pollGeneration else { return }
logger.log("poll failed: \(String(describing: error), privacy: .public)")
stateError = "Media server unreachable: \(error.localizedDescription)"
}
}
/// Reconciles the local player against a server state snapshot. Exposed
/// as internal (not private) so tests can drive deterministic state
/// transitions without a live server.
func reconcile(_ s: MediaState) {
choices = s.choices
pendingJobs = s.pending
// Keep the published desired state in sync with the server so the UI's
// play/pause icon and status text reflect reality (the server is the
// source of truth for per-room desired state).
desired = s.desired ?? "stopped"
let track = s.track
if let track, track.id != currentTrackID {
loadTrack(track)
} else if let track, let audioURL = track.audioURL, loadedAudioPath != audioURL {
// Track was loaded before it had a stream (still downloading) and
// now has one — wire it up so playback can start without a reload.
wireSource(audioURL)
}
if track == nil, currentTrackID != nil, s.desired == "stopped" {
clearMedia()
}
applyDesiredState(s, track: track)
if let hint = s.seekHint, hint.ts != appliedSeekTs, duration.isFinite, duration > 0 {
appliedSeekTs = hint.ts
let target = min(hint.position, duration)
if target.isFinite {
seek(to: max(0, target))
}
}
}
private func applyDesiredState(_ s: MediaState, track: MediaTrack?) {
guard let desiredValue = s.desired else { return }
switch desiredValue {
case "playing":
// Suppress play during an active audio interruption (phone call,
// Siri). Without this guard the next 1.5s poll would restart audio
// immediately after .began paused it. The player.rate.isZero check
// makes this idempotent: once playing, subsequent polls are no-ops.
// This also handles the late-audio case where desired=playing
// arrives before audio_url is set; when the source is later wired
// via wireSource(), the next poll retries play because rate is
// still zero.
// Also suppress while resumeSuppressed is true (no-resume
// interruption ended, server not yet confirmed paused).
guard !isInterruptionActive, !resumeSuppressed else { return }
if let track, track.audioURL != nil, loadedAudioPath != nil, player.rate.isZero {
player.play()
}
case "paused", "stopped":
if !player.rate.isZero {
player.pause()
}
// The server has confirmed a coherent paused/stopped state. If we
// were suppressing resume due to a no-resume interruption, the
// pause control succeeded (or the user paused independently), so
// it is safe to clear the suppression flag.
if resumeSuppressed {
resumeSuppressed = false
logger.log("resume suppression cleared: server confirmed \(desiredValue)")
}
default:
break
}
}
private func loadTrack(_ track: MediaTrack) {
currentTrackID = track.id
self.track = track
if let audioURL = track.audioURL {
wireSource(audioURL)
} else {
// Downloading or failed: no stream yet. Reconcile wires the stream
// URL in via polling as soon as one is available.
clearCurrentAsset()
player.replaceCurrentItem(with: nil)
}
}
private func wireSource(_ path: String) {
guard let url = URL(string: path) else { return }
loadedAudioPath = path
let pinned = mediaLoader.rewrite(url)
// Remove the previous asset's loader mapping before wiring a new one so
// the loaderURLs dictionary stays bounded (Finding 11 / Item 7).
clearCurrentAsset()
let asset = mediaLoader.assetWithPinnedURL(pinned)
currentAsset = asset
logger.log("wiring source \(pinned.absoluteString, privacy: .public)")
let item = AVPlayerItem(asset: asset)
observeItem(item)
player.replaceCurrentItem(with: item)
}
/// Removes the loader mapping for the current asset (if any) and clears the
/// reference. Called before wiring a new source or clearing media so the
/// resource loader's internal dictionaries stay bounded.
private func clearCurrentAsset() {
if let asset = currentAsset {
mediaLoader.removeLoader(asset)
currentAsset = nil
}
}
// MARK: Player events
private func observePlayer() {
if timeObserver == nil {
timeObserver = player.addPeriodicTimeObserver(
forInterval: CMTime(seconds: 0.5, preferredTimescale: 600), queue: .main
) { [weak self] time in
Task { @MainActor [weak self] in
guard let self else { return }
self.currentTime = time.seconds
if let itemDuration = self.player.currentItem?.duration {
let seconds = itemDuration.seconds
if seconds.isFinite {
self.duration = seconds
}
}
if !self.player.rate.isZero, Date().timeIntervalSince(self.lastReportDate) >= 5 {
self.lastReportDate = Date()
self.report("playing")
}
}
}
}
}
// MARK: - Playback state observation
/// Observe the player's actual rate so every real play/pause transition is
/// reported promptly, mirroring the web UI's `play`/`pause` listeners. The
/// KVO block runs on an arbitrary queue; hop to MainActor before touching
/// any controller state.
private func observePlaybackState() {
rateObservation = player.observe(\.rate, options: [.initial, .new]) { [weak self] player, _ in
Task { @MainActor [weak self] in
guard let self else { return }
guard self.currentTrackID != nil else { return }
if player.rate > 0 {
// Actual playback began (user play, interruption resume, or
// reconcile-driven play). Report + publish the transition.
self.lastReportDate = Date()
self.report("playing")
self.publishMediaState("playing")
} else if player.currentItem != nil {
// Rate dropped to zero with an item still loaded: a pause
// that didn't originate from a user action (interruption,
// reconcile). User-initiated pauses already report before
// this fires; the lastReportDate guard makes duplicates no-ops.
if Date().timeIntervalSince(self.lastReportDate) >= 1 {
self.lastReportDate = Date()
self.report("paused")
self.publishMediaState("paused")
}
}
}
}
}
private func observeItem(_ item: AVPlayerItem?) {
NotificationCenter.default.removeObserver(
self, name: .AVPlayerItemDidPlayToEndTime, object: nil
)
guard let item else { return }
NotificationCenter.default.addObserver(
self, selector: #selector(playerItemDidEnd(_:)),
name: .AVPlayerItemDidPlayToEndTime, object: item
)
statusCancellable = item.publisher(for: \.status).sink { status in
switch status {
case .readyToPlay:
logger.log("item readyToPlay")
case .failed:
logger.log("item failed: \(String(describing: item.error), privacy: .public)")
default:
break
}
}
}
@objc private func playerItemDidEnd(_ notification: Notification) {
lastReportDate = Date()
report("stopped")
publishMediaState("stopped")
Task { [weak self] in
guard let self, let room = self.room else { return }
try? await self.api.control(room: room, action: "stop")
}
}
// MARK: Reporting + publishing
private func report(_ state: String, trackID: String? = nil) {
guard let room else { return }
let position = currentTime.isFinite ? currentTime : 0
var durationValue: Double?
if duration.isFinite, duration > 0 {
durationValue = duration
}
let resolvedTrackID = trackID ?? currentTrackID
Task { [weak self] in
guard let self else { return }
do {
try await self.api.report(room: room, state: state, trackID: resolvedTrackID, position: position, duration: durationValue)
} catch {
logger.log("report failed: \(String(describing: error), privacy: .public)")
}
}
}
private func publishMediaState(_ state: String, trackID: String? = nil) {
guard let session else { return }
// Capture the exact room and generation before spawning so a delayed
// publish from an old lifecycle cannot escape into a new conversation
// room (Finding 10).
let targetRoom = session.room
let generation = session.sessionGeneration
let pollGen = pollGeneration
let resolvedTrackID = trackID ?? currentTrackID
Task { [weak self] in
guard let self, self.pollGeneration == pollGen else { return }
guard session.sessionGeneration == generation, session.room === targetRoom else { return }
do {
try await session.controlChannel.sendMediaState(state, trackID: resolvedTrackID, room: targetRoom)
} catch {
logger.log("media_state publish failed: \(String(describing: error), privacy: .public)")
}
}
}
// MARK: Play requests (search results, choices, library)
func playVideo(videoID: String?, title: String?, channel: String?) async {
let targetRoom = room ?? "default"
do {
try await api.play(room: targetRoom, videoID: videoID, title: title, channel: channel)
logger.log("play requested video_id=\(videoID ?? "nil", privacy: .public) room=\(targetRoom, privacy: .public)")
} catch {
logger.log("play failed: \(String(describing: error), privacy: .public)")
}
}
func playTrack(id: String) async {
let targetRoom = room ?? "default"
do {
try await api.play(room: targetRoom, trackID: id)
logger.log("play requested track_id=\(id, privacy: .public) room=\(targetRoom, privacy: .public)")
} catch {
logger.log("play failed: \(String(describing: error), privacy: .public)")
}
}
// MARK: Transport controls (parity with web UI play/pause/seek/stop)
func togglePlayPause() async {
guard currentTrackID != nil, loadedAudioPath != nil else { return }
if player.rate.isZero {
// Explicit user play: clear any resume suppression (the user is
// deliberately overriding the no-resume interruption state).
resumeSuppressed = false
// The rate observer reports + publishes the actual transition;
// the control call updates server desired state.
player.play()
if let room { try? await api.control(room: room, action: "resume") }
} else {
lastReportDate = Date()
report("paused")
publishMediaState("paused")
player.pause()
if let room { try? await api.control(room: room, action: "pause") }
}
}
func seek(by delta: Double) async {
guard loadedAudioPath != nil else { return }
let target = max(0, min(duration > 0 ? duration - 0.5 : .infinity, currentTime + delta))
if target.isFinite {
seek(to: target)
}
}
func seek(to position: Double) {
player.seek(to: CMTime(seconds: position, preferredTimescale: 600))
currentTime = position
report(player.rate.isZero ? "paused" : "playing")
}
func stop() async {
// Capture the track ID before clearing so the async report/publish
// tasks can read a non-nil value even after clearMedia() runs.
let trackID = currentTrackID
report("stopped", trackID: trackID)
clearMedia()
if let room { try? await api.control(room: room, action: "stop") }
publishMediaState("stopped", trackID: trackID)
}
private func clearMedia() {
currentTrackID = nil
track = nil
loadedAudioPath = nil
// Remove the loader mapping for the asset being cleared so the
// resource loader's internal dictionaries stay bounded (Item 7).
clearCurrentAsset()
observeItem(nil)
player.replaceCurrentItem(with: nil)
currentTime = 0
duration = 0
stateError = nil
}
// MARK: Volume
func applyVolume() {
player.volume = Float(settings.volume)
}
func setVolume(_ volume: Double) {
settings.volume = volume
player.volume = Float(volume)
}
}
@@ -1,220 +0,0 @@
import AVFoundation
import Foundation
import os
private let logger = Logger(subsystem: "dev.shane.voice", category: "media")
/// Proxies {base}/media/* bytes to AVPlayer over the pinned-TLS session, since
/// AVPlayer cannot trust the self-signed LAN cert. AVPlayer loads https natively
/// (ignoring registered URLProtocol subclasses), so the asset uses a custom
/// `pinned-media` scheme and this delegate answers every data request by
/// fetching the real https URL through TLSPinning.session(), forwarding Range.
///
/// **Lifecycle invariants (Finding 11 / Iteration 06 Item 7):**
/// - Tasks are keyed by `ObjectIdentifier(loadingRequest)` directly, not a
/// derived hash, so two distinct requests can never collide.
/// - All mutable state (`loaderURLs`, `activeTasks`) is accessed only on the
/// main queue: the AVAssetResourceLoader delegate is set with `queue: .main`
/// and the pinned URLSession uses `delegateQueue: .main`, so both the
/// resource-loading callbacks and the data-task completion handlers run on
/// the same serialization domain. This is enforced in code, not by comment.
/// - For HTTP 206, `contentLength` is set only when `Content-Range` provides a
/// known total; it never falls back to the range's `expectedContentLength`.
/// - For HTTP 200, `expectedContentLength` is already the full resource size.
/// - Cleanup (task removal) occurs on success, error, cancellation, and item
/// replacement via `removeLoader`.
final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate {
static let scheme = "pinned-media"
private let base: URL
/// Maps each asset's resource loader to the real https URL it proxies.
/// Accessed only on the main queue (the delegate queue).
private var loaderURLs: [ObjectIdentifier: URL] = [:]
/// Maps each loading request's identity to its upstream data task so
/// cancellation can cancel the in-flight download. Keyed directly by
/// `ObjectIdentifier(loadingRequest)` — no hash derivation, no collisions.
private var activeTasks: [ObjectIdentifier: URLSessionDataTask] = [:]
/// Shared pinned session, created once and invalidated on deinit.
/// Uses `delegateQueue: .main` so all completion handlers (and therefore
/// all mutations of `activeTasks`) run on the main queue — the same
/// serialization domain as the AVAssetResourceLoader delegate callbacks
/// (which are also dispatched to `.main`). This makes map access truly
/// serialized in code, not just by convention.
private lazy var pinnedSession: URLSession = {
let config = URLSessionConfiguration.default
config.timeoutIntervalForRequest = 30
return URLSession(configuration: config, delegate: PinnedTrustDelegate(), delegateQueue: .main)
}()
init(base: URL) {
self.base = base
}
deinit {
pinnedSession.invalidateAndCancel()
}
/// Maps a media URL (absolute https or /media/... path) to the pinned-media
/// scheme so AVPlayer routes it through this delegate instead of loading
/// https natively. Built as a string because URLComponents.url returns nil
/// for schemes not in Foundation's known-scheme list.
func rewrite(_ url: URL) -> URL {
let absolute: String
if let scheme = url.scheme?.lowercased(), scheme.hasPrefix("http") {
var c = URLComponents(url: url, resolvingAgainstBaseURL: false)!
c.scheme = Self.scheme
return c.url!
} else {
let path = url.absoluteString.hasPrefix("/") ? String(url.absoluteString.dropFirst()) : url.absoluteString
guard let host = base.host, !host.isEmpty else { return url }
let port = base.port.map { ":\($0)" } ?? ""
absolute = "\(Self.scheme)://\(host)\(port)/\(path)"
}
return URL(string: absolute) ?? url
}
/// Maps a pinned-media URL back to the real https URL it proxies.
func resolve(_ url: URL) -> URL {
guard let scheme = url.scheme?.lowercased(), scheme == Self.scheme else { return url }
var c = URLComponents(url: url, resolvingAgainstBaseURL: false)!
c.scheme = "https"
return c.url ?? url
}
/// Creates an AVURLAsset that routes through this loader's delegate. The
/// mapping from the asset's resource loader to the real URL is recorded so
/// `resourceLoader(_:shouldWaitForLoadingOfRequestedResource:)` can resolve
/// it. All mutations happen on the main queue (the delegate queue).
func assetWithPinnedURL(_ pinned: URL) -> AVURLAsset {
let asset = AVURLAsset(url: pinned)
let realURL = resolve(pinned)
// The delegate queue is .main, so this runs on the main thread.
loaderURLs[ObjectIdentifier(asset.resourceLoader)] = realURL
asset.resourceLoader.setDelegate(self, queue: .main)
return asset
}
/// Removes the loader mapping and any in-flight tasks for an asset that is
/// no longer needed. Called by MediaPlayerController when replacing or
/// clearing media items to bound the dictionaries (Finding 11).
func removeLoader(_ asset: AVURLAsset) {
let key = ObjectIdentifier(asset.resourceLoader)
loaderURLs[key] = nil
// Cancel any in-flight tasks. URLSessionDataTask has no isCancelled
// property; calling cancel() on an already-cancelled task is a no-op.
for (_, task) in activeTasks {
task.cancel()
}
activeTasks.removeAll()
}
func resourceLoader(
_ resourceLoader: AVAssetResourceLoader,
shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest
) -> Bool {
// All state access is on the main queue (the delegate queue).
guard let dataRequest = loadingRequest.dataRequest else { return false }
guard let url = loaderURLs[ObjectIdentifier(resourceLoader)] else {
logger.log("no resolved URL for resource loader")
loadingRequest.finishLoading(with: URLError(.badURL))
return false
}
var upstream = URLRequest(url: url)
let offset = dataRequest.requestedOffset
let length = Int64(dataRequest.requestedLength)
if length > 0 {
upstream.setValue("bytes=\(offset)-\(offset + length - 1)", forHTTPHeaderField: "Range")
} else {
upstream.setValue("bytes=\(offset)-", forHTTPHeaderField: "Range")
}
logger.log("fetching \(url.absoluteString, privacy: .public) range=\(upstream.value(forHTTPHeaderField: "Range") ?? "-", privacy: .public)")
// Key the task by the loading request's identity directly — no hash
// derivation, so two distinct requests can never collide.
let requestKey = ObjectIdentifier(loadingRequest)
let task = pinnedSession.dataTask(with: upstream) { [weak self] data, response, error in
guard let self else { return }
// Cleanup: remove the task from the active map regardless of outcome.
// Runs on .main (the session's delegateQueue), same domain as all
// other map mutations.
self.activeTasks[requestKey] = nil
if let error {
logger.log("fetch failed \(url.absoluteString, privacy: .public): \(error.localizedDescription, privacy: .public)")
loadingRequest.finishLoading(with: error)
return
}
guard let http = response as? HTTPURLResponse else {
loadingRequest.finishLoading(with: URLError(.badServerResponse))
return
}
// Reject non-200/206 responses (404, 500, etc.) instead of feeding
// error JSON into AVPlayer as media bytes.
guard http.statusCode == 200 || http.statusCode == 206 else {
logger.log("unexpected status \(http.statusCode) for \(url.absoluteString, privacy: .public)")
loadingRequest.finishLoading(with: URLError(.badServerResponse))
return
}
if let info = loadingRequest.contentInformationRequest {
info.contentType = http.mimeType ?? "application/octet-stream"
// Content-length handling (Iteration 05 Item 7):
// - For 206: set full content length ONLY when Content-Range
// provides a known total. Never fall back to the range's
// expectedContentLength (that would tell AVPlayer the total
// resource is only as large as this one range).
// - For 200: expectedContentLength is already the full size.
if http.statusCode == 206 {
if let total = Self.parseTotalLength(http), total >= 0 {
info.contentLength = total
}
// If Content-Range has no known total (e.g. "bytes 0-1023/*"),
// do NOT set contentLength — AVPlayer will handle it as
// unknown-length, which is correct for a range response.
} else {
// 200: expectedContentLength is the full resource size.
if http.expectedContentLength >= 0 {
info.contentLength = Int64(http.expectedContentLength)
}
}
info.isByteRangeAccessSupported = true
}
if let data {
dataRequest.respond(with: data)
}
loadingRequest.finishLoading()
}
activeTasks[requestKey] = task
task.resume()
return true
}
func resourceLoader(
_ resourceLoader: AVAssetResourceLoader,
didCancel loadingRequest: AVAssetResourceLoadingRequest
) {
// Cancel the upstream data task so a seek or item swap doesn't keep
// downloading cancelled ranges. Keyed by ObjectIdentifier directly.
let key = ObjectIdentifier(loadingRequest)
activeTasks[key]?.cancel()
activeTasks[key] = nil
logger.log("resource loader cancelled request, upstream task cancelled")
}
/// Parses the total resource length from an HTTP 206 response's
/// `Content-Range` header (e.g. "bytes 0-1023/4096" → 4096). Returns nil
/// for non-206 responses, missing headers, unknown totals ("*"), or
/// malformed values.
static func parseTotalLength(_ response: HTTPURLResponse) -> Int64? {
if response.statusCode == 206,
let contentRange = response.value(forHTTPHeaderField: "Content-Range") {
// Format: "bytes <start>-<end>/<total>" or "bytes <start>-<end>/*"
let afterSlash = contentRange.split(separator: "/").last?.trimmingCharacters(in: .whitespaces)
if let afterSlash, afterSlash != "*", let total = Int64(afterSlash) {
return total
}
}
return nil
}
}
-125
View File
@@ -1,125 +0,0 @@
import Foundation
import Testing
@testable import Voice
/// Minimal HTTP/1.1 stub built on BSD sockets that answers every request with a
/// successful JSON object body (mirroring the real media server's
/// `{"ok": true, ...}` shape). Used to prove MediaAPI no-response POSTs succeed
/// against an object body instead of throwing decodeFailed.
@MainActor
final class MediaAPIStubServer {
private(set) var url: URL!
nonisolated(unsafe) private var serverFD: Int32 = -1
nonisolated(unsafe) private var acceptThread: Thread?
nonisolated(unsafe) private var running = false
nonisolated(unsafe) private var lastPath: String?
func start() throws {
let fd = socket(AF_INET, SOCK_STREAM, 0)
guard fd >= 0 else { throw MediaAPIError.decodeFailed }
var yes: Int32 = 1
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout<Int32>.size))
var addr = sockaddr_in()
addr.sin_family = sa_family_t(AF_INET)
addr.sin_port = in_port_t(0).bigEndian
addr.sin_addr = in_addr(s_addr: inet_addr("127.0.0.1"))
let bindResult = withUnsafePointer(to: &addr) { ptr in
ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in
bind(fd, sa, socklen_t(MemoryLayout<sockaddr_in>.size))
}
}
guard bindResult == 0 else { close(fd); throw MediaAPIError.decodeFailed }
guard listen(fd, 8) == 0 else { close(fd); throw MediaAPIError.decodeFailed }
var bound = sockaddr_in()
var len = socklen_t(MemoryLayout<sockaddr_in>.size)
withUnsafeMutablePointer(to: &bound) { ptr in
_ = ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in
getsockname(fd, sa, &len)
}
}
let port = Int(UInt16(bigEndian: bound.sin_port))
serverFD = fd
running = true
url = URL(string: "http://127.0.0.1:\(port)")!
let thread = Thread { [weak self] in
self?.acceptLoop()
}
acceptThread = thread
thread.start()
}
func stop() {
running = false
if serverFD >= 0 { close(serverFD) }
serverFD = -1
acceptThread?.cancel()
}
var lastRequestPath: String? { lastPath }
nonisolated private func acceptLoop() {
while running {
var clientAddr = sockaddr_in()
var len = socklen_t(MemoryLayout<sockaddr_in>.size)
let clientFD = withUnsafeMutablePointer(to: &clientAddr) { ptr in
ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in
accept(serverFD, sa, &len)
}
}
guard clientFD >= 0 else { continue }
handleClient(clientFD)
}
}
nonisolated private func handleClient(_ clientFD: Int32) {
defer { close(clientFD) }
var buffer = [UInt8](repeating: 0, count: 4096)
let bytesRead = read(clientFD, &buffer, buffer.count)
guard bytesRead > 0 else { return }
guard let request = String(bytes: buffer[0..<bytesRead], encoding: .utf8) else { return }
var path = ""
if let firstLine = request.split(separator: "\r\n").first {
let parts = firstLine.split(separator: " ")
if parts.count >= 2 { path = String(parts[1]) }
}
lastPath = path
let body = #"{"ok": true, "room": "test", "desired": "stopped"}"#
let response = """
HTTP/1.1 200 OK\r
Content-Type: application/json\r
Content-Length: \(body.utf8.count)\r
Connection: close\r
\(body)
"""
_ = response.withCString { cstr in
write(clientFD, cstr, response.utf8.count)
}
}
}
@MainActor
struct MediaAPITests {
@Test("successful JSON object response does not throw decodeFailed")
func successfulObjectResponse() async throws {
let stub = MediaAPIStubServer()
try stub.start()
defer { stub.stop() }
// Bypass the https-only parse path: build the config directly.
let base = stub.url!
let config = ServerConfig(baseURL: base, livekitURL: base)
let api = MediaAPI(config: config)
try await api.control(room: "test", action: "pause")
try await api.report(room: "test", state: "paused", trackID: nil, position: 0, duration: nil)
try await api.play(room: "test", videoID: nil, title: nil, channel: nil, trackID: nil, startPaused: nil)
#expect(stub.lastRequestPath == "/media/api/play")
}
}
-124
View File
@@ -1,124 +0,0 @@
import Foundation
import Testing
@testable import Voice
@MainActor
struct MediaReconcileTests {
private func makeController() -> MediaPlayerController {
let config = ServerConfig(
baseURL: URL(string: "http://127.0.0.1:9")!,
livekitURL: URL(string: "ws://127.0.0.1:9")!
)
let api = MediaAPI(config: config)
return MediaPlayerController(api: api, settings: SettingsStore())
}
private func makeState(
desired: String? = "stopped",
trackID: String? = nil,
audioURL: String? = nil
) -> MediaState {
let track: MediaTrack? = trackID != nil ? MediaTrack(
id: trackID!,
kind: nil,
videoID: nil,
title: nil,
channel: nil,
duration: nil,
status: nil,
error: nil,
audioURL: audioURL,
artURL: nil
) : nil
return MediaState(
room: "default",
track: track,
desired: desired,
reported: nil,
seekHint: nil,
choices: [],
pending: [],
version: 1
)
}
@Test("reconcile updates published desired to playing")
func reconcileDesiredPlaying() {
let controller = makeController()
#expect(controller.desired == "stopped")
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
}
@Test("reconcile updates published desired to paused")
func reconcileDesiredPaused() {
let controller = makeController()
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
controller.reconcile(makeState(desired: "paused"))
#expect(controller.desired == "paused")
}
@Test("reconcile updates published desired to stopped")
func reconcileDesiredStopped() {
let controller = makeController()
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
controller.reconcile(makeState(desired: "stopped"))
#expect(controller.desired == "stopped")
}
@Test("reconcile with nil desired defaults to stopped")
func reconcileNilDesiredDefaultsToStopped() {
let controller = makeController()
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
controller.reconcile(makeState(desired: nil))
#expect(controller.desired == "stopped")
}
@Test("late audio: desired=playing before audio_url, then audio_url arrives")
func lateAudioReconciliation() {
let controller = makeController()
// Phase 1: server says playing but the track has no audio_url yet
// (still downloading). The source is not wired, so play cannot start.
let stateNoURL = makeState(desired: "playing", trackID: "track-1", audioURL: nil)
controller.reconcile(stateNoURL)
#expect(controller.desired == "playing")
#expect(controller.track?.id == "track-1")
// Player rate should still be zero because no source was wired.
#expect(controller.player.rate.isZero)
// Phase 2: the same track now has an audio_url. The reconcile path
// wires the source (wireSource) and then applyDesiredState retries
// play because desired is still "playing" and rate is still zero.
let stateWithURL = makeState(desired: "playing", trackID: "track-1", audioURL: "/stream/track-1.mp3")
controller.reconcile(stateWithURL)
#expect(controller.desired == "playing")
// The source should now be wired (loadedAudioPath is set internally).
// We can't directly observe loadedAudioPath, but we can verify the
// player has a current item after wireSource ran.
#expect(controller.player.currentItem != nil)
}
@Test("startDefaultPolling restarts after endConversation keeps scope")
func pollingRestartsAfterEndConversation() async {
let controller = makeController()
controller.startDefaultPolling()
await controller.endConversation()
controller.startDefaultPolling()
try? await Task.sleep(for: .milliseconds(50))
}
@Test("togglePlayPause without a loaded track is a no-op")
func toggleWithoutTrack() async {
let controller = makeController()
await controller.togglePlayPause()
#expect(controller.track == nil)
}
}
-126
View File
@@ -1,126 +0,0 @@
import Foundation
import Testing
@testable import Voice
/// Focused tests for PinnedMediaResourceLoader.parseTotalLength, covering
/// 200, 206 with known total, 206 with unknown total (*), and malformed
/// Content-Range headers (Finding 1 / 11).
struct PinnedMediaResourceLoaderTests {
private func makeResponse(status: Int, contentRange: String? = nil, expectedLength: Int64 = -1) -> HTTPURLResponse {
var headers: [String: String] = [:]
if let contentRange {
headers["Content-Range"] = contentRange
}
return HTTPURLResponse(
url: URL(string: "https://example.com/media/test.mp3")!,
statusCode: status,
httpVersion: "HTTP/1.1",
headerFields: headers
)!
}
@Test("206 with known total parses Content-Range total")
func parse206KnownTotal() {
let response = makeResponse(status: 206, contentRange: "bytes 0-1023/4096", expectedLength: 1024)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == 4096)
}
@Test("206 with unknown total (*) returns nil")
func parse206UnknownTotal() {
let response = makeResponse(status: 206, contentRange: "bytes 0-1023/*", expectedLength: 1024)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("200 response returns nil (expectedContentLength is already the full size)")
func parse200ReturnsNil() {
let response = makeResponse(status: 200, expectedLength: 8192)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("206 without Content-Range header returns nil")
func parse206NoContentRange() {
let response = makeResponse(status: 206, expectedLength: 512)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("malformed Content-Range returns nil")
func parseMalformedContentRange() {
let response = makeResponse(status: 206, contentRange: "garbage", expectedLength: 100)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("206 with large total parses correctly")
func parseLargeTotal() {
let response = makeResponse(status: 206, contentRange: "bytes 1048576-2097151/1073741824", expectedLength: 1048576)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == 1_073_741_824)
}
@Test("streamURL constructs endpoint without downloading")
func streamURLConstruction() {
let config = ServerConfig(
baseURL: URL(string: "https://192.168.86.2:9178")!,
livekitURL: URL(string: "ws://192.168.86.2:7880")!
)
let api = MediaAPI(config: config)
let url = api.streamURL(trackID: "track-123")
// streamURL builds under the /media/api base (MediaAPI.baseURL).
#expect(url.absoluteString == "https://192.168.86.2:9178/media/api/stream/track-123")
}
// MARK: Rewrite / Resolve round-trip (Item 7)
@Test("rewrite maps https URL to pinned-media scheme and resolve inverts it")
func rewriteResolveRoundTrip() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let original = URL(string: "https://192.168.86.2:9178/media/track-123.mp3")!
let pinned = loader.rewrite(original)
#expect(pinned.scheme == "pinned-media")
#expect(pinned.absoluteString == "pinned-media://192.168.86.2:9178/media/track-123.mp3")
let resolved = loader.resolve(pinned)
#expect(resolved.absoluteString == original.absoluteString)
}
@Test("rewrite maps path-only URL using base host and port")
func rewritePathOnly() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let pathURL = URL(string: "/media/track-456.mp3")!
let pinned = loader.rewrite(pathURL)
#expect(pinned.scheme == "pinned-media")
#expect(pinned.absoluteString == "pinned-media://192.168.86.2:9178/media/track-456.mp3")
}
@Test("resolve leaves non-pinned URLs unchanged")
func resolveNonPinned() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let https = URL(string: "https://example.com/foo.mp3")!
#expect(loader.resolve(https) == https)
}
// MARK: Asset lifecycle (Item 7)
@Test("assetWithPinnedURL creates asset and removeLoader clears mapping")
func assetLifecycle() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let pinned = loader.rewrite(URL(string: "https://192.168.86.2:9178/media/test.mp3")!)
let asset = loader.assetWithPinnedURL(pinned)
// The asset should have a valid URL matching the pinned scheme.
#expect(asset.url.scheme == PinnedMediaResourceLoader.scheme)
// Remove the loader mapping — subsequent calls should not crash.
loader.removeLoader(asset)
}
@Test("removeLoader is idempotent (safe to call twice)")
func removeLoaderIdempotent() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let pinned = loader.rewrite(URL(string: "https://192.168.86.2:9178/media/test.mp3")!)
let asset = loader.assetWithPinnedURL(pinned)
loader.removeLoader(asset)
// Second call should be a no-op, not a crash.
loader.removeLoader(asset)
}
}
+64
View File
@@ -0,0 +1,64 @@
import XCTest
/// Opt-in tests against the actual NAS deployment. No injected replies.
final class VoiceLiveTests: XCTestCase {
private var app: XCUIApplication!
override func record(_ issue: XCTIssue) {
if let app, app.state == .runningForeground {
capture("Failure state")
let hierarchy = XCTAttachment(string: app.debugDescription)
hierarchy.name = "Failure UI hierarchy"
hierarchy.lifetime = .keepAlways
add(hierarchy)
}
super.record(issue)
}
override func tearDownWithError() throws {
app?.terminate()
}
override func setUpWithError() throws {
try XCTSkipUnless(ProcessInfo.processInfo.environment["HOPE_RUN_LIVE_TESTS"] == "1", "Requires the live Hope server")
continueAfterFailure = false
app = XCUIApplication()
app.launchArguments = ["--ui-testing", "--reset-ui-testing", "--live-testing"]
app.launch()
}
func testLiveConversationAndBackgroundTask() {
app.buttons["Start chatting"].tap()
let composer = app.descendants(matching: .any).matching(identifier: "Message Hope").firstMatch
XCTAssertTrue(composer.waitForExistence(timeout: 60), "Agent must join the real LiveKit room")
composer.tap()
composer.typeText("What is six times seven? Reply in one short sentence.")
app.buttons["Send message"].tap()
let reply = app.staticTexts.matching(NSPredicate(format: "label CONTAINS[c] 'forty-two' OR label CONTAINS[c] 'forty two' OR label CONTAINS '42'")).firstMatch
XCTAssertTrue(reply.waitForExistence(timeout: 60), "Qwen must produce a real response")
capture("Live Qwen reply")
composer.tap()
composer.typeText("Compare espresso and filter coffee in a short background task. Use your existing knowledge; no web search needed.")
app.buttons["Send message"].tap()
app.tabBars.buttons["Activity"].tap()
let complete = app.staticTexts["Complete"].firstMatch
XCTAssertTrue(complete.waitForExistence(timeout: 90), "Background task must finish and deliver its result")
capture("Live background result")
app.tabBars.buttons["Talk"].tap()
app.buttons["End conversation"].tap()
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20))
// A second room must connect cleanly after the first was torn down.
app.buttons["Start chatting"].tap()
XCTAssertTrue(composer.waitForExistence(timeout: 60))
app.buttons["End conversation"].tap()
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20))
}
private func capture(_ name: String) {
let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = name
attachment.lifetime = .keepAlways
add(attachment)
}
}
+123
View File
@@ -0,0 +1,123 @@
import XCTest
/// Real UI interactions with isolated preferences. These cover the client,
/// including offline recovery; they do not claim to verify the live backend.
final class VoiceUITests: XCTestCase {
private var app: XCUIApplication!
override func setUpWithError() throws {
continueAfterFailure = false
app = XCUIApplication()
app.launchArguments = ["--ui-testing", "--reset-ui-testing"]
}
func testWelcomeNavigationAndOfflineRecovery() {
app.launch()
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 10))
XCTAssertTrue(app.staticTexts["Your microphone starts after you connect."].exists)
app.buttons["Type"].tap()
XCTAssertTrue(app.buttons["Start chatting"].exists)
XCTAssertTrue(app.staticTexts["Your microphone stays off."].exists)
app.buttons["Start chatting"].tap()
XCTAssertTrue(app.buttons["Try again"].waitForExistence(timeout: 20))
XCTAssertTrue(app.staticTexts["Couldn’t connect"].exists)
app.buttons["Settings"].tap()
XCTAssertTrue(app.textFields["Server URL"].waitForExistence(timeout: 5))
capture("Offline recovery")
}
func testVoiceSelectionPersistsAndURLValidation() {
app.launch()
app.buttons["Settings"].tap()
app.buttons.matching(NSPredicate(format: "label CONTAINS 'Speaking voice'")).firstMatch.tap()
let voice = app.buttons.matching(NSPredicate(format: "label CONTAINS 'Jenny'")).firstMatch
XCTAssertTrue(voice.waitForExistence(timeout: 5))
voice.tap()
XCTAssertTrue(app.buttons.matching(NSPredicate(format: "label CONTAINS 'Speaking voice' AND label CONTAINS 'Jenny'")).firstMatch.exists)
let server = app.textFields["Server URL"]
server.tap()
let original = server.value as? String ?? ""
server.typeText(String(repeating: XCUIKeyboardKey.delete.rawValue, count: original.count) + "http://example.com")
app.buttons["Save server address"].tap()
XCTAssertTrue(app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'must use https'")).firstMatch.waitForExistence(timeout: 5))
let invalid = server.value as? String ?? ""
server.tap()
server.typeText(String(repeating: XCUIKeyboardKey.delete.rawValue, count: invalid.count) + " https://127.0.0.1:1 ")
app.buttons["Save server address"].tap()
XCTAssertEqual(server.value as? String, "https://127.0.0.1:1")
XCTAssertTrue(app.staticTexts["Server address saved."].waitForExistence(timeout: 5))
let unchanged = server.value as? String ?? ""
server.tap()
server.typeText(String(repeating: XCUIKeyboardKey.delete.rawValue, count: unchanged.count) + " https://127.0.0.1:2 ")
app.buttons["Save server address"].tap()
Thread.sleep(forTimeInterval: 2)
XCTAssertTrue(app.navigationBars["Settings"].exists)
XCTAssertTrue(app.staticTexts["Server address saved."].exists)
XCTAssertEqual(server.value as? String, "https://127.0.0.1:2")
app.terminate()
app.launchArguments = ["--ui-testing"]
app.launch()
app.buttons["Settings"].tap()
XCTAssertTrue(app.buttons.matching(NSPredicate(format: "label CONTAINS 'Speaking voice' AND label CONTAINS 'Jenny'")).firstMatch.waitForExistence(timeout: 5))
XCTAssertEqual(app.textFields["Server URL"].value as? String, "https://127.0.0.1:2")
}
func testTranscriptClearConfirmationAndActivityDetails() {
app.launchArguments.append("--ui-fixtures")
app.launch()
XCTAssertTrue(app.staticTexts["Help me plan a quiet weekend."].waitForExistence(timeout: 10))
app.buttons["Conversation options"].tap()
app.buttons["Clear conversation"].tap()
app.buttons["Cancel"].tap()
XCTAssertTrue(app.staticTexts["Help me plan a quiet weekend."].exists)
app.buttons["Conversation options"].tap()
app.buttons["Clear conversation"].tap()
app.buttons["Clear conversation"].tap()
XCTAssertTrue(app.buttons["Start chatting"].exists)
XCTAssertFalse(app.staticTexts["Help me plan a quiet weekend."].exists)
app.tabBars.buttons["Activity"].tap()
XCTAssertTrue(app.staticTexts["Needs attention"].waitForExistence(timeout: 5))
XCTAssertTrue(app.staticTexts["Search backend timed out after 30s"].exists)
app.staticTexts["Search backend timed out after 30s"].tap()
XCTAssertTrue(app.staticTexts["Fetching benchmark articles…"].waitForExistence(timeout: 5))
capture("Activity details")
}
func testRestoredTranscriptStaysAtTopUntilLatest() {
app.launchArguments.append("--ui-long-transcript")
app.launch()
let first = app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'Earlier message 1:'")).firstMatch
let last = app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'Earlier message 30:'")).firstMatch
XCTAssertTrue(first.waitForExistence(timeout: 10))
XCTAssertTrue(app.buttons["Latest"].exists)
XCTAssertFalse(last.isHittable)
app.buttons["Latest"].tap()
XCTAssertTrue(last.waitForExistence(timeout: 5))
XCTAssertTrue(last.isHittable)
}
func testLargeTypeNavigation() {
app.launchArguments += ["-UIPreferredContentSizeCategoryName", "UICTContentSizeCategoryAccessibilityXXXL"]
app.launch()
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 10))
XCTAssertTrue(app.buttons["Start chatting"].isHittable)
app.tabBars.buttons["Activity"].tap()
XCTAssertTrue(app.staticTexts["No tasks yet"].waitForExistence(timeout: 5))
app.buttons["Settings"].tap()
XCTAssertTrue(app.navigationBars["Settings"].exists)
capture("Accessibility text size")
}
private func capture(_ name: String) {
let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = name
attachment.lifetime = .keepAlways
add(attachment)
}
}
+143 -49
View File
@@ -12,29 +12,27 @@
28659D009399CA9C4FE57B74 /* ControlChannel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 61C71C48230C683D199D6F4B /* ControlChannel.swift */; };
302BEC55CA9A10E49C307977 /* ServerConfig.swift in Sources */ = {isa = PBXBuildFile; fileRef = F5683993A1DE677121FFA1F3 /* ServerConfig.swift */; };
36DF44D9A7896F18A064847D /* ConversationView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */; };
3E01A0AF44FD3324FA72C43C /* OrbView.swift in Sources */ = {isa = PBXBuildFile; fileRef = A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */; };
57E6476135CE30495E02D723 /* VoiceUITests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */; };
64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */ = {isa = PBXBuildFile; fileRef = A0C23305A6C55A83F8617D9D /* ConversationState.swift */; };
70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */; };
720C26EC163AE46713B02DD8 /* TLSPinning.swift in Sources */ = {isa = PBXBuildFile; fileRef = D9B40076934C28BA4B92D808 /* TLSPinning.swift */; };
72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */; };
73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */; };
7A8CC686530A5920338EDF5A /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 26922A8D6467BD51ED45B63B /* Assets.xcassets */; };
89B47754E5C39418BFD839E6 /* TranscriptHistory.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68A45228EF59186CFC219F2E /* TranscriptHistory.swift */; };
8DE1363CC0119B5AF9E4E117 /* ConnectionTestView.swift in Sources */ = {isa = PBXBuildFile; fileRef = AC4B9073EE8EAB51F8C55760 /* ConnectionTestView.swift */; };
943BCE971A01F220FAE9E250 /* TaskPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E85BACA267ADD462306C9920 /* TaskPanelView.swift */; };
9461C0886FA294FDD9742E51 /* TranscriptListView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BD31F76E1880AC76ED31DD4D /* TranscriptListView.swift */; };
98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 48F76881B4AE51DEE74511CC /* SettingsView.swift */; };
A5868238CF4958D6D0C5908E /* TaskStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0B60FBBB9C47BB9ADACA7CA /* TaskStore.swift */; };
A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */; };
A66BC894BEEC870D879F991F /* LightwellView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 87DD77AA383E1ECCEEEFB399 /* LightwellView.swift */; };
A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */; };
B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */; };
BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */; };
C00D1878C2AB3D8B6F9562B6 /* AgentMonitor.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */; };
C956D0FDFAC43925979E4077 /* MediaAPI.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9191AD60FF96E44F191864E3 /* MediaAPI.swift */; };
CDE38FE9679472D770C601E6 /* VoiceSession.swift in Sources */ = {isa = PBXBuildFile; fileRef = EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */; };
D18AD347DFF72AB6361165BA /* TokenClient.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9681E7410B684D2D38752A70 /* TokenClient.swift */; };
D2E343C8B8657316A993698E /* TranscriptHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */; };
D958BE8EE0FF9568F8FABB11 /* Nearlight.swift in Sources */ = {isa = PBXBuildFile; fileRef = CA6FA2A93116B237CEA712E6 /* Nearlight.swift */; };
D9A0059E342955A06A021D6C /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = A6BAD55715F383C0DB5F306F /* LiveKit */; };
DE1F24919E906EE13319C7A3 /* MediaPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */; };
EE0C5762EC97B123A1880FFB /* Voices.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2E5DF207EDB787F63CB3A64D /* Voices.swift */; };
F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 10C194656F19E89DFD656A91 /* MediaAPITests.swift */; };
FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */; };
/* End PBXBuildFile section */
@@ -46,33 +44,39 @@
remoteGlobalIDString = 57E3A292BDE525A22EB7ADE4;
remoteInfo = Voice_iOS;
};
ED7A5BB58354A8C9A5994257 /* PBXContainerItemProxy */ = {
isa = PBXContainerItemProxy;
containerPortal = 201BFA602F0A78910611E5F5 /* Project object */;
proxyType = 1;
remoteGlobalIDString = 57E3A292BDE525A22EB7ADE4;
remoteInfo = Voice_iOS;
};
/* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */
09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaReconcileTests.swift; sourceTree = "<group>"; };
0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AgentMonitor.swift; sourceTree = "<group>"; };
10C194656F19E89DFD656A91 /* MediaAPITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPITests.swift; sourceTree = "<group>"; };
20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerReplacementCoordinatorTests.swift; sourceTree = "<group>"; };
26922A8D6467BD51ED45B63B /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = "<group>"; };
2A4C3CE2CA6FF828452803D1 /* Voice.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = Voice.app; sourceTree = BUILT_PRODUCTS_DIR; };
2E5DF207EDB787F63CB3A64D /* Voices.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Voices.swift; sourceTree = "<group>"; };
2FED8976B60FBC13BC980FCB /* VoiceApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceApp.swift; sourceTree = "<group>"; };
3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlayerController.swift; sourceTree = "<group>"; };
3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationView.swift; sourceTree = "<group>"; };
48F76881B4AE51DEE74511CC /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = "<group>"; };
5759EB570DB87BBC121D946D /* VoiceUITests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceLiveTests.swift; sourceTree = "<group>"; };
61C71C48230C683D199D6F4B /* ControlChannel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ControlChannel.swift; sourceTree = "<group>"; };
68A45228EF59186CFC219F2E /* TranscriptHistory.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHistory.swift; sourceTree = "<group>"; };
68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPanelView.swift; sourceTree = "<group>"; };
6D0D153557BB1C756ED04C61 /* SettingsStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsStore.swift; sourceTree = "<group>"; };
6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoader.swift; sourceTree = "<group>"; };
8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoaderTests.swift; sourceTree = "<group>"; };
9191AD60FF96E44F191864E3 /* MediaAPI.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPI.swift; sourceTree = "<group>"; };
8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceUITests.swift; sourceTree = "<group>"; };
87DD77AA383E1ECCEEEFB399 /* LightwellView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = LightwellView.swift; sourceTree = "<group>"; };
9681E7410B684D2D38752A70 /* TokenClient.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TokenClient.swift; sourceTree = "<group>"; };
A0C23305A6C55A83F8617D9D /* ConversationState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationState.swift; sourceTree = "<group>"; };
A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = OrbView.swift; sourceTree = "<group>"; };
AC4B9073EE8EAB51F8C55760 /* ConnectionTestView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConnectionTestView.swift; sourceTree = "<group>"; };
BD31F76E1880AC76ED31DD4D /* TranscriptListView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptListView.swift; sourceTree = "<group>"; };
BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationStateTests.swift; sourceTree = "<group>"; };
C0B60FBBB9C47BB9ADACA7CA /* TaskStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TaskStore.swift; sourceTree = "<group>"; };
C22975C69240AACD3418091F /* VoiceTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
CA6FA2A93116B237CEA712E6 /* Nearlight.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Nearlight.swift; sourceTree = "<group>"; };
D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHandler.swift; sourceTree = "<group>"; };
D9B40076934C28BA4B92D808 /* TLSPinning.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TLSPinning.swift; sourceTree = "<group>"; };
E85BACA267ADD462306C9920 /* TaskPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TaskPanelView.swift; sourceTree = "<group>"; };
@@ -93,14 +97,6 @@
/* End PBXFrameworksBuildPhase section */
/* Begin PBXGroup section */
0D2E974CBC286D7C886BA814 /* Media */ = {
isa = PBXGroup;
children = (
68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */,
);
path = Media;
sourceTree = "<group>";
};
0DF0591DF1D767F89099F62C /* Settings */ = {
isa = PBXGroup;
children = (
@@ -124,6 +120,15 @@
path = LiveKit;
sourceTree = "<group>";
};
6735DFBC89D9A939C44D763A /* UITests */ = {
isa = PBXGroup;
children = (
5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */,
8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */,
);
path = UITests;
sourceTree = "<group>";
};
6B2AB34274D9B37E77178C6C /* Networking */ = {
isa = PBXGroup;
children = (
@@ -139,6 +144,7 @@
children = (
2A4C3CE2CA6FF828452803D1 /* Voice.app */,
C22975C69240AACD3418091F /* VoiceTests.xctest */,
5759EB570DB87BBC121D946D /* VoiceUITests.xctest */,
);
name = Products;
sourceTree = "<group>";
@@ -147,7 +153,8 @@
isa = PBXGroup;
children = (
3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */,
A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */,
87DD77AA383E1ECCEEEFB399 /* LightwellView.swift */,
BD31F76E1880AC76ED31DD4D /* TranscriptListView.swift */,
);
path = Conversation;
sourceTree = "<group>";
@@ -156,7 +163,6 @@
isa = PBXGroup;
children = (
84ADAF12A43A57888CA178A6 /* Conversation */,
0D2E974CBC286D7C886BA814 /* Media */,
0DF0591DF1D767F89099F62C /* Settings */,
F168B25F8B3C6F36B6E88228 /* Tasks */,
);
@@ -166,25 +172,18 @@
B3D0379BEBE1E5F4AA58DA2D = {
isa = PBXGroup;
children = (
EB99F8CAC2A1FB46873C1E48 /* Resources */,
FCC61E9990B71498171F9D34 /* Sources */,
DC8495F2B27E1770E6620A05 /* Tests */,
6735DFBC89D9A939C44D763A /* UITests */,
6F99423E59017FED59B4D71D /* Products */,
);
sourceTree = "<group>";
};
C12426487989358DE21A5426 /* Media */ = {
isa = PBXGroup;
children = (
9191AD60FF96E44F191864E3 /* MediaAPI.swift */,
3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */,
6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */,
);
path = Media;
sourceTree = "<group>";
};
CE49A38E40AAF659A9281B85 /* App */ = {
isa = PBXGroup;
children = (
CA6FA2A93116B237CEA712E6 /* Nearlight.swift */,
6D0D153557BB1C756ED04C61 /* SettingsStore.swift */,
2FED8976B60FBC13BC980FCB /* VoiceApp.swift */,
2E5DF207EDB787F63CB3A64D /* Voices.swift */,
@@ -196,15 +195,20 @@
isa = PBXGroup;
children = (
BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */,
10C194656F19E89DFD656A91 /* MediaAPITests.swift */,
09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */,
8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */,
20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */,
FAEF89B2DEEE68F7168A8014 /* TranscriptDedupeTests.swift */,
);
path = Tests;
sourceTree = "<group>";
};
EB99F8CAC2A1FB46873C1E48 /* Resources */ = {
isa = PBXGroup;
children = (
26922A8D6467BD51ED45B63B /* Assets.xcassets */,
);
path = Resources;
sourceTree = "<group>";
};
F168B25F8B3C6F36B6E88228 /* Tasks */ = {
isa = PBXGroup;
children = (
@@ -219,7 +223,6 @@
CE49A38E40AAF659A9281B85 /* App */,
8DA8BEF2EE3E0F4181A94EB9 /* Features */,
2820A418F921E18A2FA7E4FB /* LiveKit */,
C12426487989358DE21A5426 /* Media */,
6B2AB34274D9B37E77178C6C /* Networking */,
);
path = Sources;
@@ -246,11 +249,30 @@
productReference = C22975C69240AACD3418091F /* VoiceTests.xctest */;
productType = "com.apple.product-type.bundle.unit-test";
};
1C8CC2859E99A4AC83FC6374 /* VoiceUITests */ = {
isa = PBXNativeTarget;
buildConfigurationList = 286F46F3E4A225D8ABDCEA48 /* Build configuration list for PBXNativeTarget "VoiceUITests" */;
buildPhases = (
889523F0856C91F5A5E8FD41 /* Sources */,
);
buildRules = (
);
dependencies = (
A506145EF5A1B3978391C33C /* PBXTargetDependency */,
);
name = VoiceUITests;
packageProductDependencies = (
);
productName = VoiceUITests;
productReference = 5759EB570DB87BBC121D946D /* VoiceUITests.xctest */;
productType = "com.apple.product-type.bundle.ui-testing";
};
57E3A292BDE525A22EB7ADE4 /* Voice_iOS */ = {
isa = PBXNativeTarget;
buildConfigurationList = 67CE95EFE28F976D1CF1A59A /* Build configuration list for PBXNativeTarget "Voice_iOS" */;
buildPhases = (
548DD373FE3F2A49C30AA9CB /* Sources */,
AD64D238C9844386DB9B2B44 /* Resources */,
F79CBCE3DCE9D63A26B167D9 /* Frameworks */,
);
buildRules = (
@@ -274,7 +296,11 @@
BuildIndependentTargetsInParallel = YES;
LastUpgradeCheck = 1430;
TargetAttributes = {
1C8CC2859E99A4AC83FC6374 = {
TestTargetID = 57E3A292BDE525A22EB7ADE4;
};
57E3A292BDE525A22EB7ADE4 = {
DevelopmentTeam = YJFYRB92NW;
ProvisioningStyle = Automatic;
};
};
@@ -296,12 +322,24 @@
projectDirPath = "";
projectRoot = "";
targets = (
1385EEF29189F331F2634382 /* VoiceTests_iOS */,
57E3A292BDE525A22EB7ADE4 /* Voice_iOS */,
1385EEF29189F331F2634382 /* VoiceTests_iOS */,
1C8CC2859E99A4AC83FC6374 /* VoiceUITests */,
);
};
/* End PBXProject section */
/* Begin PBXResourcesBuildPhase section */
AD64D238C9844386DB9B2B44 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
7A8CC686530A5920338EDF5A /* Assets.xcassets in Resources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXResourcesBuildPhase section */
/* Begin PBXSourcesBuildPhase section */
548DD373FE3F2A49C30AA9CB /* Sources */ = {
isa = PBXSourcesBuildPhase;
@@ -312,11 +350,8 @@
28659D009399CA9C4FE57B74 /* ControlChannel.swift in Sources */,
64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */,
36DF44D9A7896F18A064847D /* ConversationView.swift in Sources */,
C956D0FDFAC43925979E4077 /* MediaAPI.swift in Sources */,
DE1F24919E906EE13319C7A3 /* MediaPanelView.swift in Sources */,
73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */,
3E01A0AF44FD3324FA72C43C /* OrbView.swift in Sources */,
A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */,
A66BC894BEEC870D879F991F /* LightwellView.swift in Sources */,
D958BE8EE0FF9568F8FABB11 /* Nearlight.swift in Sources */,
302BEC55CA9A10E49C307977 /* ServerConfig.swift in Sources */,
A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */,
98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */,
@@ -326,20 +361,27 @@
D18AD347DFF72AB6361165BA /* TokenClient.swift in Sources */,
D2E343C8B8657316A993698E /* TranscriptHandler.swift in Sources */,
89B47754E5C39418BFD839E6 /* TranscriptHistory.swift in Sources */,
9461C0886FA294FDD9742E51 /* TranscriptListView.swift in Sources */,
1ECD63F10A8EAF33FD2A5B62 /* VoiceApp.swift in Sources */,
CDE38FE9679472D770C601E6 /* VoiceSession.swift in Sources */,
EE0C5762EC97B123A1880FFB /* Voices.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
889523F0856C91F5A5E8FD41 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */,
57E6476135CE30495E02D723 /* VoiceUITests.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
C735E25A3F84D701AEBFC3F1 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */,
F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */,
72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */,
B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */,
BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */,
1813A1CB0921D04FCAEC92D0 /* TranscriptDedupeTests.swift in Sources */,
);
@@ -353,6 +395,11 @@
target = 57E3A292BDE525A22EB7ADE4 /* Voice_iOS */;
targetProxy = 1C0045D170F971FC571D0E54 /* PBXContainerItemProxy */;
};
A506145EF5A1B3978391C33C /* PBXTargetDependency */ = {
isa = PBXTargetDependency;
target = 57E3A292BDE525A22EB7ADE4 /* Voice_iOS */;
targetProxy = ED7A5BB58354A8C9A5994257 /* PBXContainerItemProxy */;
};
/* End PBXTargetDependency section */
/* Begin XCBuildConfiguration section */
@@ -438,6 +485,24 @@
};
name = Debug;
};
452ED10E7C45C3EDEDC466B5 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
GENERATE_INFOPLIST_FILE = YES;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = dev.shane.voice.uitests;
SDKROOT = iphoneos;
SWIFT_VERSION = 5.9;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_TARGET_NAME = Voice_iOS;
};
name = Release;
};
8CD9F2FF6EF1EB5438545350 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
@@ -464,6 +529,7 @@
CODE_SIGN_IDENTITY = "iPhone Developer";
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = YJFYRB92NW;
GENERATE_INFOPLIST_FILE = NO;
INFOPLIST_FILE = Resources/Info.plist;
LD_RUNPATH_SEARCH_PATHS = (
@@ -486,6 +552,7 @@
CODE_SIGN_IDENTITY = "iPhone Developer";
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = YJFYRB92NW;
GENERATE_INFOPLIST_FILE = NO;
INFOPLIST_FILE = Resources/Info.plist;
LD_RUNPATH_SEARCH_PATHS = (
@@ -557,9 +624,36 @@
};
name = Release;
};
EEDD1CAF56A9063A94459A9B /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
GENERATE_INFOPLIST_FILE = YES;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = dev.shane.voice.uitests;
SDKROOT = iphoneos;
SWIFT_VERSION = 5.9;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_TARGET_NAME = Voice_iOS;
};
name = Debug;
};
/* End XCBuildConfiguration section */
/* Begin XCConfigurationList section */
286F46F3E4A225D8ABDCEA48 /* Build configuration list for PBXNativeTarget "VoiceUITests" */ = {
isa = XCConfigurationList;
buildConfigurations = (
EEDD1CAF56A9063A94459A9B /* Debug */,
452ED10E7C45C3EDEDC466B5 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Debug;
};
67CE95EFE28F976D1CF1A59A /* Build configuration list for PBXNativeTarget "Voice_iOS" */ = {
isa = XCConfigurationList;
buildConfigurations = (
+44
View File
@@ -0,0 +1,44 @@
# NAS deployment — 2026-09-08
Historical deployment record, not a current health assertion. The iOS music player
was removed September 12; the media/AAC details below concern the retained backend,
not a current iOS feature. Reinspect server state before making deployment changes.
The `voice` container on `192.168.86.2` hosts LiveKit, the voice agent,
token service, music API and nginx. The app connects to
`https://192.168.86.2:9178`. Docker uses host networking and an
`unless-stopped` restart policy.
- Image: `hope-voice-api-voice:68f4c46-ios-audio`.
- LLM: **qwen2.5-vl-3b**, through OpenWebUI at `http://192.168.86.2:3005/api`.
- Speech: existing Azure configuration, unchanged.
- Persistent certs, skills and music: `/mnt/user/dev/hope-voice-api` mounts.
- Build source: `/mnt/user/dev/hope-voice-deploy-68f4c46`, a separate clean
archive plus the iOS audio compatibility patch. The pre-existing dirty
backend checkout was not overwritten.
- Local compatibility source: sibling `hope-voice-api/web/media_server.py`;
regression test: `hope-voice-api/tests/test_ios_audio.py`.
The music service retains original WebM files. An explicit `?format=m4a`
request generates and caches AAC audio under `media/files/ios`, serving
byte ranges for iOS playback and seeking. Browser requests remain unchanged.
## Health and rollback
```sh
ssh [email protected] 'docker inspect --format "{{.State.Status}} {{.State.Health.Status}}" voice'
```
The stopped container `voice-before-ios-audio` retains the baseline deployment.
If rollback is needed, first inspect both containers, then stop the current
`voice`, rename it to a unique recovery name, rename `voice-before-ios-audio`
to `voice`, and start it. This restores the baseline without deleting images,
containers or the music library; it does not include iOS AAC conversion.
`scripts/deploy-backend.sh` records the launch configuration and refuses to
replace an existing `voice` container. Run it on the NAS only after building
the named image and explicitly handling the existing container. It reads
credentials on the NAS and never prints them. The backend compatibility source was
not pushed during that deployment; recheck the sibling repository before acting.
An iOS push does not publish it. Future backend CI deployments must include the
compatibility patch to preserve it.
+19 -23
View File
@@ -1,18 +1,25 @@
# Device testing checklist
Simulator verification covered connect/disconnect, transcripts (text mode), media
reconciliation/UI, tasks panel, and the hardening pass. The items below need a **real
Simulator verification covered connect/disconnect, transcripts (text mode), tasks
panel, and the hardening pass. The items below need a **real
iPhone on the LAN** (`https://192.168.86.2:9178`) because mic input, WebRTC capture,
and audio session behavior don't work in this simulator environment (CoreAudio `-4010`,
0 input channels).
Setup: build with Xcode for a physical device (or `xcodebuild ... -destination`), install,
ensure the server cert pin is current (see README — re-export after any server restart).
Setup: recheck device availability and signing, then build/install with Xcode or
the installed XcodeBuildMCP skill workflow. The phone was paired and the Nearlight
build launched successfully on September 12; that does not prove today's availability
or a still-valid provisioning profile. Verify the server certificate pin after an
actual certificate rotation, not automatically after every container restart.
For each check, record date, source revision, device/iOS version, backend/model,
observed result and supporting logs. Leave unchecked items unqualified; installing
the app or passing a simulator state test is not equivalent to hearing a reply.
## 1. Mic input with real speech
- [ ] Start → agent joins → mic enables **without** `-4010`.
- [ ] Speak: orb/status transitions to **listening**; mic meter moves with voice level.
- [ ] Speak: Lightwell/status transitions to **listening**; mic meter moves with voice level.
- [ ] Stop speaking, wait >2 s while agent is silent → **thinking** chip appears.
- [ ] Agent replies: state goes to **speaking** (remote audio drives it), TTS audible.
- [ ] User transcript finals arrive in the list with correct role/timestamps.
@@ -21,7 +28,7 @@ ensure the server cert pin is current (see README — re-export after any server
- [ ] While the agent is speaking, start talking → conversation continues without a
"barge-in" glitch; state returns to listening/thinking as expected.
- [ ] Text mode: barge-in hint line visible while in text mode.
- [ ] Switching to Type leaves microphone capture off and text messaging usable.
## 3. Wake lock (screen off)
@@ -31,29 +38,18 @@ ensure the server cert pin is current (see README — re-export after any server
side button to lock the device. It only disables the automatic idle timeout.
- [ ] After hang-up, auto-lock returns (idle timer re-enabled).
## 4. Audible media playback
- [ ] Requires a track in an iOS-decodable format (m4a/aac/mp3) — current library tracks
are webm and will fail with `-11828` until the server serves a supported format.
- [ ] Play a library track: audio is audible, now-playing card shows art/title/status.
- [ ] Pause / ±10 s / stop / seek all work; agent mics off while playing
(`media_state` published), mics back on when stopped.
- [ ] End conversation while music was playing before → playback resumes.
## 5. Audio interruption (phone call / Siri)
## 4. Audio interruption (phone call / Siri)
- [ ] Mid-conversation, trigger an interruption (incoming call or "Hey Siri").
- [ ] Mic is muted on interruption begin; conversation/media pause cleanly.
- [ ] On end (with `shouldResume`), mic and media resume automatically.
- [ ] Media panel: AVPlayer pauses on interruption and resumes after.
- [ ] On end (with `shouldResume`), mic and conversation resume automatically.
## 6. Mic permission denied fallback
## 5. Mic permission denied fallback
- [ ] Deny microphone permission in iOS Settings for the app, relaunch.
- [ ] Start → session reaches **ready** despite mic failure (non-fatal); orange error
banner shows the mic error; text mode remains fully usable.
- [ ] Start → session reaches **ready** despite mic failure (non-fatal); a visible
error explains the problem and offers iPhone Settings recovery; Type remains usable.
## 7. General device sanity
## 6. General device sanity
- [ ] Connect / hang-up / reconnect matrix (3×) — no crashes, state resets cleanly.
- [ ] Change server URL in Settings while idle → session recreates without crash.
+89
View File
@@ -0,0 +1,89 @@
# Engineering learnings
Updated 2026-09-19 from repository source, recorded test evidence, and prior debugging.
This is guidance, not a claim that hardware or server checks were rerun today.
Read [PROJECT_STATUS.md](PROJECT_STATUS.md) for the current product scope.
## Preserve behavior across UI changes
- Observe nested observable objects directly where their data is rendered. Watching
only the parent session does not guarantee transcript/state/task updates redraw.
- Keep one conversation session across tabs. Server replacement must remain
latest-wins and reject late callbacks from a torn-down session.
- Keep Settings presentation outside the replaced session's identity. Otherwise
saving a changed server can destroy the navigation stack and its success feedback.
- Treat normalization as a programmatic edit: trimming a saved URL must not clear
its own confirmation. Clear confirmation on the next genuine user edit instead.
- A restored transcript is reading history, not a request to jump to the bottom.
User scrolling, text selection, and VoiceOver row focus suspend auto-follow;
explicit **Latest** restores it. Test the reading position, not just row existence.
- Gate Send on transport readiness, reconnect state and pending sends. Retain an
unsent draft on failure. Type mode must not enable microphone capture.
- A ready room can still have a microphone failure. Show the actual error and its
recovery action without disabling the usable text path.
These are regression invariants of the current Nearlight implementation, not a
reason to restore the removed music UI or the canceled ChatGPT migration.
## Match evidence to the claim
- A token/TLS check is not an agent-join test; Docker health is not a Qwen-response
test; app installation is not proof of microphone capture or audible output.
- Offline UI fixtures must use isolated preferences and deterministic local failure
endpoints. Never reset production preferences to make a UI test pass.
- Keep live tests opt-in. Record which model/server was used, what was actually
asserted, skips, result bundles, and any cleanup of test-created state.
- Test repeated lifecycle sequences, not just one successful launch. Repeat runs
exposed an intermittent playback timeout that a single green run missed.
- After an interrupted tool call, inspect the existing process/result before
restarting. A lost tool handle does not imply the underlying test stopped.
- When UI snapshots expose no targets, cross-check screenshots and XCTest results;
this occurred despite a rendered app and was not sufficient evidence of a crash.
- Keep failed-run evidence. A later pass does not explain the earlier failure.
Source review and automated large-text checks also do not replace manual VoiceOver
or real-device speech/interruption testing.
## Device connection and signing
Diagnose the layers separately: USB detection → pairing/trust → developer/signing
readiness → install/launch → live voice behavior. An unlocked phone may still be
unpaired. Once USB detects it, investigate pairing rather than repeatedly blaming
the cable. Query fresh device details; do not reuse an old availability report.
The phone was eventually paired and the final Nearlight build installed/launched
on September 12. That supersedes the earlier Finder/pairing blocker. Recheck its
current availability and provisioning before the next run; the recorded personal
provisioning was short-lived. Do not reset privacy settings or erase a device as a
routine diagnostic step.
## Project generation and assets
`project.yml` is the source for `Voice.xcodeproj`; the scheme is `Voice_iOS`, bundle
ID `dev.shane.voice`. Adding an icon file alone is insufficient: the asset catalog
must also be included in the resource build phase. The September 12 follow-up
commits added the Lightwell icon and regenerated the project to include it.
Regenerate deliberately and review generated diffs alongside source changes.
## NAS deployment and historical audio findings
- Inspect actual containers, images, mounts and health before recreating anything.
A user concern about deletion is a reason to check, not proof of deletion.
- Preserve the persistent certificate. A container restart need not rotate it;
update the app pin only after verifying an actual certificate change.
- Stage clean deployment source separately from a dirty remote checkout. Retain
library/configuration mounts and a recoverable previous container.
- Record the deployed image and source patch. Cross-repository changes must be
published separately, with any push-triggered deployment reviewed first.
- Do not print environment files, API keys, or full credential-bearing configuration.
Historical only—the embedded player no longer exists in this app:
- WebM/Opus library downloads needed cached AAC for the iOS playback path tested.
Preserve originals and verify byte ranges/cache behavior separately from conversion.
- AVFoundation's resource-loader content type required a UTI, not an HTTP MIME type.
- Buffering an entire 6 MB response before returning audio reproduced an AVPlayer
timeout. Bounded 256 KB range delivery passed three subsequent live repetitions.
These findings do not establish support for every radio format or indefinite stream.
See [DEPLOYMENT.md](DEPLOYMENT.md) and [VERIFICATION.md](VERIFICATION.md) for recorded
deployment and test history. Do not reintroduce the client player to preserve history.
+170
View File
@@ -0,0 +1,170 @@
# Project status and handoff
Last updated: 2026-09-19 (documentation review; no new runtime qualification).
Reusable implementation, testing, device and deployment lessons are in
[ENGINEERING_LEARNINGS.md](ENGINEERING_LEARNINGS.md). September 12 test/install
results below are historical evidence, not a September 19 health check.
## Current direction
Keep **Qwen 2.5**. The user explicitly canceled the earlier request to switch to
a ChatGPT subscription/device-token login. Do not revive that migration because
an old saved goal mentions it. Current work is the native iOS UI/UX redesign
and end-to-end qualification.
**2026-09-12: the embedded music player was removed** by user decision — the
feature is gone from the app (Media tab, `Sources/Media/`, `MediaPanelView`,
`MediaAPI`, `MediaPlayerController`, `PinnedMediaResourceLoader`, media tests,
and the `media_state` control publish). The backend media API still exists and
is untouched. The backend AAC compatibility patch (below) is now irrelevant to
the iOS app but remains deployed.
The **Nearlight** native redesign is implemented across Talk, Activity, Settings,
pickers, task details, and connection diagnostics. Current simulator qualification
is green, and GPT-6 Astra's final blocker re-verification returned **PASS**. The
final source also produced a signed generic-iOS build, and once the phone
reconnected the final build was installed and launched on the paired iPhone.
Hardware audio and interruption qualification remains open.
## Completed work
### Native interface and usability
- The app now has native **Talk** and **Activity** tabs sharing one session;
changing tabs never ends a conversation.
- Implemented Astra's **Nearlight** direction: an adaptive plum/rosewood semantic
palette, restrained rounded typography, shared spacing/radius/motion tokens, and
the compact Lightwell aperture as the conversation state signature.
- Rebuilt Talk around a clear pre-connection welcome, Voice/Type choice, explicit
start/end controls, compact active status, microphone meter, persistent Type
draft, and a dock that keeps primary actions reachable.
- Reworked the transcript as a readable shared record with speaker/time metadata,
live partials, copy/selection, explicit destructive confirmation, and measured
follow-latest behavior. Restored history stays at the top; manual reading,
selection, and VoiceOver row focus suspend following until **Latest**.
- Made connection and ready-session microphone failures visible without discarding
the real error. Permission denial exposes an iPhone Settings recovery action;
Type mode remains available.
- Reworked Settings around voice, mode, model, and connection. Server edits are
validated and explicitly saved; normalization confirmation survives session
replacement and clears on the next user edit.
- Improved Activity hierarchy, active-count badge, empty state, state-specific
rows, and task detail presentation.
- Removed forced dark appearance; all redesigned surfaces support light/dark mode,
Dynamic Type, Reduce Motion, increased contrast branches, and 44-point-or-larger
interaction targets where applicable.
### Functional fixes
- Observed nested transcript, conversation-state, and task objects so asynchronous
updates render.
- Preserved latest-wins server replacement while keeping an open Settings
navigation stack and its success feedback intact.
- Preserved Type-mode microphone privacy, pending-message send gating, clean session
teardown, transcript persistence/deduplication, and interruption ordering.
- All media-related client behavior was removed with the music player on 2026-09-12.
### Tests and tooling
- Follow-up commits added the Lightwell app icon (`7ac8a1c`) and regenerated the
project to include its asset-catalog resource phase (`85169d6`). The qualification
results below predate those follow-ups; do not describe them as a fresh full run
of the current HEAD.
- Installed XcodeBuildMCP and its skills; used it for simulator build/run/test work.
- The current combined simulator run passed **50 tests, 0 failures**, with the
opt-in live suite skipped once. Five offline UI scenarios cover recovery,
settings validation/persistence and save feedback, transcript/task flows,
restored-history reading position, and accessibility-size navigation.
- A post-focus-change targeted restored-history UI regression passed independently.
- DEBUG UI fixtures use isolated preferences; ordinary application preferences and
the NAS are not modified by offline UI runs.
- GPT-6 Astra completed the design specification and four verification rounds. Its
final scoped review found B1–B3 resolved, N1/N2 preserved, and no concrete
remaining release blocker. Accepted visual/adaptive/accessibility polish remains
documented as debt, not as exhaustive WCAG or VoiceOver qualification.
## Backend deployment and cross-repository dependency
SSH host: `[email protected]` (Unraid). App URL: `https://192.168.86.2:9178`.
- `voice` image: `hope-voice-api-voice:68f4c46-ios-audio`, host networking,
restart policy `unless-stopped`. Includes LiveKit, voice agent, token service,
media API, and nginx.
- LLM: `qwen2.5-vl-3b` through existing OpenWebUI at
`http://192.168.86.2:3005/api`. Existing Azure speech configuration retained.
- `voice`, `open-webui`, and `local-vl-models` were present and healthy when the
user suspected deletion. Do not recreate healthy services unnecessarily.
- Original library, certificates, and skills were retained under
`/mnt/user/dev/hope-voice-api`; the dirty remote checkout was not overwritten.
- Separate deployment source: `/mnt/user/dev/hope-voice-deploy-68f4c46`.
- Stopped rollback container: `voice-before-ios-audio`.
- Backend AAC conversion preserves original WebM files and serves cached AAC with
byte-range support. Browser/raw requests retain their original behavior.
**Important:** the backend patch and its test are in a different local repository:
`/Users/shane/dev/hope-voice-api/web/media_server.py` and
`/Users/shane/dev/hope-voice-api/tests/test_ios_audio.py`. They were still uncommitted
at the earlier deployment handoff; their current sibling-repository status has not
been rechecked in this documentation update. Committing/pushing the iOS repository
does **not** publish those files.
The deployed image contains the patch, but a future backend CI rebuild can lose it
unless it is committed and deployed through that repository's workflow. Inspect
that workflow before pushing: it may replace the live container.
Launch configuration and rollback details: [DEPLOYMENT.md](DEPLOYMENT.md).
Do not copy credentials or `.env` contents into documentation or commits.
## Verification performed
Project `Voice.xcodeproj`, scheme `Voice_iOS`, iPhone 17 Pro simulator
`AFDA137E-1657-428E-BF01-E5CD7989772C`.
- Current combined run: **50 passed, 0 failed, 1 live skip** in 104.9 seconds.
Result bundle:
`test_sim_2026-09-12T14-03-09-058Z_pid95846_2ca9da7e.xcresult`.
- Post-focus-change restored-history regression: **1 passed, 0 failed**.
Result bundle:
`test_sim_2026-09-12T14-05-21-070Z_pid98187_bd3e531a.xcresult`.
- A fresh isolated simulator launch showed restored history top-anchored with
**Latest** visible; the UI regression then tapped Latest and verified the final
event became visible.
- Astra's targeted isolated runtime check verified a padded changed-address save
stays in Settings, normalizes the field, shows its checkmark and confirmation,
and clears both only after the next user edit. Final verdict: **PASS**.
- iPhone 14 Pro Max `BF93D35E-0853-5470-A125-6EB0DB3F17AB` was paired, signed,
installed, trusted, and launched successfully. The final Nearlight build
(`/tmp/voice-ios-device-derived/Build/Products/Debug-iphoneos/Voice.app`,
`dev.shane.voice`) was reinstalled and launched via `devicectl` after the phone
reconnected. Free provisioning expires after seven days; reinstall to re-sign.
- Historical live NAS runs verified Qwen replies, background task completion, and
reconnect. Backend integration remains 16 passed, 7 skipped, 2 deselected.
Exact historical qualifications: [VERIFICATION.md](VERIFICATION.md). Artifacts
live on this Mac under the XcodeBuildMCP workspace, not in Git.
## What remains, in order
1. **Qualify real audio:** microphone capture, TTS audibility, speaker/headphones,
barge-in, Siri/phone interruptions, lock/background behavior,
microphone-permission-denied fallback, and network-loss recovery. These require
the user's speech/listening or a real interruption; simulator state tests do not
prove them. Follow [DEVICE_TESTING.md](DEVICE_TESTING.md).
2. **Broader live regression:** exercise slow-network cancellation, network-loss
recovery, repeated server replacement, and voice/model selection against the
live agent.
3. **Manual accessibility/visual qualification:** inspect populated/error screens
in light and dark mode, accessibility text sizes, narrow/landscape layouts, and
a real VoiceOver focus session. Astra accepted the remaining presentation debt
for this personal-app release; that is not proof of exhaustive conformance.
4. **Publish the backend compatibility source safely** in the sibling repository
and ensure its normal deployment preserves mounts, certificates, and Qwen
settings. This is lower priority because the iOS app no longer consumes the
media API, but the deployed container still serves it for the web UI.
## Repository handoff
The iOS work and this report belong to branch `feat/ios-app`, remote `origin`
(`https://git.srmr.xyz/snowspeeder/hope-voice-ios.git`). Keep the Nearlight
implementation, tests, physical-device notes, and this handoff together when
committing. Check Git history and remote status for the final publish result.
+91
View File
@@ -0,0 +1,91 @@
# Verification — 2026-09-12
Documentation reviewed 2026-09-19; no tests rerun for that update. These results
predate the subsequent app-icon and generated-resource-phase commits. See
[ENGINEERING_LEARNINGS.md](ENGINEERING_LEARNINGS.md) for reusable lessons.
## Scope
Qwen 2.5 remains configured; the canceled subscription-login migration was not
revived. The embedded music player was removed by user decision. The current app
is the Astra-designed **Nearlight** native interface with shared Talk and Activity
tabs, adaptive semantic styling, the Lightwell conversation indicator, rebuilt
conversation controls, readable transcripts, actionable failure states, explicit
Settings saves, and revised Activity/detail surfaces.
## Current iOS evidence
Project: `Voice.xcodeproj`; scheme: `Voice_iOS`; simulator: iPhone 17 Pro
`AFDA137E-1657-428E-BF01-E5CD7989772C`.
- Full combined simulator run: **50 passed, 0 failed, 1 live skip** in 104.9
seconds. It includes unit/state tests and five offline UI scenarios.
- The offline UI scenarios cover navigation/offline recovery; persisted voice and
server validation; normalized same-address and changed-address save feedback;
transcript clearing and Activity details; restored-history reading position and
explicit Latest recovery; and accessibility-size navigation.
- A second targeted run after the final VoiceOver eligibility strengthening passed
`testRestoredTranscriptStaysAtTopUntilLatest`: **1 passed, 0 failed**.
- Fresh runtime inspection showed the long restored transcript at its beginning
with the **Latest** affordance visible. The regression tapped Latest and verified
the final row became visible.
- GPT-6 Astra re-read its design specification and prior findings, inspected the
final source, and performed an isolated Settings runtime reproduction. Its
round-four verdict was **PASS**: reading eligibility, ready-session microphone
recovery visibility, and save-confirmation lifetime were resolved; the earlier
selection and pending-Send fixes remained intact.
- Astra's Settings reproduction saved ` https://127.0.0.1:2 ` using isolated
UI-test preferences. Settings stayed open, the field normalized, and the
checkmark plus “Server address saved.” remained visible. The next user edit
cleared both indicators. It did not touch normal preferences or the NAS.
Result bundles under
`~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/result-bundles/`:
- Full run:
`test_sim_2026-09-12T14-03-09-058Z_pid95846_2ca9da7e.xcresult`
- Targeted restored-history run:
`test_sim_2026-09-12T14-05-21-070Z_pid98187_bd3e531a.xcresult`
## Physical-device evidence
iPhone 14 Pro Max `BF93D35E-0853-5470-A125-6EB0DB3F17AB` was paired, built for,
signed with personal team `YJFYRB92NW`, installed, trusted, and launched through
`devicectl`. This proves device signing/install/launch only, not microphone or
speaker behavior. Free provisioning expires after seven days.
The final source also produced a signed device build and, once the phone
reconnected, `devicectl` installed and launched it successfully on 2026-09-12.
This proves device signing/install/launch of the final binary, not microphone or
speaker behavior.
## Historical live and backend evidence
Earlier live NAS runs verified Qwen replies, background task completion,
disconnect/reconnect, and three consecutive Qwen/task/reconnect iterations. Older
music playback evidence is historical only: the player and its tests no longer
exist in the iOS app.
Backend integration previously passed 16 tests with 7 skipped and 2 deselected.
The AAC compatibility regression passed and the deployed byte-range request
returned HTTP 206. That patch remains in a separate backend repository and is not
published by this iOS branch.
## Not yet qualified
- Audible speaker/headphone output, physical microphone capture, barge-in,
phone/Siri interruption recovery, lock/background behavior, and live
network-loss recovery.
- A real microphone-permission-denied session. The visibility/recovery branches
are source-reviewed, but no permission was changed during final verification.
- A full VoiceOver audio/focus session, exhaustive contrast audit, every Dynamic
Type/layout combination, or strict conformance to every Nearlight design detail.
Astra explicitly accepted the remaining visual/adaptive/presentation debt for
this personal release.
- A fresh build/install and regression run after the subsequent app-icon/resource
commits. The September 12 Nearlight install succeeded; current phone availability
and provisioning have not been rechecked here.
Follow [DEVICE_TESTING.md](DEVICE_TESTING.md) for hardware qualification and
[PROJECT_STATUS.md](PROJECT_STATUS.md) for current priorities and deployment
dependencies.
+16 -2
View File
@@ -14,11 +14,11 @@ targets:
platform: [iOS]
sources:
- Sources
- path: Resources/Assets.xcassets
buildPhase: resources
dependencies:
- package: LiveKit
product: LiveKit
resources:
- path: Resources/Assets.xcassets
info:
path: Resources/Info.plist
properties:
@@ -41,6 +41,7 @@ targets:
TARGETED_DEVICE_FAMILY: "1"
CURRENT_PROJECT_VERSION: 1
MARKETING_VERSION: "0.1.0"
DEVELOPMENT_TEAM: YJFYRB92NW
CODE_SIGN_STYLE: Automatic
VoiceTests:
type: bundle.unit-test
@@ -54,3 +55,16 @@ targets:
SWIFT_VERSION: "5.9"
GENERATE_INFOPLIST_FILE: YES
PRODUCT_BUNDLE_IDENTIFIER: dev.shane.voice.tests
VoiceUITests:
type: bundle.ui-testing
platform: iOS
sources:
- UITests
dependencies:
- target: Voice_iOS
settings:
base:
SWIFT_VERSION: "5.9"
GENERATE_INFOPLIST_FILE: YES
PRODUCT_BUNDLE_IDENTIFIER: dev.shane.voice.uitests
TEST_TARGET_NAME: Voice_iOS
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Run on xNAS after building the clean backend source into the pinned image.
# Credentials stay on the NAS. Existing checkout, library and certs are retained.
set -euo pipefail
deployment_source=/mnt/user/dev/hope-voice-deploy-68f4c46
deployment_data=/mnt/user/dev/hope-voice-api
deployment_image=${HOPE_DEPLOY_IMAGE:-hope-voice-api-voice:68f4c46-ios-audio}
test -f "$deployment_data/.env"
docker image inspect "$deployment_image" >/dev/null
if docker container inspect voice >/dev/null 2>&1; then
echo 'A voice container already exists; inspect it before replacing it.' >&2
exit 1
fi
# Recover the OpenWebUI credential used by this exact backend deployment source.
LLM_API_KEY=$(sed -n 's/.*-e LLM_API_KEY=\([^ ]*\).*/\1/p' "$deployment_source/.gitea/workflows/rebuild.yml")
test -n "$LLM_API_KEY"
export LLM_API_KEY
export LLM_BASE_URL=http://192.168.86.2:3005/api
export LLM_MODEL=qwen2.5-vl-3b
docker run -d --name voice --network host --restart unless-stopped \
--env-file "$deployment_data/.env" \
--env LLM_BASE_URL --env LLM_MODEL --env LLM_API_KEY \
--env LIVEKIT_URL=http://localhost:7880 \
--env VOICE_USERS_FILE=/etc/hope/users.json \
--env VOICE_DEFAULT_USER=shane --env MEDIA_DIR=/data/media \
--env MEDIA_SERVER_PORT=8093 --env WEB_PORT=9178 \
--env YTDLP_COOKIES=/data/media/cookies.txt \
--volume "$deployment_data/certs:/etc/voice/certs" \
--volume "$deployment_data/skills:/skills" \
--volume "$deployment_data/media:/data/media" \
--volume "$deployment_source/livekit.yaml:/etc/livekit.yaml:ro" \
--health-cmd='curl -sfk https://localhost:9178/ -o /dev/null && curl -sf http://localhost:7880/ -o /dev/null && curl -sf http://localhost:8093/media/api/healthz -o /dev/null' \
--health-interval=15s --health-timeout=5s --health-retries=3 --health-start-period=30s \
"$deployment_image"