Redesign Hope iOS UI, harden playback, and document project handoff

This commit is contained in:
snowspeeder committed 2026-09-09 20:23:44 -04:00
1 parent 40aaf4fea7
commit 24f95639dc
24 files changed
+1460 -584

No files matched your search

+2
View File
@@ -1,3 +1,5 @@
# AGENTS.md
- On startup, read `docs/PROJECT_STATUS.md` before planning or changing this project. It is the current handoff report and records decisions, completed work, verification, deployment dependencies, and remaining tasks.
- Keep `docs/PROJECT_STATUS.md` updated as work progresses, especially after implementation, testing, deployment, or changes to blockers and user decisions. Distinguish verified results from assumptions, and update it before handing off or committing material changes.
- If using XcodeBuildMCP, use the installed XcodeBuildMCP skill before calling XcodeBuildMCP tools.
+13 -12
View File
@@ -2,11 +2,11 @@
Native SwiftUI iOS app (iOS 17+) that talks to the [Hope Voice](https://github.com/) agent:
voice/text conversation, transcript history, voice & model pickers, media player, and a
background-tasks panel. Mirrors the core web UI of `hope-voice-api`; no server-side changes.
background-tasks panel. Uses `hope-voice-api`, including cached AAC conversion for iOS music playback.
> **Status:** Core functionality is implemented and verified in simulator to the extent the
> environment allows. Full device qualification (mic capture, audible media playback,
> interruption behavior) and the Hope visual redesign (`ui-ux-spec.md`) remain pending —
> interruption behavior) remains pending. The native Talk/Music/Activity redesign is implemented —
> see [PLAN.md](PLAN.md) and [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md).
## Prerequisites
@@ -37,9 +37,9 @@ session when idle). The app derives LiveKit signaling and media endpoints from t
### TLS cert pinning
The deployment uses a self-signed cert that is **regenerated at container start**, so the
pinned SHA-256 in `Sources/Networking/TLSPinning.swift` breaks on every cert rotation.
Re-export after a server restart:
The deployment uses a persistent self-signed cert. The entrypoint regenerates it if
missing or issued for a different IP. The pinned SHA-256 in
`Sources/Networking/TLSPinning.swift` must match after any actual cert rotation:
```sh
openssl s_client -connect 192.168.86.2:9178 </dev/null 2>/dev/null \
@@ -53,7 +53,7 @@ fail with a pin mismatch (not a network error).
- **Conversation** — LiveKit room with the Hope agent; voice mode (mic + TTS) and text mode
(composer), toggleable and persisted. State machine (idle/connecting/listening/thinking/
speaking) driven by audio levels, orb animation, mic meter, barge-in hint.
speaking) driven by audio levels, with a compact status indicator and mic meter.
- **Transcripts** — partials updated in place, finals with timestamps, delivery-marker
stripping, last-50 history persisted to UserDefaults, clear button.
- **Voice & model pickers** — 13 Azure TTS voices; LLM models from the agent's `models`
@@ -62,23 +62,24 @@ fail with a pin mismatch (not a network error).
±10 s, stop), seek + volume sliders, search with "in library" tags, agent-offered choices,
library list (play/remove), add-stream form. AVPlayer-backed with a pinned-TLS Range proxy;
1.5 s `/state` polling and full web-parity reconciliation/reporting.
- **Background tasks panel** — live `task_event` cards (steps, result, error) with an active
count badge in the toolbar.
- **Activity tab** — live `task_event` updates (steps, result, error) with an active
count badge on the tab.
## Known limitations
- **Simulator mic**: CoreAudio returns `-4010` (no usable input source) in this environment;
voice capture must be verified on a real device. Text mode works in the simulator.
- **webm media**: iOS AVPlayer cannot decode webm/opus (`-11828`). Library tracks from yt-dlp
are webm, so audible playback requires the server to serve m4a/aac/mp3. Everything up to
decode (pinning, Range proxy, reconciliation, UI) is verified; no client change needed once
a supported format exists.
- **Music compatibility**: library WebM files are requested as cached AAC on iOS.
This requires the deployed backend compatibility patch; see [deployment](docs/DEPLOYMENT.md).
- **Cert pin rotation**: see above — re-export the pin after each server cert regeneration.
- **LiveKit signaling** uses plain `ws://192.168.86.2:7880` (not wss) because the Swift SDK
cannot trust the self-signed cert on its WebSocket; token + media still go over pinned TLS.
## Docs
- [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) — current handoff, decisions, and remaining work; read first.
- [docs/VERIFICATION.md](docs/VERIFICATION.md) — test evidence and remaining qualification.
- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — live NAS configuration and rollback.
- [PLAN.md](PLAN.md) — phase-by-phase progress, verified facts, open items.
- [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md) — real-device qualification checklist.
- [UPDATE_PATH.md](UPDATE_PATH.md) — original plan & protocol reference (read-only).
+29 -6
View File
@@ -3,19 +3,19 @@ import Combine
final class SettingsStore: ObservableObject {
@Published var serverURL: String {
didSet { UserDefaults.standard.set(serverURL, forKey: Keys.serverURL) }
didSet { AppPreferences.store.set(serverURL, forKey: Keys.serverURL) }
}
@Published var voiceID: String {
didSet { UserDefaults.standard.set(voiceID, forKey: Keys.voiceID) }
didSet { AppPreferences.store.set(voiceID, forKey: Keys.voiceID) }
}
@Published var modelID: String? {
didSet { UserDefaults.standard.set(modelID, forKey: Keys.modelID) }
didSet { AppPreferences.store.set(modelID, forKey: Keys.modelID) }
}
@Published var textMode: Bool {
didSet { UserDefaults.standard.set(textMode, forKey: Keys.textMode) }
didSet { AppPreferences.store.set(textMode, forKey: Keys.textMode) }
}
@Published var volume: Double {
didSet { UserDefaults.standard.set(volume, forKey: Keys.volume) }
didSet { AppPreferences.store.set(volume, forKey: Keys.volume) }
}
private enum Keys {
@@ -29,7 +29,7 @@ final class SettingsStore: ObservableObject {
static let defaultServerURL = "https://192.168.86.2:9178"
init() {
let d = UserDefaults.standard
let d = AppPreferences.store
// Sanitize legacy values: earlier versions wrote the field to storage on
// every keystroke, so an invalid half-typed URL could be persisted and
// shipped to the network layer. Fall back to the documented default and
@@ -54,3 +54,26 @@ final class SettingsStore: ObservableObject {
serverURL.trimmingCharacters(in: .whitespacesAndNewlines)
}
}
enum AppPreferences {
static let store: UserDefaults = {
#if DEBUG
if ProcessInfo.processInfo.arguments.contains("--ui-testing") {
let suite = "dev.shane.voice.ui-testing"
let defaults = UserDefaults(suiteName: suite)!
if ProcessInfo.processInfo.arguments.contains("--reset-ui-testing") {
defaults.removePersistentDomain(forName: suite)
// Deterministic offline errors; UI tests must not contact the NAS.
defaults.set("https://127.0.0.1:1", forKey: "serverURL")
}
if ProcessInfo.processInfo.arguments.contains("--live-testing") {
defaults.set(SettingsStore.defaultServerURL, forKey: "serverURL")
defaults.set(true, forKey: "textMode")
defaults.set("qwen2.5-vl-3b", forKey: "modelID")
}
return defaults
}
#endif
return .standard
}()
}
+9 -1
View File
@@ -12,7 +12,15 @@ struct VoiceApp: App {
var body: some Scene {
WindowGroup {
ConversationView(settings: settings)
.preferredColorScheme(.dark)
.tint(HopeStyle.tint)
}
}
}
enum HopeStyle {
static let tint = Color(uiColor: UIColor { traits in
traits.userInterfaceStyle == .dark
? UIColor(red: 0.43, green: 0.68, blue: 1, alpha: 1)
: UIColor(red: 0.12, green: 0.32, blue: 0.68, alpha: 1)
})
}
@@ -1,343 +1,347 @@
import SwiftUI
import Combine
/// Owns one session across all tabs. Changing tabs never ends a conversation.
struct ConversationView: View {
@ObservedObject var settings: SettingsStore
@State private var sessionID = UUID()
@State private var session: VoiceSession
/// The exact server URL the current session was constructed with. Used to
/// avoid redundant session replacement when the applied URL hasn't actually
/// changed (e.g. re-applying the same value from Settings).
@State private var activeServerURL: String
/// Identity guard: prevents a second `.onChange` from starting a concurrent
/// session replacement while the first is still disconnecting the old one.
@State private var isReplacingSession = false
/// Latest accepted server URL that has not yet been applied to a session.
/// Coalesced: rapid Applies during an active replacement retain the newest
/// value rather than dropping it. A single retained worker loop drains
/// arbitrary pending URLs until none remain (Iteration 08 Item 2).
@State private var pendingServerURL: String?
@State private var composerText = ""
@State private var connectError: String?
@State private var tick = 0
@State private var isReplacingSession = false
@State private var sessionID = UUID()
@State private var selectedTab = 0
init(settings: SettingsStore) {
self.settings = settings
let initialSession = Self.makeSession(settings: settings)
_session = State(initialValue: initialSession)
_activeServerURL = State(initialValue: settings.trimmedServerURL)
_session = State(initialValue: Self.makeSession(settings.trimmedServerURL, settings: settings))
}
/// Creates a session from the configured server URL. `SettingsStore.init`
/// guarantees the stored value is always valid (legacy invalid values are
/// sanitized to the documented default), so the failure branch is a safety
/// net only.
private static func makeSession(settings: SettingsStore) -> VoiceSession {
let config = ServerConfig.parse(settings.trimmedServerURL)
switch config {
case .success(let value):
return VoiceSession(config: value, tokenClient: TokenClient(config: value), settings: settings)
case .failure:
// Defensive: should be unreachable because SettingsStore sanitizes.
let fallback = ServerConfig.parse(SettingsStore.defaultServerURL)
if case .success(let value) = fallback {
settings.serverURL = SettingsStore.defaultServerURL
return VoiceSession(config: value, tokenClient: TokenClient(config: value), settings: settings)
private static func makeSession(_ url: String, settings: SettingsStore) -> VoiceSession {
guard case .success(let config) = ServerConfig.parse(url) else {
preconditionFailure("SettingsStore must supply a validated server URL")
}
fatalError("Unreachable: default server URL is valid")
let session = VoiceSession(config: config, tokenClient: TokenClient(config: config), settings: settings)
#if DEBUG
if ProcessInfo.processInfo.arguments.contains("--ui-fixtures") {
session.transcriptHandler.receiveDataTranscript(role: .user, text: "Help me plan a quiet weekend.")
session.transcriptHandler.receiveDataTranscript(role: .agent, text: "Let’s leave some room to breathe. A walk, a good book, and time for your favorite music would be a lovely start.")
session.taskStore.injectSampleEvents()
}
#endif
return session
}
var body: some View {
TabView(selection: $selectedTab) {
NavigationStack {
VStack(spacing: 0) {
statusHeader
TranscriptListView(
events: session.transcriptHandler.events,
onClear: {
// Atomically clear the canonical visible collection,
// dedupe state, and persistence so Clear is truthful.
session.transcriptHandler.clearAll()
ConversationContent(settings: settings, session: session)
.toolbar { settingsLink }
}
)
composer
}
.navigationTitle("Hope Voice")
.toolbar { toolbarContent }
.id(sessionID)
.onAppear {
if case .failure(let error) = ServerConfig.parse(settings.trimmedServerURL) {
connectError = error.localizedDescription
}
}
.onChange(of: settings.textMode) { _, newValue in
guard session.isReady else { return }
Task { await session.applyTextMode(newValue) }
}
}
.onReceive(session.objectWillChange.receive(on: RunLoop.main)) { _ in
tick &+= 1
}
.onChange(of: settings.serverURL) { _, newValue in
// The URL field now uses explicit Apply semantics (SettingsView),
// so this fires only when the user has applied a value that passed
// validation. Coalesce rapid accepted values: every accepted Apply
// writes `pendingServerURL` (latest wins). A single retained worker
// loop drains arbitrary pending URLs until none remain, covering
// A→B→C semantics where C arrives while B is disconnecting
// (Iteration 08 Item 2).
let trimmed = newValue.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty, case .success = ServerConfig.parse(trimmed) else { return }
.tabItem { Label("Talk", systemImage: "waveform") }
.tag(0)
pendingServerURL = trimmed
// Start the worker if not already active. The loop consumes
// pending values until empty, so a single Task handles any number
// of rapid Applies without nested tasks.
guard !isReplacingSession else { return }
isReplacingSession = true
Task { @MainActor [settings] in
while let url = self.pendingServerURL {
self.pendingServerURL = nil
guard url != self.activeServerURL else { continue }
let oldSession = self.session
await oldSession.disconnect()
self.activeServerURL = url
self.sessionID = UUID()
// Construct from the captured validated config, not by
// rereading settings.serverURL (which may have changed
// during disconnect).
switch ServerConfig.parse(url) {
case .success(let config):
self.session = VoiceSession(config: config, tokenClient: TokenClient(config: config), settings: settings)
case .failure:
let fallback = ServerConfig.parse(SettingsStore.defaultServerURL)
if case .success(let value) = fallback {
self.session = VoiceSession(config: value, tokenClient: TokenClient(config: value), settings: settings)
} else {
fatalError("Unreachable: default server URL is valid")
}
}
}
self.isReplacingSession = false
}
}
}
private var statusHeader: some View {
VStack(spacing: 12) {
OrbView(state: orbState)
Text(statusText)
.font(.headline)
if session.state == .ready, session.stateModel.isThinkingChipVisible {
Label("Thinking…", systemImage: "ellipsis")
.font(.caption)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(.orange.opacity(0.2), in: Capsule())
}
if session.state == .ready, !settings.textMode {
micMeter
}
if session.state == .ready, settings.textMode {
Text("Barge in anytime — type to reply")
.font(.caption)
.foregroundStyle(.secondary)
}
if let agent = session.agent {
Text("Agent: \(agent.name ?? agent.identity)")
.font(.caption)
.foregroundStyle(.secondary)
}
if session.state == .ready {
Text("\(Voices.display(for: settings.voiceID) ?? settings.voiceID) · \(session.selectedModel ?? settings.modelID ?? "—")")
.font(.caption)
.foregroundStyle(.secondary)
}
if session.state == .ready {
HStack(spacing: 16) {
// The mic toggle is disabled in text mode to enforce the
// privacy invariant at the UI level as well (Finding 4).
Toggle(isOn: Binding(
get: { !session.isMuted },
set: { newValue in Task { await session.setMuted(!newValue) } }
)) {
Label(session.isMuted ? "Muted" : "Mic on", systemImage: session.isMuted ? "mic.slash" : "mic")
}
.toggleStyle(.button)
.disabled(settings.textMode)
Toggle("Text mode", isOn: textModeBinding)
.toggleStyle(.button)
}
}
if session.state == .ready, session.isReconnecting {
Label("Connection lost — reconnecting…", systemImage: "arrow.triangle.2.circlepath")
.font(.footnote)
.foregroundStyle(.orange)
.frame(maxWidth: .infinity, alignment: .leading)
}
if let micError = session.micError {
Label(micError, systemImage: "exclamationmark.triangle")
.font(.footnote)
.foregroundStyle(.orange)
.frame(maxWidth: .infinity, alignment: .leading)
}
if let connectError {
Label(connectError, systemImage: "exclamationmark.triangle")
.font(.footnote)
.foregroundStyle(.red)
}
}
.padding()
}
private var micMeter: some View {
GeometryReader { geo in
ZStack(alignment: .leading) {
Capsule()
.fill(Color(.systemFill))
Capsule()
.fill(Color.green.gradient)
.frame(width: min(geo.size.width, geo.size.width * session.stateModel.micLevel * 2.5))
}
}
.frame(height: 4)
}
private var textModeBinding: Binding<Bool> {
$settings.textMode
}
private var composer: some View {
HStack(spacing: 8) {
switch session.state {
case .idle, .failed:
Button(action: startSession) {
Label("Start", systemImage: "phone.fill")
}
.buttonStyle(.borderedProminent)
case .connecting, .waitingAgent, .tearingDown:
ProgressView()
case .ready:
TextField(settings.textMode ? "Type a message…" : "Voice mode active",
text: $composerText,
axis: .vertical)
.textFieldStyle(.roundedBorder)
.lineLimit(1...4)
.disabled(!settings.textMode)
Button(action: sendComposer) {
Image(systemName: "arrow.up.circle.fill")
.font(.title2)
}
.accessibilityLabel("Send message")
.disabled(!settings.textMode || composerText.isEmpty)
Button(role: .destructive, action: endSession) {
Image(systemName: "phone.down.fill")
}
.accessibilityLabel("End conversation")
}
}
.padding()
}
@ToolbarContentBuilder
private var toolbarContent: some ToolbarContent {
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
TaskPanelView(taskStore: session.taskStore)
} label: {
Image(systemName: "tray.full")
.overlay(alignment: .topTrailing) {
if session.taskStore.activeCount > 0 {
Text("\(session.taskStore.activeCount)")
.font(.caption2.weight(.bold))
.foregroundStyle(.white)
.padding(3)
.background(Color.red, in: Circle())
.offset(x: 6, y: -6)
}
}
}
.accessibilityLabel("Background tasks")
}
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
NavigationStack {
MediaPanelView(playerController: session.mediaController, settings: settings)
} label: {
Image(systemName: "music.note")
.toolbar { settingsLink }
}
.accessibilityLabel("Media player")
.tabItem { Label("Music", systemImage: "music.note") }
.tag(1)
ActivityTab(taskStore: session.taskStore, settings: settings, session: session)
.tag(2)
}
.id(sessionID)
.tint(HopeStyle.tint)
.onChange(of: settings.textMode) { _, enabled in
Task { await session.applyTextMode(enabled) }
}
.onChange(of: settings.serverURL) { _, value in replaceServer(value) }
}
@ToolbarContentBuilder private var settingsLink: some ToolbarContent {
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
SettingsView(settings: settings, session: session)
} label: {
Image(systemName: "gearshape")
Image(systemName: "gearshape").frame(minWidth: 44, minHeight: 44)
}
.accessibilityLabel("Settings")
}
}
private var orbState: OrbView.OrbState {
switch session.state {
case .idle: return .idle
case .connecting, .waitingAgent, .tearingDown: return .connecting
case .ready:
guard !settings.textMode else { return .idle }
switch session.stateModel.phase {
case .idle: return .idle
case .connecting: return .connecting
case .listening: return .listening
case .thinking: return .thinking
case .speaking: return .speaking
private func replaceServer(_ value: String) {
let url = value.trimmingCharacters(in: .whitespacesAndNewlines)
guard case .success = ServerConfig.parse(url) else { return }
pendingServerURL = url
guard !isReplacingSession else { return }
isReplacingSession = true
Task { @MainActor in
// Coalesce rapid Apply actions; construct from the captured URL.
while let next = pendingServerURL {
pendingServerURL = nil
guard next != activeServerURL else { continue }
await session.disconnect()
session.mediaController.resetScope()
activeServerURL = next
session = Self.makeSession(next, settings: settings)
sessionID = UUID()
}
case .failed: return .failed
isReplacingSession = false
}
}
}
private struct ActivityTab: View {
@ObservedObject var taskStore: TaskStore
@ObservedObject var settings: SettingsStore
@ObservedObject var session: VoiceSession
var body: some View {
NavigationStack {
TaskPanelView(taskStore: taskStore)
.toolbar {
ToolbarItem(placement: .topBarTrailing) {
NavigationLink {
SettingsView(settings: settings, session: session)
} label: {
Image(systemName: "gearshape").frame(minWidth: 44, minHeight: 44)
}
.accessibilityLabel("Settings")
}
}
}
.tabItem { Label("Activity", systemImage: "checklist") }
.badge(taskStore.activeCount)
}
}
private struct ConversationContent: View {
@ObservedObject var settings: SettingsStore
@ObservedObject var session: VoiceSession
@ObservedObject private var transcripts: TranscriptHandler
@ObservedObject private var conversation: ConversationStateModel
@State private var composerText = ""
@State private var sendError: String?
@State private var isSending = false
@FocusState private var composerFocused: Bool
init(settings: SettingsStore, session: VoiceSession) {
self.settings = settings
self.session = session
transcripts = session.transcriptHandler
conversation = session.stateModel
}
var body: some View {
VStack(spacing: 0) {
if transcripts.events.isEmpty && !session.isReady {
welcome
} else {
connectionStatus.padding(.horizontal, 20).padding(.vertical, 12)
TranscriptListView(events: transcripts.events, onClear: transcripts.clearAll)
}
}
.background(Color(.systemBackground))
.navigationTitle("Hope")
.safeAreaInset(edge: .bottom, spacing: 0) { conversationControls }
.onChange(of: session.state) { _, state in
if state == .ready && settings.textMode { composerFocused = true }
}
}
private var statusText: String {
switch session.state {
case .idle: return "Ready to connect"
case .connecting: return "Connecting…"
case .waitingAgent: return "Waiting for agent…"
case .tearingDown: return "Cleaning up…"
case .ready:
if settings.textMode { return "Text mode" }
switch session.stateModel.phase {
case .listening: return "Listening"
case .thinking: return "Thinking it through"
case .speaking: return "Hope is speaking"
default: return "Ready when you are"
private var welcome: some View {
ScrollView {
VStack(alignment: .leading, spacing: 28) {
OrbView(state: orbState).padding(.top, 24).accessibilityHidden(true)
VStack(alignment: .leading, spacing: 12) {
Text("A little help.\nA good conversation.")
.font(.largeTitle.weight(.semibold))
.fixedSize(horizontal: false, vertical: true)
Text("Talk things through, find something out, or put on a favorite song.")
.font(.body).foregroundStyle(.secondary)
}
case .failed(let message): return "Failed: \(message)"
VStack(alignment: .leading, spacing: 16) {
Label("Think out loud", systemImage: "bubble.left.and.bubble.right")
Label("Find an answer", systemImage: "sparkle.magnifyingglass")
Label("Make time for music", systemImage: "music.note")
}
.font(.subheadline).foregroundStyle(.secondary)
}
.frame(maxWidth: .infinity, alignment: .leading)
.padding(.horizontal, 24).padding(.bottom, 24)
}
}
private func startSession() {
connectError = nil
// Starting a conversation preserves the restored last-50 history; only
// the explicit Clear button erases it.
private var connectionStatus: some View {
HStack(spacing: 12) {
Image(systemName: statusSymbol).font(.title3)
.foregroundStyle(session.state.isFailed ? Color.red : HopeStyle.tint)
.frame(width: 32, height: 36).accessibilityHidden(true)
VStack(alignment: .leading, spacing: 3) {
Text(statusTitle).font(.headline)
Text(statusDetail).font(.subheadline).foregroundStyle(.secondary)
}
Spacer(minLength: 0)
if session.isReady && !settings.textMode && !session.isMuted {
ProgressView(value: min(conversation.micLevel * 2.5, 1))
.frame(width: 46).accessibilityLabel("Microphone level")
}
}
.accessibilityElement(children: .combine)
}
private var conversationControls: some View {
VStack(spacing: 12) {
if let failure = failureMessage {
VStack(alignment: .leading, spacing: 6) {
Label("Couldn’t connect to Hope", systemImage: "wifi.exclamationmark")
.font(.subheadline.weight(.semibold))
Text(failure).font(.footnote).foregroundStyle(.secondary)
NavigationLink("Check connection settings") {
SettingsView(settings: settings, session: session)
}
.font(.subheadline).frame(minHeight: 44)
}
.frame(maxWidth: .infinity, alignment: .leading)
}
if session.isReconnecting {
Label("Reconnecting… Your conversation is still here.", systemImage: "wifi.exclamationmark")
.font(.footnote)
}
if let error = session.micError, !settings.textMode {
VStack(alignment: .leading, spacing: 4) {
Text(error).font(.footnote)
Button("Switch to typing") { settings.textMode = true }.frame(minHeight: 44)
}
.frame(maxWidth: .infinity, alignment: .leading)
}
if let sendError {
Text(sendError).font(.footnote).foregroundStyle(.red)
.frame(maxWidth: .infinity, alignment: .leading)
}
Picker("Conversation mode", selection: $settings.textMode) {
Label("Voice", systemImage: "mic").tag(false)
Label("Type", systemImage: "keyboard").tag(true)
}
.pickerStyle(.segmented).disabled(isBusy)
if session.isReady {
if settings.textMode { textComposer }
else {
HStack(spacing: 12) {
Button {
Task { await session.setMuted(!session.isMuted) }
} label: {
Label(session.isMuted ? "Unmute" : "Mute", systemImage: session.isMuted ? "mic.slash.fill" : "mic.fill")
.frame(maxWidth: .infinity, minHeight: 44)
}
.buttonStyle(.bordered)
endButton
}
}
} else {
Button {
sendError = nil
Task { await session.connect() }
} label: {
HStack(spacing: 10) {
if isBusy { ProgressView().tint(.white) }
else { Image(systemName: settings.textMode ? "bubble.left.fill" : "waveform") }
Text(isBusy ? statusTitle : session.state.isFailed ? "Try again" : settings.textMode ? "Start chatting" : "Let’s talk")
.font(.headline)
}
.frame(maxWidth: .infinity, minHeight: 48)
}
.buttonStyle(.borderedProminent).disabled(isBusy)
Text(settings.textMode ? "Your microphone stays off." : "Your microphone turns on when you connect.")
.font(.footnote).foregroundStyle(.secondary).multilineTextAlignment(.center)
}
}
.padding(.horizontal, 20).padding(.top, 12).padding(.bottom, 16)
.background(.bar)
}
private func endSession() {
private var textComposer: some View {
VStack(spacing: 6) {
HStack(alignment: .bottom, spacing: 8) {
TextField("Message Hope", text: $composerText, axis: .vertical)
.lineLimit(1...5).padding(12)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 12))
.focused($composerFocused).disabled(isSending)
.accessibilityLabel("Message Hope")
Button(action: sendComposer) {
if isSending { ProgressView().frame(width: 44, height: 44) }
else { Image(systemName: "arrow.up.circle.fill").font(.title).frame(width: 44, height: 44) }
}
.disabled(composerText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || isSending || session.isReconnecting)
.accessibilityLabel("Send message")
}
endButton
}
}
private var endButton: some View {
Button(role: .destructive) {
composerFocused = false
Task { await session.disconnect() }
} label: {
Label("End conversation", systemImage: "phone.down.fill")
.font(.subheadline.weight(.medium)).frame(minHeight: 44)
}
.buttonStyle(.bordered)
}
private var isBusy: Bool {
session.state == .connecting || session.state == .waitingAgent || session.state == .tearingDown
}
private var failureMessage: String? {
if case .failed(let message) = session.state { return message }
return nil
}
private var statusTitle: String {
switch session.state {
case .idle: return "Ready when you are"
case .connecting: return "Connecting…"
case .waitingAgent: return "Getting Hope ready…"
case .tearingDown: return "Ending conversation…"
case .failed: return "Connection unavailable"
case .ready:
if settings.textMode { return "You’re chatting with Hope" }
if session.isMuted { return "Microphone muted" }
switch conversation.phase {
case .thinking: return "Thinking it through…"
case .speaking: return "Hope is speaking"
default: return "I’m listening"
}
}
}
private var statusDetail: String {
if !session.isReady { return "Your recent conversation is below." }
if settings.textMode { return "Microphone off · Type whenever you’re ready" }
if session.isMuted { return "You can still hear Hope." }
return conversation.phase == .speaking ? "You can interrupt at any time." : "Speak naturally. Take your time."
}
private var statusSymbol: String {
session.state.isFailed ? "wifi.exclamationmark" : !session.isReady ? "bubble.left.and.bubble.right" : settings.textMode ? "keyboard" : session.isMuted ? "mic.slash" : "waveform"
}
private var orbState: OrbView.OrbState {
if session.state.isFailed { return .failed }
return isBusy ? .connecting : .idle
}
private func sendComposer() {
let text = composerText.trimmingCharacters(in: .whitespacesAndNewlines)
guard !text.isEmpty else { return }
composerText = ""
guard !text.isEmpty, !isSending else { return }
isSending = true
sendError = nil
Task {
do {
try await session.sendText(text)
composerText = ""
} catch {
connectError = error.localizedDescription
sendError = "Your message wasn’t sent. Try again when connected."
}
isSending = false
}
}
}
#Preview {
ConversationView(settings: SettingsStore())
}
#Preview { ConversationView(settings: SettingsStore()) }
+75 -112
View File
@@ -1,150 +1,113 @@
import SwiftUI
struct OrbView: View {
enum OrbState {
case idle
case connecting
case listening
case speaking
case thinking
case failed
}
enum OrbState { case idle, connecting, listening, speaking, thinking, failed }
let state: OrbState
@State private var animating = false
private var baseColor: Color {
switch state {
case .idle, .connecting: .blue
case .listening: .green
case .speaking, .thinking: .orange
case .failed: .red
}
}
private var scale: CGFloat {
switch state {
case .connecting, .listening, .speaking, .thinking: 1.1
default: 1.0
}
}
private var label: String {
switch state {
case .idle: "Idle"
case .connecting: "Connecting"
case .listening: "Listening"
case .speaking: "Speaking"
case .thinking: "Thinking"
case .failed: "Connection failed"
}
}
var body: some View {
let shape = Circle()
.fill(baseColor.gradient)
.frame(width: 120, height: 120)
let animated = shape
.scaleEffect(animating ? scale : 0.95)
.opacity(state == .idle || state == .failed ? 0.7 : 1.0)
let anim: Animation? = state == .idle || state == .failed
? nil
: .easeInOut(duration: 0.8).repeatForever(autoreverses: true)
animated
.animation(anim, value: animating)
.onAppear { animating = true }
.onChange(of: state) { _, _ in animating = true }
.accessibilityElement()
.accessibilityLabel("Conversation status")
.accessibilityValue(label)
ZStack {
Circle().fill(HopeStyle.tint.opacity(0.10))
Image(systemName: state == .failed ? "wifi.exclamationmark" : "waveform")
.font(.largeTitle.weight(.medium))
.foregroundStyle(HopeStyle.tint)
}
.frame(width: 76, height: 76)
.accessibilityLabel("Hope")
}
}
struct TranscriptListView: View {
/// The single canonical visible transcript (restored persisted finals plus
/// current live events). The view renders exactly this collection — there is
/// no second "restored" store, so the UI and persistence can never diverge.
let events: [TranscriptEvent]
var onClear: () -> Void
@State private var confirmClear = false
@State private var followingLatest = true
@Environment(\.accessibilityReduceMotion) private var reduceMotion
private static let deliveryMarkers = #"\[(?:appreciative|calm|concerned|curious|encouraging|excited|happy|joking|quiet|reassuring|reflective|sad|serious|surprised|sympathetic|urgent|whispering|laughter|sighing|breathing|neutral)\]\s*"#
private static let timestampFormatter: DateFormatter = {
let formatter = DateFormatter()
formatter.dateStyle = .none
formatter.timeStyle = .short
return formatter
}()
private var visibleRows: [TranscriptEvent] { events }
var body: some View {
ScrollViewReader { proxy in
VStack(alignment: .leading, spacing: 0) {
VStack(spacing: 0) {
HStack {
Text("Transcript")
.font(.caption.bold())
.foregroundStyle(.secondary)
Text("Conversation").font(.subheadline.weight(.semibold))
Spacer()
Button(role: .destructive, action: onClear) {
Label("Clear", systemImage: "trash")
.font(.caption)
Menu {
Button("Clear conversation", systemImage: "trash", role: .destructive) { confirmClear = true }
} label: {
Image(systemName: "ellipsis").frame(width: 44, height: 44)
}
.accessibilityLabel("Conversation options")
.disabled(events.isEmpty)
}
.padding(.horizontal)
.padding(.bottom, 4)
.padding(.horizontal, 20)
ScrollView {
LazyVStack(alignment: .leading, spacing: 8) {
ForEach(visibleRows) { event in
transcriptRow(event)
LazyVStack(alignment: .leading, spacing: 20) {
if events.isEmpty {
Text("Your conversation will appear here.")
.font(.body).foregroundStyle(.secondary)
.padding(.vertical, 32)
}
ForEach(events) { event in transcriptRow(event) }
Color.clear.frame(height: 1).id("latest")
.onAppear { followingLatest = true }
.onDisappear { followingLatest = false }
}
.padding(.horizontal)
.padding(.horizontal, 20).padding(.bottom, 20)
}
.onChange(of: visibleRows.count) { _, _ in
if let last = visibleRows.last {
withAnimation { proxy.scrollTo(last.id, anchor: .bottom) }
.scrollDismissesKeyboard(.interactively)
.defaultScrollAnchor(.bottom)
.onChange(of: events.last?.text) { _, _ in
if followingLatest { proxy.scrollTo("latest", anchor: .bottom) }
}
.onChange(of: events.count) { _, _ in
if followingLatest { proxy.scrollTo("latest", anchor: .bottom) }
}
if !followingLatest && !events.isEmpty {
Button {
withAnimation(reduceMotion ? nil : .easeOut(duration: 0.2)) {
proxy.scrollTo("latest", anchor: .bottom)
}
} label: {
Label("Latest messages", systemImage: "arrow.down").font(.subheadline)
.frame(minHeight: 44)
}
}
}
}
.alert("Clear this conversation?", isPresented: $confirmClear) {
Button("Clear conversation", role: .destructive, action: onClear)
Button("Cancel", role: .cancel) {}
} message: {
Text("This removes the transcript saved on this iPhone. It doesn’t reset Hope’s memory or end your current conversation.")
}
}
private func transcriptRow(_ event: TranscriptEvent) -> some View {
let roleName = event.role == .user ? "You" : "Hope"
let nameColor: Color = event.role == .user ? .secondary : .blue
let rowBackground: Color = event.role == .user
? Color(.secondarySystemBackground)
: Color.blue.opacity(0.08)
let textOpacity: Double = event.isFinal ? 1.0 : 0.55
return VStack(alignment: .leading, spacing: 2) {
HStack {
Text(roleName)
.font(.caption2.bold())
.foregroundStyle(nameColor)
Spacer()
Text(Self.timestampFormatter.string(from: event.timestamp))
.font(.caption2)
.foregroundStyle(.tertiary)
let isUser = event.role == .user
let text = event.text.replacingOccurrences(of: Self.deliveryMarkers, with: "", options: .regularExpression)
.trimmingCharacters(in: .whitespaces)
return HStack(alignment: .top, spacing: 0) {
if isUser { Spacer(minLength: 32) }
VStack(alignment: .leading, spacing: 7) {
HStack(spacing: 8) {
Text(isUser ? "You" : "Hope").font(.subheadline.weight(.semibold))
Text(event.timestamp, style: .time).font(.caption).foregroundStyle(.secondary)
if !event.isFinal {
Text("Live").font(.caption).foregroundStyle(.secondary)
}
Text(stripped(event.text))
.font(.body)
.opacity(textOpacity)
}
Text(text).font(.body).textSelection(.enabled)
.fixedSize(horizontal: false, vertical: true)
}
.padding(isUser ? 14 : 0)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(8)
.background(rowBackground)
.cornerRadius(10)
.background(isUser ? Color(.secondarySystemBackground) : .clear, in: RoundedRectangle(cornerRadius: 16))
.contextMenu {
Button("Copy message", systemImage: "doc.on.doc") { UIPasteboard.general.string = text }
}
if !isUser { Spacer(minLength: 16) }
}
.accessibilityElement(children: .combine)
.id(event.id)
}
private func stripped(_ text: String) -> String {
text.replacingOccurrences(
of: Self.deliveryMarkers,
with: "",
options: .regularExpression
).trimmingCharacters(in: .whitespaces)
}
}
+156 -62
View File
@@ -14,31 +14,76 @@ struct MediaPanelView: View {
@State private var streamURL = ""
@State private var streamMessage: String?
@State private var trackToDelete: MediaTrack?
@State private var browseMode = 0
@State private var showAddStream = false
@State private var isAddingStream = false
@State private var isLoadingLibrary = false
@State private var seekPosition: Double = 0
@State private var isSeeking = false
var body: some View {
ScrollView {
VStack(spacing: 20) {
if let stateError = playerController.stateError {
Label(stateError, systemImage: "wifi.exclamationmark")
.font(.subheadline)
.foregroundStyle(.red)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(12)
.background(Color.red.opacity(0.1), in: RoundedRectangle(cornerRadius: 14))
LazyVStack(alignment: .leading, spacing: 24) {
if playerController.stateError != nil {
VStack(alignment: .leading, spacing: 8) {
Label("Music server unavailable", systemImage: "wifi.exclamationmark")
.font(.headline)
Text("Check that your Hope server is running and you’re connected to its network. We’ll reconnect automatically.")
.font(.subheadline).foregroundStyle(.secondary)
}
.frame(maxWidth: .infinity, alignment: .leading)
}
if playerController.track != nil {
VStack(spacing: 20) {
nowPlayingCard
transportControls
seekSection
transportControls
volumeSection
}
.padding(16)
.background(Color(.secondarySystemGroupedBackground), in: RoundedRectangle(cornerRadius: 16))
}
if let error = playerController.playbackError {
Label(error, systemImage: "exclamationmark.circle")
.font(.subheadline).foregroundStyle(.red)
}
pendingSection
choicesSection
searchSection
librarySection
addStreamSection
Picker("Browse music", selection: $browseMode) {
Text("Library").tag(0)
Text("Search").tag(1)
}.pickerStyle(.segmented)
if browseMode == 0 { librarySection }
else { searchSection }
}
.padding()
}
.navigationTitle("Media")
.background(Color(.systemGroupedBackground))
.navigationTitle("Music")
.toolbar {
ToolbarItem(placement: .topBarTrailing) {
Button {
streamMessage = nil
showAddStream = true
} label: {
Image(systemName: "plus").frame(minWidth: 44, minHeight: 44)
}
.accessibilityLabel("Add radio station")
}
}
.sheet(isPresented: $showAddStream) {
NavigationStack {
Form { addStreamSection }
.navigationTitle("Add a station")
.navigationBarTitleDisplayMode(.inline)
.toolbar {
ToolbarItem(placement: .cancellationAction) {
Button("Cancel") { showAddStream = false }.disabled(isAddingStream)
}
}
}
.interactiveDismissDisabled(isAddingStream)
}
.onAppear {
playerController.startDefaultPolling()
refreshLibrary()
@@ -60,18 +105,7 @@ struct MediaPanelView: View {
private var nowPlayingCard: some View {
HStack(spacing: 12) {
if let artURL = playerController.track?.artURL, let url = URL(string: absolute(artURL)) {
AsyncImage(url: url) { phase in
switch phase {
case .success(let image):
image.resizable().scaledToFill()
default:
Image(systemName: "music.note")
.font(.title2)
.foregroundStyle(.secondary)
.frame(maxWidth: .infinity, maxHeight: .infinity)
.background(Color(.systemGray6))
}
}
PinnedArtwork(url: url)
.frame(width: 72, height: 72)
.clipShape(RoundedRectangle(cornerRadius: 10))
} else {
@@ -92,7 +126,7 @@ struct MediaPanelView: View {
Text(subtitle)
.font(.subheadline)
.foregroundStyle(.secondary)
.lineLimit(1)
.lineLimit(2)
if playerController.track != nil {
Text(statusText)
.font(.caption)
@@ -101,8 +135,6 @@ struct MediaPanelView: View {
}
Spacer()
}
.padding(12)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14))
}
private var transportControls: some View {
@@ -110,13 +142,15 @@ struct MediaPanelView: View {
Button(action: seekBackward) {
Image(systemName: "gobackward.10")
.font(.title3)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Seek back 10 seconds")
.disabled(playerController.track == nil)
.disabled(playerController.duration <= 0 || playerController.track?.kind == "stream")
Button(action: togglePlayPause) {
Image(systemName: playerController.desired == "playing" && playerController.track != nil ? "pause.circle.fill" : "play.circle.fill")
.font(.system(size: 44))
.font(.largeTitle)
.frame(width: 56, height: 56)
.foregroundStyle(playerController.track == nil ? Color.secondary : Color.accentColor)
}
.accessibilityLabel(playerController.desired == "playing" && playerController.track != nil ? "Pause" : "Play")
@@ -125,13 +159,15 @@ struct MediaPanelView: View {
Button(action: seekForward) {
Image(systemName: "goforward.10")
.font(.title3)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Seek forward 10 seconds")
.disabled(playerController.track == nil)
.disabled(playerController.duration <= 0 || playerController.track?.kind == "stream")
Button(role: .destructive, action: stopPlayback) {
Image(systemName: "stop.circle")
.font(.title2)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Stop playback")
.disabled(playerController.track == nil)
@@ -140,14 +176,23 @@ struct MediaPanelView: View {
private var seekSection: some View {
VStack(spacing: 4) {
if playerController.track != nil {
if playerController.track?.kind == "stream" {
Label("Live radio", systemImage: "dot.radiowaves.left.and.right")
.font(.subheadline).foregroundStyle(.secondary)
} else if playerController.track != nil {
Slider(
value: Binding(
get: { playerController.currentTime },
set: { newValue in Task { playerController.seek(to: newValue) } }
get: { min(isSeeking ? seekPosition : playerController.currentTime, max(playerController.duration, 1)) },
set: { seekPosition = $0 }
),
in: 0...max(playerController.duration, 1)
in: 0...max(playerController.duration, 1),
onEditingChanged: { editing in
isSeeking = editing
if !editing { playerController.seek(to: seekPosition) }
}
)
.disabled(playerController.duration <= 0)
.accessibilityLabel("Playback position")
HStack {
Text(fmtTime(playerController.currentTime))
.font(.caption.monospacedDigit())
@@ -172,6 +217,7 @@ struct MediaPanelView: View {
Image(systemName: "speaker.fill")
.foregroundStyle(.secondary)
Slider(value: $settings.volume, in: 0...1)
.accessibilityLabel("Music volume")
.onChange(of: settings.volume) { _, newValue in
playerController.setVolume(newValue)
}
@@ -227,15 +273,19 @@ struct MediaPanelView: View {
private var searchSection: some View {
VStack(alignment: .leading, spacing: 8) {
Text("Search YouTube Music")
Text("Find your next favorite")
.font(.headline)
Text("Search songs and artists on YouTube Music.")
.font(.subheadline).foregroundStyle(.secondary)
HStack(spacing: 8) {
TextField("Song, artist, lyrics…", text: $searchQuery)
.textFieldStyle(.roundedBorder)
.submitLabel(.search)
.onSubmit(performSearch)
Button("Find", action: performSearch)
Button("Search", action: performSearch)
.accessibilityIdentifier("music.search")
.buttonStyle(.bordered)
.frame(minHeight: 44)
.disabled(searchQuery.trimmingCharacters(in: .whitespaces).isEmpty || isSearching)
}
if isSearching {
@@ -251,7 +301,7 @@ struct MediaPanelView: View {
.foregroundStyle(.red)
} else if let results = searchResults {
if results.isEmpty {
Text("No results.")
Text("No matches. Try a different song or artist.")
.font(.subheadline)
.foregroundStyle(.secondary)
} else {
@@ -286,19 +336,24 @@ struct MediaPanelView: View {
.font(.headline)
Spacer()
Button(action: refreshLibrary) {
Image(systemName: "arrow.clockwise")
Image(systemName: "arrow.clockwise").frame(width: 44, height: 44)
}
.disabled(isLoadingLibrary)
.accessibilityLabel("Refresh library")
}
if let libraryError {
if let libraryError, playerController.stateError == nil {
Text(libraryError)
.font(.subheadline)
.foregroundStyle(.red)
} else if let tracks = libraryTracks {
if tracks.isEmpty {
Text("Downloaded songs and streams live here.")
.font(.subheadline)
.foregroundStyle(.secondary)
ContentUnavailableView {
Label("A home for your favorites", systemImage: "music.note.list")
} description: {
Text("Search for a song to save it here, or add a radio station using the + button.")
} actions: {
Button("Find music") { browseMode = 1 }.buttonStyle(.bordered)
}
} else {
ForEach(tracks, id: \.id) { track in
HStack(spacing: 8) {
@@ -319,23 +374,24 @@ struct MediaPanelView: View {
Image(systemName: "xmark")
.font(.subheadline.weight(.semibold))
.foregroundStyle(.red)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Remove \(track.title ?? track.id)")
}
}
}
} else if isLoadingLibrary {
ProgressView("Loading your library…").frame(maxWidth: .infinity).padding(.vertical, 24)
}
}
}
private var addStreamSection: some View {
VStack(alignment: .leading, spacing: 8) {
Text("Add live stream")
.font(.headline)
Section {
TextField("Station name", text: $streamName)
.textFieldStyle(.roundedBorder)
.accessibilityLabel("Station name")
TextField("https://… audio stream URL", text: $streamURL)
.textFieldStyle(.roundedBorder)
.accessibilityLabel("Audio stream URL")
.keyboardType(.URL)
.autocorrectionDisabled()
.textContentType(.URL)
@@ -345,9 +401,10 @@ struct MediaPanelView: View {
.font(.subheadline)
.foregroundStyle(streamMessage.hasPrefix("Added") ? Color.green : Color.red)
}
Button("Add", action: addStream)
.buttonStyle(.bordered)
.disabled(streamURL.trimmingCharacters(in: .whitespaces).isEmpty)
Button(isAddingStream ? "Adding station…" : "Add station", action: addStream)
.disabled(streamURL.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || isAddingStream)
} footer: {
Text("Use a direct MP3 or AAC audio stream address, rather than the station’s website.")
}
}
@@ -361,22 +418,22 @@ struct MediaPanelView: View {
VStack(alignment: .leading, spacing: 2) {
Text(title)
.font(.subheadline.weight(.semibold))
.lineLimit(1)
.lineLimit(2)
if !detail.isEmpty {
Text(detail)
.font(.caption)
.foregroundStyle(.secondary)
.lineLimit(1)
.lineLimit(2)
}
}
Spacer()
Button("Play", action: onPlay)
.buttonStyle(.bordered)
.controlSize(.small)
Button(action: onPlay) {
Image(systemName: "play.fill").frame(width: 44, height: 44)
}
.accessibilityLabel("Play \(title)")
.disabled(!playEnabled)
}
.padding(10)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 12))
.padding(.vertical, 10)
}
private func rowDetail(channel: String?, duration: Double?, status: String? = nil, inLibrary: Bool? = nil) -> String {
@@ -389,8 +446,8 @@ struct MediaPanelView: View {
}
private func performSearch() {
let query = searchQuery.trimmingCharacters(in: .whitespaces)
guard !query.isEmpty else { return }
let query = searchQuery.trimmingCharacters(in: .whitespacesAndNewlines)
guard !query.isEmpty, !isSearching else { return }
isSearching = true
searchError = nil
Task {
@@ -405,7 +462,10 @@ struct MediaPanelView: View {
}
private func refreshLibrary() {
guard !isLoadingLibrary else { return }
isLoadingLibrary = true
Task {
defer { isLoadingLibrary = false }
do {
let library = try await MediaAPI(config: serverConfig).library()
libraryTracks = library.tracks
@@ -417,15 +477,22 @@ struct MediaPanelView: View {
}
private func addStream() {
let url = streamURL.trimmingCharacters(in: .whitespaces)
guard !url.isEmpty else { return }
let url = streamURL.trimmingCharacters(in: .whitespacesAndNewlines)
guard !isAddingStream else { return }
guard let parsed = URL(string: url), ["https", "http"].contains(parsed.scheme?.lowercased() ?? ""), parsed.host != nil else {
streamMessage = "Enter a complete http:// or https:// audio stream address."
return
}
isAddingStream = true
let name = streamName.trimmingCharacters(in: .whitespaces).isEmpty ? url : streamName.trimmingCharacters(in: .whitespaces)
Task {
defer { isAddingStream = false }
do {
try await MediaAPI(config: serverConfig).addStream(room: "default", name: name, url: url)
streamMessage = "Added \(name)"
streamName = ""
streamURL = ""
showAddStream = false
refreshLibrary()
} catch {
streamMessage = "Could not add stream: \(error.localizedDescription)"
@@ -514,3 +581,30 @@ struct MediaPanelView: View {
return h > 0 ? "\(h):\(String(format: "%02d", m)):\(String(format: "%02d", s))" : "\(m):\(String(format: "%02d", s))"
}
}
/// Artwork uses the same trust policy as token and media requests.
private struct PinnedArtwork: View {
let url: URL
@State private var artwork: UIImage?
var body: some View {
Group {
if let artwork { Image(uiImage: artwork).resizable().scaledToFill() }
else {
Image(systemName: "music.note").font(.title2).foregroundStyle(.secondary)
.frame(maxWidth: .infinity, maxHeight: .infinity)
.background(Color(.tertiarySystemFill))
}
}
.accessibilityHidden(true)
.task(id: url) {
artwork = nil
let session = TLSPinning.session()
defer { session.invalidateAndCancel() }
guard let (data, response) = try? await session.data(from: url),
(response as? HTTPURLResponse)?.statusCode == 200,
!Task.isCancelled else { return }
artwork = UIImage(data: data)
}
}
}
@@ -26,7 +26,7 @@ struct ConnectionTestView: View {
if isBusy {
ProgressView()
} else {
Text("Get token")
Text("Test connection")
}
Spacer()
}
@@ -44,7 +44,7 @@ struct ConnectionTestView: View {
LabeledContent("User") {
Text(response.user.displayName ?? response.user.id)
}
Label("Token endpoint and TLS are working.", systemImage: "checkmark.circle")
Label("Connected to your server. Start a conversation to check Hope’s voice and replies.", systemImage: "checkmark.circle")
.foregroundStyle(.green)
}
case .failure(let error):
@@ -57,6 +57,7 @@ struct ConnectionTestView: View {
}
}
.navigationTitle("Connection test")
.navigationBarTitleDisplayMode(.inline)
}
private func runTest() {
+44 -32
View File
@@ -30,18 +30,48 @@ struct SettingsView: View {
var body: some View {
Form {
Section("Server") {
Section {
Button {
showVoicePicker = true
} label: {
LabeledContent("Speaking voice", value: Voices.display(for: settings.voiceID) ?? settings.voiceID)
}
Picker("Conversation mode", selection: $settings.textMode) {
Text("Voice").tag(false)
Text("Type · Microphone off").tag(true)
}
} header: {
Text("Make it yours")
} footer: {
Text("Voice and typing preferences are saved on this iPhone. You can switch modes from the Talk tab.")
}
Section {
if session.isReady, !session.models.isEmpty {
Button {
showModelPicker = true
} label: {
LabeledContent("Model", value: session.selectedModel ?? settings.modelID ?? "Server default")
}
} else {
LabeledContent("Model", value: settings.modelID ?? "Server default")
Text("Connect from Talk to see the models available on your server.")
.font(.footnote).foregroundStyle(.secondary)
}
} header: { Text("Intelligence") }
Section {
TextField(SettingsStore.defaultServerURL, text: $urlText)
.keyboardType(.URL)
.autocorrectionDisabled()
.textInputAutocapitalization(.never)
.accessibilityLabel("Server URL")
.onAppear { urlText = settings.serverURL }
if let urlError {
Text(urlError)
.font(.footnote)
.foregroundStyle(.red)
}
Button("Apply server URL") { applyServerURL() }
.disabled(!canApplyServerURL || urlText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty)
Button("Save server address") { applyServerURL() }
.disabled(!canApplyServerURL || urlText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || urlText == settings.serverURL)
if !canApplyServerURL {
Text(session.state == .tearingDown
? "Waiting for failure cleanup to finish…"
@@ -49,37 +79,17 @@ struct SettingsView: View {
.font(.footnote)
.foregroundStyle(.secondary)
}
}
Section("Voice") {
Button {
showVoicePicker = true
} label: {
LabeledContent("Voice", value: Voices.display(for: settings.voiceID) ?? settings.voiceID)
}
}
Section("Model") {
if session.isReady, !session.models.isEmpty {
Button {
showModelPicker = true
} label: {
LabeledContent("Model", value: session.selectedModel ?? settings.modelID ?? "—")
}
} else {
Text("Start a conversation to load the model list.")
.font(.footnote)
.foregroundStyle(.secondary)
}
}
Section("Interaction") {
Toggle("Text mode", isOn: $settings.textMode)
}
Section {
NavigationLink("Connection test") {
ConnectionTestView(settings: settings)
}
} header: {
Text("Connection")
} footer: {
Text("Hope runs on your own server. Your iPhone needs to be on the same network or a network that can reach it.")
}
}
.navigationTitle("Settings")
.navigationBarTitleDisplayMode(.inline)
.sheet(isPresented: $showVoicePicker) {
VoicePickerView(settings: settings, session: session)
}
@@ -115,7 +125,7 @@ struct SettingsView: View {
struct VoicePickerView: View {
@ObservedObject var settings: SettingsStore
var session: VoiceSession
@ObservedObject var session: VoiceSession
@Environment(\.dismiss) private var dismiss
var body: some View {
@@ -132,10 +142,11 @@ struct VoicePickerView: View {
if settings.voiceID == voice.id {
Spacer()
Image(systemName: "checkmark")
.foregroundStyle(.blue)
.foregroundStyle(HopeStyle.tint)
}
}
}
.accessibilityAddTraits(settings.voiceID == voice.id ? .isSelected : [])
}
}
.navigationTitle("Voice")
@@ -150,7 +161,7 @@ struct VoicePickerView: View {
struct ModelPickerView: View {
@ObservedObject var settings: SettingsStore
var session: VoiceSession
@ObservedObject var session: VoiceSession
@Environment(\.dismiss) private var dismiss
private var activeModelID: String? {
@@ -176,10 +187,11 @@ struct ModelPickerView: View {
if activeModelID == model.id {
Spacer()
Image(systemName: "checkmark")
.foregroundStyle(.blue)
.foregroundStyle(HopeStyle.tint)
}
}
}
.accessibilityAddTraits(activeModelID == model.id ? .isSelected : [])
}
}
.navigationTitle("Model")
+47 -43
View File
@@ -5,10 +5,12 @@ struct TaskPanelView: View {
var body: some View {
ScrollView {
VStack(spacing: 12) {
LazyVStack(alignment: .leading, spacing: 20) {
if taskStore.sortedTasks.isEmpty {
emptyState
} else {
Text(taskStore.activeCount > 0 ? "Hope is working on \(taskStore.activeCount) task\(taskStore.activeCount == 1 ? "" : "s"). You can keep talking." : "Results from the things you’ve asked Hope to work on.")
.font(.subheadline).foregroundStyle(.secondary)
ForEach(taskStore.sortedTasks) { task in
TaskCardView(task: task)
}
@@ -16,57 +18,48 @@ struct TaskPanelView: View {
}
.padding()
}
.navigationTitle("Background tasks")
#if DEBUG
.toolbar {
ToolbarItem(placement: .topBarTrailing) {
Button("Sample", action: { taskStore.injectSampleEvents() })
.font(.caption)
}
}
#endif
.navigationTitle("Activity")
.background(Color(.systemGroupedBackground))
}
private var emptyState: some View {
VStack(spacing: 8) {
Text("Nothing running")
.font(.headline)
Text("Ask Hope to research or compare something.")
.font(.subheadline)
.foregroundStyle(.secondary)
.multilineTextAlignment(.center)
ContentUnavailableView {
Label("A little help, in the background", systemImage: "checklist")
} description: {
Text("Ask Hope to research a topic or compare your options. Follow the progress and read the results here.")
}
.frame(maxWidth: .infinity)
.padding(.vertical, 48)
.padding(.top, 48)
}
}
struct TaskCardView: View {
let task: TaskItem
@State private var showSteps = false
var body: some View {
VStack(alignment: .leading, spacing: 8) {
HStack(spacing: 8) {
statusDot
Text(task.description)
.font(.subheadline.weight(.semibold))
.lineLimit(2)
VStack(alignment: .leading, spacing: 14) {
HStack {
Label(statusTitle, systemImage: statusSymbol)
.font(.subheadline.weight(.medium))
.foregroundStyle(dotColor)
Spacer()
Text(task.createdAt, style: .time).font(.caption).foregroundStyle(.secondary)
}
Text(task.description).font(.headline).fixedSize(horizontal: false, vertical: true)
if !task.steps.isEmpty {
VStack(alignment: .leading, spacing: 4) {
DisclosureGroup("\(task.steps.count) progress updates", isExpanded: $showSteps) {
VStack(alignment: .leading, spacing: 12) {
ForEach(task.steps) { step in
HStack(alignment: .top, spacing: 6) {
VStack(alignment: .leading, spacing: 4) {
Text(stepLabel(step.role))
.font(.caption2.weight(.semibold))
.foregroundStyle(stepColor(step.role))
.font(.caption.weight(.semibold)).foregroundStyle(.secondary)
Text(step.content ?? "")
.font(.caption)
.foregroundStyle(stepColor(step.role))
.lineLimit(3)
.font(.subheadline).textSelection(.enabled)
}
}
}.padding(.top, 8)
}
.font(.subheadline)
}
if let result = task.result {
block(result, color: .green, background: Color.green.opacity(0.12))
@@ -75,21 +68,33 @@ struct TaskCardView: View {
block(error, color: .red, background: Color.red.opacity(0.12))
}
}
.padding(12)
.padding(16)
.frame(maxWidth: .infinity, alignment: .leading)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14))
.background(Color(.secondarySystemGroupedBackground), in: RoundedRectangle(cornerRadius: 16))
}
private var statusDot: some View {
Circle()
.fill(dotColor)
.frame(width: 9, height: 9)
private var statusTitle: String {
switch task.status {
case .pending: "Queued"
case .running: "In progress"
case .completed: "Complete"
case .failed: "Needs attention"
}
}
private var statusSymbol: String {
switch task.status {
case .pending: "clock"
case .running: "arrow.triangle.2.circlepath"
case .completed: "checkmark.circle.fill"
case .failed: "exclamationmark.circle"
}
}
private var dotColor: Color {
switch task.status {
case .pending: return .gray
case .running: return .blue
case .running: return HopeStyle.tint
case .completed: return .green
case .failed: return .red
}
@@ -97,11 +102,10 @@ struct TaskCardView: View {
private func block(_ text: String, color: Color, background: Color) -> some View {
Text(text)
.font(.subheadline)
.foregroundStyle(color)
.font(.body)
.foregroundStyle(.primary)
.textSelection(.enabled)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(10)
.background(background, in: RoundedRectangle(cornerRadius: 10))
}
private func stepLabel(_ role: String?) -> String {
+2 -2
View File
@@ -20,7 +20,7 @@ final class TranscriptHistory: ObservableObject {
static let maxEntries = 50
init() {
if let raw = UserDefaults.standard.string(forKey: Self.key),
if let raw = AppPreferences.store.string(forKey: Self.key),
let data = raw.data(using: .utf8),
let decoded = try? JSONDecoder().decode([HistoryEntry].self, from: data) {
entries = Array(decoded.suffix(Self.maxEntries))
@@ -45,6 +45,6 @@ final class TranscriptHistory: ObservableObject {
private func persist() {
guard let data = try? JSONEncoder().encode(entries) else { return }
UserDefaults.standard.set(String(decoding: data, as: UTF8.self), forKey: Self.key)
AppPreferences.store.set(String(decoding: data, as: UTF8.self), forKey: Self.key)
}
}
+2 -2
View File
@@ -127,10 +127,10 @@ final class MediaAPI {
private let baseURL: URL
private let session: URLSession
init(config: ServerConfig) {
init(config: ServerConfig, session: URLSession? = nil) {
self.serverBase = config.baseURL
self.baseURL = config.baseURL.appendingPathComponent("media/api")
self.session = TLSPinning.session()
self.session = session ?? TLSPinning.session()
}
func state(room: String) async throws -> MediaState {
+49 -11
View File
@@ -17,6 +17,7 @@ final class MediaPlayerController: ObservableObject {
@Published private(set) var pendingJobs: [MediaPendingJob] = []
@Published private(set) var choices: [MediaChoice] = []
@Published private(set) var stateError: String?
@Published private(set) var playbackError: String?
let player = AVPlayer()
@@ -224,6 +225,7 @@ final class MediaPlayerController: ObservableObject {
pollGeneration += 1
room = nil
wasPlayingBeforeTalk = false
clearMedia()
}
private func startPolling() {
@@ -278,6 +280,7 @@ final class MediaPlayerController: ObservableObject {
desired = s.desired ?? "stopped"
let track = s.track
self.track = track
if let track, track.id != currentTrackID {
loadTrack(track)
} else if let track, let audioURL = track.audioURL, loadedAudioPath != audioURL {
@@ -349,7 +352,17 @@ final class MediaPlayerController: ObservableObject {
}
private func wireSource(_ path: String) {
guard let url = URL(string: path) else { return }
// A complete file is required for the server's cached AAC conversion.
// Keep showing download progress; the next ready-state poll wires it.
guard track?.kind != "youtube" || track?.status != "downloading" else { return }
guard var components = URLComponents(string: path) else { return }
if track?.kind == "youtube", components.path.hasPrefix("/media/api/stream/") {
var items = components.queryItems ?? []
items.removeAll { $0.name == "format" }
items.append(URLQueryItem(name: "format", value: "m4a"))
components.queryItems = items
}
guard let url = components.url else { return }
loadedAudioPath = path
let pinned = mediaLoader.rewrite(url)
// Remove the previous asset's loader mapping before wiring a new one so
@@ -439,14 +452,17 @@ final class MediaPlayerController: ObservableObject {
self, selector: #selector(playerItemDidEnd(_:)),
name: .AVPlayerItemDidPlayToEndTime, object: item
)
statusCancellable = item.publisher(for: \.status).sink { status in
statusCancellable = item.publisher(for: \.status).sink { [weak self, weak item] status in
Task { @MainActor in
guard let self, let item, self.player.currentItem === item else { return }
switch status {
case .readyToPlay:
logger.log("item readyToPlay")
self.playbackError = nil
case .failed:
self.playbackError = "This audio couldn’t play on your iPhone. Try another track or an MP3/AAC station."
logger.log("item failed: \(String(describing: item.error), privacy: .public)")
default:
break
default: break
}
}
}
}
@@ -456,15 +472,15 @@ final class MediaPlayerController: ObservableObject {
report("stopped")
publishMediaState("stopped")
Task { [weak self] in
guard let self, let room = self.room else { return }
try? await self.api.control(room: room, action: "stop")
guard let self else { return }
try? await self.api.control(room: self.room ?? "default", action: "stop")
}
}
// MARK: Reporting + publishing
private func report(_ state: String, trackID: String? = nil) {
guard let room else { return }
let room = room ?? "default"
let position = currentTime.isFinite ? currentTime : 0
var durationValue: Double?
if duration.isFinite, duration > 0 {
@@ -505,20 +521,28 @@ final class MediaPlayerController: ObservableObject {
func playVideo(videoID: String?, title: String?, channel: String?) async {
let targetRoom = room ?? "default"
let generation = pollGeneration
playbackError = nil
do {
try await api.play(room: targetRoom, videoID: videoID, title: title, channel: channel)
logger.log("play requested video_id=\(videoID ?? "nil", privacy: .public) room=\(targetRoom, privacy: .public)")
} catch {
guard generation == pollGeneration else { return }
playbackError = "Couldn’t play this song. Check your connection and try again."
logger.log("play failed: \(String(describing: error), privacy: .public)")
}
}
func playTrack(id: String) async {
let targetRoom = room ?? "default"
let generation = pollGeneration
playbackError = nil
do {
try await api.play(room: targetRoom, trackID: id)
logger.log("play requested track_id=\(id, privacy: .public) room=\(targetRoom, privacy: .public)")
} catch {
guard generation == pollGeneration else { return }
playbackError = "Couldn’t play this track. Check your connection and try again."
logger.log("play failed: \(String(describing: error), privacy: .public)")
}
}
@@ -534,13 +558,13 @@ final class MediaPlayerController: ObservableObject {
// The rate observer reports + publishes the actual transition;
// the control call updates server desired state.
player.play()
if let room { try? await api.control(room: room, action: "resume") }
await sendControl("resume")
} else {
lastReportDate = Date()
report("paused")
publishMediaState("paused")
player.pause()
if let room { try? await api.control(room: room, action: "pause") }
await sendControl("pause")
}
}
@@ -553,6 +577,7 @@ final class MediaPlayerController: ObservableObject {
}
func seek(to position: Double) {
guard position.isFinite, position >= 0 else { return }
player.seek(to: CMTime(seconds: position, preferredTimescale: 600))
currentTime = position
report(player.rate.isZero ? "paused" : "playing")
@@ -564,11 +589,12 @@ final class MediaPlayerController: ObservableObject {
let trackID = currentTrackID
report("stopped", trackID: trackID)
clearMedia()
if let room { try? await api.control(room: room, action: "stop") }
await sendControl("stop")
publishMediaState("stopped", trackID: trackID)
}
private func clearMedia() {
playbackError = nil
currentTrackID = nil
track = nil
loadedAudioPath = nil
@@ -582,6 +608,18 @@ final class MediaPlayerController: ObservableObject {
stateError = nil
}
private func sendControl(_ action: String) async {
let generation = pollGeneration
do {
try await api.control(room: room ?? "default", action: action)
guard generation == pollGeneration else { return }
playbackError = nil
} catch {
guard generation == pollGeneration else { return }
playbackError = "Couldn’t update playback on the server. Check your connection and try again."
}
}
// MARK: Volume
func applyVolume() {
+29 -11
View File
@@ -1,5 +1,6 @@
import AVFoundation
import Foundation
import UniformTypeIdentifiers
import os
private let logger = Logger(subsystem: "dev.shane.voice", category: "media")
@@ -114,21 +115,29 @@ final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate {
shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest
) -> Bool {
// All state access is on the main queue (the delegate queue).
guard let dataRequest = loadingRequest.dataRequest else { return false }
guard loadingRequest.dataRequest != nil else { return false }
guard let url = loaderURLs[ObjectIdentifier(resourceLoader)] else {
logger.log("no resolved URL for resource loader")
loadingRequest.finishLoading(with: URLError(.badURL))
return false
}
var upstream = URLRequest(url: url)
let offset = dataRequest.requestedOffset
let length = Int64(dataRequest.requestedLength)
if length > 0 {
upstream.setValue("bytes=\(offset)-\(offset + length - 1)", forHTTPHeaderField: "Range")
} else {
upstream.setValue("bytes=\(offset)-", forHTTPHeaderField: "Range")
fetchChunk(loadingRequest, url: url)
return true
}
/// Return bounded chunks promptly instead of buffering an entire song before
/// AVPlayer receives its first usable bytes. The same loading request stays
/// open until its requested range (or EOF) has been delivered.
private func fetchChunk(_ loadingRequest: AVAssetResourceLoadingRequest, url: URL) {
guard !loadingRequest.isCancelled, let dataRequest = loadingRequest.dataRequest else { return }
var upstream = URLRequest(url: url)
let offset = max(dataRequest.requestedOffset, dataRequest.currentOffset)
let length = Int64(dataRequest.requestedLength)
let requestedEnd = dataRequest.requestedOffset + length
let end = dataRequest.requestsAllDataToEndOfResource || length == 0
? offset + 262_143 : min(requestedEnd - 1, offset + 262_143)
upstream.setValue("bytes=\(offset)-\(end)", forHTTPHeaderField: "Range")
logger.log("fetching \(url.absoluteString, privacy: .public) range=\(upstream.value(forHTTPHeaderField: "Range") ?? "-", privacy: .public)")
// Key the task by the loading request's identity directly — no hash
@@ -140,6 +149,7 @@ final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate {
// Runs on .main (the session's delegateQueue), same domain as all
// other map mutations.
self.activeTasks[requestKey] = nil
guard !loadingRequest.isCancelled else { return }
if let error {
logger.log("fetch failed \(url.absoluteString, privacy: .public): \(error.localizedDescription, privacy: .public)")
@@ -158,7 +168,8 @@ final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate {
return
}
if let info = loadingRequest.contentInformationRequest {
info.contentType = http.mimeType ?? "application/octet-stream"
// AVFoundation requires a UTI, not the HTTP MIME string.
info.contentType = http.mimeType.flatMap { UTType(mimeType: $0)?.identifier }
// Content-length handling (Iteration 05 Item 7):
// - For 206: set full content length ONLY when Content-Range
// provides a known total. Never fall back to the range's
@@ -180,14 +191,21 @@ final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate {
}
info.isByteRangeAccessSupported = true
}
if let data {
if let data, !data.isEmpty {
dataRequest.respond(with: data)
let nextOffset = offset + Int64(data.count)
let total = Self.parseTotalLength(http)
let reachedEOF = total.map { nextOffset >= $0 } ?? (nextOffset <= end)
let needsMore = dataRequest.requestsAllDataToEndOfResource || nextOffset < requestedEnd
if http.statusCode == 206, !reachedEOF, needsMore {
self.fetchChunk(loadingRequest, url: url)
return
}
}
loadingRequest.finishLoading()
}
activeTasks[requestKey] = task
task.resume()
return true
}
func resourceLoader(
+89
View File
@@ -0,0 +1,89 @@
import Foundation
import Testing
@testable import Voice
private final class PlaybackProtocol: URLProtocol, @unchecked Sendable {
static let lock = NSLock()
nonisolated(unsafe) static var status = 200
nonisolated(unsafe) static var requests: [URLRequest] = []
override class func canInit(with request: URLRequest) -> Bool { true }
override class func canonicalRequest(for request: URLRequest) -> URLRequest { request }
override func startLoading() {
Self.lock.lock()
Self.requests.append(request)
let code = Self.status
Self.lock.unlock()
let response = HTTPURLResponse(url: request.url!, statusCode: code, httpVersion: nil, headerFields: ["Content-Type": "application/json"])!
client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed)
client?.urlProtocol(self, didLoad: Data("{\"ok\":true}".utf8))
client?.urlProtocolDidFinishLoading(self)
}
override func stopLoading() {}
}
@Suite(.serialized)
@MainActor struct MediaPlaybackTests {
private func controller(status: Int = 200) -> MediaPlayerController {
PlaybackProtocol.lock.withLock {
PlaybackProtocol.status = status
PlaybackProtocol.requests = []
}
let configuration = URLSessionConfiguration.ephemeral
configuration.protocolClasses = [PlaybackProtocol.self]
let url = URL(string: "https://media.test")!
let api = MediaAPI(config: ServerConfig(baseURL: url, livekitURL: url), session: URLSession(configuration: configuration))
return MediaPlayerController(api: api, settings: SettingsStore())
}
@Test("standalone playback sends stop to the default server room")
func standaloneStop() async throws {
let player = controller()
await player.stop()
let calls = PlaybackProtocol.lock.withLock { PlaybackProtocol.requests }
let stop = try #require(calls.first { $0.url?.path == "/media/api/control" })
// URLSession may expose the encoded body as an InputStream to URLProtocol.
let body: Data
if let data = stop.httpBody { body = data }
else {
let stream = try #require(stop.httpBodyStream)
stream.open()
defer { stream.close() }
var data = Data()
var buffer = [UInt8](repeating: 0, count: 1024)
while stream.hasBytesAvailable {
let count = stream.read(&buffer, maxLength: buffer.count)
if count <= 0 { break }
data.append(contentsOf: buffer.prefix(count))
}
body = data
}
let json = try #require(JSONSerialization.jsonObject(with: body) as? [String: Any])
#expect(json["room"] as? String == "default")
#expect(json["action"] as? String == "stop")
}
@Test("failed play requests are visible and a successful retry clears the error")
func failureAndRetry() async {
let player = controller(status: 503)
await player.playTrack(id: "song")
#expect(player.playbackError != nil)
PlaybackProtocol.lock.withLock { PlaybackProtocol.status = 200 }
await player.playTrack(id: "song")
#expect(player.playbackError == nil)
}
@Test("metadata refreshes when a download becomes ready")
func readyMetadata() {
let player = controller()
func state(_ status: String) -> MediaState {
MediaState(room: "default", track: MediaTrack(id: "song", kind: "youtube", videoID: nil, title: status, channel: nil, duration: 180, status: status, error: nil, audioURL: nil, artURL: nil), desired: "paused", reported: nil, seekHint: nil, choices: [], pending: [], version: 1)
}
player.reconcile(state("downloading"))
player.reconcile(state("ready"))
#expect(player.track?.status == "ready")
#expect(player.track?.title == "ready")
player.resetScope()
#expect(player.track == nil)
#expect(player.player.currentItem == nil)
}
}
+97
View File
@@ -0,0 +1,97 @@
import XCTest
/// Opt-in tests against the actual NAS deployment. No injected replies or media.
final class VoiceLiveTests: XCTestCase {
private var app: XCUIApplication!
private var startedMusic = false
override func record(_ issue: XCTIssue) {
if let app, app.state == .runningForeground {
capture("Failure state")
let hierarchy = XCTAttachment(string: app.debugDescription)
hierarchy.name = "Failure UI hierarchy"
hierarchy.lifetime = .keepAlways
add(hierarchy)
}
super.record(issue)
}
override func tearDownWithError() throws {
defer { app?.terminate() }
guard let app, app.state == .runningForeground else { return }
// Stop only playback initiated by this test, including assertion failures.
// Do not leave the shared default room playing between test runs.
if startedMusic, app.buttons["Stop playback"].exists {
app.buttons["Stop playback"].tap()
}
}
override func setUpWithError() throws {
try XCTSkipUnless(ProcessInfo.processInfo.environment["HOPE_RUN_LIVE_TESTS"] == "1", "Requires the live Hope server")
continueAfterFailure = false
app = XCUIApplication()
startedMusic = false
app.launchArguments = ["--ui-testing", "--reset-ui-testing", "--live-testing"]
app.launch()
}
func testLiveConversationAndBackgroundTask() {
app.buttons["Start chatting"].tap()
let composer = app.descendants(matching: .any).matching(identifier: "Message Hope").firstMatch
XCTAssertTrue(composer.waitForExistence(timeout: 60), "Agent must join the real LiveKit room")
composer.tap()
composer.typeText("What is six times seven? Reply in one short sentence.")
app.buttons["Send message"].tap()
let reply = app.staticTexts.matching(NSPredicate(format: "label CONTAINS[c] 'forty-two' OR label CONTAINS[c] 'forty two' OR label CONTAINS '42'")).firstMatch
XCTAssertTrue(reply.waitForExistence(timeout: 60), "Qwen must produce a real response")
capture("Live Qwen reply")
composer.tap()
composer.typeText("Compare espresso and filter coffee in a short background task. Use your existing knowledge; no web search needed.")
app.buttons["Send message"].tap()
app.tabBars.buttons["Activity"].tap()
let complete = app.staticTexts["Complete"].firstMatch
XCTAssertTrue(complete.waitForExistence(timeout: 90), "Background task must finish and deliver its result")
capture("Live background result")
app.tabBars.buttons["Talk"].tap()
app.buttons["End conversation"].tap()
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20))
// A second room must connect cleanly after the first was torn down.
app.buttons["Start chatting"].tap()
XCTAssertTrue(composer.waitForExistence(timeout: 60))
app.buttons["End conversation"].tap()
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20))
}
func testLiveLibraryPlayback() {
app.tabBars.buttons["Music"].tap()
let play = app.buttons.matching(NSPredicate(format: "label BEGINSWITH 'Play Sheryl Crow'")).firstMatch
XCTAssertTrue(play.waitForExistence(timeout: 20), "Existing library must load from NAS")
startedMusic = true
play.tap()
XCTAssertTrue(app.buttons["Pause"].waitForExistence(timeout: 30))
// Elapsed media time proves AVPlayer decoded and advanced, beyond a server desired-state label.
let position = app.sliders["Playback position"]
XCTAssertTrue(position.waitForExistence(timeout: 10))
let advanced = NSPredicate { _, _ in
let value = position.value as? String ?? "0%"
return value != "0%" && value != "0" && !value.isEmpty
}
expectation(for: advanced, evaluatedWith: position)
waitForExpectations(timeout: 35)
app.buttons["Pause"].tap()
XCTAssertTrue(app.buttons["Play"].waitForExistence(timeout: 10))
app.buttons["Seek forward 10 seconds"].tap()
capture("Live music playback")
app.buttons["Stop playback"].tap()
XCTAssertFalse(app.buttons["Pause"].exists)
startedMusic = false
}
private func capture(_ name: String) {
let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = name
attachment.lifetime = .keepAlways
add(attachment)
}
}
+113
View File
@@ -0,0 +1,113 @@
import XCTest
/// Real UI interactions with isolated preferences. These cover the client,
/// including offline recovery; they do not claim to verify the live backend.
final class VoiceUITests: XCTestCase {
private var app: XCUIApplication!
override func setUpWithError() throws {
continueAfterFailure = false
app = XCUIApplication()
app.launchArguments = ["--ui-testing", "--reset-ui-testing"]
}
func testWelcomeNavigationAndOfflineRecovery() {
app.launch()
XCTAssertTrue(app.buttons["Let’s talk"].waitForExistence(timeout: 10))
XCTAssertTrue(app.staticTexts["Your microphone turns on when you connect."].exists)
app.buttons["Type"].tap()
XCTAssertTrue(app.buttons["Start chatting"].exists)
XCTAssertTrue(app.staticTexts["Your microphone stays off."].exists)
app.buttons["Start chatting"].tap()
XCTAssertTrue(app.buttons["Try again"].waitForExistence(timeout: 20))
XCTAssertTrue(app.staticTexts["Couldn’t connect to Hope"].exists)
app.buttons["Check connection settings"].tap()
XCTAssertTrue(app.textFields["Server URL"].waitForExistence(timeout: 5))
capture("Offline recovery")
}
func testVoiceSelectionPersistsAndURLValidation() {
app.launch()
app.buttons["Settings"].tap()
app.buttons.matching(NSPredicate(format: "label CONTAINS 'Speaking voice'")).firstMatch.tap()
let voice = app.buttons.matching(NSPredicate(format: "label CONTAINS 'Jenny'")).firstMatch
XCTAssertTrue(voice.waitForExistence(timeout: 5))
voice.tap()
XCTAssertTrue(app.buttons.matching(NSPredicate(format: "label CONTAINS 'Speaking voice' AND label CONTAINS 'Jenny'")).firstMatch.exists)
let server = app.textFields["Server URL"]
server.tap()
let old = server.value as? String ?? ""
server.typeText(String(repeating: XCUIKeyboardKey.delete.rawValue, count: old.count) + "http://example.com")
app.buttons["Save server address"].tap()
XCTAssertTrue(app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'must use https'")).firstMatch.waitForExistence(timeout: 5))
app.terminate()
app.launchArguments = ["--ui-testing"]
app.launch()
app.buttons["Settings"].tap()
XCTAssertTrue(app.buttons.matching(NSPredicate(format: "label CONTAINS 'Speaking voice' AND label CONTAINS 'Jenny'")).firstMatch.waitForExistence(timeout: 5))
XCTAssertEqual(app.textFields["Server URL"].value as? String, "https://127.0.0.1:1")
}
func testMusicSearchAndStationValidation() {
app.launch()
app.tabBars.buttons["Music"].tap()
XCTAssertTrue(app.buttons["Add radio station"].waitForExistence(timeout: 5))
XCTAssertTrue(app.staticTexts["Music server unavailable"].waitForExistence(timeout: 10))
app.segmentedControls.buttons["Search"].tap()
let query = app.textFields["Song, artist, lyrics…"]
query.tap()
query.typeText("Piano")
app.buttons["music.search"].tap()
XCTAssertTrue(app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'connect to the server'")).firstMatch.waitForExistence(timeout: 15))
app.buttons["Add radio station"].tap()
let url = app.textFields["Audio stream URL"]
XCTAssertTrue(url.waitForExistence(timeout: 5))
url.tap()
url.typeText("not-a-url")
app.buttons["Add station"].tap()
XCTAssertTrue(app.staticTexts["Enter a complete http:// or https:// audio stream address."].exists)
app.buttons["Cancel"].tap()
XCTAssertTrue(app.buttons["Add radio station"].exists)
capture("Music search")
}
func testTranscriptClearConfirmationAndActivityDetails() {
app.launchArguments.append("--ui-fixtures")
app.launch()
XCTAssertTrue(app.staticTexts["Help me plan a quiet weekend."].waitForExistence(timeout: 10))
app.buttons["Conversation options"].tap()
app.buttons["Clear conversation"].tap()
app.buttons["Cancel"].tap()
XCTAssertTrue(app.staticTexts["Help me plan a quiet weekend."].exists)
app.buttons["Conversation options"].tap()
app.buttons["Clear conversation"].tap()
app.buttons["Clear conversation"].tap()
XCTAssertTrue(app.buttons["Let’s talk"].exists)
XCTAssertFalse(app.staticTexts["Help me plan a quiet weekend."].exists)
app.tabBars.buttons["Activity"].tap()
XCTAssertTrue(app.staticTexts["Needs attention"].waitForExistence(timeout: 5))
XCTAssertTrue(app.staticTexts["Search backend timed out after 30s"].exists)
app.buttons["1 progress updates"].tap()
XCTAssertTrue(app.staticTexts["Fetching benchmark articles…"].exists)
capture("Activity details")
}
func testLargeTypeNavigation() {
app.launchArguments += ["-UIPreferredContentSizeCategoryName", "UICTContentSizeCategoryAccessibilityXXXL"]
app.launch()
XCTAssertTrue(app.buttons["Let’s talk"].waitForExistence(timeout: 10))
XCTAssertTrue(app.buttons["Let’s talk"].isHittable)
app.tabBars.buttons["Activity"].tap()
XCTAssertTrue(app.staticTexts["A little help, in the background"].waitForExistence(timeout: 5))
app.buttons["Settings"].tap()
XCTAssertTrue(app.navigationBars["Settings"].exists)
capture("Accessibility text size")
}
private func capture(_ name: String) {
let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = name
attachment.lifetime = .keepAlways
add(attachment)
}
}
+109 -1
View File
@@ -13,7 +13,9 @@
302BEC55CA9A10E49C307977 /* ServerConfig.swift in Sources */ = {isa = PBXBuildFile; fileRef = F5683993A1DE677121FFA1F3 /* ServerConfig.swift */; };
36DF44D9A7896F18A064847D /* ConversationView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */; };
3E01A0AF44FD3324FA72C43C /* OrbView.swift in Sources */ = {isa = PBXBuildFile; fileRef = A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */; };
57E6476135CE30495E02D723 /* VoiceUITests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */; };
64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */ = {isa = PBXBuildFile; fileRef = A0C23305A6C55A83F8617D9D /* ConversationState.swift */; };
70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */; };
720C26EC163AE46713B02DD8 /* TLSPinning.swift in Sources */ = {isa = PBXBuildFile; fileRef = D9B40076934C28BA4B92D808 /* TLSPinning.swift */; };
72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */; };
73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */; };
@@ -24,6 +26,7 @@
A5868238CF4958D6D0C5908E /* TaskStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0B60FBBB9C47BB9ADACA7CA /* TaskStore.swift */; };
A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */; };
A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */; };
AF719B1668ADA929AA0237BF /* MediaPlaybackTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */; };
B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */; };
BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */; };
C00D1878C2AB3D8B6F9562B6 /* AgentMonitor.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */; };
@@ -46,6 +49,13 @@
remoteGlobalIDString = 57E3A292BDE525A22EB7ADE4;
remoteInfo = Voice_iOS;
};
ED7A5BB58354A8C9A5994257 /* PBXContainerItemProxy */ = {
isa = PBXContainerItemProxy;
containerPortal = 201BFA602F0A78910611E5F5 /* Project object */;
proxyType = 1;
remoteGlobalIDString = 57E3A292BDE525A22EB7ADE4;
remoteInfo = Voice_iOS;
};
/* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */
@@ -59,11 +69,14 @@
3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlayerController.swift; sourceTree = "<group>"; };
3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationView.swift; sourceTree = "<group>"; };
48F76881B4AE51DEE74511CC /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = "<group>"; };
5759EB570DB87BBC121D946D /* VoiceUITests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceLiveTests.swift; sourceTree = "<group>"; };
61C71C48230C683D199D6F4B /* ControlChannel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ControlChannel.swift; sourceTree = "<group>"; };
68A45228EF59186CFC219F2E /* TranscriptHistory.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHistory.swift; sourceTree = "<group>"; };
68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPanelView.swift; sourceTree = "<group>"; };
6D0D153557BB1C756ED04C61 /* SettingsStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsStore.swift; sourceTree = "<group>"; };
6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoader.swift; sourceTree = "<group>"; };
8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceUITests.swift; sourceTree = "<group>"; };
8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoaderTests.swift; sourceTree = "<group>"; };
9191AD60FF96E44F191864E3 /* MediaAPI.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPI.swift; sourceTree = "<group>"; };
9681E7410B684D2D38752A70 /* TokenClient.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TokenClient.swift; sourceTree = "<group>"; };
@@ -75,6 +88,7 @@
C22975C69240AACD3418091F /* VoiceTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHandler.swift; sourceTree = "<group>"; };
D9B40076934C28BA4B92D808 /* TLSPinning.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TLSPinning.swift; sourceTree = "<group>"; };
E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlaybackTests.swift; sourceTree = "<group>"; };
E85BACA267ADD462306C9920 /* TaskPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TaskPanelView.swift; sourceTree = "<group>"; };
EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceSession.swift; sourceTree = "<group>"; };
F5683993A1DE677121FFA1F3 /* ServerConfig.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerConfig.swift; sourceTree = "<group>"; };
@@ -124,6 +138,15 @@
path = LiveKit;
sourceTree = "<group>";
};
6735DFBC89D9A939C44D763A /* UITests */ = {
isa = PBXGroup;
children = (
5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */,
8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */,
);
path = UITests;
sourceTree = "<group>";
};
6B2AB34274D9B37E77178C6C /* Networking */ = {
isa = PBXGroup;
children = (
@@ -139,6 +162,7 @@
children = (
2A4C3CE2CA6FF828452803D1 /* Voice.app */,
C22975C69240AACD3418091F /* VoiceTests.xctest */,
5759EB570DB87BBC121D946D /* VoiceUITests.xctest */,
);
name = Products;
sourceTree = "<group>";
@@ -168,6 +192,7 @@
children = (
FCC61E9990B71498171F9D34 /* Sources */,
DC8495F2B27E1770E6620A05 /* Tests */,
6735DFBC89D9A939C44D763A /* UITests */,
6F99423E59017FED59B4D71D /* Products */,
);
sourceTree = "<group>";
@@ -197,6 +222,7 @@
children = (
BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */,
10C194656F19E89DFD656A91 /* MediaAPITests.swift */,
E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */,
09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */,
8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */,
20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */,
@@ -246,6 +272,24 @@
productReference = C22975C69240AACD3418091F /* VoiceTests.xctest */;
productType = "com.apple.product-type.bundle.unit-test";
};
1C8CC2859E99A4AC83FC6374 /* VoiceUITests */ = {
isa = PBXNativeTarget;
buildConfigurationList = 286F46F3E4A225D8ABDCEA48 /* Build configuration list for PBXNativeTarget "VoiceUITests" */;
buildPhases = (
889523F0856C91F5A5E8FD41 /* Sources */,
);
buildRules = (
);
dependencies = (
A506145EF5A1B3978391C33C /* PBXTargetDependency */,
);
name = VoiceUITests;
packageProductDependencies = (
);
productName = VoiceUITests;
productReference = 5759EB570DB87BBC121D946D /* VoiceUITests.xctest */;
productType = "com.apple.product-type.bundle.ui-testing";
};
57E3A292BDE525A22EB7ADE4 /* Voice_iOS */ = {
isa = PBXNativeTarget;
buildConfigurationList = 67CE95EFE28F976D1CF1A59A /* Build configuration list for PBXNativeTarget "Voice_iOS" */;
@@ -274,6 +318,9 @@
BuildIndependentTargetsInParallel = YES;
LastUpgradeCheck = 1430;
TargetAttributes = {
1C8CC2859E99A4AC83FC6374 = {
TestTargetID = 57E3A292BDE525A22EB7ADE4;
};
57E3A292BDE525A22EB7ADE4 = {
ProvisioningStyle = Automatic;
};
@@ -296,8 +343,9 @@
projectDirPath = "";
projectRoot = "";
targets = (
1385EEF29189F331F2634382 /* VoiceTests_iOS */,
57E3A292BDE525A22EB7ADE4 /* Voice_iOS */,
1385EEF29189F331F2634382 /* VoiceTests_iOS */,
1C8CC2859E99A4AC83FC6374 /* VoiceUITests */,
);
};
/* End PBXProject section */
@@ -332,12 +380,22 @@
);
runOnlyForDeploymentPostprocessing = 0;
};
889523F0856C91F5A5E8FD41 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */,
57E6476135CE30495E02D723 /* VoiceUITests.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
C735E25A3F84D701AEBFC3F1 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */,
F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */,
AF719B1668ADA929AA0237BF /* MediaPlaybackTests.swift in Sources */,
72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */,
B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */,
BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */,
@@ -353,6 +411,11 @@
target = 57E3A292BDE525A22EB7ADE4 /* Voice_iOS */;
targetProxy = 1C0045D170F971FC571D0E54 /* PBXContainerItemProxy */;
};
A506145EF5A1B3978391C33C /* PBXTargetDependency */ = {
isa = PBXTargetDependency;
target = 57E3A292BDE525A22EB7ADE4 /* Voice_iOS */;
targetProxy = ED7A5BB58354A8C9A5994257 /* PBXContainerItemProxy */;
};
/* End PBXTargetDependency section */
/* Begin XCBuildConfiguration section */
@@ -438,6 +501,24 @@
};
name = Debug;
};
452ED10E7C45C3EDEDC466B5 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
GENERATE_INFOPLIST_FILE = YES;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = dev.shane.voice.uitests;
SDKROOT = iphoneos;
SWIFT_VERSION = 5.9;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_TARGET_NAME = Voice_iOS;
};
name = Release;
};
8CD9F2FF6EF1EB5438545350 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
@@ -557,9 +638,36 @@
};
name = Release;
};
EEDD1CAF56A9063A94459A9B /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
GENERATE_INFOPLIST_FILE = YES;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
"@loader_path/Frameworks",
);
PRODUCT_BUNDLE_IDENTIFIER = dev.shane.voice.uitests;
SDKROOT = iphoneos;
SWIFT_VERSION = 5.9;
TARGETED_DEVICE_FAMILY = "1,2";
TEST_TARGET_NAME = Voice_iOS;
};
name = Debug;
};
/* End XCBuildConfiguration section */
/* Begin XCConfigurationList section */
286F46F3E4A225D8ABDCEA48 /* Build configuration list for PBXNativeTarget "VoiceUITests" */ = {
isa = XCConfigurationList;
buildConfigurations = (
EEDD1CAF56A9063A94459A9B /* Debug */,
452ED10E7C45C3EDEDC466B5 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Debug;
};
67CE95EFE28F976D1CF1A59A /* Build configuration list for PBXNativeTarget "Voice_iOS" */ = {
isa = XCConfigurationList;
buildConfigurations = (
+38
View File
@@ -0,0 +1,38 @@
# NAS deployment — 2026-09-08
The `voice` container on `192.168.86.2` hosts LiveKit, the voice agent,
token service, music API and nginx. The app connects to
`https://192.168.86.2:9178`. Docker uses host networking and an
`unless-stopped` restart policy.
- Image: `hope-voice-api-voice:68f4c46-ios-audio`.
- LLM: **qwen2.5-vl-3b**, through OpenWebUI at `http://192.168.86.2:3005/api`.
- Speech: existing Azure configuration, unchanged.
- Persistent certs, skills and music: `/mnt/user/dev/hope-voice-api` mounts.
- Build source: `/mnt/user/dev/hope-voice-deploy-68f4c46`, a separate clean
archive plus the iOS audio compatibility patch. The pre-existing dirty
backend checkout was not overwritten.
- Local compatibility source: sibling `hope-voice-api/web/media_server.py`;
regression test: `hope-voice-api/tests/test_ios_audio.py`.
The music service retains original WebM files. An explicit `?format=m4a`
request generates and caches AAC audio under `media/files/ios`, serving
byte ranges for iOS playback and seeking. Browser requests remain unchanged.
## Health and rollback
```sh
ssh [email protected] 'docker inspect --format "{{.State.Status}} {{.State.Health.Status}}" voice'
```
The stopped container `voice-before-ios-audio` retains the baseline deployment.
If rollback is needed, first inspect both containers, then stop the current
`voice`, rename it to a unique recovery name, rename `voice-before-ios-audio`
to `voice`, and start it. This restores the baseline without deleting images,
containers or the music library; it does not include iOS AAC conversion.
`scripts/deploy-backend.sh` records the launch configuration and refuses to
replace an existing `voice` container. Run it on the NAS only after building
the named image and explicitly handling the existing container. It reads
credentials on the NAS and never prints them. No source changes were pushed;
future CI deployments must include the compatibility patch to preserve it.
+2 -2
View File
@@ -33,8 +33,8 @@ ensure the server cert pin is current (see README — re-export after any server
## 4. Audible media playback
- [ ] Requires a track in an iOS-decodable format (m4a/aac/mp3) — current library tracks
are webm and will fail with `-11828` until the server serves a supported format.
- [ ] The deployed server now serves cached AAC for library tracks requested with
`?format=m4a`; verify actual speaker/headphone output on hardware.
- [ ] Play a library track: audio is audible, now-playing card shows art/title/status.
- [ ] Pause / ±10 s / stop / seek all work; agent mics off while playing
(`media_state` published), mics back on when stopped.
+139
View File
@@ -0,0 +1,139 @@
# Project status and handoff
Last updated: 2026-09-09.
## Current direction
Keep **Qwen 2.5**. The user explicitly canceled the earlier request to switch to
a ChatGPT subscription/device-token login. Do not revive that migration because
an old saved goal mentions it. Current work is the native iOS UI/UX redesign,
restoring the music/AI server, and end-to-end qualification.
The UI and playback changes are implemented, and the NAS deployment was verified
on September 8. Full app qualification is **not complete**: physical-device tests
and some broader live scenarios remain. This report summarizes prior evidence;
server health and device availability must be rechecked when resuming.
## Completed work
### Native interface and usability
- Reorganized the app into native **Talk, Music, Activity** tabs, sharing one session.
- Replaced the large animated orb with a compact waveform/status presentation;
added a clearer welcome screen, Voice/Type selection before connecting, explicit
start/end controls, microphone meter, and recoverable connection errors.
- Improved transcript readability, timestamps, live updates, copy/selection,
follow-latest scrolling, and explicit Cancel/Clear confirmation.
- Reworked Settings around voice, input mode, model, and connection; server changes
require an explicit validated save. Voice selection persists.
- Improved Music library/search navigation, now-playing controls, seek/volume,
empty/loading/error states, and validated station-entry sheet.
- Improved Activity hierarchy and task detail presentation with an active-task badge.
- Removed forced dark appearance; adopted adaptive system styling, Dynamic Type,
and larger interaction targets, informed by the Impeccable native-iOS guidance.
### Functional fixes
- Observed nested transcript/state/task objects so asynchronous updates render.
- Preserved latest-wins server replacement; clear old media when switching servers.
- Sent standalone media controls and reports to the default server room.
- Refreshed track metadata as downloads become ready; exposed request/decoder errors.
- Loaded artwork through pinned TLS for the NAS self-signed certificate.
- Requested cached AAC (`?format=m4a`) for YouTube library tracks on iOS.
- Corrected AVFoundation's content type to a UTI rather than an HTTP MIME string.
- Changed the pinned media loader to deliver 256 KB byte ranges instead of buffering
an entire song before delivering usable audio, addressing a reproduced timeout.
### Tests and tooling
- Installed XcodeBuildMCP and its skills; used it for simulator build/run/test work.
- Added the `VoiceUITests` target in `project.yml` and regenerated the Xcode project.
- Added isolated DEBUG UI-test preferences and optional fixtures; normal application
preferences are not reset by those UI-test launch arguments.
- Added media-control/error/metadata tests and opt-in real-NAS UI tests.
- Live tests retain failure screenshots/UI hierarchy and attempt to stop music
they initiated even when an assertion fails.
## Backend deployment and cross-repository dependency
SSH host: `[email protected]` (Unraid). App URL: `https://192.168.86.2:9178`.
- `voice` image: `hope-voice-api-voice:68f4c46-ios-audio`, host networking,
restart policy `unless-stopped`. Includes LiveKit, voice agent, token service,
media API, and nginx.
- LLM: `qwen2.5-vl-3b` through existing OpenWebUI at
`http://192.168.86.2:3005/api`. Existing Azure speech configuration retained.
- `voice`, `open-webui`, and `local-vl-models` were present and healthy when the
user suspected deletion. Do not recreate healthy services unnecessarily.
- Original library, certificates, and skills were retained under
`/mnt/user/dev/hope-voice-api`; the dirty remote checkout was not overwritten.
- Separate deployment source: `/mnt/user/dev/hope-voice-deploy-68f4c46`.
- Stopped rollback container: `voice-before-ios-audio`.
- Backend AAC conversion preserves original WebM files and serves cached AAC with
byte-range support. Browser/raw requests retain their original behavior.
**Important:** the backend patch and its test are in a different local repository:
`/Users/shane/dev/hope-voice-api/web/media_server.py` and
`/Users/shane/dev/hope-voice-api/tests/test_ios_audio.py`. They were still uncommitted
at this handoff. Committing/pushing `voice-ios` does **not** publish those files.
The deployed image contains the patch, but a future backend CI rebuild can lose it
unless it is committed and deployed through that repository's workflow. Inspect
that workflow before pushing: it may replace the live container.
Launch configuration and rollback details: [DEPLOYMENT.md](DEPLOYMENT.md).
Do not copy credentials or `.env` contents into documentation or commits.
## Verification performed
Project `Voice.xcodeproj`, scheme `Voice_iOS`, iPhone 17 Pro simulator.
- 68 unit tests passed; five offline UI tests passed (navigation/error recovery,
settings persistence/validation, music validation, transcript/task UI, large text).
- Real NAS tests verified Qwen replies, background task completion, reconnect,
library loading, advancing playback time, pause, forward-seek control and stop.
- Qwen/task/reconnect passed three consecutive iterations.
- Music initially failed intermittently. Logs captured an AVPlayer timeout while
buffering a whole 6 MB response. After bounded-range delivery, three consecutive
live playback iterations passed. The earlier isolated agent-join failure remains
unexplained; do not claim all intermittent connection issues are resolved.
- Backend integration: 16 passed, 7 skipped, 2 deselected.
- Backend AAC regression: one passed (conversion, cache reuse, original preservation,
and byte ranges); deployed range request returned HTTP 206.
- The final bounded-range change was verified by focused playback repetitions,
not a new full combined regression run.
Exact result-bundle names and qualifications: [VERIFICATION.md](VERIFICATION.md).
Artifacts live on this Mac under the XcodeBuildMCP workspace, not in Git.
## What remains, in order
1. **Pair the physical iPhone.** Latest check detected it on USB and reported iOS
26.6.1, device ID `BF93D35E-0853-5470-A125-6EB0DB3F17AB`, state `unpaired`.
The user had unlocked it but could not see it in Finder. Last guidance was to
enable iOS devices in Finder's Sidebar settings and accept any accessory/Trust
prompts. Recheck current state rather than repeating old cable assumptions.
2. **Build, sign, install and run on that iPhone.** No successful device install is
recorded. Check Developer Mode, trust, signing team/profile, and LAN access.
3. **Qualify real audio:** microphone capture, TTS audibility, speaker/headphones,
barge-in, media/voice handoff, Siri/phone interruptions, lock/background behavior,
microphone-permission-denied fallback, and network-loss recovery. Some checks
require the user's speech/listening or a real interruption; simulator state tests
do not prove them. Follow [DEVICE_TESTING.md](DEVICE_TESTING.md).
4. **Broader regression:** rerun the full suite after the range-loader change; test
longer playback, seek near EOF, slow network/cancellation and server replacement.
Exercise new downloads, test-owned station add/play/remove, and voice/model
selection against the live agent. Do not delete the user's existing library song.
5. **Final visual/accessibility pass:** inspect all populated/error screens in light
and dark mode, large text and small-device layouts; test VoiceOver and focus.
The existing large-text navigation test is not a full accessibility audit.
6. **Publish the backend compatibility source safely** in the sibling repository
and ensure its normal deployment preserves mounts, certificates and Qwen settings.
7. Update this report with results and remaining limitations. Do not mark the whole
app qualified until the outstanding end-to-end gates have evidence.
## Repository handoff
The iOS work and this report belong to branch `feat/ios-app`, remote `origin`
(`https://git.srmr.xyz/snowspeeder/voice-ios.git`). The requested handoff commit
includes the implementation, tests, deployment notes and startup instructions.
Check Git history/remote status for the final commit and push result.
+77
View File
@@ -0,0 +1,77 @@
# Verification — 2026-09-08
## Scope and implementation
The requested model migration was canceled: Qwen 2.5 remains configured.
The native redesign separates Talk, Music and Activity; simplifies the welcome
and connection controls; supports light/dark appearance and Dynamic Type;
adds recoverable error states; and gives transcript clearing explicit confirmation.
The Impeccable native-iOS guidance informed system controls, hierarchy, readable
spacing and touch targets. Settings, transcripts and task updates are observed
directly so asynchronous changes render promptly.
Music fixes include standalone control requests, current download metadata,
visible failures, pinned artwork, server-change cleanup, cached AAC delivery,
and correct AVFoundation content-type identifiers in the pinned resource loader.
## Evidence
Project: `Voice.xcodeproj`; scheme: `Voice_iOS`; simulator: iPhone 17 Pro.
- 68 unit tests passed, including standalone media controls, retry error clearing,
download metadata, transcript persistence/deduplication, session replacement,
and interruption state handling.
- Five offline UI tests passed: navigation/offline recovery, voice persistence and
URL validation, music search failure/station validation, transcript confirmation
and task details, and accessibility-size navigation.
- Two live UI tests passed together against the NAS at 17:56 UTC: real Qwen
response, background task completion, disconnect/reconnect, library playback
with advancing media time, pause, forward seek and stop.
- Backend integration run: 16 passed, 7 skipped, 2 deselected. This is not a claim
that the skipped or slow integration cases were verified.
- AAC conversion regression: one passed, checking original-file preservation,
AAC output, cache reuse and HTTP byte ranges. Deployed range request returned 206.
XcodeBuildMCP result bundles are under
`~/Library/Developer/XcodeBuildMCP/workspaces/voice-ios-9eb036fdea81/result-bundles/`:
- Unit/offline UI: `test_sim_2026-09-08T11-48-52-164Z_pid87175_79b1a610.xcresult`.
This combined run's two live cases failed after long stalls; its unit/offline
results above passed.
- Latest live pair: `test_sim_2026-09-08T17-56-51-984Z_pid30949_ec504db2.xcresult`.
- Earlier live rerun: `test_sim_2026-09-08T17-46-07-841Z_pid30949_bd804a04.xcresult`
failed agent join and advancing playback. Their intermittent cause has not
been established. The subsequent passing pair did not require a server restart.
## Not yet qualified
Passing simulator tests do not prove audible speaker/headphone output, microphone
capture, barge-in, phone/Siri interruption recovery, background audio, or network
loss recovery on hardware. Follow `DEVICE_TESTING.md` with a physical iPhone on
the LAN. Navigation at large text sizes is tested, not a complete VoiceOver audit.
Live tests use an existing library track; they do not prove new YouTube downloads
or every radio stream format.
## Repeatability follow-up — 18:04 UTC
Three consecutive Qwen/task/reconnect iterations passed. Music passed twice and
failed on the third iteration; runtime logs captured AVPlayer's request timeout
while the TLS loader buffered a whole 6 MB response. The loader now delivers
bounded 256 KB ranges to AVPlayer, retaining cancellation and EOF handling.
After that change, **all three consecutive live playback iterations passed**
in about 13 seconds each. This addresses the observed music buffering failure;
the earlier isolated agent-join failure has not been reproduced or explained.
- Reproduction bundle: `test_sim_2026-09-08T18-00-03-993Z_pid30949_49da9d2c.xcresult`.
- Post-fix playback bundle: `test_sim_2026-09-08T18-04-30-014Z_pid30949_df323119.xcresult`.
- Live tests now attach failure screenshots and UI hierarchy, and attempt to stop
test-initiated music during teardown even after an assertion failure.
The initial hardware check found an unavailable iPhone. A later USB check detected
the phone running iOS 26.6.1, explicitly marked **unpaired** by XcodeBuildMCP.
No successful device install or hardware qualification is recorded. See
`PROJECT_STATUS.md` for the current pairing blocker and next steps.
Deployment and rollback are recorded in `DEPLOYMENT.md`. The NAS image includes
the compatibility patch; the sibling backend source still requires its own commit
and safe deployment workflow. See `PROJECT_STATUS.md` for repository handoff scope.
+13
View File
@@ -54,3 +54,16 @@ targets:
SWIFT_VERSION: "5.9"
GENERATE_INFOPLIST_FILE: YES
PRODUCT_BUNDLE_IDENTIFIER: dev.shane.voice.tests
VoiceUITests:
type: bundle.ui-testing
platform: iOS
sources:
- UITests
dependencies:
- target: Voice_iOS
settings:
base:
SWIFT_VERSION: "5.9"
GENERATE_INFOPLIST_FILE: YES
PRODUCT_BUNDLE_IDENTIFIER: dev.shane.voice.uitests
TEST_TARGET_NAME: Voice_iOS
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Run on xNAS after building the clean backend source into the pinned image.
# Credentials stay on the NAS. Existing checkout, library and certs are retained.
set -euo pipefail
deployment_source=/mnt/user/dev/hope-voice-deploy-68f4c46
deployment_data=/mnt/user/dev/hope-voice-api
deployment_image=${HOPE_DEPLOY_IMAGE:-hope-voice-api-voice:68f4c46-ios-audio}
test -f "$deployment_data/.env"
docker image inspect "$deployment_image" >/dev/null
if docker container inspect voice >/dev/null 2>&1; then
echo 'A voice container already exists; inspect it before replacing it.' >&2
exit 1
fi
# Recover the OpenWebUI credential used by this exact backend deployment source.
LLM_API_KEY=$(sed -n 's/.*-e LLM_API_KEY=\([^ ]*\).*/\1/p' "$deployment_source/.gitea/workflows/rebuild.yml")
test -n "$LLM_API_KEY"
export LLM_API_KEY
export LLM_BASE_URL=http://192.168.86.2:3005/api
export LLM_MODEL=qwen2.5-vl-3b
docker run -d --name voice --network host --restart unless-stopped \
--env-file "$deployment_data/.env" \
--env LLM_BASE_URL --env LLM_MODEL --env LLM_API_KEY \
--env LIVEKIT_URL=http://localhost:7880 \
--env VOICE_USERS_FILE=/etc/hope/users.json \
--env VOICE_DEFAULT_USER=shane --env MEDIA_DIR=/data/media \
--env MEDIA_SERVER_PORT=8093 --env WEB_PORT=9178 \
--env YTDLP_COOKIES=/data/media/cookies.txt \
--volume "$deployment_data/certs:/etc/voice/certs" \
--volume "$deployment_data/skills:/skills" \
--volume "$deployment_data/media:/data/media" \
--volume "$deployment_source/livekit.yaml:/etc/livekit.yaml:ro" \
--health-cmd='curl -sfk https://localhost:9178/ -o /dev/null && curl -sf http://localhost:7880/ -o /dev/null && curl -sf http://localhost:8093/media/api/healthz -o /dev/null' \
--health-interval=15s --health-timeout=5s --health-retries=3 --health-start-period=30s \
"$deployment_image"