Remove embedded music player

The music feature is retired: delete the Media tab, MediaAPI,
MediaPlayerController, PinnedMediaResourceLoader, MediaPanelView, the
media_state control publish, media scope handling in VoiceSession, the
volume setting, and all media tests. App is now Talk + Activity tabs.

Backend media API in hope-voice-api is untouched (still serves the web UI).
This commit is contained in:
snowspeeder committed 2026-09-12 06:26:58 -04:00
1 parent 3620a2387b
commit ed329213de
18 files changed
+55 -2393

No files matched your search

+8 -14
View File
@@ -1,19 +1,19 @@
# Voice — iOS client for Hope Voice # Voice — iOS client for Hope Voice
Native SwiftUI iOS app (iOS 17+) that talks to the [Hope Voice](https://github.com/) agent: Native SwiftUI iOS app (iOS 17+) that talks to the [Hope Voice](https://github.com/) agent:
voice/text conversation, transcript history, voice & model pickers, media player, and a voice/text conversation, transcript history, voice & model pickers, and a background-tasks
background-tasks panel. Uses `hope-voice-api`, including cached AAC conversion for iOS music playback. panel. Uses `hope-voice-api`.
> **Status:** Core functionality is implemented and verified in simulator to the extent the > **Status:** Core functionality is implemented and verified in simulator to the extent the
> environment allows. Full device qualification (mic capture, audible media playback, > environment allows. Full device qualification (mic capture, interruption behavior)
> interruption behavior) remains pending. The native Talk/Music/Activity redesign is implemented — > remains pending. The native Talk/Activity redesign is implemented —
> see [PLAN.md](PLAN.md) and [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md). > see [PLAN.md](PLAN.md) and [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md).
## Prerequisites ## Prerequisites
- Xcode 16+ (Swift 6), iOS 17 simulator or a real iPhone on the LAN. - Xcode 16+ (Swift 6), iOS 17 simulator or a real iPhone on the LAN.
- [xcodegen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`). - [xcodegen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`).
- The `voice` container running on Unraid (LiveKit + token server + media API). - The `voice` container running on Unraid (LiveKit + token server).
- Mic: the simulator mic is unreliable in this environment (CoreAudio `-4010`, 0 input - Mic: the simulator mic is unreliable in this environment (CoreAudio `-4010`, 0 input
channels) — voice mode needs a real device. If the simulator was wiped, grant permission channels) — voice mode needs a real device. If the simulator was wiped, grant permission
first: `xcrun simctl privacy booted grant microphone dev.shane.voice`. first: `xcrun simctl privacy booted grant microphone dev.shane.voice`.
@@ -33,7 +33,7 @@ To run in the simulator, use Xcode or `xcrun simctl` with the built app.
Default base URL is `https://192.168.86.2:9178` (see `Sources/App/SettingsStore.swift`). Default base URL is `https://192.168.86.2:9178` (see `Sources/App/SettingsStore.swift`).
Change it at runtime in **Settings → Server URL** (the app disconnects and recreates the Change it at runtime in **Settings → Server URL** (the app disconnects and recreates the
session when idle). The app derives LiveKit signaling and media endpoints from this base URL. session when idle). The app derives the LiveKit signaling endpoint from this base URL.
### TLS cert pinning ### TLS cert pinning
@@ -46,7 +46,7 @@ openssl s_client -connect 192.168.86.2:9178 </dev/null 2>/dev/null \
| openssl x509 -outform der | openssl dgst -sha256 | openssl x509 -outform der | openssl dgst -sha256
``` ```
Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token/media calls Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token calls
fail with a pin mismatch (not a network error). fail with a pin mismatch (not a network error).
## Features ## Features
@@ -58,10 +58,6 @@ fail with a pin mismatch (not a network error).
stripping, last-50 history persisted to UserDefaults, clear button. stripping, last-50 history persisted to UserDefaults, clear button.
- **Voice & model pickers** — 13 Azure TTS voices; LLM models from the agent's `models` - **Voice & model pickers** — 13 Azure TTS voices; LLM models from the agent's `models`
message. Both persist and sync on join (`set_voice`/`set_model`). message. Both persist and sync on join (`set_voice`/`set_model`).
- **Media player** — now-playing card (art/title/channel/status), transport (play/pause,
±10 s, stop), seek + volume sliders, search with "in library" tags, agent-offered choices,
library list (play/remove), add-stream form. AVPlayer-backed with a pinned-TLS Range proxy;
1.5 s `/state` polling and full web-parity reconciliation/reporting.
- **Activity tab** — live `task_event` updates (steps, result, error) with an active - **Activity tab** — live `task_event` updates (steps, result, error) with an active
count badge on the tab. count badge on the tab.
@@ -69,11 +65,9 @@ fail with a pin mismatch (not a network error).
- **Simulator mic**: CoreAudio returns `-4010` (no usable input source) in this environment; - **Simulator mic**: CoreAudio returns `-4010` (no usable input source) in this environment;
voice capture must be verified on a real device. Text mode works in the simulator. voice capture must be verified on a real device. Text mode works in the simulator.
- **Music compatibility**: library WebM files are requested as cached AAC on iOS.
This requires the deployed backend compatibility patch; see [deployment](docs/DEPLOYMENT.md).
- **Cert pin rotation**: see above — re-export the pin after each server cert regeneration. - **Cert pin rotation**: see above — re-export the pin after each server cert regeneration.
- **LiveKit signaling** uses plain `ws://192.168.86.2:7880` (not wss) because the Swift SDK - **LiveKit signaling** uses plain `ws://192.168.86.2:7880` (not wss) because the Swift SDK
cannot trust the self-signed cert on its WebSocket; token + media still go over pinned TLS. cannot trust the self-signed cert on its WebSocket; token calls still go over pinned TLS.
## Docs ## Docs
-5
View File
@@ -14,16 +14,12 @@ final class SettingsStore: ObservableObject {
@Published var textMode: Bool { @Published var textMode: Bool {
didSet { AppPreferences.store.set(textMode, forKey: Keys.textMode) } didSet { AppPreferences.store.set(textMode, forKey: Keys.textMode) }
} }
@Published var volume: Double {
didSet { AppPreferences.store.set(volume, forKey: Keys.volume) }
}
private enum Keys { private enum Keys {
static let serverURL = "serverURL" static let serverURL = "serverURL"
static let voiceID = "voiceID" static let voiceID = "voiceID"
static let modelID = "modelID" static let modelID = "modelID"
static let textMode = "textMode" static let textMode = "textMode"
static let volume = "volume"
} }
static let defaultServerURL = "https://192.168.86.2:9178" static let defaultServerURL = "https://192.168.86.2:9178"
@@ -47,7 +43,6 @@ final class SettingsStore: ObservableObject {
voiceID = Voices.all.contains { $0.id == storedVoice } ? (storedVoice ?? "") : Voices.defaultID voiceID = Voices.all.contains { $0.id == storedVoice } ? (storedVoice ?? "") : Voices.defaultID
modelID = d.string(forKey: Keys.modelID) modelID = d.string(forKey: Keys.modelID)
textMode = d.bool(forKey: Keys.textMode) textMode = d.bool(forKey: Keys.textMode)
volume = d.object(forKey: Keys.volume) as? Double ?? 0.8
} }
var trimmedServerURL: String { var trimmedServerURL: String {
@@ -40,15 +40,8 @@ struct ConversationView: View {
.tabItem { Label("Talk", systemImage: "waveform") } .tabItem { Label("Talk", systemImage: "waveform") }
.tag(0) .tag(0)
NavigationStack {
MediaPanelView(playerController: session.mediaController, settings: settings)
.toolbar { settingsLink }
}
.tabItem { Label("Music", systemImage: "music.note") }
.tag(1)
ActivityTab(taskStore: session.taskStore, settings: settings, session: session) ActivityTab(taskStore: session.taskStore, settings: settings, session: session)
.tag(2) .tag(1)
} }
.id(sessionID) .id(sessionID)
.tint(HopeStyle.tint) .tint(HopeStyle.tint)
@@ -79,9 +72,7 @@ struct ConversationView: View {
// Coalesce rapid Apply actions; construct from the captured URL. // Coalesce rapid Apply actions; construct from the captured URL.
while let next = pendingServerURL { while let next = pendingServerURL {
pendingServerURL = nil pendingServerURL = nil
guard next != activeServerURL else { continue }
await session.disconnect() await session.disconnect()
session.mediaController.resetScope()
activeServerURL = next activeServerURL = next
session = Self.makeSession(next, settings: settings) session = Self.makeSession(next, settings: settings)
sessionID = UUID() sessionID = UUID()
@@ -157,13 +148,12 @@ private struct ConversationContent: View {
Text("A little help.\nA good conversation.") Text("A little help.\nA good conversation.")
.font(.largeTitle.weight(.semibold)) .font(.largeTitle.weight(.semibold))
.fixedSize(horizontal: false, vertical: true) .fixedSize(horizontal: false, vertical: true)
Text("Talk things through, find something out, or put on a favorite song.") Text("Talk things through, find something out, or get something done.")
.font(.body).foregroundStyle(.secondary) .font(.body).foregroundStyle(.secondary)
} }
VStack(alignment: .leading, spacing: 16) { VStack(alignment: .leading, spacing: 16) {
Label("Think out loud", systemImage: "bubble.left.and.bubble.right") Label("Think out loud", systemImage: "bubble.left.and.bubble.right")
Label("Find an answer", systemImage: "sparkle.magnifyingglass") Label("Find an answer", systemImage: "sparkle.magnifyingglass")
Label("Make time for music", systemImage: "music.note")
} }
.font(.subheadline).foregroundStyle(.secondary) .font(.subheadline).foregroundStyle(.secondary)
} }
-610
View File
@@ -1,610 +0,0 @@
import SwiftUI
struct MediaPanelView: View {
@ObservedObject var playerController: MediaPlayerController
@ObservedObject var settings: SettingsStore
@State private var searchQuery = ""
@State private var searchResults: [MediaSearchResult]?
@State private var isSearching = false
@State private var searchError: String?
@State private var libraryTracks: [MediaTrack]?
@State private var libraryError: String?
@State private var streamName = ""
@State private var streamURL = ""
@State private var streamMessage: String?
@State private var trackToDelete: MediaTrack?
@State private var browseMode = 0
@State private var showAddStream = false
@State private var isAddingStream = false
@State private var isLoadingLibrary = false
@State private var seekPosition: Double = 0
@State private var isSeeking = false
var body: some View {
ScrollView {
LazyVStack(alignment: .leading, spacing: 24) {
if playerController.stateError != nil {
VStack(alignment: .leading, spacing: 8) {
Label("Music server unavailable", systemImage: "wifi.exclamationmark")
.font(.headline)
Text("Check that your Hope server is running and you’re connected to its network. We’ll reconnect automatically.")
.font(.subheadline).foregroundStyle(.secondary)
}
.frame(maxWidth: .infinity, alignment: .leading)
}
if playerController.track != nil {
VStack(spacing: 20) {
nowPlayingCard
seekSection
transportControls
volumeSection
}
.padding(16)
.background(Color(.secondarySystemGroupedBackground), in: RoundedRectangle(cornerRadius: 16))
}
if let error = playerController.playbackError {
Label(error, systemImage: "exclamationmark.circle")
.font(.subheadline).foregroundStyle(.red)
}
pendingSection
choicesSection
Picker("Browse music", selection: $browseMode) {
Text("Library").tag(0)
Text("Search").tag(1)
}.pickerStyle(.segmented)
if browseMode == 0 { librarySection }
else { searchSection }
}
.padding()
}
.background(Color(.systemGroupedBackground))
.navigationTitle("Music")
.toolbar {
ToolbarItem(placement: .topBarTrailing) {
Button {
streamMessage = nil
showAddStream = true
} label: {
Image(systemName: "plus").frame(minWidth: 44, minHeight: 44)
}
.accessibilityLabel("Add radio station")
}
}
.sheet(isPresented: $showAddStream) {
NavigationStack {
Form { addStreamSection }
.navigationTitle("Add a station")
.navigationBarTitleDisplayMode(.inline)
.toolbar {
ToolbarItem(placement: .cancellationAction) {
Button("Cancel") { showAddStream = false }.disabled(isAddingStream)
}
}
}
.interactiveDismissDisabled(isAddingStream)
}
.onAppear {
playerController.startDefaultPolling()
refreshLibrary()
}
.alert("Remove from library?", isPresented: Binding(
get: { trackToDelete != nil },
set: { if !$0 { trackToDelete = nil } }
)) {
Button("Remove", role: .destructive) {
guard let track = trackToDelete else { return }
Task { await deleteTrack(track) }
}
Button("Cancel", role: .cancel) { trackToDelete = nil }
} message: {
Text(trackToDelete?.title ?? "")
}
}
private var nowPlayingCard: some View {
HStack(spacing: 12) {
if let artURL = playerController.track?.artURL, let url = URL(string: absolute(artURL)) {
PinnedArtwork(url: url)
.frame(width: 72, height: 72)
.clipShape(RoundedRectangle(cornerRadius: 10))
} else {
RoundedRectangle(cornerRadius: 10)
.fill(Color(.systemGray6))
.frame(width: 72, height: 72)
.overlay(
Image(systemName: "music.note")
.font(.title2)
.foregroundStyle(.secondary)
)
}
VStack(alignment: .leading, spacing: 4) {
Text(title)
.font(.headline)
.lineLimit(2)
Text(subtitle)
.font(.subheadline)
.foregroundStyle(.secondary)
.lineLimit(2)
if playerController.track != nil {
Text(statusText)
.font(.caption)
.foregroundStyle(playerController.desired == "playing" ? Color.green : Color.secondary)
}
}
Spacer()
}
}
private var transportControls: some View {
HStack(spacing: 24) {
Button(action: seekBackward) {
Image(systemName: "gobackward.10")
.font(.title3)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Seek back 10 seconds")
.disabled(playerController.duration <= 0 || playerController.track?.kind == "stream")
Button(action: togglePlayPause) {
Image(systemName: playerController.desired == "playing" && playerController.track != nil ? "pause.circle.fill" : "play.circle.fill")
.font(.largeTitle)
.frame(width: 56, height: 56)
.foregroundStyle(playerController.track == nil ? Color.secondary : Color.accentColor)
}
.accessibilityLabel(playerController.desired == "playing" && playerController.track != nil ? "Pause" : "Play")
.disabled(playerController.track == nil)
Button(action: seekForward) {
Image(systemName: "goforward.10")
.font(.title3)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Seek forward 10 seconds")
.disabled(playerController.duration <= 0 || playerController.track?.kind == "stream")
Button(role: .destructive, action: stopPlayback) {
Image(systemName: "stop.circle")
.font(.title2)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Stop playback")
.disabled(playerController.track == nil)
}
}
private var seekSection: some View {
VStack(spacing: 4) {
if playerController.track?.kind == "stream" {
Label("Live radio", systemImage: "dot.radiowaves.left.and.right")
.font(.subheadline).foregroundStyle(.secondary)
} else if playerController.track != nil {
Slider(
value: Binding(
get: { min(isSeeking ? seekPosition : playerController.currentTime, max(playerController.duration, 1)) },
set: { seekPosition = $0 }
),
in: 0...max(playerController.duration, 1),
onEditingChanged: { editing in
isSeeking = editing
if !editing { playerController.seek(to: seekPosition) }
}
)
.disabled(playerController.duration <= 0)
.accessibilityLabel("Playback position")
HStack {
Text(fmtTime(playerController.currentTime))
.font(.caption.monospacedDigit())
.foregroundStyle(.secondary)
Spacer()
Text(fmtTime(playerController.duration))
.font(.caption.monospacedDigit())
.foregroundStyle(.secondary)
}
} else {
Text("Nothing loaded")
.font(.subheadline)
.foregroundStyle(.secondary)
.frame(maxWidth: .infinity, alignment: .center)
.padding(.vertical, 8)
}
}
}
private var volumeSection: some View {
HStack(spacing: 12) {
Image(systemName: "speaker.fill")
.foregroundStyle(.secondary)
Slider(value: $settings.volume, in: 0...1)
.accessibilityLabel("Music volume")
.onChange(of: settings.volume) { _, newValue in
playerController.setVolume(newValue)
}
Image(systemName: "speaker.wave.3.fill")
.foregroundStyle(.secondary)
}
}
@ViewBuilder
private var pendingSection: some View {
if !playerController.pendingJobs.isEmpty {
VStack(alignment: .leading, spacing: 8) {
ForEach(playerController.pendingJobs, id: \.jobID) { job in
HStack {
Text(job.stage == "queued" ? "Queued…" : "Downloading…")
.font(.subheadline)
Spacer()
Text("\(Int((job.progress ?? 0)))%")
.font(.caption.monospacedDigit())
.foregroundStyle(.secondary)
}
}
}
.padding(12)
.background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14))
}
}
@ViewBuilder
private var choicesSection: some View {
if !playerController.choices.isEmpty {
VStack(alignment: .leading, spacing: 8) {
Text("Hope's suggestions")
.font(.headline)
ForEach(playerController.choices, id: \.videoID) { choice in
mediaRow(
title: choice.title ?? choice.videoID ?? "Unknown",
detail: rowDetail(channel: choice.channel, duration: choice.duration),
playEnabled: true
) {
Task {
await playerController.playVideo(
videoID: choice.videoID,
title: choice.title,
channel: choice.channel
)
}
}
}
}
}
}
private var searchSection: some View {
VStack(alignment: .leading, spacing: 8) {
Text("Find your next favorite")
.font(.headline)
Text("Search songs and artists on YouTube Music.")
.font(.subheadline).foregroundStyle(.secondary)
HStack(spacing: 8) {
TextField("Song, artist, lyrics…", text: $searchQuery)
.textFieldStyle(.roundedBorder)
.submitLabel(.search)
.onSubmit(performSearch)
Button("Search", action: performSearch)
.accessibilityIdentifier("music.search")
.buttonStyle(.bordered)
.frame(minHeight: 44)
.disabled(searchQuery.trimmingCharacters(in: .whitespaces).isEmpty || isSearching)
}
if isSearching {
HStack(spacing: 8) {
ProgressView()
Text("Searching YouTube Music…")
.font(.subheadline)
.foregroundStyle(.secondary)
}
} else if let searchError {
Text(searchError)
.font(.subheadline)
.foregroundStyle(.red)
} else if let results = searchResults {
if results.isEmpty {
Text("No matches. Try a different song or artist.")
.font(.subheadline)
.foregroundStyle(.secondary)
} else {
ForEach(results, id: \.videoID) { result in
mediaRow(
title: result.title ?? result.videoID ?? "Unknown",
detail: rowDetail(channel: result.channel, duration: result.duration, inLibrary: result.inLibrary),
playEnabled: true
) {
Task {
if let trackID = result.trackID {
await playerController.playTrack(id: trackID)
} else {
await playerController.playVideo(
videoID: result.videoID,
title: result.title,
channel: result.channel
)
}
}
}
}
}
}
}
}
private var librarySection: some View {
VStack(alignment: .leading, spacing: 8) {
HStack {
Text("Library")
.font(.headline)
Spacer()
Button(action: refreshLibrary) {
Image(systemName: "arrow.clockwise").frame(width: 44, height: 44)
}
.disabled(isLoadingLibrary)
.accessibilityLabel("Refresh library")
}
if let libraryError, playerController.stateError == nil {
Text(libraryError)
.font(.subheadline)
.foregroundStyle(.red)
} else if let tracks = libraryTracks {
if tracks.isEmpty {
ContentUnavailableView {
Label("A home for your favorites", systemImage: "music.note.list")
} description: {
Text("Search for a song to save it here, or add a radio station using the + button.")
} actions: {
Button("Find music") { browseMode = 1 }.buttonStyle(.bordered)
}
} else {
ForEach(tracks, id: \.id) { track in
HStack(spacing: 8) {
mediaRow(
title: track.title ?? track.id,
detail: rowDetail(
channel: track.channel ?? (track.kind == "stream" ? "Live stream" : ""),
duration: track.duration,
status: track.status
),
playEnabled: track.status == "ready"
) {
Task { await playerController.playTrack(id: track.id) }
}
Button(role: .destructive) {
trackToDelete = track
} label: {
Image(systemName: "xmark")
.font(.subheadline.weight(.semibold))
.foregroundStyle(.red)
.frame(width: 44, height: 44)
}
.accessibilityLabel("Remove \(track.title ?? track.id)")
}
}
}
} else if isLoadingLibrary {
ProgressView("Loading your library…").frame(maxWidth: .infinity).padding(.vertical, 24)
}
}
}
private var addStreamSection: some View {
Section {
TextField("Station name", text: $streamName)
.accessibilityLabel("Station name")
TextField("https://… audio stream URL", text: $streamURL)
.accessibilityLabel("Audio stream URL")
.keyboardType(.URL)
.autocorrectionDisabled()
.textContentType(.URL)
.textInputAutocapitalization(.never)
if let streamMessage {
Text(streamMessage)
.font(.subheadline)
.foregroundStyle(streamMessage.hasPrefix("Added") ? Color.green : Color.red)
}
Button(isAddingStream ? "Adding station…" : "Add station", action: addStream)
.disabled(streamURL.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || isAddingStream)
} footer: {
Text("Use a direct MP3 or AAC audio stream address, rather than the station’s website.")
}
}
private func mediaRow(
title: String,
detail: String,
playEnabled: Bool,
onPlay: @escaping () -> Void
) -> some View {
HStack(spacing: 10) {
VStack(alignment: .leading, spacing: 2) {
Text(title)
.font(.subheadline.weight(.semibold))
.lineLimit(2)
if !detail.isEmpty {
Text(detail)
.font(.caption)
.foregroundStyle(.secondary)
.lineLimit(2)
}
}
Spacer()
Button(action: onPlay) {
Image(systemName: "play.fill").frame(width: 44, height: 44)
}
.accessibilityLabel("Play \(title)")
.disabled(!playEnabled)
}
.padding(.vertical, 10)
}
private func rowDetail(channel: String?, duration: Double?, status: String? = nil, inLibrary: Bool? = nil) -> String {
var parts: [String] = []
if let channel, !channel.isEmpty { parts.append(channel) }
if let duration, duration > 0 { parts.append(fmtTime(duration)) }
if let status, status != "ready" { parts.append(status) }
if inLibrary == true { parts.append("in library") }
return parts.joined(separator: " · ")
}
private func performSearch() {
let query = searchQuery.trimmingCharacters(in: .whitespacesAndNewlines)
guard !query.isEmpty, !isSearching else { return }
isSearching = true
searchError = nil
Task {
do {
let response = try await MediaAPI(config: serverConfig).search(room: "default", query: query, limit: 6)
searchResults = response.results
} catch {
searchError = error.localizedDescription
}
isSearching = false
}
}
private func refreshLibrary() {
guard !isLoadingLibrary else { return }
isLoadingLibrary = true
Task {
defer { isLoadingLibrary = false }
do {
let library = try await MediaAPI(config: serverConfig).library()
libraryTracks = library.tracks
libraryError = nil
} catch {
libraryError = error.localizedDescription
}
}
}
private func addStream() {
let url = streamURL.trimmingCharacters(in: .whitespacesAndNewlines)
guard !isAddingStream else { return }
guard let parsed = URL(string: url), ["https", "http"].contains(parsed.scheme?.lowercased() ?? ""), parsed.host != nil else {
streamMessage = "Enter a complete http:// or https:// audio stream address."
return
}
isAddingStream = true
let name = streamName.trimmingCharacters(in: .whitespaces).isEmpty ? url : streamName.trimmingCharacters(in: .whitespaces)
Task {
defer { isAddingStream = false }
do {
try await MediaAPI(config: serverConfig).addStream(room: "default", name: name, url: url)
streamMessage = "Added \(name)"
streamName = ""
streamURL = ""
showAddStream = false
refreshLibrary()
} catch {
streamMessage = "Could not add stream: \(error.localizedDescription)"
}
}
}
private func deleteTrack(_ track: MediaTrack) async {
do {
try await MediaAPI(config: serverConfig).deleteTrack(id: track.id)
if playerController.track?.id == track.id {
Task { await playerController.stop() }
}
refreshLibrary()
} catch {
libraryError = error.localizedDescription
}
}
private var serverConfig: ServerConfig {
// Use the configured URL directly. If it's invalid, fall back to the
// documented default (media operations are less critical than the main
// session, and the Settings view will show a validation error).
if case .success(let config) = ServerConfig.parse(settings.trimmedServerURL) {
return config
}
let fallback = ServerConfig.parse("https://192.168.86.2:9178")
if case .success(let config) = fallback {
return config
}
fatalError("Unreachable: default server URL is valid")
}
private var title: String {
playerController.track?.title ?? "Nothing loaded"
}
private var subtitle: String {
guard let track = playerController.track else { return "" }
if track.status == "failed" {
return "Download failed — pick another result or try again."
}
if track.status == "downloading" {
return track.channel.map { "\($0) · downloading" } ?? "Downloading…"
}
return track.channel ?? (track.kind == "stream" ? "Live stream" : "")
}
private var statusText: String {
switch playerController.desired {
case "playing": return "Playing"
case "paused": return "Paused"
default: return "Stopped"
}
}
private func absolute(_ path: String) -> String {
if path.hasPrefix("http") { return path }
let base = serverConfig.baseURL.absoluteString
let suffix = path.hasPrefix("/") ? path : "/" + path
return base + suffix
}
private func togglePlayPause() {
Task { await playerController.togglePlayPause() }
}
private func seekBackward() {
Task { await playerController.seek(by: -10) }
}
private func seekForward() {
Task { await playerController.seek(by: 10) }
}
private func stopPlayback() {
Task { await playerController.stop() }
}
private func fmtTime(_ sec: Double) -> String {
guard sec.isFinite, sec >= 0 else { return "0:00" }
let total = Int(sec)
let s = total % 60
let m = (total / 60) % 60
let h = total / 3600
return h > 0 ? "\(h):\(String(format: "%02d", m)):\(String(format: "%02d", s))" : "\(m):\(String(format: "%02d", s))"
}
}
/// Artwork uses the same trust policy as token and media requests.
private struct PinnedArtwork: View {
let url: URL
@State private var artwork: UIImage?
var body: some View {
Group {
if let artwork { Image(uiImage: artwork).resizable().scaledToFill() }
else {
Image(systemName: "music.note").font(.title2).foregroundStyle(.secondary)
.frame(maxWidth: .infinity, maxHeight: .infinity)
.background(Color(.tertiarySystemFill))
}
}
.accessibilityHidden(true)
.task(id: url) {
artwork = nil
let session = TLSPinning.session()
defer { session.invalidateAndCancel() }
guard let (data, response) = try? await session.data(from: url),
(response as? HTTPURLResponse)?.statusCode == 200,
!Task.isCancelled else { return }
artwork = UIImage(data: data)
}
}
}
+1 -6
View File
@@ -2,7 +2,7 @@ import Foundation
import LiveKit import LiveKit
/// Outbound helpers for agent control topics and inbound parsing of /// Outbound helpers for agent control topics and inbound parsing of
/// voice-control / conversation-control / media-control data messages. /// voice-control / conversation-control data messages.
@MainActor @MainActor
final class ControlChannel: NSObject, RoomDelegate { final class ControlChannel: NSObject, RoomDelegate {
@@ -109,11 +109,6 @@ final class ControlChannel: NSObject, RoomDelegate {
try await publish(["type": "set_mode", "mode": mode], topic: "conversation-control", room: room) try await publish(["type": "set_mode", "mode": mode], topic: "conversation-control", room: room)
} }
func sendMediaState(_ state: String, trackID: String?, room: Room) async throws {
var object: [String: Any] = ["type": "media_state", "state": state]
if let trackID { object["track_id"] = trackID }
try await publish(object, topic: "media-control", room: room)
}
} }
// MARK: - Minimal JSON value decoder for flexible agent payloads // MARK: - Minimal JSON value decoder for flexible agent payloads
+4 -14
View File
@@ -48,7 +48,6 @@ final class VoiceSession: ObservableObject {
let transcriptHandler = TranscriptHandler() let transcriptHandler = TranscriptHandler()
let stateModel = ConversationStateModel() let stateModel = ConversationStateModel()
let taskStore = TaskStore() let taskStore = TaskStore()
let mediaController: MediaPlayerController
private let config: ServerConfig private let config: ServerConfig
private let tokenClient: TokenClient private let tokenClient: TokenClient
private let settings: SettingsStore private let settings: SettingsStore
@@ -90,9 +89,6 @@ final class VoiceSession: ObservableObject {
self.config = config self.config = config
self.tokenClient = tokenClient self.tokenClient = tokenClient
self.settings = settings self.settings = settings
let mediaAPI = MediaAPI(config: config)
self.mediaController = MediaPlayerController(api: mediaAPI, settings: settings)
mediaController.attach(session: self)
room.add(delegate: self) room.add(delegate: self)
room.add(delegate: controlChannel) room.add(delegate: controlChannel)
room.add(delegate: transcriptHandler) room.add(delegate: transcriptHandler)
@@ -211,7 +207,6 @@ final class VoiceSession: ObservableObject {
do { do {
let token = try await tokenClient.fetchToken() let token = try await tokenClient.fetchToken()
logger.log("token ok room=\(token.room, privacy: .public)") logger.log("token ok room=\(token.room, privacy: .public)")
await mediaController.enterMediaScope(roomName: token.room)
try await room.connect(url: config.livekitURL.absoluteString, token: token.token, connectOptions: makeConnectOptions()) try await room.connect(url: config.livekitURL.absoluteString, token: token.token, connectOptions: makeConnectOptions())
result = .success(()) result = .success(())
} catch { } catch {
@@ -263,8 +258,7 @@ final class VoiceSession: ObservableObject {
private func handleAgentTimeout(generation: Int) async { private func handleAgentTimeout(generation: Int) async {
guard generation == sessionGeneration, state == .waitingAgent else { return } guard generation == sessionGeneration, state == .waitingAgent else { return }
guard !reconnectAttempted else { guard !reconnectAttempted else {
// Final failure: tear down the live room, restore the idle timer, // Final failure: tear down the live room and restore the idle timer.
// and end the media scope so no stale room or wake lock survives.
logger.log("agent timeout final, tearing down") logger.log("agent timeout final, tearing down")
await teardownForFailure(message: "Agent never joined. Is the agent worker running?") await teardownForFailure(message: "Agent never joined. Is the agent worker running?")
return return
@@ -281,8 +275,7 @@ final class VoiceSession: ObservableObject {
/// current generation (so stale callbacks become no-ops), clear all /// current generation (so stale callbacks become no-ops), clear all
/// per-room/session ephemeral published fields (agent, mute state, model /// per-room/session ephemeral published fields (agent, mute state, model
/// list, error state, interruption ownership), disconnect and recreate the /// list, error state, interruption ownership), disconnect and recreate the
/// room, stop conversation state, restore the idle timer, and end the media /// room, stop conversation state, restore the idle timer, and publish
/// scope. Only after all cleanup completes does the session publish
/// `.failed(message)`, making retry safe: the old room is fully torn down /// `.failed(message)`, making retry safe: the old room is fully torn down
/// before Start becomes available again. /// before Start becomes available again.
/// ///
@@ -308,7 +301,6 @@ final class VoiceSession: ObservableObject {
interruptionSequence += 1 interruptionSequence += 1
stateModel.stop() stateModel.stop()
UIApplication.shared.isIdleTimerDisabled = false UIApplication.shared.isIdleTimerDisabled = false
await mediaController.endConversation()
await teardownRoom() await teardownRoom()
// Final reset immediately before publishing .failed: any old-room // Final reset immediately before publishing .failed: any old-room
@@ -540,15 +532,13 @@ final class VoiceSession: ObservableObject {
interruptionSequence += 1 interruptionSequence += 1
stateModel.stop() stateModel.stop()
UIApplication.shared.isIdleTimerDisabled = false UIApplication.shared.isIdleTimerDisabled = false
await mediaController.endConversation()
await teardownRoom() await teardownRoom()
// Final reset before publishing .idle: any old-room callback delivered // Final reset before publishing .idle: any old-room callback delivered
// during the teardownRoom() await may have repopulated ephemeral fields. // during the teardownRoom() await may have repopulated ephemeral fields.
// This ensures the idle state is clean for server replacement (Iteration 09). // This ensures the idle state is clean for server replacement (Iteration 09).
resetEphemeralFields() resetEphemeralFields()
// Publish .idle only after media and room teardown have completed. // Publish .idle only after room teardown has completed: the old session
// This makes idle a truthful readiness signal for server replacement: // is fully torn down before the UI shows "ready to start".
// the old session is fully torn down before the UI shows "ready to start".
state = .idle state = .idle
} }
-249
View File
@@ -1,249 +0,0 @@
import Foundation
struct MediaTrack: Decodable, Equatable {
let id: String
let kind: String?
let videoID: String?
let title: String?
let channel: String?
let duration: Double?
let status: String?
let error: String?
let audioURL: String?
let artURL: String?
enum CodingKeys: String, CodingKey {
case id, kind, title, channel, duration, status, error
case videoID = "video_id"
case audioURL = "audio_url"
case artURL = "art_url"
}
}
struct MediaPendingJob: Decodable, Equatable {
let jobID: String?
let trackID: String?
let stage: String?
let progress: Double?
let error: String?
enum CodingKeys: String, CodingKey {
case stage, progress, error
case jobID = "job_id"
case trackID = "track_id"
}
}
struct MediaSeekHint: Decodable, Equatable {
let ts: Double
let position: Double
}
struct MediaState: Decodable, Equatable {
let room: String?
let track: MediaTrack?
let desired: String?
let reported: MediaReported?
let seekHint: MediaSeekHint?
let choices: [MediaChoice]
let pending: [MediaPendingJob]
let version: Int?
enum CodingKeys: String, CodingKey {
case room, track, desired, reported, choices, pending, version
case seekHint = "seek_hint"
}
}
struct MediaReported: Decodable, Equatable {
let trackID: String?
let state: String?
let position: Double?
let duration: Double?
let ts: Double?
enum CodingKeys: String, CodingKey {
case state, position, duration, ts
case trackID = "track_id"
}
}
struct MediaChoice: Decodable, Equatable {
let videoID: String?
let title: String?
let channel: String?
let duration: Double?
enum CodingKeys: String, CodingKey {
case title, channel, duration
case videoID = "video_id"
}
}
struct MediaLibrary: Decodable, Equatable {
let tracks: [MediaTrack]
let version: Int?
}
struct MediaSearchResult: Decodable, Equatable {
let videoID: String?
let trackID: String?
let title: String?
let channel: String?
let duration: Double?
let inLibrary: Bool?
enum CodingKeys: String, CodingKey {
case title, channel, duration
case videoID = "video_id"
case trackID = "track_id"
case inLibrary = "in_library"
}
}
struct MediaSearchResponse: Decodable, Equatable {
let results: [MediaSearchResult]
}
enum MediaAPIError: LocalizedError {
case invalidServerURL(String)
case httpStatus(Int, String)
case decodeFailed
var errorDescription: String? {
switch self {
case .invalidServerURL(let raw):
return "Invalid server URL: \(raw)"
case .httpStatus(let code, let detail):
return "Media API request failed with HTTP \(code). \(detail)"
case .decodeFailed:
return "Unexpected response from the media endpoint."
}
}
}
final class MediaAPI {
let serverBase: URL
private let baseURL: URL
private let session: URLSession
init(config: ServerConfig, session: URLSession? = nil) {
self.serverBase = config.baseURL
self.baseURL = config.baseURL.appendingPathComponent("media/api")
self.session = session ?? TLSPinning.session()
}
func state(room: String) async throws -> MediaState {
var components = URLComponents(url: baseURL.appendingPathComponent("state"), resolvingAgainstBaseURL: false)!
components.queryItems = [URLQueryItem(name: "room", value: room)]
return try await get(MediaState.self, url: components.url!)
}
func report(room: String, state: String, trackID: String?, position: Double, duration: Double?) async throws {
var body: [String: Any] = ["room": room, "state": state, "position": position]
if let trackID { body["track_id"] = trackID }
if let duration { body["duration"] = duration }
try await post("report", body: body)
}
func play(room: String, videoID: String? = nil, title: String? = nil, channel: String? = nil, trackID: String? = nil, startPaused: Bool? = nil) async throws {
var body: [String: Any] = ["room": room]
if let videoID { body["video_id"] = videoID }
if let title { body["title"] = title }
if let channel { body["channel"] = channel }
if let trackID { body["track_id"] = trackID }
if let startPaused { body["start_paused"] = startPaused }
try await post("play", body: body)
}
func control(room: String, action: String) async throws {
try await post("control", body: ["room": room, "action": action])
}
func search(room: String, query: String, limit: Int = 6) async throws -> MediaSearchResponse {
try await postDecoded("search", body: ["room": room, "query": query, "limit": limit], as: MediaSearchResponse.self)
}
func library() async throws -> MediaLibrary {
try await get(MediaLibrary.self, url: baseURL.appendingPathComponent("library"))
}
func deleteTrack(id: String) async throws {
var request = URLRequest(url: baseURL.appendingPathComponent("tracks").appending(path: id))
request.httpMethod = "DELETE"
let (data, response) = try await session.data(for: request)
guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed }
guard (200..<300).contains(http.statusCode) else {
throw MediaAPIError.httpStatus(http.statusCode, String(data: data, encoding: .utf8) ?? "")
}
}
func addStream(room: String, name: String, url: String) async throws {
try await post("streams", body: ["room": room, "name": name, "url": url])
}
/// Returns the progressive-media endpoint URL for a track. The `/stream/{id}`
/// path serves raw media bytes (not a text URL), so this constructs the
/// endpoint without downloading or decoding the body.
func streamURL(trackID: String) -> URL {
baseURL.appendingPathComponent("stream").appending(path: trackID)
}
func healthz() async throws -> Bool {
let data = try await getRaw(url: baseURL.appendingPathComponent("healthz"))
guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { return false }
return json["ok"] as? Bool ?? false
}
private func get<T: Decodable>(_ type: T.Type, url: URL) async throws -> T {
let data = try await getRaw(url: url)
do {
return try JSONDecoder().decode(T.self, from: data)
} catch {
throw MediaAPIError.decodeFailed
}
}
private func getRaw(url: URL) async throws -> Data {
let (data, response) = try await session.data(from: url)
guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed }
guard (200..<300).contains(http.statusCode) else {
throw MediaAPIError.httpStatus(http.statusCode, String(data: data, encoding: .utf8) ?? "")
}
return data
}
/// Fire-and-forget POST that only validates HTTP success. The media server
/// returns `{"ok": true, ...snapshot}` for /play, /control, /report and
/// /streams — callers don't need the body, so decoding it (as Data or any
/// typed model) would be wasted work at best and a spurious failure mode.
private func post(_ path: String, body: [String: Any]) async throws {
_ = try await postRaw(path, body: body)
}
private func postDecoded<T: Decodable>(_ path: String, body: [String: Any], as type: T.Type) async throws -> T {
let data = try await postRaw(path, body: body)
do {
return try JSONDecoder().decode(T.self, from: data)
} catch {
throw MediaAPIError.decodeFailed
}
}
private func postRaw(_ path: String, body: [String: Any]) async throws -> Data {
var request = URLRequest(url: baseURL.appendingPathComponent(path))
request.httpMethod = "POST"
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
request.httpBody = try JSONSerialization.data(withJSONObject: body)
let (data, response) = try await session.data(for: request)
guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed }
guard (200..<300).contains(http.statusCode) else {
var detail = String(data: data, encoding: .utf8) ?? ""
if let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], let error = json["error"] as? String {
detail = error
}
throw MediaAPIError.httpStatus(http.statusCode, detail)
}
return data
}
}
-633
View File
@@ -1,633 +0,0 @@
import AVFoundation
import Combine
import Foundation
import os
private let logger = Logger(subsystem: "dev.shane.voice", category: "media")
/// AVPlayer-backed media player. The media server holds per-room desired state;
/// this controller polls /state and reconciles the local player against it,
/// porting the web UI's reconciliation rules (app.js reconcileMedia).
@MainActor
final class MediaPlayerController: ObservableObject {
@Published private(set) var track: MediaTrack?
@Published private(set) var desired: String = "stopped"
@Published private(set) var currentTime: Double = 0
@Published private(set) var duration: Double = 0
@Published private(set) var pendingJobs: [MediaPendingJob] = []
@Published private(set) var choices: [MediaChoice] = []
@Published private(set) var stateError: String?
@Published private(set) var playbackError: String?
let player = AVPlayer()
private let api: MediaAPI
private let settings: SettingsStore
private let mediaLoader: PinnedMediaResourceLoader
private weak var session: VoiceSession?
private var room: String?
private var currentTrackID: String?
private var loadedAudioPath: String?
private var currentAsset: AVURLAsset?
private var appliedSeekTs: Double = 0
private var wasPlayingBeforeTalk = false
private var lastReportDate = Date.distantPast
private var pollTask: Task<Void, Never>?
private var timeObserver: Any?
private var statusCancellable: AnyCancellable?
private var rateObservation: NSKeyValueObservation?
/// Bumped on every room-scope change so in-flight /state polls from the
/// previous scope cannot reconcile into the new one (port of the web UI's
/// `=== room` checks).
private var pollGeneration = 0
init(api: MediaAPI, settings: SettingsStore) {
self.api = api
self.settings = settings
self.mediaLoader = PinnedMediaResourceLoader(base: api.serverBase)
player.volume = Float(settings.volume)
NotificationCenter.default.addObserver(
self, selector: #selector(handleInterruption),
name: AVAudioSession.interruptionNotification, object: nil
)
}
deinit {
if let timeObserver {
player.removeTimeObserver(timeObserver)
}
rateObservation?.invalidate()
NotificationCenter.default.removeObserver(self)
}
/// Whether an audio session interruption is currently active. While true,
/// reconciliation must not call player.play() (the next poll would restart
/// audio during a phone call or Siri interaction).
private var isInterruptionActive = false
/// Separate resume-suppression flag: set when an interruption ends without
/// `.shouldResume` and the pause control to the server has not yet been
/// confirmed. While true, reconciliation must not call player.play() even
/// after `isInterruptionActive` is cleared. Cleared only by an explicit user
/// play/resume action or a confirmed coherent paused state from the server.
private var resumeSuppressed = false
/// Whether playback was actually active (rate > 0) when the interruption
/// began. Used to decide whether a `.shouldResume` end should actually
/// resume (a track that was already paused before the interruption must not
/// be force-resumed).
private var wasPlayingBeforeInterruption = false
/// Sequence number for media interruption operations. Incremented on every
/// `.began`; a resume task captures the sequence and rechecks it immediately
/// before `player.play()`. A newer `.began` makes an old resume task
/// incapable of playing (Iteration 07 Item 5).
private var mediaInterruptionSequence = 0
/// The retained operation chain for media interruption control work. Both
/// pause and resume operations are assigned here so a newer begin cancels
/// and supersedes prior control work.
private var mediaInterruptionTask: Task<Void, Never>?
@objc private func handleInterruption(_ notification: Notification) {
guard let info = notification.userInfo else { return }
guard let typeValue = info[AVAudioSessionInterruptionTypeKey] as? UInt,
let type = AVAudioSession.InterruptionType(rawValue: typeValue) else { return }
switch type {
case .began:
// Increment the sequence so any in-flight resume task from a prior
// interruption sees a stale sequence and cannot play.
mediaInterruptionSequence += 1
let seq = mediaInterruptionSequence
// Cancel/supersede prior control work (resume or pause) so it cannot
// complete out of order.
mediaInterruptionTask?.cancel()
isInterruptionActive = true
wasPlayingBeforeInterruption = !player.rate.isZero
let wasPlaying = wasPlayingBeforeInterruption
player.pause()
logger.log("audio interruption began, paused media (wasPlaying=\(wasPlaying), seq=\(seq))")
case .ended:
defer { isInterruptionActive = false }
applyVolume()
let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
let shouldResume = AVAudioSession.InterruptionOptions(rawValue: optionsValue).contains(.shouldResume)
if shouldResume, wasPlayingBeforeInterruption, currentTrackID != nil, desired == "playing", loadedAudioPath != nil {
// Synchronize the server before resuming local playback so the
// two never diverge. Capture generation/room/sequence before the
// async hop. The task is retained in mediaInterruptionTask so a
// newer .began can cancel it.
let gen = pollGeneration
let targetRoom = room
let seq = mediaInterruptionSequence
mediaInterruptionTask = Task { [weak self] in
guard let self else { return }
if let targetRoom {
do {
try await self.api.control(room: targetRoom, action: "resume")
logger.log("interruption resume: server synchronized (seq=\(seq))")
} catch {
logger.log("interruption resume: server sync failed (poll will retry): \(error.localizedDescription)")
}
}
// Recheck the sequence immediately before play: a newer .began
// increments mediaInterruptionSequence, making this stale task
// incapable of restarting audio during the newer interruption.
guard !Task.isCancelled,
self.mediaInterruptionSequence == seq,
!self.isInterruptionActive,
self.pollGeneration == gen,
self.room == targetRoom else { return }
self.player.play()
logger.log("audio interruption ended, resumed media (seq=\(seq))")
}
} else {
// No-resume path: keep suppression even if the pause control
// fails. The flag is cleared only by an explicit user play/resume
// action or a confirmed coherent paused state from the server.
resumeSuppressed = true
let seq = mediaInterruptionSequence
if let room {
mediaInterruptionTask = Task { [weak self] in
guard let self else { return }
do {
try await self.api.control(room: room, action: "pause")
logger.log("audio interruption ended without resume, sent pause to server (seq=\(seq))")
} catch {
// Keep suppression even on failure — the next poll
// will see desired=="paused" and clear it.
logger.log("audio interruption ended without resume, pause control failed (suppression retained): \(error.localizedDescription)")
}
}
}
}
@unknown default:
break
}
}
// MARK: Lifecycle
func attach(session: VoiceSession) {
self.session = session
observePlayer()
observePlaybackState()
applyVolume()
}
/// Called just before joining the LiveKit room. Carries whatever is loaded
/// into the new room-scoped player state, paused (talking pauses music).
func enterMediaScope(roomName: String) async {
pollGeneration += 1
let wasPlaying = currentTrackID != nil && !player.rate.isZero
wasPlayingBeforeTalk = wasPlaying
let carriedTrack = currentTrackID
room = roomName
if let carriedTrack {
do {
try await api.play(room: roomName, trackID: carriedTrack, startPaused: true)
logger.log("media scope carry-over: \(carriedTrack, privacy: .public) start_paused")
} catch {
logger.log("media scope transfer failed: \(String(describing: error), privacy: .public)")
}
player.pause()
}
startPolling()
}
/// Called when the conversation ends. If music was playing before the talk,
/// hand the room back to the player. The previous scope (room name) is kept
/// so the panel can resume polling it via `startDefaultPolling()`; a fresh
/// conversation re-scopes via `enterMediaScope`. Polling restarts here so
/// the resumed state stays visible without the panel being open.
func endConversation() async {
stopPolling()
pollGeneration += 1
if wasPlayingBeforeTalk {
wasPlayingBeforeTalk = false
guard let room else { return }
do {
try await api.control(room: room, action: "resume")
logger.log("resumed playback after talk ended")
} catch {
logger.log("post-talk resume failed (poll will retry): \(String(describing: error), privacy: .public)")
}
}
startPolling()
}
/// Drop the room scope entirely (app-level teardown). After this, polling
/// targets the default room again.
func resetScope() {
stopPolling()
pollGeneration += 1
room = nil
wasPlayingBeforeTalk = false
clearMedia()
}
private func startPolling() {
guard pollTask == nil else { return }
pollTask = Task { [weak self] in
while !Task.isCancelled {
await self?.pollState()
try? await Task.sleep(for: .milliseconds(1500))
}
}
}
private func stopPolling() {
pollTask?.cancel()
pollTask = nil
}
/// Resume polling whenever the panel becomes visible and nothing is being
/// polled yet — whether that's the default room (no session) or a previous
/// conversation's room scope (kept after `endConversation`).
func startDefaultPolling() {
guard pollTask == nil else { return }
startPolling()
}
// MARK: Poll + reconcile (ported from app.js pollMediaState/reconcileMedia)
private func pollState() async {
let generation = pollGeneration
let scope = room ?? "default"
do {
let state = try await api.state(room: scope)
guard generation == pollGeneration else { return }
stateError = nil
reconcile(state)
} catch {
guard generation == pollGeneration else { return }
logger.log("poll failed: \(String(describing: error), privacy: .public)")
stateError = "Media server unreachable: \(error.localizedDescription)"
}
}
/// Reconciles the local player against a server state snapshot. Exposed
/// as internal (not private) so tests can drive deterministic state
/// transitions without a live server.
func reconcile(_ s: MediaState) {
choices = s.choices
pendingJobs = s.pending
// Keep the published desired state in sync with the server so the UI's
// play/pause icon and status text reflect reality (the server is the
// source of truth for per-room desired state).
desired = s.desired ?? "stopped"
let track = s.track
self.track = track
if let track, track.id != currentTrackID {
loadTrack(track)
} else if let track, let audioURL = track.audioURL, loadedAudioPath != audioURL {
// Track was loaded before it had a stream (still downloading) and
// now has one — wire it up so playback can start without a reload.
wireSource(audioURL)
}
if track == nil, currentTrackID != nil, s.desired == "stopped" {
clearMedia()
}
applyDesiredState(s, track: track)
if let hint = s.seekHint, hint.ts != appliedSeekTs, duration.isFinite, duration > 0 {
appliedSeekTs = hint.ts
let target = min(hint.position, duration)
if target.isFinite {
seek(to: max(0, target))
}
}
}
private func applyDesiredState(_ s: MediaState, track: MediaTrack?) {
guard let desiredValue = s.desired else { return }
switch desiredValue {
case "playing":
// Suppress play during an active audio interruption (phone call,
// Siri). Without this guard the next 1.5s poll would restart audio
// immediately after .began paused it. The player.rate.isZero check
// makes this idempotent: once playing, subsequent polls are no-ops.
// This also handles the late-audio case where desired=playing
// arrives before audio_url is set; when the source is later wired
// via wireSource(), the next poll retries play because rate is
// still zero.
// Also suppress while resumeSuppressed is true (no-resume
// interruption ended, server not yet confirmed paused).
guard !isInterruptionActive, !resumeSuppressed else { return }
if let track, track.audioURL != nil, loadedAudioPath != nil, player.rate.isZero {
player.play()
}
case "paused", "stopped":
if !player.rate.isZero {
player.pause()
}
// The server has confirmed a coherent paused/stopped state. If we
// were suppressing resume due to a no-resume interruption, the
// pause control succeeded (or the user paused independently), so
// it is safe to clear the suppression flag.
if resumeSuppressed {
resumeSuppressed = false
logger.log("resume suppression cleared: server confirmed \(desiredValue)")
}
default:
break
}
}
private func loadTrack(_ track: MediaTrack) {
currentTrackID = track.id
self.track = track
if let audioURL = track.audioURL {
wireSource(audioURL)
} else {
// Downloading or failed: no stream yet. Reconcile wires the stream
// URL in via polling as soon as one is available.
clearCurrentAsset()
player.replaceCurrentItem(with: nil)
}
}
private func wireSource(_ path: String) {
// A complete file is required for the server's cached AAC conversion.
// Keep showing download progress; the next ready-state poll wires it.
guard track?.kind != "youtube" || track?.status != "downloading" else { return }
guard var components = URLComponents(string: path) else { return }
if track?.kind == "youtube", components.path.hasPrefix("/media/api/stream/") {
var items = components.queryItems ?? []
items.removeAll { $0.name == "format" }
items.append(URLQueryItem(name: "format", value: "m4a"))
components.queryItems = items
}
guard let url = components.url else { return }
loadedAudioPath = path
let pinned = mediaLoader.rewrite(url)
// Remove the previous asset's loader mapping before wiring a new one so
// the loaderURLs dictionary stays bounded (Finding 11 / Item 7).
clearCurrentAsset()
let asset = mediaLoader.assetWithPinnedURL(pinned)
currentAsset = asset
logger.log("wiring source \(pinned.absoluteString, privacy: .public)")
let item = AVPlayerItem(asset: asset)
observeItem(item)
player.replaceCurrentItem(with: item)
}
/// Removes the loader mapping for the current asset (if any) and clears the
/// reference. Called before wiring a new source or clearing media so the
/// resource loader's internal dictionaries stay bounded.
private func clearCurrentAsset() {
if let asset = currentAsset {
mediaLoader.removeLoader(asset)
currentAsset = nil
}
}
// MARK: Player events
private func observePlayer() {
if timeObserver == nil {
timeObserver = player.addPeriodicTimeObserver(
forInterval: CMTime(seconds: 0.5, preferredTimescale: 600), queue: .main
) { [weak self] time in
Task { @MainActor [weak self] in
guard let self else { return }
self.currentTime = time.seconds
if let itemDuration = self.player.currentItem?.duration {
let seconds = itemDuration.seconds
if seconds.isFinite {
self.duration = seconds
}
}
if !self.player.rate.isZero, Date().timeIntervalSince(self.lastReportDate) >= 5 {
self.lastReportDate = Date()
self.report("playing")
}
}
}
}
}
// MARK: - Playback state observation
/// Observe the player's actual rate so every real play/pause transition is
/// reported promptly, mirroring the web UI's `play`/`pause` listeners. The
/// KVO block runs on an arbitrary queue; hop to MainActor before touching
/// any controller state.
private func observePlaybackState() {
rateObservation = player.observe(\.rate, options: [.initial, .new]) { [weak self] player, _ in
Task { @MainActor [weak self] in
guard let self else { return }
guard self.currentTrackID != nil else { return }
if player.rate > 0 {
// Actual playback began (user play, interruption resume, or
// reconcile-driven play). Report + publish the transition.
self.lastReportDate = Date()
self.report("playing")
self.publishMediaState("playing")
} else if player.currentItem != nil {
// Rate dropped to zero with an item still loaded: a pause
// that didn't originate from a user action (interruption,
// reconcile). User-initiated pauses already report before
// this fires; the lastReportDate guard makes duplicates no-ops.
if Date().timeIntervalSince(self.lastReportDate) >= 1 {
self.lastReportDate = Date()
self.report("paused")
self.publishMediaState("paused")
}
}
}
}
}
private func observeItem(_ item: AVPlayerItem?) {
NotificationCenter.default.removeObserver(
self, name: .AVPlayerItemDidPlayToEndTime, object: nil
)
guard let item else { return }
NotificationCenter.default.addObserver(
self, selector: #selector(playerItemDidEnd(_:)),
name: .AVPlayerItemDidPlayToEndTime, object: item
)
statusCancellable = item.publisher(for: \.status).sink { [weak self, weak item] status in
Task { @MainActor in
guard let self, let item, self.player.currentItem === item else { return }
switch status {
case .readyToPlay:
self.playbackError = nil
case .failed:
self.playbackError = "This audio couldn’t play on your iPhone. Try another track or an MP3/AAC station."
logger.log("item failed: \(String(describing: item.error), privacy: .public)")
default: break
}
}
}
}
@objc private func playerItemDidEnd(_ notification: Notification) {
lastReportDate = Date()
report("stopped")
publishMediaState("stopped")
Task { [weak self] in
guard let self else { return }
try? await self.api.control(room: self.room ?? "default", action: "stop")
}
}
// MARK: Reporting + publishing
private func report(_ state: String, trackID: String? = nil) {
let room = room ?? "default"
let position = currentTime.isFinite ? currentTime : 0
var durationValue: Double?
if duration.isFinite, duration > 0 {
durationValue = duration
}
let resolvedTrackID = trackID ?? currentTrackID
Task { [weak self] in
guard let self else { return }
do {
try await self.api.report(room: room, state: state, trackID: resolvedTrackID, position: position, duration: durationValue)
} catch {
logger.log("report failed: \(String(describing: error), privacy: .public)")
}
}
}
private func publishMediaState(_ state: String, trackID: String? = nil) {
guard let session else { return }
// Capture the exact room and generation before spawning so a delayed
// publish from an old lifecycle cannot escape into a new conversation
// room (Finding 10).
let targetRoom = session.room
let generation = session.sessionGeneration
let pollGen = pollGeneration
let resolvedTrackID = trackID ?? currentTrackID
Task { [weak self] in
guard let self, self.pollGeneration == pollGen else { return }
guard session.sessionGeneration == generation, session.room === targetRoom else { return }
do {
try await session.controlChannel.sendMediaState(state, trackID: resolvedTrackID, room: targetRoom)
} catch {
logger.log("media_state publish failed: \(String(describing: error), privacy: .public)")
}
}
}
// MARK: Play requests (search results, choices, library)
func playVideo(videoID: String?, title: String?, channel: String?) async {
let targetRoom = room ?? "default"
let generation = pollGeneration
playbackError = nil
do {
try await api.play(room: targetRoom, videoID: videoID, title: title, channel: channel)
logger.log("play requested video_id=\(videoID ?? "nil", privacy: .public) room=\(targetRoom, privacy: .public)")
} catch {
guard generation == pollGeneration else { return }
playbackError = "Couldn’t play this song. Check your connection and try again."
logger.log("play failed: \(String(describing: error), privacy: .public)")
}
}
func playTrack(id: String) async {
let targetRoom = room ?? "default"
let generation = pollGeneration
playbackError = nil
do {
try await api.play(room: targetRoom, trackID: id)
logger.log("play requested track_id=\(id, privacy: .public) room=\(targetRoom, privacy: .public)")
} catch {
guard generation == pollGeneration else { return }
playbackError = "Couldn’t play this track. Check your connection and try again."
logger.log("play failed: \(String(describing: error), privacy: .public)")
}
}
// MARK: Transport controls (parity with web UI play/pause/seek/stop)
func togglePlayPause() async {
guard currentTrackID != nil, loadedAudioPath != nil else { return }
if player.rate.isZero {
// Explicit user play: clear any resume suppression (the user is
// deliberately overriding the no-resume interruption state).
resumeSuppressed = false
// The rate observer reports + publishes the actual transition;
// the control call updates server desired state.
player.play()
await sendControl("resume")
} else {
lastReportDate = Date()
report("paused")
publishMediaState("paused")
player.pause()
await sendControl("pause")
}
}
func seek(by delta: Double) async {
guard loadedAudioPath != nil else { return }
let target = max(0, min(duration > 0 ? duration - 0.5 : .infinity, currentTime + delta))
if target.isFinite {
seek(to: target)
}
}
func seek(to position: Double) {
guard position.isFinite, position >= 0 else { return }
player.seek(to: CMTime(seconds: position, preferredTimescale: 600))
currentTime = position
report(player.rate.isZero ? "paused" : "playing")
}
func stop() async {
// Capture the track ID before clearing so the async report/publish
// tasks can read a non-nil value even after clearMedia() runs.
let trackID = currentTrackID
report("stopped", trackID: trackID)
clearMedia()
await sendControl("stop")
publishMediaState("stopped", trackID: trackID)
}
private func clearMedia() {
playbackError = nil
currentTrackID = nil
track = nil
loadedAudioPath = nil
// Remove the loader mapping for the asset being cleared so the
// resource loader's internal dictionaries stay bounded (Item 7).
clearCurrentAsset()
observeItem(nil)
player.replaceCurrentItem(with: nil)
currentTime = 0
duration = 0
stateError = nil
}
private func sendControl(_ action: String) async {
let generation = pollGeneration
do {
try await api.control(room: room ?? "default", action: action)
guard generation == pollGeneration else { return }
playbackError = nil
} catch {
guard generation == pollGeneration else { return }
playbackError = "Couldn’t update playback on the server. Check your connection and try again."
}
}
// MARK: Volume
func applyVolume() {
player.volume = Float(settings.volume)
}
func setVolume(_ volume: Double) {
settings.volume = volume
player.volume = Float(volume)
}
}
@@ -1,238 +0,0 @@
import AVFoundation
import Foundation
import UniformTypeIdentifiers
import os
private let logger = Logger(subsystem: "dev.shane.voice", category: "media")
/// Proxies {base}/media/* bytes to AVPlayer over the pinned-TLS session, since
/// AVPlayer cannot trust the self-signed LAN cert. AVPlayer loads https natively
/// (ignoring registered URLProtocol subclasses), so the asset uses a custom
/// `pinned-media` scheme and this delegate answers every data request by
/// fetching the real https URL through TLSPinning.session(), forwarding Range.
///
/// **Lifecycle invariants (Finding 11 / Iteration 06 Item 7):**
/// - Tasks are keyed by `ObjectIdentifier(loadingRequest)` directly, not a
/// derived hash, so two distinct requests can never collide.
/// - All mutable state (`loaderURLs`, `activeTasks`) is accessed only on the
/// main queue: the AVAssetResourceLoader delegate is set with `queue: .main`
/// and the pinned URLSession uses `delegateQueue: .main`, so both the
/// resource-loading callbacks and the data-task completion handlers run on
/// the same serialization domain. This is enforced in code, not by comment.
/// - For HTTP 206, `contentLength` is set only when `Content-Range` provides a
/// known total; it never falls back to the range's `expectedContentLength`.
/// - For HTTP 200, `expectedContentLength` is already the full resource size.
/// - Cleanup (task removal) occurs on success, error, cancellation, and item
/// replacement via `removeLoader`.
final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate {
static let scheme = "pinned-media"
private let base: URL
/// Maps each asset's resource loader to the real https URL it proxies.
/// Accessed only on the main queue (the delegate queue).
private var loaderURLs: [ObjectIdentifier: URL] = [:]
/// Maps each loading request's identity to its upstream data task so
/// cancellation can cancel the in-flight download. Keyed directly by
/// `ObjectIdentifier(loadingRequest)` — no hash derivation, no collisions.
private var activeTasks: [ObjectIdentifier: URLSessionDataTask] = [:]
/// Shared pinned session, created once and invalidated on deinit.
/// Uses `delegateQueue: .main` so all completion handlers (and therefore
/// all mutations of `activeTasks`) run on the main queue — the same
/// serialization domain as the AVAssetResourceLoader delegate callbacks
/// (which are also dispatched to `.main`). This makes map access truly
/// serialized in code, not just by convention.
private lazy var pinnedSession: URLSession = {
let config = URLSessionConfiguration.default
config.timeoutIntervalForRequest = 30
return URLSession(configuration: config, delegate: PinnedTrustDelegate(), delegateQueue: .main)
}()
init(base: URL) {
self.base = base
}
deinit {
pinnedSession.invalidateAndCancel()
}
/// Maps a media URL (absolute https or /media/... path) to the pinned-media
/// scheme so AVPlayer routes it through this delegate instead of loading
/// https natively. Built as a string because URLComponents.url returns nil
/// for schemes not in Foundation's known-scheme list.
func rewrite(_ url: URL) -> URL {
let absolute: String
if let scheme = url.scheme?.lowercased(), scheme.hasPrefix("http") {
var c = URLComponents(url: url, resolvingAgainstBaseURL: false)!
c.scheme = Self.scheme
return c.url!
} else {
let path = url.absoluteString.hasPrefix("/") ? String(url.absoluteString.dropFirst()) : url.absoluteString
guard let host = base.host, !host.isEmpty else { return url }
let port = base.port.map { ":\($0)" } ?? ""
absolute = "\(Self.scheme)://\(host)\(port)/\(path)"
}
return URL(string: absolute) ?? url
}
/// Maps a pinned-media URL back to the real https URL it proxies.
func resolve(_ url: URL) -> URL {
guard let scheme = url.scheme?.lowercased(), scheme == Self.scheme else { return url }
var c = URLComponents(url: url, resolvingAgainstBaseURL: false)!
c.scheme = "https"
return c.url ?? url
}
/// Creates an AVURLAsset that routes through this loader's delegate. The
/// mapping from the asset's resource loader to the real URL is recorded so
/// `resourceLoader(_:shouldWaitForLoadingOfRequestedResource:)` can resolve
/// it. All mutations happen on the main queue (the delegate queue).
func assetWithPinnedURL(_ pinned: URL) -> AVURLAsset {
let asset = AVURLAsset(url: pinned)
let realURL = resolve(pinned)
// The delegate queue is .main, so this runs on the main thread.
loaderURLs[ObjectIdentifier(asset.resourceLoader)] = realURL
asset.resourceLoader.setDelegate(self, queue: .main)
return asset
}
/// Removes the loader mapping and any in-flight tasks for an asset that is
/// no longer needed. Called by MediaPlayerController when replacing or
/// clearing media items to bound the dictionaries (Finding 11).
func removeLoader(_ asset: AVURLAsset) {
let key = ObjectIdentifier(asset.resourceLoader)
loaderURLs[key] = nil
// Cancel any in-flight tasks. URLSessionDataTask has no isCancelled
// property; calling cancel() on an already-cancelled task is a no-op.
for (_, task) in activeTasks {
task.cancel()
}
activeTasks.removeAll()
}
func resourceLoader(
_ resourceLoader: AVAssetResourceLoader,
shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest
) -> Bool {
// All state access is on the main queue (the delegate queue).
guard loadingRequest.dataRequest != nil else { return false }
guard let url = loaderURLs[ObjectIdentifier(resourceLoader)] else {
logger.log("no resolved URL for resource loader")
loadingRequest.finishLoading(with: URLError(.badURL))
return false
}
fetchChunk(loadingRequest, url: url)
return true
}
/// Return bounded chunks promptly instead of buffering an entire song before
/// AVPlayer receives its first usable bytes. The same loading request stays
/// open until its requested range (or EOF) has been delivered.
private func fetchChunk(_ loadingRequest: AVAssetResourceLoadingRequest, url: URL) {
guard !loadingRequest.isCancelled, let dataRequest = loadingRequest.dataRequest else { return }
var upstream = URLRequest(url: url)
let offset = max(dataRequest.requestedOffset, dataRequest.currentOffset)
let length = Int64(dataRequest.requestedLength)
let requestedEnd = dataRequest.requestedOffset + length
let end = dataRequest.requestsAllDataToEndOfResource || length == 0
? offset + 262_143 : min(requestedEnd - 1, offset + 262_143)
upstream.setValue("bytes=\(offset)-\(end)", forHTTPHeaderField: "Range")
logger.log("fetching \(url.absoluteString, privacy: .public) range=\(upstream.value(forHTTPHeaderField: "Range") ?? "-", privacy: .public)")
// Key the task by the loading request's identity directly — no hash
// derivation, so two distinct requests can never collide.
let requestKey = ObjectIdentifier(loadingRequest)
let task = pinnedSession.dataTask(with: upstream) { [weak self] data, response, error in
guard let self else { return }
// Cleanup: remove the task from the active map regardless of outcome.
// Runs on .main (the session's delegateQueue), same domain as all
// other map mutations.
self.activeTasks[requestKey] = nil
guard !loadingRequest.isCancelled else { return }
if let error {
logger.log("fetch failed \(url.absoluteString, privacy: .public): \(error.localizedDescription, privacy: .public)")
loadingRequest.finishLoading(with: error)
return
}
guard let http = response as? HTTPURLResponse else {
loadingRequest.finishLoading(with: URLError(.badServerResponse))
return
}
// Reject non-200/206 responses (404, 500, etc.) instead of feeding
// error JSON into AVPlayer as media bytes.
guard http.statusCode == 200 || http.statusCode == 206 else {
logger.log("unexpected status \(http.statusCode) for \(url.absoluteString, privacy: .public)")
loadingRequest.finishLoading(with: URLError(.badServerResponse))
return
}
if let info = loadingRequest.contentInformationRequest {
// AVFoundation requires a UTI, not the HTTP MIME string.
info.contentType = http.mimeType.flatMap { UTType(mimeType: $0)?.identifier }
// Content-length handling (Iteration 05 Item 7):
// - For 206: set full content length ONLY when Content-Range
// provides a known total. Never fall back to the range's
// expectedContentLength (that would tell AVPlayer the total
// resource is only as large as this one range).
// - For 200: expectedContentLength is already the full size.
if http.statusCode == 206 {
if let total = Self.parseTotalLength(http), total >= 0 {
info.contentLength = total
}
// If Content-Range has no known total (e.g. "bytes 0-1023/*"),
// do NOT set contentLength — AVPlayer will handle it as
// unknown-length, which is correct for a range response.
} else {
// 200: expectedContentLength is the full resource size.
if http.expectedContentLength >= 0 {
info.contentLength = Int64(http.expectedContentLength)
}
}
info.isByteRangeAccessSupported = true
}
if let data, !data.isEmpty {
dataRequest.respond(with: data)
let nextOffset = offset + Int64(data.count)
let total = Self.parseTotalLength(http)
let reachedEOF = total.map { nextOffset >= $0 } ?? (nextOffset <= end)
let needsMore = dataRequest.requestsAllDataToEndOfResource || nextOffset < requestedEnd
if http.statusCode == 206, !reachedEOF, needsMore {
self.fetchChunk(loadingRequest, url: url)
return
}
}
loadingRequest.finishLoading()
}
activeTasks[requestKey] = task
task.resume()
}
func resourceLoader(
_ resourceLoader: AVAssetResourceLoader,
didCancel loadingRequest: AVAssetResourceLoadingRequest
) {
// Cancel the upstream data task so a seek or item swap doesn't keep
// downloading cancelled ranges. Keyed by ObjectIdentifier directly.
let key = ObjectIdentifier(loadingRequest)
activeTasks[key]?.cancel()
activeTasks[key] = nil
logger.log("resource loader cancelled request, upstream task cancelled")
}
/// Parses the total resource length from an HTTP 206 response's
/// `Content-Range` header (e.g. "bytes 0-1023/4096" → 4096). Returns nil
/// for non-206 responses, missing headers, unknown totals ("*"), or
/// malformed values.
static func parseTotalLength(_ response: HTTPURLResponse) -> Int64? {
if response.statusCode == 206,
let contentRange = response.value(forHTTPHeaderField: "Content-Range") {
// Format: "bytes <start>-<end>/<total>" or "bytes <start>-<end>/*"
let afterSlash = contentRange.split(separator: "/").last?.trimmingCharacters(in: .whitespaces)
if let afterSlash, afterSlash != "*", let total = Int64(afterSlash) {
return total
}
}
return nil
}
}
-125
View File
@@ -1,125 +0,0 @@
import Foundation
import Testing
@testable import Voice
/// Minimal HTTP/1.1 stub built on BSD sockets that answers every request with a
/// successful JSON object body (mirroring the real media server's
/// `{"ok": true, ...}` shape). Used to prove MediaAPI no-response POSTs succeed
/// against an object body instead of throwing decodeFailed.
@MainActor
final class MediaAPIStubServer {
private(set) var url: URL!
nonisolated(unsafe) private var serverFD: Int32 = -1
nonisolated(unsafe) private var acceptThread: Thread?
nonisolated(unsafe) private var running = false
nonisolated(unsafe) private var lastPath: String?
func start() throws {
let fd = socket(AF_INET, SOCK_STREAM, 0)
guard fd >= 0 else { throw MediaAPIError.decodeFailed }
var yes: Int32 = 1
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout<Int32>.size))
var addr = sockaddr_in()
addr.sin_family = sa_family_t(AF_INET)
addr.sin_port = in_port_t(0).bigEndian
addr.sin_addr = in_addr(s_addr: inet_addr("127.0.0.1"))
let bindResult = withUnsafePointer(to: &addr) { ptr in
ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in
bind(fd, sa, socklen_t(MemoryLayout<sockaddr_in>.size))
}
}
guard bindResult == 0 else { close(fd); throw MediaAPIError.decodeFailed }
guard listen(fd, 8) == 0 else { close(fd); throw MediaAPIError.decodeFailed }
var bound = sockaddr_in()
var len = socklen_t(MemoryLayout<sockaddr_in>.size)
withUnsafeMutablePointer(to: &bound) { ptr in
_ = ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in
getsockname(fd, sa, &len)
}
}
let port = Int(UInt16(bigEndian: bound.sin_port))
serverFD = fd
running = true
url = URL(string: "http://127.0.0.1:\(port)")!
let thread = Thread { [weak self] in
self?.acceptLoop()
}
acceptThread = thread
thread.start()
}
func stop() {
running = false
if serverFD >= 0 { close(serverFD) }
serverFD = -1
acceptThread?.cancel()
}
var lastRequestPath: String? { lastPath }
nonisolated private func acceptLoop() {
while running {
var clientAddr = sockaddr_in()
var len = socklen_t(MemoryLayout<sockaddr_in>.size)
let clientFD = withUnsafeMutablePointer(to: &clientAddr) { ptr in
ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in
accept(serverFD, sa, &len)
}
}
guard clientFD >= 0 else { continue }
handleClient(clientFD)
}
}
nonisolated private func handleClient(_ clientFD: Int32) {
defer { close(clientFD) }
var buffer = [UInt8](repeating: 0, count: 4096)
let bytesRead = read(clientFD, &buffer, buffer.count)
guard bytesRead > 0 else { return }
guard let request = String(bytes: buffer[0..<bytesRead], encoding: .utf8) else { return }
var path = ""
if let firstLine = request.split(separator: "\r\n").first {
let parts = firstLine.split(separator: " ")
if parts.count >= 2 { path = String(parts[1]) }
}
lastPath = path
let body = #"{"ok": true, "room": "test", "desired": "stopped"}"#
let response = """
HTTP/1.1 200 OK\r
Content-Type: application/json\r
Content-Length: \(body.utf8.count)\r
Connection: close\r
\(body)
"""
_ = response.withCString { cstr in
write(clientFD, cstr, response.utf8.count)
}
}
}
@MainActor
struct MediaAPITests {
@Test("successful JSON object response does not throw decodeFailed")
func successfulObjectResponse() async throws {
let stub = MediaAPIStubServer()
try stub.start()
defer { stub.stop() }
// Bypass the https-only parse path: build the config directly.
let base = stub.url!
let config = ServerConfig(baseURL: base, livekitURL: base)
let api = MediaAPI(config: config)
try await api.control(room: "test", action: "pause")
try await api.report(room: "test", state: "paused", trackID: nil, position: 0, duration: nil)
try await api.play(room: "test", videoID: nil, title: nil, channel: nil, trackID: nil, startPaused: nil)
#expect(stub.lastRequestPath == "/media/api/play")
}
}
-89
View File
@@ -1,89 +0,0 @@
import Foundation
import Testing
@testable import Voice
private final class PlaybackProtocol: URLProtocol, @unchecked Sendable {
static let lock = NSLock()
nonisolated(unsafe) static var status = 200
nonisolated(unsafe) static var requests: [URLRequest] = []
override class func canInit(with request: URLRequest) -> Bool { true }
override class func canonicalRequest(for request: URLRequest) -> URLRequest { request }
override func startLoading() {
Self.lock.lock()
Self.requests.append(request)
let code = Self.status
Self.lock.unlock()
let response = HTTPURLResponse(url: request.url!, statusCode: code, httpVersion: nil, headerFields: ["Content-Type": "application/json"])!
client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed)
client?.urlProtocol(self, didLoad: Data("{\"ok\":true}".utf8))
client?.urlProtocolDidFinishLoading(self)
}
override func stopLoading() {}
}
@Suite(.serialized)
@MainActor struct MediaPlaybackTests {
private func controller(status: Int = 200) -> MediaPlayerController {
PlaybackProtocol.lock.withLock {
PlaybackProtocol.status = status
PlaybackProtocol.requests = []
}
let configuration = URLSessionConfiguration.ephemeral
configuration.protocolClasses = [PlaybackProtocol.self]
let url = URL(string: "https://media.test")!
let api = MediaAPI(config: ServerConfig(baseURL: url, livekitURL: url), session: URLSession(configuration: configuration))
return MediaPlayerController(api: api, settings: SettingsStore())
}
@Test("standalone playback sends stop to the default server room")
func standaloneStop() async throws {
let player = controller()
await player.stop()
let calls = PlaybackProtocol.lock.withLock { PlaybackProtocol.requests }
let stop = try #require(calls.first { $0.url?.path == "/media/api/control" })
// URLSession may expose the encoded body as an InputStream to URLProtocol.
let body: Data
if let data = stop.httpBody { body = data }
else {
let stream = try #require(stop.httpBodyStream)
stream.open()
defer { stream.close() }
var data = Data()
var buffer = [UInt8](repeating: 0, count: 1024)
while stream.hasBytesAvailable {
let count = stream.read(&buffer, maxLength: buffer.count)
if count <= 0 { break }
data.append(contentsOf: buffer.prefix(count))
}
body = data
}
let json = try #require(JSONSerialization.jsonObject(with: body) as? [String: Any])
#expect(json["room"] as? String == "default")
#expect(json["action"] as? String == "stop")
}
@Test("failed play requests are visible and a successful retry clears the error")
func failureAndRetry() async {
let player = controller(status: 503)
await player.playTrack(id: "song")
#expect(player.playbackError != nil)
PlaybackProtocol.lock.withLock { PlaybackProtocol.status = 200 }
await player.playTrack(id: "song")
#expect(player.playbackError == nil)
}
@Test("metadata refreshes when a download becomes ready")
func readyMetadata() {
let player = controller()
func state(_ status: String) -> MediaState {
MediaState(room: "default", track: MediaTrack(id: "song", kind: "youtube", videoID: nil, title: status, channel: nil, duration: 180, status: status, error: nil, audioURL: nil, artURL: nil), desired: "paused", reported: nil, seekHint: nil, choices: [], pending: [], version: 1)
}
player.reconcile(state("downloading"))
player.reconcile(state("ready"))
#expect(player.track?.status == "ready")
#expect(player.track?.title == "ready")
player.resetScope()
#expect(player.track == nil)
#expect(player.player.currentItem == nil)
}
}
-124
View File
@@ -1,124 +0,0 @@
import Foundation
import Testing
@testable import Voice
@MainActor
struct MediaReconcileTests {
private func makeController() -> MediaPlayerController {
let config = ServerConfig(
baseURL: URL(string: "http://127.0.0.1:9")!,
livekitURL: URL(string: "ws://127.0.0.1:9")!
)
let api = MediaAPI(config: config)
return MediaPlayerController(api: api, settings: SettingsStore())
}
private func makeState(
desired: String? = "stopped",
trackID: String? = nil,
audioURL: String? = nil
) -> MediaState {
let track: MediaTrack? = trackID != nil ? MediaTrack(
id: trackID!,
kind: nil,
videoID: nil,
title: nil,
channel: nil,
duration: nil,
status: nil,
error: nil,
audioURL: audioURL,
artURL: nil
) : nil
return MediaState(
room: "default",
track: track,
desired: desired,
reported: nil,
seekHint: nil,
choices: [],
pending: [],
version: 1
)
}
@Test("reconcile updates published desired to playing")
func reconcileDesiredPlaying() {
let controller = makeController()
#expect(controller.desired == "stopped")
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
}
@Test("reconcile updates published desired to paused")
func reconcileDesiredPaused() {
let controller = makeController()
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
controller.reconcile(makeState(desired: "paused"))
#expect(controller.desired == "paused")
}
@Test("reconcile updates published desired to stopped")
func reconcileDesiredStopped() {
let controller = makeController()
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
controller.reconcile(makeState(desired: "stopped"))
#expect(controller.desired == "stopped")
}
@Test("reconcile with nil desired defaults to stopped")
func reconcileNilDesiredDefaultsToStopped() {
let controller = makeController()
controller.reconcile(makeState(desired: "playing"))
#expect(controller.desired == "playing")
controller.reconcile(makeState(desired: nil))
#expect(controller.desired == "stopped")
}
@Test("late audio: desired=playing before audio_url, then audio_url arrives")
func lateAudioReconciliation() {
let controller = makeController()
// Phase 1: server says playing but the track has no audio_url yet
// (still downloading). The source is not wired, so play cannot start.
let stateNoURL = makeState(desired: "playing", trackID: "track-1", audioURL: nil)
controller.reconcile(stateNoURL)
#expect(controller.desired == "playing")
#expect(controller.track?.id == "track-1")
// Player rate should still be zero because no source was wired.
#expect(controller.player.rate.isZero)
// Phase 2: the same track now has an audio_url. The reconcile path
// wires the source (wireSource) and then applyDesiredState retries
// play because desired is still "playing" and rate is still zero.
let stateWithURL = makeState(desired: "playing", trackID: "track-1", audioURL: "/stream/track-1.mp3")
controller.reconcile(stateWithURL)
#expect(controller.desired == "playing")
// The source should now be wired (loadedAudioPath is set internally).
// We can't directly observe loadedAudioPath, but we can verify the
// player has a current item after wireSource ran.
#expect(controller.player.currentItem != nil)
}
@Test("startDefaultPolling restarts after endConversation keeps scope")
func pollingRestartsAfterEndConversation() async {
let controller = makeController()
controller.startDefaultPolling()
await controller.endConversation()
controller.startDefaultPolling()
try? await Task.sleep(for: .milliseconds(50))
}
@Test("togglePlayPause without a loaded track is a no-op")
func toggleWithoutTrack() async {
let controller = makeController()
await controller.togglePlayPause()
#expect(controller.track == nil)
}
}
-126
View File
@@ -1,126 +0,0 @@
import Foundation
import Testing
@testable import Voice
/// Focused tests for PinnedMediaResourceLoader.parseTotalLength, covering
/// 200, 206 with known total, 206 with unknown total (*), and malformed
/// Content-Range headers (Finding 1 / 11).
struct PinnedMediaResourceLoaderTests {
private func makeResponse(status: Int, contentRange: String? = nil, expectedLength: Int64 = -1) -> HTTPURLResponse {
var headers: [String: String] = [:]
if let contentRange {
headers["Content-Range"] = contentRange
}
return HTTPURLResponse(
url: URL(string: "https://example.com/media/test.mp3")!,
statusCode: status,
httpVersion: "HTTP/1.1",
headerFields: headers
)!
}
@Test("206 with known total parses Content-Range total")
func parse206KnownTotal() {
let response = makeResponse(status: 206, contentRange: "bytes 0-1023/4096", expectedLength: 1024)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == 4096)
}
@Test("206 with unknown total (*) returns nil")
func parse206UnknownTotal() {
let response = makeResponse(status: 206, contentRange: "bytes 0-1023/*", expectedLength: 1024)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("200 response returns nil (expectedContentLength is already the full size)")
func parse200ReturnsNil() {
let response = makeResponse(status: 200, expectedLength: 8192)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("206 without Content-Range header returns nil")
func parse206NoContentRange() {
let response = makeResponse(status: 206, expectedLength: 512)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("malformed Content-Range returns nil")
func parseMalformedContentRange() {
let response = makeResponse(status: 206, contentRange: "garbage", expectedLength: 100)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil)
}
@Test("206 with large total parses correctly")
func parseLargeTotal() {
let response = makeResponse(status: 206, contentRange: "bytes 1048576-2097151/1073741824", expectedLength: 1048576)
#expect(PinnedMediaResourceLoader.parseTotalLength(response) == 1_073_741_824)
}
@Test("streamURL constructs endpoint without downloading")
func streamURLConstruction() {
let config = ServerConfig(
baseURL: URL(string: "https://192.168.86.2:9178")!,
livekitURL: URL(string: "ws://192.168.86.2:7880")!
)
let api = MediaAPI(config: config)
let url = api.streamURL(trackID: "track-123")
// streamURL builds under the /media/api base (MediaAPI.baseURL).
#expect(url.absoluteString == "https://192.168.86.2:9178/media/api/stream/track-123")
}
// MARK: Rewrite / Resolve round-trip (Item 7)
@Test("rewrite maps https URL to pinned-media scheme and resolve inverts it")
func rewriteResolveRoundTrip() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let original = URL(string: "https://192.168.86.2:9178/media/track-123.mp3")!
let pinned = loader.rewrite(original)
#expect(pinned.scheme == "pinned-media")
#expect(pinned.absoluteString == "pinned-media://192.168.86.2:9178/media/track-123.mp3")
let resolved = loader.resolve(pinned)
#expect(resolved.absoluteString == original.absoluteString)
}
@Test("rewrite maps path-only URL using base host and port")
func rewritePathOnly() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let pathURL = URL(string: "/media/track-456.mp3")!
let pinned = loader.rewrite(pathURL)
#expect(pinned.scheme == "pinned-media")
#expect(pinned.absoluteString == "pinned-media://192.168.86.2:9178/media/track-456.mp3")
}
@Test("resolve leaves non-pinned URLs unchanged")
func resolveNonPinned() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let https = URL(string: "https://example.com/foo.mp3")!
#expect(loader.resolve(https) == https)
}
// MARK: Asset lifecycle (Item 7)
@Test("assetWithPinnedURL creates asset and removeLoader clears mapping")
func assetLifecycle() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let pinned = loader.rewrite(URL(string: "https://192.168.86.2:9178/media/test.mp3")!)
let asset = loader.assetWithPinnedURL(pinned)
// The asset should have a valid URL matching the pinned scheme.
#expect(asset.url.scheme == PinnedMediaResourceLoader.scheme)
// Remove the loader mapping — subsequent calls should not crash.
loader.removeLoader(asset)
}
@Test("removeLoader is idempotent (safe to call twice)")
func removeLoaderIdempotent() {
let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!)
let pinned = loader.rewrite(URL(string: "https://192.168.86.2:9178/media/test.mp3")!)
let asset = loader.assetWithPinnedURL(pinned)
loader.removeLoader(asset)
// Second call should be a no-op, not a crash.
loader.removeLoader(asset)
}
}
+2 -35
View File
@@ -1,9 +1,8 @@
import XCTest import XCTest
/// Opt-in tests against the actual NAS deployment. No injected replies or media. /// Opt-in tests against the actual NAS deployment. No injected replies.
final class VoiceLiveTests: XCTestCase { final class VoiceLiveTests: XCTestCase {
private var app: XCUIApplication! private var app: XCUIApplication!
private var startedMusic = false
override func record(_ issue: XCTIssue) { override func record(_ issue: XCTIssue) {
if let app, app.state == .runningForeground { if let app, app.state == .runningForeground {
@@ -17,20 +16,13 @@ final class VoiceLiveTests: XCTestCase {
} }
override func tearDownWithError() throws { override func tearDownWithError() throws {
defer { app?.terminate() } app?.terminate()
guard let app, app.state == .runningForeground else { return }
// Stop only playback initiated by this test, including assertion failures.
// Do not leave the shared default room playing between test runs.
if startedMusic, app.buttons["Stop playback"].exists {
app.buttons["Stop playback"].tap()
}
} }
override func setUpWithError() throws { override func setUpWithError() throws {
try XCTSkipUnless(ProcessInfo.processInfo.environment["HOPE_RUN_LIVE_TESTS"] == "1", "Requires the live Hope server") try XCTSkipUnless(ProcessInfo.processInfo.environment["HOPE_RUN_LIVE_TESTS"] == "1", "Requires the live Hope server")
continueAfterFailure = false continueAfterFailure = false
app = XCUIApplication() app = XCUIApplication()
startedMusic = false
app.launchArguments = ["--ui-testing", "--reset-ui-testing", "--live-testing"] app.launchArguments = ["--ui-testing", "--reset-ui-testing", "--live-testing"]
app.launch() app.launch()
} }
@@ -63,31 +55,6 @@ final class VoiceLiveTests: XCTestCase {
XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20)) XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20))
} }
func testLiveLibraryPlayback() {
app.tabBars.buttons["Music"].tap()
let play = app.buttons.matching(NSPredicate(format: "label BEGINSWITH 'Play Sheryl Crow'")).firstMatch
XCTAssertTrue(play.waitForExistence(timeout: 20), "Existing library must load from NAS")
startedMusic = true
play.tap()
XCTAssertTrue(app.buttons["Pause"].waitForExistence(timeout: 30))
// Elapsed media time proves AVPlayer decoded and advanced, beyond a server desired-state label.
let position = app.sliders["Playback position"]
XCTAssertTrue(position.waitForExistence(timeout: 10))
let advanced = NSPredicate { _, _ in
let value = position.value as? String ?? "0%"
return value != "0%" && value != "0" && !value.isEmpty
}
expectation(for: advanced, evaluatedWith: position)
waitForExpectations(timeout: 35)
app.buttons["Pause"].tap()
XCTAssertTrue(app.buttons["Play"].waitForExistence(timeout: 10))
app.buttons["Seek forward 10 seconds"].tap()
capture("Live music playback")
app.buttons["Stop playback"].tap()
XCTAssertFalse(app.buttons["Pause"].exists)
startedMusic = false
}
private func capture(_ name: String) { private func capture(_ name: String) {
let attachment = XCTAttachment(screenshot: app.screenshot()) let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = name attachment.name = name
-23
View File
@@ -48,29 +48,6 @@ final class VoiceUITests: XCTestCase {
XCTAssertEqual(app.textFields["Server URL"].value as? String, "https://127.0.0.1:1") XCTAssertEqual(app.textFields["Server URL"].value as? String, "https://127.0.0.1:1")
} }
func testMusicSearchAndStationValidation() {
app.launch()
app.tabBars.buttons["Music"].tap()
XCTAssertTrue(app.buttons["Add radio station"].waitForExistence(timeout: 5))
XCTAssertTrue(app.staticTexts["Music server unavailable"].waitForExistence(timeout: 10))
app.segmentedControls.buttons["Search"].tap()
let query = app.textFields["Song, artist, lyrics…"]
query.tap()
query.typeText("Piano")
app.buttons["music.search"].tap()
XCTAssertTrue(app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'connect to the server'")).firstMatch.waitForExistence(timeout: 15))
app.buttons["Add radio station"].tap()
let url = app.textFields["Audio stream URL"]
XCTAssertTrue(url.waitForExistence(timeout: 5))
url.tap()
url.typeText("not-a-url")
app.buttons["Add station"].tap()
XCTAssertTrue(app.staticTexts["Enter a complete http:// or https:// audio stream address."].exists)
app.buttons["Cancel"].tap()
XCTAssertTrue(app.buttons["Add radio station"].exists)
capture("Music search")
}
func testTranscriptClearConfirmationAndActivityDetails() { func testTranscriptClearConfirmationAndActivityDetails() {
app.launchArguments.append("--ui-fixtures") app.launchArguments.append("--ui-fixtures")
app.launch() app.launch()
-48
View File
@@ -17,27 +17,19 @@
64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */ = {isa = PBXBuildFile; fileRef = A0C23305A6C55A83F8617D9D /* ConversationState.swift */; }; 64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */ = {isa = PBXBuildFile; fileRef = A0C23305A6C55A83F8617D9D /* ConversationState.swift */; };
70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */; }; 70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */; };
720C26EC163AE46713B02DD8 /* TLSPinning.swift in Sources */ = {isa = PBXBuildFile; fileRef = D9B40076934C28BA4B92D808 /* TLSPinning.swift */; }; 720C26EC163AE46713B02DD8 /* TLSPinning.swift in Sources */ = {isa = PBXBuildFile; fileRef = D9B40076934C28BA4B92D808 /* TLSPinning.swift */; };
72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */; };
73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */; };
89B47754E5C39418BFD839E6 /* TranscriptHistory.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68A45228EF59186CFC219F2E /* TranscriptHistory.swift */; }; 89B47754E5C39418BFD839E6 /* TranscriptHistory.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68A45228EF59186CFC219F2E /* TranscriptHistory.swift */; };
8DE1363CC0119B5AF9E4E117 /* ConnectionTestView.swift in Sources */ = {isa = PBXBuildFile; fileRef = AC4B9073EE8EAB51F8C55760 /* ConnectionTestView.swift */; }; 8DE1363CC0119B5AF9E4E117 /* ConnectionTestView.swift in Sources */ = {isa = PBXBuildFile; fileRef = AC4B9073EE8EAB51F8C55760 /* ConnectionTestView.swift */; };
943BCE971A01F220FAE9E250 /* TaskPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E85BACA267ADD462306C9920 /* TaskPanelView.swift */; }; 943BCE971A01F220FAE9E250 /* TaskPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E85BACA267ADD462306C9920 /* TaskPanelView.swift */; };
98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 48F76881B4AE51DEE74511CC /* SettingsView.swift */; }; 98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 48F76881B4AE51DEE74511CC /* SettingsView.swift */; };
A5868238CF4958D6D0C5908E /* TaskStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0B60FBBB9C47BB9ADACA7CA /* TaskStore.swift */; }; A5868238CF4958D6D0C5908E /* TaskStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0B60FBBB9C47BB9ADACA7CA /* TaskStore.swift */; };
A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */; };
A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */; }; A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */; };
AF719B1668ADA929AA0237BF /* MediaPlaybackTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */; };
B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */; };
BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */; }; BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */; };
C00D1878C2AB3D8B6F9562B6 /* AgentMonitor.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */; }; C00D1878C2AB3D8B6F9562B6 /* AgentMonitor.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */; };
C956D0FDFAC43925979E4077 /* MediaAPI.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9191AD60FF96E44F191864E3 /* MediaAPI.swift */; };
CDE38FE9679472D770C601E6 /* VoiceSession.swift in Sources */ = {isa = PBXBuildFile; fileRef = EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */; }; CDE38FE9679472D770C601E6 /* VoiceSession.swift in Sources */ = {isa = PBXBuildFile; fileRef = EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */; };
D18AD347DFF72AB6361165BA /* TokenClient.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9681E7410B684D2D38752A70 /* TokenClient.swift */; }; D18AD347DFF72AB6361165BA /* TokenClient.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9681E7410B684D2D38752A70 /* TokenClient.swift */; };
D2E343C8B8657316A993698E /* TranscriptHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */; }; D2E343C8B8657316A993698E /* TranscriptHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */; };
D9A0059E342955A06A021D6C /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = A6BAD55715F383C0DB5F306F /* LiveKit */; }; D9A0059E342955A06A021D6C /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = A6BAD55715F383C0DB5F306F /* LiveKit */; };
DE1F24919E906EE13319C7A3 /* MediaPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */; };
EE0C5762EC97B123A1880FFB /* Voices.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2E5DF207EDB787F63CB3A64D /* Voices.swift */; }; EE0C5762EC97B123A1880FFB /* Voices.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2E5DF207EDB787F63CB3A64D /* Voices.swift */; };
F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 10C194656F19E89DFD656A91 /* MediaAPITests.swift */; };
FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */; }; FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */; };
/* End PBXBuildFile section */ /* End PBXBuildFile section */
@@ -59,26 +51,19 @@
/* End PBXContainerItemProxy section */ /* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */ /* Begin PBXFileReference section */
09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaReconcileTests.swift; sourceTree = "<group>"; };
0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AgentMonitor.swift; sourceTree = "<group>"; }; 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AgentMonitor.swift; sourceTree = "<group>"; };
10C194656F19E89DFD656A91 /* MediaAPITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPITests.swift; sourceTree = "<group>"; };
20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerReplacementCoordinatorTests.swift; sourceTree = "<group>"; }; 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerReplacementCoordinatorTests.swift; sourceTree = "<group>"; };
2A4C3CE2CA6FF828452803D1 /* Voice.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = Voice.app; sourceTree = BUILT_PRODUCTS_DIR; }; 2A4C3CE2CA6FF828452803D1 /* Voice.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = Voice.app; sourceTree = BUILT_PRODUCTS_DIR; };
2E5DF207EDB787F63CB3A64D /* Voices.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Voices.swift; sourceTree = "<group>"; }; 2E5DF207EDB787F63CB3A64D /* Voices.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Voices.swift; sourceTree = "<group>"; };
2FED8976B60FBC13BC980FCB /* VoiceApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceApp.swift; sourceTree = "<group>"; }; 2FED8976B60FBC13BC980FCB /* VoiceApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceApp.swift; sourceTree = "<group>"; };
3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlayerController.swift; sourceTree = "<group>"; };
3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationView.swift; sourceTree = "<group>"; }; 3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationView.swift; sourceTree = "<group>"; };
48F76881B4AE51DEE74511CC /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = "<group>"; }; 48F76881B4AE51DEE74511CC /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = "<group>"; };
5759EB570DB87BBC121D946D /* VoiceUITests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; 5759EB570DB87BBC121D946D /* VoiceUITests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceLiveTests.swift; sourceTree = "<group>"; }; 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceLiveTests.swift; sourceTree = "<group>"; };
61C71C48230C683D199D6F4B /* ControlChannel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ControlChannel.swift; sourceTree = "<group>"; }; 61C71C48230C683D199D6F4B /* ControlChannel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ControlChannel.swift; sourceTree = "<group>"; };
68A45228EF59186CFC219F2E /* TranscriptHistory.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHistory.swift; sourceTree = "<group>"; }; 68A45228EF59186CFC219F2E /* TranscriptHistory.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHistory.swift; sourceTree = "<group>"; };
68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPanelView.swift; sourceTree = "<group>"; };
6D0D153557BB1C756ED04C61 /* SettingsStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsStore.swift; sourceTree = "<group>"; }; 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsStore.swift; sourceTree = "<group>"; };
6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoader.swift; sourceTree = "<group>"; };
8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceUITests.swift; sourceTree = "<group>"; }; 8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceUITests.swift; sourceTree = "<group>"; };
8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoaderTests.swift; sourceTree = "<group>"; };
9191AD60FF96E44F191864E3 /* MediaAPI.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPI.swift; sourceTree = "<group>"; };
9681E7410B684D2D38752A70 /* TokenClient.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TokenClient.swift; sourceTree = "<group>"; }; 9681E7410B684D2D38752A70 /* TokenClient.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TokenClient.swift; sourceTree = "<group>"; };
A0C23305A6C55A83F8617D9D /* ConversationState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationState.swift; sourceTree = "<group>"; }; A0C23305A6C55A83F8617D9D /* ConversationState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationState.swift; sourceTree = "<group>"; };
A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = OrbView.swift; sourceTree = "<group>"; }; A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = OrbView.swift; sourceTree = "<group>"; };
@@ -88,7 +73,6 @@
C22975C69240AACD3418091F /* VoiceTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; C22975C69240AACD3418091F /* VoiceTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHandler.swift; sourceTree = "<group>"; }; D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHandler.swift; sourceTree = "<group>"; };
D9B40076934C28BA4B92D808 /* TLSPinning.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TLSPinning.swift; sourceTree = "<group>"; }; D9B40076934C28BA4B92D808 /* TLSPinning.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TLSPinning.swift; sourceTree = "<group>"; };
E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlaybackTests.swift; sourceTree = "<group>"; };
E85BACA267ADD462306C9920 /* TaskPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TaskPanelView.swift; sourceTree = "<group>"; }; E85BACA267ADD462306C9920 /* TaskPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TaskPanelView.swift; sourceTree = "<group>"; };
EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceSession.swift; sourceTree = "<group>"; }; EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceSession.swift; sourceTree = "<group>"; };
F5683993A1DE677121FFA1F3 /* ServerConfig.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerConfig.swift; sourceTree = "<group>"; }; F5683993A1DE677121FFA1F3 /* ServerConfig.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerConfig.swift; sourceTree = "<group>"; };
@@ -107,14 +91,6 @@
/* End PBXFrameworksBuildPhase section */ /* End PBXFrameworksBuildPhase section */
/* Begin PBXGroup section */ /* Begin PBXGroup section */
0D2E974CBC286D7C886BA814 /* Media */ = {
isa = PBXGroup;
children = (
68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */,
);
path = Media;
sourceTree = "<group>";
};
0DF0591DF1D767F89099F62C /* Settings */ = { 0DF0591DF1D767F89099F62C /* Settings */ = {
isa = PBXGroup; isa = PBXGroup;
children = ( children = (
@@ -180,7 +156,6 @@
isa = PBXGroup; isa = PBXGroup;
children = ( children = (
84ADAF12A43A57888CA178A6 /* Conversation */, 84ADAF12A43A57888CA178A6 /* Conversation */,
0D2E974CBC286D7C886BA814 /* Media */,
0DF0591DF1D767F89099F62C /* Settings */, 0DF0591DF1D767F89099F62C /* Settings */,
F168B25F8B3C6F36B6E88228 /* Tasks */, F168B25F8B3C6F36B6E88228 /* Tasks */,
); );
@@ -197,16 +172,6 @@
); );
sourceTree = "<group>"; sourceTree = "<group>";
}; };
C12426487989358DE21A5426 /* Media */ = {
isa = PBXGroup;
children = (
9191AD60FF96E44F191864E3 /* MediaAPI.swift */,
3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */,
6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */,
);
path = Media;
sourceTree = "<group>";
};
CE49A38E40AAF659A9281B85 /* App */ = { CE49A38E40AAF659A9281B85 /* App */ = {
isa = PBXGroup; isa = PBXGroup;
children = ( children = (
@@ -221,10 +186,6 @@
isa = PBXGroup; isa = PBXGroup;
children = ( children = (
BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */, BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */,
10C194656F19E89DFD656A91 /* MediaAPITests.swift */,
E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */,
09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */,
8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */,
20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */, 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */,
FAEF89B2DEEE68F7168A8014 /* TranscriptDedupeTests.swift */, FAEF89B2DEEE68F7168A8014 /* TranscriptDedupeTests.swift */,
); );
@@ -245,7 +206,6 @@
CE49A38E40AAF659A9281B85 /* App */, CE49A38E40AAF659A9281B85 /* App */,
8DA8BEF2EE3E0F4181A94EB9 /* Features */, 8DA8BEF2EE3E0F4181A94EB9 /* Features */,
2820A418F921E18A2FA7E4FB /* LiveKit */, 2820A418F921E18A2FA7E4FB /* LiveKit */,
C12426487989358DE21A5426 /* Media */,
6B2AB34274D9B37E77178C6C /* Networking */, 6B2AB34274D9B37E77178C6C /* Networking */,
); );
path = Sources; path = Sources;
@@ -360,11 +320,7 @@
28659D009399CA9C4FE57B74 /* ControlChannel.swift in Sources */, 28659D009399CA9C4FE57B74 /* ControlChannel.swift in Sources */,
64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */, 64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */,
36DF44D9A7896F18A064847D /* ConversationView.swift in Sources */, 36DF44D9A7896F18A064847D /* ConversationView.swift in Sources */,
C956D0FDFAC43925979E4077 /* MediaAPI.swift in Sources */,
DE1F24919E906EE13319C7A3 /* MediaPanelView.swift in Sources */,
73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */,
3E01A0AF44FD3324FA72C43C /* OrbView.swift in Sources */, 3E01A0AF44FD3324FA72C43C /* OrbView.swift in Sources */,
A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */,
302BEC55CA9A10E49C307977 /* ServerConfig.swift in Sources */, 302BEC55CA9A10E49C307977 /* ServerConfig.swift in Sources */,
A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */, A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */,
98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */, 98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */,
@@ -394,10 +350,6 @@
buildActionMask = 2147483647; buildActionMask = 2147483647;
files = ( files = (
FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */, FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */,
F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */,
AF719B1668ADA929AA0237BF /* MediaPlaybackTests.swift in Sources */,
72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */,
B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */,
BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */, BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */,
1813A1CB0921D04FCAEC92D0 /* TranscriptDedupeTests.swift in Sources */, 1813A1CB0921D04FCAEC92D0 /* TranscriptDedupeTests.swift in Sources */,
); );
+6 -17
View File
@@ -1,7 +1,7 @@
# Device testing checklist # Device testing checklist
Simulator verification covered connect/disconnect, transcripts (text mode), media Simulator verification covered connect/disconnect, transcripts (text mode), tasks
reconciliation/UI, tasks panel, and the hardening pass. The items below need a **real panel, and the hardening pass. The items below need a **real
iPhone on the LAN** (`https://192.168.86.2:9178`) because mic input, WebRTC capture, iPhone on the LAN** (`https://192.168.86.2:9178`) because mic input, WebRTC capture,
and audio session behavior don't work in this simulator environment (CoreAudio `-4010`, and audio session behavior don't work in this simulator environment (CoreAudio `-4010`,
0 input channels). 0 input channels).
@@ -31,29 +31,18 @@ ensure the server cert pin is current (see README — re-export after any server
side button to lock the device. It only disables the automatic idle timeout. side button to lock the device. It only disables the automatic idle timeout.
- [ ] After hang-up, auto-lock returns (idle timer re-enabled). - [ ] After hang-up, auto-lock returns (idle timer re-enabled).
## 4. Audible media playback ## 4. Audio interruption (phone call / Siri)
- [ ] The deployed server now serves cached AAC for library tracks requested with
`?format=m4a`; verify actual speaker/headphone output on hardware.
- [ ] Play a library track: audio is audible, now-playing card shows art/title/status.
- [ ] Pause / ±10 s / stop / seek all work; agent mics off while playing
(`media_state` published), mics back on when stopped.
- [ ] End conversation while music was playing before → playback resumes.
## 5. Audio interruption (phone call / Siri)
- [ ] Mid-conversation, trigger an interruption (incoming call or "Hey Siri"). - [ ] Mid-conversation, trigger an interruption (incoming call or "Hey Siri").
- [ ] Mic is muted on interruption begin; conversation/media pause cleanly. - [ ] On end (with `shouldResume`), mic and conversation resume automatically.
- [ ] On end (with `shouldResume`), mic and media resume automatically.
- [ ] Media panel: AVPlayer pauses on interruption and resumes after.
## 6. Mic permission denied fallback ## 5. Mic permission denied fallback
- [ ] Deny microphone permission in iOS Settings for the app, relaunch. - [ ] Deny microphone permission in iOS Settings for the app, relaunch.
- [ ] Start → session reaches **ready** despite mic failure (non-fatal); orange error - [ ] Start → session reaches **ready** despite mic failure (non-fatal); orange error
banner shows the mic error; text mode remains fully usable. banner shows the mic error; text mode remains fully usable.
## 7. General device sanity ## 6. General device sanity
- [ ] Connect / hang-up / reconnect matrix (3×) — no crashes, state resets cleanly. - [ ] Connect / hang-up / reconnect matrix (3×) — no crashes, state resets cleanly.
- [ ] Change server URL in Settings while idle → session recreates without crash. - [ ] Change server URL in Settings while idle → session recreates without crash.
+32 -25
View File
@@ -1,13 +1,20 @@
# Project status and handoff # Project status and handoff
Last updated: 2026-09-09. Last updated: 2026-09-12.
## Current direction ## Current direction
Keep **Qwen 2.5**. The user explicitly canceled the earlier request to switch to Keep **Qwen 2.5**. The user explicitly canceled the earlier request to switch to
a ChatGPT subscription/device-token login. Do not revive that migration because a ChatGPT subscription/device-token login. Do not revive that migration because
an old saved goal mentions it. Current work is the native iOS UI/UX redesign, an old saved goal mentions it. Current work is the native iOS UI/UX redesign
restoring the music/AI server, and end-to-end qualification. and end-to-end qualification.
**2026-09-12: the embedded music player was removed** by user decision — the
feature is gone from the app (Media tab, `Sources/Media/`, `MediaPanelView`,
`MediaAPI`, `MediaPlayerController`, `PinnedMediaResourceLoader`, media tests,
and the `media_state` control publish). The backend media API still exists and
is untouched. The backend AAC compatibility patch (below) is now irrelevant to
the iOS app but remains deployed.
The UI and playback changes are implemented, and the NAS deployment was verified The UI and playback changes are implemented, and the NAS deployment was verified
on September 8. Full app qualification is **not complete**: physical-device tests on September 8. Full app qualification is **not complete**: physical-device tests
@@ -19,6 +26,8 @@ server health and device availability must be rechecked when resuming.
### Native interface and usability ### Native interface and usability
- Reorganized the app into native **Talk, Music, Activity** tabs, sharing one session. - Reorganized the app into native **Talk, Music, Activity** tabs, sharing one session.
(The Music tab was removed again on 2026-09-12 by user decision; the app now has
**Talk** and **Activity** tabs.)
- Replaced the large animated orb with a compact waveform/status presentation; - Replaced the large animated orb with a compact waveform/status presentation;
added a clearer welcome screen, Voice/Type selection before connecting, explicit added a clearer welcome screen, Voice/Type selection before connecting, explicit
start/end controls, microphone meter, and recoverable connection errors. start/end controls, microphone meter, and recoverable connection errors.
@@ -27,7 +36,7 @@ server health and device availability must be rechecked when resuming.
- Reworked Settings around voice, input mode, model, and connection; server changes - Reworked Settings around voice, input mode, model, and connection; server changes
require an explicit validated save. Voice selection persists. require an explicit validated save. Voice selection persists.
- Improved Music library/search navigation, now-playing controls, seek/volume, - Improved Music library/search navigation, now-playing controls, seek/volume,
empty/loading/error states, and validated station-entry sheet. empty/loading/error states, and validated station-entry sheet. (Removed 2026-09-12.)
- Improved Activity hierarchy and task detail presentation with an active-task badge. - Improved Activity hierarchy and task detail presentation with an active-task badge.
- Removed forced dark appearance; adopted adaptive system styling, Dynamic Type, - Removed forced dark appearance; adopted adaptive system styling, Dynamic Type,
and larger interaction targets, informed by the Impeccable native-iOS guidance. and larger interaction targets, informed by the Impeccable native-iOS guidance.
@@ -35,14 +44,11 @@ server health and device availability must be rechecked when resuming.
### Functional fixes ### Functional fixes
- Observed nested transcript/state/task objects so asynchronous updates render. - Observed nested transcript/state/task objects so asynchronous updates render.
- Preserved latest-wins server replacement; clear old media when switching servers. - Preserved latest-wins server replacement.
- Sent standalone media controls and reports to the default server room. (All media-related behaviors — media clearing on server switch, standalone media
- Refreshed track metadata as downloads become ready; exposed request/decoder errors. controls/reports, track metadata refresh, pinned-TLS artwork, cached-AAC requests,
- Loaded artwork through pinned TLS for the NAS self-signed certificate. UTI content-type fix, bounded-range media loader — were removed with the music
- Requested cached AAC (`?format=m4a`) for YouTube library tracks on iOS. player on 2026-09-12.)
- Corrected AVFoundation's content type to a UTI rather than an HTTP MIME string.
- Changed the pinned media loader to deliver 256 KB byte ranges instead of buffering
an entire song before delivering usable audio, addressing a reproduced timeout.
### Tests and tooling ### Tests and tooling
@@ -50,9 +56,10 @@ server health and device availability must be rechecked when resuming.
- Added the `VoiceUITests` target in `project.yml` and regenerated the Xcode project. - Added the `VoiceUITests` target in `project.yml` and regenerated the Xcode project.
- Added isolated DEBUG UI-test preferences and optional fixtures; normal application - Added isolated DEBUG UI-test preferences and optional fixtures; normal application
preferences are not reset by those UI-test launch arguments. preferences are not reset by those UI-test launch arguments.
- Added media-control/error/metadata tests and opt-in real-NAS UI tests. - Added media-control/error/metadata tests and opt-in real-NAS UI tests. (Media tests
were removed with the music player on 2026-09-12.)
- Live tests retain failure screenshots/UI hierarchy and attempt to stop music - Live tests retain failure screenshots/UI hierarchy and attempt to stop music
they initiated even when an assertion fails. they initiated even when an assertion fails. (Music playback test removed 2026-09-12.)
## Backend deployment and cross-repository dependency ## Backend deployment and cross-repository dependency
@@ -87,15 +94,14 @@ Do not copy credentials or `.env` contents into documentation or commits.
Project `Voice.xcodeproj`, scheme `Voice_iOS`, iPhone 17 Pro simulator. Project `Voice.xcodeproj`, scheme `Voice_iOS`, iPhone 17 Pro simulator.
- 68 unit tests passed; five offline UI tests passed (navigation/error recovery, - 49 unit tests passed (after the 2026-09-12 music-player removal; previously 68 with
settings persistence/validation, music validation, transcript/task UI, large text). media tests); four offline UI tests passed (navigation/error recovery,
settings persistence/validation, transcript/task UI, large text).
- Real NAS tests verified Qwen replies, background task completion, reconnect, - Real NAS tests verified Qwen replies, background task completion, reconnect,
library loading, advancing playback time, pause, forward-seek control and stop. library loading, advancing playback time, pause, forward-seek control and stop.
(Playback evidence predates the music-player removal; the feature no longer exists
in the app.)
- Qwen/task/reconnect passed three consecutive iterations. - Qwen/task/reconnect passed three consecutive iterations.
- Music initially failed intermittently. Logs captured an AVPlayer timeout while
buffering a whole 6 MB response. After bounded-range delivery, three consecutive
live playback iterations passed. The earlier isolated agent-join failure remains
unexplained; do not claim all intermittent connection issues are resolved.
- Backend integration: 16 passed, 7 skipped, 2 deselected. - Backend integration: 16 passed, 7 skipped, 2 deselected.
- Backend AAC regression: one passed (conversion, cache reuse, original preservation, - Backend AAC regression: one passed (conversion, cache reuse, original preservation,
and byte ranges); deployed range request returned HTTP 206. and byte ranges); deployed range request returned HTTP 206.
@@ -115,19 +121,20 @@ Artifacts live on this Mac under the XcodeBuildMCP workspace, not in Git.
2. **Build, sign, install and run on that iPhone.** No successful device install is 2. **Build, sign, install and run on that iPhone.** No successful device install is
recorded. Check Developer Mode, trust, signing team/profile, and LAN access. recorded. Check Developer Mode, trust, signing team/profile, and LAN access.
3. **Qualify real audio:** microphone capture, TTS audibility, speaker/headphones, 3. **Qualify real audio:** microphone capture, TTS audibility, speaker/headphones,
barge-in, media/voice handoff, Siri/phone interruptions, lock/background behavior, barge-in, Siri/phone interruptions, lock/background behavior,
microphone-permission-denied fallback, and network-loss recovery. Some checks microphone-permission-denied fallback, and network-loss recovery. Some checks
require the user's speech/listening or a real interruption; simulator state tests require the user's speech/listening or a real interruption; simulator state tests
do not prove them. Follow [DEVICE_TESTING.md](DEVICE_TESTING.md). do not prove them. Follow [DEVICE_TESTING.md](DEVICE_TESTING.md).
4. **Broader regression:** rerun the full suite after the range-loader change; test 4. **Broader regression:** rerun the full suite after the music-player removal;
longer playback, seek near EOF, slow network/cancellation and server replacement. test slow network/cancellation and server replacement. Exercise voice/model
Exercise new downloads, test-owned station add/play/remove, and voice/model selection against the live agent.
selection against the live agent. Do not delete the user's existing library song.
5. **Final visual/accessibility pass:** inspect all populated/error screens in light 5. **Final visual/accessibility pass:** inspect all populated/error screens in light
and dark mode, large text and small-device layouts; test VoiceOver and focus. and dark mode, large text and small-device layouts; test VoiceOver and focus.
The existing large-text navigation test is not a full accessibility audit. The existing large-text navigation test is not a full accessibility audit.
6. **Publish the backend compatibility source safely** in the sibling repository 6. **Publish the backend compatibility source safely** in the sibling repository
and ensure its normal deployment preserves mounts, certificates and Qwen settings. and ensure its normal deployment preserves mounts, certificates and Qwen settings.
(Lower priority now: the iOS app no longer consumes the media API, but the deployed
container still serves it for the web UI.)
7. Update this report with results and remaining limitations. Do not mark the whole 7. Update this report with results and remaining limitations. Do not mark the whole
app qualified until the outstanding end-to-end gates have evidence. app qualified until the outstanding end-to-end gates have evidence.