From ed329213deef2f83bd7fb3d6e68476b40419cd8a Mon Sep 17 00:00:00 2001 From: Shane Maynard Date: Sat, 12 Sep 2026 06:26:58 -0400 Subject: [PATCH] Remove embedded music player The music feature is retired: delete the Media tab, MediaAPI, MediaPlayerController, PinnedMediaResourceLoader, MediaPanelView, the media_state control publish, media scope handling in VoiceSession, the volume setting, and all media tests. App is now Talk + Activity tabs. Backend media API in hope-voice-api is untouched (still serves the web UI). --- README.md | 22 +- Sources/App/SettingsStore.swift | 5 - .../Conversation/ConversationView.swift | 14 +- Sources/Features/Media/MediaPanelView.swift | 610 ----------------- Sources/LiveKit/ControlChannel.swift | 7 +- Sources/LiveKit/VoiceSession.swift | 18 +- Sources/Media/MediaAPI.swift | 249 ------- Sources/Media/MediaPlayerController.swift | 633 ------------------ Sources/Media/PinnedMediaResourceLoader.swift | 238 ------- Tests/MediaAPITests.swift | 125 ---- Tests/MediaPlaybackTests.swift | 89 --- Tests/MediaReconcileTests.swift | 124 ---- Tests/PinnedMediaResourceLoaderTests.swift | 126 ---- UITests/VoiceLiveTests.swift | 37 +- UITests/VoiceUITests.swift | 23 - Voice.xcodeproj/project.pbxproj | 48 -- docs/DEVICE_TESTING.md | 23 +- docs/PROJECT_STATUS.md | 57 +- 18 files changed, 55 insertions(+), 2393 deletions(-) delete mode 100644 Sources/Features/Media/MediaPanelView.swift delete mode 100644 Sources/Media/MediaAPI.swift delete mode 100644 Sources/Media/MediaPlayerController.swift delete mode 100644 Sources/Media/PinnedMediaResourceLoader.swift delete mode 100644 Tests/MediaAPITests.swift delete mode 100644 Tests/MediaPlaybackTests.swift delete mode 100644 Tests/MediaReconcileTests.swift delete mode 100644 Tests/PinnedMediaResourceLoaderTests.swift diff --git a/README.md b/README.md index 7720fb6..da89d32 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,19 @@ # Voice — iOS client for Hope Voice Native SwiftUI iOS app (iOS 17+) that talks to the [Hope Voice](https://github.com/) agent: -voice/text conversation, transcript history, voice & model pickers, media player, and a -background-tasks panel. Uses `hope-voice-api`, including cached AAC conversion for iOS music playback. +voice/text conversation, transcript history, voice & model pickers, and a background-tasks +panel. Uses `hope-voice-api`. > **Status:** Core functionality is implemented and verified in simulator to the extent the -> environment allows. Full device qualification (mic capture, audible media playback, -> interruption behavior) remains pending. The native Talk/Music/Activity redesign is implemented — +> environment allows. Full device qualification (mic capture, interruption behavior) +> remains pending. The native Talk/Activity redesign is implemented — > see [PLAN.md](PLAN.md) and [docs/DEVICE_TESTING.md](docs/DEVICE_TESTING.md). ## Prerequisites - Xcode 16+ (Swift 6), iOS 17 simulator or a real iPhone on the LAN. - [xcodegen](https://github.com/yonaskolb/XcodeGen) (`brew install xcodegen`). -- The `voice` container running on Unraid (LiveKit + token server + media API). +- The `voice` container running on Unraid (LiveKit + token server). - Mic: the simulator mic is unreliable in this environment (CoreAudio `-4010`, 0 input channels) — voice mode needs a real device. If the simulator was wiped, grant permission first: `xcrun simctl privacy booted grant microphone dev.shane.voice`. @@ -33,7 +33,7 @@ To run in the simulator, use Xcode or `xcrun simctl` with the built app. Default base URL is `https://192.168.86.2:9178` (see `Sources/App/SettingsStore.swift`). Change it at runtime in **Settings → Server URL** (the app disconnects and recreates the -session when idle). The app derives LiveKit signaling and media endpoints from this base URL. +session when idle). The app derives the LiveKit signaling endpoint from this base URL. ### TLS cert pinning @@ -46,7 +46,7 @@ openssl s_client -connect 192.168.86.2:9178 /dev/null \ | openssl x509 -outform der | openssl dgst -sha256 ``` -Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token/media calls +Paste the new hash into `TLSPinning.pinnedCertSHA256Hex`. Until you do, token calls fail with a pin mismatch (not a network error). ## Features @@ -58,10 +58,6 @@ fail with a pin mismatch (not a network error). stripping, last-50 history persisted to UserDefaults, clear button. - **Voice & model pickers** — 13 Azure TTS voices; LLM models from the agent's `models` message. Both persist and sync on join (`set_voice`/`set_model`). -- **Media player** — now-playing card (art/title/channel/status), transport (play/pause, - ±10 s, stop), seek + volume sliders, search with "in library" tags, agent-offered choices, - library list (play/remove), add-stream form. AVPlayer-backed with a pinned-TLS Range proxy; - 1.5 s `/state` polling and full web-parity reconciliation/reporting. - **Activity tab** — live `task_event` updates (steps, result, error) with an active count badge on the tab. @@ -69,11 +65,9 @@ fail with a pin mismatch (not a network error). - **Simulator mic**: CoreAudio returns `-4010` (no usable input source) in this environment; voice capture must be verified on a real device. Text mode works in the simulator. -- **Music compatibility**: library WebM files are requested as cached AAC on iOS. - This requires the deployed backend compatibility patch; see [deployment](docs/DEPLOYMENT.md). - **Cert pin rotation**: see above — re-export the pin after each server cert regeneration. - **LiveKit signaling** uses plain `ws://192.168.86.2:7880` (not wss) because the Swift SDK - cannot trust the self-signed cert on its WebSocket; token + media still go over pinned TLS. + cannot trust the self-signed cert on its WebSocket; token calls still go over pinned TLS. ## Docs diff --git a/Sources/App/SettingsStore.swift b/Sources/App/SettingsStore.swift index b4f2ed1..92b221b 100644 --- a/Sources/App/SettingsStore.swift +++ b/Sources/App/SettingsStore.swift @@ -14,16 +14,12 @@ final class SettingsStore: ObservableObject { @Published var textMode: Bool { didSet { AppPreferences.store.set(textMode, forKey: Keys.textMode) } } - @Published var volume: Double { - didSet { AppPreferences.store.set(volume, forKey: Keys.volume) } - } private enum Keys { static let serverURL = "serverURL" static let voiceID = "voiceID" static let modelID = "modelID" static let textMode = "textMode" - static let volume = "volume" } static let defaultServerURL = "https://192.168.86.2:9178" @@ -47,7 +43,6 @@ final class SettingsStore: ObservableObject { voiceID = Voices.all.contains { $0.id == storedVoice } ? (storedVoice ?? "") : Voices.defaultID modelID = d.string(forKey: Keys.modelID) textMode = d.bool(forKey: Keys.textMode) - volume = d.object(forKey: Keys.volume) as? Double ?? 0.8 } var trimmedServerURL: String { diff --git a/Sources/Features/Conversation/ConversationView.swift b/Sources/Features/Conversation/ConversationView.swift index 423a5ae..d956a54 100644 --- a/Sources/Features/Conversation/ConversationView.swift +++ b/Sources/Features/Conversation/ConversationView.swift @@ -40,15 +40,8 @@ struct ConversationView: View { .tabItem { Label("Talk", systemImage: "waveform") } .tag(0) - NavigationStack { - MediaPanelView(playerController: session.mediaController, settings: settings) - .toolbar { settingsLink } - } - .tabItem { Label("Music", systemImage: "music.note") } - .tag(1) - ActivityTab(taskStore: session.taskStore, settings: settings, session: session) - .tag(2) + .tag(1) } .id(sessionID) .tint(HopeStyle.tint) @@ -79,9 +72,7 @@ struct ConversationView: View { // Coalesce rapid Apply actions; construct from the captured URL. while let next = pendingServerURL { pendingServerURL = nil - guard next != activeServerURL else { continue } await session.disconnect() - session.mediaController.resetScope() activeServerURL = next session = Self.makeSession(next, settings: settings) sessionID = UUID() @@ -157,13 +148,12 @@ private struct ConversationContent: View { Text("A little help.\nA good conversation.") .font(.largeTitle.weight(.semibold)) .fixedSize(horizontal: false, vertical: true) - Text("Talk things through, find something out, or put on a favorite song.") + Text("Talk things through, find something out, or get something done.") .font(.body).foregroundStyle(.secondary) } VStack(alignment: .leading, spacing: 16) { Label("Think out loud", systemImage: "bubble.left.and.bubble.right") Label("Find an answer", systemImage: "sparkle.magnifyingglass") - Label("Make time for music", systemImage: "music.note") } .font(.subheadline).foregroundStyle(.secondary) } diff --git a/Sources/Features/Media/MediaPanelView.swift b/Sources/Features/Media/MediaPanelView.swift deleted file mode 100644 index 09177ec..0000000 --- a/Sources/Features/Media/MediaPanelView.swift +++ /dev/null @@ -1,610 +0,0 @@ -import SwiftUI - -struct MediaPanelView: View { - @ObservedObject var playerController: MediaPlayerController - @ObservedObject var settings: SettingsStore - - @State private var searchQuery = "" - @State private var searchResults: [MediaSearchResult]? - @State private var isSearching = false - @State private var searchError: String? - @State private var libraryTracks: [MediaTrack]? - @State private var libraryError: String? - @State private var streamName = "" - @State private var streamURL = "" - @State private var streamMessage: String? - @State private var trackToDelete: MediaTrack? - @State private var browseMode = 0 - @State private var showAddStream = false - @State private var isAddingStream = false - @State private var isLoadingLibrary = false - @State private var seekPosition: Double = 0 - @State private var isSeeking = false - - var body: some View { - ScrollView { - LazyVStack(alignment: .leading, spacing: 24) { - if playerController.stateError != nil { - VStack(alignment: .leading, spacing: 8) { - Label("Music server unavailable", systemImage: "wifi.exclamationmark") - .font(.headline) - Text("Check that your Hope server is running and you’re connected to its network. We’ll reconnect automatically.") - .font(.subheadline).foregroundStyle(.secondary) - } - .frame(maxWidth: .infinity, alignment: .leading) - } - if playerController.track != nil { - VStack(spacing: 20) { - nowPlayingCard - seekSection - transportControls - volumeSection - } - .padding(16) - .background(Color(.secondarySystemGroupedBackground), in: RoundedRectangle(cornerRadius: 16)) - } - if let error = playerController.playbackError { - Label(error, systemImage: "exclamationmark.circle") - .font(.subheadline).foregroundStyle(.red) - } - pendingSection - choicesSection - Picker("Browse music", selection: $browseMode) { - Text("Library").tag(0) - Text("Search").tag(1) - }.pickerStyle(.segmented) - if browseMode == 0 { librarySection } - else { searchSection } - } - .padding() - } - .background(Color(.systemGroupedBackground)) - .navigationTitle("Music") - .toolbar { - ToolbarItem(placement: .topBarTrailing) { - Button { - streamMessage = nil - showAddStream = true - } label: { - Image(systemName: "plus").frame(minWidth: 44, minHeight: 44) - } - .accessibilityLabel("Add radio station") - } - } - .sheet(isPresented: $showAddStream) { - NavigationStack { - Form { addStreamSection } - .navigationTitle("Add a station") - .navigationBarTitleDisplayMode(.inline) - .toolbar { - ToolbarItem(placement: .cancellationAction) { - Button("Cancel") { showAddStream = false }.disabled(isAddingStream) - } - } - } - .interactiveDismissDisabled(isAddingStream) - } - .onAppear { - playerController.startDefaultPolling() - refreshLibrary() - } - .alert("Remove from library?", isPresented: Binding( - get: { trackToDelete != nil }, - set: { if !$0 { trackToDelete = nil } } - )) { - Button("Remove", role: .destructive) { - guard let track = trackToDelete else { return } - Task { await deleteTrack(track) } - } - Button("Cancel", role: .cancel) { trackToDelete = nil } - } message: { - Text(trackToDelete?.title ?? "") - } - } - - private var nowPlayingCard: some View { - HStack(spacing: 12) { - if let artURL = playerController.track?.artURL, let url = URL(string: absolute(artURL)) { - PinnedArtwork(url: url) - .frame(width: 72, height: 72) - .clipShape(RoundedRectangle(cornerRadius: 10)) - } else { - RoundedRectangle(cornerRadius: 10) - .fill(Color(.systemGray6)) - .frame(width: 72, height: 72) - .overlay( - Image(systemName: "music.note") - .font(.title2) - .foregroundStyle(.secondary) - ) - } - - VStack(alignment: .leading, spacing: 4) { - Text(title) - .font(.headline) - .lineLimit(2) - Text(subtitle) - .font(.subheadline) - .foregroundStyle(.secondary) - .lineLimit(2) - if playerController.track != nil { - Text(statusText) - .font(.caption) - .foregroundStyle(playerController.desired == "playing" ? Color.green : Color.secondary) - } - } - Spacer() - } - } - - private var transportControls: some View { - HStack(spacing: 24) { - Button(action: seekBackward) { - Image(systemName: "gobackward.10") - .font(.title3) - .frame(width: 44, height: 44) - } - .accessibilityLabel("Seek back 10 seconds") - .disabled(playerController.duration <= 0 || playerController.track?.kind == "stream") - - Button(action: togglePlayPause) { - Image(systemName: playerController.desired == "playing" && playerController.track != nil ? "pause.circle.fill" : "play.circle.fill") - .font(.largeTitle) - .frame(width: 56, height: 56) - .foregroundStyle(playerController.track == nil ? Color.secondary : Color.accentColor) - } - .accessibilityLabel(playerController.desired == "playing" && playerController.track != nil ? "Pause" : "Play") - .disabled(playerController.track == nil) - - Button(action: seekForward) { - Image(systemName: "goforward.10") - .font(.title3) - .frame(width: 44, height: 44) - } - .accessibilityLabel("Seek forward 10 seconds") - .disabled(playerController.duration <= 0 || playerController.track?.kind == "stream") - - Button(role: .destructive, action: stopPlayback) { - Image(systemName: "stop.circle") - .font(.title2) - .frame(width: 44, height: 44) - } - .accessibilityLabel("Stop playback") - .disabled(playerController.track == nil) - } - } - - private var seekSection: some View { - VStack(spacing: 4) { - if playerController.track?.kind == "stream" { - Label("Live radio", systemImage: "dot.radiowaves.left.and.right") - .font(.subheadline).foregroundStyle(.secondary) - } else if playerController.track != nil { - Slider( - value: Binding( - get: { min(isSeeking ? seekPosition : playerController.currentTime, max(playerController.duration, 1)) }, - set: { seekPosition = $0 } - ), - in: 0...max(playerController.duration, 1), - onEditingChanged: { editing in - isSeeking = editing - if !editing { playerController.seek(to: seekPosition) } - } - ) - .disabled(playerController.duration <= 0) - .accessibilityLabel("Playback position") - HStack { - Text(fmtTime(playerController.currentTime)) - .font(.caption.monospacedDigit()) - .foregroundStyle(.secondary) - Spacer() - Text(fmtTime(playerController.duration)) - .font(.caption.monospacedDigit()) - .foregroundStyle(.secondary) - } - } else { - Text("Nothing loaded") - .font(.subheadline) - .foregroundStyle(.secondary) - .frame(maxWidth: .infinity, alignment: .center) - .padding(.vertical, 8) - } - } - } - - private var volumeSection: some View { - HStack(spacing: 12) { - Image(systemName: "speaker.fill") - .foregroundStyle(.secondary) - Slider(value: $settings.volume, in: 0...1) - .accessibilityLabel("Music volume") - .onChange(of: settings.volume) { _, newValue in - playerController.setVolume(newValue) - } - Image(systemName: "speaker.wave.3.fill") - .foregroundStyle(.secondary) - } - } - - @ViewBuilder - private var pendingSection: some View { - if !playerController.pendingJobs.isEmpty { - VStack(alignment: .leading, spacing: 8) { - ForEach(playerController.pendingJobs, id: \.jobID) { job in - HStack { - Text(job.stage == "queued" ? "Queued…" : "Downloading…") - .font(.subheadline) - Spacer() - Text("\(Int((job.progress ?? 0)))%") - .font(.caption.monospacedDigit()) - .foregroundStyle(.secondary) - } - } - } - .padding(12) - .background(Color(.secondarySystemBackground), in: RoundedRectangle(cornerRadius: 14)) - } - } - - @ViewBuilder - private var choicesSection: some View { - if !playerController.choices.isEmpty { - VStack(alignment: .leading, spacing: 8) { - Text("Hope's suggestions") - .font(.headline) - ForEach(playerController.choices, id: \.videoID) { choice in - mediaRow( - title: choice.title ?? choice.videoID ?? "Unknown", - detail: rowDetail(channel: choice.channel, duration: choice.duration), - playEnabled: true - ) { - Task { - await playerController.playVideo( - videoID: choice.videoID, - title: choice.title, - channel: choice.channel - ) - } - } - } - } - } - } - - private var searchSection: some View { - VStack(alignment: .leading, spacing: 8) { - Text("Find your next favorite") - .font(.headline) - Text("Search songs and artists on YouTube Music.") - .font(.subheadline).foregroundStyle(.secondary) - HStack(spacing: 8) { - TextField("Song, artist, lyrics…", text: $searchQuery) - .textFieldStyle(.roundedBorder) - .submitLabel(.search) - .onSubmit(performSearch) - Button("Search", action: performSearch) - .accessibilityIdentifier("music.search") - .buttonStyle(.bordered) - .frame(minHeight: 44) - .disabled(searchQuery.trimmingCharacters(in: .whitespaces).isEmpty || isSearching) - } - if isSearching { - HStack(spacing: 8) { - ProgressView() - Text("Searching YouTube Music…") - .font(.subheadline) - .foregroundStyle(.secondary) - } - } else if let searchError { - Text(searchError) - .font(.subheadline) - .foregroundStyle(.red) - } else if let results = searchResults { - if results.isEmpty { - Text("No matches. Try a different song or artist.") - .font(.subheadline) - .foregroundStyle(.secondary) - } else { - ForEach(results, id: \.videoID) { result in - mediaRow( - title: result.title ?? result.videoID ?? "Unknown", - detail: rowDetail(channel: result.channel, duration: result.duration, inLibrary: result.inLibrary), - playEnabled: true - ) { - Task { - if let trackID = result.trackID { - await playerController.playTrack(id: trackID) - } else { - await playerController.playVideo( - videoID: result.videoID, - title: result.title, - channel: result.channel - ) - } - } - } - } - } - } - } - } - - private var librarySection: some View { - VStack(alignment: .leading, spacing: 8) { - HStack { - Text("Library") - .font(.headline) - Spacer() - Button(action: refreshLibrary) { - Image(systemName: "arrow.clockwise").frame(width: 44, height: 44) - } - .disabled(isLoadingLibrary) - .accessibilityLabel("Refresh library") - } - if let libraryError, playerController.stateError == nil { - Text(libraryError) - .font(.subheadline) - .foregroundStyle(.red) - } else if let tracks = libraryTracks { - if tracks.isEmpty { - ContentUnavailableView { - Label("A home for your favorites", systemImage: "music.note.list") - } description: { - Text("Search for a song to save it here, or add a radio station using the + button.") - } actions: { - Button("Find music") { browseMode = 1 }.buttonStyle(.bordered) - } - } else { - ForEach(tracks, id: \.id) { track in - HStack(spacing: 8) { - mediaRow( - title: track.title ?? track.id, - detail: rowDetail( - channel: track.channel ?? (track.kind == "stream" ? "Live stream" : ""), - duration: track.duration, - status: track.status - ), - playEnabled: track.status == "ready" - ) { - Task { await playerController.playTrack(id: track.id) } - } - Button(role: .destructive) { - trackToDelete = track - } label: { - Image(systemName: "xmark") - .font(.subheadline.weight(.semibold)) - .foregroundStyle(.red) - .frame(width: 44, height: 44) - } - .accessibilityLabel("Remove \(track.title ?? track.id)") - } - } - } - } else if isLoadingLibrary { - ProgressView("Loading your library…").frame(maxWidth: .infinity).padding(.vertical, 24) - } - } - } - - private var addStreamSection: some View { - Section { - TextField("Station name", text: $streamName) - .accessibilityLabel("Station name") - TextField("https://… audio stream URL", text: $streamURL) - .accessibilityLabel("Audio stream URL") - .keyboardType(.URL) - .autocorrectionDisabled() - .textContentType(.URL) - .textInputAutocapitalization(.never) - if let streamMessage { - Text(streamMessage) - .font(.subheadline) - .foregroundStyle(streamMessage.hasPrefix("Added") ? Color.green : Color.red) - } - Button(isAddingStream ? "Adding station…" : "Add station", action: addStream) - .disabled(streamURL.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty || isAddingStream) - } footer: { - Text("Use a direct MP3 or AAC audio stream address, rather than the station’s website.") - } - } - - private func mediaRow( - title: String, - detail: String, - playEnabled: Bool, - onPlay: @escaping () -> Void - ) -> some View { - HStack(spacing: 10) { - VStack(alignment: .leading, spacing: 2) { - Text(title) - .font(.subheadline.weight(.semibold)) - .lineLimit(2) - if !detail.isEmpty { - Text(detail) - .font(.caption) - .foregroundStyle(.secondary) - .lineLimit(2) - } - } - Spacer() - Button(action: onPlay) { - Image(systemName: "play.fill").frame(width: 44, height: 44) - } - .accessibilityLabel("Play \(title)") - .disabled(!playEnabled) - } - .padding(.vertical, 10) - } - - private func rowDetail(channel: String?, duration: Double?, status: String? = nil, inLibrary: Bool? = nil) -> String { - var parts: [String] = [] - if let channel, !channel.isEmpty { parts.append(channel) } - if let duration, duration > 0 { parts.append(fmtTime(duration)) } - if let status, status != "ready" { parts.append(status) } - if inLibrary == true { parts.append("in library") } - return parts.joined(separator: " · ") - } - - private func performSearch() { - let query = searchQuery.trimmingCharacters(in: .whitespacesAndNewlines) - guard !query.isEmpty, !isSearching else { return } - isSearching = true - searchError = nil - Task { - do { - let response = try await MediaAPI(config: serverConfig).search(room: "default", query: query, limit: 6) - searchResults = response.results - } catch { - searchError = error.localizedDescription - } - isSearching = false - } - } - - private func refreshLibrary() { - guard !isLoadingLibrary else { return } - isLoadingLibrary = true - Task { - defer { isLoadingLibrary = false } - do { - let library = try await MediaAPI(config: serverConfig).library() - libraryTracks = library.tracks - libraryError = nil - } catch { - libraryError = error.localizedDescription - } - } - } - - private func addStream() { - let url = streamURL.trimmingCharacters(in: .whitespacesAndNewlines) - guard !isAddingStream else { return } - guard let parsed = URL(string: url), ["https", "http"].contains(parsed.scheme?.lowercased() ?? ""), parsed.host != nil else { - streamMessage = "Enter a complete http:// or https:// audio stream address." - return - } - isAddingStream = true - let name = streamName.trimmingCharacters(in: .whitespaces).isEmpty ? url : streamName.trimmingCharacters(in: .whitespaces) - Task { - defer { isAddingStream = false } - do { - try await MediaAPI(config: serverConfig).addStream(room: "default", name: name, url: url) - streamMessage = "Added \(name)" - streamName = "" - streamURL = "" - showAddStream = false - refreshLibrary() - } catch { - streamMessage = "Could not add stream: \(error.localizedDescription)" - } - } - } - - private func deleteTrack(_ track: MediaTrack) async { - do { - try await MediaAPI(config: serverConfig).deleteTrack(id: track.id) - if playerController.track?.id == track.id { - Task { await playerController.stop() } - } - refreshLibrary() - } catch { - libraryError = error.localizedDescription - } - } - - private var serverConfig: ServerConfig { - // Use the configured URL directly. If it's invalid, fall back to the - // documented default (media operations are less critical than the main - // session, and the Settings view will show a validation error). - if case .success(let config) = ServerConfig.parse(settings.trimmedServerURL) { - return config - } - let fallback = ServerConfig.parse("https://192.168.86.2:9178") - if case .success(let config) = fallback { - return config - } - fatalError("Unreachable: default server URL is valid") - } - - private var title: String { - playerController.track?.title ?? "Nothing loaded" - } - - private var subtitle: String { - guard let track = playerController.track else { return "" } - if track.status == "failed" { - return "Download failed — pick another result or try again." - } - if track.status == "downloading" { - return track.channel.map { "\($0) · downloading" } ?? "Downloading…" - } - return track.channel ?? (track.kind == "stream" ? "Live stream" : "") - } - - private var statusText: String { - switch playerController.desired { - case "playing": return "Playing" - case "paused": return "Paused" - default: return "Stopped" - } - } - - private func absolute(_ path: String) -> String { - if path.hasPrefix("http") { return path } - let base = serverConfig.baseURL.absoluteString - let suffix = path.hasPrefix("/") ? path : "/" + path - return base + suffix - } - - private func togglePlayPause() { - Task { await playerController.togglePlayPause() } - } - - private func seekBackward() { - Task { await playerController.seek(by: -10) } - } - - private func seekForward() { - Task { await playerController.seek(by: 10) } - } - - private func stopPlayback() { - Task { await playerController.stop() } - } - - private func fmtTime(_ sec: Double) -> String { - guard sec.isFinite, sec >= 0 else { return "0:00" } - let total = Int(sec) - let s = total % 60 - let m = (total / 60) % 60 - let h = total / 3600 - return h > 0 ? "\(h):\(String(format: "%02d", m)):\(String(format: "%02d", s))" : "\(m):\(String(format: "%02d", s))" - } -} - -/// Artwork uses the same trust policy as token and media requests. -private struct PinnedArtwork: View { - let url: URL - @State private var artwork: UIImage? - - var body: some View { - Group { - if let artwork { Image(uiImage: artwork).resizable().scaledToFill() } - else { - Image(systemName: "music.note").font(.title2).foregroundStyle(.secondary) - .frame(maxWidth: .infinity, maxHeight: .infinity) - .background(Color(.tertiarySystemFill)) - } - } - .accessibilityHidden(true) - .task(id: url) { - artwork = nil - let session = TLSPinning.session() - defer { session.invalidateAndCancel() } - guard let (data, response) = try? await session.data(from: url), - (response as? HTTPURLResponse)?.statusCode == 200, - !Task.isCancelled else { return } - artwork = UIImage(data: data) - } - } -} diff --git a/Sources/LiveKit/ControlChannel.swift b/Sources/LiveKit/ControlChannel.swift index f10bd19..6b54aee 100644 --- a/Sources/LiveKit/ControlChannel.swift +++ b/Sources/LiveKit/ControlChannel.swift @@ -2,7 +2,7 @@ import Foundation import LiveKit /// Outbound helpers for agent control topics and inbound parsing of -/// voice-control / conversation-control / media-control data messages. +/// voice-control / conversation-control data messages. @MainActor final class ControlChannel: NSObject, RoomDelegate { @@ -109,11 +109,6 @@ final class ControlChannel: NSObject, RoomDelegate { try await publish(["type": "set_mode", "mode": mode], topic: "conversation-control", room: room) } - func sendMediaState(_ state: String, trackID: String?, room: Room) async throws { - var object: [String: Any] = ["type": "media_state", "state": state] - if let trackID { object["track_id"] = trackID } - try await publish(object, topic: "media-control", room: room) - } } // MARK: - Minimal JSON value decoder for flexible agent payloads diff --git a/Sources/LiveKit/VoiceSession.swift b/Sources/LiveKit/VoiceSession.swift index 0fad365..1e8eb65 100644 --- a/Sources/LiveKit/VoiceSession.swift +++ b/Sources/LiveKit/VoiceSession.swift @@ -48,7 +48,6 @@ final class VoiceSession: ObservableObject { let transcriptHandler = TranscriptHandler() let stateModel = ConversationStateModel() let taskStore = TaskStore() - let mediaController: MediaPlayerController private let config: ServerConfig private let tokenClient: TokenClient private let settings: SettingsStore @@ -90,9 +89,6 @@ final class VoiceSession: ObservableObject { self.config = config self.tokenClient = tokenClient self.settings = settings - let mediaAPI = MediaAPI(config: config) - self.mediaController = MediaPlayerController(api: mediaAPI, settings: settings) - mediaController.attach(session: self) room.add(delegate: self) room.add(delegate: controlChannel) room.add(delegate: transcriptHandler) @@ -211,7 +207,6 @@ final class VoiceSession: ObservableObject { do { let token = try await tokenClient.fetchToken() logger.log("token ok room=\(token.room, privacy: .public)") - await mediaController.enterMediaScope(roomName: token.room) try await room.connect(url: config.livekitURL.absoluteString, token: token.token, connectOptions: makeConnectOptions()) result = .success(()) } catch { @@ -263,8 +258,7 @@ final class VoiceSession: ObservableObject { private func handleAgentTimeout(generation: Int) async { guard generation == sessionGeneration, state == .waitingAgent else { return } guard !reconnectAttempted else { - // Final failure: tear down the live room, restore the idle timer, - // and end the media scope so no stale room or wake lock survives. + // Final failure: tear down the live room and restore the idle timer. logger.log("agent timeout final, tearing down") await teardownForFailure(message: "Agent never joined. Is the agent worker running?") return @@ -281,8 +275,7 @@ final class VoiceSession: ObservableObject { /// current generation (so stale callbacks become no-ops), clear all /// per-room/session ephemeral published fields (agent, mute state, model /// list, error state, interruption ownership), disconnect and recreate the - /// room, stop conversation state, restore the idle timer, and end the media - /// scope. Only after all cleanup completes does the session publish + /// room, stop conversation state, restore the idle timer, and publish /// `.failed(message)`, making retry safe: the old room is fully torn down /// before Start becomes available again. /// @@ -308,7 +301,6 @@ final class VoiceSession: ObservableObject { interruptionSequence += 1 stateModel.stop() UIApplication.shared.isIdleTimerDisabled = false - await mediaController.endConversation() await teardownRoom() // Final reset immediately before publishing .failed: any old-room @@ -540,15 +532,13 @@ final class VoiceSession: ObservableObject { interruptionSequence += 1 stateModel.stop() UIApplication.shared.isIdleTimerDisabled = false - await mediaController.endConversation() await teardownRoom() // Final reset before publishing .idle: any old-room callback delivered // during the teardownRoom() await may have repopulated ephemeral fields. // This ensures the idle state is clean for server replacement (Iteration 09). resetEphemeralFields() - // Publish .idle only after media and room teardown have completed. - // This makes idle a truthful readiness signal for server replacement: - // the old session is fully torn down before the UI shows "ready to start". + // Publish .idle only after room teardown has completed: the old session + // is fully torn down before the UI shows "ready to start". state = .idle } diff --git a/Sources/Media/MediaAPI.swift b/Sources/Media/MediaAPI.swift deleted file mode 100644 index 09936c3..0000000 --- a/Sources/Media/MediaAPI.swift +++ /dev/null @@ -1,249 +0,0 @@ -import Foundation - -struct MediaTrack: Decodable, Equatable { - let id: String - let kind: String? - let videoID: String? - let title: String? - let channel: String? - let duration: Double? - let status: String? - let error: String? - let audioURL: String? - let artURL: String? - - enum CodingKeys: String, CodingKey { - case id, kind, title, channel, duration, status, error - case videoID = "video_id" - case audioURL = "audio_url" - case artURL = "art_url" - } -} - -struct MediaPendingJob: Decodable, Equatable { - let jobID: String? - let trackID: String? - let stage: String? - let progress: Double? - let error: String? - - enum CodingKeys: String, CodingKey { - case stage, progress, error - case jobID = "job_id" - case trackID = "track_id" - } -} - -struct MediaSeekHint: Decodable, Equatable { - let ts: Double - let position: Double -} - -struct MediaState: Decodable, Equatable { - let room: String? - let track: MediaTrack? - let desired: String? - let reported: MediaReported? - let seekHint: MediaSeekHint? - let choices: [MediaChoice] - let pending: [MediaPendingJob] - let version: Int? - - enum CodingKeys: String, CodingKey { - case room, track, desired, reported, choices, pending, version - case seekHint = "seek_hint" - } -} - -struct MediaReported: Decodable, Equatable { - let trackID: String? - let state: String? - let position: Double? - let duration: Double? - let ts: Double? - - enum CodingKeys: String, CodingKey { - case state, position, duration, ts - case trackID = "track_id" - } -} - -struct MediaChoice: Decodable, Equatable { - let videoID: String? - let title: String? - let channel: String? - let duration: Double? - - enum CodingKeys: String, CodingKey { - case title, channel, duration - case videoID = "video_id" - } -} - -struct MediaLibrary: Decodable, Equatable { - let tracks: [MediaTrack] - let version: Int? -} - -struct MediaSearchResult: Decodable, Equatable { - let videoID: String? - let trackID: String? - let title: String? - let channel: String? - let duration: Double? - let inLibrary: Bool? - - enum CodingKeys: String, CodingKey { - case title, channel, duration - case videoID = "video_id" - case trackID = "track_id" - case inLibrary = "in_library" - } -} - -struct MediaSearchResponse: Decodable, Equatable { - let results: [MediaSearchResult] -} - -enum MediaAPIError: LocalizedError { - case invalidServerURL(String) - case httpStatus(Int, String) - case decodeFailed - - var errorDescription: String? { - switch self { - case .invalidServerURL(let raw): - return "Invalid server URL: \(raw)" - case .httpStatus(let code, let detail): - return "Media API request failed with HTTP \(code). \(detail)" - case .decodeFailed: - return "Unexpected response from the media endpoint." - } - } -} - -final class MediaAPI { - let serverBase: URL - private let baseURL: URL - private let session: URLSession - - init(config: ServerConfig, session: URLSession? = nil) { - self.serverBase = config.baseURL - self.baseURL = config.baseURL.appendingPathComponent("media/api") - self.session = session ?? TLSPinning.session() - } - - func state(room: String) async throws -> MediaState { - var components = URLComponents(url: baseURL.appendingPathComponent("state"), resolvingAgainstBaseURL: false)! - components.queryItems = [URLQueryItem(name: "room", value: room)] - return try await get(MediaState.self, url: components.url!) - } - - func report(room: String, state: String, trackID: String?, position: Double, duration: Double?) async throws { - var body: [String: Any] = ["room": room, "state": state, "position": position] - if let trackID { body["track_id"] = trackID } - if let duration { body["duration"] = duration } - try await post("report", body: body) - } - - func play(room: String, videoID: String? = nil, title: String? = nil, channel: String? = nil, trackID: String? = nil, startPaused: Bool? = nil) async throws { - var body: [String: Any] = ["room": room] - if let videoID { body["video_id"] = videoID } - if let title { body["title"] = title } - if let channel { body["channel"] = channel } - if let trackID { body["track_id"] = trackID } - if let startPaused { body["start_paused"] = startPaused } - try await post("play", body: body) - } - - func control(room: String, action: String) async throws { - try await post("control", body: ["room": room, "action": action]) - } - - func search(room: String, query: String, limit: Int = 6) async throws -> MediaSearchResponse { - try await postDecoded("search", body: ["room": room, "query": query, "limit": limit], as: MediaSearchResponse.self) - } - - func library() async throws -> MediaLibrary { - try await get(MediaLibrary.self, url: baseURL.appendingPathComponent("library")) - } - - func deleteTrack(id: String) async throws { - var request = URLRequest(url: baseURL.appendingPathComponent("tracks").appending(path: id)) - request.httpMethod = "DELETE" - let (data, response) = try await session.data(for: request) - guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed } - guard (200..<300).contains(http.statusCode) else { - throw MediaAPIError.httpStatus(http.statusCode, String(data: data, encoding: .utf8) ?? "") - } - } - - func addStream(room: String, name: String, url: String) async throws { - try await post("streams", body: ["room": room, "name": name, "url": url]) - } - - /// Returns the progressive-media endpoint URL for a track. The `/stream/{id}` - /// path serves raw media bytes (not a text URL), so this constructs the - /// endpoint without downloading or decoding the body. - func streamURL(trackID: String) -> URL { - baseURL.appendingPathComponent("stream").appending(path: trackID) - } - - func healthz() async throws -> Bool { - let data = try await getRaw(url: baseURL.appendingPathComponent("healthz")) - guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { return false } - return json["ok"] as? Bool ?? false - } - - private func get(_ type: T.Type, url: URL) async throws -> T { - let data = try await getRaw(url: url) - do { - return try JSONDecoder().decode(T.self, from: data) - } catch { - throw MediaAPIError.decodeFailed - } - } - - private func getRaw(url: URL) async throws -> Data { - let (data, response) = try await session.data(from: url) - guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed } - guard (200..<300).contains(http.statusCode) else { - throw MediaAPIError.httpStatus(http.statusCode, String(data: data, encoding: .utf8) ?? "") - } - return data - } - - /// Fire-and-forget POST that only validates HTTP success. The media server - /// returns `{"ok": true, ...snapshot}` for /play, /control, /report and - /// /streams — callers don't need the body, so decoding it (as Data or any - /// typed model) would be wasted work at best and a spurious failure mode. - private func post(_ path: String, body: [String: Any]) async throws { - _ = try await postRaw(path, body: body) - } - - private func postDecoded(_ path: String, body: [String: Any], as type: T.Type) async throws -> T { - let data = try await postRaw(path, body: body) - do { - return try JSONDecoder().decode(T.self, from: data) - } catch { - throw MediaAPIError.decodeFailed - } - } - - private func postRaw(_ path: String, body: [String: Any]) async throws -> Data { - var request = URLRequest(url: baseURL.appendingPathComponent(path)) - request.httpMethod = "POST" - request.setValue("application/json", forHTTPHeaderField: "Content-Type") - request.httpBody = try JSONSerialization.data(withJSONObject: body) - let (data, response) = try await session.data(for: request) - guard let http = response as? HTTPURLResponse else { throw MediaAPIError.decodeFailed } - guard (200..<300).contains(http.statusCode) else { - var detail = String(data: data, encoding: .utf8) ?? "" - if let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], let error = json["error"] as? String { - detail = error - } - throw MediaAPIError.httpStatus(http.statusCode, detail) - } - return data - } -} diff --git a/Sources/Media/MediaPlayerController.swift b/Sources/Media/MediaPlayerController.swift deleted file mode 100644 index 29aecd3..0000000 --- a/Sources/Media/MediaPlayerController.swift +++ /dev/null @@ -1,633 +0,0 @@ -import AVFoundation -import Combine -import Foundation -import os - -private let logger = Logger(subsystem: "dev.shane.voice", category: "media") - -/// AVPlayer-backed media player. The media server holds per-room desired state; -/// this controller polls /state and reconciles the local player against it, -/// porting the web UI's reconciliation rules (app.js reconcileMedia). -@MainActor -final class MediaPlayerController: ObservableObject { - @Published private(set) var track: MediaTrack? - @Published private(set) var desired: String = "stopped" - @Published private(set) var currentTime: Double = 0 - @Published private(set) var duration: Double = 0 - @Published private(set) var pendingJobs: [MediaPendingJob] = [] - @Published private(set) var choices: [MediaChoice] = [] - @Published private(set) var stateError: String? - @Published private(set) var playbackError: String? - - let player = AVPlayer() - - private let api: MediaAPI - private let settings: SettingsStore - private let mediaLoader: PinnedMediaResourceLoader - private weak var session: VoiceSession? - - private var room: String? - private var currentTrackID: String? - private var loadedAudioPath: String? - private var currentAsset: AVURLAsset? - private var appliedSeekTs: Double = 0 - private var wasPlayingBeforeTalk = false - private var lastReportDate = Date.distantPast - private var pollTask: Task? - private var timeObserver: Any? - private var statusCancellable: AnyCancellable? - private var rateObservation: NSKeyValueObservation? - - /// Bumped on every room-scope change so in-flight /state polls from the - /// previous scope cannot reconcile into the new one (port of the web UI's - /// `=== room` checks). - private var pollGeneration = 0 - - init(api: MediaAPI, settings: SettingsStore) { - self.api = api - self.settings = settings - self.mediaLoader = PinnedMediaResourceLoader(base: api.serverBase) - player.volume = Float(settings.volume) - NotificationCenter.default.addObserver( - self, selector: #selector(handleInterruption), - name: AVAudioSession.interruptionNotification, object: nil - ) - } - - deinit { - if let timeObserver { - player.removeTimeObserver(timeObserver) - } - rateObservation?.invalidate() - NotificationCenter.default.removeObserver(self) - } - - /// Whether an audio session interruption is currently active. While true, - /// reconciliation must not call player.play() (the next poll would restart - /// audio during a phone call or Siri interaction). - private var isInterruptionActive = false - - /// Separate resume-suppression flag: set when an interruption ends without - /// `.shouldResume` and the pause control to the server has not yet been - /// confirmed. While true, reconciliation must not call player.play() even - /// after `isInterruptionActive` is cleared. Cleared only by an explicit user - /// play/resume action or a confirmed coherent paused state from the server. - private var resumeSuppressed = false - - /// Whether playback was actually active (rate > 0) when the interruption - /// began. Used to decide whether a `.shouldResume` end should actually - /// resume (a track that was already paused before the interruption must not - /// be force-resumed). - private var wasPlayingBeforeInterruption = false - - /// Sequence number for media interruption operations. Incremented on every - /// `.began`; a resume task captures the sequence and rechecks it immediately - /// before `player.play()`. A newer `.began` makes an old resume task - /// incapable of playing (Iteration 07 Item 5). - private var mediaInterruptionSequence = 0 - - /// The retained operation chain for media interruption control work. Both - /// pause and resume operations are assigned here so a newer begin cancels - /// and supersedes prior control work. - private var mediaInterruptionTask: Task? - - @objc private func handleInterruption(_ notification: Notification) { - guard let info = notification.userInfo else { return } - guard let typeValue = info[AVAudioSessionInterruptionTypeKey] as? UInt, - let type = AVAudioSession.InterruptionType(rawValue: typeValue) else { return } - switch type { - case .began: - // Increment the sequence so any in-flight resume task from a prior - // interruption sees a stale sequence and cannot play. - mediaInterruptionSequence += 1 - let seq = mediaInterruptionSequence - // Cancel/supersede prior control work (resume or pause) so it cannot - // complete out of order. - mediaInterruptionTask?.cancel() - isInterruptionActive = true - wasPlayingBeforeInterruption = !player.rate.isZero - let wasPlaying = wasPlayingBeforeInterruption - player.pause() - logger.log("audio interruption began, paused media (wasPlaying=\(wasPlaying), seq=\(seq))") - case .ended: - defer { isInterruptionActive = false } - applyVolume() - let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0 - let shouldResume = AVAudioSession.InterruptionOptions(rawValue: optionsValue).contains(.shouldResume) - if shouldResume, wasPlayingBeforeInterruption, currentTrackID != nil, desired == "playing", loadedAudioPath != nil { - // Synchronize the server before resuming local playback so the - // two never diverge. Capture generation/room/sequence before the - // async hop. The task is retained in mediaInterruptionTask so a - // newer .began can cancel it. - let gen = pollGeneration - let targetRoom = room - let seq = mediaInterruptionSequence - mediaInterruptionTask = Task { [weak self] in - guard let self else { return } - if let targetRoom { - do { - try await self.api.control(room: targetRoom, action: "resume") - logger.log("interruption resume: server synchronized (seq=\(seq))") - } catch { - logger.log("interruption resume: server sync failed (poll will retry): \(error.localizedDescription)") - } - } - // Recheck the sequence immediately before play: a newer .began - // increments mediaInterruptionSequence, making this stale task - // incapable of restarting audio during the newer interruption. - guard !Task.isCancelled, - self.mediaInterruptionSequence == seq, - !self.isInterruptionActive, - self.pollGeneration == gen, - self.room == targetRoom else { return } - self.player.play() - logger.log("audio interruption ended, resumed media (seq=\(seq))") - } - } else { - // No-resume path: keep suppression even if the pause control - // fails. The flag is cleared only by an explicit user play/resume - // action or a confirmed coherent paused state from the server. - resumeSuppressed = true - let seq = mediaInterruptionSequence - if let room { - mediaInterruptionTask = Task { [weak self] in - guard let self else { return } - do { - try await self.api.control(room: room, action: "pause") - logger.log("audio interruption ended without resume, sent pause to server (seq=\(seq))") - } catch { - // Keep suppression even on failure — the next poll - // will see desired=="paused" and clear it. - logger.log("audio interruption ended without resume, pause control failed (suppression retained): \(error.localizedDescription)") - } - } - } - } - @unknown default: - break - } - } - - // MARK: Lifecycle - - func attach(session: VoiceSession) { - self.session = session - observePlayer() - observePlaybackState() - applyVolume() - } - - /// Called just before joining the LiveKit room. Carries whatever is loaded - /// into the new room-scoped player state, paused (talking pauses music). - func enterMediaScope(roomName: String) async { - pollGeneration += 1 - let wasPlaying = currentTrackID != nil && !player.rate.isZero - wasPlayingBeforeTalk = wasPlaying - let carriedTrack = currentTrackID - room = roomName - if let carriedTrack { - do { - try await api.play(room: roomName, trackID: carriedTrack, startPaused: true) - logger.log("media scope carry-over: \(carriedTrack, privacy: .public) start_paused") - } catch { - logger.log("media scope transfer failed: \(String(describing: error), privacy: .public)") - } - player.pause() - } - startPolling() - } - - /// Called when the conversation ends. If music was playing before the talk, - /// hand the room back to the player. The previous scope (room name) is kept - /// so the panel can resume polling it via `startDefaultPolling()`; a fresh - /// conversation re-scopes via `enterMediaScope`. Polling restarts here so - /// the resumed state stays visible without the panel being open. - func endConversation() async { - stopPolling() - pollGeneration += 1 - if wasPlayingBeforeTalk { - wasPlayingBeforeTalk = false - guard let room else { return } - do { - try await api.control(room: room, action: "resume") - logger.log("resumed playback after talk ended") - } catch { - logger.log("post-talk resume failed (poll will retry): \(String(describing: error), privacy: .public)") - } - } - startPolling() - } - - /// Drop the room scope entirely (app-level teardown). After this, polling - /// targets the default room again. - func resetScope() { - stopPolling() - pollGeneration += 1 - room = nil - wasPlayingBeforeTalk = false - clearMedia() - } - - private func startPolling() { - guard pollTask == nil else { return } - pollTask = Task { [weak self] in - while !Task.isCancelled { - await self?.pollState() - try? await Task.sleep(for: .milliseconds(1500)) - } - } - } - - private func stopPolling() { - pollTask?.cancel() - pollTask = nil - } - - /// Resume polling whenever the panel becomes visible and nothing is being - /// polled yet — whether that's the default room (no session) or a previous - /// conversation's room scope (kept after `endConversation`). - func startDefaultPolling() { - guard pollTask == nil else { return } - startPolling() - } - - // MARK: Poll + reconcile (ported from app.js pollMediaState/reconcileMedia) - - private func pollState() async { - let generation = pollGeneration - let scope = room ?? "default" - do { - let state = try await api.state(room: scope) - guard generation == pollGeneration else { return } - stateError = nil - reconcile(state) - } catch { - guard generation == pollGeneration else { return } - logger.log("poll failed: \(String(describing: error), privacy: .public)") - stateError = "Media server unreachable: \(error.localizedDescription)" - } - } - - /// Reconciles the local player against a server state snapshot. Exposed - /// as internal (not private) so tests can drive deterministic state - /// transitions without a live server. - func reconcile(_ s: MediaState) { - choices = s.choices - pendingJobs = s.pending - // Keep the published desired state in sync with the server so the UI's - // play/pause icon and status text reflect reality (the server is the - // source of truth for per-room desired state). - desired = s.desired ?? "stopped" - - let track = s.track - self.track = track - if let track, track.id != currentTrackID { - loadTrack(track) - } else if let track, let audioURL = track.audioURL, loadedAudioPath != audioURL { - // Track was loaded before it had a stream (still downloading) and - // now has one — wire it up so playback can start without a reload. - wireSource(audioURL) - } - if track == nil, currentTrackID != nil, s.desired == "stopped" { - clearMedia() - } - - applyDesiredState(s, track: track) - - if let hint = s.seekHint, hint.ts != appliedSeekTs, duration.isFinite, duration > 0 { - appliedSeekTs = hint.ts - let target = min(hint.position, duration) - if target.isFinite { - seek(to: max(0, target)) - } - } - } - - private func applyDesiredState(_ s: MediaState, track: MediaTrack?) { - guard let desiredValue = s.desired else { return } - switch desiredValue { - case "playing": - // Suppress play during an active audio interruption (phone call, - // Siri). Without this guard the next 1.5s poll would restart audio - // immediately after .began paused it. The player.rate.isZero check - // makes this idempotent: once playing, subsequent polls are no-ops. - // This also handles the late-audio case where desired=playing - // arrives before audio_url is set; when the source is later wired - // via wireSource(), the next poll retries play because rate is - // still zero. - // Also suppress while resumeSuppressed is true (no-resume - // interruption ended, server not yet confirmed paused). - guard !isInterruptionActive, !resumeSuppressed else { return } - if let track, track.audioURL != nil, loadedAudioPath != nil, player.rate.isZero { - player.play() - } - case "paused", "stopped": - if !player.rate.isZero { - player.pause() - } - // The server has confirmed a coherent paused/stopped state. If we - // were suppressing resume due to a no-resume interruption, the - // pause control succeeded (or the user paused independently), so - // it is safe to clear the suppression flag. - if resumeSuppressed { - resumeSuppressed = false - logger.log("resume suppression cleared: server confirmed \(desiredValue)") - } - default: - break - } - } - - private func loadTrack(_ track: MediaTrack) { - currentTrackID = track.id - self.track = track - if let audioURL = track.audioURL { - wireSource(audioURL) - } else { - // Downloading or failed: no stream yet. Reconcile wires the stream - // URL in via polling as soon as one is available. - clearCurrentAsset() - player.replaceCurrentItem(with: nil) - } - } - - private func wireSource(_ path: String) { - // A complete file is required for the server's cached AAC conversion. - // Keep showing download progress; the next ready-state poll wires it. - guard track?.kind != "youtube" || track?.status != "downloading" else { return } - guard var components = URLComponents(string: path) else { return } - if track?.kind == "youtube", components.path.hasPrefix("/media/api/stream/") { - var items = components.queryItems ?? [] - items.removeAll { $0.name == "format" } - items.append(URLQueryItem(name: "format", value: "m4a")) - components.queryItems = items - } - guard let url = components.url else { return } - loadedAudioPath = path - let pinned = mediaLoader.rewrite(url) - // Remove the previous asset's loader mapping before wiring a new one so - // the loaderURLs dictionary stays bounded (Finding 11 / Item 7). - clearCurrentAsset() - let asset = mediaLoader.assetWithPinnedURL(pinned) - currentAsset = asset - logger.log("wiring source \(pinned.absoluteString, privacy: .public)") - let item = AVPlayerItem(asset: asset) - observeItem(item) - player.replaceCurrentItem(with: item) - } - - /// Removes the loader mapping for the current asset (if any) and clears the - /// reference. Called before wiring a new source or clearing media so the - /// resource loader's internal dictionaries stay bounded. - private func clearCurrentAsset() { - if let asset = currentAsset { - mediaLoader.removeLoader(asset) - currentAsset = nil - } - } - - // MARK: Player events - - private func observePlayer() { - if timeObserver == nil { - timeObserver = player.addPeriodicTimeObserver( - forInterval: CMTime(seconds: 0.5, preferredTimescale: 600), queue: .main - ) { [weak self] time in - Task { @MainActor [weak self] in - guard let self else { return } - self.currentTime = time.seconds - if let itemDuration = self.player.currentItem?.duration { - let seconds = itemDuration.seconds - if seconds.isFinite { - self.duration = seconds - } - } - if !self.player.rate.isZero, Date().timeIntervalSince(self.lastReportDate) >= 5 { - self.lastReportDate = Date() - self.report("playing") - } - } - } - } - } - - // MARK: - Playback state observation - - /// Observe the player's actual rate so every real play/pause transition is - /// reported promptly, mirroring the web UI's `play`/`pause` listeners. The - /// KVO block runs on an arbitrary queue; hop to MainActor before touching - /// any controller state. - private func observePlaybackState() { - rateObservation = player.observe(\.rate, options: [.initial, .new]) { [weak self] player, _ in - Task { @MainActor [weak self] in - guard let self else { return } - guard self.currentTrackID != nil else { return } - if player.rate > 0 { - // Actual playback began (user play, interruption resume, or - // reconcile-driven play). Report + publish the transition. - self.lastReportDate = Date() - self.report("playing") - self.publishMediaState("playing") - } else if player.currentItem != nil { - // Rate dropped to zero with an item still loaded: a pause - // that didn't originate from a user action (interruption, - // reconcile). User-initiated pauses already report before - // this fires; the lastReportDate guard makes duplicates no-ops. - if Date().timeIntervalSince(self.lastReportDate) >= 1 { - self.lastReportDate = Date() - self.report("paused") - self.publishMediaState("paused") - } - } - } - } - } - - private func observeItem(_ item: AVPlayerItem?) { - NotificationCenter.default.removeObserver( - self, name: .AVPlayerItemDidPlayToEndTime, object: nil - ) - guard let item else { return } - NotificationCenter.default.addObserver( - self, selector: #selector(playerItemDidEnd(_:)), - name: .AVPlayerItemDidPlayToEndTime, object: item - ) - statusCancellable = item.publisher(for: \.status).sink { [weak self, weak item] status in - Task { @MainActor in - guard let self, let item, self.player.currentItem === item else { return } - switch status { - case .readyToPlay: - self.playbackError = nil - case .failed: - self.playbackError = "This audio couldn’t play on your iPhone. Try another track or an MP3/AAC station." - logger.log("item failed: \(String(describing: item.error), privacy: .public)") - default: break - } - } - } - } - - @objc private func playerItemDidEnd(_ notification: Notification) { - lastReportDate = Date() - report("stopped") - publishMediaState("stopped") - Task { [weak self] in - guard let self else { return } - try? await self.api.control(room: self.room ?? "default", action: "stop") - } - } - - // MARK: Reporting + publishing - - private func report(_ state: String, trackID: String? = nil) { - let room = room ?? "default" - let position = currentTime.isFinite ? currentTime : 0 - var durationValue: Double? - if duration.isFinite, duration > 0 { - durationValue = duration - } - let resolvedTrackID = trackID ?? currentTrackID - Task { [weak self] in - guard let self else { return } - do { - try await self.api.report(room: room, state: state, trackID: resolvedTrackID, position: position, duration: durationValue) - } catch { - logger.log("report failed: \(String(describing: error), privacy: .public)") - } - } - } - - private func publishMediaState(_ state: String, trackID: String? = nil) { - guard let session else { return } - // Capture the exact room and generation before spawning so a delayed - // publish from an old lifecycle cannot escape into a new conversation - // room (Finding 10). - let targetRoom = session.room - let generation = session.sessionGeneration - let pollGen = pollGeneration - let resolvedTrackID = trackID ?? currentTrackID - Task { [weak self] in - guard let self, self.pollGeneration == pollGen else { return } - guard session.sessionGeneration == generation, session.room === targetRoom else { return } - do { - try await session.controlChannel.sendMediaState(state, trackID: resolvedTrackID, room: targetRoom) - } catch { - logger.log("media_state publish failed: \(String(describing: error), privacy: .public)") - } - } - } - - // MARK: Play requests (search results, choices, library) - - func playVideo(videoID: String?, title: String?, channel: String?) async { - let targetRoom = room ?? "default" - let generation = pollGeneration - playbackError = nil - do { - try await api.play(room: targetRoom, videoID: videoID, title: title, channel: channel) - logger.log("play requested video_id=\(videoID ?? "nil", privacy: .public) room=\(targetRoom, privacy: .public)") - } catch { - guard generation == pollGeneration else { return } - playbackError = "Couldn’t play this song. Check your connection and try again." - logger.log("play failed: \(String(describing: error), privacy: .public)") - } - } - - func playTrack(id: String) async { - let targetRoom = room ?? "default" - let generation = pollGeneration - playbackError = nil - do { - try await api.play(room: targetRoom, trackID: id) - logger.log("play requested track_id=\(id, privacy: .public) room=\(targetRoom, privacy: .public)") - } catch { - guard generation == pollGeneration else { return } - playbackError = "Couldn’t play this track. Check your connection and try again." - logger.log("play failed: \(String(describing: error), privacy: .public)") - } - } - - // MARK: Transport controls (parity with web UI play/pause/seek/stop) - - func togglePlayPause() async { - guard currentTrackID != nil, loadedAudioPath != nil else { return } - if player.rate.isZero { - // Explicit user play: clear any resume suppression (the user is - // deliberately overriding the no-resume interruption state). - resumeSuppressed = false - // The rate observer reports + publishes the actual transition; - // the control call updates server desired state. - player.play() - await sendControl("resume") - } else { - lastReportDate = Date() - report("paused") - publishMediaState("paused") - player.pause() - await sendControl("pause") - } - } - - func seek(by delta: Double) async { - guard loadedAudioPath != nil else { return } - let target = max(0, min(duration > 0 ? duration - 0.5 : .infinity, currentTime + delta)) - if target.isFinite { - seek(to: target) - } - } - - func seek(to position: Double) { - guard position.isFinite, position >= 0 else { return } - player.seek(to: CMTime(seconds: position, preferredTimescale: 600)) - currentTime = position - report(player.rate.isZero ? "paused" : "playing") - } - - func stop() async { - // Capture the track ID before clearing so the async report/publish - // tasks can read a non-nil value even after clearMedia() runs. - let trackID = currentTrackID - report("stopped", trackID: trackID) - clearMedia() - await sendControl("stop") - publishMediaState("stopped", trackID: trackID) - } - - private func clearMedia() { - playbackError = nil - currentTrackID = nil - track = nil - loadedAudioPath = nil - // Remove the loader mapping for the asset being cleared so the - // resource loader's internal dictionaries stay bounded (Item 7). - clearCurrentAsset() - observeItem(nil) - player.replaceCurrentItem(with: nil) - currentTime = 0 - duration = 0 - stateError = nil - } - - private func sendControl(_ action: String) async { - let generation = pollGeneration - do { - try await api.control(room: room ?? "default", action: action) - guard generation == pollGeneration else { return } - playbackError = nil - } catch { - guard generation == pollGeneration else { return } - playbackError = "Couldn’t update playback on the server. Check your connection and try again." - } - } - - // MARK: Volume - - func applyVolume() { - player.volume = Float(settings.volume) - } - - func setVolume(_ volume: Double) { - settings.volume = volume - player.volume = Float(volume) - } -} diff --git a/Sources/Media/PinnedMediaResourceLoader.swift b/Sources/Media/PinnedMediaResourceLoader.swift deleted file mode 100644 index a4acdd2..0000000 --- a/Sources/Media/PinnedMediaResourceLoader.swift +++ /dev/null @@ -1,238 +0,0 @@ -import AVFoundation -import Foundation -import UniformTypeIdentifiers -import os - -private let logger = Logger(subsystem: "dev.shane.voice", category: "media") - -/// Proxies {base}/media/* bytes to AVPlayer over the pinned-TLS session, since -/// AVPlayer cannot trust the self-signed LAN cert. AVPlayer loads https natively -/// (ignoring registered URLProtocol subclasses), so the asset uses a custom -/// `pinned-media` scheme and this delegate answers every data request by -/// fetching the real https URL through TLSPinning.session(), forwarding Range. -/// -/// **Lifecycle invariants (Finding 11 / Iteration 06 Item 7):** -/// - Tasks are keyed by `ObjectIdentifier(loadingRequest)` directly, not a -/// derived hash, so two distinct requests can never collide. -/// - All mutable state (`loaderURLs`, `activeTasks`) is accessed only on the -/// main queue: the AVAssetResourceLoader delegate is set with `queue: .main` -/// and the pinned URLSession uses `delegateQueue: .main`, so both the -/// resource-loading callbacks and the data-task completion handlers run on -/// the same serialization domain. This is enforced in code, not by comment. -/// - For HTTP 206, `contentLength` is set only when `Content-Range` provides a -/// known total; it never falls back to the range's `expectedContentLength`. -/// - For HTTP 200, `expectedContentLength` is already the full resource size. -/// - Cleanup (task removal) occurs on success, error, cancellation, and item -/// replacement via `removeLoader`. -final class PinnedMediaResourceLoader: NSObject, AVAssetResourceLoaderDelegate { - - static let scheme = "pinned-media" - - private let base: URL - /// Maps each asset's resource loader to the real https URL it proxies. - /// Accessed only on the main queue (the delegate queue). - private var loaderURLs: [ObjectIdentifier: URL] = [:] - /// Maps each loading request's identity to its upstream data task so - /// cancellation can cancel the in-flight download. Keyed directly by - /// `ObjectIdentifier(loadingRequest)` — no hash derivation, no collisions. - private var activeTasks: [ObjectIdentifier: URLSessionDataTask] = [:] - /// Shared pinned session, created once and invalidated on deinit. - /// Uses `delegateQueue: .main` so all completion handlers (and therefore - /// all mutations of `activeTasks`) run on the main queue — the same - /// serialization domain as the AVAssetResourceLoader delegate callbacks - /// (which are also dispatched to `.main`). This makes map access truly - /// serialized in code, not just by convention. - private lazy var pinnedSession: URLSession = { - let config = URLSessionConfiguration.default - config.timeoutIntervalForRequest = 30 - return URLSession(configuration: config, delegate: PinnedTrustDelegate(), delegateQueue: .main) - }() - - init(base: URL) { - self.base = base - } - - deinit { - pinnedSession.invalidateAndCancel() - } - - /// Maps a media URL (absolute https or /media/... path) to the pinned-media - /// scheme so AVPlayer routes it through this delegate instead of loading - /// https natively. Built as a string because URLComponents.url returns nil - /// for schemes not in Foundation's known-scheme list. - func rewrite(_ url: URL) -> URL { - let absolute: String - if let scheme = url.scheme?.lowercased(), scheme.hasPrefix("http") { - var c = URLComponents(url: url, resolvingAgainstBaseURL: false)! - c.scheme = Self.scheme - return c.url! - } else { - let path = url.absoluteString.hasPrefix("/") ? String(url.absoluteString.dropFirst()) : url.absoluteString - guard let host = base.host, !host.isEmpty else { return url } - let port = base.port.map { ":\($0)" } ?? "" - absolute = "\(Self.scheme)://\(host)\(port)/\(path)" - } - return URL(string: absolute) ?? url - } - - /// Maps a pinned-media URL back to the real https URL it proxies. - func resolve(_ url: URL) -> URL { - guard let scheme = url.scheme?.lowercased(), scheme == Self.scheme else { return url } - var c = URLComponents(url: url, resolvingAgainstBaseURL: false)! - c.scheme = "https" - return c.url ?? url - } - - /// Creates an AVURLAsset that routes through this loader's delegate. The - /// mapping from the asset's resource loader to the real URL is recorded so - /// `resourceLoader(_:shouldWaitForLoadingOfRequestedResource:)` can resolve - /// it. All mutations happen on the main queue (the delegate queue). - func assetWithPinnedURL(_ pinned: URL) -> AVURLAsset { - let asset = AVURLAsset(url: pinned) - let realURL = resolve(pinned) - // The delegate queue is .main, so this runs on the main thread. - loaderURLs[ObjectIdentifier(asset.resourceLoader)] = realURL - asset.resourceLoader.setDelegate(self, queue: .main) - return asset - } - - /// Removes the loader mapping and any in-flight tasks for an asset that is - /// no longer needed. Called by MediaPlayerController when replacing or - /// clearing media items to bound the dictionaries (Finding 11). - func removeLoader(_ asset: AVURLAsset) { - let key = ObjectIdentifier(asset.resourceLoader) - loaderURLs[key] = nil - // Cancel any in-flight tasks. URLSessionDataTask has no isCancelled - // property; calling cancel() on an already-cancelled task is a no-op. - for (_, task) in activeTasks { - task.cancel() - } - activeTasks.removeAll() - } - - func resourceLoader( - _ resourceLoader: AVAssetResourceLoader, - shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest - ) -> Bool { - // All state access is on the main queue (the delegate queue). - guard loadingRequest.dataRequest != nil else { return false } - guard let url = loaderURLs[ObjectIdentifier(resourceLoader)] else { - logger.log("no resolved URL for resource loader") - loadingRequest.finishLoading(with: URLError(.badURL)) - return false - } - - fetchChunk(loadingRequest, url: url) - return true - } - - /// Return bounded chunks promptly instead of buffering an entire song before - /// AVPlayer receives its first usable bytes. The same loading request stays - /// open until its requested range (or EOF) has been delivered. - private func fetchChunk(_ loadingRequest: AVAssetResourceLoadingRequest, url: URL) { - guard !loadingRequest.isCancelled, let dataRequest = loadingRequest.dataRequest else { return } - var upstream = URLRequest(url: url) - let offset = max(dataRequest.requestedOffset, dataRequest.currentOffset) - let length = Int64(dataRequest.requestedLength) - let requestedEnd = dataRequest.requestedOffset + length - let end = dataRequest.requestsAllDataToEndOfResource || length == 0 - ? offset + 262_143 : min(requestedEnd - 1, offset + 262_143) - upstream.setValue("bytes=\(offset)-\(end)", forHTTPHeaderField: "Range") - logger.log("fetching \(url.absoluteString, privacy: .public) range=\(upstream.value(forHTTPHeaderField: "Range") ?? "-", privacy: .public)") - - // Key the task by the loading request's identity directly — no hash - // derivation, so two distinct requests can never collide. - let requestKey = ObjectIdentifier(loadingRequest) - let task = pinnedSession.dataTask(with: upstream) { [weak self] data, response, error in - guard let self else { return } - // Cleanup: remove the task from the active map regardless of outcome. - // Runs on .main (the session's delegateQueue), same domain as all - // other map mutations. - self.activeTasks[requestKey] = nil - guard !loadingRequest.isCancelled else { return } - - if let error { - logger.log("fetch failed \(url.absoluteString, privacy: .public): \(error.localizedDescription, privacy: .public)") - loadingRequest.finishLoading(with: error) - return - } - guard let http = response as? HTTPURLResponse else { - loadingRequest.finishLoading(with: URLError(.badServerResponse)) - return - } - // Reject non-200/206 responses (404, 500, etc.) instead of feeding - // error JSON into AVPlayer as media bytes. - guard http.statusCode == 200 || http.statusCode == 206 else { - logger.log("unexpected status \(http.statusCode) for \(url.absoluteString, privacy: .public)") - loadingRequest.finishLoading(with: URLError(.badServerResponse)) - return - } - if let info = loadingRequest.contentInformationRequest { - // AVFoundation requires a UTI, not the HTTP MIME string. - info.contentType = http.mimeType.flatMap { UTType(mimeType: $0)?.identifier } - // Content-length handling (Iteration 05 Item 7): - // - For 206: set full content length ONLY when Content-Range - // provides a known total. Never fall back to the range's - // expectedContentLength (that would tell AVPlayer the total - // resource is only as large as this one range). - // - For 200: expectedContentLength is already the full size. - if http.statusCode == 206 { - if let total = Self.parseTotalLength(http), total >= 0 { - info.contentLength = total - } - // If Content-Range has no known total (e.g. "bytes 0-1023/*"), - // do NOT set contentLength — AVPlayer will handle it as - // unknown-length, which is correct for a range response. - } else { - // 200: expectedContentLength is the full resource size. - if http.expectedContentLength >= 0 { - info.contentLength = Int64(http.expectedContentLength) - } - } - info.isByteRangeAccessSupported = true - } - if let data, !data.isEmpty { - dataRequest.respond(with: data) - let nextOffset = offset + Int64(data.count) - let total = Self.parseTotalLength(http) - let reachedEOF = total.map { nextOffset >= $0 } ?? (nextOffset <= end) - let needsMore = dataRequest.requestsAllDataToEndOfResource || nextOffset < requestedEnd - if http.statusCode == 206, !reachedEOF, needsMore { - self.fetchChunk(loadingRequest, url: url) - return - } - } - loadingRequest.finishLoading() - } - activeTasks[requestKey] = task - task.resume() - } - - func resourceLoader( - _ resourceLoader: AVAssetResourceLoader, - didCancel loadingRequest: AVAssetResourceLoadingRequest - ) { - // Cancel the upstream data task so a seek or item swap doesn't keep - // downloading cancelled ranges. Keyed by ObjectIdentifier directly. - let key = ObjectIdentifier(loadingRequest) - activeTasks[key]?.cancel() - activeTasks[key] = nil - logger.log("resource loader cancelled request, upstream task cancelled") - } - - /// Parses the total resource length from an HTTP 206 response's - /// `Content-Range` header (e.g. "bytes 0-1023/4096" → 4096). Returns nil - /// for non-206 responses, missing headers, unknown totals ("*"), or - /// malformed values. - static func parseTotalLength(_ response: HTTPURLResponse) -> Int64? { - if response.statusCode == 206, - let contentRange = response.value(forHTTPHeaderField: "Content-Range") { - // Format: "bytes -/" or "bytes -/*" - let afterSlash = contentRange.split(separator: "/").last?.trimmingCharacters(in: .whitespaces) - if let afterSlash, afterSlash != "*", let total = Int64(afterSlash) { - return total - } - } - return nil - } -} diff --git a/Tests/MediaAPITests.swift b/Tests/MediaAPITests.swift deleted file mode 100644 index 0bb7d42..0000000 --- a/Tests/MediaAPITests.swift +++ /dev/null @@ -1,125 +0,0 @@ -import Foundation -import Testing -@testable import Voice - -/// Minimal HTTP/1.1 stub built on BSD sockets that answers every request with a -/// successful JSON object body (mirroring the real media server's -/// `{"ok": true, ...}` shape). Used to prove MediaAPI no-response POSTs succeed -/// against an object body instead of throwing decodeFailed. -@MainActor -final class MediaAPIStubServer { - private(set) var url: URL! - nonisolated(unsafe) private var serverFD: Int32 = -1 - nonisolated(unsafe) private var acceptThread: Thread? - nonisolated(unsafe) private var running = false - nonisolated(unsafe) private var lastPath: String? - - func start() throws { - let fd = socket(AF_INET, SOCK_STREAM, 0) - guard fd >= 0 else { throw MediaAPIError.decodeFailed } - var yes: Int32 = 1 - setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout.size)) - - var addr = sockaddr_in() - addr.sin_family = sa_family_t(AF_INET) - addr.sin_port = in_port_t(0).bigEndian - addr.sin_addr = in_addr(s_addr: inet_addr("127.0.0.1")) - let bindResult = withUnsafePointer(to: &addr) { ptr in - ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in - bind(fd, sa, socklen_t(MemoryLayout.size)) - } - } - guard bindResult == 0 else { close(fd); throw MediaAPIError.decodeFailed } - guard listen(fd, 8) == 0 else { close(fd); throw MediaAPIError.decodeFailed } - - var bound = sockaddr_in() - var len = socklen_t(MemoryLayout.size) - withUnsafeMutablePointer(to: &bound) { ptr in - _ = ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in - getsockname(fd, sa, &len) - } - } - let port = Int(UInt16(bigEndian: bound.sin_port)) - serverFD = fd - running = true - url = URL(string: "http://127.0.0.1:\(port)")! - - let thread = Thread { [weak self] in - self?.acceptLoop() - } - acceptThread = thread - thread.start() - } - - func stop() { - running = false - if serverFD >= 0 { close(serverFD) } - serverFD = -1 - acceptThread?.cancel() - } - - var lastRequestPath: String? { lastPath } - - nonisolated private func acceptLoop() { - while running { - var clientAddr = sockaddr_in() - var len = socklen_t(MemoryLayout.size) - let clientFD = withUnsafeMutablePointer(to: &clientAddr) { ptr in - ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in - accept(serverFD, sa, &len) - } - } - guard clientFD >= 0 else { continue } - handleClient(clientFD) - } - } - - nonisolated private func handleClient(_ clientFD: Int32) { - defer { close(clientFD) } - var buffer = [UInt8](repeating: 0, count: 4096) - let bytesRead = read(clientFD, &buffer, buffer.count) - guard bytesRead > 0 else { return } - guard let request = String(bytes: buffer[0..= 2 { path = String(parts[1]) } - } - lastPath = path - - let body = #"{"ok": true, "room": "test", "desired": "stopped"}"# - let response = """ - HTTP/1.1 200 OK\r - Content-Type: application/json\r - Content-Length: \(body.utf8.count)\r - Connection: close\r - - \(body) - """ - _ = response.withCString { cstr in - write(clientFD, cstr, response.utf8.count) - } - } -} - -@MainActor -struct MediaAPITests { - @Test("successful JSON object response does not throw decodeFailed") - func successfulObjectResponse() async throws { - let stub = MediaAPIStubServer() - try stub.start() - defer { stub.stop() } - - // Bypass the https-only parse path: build the config directly. - let base = stub.url! - let config = ServerConfig(baseURL: base, livekitURL: base) - let api = MediaAPI(config: config) - - try await api.control(room: "test", action: "pause") - try await api.report(room: "test", state: "paused", trackID: nil, position: 0, duration: nil) - try await api.play(room: "test", videoID: nil, title: nil, channel: nil, trackID: nil, startPaused: nil) - - #expect(stub.lastRequestPath == "/media/api/play") - } -} diff --git a/Tests/MediaPlaybackTests.swift b/Tests/MediaPlaybackTests.swift deleted file mode 100644 index fd5e70b..0000000 --- a/Tests/MediaPlaybackTests.swift +++ /dev/null @@ -1,89 +0,0 @@ -import Foundation -import Testing -@testable import Voice - -private final class PlaybackProtocol: URLProtocol, @unchecked Sendable { - static let lock = NSLock() - nonisolated(unsafe) static var status = 200 - nonisolated(unsafe) static var requests: [URLRequest] = [] - override class func canInit(with request: URLRequest) -> Bool { true } - override class func canonicalRequest(for request: URLRequest) -> URLRequest { request } - override func startLoading() { - Self.lock.lock() - Self.requests.append(request) - let code = Self.status - Self.lock.unlock() - let response = HTTPURLResponse(url: request.url!, statusCode: code, httpVersion: nil, headerFields: ["Content-Type": "application/json"])! - client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed) - client?.urlProtocol(self, didLoad: Data("{\"ok\":true}".utf8)) - client?.urlProtocolDidFinishLoading(self) - } - override func stopLoading() {} -} - -@Suite(.serialized) -@MainActor struct MediaPlaybackTests { - private func controller(status: Int = 200) -> MediaPlayerController { - PlaybackProtocol.lock.withLock { - PlaybackProtocol.status = status - PlaybackProtocol.requests = [] - } - let configuration = URLSessionConfiguration.ephemeral - configuration.protocolClasses = [PlaybackProtocol.self] - let url = URL(string: "https://media.test")! - let api = MediaAPI(config: ServerConfig(baseURL: url, livekitURL: url), session: URLSession(configuration: configuration)) - return MediaPlayerController(api: api, settings: SettingsStore()) - } - - @Test("standalone playback sends stop to the default server room") - func standaloneStop() async throws { - let player = controller() - await player.stop() - let calls = PlaybackProtocol.lock.withLock { PlaybackProtocol.requests } - let stop = try #require(calls.first { $0.url?.path == "/media/api/control" }) - // URLSession may expose the encoded body as an InputStream to URLProtocol. - let body: Data - if let data = stop.httpBody { body = data } - else { - let stream = try #require(stop.httpBodyStream) - stream.open() - defer { stream.close() } - var data = Data() - var buffer = [UInt8](repeating: 0, count: 1024) - while stream.hasBytesAvailable { - let count = stream.read(&buffer, maxLength: buffer.count) - if count <= 0 { break } - data.append(contentsOf: buffer.prefix(count)) - } - body = data - } - let json = try #require(JSONSerialization.jsonObject(with: body) as? [String: Any]) - #expect(json["room"] as? String == "default") - #expect(json["action"] as? String == "stop") - } - - @Test("failed play requests are visible and a successful retry clears the error") - func failureAndRetry() async { - let player = controller(status: 503) - await player.playTrack(id: "song") - #expect(player.playbackError != nil) - PlaybackProtocol.lock.withLock { PlaybackProtocol.status = 200 } - await player.playTrack(id: "song") - #expect(player.playbackError == nil) - } - - @Test("metadata refreshes when a download becomes ready") - func readyMetadata() { - let player = controller() - func state(_ status: String) -> MediaState { - MediaState(room: "default", track: MediaTrack(id: "song", kind: "youtube", videoID: nil, title: status, channel: nil, duration: 180, status: status, error: nil, audioURL: nil, artURL: nil), desired: "paused", reported: nil, seekHint: nil, choices: [], pending: [], version: 1) - } - player.reconcile(state("downloading")) - player.reconcile(state("ready")) - #expect(player.track?.status == "ready") - #expect(player.track?.title == "ready") - player.resetScope() - #expect(player.track == nil) - #expect(player.player.currentItem == nil) - } -} diff --git a/Tests/MediaReconcileTests.swift b/Tests/MediaReconcileTests.swift deleted file mode 100644 index 55c3761..0000000 --- a/Tests/MediaReconcileTests.swift +++ /dev/null @@ -1,124 +0,0 @@ -import Foundation -import Testing -@testable import Voice - -@MainActor -struct MediaReconcileTests { - private func makeController() -> MediaPlayerController { - let config = ServerConfig( - baseURL: URL(string: "http://127.0.0.1:9")!, - livekitURL: URL(string: "ws://127.0.0.1:9")! - ) - let api = MediaAPI(config: config) - return MediaPlayerController(api: api, settings: SettingsStore()) - } - - private func makeState( - desired: String? = "stopped", - trackID: String? = nil, - audioURL: String? = nil - ) -> MediaState { - let track: MediaTrack? = trackID != nil ? MediaTrack( - id: trackID!, - kind: nil, - videoID: nil, - title: nil, - channel: nil, - duration: nil, - status: nil, - error: nil, - audioURL: audioURL, - artURL: nil - ) : nil - return MediaState( - room: "default", - track: track, - desired: desired, - reported: nil, - seekHint: nil, - choices: [], - pending: [], - version: 1 - ) - } - - @Test("reconcile updates published desired to playing") - func reconcileDesiredPlaying() { - let controller = makeController() - #expect(controller.desired == "stopped") - - controller.reconcile(makeState(desired: "playing")) - #expect(controller.desired == "playing") - } - - @Test("reconcile updates published desired to paused") - func reconcileDesiredPaused() { - let controller = makeController() - controller.reconcile(makeState(desired: "playing")) - #expect(controller.desired == "playing") - - controller.reconcile(makeState(desired: "paused")) - #expect(controller.desired == "paused") - } - - @Test("reconcile updates published desired to stopped") - func reconcileDesiredStopped() { - let controller = makeController() - controller.reconcile(makeState(desired: "playing")) - #expect(controller.desired == "playing") - - controller.reconcile(makeState(desired: "stopped")) - #expect(controller.desired == "stopped") - } - - @Test("reconcile with nil desired defaults to stopped") - func reconcileNilDesiredDefaultsToStopped() { - let controller = makeController() - controller.reconcile(makeState(desired: "playing")) - #expect(controller.desired == "playing") - - controller.reconcile(makeState(desired: nil)) - #expect(controller.desired == "stopped") - } - - @Test("late audio: desired=playing before audio_url, then audio_url arrives") - func lateAudioReconciliation() { - let controller = makeController() - - // Phase 1: server says playing but the track has no audio_url yet - // (still downloading). The source is not wired, so play cannot start. - let stateNoURL = makeState(desired: "playing", trackID: "track-1", audioURL: nil) - controller.reconcile(stateNoURL) - #expect(controller.desired == "playing") - #expect(controller.track?.id == "track-1") - // Player rate should still be zero because no source was wired. - #expect(controller.player.rate.isZero) - - // Phase 2: the same track now has an audio_url. The reconcile path - // wires the source (wireSource) and then applyDesiredState retries - // play because desired is still "playing" and rate is still zero. - let stateWithURL = makeState(desired: "playing", trackID: "track-1", audioURL: "/stream/track-1.mp3") - controller.reconcile(stateWithURL) - #expect(controller.desired == "playing") - // The source should now be wired (loadedAudioPath is set internally). - // We can't directly observe loadedAudioPath, but we can verify the - // player has a current item after wireSource ran. - #expect(controller.player.currentItem != nil) - } - - @Test("startDefaultPolling restarts after endConversation keeps scope") - func pollingRestartsAfterEndConversation() async { - let controller = makeController() - controller.startDefaultPolling() - await controller.endConversation() - controller.startDefaultPolling() - try? await Task.sleep(for: .milliseconds(50)) - } - - @Test("togglePlayPause without a loaded track is a no-op") - func toggleWithoutTrack() async { - let controller = makeController() - await controller.togglePlayPause() - #expect(controller.track == nil) - } -} diff --git a/Tests/PinnedMediaResourceLoaderTests.swift b/Tests/PinnedMediaResourceLoaderTests.swift deleted file mode 100644 index 3c3f306..0000000 --- a/Tests/PinnedMediaResourceLoaderTests.swift +++ /dev/null @@ -1,126 +0,0 @@ -import Foundation -import Testing -@testable import Voice - -/// Focused tests for PinnedMediaResourceLoader.parseTotalLength, covering -/// 200, 206 with known total, 206 with unknown total (*), and malformed -/// Content-Range headers (Finding 1 / 11). -struct PinnedMediaResourceLoaderTests { - - private func makeResponse(status: Int, contentRange: String? = nil, expectedLength: Int64 = -1) -> HTTPURLResponse { - var headers: [String: String] = [:] - if let contentRange { - headers["Content-Range"] = contentRange - } - return HTTPURLResponse( - url: URL(string: "https://example.com/media/test.mp3")!, - statusCode: status, - httpVersion: "HTTP/1.1", - headerFields: headers - )! - } - - @Test("206 with known total parses Content-Range total") - func parse206KnownTotal() { - let response = makeResponse(status: 206, contentRange: "bytes 0-1023/4096", expectedLength: 1024) - #expect(PinnedMediaResourceLoader.parseTotalLength(response) == 4096) - } - - @Test("206 with unknown total (*) returns nil") - func parse206UnknownTotal() { - let response = makeResponse(status: 206, contentRange: "bytes 0-1023/*", expectedLength: 1024) - #expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil) - } - - @Test("200 response returns nil (expectedContentLength is already the full size)") - func parse200ReturnsNil() { - let response = makeResponse(status: 200, expectedLength: 8192) - #expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil) - } - - @Test("206 without Content-Range header returns nil") - func parse206NoContentRange() { - let response = makeResponse(status: 206, expectedLength: 512) - #expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil) - } - - @Test("malformed Content-Range returns nil") - func parseMalformedContentRange() { - let response = makeResponse(status: 206, contentRange: "garbage", expectedLength: 100) - #expect(PinnedMediaResourceLoader.parseTotalLength(response) == nil) - } - - @Test("206 with large total parses correctly") - func parseLargeTotal() { - let response = makeResponse(status: 206, contentRange: "bytes 1048576-2097151/1073741824", expectedLength: 1048576) - #expect(PinnedMediaResourceLoader.parseTotalLength(response) == 1_073_741_824) - } - - @Test("streamURL constructs endpoint without downloading") - func streamURLConstruction() { - let config = ServerConfig( - baseURL: URL(string: "https://192.168.86.2:9178")!, - livekitURL: URL(string: "ws://192.168.86.2:7880")! - ) - let api = MediaAPI(config: config) - let url = api.streamURL(trackID: "track-123") - // streamURL builds under the /media/api base (MediaAPI.baseURL). - #expect(url.absoluteString == "https://192.168.86.2:9178/media/api/stream/track-123") - } - - // MARK: Rewrite / Resolve round-trip (Item 7) - - @Test("rewrite maps https URL to pinned-media scheme and resolve inverts it") - func rewriteResolveRoundTrip() { - let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!) - let original = URL(string: "https://192.168.86.2:9178/media/track-123.mp3")! - let pinned = loader.rewrite(original) - #expect(pinned.scheme == "pinned-media") - #expect(pinned.absoluteString == "pinned-media://192.168.86.2:9178/media/track-123.mp3") - - let resolved = loader.resolve(pinned) - #expect(resolved.absoluteString == original.absoluteString) - } - - @Test("rewrite maps path-only URL using base host and port") - func rewritePathOnly() { - let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!) - let pathURL = URL(string: "/media/track-456.mp3")! - let pinned = loader.rewrite(pathURL) - #expect(pinned.scheme == "pinned-media") - #expect(pinned.absoluteString == "pinned-media://192.168.86.2:9178/media/track-456.mp3") - } - - @Test("resolve leaves non-pinned URLs unchanged") - func resolveNonPinned() { - let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!) - let https = URL(string: "https://example.com/foo.mp3")! - #expect(loader.resolve(https) == https) - } - - // MARK: Asset lifecycle (Item 7) - - @Test("assetWithPinnedURL creates asset and removeLoader clears mapping") - func assetLifecycle() { - let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!) - let pinned = loader.rewrite(URL(string: "https://192.168.86.2:9178/media/test.mp3")!) - let asset = loader.assetWithPinnedURL(pinned) - - // The asset should have a valid URL matching the pinned scheme. - #expect(asset.url.scheme == PinnedMediaResourceLoader.scheme) - - // Remove the loader mapping — subsequent calls should not crash. - loader.removeLoader(asset) - } - - @Test("removeLoader is idempotent (safe to call twice)") - func removeLoaderIdempotent() { - let loader = PinnedMediaResourceLoader(base: URL(string: "https://192.168.86.2:9178")!) - let pinned = loader.rewrite(URL(string: "https://192.168.86.2:9178/media/test.mp3")!) - let asset = loader.assetWithPinnedURL(pinned) - - loader.removeLoader(asset) - // Second call should be a no-op, not a crash. - loader.removeLoader(asset) - } -} diff --git a/UITests/VoiceLiveTests.swift b/UITests/VoiceLiveTests.swift index 15f6978..cd768e4 100644 --- a/UITests/VoiceLiveTests.swift +++ b/UITests/VoiceLiveTests.swift @@ -1,9 +1,8 @@ import XCTest -/// Opt-in tests against the actual NAS deployment. No injected replies or media. +/// Opt-in tests against the actual NAS deployment. No injected replies. final class VoiceLiveTests: XCTestCase { private var app: XCUIApplication! - private var startedMusic = false override func record(_ issue: XCTIssue) { if let app, app.state == .runningForeground { @@ -17,20 +16,13 @@ final class VoiceLiveTests: XCTestCase { } override func tearDownWithError() throws { - defer { app?.terminate() } - guard let app, app.state == .runningForeground else { return } - // Stop only playback initiated by this test, including assertion failures. - // Do not leave the shared default room playing between test runs. - if startedMusic, app.buttons["Stop playback"].exists { - app.buttons["Stop playback"].tap() - } + app?.terminate() } override func setUpWithError() throws { try XCTSkipUnless(ProcessInfo.processInfo.environment["HOPE_RUN_LIVE_TESTS"] == "1", "Requires the live Hope server") continueAfterFailure = false app = XCUIApplication() - startedMusic = false app.launchArguments = ["--ui-testing", "--reset-ui-testing", "--live-testing"] app.launch() } @@ -63,31 +55,6 @@ final class VoiceLiveTests: XCTestCase { XCTAssertTrue(app.buttons["Start chatting"].waitForExistence(timeout: 20)) } - func testLiveLibraryPlayback() { - app.tabBars.buttons["Music"].tap() - let play = app.buttons.matching(NSPredicate(format: "label BEGINSWITH 'Play Sheryl Crow'")).firstMatch - XCTAssertTrue(play.waitForExistence(timeout: 20), "Existing library must load from NAS") - startedMusic = true - play.tap() - XCTAssertTrue(app.buttons["Pause"].waitForExistence(timeout: 30)) - // Elapsed media time proves AVPlayer decoded and advanced, beyond a server desired-state label. - let position = app.sliders["Playback position"] - XCTAssertTrue(position.waitForExistence(timeout: 10)) - let advanced = NSPredicate { _, _ in - let value = position.value as? String ?? "0%" - return value != "0%" && value != "0" && !value.isEmpty - } - expectation(for: advanced, evaluatedWith: position) - waitForExpectations(timeout: 35) - app.buttons["Pause"].tap() - XCTAssertTrue(app.buttons["Play"].waitForExistence(timeout: 10)) - app.buttons["Seek forward 10 seconds"].tap() - capture("Live music playback") - app.buttons["Stop playback"].tap() - XCTAssertFalse(app.buttons["Pause"].exists) - startedMusic = false - } - private func capture(_ name: String) { let attachment = XCTAttachment(screenshot: app.screenshot()) attachment.name = name diff --git a/UITests/VoiceUITests.swift b/UITests/VoiceUITests.swift index 8458fd4..59aad2e 100644 --- a/UITests/VoiceUITests.swift +++ b/UITests/VoiceUITests.swift @@ -48,29 +48,6 @@ final class VoiceUITests: XCTestCase { XCTAssertEqual(app.textFields["Server URL"].value as? String, "https://127.0.0.1:1") } - func testMusicSearchAndStationValidation() { - app.launch() - app.tabBars.buttons["Music"].tap() - XCTAssertTrue(app.buttons["Add radio station"].waitForExistence(timeout: 5)) - XCTAssertTrue(app.staticTexts["Music server unavailable"].waitForExistence(timeout: 10)) - app.segmentedControls.buttons["Search"].tap() - let query = app.textFields["Song, artist, lyrics…"] - query.tap() - query.typeText("Piano") - app.buttons["music.search"].tap() - XCTAssertTrue(app.staticTexts.matching(NSPredicate(format: "label CONTAINS 'connect to the server'")).firstMatch.waitForExistence(timeout: 15)) - app.buttons["Add radio station"].tap() - let url = app.textFields["Audio stream URL"] - XCTAssertTrue(url.waitForExistence(timeout: 5)) - url.tap() - url.typeText("not-a-url") - app.buttons["Add station"].tap() - XCTAssertTrue(app.staticTexts["Enter a complete http:// or https:// audio stream address."].exists) - app.buttons["Cancel"].tap() - XCTAssertTrue(app.buttons["Add radio station"].exists) - capture("Music search") - } - func testTranscriptClearConfirmationAndActivityDetails() { app.launchArguments.append("--ui-fixtures") app.launch() diff --git a/Voice.xcodeproj/project.pbxproj b/Voice.xcodeproj/project.pbxproj index a7d78db..ab68eb4 100644 --- a/Voice.xcodeproj/project.pbxproj +++ b/Voice.xcodeproj/project.pbxproj @@ -17,27 +17,19 @@ 64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */ = {isa = PBXBuildFile; fileRef = A0C23305A6C55A83F8617D9D /* ConversationState.swift */; }; 70E4B860147494D9657E8C01 /* VoiceLiveTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */; }; 720C26EC163AE46713B02DD8 /* TLSPinning.swift in Sources */ = {isa = PBXBuildFile; fileRef = D9B40076934C28BA4B92D808 /* TLSPinning.swift */; }; - 72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */; }; - 73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */; }; 89B47754E5C39418BFD839E6 /* TranscriptHistory.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68A45228EF59186CFC219F2E /* TranscriptHistory.swift */; }; 8DE1363CC0119B5AF9E4E117 /* ConnectionTestView.swift in Sources */ = {isa = PBXBuildFile; fileRef = AC4B9073EE8EAB51F8C55760 /* ConnectionTestView.swift */; }; 943BCE971A01F220FAE9E250 /* TaskPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = E85BACA267ADD462306C9920 /* TaskPanelView.swift */; }; 98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 48F76881B4AE51DEE74511CC /* SettingsView.swift */; }; A5868238CF4958D6D0C5908E /* TaskStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0B60FBBB9C47BB9ADACA7CA /* TaskStore.swift */; }; - A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */; }; A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */; }; - AF719B1668ADA929AA0237BF /* MediaPlaybackTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */; }; - B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */; }; BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */; }; C00D1878C2AB3D8B6F9562B6 /* AgentMonitor.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */; }; - C956D0FDFAC43925979E4077 /* MediaAPI.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9191AD60FF96E44F191864E3 /* MediaAPI.swift */; }; CDE38FE9679472D770C601E6 /* VoiceSession.swift in Sources */ = {isa = PBXBuildFile; fileRef = EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */; }; D18AD347DFF72AB6361165BA /* TokenClient.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9681E7410B684D2D38752A70 /* TokenClient.swift */; }; D2E343C8B8657316A993698E /* TranscriptHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */; }; D9A0059E342955A06A021D6C /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = A6BAD55715F383C0DB5F306F /* LiveKit */; }; - DE1F24919E906EE13319C7A3 /* MediaPanelView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */; }; EE0C5762EC97B123A1880FFB /* Voices.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2E5DF207EDB787F63CB3A64D /* Voices.swift */; }; - F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 10C194656F19E89DFD656A91 /* MediaAPITests.swift */; }; FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */; }; /* End PBXBuildFile section */ @@ -59,26 +51,19 @@ /* End PBXContainerItemProxy section */ /* Begin PBXFileReference section */ - 09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaReconcileTests.swift; sourceTree = ""; }; 0FC167F6658CFEC9546A7E9F /* AgentMonitor.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AgentMonitor.swift; sourceTree = ""; }; - 10C194656F19E89DFD656A91 /* MediaAPITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPITests.swift; sourceTree = ""; }; 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerReplacementCoordinatorTests.swift; sourceTree = ""; }; 2A4C3CE2CA6FF828452803D1 /* Voice.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = Voice.app; sourceTree = BUILT_PRODUCTS_DIR; }; 2E5DF207EDB787F63CB3A64D /* Voices.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Voices.swift; sourceTree = ""; }; 2FED8976B60FBC13BC980FCB /* VoiceApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceApp.swift; sourceTree = ""; }; - 3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlayerController.swift; sourceTree = ""; }; 3F3FA9F420065D2127B0AEC0 /* ConversationView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationView.swift; sourceTree = ""; }; 48F76881B4AE51DEE74511CC /* SettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsView.swift; sourceTree = ""; }; 5759EB570DB87BBC121D946D /* VoiceUITests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceUITests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; 5F6A697985B36B3A1A3F2AD5 /* VoiceLiveTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceLiveTests.swift; sourceTree = ""; }; 61C71C48230C683D199D6F4B /* ControlChannel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ControlChannel.swift; sourceTree = ""; }; 68A45228EF59186CFC219F2E /* TranscriptHistory.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHistory.swift; sourceTree = ""; }; - 68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPanelView.swift; sourceTree = ""; }; 6D0D153557BB1C756ED04C61 /* SettingsStore.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SettingsStore.swift; sourceTree = ""; }; - 6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoader.swift; sourceTree = ""; }; 8309AC6885ED7EB0047AE211 /* VoiceUITests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceUITests.swift; sourceTree = ""; }; - 8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PinnedMediaResourceLoaderTests.swift; sourceTree = ""; }; - 9191AD60FF96E44F191864E3 /* MediaAPI.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaAPI.swift; sourceTree = ""; }; 9681E7410B684D2D38752A70 /* TokenClient.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TokenClient.swift; sourceTree = ""; }; A0C23305A6C55A83F8617D9D /* ConversationState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ConversationState.swift; sourceTree = ""; }; A62BAE22E3A165DCB5DFFEE1 /* OrbView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = OrbView.swift; sourceTree = ""; }; @@ -88,7 +73,6 @@ C22975C69240AACD3418091F /* VoiceTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = VoiceTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; D82D665165A7BC1A4CBCCC6F /* TranscriptHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptHandler.swift; sourceTree = ""; }; D9B40076934C28BA4B92D808 /* TLSPinning.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TLSPinning.swift; sourceTree = ""; }; - E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MediaPlaybackTests.swift; sourceTree = ""; }; E85BACA267ADD462306C9920 /* TaskPanelView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TaskPanelView.swift; sourceTree = ""; }; EB58EEF9490F81FB2E1CE31C /* VoiceSession.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VoiceSession.swift; sourceTree = ""; }; F5683993A1DE677121FFA1F3 /* ServerConfig.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ServerConfig.swift; sourceTree = ""; }; @@ -107,14 +91,6 @@ /* End PBXFrameworksBuildPhase section */ /* Begin PBXGroup section */ - 0D2E974CBC286D7C886BA814 /* Media */ = { - isa = PBXGroup; - children = ( - 68FACB1A5EA6FEFD1D2613E9 /* MediaPanelView.swift */, - ); - path = Media; - sourceTree = ""; - }; 0DF0591DF1D767F89099F62C /* Settings */ = { isa = PBXGroup; children = ( @@ -180,7 +156,6 @@ isa = PBXGroup; children = ( 84ADAF12A43A57888CA178A6 /* Conversation */, - 0D2E974CBC286D7C886BA814 /* Media */, 0DF0591DF1D767F89099F62C /* Settings */, F168B25F8B3C6F36B6E88228 /* Tasks */, ); @@ -197,16 +172,6 @@ ); sourceTree = ""; }; - C12426487989358DE21A5426 /* Media */ = { - isa = PBXGroup; - children = ( - 9191AD60FF96E44F191864E3 /* MediaAPI.swift */, - 3964F565A6C6A06399EBA4DF /* MediaPlayerController.swift */, - 6DC3AC57ED58FC517E212F19 /* PinnedMediaResourceLoader.swift */, - ); - path = Media; - sourceTree = ""; - }; CE49A38E40AAF659A9281B85 /* App */ = { isa = PBXGroup; children = ( @@ -221,10 +186,6 @@ isa = PBXGroup; children = ( BDD0AA8B0E9B45A325D1E5E7 /* ConversationStateTests.swift */, - 10C194656F19E89DFD656A91 /* MediaAPITests.swift */, - E492DE95BFFF99D7AC1E8970 /* MediaPlaybackTests.swift */, - 09FB71841B3D72CD1D69E47A /* MediaReconcileTests.swift */, - 8A20F8AFF1AAC5A944173549 /* PinnedMediaResourceLoaderTests.swift */, 20A82F8996DEA166C94A7956 /* ServerReplacementCoordinatorTests.swift */, FAEF89B2DEEE68F7168A8014 /* TranscriptDedupeTests.swift */, ); @@ -245,7 +206,6 @@ CE49A38E40AAF659A9281B85 /* App */, 8DA8BEF2EE3E0F4181A94EB9 /* Features */, 2820A418F921E18A2FA7E4FB /* LiveKit */, - C12426487989358DE21A5426 /* Media */, 6B2AB34274D9B37E77178C6C /* Networking */, ); path = Sources; @@ -360,11 +320,7 @@ 28659D009399CA9C4FE57B74 /* ControlChannel.swift in Sources */, 64291BBC8D0DAD47E07E94E0 /* ConversationState.swift in Sources */, 36DF44D9A7896F18A064847D /* ConversationView.swift in Sources */, - C956D0FDFAC43925979E4077 /* MediaAPI.swift in Sources */, - DE1F24919E906EE13319C7A3 /* MediaPanelView.swift in Sources */, - 73DA73811F8AEB2AFE340D10 /* MediaPlayerController.swift in Sources */, 3E01A0AF44FD3324FA72C43C /* OrbView.swift in Sources */, - A86EFB492F40C26D4AA315E9 /* PinnedMediaResourceLoader.swift in Sources */, 302BEC55CA9A10E49C307977 /* ServerConfig.swift in Sources */, A94F6C509E3FE215115B2C58 /* SettingsStore.swift in Sources */, 98BA1A0F92DB7B98A8BA7814 /* SettingsView.swift in Sources */, @@ -394,10 +350,6 @@ buildActionMask = 2147483647; files = ( FC696C6E74057D4C93E6DA67 /* ConversationStateTests.swift in Sources */, - F26037AD028ED105E127CBD3 /* MediaAPITests.swift in Sources */, - AF719B1668ADA929AA0237BF /* MediaPlaybackTests.swift in Sources */, - 72F3B4AEC214521015200583 /* MediaReconcileTests.swift in Sources */, - B94FAFA796A9AF6011B03DEC /* PinnedMediaResourceLoaderTests.swift in Sources */, BCAB602C722D74F6C96C5E36 /* ServerReplacementCoordinatorTests.swift in Sources */, 1813A1CB0921D04FCAEC92D0 /* TranscriptDedupeTests.swift in Sources */, ); diff --git a/docs/DEVICE_TESTING.md b/docs/DEVICE_TESTING.md index d0efede..759c97b 100644 --- a/docs/DEVICE_TESTING.md +++ b/docs/DEVICE_TESTING.md @@ -1,7 +1,7 @@ # Device testing checklist -Simulator verification covered connect/disconnect, transcripts (text mode), media -reconciliation/UI, tasks panel, and the hardening pass. The items below need a **real +Simulator verification covered connect/disconnect, transcripts (text mode), tasks +panel, and the hardening pass. The items below need a **real iPhone on the LAN** (`https://192.168.86.2:9178`) because mic input, WebRTC capture, and audio session behavior don't work in this simulator environment (CoreAudio `-4010`, 0 input channels). @@ -31,29 +31,18 @@ ensure the server cert pin is current (see README — re-export after any server side button to lock the device. It only disables the automatic idle timeout. - [ ] After hang-up, auto-lock returns (idle timer re-enabled). -## 4. Audible media playback - -- [ ] The deployed server now serves cached AAC for library tracks requested with - `?format=m4a`; verify actual speaker/headphone output on hardware. -- [ ] Play a library track: audio is audible, now-playing card shows art/title/status. -- [ ] Pause / ±10 s / stop / seek all work; agent mics off while playing - (`media_state` published), mics back on when stopped. -- [ ] End conversation while music was playing before → playback resumes. - -## 5. Audio interruption (phone call / Siri) +## 4. Audio interruption (phone call / Siri) - [ ] Mid-conversation, trigger an interruption (incoming call or "Hey Siri"). -- [ ] Mic is muted on interruption begin; conversation/media pause cleanly. -- [ ] On end (with `shouldResume`), mic and media resume automatically. -- [ ] Media panel: AVPlayer pauses on interruption and resumes after. +- [ ] On end (with `shouldResume`), mic and conversation resume automatically. -## 6. Mic permission denied fallback +## 5. Mic permission denied fallback - [ ] Deny microphone permission in iOS Settings for the app, relaunch. - [ ] Start → session reaches **ready** despite mic failure (non-fatal); orange error banner shows the mic error; text mode remains fully usable. -## 7. General device sanity +## 6. General device sanity - [ ] Connect / hang-up / reconnect matrix (3×) — no crashes, state resets cleanly. - [ ] Change server URL in Settings while idle → session recreates without crash. diff --git a/docs/PROJECT_STATUS.md b/docs/PROJECT_STATUS.md index 9357c0a..81ff42e 100644 --- a/docs/PROJECT_STATUS.md +++ b/docs/PROJECT_STATUS.md @@ -1,13 +1,20 @@ # Project status and handoff -Last updated: 2026-09-09. +Last updated: 2026-09-12. ## Current direction Keep **Qwen 2.5**. The user explicitly canceled the earlier request to switch to a ChatGPT subscription/device-token login. Do not revive that migration because -an old saved goal mentions it. Current work is the native iOS UI/UX redesign, -restoring the music/AI server, and end-to-end qualification. +an old saved goal mentions it. Current work is the native iOS UI/UX redesign +and end-to-end qualification. + +**2026-09-12: the embedded music player was removed** by user decision — the +feature is gone from the app (Media tab, `Sources/Media/`, `MediaPanelView`, +`MediaAPI`, `MediaPlayerController`, `PinnedMediaResourceLoader`, media tests, +and the `media_state` control publish). The backend media API still exists and +is untouched. The backend AAC compatibility patch (below) is now irrelevant to +the iOS app but remains deployed. The UI and playback changes are implemented, and the NAS deployment was verified on September 8. Full app qualification is **not complete**: physical-device tests @@ -19,6 +26,8 @@ server health and device availability must be rechecked when resuming. ### Native interface and usability - Reorganized the app into native **Talk, Music, Activity** tabs, sharing one session. + (The Music tab was removed again on 2026-09-12 by user decision; the app now has + **Talk** and **Activity** tabs.) - Replaced the large animated orb with a compact waveform/status presentation; added a clearer welcome screen, Voice/Type selection before connecting, explicit start/end controls, microphone meter, and recoverable connection errors. @@ -27,7 +36,7 @@ server health and device availability must be rechecked when resuming. - Reworked Settings around voice, input mode, model, and connection; server changes require an explicit validated save. Voice selection persists. - Improved Music library/search navigation, now-playing controls, seek/volume, - empty/loading/error states, and validated station-entry sheet. + empty/loading/error states, and validated station-entry sheet. (Removed 2026-09-12.) - Improved Activity hierarchy and task detail presentation with an active-task badge. - Removed forced dark appearance; adopted adaptive system styling, Dynamic Type, and larger interaction targets, informed by the Impeccable native-iOS guidance. @@ -35,14 +44,11 @@ server health and device availability must be rechecked when resuming. ### Functional fixes - Observed nested transcript/state/task objects so asynchronous updates render. -- Preserved latest-wins server replacement; clear old media when switching servers. -- Sent standalone media controls and reports to the default server room. -- Refreshed track metadata as downloads become ready; exposed request/decoder errors. -- Loaded artwork through pinned TLS for the NAS self-signed certificate. -- Requested cached AAC (`?format=m4a`) for YouTube library tracks on iOS. -- Corrected AVFoundation's content type to a UTI rather than an HTTP MIME string. -- Changed the pinned media loader to deliver 256 KB byte ranges instead of buffering - an entire song before delivering usable audio, addressing a reproduced timeout. +- Preserved latest-wins server replacement. + (All media-related behaviors — media clearing on server switch, standalone media + controls/reports, track metadata refresh, pinned-TLS artwork, cached-AAC requests, + UTI content-type fix, bounded-range media loader — were removed with the music + player on 2026-09-12.) ### Tests and tooling @@ -50,9 +56,10 @@ server health and device availability must be rechecked when resuming. - Added the `VoiceUITests` target in `project.yml` and regenerated the Xcode project. - Added isolated DEBUG UI-test preferences and optional fixtures; normal application preferences are not reset by those UI-test launch arguments. -- Added media-control/error/metadata tests and opt-in real-NAS UI tests. +- Added media-control/error/metadata tests and opt-in real-NAS UI tests. (Media tests + were removed with the music player on 2026-09-12.) - Live tests retain failure screenshots/UI hierarchy and attempt to stop music - they initiated even when an assertion fails. + they initiated even when an assertion fails. (Music playback test removed 2026-09-12.) ## Backend deployment and cross-repository dependency @@ -87,15 +94,14 @@ Do not copy credentials or `.env` contents into documentation or commits. Project `Voice.xcodeproj`, scheme `Voice_iOS`, iPhone 17 Pro simulator. -- 68 unit tests passed; five offline UI tests passed (navigation/error recovery, - settings persistence/validation, music validation, transcript/task UI, large text). +- 49 unit tests passed (after the 2026-09-12 music-player removal; previously 68 with + media tests); four offline UI tests passed (navigation/error recovery, + settings persistence/validation, transcript/task UI, large text). - Real NAS tests verified Qwen replies, background task completion, reconnect, library loading, advancing playback time, pause, forward-seek control and stop. + (Playback evidence predates the music-player removal; the feature no longer exists + in the app.) - Qwen/task/reconnect passed three consecutive iterations. -- Music initially failed intermittently. Logs captured an AVPlayer timeout while - buffering a whole 6 MB response. After bounded-range delivery, three consecutive - live playback iterations passed. The earlier isolated agent-join failure remains - unexplained; do not claim all intermittent connection issues are resolved. - Backend integration: 16 passed, 7 skipped, 2 deselected. - Backend AAC regression: one passed (conversion, cache reuse, original preservation, and byte ranges); deployed range request returned HTTP 206. @@ -115,19 +121,20 @@ Artifacts live on this Mac under the XcodeBuildMCP workspace, not in Git. 2. **Build, sign, install and run on that iPhone.** No successful device install is recorded. Check Developer Mode, trust, signing team/profile, and LAN access. 3. **Qualify real audio:** microphone capture, TTS audibility, speaker/headphones, - barge-in, media/voice handoff, Siri/phone interruptions, lock/background behavior, + barge-in, Siri/phone interruptions, lock/background behavior, microphone-permission-denied fallback, and network-loss recovery. Some checks require the user's speech/listening or a real interruption; simulator state tests do not prove them. Follow [DEVICE_TESTING.md](DEVICE_TESTING.md). -4. **Broader regression:** rerun the full suite after the range-loader change; test - longer playback, seek near EOF, slow network/cancellation and server replacement. - Exercise new downloads, test-owned station add/play/remove, and voice/model - selection against the live agent. Do not delete the user's existing library song. +4. **Broader regression:** rerun the full suite after the music-player removal; + test slow network/cancellation and server replacement. Exercise voice/model + selection against the live agent. 5. **Final visual/accessibility pass:** inspect all populated/error screens in light and dark mode, large text and small-device layouts; test VoiceOver and focus. The existing large-text navigation test is not a full accessibility audit. 6. **Publish the backend compatibility source safely** in the sibling repository and ensure its normal deployment preserves mounts, certificates and Qwen settings. + (Lower priority now: the iOS app no longer consumes the media API, but the deployed + container still serves it for the web UI.) 7. Update this report with results and remaining limitations. Do not mark the whole app qualified until the outstanding end-to-end gates have evidence.