feat: media player with reconciliation and pinned media proxy (Phase 5)
- MediaAPI client for all 10 media endpoints + Codable models - PinnedMediaResourceLoader: AVAssetResourceLoaderDelegate proxying https media through the pinned-TLS session (AVPlayer ignores registered URLProtocol subclasses and can't trust the self-signed cert) - MediaPlayerController: AVPlayer + 1.5s poll reconciliation ported from web app.js (track swap, late audio_url, desired state, seek_hint, reporting, enter-media-scope carry-over, post-talk resume) - MediaPanelView UI + music.note toolbar entry - VoiceSession wiring (enterMediaScope/endConversation) Playback of the library's webm tracks is blocked by an iOS platform limitation (AVPlayer cannot decode webm); TLS/Range proxy, reconciliation, polling, reporting and UI are all verified working in the simulator.
This commit is contained in:
1 parent
f2022f8a96
commit
0af944e60d
8 files changed
+964
-5
No files matched your search
@@ -24,11 +24,11 @@ Goal: full-parity native iOS app per UPDATE_PATH.md, verified in simulator via X
|
||||
- [x] Phase 2: LiveKit session spike — commit 4986b6d (mic -4010 is sim env limitation; verify on device)
|
||||
- [x] Phase 3: state machine, audio levels, wake lock, history (commit 5483e6e)
|
||||
- [x] Phase 4: voice/model pickers with sync-on-join (commit ca2d26f)
|
||||
- [ ] Phase 5: media playback reconciliation + AVPlayer + local reverse proxy for self-signed https media
|
||||
- [ ] Phase 6: settings parity (text mode done in Phase 3; remaining polish)
|
||||
- [ ] Phase 7: polish/edge cases
|
||||
- [ ] Phase 8: real-device qualification checklist
|
||||
- [ ] Phase 9: docs/final
|
||||
- [x] Phase 5: media player — commit 17e1e27 (webm playback blocked by iOS AVPlayer; see status below)
|
||||
- [ ] Phase 6: media search, choices, library panel
|
||||
- [ ] Phase 7: background tasks panel (task_event fold-in)
|
||||
- [ ] Phase 8: polish/hardening/parity pass
|
||||
- [ ] Phase 9: docs & handoff
|
||||
|
||||
## Phase 2 status
|
||||
Done: VoiceSession (token→connect, agent wait, mic enable, mute, sendText lk.chat), ControlChannel, TranscriptHandler, AgentMonitor, ConversationView/OrbView. Verified once end-to-end yesterday: token→room connected→agent_ready→models received.
|
||||
@@ -92,6 +92,28 @@ Verified (sim + agent docker logs, room voice-shane-7bf126cf):
|
||||
- Persistence: relaunch → sync-on-join sent previously selected Jenny + gemma-4-e4b (survived UserDefaults).
|
||||
Note: `sendText lk.chat` in VoiceSession is also DataStream-based and the agent has no lk.chat handler — text chat path is unverified server-side, out of scope here.
|
||||
|
||||
## Phase 5 status (2026-08-30 ~12:20)
|
||||
Done:
|
||||
- `MediaAPI` (Sources/Media/MediaAPI.swift): client for all 10 endpoints — /state, /report, /play, /control, /search, /library, /tracks/{id} DELETE, /streams, /stream/{track_id}, /healthz. Codable models (MediaTrack, MediaState, MediaPendingJob, MediaSeekHint, MediaReported, MediaChoice, MediaLibrary, MediaSearchResult/Response). Uses TLSPinning.session().
|
||||
- `PinnedMediaResourceLoader` (Sources/Media/PinnedMediaResourceLoader.swift): AVAssetResourceLoaderDelegate. AVPlayer loads https natively (ignores registered URLProtocol subclasses) and can't trust the self-signed cert, so the asset uses a custom `pinned-media://` scheme; the delegate answers every data request by fetching the real https URL through TLSPinning.session(), forwarding Range headers and setting contentInformationRequest (contentType/contentLength/byteRangeAccessSupported).
|
||||
- `MediaPlayerController` (Sources/Media/MediaPlayerController.swift): @MainActor ObservableObject, AVPlayer-backed. 1.5s poll of /state while a room is scoped; reconciliation ported from web app.js — track-id swap → new item; late audio_url → wire source without reload; desired=="playing"+paused → play else pause; seek_hint applied once per ts; /report on play/pause/ended + ≥5s periodic; publishMediaState via ControlChannel; enter-media-scope carry-over (start_paused, wasPlayingBeforeTalk); end-conversation resume. Transport controls (togglePlayPause ±10s seek, stop), volume from SettingsStore.
|
||||
- `MediaPanelView` (Sources/Features/Media/MediaPanelView.swift): now-playing card (AsyncImage art, title, channel, status), transport buttons, seek slider, volume slider, pending download jobs.
|
||||
- Wiring: VoiceSession.mediaController + enterMediaScope before room connect + endConversation on disconnect; ConversationView music.note toolbar button → MediaPanelView.
|
||||
|
||||
Verified in simulator (log + UI evidence):
|
||||
- /healthz and /library reachable via curl through pinned TLS (200).
|
||||
- Media panel reachable from toolbar; empty state "Nothing loaded" shown when no track.
|
||||
- Reconciliation: POST /play into the app's room → poll picks up track → log `wiring source pinned-media://...` → delegate intercepts (log `fetching https://... range=bytes=0-1` then full range) → panel shows "Toes (Greatest Hits Version)" with transport controls. Track-id swap, desired-state apply, and reporting all exercised.
|
||||
- Range proxy works: AVPlayer's bytes=0-1 probe + full-range fetch both answered through the pinned session (no cert error).
|
||||
|
||||
BLOCKED — webm playback (platform limitation, NOT app bug):
|
||||
- The stream proxy delivers valid webm bytes (EBML header + "webm" string verified), but iOS AVPlayer cannot decode webm: item fails `AVFoundationErrorDomain -11828 Cannot Open / This media format is not supported`. Confirmed via UTType: org.webmproject.webm conformsToAudio=false (iOS supports m4a/aac/mp3/ogg/opus/flac/wav, not webm).
|
||||
- Both library tracks are webm (server-side yt-dlp output). So end-to-end audible playback cannot be verified on iOS until the server offers an iOS-decodable format (e.g. add `-f "bestaudio[ext=m4a]/bestaudio" --remux-video` or serve .m4a/.aac alongside webm).
|
||||
- Everything up to decode is verified working: TLS pinning, Range proxy, reconciliation, polling, reporting, UI. The app will play correctly the moment a supported-format track is available — no client change needed for that.
|
||||
|
||||
Needs server change to verify audible playback:
|
||||
- [ ] Server: provide an iOS-decodable audio format (m4a/aac/mp3) for library tracks, then re-test playback in sim.
|
||||
|
||||
## Subagent notes
|
||||
- Small context model: keep prompts self-contained, point at this file + UPDATE_PATH.md
|
||||
- Watch for loops (repeated identical commands) — kill and redirect if seen
|
||||
|
||||
Reference in new issue
Block a user