feat(apple): iOS/iPadOS client — touch, pointer lock, shared SwiftUI shell
ci / rust (push) Has been cancelled
ci / rust (push) Has been cancelled
The whole client now runs on iPadOS/iOS from the same sources, first-lit live in the
iPad simulator against the real host at 1280x720@60 (60 fps on the HUD, capture state
machine active, mic permission flow shown).
- PunktfunkCore.xcframework grows iOS device + universal-simulator slices
(BUILD_IOS=1; rustup targets aarch64-apple-ios{,-sim} + x86_64-apple-ios).
- The decode pump is extracted into a shared StreamPump (identical IDR re-gate logic on
both platforms); the iOS StreamView (StreamViewIOS.swift) has the same name/signature
as the macOS one, so ContentView & co. are byte-identical across platforms — hosted
in a UIViewController for prefersPointerLocked (the iPadOS cursor capture; see README
note 9 for the UIHostingController forwarding caveat).
- Touch is always forwarded: per-finger wire ids, coordinates mapped through the
aspect-fit letterbox into LIVE host-mode pixels (surface == host mode, identity
rescale host-side; follows mid-stream requestMode switches).
- InputCapture is cross-platform: GC works the same on iPadOS, ⌘⎋ is detected from the
HID stream there; stale-⌘ tracking after focus loss fixed on both platforms
(releaseAll now drops the modifier/latch state — a ⌘ released in another app
otherwise hijacked Esc forever).
- SessionAudio: AVAudioSession on iOS (.playAndRecord + .defaultToSpeaker — without it
iPhones route host audio to the EARPIECE; deactivated with
notifyOthersOnDeactivation on stop so interrupted background audio resumes); HAL
device pinning + the Settings pickers stay macOS-only.
- New Punktfunk-iOS app target (shared synchronized sources, generated Info.plist with
mic + local-network usage descriptions — QUIC to a LAN host trips local network
privacy on real devices — scene manifest + indirect input events for Stage Manager /
external displays), shared scheme, macOS min-window frames gated off iOS.
For the iPad-on-an-external-screen idea: with multiple scenes + indirect input enabled,
Stage Manager iPads can drag the punktfunk window onto the external display and drive
the PC with keyboard/mouse/touch. Known gaps (README note 9): the pointer-lock
preference isn't consulted through UIHostingController (relative mouse works, the local
cursor just stays visible) and AVAudioSession interruptions don't auto-restart audio.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
// The platform-independent heart of the presenters: one thread pulling AUs from the
|
||||
// connection into an AVSampleBufferDisplayLayer, with the format description refreshed
|
||||
// on every IDR (the host opens with an IDR carrying in-band parameter sets; recovery
|
||||
// keyframes re-send them — there is no out-of-band extradata, ever). Shared by the
|
||||
// macOS StreamLayerView and the iOS/iPadOS stream view.
|
||||
|
||||
import AVFoundation
|
||||
import Foundation
|
||||
|
||||
/// Cancellation handle owned by exactly one pump thread — a restart hands the old pump
|
||||
/// its own token, so it can never be revived by a newer start().
|
||||
private final class PumpToken: @unchecked Sendable {
|
||||
private let lock = NSLock()
|
||||
private var live = true
|
||||
var isLive: Bool {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
return live
|
||||
}
|
||||
func cancel() {
|
||||
lock.lock()
|
||||
live = false
|
||||
lock.unlock()
|
||||
}
|
||||
}
|
||||
|
||||
/// One pump per instance; create a fresh StreamPump per start (cancel is permanent).
|
||||
final class StreamPump {
|
||||
private let token = PumpToken()
|
||||
|
||||
/// Pump thread: pull AUs, wrap, enqueue. Non-IDR AUs before the first format
|
||||
/// description are dropped. `onFrame`/`onSessionEnd` fire on the pump thread.
|
||||
func start(
|
||||
connection: PunktfunkConnection,
|
||||
layer: AVSampleBufferDisplayLayer,
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)?,
|
||||
onSessionEnd: (@Sendable () -> Void)?
|
||||
) {
|
||||
let token = token
|
||||
layer.flush() // drop any frames a previous connection left queued
|
||||
|
||||
let thread = Thread {
|
||||
var format: CMVideoFormatDescription?
|
||||
while token.isLive {
|
||||
do {
|
||||
guard let au = try connection.nextAU(timeoutMs: 100) else { continue }
|
||||
onFrame?(au)
|
||||
if let f = AnnexB.formatDescription(fromIDR: au.data) {
|
||||
format = f // refreshed on every IDR (mode changes included)
|
||||
}
|
||||
if layer.status == .failed {
|
||||
// Decode wedged: flush and re-gate on the next in-band parameter
|
||||
// sets — resuming with a delta frame can't recover. (A
|
||||
// request-IDR channel on punktfunk/1 is a host-side TODO; with the
|
||||
// host's infinite GOP this may otherwise stay black until the
|
||||
// next recovery keyframe.)
|
||||
layer.flush()
|
||||
format = AnnexB.formatDescription(fromIDR: au.data)
|
||||
}
|
||||
guard let f = format,
|
||||
let sample = AnnexB.sampleBuffer(au: au, format: f),
|
||||
token.isLive // don't enqueue a stale frame after a restart
|
||||
else { continue }
|
||||
layer.enqueue(sample)
|
||||
} catch {
|
||||
if token.isLive {
|
||||
onSessionEnd?()
|
||||
}
|
||||
break // session closed
|
||||
}
|
||||
}
|
||||
}
|
||||
thread.name = "punktfunk-pump"
|
||||
thread.qualityOfService = .userInteractive
|
||||
thread.start()
|
||||
}
|
||||
|
||||
/// Stop pumping (≤ one poll timeout). Does not close the connection.
|
||||
func stop() {
|
||||
token.cancel()
|
||||
}
|
||||
|
||||
deinit { token.cancel() }
|
||||
}
|
||||
Reference in New Issue
Block a user