fix(apple/iOS): immersive streaming — edge-to-edge, no status bar, hidden cursor, native default mode
ci / rust (push) Has been cancelled

Streaming on iPad left the status bar up and the video boxed inside the safe areas, on
top of a 16:9 default mode letterboxing on the 4:3 screen, with the iPadOS cursor
hovering over the video. The session view is now immersive on iOS:

- .ignoresSafeArea + .statusBarHidden + .persistentSystemOverlays(.hidden) for the
  session only (home gets its chrome back on disconnect).
- First run seeds the stream mode from the device's native screen
  (UIScreen.nativeBounds + maximumFramesPerSecond) instead of 1920×1080 — verified
  live: a fresh install negotiated the iPad's 2752×2064 with the host. macOS keeps the
  1080p default (a desktop window is not the screen).
- The iPadOS cursor hides while over the video (UIPointerInteraction .hidden(),
  re-resolved on capture toggles) — the host renders its own cursor from our deltas;
  true pointer lock through UIHostingController remains the documented gap.

Found along the way (host-side, not fixed here): at very high modes a keyframe burst
can fill the UDP send buffer and m3 treats the sendmmsg WouldBlock as fatal
("session ended with error: submit_frame: WouldBlock") instead of backpressuring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-06-11 12:44:29 +02:00
parent 57f7e32c24
commit 154da2dc58
3 changed files with 51 additions and 8 deletions
+8 -5
View File
@@ -188,11 +188,14 @@ signing, bundle id `io.unom.punktfunk`. Notes:
pickers are macOS-only). For the iPad-with-external-display setup: the target pickers are macOS-only). For the iPad-with-external-display setup: the target
enables multiple scenes + indirect input events — on Stage Manager iPads, drag the enables multiple scenes + indirect input events — on Stage Manager iPads, drag the
punktfunk window onto the external screen and the stream runs there with full punktfunk window onto the external screen and the stream runs there with full
keyboard/mouse/touch. Known gaps: `prefersPointerLocked` is declared on the stream keyboard/mouse/touch. While streaming the session is immersive (edge-to-edge,
view controller but UIHostingController doesn't forward the preference from status bar + home indicator hidden) and the iPadOS cursor is hidden over the video
representable children, so the system cursor stays visible (relative-mouse (`UIPointerInteraction` `.hidden()` — visible again when ⌘⎋ releases capture); on
forwarding works regardless — fixing it means putting the controller into the UIKit iOS first run the stream mode defaults to the device's native screen so the video
presentation chain, e.g. a full-screen UIKit presentation on session start); and fills the display. Known gaps: true pointer LOCK (`prefersPointerLocked`) isn't
consulted through UIHostingController, so the hidden cursor can still drift onto a
second screen (fixing it means putting the controller into the UIKit presentation
chain); and
AVAudioSession interruptions (calls, Siri) don't auto-restart the audio engines yet AVAudioSession interruptions (calls, Siri) don't auto-restart the audio engines yet
(reconnect recovers). (reconnect recovers).
@@ -40,7 +40,10 @@ struct ContentView: View {
home home
} }
} }
.onAppear { autoConnectIfAsked() } .onAppear {
seedDefaultModeIfNeeded()
autoConnectIfAsked()
}
.onDisappear { model.disconnect() } // window closed mid-session (Cmd+N spawns more) .onDisappear { model.disconnect() } // window closed mid-session (Cmd+N spawns more)
// On the outer Group so the sheet survives the trust-prompt home transition // On the outer Group so the sheet survives the trust-prompt home transition
// (the "Pair with PIN instead" path disconnects first the host's accept loop // (the "Pair with PIN instead" path disconnects first the host's accept loop
@@ -77,8 +80,15 @@ struct ContentView: View {
} }
#if os(macOS) #if os(macOS)
.frame(minWidth: 640, minHeight: 360) .frame(minWidth: 640, minHeight: 360)
#endif
.background(Color.black) .background(Color.black)
#else
// Streaming is immersive: edge-to-edge under the status bar and home
// indicator, both hidden for the session (they return with the hosts grid).
.background(Color.black)
.ignoresSafeArea()
.statusBarHidden(true)
.persistentSystemOverlays(.hidden)
#endif
} }
// MARK: - Home (hosts grid) // MARK: - Home (hosts grid)
@@ -243,6 +253,21 @@ struct ContentView: View {
} }
} }
/// First run on iOS: default the stream mode to this device's native screen so the
/// video fills the display instead of letterboxing 1920×1080 onto a 4:3 iPad. (The
/// compiled-in AppStorage defaults only apply until any value is saved; macOS keeps
/// 1080p a desktop window is not the screen.)
private func seedDefaultModeIfNeeded() {
#if os(iOS)
let defaults = UserDefaults.standard
guard defaults.object(forKey: "punktfunk.width") == nil else { return }
let bounds = UIScreen.main.nativeBounds // portrait-oriented pixels
defaults.set(Int(max(bounds.width, bounds.height)), forKey: "punktfunk.width")
defaults.set(Int(min(bounds.width, bounds.height)), forKey: "punktfunk.height")
defaults.set(UIScreen.main.maximumFramesPerSecond, forKey: "punktfunk.hz")
#endif
}
private func connect(_ host: StoredHost) { private func connect(_ host: StoredHost) {
model.connect( model.connect(
to: host, to: host,
@@ -66,12 +66,13 @@ public struct StreamView: UIViewControllerRepresentable {
} }
} }
public final class StreamViewController: UIViewController { public final class StreamViewController: UIViewController, UIPointerInteractionDelegate {
public private(set) var connection: PunktfunkConnection? public private(set) var connection: PunktfunkConnection?
private var pump: StreamPump? private var pump: StreamPump?
private var inputCapture: InputCapture? private var inputCapture: InputCapture?
private var captured = false private var captured = false
private var observers: [NSObjectProtocol] = [] private var observers: [NSObjectProtocol] = []
private var pointerInteraction: UIPointerInteraction?
var onCaptureChange: ((Bool) -> Void)? var onCaptureChange: ((Bool) -> Void)?
@@ -89,6 +90,19 @@ public final class StreamViewController: UIViewController {
public override func loadView() { public override func loadView() {
view = StreamLayerUIView() view = StreamLayerUIView()
// Hide the iPadOS cursor while it hovers the video: the host renders its own
// cursor from our raw deltas, so the local one only diverges from it. (True
// pointer LOCK prefersPointerLocked isn't consulted through
// UIHostingController; this hides the pointer without locking it.)
let interaction = UIPointerInteraction(delegate: self)
view.addInteraction(interaction)
pointerInteraction = interaction
}
public func pointerInteraction(
_ interaction: UIPointerInteraction, styleFor region: UIPointerRegion
) -> UIPointerStyle? {
captured ? .hidden() : nil
} }
public override var prefersPointerLocked: Bool { captured } public override var prefersPointerLocked: Bool { captured }
@@ -168,6 +182,7 @@ public final class StreamViewController: UIViewController {
captured = false captured = false
} }
setNeedsUpdateOfPrefersPointerLocked() setNeedsUpdateOfPrefersPointerLocked()
pointerInteraction?.invalidate() // re-resolve the hidden/visible pointer style
let onCaptureChange = onCaptureChange let onCaptureChange = onCaptureChange
let captured = captured let captured = captured
DispatchQueue.main.async { onCaptureChange?(captured) } DispatchQueue.main.async { onCaptureChange?(captured) }