feat(apple): gamepad UI v2 — controller settings + add host, aurora, macOS
Sources reorganized (client: Home/Session/Settings/Stores/Support/Trust; kit: Audio/Connection/Gamepad/Input/Support/Video/Views) with the big files split along the same seams. The gamepad mode is couch-complete, and now on macOS too (the living-room Mac case), not just iOS/iPadOS: - GamepadSettingsView: a console-style, fully controller-navigable settings screen (X from the launcher) — up/down moves focus, left/right steps values (clamped, boundary thud), A cycles/toggles, B closes; the focused row shows a one-line description. Backed by GamepadMenuList, the vertical sibling of GamepadCarousel, and SettingsOptions — the option lists hoisted out of SettingsView statics and shared by the touch, tvOS and gamepad settings. - GamepadAddHostView + GamepadKeyboard: register a host end to end with a pad — field rows open an on-screen controller keyboard (dpad grid, A types, X backspaces, B done); the launcher carousel ends in an Add Host tile, so the dead-end "add one with touch first" empty state is gone. - Launcher polish: contextual hint bar with the pad's real button glyphs, controller name + battery chip, one shared console chrome. - GamepadScreenBackground: an animated aurora (TimelineView-driven drifting blobs in the brand's violet family, breathing radii, slow hue shift, legibility scrim; freezes under Reduce Motion). Pure SwiftUI on purpose — a .metal library only bundles reliably in one of the two build systems (SPM vs the xcodeproj's synced folders) these sources compile under. - macOS port: settings/add-host/library present as sized sheets (a macOS sheet takes its content's IDEAL size, and the GeometryReader-driven screens collapsed to nothing), NSScreen-based mode lists, scroll indicators .never (the "always show scroll bars" setting overrides .hidden), tray scrims so scrolled rows dim under the pinned title/hints, extra title clearance, and a PUNKTFUNK_FORCE_GAMEPAD_UI=1 dev hook — launcher/settings/add-host/keyboard/ library render-verified live on a real Mac + LAN hosts. - GamepadMenuInput: X button support, and (re)start now snapshots held buttons so a controller handoff press never fires twice (the B that closed the keyboard no longer also cancels the screen underneath). - Cleanups: one "Connection failed" alert in ContentView instead of one per home screen; HostDiscovery.advertises/unsaved shared by both home screens. - host: can_encode_444 stub for the non-Linux/Windows host build (the macOS synthetic-source loopback used by the Swift tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,612 @@
|
||||
// SwiftUI presentation: AVSampleBufferDisplayLayer fed straight from the punktfunk/1 connection.
|
||||
//
|
||||
// Stage-1 presenter (see README): the layer accepts *compressed* HEVC sample buffers and
|
||||
// does hardware decode + display itself — fastest path to pixels, IOSurface-backed
|
||||
// zero-copy on Apple silicon. Stage 2 (explicit VTDecompressionSession + CAMetalLayer)
|
||||
// replaces this when we start tuning frame pacing / measuring glass-to-glass.
|
||||
//
|
||||
// The view also owns the input-capture state machine (Moonlight-style): capture is a
|
||||
// deliberate, reversible state — engaged when the stream starts and when the user clicks
|
||||
// into the video, released by ⌘⎋ or focus loss, and NEVER engaged by mere app
|
||||
// activation (the click that activates the window may be a title-bar drag or a resize —
|
||||
// warping the cursor there is exactly the intrusiveness this design removes). While
|
||||
// released, nothing is forwarded to the host and the local cursor is free.
|
||||
//
|
||||
// macOS-first (NSViewRepresentable); the iOS variant is the same layer under
|
||||
// UIViewRepresentable.
|
||||
|
||||
#if os(macOS)
|
||||
import AppKit
|
||||
import AVFoundation
|
||||
import SwiftUI
|
||||
import os
|
||||
|
||||
/// Same diagnostic switch as InputCapture: PUNKTFUNK_INPUT_DEBUG=1 logs when the macOS
|
||||
/// NSEvent mouse monitor (relative motion + buttons) is installed/removed, so the user can
|
||||
/// confirm the new motion path is actually live for a session.
|
||||
private let streamInputLog = Logger(subsystem: "io.unom.punktfunk", category: "input")
|
||||
private let streamInputDebug =
|
||||
ProcessInfo.processInfo.environment["PUNKTFUNK_INPUT_DEBUG"] == "1"
|
||||
|
||||
/// Hides the LOCAL cursor while captured. The host renders its own cursor, and the local
|
||||
/// one both diverges from it (the host applies acceleration/clamping to our deltas) and
|
||||
/// can wander out of the window — a click there would focus another app. So while captured
|
||||
/// we do what Moonlight does: warp the cursor into the view, freeze it
|
||||
/// (`CGAssociateMouseAndMouseCursorPosition(false)` — under which NSEvent mouseMoved/
|
||||
/// dragged deltas become the relative motion StreamLayerView forwards), and hide it.
|
||||
/// hide/unhide and associate are balanced via `captured`.
|
||||
///
|
||||
/// In CLIENT-SIDE-CURSOR mode (gamescope, whose capture carries no host cursor) this is a
|
||||
/// no-op: the local cursor stays visible and free, and StreamLayerView forwards ABSOLUTE
|
||||
/// positions instead — the visible system cursor IS the on-screen cursor. `disassociate`
|
||||
/// selects between the two; `release()` only undoes what `capture` actually did.
|
||||
private final class CursorCapture {
|
||||
private var captured = false
|
||||
/// Whether the engaged capture actually disassociated+hid (false in cursor-visible mode),
|
||||
/// so `release()` only restores when it must.
|
||||
private var disassociated = false
|
||||
|
||||
/// Returns whether capture actually engaged. It can fail mid app-activation — the click
|
||||
/// that reactivates the app delivers `mouseDown` before the app is frontmost, and
|
||||
/// `CGAssociateMouseAndMouseCursorPosition` is refused then — so the caller must stay
|
||||
/// released and let the NEXT click retry, never latching a half-captured state. With
|
||||
/// `disassociate: false` (cursor-visible mode) it always engages — there is no grab to
|
||||
/// be refused, the cursor stays free and visible.
|
||||
func capture(in view: NSView, disassociate: Bool) -> Bool {
|
||||
guard !captured, let window = view.window, view.bounds.width > 0 else { return false }
|
||||
if disassociate {
|
||||
// Park the cursor mid-view so a click can't land in (and activate) another app.
|
||||
let rectOnScreen = window.convertToScreen(view.convert(view.bounds, to: nil))
|
||||
let primaryHeight = NSScreen.screens.first?.frame.height ?? 0
|
||||
CGWarpMouseCursorPosition(
|
||||
CGPoint(x: rectOnScreen.midX, y: primaryHeight - rectOnScreen.midY))
|
||||
guard CGAssociateMouseAndMouseCursorPosition(0) == .success else { return false }
|
||||
NSCursor.hide()
|
||||
}
|
||||
captured = true
|
||||
disassociated = disassociate
|
||||
return true
|
||||
}
|
||||
|
||||
func release() {
|
||||
guard captured else { return }
|
||||
if disassociated {
|
||||
CGAssociateMouseAndMouseCursorPosition(1)
|
||||
NSCursor.unhide()
|
||||
}
|
||||
captured = false
|
||||
disassociated = false
|
||||
}
|
||||
}
|
||||
|
||||
public struct StreamView: NSViewRepresentable {
|
||||
private let connection: PunktfunkConnection
|
||||
private let captureEnabled: Bool
|
||||
private let onCaptureChange: ((Bool) -> Void)?
|
||||
private let onFrame: (@Sendable (AccessUnit) -> Void)?
|
||||
private let onSessionEnd: (@Sendable () -> Void)?
|
||||
private let presentMeter: LatencyMeter?
|
||||
private let presentTailMeter: LatencyMeter?
|
||||
|
||||
/// `onFrame`/`onSessionEnd` fire on the pump thread — hop to the main actor for UI.
|
||||
/// `captureEnabled: false` disables input capture entirely while UI (e.g. a trust
|
||||
/// prompt) is layered over the stream; flipping it to true auto-engages capture
|
||||
/// once. `onCaptureChange` (main thread) reports engage/release — drive the HUD's
|
||||
/// "click to capture" / "⌘⎋ releases" hint with it. `presentMeter` records capture→present
|
||||
/// and `presentTailMeter` decode→present when the stage-2 presenter is active.
|
||||
public init(
|
||||
connection: PunktfunkConnection,
|
||||
captureEnabled: Bool = true,
|
||||
onCaptureChange: ((Bool) -> Void)? = nil,
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
|
||||
onSessionEnd: (@Sendable () -> Void)? = nil,
|
||||
presentMeter: LatencyMeter? = nil,
|
||||
presentTailMeter: LatencyMeter? = nil
|
||||
) {
|
||||
self.connection = connection
|
||||
self.captureEnabled = captureEnabled
|
||||
self.onCaptureChange = onCaptureChange
|
||||
self.onFrame = onFrame
|
||||
self.onSessionEnd = onSessionEnd
|
||||
self.presentMeter = presentMeter
|
||||
self.presentTailMeter = presentTailMeter
|
||||
}
|
||||
|
||||
public func makeNSView(context: Context) -> StreamLayerView {
|
||||
let view = StreamLayerView()
|
||||
view.onCaptureChange = onCaptureChange
|
||||
view.captureEnabled = captureEnabled
|
||||
view.presentMeter = presentMeter
|
||||
view.presentTailMeter = presentTailMeter
|
||||
view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
|
||||
return view
|
||||
}
|
||||
|
||||
public func updateNSView(_ view: StreamLayerView, context: Context) {
|
||||
view.onCaptureChange = onCaptureChange
|
||||
view.captureEnabled = captureEnabled
|
||||
view.presentMeter = presentMeter
|
||||
view.presentTailMeter = presentTailMeter
|
||||
// SwiftUI reuses the NSView across state changes — repoint the pump only when the
|
||||
// connection identity actually changed.
|
||||
if view.connection !== connection {
|
||||
view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
|
||||
}
|
||||
}
|
||||
|
||||
public static func dismantleNSView(_ view: StreamLayerView, coordinator: ()) {
|
||||
view.stop()
|
||||
}
|
||||
}
|
||||
|
||||
public final class StreamLayerView: NSView {
|
||||
private let displayLayer = AVSampleBufferDisplayLayer()
|
||||
/// Record capture→present / decode→present when the stage-2 presenter is active.
|
||||
/// Consulted at start().
|
||||
var presentMeter: LatencyMeter?
|
||||
var presentTailMeter: LatencyMeter?
|
||||
/// The shared presenter stack: stage-2 (CAMetalLayer sublayer + display link) with the
|
||||
/// stage-1 StreamPump → displayLayer path as the Metal-unavailable / DEBUG fallback.
|
||||
private let presenter = SessionPresenter()
|
||||
public private(set) var connection: PunktfunkConnection?
|
||||
private let cursorCapture = CursorCapture()
|
||||
private var inputCapture: InputCapture?
|
||||
private var appObservers: [NSObjectProtocol] = []
|
||||
private var windowObservers: [NSObjectProtocol] = []
|
||||
/// Local NSEvent monitor carrying relative mouse MOTION + BUTTONS to the host while
|
||||
/// captured (GCMouse's own delivery proved unreliable on macOS — see InputCapture).
|
||||
/// Installed on engage, removed on release; nil while not captured.
|
||||
private var mouseEventMonitor: Any?
|
||||
/// The window's `acceptsMouseMovedEvents` value before client-side-cursor capture raised
|
||||
/// it (nil = not raised by us); restored on release so we leave the window as we found it.
|
||||
private var savedAcceptsMouseMoved: Bool?
|
||||
|
||||
/// Whether input capture is currently engaged (cursor hidden+frozen, mouse/keyboard
|
||||
/// forwarded). Main-thread only.
|
||||
public private(set) var captured = false
|
||||
|
||||
/// Client-side-cursor mode: when true the local system cursor stays VISIBLE over the
|
||||
/// stream and the mouse monitor forwards ABSOLUTE positions (the visible cursor is the
|
||||
/// on-screen cursor — gamescope draws none, so no double cursor); when false the existing
|
||||
/// captured/disassociated relative path runs unchanged. Initialized at session start from
|
||||
/// the `cursorMode` setting + the host's resolved compositor, toggled live by ⌘⇧C. A live
|
||||
/// flip re-engages capture in the new mode so disassociation + the abs/rel choice swap
|
||||
/// atomically. Main-thread only.
|
||||
private var cursorVisible = false
|
||||
/// One-shot auto-engage request (stream start, trust confirmed) — attempted as soon
|
||||
/// as the view is in a window with real bounds, then dropped, so it can never fire
|
||||
/// surprisingly later (e.g. on a resize).
|
||||
private var pendingAutoCapture = false
|
||||
|
||||
/// Reports engage/release on the main thread.
|
||||
public var onCaptureChange: ((Bool) -> Void)?
|
||||
|
||||
/// Main-thread only. False = input capture disabled outright (UI layered over the
|
||||
/// stream); flipping to true auto-engages once.
|
||||
public var captureEnabled = true {
|
||||
didSet {
|
||||
guard captureEnabled != oldValue else { return }
|
||||
if captureEnabled {
|
||||
requestAutoCapture()
|
||||
} else {
|
||||
releaseCapture()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public override init(frame: NSRect) {
|
||||
super.init(frame: frame)
|
||||
displayLayer.videoGravity = .resizeAspect
|
||||
layer = displayLayer // layer-hosting: assign before wantsLayer
|
||||
wantsLayer = true
|
||||
// Focus loss releases capture. Becoming active does NOT re-engage: the click
|
||||
// that activates the window may be on the title bar (a drag) or a resize edge —
|
||||
// the user clicks into the video (or hits ⌘⎋) when they want capture back.
|
||||
appObservers.append(NotificationCenter.default.addObserver(
|
||||
forName: NSApplication.didResignActiveNotification, object: nil, queue: .main
|
||||
) { [weak self] _ in
|
||||
self?.releaseCapture()
|
||||
})
|
||||
}
|
||||
|
||||
public required init?(coder: NSCoder) { fatalError("not used") }
|
||||
|
||||
public override func viewDidMoveToWindow() {
|
||||
super.viewDidMoveToWindow()
|
||||
windowObservers.forEach(NotificationCenter.default.removeObserver(_:))
|
||||
windowObservers.removeAll()
|
||||
guard let window else {
|
||||
releaseCapture()
|
||||
return
|
||||
}
|
||||
// ⌘-key-equivalents stay live while captured, so Settings (⌘,), a new window
|
||||
// (⌘N), or Minimize (⌘M) can take key status without the APP resigning active —
|
||||
// capture must release then too, or the new window inherits a hidden, frozen
|
||||
// cursor and its local typing is double-delivered to the host.
|
||||
for name in [NSWindow.didResignKeyNotification, NSWindow.didMiniaturizeNotification] {
|
||||
windowObservers.append(NotificationCenter.default.addObserver(
|
||||
forName: name, object: window, queue: .main
|
||||
) { [weak self] _ in
|
||||
self?.releaseCapture()
|
||||
})
|
||||
}
|
||||
// Becoming key RETRIES a still-pending session-start auto-capture — the case where a
|
||||
// session began (reconnect) while this window wasn't key yet, so engageCapture(fromClick:
|
||||
// false) was refused by its key-window guard and, with no retry, capture stayed off and
|
||||
// input dead. This is a no-op once capture engaged (pendingAutoCapture is cleared) and
|
||||
// after a manual ⌘⎋/focus-loss release (the flag is already false), so it does NOT
|
||||
// resurrect the deliberately-rejected "auto-grab on every activation" behavior.
|
||||
windowObservers.append(NotificationCenter.default.addObserver(
|
||||
forName: NSWindow.didBecomeKeyNotification, object: window, queue: .main
|
||||
) { [weak self] _ in
|
||||
self?.attemptPendingCapture()
|
||||
})
|
||||
attemptPendingCapture()
|
||||
}
|
||||
|
||||
public override func layout() {
|
||||
super.layout()
|
||||
attemptPendingCapture() // bounds become real here on first presentation
|
||||
layoutPresenter() // keep the stage-2 sublayer aspect-fit to the view
|
||||
}
|
||||
|
||||
public override func setFrameSize(_ newSize: NSSize) {
|
||||
super.setFrameSize(newSize)
|
||||
// `layout()` isn't guaranteed on a manual-frame (no-Auto-Layout) live resize, so the
|
||||
// stage-2 metal sublayer's frame could stay at the old size while the view grows —
|
||||
// the compositor then upscales a too-small layer and the video turns blocky. Re-fit
|
||||
// here too so it always tracks the window's size (no stale upscale).
|
||||
layoutPresenter()
|
||||
}
|
||||
|
||||
// MARK: - Capture state machine
|
||||
|
||||
/// Clicking into the video engages capture; that click is local (engagement), so
|
||||
/// InputCapture suppresses its press/release toward the host. Clicks while captured
|
||||
/// are the host's (GC forwards them) — nothing to do here.
|
||||
public override func mouseDown(with event: NSEvent) {
|
||||
if streamInputDebug {
|
||||
streamInputLog.debug(
|
||||
"mouseDown: captureEnabled=\(self.captureEnabled, privacy: .public) captured=\(self.captured, privacy: .public)")
|
||||
}
|
||||
if captureEnabled, !captured {
|
||||
engageCapture(fromClick: true)
|
||||
return
|
||||
}
|
||||
super.mouseDown(with: event)
|
||||
}
|
||||
|
||||
/// A click from another app counts (one click into the video captures, not two).
|
||||
public override func acceptsFirstMouse(for event: NSEvent?) -> Bool { true }
|
||||
|
||||
/// The engage click is complete — drop its suppression latch (see InputCapture;
|
||||
/// guards against GC delivering both halves of the click before our mouseDown).
|
||||
public override func mouseUp(with event: NSEvent) {
|
||||
inputCapture?.endClickSuppression()
|
||||
super.mouseUp(with: event)
|
||||
}
|
||||
|
||||
/// Scroll is forwarded from here, not from GCMouse: trackpad/Magic Mouse gestures
|
||||
/// never reach GameController's scroll dpad. While captured the cursor is parked
|
||||
/// mid-view, so this view receives every scroll event. Precise (gesture) deltas are
|
||||
/// pixels — ~0.1 wheel notch per pixel (SDL's factor) → ×12 for WHEEL_DELTA(120);
|
||||
/// classic wheels report lines, one notch = ±1 → ×120. Signs pass through as-is,
|
||||
/// preserving the user's local (natural-)scrolling preference.
|
||||
public override func scrollWheel(with event: NSEvent) {
|
||||
guard captured, let inputCapture else {
|
||||
super.scrollWheel(with: event)
|
||||
return
|
||||
}
|
||||
let scale: Float = event.hasPreciseScrollingDeltas ? 12 : 120
|
||||
inputCapture.sendScroll(
|
||||
dx: Float(event.scrollingDeltaX) * scale,
|
||||
dy: Float(event.scrollingDeltaY) * scale)
|
||||
}
|
||||
|
||||
// While captured, the view is first responder and SENDS key events to the host straight
|
||||
// from NSEvent — GCKeyboard delivery proved unreliable on macOS (the same GameController
|
||||
// quirk that killed GCMouse motion, fixed in e414ec0), so the macOS GCKeyboard send path
|
||||
// is disabled and NSEvent is the single source. We map NSEvent.keyCode (a Carbon virtual
|
||||
// keycode) → Windows VK and forward via InputCapture.sendKey, then CONSUME (return without
|
||||
// super) to stop the responder chain's "unhandled keyDown" beep. Keys with no VK mapping
|
||||
// are still consumed while captured so they don't beep either. The ⌘⎋ toggle's Esc is
|
||||
// swallowed upstream by InputCapture's keyDown monitor (suppressedVK), so it never gets
|
||||
// here as a send; ⌘-combos still arrive via performKeyEquivalent and stay functional (⌘D).
|
||||
// Modifier keys never fire keyDown/keyUp — they come through flagsChanged below.
|
||||
public override var acceptsFirstResponder: Bool { true }
|
||||
// A click after the app was inactive (Cmd-Tab away and back) must reach mouseDown so the
|
||||
// user can re-capture — the deliberate design is that becoming active does NOT auto-grab;
|
||||
// you click into the video. Default NSViews aren't key-view candidates, which can drop
|
||||
// that first click; opting in keeps the view a valid click/responder target.
|
||||
public override var canBecomeKeyView: Bool { true }
|
||||
public override func keyDown(with event: NSEvent) {
|
||||
if captured {
|
||||
if let ic = inputCapture, let vk = InputCapture.keyCodeToVK[event.keyCode] {
|
||||
ic.sendKey(vk, down: true) // autorepeat (event.isARepeat) passes through — fine for VK
|
||||
}
|
||||
return // consume even unmapped keys while captured (no beep)
|
||||
}
|
||||
super.keyDown(with: event)
|
||||
}
|
||||
public override func keyUp(with event: NSEvent) {
|
||||
if captured {
|
||||
if let ic = inputCapture, let vk = InputCapture.keyCodeToVK[event.keyCode] {
|
||||
ic.sendKey(vk, down: false)
|
||||
}
|
||||
return
|
||||
}
|
||||
super.keyUp(with: event)
|
||||
}
|
||||
/// Modifier keys (shift/control/option/command) arrive ONLY as flagsChanged on macOS,
|
||||
/// never keyDown/keyUp — InputCapture diffs the raw flags to recover each L/R down/up.
|
||||
public override func flagsChanged(with event: NSEvent) {
|
||||
if captured, let inputCapture {
|
||||
inputCapture.handleFlagsChanged(UInt(event.modifierFlags.rawValue))
|
||||
return
|
||||
}
|
||||
super.flagsChanged(with: event)
|
||||
}
|
||||
|
||||
private func requestAutoCapture() {
|
||||
pendingAutoCapture = true
|
||||
attemptPendingCapture()
|
||||
}
|
||||
|
||||
private func attemptPendingCapture() {
|
||||
guard pendingAutoCapture, window != nil, bounds.width > 0 else { return }
|
||||
engageCapture(fromClick: false)
|
||||
// Clear the one-shot only once it ACTUALLY engaged. If the engage was refused — the
|
||||
// app/window isn't key yet (common right after a reconnect), or the cursor grab raced
|
||||
// app activation — leave it armed so didBecomeKey (or the next layout pass) retries.
|
||||
// This stays scoped to session start: a later manual release (⌘⎋, focus loss) doesn't
|
||||
// re-arm it, so it never resurrects auto-grab-on-activation.
|
||||
if captured { pendingAutoCapture = false }
|
||||
}
|
||||
|
||||
private func engageCapture(fromClick: Bool) {
|
||||
// A click is explicit intent AND may arrive mid-activation (acceptsFirstMouse:
|
||||
// NSApp.isActive / isKeyWindow are still false for the click coming in from
|
||||
// another app) — only the auto-engage paths require already-held key status.
|
||||
// `connection != nil` is the session-active gate (presenter internals are opaque here).
|
||||
guard captureEnabled, !captured, connection != nil, window != nil,
|
||||
fromClick || (NSApp.isActive && window?.isKeyWindow == true)
|
||||
else { return }
|
||||
// If the cursor grab is refused (e.g. the reactivating click arrives before the app is
|
||||
// frontmost), stay released so the NEXT click retries — never latch captured=true over
|
||||
// a free cursor, which would make mouseDown's `!captured` guard reject every later click.
|
||||
// In client-side-cursor mode there is no grab (the cursor stays visible) — capture
|
||||
// always engages and the monitor forwards absolute positions instead.
|
||||
guard cursorCapture.capture(in: self, disassociate: !cursorVisible) else { return }
|
||||
inputCapture?.setForwarding(true, suppressClick: fromClick)
|
||||
// Install AFTER the warp + setForwarding: the engage warp generates no forwarded
|
||||
// delta (the monitor isn't up yet), and the engage click's suppression latch is
|
||||
// already armed, so the monitor only ever sees genuine post-engage input.
|
||||
installMouseMonitor()
|
||||
captured = true
|
||||
window?.makeFirstResponder(self)
|
||||
notifyCaptureChange(true)
|
||||
}
|
||||
|
||||
private func releaseCapture() {
|
||||
guard captured else { return }
|
||||
removeMouseMonitor()
|
||||
cursorCapture.release()
|
||||
inputCapture?.setForwarding(false)
|
||||
captured = false
|
||||
notifyCaptureChange(false)
|
||||
}
|
||||
|
||||
/// A single local monitor for motion + buttons, installed only while captured. A local
|
||||
/// monitor is more robust than view overrides for relative motion: it sidesteps the
|
||||
/// `window.acceptsMouseMovedEvents`/tracking-area/responder-chain requirements, and
|
||||
/// since the cursor is frozen mid-view while captured every such event belongs here.
|
||||
/// ALL four motion types are covered so motion keeps flowing during a button-held drag,
|
||||
/// not just `.mouseMoved`. NSEvent deltas under disassociation are OS-pointer-
|
||||
/// acceleration-applied (not raw HID) — what Moonlight's macOS client ships; if the
|
||||
/// host re-accelerates there's mild double-acceleration, acceptable and fixable later
|
||||
/// via IOHID. Events are returned (not swallowed): the cursor is frozen, so they're
|
||||
/// inert locally.
|
||||
///
|
||||
/// In client-side-cursor mode the cursor is NOT frozen, so bare `.mouseMoved` events are
|
||||
/// only generated while `window.acceptsMouseMovedEvents` is true — we enable it here and
|
||||
/// restore it on removal so absolute hover-motion keeps flowing without a click held.
|
||||
private func installMouseMonitor() {
|
||||
guard mouseEventMonitor == nil else { return }
|
||||
if cursorVisible {
|
||||
savedAcceptsMouseMoved = window?.acceptsMouseMovedEvents
|
||||
window?.acceptsMouseMovedEvents = true
|
||||
}
|
||||
mouseEventMonitor = NSEvent.addLocalMonitorForEvents(matching: [
|
||||
.mouseMoved, .leftMouseDragged, .rightMouseDragged, .otherMouseDragged,
|
||||
.leftMouseDown, .leftMouseUp, .rightMouseDown, .rightMouseUp,
|
||||
.otherMouseDown, .otherMouseUp,
|
||||
]) { [weak self] event in
|
||||
guard let self, self.captured, let ic = self.inputCapture else { return event }
|
||||
switch event.type {
|
||||
case .mouseMoved, .leftMouseDragged, .rightMouseDragged, .otherMouseDragged:
|
||||
if self.cursorVisible {
|
||||
// Client-side cursor: forward the ABSOLUTE position (mapped through the
|
||||
// aspect-fit letterbox into host pixels), the same path the iPad pointer
|
||||
// fallback uses. Events in the letterbox bars are dropped (nil host point).
|
||||
if let p = self.hostPoint(from: event) {
|
||||
ic.sendMouseAbs(x: p.x, y: p.y, surfaceWidth: p.w, surfaceHeight: p.h)
|
||||
}
|
||||
} else {
|
||||
ic.sendMotion(dx: Float(event.deltaX), dy: Float(event.deltaY)) // no y-negation
|
||||
}
|
||||
case .leftMouseDown: ic.sendMouseButton(1, pressed: true)
|
||||
case .leftMouseUp: ic.sendMouseButton(1, pressed: false)
|
||||
case .rightMouseDown: ic.sendMouseButton(3, pressed: true)
|
||||
case .rightMouseUp: ic.sendMouseButton(3, pressed: false)
|
||||
case .otherMouseDown: ic.sendMouseButton(self.wireButton(for: event), pressed: true)
|
||||
case .otherMouseUp: ic.sendMouseButton(self.wireButton(for: event), pressed: false)
|
||||
default: break
|
||||
}
|
||||
return event
|
||||
}
|
||||
if streamInputDebug { streamInputLog.debug("mouse NSEvent monitor installed (capture engaged)") }
|
||||
}
|
||||
|
||||
private func removeMouseMonitor() {
|
||||
if let monitor = mouseEventMonitor {
|
||||
NSEvent.removeMonitor(monitor)
|
||||
mouseEventMonitor = nil
|
||||
if streamInputDebug { streamInputLog.debug("mouse NSEvent monitor removed (capture released)") }
|
||||
}
|
||||
// Restore the window's prior mouse-moved-events setting if we raised it (cursor mode).
|
||||
if let saved = savedAcceptsMouseMoved {
|
||||
window?.acceptsMouseMovedEvents = saved
|
||||
savedAcceptsMouseMoved = nil
|
||||
}
|
||||
}
|
||||
|
||||
/// One host-pixel point on the negotiated output, with the surface dimensions the host
|
||||
/// rescales against (surface == host mode, so the host applies no extra scaling).
|
||||
private struct HostPoint { let x: Int32; let y: Int32; let w: UInt32; let h: UInt32 }
|
||||
|
||||
/// Map an NSEvent's cursor location into host-mode pixels for the client-side-cursor
|
||||
/// (absolute) path. NSEvent.locationInWindow is window space, origin BOTTOM-left (+y up);
|
||||
/// we convert to this view's space, FLIP y to the host's top-left (+y down) convention,
|
||||
/// then aspect-fit-letterbox into the host mode exactly like the iOS touch/pointer path.
|
||||
/// Returns nil for events in the letterbox bars (outside the video rect) so the host's
|
||||
/// cursor isn't dragged onto a black edge, and until a mode is negotiated.
|
||||
private func hostPoint(from event: NSEvent) -> HostPoint? {
|
||||
guard let connection else { return nil }
|
||||
let mode = connection.currentMode()
|
||||
guard mode.width > 0, mode.height > 0 else { return nil }
|
||||
// Window → view coords (non-flipped: origin bottom-left), then flip y into view-top-left.
|
||||
let inView = convert(event.locationInWindow, from: nil)
|
||||
let p = CGPoint(x: inView.x, y: bounds.height - inView.y)
|
||||
// The video occupies the aspect-fit rect inside the (non-flipped) bounds; AVMakeRect's
|
||||
// origin is bottom-left, so flip its minY too to match p's top-left space.
|
||||
let fit = AVMakeRect(
|
||||
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
|
||||
insideRect: bounds)
|
||||
guard fit.width > 0, fit.height > 0 else { return nil }
|
||||
let videoMinYTop = bounds.height - fit.maxY
|
||||
let u = (p.x - fit.minX) / fit.width
|
||||
let v = (p.y - videoMinYTop) / fit.height
|
||||
guard u >= 0, u <= 1, v >= 0, v <= 1 else { return nil } // letterbox bars
|
||||
let hx = Int32((u * CGFloat(mode.width)).rounded()
|
||||
.clamped(to: 0...CGFloat(mode.width - 1)))
|
||||
let hy = Int32((v * CGFloat(mode.height)).rounded()
|
||||
.clamped(to: 0...CGFloat(mode.height - 1)))
|
||||
return HostPoint(x: hx, y: hy, w: mode.width, h: mode.height)
|
||||
}
|
||||
|
||||
/// NSEvent `buttonNumber` → GameStream wire id for the "other" buttons: 2 = middle,
|
||||
/// 3 = first side (X1), 4 = second side (X2). Unknown extras fall back to middle.
|
||||
private func wireButton(for event: NSEvent) -> UInt32 {
|
||||
switch event.buttonNumber {
|
||||
case 2: return 2 // middle
|
||||
case 3: return 4 // X1
|
||||
case 4: return 5 // X2
|
||||
default: return 2
|
||||
}
|
||||
}
|
||||
|
||||
/// Engage/release can run inside a SwiftUI update pass (captureEnabled flips in
|
||||
/// updateNSView; release in dismantleNSView) — publishing model state synchronously
|
||||
/// there is undefined behavior, so the callback is deferred a runloop turn.
|
||||
private func notifyCaptureChange(_ captured: Bool) {
|
||||
guard let onCaptureChange else { return }
|
||||
DispatchQueue.main.async { onCaptureChange(captured) }
|
||||
}
|
||||
|
||||
// MARK: - Session start/stop
|
||||
|
||||
/// Wire up input capture and start the presenter (see SessionPresenter for the
|
||||
/// stage-2/stage-1 choice). `onFrame` fires per AU at receipt; `onSessionEnd` on close.
|
||||
public func start(
|
||||
connection: PunktfunkConnection,
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
|
||||
onSessionEnd: (@Sendable () -> Void)? = nil
|
||||
) {
|
||||
stop()
|
||||
self.connection = connection
|
||||
|
||||
// The view owns the session's input capture: handlers attach now, but nothing is
|
||||
// forwarded until capture engages (captureEnabled + auto-engage or a click).
|
||||
let capture = InputCapture(connection: connection)
|
||||
capture.onToggleCapture = { [weak self] in
|
||||
// The ⌘⎋ monitor is app-wide — only the key window's stream owns the toggle
|
||||
// (two stream windows would otherwise flip each other's capture).
|
||||
guard let self, self.window?.isKeyWindow == true else { return }
|
||||
if self.captured {
|
||||
self.releaseCapture()
|
||||
} else {
|
||||
self.engageCapture(fromClick: false)
|
||||
}
|
||||
}
|
||||
capture.onPreempted = { [weak self] in
|
||||
// A newer session took the GC handler slots — staying "captured" here would
|
||||
// be a cursor trap with dead input.
|
||||
self?.releaseCapture()
|
||||
}
|
||||
// ⌘⇧C flips the client-side cursor live. Only the key window's stream owns it (same
|
||||
// guard as the ⌘⎋ capture toggle). Re-engage capture in the new mode so disassociation
|
||||
// and the absolute/relative forwarding choice swap atomically — releaseCapture restores
|
||||
// the old mode's grab (if any), engageCapture installs the new one.
|
||||
// ⌘⇧C would flip the client-side cursor live — NEUTERED while the feature is disabled
|
||||
// (see the cursorVisible resolution below): toggling it on under gamescope's relative-only
|
||||
// input traps the pointer. Restore this body when absolute/synthetic-cursor support lands.
|
||||
capture.onToggleCursor = {}
|
||||
capture.start()
|
||||
inputCapture = capture
|
||||
|
||||
// Client-side cursor is TEMPORARILY DISABLED. It positions the host cursor with ABSOLUTE
|
||||
// events, but gamescope's input socket (EIS) grants only a relative pointer, so those are
|
||||
// silently dropped — the pointer never moves and clicks/scroll land on the stuck position
|
||||
// (looks like "all input dead"). gamescope is exactly the compositor Auto enabled it for.
|
||||
// Forced off until per-compositor gating (KWin/GNOME/Sway have absolute) or a synthetic-
|
||||
// cursor-over-relative path lands; the resolution logic below is kept for that. See the
|
||||
// ⌘⇧C handler (also neutered) and the cursorMode setting (hidden).
|
||||
cursorVisible = false
|
||||
_ = connection.resolvedCompositor // (was: Auto → gamescope; kept to document intent)
|
||||
|
||||
// Presenter choice + lifecycle live in SessionPresenter (shared with iOS/tvOS): stage-2
|
||||
// (explicit VTDecompressionSession decode + a CAMetalLayer/display-link present) by
|
||||
// default, the stage-1 pump as the Metal-missing / DEBUG fallback. The link comes from
|
||||
// NSView.displayLink so it tracks the display this view is on.
|
||||
presenter.start(
|
||||
connection: connection,
|
||||
baseLayer: displayLayer,
|
||||
presentMeter: presentMeter,
|
||||
presentTailMeter: presentTailMeter,
|
||||
makeDisplayLink: { displayLink(target: $0, selector: $1) },
|
||||
onFrame: onFrame,
|
||||
onSessionEnd: onSessionEnd)
|
||||
layoutPresenter()
|
||||
requestAutoCapture() // entering a session is the deliberate "capture me" moment
|
||||
}
|
||||
|
||||
/// Aspect-fit the stage-2 metal sublayer to the view; refresh contentsScale on a
|
||||
/// retina↔non-retina move (see SessionPresenter.layout).
|
||||
private func layoutPresenter() {
|
||||
presenter.layout(in: bounds, contentsScale: window?.backingScaleFactor ?? 1)
|
||||
}
|
||||
|
||||
public override func viewDidChangeBackingProperties() {
|
||||
super.viewDidChangeBackingProperties()
|
||||
layoutPresenter() // backing scale changed (e.g. moved to a non-retina display)
|
||||
}
|
||||
|
||||
/// Stop pumping (≤ one poll timeout). Does not close the connection — that stays with
|
||||
/// whoever owns it (PunktfunkConnection.close() is safe alongside a draining pump).
|
||||
public func stop() {
|
||||
releaseCapture()
|
||||
removeMouseMonitor() // belt-and-suspenders: releaseCapture no-ops if not captured
|
||||
inputCapture?.stop()
|
||||
inputCapture = nil
|
||||
presenter.stop()
|
||||
connection = nil
|
||||
}
|
||||
|
||||
deinit {
|
||||
removeMouseMonitor()
|
||||
appObservers.forEach(NotificationCenter.default.removeObserver(_:))
|
||||
windowObservers.forEach(NotificationCenter.default.removeObserver(_:))
|
||||
presenter.stop() // invalidate the display link + stop the pipeline if stop() was missed
|
||||
}
|
||||
}
|
||||
#endif
|
||||
@@ -0,0 +1,620 @@
|
||||
// iOS/iPadOS presenter: the same AVSampleBufferDisplayLayer + StreamPump as macOS,
|
||||
// hosted in a UIViewController so the scene can pointer-lock (the iPadOS equivalent of
|
||||
// the Mac's cursor capture — with a hardware mouse/trackpad the system cursor is hidden
|
||||
// and GCMouse's raw deltas drive the host cursor alone; the system only honors the lock
|
||||
// fullscreen-and-frontmost, so in Stage Manager it degrades to Mac-style "both cursors
|
||||
// visible" forwarding).
|
||||
//
|
||||
// FINGER touch and INDIRECT POINTER (mouse/trackpad) are routed apart by UITouch.type.
|
||||
// Direct fingers (and Pencil) always forward as wire touches — every finger maps to a touch
|
||||
// id, coordinates mapped through the aspect-fit letterbox into host-mode pixels (surface ==
|
||||
// host mode, so the host's rescale is the identity).
|
||||
//
|
||||
// A hardware mouse/trackpad is a pointer, not a finger. When the scene is pointer-LOCKED
|
||||
// (full-screen + frontmost iPad, and the user hasn't disabled pointer capture in Settings —
|
||||
// see PointerLockChain, which steers the lock request through SwiftUI's hosting controllers)
|
||||
// GCMouse delivers raw relative deltas and the system hides the cursor — the gaming-grade path.
|
||||
// InputCapture handles EVERY connected mouse (GCMouse.mice), not just the current one, so a
|
||||
// trackpad + a second pointer (e.g. a Universal Control mouse) both drive. When the scene CAN'T
|
||||
// lock (Stage Manager, not frontmost, iPhone, capture disabled) the system shows its own cursor
|
||||
// and routes the mouse through UIKit's pointer path: hover + indirect-pointer touches, which we
|
||||
// forward as ABSOLUTE cursor position (+ buttons) so the host cursor tracks the visible local one.
|
||||
// We never forward an indirect pointer as a touch — doing so hid the cursor and made the host see
|
||||
// taps instead of a moving mouse. The two paths are mutually exclusive on `gcMouseForwarding`
|
||||
// (== locked): GCMouse forwards only WHILE locked, the UIKit indirect path (motion, buttons AND
|
||||
// scroll) only while NOT locked — so a pointer that emits both channels under lock can't double-send.
|
||||
// Hardware keyboard forwarding shares InputCapture with macOS — auto-engaged when streaming
|
||||
// starts, ⌘⎋ toggles (detected from the HID stream; there is no NSEvent monitor here).
|
||||
//
|
||||
// The public type is named StreamView like its macOS twin (each is platform-gated), so
|
||||
// the SwiftUI app layer is identical on both platforms.
|
||||
|
||||
#if os(iOS) || os(tvOS)
|
||||
import AVFoundation
|
||||
import GameController
|
||||
import PunktfunkCore
|
||||
import SwiftUI
|
||||
import UIKit
|
||||
import os
|
||||
|
||||
/// Same diagnostic switch as InputCapture (PUNKTFUNK_INPUT_DEBUG=1): on iOS we log the
|
||||
/// resolved pointer-lock state each time capture engages, so the user can see whether the
|
||||
/// scene actually locked (GCMouse only delivers deltas while it did) or whether we're on
|
||||
/// the touch fallback.
|
||||
private let iosInputLog = Logger(subsystem: "io.unom.punktfunk", category: "input")
|
||||
private let iosInputDebug = ProcessInfo.processInfo.environment["PUNKTFUNK_INPUT_DEBUG"] == "1"
|
||||
|
||||
public struct StreamView: UIViewControllerRepresentable {
|
||||
private let connection: PunktfunkConnection
|
||||
private let captureEnabled: Bool
|
||||
private let onCaptureChange: ((Bool) -> Void)?
|
||||
private let onFrame: (@Sendable (AccessUnit) -> Void)?
|
||||
private let onSessionEnd: (@Sendable () -> Void)?
|
||||
private let presentMeter: LatencyMeter?
|
||||
private let presentTailMeter: LatencyMeter?
|
||||
|
||||
public init(
|
||||
connection: PunktfunkConnection,
|
||||
captureEnabled: Bool = true,
|
||||
onCaptureChange: ((Bool) -> Void)? = nil,
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
|
||||
onSessionEnd: (@Sendable () -> Void)? = nil,
|
||||
presentMeter: LatencyMeter? = nil,
|
||||
presentTailMeter: LatencyMeter? = nil
|
||||
) {
|
||||
self.connection = connection
|
||||
self.captureEnabled = captureEnabled
|
||||
self.onCaptureChange = onCaptureChange
|
||||
self.onFrame = onFrame
|
||||
self.onSessionEnd = onSessionEnd
|
||||
self.presentMeter = presentMeter
|
||||
self.presentTailMeter = presentTailMeter
|
||||
}
|
||||
|
||||
public func makeUIViewController(context: Context) -> StreamViewController {
|
||||
let controller = StreamViewController()
|
||||
controller.onCaptureChange = onCaptureChange
|
||||
controller.captureEnabled = captureEnabled
|
||||
controller.presentMeter = presentMeter
|
||||
controller.presentTailMeter = presentTailMeter
|
||||
controller.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
|
||||
return controller
|
||||
}
|
||||
|
||||
public func updateUIViewController(_ controller: StreamViewController, context: Context) {
|
||||
controller.onCaptureChange = onCaptureChange
|
||||
controller.captureEnabled = captureEnabled
|
||||
controller.presentMeter = presentMeter
|
||||
controller.presentTailMeter = presentTailMeter
|
||||
if controller.connection !== connection {
|
||||
controller.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
|
||||
}
|
||||
}
|
||||
|
||||
public static func dismantleUIViewController(
|
||||
_ controller: StreamViewController, coordinator: ()
|
||||
) {
|
||||
controller.stop()
|
||||
}
|
||||
}
|
||||
|
||||
public final class StreamViewController: UIViewController {
|
||||
public private(set) var connection: PunktfunkConnection?
|
||||
private var observers: [NSObjectProtocol] = []
|
||||
/// Record capture→present / decode→present when the stage-2 presenter is active.
|
||||
/// Consulted at start().
|
||||
var presentMeter: LatencyMeter?
|
||||
var presentTailMeter: LatencyMeter?
|
||||
/// The shared presenter stack: stage-2 (CAMetalLayer sublayer + display link) with the
|
||||
/// stage-1 StreamPump → displayLayer path as the Metal-unavailable / DEBUG fallback.
|
||||
private let presenter = SessionPresenter()
|
||||
#if os(iOS)
|
||||
private var inputCapture: InputCapture?
|
||||
fileprivate var captured = false
|
||||
private var pointerInteraction: UIPointerInteraction?
|
||||
/// Capture state at the last resign, restored on the next foreground — otherwise the
|
||||
/// mouse/keyboard stay released after navigating out and nothing re-grabs them.
|
||||
private var wasCapturedOnResign = false
|
||||
#endif
|
||||
|
||||
/// Reads whether the scene's pointer is actually locked right now; nil = state
|
||||
/// unavailable (no scene yet, or pre-availability). Only while this is true does GCMouse
|
||||
/// deliver relative deltas — otherwise the touch path carries input.
|
||||
private func pointerLockEngaged() -> Bool? {
|
||||
#if os(iOS)
|
||||
return view.window?.windowScene?.pointerLockState?.isLocked
|
||||
#else
|
||||
return nil
|
||||
#endif
|
||||
}
|
||||
|
||||
var onCaptureChange: ((Bool) -> Void)?
|
||||
|
||||
var captureEnabled = true {
|
||||
didSet {
|
||||
guard captureEnabled != oldValue else { return }
|
||||
#if os(iOS)
|
||||
setCaptured(captureEnabled)
|
||||
#endif
|
||||
}
|
||||
}
|
||||
|
||||
private var streamView: StreamLayerUIView {
|
||||
// swiftlint:disable:next force_cast
|
||||
view as! StreamLayerUIView
|
||||
}
|
||||
|
||||
public override func loadView() {
|
||||
view = StreamLayerUIView()
|
||||
// Re-size the stage-2 drawable if the display scale changes without a bounds change (e.g.
|
||||
// moving to an external display at a different scale) — the iOS analogue of macOS's
|
||||
// viewDidChangeBackingProperties relayout. The handler takes the VC as its argument, so it
|
||||
// doesn't capture self (no retain cycle with the registration).
|
||||
registerForTraitChanges([UITraitDisplayScale.self]) { (vc: StreamViewController, _) in
|
||||
vc.layoutMetalLayer()
|
||||
}
|
||||
#if os(iOS)
|
||||
// Hide the iPadOS cursor while it hovers the video: the host renders its own
|
||||
// cursor from our deltas, so the local one only diverges from it. This hides the
|
||||
// pointer; true pointer LOCK (below) is what makes GCMouse deliver relative deltas
|
||||
// — and the system only grants it on a full-screen, frontmost iPad scene.
|
||||
let interaction = UIPointerInteraction(delegate: self)
|
||||
view.addInteraction(interaction)
|
||||
pointerInteraction = interaction
|
||||
#endif
|
||||
}
|
||||
|
||||
#if os(iOS)
|
||||
/// Whether the user wants the mouse/trackpad pointer CAPTURED (pointer lock → relative
|
||||
/// movement, the gaming default) rather than forwarded as an absolute position (desktop
|
||||
/// use). Read live from UserDefaults so it tracks the Settings toggle; defaults to on when
|
||||
/// unset. iPad-only — gated again in `prefersPointerLocked`.
|
||||
private var pointerCaptureEnabled: Bool {
|
||||
UserDefaults.standard.object(forKey: DefaultsKey.pointerCapture) as? Bool ?? true
|
||||
}
|
||||
|
||||
/// Whether the pointer should be CAPTURED right now: iPad, capture engaged, and the user
|
||||
/// hasn't opted into the absolute (desktop) pointer. The system additionally requires
|
||||
/// full-screen + frontmost and may drop the lock (Slide Over/Stage Manager/backgrounding) —
|
||||
/// syncPointerLock() handles the actual grant/drop and falls back to absolute when unlocked.
|
||||
private var wantsPointerLock: Bool {
|
||||
captured && pointerCaptureEnabled && UIDevice.current.userInterfaceIdiom == .pad
|
||||
}
|
||||
|
||||
public override var prefersPointerLocked: Bool { wantsPointerLock }
|
||||
public override var prefersHomeIndicatorAutoHidden: Bool { true }
|
||||
|
||||
// NOTE: we deliberately do NOT override `childViewControllerForPointerLock`. The default
|
||||
// returns nil, which tells the system to use THIS controller's own `prefersPointerLocked` —
|
||||
// exactly what we want, since `PointerLockChain` forces our SwiftUI ancestors to forward the
|
||||
// downward walk to us and we are the terminal anchor. Returning `self` here would make the
|
||||
// system ask the same controller forever (it keeps delegating to the returned child) →
|
||||
// unbounded recursion → stack overflow once the chain actually reaches us.
|
||||
|
||||
/// (Re)build or tear down the forced pointer-lock forwarding chain from this controller to the
|
||||
/// window root so the system actually resolves our `prefersPointerLocked`. Safe to call
|
||||
/// repeatedly — it no-ops until the view is in a window with a parent chain, and re-runs from
|
||||
/// the appearance/parent callbacks once SwiftUI has placed us.
|
||||
private func updatePointerLockChain() {
|
||||
// Engaging needs a live parent chain to the window root; disengaging is always safe and
|
||||
// must run even after the view has left the window (session teardown) so the stamped
|
||||
// SwiftUI ancestors are cleared.
|
||||
if wantsPointerLock, view.window != nil {
|
||||
PointerLockChain.engage(self)
|
||||
} else {
|
||||
PointerLockChain.disengage(self)
|
||||
}
|
||||
}
|
||||
|
||||
public override func viewDidAppear(_ animated: Bool) {
|
||||
super.viewDidAppear(animated)
|
||||
// SwiftUI places us in the hierarchy AFTER start()'s setCaptured(true), and may reparent us
|
||||
// later — re-anchor the chain here so a lock requested before we had a parent still lands.
|
||||
updatePointerLockChain()
|
||||
}
|
||||
|
||||
public override func didMove(toParent parent: UIViewController?) {
|
||||
super.didMove(toParent: parent)
|
||||
updatePointerLockChain() // chain shape changed — re-anchor (or no-op if not yet in a window)
|
||||
}
|
||||
#endif
|
||||
|
||||
func start(
|
||||
connection: PunktfunkConnection,
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)?,
|
||||
onSessionEnd: (@Sendable () -> Void)?
|
||||
) {
|
||||
stop()
|
||||
self.connection = connection
|
||||
loadViewIfNeeded()
|
||||
#if os(iOS)
|
||||
// Fresh session: drop any resign/foreground capture-restore state left over from a
|
||||
// prior session (stop() doesn't clear it). Otherwise a stale `true` could later
|
||||
// re-engage capture on a foreground that the new session never asked for.
|
||||
wasCapturedOnResign = false
|
||||
// Read the LIVE mode per touch batch — an accepted requestMode() mid-stream
|
||||
// changes the letterbox, and touches must follow it.
|
||||
streamView.currentHostMode = { [weak connection] in
|
||||
guard let connection else { return .zero }
|
||||
let mode = connection.currentMode()
|
||||
return CGSize(width: Double(mode.width), height: Double(mode.height))
|
||||
}
|
||||
streamView.onTouchEvent = { [weak self, weak connection] event in
|
||||
// Touch IS the intent during a trusted session, but must not leak to the host
|
||||
// while a trust prompt is up (captureEnabled == false) — gate it on that. The
|
||||
// ⌘⎋ mouse/keyboard toggle (captured) deliberately does NOT gate touch.
|
||||
guard self?.captureEnabled == true else { return }
|
||||
connection?.send(event)
|
||||
}
|
||||
// Indirect pointer (mouse/trackpad) WITHOUT a lock → absolute cursor + buttons + scroll.
|
||||
// While the scene is pointer-LOCKED the GCMouse path owns motion AND buttons AND scroll, so
|
||||
// the whole UIKit indirect path is gated off here (`gcMouseForwarding`). The trackpad and a
|
||||
// mouse BOTH report through GCMouse under lock and ALSO emit UIKit indirect-pointer events
|
||||
// (pinned at the locked position) — without this gate a click double-sends (GCMouse + UIKit)
|
||||
// and a second pointer (e.g. a Universal Control mouse) competes with the trackpad. The gate
|
||||
// is the exact mirror of the GCMouse handlers, which fire only while locked.
|
||||
streamView.onPointerMoveAbs = { [weak self] p in
|
||||
guard let self, self.inputCapture?.gcMouseForwarding == false else { return }
|
||||
self.inputCapture?.sendMouseAbs(
|
||||
x: p.x, y: p.y, surfaceWidth: p.w, surfaceHeight: p.h)
|
||||
}
|
||||
streamView.onPointerButton = { [weak self] button, down in
|
||||
guard let self, self.inputCapture?.gcMouseForwarding == false else { return }
|
||||
self.inputCapture?.sendMouseButton(button, pressed: down)
|
||||
}
|
||||
streamView.onScroll = { [weak self] dx, dy in
|
||||
guard let self, self.inputCapture?.gcMouseForwarding == false else { return }
|
||||
self.inputCapture?.sendScroll(dx: dx, dy: dy)
|
||||
}
|
||||
|
||||
let capture = InputCapture(connection: connection)
|
||||
capture.onToggleCapture = { [weak self] in
|
||||
guard let self else { return }
|
||||
self.setCaptured(!self.captured)
|
||||
}
|
||||
capture.onPreempted = { [weak self] in
|
||||
self?.setCaptured(false)
|
||||
}
|
||||
capture.start()
|
||||
inputCapture = capture
|
||||
#endif
|
||||
|
||||
// Presenter choice + lifecycle live in SessionPresenter (shared with macOS): stage-2
|
||||
// (explicit VTDecompressionSession decode + a CAMetalLayer/display-link present) by
|
||||
// default, the stage-1 pump as the Metal-missing / DEBUG fallback.
|
||||
presenter.start(
|
||||
connection: connection,
|
||||
baseLayer: streamView.displayLayer,
|
||||
presentMeter: presentMeter,
|
||||
presentTailMeter: presentTailMeter,
|
||||
makeDisplayLink: { CADisplayLink(target: $0, selector: $1) },
|
||||
onFrame: onFrame,
|
||||
onSessionEnd: onSessionEnd)
|
||||
layoutMetalLayer()
|
||||
|
||||
#if os(iOS)
|
||||
// GC only delivers while active; everything held is flushed by InputCapture's
|
||||
// own resign observer — here we just mirror the capture state for the HUD and
|
||||
// the pointer lock.
|
||||
observers.append(NotificationCenter.default.addObserver(
|
||||
forName: UIApplication.willResignActiveNotification, object: nil, queue: .main
|
||||
) { [weak self] _ in
|
||||
guard let self else { return }
|
||||
self.wasCapturedOnResign = self.captured
|
||||
self.setCaptured(false)
|
||||
})
|
||||
// Returning to the foreground restores the capture the user had before leaving —
|
||||
// without this the mouse/keyboard stay released and nothing re-grabs them (touch
|
||||
// always plays regardless). The macOS twin re-engages on a click into the video.
|
||||
observers.append(NotificationCenter.default.addObserver(
|
||||
forName: UIApplication.didBecomeActiveNotification, object: nil, queue: .main
|
||||
) { [weak self] _ in
|
||||
// inputCapture != nil: don't try to restore before this session's capture is wired
|
||||
// up — setForwarding would silently no-op on the nil handlers and leave input dead.
|
||||
guard let self, self.wasCapturedOnResign, self.captureEnabled,
|
||||
self.connection != nil, self.inputCapture != nil
|
||||
else { return }
|
||||
self.setCaptured(true)
|
||||
})
|
||||
// The system can grant or drop the lock without us asking (Slide Over, Stage Manager,
|
||||
// entering/leaving foregroundActive). Re-resolve the mouse routing on every change:
|
||||
// GCMouse (locked) vs the absolute UIKit pointer path (unlocked), and the
|
||||
// hidden-vs-visible local cursor.
|
||||
observers.append(NotificationCenter.default.addObserver(
|
||||
forName: UIPointerLockState.didChangeNotification, object: nil, queue: .main
|
||||
) { [weak self] _ in
|
||||
self?.syncPointerLock()
|
||||
})
|
||||
|
||||
if captureEnabled {
|
||||
setCaptured(true) // entering a session is the deliberate "capture me" moment
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
func stop() {
|
||||
observers.forEach(NotificationCenter.default.removeObserver(_:))
|
||||
observers.removeAll()
|
||||
#if os(iOS)
|
||||
setCaptured(false)
|
||||
inputCapture?.stop()
|
||||
inputCapture = nil
|
||||
streamView.onTouchEvent = nil
|
||||
streamView.onPointerMoveAbs = nil
|
||||
streamView.onPointerButton = nil
|
||||
streamView.onScroll = nil
|
||||
streamView.currentHostMode = nil
|
||||
#endif
|
||||
presenter.stop()
|
||||
connection = nil
|
||||
}
|
||||
|
||||
public override func viewDidLayoutSubviews() {
|
||||
super.viewDidLayoutSubviews()
|
||||
layoutMetalLayer()
|
||||
}
|
||||
|
||||
/// The display scale to render the metal drawable at. `traitCollection.displayScale` is the
|
||||
/// canonical render scale and is reliable once the controller is in the hierarchy;
|
||||
/// `view.contentScaleFactor` can read 1.0 before the view attaches to a window/screen, which
|
||||
/// would size the drawable at point resolution → a pixelated, upscaled mess. Falls back to the
|
||||
/// main screen scale if the trait is still unspecified.
|
||||
private var renderScale: CGFloat {
|
||||
let s = traitCollection.displayScale
|
||||
return s > 0 ? s : UIScreen.main.scale
|
||||
}
|
||||
|
||||
/// Aspect-fit the stage-2 metal sublayer to the view at the canonical render scale
|
||||
/// (see SessionPresenter.layout).
|
||||
private func layoutMetalLayer() {
|
||||
presenter.layout(in: streamView.bounds, contentsScale: renderScale)
|
||||
}
|
||||
|
||||
#if os(iOS)
|
||||
private func setCaptured(_ on: Bool) {
|
||||
if on {
|
||||
// `connection != nil` is the session-active gate (presenter internals are opaque here).
|
||||
guard captureEnabled, !captured, connection != nil else { return }
|
||||
inputCapture?.setForwarding(true)
|
||||
captured = true
|
||||
} else {
|
||||
guard captured else { return }
|
||||
inputCapture?.setForwarding(false)
|
||||
captured = false
|
||||
}
|
||||
setNeedsUpdateOfPrefersPointerLocked()
|
||||
updatePointerLockChain() // (re)anchor the SwiftUI ancestors so the lock actually resolves
|
||||
syncPointerLock() // resolve cursor + GCMouse/absolute routing for the current state
|
||||
let onCaptureChange = onCaptureChange
|
||||
let captured = captured
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
onCaptureChange?(captured)
|
||||
// The lock request is async — the resolved state can land a runloop later, and the
|
||||
// initial grant may precede our didChange observer, so re-resolve the routing here.
|
||||
self?.syncPointerLock()
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve the mouse routing for the scene's CURRENT pointer-lock state: GCMouse (relative
|
||||
/// deltas + buttons) while locked, the absolute UIKit pointer path while not, and the
|
||||
/// hidden-vs-visible local cursor to match. Idempotent — safe to call on every lock-state
|
||||
/// change and capture toggle. Main queue.
|
||||
private func syncPointerLock() {
|
||||
let locked = pointerLockEngaged() == true
|
||||
let useGCMouse = captured && locked
|
||||
// Lock dropped (or capture ended) while the GCMouse path held a button down: once
|
||||
// gcMouseForwarding flips false its release handler is gated off, so flush any held
|
||||
// mouse button here before the switch — otherwise it sticks down on the host.
|
||||
if inputCapture?.gcMouseForwarding == true, !useGCMouse {
|
||||
inputCapture?.releaseMouseButtons()
|
||||
}
|
||||
inputCapture?.gcMouseForwarding = useGCMouse
|
||||
pointerInteraction?.invalidate() // re-resolve the hidden/visible cursor for the state
|
||||
if iosInputDebug {
|
||||
iosInputLog.debug(
|
||||
"pointer lock isLocked=\(locked, privacy: .public) captured=\(self.captured, privacy: .public)")
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
deinit {
|
||||
observers.forEach(NotificationCenter.default.removeObserver(_:))
|
||||
presenter.stop() // invalidate the display link + stop the pipeline if stop() was missed
|
||||
}
|
||||
}
|
||||
|
||||
#if os(iOS)
|
||||
extension StreamViewController: UIPointerInteractionDelegate {
|
||||
public func pointerInteraction(
|
||||
_ interaction: UIPointerInteraction, styleFor region: UIPointerRegion
|
||||
) -> UIPointerStyle? {
|
||||
// Hide the local cursor only when the scene is actually pointer-LOCKED — then the
|
||||
// host renders its own cursor from GCMouse deltas and a visible local one would just
|
||||
// diverge. When the lock isn't held the cursor stays VISIBLE so the user can aim; the
|
||||
// pointer is forwarded as an absolute position, both cursors tracking together.
|
||||
captured && pointerLockEngaged() == true ? .hidden() : nil
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
/// The layer-backed video surface + touch source. Touches are mapped through the
|
||||
/// aspect-fit letterbox into host-mode pixels (surface == host mode, so the host-side
|
||||
/// rescale is the identity); touches outside the video area are clamped onto its edge.
|
||||
final class StreamLayerUIView: UIView {
|
||||
override class var layerClass: AnyClass { AVSampleBufferDisplayLayer.self }
|
||||
var displayLayer: AVSampleBufferDisplayLayer {
|
||||
// swiftlint:disable:next force_cast
|
||||
layer as! AVSampleBufferDisplayLayer
|
||||
}
|
||||
|
||||
#if os(iOS)
|
||||
/// A position already mapped into host-mode pixels, with the surface dims the host
|
||||
/// rescales against (== host mode, so its rescale is the identity).
|
||||
struct HostPoint { let x: Int32; let y: Int32; let w: UInt32; let h: UInt32 }
|
||||
|
||||
/// Reads the LIVE negotiated mode in pixels (the touch/pointer coordinate space).
|
||||
var currentHostMode: (() -> CGSize)?
|
||||
/// Direct fingers / Pencil → wire touch events.
|
||||
var onTouchEvent: ((PunktfunkInputEvent) -> Void)?
|
||||
/// Indirect pointer (mouse/trackpad with no lock) → absolute cursor moves.
|
||||
var onPointerMoveAbs: ((HostPoint) -> Void)?
|
||||
/// Indirect-pointer buttons (GameStream ids: 1=left 3=right); `down` = press.
|
||||
var onPointerButton: ((_ button: UInt32, _ down: Bool) -> Void)?
|
||||
/// Trackpad two-finger / wheel scroll (no lock) → host scroll deltas, WHEEL(120)-scaled.
|
||||
var onScroll: ((_ dx: Float, _ dy: Float) -> Void)?
|
||||
|
||||
/// Wire touch ids per active direct UITouch; ids are reused after the touch ends.
|
||||
private var touchIDs: [ObjectIdentifier: UInt32] = [:]
|
||||
/// GameStream button held per active indirect-pointer touch (one click/drag session);
|
||||
/// released when that touch ends.
|
||||
private var pointerButtons: [ObjectIdentifier: UInt32] = [:]
|
||||
#endif
|
||||
|
||||
override init(frame: CGRect) {
|
||||
super.init(frame: frame)
|
||||
displayLayer.videoGravity = .resizeAspect
|
||||
#if os(iOS)
|
||||
isMultipleTouchEnabled = true
|
||||
// Button-less mouse/trackpad movement (no lock) arrives as hover, not touches —
|
||||
// forward it as absolute cursor moves so the host cursor tracks without a click held.
|
||||
addGestureRecognizer(
|
||||
UIHoverGestureRecognizer(target: self, action: #selector(handleHover)))
|
||||
// Trackpad two-finger / wheel scroll → a scroll-ONLY pan: allowedTouchTypes = []
|
||||
// rejects finger drags (those stay host touches), allowedScrollTypesMask accepts the
|
||||
// indirect scroll devices. Forwarded as host scroll deltas.
|
||||
let scrollPan = UIPanGestureRecognizer(target: self, action: #selector(handleScroll))
|
||||
scrollPan.allowedScrollTypesMask = .all
|
||||
scrollPan.allowedTouchTypes = []
|
||||
addGestureRecognizer(scrollPan)
|
||||
#endif
|
||||
backgroundColor = .black
|
||||
}
|
||||
|
||||
@available(*, unavailable)
|
||||
required init?(coder: NSCoder) { fatalError("not used") }
|
||||
|
||||
#if os(iOS)
|
||||
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
|
||||
route(touches, event: event, kind: .down)
|
||||
}
|
||||
override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
|
||||
route(touches, event: event, kind: .move)
|
||||
}
|
||||
override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
|
||||
route(touches, event: event, kind: .up)
|
||||
}
|
||||
override func touchesCancelled(_ touches: Set<UITouch>, with event: UIEvent?) {
|
||||
route(touches, event: event, kind: .up)
|
||||
}
|
||||
|
||||
private enum TouchKind { case down, move, up }
|
||||
|
||||
/// Split a touch batch by kind: an INDIRECT POINTER (mouse/trackpad with no lock) drives
|
||||
/// the host cursor as an absolute mouse; everything else (direct finger, Pencil) is a host
|
||||
/// touch. Mixed batches are possible, so partition rather than branch on the first touch.
|
||||
private func route(_ touches: Set<UITouch>, event: UIEvent?, kind: TouchKind) {
|
||||
var fingers: Set<UITouch> = []
|
||||
for touch in touches {
|
||||
if touch.type == .indirectPointer {
|
||||
handleIndirectPointer(touch, event: event, kind: kind)
|
||||
} else {
|
||||
fingers.insert(touch)
|
||||
}
|
||||
}
|
||||
if !fingers.isEmpty { forwardTouches(fingers, kind: kind) }
|
||||
}
|
||||
|
||||
/// An indirect-pointer touch is a button-held click/drag session: forward its position as
|
||||
/// an absolute cursor move and its button as a mouse button (down on begin, up on end).
|
||||
private func handleIndirectPointer(_ touch: UITouch, event: UIEvent?, kind: TouchKind) {
|
||||
let key = ObjectIdentifier(touch)
|
||||
let host = hostPoint(from: touch.location(in: self))
|
||||
switch kind {
|
||||
case .down:
|
||||
let button = Self.gsButton(for: event?.buttonMask ?? .primary)
|
||||
pointerButtons[key] = button
|
||||
if let host { onPointerMoveAbs?(host) } // place the cursor, then press
|
||||
onPointerButton?(button, true)
|
||||
case .move:
|
||||
if let host { onPointerMoveAbs?(host) }
|
||||
case .up:
|
||||
if let host { onPointerMoveAbs?(host) }
|
||||
if let button = pointerButtons.removeValue(forKey: key) {
|
||||
onPointerButton?(button, false)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func forwardTouches(_ touches: Set<UITouch>, kind: TouchKind) {
|
||||
guard onTouchEvent != nil else { return }
|
||||
for touch in touches {
|
||||
let key = ObjectIdentifier(touch)
|
||||
let id: UInt32
|
||||
switch kind {
|
||||
case .down:
|
||||
id = nextFreeID()
|
||||
touchIDs[key] = id
|
||||
case .move, .up:
|
||||
guard let known = touchIDs[key] else { continue }
|
||||
id = known
|
||||
}
|
||||
if kind == .up {
|
||||
touchIDs.removeValue(forKey: key)
|
||||
onTouchEvent?(.touchUp(id: id))
|
||||
continue
|
||||
}
|
||||
guard let h = hostPoint(from: touch.location(in: self)) else { continue }
|
||||
onTouchEvent?(
|
||||
kind == .down
|
||||
? .touchDown(id: id, x: h.x, y: h.y, surfaceWidth: h.w, surfaceHeight: h.h)
|
||||
: .touchMove(id: id, x: h.x, y: h.y, surfaceWidth: h.w, surfaceHeight: h.h))
|
||||
}
|
||||
}
|
||||
|
||||
/// Button-less mouse/trackpad movement (no lock) → absolute cursor move.
|
||||
@objc private func handleHover(_ recognizer: UIHoverGestureRecognizer) {
|
||||
switch recognizer.state {
|
||||
case .began, .changed:
|
||||
if let h = hostPoint(from: recognizer.location(in: self)) { onPointerMoveAbs?(h) }
|
||||
default:
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/// Trackpad / wheel scroll (no lock) → host scroll deltas. The translation is consumed
|
||||
/// each callback so the next is a fresh delta. Sign/scale are tunable (≈ one notch per
|
||||
/// ~10 pt): finger up scrolls up (host +y), x passes through — the host WHEEL convention.
|
||||
@objc private func handleScroll(_ g: UIPanGestureRecognizer) {
|
||||
guard g.state == .began || g.state == .changed else { return }
|
||||
let t = g.translation(in: self)
|
||||
g.setTranslation(.zero, in: self)
|
||||
onScroll?(Float(t.x) * 12, Float(-t.y) * 12)
|
||||
}
|
||||
|
||||
/// Map a view-space point through the aspect-fit letterbox into host-mode pixels; points
|
||||
/// outside the video area clamp onto its edge. nil until a mode is negotiated.
|
||||
private func hostPoint(from p: CGPoint) -> HostPoint? {
|
||||
guard let hostMode = currentHostMode?(), hostMode.width > 0, hostMode.height > 0
|
||||
else { return nil }
|
||||
let video = AVMakeRect(aspectRatio: hostMode, insideRect: bounds)
|
||||
guard video.width > 0, video.height > 0 else { return nil }
|
||||
let x = Int32(((p.x - video.minX) / video.width * hostMode.width)
|
||||
.rounded().clamped(to: 0...(hostMode.width - 1)))
|
||||
let y = Int32(((p.y - video.minY) / video.height * hostMode.height)
|
||||
.rounded().clamped(to: 0...(hostMode.height - 1)))
|
||||
return HostPoint(x: x, y: y, w: UInt32(hostMode.width), h: UInt32(hostMode.height))
|
||||
}
|
||||
|
||||
/// `.secondary` (right button / two-finger click) → GameStream right (3); else left (1).
|
||||
private static func gsButton(for mask: UIEvent.ButtonMask) -> UInt32 {
|
||||
mask.contains(.secondary) ? 3 : 1
|
||||
}
|
||||
|
||||
private func nextFreeID() -> UInt32 {
|
||||
var id: UInt32 = 0
|
||||
while touchIDs.values.contains(id) { id += 1 }
|
||||
return id
|
||||
}
|
||||
#endif
|
||||
}
|
||||
#endif
|
||||
Reference in New Issue
Block a user