A 6-agent adversarial audit of the client (11 confirmed of 39 findings, the rest
filtered) drove these:
- fix: SessionAudio ring buffer — guard a write larger than the ring (would push
readIdx past writeIdx and corrupt the buffer; never happens, but guard not corrupt).
- fix: CADisplayLink retain cycle (stage-2 presenter) — a weak-target DisplayLinkProxy
so the view can deallocate (the link retains its target); stage-2 teardown added to
both StreamView/StreamViewController deinits as a safety net.
- fix: GamepadFeedback deinit { flag.stop() } — the drain thread holds the connection
strongly and self weakly, so an abrupt teardown without stop() would leak it.
- refactor: centralize the 12 UserDefaults/@AppStorage key literals (scattered across
8 files) into one DefaultsKey enum — a typo silently splits a setting's reader from
its writer.
- docs: RumbleRenderer @unchecked Sendable invariant; the HID digit-row table; the
stage-2 layer compositing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opt-in (Settings -> Presenter; `punktfunk.presenter`, default stage-1). Stage-1's
AVSampleBufferDisplayLayer decodes AND presents internally with no per-frame
callback, so neither decode nor present can be stamped or hand-paced. Stage-2
takes explicit control:
- VideoDecoder: VTDecompressionSession, async output callback stamps
decode-completion, session rebuilt on every IDR / format change. Unit-tested
(testVideoDecoderAsyncCallbackDeliversPixels).
- MetalVideoPresenter: CAMetalLayer + CVMetalTextureCache + a runtime-compiled
BT.709 limited-range NV12->RGB shader, present at the next vsync. The
CVMetalTextures + pixel buffer are held until the GPU completes.
- Stage2Pipeline: pump thread -> decoder -> newest-ready 1-slot ring; the hosting
view's display link drains it once per vsync and stamps capture->present
(the display-link target time projected into CLOCK_REALTIME).
- LatencyMeter gains record(ptsNs:atNs:offsetNs:); the HUD shows a capture->present
(glass-to-glass, modulo host render->capture) line, skew-corrected via
clockOffsetNs. Measured live ~11 ms p50 vs ~2.2 ms capture->client.
- StreamView / StreamViewIOS host the CAMetalLayer as a sublayer + a CADisplayLink
(NSView.displayLink on macOS) when stage-2; input capture + HUD unchanged. The
session-active gates switch from `pump != nil` to `connection != nil` so capture
engages without a StreamPump.
Validated: builds macOS/iOS/tvOS; the decode half is unit-tested; the Metal
present is live-validated on glass (correct image + the capture->present number).
Colorspace is BT.709 SDR for now; 10-bit/HDR + a pacing policy are later.
Plan: docs-site/content/docs/apple-stage2-presenter.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Input handling, building on macOS/iOS/tvOS:
- macOS recapture after navigating out: engageCapture no longer latches
captured=true when the cursor grab is refused mid app-activation (which left
a free cursor that no later click could re-grab); cursorCapture.capture() now
reports success. + canBecomeKeyView.
- iOS/iPadOS recapture: restore the prior capture on didBecomeActive (nothing
re-grabbed mouse/keyboard on return before).
- iPad indirect pointer (no lock) is forwarded as an absolute MOUSE (move +
buttons + scroll via hover / UITouch.indirectPointer), not as touch, with the
local cursor visible; GCMouse owns the locked regime, gated so the two never
double-send. Adds the MouseMoveAbs wire helper.
- Trackpad scroll on iOS (was entirely missing): GCMouse scroll dpad when
locked + a scroll-only UIPanGestureRecognizer otherwise.
- tvOS: no focusable control during play (a focusable Disconnect button ate the
controller's A in the focus engine); Siri Remote Menu disconnects.
- Don't leak touch to the host under the TOFU trust prompt (gate on
captureEnabled).
LAN discovery: HostDiscovery (NWBrowser over _punktfunk._udp, the host's
crate::discovery advert) resolves each service to IP:port and parses the TXT
(fp advisory, pair, id); an "On this network" section in the grid (tap to save
+ connect, or pair if required). iOS/tvOS get NSBonjourServices via a merged
Config/Info.plist. Integration-tested end to end against a fake NWListener advert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side logs proved the macOS client sent keyboard + scroll but ZERO relative
mouse-motion and ZERO button events for an entire session — the user was moving
the mouse the whole time. Root cause is client-side: GCMouse's
mouseMovedHandler/pressedChangedHandler silently never fired on the live Mac
(a documented GameController quirk) while GCKeyboard worked and scroll already
rode NSEvent. So motion/buttons were the only input on a GCMouse-only path, and
that path was dead.
macOS: stop relying on GCMouse for motion/buttons (compiled out with
#if !os(macOS)); drive them from a local NSEvent monitor installed only while
captured — the same channel scrollWheel already uses successfully. Under
CGAssociateMouseAndMouseCursorPosition(false) the mouseMoved/dragged deltaX/deltaY
ARE the relative motion (OS-acceleration-applied, exactly what Moonlight's macOS
client ships). All four motion event types are covered so motion keeps flowing
during a button-held drag; buttons map left/right/middle/X1/X2 through the
existing engage-click-suppression + release-on-blur logic. NSEvent deltaY is
already screen-space (+y down) so, unlike the GCMouse path, it is NOT negated.
iPad: the input failure there was a different cause — GCMouse only delivers
relative deltas while the scene holds a true pointer LOCK, which the system grants
only to a full-screen, frontmost iPad scene and which UIHostingController doesn't
consult for children. Gate prefersPointerLocked to iPad + captured, add
childViewControllerForPointerLock so a reparenting container forwards the lock
decision to this VC, and log the resolved lock state. Touch remains the
unconditional fallback.
Adds a PUNKTFUNK_INPUT_DEBUG=1 switch (os.Logger, throttled) so motion/buttons
being SENT is verifiable on-device without host-side logs. iOS GCMouse path
otherwise unchanged; GCKeyboard unchanged on both. Researched + adversarially
reviewed; Swift builds only on a Mac, so this is unverified-compiled here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The same app now runs on tvOS (target Punktfunk-tvOS, bundle io.unom.punktfunk.tvos),
validated live against the box: vkcube at 1280x720@60, 60 fps in the Apple TV 4K
simulator, glass HUD with a focusable Disconnect button.
- PunktfunkCore.xcframework grows tvOS device + universal-simulator slices. These are
TIER-3 Rust targets (no prebuilt std): BUILD_TVOS=1 builds them with nightly and
-Zbuild-std from rust-src — the full quic stack (quinn/rustls-ring/tokio) compiles
for tvOS unchanged.
- The UIKit stream view covers iOS AND tvOS, with pointer interaction, pointer lock,
touch forwarding and InputCapture gated to iOS — tvOS is view-only until gamepad
capture lands (the natural tvOS input).
- SessionAudio on tvOS: .playback session, no mic (no app-accessible microphone).
- App chrome gates: keyboardShortcut/textSelection/controlSize/statusBarHidden are
iOS/macOS-only; host cards use the focus-native .card button style on tvOS; the
Audio settings section hides (system-routed); mode seeding works from the TV screen
(1920x1080@60).
- Package platforms += .tvOS(.v17); new Xcode target + shared scheme
(TARGETED_DEVICE_FAMILY 3, local-network usage description included).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Streaming on iPad left the status bar up and the video boxed inside the safe areas, on
top of a 16:9 default mode letterboxing on the 4:3 screen, with the iPadOS cursor
hovering over the video. The session view is now immersive on iOS:
- .ignoresSafeArea + .statusBarHidden + .persistentSystemOverlays(.hidden) for the
session only (home gets its chrome back on disconnect).
- First run seeds the stream mode from the device's native screen
(UIScreen.nativeBounds + maximumFramesPerSecond) instead of 1920×1080 — verified
live: a fresh install negotiated the iPad's 2752×2064 with the host. macOS keeps the
1080p default (a desktop window is not the screen).
- The iPadOS cursor hides while over the video (UIPointerInteraction .hidden(),
re-resolved on capture toggles) — the host renders its own cursor from our deltas;
true pointer lock through UIHostingController remains the documented gap.
Found along the way (host-side, not fixed here): at very high modes a keyframe burst
can fill the UDP send buffer and m3 treats the sendmmsg WouldBlock as fatal
("session ended with error: submit_frame: WouldBlock") instead of backpressuring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The whole client now runs on iPadOS/iOS from the same sources, first-lit live in the
iPad simulator against the real host at 1280x720@60 (60 fps on the HUD, capture state
machine active, mic permission flow shown).
- PunktfunkCore.xcframework grows iOS device + universal-simulator slices
(BUILD_IOS=1; rustup targets aarch64-apple-ios{,-sim} + x86_64-apple-ios).
- The decode pump is extracted into a shared StreamPump (identical IDR re-gate logic on
both platforms); the iOS StreamView (StreamViewIOS.swift) has the same name/signature
as the macOS one, so ContentView & co. are byte-identical across platforms — hosted
in a UIViewController for prefersPointerLocked (the iPadOS cursor capture; see README
note 9 for the UIHostingController forwarding caveat).
- Touch is always forwarded: per-finger wire ids, coordinates mapped through the
aspect-fit letterbox into LIVE host-mode pixels (surface == host mode, identity
rescale host-side; follows mid-stream requestMode switches).
- InputCapture is cross-platform: GC works the same on iPadOS, ⌘⎋ is detected from the
HID stream there; stale-⌘ tracking after focus loss fixed on both platforms
(releaseAll now drops the modifier/latch state — a ⌘ released in another app
otherwise hijacked Esc forever).
- SessionAudio: AVAudioSession on iOS (.playAndRecord + .defaultToSpeaker — without it
iPhones route host audio to the EARPIECE; deactivated with
notifyOthersOnDeactivation on stop so interrupted background audio resumes); HAL
device pinning + the Settings pickers stay macOS-only.
- New Punktfunk-iOS app target (shared synchronized sources, generated Info.plist with
mic + local-network usage descriptions — QUIC to a LAN host trips local network
privacy on real devices — scene manifest + indirect input events for Stage Manager /
external displays), shared scheme, macOS min-window frames gated off iOS.
For the iPad-on-an-external-screen idea: with multiple scenes + indirect input enabled,
Stage Manager iPads can drag the punktfunk window onto the external display and drive
the PC with keyboard/mouse/touch. Known gaps (README note 9): the pointer-lock
preference isn't consulted through UIHostingController (relative mouse works, the local
cursor just stays visible) and AVAudioSession interruptions don't auto-restart audio.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
enricobuehler