Input handling, building on macOS/iOS/tvOS:
- macOS recapture after navigating out: engageCapture no longer latches
captured=true when the cursor grab is refused mid app-activation (which left
a free cursor that no later click could re-grab); cursorCapture.capture() now
reports success. + canBecomeKeyView.
- iOS/iPadOS recapture: restore the prior capture on didBecomeActive (nothing
re-grabbed mouse/keyboard on return before).
- iPad indirect pointer (no lock) is forwarded as an absolute MOUSE (move +
buttons + scroll via hover / UITouch.indirectPointer), not as touch, with the
local cursor visible; GCMouse owns the locked regime, gated so the two never
double-send. Adds the MouseMoveAbs wire helper.
- Trackpad scroll on iOS (was entirely missing): GCMouse scroll dpad when
locked + a scroll-only UIPanGestureRecognizer otherwise.
- tvOS: no focusable control during play (a focusable Disconnect button ate the
controller's A in the focus engine); Siri Remote Menu disconnects.
- Don't leak touch to the host under the TOFU trust prompt (gate on
captureEnabled).
LAN discovery: HostDiscovery (NWBrowser over _punktfunk._udp, the host's
crate::discovery advert) resolves each service to IP:port and parses the TXT
(fp advisory, pair, id); an "On this network" section in the grid (tap to save
+ connect, or pair if required). iOS/tvOS get NSBonjourServices via a merged
Config/Info.plist. Integration-tested end to end against a fake NWListener advert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
macOS GCKeyboard delivery is flaky — the same GameController quirk that
killed GCMouse motion (e414ec0). Keyboard input intermittently failed to
reach the host (e.g. typing in a gamescope game). Switch the macOS key
source to NSEvent, mirroring the mouse fix:
- StreamLayerView.keyDown/keyUp map NSEvent.keyCode (Carbon virtual
keycode) → Windows VK via the new InputCapture.keyCodeToVK table and
forward through InputCapture.sendKey, then consume the event (no beep).
- flagsChanged drives InputCapture.handleFlagsChanged, which diffs the raw
modifier flags to recover each L/R modifier down/up (modifiers never fire
keyDown/keyUp on macOS) and emits the same L/R VKs hidToVK already does.
- The macOS GCKeyboard keyChangedHandler is disabled (#if !os(macOS)) so it
can't double-send; iOS keeps the GCKeyboard path unchanged.
sendKey honors the ⌘⎋ capture-toggle suppressedVK latch and tracks into
pressedVKs so releaseAll()/blur flushes anything still held. The emitted
VKs are identical to the existing HID path, so the host (vk_to_evdev)
needs no change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side logs proved the macOS client sent keyboard + scroll but ZERO relative
mouse-motion and ZERO button events for an entire session — the user was moving
the mouse the whole time. Root cause is client-side: GCMouse's
mouseMovedHandler/pressedChangedHandler silently never fired on the live Mac
(a documented GameController quirk) while GCKeyboard worked and scroll already
rode NSEvent. So motion/buttons were the only input on a GCMouse-only path, and
that path was dead.
macOS: stop relying on GCMouse for motion/buttons (compiled out with
#if !os(macOS)); drive them from a local NSEvent monitor installed only while
captured — the same channel scrollWheel already uses successfully. Under
CGAssociateMouseAndMouseCursorPosition(false) the mouseMoved/dragged deltaX/deltaY
ARE the relative motion (OS-acceleration-applied, exactly what Moonlight's macOS
client ships). All four motion event types are covered so motion keeps flowing
during a button-held drag; buttons map left/right/middle/X1/X2 through the
existing engage-click-suppression + release-on-blur logic. NSEvent deltaY is
already screen-space (+y down) so, unlike the GCMouse path, it is NOT negated.
iPad: the input failure there was a different cause — GCMouse only delivers
relative deltas while the scene holds a true pointer LOCK, which the system grants
only to a full-screen, frontmost iPad scene and which UIHostingController doesn't
consult for children. Gate prefersPointerLocked to iPad + captured, add
childViewControllerForPointerLock so a reparenting container forwards the lock
decision to this VC, and log the resolved lock state. Touch remains the
unconditional fallback.
Adds a PUNKTFUNK_INPUT_DEBUG=1 switch (os.Logger, throttled) so motion/buttons
being SENT is verifiable on-device without host-side logs. iOS GCMouse path
otherwise unchanged; GCKeyboard unchanged on both. Researched + adversarially
reviewed; Swift builds only on a Mac, so this is unverified-compiled here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The whole client now runs on iPadOS/iOS from the same sources, first-lit live in the
iPad simulator against the real host at 1280x720@60 (60 fps on the HUD, capture state
machine active, mic permission flow shown).
- PunktfunkCore.xcframework grows iOS device + universal-simulator slices
(BUILD_IOS=1; rustup targets aarch64-apple-ios{,-sim} + x86_64-apple-ios).
- The decode pump is extracted into a shared StreamPump (identical IDR re-gate logic on
both platforms); the iOS StreamView (StreamViewIOS.swift) has the same name/signature
as the macOS one, so ContentView & co. are byte-identical across platforms — hosted
in a UIViewController for prefersPointerLocked (the iPadOS cursor capture; see README
note 9 for the UIHostingController forwarding caveat).
- Touch is always forwarded: per-finger wire ids, coordinates mapped through the
aspect-fit letterbox into LIVE host-mode pixels (surface == host mode, identity
rescale host-side; follows mid-stream requestMode switches).
- InputCapture is cross-platform: GC works the same on iPadOS, ⌘⎋ is detected from the
HID stream there; stale-⌘ tracking after focus loss fixed on both platforms
(releaseAll now drops the modifier/latch state — a ⌘ released in another app
otherwise hijacked Esc forever).
- SessionAudio: AVAudioSession on iOS (.playAndRecord + .defaultToSpeaker — without it
iPhones route host audio to the EARPIECE; deactivated with
notifyOthersOnDeactivation on stop so interrupted background audio resumes); HAL
device pinning + the Settings pickers stay macOS-only.
- New Punktfunk-iOS app target (shared synchronized sources, generated Info.plist with
mic + local-network usage descriptions — QUIC to a LAN host trips local network
privacy on real devices — scene manifest + indirect input events for Stage Manager /
external displays), shared scheme, macOS min-window frames gated off iOS.
For the iPad-on-an-external-screen idea: with multiple scenes + indirect input enabled,
Stage Manager iPads can drag the punktfunk window onto the external display and drive
the PC with keyboard/mouse/touch. Known gaps (README note 9): the pointer-lock
preference isn't consulted through UIHostingController (relative mouse works, the local
cursor just stays visible) and AVAudioSession interruptions don't auto-restart audio.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scroll was wired to GCMouse's scroll dpad, which only fires for plain HID wheel
deltas — trackpad and Magic Mouse scrolling are gesture events that never reach
GameController, so scrolling was dead on the default Mac setups. The stream view now
overrides scrollWheel (while captured the cursor is parked mid-view, so it receives
every scroll event) and feeds InputCapture.sendScroll: precise gesture deltas are
pixels (~0.1 notch/px, SDL's factor → ×12 for WHEEL_DELTA(120)), classic wheels are
lines (×120), fractional remainders accumulate, and the GC scroll handler is gone so
wheel mice can't double-deliver. Signs pass through as-is, preserving the local
(natural-)scrolling preference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Capture used to engage whenever the app became active, so the click that activates the
window — on the title bar (a drag) or a resize edge — got the cursor warped away
mid-gesture, and raw deltas kept streaming to the host while the user fought the window.
Reworked Moonlight-style, with capture as a deliberate, reversible state owned by
StreamLayerView:
- Engage: automatically once when the stream starts / trust is confirmed (one-shot, can
never fire surprisingly later), or by clicking into the video (that click's
press/release are suppressed toward the host; acceptsFirstMouse makes it one click
from another app). NEVER on app re-activation.
- Release: ⌘⎋ (toggles, key-window-scoped), focus loss — now including same-app window
switches (⌘, / ⌘N / ⌘M resign key without resigning the app; previously the new
window inherited a hidden frozen cursor and its typing was double-delivered to the
host) — and disconnect.
- While released: nothing is forwarded (InputCapture.forwarding gates the GC handlers;
held keys/buttons are flushed host-side so nothing sticks), the cursor is free, and
the HUD (now showing the capture state) is clickable.
- The no-beep behavior moved from the NSEvent monitor to first-responder key
consumption — swallowing at the monitor risked starving GC's own delivery (the
"input broken altogether" report). The monitor now only intercepts ⌘⎋.
- Adversarial-review fixes: a second session preempts the previous one cleanly instead
of leaving it captured with dead GC handlers (onPreempted); the engage click's
suppression latch can't outlive the click (mouseUp backstop); ⌘⎋'s physical Esc can't
type into the host in either toggle direction (suppressedVK latch + Esc-while-⌘
guard); capture callbacks defer out of the SwiftUI update pass.
Validated live against the box: 16185 input datagrams injected during a captured
session (gamescope EIS), title-bar drag/resize free while released, and visible
cursor + typing on a streamed KWin desktop, all user-confirmed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The app grows from a dev connect form into a real client shell:
- Home is a grid of saved hosts (UserDefaults-persisted; context menu: Remove / Forget
Identity), "+" in the toolbar opens the add-host sheet, the stream mode moved into
Settings (⌘, / gear) — native resolution stays the only mode, no scaling.
- Trust is now explicit: the protocol always supported certificate pinning, but the app
passed no pin and discarded the observed fingerprint — silently trusting any host.
First connect now shows the host's SHA-256 fingerprint (compare with the "clients pin
this fingerprint" line in the host log) over the live-but-blurred stream; the stream
must pump immediately (the opening IDR is the only guaranteed one), so StreamView gains
a capturesCursor switch to keep the cursor free while the prompt needs clicking, and
input capture starts only after confirmation. Trusting pins the fingerprint per host;
a changed host identity then refuses to connect.
- PUNKTFUNK_AUTOCONNECT keeps working (auto-trusts, doesn't touch the saved hosts).
Host→client authorization (pairing PIN) remains a punktfunk-core roadmap item — the host
still accepts any client that can reach its port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The host renders its own cursor from our raw deltas, so the local macOS cursor both
stays visible and drifts away from the remote one — and it can wander out of the window,
where a click focuses another app. While the stream has focus, do what Moonlight does:
warp the cursor mid-view, disconnect it from mouse movement
(CGAssociateMouseAndMouseCursorPosition(false) — GCMouse still delivers raw HID deltas),
and hide it. Released on app deactivation (Cmd+Tab is the escape hatch), view teardown,
and disconnect; re-captured when the stream regains focus. The HUD's Disconnect gains ⌘D
since a hidden, frozen cursor can't click it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full project rename, decided 2026-06-10:
- Crates/binaries: punktfunk-core / punktfunk-host / punktfunk-client-rs.
- C ABI: punktfunk_* symbols, Punktfunk* types, include/punktfunk_core.h,
PUNKTFUNK_FEATURE_QUIC guard (header regenerated; cbindgen renames updated, incl.
PUNKTFUNK_BTN_*/PUNKTFUNK_AXIS_* wire constants).
- Protocol: punktfunk/1 — control-plane magic LMN1 → PKF1, nonce salt lmn1 → pkf1.
WIRE BREAK: clients must be rebuilt from this revision.
- Env knobs: PUNKTFUNK_VIDEO_SOURCE / PUNKTFUNK_COMPOSITOR / PUNKTFUNK_ZEROCOPY / ….
- Host config dir: ~/.config/punktfunk (the box's dir was migrated in place — the
persistent identity is unchanged, pinned fingerprints stay valid).
- Swift package: PunktfunkKit + PunktfunkCore.xcframework + PunktfunkConnection
(Sources/PunktfunkClient app + tests renamed with it); build-xcframework.sh updated.
- scripts/: 60-punktfunk.rules, punktfunk-host.service; OpenAPI doc regenerated.
Also: scripts/headless/run-headless-kde.sh — full headless Plasma bringup. Root cause of
"desktop but no apps/settings" over the stream: plasmashell launched without
XDG_MENU_PREFIX=plasma-, so the launcher resolved a nonexistent applications.menu and
rendered an empty menu. The script sets the complete KDE session env (menu prefix,
KDE_FULL_SESSION, session version) and rebuilds ksycoca before starting plasmashell.
Gate: 97/97 tests, clippy -D warnings (both feature sets), fmt, C-ABI harness PASS,
zero lumen references left outside .git.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler