Modifier keys arrive only as flagsChanged, and the direction was recovered
by diffing the device-dependent L/R bits (NX_DEVICE*KEYMASK) alone. Those
bits are undocumented and some keyboards omit them (only the class bit,
e.g. NX_CONTROLMASK, is set), so the diff saw no transition and the key
never reached the host — no Ctrl shortcuts. SDL/Moonlight key off the
event's keyCode for exactly this reason; do the same: keyCode names the
changed key, the class bit says up, the device bits (when present) pick
the side, and a tracked-held-state flip covers keyboards without them.
PUNKTFUNK_INPUT_DEBUG=1 now also logs every flagsChanged (keyCode + raw
flags) so a field report is diagnosable from client logs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mic uplink handed the host pure digital silence on a multi-channel
interface: AVAudioConverter's N→stereo downmix takes channels 0/1, but a
pro interface puts the mic on ONE higher discrete channel. Fold the input
to a mono bus ourselves instead — pick the mic's channel (or sum all) and
resample that to the encoder's 48 kHz stereo, so the silent 0/1 downmix
never happens.
- New "Microphone channel" setting (macOS): Auto (sum every channel — a
lone hot mic passes at full level) or pin 1-based channel N. Picker
appears only for multi-channel devices, driven by the device's input
channel count.
- Diagnostics that make this class of failure self-naming next session:
log the actual live capture device + format + fold mode, warn on a
silent UID fallback, and a one-shot silence tripwire on the EXTRACTED
signal (WARN on 10 s of zeros, else peak dBFS).
- foldToMono extracted as a pure, unit-tested helper (pin / sum-clamp x
interleaved / deinterleaved / mono / out-of-range).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consumes the 0xCF host-timing plane (99cb8dd) on all four GUI clients: each
keeps a bounded pending ring of receipt samples keyed by pts, matches the
host's per-AU capture→sent reports against it, and the HUD equation becomes
= host 3.1 + network 6.7 + decode 2.1 + display 2.3
falling back to the combined `= host+network …` term whenever no timing
matched the window (old host / datagram loss) — same total, one split
fewer, never a misleading zero. Apple additionally gains the split as the
only equation line under the stage-1 fallback presenter (receipt is
presenter-independent), a `nextHostTiming` wrapper with its own plane lock,
and a unit-tested `HostNetworkSplitter`; Android extends the JNI stats
array 16→18 doubles (0–15 unchanged); Windows/Linux thread the split
through `Stats` into the HUD and the headless/debug logs.
Docs updated: design/stats-unification.md Phase 2 → implemented (wire
format, fallback semantics), and the docs-site matrix's Sunshine "Host
processing latency" row is now a direct match (ours includes the paced
send; avg vs p50).
Verified here: linux client clippy -D warnings green on the live tree,
windows stub check + hand-verified diff, android cargo-ndk arm64 check
green, apple loopback test extended (needs the rebuilt xcframework + swift
test on the mac). On-glass: pending on all platforms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One stat model everywhere (design/stats-unification.md): four measurement
points (capture/received/decoded/displayed), three stages that tile the
interval exactly, and a HUD that shows the addition explicitly —
end-to-end 14.2 ms p50 · 19.8 p95 · capture→on-glass
= host+network 9.8 + decode 2.1 + display 2.3
replacing each client's ad-hoc mix of overlapping absolutes (the Apple HUD's
three arrow lines that looked sequential but weren't), mean-vs-median decode
times (Windows/Linux), missing same-host-clock flags (Windows/Linux), and
three different names for the same capture→received measurement (probe's
"reassembled", Apple/Android's "client", Windows/Linux's post-decode "lat").
Per client: Apple threads receivedNs through the VT decode via the frame
refcon bit pattern so the decode stage exists at all (stage-1 fallback
honestly degrades to a capture→received headline); Windows carries
FrameTimes through the existing frame channel to the render thread and adds
e2e p50/p95 post-Present; Linux stamps received at AU pop and rides
decoded_ns on DecodedFrame to the paintable-set site; Android pairs receipt
stamps with MediaCodec output buffers via the codec's pts round-trip (JNI
stats array 14→16 doubles, indexes 0-13 unchanged). fps now uniformly counts
received AUs; lost/(received+lost) per window, hidden at zero.
docs-site gains "Understanding the Stats Overlay": what each line means, why
the equation only approximately sums (percentiles), and a line-by-line
Moonlight/Sunshine matrix — including that Moonlight has no end-to-end
number and its "network latency" is an ENet control RTT, so punktfunk's
headline must not be compared against any single Moonlight line.
Verified here: linux client + probe + core check/clippy/fmt green, android
native cargo-ndk arm64 check green. Pending: Windows CI + on-glass, swift
test on the mac, on-device Android.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The two touch clients had exactly complementary gaps: iOS forwarded fingers
ONLY as raw wire touches (no way to drive the host cursor from the touch
screen), Android had the two mouse modes but no passthrough. Both now share
one three-way "Touch input" setting: Trackpad (default) / Direct pointer /
Touch passthrough.
iOS/iPadOS: Input/TouchMouse.swift ports the Android gesture engine 1:1
(same px-based acceleration curve; tap=click, two-finger tap=right-click,
two-finger drag=scroll, tap-then-drag=held drag, three-finger tap=stats
HUD via the shared hudEnabled default); direct-pointer mode maps through
the aspect-fit letterbox; the previous always-on behavior lives on as the
passthrough option. The mode latches per gesture (a Settings change never
splits one gesture across models), touchesCancelled releases held state
without synthesizing a click, and session stop flushes a mid-drag button.
Settings picker on iPhone + iPad next to the iPad-only pointer-capture
toggle. Deliberate default change: trackpad, not passthrough.
Android: new nativeSendTouch JNI shim → wire TouchDown/Move/Up (the host
already injects real touch on every backend — libei touchscreen, wlroots,
KWin fake-input, SendInput); streamTouchPassthrough forwards every finger
with stable ids and lifts still-held contacts on teardown; the trackpadMode
Boolean becomes the TouchMode enum (old pref migrated on load, never
rewritten) with a Settings dropdown.
Verified: macOS swift build + full suite (incl. new TouchMouseTests), iOS
Simulator Swift compile, cargo check/fmt/clippy on the native crate, Kotlin
app+kit compile + unit tests. On-glass feel of the iOS ballistics and
Android passthrough against a touch-aware app still pending.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ground-up RumbleRenderer rewrite around one principle: rumble is idempotent
state on a lossy channel, and the actuator's divergence from it must be
bounded, not best-effort. The old renderer rebuilt an infinite-duration
CHHapticAdvancedPatternPlayer per 0xCA datagram via an async stop; one stop
lost inside CoreHaptics left an unstoppable player buzzing forever (the
"entered the menu and rumble never stopped" bug).
- Finite 4 s segments, never infinite events — a leaked player self-silences;
steady levels re-arm seamlessly ON the engine timeline (no stop/start race)
- GamepadFeedback drains the rumble plane DRY per cycle, newest-wins (was one
datagram per 8 ms through a 16-deep drop-newest queue = lag + shed stops)
- Host 500 ms state refreshes dedupe to a liveness stamp; zero applies
immediately; nonzero ramps throttle to one rebake/25 ms per motor
- Throwing player stop escalates to engine.stop() (kills leaked players);
1.6 s staleness watchdog (Policy.session) force-silences on a dead channel;
the test panel holds levels via Policy.manual
- Plain makePlayer, NEVER makeAdvancedPlayer: gamecontrollerd's controller
haptics server advertises `adv players: 0`, and iOS 27 beta 2 hard-drops
advanced loads with an XPC decode fault (-4811/4097, rumble silently dead).
Live-verified on an iOS 27 beta 2 iPhone: DualSense rumble works
- Split-handle engines fall back to one combined .default engine on repeated
failure; renderer publishes health transitions and the test panel shows
them (a refused system service no longer reads as silent app breakage)
- Per-motor sharpness on split handles (0.3 heavy / 0.7 light); macOS
DualSense raw-HID path gains a ~1 s keepalive re-write while nonzero
- RumbleTuningTests pin the scheduling math, tuning relations, and a
queue/ticker teardown smoke test
Stuck-rumble streaming repro revalidation on glass still pending.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sources reorganized (client: Home/Session/Settings/Stores/Support/Trust; kit:
Audio/Connection/Gamepad/Input/Support/Video/Views) with the big files split
along the same seams.
The gamepad mode is couch-complete, and now on macOS too (the living-room
Mac case), not just iOS/iPadOS:
- GamepadSettingsView: a console-style, fully controller-navigable settings
screen (X from the launcher) — up/down moves focus, left/right steps values
(clamped, boundary thud), A cycles/toggles, B closes; the focused row shows a
one-line description. Backed by GamepadMenuList, the vertical sibling of
GamepadCarousel, and SettingsOptions — the option lists hoisted out of
SettingsView statics and shared by the touch, tvOS and gamepad settings.
- GamepadAddHostView + GamepadKeyboard: register a host end to end with a pad
— field rows open an on-screen controller keyboard (dpad grid, A types,
X backspaces, B done); the launcher carousel ends in an Add Host tile, so
the dead-end "add one with touch first" empty state is gone.
- Launcher polish: contextual hint bar with the pad's real button glyphs,
controller name + battery chip, one shared console chrome.
- GamepadScreenBackground: an animated aurora (TimelineView-driven drifting
blobs in the brand's violet family, breathing radii, slow hue shift,
legibility scrim; freezes under Reduce Motion). Pure SwiftUI on purpose — a
.metal library only bundles reliably in one of the two build systems (SPM vs
the xcodeproj's synced folders) these sources compile under.
- macOS port: settings/add-host/library present as sized sheets (a macOS sheet
takes its content's IDEAL size, and the GeometryReader-driven screens
collapsed to nothing), NSScreen-based mode lists, scroll indicators .never
(the "always show scroll bars" setting overrides .hidden), tray scrims so
scrolled rows dim under the pinned title/hints, extra title clearance, and a
PUNKTFUNK_FORCE_GAMEPAD_UI=1 dev hook — launcher/settings/add-host/keyboard/
library render-verified live on a real Mac + LAN hosts.
- GamepadMenuInput: X button support, and (re)start now snapshots held buttons
so a controller handoff press never fires twice (the B that closed the
keyboard no longer also cancels the screen underneath).
- Cleanups: one "Connection failed" alert in ContentView instead of one per
home screen; HostDiscovery.advertises/unsaved shared by both home screens.
- host: can_encode_444 stub for the non-Linux/Windows host build (the macOS
synthetic-source loopback used by the Swift tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rounds out codec negotiation across the last three clients — each advertises what it can decode,
builds its decoder from the resolved Welcome.codec, and exposes a "Video codec" preference picker.
**Windows** (Rust, mirrors Linux): `decodable_codecs()` + `ffmpeg_codec_id()`; the D3D11VA and
software FFmpeg decoders (and the mid-session D3D11VA→software demotion) open the negotiated codec
instead of hardcoding HEVC; settings gain a `codec` field + reactor ComboBox; `--codec` CLI flag.
**Apple** (Swift/C-ABI): AnnexB is now codec-aware — a `VideoCodec` enum drives H.264 vs HEVC NAL
parsing / parameter-set extraction (`CMVideoFormatDescriptionCreateFromH264ParameterSets` for H.264,
no VPS) and AVCC repacking; `PunktfunkConnection` advertises H264|HEVC via `punktfunk_connect_ex7`,
reads `resolvedCodec` (`punktfunk_connection_codec`), and threads `videoCodec` into the stage-1/2
pipelines + `VideoDecoder`; SettingsView "Video codec" Picker (auto/HEVC/H.264). AV1 is left out
(hosts don't emit it on the native path, and it's not an AnnexB codec). Test call sites updated.
**Android** (Kotlin + Rust JNI): the JNI `nativeConnect` gains `preferredCodec`; the native decode
loop picks the AMediaCodec MIME (`video/hevc`|`video/avc`) from `connector.codec` and advertises
H264|HEVC; Settings `codec` field + Compose dropdown.
Core/host/probe/Linux clippy + tests green (unchanged from 2a). Windows/Apple/Android compile on
their platform CI (this Linux box can't build them — Windows toolchain / Xcode / the Android NDK's
opus-cmake toolchain). All follow the Linux client's validated pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stream reliability
- Default to the stage-2 presenter (VTDecompressionSession + CAMetalLayer): it detects
and recovers a wedged decoder, where stage-1's AVSampleBufferDisplayLayer freezes hard
on a lost HEVC reference frame with no app-side recovery (confirmed Apple limitation).
Stage 1 is now a DEBUG-only presenter toggle, plus the automatic no-Metal fallback.
- Stage-2 pixel-perfect: render the drawable at the decoded size (shader stays 1:1 =
identity) and let the layer's contentsGravity scale via the system compositor — the
same path stage-1's videoGravity used — instead of scaling in-shader.
- Loss recovery in both pumps is now a persistent awaitingIDR want, retried until an IDR
actually lands, so a keyframe request swallowed by the throttle can't strand a frozen
frame; 100 ms keyframe throttle to match the Android path.
- Fix "Publishing changes from within view updates": defer the HostStore writes out of
the .onChange(of: model.phase) callback.
- Move AVAudioSession setActive/setCategory off the main thread (async on a shared serial
queue) to stop the UI-stall warning.
Controllers
- Rumble: capped-exponential backoff when the gamecontrollerd.haptics XPC breaks (-4811)
so a transient server interruption self-heals instead of cascading; playsHapticsOnly so
a controller engine doesn't join the always-active streaming audio session.
- Host cards: iPad pointer "magnet" hover effect; iPhone press scale + light haptic.
UI / design
- Ship Geist (SIL OFL 1.1) as the app font (bundled OTFs + registration), with the
license surfaced in Acknowledgements.
- Restructure iOS/iPadOS Settings into a category NavigationSplitView; resolution wheel
with custom-resolution entry; 10-bit HDR toggle in Display.
- Industrial host-card redesign (left-aligned, bold, brand monogram tiles).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GameController's CHHapticEngine never reaches the DualSense's motors on macOS — its
adaptive triggers and lightbar work, but rumble stays silent (a documented platform
gap). Drive the motors directly via the DualSense HID output report instead, the way
SDL and the Linux hid-playstation driver do — the same report that already rumbles
the pad on a Linux host. Confirmed live on macOS.
- DualSenseHID (macOS): opens the Sony DualSense via IOHIDManager and writes the USB
(0x02, 48 bytes) and Bluetooth (0x31, 78 bytes + CRC32) output reports through
IOHIDDeviceSetReport. Allowed under the App Sandbox by the existing device.usb +
device.bluetooth entitlements; coexists with GameController (non-seized open).
Flags mirror the kernel driver (COMPATIBLE_VIBRATION | HAPTICS_SELECT +
COMPATIBLE_VIBRATION2); valid_flag1 = 0 so a rumble report leaves the
GameController-managed lightbar / triggers / player LEDs untouched.
- RumbleRenderer routes a DualSense to the HID backend and keeps CoreHaptics for
every other pad, fixing both live sessions and the test panel (shared renderer).
- CoreHaptics path reworked too: bake the target intensity + an explicit sharpness
into the continuous event (the dynamic-parameter scaling is silent on controller
engines) and tear down outside the inout access to fix a latent exclusivity hazard.
Adds a DEBUG-only Settings -> Controllers -> "Test Controller" panel (ControllerTestView
+ ControllerTester) that shows live input and fires rumble / adaptive triggers /
lightbar / player LEDs straight at the pad, with a readout of the active rumble backend
("DualSense HID - USB/Bluetooth"). Used to validate the fix.
Tests: DualSenseHIDTests pins the USB/BT report layout and the BT CRC32 (canonical
0xCBF43926 check vector). Debug + release build clean; gamepad suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two bodies of work in one commit (the rename moved files the fixes also touched).
Naming/structure cleanup (pre-launch):
- Host modules m3.rs->punktfunk1.rs, m0.rs->spike.rs; CLI m3-host->punktfunk1-host,
m0->spike; bare `punktfunk-host` now prints help. Types M3Options/M3Source->
Punktfunk1Options/Punktfunk1Source.
- Clients consolidated out of crates/ into clients/: punktfunk-client-rs->
clients/probe (crate punktfunk-probe), client-linux->clients/linux,
client-windows->clients/windows, punktfunk-android->clients/android/native
(crate punktfunk-client-android; kept [lib] name=punktfunk_android so the JNI
contract is unchanged). crates/ now holds only core + host.
- Milestone codes M0-M4 purged from code/CLI/CLAUDE.md/README/docs/docs-site,
kept only in docs/implementation-plan.md. docs/m2-plan.md->
docs/gamestream-host-plan.md. CI/gradle/flatpak paths updated.
Client loss-recovery (video froze and never recovered after a brief drop):
- Export punktfunk_connection_frames_dropped through the C ABI (the core already
tracked it for the client keyframe-recovery loop; it was never reachable from
the ABI clients). Regenerated punktfunk_core.h.
- Apple (StreamPump + Stage2Pipeline) and Android (decode.rs) now poll
frames_dropped and request a keyframe when it climbs -- the same loss-driven
recovery Linux/Windows already had. Under infinite GOP the decoder silently
conceals reference-missing frames, so the decode-error trigger rarely fires.
Apple rumble robustness (worked then went spotty -- DualSense + Xbox):
- Add CHHapticEngine stopped/reset handlers (rebuild on app background / audio
interruption / server reset) and drop the permanent `broken` latch on a
transient drive failure; latch only when the controller truly has no haptics.
- Surface swallowed SDL set_rumble errors on Linux/Windows + diagnostic logging.
Verified: cargo build/clippy/fmt --workspace, C-ABI harness, header drift.
Not runnable on this box (verify in CI): Gitea workflows, gradle/Android,
flatpak, Swift/decky.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan step 3 — the Apple client surfaces the host's game library, behind a feature
flag (`DefaultsKey.libraryEnabled`, default OFF). Browsing only; launching a chosen
title is step 4.
- PunktfunkKit `LibraryClient`: Codable GameEntry/Artwork/LaunchSpec mirroring
crates/punktfunk-host/src/library.rs, and an async fetch of GET /api/v1/library
with a bearer token. Typed LibraryError guides setup (the common case is "needs a
--mgmt-token"). `Artwork.posterCandidates` = portrait → header → hero.
- `LibraryView`: cross-platform poster grid (LazyVGrid, AsyncImage that walks the art
candidates past load failures to a text placeholder), a store badge, and an inline
Connection form (mgmt port + token) that surfaces when the API is unreachable / 401
/ no token set. Read-only.
- StoredHost gains `mgmtPort`/`mgmtToken` (the mgmt API is a distinct port from the
data plane and needs a token off-loopback). Both OPTIONAL — synthesized Decodable
ignores property defaults but treats a missing Optional as nil, so older saved
hosts decode unchanged (a defaulted non-optional would wipe the list). HostStore.setMgmt.
- Entry point: a flag-gated "Browse Library…" host-card context action → LibraryView
(sheet on macOS/iOS, pushed on tvOS), mirroring the pair/speed-test plumbing. Plus a
Settings "Experimental" toggle.
Can't compile Swift on the Linux dev box; CI (apple.yml: swift build + swift test on
the mac mini) verifies the macOS path. Added LibraryClientTests (decode + art order)
for `swift test`. iOS/tvOS-only branches mirror existing patterns. Live-verify on the
Mac pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opt-in (Settings -> Presenter; `punktfunk.presenter`, default stage-1). Stage-1's
AVSampleBufferDisplayLayer decodes AND presents internally with no per-frame
callback, so neither decode nor present can be stamped or hand-paced. Stage-2
takes explicit control:
- VideoDecoder: VTDecompressionSession, async output callback stamps
decode-completion, session rebuilt on every IDR / format change. Unit-tested
(testVideoDecoderAsyncCallbackDeliversPixels).
- MetalVideoPresenter: CAMetalLayer + CVMetalTextureCache + a runtime-compiled
BT.709 limited-range NV12->RGB shader, present at the next vsync. The
CVMetalTextures + pixel buffer are held until the GPU completes.
- Stage2Pipeline: pump thread -> decoder -> newest-ready 1-slot ring; the hosting
view's display link drains it once per vsync and stamps capture->present
(the display-link target time projected into CLOCK_REALTIME).
- LatencyMeter gains record(ptsNs:atNs:offsetNs:); the HUD shows a capture->present
(glass-to-glass, modulo host render->capture) line, skew-corrected via
clockOffsetNs. Measured live ~11 ms p50 vs ~2.2 ms capture->client.
- StreamView / StreamViewIOS host the CAMetalLayer as a sublayer + a CADisplayLink
(NSView.displayLink on macOS) when stage-2; input capture + HUD unchanged. The
session-active gates switch from `pump != nil` to `connection != nil` so capture
engages without a StreamPump.
Validated: builds macOS/iOS/tvOS; the decode half is unit-tested; the Metal
present is live-validated on glass (correct image + the capture->present number).
Colorspace is BT.709 SDR for now; 10-bit/HDR + a pacing policy are later.
Plan: docs-site/content/docs/apple-stage2-presenter.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Input handling, building on macOS/iOS/tvOS:
- macOS recapture after navigating out: engageCapture no longer latches
captured=true when the cursor grab is refused mid app-activation (which left
a free cursor that no later click could re-grab); cursorCapture.capture() now
reports success. + canBecomeKeyView.
- iOS/iPadOS recapture: restore the prior capture on didBecomeActive (nothing
re-grabbed mouse/keyboard on return before).
- iPad indirect pointer (no lock) is forwarded as an absolute MOUSE (move +
buttons + scroll via hover / UITouch.indirectPointer), not as touch, with the
local cursor visible; GCMouse owns the locked regime, gated so the two never
double-send. Adds the MouseMoveAbs wire helper.
- Trackpad scroll on iOS (was entirely missing): GCMouse scroll dpad when
locked + a scroll-only UIPanGestureRecognizer otherwise.
- tvOS: no focusable control during play (a focusable Disconnect button ate the
controller's A in the focus engine); Siri Remote Menu disconnects.
- Don't leak touch to the host under the TOFU trust prompt (gate on
captureEnabled).
LAN discovery: HostDiscovery (NWBrowser over _punktfunk._udp, the host's
crate::discovery advert) resolves each service to IP:port and parses the TXT
(fp advisory, pair, id); an "On this network" section in the grid (tap to save
+ connect, or pair if required). iOS/tvOS get NSBonjourServices via a merged
Config/Info.plist. Integration-tested end to end against a fake NWListener advert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple speed test asked for only 400 Mbps, capping the measured throughput
there and hiding the link's real headroom. Request the host's full
MAX_PROBE_KBPS (3 Gbps) instead, and raise the recommended-bitrate clamp from
500 Mbps to the host's 2 Gbps session ceiling so a fast measurement yields a
usable recommendation.
Also fix the stale caps left when the host clamps were raised (0cd02e7): the
resolved-bitrate range and the probe doc comments (abi.rs, client.rs,
regenerated header), plus the section 9 roadmap copy, now read 3 Gbps probe /
2 Gbps session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple client now consumes the connector's clock offset. PunktfunkConnection
reads punktfunk_connection_clock_offset_ns into clockOffsetNs at connect; a new
LatencyMeter (PunktfunkKit, NSLock + percentiles, mirrors FrameMeter) records each
AU's capture->client-receipt latency = now(CLOCK_REALTIME) + offset - pts_ns, and
SessionModel drains p50/p95 into the macOS HUD ("capture->client N/N ms p50/p95",
"(same-host)" when the host didn't answer the skew handshake). Wired at the
existing onFrame hook in ContentView — additive, no change to the decode/present
path. Unit test for the meter (percentiles, skew flag, absurd-value guard).
This is the first cross-machine latency the real Apple client reports. SCOPE:
stage-1 AVSampleBufferDisplayLayer decodes+presents compressed samples internally
with no per-frame callback, so this excludes decode+present; true decode->present
needs the stage-2 presenter (VTDecompressionSession + CAMetalLayer). Rebuild
PunktfunkCore.xcframework (for the new C getter) before swift build/test on a Mac.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple client grows full gamepad support and punktfunk/1 learns to negotiate
the virtual pad type:
- Protocol: Hello carries a GamepadPref byte (offset 21, the same trailing-byte
back-compat pattern as the compositor; echoed resolved in Welcome at 54).
Host precedence: explicit client choice > PUNKTFUNK_GAMEPAD env > Xbox 360,
DualSense (UHID) only where available. ABI: punktfunk_connect_ex2 +
punktfunk_connection_gamepad (connect_ex delegates; ABI_VERSION stays 2 — the
trailing byte IS the compat mechanism). punktfunk-client-rs gets --gamepad.
- Swift client: GamepadManager (app-lifetime discovery + selection — Settings
lists every controller with capabilities/battery/"In use"; exactly ONE pad
forwards as pad 0, auto = most recently connected, or pinned), GamepadCapture
(snapshot-diff button/axis events, DualSense touchpad + ~250 Hz motion on the
rich-input plane, held state released on switch/deactivate/stop),
GamepadFeedback (rumble → CoreHaptics per-handle engines; lightbar →
GCDeviceLight; player LEDs → playerIndex; adaptive-trigger blocks → the
table-driven DualSenseTriggerEffect parser → GCDualSenseAdaptiveTrigger,
exact for the 10-zone positional modes). The pad type auto-resolves from the
physical controller at connect time, user-overridable in Settings.
- Host DualSense fixes surfaced by adversarial review against hid-playstation /
SDL / Nielk1 ground truth: input-report sensor/touch offsets were off by one
(the kernel read garbage motion + phantom touches), the L2/R2 trigger blocks
were swapped (the report is right-trigger-first), feedback now gates on the
report's valid-flags (a plain rumble write no longer blanks lightbar/
triggers), and the touchpad rescale clamps to the advertised ABS_MT extents.
- Tests: Hello/Welcome trailing-byte back-compat, pick_gamepad precedence,
byte-exact input-report layout, valid-flag gating, per-mode trigger-parser
table (incl. packed 3-bit zones), wire conversions, and a scripted loopback
feedback burst (PUNKTFUNK_TEST_FEEDBACK=1) asserted through the xcframework
on the rumble + HID-output planes.
Validated: cargo test/clippy/fmt green on macOS + Linux (61 host tests), swift
build/test green, test-loopback.sh green, tvOS/iOS targets compile. DualSense
motion sign/scale is derived from the calibration blob, not yet live-verified
(constants isolated in GamepadWire).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both directions of the audio plane, on CoreAudio's built-in Opus codec
(kAudioFormatOpus — no bundled libopus; OpusCodec.swift, round trip unit-tested):
- Playback: a drain thread pulls nextAudio() packets, decodes, and writes a priming
jitter ring feeding an AVAudioSourceNode (~20 ms prefill, adaptive to the device's
render quantum so large-buffer devices don't oscillate prime/dropout; a high-water
clamp sheds stall backlog so one network hiccup can't permanently lag audio behind
video; underrun re-primes — one dip, not sustained crackle).
- Mic: a second engine taps the input device, resamples to 48 kHz stereo, Opus-encodes
20 ms chunks and sendMic()s them into the host's virtual PipeWire source. Permission
via AVCaptureDevice (NSMicrophoneUsageDescription added to the Xcode target).
- Settings: Speaker + Microphone pickers (CoreAudio HAL enumeration, persisted by
device UID — "System default" leaves the engine unpinned so it follows macOS device
changes) and a "Send microphone" toggle (default on). Applies from the next session.
- Audio starts with streaming, never during the trust prompt (no host sound — and no
mic uplink — before the user trusted the host); teardown stops audio before close().
Adversarial-review fixes baked in: stop() and the dangling mic-permission callback
share one lock+flag protocol (no hot mic with no owner), the connect-success handler
bails when the attempt was abandoned mid-handshake (no session/mic for a dead window),
SessionAudio gets a deinit backstop (a dropped instance can't pin the connection via
its drain thread), and the render scratch buffer is block-owned (was leaked per
session).
Verified live against the box: remote test decodes 100 host Opus packets to PCM and
the host opens its virtual mic on the first uplinked frame ("punktfunk/1 virtual mic
ready"); on-glass session runs with both engines up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopts the new ABI surface (still v2, additive):
- PunktfunkConnection.sendMic(_:seq:ptsNs:) — Opus mic frames (48 kHz) to the host's
virtual PipeWire source; enqueue-only, empty data = DTX silence. Wiring the actual
Mac microphone (AVAudioEngine input → Opus) into the app is the follow-up, alongside
audio playback (README note 5).
- PunktfunkInputEvent.touchDown/touchMove/touchUp — absolute pixels + surface size in
flags, host injects via libei ei_touchscreen. Built for the iOS variant; nothing on
macOS emits them yet.
- Loopback round trip now also sends touch events and mic frames (incl. a DTX frame)
through the wrapper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopts punktfunk_connect_ex from the compositor-selection batch: a Compositor enum on
PunktfunkConnection (auto/kwin/wlroots/mutter/gamescope, with the host's name aliases
for env parsing), a "Host compositor" picker in Settings (default Automatic — a
concrete choice is honored only if that backend is available host-side), and
PUNKTFUNK_COMPOSITOR / PUNKTFUNK_REMOTE_COMPOSITOR pass-throughs for the autoconnect
dev hook and the remote first-light test. The wire change is backward-compatible
(optional trailing byte), so no behavior changes at the default.
Validated live against the box: host with no compositor env (auto-detect = KWin)
logged "honoring client compositor request compositor=gamescope" and streamed 60/60
decoded frames from the spawned gamescope.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pairing/renegotiation batch bumped the punktfunk/1 ABI to v2 and the host now
hard-rejects v1 Hellos (m3.rs), so streaming from the Mac was dead until the bundled
PunktfunkCore.xcframework is rebuilt — it is gitignored, so that is a per-checkout step:
bash scripts/build-xcframework.sh. The Swift wrapper itself was already adapted upstream;
this lands the app on top of it.
- ClientIdentityStore: persistent client identity in the login Keychain, presented on
every connect so paired hosts recognize this Mac. Keychain access failure throws
instead of regenerating (a fresh identity would silently un-pair this Mac from every
--require-pairing host); a lost first-run race resolves toward the stored identity;
pairing uses the strict loadForPairing() so a memory-only identity can't strand a
ceremony.
- PairSheet: the SPAKE2 PIN ceremony, reachable from a host card's context menu and from
the trust prompt's "Pair with PIN instead…" (which drops the live session first — the
host's accept loop is sequential). Success pins the verified fingerprint and connects;
an in-flight ceremony self-discards when the sheet is dismissed, so a late success
can't pin + auto-connect behind the user's back. Wrong PIN and Keychain failures get
distinct, actionable error text.
- Tests: identity unit tests; the full pairing ceremony + --require-pairing gate on
loopback (test-loopback.sh arms a second host, parses its PIN from the log, and gives
both hosts throwaway config homes — no more writes to the real ~/.config/punktfunk);
remote pairing + pinned stream over the LAN (PUNKTFUNK_REMOTE_PIN, _PORT).
Validated live against the box: SPAKE2 ceremony with the host's arming PIN → verified
fingerprint → pinned + identified 720p60 session (host persisted the client identity);
first light 60/60 AUs decoded to pixels; vkcube on glass through the app.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full project rename, decided 2026-06-10:
- Crates/binaries: punktfunk-core / punktfunk-host / punktfunk-client-rs.
- C ABI: punktfunk_* symbols, Punktfunk* types, include/punktfunk_core.h,
PUNKTFUNK_FEATURE_QUIC guard (header regenerated; cbindgen renames updated, incl.
PUNKTFUNK_BTN_*/PUNKTFUNK_AXIS_* wire constants).
- Protocol: punktfunk/1 — control-plane magic LMN1 → PKF1, nonce salt lmn1 → pkf1.
WIRE BREAK: clients must be rebuilt from this revision.
- Env knobs: PUNKTFUNK_VIDEO_SOURCE / PUNKTFUNK_COMPOSITOR / PUNKTFUNK_ZEROCOPY / ….
- Host config dir: ~/.config/punktfunk (the box's dir was migrated in place — the
persistent identity is unchanged, pinned fingerprints stay valid).
- Swift package: PunktfunkKit + PunktfunkCore.xcframework + PunktfunkConnection
(Sources/PunktfunkClient app + tests renamed with it); build-xcframework.sh updated.
- scripts/: 60-punktfunk.rules, punktfunk-host.service; OpenAPI doc regenerated.
Also: scripts/headless/run-headless-kde.sh — full headless Plasma bringup. Root cause of
"desktop but no apps/settings" over the stream: plasmashell launched without
XDG_MENU_PREFIX=plasma-, so the launcher resolved a nonexistent applications.menu and
rendered an empty menu. The script sets the complete KDE session env (menu prefix,
KDE_FULL_SESSION, session version) and rebuilds ksycoca before starting plasmashell.
Gate: 97/97 tests, clippy -D warnings (both feature sets), fmt, C-ABI harness PASS,
zero lumen references left outside .git.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clients/apple scaffold is now a working macOS client, validated live against this
repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC +
AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60,
mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector
(~3.7k events injected in one session).
LumenKit:
- LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as
integers while the enum constants import as a distinct Swift type — bridge by rawValue);
close() is now safe from any thread (a close flag + pumpLock held across the blocking
poll enforce the C contract "never close with a next_au in flight"; flag prevents
lock-starvation by back-to-back polls).
- StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate
on the next in-band parameter sets when the layer fails, no stale enqueue after restart.
- InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away),
pressed-state tracking with release-all on focus loss and stop() (nothing sticks down
host-side), global-singleton ownership guard (GC has one handler slot per process),
X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs.
- LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD,
LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs.
- Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded
HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels);
test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of
c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN).
Host/build fixes that fell out:
- The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching
are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule.
- Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the
ei/wl axes, but GameStream's horizontal convention is positive = right
(moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also
un-inverts real Moonlight clients.
- AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's
policy, instead of leaking them into the preceding NAL.
- build-xcframework.sh: deployment targets pinned to the package floor + an otool guard —
cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship
too-new minos objects.
Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified):
14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged
here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
