Key insight (from the user): a fresh RECONNECT shows the secure desktop but the
live transition does not — so the difference is what a fresh session does that
the live switch skipped. A reconnect runs build() = REMOVE + fresh ADD of the
SudoVDA monitor + re-isolate + a fresh capturer; the live transition instead
reused the session-start output (created while on the NORMAL desktop), which goes
born-lost (ACCESS_LOST storm → black) on the secure desktop.
Fix: virtual_stream_relay now calls build() on EVERY source switch (both WGC→DDA
and DDA→WGC), then opens DDA on the new target for secure / uses the fresh helper
for normal. This makes each transition equivalent to the reconnect that works —
fixing both the WGC→DDA cutover (secure desktop now in the clean output state DDA
can duplicate) and the DDA→WGC cutover (a fresh helper's first frame is its
opening IDR, so await_idr clears immediately instead of waiting on a wedged
helper). Costs a ~1-2s rebuild per transition, acceptable for UAC/lock events.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The secure (Winlogon: UAC/lock/login) desktop presents via fullscreen
independent-flip/MPO — it scans out bypassing DWM composition, so DXGI Desktop
Duplication returns born-lost DXGI_ERROR_ACCESS_LOST (the client sees black; the
UAC only "flashes" during the brief composed transition). Confirmed live: stable
4090 LUID across the storm (NOT reparenting) on an FP16 HDR output, recovering
only when the screen changes.
Fix (non-input, no system-wide registry change): capture/composed_flip.rs keeps a
tiny click-through near-invisible TOPMOST LAYERED window alive on the current
input desktop. Any visible window on the output disqualifies independent-flip →
DWM composites → DDA can capture. A dedicated thread follows the input desktop
(Default↔Winlogon) and recreates the window there on each switch (a window is
bound to its desktop), re-asserting topmost + pumping messages every 200ms.
Started for the two-process stream's lifetime; gated by PUNKTFUNK_FORCE_COMPOSED
(default on, =0 to disable). Needs GENERIC_ALL on OpenInputDesktop for
DESKTOP_CREATEWINDOW (0x80070005 otherwise). Validated: overlay creates on the
Default desktop; live lock test pending.
Also includes SET_RENDER_ADAPTER (sudovda.rs, Apollo item #16): pins the IDD
render GPU to the NVENC GPU before ADD — issued + accepted live, though the
secure-desktop storm was proven to be independent-flip (stable LUID), not
reparenting, so it's correctness/hygiene here rather than this bug's fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A WGC-helper exit (crash, or a console disconnect killing its session) used to
end the stream. Now virtual_stream_relay rebuilds the output + helper and resumes
on the new helper's opening IDR. Rebuild — not respawn-on-the-old-target —
because an abruptly-killed helper leaves the SudoVDA's DXGI output briefly
unresolvable ("no DXGI output for target N yet"), and a console reconnect needs
a fresh output in the new session; `build` (the same path reconfigure uses)
recreates both. Bounded: 500ms backoff per attempt, give up after
MAX_HELPER_FAILS (20) consecutive failures; the counter resets on the first
relayed frame.
Live-validated on the RTX 4090 (host as SYSTEM): force-killed the helper PID
mid-stream → exactly one "WGC helper exited — rebuilt output + helper fails=1" →
the stream recovered and client-rs decoded 645 HEVC Main-10 frames continuously
across the kill (an earlier respawn-on-stale-target attempt storm-failed with
"no DXGI output", which the rebuild fixes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PUNKTFUNK_SECURE_TEST_PERIOD_MS=N drives a square-wave secure/normal toggle in
virtual_stream_relay (instead of the real DesktopWatcher), to exercise the
mid-session helper↔DDA mux without a live UAC/lock. Gated behind the env var,
in the style of PUNKTFUNK_VIDEO_DROP / PUNKTFUNK_FEC_PCT.
Live-validated on the RTX 4090 (host as SYSTEM): with a 4s toggle the mux
switched secure(DDA)↔normal(WGC relay) cleanly 5× in one session and the client
decoded 308 HEVC Main-10 frames continuously across every switch — the
wait-for-IDR latch held with no decode break. The real Winlogon DDA capture is
pre-proven by the single-process secure path (a08df8c); the toggle exercises the
new surface (the mux). Doc updated with the validation + the SYSTEM-mode audio
caveat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`virtual_stream_relay` now muxes the AU source by input desktop. A DesktopWatcher
(SYSTEM-only Winlogon-name poll) drives it: the user-session WGC helper relay
feeds the normal (Default) desktop; the host's OWN DDA capturer+encoder — opened
lazily on the first secure transition, on the same SudoVDA target with a no-op
keepalive (the host still holds the real isolation owner) — captures the secure
(Winlogon: UAC/lock/login) desktop that WGC can't see. Every switch latches
"wait for IDR" and forces the now-active source to emit a keyframe (the two
encoders keep independent infinite-GOP state, so the client must resume on an
IDR); returning to the helper also drains its stale buffered AUs first.
Reconfigure drops the stale-target DDA; keyframe requests route to the live
source. Send path (FEC/seal/paced-send) unchanged.
Also: wgc_relay gains try_recv (drain on switch-back); open_dda takes dims as
args (avoids a closure borrow of the reassigned cur_mode); the forward! macro
returns bool with `break 'outer` at the call site (no in-macro label hygiene).
cfg-gated windows-only. Live validation (UAC switch over a session) pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SYSTEM host now sources the normal-desktop video from a user-session WGC
helper instead of capturing in-process (WGC won't activate as SYSTEM). New
`capture/wgc_relay.rs`: `HelperRelay::spawn` launches `m3-host wgc-helper` in the
interactive user session via CreateProcessAsUserW (WTSQueryUserToken →
DuplicateTokenEx(TokenPrimary) → lpDesktop="winsta0\\default", CREATE_NO_WINDOW)
with three anonymous pipes — stdout (framed Annex-B AUs → parsed back to
RelayAu), stdin (control: force-keyframe), stderr (helper logs → host tracing).
The host holds the SudoVDA keepalive (sole isolation/topology owner); the helper
captures by GDI name only.
m3.rs: `virtual_stream` dispatches to the new `virtual_stream_relay` when
`should_use_helper()` (running as SYSTEM, or PUNKTFUNK_FORCE_HELPER; disable with
PUNKTFUNK_NO_HELPER). The relay loop feeds the existing send thread — same
FEC/seal/paced-send path. Reconfigure rebuilds the output + re-spawns the helper;
keyframe requests forward over the control pipe; helper pts_ns (same-machine
monotonic clock) is used directly as capture_ns. Disconnect ends the stream
(step 6 adds the relaunch watchdog).
wgc_helper.rs: reads the stdin control byte to request an IDR; --bit-depth flag
threaded through so SDR 10-bit (Main10) negotiation reaches the helper's encoder.
cfg-gated windows-only; Linux/macOS build unaffected. Step 5 (DesktopWatcher mux
to host DDA on the Winlogon secure desktop) is next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds true HDR (BT.2020 PQ) and 10-bit (HEVC Main10) streaming, negotiated so an
8-bit/SDR client is never sent a stream it can't decode, plus a robust fix for the
capture losing the stream across a secure-desktop transition.
Protocol (punktfunk-core/quic.rs):
- Hello gains `video_caps` (VIDEO_CAP_10BIT / VIDEO_CAP_HDR), Welcome gains `bit_depth`,
both as optional trailing bytes (back-compat). client-rs advertises 10-bit via
PUNKTFUNK_CLIENT_10BIT; the connector advertises 0 for now (in-band detection drives
the native clients). Regenerated punktfunk_core.h.
Windows host:
- 10-bit Main10: host enables it only when the client advertised VIDEO_CAP_10BIT AND
PUNKTFUNK_10BIT is set; threaded through open_video → NVENC (profile Main10,
pixelBitDepthMinus8).
- HDR: when the captured desktop is scRGB FP16 (R16G16B16A16_FLOAT, HDR on), copy it to
an FP16 surface, composite the cursor there, convert scRGB → BT.2020 PQ 10-bit
(R10G10B10A2) via a shader, and encode HEVC Main10 with the BT.2020/PQ colour VUI
(ABGR10 input). Fixes the freeze + cursor-trail that came from feeding FP16 into the
BGRA path. Reacts dynamically to the HDR toggle.
- Capture recovery: rebuild is now a single NON-BLOCKING attempt, throttled to ~4×/s,
repeating the last good frame between attempts (format-tagged last_present). During a
secure-desktop dwell SudoVDA's output is gone; the old blocking 12 s retry starved the
send loop for seconds so the client timed out and disconnected — now the session stays
fed (frozen) until the desktop returns. Also seeds a black frame on recovery.
Apple client (PunktfunkKit):
- Detects HDR in-band from the stream VUI (PQ transfer function), decodes to 10-bit P010,
and presents via an rgba16Float + BT.2020 PQ CAMetalLayer with EDR; SDR path unchanged.
Switches automatically on a mid-session HDR toggle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the host→client rumble path on Windows, the analogue of the Linux
uinput EV_FF read loop: a game's force-feedback on the virtual Xbox 360
pad is delivered by ViGEm's notification API (`request_notification` →
`spawn_thread`, gated by the crate's `unstable_xtarget_notification`
feature). A per-pad background thread stores the latest motor levels;
`pump_rumble` relays changes to the client on the universal 0xCA plane
(motors scaled 0..255 → 0..65535). Dropping the target aborts the
notification, so the thread exits with the session. Live verification
still needs a physical pad.
Also fix the Windows backends' clippy debt — these modules are cfg-
excluded from Linux CI, so `clippy -D warnings` never saw them, and the
VM's rustc 1.96 clippy is stricter on shared code than the CI image:
- dxgi: manual checked division → checked_div().map_or
- sendinput: `x = x | y` → `x |= y`
- sudovda: `.then(|| ptr)` → `.then_some(ptr)`
- m3 pick_compositor: drop the needless early return (match form)
- m3 resolve_compositor: Windows arm is a tail expr, not `return`
All Windows backends now build + clippy clean (default and --features
nvenc); Linux unaffected (fmt/clippy/check green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"Controller disconnected every few seconds" (Forza Horizon, held steady): the
virtual UHID DualSense emitted HID report 0x01 ONLY on state change, but a real
DualSense streams it continuously (~250 Hz). When the player holds the
wheel/throttle steady the client sends no wire events, so the host wrote nothing
and /dev/uhid went silent for seconds — the kernel hid-playstation driver / Proton
/ SDL treat that as an unplugged controller. (The uinput X-Box pad is immune:
evdev holds last-known state with no periodic-report requirement.)
Add DualSenseManager::heartbeat(max_gap): re-emit each live pad's CURRENT report
when it's been silent for max_gap (idempotent — a stale-but-correct frame, never a
phantom input; write_state bumps seq+timestamp). write() resets the per-pad timer,
so an actively-used pad emits no extra reports — the heartbeat only fills genuine
silence. PadBackend::heartbeat() drives it at an 8 ms gap (~125 Hz) for DualSense
(no-op for X-Box), called every input-thread tick (the loop already runs ≤4 ms).
GET_REPORT feature replies + the pad lifecycle were ruled out by the investigation
(pad is created once, never torn down mid-session). Compiles, clippy/fmt clean, 78
host tests pass. Verify on the box: held-idle DualSense stays present in evtest /
no SDL CONTROLLERDEVICEREMOVED; Forza no longer toasts "controller disconnected".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `m3` audio_thread (desktop capture → Opus 48 kHz stereo 5 ms CBR →
AUDIO_MAGIC datagrams) now runs on Windows, fed by the WASAPI loopback
capturer. The `opus` crate vendors libopus via `audiopus_sys` + cmake
(no system lib / vcpkg), so it builds on MSVC — moved into a
`cfg(any(linux, windows))` deps table and widened the audio_thread cfg
to match (the stub now only covers other targets, e.g. macOS).
Build note: CMake 4 rejects libopus's old `cmake_minimum_required`;
set `CMAKE_POLICY_VERSION_MINIMUM=3.5` when building the host on Windows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two parked follow-ups from the session-aware host work:
#3 — KWin/Mutter virtual output not set primary. The auto-detected desktop path
*is* "stream this desktop", but the per-session virtual output wasn't promoted to
primary, so KDE/GNOME panels + windows stayed on an unstreamed real output and the
streamed screen showed only wallpaper. apply_session_env now defaults
PUNKTFUNK_KWIN_VIRTUAL_PRIMARY / PUNKTFUNK_MUTTER_VIRTUAL_PRIMARY on for the
auto path (explicit config still wins), so the streamed output becomes the sole
desktop.
#2 — input flaky after a mid-stream Gaming->Desktop switch. The xdg portal
(D-Bus-activated) and the systemd --user env still pointed at the old session, so
the host's RemoteDesktop portal opened against a half-stale env: it accepted
events but they didn't reach the compositor until a reconnect. New
vdisplay::settle_desktop_portal() pushes the live session env into the
systemd/D-Bus activation environment and (for KWin) restarts the portal so it
re-reads it, mirroring a fresh desktop login (and the existing wlroots portal
restart). Called from the mid-stream switch rebuild slot before the injector
reopens. GNOME uses Mutter's direct EIS, so it only gets the env push.
Compiles, clippy/fmt clean, 78 host tests pass. Live validation on the Bazzite
box next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The punktfunk/1 control plane already compiled on Windows; these wire the last gaps so the host actually runs: config_dir falls back to %APPDATA% (HOME\.config when set), paired_path uses it, hostname from COMPUTERNAME, and resolve_compositor short-circuits the Linux session-detection on Windows (SudoVDA is the single backend; vdisplay::open ignores the compositor arg). Validated live on the VM: m3-host creates its identity, binds the QUIC endpoint (fingerprint logged), advertises mDNS (_punktfunk._udp, host from COMPUTERNAME), and accepts sessions. GPU-less validations green: m0 synthetic->openh264->core FEC loopback (120/120, 0 mismatches) and the m3 c_abi_connection_roundtrip control-plane test. Full session capture (SudoVDA->DXGI) + NVENC remain GPU-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes from live Bazzite testing of the managed-Gaming + mid-stream work:
1. Input now FOLLOWS the active session. The host-lifetime injector was pinned to
the first backend it opened and only reopened on an inject FAILURE — but with
Feature A keeping the managed gamescope warm, its EIS socket stays alive, so a
switch to the KDE desktop + reconnect kept injecting into the idle gamescope
(input silently dead on KDE). injector_service_thread now compares the
resolved input backend (default_backend() ← PUNKTFUNK_INPUT_BACKEND, set per
connect by apply_input_env, and on a mid-stream switch) each event and reopens
when it changes. Fixes input on a Gaming->Desktop reconnect AND Feature B's
mid-stream input re-route, with no plumbing.
2. Debounced TV-restore no longer yanks you back to gaming. do_restore_tv_session
now checks detect_active_session(): if a desktop session is active (the user
switched), it tears down the idle managed gamescope but does NOT restart the
gaming autologin. Observed live: the restore fired and restarted
gamescope-session-plus@ogui-steam while the client was already on the KDE
desktop.
Also: document PUNKTFUNK_SESSION_WATCH (Feature B opt-in) in the Bazzite host.env
and correct the managed-default description. Compiles, clippy/fmt clean, 78 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Feature B: while streaming, follow a Gaming<->Desktop switch on the box without
a reconnect. A ~1s watcher thread (session_watcher_loop) self-baselines on the
live ActiveKind and, when it changes and stays changed for a 3s debounce (the
old/new compositors coexist briefly during a switch), sends a SessionSwitch to
the encode loop. The loop's new rebuild slot — taking precedence over a queued
mode change — retargets the process env (apply_session_env/apply_input_env) and
rebuilds the WHOLE backend in place at the SAME client mode (vdisplay::open +
build_pipeline_with_retry), reusing the proven mode-switch rebuild path: the
Session + send thread (QUIC control + UDP data plane + side planes) stay up, the
client sees a brief freeze then an IDR. Old pipeline kept on a rebuild failure
(transient vs permanent classified via is_permanent_build_error). Input
re-routes via the host-lifetime injector's lazy reopen against the new
PUNKTFUNK_INPUT_BACKEND.
Opt-in via PUNKTFUNK_SESSION_WATCH (off by default; never under an explicit
PUNKTFUNK_COMPOSITOR pin), so it lands inert and is promoted to default only
after live validation on a real Bazzite Gaming<->KDE flip. The watcher snapshots
the SessionEnv so only the encode thread writes process env.
Compiles, clippy/fmt clean, 78 host tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Feature A: in Gaming Mode, default to a host-managed gamescope at the CLIENT's
mode (tear the TV's autologin down on connect) instead of attaching to the
running TV session — so the client receives ITS resolution (capture == encode ==
client mode, fixing the InitializeEncoder size mismatch the attach path hit),
not the TV's 4K.
Reliability is the debounce: restore_managed_session() now SCHEDULES the TV
restore RESTORE_DEBOUNCE (5s) after the last disconnect via a host-lifetime
worker, instead of restoring immediately per-disconnect. A reconnect inside the
window cancels the pending restore and reuses the still-warm managed session
(create_managed_session clears PENDING_RESTORE at the top) — so a quick reconnect
(e.g. a controller hiccup) never triggers a gamescope stop/relaunch, which is the
per-connect churn that leaked NVIDIA GPU context on F44 (the black-screen
reconnect).
- vdisplay/gamescope.rs: PENDING_RESTORE + RESTORE_DEBOUNCE; schedule_restore_tv_session
(debounced), do_restore_tv_session (the actual restore, worker-driven),
start_restore_worker (100ms tick, RAII keepalive handle). create_managed_session
cancels the pending restore + reuse path unchanged.
- vdisplay.rs: apply_input_env flips gamescope to managed-DEFAULT; PUNKTFUNK_GAMESCOPE_ATTACH
(or an explicit _NODE) opts back to attach for couch-on-TV; _MANAGED forces managed.
restore_managed_session schedules; new start_restore_worker wrapper.
- m3.rs serve(): hold the restore worker for the host lifetime.
- bazzite host.env: document managed-default + the ATTACH opt-out.
Compiles, clippy-clean, 78 host tests pass. F44 single stop/start leak to be
verified live on the box.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bazzite/SteamOS boxes flip between Steam Gaming Mode (gamescope) and a
KDE/GNOME desktop. The host statically read PUNKTFUNK_COMPOSITOR /
XDG_CURRENT_DESKTOP once, so switching to Desktop Mode failed the stream, and
the gamescope managed-session path stopped+relaunched the autologin per connect
— leaking GPU context on F44 (reconnect → black screen).
Replace the static read with a runtime probe of the live session and route each
connect to the right backend, churn-free:
- vdisplay::detect_active_session() probes /proc for the running compositor of
our uid (gamescope|kwin_wayland|gnome-shell|sway, desktop outranks a leftover
gamescope) + scans the runtime dir for the live wayland-* socket. Returns an
ActiveKind + the SessionEnv (WAYLAND_DISPLAY/XDG_RUNTIME_DIR/DBUS/
XDG_CURRENT_DESKTOP) that targets it.
- apply_session_env() writes that into the process env per connect (host serves
one session at a time), so every backend (capture + input) opens against the
live session; apply_input_env() points input at the matching backend and
selects gamescope ATTACH (no managed restart) unless PUNKTFUNK_GAMESCOPE_MANAGED.
- resolve_compositor() (native path) auto-detects + applies; explicit
PUNKTFUNK_COMPOSITOR still wins (legacy/CI/forcing). detect() is now
active-aware for the GameStream/mgmt callers too.
- Bazzite host.env drops the static gamescope force; documents auto-detection
+ the optional overrides.
Result: Desktop Mode → KWin/Mutter virtual output at the client's mode
(churn-free, the reliable path); Gaming Mode → attach to the running gamescope
(no SIGSEGV/GPU leak on reconnect). Compiles + clippy-clean; 78 host tests pass.
Live validation on the Bazzite box pending (box offline).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a Bazzite host that autologins into gaming mode on a physical display (the F44
default: gamescope-session-plus@ogui-steam on the TV), Steam — single-instance — is
held by that session, which renders to the TV's native mode. The host-managed session
then can't start its own Steam, so it captured the TV's 4K output instead of the
client's mode (stretched). On F43 the box wasn't in gaming mode, so the host's Steam
was the only one.
Fix: on connect, the host-managed gamescope path stops any running autologin
`gamescope-session-plus@*` unit (frees Steam) before launching its own session at the
client's mode; on client disconnect (`restore_tv_session`, called from serve_session
teardown) it stops our session and restarts the autologin one, so the TV returns to
gaming mode by default when no one is streaming. Stopping the `--user` unit sticks
(Relogin only fires on the full logind session ending — verified live), so no sddm
config change is needed. Cost: a Steam cold-start per connect, given single-instance.
No-op on non-Bazzite / headless boxes (nothing to stop → nothing to restore).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two DualSense (UHID) fixes surfaced live on the Bazzite host:
- Battery: serialize_state never set the input report's status byte (struct off 52 →
r[53]), so hid-playstation read battery capacity 0 and SteamOS warned "low battery"
even on a fully-charged pad. Set it to 0x0A (discharging, low nibble 0xA → 100 %) —
a virtual pad has no real cell. (Forwarding the client pad's real charge is a later
feature.) Regression assert added to the layout test.
- Rumble diagnostic: log the silent→active transition when forwarding a buzz on the
0xCA plane, so a live test can tell "host never receives rumble from the game"
(Steam Input / parse) apart from "client doesn't render it". Once per buzz, no spam.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan step 4 (plumbing + host behavior). A client can ask the host to launch a
library title on connect; the host resolves it against ITS OWN library and runs it
in the session — the client sends only the store-qualified id, never a command, so a
remote peer can't inject one.
- Protocol (quic.rs): `Hello.launch: Option<String>` (the GameEntry id). Appended
after `name`; when launch is present but name absent, a zero-length name placeholder
keeps the offset deterministic — so a Hello with neither field stays byte-identical
to the bitrate-era 26-byte form (test-asserted). Old peers ignore it; new hosts
decode None from old clients. Round-trip + back-compat + truncation tests.
- Host: `library::launch_command(id)` resolves id → command via the host's own library —
`steam_appid` → `steam steam://rungameid/<appid>` (appid validated as digits, the only
client-influenced part), `command` → the host-stored command verbatim (trusted, never
from the client). m3.rs sets PUNKTFUNK_GAMESCOPE_APP from it before bringup, exactly
as the GameStream /launch path does (one session at a time). Unit-tested incl. an
injection-attempt guard. Takes effect on the bare-spawn gamescope path; a no-op on a
shared desktop / attach-to-existing session.
- C ABI: `punktfunk_connect_ex4` adds `launch_id` (NULL = none); `_ex3` now delegates to
it. Threaded through NativeClient::connect → WorkerArgs → Hello.
- client-rs gains `--launch ID` (headless testing); client-linux passes None (no picker
yet). Header regenerated.
Next: the Apple library grid passes the picked id via punktfunk_connect_ex4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The video data plane is a raw UDP socket separate from the QUIC control connection. On a flat LAN
the host can send straight to the client, but across NAT or a stateful inter-VLAN firewall the
unsolicited host→client video is rejected (ICMP port-unreachable → the session dies immediately,
while control/audio/input keep working since they ride the client-initiated QUIC). Observed live:
a client on 192.168.6.2 streaming from a host on 192.168.1.48.
Fix: client-initiated hole-punching. The client sends PUNCH_MAGIC datagrams from its data socket
to the host's advertised data port (Welcome.udp_port); that opens the firewall/NAT return path and
lets the host learn the client's OBSERVED source (the NAT-translated address, not the client's
reported private one). The host (UdpTransport::connect_via_punch) waits ≤2.5s for the first punch
and streams there, falling back to the client-reported address for clients that don't punch
(flat-LAN behaviour unchanged). The client keeps a low-rate keepalive so a stateful firewall's idle
timeout can't close the path during a static, low-bitrate scene. Wired into client-rs and the
NativeClient connector (covers the Linux + Apple clients; the Apple app needs an xcframework rebuild
to pick up the new core).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pointer/keyboard injector is host-lifetime (one EIS connection for every punktfunk/1
session), so its existing release_all only fires on EIS disconnect — never when a *client*
session ends. A button still down at an abrupt client disconnect therefore stayed latched in
the compositor: Mutter keeps the destroyed press's implicit pointer grab, so after reconnect a
stuck left-button-down turns every motion into a drag (windows move, text selects) while a
fresh click's press is swallowed — clicking buttons and text inputs does nothing. Only the one
held button is affected; keyboard and the other buttons are fine, exactly as reported.
Fix: input_thread now tracks the buttons/keys the client holds and, when the session ends,
synthesizes the matching up-events through the host-lifetime injector (whose EIS connection —
and the dangling grab — outlive the session). Backend-agnostic (normal inject path), so it
covers libei/EIS, wlr and uinput alike.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The host sealed every packet with ~3 heap allocations: aes-gcm's convenience
encrypt() allocates the ciphertext Vec, seal_for_wire allocates the seq||ct||tag
wire Vec, and seal_frame allocated a fresh Vec<Vec<u8>> per frame. At line rate
(~250k–500k pkt/s for 2.5–5 Gbps) that's the single-core allocator wall.
- SessionCrypto::seal_in_place uses AeadInPlace::encrypt_in_place_detached to
encrypt into the caller's buffer and write the detached tag at the end —
byte-identical to seal's ciphertext||tag, no allocation (unit-tested for byte
equality + decrypt).
- Session keeps a wire_pool the caller returns via reclaim_wires; seal_frame
seals each packet in place into the reused buffers (clear() keeps capacity), so
after warmup there's no per-packet ciphertext/wire allocation. paced_submit and
submit_frame reclaim the pool after sending.
End-to-end encrypted/lossless multi-frame tests stay green (validates the pool
reuse doesn't corrupt across frames). Next: write packetize directly into a
contiguous send buffer (kills the remaining shard allocs + GSO's coalescing copy).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Groundwork for multi-Gbps (2.5G link here, 5G to the Mac Studio). The encoder is
pixel-rate bound, not bitrate bound, so these unblock the transport:
- MAX_BITRATE_KBPS 2G -> 8G, MAX_PROBE_KBPS 3G -> 10G (the cap was policy, not a
hardware limit — NVENC emits multi-Gbps trivially with the 2-way split).
- Welcome shard_payload 1200 -> 1452: fills a 1500 MTU, ~17% fewer packets for
free (even size, FEC-safe; negotiated so the client follows).
- PUNKTFUNK_FEC_PCT env overrides the 20% FEC default — a clean wired LAN can drop
it (every recovery shard is wire bytes+packets); 0 disables FEC.
Next: UDP GSO (the dominant lever — research shows ~2.4x throughput / ~40x fewer
syscalls; sendmmsg batching alone is insufficient) + in-place AES-GCM seal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the intermittent first-connect freeze. The host streams infinite GOP — one
opening IDR, then P-frames only (recovery keyframes just on loss) — so when the
client's decoder wedges on the cold first session (a lost/corrupt opening IDR, a
bad early P-frame) the picture stays frozen until the far-off next keyframe. The
client had no way to ask for one; now it does.
Add a RequestKeyframe control message (client -> host, reliable control stream),
mirroring Reconfigure:
- core: quic.rs RequestKeyframe (type 0x03) + roundtrip test; client.rs
CtrlRequest::Keyframe + NativeClient::request_keyframe; abi.rs
punktfunk_connection_request_keyframe (header regenerated).
- host: m3.rs decodes it in the control loop and signals the encode loop, which
coalesces a burst and calls enc.request_keyframe() — wiring the existing
NvencEncoder hook (force_kf -> next frame pict_type=I), the same recovery the
GameStream path already had via force_idr.
- apple: PunktfunkConnection.requestKeyframe(); StreamPump (stage-1) requests on
layer.status==.failed; Stage2Pipeline (stage-2) on a sync submit failure and on
the async decode-error callback via a thread-safe KeyframeRecovery. All
throttled to <=1/250ms (the decode stays wedged for several frames until the IDR
lands, so per-frame requests would flood the control stream).
Self-healing: a lost recovery IDR is re-requested after the throttle; the host
coalesces bursts into a single IDR.
Validated: cargo fmt + clippy clean; core + host test suites green (incl. new
request_keyframe_roundtrip); swift build + test (39 passed); xcframework rebuilt
(all 5 slices), header regenerated with no unrelated drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An identified-but-unpaired device that knocks on a pairing-required host is now
held as a pending request the operator approves from the web console — pairing it
with no PIN fetched out of band — instead of a flat reject.
- core: Hello gains an optional trailing device name (len u8 || UTF-8, ≤64,
same trailing-back-compat pattern as compositor/gamepad/bitrate). client-rs
--name sends it; the connector sends None (fingerprint-derived label).
- native_pairing: in-memory pending queue (note_pending dedups by fingerprint,
evicts the least-recently-active past a 32 cap, 10-min TTL); approve_pending
pins the fingerprint, deny drops it. Names are sanitized (strip control/ANSI/
bidi — untrusted wire input); add()/remove() roll back in-memory on a persist
failure; pairing clears any stale pending knock.
- m3: the require_pairing gate records the knock (sanitized label) before
rejecting; anonymous (certless) clients record nothing.
- mgmt: GET /native/pending, POST /native/pending/{id}/approve (optional {name})
and /deny; OpenAPI + tests; docs/api/openapi.json regenerated.
- web: a "Waiting for approval" section on the Pairing page (live-poll, Approve/
Deny, error-surfaced via QueryState); en+de strings.
- Also completes an in-progress NativeClient Sync refactor (receivers behind
per-plane mutexes) that was left half-applied in the tree.
Adversarially reviewed (4 lenses + 3-vote verify); the confirmed findings are
fixed here. Validated live on the GNOME box: knock (with a wire name, and a
malicious ANSI/bidi name that got neutralized) → pending → approve → the same
identity streams real video. Full workspace tests + clippy + fmt green; web tsc
clean. Roadmap §8b-1 marked done; §8b-2 (peer-push approval) is the client
follow-up. See docs-site pairing page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The accept loop no longer awaits each session inline — it spawns each onto a
JoinSet, bounded by a semaphore (--max-concurrent, default 4: a NVENC session
bound; overflow clients wait in QUIC's accept backlog until a slot frees). The
QUIC handshake stays in the accept loop so a failed handshake (e.g. a pin
mismatch where the client aborts) doesn't consume a session slot or block
accepting the next client; the slow part (control handshake, pairing, the
capture/encode pipeline) runs in the spawned task.
Each session already had its own virtual output + NVENC encoder; the
host-lifetime input/audio/mic services stay shared — the natural "multiple
devices viewing/controlling the same desktop" semantic on kwin/mutter/wlroots.
gamescope's independent-desktops (per-session input/audio) isolation is a
follow-up. New M3Options.max_concurrent + the `--max-concurrent` CLI flag.
Validated live (GNOME box): two clients connected at once -> two independent
Mutter virtual outputs (720p60 + 1080p60) streaming simultaneously (39 MB +
48 MB). All 61 host tests green (the c_abi/pairing tests exercise the new loop +
the failed-handshake-doesn't-count semantics).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ClockProbe/ClockEcho on the QUIC control stream — 8 NTP-style rounds right after
Start; the min-RTT sample gives the host-client clock offset (clock_offset_ns
estimator in punktfunk-core). The client adds the offset to its receive instant
before differencing against the AU pts_ns, so the capture->reassembled latency
percentiles are valid across machines (skew_corrected=true), not just same-host.
Back-compat: an old host that doesn't answer the probe times out and the client
falls back to a shared-clock assumption (skew_corrected=false).
Host adds one ClockProbe dispatch arm in the control task; the client runs
clock_sync after Start, before the --remode/--speed-test tasks take the stream.
Validated cross-LAN (GNOME box -> dev box): offset ~ -1.57 ms (reproducible),
rtt ~140 us, p50 1.30 ms skew-corrected capture->reassembled — the offset is
exactly the systematic error the handshake removes. Unit tests for the message
codecs and the min-RTT offset estimator.
Roadmap §12: skew handshake done; remaining for true glass-to-glass is the Apple
client present-stamp (decode->present) plus the host render->capture term.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both the unified host (serve --native) and standalone m3-host now advertise the
native punktfunk/1 service over mDNS (_punktfunk._udp) — the analogue of the
GameStream _nvstream._tcp advert. TXT records carry proto, the host cert
fingerprint (fp, the value clients pin), the pairing requirement
(pair=required|optional), and the host id. New crate::discovery module, wired
into m3::serve so both host entry points get it; best-effort, never blocks
streaming (--connect always works).
Client gains `punktfunk-client-rs --discover [SECS]`: browses the LAN and prints
each host (name, addr:port, pairing, fingerprint), then exits. Apple clients
browse the same service natively via NWBrowser (service type + TXT keys are the
contract).
Validated cross-LAN: the dev box discovered the GNOME-box appliance
(pair=required) and a standalone synthetic host (pair=optional); fingerprint and
pairing state correct in both.
Also refresh the now-stale sendmmsg caveat in the bitrate doc (batched/paced send
landed + validated to 1 Gbps) and mark the encode|send thread split done in §12.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bigger-bet #1 from the latency plan. virtual_stream ran capture+encode+seal+
paced-send on ONE thread, so frame N+1's capture/encode couldn't start until
frame N's entire paced tail had left the wire — the pacing budget (~0.9×interval)
was serialized in front of the next encode. Port GameStream's spawn_sender model
to the native path:
- A dedicated send thread (`send_loop`) owns the WHOLE Session (so no socket
clone or shared/Arc stats needed — `seal_frame` mutates the nonce, `send_sealed`
+ the probe bursts all live there) and does FEC+seal + microburst-paced send.
- The encode thread captures+encodes + handles reconfig and hands each AU over a
bounded sync_channel(3) as a FrameMsg (data, capture_ns, flags, deadline,
encode_us). It BLOCKS on backpressure if the send falls behind — frames slow
down rather than a dropped frame freezing the infinite-GOP stream (we don't
drop). Clean shutdown: drop the channel → send thread drains/exits → join.
- Probes (run_probe_burst) move to the send thread since they need the Session; a
burst naturally pauses video (the encode thread blocks on the full channel).
- Per-frame encode_us/pace_us histogram moved to the send thread (carries
encode_us in the FrameMsg) and now reflects the overlap.
Removes the encode↔paced-tail serialization (~2-8 ms @60-120 fps), independent of
the pacing policy, no quality cost. Substrate for the future NVENC slice wrapper.
Verified live on this box (appliance restarted onto it): a client streamed the
KWin desktop (1.49 MB H.265, clean, no panic) and a 200 Mbps speed-test probe
completed through the send thread (0 drops). Build + clippy + fmt green.
Real-NIC sustained soak (reconfig under load, line-rate, mode switches) pending
the Ubuntu third host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From a bug-hunt + unsafe-audit pass (4 reviewers + adversarial verify). It
confirmed ZERO real bugs in the recent batched/paced data-plane work — these are
the surfaced cleanups + one genuine soundness fix:
- SOUNDNESS (reduce unsafe): inject/gamepad.rs::pump_ff did `ptr::read` of an
InputEventRaw (align 8, holds a timeval) out of a 1-aligned [u8; N] buffer — UB
per the reference (x86_64 tolerates it, but it can miscompile under LTO). Use
ptr::read_unaligned + a SAFETY note. Zero behavior change.
- recv parity: recv_batch (recvmmsg) didn't drop an oversized/truncated datagram
the way scalar recv does — poll_frame now skips a message whose len fills the
buffer (> MAX_DATAGRAM_BYTES), matching recv's `n >= RECV_BUF` drop. (AEAD
already rejected these on encrypted sessions; this restores the documented
invariant on the batched path.)
- dedup unsafe FFI: factor the identical mmsghdr-from-iovec construction out of
send_batch + recv_batch into one `mmsghdrs()` helper — the raw-pointer
scaffolding + its lifetime SAFETY note now live in one place.
- docs: TARGET_SOCKBUF no longer calls paced sending future work (it landed,
m3.rs::paced_submit); gamescope.rs input is no longer "(TODO)" (wired +
live-validated); the PUNKTFUNK_PERF `wire_mbps` field is renamed `tx_mbps` and
noted as attempted/sealed bytes (send_dropped shows what didn't reach the wire).
Full suite (35 + loopback round-trip + 6) + clippy + fmt green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From the latency investigation: the freeze-fix pacing (paced_submit) was the
single biggest software-controllable latency term — it unconditionally spread
EVERY multi-chunk frame over ~90% of the frame interval, adding up to ~7.5 ms
@120 / ~15 ms @60 to a frame's last packet even when the frame was small or the
link idle. Recover that on the common case while keeping the freeze fix:
- Microburst-cap pacing: a frame whose sealed size is <= a cap (default 128 KB,
PUNKTFUNK_PACE_BURST_KB) goes out in ONE immediate burst — no pacing latency.
Only the OVERFLOW of a bigger frame (IDR / sustained high bitrate, the bursts
that actually overran the tx buffer and froze) is spread. 128 KB is well under
the ~150 Mbps@60 frame size where drops began, so the default is safe; raise it
after confirming send_dropped stays 0 on a given link. Still never slower than
unpaced (budget collapses to 0 with no slack). seal-once/in-order nonce
preserved — chunks are split, never reordered or re-sealed.
- Per-frame instrumentation (PUNKTFUNK_PERF, zero-cost off): encode_us +
pace_us (the pacing tail) p50/p99/max histograms + immediate-vs-paced frame
counts in the periodic perf line, so the pacing tail is finally visible and the
cap is tunable against real numbers.
Host builds + clippy + fmt green. NOT yet deployed to the running hosts (still on
the safe full-pacing A+B build) — needs the user's LAN soak to validate the cap
doesn't reintroduce send_dropped before raising it. Deferred bigger bets (need
real-NIC/GPU/Mac validation): encode|send thread split on the native path,
CUDA stream+event (one redundant sync), NVENC slice wrapper, stage-2 Apple
presenter, glass-to-glass probe — see docs/roadmap.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Increment B of the send-path rework — the actual fix for "freezes get more
common over ~150 Mbps, no image at all at 400 Mbps" on the native path. Cause:
the encoder emits a frame and submit_frame blasted ALL its packets at once into
the NIC; a real link drops the line-rate burst (host send buffer EAGAINs), and
under infinite GOP one dropped frame freezes the decode until the next keyframe.
(The speed-test probe showed 0 drops at 400 Mbps because the probe is self-paced;
real video wasn't.)
Adaptive pacing, no extra thread, no regression:
- Session splits into seal_frame (FEC + packetize + seal → wire packets, no
send) and send_sealed (one batched sendmmsg of a chunk, counts drops);
submit_frame is now their composition (synthetic + probe paths unchanged).
- virtual_stream's paced_submit seals a frame then sends it in 16-packet chunks
spread over ~90% of the time until the next frame is due. At 60 fps desktop
(fast encode → lots of slack) the frame spreads across the interval → no NIC
burst → no freeze. At 240 fps@5K (encode ≈ interval → ~0 slack) the budget
collapses and every chunk goes out immediately → never slower than before.
Core suite (34 + loopback round-trip + 6) + clippy + fmt green. The seal/send
split is covered by the existing loopback tests; the pacing is host timing,
verified by review (live-test needs a real NIC — your Mac at a raised bitrate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First step of 1 Gbps+ readiness (the whole point of the GF(2^16) Leopard FEC):
make 1 Gbps configurable and its dominant failure mode observable, before the
real transport work (sendmmsg + paced encode|send split) lands.
Investigation (6-way) verdict: we're ~halfway, and it's mostly clamps plus one
real piece of work. The integer/type path, FEC (a 1 Gbps frame is only a few
hundred shards in one GF(2^16) block, far under the 65535 ceiling), AES-GCM
(AES-NI, ~10-25x headroom), and the M1 reassembler bounds (fully derived from
the negotiated FecConfig) are ALL already 1 Gbps-ready and untouched.
This commit (the configurable + observable foundation):
- m3.rs: MAX_BITRATE_KBPS 500_000 -> 2_000_000 (2 Gbps headroom over the 1 Gbps+
target); MAX_PROBE_KBPS 1_000_000 -> 3_000_000 (probe can demonstrate headroom
ABOVE the session cap so a client can confidently pick a 1 Gbps+ bitrate).
- transport/udp.rs: TARGET_SOCKBUF 8 MB -> 32 MB (a multi-MB IDR keyframe burst
no longer fills the buffer); scripts/99-punktfunk-net.conf bumped to match.
- Observability: Transport::send now returns Ok(true|false) (false = WouldBlock
send-buffer drop, previously a silent Ok(())). Session counts these as a new
`packets_send_dropped` stat (distinct from recv-side packets_dropped) — in
Stats, the C ABI PunktfunkStats (header regenerated), a PUNKTFUNK_PERF periodic
wire-Mbps + drop dump in virtual_stream, and the speed-test probe completion
log. This is the dominant 1 Gbps+ loss mode and was invisible.
Loopback-verified: a probe now runs at 1.2 Gbps target (no longer truncated to
1 Gbps) with the drop counter live. NOT yet a sustained-1-Gbps proof — the
single-send()-per-packet native path is the next, real piece of work (port the
proven GameStream sendmmsg + paced send thread into the core Transport).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related additions to the native protocol, host-side (the client side of
each is exposed over the C ABI so the platform clients can wire it up).
Bitrate negotiation
- Hello/Welcome carry `bitrate_kbps` (appended trailing-byte field, back-compat:
old peers decode 0 = host default). The client requests a rate; the host
clamps it to [500 kbps, 500 Mbps] (or its 20 Mbps default when 0) and echoes
the resolved value in Welcome. Replaces the hardcoded 20 Mbps NVENC bitrate in
m3.rs — threaded through virtual_stream → build_pipeline → open_video, applied
on the initial mode and every reconfigure rebuild.
- C ABI: punktfunk_connect_ex3(..., bitrate_kbps, ...) (ex2 delegates with 0);
punktfunk_connection_bitrate() reads the resolved value.
Speed test (bandwidth probe)
- New typed control messages ProbeRequest{target_kbps,duration_ms} (0x20) /
ProbeResult{bytes_sent,packets_sent,duration_ms} (0x21), plus a FLAG_PROBE
packet flag. The client asks the host to burst zero-filled, FLAG_PROBE-tagged
access units over the data plane at a target goodput for a duration (clamped
≤ 1 Gbps / ≤ 5 s), pacing by a bytes-allowed budget; video pauses for the
burst. The host reports what it actually sent; the client measures received
bytes + window → goodput and loss. Probe filler is never fed to the decoder
(diverted in the connector pump and the reference client's poll loop).
- The host control task now multiplexes Reconfigure + ProbeRequest (inbound)
and ProbeResult (outbound) over select!; a probe channel reaches the
data-plane thread (both virtual and synthetic sources).
- Connector: NativeClient::request_probe()/probe_result() with an internal
accumulator; C ABI punktfunk_connection_speed_test() +
punktfunk_connection_probe_result() → PunktfunkProbeResult.
- punktfunk-client-rs gains `--bitrate KBPS` and `--speed-test KBPS:MS` (its own
loop measures + logs goodput/loss) for loopback verification.
Validated on loopback (synthetic source): a 20 Mbps / 2 s probe measured
20050 kbps at 0% loss, bitrate negotiated (0→20000 and 50000→50000), and the
interleaved probe AUs were correctly excluded from frame verification
(mismatched=0). Wire codecs + trailing-byte back-compat have unit tests. C
header regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple client grows full gamepad support and punktfunk/1 learns to negotiate
the virtual pad type:
- Protocol: Hello carries a GamepadPref byte (offset 21, the same trailing-byte
back-compat pattern as the compositor; echoed resolved in Welcome at 54).
Host precedence: explicit client choice > PUNKTFUNK_GAMEPAD env > Xbox 360,
DualSense (UHID) only where available. ABI: punktfunk_connect_ex2 +
punktfunk_connection_gamepad (connect_ex delegates; ABI_VERSION stays 2 — the
trailing byte IS the compat mechanism). punktfunk-client-rs gets --gamepad.
- Swift client: GamepadManager (app-lifetime discovery + selection — Settings
lists every controller with capabilities/battery/"In use"; exactly ONE pad
forwards as pad 0, auto = most recently connected, or pinned), GamepadCapture
(snapshot-diff button/axis events, DualSense touchpad + ~250 Hz motion on the
rich-input plane, held state released on switch/deactivate/stop),
GamepadFeedback (rumble → CoreHaptics per-handle engines; lightbar →
GCDeviceLight; player LEDs → playerIndex; adaptive-trigger blocks → the
table-driven DualSenseTriggerEffect parser → GCDualSenseAdaptiveTrigger,
exact for the 10-zone positional modes). The pad type auto-resolves from the
physical controller at connect time, user-overridable in Settings.
- Host DualSense fixes surfaced by adversarial review against hid-playstation /
SDL / Nielk1 ground truth: input-report sensor/touch offsets were off by one
(the kernel read garbage motion + phantom touches), the L2/R2 trigger blocks
were swapped (the report is right-trigger-first), feedback now gates on the
report's valid-flags (a plain rumble write no longer blanks lightbar/
triggers), and the touchpad rescale clamps to the advertised ABS_MT extents.
- Tests: Hello/Welcome trailing-byte back-compat, pick_gamepad precedence,
byte-exact input-report layout, valid-flag gating, per-mode trigger-parser
table (incl. packed 3-bit zones), wire conversions, and a scripted loopback
feedback burst (PUNKTFUNK_TEST_FEEDBACK=1) asserted through the xcframework
on the rumble + HID-output planes.
Validated: cargo test/clippy/fmt green on macOS + Linux (61 host tests), swift
build/test green, test-loopback.sh green, tvOS/iOS targets compile. DualSense
motion sign/scale is derived from the calibration blob, not yet live-verified
(constants isolated in GamepadWire).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An open punktfunk/1 host any LAN device can trust-on-first-use and stream from is
insecure. The unified host now gates native sessions on pairing by DEFAULT: a client
must complete the SPAKE2 PIN ceremony (armed from the web console) before it's
admitted; paired devices persist. `serve --open` keeps the old TOFU behavior for
trusted single-user setups.
native_serve_opts now takes a NativeServe { port, require_pairing }; parse_serve
builds it with require_pairing = !--open. GameStream pairing (separate) is unchanged.
The require_pairing gate + ceremony are already covered by m3::pairing_ceremony_and_gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`serve --native` now runs the GameStream host AND the native punktfunk/1 (QUIC)
host in ONE process, sharing a single NativePairing handle with the management API
— so native pairing is operable from the web console instead of journalctl.
- gamestream::serve gains a native_port: spawns crate::m3::serve in the same
runtime and passes the shared NativePairing to mgmt::run. Validated live: one
process binds both RTSP 48010 and QUIC 9777.
- mgmt API: new `native` endpoints — GET /native/pair (status), POST
/native/pair/arm (mint a fresh, time-limited PIN to DISPLAY), DELETE /native/pair
(disarm), GET/DELETE /native/clients (list/unpair). GameStream-only hosts report
enabled:false. OpenAPI regenerated (checked-in doc + drift test).
- main.rs: serve --native / --native-port flags.
The native host arms pairing on demand (the operator reads the PIN from the
console; the SPAKE2 ceremony is host-shows-PIN). New mgmt + native_pairing tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Groundwork for web-UI-driven native (punktfunk/1) pairing. Replaces m3's fixed
startup PIN + local paired store with a shared `NativePairing` (new module):
arm-on-demand with a fresh, time-limited PIN (`arm(ttl)`), `current_pin()` read
per ceremony so a lapsed window stops pairing, plus the trust store (list/add/
remove/is_paired) and a `status()` snapshot. The management API (next commit) and
the QUIC accept loop share one handle. CLI `--allow-pairing`/`--require-pairing`
still arm at startup (no expiry, PIN logged) — back-compat. m3 pairing ceremony +
gate and the C-ABI roundtrip stay green; new unit tests for arm/expire/pair.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PUNKTFUNK_GAMEPAD=dualsense now routes a session's gamepad through a real virtual
DualSense (UHID + hid-playstation) end to end:
- host: a `PadBackend` enum (m3.rs) selects `GamepadManager` (uinput xpad, default)
or the new `DualSenseManager` (dualsense.rs) per session. The manager keeps each
pad's full DsState so touchpad + motion (rich-input plane) persist across
button/stick frames, and services the !Send /dev/uhid fd only on the input thread
(which cycles <=4ms, so the GET_REPORT init handshake completes).
- feedback: `service()` now returns `DsFeedback { hidout, rumble }`. Motor rumble
stays on the universal 0xCA plane (so non-DualSense clients still feel it; manager
dedups change); lightbar / player LEDs / adaptive-trigger effects ride the new
0xCD HID-output plane (host->client) as `HidOutput`.
- rich input: touchpad contacts + motion ride the 0xCC plane (client->host) as
`RichInput`, applied via `DualSenseManager::apply_rich` (merged with button state;
touch normalized 0..65535 -> the touchpad resolution).
- connector + C ABI: `NativeClient::next_hidout` / `send_rich_input`, exported as
`punktfunk_connection_next_hidout` (-> PunktfunkHidOutput) and
`punktfunk_connection_send_rich_input` (<- PunktfunkRichInput); header regenerated.
- reference client: `--rich-input-test` drives the DualSense touchpad + motion and
logs the 0xCD feedback that comes back.
Validated live on-box: a synthetic-source m3-host + client-rs created the real
kernel DualSense, drove 0xCC, and decoded 12 live 0xCD events (the kernel's actual
lightbar/trigger init reports) with the data plane unaffected (600/600 frames).
Adversarial review fixes folded in: the input loop no longer skips the rich drain +
feedback pump on a dropped gamepad event, and the touch contact id is clamped to its
slot. Remaining: the Apple client renders triggers/rumble on a real DualSense.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new features were Linux-built only and broke the documented macOS gate
(cargo build/test/clippy --workspace) four ways, all fixed following the existing
platform-gating conventions:
- m3.rs: mic_service_thread split into the Linux worker and a non-Linux stub that
drains and drops (sessions still count the datagrams) — opus/PipeWire are
Linux-gated deps, same pattern as audio_thread.
- punktfunk-client-rs: the new `opus` dependency moved into the Linux target table and
--mic-test gated with a warn-and-skip stub (only the synthetic-tone test rig needs
the encoder; the mic uplink itself is portable).
- gamestream/audio.rs: SAMPLE_RATE import gated to any(linux, test) (the frame_sizing
test uses it everywhere, the data plane only on Linux).
- tests/c_abi.rs: the harness's macOS link flags gained Security + CoreFoundation —
the quic feature now pulls rustls's platform verifier into the staticlib.
Also: two clippy match-ref-pats lints in the new rich-input/HID-output decoders
(clippy -D warnings is the repo gate), the regenerated punktfunk_core.h committed (the
checked-in copy predated the rich-input/HID-output constants — CI fails on drift), and
web's inlang cache dir gitignored.
cargo build/test/clippy/fmt --workspace: green on macOS, 122 tests passing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The inverse of the host→client audio path: the client's mic, Opus-encoded, rides a
new 0xCB QUIC datagram to the host, which decodes it into a virtual PipeWire
Audio/Source its apps can record from (voice chat, etc.).
Protocol (punktfunk-core):
- MIC_MAGIC 0xCB + encode/decode_mic_datagram (mirror of the 0xC9 audio datagram).
- NativeClient::send_mic(seq, pts_ns, opus) over a new outbound channel + worker task
(mirror of send_input); C ABI punktfunk_connection_send_mic for native clients.
Host:
- audio::VirtualMic + PwMicSource: a PipeWire output stream tagged media.class=
Audio/Source (Direction::Output) — a recordable microphone node, fed decoded PCM.
- MicService: host-lifetime owner of the source + Opus decoder (mirror of
InjectorService / the audio capturer slot); lazily opened, persists across sessions,
self-heals. The per-session datagram reader now demuxes 0xCB→mic / 0xC8→input over a
single read_datagram loop (two loops would race).
- Adaptive jitter buffer in the producer: primes to ~3 consumer quanta before emitting,
so the 5 ms push / N ms pull clock skew never underruns — without it ~58% of output
was silence; with it, glitch-free across consumer quanta.
Client: punktfunk-client-rs --mic-test streams a synthetic 440 Hz Opus tone as the mic
uplink (opus dep added) for end-to-end validation without a real microphone.
Validated live on headless KWin: client tone → host source → pw-record shows the
punktfunk-mic Audio/Source node, 440 Hz dominant (Goertzel power 20.7 vs <0.001
elsewhere), RMS 0.179 ≈ the ideal 0.177, 0.3–0.4% silence at both 256 ms and 10 ms
consumer quanta. Tests +1 (mic datagram roundtrip); workspace green, clippy/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under rapid client reconnects, KWin's libei/EIS input setup intermittently wedged
with "EIS setup timed out", causing total input loss for affected sessions. Root
cause: each punktfunk/1 session opened (and tore down) its own RemoteDesktop-portal
CreateSession for pointer/keyboard injection, and back-to-back reconnects raced a
prior session's portal teardown before it settled.
LibeiInjector is only a Send channel handle to a worker thread that owns the portal
session, so the injector can live for the whole host run instead of per session.
Adds InjectorService: one host-lifetime thread owns the (!Send) injector, opened
ONCE (lazily, on the first event) and reused across every session — the portal grant
is established a single time and held. Sessions forward pointer/keyboard events to it
over a clonable Send channel; gamepads stay per-session (uinput, no portal). The
service self-heals — reopen after a 2s backoff if open fails or the backend worker
dies (covers a gamescope EIS socket that respawns with its nested session).
Mirrors the existing host-lifetime audio-capturer slot; the audio capturer is Send
(a slot works), the injector is !Send (needs the owning thread + channel).
Validated live on headless KWin: 8 rapid back-to-back input sessions →
"input injector ready (host-lifetime)" exactly once, ZERO "EIS setup timed out",
8/8 sessions injected input. Tests green, clippy/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A client can now request which compositor backend the host drives its virtual
output on (gamescope/KWin/Mutter/wlroots). The host honors the request if that
backend is available, else falls back to auto-detect and reports the resolved
choice back — wire-compatible both directions (no ABI bump).
Protocol (punktfunk-core):
- New CompositorPref (config.rs): Auto|Kwin|Wlroots|Mutter|Gamescope with
u8/name mappings. Appended as one optional byte to Hello (client preference)
and Welcome (host's resolved choice). Both decoders already tolerate trailing
bytes, so old↔new interop is preserved — ABI_VERSION stays 2. Round-trip +
back-compat (truncated-message) tests.
- C ABI: punktfunk_connect_ex(compositor) + PUNKTFUNK_COMPOSITOR_* constants;
punktfunk_connect delegates with AUTO, so the existing symbol is unchanged.
NativeClient::connect / worker_main thread the preference through.
Host:
- vdisplay::available() enumerates usable backends via cheap, side-effect-free
probes (KWin zkde global, gamescope binary+version, GNOME/Sway env), plus
Compositor id/label/as_pref/from_pref/all helpers.
- m3 handshake resolves the preference to a concrete backend during the
handshake (pick_compositor pure + resolved logging), reports it in Welcome,
and threads it into virtual_stream (replacing the unconditional detect()).
- mgmt GET /v1/compositors lists every backend with availability + the
auto-detected default (OpenAPI regenerated).
Client:
- punktfunk-client-rs --compositor NAME; logs the host's resolved choice from
the Welcome ("session offer … compositor=…").
Web console:
- Host page gains a Compositors card (availability + default badges) via the
codegen'd useListCompositors hook; en/de strings added.
Also fixes a pre-existing, env-dependent test-isolation bug:
mgmt::tests::paired_clients_list_and_unpair seeded the real
~/.config/punktfunk/paired.json (AppState::new loads it), so a real
GameStream-paired client leaked into body[0] on a dev box — now cleared first.
Live-validated against headless KWin: --compositor kwin honored, --compositor
mutter falls back to kwin (available=[kwin, gamescope]), resolved choice
round-trips to the client. Tests: +6 (wire/back-compat, resolution precedence,
endpoint); workspace green, clippy/fmt clean, C ABI harness PASS at abi_version=2,
web typecheck + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardens the virtual-display → capture → encode bring-up against the transient
failures that surfaced as black screens / wrong refresh on cold KDE sessions.
- m3: build_pipeline_with_retry wraps the initial vd.create() + first-frame with
bounded exponential backoff (4 attempts, 500ms→2s). is_permanent_build_error
classifies config/version/missing-tool failures so they fail fast instead of
burning the retry budget. Encoder + frame clock now pace to the *achieved*
refresh reported in VirtualOutput::preferred_mode, not the requested rate.
- capture/linux: PortalCapturer::Drop sends a pipewire channel quit and joins the
thread, so a dropped/failed/retried capturer releases its PipeWire thread + EGL/
CUDA context promptly instead of leaking it to process exit. First-frame timeout
now reports the node id and distinguishes "format never negotiated" from
"negotiated but no buffers arrived" via a negotiated flag set in param_changed.
- vdisplay/kwin: set_custom_refresh reads back the active mode from kscreen-doctor
and returns the refresh KWin actually gave us (a rejected custom mode silently
leaves the output at 60Hz); create() carries it into preferred_mode.
- vdisplay/gamescope: find_gamescope_node requires the Video/Source object (the
node.name=gamescope tag is on two objects; the other wedges the link); a version
check warns on <3.16.22 (the PipeWire-1.6 capture-deadlock signature).
Live-validated against headless KWin: 720p120 build with requested=120 achieved=120,
zero-copy CUDA frames, and no per-session thread accumulation across back-to-back
sessions. Tests: +3 unit (retry classifier, gamescope version parse); 49 host tests
green, clippy/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triaged the multi-agent review of the renegotiation + pairing + Sway + AV1/surround batch
(1 critical, 11 major/minor confirmed). Fixes:
CRITICAL — PIN pairing was offline-brute-forceable. The HMAC-of-PIN proof let an active
MITM who terminates the TOFU ceremony recover the 4-digit PIN by offline dictionary search
(all other inputs observable) and forge a correctly-bound proof. Replaced with **SPAKE2**
(balanced PAKE, `spake2` crate) + key-confirmation MACs, binding both cert fingerprints as
the SPAKE2 identities: an attacker gets exactly ONE online guess, no offline search, and
mismatched cert views (a real MITM) never reach a shared key. Also reworked the UX to an
"arming PIN" — one PIN per arming window shown at host startup (the SPAKE2 client needs the
PIN to build its first message, so it can't be minted per-connection). Validated live:
wrong PIN rejected in 0.1s, right PIN pairs + persists + the paired identity streams.
Pairing hardening: `--allow-pairing`/`--require-pairing` must arm pairing (default rejects
unsolicited ceremonies); per-host cooldown bounds online guessing; the client flushes its
CONNECTION_CLOSE so a refused ceremony can't wedge the sequential host for the full timeout;
atomic (temp+rename) paired-store writes.
Protocol: control/pairing messages use a distinct CTL_MAGIC (PKFc) — fully disjoint from
the positional Hello namespace (a future abi_version can't be misparsed as a control
message); all typed decodes are length-exact. ABI_VERSION → 2 (punktfunk_connect signature
gained the identity params; header regenerated).
Renegotiation: drain the reconfig channel to the NEWEST mode (one rebuild, not one per
stale step); validate refresh_hz; build the new pipeline BEFORE dropping the old so a
rebuild failure keeps the session on its current mode instead of killing it.
GameStream: packetDuration snaps to {5,10} (an in-between value isn't a legal Opus frame
size and would kill audio). Sway: chooser file moved to $XDG_RUNTIME_DIR (was a fixed
world-writable /tmp path — DoS / capture-misdirection by another local user).
Swift: fixed two compile breakers in the new pairing/identity APIs (Int32 status .rawValue,
UInt cap cast). New SPAKE2 + namespace-disjointness + pairing-roundtrip unit tests; the
in-process pairing test now also exercises the arming PIN + cooldown. 114 tests green,
clippy -D warnings clean (both feature sets), fmt, C-ABI harness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renegotiation (no reconnect on resize): the handshake bi-stream stays open; the client
sends Reconfigure{mode} (typed post-handshake message), the host validates + acks
Reconfigured and rebuilds capture/encoder/virtual output at the new mode while the data
plane (keys, ports, FEC) runs untouched — the first new-mode AU is an IDR with in-band
parameter sets. NativeClient::request_mode / punktfunk_connection_request_mode; mode()
reflects the active mode. Validated live on KWin: one continuous stream, 225 frames
@1280x720 then 395 @1920x1080, ~90 ms pipeline rebuild (ffprobe shows both resolutions).
PIN pairing (mutual trust, kills TOFU MITM): clients get persistent self-signed
identities presented via QUIC client auth (generate_identity / client auth offered but
optional server-side — legacy clients still connect). Ceremony on the control stream:
PairRequest{name} → host shows a 4-digit PIN (log) + PairChallenge{salt} → client proves
with HMAC-SHA256(PIN‖salt, client_fp‖host_fp) — binding both certs means a MITM can't
forward a proof, single attempt per PIN, constant-time compare → PairResult; host
persists the fingerprint (~/.config/punktfunk/punktfunk1-paired.json), client pins the
host's. m3-host --require-pairing gates sessions on the paired set.
NativeClient::pair + punktfunk_pair/punktfunk_generate_identity in the ABI; reference
client: --pair PIN --name LABEL + auto-generated persistent identity, --remode for live
renegotiation testing. Swift wrapper: ClientIdentity/generateIdentity()/pair(),
requestMode()/currentMode(); README handoff updated.
Tested: reconfigure/pairing wire roundtrips, C-ABI mode switch ack, full in-process
ceremony (wrong PIN → Crypto, anonymous-vs-gate rejection, success → pinned session);
live wrong-PIN ceremony against the serving host (PIN logged, proof rejected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codec negotiation (M2 polish):
- ServerCodecModeSupport now advertises what we encode: H264|HEVC|AV1_MAIN8
= 65793 (flags verified against moonlight-common-c Limelight.h). The old
placeholder 3843 wrongly claimed HEVC Main10 + 4:4:4 and no AV1. Main10
bits stay off on purpose: Moonlight ties 10-bit to HDR, and capture is
8-bit SDR BGRx with no HDR metadata path (av1_nvenc -highbitdepth was
validated working for later).
- RTSP ANNOUNCE: bitStreamFormat 0/1/2 -> H264/HEVC/AV1 (already plumbed to
av1_nvenc; validated e2e via `m0 --codec av1` + ffprobe av01), and a
dynamicRangeMode!=0 request now logs + falls back to 8-bit SDR.
Surround audio (M2 polish):
- ANNOUNCE x-nv-audio.surround.{numChannels,AudioQuality} +
x-nv-aqos.packetDuration -> per-session AudioParams; DESCRIBE advertises
all six Opus configs (normal before HQ per channel count). Normal-quality
mappings are pre-rotated for the client's GFE-order LFE swap
(RtspConnection.c, verified verbatim) so its derived decoder mapping
equals our encoder mapping — including 7.1, where Sunshine's rotate only
covers [3,6) and scrambles LFE/SL/SR.
- 5.1/7.1 encode via libopus multistream (audiopus_sys, the sys layer the
opus crate already links) with Sunshine's layouts/bitrates, RAII wrapper;
the live-validated stereo wire is byte-identical (plain Opus, no FEC).
- Surround sessions add Sunshine-style RS(4,2) audio FEC (packetType 127 +
AUDIO_FEC_HEADER, the OpenFEC parity matrix both ends hardcode, nanors
gemm semantics verified from nanors/rs.c).
- PipeWire capture generalized to the negotiated channel count with explicit
FL FR FC LFE RL RR [SL SR] positions; missing sink channels are zero-
filled by the channel-mixer. PwAudioCapturer now tears down cleanly on
Drop (pipewire channel -> loop quit), so a channel-count change can
reopen without leaking a capture stream.
Tests: serverinfo mask, RTSP codec/audio param parsing, DESCRIBE contents,
surround-params strings + client-swap round trip, FEC parity self-recovery
and packet layout, real-codec 5.1 channel-identity round trip, and an
ignored live test (ran green against a 6ch null sink monitor).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler