The capture-architecture reset from the research: add a Windows.Graphics.Capture (WGC) backend that
captures the COMPOSED desktop — including the overlay/independent-flip/MPO planes DXGI Desktop
Duplication misses — which structurally fixes the frozen HDR animations + video (proven live: a WGC
frame decodes to the real 5120x1440 HDR content DDA freezes on). It reuses the whole pipeline
unchanged: the WGC frame's GPU texture → same scRGB→BT.2020-PQ shader → NVENC zero-copy; the OS
composites the cursor (IsCursorCaptureEnabled) so no manual cursor pass. crates/punktfunk-host/src/
capture/wgc.rs; find_output/make_device/HdrConverter/nudge_cursor_onto made pub(crate) for reuse.
Reliability findings + mitigations (live on the RTX 4090):
- WGC can't activate under the SYSTEM account (0x80070424) — it needs the interactive user token. The
host must run as the user for WGC (run.cmd: drop PsExec -s). DDA still needs SYSTEM for the secure
desktop — that token reconciliation (impersonation) is the remaining task.
- WGC's Direct3D11CaptureFramePool::CreateFreeThreaded intermittently HANGS on the headless SudoVDA
(IddCx) display, correlated with accumulated SudoVDA churn (failed REMOVEs leaving lingering
displays); clean-state opens reliably. Since it's a blocking hang, capture_virtual_output runs WGC
open on a watchdog thread with a 5s timeout and falls back to DDA on hang/error — the session is
NEVER left black: WGC when it opens (fixed animations), DDA otherwise. First-frame nudge added (WGC
fires FrameArrived on change; a static desktop otherwise never delivers the first frame).
- Default WGC; PUNKTFUNK_CAPTURE=dda forces DDA. DDA path unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live-validated Mac <-> RTX 4090 at the display's native 5120x1440@240:
- Resolution: set_active_mode enumerates the IDD's advertised modes and sets the
requested resolution at the best supported refresh (keeps 5120x1440@240; no more
silent fallback to the 1080p OS default when an exact mode is briefly unavailable).
- Bitrate auto-cap: NVENC init probes and steps the average bitrate down to the GPU's
codec-level max so a high client bitrate connects (matches the Linux host; we do not
split NVENC sessions).
- Mouse cursor: DXGI duplication excludes the HW cursor; capture the pointer
shape/position (GetFramePointerShape) and GPU-composite it before NVENC. Color cursors
alpha-blend; masked-color (the text I-beam) uses an INV_DEST_COLOR inversion blend so
the caret inverts the screen and shows on any background (no black box); monochrome
handled too.
- Secure desktop (lock / login / UAC): run as SYSTEM in the interactive session, follow
the input desktop via SetThreadDesktop, and on the WinSta switch recreate the D3D11
device and re-resolve the virtual output's GDI name from the stable SudoVDA target id
(the name changes across the topology rebuild; the old failure hunted the stale
\\.\DISPLAYn and dropped). ACCESS_LOST / INVALID_CALL / device-removed are recoverable,
and a mid-stream resolution change is followed (capturer + NVENC re-init at the new
size). isolate_displays detaches other monitors so Winlogon renders to the virtual
output. One real session recovered 1012 desktop switches and completed cleanly.
Windows-only backends; Linux/macOS unaffected. Builds clean on x86_64-pc-windows-msvc.
Deployment (windowless SYSTEM launch via PsExec + hidden VBScript) documented in
docs/windows-host.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the host→client rumble path on Windows, the analogue of the Linux
uinput EV_FF read loop: a game's force-feedback on the virtual Xbox 360
pad is delivered by ViGEm's notification API (`request_notification` →
`spawn_thread`, gated by the crate's `unstable_xtarget_notification`
feature). A per-pad background thread stores the latest motor levels;
`pump_rumble` relays changes to the client on the universal 0xCA plane
(motors scaled 0..255 → 0..65535). Dropping the target aborts the
notification, so the thread exits with the session. Live verification
still needs a physical pad.
Also fix the Windows backends' clippy debt — these modules are cfg-
excluded from Linux CI, so `clippy -D warnings` never saw them, and the
VM's rustc 1.96 clippy is stricter on shared code than the CI image:
- dxgi: manual checked division → checked_div().map_or
- sendinput: `x = x | y` → `x |= y`
- sudovda: `.then(|| ptr)` → `.then_some(ptr)`
- m3 pick_compositor: drop the needless early return (match form)
- m3 resolve_compositor: Windows arm is a tail expr, not `return`
All Windows backends now build + clippy clean (default and --features
nvenc); Linux unaffected (fmt/clippy/check green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `m3` audio_thread (desktop capture → Opus 48 kHz stereo 5 ms CBR →
AUDIO_MAGIC datagrams) now runs on Windows, fed by the WASAPI loopback
capturer. The `opus` crate vendors libopus via `audiopus_sys` + cmake
(no system lib / vcpkg), so it builds on MSVC — moved into a
`cfg(any(linux, windows))` deps table and widened the audio_thread cfg
to match (the stub now only covers other targets, e.g. macOS).
Build note: CMake 4 rejects libopus's old `cmake_minimum_required`;
set `CMAKE_POLICY_VERSION_MINIMUM=3.5` when building the host on Windows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows GamepadManager via vigem-client (ViGEmBus) — the uinput-xpad analogue: one virtual Xbox 360 controller per client pad index, created lazily on first State. GameStream/Moonlight already uses the XInput conventions (low-16 button bits, sticks -32768..32767 +Y up, triggers 0..255), so the GamepadFrame->XGamepad mapping is 1:1. Replaces the non-Linux GamepadManager stub (same new/handle/pump_rumble API the m3 PadBackend drives, so no m3 change). Graceful when ViGEmBus is absent (gamepad disabled, session continues). Compiles clean on Windows + Linux; live-test needs the ViGEmBus driver + a physical pad. Rumble back-channel is a TODO (ViGEm notification API).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zero-copy capture->encode on the GPU via the raw NVENC API (nvidia_video_codec_sdk sys + ENCODE_API; the safe wrapper is CUDA-only). Opens an NV_ENC_DEVICE_TYPE_DIRECTX session on the SAME ID3D11Device as the DXGI capturer (carried on the new FramePayload::D3d11), registers a pool of BGRA textures once, CopyResources each captured texture in and encode_picture; CBR/ULL, infinite GOP, P-only, forced-IDR for RFI. The DXGI capturer gains a D3D11 zero-copy output (selected, like the encoder, by PUNKTFUNK_ENCODER=nvenc) so capture+encode share textures.
OFF by default (the nvenc feature pulls the NVENC SDK + cudarc): the default Windows host links without it (openh264 path). cudarc builds toolkit-less via the SDK ci-check feature (dynamic-loading). At link time --features nvenc needs nvencodeapi.lib (NVENC SDK, or an import lib generated from the driver's nvEncodeAPI64.dll) on PUNKTFUNK_NVENC_LIB_DIR. Both default and --features nvenc builds validated to compile+link GPU-less on the VM (import lib generated from the driver DLL). Runtime needs a real NVIDIA GPU.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows Capturer via DXGI Desktop Duplication: create a D3D11 device on the SudoVDA adapter (by LUID), find the matching output (by GDI name), DuplicateOutput, and per AcquireNextFrame copy the desktop into a CPU-readable staging texture -> tightly-packed BGRA (FramePayload::Cpu, feeds the openh264 software encoder GPU-lessly). Handles WAIT_TIMEOUT (reuse last frame) and ACCESS_LOST (re-duplicate). Adds FramePayload::D3d11(D3d11Frame) for the future NVENC zero-copy path, and a VirtualOutput.win_capture identity (adapter LUID + GDI name) carried out of the SudoVDA backend. Pure helpers (pack_luid/gdi_name_matches/depad_bgra) unit-tested on the VM; the live duplication path needs a real GPU + an activated SudoVDA monitor. Compiles clean on Windows + Linux.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows AudioCapturer via the wasapi crate (0.23): loopback the default render endpoint (Render device + Direction::Capture + shared mode => STREAMFLAGS_LOOPBACK) at 48 kHz stereo f32 with autoconvert, feeding the existing Opus path with no resampling. Dedicated COM-MTA thread owns the !Send WASAPI objects; interleaved f32 chunks leave over a bounded lossy channel; RAII Drop stops + joins. Bring-up handshake reports a missing endpoint as Err so a session continues without audio. open_audio_capture Windows factory arm + module. Init chain validated live on the VM (open succeeds; next_chunk waits on a silent system). Virtual mic deferred (no Windows virtual-audio endpoint). m3 audio_thread wiring + opus hoist land with the integration task.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows Encoder impl via the openh264 crate (statically-bundled, BSD-2): low-latency screen-content config (Baseline/no-B-frames, bitrate RC, BT.709 limited, near-infinite GOP + forced-IDR recovery via request_keyframe), packed CPU pixels (BGRx/BGRA/RGB/RGBA/RGBx/BGR) -> I420 -> AnnexB with in-band SPS/PPS each IDR. Synchronous: submit encodes immediately, poll hands back the one AU, flush is a no-op. Windows open_video factory selects it (PUNKTFUNK_ENCODER=software|nvenc|auto; NVENC arm lands later), H.264-only with a clear error otherwise, SW bitrate ceiling. Unit-tested live on the VM: synthetic BGRx -> AnnexB IDR + SPS NAL. Unblocks the GPU-less capture->encode->FEC->send pipeline. Compiles clean on Windows + Linux.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows InputInjector via SendInput (Win32 KeyboardAndMouse), mirroring the wlroots backend: absolute mouse (MOUSEEVENTF_VIRTUALDESK normalized to the virtual desktop), relative mouse, scancode keyboard (MapVirtualKeyExW + extended-key flagging), scroll (no sign flip — Windows wheel matches GameStream), buttons. Client already sends Windows VK codes (no keycode table). Reattaches the thread to the input desktop (OpenInputDesktop/SetThreadDesktop) to survive UAC/lock switches. New Backend::SendInput, the Windows auto-default in default_backend(), open() arm, windows-crate features. Compiles clean on Windows + Linux. Live injection validates with the in-session host run (SendInput is desktop-isolated from an SSH network logon).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows VirtualDisplay backend driving SudoVDA (the Apollo IDD) via its DeviceIoControl IOCTL protocol: open by interface GUID, ADD at the client's exact WxH@Hz (mode baked into the IOCTL, no EDID seeding), mandatory watchdog ping thread, QueryDisplayConfig name resolution, RAII Drop -> REMOVE. Wired behind the existing VirtualDisplay trait (open()/probe() Windows arms). Validated live on the GPU-less VM (standalone + via the trait, env-gated test): version 0.2.1, ADD 1920x1080@60 -> target, watchdog hold, REMOVE. Monitor activation into a WDDM path (-> capturable \\.\DisplayN) needs a real GPU and is deferred with capture/NVENC. docs/windows-host.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 of moving the library off a manual mgmt token: the management API now serves
over HTTPS with the host's persistent identity (the cert clients already pin) and
OPTIONAL client-cert auth. A request is authorized if EITHER the peer presented a
client certificate whose SHA-256 is in the punktfunk/1 paired store (the same trust the
QUIC data plane uses — so a paired native client needs no token), OR it carries the
bearer token (the web console / admin). `/health` stays open.
axum-server can't surface the peer cert to a handler, so `serve_https` runs the rustls
handshake itself (tokio-rustls), reads the verified peer certificate, and serves the
axum Router over hyper with the fingerprint attached to each request; `require_auth`
checks it against `NativePairing::is_paired`. The verifier reuses the GameStream
AcceptAnyClientCert, parameterized to make client auth optional (a browser with no cert
still completes the handshake and falls back to the token).
Validated live: paired cert → 200, unpaired cert / no creds / bad token → 401, bearer
→ 200, /health open. (Note: the API is now HTTPS with a self-signed cert — a browser
shows a one-time trust prompt; native clients pin by fingerprint.)
Next: Apple client presents its identity over mTLS (drops the token field); embed the
web console; enable HTTPS mgmt by default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codec negotiation (M2 polish):
- ServerCodecModeSupport now advertises what we encode: H264|HEVC|AV1_MAIN8
= 65793 (flags verified against moonlight-common-c Limelight.h). The old
placeholder 3843 wrongly claimed HEVC Main10 + 4:4:4 and no AV1. Main10
bits stay off on purpose: Moonlight ties 10-bit to HDR, and capture is
8-bit SDR BGRx with no HDR metadata path (av1_nvenc -highbitdepth was
validated working for later).
- RTSP ANNOUNCE: bitStreamFormat 0/1/2 -> H264/HEVC/AV1 (already plumbed to
av1_nvenc; validated e2e via `m0 --codec av1` + ffprobe av01), and a
dynamicRangeMode!=0 request now logs + falls back to 8-bit SDR.
Surround audio (M2 polish):
- ANNOUNCE x-nv-audio.surround.{numChannels,AudioQuality} +
x-nv-aqos.packetDuration -> per-session AudioParams; DESCRIBE advertises
all six Opus configs (normal before HQ per channel count). Normal-quality
mappings are pre-rotated for the client's GFE-order LFE swap
(RtspConnection.c, verified verbatim) so its derived decoder mapping
equals our encoder mapping — including 7.1, where Sunshine's rotate only
covers [3,6) and scrambles LFE/SL/SR.
- 5.1/7.1 encode via libopus multistream (audiopus_sys, the sys layer the
opus crate already links) with Sunshine's layouts/bitrates, RAII wrapper;
the live-validated stereo wire is byte-identical (plain Opus, no FEC).
- Surround sessions add Sunshine-style RS(4,2) audio FEC (packetType 127 +
AUDIO_FEC_HEADER, the OpenFEC parity matrix both ends hardcode, nanors
gemm semantics verified from nanors/rs.c).
- PipeWire capture generalized to the negotiated channel count with explicit
FL FR FC LFE RL RR [SL SR] positions; missing sink channels are zero-
filled by the channel-mixer. PwAudioCapturer now tears down cleanly on
Drop (pipewire channel -> loop quit), so a channel-count change can
reopen without leaking a capture stream.
Tests: serverinfo mask, RTSP codec/audio param parsing, DESCRIBE contents,
surround-params strings + client-swap round trip, FEC parity self-recovery
and packet layout, real-codec 5.1 channel-identity round trip, and an
ignored live test (ran green against a 6ch null sink monitor).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full project rename, decided 2026-06-10:
- Crates/binaries: punktfunk-core / punktfunk-host / punktfunk-client-rs.
- C ABI: punktfunk_* symbols, Punktfunk* types, include/punktfunk_core.h,
PUNKTFUNK_FEATURE_QUIC guard (header regenerated; cbindgen renames updated, incl.
PUNKTFUNK_BTN_*/PUNKTFUNK_AXIS_* wire constants).
- Protocol: punktfunk/1 — control-plane magic LMN1 → PKF1, nonce salt lmn1 → pkf1.
WIRE BREAK: clients must be rebuilt from this revision.
- Env knobs: PUNKTFUNK_VIDEO_SOURCE / PUNKTFUNK_COMPOSITOR / PUNKTFUNK_ZEROCOPY / ….
- Host config dir: ~/.config/punktfunk (the box's dir was migrated in place — the
persistent identity is unchanged, pinned fingerprints stay valid).
- Swift package: PunktfunkKit + PunktfunkCore.xcframework + PunktfunkConnection
(Sources/PunktfunkClient app + tests renamed with it); build-xcframework.sh updated.
- scripts/: 60-punktfunk.rules, punktfunk-host.service; OpenAPI doc regenerated.
Also: scripts/headless/run-headless-kde.sh — full headless Plasma bringup. Root cause of
"desktop but no apps/settings" over the stream: plasmashell launched without
XDG_MENU_PREFIX=plasma-, so the launcher resolved a nonexistent applications.menu and
rendered an empty menu. The script sets the complete KDE session env (menu prefix,
KDE_FULL_SESSION, session version) and rebuilds ksycoca before starting plasmashell.
Gate: 97/97 tests, clippy -D warnings (both feature sets), fmt, C-ABI harness PASS,
zero lumen references left outside .git.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler