A single DXGI cursor shape can need BOTH an alpha-blended layer AND a
screen-inverting (XOR) layer at once — a masked-color text I-beam (opaque
hot-spot + inverting bar) or a monochrome cursor mixing opaque and invert
pixels. The old path produced ONE BGRA image per shape and picked ONE blend
(cursor_invert) for the whole shape, so such mixed cursors rendered wrong
(masked-color opaque pixels forced through the invert blend; monochrome
(AND=1,XOR=1) invert pixels approximated as solid black).
Port Apollo/Sunshine's decomposition: convert_pointer_shape now returns a
CursorShape with optional alpha/xor layers; CursorCompositor holds tex_alpha
+ tex_xor and draw_layer renders each with its own blend (alpha = src-over,
HDR-scaled; XOR = inversion, unscaled — it operates on the framebuffer
reference). The CPU software path blends both layers too. Empty layers are
never uploaded or drawn. Removes the single cursor_invert flag.
Fixes#13 in docs/apollo-comparison.md. Independently reviewed (ship);
Windows-only code — compile verified by CI / dev VM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause (confirmed live: "black until I pressed a key, then the image came
back"): the secure desktop (lock/login/UAC) is STATIC, and DXGI Desktop
Duplication only emits a frame on CHANGE. On the normal→secure switch the
duplication is rebuilt (recreate_dupl / try_reduplicate), and we then SEEDED A
BLACK frame as last_present — which the static secure desktop never replaced
(no change-frame) until the user pressed a key. So we streamed black.
Fix: after rebuilding the duplication, CAPTURE the current desktop frame instead
of seeding black. A freshly-created duplication's first AcquireNextFrame returns
the full current desktop; grab it and present it. New `present_acquired` factors
the frame-processing out of `acquire`; both recovery paths now call it:
- recreate_dupl: after adopting the new duplication, acquire+present the real
frame (born-lost ACCESS_LOST / no-initial-frame → seed black as fallback and
let the 250ms-throttled caller retry — a brief flash, then real content).
- try_reduplicate: adopt-first, then capture its probe frame (was discarded).
Also (independently-correct safe fixes, per the adversarial review):
- DesktopWatcher computes the current desktop synchronously in start() before
returning, so a session that begins on the secure desktop (reconnect to a
locked box) doesn't relay one stale normal-desktop frame (the "flash").
- DuplCapturer::open reasserts SudoVDA isolation at open time (mirrors
recreate_dupl) — forces the secure desktop back onto the virtual output if a
lock/UAC re-attached a physical monitor.
- Instrumentation: dbg_black_seeds counter + a throttled warn when black is
seeded, and an info when a real secure-desktop frame is captured on recovery.
Pending: the user's real-lock smoke test on the 4090 (a headless PsExec
LockWorkStation runs as SYSTEM and can't lock an interactive session, so this
must be validated with an actual lock).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The capture-architecture reset from the research: add a Windows.Graphics.Capture (WGC) backend that
captures the COMPOSED desktop — including the overlay/independent-flip/MPO planes DXGI Desktop
Duplication misses — which structurally fixes the frozen HDR animations + video (proven live: a WGC
frame decodes to the real 5120x1440 HDR content DDA freezes on). It reuses the whole pipeline
unchanged: the WGC frame's GPU texture → same scRGB→BT.2020-PQ shader → NVENC zero-copy; the OS
composites the cursor (IsCursorCaptureEnabled) so no manual cursor pass. crates/punktfunk-host/src/
capture/wgc.rs; find_output/make_device/HdrConverter/nudge_cursor_onto made pub(crate) for reuse.
Reliability findings + mitigations (live on the RTX 4090):
- WGC can't activate under the SYSTEM account (0x80070424) — it needs the interactive user token. The
host must run as the user for WGC (run.cmd: drop PsExec -s). DDA still needs SYSTEM for the secure
desktop — that token reconciliation (impersonation) is the remaining task.
- WGC's Direct3D11CaptureFramePool::CreateFreeThreaded intermittently HANGS on the headless SudoVDA
(IddCx) display, correlated with accumulated SudoVDA churn (failed REMOVEs leaving lingering
displays); clean-state opens reliably. Since it's a blocking hang, capture_virtual_output runs WGC
open on a watchdog thread with a 5s timeout and falls back to DDA on hang/error — the session is
NEVER left black: WGC when it opens (fixed animations), DDA otherwise. First-frame nudge added (WGC
fires FrameArrived on change; a static desktop otherwise never delivers the first frame).
- Default WGC; PUNKTFUNK_CAPTURE=dda forces DDA. DDA path unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The HDR path produced a constant ACCESS_LOST churn during real desktop activity (window resize /
Start menu / DWM transitions): the duplication keeps getting invalidated but the OUTPUT stays valid
(probe passes — 0 born-lost over 72 rebuilds). The old recovery did a FULL rebuild (new device +
factory) on every loss, which re-inits NVENC + seeds black + was throttled to 4x/s → mostly-frozen,
re-init churn = "broken animations".
Now recovery is tiered (mirrors Sunshine): try_reduplicate() does a fresh DuplicateOutput on the
EXISTING device+output — no new device, so NO encoder re-init, NO black seed, gpu_copy/HDR
textures/last_present kept → frames resume immediately. Only a genuine output loss (secure-desktop
switch) or a dead device (DEVICE_REMOVED/RESET) falls back to the full, throttled recreate_dupl.
Both paths probe the new duplication and reject a born-lost one.
Validated synthetically (1080p60 + 5120x1440@240 HDR): pipeline stable, 0 churn, frames flow. The
real-desktop churn needs live validation (can't synthesize DWM animations). Secure-desktop "UI never
appears in-session" is a separate issue (output gone in-session; only a fresh monitor re-add works) —
still open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- HDR cursor: sRGB→linear decode + scale to HDR graphics white (PUNKTFUNK_HDR_CURSOR_NITS, default
203 per BT.2408) in the FP16 cursor composite, so it's no longer ~2.5x too dim. SDR path unchanged;
the masked-color (I-beam) inversion blend left unscaled. Cursor cbuffer widened 16→32 + bound to PS.
(Validated live: cursor now correct brightness in HDR.)
- Secure-desktop recovery: recreate_dupl now PROBES the rebuilt duplication with a 50ms
AcquireNextFrame and only adopts it when live (Ok/WAIT_TIMEOUT); a born-lost one (immediate
ACCESS_LOST) is dropped so the caller repeats the last frame + retries. Plus reassert_isolation()
re-detaches physical displays on every recovery (re-routing the secure/HDR desktop to the virtual
output, the delta a fresh reconnect has). NOTE: the born-lost ACCESS_LOST storm in HDR is NOT yet
resolved by these — still under investigation (animations/secure-UI/cursor-trail in HDR remain).
- docs/windows-client-bootstrap.md: handoff for the native Windows Rust client (windows-rs Reactor +
WinUI 3 SwapChainPanel, D3D11VA decode, WASAPI audio, SDL3 input; ports crates/punktfunk-client-linux;
10-bit/HDR present; dev boxes + gotchas).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds true HDR (BT.2020 PQ) and 10-bit (HEVC Main10) streaming, negotiated so an
8-bit/SDR client is never sent a stream it can't decode, plus a robust fix for the
capture losing the stream across a secure-desktop transition.
Protocol (punktfunk-core/quic.rs):
- Hello gains `video_caps` (VIDEO_CAP_10BIT / VIDEO_CAP_HDR), Welcome gains `bit_depth`,
both as optional trailing bytes (back-compat). client-rs advertises 10-bit via
PUNKTFUNK_CLIENT_10BIT; the connector advertises 0 for now (in-band detection drives
the native clients). Regenerated punktfunk_core.h.
Windows host:
- 10-bit Main10: host enables it only when the client advertised VIDEO_CAP_10BIT AND
PUNKTFUNK_10BIT is set; threaded through open_video → NVENC (profile Main10,
pixelBitDepthMinus8).
- HDR: when the captured desktop is scRGB FP16 (R16G16B16A16_FLOAT, HDR on), copy it to
an FP16 surface, composite the cursor there, convert scRGB → BT.2020 PQ 10-bit
(R10G10B10A2) via a shader, and encode HEVC Main10 with the BT.2020/PQ colour VUI
(ABGR10 input). Fixes the freeze + cursor-trail that came from feeding FP16 into the
BGRA path. Reacts dynamically to the HDR toggle.
- Capture recovery: rebuild is now a single NON-BLOCKING attempt, throttled to ~4×/s,
repeating the last good frame between attempts (format-tagged last_present). During a
secure-desktop dwell SudoVDA's output is gone; the old blocking 12 s retry starved the
send loop for seconds so the client timed out and disconnected — now the session stays
fed (frozen) until the desktop returns. Also seeds a black frame on recovery.
Apple client (PunktfunkKit):
- Detects HDR in-band from the stream VUI (PQ transfer function), decodes to 10-bit P010,
and presents via an rgba16Float + BT.2020 PQ CAMetalLayer with EDR; SDR path unchanged.
Switches automatically on a mid-session HDR toggle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live-validated Mac <-> RTX 4090 at the display's native 5120x1440@240:
- Resolution: set_active_mode enumerates the IDD's advertised modes and sets the
requested resolution at the best supported refresh (keeps 5120x1440@240; no more
silent fallback to the 1080p OS default when an exact mode is briefly unavailable).
- Bitrate auto-cap: NVENC init probes and steps the average bitrate down to the GPU's
codec-level max so a high client bitrate connects (matches the Linux host; we do not
split NVENC sessions).
- Mouse cursor: DXGI duplication excludes the HW cursor; capture the pointer
shape/position (GetFramePointerShape) and GPU-composite it before NVENC. Color cursors
alpha-blend; masked-color (the text I-beam) uses an INV_DEST_COLOR inversion blend so
the caret inverts the screen and shows on any background (no black box); monochrome
handled too.
- Secure desktop (lock / login / UAC): run as SYSTEM in the interactive session, follow
the input desktop via SetThreadDesktop, and on the WinSta switch recreate the D3D11
device and re-resolve the virtual output's GDI name from the stable SudoVDA target id
(the name changes across the topology rebuild; the old failure hunted the stale
\\.\DISPLAYn and dropped). ACCESS_LOST / INVALID_CALL / device-removed are recoverable,
and a mid-stream resolution change is followed (capturer + NVENC re-init at the new
size). isolate_displays detaches other monitors so Winlogon renders to the virtual
output. One real session recovered 1012 desktop switches and completed cleanly.
Windows-only backends; Linux/macOS unaffected. Builds clean on x86_64-pc-windows-msvc.
Deployment (windowless SYSTEM launch via PsExec + hidden VBScript) documented in
docs/windows-host.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated live on an RTX 4090 (Windows 11) host streaming to the Rust
reference client over the LAN: SudoVDA virtual display → DXGI Desktop
Duplication (D3D11 zero-copy) → NVENC HEVC → punktfunk/1. 720p60 and
1080p60 both clean (181 / 177 frames, 0 mismatched, p50 1.6 / 3.45 ms
cross-machine), coexisting with Apollo. Two real-hardware bugs the
GPU-less VM couldn't surface:
- DXGI capturer: the SudoVDA virtual monitor's DXGI output is enumerated
under the GPU that *renders* it (the 4090, LUID 0x15df6), NOT under the
SudoVDA "adapter" LUID SudoVDA reports (0x23276). Restricting the output
search to that LUID found nothing → "adapter has no output named
\\.\DISPLAYn". Now search ALL adapters for the GDI name, bind the D3D11
device to whichever adapter exposes it (NVENC then shares that device),
with a settle-retry (the output appears a beat after display creation)
and topology logging.
- native_pairing / apps: keyed config paths off raw $HOME, which a Windows
service/scheduled-task context doesn't set → "HOME unset" hard-fail at
m3-host startup. Route both through gamestream::config_dir(), which falls
back to %APPDATA% on Windows (cert/paired/apps now under AppData\Roaming).
clippy -D warnings + build green on x86_64-pc-windows-msvc (default and
--features nvenc) and Linux (78/78 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the host→client rumble path on Windows, the analogue of the Linux
uinput EV_FF read loop: a game's force-feedback on the virtual Xbox 360
pad is delivered by ViGEm's notification API (`request_notification` →
`spawn_thread`, gated by the crate's `unstable_xtarget_notification`
feature). A per-pad background thread stores the latest motor levels;
`pump_rumble` relays changes to the client on the universal 0xCA plane
(motors scaled 0..255 → 0..65535). Dropping the target aborts the
notification, so the thread exits with the session. Live verification
still needs a physical pad.
Also fix the Windows backends' clippy debt — these modules are cfg-
excluded from Linux CI, so `clippy -D warnings` never saw them, and the
VM's rustc 1.96 clippy is stricter on shared code than the CI image:
- dxgi: manual checked division → checked_div().map_or
- sendinput: `x = x | y` → `x |= y`
- sudovda: `.then(|| ptr)` → `.then_some(ptr)`
- m3 pick_compositor: drop the needless early return (match form)
- m3 resolve_compositor: Windows arm is a tail expr, not `return`
All Windows backends now build + clippy clean (default and --features
nvenc); Linux unaffected (fmt/clippy/check green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zero-copy capture->encode on the GPU via the raw NVENC API (nvidia_video_codec_sdk sys + ENCODE_API; the safe wrapper is CUDA-only). Opens an NV_ENC_DEVICE_TYPE_DIRECTX session on the SAME ID3D11Device as the DXGI capturer (carried on the new FramePayload::D3d11), registers a pool of BGRA textures once, CopyResources each captured texture in and encode_picture; CBR/ULL, infinite GOP, P-only, forced-IDR for RFI. The DXGI capturer gains a D3D11 zero-copy output (selected, like the encoder, by PUNKTFUNK_ENCODER=nvenc) so capture+encode share textures.
OFF by default (the nvenc feature pulls the NVENC SDK + cudarc): the default Windows host links without it (openh264 path). cudarc builds toolkit-less via the SDK ci-check feature (dynamic-loading). At link time --features nvenc needs nvencodeapi.lib (NVENC SDK, or an import lib generated from the driver's nvEncodeAPI64.dll) on PUNKTFUNK_NVENC_LIB_DIR. Both default and --features nvenc builds validated to compile+link GPU-less on the VM (import lib generated from the driver DLL). Runtime needs a real NVIDIA GPU.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows Capturer via DXGI Desktop Duplication: create a D3D11 device on the SudoVDA adapter (by LUID), find the matching output (by GDI name), DuplicateOutput, and per AcquireNextFrame copy the desktop into a CPU-readable staging texture -> tightly-packed BGRA (FramePayload::Cpu, feeds the openh264 software encoder GPU-lessly). Handles WAIT_TIMEOUT (reuse last frame) and ACCESS_LOST (re-duplicate). Adds FramePayload::D3d11(D3d11Frame) for the future NVENC zero-copy path, and a VirtualOutput.win_capture identity (adapter LUID + GDI name) carried out of the SudoVDA backend. Pure helpers (pack_luid/gdi_name_matches/depad_bgra) unit-tested on the VM; the live duplication path needs a real GPU + an activated SudoVDA monitor. Compiles clean on Windows + Linux.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler