feat(windows): pf-vdisplay IDD-push — HDR + pipelined zero-copy capture
apple / swift (push) Successful in 1m4s
windows-host / package (push) Successful in 6m28s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m14s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m10s
release / apple (push) Successful in 7m53s
android / android (push) Successful in 10m33s
ci / web (push) Successful in 44s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 3m4s
ci / docs-site (push) Successful in 53s
ci / rust (push) Successful in 12m22s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m11s
apple / screenshots (push) Successful in 5m24s
deb / build-publish (push) Successful in 3m16s
decky / build-publish (push) Successful in 21s
ci / bench (push) Successful in 4m42s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 27s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m34s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m42s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m13s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 47s
flatpak / build-publish (push) Successful in 4m24s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m5s
docker / deploy-docs (push) Successful in 25s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 7m44s

HDR (display-driven, matching the WGC path):
- CTA-861.3 HDR EDID (BT.2020 primaries + HDR Static Metadata block) so Windows
  offers "Use HDR" on the virtual display. The host FOLLOWS the display's live
  advanced-color state, recreating the shared ring at the matching format
  (FP16 in HDR / BGRA in SDR) on a toggle — no freeze.
- Always emit Main10/BT.2020-PQ Rgb10a2 while the display is HDR; the client
  auto-detects PQ from the HEVC VUI (clients under-report VIDEO_CAP_10BIT).
  Generic HDR10 mastering SEI on every IDR.
- Generation-tagged `latest` (gen<<40|seq<<8|slot) + driver `is_stale` re-attach
  kill the toggle-time garbage frame and any stale-ring read.

Perf:
- Pipeline the encode loop (Capturer::pipeline_depth; IDD-push = 2): submit N+1
  before polling N so the convert/copy on the 3D engine overlaps the NVENC encode
  of N on the ASIC. PUNKTFUNK_IDD_DEPTH overrides (1 = synchronous).
- Rotating host output ring (OUT_RING) so the in-flight encode and the next
  convert never touch the same texture.
- HDR converts directly from the keyed-mutex slot's SRV into the output ring
  (drops the redundant slot->fp16 scratch copy); SDR copies the BGRA slot in.
  The slot mutex is held only across the convert/copy, not the encode.
  RING_LEN 3->6 for publish headroom.
- Capture-health diagnostic: new_fps vs repeat_fps under PUNKTFUNK_PERF (a low
  new_fps at a high send rate means the source isn't compositing, not an encode
  stall).

Validated live on the RTX box: 5120x1440@240 HDR streams; driver composes
~180 new fps, encode 240 fps @ ~4.3 ms p50.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-24 00:35:52 +02:00
parent c5dab484df
commit e2c9bfd3d9
26 changed files with 2962 additions and 313 deletions
+219
View File
@@ -115,6 +115,225 @@ binary keeps running (silently). **Devnode hygiene:** create the root devnode wi
on every `pnputil /add-driver` (they have `hwid root\pf_vdisplay`, so the driver install re-materializes
them). The production installer must use a single `nefconc`/INF-created node and never `devgen`.
## P2 — direct frame push (kill DDA): design & decision record
Status: **in progress.** P1 ships frames the old way (the driver drains its swap-chain and DDA/WGC
re-captures the composited desktop). P2 makes the driver *publish* each swap-chain frame to the host
directly, so we can retire Desktop Duplication and its multi-GPU survival code. Built behind
`PUNKTFUNK_IDD_PUSH`, A/B'd against DDA, and only then made the default.
### The decisive finding: producer and consumer are both in Session 0
The whole transport design hinged on one unknown — same-session or cross-session? **Measured on the
RTX box (2026-06-22):** the pf-vdisplay host process is `WUDFHost.exe` with
`-DeviceGroupId:pfVDisplayGroup`, running in **Session 0**; the punktfunk host service is `LocalSystem`,
also **Session 0**. So the swap-chain processor thread (spawned by our own `thread::spawn` inside the
driver, i.e. in `WUDFHost`) and the encoder live in the **same session**. This is the easy case:
- A D3D11 **shared keyed-mutex texture** created in the driver can be opened by name in the host with
`ID3D11Device1::OpenSharedResourceByName` — both devices created on the **same render-adapter LUID**
(which the driver already reports out of the `ADD` IOCTL via `OsAdapterLuid`, surfaced as
`WinCaptureTarget::adapter_luid`).
- Named kernel objects resolve through Session 0's shared `\BaseNamedObjects`, so **no `Global\`
prefix / `SeCreateGlobalPrivilege` gymnastics** are needed (kept the names unprefixed; documented
that this relies on both processes being Session 0). The Looking-Glass cross-*VM* shared-memory
device is unnecessary — this is cross-*process*, same-session, on one GPU.
This collapses the "Session-0 cross-process transport is the long pole" risk from the original plan.
### Transport: a ring of shared keyed-mutex textures + a metadata header + an event
A single ping-pong keyed mutex would couple the driver's present rate to the host's consume rate — and
**the swap-chain thread must never block** (a stalled `IddCxSwapChainReleaseAndAcquire`/processing loop
freezes DWM compositing system-wide). So, the Looking-Glass shape — multiple frame buffers, newest
wins:
- **Ring** of `N` (default 3) shared textures, `RESOURCE_MISC_SHARED_NTHANDLE |
SHARED_KEYEDMUTEX`, fixed size for the session. A **generation** counter bumps on a mode change
(resize): the driver tears down + recreates the ring at the new size, the host notices the
generation change and re-opens.
- **Named metadata header** (`CreateFileMapping`): `{magic, version, generation, width, height,
dxgi_format, ring_len, latest}` where `latest` packs `{write_index, monotonic sequence}` published
*after* the copy completes. Plain (unprefixed) names — Session-0 shared namespace.
- **Frame-ready auto-reset event** so the consumer waits instead of spinning.
- **Producer (driver, per acquired frame):** pick `(latest_index + 1) % N`; **try**-acquire that
slot's keyed mutex with a 0 ms timeout (if the host still holds it — rare with 3 slots — reuse the
current slot or skip, **never block**); `CopyResource` the acquired `MetaData.pSurface` into the
slot; release the mutex; publish `{index, ++seq}`; `SetEvent`. Then `FinishedProcessingFrame` as
today.
- **Consumer (host `IddPushCapturer`):** `WaitForSingleObject(event, timeout)`; read `latest`; if `seq`
advanced, acquire that slot's mutex, `CopyResource` into an owned NVENC-input texture, release, yield
`FramePayload::D3d11{texture, device}` — straight into the existing zero-copy NVENC path. No DDA, no
CPU readback.
### What P2 removes vs. keeps
- **Removes:** `capture/dxgi.rs`'s `DXGI_ERROR_ACCESS_LOST`/`MODE_CHANGE_IN_PROGRESS` re-duplication
churn, the legacy-`DuplicateOutput` fallback, and **`install_gpu_pref_hook()` (the `win32u.dll`
patch)** — by **pinning the render adapter to the encoder GPU** (`IddCxAdapterSetRenderAdapter`, the
existing `SET_RENDER_ADAPTER` IOCTL, driven before `ADD`), so the OS never reparents the output and
the shared texture + NVENC share one device by construction.
- **Keeps:** display **topology** (making the virtual display the composited desktop) and the
**watchdog** (now ours). The **two-process WGC secure-desktop relay** stays until we confirm the IDD
push also delivers the secure (Winlogon) desktop; if it does, that retires too.
### On-glass attempt 2026-06-22 — code complete, blocked at driver load
The full transport (driver publisher + host `IddPushCapturer` + render-LUID robustness + in-process
routing) is written and compiles clean. The first on-glass A/B exposed several real things and one
hard blocker:
- **The service captures in a Session-1 WGC helper, not in-process.** `should_use_helper()` returns
true for a SYSTEM service, so it spawns a user-session helper that does capture **and input
injection**. IDD-push must capture **in-process in Session 0** (where the driver publishes) — wired
via `should_use_helper()` returning false for `PUNKTFUNK_IDD_PUSH`. **Caveat:** `SendInput` from
Session 0 can't reach the user's Session-1 desktop, so in-process IDD-push has **no working input**
yet. Production needs either a Session-1 input-only helper, or `Global\`-namespaced shared textures
so a Session-1 helper consumes IDD-push for both video + input.
- **`SET_RENDER_ADAPTER` is ignored by the driver** (the IDD lands on a different adapter than pinned:
observed IDD adapter `0xd60722` vs pinned 4090 `0x15de1`). The render-LUID-in-header path makes the
host bind correctly regardless, but the driver should be made to actually honor the pin (or the host
must copy across adapters) so NVENC gets a 4090 surface.
- **Cursor is included** in the IddCx composited frame (DDA strips it) — so the host-side cursor
compositor (P2.5) is likely unnecessary for this path.
- **`FAILED_POST_START` was a red herring (churn, not the binary).** Comparing the 2157 (works) and
the `frame_transport` DLL import tables: **identical** (same 8 DLLs; the size/hash delta is just the
Authenticode signature). A clean install **+ reboot** (no `restart-device`/`disable-enable`/kill in
between) loads the `frame_transport` driver to **`OK`**. The earlier `FAILED_POST_START` was the
device wedging from the hot-reload churn (the deploy gotchas above). **Lesson: deploy = install +
reboot, full stop.**
- **THE REAL BLOCKER — the driver can't CREATE the shared objects.** With the driver loaded clean and
the monitor active, the host's `IddPushCapturer` still times out: `pfvd-hdr-<target> never appeared`.
The driver's own `OutputDebugString` is invisible (UMDF redirects it to ETW, not DebugView — verified
with a working DBWIN self-test), so a **file-logging** driver build was tried — and it wrote **no
file at all**, even though `init()` runs in `DriverEntry`, the device is `OK`, WUDFHost runs as
`LocalService`, and `C:\Users\Public` is world-writable. **WUDFHost runs with a restricted token: it
can neither write the filesystem nor create named kernel objects** (`CreateFileMappingW`/`CreateEventW`/
`CreateSharedHandle`), so `FramePublisher::new` fails silently. This is exactly why the **gamepad UMDF
drivers invert it**: `inject/dualsense_windows.rs` — *"the host creates the section (privileged → a
permissive SDDL so the WUDFHost can open it); the driver maps it"* — `Global\pfds-shm-<idx>` + SDDL
`D:(A;;GA;;;WD)`. **Fix: invert frame-push to match.** The HOST creates the header + event + ring
textures (`Global\` names, `D:(A;;GA;;;WD)` SDDL); the DRIVER only OPENS them, writes its actual
render LUID + a status code back into the host-created header (so we get driver visibility through the
host log), and runs the copy loop. The host creates the textures on the render adapter the driver
reports.
- **Also unresolved: `SET_RENDER_ADAPTER` appears ignored** (the host's pin to the 4090 vs the ADD-reply
adapter differ every time). The inverted header carries the driver's *actual* render LUID so the host
can create textures + run NVENC on the right adapter — but if that's the iGPU, NVENC (NVIDIA) can't
encode it, so the driver must be made to honor the pin (or the host must cross-adapter copy). Needs its
own investigation.
**Driver deploy gotchas learned (this box):** hot-reloading a UMDF display driver is unreliable —
`pnputil /restart-device` does NOT restart WUDFHost (old image stays mapped), `Disable/Enable-PnpDevice`
errors on the root-enumerated IDD, and **killing WUDFHost invalidates the host's cached `{e5bcc234}`
control handle** (every ADD then fails `0x80070006`, and the device can wedge to `FAILED_POST_START`).
A **reboot** loads a freshly-installed build cleanly. **Recovery** from a broken build is clean and
reboot-free: `pnputil /delete-driver <oemNN>.inf /uninstall` removes the bad package and the device
rebinds the previous (validated) package in the DriverStore — restored 2157 → `OK` immediately.
### On-glass attempt 2 (2026-06-23) — inversion works; in-process Session-0 path is a dead end
Implemented the **inversion** (host creates the header + event + ring textures with the
`D:(A;;GA;;;WD)` SDDL, driver only opens them) + a per-attempt **generation** (kills the
`DXGI_ERROR_NAME_ALREADY_EXISTS` retry collisions) + a fixed-name **`Global\pfvd-dbg` debug channel**
(structured counters the driver writes, since UMDF/ETW + the restricted token block its other logs).
Results on the RTX box:
- ✅ The host **creates the shared ring every time** (`created shared ring … render_luid=…`) — the
privileged-create / restricted-open split is sound.
- ✅ No more name collisions (generation fix).
-**The driver writes NOTHING** — debug block all zeros, crucially `run_core_entries=0`. The
swap-chain processor **never runs**, i.e. the OS **never assigns a swap-chain** to the virtual
monitor in this path.
**Root cause: an IddCx monitor only gets a swap-chain when something PRESENTS to it, and the in-process
path has no presenter.** The host + the CCD topology-isolate run in **Session 0, which has no DWM /
compositor**. The WGC path works because its capture helper lives in **Session 1**, where DWM composes
the desktop onto the display (that composition is the swap-chain trigger). So in-process Session-0
IDD-push gets no frames to push, full stop — a **fundamental** barrier, not a fixable bug. The original
plan's "Session-0 transport is the long pole" was right, but the long pole turned out to be *triggering
presentation*, not the shared-memory mechanics (those work).
**Consequence:** the only viable IDD-push shape is **option 3 — a Session-1 helper drives presentation +
consumes the `Global\` ring** (the inversion built here is exactly what it needs). But it carries an
unretired risk: it's still unproven whether the swap-chain gets assigned even with a Session-1 consumer
that isn't WGC. Until that's answered, **DDA/WGC stays the shipping Windows capture path** — it works.
All the IDD-push code (driver open-side + host create-side + debug channel) is written, compiles, and is
gated behind `PUNKTFUNK_IDD_PUSH` (off), so it's dormant and harmless.
### CONCLUSION (2026-06-23): IDD-push is not viable for bare-metal capture — the swap-chain is never assigned
After the inversion + a fixed-name debug channel + a host-created-ring observer + an autonomous
loopback test harness (`punktfunk-probe` → the SYSTEM service, paired via the mgmt API), the question
"does the driver's swap-chain processor ever run?" was answered **definitively: no.** The driver's
`run_core` is **never entered**`run_core_entries=0` in *every* configuration tested:
- in-process (Session 0) and WGC-triggered (Session 1 helper) sessions,
- a user-created ring AND a host-created (LocalSystem) ring with a permissive `D:(A;;GA;;;WD)` SDDL,
- with and without a Low-IL (`S:(ML;;NW;;;LW)`) mandatory label,
- with WUDFHost confirmed **not** an AppContainer (`IsAppContainer=0`),
— even while WGC simultaneously captured the same virtual monitor's composition and streamed multi-MB
of HEVC. The gamepad UMDF drivers prove a UMDF driver *can* open + write a host-created `Global\`
section on this box, so the driver writing nothing is **not** an access problem — `run_core` simply
does not run.
**Root cause (researched + ecosystem-confirmed):** an IddCx virtual monitor only receives a swap-chain
(`EVT_IDD_CX_MONITOR_ASSIGN_SWAPCHAIN`) when the OS **presents/scans-out** to it, which requires a real
presentation consumer. **WGC/DDA capture of the composed desktop does NOT count** — it reads DWM's
composition, bypassing the driver's swap-chain. With no physical scanout and no consumer that routes
*through the driver*, the path stays inactive (`IDDCX_PATH_FLAGS=0`) and `ASSIGN_SWAPCHAIN` never fires.
Confirming evidence:
- **Every bare-metal virtual-display capture project uses WGC/DDA, not the driver swap-chain:** SudoVDA
(its swap-chain loop acquires-and-discards), Apollo/Sunshine (DDA + WGC backends), virtual-display-rs
(discards), parsec-vdd (no frame path). Only **Looking Glass** consumes the driver swap-chain — and
only because a **VM guest scans out** the display (the consumer). We have no equivalent on bare metal.
- Microsoft's own unanswered Q&A (learn.microsoft.com/answers 4096179) reports the identical symptom for
the IddSampleDriver: virtual display "always inactive," `ASSIGN_SWAPCHAIN` never runs.
**Verdict:** the "driver consumes its swap-chain and pushes frames" architecture (P2 / Looking-Glass
style) **cannot get frames** for punktfunk's bare-metal, whole-desktop, capture-only use case. The
shared-memory transport machinery (host-creates / driver-opens, the gamepad pattern) is all sound and
proven to *create*, but there is nothing for the driver to publish. **DDA/WGC remains the only viable
Windows capture path**, which is exactly what the entire ecosystem does. The IDD-push code stays
in-tree, compiles, and is gated `off` (`PUNKTFUNK_IDD_PUSH`) — dormant and harmless — documenting the
attempt so it isn't re-tried. "Better performance/lower overhead" must come from optimizing the WGC/DDA
path (e.g. trimming the Session-0↔Session-1 relay, zero-copy encode), not from IDD-push.
The only unexplored avenue is **driver-side** (a different adapter/monitor/path setup that might make the
OS treat the virtual display as a presentation target) — but it needs a reboot to test, the MS Q&A
suggests it's unsolved, and the unanimous ecosystem choice of WGC/DDA argues it's a dead end.
**Final exhaustion (2026-06-23, follow-up): both remaining avenues closed.**
- **Option 3 (present source) — TESTED, failed.** Added a present-trigger to the Session-1 WGC helper:
it successfully created a D3D11 swapchain on the virtual display and presented continuously (WGC even
captured the flashing window). The driver stayed `run_core_entries=0` / `frames_acquired=0`. So an
active *present source* on the display does NOT make the OS assign the driver's swap-chain either —
DWM composes the present onto the display (capturable) without routing it through the driver's
swap-chain.
- **Option 2 (driver flag) — closed by analysis.** The present-trigger succeeding proves the **path is
already active** (a swapchain presents to the display fine); the missing piece is **scanout routed
through the driver**, which the OS does only for a real consumer (physical display / VM guest / RDP).
The one IddCx flag for that — `IDDCX_ADAPTER_FLAGS_REMOTE_SESSION_DRIVER` — requires the **RDP
protocol stack** as the consumer, which bare-metal console capture has no equivalent of.
**Verdict is final:** IDD-push needs a presentation consumer (scanout / VM guest / RDP) that bare-metal
console desktop-capture fundamentally cannot provide. No host-side capture, no in-process path, no
present source, and no available driver flag overcomes it. WGC (normal desktop) + DDA (secure desktop)
is the only viable Windows capture path — as the entire ecosystem already does. The IDD-push +
present-trigger code stays in-tree, gated off, as the documented record of the attempt.
### Known gaps the build-out must close (tracked as P2.* tasks)
- **Cursor.** DDA/WGC composite the HW cursor host-side from frame-info; the IDD path delivers the
cursor separately (`IddCxMonitorSetupHardwareCursor` event → `QueryHardwareCursor`). The prototype
may ship cursor-less; the build-out wires the IDD cursor into the existing `CursorCompositor`.
- **HDR.** The default IddCx swap-chain surface is 8-bit `B8G8R8A8`; FP16/HDR needs the **IddCx 1.11
D3D12 acquire path** (`SetDevice2`/`ReleaseAndAcquireBuffer2``ID3D12Resource`). Build against
1.10, runtime-gate 1.11. SDR-only for the prototype.
## Why we'd do this
The user's goals, mapped to outcomes: