unom/punktfunk
11
0
Fork 5
Code Issues 14 Pull Requests 39 Actions 30 Packages Projects 1 Releases 60 Wiki Activity

fix(vdisplay/encode): feed NVENC packed RGB for HDR 4:4:4 #599

Open
enricobuehler wants to merge 2 commits from fix/vdisplay-hdr-444 into wp/vp-p4-drain-clock
pull from: fix/vdisplay-hdr-444
merge into: :wp/vp-p4-drain-clock
Conversation 0 Commits 2 Files Changed 9
+443 -114
enricobuehler commented 2026-09-03 10:50:44 +00:00
Owner
Copy Link
Copy Source

HDR combined with 4:4:4 regressed at the cutover, and the driver told the host otherwise. Found while tracing what gate 3's 4:4:4 parity clause could actually be measured with.

What was wrong

spec_for mapped every HDR session to a 10-bit planar input, consulting chroma444 only for the PyroWave arm. NVENC then computed want_444 = chroma_444 && full_chroma_input, which is false for a 10-bit planar input, so it encoded 4:2:0.

The interesting part is the lie, and it is not quite what it looked like. NVENC does own an input-format gate — "4:4:4 negotiated but the capturer delivered subsampled YUV" — but it runs at the first submit. open() was a pure struct build that ignored its _format argument, so caps(), which the driver reads immediately and returns in the SET_ENCODE reply before any frame exists, echoed the request back. The check was not missing; it ran too late to reach the reply. The host's own mismatch warning at stream.rs:3898 was being silenced by that echo, and NVENC's later complaint never left the driver host process.

The fix

Honest capabilities, at the point they are computed. NvencD3d11Encoder::open now uses its format argument to seed buffer_fmt and to clear the effective chroma_444 when that format is not packed RGB. open and submit share two new helpers so they cannot drift apart again. chroma_444_requested keeps the negotiated value, so a later RGB re-initialisation recovers 4:4:4 exactly as before. That fixes it for every caller of pf-encode-win, not only the driver.

The capability is restored, not just declared absent. EncodeInput::Rgb10 drives the HdrRgb10Converter that was already in the tree and unused, into R10G10B10A2_UNORM pool slots with BIND_RENDER_TARGET — the same bind flags the pre-cutover out-ring used, checked against the commit before the cutover. Cursor blend, deferred scratch and fence paths needed no change, since Rgb10 reads FP16 and lands in the existing deferred-scratch arm.

Restricting packed RGB to NVENC narrows nothing: the handshake only ever asks 4:4:4 of NVENC or PyroWave.

One structural call worth reviewing

The input-kind decision moved into pf-driver-proto::encode::EncodeInput (choose plus full_chroma), with the driver's InputKind now an alias. That is what lets the test run on macOS, and it puts the choice next to the reply it has to agree with rather than two crates away.

Probe

EncodeProbeRequest::flags gains HDR and 4:4:4 bits — no wire change, flags was reserved. spec() now calls the same InputKind::choose that SET_ENCODE calls, so the probe exercises the shipping decision instead of a copy, which is why the old probe could never have caught this: it hard-coded 8-bit 4:2:0. The open line logs the chosen input and the resulting chroma_444. Fixed in passing: the probe reply hard-coded backend_opened = 1, so it always claimed NVENC.

Test

a_444_request_picks_a_full_chroma_input — a table over backend, HDR and 4:4:4 pinning the chosen input, plus a loop asserting that both backends the host negotiates 4:4:4 for land on a full-chroma input at both depths. The HDR-plus-4:4:4 row is the regression. It runs on macOS: 52 tests pass in pf-driver-proto.

CI, and a correction

I claimed in #594 that the clippy fix turns windows-drivers.yml green. That was wrong, and this branch proved it. A dispatch of that workflow fails 40 seconds in, before any Rust compiles, in pyrowave-sys's CMake with a path-too-long error, because the runner's checkout root pushes a tlog past the Windows path limit — exactly the hazard the workflow's own comment predicted. The same job fails identically on the base branch, so this lane is red on the whole stack for an infrastructure reason, not a code one. The clippy error was real and is still worth fixing; it simply was not what that lane was dying on.

The driver therefore builds by hand instead: all four CI steps run in a short path on the build box, every exit code 0.

Residual

caps().chroma_444 still cannot reflect the GPU's own HEVC 4:4:4 support at driver open, because that is only known after a device exists. The host already gates it before the handshake, so the only exposure is a multi-GPU box resolving a different adapter host-side than driver-side.

HDR combined with 4:4:4 regressed at the cutover, and the driver told the host otherwise. Found while tracing what gate 3's 4:4:4 parity clause could actually be measured with. ## What was wrong `spec_for` mapped **every** HDR session to a 10-bit planar input, consulting `chroma444` only for the PyroWave arm. NVENC then computed `want_444 = chroma_444 && full_chroma_input`, which is false for a 10-bit planar input, so it encoded 4:2:0. The interesting part is the lie, and it is not quite what it looked like. NVENC **does** own an input-format gate — "4:4:4 negotiated but the capturer delivered subsampled YUV" — but it runs at the first `submit`. `open()` was a pure struct build that ignored its `_format` argument, so `caps()`, which the driver reads immediately and returns in the `SET_ENCODE` reply before any frame exists, echoed the request back. **The check was not missing; it ran too late to reach the reply.** The host's own mismatch warning at `stream.rs:3898` was being silenced by that echo, and NVENC's later complaint never left the driver host process. ## The fix **Honest capabilities, at the point they are computed.** `NvencD3d11Encoder::open` now uses its `format` argument to seed `buffer_fmt` and to clear the effective `chroma_444` when that format is not packed RGB. `open` and `submit` share two new helpers so they cannot drift apart again. `chroma_444_requested` keeps the negotiated value, so a later RGB re-initialisation recovers 4:4:4 exactly as before. That fixes it for every caller of `pf-encode-win`, not only the driver. **The capability is restored, not just declared absent.** `EncodeInput::Rgb10` drives the `HdrRgb10Converter` that was already in the tree and unused, into `R10G10B10A2_UNORM` pool slots with `BIND_RENDER_TARGET` — the same bind flags the pre-cutover out-ring used, checked against the commit before the cutover. Cursor blend, deferred scratch and fence paths needed no change, since `Rgb10` reads FP16 and lands in the existing deferred-scratch arm. Restricting packed RGB to NVENC narrows nothing: the handshake only ever asks 4:4:4 of NVENC or PyroWave. ## One structural call worth reviewing The input-kind decision moved into `pf-driver-proto::encode::EncodeInput` (`choose` plus `full_chroma`), with the driver's `InputKind` now an alias. That is what lets the test run on macOS, and it puts the choice next to the reply it has to agree with rather than two crates away. ## Probe `EncodeProbeRequest::flags` gains HDR and 4:4:4 bits — no wire change, `flags` was reserved. `spec()` now calls the same `InputKind::choose` that `SET_ENCODE` calls, so the probe exercises the shipping decision instead of a copy, which is why the old probe could never have caught this: it hard-coded 8-bit 4:2:0. The open line logs the chosen input and the resulting `chroma_444`. Fixed in passing: the probe reply hard-coded `backend_opened = 1`, so it always claimed NVENC. ## Test `a_444_request_picks_a_full_chroma_input` — a table over backend, HDR and 4:4:4 pinning the chosen input, plus a loop asserting that both backends the host negotiates 4:4:4 for land on a full-chroma input at both depths. The HDR-plus-4:4:4 row is the regression. **It runs on macOS**: 52 tests pass in `pf-driver-proto`. ## CI, and a correction I claimed in #594 that the clippy fix turns `windows-drivers.yml` green. **That was wrong, and this branch proved it.** A dispatch of that workflow fails 40 seconds in, before any Rust compiles, in `pyrowave-sys`'s CMake with a path-too-long error, because the runner's checkout root pushes a tlog past the Windows path limit — exactly the hazard the workflow's own comment predicted. The **same job fails identically on the base branch**, so this lane is red on the whole stack for an infrastructure reason, not a code one. The clippy error was real and is still worth fixing; it simply was not what that lane was dying on. The driver therefore builds by hand instead: all four CI steps run in a short path on the build box, every exit code 0. ## Residual `caps().chroma_444` still cannot reflect the GPU's own HEVC 4:4:4 support at driver open, because that is only known after a device exists. The host already gates it before the handshake, so the only exposure is a multi-GPU box resolving a different adapter host-side than driver-side.
enricobuehler added 1 commit 2026-09-03 10:50:45 +00:00
fix(vdisplay/encode): feed NVENC packed RGB for HDR 4:4:4
windows-drivers / probe-and-proto (pull_request) Successful in 23s
Details
windows-drivers / driver-build (pull_request) Failing after 53s
Details
ci / web (pull_request) Failing after 2m30s
Details
ci / rust-arm64 (pull_request) Successful in 3m55s
Details
ci / docs-site (pull_request) Successful in 2m9s
Details
ci / decky-typecheck (pull_request) Successful in 14s
Details
ci / sdk-plugin-kit (pull_request) Successful in 1m7s
Details
ci / rust (pull_request) Failing after 5m21s
Details
ci / bun-nix (pull_request) Successful in 51s
Details
secret-scan / gitleaks (pull_request) Successful in 16s
Details
ci / docs-drift (pull_request) Failing after 58s
Details
android / android (pull_request) Successful in 6m55s
Details
macos-host / check (pull_request) Canceled after 0s
Details
fc5d7f8282 
Moving the encoder into the driver dropped the packed-RGB input the host
used to build for HDR with 4:4:4: spec_for mapped every HDR session to
P010, so NVENC saw subsampled YUV in a 4:2:0 session while the SET_ENCODE
reply, sent before the first frame, still promised full chroma. A client
that negotiated both was told 4:4:4 with nothing on the wire to say
otherwise.

The input choice moves to pf-driver-proto, beside the reply it has to
agree with, so one table test on macOS pins it. NVENC's caps() now reads
the format the session opened with rather than the request alone, which
makes the answer honest for every caller of the crate. HdrRgb10Converter
was already sitting there unused.

The S5 probe could not have caught this: depth, chroma and the 4:4:4 flag
were hard-coded to 8-bit 4:2:0. Two request flags now drive the 10-bit
and full-chroma inputs, with a check that the desktop presents the
surface format that input reads.
enricobuehler added 1 commit 2026-09-03 11:00:44 +00:00
test(vdisplay/probe): drive the HDR 4:4:4 matrix on the box
macos-host / check (pull_request) Waiting to run
Details
ci / decky-typecheck (pull_request) Successful in 17s
Details
ci / bun-nix (pull_request) Successful in 18s
Details
ci / docs-site (pull_request) Successful in 55s
Details
ci / web (pull_request) Failing after 56s
Details
secret-scan / gitleaks (pull_request) Successful in 18s
Details
ci / docs-drift (pull_request) Failing after 38s
Details
ci / sdk-plugin-kit (pull_request) Successful in 1m36s
Details
windows-drivers / probe-and-proto (pull_request) Successful in 2m2s
Details
ci / rust-arm64 (pull_request) Successful in 5m6s
Details
windows-drivers / driver-build (pull_request) Failing after 4m13s
Details
ci / rust (pull_request) Failing after 8m32s
Details
android / android (pull_request) Successful in 11m2s
Details
ea60f5df5a 
The loopback client cannot negotiate 4:4:4, so the driver probe is the only
instrument that reaches the packed-RGB path on glass. It could not reach it
either: nothing turns advanced colour on for a virtual display the host
service is not managing, so every HDR row would have failed on its surface
format.

The probe run now sets the display to the colour mode its depth needs and
prints what stuck, and the reply's name field carries the chosen input beside
the chroma the caps reported. Both travel in the test's own output, so a row
is judged without reading the driver log.

live-hdr444.ps1 sequences the four rows through the existing s5-prep and
probe-run launchers, then names the disagreement between an input and its
advertised chroma as the regression it is.
Some required checks failed
macos-host / check (pull_request) Waiting to run
Details
ci / decky-typecheck (pull_request) Successful in 17s
Details
ci / bun-nix (pull_request) Successful in 18s
Details
ci / docs-site (pull_request) Successful in 55s
Details
ci / web (pull_request) Failing after 56s
Details
secret-scan / gitleaks (pull_request) Successful in 18s
Details
ci / docs-drift (pull_request) Failing after 38s
Details
ci / sdk-plugin-kit (pull_request) Successful in 1m36s
Details
windows-drivers / probe-and-proto (pull_request) Successful in 2m2s
Details
ci / rust-arm64 (pull_request) Successful in 5m6s
Details
windows-drivers / driver-build (pull_request) Failing after 4m13s
Details
ci / rust (pull_request) Failing after 8m32s
Details
android / android (pull_request) Successful in 11m2s
Details
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/vdisplay-hdr-444:fix/vdisplay-hdr-444
git checkout fix/vdisplay-hdr-444
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
FernandoMX FlipperDan HitFrostbite Seenko bluelightspecial enricobuehler (Enrico Bühler) kszlim luxus nannku neoney sonar
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unom/punktfunk#599