feat(encode/nvenc): an HDR capture stays zero-copy on NVIDIA

AMD/Intel needed no new encoder code for HDR — the VAAPI path already ingests
an XR30 dmabuf into `format=p010:out_color_matrix=bt2020`. NVIDIA did: the
10-bit formats were excluded from the GPU import outright, so an HDR session
fell back to a CPU readback plus swscale, which is the one thing the capture
path is not allowed to ship.

It turns out no CSC kernel is needed. NVENC ingests packed 10-bit RGB natively
as `ARGB10`/`ABGR10` and does the conversion itself following the configured
VUI matrix — which `apply_low_latency_config` already sets to BT.2020 NCL for
an HDR session. So the frame travels LINEAR dmabuf → Vulkan bridge → CUDA →
NVENC unconverted: no host CSC pass, no depth loss, no extra work on a
contended SM.

* invariant 1 is restated rather than dropped: HDR must never take the TILED
  EGL de-tile blit (it renders into an 8-bit `GL_RGBA8` texture). The HDR pods
  are LINEAR-only by construction, so the plan may build the importer; the
  per-frame gate — which sees the negotiated modifier the plan cannot — is what
  enforces the tiled half, and falls back to the CPU path if a producer ever
  ignores our offer.
* …but only where the encoder can actually take the payload
  (`linux_hdr_cuda_ok`). libav's HDR route builds a P010 hardware frames
  context and swscales into it, so on a host without the direct-SDK backend a
  packed-2:10:10:10 CUDA buffer would land in a P010 surface as garbage. Those
  keep the CPU path.
* `nvenc_cuda` stops pinning 8-bit/SDR. Depth and HDR now follow the INPUT
  format, like the Windows backend: a 10-bit session whose capture came back
  8-bit encodes AND labels 8-bit rather than mislabelling.
* the cursor-blend compute shader gains two 10-bit modes, so the pointer
  gamescope leaves out of its node survives the HDR path. Same display-referred
  blend the CPU path's `composite_cursor_rgb10` already does — the samples are
  PQ, and a real sRGB→PQ cursor LUT is polish, not correctness for a pointer.
This commit is contained in:
2026-07-28 18:03:30 +02:00
parent 689297c86e
commit 17f824c3e9
5 changed files with 224 additions and 52 deletions
+33
View File
@@ -365,6 +365,12 @@ fn open_video_backend_linux(
// An HDR session (10-bit + a PQ/BT.2020 capture format) must skip the Vulkan Video
// backend — it hardcodes an 8-bit 4:2:0 BT.709 CSC — and take the libav VAAPI path,
// which has the P010/Main10/PQ wiring. SDR sessions keep the Vulkan default.
//
// Two things ride this switch, and both are the accepted cost of AMD/Intel HDR until
// Vulkan Video learns 10-bit: the Vulkan backend's real RFI loss recovery, and its
// compute-CSC **cursor blend**. A gamescope HDR session therefore streams without the
// host-composited XFixes pointer (gamescope has no embedded-cursor mode to fall back
// to) — `open_video`'s `blends_cursor` backstop logs it per session.
#[cfg(feature = "vulkan-encode")]
if matches!(codec, Codec::H265 | Codec::Av1)
&& vulkan_encode_enabled()
@@ -1002,6 +1008,33 @@ pub fn linux_native_nv12_ok(codec: Codec) -> bool {
}
}
/// Can this host's encode path ingest a **packed 10-bit PQ/BT.2020 CUDA payload** — i.e. may an
/// HDR capture stay zero-copy on NVIDIA?
///
/// Only the direct-SDK NVENC backend can: it registers the buffer as an `ARGB10`/`ABGR10` input
/// surface and does the BT.2020 CSC in the encoder itself. The libav fallback cannot — its HDR
/// route builds a **P010** hardware frames context and swscales the RGB into it, so handing it a
/// packed-10-bit CUDA buffer would copy 2:10:10:10 words into a P010 surface and stream garbage.
/// So when the direct path is compiled out or vetoed (`PUNKTFUNK_NVENC_DIRECT=0`), the capturer
/// must NOT build the importer for an HDR session and the frames take the CPU path instead — the
/// same route HDR took before the direct path learned 10-bit.
///
/// Resolved by the host facade into [`pf_capture::ZeroCopyPolicy`], like every other
/// encode-backend fact capture is allowed to know (the one-way capture→encode edge).
#[cfg(target_os = "linux")]
pub fn linux_hdr_cuda_ok() -> bool {
#[cfg(feature = "nvenc")]
{
// Same two terms `open_nvenc_probed` uses to take the direct arm — minus `cuda`, which is
// the very thing the caller is deciding.
nvenc_direct_enabled() && !linux_zero_copy_is_vaapi()
}
#[cfg(not(feature = "nvenc"))]
{
false
}
}
/// Whether the encode backend this session will resolve to composites [`CapturedFrame::cursor`]
/// ([`EncoderCaps::blends_cursor`]) — answered BEFORE capture opens, so the host plans cursor
/// delivery honestly instead of discovering a cursorless stream after the fact (the