feat(encode/vulkan): probe the device instead of guessing — AV1 10-bit + zero-CSC HDR

Three fixes to the same mistake: deciding what the Vulkan Video backend can do
from a table in our heads rather than from the driver, and routing everything
that didn't fit to libav VAAPI — where a session loses real RFI recovery and
the cursor blend for no reason the hardware asked for.

**Capability probe, per codec AND depth.** `probe_encode_support`'s "is there
an encode queue" boolean becomes `VulkanEncodeCaps { supported, eight_bit,
ten_bit }`, answered by `vkGetPhysicalDeviceVideoCapabilitiesKHR` against the
very profile chain the session open builds. So the dispatcher's prediction
cannot disagree with reality: a capable device keeps the Vulkan path, an
incapable one routes to VAAPI BEFORE burning a failed open, and the
cursor-blend mirror stays honest for free. This is the shape the direct-SDK
NVENC path already uses for its codec GUIDs.

**AV1 10-bit.** It was excluded on a guess about driver coverage; now the
device answers. `color_config()` carries `high_bitdepth` + the BT.2020/PQ CICP
triplet in both the `StdVideoAV1ColorConfig` and the sequence-header OBU we
bit-pack ourselves — they must stay identical or the driver's frame OBUs parse
against a header we didn't write. `high_bitdepth` sits BEFORE the CICP bytes,
so getting it wrong doesn't just mislabel the depth, it puts every following
field one bit out of phase; the new test reads the packed bits back.

**Zero-CSC RGB-direct in HDR.** The EFC probe assumed BT.709 and BGRA. It now
asks for the model this session's colourimetry needs (`MODEL_YCBCR_2020` for
10-bit — the extension has always had it) and for the CAPTURED format as an
encode-source format, and the session create-info selects the matching model.
An HDR session with no pointer to composite therefore hands the captured
buffer straight to the fixed-function front end and runs no host CSC at all.
Sessions that DO composite a pointer keep the compute CSC, unchanged: the EFC
cannot blend, and that rule outranks everything.

Also: `can_encode_10bit` on AMD/Intel now reports the union of VAAPI's and
Vulkan Video's answers instead of VAAPI's alone. `open_amd_intel` tries Vulkan
first and falls back, so either one being able to encode Main10 makes the
session 10-bit-capable — answering `false` because only one of them said yes
stranded encodable HDR sessions at 8 bits.

crates/pf-encode/src/enc/linux/vk_build.rs

@@ -20,17 +20,25 @@ pub(super) fn align_up(v: u64, a: u64) -> u64 {
 }
 
 /// Probe for the RGB-direct encode source (design/vulkan-rgb-direct-encode.md): can this device
-/// take the captured RGB dmabuf directly, with the VCN EFC front-end doing the 709-narrow CSC,
-/// via `VK_VALVE_video_encode_rgb_conversion` (RADV since Mesa 26.0, gated on EFC hardware)?
+/// take the captured RGB dmabuf directly, with the VCN EFC front-end doing the CSC, via
+/// `VK_VALVE_video_encode_rgb_conversion` (RADV since Mesa 26.0, gated on EFC hardware)?
 /// `Ok((x_offset, y_offset))` carries the chroma-siting bits a session must be created with
 /// (the preferred available bit per axis); `Err` is the first missing requirement, logged as
 /// the open-time verdict.
+///
+/// `ten_bit` + `src_fmt` describe the session being planned: an HDR one needs the EFC to advertise
+/// the BT.2020 model (not 709) and the 10-bit packed-RGB `src_fmt` as an encode-source format.
+/// Both are hardware facts, so an EFC that cannot do HDR simply reports `Err` and the session
+/// takes the compute CSC — no fallback all the way out to VAAPI.
+#[allow(clippy::too_many_arguments)]
 pub(super) unsafe fn probe_rgb_direct(
     instance: &ash::Instance,
     vq_inst: &ash::khr::video_queue::Instance,
     pd: vk::PhysicalDevice,
     codec_op: vk::VideoCodecOperationFlagsKHR,
     av1: bool,
+    ten_bit: bool,
+    src_fmt: vk::Format,
 ) -> Result<(u32, u32), &'static str> {
     use crate::vk_av1_encode as av1b;
     use crate::vk_valve_rgb as vrgb;
@@ -58,10 +66,11 @@ pub(super) unsafe fn probe_rgb_direct(
     if feat.video_encode_rgb_conversion == vk::FALSE {
         return Err("no-feature");
     }
-    // 3. Capabilities under the rgb-chained profile — the conversion must cover the compute
-    //    CSC's colour math (rgb2yuv.comp: BT.709, narrow range; chroma siting is looser, see
-    //    below). The profile chain is the same one every rgb-direct consumer presents.
-    let mut ps = RgbProfileStack::new(codec_op);
+    // 3. Capabilities under the rgb-chained profile — the conversion must cover the colour math
+    //    the compute CSC would otherwise do (`rgb2yuv.comp`: BT.709 narrow; `rgb2yuv10.comp`:
+    //    BT.2020 narrow), at this session's depth. Chroma siting is looser, see below. The
+    //    profile chain is the same one every rgb-direct consumer presents.
+    let mut ps = RgbProfileStack::new(codec_op, ten_bit);
     let profile = *ps.wire(av1);
     let mut rgb_caps = vrgb::VideoEncodeRgbConversionCapabilitiesVALVE {
         s_type: vrgb::stype(vrgb::ST_CAPABILITIES),
@@ -103,10 +112,13 @@ pub(super) unsafe fn probe_rgb_direct(
             None
         }
     };
-    if rgb_caps.rgb_models & vrgb::MODEL_YCBCR_709 == 0
-        || rgb_caps.rgb_ranges & vrgb::RANGE_NARROW == 0
-    {
-        return Err("no-709-narrow");
+    let want_model = rgb_model_for(ten_bit);
+    if rgb_caps.rgb_models & want_model == 0 || rgb_caps.rgb_ranges & vrgb::RANGE_NARROW == 0 {
+        return Err(if ten_bit {
+            "no-2020-narrow"
+        } else {
+            "no-709-narrow"
+        });
     }
     let (Some(x_offset), Some(y_offset)) = (
         pick(rgb_caps.x_chroma_offsets),
@@ -114,8 +126,10 @@ pub(super) unsafe fn probe_rgb_direct(
     ) else {
         return Err("no-chroma-siting");
     };
-    // 4. The encode-src format set under this profile must offer BGRA with DRM-modifier tiling —
-    //    the capture hands LINEAR BGRx dmabufs (fourcc XR24), which import as B8G8R8A8_UNORM.
+    // 4. The encode-src format set under this profile must offer the CAPTURED format with
+    //    DRM-modifier tiling — LINEAR BGRx dmabufs (fourcc XR24) import as B8G8R8A8_UNORM, and a
+    //    10-bit PQ capture (XR30/XB30) as one of the packed 2:10:10:10 formats. A device whose
+    //    EFC handles 8-bit RGB but not 10-bit lands here rather than at the session create.
     let profile_arr = [profile];
     let plist = vk::VideoProfileListInfoKHR::default().profiles(&profile_arr);
     let mut fmt_info = vk::PhysicalDeviceVideoFormatInfoKHR::default()
@@ -132,15 +146,31 @@ pub(super) unsafe fn probe_rgb_direct(
     if r != vk::Result::SUCCESS && r != vk::Result::INCOMPLETE {
         return Err("no-rgb-format");
     }
-    if !props[..count as usize].iter().any(|p| {
-        p.format == vk::Format::B8G8R8A8_UNORM
-            && p.image_tiling == vk::ImageTiling::DRM_FORMAT_MODIFIER_EXT
-    }) {
-        return Err("no-bgra-modifier-tiling");
+    if !props[..count as usize]
+        .iter()
+        .any(|p| p.format == src_fmt && p.image_tiling == vk::ImageTiling::DRM_FORMAT_MODIFIER_EXT)
+    {
+        return Err(if ten_bit {
+            "no-rgb10-modifier-tiling"
+        } else {
+            "no-bgra-modifier-tiling"
+        });
     }
     Ok((x_offset, y_offset))
 }
 
+/// The EFC colour model a session of this depth needs: BT.709 for SDR, BT.2020 for HDR — the same
+/// matrices the two compute-CSC shaders implement, so the encode source is interchangeable and the
+/// SPS/sequence-header colour signalling is correct either way.
+pub(super) fn rgb_model_for(ten_bit: bool) -> u32 {
+    use crate::vk_valve_rgb as vrgb;
+    if ten_bit {
+        vrgb::MODEL_YCBCR_2020
+    } else {
+        vrgb::MODEL_YCBCR_709
+    }
+}
+
 pub(super) unsafe fn make_video_image(
     device: &ash::Device,
     mp: &vk::PhysicalDeviceMemoryProperties,
@@ -708,9 +738,10 @@ fn leb128(mut v: u64) -> Vec<u8> {
 
 /// Bit-pack a `sequence_header_obu` (AV1 spec §5.5) into a size-delimited OBU. The field values here
 /// MUST mirror the `StdVideoAV1SequenceHeader` handed to the driver in `build_parameters_av1` so the
-/// driver-emitted frame OBUs parse against this header. Single operating point, 8-bit 4:2:0,
-/// order-hint on, CDEF+restoration+filter-intra allowed, everything exotic (compound/warp/superres)
-/// disabled — the profile our single-reference P-frame encoder actually uses.
+/// driver-emitted frame OBUs parse against this header. Single operating point, 4:2:0 at 8 or 10
+/// bits, order-hint on, CDEF+restoration+filter-intra allowed, everything exotic
+/// (compound/warp/superres) disabled — the profile our single-reference P-frame encoder uses.
+#[allow(clippy::too_many_arguments)]
 fn av1_sequence_header_obu(
     sb128: bool,
     fwb: u32,
@@ -719,6 +750,7 @@ fn av1_sequence_header_obu(
     max_h_m1: u32,
     order_hint_bits_minus_1: u32,
     seq_level_idx: u32,
+    ten_bit: bool,
 ) -> Vec<u8> {
     let mut w = Av1BitWriter::new();
     w.put(0, 3); // seq_profile = MAIN
@@ -753,19 +785,28 @@ fn av1_sequence_header_obu(
     w.bit(0); // enable_superres
     w.bit(0); // enable_cdef
     w.bit(0); // enable_restoration
-              // color_config() (AV1 spec §5.5.2): 8-bit 4:2:0, BT.709 limited — the CSC this
-              // backend's `rgb2yuv.comp` actually performs. AV1 has no VUI, so the CICP triplet
-              // lives here; omitting it (color_description_present_flag = 0) left the stream
-              // "unspecified" and vendor TV decoders guess colorimetry from resolution.
-              // CP_BT_709/TC_BT_709/MC_BT_709 avoids the spec's sRGB special case (which would
-              // force color_range = 1 and drop the explicit range bit), so the field order below
-              // is the same as the unspecified form plus the three CICP bytes.
-    w.bit(0); // high_bitdepth
+              // color_config() (AV1 spec §5.5.2): 4:2:0 at the session's depth, limited range,
+              // carrying the CSC this backend actually performed — BT.709 for `rgb2yuv.comp`,
+              // BT.2020 + PQ for `rgb2yuv10.comp` (or the EFC's equivalent). AV1 has no VUI, so
+              // the CICP triplet lives here; omitting it (color_description_present_flag = 0)
+              // left the stream "unspecified" and vendor TV decoders guess colorimetry from
+              // resolution. Neither triplet hits the spec's sRGB special case (which would force
+              // color_range = 1 and drop the explicit range bit), so the field order below is the
+              // same as the unspecified form plus the three CICP bytes.
+              //
+              // `high_bitdepth` alone encodes 10-bit here: `twelve_bit` follows it ONLY for
+              // seq_profile 2, and ours is MAIN (0).
+    w.bit(ten_bit as u32); // high_bitdepth -> BitDepth = 10
     w.bit(0); // mono_chrome
     w.bit(1); // color_description_present_flag
-    w.put(1, 8); // color_primaries = CP_BT_709
-    w.put(1, 8); // transfer_characteristics = TC_BT_709
-    w.put(1, 8); // matrix_coefficients = MC_BT_709
+    let (prim, trc, mat) = if ten_bit {
+        (9u32, 16u32, 9u32)
+    } else {
+        (1, 1, 1)
+    };
+    w.put(prim, 8); // color_primaries         (1 = BT.709, 9 = BT.2020)
+    w.put(trc, 8); // transfer_characteristics (1 = BT.709, 16 = SMPTE 2084)
+    w.put(mat, 8); // matrix_coefficients      (1 = BT.709, 9 = BT.2020 NCL)
     w.bit(0); // color_range (studio/limited)
     w.put(0, 2); // chroma_sample_position = CSP_UNKNOWN (subsampling_x==subsampling_y==1 for profile 0)
     w.bit(0); // separate_uv_delta_q
@@ -796,6 +837,9 @@ pub(super) unsafe fn build_parameters_av1(
     max_level: ash::vk::native::StdVideoAV1Level,
     sb128: bool,
     quality_level: u32,
+    // 10-bit HDR session — must agree with the video profile the session was CREATED with
+    // (`open_inner`'s `ten_bit`) and with the OBU packed below.
+    ten_bit: bool,
 ) -> Result<(vk::VideoSessionParametersKHR, Vec<u8>, Vec<u8>)> {
     use crate::vk_av1_encode as av1;
     use ash::vk::native as hh;
@@ -806,21 +850,33 @@ pub(super) unsafe fn build_parameters_av1(
     let seq_level_idx = max_level; // StdVideoAV1Level's numeric value IS the AV1 seq_level_idx
 
     // ---- Std sequence header (must match the OBU packed below) ----
-    // BT.709 limited, mirroring the `color_config()` bits `av1_sequence_header_obu` packs — the two
-    // MUST stay identical or the driver's frame OBUs parse against a header we didn't write.
-    // `color_range` stays 0 (studio swing); only the description flag + CICP triplet change.
+    // Limited range at the session's depth, mirroring the `color_config()` bits
+    // `av1_sequence_header_obu` packs — the two MUST stay identical or the driver's frame OBUs
+    // parse against a header we didn't write. `color_range` stays 0 (studio swing).
     let mut cc_flags: hh::StdVideoAV1ColorConfigFlags = std::mem::zeroed();
     cc_flags.set_color_description_present_flag(1);
     let mut cc: hh::StdVideoAV1ColorConfig = std::mem::zeroed();
     cc.flags = cc_flags;
-    cc.BitDepth = 8;
+    // The Std struct carries the DEPTH; the driver derives the OBU's `high_bitdepth` from it.
+    cc.BitDepth = if ten_bit { 10 } else { 8 };
     cc.subsampling_x = 1;
     cc.subsampling_y = 1;
-    cc.color_primaries = hh::StdVideoAV1ColorPrimaries_STD_VIDEO_AV1_COLOR_PRIMARIES_BT_709;
-    cc.transfer_characteristics =
-        hh::StdVideoAV1TransferCharacteristics_STD_VIDEO_AV1_TRANSFER_CHARACTERISTICS_BT_709;
-    cc.matrix_coefficients =
-        hh::StdVideoAV1MatrixCoefficients_STD_VIDEO_AV1_MATRIX_COEFFICIENTS_BT_709;
+    let (prim, trc, mat) = if ten_bit {
+        (
+            hh::StdVideoAV1ColorPrimaries_STD_VIDEO_AV1_COLOR_PRIMARIES_BT_2020,
+            hh::StdVideoAV1TransferCharacteristics_STD_VIDEO_AV1_TRANSFER_CHARACTERISTICS_SMPTE_2084,
+            hh::StdVideoAV1MatrixCoefficients_STD_VIDEO_AV1_MATRIX_COEFFICIENTS_BT_2020_NCL,
+        )
+    } else {
+        (
+            hh::StdVideoAV1ColorPrimaries_STD_VIDEO_AV1_COLOR_PRIMARIES_BT_709,
+            hh::StdVideoAV1TransferCharacteristics_STD_VIDEO_AV1_TRANSFER_CHARACTERISTICS_BT_709,
+            hh::StdVideoAV1MatrixCoefficients_STD_VIDEO_AV1_MATRIX_COEFFICIENTS_BT_709,
+        )
+    };
+    cc.color_primaries = prim;
+    cc.transfer_characteristics = trc;
+    cc.matrix_coefficients = mat;
     cc.chroma_sample_position =
         hh::StdVideoAV1ChromaSamplePosition_STD_VIDEO_AV1_CHROMA_SAMPLE_POSITION_UNKNOWN;
 
@@ -891,6 +947,7 @@ pub(super) unsafe fn build_parameters_av1(
         h - 1,
         order_hint_bits_minus_1,
         seq_level_idx,
+        ten_bit,
     );
     let mut keyframe_prefix = td.clone();
     keyframe_prefix.extend_from_slice(&seq_obu);
@@ -911,7 +968,7 @@ mod tests {
         fwb: u32,
         fhb: u32,
         seq_level_idx: u32,
-    ) -> (u8, u8, u8, u8, u8) {
+    ) -> (u8, u8, u8, u8, u8, u8) {
         // obu_header (1 byte) + leb128 size — the payload starts after both.
         assert_eq!(
             obu[0], 0x0a,
@@ -972,8 +1029,9 @@ mod tests {
         take(1); // enable_cdef
         take(1); // enable_restoration
 
-        // color_config()
-        assert_eq!(take(1), 0, "high_bitdepth (8-bit)");
+        // color_config(). `high_bitdepth` is returned rather than asserted — the 10-bit case
+        // below is the whole point of reading it.
+        let high_bitdepth = take(1) as u8;
         assert_eq!(take(1), 0, "mono_chrome");
         let described = take(1) as u8;
         let (cp, tc, mc) = if described == 1 {
@@ -986,7 +1044,7 @@ mod tests {
         assert_eq!(take(1), 0, "separate_uv_delta_q");
         assert_eq!(take(1), 0, "film_grain_params_present");
         assert_eq!(take(1), 1, "trailing_one_bit");
-        (described, cp, tc, mc, range)
+        (high_bitdepth, described, cp, tc, mc, range)
     }
 
     /// The sequence header must SIGNAL BT.709 limited — the CSC `rgb2yuv.comp` actually performs.
@@ -1001,8 +1059,9 @@ mod tests {
         // 1920x1080: av_log2 gives 10/10 frame-size bits; level 4.0 (seq_level_idx 8) exercises
         // the seq_tier branch, and sb128 both ways since it sits above color_config.
         for (sb128, level) in [(false, 8u32), (true, 5u32)] {
-            let obu = av1_sequence_header_obu(sb128, 10, 10, 1919, 1079, 7, level);
-            let (described, cp, tc, mc, range) = read_color_config(&obu, 10, 10, level);
+            let obu = av1_sequence_header_obu(sb128, 10, 10, 1919, 1079, 7, level, false);
+            let (depth10, described, cp, tc, mc, range) = read_color_config(&obu, 10, 10, level);
+            assert_eq!(depth10, 0, "high_bitdepth (8-bit session)");
             assert_eq!(
                 described, 1,
                 "color_description_present_flag (sb128={sb128})"
@@ -1015,4 +1074,24 @@ mod tests {
             assert_eq!(range, 0, "color_range = studio/limited swing");
         }
     }
+
+    /// …and a 10-bit session must signal BT.2020 + PQ with `high_bitdepth` set. Same reason the
+    /// 8-bit twin exists, plus one that is specific to AV1: `high_bitdepth` sits BEFORE the CICP
+    /// bytes in `color_config()`, so getting it wrong does not just mislabel the depth — every
+    /// field after it parses one bit out of phase.
+    #[test]
+    fn av1_sequence_header_signals_bt2020_pq_at_10_bit() {
+        for (sb128, level) in [(false, 8u32), (true, 5u32)] {
+            let obu = av1_sequence_header_obu(sb128, 10, 10, 1919, 1079, 7, level, true);
+            let (depth10, described, cp, tc, mc, range) = read_color_config(&obu, 10, 10, level);
+            assert_eq!(depth10, 1, "high_bitdepth (sb128={sb128})");
+            assert_eq!(described, 1, "color_description_present_flag");
+            assert_eq!(
+                (cp, tc, mc),
+                (9, 16, 9),
+                "CICP BT.2020 primaries / SMPTE 2084 transfer / BT.2020-NCL matrix"
+            );
+            assert_eq!(range, 0, "color_range = studio/limited swing");
+        }
+    }
 }

crates/pf-encode/src/enc/linux/vulkan_video.rs

@@ -33,6 +33,22 @@ const P010: vk::Format = vk::Format::G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16;
 /// The session's 4:2:0 picture + DPB format for its bit depth. One function so the session
 /// create-info, the DPB image, its views and the per-frame encode source cannot drift apart —
 /// they must all name the SAME format or session creation fails.
+const fn component_depth(ten_bit: bool) -> vk::VideoComponentBitDepthFlagsKHR {
+    if ten_bit {
+        vk::VideoComponentBitDepthFlagsKHR::TYPE_10
+    } else {
+        vk::VideoComponentBitDepthFlagsKHR::TYPE_8
+    }
+}
+
+const fn h265_profile_idc(ten_bit: bool) -> vk::native::StdVideoH265ProfileIdc {
+    if ten_bit {
+        vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN_10
+    } else {
+        vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN
+    }
+}
+
 const fn yuv_format(hdr: bool) -> vk::Format {
     if hdr {
         P010
@@ -168,7 +184,7 @@ struct RgbProfileStack {
 }
 
 impl RgbProfileStack {
-    fn new(codec_op: vk::VideoCodecOperationFlagsKHR) -> Self {
+    fn new(codec_op: vk::VideoCodecOperationFlagsKHR, ten_bit: bool) -> Self {
         use super::vk_av1_encode as av1b;
         use super::vk_valve_rgb as vrgb;
         Self {
@@ -181,9 +197,8 @@ impl RgbProfileStack {
                 .video_usage_hints(vk::VideoEncodeUsageFlagsKHR::STREAMING)
                 .video_content_hints(vk::VideoEncodeContentFlagsKHR::RENDERED)
                 .tuning_mode(vk::VideoEncodeTuningModeKHR::ULTRA_LOW_LATENCY),
-            h265: vk::VideoEncodeH265ProfileInfoKHR::default().std_profile_idc(
-                vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN,
-            ),
+            h265: vk::VideoEncodeH265ProfileInfoKHR::default()
+                .std_profile_idc(h265_profile_idc(ten_bit)),
             av1: av1b::VideoEncodeAV1ProfileInfoKHR {
                 s_type: av1b::stype(av1b::ST_PROFILE_INFO),
                 p_next: std::ptr::null(),
@@ -192,8 +207,8 @@ impl RgbProfileStack {
             profile: vk::VideoProfileInfoKHR::default()
                 .video_codec_operation(codec_op)
                 .chroma_subsampling(vk::VideoChromaSubsamplingFlagsKHR::TYPE_420)
-                .luma_bit_depth(vk::VideoComponentBitDepthFlagsKHR::TYPE_8)
-                .chroma_bit_depth(vk::VideoComponentBitDepthFlagsKHR::TYPE_8),
+                .luma_bit_depth(component_depth(ten_bit))
+                .chroma_bit_depth(component_depth(ten_bit)),
         }
     }
 
@@ -221,26 +236,26 @@ struct NativeProfileStack {
 }
 
 impl NativeProfileStack {
-    fn new(codec_op: vk::VideoCodecOperationFlagsKHR) -> Self {
+    fn new(codec_op: vk::VideoCodecOperationFlagsKHR, ten_bit: bool) -> Self {
         use super::vk_av1_encode as av1b;
         Self {
             usage: vk::VideoEncodeUsageInfoKHR::default()
                 .video_usage_hints(vk::VideoEncodeUsageFlagsKHR::STREAMING)
                 .video_content_hints(vk::VideoEncodeContentFlagsKHR::RENDERED)
                 .tuning_mode(vk::VideoEncodeTuningModeKHR::ULTRA_LOW_LATENCY),
-            h265: vk::VideoEncodeH265ProfileInfoKHR::default().std_profile_idc(
-                vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN,
-            ),
+            h265: vk::VideoEncodeH265ProfileInfoKHR::default()
+                .std_profile_idc(h265_profile_idc(ten_bit)),
             av1: av1b::VideoEncodeAV1ProfileInfoKHR {
                 s_type: av1b::stype(av1b::ST_PROFILE_INFO),
                 p_next: std::ptr::null(),
+                // AV1 Main covers 8 AND 10 bits — the depth rides `VideoProfileInfoKHR` alone.
                 std_profile: vk::native::StdVideoAV1Profile_STD_VIDEO_AV1_PROFILE_MAIN,
             },
             profile: vk::VideoProfileInfoKHR::default()
                 .video_codec_operation(codec_op)
                 .chroma_subsampling(vk::VideoChromaSubsamplingFlagsKHR::TYPE_420)
-                .luma_bit_depth(vk::VideoComponentBitDepthFlagsKHR::TYPE_8)
-                .chroma_bit_depth(vk::VideoComponentBitDepthFlagsKHR::TYPE_8),
+                .luma_bit_depth(component_depth(ten_bit))
+                .chroma_bit_depth(component_depth(ten_bit)),
         }
     }
 
@@ -260,7 +275,7 @@ impl NativeProfileStack {
 /// profile rebuilds — the two must agree, profile identity is by value).
 /// The physical device + encode queue family a session runs on: the FIRST device exposing a
 /// `VIDEO_ENCODE` queue family that advertises `codec_op` (llvmpipe advertises none, so it drops
-/// out implicitly). Shared by [`VulkanVideoEncoder::open_inner`] and [`probe_encode_support`].
+/// out implicitly). Shared by [`VulkanVideoEncoder::open_inner`] and [`probe_encode_caps`].
 ///
 /// # Safety
 /// `instance` must be a live `ash::Instance` and `devices` handles enumerated from it.
@@ -291,59 +306,104 @@ unsafe fn find_encode_device(
     None
 }
 
-/// Can this GPU + driver open a Vulkan Video **encode** session for `codec` at all?
+/// What this device's Vulkan Video **encode** stack can actually do for one codec — the caps
+/// probe the negotiation stands on, so a session is never planned around a guess.
 ///
-/// This is the verdict the native-NV12 capture negotiation needs BEFORE it commits
-/// ([`crate::linux_native_nv12_ok`]): once the producer hands over two-plane NV12 there is no VAAPI
-/// fallback — libav would import it as packed RGB — so [`crate::open_video`] deliberately makes
-/// that open-failure fatal, and a Mesa built without `VK_KHR_video_encode_h265` therefore kills the
-/// session at its first frame instead of streaming on VAAPI.
+/// Two questions, and they are genuinely independent: an encode queue for the codec operation may
+/// exist while the silicon declines the 10-bit profile (older VCN, an Intel generation without
+/// Main10 encode). Answering only the first is what used to push every HDR session to libav VAAPI
+/// — losing real RFI recovery and the compute CSC's cursor blend for nothing on hardware that
+/// could do it.
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
+pub(crate) struct VulkanEncodeCaps {
+    /// An encode queue advertises this codec's operation.
+    pub supported: bool,
+    /// The device accepts an 8-bit 4:2:0 profile for it (the ordinary SDR session).
+    pub eight_bit: bool,
+    /// …and a 10-bit one: HEVC Main10 / AV1 Main at 10 bits, the HDR session's profile.
+    pub ten_bit: bool,
+}
+
+/// Probe [`VulkanEncodeCaps`] for `codec`. Uncached — [`crate::vulkan_encode_caps`] owns the
+/// per-(GPU, codec) cache, exactly as it did for the old boolean.
 ///
-/// Deliberately the FIRST check `open_inner` performs and hard-fails on, and nothing more: the same
-/// [`find_encode_device`] scan, run against the same codec op. That makes it provably no stricter
-/// than the open, so a failure here can only ever name a session that would have died anyway — it
-/// can never talk a working host out of the fast path. It is also cheap: one instance plus
-/// physical-device queries, no logical device, no video session, no VRAM. The later stages
-/// (`create_device`, `create_video_session`, the capability query) can still fail for reasons this
-/// does not model; those keep the session on the packed-RGB negotiation the ordinary way, because
-/// the capture format is only committed once this said yes.
-///
-/// Probed PER CODEC, not once: `codec_op_for` selects a different queue-family bit for AV1, and
-/// HEVC-encode-without-AV1-encode is the common VCN/ANV configuration.
-pub(crate) fn probe_encode_support(codec: Codec) -> Result<(), &'static str> {
+/// The depth answers come from `vkGetPhysicalDeviceVideoCapabilitiesKHR` against a profile built
+/// at that depth, which is the same query the session open makes: a `true` here means the open
+/// will get past the profile gate, and a `false` means it would have failed. No second source of
+/// truth to drift.
+pub(crate) fn probe_encode_caps(codec: Codec) -> VulkanEncodeCaps {
     if !matches!(codec, Codec::H265 | Codec::Av1) {
-        return Err("the Vulkan Video backend encodes HEVC + AV1 only");
+        return VulkanEncodeCaps::default();
     }
-    let codec_op = codec_op_for(codec == Codec::Av1);
+    let av1 = codec == Codec::Av1;
+    let codec_op = codec_op_for(av1);
     // SAFETY: creates one Vulkan instance and issues only physical-device queries against it, then
     // destroys it on EVERY path below before returning — no handle derived from it escapes, and
     // nothing outside this call observes it. `Entry::load` only dlopens the loader (a missing
     // libvulkan returns `Err`), touching no process state the rest of the crate relies on.
-    // `find_encode_device` gets a live instance and handles enumerated from it, as it requires.
+    // `find_encode_device` gets a live instance and handles enumerated from it, as it requires;
+    // `depth_supported` gets that same live instance plus the physical device it returned.
     unsafe {
         let Ok(entry) = ash::Entry::load() else {
-            return Err("no Vulkan loader");
+            return VulkanEncodeCaps::default();
         };
         let app = vk::ApplicationInfo::default().api_version(vk::API_VERSION_1_3);
         let Ok(instance) = entry.create_instance(
             &vk::InstanceCreateInfo::default().application_info(&app),
             None,
         ) else {
-            return Err("vkCreateInstance failed");
+            return VulkanEncodeCaps::default();
         };
         let found = match instance.enumerate_physical_devices() {
-            Ok(devices) => find_encode_device(&instance, &devices, codec_op).is_some(),
-            Err(_) => false,
+            Ok(devices) => find_encode_device(&instance, &devices, codec_op).map(|(pd, _)| pd),
+            Err(_) => None,
+        };
+        let caps = match found {
+            Some(pd) => {
+                let vq_inst = ash::khr::video_queue::Instance::new(&entry, &instance);
+                VulkanEncodeCaps {
+                    supported: true,
+                    eight_bit: depth_supported(&vq_inst, pd, codec_op, av1, false),
+                    ten_bit: depth_supported(&vq_inst, pd, codec_op, av1, true),
+                }
+            }
+            None => VulkanEncodeCaps::default(),
         };
         instance.destroy_instance(None);
-        if found {
-            Ok(())
-        } else {
-            Err("no VK_KHR_video_encode queue for this codec on any device")
-        }
+        caps
     }
 }
 
+/// Does `pd` accept an encode profile for `codec_op` at this bit depth? The profile chain is
+/// byte-identical to the one [`VulkanVideoEncoder::open_inner`] builds — that is the point.
+///
+/// # Safety
+/// `vq_inst` must wrap the live instance `pd` was enumerated from.
+unsafe fn depth_supported(
+    vq_inst: &ash::khr::video_queue::Instance,
+    pd: vk::PhysicalDevice,
+    codec_op: vk::VideoCodecOperationFlagsKHR,
+    av1: bool,
+    ten_bit: bool,
+) -> bool {
+    let mut ps = NativeProfileStack::new(codec_op, ten_bit);
+    let profile = *ps.wire(av1);
+    let mut h265_caps = vk::VideoEncodeH265CapabilitiesKHR::default();
+    let mut av1_caps: super::vk_av1_encode::VideoEncodeAV1CapabilitiesKHR = std::mem::zeroed();
+    av1_caps.s_type = super::vk_av1_encode::stype(super::vk_av1_encode::ST_CAPABILITIES);
+    let mut enc_caps = vk::VideoEncodeCapabilitiesKHR::default();
+    let mut caps = vk::VideoCapabilitiesKHR::default();
+    if av1 {
+        av1_caps.p_next = &mut enc_caps as *mut _ as *mut c_void;
+        caps.p_next = &mut av1_caps as *mut _ as *mut c_void;
+    } else {
+        h265_caps.p_next = &mut enc_caps as *mut _ as *mut c_void;
+        caps.p_next = &mut h265_caps as *mut _ as *mut c_void;
+    }
+    (vq_inst.fp().get_physical_device_video_capabilities_khr)(pd, &profile, &mut caps)
+        == vk::Result::SUCCESS
+}
+
 fn codec_op_for(av1: bool) -> vk::VideoCodecOperationFlagsKHR {
     if av1 {
         vk::VideoCodecOperationFlagsKHR::from_raw(
@@ -639,6 +699,11 @@ pub struct VulkanVideoEncoder {
     /// aligned and an undersized direct source is the OOB-read class behind the 2026-07-20 field
     /// GPU reset (those paths keep their aligned-size sources/staging).
     native_nv12: bool,
+    /// This is a 10-bit (HDR) session: Main10 / AV1-10 profile, `P010` picture + DPB, and either
+    /// the BT.2020 compute CSC or the EFC's BT.2020 conversion. Every profile chain rebuilt after
+    /// `open` (the per-buffer dmabuf imports) must present the SAME depth, so it is carried here
+    /// rather than re-derived.
+    ten_bit: bool,
 
     /// A [`reconfigure_bitrate`](Encoder::reconfigure_bitrate) rate not yet installed in the video
     /// session. The next `record_submit` emits an `ENCODE_RATE_CONTROL` control command carrying it
@@ -694,29 +759,25 @@ impl VulkanVideoEncoder {
     ) -> Result<Self> {
         let native_nv12 = format == PixelFormat::Nv12;
         // HDR: a packed 10-bit PQ/BT.2020 capture (a gamescope output off our `pipewire-hdr`
-        // build) encodes HEVC Main10 through the 10-bit compute CSC. HEVC only — AV1 10-bit
-        // encode has far thinner driver coverage, and `open_amd_intel` keeps those sessions on
-        // libav VAAPI rather than gambling a session open on it.
-        let hdr = format.is_hdr_rgb10();
-        if hdr && codec != Codec::H265 {
-            bail!(
-                "vulkan-encode: 10-bit HDR is HEVC Main10 only here (got {codec:?}) — the \
-                 dispatcher should have routed this session to VAAPI"
-            );
-        }
+        // build, or the GNOME 50+ portal monitor mirror). BOTH codecs and BOTH encode sources
+        // serve it — which of them this device can actually do is `probe_encode_caps`' answer,
+        // consulted by the dispatcher before we get here and re-checked by the profile query
+        // inside the open.
+        let ten_bit = format.is_hdr_rgb10();
+        // The RGB-direct (EFC) source needs the captured format as the session's picture format.
+        // `pixel_to_vk` covers every format the capture can hand a GPU session; the BGRA default
+        // only preserves the old behaviour for the CPU-only layouts, which never reach that arm.
+        let src_rgb_fmt = pixel_to_vk(format).unwrap_or(vk::Format::B8G8R8A8_UNORM);
         // A cursor-blend session must keep the compute-CSC path — the only arm with the cursor
         // blend — so it outranks an explicit RGB-direct pin (EFC cannot composite; the
-        // negotiation promised the client a composited pointer). HDR pins the same arm for a
-        // different reason: the EFC's fixed-function conversion is 8-bit BT.709 narrow with no
-        // knob for BT.2020, so it cannot express this session's colourimetry at all.
-        if (cursor_blend || hdr) && rgb_request() == Some(true) {
+        // negotiation promised the client a composited pointer).
+        if cursor_blend && rgb_request() == Some(true) {
             tracing::info!(
-                hdr,
-                "PUNKTFUNK_VULKAN_RGB_DIRECT=1 ignored for this session — the EFC front-end can \
-                 neither composite a pointer nor convert BT.2020 10-bit; using the compute-CSC path"
+                "PUNKTFUNK_VULKAN_RGB_DIRECT=1 ignored for this session — it composites the \
+                 pointer, which the EFC front-end cannot; using the compute-CSC path"
             );
         }
-        let want_rgb = !native_nv12 && !cursor_blend && !hdr && rgb_request().unwrap_or(true);
+        let want_rgb = !native_nv12 && !cursor_blend && rgb_request().unwrap_or(true);
         Self::open_opts_inner(
             codec,
             width,
@@ -725,7 +786,8 @@ impl VulkanVideoEncoder {
             bitrate_bps,
             want_rgb,
             native_nv12,
-            hdr,
+            ten_bit,
+            src_rgb_fmt,
         )
     }
 
@@ -752,6 +814,7 @@ impl VulkanVideoEncoder {
             want_rgb,
             false,
             false,
+            vk::Format::B8G8R8A8_UNORM,
         )
     }
 
@@ -764,7 +827,8 @@ impl VulkanVideoEncoder {
         bitrate_bps: u64,
         want_rgb: bool,
         native_nv12: bool,
-        hdr: bool,
+        ten_bit: bool,
+        src_rgb_fmt: vk::Format,
     ) -> Result<Self> {
         if !matches!(codec, Codec::H265 | Codec::Av1) {
             bail!("vulkan-encode backend supports HEVC + AV1 only (got {codec:?})");
@@ -787,7 +851,8 @@ impl VulkanVideoEncoder {
                 bitrate_bps.max(1_000_000),
                 want_rgb,
                 native_nv12,
-                hdr,
+                ten_bit,
+                src_rgb_fmt,
             )
         }
     }
@@ -805,6 +870,7 @@ impl VulkanVideoEncoder {
         native_nv12: bool,
         // NOT `hdr`: this fn already binds that name to the parameter-set HEADER bytes below.
         ten_bit: bool,
+        src_rgb_fmt: vk::Format,
     ) -> Result<Self> {
         use super::vk_av1_encode as av1b;
         use super::vk_valve_rgb as vrgb;
@@ -863,7 +929,7 @@ impl VulkanVideoEncoder {
         let rgb_probe = if native_nv12 {
             Err("not-probed(native NV12 source selected)")
         } else {
-            probe_rgb_direct(&instance, &vq_inst, pd, codec_op, av1)
+            probe_rgb_direct(&instance, &vq_inst, pd, codec_op, av1, ten_bit, src_rgb_fmt)
         };
         let rgb_cfg: Option<RgbDirect> = match (&rgb_probe, want_rgb) {
             (Ok((x, y)), true) => {
@@ -925,11 +991,7 @@ impl VulkanVideoEncoder {
             perform_encode_rgb_conversion: vk::TRUE,
         };
         let mut h265_profile =
-            vk::VideoEncodeH265ProfileInfoKHR::default().std_profile_idc(if ten_bit {
-                vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN_10
-            } else {
-                vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN
-            });
+            vk::VideoEncodeH265ProfileInfoKHR::default().std_profile_idc(h265_profile_idc(ten_bit));
         let mut av1_profile = av1b::VideoEncodeAV1ProfileInfoKHR {
             s_type: av1b::stype(av1b::ST_PROFILE_INFO),
             p_next: std::ptr::null(),
@@ -946,11 +1008,7 @@ impl VulkanVideoEncoder {
         // below with VIDEO_PROFILE_FORMAT_NOT_SUPPORTED, which fails the open — and a failed
         // Vulkan open falls back to libav VAAPI in `open_amd_intel`. That is the whole 10-bit
         // capability gate: no separate probe, and no way to reach a half-configured session.
-        let depth = if ten_bit {
-            vk::VideoComponentBitDepthFlagsKHR::TYPE_10
-        } else {
-            vk::VideoComponentBitDepthFlagsKHR::TYPE_8
-        };
+        let depth = component_depth(ten_bit);
         let mut profile = vk::VideoProfileInfoKHR::default()
             .video_codec_operation(codec_op)
             .chroma_subsampling(vk::VideoChromaSubsamplingFlagsKHR::TYPE_420)
@@ -1175,7 +1233,7 @@ impl VulkanVideoEncoder {
         let mut rgb_sci = vrgb::VideoEncodeSessionRgbConversionCreateInfoVALVE {
             s_type: vrgb::stype(vrgb::ST_SESSION_CREATE_INFO),
             p_next: std::ptr::null(),
-            rgb_model: vrgb::MODEL_YCBCR_709,
+            rgb_model: rgb_model_for(ten_bit),
             rgb_range: vrgb::RANGE_NARROW,
             x_chroma_offset: rgb_cfg.as_ref().map_or(0, |c| c.x_offset),
             y_chroma_offset: rgb_cfg.as_ref().map_or(0, |c| c.y_offset),
@@ -1184,7 +1242,7 @@ impl VulkanVideoEncoder {
             .queue_family_index(encode_family)
             .video_profile(&profile)
             .picture_format(if rgb_cfg.is_some() {
-                vk::Format::B8G8R8A8_UNORM
+                src_rgb_fmt
             } else {
                 yuv_format(ten_bit)
             })
@@ -1290,6 +1348,7 @@ impl VulkanVideoEncoder {
                 av1_caps.max_level,
                 av1_superblock128,
                 quality_level,
+                ten_bit,
             )?
         } else {
             let (p, hdr) = build_parameters_h265(
@@ -1466,7 +1525,7 @@ impl VulkanVideoEncoder {
                 rgb_cfg
                     .as_ref()
                     .is_some_and(|c| c.padded)
-                    .then_some(vk::Format::B8G8R8A8_UNORM),
+                    .then_some(src_rgb_fmt),
                 ten_bit,
                 guard.frames.last_mut().expect("frame just pushed"),
             )?;
@@ -1527,6 +1586,7 @@ impl VulkanVideoEncoder {
             cpu_expand: Vec::new(),
             rgb: rgb_cfg,
             native_nv12,
+            ten_bit,
             pending_bitrate: None,
             width: w,
             height: h,
@@ -1687,7 +1747,8 @@ impl VulkanVideoEncoder {
         ch: u32,
     ) -> Result<(vk::Image, vk::DeviceMemory, vk::ImageView)> {
         if self.native_nv12 {
-            let mut ps = NativeProfileStack::new(codec_op_for(self.codec == Codec::Av1));
+            let mut ps =
+                NativeProfileStack::new(codec_op_for(self.codec == Codec::Av1), self.ten_bit);
             let profile = *ps.wire(self.codec == Codec::Av1);
             let arr = [profile];
             let mut plist = vk::VideoProfileListInfoKHR::default().profiles(&arr);
@@ -1716,7 +1777,7 @@ impl VulkanVideoEncoder {
                 None,
             )
         } else if self.rgb.is_some() {
-            let mut ps = RgbProfileStack::new(codec_op_for(self.codec == Codec::Av1));
+            let mut ps = RgbProfileStack::new(codec_op_for(self.codec == Codec::Av1), self.ten_bit);
             let profile = *ps.wire(self.codec == Codec::Av1);
             let arr = [profile];
             let mut plist = vk::VideoProfileListInfoKHR::default().profiles(&arr);
@@ -1873,7 +1934,7 @@ impl VulkanVideoEncoder {
                 // usage, and shared with the encode queue (the compute queue only copies into
                 // it; the semaphore orders the hand-off, CONCURRENT avoids a QFOT).
                 let av1 = self.codec == Codec::Av1;
-                let mut ps = RgbProfileStack::new(codec_op_for(av1));
+                let mut ps = RgbProfileStack::new(codec_op_for(av1), self.ten_bit);
                 let profile = *ps.wire(av1);
                 let arr = [profile];
                 let mut plist = vk::VideoProfileListInfoKHR::default().profiles(&arr);
@@ -4232,7 +4293,7 @@ impl Drop for VulkanVideoEncoder {
 mod build;
 use self::build::{
     align_up, build_parameters_av1, build_parameters_h265, make_frame, make_video_image,
-    probe_rgb_direct,
+    probe_rgb_direct, rgb_model_for,
 };
 
 #[cfg(test)]

crates/pf-encode/src/lib.rs

@@ -362,20 +362,22 @@ fn open_video_backend_linux(
     // the stream never dies over the new path. `format`/`bit_depth`/`chroma` only matter to VAAPI
     // — the Vulkan backend imports the dmabuf and does its own CSC.
     let open_amd_intel = || -> Result<(Box<dyn Encoder>, &'static str)> {
-        // HDR (10-bit + a PQ/BT.2020 capture format) takes the Vulkan backend for **HEVC**: its
-        // compute CSC has a 10-bit BT.2020 variant and the session opens Main10. That keeps the
-        // two things an HDR session would otherwise lose here — real RFI recovery and the
-        // compute-CSC cursor blend, which is the only way a gamescope pointer reaches the stream
-        // (gamescope has no embedded-cursor mode to fall back to).
+        // HDR (10-bit + a PQ/BT.2020 capture format) keeps this backend for EITHER codec, as
+        // long as the device says it can: the compute CSC has a BT.2020 10-bit variant, the EFC
+        // has a BT.2020 conversion, and the session opens Main10 / AV1-at-10. That keeps the two
+        // things an HDR session would otherwise lose — real RFI recovery, and the compute CSC's
+        // cursor blend, which is the only way a gamescope pointer reaches the stream (gamescope
+        // has no embedded-cursor mode to fall back to).
         //
-        // AV1 10-bit stays on libav VAAPI: encode-side driver coverage for it is far thinner than
-        // HEVC Main10, and this is not the place to gamble a session open. A device that cannot
-        // do Main10 either fails `get_physical_device_video_capabilities` inside the open and
-        // lands on the VAAPI fallback below — the same net as any other unsupported config.
+        // `vulkan_encode_available_at` is the device's own answer to the same profile query the
+        // open will make, so this is a prediction that cannot disagree with reality — and a `no`
+        // routes to libav VAAPI here rather than burning a failed session open first.
+        #[cfg(feature = "vulkan-encode")]
+        let ten_bit_session = bit_depth == 10 && format.is_hdr_rgb10();
         #[cfg(feature = "vulkan-encode")]
         if matches!(codec, Codec::H265 | Codec::Av1)
             && vulkan_encode_enabled()
-            && !(bit_depth == 10 && format.is_hdr_rgb10() && codec != Codec::H265)
+            && vulkan_encode_available_at(codec, ten_bit_session)
         {
             match vulkan_video::VulkanVideoEncoder::open(
                 codec,
@@ -1046,14 +1048,9 @@ pub fn linux_hdr_cuda_ok() -> bool {
 ///
 /// `cuda_planned` is the caller's prediction of a CUDA capture payload (NVIDIA + zero-copy — the
 /// prediction `SessionPlan` already makes); `ten_bit` the negotiated depth. Both shift the
-/// dispatch: a CPU payload keeps NVIDIA on libav NVENC (no blend), and a 10-bit **AV1** session
-/// skips Vulkan Video for libav VAAPI (no blend) — 10-bit HEVC does not, its compute CSC has a
-/// BT.2020 Main10 variant that blends like the 8-bit one.
-///
-/// Whether the DEVICE can encode Main10 is only settled when the session opens (the profile query
-/// is what answers it); a device that cannot falls back to VAAPI there, and `open_video`'s
-/// `blends_cursor` backstop logs the divergence. Same shape as every other open-time fallback
-/// this prediction cannot see.
+/// dispatch: a CPU payload keeps NVIDIA on libav NVENC (no blend), and a 10-bit session keeps
+/// Vulkan Video (which blends) only where the device advertises a 10-bit profile for that codec —
+/// `vulkan_encode_available_at`, the same query the open makes.
 #[cfg(target_os = "linux")]
 pub fn cursor_blend_capable(codec: Codec, cuda_planned: bool, ten_bit: bool) -> bool {
     // A negotiated PyroWave session routes to that backend before the pref is consulted
@@ -1076,11 +1073,11 @@ pub fn cursor_blend_capable(codec: Codec, cuda_planned: bool, ten_bit: bool) ->
         // the device probe runs last (it opens a Vulkan instance, cached per GPU+codec).
         #[cfg(feature = "vulkan-encode")]
         {
+            // Mirrors `open_amd_intel` exactly, depth included — the device answers whether a
+            // 10-bit profile for this codec exists, so the prediction and the open agree.
             matches!(codec, Codec::H265 | Codec::Av1)
                 && vulkan_encode_enabled()
-                // 10-bit is HEVC Main10 only on this backend — mirrors `open_amd_intel`.
-                && !(ten_bit && codec != Codec::H265)
-                && vulkan_encode_available(codec)
+                && vulkan_encode_available_at(codec, ten_bit)
         }
         #[cfg(not(feature = "vulkan-encode"))]
         {
@@ -1134,30 +1131,59 @@ fn cursor_blend_capable_for(
 /// reports what actually did.
 #[cfg(all(target_os = "linux", feature = "vulkan-encode"))]
 fn vulkan_encode_available(codec: Codec) -> bool {
+    vulkan_encode_caps(codec).supported
+}
+
+/// Can the Vulkan Video backend encode `codec` at this depth on the selected GPU? The gate that
+/// decides whether an HDR session gets the good path (real RFI + the compute CSC's cursor blend)
+/// or falls back to libav VAAPI — asked per codec, because a device can advertise HEVC Main10 and
+/// still decline 10-bit AV1.
+#[cfg(target_os = "linux")]
+fn vulkan_encode_available_at(codec: Codec, ten_bit: bool) -> bool {
+    let caps = vulkan_encode_caps(codec);
+    caps.supported
+        && if ten_bit {
+            caps.ten_bit
+        } else {
+            caps.eight_bit
+        }
+}
+
+/// The device's Vulkan Video encode capabilities for `codec`, cached per (selected GPU, codec) —
+/// a web-console GPU change re-probes on the new adapter before the next Welcome, mirroring
+/// `can_encode_444`/`can_encode_10bit`. The probe creates and destroys its own Vulkan instance,
+/// so it is worth caching but safe to call from anywhere.
+#[cfg(target_os = "linux")]
+fn vulkan_encode_caps(codec: Codec) -> vulkan_video::VulkanEncodeCaps {
     use std::collections::HashMap;
     use std::sync::{Mutex, OnceLock};
-    static CACHE: OnceLock<Mutex<HashMap<(String, &'static str), bool>>> = OnceLock::new();
+    #[allow(clippy::type_complexity)]
+    static CACHE: OnceLock<Mutex<HashMap<(String, &'static str), vulkan_video::VulkanEncodeCaps>>> =
+        OnceLock::new();
     let key = (pf_gpu::selection_key(), codec.label());
     let cache = CACHE.get_or_init(|| Mutex::new(HashMap::new()));
     if let Some(v) = cache.lock().unwrap().get(&key) {
         return *v;
     }
-    let verdict = vulkan_video::probe_encode_support(codec);
-    let ok = verdict.is_ok();
-    match &verdict {
-        Ok(()) => tracing::info!(
+    let caps = vulkan_video::probe_encode_caps(codec);
+    if caps.supported {
+        tracing::info!(
             ?codec,
-            "Vulkan Video encode probed OK — producer-native NV12 capture is eligible"
-        ),
-        Err(why) => tracing::info!(
+            eight_bit = caps.eight_bit,
+            ten_bit = caps.ten_bit,
+            "Vulkan Video encode probed — producer-native NV12 capture is eligible, and a 10-bit \
+             session keeps this backend (real RFI + the compute CSC's cursor blend) where the \
+             device accepts the profile"
+        );
+    } else {
+        tracing::info!(
             ?codec,
-            why = *why,
-            "Vulkan Video encode unavailable — keeping the packed-RGB capture negotiation \
-             (the native-NV12 path has no VAAPI fallback)"
-        ),
+            "Vulkan Video encode unavailable (no encode queue for this codec) — keeping the \
+             packed-RGB capture negotiation (the native-NV12 path has no VAAPI fallback)"
+        );
     }
-    cache.lock().unwrap().insert(key, ok);
-    ok
+    cache.lock().unwrap().insert(key, caps);
+    caps
 }
 
 /// Cheap, side-effect-free NVIDIA-presence probe for the `auto` backend selector: the NVIDIA
@@ -1503,13 +1529,16 @@ pub fn can_encode_10bit(codec: Codec) -> bool {
     let supported = {
         #[cfg(target_os = "linux")]
         {
-            // NVENC (libav, the HDR P010 swscale path) or VAAPI (P010 upload / dmabuf graph),
-            // probed by opening a tiny real Main10 encoder — the same honesty contract as
-            // `can_encode_444`. Vulkan-video and the direct-SDK CUDA path stay 8-bit; a 10-bit
-            // session routes around them (see `open_video_backend`). NOTE: encode capability is
-            // only half the Linux gate — the capture side (GNOME 50+ portal monitor in HDR mode)
-            // is resolved separately by the host (`capturer_supports_hdr` / the GameStream RTSP
-            // honor), since this probe can't know what the compositor will negotiate.
+            // NVENC (libav, the HDR P010 swscale path — the direct-SDK CUDA path encodes the
+            // same GPU's Main10, so this is a truthful proxy for it too) or, on AMD/Intel, VAAPI
+            // OR Vulkan Video. Both of the latter are asked because either can serve the session:
+            // `open_amd_intel` tries Vulkan first and falls back to VAAPI, so 10-bit is available
+            // if EITHER says yes, and answering `false` when only one does would strand a
+            // perfectly encodable HDR session at 8 bits. NOTE: encode capability is only half the
+            // Linux gate — the capture side (a gamescope HDR output, or a GNOME 50+ portal
+            // monitor in HDR mode) is resolved separately by the host
+            // (`capture::capturer_supports_hdr_for` / the GameStream RTSP honor), since this
+            // probe can't know what the compositor will negotiate.
             // Resolve through the SAME helper `can_encode_444` uses (and which mirrors
             // `open_video`'s dispatch): `linux_auto_is_vaapi` ignores `encoder_pref`, so on a box
             // that forces a backend — e.g. `encoder_pref = "vaapi"` on an NVIDIA host — this probe
@@ -1517,7 +1546,17 @@ pub fn can_encode_10bit(codec: Codec) -> bool {
             // depth (plus the HDR/SDR colour label derived from it) would describe a backend that
             // never runs. That is exactly the dishonesty this probe exists to prevent.
             if linux_zero_copy_is_vaapi() {
-                vaapi::probe_can_encode_10bit(codec)
+                let vulkan10 = {
+                    #[cfg(feature = "vulkan-encode")]
+                    {
+                        vulkan_encode_enabled() && vulkan_encode_available_at(codec, true)
+                    }
+                    #[cfg(not(feature = "vulkan-encode"))]
+                    {
+                        false
+                    }
+                };
+                vulkan10 || vaapi::probe_can_encode_10bit(codec)
             } else {
                 linux::probe_can_encode_10bit(codec)
             }

docs-site/content/docs/gamescope.md

@@ -158,10 +158,11 @@ Two things to know:
 - **SDR content rides the same PQ stream.** The desktop, the Steam overlay and SDR games are mapped
   into the HDR container at `PUNKTFUNK_GAMESCOPE_SDR_NITS` (400 by default). If white looks too
   bright or too dim on your TV, that is the knob.
-- **HDR picks HEVC on AMD and Intel.** A 10-bit AV1 session falls back to a slower encode path
-  that also can't draw the mouse pointer gamescope leaves out of its capture — so if you stream
-  Gaming Mode in HDR from an AMD or Intel box, leave the codec on HEVC. NVIDIA is unaffected
-  either way.
+- **On AMD and Intel, HDR follows what the GPU can encode.** The host asks the driver whether it
+  can encode your codec at 10 bits and picks the fast path when it can — which current AMD and
+  Intel GPUs do for HEVC, and newer ones for AV1 too. On a GPU that declines, the session still
+  streams HDR through a slower path, but the mouse pointer gamescope leaves out of its capture
+  can't be drawn back in there. `punktfunk-host hdr-probe` reports what your box answered.
 
 ## Known limits
 

docs/releases/v0.21.0.md

@@ -27,9 +27,7 @@ the Punktfunk sysext; there's an Arch package, a NixOS option, and a build scrip
 else. `punktfunk-host hdr-probe` tells you exactly which pieces are in place.
 
 Opt-in for this release while it soaks: without the knob, or without the extra package, the
-gamescope path streams SDR exactly as before. One thing to know on AMD and Intel: keep the codec on
-HEVC — a 10-bit AV1 session falls back to a slower encode path that also loses the on-screen mouse
-pointer. NVIDIA is unaffected either way.
+gamescope path streams SDR exactly as before.
 
 ## Fixed: newer KDE Plasma silently fell back to a slower, less reliable way to arrange displays
 
@@ -46,8 +44,10 @@ On current KDE Plasma (6.7 and newer), Punktfunk's direct way of talking to the
 - **KWin's in-process output-management path now also binds `kde_output_device_registry_v2`**, not just the per-output `kde_output_device_v2` globals it originally shipped with — KWin 6.7 stopped advertising the latter, so the module written specifically to avoid shelling out to `kscreen-doctor` (0.19.x) saw zero devices and silently degraded to it on every session. Both models are supported now; registry-sourced devices arrive one Wayland round-trip later, so the handshake gains one conditional extra barrier.
 - **gamescope HDR is decided statically, before the display exists.** The punktfunk/1 Welcome fixes a session's bit depth up front and cannot take it back (PQ frames on an 8-bit encoder are a deliberate hard error), so the capability answer is the identity of the gamescope binary the host will spawn — a `+pfhdr` marker in its `--version` banner, probed once per boot — never an optimistic negotiation. The capture-side gate became source-aware (`capturer_supports_hdr_for(compositor)`), the HDR negotiation-failure latch became per-source (a wedged monitor mirror no longer disables a gamescope session's HDR, or vice versa), and the keep-alive reuse key gained `hdr` so a display brought up SDR can never be handed to an HDR session. The GameStream plane's live BT.2100 monitor probe is now scoped to the portal source — a headless gamescope box has no monitor to be in HDR mode.
 - **The gamescope patch mirrors code already in gamescope's tree.** Its PipeWire node additionally offers `xRGB_210LE`/`xBGR_210LE` with MANDATORY SMPTE ST.2084 + BT.2020 properties, mapped to `DRM_FORMAT_XRGB2101010`/`XBGR2101010` — the same 10-bit capture texture the HDR AVIF screenshot path allocates — and `paint_pipewire()` composites into them with the HDR screenshot LUT set and `EOTF_PQ`, which is exactly the `bHDRScreenshot` branch. The new formats are listed last, so every existing consumer keeps negotiating the 8-bit stream bit-for-bit, and the fixed PQ container is deliberately *not* conditional on the focused app being HDR (a stream's colourimetry must not follow what the game happens to render).
-- **Vulkan Video learned 10-bit** (HEVC Main10), which is what keeps AMD/Intel HDR on the good path: an HDR session opens a Main10 video profile with 10-bit component depths, a `G10X6…3PACK16` picture and DPB, an SPS carrying `bit_depth_*_minus8 = 2` and the BT.2020/PQ CICP triplet, and a new `rgb2yuv10.comp` — the 8-bit BT.709 shader's twin, doing a pure BT.2020 NCL matrix on the already-PQ-encoded samples (there is no transfer function to apply, and applying one would be wrong) and writing 10-bit values into the high bits of `R16`/`RG16` scratch planes that are size-compatible with the picture's. That keeps the two things HDR would otherwise cost on this vendor: real RFI loss recovery, and the compute CSC's cursor blend — the only way a gamescope pointer reaches the stream at all. 10-bit **AV1** still routes to libav VAAPI: encode-side driver coverage for it is far thinner than Main10. A device that can't do Main10 fails the profile query inside the open and lands on the same VAAPI fallback as any other unsupported config.
-- **AMD/Intel HDR needs no new encoder code on the VAAPI fallback**; NVIDIA gained a zero-copy leg. The VAAPI path already ingested XR30 dmabufs into `format=p010:out_color_matrix=bt2020`. On NVIDIA the packed 10-bit frame now travels LINEAR dmabuf → Vulkan bridge → CUDA → NVENC `ARGB10`/`ABGR10`, letting NVENC do the BT.2020 CSC itself: no host CSC pass, no depth loss, and the cursor-blend compute shader gained two 10-bit modes so the pointer survives. The tiled EGL de-tile blit is still 8-bit and HDR never routes through it. A host without the direct-SDK NVENC backend keeps HDR on the CPU path, since libav's HDR route swscales into a P010 hardware frame that a packed-10-bit CUDA buffer cannot fill.
+- **Vulkan Video learned 10-bit**, which is what keeps AMD/Intel HDR on the good path: an HDR session opens a 10-bit video profile (HEVC Main10 / AV1 Main at 10 bits) with a `G10X6…3PACK16` picture and DPB, headers carrying the depth and the BT.2020/PQ CICP triplet — an SPS `bit_depth_*_minus8 = 2` for HEVC, `high_bitdepth` plus the matching sequence-header OBU bits for AV1 — and a new `rgb2yuv10.comp`: the 8-bit BT.709 shader's twin, doing a pure BT.2020 NCL matrix on the already-PQ-encoded samples (there is no transfer function to apply, and applying one would be wrong) and writing 10-bit values into the high bits of `R16`/`RG16` scratch planes that are size-compatible with the picture's. That keeps the two things HDR would otherwise cost on this vendor: real RFI loss recovery, and the compute CSC's cursor blend — the only way a gamescope pointer reaches the stream at all.
+- **The Vulkan encode backend is now capability-probed per codec AND depth**, mirroring what the direct-SDK NVENC path already does with its GUID probe. `vkGetPhysicalDeviceVideoCapabilitiesKHR` is asked against the very profile the session open will build, so the dispatcher's prediction cannot disagree with reality: a device that can encode 10-bit keeps the Vulkan path, one that can't routes to libav VAAPI *before* burning a failed session open, and `can_encode_10bit` reports the union of what VAAPI and Vulkan Video can do rather than VAAPI's answer alone (which was under-reporting 10-bit on hardware that could do it).
+- **The zero-CSC RGB-direct (EFC) source works in HDR too.** The `VK_VALVE_video_encode_rgb_conversion` probe now asks for the BT.2020 model and the captured 10-bit packed-RGB format instead of assuming BT.709/BGRA, and the session selects the matching model — so an HDR session with no pointer to composite (the GameStream desktop mirror) hands the captured buffer straight to the encoder's fixed-function front end and runs no host CSC at all. Sessions that DO composite a pointer keep the compute CSC, as before: the EFC cannot blend.
+- **NVIDIA gained a zero-copy HDR leg**, and the VAAPI fallback needed no new encoder code. The VAAPI path already ingested XR30 dmabufs into `format=p010:out_color_matrix=bt2020`. On NVIDIA the packed 10-bit frame now travels LINEAR dmabuf → Vulkan bridge → CUDA → NVENC `ARGB10`/`ABGR10`, letting NVENC do the BT.2020 CSC itself: no host CSC pass, no depth loss, and the cursor-blend compute shader gained two 10-bit modes so the pointer survives. The tiled EGL de-tile blit is still 8-bit and HDR never routes through it. A host without the direct-SDK NVENC backend keeps HDR on the CPU path, since libav's HDR route swscales into a P010 hardware frame that a packed-10-bit CUDA buffer cannot fill.
 - **CI-only fix:** the winget release-verification step's `envs:` allow-list was a step-level sibling of `with:`/`env:` instead of nested inside `with:`, so `appleboy/ssh-action` never actually received it as an input and the step kept failing on every tag. Moved to match how `REGISTRY_TOKEN` is already forwarded elsewhere.
 
 See [v0.20.1](v0.20.1.md) for the fixes carried by that release (Windows install/winget, GameStream opt-in default, the gamescope Game Mode takeover hardening, a laptop-panel stall fix, and the PyroWave latency-creep bundle) — all included here too, since this release supersedes it.