Three fixes to the same mistake: deciding what the Vulkan Video backend can do
from a table in our heads rather than from the driver, and routing everything
that didn't fit to libav VAAPI — where a session loses real RFI recovery and
the cursor blend for no reason the hardware asked for.
**Capability probe, per codec AND depth.** `probe_encode_support`'s "is there
an encode queue" boolean becomes `VulkanEncodeCaps { supported, eight_bit,
ten_bit }`, answered by `vkGetPhysicalDeviceVideoCapabilitiesKHR` against the
very profile chain the session open builds. So the dispatcher's prediction
cannot disagree with reality: a capable device keeps the Vulkan path, an
incapable one routes to VAAPI BEFORE burning a failed open, and the
cursor-blend mirror stays honest for free. This is the shape the direct-SDK
NVENC path already uses for its codec GUIDs.
**AV1 10-bit.** It was excluded on a guess about driver coverage; now the
device answers. `color_config()` carries `high_bitdepth` + the BT.2020/PQ CICP
triplet in both the `StdVideoAV1ColorConfig` and the sequence-header OBU we
bit-pack ourselves — they must stay identical or the driver's frame OBUs parse
against a header we didn't write. `high_bitdepth` sits BEFORE the CICP bytes,
so getting it wrong doesn't just mislabel the depth, it puts every following
field one bit out of phase; the new test reads the packed bits back.
**Zero-CSC RGB-direct in HDR.** The EFC probe assumed BT.709 and BGRA. It now
asks for the model this session's colourimetry needs (`MODEL_YCBCR_2020` for
10-bit — the extension has always had it) and for the CAPTURED format as an
encode-source format, and the session create-info selects the matching model.
An HDR session with no pointer to composite therefore hands the captured
buffer straight to the fixed-function front end and runs no host CSC at all.
Sessions that DO composite a pointer keep the compute CSC, unchanged: the EFC
cannot blend, and that rule outranks everything.
Also: `can_encode_10bit` on AMD/Intel now reports the union of VAAPI's and
Vulkan Video's answers instead of VAAPI's alone. `open_amd_intel` tries Vulkan
first and falls back, so either one being able to encode Main10 makes the
session 10-bit-capable — answering `false` because only one of them said yes
stranded encodable HDR sessions at 8 bits.
enricobuehler
cb4690f216