The Vulkan Video backend was 8-bit for no structural reason — the API has
`VK_VIDEO_COMPONENT_BIT_DEPTH_10_BIT` and `PROFILE_IDC_MAIN_10` in the very
fields this pinned to 8 and MAIN, and AMD VCN and Intel both encode Main10.
It was six hardcoded sites, and the cost of leaving them was paid twice over:
an HDR session had to take libav VAAPI, losing real RFI loss recovery AND the
compute CSC's cursor blend — which on gamescope is the only way the pointer
reaches the stream at all, since gamescope has no embedded-cursor mode.
An HDR session now opens a Main10 profile with 10-bit component depths, a
`G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16` picture + DPB, and an SPS carrying
`bit_depth_*_minus8 = 2` with the BT.2020/PQ CICP triplet instead of BT.709.
`rgb2yuv10.comp` is the CSC's twin, and the two interesting parts of it are:
* it is a PURE 3x3 matrix. The samples arrive already PQ-encoded (gamescope
composites into the PQ container), so BT.2020 NCL applies to the code values
as they are — there is no transfer function to apply here and applying one
would be wrong;
* the scratch planes are `R16`/`RG16`, not the picture's plane formats. The
10-bit ycbcr plane formats are not storage-image formats, so the shader
writes the value into the HIGH bits by hand (`code10 << 6`, hence the
`64/65535` factor and not `1/1023`) into planes that are merely
SIZE-compatible with the picture's — which is all `vkCmdCopyImage` requires.
Scope and safety:
* HEVC only. AV1 10-bit encode has far thinner driver coverage, and a session
open is not the place to gamble on it — those stay on VAAPI, as does a device
that fails the Main10 profile query inside the open (the pre-existing "failed
Vulkan open falls back to VAAPI" net, no new probe needed).
* HDR pins the compute-CSC arm over the EFC RGB-direct one, which the EFC could
not serve anyway: its fixed-function conversion is 8-bit BT.709 narrow with
no knob for BT.2020.
* `open_inner` binds `hdr` to the parameter-set HEADER bytes, so the depth flag
is `ten_bit` there — the one name collision this change had to route around.
enricobuehler
479f0965ee