gamescope's capture textures are mappable, hence linear-tiled, and NVIDIA does
not implement linear-tiled STORAGE for A2R10G10B10_UNORM_PACK32. Upstream says
it plainly in rendervulkan.cpp: "imageStore lands in XBGR order there, swapping
R/B". So the composite writes XBGR bytes into a buffer still LABELLED
XRGB2101010, and our patch's spa_format_to_drm() derives that label from the
negotiated SPA format alone, never asking the hardware what it can actually
write.
The host then believed the label, correctly at every step:
xRGB_210LE -> PixelFormat::X2Rgb10 -> NV_ENC_BUFFER_FORMAT_ARGB10. DRM
XRGB2101010 really is "B in the low 10 bits" and NVENC ARGB10 really is "B in
the lowest 10 bits"; the Windows twin (R10G10B10A2 -> ABGR10) is correct by the
same rule. Every mapping audits clean because the label was right and only the
CONTENT was wrong -- which is why this survived a full trace of both ends.
Fix the preference host-side: offer xBGR_210LE FIRST. The first compatible
consumer pod wins, so that is what a gamescope session lands on, and an
XBGR2101010 texture is one NVIDIA writes in its own order -- label and content
agree. It costs nothing elsewhere: A2B10G10R10_UNORM_PACK32 is the universally
supported packed-10 format, it is what upstream's own fallback picks, and
X2Bgr10 has a first-class encoder path (NVENC ABGR10, VAAPI X2BGR10LE).
xRGB_210LE stays as the second pod so a producer offering only it can still
negotiate HDR instead of dropping to the SDR downgrade.
Doing it here rather than in the patch set is deliberate: the real fix is for
spa_format_to_drm() to offer only what vulkan_get_rgb10_capture_format()
reports, but that function landed after 3.16.25 and the pin is
3.16.25-7-g60561e2+pfhdr4 (0 "2101010" strings in the shipped binary), so the
deployed gamescope cannot self-correct. This ships in the host binary with no
gamescope rebuild.
Field-confirmed on the RTX 5070 Ti Bazzite host with 0.26.0, and confirmed
host-side rather than client-side by reproducing the identical swap from two
unrelated clients (16" MacBook Pro and Mac Studio). SDR was never affected --
it takes no packed-10 path.
Gate (pf-lxcheck2, linux/amd64): fmt clean, clippy --all-targets -D warnings
clean, cargo test -p pf-capture 60 passed / 0 failed incl. the new
hdr_offers_xbgr_before_xrgb order pin.
enricobuehler
97928516a0