Every NVIDIA gamescope HDR stream had red and blue swapped — and a sysext step added in a release was unreachable forever #143

Merged

enricobuehler merged 2 commits from worktree-hdr-rb-swap-nvidia into main 2026-08-09 15:34:05 +00:00

Conversation 0 Commits 2 Files Changed 3

+104 -7

enricobuehler commented 2026-08-09 15:28:12 +00:00

Owner

Copy Link

Copy Source

Two field bugs from .41 (Bazzite, RTX 5070 Ti) on 0.26.0-1. Both are fixed and verified on the box.

1. Red and blue swapped on every NVIDIA HDR stream

fix(pf-capture): every NVIDIA HDR stream had red and blue swapped

gamescope allocates its PipeWire capture textures mappable, hence linear-tiled, and composites into them with imageStore. NVIDIA does not implement linear-tiled STORAGE for A2R10G10B10_UNORM_PACK32. Upstream says it plainly in rendervulkan.cpp:

// XRGB2101010 storage images are an optional Vulkan feature that NVIDIA hardware
// doesn't support, and imageStore lands in XBGR order there, swapping R/B.

So the composite writes XBGR bytes into a buffer still labelled XRGB2101010, and our patch's spa_format_to_drm() derives that label from the negotiated SPA format alone, never asking the hardware what it can actually write.

Why this resisted diagnosis: every mapping in the chain is individually correct. DRM XRGB2101010 really is "B in the low 10 bits"; NVENC ARGB10 really is "B in the lowest 10 bits"; the Windows twin R10G10B10A2 → ABGR10 is correct by the same rule. All of them audit clean, because the label was right and only the content was wrong.

⚠️ The trap worth remembering: do not apply the "reverse the name" rule to _PACK32 formats. It is correct for Vulkan's 8-bit byte-ordered formats (XRGB8888 → B8G8R8A8) and wrong for _PACK32, which is word-ordered like DRM.

The fix

Offer xBGR_210LE first (new HDR_FORMAT_ORDER, consumed by pipewire.rs). The first compatible consumer pod wins, so that is what a gamescope session lands on, and an XBGR2101010 texture is one NVIDIA writes in its own order — label and content agree. xRGB_210LE stays as the second pod so a producer offering only it can still negotiate HDR rather than dropping to the SDR downgrade. Test hdr_offers_xbgr_before_xrgb pins the order.

Done host-side deliberately: the upstream-correct fix is for spa_format_to_drm() to follow vulkan_get_rgb10_capture_format(), but that function landed after 3.16.25 and our pin is 3.16.25-7-g60561e2+pfhdr4 (0 2101010 strings in the shipped binary), so the deployed gamescope cannot self-correct. This ships in the host binary with no gamescope rebuild. Moving the pin is filed as follow-up.

Evidence

A probe reproducing gamescope's own vulkan_get_rgb10_capture_format() check, run on both vendors:

GPU	A2R10G10B10 (XRGB)	A2B10G10R10 (XBGR)
RTX 5070 Ti	NO — storage=false, linearTilingFeatures=0x1dc01	YES — 0x1dc03
Radeon 780M (RADV Phoenix)	YES — 0x1dd83	YES — 0x1dd83
llvmpipe (control)	NO — storage=false	YES

On NVIDIA the only differing bit is STORAGE_IMAGE — precisely the capability imageStore needs.

Confirmed host-side, not client-side, by reproducing the identical swap from two unrelated clients (16" MacBook Pro and Mac Studio). SDR was never affected — it takes no packed-10 path.

On glass after the fix (locally built host running on .41):

HDR capture: offering xBGR_210LE/xRGB_210LE LINEAR dmabufs ...
pipewire format negotiated ... spa_format=VideoFormat::xBGR_210LE mapped=Some(X2Bgr10) hdr=true
NVENC CUDA session ready mode=5120x1440@240 bit_depth=10 fmt=NV_ENC_BUFFER_FORMAT_ABGR10

Colours confirmed correct by the reporter.

AMD regression check

Both mechanisms that could regress were verified directly:

1. Compositor side — AMD supports A2B10G10R10 linear STORAGE+SAMPLED with byte-identical feature bits to A2R10G10B10 (0x1dd83), so the format this change selects is composited correctly. XBGR is also the more broadly supported of the two (it is what upstream's own fallback picks, and the only one llvmpipe supports).
2. Encode side — handling is symmetric across every Linux backend (vaapi.rs:63/64 + 825/826, vk_util.rs:88/89, nvenc_cuda.rs:587/589 + 634/635, mod.rs:47/48 and a shared match arm at :175); the only single-format sites are NVENC/Vulkan test fixtures. On the AMD box, ffmpeg lists x2bgr10le as IO (same as x2rgb10le) and a real swscale conversion produced the expected 307,200 bytes.

Caveat stated plainly: a full end-to-end gamescope HDR stream on AMD was not run — the AMD box is a headless VM without the patched gamescope. What was verified is the two mechanisms by which this change could regress AMD, each directly.

2. A sysext post_merge step added in a release was unreachable forever

fix(packaging): a post_merge step added in a release was unreachable forever

A sysext upgrade is driven by the script from the old image — /usr/bin/punktfunk-sysext is replaced by the very systemd-sysext refresh that runs mid-upgrade — so a post_merge step added in the new release is executed by nobody. The old script does not have it, and the new script never gets a turn: from then on update matches the "already on $cur" branch and returns before post_merge. The step is permanently unreachable on exactly the installs that need it, and nothing says so.

The casualty was the punktfunk group, which post_merge learned to create in 0.26.0 (62a6fa9f). 0.25.0's script drove the upgrade, so the group was never created, and every punktfunk-sysext update since has said "nothing to do". pf-dm-helper gates on membership in that group, so it refused every caller — pkexec authorised it and the helper then declined itself — and every managed gamescope takeover fell back to "stopping the display manager needs privilege", leaving sddm's autologin Relogin loop churning logind sessions for the whole stream.

Changes: re-run post_merge when already current (everything in it is idempotent, so convergence is the honest behaviour and "nothing to do" was a lie about host state); add an explicit reapply verb; and print the membership hint, since creating the group is necessary but not sufficient and the difference is invisible until a stream fails. Joining stays opt-in by design — writing vhci attach materialises an arbitrary emulated USB device — so post_merge names the exact usermod rather than doing it. Matched with grep -qx so punktfunk-update does not read as punktfunk.

Verified live on .41: group absent → reapply → punktfunk:x:965: created; hint fired naming the user and went silent after joining; pf-dm-helper with a member UID now reaches verb dispatch while a non-member is still refused.

Gates

pf-lxcheck2, linux/amd64: cargo fmt --check clean, cargo clippy --locked --all-targets -p pf-capture -- -D warnings clean, cargo test -p pf-capture 60 passed / 0 failed including the new order pin. Host built natively on f43 against the box's own FFmpeg (libavcodec.so.61.19.101) — 7m35s, 0 errors. bash -n clean on the sysext script; shellcheck clean apart from a pre-existing SC1091 that fires on the unmodified file too.

Two field bugs from `.41` (Bazzite, RTX 5070 Ti) on 0.26.0-1. Both are fixed and verified on the box. ## 1. Red and blue swapped on every NVIDIA HDR stream `fix(pf-capture): every NVIDIA HDR stream had red and blue swapped` gamescope allocates its PipeWire capture textures mappable, hence **linear**-tiled, and composites into them with `imageStore`. NVIDIA does not implement linear-tiled `STORAGE` for `A2R10G10B10_UNORM_PACK32`. Upstream says it plainly in `rendervulkan.cpp`: > `// XRGB2101010 storage images are an optional Vulkan feature that NVIDIA hardware` > `// doesn't support, and imageStore lands in XBGR order there, swapping R/B.` So the composite writes **XBGR bytes into a buffer still labelled `XRGB2101010`**, and our patch's `spa_format_to_drm()` derives that label from the negotiated SPA format alone, never asking the hardware what it can actually write. **Why this resisted diagnosis: every mapping in the chain is individually correct.** DRM `XRGB2101010` really is "B in the low 10 bits"; NVENC `ARGB10` really is "B in the lowest 10 bits"; the Windows twin `R10G10B10A2 → ABGR10` is correct by the same rule. All of them audit clean, because the label was right and only the *content* was wrong. ⚠️ The trap worth remembering: **do not apply the "reverse the name" rule to `_PACK32` formats.** It is correct for Vulkan's 8-bit *byte-ordered* formats (`XRGB8888 → B8G8R8A8`) and wrong for `_PACK32`, which is word-ordered like DRM. ### The fix Offer `xBGR_210LE` **first** (new `HDR_FORMAT_ORDER`, consumed by `pipewire.rs`). The first compatible consumer pod wins, so that is what a gamescope session lands on, and an `XBGR2101010` texture is one NVIDIA writes in its own order — label and content agree. `xRGB_210LE` stays as the second pod so a producer offering only it can still negotiate HDR rather than dropping to the SDR downgrade. Test `hdr_offers_xbgr_before_xrgb` pins the order. Done host-side deliberately: the upstream-correct fix is for `spa_format_to_drm()` to follow `vulkan_get_rgb10_capture_format()`, but that function landed **after 3.16.25** and our pin is `3.16.25-7-g60561e2+pfhdr4` (0 `2101010` strings in the shipped binary), so the deployed gamescope cannot self-correct. This ships in the host binary with no gamescope rebuild. Moving the pin is filed as follow-up. ### Evidence A probe reproducing gamescope's own `vulkan_get_rgb10_capture_format()` check, run on both vendors: | GPU | `A2R10G10B10` (XRGB) | `A2B10G10R10` (XBGR) | |---|---|---| | RTX 5070 Ti | **NO** — `storage=false`, `linearTilingFeatures=0x1dc01` | **YES** — `0x1dc03` | | Radeon 780M (RADV Phoenix) | **YES** — `0x1dd83` | **YES** — `0x1dd83` | | llvmpipe (control) | NO — `storage=false` | YES | On NVIDIA the only differing bit is `STORAGE_IMAGE` — precisely the capability `imageStore` needs. **Confirmed host-side, not client-side**, by reproducing the identical swap from two unrelated clients (16" MacBook Pro and Mac Studio). SDR was never affected — it takes no packed-10 path. **On glass after the fix** (locally built host running on `.41`): ``` HDR capture: offering xBGR_210LE/xRGB_210LE LINEAR dmabufs ... pipewire format negotiated ... spa_format=VideoFormat::xBGR_210LE mapped=Some(X2Bgr10) hdr=true NVENC CUDA session ready mode=5120x1440@240 bit_depth=10 fmt=NV_ENC_BUFFER_FORMAT_ABGR10 ``` Colours confirmed correct by the reporter. ### AMD regression check Both mechanisms that could regress were verified directly: 1. **Compositor side** — AMD supports `A2B10G10R10` linear `STORAGE+SAMPLED` with *byte-identical* feature bits to `A2R10G10B10` (`0x1dd83`), so the format this change selects is composited correctly. XBGR is also the more broadly supported of the two (it is what upstream's own fallback picks, and the only one llvmpipe supports). 2. **Encode side** — handling is symmetric across every Linux backend (`vaapi.rs:63/64` + `825/826`, `vk_util.rs:88/89`, `nvenc_cuda.rs:587/589` + `634/635`, `mod.rs:47/48` and a shared match arm at `:175`); the only single-format sites are NVENC/Vulkan test fixtures. On the AMD box, ffmpeg lists `x2bgr10le` as `IO` (same as `x2rgb10le`) and a real swscale conversion produced the expected 307,200 bytes. Caveat stated plainly: a full end-to-end gamescope HDR *stream* on AMD was not run — the AMD box is a headless VM without the patched gamescope. What was verified is the two mechanisms by which this change could regress AMD, each directly. ## 2. A sysext post_merge step added in a release was unreachable forever `fix(packaging): a post_merge step added in a release was unreachable forever` A sysext upgrade is driven by the script from the **old** image — `/usr/bin/punktfunk-sysext` is replaced by the very `systemd-sysext refresh` that runs mid-upgrade — so a `post_merge` step *added* in the new release is executed by nobody. The old script does not have it, and the new script never gets a turn: from then on `update` matches the "already on `$cur`" branch and returns before `post_merge`. The step is permanently unreachable on exactly the installs that need it, and nothing says so. The casualty was the `punktfunk` group, which `post_merge` learned to create in 0.26.0 (`62a6fa9f`). 0.25.0's script drove the upgrade, so the group was never created, and every `punktfunk-sysext update` since has said "nothing to do". `pf-dm-helper` gates on membership in that group, so it refused every caller — pkexec authorised it and the helper then declined itself — and every managed gamescope takeover fell back to *"stopping the display manager needs privilege"*, leaving sddm's autologin Relogin loop churning logind sessions for the whole stream. Changes: re-run `post_merge` when already current (everything in it is idempotent, so convergence is the honest behaviour and "nothing to do" was a lie about host state); add an explicit `reapply` verb; and print the membership hint, since creating the group is necessary but not sufficient and the difference is invisible until a stream fails. Joining stays opt-in by design — writing vhci `attach` materialises an arbitrary emulated USB device — so `post_merge` names the exact `usermod` rather than doing it. Matched with `grep -qx` so `punktfunk-update` does not read as `punktfunk`. Verified live on `.41`: group **absent** → `reapply` → `punktfunk:x:965:` created; hint fired naming the user and went silent after joining; `pf-dm-helper` with a member UID now reaches verb dispatch while a non-member is still refused. ## Gates `pf-lxcheck2`, `linux/amd64`: `cargo fmt --check` clean, `cargo clippy --locked --all-targets -p pf-capture -- -D warnings` clean, `cargo test -p pf-capture` **60 passed / 0 failed** including the new order pin. Host built natively on f43 against the box's own FFmpeg (`libavcodec.so.61.19.101`) — 7m35s, 0 errors. `bash -n` clean on the sysext script; shellcheck clean apart from a pre-existing `SC1091` that fires on the unmodified file too.

enricobuehler added 2 commits 2026-08-09 15:28:15 +00:00

fix(pf-capture): every NVIDIA HDR stream had red and blue swapped ... 97928516a0

gamescope's capture textures are mappable, hence linear-tiled, and NVIDIA does
not implement linear-tiled STORAGE for A2R10G10B10_UNORM_PACK32. Upstream says
it plainly in rendervulkan.cpp: "imageStore lands in XBGR order there, swapping
R/B". So the composite writes XBGR bytes into a buffer still LABELLED
XRGB2101010, and our patch's spa_format_to_drm() derives that label from the
negotiated SPA format alone, never asking the hardware what it can actually
write.

The host then believed the label, correctly at every step:
xRGB_210LE -> PixelFormat::X2Rgb10 -> NV_ENC_BUFFER_FORMAT_ARGB10. DRM
XRGB2101010 really is "B in the low 10 bits" and NVENC ARGB10 really is "B in
the lowest 10 bits"; the Windows twin (R10G10B10A2 -> ABGR10) is correct by the
same rule. Every mapping audits clean because the label was right and only the
CONTENT was wrong -- which is why this survived a full trace of both ends.

Fix the preference host-side: offer xBGR_210LE FIRST. The first compatible
consumer pod wins, so that is what a gamescope session lands on, and an
XBGR2101010 texture is one NVIDIA writes in its own order -- label and content
agree. It costs nothing elsewhere: A2B10G10R10_UNORM_PACK32 is the universally
supported packed-10 format, it is what upstream's own fallback picks, and
X2Bgr10 has a first-class encoder path (NVENC ABGR10, VAAPI X2BGR10LE).
xRGB_210LE stays as the second pod so a producer offering only it can still
negotiate HDR instead of dropping to the SDR downgrade.

Doing it here rather than in the patch set is deliberate: the real fix is for
spa_format_to_drm() to offer only what vulkan_get_rgb10_capture_format()
reports, but that function landed after 3.16.25 and the pin is
3.16.25-7-g60561e2+pfhdr4 (0 "2101010" strings in the shipped binary), so the
deployed gamescope cannot self-correct. This ships in the host binary with no
gamescope rebuild.

Field-confirmed on the RTX 5070 Ti Bazzite host with 0.26.0, and confirmed
host-side rather than client-side by reproducing the identical swap from two
unrelated clients (16" MacBook Pro and Mac Studio). SDR was never affected --
it takes no packed-10 path.

Gate (pf-lxcheck2, linux/amd64): fmt clean, clippy --all-targets -D warnings
clean, cargo test -p pf-capture 60 passed / 0 failed incl. the new
hdr_offers_xbgr_before_xrgb order pin.

fix(packaging): a post_merge step added in a release was unreachable forever ... 0ab17ee81d

A sysext upgrade is driven by the script from the OLD image -- /usr/bin/punktfunk-sysext
is replaced by the very `systemd-sysext refresh` that runs mid-upgrade -- so a
post_merge step ADDED in the new release is executed by nobody. The old script
does not have it, and the new script never gets a turn: from then on `update`
matches the "already on $cur" branch and returns before post_merge. The step is
permanently unreachable on exactly the installs that need it, and nothing says so.

Field-proven on the Bazzite host that took 0.25.0 -> 0.26.0 (2026-08-09). The
casualty was the `punktfunk` group, which post_merge learned to create in 0.26.0
(62a6fa9f): 0.25.0's script ran the upgrade, so the group was never created, and
every `punktfunk-sysext update` since has said "nothing to do". `pf-dm-helper`
gates on membership in that group, so it refused every caller -- pkexec authorised
it and the helper then declined itself -- and every managed gamescope takeover fell
back to "stopping the display manager needs privilege", leaving sddm's autologin
Relogin loop churning logind sessions for the whole stream.

Re-run post_merge when already current. Everything in it is idempotent (guarded
getent/groupadd, `install` of /etc mirrors, udevadm reload/trigger, sysctl,
modprobe), so convergence is the honest behaviour and "nothing to do" was a lie
about host state. Add an explicit `reapply` verb too, so the steps a sysext image
cannot carry can be re-applied without reinstalling the image.

Also print the membership hint. Creating the group is necessary but NOT sufficient
and the difference is invisible until a stream fails: joining stays opt-in by
design (writing vhci `attach` materialises an arbitrary emulated USB device), so
post_merge now names the exact usermod when SUDO_USER is not a member. Matched with
`grep -qx` so `punktfunk-update` does not read as `punktfunk`.

bash -n clean; shellcheck clean apart from the pre-existing SC1091 on
`. /etc/os-release`, which fires on the unmodified file too.

enricobuehler merged commit 003ce8bea7 into main 2026-08-09 15:34:05 +00:00

enricobuehler deleted branch worktree-hdr-rb-swap-nvidia 2026-08-09 15:34:13 +00:00

enricobuehler referenced this issue from a commit 2026-08-09 18:48:51 +00:00

fix(packaging/gamescope): bump the pin past upstream's capture-format probe, and sign the RPM

enricobuehler referenced this pull request 2026-08-09 18:49:35 +00:00

Nobara could never use the patched gamescope — and the RPM it was told to install was unsigned #144

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX enricobuehler (Enrico Bühler) kszlim nannku neoney sassycrown

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#143