First AMD test (Steam Deck, Mesa radeonsi) showed a mostly-green image
with red whites — the classic fingerprint of NV12 chroma read as 0.
Root cause (confirmed against FFmpeg/GTK/mpv source): FFmpeg's VAAPI
export uses VA_EXPORT_SURFACE_SEPARATE_LAYERS unconditionally, so an
NV12 surface comes back as TWO single-plane layers — layers[0]=R8
(luma), layers[1]=GR88 (chroma) — sharing one object/fd, the UV plane
reached via offset. map_dmabuf took layers[0] only and used its format
(R8) as the GTK fourcc, so GdkDmabufTexture got a luma-only texture
with the chroma plane dropped → chroma defaults to 0 → green field,
red highlights.
Fix (matches mpv's dmabuf_interop_gl flatten pattern):
- Derive the combined fourcc from the decoder's sw_format
(AVHWFramesContext.sw_format → NV12 → DRM_FORMAT_NV12), NOT from the
per-plane component formats. The frame format is absent from the
separate-layer descriptor and must be deduced from sw_format.
- Flatten every plane across every layer in declared order (Y then UV),
each with its own fd (objects[plane.object_index].fd), offset, pitch.
- One-time descriptor dump (objects/layers/formats/modifier) so a new
driver's real layout is visible in the logs.
- Unit test locks the DRM FourCC magic numbers (NV12=0x3231564e).
Software decode (swscale, reads colorspace from the VUI) was always
correct, which isolated the bug to this path. PUNKTFUNK_DECODER=software
is the immediate workaround on an un-rebuilt binary. Awaiting Steam Deck
reconfirm (no AMD VAAPI on the NVIDIA dev box to live-verify).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler
64b167946f