enricobuehlerandClaude Opus 5 09aa2db37c fix(encode): probe Vulkan encode before committing capture to producer-native NV12 ...

`linux_native_nv12_ok` is the verdict the host threads into capture negotiation
(session_plan -> OutputFormat::nv12_native -> ZeroCopyPolicy::native_nv12_session
-> pf-capture's prefer_native_nv12). Its doc claimed "the backend must be eligible
to open", but it asked only three static questions — codec is H265/AV1, an env
default, and a pref denylist — and never whether this GPU has an encode queue at
all. It could not: vulkan_video.rs exposed no probe.

That verdict is uniquely load-bearing. Once the producer has been asked for
two-plane NV12 there is NO fallback: open_video deliberately makes a failed Vulkan
open FATAL for an NV12 capture rather than degrading to libav VAAPI, because VAAPI
would import that buffer as packed RGB and stream silent garbage. So on a gamescope
host whose Mesa lacks Vulkan HEVC encode the session died at its first frame, while
the same host with PUNKTFUNK_PIPEWIRE_NV12=0 streamed fine — and the comment
promising such a device "degrades gracefully to the old backend rather than
breaking the stream" was stale on every gamescope host.

Two changes:

1. The gate. The third conjunct was a denylist of the EXPLICIT prefs that skip
   Vulkan Video ("nvenc"|"nvidia"|"cuda"|"pyrowave"), which silently missed the one
   that matters: the DEFAULT encoder_pref is "", and "" resolves to auto, which on
   an NVIDIA box opens NVENC. A stock NVIDIA host passed this gate. It now consults
   `linux_zero_copy_is_vaapi`, which layers the pref on top of the same auto
   decision open_video makes — what the note on `linux_auto_is_vaapi` says a
   capability probe must use, and what the downstream consumer of this very verdict
   already uses. This alone was worth fixing; a probe added behind the old gate
   would have opened a Vulkan instance on every NVIDIA handshake.

2. The probe. `vulkan_video::probe_encode_support` runs the FIRST check open_inner
   performs and hard-fails on — the physical-device + encode-queue-family scan for
   this codec's op — and nothing more, so it is provably no stricter than the open
   and can never talk a working host out of the fast path. The scan is now a shared
   `find_encode_device` used by BOTH, because a probe that mirrors a dispatch goes
   stale the first time the dispatch grows a case (the failure open_video's
   backend-label note already records). Cost: one instance plus physical-device
   queries — no logical device, no video session, no VRAM — cached per (selected
   GPU, codec) in the can_encode_10bit idiom, with the probe run outside the lock.
   Per codec, not once: codec_op_for selects a different queue-family bit for AV1,
   and HEVC-encode-without-AV1-encode is the common VCN/ANV configuration.

Later stages (create_device, create_video_session, the capability query) can still
fail for reasons the probe does not model. Those are harmless: the capture format
is only committed once the probe says yes, and everything after keeps the ordinary
packed-RGB negotiation.

Verified `-D warnings --all-targets` + tests on Linux (default; shipped
nvenc+vulkan-encode+pyrowave). The behaviour needs an on-glass check on the bazzite
RADV box — including the negative case, which `RADV_DEBUG=novideo` reproduces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-25 00:32:37 +02:00

..

enc

fix(encode): probe Vulkan encode before committing capture to producer-native NV12

2026-07-25 00:32:37 +02:00

lib.rs

fix(encode): probe Vulkan encode before committing capture to producer-native NV12

2026-07-25 00:32:37 +02:00