From 310b85f1550b4b7897b4e6e94586f20013df6087 Mon Sep 17 00:00:00 2001
From: enricobuehler <buehler@unom.io>
Date: Sat, 25 Jul 2026 03:17:45 +0200
Subject: [PATCH] fix(encode): a forced-Vulkan pref must not advertise codecs
 that arm will refuse
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

With `PUNKTFUNK_ENCODER=vulkan`, `open_video_backend`'s vulkan arm bails outright for
anything that is not HEVC/AV1 ("the Vulkan Video encoder supports HEVC + AV1; the
session negotiated {codec:?}"). But `host_wire_caps` for that same pref fell through
to the VAAPI probe / static superset, which includes H.264 — so the host advertised
H.264, a client could negotiate it, and the session died at encoder open.

The pref now contributes a CEILING that is intersected with the device probe, never a
replacement for it. That distinction is the whole fix: pinning a static HEVC|AV1
would have ADDED AV1 on the AMD/Intel hosts whose probe currently withholds it
(pre-RDNA3, pre-Arc), re-creating this very bug for a different codec. Intersecting
can only ever narrow.

Without the `vulkan-encode` feature the pref cannot open anything at all — that arm
bails with "requires a build with --features vulkan-encode" — so the ceiling is
empty there rather than optimistic.

Costs nothing on the handshake path: a `&str` match plus a const-folded `cfg!`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 crates/pf-encode/src/lib.rs | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/crates/pf-encode/src/lib.rs b/crates/pf-encode/src/lib.rs
index bdca9e21..c51777ae 100644
--- a/crates/pf-encode/src/lib.rs
+++ b/crates/pf-encode/src/lib.rs
@@ -78,13 +78,34 @@ impl Codec {
                 ) {
                     return punktfunk_core::quic::CODEC_H264;
                 }
+                // A pref that FORCES the raw Vulkan Video backend can only ever serve what that
+                // backend encodes: `open_video_backend`'s `vulkan` arm bails outright for anything
+                // that is not HEVC/AV1 ("the Vulkan Video encoder supports HEVC + AV1; the session
+                // negotiated {codec:?}"). Advertising H.264 there let a client negotiate it and die
+                // at encoder open. Without the `vulkan-encode` feature that arm cannot open at all,
+                // so the pref advertises nothing.
+                //
+                // This is a CEILING intersected with the device probe below, never a replacement
+                // for it: pinning a static HEVC|AV1 would ADD AV1 on the AMD/Intel hosts whose probe
+                // currently withholds it (pre-RDNA3, pre-Arc), i.e. it would re-create this very bug
+                // for a different codec. Intersecting can only narrow.
+                let pref_ceiling: u8 = match pf_host_config::config().encoder_pref.as_str() {
+                    "vulkan" | "vulkan-video" => {
+                        if cfg!(feature = "vulkan-encode") {
+                            punktfunk_core::quic::CODEC_HEVC | punktfunk_core::quic::CODEC_AV1
+                        } else {
+                            0
+                        }
+                    }
+                    _ => GPU_SUPERSET,
+                };
                 if linux_zero_copy_is_vaapi() {
                     if let Some(m) = vaapi_codec_support().wire_mask() {
-                        return m;
+                        return m & pref_ceiling;
                     }
                 }
                 // NVENC (static superset, like GameStream) — or an empty VAAPI probe (see above).
-                GPU_SUPERSET
+                GPU_SUPERSET & pref_ceiling
             }
             #[cfg(target_os = "windows")]
             {