fix(encode/vulkan): VBR instead of CBR — the driver was stuffing ~98% filler into every calm stream

Second attempt at WP6.3; the first (ce543668) was withdrawn after its tight
CBR window measured as a 36x bandwidth regression (97% filler NALs). The
correction that unlocked this one: that measurement's BASELINE row was an
8-frame artifact. Under the shipped 1000ms/500ms CBR window a calm stream
overflows the CPB once the initial fill drains (~30 frames at 10Mbps/60fps)
and RADV then pads every frame to the exact rate share, forever. Measured on
the 780M (1280x720@60, 10Mbps, calm content): 64 frames = 97.5% filler;
300 frames = 5.63MB at 98.5% filler where this commit ships 83KB at 0%.
AV1: 99.6% -> 0%. The status quo was ~the full target bitrate of zeros on
every idle AMD/Intel Vulkan-encode desktop and Steam Deck — and stuffing to
exactly the target permanently satisfied the ABR calm brake
(actual >= 3/4 * current), the ratchet WP6.3's withdrawal feared from the
tight window, live in the shipped code all along.

The fix reads VkVideoEncodeCapabilitiesKHR::rateControlModes (previously
ignored — rateControlMode was hardcoded CBR with no capability check) and
installs VBR with average == max plus the house ~1-frame window
(vbv_window_ms, PUNKTFUNK_VBV_FRAMES-scaled) when the driver advertises VBR.
VBR permits underspend, the exact missing degree of freedom: Vulkan exposes
no filler-suppression control (AMF's filler_data=false / NVENC's default-off
have no VK equivalent), so the MODE is the only lever. CBR-only drivers keep
the loose window untouched — tightening it under CBR just starts the stuffing
30 frames earlier. Drivers advertising neither mode (ANV per current Mesa)
keep the pre-existing CBR install, now WARN-logged.

No pacing claim, deliberately: burst A/B on the 780M is byte-identical
between 1000ms CBR and 17ms VBR (max AU 1.19MB in both) — this firmware
ignores the window for QP decisions entirely. The payload is filler
elimination. PUNKTFUNK_VULKAN_RC=cbr|vbr is the field escape hatch and the
on-box A/B control (two withdrawn attempts bought that insurance).

Also on the same caps struct: maxBitrate is now read and clamps open +
retarget (RADV reports 1 Gbps — within 5% of the 4K120 ABR targets), and
applied_bitrate_bps() reports the encoder-side truth (pending-first, so the
session loop's read right after reconfigure_bitrate sees the clamp) — without
it a binding clamp would feed the ABR a phantom base, the trap the trait doc
names.

And the one-frame VUID-vkCmdBeginVideoCodingKHR-pBeginInfo-08254 violation
found in the withdrawal review: record_submit promoted a pending retarget
into self.bitrate BEFORE recording whenever first_frame was set, so after a
mid-stream reset() (which preserves the pending rate and rc_installed) the
begin-coding declaration named a rate the session had not installed — and the
two triggers, ABR retarget and the stall watchdog, correlate. Now the
declaration always names the session's current rate and the RESET install
carries the pending one via its own struct; promotion stays in
post_submit_bookkeeping. The extended validation-layer test reproduces the
retarget-then-reset coincidence: exactly one 08254 on the pre-fix build, zero
on this one (RADV PHOENIX, on glass).

Gates: docker amd64 legs green; Windows .173 seven legs (34 passed);
.25 full-suite parity vs origin/main (identical CUDA-only failures) + all 9
vulkan on-glass tests + validation layers clean.

WP6.3, plus WP7.1's ms-form half (vbv_window_ms).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

crates/pf-encode/src/enc/codec.rs

@@ -520,6 +520,35 @@ pub(crate) fn vbv_frames_env() -> f64 {
         .unwrap_or(1.0)
 }
 
+/// The same HRD/VBV window as [`vbv_frames_env`], expressed the way the Vulkan Video encode API
+/// wants it: `(virtualBufferSizeInMs, initialVirtualBufferSizeInMs)`.
+///
+/// Every other backend states the window in **bits** (`bitrate / fps × frames`); Vulkan states it
+/// in **milliseconds**. `vulkan_video.rs` consumes this ONLY when the driver advertises VBR
+/// (WP6.3): a tight window under CBR makes the driver stuff underspent frames with filler NALs up
+/// to the exact rate share — measured 97 % filler on the 780M — because CBR must keep the CPB from
+/// overflowing and Vulkan exposes no filler-suppression control. VBR permits the underspend, so
+/// the tight window only ever *bounds* a complex frame.
+///
+/// The initial fill stays at half the window, preserving the RATIO the hardcoded (1000, 500)
+/// pair had — the direct-NVENC house shape uses a FULL-window initial fill instead; measured on
+/// RADV the difference is inert (the firmware showed no window sensitivity at all). Both
+/// VUIDs on `VkVideoEncodeRateControlInfoKHR`'s window fields are satisfied by construction: the
+/// window clamps to `>= 1` so it is non-zero, and `window / 2 <= window` always
+/// (`VUID-...-08358` is `<=`, relaxed in Vulkan 1.3.299).
+///
+/// Carries its only caller's gate: `vulkan_video.rs` is the sole ms-form consumer, and with the
+/// crate-wide `allow(dead_code)` gone (WP0.3) an item unused in ANY feature combination is a hard
+/// error — this is dead on every Windows leg.
+#[cfg(all(target_os = "linux", feature = "vulkan-encode"))]
+pub(crate) fn vbv_window_ms(fps: u32) -> (u32, u32) {
+    let frames = vbv_frames_env();
+    let ms = (frames * 1000.0 / fps.max(1) as f64).round();
+    // `f64 as u32` saturates at the bounds in Rust, so an absurd `PUNKTFUNK_VBV_FRAMES` cannot wrap.
+    let window = (ms as u32).max(1);
+    (window, window / 2)
+}
+
 /// Validate a requested encode resolution before we allocate buffers or open NVENC. Rejects
 /// zero/odd-sized and out-of-range modes with a clear error instead of letting buffer math
 /// overflow or the encoder open fail with an opaque NVENC code. A client can request any
@@ -575,6 +604,31 @@ pub fn validate_dimensions(codec: Codec, width: u32, height: u32) -> Result<()>
 mod tests {
     use super::*;
 
+    /// WP6.3. The window VUIDs on `VkVideoEncodeRateControlInfoKHR` are the whole contract of
+    /// this helper, and both are edge cases: the window must be non-zero (a high-refresh mode
+    /// rounds a sub-1 ms window down to nothing) and the initial fill must be at most the window
+    /// (`<=` — `VUID-...-08358` was relaxed in 1.3.299). Env-free so it pins the default shape —
+    /// the scaled cases belong to whoever sets `PUNKTFUNK_VBV_FRAMES`. Carries the helper's own
+    /// cfg gate (see its note), so it runs on the Linux `vulkan-encode` leg.
+    #[cfg(all(target_os = "linux", feature = "vulkan-encode"))]
+    #[test]
+    fn vbv_window_is_about_one_frame_and_always_legal() {
+        // The house default is ~1 frame interval, not the 1000 ms the Vulkan backend hardwired.
+        assert_eq!(vbv_window_ms(60).0, 17); // 16.67 ms
+        assert_eq!(vbv_window_ms(30).0, 33);
+        assert_eq!(vbv_window_ms(240).0, 4);
+        for fps in [1, 24, 30, 60, 120, 144, 240, 480, 1000, 4000, u32::MAX] {
+            let (window, initial) = vbv_window_ms(fps);
+            assert!(window > 0, "virtualBufferSizeInMs must be > 0 (fps {fps})");
+            assert!(
+                initial <= window,
+                "initialVirtualBufferSizeInMs must be <= virtualBufferSizeInMs (fps {fps})"
+            );
+        }
+        // fps 0 must not divide by zero — `open` clamps, but the helper is called directly too.
+        assert!(vbv_window_ms(0).0 > 0);
+    }
+
     #[test]
     fn rejects_zero_and_odd_dimensions() {
         assert!(validate_dimensions(Codec::H265, 0, 1080).is_err());