fix(encode): bound GPU waits, validate encode status, repair command-buffer and cache invariants

Five of the nine medium findings from the pf-encode sweep. The remaining four
need cross-crate plumbing or an unwind refactor and are deliberately left out.

- vulkan_video `enqueue` waited on the backpressure fence with `u64::MAX`. That
  wait runs ON the host encode thread — the same thread the stall watchdog's
  `reset()` would run on — so a wedged GPU parked the one thread that could
  recover the session: no error, no reset, and teardown blocking on the join.
  This is the DEFAULT encode path for AMD/Intel Linux hosts (both shipped build
  recipes enable `vulkan-encode` and `vulkan_encode_enabled()` defaults true).
  Now bounded by ENCODE_FENCE_TIMEOUT_NS with expiry surfaced as an error.

- vulkan_video `import_cached` evicted a cached dmabuf import and destroyed its
  image/view/memory with no fence wait, while up to `ring_depth - 1` submitted
  frames may still reference it — a GPU-side use-after-free. `Drop` and `reset`
  both idle first; this was the one unguarded destroy. Now idles before the
  eviction loop, guarded on the length test so the steady state pays nothing.

- vulkan_video `read_slot` never asked for the encode's operation status, so a
  FAILED encode was indistinguishable from a successful one and its feedback
  was read as if it described real bitstream. Now requests WITH_STATUS_KHR and
  refuses anything that is not COMPLETE.

- linux/pyrowave `encode_frame` opens its recording window early and has six
  fallible steps inside it; every one returned with `cmd` still RECORDING, and
  nothing repaired it (one `begin_command_buffer` in the file, and neither
  `reset()` nor `Drop` touches `cmd`), so the next frame called `begin` on a
  recording buffer — invalid usage. `submit` now resets the buffer on error;
  legal on all these paths since the pool carries RESET_COMMAND_BUFFER and the
  buffer is not pending.

- windows/pyrowave `encode_frame` ignored `frame.width`/`height` and imported
  planes at the encoder's configured extent, so a ring recreate at a new mode
  (the IDD capturer does this autonomously on a confirmed descriptor change)
  read the planes under a stale VkImageCreateInfo. Added the size guard its QSV
  and AMF siblings already carry, and keyed the plane cache on
  (address, width, height) so a recycled COM address cannot resurrect an import
  of a different size. NOTE: a recycle at the SAME size is still theoretically
  possible; the complete fix keys on the capturer's ring generation and needs
  that plumbed onto `PyroFrameShare`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

crates/pf-encode/src/enc/linux/pyrowave.rs

@@ -1176,7 +1176,24 @@ impl Encoder for PyroWaveEncoder {
     fn submit(&mut self, frame: &CapturedFrame) -> Result<()> {
         // SAFETY: single-threaded encoder; `encode_frame` records/submits on handles this
         // struct owns and waits its own fence before touching results.
-        unsafe { self.encode_frame(frame) }
+        let r = unsafe { self.encode_frame(frame) };
+        if r.is_err() {
+            // `encode_frame` opens the recording window early and has several fallible steps
+            // inside it (cursor prep, dmabuf import, format mapping, the CPU-RGB staging path,
+            // an unsupported-payload bail, and the encode call itself). Every one returns with
+            // `self.cmd` still RECORDING, and nothing downstream repairs it — there is exactly
+            // one `begin_command_buffer` in this file and `reset()`/`Drop` never touch `cmd` —
+            // so the NEXT frame would call `begin` on a recording buffer, which is invalid usage.
+            // Legal here on every path: the pool carries RESET_COMMAND_BUFFER and the buffer is
+            // not pending (we never reached the submit, or the submit itself failed).
+            // SAFETY: `self.cmd` is owned by this encoder and, on these paths, not in flight.
+            unsafe {
+                let _ = self
+                    .device
+                    .reset_command_buffer(self.cmd, vk::CommandBufferResetFlags::empty());
+            }
+        }
+        r
     }
 
     fn caps(&self) -> EncoderCaps {

crates/pf-encode/src/enc/linux/vulkan_video.rs

@@ -37,6 +37,12 @@ const DPB_SLOTS: u32 = 8;
 /// latency — on-glass validated as rock-solid at 1080p@240, so it is the real-time default;
 /// backpressure kicks in at the 2nd unread frame. Distinct from `DPB_SLOTS` (reference pool).
 const RING_DEFAULT: usize = 2;
+
+/// Ceiling on any blocking GPU fence wait on the encode thread (5 s). Generous against a real
+/// encode (single-digit ms even on a loaded GPU) and against a driver hiccup, but finite: this is
+/// the thread the stall watchdog's `reset()` runs on, so an unbounded wait would deadlock the very
+/// path that recovers the session. Matches the Windows NVENC retrieve-thread budget.
+const ENCODE_FENCE_TIMEOUT_NS: u64 = 5_000_000_000;
 /// AV1 base quantizer index (0..=255) seeded into every frame. CBR rate control overrides it per
 /// frame; it only matters as the starting point and for the (rate-control-ignored) constant-Q path.
 const AV1_BASE_Q_IDX: u8 = 128;
@@ -868,6 +874,15 @@ impl VulkanVideoEncoder {
         let (img, mem, view) = self.import_dmabuf(d, cw, ch)?;
         // Bound the cache; evict oldest (FIFO). A stable PipeWire pool never trips this in steady state
         // (all imports resident); it only cycles across a pool change (which also rebuilds the session).
+        // Up to `ring_depth - 1` submitted frames may still be executing against a cached image
+        // (`enqueue` only drains down to `frames.len()`, and `record_submit` imports before it
+        // records), so destroying an evicted import here is a GPU-side use-after-free. `Drop` and
+        // `reset()` both idle the device first; this was the one unguarded destroy. Guarded on the
+        // length test so the steady-state path — where the cache is resident and never evicts —
+        // pays nothing.
+        if self.import_cache.len() >= IMPORT_CACHE_CAP {
+            let _ = self.device.device_wait_idle();
+        }
         while self.import_cache.len() >= IMPORT_CACHE_CAP {
             let (_, _, oi, om, ov) = self.import_cache.remove(0);
             self.device.destroy_image_view(ov, None);
@@ -1849,9 +1864,39 @@ impl VulkanVideoEncoder {
     unsafe fn read_slot(&mut self, slot: usize) -> Result<EncodedFrame> {
         let dev = self.device.clone();
         let f = &self.frames[slot];
-        let mut fb = [[0u32; 2]; 1];
+        // Ask for the operation status alongside the two feedback words: without it a FAILED encode
-        dev.get_query_pool_results(f.query_pool, 0, &mut fb, vk::QueryResultFlags::WAIT)?;
+        // is indistinguishable from a successful one, and its offset/bytes-written are read as if
-        let (off, len) = (fb[0][0] as usize, fb[0][1] as usize);
+        // they described real bitstream. The status rides as a trailing element (signed:
+        // `VkQueryResultStatusKHR` is >0 COMPLETE, 0 NOT_READY, <0 error).
+        let mut fb = [[0i32; 3]; 1];
+        dev.get_query_pool_results(
+            f.query_pool,
+            0,
+            &mut fb,
+            vk::QueryResultFlags::WAIT | vk::QueryResultFlags::WITH_STATUS_KHR,
+        )?;
+        let status = fb[0][2];
+        if status <= 0 {
+            anyhow::bail!(
+                "vulkan-encode: encode feedback for slot {slot} reports status {status} \
+                 (not COMPLETE) — dropping the frame rather than shipping its bitstream"
+            );
+        }
+        let fb = [[fb[0][0] as u32, fb[0][1] as u32]];
+        // The (offset, bytes-written) pair is driver-reported: validate it against the bitstream
+        // allocation BEFORE mapping, or the `from_raw_parts` below reads outside the buffer and
+        // ships whatever it finds straight onto the wire. Checked in u64 so the add cannot wrap,
+        // and before `map_memory` so there is no unmap to unwind on the error path.
+        let (off64, len64) = (fb[0][0] as u64, fb[0][1] as u64);
+        if off64.saturating_add(len64) > self.bs_size {
+            anyhow::bail!(
+                "vulkan-encode: driver reported bitstream feedback offset={off64} \
+                 bytes_written={len64}, outside the {} byte bitstream buffer — the encode likely \
+                 overflowed its destination range",
+                self.bs_size
+            );
+        }
+        let (off, len) = (off64 as usize, len64 as usize);
         let p =
             dev.map_memory(f.bs_mem, 0, vk::WHOLE_SIZE, vk::MemoryMapFlags::empty())? as *const u8;
         let prefix: &[u8] = if f.keyframe {
@@ -1879,8 +1924,25 @@ impl VulkanVideoEncoder {
         // and free it — that oldest slot is exactly the round-robin `ring` cursor we reuse next.
         while self.in_flight.len() >= self.frames.len() {
             let slot = self.in_flight.pop_front().unwrap();
-            self.device
+            // Bounded, not `u64::MAX`: this runs ON the host encode thread, which is also the
-                .wait_for_fences(&[self.frames[slot].fence], true, u64::MAX)?;
+            // thread the stall watchdog's `reset()` would run on. An infinite wait against a
+            // wedged GPU/driver therefore parks the one thread that could recover the session —
+            // it never errors, never resets, and teardown blocks joining it. Surfacing expiry as
+            // an error hands control back to the existing recovery path (same convention as the
+            // pyrowave and Windows NVENC backends).
+            match self.device.wait_for_fences(
+                &[self.frames[slot].fence],
+                true,
+                ENCODE_FENCE_TIMEOUT_NS,
+            ) {
+                Ok(()) => {}
+                Err(vk::Result::TIMEOUT) => anyhow::bail!(
+                    "vulkan-encode: fence for slot {slot} did not signal within {} ms — GPU or \
+                     driver wedged; failing the submit so the session can reset",
+                    ENCODE_FENCE_TIMEOUT_NS / 1_000_000
+                ),
+                Err(e) => return Err(e.into()),
+            }
             let done = self.read_slot(slot)?;
             self.pending.push_back(done);
         }
@@ -1923,7 +1985,28 @@ impl Encoder for VulkanVideoEncoder {
         if first_frame < 0 || first_frame > last_frame {
             return false;
         }
+        // Taint sweep BEFORE picking the anchor (the fecbec2d fix AMF and QSV got; this backend was
+        // carved out one commit later and never received it). "Resident and older than THIS loss" is
+        // not the same as "the client decoded it": after an earlier loss [a,b] was recovered at wire
+        // r, everything in [a, r-1] is undecodable at the client — the lost frames plus every frame
+        // that predicted through the gap. Those wires stay valid anchor candidates here until the
+        // 8-slot ring rolls them out, so a LATER loss can anchor on one and ship corruption tagged
+        // `recovery_anchor` — which is the client's definitive re-anchor signal (reanchor.rs), so it
+        // lifts the post-loss freeze onto a picture built from a reference it never had.
+        //
+        // Blank `slot_wire` ONLY. `slot_poc` must keep naming every physically-resident DPB picture
+        // for `build_h265_rps_s0`, or a conforming decoder evicts them and the anchor references a
+        // picture the client already dropped. `slot_wire` is the RFI/loss domain; `slot_poc` is the
+        // reference-delta domain. `prev_slot` and the normal P-frame path are indices, not wires, so
+        // ordinary prediction is unaffected.
+        for w in self.slot_wire.iter_mut() {
+            if *w >= first_frame {
+                *w = -1;
+            }
+        }
         // Can we anchor a clean P-frame to a resident slot strictly older than the loss?
+        // (A sweep that empties every candidate yields `None` here and declines the RFI, matching
+        // `qsv_live_ltr_rfi_taint_sweep_declines`.)
         match pick_recovery_slot(&self.slot_wire, first_frame) {
             Some(_) => {
                 self.pending_loss = Some(first_frame);
@@ -2697,6 +2780,62 @@ mod tests {
         assert_eq!(pick_recovery_slot(&[-1; 8], 5), None);
     }
 
+    /// The taint sweep (fecbec2d's fix, ported here): a slot encoded inside an EARLIER, still
+    /// unrepaired loss window must not become the "known-good" anchor of a LATER loss. Without the
+    /// sweep, `pick_recovery_slot` accepts it — it is resident and its wire is below the second
+    /// loss start — and the frame ships tagged `recovery_anchor`, lifting the client's freeze onto
+    /// a reference it never decoded.
+    #[test]
+    fn taint_sweep_excludes_slots_from_an_earlier_loss() {
+        // Apply the sweep exactly as `invalidate_ref_frames` does.
+        fn sweep(wires: &mut [i64], loss_first: i64) {
+            for w in wires.iter_mut() {
+                if *w >= loss_first {
+                    *w = -1;
+                }
+            }
+        }
+
+        // Slots hold wires 0..7. Loss 1 starts at wire 4, so wires 4..7 are undecodable at the
+        // client. A second loss report arrives at wire 6 while they are all still resident.
+        let tainted = [4i64, 5, 6, 7];
+
+        // WITHOUT the sweep this is the bug: the newest wire below 6 is wire 5 — squarely inside
+        // loss 1's unrepaired window — and it would be served as the "known-good" anchor.
+        let unswept = [0i64, 1, 2, 3, 4, 5, 6, 7];
+        let picked = pick_recovery_slot(&unswept, 6).expect("unswept picks something");
+        assert!(
+            tainted.contains(&unswept[picked]),
+            "precondition: without the sweep the anchor comes from the earlier loss window"
+        );
+
+        // WITH the sweep, loss 1 blanks 4..7, so loss 2 can only reach genuinely clean wires.
+        let mut wires = unswept;
+        sweep(&mut wires, 4);
+        assert_eq!(wires, [0, 1, 2, 3, -1, -1, -1, -1]);
+        let picked = pick_recovery_slot(&wires, 6).expect("clean wires remain");
+        assert_eq!(picked, 3, "newest clean survivor is wire 3");
+        assert!(!tainted.contains(&wires[picked]));
+
+        // Encoding resumes after recovery; wires 8..11 refill the swept slots and are clean. A
+        // later loss at wire 10 legitimately anchors on wire 9 — the sweep must not over-reject.
+        wires[4] = 8;
+        wires[5] = 9;
+        wires[6] = 10;
+        wires[7] = 11;
+        sweep(&mut wires, 10);
+        assert_eq!(
+            pick_recovery_slot(&wires, 10),
+            Some(5),
+            "wire 9 is post-recovery, clean"
+        );
+
+        // A loss covering every live wire leaves nothing clean → decline, caller serves an IDR.
+        let mut all = [5i64, 6, 7, 8, 9, 10, 11, 12];
+        sweep(&mut all, 5);
+        assert_eq!(pick_recovery_slot(&all, 5), None);
+    }
+
     /// The full-retention RPS: every resident picture is listed (so the decoder keeps it), the
     /// setup slot's dying occupant is not, and `used_by_curr_pic` marks exactly the real reference.
     #[test]

crates/pf-encode/src/enc/windows/pyrowave.rs

@@ -43,6 +43,9 @@ const BS_SLACK: usize = 256 * 1024;
 /// (a desktop-switch device recreate), in which case the stale imports are evicted + destroyed.
 const IMPORT_CACHE_CAP: usize = 8;
 
+/// Plane-import cache key: the texture's COM address plus the extent it was imported at.
+type PlaneKey = (isize, u32, u32);
+
 // --- Vulkan enum values not surfaced by pyrowave-sys' bindgen (only enums *reachable* from the
 // pyrowave C API are generated; these plain #define / flags-typedef values are stable spec
 // constants). bindgen renders every reachable Vulkan enum as a `u32` type alias, so these u32
@@ -136,8 +139,8 @@ pub struct PyroWaveEncoder {
     // Imported plane textures, cached by the out-ring texture's raw pointer (stable per ring slot):
     // the full-res R8 Y plane and the half-res R8G8 CbCr plane, imported SEPARATELY (a single planar
     // NV12 import is unreliable on NVIDIA at arbitrary sizes).
-    y_images: Vec<(isize, pw::pyrowave_image)>,
+    y_images: Vec<(PlaneKey, pw::pyrowave_image)>,
-    cbcr_images: Vec<(isize, pw::pyrowave_image)>,
+    cbcr_images: Vec<(PlaneKey, pw::pyrowave_image)>,
 
     width: u32,
     height: u32,
@@ -351,10 +354,16 @@ impl PyroWaveEncoder {
     ///
     /// # Safety
     /// Same contract as [`import_plane`].
+    /// Keyed on `(texture address, width, height)` rather than the bare address: the COM pointer
+    /// carries no reference here, so a released texture's address can be recycled by a later
+    /// allocation and return an import describing the WRONG surface. Folding the extent in means a
+    /// recycled address at a different size can never alias. (A recycle at the SAME size is still
+    /// possible in principle — the complete fix is to key on the capturer's ring generation, which
+    /// needs that generation plumbed onto `PyroFrameShare`.)
     unsafe fn cached_plane(
-        cache: &mut Vec<(isize, pw::pyrowave_image)>,
+        cache: &mut Vec<(PlaneKey, pw::pyrowave_image)>,
         make: impl FnOnce() -> Result<pw::pyrowave_image>,
-        key: isize,
+        key: PlaneKey,
     ) -> Result<pw::pyrowave_image> {
         if let Some((_, img)) = cache.iter().find(|(k, _)| *k == key) {
             return Ok(*img);
@@ -423,6 +432,21 @@ impl PyroWaveEncoder {
             !self.pw_enc.is_null(),
             "pyrowave: encode after a failed reset (encoder was destroyed and not rebuilt)"
         );
+        // The plane textures are imported at the encoder's CONFIGURED extent, not the frame's, so a
+        // capture that changed size would be read under a stale `VkImageCreateInfo`. This is
+        // reachable without any client Reconfigure: the IDD capturer autonomously recreates its ring
+        // on a confirmed display-descriptor change (e.g. a fullscreen game mode-setting the virtual
+        // display). Refuse instead — the session must reopen the encoder at the new mode. Mirrors
+        // the guard the QSV and AMF backends already carry.
+        anyhow::ensure!(
+            frame.width == self.width && frame.height == self.height,
+            "pyrowave: captured frame {}x{} != encoder {}x{} (the capturer recreated its ring at a \
+             new mode — the encoder must be reopened)",
+            frame.width,
+            frame.height,
+            self.width,
+            self.height
+        );
         let FramePayload::D3d11(d3d) = &frame.payload else {
             bail!("pyrowave (Windows) needs a D3D11 frame (the capturer must be in pyrowave mode)")
         };
@@ -465,7 +489,7 @@ impl PyroWaveEncoder {
         };
         let pw_dev = self.pw_dev;
         let y_img = {
-            let key = d3d.texture.as_raw() as isize;
+            let key = (d3d.texture.as_raw() as isize, w, h);
             let tex = &d3d.texture;
             Self::cached_plane(
                 &mut self.y_images,
@@ -474,7 +498,7 @@ impl PyroWaveEncoder {
             )?
         };
         let cbcr_img = {
-            let key = share.cbcr.as_raw() as isize;
+            let key = (share.cbcr.as_raw() as isize, cw, ch);
             let tex = &share.cbcr;
             Self::cached_plane(
                 &mut self.cbcr_images,

crates/pf-encode/src/enc/windows/qsv.rs

@@ -714,7 +714,16 @@ pub struct QsvEncoder {
     /// `EncoderCaps::supports_rfi` and all per-frame marking/forcing below.
     ltr_active: bool,
     /// The wire frame index stored in each LTR slot (`None` = never marked).
+    ///
+    /// This mirrors the HARDWARE DPB, so an entry must not be cleared merely because we distrust
+    /// it: nulling issues no VPL call, and the encoder keeps the frame marked long-term until that
+    /// `LongTermIdx` is re-marked or an IDR flushes it. Distrust is recorded in `ltr_tainted`
+    /// instead, so the rejection list can still NAME the entry the hardware is holding.
     ltr_slots: [Option<i64>; NUM_LTR_SLOTS],
+    /// Per-slot taint from `invalidate_ref_frames`' sweep: the mark is still live in the hardware
+    /// DPB but was encoded inside the client's corrupt window, so it may not anchor a recovery —
+    /// it must be REJECTED instead. Cleared wherever the slot is re-marked or the DPB is flushed.
+    ltr_tainted: [bool; NUM_LTR_SLOTS],
     next_ltr_slot: usize,
     ltr_mark_interval: i64,
     /// Set by `invalidate_ref_frames`: the slot the next submitted frame force-references.
@@ -789,6 +798,7 @@ impl QsvEncoder {
             ir_active: false,
             ltr_active: false,
             ltr_slots: [None; NUM_LTR_SLOTS],
+            ltr_tainted: [false; NUM_LTR_SLOTS],
             next_ltr_slot: 0,
             ltr_mark_interval: ltr_mark_interval(fps),
             pending_force: None,
@@ -913,6 +923,7 @@ impl QsvEncoder {
         self.ltr_active = ltr_active;
         self.ir_active = ir_active;
         self.ltr_slots = [None; NUM_LTR_SLOTS];
+        self.ltr_tainted = [false; NUM_LTR_SLOTS];
         self.next_ltr_slot = 0;
         self.pending_force = None;
         self.hdr_applied = self.hdr_meta;
@@ -1038,6 +1049,7 @@ impl Encoder for QsvEncoder {
                 // An IDR voids the decoder's reference buffers — drop stale slots and any
                 // queued force; the mark cadence below re-anchors on the IDR itself.
                 self.ltr_slots = [None; NUM_LTR_SLOTS];
+                self.ltr_tainted = [false; NUM_LTR_SLOTS]; // the IDR flushed the DPB with them
                 self.next_ltr_slot = 0;
                 self.pending_force = None;
             } else if self.ltr_test_force_at == Some(cur_idx) {
@@ -1053,7 +1065,9 @@ impl Encoder for QsvEncoder {
                 // emptied the slot since the force was queued. An empty slot means there is
                 // nothing clean to re-reference — the frame must ship as a plain P WITHOUT the
                 // `recovery_anchor` tag (the client lifts its post-loss freeze on that tag).
-                if let Some(idx) = self.ltr_slots[slot] {
+                // The slot is no longer emptied by the sweep, so test the taint flag too — a
+                // tainted slot is exactly the "nothing clean to re-reference" case.
+                if let Some(idx) = self.ltr_slots[slot].filter(|_| !self.ltr_tainted[slot]) {
                     force_ltr = Some((slot, idx));
                     recovery_anchor = true;
                 }
@@ -1061,6 +1075,9 @@ impl Encoder for QsvEncoder {
             if force_ltr.is_none() && (forced || cur_idx % self.ltr_mark_interval == 0) {
                 let slot = self.next_ltr_slot;
                 self.ltr_slots[slot] = Some(cur_idx);
+                // Re-marking replaces the hardware's LongTermIdx: the tainted frame is gone from
+                // the DPB and this slot is clean again.
+                self.ltr_tainted[slot] = false;
                 self.next_ltr_slot = (self.next_ltr_slot + 1) % NUM_LTR_SLOTS;
                 mark_slot = Some(slot);
             }
@@ -1323,13 +1340,23 @@ impl Encoder for QsvEncoder {
         // loss ships corruption as the recovery anchor, and every subsequent mark re-samples
         // the soup — the sustained-loss field failure where the picture never healed. Dropped
         // slots stay dropped; the cadence re-marks a clean frame within ~1/4 s.
-        for marked in self.ltr_slots.iter_mut() {
+        //
+        // Mark tainted rather than clearing: `ltr_slots` mirrors the HARDWARE DPB, and nulling an
+        // entry issues no VPL call — the frame stays marked long-term in the encoder. Clearing it
+        // made the rejection list below (which iterates the post-sweep mirror and only names `Some`
+        // slots) silently SKIP the one entry the sweep exists to distrust, so the recovery frame
+        // could still predict from it. With two slots the "exactly one swept" case is the modal
+        // one, and it was the broken one.
+        for (slot, marked) in self.ltr_slots.iter().enumerate() {
             if marked.is_some_and(|idx| idx >= first) {
-                *marked = None;
+                self.ltr_tainted[slot] = true;
             }
         }
         let mut best: Option<(usize, i64)> = None;
         for (slot, marked) in self.ltr_slots.iter().enumerate() {
+            if self.ltr_tainted[slot] {
+                continue; // still in the DPB, but encoded inside the corrupt window
+            }
             if let Some(idx) = *marked {
                 if idx < first && best.is_none_or(|(_, b)| idx > b) {
                     best = Some((slot, idx));
@@ -1454,6 +1481,7 @@ impl Encoder for QsvEncoder {
                 self.ltr_active = ltr;
                 self.ir_active = ir;
                 self.ltr_slots = [None; NUM_LTR_SLOTS];
+                self.ltr_tainted = [false; NUM_LTR_SLOTS];
                 self.next_ltr_slot = 0;
                 self.pending_force = None;
                 if let Some(inner) = self.inner.as_mut() {