fix(encode): bound GPU waits, validate encode status, repair command-buffer and cache invariants
Five of the nine medium findings from the pf-encode sweep. The remaining four need cross-crate plumbing or an unwind refactor and are deliberately left out. - vulkan_video `enqueue` waited on the backpressure fence with `u64::MAX`. That wait runs ON the host encode thread — the same thread the stall watchdog's `reset()` would run on — so a wedged GPU parked the one thread that could recover the session: no error, no reset, and teardown blocking on the join. This is the DEFAULT encode path for AMD/Intel Linux hosts (both shipped build recipes enable `vulkan-encode` and `vulkan_encode_enabled()` defaults true). Now bounded by ENCODE_FENCE_TIMEOUT_NS with expiry surfaced as an error. - vulkan_video `import_cached` evicted a cached dmabuf import and destroyed its image/view/memory with no fence wait, while up to `ring_depth - 1` submitted frames may still reference it — a GPU-side use-after-free. `Drop` and `reset` both idle first; this was the one unguarded destroy. Now idles before the eviction loop, guarded on the length test so the steady state pays nothing. - vulkan_video `read_slot` never asked for the encode's operation status, so a FAILED encode was indistinguishable from a successful one and its feedback was read as if it described real bitstream. Now requests WITH_STATUS_KHR and refuses anything that is not COMPLETE. - linux/pyrowave `encode_frame` opens its recording window early and has six fallible steps inside it; every one returned with `cmd` still RECORDING, and nothing repaired it (one `begin_command_buffer` in the file, and neither `reset()` nor `Drop` touches `cmd`), so the next frame called `begin` on a recording buffer — invalid usage. `submit` now resets the buffer on error; legal on all these paths since the pool carries RESET_COMMAND_BUFFER and the buffer is not pending. - windows/pyrowave `encode_frame` ignored `frame.width`/`height` and imported planes at the encoder's configured extent, so a ring recreate at a new mode (the IDD capturer does this autonomously on a confirmed descriptor change) read the planes under a stale VkImageCreateInfo. Added the size guard its QSV and AMF siblings already carry, and keyed the plane cache on (address, width, height) so a recycled COM address cannot resurrect an import of a different size. NOTE: a recycle at the SAME size is still theoretically possible; the complete fix keys on the capturer's ring generation and needs that plumbed onto `PyroFrameShare`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1176,7 +1176,24 @@ impl Encoder for PyroWaveEncoder {
|
||||
fn submit(&mut self, frame: &CapturedFrame) -> Result<()> {
|
||||
// SAFETY: single-threaded encoder; `encode_frame` records/submits on handles this
|
||||
// struct owns and waits its own fence before touching results.
|
||||
unsafe { self.encode_frame(frame) }
|
||||
let r = unsafe { self.encode_frame(frame) };
|
||||
if r.is_err() {
|
||||
// `encode_frame` opens the recording window early and has several fallible steps
|
||||
// inside it (cursor prep, dmabuf import, format mapping, the CPU-RGB staging path,
|
||||
// an unsupported-payload bail, and the encode call itself). Every one returns with
|
||||
// `self.cmd` still RECORDING, and nothing downstream repairs it — there is exactly
|
||||
// one `begin_command_buffer` in this file and `reset()`/`Drop` never touch `cmd` —
|
||||
// so the NEXT frame would call `begin` on a recording buffer, which is invalid usage.
|
||||
// Legal here on every path: the pool carries RESET_COMMAND_BUFFER and the buffer is
|
||||
// not pending (we never reached the submit, or the submit itself failed).
|
||||
// SAFETY: `self.cmd` is owned by this encoder and, on these paths, not in flight.
|
||||
unsafe {
|
||||
let _ = self
|
||||
.device
|
||||
.reset_command_buffer(self.cmd, vk::CommandBufferResetFlags::empty());
|
||||
}
|
||||
}
|
||||
r
|
||||
}
|
||||
|
||||
fn caps(&self) -> EncoderCaps {
|
||||
|
||||
Reference in New Issue
Block a user