fix(encode): bound GPU waits, validate encode status, repair command-buffer and cache invariants
Five of the nine medium findings from the pf-encode sweep. The remaining four need cross-crate plumbing or an unwind refactor and are deliberately left out. - vulkan_video `enqueue` waited on the backpressure fence with `u64::MAX`. That wait runs ON the host encode thread — the same thread the stall watchdog's `reset()` would run on — so a wedged GPU parked the one thread that could recover the session: no error, no reset, and teardown blocking on the join. This is the DEFAULT encode path for AMD/Intel Linux hosts (both shipped build recipes enable `vulkan-encode` and `vulkan_encode_enabled()` defaults true). Now bounded by ENCODE_FENCE_TIMEOUT_NS with expiry surfaced as an error. - vulkan_video `import_cached` evicted a cached dmabuf import and destroyed its image/view/memory with no fence wait, while up to `ring_depth - 1` submitted frames may still reference it — a GPU-side use-after-free. `Drop` and `reset` both idle first; this was the one unguarded destroy. Now idles before the eviction loop, guarded on the length test so the steady state pays nothing. - vulkan_video `read_slot` never asked for the encode's operation status, so a FAILED encode was indistinguishable from a successful one and its feedback was read as if it described real bitstream. Now requests WITH_STATUS_KHR and refuses anything that is not COMPLETE. - linux/pyrowave `encode_frame` opens its recording window early and has six fallible steps inside it; every one returned with `cmd` still RECORDING, and nothing repaired it (one `begin_command_buffer` in the file, and neither `reset()` nor `Drop` touches `cmd`), so the next frame called `begin` on a recording buffer — invalid usage. `submit` now resets the buffer on error; legal on all these paths since the pool carries RESET_COMMAND_BUFFER and the buffer is not pending. - windows/pyrowave `encode_frame` ignored `frame.width`/`height` and imported planes at the encoder's configured extent, so a ring recreate at a new mode (the IDD capturer does this autonomously on a confirmed descriptor change) read the planes under a stale VkImageCreateInfo. Added the size guard its QSV and AMF siblings already carry, and keyed the plane cache on (address, width, height) so a recycled COM address cannot resurrect an import of a different size. NOTE: a recycle at the SAME size is still theoretically possible; the complete fix keys on the capturer's ring generation and needs that plumbed onto `PyroFrameShare`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -43,6 +43,9 @@ const BS_SLACK: usize = 256 * 1024;
|
||||
/// (a desktop-switch device recreate), in which case the stale imports are evicted + destroyed.
|
||||
const IMPORT_CACHE_CAP: usize = 8;
|
||||
|
||||
/// Plane-import cache key: the texture's COM address plus the extent it was imported at.
|
||||
type PlaneKey = (isize, u32, u32);
|
||||
|
||||
// --- Vulkan enum values not surfaced by pyrowave-sys' bindgen (only enums *reachable* from the
|
||||
// pyrowave C API are generated; these plain #define / flags-typedef values are stable spec
|
||||
// constants). bindgen renders every reachable Vulkan enum as a `u32` type alias, so these u32
|
||||
@@ -136,8 +139,8 @@ pub struct PyroWaveEncoder {
|
||||
// Imported plane textures, cached by the out-ring texture's raw pointer (stable per ring slot):
|
||||
// the full-res R8 Y plane and the half-res R8G8 CbCr plane, imported SEPARATELY (a single planar
|
||||
// NV12 import is unreliable on NVIDIA at arbitrary sizes).
|
||||
y_images: Vec<(isize, pw::pyrowave_image)>,
|
||||
cbcr_images: Vec<(isize, pw::pyrowave_image)>,
|
||||
y_images: Vec<(PlaneKey, pw::pyrowave_image)>,
|
||||
cbcr_images: Vec<(PlaneKey, pw::pyrowave_image)>,
|
||||
|
||||
width: u32,
|
||||
height: u32,
|
||||
@@ -351,10 +354,16 @@ impl PyroWaveEncoder {
|
||||
///
|
||||
/// # Safety
|
||||
/// Same contract as [`import_plane`].
|
||||
/// Keyed on `(texture address, width, height)` rather than the bare address: the COM pointer
|
||||
/// carries no reference here, so a released texture's address can be recycled by a later
|
||||
/// allocation and return an import describing the WRONG surface. Folding the extent in means a
|
||||
/// recycled address at a different size can never alias. (A recycle at the SAME size is still
|
||||
/// possible in principle — the complete fix is to key on the capturer's ring generation, which
|
||||
/// needs that generation plumbed onto `PyroFrameShare`.)
|
||||
unsafe fn cached_plane(
|
||||
cache: &mut Vec<(isize, pw::pyrowave_image)>,
|
||||
cache: &mut Vec<(PlaneKey, pw::pyrowave_image)>,
|
||||
make: impl FnOnce() -> Result<pw::pyrowave_image>,
|
||||
key: isize,
|
||||
key: PlaneKey,
|
||||
) -> Result<pw::pyrowave_image> {
|
||||
if let Some((_, img)) = cache.iter().find(|(k, _)| *k == key) {
|
||||
return Ok(*img);
|
||||
@@ -423,6 +432,21 @@ impl PyroWaveEncoder {
|
||||
!self.pw_enc.is_null(),
|
||||
"pyrowave: encode after a failed reset (encoder was destroyed and not rebuilt)"
|
||||
);
|
||||
// The plane textures are imported at the encoder's CONFIGURED extent, not the frame's, so a
|
||||
// capture that changed size would be read under a stale `VkImageCreateInfo`. This is
|
||||
// reachable without any client Reconfigure: the IDD capturer autonomously recreates its ring
|
||||
// on a confirmed display-descriptor change (e.g. a fullscreen game mode-setting the virtual
|
||||
// display). Refuse instead — the session must reopen the encoder at the new mode. Mirrors
|
||||
// the guard the QSV and AMF backends already carry.
|
||||
anyhow::ensure!(
|
||||
frame.width == self.width && frame.height == self.height,
|
||||
"pyrowave: captured frame {}x{} != encoder {}x{} (the capturer recreated its ring at a \
|
||||
new mode — the encoder must be reopened)",
|
||||
frame.width,
|
||||
frame.height,
|
||||
self.width,
|
||||
self.height
|
||||
);
|
||||
let FramePayload::D3d11(d3d) = &frame.payload else {
|
||||
bail!("pyrowave (Windows) needs a D3D11 frame (the capturer must be in pyrowave mode)")
|
||||
};
|
||||
@@ -465,7 +489,7 @@ impl PyroWaveEncoder {
|
||||
};
|
||||
let pw_dev = self.pw_dev;
|
||||
let y_img = {
|
||||
let key = d3d.texture.as_raw() as isize;
|
||||
let key = (d3d.texture.as_raw() as isize, w, h);
|
||||
let tex = &d3d.texture;
|
||||
Self::cached_plane(
|
||||
&mut self.y_images,
|
||||
@@ -474,7 +498,7 @@ impl PyroWaveEncoder {
|
||||
)?
|
||||
};
|
||||
let cbcr_img = {
|
||||
let key = share.cbcr.as_raw() as isize;
|
||||
let key = (share.cbcr.as_raw() as isize, cw, ch);
|
||||
let tex = &share.cbcr;
|
||||
Self::cached_plane(
|
||||
&mut self.cbcr_images,
|
||||
|
||||
Reference in New Issue
Block a user