fix(clients): shared-VkQueue race + swapchain recreate destroy-in-use — the intermittent device-lost stream killer
apple / swift (push) Successful in 1m7s
ci / web (push) Successful in 1m9s
ci / docs-site (push) Successful in 1m18s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m41s
decky / build-publish (push) Successful in 21s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m43s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m34s
ci / bench (push) Successful in 6m12s
apple / screenshots (push) Successful in 5m18s
docker / deploy-docs (push) Successful in 27s
flatpak / build-publish (push) Failing after 5m25s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m50s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 17m35s
arch / build-publish (push) Successful in 19m55s
deb / build-publish (push) Successful in 18m59s
ci / rust (push) Successful in 20m9s
android / android (push) Successful in 20m11s
apple / swift (push) Successful in 1m7s
ci / web (push) Successful in 1m9s
ci / docs-site (push) Successful in 1m18s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m41s
decky / build-publish (push) Successful in 21s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m43s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m34s
ci / bench (push) Successful in 6m12s
apple / screenshots (push) Successful in 5m18s
docker / deploy-docs (push) Successful in 27s
flatpak / build-publish (push) Failing after 5m25s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m50s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 17m35s
arch / build-publish (push) Successful in 19m55s
deb / build-publish (push) Successful in 18m59s
ci / rust (push) Successful in 20m9s
android / android (push) Successful in 20m11s
Live-diagnosed on the RTX box during the adaptive-bitrate A/B: 2 of 3 streams died with VK_ERROR_DEVICE_LOST at stream start or at a mid-stream encoder rebuild, then zombied — the demote-to-software path rebuilt the decoder against the same dead device, FFmpeg wedged inside the rebuild, and the pump flushed a never-draining backlog every 2 s forever. No OS TDR: the client's own Vulkan misuse. Two causes: 1. The presenter creates ONE graphics-family queue and hands FFmpeg's AVVulkanDeviceContext the same family (nb_graphics_queues=1 ⇒ queue 0) for its transfer/compute prep — so the pump thread and the presenter thread submitted to the SAME VkQueue with no shared lock, violating vkQueueSubmit's external-sync rule exactly when FFmpeg puts work on the graphics queue (decoder open / frames-context rebuild = stream start and every ABR encoder re-target). New guard-less QueueLock (FFmpeg's lock_queue/unlock_queue callbacks are a raw pair) shared by all four queue users: FFmpeg (callbacks installed via user_opaque), the presenter's submit/present/wait-idle, and the Skia overlay's flushes. 2. Swapchain recreation destroyed the old swapchain + render semaphores after ONE fence cycle — the fence proves our submit, not the presentation engine's semaphore consumption (VUID-vkDestroySemaphore-05149 + VUID-vkDestroySwapchainKHR-01282 on every recreate). Recreate now drains the queue (vkQueueWaitIdle under the shared lock — safe now that FFmpeg honours it) and destroys immediately; the deferred DisplayGarbage machinery is gone. Resilience: VK_ERROR_DEVICE_LOST anywhere in a present error chain is now fatal — the run loop fails the session loudly instead of demoting to software on a dead device (the zombie path). Verified on the RTX box (RTX 4090 → host .21) under VK_LAYER_KHRONOS_validation: 3/3 stream start/stop cycles clean, then 8 mid-stream encoder rebuilds in one session (4 ABR down-steps under 10% induced loss + 4 clean-link recovery up-steps — the exact scenarios that previously killed the device): 0 device losses, 0 wedges, both recreate VUIDs gone (previously fired on every path). Remaining validation messages are FFmpeg's own video-session VUIDs, untouched by this change. Linux: clippy -D warnings + tests green (home-worker-2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -797,6 +797,79 @@ impl VaapiDecoder {
|
||||
}
|
||||
}
|
||||
|
||||
/// Guard-less mutex serializing every `vkQueueSubmit`/`vkQueuePresentKHR`/
|
||||
/// `vkQueueWaitIdle` on the device the presenter shares with FFmpeg.
|
||||
///
|
||||
/// Why it exists: the presenter created the device with ONE graphics-family queue and
|
||||
/// told FFmpeg's `AVVulkanDeviceContext` to use that same family (`nb_graphics_queues
|
||||
/// = 1` ⇒ queue index 0) for its transfer/compute prep work — so the presenter thread
|
||||
/// and the session pump thread were submitting to the SAME `VkQueue` with no shared
|
||||
/// lock. `vkQueueSubmit` requires external synchronization on the queue; the race
|
||||
/// surfaced as intermittent `VK_ERROR_DEVICE_LOST` at exactly the moments FFmpeg puts
|
||||
/// work on the graphics queue (decoder open / frames-context rebuild — i.e. stream
|
||||
/// start and every adaptive-bitrate encoder rebuild; live-diagnosed 2026-07-09).
|
||||
///
|
||||
/// FFmpeg's hook for this is the `lock_queue`/`unlock_queue` callback pair on
|
||||
/// `AVVulkanDeviceContext` — a raw lock/unlock shape with no RAII scope, hence this
|
||||
/// guard-less primitive (`std::sync::Mutex`'s guard can't cross the C callbacks).
|
||||
/// Contention is a handful of µs-scale critical sections per frame; a plain
|
||||
/// Mutex+Condvar is more than enough.
|
||||
pub struct QueueLock {
|
||||
locked: std::sync::Mutex<bool>,
|
||||
cv: std::sync::Condvar,
|
||||
}
|
||||
|
||||
impl QueueLock {
|
||||
#[allow(clippy::new_without_default)]
|
||||
pub fn new() -> QueueLock {
|
||||
QueueLock {
|
||||
locked: std::sync::Mutex::new(false),
|
||||
cv: std::sync::Condvar::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Block until the queue is free, then take it. Pair with [`QueueLock::unlock`]
|
||||
/// (FFmpeg's callbacks), or use [`QueueLock::guard`] from Rust callers.
|
||||
pub fn lock(&self) {
|
||||
let mut g = self
|
||||
.locked
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
while *g {
|
||||
g = self
|
||||
.cv
|
||||
.wait(g)
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
}
|
||||
*g = true;
|
||||
}
|
||||
|
||||
pub fn unlock(&self) {
|
||||
let mut g = self
|
||||
.locked
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
*g = false;
|
||||
drop(g);
|
||||
self.cv.notify_one();
|
||||
}
|
||||
|
||||
/// RAII form for Rust call sites (presenter submits/presents, Skia flushes).
|
||||
pub fn guard(&self) -> QueueLockGuard<'_> {
|
||||
self.lock();
|
||||
QueueLockGuard(self)
|
||||
}
|
||||
}
|
||||
|
||||
/// Releases the [`QueueLock`] on drop.
|
||||
pub struct QueueLockGuard<'a>(&'a QueueLock);
|
||||
|
||||
impl Drop for QueueLockGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
self.0.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
/// The presenter's Vulkan device handles, exported so FFmpeg's Vulkan Video decoder
|
||||
/// runs on the SAME device the presenter samples from — the whole point: the decoded
|
||||
/// VkImage is composited directly, no interop, no copy (plan: Vulkan Video phase).
|
||||
@@ -839,6 +912,10 @@ pub struct VulkanDecodeDevice {
|
||||
/// backend creates its decode device on the SAME adapter so shared textures never cross
|
||||
/// GPUs. `None` when not reported (or off Windows, where it's unused).
|
||||
pub adapter_luid: Option<[u8; 8]>,
|
||||
/// The device's shared queue lock (see [`QueueLock`]). The presenter holds it around
|
||||
/// its own submits/presents; the decoder wires it into FFmpeg's
|
||||
/// `lock_queue`/`unlock_queue` callbacks so both sides serialize on the same queues.
|
||||
pub queue_lock: std::sync::Arc<QueueLock>,
|
||||
}
|
||||
|
||||
/// `fourcc(a,b,c,d)` — the DRM FourCC packing (little-endian, `a | b<<8 | c<<16 | d<<24`).
|
||||
@@ -934,6 +1011,36 @@ struct VkCtxStorage {
|
||||
f11: pf_ffvk::VkPhysicalDeviceVulkan11Features,
|
||||
f12: pf_ffvk::VkPhysicalDeviceVulkan12Features,
|
||||
f13: pf_ffvk::VkPhysicalDeviceVulkan13Features,
|
||||
/// Keeps the shared queue lock alive for `AVHWDeviceContext.user_opaque` — the
|
||||
/// `lock_queue`/`unlock_queue` trampolines below dereference it for as long as the
|
||||
/// hw device context can fire them.
|
||||
_queue_lock: std::sync::Arc<QueueLock>,
|
||||
}
|
||||
|
||||
/// FFmpeg `AVVulkanDeviceContext.lock_queue` trampoline: take the device's shared
|
||||
/// [`QueueLock`] (stashed in `AVHWDeviceContext.user_opaque`; owned by
|
||||
/// [`VkCtxStorage`], which outlives the context). Replaces FFmpeg's internal default,
|
||||
/// which only serializes FFmpeg against itself — the presenter submits to the same
|
||||
/// graphics queue from another thread and holds this same lock around its calls.
|
||||
unsafe extern "C" fn ffvk_lock_queue(
|
||||
ctx: *mut pf_ffvk::AVHWDeviceContext,
|
||||
_queue_family: u32,
|
||||
_index: u32,
|
||||
) {
|
||||
let dev = ctx as *mut ffmpeg::ffi::AVHWDeviceContext;
|
||||
let lock = (*dev).user_opaque as *const QueueLock;
|
||||
(*lock).lock();
|
||||
}
|
||||
|
||||
/// The matching `unlock_queue` trampoline — see [`ffvk_lock_queue`].
|
||||
unsafe extern "C" fn ffvk_unlock_queue(
|
||||
ctx: *mut pf_ffvk::AVHWDeviceContext,
|
||||
_queue_family: u32,
|
||||
_index: u32,
|
||||
) {
|
||||
let dev = ctx as *mut ffmpeg::ffi::AVHWDeviceContext;
|
||||
let lock = (*dev).user_opaque as *const QueueLock;
|
||||
(*lock).unlock();
|
||||
}
|
||||
|
||||
impl VulkanDecoder {
|
||||
@@ -957,6 +1064,7 @@ impl VulkanDecoder {
|
||||
f11: std::mem::zeroed(),
|
||||
f12: std::mem::zeroed(),
|
||||
f13: std::mem::zeroed(),
|
||||
_queue_lock: vk.queue_lock.clone(),
|
||||
});
|
||||
store.inst_ptrs = store._inst.iter().map(|c| c.as_ptr()).collect();
|
||||
store.dev_ptrs = store._dev.iter().map(|c| c.as_ptr()).collect();
|
||||
@@ -1030,6 +1138,16 @@ impl VulkanDecoder {
|
||||
(*hwctx).nb_qf = 2;
|
||||
}
|
||||
|
||||
// Shared-queue external sync (see [`QueueLock`]): FFmpeg must take the
|
||||
// same lock the presenter holds around its own submits/presents — set
|
||||
// BEFORE init so FFmpeg never installs its internal defaults (which only
|
||||
// serialize FFmpeg against itself; the cross-thread race with the
|
||||
// presenter's queue was an intermittent VK_ERROR_DEVICE_LOST).
|
||||
(*devctx).user_opaque =
|
||||
std::sync::Arc::as_ptr(&store._queue_lock) as *mut std::ffi::c_void;
|
||||
(*hwctx).lock_queue = Some(ffvk_lock_queue);
|
||||
(*hwctx).unlock_queue = Some(ffvk_unlock_queue);
|
||||
|
||||
let r = ffi::av_hwdevice_ctx_init(hw_device);
|
||||
if r < 0 {
|
||||
ffi::av_buffer_unref(&mut hw_device);
|
||||
|
||||
Reference in New Issue
Block a user