fix(clients): shared-VkQueue race + swapchain recreate destroy-in-use — the intermittent device-lost stream killer
apple / swift (push) Successful in 1m7s
ci / web (push) Successful in 1m9s
ci / docs-site (push) Successful in 1m18s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m41s
decky / build-publish (push) Successful in 21s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m43s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m34s
ci / bench (push) Successful in 6m12s
apple / screenshots (push) Successful in 5m18s
docker / deploy-docs (push) Successful in 27s
flatpak / build-publish (push) Failing after 5m25s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m50s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 17m35s
arch / build-publish (push) Successful in 19m55s
deb / build-publish (push) Successful in 18m59s
ci / rust (push) Successful in 20m9s
android / android (push) Successful in 20m11s
apple / swift (push) Successful in 1m7s
ci / web (push) Successful in 1m9s
ci / docs-site (push) Successful in 1m18s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m41s
decky / build-publish (push) Successful in 21s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m43s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m34s
ci / bench (push) Successful in 6m12s
apple / screenshots (push) Successful in 5m18s
docker / deploy-docs (push) Successful in 27s
flatpak / build-publish (push) Failing after 5m25s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m50s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 17m35s
arch / build-publish (push) Successful in 19m55s
deb / build-publish (push) Successful in 18m59s
ci / rust (push) Successful in 20m9s
android / android (push) Successful in 20m11s
Live-diagnosed on the RTX box during the adaptive-bitrate A/B: 2 of 3 streams died with VK_ERROR_DEVICE_LOST at stream start or at a mid-stream encoder rebuild, then zombied — the demote-to-software path rebuilt the decoder against the same dead device, FFmpeg wedged inside the rebuild, and the pump flushed a never-draining backlog every 2 s forever. No OS TDR: the client's own Vulkan misuse. Two causes: 1. The presenter creates ONE graphics-family queue and hands FFmpeg's AVVulkanDeviceContext the same family (nb_graphics_queues=1 ⇒ queue 0) for its transfer/compute prep — so the pump thread and the presenter thread submitted to the SAME VkQueue with no shared lock, violating vkQueueSubmit's external-sync rule exactly when FFmpeg puts work on the graphics queue (decoder open / frames-context rebuild = stream start and every ABR encoder re-target). New guard-less QueueLock (FFmpeg's lock_queue/unlock_queue callbacks are a raw pair) shared by all four queue users: FFmpeg (callbacks installed via user_opaque), the presenter's submit/present/wait-idle, and the Skia overlay's flushes. 2. Swapchain recreation destroyed the old swapchain + render semaphores after ONE fence cycle — the fence proves our submit, not the presentation engine's semaphore consumption (VUID-vkDestroySemaphore-05149 + VUID-vkDestroySwapchainKHR-01282 on every recreate). Recreate now drains the queue (vkQueueWaitIdle under the shared lock — safe now that FFmpeg honours it) and destroys immediately; the deferred DisplayGarbage machinery is gone. Resilience: VK_ERROR_DEVICE_LOST anywhere in a present error chain is now fatal — the run loop fails the session loudly instead of demoting to software on a dead device (the zombie path). Verified on the RTX box (RTX 4090 → host .21) under VK_LAYER_KHRONOS_validation: 3/3 stream start/stop cycles clean, then 8 mid-stream encoder rebuilds in one session (4 ABR down-steps under 10% induced loss + 4 clean-link recovery up-steps — the exact scenarios that previously killed the device): 0 device losses, 0 wedges, both recreate VUIDs gone (previously fired on every path). Remaining validation messages are FFmpeg's own video-session VUIDs, untouched by this change. Linux: clippy -D warnings + tests green (home-worker-2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -235,6 +235,18 @@ impl StreamState {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a present error is `VK_ERROR_DEVICE_LOST` anywhere in its chain. A lost
|
||||
/// device is unrecoverable by spec — every object on it (decoder frames, swapchain,
|
||||
/// the Skia context) is dead, and the demote-to-software path would rebuild the
|
||||
/// decoder against that same dead device (observed live 2026-07-09: FFmpeg wedges
|
||||
/// inside the rebuild, the decode thread never returns, and the client zombies with
|
||||
/// the pump flushing a never-draining backlog every 2 s). The only correct response
|
||||
/// is to fail the session loudly and let the shell relaunch.
|
||||
fn device_lost(e: &anyhow::Error) -> bool {
|
||||
e.chain()
|
||||
.any(|c| c.downcast_ref::<ash::vk::Result>() == Some(&ash::vk::Result::ERROR_DEVICE_LOST))
|
||||
}
|
||||
|
||||
fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>> {
|
||||
// Before any window exists: unpackaged runs adopt the shell's AppUserModelID so the
|
||||
// shell⇄session windows group as one taskbar app (win32.rs; MSIX identity wins).
|
||||
@@ -789,8 +801,13 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
|
||||
// Import/CSC failure is survivable (the stream continues on
|
||||
// the next frame) — but a streak means this box can't do the
|
||||
// hw path: demote the decoder to software, same contract as
|
||||
// the GTK presenter's GL-converter failures.
|
||||
// the GTK presenter's GL-converter failures. A lost DEVICE
|
||||
// is not survivable and must not demote — see [`device_lost`].
|
||||
Err(e) => {
|
||||
if device_lost(&e) {
|
||||
return Err(e)
|
||||
.context("GPU device lost — the session cannot continue");
|
||||
}
|
||||
st.hw_fails += 1;
|
||||
tracing::warn!(error = %format!("{e:#}"), fails = st.hw_fails,
|
||||
"hardware present failed");
|
||||
@@ -832,6 +849,11 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
|
||||
p
|
||||
}
|
||||
Err(e) => {
|
||||
// Lost device ⇒ unrecoverable, never demote ([`device_lost`]).
|
||||
if device_lost(&e) {
|
||||
return Err(e)
|
||||
.context("GPU device lost — the session cannot continue");
|
||||
}
|
||||
st.hw_fails += 1;
|
||||
tracing::warn!(error = %format!("{e:#}"), fails = st.hw_fails,
|
||||
"hardware present failed");
|
||||
@@ -873,6 +895,11 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
|
||||
p
|
||||
}
|
||||
Err(e) => {
|
||||
// Lost device ⇒ unrecoverable, never demote ([`device_lost`]).
|
||||
if device_lost(&e) {
|
||||
return Err(e)
|
||||
.context("GPU device lost — the session cannot continue");
|
||||
}
|
||||
st.hw_fails += 1;
|
||||
tracing::warn!(error = %format!("{e:#}"), fails = st.hw_fails,
|
||||
"vulkan-video present failed");
|
||||
|
||||
Reference in New Issue
Block a user