fix(host): NVENC open-failure resilience — backoff, failed-open hygiene, self-diagnosis
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 51s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
apple / swift (push) Successful in 1m9s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m48s
docker / deploy-docs (push) Successful in 24s
apple / screenshots (push) Successful in 5m18s
windows-host / package (push) Successful in 9m16s
arch / build-publish (push) Successful in 10m53s
android / android (push) Successful in 11m58s
deb / build-publish (push) Successful in 18m7s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m19s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m28s
ci / rust (push) Successful in 24m56s
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 51s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
apple / swift (push) Successful in 1m9s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m48s
docker / deploy-docs (push) Successful in 24s
apple / screenshots (push) Successful in 5m18s
windows-host / package (push) Successful in 9m16s
arch / build-publish (push) Successful in 10m53s
android / android (push) Successful in 11m58s
deb / build-publish (push) Successful in 18m7s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m19s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m28s
ci / rust (push) Successful in 24m56s
Field report (Linux direct NVENC): after a codec switch, every session open failed with NV_ENC_ERR_INVALID_VERSION until the host process was restarted — so the poisoned state is per-process, not a driver install issue. On-hardware investigation (RTX 5070 Ti, 610.43.03) could not reproduce it with clean codec cycles, dirty teardowns, or open/destroy storms, but established the failure class: the driver enforces a per-process concurrent-session cap (12 there, status INCOMPATIBLE_CLIENT_KEY; other branches report differently) whose exhaustion is exactly this signature — persistent open failures healed only by a process restart. Harden every path that can feed or mask that state: * Rebuild backoff: the in-place encoder-rebuild retries slept one frame interval, so all 5 attempts burned within ~40 ms at 120 Hz — no driver-side transient (deferred teardown of the previous session, engine reset) can clear that fast. Exponential backoff 100 ms → 1.6 s (~3 s total) so transients heal instead of killing the session. * Destroy-on-failed-open (Linux + Windows, all four open sites): the NVENC docs require NvEncDestroyEncoder even when OpenEncodeSessionEx FAILS — the driver may have allocated the session slot before erroring. Without it a retry burst against a transient leaks slots toward the cap, converting the transient into permanent exhaustion. * Teardown: a destroy_encoder failure (a session slot the driver may keep) is now logged with its status instead of silently discarded. * One-shot self-diagnosis on a failed session open (Linux): retry the raw open on a fresh dedicated CUDA context and log which of the three causes applies — shared-context poisoned (fresh works), driver-level skew/exhaustion/GPU loss (fresh fails the same way), or CUDA itself unhealthy (no fresh context) — so the next field report pinpoints the root cause with zero reporter effort. On-hardware regression tests (RTX box .21, all green): codec-switch reopen cycle (H265→AV1→H265→H264→H265), dirty teardown with in-flight encodes, and the full open-failure→diagnosis→in-place-recovery path via real session-cap exhaustion. Existing RFI/reconfigure/4:4:4 smokes still pass; clippy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -4935,10 +4935,19 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
|
||||
max = MAX_ENCODER_RESETS,
|
||||
"encoder submit failed — encoder rebuilt in place, forcing an IDR");
|
||||
last_au_at = std::time::Instant::now();
|
||||
// Re-pace from the rebuild and retry this frame next tick (gives the fresh encoder
|
||||
// one frame period to come up instead of hammering it in a hot loop).
|
||||
next = std::time::Instant::now() + interval;
|
||||
std::thread::sleep(interval);
|
||||
// Back off exponentially between rebuild attempts (100 ms → 1.6 s, ~3 s total across
|
||||
// the reset budget). One frame period is NOT enough: a 2026-07 field report showed all
|
||||
// 5 resets burning within 40 ms at 120 Hz against a driver-side condition (NVENC
|
||||
// session open failing after a codec switch) that no 8 ms retry could outlive — any
|
||||
// transient like the previous session's deferred driver teardown needs real time. A
|
||||
// genuinely dead encoder now costs ~3 s before the session ends with the terminal
|
||||
// error, which the client's stall UI already covers.
|
||||
let backoff = std::cmp::max(
|
||||
interval,
|
||||
std::time::Duration::from_millis(100u64 << (encoder_resets - 1).min(4)),
|
||||
);
|
||||
next = std::time::Instant::now() + backoff;
|
||||
std::thread::sleep(backoff);
|
||||
continue;
|
||||
}
|
||||
let submit_us = if measure {
|
||||
|
||||
Reference in New Issue
Block a user