fix(host): NVENC open-failure resilience — backoff, failed-open hygiene, self-diagnosis
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 51s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
apple / swift (push) Successful in 1m9s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m48s
docker / deploy-docs (push) Successful in 24s
apple / screenshots (push) Successful in 5m18s
windows-host / package (push) Successful in 9m16s
arch / build-publish (push) Successful in 10m53s
android / android (push) Successful in 11m58s
deb / build-publish (push) Successful in 18m7s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m19s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m28s
ci / rust (push) Successful in 24m56s

Field report (Linux direct NVENC): after a codec switch, every session open
failed with NV_ENC_ERR_INVALID_VERSION until the host process was restarted —
so the poisoned state is per-process, not a driver install issue. On-hardware
investigation (RTX 5070 Ti, 610.43.03) could not reproduce it with clean codec
cycles, dirty teardowns, or open/destroy storms, but established the failure
class: the driver enforces a per-process concurrent-session cap (12 there,
status INCOMPATIBLE_CLIENT_KEY; other branches report differently) whose
exhaustion is exactly this signature — persistent open failures healed only by
a process restart. Harden every path that can feed or mask that state:

* Rebuild backoff: the in-place encoder-rebuild retries slept one frame
  interval, so all 5 attempts burned within ~40 ms at 120 Hz — no driver-side
  transient (deferred teardown of the previous session, engine reset) can
  clear that fast. Exponential backoff 100 ms → 1.6 s (~3 s total) so
  transients heal instead of killing the session.
* Destroy-on-failed-open (Linux + Windows, all four open sites): the NVENC
  docs require NvEncDestroyEncoder even when OpenEncodeSessionEx FAILS — the
  driver may have allocated the session slot before erroring. Without it a
  retry burst against a transient leaks slots toward the cap, converting the
  transient into permanent exhaustion.
* Teardown: a destroy_encoder failure (a session slot the driver may keep) is
  now logged with its status instead of silently discarded.
* One-shot self-diagnosis on a failed session open (Linux): retry the raw open
  on a fresh dedicated CUDA context and log which of the three causes applies
  — shared-context poisoned (fresh works), driver-level skew/exhaustion/GPU
  loss (fresh fails the same way), or CUDA itself unhealthy (no fresh context)
  — so the next field report pinpoints the root cause with zero reporter
  effort.

On-hardware regression tests (RTX box .21, all green): codec-switch reopen
cycle (H265→AV1→H265→H264→H265), dirty teardown with in-flight encodes, and
the full open-failure→diagnosis→in-place-recovery path via real session-cap
exhaustion. Existing RFI/reconfigure/4:4:4 smokes still pass; clippy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-16 01:57:36 +02:00
parent f901bedf22
commit d381cdf7f4
4 changed files with 344 additions and 21 deletions
+13 -4
View File
@@ -4935,10 +4935,19 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
max = MAX_ENCODER_RESETS,
"encoder submit failed — encoder rebuilt in place, forcing an IDR");
last_au_at = std::time::Instant::now();
// Re-pace from the rebuild and retry this frame next tick (gives the fresh encoder
// one frame period to come up instead of hammering it in a hot loop).
next = std::time::Instant::now() + interval;
std::thread::sleep(interval);
// Back off exponentially between rebuild attempts (100 ms → 1.6 s, ~3 s total across
// the reset budget). One frame period is NOT enough: a 2026-07 field report showed all
// 5 resets burning within 40 ms at 120 Hz against a driver-side condition (NVENC
// session open failing after a codec switch) that no 8 ms retry could outlive — any
// transient like the previous session's deferred driver teardown needs real time. A
// genuinely dead encoder now costs ~3 s before the session ends with the terminal
// error, which the client's stall UI already covers.
let backoff = std::cmp::max(
interval,
std::time::Duration::from_millis(100u64 << (encoder_resets - 1).min(4)),
);
next = std::time::Instant::now() + backoff;
std::thread::sleep(backoff);
continue;
}
let submit_us = if measure {