fix(host): NVENC open-failure resilience — backoff, failed-open hygiene, self-diagnosis

Field report (Linux direct NVENC): after a codec switch, every session open
failed with NV_ENC_ERR_INVALID_VERSION until the host process was restarted —
so the poisoned state is per-process, not a driver install issue. On-hardware
investigation (RTX 5070 Ti, 610.43.03) could not reproduce it with clean codec
cycles, dirty teardowns, or open/destroy storms, but established the failure
class: the driver enforces a per-process concurrent-session cap (12 there,
status INCOMPATIBLE_CLIENT_KEY; other branches report differently) whose
exhaustion is exactly this signature — persistent open failures healed only by
a process restart. Harden every path that can feed or mask that state:

* Rebuild backoff: the in-place encoder-rebuild retries slept one frame
  interval, so all 5 attempts burned within ~40 ms at 120 Hz — no driver-side
  transient (deferred teardown of the previous session, engine reset) can
  clear that fast. Exponential backoff 100 ms → 1.6 s (~3 s total) so
  transients heal instead of killing the session.
* Destroy-on-failed-open (Linux + Windows, all four open sites): the NVENC
  docs require NvEncDestroyEncoder even when OpenEncodeSessionEx FAILS — the
  driver may have allocated the session slot before erroring. Without it a
  retry burst against a transient leaks slots toward the cap, converting the
  transient into permanent exhaustion.
* Teardown: a destroy_encoder failure (a session slot the driver may keep) is
  now logged with its status instead of silently discarded.
* One-shot self-diagnosis on a failed session open (Linux): retry the raw open
  on a fresh dedicated CUDA context and log which of the three causes applies
  — shared-context poisoned (fresh works), driver-level skew/exhaustion/GPU
  loss (fresh fails the same way), or CUDA itself unhealthy (no fresh context)
  — so the next field report pinpoints the root cause with zero reporter
  effort.

On-hardware regression tests (RTX box .21, all green): codec-switch reopen
cycle (H265→AV1→H265→H264→H265), dirty teardown with in-flight encodes, and
the full open-failure→diagnosis→in-place-recovery path via real session-cap
exhaustion. Existing RFI/reconfigure/4:4:4 smokes still pass; clippy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

crates/punktfunk-host/src/encode.rs

@@ -1379,6 +1379,11 @@ mod nvenc;
 #[cfg(all(target_os = "linux", feature = "nvenc"))]
 #[path = "encode/linux/nvenc_cuda.rs"]
 mod nvenc_cuda;
+// Actionable `NVENCSTATUS` → cause mapping shared by both direct-NVENC backends, so a failed
+// session open logs "update/reboot the driver" instead of the old misleading "(no NVIDIA GPU?)".
+#[cfg(all(any(target_os = "linux", target_os = "windows"), feature = "nvenc"))]
+#[path = "encode/nvenc_status.rs"]
+mod nvenc_status;
 // Software (openh264) H.264 encoder — the GPU-less path on BOTH Windows and Linux (a headless /
 // GPU-less test box, or a fallback when no hardware encoder is available). Platform-agnostic: it
 // consumes CPU RGB `CapturedFrame`s and the statically-bundled openh264 build.

crates/punktfunk-host/src/encode/linux/nvenc_cuda.rs

@@ -31,6 +31,7 @@
 // Every `unsafe` block / impl in this file carries a `// SAFETY:` proof; enforce it.
 #![deny(clippy::undocumented_unsafe_blocks)]
 
+use super::nvenc_status;
 use super::{ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps};
 use crate::capture::{CapturedFrame, FramePayload};
 use crate::zerocopy::cuda::{self, InputSurface};
@@ -317,6 +318,9 @@ pub struct NvencCudaEncoder {
     cursor: Option<cuda::CursorBlend>,
     cursor_tried: bool,
     cursor_serial: u64,
+    /// One-shot latch for [`diagnose_failed_open`](Self::diagnose_failed_open) so a rebuild-retry
+    /// burst (the session loop's bounded encoder resets) logs the diagnosis once, not per attempt.
+    diagnosed: bool,
 }
 
 // SAFETY: the `!Send` fields are the raw NVENC session handle (`encoder`), the shared `CUcontext`
@@ -382,6 +386,7 @@ impl NvencCudaEncoder {
             cursor: None,
             cursor_tried: false,
             cursor_serial: u64::MAX,
+            diagnosed: false,
             inited: false,
             rfi_supported: false,
             custom_vbv: false,
@@ -407,7 +412,16 @@ impl NvencCudaEncoder {
         for &bs in &self.bitstreams {
             let _ = (api().destroy_bitstream_buffer)(self.encoder, bs);
         }
-        let _ = (api().destroy_encoder)(self.encoder);
+        // A destroy failure means the driver may still hold this session's slot (the concurrent-
+        // session cap is per process and only a restart clears a leak) — make it visible instead
+        // of silently discarding the status.
+        if let Err(e) = (api().destroy_encoder)(self.encoder).nv_ok() {
+            tracing::warn!(
+                status = ?e,
+                "NVENC destroy_encoder failed at teardown — the driver may have leaked this \
+                 session's slot toward the concurrent-session cap"
+            );
+        }
         self.ring.clear(); // drops the InputSurfaces, freeing their CUDA allocations
         self.bitstreams.clear();
         self.pending.clear();
@@ -446,11 +460,19 @@ impl NvencCudaEncoder {
             ..Default::default()
         };
         let mut enc: *mut c_void = ptr::null_mut();
-        (api().open_encode_session_ex)(&mut params, &mut enc)
-            .nv_ok()
-            .map_err(|e| {
-                anyhow!("NVENC open_encode_session_ex (caps probe): {e:?} (no NVIDIA GPU?)")
-            })?;
+        if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
+            // The NVENC docs require NvEncDestroyEncoder even after a FAILED open (the driver may
+            // have allocated the session slot before erroring) — without it, every failed open in
+            // a retry loop leaks a slot toward the concurrent-session cap, turning a transient
+            // failure into permanent exhaustion that only a host restart clears.
+            if !enc.is_null() {
+                let _ = (api().destroy_encoder)(enc);
+            }
+            return Err(nvenc_status::call_err(
+                "open_encode_session_ex (caps probe)",
+                e,
+            ));
+        }
         let wmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_WIDTH_MAX);
         let hmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_HEIGHT_MAX);
         let yuv444 = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_YUV444_ENCODE);
@@ -490,6 +512,62 @@ impl NvencCudaEncoder {
         Ok(())
     }
 
+    /// One-shot self-diagnosis for a failed session open (2026-07 field report: after a codec
+    /// switch every open returned `NV_ENC_ERR_INVALID_VERSION` until the HOST PROCESS was
+    /// restarted — so the poisoned state is per-process, not the driver install). Retries the raw
+    /// open on a FRESH dedicated CUDA context to split the candidate causes apart in the log:
+    ///   * fresh context WORKS  → the shared process context (or its NVENC association) is in a
+    ///     bad state — a host bug to report;
+    ///   * fresh context fails the SAME way → driver-level: userspace/kernel version skew,
+    ///     concurrent-session-cap exhaustion (leaked sessions), or a lost/reset GPU;
+    ///   * no fresh context AT ALL → CUDA itself is unhealthy in this process.
+    ///
+    /// Log-only (the caller still fails the open); latched per encoder so a reset burst logs once.
+    fn diagnose_failed_open(&mut self) {
+        if self.diagnosed {
+            return;
+        }
+        self.diagnosed = true;
+        let fresh = cuda::with_fresh_context(|ctx| {
+            let mut params = nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS {
+                version: nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER,
+                deviceType: nv::NV_ENC_DEVICE_TYPE::NV_ENC_DEVICE_TYPE_CUDA,
+                device: ctx,
+                apiVersion: nv::NVENCAPI_VERSION,
+                ..Default::default()
+            };
+            let mut enc: *mut c_void = ptr::null_mut();
+            // SAFETY: `params`/`enc` are live stack locals across the synchronous call; `ctx` is
+            // the live diagnostic context `with_fresh_context` just created. Any session the probe
+            // opened (even on a failed status, per the NVENC docs) is destroyed exactly once here.
+            unsafe {
+                let st = (api().open_encode_session_ex)(&mut params, &mut enc);
+                if !enc.is_null() {
+                    let _ = (api().destroy_encoder)(enc);
+                }
+                st
+            }
+        });
+        match fresh {
+            Ok(nv::NVENCSTATUS::NV_ENC_SUCCESS) => tracing::error!(
+                "NVENC self-diagnosis: the session opens FINE on a fresh CUDA context — the \
+                 host's shared CUDA context is in a bad state (host bug; please report this log)"
+            ),
+            Ok(st) => tracing::error!(
+                fresh_ctx_status = ?st,
+                "NVENC self-diagnosis: the open fails on a fresh CUDA context too — driver-level \
+                 cause: {}",
+                nvenc_status::explain(st)
+            ),
+            Err(e) => tracing::error!(
+                error = %format!("{e:#}"),
+                "NVENC self-diagnosis: could not create a fresh CUDA context — CUDA itself is \
+                 unhealthy in this process (GPU reset/fell off the bus, or a poisoned driver \
+                 state); a host restart should clear it"
+            ),
+        }
+    }
+
     /// Author the session's `NV_ENC_CONFIG` at `bitrate` (bps): the P1/ULL preset (queried on
     /// `enc`) seeded with the RC/tier/chroma/VUI/DPB shape this backend always runs. ONE builder
     /// shared by [`try_open_session`] and [`Encoder::reconfigure_bitrate`], so an in-place rate
@@ -512,7 +590,7 @@ impl NvencCudaEncoder {
             &mut preset,
         )
         .nv_ok()
-        .map_err(|e| anyhow!("get_encode_preset_config_ex: {e:?}"))?;
+        .map_err(|e| nvenc_status::call_err("get_encode_preset_config_ex", e))?;
         let mut cfg = preset.presetCfg;
 
         // CBR, infinite GOP, P-only, ~1-frame VBV (mirror the Windows/Linux-libav RC config).
@@ -681,9 +759,14 @@ impl NvencCudaEncoder {
             ..Default::default()
         };
         let mut enc: *mut c_void = ptr::null_mut();
-        (api().open_encode_session_ex)(&mut params, &mut enc)
-            .nv_ok()
-            .map_err(|e| anyhow!("NVENC open_encode_session_ex: {e:?} (no NVIDIA GPU?)"))?;
+        if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
+            // Destroy-on-failed-open, as in `query_caps`: a failed open may still hold a session
+            // slot that must be released.
+            if !enc.is_null() {
+                let _ = (api().destroy_encoder)(enc);
+            }
+            return Err(nvenc_status::call_err("open_encode_session_ex", e));
+        }
 
         let mut cfg = match self.build_config(enc, bitrate) {
             Ok(cfg) => cfg,
@@ -698,7 +781,7 @@ impl NvencCudaEncoder {
             Ok(()) => Ok(enc),
             Err(e) => {
                 let _ = (api().destroy_encoder)(enc);
-                Err(anyhow!("initialize_encoder: {e:?}"))
+                Err(nvenc_status::call_err("initialize_encoder", e))
             }
         }
     }
@@ -718,7 +801,12 @@ impl NvencCudaEncoder {
             self.cu_ctx = cuda::context().context("shared CUDA context (Linux direct NVENC)")?;
             cuda::make_current().context("cuCtxSetCurrent (encode thread)")?;
 
-            self.query_caps()?;
+            if let Err(e) = self.query_caps() {
+                // The one place every session-open failure funnels through (the probe is the first
+                // open of any session) — run the one-shot self-diagnosis before propagating.
+                self.diagnose_failed_open();
+                return Err(e);
+            }
             const FLOOR_BPS: u64 = 10_000_000;
             let requested_bps = self.bitrate_bps;
             // 2-way NVENC split-frame encoding (Ada dual-NVENC) above ~1 Gpix/s; env override
@@ -818,7 +906,7 @@ impl NvencCudaEncoder {
                 };
                 (api().create_bitstream_buffer)(enc, &mut cb)
                     .nv_ok()
-                    .map_err(|e| anyhow!("create_bitstream_buffer: {e:?}"))?;
+                    .map_err(|e| nvenc_status::call_err("create_bitstream_buffer", e))?;
                 self.bitstreams.push(cb.bitstreamBuffer);
             }
 
@@ -849,7 +937,7 @@ impl NvencCudaEncoder {
                 };
                 (api().register_resource)(self.encoder, &mut rr)
                     .nv_ok()
-                    .map_err(|e| anyhow!("register_resource (CUDADEVICEPTR): {e:?}"))?;
+                    .map_err(|e| nvenc_status::call_err("register_resource (CUDADEVICEPTR)", e))?;
                 self.ring.push(RingSlot {
                     surface,
                     reg: rr.registeredResource,
@@ -1014,7 +1102,7 @@ impl Encoder for NvencCudaEncoder {
             };
             (api().map_input_resource)(self.encoder, &mut mp)
                 .nv_ok()
-                .map_err(|e| anyhow!("map_input_resource: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("map_input_resource", e))?;
 
             let pts = self.frame_idx as u64;
             self.frame_idx += 1;
@@ -1086,7 +1174,7 @@ impl Encoder for NvencCudaEncoder {
             }
             (api().encode_picture)(self.encoder, &mut pic)
                 .nv_ok()
-                .map_err(|e| anyhow!("encode_picture: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("encode_picture", e))?;
             self.pending.push_back((
                 self.bitstreams[slot],
                 mp.mappedResource,
@@ -1186,7 +1274,7 @@ impl Encoder for NvencCudaEncoder {
             };
             (api().lock_bitstream)(self.encoder, &mut lock)
                 .nv_ok()
-                .map_err(|e| anyhow!("lock_bitstream: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("lock_bitstream", e))?;
             let data = std::slice::from_raw_parts(
                 lock.bitstreamBufferPtr as *const u8,
                 lock.bitstreamSizeInBytes as usize,
@@ -1198,7 +1286,7 @@ impl Encoder for NvencCudaEncoder {
             );
             (api().unlock_bitstream)(self.encoder, bs)
                 .nv_ok()
-                .map_err(|e| anyhow!("unlock_bitstream: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("unlock_bitstream", e))?;
             if !map.is_null() {
                 let _ = (api().unmap_input_resource)(self.encoder, map);
             }
@@ -1514,4 +1602,161 @@ mod tests {
             "negative first → decline"
         );
     }
+
+    fn open_h265() -> NvencCudaEncoder {
+        NvencCudaEncoder::open(
+            Codec::H265,
+            PixelFormat::Nv12,
+            1280,
+            720,
+            60,
+            20_000_000,
+            true,
+            8,
+            ChromaFormat::Yuv420,
+        )
+        .expect("open NVENC CUDA encoder")
+    }
+
+    /// ON-HARDWARE: the codec-switch lifecycle from the 2026-07 field report ("switching the
+    /// codec leaves the host unable to bring the encoder up until a restart") — cycle sessions
+    /// across codecs in ONE process, clean drain per leg. Every leg must open and encode. Run:
+    ///   cargo test -p punktfunk-host --features nvenc -- --ignored nvenc_cuda_codec_switch --nocapture
+    #[test]
+    #[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box (.21)"]
+    fn nvenc_cuda_codec_switch_reopen() {
+        const W: u32 = 1280;
+        const H: u32 = 720;
+        crate::zerocopy::cuda::make_current().expect("shared CUDA context current");
+        for (leg, codec) in [
+            Codec::H265,
+            Codec::Av1,
+            Codec::H265,
+            Codec::H264,
+            Codec::H265,
+        ]
+        .into_iter()
+        .enumerate()
+        {
+            let mut enc = NvencCudaEncoder::open(
+                codec,
+                PixelFormat::Nv12,
+                W,
+                H,
+                60,
+                20_000_000,
+                true,
+                8,
+                ChromaFormat::Yuv420,
+            )
+            .expect("open");
+            for f in 0..4u32 {
+                let frame = nv12_frame(W, H, f);
+                enc.submit_indexed(&frame, f)
+                    .unwrap_or_else(|e| panic!("leg {leg} {codec:?} submit failed: {e:#}"));
+                while enc.poll().expect("poll").is_some() {}
+            }
+            drop(enc);
+        }
+        println!("nvenc_cuda codec-switch: 5 legs across H265/AV1/H264, all clean");
+    }
+
+    /// ON-HARDWARE: dirty teardown — drop encoders with encodes still in flight (what a
+    /// mid-stream session kill does), several times, then a fresh session must still open. Guards
+    /// the teardown-with-pending path against driver-side session-slot leaks.
+    #[test]
+    #[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box (.21)"]
+    fn nvenc_cuda_dirty_teardown_reopen() {
+        const W: u32 = 1280;
+        const H: u32 = 720;
+        crate::zerocopy::cuda::make_current().expect("shared CUDA context current");
+        for round in 0..3 {
+            let mut enc = open_h265();
+            for f in 0..4u32 {
+                let frame = nv12_frame(W, H, f);
+                enc.submit_indexed(&frame, f)
+                    .unwrap_or_else(|e| panic!("round {round} submit {f} failed: {e:#}"));
+            }
+            drop(enc); // teardown with 4 in-flight encodes
+        }
+        let mut enc = open_h265();
+        let frame = nv12_frame(W, H, 0);
+        enc.submit_indexed(&frame, 0)
+            .expect("reopen after dirty teardowns");
+        while enc.poll().expect("poll").is_some() {}
+        println!("nvenc_cuda dirty-teardown: 3 dirty drops, reopen clean");
+    }
+
+    /// ON-HARDWARE: the session-open failure path end to end — exhaust the driver's concurrent-
+    /// session cap with raw opens, assert a real encoder open fails with the actionable error
+    /// (and fires the one-shot self-diagnosis), then free the slots and assert the SAME encoder
+    /// rebuilds in place and produces an AU. This is the transient the session loop's rebuild
+    /// backoff is sized to outlive; on the RTX 5070 Ti (driver 610.43.03) the cap is 12 sessions
+    /// and the failure status is `NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY`.
+    #[test]
+    #[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box (.21)"]
+    fn nvenc_cuda_open_failure_diagnosis_and_recovery() {
+        const W: u32 = 1280;
+        const H: u32 = 720;
+        crate::zerocopy::cuda::make_current().expect("shared CUDA context current");
+        try_api().expect("nvenc api");
+        let shared = cuda::context().expect("shared ctx");
+
+        let open_raw = |device: *mut c_void| -> (nv::NVENCSTATUS, *mut c_void) {
+            let mut params = nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS {
+                version: nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER,
+                deviceType: nv::NV_ENC_DEVICE_TYPE::NV_ENC_DEVICE_TYPE_CUDA,
+                device,
+                apiVersion: nv::NVENCAPI_VERSION,
+                ..Default::default()
+            };
+            let mut enc: *mut c_void = ptr::null_mut();
+            // SAFETY: live params/out-param across the synchronous call; test-only.
+            let st = unsafe { (api().open_encode_session_ex)(&mut params, &mut enc) };
+            (st, enc)
+        };
+
+        // Exhaust the concurrent-session cap.
+        let mut held = Vec::new();
+        loop {
+            let (st, enc) = open_raw(shared);
+            if st != nv::NVENCSTATUS::NV_ENC_SUCCESS {
+                if !enc.is_null() {
+                    // SAFETY: destroy the failed-open residue per the NVENC docs.
+                    unsafe {
+                        let _ = (api().destroy_encoder)(enc);
+                    }
+                }
+                break;
+            }
+            held.push(enc);
+        }
+        assert!(!held.is_empty(), "expected a finite session cap");
+
+        // A real encoder open must now fail (lazy init → caps probe) with the actionable error.
+        let mut enc = open_h265();
+        let frame = nv12_frame(W, H, 0);
+        let err = enc
+            .submit_indexed(&frame, 0)
+            .expect_err("submit must fail while the cap is exhausted");
+        println!("at-cap error (self-diagnosis logged alongside): {err:#}");
+
+        // The transient clears (slots freed) → the SAME encoder rebuilds in place and encodes.
+        for e in held {
+            // SAFETY: e came from a successful raw open above; destroyed exactly once.
+            unsafe {
+                let _ = (api().destroy_encoder)(e);
+            }
+        }
+        assert!(enc.reset(), "in-place reset must be available");
+        let frame = nv12_frame(W, H, 1);
+        enc.submit_indexed(&frame, 1)
+            .expect("rebuild after the transient cleared");
+        let mut got = false;
+        while enc.poll().expect("poll").is_some() {
+            got = true;
+        }
+        assert!(got, "recovered encoder must produce an AU");
+        println!("nvenc_cuda open-failure recovery: cap hit → diagnosed → recovered in place");
+    }
 }

crates/punktfunk-host/src/encode/nvenc_status.rs

@@ -0,0 +1,80 @@
+//! Actionable explanations for `NVENCSTATUS` failures — shared by the direct-SDK NVENC backends on
+//! Windows (`encode/windows/nvenc.rs`) and Linux (`encode/linux/nvenc_cuda.rs`).
+//!
+//! Every NVENC entry-point failure used to be annotated `(no NVIDIA GPU?)`, which actively misled
+//! triage: the direct-NVENC path only loads on a machine that HAS an NVIDIA GPU, and the failure a
+//! user actually hit — `NV_ENC_ERR_INVALID_VERSION` from a userspace/kernel driver version skew,
+//! fixed by a reboot — has nothing to do with a missing GPU. This maps each status to what it really
+//! means and what the operator should do, and folds that cause into the `anyhow::Error` at
+//! construction, so every downstream `{e:#}` log (the encode-recovery loop, session teardown) says
+//! the useful thing without extra plumbing.
+
+use nvidia_video_codec_sdk::sys::nvEncodeAPI as nv;
+
+/// A one-line, operator-actionable cause for an NVENC status. Does not repeat the raw code —
+/// callers print that alongside (see [`call_err`]). Public for the few sites that build a
+/// `String`/`format!` error instead of an `anyhow::Error`.
+pub(super) fn explain(status: nv::NVENCSTATUS) -> String {
+    match status {
+        // The one this whole module exists for: a version word the driver rejects. Either the
+        // driver is genuinely older than our headers, or (the sneaky case) the userspace
+        // `libnvidia-encode` reports a new-enough version to the pre-flight probe but the running
+        // kernel module is older and rejects the session — the classic "updated the driver, didn't
+        // reboot" skew. Both heal the same way.
+        nv::NVENCSTATUS::NV_ENC_ERR_INVALID_VERSION => format!(
+            "the NVIDIA driver is older than this build's NVENC headers (needs NVENC API {}.{} or \
+             newer), or the userspace and kernel-module driver versions are mismatched — common \
+             right after a driver update without a reboot. Update the NVIDIA driver, or reboot if \
+             you just updated it (a host restart is the usual fix).",
+            nv::NVENCAPI_MAJOR_VERSION,
+            nv::NVENCAPI_MINOR_VERSION,
+        ),
+        nv::NVENCSTATUS::NV_ENC_ERR_NO_ENCODE_DEVICE => {
+            "this GPU exposes no usable NVENC engine — it has no hardware video encoder, or NVENC is \
+             disabled on this card"
+                .to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_UNSUPPORTED_DEVICE => {
+            "this GPU model is not supported by NVENC".to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_INVALID_ENCODERDEVICE
+        | nv::NVENCSTATUS::NV_ENC_ERR_INVALID_DEVICE => {
+            "the device/context handed to NVENC is invalid — a GPU reset or driver reload can cause \
+             this"
+                .to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_DEVICE_NOT_EXIST => {
+            "the NVENC device no longer exists — the driver reset, or the GPU fell off the bus"
+                .to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_OUT_OF_MEMORY => "the GPU is out of memory".to_string(),
+        nv::NVENCSTATUS::NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY => {
+            "NVENC rejected the client key — the GeForce concurrent-NVENC-session limit was reached, \
+             or the driver is unlicensed for this many encoders"
+                .to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_UNIMPLEMENTED
+        | nv::NVENCSTATUS::NV_ENC_ERR_UNSUPPORTED_PARAM => {
+            "this driver/GPU does not implement the requested NVENC encode mode".to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_INVALID_PARAM => {
+            "NVENC rejected a parameter — an encode mode this GPU does not support".to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_ENCODER_BUSY => {
+            "the NVENC engine is busy — retry, or reduce the number of concurrent encode sessions"
+                .to_string()
+        }
+        nv::NVENCSTATUS::NV_ENC_ERR_GENERIC => {
+            "the NVIDIA driver returned a generic NVENC failure — check dmesg and the driver install"
+                .to_string()
+        }
+        other => format!("unexpected NVENC status ({other:?})"),
+    }
+}
+
+/// Build an actionable `anyhow::Error` for a failed NVENC entry-point call. `call` names the API
+/// (e.g. `"open_encode_session_ex"`); the message carries both the raw status and its real-world
+/// cause, so triage never again reads a version mismatch as "(no NVIDIA GPU?)".
+pub(super) fn call_err(call: &str, status: nv::NVENCSTATUS) -> anyhow::Error {
+    anyhow::anyhow!("NVENC {call} failed: {status:?} — {}", explain(status))
+}

crates/punktfunk-host/src/encode/windows/nvenc.rs

@@ -36,6 +36,7 @@
 // Every `unsafe` block / impl in this file carries a `// SAFETY:` proof; enforce it.
 #![deny(clippy::undocumented_unsafe_blocks)]
 
+use super::nvenc_status;
 use super::{ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps};
 use crate::capture::{CapturedFrame, FramePayload, PixelFormat};
 use anyhow::{anyhow, bail, Context, Result};
@@ -381,7 +382,10 @@ fn retrieve_loop(
                         let _ = (api().unlock_bitstream)(enc as *mut c_void, job.bs as *mut c_void);
                         Ok((data, keyframe))
                     }
-                    Err(e) => Err(format!("lock_bitstream (async): {e:?}")),
+                    Err(e) => Err(format!(
+                        "lock_bitstream (async): {e:?} — {}",
+                        nvenc_status::explain(e)
+                    )),
                 }
             }
         };
@@ -585,7 +589,16 @@ impl NvencD3d11Encoder {
         for &bs in &self.bitstreams {
             let _ = (api().destroy_bitstream_buffer)(self.encoder, bs);
         }
-        let _ = (api().destroy_encoder)(self.encoder);
+        // A destroy failure means the driver may still hold this session's slot (the concurrent-
+        // session cap is per process and only a restart clears a leak) — make it visible instead
+        // of silently discarding the status.
+        if let Err(e) = (api().destroy_encoder)(self.encoder).nv_ok() {
+            tracing::warn!(
+                status = ?e,
+                "NVENC destroy_encoder failed at teardown — the driver may have leaked this \
+                 session's slot toward the concurrent-session cap"
+            );
+        }
         // Return this session's units to the budget (see LIVE_SESSION_UNITS).
         LIVE_SESSION_UNITS.fetch_sub(self.session_units, std::sync::atomic::Ordering::Relaxed);
         self.session_units = 0;
@@ -633,11 +646,19 @@ impl NvencD3d11Encoder {
             ..Default::default()
         };
         let mut enc: *mut c_void = ptr::null_mut();
-        (api().open_encode_session_ex)(&mut params, &mut enc)
-            .nv_ok()
-            .map_err(|e| {
-                anyhow!("NVENC open_encode_session_ex (caps probe): {e:?} (no NVIDIA GPU?)")
-            })?;
+        if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
+            // The NVENC docs require NvEncDestroyEncoder even after a FAILED open (the driver may
+            // have allocated the session slot before erroring) — without it, every failed open in
+            // a retry loop leaks a slot toward the concurrent-session cap, turning a transient
+            // failure into permanent exhaustion that only a host restart clears.
+            if !enc.is_null() {
+                let _ = (api().destroy_encoder)(enc);
+            }
+            return Err(nvenc_status::call_err(
+                "open_encode_session_ex (caps probe)",
+                e,
+            ));
+        }
         let wmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_WIDTH_MAX);
         let hmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_HEIGHT_MAX);
         let ten_bit = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_10BIT_ENCODE);
@@ -712,7 +733,7 @@ impl NvencD3d11Encoder {
             &mut preset,
         )
         .nv_ok()
-        .map_err(|e| anyhow!("get_encode_preset_config_ex: {e:?}"))?;
+        .map_err(|e| nvenc_status::call_err("get_encode_preset_config_ex", e))?;
         let mut cfg = preset.presetCfg;
 
         // Mirror the Linux RC config: CBR, infinite GOP, P-only, ~1-frame VBV.
@@ -946,9 +967,14 @@ impl NvencD3d11Encoder {
             ..Default::default()
         };
         let mut enc: *mut c_void = ptr::null_mut();
-        (api().open_encode_session_ex)(&mut params, &mut enc)
-            .nv_ok()
-            .map_err(|e| anyhow!("NVENC open_encode_session_ex: {e:?} (no NVIDIA GPU?)"))?;
+        if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
+            // Destroy-on-failed-open, as in `query_caps`: a failed open may still hold a session
+            // slot that must be released.
+            if !enc.is_null() {
+                let _ = (api().destroy_encoder)(enc);
+            }
+            return Err(nvenc_status::call_err("open_encode_session_ex", e));
+        }
 
         let mut cfg = match self.build_config(enc, bitrate) {
             Ok(cfg) => cfg,
@@ -963,7 +989,7 @@ impl NvencD3d11Encoder {
             Ok(()) => Ok(enc),
             Err(e) => {
                 let _ = (api().destroy_encoder)(enc);
-                Err(anyhow!("initialize_encoder: {e:?}"))
+                Err(nvenc_status::call_err("initialize_encoder", e))
             }
         }
     }
@@ -1139,7 +1165,7 @@ impl NvencD3d11Encoder {
                 };
                 (api().create_bitstream_buffer)(enc, &mut cb)
                     .nv_ok()
-                    .map_err(|e| anyhow!("create_bitstream_buffer: {e:?}"))?;
+                    .map_err(|e| nvenc_status::call_err("create_bitstream_buffer", e))?;
                 self.bitstreams.push(cb.bitstreamBuffer);
             }
             // Async retrieve: one auto-reset completion event per pool bitstream, registered with
@@ -1156,7 +1182,7 @@ impl NvencD3d11Encoder {
                     };
                     (api().register_async_event)(enc, &mut ep)
                         .nv_ok()
-                        .map_err(|e| anyhow!("register_async_event: {e:?}"))?;
+                        .map_err(|e| nvenc_status::call_err("register_async_event", e))?;
                     self.events.push(ev.0 as usize);
                 }
                 let (work_tx, work_rx) = mpsc::sync_channel::<RetrieveJob>(POOL);
@@ -1372,7 +1398,7 @@ impl Encoder for NvencD3d11Encoder {
                 };
                 (api().register_resource)(self.encoder, &mut rr)
                     .nv_ok()
-                    .map_err(|e| anyhow!("register_resource: {e:?}"))?;
+                    .map_err(|e| nvenc_status::call_err("register_resource", e))?;
                 self.regs
                     .insert(key, (rr.registeredResource, frame.texture.clone()));
             }
@@ -1385,7 +1411,7 @@ impl Encoder for NvencD3d11Encoder {
             };
             (api().map_input_resource)(self.encoder, &mut mp)
                 .nv_ok()
-                .map_err(|e| anyhow!("map_input_resource: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("map_input_resource", e))?;
 
             let pts = self.frame_idx as u64;
             self.frame_idx += 1;
@@ -1469,7 +1495,7 @@ impl Encoder for NvencD3d11Encoder {
             }
             (api().encode_picture)(self.encoder, &mut pic)
                 .nv_ok()
-                .map_err(|e| anyhow!("encode_picture: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("encode_picture", e))?;
             self.pending.push_back((
                 self.bitstreams[slot],
                 mp.mappedResource,
@@ -1629,7 +1655,7 @@ impl Encoder for NvencD3d11Encoder {
             };
             (api().lock_bitstream)(self.encoder, &mut lock)
                 .nv_ok()
-                .map_err(|e| anyhow!("lock_bitstream: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("lock_bitstream", e))?;
             let data = std::slice::from_raw_parts(
                 lock.bitstreamBufferPtr as *const u8,
                 lock.bitstreamSizeInBytes as usize,
@@ -1641,7 +1667,7 @@ impl Encoder for NvencD3d11Encoder {
             );
             (api().unlock_bitstream)(self.encoder, bs)
                 .nv_ok()
-                .map_err(|e| anyhow!("unlock_bitstream: {e:?}"))?;
+                .map_err(|e| nvenc_status::call_err("unlock_bitstream", e))?;
             if !map.is_null() {
                 let _ = (api().unmap_input_resource)(self.encoder, map);
             }
@@ -1762,8 +1788,9 @@ pub fn probe_can_encode_444(codec: Codec) -> bool {
     // `EnumAdapterByLuid` return owned COM objects or err (→ default-adapter fallback).
     // `D3D11CreateDevice` (explicit adapter + UNKNOWN driver type, or NULL adapter + HARDWARE)
     // fills `device` or returns Err (→ false). `open_encode_session_ex` opens an NVENC session
-    // against that device's raw pointer (valid while `device` is held) or errors (→ false, tearing
-    // nothing down). `get_encode_caps` reads one scalar cap into `val` via the loaded API table.
+    // against that device's raw pointer (valid while `device` is held) or errors (→ false, after
+    // destroying any residue session the failed open left — the docs require it).
+    // `get_encode_caps` reads one scalar cap into `val` via the loaded API table.
     // `destroy_encoder` frees the session exactly once; `device`/its context drop with the COM
     // wrappers. No handle escapes this call and nothing runs concurrently.
     unsafe {
@@ -1816,6 +1843,10 @@ pub fn probe_can_encode_444(codec: Codec) -> bool {
             .nv_ok()
             .is_err()
         {
+            // Destroy-on-failed-open: a failed open may still hold a session slot.
+            if !enc.is_null() {
+                let _ = (api().destroy_encoder)(enc);
+            }
             return false;
         }
         let mut param = nv::NV_ENC_CAPS_PARAM {

crates/punktfunk-host/src/linux/zerocopy/cuda.rs

@@ -120,6 +120,7 @@ struct CudaApi {
     cuInit: unsafe extern "C" fn(c_uint) -> CUresult,
     cuDeviceGet: unsafe extern "C" fn(*mut CUdevice, c_int) -> CUresult,
     cuCtxCreate_v2: unsafe extern "C" fn(*mut CUcontext, c_uint, CUdevice) -> CUresult,
+    cuCtxDestroy_v2: unsafe extern "C" fn(CUcontext) -> CUresult,
     cuCtxSetCurrent: unsafe extern "C" fn(CUcontext) -> CUresult,
     cuMemAllocPitch_v2:
         unsafe extern "C" fn(*mut CUdeviceptr, *mut usize, usize, usize, c_uint) -> CUresult,
@@ -214,6 +215,7 @@ fn cuda_api() -> Option<&'static CudaApi> {
                 cuInit: *lib.get(b"cuInit\0").ok()?,
                 cuDeviceGet: *lib.get(b"cuDeviceGet\0").ok()?,
                 cuCtxCreate_v2: *lib.get(b"cuCtxCreate_v2\0").ok()?,
+                cuCtxDestroy_v2: *lib.get(b"cuCtxDestroy_v2\0").ok()?,
                 cuCtxSetCurrent: *lib.get(b"cuCtxSetCurrent\0").ok()?,
                 cuMemAllocPitch_v2: *lib.get(b"cuMemAllocPitch_v2\0").ok()?,
                 cuMemFree_v2: *lib.get(b"cuMemFree_v2\0").ok()?,
@@ -275,6 +277,12 @@ unsafe fn cuCtxCreate_v2(pctx: *mut CUcontext, flags: c_uint, dev: CUdevice) ->
         None => CU_ERROR_NOT_LOADED,
     }
 }
+unsafe fn cuCtxDestroy_v2(ctx: CUcontext) -> CUresult {
+    match cuda_api() {
+        Some(a) => (a.cuCtxDestroy_v2)(ctx),
+        None => CU_ERROR_NOT_LOADED,
+    }
+}
 unsafe fn cuCtxSetCurrent(ctx: CUcontext) -> CUresult {
     match cuda_api() {
         Some(a) => (a.cuCtxSetCurrent)(ctx),
@@ -610,6 +618,37 @@ pub fn make_current() -> Result<()> {
     unsafe { ck(cuCtxSetCurrent(ctx), "cuCtxSetCurrent") }
 }
 
+/// DIAGNOSTIC-ONLY: create a fresh dedicated context on device 0, run `probe` with it, destroy it,
+/// and restore the shared context as current. Used by the NVENC backend's session-open
+/// self-diagnosis to split "this process's shared context is in a bad state" (probe succeeds on
+/// the fresh context) from a driver-level condition like version skew or session exhaustion
+/// (probe fails there too). Never used on a hot path.
+pub fn with_fresh_context<R>(probe: impl FnOnce(CUcontext) -> R) -> Result<R> {
+    if cuda_api().is_none() {
+        bail!("libcuda.so.1 not available");
+    }
+    // SAFETY: the driver table is present (checked above). `cuInit(0)` is idempotent. `&mut dev`/
+    // `&mut ctx` are live, distinct stack out-params for their synchronous calls. On success `ctx`
+    // is a valid dedicated context, destroyed exactly once below; the shared context is restored
+    // as current afterwards (creation left the fresh one current on this thread).
+    unsafe {
+        ck(cuInit(0), "cuInit")?;
+        let mut dev: CUdevice = 0;
+        ck(cuDeviceGet(&mut dev, 0), "cuDeviceGet")?;
+        let mut ctx: CUcontext = std::ptr::null_mut();
+        ck(
+            cuCtxCreate_v2(&mut ctx, CU_CTX_SCHED_BLOCKING_SYNC, dev),
+            "cuCtxCreate_v2 (diagnostic)",
+        )?;
+        let r = probe(ctx);
+        let _ = cuCtxDestroy_v2(ctx);
+        if let Some(c) = CONTEXT.get() {
+            let _ = cuCtxSetCurrent(c.0);
+        }
+        Ok(r)
+    }
+}
+
 thread_local! {
     /// Per-thread copy stream. `None` until first use; `Some(null)` means "creation failed, use the
     /// default (NULL) stream". Per-thread (not shared) so each worker's `cuStreamSynchronize` waits

crates/punktfunk-host/src/punktfunk1.rs

@@ -4919,16 +4919,35 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
             if encoder_resets > MAX_ENCODER_RESETS
                 || !reset_stalled_encoder(&mut enc, &mut inflight)
             {
+                // Terminal: rebuilds are exhausted (or the backend can't rebuild in place). Say so
+                // plainly with the underlying cause — the per-reset lines above only ever repeat
+                // "rebuilt in place", so without this the session just vanishes. The error carries
+                // its own actionable text now (e.g. an NVENC version mismatch → "update/reboot the
+                // driver"), so this is the one line an operator needs.
+                tracing::error!(
+                    error = %format!("{e:#}"),
+                    resets = encoder_resets,
+                    "encoder did not recover after repeated in-place rebuilds — ending the video \
+                     session (see the error above for the cause)");
                 return Err(e).context("encoder submit");
             }
             tracing::error!(error = %format!("{e:#}"), reset = encoder_resets,
                 max = MAX_ENCODER_RESETS,
                 "encoder submit failed — encoder rebuilt in place, forcing an IDR");
             last_au_at = std::time::Instant::now();
-            // Re-pace from the rebuild and retry this frame next tick (gives the fresh encoder
-            // one frame period to come up instead of hammering it in a hot loop).
-            next = std::time::Instant::now() + interval;
-            std::thread::sleep(interval);
+            // Back off exponentially between rebuild attempts (100 ms → 1.6 s, ~3 s total across
+            // the reset budget). One frame period is NOT enough: a 2026-07 field report showed all
+            // 5 resets burning within 40 ms at 120 Hz against a driver-side condition (NVENC
+            // session open failing after a codec switch) that no 8 ms retry could outlive — any
+            // transient like the previous session's deferred driver teardown needs real time. A
+            // genuinely dead encoder now costs ~3 s before the session ends with the terminal
+            // error, which the client's stall UI already covers.
+            let backoff = std::cmp::max(
+                interval,
+                std::time::Duration::from_millis(100u64 << (encoder_resets - 1).min(4)),
+            );
+            next = std::time::Instant::now() + backoff;
+            std::thread::sleep(backoff);
             continue;
         }
         let submit_us = if measure {