fix(host): NVENC open-failure resilience — backoff, failed-open hygiene, self-diagnosis
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 51s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
apple / swift (push) Successful in 1m9s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m48s
docker / deploy-docs (push) Successful in 24s
apple / screenshots (push) Successful in 5m18s
windows-host / package (push) Successful in 9m16s
arch / build-publish (push) Successful in 10m53s
android / android (push) Successful in 11m58s
deb / build-publish (push) Successful in 18m7s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m19s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m28s
ci / rust (push) Successful in 24m56s
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 51s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
apple / swift (push) Successful in 1m9s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m48s
docker / deploy-docs (push) Successful in 24s
apple / screenshots (push) Successful in 5m18s
windows-host / package (push) Successful in 9m16s
arch / build-publish (push) Successful in 10m53s
android / android (push) Successful in 11m58s
deb / build-publish (push) Successful in 18m7s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m19s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m28s
ci / rust (push) Successful in 24m56s
Field report (Linux direct NVENC): after a codec switch, every session open failed with NV_ENC_ERR_INVALID_VERSION until the host process was restarted — so the poisoned state is per-process, not a driver install issue. On-hardware investigation (RTX 5070 Ti, 610.43.03) could not reproduce it with clean codec cycles, dirty teardowns, or open/destroy storms, but established the failure class: the driver enforces a per-process concurrent-session cap (12 there, status INCOMPATIBLE_CLIENT_KEY; other branches report differently) whose exhaustion is exactly this signature — persistent open failures healed only by a process restart. Harden every path that can feed or mask that state: * Rebuild backoff: the in-place encoder-rebuild retries slept one frame interval, so all 5 attempts burned within ~40 ms at 120 Hz — no driver-side transient (deferred teardown of the previous session, engine reset) can clear that fast. Exponential backoff 100 ms → 1.6 s (~3 s total) so transients heal instead of killing the session. * Destroy-on-failed-open (Linux + Windows, all four open sites): the NVENC docs require NvEncDestroyEncoder even when OpenEncodeSessionEx FAILS — the driver may have allocated the session slot before erroring. Without it a retry burst against a transient leaks slots toward the cap, converting the transient into permanent exhaustion. * Teardown: a destroy_encoder failure (a session slot the driver may keep) is now logged with its status instead of silently discarded. * One-shot self-diagnosis on a failed session open (Linux): retry the raw open on a fresh dedicated CUDA context and log which of the three causes applies — shared-context poisoned (fresh works), driver-level skew/exhaustion/GPU loss (fresh fails the same way), or CUDA itself unhealthy (no fresh context) — so the next field report pinpoints the root cause with zero reporter effort. On-hardware regression tests (RTX box .21, all green): codec-switch reopen cycle (H265→AV1→H265→H264→H265), dirty teardown with in-flight encodes, and the full open-failure→diagnosis→in-place-recovery path via real session-cap exhaustion. Existing RFI/reconfigure/4:4:4 smokes still pass; clippy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -318,6 +318,9 @@ pub struct NvencCudaEncoder {
|
|||||||
cursor: Option<cuda::CursorBlend>,
|
cursor: Option<cuda::CursorBlend>,
|
||||||
cursor_tried: bool,
|
cursor_tried: bool,
|
||||||
cursor_serial: u64,
|
cursor_serial: u64,
|
||||||
|
/// One-shot latch for [`diagnose_failed_open`](Self::diagnose_failed_open) so a rebuild-retry
|
||||||
|
/// burst (the session loop's bounded encoder resets) logs the diagnosis once, not per attempt.
|
||||||
|
diagnosed: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
// SAFETY: the `!Send` fields are the raw NVENC session handle (`encoder`), the shared `CUcontext`
|
// SAFETY: the `!Send` fields are the raw NVENC session handle (`encoder`), the shared `CUcontext`
|
||||||
@@ -383,6 +386,7 @@ impl NvencCudaEncoder {
|
|||||||
cursor: None,
|
cursor: None,
|
||||||
cursor_tried: false,
|
cursor_tried: false,
|
||||||
cursor_serial: u64::MAX,
|
cursor_serial: u64::MAX,
|
||||||
|
diagnosed: false,
|
||||||
inited: false,
|
inited: false,
|
||||||
rfi_supported: false,
|
rfi_supported: false,
|
||||||
custom_vbv: false,
|
custom_vbv: false,
|
||||||
@@ -408,7 +412,16 @@ impl NvencCudaEncoder {
|
|||||||
for &bs in &self.bitstreams {
|
for &bs in &self.bitstreams {
|
||||||
let _ = (api().destroy_bitstream_buffer)(self.encoder, bs);
|
let _ = (api().destroy_bitstream_buffer)(self.encoder, bs);
|
||||||
}
|
}
|
||||||
let _ = (api().destroy_encoder)(self.encoder);
|
// A destroy failure means the driver may still hold this session's slot (the concurrent-
|
||||||
|
// session cap is per process and only a restart clears a leak) — make it visible instead
|
||||||
|
// of silently discarding the status.
|
||||||
|
if let Err(e) = (api().destroy_encoder)(self.encoder).nv_ok() {
|
||||||
|
tracing::warn!(
|
||||||
|
status = ?e,
|
||||||
|
"NVENC destroy_encoder failed at teardown — the driver may have leaked this \
|
||||||
|
session's slot toward the concurrent-session cap"
|
||||||
|
);
|
||||||
|
}
|
||||||
self.ring.clear(); // drops the InputSurfaces, freeing their CUDA allocations
|
self.ring.clear(); // drops the InputSurfaces, freeing their CUDA allocations
|
||||||
self.bitstreams.clear();
|
self.bitstreams.clear();
|
||||||
self.pending.clear();
|
self.pending.clear();
|
||||||
@@ -447,9 +460,19 @@ impl NvencCudaEncoder {
|
|||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
let mut enc: *mut c_void = ptr::null_mut();
|
let mut enc: *mut c_void = ptr::null_mut();
|
||||||
(api().open_encode_session_ex)(&mut params, &mut enc)
|
if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
|
||||||
.nv_ok()
|
// The NVENC docs require NvEncDestroyEncoder even after a FAILED open (the driver may
|
||||||
.map_err(|e| nvenc_status::call_err("open_encode_session_ex (caps probe)", e))?;
|
// have allocated the session slot before erroring) — without it, every failed open in
|
||||||
|
// a retry loop leaks a slot toward the concurrent-session cap, turning a transient
|
||||||
|
// failure into permanent exhaustion that only a host restart clears.
|
||||||
|
if !enc.is_null() {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
|
return Err(nvenc_status::call_err(
|
||||||
|
"open_encode_session_ex (caps probe)",
|
||||||
|
e,
|
||||||
|
));
|
||||||
|
}
|
||||||
let wmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_WIDTH_MAX);
|
let wmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_WIDTH_MAX);
|
||||||
let hmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_HEIGHT_MAX);
|
let hmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_HEIGHT_MAX);
|
||||||
let yuv444 = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_YUV444_ENCODE);
|
let yuv444 = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_YUV444_ENCODE);
|
||||||
@@ -489,6 +512,62 @@ impl NvencCudaEncoder {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One-shot self-diagnosis for a failed session open (2026-07 field report: after a codec
|
||||||
|
/// switch every open returned `NV_ENC_ERR_INVALID_VERSION` until the HOST PROCESS was
|
||||||
|
/// restarted — so the poisoned state is per-process, not the driver install). Retries the raw
|
||||||
|
/// open on a FRESH dedicated CUDA context to split the candidate causes apart in the log:
|
||||||
|
/// * fresh context WORKS → the shared process context (or its NVENC association) is in a
|
||||||
|
/// bad state — a host bug to report;
|
||||||
|
/// * fresh context fails the SAME way → driver-level: userspace/kernel version skew,
|
||||||
|
/// concurrent-session-cap exhaustion (leaked sessions), or a lost/reset GPU;
|
||||||
|
/// * no fresh context AT ALL → CUDA itself is unhealthy in this process.
|
||||||
|
///
|
||||||
|
/// Log-only (the caller still fails the open); latched per encoder so a reset burst logs once.
|
||||||
|
fn diagnose_failed_open(&mut self) {
|
||||||
|
if self.diagnosed {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
self.diagnosed = true;
|
||||||
|
let fresh = cuda::with_fresh_context(|ctx| {
|
||||||
|
let mut params = nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS {
|
||||||
|
version: nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER,
|
||||||
|
deviceType: nv::NV_ENC_DEVICE_TYPE::NV_ENC_DEVICE_TYPE_CUDA,
|
||||||
|
device: ctx,
|
||||||
|
apiVersion: nv::NVENCAPI_VERSION,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
let mut enc: *mut c_void = ptr::null_mut();
|
||||||
|
// SAFETY: `params`/`enc` are live stack locals across the synchronous call; `ctx` is
|
||||||
|
// the live diagnostic context `with_fresh_context` just created. Any session the probe
|
||||||
|
// opened (even on a failed status, per the NVENC docs) is destroyed exactly once here.
|
||||||
|
unsafe {
|
||||||
|
let st = (api().open_encode_session_ex)(&mut params, &mut enc);
|
||||||
|
if !enc.is_null() {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
|
st
|
||||||
|
}
|
||||||
|
});
|
||||||
|
match fresh {
|
||||||
|
Ok(nv::NVENCSTATUS::NV_ENC_SUCCESS) => tracing::error!(
|
||||||
|
"NVENC self-diagnosis: the session opens FINE on a fresh CUDA context — the \
|
||||||
|
host's shared CUDA context is in a bad state (host bug; please report this log)"
|
||||||
|
),
|
||||||
|
Ok(st) => tracing::error!(
|
||||||
|
fresh_ctx_status = ?st,
|
||||||
|
"NVENC self-diagnosis: the open fails on a fresh CUDA context too — driver-level \
|
||||||
|
cause: {}",
|
||||||
|
nvenc_status::explain(st)
|
||||||
|
),
|
||||||
|
Err(e) => tracing::error!(
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
"NVENC self-diagnosis: could not create a fresh CUDA context — CUDA itself is \
|
||||||
|
unhealthy in this process (GPU reset/fell off the bus, or a poisoned driver \
|
||||||
|
state); a host restart should clear it"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Author the session's `NV_ENC_CONFIG` at `bitrate` (bps): the P1/ULL preset (queried on
|
/// Author the session's `NV_ENC_CONFIG` at `bitrate` (bps): the P1/ULL preset (queried on
|
||||||
/// `enc`) seeded with the RC/tier/chroma/VUI/DPB shape this backend always runs. ONE builder
|
/// `enc`) seeded with the RC/tier/chroma/VUI/DPB shape this backend always runs. ONE builder
|
||||||
/// shared by [`try_open_session`] and [`Encoder::reconfigure_bitrate`], so an in-place rate
|
/// shared by [`try_open_session`] and [`Encoder::reconfigure_bitrate`], so an in-place rate
|
||||||
@@ -680,9 +759,14 @@ impl NvencCudaEncoder {
|
|||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
let mut enc: *mut c_void = ptr::null_mut();
|
let mut enc: *mut c_void = ptr::null_mut();
|
||||||
(api().open_encode_session_ex)(&mut params, &mut enc)
|
if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
|
||||||
.nv_ok()
|
// Destroy-on-failed-open, as in `query_caps`: a failed open may still hold a session
|
||||||
.map_err(|e| nvenc_status::call_err("open_encode_session_ex", e))?;
|
// slot that must be released.
|
||||||
|
if !enc.is_null() {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
|
return Err(nvenc_status::call_err("open_encode_session_ex", e));
|
||||||
|
}
|
||||||
|
|
||||||
let mut cfg = match self.build_config(enc, bitrate) {
|
let mut cfg = match self.build_config(enc, bitrate) {
|
||||||
Ok(cfg) => cfg,
|
Ok(cfg) => cfg,
|
||||||
@@ -717,7 +801,12 @@ impl NvencCudaEncoder {
|
|||||||
self.cu_ctx = cuda::context().context("shared CUDA context (Linux direct NVENC)")?;
|
self.cu_ctx = cuda::context().context("shared CUDA context (Linux direct NVENC)")?;
|
||||||
cuda::make_current().context("cuCtxSetCurrent (encode thread)")?;
|
cuda::make_current().context("cuCtxSetCurrent (encode thread)")?;
|
||||||
|
|
||||||
self.query_caps()?;
|
if let Err(e) = self.query_caps() {
|
||||||
|
// The one place every session-open failure funnels through (the probe is the first
|
||||||
|
// open of any session) — run the one-shot self-diagnosis before propagating.
|
||||||
|
self.diagnose_failed_open();
|
||||||
|
return Err(e);
|
||||||
|
}
|
||||||
const FLOOR_BPS: u64 = 10_000_000;
|
const FLOOR_BPS: u64 = 10_000_000;
|
||||||
let requested_bps = self.bitrate_bps;
|
let requested_bps = self.bitrate_bps;
|
||||||
// 2-way NVENC split-frame encoding (Ada dual-NVENC) above ~1 Gpix/s; env override
|
// 2-way NVENC split-frame encoding (Ada dual-NVENC) above ~1 Gpix/s; env override
|
||||||
@@ -1513,4 +1602,161 @@ mod tests {
|
|||||||
"negative first → decline"
|
"negative first → decline"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn open_h265() -> NvencCudaEncoder {
|
||||||
|
NvencCudaEncoder::open(
|
||||||
|
Codec::H265,
|
||||||
|
PixelFormat::Nv12,
|
||||||
|
1280,
|
||||||
|
720,
|
||||||
|
60,
|
||||||
|
20_000_000,
|
||||||
|
true,
|
||||||
|
8,
|
||||||
|
ChromaFormat::Yuv420,
|
||||||
|
)
|
||||||
|
.expect("open NVENC CUDA encoder")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// ON-HARDWARE: the codec-switch lifecycle from the 2026-07 field report ("switching the
|
||||||
|
/// codec leaves the host unable to bring the encoder up until a restart") — cycle sessions
|
||||||
|
/// across codecs in ONE process, clean drain per leg. Every leg must open and encode. Run:
|
||||||
|
/// cargo test -p punktfunk-host --features nvenc -- --ignored nvenc_cuda_codec_switch --nocapture
|
||||||
|
#[test]
|
||||||
|
#[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box (.21)"]
|
||||||
|
fn nvenc_cuda_codec_switch_reopen() {
|
||||||
|
const W: u32 = 1280;
|
||||||
|
const H: u32 = 720;
|
||||||
|
crate::zerocopy::cuda::make_current().expect("shared CUDA context current");
|
||||||
|
for (leg, codec) in [
|
||||||
|
Codec::H265,
|
||||||
|
Codec::Av1,
|
||||||
|
Codec::H265,
|
||||||
|
Codec::H264,
|
||||||
|
Codec::H265,
|
||||||
|
]
|
||||||
|
.into_iter()
|
||||||
|
.enumerate()
|
||||||
|
{
|
||||||
|
let mut enc = NvencCudaEncoder::open(
|
||||||
|
codec,
|
||||||
|
PixelFormat::Nv12,
|
||||||
|
W,
|
||||||
|
H,
|
||||||
|
60,
|
||||||
|
20_000_000,
|
||||||
|
true,
|
||||||
|
8,
|
||||||
|
ChromaFormat::Yuv420,
|
||||||
|
)
|
||||||
|
.expect("open");
|
||||||
|
for f in 0..4u32 {
|
||||||
|
let frame = nv12_frame(W, H, f);
|
||||||
|
enc.submit_indexed(&frame, f)
|
||||||
|
.unwrap_or_else(|e| panic!("leg {leg} {codec:?} submit failed: {e:#}"));
|
||||||
|
while enc.poll().expect("poll").is_some() {}
|
||||||
|
}
|
||||||
|
drop(enc);
|
||||||
|
}
|
||||||
|
println!("nvenc_cuda codec-switch: 5 legs across H265/AV1/H264, all clean");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// ON-HARDWARE: dirty teardown — drop encoders with encodes still in flight (what a
|
||||||
|
/// mid-stream session kill does), several times, then a fresh session must still open. Guards
|
||||||
|
/// the teardown-with-pending path against driver-side session-slot leaks.
|
||||||
|
#[test]
|
||||||
|
#[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box (.21)"]
|
||||||
|
fn nvenc_cuda_dirty_teardown_reopen() {
|
||||||
|
const W: u32 = 1280;
|
||||||
|
const H: u32 = 720;
|
||||||
|
crate::zerocopy::cuda::make_current().expect("shared CUDA context current");
|
||||||
|
for round in 0..3 {
|
||||||
|
let mut enc = open_h265();
|
||||||
|
for f in 0..4u32 {
|
||||||
|
let frame = nv12_frame(W, H, f);
|
||||||
|
enc.submit_indexed(&frame, f)
|
||||||
|
.unwrap_or_else(|e| panic!("round {round} submit {f} failed: {e:#}"));
|
||||||
|
}
|
||||||
|
drop(enc); // teardown with 4 in-flight encodes
|
||||||
|
}
|
||||||
|
let mut enc = open_h265();
|
||||||
|
let frame = nv12_frame(W, H, 0);
|
||||||
|
enc.submit_indexed(&frame, 0)
|
||||||
|
.expect("reopen after dirty teardowns");
|
||||||
|
while enc.poll().expect("poll").is_some() {}
|
||||||
|
println!("nvenc_cuda dirty-teardown: 3 dirty drops, reopen clean");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// ON-HARDWARE: the session-open failure path end to end — exhaust the driver's concurrent-
|
||||||
|
/// session cap with raw opens, assert a real encoder open fails with the actionable error
|
||||||
|
/// (and fires the one-shot self-diagnosis), then free the slots and assert the SAME encoder
|
||||||
|
/// rebuilds in place and produces an AU. This is the transient the session loop's rebuild
|
||||||
|
/// backoff is sized to outlive; on the RTX 5070 Ti (driver 610.43.03) the cap is 12 sessions
|
||||||
|
/// and the failure status is `NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY`.
|
||||||
|
#[test]
|
||||||
|
#[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box (.21)"]
|
||||||
|
fn nvenc_cuda_open_failure_diagnosis_and_recovery() {
|
||||||
|
const W: u32 = 1280;
|
||||||
|
const H: u32 = 720;
|
||||||
|
crate::zerocopy::cuda::make_current().expect("shared CUDA context current");
|
||||||
|
try_api().expect("nvenc api");
|
||||||
|
let shared = cuda::context().expect("shared ctx");
|
||||||
|
|
||||||
|
let open_raw = |device: *mut c_void| -> (nv::NVENCSTATUS, *mut c_void) {
|
||||||
|
let mut params = nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS {
|
||||||
|
version: nv::NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER,
|
||||||
|
deviceType: nv::NV_ENC_DEVICE_TYPE::NV_ENC_DEVICE_TYPE_CUDA,
|
||||||
|
device,
|
||||||
|
apiVersion: nv::NVENCAPI_VERSION,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
let mut enc: *mut c_void = ptr::null_mut();
|
||||||
|
// SAFETY: live params/out-param across the synchronous call; test-only.
|
||||||
|
let st = unsafe { (api().open_encode_session_ex)(&mut params, &mut enc) };
|
||||||
|
(st, enc)
|
||||||
|
};
|
||||||
|
|
||||||
|
// Exhaust the concurrent-session cap.
|
||||||
|
let mut held = Vec::new();
|
||||||
|
loop {
|
||||||
|
let (st, enc) = open_raw(shared);
|
||||||
|
if st != nv::NVENCSTATUS::NV_ENC_SUCCESS {
|
||||||
|
if !enc.is_null() {
|
||||||
|
// SAFETY: destroy the failed-open residue per the NVENC docs.
|
||||||
|
unsafe {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
held.push(enc);
|
||||||
|
}
|
||||||
|
assert!(!held.is_empty(), "expected a finite session cap");
|
||||||
|
|
||||||
|
// A real encoder open must now fail (lazy init → caps probe) with the actionable error.
|
||||||
|
let mut enc = open_h265();
|
||||||
|
let frame = nv12_frame(W, H, 0);
|
||||||
|
let err = enc
|
||||||
|
.submit_indexed(&frame, 0)
|
||||||
|
.expect_err("submit must fail while the cap is exhausted");
|
||||||
|
println!("at-cap error (self-diagnosis logged alongside): {err:#}");
|
||||||
|
|
||||||
|
// The transient clears (slots freed) → the SAME encoder rebuilds in place and encodes.
|
||||||
|
for e in held {
|
||||||
|
// SAFETY: e came from a successful raw open above; destroyed exactly once.
|
||||||
|
unsafe {
|
||||||
|
let _ = (api().destroy_encoder)(e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(enc.reset(), "in-place reset must be available");
|
||||||
|
let frame = nv12_frame(W, H, 1);
|
||||||
|
enc.submit_indexed(&frame, 1)
|
||||||
|
.expect("rebuild after the transient cleared");
|
||||||
|
let mut got = false;
|
||||||
|
while enc.poll().expect("poll").is_some() {
|
||||||
|
got = true;
|
||||||
|
}
|
||||||
|
assert!(got, "recovered encoder must produce an AU");
|
||||||
|
println!("nvenc_cuda open-failure recovery: cap hit → diagnosed → recovered in place");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -589,7 +589,16 @@ impl NvencD3d11Encoder {
|
|||||||
for &bs in &self.bitstreams {
|
for &bs in &self.bitstreams {
|
||||||
let _ = (api().destroy_bitstream_buffer)(self.encoder, bs);
|
let _ = (api().destroy_bitstream_buffer)(self.encoder, bs);
|
||||||
}
|
}
|
||||||
let _ = (api().destroy_encoder)(self.encoder);
|
// A destroy failure means the driver may still hold this session's slot (the concurrent-
|
||||||
|
// session cap is per process and only a restart clears a leak) — make it visible instead
|
||||||
|
// of silently discarding the status.
|
||||||
|
if let Err(e) = (api().destroy_encoder)(self.encoder).nv_ok() {
|
||||||
|
tracing::warn!(
|
||||||
|
status = ?e,
|
||||||
|
"NVENC destroy_encoder failed at teardown — the driver may have leaked this \
|
||||||
|
session's slot toward the concurrent-session cap"
|
||||||
|
);
|
||||||
|
}
|
||||||
// Return this session's units to the budget (see LIVE_SESSION_UNITS).
|
// Return this session's units to the budget (see LIVE_SESSION_UNITS).
|
||||||
LIVE_SESSION_UNITS.fetch_sub(self.session_units, std::sync::atomic::Ordering::Relaxed);
|
LIVE_SESSION_UNITS.fetch_sub(self.session_units, std::sync::atomic::Ordering::Relaxed);
|
||||||
self.session_units = 0;
|
self.session_units = 0;
|
||||||
@@ -637,9 +646,19 @@ impl NvencD3d11Encoder {
|
|||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
let mut enc: *mut c_void = ptr::null_mut();
|
let mut enc: *mut c_void = ptr::null_mut();
|
||||||
(api().open_encode_session_ex)(&mut params, &mut enc)
|
if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
|
||||||
.nv_ok()
|
// The NVENC docs require NvEncDestroyEncoder even after a FAILED open (the driver may
|
||||||
.map_err(|e| nvenc_status::call_err("open_encode_session_ex (caps probe)", e))?;
|
// have allocated the session slot before erroring) — without it, every failed open in
|
||||||
|
// a retry loop leaks a slot toward the concurrent-session cap, turning a transient
|
||||||
|
// failure into permanent exhaustion that only a host restart clears.
|
||||||
|
if !enc.is_null() {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
|
return Err(nvenc_status::call_err(
|
||||||
|
"open_encode_session_ex (caps probe)",
|
||||||
|
e,
|
||||||
|
));
|
||||||
|
}
|
||||||
let wmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_WIDTH_MAX);
|
let wmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_WIDTH_MAX);
|
||||||
let hmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_HEIGHT_MAX);
|
let hmax = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_HEIGHT_MAX);
|
||||||
let ten_bit = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_10BIT_ENCODE);
|
let ten_bit = self.get_cap(enc, nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_10BIT_ENCODE);
|
||||||
@@ -948,9 +967,14 @@ impl NvencD3d11Encoder {
|
|||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
let mut enc: *mut c_void = ptr::null_mut();
|
let mut enc: *mut c_void = ptr::null_mut();
|
||||||
(api().open_encode_session_ex)(&mut params, &mut enc)
|
if let Err(e) = (api().open_encode_session_ex)(&mut params, &mut enc).nv_ok() {
|
||||||
.nv_ok()
|
// Destroy-on-failed-open, as in `query_caps`: a failed open may still hold a session
|
||||||
.map_err(|e| nvenc_status::call_err("open_encode_session_ex", e))?;
|
// slot that must be released.
|
||||||
|
if !enc.is_null() {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
|
return Err(nvenc_status::call_err("open_encode_session_ex", e));
|
||||||
|
}
|
||||||
|
|
||||||
let mut cfg = match self.build_config(enc, bitrate) {
|
let mut cfg = match self.build_config(enc, bitrate) {
|
||||||
Ok(cfg) => cfg,
|
Ok(cfg) => cfg,
|
||||||
@@ -1764,8 +1788,9 @@ pub fn probe_can_encode_444(codec: Codec) -> bool {
|
|||||||
// `EnumAdapterByLuid` return owned COM objects or err (→ default-adapter fallback).
|
// `EnumAdapterByLuid` return owned COM objects or err (→ default-adapter fallback).
|
||||||
// `D3D11CreateDevice` (explicit adapter + UNKNOWN driver type, or NULL adapter + HARDWARE)
|
// `D3D11CreateDevice` (explicit adapter + UNKNOWN driver type, or NULL adapter + HARDWARE)
|
||||||
// fills `device` or returns Err (→ false). `open_encode_session_ex` opens an NVENC session
|
// fills `device` or returns Err (→ false). `open_encode_session_ex` opens an NVENC session
|
||||||
// against that device's raw pointer (valid while `device` is held) or errors (→ false, tearing
|
// against that device's raw pointer (valid while `device` is held) or errors (→ false, after
|
||||||
// nothing down). `get_encode_caps` reads one scalar cap into `val` via the loaded API table.
|
// destroying any residue session the failed open left — the docs require it).
|
||||||
|
// `get_encode_caps` reads one scalar cap into `val` via the loaded API table.
|
||||||
// `destroy_encoder` frees the session exactly once; `device`/its context drop with the COM
|
// `destroy_encoder` frees the session exactly once; `device`/its context drop with the COM
|
||||||
// wrappers. No handle escapes this call and nothing runs concurrently.
|
// wrappers. No handle escapes this call and nothing runs concurrently.
|
||||||
unsafe {
|
unsafe {
|
||||||
@@ -1818,6 +1843,10 @@ pub fn probe_can_encode_444(codec: Codec) -> bool {
|
|||||||
.nv_ok()
|
.nv_ok()
|
||||||
.is_err()
|
.is_err()
|
||||||
{
|
{
|
||||||
|
// Destroy-on-failed-open: a failed open may still hold a session slot.
|
||||||
|
if !enc.is_null() {
|
||||||
|
let _ = (api().destroy_encoder)(enc);
|
||||||
|
}
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
let mut param = nv::NV_ENC_CAPS_PARAM {
|
let mut param = nv::NV_ENC_CAPS_PARAM {
|
||||||
|
|||||||
@@ -120,6 +120,7 @@ struct CudaApi {
|
|||||||
cuInit: unsafe extern "C" fn(c_uint) -> CUresult,
|
cuInit: unsafe extern "C" fn(c_uint) -> CUresult,
|
||||||
cuDeviceGet: unsafe extern "C" fn(*mut CUdevice, c_int) -> CUresult,
|
cuDeviceGet: unsafe extern "C" fn(*mut CUdevice, c_int) -> CUresult,
|
||||||
cuCtxCreate_v2: unsafe extern "C" fn(*mut CUcontext, c_uint, CUdevice) -> CUresult,
|
cuCtxCreate_v2: unsafe extern "C" fn(*mut CUcontext, c_uint, CUdevice) -> CUresult,
|
||||||
|
cuCtxDestroy_v2: unsafe extern "C" fn(CUcontext) -> CUresult,
|
||||||
cuCtxSetCurrent: unsafe extern "C" fn(CUcontext) -> CUresult,
|
cuCtxSetCurrent: unsafe extern "C" fn(CUcontext) -> CUresult,
|
||||||
cuMemAllocPitch_v2:
|
cuMemAllocPitch_v2:
|
||||||
unsafe extern "C" fn(*mut CUdeviceptr, *mut usize, usize, usize, c_uint) -> CUresult,
|
unsafe extern "C" fn(*mut CUdeviceptr, *mut usize, usize, usize, c_uint) -> CUresult,
|
||||||
@@ -214,6 +215,7 @@ fn cuda_api() -> Option<&'static CudaApi> {
|
|||||||
cuInit: *lib.get(b"cuInit\0").ok()?,
|
cuInit: *lib.get(b"cuInit\0").ok()?,
|
||||||
cuDeviceGet: *lib.get(b"cuDeviceGet\0").ok()?,
|
cuDeviceGet: *lib.get(b"cuDeviceGet\0").ok()?,
|
||||||
cuCtxCreate_v2: *lib.get(b"cuCtxCreate_v2\0").ok()?,
|
cuCtxCreate_v2: *lib.get(b"cuCtxCreate_v2\0").ok()?,
|
||||||
|
cuCtxDestroy_v2: *lib.get(b"cuCtxDestroy_v2\0").ok()?,
|
||||||
cuCtxSetCurrent: *lib.get(b"cuCtxSetCurrent\0").ok()?,
|
cuCtxSetCurrent: *lib.get(b"cuCtxSetCurrent\0").ok()?,
|
||||||
cuMemAllocPitch_v2: *lib.get(b"cuMemAllocPitch_v2\0").ok()?,
|
cuMemAllocPitch_v2: *lib.get(b"cuMemAllocPitch_v2\0").ok()?,
|
||||||
cuMemFree_v2: *lib.get(b"cuMemFree_v2\0").ok()?,
|
cuMemFree_v2: *lib.get(b"cuMemFree_v2\0").ok()?,
|
||||||
@@ -275,6 +277,12 @@ unsafe fn cuCtxCreate_v2(pctx: *mut CUcontext, flags: c_uint, dev: CUdevice) ->
|
|||||||
None => CU_ERROR_NOT_LOADED,
|
None => CU_ERROR_NOT_LOADED,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
unsafe fn cuCtxDestroy_v2(ctx: CUcontext) -> CUresult {
|
||||||
|
match cuda_api() {
|
||||||
|
Some(a) => (a.cuCtxDestroy_v2)(ctx),
|
||||||
|
None => CU_ERROR_NOT_LOADED,
|
||||||
|
}
|
||||||
|
}
|
||||||
unsafe fn cuCtxSetCurrent(ctx: CUcontext) -> CUresult {
|
unsafe fn cuCtxSetCurrent(ctx: CUcontext) -> CUresult {
|
||||||
match cuda_api() {
|
match cuda_api() {
|
||||||
Some(a) => (a.cuCtxSetCurrent)(ctx),
|
Some(a) => (a.cuCtxSetCurrent)(ctx),
|
||||||
@@ -610,6 +618,37 @@ pub fn make_current() -> Result<()> {
|
|||||||
unsafe { ck(cuCtxSetCurrent(ctx), "cuCtxSetCurrent") }
|
unsafe { ck(cuCtxSetCurrent(ctx), "cuCtxSetCurrent") }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// DIAGNOSTIC-ONLY: create a fresh dedicated context on device 0, run `probe` with it, destroy it,
|
||||||
|
/// and restore the shared context as current. Used by the NVENC backend's session-open
|
||||||
|
/// self-diagnosis to split "this process's shared context is in a bad state" (probe succeeds on
|
||||||
|
/// the fresh context) from a driver-level condition like version skew or session exhaustion
|
||||||
|
/// (probe fails there too). Never used on a hot path.
|
||||||
|
pub fn with_fresh_context<R>(probe: impl FnOnce(CUcontext) -> R) -> Result<R> {
|
||||||
|
if cuda_api().is_none() {
|
||||||
|
bail!("libcuda.so.1 not available");
|
||||||
|
}
|
||||||
|
// SAFETY: the driver table is present (checked above). `cuInit(0)` is idempotent. `&mut dev`/
|
||||||
|
// `&mut ctx` are live, distinct stack out-params for their synchronous calls. On success `ctx`
|
||||||
|
// is a valid dedicated context, destroyed exactly once below; the shared context is restored
|
||||||
|
// as current afterwards (creation left the fresh one current on this thread).
|
||||||
|
unsafe {
|
||||||
|
ck(cuInit(0), "cuInit")?;
|
||||||
|
let mut dev: CUdevice = 0;
|
||||||
|
ck(cuDeviceGet(&mut dev, 0), "cuDeviceGet")?;
|
||||||
|
let mut ctx: CUcontext = std::ptr::null_mut();
|
||||||
|
ck(
|
||||||
|
cuCtxCreate_v2(&mut ctx, CU_CTX_SCHED_BLOCKING_SYNC, dev),
|
||||||
|
"cuCtxCreate_v2 (diagnostic)",
|
||||||
|
)?;
|
||||||
|
let r = probe(ctx);
|
||||||
|
let _ = cuCtxDestroy_v2(ctx);
|
||||||
|
if let Some(c) = CONTEXT.get() {
|
||||||
|
let _ = cuCtxSetCurrent(c.0);
|
||||||
|
}
|
||||||
|
Ok(r)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
thread_local! {
|
thread_local! {
|
||||||
/// Per-thread copy stream. `None` until first use; `Some(null)` means "creation failed, use the
|
/// Per-thread copy stream. `None` until first use; `Some(null)` means "creation failed, use the
|
||||||
/// default (NULL) stream". Per-thread (not shared) so each worker's `cuStreamSynchronize` waits
|
/// default (NULL) stream". Per-thread (not shared) so each worker's `cuStreamSynchronize` waits
|
||||||
|
|||||||
@@ -4935,10 +4935,19 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
|
|||||||
max = MAX_ENCODER_RESETS,
|
max = MAX_ENCODER_RESETS,
|
||||||
"encoder submit failed — encoder rebuilt in place, forcing an IDR");
|
"encoder submit failed — encoder rebuilt in place, forcing an IDR");
|
||||||
last_au_at = std::time::Instant::now();
|
last_au_at = std::time::Instant::now();
|
||||||
// Re-pace from the rebuild and retry this frame next tick (gives the fresh encoder
|
// Back off exponentially between rebuild attempts (100 ms → 1.6 s, ~3 s total across
|
||||||
// one frame period to come up instead of hammering it in a hot loop).
|
// the reset budget). One frame period is NOT enough: a 2026-07 field report showed all
|
||||||
next = std::time::Instant::now() + interval;
|
// 5 resets burning within 40 ms at 120 Hz against a driver-side condition (NVENC
|
||||||
std::thread::sleep(interval);
|
// session open failing after a codec switch) that no 8 ms retry could outlive — any
|
||||||
|
// transient like the previous session's deferred driver teardown needs real time. A
|
||||||
|
// genuinely dead encoder now costs ~3 s before the session ends with the terminal
|
||||||
|
// error, which the client's stall UI already covers.
|
||||||
|
let backoff = std::cmp::max(
|
||||||
|
interval,
|
||||||
|
std::time::Duration::from_millis(100u64 << (encoder_resets - 1).min(4)),
|
||||||
|
);
|
||||||
|
next = std::time::Instant::now() + backoff;
|
||||||
|
std::thread::sleep(backoff);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let submit_us = if measure {
|
let submit_us = if measure {
|
||||||
|
|||||||
Reference in New Issue
Block a user