fix(encode/nvenc): cache the level ceiling, expose the applied bitrate, force 2-way split at 4K120

Three encode-time fixes from the 4K120 field analysis (sessions pinned ~107 of
120 fps, 14-15 ms reported encode):

- Ceiling truth: the codec-level bitrate clamp (binary search at session open)
  now records its result in a process-lifetime advisory cache keyed by
  GPU/config, so an ABR overshoot opens — or in-place reconfigures — straight
  AT the ceiling instead of re-running the ~6-open search (and a full rebuild
  + IDR) on every overshoot. New `Encoder::applied_bitrate_bps()` exposes the
  post-clamp rate so the session loop can stop pacing/acking a phantom
  requested rate (consumed host-side in a follow-up).

- Clamp-search hygiene: only NVENC parameter/caps rejections steer the search
  now (`NvCallError` keeps the raw status downcastable); a transient failure
  (busy engine, session limit, OOM, driver skew) propagates instead of
  shrinking into — and now caching — a bogus ceiling. The floor fallback also
  records the split mode it actually opened with, so a later reconfigure
  re-presents the real session params.

- Split-frame selection: one shared `resolve_split_mode` replaces the two
  byte-identical direct-SDK copies; the force-2-way threshold moves to
  `SPLIT_FORCE_PIXEL_RATE` (950 Mpix/s, shared with the libav path) because
  4K120 = 995,328,000 px/s missed the old `> 1e9` gate by 0.47% and stayed on
  AUTO — which never engages at 2160 px height, leaving the second NVENC
  engine idle in exactly the mode the threshold existed for. The Linux
  session-ready info! line now carries the final split mode (journals are
  INFO+; this was undiagnosable from user logs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 01:44:54 +02:00
co-authored by Claude Fable 5
parent 42e5f5ad1e
commit c1d54b835b
10 changed files with 498 additions and 111 deletions
+22 -1
View File
@@ -112,7 +112,7 @@ impl AuChunk {
} }
/// Codec selection negotiated with the client. /// Codec selection negotiated with the client.
#[derive(Clone, Copy, Debug, PartialEq, Eq)] #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum Codec { pub enum Codec {
H264, H264,
H265, H265,
@@ -399,6 +399,16 @@ pub trait Encoder: Send {
fn reconfigure_bitrate(&mut self, _bps: u64) -> bool { fn reconfigure_bitrate(&mut self, _bps: u64) -> bool {
false false
} }
/// The bitrate (bps) the encoder is ACTUALLY running at (or will open at, for a lazily-opened
/// backend) — the encoder-side truth after any internal clamp, e.g. the direct-NVENC
/// codec-level ceiling search. The session loop reads this after every open/reconfigure and
/// stores IT, not the requested rate, as the live bitrate — so the send pacer, the console
/// and the client controller's ack all track what the ASIC really targets (a controller fed
/// the requested rate keeps climbing from a phantom base, §ABR overdrive). `None` (the
/// default) = the backend doesn't track an applied rate; the caller keeps the requested one.
fn applied_bitrate_bps(&self) -> Option<u64> {
None
}
/// Wire-chunk the encoder's AUs at the session's shard payload size (the PyroWave /// Wire-chunk the encoder's AUs at the session's shard payload size (the PyroWave
/// datagram-aligned mode, plan §4.4): every `shard_payload` window of the emitted AU /// datagram-aligned mode, plan §4.4): every `shard_payload` window of the emitted AU
/// starts a fresh self-delimiting codec packet, zero-padded to the window — so a lost /// starts a fresh self-delimiting codec packet, zero-padded to the window — so a lost
@@ -452,6 +462,17 @@ impl Codec {
} }
} }
/// Pixel rate (luma samples/s) at or above which NVENC split-frame encoding is FORCED 2-way —
/// one number shared by the direct-SDK selector (`nvenc_core::resolve_split_mode`) and the libav
/// `split_encode_mode` option author (`linux::NvencEncoder`), so the two paths can never disagree
/// about which modes split. A single NVENC engine tops out ~1 Gpix/s on HEVC, and AUTO doesn't
/// engage below ~2112 px height, so the sessions that need the second engine must be forced. Set
/// BELOW 1 Gpix/s deliberately: 4K120 — the mode this threshold exists for — is 3840×2160×120 =
/// 995,328,000, which a `> 1_000_000_000` gate missed by 0.47% and left on AUTO (pinned ~107 fps
/// on a 4090). 950 M keeps margin for fractional refresh rates while leaving 1440p240 (884.7 M,
/// comfortably single-engine) on AUTO.
pub const SPLIT_FORCE_PIXEL_RATE: u64 = 950_000_000;
/// `PUNKTFUNK_VBV_FRAMES` — HRD/VBV size in frame intervals (default 1.0, the strict low-latency /// `PUNKTFUNK_VBV_FRAMES` — HRD/VBV size in frame intervals (default 1.0, the strict low-latency
/// shape every backend ships: each frame must fit its rate share, keeping frame sizes uniform for /// shape every backend ships: each frame must fit its rate share, keeping frame sizes uniform for
/// the pacer). The AMF/VAAPI/QSV paths parse the same variable locally; this helper brings the /// the pacer). The AMF/VAAPI/QSV paths parse the same variable locally; this helper brings the
+8 -4
View File
@@ -439,15 +439,19 @@ impl NvencEncoder {
} }
// Split-frame encode across both NVENC engines (GB203 has 2) when the pixel rate exceeds // Split-frame encode across both NVENC engines (GB203 has 2) when the pixel rate exceeds
// a single engine's HEVC capacity (~1 Gpix/s); e.g. 5120x1440@240 = 1.77 Gpix/s needs it, // a single engine's HEVC capacity; e.g. 5120x1440@240 = 1.77 Gpix/s needs it, @120
// @120 = 0.88 Gpix/s does not. HEVC/AV1 only (not H.264). AUTO won't engage below ~2112px // (0.88 Gpix/s) does not. HEVC/AV1 only (not H.264). AUTO won't engage below ~2112px
// height, so we force `2`; below the threshold we leave it AUTO (split costs ~2% BD-rate). // height, so we force `2`; below the threshold we leave it AUTO (split costs ~2% BD-rate).
// Output is standard HEVC — transparent to the client. Override with PUNKTFUNK_SPLIT_ENCODE. // Threshold shared with the direct-SDK selector ([`super::SPLIT_FORCE_PIXEL_RATE`] — set
// so 4K120 = 995.3 Mpix/s forces, which `> 1e9` famously missed by 0.47%). Output is
// standard HEVC — transparent to the client. Override with PUNKTFUNK_SPLIT_ENCODE.
let pix_rate = width as u64 * height as u64 * fps as u64; let pix_rate = width as u64 * height as u64 * fps as u64;
let split = std::env::var("PUNKTFUNK_SPLIT_ENCODE").ok(); let split = std::env::var("PUNKTFUNK_SPLIT_ENCODE").ok();
match split.as_deref() { match split.as_deref() {
Some(mode) => opts.set("split_encode_mode", mode), Some(mode) => opts.set("split_encode_mode", mode),
None if matches!(codec, Codec::H265 | Codec::Av1) && pix_rate > 1_000_000_000 => { None if matches!(codec, Codec::H265 | Codec::Av1)
&& pix_rate >= super::SPLIT_FORCE_PIXEL_RATE =>
{
opts.set("split_encode_mode", "2"); opts.set("split_encode_mode", "2");
tracing::info!( tracing::info!(
pix_rate, pix_rate,
+100 -40
View File
@@ -61,8 +61,9 @@
#![deny(clippy::undocumented_unsafe_blocks)] #![deny(clippy::undocumented_unsafe_blocks)]
use super::nvenc_core::{ use super::nvenc_core::{
apply_low_latency_config, build_init_params, codec_guid, resolve_slices, resolve_subframe, apply_low_latency_config, build_init_params, cached_ceiling, codec_guid, resolve_slices,
LowLatencyConfig, NvStatusExt, RFI_DPB, resolve_split_mode, resolve_subframe, store_ceiling, CeilingKey, LowLatencyConfig, NvStatusExt,
RFI_DPB,
}; };
use super::nvenc_status; use super::nvenc_status;
use super::{AuChunk, ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps}; use super::{AuChunk, ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps};
@@ -1000,6 +1001,23 @@ impl NvencCudaEncoder {
Ok(cfg) Ok(cfg)
} }
/// This session config's identity in the process-lifetime bitrate-ceiling cache
/// (`nvenc_core::{cached_ceiling, store_ceiling}`). GPU identity is the process-global shared
/// `CUcontext` pointer — one context per process, stable for its lifetime; only valid once
/// `cu_ctx` is bound (`init_session` start), which every caller is downstream of.
fn ceiling_key(&self, split_mode: u32) -> CeilingKey {
CeilingKey {
gpu: self.cu_ctx as u64,
codec: self.codec,
width: self.width,
height: self.height,
fps: self.fps,
bit_depth: self.bit_depth,
chroma_444: self.chroma_444,
split_mode,
}
}
/// Open + configure + initialize ONE NVENC CUDA session at `bitrate` (bps) and `split_mode`. /// Open + configure + initialize ONE NVENC CUDA session at `bitrate` (bps) and `split_mode`.
/// Returns the session handle, or destroys it and returns the error. /// Returns the session handle, or destroys it and returns the error.
unsafe fn try_open_session(&self, bitrate: u64, split_mode: u32) -> Result<*mut c_void> { unsafe fn try_open_session(&self, bitrate: u64, split_mode: u32) -> Result<*mut c_void> {
@@ -1074,58 +1092,71 @@ impl NvencCudaEncoder {
} }
const FLOOR_BPS: u64 = 10_000_000; const FLOOR_BPS: u64 = 10_000_000;
let requested_bps = self.bitrate_bps; let requested_bps = self.bitrate_bps;
// 2-way NVENC split-frame encoding (Ada dual-NVENC) above ~1 Gpix/s; env override // 2-way NVENC split-frame encoding (Ada dual-NVENC) — shared selector, see
// PUNKTFUNK_SPLIT_ENCODE = 0/disable | 1/auto | 2 | 3. HEVC/AV1 only. // [`resolve_split_mode`] for the precedence (env override / 10-bit / pixel rate).
let pixel_rate = self.width as u64 * self.height as u64 * self.fps.max(1) as u64; let pixel_rate = self.width as u64 * self.height as u64 * self.fps.max(1) as u64;
let mut split_mode: u32 = match std::env::var("PUNKTFUNK_SPLIT_ENCODE").ok().as_deref() let split_mode: u32 = resolve_split_mode(self.bit_depth, pixel_rate);
{
Some("0") | Some("disable") => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32
}
Some("1") | Some("auto") => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_AUTO_FORCED_MODE as u32
}
Some("3") => nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_THREE_FORCED_MODE as u32,
Some("2") => nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_TWO_FORCED_MODE as u32,
_ if self.bit_depth >= 10 => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32
}
_ if pixel_rate > 1_000_000_000 => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_TWO_FORCED_MODE as u32
}
_ => nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_AUTO_MODE as u32,
};
const CLAMP_TOL_BPS: u64 = 20_000_000; const CLAMP_TOL_BPS: u64 = 20_000_000;
let mut probe = self.try_open_session(requested_bps, split_mode); // Ceiling cache (process lifetime, `nvenc_core`): a prior clamp search already found
// Disambiguate a forced-split rejection from a bitrate-cap rejection. // this config's max accepted rate — open straight AT the ceiling instead of paying
// the ~6-open binary search (and its session churn) on every ABR overshoot.
let mut target_bps = requested_bps;
if let Some(ceiling) = cached_ceiling(&self.ceiling_key(split_mode)) {
if requested_bps > ceiling {
tracing::info!(
requested_mbps = requested_bps / 1_000_000,
ceiling_mbps = ceiling / 1_000_000,
"NVENC (Linux): requested bitrate above the cached codec-level ceiling — \
opening at the ceiling"
);
target_bps = ceiling;
}
}
let mut probe = self.try_open_session(target_bps, split_mode);
// The cache is advisory: a stale entry (driver change, identity collision) must not
// wedge the open — retry the requested rate and let the search below rediscover.
if probe.is_err() && target_bps < requested_bps {
target_bps = requested_bps;
probe = self.try_open_session(requested_bps, split_mode);
}
// Disambiguate a forced-split rejection from a bitrate-cap rejection. `used_split`
// tracks the mode sessions ACTUALLY open with from here on — it feeds
// `self.split_mode` (a reconfigure must re-present it) and the ceiling-cache key.
let mut used_split = split_mode;
let split_on = let split_on =
split_mode != nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32; split_mode != nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32;
if probe.is_err() && split_on { if probe.is_err() && split_on {
let no_split = nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32; let no_split = nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32;
if let Ok(e) = self.try_open_session(requested_bps, no_split) { if let Ok(e) = self.try_open_session(target_bps, no_split) {
tracing::warn!( tracing::warn!(
"NVENC (Linux): split-encode rejected by codec/config — disabled" "NVENC (Linux): split-encode rejected by codec/config — disabled"
); );
split_mode = no_split; used_split = no_split;
probe = Ok(e); probe = Ok(e);
} }
} }
let enc = match probe { let enc = match probe {
Ok(enc) => { Ok(enc) => {
self.bitrate_bps = requested_bps; self.bitrate_bps = target_bps;
enc enc
} }
// Only a parameter/caps rejection means "the bitrate is above the codec-level
// ceiling". A transient failure (busy engine, session limit, OOM, device loss,
// version skew) must propagate — a search steered by it would discover, and
// cache, a bogus ceiling.
Err(e) if !nvenc_status::is_param_rejection(&e) => return Err(e),
Err(_) => { Err(_) => {
// Requested bitrate exceeds the codec-level ceiling — binary-search the max accepted. // Requested bitrate exceeds the codec-level ceiling — binary-search the max accepted.
let mut lo = FLOOR_BPS; let mut lo = FLOOR_BPS;
let mut hi = requested_bps; let mut hi = target_bps;
let mut best: *mut c_void = ptr::null_mut(); let mut best: *mut c_void = ptr::null_mut();
let mut best_bps = 0u64; let mut best_bps = 0u64;
while hi > lo + CLAMP_TOL_BPS { while hi > lo + CLAMP_TOL_BPS {
let mid = lo + (hi - lo) / 2; let mid = lo + (hi - lo) / 2;
match self.try_open_session(mid, split_mode) { match self.try_open_session(mid, used_split) {
Ok(e) => { Ok(e) => {
if !best.is_null() { if !best.is_null() {
let _ = (api().destroy_encoder)(best); let _ = (api().destroy_encoder)(best);
@@ -1134,18 +1165,30 @@ impl NvencCudaEncoder {
best_bps = mid; best_bps = mid;
lo = mid; lo = mid;
} }
Err(_) => hi = mid, Err(e) if nvenc_status::is_param_rejection(&e) => hi = mid,
Err(e) => {
// Environmental mid-search failure: don't let it shrink the
// search — release the partial result and propagate.
if !best.is_null() {
let _ = (api().destroy_encoder)(best);
}
return Err(e);
}
} }
} }
if best.is_null() { if best.is_null() {
let no_split = let no_split =
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32; nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32;
best = self best = match self.try_open_session(FLOOR_BPS, used_split) {
.try_open_session(FLOOR_BPS, split_mode) Ok(e) => e,
.or_else(|_| self.try_open_session(FLOOR_BPS, no_split)) Err(_) => {
.context( let e = self.try_open_session(FLOOR_BPS, no_split).context(
"NVENC initialize_encoder rejected even at the floor bitrate", "NVENC initialize_encoder rejected even at the floor bitrate",
)?; )?;
used_split = no_split;
e
}
};
best_bps = FLOOR_BPS; best_bps = FLOOR_BPS;
} }
tracing::warn!( tracing::warn!(
@@ -1153,15 +1196,13 @@ impl NvencCudaEncoder {
clamped_mbps = best_bps / 1_000_000, clamped_mbps = best_bps / 1_000_000,
"NVENC (Linux): requested bitrate above the GPU codec-level ceiling — clamped" "NVENC (Linux): requested bitrate above the GPU codec-level ceiling — clamped"
); );
store_ceiling(self.ceiling_key(used_split), best_bps);
self.bitrate_bps = best_bps; self.bitrate_bps = best_bps;
best best
} }
}; };
self.encoder = enc; self.encoder = enc;
// (Best effort: the floor fallback above may have succeeded split-disabled without self.split_mode = used_split;
// updating `split_mode` — a later reconfigure then presents the forced mode, NVENC
// rejects it, and the caller's rebuild fallback covers the mismatch.)
self.split_mode = split_mode;
// Output bitstream pool. // Output bitstream pool.
for _ in 0..POOL { for _ in 0..POOL {
@@ -1345,6 +1386,10 @@ impl NvencCudaEncoder {
mbps = self.bitrate_bps / 1_000_000, mbps = self.bitrate_bps / 1_000_000,
codec = ?self.codec_guid, codec = ?self.codec_guid,
fmt = ?self.buffer_fmt, fmt = ?self.buffer_fmt,
// The FINAL split mode (post any rejection fallback) at INFO — journals run
// INFO+, and "did 4K120 actually split across engines?" was undiagnosable from
// a user log without it (Windows only had a debug! at selection time).
split_mode = self.split_mode,
"NVENC CUDA session ready" "NVENC CUDA session ready"
); );
Ok(()) Ok(())
@@ -2061,6 +2106,15 @@ impl Encoder for NvencCudaEncoder {
self.bitrate_bps = bps; self.bitrate_bps = bps;
return true; return true;
} }
// Cached codec-level ceiling: clamp the target BEFORE the driver call, so a known
// overshoot retargets to the ceiling IN PLACE instead of bouncing off the driver into
// the caller's full-rebuild fallback (an IDR plus ~half a second of session churn per
// ABR overshoot on the pre-cache path). The caller reads the clamp back through
// [`Encoder::applied_bitrate_bps`].
let bps = match cached_ceiling(&self.ceiling_key(self.split_mode)) {
Some(ceiling) => bps.min(ceiling),
None => bps,
};
// SAFETY: `inited` ⟹ `self.encoder` is the live session and every call here runs on the // SAFETY: `inited` ⟹ `self.encoder` is the live session and every call here runs on the
// encode thread with no NVENC call in flight (the session loop calls this between // encode thread with no NVENC call in flight (the session loop calls this between
// submit/poll). `build_config` only queries the preset on that session; `cfg` outlives // submit/poll). `build_config` only queries the preset on that session; `cfg` outlives
@@ -2108,6 +2162,12 @@ impl Encoder for NvencCudaEncoder {
} }
} }
fn applied_bitrate_bps(&self) -> Option<u64> {
// `bitrate_bps` is the post-clamp truth: the open path's ceiling search and the
// reconfigure path's cache clamp both write what the session ACTUALLY targets.
Some(self.bitrate_bps)
}
fn flush(&mut self) -> Result<()> { fn flush(&mut self) -> Result<()> {
Ok(()) // P1/ULL + frameIntervalP=1: each submit yields its AU; no internal queue to drain. Ok(()) // P1/ULL + frameIntervalP=1: each submit yields its AU; no internal queue to drain.
} }
+149
View File
@@ -67,6 +67,155 @@ pub(super) fn resolve_subframe(default_on: bool) -> bool {
} }
} }
/// Resolved NVENC split-frame encode mode for a session — ONE selector shared by the Windows and
/// Linux direct-SDK backends (they had drifted into byte-identical duplicates, one of which
/// logged and one didn't). Precedence:
/// 1. `PUNKTFUNK_SPLIT_ENCODE` = `0`/`disable` | `1`/`auto` (AUTO_FORCED) | `2` | `3` — operator
/// override, always wins.
/// 2. 10-bit → DISABLE: 2-way split is measurably SLOWER on Ada for Main10 — at 5120×1440@240
/// forced-2 took 7.6 ms/frame (~131 fps) vs 2.8 ms (~357 fps) single-engine (the split/merge
/// overhead dominates), and a single engine handles 5K@240 Main10 well under budget. This was
/// the "broken animations in HDR" cap at ~131 fps.
/// 3. Pixel rate ≥ [`super::SPLIT_FORCE_PIXEL_RATE`] → force 2-way (AUTO never engages below
/// ~2112 px height, so 4K120 must be forced onto the second engine).
/// 4. Else AUTO (the ~2% BD-rate split cost isn't worth it at low pixel rates).
///
/// The caller still owns the rejection fallback (retry split-disabled) — a codec/config that
/// rejects the chosen mode downgrades at open, not here.
pub(super) fn resolve_split_mode(bit_depth: u8, pixel_rate: u64) -> u32 {
use nv::NV_ENC_SPLIT_ENCODE_MODE as M;
let mode = match std::env::var("PUNKTFUNK_SPLIT_ENCODE").ok().as_deref() {
Some("0") | Some("disable") => M::NV_ENC_SPLIT_DISABLE_MODE as u32,
Some("1") | Some("auto") => M::NV_ENC_SPLIT_AUTO_FORCED_MODE as u32,
Some("3") => M::NV_ENC_SPLIT_THREE_FORCED_MODE as u32,
Some("2") => M::NV_ENC_SPLIT_TWO_FORCED_MODE as u32,
_ if bit_depth >= 10 => M::NV_ENC_SPLIT_DISABLE_MODE as u32,
_ if pixel_rate >= super::SPLIT_FORCE_PIXEL_RATE => M::NV_ENC_SPLIT_TWO_FORCED_MODE as u32,
_ => M::NV_ENC_SPLIT_AUTO_MODE as u32,
};
tracing::debug!(
split_mode = mode,
bit_depth,
pixel_rate,
"NVENC split-encode mode selected"
);
mode
}
/// One session config's identity for the process-lifetime bitrate-ceiling cache
/// ([`cached_ceiling`]/[`store_ceiling`]). Everything the driver's codec-level validation keys
/// off: the GPU (different NVENC generations have different level ceilings), dims/fps (the luma
/// rate selects the level), depth/chroma (they select the profile) and the split mode the
/// sessions ACTUALLY opened with (a split session budgets per engine).
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
pub(super) struct CeilingKey {
/// GPU identity — Linux: the process-global shared `CUcontext` pointer; Windows: the render
/// adapter LUID (0 when unresolved). Best effort: the cache is advisory (see
/// [`cached_ceiling`]), so a colliding identity costs one failed open + re-search, never a
/// wrong session.
pub gpu: u64,
pub codec: Codec,
pub width: u32,
pub height: u32,
pub fps: u32,
pub bit_depth: u8,
pub chroma_444: bool,
pub split_mode: u32,
}
fn ceilings() -> &'static std::sync::Mutex<std::collections::HashMap<CeilingKey, u64>> {
static CEILINGS: std::sync::OnceLock<
std::sync::Mutex<std::collections::HashMap<CeilingKey, u64>>,
> = std::sync::OnceLock::new();
CEILINGS.get_or_init(Default::default)
}
/// The codec-level bitrate ceiling (bps) a previous clamp search discovered for `key` this
/// process lifetime, if any. ADVISORY: the consumer must treat a failed open at the cached value
/// as a stale entry (fall back to the full search, which rewrites it via [`store_ceiling`]) —
/// that self-healing is what lets the key's GPU identity be best-effort. What this buys: an ABR
/// overshoot on a config whose ceiling is already known opens (or in-place reconfigures) straight
/// AT the ceiling instead of re-running the ~6-open binary search and its ~half-second of session
/// churn per rebuild.
pub(super) fn cached_ceiling(key: &CeilingKey) -> Option<u64> {
ceilings().lock().unwrap().get(key).copied()
}
/// Record the clamp search's discovered max accepted bitrate (bps) for `key`.
pub(super) fn store_ceiling(key: CeilingKey, bps: u64) {
ceilings().lock().unwrap().insert(key, bps);
}
#[cfg(test)]
mod tests {
use super::*;
use nv::NV_ENC_SPLIT_ENCODE_MODE as M;
// These assume PUNKTFUNK_SPLIT_ENCODE is unset (CI); an operator override deliberately wins.
#[test]
fn split_forces_two_way_at_4k120() {
// The regression this threshold constant exists for: 3840×2160×120 = 995,328,000 sat
// 0.47% under the old `> 1_000_000_000` gate and stayed AUTO — pinned ~107 fps on a
// 4090 because AUTO never engages at 2160 px height.
let four_k_120 = 3840u64 * 2160 * 120;
assert_eq!(
resolve_split_mode(8, four_k_120),
M::NV_ENC_SPLIT_TWO_FORCED_MODE as u32
);
}
#[test]
fn split_leaves_1440p240_auto() {
// 884.7 Mpix/s is comfortably single-engine — the threshold move must not drag it in.
let qhd_240 = 2560u64 * 1440 * 240;
assert_eq!(
resolve_split_mode(8, qhd_240),
M::NV_ENC_SPLIT_AUTO_MODE as u32
);
}
#[test]
fn split_disabled_for_10bit_even_at_high_pixel_rate() {
// The measured Main10 rule: split/merge overhead dominates 10-bit on Ada (7.6 ms forced-2
// vs 2.8 ms single-engine at 5K240) — 10-bit precedes the pixel-rate arm.
let five_k_240 = 5120u64 * 1440 * 240;
assert_eq!(
resolve_split_mode(10, five_k_240),
M::NV_ENC_SPLIT_DISABLE_MODE as u32
);
}
#[test]
fn ceiling_cache_round_trips_and_keys_precisely() {
let key = CeilingKey {
gpu: 0xB0B0,
codec: Codec::H265,
width: 3840,
height: 2160,
fps: 120,
bit_depth: 8,
chroma_444: false,
split_mode: M::NV_ENC_SPLIT_TWO_FORCED_MODE as u32,
};
assert_eq!(cached_ceiling(&key), None);
store_ceiling(key, 794_000_000);
assert_eq!(cached_ceiling(&key), Some(794_000_000));
// Any config-identity change is a different ceiling — a miss, never a wrong clamp.
assert_eq!(cached_ceiling(&CeilingKey { fps: 60, ..key }), None);
assert_eq!(
cached_ceiling(&CeilingKey {
split_mode: M::NV_ENC_SPLIT_DISABLE_MODE as u32,
..key
}),
None
);
// A re-search overwrites (the advisory-cache stale-entry path).
store_ceiling(key, 620_000_000);
assert_eq!(cached_ceiling(&key), Some(620_000_000));
}
}
/// Reference-frame DPB depth when RFI is supported (Apollo uses 5). A deeper DPB lets an invalidated /// Reference-frame DPB depth when RFI is supported (Apollo uses 5). A deeper DPB lets an invalidated
/// reference fall back to an older still-valid frame instead of a full IDR; `numRefL0 = 1` keeps each /// reference fall back to an older still-valid frame instead of a full IDR; `numRefL0 = 1` keeps each
/// P-frame single-reference for low latency. Also the window the backends' `invalidate_ref_frames` /// P-frame single-reference for low latency. Also the window the backends' `invalidate_ref_frames`
+35 -3
View File
@@ -72,9 +72,41 @@ pub(super) fn explain(status: nv::NVENCSTATUS) -> String {
} }
} }
/// Typed root of a failed NVENC entry-point call: carries the raw status so callers can classify
/// the failure class, not just print it — the bitrate-clamp search must only read a
/// parameter/caps rejection as "above the codec-level ceiling"; a transient failure shrinking the
/// search would discover (and cache) a bogus ceiling. Recover it through an `anyhow` chain with
/// `err.downcast_ref::<NvCallError>()` (see [`is_param_rejection`]).
#[derive(Debug)]
pub(super) struct NvCallError(pub(super) nv::NVENCSTATUS);
impl std::fmt::Display for NvCallError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{:?} — {}", self.0, explain(self.0))
}
}
impl std::error::Error for NvCallError {}
/// Whether `err` is an NVENC parameter/capability rejection: the driver understood the request
/// and says THIS config is not encodable — the clamp search's "bitrate above the ceiling"
/// evidence. Everything else (busy engine, session limit, OOM, device loss, version skew) is
/// environmental and must propagate instead of steering the search.
pub(super) fn is_param_rejection(err: &anyhow::Error) -> bool {
matches!(
err.downcast_ref::<NvCallError>(),
Some(NvCallError(
nv::NVENCSTATUS::NV_ENC_ERR_INVALID_PARAM
| nv::NVENCSTATUS::NV_ENC_ERR_UNSUPPORTED_PARAM
| nv::NVENCSTATUS::NV_ENC_ERR_UNIMPLEMENTED,
))
)
}
/// Build an actionable `anyhow::Error` for a failed NVENC entry-point call. `call` names the API /// Build an actionable `anyhow::Error` for a failed NVENC entry-point call. `call` names the API
/// (e.g. `"open_encode_session_ex"`); the message carries both the raw status and its real-world /// (e.g. `"open_encode_session_ex"`); the chain carries both the raw status and its real-world
/// cause, so triage never again reads a version mismatch as "(no NVIDIA GPU?)". /// cause, so triage never again reads a version mismatch as "(no NVIDIA GPU?)". The
/// [`NvCallError`] root keeps the status downcastable for failure-class checks.
pub(super) fn call_err(call: &str, status: nv::NVENCSTATUS) -> anyhow::Error { pub(super) fn call_err(call: &str, status: nv::NVENCSTATUS) -> anyhow::Error {
anyhow::anyhow!("NVENC {call} failed: {status:?} — {}", explain(status)) anyhow::Error::new(NvCallError(status)).context(format!("NVENC {call} failed"))
} }
+108 -59
View File
@@ -37,8 +37,9 @@
#![deny(clippy::undocumented_unsafe_blocks)] #![deny(clippy::undocumented_unsafe_blocks)]
use super::nvenc_core::{ use super::nvenc_core::{
apply_low_latency_config, build_init_params, codec_guid, resolve_slices, resolve_subframe, apply_low_latency_config, build_init_params, cached_ceiling, codec_guid, resolve_slices,
LowLatencyConfig, NvStatusExt, RFI_DPB, resolve_split_mode, resolve_subframe, store_ceiling, CeilingKey, LowLatencyConfig, NvStatusExt,
RFI_DPB,
}; };
use super::nvenc_status; use super::nvenc_status;
use super::{ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps}; use super::{ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps};
@@ -754,6 +755,27 @@ impl NvencD3d11Encoder {
Ok(cfg) Ok(cfg)
} }
/// This session config's identity in the process-lifetime bitrate-ceiling cache
/// (`nvenc_core::{cached_ceiling, store_ceiling}`). GPU identity is the selected render
/// adapter's LUID — the adapter the capturer's device (and so this session) lives on; `0`
/// when unresolved. Best effort by design: the cache is advisory, a colliding identity costs
/// one failed open + re-search, never a wrong session.
fn ceiling_key(&self, split_mode: u32) -> CeilingKey {
let gpu = pf_gpu::resolve_render_adapter_luid()
.map(|l| ((l.HighPart as u32 as u64) << 32) | l.LowPart as u64)
.unwrap_or(0);
CeilingKey {
gpu,
codec: self.codec,
width: self.width,
height: self.height,
fps: self.fps,
bit_depth: self.bit_depth,
chroma_444: self.chroma_444,
split_mode,
}
}
/// Open + configure + initialize ONE NVENC session at `bitrate` (bps) and `split_mode`. Returns /// Open + configure + initialize ONE NVENC session at `bitrate` (bps) and `split_mode`. Returns
/// the session handle, or destroys it and returns the error. NVENC has no re-init after a failed /// the session handle, or destroys it and returns the error. NVENC has no re-init after a failed
/// `initialize_encoder`, so the bitrate-clamp search in `init_session` calls this once per probe. /// `initialize_encoder`, so the bitrate-clamp search in `init_session` calls this once per probe.
@@ -834,43 +856,14 @@ impl NvencD3d11Encoder {
// gets the highest the GPU can actually do, not a coarse fraction of it. // gets the highest the GPU can actually do, not a coarse fraction of it.
const FLOOR_BPS: u64 = 10_000_000; const FLOOR_BPS: u64 = 10_000_000;
let requested_bps = self.bitrate_bps; let requested_bps = self.bitrate_bps;
// 2-way NVENC split-frame encoding (Ada dual-NVENC) — the high-pixel-rate throughput lever // 2-way NVENC split-frame encoding (Ada dual-NVENC) — the high-pixel-rate throughput lever.
// the Linux host enables via libavcodec `split_encode_mode`. A single Ada NVENC session tops // A single Ada NVENC session tops out ~0.8-1 Gpix/s, so at high motion a 5K@240
// out ~0.8 Gpix/s, so at high motion a 5K@240 (1.77 Gpix/s) frame takes ~8 ms to encode and // (1.77 Gpix/s) frame takes ~8 ms to encode and the rate caps ~125 fps; splitting across
// the rate caps ~125 fps; splitting across both engines roughly halves that. Force 2-way // both engines roughly halves that. Shared selector — see [`resolve_split_mode`] for the
// above ~1 Gpix/s (matching encode/linux.rs), AUTO below (the ~2% BD-rate cost isn't worth // precedence (env override / the measured Main10 don't-split rule / pixel rate).
// it at low pixel rates). Env override PUNKTFUNK_SPLIT_ENCODE = 0/disable | 1/auto | 2 | 3. // The init-failure fallback below disables it if a codec/config rejects it.
// HEVC/AV1 only; the init-failure fallback below disables it if a codec/config rejects it.
let pixel_rate = self.width as u64 * self.height as u64 * self.fps.max(1) as u64; let pixel_rate = self.width as u64 * self.height as u64 * self.fps.max(1) as u64;
let mut split_mode: u32 = match std::env::var("PUNKTFUNK_SPLIT_ENCODE").ok().as_deref() let split_mode: u32 = resolve_split_mode(self.bit_depth, pixel_rate);
{
Some("0") | Some("disable") => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32
}
Some("1") | Some("auto") => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_AUTO_FORCED_MODE as u32
}
Some("3") => nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_THREE_FORCED_MODE as u32,
Some("2") => nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_TWO_FORCED_MODE as u32,
// Main10 (10-bit / HDR): 2-way split is measurably SLOWER on Ada — at 5120x1440@240
// Main10, forced-2 took 7.6 ms/frame (~131 fps) vs 2.8 ms (~357 fps) single-engine
// (the split/merge overhead dominates for 10-bit). A single Ada NVENC engine already
// handles 5K@240 Main10 well under the 4.17 ms budget, so DON'T split — splitting was
// the "broken animations in HDR" (the stream capped at ~131 fps). Env still overrides.
_ if self.bit_depth >= 10 => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32
}
_ if pixel_rate > 1_000_000_000 => {
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_TWO_FORCED_MODE as u32
}
_ => nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_AUTO_MODE as u32,
};
tracing::debug!(
split_mode,
bit_depth = self.bit_depth,
pixel_rate,
"NVENC split-encode mode selected"
);
// Find the highest bitrate the GPU's codec LEVEL accepts and CLAMP to it. NVENC rejects // Find the highest bitrate the GPU's codec LEVEL accepts and CLAMP to it. NVENC rejects
// `initialize_encoder` (InvalidParam) when the bitrate exceeds the level ceiling (e.g. a // `initialize_encoder` (InvalidParam) when the bitrate exceeds the level ceiling (e.g. a
// 1 Gbps request on HEVC). Strategy: try the requested rate; if the only problem is a forced // 1 Gbps request on HEVC). Strategy: try the requested rate; if the only problem is a forced
@@ -884,39 +877,69 @@ impl NvencD3d11Encoder {
// built in the right mode from the start. // built in the right mode from the start.
let use_async = self.async_supported && async_retrieve_requested(); let use_async = self.async_supported && async_retrieve_requested();
let mut probe = self.try_open_session(device, requested_bps, split_mode, use_async); // Ceiling cache (process lifetime, `nvenc_core`): a prior clamp search already found
// this config's max accepted rate — open straight AT the ceiling instead of paying
// the ~6-open binary search (and its session churn) on every ABR overshoot.
let mut target_bps = requested_bps;
if let Some(ceiling) = cached_ceiling(&self.ceiling_key(split_mode)) {
if requested_bps > ceiling {
tracing::info!(
requested_mbps = requested_bps / 1_000_000,
ceiling_mbps = ceiling / 1_000_000,
"NVENC: requested bitrate above the cached codec-level ceiling — opening \
at the ceiling"
);
target_bps = ceiling;
}
}
let mut probe = self.try_open_session(device, target_bps, split_mode, use_async);
// The cache is advisory: a stale entry (driver change, identity collision) must not
// wedge the open — retry the requested rate and let the search below rediscover.
if probe.is_err() && target_bps < requested_bps {
target_bps = requested_bps;
probe = self.try_open_session(device, requested_bps, split_mode, use_async);
}
// Disambiguate a forced-split rejection from a bitrate-cap rejection: retry once at the // Disambiguate a forced-split rejection from a bitrate-cap rejection: retry once at the
// requested rate with split disabled — if THAT succeeds, split was the problem, not bitrate. // requested rate with split disabled — if THAT succeeds, split was the problem, not bitrate.
// ANY non-disabled mode can be the rejection — AUTO included: AV1 rejects the whole // ANY non-disabled mode can be the rejection — AUTO included: AV1 rejects the whole
// init with INVALID_PARAM on drivers/configs where auto split isn't valid for it, // init with INVALID_PARAM on drivers/configs where auto split isn't valid for it,
// which then masqueraded as a bitrate cap and failed "even at the floor". // which then masqueraded as a bitrate cap and failed "even at the floor".
// `used_split` tracks the mode sessions ACTUALLY open with from here on — it feeds
// `self.split_mode` (a reconfigure must re-present it) and the ceiling-cache key.
let mut used_split = split_mode;
let split_on = let split_on =
split_mode != nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32; split_mode != nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32;
if probe.is_err() && split_on { if probe.is_err() && split_on {
let no_split = nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32; let no_split = nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32;
if let Ok(e) = self.try_open_session(device, requested_bps, no_split, use_async) { if let Ok(e) = self.try_open_session(device, target_bps, no_split, use_async) {
tracing::warn!("NVENC: split-encode rejected by codec/config — disabled"); tracing::warn!("NVENC: split-encode rejected by codec/config — disabled");
split_mode = no_split; used_split = no_split;
probe = Ok(e); probe = Ok(e);
} }
} }
let enc = match probe { let enc = match probe {
Ok(enc) => { Ok(enc) => {
self.bitrate_bps = requested_bps; self.bitrate_bps = target_bps;
enc enc
} }
// Only a parameter/caps rejection means "the bitrate is above the codec-level
// ceiling". A transient failure (busy engine, session limit, OOM, device loss,
// version skew) must propagate — a search steered by it would discover, and
// cache, a bogus ceiling.
Err(e) if !nvenc_status::is_param_rejection(&e) => return Err(e),
Err(_) => { Err(_) => {
// Requested bitrate exceeds the codec-level ceiling — binary-search the max accepted. // Requested bitrate exceeds the codec-level ceiling — binary-search the max accepted.
// `lo` is the highest known-good rate (FLOOR is assumed to fit), `hi` the lowest // `lo` is the highest known-good rate (FLOOR is assumed to fit), `hi` the lowest
// rejected; `best` holds the live session at `lo` so we end up with the clamped one. // rejected; `best` holds the live session at `lo` so we end up with the clamped one.
let mut lo = FLOOR_BPS; let mut lo = FLOOR_BPS;
let mut hi = requested_bps; let mut hi = target_bps;
let mut best: *mut c_void = ptr::null_mut(); let mut best: *mut c_void = ptr::null_mut();
let mut best_bps = 0u64; let mut best_bps = 0u64;
while hi > lo + CLAMP_TOL_BPS { while hi > lo + CLAMP_TOL_BPS {
let mid = lo + (hi - lo) / 2; let mid = lo + (hi - lo) / 2;
match self.try_open_session(device, mid, split_mode, use_async) { match self.try_open_session(device, mid, used_split, use_async) {
Ok(e) => { Ok(e) => {
if !best.is_null() { if !best.is_null() {
let _ = (api().destroy_encoder)(best); let _ = (api().destroy_encoder)(best);
@@ -925,7 +948,15 @@ impl NvencD3d11Encoder {
best_bps = mid; best_bps = mid;
lo = mid; lo = mid;
} }
Err(_) => hi = mid, Err(e) if nvenc_status::is_param_rejection(&e) => hi = mid,
Err(e) => {
// Environmental mid-search failure: don't let it shrink the
// search — release the partial result and propagate.
if !best.is_null() {
let _ = (api().destroy_encoder)(best);
}
return Err(e);
}
} }
} }
if best.is_null() { if best.is_null() {
@@ -933,14 +964,19 @@ impl NvencD3d11Encoder {
// trying split-disabled in case a forced split (not the bitrate) is the blocker. // trying split-disabled in case a forced split (not the bitrate) is the blocker.
let no_split = let no_split =
nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32; nv::NV_ENC_SPLIT_ENCODE_MODE::NV_ENC_SPLIT_DISABLE_MODE as u32;
best = self best = match self.try_open_session(device, FLOOR_BPS, used_split, use_async)
.try_open_session(device, FLOOR_BPS, split_mode, use_async) {
.or_else(|_| { Ok(e) => e,
self.try_open_session(device, FLOOR_BPS, no_split, use_async) Err(_) => {
}) let e = self
.context( .try_open_session(device, FLOOR_BPS, no_split, use_async)
"NVENC initialize_encoder rejected even at the floor bitrate", .context(
)?; "NVENC initialize_encoder rejected even at the floor bitrate",
)?;
used_split = no_split;
e
}
};
best_bps = FLOOR_BPS; best_bps = FLOOR_BPS;
} }
tracing::warn!( tracing::warn!(
@@ -948,21 +984,19 @@ impl NvencD3d11Encoder {
clamped_mbps = best_bps / 1_000_000, clamped_mbps = best_bps / 1_000_000,
"NVENC: requested bitrate above the GPU codec-level ceiling — clamped to the max accepted" "NVENC: requested bitrate above the GPU codec-level ceiling — clamped to the max accepted"
); );
store_ceiling(self.ceiling_key(used_split), best_bps);
self.bitrate_bps = best_bps; self.bitrate_bps = best_bps;
best best
} }
}; };
self.encoder = enc; self.encoder = enc;
// Session init params a later `reconfigure_bitrate` must re-present verbatim. (Best // Session init params a later `reconfigure_bitrate` must re-present verbatim.
// effort: the floor fallback above may have succeeded split-disabled without updating self.split_mode = used_split;
// `split_mode` — a reconfigure then presents the forced mode, NVENC rejects it, and
// the caller's rebuild fallback covers the mismatch.)
self.split_mode = split_mode;
self.session_async = use_async; self.session_async = use_async;
// Session-budget accounting (Stage W3): record what this open holds so admission can // Session-budget accounting (Stage W3): record what this open holds so admission can
// decline a parallel display the hardware can't afford. Weighted by the FINAL split // decline a parallel display the hardware can't afford. Weighted by the FINAL split
// mode (a split session occupies one hardware session per engine). // mode (a split session occupies one hardware session per engine).
self.session_units = split_mode_units(split_mode); self.session_units = split_mode_units(used_split);
LIVE_SESSION_UNITS.fetch_add(self.session_units, std::sync::atomic::Ordering::Relaxed); LIVE_SESSION_UNITS.fetch_add(self.session_units, std::sync::atomic::Ordering::Relaxed);
// (The clamp path above already logs the requested→clamped bitrate at warn; no second // (The clamp path above already logs the requested→clamped bitrate at warn; no second
// info line for the same event here.) // info line for the same event here.)
@@ -1552,6 +1586,15 @@ impl Encoder for NvencD3d11Encoder {
self.bitrate_bps = bps; self.bitrate_bps = bps;
return true; return true;
} }
// Cached codec-level ceiling: clamp the target BEFORE the driver call, so a known
// overshoot retargets to the ceiling IN PLACE instead of bouncing off the driver into
// the caller's full-rebuild fallback (an IDR plus ~half a second of session churn per
// ABR overshoot on the pre-cache path). The caller reads the clamp back through
// [`Encoder::applied_bitrate_bps`].
let bps = match cached_ceiling(&self.ceiling_key(self.split_mode)) {
Some(ceiling) => bps.min(ceiling),
None => bps,
};
// SAFETY: `inited` ⟹ `self.encoder` is the live session and this runs on the encode // SAFETY: `inited` ⟹ `self.encoder` is the live session and this runs on the encode
// thread between submit/poll (`nvEncReconfigureEncoder` is a submit-side call, the // thread between submit/poll (`nvEncReconfigureEncoder` is a submit-side call, the
// sanctioned side of the two-thread async split — the retrieve thread only ever locks // sanctioned side of the two-thread async split — the retrieve thread only ever locks
@@ -1600,6 +1643,12 @@ impl Encoder for NvencD3d11Encoder {
} }
} }
fn applied_bitrate_bps(&self) -> Option<u64> {
// `bitrate_bps` is the post-clamp truth: the open path's ceiling search and the
// reconfigure path's cache clamp both write what the session ACTUALLY targets.
Some(self.bitrate_bps)
}
fn flush(&mut self) -> Result<()> { fn flush(&mut self) -> Result<()> {
Ok(()) // P1/ULL + frameIntervalP=1: each submit yields its AU; no internal queue to drain. Ok(()) // P1/ULL + frameIntervalP=1: each submit yields its AU; no internal queue to drain.
} }
+6
View File
@@ -243,6 +243,12 @@ impl Encoder for TrackedEncoder {
fn reconfigure_bitrate(&mut self, bps: u64) -> bool { fn reconfigure_bitrate(&mut self, bps: u64) -> bool {
self.inner.reconfigure_bitrate(bps) self.inner.reconfigure_bitrate(bps)
} }
// Forwarded (the same trap class as `set_wire_chunking`): the unforwarded default `None`
// would tell the session loop "no encoder truth here" and it would keep pacing/acking the
// requested rate even where NVENC clamped to the codec-level ceiling.
fn applied_bitrate_bps(&self) -> Option<u64> {
self.inner.applied_bitrate_bps()
}
fn flush(&mut self) -> Result<()> { fn flush(&mut self) -> Result<()> {
self.inner.flush() self.inner.flush()
} }
+18
View File
@@ -985,6 +985,18 @@ async fn serve_session(
// full IDR when the encoder supports it (native-AMF LTR / Windows NVENC). // full IDR when the encoder supports it (native-AMF LTR / Windows NVENC).
let (rfi_tx, rfi_rx) = std::sync::mpsc::channel::<(u32, u32)>(); let (rfi_tx, rfi_rx) = std::sync::mpsc::channel::<(u32, u32)>();
let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::<u32>(); let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::<u32>();
// Encoder-truth bridge, data plane → control task (§ABR overdrive). The encode loop publishes
// here; the control task reads at `SetBitrate`-resolve time, so the ack the client's
// controller climbs from tracks what the encoder ACTUALLY does, not what was asked:
// - `live_bitrate`: the encoder's applied rate (kbps) — also the send pacer's/console's view.
// - `encoder_ceiling_kbps`: the discovered codec-level ceiling (0 = none discovered yet);
// resolves land at min(policy clamp, ceiling), so overshoots stop costing rebuilds.
// - `cadence_degraded`: encode can't hold the frame cadence — a climb is refused (acked at
// the current rate); the network isn't the bottleneck, more bits are anti-medicine.
// Plain atomics, not a channel: only the freshest value matters, and only at resolve time.
let live_bitrate = Arc::new(AtomicU32::new(welcome.bitrate_kbps));
let encoder_ceiling_kbps = Arc::new(AtomicU32::new(0));
let cadence_degraded = Arc::new(AtomicBool::new(false));
let (probe_tx, probe_rx) = std::sync::mpsc::channel::<ProbeRequest>(); let (probe_tx, probe_rx) = std::sync::mpsc::channel::<ProbeRequest>();
let (probe_result_tx, probe_result_rx) = tokio::sync::mpsc::unbounded_channel::<ProbeResult>(); let (probe_result_tx, probe_result_rx) = tokio::sync::mpsc::unbounded_channel::<ProbeResult>();
// Mode-switch outcome, data plane → control task (same pattern as `probe_result_tx`): the accept // Mode-switch outcome, data plane → control task (same pattern as `probe_result_tx`): the accept
@@ -1036,6 +1048,9 @@ async fn serve_session(
live_reconfig_ok, live_reconfig_ok,
adaptive_fec, adaptive_fec,
session_bitrate_kbps, session_bitrate_kbps,
live_bitrate.clone(),
encoder_ceiling_kbps.clone(),
cadence_degraded.clone(),
fec_target_ctl, fec_target_ctl,
reconfig_tx, reconfig_tx,
keyframe_tx, keyframe_tx,
@@ -1429,6 +1444,9 @@ async fn serve_session(
bitrate_rx, bitrate_rx,
compositor, compositor,
bitrate_kbps, bitrate_kbps,
live_bitrate,
encoder_ceiling_kbps,
cadence_degraded,
bitrate_auto, bitrate_auto,
bit_depth, bit_depth,
chroma, chroma,
+32 -1
View File
@@ -22,6 +22,13 @@ pub(super) async fn run(
live_reconfig_ok: bool, live_reconfig_ok: bool,
adaptive_fec: bool, adaptive_fec: bool,
session_bitrate_kbps: u32, session_bitrate_kbps: u32,
/// Encoder-truth bridge (data plane → here, §ABR overdrive): the encoder's live applied rate,
/// its discovered codec-level ceiling (0 = unknown), and the "encode can't hold cadence"
/// flag. Read at `SetBitrate`-resolve time so the ack — the base the client's controller
/// climbs from — never promises a rate the encoder won't run at.
live_bitrate: Arc<AtomicU32>,
encoder_ceiling_kbps: Arc<AtomicU32>,
cadence_degraded: Arc<AtomicBool>,
fec_target_ctl: Arc<AtomicU8>, fec_target_ctl: Arc<AtomicU8>,
reconfig_tx: std::sync::mpsc::Sender<punktfunk_core::Mode>, reconfig_tx: std::sync::mpsc::Sender<punktfunk_core::Mode>,
keyframe_tx: std::sync::mpsc::Sender<()>, keyframe_tx: std::sync::mpsc::Sender<()>,
@@ -161,7 +168,31 @@ pub(super) async fn run(
); );
session_bitrate_kbps session_bitrate_kbps
} else { } else {
resolve_bitrate_kbps(req.bitrate_kbps) let mut r = resolve_bitrate_kbps(req.bitrate_kbps);
// Encoder truth (§ABR overdrive): the ack below is the base the
// client's controller climbs from, so it must not promise past the
// encoder's discovered codec-level ceiling — the pre-fix path acked
// 1.01 Gbps while the ASIC ran 794 Mbps, and the controller climbed
// from the phantom number forever (a ~0.6 s rebuild + IDR per step).
let ceiling = encoder_ceiling_kbps.load(Ordering::Relaxed);
if ceiling != 0 && r > ceiling {
r = ceiling;
}
// Climb refusal while encode can't hold cadence: on a fat LAN no
// network signal ever stops the climb, and past the compute knee more
// bits only deepen the miss. Resolve a CLIMB to the current applied
// rate (descents pass — they're the cure); the short ack teaches the
// client controller its ceiling.
let live = live_bitrate.load(Ordering::Relaxed);
if cadence_degraded.load(Ordering::Relaxed) && live != 0 && r > live {
tracing::info!(
requested_kbps = req.bitrate_kbps,
held_kbps = live,
"bitrate climb refused — encode is behind cadence"
);
r = live;
}
r
}; };
tracing::debug!( tracing::debug!(
requested_kbps = req.bitrate_kbps, requested_kbps = req.bitrate_kbps,
+20 -3
View File
@@ -909,6 +909,20 @@ pub(super) struct SessionContext {
pub(super) compositor: crate::vdisplay::Compositor, pub(super) compositor: crate::vdisplay::Compositor,
/// Negotiated encoder bitrate (kbps). /// Negotiated encoder bitrate (kbps).
pub(super) bitrate_kbps: u32, pub(super) bitrate_kbps: u32,
/// The encoder's live APPLIED rate (kbps) — shared with the send pacer, the web console, the
/// mgmt registry AND the control task (which acks climbs against it). The encode loop stores
/// `Encoder::applied_bitrate_bps` here after every apply, so everything downstream tracks
/// what the ASIC really targets, not what was requested (§ABR overdrive).
pub(super) live_bitrate: Arc<AtomicU32>,
/// The encoder's discovered codec-level bitrate ceiling (kbps; 0 = none discovered): written
/// when an apply comes back short, read by this loop (pre-clamp incoming requests — a
/// request already AT the ceiling then costs nothing) and by the control task (truthful
/// acks from the first post-discovery request).
pub(super) encoder_ceiling_kbps: Arc<AtomicU32>,
/// "Encode can't hold the frame cadence" (the escalation leaky bucket is elevated, or the
/// session escalated): while set, the control task refuses bitrate CLIMBS — the network
/// isn't the bottleneck, feeding the encoder more bits deepens the miss.
pub(super) cadence_degraded: Arc<AtomicBool>,
/// The client asked for "Automatic" (`Hello::bitrate_kbps == 0`), so `bitrate_kbps` came from /// The client asked for "Automatic" (`Hello::bitrate_kbps == 0`), so `bitrate_kbps` came from
/// the host's codec-aware default. For PyroWave that default is the ~1.6 bpp operating point of /// the host's codec-aware default. For PyroWave that default is the ~1.6 bpp operating point of
/// the NEGOTIATED MODE (`resolve_bitrate_kbps_for`) — a mid-stream mode switch re-resolves it /// the NEGOTIATED MODE (`resolve_bitrate_kbps_for`) — a mid-stream mode switch re-resolves it
@@ -1035,6 +1049,9 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
bitrate_rx, bitrate_rx,
compositor, compositor,
mut bitrate_kbps, mut bitrate_kbps,
live_bitrate,
encoder_ceiling_kbps,
cadence_degraded,
bitrate_auto, bitrate_auto,
bit_depth, bit_depth,
// The resolved chroma is already captured in `plan` (above); ignore the duplicate here. // The resolved chroma is already captured in `plan` (above); ignore the duplicate here.
@@ -1254,9 +1271,9 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
// The bounded channel applies backpressure (the encode thread blocks if the send falls behind, // The bounded channel applies backpressure (the encode thread blocks if the send falls behind,
// so frames slow down rather than a dropped frame freezing the infinite-GOP stream). // so frames slow down rather than a dropped frame freezing the infinite-GOP stream).
let (frame_tx, frame_rx) = std::sync::mpsc::sync_channel::<SendMsg>(3); let (frame_tx, frame_rx) = std::sync::mpsc::sync_channel::<SendMsg>(3);
// Live encoder bitrate, shared with the send thread's stats sample: a mid-stream adaptive // `live_bitrate` (SessionContext) is shared with the send thread's stats sample AND the
// bitrate change (bitrate_rx below) updates it so the console shows the actual target. // control task: a mid-stream adaptive bitrate change (bitrate_rx below) stores the
let live_bitrate = Arc::new(AtomicU32::new(bitrate_kbps)); // encoder-APPLIED rate, so the console, pacer and climb-refusal acks all see the truth.
// Live session mode, same pattern (H3): a mid-stream mode switch (reconfig below) updates it so // Live session mode, same pattern (H3): a mid-stream mode switch (reconfig below) updates it so
// a stats capture armed after a resize registers the real mode. Seeded with the refresh the // a stats capture armed after a resize registers the real mode. Seeded with the refresh the
// initial build actually achieved (`interval_hz`), not the request — KWin may cap a virtual // initial build actually achieved (`interval_hz`), not the request — KWin may cap a virtual