feat(host/encode): de-escalate the latency escalation once cadence holds clean
ci / web (push) Successful in 46s
ci / docs-site (push) Successful in 1m7s
decky / build-publish (push) Successful in 32s
ci / bench (push) Successful in 6m12s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 42s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 6m44s
deb / build-publish (push) Successful in 12m39s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 11s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
deb / build-publish-host (push) Successful in 13m29s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 6m35s
android / android (push) Successful in 15m57s
arch / build-publish (push) Successful in 16m18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9m21s
docker / deploy-docs (push) Successful in 23s
windows-host / package (push) Successful in 17m51s
apple / swift (push) Successful in 5m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m32s
ci / rust (push) Successful in 21m53s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m20s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 4m40s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m45s
flatpak / build-publish (push) Successful in 6m24s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m2s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m35s
release / apple (push) Successful in 30m44s
apple / screenshots (push) Successful in 24m33s
ci / web (push) Successful in 46s
ci / docs-site (push) Successful in 1m7s
decky / build-publish (push) Successful in 32s
ci / bench (push) Successful in 6m12s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 42s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 6m44s
deb / build-publish (push) Successful in 12m39s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 11s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
deb / build-publish-host (push) Successful in 13m29s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 6m35s
android / android (push) Successful in 15m57s
arch / build-publish (push) Successful in 16m18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9m21s
docker / deploy-docs (push) Successful in 23s
windows-host / package (push) Successful in 17m51s
apple / swift (push) Successful in 5m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m32s
ci / rust (push) Successful in 21m53s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m20s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 4m40s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m45s
flatpak / build-publish (push) Successful in 6m24s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m2s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m35s
release / apple (push) Successful in 30m44s
apple / screenshots (push) Successful in 24m33s
Escalate-and-hold's missing half. The contention escalation (capture depth, then the NVENC pipelined retrieve) was permanent: one sustained overrun — even one CAUSED by the ABR overdrive's rebuild storms — cost the session its depth-1 latency and its sub-frame streaming forever, and `encode_us` reported queue depth instead of ASIC time for the rest of the session. - The leaky bucket now keeps scoring after escalation. A sustained every-frame-on-cadence run (~5 s at 120 fps) winds back one stage in reverse order: pipelined retrieve first (its rebuild restores the IO-stream binding and sub-frame chunked streaming), then capture depth back to 1. Attempts are paced by an exponential backoff (1 → 5 → 25 min, capped) — a workload that truly needs the escalation converges to keeping it, but never a permanent latch. - NVENC (Linux) implements `set_pipelined(false)`: a `want_sync` latch handled at the same drained safe point as the engage side (`maybe_disengage_async` mirrors `maybe_engage_async`); the lazy sync re-init re-arms everything and opens on an IDR. The stream loop polls until the switch lands, then re-runs the escalation warmup so the wind-back's own stall can't re-escalate it. `PUNKTFUNK_NVENC_ASYNC=1` (operator-pinned async) refuses the wind-back; the trait doc now specifies the two-way contract. - While escalated, `cadence_degraded` stays latched (bitrate climbs refused) even with the bucket drained: the headroom is spent, and climbs resuming mid-escalation would saw against it and starve the clean run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1457,6 +1457,22 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
// network, is the bottleneck" evidence — enough to flag `cadence_degraded` (the control task
|
||||
// then refuses bitrate CLIMBS) well before the session pays a latency escalation for it.
|
||||
const DEPTH_DEGRADE: u32 = 10;
|
||||
// De-escalation (the escalate-and-hold v1's missing half): a sustained clean run at the
|
||||
// escalated setting (~5 s at 120 fps, every frame on cadence) earns ONE attempt at winding
|
||||
// back — reverse order of the escalation, pipelined retrieve first (its rebuild restores
|
||||
// sub-frame streaming and the IO-stream binding), then capture depth back to 1. Each
|
||||
// attempt costs the wind-back rebuild's IDR, so attempts are paced by an exponential
|
||||
// backoff (1 → 5 → 25 min, capped) — a workload that genuinely needs the escalation
|
||||
// converges to keeping it, but NEVER a permanent latch: a latch plus the ABR sawtooth
|
||||
// pinned sessions at the floor with the escalation stuck.
|
||||
const DEESCALATE_CLEAN_FRAMES: u32 = 600;
|
||||
const DEESCALATE_BACKOFF_START: std::time::Duration = std::time::Duration::from_secs(60);
|
||||
const DEESCALATE_BACKOFF_MAX: std::time::Duration = std::time::Duration::from_secs(25 * 60);
|
||||
let mut pipelined_active = false;
|
||||
let mut deescalating = false;
|
||||
let mut ahead_run: u32 = 0;
|
||||
let mut deescalate_not_before: Option<std::time::Instant> = None;
|
||||
let mut deescalate_backoff = DEESCALATE_BACKOFF_START;
|
||||
while !stop.load(Ordering::SeqCst) && std::time::Instant::now() < deadline {
|
||||
// Mid-stream session switch (the box flipped Gaming↔Desktop): rebuild the WHOLE backend in
|
||||
// place — a different compositor at the SAME client mode — keeping the Session + send thread
|
||||
@@ -1794,6 +1810,7 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
// escalation — clean slate + re-run the warmup before judging again.
|
||||
behind_score = 0;
|
||||
depth_frames = 0;
|
||||
ahead_run = 0;
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), to_kbps = new_kbps,
|
||||
@@ -2618,7 +2635,7 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
// two-thread lock moves the encode wait off this loop so capture/submit keep cadence,
|
||||
// at ~one tick of AU latency. `enc.set_pipelined` may decline (unsupported backend or
|
||||
// an explicit PUNKTFUNK_NVENC_ASYNC=0); either way it is asked exactly once.
|
||||
if idd_adaptive_enabled() && (cur_depth < max_depth || !pipeline_asked) {
|
||||
if idd_adaptive_enabled() {
|
||||
depth_frames += 1;
|
||||
if depth_frames > DEPTH_WARMUP_FRAMES {
|
||||
let behind = std::time::Instant::now() >= next;
|
||||
@@ -2627,34 +2644,89 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
} else {
|
||||
behind_score.saturating_sub(1)
|
||||
};
|
||||
let escalated = cur_depth > 1 || pipelined_active || deescalating;
|
||||
// Export "encode can't hold cadence" for the control task's climb refusal.
|
||||
// Stored BEFORE the escalate check: the firing iteration writes `true`, and a
|
||||
// final-stage escalation freezes this whole block (the guard above goes false)
|
||||
// — so an escalated session stays flagged, which is exactly right: its climb
|
||||
// headroom is spent until something (a down-step, de-escalation) changes.
|
||||
cadence_degraded.store(behind_score >= DEPTH_DEGRADE, Ordering::Relaxed);
|
||||
if behind_score >= DEPTH_ESCALATE {
|
||||
// An escalated session stays flagged even with the bucket drained: its climb
|
||||
// headroom is spent, and letting climbs resume would saw against the
|
||||
// escalation and starve the de-escalation clean run below.
|
||||
cadence_degraded.store(
|
||||
escalated || behind_score >= DEPTH_DEGRADE,
|
||||
Ordering::Relaxed,
|
||||
);
|
||||
if deescalating {
|
||||
// A requested wind-back completes at the encoder's drained safe point —
|
||||
// poll it (the call is a cheap latch check until then).
|
||||
if !enc.set_pipelined(false) {
|
||||
deescalating = false;
|
||||
pipelined_active = false;
|
||||
// Re-arm the ask: a future sustained overrun may escalate again (the
|
||||
// backoff below paces how soon another wind-back may follow it).
|
||||
pipeline_asked = false;
|
||||
tracing::info!(
|
||||
"encoder pipelined retrieve de-escalated — sync retrieve (and \
|
||||
sub-frame streaming, where armed) restored; re-monitoring cadence"
|
||||
);
|
||||
// The wind-back rebuild's own stall must not re-escalate on the spot.
|
||||
behind_score = 0;
|
||||
depth_frames = 0;
|
||||
ahead_run = 0;
|
||||
}
|
||||
} else if behind_score >= DEPTH_ESCALATE
|
||||
&& (cur_depth < max_depth || !pipeline_asked)
|
||||
{
|
||||
if cur_depth < max_depth {
|
||||
cur_depth = max_depth;
|
||||
tracing::info!(
|
||||
depth = cur_depth,
|
||||
"IDD pipeline depth escalated — encode can't hold cadence at depth-1 \
|
||||
(GPU contention); pipelining for the rest of the session (latency \
|
||||
(GPU contention); pipelining until cadence holds clean (latency \
|
||||
trade for throughput)"
|
||||
);
|
||||
} else {
|
||||
pipeline_asked = true;
|
||||
if enc.set_pipelined(true) {
|
||||
pipelined_active = enc.set_pipelined(true);
|
||||
if pipelined_active {
|
||||
tracing::info!(
|
||||
"encoder pipelined retrieve escalated — encode can't hold \
|
||||
cadence and the capturer has no depth to give; the encode wait \
|
||||
moves off the loop for the rest of the session (latency trade \
|
||||
moves off the loop until cadence holds clean (latency trade \
|
||||
for throughput)"
|
||||
);
|
||||
}
|
||||
}
|
||||
// Give the action time to take effect before judging again.
|
||||
behind_score = 0;
|
||||
ahead_run = 0;
|
||||
} else if escalated {
|
||||
// De-escalation: a sustained every-frame-on-cadence run at the escalated
|
||||
// setting is the evidence the contention passed (a lower ABR rate, the
|
||||
// game scene lightened) — wind back in reverse order, paced by the
|
||||
// exponential backoff (see the consts above).
|
||||
ahead_run = if behind { 0 } else { ahead_run + 1 };
|
||||
if ahead_run >= DEESCALATE_CLEAN_FRAMES
|
||||
&& deescalate_not_before.is_none_or(|t| std::time::Instant::now() >= t)
|
||||
{
|
||||
ahead_run = 0;
|
||||
deescalate_not_before =
|
||||
Some(std::time::Instant::now() + deescalate_backoff);
|
||||
deescalate_backoff = (deescalate_backoff * 5).min(DEESCALATE_BACKOFF_MAX);
|
||||
if pipelined_active {
|
||||
tracing::info!(
|
||||
"cadence held clean while escalated — winding the pipelined \
|
||||
retrieve back (latency recovery; costs one IDR)"
|
||||
);
|
||||
deescalating = true;
|
||||
} else if cur_depth > 1 {
|
||||
cur_depth = 1;
|
||||
tracing::info!(
|
||||
depth = cur_depth,
|
||||
"IDD pipeline depth de-escalated — cadence held clean at the \
|
||||
escalated depth (latency recovery)"
|
||||
);
|
||||
behind_score = 0;
|
||||
depth_frames = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user