feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike. Client (pf-client-core): after a reference loss the hardware decoder conceals the missing-reference deltas (on RADV, a gray plate with new motion painted over) and returns Ok. The pump now freezes on the last good picture until a clean re-anchor instead of showing the concealment — lifting on a real IDR, an intra-refresh recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The frame_index gap is the early, precise loss signal and drives an RFI request. Host recovery signals (inert unless the backend supports them): - USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR). - USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame. AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the only clean-recovery lever is user LTR: mark frames as long-term references, and on loss force the next frame to re-reference the newest known-good one — a clean P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR before the loss; a range older than the live slots falls back to a keyframe. Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi(). Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success anchors the keyframe cooldown so the client's frames_dropped echo of the same loss is coalesced away rather than emitting a redundant IDR. Spike: synthetic NV12 GPU source for headless AMF encoder testing. Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests (incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux; punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90 forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -34,7 +34,7 @@ use punktfunk_core::packet::{FLAG_PIC, FLAG_PROBE, FLAG_SOF};
|
||||
use punktfunk_core::quic::{
|
||||
endpoint, io, BitrateChanged, ClockEcho, ClockProbe, ColorInfo, Hello, LossReport,
|
||||
PairChallenge, PairProof, PairRequest, PairResult, ProbeRequest, ProbeResult, Reconfigure,
|
||||
Reconfigured, RequestKeyframe, SetBitrate, Start, Welcome,
|
||||
Reconfigured, RequestKeyframe, RfiRequest, SetBitrate, Start, Welcome,
|
||||
};
|
||||
use punktfunk_core::transport::UdpTransport;
|
||||
use punktfunk_core::Session;
|
||||
@@ -1124,6 +1124,10 @@ async fn serve_session(
|
||||
// (inbound requests, outbound probe results) are multiplexed with `select!`.
|
||||
let (reconfig_tx, reconfig_rx) = std::sync::mpsc::channel::<punktfunk_core::Mode>();
|
||||
let (keyframe_tx, keyframe_rx) = std::sync::mpsc::channel::<()>();
|
||||
// Client LTR-RFI recovery: the control task forwards each `RfiRequest`'s lost-frame range here;
|
||||
// the encode loop prefers `Encoder::invalidate_ref_frames` (a clean re-anchor P-frame) over a
|
||||
// full IDR when the encoder supports it (native-AMF LTR / Windows NVENC).
|
||||
let (rfi_tx, rfi_rx) = std::sync::mpsc::channel::<(u32, u32)>();
|
||||
let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::<u32>();
|
||||
let (probe_tx, probe_rx) = std::sync::mpsc::channel::<ProbeRequest>();
|
||||
let (probe_result_tx, mut probe_result_rx) =
|
||||
@@ -1199,6 +1203,19 @@ async fn serve_session(
|
||||
if keyframe_tx.send(()).is_err() {
|
||||
break; // data plane gone
|
||||
}
|
||||
} else if let Ok(req) = RfiRequest::decode(&msg) {
|
||||
// Client LTR-RFI recovery: it lost the frame range `[first, last]` and asks
|
||||
// the encoder to re-reference a known-good older frame instead of paying for
|
||||
// a full IDR. The encode loop attempts `invalidate_ref_frames`, falling back
|
||||
// to a coalesced keyframe when the encoder can't (range too old / no RFI).
|
||||
tracing::debug!(
|
||||
first = req.first_frame,
|
||||
last = req.last_frame,
|
||||
"client requested reference-frame invalidation (loss recovery)"
|
||||
);
|
||||
if rfi_tx.send((req.first_frame, req.last_frame)).is_err() {
|
||||
break; // data plane gone
|
||||
}
|
||||
} else if let Ok(rep) = LossReport::decode(&msg) {
|
||||
// Adaptive FEC: size recovery to the loss the client is seeing. The data-plane
|
||||
// send loop reads `fec_target_ctl` and applies it per frame. Ignored when FEC
|
||||
@@ -1590,6 +1607,7 @@ async fn serve_session(
|
||||
quit: quit_stream,
|
||||
reconfig: reconfig_rx,
|
||||
keyframe: keyframe_rx,
|
||||
rfi: rfi_rx,
|
||||
bitrate_rx,
|
||||
compositor,
|
||||
bitrate_kbps,
|
||||
@@ -2396,6 +2414,29 @@ fn audio_thread(
|
||||
tracing::warn!("punktfunk/1 audio requires Linux or Windows — session continues without it");
|
||||
}
|
||||
|
||||
/// Advance the intra-refresh wave position and decide whether this emitted AU is a wave boundary
|
||||
/// that should carry [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT).
|
||||
///
|
||||
/// `ir_wave_pos` counts frames since the last IDR/wave start; a real IDR re-phases it to 0 (an IDR
|
||||
/// restarts the encoder's wave AND is itself a clean anchor, so it is never additionally marked).
|
||||
/// Every `period`-th non-IDR AU is a boundary — the client lifts its post-loss freeze on the SECOND
|
||||
/// such mark. Pure so the marking cadence is unit-tested without a GPU (see the pump's use in the
|
||||
/// encode-poll loop).
|
||||
fn mark_recovery_boundary(ir_wave_pos: &mut u32, is_keyframe: bool, period: u32) -> bool {
|
||||
if is_keyframe {
|
||||
*ir_wave_pos = 0;
|
||||
false
|
||||
} else {
|
||||
*ir_wave_pos += 1;
|
||||
if *ir_wave_pos >= period {
|
||||
*ir_wave_pos = 0;
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn synthetic_stream(
|
||||
session: &mut Session,
|
||||
frames: u32,
|
||||
@@ -3396,6 +3437,9 @@ struct SessionContext {
|
||||
reconfig: std::sync::mpsc::Receiver<punktfunk_core::Mode>,
|
||||
/// Client decode-recovery keyframe requests.
|
||||
keyframe: std::sync::mpsc::Receiver<()>,
|
||||
/// Client LTR-RFI recovery requests — the lost-frame range `(first, last)`. The encode loop
|
||||
/// prefers `Encoder::invalidate_ref_frames` over a full IDR when the encoder supports it.
|
||||
rfi: std::sync::mpsc::Receiver<(u32, u32)>,
|
||||
/// Accepted mid-stream bitrate changes (adaptive bitrate, already clamped) — the encoder
|
||||
/// alone is rebuilt in place at the new rate; capture + virtual output are untouched.
|
||||
bitrate_rx: std::sync::mpsc::Receiver<u32>,
|
||||
@@ -3467,6 +3511,7 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
|
||||
quit,
|
||||
reconfig,
|
||||
keyframe,
|
||||
rfi,
|
||||
bitrate_rx,
|
||||
compositor,
|
||||
mut bitrate_kbps,
|
||||
@@ -3684,6 +3729,11 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
|
||||
// Self-diagnosis for the periodic-stutter class: warns when the served recovery IDRs settle
|
||||
// into a stable multi-second rhythm (see [`crate::metronome::Metronome`]).
|
||||
let mut recovery_cadence = crate::metronome::Metronome::new();
|
||||
// Position within the current intra-refresh wave (frames since the last IDR/wave start). Only
|
||||
// meaningful on a `caps().intra_refresh_recovery` encoder; the pump tags every wave-boundary AU
|
||||
// with `USER_FLAG_RECOVERY_POINT` so the client can lift its post-loss freeze on a clean
|
||||
// re-anchor without a full IDR. Re-phased to 0 at each emitted IDR (which restarts the wave).
|
||||
let mut ir_wave_pos: u32 = 0;
|
||||
// Per-stage latency breakdown (PUNKTFUNK_PERF): per-call µs for the GPU-bound stages so we see
|
||||
// exactly where the capture→encoded latency goes — cap=try_latest (ring read + colour convert),
|
||||
// submit=encode_picture launch, wait=lock_bitstream (the scheduling wait + ASIC encode, the one
|
||||
@@ -3900,6 +3950,33 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
|
||||
while keyframe.try_recv().is_ok() {
|
||||
want_kf = true;
|
||||
}
|
||||
// Client LTR-RFI recovery: prefer re-referencing a known-good older frame (a clean recovery
|
||||
// P-frame — no 20-40× IDR spike) over a full keyframe when the encoder supports it (native
|
||||
// AMF LTR / Windows NVENC). Drain the backlog (the client re-requests until the recovery
|
||||
// frame lands) coalesced to the widest lost range. Attempt the invalidate only when a full
|
||||
// IDR isn't already queued — an explicit keyframe request means a fully wedged decoder that
|
||||
// needs the IDR, which supersedes an RFI recovery. A failure (range older than the encoder's
|
||||
// live references, or no RFI backend) falls through to the coalesced keyframe path below.
|
||||
let mut rfi_range: Option<(u32, u32)> = None;
|
||||
while let Ok((first, last)) = rfi.try_recv() {
|
||||
rfi_range = Some(match rfi_range {
|
||||
Some((pf, pl)) => (pf.min(first), pl.max(last)),
|
||||
None => (first, last),
|
||||
});
|
||||
}
|
||||
if !want_kf {
|
||||
if let Some((first, last)) = rfi_range {
|
||||
if enc.caps().supports_rfi && enc.invalidate_ref_frames(first as i64, last as i64) {
|
||||
// The RFI recovered the loss with a clean re-anchor P-frame (no IDR). Anchor the
|
||||
// keyframe cooldown so the client's echo of the SAME loss — its frames_dropped-
|
||||
// driven keyframe request, arriving ~one loss-window later — is coalesced away
|
||||
// instead of emitting a redundant full IDR right after the cheap recovery.
|
||||
last_forced_idr = Some(std::time::Instant::now());
|
||||
} else {
|
||||
want_kf = true; // range too old / no RFI backend → coalesced keyframe below
|
||||
}
|
||||
}
|
||||
}
|
||||
if want_kf {
|
||||
// Clients request a keyframe on EVERY FEC-unrecoverable frame (`frames_dropped` polling)
|
||||
// and keep asking until the IDR actually arrives + decodes — a full round-trip on a link
|
||||
@@ -4225,11 +4302,28 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
|
||||
last_au_at = std::time::Instant::now();
|
||||
encoder_resets = 0;
|
||||
let (cap_ns, sub_ns, deadline) = inflight.pop_front().expect("inflight non-empty");
|
||||
let flags = if au.keyframe {
|
||||
let mut flags = if au.keyframe {
|
||||
(FLAG_PIC | FLAG_SOF) as u32
|
||||
} else {
|
||||
FLAG_PIC as u32
|
||||
};
|
||||
// Intra-refresh recovery marking (inert unless the backend validated its constrained GDR
|
||||
// via `intra_refresh_recovery`): tag every wave-boundary AU with USER_FLAG_RECOVERY_POINT
|
||||
// so the client lifts its post-loss freeze on the second mark — a proven clean re-anchor —
|
||||
// instead of forcing a full IDR. See [`mark_recovery_boundary`] for the cadence.
|
||||
let caps = enc.caps();
|
||||
if caps.intra_refresh_recovery
|
||||
&& caps.intra_refresh_period > 0
|
||||
&& mark_recovery_boundary(&mut ir_wave_pos, au.keyframe, caps.intra_refresh_period)
|
||||
{
|
||||
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_POINT;
|
||||
}
|
||||
// Reference-frame-invalidation recovery frame (AMD LTR force-reference): a clean P-frame
|
||||
// off a known-good reference. Tag it so the client lifts its post-loss freeze on this one
|
||||
// AU without an IDR — the definitive single-frame re-anchor (see USER_FLAG_RECOVERY_ANCHOR).
|
||||
if au.recovery_anchor {
|
||||
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR;
|
||||
}
|
||||
// Re-send the HDR mastering metadata (0xCE) on each keyframe (a decoder-resync point) and
|
||||
// whenever it changed, so a client that dropped the best-effort datagram re-converges.
|
||||
if let Some(m) = last_hdr_meta {
|
||||
@@ -4654,6 +4748,32 @@ mod tests {
|
||||
assert!(reconfig_allowed(None, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recovery_marks_land_every_period_and_rephase_at_idr() {
|
||||
let period = 4;
|
||||
let mut pos = 0u32;
|
||||
// Frames 1..=3 are mid-wave (no mark), frame 4 is the boundary; then it repeats.
|
||||
let marks: Vec<bool> = (0..10)
|
||||
.map(|_| mark_recovery_boundary(&mut pos, false, period))
|
||||
.collect();
|
||||
assert_eq!(
|
||||
marks,
|
||||
vec![false, false, false, true, false, false, false, true, false, false]
|
||||
);
|
||||
|
||||
// An IDR mid-wave re-phases: the counter restarts, so the next boundary is a full period
|
||||
// later (an IDR is itself a clean anchor, so it is not additionally marked).
|
||||
let mut pos = 0u32;
|
||||
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1
|
||||
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2
|
||||
assert!(!mark_recovery_boundary(&mut pos, true, period)); // IDR → pos 0, no mark
|
||||
// Now a fresh full period is needed, not just the 2 remaining frames.
|
||||
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1
|
||||
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2
|
||||
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 3
|
||||
assert!(mark_recovery_boundary(&mut pos, false, period)); // pos 4 → mark
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pad_snapshot_replaces_state_and_seq_gates() {
|
||||
use punktfunk_core::input::{gamepad, GamepadSnapshot};
|
||||
|
||||
Reference in New Issue
Block a user