feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike. Client (pf-client-core): after a reference loss the hardware decoder conceals the missing-reference deltas (on RADV, a gray plate with new motion painted over) and returns Ok. The pump now freezes on the last good picture until a clean re-anchor instead of showing the concealment — lifting on a real IDR, an intra-refresh recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The frame_index gap is the early, precise loss signal and drives an RFI request. Host recovery signals (inert unless the backend supports them): - USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR). - USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame. AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the only clean-recovery lever is user LTR: mark frames as long-term references, and on loss force the next frame to re-reference the newest known-good one — a clean P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR before the loss; a range older than the live slots falls back to a keyframe. Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi(). Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success anchors the keyframe cooldown so the client's frames_dropped echo of the same loss is coalesced away rather than emitting a redundant IDR. Spike: synthetic NV12 GPU source for headless AMF encoder testing. Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests (incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux; punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90 forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -104,6 +104,81 @@ pub struct Stats {
|
||||
/// IDR (or a mid-GOP join) unfreezes almost immediately instead of never.
|
||||
const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3;
|
||||
|
||||
/// Longest the pump holds the last good frame waiting for a post-loss re-anchor keyframe before it
|
||||
/// gives up and resumes display. After a reference loss the hardware decoder does not error — it
|
||||
/// conceals the reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as
|
||||
/// a gray plate with the new frame's motion painted over it) and returns Ok, so displaying them is
|
||||
/// the "gray frames mid-stream" artifact. We instead freeze on the last good picture until a fresh
|
||||
/// IDR re-anchors decode — the behaviour NVIDIA already shows (its DISTINCT output image + different
|
||||
/// concealment reads as a brief freeze, not gray). This cap only bounds the freeze when recovery
|
||||
/// genuinely stalls (host ignores the request, or an RFI recovery that never emits a keyframe), so a
|
||||
/// glitch can never become a permanent freeze. A recovery IDR round-trips well under this on any
|
||||
/// live link.
|
||||
const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500);
|
||||
|
||||
/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
|
||||
/// latest frame gap before the pump lifts its freeze on an IDR-free stream. TWO, not one: with a
|
||||
/// continuous rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a
|
||||
/// loss is only partially healed — stripes swept BEFORE the loss still reference the lost frame — and
|
||||
/// lifting there would flash a partially-stale picture. The SECOND boundary guarantees a full wave
|
||||
/// swept entirely after the loss, so the picture is clean. This stays correct under repeated loss
|
||||
/// because every new gap resets the count. The cost is up to ~2 wave periods of holding the last good
|
||||
/// frame — the deliberate "hold longer, never show garbage" trade.
|
||||
///
|
||||
/// [`USER_FLAG_RECOVERY_POINT`]: punktfunk_core::packet::USER_FLAG_RECOVERY_POINT
|
||||
const REANCHOR_MARKS_TO_LIFT: u32 = 2;
|
||||
|
||||
/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark
|
||||
/// pushes the freeze deadline out by this much, so a live mark stream (the host actively healing via
|
||||
/// its wave) keeps the client patiently holding the last good frame instead of tripping the IDR
|
||||
/// floor mid-heal. Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the
|
||||
/// marks STOP (heal stalled, or the host isn't running intra-refresh) the deadline lapses and the
|
||||
/// normal recovery-IDR floor fires, so a real stall still recovers.
|
||||
const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500);
|
||||
|
||||
/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is
|
||||
/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters
|
||||
/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small
|
||||
/// positive delta is a forward gap (missing frames whose dependents will decode against absent
|
||||
/// references); a delta in the top half is an index behind us.
|
||||
fn index_gap(expected: u32, got: u32) -> Option<u32> {
|
||||
let ahead = got.wrapping_sub(expected);
|
||||
(ahead != 0 && ahead < u32::MAX / 2).then_some(ahead)
|
||||
}
|
||||
|
||||
/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze.
|
||||
///
|
||||
/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried
|
||||
/// [`USER_FLAG_RECOVERY_ANCHOR`](punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR), the host's
|
||||
/// definitive single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a
|
||||
/// known-good reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait.
|
||||
/// `has_mark` — this AU carried [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
|
||||
/// a host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks
|
||||
/// seen since the latest gap.
|
||||
///
|
||||
/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0
|
||||
/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested
|
||||
/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only
|
||||
/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and
|
||||
/// lifts immediately.
|
||||
fn reanchor_after_frame(
|
||||
is_keyframe: bool,
|
||||
has_anchor: bool,
|
||||
has_mark: bool,
|
||||
marks: u32,
|
||||
) -> (bool, u32) {
|
||||
let marks = if has_mark {
|
||||
marks.saturating_add(1)
|
||||
} else {
|
||||
marks
|
||||
};
|
||||
if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT {
|
||||
(true, 0)
|
||||
} else {
|
||||
(false, marks)
|
||||
}
|
||||
}
|
||||
|
||||
/// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs).
|
||||
/// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old
|
||||
/// host (no 0xCF at all) this just caps the dead-weight ring.
|
||||
@@ -319,6 +394,20 @@ fn pump(
|
||||
// never drops, so the drop-count trigger below stays silent and the stream freezes
|
||||
// on the last good frame. A short streak forces a fresh IDR to re-anchor.
|
||||
let mut no_output_streak = 0u32;
|
||||
// Freeze-until-reanchor: armed the moment we request a recovery keyframe (loss, decode error, or
|
||||
// a no-output streak), it withholds the decoder's concealed frames from the presenter — which
|
||||
// then redraws the last good picture — until a fresh keyframe re-anchors decode. See
|
||||
// [`REANCHOR_FREEZE_MAX`] for why this exists and its backstop deadline.
|
||||
let mut awaiting_reanchor = false;
|
||||
let mut reanchor_deadline: Option<Instant> = None;
|
||||
// Host intra-refresh recovery marks seen since the latest gap (see [`REANCHOR_MARKS_TO_LIFT`]).
|
||||
// Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks.
|
||||
let mut recovery_marks: u32 = 0;
|
||||
// The frame_index we expect next (the host numbers frames consecutively). A jump means a frame
|
||||
// went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms
|
||||
// ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of
|
||||
// the loss window, by which point the concealment already reached the screen).
|
||||
let mut next_expected_index: Option<u32> = None;
|
||||
|
||||
let end: Option<String> = loop {
|
||||
if stop.load(Ordering::SeqCst) {
|
||||
@@ -334,9 +423,90 @@ fn pump(
|
||||
// fps / goodput count every received AU (spec), decoded or not.
|
||||
frames_n += 1;
|
||||
bytes_n += frame.data.len() as u64;
|
||||
// Reference-continuity gate: the host numbers frames consecutively, so a jump in
|
||||
// frame_index means a frame is missing (lost, or an out-of-order straggler the
|
||||
// reassembler emitted a newer frame ahead of) and this AU references a picture we
|
||||
// never decoded. On RADV the decoder conceals that as a gray plate with the new
|
||||
// motion on top — the reported artifact, and it shows most on high-motion frames (a
|
||||
// full-screen pan bursts far more packets than a static desktop or a UFO-test's small
|
||||
// moving sprite, so it is the frame that loses shards). Arm the freeze at the FIRST
|
||||
// such frame — ~120 ms before `frames_dropped` would — so the gray never reaches the
|
||||
// screen; recovery IDRs stay on the existing throttled path (see the arm below).
|
||||
match next_expected_index {
|
||||
Some(exp) if frame.frame_index == exp => {
|
||||
next_expected_index = Some(exp.wrapping_add(1)); // contiguous
|
||||
}
|
||||
// A forward gap: hold the last good frame — but DO NOT ask for a keyframe here.
|
||||
// Hiding the concealment is free (the presenter redraws the last picture); an IDR
|
||||
// is not — at 4K120 it is a multi-megabyte frame and a visible stutter, and it can
|
||||
// re-trigger the very burst loss that caused this. The existing loss recovery below
|
||||
// (`frames_dropped`, host-coalesced + throttled) still requests it at exactly the
|
||||
// cadence it did before this change, so we add zero IDR pressure per pan. A
|
||||
// straggler behind us (`index_gap` → None) leaves the expectation put so the real
|
||||
// gap still trips.
|
||||
Some(exp) => {
|
||||
if let Some(gap) = index_gap(exp, frame.frame_index) {
|
||||
let now = Instant::now();
|
||||
awaiting_reanchor = true;
|
||||
recovery_marks = 0;
|
||||
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
|
||||
next_expected_index = Some(frame.frame_index.wrapping_add(1));
|
||||
// The gap carries the PRECISE lost range — [first missing, newest
|
||||
// received - 1] — so this is the one recovery signal that can drive true
|
||||
// reference-frame invalidation. Prefer an RFI request over a keyframe: an
|
||||
// RFI-capable host (AMD LTR / NVENC) re-references a known-good picture and
|
||||
// emits a clean P-frame tagged USER_FLAG_RECOVERY_ANCHOR (the freeze lifts
|
||||
// on ONE frame, no 20-40× IDR spike); an incapable/old host forces a
|
||||
// host-coalesced IDR instead, or ignores it (then the frames_dropped /
|
||||
// overdue keyframe paths below are the backstop). Throttled with those
|
||||
// paths (one recovery ask per 100 ms) so a burst of gaps — a full-screen
|
||||
// pan shedding shards — can't storm the control stream. This fires ~120 ms
|
||||
// before frames_dropped would, so recovery also starts sooner.
|
||||
if last_kf_req
|
||||
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
|
||||
{
|
||||
last_kf_req = Some(now);
|
||||
let _ = connector
|
||||
.request_rfi(exp, frame.frame_index.wrapping_sub(1));
|
||||
}
|
||||
tracing::trace!(gap, "frame gap — RFI recovery, holding last frame until re-anchor");
|
||||
}
|
||||
}
|
||||
None => next_expected_index = Some(frame.frame_index.wrapping_add(1)),
|
||||
}
|
||||
match decoder.decode(&frame.data) {
|
||||
Ok(Some(image)) => {
|
||||
no_output_streak = 0; // a decoded frame — the anchor holds
|
||||
// Host-signalled intra-refresh recovery mark: on an IDR-free intra-refresh
|
||||
// stream this wave-boundary flag is the only clean point the client can honor
|
||||
// (the decoder never flags the re-anchor — the coded frame stays `P`). A live
|
||||
// mark stream also means the host is actively healing, so push the backstop out
|
||||
// rather than trip a mid-heal IDR (see `RECOVERY_MARK_PATIENCE`).
|
||||
let has_mark =
|
||||
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_POINT != 0;
|
||||
// The host's definitive single-frame re-anchor: an LTR-RFI recovery frame (a
|
||||
// clean P-frame off a known-good reference), the AMD twin of an IDR re-anchor
|
||||
// but without the spike. It lifts on the FIRST occurrence.
|
||||
let has_anchor =
|
||||
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR != 0;
|
||||
if has_mark && awaiting_reanchor {
|
||||
reanchor_deadline = Some(Instant::now() + RECOVERY_MARK_PATIENCE);
|
||||
}
|
||||
// A fresh clean re-anchor lifts the freeze and shows this frame: a real intra
|
||||
// keyframe (IDR, always clean), an LTR-RFI recovery anchor (also whole), OR the
|
||||
// second recovery mark since the gap (the first wave boundary is only
|
||||
// half-healed — see `reanchor_after_frame`).
|
||||
let (lift, marks) = reanchor_after_frame(
|
||||
image.is_keyframe(),
|
||||
has_anchor,
|
||||
has_mark,
|
||||
recovery_marks,
|
||||
);
|
||||
recovery_marks = marks;
|
||||
if lift {
|
||||
awaiting_reanchor = false;
|
||||
reanchor_deadline = None;
|
||||
}
|
||||
total_frames += 1;
|
||||
dec_path = match &image {
|
||||
DecodedImage::Cpu(_) => "software",
|
||||
@@ -391,11 +561,20 @@ fn pump(
|
||||
DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)),
|
||||
_ => None,
|
||||
};
|
||||
let _ = frame_tx.force_send(DecodedFrame {
|
||||
pts_ns: frame.pts_ns,
|
||||
decoded_ns,
|
||||
image,
|
||||
});
|
||||
if awaiting_reanchor {
|
||||
// Post-loss concealment: withhold this frame (it references a lost/gray
|
||||
// reference) so the presenter keeps redrawing the last good picture
|
||||
// rather than flashing the decoder's gray plate. Dropped here — the
|
||||
// hw-decode stat below still samples via `hw_fence` (raw handle + value,
|
||||
// valid past the guard). Cleared by the next keyframe or the backstop.
|
||||
tracing::trace!("holding last frame — awaiting post-loss re-anchor");
|
||||
} else {
|
||||
let _ = frame_tx.force_send(DecodedFrame {
|
||||
pts_ns: frame.pts_ns,
|
||||
decoded_ns,
|
||||
image,
|
||||
});
|
||||
}
|
||||
// `decode` stage: received→decode COMPLETE, single clock.
|
||||
match hw_fence {
|
||||
Some((sem, value)) => {
|
||||
@@ -424,6 +603,12 @@ fn pump(
|
||||
// trip before asking again instead of flooding.
|
||||
if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK {
|
||||
let now = Instant::now();
|
||||
// Wedged on missing references: hold the last good frame until re-anchor
|
||||
// (armed even when the IDR request itself is throttled — the stream is broken
|
||||
// regardless of whether we ask again this iteration).
|
||||
awaiting_reanchor = true;
|
||||
recovery_marks = 0;
|
||||
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
|
||||
if last_kf_req
|
||||
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
|
||||
{
|
||||
@@ -451,6 +636,9 @@ fn pump(
|
||||
// through the same throttle as loss recovery below.
|
||||
if decoder.take_keyframe_request() {
|
||||
let now = Instant::now();
|
||||
awaiting_reanchor = true;
|
||||
recovery_marks = 0;
|
||||
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
|
||||
if last_kf_req
|
||||
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
|
||||
{
|
||||
@@ -487,12 +675,33 @@ fn pump(
|
||||
if dropped > last_dropped {
|
||||
last_dropped = dropped;
|
||||
let now = Instant::now();
|
||||
// A dropped AU means the frames after it reference a picture we never decoded — the
|
||||
// decoder will conceal them (gray on RADV). Freeze on the last good frame until a fresh
|
||||
// IDR re-anchors, so the concealment never reaches the screen.
|
||||
awaiting_reanchor = true;
|
||||
recovery_marks = 0;
|
||||
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
|
||||
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
|
||||
last_kf_req = Some(now);
|
||||
let _ = connector.request_keyframe();
|
||||
tracing::debug!(dropped, "requested keyframe (loss recovery)");
|
||||
}
|
||||
}
|
||||
// Re-anchor overdue: the freeze has held the whole window with no keyframe — a lost recovery
|
||||
// IDR, or a benign reorder that produced no `frames_dropped` and so requested none. Do NOT
|
||||
// resume to gray (the one thing worse than a freeze): keep holding the last good frame and
|
||||
// (re-)request a keyframe, throttled + host-coalesced, so a CLEAN re-anchor is what un-freezes
|
||||
// us. A genuinely dead stream — host gone, link collapsed — is caught by the QUIC idle-timeout
|
||||
// watchdog (returns to the menu), never by painting the decoder's concealment.
|
||||
if awaiting_reanchor && reanchor_deadline.is_some_and(|d| Instant::now() >= d) {
|
||||
let now = Instant::now();
|
||||
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
|
||||
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
|
||||
last_kf_req = Some(now);
|
||||
let _ = connector.request_keyframe();
|
||||
tracing::debug!("re-anchor overdue — still holding, re-requesting keyframe");
|
||||
}
|
||||
}
|
||||
|
||||
if window_start.elapsed() >= Duration::from_secs(1) {
|
||||
let secs = window_start.elapsed().as_secs_f32();
|
||||
@@ -614,3 +823,107 @@ fn spawn_audio(
|
||||
.map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled"))
|
||||
.ok()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{index_gap, reanchor_after_frame, REANCHOR_MARKS_TO_LIFT};
|
||||
|
||||
// Simulate the pump's re-anchor state across a sequence of decoded frames: each `(is_keyframe,
|
||||
// has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based)
|
||||
// at which the freeze first lifts, or `None` if it never does. `gap_before` reset points model a
|
||||
// fresh loss re-arming the freeze (the pump zeroes the count at every gap/arm site).
|
||||
fn lift_at(frames: &[(bool, bool)]) -> Option<usize> {
|
||||
let mut marks = 0u32;
|
||||
for (i, &(is_kf, has_mark)) in frames.iter().enumerate() {
|
||||
// The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised
|
||||
// by `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here.
|
||||
let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks);
|
||||
marks = m;
|
||||
if lift {
|
||||
return Some(i);
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_single_recovery_mark_does_not_lift() {
|
||||
// The first wave boundary after a loss is only half-healed — one mark must hold the freeze.
|
||||
assert_eq!(REANCHOR_MARKS_TO_LIFT, 2);
|
||||
assert_eq!(lift_at(&[(false, true)]), None);
|
||||
assert_eq!(lift_at(&[(false, false), (false, true), (false, false)]), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_second_recovery_mark_lifts() {
|
||||
// Two marks = a full wave swept after the loss → clean re-anchor.
|
||||
assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1));
|
||||
assert_eq!(
|
||||
lift_at(&[(false, false), (false, true), (false, false), (false, true)]),
|
||||
Some(3)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_real_keyframe_lifts_immediately() {
|
||||
// An IDR is always a clean anchor — no marks needed.
|
||||
assert_eq!(lift_at(&[(true, false)]), Some(0));
|
||||
assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_fresh_gap_resets_the_mark_count() {
|
||||
// The pump zeroes `recovery_marks` at each arm site, so one mark before a new gap plus one
|
||||
// after must NOT lift — the model resets the running count to imitate that.
|
||||
let mut marks = 0u32;
|
||||
let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap)
|
||||
marks = m;
|
||||
assert_eq!(marks, 1);
|
||||
marks = 0; // a new gap re-arms the freeze → count reset
|
||||
let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave
|
||||
assert!(!lift, "a single post-gap mark must not lift");
|
||||
assert_eq!(m, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_rfi_anchor_lifts_immediately() {
|
||||
// An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good
|
||||
// reference), so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait.
|
||||
let (lift, marks) = reanchor_after_frame(false, true, false, 0);
|
||||
assert!(lift, "an RFI anchor must lift the freeze immediately");
|
||||
assert_eq!(marks, 0, "a lift resets the running mark count");
|
||||
// Even with zero prior marks and no keyframe, the anchor alone is sufficient.
|
||||
let (lift, _) = reanchor_after_frame(false, true, true, 1);
|
||||
assert!(lift, "an anchor lifts regardless of the pending mark count");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn contiguous_indices_are_not_a_gap() {
|
||||
assert_eq!(index_gap(5, 5), None);
|
||||
assert_eq!(index_gap(0, 0), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_forward_jump_reports_the_skip_count() {
|
||||
assert_eq!(index_gap(5, 6), Some(1)); // one frame missing
|
||||
assert_eq!(index_gap(5, 9), Some(4));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_straggler_behind_us_is_not_a_gap() {
|
||||
// The reassembler emitted a newer frame first; the late one must not re-arm.
|
||||
assert_eq!(index_gap(9, 5), None);
|
||||
assert_eq!(index_gap(1, 0), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_index_counter_wraps_cleanly() {
|
||||
// last frame = u32::MAX, so the next expected wraps to 0.
|
||||
assert_eq!(index_gap(0, 0), None); // contiguous across the wrap
|
||||
// waiting on u32::MAX, frame 0 arrived → MAX was skipped.
|
||||
assert_eq!(index_gap(u32::MAX, 0), Some(1));
|
||||
assert_eq!(index_gap(u32::MAX, 2), Some(3));
|
||||
// an old frame arriving just after the wrap is still a straggler.
|
||||
assert_eq!(index_gap(0, u32::MAX), None);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -98,6 +98,10 @@ pub struct VkVideoFrame {
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I): the stream's re-anchor point. The pump resumes display on
|
||||
/// one after suppressing the concealed frames a reference loss leaves in its wake (on
|
||||
/// RADV a lost reference decodes to a gray plate with the new motion painted on top).
|
||||
pub keyframe: bool,
|
||||
/// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive
|
||||
/// until the presenter's fence proves the GPU reads done — same mechanism as the
|
||||
/// VAAPI path's DRM guard.
|
||||
@@ -143,6 +147,44 @@ impl ColorDesc {
|
||||
}
|
||||
}
|
||||
|
||||
/// True if the decoder tagged this frame as a full IDR keyframe — a guaranteed clean re-anchor
|
||||
/// after which the picture is loss-free, so the pump can lift a post-loss display freeze here.
|
||||
///
|
||||
/// Keys off `AV_FRAME_FLAG_KEY` (with `pict_type == I` as a belt for decoders that fill pict_type
|
||||
/// but not the flag). NOTE: FFmpeg's H.264/HEVC decode layer sets this flag **only for true IDR
|
||||
/// frames**, never for an *intra-refresh recovery point*. H.264 flags key only when a picture's
|
||||
/// `recovery_frame_cnt == 0` (a moving band uses `> 0`); HEVC clears the flag on every non-IRAP
|
||||
/// frame regardless of the recovery-point SEI. So an intra-refresh host (NVENC/AMF/QSV) heals the
|
||||
/// picture over N P-frames with no decoded frame ever flagged key — this function cannot detect
|
||||
/// that clean point, and the pump would freeze until the `REANCHOR_FREEZE_MAX` backstop (in
|
||||
/// `session.rs`) forces a real IDR. Detecting an intra-refresh re-anchor requires an out-of-band
|
||||
/// host wire signal on the AU that completes the wave; that is not yet plumbed.
|
||||
///
|
||||
/// # Safety
|
||||
/// `frame` must point to a valid `AVFrame` alive for the duration of the call.
|
||||
pub unsafe fn frame_is_keyframe(frame: *const ffmpeg::ffi::AVFrame) -> bool {
|
||||
// SAFETY: caller guarantees a live AVFrame; plain field reads.
|
||||
unsafe {
|
||||
((*frame).flags & ffmpeg::ffi::AV_FRAME_FLAG_KEY) != 0
|
||||
|| (*frame).pict_type == ffmpeg::ffi::AVPictureType::AV_PICTURE_TYPE_I
|
||||
}
|
||||
}
|
||||
|
||||
impl DecodedImage {
|
||||
/// Whether the frame is an intra keyframe — see [`frame_is_keyframe`]. The pump uses
|
||||
/// this as the stream's re-anchor signal after a loss.
|
||||
pub fn is_keyframe(&self) -> bool {
|
||||
match self {
|
||||
DecodedImage::Cpu(f) => f.keyframe,
|
||||
#[cfg(target_os = "linux")]
|
||||
DecodedImage::Dmabuf(f) => f.keyframe,
|
||||
DecodedImage::VkFrame(f) => f.keyframe,
|
||||
#[cfg(windows)]
|
||||
DecodedImage::D3d11(f) => f.keyframe,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The Y′CbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant
|
||||
/// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient
|
||||
/// implementation every presenter derives its CSC from (Vulkan push constants, the Windows
|
||||
@@ -205,6 +247,8 @@ pub struct CpuFrame {
|
||||
/// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries
|
||||
/// baked in — the presenter tags the texture so GTK tone-maps it.
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
|
||||
pub keyframe: bool,
|
||||
}
|
||||
|
||||
/// A decoded frame still on the GPU: dmabuf fds + plane layout for
|
||||
@@ -222,6 +266,8 @@ pub struct DmabufFrame {
|
||||
/// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709
|
||||
/// narrow for SDR, BT.2020 PQ for an HDR stream).
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
|
||||
pub keyframe: bool,
|
||||
pub guard: DrmFrameGuard,
|
||||
}
|
||||
|
||||
@@ -644,6 +690,9 @@ impl SoftwareDecoder {
|
||||
stride: dst_linesize[0] as usize,
|
||||
rgba,
|
||||
color,
|
||||
// `is_key()` reads the same intra flag `frame_is_keyframe` derives from pict_type
|
||||
// for the hardware paths; ffmpeg-next handles the FFmpeg-version binding split.
|
||||
keyframe: frame.is_key(),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -844,6 +893,7 @@ impl VaapiDecoder {
|
||||
// SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after
|
||||
// this returns); plain CICP field reads.
|
||||
color: ColorDesc::from_raw(self.frame),
|
||||
keyframe: frame_is_keyframe(self.frame),
|
||||
guard,
|
||||
})
|
||||
}
|
||||
@@ -1363,6 +1413,7 @@ impl VulkanDecoder {
|
||||
width: (*self.frame).width as u32,
|
||||
height: (*self.frame).height as u32,
|
||||
color: ColorDesc::from_raw(self.frame),
|
||||
keyframe: frame_is_keyframe(self.frame),
|
||||
guard: DrmFrameGuard(clone),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -99,6 +99,9 @@ pub struct D3d11Frame {
|
||||
/// BT.709 full-range RGB — regardless of the stream's own CICP (a PQ stream was
|
||||
/// tone-mapped). The presenter keys SDR/HDR handling off this, so it always reads SDR.
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See
|
||||
/// `crate::video::VkVideoFrame`.
|
||||
pub keyframe: bool,
|
||||
/// The ring slot's NT shared handle (`IDXGIResource1::CreateSharedHandle`), stable for the
|
||||
/// ring's lifetime. Raw `isize` so the frame crosses the pump→presenter channel.
|
||||
pub handle: isize,
|
||||
@@ -692,6 +695,8 @@ impl D3d11vaDecoder {
|
||||
matrix: 0, // identity — RGB
|
||||
full_range: true,
|
||||
},
|
||||
// SAFETY: `self.frame` is the live decoded AVFrame for this call.
|
||||
keyframe: crate::video::frame_is_keyframe(self.frame),
|
||||
handle,
|
||||
generation,
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user