feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s

Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets
AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike.

Client (pf-client-core): after a reference loss the hardware decoder conceals the
missing-reference deltas (on RADV, a gray plate with new motion painted over) and
returns Ok. The pump now freezes on the last good picture until a clean re-anchor
instead of showing the concealment — lifting on a real IDR, an intra-refresh
recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The
frame_index gap is the early, precise loss signal and drives an RFI request.

Host recovery signals (inert unless the backend supports them):
- USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR).
- USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame.

AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API
has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the
only clean-recovery lever is user LTR: mark frames as long-term references, and on
loss force the next frame to re-reference the newest known-good one — a clean
P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for
AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR
before the loss; a range older than the live slots falls back to a keyframe.

Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi().
Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success
anchors the keyframe cooldown so the client's frames_dropped echo of the same loss
is coalesced away rather than emitting a redundant IDR.

Spike: synthetic NV12 GPU source for headless AMF encoder testing.

Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests
(incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux;
punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90
forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-11 17:31:17 +02:00
parent 890c7531d8
commit e55ff1bb28
21 changed files with 1228 additions and 33 deletions
+318 -5
View File
@@ -104,6 +104,81 @@ pub struct Stats {
/// IDR (or a mid-GOP join) unfreezes almost immediately instead of never.
const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3;
/// Longest the pump holds the last good frame waiting for a post-loss re-anchor keyframe before it
/// gives up and resumes display. After a reference loss the hardware decoder does not error — it
/// conceals the reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as
/// a gray plate with the new frame's motion painted over it) and returns Ok, so displaying them is
/// the "gray frames mid-stream" artifact. We instead freeze on the last good picture until a fresh
/// IDR re-anchors decode — the behaviour NVIDIA already shows (its DISTINCT output image + different
/// concealment reads as a brief freeze, not gray). This cap only bounds the freeze when recovery
/// genuinely stalls (host ignores the request, or an RFI recovery that never emits a keyframe), so a
/// glitch can never become a permanent freeze. A recovery IDR round-trips well under this on any
/// live link.
const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500);
/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
/// latest frame gap before the pump lifts its freeze on an IDR-free stream. TWO, not one: with a
/// continuous rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a
/// loss is only partially healed — stripes swept BEFORE the loss still reference the lost frame — and
/// lifting there would flash a partially-stale picture. The SECOND boundary guarantees a full wave
/// swept entirely after the loss, so the picture is clean. This stays correct under repeated loss
/// because every new gap resets the count. The cost is up to ~2 wave periods of holding the last good
/// frame — the deliberate "hold longer, never show garbage" trade.
///
/// [`USER_FLAG_RECOVERY_POINT`]: punktfunk_core::packet::USER_FLAG_RECOVERY_POINT
const REANCHOR_MARKS_TO_LIFT: u32 = 2;
/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark
/// pushes the freeze deadline out by this much, so a live mark stream (the host actively healing via
/// its wave) keeps the client patiently holding the last good frame instead of tripping the IDR
/// floor mid-heal. Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the
/// marks STOP (heal stalled, or the host isn't running intra-refresh) the deadline lapses and the
/// normal recovery-IDR floor fires, so a real stall still recovers.
const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500);
/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is
/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters
/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small
/// positive delta is a forward gap (missing frames whose dependents will decode against absent
/// references); a delta in the top half is an index behind us.
fn index_gap(expected: u32, got: u32) -> Option<u32> {
let ahead = got.wrapping_sub(expected);
(ahead != 0 && ahead < u32::MAX / 2).then_some(ahead)
}
/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze.
///
/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried
/// [`USER_FLAG_RECOVERY_ANCHOR`](punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR), the host's
/// definitive single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a
/// known-good reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait.
/// `has_mark` — this AU carried [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// a host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks
/// seen since the latest gap.
///
/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0
/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested
/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only
/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and
/// lifts immediately.
fn reanchor_after_frame(
is_keyframe: bool,
has_anchor: bool,
has_mark: bool,
marks: u32,
) -> (bool, u32) {
let marks = if has_mark {
marks.saturating_add(1)
} else {
marks
};
if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT {
(true, 0)
} else {
(false, marks)
}
}
/// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs).
/// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old
/// host (no 0xCF at all) this just caps the dead-weight ring.
@@ -319,6 +394,20 @@ fn pump(
// never drops, so the drop-count trigger below stays silent and the stream freezes
// on the last good frame. A short streak forces a fresh IDR to re-anchor.
let mut no_output_streak = 0u32;
// Freeze-until-reanchor: armed the moment we request a recovery keyframe (loss, decode error, or
// a no-output streak), it withholds the decoder's concealed frames from the presenter — which
// then redraws the last good picture — until a fresh keyframe re-anchors decode. See
// [`REANCHOR_FREEZE_MAX`] for why this exists and its backstop deadline.
let mut awaiting_reanchor = false;
let mut reanchor_deadline: Option<Instant> = None;
// Host intra-refresh recovery marks seen since the latest gap (see [`REANCHOR_MARKS_TO_LIFT`]).
// Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks.
let mut recovery_marks: u32 = 0;
// The frame_index we expect next (the host numbers frames consecutively). A jump means a frame
// went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms
// ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of
// the loss window, by which point the concealment already reached the screen).
let mut next_expected_index: Option<u32> = None;
let end: Option<String> = loop {
if stop.load(Ordering::SeqCst) {
@@ -334,9 +423,90 @@ fn pump(
// fps / goodput count every received AU (spec), decoded or not.
frames_n += 1;
bytes_n += frame.data.len() as u64;
// Reference-continuity gate: the host numbers frames consecutively, so a jump in
// frame_index means a frame is missing (lost, or an out-of-order straggler the
// reassembler emitted a newer frame ahead of) and this AU references a picture we
// never decoded. On RADV the decoder conceals that as a gray plate with the new
// motion on top — the reported artifact, and it shows most on high-motion frames (a
// full-screen pan bursts far more packets than a static desktop or a UFO-test's small
// moving sprite, so it is the frame that loses shards). Arm the freeze at the FIRST
// such frame — ~120 ms before `frames_dropped` would — so the gray never reaches the
// screen; recovery IDRs stay on the existing throttled path (see the arm below).
match next_expected_index {
Some(exp) if frame.frame_index == exp => {
next_expected_index = Some(exp.wrapping_add(1)); // contiguous
}
// A forward gap: hold the last good frame — but DO NOT ask for a keyframe here.
// Hiding the concealment is free (the presenter redraws the last picture); an IDR
// is not — at 4K120 it is a multi-megabyte frame and a visible stutter, and it can
// re-trigger the very burst loss that caused this. The existing loss recovery below
// (`frames_dropped`, host-coalesced + throttled) still requests it at exactly the
// cadence it did before this change, so we add zero IDR pressure per pan. A
// straggler behind us (`index_gap` → None) leaves the expectation put so the real
// gap still trips.
Some(exp) => {
if let Some(gap) = index_gap(exp, frame.frame_index) {
let now = Instant::now();
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
next_expected_index = Some(frame.frame_index.wrapping_add(1));
// The gap carries the PRECISE lost range — [first missing, newest
// received - 1] — so this is the one recovery signal that can drive true
// reference-frame invalidation. Prefer an RFI request over a keyframe: an
// RFI-capable host (AMD LTR / NVENC) re-references a known-good picture and
// emits a clean P-frame tagged USER_FLAG_RECOVERY_ANCHOR (the freeze lifts
// on ONE frame, no 20-40× IDR spike); an incapable/old host forces a
// host-coalesced IDR instead, or ignores it (then the frames_dropped /
// overdue keyframe paths below are the backstop). Throttled with those
// paths (one recovery ask per 100 ms) so a burst of gaps — a full-screen
// pan shedding shards — can't storm the control stream. This fires ~120 ms
// before frames_dropped would, so recovery also starts sooner.
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now);
let _ = connector
.request_rfi(exp, frame.frame_index.wrapping_sub(1));
}
tracing::trace!(gap, "frame gap — RFI recovery, holding last frame until re-anchor");
}
}
None => next_expected_index = Some(frame.frame_index.wrapping_add(1)),
}
match decoder.decode(&frame.data) {
Ok(Some(image)) => {
no_output_streak = 0; // a decoded frame — the anchor holds
// Host-signalled intra-refresh recovery mark: on an IDR-free intra-refresh
// stream this wave-boundary flag is the only clean point the client can honor
// (the decoder never flags the re-anchor — the coded frame stays `P`). A live
// mark stream also means the host is actively healing, so push the backstop out
// rather than trip a mid-heal IDR (see `RECOVERY_MARK_PATIENCE`).
let has_mark =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_POINT != 0;
// The host's definitive single-frame re-anchor: an LTR-RFI recovery frame (a
// clean P-frame off a known-good reference), the AMD twin of an IDR re-anchor
// but without the spike. It lifts on the FIRST occurrence.
let has_anchor =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR != 0;
if has_mark && awaiting_reanchor {
reanchor_deadline = Some(Instant::now() + RECOVERY_MARK_PATIENCE);
}
// A fresh clean re-anchor lifts the freeze and shows this frame: a real intra
// keyframe (IDR, always clean), an LTR-RFI recovery anchor (also whole), OR the
// second recovery mark since the gap (the first wave boundary is only
// half-healed — see `reanchor_after_frame`).
let (lift, marks) = reanchor_after_frame(
image.is_keyframe(),
has_anchor,
has_mark,
recovery_marks,
);
recovery_marks = marks;
if lift {
awaiting_reanchor = false;
reanchor_deadline = None;
}
total_frames += 1;
dec_path = match &image {
DecodedImage::Cpu(_) => "software",
@@ -391,11 +561,20 @@ fn pump(
DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)),
_ => None,
};
let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns,
decoded_ns,
image,
});
if awaiting_reanchor {
// Post-loss concealment: withhold this frame (it references a lost/gray
// reference) so the presenter keeps redrawing the last good picture
// rather than flashing the decoder's gray plate. Dropped here — the
// hw-decode stat below still samples via `hw_fence` (raw handle + value,
// valid past the guard). Cleared by the next keyframe or the backstop.
tracing::trace!("holding last frame — awaiting post-loss re-anchor");
} else {
let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns,
decoded_ns,
image,
});
}
// `decode` stage: received→decode COMPLETE, single clock.
match hw_fence {
Some((sem, value)) => {
@@ -424,6 +603,12 @@ fn pump(
// trip before asking again instead of flooding.
if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK {
let now = Instant::now();
// Wedged on missing references: hold the last good frame until re-anchor
// (armed even when the IDR request itself is throttled — the stream is broken
// regardless of whether we ask again this iteration).
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
@@ -451,6 +636,9 @@ fn pump(
// through the same throttle as loss recovery below.
if decoder.take_keyframe_request() {
let now = Instant::now();
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
@@ -487,12 +675,33 @@ fn pump(
if dropped > last_dropped {
last_dropped = dropped;
let now = Instant::now();
// A dropped AU means the frames after it reference a picture we never decoded — the
// decoder will conceal them (gray on RADV). Freeze on the last good frame until a fresh
// IDR re-anchors, so the concealment never reaches the screen.
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!(dropped, "requested keyframe (loss recovery)");
}
}
// Re-anchor overdue: the freeze has held the whole window with no keyframe — a lost recovery
// IDR, or a benign reorder that produced no `frames_dropped` and so requested none. Do NOT
// resume to gray (the one thing worse than a freeze): keep holding the last good frame and
// (re-)request a keyframe, throttled + host-coalesced, so a CLEAN re-anchor is what un-freezes
// us. A genuinely dead stream — host gone, link collapsed — is caught by the QUIC idle-timeout
// watchdog (returns to the menu), never by painting the decoder's concealment.
if awaiting_reanchor && reanchor_deadline.is_some_and(|d| Instant::now() >= d) {
let now = Instant::now();
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!("re-anchor overdue — still holding, re-requesting keyframe");
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32();
@@ -614,3 +823,107 @@ fn spawn_audio(
.map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled"))
.ok()
}
#[cfg(test)]
mod tests {
use super::{index_gap, reanchor_after_frame, REANCHOR_MARKS_TO_LIFT};
// Simulate the pump's re-anchor state across a sequence of decoded frames: each `(is_keyframe,
// has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based)
// at which the freeze first lifts, or `None` if it never does. `gap_before` reset points model a
// fresh loss re-arming the freeze (the pump zeroes the count at every gap/arm site).
fn lift_at(frames: &[(bool, bool)]) -> Option<usize> {
let mut marks = 0u32;
for (i, &(is_kf, has_mark)) in frames.iter().enumerate() {
// The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised
// by `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here.
let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks);
marks = m;
if lift {
return Some(i);
}
}
None
}
#[test]
fn a_single_recovery_mark_does_not_lift() {
// The first wave boundary after a loss is only half-healed — one mark must hold the freeze.
assert_eq!(REANCHOR_MARKS_TO_LIFT, 2);
assert_eq!(lift_at(&[(false, true)]), None);
assert_eq!(lift_at(&[(false, false), (false, true), (false, false)]), None);
}
#[test]
fn the_second_recovery_mark_lifts() {
// Two marks = a full wave swept after the loss → clean re-anchor.
assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1));
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false), (false, true)]),
Some(3)
);
}
#[test]
fn a_real_keyframe_lifts_immediately() {
// An IDR is always a clean anchor — no marks needed.
assert_eq!(lift_at(&[(true, false)]), Some(0));
assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1));
}
#[test]
fn a_fresh_gap_resets_the_mark_count() {
// The pump zeroes `recovery_marks` at each arm site, so one mark before a new gap plus one
// after must NOT lift — the model resets the running count to imitate that.
let mut marks = 0u32;
let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap)
marks = m;
assert_eq!(marks, 1);
marks = 0; // a new gap re-arms the freeze → count reset
let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave
assert!(!lift, "a single post-gap mark must not lift");
assert_eq!(m, 1);
}
#[test]
fn an_rfi_anchor_lifts_immediately() {
// An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good
// reference), so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait.
let (lift, marks) = reanchor_after_frame(false, true, false, 0);
assert!(lift, "an RFI anchor must lift the freeze immediately");
assert_eq!(marks, 0, "a lift resets the running mark count");
// Even with zero prior marks and no keyframe, the anchor alone is sufficient.
let (lift, _) = reanchor_after_frame(false, true, true, 1);
assert!(lift, "an anchor lifts regardless of the pending mark count");
}
#[test]
fn contiguous_indices_are_not_a_gap() {
assert_eq!(index_gap(5, 5), None);
assert_eq!(index_gap(0, 0), None);
}
#[test]
fn a_forward_jump_reports_the_skip_count() {
assert_eq!(index_gap(5, 6), Some(1)); // one frame missing
assert_eq!(index_gap(5, 9), Some(4));
}
#[test]
fn a_straggler_behind_us_is_not_a_gap() {
// The reassembler emitted a newer frame first; the late one must not re-arm.
assert_eq!(index_gap(9, 5), None);
assert_eq!(index_gap(1, 0), None);
}
#[test]
fn the_index_counter_wraps_cleanly() {
// last frame = u32::MAX, so the next expected wraps to 0.
assert_eq!(index_gap(0, 0), None); // contiguous across the wrap
// waiting on u32::MAX, frame 0 arrived → MAX was skipped.
assert_eq!(index_gap(u32::MAX, 0), Some(1));
assert_eq!(index_gap(u32::MAX, 2), Some(3));
// an old frame arriving just after the wrap is still a straggler.
assert_eq!(index_gap(0, u32::MAX), None);
}
}
+51
View File
@@ -98,6 +98,10 @@ pub struct VkVideoFrame {
pub width: u32,
pub height: u32,
pub color: ColorDesc,
/// Intra keyframe (IDR/I): the stream's re-anchor point. The pump resumes display on
/// one after suppressing the concealed frames a reference loss leaves in its wake (on
/// RADV a lost reference decodes to a gray plate with the new motion painted on top).
pub keyframe: bool,
/// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive
/// until the presenter's fence proves the GPU reads done — same mechanism as the
/// VAAPI path's DRM guard.
@@ -143,6 +147,44 @@ impl ColorDesc {
}
}
/// True if the decoder tagged this frame as a full IDR keyframe — a guaranteed clean re-anchor
/// after which the picture is loss-free, so the pump can lift a post-loss display freeze here.
///
/// Keys off `AV_FRAME_FLAG_KEY` (with `pict_type == I` as a belt for decoders that fill pict_type
/// but not the flag). NOTE: FFmpeg's H.264/HEVC decode layer sets this flag **only for true IDR
/// frames**, never for an *intra-refresh recovery point*. H.264 flags key only when a picture's
/// `recovery_frame_cnt == 0` (a moving band uses `> 0`); HEVC clears the flag on every non-IRAP
/// frame regardless of the recovery-point SEI. So an intra-refresh host (NVENC/AMF/QSV) heals the
/// picture over N P-frames with no decoded frame ever flagged key — this function cannot detect
/// that clean point, and the pump would freeze until the `REANCHOR_FREEZE_MAX` backstop (in
/// `session.rs`) forces a real IDR. Detecting an intra-refresh re-anchor requires an out-of-band
/// host wire signal on the AU that completes the wave; that is not yet plumbed.
///
/// # Safety
/// `frame` must point to a valid `AVFrame` alive for the duration of the call.
pub unsafe fn frame_is_keyframe(frame: *const ffmpeg::ffi::AVFrame) -> bool {
// SAFETY: caller guarantees a live AVFrame; plain field reads.
unsafe {
((*frame).flags & ffmpeg::ffi::AV_FRAME_FLAG_KEY) != 0
|| (*frame).pict_type == ffmpeg::ffi::AVPictureType::AV_PICTURE_TYPE_I
}
}
impl DecodedImage {
/// Whether the frame is an intra keyframe — see [`frame_is_keyframe`]. The pump uses
/// this as the stream's re-anchor signal after a loss.
pub fn is_keyframe(&self) -> bool {
match self {
DecodedImage::Cpu(f) => f.keyframe,
#[cfg(target_os = "linux")]
DecodedImage::Dmabuf(f) => f.keyframe,
DecodedImage::VkFrame(f) => f.keyframe,
#[cfg(windows)]
DecodedImage::D3d11(f) => f.keyframe,
}
}
}
/// The YCbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant
/// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient
/// implementation every presenter derives its CSC from (Vulkan push constants, the Windows
@@ -205,6 +247,8 @@ pub struct CpuFrame {
/// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries
/// baked in — the presenter tags the texture so GTK tone-maps it.
pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
pub keyframe: bool,
}
/// A decoded frame still on the GPU: dmabuf fds + plane layout for
@@ -222,6 +266,8 @@ pub struct DmabufFrame {
/// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709
/// narrow for SDR, BT.2020 PQ for an HDR stream).
pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
pub keyframe: bool,
pub guard: DrmFrameGuard,
}
@@ -644,6 +690,9 @@ impl SoftwareDecoder {
stride: dst_linesize[0] as usize,
rgba,
color,
// `is_key()` reads the same intra flag `frame_is_keyframe` derives from pict_type
// for the hardware paths; ffmpeg-next handles the FFmpeg-version binding split.
keyframe: frame.is_key(),
})
}
}
@@ -844,6 +893,7 @@ impl VaapiDecoder {
// SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after
// this returns); plain CICP field reads.
color: ColorDesc::from_raw(self.frame),
keyframe: frame_is_keyframe(self.frame),
guard,
})
}
@@ -1363,6 +1413,7 @@ impl VulkanDecoder {
width: (*self.frame).width as u32,
height: (*self.frame).height as u32,
color: ColorDesc::from_raw(self.frame),
keyframe: frame_is_keyframe(self.frame),
guard: DrmFrameGuard(clone),
})
}
+5
View File
@@ -99,6 +99,9 @@ pub struct D3d11Frame {
/// BT.709 full-range RGB — regardless of the stream's own CICP (a PQ stream was
/// tone-mapped). The presenter keys SDR/HDR handling off this, so it always reads SDR.
pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See
/// `crate::video::VkVideoFrame`.
pub keyframe: bool,
/// The ring slot's NT shared handle (`IDXGIResource1::CreateSharedHandle`), stable for the
/// ring's lifetime. Raw `isize` so the frame crosses the pump→presenter channel.
pub handle: isize,
@@ -692,6 +695,8 @@ impl D3d11vaDecoder {
matrix: 0, // identity — RGB
full_range: true,
},
// SAFETY: `self.frame` is the live decoded AVFrame for this call.
keyframe: crate::video::frame_is_keyframe(self.frame),
handle,
generation,
})