From e55ff1bb287ee6e73e1fa64bbe63f9394c0640a0 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Sat, 11 Jul 2026 17:31:17 +0200 Subject: [PATCH] =?UTF-8?q?feat(recovery):=20clean=20mid-stream=20loss=20r?= =?UTF-8?q?ecovery=20=E2=80=94=20freeze-until-reanchor=20+=20AMD=20LTR-RFI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike. Client (pf-client-core): after a reference loss the hardware decoder conceals the missing-reference deltas (on RADV, a gray plate with new motion painted over) and returns Ok. The pump now freezes on the last good picture until a clean re-anchor instead of showing the concealment — lifting on a real IDR, an intra-refresh recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The frame_index gap is the early, precise loss signal and drives an RFI request. Host recovery signals (inert unless the backend supports them): - USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR). - USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame. AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the only clean-recovery lever is user LTR: mark frames as long-term references, and on loss force the next frame to re-reference the newest known-good one — a clean P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR before the loss; a range older than the live slots falls back to a keyframe. Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi(). Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success anchors the keyframe cooldown so the client's frames_dropped echo of the same loss is coalesced away rather than emitting a redundant IDR. Spike: synthetic NV12 GPU source for headless AMF encoder testing. Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests (incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux; punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90 forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR). Co-Authored-By: Claude Opus 4.8 --- crates/pf-client-core/src/session.rs | 323 ++++++++++++++++- crates/pf-client-core/src/video.rs | 51 +++ crates/pf-client-core/src/video_d3d11.rs | 5 + crates/punktfunk-core/src/client.rs | 26 +- crates/punktfunk-core/src/packet.rs | 19 + crates/punktfunk-core/src/quic/msgs.rs | 42 +++ crates/punktfunk-core/src/quic/tests.rs | 29 ++ crates/punktfunk-host/src/capture.rs | 3 + .../src/capture/windows/synthetic_nv12.rs | 181 +++++++++ crates/punktfunk-host/src/encode.rs | 33 +- crates/punktfunk-host/src/encode/linux/mod.rs | 16 + .../punktfunk-host/src/encode/linux/vaapi.rs | 1 + crates/punktfunk-host/src/encode/sw.rs | 1 + .../punktfunk-host/src/encode/windows/amf.rs | 342 +++++++++++++++++- .../src/encode/windows/ffmpeg_win.rs | 1 + .../src/encode/windows/nvenc.rs | 4 + crates/punktfunk-host/src/main.rs | 6 +- crates/punktfunk-host/src/pipeline.rs | 4 + crates/punktfunk-host/src/punktfunk1.rs | 124 ++++++- crates/punktfunk-host/src/spike.rs | 26 ++ include/punktfunk_core.h | 24 ++ 21 files changed, 1228 insertions(+), 33 deletions(-) create mode 100644 crates/punktfunk-host/src/capture/windows/synthetic_nv12.rs diff --git a/crates/pf-client-core/src/session.rs b/crates/pf-client-core/src/session.rs index 9e92cd9d..defd2f1c 100644 --- a/crates/pf-client-core/src/session.rs +++ b/crates/pf-client-core/src/session.rs @@ -104,6 +104,81 @@ pub struct Stats { /// IDR (or a mid-GOP join) unfreezes almost immediately instead of never. const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3; +/// Longest the pump holds the last good frame waiting for a post-loss re-anchor keyframe before it +/// gives up and resumes display. After a reference loss the hardware decoder does not error — it +/// conceals the reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as +/// a gray plate with the new frame's motion painted over it) and returns Ok, so displaying them is +/// the "gray frames mid-stream" artifact. We instead freeze on the last good picture until a fresh +/// IDR re-anchors decode — the behaviour NVIDIA already shows (its DISTINCT output image + different +/// concealment reads as a brief freeze, not gray). This cap only bounds the freeze when recovery +/// genuinely stalls (host ignores the request, or an RFI recovery that never emits a keyframe), so a +/// glitch can never become a permanent freeze. A recovery IDR round-trips well under this on any +/// live link. +const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500); + +/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the +/// latest frame gap before the pump lifts its freeze on an IDR-free stream. TWO, not one: with a +/// continuous rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a +/// loss is only partially healed — stripes swept BEFORE the loss still reference the lost frame — and +/// lifting there would flash a partially-stale picture. The SECOND boundary guarantees a full wave +/// swept entirely after the loss, so the picture is clean. This stays correct under repeated loss +/// because every new gap resets the count. The cost is up to ~2 wave periods of holding the last good +/// frame — the deliberate "hold longer, never show garbage" trade. +/// +/// [`USER_FLAG_RECOVERY_POINT`]: punktfunk_core::packet::USER_FLAG_RECOVERY_POINT +const REANCHOR_MARKS_TO_LIFT: u32 = 2; + +/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark +/// pushes the freeze deadline out by this much, so a live mark stream (the host actively healing via +/// its wave) keeps the client patiently holding the last good frame instead of tripping the IDR +/// floor mid-heal. Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the +/// marks STOP (heal stalled, or the host isn't running intra-refresh) the deadline lapses and the +/// normal recovery-IDR floor fires, so a real stall still recovers. +const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500); + +/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is +/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters +/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small +/// positive delta is a forward gap (missing frames whose dependents will decode against absent +/// references); a delta in the top half is an index behind us. +fn index_gap(expected: u32, got: u32) -> Option { + let ahead = got.wrapping_sub(expected); + (ahead != 0 && ahead < u32::MAX / 2).then_some(ahead) +} + +/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze. +/// +/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried +/// [`USER_FLAG_RECOVERY_ANCHOR`](punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR), the host's +/// definitive single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a +/// known-good reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait. +/// `has_mark` — this AU carried [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT), +/// a host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks +/// seen since the latest gap. +/// +/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0 +/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested +/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only +/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and +/// lifts immediately. +fn reanchor_after_frame( + is_keyframe: bool, + has_anchor: bool, + has_mark: bool, + marks: u32, +) -> (bool, u32) { + let marks = if has_mark { + marks.saturating_add(1) + } else { + marks + }; + if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT { + (true, 0) + } else { + (false, marks) + } +} + /// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs). /// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old /// host (no 0xCF at all) this just caps the dead-weight ring. @@ -319,6 +394,20 @@ fn pump( // never drops, so the drop-count trigger below stays silent and the stream freezes // on the last good frame. A short streak forces a fresh IDR to re-anchor. let mut no_output_streak = 0u32; + // Freeze-until-reanchor: armed the moment we request a recovery keyframe (loss, decode error, or + // a no-output streak), it withholds the decoder's concealed frames from the presenter — which + // then redraws the last good picture — until a fresh keyframe re-anchors decode. See + // [`REANCHOR_FREEZE_MAX`] for why this exists and its backstop deadline. + let mut awaiting_reanchor = false; + let mut reanchor_deadline: Option = None; + // Host intra-refresh recovery marks seen since the latest gap (see [`REANCHOR_MARKS_TO_LIFT`]). + // Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks. + let mut recovery_marks: u32 = 0; + // The frame_index we expect next (the host numbers frames consecutively). A jump means a frame + // went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms + // ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of + // the loss window, by which point the concealment already reached the screen). + let mut next_expected_index: Option = None; let end: Option = loop { if stop.load(Ordering::SeqCst) { @@ -334,9 +423,90 @@ fn pump( // fps / goodput count every received AU (spec), decoded or not. frames_n += 1; bytes_n += frame.data.len() as u64; + // Reference-continuity gate: the host numbers frames consecutively, so a jump in + // frame_index means a frame is missing (lost, or an out-of-order straggler the + // reassembler emitted a newer frame ahead of) and this AU references a picture we + // never decoded. On RADV the decoder conceals that as a gray plate with the new + // motion on top — the reported artifact, and it shows most on high-motion frames (a + // full-screen pan bursts far more packets than a static desktop or a UFO-test's small + // moving sprite, so it is the frame that loses shards). Arm the freeze at the FIRST + // such frame — ~120 ms before `frames_dropped` would — so the gray never reaches the + // screen; recovery IDRs stay on the existing throttled path (see the arm below). + match next_expected_index { + Some(exp) if frame.frame_index == exp => { + next_expected_index = Some(exp.wrapping_add(1)); // contiguous + } + // A forward gap: hold the last good frame — but DO NOT ask for a keyframe here. + // Hiding the concealment is free (the presenter redraws the last picture); an IDR + // is not — at 4K120 it is a multi-megabyte frame and a visible stutter, and it can + // re-trigger the very burst loss that caused this. The existing loss recovery below + // (`frames_dropped`, host-coalesced + throttled) still requests it at exactly the + // cadence it did before this change, so we add zero IDR pressure per pan. A + // straggler behind us (`index_gap` → None) leaves the expectation put so the real + // gap still trips. + Some(exp) => { + if let Some(gap) = index_gap(exp, frame.frame_index) { + let now = Instant::now(); + awaiting_reanchor = true; + recovery_marks = 0; + reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX); + next_expected_index = Some(frame.frame_index.wrapping_add(1)); + // The gap carries the PRECISE lost range — [first missing, newest + // received - 1] — so this is the one recovery signal that can drive true + // reference-frame invalidation. Prefer an RFI request over a keyframe: an + // RFI-capable host (AMD LTR / NVENC) re-references a known-good picture and + // emits a clean P-frame tagged USER_FLAG_RECOVERY_ANCHOR (the freeze lifts + // on ONE frame, no 20-40× IDR spike); an incapable/old host forces a + // host-coalesced IDR instead, or ignores it (then the frames_dropped / + // overdue keyframe paths below are the backstop). Throttled with those + // paths (one recovery ask per 100 ms) so a burst of gaps — a full-screen + // pan shedding shards — can't storm the control stream. This fires ~120 ms + // before frames_dropped would, so recovery also starts sooner. + if last_kf_req + .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) + { + last_kf_req = Some(now); + let _ = connector + .request_rfi(exp, frame.frame_index.wrapping_sub(1)); + } + tracing::trace!(gap, "frame gap — RFI recovery, holding last frame until re-anchor"); + } + } + None => next_expected_index = Some(frame.frame_index.wrapping_add(1)), + } match decoder.decode(&frame.data) { Ok(Some(image)) => { no_output_streak = 0; // a decoded frame — the anchor holds + // Host-signalled intra-refresh recovery mark: on an IDR-free intra-refresh + // stream this wave-boundary flag is the only clean point the client can honor + // (the decoder never flags the re-anchor — the coded frame stays `P`). A live + // mark stream also means the host is actively healing, so push the backstop out + // rather than trip a mid-heal IDR (see `RECOVERY_MARK_PATIENCE`). + let has_mark = + frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_POINT != 0; + // The host's definitive single-frame re-anchor: an LTR-RFI recovery frame (a + // clean P-frame off a known-good reference), the AMD twin of an IDR re-anchor + // but without the spike. It lifts on the FIRST occurrence. + let has_anchor = + frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR != 0; + if has_mark && awaiting_reanchor { + reanchor_deadline = Some(Instant::now() + RECOVERY_MARK_PATIENCE); + } + // A fresh clean re-anchor lifts the freeze and shows this frame: a real intra + // keyframe (IDR, always clean), an LTR-RFI recovery anchor (also whole), OR the + // second recovery mark since the gap (the first wave boundary is only + // half-healed — see `reanchor_after_frame`). + let (lift, marks) = reanchor_after_frame( + image.is_keyframe(), + has_anchor, + has_mark, + recovery_marks, + ); + recovery_marks = marks; + if lift { + awaiting_reanchor = false; + reanchor_deadline = None; + } total_frames += 1; dec_path = match &image { DecodedImage::Cpu(_) => "software", @@ -391,11 +561,20 @@ fn pump( DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)), _ => None, }; - let _ = frame_tx.force_send(DecodedFrame { - pts_ns: frame.pts_ns, - decoded_ns, - image, - }); + if awaiting_reanchor { + // Post-loss concealment: withhold this frame (it references a lost/gray + // reference) so the presenter keeps redrawing the last good picture + // rather than flashing the decoder's gray plate. Dropped here — the + // hw-decode stat below still samples via `hw_fence` (raw handle + value, + // valid past the guard). Cleared by the next keyframe or the backstop. + tracing::trace!("holding last frame — awaiting post-loss re-anchor"); + } else { + let _ = frame_tx.force_send(DecodedFrame { + pts_ns: frame.pts_ns, + decoded_ns, + image, + }); + } // `decode` stage: received→decode COMPLETE, single clock. match hw_fence { Some((sem, value)) => { @@ -424,6 +603,12 @@ fn pump( // trip before asking again instead of flooding. if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK { let now = Instant::now(); + // Wedged on missing references: hold the last good frame until re-anchor + // (armed even when the IDR request itself is throttled — the stream is broken + // regardless of whether we ask again this iteration). + awaiting_reanchor = true; + recovery_marks = 0; + reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX); if last_kf_req .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { @@ -451,6 +636,9 @@ fn pump( // through the same throttle as loss recovery below. if decoder.take_keyframe_request() { let now = Instant::now(); + awaiting_reanchor = true; + recovery_marks = 0; + reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX); if last_kf_req .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { @@ -487,12 +675,33 @@ fn pump( if dropped > last_dropped { last_dropped = dropped; let now = Instant::now(); + // A dropped AU means the frames after it reference a picture we never decoded — the + // decoder will conceal them (gray on RADV). Freeze on the last good frame until a fresh + // IDR re-anchors, so the concealment never reaches the screen. + awaiting_reanchor = true; + recovery_marks = 0; + reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX); if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { last_kf_req = Some(now); let _ = connector.request_keyframe(); tracing::debug!(dropped, "requested keyframe (loss recovery)"); } } + // Re-anchor overdue: the freeze has held the whole window with no keyframe — a lost recovery + // IDR, or a benign reorder that produced no `frames_dropped` and so requested none. Do NOT + // resume to gray (the one thing worse than a freeze): keep holding the last good frame and + // (re-)request a keyframe, throttled + host-coalesced, so a CLEAN re-anchor is what un-freezes + // us. A genuinely dead stream — host gone, link collapsed — is caught by the QUIC idle-timeout + // watchdog (returns to the menu), never by painting the decoder's concealment. + if awaiting_reanchor && reanchor_deadline.is_some_and(|d| Instant::now() >= d) { + let now = Instant::now(); + reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX); + if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { + last_kf_req = Some(now); + let _ = connector.request_keyframe(); + tracing::debug!("re-anchor overdue — still holding, re-requesting keyframe"); + } + } if window_start.elapsed() >= Duration::from_secs(1) { let secs = window_start.elapsed().as_secs_f32(); @@ -614,3 +823,107 @@ fn spawn_audio( .map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled")) .ok() } + +#[cfg(test)] +mod tests { + use super::{index_gap, reanchor_after_frame, REANCHOR_MARKS_TO_LIFT}; + + // Simulate the pump's re-anchor state across a sequence of decoded frames: each `(is_keyframe, + // has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based) + // at which the freeze first lifts, or `None` if it never does. `gap_before` reset points model a + // fresh loss re-arming the freeze (the pump zeroes the count at every gap/arm site). + fn lift_at(frames: &[(bool, bool)]) -> Option { + let mut marks = 0u32; + for (i, &(is_kf, has_mark)) in frames.iter().enumerate() { + // The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised + // by `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here. + let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks); + marks = m; + if lift { + return Some(i); + } + } + None + } + + #[test] + fn a_single_recovery_mark_does_not_lift() { + // The first wave boundary after a loss is only half-healed — one mark must hold the freeze. + assert_eq!(REANCHOR_MARKS_TO_LIFT, 2); + assert_eq!(lift_at(&[(false, true)]), None); + assert_eq!(lift_at(&[(false, false), (false, true), (false, false)]), None); + } + + #[test] + fn the_second_recovery_mark_lifts() { + // Two marks = a full wave swept after the loss → clean re-anchor. + assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1)); + assert_eq!( + lift_at(&[(false, false), (false, true), (false, false), (false, true)]), + Some(3) + ); + } + + #[test] + fn a_real_keyframe_lifts_immediately() { + // An IDR is always a clean anchor — no marks needed. + assert_eq!(lift_at(&[(true, false)]), Some(0)); + assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1)); + } + + #[test] + fn a_fresh_gap_resets_the_mark_count() { + // The pump zeroes `recovery_marks` at each arm site, so one mark before a new gap plus one + // after must NOT lift — the model resets the running count to imitate that. + let mut marks = 0u32; + let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap) + marks = m; + assert_eq!(marks, 1); + marks = 0; // a new gap re-arms the freeze → count reset + let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave + assert!(!lift, "a single post-gap mark must not lift"); + assert_eq!(m, 1); + } + + #[test] + fn an_rfi_anchor_lifts_immediately() { + // An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good + // reference), so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait. + let (lift, marks) = reanchor_after_frame(false, true, false, 0); + assert!(lift, "an RFI anchor must lift the freeze immediately"); + assert_eq!(marks, 0, "a lift resets the running mark count"); + // Even with zero prior marks and no keyframe, the anchor alone is sufficient. + let (lift, _) = reanchor_after_frame(false, true, true, 1); + assert!(lift, "an anchor lifts regardless of the pending mark count"); + } + + #[test] + fn contiguous_indices_are_not_a_gap() { + assert_eq!(index_gap(5, 5), None); + assert_eq!(index_gap(0, 0), None); + } + + #[test] + fn a_forward_jump_reports_the_skip_count() { + assert_eq!(index_gap(5, 6), Some(1)); // one frame missing + assert_eq!(index_gap(5, 9), Some(4)); + } + + #[test] + fn a_straggler_behind_us_is_not_a_gap() { + // The reassembler emitted a newer frame first; the late one must not re-arm. + assert_eq!(index_gap(9, 5), None); + assert_eq!(index_gap(1, 0), None); + } + + #[test] + fn the_index_counter_wraps_cleanly() { + // last frame = u32::MAX, so the next expected wraps to 0. + assert_eq!(index_gap(0, 0), None); // contiguous across the wrap + // waiting on u32::MAX, frame 0 arrived → MAX was skipped. + assert_eq!(index_gap(u32::MAX, 0), Some(1)); + assert_eq!(index_gap(u32::MAX, 2), Some(3)); + // an old frame arriving just after the wrap is still a straggler. + assert_eq!(index_gap(0, u32::MAX), None); + } +} diff --git a/crates/pf-client-core/src/video.rs b/crates/pf-client-core/src/video.rs index 968761ba..da0103dd 100644 --- a/crates/pf-client-core/src/video.rs +++ b/crates/pf-client-core/src/video.rs @@ -98,6 +98,10 @@ pub struct VkVideoFrame { pub width: u32, pub height: u32, pub color: ColorDesc, + /// Intra keyframe (IDR/I): the stream's re-anchor point. The pump resumes display on + /// one after suppressing the concealed frames a reference loss leaves in its wake (on + /// RADV a lost reference decodes to a gray plate with the new motion painted on top). + pub keyframe: bool, /// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive /// until the presenter's fence proves the GPU reads done — same mechanism as the /// VAAPI path's DRM guard. @@ -143,6 +147,44 @@ impl ColorDesc { } } +/// True if the decoder tagged this frame as a full IDR keyframe — a guaranteed clean re-anchor +/// after which the picture is loss-free, so the pump can lift a post-loss display freeze here. +/// +/// Keys off `AV_FRAME_FLAG_KEY` (with `pict_type == I` as a belt for decoders that fill pict_type +/// but not the flag). NOTE: FFmpeg's H.264/HEVC decode layer sets this flag **only for true IDR +/// frames**, never for an *intra-refresh recovery point*. H.264 flags key only when a picture's +/// `recovery_frame_cnt == 0` (a moving band uses `> 0`); HEVC clears the flag on every non-IRAP +/// frame regardless of the recovery-point SEI. So an intra-refresh host (NVENC/AMF/QSV) heals the +/// picture over N P-frames with no decoded frame ever flagged key — this function cannot detect +/// that clean point, and the pump would freeze until the `REANCHOR_FREEZE_MAX` backstop (in +/// `session.rs`) forces a real IDR. Detecting an intra-refresh re-anchor requires an out-of-band +/// host wire signal on the AU that completes the wave; that is not yet plumbed. +/// +/// # Safety +/// `frame` must point to a valid `AVFrame` alive for the duration of the call. +pub unsafe fn frame_is_keyframe(frame: *const ffmpeg::ffi::AVFrame) -> bool { + // SAFETY: caller guarantees a live AVFrame; plain field reads. + unsafe { + ((*frame).flags & ffmpeg::ffi::AV_FRAME_FLAG_KEY) != 0 + || (*frame).pict_type == ffmpeg::ffi::AVPictureType::AV_PICTURE_TYPE_I + } +} + +impl DecodedImage { + /// Whether the frame is an intra keyframe — see [`frame_is_keyframe`]. The pump uses + /// this as the stream's re-anchor signal after a loss. + pub fn is_keyframe(&self) -> bool { + match self { + DecodedImage::Cpu(f) => f.keyframe, + #[cfg(target_os = "linux")] + DecodedImage::Dmabuf(f) => f.keyframe, + DecodedImage::VkFrame(f) => f.keyframe, + #[cfg(windows)] + DecodedImage::D3d11(f) => f.keyframe, + } + } +} + /// The Y′CbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant /// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient /// implementation every presenter derives its CSC from (Vulkan push constants, the Windows @@ -205,6 +247,8 @@ pub struct CpuFrame { /// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries /// baked in — the presenter tags the texture so GTK tone-maps it. pub color: ColorDesc, + /// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`]. + pub keyframe: bool, } /// A decoded frame still on the GPU: dmabuf fds + plane layout for @@ -222,6 +266,8 @@ pub struct DmabufFrame { /// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709 /// narrow for SDR, BT.2020 PQ for an HDR stream). pub color: ColorDesc, + /// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`]. + pub keyframe: bool, pub guard: DrmFrameGuard, } @@ -644,6 +690,9 @@ impl SoftwareDecoder { stride: dst_linesize[0] as usize, rgba, color, + // `is_key()` reads the same intra flag `frame_is_keyframe` derives from pict_type + // for the hardware paths; ffmpeg-next handles the FFmpeg-version binding split. + keyframe: frame.is_key(), }) } } @@ -844,6 +893,7 @@ impl VaapiDecoder { // SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after // this returns); plain CICP field reads. color: ColorDesc::from_raw(self.frame), + keyframe: frame_is_keyframe(self.frame), guard, }) } @@ -1363,6 +1413,7 @@ impl VulkanDecoder { width: (*self.frame).width as u32, height: (*self.frame).height as u32, color: ColorDesc::from_raw(self.frame), + keyframe: frame_is_keyframe(self.frame), guard: DrmFrameGuard(clone), }) } diff --git a/crates/pf-client-core/src/video_d3d11.rs b/crates/pf-client-core/src/video_d3d11.rs index 9e5694ee..c0f6ef06 100644 --- a/crates/pf-client-core/src/video_d3d11.rs +++ b/crates/pf-client-core/src/video_d3d11.rs @@ -99,6 +99,9 @@ pub struct D3d11Frame { /// BT.709 full-range RGB — regardless of the stream's own CICP (a PQ stream was /// tone-mapped). The presenter keys SDR/HDR handling off this, so it always reads SDR. pub color: ColorDesc, + /// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See + /// `crate::video::VkVideoFrame`. + pub keyframe: bool, /// The ring slot's NT shared handle (`IDXGIResource1::CreateSharedHandle`), stable for the /// ring's lifetime. Raw `isize` so the frame crosses the pump→presenter channel. pub handle: isize, @@ -692,6 +695,8 @@ impl D3d11vaDecoder { matrix: 0, // identity — RGB full_range: true, }, + // SAFETY: `self.frame` is the live decoded AVFrame for this call. + keyframe: crate::video::frame_is_keyframe(self.frame), handle, generation, }) diff --git a/crates/punktfunk-core/src/client.rs b/crates/punktfunk-core/src/client.rs index 0ea67267..9ce5eb2c 100644 --- a/crates/punktfunk-core/src/client.rs +++ b/crates/punktfunk-core/src/client.rs @@ -19,7 +19,8 @@ use crate::packet::FLAG_PROBE; use crate::quic::{ accept_resync, endpoint, io, wall_clock_ns, window_loss_ppm, BitrateChanged, ClockEcho, ClockResync, ColorInfo, HdrMeta, Hello, HidOutput, LossReport, ProbeRequest, ProbeResult, - Reconfigure, Reconfigured, RequestKeyframe, ResyncStep, RichInput, SetBitrate, Start, Welcome, + Reconfigure, Reconfigured, RequestKeyframe, RfiRequest, ResyncStep, RichInput, SetBitrate, Start, + Welcome, }; use crate::session::{Frame, Session}; use crate::transport::UdpTransport; @@ -49,6 +50,10 @@ enum CtrlRequest { Mode(Mode), Probe(ProbeRequest), Keyframe, + /// Reference-frame-invalidation recovery: the client saw a `frame_index` gap and reports the + /// invalidation range so an RFI-capable host re-references a known-good picture instead of + /// forcing a full IDR. See [`RfiRequest`]. + Rfi(RfiRequest), Loss(LossReport), /// Adaptive bitrate: ask the host to re-target its encoder (kbps). Sent by the pump's /// [`BitrateController`] when the user's bitrate setting is Automatic. @@ -868,6 +873,24 @@ impl NativeClient { .map_err(|_| PunktfunkError::Closed) } + /// Ask the host to recover from loss by **reference-frame invalidation** rather than a full IDR: + /// the client reports the range `[first_frame, last_frame]` of access units it can no longer trust + /// (from the first missing `frame_index` through the newest received). An RFI-capable host + /// re-references a known-good picture before `first_frame` (AMD LTR / NVENC RFI) and emits a clean + /// P-frame tagged [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`]; a host that can't RFI forces an IDR + /// instead (same as [`request_keyframe`](Self::request_keyframe)). Non-blocking, fire-and-forget — + /// the recovered frame is the only ack; throttle it like the keyframe request. Prefer this over + /// `request_keyframe` on loss so AMD/RFI hosts avoid the IDR spike; the keyframe request remains + /// the backstop when the recovery frame itself is lost. + pub fn request_rfi(&self, first_frame: u32, last_frame: u32) -> Result<()> { + self.ctrl_tx + .try_send(CtrlRequest::Rfi(RfiRequest { + first_frame, + last_frame, + })) + .map_err(|_| PunktfunkError::Closed) + } + /// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't /// rebuild them). A video loop polls this and calls [`request_keyframe`](Self::request_keyframe) /// when it increases — the correct loss trigger under infinite GOP, where unrecoverable loss @@ -1511,6 +1534,7 @@ async fn worker_main(args: WorkerArgs) { CtrlRequest::Mode(m) => Reconfigure { mode: m }.encode(), CtrlRequest::Probe(p) => p.encode(), CtrlRequest::Keyframe => RequestKeyframe.encode(), + CtrlRequest::Rfi(r) => r.encode(), CtrlRequest::Loss(r) => r.encode(), CtrlRequest::SetBitrate(k) => SetBitrate { bitrate_kbps: k }.encode(), CtrlRequest::ClockResync => { diff --git a/crates/punktfunk-core/src/packet.rs b/crates/punktfunk-core/src/packet.rs index eec38c6e..981efe6f 100644 --- a/crates/punktfunk-core/src/packet.rs +++ b/crates/punktfunk-core/src/packet.rs @@ -35,6 +35,25 @@ pub const FLAG_SOF: u8 = 0x4; /// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it). pub const FLAG_PROBE: u8 = 0x8; +/// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client +/// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that +/// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is +/// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its +/// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible +/// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host +/// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four. +pub const USER_FLAG_RECOVERY_POINT: u32 = 0x10; + +/// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike +/// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss +/// is only half-healed so the client waits for the second), this marks an access unit the host coded +/// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation** +/// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client +/// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts +/// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets +/// `AV_FRAME_FLAG_KEY` — this host flag is the only signal. +pub const USER_FLAG_RECOVERY_ANCHOR: u32 = 0x20; + /// Crypto framing overhead [`Session`](crate::session::Session) adds when encrypting: /// an 8-byte sequence prefix plus the GCM tag. pub const CRYPTO_OVERHEAD: usize = 8 + crate::crypto::TAG_LEN; diff --git a/crates/punktfunk-core/src/quic/msgs.rs b/crates/punktfunk-core/src/quic/msgs.rs index 7e16acd7..36c69d16 100644 --- a/crates/punktfunk-core/src/quic/msgs.rs +++ b/crates/punktfunk-core/src/quic/msgs.rs @@ -355,6 +355,24 @@ pub struct Reconfigured { #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct RequestKeyframe; +/// `client → host`: reference-frame-invalidation recovery — the loss-aware sibling of +/// [`RequestKeyframe`]. The client detected a `frame_index` gap and reports the range `[first_frame, +/// last_frame]` of access units it can no longer trust (from the first missing index through the +/// newest received). Instead of a full IDR (a 20-40× spike that deepens the loss it recovers), a host +/// whose encoder supports RFI re-references a known-good picture *before* `first_frame` — an AMD LTR +/// force-reference or an NVENC `nvEncInvalidateRefFrames` — emitting a single clean P-frame it tags +/// [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its freeze on it. A host that +/// can't RFI (no valid reference / libavcodec backend) forces an IDR instead, exactly as for a bare +/// [`RequestKeyframe`]; a host that predates this ignores the unknown message and the client's +/// keyframe backstop still recovers. Fire-and-forget — the recovered frame is the only ack. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct RfiRequest { + /// First access-unit `frame_index` the client can no longer trust (the gap start). + pub first_frame: u32, + /// Newest received `frame_index` at the time of the report (the invalidation range end). + pub last_frame: u32, +} + /// `client → host`, periodic: the client's observed data-plane loss, so the host can size FEC to /// the link instead of a flat percentage (adaptive FEC). `loss_ppm` is parts-per-million of shards /// that arrived missing-but-recovered (plus a bump when frames went unrecoverable) over the report @@ -467,6 +485,8 @@ pub const MSG_LOSS_REPORT: u8 = 0x04; pub const MSG_SET_BITRATE: u8 = 0x05; /// Type byte of [`BitrateChanged`]. pub const MSG_BITRATE_CHANGED: u8 = 0x06; +/// Type byte of [`RfiRequest`]. +pub const MSG_RFI_REQUEST: u8 = 0x07; /// Type byte of [`ProbeRequest`]. pub const MSG_PROBE_REQUEST: u8 = 0x20; /// Type byte of [`ProbeResult`]. @@ -1032,6 +1052,28 @@ impl RequestKeyframe { } } +impl RfiRequest { + pub fn encode(&self) -> Vec { + // magic[0..4] type[4] first_frame[5..9] last_frame[9..13] + let mut b = Vec::with_capacity(13); + b.extend_from_slice(CTL_MAGIC); + b.push(MSG_RFI_REQUEST); + b.extend_from_slice(&self.first_frame.to_le_bytes()); + b.extend_from_slice(&self.last_frame.to_le_bytes()); + b + } + + pub fn decode(b: &[u8]) -> Result { + if b.len() != 13 || &b[0..4] != CTL_MAGIC || b[4] != MSG_RFI_REQUEST { + return Err(PunktfunkError::InvalidArg("bad RfiRequest")); + } + Ok(RfiRequest { + first_frame: u32::from_le_bytes(b[5..9].try_into().unwrap()), + last_frame: u32::from_le_bytes(b[9..13].try_into().unwrap()), + }) + } +} + impl LossReport { pub fn encode(&self) -> Vec { // magic[0..4] type[4] loss_ppm[5..9] diff --git a/crates/punktfunk-core/src/quic/tests.rs b/crates/punktfunk-core/src/quic/tests.rs index dbc36240..9e09a804 100644 --- a/crates/punktfunk-core/src/quic/tests.rs +++ b/crates/punktfunk-core/src/quic/tests.rs @@ -633,6 +633,35 @@ fn request_keyframe_roundtrip() { assert!(RequestKeyframe::decode(&[bytes.as_slice(), &[0]].concat()).is_err()); } +#[test] +fn rfi_request_roundtrip() { + for (first_frame, last_frame) in [(0u32, 0u32), (40, 47), (5, 5), (1_000_000, u32::MAX)] { + let r = RfiRequest { + first_frame, + last_frame, + }; + assert_eq!(RfiRequest::decode(&r.encode()).unwrap(), r); + } + // Disjoint from the bare keyframe request (its loss-unaware sibling) and others: type byte + length. + assert!(RfiRequest::decode(&RequestKeyframe.encode()).is_err()); + assert!(RequestKeyframe::decode( + &RfiRequest { + first_frame: 1, + last_frame: 2 + } + .encode() + ) + .is_err()); + // Exact length — no trailing bytes. + let bytes = RfiRequest { + first_frame: 3, + last_frame: 9, + } + .encode(); + assert!(RfiRequest::decode(&[bytes.as_slice(), &[0]].concat()).is_err()); + assert!(RfiRequest::decode(&bytes[..bytes.len() - 1]).is_err()); +} + #[test] fn loss_report_roundtrip() { for loss_ppm in [0u32, 1, 12_345, 50_000, 1_000_000] { diff --git a/crates/punktfunk-host/src/capture.rs b/crates/punktfunk-host/src/capture.rs index b4f6ad88..39d8b698 100644 --- a/crates/punktfunk-host/src/capture.rs +++ b/crates/punktfunk-host/src/capture.rs @@ -466,5 +466,8 @@ pub mod dxgi; #[cfg(target_os = "windows")] #[path = "capture/windows/idd_push.rs"] pub mod idd_push; +#[cfg(target_os = "windows")] +#[path = "capture/windows/synthetic_nv12.rs"] +pub mod synthetic_nv12; #[cfg(target_os = "linux")] mod linux; diff --git a/crates/punktfunk-host/src/capture/windows/synthetic_nv12.rs b/crates/punktfunk-host/src/capture/windows/synthetic_nv12.rs new file mode 100644 index 00000000..c691fbce --- /dev/null +++ b/crates/punktfunk-host/src/capture/windows/synthetic_nv12.rs @@ -0,0 +1,181 @@ +//! A headless synthetic **NV12 D3D11** capture source for exercising the GPU encoders on Windows +//! without a real capture session. +//! +//! The native AMF path (and the D3D11 zero-copy NVENC/QSV paths) require an NV12 texture that lives +//! on the GPU — the CPU-Bgrx [`SyntheticCapturer`](crate::capture::SyntheticCapturer) can't provide +//! one, and DXGI Desktop Duplication can't create one under an ssh session-0 (E_ACCESSDENIED). This +//! source builds an NV12 texture on the selected render adapter and fills it with a **moving** luma +//! ramp each frame, so the encoder sees genuine motion (P-frame residuals + the intra-refresh wave +//! under content change) — exactly what an intra-refresh recovery validation needs. Driven by +//! `spike --source synthetic-nv12`. + +use crate::capture::dxgi::{make_device, D3d11Frame}; +use crate::capture::{CapturedFrame, Capturer, FramePayload, PixelFormat}; +use anyhow::{Context, Result}; +use windows::Win32::Graphics::Direct3D11::{ + ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D, D3D11_BIND_SHADER_RESOURCE, + D3D11_CPU_ACCESS_WRITE, D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_WRITE, D3D11_TEXTURE2D_DESC, + D3D11_USAGE, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING, +}; +use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_NV12, DXGI_SAMPLE_DESC}; +use windows::Win32::Graphics::Dxgi::{CreateDXGIFactory1, IDXGIAdapter1, IDXGIFactory4}; + +/// Synthetic NV12 frames on the GPU. Owns its own D3D11 device + immediate context and two NV12 +/// textures: a CPU-writable STAGING scratch it fills each frame, and a DEFAULT texture it copies +/// into and hands to the encoder. The encoder copies out of the DEFAULT texture synchronously +/// (spike drives capture→submit→poll on one thread), so reusing one DEFAULT texture is sound. +pub struct SyntheticNv12Capturer { + device: ID3D11Device, + context: ID3D11DeviceContext, + default_tex: ID3D11Texture2D, + staging: ID3D11Texture2D, + width: u32, + height: u32, + fps: u32, + frame_idx: u64, +} + +// SAFETY: mirrors `D3d11Frame`'s reasoning — the device is created free-threaded (`make_device` +// passes no `SINGLETHREADED` flag) and D3D11 uses interlocked COM refcounting, so moving the whole +// capturer (device + immediate context + textures) to its owning thread and using it only there is +// sound. The value is moved, never aliased (no `Sync`), so the single-threaded immediate context is +// never touched concurrently. +unsafe impl Send for SyntheticNv12Capturer {} + +impl SyntheticNv12Capturer { + pub fn new(width: u32, height: u32, fps: u32) -> Result { + // NV12 is 4:2:0 — both dimensions must be even (the chroma plane is width/2 × height/2). + let width = (width & !1).max(2); + let height = (height & !1).max(2); + // SAFETY: a self-contained builder owning every handle it creates; each COM call is checked + // and the returned owners drop with their wrappers. + unsafe { + let adapter = resolve_render_adapter().context("resolve render adapter for NV12 source")?; + let (device, context) = make_device(&adapter).context("create D3D11 device")?; + let default_tex = create_nv12( + &device, + width, + height, + D3D11_USAGE_DEFAULT, + 0, + D3D11_BIND_SHADER_RESOURCE.0 as u32, + ) + .context("create NV12 default texture")?; + let staging = create_nv12( + &device, + width, + height, + D3D11_USAGE_STAGING, + D3D11_CPU_ACCESS_WRITE.0 as u32, + 0, + ) + .context("create NV12 staging texture")?; + Ok(SyntheticNv12Capturer { + device, + context, + default_tex, + staging, + width, + height, + fps, + frame_idx: 0, + }) + } + } +} + +impl Capturer for SyntheticNv12Capturer { + fn next_frame(&mut self) -> Result { + let pts_ns = self.frame_idx * 1_000_000_000 / self.fps.max(1) as u64; + // SAFETY: Map/Unmap/CopyResource on this capturer's own single-threaded immediate context; + // all writes stay within the mapped NV12 surface (Y: H rows of RowPitch; UV: H/2 rows of + // RowPitch beginning at RowPitch*H — the standard NV12 plane layout). + unsafe { + let mut map = D3D11_MAPPED_SUBRESOURCE::default(); + self.context + .Map(&self.staging, 0, D3D11_MAP_WRITE, 0, Some(&mut map)) + .context("Map(NV12 staging)")?; + let pitch = map.RowPitch as usize; + let base = map.pData as *mut u8; + // A diagonal luma ramp that shifts 4 codes/frame — strong, deterministic motion. + let shift = (self.frame_idx as u32).wrapping_mul(4); + for y in 0..self.height { + let row = base.add(y as usize * pitch); + for x in 0..self.width { + *row.add(x as usize) = x.wrapping_add(y).wrapping_add(shift) as u8; + } + } + // UV plane (neutral gray = 128) at offset RowPitch*H: H/2 rows, `width` bytes each + // (width/2 interleaved Cb,Cr pairs). + let uv = base.add(pitch * self.height as usize); + for r in 0..(self.height / 2) { + let row = uv.add(r as usize * pitch); + for c in 0..self.width { + *row.add(c as usize) = 128; + } + } + self.context.Unmap(&self.staging, 0); + self.context.CopyResource(&self.default_tex, &self.staging); + } + self.frame_idx += 1; + Ok(CapturedFrame { + width: self.width, + height: self.height, + pts_ns, + format: PixelFormat::Nv12, + payload: FramePayload::D3d11(D3d11Frame { + texture: self.default_tex.clone(), + device: self.device.clone(), + }), + }) + } +} + +/// Resolve the same render adapter the encoder will pick (`PUNKTFUNK_RENDER_ADAPTER` / preference / +/// max-VRAM LUID), falling back to adapter 0. +/// +/// # Safety +/// Calls DXGI factory/adapter enumeration; returns owned COM objects or an error. +unsafe fn resolve_render_adapter() -> Result { + let factory: IDXGIFactory4 = CreateDXGIFactory1().context("CreateDXGIFactory1")?; + if let Some(luid) = crate::win_adapter::resolve_render_adapter_luid() { + if let Ok(a) = factory.EnumAdapterByLuid::(luid) { + return Ok(a); + } + } + factory.EnumAdapters1(0).context("EnumAdapters1(0)") +} + +/// Create an NV12 `Texture2D` with the given usage/CPU-access/bind flags. +/// +/// # Safety +/// `device` must be a live D3D11 device; the returned texture is owned by the caller. +unsafe fn create_nv12( + device: &ID3D11Device, + width: u32, + height: u32, + usage: D3D11_USAGE, + cpu_access: u32, + bind: u32, +) -> Result { + let desc = D3D11_TEXTURE2D_DESC { + Width: width, + Height: height, + MipLevels: 1, + ArraySize: 1, + Format: DXGI_FORMAT_NV12, + SampleDesc: DXGI_SAMPLE_DESC { + Count: 1, + Quality: 0, + }, + Usage: usage, + BindFlags: bind, + CPUAccessFlags: cpu_access, + ..Default::default() + }; + let mut tex: Option = None; + device + .CreateTexture2D(&desc, None, Some(&mut tex)) + .context("CreateTexture2D(NV12)")?; + tex.context("CreateTexture2D returned a null NV12 texture") +} diff --git a/crates/punktfunk-host/src/encode.rs b/crates/punktfunk-host/src/encode.rs index d4ee11cf..fc288ad2 100644 --- a/crates/punktfunk-host/src/encode.rs +++ b/crates/punktfunk-host/src/encode.rs @@ -19,6 +19,13 @@ pub struct EncodedFrame { pub pts_ns: u64, /// True for IDR/keyframes (sets the SOF/keyframe wire flags). pub keyframe: bool, + /// True when this AU is a **reference-frame-invalidation recovery frame** — a clean P-frame the + /// encoder coded against a known-good reference in response to + /// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) (AMD LTR force-reference). The pump + /// tags it [`punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its post-loss + /// freeze on it without an IDR. Only the native-AMF LTR path sets it; every other backend leaves + /// it `false` (their RFI, when present, re-references transparently with no distinct clean-point AU). + pub recovery_anchor: bool, } /// Codec selection negotiated with the client. @@ -208,12 +215,28 @@ pub struct EncoderCaps { /// the encoder's real chroma disagrees with what was negotiated (the in-band SPS is authoritative /// for the decoder either way). pub chroma_444: bool, - /// The encoder runs a periodic **intra-refresh wave** (a moving band of intra blocks + - /// recovery-point SEI, no periodic IDR): FEC-unrecoverable loss self-heals within one wave, so - /// the session glue rate-limits client keyframe requests instead of answering each with a full - /// IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC sets it when - /// `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/software never do. + /// The encoder runs a periodic **intra-refresh wave** — a moving band of intra blocks that + /// re-codes the whole picture over ~0.5 s, no periodic IDR. FEC-unrecoverable loss self-heals as + /// the band sweeps, so the session glue rate-limits client keyframe requests instead of answering + /// each with a full IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC / AMF + /// set it when `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/QSV/software never + /// do. NOTE — the wave carries NO decoder-visible clean-point: FFmpeg never sets `AV_FRAME_FLAG_KEY` + /// at a recovery point (H.264 flags key only when `recovery_frame_cnt == 0`; HEVC only on IRAP), + /// and AMF emits no recovery-point SEI at all. So this cap ALONE does not let the client lift its + /// post-loss freeze without an IDR — that needs [`intra_refresh_recovery`](Self::intra_refresh_recovery). pub intra_refresh: bool, + /// The intra-refresh wave is a *validated constrained GDR* — verified on real hardware to fully + /// heal a lost picture within one wave period with no residual artifacts. Only then does the host + /// tag each wave-boundary AU with [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT), + /// so the client can lift its freeze on the second mark (a proven clean re-anchor) instead of + /// waiting out its backstop and forcing a full IDR. Default `false` on every backend until on-glass + /// validation flips it — an un-validated encoder keeps the IDR recovery path, so this is inert and + /// cannot regress. Meaningless unless [`intra_refresh`](Self::intra_refresh) is also set. + pub intra_refresh_recovery: bool, + /// Length of the intra-refresh wave in frames — the boundary period the host marks on (it sets + /// `USER_FLAG_RECOVERY_POINT` on every Nth emitted AU, re-phased at each IDR). 0 when intra-refresh + /// is off. Only consulted when [`intra_refresh_recovery`](Self::intra_refresh_recovery) is set. + pub intra_refresh_period: u32, } /// A hardware encoder. One per session; runs on the encode thread. diff --git a/crates/punktfunk-host/src/encode/linux/mod.rs b/crates/punktfunk-host/src/encode/linux/mod.rs index 615d6e83..4cf950e0 100644 --- a/crates/punktfunk-host/src/encode/linux/mod.rs +++ b/crates/punktfunk-host/src/encode/linux/mod.rs @@ -177,6 +177,10 @@ pub struct NvencEncoder { /// Opened in intra-refresh mode (surfaced via [`caps`](Encoder::caps) so the session glue /// rate-limits forced IDRs — the wave heals loss without them). intra_refresh: bool, + /// Resolved wave length in frames when [`intra_refresh`](Self::intra_refresh), else 0. Cached at + /// open so the pump's per-AU `caps()` doesn't re-read `PUNKTFUNK_IR_PERIOD_FRAMES`; the pump marks + /// every Nth AU with `USER_FLAG_RECOVERY_POINT` for the client's clean re-anchor. + intra_refresh_period: u32, } // `CudaHw` holds raw `AVBufferRef`s and `sws_444` a raw `SwsContext`; the encoder lives on a single @@ -525,6 +529,11 @@ impl NvencEncoder { frame_idx: 0, force_kf: false, intra_refresh, + intra_refresh_period: if intra_refresh { + intra_refresh_period(fps).max(1) as u32 + } else { + 0 + }, }) } } @@ -536,6 +545,12 @@ impl Encoder for NvencEncoder { // convert. RFI/HDR-SEI stay unsupported on libavcodec NVENC (the trait defaults). chroma_444: self.want_444, intra_refresh: self.intra_refresh, + // NVENC intra-refresh is purpose-built GDR loss recovery (moving band + recovery-point + // SEI): the wave heals a lost picture within one period, so mark the boundary AUs and let + // the client re-anchor on them instead of forcing a full IDR. Tied to `intra_refresh` + // (already the `PUNKTFUNK_INTRA_REFRESH` opt-in), unlike AMF/QSV which stay unvalidated. + intra_refresh_recovery: self.intra_refresh, + intra_refresh_period: self.intra_refresh_period, ..super::EncoderCaps::default() } } @@ -578,6 +593,7 @@ impl Encoder for NvencEncoder { data, pts_ns, keyframe: pkt.is_key(), + recovery_anchor: false, })) } // No packet ready yet (need another input frame). diff --git a/crates/punktfunk-host/src/encode/linux/vaapi.rs b/crates/punktfunk-host/src/encode/linux/vaapi.rs index 698f6e33..62972455 100644 --- a/crates/punktfunk-host/src/encode/linux/vaapi.rs +++ b/crates/punktfunk-host/src/encode/linux/vaapi.rs @@ -294,6 +294,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result, + /// LTR-RFI recovery property names (design: the AMD twin of NVENC intra-refresh recovery). + /// `None` on AV1 — its reference management uses a frame-marking OBU mechanism this path does + /// not drive, so LTR recovery is AVC/HEVC-only. + ltr: Option, +} + +/// The four AMF LTR (long-term-reference) property names, codec-prefixed (AVC bare, HEVC `Hevc*`). +/// Two are static (`max_*`, set once at open); two are per-frame (`mark`/`force`, set on the input +/// surface each `submit`). Together they let a loss re-reference a known-good older frame — a clean +/// P-frame instead of a 20–40× IDR spike. +struct LtrProps { + /// `MaxOfLTRFrames` — number of user LTR slots (we request [`NUM_LTR_SLOTS`]). + max_ltr_frames: PCWSTR, + /// `MaxNumRefFrames` — reference-picture budget; must exceed 1 for LTR to engage. + max_num_ref_frames: PCWSTR, + /// `MarkCurrentWithLTRIndex` (per-frame) — tag the current frame as long-term reference slot N. + mark_ltr_index: PCWSTR, + /// `ForceLTRReferenceBitfield` (per-frame) — force the current frame to reference only the LTR + /// slots in the bitfield (`1< CodecProps { out_primaries: w!("OutColorPrimaries"), hdr_metadata: None, intra_refresh: Some((w!("IntraRefreshMBsNumberPerSlot"), 16)), + ltr: Some(LtrProps { + max_ltr_frames: w!("MaxOfLTRFrames"), + max_num_ref_frames: w!("MaxNumRefFrames"), + mark_ltr_index: w!("MarkCurrentWithLTRIndex"), + force_ltr_bitfield: w!("ForceLTRReferenceBitfield"), + }), }, Codec::H265 => CodecProps { component: w!("AMFVideoEncoderHW_HEVC"), @@ -716,6 +742,12 @@ fn codec_props(codec: Codec) -> CodecProps { out_primaries: w!("HevcOutColorPrimaries"), hdr_metadata: Some(w!("HevcInHDRMetadata")), intra_refresh: Some((w!("HevcIntraRefreshCTBsNumberPerSlot"), 64)), + ltr: Some(LtrProps { + max_ltr_frames: w!("HevcMaxOfLTRFrames"), + max_num_ref_frames: w!("HevcMaxNumRefFrames"), + mark_ltr_index: w!("HevcMarkCurrentWithLTRIndex"), + force_ltr_bitfield: w!("HevcForceLTRReferenceBitfield"), + }), }, Codec::Av1 => CodecProps { component: w!("AMFVideoEncoderHW_AV1"), @@ -743,6 +775,7 @@ fn codec_props(codec: Codec) -> CodecProps { out_primaries: w!("Av1OutputColorPrimaries"), hdr_metadata: Some(w!("Av1InHDRMetadata")), intra_refresh: None, + ltr: None, }, } } @@ -797,6 +830,45 @@ fn intra_refresh_period(fps: u32) -> u32 { .unwrap_or_else(|| (fps.max(16) / 2).max(2)) } +/// Number of user-controlled LTR slots. AMD exposes up to 2; two rotating slots hold a sliding pair +/// of recent long-term references, so a loss can re-reference the newest one *before* the loss point. +const NUM_LTR_SLOTS: usize = 2; + +/// AMD's real clean loss-recovery path (the NVENC-RFI twin): the encoder marks frames as long-term +/// references, and on loss forces a later frame to re-reference a known-good one — a clean P-frame, +/// not a 20-40× IDR spike. On by default when the driver supports it (AMF intra-refresh cannot heal — +/// no constrained-intra-prediction property exists in the API, header-confirmed + PSNR-proven — and +/// LTR is mutually exclusive with it, so LTR wins). `PUNKTFUNK_NO_AMF_LTR=1` forces the old full-IDR +/// recovery for debugging. +fn ltr_disabled() -> bool { + std::env::var("PUNKTFUNK_NO_AMF_LTR") + .map(|v| matches!(v.trim(), "1" | "true" | "yes" | "on")) + .unwrap_or(false) +} + +/// Cadence (frames) between LTR marks — a fresh long-term reference roughly every half second by +/// default (`PUNKTFUNK_LTR_INTERVAL_FRAMES` overrides). With [`NUM_LTR_SLOTS`] slots this keeps ~one +/// second of recent references, so a loss up to ~1 s old still has a known-good frame to force; a +/// smaller interval means the forced reference is more recent (a smaller recovery-frame residual). +fn ltr_mark_interval(fps: u32) -> i64 { + std::env::var("PUNKTFUNK_LTR_INTERVAL_FRAMES") + .ok() + .and_then(|s| s.parse::().ok()) + .filter(|v| *v >= 1) + .unwrap_or_else(|| (fps.max(2) / 2).max(1) as i64) +} + +/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N` the encoder +/// self-triggers its real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path, so a +/// headless spike run can exercise LTR recovery end-to-end (mark → force → recovery-anchor tag) +/// without a live client sending an [`RfiRequest`](punktfunk_core::quic::RfiRequest). `None` normally. +fn ltr_test_force_at() -> Option { + std::env::var("PUNKTFUNK_LTR_FORCE_AT") + .ok() + .and_then(|s| s.parse::().ok()) + .filter(|v| *v > 0) +} + // --------------------------------------------------------------------------------------------- // Owned-pointer guards (release exactly once; Terminate before Release for context/component, // mirroring amfenc.c's teardown order). @@ -930,11 +1002,12 @@ struct Inner { dctx: ID3D11DeviceContext, ring: Vec, next: usize, - /// (pts_ns, forced-IDR) per submitted-but-unretrieved frame, FIFO — the AMF encoder emits - /// AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`. Its length is - /// the count of input surfaces AMF still holds, so `submit` bounds it below [`RING`] to keep - /// the input ring from being overwritten under it. - pending: VecDeque<(u64, bool)>, + /// (pts_ns, forced-IDR, recovery-anchor) per submitted-but-unretrieved frame, FIFO — the AMF + /// encoder emits AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`. + /// The third field tags the LTR-RFI re-anchor frame so the AU carries `recovery_anchor` for the + /// client's freeze-lift. Its length is the count of input surfaces AMF still holds, so `submit` + /// bounds it below [`RING`] to keep the input ring from being overwritten under it. + pending: VecDeque<(u64, bool, bool)>, /// AUs already pulled by `submit`'s backpressure drain, waiting to be handed out by `poll` /// (FIFO, strictly older than anything still in `pending`). Empty in the steady state — only /// fills when the encoder falls behind and `submit` drains to free an input slot. @@ -988,6 +1061,26 @@ pub struct AmfEncoder { /// gates [`EncoderCaps::intra_refresh`] so keyframe-request rate-limiting only happens when /// the wave really runs. ir_active: bool, + // --- Long-Term-Reference reference-frame-invalidation recovery (the AMD RFI path) --- + /// The driver accepted the LTR properties at open — gates [`EncoderCaps::supports_rfi`] and all + /// the per-frame LTR marking/forcing below. When true, intra-refresh is NOT set (mutually + /// exclusive) and loss recovery re-references a known-good LTR instead of forcing a full IDR. + ltr_active: bool, + /// The `frame_idx` currently stored in each of the two LTR slots (`None` = never marked). On loss + /// the newest slot with an index *before* the loss is the known-good reference to force. + ltr_slots: [Option; NUM_LTR_SLOTS], + /// The slot the next LTR mark writes (round-robins `0,1,0,1,…` so the two slots hold a sliding + /// pair of recent references). + next_ltr_slot: usize, + /// Cadence (frames) between LTR marks — a fresh long-term reference roughly this often. + ltr_mark_interval: i64, + /// Set by [`invalidate_ref_frames`](Encoder::invalidate_ref_frames): the LTR slot the *next* + /// submitted frame must force-reference (`ForceLTRReferenceBitfield`). Consumed on that submit. + pending_force: Option, + /// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N`, self-trigger the + /// real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path so a headless spike run can + /// exercise LTR recovery end-to-end without a live client. `None` in normal operation. + ltr_test_force_at: Option, /// Consecutive [`reset`](Self::reset)s that have NOT been followed by a produced AU (cleared in /// `poll` on any output). An in-place `Terminate`+re-`Init` heals a transient component stall, /// but it re-inits the SAME context — so if the fault is the context / VCN session itself (the @@ -1084,17 +1177,33 @@ impl AmfEncoder { force_kf: false, hdr_meta: None, ir_active: false, + ltr_active: false, + ltr_slots: [None; NUM_LTR_SLOTS], + next_ltr_slot: 0, + ltr_mark_interval: ltr_mark_interval(fps), + pending_force: None, + ltr_test_force_at: ltr_test_force_at(), resets_without_output: 0, }) } + /// Whether this encoder should *attempt* the LTR-RFI recovery path (design: the AMD twin of + /// NVENC intra-refresh recovery). Gated to AVC/HEVC — AMF exposes user LTR only for those two + /// codecs — and defeatable via `PUNKTFUNK_NO_AMF_LTR`. Whether the driver actually *accepts* the + /// properties is a separate question answered by [`apply_static_props`], which sets `ltr_active`. + fn ltr_wanted(&self) -> bool { + !ltr_disabled() && matches!(self.codec, Codec::H264 | Codec::H265) + } + /// Apply the static encoder configuration (design §3.4 — the native mirror of the ffmpeg /// opts block in `open_win_encoder`). Called before `Init`, and again on a `reset()` /// re-`Init` (Terminate does not guarantee property retention across every driver). - /// Returns whether the intra-refresh wave was requested AND accepted by this driver — the - /// caller stores it so [`Encoder::caps`] only rate-limits keyframe requests when the wave - /// really runs. - unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result { + /// Returns `(ir_active, ltr_active)`: whether the intra-refresh wave / the LTR-RFI slots were + /// requested AND accepted by this driver. The two are mutually exclusive (LTR wins when both are + /// wanted). The caller stores both — `ir_active` so [`Encoder::caps`] only rate-limits keyframe + /// requests when a wave runs, `ltr_active` so [`Encoder::caps`] advertises `supports_rfi` and the + /// per-frame mark/force logic in `submit` only fires when the slots exist. + unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<(bool, bool)> { let p = &self.props; // Usage first: it "fully configures parameter set" — everything after is an override. set_prop( @@ -1145,7 +1254,39 @@ impl AmfEncoder { // whole picture refreshes every `period` frames — per-slot units = ceil(total blocks / // period). Optional by VCN generation; the return value gates `caps().intra_refresh`. let mut ir_active = false; - if let Some((name, block)) = p.intra_refresh { + let mut ltr_active = false; + if let Some(ltr) = p.ltr.as_ref().filter(|_| self.ltr_wanted()) { + // LTR-RFI recovery (design: the AMD twin of NVENC intra-refresh recovery). Request + // NUM_LTR_SLOTS user-controlled long-term references. LTR needs >1 reference frames and + // is MUTUALLY EXCLUSIVE with intra-refresh (AMF disables one if both are set), so the + // intra-refresh block below is skipped whenever LTR engages. + let ref_ok = set_prop( + comp, + ltr.max_num_ref_frames, + AmfVariant::from_i64(NUM_LTR_SLOTS as i64), + false, + )?; + let ltr_ok = set_prop( + comp, + ltr.max_ltr_frames, + AmfVariant::from_i64(NUM_LTR_SLOTS as i64), + false, + )?; + ltr_active = ref_ok && ltr_ok; + if ltr_active { + tracing::info!( + slots = NUM_LTR_SLOTS, + mark_interval = self.ltr_mark_interval, + "AMF LTR-RFI recovery enabled (loss recovery re-references a known-good LTR, not a full IDR)" + ); + } else { + tracing::warn!( + ref_ok, + ltr_ok, + "this VCN/driver rejected an LTR property — loss recovery stays full-IDR" + ); + } + } else if let Some((name, block)) = p.intra_refresh { if intra_refresh_requested() { let period = intra_refresh_period(self.fps); let blocks = self.width.div_ceil(block) * self.height.div_ceil(block); @@ -1273,7 +1414,7 @@ impl AmfEncoder { AmfVariant::from_i64(primaries), self.ten_bit, )?; - Ok(ir_active) + Ok((ir_active, ltr_active)) } /// Build (or rebuild, on a capture-device change) the AMF context + encoder component on the @@ -1323,7 +1464,7 @@ impl AmfEncoder { bail!("AMF CreateComponent returned null"); } let comp = Component(comp); - let ir_active = self.apply_static_props(comp.0)?; + let (ir_active, ltr_active) = self.apply_static_props(comp.0)?; let fmt = if self.ten_bit { sys::AMF_SURFACE_P010 } else { @@ -1334,6 +1475,14 @@ impl AmfEncoder { "AMF encoder Init", )?; self.ir_active = ir_active; + // A rebuilt component starts with fresh (empty) LTR slots — a new context has no + // reference history, so any prior marks are void and the first frame re-IDRs anyway. + self.ltr_active = ltr_active; + if ltr_active { + self.ltr_slots = [None; NUM_LTR_SLOTS]; + self.next_ltr_slot = 0; + self.pending_force = None; + } // Owned input ring on the capturer's device (design §3.2): RENDER_TARGET | // SHADER_RESOURCE, the same bind flags the validated ffmpeg zero-copy pool uses. @@ -1594,7 +1743,7 @@ enum DrainOutcome { /// single encode thread with no other AMF call to this component in flight. unsafe fn drain_one_output( comp: *mut sys::AmfComponent, - pending: &mut VecDeque<(u64, bool)>, + pending: &mut VecDeque<(u64, bool, bool)>, output_data_type: PCWSTR, output_key_max: i64, ) -> Result { @@ -1641,11 +1790,12 @@ unsafe fn drain_one_output( bail!("AMF output buffer is empty"); } let au = std::slice::from_raw_parts(native as *const u8, size).to_vec(); - let (pts_ns, forced) = pending.pop_front().unwrap_or((0, false)); + let (pts_ns, forced, recovery_anchor) = pending.pop_front().unwrap_or((0, false, false)); Ok(DrainOutcome::Frame(EncodedFrame { data: au, pts_ns, keyframe: key_prop || forced, + recovery_anchor, })) } @@ -1689,9 +1839,57 @@ impl Encoder for AmfEncoder { expected ); self.ensure_inner(&frame.device)?; + let cur_idx = self.frame_idx; let forced = std::mem::take(&mut self.force_kf) || self.frame_idx == 0; let pts_100ns = self.frame_idx * 10_000_000 / self.fps.max(1) as i64; self.frame_idx += 1; + // --- LTR-RFI per-frame decisions (design: the AMD twin of NVENC intra-refresh recovery) --- + // Decided here, before borrowing `inner`, because the test hook re-enters `&mut self` + // (`invalidate_ref_frames`) and the mark cadence mutates the slot bookkeeping. The two + // per-frame property names are copied out (PCWSTR is Copy) so the unsafe surface block can + // set them without re-borrowing `self.props` under the live `inner` borrow. + let ltr_names = self + .props + .ltr + .as_ref() + .map(|l| (l.mark_ltr_index, l.force_ltr_bitfield)); + let mut mark_slot: Option = None; + let mut force_slot: Option = None; + let mut recovery_anchor = false; + if self.ltr_active { + if forced { + // An IDR resets the decoder's reference buffers — every prior LTR mark is void. + // Re-anchor from scratch: drop the stale slots (the mark cadence below tags the IDR + // as the first fresh long-term reference) and cancel any force queued against them. + self.ltr_slots = [None; NUM_LTR_SLOTS]; + self.next_ltr_slot = 0; + self.pending_force = None; + } else if self.ltr_test_force_at == Some(cur_idx) { + // Spike-only validation hook: self-trigger the real invalidate path so a headless + // run exercises mark → force → recovery-anchor without a live client's RfiRequest. + let triggered = self.invalidate_ref_frames(cur_idx, cur_idx); + tracing::info!( + frame = cur_idx, + triggered, + "AMF LTR test hook fired invalidate_ref_frames" + ); + } + // Apply a queued force (from invalidate_ref_frames / the test hook) to THIS frame: it + // becomes the clean re-anchor P-frame the client lifts its post-loss freeze on. + if let Some(slot) = self.pending_force.take() { + force_slot = Some(slot); + recovery_anchor = true; + } + // Mark cadence: refresh a long-term reference on every IDR and every `ltr_mark_interval` + // frames — but never on the recovery frame itself (marking rotates `next_ltr_slot` and + // could overwrite the very slot being forced; the next cadence mark re-establishes it). + if force_slot.is_none() && (forced || cur_idx % self.ltr_mark_interval == 0) { + let slot = self.next_ltr_slot; + self.ltr_slots[slot] = Some(cur_idx); + self.next_ltr_slot = (self.next_ltr_slot + 1) % NUM_LTR_SLOTS; + mark_slot = Some(slot); + } + } let inner = self.inner.as_mut().expect("ensure_inner succeeded"); // Push the HDR mastering metadata when it changed (or a rebuilt component lost it) — a // dynamic property, so mid-stream regrades take effect on the next IDR. Best-effort: a @@ -1831,6 +2029,47 @@ impl Encoder for AmfEncoder { Codec::Av1 => {} } } + // LTR-RFI per-frame properties (design: the AMD twin of NVENC intra-refresh recovery). + // `mark_slot`/`force_slot` were decided above. Marking tags the current frame as a + // long-term reference; forcing makes it re-reference a known-good LTR — a clean P-frame + // that breaks the corrupted short-term chain after a loss, no 20-40× IDR. Best-effort: + // a rejecting driver just leaves the client on its keyframe-request fallback. + if let Some((mark_name, force_name)) = ltr_names { + if let Some(slot) = mark_slot { + let r = ((*(*surf.0).vtbl).set_property)( + surf.0, + mark_name.0, + AmfVariant::from_i64(slot as i64), + ); + if r != sys::AMF_OK { + tracing::warn!( + slot, + result = %format!("{} ({r})", result_name(r)), + "AMF LTR mark rejected" + ); + } + } + if let Some(slot) = force_slot { + let r = ((*(*surf.0).vtbl).set_property)( + surf.0, + force_name.0, + AmfVariant::from_i64(1_i64 << slot), + ); + if r == sys::AMF_OK { + tracing::info!( + slot, + frame = cur_idx, + "AMF LTR-RFI: re-referencing known-good LTR (clean recovery, no IDR)" + ); + } else { + tracing::warn!( + slot, + result = %format!("{} ({r})", result_name(r)), + "AMF LTR force-reference rejected — client stays frozen until its IDR fallback" + ); + } + } + } let mut r = ((*(*inner.comp.0).vtbl).submit_input)(inner.comp.0, surf.0); // Backstop back-pressure: the in-flight bound above already keeps a slot free, but if // AMF's own input queue is momentarily full, AMF_INPUT_FULL is "busy, drain me and @@ -1873,7 +2112,7 @@ impl Encoder for AmfEncoder { } } } - inner.pending.push_back((captured.pts_ns, forced)); + inner.pending.push_back((captured.pts_ns, forced, recovery_anchor)); Ok(()) } @@ -1887,11 +2126,63 @@ impl Encoder for AmfEncoder { self.hdr_meta = meta; } + /// LTR-RFI recovery (the AMD twin of the Windows NVENC `nvEncInvalidateRefFrames` path): a loss + /// of client frames `[first, last]` is answered by forcing the *next* submitted frame to + /// re-reference the newest long-term reference marked *before* the loss — a clean P-frame the + /// client can decode against a picture it still holds, instead of a 20-40× IDR spike. + /// + /// Returns `true` when a usable pre-loss LTR exists (so the caller must NOT also force an IDR); + /// `false` when the loss predates every live LTR — then the only correct recovery is a keyframe, + /// and the caller falls back to [`request_keyframe`](Self::request_keyframe). Runs on the encode + /// thread (like submit/poll); the force is applied on the next `submit`. + fn invalidate_ref_frames(&mut self, first: i64, last: i64) -> bool { + // No live LTR session (driver declined the slots, or AV1 which has no user-LTR path) or a + // nonsense range → caller forces a full IDR. + if !self.ltr_active || first < 0 || first > last { + return false; + } + // Pick the newest LTR strictly OLDER than the loss: the most recent known-good reference the + // client still holds, so re-referencing it costs the least (smallest recovery-frame residual). + // Frame numbers are 1:1 with the client's (both count submissions in order — see the NVENC + // path), so `ltr_slots` (which store `frame_idx`) compare directly against `first`. + let mut best: Option<(usize, i64)> = None; + for (slot, marked) in self.ltr_slots.iter().enumerate() { + if let Some(idx) = *marked { + if idx < first && best.is_none_or(|(_, b)| idx > b) { + best = Some((slot, idx)); + } + } + } + match best { + Some((slot, ltr_frame)) => { + // Queue the force for the next submit; that frame ships tagged `recovery_anchor`. + self.pending_force = Some(slot); + tracing::info!( + first, + last, + slot, + ltr_frame, + "AMF LTR-RFI: forcing the next frame to re-reference a known-good LTR (no IDR)" + ); + true + } + None => { + tracing::info!( + first, + last, + "AMF LTR-RFI: no live LTR older than the loss — falling back to IDR recovery" + ); + false + } + } + } + fn caps(&self) -> EncoderCaps { EncoderCaps { - // AMF has no NVENC-style reference invalidation — the intra-refresh wave is the - // loss-recovery substitute; without it every unrecoverable loss costs an IDR. - supports_rfi: false, + // LTR-RFI: AMD's reference invalidation is the user long-term-reference path (mark a + // frame, force a later one to re-reference it). True only when the live driver accepted + // the LTR slots at open — otherwise loss recovery falls back to a full IDR. + supports_rfi: self.ltr_active, // In-band mastering/CLL via `*InHDRMetadata` (HEVC SEI / AV1 metadata OBU); AVC has // no such property (and no HDR sessions negotiate H.264). supports_hdr_metadata: self.ten_bit && self.props.hdr_metadata.is_some(), @@ -1901,6 +2192,11 @@ impl Encoder for AmfEncoder { // accepted the property (queried per loss event, so the post-first-frame value is // what the session glue's IDR rate-limiting sees). intra_refresh: self.ir_active, + // Not yet: the AMD VCN wave heals in principle, but its constrained-GDR + // heal-within-a-period is unvalidated on-glass and AMF emits no recovery-point SEI, so + // the host keeps the IDR recovery path. Flip both once verified on real hardware. + intra_refresh_recovery: false, + intra_refresh_period: 0, } } @@ -1992,6 +2288,7 @@ impl Encoder for AmfEncoder { self.inner = None; self.bound_device = 0; self.ir_active = false; + self.ltr_active = false; return true; } let inner = self @@ -2016,8 +2313,14 @@ impl Encoder for AmfEncoder { sys::AMF_SURFACE_NV12 }; match self.apply_static_props(comp) { - Ok(ir) => { + Ok((ir, ltr)) => { self.ir_active = ir; + // Re-Init voids the reference history: the rebuilt stream restarts at IDR with + // empty LTR slots, so any prior marks are stale and must be dropped. + self.ltr_active = ltr; + self.ltr_slots = [None; NUM_LTR_SLOTS]; + self.next_ltr_slot = 0; + self.pending_force = None; ((*(*comp).vtbl).init)(comp, fmt, self.width as i32, self.height as i32) == sys::AMF_OK } @@ -2030,6 +2333,7 @@ impl Encoder for AmfEncoder { ); } else { self.ir_active = false; + self.ltr_active = false; // Full teardown; the next submit reopens context + component on the current device. tracing::warn!("AMF in-place re-Init failed — full context teardown, reopening lazily"); self.inner = None; diff --git a/crates/punktfunk-host/src/encode/windows/ffmpeg_win.rs b/crates/punktfunk-host/src/encode/windows/ffmpeg_win.rs index dbf575de..8e548549 100644 --- a/crates/punktfunk-host/src/encode/windows/ffmpeg_win.rs +++ b/crates/punktfunk-host/src/encode/windows/ffmpeg_win.rs @@ -339,6 +339,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result Result { "--source" => { source = match next()?.as_str() { "synthetic" => Source::Synthetic, + "synthetic-nv12" => Source::SyntheticNv12, "portal" => Source::Portal, "kwin-virtual" => Source::KwinVirtual, other => { - bail!("unknown --source '{other}' (synthetic|portal|kwin-virtual)") + bail!( + "unknown --source '{other}' \ + (synthetic|synthetic-nv12|portal|kwin-virtual)" + ) } } } diff --git a/crates/punktfunk-host/src/pipeline.rs b/crates/punktfunk-host/src/pipeline.rs index 71f338a3..7b227802 100644 --- a/crates/punktfunk-host/src/pipeline.rs +++ b/crates/punktfunk-host/src/pipeline.rs @@ -26,12 +26,16 @@ pub fn pump_once( data, pts_ns, keyframe, + recovery_anchor, }) = encoder.poll()? { let mut flags = FLAG_PIC as u32; if keyframe { flags |= FLAG_SOF as u32; } + if recovery_anchor { + flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR; + } // core does FEC + packetize + pace + send. session.submit_frame(&data, pts_ns, flags)?; } diff --git a/crates/punktfunk-host/src/punktfunk1.rs b/crates/punktfunk-host/src/punktfunk1.rs index 21a362d3..379ac14a 100644 --- a/crates/punktfunk-host/src/punktfunk1.rs +++ b/crates/punktfunk-host/src/punktfunk1.rs @@ -34,7 +34,7 @@ use punktfunk_core::packet::{FLAG_PIC, FLAG_PROBE, FLAG_SOF}; use punktfunk_core::quic::{ endpoint, io, BitrateChanged, ClockEcho, ClockProbe, ColorInfo, Hello, LossReport, PairChallenge, PairProof, PairRequest, PairResult, ProbeRequest, ProbeResult, Reconfigure, - Reconfigured, RequestKeyframe, SetBitrate, Start, Welcome, + Reconfigured, RequestKeyframe, RfiRequest, SetBitrate, Start, Welcome, }; use punktfunk_core::transport::UdpTransport; use punktfunk_core::Session; @@ -1124,6 +1124,10 @@ async fn serve_session( // (inbound requests, outbound probe results) are multiplexed with `select!`. let (reconfig_tx, reconfig_rx) = std::sync::mpsc::channel::(); let (keyframe_tx, keyframe_rx) = std::sync::mpsc::channel::<()>(); + // Client LTR-RFI recovery: the control task forwards each `RfiRequest`'s lost-frame range here; + // the encode loop prefers `Encoder::invalidate_ref_frames` (a clean re-anchor P-frame) over a + // full IDR when the encoder supports it (native-AMF LTR / Windows NVENC). + let (rfi_tx, rfi_rx) = std::sync::mpsc::channel::<(u32, u32)>(); let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::(); let (probe_tx, probe_rx) = std::sync::mpsc::channel::(); let (probe_result_tx, mut probe_result_rx) = @@ -1199,6 +1203,19 @@ async fn serve_session( if keyframe_tx.send(()).is_err() { break; // data plane gone } + } else if let Ok(req) = RfiRequest::decode(&msg) { + // Client LTR-RFI recovery: it lost the frame range `[first, last]` and asks + // the encoder to re-reference a known-good older frame instead of paying for + // a full IDR. The encode loop attempts `invalidate_ref_frames`, falling back + // to a coalesced keyframe when the encoder can't (range too old / no RFI). + tracing::debug!( + first = req.first_frame, + last = req.last_frame, + "client requested reference-frame invalidation (loss recovery)" + ); + if rfi_tx.send((req.first_frame, req.last_frame)).is_err() { + break; // data plane gone + } } else if let Ok(rep) = LossReport::decode(&msg) { // Adaptive FEC: size recovery to the loss the client is seeing. The data-plane // send loop reads `fec_target_ctl` and applies it per frame. Ignored when FEC @@ -1590,6 +1607,7 @@ async fn serve_session( quit: quit_stream, reconfig: reconfig_rx, keyframe: keyframe_rx, + rfi: rfi_rx, bitrate_rx, compositor, bitrate_kbps, @@ -2396,6 +2414,29 @@ fn audio_thread( tracing::warn!("punktfunk/1 audio requires Linux or Windows — session continues without it"); } +/// Advance the intra-refresh wave position and decide whether this emitted AU is a wave boundary +/// that should carry [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT). +/// +/// `ir_wave_pos` counts frames since the last IDR/wave start; a real IDR re-phases it to 0 (an IDR +/// restarts the encoder's wave AND is itself a clean anchor, so it is never additionally marked). +/// Every `period`-th non-IDR AU is a boundary — the client lifts its post-loss freeze on the SECOND +/// such mark. Pure so the marking cadence is unit-tested without a GPU (see the pump's use in the +/// encode-poll loop). +fn mark_recovery_boundary(ir_wave_pos: &mut u32, is_keyframe: bool, period: u32) -> bool { + if is_keyframe { + *ir_wave_pos = 0; + false + } else { + *ir_wave_pos += 1; + if *ir_wave_pos >= period { + *ir_wave_pos = 0; + true + } else { + false + } + } +} + fn synthetic_stream( session: &mut Session, frames: u32, @@ -3396,6 +3437,9 @@ struct SessionContext { reconfig: std::sync::mpsc::Receiver, /// Client decode-recovery keyframe requests. keyframe: std::sync::mpsc::Receiver<()>, + /// Client LTR-RFI recovery requests — the lost-frame range `(first, last)`. The encode loop + /// prefers `Encoder::invalidate_ref_frames` over a full IDR when the encoder supports it. + rfi: std::sync::mpsc::Receiver<(u32, u32)>, /// Accepted mid-stream bitrate changes (adaptive bitrate, already clamped) — the encoder /// alone is rebuilt in place at the new rate; capture + virtual output are untouched. bitrate_rx: std::sync::mpsc::Receiver, @@ -3467,6 +3511,7 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> { quit, reconfig, keyframe, + rfi, bitrate_rx, compositor, mut bitrate_kbps, @@ -3684,6 +3729,11 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> { // Self-diagnosis for the periodic-stutter class: warns when the served recovery IDRs settle // into a stable multi-second rhythm (see [`crate::metronome::Metronome`]). let mut recovery_cadence = crate::metronome::Metronome::new(); + // Position within the current intra-refresh wave (frames since the last IDR/wave start). Only + // meaningful on a `caps().intra_refresh_recovery` encoder; the pump tags every wave-boundary AU + // with `USER_FLAG_RECOVERY_POINT` so the client can lift its post-loss freeze on a clean + // re-anchor without a full IDR. Re-phased to 0 at each emitted IDR (which restarts the wave). + let mut ir_wave_pos: u32 = 0; // Per-stage latency breakdown (PUNKTFUNK_PERF): per-call µs for the GPU-bound stages so we see // exactly where the capture→encoded latency goes — cap=try_latest (ring read + colour convert), // submit=encode_picture launch, wait=lock_bitstream (the scheduling wait + ASIC encode, the one @@ -3900,6 +3950,33 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> { while keyframe.try_recv().is_ok() { want_kf = true; } + // Client LTR-RFI recovery: prefer re-referencing a known-good older frame (a clean recovery + // P-frame — no 20-40× IDR spike) over a full keyframe when the encoder supports it (native + // AMF LTR / Windows NVENC). Drain the backlog (the client re-requests until the recovery + // frame lands) coalesced to the widest lost range. Attempt the invalidate only when a full + // IDR isn't already queued — an explicit keyframe request means a fully wedged decoder that + // needs the IDR, which supersedes an RFI recovery. A failure (range older than the encoder's + // live references, or no RFI backend) falls through to the coalesced keyframe path below. + let mut rfi_range: Option<(u32, u32)> = None; + while let Ok((first, last)) = rfi.try_recv() { + rfi_range = Some(match rfi_range { + Some((pf, pl)) => (pf.min(first), pl.max(last)), + None => (first, last), + }); + } + if !want_kf { + if let Some((first, last)) = rfi_range { + if enc.caps().supports_rfi && enc.invalidate_ref_frames(first as i64, last as i64) { + // The RFI recovered the loss with a clean re-anchor P-frame (no IDR). Anchor the + // keyframe cooldown so the client's echo of the SAME loss — its frames_dropped- + // driven keyframe request, arriving ~one loss-window later — is coalesced away + // instead of emitting a redundant full IDR right after the cheap recovery. + last_forced_idr = Some(std::time::Instant::now()); + } else { + want_kf = true; // range too old / no RFI backend → coalesced keyframe below + } + } + } if want_kf { // Clients request a keyframe on EVERY FEC-unrecoverable frame (`frames_dropped` polling) // and keep asking until the IDR actually arrives + decodes — a full round-trip on a link @@ -4225,11 +4302,28 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> { last_au_at = std::time::Instant::now(); encoder_resets = 0; let (cap_ns, sub_ns, deadline) = inflight.pop_front().expect("inflight non-empty"); - let flags = if au.keyframe { + let mut flags = if au.keyframe { (FLAG_PIC | FLAG_SOF) as u32 } else { FLAG_PIC as u32 }; + // Intra-refresh recovery marking (inert unless the backend validated its constrained GDR + // via `intra_refresh_recovery`): tag every wave-boundary AU with USER_FLAG_RECOVERY_POINT + // so the client lifts its post-loss freeze on the second mark — a proven clean re-anchor — + // instead of forcing a full IDR. See [`mark_recovery_boundary`] for the cadence. + let caps = enc.caps(); + if caps.intra_refresh_recovery + && caps.intra_refresh_period > 0 + && mark_recovery_boundary(&mut ir_wave_pos, au.keyframe, caps.intra_refresh_period) + { + flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_POINT; + } + // Reference-frame-invalidation recovery frame (AMD LTR force-reference): a clean P-frame + // off a known-good reference. Tag it so the client lifts its post-loss freeze on this one + // AU without an IDR — the definitive single-frame re-anchor (see USER_FLAG_RECOVERY_ANCHOR). + if au.recovery_anchor { + flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR; + } // Re-send the HDR mastering metadata (0xCE) on each keyframe (a decoder-resync point) and // whenever it changed, so a client that dropped the best-effort datagram re-converges. if let Some(m) = last_hdr_meta { @@ -4654,6 +4748,32 @@ mod tests { assert!(reconfig_allowed(None, false)); } + #[test] + fn recovery_marks_land_every_period_and_rephase_at_idr() { + let period = 4; + let mut pos = 0u32; + // Frames 1..=3 are mid-wave (no mark), frame 4 is the boundary; then it repeats. + let marks: Vec = (0..10) + .map(|_| mark_recovery_boundary(&mut pos, false, period)) + .collect(); + assert_eq!( + marks, + vec![false, false, false, true, false, false, false, true, false, false] + ); + + // An IDR mid-wave re-phases: the counter restarts, so the next boundary is a full period + // later (an IDR is itself a clean anchor, so it is not additionally marked). + let mut pos = 0u32; + assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1 + assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2 + assert!(!mark_recovery_boundary(&mut pos, true, period)); // IDR → pos 0, no mark + // Now a fresh full period is needed, not just the 2 remaining frames. + assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1 + assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2 + assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 3 + assert!(mark_recovery_boundary(&mut pos, false, period)); // pos 4 → mark + } + #[test] fn pad_snapshot_replaces_state_and_seq_gates() { use punktfunk_core::input::{gamepad, GamepadSnapshot}; diff --git a/crates/punktfunk-host/src/spike.rs b/crates/punktfunk-host/src/spike.rs index c8c5173e..219e374c 100644 --- a/crates/punktfunk-host/src/spike.rs +++ b/crates/punktfunk-host/src/spike.rs @@ -22,6 +22,11 @@ use std::time::Instant; pub enum Source { /// Deterministic moving BGRx test pattern — no capture session required. Synthetic, + /// Deterministic moving NV12 texture on the GPU (Windows only) — no capture session required. + /// Feeds the native AMF / D3D11 zero-copy encoders, which demand an NV12 GPU texture the CPU + /// `Synthetic` source can't give them. Used to validate GPU-encoder behaviour (e.g. AMF + /// intra-refresh) headlessly. + SyntheticNv12, /// Live monitor via the xdg ScreenCast portal + PipeWire. Portal, /// KWin virtual output created at `width`x`height` (zkde_screencast). Lets us validate @@ -56,6 +61,27 @@ pub fn run(opts: Options) -> Result<()> { ); Box::new(SyntheticCapturer::new(opts.width, opts.height, opts.fps)) } + Source::SyntheticNv12 => { + #[cfg(target_os = "windows")] + { + tracing::info!( + width = opts.width, + height = opts.height, + fps = opts.fps, + "spike source: synthetic NV12 GPU texture (moving luma ramp)" + ); + Box::new( + capture::synthetic_nv12::SyntheticNv12Capturer::new( + opts.width, opts.height, opts.fps, + ) + .context("open synthetic NV12 capturer")?, + ) + } + #[cfg(not(target_os = "windows"))] + { + anyhow::bail!("--source synthetic-nv12 is Windows-only (native AMF / D3D11 encoders)"); + } + } Source::Portal => { tracing::info!("spike source: xdg ScreenCast portal (live monitor)"); capture::open_portal_monitor().context("open portal capturer")? diff --git a/include/punktfunk_core.h b/include/punktfunk_core.h index 52faa565..2500aeb4 100644 --- a/include/punktfunk_core.h +++ b/include/punktfunk_core.h @@ -255,6 +255,25 @@ // feeding them to the decoder. Punktfunk/1 only (GameStream never sets it). #define FLAG_PROBE 8 +// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client +// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that +// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is +// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its +// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible +// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host +// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four. +#define USER_FLAG_RECOVERY_POINT 16 + +// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike +// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss +// is only half-healed so the client waits for the second), this marks an access unit the host coded +// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation** +// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client +// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts +// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets +// `AV_FRAME_FLAG_KEY` — this host flag is the only signal. +#define USER_FLAG_RECOVERY_ANCHOR 32 + // Largest UDP datagram the core will send or accept. `Config::validate` bounds // `shard_payload` so `HEADER_LEN + shard_payload + CRYPTO_OVERHEAD ≤ MAX_DATAGRAM_BYTES`. #define MAX_DATAGRAM_BYTES 2048 @@ -462,6 +481,11 @@ #define MSG_BITRATE_CHANGED 6 #endif +#if defined(PUNKTFUNK_FEATURE_QUIC) +// Type byte of [`RfiRequest`]. +#define MSG_RFI_REQUEST 7 +#endif + #if defined(PUNKTFUNK_FEATURE_QUIC) // Type byte of [`ProbeRequest`]. #define MSG_PROBE_REQUEST 32