feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike. Client (pf-client-core): after a reference loss the hardware decoder conceals the missing-reference deltas (on RADV, a gray plate with new motion painted over) and returns Ok. The pump now freezes on the last good picture until a clean re-anchor instead of showing the concealment — lifting on a real IDR, an intra-refresh recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The frame_index gap is the early, precise loss signal and drives an RFI request. Host recovery signals (inert unless the backend supports them): - USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR). - USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame. AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the only clean-recovery lever is user LTR: mark frames as long-term references, and on loss force the next frame to re-reference the newest known-good one — a clean P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR before the loss; a range older than the live slots falls back to a keyframe. Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi(). Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success anchors the keyframe cooldown so the client's frames_dropped echo of the same loss is coalesced away rather than emitting a redundant IDR. Spike: synthetic NV12 GPU source for headless AMF encoder testing. Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests (incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux; punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90 forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -98,6 +98,10 @@ pub struct VkVideoFrame {
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I): the stream's re-anchor point. The pump resumes display on
|
||||
/// one after suppressing the concealed frames a reference loss leaves in its wake (on
|
||||
/// RADV a lost reference decodes to a gray plate with the new motion painted on top).
|
||||
pub keyframe: bool,
|
||||
/// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive
|
||||
/// until the presenter's fence proves the GPU reads done — same mechanism as the
|
||||
/// VAAPI path's DRM guard.
|
||||
@@ -143,6 +147,44 @@ impl ColorDesc {
|
||||
}
|
||||
}
|
||||
|
||||
/// True if the decoder tagged this frame as a full IDR keyframe — a guaranteed clean re-anchor
|
||||
/// after which the picture is loss-free, so the pump can lift a post-loss display freeze here.
|
||||
///
|
||||
/// Keys off `AV_FRAME_FLAG_KEY` (with `pict_type == I` as a belt for decoders that fill pict_type
|
||||
/// but not the flag). NOTE: FFmpeg's H.264/HEVC decode layer sets this flag **only for true IDR
|
||||
/// frames**, never for an *intra-refresh recovery point*. H.264 flags key only when a picture's
|
||||
/// `recovery_frame_cnt == 0` (a moving band uses `> 0`); HEVC clears the flag on every non-IRAP
|
||||
/// frame regardless of the recovery-point SEI. So an intra-refresh host (NVENC/AMF/QSV) heals the
|
||||
/// picture over N P-frames with no decoded frame ever flagged key — this function cannot detect
|
||||
/// that clean point, and the pump would freeze until the `REANCHOR_FREEZE_MAX` backstop (in
|
||||
/// `session.rs`) forces a real IDR. Detecting an intra-refresh re-anchor requires an out-of-band
|
||||
/// host wire signal on the AU that completes the wave; that is not yet plumbed.
|
||||
///
|
||||
/// # Safety
|
||||
/// `frame` must point to a valid `AVFrame` alive for the duration of the call.
|
||||
pub unsafe fn frame_is_keyframe(frame: *const ffmpeg::ffi::AVFrame) -> bool {
|
||||
// SAFETY: caller guarantees a live AVFrame; plain field reads.
|
||||
unsafe {
|
||||
((*frame).flags & ffmpeg::ffi::AV_FRAME_FLAG_KEY) != 0
|
||||
|| (*frame).pict_type == ffmpeg::ffi::AVPictureType::AV_PICTURE_TYPE_I
|
||||
}
|
||||
}
|
||||
|
||||
impl DecodedImage {
|
||||
/// Whether the frame is an intra keyframe — see [`frame_is_keyframe`]. The pump uses
|
||||
/// this as the stream's re-anchor signal after a loss.
|
||||
pub fn is_keyframe(&self) -> bool {
|
||||
match self {
|
||||
DecodedImage::Cpu(f) => f.keyframe,
|
||||
#[cfg(target_os = "linux")]
|
||||
DecodedImage::Dmabuf(f) => f.keyframe,
|
||||
DecodedImage::VkFrame(f) => f.keyframe,
|
||||
#[cfg(windows)]
|
||||
DecodedImage::D3d11(f) => f.keyframe,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The Y′CbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant
|
||||
/// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient
|
||||
/// implementation every presenter derives its CSC from (Vulkan push constants, the Windows
|
||||
@@ -205,6 +247,8 @@ pub struct CpuFrame {
|
||||
/// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries
|
||||
/// baked in — the presenter tags the texture so GTK tone-maps it.
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
|
||||
pub keyframe: bool,
|
||||
}
|
||||
|
||||
/// A decoded frame still on the GPU: dmabuf fds + plane layout for
|
||||
@@ -222,6 +266,8 @@ pub struct DmabufFrame {
|
||||
/// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709
|
||||
/// narrow for SDR, BT.2020 PQ for an HDR stream).
|
||||
pub color: ColorDesc,
|
||||
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
|
||||
pub keyframe: bool,
|
||||
pub guard: DrmFrameGuard,
|
||||
}
|
||||
|
||||
@@ -644,6 +690,9 @@ impl SoftwareDecoder {
|
||||
stride: dst_linesize[0] as usize,
|
||||
rgba,
|
||||
color,
|
||||
// `is_key()` reads the same intra flag `frame_is_keyframe` derives from pict_type
|
||||
// for the hardware paths; ffmpeg-next handles the FFmpeg-version binding split.
|
||||
keyframe: frame.is_key(),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -844,6 +893,7 @@ impl VaapiDecoder {
|
||||
// SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after
|
||||
// this returns); plain CICP field reads.
|
||||
color: ColorDesc::from_raw(self.frame),
|
||||
keyframe: frame_is_keyframe(self.frame),
|
||||
guard,
|
||||
})
|
||||
}
|
||||
@@ -1363,6 +1413,7 @@ impl VulkanDecoder {
|
||||
width: (*self.frame).width as u32,
|
||||
height: (*self.frame).height as u32,
|
||||
color: ColorDesc::from_raw(self.frame),
|
||||
keyframe: frame_is_keyframe(self.frame),
|
||||
guard: DrmFrameGuard(clone),
|
||||
})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user