feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike. Client (pf-client-core): after a reference loss the hardware decoder conceals the missing-reference deltas (on RADV, a gray plate with new motion painted over) and returns Ok. The pump now freezes on the last good picture until a clean re-anchor instead of showing the concealment — lifting on a real IDR, an intra-refresh recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The frame_index gap is the early, precise loss signal and drives an RFI request. Host recovery signals (inert unless the backend supports them): - USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR). - USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame. AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the only clean-recovery lever is user LTR: mark frames as long-term references, and on loss force the next frame to re-reference the newest known-good one — a clean P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR before the loss; a range older than the live slots falls back to a keyframe. Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi(). Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success anchors the keyframe cooldown so the client's frames_dropped echo of the same loss is coalesced away rather than emitting a redundant IDR. Spike: synthetic NV12 GPU source for headless AMF encoder testing. Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests (incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux; punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90 forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -19,7 +19,8 @@ use crate::packet::FLAG_PROBE;
|
||||
use crate::quic::{
|
||||
accept_resync, endpoint, io, wall_clock_ns, window_loss_ppm, BitrateChanged, ClockEcho,
|
||||
ClockResync, ColorInfo, HdrMeta, Hello, HidOutput, LossReport, ProbeRequest, ProbeResult,
|
||||
Reconfigure, Reconfigured, RequestKeyframe, ResyncStep, RichInput, SetBitrate, Start, Welcome,
|
||||
Reconfigure, Reconfigured, RequestKeyframe, RfiRequest, ResyncStep, RichInput, SetBitrate, Start,
|
||||
Welcome,
|
||||
};
|
||||
use crate::session::{Frame, Session};
|
||||
use crate::transport::UdpTransport;
|
||||
@@ -49,6 +50,10 @@ enum CtrlRequest {
|
||||
Mode(Mode),
|
||||
Probe(ProbeRequest),
|
||||
Keyframe,
|
||||
/// Reference-frame-invalidation recovery: the client saw a `frame_index` gap and reports the
|
||||
/// invalidation range so an RFI-capable host re-references a known-good picture instead of
|
||||
/// forcing a full IDR. See [`RfiRequest`].
|
||||
Rfi(RfiRequest),
|
||||
Loss(LossReport),
|
||||
/// Adaptive bitrate: ask the host to re-target its encoder (kbps). Sent by the pump's
|
||||
/// [`BitrateController`] when the user's bitrate setting is Automatic.
|
||||
@@ -868,6 +873,24 @@ impl NativeClient {
|
||||
.map_err(|_| PunktfunkError::Closed)
|
||||
}
|
||||
|
||||
/// Ask the host to recover from loss by **reference-frame invalidation** rather than a full IDR:
|
||||
/// the client reports the range `[first_frame, last_frame]` of access units it can no longer trust
|
||||
/// (from the first missing `frame_index` through the newest received). An RFI-capable host
|
||||
/// re-references a known-good picture before `first_frame` (AMD LTR / NVENC RFI) and emits a clean
|
||||
/// P-frame tagged [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`]; a host that can't RFI forces an IDR
|
||||
/// instead (same as [`request_keyframe`](Self::request_keyframe)). Non-blocking, fire-and-forget —
|
||||
/// the recovered frame is the only ack; throttle it like the keyframe request. Prefer this over
|
||||
/// `request_keyframe` on loss so AMD/RFI hosts avoid the IDR spike; the keyframe request remains
|
||||
/// the backstop when the recovery frame itself is lost.
|
||||
pub fn request_rfi(&self, first_frame: u32, last_frame: u32) -> Result<()> {
|
||||
self.ctrl_tx
|
||||
.try_send(CtrlRequest::Rfi(RfiRequest {
|
||||
first_frame,
|
||||
last_frame,
|
||||
}))
|
||||
.map_err(|_| PunktfunkError::Closed)
|
||||
}
|
||||
|
||||
/// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't
|
||||
/// rebuild them). A video loop polls this and calls [`request_keyframe`](Self::request_keyframe)
|
||||
/// when it increases — the correct loss trigger under infinite GOP, where unrecoverable loss
|
||||
@@ -1511,6 +1534,7 @@ async fn worker_main(args: WorkerArgs) {
|
||||
CtrlRequest::Mode(m) => Reconfigure { mode: m }.encode(),
|
||||
CtrlRequest::Probe(p) => p.encode(),
|
||||
CtrlRequest::Keyframe => RequestKeyframe.encode(),
|
||||
CtrlRequest::Rfi(r) => r.encode(),
|
||||
CtrlRequest::Loss(r) => r.encode(),
|
||||
CtrlRequest::SetBitrate(k) => SetBitrate { bitrate_kbps: k }.encode(),
|
||||
CtrlRequest::ClockResync => {
|
||||
|
||||
@@ -35,6 +35,25 @@ pub const FLAG_SOF: u8 = 0x4;
|
||||
/// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it).
|
||||
pub const FLAG_PROBE: u8 = 0x8;
|
||||
|
||||
/// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client
|
||||
/// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that
|
||||
/// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is
|
||||
/// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its
|
||||
/// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible
|
||||
/// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host
|
||||
/// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four.
|
||||
pub const USER_FLAG_RECOVERY_POINT: u32 = 0x10;
|
||||
|
||||
/// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike
|
||||
/// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss
|
||||
/// is only half-healed so the client waits for the second), this marks an access unit the host coded
|
||||
/// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation**
|
||||
/// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client
|
||||
/// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts
|
||||
/// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets
|
||||
/// `AV_FRAME_FLAG_KEY` — this host flag is the only signal.
|
||||
pub const USER_FLAG_RECOVERY_ANCHOR: u32 = 0x20;
|
||||
|
||||
/// Crypto framing overhead [`Session`](crate::session::Session) adds when encrypting:
|
||||
/// an 8-byte sequence prefix plus the GCM tag.
|
||||
pub const CRYPTO_OVERHEAD: usize = 8 + crate::crypto::TAG_LEN;
|
||||
|
||||
@@ -355,6 +355,24 @@ pub struct Reconfigured {
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct RequestKeyframe;
|
||||
|
||||
/// `client → host`: reference-frame-invalidation recovery — the loss-aware sibling of
|
||||
/// [`RequestKeyframe`]. The client detected a `frame_index` gap and reports the range `[first_frame,
|
||||
/// last_frame]` of access units it can no longer trust (from the first missing index through the
|
||||
/// newest received). Instead of a full IDR (a 20-40× spike that deepens the loss it recovers), a host
|
||||
/// whose encoder supports RFI re-references a known-good picture *before* `first_frame` — an AMD LTR
|
||||
/// force-reference or an NVENC `nvEncInvalidateRefFrames` — emitting a single clean P-frame it tags
|
||||
/// [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its freeze on it. A host that
|
||||
/// can't RFI (no valid reference / libavcodec backend) forces an IDR instead, exactly as for a bare
|
||||
/// [`RequestKeyframe`]; a host that predates this ignores the unknown message and the client's
|
||||
/// keyframe backstop still recovers. Fire-and-forget — the recovered frame is the only ack.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct RfiRequest {
|
||||
/// First access-unit `frame_index` the client can no longer trust (the gap start).
|
||||
pub first_frame: u32,
|
||||
/// Newest received `frame_index` at the time of the report (the invalidation range end).
|
||||
pub last_frame: u32,
|
||||
}
|
||||
|
||||
/// `client → host`, periodic: the client's observed data-plane loss, so the host can size FEC to
|
||||
/// the link instead of a flat percentage (adaptive FEC). `loss_ppm` is parts-per-million of shards
|
||||
/// that arrived missing-but-recovered (plus a bump when frames went unrecoverable) over the report
|
||||
@@ -467,6 +485,8 @@ pub const MSG_LOSS_REPORT: u8 = 0x04;
|
||||
pub const MSG_SET_BITRATE: u8 = 0x05;
|
||||
/// Type byte of [`BitrateChanged`].
|
||||
pub const MSG_BITRATE_CHANGED: u8 = 0x06;
|
||||
/// Type byte of [`RfiRequest`].
|
||||
pub const MSG_RFI_REQUEST: u8 = 0x07;
|
||||
/// Type byte of [`ProbeRequest`].
|
||||
pub const MSG_PROBE_REQUEST: u8 = 0x20;
|
||||
/// Type byte of [`ProbeResult`].
|
||||
@@ -1032,6 +1052,28 @@ impl RequestKeyframe {
|
||||
}
|
||||
}
|
||||
|
||||
impl RfiRequest {
|
||||
pub fn encode(&self) -> Vec<u8> {
|
||||
// magic[0..4] type[4] first_frame[5..9] last_frame[9..13]
|
||||
let mut b = Vec::with_capacity(13);
|
||||
b.extend_from_slice(CTL_MAGIC);
|
||||
b.push(MSG_RFI_REQUEST);
|
||||
b.extend_from_slice(&self.first_frame.to_le_bytes());
|
||||
b.extend_from_slice(&self.last_frame.to_le_bytes());
|
||||
b
|
||||
}
|
||||
|
||||
pub fn decode(b: &[u8]) -> Result<RfiRequest> {
|
||||
if b.len() != 13 || &b[0..4] != CTL_MAGIC || b[4] != MSG_RFI_REQUEST {
|
||||
return Err(PunktfunkError::InvalidArg("bad RfiRequest"));
|
||||
}
|
||||
Ok(RfiRequest {
|
||||
first_frame: u32::from_le_bytes(b[5..9].try_into().unwrap()),
|
||||
last_frame: u32::from_le_bytes(b[9..13].try_into().unwrap()),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl LossReport {
|
||||
pub fn encode(&self) -> Vec<u8> {
|
||||
// magic[0..4] type[4] loss_ppm[5..9]
|
||||
|
||||
@@ -633,6 +633,35 @@ fn request_keyframe_roundtrip() {
|
||||
assert!(RequestKeyframe::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rfi_request_roundtrip() {
|
||||
for (first_frame, last_frame) in [(0u32, 0u32), (40, 47), (5, 5), (1_000_000, u32::MAX)] {
|
||||
let r = RfiRequest {
|
||||
first_frame,
|
||||
last_frame,
|
||||
};
|
||||
assert_eq!(RfiRequest::decode(&r.encode()).unwrap(), r);
|
||||
}
|
||||
// Disjoint from the bare keyframe request (its loss-unaware sibling) and others: type byte + length.
|
||||
assert!(RfiRequest::decode(&RequestKeyframe.encode()).is_err());
|
||||
assert!(RequestKeyframe::decode(
|
||||
&RfiRequest {
|
||||
first_frame: 1,
|
||||
last_frame: 2
|
||||
}
|
||||
.encode()
|
||||
)
|
||||
.is_err());
|
||||
// Exact length — no trailing bytes.
|
||||
let bytes = RfiRequest {
|
||||
first_frame: 3,
|
||||
last_frame: 9,
|
||||
}
|
||||
.encode();
|
||||
assert!(RfiRequest::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
|
||||
assert!(RfiRequest::decode(&bytes[..bytes.len() - 1]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn loss_report_roundtrip() {
|
||||
for loss_ppm in [0u32, 1, 12_345, 50_000, 1_000_000] {
|
||||
|
||||
Reference in New Issue
Block a user