fix(encode): harden loss-recovery correctness across host encoders (F1–F7)
Phases 1–4 of design/encoder-recovery-hardening.md — make the shipped RFI/ freeze-until-reanchor recovery honest and rebuild-safe across every backend. F1 — frame-index domain desync: the encode loop now owns a session-lifetime `au_seq`; `Encoder::submit_indexed(au_seq + inflight)` pins NVENC inputTimeStamp and AMF LTR slots to the WIRE frame index, so `invalidate_ref_frames` compares client frame numbers in the same domain and survives adaptive-bitrate rebuilds (an internal counter desynced on the first rebuild → RFI silently dead / an AMF force-ref onto a never-decoded frame). `FrameMsg.frame_index` → `Session::seal_frame_at`; GameStream gets the same via `VideoPacketizer:: packetize(.., Some(idx))`. F2 — Windows NVENC left the client frozen ~1s per loss: NVENC RFI was transparent (no anchor tag) while the session glue armed the 750ms IDR cooldown, so the freeze only lifted on the ~1s keyframe re-ask. NVENC now mirrors AMF — `pending_anchor` tags the first post-invalidate AU (the clean re-anchor P-frame) `recovery_anchor`, incl. the covering-range dedupe re-arm; the client lifts at ~RTT. F3 — speed-test probe filler burned video frame indexes: moved to its own index space (`Packetizer::alloc_probe_index` + `Session::submit_probe_frame`) with a second client reassembly window routed on FLAG_PROBE, gated on the new VIDEO_CAP_PROBE_SEQ Hello bit (mid-session probes declined for older clients). F4 — RFI range sanity cap: forward gaps wider than `packet::RFI_MAX_RANGE` (256) resync via keyframe instead of an out-of-range RFI, host- and client-side (client huge-gap → keyframe in `RfiRecovery::observe` + the pf-client-core pump). F5 — reset() parity: Windows NVENC (teardown + lazy re-init), Linux VAAPI (drop-inner), Linux NVENC (reopen from stored OpenArgs) now give the stall watchdog a heal lever instead of ending the session. F6 — sw.rs `pending: VecDeque` (was `Option`), killing the silent AU drop at capturer pipeline depth > 1. F7 — doc sweep on the RFI/anchor comments. Verified: punktfunk-core lib tests (macOS + Linux), full punktfunk-host suite on Linux (RTX 5070 Ti), Windows compile. Owed: the on-glass client matrix (F2 freeze A/B, AMF LTR spike across a bitrate rebuild). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -21,10 +21,13 @@ pub struct EncodedFrame {
|
||||
pub keyframe: bool,
|
||||
/// True when this AU is a **reference-frame-invalidation recovery frame** — a clean P-frame the
|
||||
/// encoder coded against a known-good reference in response to
|
||||
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) (AMD LTR force-reference). The pump
|
||||
/// tags it [`punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its post-loss
|
||||
/// freeze on it without an IDR. Only the native-AMF LTR path sets it; every other backend leaves
|
||||
/// it `false` (their RFI, when present, re-references transparently with no distinct clean-point AU).
|
||||
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames). The pump tags it
|
||||
/// [`punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its post-loss
|
||||
/// freeze on it without an IDR. Set by BOTH RFI backends: native AMF (the LTR force-reference
|
||||
/// frame) and Windows direct-NVENC (the first frame encoded after `nvEncInvalidateRefFrames` —
|
||||
/// the invalidation applies at the next `encode_picture`, so that AU is by construction the
|
||||
/// clean re-anchor). Without it the client's freeze can only lift on an IDR — which the host
|
||||
/// suppresses after a successful RFI (the cooldown), a ~1 s frozen stall per loss event.
|
||||
pub recovery_anchor: bool,
|
||||
}
|
||||
|
||||
@@ -201,8 +204,9 @@ pub struct EncoderCaps {
|
||||
/// The encoder can perform real reference-frame invalidation — i.e.
|
||||
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) can return `true`. When `false`
|
||||
/// the caller skips that always-`false` call and forces a keyframe directly on loss recovery.
|
||||
/// Only the Windows direct-NVENC path implements RFI; libavcodec (Linux NVENC), VAAPI and
|
||||
/// AMF/QSV always keyframe.
|
||||
/// Two backends implement RFI: Windows direct-NVENC (`nvEncInvalidateRefFrames`) and native
|
||||
/// AMF (user-LTR force-reference, when the driver accepted the LTR slots at open). The
|
||||
/// libavcodec paths (Linux NVENC, VAAPI, QSV) can't express it and always keyframe.
|
||||
pub supports_rfi: bool,
|
||||
/// The encoder emits in-band HDR mastering/CLL SEI from [`set_hdr_meta`](Encoder::set_hdr_meta).
|
||||
/// When `false`, `set_hdr_meta` is a no-op and no in-band grade reaches the client. Only the
|
||||
@@ -242,6 +246,22 @@ pub struct EncoderCaps {
|
||||
/// A hardware encoder. One per session; runs on the encode thread.
|
||||
pub trait Encoder: Send {
|
||||
fn submit(&mut self, frame: &CapturedFrame) -> Result<()>;
|
||||
/// [`submit`](Self::submit) with the **wire frame index** this frame's AU will carry — the
|
||||
/// number the packetizer stamps on it and the client's loss reports/RFI requests name. The
|
||||
/// session glue predicts it exactly as `AUs sent so far + frames in flight` (AUs are emitted
|
||||
/// FIFO, one per submission; anything that would break the prediction — an in-place reset, a
|
||||
/// device-change teardown, an encoder rebuild — forfeits the in-flight frames on BOTH sides
|
||||
/// and clears the encoder's reference state, so stale predictions die with it). The RFI
|
||||
/// backends pin their frame numbering (LTR marks, DPB timestamps) to this so
|
||||
/// [`invalidate_ref_frames`](Self::invalidate_ref_frames) compares client frame numbers
|
||||
/// against the same domain — an encoder-internal counter desyncs from the wire on the first
|
||||
/// mid-stream rebuild (adaptive bitrate steps do this under congestion, exactly when losses
|
||||
/// happen). Default: ignore the index and delegate to `submit` (backends without per-frame
|
||||
/// reference bookkeeping don't care).
|
||||
fn submit_indexed(&mut self, frame: &CapturedFrame, wire_index: u32) -> Result<()> {
|
||||
let _ = wire_index;
|
||||
self.submit(frame)
|
||||
}
|
||||
/// This encoder's static [capabilities](EncoderCaps) (RFI, HDR SEI), so the session glue can
|
||||
/// route by query rather than rely on the no-op/`false` defaults of
|
||||
/// [`invalidate_ref_frames`](Self::invalidate_ref_frames) / [`set_hdr_meta`](Self::set_hdr_meta).
|
||||
@@ -259,13 +279,14 @@ pub trait Encoder: Send {
|
||||
/// Default: no-op (SDR encoders / libavcodec paths that don't attach it yet). Cheap to call
|
||||
/// every frame; only the direct-NVENC path consumes it.
|
||||
fn set_hdr_meta(&mut self, _meta: Option<punktfunk_core::quic::HdrMeta>) {}
|
||||
/// Invalidate a contiguous range of previously-encoded reference frames (client frame numbers,
|
||||
/// as reported in a loss-recovery request) so the encoder re-references an older still-valid
|
||||
/// frame instead of emitting a full IDR. Returns `true` if a real reference invalidation was
|
||||
/// performed; `false` means the encoder couldn't (range older than the DPB, or the backend has
|
||||
/// no RFI) and the caller should fall back to [`request_keyframe`](Self::request_keyframe).
|
||||
/// Default: `false` — only the Windows direct-NVENC path implements true RFI; libavcodec
|
||||
/// (Linux NVENC) and VAAPI can't express `nvEncInvalidateRefFrames`, so they keyframe.
|
||||
/// Invalidate a contiguous range of previously-encoded reference frames (client frame numbers
|
||||
/// — WIRE frame indexes, the domain [`submit_indexed`](Self::submit_indexed) pins the encoder's
|
||||
/// bookkeeping to) so the encoder re-references an older still-valid frame instead of emitting
|
||||
/// a full IDR. Returns `true` if a real reference invalidation was performed; `false` means the
|
||||
/// encoder couldn't (range older than the DPB/LTR history, or the backend has no RFI) and the
|
||||
/// caller should fall back to [`request_keyframe`](Self::request_keyframe). Default: `false` —
|
||||
/// the Windows direct-NVENC path (`nvEncInvalidateRefFrames`) and native AMF (LTR
|
||||
/// force-reference) implement true RFI; the libavcodec paths can't express it, so they keyframe.
|
||||
fn invalidate_ref_frames(&mut self, _first_frame: i64, _last_frame: i64) -> bool {
|
||||
false
|
||||
}
|
||||
@@ -441,6 +462,9 @@ impl Encoder for TrackedEncoder {
|
||||
fn submit(&mut self, frame: &CapturedFrame) -> Result<()> {
|
||||
self.inner.submit(frame)
|
||||
}
|
||||
fn submit_indexed(&mut self, frame: &CapturedFrame, wire_index: u32) -> Result<()> {
|
||||
self.inner.submit_indexed(frame, wire_index)
|
||||
}
|
||||
fn caps(&self) -> EncoderCaps {
|
||||
self.inner.caps()
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user