unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 25 Wiki Activity

fix(encode): harden loss-recovery correctness across host encoders (F1–F7)

Browse Source 
Phases 1–4 of design/encoder-recovery-hardening.md — make the shipped RFI/
freeze-until-reanchor recovery honest and rebuild-safe across every backend.

F1 — frame-index domain desync: the encode loop now owns a session-lifetime
`au_seq`; `Encoder::submit_indexed(au_seq + inflight)` pins NVENC inputTimeStamp
and AMF LTR slots to the WIRE frame index, so `invalidate_ref_frames` compares
client frame numbers in the same domain and survives adaptive-bitrate rebuilds
(an internal counter desynced on the first rebuild → RFI silently dead / an AMF
force-ref onto a never-decoded frame). `FrameMsg.frame_index` →
`Session::seal_frame_at`; GameStream gets the same via `VideoPacketizer::
packetize(.., Some(idx))`.

F2 — Windows NVENC left the client frozen ~1s per loss: NVENC RFI was
transparent (no anchor tag) while the session glue armed the 750ms IDR cooldown,
so the freeze only lifted on the ~1s keyframe re-ask. NVENC now mirrors AMF —
`pending_anchor` tags the first post-invalidate AU (the clean re-anchor
P-frame) `recovery_anchor`, incl. the covering-range dedupe re-arm; the client
lifts at ~RTT.

F3 — speed-test probe filler burned video frame indexes: moved to its own index
space (`Packetizer::alloc_probe_index` + `Session::submit_probe_frame`) with a
second client reassembly window routed on FLAG_PROBE, gated on the new
VIDEO_CAP_PROBE_SEQ Hello bit (mid-session probes declined for older clients).

F4 — RFI range sanity cap: forward gaps wider than `packet::RFI_MAX_RANGE` (256)
resync via keyframe instead of an out-of-range RFI, host- and client-side
(client huge-gap → keyframe in `RfiRecovery::observe` + the pf-client-core pump).

F5 — reset() parity: Windows NVENC (teardown + lazy re-init), Linux VAAPI
(drop-inner), Linux NVENC (reopen from stored OpenArgs) now give the stall
watchdog a heal lever instead of ending the session.

F6 — sw.rs `pending: VecDeque` (was `Option`), killing the silent AU drop at
capturer pipeline depth > 1. F7 — doc sweep on the RFI/anchor comments.

Verified: punktfunk-core lib tests (macOS + Linux), full punktfunk-host suite on
Linux (RTX 5070 Ti), Windows compile. Owed: the on-glass client matrix (F2
freeze A/B, AMF LTR spike across a bitrate rebuild).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Enrico Bühler
2026-07-12 11:17:19 +02:00
parent 0dc414f197
commit fdda7144ed
15 changed files with 801 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/punktfunk-host/src/gamestream/stream.rs
+39 -8
View File
@@ -436,7 +436,10 @@ fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec<u8>]) -> std::io::Result<()> {
/// behind encode (measured ~3 ms/frame at 4K, which capped GameStream's frame rate well below what
/// the encoder alone can sustain).
struct RawFrame {
aus: Vec<(Vec<u8>, FrameType)>,
/// `(bitstream, type, wire frameIndex)` per AU. The stream loop assigns the index (it owns
/// the numbering — see its `au_seq`), so the encoder's RFI bookkeeping stays 1:1 with what
/// Moonlight sees across mid-stream encoder rebuilds.
aus: Vec<(Vec<u8>, FrameType, u32)>,
ts: u32,
}
@@ -460,8 +463,8 @@ fn spawn_packetizer(
crate::punktfunk1::boost_thread_priority(false);
while let Ok(frame) = rx.recv() {
let mut batch: PacketBatch = Vec::new();
for (au, ft) in frame.aus {
batch.extend(pk.packetize(&au, ft, frame.ts));
for (au, ft, idx) in frame.aus {
batch.extend(pk.packetize(&au, ft, frame.ts, Some(idx)));
}
if batch.is_empty() {
continue;
@@ -660,6 +663,16 @@ fn stream_body(
// routed through the same coalesce gate as client IDR requests so a burst of drops (congestion)
// can't become an IDR storm.
let mut recover_after_drop = false;
// The stream's wire frameIndex numbering, owned HERE (the index of the next AU handed to the
// packetizer thread; a dropped-at-the-queue frame consumes none). A submission's future index
// is `au_seq + enc_inflight` (AUs are emitted FIFO, one per submission); passing it to
// `Encoder::submit_indexed` keeps the encoder's RFI bookkeeping 1:1 with Moonlight's frame
// numbers across the in-place encoder rebuild above (an internal counter would desync there).
// A pipeline-head drop desyncs the prediction by the dropped AU count for the frames already
// in flight — bounded and self-healing: the drop arms `recover_after_drop`, whose forced IDR
// resets the encoder's reference state (stale LTR/DPB bookkeeping dies with it).
let mut au_seq: u32 = 0;
let mut enc_inflight: u32 = 0;
while running.load(Ordering::SeqCst) {
let tick = Instant::now();
@@ -728,6 +741,10 @@ fn stream_body(
enc.request_keyframe();
last_keyframe = Some(Instant::now());
next_frame = Instant::now();
// The old encoder died with its in-flight submissions — their AUs will never
// arrive, so the numbering prediction restarts at `au_seq` (the fresh encoder's
// reference state is empty, so the reused predictions meet no stale bookkeeping).
enc_inflight = 0;
tracing::info!("gamestream: source rebuilt — stream continues");
continue;
}
@@ -742,7 +759,13 @@ fn stream_body(
if let Some((first, last)) = rfi_range.lock().unwrap().take() {
// Prefer reference-frame invalidation when the encoder supports it (no costly IDR
// spike); otherwise — or if the range is too old to invalidate — fall back to a keyframe.
if !(supports_rfi && enc.invalidate_ref_frames(first, last)) {
// Sanity-cap the range first: wider than RFI_MAX_RANGE exceeds any encoder's reference
// history (or is a phantom range from a desynced counter) — keyframe, never a
// force-reference that could ship corruption as a clean frame.
let width = (last as u32).wrapping_sub(first as u32);
if width > punktfunk_core::packet::RFI_MAX_RANGE
|| !(supports_rfi && enc.invalidate_ref_frames(first, last))
{
want_keyframe = true;
}
}
@@ -766,21 +789,27 @@ fn stream_body(
tracing::debug!("video: keyframe request coalesced (IDR still in flight)");
}
}
enc.submit(&frame).context("encoder submit")?;
enc.submit_indexed(&frame, au_seq.wrapping_add(enc_inflight))
.context("encoder submit")?;
enc_inflight = enc_inflight.wrapping_add(1);
let t_enc = tick.elapsed();
// 90 kHz RTP timestamp from wall-clock, so a variable capture rate stays correct.
let ts = (stream_start.elapsed().as_secs_f64() * 90_000.0) as u32;
// Drain the encoder's access units (owned buffers) — FEC/packetization runs on the
// packetizer thread, off this loop, so it never serializes behind encode.
let mut aus: Vec<(Vec<u8>, FrameType)> = Vec::new();
// packetizer thread, off this loop, so it never serializes behind encode. Each AU is
// stamped with its wire frameIndex here (`au_seq + position`); the numbering only
// ADVANCES if the batch is actually enqueued below (a dropped batch consumes none).
let mut aus: Vec<(Vec<u8>, FrameType, u32)> = Vec::new();
while let Some(au) = enc.poll().context("encoder poll")? {
let ft = if au.keyframe {
FrameType::Idr
} else {
FrameType::P
};
aus.push((au.data, ft));
let idx = au_seq.wrapping_add(aus.len() as u32);
aus.push((au.data, ft, idx));
enc_inflight = enc_inflight.saturating_sub(1);
}
let t_pkt = tick.elapsed();
@@ -788,9 +817,11 @@ fn stream_body(
// (packetizer, or the paced sender behind it) is behind — drop this frame (FEC/RFI covers the
// client) and keep encoding, so a downstream stall can never cap the encode rate.
if !aus.is_empty() {
let batch_len = aus.len() as u32;
match raw_tx.try_send(RawFrame { aus, ts }) {
Ok(()) => {
sent_batches += 1;
au_seq = au_seq.wrapping_add(batch_len);
}
Err(std::sync::mpsc::TrySendError::Full(_)) => {
dropped_batches += 1;