unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 26 Wiki Activity

fix(encode): harden loss-recovery correctness across host encoders (F1–F7)

Browse Source 
Phases 1–4 of design/encoder-recovery-hardening.md — make the shipped RFI/
freeze-until-reanchor recovery honest and rebuild-safe across every backend.

F1 — frame-index domain desync: the encode loop now owns a session-lifetime
`au_seq`; `Encoder::submit_indexed(au_seq + inflight)` pins NVENC inputTimeStamp
and AMF LTR slots to the WIRE frame index, so `invalidate_ref_frames` compares
client frame numbers in the same domain and survives adaptive-bitrate rebuilds
(an internal counter desynced on the first rebuild → RFI silently dead / an AMF
force-ref onto a never-decoded frame). `FrameMsg.frame_index` →
`Session::seal_frame_at`; GameStream gets the same via `VideoPacketizer::
packetize(.., Some(idx))`.

F2 — Windows NVENC left the client frozen ~1s per loss: NVENC RFI was
transparent (no anchor tag) while the session glue armed the 750ms IDR cooldown,
so the freeze only lifted on the ~1s keyframe re-ask. NVENC now mirrors AMF —
`pending_anchor` tags the first post-invalidate AU (the clean re-anchor
P-frame) `recovery_anchor`, incl. the covering-range dedupe re-arm; the client
lifts at ~RTT.

F3 — speed-test probe filler burned video frame indexes: moved to its own index
space (`Packetizer::alloc_probe_index` + `Session::submit_probe_frame`) with a
second client reassembly window routed on FLAG_PROBE, gated on the new
VIDEO_CAP_PROBE_SEQ Hello bit (mid-session probes declined for older clients).

F4 — RFI range sanity cap: forward gaps wider than `packet::RFI_MAX_RANGE` (256)
resync via keyframe instead of an out-of-range RFI, host- and client-side
(client huge-gap → keyframe in `RfiRecovery::observe` + the pf-client-core pump).

F5 — reset() parity: Windows NVENC (teardown + lazy re-init), Linux VAAPI
(drop-inner), Linux NVENC (reopen from stored OpenArgs) now give the stall
watchdog a heal lever instead of ending the session.

F6 — sw.rs `pending: VecDeque` (was `Option`), killing the silent AU drop at
capturer pipeline depth > 1. F7 — doc sweep on the RFI/anchor comments.

Verified: punktfunk-core lib tests (macOS + Linux), full punktfunk-host suite on
Linux (RTX 5070 Ti), Windows compile. Owed: the on-glass client matrix (F2
freeze A/B, AMF LTR spike across a bitrate rebuild).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Enrico Bühler
2026-07-12 11:17:19 +02:00
parent 0dc414f197
commit fdda7144ed
15 changed files with 801 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/punktfunk-host/src/gamestream/stream.rs
+39 -8
View File
@@ -436,7 +436,10 @@ fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec<u8>]) -> std::io::Result<()> {
/// behind encode (measured ~3 ms/frame at 4K, which capped GameStream's frame rate well below what
/// the encoder alone can sustain).
struct RawFrame {
aus: Vec<(Vec<u8>, FrameType)>,
/// `(bitstream, type, wire frameIndex)` per AU. The stream loop assigns the index (it owns
/// the numbering — see its `au_seq`), so the encoder's RFI bookkeeping stays 1:1 with what
/// Moonlight sees across mid-stream encoder rebuilds.
aus: Vec<(Vec<u8>, FrameType, u32)>,
ts: u32,
}
@@ -460,8 +463,8 @@ fn spawn_packetizer(
crate::punktfunk1::boost_thread_priority(false);
while let Ok(frame) = rx.recv() {
let mut batch: PacketBatch = Vec::new();
for (au, ft) in frame.aus {
batch.extend(pk.packetize(&au, ft, frame.ts));
for (au, ft, idx) in frame.aus {
batch.extend(pk.packetize(&au, ft, frame.ts, Some(idx)));
}
if batch.is_empty() {
continue;
@@ -660,6 +663,16 @@ fn stream_body(
// routed through the same coalesce gate as client IDR requests so a burst of drops (congestion)
// can't become an IDR storm.
let mut recover_after_drop = false;
// The stream's wire frameIndex numbering, owned HERE (the index of the next AU handed to the
// packetizer thread; a dropped-at-the-queue frame consumes none). A submission's future index
// is `au_seq + enc_inflight` (AUs are emitted FIFO, one per submission); passing it to
// `Encoder::submit_indexed` keeps the encoder's RFI bookkeeping 1:1 with Moonlight's frame
// numbers across the in-place encoder rebuild above (an internal counter would desync there).
// A pipeline-head drop desyncs the prediction by the dropped AU count for the frames already
// in flight — bounded and self-healing: the drop arms `recover_after_drop`, whose forced IDR
// resets the encoder's reference state (stale LTR/DPB bookkeeping dies with it).
let mut au_seq: u32 = 0;
let mut enc_inflight: u32 = 0;
while running.load(Ordering::SeqCst) {
let tick = Instant::now();
@@ -728,6 +741,10 @@ fn stream_body(
enc.request_keyframe();
last_keyframe = Some(Instant::now());
next_frame = Instant::now();
// The old encoder died with its in-flight submissions — their AUs will never
// arrive, so the numbering prediction restarts at `au_seq` (the fresh encoder's
// reference state is empty, so the reused predictions meet no stale bookkeeping).
enc_inflight = 0;
tracing::info!("gamestream: source rebuilt — stream continues");
continue;
}
@@ -742,7 +759,13 @@ fn stream_body(
if let Some((first, last)) = rfi_range.lock().unwrap().take() {
// Prefer reference-frame invalidation when the encoder supports it (no costly IDR
// spike); otherwise — or if the range is too old to invalidate — fall back to a keyframe.
if !(supports_rfi && enc.invalidate_ref_frames(first, last)) {
// Sanity-cap the range first: wider than RFI_MAX_RANGE exceeds any encoder's reference
// history (or is a phantom range from a desynced counter) — keyframe, never a
// force-reference that could ship corruption as a clean frame.
let width = (last as u32).wrapping_sub(first as u32);
if width > punktfunk_core::packet::RFI_MAX_RANGE
|| !(supports_rfi && enc.invalidate_ref_frames(first, last))
{
want_keyframe = true;
}
}
@@ -766,21 +789,27 @@ fn stream_body(
tracing::debug!("video: keyframe request coalesced (IDR still in flight)");
}
}
enc.submit(&frame).context("encoder submit")?;
enc.submit_indexed(&frame, au_seq.wrapping_add(enc_inflight))
.context("encoder submit")?;
enc_inflight = enc_inflight.wrapping_add(1);
let t_enc = tick.elapsed();
// 90 kHz RTP timestamp from wall-clock, so a variable capture rate stays correct.
let ts = (stream_start.elapsed().as_secs_f64() * 90_000.0) as u32;
// Drain the encoder's access units (owned buffers) — FEC/packetization runs on the
// packetizer thread, off this loop, so it never serializes behind encode.
let mut aus: Vec<(Vec<u8>, FrameType)> = Vec::new();
// packetizer thread, off this loop, so it never serializes behind encode. Each AU is
// stamped with its wire frameIndex here (`au_seq + position`); the numbering only
// ADVANCES if the batch is actually enqueued below (a dropped batch consumes none).
let mut aus: Vec<(Vec<u8>, FrameType, u32)> = Vec::new();
while let Some(au) = enc.poll().context("encoder poll")? {
let ft = if au.keyframe {
FrameType::Idr
} else {
FrameType::P
};
aus.push((au.data, ft));
let idx = au_seq.wrapping_add(aus.len() as u32);
aus.push((au.data, ft, idx));
enc_inflight = enc_inflight.saturating_sub(1);
}
let t_pkt = tick.elapsed();
@@ -788,9 +817,11 @@ fn stream_body(
// (packetizer, or the paced sender behind it) is behind — drop this frame (FEC/RFI covers the
// client) and keep encoding, so a downstream stall can never cap the encode rate.
if !aus.is_empty() {
let batch_len = aus.len() as u32;
match raw_tx.try_send(RawFrame { aus, ts }) {
Ok(()) => {
sent_batches += 1;
au_seq = au_seq.wrapping_add(batch_len);
}
Err(std::sync::mpsc::TrySendError::Full(_)) => {
dropped_batches += 1;
crates/punktfunk-host/src/gamestream/video.rs
+15 -7
View File
@@ -69,14 +69,22 @@ impl VideoPacketizer {
}
/// Packetize one encoded AU into wire datagrams (data shards + Cauchy RS parity shards).
///
/// `frame_index`: `Some(i)` stamps the caller's index (the stream loop owns the numbering so
/// the encoder's RFI bookkeeping stays 1:1 with the wire across mid-stream encoder rebuilds —
/// see `Encoder::submit_indexed`); `None` draws from the internal counter (tests/harnesses).
pub fn packetize(
&mut self,
au: &[u8],
frame_type: FrameType,
timestamp_90k: u32,
frame_index: Option<u32>,
) -> Vec<Vec<u8>> {
let frame_index = self.frame_index;
self.frame_index = self.frame_index.wrapping_add(1);
let frame_index = frame_index.unwrap_or_else(|| {
let i = self.frame_index;
self.frame_index = i.wrapping_add(1);
i
});
let pps = self.payload_per_shard;
let blocksize = SHARD_HEADER + pps; // = packet_size + 16
let pct = self.fec_percentage;
@@ -235,7 +243,7 @@ mod tests {
let mut pk = VideoPacketizer::new(1392, 0, 0); // data-only; pps = 1392+16-32 = 1376
assert_eq!(pk.payload_per_shard, 1376);
let au = vec![0xABu8; 4000]; // 8+4000 = 4008 → ceil(4008/1376) = 3 data shards
let pkts = pk.packetize(&au, FrameType::Idr, 90_000);
let pkts = pk.packetize(&au, FrameType::Idr, 90_000, None);
assert_eq!(pkts.len(), 3);
for p in &pkts {
assert_eq!(p.len(), SHARD_HEADER + 1376);
@@ -266,7 +274,7 @@ mod tests {
for ps in [0usize, 15, 16, 17, 32] {
let mut pk = VideoPacketizer::new(ps, 20, 2);
assert!(pk.payload_per_shard >= 1, "pps must never be 0 (ps={ps})");
let _ = pk.packetize(&[0xCDu8; 200], FrameType::Idr, 0); // must not panic
let _ = pk.packetize(&[0xCDu8; 200], FrameType::Idr, 0, None); // must not panic
}
}
@@ -274,7 +282,7 @@ mod tests {
fn multi_block_split() {
let mut pk = VideoPacketizer::new(1392, 0, 0); // data-only
let au = vec![0u8; 600_000];
let pkts = pk.packetize(&au, FrameType::P, 0);
let pkts = pk.packetize(&au, FrameType::P, 0, None);
let total = (8 + au.len()).div_ceil(1376);
assert_eq!(pkts.len(), total);
let n_blocks = total.div_ceil(255).clamp(1, 4);
@@ -286,7 +294,7 @@ mod tests {
fn emits_parity_shards() {
let mut pk = VideoPacketizer::new(1392, 20, 0); // pps = 1376, 20% FEC
let au = vec![0xABu8; 4000]; // 8+4000 = 4008 → 3 data shards (k=3)
let pkts = pk.packetize(&au, FrameType::Idr, 0);
let pkts = pk.packetize(&au, FrameType::Idr, 0, None);
// m = ceil(3*20/100) = 1 parity shard → 4 packets; wire_pct = 100*1/3 = 33.
assert_eq!(pkts.len(), 4);
for p in &pkts {
@@ -313,7 +321,7 @@ mod tests {
fn parity_recovers_full_datagram_incl_flags() {
let mut pk = VideoPacketizer::new(1392, 50, 0); // high pct → plenty of parity
let au = vec![0x5Au8; 4000]; // k = 3
let pkts = pk.packetize(&au, FrameType::Idr, 0);
let pkts = pk.packetize(&au, FrameType::Idr, 0, None);
let k = 3usize;
let m = pkts.len() - k;
assert!(m >= 1);