feat(core,host,client): PyroWave datagram-aligned packets + partial-frame delivery (Phase 4, §4.4)
ci / web (push) Successful in 1m10s
ci / docs-site (push) Successful in 1m16s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 12s
decky / build-publish (push) Successful in 27s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 10s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 10s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
apple / swift (push) Successful in 4m59s
ci / rust (push) Failing after 6m17s
ci / bench (push) Successful in 6m36s
docker / deploy-docs (push) Successful in 23s
flatpak / build-publish (push) Failing after 8m29s
arch / build-publish (push) Successful in 11m11s
android / android (push) Successful in 12m33s
deb / build-publish (push) Successful in 14m8s
windows-host / package (push) Successful in 14m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m2s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m4s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m29s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m6s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 5m15s
release / apple (push) Successful in 26m8s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m48s
apple / screenshots (push) Successful in 20m29s

PyroWave AUs now packetize on the negotiated shard payload, so a lost datagram
costs a few wavelet blocks of localized blur rather than a whole frame — and the
client can render an aged-out lossy frame instead of freezing until the next one.

Host (opt-in, PyroWave only):
- The encoder packetizes at the shard payload behind a 4-byte window prefix
  (used-len u16 + kind u16). Whole packets pack into WIN_PACKED windows; a packet
  too large for one shard (PyroWave 32x32 blocks are atomic and can exceed a
  shard) rides a WIN_FRAG_FIRST/CONT/LAST chain. `set_wire_chunking()` joins the
  Encoder trait (forwarded through TrackedEncoder — the silent-no-op trap);
  EncodedFrame.chunk_aligned marks the AU.
- virtual_stream tags the AU with USER_FLAG_CHUNK_ALIGNED and re-applies chunking
  after every encoder (re)build, the adaptive-bitrate rebuild included.

Core:
- USER_FLAG_CHUNK_ALIGNED (0x40) wire bit. Reassembler opt-in
  (set_deliver_partial): a chunk-aligned frame that ages out with holes is handed
  over as Frame{complete:false} — received shards at their exact offsets, missing
  ranges zero-filled — instead of being dropped. Partials age out on a tight 30ms
  fuse (PARTIAL_WINDOW_NS) instead of the 120ms loss window: each frame is
  independently decodable, so an ancient partial has no value in a live stream.
  Newest-wins. A partial still counts as dropped for loss reporting.

Client (PyroWave decode):
- The session opts in when codec == PyroWave. The decoder walks the AU
  window-by-window, skipping zero (missing) windows and reassembling FRAG chains,
  then decodes whatever survived. A newest-decoded-index guard drops partials the
  pump has already moved past (no time-travel present).

Also fixes a redundant-closure clippy nit in the PyroWave planar-present path.

Validated on an RTX 5070 Ti under 2% netem loss with FEC pinned off: 60fps
sustained entirely via partials, e2e 43ms p50 (146ms before the fuse) vs 23ms
lossless, no keyframe-recovery chatter. Tests green: core 149, host 310 + the
GPU-gated encoder smoke (framed-window walk + FRAG reassembly + upstream
round-trip), client 26; clippy clean on the pyrowave feature combos.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-15 11:14:24 +02:00
parent 1fc9ef0050
commit 705a8baddf
21 changed files with 619 additions and 33 deletions
@@ -652,6 +652,7 @@ impl Encoder for NvencEncoder {
pts_ns,
keyframe: pkt.is_key(),
recovery_anchor: false,
chunk_aligned: false,
}))
}
// No packet ready yet (need another input frame).
@@ -1148,6 +1148,7 @@ impl Encoder for NvencCudaEncoder {
pts_ns,
keyframe,
recovery_anchor: anchor,
chunk_aligned: false,
}))
}
}
@@ -43,6 +43,13 @@ const IMPORT_CACHE_CAP: usize = 16;
/// Headroom over the per-frame rate budget for the packetized bitstream (block headers + meta;
/// the rate controller itself never exceeds the budget).
const BS_SLACK: usize = 256 * 1024;
/// Chunked-mode window framing (§4.4): 4-byte prefix per shard-sized window.
const WINDOW_PREFIX: usize = 4;
/// Window kinds: whole packets / an oversized packet's fragments.
const WIN_PACKED: u16 = 0;
const WIN_FRAG_FIRST: u16 = 1;
const WIN_FRAG_CONT: u16 = 2;
const WIN_FRAG_LAST: u16 = 3;
/// The DRM modifiers the PyroWave device can import as a SAMPLED image of the capture's
/// packed-RGB format. The capture advertises these for the pyrowave passthrough instead of
@@ -178,6 +185,10 @@ pub struct PyroWaveEncoder {
fps: u32,
/// Per-frame bitstream budget (hard CBR): `bitrate / (8 * fps)`.
frame_budget: usize,
/// Datagram-aligned mode (plan §4.4): packetize at this boundary and pad every codec
/// packet to it, so each wire shard carries whole self-delimiting packets. `None` =
/// one packet per AU (the dense MVP shape).
wire_chunk: Option<usize>,
bitstream: Vec<u8>,
pending: VecDeque<EncodedFrame>,
frame_count: u64,
@@ -532,6 +543,7 @@ impl PyroWaveEncoder {
height: h,
fps,
frame_budget,
wire_chunk: None,
bitstream: Vec::new(),
pending: VecDeque::new(),
frame_count: 0,
@@ -865,31 +877,100 @@ impl PyroWaveEncoder {
dev.wait_for_fences(&[self.fence], true, 5_000_000_000)
.context("pyrowave encode fence")?;
// ---- packetize: boundary = whole buffer, so the AU is exactly one pyrowave packet ----
// ---- packetize ----
// Dense (default): boundary = whole buffer → the AU is exactly one pyrowave packet.
// Datagram-aligned (§4.4, `set_wire_chunking`): boundary = the wire shard payload;
// each codec packet is zero-padded to the boundary so every shard carries whole
// self-delimiting packets — the client windows its parse and a lost shard costs
// only those blocks. Padding cost is small: the packetizer fills close to the
// boundary by design.
let cap = self.frame_budget + BS_SLACK;
self.bitstream.resize(cap, 0);
// Chunked mode reserves 4 bytes per window for the framing prefix.
let boundary = self.wire_chunk.map(|c| c - WINDOW_PREFIX).unwrap_or(cap);
let mut n: usize = 0;
pw_check(
pw::pyrowave_encoder_compute_num_packets(self.pw_enc, cap, &mut n),
pw::pyrowave_encoder_compute_num_packets(self.pw_enc, boundary, &mut n),
"compute_num_packets",
)?;
if n != 1 {
bail!("pyrowave: expected a single packet at boundary {cap}, got {n}");
if n == 0 || (self.wire_chunk.is_none() && n != 1) {
bail!("pyrowave: unexpected packet count {n} at boundary {boundary}");
}
let mut packet = pw::pyrowave_packet { offset: 0, size: 0 };
let mut packets = vec![pw::pyrowave_packet { offset: 0, size: 0 }; n];
let mut out_n: usize = 0;
pw_check(
pw::pyrowave_encoder_packetize(
self.pw_enc,
&mut packet,
cap,
packets.as_mut_ptr(),
boundary,
&mut out_n,
self.bitstream.as_mut_ptr() as *mut std::ffi::c_void,
cap,
),
"packetize",
)?;
let au = self.bitstream[packet.offset..packet.offset + packet.size].to_vec();
packets.truncate(out_n.max(1));
let au = if let Some(chunk) = self.wire_chunk {
// Window framing (§4.4): each `chunk`-sized window opens with a 4-byte prefix
// (u16 used-length + u16 kind) and carries either WHOLE self-delimiting codec
// packets (PACKED — several small ones share a window) or one fragment of an
// oversized packet (FRAG chain — pyrowave 32×32 blocks are atomic and may
// exceed a shard). A lost shard zeroes its window (used = 0) — the receiver
// skips it and drops any fragment chain it interrupts.
let payload_max = chunk - WINDOW_PREFIX;
let mut au: Vec<u8> = Vec::with_capacity((packets.len() + 1) * chunk);
// The currently-open PACKED window: (start offset of its prefix, bytes used).
let mut open: Option<(usize, usize)> = None;
let close = |au: &mut Vec<u8>, open: &mut Option<(usize, usize)>, chunk: usize| {
if let Some((start, used)) = open.take() {
au[start..start + 2].copy_from_slice(&(used as u16).to_le_bytes());
au[start + 2..start + 4].copy_from_slice(&WIN_PACKED.to_le_bytes());
au.resize(start + chunk, 0);
}
};
for p in &packets {
let bytes = &self.bitstream[p.offset..p.offset + p.size];
if p.size <= payload_max {
let fits = open.is_some_and(|(_, used)| used + p.size <= payload_max);
if !fits {
close(&mut au, &mut open, chunk);
let start = au.len();
au.resize(start + WINDOW_PREFIX, 0);
open = Some((start, 0));
}
au.extend_from_slice(bytes);
if let Some((_, used)) = open.as_mut() {
*used += p.size;
}
} else {
// Oversized packet: its own FRAG chain of full windows.
close(&mut au, &mut open, chunk);
let mut off = 0usize;
while off < p.size {
let take = (p.size - off).min(payload_max);
let kind = if off == 0 {
WIN_FRAG_FIRST
} else if off + take == p.size {
WIN_FRAG_LAST
} else {
WIN_FRAG_CONT
};
let start = au.len();
au.resize(start + WINDOW_PREFIX, 0);
au[start..start + 2].copy_from_slice(&(take as u16).to_le_bytes());
au[start + 2..start + 4].copy_from_slice(&kind.to_le_bytes());
au.extend_from_slice(&bytes[off..off + take]);
au.resize(start + chunk, 0);
off += take;
}
}
}
close(&mut au, &mut open, chunk);
au
} else {
let p = &packets[0];
self.bitstream[p.offset..p.offset + p.size].to_vec()
};
self.frame_count += 1;
self.pending.push_back(EncodedFrame {
data: au,
@@ -898,6 +979,7 @@ impl PyroWaveEncoder {
// whole recovery story (plan §1.2).
keyframe: true,
recovery_anchor: false,
chunk_aligned: self.wire_chunk.is_some(),
});
Ok(())
}
@@ -960,6 +1042,17 @@ impl Encoder for PyroWaveEncoder {
true
}
fn set_wire_chunking(&mut self, shard_payload: usize) {
// Sanity floor: a boundary below one block header + payload word is meaningless.
if shard_payload >= 64 {
self.wire_chunk = Some(shard_payload);
tracing::info!(
shard_payload,
"pyrowave: datagram-aligned packetization on (partial-frame loss mode)"
);
}
}
fn flush(&mut self) -> Result<()> {
// Synchronous per-frame encode: nothing buffered beyond `pending`.
Ok(())
@@ -1127,6 +1220,83 @@ mod tests {
);
}
// Datagram-aligned mode (§4.4): every emitted AU is a whole number of framed
// windows — 4-byte prefix (used-length + kind), whole packets or FRAG chains for
// oversized atomic blocks, zero padding after `used`. Walking + reassembling the
// fragments must reproduce a decodable packet stream.
enc.set_wire_chunking(1408);
enc.submit(&cpu_frame(w, h, 500, [90, 60, 30, 255]))
.expect("chunked submit");
let au = enc.poll().expect("poll").expect("chunked AU");
assert!(au.chunk_aligned);
assert_eq!(au.data.len() % 1408, 0, "AU is a whole number of windows");
// SAFETY: test-only FFI with locally-owned buffers.
unsafe {
let mut dev: pw::pyrowave_device = std::ptr::null_mut();
assert_eq!(
pw::pyrowave_create_default_device(&mut dev),
pw::pyrowave_result_PYROWAVE_SUCCESS
);
let dinfo = pw::pyrowave_decoder_create_info {
device: dev,
width: w as i32,
height: h as i32,
chroma: pw::pyrowave_chroma_subsampling_PYROWAVE_CHROMA_SUBSAMPLING_420,
fragment_path: false,
};
let mut dec: pw::pyrowave_decoder = std::ptr::null_mut();
assert_eq!(
pw::pyrowave_decoder_create(&dinfo, &mut dec),
pw::pyrowave_result_PYROWAVE_SUCCESS
);
let mut frag: Vec<u8> = Vec::new();
let mut pushed = 0usize;
for win in au.data.chunks(1408) {
let used = u16::from_le_bytes([win[0], win[1]]) as usize;
let kind = u16::from_le_bytes([win[2], win[3]]);
assert!(4 + used <= win.len(), "window overrun");
assert!(win[4 + used..].iter().all(|&b| b == 0), "non-zero padding");
let body = &win[4..4 + used];
match kind {
0 => {
assert_eq!(
pw::pyrowave_decoder_push_packet(
dec,
body.as_ptr() as *const _,
body.len()
),
pw::pyrowave_result_PYROWAVE_SUCCESS
);
pushed += body.len();
}
1 => frag = body.to_vec(),
2 => frag.extend_from_slice(body),
3 => {
frag.extend_from_slice(body);
assert_eq!(
pw::pyrowave_decoder_push_packet(
dec,
frag.as_ptr() as *const _,
frag.len()
),
pw::pyrowave_result_PYROWAVE_SUCCESS
);
pushed += frag.len();
frag.clear();
}
k => panic!("unknown window kind {k}"),
}
}
assert!(pushed > 0, "chunked AU carries real packets");
assert!(
pw::pyrowave_decoder_decode_is_ready(dec, false),
"chunked AU incomplete after framed walk"
);
pw::pyrowave_decoder_destroy(dec);
pw::pyrowave_device_destroy(dev);
}
enc.set_wire_chunking(0); // below the floor — back to dense
// In-place rate retarget + encoder rebuild both keep encoding.
assert!(enc.reconfigure_bitrate(100_000_000));
assert!(enc.reset());
@@ -297,6 +297,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<Option<En
pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(),
recovery_anchor: false,
chunk_aligned: false,
}))
}
Err(ffmpeg::Error::Other { errno })
@@ -1713,6 +1713,7 @@ impl VulkanVideoEncoder {
pts_ns: f.pts_ns,
keyframe: f.keyframe,
recovery_anchor: f.recovery_anchor,
chunk_aligned: false,
})
}