feat(core,host,client): PyroWave datagram-aligned packets + partial-frame delivery (Phase 4, §4.4)
ci / web (push) Successful in 1m10s
ci / docs-site (push) Successful in 1m16s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 12s
decky / build-publish (push) Successful in 27s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 10s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 10s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
apple / swift (push) Successful in 4m59s
ci / rust (push) Failing after 6m17s
ci / bench (push) Successful in 6m36s
docker / deploy-docs (push) Successful in 23s
flatpak / build-publish (push) Failing after 8m29s
arch / build-publish (push) Successful in 11m11s
android / android (push) Successful in 12m33s
deb / build-publish (push) Successful in 14m8s
windows-host / package (push) Successful in 14m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m2s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m4s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m29s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m6s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 5m15s
release / apple (push) Successful in 26m8s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m48s
apple / screenshots (push) Successful in 20m29s
ci / web (push) Successful in 1m10s
ci / docs-site (push) Successful in 1m16s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 12s
decky / build-publish (push) Successful in 27s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 10s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 10s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
apple / swift (push) Successful in 4m59s
ci / rust (push) Failing after 6m17s
ci / bench (push) Successful in 6m36s
docker / deploy-docs (push) Successful in 23s
flatpak / build-publish (push) Failing after 8m29s
arch / build-publish (push) Successful in 11m11s
android / android (push) Successful in 12m33s
deb / build-publish (push) Successful in 14m8s
windows-host / package (push) Successful in 14m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m2s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m4s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m29s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m6s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 5m15s
release / apple (push) Successful in 26m8s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m48s
apple / screenshots (push) Successful in 20m29s
PyroWave AUs now packetize on the negotiated shard payload, so a lost datagram
costs a few wavelet blocks of localized blur rather than a whole frame — and the
client can render an aged-out lossy frame instead of freezing until the next one.
Host (opt-in, PyroWave only):
- The encoder packetizes at the shard payload behind a 4-byte window prefix
(used-len u16 + kind u16). Whole packets pack into WIN_PACKED windows; a packet
too large for one shard (PyroWave 32x32 blocks are atomic and can exceed a
shard) rides a WIN_FRAG_FIRST/CONT/LAST chain. `set_wire_chunking()` joins the
Encoder trait (forwarded through TrackedEncoder — the silent-no-op trap);
EncodedFrame.chunk_aligned marks the AU.
- virtual_stream tags the AU with USER_FLAG_CHUNK_ALIGNED and re-applies chunking
after every encoder (re)build, the adaptive-bitrate rebuild included.
Core:
- USER_FLAG_CHUNK_ALIGNED (0x40) wire bit. Reassembler opt-in
(set_deliver_partial): a chunk-aligned frame that ages out with holes is handed
over as Frame{complete:false} — received shards at their exact offsets, missing
ranges zero-filled — instead of being dropped. Partials age out on a tight 30ms
fuse (PARTIAL_WINDOW_NS) instead of the 120ms loss window: each frame is
independently decodable, so an ancient partial has no value in a live stream.
Newest-wins. A partial still counts as dropped for loss reporting.
Client (PyroWave decode):
- The session opts in when codec == PyroWave. The decoder walks the AU
window-by-window, skipping zero (missing) windows and reassembling FRAG chains,
then decodes whatever survived. A newest-decoded-index guard drops partials the
pump has already moved past (no time-travel present).
Also fixes a redundant-closure clippy nit in the PyroWave planar-present path.
Validated on an RTX 5070 Ti under 2% netem loss with FEC pinned off: 60fps
sustained entirely via partials, e2e 43ms p50 (146ms before the fuse) vs 23ms
lossless, no keyframe-recovery chatter. Tests green: core 149, host 310 + the
GPU-gated encoder smoke (framed-window walk + FRAG reassembly + upstream
round-trip), client 26; clippy clean on the pyrowave feature combos.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -289,7 +289,12 @@ fn pump(
|
||||
let built = if connector.codec == punktfunk_core::quic::CODEC_PYROWAVE {
|
||||
let mode = connector.mode();
|
||||
match params.vulkan.as_ref() {
|
||||
Some(vk) => Decoder::new_pyrowave(vk, mode.width, mode.height),
|
||||
Some(vk) => Decoder::new_pyrowave(
|
||||
vk,
|
||||
mode.width,
|
||||
mode.height,
|
||||
connector.shard_payload as usize,
|
||||
),
|
||||
None => Err(anyhow::anyhow!(
|
||||
"pyrowave session without a presenter device"
|
||||
)),
|
||||
@@ -324,6 +329,8 @@ fn pump(
|
||||
// step, drift) keep the capture-clock latency stats honest — never cached at session start.
|
||||
let clock_offset_live = connector.clock_offset_shared();
|
||||
let mut total_frames = 0u64;
|
||||
// Newest frame index handed to the decoder — the staleness bar for late partials.
|
||||
let mut newest_decoded_idx: Option<u32> = None;
|
||||
let mut window_start = Instant::now();
|
||||
let mut frames_n = 0u32;
|
||||
let mut bytes_n = 0u64;
|
||||
@@ -432,7 +439,21 @@ fn pump(
|
||||
}
|
||||
None => next_expected_index = Some(frame.frame_index.wrapping_add(1)),
|
||||
}
|
||||
match decoder.decode(&frame.data) {
|
||||
// A PARTIAL that lost the race (a newer frame already decoded) is pure
|
||||
// time travel — skip it; each PyroWave frame is independent, so nothing
|
||||
// downstream needs it. Completes keep the normal path (reorder is handled
|
||||
// by the continuity gate).
|
||||
if !frame.complete
|
||||
&& newest_decoded_idx
|
||||
.is_some_and(|n: u32| n.wrapping_sub(frame.frame_index) <= u32::MAX / 2)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
newest_decoded_idx = Some(match newest_decoded_idx {
|
||||
Some(n) if frame.frame_index.wrapping_sub(n) > u32::MAX / 2 => n,
|
||||
_ => frame.frame_index,
|
||||
});
|
||||
match decoder.decode_frame(&frame.data, frame.flags, frame.complete) {
|
||||
Ok(Some(image)) => {
|
||||
// Fold this decoded frame through the shared freeze gate: it reads the AU's
|
||||
// re-anchor wire flags (FLAG_SOF IDR marker / RECOVERY_ANCHOR / RECOVERY_POINT),
|
||||
|
||||
@@ -572,10 +572,18 @@ impl Decoder {
|
||||
/// compute on the presenter's device, no FFmpeg. `codec_id` is irrelevant (kept as
|
||||
/// HEVC so an — impossible — demotion path stays well-formed).
|
||||
#[cfg(all(target_os = "linux", feature = "pyrowave"))]
|
||||
pub fn new_pyrowave(vk: &VulkanDecodeDevice, width: u32, height: u32) -> Result<Decoder> {
|
||||
pub fn new_pyrowave(
|
||||
vk: &VulkanDecodeDevice,
|
||||
width: u32,
|
||||
height: u32,
|
||||
shard_payload: usize,
|
||||
) -> Result<Decoder> {
|
||||
Ok(Decoder {
|
||||
backend: Backend::PyroWave(Box::new(crate::video_pyrowave::PyroWaveDecoder::new(
|
||||
vk, width, height,
|
||||
vk,
|
||||
width,
|
||||
height,
|
||||
shard_payload,
|
||||
)?)),
|
||||
codec_id: ffmpeg::codec::Id::HEVC,
|
||||
vaapi_fails: 0,
|
||||
@@ -610,8 +618,24 @@ impl Decoder {
|
||||
/// pump asks the host for a fresh IDR — under the infinite GOP nothing else resyncs a
|
||||
/// rebuilt/erroring decoder, so skipping this leaves the picture gray/frozen for good.
|
||||
pub fn decode(&mut self, au: &[u8]) -> Result<Option<DecodedImage>> {
|
||||
self.decode_frame(au, 0, true)
|
||||
}
|
||||
|
||||
/// [`decode`](Self::decode) with the AU's wire facts: `user_flags` (chunk-aligned AUs
|
||||
/// are parsed in shard windows — [`punktfunk_core::packet::USER_FLAG_CHUNK_ALIGNED`])
|
||||
/// and completeness (`false` = a partial delivery; only the PyroWave backend decodes
|
||||
/// those — as one frame of localized blur, plan §4.4).
|
||||
pub fn decode_frame(
|
||||
&mut self,
|
||||
au: &[u8],
|
||||
user_flags: u32,
|
||||
complete: bool,
|
||||
) -> Result<Option<DecodedImage>> {
|
||||
let result = match &mut self.backend {
|
||||
Backend::Vulkan(v) => v.decode(au).map(|f| f.map(DecodedImage::VkFrame)),
|
||||
Backend::Vulkan(v) => {
|
||||
debug_assert!(complete, "partial AUs are pyrowave-only");
|
||||
v.decode(au).map(|f| f.map(DecodedImage::VkFrame))
|
||||
}
|
||||
#[cfg(target_os = "linux")]
|
||||
Backend::Vaapi(v) => v.decode(au).map(|f| f.map(DecodedImage::Dmabuf)),
|
||||
#[cfg(windows)]
|
||||
@@ -620,7 +644,12 @@ impl Decoder {
|
||||
// error; the pump surfaces it and the session falls back to HEVC by
|
||||
// renegotiation (plan §4.6), not by decoder swap.
|
||||
#[cfg(all(target_os = "linux", feature = "pyrowave"))]
|
||||
Backend::PyroWave(p) => return Ok(p.decode(au)?.map(DecodedImage::PyroWave)),
|
||||
Backend::PyroWave(p) => {
|
||||
let aligned = user_flags & punktfunk_core::packet::USER_FLAG_CHUNK_ALIGNED != 0;
|
||||
return Ok(p
|
||||
.decode_frame(au, aligned, complete)?
|
||||
.map(DecodedImage::PyroWave));
|
||||
}
|
||||
Backend::Software(s) => return Ok(s.decode(au)?.map(DecodedImage::Cpu)),
|
||||
};
|
||||
match result {
|
||||
|
||||
@@ -190,6 +190,9 @@ pub struct PyroWaveDecoder {
|
||||
fence: vk::Fence,
|
||||
width: u32,
|
||||
height: u32,
|
||||
/// The wire shard payload — the parse-window size for chunk-aligned AUs (§4.4): each
|
||||
/// window holds whole self-delimiting codec packets, zero-padded to the window.
|
||||
wire_window: usize,
|
||||
}
|
||||
|
||||
// SAFETY: used only from the single decode thread; the shared-queue accesses go through
|
||||
@@ -197,7 +200,12 @@ pub struct PyroWaveDecoder {
|
||||
unsafe impl Send for PyroWaveDecoder {}
|
||||
|
||||
impl PyroWaveDecoder {
|
||||
pub fn new(vkd: &VulkanDecodeDevice, width: u32, height: u32) -> Result<PyroWaveDecoder> {
|
||||
pub fn new(
|
||||
vkd: &VulkanDecodeDevice,
|
||||
width: u32,
|
||||
height: u32,
|
||||
shard_payload: usize,
|
||||
) -> Result<PyroWaveDecoder> {
|
||||
if !vkd.pyrowave_decode {
|
||||
bail!("presenter device lacks the PyroWave compute feature set");
|
||||
}
|
||||
@@ -207,13 +215,14 @@ impl PyroWaveDecoder {
|
||||
// SAFETY: the handles in `vkd` are the presenter's live instance/device (it
|
||||
// outlives the decoder — same contract the FFmpeg Vulkan backend relies on);
|
||||
// `Hold` pins the reconstructed create-infos for the pyrowave device's lifetime.
|
||||
unsafe { Self::new_inner(vkd, width, height) }
|
||||
unsafe { Self::new_inner(vkd, width, height, shard_payload) }
|
||||
}
|
||||
|
||||
unsafe fn new_inner(
|
||||
vkd: &VulkanDecodeDevice,
|
||||
width: u32,
|
||||
height: u32,
|
||||
shard_payload: usize,
|
||||
) -> Result<PyroWaveDecoder> {
|
||||
let static_fn = ash::StaticFn {
|
||||
get_instance_proc_addr: std::mem::transmute::<usize, vk::PFN_vkGetInstanceProcAddr>(
|
||||
@@ -380,25 +389,113 @@ impl PyroWaveDecoder {
|
||||
fence,
|
||||
width,
|
||||
height,
|
||||
wire_window: shard_payload.max(64),
|
||||
})
|
||||
}
|
||||
|
||||
/// One AU in → one frame out (the AU is a complete pyrowave frame: one packet).
|
||||
pub fn decode(&mut self, au: &[u8]) -> Result<Option<PyroWavePlanarFrame>> {
|
||||
/// One AU in → one frame out. `aligned` = the AU is shard-window chunked (each
|
||||
/// `wire_window` holds whole self-delimiting packets, zero-padded — walk and strip);
|
||||
/// `complete` = every shard arrived (a partial decodes anyway: missing blocks are
|
||||
/// localized blur for exactly this frame, §4.4).
|
||||
pub fn decode_frame(
|
||||
&mut self,
|
||||
au: &[u8],
|
||||
aligned: bool,
|
||||
complete: bool,
|
||||
) -> Result<Option<PyroWavePlanarFrame>> {
|
||||
// SAFETY: single decode thread; all handles owned/pinned by `self`; queue access
|
||||
// serialized under the device-wide QueueLock; the fence bounds GPU completion
|
||||
// before the frame is handed to the presenter.
|
||||
unsafe { self.decode_inner(au) }
|
||||
unsafe { self.decode_inner(au, aligned, complete) }
|
||||
}
|
||||
|
||||
unsafe fn decode_inner(&mut self, au: &[u8]) -> Result<Option<PyroWavePlanarFrame>> {
|
||||
pw_check(
|
||||
pw::pyrowave_decoder_push_packet(self.pw_dec, au.as_ptr() as *const c_void, au.len()),
|
||||
"push_packet",
|
||||
)?;
|
||||
// The reassembler delivers complete AUs only, so a frame is ready per push; a
|
||||
// stale/duplicate packet (sequence rewind) simply isn't — skip, no error.
|
||||
if !pw::pyrowave_decoder_decode_is_ready(self.pw_dec, false) {
|
||||
/// Consume one framed shard window (§4.4): a 4-byte prefix (u16 used-length + u16
|
||||
/// kind) then either WHOLE self-delimiting codec packets (PACKED) or one fragment of
|
||||
/// an oversized packet (FRAG chain). A lost shard arrives as a zeroed window
|
||||
/// (used = 0) — skipped, and it breaks any fragment chain it interrupts (that
|
||||
/// packet's blocks are unusable without their end; dropping them is the §4.4 blur).
|
||||
unsafe fn push_window(&mut self, win: &[u8], frag: &mut Vec<u8>) -> Result<()> {
|
||||
if win.len() < 4 {
|
||||
return Ok(());
|
||||
}
|
||||
let used = u16::from_le_bytes([win[0], win[1]]) as usize;
|
||||
let kind = u16::from_le_bytes([win[2], win[3]]);
|
||||
if used == 0 || 4 + used > win.len() {
|
||||
frag.clear(); // missing / garbage window — drop any chain in progress
|
||||
return Ok(());
|
||||
}
|
||||
let body = &win[4..4 + used];
|
||||
match kind {
|
||||
0 => {
|
||||
frag.clear();
|
||||
pw_check(
|
||||
pw::pyrowave_decoder_push_packet(
|
||||
self.pw_dec,
|
||||
body.as_ptr() as *const c_void,
|
||||
body.len(),
|
||||
),
|
||||
"push_packet",
|
||||
)
|
||||
}
|
||||
1 => {
|
||||
frag.clear();
|
||||
frag.extend_from_slice(body);
|
||||
Ok(())
|
||||
}
|
||||
2 => {
|
||||
if !frag.is_empty() {
|
||||
frag.extend_from_slice(body);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
3 => {
|
||||
if !frag.is_empty() {
|
||||
frag.extend_from_slice(body);
|
||||
let r = pw_check(
|
||||
pw::pyrowave_decoder_push_packet(
|
||||
self.pw_dec,
|
||||
frag.as_ptr() as *const c_void,
|
||||
frag.len(),
|
||||
),
|
||||
"push_packet (fragmented)",
|
||||
);
|
||||
frag.clear();
|
||||
return r;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
_ => {
|
||||
frag.clear();
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
unsafe fn decode_inner(
|
||||
&mut self,
|
||||
au: &[u8],
|
||||
aligned: bool,
|
||||
complete: bool,
|
||||
) -> Result<Option<PyroWavePlanarFrame>> {
|
||||
if aligned {
|
||||
let mut frag: Vec<u8> = Vec::new();
|
||||
for win in au.chunks(self.wire_window) {
|
||||
self.push_window(win, &mut frag)?;
|
||||
}
|
||||
} else {
|
||||
pw_check(
|
||||
pw::pyrowave_decoder_push_packet(
|
||||
self.pw_dec,
|
||||
au.as_ptr() as *const c_void,
|
||||
au.len(),
|
||||
),
|
||||
"push_packet",
|
||||
)?;
|
||||
}
|
||||
// A complete AU that isn't ready is a stale/duplicate (sequence rewind) — skip.
|
||||
// A PARTIAL is decoded regardless: missing wavelet blocks reconstruct as zeros,
|
||||
// i.e. localized blur for exactly this one frame (the next is complete again).
|
||||
if complete && !pw::pyrowave_decoder_decode_is_ready(self.pw_dec, false) {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user