feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s

Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets
AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike.

Client (pf-client-core): after a reference loss the hardware decoder conceals the
missing-reference deltas (on RADV, a gray plate with new motion painted over) and
returns Ok. The pump now freezes on the last good picture until a clean re-anchor
instead of showing the concealment — lifting on a real IDR, an intra-refresh
recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The
frame_index gap is the early, precise loss signal and drives an RFI request.

Host recovery signals (inert unless the backend supports them):
- USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR).
- USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame.

AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API
has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the
only clean-recovery lever is user LTR: mark frames as long-term references, and on
loss force the next frame to re-reference the newest known-good one — a clean
P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for
AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR
before the loss; a range older than the live slots falls back to a keyframe.

Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi().
Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success
anchors the keyframe cooldown so the client's frames_dropped echo of the same loss
is coalesced away rather than emitting a redundant IDR.

Spike: synthetic NV12 GPU source for headless AMF encoder testing.

Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests
(incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux;
punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90
forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-11 17:31:17 +02:00
parent 890c7531d8
commit e55ff1bb28
21 changed files with 1228 additions and 33 deletions
+3
View File
@@ -466,5 +466,8 @@ pub mod dxgi;
#[cfg(target_os = "windows")]
#[path = "capture/windows/idd_push.rs"]
pub mod idd_push;
#[cfg(target_os = "windows")]
#[path = "capture/windows/synthetic_nv12.rs"]
pub mod synthetic_nv12;
#[cfg(target_os = "linux")]
mod linux;
@@ -0,0 +1,181 @@
//! A headless synthetic **NV12 D3D11** capture source for exercising the GPU encoders on Windows
//! without a real capture session.
//!
//! The native AMF path (and the D3D11 zero-copy NVENC/QSV paths) require an NV12 texture that lives
//! on the GPU — the CPU-Bgrx [`SyntheticCapturer`](crate::capture::SyntheticCapturer) can't provide
//! one, and DXGI Desktop Duplication can't create one under an ssh session-0 (E_ACCESSDENIED). This
//! source builds an NV12 texture on the selected render adapter and fills it with a **moving** luma
//! ramp each frame, so the encoder sees genuine motion (P-frame residuals + the intra-refresh wave
//! under content change) — exactly what an intra-refresh recovery validation needs. Driven by
//! `spike --source synthetic-nv12`.
use crate::capture::dxgi::{make_device, D3d11Frame};
use crate::capture::{CapturedFrame, Capturer, FramePayload, PixelFormat};
use anyhow::{Context, Result};
use windows::Win32::Graphics::Direct3D11::{
ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D, D3D11_BIND_SHADER_RESOURCE,
D3D11_CPU_ACCESS_WRITE, D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_WRITE, D3D11_TEXTURE2D_DESC,
D3D11_USAGE, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING,
};
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_NV12, DXGI_SAMPLE_DESC};
use windows::Win32::Graphics::Dxgi::{CreateDXGIFactory1, IDXGIAdapter1, IDXGIFactory4};
/// Synthetic NV12 frames on the GPU. Owns its own D3D11 device + immediate context and two NV12
/// textures: a CPU-writable STAGING scratch it fills each frame, and a DEFAULT texture it copies
/// into and hands to the encoder. The encoder copies out of the DEFAULT texture synchronously
/// (spike drives capture→submit→poll on one thread), so reusing one DEFAULT texture is sound.
pub struct SyntheticNv12Capturer {
device: ID3D11Device,
context: ID3D11DeviceContext,
default_tex: ID3D11Texture2D,
staging: ID3D11Texture2D,
width: u32,
height: u32,
fps: u32,
frame_idx: u64,
}
// SAFETY: mirrors `D3d11Frame`'s reasoning — the device is created free-threaded (`make_device`
// passes no `SINGLETHREADED` flag) and D3D11 uses interlocked COM refcounting, so moving the whole
// capturer (device + immediate context + textures) to its owning thread and using it only there is
// sound. The value is moved, never aliased (no `Sync`), so the single-threaded immediate context is
// never touched concurrently.
unsafe impl Send for SyntheticNv12Capturer {}
impl SyntheticNv12Capturer {
pub fn new(width: u32, height: u32, fps: u32) -> Result<Self> {
// NV12 is 4:2:0 — both dimensions must be even (the chroma plane is width/2 × height/2).
let width = (width & !1).max(2);
let height = (height & !1).max(2);
// SAFETY: a self-contained builder owning every handle it creates; each COM call is checked
// and the returned owners drop with their wrappers.
unsafe {
let adapter = resolve_render_adapter().context("resolve render adapter for NV12 source")?;
let (device, context) = make_device(&adapter).context("create D3D11 device")?;
let default_tex = create_nv12(
&device,
width,
height,
D3D11_USAGE_DEFAULT,
0,
D3D11_BIND_SHADER_RESOURCE.0 as u32,
)
.context("create NV12 default texture")?;
let staging = create_nv12(
&device,
width,
height,
D3D11_USAGE_STAGING,
D3D11_CPU_ACCESS_WRITE.0 as u32,
0,
)
.context("create NV12 staging texture")?;
Ok(SyntheticNv12Capturer {
device,
context,
default_tex,
staging,
width,
height,
fps,
frame_idx: 0,
})
}
}
}
impl Capturer for SyntheticNv12Capturer {
fn next_frame(&mut self) -> Result<CapturedFrame> {
let pts_ns = self.frame_idx * 1_000_000_000 / self.fps.max(1) as u64;
// SAFETY: Map/Unmap/CopyResource on this capturer's own single-threaded immediate context;
// all writes stay within the mapped NV12 surface (Y: H rows of RowPitch; UV: H/2 rows of
// RowPitch beginning at RowPitch*H — the standard NV12 plane layout).
unsafe {
let mut map = D3D11_MAPPED_SUBRESOURCE::default();
self.context
.Map(&self.staging, 0, D3D11_MAP_WRITE, 0, Some(&mut map))
.context("Map(NV12 staging)")?;
let pitch = map.RowPitch as usize;
let base = map.pData as *mut u8;
// A diagonal luma ramp that shifts 4 codes/frame — strong, deterministic motion.
let shift = (self.frame_idx as u32).wrapping_mul(4);
for y in 0..self.height {
let row = base.add(y as usize * pitch);
for x in 0..self.width {
*row.add(x as usize) = x.wrapping_add(y).wrapping_add(shift) as u8;
}
}
// UV plane (neutral gray = 128) at offset RowPitch*H: H/2 rows, `width` bytes each
// (width/2 interleaved Cb,Cr pairs).
let uv = base.add(pitch * self.height as usize);
for r in 0..(self.height / 2) {
let row = uv.add(r as usize * pitch);
for c in 0..self.width {
*row.add(c as usize) = 128;
}
}
self.context.Unmap(&self.staging, 0);
self.context.CopyResource(&self.default_tex, &self.staging);
}
self.frame_idx += 1;
Ok(CapturedFrame {
width: self.width,
height: self.height,
pts_ns,
format: PixelFormat::Nv12,
payload: FramePayload::D3d11(D3d11Frame {
texture: self.default_tex.clone(),
device: self.device.clone(),
}),
})
}
}
/// Resolve the same render adapter the encoder will pick (`PUNKTFUNK_RENDER_ADAPTER` / preference /
/// max-VRAM LUID), falling back to adapter 0.
///
/// # Safety
/// Calls DXGI factory/adapter enumeration; returns owned COM objects or an error.
unsafe fn resolve_render_adapter() -> Result<IDXGIAdapter1> {
let factory: IDXGIFactory4 = CreateDXGIFactory1().context("CreateDXGIFactory1")?;
if let Some(luid) = crate::win_adapter::resolve_render_adapter_luid() {
if let Ok(a) = factory.EnumAdapterByLuid::<IDXGIAdapter1>(luid) {
return Ok(a);
}
}
factory.EnumAdapters1(0).context("EnumAdapters1(0)")
}
/// Create an NV12 `Texture2D` with the given usage/CPU-access/bind flags.
///
/// # Safety
/// `device` must be a live D3D11 device; the returned texture is owned by the caller.
unsafe fn create_nv12(
device: &ID3D11Device,
width: u32,
height: u32,
usage: D3D11_USAGE,
cpu_access: u32,
bind: u32,
) -> Result<ID3D11Texture2D> {
let desc = D3D11_TEXTURE2D_DESC {
Width: width,
Height: height,
MipLevels: 1,
ArraySize: 1,
Format: DXGI_FORMAT_NV12,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: usage,
BindFlags: bind,
CPUAccessFlags: cpu_access,
..Default::default()
};
let mut tex: Option<ID3D11Texture2D> = None;
device
.CreateTexture2D(&desc, None, Some(&mut tex))
.context("CreateTexture2D(NV12)")?;
tex.context("CreateTexture2D returned a null NV12 texture")
}
+28 -5
View File
@@ -19,6 +19,13 @@ pub struct EncodedFrame {
pub pts_ns: u64,
/// True for IDR/keyframes (sets the SOF/keyframe wire flags).
pub keyframe: bool,
/// True when this AU is a **reference-frame-invalidation recovery frame** — a clean P-frame the
/// encoder coded against a known-good reference in response to
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) (AMD LTR force-reference). The pump
/// tags it [`punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its post-loss
/// freeze on it without an IDR. Only the native-AMF LTR path sets it; every other backend leaves
/// it `false` (their RFI, when present, re-references transparently with no distinct clean-point AU).
pub recovery_anchor: bool,
}
/// Codec selection negotiated with the client.
@@ -208,12 +215,28 @@ pub struct EncoderCaps {
/// the encoder's real chroma disagrees with what was negotiated (the in-band SPS is authoritative
/// for the decoder either way).
pub chroma_444: bool,
/// The encoder runs a periodic **intra-refresh wave** (a moving band of intra blocks +
/// recovery-point SEI, no periodic IDR): FEC-unrecoverable loss self-heals within one wave, so
/// the session glue rate-limits client keyframe requests instead of answering each with a full
/// IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC sets it when
/// `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/software never do.
/// The encoder runs a periodic **intra-refresh wave** a moving band of intra blocks that
/// re-codes the whole picture over ~0.5 s, no periodic IDR. FEC-unrecoverable loss self-heals as
/// the band sweeps, so the session glue rate-limits client keyframe requests instead of answering
/// each with a full IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC / AMF
/// set it when `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/QSV/software never
/// do. NOTE — the wave carries NO decoder-visible clean-point: FFmpeg never sets `AV_FRAME_FLAG_KEY`
/// at a recovery point (H.264 flags key only when `recovery_frame_cnt == 0`; HEVC only on IRAP),
/// and AMF emits no recovery-point SEI at all. So this cap ALONE does not let the client lift its
/// post-loss freeze without an IDR — that needs [`intra_refresh_recovery`](Self::intra_refresh_recovery).
pub intra_refresh: bool,
/// The intra-refresh wave is a *validated constrained GDR* — verified on real hardware to fully
/// heal a lost picture within one wave period with no residual artifacts. Only then does the host
/// tag each wave-boundary AU with [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// so the client can lift its freeze on the second mark (a proven clean re-anchor) instead of
/// waiting out its backstop and forcing a full IDR. Default `false` on every backend until on-glass
/// validation flips it — an un-validated encoder keeps the IDR recovery path, so this is inert and
/// cannot regress. Meaningless unless [`intra_refresh`](Self::intra_refresh) is also set.
pub intra_refresh_recovery: bool,
/// Length of the intra-refresh wave in frames — the boundary period the host marks on (it sets
/// `USER_FLAG_RECOVERY_POINT` on every Nth emitted AU, re-phased at each IDR). 0 when intra-refresh
/// is off. Only consulted when [`intra_refresh_recovery`](Self::intra_refresh_recovery) is set.
pub intra_refresh_period: u32,
}
/// A hardware encoder. One per session; runs on the encode thread.
@@ -177,6 +177,10 @@ pub struct NvencEncoder {
/// Opened in intra-refresh mode (surfaced via [`caps`](Encoder::caps) so the session glue
/// rate-limits forced IDRs — the wave heals loss without them).
intra_refresh: bool,
/// Resolved wave length in frames when [`intra_refresh`](Self::intra_refresh), else 0. Cached at
/// open so the pump's per-AU `caps()` doesn't re-read `PUNKTFUNK_IR_PERIOD_FRAMES`; the pump marks
/// every Nth AU with `USER_FLAG_RECOVERY_POINT` for the client's clean re-anchor.
intra_refresh_period: u32,
}
// `CudaHw` holds raw `AVBufferRef`s and `sws_444` a raw `SwsContext`; the encoder lives on a single
@@ -525,6 +529,11 @@ impl NvencEncoder {
frame_idx: 0,
force_kf: false,
intra_refresh,
intra_refresh_period: if intra_refresh {
intra_refresh_period(fps).max(1) as u32
} else {
0
},
})
}
}
@@ -536,6 +545,12 @@ impl Encoder for NvencEncoder {
// convert. RFI/HDR-SEI stay unsupported on libavcodec NVENC (the trait defaults).
chroma_444: self.want_444,
intra_refresh: self.intra_refresh,
// NVENC intra-refresh is purpose-built GDR loss recovery (moving band + recovery-point
// SEI): the wave heals a lost picture within one period, so mark the boundary AUs and let
// the client re-anchor on them instead of forcing a full IDR. Tied to `intra_refresh`
// (already the `PUNKTFUNK_INTRA_REFRESH` opt-in), unlike AMF/QSV which stay unvalidated.
intra_refresh_recovery: self.intra_refresh,
intra_refresh_period: self.intra_refresh_period,
..super::EncoderCaps::default()
}
}
@@ -578,6 +593,7 @@ impl Encoder for NvencEncoder {
data,
pts_ns,
keyframe: pkt.is_key(),
recovery_anchor: false,
}))
}
// No packet ready yet (need another input frame).
@@ -294,6 +294,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<Option<En
data,
pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(),
recovery_anchor: false,
}))
}
Err(ffmpeg::Error::Other { errno })
+1
View File
@@ -211,6 +211,7 @@ impl Encoder for OpenH264Encoder {
data,
pts_ns,
keyframe,
recovery_anchor: false,
});
}
self.frame_idx += 1;
+323 -19
View File
@@ -644,6 +644,26 @@ struct CodecProps {
/// HEVC 64-px CTBs. `None` on AV1 (v1.4.36 exposes only a mode enum, no slot-size control —
/// loss recovery stays IDR there).
intra_refresh: Option<(PCWSTR, u32)>,
/// LTR-RFI recovery property names (design: the AMD twin of NVENC intra-refresh recovery).
/// `None` on AV1 — its reference management uses a frame-marking OBU mechanism this path does
/// not drive, so LTR recovery is AVC/HEVC-only.
ltr: Option<LtrProps>,
}
/// The four AMF LTR (long-term-reference) property names, codec-prefixed (AVC bare, HEVC `Hevc*`).
/// Two are static (`max_*`, set once at open); two are per-frame (`mark`/`force`, set on the input
/// surface each `submit`). Together they let a loss re-reference a known-good older frame — a clean
/// P-frame instead of a 2040× IDR spike.
struct LtrProps {
/// `MaxOfLTRFrames` — number of user LTR slots (we request [`NUM_LTR_SLOTS`]).
max_ltr_frames: PCWSTR,
/// `MaxNumRefFrames` — reference-picture budget; must exceed 1 for LTR to engage.
max_num_ref_frames: PCWSTR,
/// `MarkCurrentWithLTRIndex` (per-frame) — tag the current frame as long-term reference slot N.
mark_ltr_index: PCWSTR,
/// `ForceLTRReferenceBitfield` (per-frame) — force the current frame to reference only the LTR
/// slots in the bitfield (`1<<N`), breaking the corrupted short-term chain after a loss.
force_ltr_bitfield: PCWSTR,
}
/// The two payload shapes `lowlatency` takes across codecs.
@@ -689,6 +709,12 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("OutColorPrimaries"),
hdr_metadata: None,
intra_refresh: Some((w!("IntraRefreshMBsNumberPerSlot"), 16)),
ltr: Some(LtrProps {
max_ltr_frames: w!("MaxOfLTRFrames"),
max_num_ref_frames: w!("MaxNumRefFrames"),
mark_ltr_index: w!("MarkCurrentWithLTRIndex"),
force_ltr_bitfield: w!("ForceLTRReferenceBitfield"),
}),
},
Codec::H265 => CodecProps {
component: w!("AMFVideoEncoderHW_HEVC"),
@@ -716,6 +742,12 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("HevcOutColorPrimaries"),
hdr_metadata: Some(w!("HevcInHDRMetadata")),
intra_refresh: Some((w!("HevcIntraRefreshCTBsNumberPerSlot"), 64)),
ltr: Some(LtrProps {
max_ltr_frames: w!("HevcMaxOfLTRFrames"),
max_num_ref_frames: w!("HevcMaxNumRefFrames"),
mark_ltr_index: w!("HevcMarkCurrentWithLTRIndex"),
force_ltr_bitfield: w!("HevcForceLTRReferenceBitfield"),
}),
},
Codec::Av1 => CodecProps {
component: w!("AMFVideoEncoderHW_AV1"),
@@ -743,6 +775,7 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("Av1OutputColorPrimaries"),
hdr_metadata: Some(w!("Av1InHDRMetadata")),
intra_refresh: None,
ltr: None,
},
}
}
@@ -797,6 +830,45 @@ fn intra_refresh_period(fps: u32) -> u32 {
.unwrap_or_else(|| (fps.max(16) / 2).max(2))
}
/// Number of user-controlled LTR slots. AMD exposes up to 2; two rotating slots hold a sliding pair
/// of recent long-term references, so a loss can re-reference the newest one *before* the loss point.
const NUM_LTR_SLOTS: usize = 2;
/// AMD's real clean loss-recovery path (the NVENC-RFI twin): the encoder marks frames as long-term
/// references, and on loss forces a later frame to re-reference a known-good one — a clean P-frame,
/// not a 20-40× IDR spike. On by default when the driver supports it (AMF intra-refresh cannot heal —
/// no constrained-intra-prediction property exists in the API, header-confirmed + PSNR-proven — and
/// LTR is mutually exclusive with it, so LTR wins). `PUNKTFUNK_NO_AMF_LTR=1` forces the old full-IDR
/// recovery for debugging.
fn ltr_disabled() -> bool {
std::env::var("PUNKTFUNK_NO_AMF_LTR")
.map(|v| matches!(v.trim(), "1" | "true" | "yes" | "on"))
.unwrap_or(false)
}
/// Cadence (frames) between LTR marks — a fresh long-term reference roughly every half second by
/// default (`PUNKTFUNK_LTR_INTERVAL_FRAMES` overrides). With [`NUM_LTR_SLOTS`] slots this keeps ~one
/// second of recent references, so a loss up to ~1 s old still has a known-good frame to force; a
/// smaller interval means the forced reference is more recent (a smaller recovery-frame residual).
fn ltr_mark_interval(fps: u32) -> i64 {
std::env::var("PUNKTFUNK_LTR_INTERVAL_FRAMES")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.filter(|v| *v >= 1)
.unwrap_or_else(|| (fps.max(2) / 2).max(1) as i64)
}
/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N` the encoder
/// self-triggers its real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path, so a
/// headless spike run can exercise LTR recovery end-to-end (mark → force → recovery-anchor tag)
/// without a live client sending an [`RfiRequest`](punktfunk_core::quic::RfiRequest). `None` normally.
fn ltr_test_force_at() -> Option<i64> {
std::env::var("PUNKTFUNK_LTR_FORCE_AT")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.filter(|v| *v > 0)
}
// ---------------------------------------------------------------------------------------------
// Owned-pointer guards (release exactly once; Terminate before Release for context/component,
// mirroring amfenc.c's teardown order).
@@ -930,11 +1002,12 @@ struct Inner {
dctx: ID3D11DeviceContext,
ring: Vec<ID3D11Texture2D>,
next: usize,
/// (pts_ns, forced-IDR) per submitted-but-unretrieved frame, FIFO — the AMF encoder emits
/// AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`. Its length is
/// the count of input surfaces AMF still holds, so `submit` bounds it below [`RING`] to keep
/// the input ring from being overwritten under it.
pending: VecDeque<(u64, bool)>,
/// (pts_ns, forced-IDR, recovery-anchor) per submitted-but-unretrieved frame, FIFO — the AMF
/// encoder emits AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`.
/// The third field tags the LTR-RFI re-anchor frame so the AU carries `recovery_anchor` for the
/// client's freeze-lift. Its length is the count of input surfaces AMF still holds, so `submit`
/// bounds it below [`RING`] to keep the input ring from being overwritten under it.
pending: VecDeque<(u64, bool, bool)>,
/// AUs already pulled by `submit`'s backpressure drain, waiting to be handed out by `poll`
/// (FIFO, strictly older than anything still in `pending`). Empty in the steady state — only
/// fills when the encoder falls behind and `submit` drains to free an input slot.
@@ -988,6 +1061,26 @@ pub struct AmfEncoder {
/// gates [`EncoderCaps::intra_refresh`] so keyframe-request rate-limiting only happens when
/// the wave really runs.
ir_active: bool,
// --- Long-Term-Reference reference-frame-invalidation recovery (the AMD RFI path) ---
/// The driver accepted the LTR properties at open — gates [`EncoderCaps::supports_rfi`] and all
/// the per-frame LTR marking/forcing below. When true, intra-refresh is NOT set (mutually
/// exclusive) and loss recovery re-references a known-good LTR instead of forcing a full IDR.
ltr_active: bool,
/// The `frame_idx` currently stored in each of the two LTR slots (`None` = never marked). On loss
/// the newest slot with an index *before* the loss is the known-good reference to force.
ltr_slots: [Option<i64>; NUM_LTR_SLOTS],
/// The slot the next LTR mark writes (round-robins `0,1,0,1,…` so the two slots hold a sliding
/// pair of recent references).
next_ltr_slot: usize,
/// Cadence (frames) between LTR marks — a fresh long-term reference roughly this often.
ltr_mark_interval: i64,
/// Set by [`invalidate_ref_frames`](Encoder::invalidate_ref_frames): the LTR slot the *next*
/// submitted frame must force-reference (`ForceLTRReferenceBitfield`). Consumed on that submit.
pending_force: Option<usize>,
/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N`, self-trigger the
/// real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path so a headless spike run can
/// exercise LTR recovery end-to-end without a live client. `None` in normal operation.
ltr_test_force_at: Option<i64>,
/// Consecutive [`reset`](Self::reset)s that have NOT been followed by a produced AU (cleared in
/// `poll` on any output). An in-place `Terminate`+re-`Init` heals a transient component stall,
/// but it re-inits the SAME context — so if the fault is the context / VCN session itself (the
@@ -1084,17 +1177,33 @@ impl AmfEncoder {
force_kf: false,
hdr_meta: None,
ir_active: false,
ltr_active: false,
ltr_slots: [None; NUM_LTR_SLOTS],
next_ltr_slot: 0,
ltr_mark_interval: ltr_mark_interval(fps),
pending_force: None,
ltr_test_force_at: ltr_test_force_at(),
resets_without_output: 0,
})
}
/// Whether this encoder should *attempt* the LTR-RFI recovery path (design: the AMD twin of
/// NVENC intra-refresh recovery). Gated to AVC/HEVC — AMF exposes user LTR only for those two
/// codecs — and defeatable via `PUNKTFUNK_NO_AMF_LTR`. Whether the driver actually *accepts* the
/// properties is a separate question answered by [`apply_static_props`], which sets `ltr_active`.
fn ltr_wanted(&self) -> bool {
!ltr_disabled() && matches!(self.codec, Codec::H264 | Codec::H265)
}
/// Apply the static encoder configuration (design §3.4 — the native mirror of the ffmpeg
/// opts block in `open_win_encoder`). Called before `Init`, and again on a `reset()`
/// re-`Init` (Terminate does not guarantee property retention across every driver).
/// Returns whether the intra-refresh wave was requested AND accepted by this driver — the
/// caller stores it so [`Encoder::caps`] only rate-limits keyframe requests when the wave
/// really runs.
unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<bool> {
/// Returns `(ir_active, ltr_active)`: whether the intra-refresh wave / the LTR-RFI slots were
/// requested AND accepted by this driver. The two are mutually exclusive (LTR wins when both are
/// wanted). The caller stores both — `ir_active` so [`Encoder::caps`] only rate-limits keyframe
/// requests when a wave runs, `ltr_active` so [`Encoder::caps`] advertises `supports_rfi` and the
/// per-frame mark/force logic in `submit` only fires when the slots exist.
unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<(bool, bool)> {
let p = &self.props;
// Usage first: it "fully configures parameter set" — everything after is an override.
set_prop(
@@ -1145,7 +1254,39 @@ impl AmfEncoder {
// whole picture refreshes every `period` frames — per-slot units = ceil(total blocks /
// period). Optional by VCN generation; the return value gates `caps().intra_refresh`.
let mut ir_active = false;
if let Some((name, block)) = p.intra_refresh {
let mut ltr_active = false;
if let Some(ltr) = p.ltr.as_ref().filter(|_| self.ltr_wanted()) {
// LTR-RFI recovery (design: the AMD twin of NVENC intra-refresh recovery). Request
// NUM_LTR_SLOTS user-controlled long-term references. LTR needs >1 reference frames and
// is MUTUALLY EXCLUSIVE with intra-refresh (AMF disables one if both are set), so the
// intra-refresh block below is skipped whenever LTR engages.
let ref_ok = set_prop(
comp,
ltr.max_num_ref_frames,
AmfVariant::from_i64(NUM_LTR_SLOTS as i64),
false,
)?;
let ltr_ok = set_prop(
comp,
ltr.max_ltr_frames,
AmfVariant::from_i64(NUM_LTR_SLOTS as i64),
false,
)?;
ltr_active = ref_ok && ltr_ok;
if ltr_active {
tracing::info!(
slots = NUM_LTR_SLOTS,
mark_interval = self.ltr_mark_interval,
"AMF LTR-RFI recovery enabled (loss recovery re-references a known-good LTR, not a full IDR)"
);
} else {
tracing::warn!(
ref_ok,
ltr_ok,
"this VCN/driver rejected an LTR property — loss recovery stays full-IDR"
);
}
} else if let Some((name, block)) = p.intra_refresh {
if intra_refresh_requested() {
let period = intra_refresh_period(self.fps);
let blocks = self.width.div_ceil(block) * self.height.div_ceil(block);
@@ -1273,7 +1414,7 @@ impl AmfEncoder {
AmfVariant::from_i64(primaries),
self.ten_bit,
)?;
Ok(ir_active)
Ok((ir_active, ltr_active))
}
/// Build (or rebuild, on a capture-device change) the AMF context + encoder component on the
@@ -1323,7 +1464,7 @@ impl AmfEncoder {
bail!("AMF CreateComponent returned null");
}
let comp = Component(comp);
let ir_active = self.apply_static_props(comp.0)?;
let (ir_active, ltr_active) = self.apply_static_props(comp.0)?;
let fmt = if self.ten_bit {
sys::AMF_SURFACE_P010
} else {
@@ -1334,6 +1475,14 @@ impl AmfEncoder {
"AMF encoder Init",
)?;
self.ir_active = ir_active;
// A rebuilt component starts with fresh (empty) LTR slots — a new context has no
// reference history, so any prior marks are void and the first frame re-IDRs anyway.
self.ltr_active = ltr_active;
if ltr_active {
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
}
// Owned input ring on the capturer's device (design §3.2): RENDER_TARGET |
// SHADER_RESOURCE, the same bind flags the validated ffmpeg zero-copy pool uses.
@@ -1594,7 +1743,7 @@ enum DrainOutcome {
/// single encode thread with no other AMF call to this component in flight.
unsafe fn drain_one_output(
comp: *mut sys::AmfComponent,
pending: &mut VecDeque<(u64, bool)>,
pending: &mut VecDeque<(u64, bool, bool)>,
output_data_type: PCWSTR,
output_key_max: i64,
) -> Result<DrainOutcome> {
@@ -1641,11 +1790,12 @@ unsafe fn drain_one_output(
bail!("AMF output buffer is empty");
}
let au = std::slice::from_raw_parts(native as *const u8, size).to_vec();
let (pts_ns, forced) = pending.pop_front().unwrap_or((0, false));
let (pts_ns, forced, recovery_anchor) = pending.pop_front().unwrap_or((0, false, false));
Ok(DrainOutcome::Frame(EncodedFrame {
data: au,
pts_ns,
keyframe: key_prop || forced,
recovery_anchor,
}))
}
@@ -1689,9 +1839,57 @@ impl Encoder for AmfEncoder {
expected
);
self.ensure_inner(&frame.device)?;
let cur_idx = self.frame_idx;
let forced = std::mem::take(&mut self.force_kf) || self.frame_idx == 0;
let pts_100ns = self.frame_idx * 10_000_000 / self.fps.max(1) as i64;
self.frame_idx += 1;
// --- LTR-RFI per-frame decisions (design: the AMD twin of NVENC intra-refresh recovery) ---
// Decided here, before borrowing `inner`, because the test hook re-enters `&mut self`
// (`invalidate_ref_frames`) and the mark cadence mutates the slot bookkeeping. The two
// per-frame property names are copied out (PCWSTR is Copy) so the unsafe surface block can
// set them without re-borrowing `self.props` under the live `inner` borrow.
let ltr_names = self
.props
.ltr
.as_ref()
.map(|l| (l.mark_ltr_index, l.force_ltr_bitfield));
let mut mark_slot: Option<usize> = None;
let mut force_slot: Option<usize> = None;
let mut recovery_anchor = false;
if self.ltr_active {
if forced {
// An IDR resets the decoder's reference buffers — every prior LTR mark is void.
// Re-anchor from scratch: drop the stale slots (the mark cadence below tags the IDR
// as the first fresh long-term reference) and cancel any force queued against them.
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
} else if self.ltr_test_force_at == Some(cur_idx) {
// Spike-only validation hook: self-trigger the real invalidate path so a headless
// run exercises mark → force → recovery-anchor without a live client's RfiRequest.
let triggered = self.invalidate_ref_frames(cur_idx, cur_idx);
tracing::info!(
frame = cur_idx,
triggered,
"AMF LTR test hook fired invalidate_ref_frames"
);
}
// Apply a queued force (from invalidate_ref_frames / the test hook) to THIS frame: it
// becomes the clean re-anchor P-frame the client lifts its post-loss freeze on.
if let Some(slot) = self.pending_force.take() {
force_slot = Some(slot);
recovery_anchor = true;
}
// Mark cadence: refresh a long-term reference on every IDR and every `ltr_mark_interval`
// frames — but never on the recovery frame itself (marking rotates `next_ltr_slot` and
// could overwrite the very slot being forced; the next cadence mark re-establishes it).
if force_slot.is_none() && (forced || cur_idx % self.ltr_mark_interval == 0) {
let slot = self.next_ltr_slot;
self.ltr_slots[slot] = Some(cur_idx);
self.next_ltr_slot = (self.next_ltr_slot + 1) % NUM_LTR_SLOTS;
mark_slot = Some(slot);
}
}
let inner = self.inner.as_mut().expect("ensure_inner succeeded");
// Push the HDR mastering metadata when it changed (or a rebuilt component lost it) — a
// dynamic property, so mid-stream regrades take effect on the next IDR. Best-effort: a
@@ -1831,6 +2029,47 @@ impl Encoder for AmfEncoder {
Codec::Av1 => {}
}
}
// LTR-RFI per-frame properties (design: the AMD twin of NVENC intra-refresh recovery).
// `mark_slot`/`force_slot` were decided above. Marking tags the current frame as a
// long-term reference; forcing makes it re-reference a known-good LTR — a clean P-frame
// that breaks the corrupted short-term chain after a loss, no 20-40× IDR. Best-effort:
// a rejecting driver just leaves the client on its keyframe-request fallback.
if let Some((mark_name, force_name)) = ltr_names {
if let Some(slot) = mark_slot {
let r = ((*(*surf.0).vtbl).set_property)(
surf.0,
mark_name.0,
AmfVariant::from_i64(slot as i64),
);
if r != sys::AMF_OK {
tracing::warn!(
slot,
result = %format!("{} ({r})", result_name(r)),
"AMF LTR mark rejected"
);
}
}
if let Some(slot) = force_slot {
let r = ((*(*surf.0).vtbl).set_property)(
surf.0,
force_name.0,
AmfVariant::from_i64(1_i64 << slot),
);
if r == sys::AMF_OK {
tracing::info!(
slot,
frame = cur_idx,
"AMF LTR-RFI: re-referencing known-good LTR (clean recovery, no IDR)"
);
} else {
tracing::warn!(
slot,
result = %format!("{} ({r})", result_name(r)),
"AMF LTR force-reference rejected — client stays frozen until its IDR fallback"
);
}
}
}
let mut r = ((*(*inner.comp.0).vtbl).submit_input)(inner.comp.0, surf.0);
// Backstop back-pressure: the in-flight bound above already keeps a slot free, but if
// AMF's own input queue is momentarily full, AMF_INPUT_FULL is "busy, drain me and
@@ -1873,7 +2112,7 @@ impl Encoder for AmfEncoder {
}
}
}
inner.pending.push_back((captured.pts_ns, forced));
inner.pending.push_back((captured.pts_ns, forced, recovery_anchor));
Ok(())
}
@@ -1887,11 +2126,63 @@ impl Encoder for AmfEncoder {
self.hdr_meta = meta;
}
/// LTR-RFI recovery (the AMD twin of the Windows NVENC `nvEncInvalidateRefFrames` path): a loss
/// of client frames `[first, last]` is answered by forcing the *next* submitted frame to
/// re-reference the newest long-term reference marked *before* the loss — a clean P-frame the
/// client can decode against a picture it still holds, instead of a 20-40× IDR spike.
///
/// Returns `true` when a usable pre-loss LTR exists (so the caller must NOT also force an IDR);
/// `false` when the loss predates every live LTR — then the only correct recovery is a keyframe,
/// and the caller falls back to [`request_keyframe`](Self::request_keyframe). Runs on the encode
/// thread (like submit/poll); the force is applied on the next `submit`.
fn invalidate_ref_frames(&mut self, first: i64, last: i64) -> bool {
// No live LTR session (driver declined the slots, or AV1 which has no user-LTR path) or a
// nonsense range → caller forces a full IDR.
if !self.ltr_active || first < 0 || first > last {
return false;
}
// Pick the newest LTR strictly OLDER than the loss: the most recent known-good reference the
// client still holds, so re-referencing it costs the least (smallest recovery-frame residual).
// Frame numbers are 1:1 with the client's (both count submissions in order — see the NVENC
// path), so `ltr_slots` (which store `frame_idx`) compare directly against `first`.
let mut best: Option<(usize, i64)> = None;
for (slot, marked) in self.ltr_slots.iter().enumerate() {
if let Some(idx) = *marked {
if idx < first && best.is_none_or(|(_, b)| idx > b) {
best = Some((slot, idx));
}
}
}
match best {
Some((slot, ltr_frame)) => {
// Queue the force for the next submit; that frame ships tagged `recovery_anchor`.
self.pending_force = Some(slot);
tracing::info!(
first,
last,
slot,
ltr_frame,
"AMF LTR-RFI: forcing the next frame to re-reference a known-good LTR (no IDR)"
);
true
}
None => {
tracing::info!(
first,
last,
"AMF LTR-RFI: no live LTR older than the loss — falling back to IDR recovery"
);
false
}
}
}
fn caps(&self) -> EncoderCaps {
EncoderCaps {
// AMF has no NVENC-style reference invalidation the intra-refresh wave is the
// loss-recovery substitute; without it every unrecoverable loss costs an IDR.
supports_rfi: false,
// LTR-RFI: AMD's reference invalidation is the user long-term-reference path (mark a
// frame, force a later one to re-reference it). True only when the live driver accepted
// the LTR slots at open — otherwise loss recovery falls back to a full IDR.
supports_rfi: self.ltr_active,
// In-band mastering/CLL via `*InHDRMetadata` (HEVC SEI / AV1 metadata OBU); AVC has
// no such property (and no HDR sessions negotiate H.264).
supports_hdr_metadata: self.ten_bit && self.props.hdr_metadata.is_some(),
@@ -1901,6 +2192,11 @@ impl Encoder for AmfEncoder {
// accepted the property (queried per loss event, so the post-first-frame value is
// what the session glue's IDR rate-limiting sees).
intra_refresh: self.ir_active,
// Not yet: the AMD VCN wave heals in principle, but its constrained-GDR
// heal-within-a-period is unvalidated on-glass and AMF emits no recovery-point SEI, so
// the host keeps the IDR recovery path. Flip both once verified on real hardware.
intra_refresh_recovery: false,
intra_refresh_period: 0,
}
}
@@ -1992,6 +2288,7 @@ impl Encoder for AmfEncoder {
self.inner = None;
self.bound_device = 0;
self.ir_active = false;
self.ltr_active = false;
return true;
}
let inner = self
@@ -2016,8 +2313,14 @@ impl Encoder for AmfEncoder {
sys::AMF_SURFACE_NV12
};
match self.apply_static_props(comp) {
Ok(ir) => {
Ok((ir, ltr)) => {
self.ir_active = ir;
// Re-Init voids the reference history: the rebuilt stream restarts at IDR with
// empty LTR slots, so any prior marks are stale and must be dropped.
self.ltr_active = ltr;
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
((*(*comp).vtbl).init)(comp, fmt, self.width as i32, self.height as i32)
== sys::AMF_OK
}
@@ -2030,6 +2333,7 @@ impl Encoder for AmfEncoder {
);
} else {
self.ir_active = false;
self.ltr_active = false;
// Full teardown; the next submit reopens context + component on the current device.
tracing::warn!("AMF in-place re-Init failed — full context teardown, reopening lazily");
self.inner = None;
@@ -339,6 +339,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<PollOutco
data,
pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(),
recovery_anchor: false,
}))
}
Err(ffmpeg::Error::Other { errno })
@@ -1157,6 +1157,7 @@ impl NvencD3d11Encoder {
data,
pts_ns,
keyframe,
recovery_anchor: false,
});
Ok(())
}
@@ -1424,6 +1425,8 @@ impl Encoder for NvencD3d11Encoder {
// The direct-NVENC path recovers via real RFI (or a forced IDR), not the Linux
// libavcodec intra-refresh mode.
intra_refresh: false,
intra_refresh_recovery: false,
intra_refresh_period: 0,
}
}
@@ -1542,6 +1545,7 @@ impl Encoder for NvencD3d11Encoder {
data,
pts_ns,
keyframe,
recovery_anchor: false,
}))
}
}
+5 -1
View File
@@ -696,10 +696,14 @@ fn parse_spike(args: &[String]) -> Result<Options> {
"--source" => {
source = match next()?.as_str() {
"synthetic" => Source::Synthetic,
"synthetic-nv12" => Source::SyntheticNv12,
"portal" => Source::Portal,
"kwin-virtual" => Source::KwinVirtual,
other => {
bail!("unknown --source '{other}' (synthetic|portal|kwin-virtual)")
bail!(
"unknown --source '{other}' \
(synthetic|synthetic-nv12|portal|kwin-virtual)"
)
}
}
}
+4
View File
@@ -26,12 +26,16 @@ pub fn pump_once(
data,
pts_ns,
keyframe,
recovery_anchor,
}) = encoder.poll()?
{
let mut flags = FLAG_PIC as u32;
if keyframe {
flags |= FLAG_SOF as u32;
}
if recovery_anchor {
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR;
}
// core does FEC + packetize + pace + send.
session.submit_frame(&data, pts_ns, flags)?;
}
+122 -2
View File
@@ -34,7 +34,7 @@ use punktfunk_core::packet::{FLAG_PIC, FLAG_PROBE, FLAG_SOF};
use punktfunk_core::quic::{
endpoint, io, BitrateChanged, ClockEcho, ClockProbe, ColorInfo, Hello, LossReport,
PairChallenge, PairProof, PairRequest, PairResult, ProbeRequest, ProbeResult, Reconfigure,
Reconfigured, RequestKeyframe, SetBitrate, Start, Welcome,
Reconfigured, RequestKeyframe, RfiRequest, SetBitrate, Start, Welcome,
};
use punktfunk_core::transport::UdpTransport;
use punktfunk_core::Session;
@@ -1124,6 +1124,10 @@ async fn serve_session(
// (inbound requests, outbound probe results) are multiplexed with `select!`.
let (reconfig_tx, reconfig_rx) = std::sync::mpsc::channel::<punktfunk_core::Mode>();
let (keyframe_tx, keyframe_rx) = std::sync::mpsc::channel::<()>();
// Client LTR-RFI recovery: the control task forwards each `RfiRequest`'s lost-frame range here;
// the encode loop prefers `Encoder::invalidate_ref_frames` (a clean re-anchor P-frame) over a
// full IDR when the encoder supports it (native-AMF LTR / Windows NVENC).
let (rfi_tx, rfi_rx) = std::sync::mpsc::channel::<(u32, u32)>();
let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::<u32>();
let (probe_tx, probe_rx) = std::sync::mpsc::channel::<ProbeRequest>();
let (probe_result_tx, mut probe_result_rx) =
@@ -1199,6 +1203,19 @@ async fn serve_session(
if keyframe_tx.send(()).is_err() {
break; // data plane gone
}
} else if let Ok(req) = RfiRequest::decode(&msg) {
// Client LTR-RFI recovery: it lost the frame range `[first, last]` and asks
// the encoder to re-reference a known-good older frame instead of paying for
// a full IDR. The encode loop attempts `invalidate_ref_frames`, falling back
// to a coalesced keyframe when the encoder can't (range too old / no RFI).
tracing::debug!(
first = req.first_frame,
last = req.last_frame,
"client requested reference-frame invalidation (loss recovery)"
);
if rfi_tx.send((req.first_frame, req.last_frame)).is_err() {
break; // data plane gone
}
} else if let Ok(rep) = LossReport::decode(&msg) {
// Adaptive FEC: size recovery to the loss the client is seeing. The data-plane
// send loop reads `fec_target_ctl` and applies it per frame. Ignored when FEC
@@ -1590,6 +1607,7 @@ async fn serve_session(
quit: quit_stream,
reconfig: reconfig_rx,
keyframe: keyframe_rx,
rfi: rfi_rx,
bitrate_rx,
compositor,
bitrate_kbps,
@@ -2396,6 +2414,29 @@ fn audio_thread(
tracing::warn!("punktfunk/1 audio requires Linux or Windows — session continues without it");
}
/// Advance the intra-refresh wave position and decide whether this emitted AU is a wave boundary
/// that should carry [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT).
///
/// `ir_wave_pos` counts frames since the last IDR/wave start; a real IDR re-phases it to 0 (an IDR
/// restarts the encoder's wave AND is itself a clean anchor, so it is never additionally marked).
/// Every `period`-th non-IDR AU is a boundary — the client lifts its post-loss freeze on the SECOND
/// such mark. Pure so the marking cadence is unit-tested without a GPU (see the pump's use in the
/// encode-poll loop).
fn mark_recovery_boundary(ir_wave_pos: &mut u32, is_keyframe: bool, period: u32) -> bool {
if is_keyframe {
*ir_wave_pos = 0;
false
} else {
*ir_wave_pos += 1;
if *ir_wave_pos >= period {
*ir_wave_pos = 0;
true
} else {
false
}
}
}
fn synthetic_stream(
session: &mut Session,
frames: u32,
@@ -3396,6 +3437,9 @@ struct SessionContext {
reconfig: std::sync::mpsc::Receiver<punktfunk_core::Mode>,
/// Client decode-recovery keyframe requests.
keyframe: std::sync::mpsc::Receiver<()>,
/// Client LTR-RFI recovery requests — the lost-frame range `(first, last)`. The encode loop
/// prefers `Encoder::invalidate_ref_frames` over a full IDR when the encoder supports it.
rfi: std::sync::mpsc::Receiver<(u32, u32)>,
/// Accepted mid-stream bitrate changes (adaptive bitrate, already clamped) — the encoder
/// alone is rebuilt in place at the new rate; capture + virtual output are untouched.
bitrate_rx: std::sync::mpsc::Receiver<u32>,
@@ -3467,6 +3511,7 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
quit,
reconfig,
keyframe,
rfi,
bitrate_rx,
compositor,
mut bitrate_kbps,
@@ -3684,6 +3729,11 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
// Self-diagnosis for the periodic-stutter class: warns when the served recovery IDRs settle
// into a stable multi-second rhythm (see [`crate::metronome::Metronome`]).
let mut recovery_cadence = crate::metronome::Metronome::new();
// Position within the current intra-refresh wave (frames since the last IDR/wave start). Only
// meaningful on a `caps().intra_refresh_recovery` encoder; the pump tags every wave-boundary AU
// with `USER_FLAG_RECOVERY_POINT` so the client can lift its post-loss freeze on a clean
// re-anchor without a full IDR. Re-phased to 0 at each emitted IDR (which restarts the wave).
let mut ir_wave_pos: u32 = 0;
// Per-stage latency breakdown (PUNKTFUNK_PERF): per-call µs for the GPU-bound stages so we see
// exactly where the capture→encoded latency goes — cap=try_latest (ring read + colour convert),
// submit=encode_picture launch, wait=lock_bitstream (the scheduling wait + ASIC encode, the one
@@ -3900,6 +3950,33 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
while keyframe.try_recv().is_ok() {
want_kf = true;
}
// Client LTR-RFI recovery: prefer re-referencing a known-good older frame (a clean recovery
// P-frame — no 20-40× IDR spike) over a full keyframe when the encoder supports it (native
// AMF LTR / Windows NVENC). Drain the backlog (the client re-requests until the recovery
// frame lands) coalesced to the widest lost range. Attempt the invalidate only when a full
// IDR isn't already queued — an explicit keyframe request means a fully wedged decoder that
// needs the IDR, which supersedes an RFI recovery. A failure (range older than the encoder's
// live references, or no RFI backend) falls through to the coalesced keyframe path below.
let mut rfi_range: Option<(u32, u32)> = None;
while let Ok((first, last)) = rfi.try_recv() {
rfi_range = Some(match rfi_range {
Some((pf, pl)) => (pf.min(first), pl.max(last)),
None => (first, last),
});
}
if !want_kf {
if let Some((first, last)) = rfi_range {
if enc.caps().supports_rfi && enc.invalidate_ref_frames(first as i64, last as i64) {
// The RFI recovered the loss with a clean re-anchor P-frame (no IDR). Anchor the
// keyframe cooldown so the client's echo of the SAME loss — its frames_dropped-
// driven keyframe request, arriving ~one loss-window later — is coalesced away
// instead of emitting a redundant full IDR right after the cheap recovery.
last_forced_idr = Some(std::time::Instant::now());
} else {
want_kf = true; // range too old / no RFI backend → coalesced keyframe below
}
}
}
if want_kf {
// Clients request a keyframe on EVERY FEC-unrecoverable frame (`frames_dropped` polling)
// and keep asking until the IDR actually arrives + decodes — a full round-trip on a link
@@ -4225,11 +4302,28 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
last_au_at = std::time::Instant::now();
encoder_resets = 0;
let (cap_ns, sub_ns, deadline) = inflight.pop_front().expect("inflight non-empty");
let flags = if au.keyframe {
let mut flags = if au.keyframe {
(FLAG_PIC | FLAG_SOF) as u32
} else {
FLAG_PIC as u32
};
// Intra-refresh recovery marking (inert unless the backend validated its constrained GDR
// via `intra_refresh_recovery`): tag every wave-boundary AU with USER_FLAG_RECOVERY_POINT
// so the client lifts its post-loss freeze on the second mark — a proven clean re-anchor —
// instead of forcing a full IDR. See [`mark_recovery_boundary`] for the cadence.
let caps = enc.caps();
if caps.intra_refresh_recovery
&& caps.intra_refresh_period > 0
&& mark_recovery_boundary(&mut ir_wave_pos, au.keyframe, caps.intra_refresh_period)
{
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_POINT;
}
// Reference-frame-invalidation recovery frame (AMD LTR force-reference): a clean P-frame
// off a known-good reference. Tag it so the client lifts its post-loss freeze on this one
// AU without an IDR — the definitive single-frame re-anchor (see USER_FLAG_RECOVERY_ANCHOR).
if au.recovery_anchor {
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR;
}
// Re-send the HDR mastering metadata (0xCE) on each keyframe (a decoder-resync point) and
// whenever it changed, so a client that dropped the best-effort datagram re-converges.
if let Some(m) = last_hdr_meta {
@@ -4654,6 +4748,32 @@ mod tests {
assert!(reconfig_allowed(None, false));
}
#[test]
fn recovery_marks_land_every_period_and_rephase_at_idr() {
let period = 4;
let mut pos = 0u32;
// Frames 1..=3 are mid-wave (no mark), frame 4 is the boundary; then it repeats.
let marks: Vec<bool> = (0..10)
.map(|_| mark_recovery_boundary(&mut pos, false, period))
.collect();
assert_eq!(
marks,
vec![false, false, false, true, false, false, false, true, false, false]
);
// An IDR mid-wave re-phases: the counter restarts, so the next boundary is a full period
// later (an IDR is itself a clean anchor, so it is not additionally marked).
let mut pos = 0u32;
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2
assert!(!mark_recovery_boundary(&mut pos, true, period)); // IDR → pos 0, no mark
// Now a fresh full period is needed, not just the 2 remaining frames.
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 3
assert!(mark_recovery_boundary(&mut pos, false, period)); // pos 4 → mark
}
#[test]
fn pad_snapshot_replaces_state_and_seq_gates() {
use punktfunk_core::input::{gamepad, GamepadSnapshot};
+26
View File
@@ -22,6 +22,11 @@ use std::time::Instant;
pub enum Source {
/// Deterministic moving BGRx test pattern — no capture session required.
Synthetic,
/// Deterministic moving NV12 texture on the GPU (Windows only) — no capture session required.
/// Feeds the native AMF / D3D11 zero-copy encoders, which demand an NV12 GPU texture the CPU
/// `Synthetic` source can't give them. Used to validate GPU-encoder behaviour (e.g. AMF
/// intra-refresh) headlessly.
SyntheticNv12,
/// Live monitor via the xdg ScreenCast portal + PipeWire.
Portal,
/// KWin virtual output created at `width`x`height` (zkde_screencast). Lets us validate
@@ -56,6 +61,27 @@ pub fn run(opts: Options) -> Result<()> {
);
Box::new(SyntheticCapturer::new(opts.width, opts.height, opts.fps))
}
Source::SyntheticNv12 => {
#[cfg(target_os = "windows")]
{
tracing::info!(
width = opts.width,
height = opts.height,
fps = opts.fps,
"spike source: synthetic NV12 GPU texture (moving luma ramp)"
);
Box::new(
capture::synthetic_nv12::SyntheticNv12Capturer::new(
opts.width, opts.height, opts.fps,
)
.context("open synthetic NV12 capturer")?,
)
}
#[cfg(not(target_os = "windows"))]
{
anyhow::bail!("--source synthetic-nv12 is Windows-only (native AMF / D3D11 encoders)");
}
}
Source::Portal => {
tracing::info!("spike source: xdg ScreenCast portal (live monitor)");
capture::open_portal_monitor().context("open portal capturer")?