4 Commits

Author SHA1 Message Date
enricobuehler e55ff1bb28 feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Has started running
Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets
AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike.

Client (pf-client-core): after a reference loss the hardware decoder conceals the
missing-reference deltas (on RADV, a gray plate with new motion painted over) and
returns Ok. The pump now freezes on the last good picture until a clean re-anchor
instead of showing the concealment — lifting on a real IDR, an intra-refresh
recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The
frame_index gap is the early, precise loss signal and drives an RFI request.

Host recovery signals (inert unless the backend supports them):
- USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR).
- USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame.

AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API
has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the
only clean-recovery lever is user LTR: mark frames as long-term references, and on
loss force the next frame to re-reference the newest known-good one — a clean
P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for
AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR
before the loss; a range older than the live slots falls back to a keyframe.

Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi().
Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success
anchors the keyframe cooldown so the client's frames_dropped echo of the same loss
is coalesced away rather than emitting a redundant IDR.

Spike: synthetic NV12 GPU source for headless AMF encoder testing.

Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests
(incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux;
punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90
forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 17:31:17 +02:00
enricobuehler 890c7531d8 Merge branch 'midstream-resize': mid-stream resolution resize
apple / swift (push) Successful in 4m19s
ci / rust (push) Failing after 46s
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 1m2s
decky / build-publish (push) Successful in 20s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
ci / bench (push) Successful in 5m17s
arch / build-publish (push) Successful in 10m44s
docker / deploy-docs (push) Successful in 11s
android / android (push) Successful in 16m43s
windows-host / package (push) Failing after 8m29s
flatpak / build-publish (push) Failing after 8m13s
deb / build-publish (push) Successful in 11m24s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m51s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m1s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m24s
release / apple (push) Successful in 24m35s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m24s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m2s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m25s
apple / screenshots (push) Successful in 19m55s
Lands the mid-stream resolution resize feature (client-driven Reconfigure so
the host's virtual display + encoder follow a resized client window without a
reconnect), all paths default OFF:

- host hardening H1-H5 + session-binary Match window (C1)
- Apple macOS/iPadOS Match-window trigger + settings (C3) and the resize
  overlay (blur + spinner) client UX
- Windows on-glass fixes: corrective-ack actual resolution + pf-vdisplay
  monitor re-arrival for out-of-list mid-stream modes
- Linux backend matrix + the live-reconfigure gate unit tests

Validated on-glass: Windows IDD-push (.173), Linux Mutter + KWin. Android
(C4) deferred; Apple full build pending on a Mac.
2026-07-11 15:59:07 +02:00
enricobuehler e6fbcecdb9 fix(clients/windows): GUI text inputs read the live value, not a stale render snapshot
ci / web (push) Successful in 50s
ci / docs-site (push) Successful in 1m5s
decky / build-publish (push) Successful in 23s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 16s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 14s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 14s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 1m0s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 2m33s
apple / swift (push) Successful in 4m24s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 2m18s
ci / bench (push) Successful in 7m10s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m17s
arch / build-publish (push) Successful in 11m38s
deb / build-publish (push) Successful in 12m6s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m9s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m46s
android / android (push) Successful in 17m7s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m48s
docker / deploy-docs (push) Successful in 22s
ci / rust (push) Successful in 26m33s
apple / screenshots (push) Successful in 19m29s
A component() page re-renders reliably only when its props change: root wraps
every screen in a stable animated border, so once the entrance tween settles the
reconciler skips that unchanged-props subtree and a page's own use_state writes
never force a re-render. Three text fields read their value at click time from
that stranded local state:

- PIN pairing sent an empty PIN, so pairing always failed with "wrong PIN, or not
  armed?" — the reported bug. The CLI --pair path bypasses the reactor and worked.
- "Add host" Connect captured the empty mount-time address and silently did
  nothing (you open the modal precisely when the host isn't being discovered, so
  no discovery tick re-renders the page while you type).
- Rename round-tripped the draft through an always-deferred AsyncSetState into a
  controlled text box, fighting the caret on fast typing and dropping the last
  character when Save was clicked before the write landed.

Fix: hold each field's live value in a use_ref cell written on every keystroke
and read at commit time (uncontrolled input), instead of a render-time snapshot.
Rename is seeded when its target changes and no longer re-renders the whole page
per keystroke. Reviewed the rest of the app (settings, speed test, library,
stream, connect/request-access/waking, forget) — all driven by root-state props
and wired correctly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:58:25 +02:00
enricobuehler 64b9d11ee6 fix(ci/windows): reclaim runner disk before building
ci / web (push) Successful in 1m2s
ci / docs-site (push) Successful in 1m2s
apple / swift (push) Successful in 4m58s
ci / bench (push) Successful in 6m47s
android-screenshots / screenshots (push) Successful in 2m58s
windows-host / package (push) Successful in 14m30s
decky / build-publish (push) Successful in 16s
ci / rust (push) Successful in 18m31s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m6s
android / android (push) Successful in 12m26s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m58s
web-screenshots / screenshots (push) Successful in 2m45s
flatpak / build-publish (push) Failing after 8m2s
linux-client-screenshots / screenshots (push) Successful in 7m33s
docker / deploy-docs (push) Successful in 19s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5m35s
arch / build-publish (push) Successful in 12m58s
deb / build-publish (push) Successful in 14m8s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m35s
release / apple (push) Successful in 28m12s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m43s
apple / screenshots (push) Successful in 18m28s
A full Windows CI pass writes ~50 GB of cargo target output into the shared
C:\t (x64) / C:\t-a64 (arm64) scratch dirs on the intentionally-small (100 GB)
windows-amd64 runner. Left to accumulate across runs, that overflowed the disk
and every build died with "no space on device" (os error 112) — bytemuck_derive,
cc, bindgen, windows, tracing-subscriber, fs4 all failing mid-compile, taking
down pf-vdisplay/host builds.

ensure-windows-toolchain.ps1 already runs first in every Windows job, so reclaim
disk there before provisioning/building: call the runner-baked reclaimer
(unom/infra installs C:\Users\Public\act-runner\clean-runner-disk.ps1 + a
scheduled task) in threshold mode so THIS job starts with headroom regardless of
when that task last ran, and keep incremental caches warm when there's room. A
small inline fallback covers a runner not yet re-baked with the reclaimer. The
whole step is best-effort — a cleanup hiccup never fails the build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 11:35:08 +02:00
24 changed files with 1325 additions and 47 deletions
+49 -10
View File
@@ -189,14 +189,21 @@ fn status_row(online: Option<bool>, badge: &str, kind: Pill) -> Element {
/// The in-tile rename editor (ContentDialog can't hold a text field): name box + save/cancel. /// The in-tile rename editor (ContentDialog can't hold a text field): name box + save/cancel.
/// No tap-to-connect while editing — a click into the box would bubble `Tapped` to the region. /// No tap-to-connect while editing — a click into the box would bubble `Tapped` to the region.
/// `initial` seeds the text box's displayed value and is CONSTANT for the life of the edit — the
/// field is uncontrolled, its live value kept in `live` (read at Save). Driving a *controlled* box
/// from an always-deferred `AsyncSetState` round-trip fights the caret on fast typing and can drop
/// the last char if Save is clicked before the write lands; an uncontrolled box + a ref sidesteps
/// both (and skips a full-page re-render per keystroke). See the seed block in `hosts_page`.
fn rename_editor( fn rename_editor(
draft: &str, initial: &str,
fp: String, fp: String,
live: HookRef<String>,
set_rename: AsyncSetState<Option<(String, String)>>, set_rename: AsyncSetState<Option<(String, String)>>,
) -> Element { ) -> Element {
let commit = { let commit = {
let (fp, draft, sr) = (fp.clone(), draft.to_string(), set_rename.clone()); let (fp, live, sr) = (fp.clone(), live.clone(), set_rename.clone());
move || { move || {
let draft = live.borrow();
let name = draft.trim(); let name = draft.trim();
if !name.is_empty() { if !name.is_empty() {
let mut known = KnownHosts::load(); let mut known = KnownHosts::load();
@@ -209,12 +216,12 @@ fn rename_editor(
} }
}; };
let on_changed = { let on_changed = {
let sr = set_rename.clone(); let live = live.clone();
move |s: String| sr.call(Some((fp.clone(), s))) move |s: String| live.set(s)
}; };
card( card(
vstack(( vstack((
text_box(draft) text_box(initial)
.placeholder_text("Host name") .placeholder_text("Host name")
.on_text_changed(on_changed), .on_text_changed(on_changed),
hstack(( hstack((
@@ -240,6 +247,14 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
let set_screen = &props.svc.set_screen; let set_screen = &props.svc.set_screen;
let set_status = &props.svc.set_status; let set_status = &props.svc.set_status;
let (manual, set_manual) = cx.use_state(String::new()); let (manual, set_manual) = cx.use_state(String::new());
// The Add-host field's live value, read by Connect at click time. This page's `use_state` is
// unreliable as the click's source of truth: while the modal is open the page usually has no
// reason to re-render (you open it precisely because the host ISN'T being discovered, so no
// discovery tick fires), and the top-down reconcile skips this unchanged-props subtree — so a
// sync `set_manual` write never re-renders the Connect button to re-capture the address, and it
// would connect to the empty mount-time value. Mirror every keystroke into this stable ref (the
// pair-screen PIN pattern). `manual` still drives the text box's displayed value.
let manual_live = cx.use_ref(String::new());
// "Add host" modal open state lives in ROOT (see `HostsProps`). // "Add host" modal open state lives in ROOT (see `HostsProps`).
let show_add = props.show_add; let show_add = props.show_add;
let set_show_add = &props.set_show_add; let set_show_add = &props.set_show_add;
@@ -249,6 +264,18 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
let rename = props.rename.clone(); let rename = props.rename.clone();
let set_forget = &props.set_forget; let set_forget = &props.set_forget;
let set_rename = &props.set_rename; let set_rename = &props.set_rename;
// The live rename draft, read at Save time (see `rename_editor`). Root `rename` carries only the
// INITIAL name, so it no longer round-trips per keystroke. Seed the draft each time the rename
// TARGET changes (start, cancel, or a switch to another host).
let rename_draft = cx.use_ref(String::new());
let rename_seed = cx.use_ref(Option::<String>::None);
{
let active = rename.as_ref().map(|(fp, _)| fp.clone());
if *rename_seed.borrow() != active {
rename_draft.set(rename.as_ref().map(|(_, n)| n.clone()).unwrap_or_default());
rename_seed.set(active);
}
}
let hover = Hover { let hover = Hover {
current: props.hover.clone(), current: props.hover.clone(),
set: props.set_hover.clone(), set: props.set_hover.clone(),
@@ -393,8 +420,13 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
for k in &known.hosts { for k in &known.hosts {
// Rust 2021 (no let-chains): match the "this tile is being renamed" case explicitly. // Rust 2021 (no let-chains): match the "this tile is being renamed" case explicitly.
if matches!(&rename, Some((fp, _)) if fp == &k.fp_hex) { if matches!(&rename, Some((fp, _)) if fp == &k.fp_hex) {
let (fp, draft) = rename.clone().unwrap(); let (fp, initial) = rename.clone().unwrap();
tiles.push(rename_editor(&draft, fp, set_rename.clone())); tiles.push(rename_editor(
&initial,
fp,
rename_draft.clone(),
set_rename.clone(),
));
continue; continue;
} }
let target = Target { let target = Target {
@@ -595,14 +627,15 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
// field). The scrim border fills the cell and is hit-testable, so it blocks the page behind; // field). The scrim border fills the cell and is hit-testable, so it blocks the page behind;
// it closes only via Cancel/Connect (a scrim tap would bubble `Tapped` up from the card too). // it closes only via Cancel/Connect (a scrim tap would bubble `Tapped` up from the card too).
let connect_manual = { let connect_manual = {
let (ctx2, ss, st, text, sa) = ( let (ctx2, ss, st, live, sa) = (
ctx.clone(), ctx.clone(),
set_screen.clone(), set_screen.clone(),
set_status.clone(), set_status.clone(),
manual.clone(), manual_live.clone(),
set_show_add.clone(), set_show_add.clone(),
); );
move || { move || {
let text = live.borrow();
let text = text.trim(); let text = text.trim();
if text.is_empty() { if text.is_empty() {
return; return;
@@ -640,7 +673,13 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
text_box(manual) text_box(manual)
.header("Address") .header("Address")
.placeholder_text("192.168.1.20 or my-pc.local") .placeholder_text("192.168.1.20 or my-pc.local")
.on_text_changed(move |s| set_manual.call(s)) .on_text_changed({
let live = manual_live.clone();
move |s: String| {
live.set(s.clone());
set_manual.call(s);
}
})
.margin(edges(0.0, 6.0, 0.0, 0.0)), .margin(edges(0.0, 6.0, 0.0, 0.0)),
hstack(( hstack((
button("Connect") button("Connect")
+20 -4
View File
@@ -14,21 +14,28 @@ pub(crate) fn pair_page(props: &Svc, cx: &mut RenderCx) -> Element {
let set_screen = &props.set_screen; let set_screen = &props.set_screen;
let set_status = &props.set_status; let set_status = &props.set_status;
let (code, set_code) = cx.use_state(String::new()); let (code, set_code) = cx.use_state(String::new());
// The PIN's live value, read directly by the click handler. This page's props (`Svc`) never
// change, and root wraps every screen in an animated `border` that compares equal once the
// entrance tween settles — so the top-down reconcile `can_skip_update`s this subtree and never
// re-renders the pair component off its *local* `use_state`. A button rebuilt only at mount
// would forever capture the empty mount-time PIN (pairing then fails as a "wrong PIN"). Mirror
// every keystroke into this stable ref instead, so the click reads exactly what was typed.
let live_pin = cx.use_ref(String::new());
let target = ctx.shared.target.lock().unwrap().clone(); let target = ctx.shared.target.lock().unwrap().clone();
let pair_btn = { let pair_btn = {
let (ctx2, ss, st, code2, target2) = ( let (ctx2, ss, st, live, target2) = (
ctx.clone(), ctx.clone(),
set_screen.clone(), set_screen.clone(),
set_status.clone(), set_status.clone(),
code.clone(), live_pin.clone(),
target.clone(), target.clone(),
); );
button("Pair & Connect") button("Pair & Connect")
.accent() .accent()
.icon(Symbol::Accept) .icon(Symbol::Accept)
.on_click(move || { .on_click(move || {
let pin = code2.trim().to_string(); let pin = live.borrow().trim().to_string();
let (ctx3, ss, st, target3) = let (ctx3, ss, st, target3) =
(ctx2.clone(), ss.clone(), st.clone(), target2.clone()); (ctx2.clone(), ss.clone(), st.clone(), target2.clone());
std::thread::spawn(move || { std::thread::spawn(move || {
@@ -109,7 +116,16 @@ pub(crate) fn pair_page(props: &Svc, cx: &mut RenderCx) -> Element {
text_box(code) text_box(code)
.placeholder_text("PIN") .placeholder_text("PIN")
.font_size(28.0) .font_size(28.0)
.on_text_changed(move |s| set_code.call(s)), .on_text_changed({
let live = live_pin.clone();
move |s: String| {
// Record the live value for the click handler (the source of truth for the
// PIN), and mirror it into `code` so the field stays correct if anything ever
// does re-render this page (theme/DPI change).
live.set(s.clone());
set_code.call(s);
}
}),
hstack((pair_btn, cancel_btn)).spacing(8.0), hstack((pair_btn, cancel_btn)).spacing(8.0),
text_block( text_block(
"Don\u{2019}t have a PIN? Request access instead and approve this device on the host \ "Don\u{2019}t have a PIN? Request access instead and approve this device on the host \
+318 -5
View File
@@ -104,6 +104,81 @@ pub struct Stats {
/// IDR (or a mid-GOP join) unfreezes almost immediately instead of never. /// IDR (or a mid-GOP join) unfreezes almost immediately instead of never.
const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3; const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3;
/// Longest the pump holds the last good frame waiting for a post-loss re-anchor keyframe before it
/// gives up and resumes display. After a reference loss the hardware decoder does not error — it
/// conceals the reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as
/// a gray plate with the new frame's motion painted over it) and returns Ok, so displaying them is
/// the "gray frames mid-stream" artifact. We instead freeze on the last good picture until a fresh
/// IDR re-anchors decode — the behaviour NVIDIA already shows (its DISTINCT output image + different
/// concealment reads as a brief freeze, not gray). This cap only bounds the freeze when recovery
/// genuinely stalls (host ignores the request, or an RFI recovery that never emits a keyframe), so a
/// glitch can never become a permanent freeze. A recovery IDR round-trips well under this on any
/// live link.
const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500);
/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
/// latest frame gap before the pump lifts its freeze on an IDR-free stream. TWO, not one: with a
/// continuous rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a
/// loss is only partially healed — stripes swept BEFORE the loss still reference the lost frame — and
/// lifting there would flash a partially-stale picture. The SECOND boundary guarantees a full wave
/// swept entirely after the loss, so the picture is clean. This stays correct under repeated loss
/// because every new gap resets the count. The cost is up to ~2 wave periods of holding the last good
/// frame — the deliberate "hold longer, never show garbage" trade.
///
/// [`USER_FLAG_RECOVERY_POINT`]: punktfunk_core::packet::USER_FLAG_RECOVERY_POINT
const REANCHOR_MARKS_TO_LIFT: u32 = 2;
/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark
/// pushes the freeze deadline out by this much, so a live mark stream (the host actively healing via
/// its wave) keeps the client patiently holding the last good frame instead of tripping the IDR
/// floor mid-heal. Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the
/// marks STOP (heal stalled, or the host isn't running intra-refresh) the deadline lapses and the
/// normal recovery-IDR floor fires, so a real stall still recovers.
const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500);
/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is
/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters
/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small
/// positive delta is a forward gap (missing frames whose dependents will decode against absent
/// references); a delta in the top half is an index behind us.
fn index_gap(expected: u32, got: u32) -> Option<u32> {
let ahead = got.wrapping_sub(expected);
(ahead != 0 && ahead < u32::MAX / 2).then_some(ahead)
}
/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze.
///
/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried
/// [`USER_FLAG_RECOVERY_ANCHOR`](punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR), the host's
/// definitive single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a
/// known-good reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait.
/// `has_mark` — this AU carried [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// a host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks
/// seen since the latest gap.
///
/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0
/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested
/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only
/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and
/// lifts immediately.
fn reanchor_after_frame(
is_keyframe: bool,
has_anchor: bool,
has_mark: bool,
marks: u32,
) -> (bool, u32) {
let marks = if has_mark {
marks.saturating_add(1)
} else {
marks
};
if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT {
(true, 0)
} else {
(false, marks)
}
}
/// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs). /// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs).
/// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old /// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old
/// host (no 0xCF at all) this just caps the dead-weight ring. /// host (no 0xCF at all) this just caps the dead-weight ring.
@@ -319,6 +394,20 @@ fn pump(
// never drops, so the drop-count trigger below stays silent and the stream freezes // never drops, so the drop-count trigger below stays silent and the stream freezes
// on the last good frame. A short streak forces a fresh IDR to re-anchor. // on the last good frame. A short streak forces a fresh IDR to re-anchor.
let mut no_output_streak = 0u32; let mut no_output_streak = 0u32;
// Freeze-until-reanchor: armed the moment we request a recovery keyframe (loss, decode error, or
// a no-output streak), it withholds the decoder's concealed frames from the presenter — which
// then redraws the last good picture — until a fresh keyframe re-anchors decode. See
// [`REANCHOR_FREEZE_MAX`] for why this exists and its backstop deadline.
let mut awaiting_reanchor = false;
let mut reanchor_deadline: Option<Instant> = None;
// Host intra-refresh recovery marks seen since the latest gap (see [`REANCHOR_MARKS_TO_LIFT`]).
// Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks.
let mut recovery_marks: u32 = 0;
// The frame_index we expect next (the host numbers frames consecutively). A jump means a frame
// went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms
// ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of
// the loss window, by which point the concealment already reached the screen).
let mut next_expected_index: Option<u32> = None;
let end: Option<String> = loop { let end: Option<String> = loop {
if stop.load(Ordering::SeqCst) { if stop.load(Ordering::SeqCst) {
@@ -334,9 +423,90 @@ fn pump(
// fps / goodput count every received AU (spec), decoded or not. // fps / goodput count every received AU (spec), decoded or not.
frames_n += 1; frames_n += 1;
bytes_n += frame.data.len() as u64; bytes_n += frame.data.len() as u64;
// Reference-continuity gate: the host numbers frames consecutively, so a jump in
// frame_index means a frame is missing (lost, or an out-of-order straggler the
// reassembler emitted a newer frame ahead of) and this AU references a picture we
// never decoded. On RADV the decoder conceals that as a gray plate with the new
// motion on top — the reported artifact, and it shows most on high-motion frames (a
// full-screen pan bursts far more packets than a static desktop or a UFO-test's small
// moving sprite, so it is the frame that loses shards). Arm the freeze at the FIRST
// such frame — ~120 ms before `frames_dropped` would — so the gray never reaches the
// screen; recovery IDRs stay on the existing throttled path (see the arm below).
match next_expected_index {
Some(exp) if frame.frame_index == exp => {
next_expected_index = Some(exp.wrapping_add(1)); // contiguous
}
// A forward gap: hold the last good frame — but DO NOT ask for a keyframe here.
// Hiding the concealment is free (the presenter redraws the last picture); an IDR
// is not — at 4K120 it is a multi-megabyte frame and a visible stutter, and it can
// re-trigger the very burst loss that caused this. The existing loss recovery below
// (`frames_dropped`, host-coalesced + throttled) still requests it at exactly the
// cadence it did before this change, so we add zero IDR pressure per pan. A
// straggler behind us (`index_gap` → None) leaves the expectation put so the real
// gap still trips.
Some(exp) => {
if let Some(gap) = index_gap(exp, frame.frame_index) {
let now = Instant::now();
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
next_expected_index = Some(frame.frame_index.wrapping_add(1));
// The gap carries the PRECISE lost range — [first missing, newest
// received - 1] — so this is the one recovery signal that can drive true
// reference-frame invalidation. Prefer an RFI request over a keyframe: an
// RFI-capable host (AMD LTR / NVENC) re-references a known-good picture and
// emits a clean P-frame tagged USER_FLAG_RECOVERY_ANCHOR (the freeze lifts
// on ONE frame, no 20-40× IDR spike); an incapable/old host forces a
// host-coalesced IDR instead, or ignores it (then the frames_dropped /
// overdue keyframe paths below are the backstop). Throttled with those
// paths (one recovery ask per 100 ms) so a burst of gaps — a full-screen
// pan shedding shards — can't storm the control stream. This fires ~120 ms
// before frames_dropped would, so recovery also starts sooner.
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now);
let _ = connector
.request_rfi(exp, frame.frame_index.wrapping_sub(1));
}
tracing::trace!(gap, "frame gap — RFI recovery, holding last frame until re-anchor");
}
}
None => next_expected_index = Some(frame.frame_index.wrapping_add(1)),
}
match decoder.decode(&frame.data) { match decoder.decode(&frame.data) {
Ok(Some(image)) => { Ok(Some(image)) => {
no_output_streak = 0; // a decoded frame — the anchor holds no_output_streak = 0; // a decoded frame — the anchor holds
// Host-signalled intra-refresh recovery mark: on an IDR-free intra-refresh
// stream this wave-boundary flag is the only clean point the client can honor
// (the decoder never flags the re-anchor — the coded frame stays `P`). A live
// mark stream also means the host is actively healing, so push the backstop out
// rather than trip a mid-heal IDR (see `RECOVERY_MARK_PATIENCE`).
let has_mark =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_POINT != 0;
// The host's definitive single-frame re-anchor: an LTR-RFI recovery frame (a
// clean P-frame off a known-good reference), the AMD twin of an IDR re-anchor
// but without the spike. It lifts on the FIRST occurrence.
let has_anchor =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR != 0;
if has_mark && awaiting_reanchor {
reanchor_deadline = Some(Instant::now() + RECOVERY_MARK_PATIENCE);
}
// A fresh clean re-anchor lifts the freeze and shows this frame: a real intra
// keyframe (IDR, always clean), an LTR-RFI recovery anchor (also whole), OR the
// second recovery mark since the gap (the first wave boundary is only
// half-healed — see `reanchor_after_frame`).
let (lift, marks) = reanchor_after_frame(
image.is_keyframe(),
has_anchor,
has_mark,
recovery_marks,
);
recovery_marks = marks;
if lift {
awaiting_reanchor = false;
reanchor_deadline = None;
}
total_frames += 1; total_frames += 1;
dec_path = match &image { dec_path = match &image {
DecodedImage::Cpu(_) => "software", DecodedImage::Cpu(_) => "software",
@@ -391,11 +561,20 @@ fn pump(
DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)), DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)),
_ => None, _ => None,
}; };
let _ = frame_tx.force_send(DecodedFrame { if awaiting_reanchor {
pts_ns: frame.pts_ns, // Post-loss concealment: withhold this frame (it references a lost/gray
decoded_ns, // reference) so the presenter keeps redrawing the last good picture
image, // rather than flashing the decoder's gray plate. Dropped here — the
}); // hw-decode stat below still samples via `hw_fence` (raw handle + value,
// valid past the guard). Cleared by the next keyframe or the backstop.
tracing::trace!("holding last frame — awaiting post-loss re-anchor");
} else {
let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns,
decoded_ns,
image,
});
}
// `decode` stage: received→decode COMPLETE, single clock. // `decode` stage: received→decode COMPLETE, single clock.
match hw_fence { match hw_fence {
Some((sem, value)) => { Some((sem, value)) => {
@@ -424,6 +603,12 @@ fn pump(
// trip before asking again instead of flooding. // trip before asking again instead of flooding.
if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK { if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK {
let now = Instant::now(); let now = Instant::now();
// Wedged on missing references: hold the last good frame until re-anchor
// (armed even when the IDR request itself is throttled — the stream is broken
// regardless of whether we ask again this iteration).
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{ {
@@ -451,6 +636,9 @@ fn pump(
// through the same throttle as loss recovery below. // through the same throttle as loss recovery below.
if decoder.take_keyframe_request() { if decoder.take_keyframe_request() {
let now = Instant::now(); let now = Instant::now();
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{ {
@@ -487,12 +675,33 @@ fn pump(
if dropped > last_dropped { if dropped > last_dropped {
last_dropped = dropped; last_dropped = dropped;
let now = Instant::now(); let now = Instant::now();
// A dropped AU means the frames after it reference a picture we never decoded — the
// decoder will conceal them (gray on RADV). Freeze on the last good frame until a fresh
// IDR re-anchors, so the concealment never reaches the screen.
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now); last_kf_req = Some(now);
let _ = connector.request_keyframe(); let _ = connector.request_keyframe();
tracing::debug!(dropped, "requested keyframe (loss recovery)"); tracing::debug!(dropped, "requested keyframe (loss recovery)");
} }
} }
// Re-anchor overdue: the freeze has held the whole window with no keyframe — a lost recovery
// IDR, or a benign reorder that produced no `frames_dropped` and so requested none. Do NOT
// resume to gray (the one thing worse than a freeze): keep holding the last good frame and
// (re-)request a keyframe, throttled + host-coalesced, so a CLEAN re-anchor is what un-freezes
// us. A genuinely dead stream — host gone, link collapsed — is caught by the QUIC idle-timeout
// watchdog (returns to the menu), never by painting the decoder's concealment.
if awaiting_reanchor && reanchor_deadline.is_some_and(|d| Instant::now() >= d) {
let now = Instant::now();
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!("re-anchor overdue — still holding, re-requesting keyframe");
}
}
if window_start.elapsed() >= Duration::from_secs(1) { if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32(); let secs = window_start.elapsed().as_secs_f32();
@@ -614,3 +823,107 @@ fn spawn_audio(
.map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled")) .map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled"))
.ok() .ok()
} }
#[cfg(test)]
mod tests {
use super::{index_gap, reanchor_after_frame, REANCHOR_MARKS_TO_LIFT};
// Simulate the pump's re-anchor state across a sequence of decoded frames: each `(is_keyframe,
// has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based)
// at which the freeze first lifts, or `None` if it never does. `gap_before` reset points model a
// fresh loss re-arming the freeze (the pump zeroes the count at every gap/arm site).
fn lift_at(frames: &[(bool, bool)]) -> Option<usize> {
let mut marks = 0u32;
for (i, &(is_kf, has_mark)) in frames.iter().enumerate() {
// The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised
// by `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here.
let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks);
marks = m;
if lift {
return Some(i);
}
}
None
}
#[test]
fn a_single_recovery_mark_does_not_lift() {
// The first wave boundary after a loss is only half-healed — one mark must hold the freeze.
assert_eq!(REANCHOR_MARKS_TO_LIFT, 2);
assert_eq!(lift_at(&[(false, true)]), None);
assert_eq!(lift_at(&[(false, false), (false, true), (false, false)]), None);
}
#[test]
fn the_second_recovery_mark_lifts() {
// Two marks = a full wave swept after the loss → clean re-anchor.
assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1));
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false), (false, true)]),
Some(3)
);
}
#[test]
fn a_real_keyframe_lifts_immediately() {
// An IDR is always a clean anchor — no marks needed.
assert_eq!(lift_at(&[(true, false)]), Some(0));
assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1));
}
#[test]
fn a_fresh_gap_resets_the_mark_count() {
// The pump zeroes `recovery_marks` at each arm site, so one mark before a new gap plus one
// after must NOT lift — the model resets the running count to imitate that.
let mut marks = 0u32;
let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap)
marks = m;
assert_eq!(marks, 1);
marks = 0; // a new gap re-arms the freeze → count reset
let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave
assert!(!lift, "a single post-gap mark must not lift");
assert_eq!(m, 1);
}
#[test]
fn an_rfi_anchor_lifts_immediately() {
// An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good
// reference), so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait.
let (lift, marks) = reanchor_after_frame(false, true, false, 0);
assert!(lift, "an RFI anchor must lift the freeze immediately");
assert_eq!(marks, 0, "a lift resets the running mark count");
// Even with zero prior marks and no keyframe, the anchor alone is sufficient.
let (lift, _) = reanchor_after_frame(false, true, true, 1);
assert!(lift, "an anchor lifts regardless of the pending mark count");
}
#[test]
fn contiguous_indices_are_not_a_gap() {
assert_eq!(index_gap(5, 5), None);
assert_eq!(index_gap(0, 0), None);
}
#[test]
fn a_forward_jump_reports_the_skip_count() {
assert_eq!(index_gap(5, 6), Some(1)); // one frame missing
assert_eq!(index_gap(5, 9), Some(4));
}
#[test]
fn a_straggler_behind_us_is_not_a_gap() {
// The reassembler emitted a newer frame first; the late one must not re-arm.
assert_eq!(index_gap(9, 5), None);
assert_eq!(index_gap(1, 0), None);
}
#[test]
fn the_index_counter_wraps_cleanly() {
// last frame = u32::MAX, so the next expected wraps to 0.
assert_eq!(index_gap(0, 0), None); // contiguous across the wrap
// waiting on u32::MAX, frame 0 arrived → MAX was skipped.
assert_eq!(index_gap(u32::MAX, 0), Some(1));
assert_eq!(index_gap(u32::MAX, 2), Some(3));
// an old frame arriving just after the wrap is still a straggler.
assert_eq!(index_gap(0, u32::MAX), None);
}
}
+51
View File
@@ -98,6 +98,10 @@ pub struct VkVideoFrame {
pub width: u32, pub width: u32,
pub height: u32, pub height: u32,
pub color: ColorDesc, pub color: ColorDesc,
/// Intra keyframe (IDR/I): the stream's re-anchor point. The pump resumes display on
/// one after suppressing the concealed frames a reference loss leaves in its wake (on
/// RADV a lost reference decodes to a gray plate with the new motion painted on top).
pub keyframe: bool,
/// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive /// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive
/// until the presenter's fence proves the GPU reads done — same mechanism as the /// until the presenter's fence proves the GPU reads done — same mechanism as the
/// VAAPI path's DRM guard. /// VAAPI path's DRM guard.
@@ -143,6 +147,44 @@ impl ColorDesc {
} }
} }
/// True if the decoder tagged this frame as a full IDR keyframe — a guaranteed clean re-anchor
/// after which the picture is loss-free, so the pump can lift a post-loss display freeze here.
///
/// Keys off `AV_FRAME_FLAG_KEY` (with `pict_type == I` as a belt for decoders that fill pict_type
/// but not the flag). NOTE: FFmpeg's H.264/HEVC decode layer sets this flag **only for true IDR
/// frames**, never for an *intra-refresh recovery point*. H.264 flags key only when a picture's
/// `recovery_frame_cnt == 0` (a moving band uses `> 0`); HEVC clears the flag on every non-IRAP
/// frame regardless of the recovery-point SEI. So an intra-refresh host (NVENC/AMF/QSV) heals the
/// picture over N P-frames with no decoded frame ever flagged key — this function cannot detect
/// that clean point, and the pump would freeze until the `REANCHOR_FREEZE_MAX` backstop (in
/// `session.rs`) forces a real IDR. Detecting an intra-refresh re-anchor requires an out-of-band
/// host wire signal on the AU that completes the wave; that is not yet plumbed.
///
/// # Safety
/// `frame` must point to a valid `AVFrame` alive for the duration of the call.
pub unsafe fn frame_is_keyframe(frame: *const ffmpeg::ffi::AVFrame) -> bool {
// SAFETY: caller guarantees a live AVFrame; plain field reads.
unsafe {
((*frame).flags & ffmpeg::ffi::AV_FRAME_FLAG_KEY) != 0
|| (*frame).pict_type == ffmpeg::ffi::AVPictureType::AV_PICTURE_TYPE_I
}
}
impl DecodedImage {
/// Whether the frame is an intra keyframe — see [`frame_is_keyframe`]. The pump uses
/// this as the stream's re-anchor signal after a loss.
pub fn is_keyframe(&self) -> bool {
match self {
DecodedImage::Cpu(f) => f.keyframe,
#[cfg(target_os = "linux")]
DecodedImage::Dmabuf(f) => f.keyframe,
DecodedImage::VkFrame(f) => f.keyframe,
#[cfg(windows)]
DecodedImage::D3d11(f) => f.keyframe,
}
}
}
/// The YCbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant /// The YCbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant
/// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient /// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient
/// implementation every presenter derives its CSC from (Vulkan push constants, the Windows /// implementation every presenter derives its CSC from (Vulkan push constants, the Windows
@@ -205,6 +247,8 @@ pub struct CpuFrame {
/// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries /// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries
/// baked in — the presenter tags the texture so GTK tone-maps it. /// baked in — the presenter tags the texture so GTK tone-maps it.
pub color: ColorDesc, pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
pub keyframe: bool,
} }
/// A decoded frame still on the GPU: dmabuf fds + plane layout for /// A decoded frame still on the GPU: dmabuf fds + plane layout for
@@ -222,6 +266,8 @@ pub struct DmabufFrame {
/// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709 /// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709
/// narrow for SDR, BT.2020 PQ for an HDR stream). /// narrow for SDR, BT.2020 PQ for an HDR stream).
pub color: ColorDesc, pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
pub keyframe: bool,
pub guard: DrmFrameGuard, pub guard: DrmFrameGuard,
} }
@@ -644,6 +690,9 @@ impl SoftwareDecoder {
stride: dst_linesize[0] as usize, stride: dst_linesize[0] as usize,
rgba, rgba,
color, color,
// `is_key()` reads the same intra flag `frame_is_keyframe` derives from pict_type
// for the hardware paths; ffmpeg-next handles the FFmpeg-version binding split.
keyframe: frame.is_key(),
}) })
} }
} }
@@ -844,6 +893,7 @@ impl VaapiDecoder {
// SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after // SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after
// this returns); plain CICP field reads. // this returns); plain CICP field reads.
color: ColorDesc::from_raw(self.frame), color: ColorDesc::from_raw(self.frame),
keyframe: frame_is_keyframe(self.frame),
guard, guard,
}) })
} }
@@ -1363,6 +1413,7 @@ impl VulkanDecoder {
width: (*self.frame).width as u32, width: (*self.frame).width as u32,
height: (*self.frame).height as u32, height: (*self.frame).height as u32,
color: ColorDesc::from_raw(self.frame), color: ColorDesc::from_raw(self.frame),
keyframe: frame_is_keyframe(self.frame),
guard: DrmFrameGuard(clone), guard: DrmFrameGuard(clone),
}) })
} }
+5
View File
@@ -99,6 +99,9 @@ pub struct D3d11Frame {
/// BT.709 full-range RGB — regardless of the stream's own CICP (a PQ stream was /// BT.709 full-range RGB — regardless of the stream's own CICP (a PQ stream was
/// tone-mapped). The presenter keys SDR/HDR handling off this, so it always reads SDR. /// tone-mapped). The presenter keys SDR/HDR handling off this, so it always reads SDR.
pub color: ColorDesc, pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See
/// `crate::video::VkVideoFrame`.
pub keyframe: bool,
/// The ring slot's NT shared handle (`IDXGIResource1::CreateSharedHandle`), stable for the /// The ring slot's NT shared handle (`IDXGIResource1::CreateSharedHandle`), stable for the
/// ring's lifetime. Raw `isize` so the frame crosses the pump→presenter channel. /// ring's lifetime. Raw `isize` so the frame crosses the pump→presenter channel.
pub handle: isize, pub handle: isize,
@@ -692,6 +695,8 @@ impl D3d11vaDecoder {
matrix: 0, // identity — RGB matrix: 0, // identity — RGB
full_range: true, full_range: true,
}, },
// SAFETY: `self.frame` is the live decoded AVFrame for this call.
keyframe: crate::video::frame_is_keyframe(self.frame),
handle, handle,
generation, generation,
}) })
+25 -1
View File
@@ -19,7 +19,8 @@ use crate::packet::FLAG_PROBE;
use crate::quic::{ use crate::quic::{
accept_resync, endpoint, io, wall_clock_ns, window_loss_ppm, BitrateChanged, ClockEcho, accept_resync, endpoint, io, wall_clock_ns, window_loss_ppm, BitrateChanged, ClockEcho,
ClockResync, ColorInfo, HdrMeta, Hello, HidOutput, LossReport, ProbeRequest, ProbeResult, ClockResync, ColorInfo, HdrMeta, Hello, HidOutput, LossReport, ProbeRequest, ProbeResult,
Reconfigure, Reconfigured, RequestKeyframe, ResyncStep, RichInput, SetBitrate, Start, Welcome, Reconfigure, Reconfigured, RequestKeyframe, RfiRequest, ResyncStep, RichInput, SetBitrate, Start,
Welcome,
}; };
use crate::session::{Frame, Session}; use crate::session::{Frame, Session};
use crate::transport::UdpTransport; use crate::transport::UdpTransport;
@@ -49,6 +50,10 @@ enum CtrlRequest {
Mode(Mode), Mode(Mode),
Probe(ProbeRequest), Probe(ProbeRequest),
Keyframe, Keyframe,
/// Reference-frame-invalidation recovery: the client saw a `frame_index` gap and reports the
/// invalidation range so an RFI-capable host re-references a known-good picture instead of
/// forcing a full IDR. See [`RfiRequest`].
Rfi(RfiRequest),
Loss(LossReport), Loss(LossReport),
/// Adaptive bitrate: ask the host to re-target its encoder (kbps). Sent by the pump's /// Adaptive bitrate: ask the host to re-target its encoder (kbps). Sent by the pump's
/// [`BitrateController`] when the user's bitrate setting is Automatic. /// [`BitrateController`] when the user's bitrate setting is Automatic.
@@ -868,6 +873,24 @@ impl NativeClient {
.map_err(|_| PunktfunkError::Closed) .map_err(|_| PunktfunkError::Closed)
} }
/// Ask the host to recover from loss by **reference-frame invalidation** rather than a full IDR:
/// the client reports the range `[first_frame, last_frame]` of access units it can no longer trust
/// (from the first missing `frame_index` through the newest received). An RFI-capable host
/// re-references a known-good picture before `first_frame` (AMD LTR / NVENC RFI) and emits a clean
/// P-frame tagged [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`]; a host that can't RFI forces an IDR
/// instead (same as [`request_keyframe`](Self::request_keyframe)). Non-blocking, fire-and-forget —
/// the recovered frame is the only ack; throttle it like the keyframe request. Prefer this over
/// `request_keyframe` on loss so AMD/RFI hosts avoid the IDR spike; the keyframe request remains
/// the backstop when the recovery frame itself is lost.
pub fn request_rfi(&self, first_frame: u32, last_frame: u32) -> Result<()> {
self.ctrl_tx
.try_send(CtrlRequest::Rfi(RfiRequest {
first_frame,
last_frame,
}))
.map_err(|_| PunktfunkError::Closed)
}
/// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't /// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't
/// rebuild them). A video loop polls this and calls [`request_keyframe`](Self::request_keyframe) /// rebuild them). A video loop polls this and calls [`request_keyframe`](Self::request_keyframe)
/// when it increases — the correct loss trigger under infinite GOP, where unrecoverable loss /// when it increases — the correct loss trigger under infinite GOP, where unrecoverable loss
@@ -1511,6 +1534,7 @@ async fn worker_main(args: WorkerArgs) {
CtrlRequest::Mode(m) => Reconfigure { mode: m }.encode(), CtrlRequest::Mode(m) => Reconfigure { mode: m }.encode(),
CtrlRequest::Probe(p) => p.encode(), CtrlRequest::Probe(p) => p.encode(),
CtrlRequest::Keyframe => RequestKeyframe.encode(), CtrlRequest::Keyframe => RequestKeyframe.encode(),
CtrlRequest::Rfi(r) => r.encode(),
CtrlRequest::Loss(r) => r.encode(), CtrlRequest::Loss(r) => r.encode(),
CtrlRequest::SetBitrate(k) => SetBitrate { bitrate_kbps: k }.encode(), CtrlRequest::SetBitrate(k) => SetBitrate { bitrate_kbps: k }.encode(),
CtrlRequest::ClockResync => { CtrlRequest::ClockResync => {
+19
View File
@@ -35,6 +35,25 @@ pub const FLAG_SOF: u8 = 0x4;
/// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it). /// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it).
pub const FLAG_PROBE: u8 = 0x8; pub const FLAG_PROBE: u8 = 0x8;
/// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client
/// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that
/// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is
/// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its
/// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible
/// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host
/// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four.
pub const USER_FLAG_RECOVERY_POINT: u32 = 0x10;
/// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike
/// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss
/// is only half-healed so the client waits for the second), this marks an access unit the host coded
/// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation**
/// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client
/// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts
/// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets
/// `AV_FRAME_FLAG_KEY` — this host flag is the only signal.
pub const USER_FLAG_RECOVERY_ANCHOR: u32 = 0x20;
/// Crypto framing overhead [`Session`](crate::session::Session) adds when encrypting: /// Crypto framing overhead [`Session`](crate::session::Session) adds when encrypting:
/// an 8-byte sequence prefix plus the GCM tag. /// an 8-byte sequence prefix plus the GCM tag.
pub const CRYPTO_OVERHEAD: usize = 8 + crate::crypto::TAG_LEN; pub const CRYPTO_OVERHEAD: usize = 8 + crate::crypto::TAG_LEN;
+42
View File
@@ -355,6 +355,24 @@ pub struct Reconfigured {
#[derive(Clone, Copy, Debug, PartialEq, Eq)] #[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct RequestKeyframe; pub struct RequestKeyframe;
/// `client → host`: reference-frame-invalidation recovery — the loss-aware sibling of
/// [`RequestKeyframe`]. The client detected a `frame_index` gap and reports the range `[first_frame,
/// last_frame]` of access units it can no longer trust (from the first missing index through the
/// newest received). Instead of a full IDR (a 20-40× spike that deepens the loss it recovers), a host
/// whose encoder supports RFI re-references a known-good picture *before* `first_frame` — an AMD LTR
/// force-reference or an NVENC `nvEncInvalidateRefFrames` — emitting a single clean P-frame it tags
/// [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its freeze on it. A host that
/// can't RFI (no valid reference / libavcodec backend) forces an IDR instead, exactly as for a bare
/// [`RequestKeyframe`]; a host that predates this ignores the unknown message and the client's
/// keyframe backstop still recovers. Fire-and-forget — the recovered frame is the only ack.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct RfiRequest {
/// First access-unit `frame_index` the client can no longer trust (the gap start).
pub first_frame: u32,
/// Newest received `frame_index` at the time of the report (the invalidation range end).
pub last_frame: u32,
}
/// `client → host`, periodic: the client's observed data-plane loss, so the host can size FEC to /// `client → host`, periodic: the client's observed data-plane loss, so the host can size FEC to
/// the link instead of a flat percentage (adaptive FEC). `loss_ppm` is parts-per-million of shards /// the link instead of a flat percentage (adaptive FEC). `loss_ppm` is parts-per-million of shards
/// that arrived missing-but-recovered (plus a bump when frames went unrecoverable) over the report /// that arrived missing-but-recovered (plus a bump when frames went unrecoverable) over the report
@@ -467,6 +485,8 @@ pub const MSG_LOSS_REPORT: u8 = 0x04;
pub const MSG_SET_BITRATE: u8 = 0x05; pub const MSG_SET_BITRATE: u8 = 0x05;
/// Type byte of [`BitrateChanged`]. /// Type byte of [`BitrateChanged`].
pub const MSG_BITRATE_CHANGED: u8 = 0x06; pub const MSG_BITRATE_CHANGED: u8 = 0x06;
/// Type byte of [`RfiRequest`].
pub const MSG_RFI_REQUEST: u8 = 0x07;
/// Type byte of [`ProbeRequest`]. /// Type byte of [`ProbeRequest`].
pub const MSG_PROBE_REQUEST: u8 = 0x20; pub const MSG_PROBE_REQUEST: u8 = 0x20;
/// Type byte of [`ProbeResult`]. /// Type byte of [`ProbeResult`].
@@ -1032,6 +1052,28 @@ impl RequestKeyframe {
} }
} }
impl RfiRequest {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] first_frame[5..9] last_frame[9..13]
let mut b = Vec::with_capacity(13);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_RFI_REQUEST);
b.extend_from_slice(&self.first_frame.to_le_bytes());
b.extend_from_slice(&self.last_frame.to_le_bytes());
b
}
pub fn decode(b: &[u8]) -> Result<RfiRequest> {
if b.len() != 13 || &b[0..4] != CTL_MAGIC || b[4] != MSG_RFI_REQUEST {
return Err(PunktfunkError::InvalidArg("bad RfiRequest"));
}
Ok(RfiRequest {
first_frame: u32::from_le_bytes(b[5..9].try_into().unwrap()),
last_frame: u32::from_le_bytes(b[9..13].try_into().unwrap()),
})
}
}
impl LossReport { impl LossReport {
pub fn encode(&self) -> Vec<u8> { pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] loss_ppm[5..9] // magic[0..4] type[4] loss_ppm[5..9]
+29
View File
@@ -633,6 +633,35 @@ fn request_keyframe_roundtrip() {
assert!(RequestKeyframe::decode(&[bytes.as_slice(), &[0]].concat()).is_err()); assert!(RequestKeyframe::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
} }
#[test]
fn rfi_request_roundtrip() {
for (first_frame, last_frame) in [(0u32, 0u32), (40, 47), (5, 5), (1_000_000, u32::MAX)] {
let r = RfiRequest {
first_frame,
last_frame,
};
assert_eq!(RfiRequest::decode(&r.encode()).unwrap(), r);
}
// Disjoint from the bare keyframe request (its loss-unaware sibling) and others: type byte + length.
assert!(RfiRequest::decode(&RequestKeyframe.encode()).is_err());
assert!(RequestKeyframe::decode(
&RfiRequest {
first_frame: 1,
last_frame: 2
}
.encode()
)
.is_err());
// Exact length — no trailing bytes.
let bytes = RfiRequest {
first_frame: 3,
last_frame: 9,
}
.encode();
assert!(RfiRequest::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
assert!(RfiRequest::decode(&bytes[..bytes.len() - 1]).is_err());
}
#[test] #[test]
fn loss_report_roundtrip() { fn loss_report_roundtrip() {
for loss_ppm in [0u32, 1, 12_345, 50_000, 1_000_000] { for loss_ppm in [0u32, 1, 12_345, 50_000, 1_000_000] {
+3
View File
@@ -466,5 +466,8 @@ pub mod dxgi;
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
#[path = "capture/windows/idd_push.rs"] #[path = "capture/windows/idd_push.rs"]
pub mod idd_push; pub mod idd_push;
#[cfg(target_os = "windows")]
#[path = "capture/windows/synthetic_nv12.rs"]
pub mod synthetic_nv12;
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
mod linux; mod linux;
@@ -0,0 +1,181 @@
//! A headless synthetic **NV12 D3D11** capture source for exercising the GPU encoders on Windows
//! without a real capture session.
//!
//! The native AMF path (and the D3D11 zero-copy NVENC/QSV paths) require an NV12 texture that lives
//! on the GPU — the CPU-Bgrx [`SyntheticCapturer`](crate::capture::SyntheticCapturer) can't provide
//! one, and DXGI Desktop Duplication can't create one under an ssh session-0 (E_ACCESSDENIED). This
//! source builds an NV12 texture on the selected render adapter and fills it with a **moving** luma
//! ramp each frame, so the encoder sees genuine motion (P-frame residuals + the intra-refresh wave
//! under content change) — exactly what an intra-refresh recovery validation needs. Driven by
//! `spike --source synthetic-nv12`.
use crate::capture::dxgi::{make_device, D3d11Frame};
use crate::capture::{CapturedFrame, Capturer, FramePayload, PixelFormat};
use anyhow::{Context, Result};
use windows::Win32::Graphics::Direct3D11::{
ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D, D3D11_BIND_SHADER_RESOURCE,
D3D11_CPU_ACCESS_WRITE, D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_WRITE, D3D11_TEXTURE2D_DESC,
D3D11_USAGE, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING,
};
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_NV12, DXGI_SAMPLE_DESC};
use windows::Win32::Graphics::Dxgi::{CreateDXGIFactory1, IDXGIAdapter1, IDXGIFactory4};
/// Synthetic NV12 frames on the GPU. Owns its own D3D11 device + immediate context and two NV12
/// textures: a CPU-writable STAGING scratch it fills each frame, and a DEFAULT texture it copies
/// into and hands to the encoder. The encoder copies out of the DEFAULT texture synchronously
/// (spike drives capture→submit→poll on one thread), so reusing one DEFAULT texture is sound.
pub struct SyntheticNv12Capturer {
device: ID3D11Device,
context: ID3D11DeviceContext,
default_tex: ID3D11Texture2D,
staging: ID3D11Texture2D,
width: u32,
height: u32,
fps: u32,
frame_idx: u64,
}
// SAFETY: mirrors `D3d11Frame`'s reasoning — the device is created free-threaded (`make_device`
// passes no `SINGLETHREADED` flag) and D3D11 uses interlocked COM refcounting, so moving the whole
// capturer (device + immediate context + textures) to its owning thread and using it only there is
// sound. The value is moved, never aliased (no `Sync`), so the single-threaded immediate context is
// never touched concurrently.
unsafe impl Send for SyntheticNv12Capturer {}
impl SyntheticNv12Capturer {
pub fn new(width: u32, height: u32, fps: u32) -> Result<Self> {
// NV12 is 4:2:0 — both dimensions must be even (the chroma plane is width/2 × height/2).
let width = (width & !1).max(2);
let height = (height & !1).max(2);
// SAFETY: a self-contained builder owning every handle it creates; each COM call is checked
// and the returned owners drop with their wrappers.
unsafe {
let adapter = resolve_render_adapter().context("resolve render adapter for NV12 source")?;
let (device, context) = make_device(&adapter).context("create D3D11 device")?;
let default_tex = create_nv12(
&device,
width,
height,
D3D11_USAGE_DEFAULT,
0,
D3D11_BIND_SHADER_RESOURCE.0 as u32,
)
.context("create NV12 default texture")?;
let staging = create_nv12(
&device,
width,
height,
D3D11_USAGE_STAGING,
D3D11_CPU_ACCESS_WRITE.0 as u32,
0,
)
.context("create NV12 staging texture")?;
Ok(SyntheticNv12Capturer {
device,
context,
default_tex,
staging,
width,
height,
fps,
frame_idx: 0,
})
}
}
}
impl Capturer for SyntheticNv12Capturer {
fn next_frame(&mut self) -> Result<CapturedFrame> {
let pts_ns = self.frame_idx * 1_000_000_000 / self.fps.max(1) as u64;
// SAFETY: Map/Unmap/CopyResource on this capturer's own single-threaded immediate context;
// all writes stay within the mapped NV12 surface (Y: H rows of RowPitch; UV: H/2 rows of
// RowPitch beginning at RowPitch*H — the standard NV12 plane layout).
unsafe {
let mut map = D3D11_MAPPED_SUBRESOURCE::default();
self.context
.Map(&self.staging, 0, D3D11_MAP_WRITE, 0, Some(&mut map))
.context("Map(NV12 staging)")?;
let pitch = map.RowPitch as usize;
let base = map.pData as *mut u8;
// A diagonal luma ramp that shifts 4 codes/frame — strong, deterministic motion.
let shift = (self.frame_idx as u32).wrapping_mul(4);
for y in 0..self.height {
let row = base.add(y as usize * pitch);
for x in 0..self.width {
*row.add(x as usize) = x.wrapping_add(y).wrapping_add(shift) as u8;
}
}
// UV plane (neutral gray = 128) at offset RowPitch*H: H/2 rows, `width` bytes each
// (width/2 interleaved Cb,Cr pairs).
let uv = base.add(pitch * self.height as usize);
for r in 0..(self.height / 2) {
let row = uv.add(r as usize * pitch);
for c in 0..self.width {
*row.add(c as usize) = 128;
}
}
self.context.Unmap(&self.staging, 0);
self.context.CopyResource(&self.default_tex, &self.staging);
}
self.frame_idx += 1;
Ok(CapturedFrame {
width: self.width,
height: self.height,
pts_ns,
format: PixelFormat::Nv12,
payload: FramePayload::D3d11(D3d11Frame {
texture: self.default_tex.clone(),
device: self.device.clone(),
}),
})
}
}
/// Resolve the same render adapter the encoder will pick (`PUNKTFUNK_RENDER_ADAPTER` / preference /
/// max-VRAM LUID), falling back to adapter 0.
///
/// # Safety
/// Calls DXGI factory/adapter enumeration; returns owned COM objects or an error.
unsafe fn resolve_render_adapter() -> Result<IDXGIAdapter1> {
let factory: IDXGIFactory4 = CreateDXGIFactory1().context("CreateDXGIFactory1")?;
if let Some(luid) = crate::win_adapter::resolve_render_adapter_luid() {
if let Ok(a) = factory.EnumAdapterByLuid::<IDXGIAdapter1>(luid) {
return Ok(a);
}
}
factory.EnumAdapters1(0).context("EnumAdapters1(0)")
}
/// Create an NV12 `Texture2D` with the given usage/CPU-access/bind flags.
///
/// # Safety
/// `device` must be a live D3D11 device; the returned texture is owned by the caller.
unsafe fn create_nv12(
device: &ID3D11Device,
width: u32,
height: u32,
usage: D3D11_USAGE,
cpu_access: u32,
bind: u32,
) -> Result<ID3D11Texture2D> {
let desc = D3D11_TEXTURE2D_DESC {
Width: width,
Height: height,
MipLevels: 1,
ArraySize: 1,
Format: DXGI_FORMAT_NV12,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: usage,
BindFlags: bind,
CPUAccessFlags: cpu_access,
..Default::default()
};
let mut tex: Option<ID3D11Texture2D> = None;
device
.CreateTexture2D(&desc, None, Some(&mut tex))
.context("CreateTexture2D(NV12)")?;
tex.context("CreateTexture2D returned a null NV12 texture")
}
+28 -5
View File
@@ -19,6 +19,13 @@ pub struct EncodedFrame {
pub pts_ns: u64, pub pts_ns: u64,
/// True for IDR/keyframes (sets the SOF/keyframe wire flags). /// True for IDR/keyframes (sets the SOF/keyframe wire flags).
pub keyframe: bool, pub keyframe: bool,
/// True when this AU is a **reference-frame-invalidation recovery frame** — a clean P-frame the
/// encoder coded against a known-good reference in response to
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) (AMD LTR force-reference). The pump
/// tags it [`punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its post-loss
/// freeze on it without an IDR. Only the native-AMF LTR path sets it; every other backend leaves
/// it `false` (their RFI, when present, re-references transparently with no distinct clean-point AU).
pub recovery_anchor: bool,
} }
/// Codec selection negotiated with the client. /// Codec selection negotiated with the client.
@@ -208,12 +215,28 @@ pub struct EncoderCaps {
/// the encoder's real chroma disagrees with what was negotiated (the in-band SPS is authoritative /// the encoder's real chroma disagrees with what was negotiated (the in-band SPS is authoritative
/// for the decoder either way). /// for the decoder either way).
pub chroma_444: bool, pub chroma_444: bool,
/// The encoder runs a periodic **intra-refresh wave** (a moving band of intra blocks + /// The encoder runs a periodic **intra-refresh wave** a moving band of intra blocks that
/// recovery-point SEI, no periodic IDR): FEC-unrecoverable loss self-heals within one wave, so /// re-codes the whole picture over ~0.5 s, no periodic IDR. FEC-unrecoverable loss self-heals as
/// the session glue rate-limits client keyframe requests instead of answering each with a full /// the band sweeps, so the session glue rate-limits client keyframe requests instead of answering
/// IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC sets it when /// each with a full IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC / AMF
/// `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/software never do. /// set it when `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/QSV/software never
/// do. NOTE — the wave carries NO decoder-visible clean-point: FFmpeg never sets `AV_FRAME_FLAG_KEY`
/// at a recovery point (H.264 flags key only when `recovery_frame_cnt == 0`; HEVC only on IRAP),
/// and AMF emits no recovery-point SEI at all. So this cap ALONE does not let the client lift its
/// post-loss freeze without an IDR — that needs [`intra_refresh_recovery`](Self::intra_refresh_recovery).
pub intra_refresh: bool, pub intra_refresh: bool,
/// The intra-refresh wave is a *validated constrained GDR* — verified on real hardware to fully
/// heal a lost picture within one wave period with no residual artifacts. Only then does the host
/// tag each wave-boundary AU with [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// so the client can lift its freeze on the second mark (a proven clean re-anchor) instead of
/// waiting out its backstop and forcing a full IDR. Default `false` on every backend until on-glass
/// validation flips it — an un-validated encoder keeps the IDR recovery path, so this is inert and
/// cannot regress. Meaningless unless [`intra_refresh`](Self::intra_refresh) is also set.
pub intra_refresh_recovery: bool,
/// Length of the intra-refresh wave in frames — the boundary period the host marks on (it sets
/// `USER_FLAG_RECOVERY_POINT` on every Nth emitted AU, re-phased at each IDR). 0 when intra-refresh
/// is off. Only consulted when [`intra_refresh_recovery`](Self::intra_refresh_recovery) is set.
pub intra_refresh_period: u32,
} }
/// A hardware encoder. One per session; runs on the encode thread. /// A hardware encoder. One per session; runs on the encode thread.
@@ -177,6 +177,10 @@ pub struct NvencEncoder {
/// Opened in intra-refresh mode (surfaced via [`caps`](Encoder::caps) so the session glue /// Opened in intra-refresh mode (surfaced via [`caps`](Encoder::caps) so the session glue
/// rate-limits forced IDRs — the wave heals loss without them). /// rate-limits forced IDRs — the wave heals loss without them).
intra_refresh: bool, intra_refresh: bool,
/// Resolved wave length in frames when [`intra_refresh`](Self::intra_refresh), else 0. Cached at
/// open so the pump's per-AU `caps()` doesn't re-read `PUNKTFUNK_IR_PERIOD_FRAMES`; the pump marks
/// every Nth AU with `USER_FLAG_RECOVERY_POINT` for the client's clean re-anchor.
intra_refresh_period: u32,
} }
// `CudaHw` holds raw `AVBufferRef`s and `sws_444` a raw `SwsContext`; the encoder lives on a single // `CudaHw` holds raw `AVBufferRef`s and `sws_444` a raw `SwsContext`; the encoder lives on a single
@@ -525,6 +529,11 @@ impl NvencEncoder {
frame_idx: 0, frame_idx: 0,
force_kf: false, force_kf: false,
intra_refresh, intra_refresh,
intra_refresh_period: if intra_refresh {
intra_refresh_period(fps).max(1) as u32
} else {
0
},
}) })
} }
} }
@@ -536,6 +545,12 @@ impl Encoder for NvencEncoder {
// convert. RFI/HDR-SEI stay unsupported on libavcodec NVENC (the trait defaults). // convert. RFI/HDR-SEI stay unsupported on libavcodec NVENC (the trait defaults).
chroma_444: self.want_444, chroma_444: self.want_444,
intra_refresh: self.intra_refresh, intra_refresh: self.intra_refresh,
// NVENC intra-refresh is purpose-built GDR loss recovery (moving band + recovery-point
// SEI): the wave heals a lost picture within one period, so mark the boundary AUs and let
// the client re-anchor on them instead of forcing a full IDR. Tied to `intra_refresh`
// (already the `PUNKTFUNK_INTRA_REFRESH` opt-in), unlike AMF/QSV which stay unvalidated.
intra_refresh_recovery: self.intra_refresh,
intra_refresh_period: self.intra_refresh_period,
..super::EncoderCaps::default() ..super::EncoderCaps::default()
} }
} }
@@ -578,6 +593,7 @@ impl Encoder for NvencEncoder {
data, data,
pts_ns, pts_ns,
keyframe: pkt.is_key(), keyframe: pkt.is_key(),
recovery_anchor: false,
})) }))
} }
// No packet ready yet (need another input frame). // No packet ready yet (need another input frame).
@@ -294,6 +294,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<Option<En
data, data,
pts_ns: pts * 1_000_000_000 / fps as u64, pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(), keyframe: pkt.is_key(),
recovery_anchor: false,
})) }))
} }
Err(ffmpeg::Error::Other { errno }) Err(ffmpeg::Error::Other { errno })
+1
View File
@@ -211,6 +211,7 @@ impl Encoder for OpenH264Encoder {
data, data,
pts_ns, pts_ns,
keyframe, keyframe,
recovery_anchor: false,
}); });
} }
self.frame_idx += 1; self.frame_idx += 1;
+323 -19
View File
@@ -644,6 +644,26 @@ struct CodecProps {
/// HEVC 64-px CTBs. `None` on AV1 (v1.4.36 exposes only a mode enum, no slot-size control — /// HEVC 64-px CTBs. `None` on AV1 (v1.4.36 exposes only a mode enum, no slot-size control —
/// loss recovery stays IDR there). /// loss recovery stays IDR there).
intra_refresh: Option<(PCWSTR, u32)>, intra_refresh: Option<(PCWSTR, u32)>,
/// LTR-RFI recovery property names (design: the AMD twin of NVENC intra-refresh recovery).
/// `None` on AV1 — its reference management uses a frame-marking OBU mechanism this path does
/// not drive, so LTR recovery is AVC/HEVC-only.
ltr: Option<LtrProps>,
}
/// The four AMF LTR (long-term-reference) property names, codec-prefixed (AVC bare, HEVC `Hevc*`).
/// Two are static (`max_*`, set once at open); two are per-frame (`mark`/`force`, set on the input
/// surface each `submit`). Together they let a loss re-reference a known-good older frame — a clean
/// P-frame instead of a 2040× IDR spike.
struct LtrProps {
/// `MaxOfLTRFrames` — number of user LTR slots (we request [`NUM_LTR_SLOTS`]).
max_ltr_frames: PCWSTR,
/// `MaxNumRefFrames` — reference-picture budget; must exceed 1 for LTR to engage.
max_num_ref_frames: PCWSTR,
/// `MarkCurrentWithLTRIndex` (per-frame) — tag the current frame as long-term reference slot N.
mark_ltr_index: PCWSTR,
/// `ForceLTRReferenceBitfield` (per-frame) — force the current frame to reference only the LTR
/// slots in the bitfield (`1<<N`), breaking the corrupted short-term chain after a loss.
force_ltr_bitfield: PCWSTR,
} }
/// The two payload shapes `lowlatency` takes across codecs. /// The two payload shapes `lowlatency` takes across codecs.
@@ -689,6 +709,12 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("OutColorPrimaries"), out_primaries: w!("OutColorPrimaries"),
hdr_metadata: None, hdr_metadata: None,
intra_refresh: Some((w!("IntraRefreshMBsNumberPerSlot"), 16)), intra_refresh: Some((w!("IntraRefreshMBsNumberPerSlot"), 16)),
ltr: Some(LtrProps {
max_ltr_frames: w!("MaxOfLTRFrames"),
max_num_ref_frames: w!("MaxNumRefFrames"),
mark_ltr_index: w!("MarkCurrentWithLTRIndex"),
force_ltr_bitfield: w!("ForceLTRReferenceBitfield"),
}),
}, },
Codec::H265 => CodecProps { Codec::H265 => CodecProps {
component: w!("AMFVideoEncoderHW_HEVC"), component: w!("AMFVideoEncoderHW_HEVC"),
@@ -716,6 +742,12 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("HevcOutColorPrimaries"), out_primaries: w!("HevcOutColorPrimaries"),
hdr_metadata: Some(w!("HevcInHDRMetadata")), hdr_metadata: Some(w!("HevcInHDRMetadata")),
intra_refresh: Some((w!("HevcIntraRefreshCTBsNumberPerSlot"), 64)), intra_refresh: Some((w!("HevcIntraRefreshCTBsNumberPerSlot"), 64)),
ltr: Some(LtrProps {
max_ltr_frames: w!("HevcMaxOfLTRFrames"),
max_num_ref_frames: w!("HevcMaxNumRefFrames"),
mark_ltr_index: w!("HevcMarkCurrentWithLTRIndex"),
force_ltr_bitfield: w!("HevcForceLTRReferenceBitfield"),
}),
}, },
Codec::Av1 => CodecProps { Codec::Av1 => CodecProps {
component: w!("AMFVideoEncoderHW_AV1"), component: w!("AMFVideoEncoderHW_AV1"),
@@ -743,6 +775,7 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("Av1OutputColorPrimaries"), out_primaries: w!("Av1OutputColorPrimaries"),
hdr_metadata: Some(w!("Av1InHDRMetadata")), hdr_metadata: Some(w!("Av1InHDRMetadata")),
intra_refresh: None, intra_refresh: None,
ltr: None,
}, },
} }
} }
@@ -797,6 +830,45 @@ fn intra_refresh_period(fps: u32) -> u32 {
.unwrap_or_else(|| (fps.max(16) / 2).max(2)) .unwrap_or_else(|| (fps.max(16) / 2).max(2))
} }
/// Number of user-controlled LTR slots. AMD exposes up to 2; two rotating slots hold a sliding pair
/// of recent long-term references, so a loss can re-reference the newest one *before* the loss point.
const NUM_LTR_SLOTS: usize = 2;
/// AMD's real clean loss-recovery path (the NVENC-RFI twin): the encoder marks frames as long-term
/// references, and on loss forces a later frame to re-reference a known-good one — a clean P-frame,
/// not a 20-40× IDR spike. On by default when the driver supports it (AMF intra-refresh cannot heal —
/// no constrained-intra-prediction property exists in the API, header-confirmed + PSNR-proven — and
/// LTR is mutually exclusive with it, so LTR wins). `PUNKTFUNK_NO_AMF_LTR=1` forces the old full-IDR
/// recovery for debugging.
fn ltr_disabled() -> bool {
std::env::var("PUNKTFUNK_NO_AMF_LTR")
.map(|v| matches!(v.trim(), "1" | "true" | "yes" | "on"))
.unwrap_or(false)
}
/// Cadence (frames) between LTR marks — a fresh long-term reference roughly every half second by
/// default (`PUNKTFUNK_LTR_INTERVAL_FRAMES` overrides). With [`NUM_LTR_SLOTS`] slots this keeps ~one
/// second of recent references, so a loss up to ~1 s old still has a known-good frame to force; a
/// smaller interval means the forced reference is more recent (a smaller recovery-frame residual).
fn ltr_mark_interval(fps: u32) -> i64 {
std::env::var("PUNKTFUNK_LTR_INTERVAL_FRAMES")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.filter(|v| *v >= 1)
.unwrap_or_else(|| (fps.max(2) / 2).max(1) as i64)
}
/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N` the encoder
/// self-triggers its real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path, so a
/// headless spike run can exercise LTR recovery end-to-end (mark → force → recovery-anchor tag)
/// without a live client sending an [`RfiRequest`](punktfunk_core::quic::RfiRequest). `None` normally.
fn ltr_test_force_at() -> Option<i64> {
std::env::var("PUNKTFUNK_LTR_FORCE_AT")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.filter(|v| *v > 0)
}
// --------------------------------------------------------------------------------------------- // ---------------------------------------------------------------------------------------------
// Owned-pointer guards (release exactly once; Terminate before Release for context/component, // Owned-pointer guards (release exactly once; Terminate before Release for context/component,
// mirroring amfenc.c's teardown order). // mirroring amfenc.c's teardown order).
@@ -930,11 +1002,12 @@ struct Inner {
dctx: ID3D11DeviceContext, dctx: ID3D11DeviceContext,
ring: Vec<ID3D11Texture2D>, ring: Vec<ID3D11Texture2D>,
next: usize, next: usize,
/// (pts_ns, forced-IDR) per submitted-but-unretrieved frame, FIFO — the AMF encoder emits /// (pts_ns, forced-IDR, recovery-anchor) per submitted-but-unretrieved frame, FIFO — the AMF
/// AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`. Its length is /// encoder emits AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`.
/// the count of input surfaces AMF still holds, so `submit` bounds it below [`RING`] to keep /// The third field tags the LTR-RFI re-anchor frame so the AU carries `recovery_anchor` for the
/// the input ring from being overwritten under it. /// client's freeze-lift. Its length is the count of input surfaces AMF still holds, so `submit`
pending: VecDeque<(u64, bool)>, /// bounds it below [`RING`] to keep the input ring from being overwritten under it.
pending: VecDeque<(u64, bool, bool)>,
/// AUs already pulled by `submit`'s backpressure drain, waiting to be handed out by `poll` /// AUs already pulled by `submit`'s backpressure drain, waiting to be handed out by `poll`
/// (FIFO, strictly older than anything still in `pending`). Empty in the steady state — only /// (FIFO, strictly older than anything still in `pending`). Empty in the steady state — only
/// fills when the encoder falls behind and `submit` drains to free an input slot. /// fills when the encoder falls behind and `submit` drains to free an input slot.
@@ -988,6 +1061,26 @@ pub struct AmfEncoder {
/// gates [`EncoderCaps::intra_refresh`] so keyframe-request rate-limiting only happens when /// gates [`EncoderCaps::intra_refresh`] so keyframe-request rate-limiting only happens when
/// the wave really runs. /// the wave really runs.
ir_active: bool, ir_active: bool,
// --- Long-Term-Reference reference-frame-invalidation recovery (the AMD RFI path) ---
/// The driver accepted the LTR properties at open — gates [`EncoderCaps::supports_rfi`] and all
/// the per-frame LTR marking/forcing below. When true, intra-refresh is NOT set (mutually
/// exclusive) and loss recovery re-references a known-good LTR instead of forcing a full IDR.
ltr_active: bool,
/// The `frame_idx` currently stored in each of the two LTR slots (`None` = never marked). On loss
/// the newest slot with an index *before* the loss is the known-good reference to force.
ltr_slots: [Option<i64>; NUM_LTR_SLOTS],
/// The slot the next LTR mark writes (round-robins `0,1,0,1,…` so the two slots hold a sliding
/// pair of recent references).
next_ltr_slot: usize,
/// Cadence (frames) between LTR marks — a fresh long-term reference roughly this often.
ltr_mark_interval: i64,
/// Set by [`invalidate_ref_frames`](Encoder::invalidate_ref_frames): the LTR slot the *next*
/// submitted frame must force-reference (`ForceLTRReferenceBitfield`). Consumed on that submit.
pending_force: Option<usize>,
/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N`, self-trigger the
/// real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path so a headless spike run can
/// exercise LTR recovery end-to-end without a live client. `None` in normal operation.
ltr_test_force_at: Option<i64>,
/// Consecutive [`reset`](Self::reset)s that have NOT been followed by a produced AU (cleared in /// Consecutive [`reset`](Self::reset)s that have NOT been followed by a produced AU (cleared in
/// `poll` on any output). An in-place `Terminate`+re-`Init` heals a transient component stall, /// `poll` on any output). An in-place `Terminate`+re-`Init` heals a transient component stall,
/// but it re-inits the SAME context — so if the fault is the context / VCN session itself (the /// but it re-inits the SAME context — so if the fault is the context / VCN session itself (the
@@ -1084,17 +1177,33 @@ impl AmfEncoder {
force_kf: false, force_kf: false,
hdr_meta: None, hdr_meta: None,
ir_active: false, ir_active: false,
ltr_active: false,
ltr_slots: [None; NUM_LTR_SLOTS],
next_ltr_slot: 0,
ltr_mark_interval: ltr_mark_interval(fps),
pending_force: None,
ltr_test_force_at: ltr_test_force_at(),
resets_without_output: 0, resets_without_output: 0,
}) })
} }
/// Whether this encoder should *attempt* the LTR-RFI recovery path (design: the AMD twin of
/// NVENC intra-refresh recovery). Gated to AVC/HEVC — AMF exposes user LTR only for those two
/// codecs — and defeatable via `PUNKTFUNK_NO_AMF_LTR`. Whether the driver actually *accepts* the
/// properties is a separate question answered by [`apply_static_props`], which sets `ltr_active`.
fn ltr_wanted(&self) -> bool {
!ltr_disabled() && matches!(self.codec, Codec::H264 | Codec::H265)
}
/// Apply the static encoder configuration (design §3.4 — the native mirror of the ffmpeg /// Apply the static encoder configuration (design §3.4 — the native mirror of the ffmpeg
/// opts block in `open_win_encoder`). Called before `Init`, and again on a `reset()` /// opts block in `open_win_encoder`). Called before `Init`, and again on a `reset()`
/// re-`Init` (Terminate does not guarantee property retention across every driver). /// re-`Init` (Terminate does not guarantee property retention across every driver).
/// Returns whether the intra-refresh wave was requested AND accepted by this driver — the /// Returns `(ir_active, ltr_active)`: whether the intra-refresh wave / the LTR-RFI slots were
/// caller stores it so [`Encoder::caps`] only rate-limits keyframe requests when the wave /// requested AND accepted by this driver. The two are mutually exclusive (LTR wins when both are
/// really runs. /// wanted). The caller stores both — `ir_active` so [`Encoder::caps`] only rate-limits keyframe
unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<bool> { /// requests when a wave runs, `ltr_active` so [`Encoder::caps`] advertises `supports_rfi` and the
/// per-frame mark/force logic in `submit` only fires when the slots exist.
unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<(bool, bool)> {
let p = &self.props; let p = &self.props;
// Usage first: it "fully configures parameter set" — everything after is an override. // Usage first: it "fully configures parameter set" — everything after is an override.
set_prop( set_prop(
@@ -1145,7 +1254,39 @@ impl AmfEncoder {
// whole picture refreshes every `period` frames — per-slot units = ceil(total blocks / // whole picture refreshes every `period` frames — per-slot units = ceil(total blocks /
// period). Optional by VCN generation; the return value gates `caps().intra_refresh`. // period). Optional by VCN generation; the return value gates `caps().intra_refresh`.
let mut ir_active = false; let mut ir_active = false;
if let Some((name, block)) = p.intra_refresh { let mut ltr_active = false;
if let Some(ltr) = p.ltr.as_ref().filter(|_| self.ltr_wanted()) {
// LTR-RFI recovery (design: the AMD twin of NVENC intra-refresh recovery). Request
// NUM_LTR_SLOTS user-controlled long-term references. LTR needs >1 reference frames and
// is MUTUALLY EXCLUSIVE with intra-refresh (AMF disables one if both are set), so the
// intra-refresh block below is skipped whenever LTR engages.
let ref_ok = set_prop(
comp,
ltr.max_num_ref_frames,
AmfVariant::from_i64(NUM_LTR_SLOTS as i64),
false,
)?;
let ltr_ok = set_prop(
comp,
ltr.max_ltr_frames,
AmfVariant::from_i64(NUM_LTR_SLOTS as i64),
false,
)?;
ltr_active = ref_ok && ltr_ok;
if ltr_active {
tracing::info!(
slots = NUM_LTR_SLOTS,
mark_interval = self.ltr_mark_interval,
"AMF LTR-RFI recovery enabled (loss recovery re-references a known-good LTR, not a full IDR)"
);
} else {
tracing::warn!(
ref_ok,
ltr_ok,
"this VCN/driver rejected an LTR property — loss recovery stays full-IDR"
);
}
} else if let Some((name, block)) = p.intra_refresh {
if intra_refresh_requested() { if intra_refresh_requested() {
let period = intra_refresh_period(self.fps); let period = intra_refresh_period(self.fps);
let blocks = self.width.div_ceil(block) * self.height.div_ceil(block); let blocks = self.width.div_ceil(block) * self.height.div_ceil(block);
@@ -1273,7 +1414,7 @@ impl AmfEncoder {
AmfVariant::from_i64(primaries), AmfVariant::from_i64(primaries),
self.ten_bit, self.ten_bit,
)?; )?;
Ok(ir_active) Ok((ir_active, ltr_active))
} }
/// Build (or rebuild, on a capture-device change) the AMF context + encoder component on the /// Build (or rebuild, on a capture-device change) the AMF context + encoder component on the
@@ -1323,7 +1464,7 @@ impl AmfEncoder {
bail!("AMF CreateComponent returned null"); bail!("AMF CreateComponent returned null");
} }
let comp = Component(comp); let comp = Component(comp);
let ir_active = self.apply_static_props(comp.0)?; let (ir_active, ltr_active) = self.apply_static_props(comp.0)?;
let fmt = if self.ten_bit { let fmt = if self.ten_bit {
sys::AMF_SURFACE_P010 sys::AMF_SURFACE_P010
} else { } else {
@@ -1334,6 +1475,14 @@ impl AmfEncoder {
"AMF encoder Init", "AMF encoder Init",
)?; )?;
self.ir_active = ir_active; self.ir_active = ir_active;
// A rebuilt component starts with fresh (empty) LTR slots — a new context has no
// reference history, so any prior marks are void and the first frame re-IDRs anyway.
self.ltr_active = ltr_active;
if ltr_active {
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
}
// Owned input ring on the capturer's device (design §3.2): RENDER_TARGET | // Owned input ring on the capturer's device (design §3.2): RENDER_TARGET |
// SHADER_RESOURCE, the same bind flags the validated ffmpeg zero-copy pool uses. // SHADER_RESOURCE, the same bind flags the validated ffmpeg zero-copy pool uses.
@@ -1594,7 +1743,7 @@ enum DrainOutcome {
/// single encode thread with no other AMF call to this component in flight. /// single encode thread with no other AMF call to this component in flight.
unsafe fn drain_one_output( unsafe fn drain_one_output(
comp: *mut sys::AmfComponent, comp: *mut sys::AmfComponent,
pending: &mut VecDeque<(u64, bool)>, pending: &mut VecDeque<(u64, bool, bool)>,
output_data_type: PCWSTR, output_data_type: PCWSTR,
output_key_max: i64, output_key_max: i64,
) -> Result<DrainOutcome> { ) -> Result<DrainOutcome> {
@@ -1641,11 +1790,12 @@ unsafe fn drain_one_output(
bail!("AMF output buffer is empty"); bail!("AMF output buffer is empty");
} }
let au = std::slice::from_raw_parts(native as *const u8, size).to_vec(); let au = std::slice::from_raw_parts(native as *const u8, size).to_vec();
let (pts_ns, forced) = pending.pop_front().unwrap_or((0, false)); let (pts_ns, forced, recovery_anchor) = pending.pop_front().unwrap_or((0, false, false));
Ok(DrainOutcome::Frame(EncodedFrame { Ok(DrainOutcome::Frame(EncodedFrame {
data: au, data: au,
pts_ns, pts_ns,
keyframe: key_prop || forced, keyframe: key_prop || forced,
recovery_anchor,
})) }))
} }
@@ -1689,9 +1839,57 @@ impl Encoder for AmfEncoder {
expected expected
); );
self.ensure_inner(&frame.device)?; self.ensure_inner(&frame.device)?;
let cur_idx = self.frame_idx;
let forced = std::mem::take(&mut self.force_kf) || self.frame_idx == 0; let forced = std::mem::take(&mut self.force_kf) || self.frame_idx == 0;
let pts_100ns = self.frame_idx * 10_000_000 / self.fps.max(1) as i64; let pts_100ns = self.frame_idx * 10_000_000 / self.fps.max(1) as i64;
self.frame_idx += 1; self.frame_idx += 1;
// --- LTR-RFI per-frame decisions (design: the AMD twin of NVENC intra-refresh recovery) ---
// Decided here, before borrowing `inner`, because the test hook re-enters `&mut self`
// (`invalidate_ref_frames`) and the mark cadence mutates the slot bookkeeping. The two
// per-frame property names are copied out (PCWSTR is Copy) so the unsafe surface block can
// set them without re-borrowing `self.props` under the live `inner` borrow.
let ltr_names = self
.props
.ltr
.as_ref()
.map(|l| (l.mark_ltr_index, l.force_ltr_bitfield));
let mut mark_slot: Option<usize> = None;
let mut force_slot: Option<usize> = None;
let mut recovery_anchor = false;
if self.ltr_active {
if forced {
// An IDR resets the decoder's reference buffers — every prior LTR mark is void.
// Re-anchor from scratch: drop the stale slots (the mark cadence below tags the IDR
// as the first fresh long-term reference) and cancel any force queued against them.
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
} else if self.ltr_test_force_at == Some(cur_idx) {
// Spike-only validation hook: self-trigger the real invalidate path so a headless
// run exercises mark → force → recovery-anchor without a live client's RfiRequest.
let triggered = self.invalidate_ref_frames(cur_idx, cur_idx);
tracing::info!(
frame = cur_idx,
triggered,
"AMF LTR test hook fired invalidate_ref_frames"
);
}
// Apply a queued force (from invalidate_ref_frames / the test hook) to THIS frame: it
// becomes the clean re-anchor P-frame the client lifts its post-loss freeze on.
if let Some(slot) = self.pending_force.take() {
force_slot = Some(slot);
recovery_anchor = true;
}
// Mark cadence: refresh a long-term reference on every IDR and every `ltr_mark_interval`
// frames — but never on the recovery frame itself (marking rotates `next_ltr_slot` and
// could overwrite the very slot being forced; the next cadence mark re-establishes it).
if force_slot.is_none() && (forced || cur_idx % self.ltr_mark_interval == 0) {
let slot = self.next_ltr_slot;
self.ltr_slots[slot] = Some(cur_idx);
self.next_ltr_slot = (self.next_ltr_slot + 1) % NUM_LTR_SLOTS;
mark_slot = Some(slot);
}
}
let inner = self.inner.as_mut().expect("ensure_inner succeeded"); let inner = self.inner.as_mut().expect("ensure_inner succeeded");
// Push the HDR mastering metadata when it changed (or a rebuilt component lost it) — a // Push the HDR mastering metadata when it changed (or a rebuilt component lost it) — a
// dynamic property, so mid-stream regrades take effect on the next IDR. Best-effort: a // dynamic property, so mid-stream regrades take effect on the next IDR. Best-effort: a
@@ -1831,6 +2029,47 @@ impl Encoder for AmfEncoder {
Codec::Av1 => {} Codec::Av1 => {}
} }
} }
// LTR-RFI per-frame properties (design: the AMD twin of NVENC intra-refresh recovery).
// `mark_slot`/`force_slot` were decided above. Marking tags the current frame as a
// long-term reference; forcing makes it re-reference a known-good LTR — a clean P-frame
// that breaks the corrupted short-term chain after a loss, no 20-40× IDR. Best-effort:
// a rejecting driver just leaves the client on its keyframe-request fallback.
if let Some((mark_name, force_name)) = ltr_names {
if let Some(slot) = mark_slot {
let r = ((*(*surf.0).vtbl).set_property)(
surf.0,
mark_name.0,
AmfVariant::from_i64(slot as i64),
);
if r != sys::AMF_OK {
tracing::warn!(
slot,
result = %format!("{} ({r})", result_name(r)),
"AMF LTR mark rejected"
);
}
}
if let Some(slot) = force_slot {
let r = ((*(*surf.0).vtbl).set_property)(
surf.0,
force_name.0,
AmfVariant::from_i64(1_i64 << slot),
);
if r == sys::AMF_OK {
tracing::info!(
slot,
frame = cur_idx,
"AMF LTR-RFI: re-referencing known-good LTR (clean recovery, no IDR)"
);
} else {
tracing::warn!(
slot,
result = %format!("{} ({r})", result_name(r)),
"AMF LTR force-reference rejected — client stays frozen until its IDR fallback"
);
}
}
}
let mut r = ((*(*inner.comp.0).vtbl).submit_input)(inner.comp.0, surf.0); let mut r = ((*(*inner.comp.0).vtbl).submit_input)(inner.comp.0, surf.0);
// Backstop back-pressure: the in-flight bound above already keeps a slot free, but if // Backstop back-pressure: the in-flight bound above already keeps a slot free, but if
// AMF's own input queue is momentarily full, AMF_INPUT_FULL is "busy, drain me and // AMF's own input queue is momentarily full, AMF_INPUT_FULL is "busy, drain me and
@@ -1873,7 +2112,7 @@ impl Encoder for AmfEncoder {
} }
} }
} }
inner.pending.push_back((captured.pts_ns, forced)); inner.pending.push_back((captured.pts_ns, forced, recovery_anchor));
Ok(()) Ok(())
} }
@@ -1887,11 +2126,63 @@ impl Encoder for AmfEncoder {
self.hdr_meta = meta; self.hdr_meta = meta;
} }
/// LTR-RFI recovery (the AMD twin of the Windows NVENC `nvEncInvalidateRefFrames` path): a loss
/// of client frames `[first, last]` is answered by forcing the *next* submitted frame to
/// re-reference the newest long-term reference marked *before* the loss — a clean P-frame the
/// client can decode against a picture it still holds, instead of a 20-40× IDR spike.
///
/// Returns `true` when a usable pre-loss LTR exists (so the caller must NOT also force an IDR);
/// `false` when the loss predates every live LTR — then the only correct recovery is a keyframe,
/// and the caller falls back to [`request_keyframe`](Self::request_keyframe). Runs on the encode
/// thread (like submit/poll); the force is applied on the next `submit`.
fn invalidate_ref_frames(&mut self, first: i64, last: i64) -> bool {
// No live LTR session (driver declined the slots, or AV1 which has no user-LTR path) or a
// nonsense range → caller forces a full IDR.
if !self.ltr_active || first < 0 || first > last {
return false;
}
// Pick the newest LTR strictly OLDER than the loss: the most recent known-good reference the
// client still holds, so re-referencing it costs the least (smallest recovery-frame residual).
// Frame numbers are 1:1 with the client's (both count submissions in order — see the NVENC
// path), so `ltr_slots` (which store `frame_idx`) compare directly against `first`.
let mut best: Option<(usize, i64)> = None;
for (slot, marked) in self.ltr_slots.iter().enumerate() {
if let Some(idx) = *marked {
if idx < first && best.is_none_or(|(_, b)| idx > b) {
best = Some((slot, idx));
}
}
}
match best {
Some((slot, ltr_frame)) => {
// Queue the force for the next submit; that frame ships tagged `recovery_anchor`.
self.pending_force = Some(slot);
tracing::info!(
first,
last,
slot,
ltr_frame,
"AMF LTR-RFI: forcing the next frame to re-reference a known-good LTR (no IDR)"
);
true
}
None => {
tracing::info!(
first,
last,
"AMF LTR-RFI: no live LTR older than the loss — falling back to IDR recovery"
);
false
}
}
}
fn caps(&self) -> EncoderCaps { fn caps(&self) -> EncoderCaps {
EncoderCaps { EncoderCaps {
// AMF has no NVENC-style reference invalidation the intra-refresh wave is the // LTR-RFI: AMD's reference invalidation is the user long-term-reference path (mark a
// loss-recovery substitute; without it every unrecoverable loss costs an IDR. // frame, force a later one to re-reference it). True only when the live driver accepted
supports_rfi: false, // the LTR slots at open — otherwise loss recovery falls back to a full IDR.
supports_rfi: self.ltr_active,
// In-band mastering/CLL via `*InHDRMetadata` (HEVC SEI / AV1 metadata OBU); AVC has // In-band mastering/CLL via `*InHDRMetadata` (HEVC SEI / AV1 metadata OBU); AVC has
// no such property (and no HDR sessions negotiate H.264). // no such property (and no HDR sessions negotiate H.264).
supports_hdr_metadata: self.ten_bit && self.props.hdr_metadata.is_some(), supports_hdr_metadata: self.ten_bit && self.props.hdr_metadata.is_some(),
@@ -1901,6 +2192,11 @@ impl Encoder for AmfEncoder {
// accepted the property (queried per loss event, so the post-first-frame value is // accepted the property (queried per loss event, so the post-first-frame value is
// what the session glue's IDR rate-limiting sees). // what the session glue's IDR rate-limiting sees).
intra_refresh: self.ir_active, intra_refresh: self.ir_active,
// Not yet: the AMD VCN wave heals in principle, but its constrained-GDR
// heal-within-a-period is unvalidated on-glass and AMF emits no recovery-point SEI, so
// the host keeps the IDR recovery path. Flip both once verified on real hardware.
intra_refresh_recovery: false,
intra_refresh_period: 0,
} }
} }
@@ -1992,6 +2288,7 @@ impl Encoder for AmfEncoder {
self.inner = None; self.inner = None;
self.bound_device = 0; self.bound_device = 0;
self.ir_active = false; self.ir_active = false;
self.ltr_active = false;
return true; return true;
} }
let inner = self let inner = self
@@ -2016,8 +2313,14 @@ impl Encoder for AmfEncoder {
sys::AMF_SURFACE_NV12 sys::AMF_SURFACE_NV12
}; };
match self.apply_static_props(comp) { match self.apply_static_props(comp) {
Ok(ir) => { Ok((ir, ltr)) => {
self.ir_active = ir; self.ir_active = ir;
// Re-Init voids the reference history: the rebuilt stream restarts at IDR with
// empty LTR slots, so any prior marks are stale and must be dropped.
self.ltr_active = ltr;
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
((*(*comp).vtbl).init)(comp, fmt, self.width as i32, self.height as i32) ((*(*comp).vtbl).init)(comp, fmt, self.width as i32, self.height as i32)
== sys::AMF_OK == sys::AMF_OK
} }
@@ -2030,6 +2333,7 @@ impl Encoder for AmfEncoder {
); );
} else { } else {
self.ir_active = false; self.ir_active = false;
self.ltr_active = false;
// Full teardown; the next submit reopens context + component on the current device. // Full teardown; the next submit reopens context + component on the current device.
tracing::warn!("AMF in-place re-Init failed — full context teardown, reopening lazily"); tracing::warn!("AMF in-place re-Init failed — full context teardown, reopening lazily");
self.inner = None; self.inner = None;
@@ -339,6 +339,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<PollOutco
data, data,
pts_ns: pts * 1_000_000_000 / fps as u64, pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(), keyframe: pkt.is_key(),
recovery_anchor: false,
})) }))
} }
Err(ffmpeg::Error::Other { errno }) Err(ffmpeg::Error::Other { errno })
@@ -1157,6 +1157,7 @@ impl NvencD3d11Encoder {
data, data,
pts_ns, pts_ns,
keyframe, keyframe,
recovery_anchor: false,
}); });
Ok(()) Ok(())
} }
@@ -1424,6 +1425,8 @@ impl Encoder for NvencD3d11Encoder {
// The direct-NVENC path recovers via real RFI (or a forced IDR), not the Linux // The direct-NVENC path recovers via real RFI (or a forced IDR), not the Linux
// libavcodec intra-refresh mode. // libavcodec intra-refresh mode.
intra_refresh: false, intra_refresh: false,
intra_refresh_recovery: false,
intra_refresh_period: 0,
} }
} }
@@ -1542,6 +1545,7 @@ impl Encoder for NvencD3d11Encoder {
data, data,
pts_ns, pts_ns,
keyframe, keyframe,
recovery_anchor: false,
})) }))
} }
} }
+5 -1
View File
@@ -696,10 +696,14 @@ fn parse_spike(args: &[String]) -> Result<Options> {
"--source" => { "--source" => {
source = match next()?.as_str() { source = match next()?.as_str() {
"synthetic" => Source::Synthetic, "synthetic" => Source::Synthetic,
"synthetic-nv12" => Source::SyntheticNv12,
"portal" => Source::Portal, "portal" => Source::Portal,
"kwin-virtual" => Source::KwinVirtual, "kwin-virtual" => Source::KwinVirtual,
other => { other => {
bail!("unknown --source '{other}' (synthetic|portal|kwin-virtual)") bail!(
"unknown --source '{other}' \
(synthetic|synthetic-nv12|portal|kwin-virtual)"
)
} }
} }
} }
+4
View File
@@ -26,12 +26,16 @@ pub fn pump_once(
data, data,
pts_ns, pts_ns,
keyframe, keyframe,
recovery_anchor,
}) = encoder.poll()? }) = encoder.poll()?
{ {
let mut flags = FLAG_PIC as u32; let mut flags = FLAG_PIC as u32;
if keyframe { if keyframe {
flags |= FLAG_SOF as u32; flags |= FLAG_SOF as u32;
} }
if recovery_anchor {
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR;
}
// core does FEC + packetize + pace + send. // core does FEC + packetize + pace + send.
session.submit_frame(&data, pts_ns, flags)?; session.submit_frame(&data, pts_ns, flags)?;
} }
+122 -2
View File
@@ -34,7 +34,7 @@ use punktfunk_core::packet::{FLAG_PIC, FLAG_PROBE, FLAG_SOF};
use punktfunk_core::quic::{ use punktfunk_core::quic::{
endpoint, io, BitrateChanged, ClockEcho, ClockProbe, ColorInfo, Hello, LossReport, endpoint, io, BitrateChanged, ClockEcho, ClockProbe, ColorInfo, Hello, LossReport,
PairChallenge, PairProof, PairRequest, PairResult, ProbeRequest, ProbeResult, Reconfigure, PairChallenge, PairProof, PairRequest, PairResult, ProbeRequest, ProbeResult, Reconfigure,
Reconfigured, RequestKeyframe, SetBitrate, Start, Welcome, Reconfigured, RequestKeyframe, RfiRequest, SetBitrate, Start, Welcome,
}; };
use punktfunk_core::transport::UdpTransport; use punktfunk_core::transport::UdpTransport;
use punktfunk_core::Session; use punktfunk_core::Session;
@@ -1124,6 +1124,10 @@ async fn serve_session(
// (inbound requests, outbound probe results) are multiplexed with `select!`. // (inbound requests, outbound probe results) are multiplexed with `select!`.
let (reconfig_tx, reconfig_rx) = std::sync::mpsc::channel::<punktfunk_core::Mode>(); let (reconfig_tx, reconfig_rx) = std::sync::mpsc::channel::<punktfunk_core::Mode>();
let (keyframe_tx, keyframe_rx) = std::sync::mpsc::channel::<()>(); let (keyframe_tx, keyframe_rx) = std::sync::mpsc::channel::<()>();
// Client LTR-RFI recovery: the control task forwards each `RfiRequest`'s lost-frame range here;
// the encode loop prefers `Encoder::invalidate_ref_frames` (a clean re-anchor P-frame) over a
// full IDR when the encoder supports it (native-AMF LTR / Windows NVENC).
let (rfi_tx, rfi_rx) = std::sync::mpsc::channel::<(u32, u32)>();
let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::<u32>(); let (bitrate_tx, bitrate_rx) = std::sync::mpsc::channel::<u32>();
let (probe_tx, probe_rx) = std::sync::mpsc::channel::<ProbeRequest>(); let (probe_tx, probe_rx) = std::sync::mpsc::channel::<ProbeRequest>();
let (probe_result_tx, mut probe_result_rx) = let (probe_result_tx, mut probe_result_rx) =
@@ -1199,6 +1203,19 @@ async fn serve_session(
if keyframe_tx.send(()).is_err() { if keyframe_tx.send(()).is_err() {
break; // data plane gone break; // data plane gone
} }
} else if let Ok(req) = RfiRequest::decode(&msg) {
// Client LTR-RFI recovery: it lost the frame range `[first, last]` and asks
// the encoder to re-reference a known-good older frame instead of paying for
// a full IDR. The encode loop attempts `invalidate_ref_frames`, falling back
// to a coalesced keyframe when the encoder can't (range too old / no RFI).
tracing::debug!(
first = req.first_frame,
last = req.last_frame,
"client requested reference-frame invalidation (loss recovery)"
);
if rfi_tx.send((req.first_frame, req.last_frame)).is_err() {
break; // data plane gone
}
} else if let Ok(rep) = LossReport::decode(&msg) { } else if let Ok(rep) = LossReport::decode(&msg) {
// Adaptive FEC: size recovery to the loss the client is seeing. The data-plane // Adaptive FEC: size recovery to the loss the client is seeing. The data-plane
// send loop reads `fec_target_ctl` and applies it per frame. Ignored when FEC // send loop reads `fec_target_ctl` and applies it per frame. Ignored when FEC
@@ -1590,6 +1607,7 @@ async fn serve_session(
quit: quit_stream, quit: quit_stream,
reconfig: reconfig_rx, reconfig: reconfig_rx,
keyframe: keyframe_rx, keyframe: keyframe_rx,
rfi: rfi_rx,
bitrate_rx, bitrate_rx,
compositor, compositor,
bitrate_kbps, bitrate_kbps,
@@ -2396,6 +2414,29 @@ fn audio_thread(
tracing::warn!("punktfunk/1 audio requires Linux or Windows — session continues without it"); tracing::warn!("punktfunk/1 audio requires Linux or Windows — session continues without it");
} }
/// Advance the intra-refresh wave position and decide whether this emitted AU is a wave boundary
/// that should carry [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT).
///
/// `ir_wave_pos` counts frames since the last IDR/wave start; a real IDR re-phases it to 0 (an IDR
/// restarts the encoder's wave AND is itself a clean anchor, so it is never additionally marked).
/// Every `period`-th non-IDR AU is a boundary — the client lifts its post-loss freeze on the SECOND
/// such mark. Pure so the marking cadence is unit-tested without a GPU (see the pump's use in the
/// encode-poll loop).
fn mark_recovery_boundary(ir_wave_pos: &mut u32, is_keyframe: bool, period: u32) -> bool {
if is_keyframe {
*ir_wave_pos = 0;
false
} else {
*ir_wave_pos += 1;
if *ir_wave_pos >= period {
*ir_wave_pos = 0;
true
} else {
false
}
}
}
fn synthetic_stream( fn synthetic_stream(
session: &mut Session, session: &mut Session,
frames: u32, frames: u32,
@@ -3396,6 +3437,9 @@ struct SessionContext {
reconfig: std::sync::mpsc::Receiver<punktfunk_core::Mode>, reconfig: std::sync::mpsc::Receiver<punktfunk_core::Mode>,
/// Client decode-recovery keyframe requests. /// Client decode-recovery keyframe requests.
keyframe: std::sync::mpsc::Receiver<()>, keyframe: std::sync::mpsc::Receiver<()>,
/// Client LTR-RFI recovery requests — the lost-frame range `(first, last)`. The encode loop
/// prefers `Encoder::invalidate_ref_frames` over a full IDR when the encoder supports it.
rfi: std::sync::mpsc::Receiver<(u32, u32)>,
/// Accepted mid-stream bitrate changes (adaptive bitrate, already clamped) — the encoder /// Accepted mid-stream bitrate changes (adaptive bitrate, already clamped) — the encoder
/// alone is rebuilt in place at the new rate; capture + virtual output are untouched. /// alone is rebuilt in place at the new rate; capture + virtual output are untouched.
bitrate_rx: std::sync::mpsc::Receiver<u32>, bitrate_rx: std::sync::mpsc::Receiver<u32>,
@@ -3467,6 +3511,7 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
quit, quit,
reconfig, reconfig,
keyframe, keyframe,
rfi,
bitrate_rx, bitrate_rx,
compositor, compositor,
mut bitrate_kbps, mut bitrate_kbps,
@@ -3684,6 +3729,11 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
// Self-diagnosis for the periodic-stutter class: warns when the served recovery IDRs settle // Self-diagnosis for the periodic-stutter class: warns when the served recovery IDRs settle
// into a stable multi-second rhythm (see [`crate::metronome::Metronome`]). // into a stable multi-second rhythm (see [`crate::metronome::Metronome`]).
let mut recovery_cadence = crate::metronome::Metronome::new(); let mut recovery_cadence = crate::metronome::Metronome::new();
// Position within the current intra-refresh wave (frames since the last IDR/wave start). Only
// meaningful on a `caps().intra_refresh_recovery` encoder; the pump tags every wave-boundary AU
// with `USER_FLAG_RECOVERY_POINT` so the client can lift its post-loss freeze on a clean
// re-anchor without a full IDR. Re-phased to 0 at each emitted IDR (which restarts the wave).
let mut ir_wave_pos: u32 = 0;
// Per-stage latency breakdown (PUNKTFUNK_PERF): per-call µs for the GPU-bound stages so we see // Per-stage latency breakdown (PUNKTFUNK_PERF): per-call µs for the GPU-bound stages so we see
// exactly where the capture→encoded latency goes — cap=try_latest (ring read + colour convert), // exactly where the capture→encoded latency goes — cap=try_latest (ring read + colour convert),
// submit=encode_picture launch, wait=lock_bitstream (the scheduling wait + ASIC encode, the one // submit=encode_picture launch, wait=lock_bitstream (the scheduling wait + ASIC encode, the one
@@ -3900,6 +3950,33 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
while keyframe.try_recv().is_ok() { while keyframe.try_recv().is_ok() {
want_kf = true; want_kf = true;
} }
// Client LTR-RFI recovery: prefer re-referencing a known-good older frame (a clean recovery
// P-frame — no 20-40× IDR spike) over a full keyframe when the encoder supports it (native
// AMF LTR / Windows NVENC). Drain the backlog (the client re-requests until the recovery
// frame lands) coalesced to the widest lost range. Attempt the invalidate only when a full
// IDR isn't already queued — an explicit keyframe request means a fully wedged decoder that
// needs the IDR, which supersedes an RFI recovery. A failure (range older than the encoder's
// live references, or no RFI backend) falls through to the coalesced keyframe path below.
let mut rfi_range: Option<(u32, u32)> = None;
while let Ok((first, last)) = rfi.try_recv() {
rfi_range = Some(match rfi_range {
Some((pf, pl)) => (pf.min(first), pl.max(last)),
None => (first, last),
});
}
if !want_kf {
if let Some((first, last)) = rfi_range {
if enc.caps().supports_rfi && enc.invalidate_ref_frames(first as i64, last as i64) {
// The RFI recovered the loss with a clean re-anchor P-frame (no IDR). Anchor the
// keyframe cooldown so the client's echo of the SAME loss — its frames_dropped-
// driven keyframe request, arriving ~one loss-window later — is coalesced away
// instead of emitting a redundant full IDR right after the cheap recovery.
last_forced_idr = Some(std::time::Instant::now());
} else {
want_kf = true; // range too old / no RFI backend → coalesced keyframe below
}
}
}
if want_kf { if want_kf {
// Clients request a keyframe on EVERY FEC-unrecoverable frame (`frames_dropped` polling) // Clients request a keyframe on EVERY FEC-unrecoverable frame (`frames_dropped` polling)
// and keep asking until the IDR actually arrives + decodes — a full round-trip on a link // and keep asking until the IDR actually arrives + decodes — a full round-trip on a link
@@ -4225,11 +4302,28 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
last_au_at = std::time::Instant::now(); last_au_at = std::time::Instant::now();
encoder_resets = 0; encoder_resets = 0;
let (cap_ns, sub_ns, deadline) = inflight.pop_front().expect("inflight non-empty"); let (cap_ns, sub_ns, deadline) = inflight.pop_front().expect("inflight non-empty");
let flags = if au.keyframe { let mut flags = if au.keyframe {
(FLAG_PIC | FLAG_SOF) as u32 (FLAG_PIC | FLAG_SOF) as u32
} else { } else {
FLAG_PIC as u32 FLAG_PIC as u32
}; };
// Intra-refresh recovery marking (inert unless the backend validated its constrained GDR
// via `intra_refresh_recovery`): tag every wave-boundary AU with USER_FLAG_RECOVERY_POINT
// so the client lifts its post-loss freeze on the second mark — a proven clean re-anchor —
// instead of forcing a full IDR. See [`mark_recovery_boundary`] for the cadence.
let caps = enc.caps();
if caps.intra_refresh_recovery
&& caps.intra_refresh_period > 0
&& mark_recovery_boundary(&mut ir_wave_pos, au.keyframe, caps.intra_refresh_period)
{
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_POINT;
}
// Reference-frame-invalidation recovery frame (AMD LTR force-reference): a clean P-frame
// off a known-good reference. Tag it so the client lifts its post-loss freeze on this one
// AU without an IDR — the definitive single-frame re-anchor (see USER_FLAG_RECOVERY_ANCHOR).
if au.recovery_anchor {
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR;
}
// Re-send the HDR mastering metadata (0xCE) on each keyframe (a decoder-resync point) and // Re-send the HDR mastering metadata (0xCE) on each keyframe (a decoder-resync point) and
// whenever it changed, so a client that dropped the best-effort datagram re-converges. // whenever it changed, so a client that dropped the best-effort datagram re-converges.
if let Some(m) = last_hdr_meta { if let Some(m) = last_hdr_meta {
@@ -4654,6 +4748,32 @@ mod tests {
assert!(reconfig_allowed(None, false)); assert!(reconfig_allowed(None, false));
} }
#[test]
fn recovery_marks_land_every_period_and_rephase_at_idr() {
let period = 4;
let mut pos = 0u32;
// Frames 1..=3 are mid-wave (no mark), frame 4 is the boundary; then it repeats.
let marks: Vec<bool> = (0..10)
.map(|_| mark_recovery_boundary(&mut pos, false, period))
.collect();
assert_eq!(
marks,
vec![false, false, false, true, false, false, false, true, false, false]
);
// An IDR mid-wave re-phases: the counter restarts, so the next boundary is a full period
// later (an IDR is itself a clean anchor, so it is not additionally marked).
let mut pos = 0u32;
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2
assert!(!mark_recovery_boundary(&mut pos, true, period)); // IDR → pos 0, no mark
// Now a fresh full period is needed, not just the 2 remaining frames.
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 1
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 2
assert!(!mark_recovery_boundary(&mut pos, false, period)); // pos 3
assert!(mark_recovery_boundary(&mut pos, false, period)); // pos 4 → mark
}
#[test] #[test]
fn pad_snapshot_replaces_state_and_seq_gates() { fn pad_snapshot_replaces_state_and_seq_gates() {
use punktfunk_core::input::{gamepad, GamepadSnapshot}; use punktfunk_core::input::{gamepad, GamepadSnapshot};
+26
View File
@@ -22,6 +22,11 @@ use std::time::Instant;
pub enum Source { pub enum Source {
/// Deterministic moving BGRx test pattern — no capture session required. /// Deterministic moving BGRx test pattern — no capture session required.
Synthetic, Synthetic,
/// Deterministic moving NV12 texture on the GPU (Windows only) — no capture session required.
/// Feeds the native AMF / D3D11 zero-copy encoders, which demand an NV12 GPU texture the CPU
/// `Synthetic` source can't give them. Used to validate GPU-encoder behaviour (e.g. AMF
/// intra-refresh) headlessly.
SyntheticNv12,
/// Live monitor via the xdg ScreenCast portal + PipeWire. /// Live monitor via the xdg ScreenCast portal + PipeWire.
Portal, Portal,
/// KWin virtual output created at `width`x`height` (zkde_screencast). Lets us validate /// KWin virtual output created at `width`x`height` (zkde_screencast). Lets us validate
@@ -56,6 +61,27 @@ pub fn run(opts: Options) -> Result<()> {
); );
Box::new(SyntheticCapturer::new(opts.width, opts.height, opts.fps)) Box::new(SyntheticCapturer::new(opts.width, opts.height, opts.fps))
} }
Source::SyntheticNv12 => {
#[cfg(target_os = "windows")]
{
tracing::info!(
width = opts.width,
height = opts.height,
fps = opts.fps,
"spike source: synthetic NV12 GPU texture (moving luma ramp)"
);
Box::new(
capture::synthetic_nv12::SyntheticNv12Capturer::new(
opts.width, opts.height, opts.fps,
)
.context("open synthetic NV12 capturer")?,
)
}
#[cfg(not(target_os = "windows"))]
{
anyhow::bail!("--source synthetic-nv12 is Windows-only (native AMF / D3D11 encoders)");
}
}
Source::Portal => { Source::Portal => {
tracing::info!("spike source: xdg ScreenCast portal (live monitor)"); tracing::info!("spike source: xdg ScreenCast portal (live monitor)");
capture::open_portal_monitor().context("open portal capturer")? capture::open_portal_monitor().context("open portal capturer")?
+24
View File
@@ -255,6 +255,25 @@
// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it). // feeding them to the decoder. Punktfunk/1 only (GameStream never sets it).
#define FLAG_PROBE 8 #define FLAG_PROBE 8
// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client
// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that
// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is
// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its
// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible
// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host
// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four.
#define USER_FLAG_RECOVERY_POINT 16
// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike
// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss
// is only half-healed so the client waits for the second), this marks an access unit the host coded
// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation**
// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client
// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts
// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets
// `AV_FRAME_FLAG_KEY` — this host flag is the only signal.
#define USER_FLAG_RECOVERY_ANCHOR 32
// Largest UDP datagram the core will send or accept. `Config::validate` bounds // Largest UDP datagram the core will send or accept. `Config::validate` bounds
// `shard_payload` so `HEADER_LEN + shard_payload + CRYPTO_OVERHEAD ≤ MAX_DATAGRAM_BYTES`. // `shard_payload` so `HEADER_LEN + shard_payload + CRYPTO_OVERHEAD ≤ MAX_DATAGRAM_BYTES`.
#define MAX_DATAGRAM_BYTES 2048 #define MAX_DATAGRAM_BYTES 2048
@@ -462,6 +481,11 @@
#define MSG_BITRATE_CHANGED 6 #define MSG_BITRATE_CHANGED 6
#endif #endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`RfiRequest`].
#define MSG_RFI_REQUEST 7
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC) #if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ProbeRequest`]. // Type byte of [`ProbeRequest`].
#define MSG_PROBE_REQUEST 32 #define MSG_PROBE_REQUEST 32
+28
View File
@@ -16,6 +16,34 @@ trap {
exit 1 exit 1
} }
# --- reclaim disk before building -----------------------------------------------------------------
# The windows-amd64 runner's system volume is intentionally small (100 GB) and a full Windows CI pass
# writes ~50 GB of cargo target output into C:\t (x64) / C:\t-a64 (arm64). Left to accumulate across
# runs that overflows the disk and the build dies with "no space on device" (os error 112) - exactly
# what took the Windows host build down. The runner bakes in a reclaimer + a scheduled task that keeps
# an idle box lean (unom/infra's setup-gitea-runner-base.ps1 ->
# C:\Users\Public\act-runner\clean-runner-disk.ps1); call it here too so THIS job starts with headroom
# regardless of when that task last ran. Threshold mode (no -Force): it only prunes when actually low,
# so incremental-compile caches survive when there's room. Best-effort - a cleanup hiccup must never
# fail the build.
$reclaimer = 'C:\Users\Public\act-runner\clean-runner-disk.ps1'
try {
if (Test-Path $reclaimer) {
& powershell.exe -NoProfile -ExecutionPolicy Bypass -File $reclaimer
}
else {
# Fallback for a runner not yet re-baked with the infra reclaimer: prune the big target dirs when low.
$freeGb = [math]::Round((Get-PSDrive C).Free / 1GB, 1)
Write-Host "[ensure-toolchain] clean-runner-disk.ps1 absent; C: free ${freeGb} GB"
if ($freeGb -lt 35) {
foreach ($d in 'C:\t', 'C:\t-a64') {
if (Test-Path $d) { Write-Host " reclaiming $d"; Remove-Item $d -Recurse -Force -ErrorAction SilentlyContinue }
}
}
}
}
catch { Write-Warning "disk reclaim step failed (non-fatal): $_" }
$ciDir = $PSScriptRoot $ciDir = $PSScriptRoot
& "$ciDir\provision-windows-wdk.ps1" & "$ciDir\provision-windows-wdk.ps1"
& "$ciDir\provision-windows-punktfunk-extras.ps1" & "$ciDir\provision-windows-punktfunk-extras.ps1"