unom/punktfunk
6
0
Fork 1
Code Issues 3 Pull Requests 3 Actions 1 Packages Projects Releases 45 Wiki Activity

feat(client): wire AV1 into the native Vulkan rung, pin-only

Browse Source 
The third codec arm in video_vk_native, AV1 admitted to native_codec and to
native_vulkan_gate by pin only. It stays out of `auto` on the same rule M5's
D3D11VA rung follows: `auto` admission is earned with hardware evidence, and
this has decoded nothing on a device.

is_integrity_warning_av1 did not exist, so the client could not have
concealed AV1 damage at all. Added, exhaustive, no wildcard: all three AV1
warnings really are damage, because AV1 has no spec-legal-but-noisy signal
to mis-classify — no reorder envelope to announce, no MMCO to rebase — and
the exhaustive match is what stops a future variant defaulting to clean.

The blocking defect review found was two safety mechanisms cancelling each
other. After a failure the decoder skipped to the next key frame answering
Ok(None), and because AV1's planner has no flush its store kept planning
cleanly, so those AUs carried no warnings and the client read them as proof
the rung works — clearing the demotion streak and resetting its clock on
every one. The streak could then never reach the threshold, which made the
never-delivered fall-through to FFmpeg-Vulkan unreachable, which is the
documented backstop for exactly three things: a level above maxLevelIdc, a
sequence header disagreeing with the Welcome, and film grain. Film grain is
the probe's own admitted assumption, so a grain stream would have frozen the
screen for the session while DecodeHealth reported run 0 — recovered.

AV1 now answers the wait with an error, as H.264 and H.265 already do
through AwaitingIdr, so all three codecs are indistinguishable to the
demotion machinery. That matters more than the extra precision of a third
state: only the H.26x paths have hardware evidence, and they are proven WITH
that behaviour.

The obvious form of that fix would have wedged the decoder. A key frame can
sit behind a skipped frame inside the same temporal unit — the vendored
vector has 24 two-frame units — so erroring out of the per-plan loop would
never reach it and the wait would never end. Skips are therefore counted per
frame and the error raised only when the whole unit was skipped, with the
metadata-only unit staying a clean Ok(None).

Also closed: a refused temporal unit left an already-decoded frame in the
ready queue, which shipped on the next AU as a clean success — putting a
picture from a refused AU on screen, clearing the streak again, and latching
delivered so the fall-through was disabled for good. The error arm now
drains and releases unshown.

MAX_DELIVERABLE is derived rather than picked: HOLD_HEADROOM minus the
pipeline's own hold, pinned to pf-vkdecode's constant so a hardcoded depth
fails the build. At the previous 8 the queue plus the presenter's 4-7 stood
against a headroom of 8, so it capped memory without preventing the
exhaustion it named, and a frame waiting 8 AUs burned 16 of the 17 query
slots — where a re-armed slot reads as Failed and becomes a fabricated
driver-corruption verdict in the very counter the Ally X signal lives in.
The trim now runs after this AU's frame is taken, or at the derived depth it
would drop a two-output unit's first frame and invert display order inside
one AU.

Its justification was also wrong: the claim that a temporal unit may carry a
show_existing_frame alongside a shown frame is disproved by this repo's own
golden — 250 units, 250 shown, zero show_existing. The bound is kept as
defence in depth against a non-conformant or multi-operating-point stream,
and now says so.

Gates: macOS fmt/clippy/392 tests, container clippy -D warnings over six
crates, 851 tests, workspace check. No hardware: the rung is pin-only and
has still never decoded a frame on a device.
This commit is contained in:
enricobuehler
2026-08-06 22:19:22 +02:00
parent cab3aa1726
commit a404830456
8 changed files with 1102 additions and 150 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/pf-client-core/Cargo.toml
+4 -3
View File
@@ -18,9 +18,10 @@ punktfunk-core = { path = "../punktfunk-core", features = ["quic"] }
# FFmpeg's Vulkan hwcontext surface (Vulkan Video decode on the presenter's device).
pf-ffvk = { path = "../pf-ffvk" }
# Native Vulkan Video decode (WP-C of the native-decode program, HEVC added by M3
# WP-2): auto's rung immediately above FFmpeg-Vulkan (2026-08-05 ladder decision), also
# pinnable via `PUNKTFUNK_DECODER=native-vulkan` — video_vk_native.rs, running
# pf-vkdecode's VkH264Decoder/VkH265Decoder on the presenter's shared device.
# WP-2, AV1 by M7 — pin only): auto's rung immediately above FFmpeg-Vulkan (2026-08-05
# ladder decision), also pinnable via `PUNKTFUNK_DECODER=native-vulkan` —
# video_vk_native.rs, running pf-vkdecode's VkH264Decoder/VkH265Decoder/VkAv1Decoder on
# the presenter's shared device.
pf-vkdecode = { path = "../pf-vkdecode" }
async-channel = "2"
crates/pf-client-core/src/lib.rs
+5 -4
View File
@@ -84,10 +84,11 @@ mod video_vaapi;
#[cfg(target_os = "linux")]
pub mod video_vaapi_native;
// Native Vulkan Video decode (WP-C of the native-decode program, HEVC added by M3
// WP-2): pf-vkdecode's H.264/H.265 decoders on the presenter's shared device — auto's
// rung immediately above FFmpeg-Vulkan (2026-08-05 ladder decision; the program is
// dropping FFmpeg from the client), also pinnable via
// `PUNKTFUNK_DECODER=native-vulkan`.
// WP-2, AV1 by M7): pf-vkdecode's H.264/H.265/AV1 decoders on the presenter's shared
// device — auto's rung immediately above FFmpeg-Vulkan (2026-08-05 ladder decision;
// the program is dropping FFmpeg from the client), also pinnable via
// `PUNKTFUNK_DECODER=native-vulkan`. The AV1 leg is PIN ONLY until it has hardware
// evidence, so an `auto` AV1 session still lands on the FFmpeg rungs.
#[cfg(any(target_os = "linux", windows))]
mod video_vk_native;
#[cfg(any(target_os = "linux", windows))]
crates/pf-client-core/src/video.rs
+175 -48
View File
@@ -18,7 +18,9 @@
//! `native-d3d11va` (Windows) pins M5's pf-dxvadec `ID3D11VideoDecoder` rung and
//! `native-vaapi` (Linux) pins M6's pf-vaadec libva rung. Both of those are reachable
//! ONLY by their pin — they are absent from every `auto` arm until they have the
//! hardware evidence M2's native rung had before IT joined `auto`):
//! hardware evidence M2's native rung had before IT joined `auto`, and M7's AV1 leg of
//! the native Vulkan rung is pin-only for the same reason: `native-vulkan` reaches it,
//! `auto` never does, so an AV1 session still lands on the FFmpeg rungs by default):
//!
//! * **Vulkan Video**: FFmpeg's Vulkan decoder running on the PRESENTER's own VkDevice
//! (its handles arrive via [`VulkanDecodeDevice`]) — the decoded VkImage feeds the
@@ -109,13 +111,14 @@ pub enum DecodedImage {
#[cfg(all(any(target_os = "linux", windows), feature = "pyrowave"))]
PyroWave(crate::video_pyrowave::PyroWavePlanarFrame),
/// Native Vulkan Video output (pf-vkdecode — auto's H.264/HEVC rung immediately
/// above FFmpeg-Vulkan, also pinnable via `PUNKTFUNK_DECODER=native-vulkan`): a
/// decoded image + per-plane views already on the PRESENTER's device — same
/// zero-copy contract as [`DecodedImage::VkFrame`], no FFmpeg involved. The
/// picture format is the stream's, carried on the frame
/// ([`NativeVkFrame::vk_format`] — NV12 for H.264 and HEVC Main, P010 for Main
/// 10, the two-plane 4:4:4 formats for RExt), never assumed. The presenter waits
/// the frame's timeline pair, transitions the layer for sampling and BACK to
/// above FFmpeg-Vulkan, plus M7's pin-only AV1 leg; pinnable via
/// `PUNKTFUNK_DECODER=native-vulkan`): a decoded image + per-plane views already
/// on the PRESENTER's device — same zero-copy contract as
/// [`DecodedImage::VkFrame`], no FFmpeg involved. The picture format is the
/// stream's, carried on the frame ([`NativeVkFrame::vk_format`] — NV12 for H.264,
/// HEVC Main and AV1 Main 8-bit, P010 for Main 10, the two-plane 4:4:4 formats for
/// RExt and AV1 High), never assumed. The presenter waits the frame's timeline
/// pair, transitions the layer for sampling and BACK to
/// [`NativeVkFrame::layout`], and releases the decoder's slot by dropping the
/// frame (its guard sends the release token).
NativeVk(NativeVkFrame),
@@ -181,6 +184,24 @@ pub struct DecodeHealth {
/// The longest [`Self::run`] of the session — the worst moment, which a
/// once-per-second sample of `run` will usually miss entirely.
pub worst_run: u32,
/// Frames that decoded CORRECTLY and were then discarded without ever being
/// shown, because the backend's deliverable queue overflowed
/// (`video_vk_native::MAX_DELIVERABLE` — a decoder making more pictures
/// display-ready per access unit than the pump can take one at a time).
///
/// Deliberately its own number and not folded into any of the three above:
/// nothing was damaged, nothing was refused and no driver failed, so counting
/// it as any of those would put a damage report on a healthy stream — and the
/// AU it happened on still showed a picture, so it must not extend
/// [`Self::run`] either. But it cannot be nothing at all: a session quietly
/// discarding a frame per AU is one running at half the frame rate it thinks
/// it is, and before this counter existed it read as perfectly clean.
///
/// Structurally 0 on every rung but native Vulkan — it is the only one with a
/// deliverable queue — and not on the session stats line today; the
/// rate-limited `warn` at the drop site is the field signal, and this is the
/// number a stats field would read.
pub dropped: u64,
/// This device answers per-op decode-status queries
/// (`queryResultStatusSupport`). When FALSE — RADV, where recording a query
/// anyway HANGS the VCN ring — [`Self::failed`] can only ever read 0, because
@@ -225,6 +246,17 @@ impl DecodeHealth {
self.run = 0;
}
}
/// Note one correctly-decoded frame discarded unshown — see [`Self::dropped`].
///
/// Separate from [`Self::note`] because it is not an AU verdict: several frames
/// can be dropped within one access unit, and the access unit itself may well
/// have shipped a picture. It touches nothing but its own counter, and in
/// particular never [`Self::run`], which answers "did the picture come back"
/// and here it did.
pub(crate) fn note_dropped(&mut self) {
self.dropped = self.dropped.saturating_add(1);
}
}
/// A raw `VkFormat` code point, carried across the ash-free boundary.
@@ -640,13 +672,15 @@ impl Drop for DrmFrameGuard {
enum Backend {
Vulkan(VulkanDecoder),
/// Native Vulkan Video H.264/HEVC (pf-vkdecode) on the presenter's device —
/// Native Vulkan Video H.264/HEVC/AV1 (pf-vkdecode) on the presenter's device —
/// auto's rung immediately above FFmpeg-Vulkan since the 2026-08-05 ladder
/// decision (WP-D closed bit-exact; the program's goal is dropping FFmpeg from
/// the client), also pinnable by name (`PUNKTFUNK_DECODER=native-vulkan`) — see
/// [`native_vulkan_gate`]. The negotiated codec picks the decoder once, at
/// construction; everything else about this backend is codec-agnostic. Errors
/// ride the SAME streak/demotion machinery as the FFmpeg-Vulkan rung.
/// construction; everything else about this backend is codec-agnostic. Its AV1
/// leg (M7) is reachable by the PIN only and never through `auto`, which is the
/// gate's decision, not this variant's. Errors ride the SAME streak/demotion
/// machinery as the FFmpeg-Vulkan rung.
/// Boxed: the decoder (planner + shipped-frame ledger) dwarfs the other variants,
/// same as PyroWave below.
NativeVulkan(Box<NativeVulkanDecoder>),
@@ -804,14 +838,12 @@ fn clears_demotion_streak(delivered: bool, concealed: bool) -> bool {
/// `VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR` — the raw flag bit within
/// [`VulkanDecodeDevice::decode_video_caps`] (this crate stays ash-free).
const VIDEO_CODEC_OP_DECODE_H264: u32 = 0x0000_0001;
/// `VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR` — its H.265 sibling. (AV1 is 0x4
/// and deliberately has no constant here: pf-vkdecode has no AV1 decoder, so the bit
/// would only invite a gate that admits a session nothing can decode.)
/// `VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR` — its H.265 sibling.
const VIDEO_CODEC_OP_DECODE_H265: u32 = 0x0000_0002;
/// `VK_VIDEO_CODEC_OPERATION_DECODE_AV1_BIT_KHR`. The Deck's VanGogh advertises
/// it alongside H.264/H.265/VP9, and it is what
/// [`av1_hardware_decodable`] reads.
/// it alongside H.264/H.265/VP9; it is what [`av1_hardware_decodable`] reads and,
/// since M7, the caps bit [`native_codec`] demands for an AV1 session.
const VIDEO_CODEC_OP_DECODE_AV1: u32 = 0x0000_0004;
/// The native decoder for a negotiated wire codec, plus the
@@ -821,13 +853,16 @@ const VIDEO_CODEC_OP_DECODE_AV1: u32 = 0x0000_0004;
/// The two are returned together on purpose: "which decoder" and "which caps bit"
/// are one fact, and splitting them is how a gate ends up admitting HEVC on an
/// H.264-only decode family (`vkCreateVideoSessionKHR` for a codec operation the
/// family cannot run is undefined behaviour, not an error). AV1 has a Vulkan decode
/// op and real hardware advertises it — but there is no AV1 decoder in pf-vkdecode,
/// so those sessions must keep falling through to the FFmpeg rungs.
/// family cannot run is undefined behaviour, not an error).
///
/// ⚠ Being here is "pf-vkdecode has a decoder", NOT "the automatic ladder may pick
/// it". AV1 (M7) is pin-only; [`native_vulkan_gate`] holds that decision, and it
/// reads this map for the codec/caps pair only.
fn native_codec(codec_id: ffmpeg::codec::Id) -> Option<(NativeCodec, u32)> {
match codec_id {
ffmpeg::codec::Id::H264 => Some((NativeCodec::H264, VIDEO_CODEC_OP_DECODE_H264)),
ffmpeg::codec::Id::HEVC => Some((NativeCodec::H265, VIDEO_CODEC_OP_DECODE_H265)),
ffmpeg::codec::Id::AV1 => Some((NativeCodec::Av1, VIDEO_CODEC_OP_DECODE_AV1)),
_ => None,
}
}
@@ -862,8 +897,9 @@ fn native_vaapi_codec(codec_id: ffmpeg::codec::Id) -> Option<pf_vaadec::Codec> {
}
/// The native Vulkan Video admission gate (WP-C of the native-decode program, widened
/// by the 2026-08-05 ladder decision and again by M3 WP-2's HEVC wiring): the
/// pf-vkdecode backend engages when `choice` asks for it — by name
/// by the 2026-08-05 ladder decision, again by M3 WP-2's HEVC wiring and again — for
/// the pin only — by M7's AV1 wiring): the pf-vkdecode backend engages when `choice`
/// asks for it — by name
/// (`PUNKTFUNK_DECODER=native-vulkan` — `choice` is env-first, so that's what carries
/// it) or as the auto family (`auto`/``/`hardware`), where native is the rung
/// immediately ABOVE FFmpeg-Vulkan: WP-D closed with bit-exact parity against
@@ -879,10 +915,19 @@ fn native_vaapi_codec(codec_id: ffmpeg::codec::Id) -> Option<pf_vaadec::Codec> {
/// pin — refuses.
///
/// Beyond the choice: the negotiated wire codec must be one pf-vkdecode speaks —
/// H.264 or H.265 ([`native_codec`]) — and the presenter's decode family must
/// H.264, H.265 or AV1 ([`native_codec`]) — and the presenter's decode family must
/// advertise THAT codec's decode operation. `video_decode` alone proves the extension
/// stack, never the codec: an AV1-only decode family exists on real hardware, and
/// H.264-only ones are the common case on older silicon. AV1 sessions refuse outright.
/// H.264-only ones are the common case on older silicon.
///
/// **AV1 (M7) is admitted by the PIN ONLY** and is absent from the `auto` family, on
/// exactly the rule M5's native D3D11VA and M6's native VAAPI rungs follow: `auto`
/// admission is earned with hardware parity and a soak, and the AV1 rung has decoded
/// nothing on hardware. An `auto` AV1 session therefore keeps landing where it landed
/// before M7 — the FFmpeg rungs — and the pin is what a lab run uses to reach the new
/// one. The per-codec choice test is the one thing that makes this gate more than a
/// codec lookup, so it lives here rather than in [`native_codec`], which stays the
/// answer to "does a decoder exist and which caps bit does it need".
///
/// What the gate deliberately does NOT check is the stream's picture SHAPE — that is
/// [`NativeVulkanDecoder::new`]'s construction-time probe, which has the negotiated
@@ -894,12 +939,18 @@ fn native_vulkan_gate(
video_decode: bool,
decode_video_caps: u32,
) -> bool {
let Some((_, codec_op)) = native_codec(codec_id) else {
let Some((codec, codec_op)) = native_codec(codec_id) else {
return false;
};
matches!(choice, "native-vulkan" | "auto" | "" | "hardware")
&& video_decode
&& decode_video_caps & codec_op != 0
let chosen = match codec {
// Hardware-proven rungs: the pin AND the whole auto family.
NativeCodec::H264 | NativeCodec::H265 => {
matches!(choice, "native-vulkan" | "auto" | "" | "hardware")
}
// Pin only, until this rung has decoded a frame on real hardware.
NativeCodec::Av1 => choice == "native-vulkan",
};
chosen && video_decode && decode_video_caps & codec_op != 0
}
/// Map a negotiated `quic` codec bit to the FFmpeg decoder id the client opens.
@@ -1136,7 +1187,8 @@ impl Decoder {
/// ([`VulkanDecodeDevice::prefer_vulkan_first`]); on H.264 and HEVC sessions the
/// native pf-vkdecode rung sits immediately above FFmpeg-Vulkan wherever the
/// ladder reaches it ([`native_vulkan_gate`] — the program is dropping FFmpeg, and
/// a native INIT failure falls through to FFmpeg-Vulkan). Linux: native → Vulkan →
/// a native INIT failure falls through to FFmpeg-Vulkan). An AV1 session does NOT
/// take it in `auto` — that leg is pin-only (M7). Linux: native → Vulkan →
/// VAAPI → software on NVIDIA and ALL AMD (`prefer_vulkan_first` is vendor-wide —
/// desktop RADV included, on-glass verdict — not just the Deck's VanGogh);
/// VAAPI → native → Vulkan → software on Intel/unknown. Windows (no VAAPI
@@ -1302,9 +1354,9 @@ impl Decoder {
tracing::warn!(
?codec_id,
video_decode = vk.is_some_and(|v| v.video_decode),
"PUNKTFUNK_DECODER=native-vulkan refused (needs an H.264 or HEVC session \
and a presenter device whose decode family advertises that codec) — \
standard ladder"
"PUNKTFUNK_DECODER=native-vulkan refused (needs an H.264, HEVC or AV1 \
session and a presenter device whose decode family advertises that \
codec) — standard ladder"
);
}
choice = "auto".to_string();
@@ -2173,6 +2225,51 @@ mod tests {
"three driver errors interleaved with concealment must still reach the \
demotion threshold — they got to {fails}"
);
// ---- The AV1 shape (M7), and the reason its recovery wait is an `Err` ----
//
// A native rung waiting to re-anchor after a failure produces no picture for
// every AU of the wait, and all three codecs say so with an ERROR: H.264 and
// H.265 through their planners' `PlanError::AwaitingIdr`, AV1 through
// `VkDecodeError::AwaitingKeyAv1`. So the streak ticks for the whole wait and
// a rung that never recovers reaches the threshold.
let mut fails = 0u32;
for errored in [true; 5] {
// the failing AU, then four skipped ones
if errored {
fails += 1;
} else if clears_demotion_streak(false, false) {
fails = 0;
}
}
assert!(fails >= VAAPI_DEMOTE_AFTER);
// The counterfactual is the whole point, and it is what the AV1 rung was
// first wired as: answer the skipped AUs with a CLEAN `Ok(None)` instead —
// no picture, no warnings, nothing to object to — and every one of them
// clears the streak. The `Err` from each failure is then alone, and
// `VAAPI_DEMOTE_AFTER` is unreachable no matter how long the session runs.
//
// The stream this strands is real and named in `NativeVulkanDecoder::new`:
// an AV1 sequence with `film_grain_params_present = 1` on a device without
// the grain decode profile fails at `ensure_state` — at EVERY key frame, and
// only at a key frame. Key frame `Err`, inter frames "clean", next key frame
// `Err`: a frozen screen for the whole session, `refused N · damaged 0 ·
// run 0` on the stats line, and the `!delivered` fall-through to
// FFmpeg-Vulkan below never reached.
let mut fails = 0u32;
for errored in [true, false, false, true, false, false, true, false, false] {
if errored {
fails += 1;
} else if clears_demotion_streak(false, false) {
fails = 0;
}
}
assert!(
fails < VAAPI_DEMOTE_AFTER,
"a recovery wait answered as a CLEAN AU zeroes the streak once per frame \
— which is why it must not be answered that way; it got to {fails}"
);
}
/// Auto's hardware order (both OSes): Vulkan-first on NVIDIA (on Linux: no usable
@@ -2198,10 +2295,6 @@ mod tests {
assert!(!decode_device(0x8086, "Intel(R) Arc(TM) Pro Graphics").prefer_vulkan_first());
}
/// The native-Vulkan admission gate (WP-C, widened by the 2026-08-05 ladder
/// decision and again by M3 WP-2's HEVC wiring): the pin AND the auto family
/// admit on a capable H.264 or HEVC session — native sits immediately above
/// FFmpeg-Vulkan because the program is dropping FFmpeg — while every explicit
/// AV1 is advertised on a HARDWARE fact, never on a decoder existing.
///
/// The standing open item M7 closes. `ffmpeg::decoder::find(AV1)` says yes
@@ -2238,11 +2331,16 @@ mod tests {
assert!(!av1_hardware_decodable(Some(&dev)));
}
/// backend pin refuses (`vulkan` names the FFmpeg-Vulkan backend specifically and
/// must keep meaning exactly that), and the codec/device legs still refuse for
/// every choice. The codec's OWN caps bit is the device leg: admitting HEVC on an
/// H.264-only decode family would create a video session for an operation the
/// family cannot run, which is undefined behaviour rather than an error.
/// The native-Vulkan admission gate (WP-C, widened by the 2026-08-05 ladder
/// decision, again by M3 WP-2's HEVC wiring and again — pin only — by M7's AV1
/// wiring): the pin AND the auto family admit on a capable H.264 or HEVC session
/// (native sits immediately above FFmpeg-Vulkan because the program is dropping
/// FFmpeg), the PIN ALONE admits AV1, every explicit other-backend pin refuses
/// (`vulkan` names the FFmpeg-Vulkan backend specifically and must keep meaning
/// exactly that), and the codec/device legs still refuse for every choice. The
/// codec's OWN caps bit is the device leg: admitting HEVC on an H.264-only decode
/// family would create a video session for an operation the family cannot run,
/// which is undefined behaviour rather than an error.
#[test]
fn native_vulkan_gate_admits_pin_and_auto_family_per_codec_on_a_capable_family() {
use ffmpeg::codec::Id;
@@ -2258,11 +2356,13 @@ mod tests {
VIDEO_CODEC_OP_DECODE_H265, 0x2,
"VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR"
);
assert_eq!(
VIDEO_CODEC_OP_DECODE_AV1, 0x4,
"VK_VIDEO_CODEC_OPERATION_DECODE_AV1_BIT_KHR"
);
const H264_OP: u32 = VIDEO_CODEC_OP_DECODE_H264;
const H265_OP: u32 = VIDEO_CODEC_OP_DECODE_H265;
// `VK_VIDEO_CODEC_OPERATION_DECODE_AV1_BIT_KHR` — a real bit on real
// hardware, and never enough on its own (no AV1 decoder exists here).
const AV1_OP: u32 = 0x4;
const AV1_OP: u32 = VIDEO_CODEC_OP_DECODE_AV1;
for choice in ["native-vulkan", "auto", "", "hardware"] {
// The pin and the whole auto family admit both codecs pf-vkdecode
// speaks, on a family that advertises the matching op…
@@ -2293,14 +2393,31 @@ mod tests {
!native_vulkan_gate(choice, Id::H264, true, H265_OP),
"{choice:?}"
);
// AV1 refuses whatever the family advertises — pf-vkdecode has no AV1
// decoder, so the session must fall through to the FFmpeg rungs.
// AV1 (M7) is PIN ONLY: `native-vulkan` reaches it, and the whole auto
// family must keep landing on the FFmpeg rungs exactly as it did before
// this rung existed. That is not a caps question — the family below
// advertises AV1 — it is the "auto admission is earned with hardware
// parity and a soak" rule, and this rung has decoded nothing.
let av1_pin = choice == "native-vulkan";
assert_eq!(
native_vulkan_gate(choice, Id::AV1, true, AV1_OP),
av1_pin,
"{choice:?}"
);
assert_eq!(
native_vulkan_gate(choice, Id::AV1, true, H264_OP | H265_OP | AV1_OP),
av1_pin,
"{choice:?}"
);
// …and the pin is still not a licence to skip the device leg: an AV1
// session on a family that does not advertise the AV1 op would create a
// video session for an operation the family cannot run.
assert!(
!native_vulkan_gate(choice, Id::AV1, true, AV1_OP),
!native_vulkan_gate(choice, Id::AV1, true, H264_OP | H265_OP),
"{choice:?}"
);
assert!(
!native_vulkan_gate(choice, Id::AV1, true, H264_OP | H265_OP | AV1_OP),
!native_vulkan_gate(choice, Id::AV1, false, AV1_OP),
"{choice:?}"
);
// No Vulkan-Video-capable presenter device.
@@ -2336,6 +2453,10 @@ mod tests {
!native_vulkan_gate(choice, Id::HEVC, true, H265_OP),
"{choice:?}"
);
assert!(
!native_vulkan_gate(choice, Id::AV1, true, AV1_OP),
"{choice:?}"
);
}
// The decoder the gate implies — the construction sites `expect()` this
// exact agreement, so a codec admitted with no decoder behind it would be a
@@ -2348,7 +2469,13 @@ mod tests {
native_codec(Id::HEVC).map(|(c, _)| c),
Some(NativeCodec::H265)
);
assert!(native_codec(Id::AV1).is_none());
// AV1 has a decoder AND the caps bit here — being in this map is what the
// pin construction path reads. Whether `auto` may pick it is the gate's
// decision above, and deliberately not this one's.
assert_eq!(
native_codec(Id::AV1),
Some((NativeCodec::Av1, VIDEO_CODEC_OP_DECODE_AV1))
);
assert!(native_codec(Id::VP9).is_none());
}
crates/pf-client-core/src/video_vk_native.rs
+678 -70
View File
File diff suppressed because it is too large Load Diff
crates/pf-vkdecode/src/decoder.rs
+25
View File
@@ -234,6 +234,26 @@ pub enum VkDecodeError {
/// produce. `ref_index` is the AV1 reference name (`LAST_FRAME` = 0 through
/// `ALTREF_FRAME` = 6), `slot` the reference slot it pointed at.
MissingReferenceAv1 { slot: u8, ref_index: u8 },
/// Every frame of this AV1 temporal unit was SKIPPED because the decoder is
/// waiting for the next key frame after a failure — nothing decoded, nothing
/// displayed.
///
/// AV1's answer to [`pf_bitstream::h264::PlanError::AwaitingIdr`], and
/// deliberately the same KIND of answer: an error, once per access unit, for
/// as long as the wait lasts. The AV1 planner has no `flush`, so the wait is
/// held in [`crate::VkAv1Decoder`] rather than in the planner — but a consumer
/// must not be able to tell the two codecs apart here, because the consumer's
/// demotion streak is what turns "this rung produces no picture" into "fall
/// through to the next rung". Answering the wait with a clean `Ok(None)`
/// instead RESETS that streak once per frame, and a rung whose every key frame
/// fails (a film-grain sequence on a device without the grain profile, a level
/// above `maxLevelIdc`, a sequence header disagreeing with the negotiation)
/// then never demotes at all: one error per key frame, cleared by the inter
/// frames between them, and a frozen screen for the whole session.
///
/// A key frame ANYWHERE in the unit clears the wait and decodes, so this is
/// returned only when the unit produced nothing at all.
AwaitingKeyAv1,
/// Plan-to-Vulkan conversion failed (caller/session bugs; `CapacityMismatch`
/// is consumed internally by the rebuild path and only surfaces if the rebuilt
/// session STILL mismatches).
@@ -302,6 +322,11 @@ impl std::fmt::Display for VkDecodeError {
no picture — the surviving references would renumber"
)
}
VkDecodeError::AwaitingKeyAv1 => write!(
f,
"every frame of this AV1 temporal unit was skipped — the decoder is \
waiting for the next key frame after a failure"
),
VkDecodeError::Convert(e) => write!(f, "plan conversion failed: {e}"),
VkDecodeError::ConvertH265(e) => write!(f, "H.265 plan conversion failed: {e}"),
VkDecodeError::Caps(e) => write!(f, "decode capabilities unusable: {e}"),
crates/pf-vkdecode/src/decoder_av1.rs
+145 -19
View File
@@ -539,6 +539,25 @@ pub(crate) fn lost_reference(warnings: &[PlanWarning]) -> Option<(u8, u8)> {
})
}
/// What [`VkAv1Decoder::decode_planned`] did with one frame of a temporal unit.
///
/// Two outcomes rather than a bare `Ok(())`, because "the plan was honoured" and
/// "the decoder is waiting for a key frame and did nothing" are opposite
/// statements about the rung, and the caller has to count the second: a unit in
/// which EVERY frame was skipped produced no picture at all and comes back as
/// [`VkDecodeError::AwaitingKeyAv1`], while a unit where a key frame cleared the
/// wait partway through decoded normally (see [`VkAv1Decoder::awaiting_key`]).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FrameOutcome {
/// The plan was carried out: submitted, or (a `show_existing_frame`) settled
/// into a display verdict without a submission. Either way the unit produced
/// this frame.
Decoded,
/// Skipped: the decoder is waiting for the next key frame after a failure and
/// this frame is undecodable by construction.
SkippedAwaitingKey,
}
/// Everything tied to ONE AV1 session generation. A stream renegotiation (extent
/// or profile — including a bit-depth, sampling or film-grain switch) retires it
/// and builds fresh.
@@ -606,21 +625,33 @@ pub struct VkAv1Decoder {
/// ([`RecoveryLatch`] docs for the whole argument).
recovery: RecoveryLatch,
/// Every frame until the next KEY frame is undecodable, and is skipped rather
/// than failed.
/// than converted.
///
/// This exists because AV1's planner has no `flush`: when a failure forces
/// [`Self::recover_dpb`] to empty this decoder's slot ledger and image
/// bindings, the PLANNER's own eight-slot store still believes those pictures
/// are resident and keeps handing out inter frames that reference them. Each
/// would fail in `plan_to_vk_av1` with `UnresolvedReference` — a real error
/// per frame, at frame rate, which reads to the integration layer as a decoder
/// that has stopped working rather than a stream waiting to re-anchor.
/// would fail in `plan_to_vk_av1` with `UnresolvedReference` — a per-frame
/// failure whose message describes a phantom reference gap rather than the
/// wait that is really in progress, and which would drag every one of those
/// frames through a conversion that cannot succeed.
///
/// So the frames between the failure and the key frame are ANSWERED like the
/// H.265 decoder answers a RASL picture after an open-GOP join: `Ok` with
/// whatever was already display-ready, planner untouched, no error and no
/// second keyframe request. A key frame (which references nothing and refreshes
/// all eight slots) clears it and decoding resumes.
/// So the frames are skipped. What they are NOT is laundered into a clean
/// answer: a temporal unit in which every frame was skipped comes back as
/// [`VkDecodeError::AwaitingKeyAv1`], once per access unit, exactly as the
/// H.264/H.265 decoders answer the same wait with their planners'
/// `PlanError::AwaitingIdr`. The three codecs must be indistinguishable here,
/// because the consumer's demotion streak is the only thing that turns "this
/// rung produces no picture" into "fall through to the next rung": a clean
/// `Ok(None)` RESETS that streak once per frame, so a rung whose every key
/// frame fails would never reach the threshold and the session would keep a
/// frozen screen with a clean bill of health. During a recovery wait the
/// decoder really has stopped working, and that is what the streak must see.
///
/// A DECODED key frame (which references nothing and refreshes all eight
/// slots) clears it and decoding resumes — including one that arrives partway
/// through a temporal unit, which is why the skip is per FRAME while the error
/// is per ACCESS UNIT.
awaiting_key: bool,
}
@@ -704,9 +735,11 @@ impl VkAv1Decoder {
/// Returns the next display-ready frame, if the planner declared one; drain the
/// rest with [`Self::take_ready`].
///
/// A frame skipped while [`Self::awaiting_key`] is set is NOT an error (its
/// docs carry the argument); nor is a `show_existing_frame` naming an empty
/// slot, which the planner reports as a warning and which simply displays
/// A temporal unit whose every frame was skipped while [`Self::awaiting_key`]
/// is set comes back as [`VkDecodeError::AwaitingKeyAv1`] — the same kind of
/// answer the H.264/H.265 decoders give for the same wait, and for the reason
/// [`Self::awaiting_key`]'s docs carry. A `show_existing_frame` naming an empty
/// slot is NOT that: the planner reports it as a warning and it simply displays
/// nothing.
///
/// Never panics. `VkDecodeError::DeviceLost` latches: every later call fails
@@ -746,22 +779,40 @@ impl VkAv1Decoder {
self.last_warnings.extend(plan.warnings.iter().cloned());
}
let mut skipped = 0usize;
for plan in &plans {
// From here the PLANNER has already advanced past this frame — its
// store holds the picture whatever happens next — so any failure below
// leaves the planner's store and this decoder's ledgers able to
// disagree. Latch the recovery rather than returning into a permanently
// wedged state.
if let Err(e) = self.decode_planned(plan, au) {
self.recovery.latch();
return Err(e);
match self.decode_planned(plan, au) {
Ok(FrameOutcome::Decoded) => {}
Ok(FrameOutcome::SkippedAwaitingKey) => skipped += 1,
Err(e) => {
self.recovery.latch();
return Err(e);
}
}
}
// Nothing in this unit decoded and nothing was displayed, because the
// decoder is still waiting for a key frame. That is an ERROR per access
// unit — [`VkDecodeError::AwaitingKeyAv1`] and [`Self::awaiting_key`] carry
// the argument — and deliberately not a latch: `recover_dpb` has already
// run, the ledgers are consistent, and re-latching would re-flush an empty
// ledger once per frame for the whole wait.
//
// Counted rather than short-circuited inside the loop, because a key frame
// may sit BEHIND a skipped frame in the same temporal unit: returning at
// the first skip would never reach it, and the wait would never end.
if whole_unit_skipped(plans.len(), skipped) {
return Err(VkDecodeError::AwaitingKeyAv1);
}
Ok(self.ready.pop_front())
}
/// One planned frame of a temporal unit.
fn decode_planned(&mut self, plan: &AuPlan, au: &[u8]) -> Result<(), VkDecodeError> {
fn decode_planned(&mut self, plan: &AuPlan, au: &[u8]) -> Result<FrameOutcome, VkDecodeError> {
// A key frame re-anchors everything: it references nothing and refreshes
// all eight slots, so it is decodable no matter what came before.
//
@@ -779,7 +830,7 @@ impl VkAv1Decoder {
show_existing = plan.dpb.stored.is_none(),
"frame skipped while awaiting the next AV1 key frame"
);
return Ok(());
return Ok(FrameOutcome::SkippedAwaitingKey);
}
// `show_existing_frame`: no decode at all. It displays a slot's contents —
@@ -792,7 +843,9 @@ impl VkAv1Decoder {
state.slots.release(id);
}
}
return Ok(());
// Decoded: nothing was submitted, but the plan was HONOURED — it
// declared a picture displayable, which is a frame the unit produced.
return Ok(FrameOutcome::Decoded);
};
// A reference the planner could not resolve: refuse before anything is
@@ -1031,7 +1084,7 @@ impl VkAv1Decoder {
state.pool.pictures[entry.image].pending = false;
}
}
Ok(())
Ok(FrameOutcome::Decoded)
}
/// Apply one plan's DPB verdicts: outputs become ready frames (their images
@@ -1622,6 +1675,28 @@ fn clears_awaiting_key(plan: &AuPlan) -> bool {
plan.picture.is_key && plan.dpb.stored.is_some()
}
/// Did a temporal unit of `planned` frames produce NOTHING because every one of
/// them was skipped waiting for a key frame — the
/// [`VkDecodeError::AwaitingKeyAv1`] condition?
///
/// A named function rather than the expression inlined at the call site because
/// both of its edges are load-bearing and neither is obvious:
///
/// * `planned == 0` is not a skip. A temporal unit can plan no frames at all (one
/// carrying only metadata or a sequence header), and that is an ordinary
/// `Ok(None)` — turning it into an error would fail access units on a perfectly
/// healthy stream.
/// * `skipped < planned` is not a skip either, and this is the case an early
/// return inside the loop would have got wrong: a key frame may sit BEHIND a
/// skipped frame in the same unit, clears the wait when it is reached, and
/// decodes. Reporting the unit as skipped there would answer an error for an
/// access unit that really did decode a picture.
///
/// Pure, so the aggregation is CPU-testable without a device.
fn whole_unit_skipped(planned: usize, skipped: usize) -> bool {
planned > 0 && skipped == planned
}
/// The stream's level, as the sequence header's FIRST operating point states it.
///
/// Operating point 0 is the full stream — the one a non-scalable decoder decodes
@@ -2789,6 +2864,57 @@ mod tests {
assert!(!clears_awaiting_key(&inter));
}
/// A recovery WAIT must reach the consumer as an ERROR, once per access unit —
/// the same answer H.264/H.265 give through their planners'
/// `PlanError::AwaitingIdr`, and the reason [`VkAv1Decoder::awaiting_key`]'s
/// docs carry: a clean `Ok(None)` resets the consumer's demotion streak once
/// per frame, so a rung whose every key frame fails (film grain on a device
/// without the grain profile; a level above `maxLevelIdc`; a sequence header
/// disagreeing with the negotiation) would never demote and the session would
/// hold a frozen screen with a clean bill of health.
///
/// What this pins is the AGGREGATION, which is where the naive fix goes wrong:
/// the error is per ACCESS UNIT while the skip is per FRAME, because a key
/// frame can sit behind a skipped frame in the same temporal unit — the
/// vendored vector has 24 units carrying two frames each.
#[test]
fn a_unit_reports_the_key_frame_wait_only_when_it_decoded_nothing_at_all() {
// The wait itself: every frame of the unit skipped.
assert!(whole_unit_skipped(1, 1), "a single-frame unit");
assert!(whole_unit_skipped(2, 2), "and a two-frame one");
// A key frame arrived partway through the unit and decoded: NOT the wait,
// whatever came before it. An early return at the first skip would have
// answered an error here and never reached the key frame at all.
assert!(!whole_unit_skipped(2, 1));
assert!(!whole_unit_skipped(3, 2));
// Nothing was skipped: the ordinary decoding case.
assert!(!whole_unit_skipped(2, 0));
// A unit that planned no frames (metadata / a sequence header on its own)
// is a clean `Ok(None)`, never an error.
assert!(!whole_unit_skipped(0, 0));
}
/// The wait's error must be DISTINGUISHABLE from the failure that started it —
/// a support engineer reading a field log has to be able to tell "the AU could
/// not be decoded" from "the decoder is waiting to re-anchor", and the two ride
/// the same `Err` channel.
#[test]
fn the_key_frame_wait_names_itself_in_the_error_text() {
let waiting = format!("{}", VkDecodeError::AwaitingKeyAv1);
assert!(waiting.contains("key frame"), "{waiting}");
assert!(waiting.contains("skipped"), "{waiting}");
// …and it is not the same message as the loss that latched the recovery.
let lost = format!(
"{}",
VkDecodeError::MissingReferenceAv1 {
slot: 3,
ref_index: 2
}
);
assert_ne!(waiting, lost);
}
/// The `refresh_frame_flags == 0` leg is real AV1 and this vector has none of
/// it — which is worth PROVING rather than assuming, because it is exactly the
/// sort of "cannot happen" that quietly exhausts a nine-slot ledger in the
crates/pf-vkdecode/src/integrity.rs
+65 -2
View File
@@ -1,7 +1,7 @@
//! Which planner warnings mean the PICTURE is damaged (M4 of the native-decode
//! program).
//!
//! Both planners emit two very different kinds of thing through one warning
//! The planners emit two very different kinds of thing through one warning
//! channel, and the split is what a consumer must branch on:
//!
//! * **Integrity** — a reference the DPB does not hold, a `frame_num` gap, an AU
@@ -24,7 +24,7 @@
//! does not actually perform — the exact shape of the `nb_queries = 0` failure the
//! program exists to end.
use crate::{H265PlanWarning, PlanWarning};
use crate::{Av1PlanWarning, H265PlanWarning, PlanWarning};
/// Does this H.264 planner warning mean the PICTURE is damaged?
///
@@ -59,6 +59,41 @@ pub fn is_integrity_warning_h265(w: &H265PlanWarning) -> bool {
}
}
/// The AV1 twin (M7). Every variant the AV1 planner has today IS damage, and that
/// is a fact about the codec rather than an oversight: AV1 puts nothing in this
/// channel that resembles h265's `NonZeroReorder` or h264's `Mmco5Rebase`. It has
/// no reorder envelope to report (no bumping process, no `max_num_reorder_pics`)
/// and no MMCO to rebase — the frame header states the whole reference update
/// outright — so the only things left to warn about are pictures that went
/// missing and an OBU walk that stopped early.
///
/// `MissingShowExisting` is the one that could be argued, and it is damage: a
/// `show_existing_frame` naming an empty slot means the picture the STREAM chose
/// to display was lost upstream. Nothing is displayed for that frame, so the
/// screen keeps the previous one — exactly the "silently stale picture" state a
/// re-anchor exists to end.
///
/// ⚠ `MissingReference` is classified here for completeness and does NOT normally
/// reach a consumer through this predicate: [`crate::VkAv1Decoder`] refuses the
/// whole access unit for it ([`crate::VkDecodeError::MissingReferenceAv1`]),
/// because AV1's `refs` array is indexed by reference NAME and there is no legal
/// substitute to write into a hole — a `-1` for a name the frame really references
/// is a spec violation whose firmware behaviour is undefined. So the AV1 rung
/// answers a lost reference as a REFUSAL, not as concealment, and it is the
/// refusal counter that moves. Classifying it as damage here anyway keeps the two
/// statements consistent for any consumer that does see the warning (and for the
/// fault harness, which asserts detection against exactly this list).
///
/// Exhaustive for the same reason as [`is_integrity_warning`]: a new AV1 warning
/// must not be able to mean "damaged" and read as clean.
pub fn is_integrity_warning_av1(w: &Av1PlanWarning) -> bool {
match w {
Av1PlanWarning::MissingReference { .. }
| Av1PlanWarning::MissingShowExisting { .. }
| Av1PlanWarning::TruncatedAu { .. } => true,
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -105,4 +140,32 @@ mod tests {
every ABR renegotiation's IDR"
);
}
/// AV1's whole warning vocabulary is damage. Note plainly what this test does
/// and does not guard, because the two are easy to confuse:
///
/// * A NEW variant is caught by the EXHAUSTIVE MATCH in
/// [`is_integrity_warning_av1`], not here — this loop enumerates the variants
/// by hand, so a fourth one would simply not appear in it. That is the whole
/// reason the function is written as a match with no `_` arm.
/// * What this test does guard is a RECLASSIFICATION: split one of these names
/// out of the `|` chain and give it a `false` arm — the shape a future
/// "spec-legal AV1 signal" would arrive in — and the assertion below fires.
/// `MissingShowExisting` is the one most likely to be argued down that way (a
/// frame that decoded nothing and displayed nothing reads as harmless), and
/// reading it as clean would leave the previous picture on the screen with no
/// re-anchor asked for.
#[test]
fn every_av1_warning_is_damage_because_av1_has_no_envelope_signal() {
for w in [
Av1PlanWarning::MissingReference {
slot: 3,
ref_index: 1,
},
Av1PlanWarning::MissingShowExisting { slot: 5 },
Av1PlanWarning::TruncatedAu { offset: 900 },
] {
assert!(is_integrity_warning_av1(&w), "{w:?} is damage");
}
}
}
crates/pf-vkdecode/src/lib.rs
+5 -4
View File
@@ -96,10 +96,10 @@
//! [`DecodedVkFrame::recovery`]). The only clean point an intra-refresh session
//! has — its wave emits no IDR — so without it a client freezes for its full
//! backstop and then forces the very IDR the wave exists to avoid.
//! - [`integrity`]: [`is_integrity_warning`] / [`is_integrity_warning_h265`], the
//! one list of warnings that mean the PICTURE is damaged. Here rather than in the
//! client so the fault harness asserts against the predicate production conceals
//! on.
//! - [`integrity`]: [`is_integrity_warning`] / [`is_integrity_warning_h265`] /
//! [`is_integrity_warning_av1`], the one list of warnings that mean the PICTURE
//! is damaged. Here rather than in the client so the fault harness asserts
//! against the predicate production conceals on.
//! - [`fault`]: [`AuFault`], deliberate decoder-input corruption
//! (`PUNKTFUNK_AU_FAULT`), inert unless armed. A detector nobody can fire is
//! exactly as trustworthy as no detector at all.
@@ -209,6 +209,7 @@ pub use images::plan_pools;
pub use images::PoolPlan;
pub use images::HOLD_HEADROOM;
pub use integrity::is_integrity_warning;
pub use integrity::is_integrity_warning_av1;
pub use integrity::is_integrity_warning_h265;
pub use params::pps_to_std;
pub use params::sps_to_std;