From a6e51215fd2c74b20acc24eebb3fbf93e9f4a108 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Thu, 6 Aug 2026 16:27:13 +0200 Subject: [PATCH] =?UTF-8?q?feat(client):=20M6's=20rung=20is=20wired=20?= =?UTF-8?q?=E2=80=94=20libva,=20dlopen'd,=20no=20libavcodec?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The native VAAPI decoder now runs end to end: pf-vaadec's plans go into libva's buffers, the surface comes back as DRM-PRIME dmabufs, and the presenter imports them exactly as it does the FFmpeg rung's. Pin-only — `PUNKTFUNK_DECODER=native-vaapi` — for the reason M5's D3D11VA rung was: `auto` admission is earned with hardware parity and a soak, and this rung has decoded nothing yet. libva is dlopen'd rather than linked, so the pf-lxcheck2 container compiles and clippies the whole thing without libva-dev, and a machine without a VAAPI runtime gets a clean refusal instead of a packaging dependency. The surface pool is not the slot map. `SlotMap::assign` hands out the lowest free slot, and a slot freed by an access unit's own removals is free by the time that unit's picture takes it — measured at 225 of the vendored vector's 250 access units. A surface bound by slot index would therefore decode, on nine frames in ten, into the surface still holding the picture on screen. So `plan_to_va` now takes the decode target as a parameter, bound by the caller at activation time the way pf-vkdecode binds a pool image, and a surface is free only when no live picture is bound to it, no output is owed for it, and no consumer holds it. Measured rather than transcribed, as everywhere else here: layout-probe.c grew the export descriptor (312 bytes, objects[4]/layers[4]), the buffer-type enumerators — VASliceParameterBufferType is 4 and VASliceDataBufferType is 5, not the 3 and 4 that counting off the header suggests — and the config, attribute and generic-value layouts. All pinned as compile-time assertions, which is how the 12-byte VAGenericValue in the first draft was caught: the C union holds a pointer, so it is 8-aligned and 16 bytes. The plane walk lives in pf-vaadec, pure and unit-tested on macOS, because it is the one structure the DRIVER writes and we read: SEPARATE_LAYERS returns NV12 as two layers, and taking layers[0] is the green screen this project has already paid for. It also refuses what it cannot express rather than guessing — a bogus object count, a plane naming an object that is not there, objects disagreeing on tiling. Own DecodedImage variant, same payload type. The physical hand-off is identical to the FFmpeg rung's, so the presenter keeps ONE arm and one demotion streak; the variant exists so the compiler asks which rung decoded wherever that matters. Both D3D11VA rungs share a variant and `1573a987` had to fix the consequence afterwards — a "native" soak that could silently have been an FFmpeg soak. Here the four uncovered matches were compile errors. Buffers are destroyed by us, not by vaEndPicture: va.h is explicit that the user must call vaDestroyBuffer, and the libva 0.x behaviour is long gone. Leaking two per picture at 60 fps exhausts the driver's store in minutes. pf-vaadec's presenter headroom was 4, written against no consumer. The Vulkan rung had already measured the client pipeline at four to seven held frames; it is 8 now, pinned to that crate's constant so a re-measurement moves both. Gates: macOS fmt/clippy/341 tests/cargo doc, and in the container clippy -D warnings over six crates, 795 tests, workspace check. Hardware legs are still owed — no AMD/Mesa or Intel box was reachable. --- Cargo.lock | 2 + clients/session/README.md | 4 +- crates/pf-client-core/Cargo.toml | 10 + crates/pf-client-core/src/lib.rs | 6 + crates/pf-client-core/src/session.rs | 9 + crates/pf-client-core/src/video.rs | 156 +- crates/pf-client-core/src/video_vaapi.rs | 2 +- .../pf-client-core/src/video_vaapi_native.rs | 1855 +++++++++++++++++ crates/pf-client-core/src/video_vulkan.rs | 2 +- crates/pf-presenter/src/run.rs | 10 +- crates/pf-vaadec/layout-probe.c | 63 + crates/pf-vaadec/src/config.rs | 38 +- crates/pf-vaadec/src/drm.rs | 466 +++++ crates/pf-vaadec/src/lib.rs | 20 +- crates/pf-vaadec/src/pic.rs | 146 +- crates/pf-vaadec/src/pic_h265.rs | 35 +- crates/pf-vaadec/src/va.rs | 13 + 17 files changed, 2780 insertions(+), 57 deletions(-) create mode 100644 crates/pf-client-core/src/video_vaapi_native.rs create mode 100644 crates/pf-vaadec/src/drm.rs diff --git a/Cargo.lock b/Cargo.lock index 74cc3b99..6b522db9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2990,11 +2990,13 @@ dependencies = [ "ash", "async-channel", "ffmpeg-next", + "libloading", "mdns-sd", "opus", "pf-dxvadec", "pf-ffvk", "pf-update-check", + "pf-vaadec", "pf-vkdecode", "pipewire", "punktfunk-core", diff --git a/clients/session/README.md b/clients/session/README.md index 8d9534f6..c8cc5572 100644 --- a/clients/session/README.md +++ b/clients/session/README.md @@ -62,7 +62,9 @@ tone-map in-shader to SDR when it doesn't (`PUNKTFUNK_TONEMAP_PEAK` tunes the ro default ≈1000 nits). The host still gates the upgrade behind its `PUNKTFUNK_10BIT` policy. -Debug/bisect knobs: `PUNKTFUNK_DECODER=native-vulkan|vulkan|vaapi|d3d11va|software`, `PUNKTFUNK_PRESENT_MODE= +Debug/bisect knobs: `PUNKTFUNK_DECODER=native-vulkan|native-vaapi|native-d3d11va|vulkan|vaapi|d3d11va|software` +(the three `native-*` values pin this program's own decoders; `native-vaapi` also takes +`PUNKTFUNK_VAAPI_DEVICE=/dev/dri/renderDNNN` to choose the GPU), `PUNKTFUNK_PRESENT_MODE= mailbox|fifo|immediate|fifo_relaxed` (default MAILBOX, FIFO where the surface offers no MAILBOX — AMD on Windows), `PUNKTFUNK_VK_DEVICE=` (multi-GPU), and `PUNKTFUNK_HW_FAULT=import` (fault every VAAPI dmabuf import — proves the three-strike diff --git a/crates/pf-client-core/Cargo.toml b/crates/pf-client-core/Cargo.toml index d9806c1e..0d39da70 100644 --- a/crates/pf-client-core/Cargo.toml +++ b/crates/pf-client-core/Cargo.toml @@ -59,6 +59,16 @@ rand = "0.9" [target.'cfg(target_os = "linux")'.dependencies] pipewire = "0.9" sdl3 = { version = "0.18", features = ["hidapi"] } +# Native VAAPI decode (M6 of the native-decode program): the hand-declared libva buffer +# layouts, the profile/format/surface decisions, the AuPlan → picparams/IQ/slice +# conversion and the DRM-PRIME export descriptor that `video_vaapi_native` marshals. +# Cross-platform on purpose — everything decidable without a device is tested by the +# ordinary macOS and container gates, exactly as pf-dxvadec does for Windows. +pf-vaadec = { path = "../pf-vaadec" } +# libva itself is dlopen'd, never linked (see `video_vaapi_native`'s module docs): the +# container can then compile and clippy the whole rung without `libva-dev`, and a machine +# without a VAAPI runtime gets a clean refusal instead of a packaging dependency. +libloading = "0.8" [target.'cfg(windows)'.dependencies] wasapi = "0.23" diff --git a/crates/pf-client-core/src/lib.rs b/crates/pf-client-core/src/lib.rs index 78955667..e95e9ff9 100644 --- a/crates/pf-client-core/src/lib.rs +++ b/crates/pf-client-core/src/lib.rs @@ -77,6 +77,12 @@ mod video_software; mod video_libav; #[cfg(target_os = "linux")] mod video_vaapi; +// Native VAAPI decode (M6 of the native-decode program): pf-vaadec's plans driven +// straight into libva, dlopen'd at runtime, exporting DRM-PRIME dmabufs the +// presenter imports — the FFmpeg-free replacement for `video_vaapi`. Pin-only for +// now (`PUNKTFUNK_DECODER=native-vaapi`). +#[cfg(target_os = "linux")] +pub mod video_vaapi_native; // Native Vulkan Video decode (WP-C of the native-decode program, HEVC added by M3 // WP-2): pf-vkdecode's H.264/H.265 decoders on the presenter's shared device — auto's // rung immediately above FFmpeg-Vulkan (2026-08-05 ladder decision; the program is diff --git a/crates/pf-client-core/src/session.rs b/crates/pf-client-core/src/session.rs index cc5708d9..c022aa4f 100644 --- a/crates/pf-client-core/src/session.rs +++ b/crates/pf-client-core/src/session.rs @@ -892,6 +892,11 @@ fn pump( DecodedImage::Cpu(_) => "software", #[cfg(target_os = "linux")] DecodedImage::Dmabuf(_) => "vaapi", + // A separate VARIANT rather than a flag on the frame, + // unlike the D3D11VA pair below — so this tag cannot be + // forgotten, only written. + #[cfg(target_os = "linux")] + DecodedImage::NativeDmabuf(_) => "native-vaapi", DecodedImage::VkFrame(_) => "vulkan", // Both D3D11VA rungs deliver this variant — they share the // hand-off ring on purpose — so the frame carries which one @@ -915,6 +920,10 @@ fn pump( DecodedImage::Cpu(c) => (c.width, c.height, "software"), #[cfg(target_os = "linux")] DecodedImage::Dmabuf(d) => (d.width, d.height, "vaapi-dmabuf"), + #[cfg(target_os = "linux")] + DecodedImage::NativeDmabuf(d) => { + (d.width, d.height, "native-vaapi-dmabuf") + } DecodedImage::VkFrame(v) => (v.width, v.height, "vulkan-video"), #[cfg(windows)] DecodedImage::D3d11(d) => (d.width, d.height, "d3d11va"), diff --git a/crates/pf-client-core/src/video.rs b/crates/pf-client-core/src/video.rs index 42869bcd..caa46550 100644 --- a/crates/pf-client-core/src/video.rs +++ b/crates/pf-client-core/src/video.rs @@ -12,11 +12,12 @@ //! native → vulkan → software on Intel/unknown. Windows: native → vulkan → d3d11va → //! software on NVIDIA/AMD, d3d11va → native → vulkan → software on Intel/unknown. //! Override: -//! `PUNKTFUNK_DECODER=vulkan|vaapi|d3d11va|software|native-vulkan|native-d3d11va` — -//! `vulkan` names the FFmpeg-Vulkan backend specifically; `native-vulkan` pins the +//! `PUNKTFUNK_DECODER=vulkan|vaapi|d3d11va|software|native-vulkan|native-d3d11va|native-vaapi` +//! — `vulkan` names the FFmpeg-Vulkan backend specifically; `native-vulkan` pins the //! pf-vkdecode decoder by name, skipping the vendor-ordered rungs ahead of it; -//! `native-d3d11va` (Windows) pins M5's pf-dxvadec `ID3D11VideoDecoder` rung, which is -//! reachable ONLY by that pin — it is absent from every `auto` arm until it has the +//! `native-d3d11va` (Windows) pins M5's pf-dxvadec `ID3D11VideoDecoder` rung and +//! `native-vaapi` (Linux) pins M6's pf-vaadec libva rung. Both of those are reachable +//! ONLY by their pin — they are absent from every `auto` arm until they have the //! hardware evidence M2's native rung had before IT joined `auto`): //! //! * **Vulkan Video**: FFmpeg's Vulkan decoder running on the PRESENTER's own VkDevice @@ -81,6 +82,20 @@ pub enum DecodedImage { Cpu(CpuFrame), #[cfg(target_os = "linux")] Dmabuf(DmabufFrame), + /// The NATIVE VAAPI rung's output (`pf-vaadec` + `video_vaapi_native`, M6) — + /// physically the same thing as [`DecodedImage::Dmabuf`], and deliberately the + /// same payload type, because the import a consumer performs is identical: + /// dmabuf fds plus a plane layout. It is a separate VARIANT purely so the two + /// rungs can never be confused for one another. + /// + /// That is not fastidiousness. Both D3D11VA rungs share one variant (they share + /// the hand-off ring on purpose), and the consequence had to be fixed in + /// `1573a987`: the `stats:` decode-path tag is derived from the variant, so a + /// "native" soak could silently have been an FFmpeg soak, and there was no way + /// to tell from the log. Here the compiler asks the question instead — every + /// `match` on `DecodedImage` must say which rung it means. + #[cfg(target_os = "linux")] + NativeDmabuf(DmabufFrame), /// FFmpeg Vulkan Video output: a VkImage already on the PRESENTER's device. VkFrame(VkVideoFrame), /// D3D11VA output copied into a shareable NT-handle texture the presenter imports @@ -485,7 +500,7 @@ impl DecodedImage { match self { DecodedImage::Cpu(f) => f.keyframe, #[cfg(target_os = "linux")] - DecodedImage::Dmabuf(f) => f.keyframe, + DecodedImage::Dmabuf(f) | DecodedImage::NativeDmabuf(f) => f.keyframe, DecodedImage::VkFrame(f) => f.keyframe, #[cfg(windows)] DecodedImage::D3d11(f) => f.keyframe, @@ -528,7 +543,7 @@ impl DecodedImage { match self { DecodedImage::Cpu(f) => (f.width, f.height), #[cfg(target_os = "linux")] - DecodedImage::Dmabuf(f) => (f.width, f.height), + DecodedImage::Dmabuf(f) | DecodedImage::NativeDmabuf(f) => (f.width, f.height), DecodedImage::VkFrame(f) => (f.width, f.height), #[cfg(windows)] DecodedImage::D3d11(f) => (f.width, f.height), @@ -581,21 +596,45 @@ pub struct DmabufPlane { pub stride: u32, } -/// Owns the mapped DRM-PRIME `AVFrame` (which in turn references the VAAPI surface). -/// Dropping it releases the surface back to the decoder pool and closes the fds. -pub struct DrmFrameGuard(pub(crate) *mut ffmpeg::ffi::AVFrame); -// SAFETY: the guard owns one `AVFrame` and frees it exactly once in `Drop`. libav's buffer +/// Keeps a decoded surface alive until the consumer's GPU reads are done: dropping +/// it releases the surface back to its decoder's pool and closes the fds. +/// +/// The consumer treats this as opaque — the presenter dups every dmabuf fd it +/// imports and simply holds the guard until its fence has been waited — so the only +/// thing the two variants differ in is WHO owns the surface. libavcodec's rungs hand +/// over a mapped `AVFrame`; the native VAAPI rung (`video_vaapi_native`, M6) owns a +/// `VASurface` from its own pool and has no `AVFrame` at all, which is precisely the +/// seam that had to be widened for it to exist. M10 deletes the FFmpeg variant and +/// this enum collapses again. +pub enum DrmFrameGuard { + /// A mapped DRM-PRIME `AVFrame` — the FFmpeg VAAPI hwaccel — or the cloned + /// `AVFrame` behind an `AVVkFrame` on the FFmpeg Vulkan path. + Av(*mut ffmpeg::ffi::AVFrame), + /// The native VAAPI rung's own owner: closes the exported PRIME fds and returns + /// the surface to the decoder's pool. + #[cfg(target_os = "linux")] + NativeVa(crate::video_vaapi_native::VaFrameGuard), +} +// SAFETY: the `Av` variant owns one `AVFrame` and frees it exactly once in `Drop`. libav's buffer // refcounts are atomic and its hwframe pool is internally locked, so releasing the frame — and with // it the VAAPI surface, back to the decoder's pool — from a different thread than the one that // mapped it is sound. That is the whole point here: the guard is handed to GTK and dropped on the // main thread while the pump thread keeps decoding. Moved, never shared; deliberately NOT `Sync`. +// The `NativeVa` variant is `Send` on its own (owned fds plus an `mpsc::Sender`) and needs no +// promise from here. unsafe impl Send for DrmFrameGuard {} impl Drop for DrmFrameGuard { fn drop(&mut self) { - // SAFETY: `self.0` is the one `AVFrame` this guard owns; `av_frame_free` releases it - // exactly once (this `Drop` runs once) and nulls the pointer through the `&mut`. - unsafe { ffmpeg::ffi::av_frame_free(&mut self.0) }; + match self { + // SAFETY: this is the one `AVFrame` the guard owns; `av_frame_free` releases it + // exactly once (this `Drop` runs once) and nulls the pointer through the `&mut`. + DrmFrameGuard::Av(frame) => unsafe { ffmpeg::ffi::av_frame_free(frame) }, + // The native guard releases through its own `Drop`, which runs as this value's + // fields are dropped — right after this match. + #[cfg(target_os = "linux")] + DrmFrameGuard::NativeVa(_) => {} + } } } @@ -613,6 +652,18 @@ enum Backend { NativeVulkan(Box), #[cfg(target_os = "linux")] Vaapi(VaapiDecoder), + /// Native VAAPI (`pf-vaadec` + `video_vaapi_native`) — M6's replacement for the + /// FFmpeg-backed [`Backend::Vaapi`] rung: libva driven straight from pf-bitstream + /// plans, dlopen'd, exporting the same DRM-PRIME dmabufs, no libavcodec. + /// **Pin-only** (`PUNKTFUNK_DECODER=native-vaapi`) and deliberately NOT in the + /// automatic ladder, on the same rule M5's native D3D11VA rung follows: `auto` + /// admission is earned with hardware parity and a soak, and this rung has decoded + /// nothing yet. Errors ride the SAME streak/demotion machinery as every other + /// hardware rung. + /// Boxed: the decoder (two planners, a display and a surface pool) dwarfs the + /// other variants. + #[cfg(target_os = "linux")] + NativeVaapi(Box), #[cfg(windows)] D3d11va(crate::video_d3d11::D3d11vaDecoder), /// Native D3D11VA (`pf-dxvadec` + `video_d3d11_native`) — M5's replacement for the @@ -791,6 +842,20 @@ fn native_d3d11_codec(codec_id: ffmpeg::codec::Id) -> Option } } +/// The native VAAPI decoder for a negotiated wire codec, or `None` for one pf-vaadec +/// cannot decode. Like its DXVA twin there is no caps bit to consult first: VAAPI +/// advertises support as a profile/entrypoint pair on the DISPLAY, which +/// [`crate::video_vaapi_native::NativeVaapiDecoder::new`] queries on the device it is +/// about to build on. +#[cfg(target_os = "linux")] +fn native_vaapi_codec(codec_id: ffmpeg::codec::Id) -> Option { + match codec_id { + ffmpeg::codec::Id::H264 => Some(pf_vaadec::Codec::H264), + ffmpeg::codec::Id::HEVC => Some(pf_vaadec::Codec::H265), + _ => None, + } +} + /// The native Vulkan Video admission gate (WP-C of the native-decode program, widened /// by the 2026-08-05 ladder decision and again by M3 WP-2's HEVC wiring): the /// pf-vkdecode backend engages when `choice` asks for it — by name @@ -1127,6 +1192,37 @@ impl Decoder { } choice = "auto".to_string(); } + // Native VAAPI (M6, pf-vaadec) — PIN ONLY, ahead of everything because a pin is a + // pin, and absent from every `auto` arm below for the same reason its D3D11VA + // sibling is: `auto` admission is earned with hardware parity and a soak. A + // refusal or an init failure logs and drops to the standard ladder (choice reads + // as `auto` from here on), so a native failure is never quieter, nor lands + // somewhere other, than an FFmpeg rung's failure. + #[cfg(target_os = "linux")] + if choice == crate::video_vaapi_native::DECODER_PIN { + match native_vaapi_codec(codec_id) { + Some(codec) => { + match crate::video_vaapi_native::NativeVaapiDecoder::new(codec, stream) { + Ok(d) => { + tracing::info!( + ?codec_id, + decoder = d.name(), + "native VAAPI hardware decode active (pf-vaadec, zero-copy dmabuf)" + ); + return done(Backend::NativeVaapi(Box::new(d))); + } + Err(e) => tracing::warn!(reason = %format!("{e:#}"), + "native VAAPI init failed — demoting to the standard ladder"), + } + } + None => tracing::warn!( + ?codec_id, + "PUNKTFUNK_DECODER=native-vaapi refused (needs an H.264 or HEVC \ + session) — standard ladder" + ), + } + choice = "auto".to_string(); + } let mut native_tried = false; if choice == "native-vulkan" { if native_vulkan_gate( @@ -1398,6 +1494,10 @@ impl Decoder { // claiming a driver verdict nothing can produce. #[cfg(windows)] Backend::NativeD3d11va(d) => Some(d.health()), + // Same shape as the DXVA rung above, for the same reason: libva has no + // per-picture decode-status query either. + #[cfg(target_os = "linux")] + Backend::NativeVaapi(d) => Some(d.health()), _ => None, } } @@ -1547,6 +1647,20 @@ impl Decoder { } #[cfg(target_os = "linux")] Backend::Vaapi(v) => v.decode(au).map(|f| f.map(DecodedImage::Dmabuf)), + #[cfg(target_os = "linux")] + Backend::NativeVaapi(v) => { + debug_assert!(complete, "partial AUs are pyrowave-only"); + let r = v.decode(au).map(|f| f.map(DecodedImage::NativeDmabuf)); + // Same split as the two native rungs above, for the same reason: this + // rung can SEE stream damage, and turning what an FFmpeg rung conceals + // silently into an error would demote it on exactly the lossy links it + // exists to diagnose. + if v.take_recovery_request() { + self.want_keyframe = true; + concealed = true; + } + r + } #[cfg(windows)] Backend::D3d11va(d) => d.decode(au).map(|f| f.map(DecodedImage::D3d11)), #[cfg(windows)] @@ -1595,6 +1709,8 @@ impl Decoder { Backend::D3d11va(_) => "D3D11VA", #[cfg(windows)] Backend::NativeD3d11va(_) => "native D3D11VA", + #[cfg(target_os = "linux")] + Backend::NativeVaapi(_) => "native VAAPI", _ => "VAAPI", }; self.vaapi_fails += 1; @@ -1644,13 +1760,21 @@ impl Decoder { // FFmpeg-Vulkan-on-Mesa error-streaking where VAAPI streams // perfectly); only when that can't be built either does the // session land on software. + // The NATIVE VAAPI rung demotes here too, and to the same place: its + // failure is a statement about pf-vaadec's submission, not about + // VAAPI, so libavcodec's decoder on the very same profile is the + // right next rung — and while that rung is pin-only, this is the + // only way a lab session that pinned it keeps hardware decode. #[cfg(target_os = "linux")] - if matches!(self.backend, Backend::Vulkan(_) | Backend::NativeVulkan(_)) { + if matches!( + self.backend, + Backend::Vulkan(_) | Backend::NativeVulkan(_) | Backend::NativeVaapi(_) + ) { match VaapiDecoder::new(self.codec_id) { Ok(v) => { tracing::warn!(error = %e, fails = self.vaapi_fails, - decoder = v.name(), - "Vulkan Video decode failing repeatedly — demoting to VAAPI"); + from = which, decoder = v.name(), + "hardware decode failing repeatedly — demoting to VAAPI"); self.backend = Backend::Vaapi(v); self.vaapi_fails = 0; self.first_fail = None; diff --git a/crates/pf-client-core/src/video_vaapi.rs b/crates/pf-client-core/src/video_vaapi.rs index 1aa79ec8..b1aeadfd 100644 --- a/crates/pf-client-core/src/video_vaapi.rs +++ b/crates/pf-client-core/src/video_vaapi.rs @@ -197,7 +197,7 @@ impl VaapiDecoder { return Err(averr("av_hwframe_map", r)); } let desc = (*drm).data[0] as *const ffi::AVDRMFrameDescriptor; - let guard = DrmFrameGuard(drm); + let guard = DrmFrameGuard::Av(drm); let d = &*desc; if d.nb_layers < 1 || d.nb_objects < 1 { bail!("DRM descriptor without layers/objects"); diff --git a/crates/pf-client-core/src/video_vaapi_native.rs b/crates/pf-client-core/src/video_vaapi_native.rs new file mode 100644 index 00000000..9118aeb6 --- /dev/null +++ b/crates/pf-client-core/src/video_vaapi_native.rs @@ -0,0 +1,1855 @@ +//! Native VAAPI decode — M6 of the native-decode program, and the FFmpeg-free +//! replacement for [`crate::video_vaapi`]. +//! +//! `pf-vaadec` turns one pf-bitstream `AuPlan` into the buffers a +//! `vaRenderPicture` call carries; this module is everything libva-shaped around +//! that: the display, the config and context, the surface pool, the submission, and +//! the DRM-PRIME export the presenter imports. Its output is +//! [`DecodedImage::NativeDmabuf`] — physically identical to what the FFmpeg VAAPI +//! rung delivers, deliberately a different variant so a log can never confuse the +//! two rungs (see that variant's docs). +//! +//! # libva is dlopen'd, never linked +//! +//! Everything here resolves `libva.so.2` and `libva-drm.so.2` at runtime. Three +//! things follow, and all three are the point: +//! +//! * `pf-client-core` gains no build-time libva dependency, so the **pf-lxcheck2 +//! container compiles and clippies this whole rung** even though it has no +//! `libva-dev`. On a program where `cfg(windows)` code could only ever be checked +//! on a box, that is the difference between a defect found on a laptop and one +//! found on hardware. +//! * A machine without libva gets a clean refusal at construction — the ladder +//! falls through exactly as it does for any other unavailable rung — instead of a +//! packaging dependency or a link error. +//! * Nothing in the shipped packages needs to change to try it. +//! +//! # The surface pool, and why it is not the slot map +//! +//! VAAPI has no DPB slots: `VAPictureH264::picture_id` is a `VASurfaceID`, and +//! `vaBeginPicture` takes the target surface itself. The slot ledger +//! ([`pf_vaadec::SlotMap`], borrowed from the Vulkan rung) is our own indirection +//! from a stable `PicId` to a small integer, and a slot is emphatically NOT a +//! surface index. +//! +//! It cannot be, because [`pf_vaadec::SlotMap::assign`] hands out the lowest free +//! slot and a slot freed by this access unit's own removals is free by then — +//! measured at **225 of the vendored vector's 250 access units** (pf-vaadec's +//! `the_setup_picture_routinely_inherits_a_just_freed_slot`). A surface bound by +//! slot index would therefore decode, on nine frames in ten, straight into the +//! surface holding the picture that was just displayed. Under zero-copy the +//! presenter is still sampling that surface: it holds the frame until its fence has +//! been waited, which is exactly what "zero-copy" costs. +//! +//! So the pool follows pf-vkdecode's image model. Surfaces outnumber slots by +//! [`pf_vaadec::config::PRESENTER_HEADROOM`], the decode target is taken from a free +//! list at activation time and bound to its slot afterwards, and a surface the +//! presenter holds simply stays off the free list until its release token comes +//! back. A surface is free when no live picture is bound to it AND no consumer holds +//! it — two conditions, tracked separately, because they end at different times. + +use std::os::fd::AsRawFd as _; +use std::os::fd::FromRawFd as _; +use std::os::fd::OwnedFd; +use std::os::raw::c_char; +use std::os::raw::c_int; +use std::os::raw::c_uint; +use std::os::raw::c_void; +use std::sync::mpsc; + +use anyhow::anyhow; +use anyhow::bail; +use anyhow::Context as _; +use anyhow::Result; + +use crate::video::DecodeHealth; +use crate::video::DmabufFrame; +use crate::video::DmabufPlane; +use crate::video::DrmFrameGuard; +use crate::video::StreamFormat; +use crate::video_color::ColorDesc; + +/// `PUNKTFUNK_DECODER=native-vaapi` — the pin that selects this rung. +/// +/// Pin-only, like M5's native D3D11VA rung was at the same stage and for the same +/// reason: the native Vulkan rung joined `auto` only after bit-exact parity on +/// several drivers and a long soak, and this one has decoded nothing yet. +pub(crate) const DECODER_PIN: &str = "native-vaapi"; + +// --------------------------------------------------------------------------- +// libva, resolved at runtime +// --------------------------------------------------------------------------- + +/// `VADisplay` — an opaque driver handle. +type VaDisplay = *mut c_void; +type VaStatus = c_int; +type VaSurfaceId = c_uint; +type VaConfigId = c_uint; +type VaContextId = c_uint; +type VaBufferId = c_uint; + +const VA_STATUS_SUCCESS: VaStatus = 0; +/// `VA_INVALID_ID` — also the "no surface" sentinel in a slot table. +const VA_INVALID_ID: c_uint = 0xffff_ffff; +/// `VA_PROGRESSIVE` — the only picture structure this rung's envelope contains. +const VA_PROGRESSIVE: c_uint = 0x0001; + +/// `VAGenericValue` — 16 bytes, value at offset 8, align 8 (measured). +/// +/// The C type's `value` is a union of `int`/`float`/`void*`/function pointer. Two +/// consequences are written out here rather than left to a Rust `union` declaration: +/// +/// * the union holds a pointer, so it is **eight-byte aligned** — which is why the +/// enum ahead of it is followed by four bytes of padding, and why the whole thing +/// is 16 bytes and not 12. The compile-time assertion below caught exactly that +/// mistake in this file; +/// * a Rust union initialised through its `i32` arm leaves the other four bytes +/// **uninitialised**, and those are the bytes a driver reading the pointer arm +/// would see. Naming the remainder and writing zero means everything crossing the +/// FFI boundary was written by us. +#[repr(C, align(8))] +#[derive(Clone, Copy)] +struct VaGenericValue { + kind: c_int, + _pad: u32, + /// The union's integer arm, first in the union — where `VAGenericValue.i` lives. + i: i32, + /// The rest of the union. Always zero. + _rest: u32, +} + +#[repr(C)] +#[derive(Clone, Copy)] +struct VaSurfaceAttrib { + kind: c_int, + flags: c_uint, + value: VaGenericValue, +} + +/// `VASurfaceAttribPixelFormat` / `VAGenericValueTypeInteger` / +/// `VA_SURFACE_ATTRIB_SETTABLE` — measured by `pf-vaadec/layout-probe.c`. +const VA_SURFACE_ATTRIB_PIXEL_FORMAT: c_int = 1; +const VA_GENERIC_VALUE_TYPE_INTEGER: c_int = 1; +const VA_SURFACE_ATTRIB_SETTABLE: c_uint = 0x0002; + +// The layouts these calls pass by value, measured (`pf-vaadec/layout-probe.c`). +const _: () = { + assert!(size_of::() == 16); + assert!(std::mem::offset_of!(VaGenericValue, i) == 8); + assert!(size_of::() == 24); + assert!(std::mem::offset_of!(VaSurfaceAttrib, flags) == 4); + assert!(std::mem::offset_of!(VaSurfaceAttrib, value) == 8); +}; + +/// The libva entry points this rung calls, resolved from `libva.so.2` and +/// `libva-drm.so.2` at runtime (the same pattern the host's NVML and CUDA loaders +/// use — no link-time dependency, absent library = clean refusal). +struct Libva { + _va: libloading::Library, + _drm: libloading::Library, + get_display_drm: unsafe extern "C" fn(c_int) -> VaDisplay, + initialize: unsafe extern "C" fn(VaDisplay, *mut c_int, *mut c_int) -> VaStatus, + terminate: unsafe extern "C" fn(VaDisplay) -> VaStatus, + error_str: unsafe extern "C" fn(VaStatus) -> *const c_char, + query_config_entrypoints: + unsafe extern "C" fn(VaDisplay, c_int, *mut c_int, *mut c_int) -> VaStatus, + max_entrypoints: unsafe extern "C" fn(VaDisplay) -> c_int, + create_config: unsafe extern "C" fn( + VaDisplay, + c_int, + c_int, + *mut c_void, + c_int, + *mut VaConfigId, + ) -> VaStatus, + destroy_config: unsafe extern "C" fn(VaDisplay, VaConfigId) -> VaStatus, + create_surfaces: unsafe extern "C" fn( + VaDisplay, + c_uint, + c_uint, + c_uint, + *mut VaSurfaceId, + c_uint, + *mut VaSurfaceAttrib, + c_uint, + ) -> VaStatus, + destroy_surfaces: unsafe extern "C" fn(VaDisplay, *mut VaSurfaceId, c_int) -> VaStatus, + create_context: unsafe extern "C" fn( + VaDisplay, + VaConfigId, + c_int, + c_int, + c_int, + *mut VaSurfaceId, + c_int, + *mut VaContextId, + ) -> VaStatus, + destroy_context: unsafe extern "C" fn(VaDisplay, VaContextId) -> VaStatus, + create_buffer: unsafe extern "C" fn( + VaDisplay, + VaContextId, + c_uint, + c_uint, + c_uint, + *mut c_void, + *mut VaBufferId, + ) -> VaStatus, + destroy_buffer: unsafe extern "C" fn(VaDisplay, VaBufferId) -> VaStatus, + begin_picture: unsafe extern "C" fn(VaDisplay, VaContextId, VaSurfaceId) -> VaStatus, + render_picture: + unsafe extern "C" fn(VaDisplay, VaContextId, *mut VaBufferId, c_int) -> VaStatus, + end_picture: unsafe extern "C" fn(VaDisplay, VaContextId) -> VaStatus, + sync_surface: unsafe extern "C" fn(VaDisplay, VaSurfaceId) -> VaStatus, + /// `vaExportSurfaceHandle(dpy, surface_id, mem_type, flags, descriptor)` — five + /// parameters, and the descriptor's type is decided by `mem_type`. + export_surface_handle: + unsafe extern "C" fn(VaDisplay, VaSurfaceId, c_uint, c_uint, *mut c_void) -> VaStatus, +} + +impl Libva { + fn load() -> Result { + // SAFETY: `Library::new` runs the trusted system libva's initialisers, and each + // `lib.get` resolves a documented libva symbol to the matching `unsafe extern "C"` + // signature transcribed from `va.h` / `va_drm.h` (by-value integers and pointers + // throughout, no callbacks). Both `Library` handles are stored in the returned + // struct, so every resolved pointer outlives its uses. + unsafe { + let va = libloading::Library::new("libva.so.2") + .context("libva.so.2 (no VAAPI runtime on this system)")?; + let drm = libloading::Library::new("libva-drm.so.2") + .context("libva-drm.so.2 (no VAAPI DRM backend on this system)")?; + // Each symbol is resolved AT the field's own type — `Library::get` is + // generic, so the struct's declared signature is what `dlsym`'s pointer + // is read as. No `transmute` anywhere: a mistyped entry point is then a + // mismatch the reader can see next to the declaration rather than a cast + // that accepts anything. Bound with `let` (not inline in the literal) so + // each borrow of the `Library` ends before it is moved into the struct. + macro_rules! get { + ($lib:expr, $name:literal) => { + *$lib + .get(concat!($name, "\0").as_bytes()) + .map_err(|e| anyhow!(concat!("dlsym ", $name, ": {}"), e))? + }; + } + let get_display_drm = get!(drm, "vaGetDisplayDRM"); + let initialize = get!(va, "vaInitialize"); + let terminate = get!(va, "vaTerminate"); + let error_str = get!(va, "vaErrorStr"); + let query_config_entrypoints = get!(va, "vaQueryConfigEntrypoints"); + let max_entrypoints = get!(va, "vaMaxNumEntrypoints"); + let create_config = get!(va, "vaCreateConfig"); + let destroy_config = get!(va, "vaDestroyConfig"); + let create_surfaces = get!(va, "vaCreateSurfaces"); + let destroy_surfaces = get!(va, "vaDestroySurfaces"); + let create_context = get!(va, "vaCreateContext"); + let destroy_context = get!(va, "vaDestroyContext"); + let create_buffer = get!(va, "vaCreateBuffer"); + let destroy_buffer = get!(va, "vaDestroyBuffer"); + let begin_picture = get!(va, "vaBeginPicture"); + let render_picture = get!(va, "vaRenderPicture"); + let end_picture = get!(va, "vaEndPicture"); + let sync_surface = get!(va, "vaSyncSurface"); + let export_surface_handle = get!(va, "vaExportSurfaceHandle"); + Ok(Libva { + get_display_drm, + initialize, + terminate, + error_str, + query_config_entrypoints, + max_entrypoints, + create_config, + destroy_config, + create_surfaces, + destroy_surfaces, + create_context, + destroy_context, + create_buffer, + destroy_buffer, + begin_picture, + render_picture, + end_picture, + sync_surface, + export_surface_handle, + _va: va, + _drm: drm, + }) + } + } + + /// libva's own text for a status code, so a driver's reason reaches the log + /// instead of a bare number. + fn err(&self, what: &str, status: VaStatus) -> anyhow::Error { + // SAFETY: `vaErrorStr` is documented total — it returns a pointer into libva's + // static string table for any input, valid while the library is loaded, which + // `&self` proves. + let text = unsafe { + let p = (self.error_str)(status); + if p.is_null() { + String::new() + } else { + std::ffi::CStr::from_ptr(p).to_string_lossy().into_owned() + } + }; + if text.is_empty() { + anyhow!("{what} failed ({status})") + } else { + anyhow!("{what} failed: {text} ({status})") + } + } + + fn check(&self, what: &str, status: VaStatus) -> Result<()> { + if status == VA_STATUS_SUCCESS { + Ok(()) + } else { + Err(self.err(what, status)) + } + } +} + +// --------------------------------------------------------------------------- +// The display +// --------------------------------------------------------------------------- + +/// An initialised `VADisplay` over a DRM render node. +struct Display { + va: Libva, + display: VaDisplay, + /// The render node. libva does NOT dup the fd it is given, so the display is + /// only valid while this is open — it is dropped after `vaTerminate`. + node: Option, + /// Which node, for the field report that asks "which GPU decoded?". + path: String, + version: (c_int, c_int), +} + +// SAFETY: the display is created and used from ONE thread (the pump), and `Send` only +// permits MOVING that ownership. libva is not safe for concurrent calls on one +// display, which is why `Sync` is deliberately absent: every path into it goes through +// `&mut NativeVaapiDecoder`, and that is the serialisation. +unsafe impl Send for Display {} + +impl Display { + /// Open a render node and initialise libva on it. + /// + /// `PUNKTFUNK_VAAPI_DEVICE` pins a node explicitly. Otherwise nodes are tried in + /// name order and the first that initialises wins — the rule libavcodec's VAAPI + /// device creation uses when given no device string, so a box that gets hardware + /// decode from the FFmpeg rung today gets the same GPU here. + /// + /// ⚠ On a multi-GPU box that is not necessarily the PRESENTER's GPU, and a dmabuf + /// exported from one GPU and imported into another either fails outright or + /// copies. The FFmpeg rung has always had this property; the env pin is the + /// escape hatch, and the chosen node is logged so a field report can name it. + fn open(va: Libva) -> Result { + if let Some(pin) = std::env::var_os("PUNKTFUNK_VAAPI_DEVICE") { + let path = pin.to_string_lossy().into_owned(); + let (display, node, version) = Display::probe(&va, &path) + .with_context(|| format!("PUNKTFUNK_VAAPI_DEVICE={path}"))?; + return Ok(Display { + va, + display, + node: Some(node), + path, + version, + }); + } + let mut nodes: Vec = std::fs::read_dir("/dev/dri") + .context("/dev/dri (no DRM devices on this machine)")? + .filter_map(|e| e.ok()) + .map(|e| e.path()) + .filter(|p| { + p.file_name() + .and_then(|n| n.to_str()) + .is_some_and(|n| n.starts_with("renderD")) + }) + .collect(); + nodes.sort(); + let mut tried: Vec = Vec::new(); + for node in &nodes { + let path = node.to_string_lossy().into_owned(); + match Display::probe(&va, &path) { + Ok((display, node, version)) => { + return Ok(Display { + va, + display, + node: Some(node), + path, + version, + }) + } + Err(e) => { + tracing::debug!(node = %path, reason = %format!("{e:#}"), "not a VAAPI device"); + tried.push(path); + } + } + } + bail!( + "no render node initialised a VAAPI display ({})", + if tried.is_empty() { + "/dev/dri has no renderD* nodes".to_string() + } else { + format!("tried {}", tried.join(", ")) + } + ) + } + + /// Try ONE node, borrowing the loaded library — so a box with several GPUs + /// dlopens libva once rather than once per node, and a failure carries only its + /// reason. + fn probe(va: &Libva, path: &str) -> Result<(VaDisplay, OwnedFd, (c_int, c_int))> { + let node = OwnedFd::from( + std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(path) + .with_context(|| format!("open {path}"))?, + ); + // SAFETY: `vaGetDisplayDRM` takes the render node's fd by value and returns an + // opaque display or null; `vaInitialize` writes the two version ints through + // the out-pointers, which are locals live across the call. The fd stays open in + // `node` for as long as the display exists — libva does not dup it. + unsafe { + let display = (va.get_display_drm)(node.as_raw_fd()); + if display.is_null() { + bail!("vaGetDisplayDRM({path}) returned no display"); + } + let (mut major, mut minor) = (0, 0); + let status = (va.initialize)(display, &mut major, &mut minor); + if status != VA_STATUS_SUCCESS { + let e = va.err("vaInitialize", status); + // The display is unusable but still allocated; terminate it so the + // driver's own state goes with the attempt. + (va.terminate)(display); + // No path in the context: every caller already names the node it + // asked about, and on a box where nothing initialises that printed + // the node twice on every line. + return Err(e); + } + Ok((display, node, (major, minor))) + } + } +} + +impl Display { + /// Does this device decode that profile? + /// + /// Asked BEFORE `vaCreateConfig` so an unsupported profile is a clean refusal + /// naming the profile, not a driver status code — the ladder falls through + /// either way, but only one of them tells a field report why. + fn require_entrypoint(&self, profile: c_int) -> Result<()> { + // SAFETY: `vaMaxNumEntrypoints` returns the array size this display needs; + // the vector is allocated to exactly that and `count` is a local written + // through by the call. + unsafe { + let max = (self.va.max_entrypoints)(self.display); + if max <= 0 { + bail!("vaMaxNumEntrypoints returned {max}"); + } + let mut entrypoints = vec![0 as c_int; max as usize]; + let mut count: c_int = 0; + self.va.check( + "vaQueryConfigEntrypoints", + (self.va.query_config_entrypoints)( + self.display, + profile, + entrypoints.as_mut_ptr(), + &mut count, + ), + )?; + let vld = pf_vaadec::VA_ENTRYPOINT_VLD as c_int; + if !entrypoints[..count.clamp(0, max) as usize].contains(&vld) { + bail!("this device has no VLD decode entrypoint for VAProfile {profile}"); + } + } + Ok(()) + } + + /// `vaCreateBuffer` with the data copied in — libva's documented behaviour for a + /// non-null `data` pointer, and what makes the caller's structs free to die + /// straight after. + fn create_buffer( + &self, + context: VaContextId, + kind: u32, + size: usize, + data: *const c_void, + ) -> Result { + let mut id: VaBufferId = VA_INVALID_ID; + // SAFETY: a live display and context; `data` points at `size` readable bytes + // for the duration of the call (the caller's live struct or slice), and `id` + // is a local written through. libva copies the payload before returning. + self.va.check("vaCreateBuffer", unsafe { + (self.va.create_buffer)( + self.display, + context, + kind as c_uint, + size as c_uint, + 1, + data.cast_mut(), + &mut id, + ) + })?; + Ok(id) + } + + /// Destroy every buffer of a submission. + /// + /// ⚠ Not optional and not automatic. `va.h` is explicit — *"The user must call + /// vaDestroyBuffer() to destroy a buffer"*, and *"a buffer can be re-used and + /// sent to the server by another Begin/Render/End sequence if vaDestroyBuffer() + /// is not called"*. The libva 0.x behaviour where `vaEndPicture` consumed them is + /// long gone; leaking two-plus buffers per picture at 60 fps exhausts the + /// driver's buffer store in minutes. + fn destroy_buffers(&self, buffers: &[VaBufferId]) { + for &b in buffers { + if b == VA_INVALID_ID { + continue; + } + // SAFETY: each id came from `create_buffer` on this display and is + // destroyed exactly once — the submission's list is consumed here. + unsafe { (self.va.destroy_buffer)(self.display, b) }; + } + } +} + +impl Drop for Display { + fn drop(&mut self) { + // SAFETY: `self.display` was initialised in `open_node` and nothing else + // terminates it; `Drop` runs once. The node fd is dropped AFTER this, which is + // the order libva requires — it holds the fd, it does not own it. + unsafe { (self.va.terminate)(self.display) }; + self.node = None; + } +} + +// --------------------------------------------------------------------------- +// The stream shape a session is built for +// --------------------------------------------------------------------------- + +/// Everything about a stream that sizes or configures a session. Any change rebuilds +/// the whole thing: the config, the context, the surface pool and the slot map all +/// derive from it, and a half-rebuilt session hands out surfaces the pool does not +/// have. (M5's review found the depth/chroma half of this missing on the D3D11 rung — +/// the Windows host flips an HDR desktop to PQ in-band with a NEW SPS at unchanged +/// size, so a shape keyed on size alone decodes 10-bit samples into an 8-bit pool.) +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct StreamShape { + coded_width: u32, + coded_height: u32, + display_width: u32, + display_height: u32, + max_dpb_frames: usize, + chroma_format_idc: u8, + bit_depth: u8, +} + +/// Which codec, and the planner that plans it. +enum Planner { + H264(Box), + H265(Box), +} + +impl Planner { + fn name(&self) -> &'static str { + match self { + Planner::H264(_) => "native-vaapi h264", + Planner::H265(_) => "native-vaapi h265", + } + } +} + +// --------------------------------------------------------------------------- +// Surface release: the consumer's half of the zero-copy contract +// --------------------------------------------------------------------------- + +/// What a shipped frame hands back when the consumer is done with it. +/// +/// `generation` is what makes a renegotiation safe: a token from a retired pool +/// names a surface index that no longer exists, and freeing that index in the NEW +/// pool would hand a live surface to the decoder as if it were spare. +#[derive(Debug, Clone, Copy)] +struct VaRelease { + surface: usize, + generation: u64, +} + +/// Holds one shipped picture's surface out of the decoder's free list, and owns the +/// fds exported for it. +/// +/// The presenter DUPS every fd it imports (`pf-presenter`'s dmabuf import says so in +/// as many words) and drops the frame — and so this guard — only after the fence for +/// its sampling submission has been waited. So "guard dropped" means "the GPU is done +/// reading", which is exactly when the surface may be decoded into again. +pub struct VaFrameGuard { + /// The exported PRIME fds, closed by this field's own drop. One per OBJECT, not + /// per plane: several planes routinely name one object, and closing a shared fd + /// twice would close an unrelated file. + _fds: Vec, + tx: mpsc::Sender, + release: VaRelease, +} + +impl Drop for VaFrameGuard { + fn drop(&mut self) { + // A dead channel means the decoder is gone; there is nothing to release to. + let _ = self.tx.send(self.release); + } +} + +// --------------------------------------------------------------------------- +// The session +// --------------------------------------------------------------------------- + +/// The live config, context and surface pool for one [`StreamShape`]. +struct Session { + shape: StreamShape, + config: VaConfigId, + context: VaContextId, + /// The pool. Indices into this are what everything else here refers to. + surfaces: Vec, + /// A consumer holds this surface. Cleared when its release token returns. + held: Vec, + /// DPB slot → pool index, rebound at ACTIVATION (module docs). `None` for a slot + /// holding no picture. + slot_surface: Vec>, + /// Decoded pictures the planner has not output yet, `(PicId, pool index)`. + /// Separate from the slot binding because the two end at different times: a + /// non-reference picture leaves the DPB immediately but still owes an output. + pending: Vec<(u64, usize)>, + slots: pf_vaadec::SlotMap, + /// The surface fourcc the pool was created with (NV12 or P010). + fourcc: u32, + /// Bumped on every rebuild; stamped into release tokens. + generation: u64, +} + +impl Session { + /// A surface bound to no live picture, owed no output, and held by no consumer. + /// + /// All three conditions, because they end at different moments: a picture leaves + /// the DPB when the planner removes it, stops being pending when it is output, + /// and stops being held when the presenter's fence has been waited — and the + /// display is usually the LAST of the three. + fn free_surface(&self) -> Option { + (0..self.surfaces.len()).find(|i| { + !self.held[*i] + && !self.slot_surface.contains(&Some(*i)) + && !self.pending.iter().any(|(_, p)| p == i) + }) + } + + /// Re-derive the slot bindings from the ledger: a slot the planner released + /// binds nothing. + /// + /// Done by reading the ledger rather than by tracking `removed` here, so there is + /// exactly one source of truth for which slots are live. `plan_to_va` has already + /// applied this AU's removals by the time it returns. + fn sync_slot_bindings(&mut self) { + let mut live = vec![false; self.slot_surface.len()]; + for (slot, _) in self.slots.held() { + if let Some(l) = live.get_mut(usize::from(slot)) { + *l = true; + } + } + for (slot, bound) in self.slot_surface.iter_mut().enumerate() { + if !live[slot] { + *bound = None; + } + } + } + + /// Slot → `VASurfaceID`, for the pictures the DPB holds RIGHT NOW. + /// + /// Built before the conversion, because references resolve against the + /// pre-removal state. An unbound slot reads [`VA_INVALID_ID`], never 0 — a zero + /// there is a plausible surface id, and the conversion only ever indexes slots + /// the ledger says are live, so the sentinel exists to make a bug in that + /// argument visible rather than silent. + fn surface_table(&self) -> Vec { + self.slot_surface + .iter() + .map(|b| b.map_or(VA_INVALID_ID, |i| self.surfaces[i])) + .collect() + } + + /// Release every libva object this session owns, in creation-reverse order. + /// Called explicitly (a `Drop` here could not reach the display). + fn destroy(mut self, d: &Display) { + // SAFETY: every handle was created on this display by `build` and is + // destroyed exactly once — `destroy` consumes `self`. Surfaces are freed + // after the context that referenced them, which is the order libva documents. + unsafe { + (d.va.destroy_context)(d.display, self.context); + (d.va.destroy_surfaces)( + d.display, + self.surfaces.as_mut_ptr(), + self.surfaces.len() as c_int, + ); + (d.va.destroy_config)(d.display, self.config); + } + } + + /// Build a config, a surface pool and a context for one stream shape. + fn build(d: &Display, codec: pf_vaadec::Codec, shape: StreamShape) -> Result { + let profile = pf_vaadec::profile_for(codec, shape.chroma_format_idc, shape.bit_depth) + .map_err(|e| anyhow!("{e}"))?; + let rt_format = pf_vaadec::rt_format(shape.chroma_format_idc, shape.bit_depth) + .map_err(|e| anyhow!("{e}"))?; + let fourcc = match shape.bit_depth { + 8 => pf_vaadec::VA_FOURCC_NV12, + 10 => pf_vaadec::VA_FOURCC_P010, + other => bail!("no VAAPI surface format for {other}-bit output"), + }; + d.require_entrypoint(profile.value)?; + + // `VAConfigAttribRTFormat` (= 0, measured) is set explicitly rather than left + // to the driver's default: on a Main 10 stream the default is the 8-bit + // format, and a decoder writing 10-bit samples into an 8-bit surface is the + // silent-narrowing failure this program refuses everywhere else. + let mut attrib = VaConfigAttrib { + kind: VA_CONFIG_ATTRIB_RT_FORMAT, + value: rt_format, + }; + let mut config: VaConfigId = VA_INVALID_ID; + // SAFETY: a live display; `attrib` and `config` are locals that outlive the + // call, and the count matches the slice length. + d.va.check("vaCreateConfig", unsafe { + (d.va.create_config)( + d.display, + profile.value, + pf_vaadec::VA_ENTRYPOINT_VLD as c_int, + (&mut attrib as *mut VaConfigAttrib).cast::(), + 1, + &mut config, + ) + })?; + + // From here every early return must destroy what has been created, so the + // fallible tail is written as a closure and unwound once. + let built = (|| -> Result { + let count = pf_vaadec::surface_count(shape.max_dpb_frames); + let mut surfaces: Vec = vec![VA_INVALID_ID; count]; + let mut pixel = VaSurfaceAttrib { + kind: VA_SURFACE_ATTRIB_PIXEL_FORMAT, + flags: VA_SURFACE_ATTRIB_SETTABLE, + value: VaGenericValue { + kind: VA_GENERIC_VALUE_TYPE_INTEGER, + _pad: 0, + // The fourcc is an i32 in libva's integer arm; the top bit is + // clear for every fourcc here, so the cast is value-preserving. + i: fourcc as i32, + _rest: 0, + }, + }; + // Surfaces are allocated at the CODED size. The conformance window is a + // display-time crop, and a pool sized to the display region would be + // short by the codec's granule padding — the scar that smears rows. + // SAFETY: live display; the surface array and the attribute are locals + // that outlive the call and the counts match their lengths. + d.va.check("vaCreateSurfaces", unsafe { + (d.va.create_surfaces)( + d.display, + rt_format, + shape.coded_width, + shape.coded_height, + surfaces.as_mut_ptr(), + count as c_uint, + &mut pixel, + 1, + ) + })?; + + let mut context: VaContextId = VA_INVALID_ID; + // SAFETY: live display and the config/surfaces just created; `context` is + // a local that outlives the call. libva copies the surface array. + let status = unsafe { + (d.va.create_context)( + d.display, + config, + shape.coded_width as c_int, + shape.coded_height as c_int, + VA_PROGRESSIVE as c_int, + surfaces.as_mut_ptr(), + count as c_int, + &mut context, + ) + }; + if let Err(e) = d.va.check("vaCreateContext", status) { + // SAFETY: destroying the surfaces this closure just created, on the + // unwind path, before they are moved into a Session. + unsafe { + (d.va.destroy_surfaces)( + d.display, + surfaces.as_mut_ptr(), + surfaces.len() as c_int, + ) + }; + return Err(e); + } + + let slots = pf_vaadec::SlotMap::new(shape.max_dpb_frames); + let slot_count = slots.capacity(); + tracing::info!( + node = %d.path, + va = format_args!("{}.{}", d.version.0, d.version.1), + profile = profile.name, + coded = format_args!("{}x{}", shape.coded_width, shape.coded_height), + display = format_args!("{}x{}", shape.display_width, shape.display_height), + bit_depth = shape.bit_depth, + surfaces = count, + dpb_slots = slot_count, + "native VAAPI decode session built" + ); + Ok(Session { + shape, + config, + context, + surfaces, + held: vec![false; count], + slot_surface: vec![None; slot_count], + pending: Vec::new(), + slots, + fourcc, + generation: 0, + }) + })(); + if built.is_err() { + // SAFETY: destroying the config created above, on the unwind path; no + // Session took ownership of it. + unsafe { (d.va.destroy_config)(d.display, config) }; + } + built + } +} + +/// `VAConfigAttrib` — 8 bytes, `{type, value}` at 0 and 4 (measured). +#[repr(C)] +#[derive(Clone, Copy)] +struct VaConfigAttrib { + kind: c_int, + value: c_uint, +} + +/// `VAConfigAttribRTFormat` — measured, and 0 is a real enumerator here rather than +/// a "left unset", which is why it is named. +const VA_CONFIG_ATTRIB_RT_FORMAT: c_int = 0; + +/// `VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2` — the export memory type that yields a +/// [`pf_vaadec::VaDrmPrimeSurfaceDescriptor`]. Measured; the older +/// `..._DRM_PRIME` (0x2000_0000) hands back a different, smaller structure. +const VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2: c_uint = 0x4000_0000; + +const _: () = { + assert!(size_of::() == 8); + assert!(std::mem::offset_of!(VaConfigAttrib, value) == 4); +}; + +// --------------------------------------------------------------------------- +// The decoder +// --------------------------------------------------------------------------- + +/// The native VAAPI rung. +pub(crate) struct NativeVaapiDecoder { + display: Display, + planner: Planner, + session: Option, + health: DecodeHealth, + /// A concealed AU asks the pump for a re-anchor, through the same one throttle + /// every other ask uses. Drained by [`Self::take_recovery_request`]. + recovery_request: bool, + generation: u64, + release_tx: mpsc::Sender, + release_rx: mpsc::Receiver, + /// Pool-index releases that arrived for a RETIRED generation, so the count of + /// what is still outstanding is honest in the log. + stale_releases: u64, +} + +impl NativeVaapiDecoder { + /// Build the rung, refusing anything this device or this crate cannot decode + /// BEFORE the ladder commits to it. + /// + /// The refusal is at construction on purpose, and it is M3 WP-2's lesson: a + /// backend that accepts a session and then refuses its first access unit has + /// already cost the ladder its fall-through — the refusal arrives as a decode + /// error, burns the demotion streak, and lands the session on a rung far below + /// the one it would have had. So the negotiated [`StreamFormat`] is probed here, + /// where "no" simply means the next rung is tried. + pub(crate) fn new(codec: pf_vaadec::Codec, stream: StreamFormat) -> Result { + let depth = stream.bit_depth; + pf_vaadec::profile_for(codec, stream.chroma_format_idc, depth) + .map_err(|e| anyhow!("{e}")) + .context("the negotiated stream shape has no VAAPI decode profile")?; + let va = Libva::load().context("libva")?; + let display = Display::open(va)?; + let planner = match codec { + pf_vaadec::Codec::H264 => Planner::H264(Box::new(pf_vaadec::H264Planner::new())), + pf_vaadec::Codec::H265 => Planner::H265(Box::new(pf_vaadec::H265Planner::new())), + }; + let (release_tx, release_rx) = mpsc::channel(); + Ok(NativeVaapiDecoder { + display, + planner, + session: None, + health: DecodeHealth { + // VAAPI has no per-picture decode-status query — there is no + // counterpart to Vulkan's `RESULT_STATUS_ONLY`, exactly as on + // D3D11VA. Saying so is what keeps "clean" and "unmeasured" + // distinguishable on the stats line: `failed` is structurally 0 + // here, and `DecodeHealth::note` enforces that rather than trusting + // this rung to never pass a verdict it cannot have. + status_queries: false, + ..DecodeHealth::default() + }, + recovery_request: false, + generation: 0, + release_tx, + release_rx, + stale_releases: 0, + }) + } + + pub(crate) fn name(&self) -> &'static str { + self.planner.name() + } + + pub(crate) fn health(&self) -> DecodeHealth { + self.health + } + + /// Drain the re-anchor request a concealed AU raised. + pub(crate) fn take_recovery_request(&mut self) -> bool { + std::mem::take(&mut self.recovery_request) + } + + /// Return surfaces the consumer has finished with to the free list. + fn drain_releases(&mut self) { + drain_releases_into( + &self.release_rx, + self.session.as_mut(), + &mut self.stale_releases, + ); + } + + /// Decode one access unit. + /// + /// `Ok(None)` means "no picture from this AU", and covers three different + /// things, deliberately none of them errors: + /// + /// * the planner output nothing yet (reordering, or the very first AUs); + /// * the picture was CONCEALED — an integrity warning says a reference was + /// substituted, so the output is released unshown, [`DecodeHealth::damaged`] + /// records it and a re-anchor is requested through the pump's one throttle. + /// Not an error, because three errors in a second demote the rung on exactly + /// the lossy links it exists to diagnose, where an FFmpeg rung conceals the + /// same event silently and keeps its job; + /// * an HEVC RASL picture skipped after an open-GOP join. `PlanError::RaslSkipped` + /// is the spec's own answer (8.1.3 NOTE) and must NEVER reach the reanchor + /// path — mapping it to an error would make every open-GOP join beg the host + /// for a keyframe it has no reason to send. + pub(crate) fn decode(&mut self, au: &[u8]) -> Result> { + self.drain_releases(); + let result = if matches!(self.planner, Planner::H264(_)) { + self.decode_h264(au) + } else { + self.decode_h265(au) + }; + // ONE verdict per access unit, folded here and nowhere else. Damage is + // reported by the codec arm rather than counted inside it, so a failure + // AFTER a clean plan (a submission, an export) is a refusal and only a + // refusal — not a clean AU that also refused, which would reset the run + // counter a support engineer reads first. + match &result { + Ok((_, damaged)) => self.health.note(*damaged, false, 0), + Err(_) => self.health.note(false, true, 0), + } + result.map(|(frame, _)| frame) + } + + fn decode_h264(&mut self, au: &[u8]) -> Result<(Option, bool)> { + let plan = match &mut self.planner { + Planner::H264(p) => p.plan_au(au).map_err(|e| anyhow!("{e:?}"))?, + Planner::H265(_) => unreachable!("dispatched on the planner's own arm"), + }; + let shape = shape_of( + plan.picture.coded_width, + plan.picture.coded_height, + plan.picture.display_crop, + plan.picture.max_dpb_frames, + plan.picture.chroma_format_idc, + 8 + plan.picture.bit_depth_luma_minus8, + )?; + let damaged = plan.warnings.iter().any(pf_vaadec::is_integrity_warning); + if !plan.warnings.is_empty() { + tracing::debug!(warnings = ?plan.warnings, damaged, "native VAAPI plan warnings"); + } + + let Self { + display, session, .. + } = self; + let s = ensure_session( + display, + session, + pf_vaadec::Codec::H264, + shape, + &mut self.generation, + )?; + let free = s + .free_surface() + .ok_or_else(|| anyhow!("surface pool exhausted ({} surfaces)", s.surfaces.len()))?; + let target = s.surfaces[free]; + let table = s.surface_table(); + let converted = pf_vaadec::plan_to_va(&plan, au, &mut s.slots, &table, target) + .map_err(|e| anyhow!("{e}"))?; + + bind_setup(s, plan.dpb.stored, free); + + let iq = Some(as_ptr(&converted.iq_matrix)); + let slice_ptrs: Vec<(*const c_void, usize)> = converted.slices.iter().map(as_ptr).collect(); + submit( + display, + s, + target, + as_ptr(&converted.pic_params), + iq, + &slice_ptrs, + &converted.slice_data, + au, + )?; + + let frame = finish( + display, + s, + &plan.dpb.outputs, + &plan.dpb.removed, + damaged, + plan.picture.is_idr, + colour_of(&plan.picture.colour), + &mut self.recovery_request, + &self.release_tx, + )?; + Ok((frame, damaged)) + } + + fn decode_h265(&mut self, au: &[u8]) -> Result<(Option, bool)> { + let plan = match &mut self.planner { + Planner::H265(p) => match p.plan_au(au) { + Ok(plan) => plan, + // The contract pf-bitstream's h265 module docs record for this + // wiring: a skipped RASL picture is an Ok-skip, never an error and + // never a re-anchor. See [`Self::decode`]. + Err(pf_vaadec::PlanErrorH265::RaslSkipped { .. }) => return Ok((None, false)), + Err(e) => return Err(anyhow!("{e:?}")), + }, + Planner::H264(_) => unreachable!("dispatched on the planner's own arm"), + }; + let shape = shape_of( + plan.picture.coded_width, + plan.picture.coded_height, + plan.picture.display_crop, + plan.picture.max_dpb_frames, + plan.picture.chroma_format_idc, + 8 + plan.picture.bit_depth_luma_minus8, + )?; + let damaged = plan + .warnings + .iter() + .any(pf_vaadec::is_integrity_warning_h265); + if !plan.warnings.is_empty() { + tracing::debug!(warnings = ?plan.warnings, damaged, "native VAAPI plan warnings"); + } + + let Self { + display, session, .. + } = self; + let s = ensure_session( + display, + session, + pf_vaadec::Codec::H265, + shape, + &mut self.generation, + )?; + let free = s + .free_surface() + .ok_or_else(|| anyhow!("surface pool exhausted ({} surfaces)", s.surfaces.len()))?; + let target = s.surfaces[free]; + let table = s.surface_table(); + let converted = pf_vaadec::plan_to_va_h265(&plan, au, &mut s.slots, &table, target) + .map_err(|e| anyhow!("{e}"))?; + + bind_setup(s, plan.dpb.stored, free); + + // The IQ matrix is submitted ONLY where the sequence codes scaling lists. + // Handing the driver an all-zero matrix on a "use the defaults" stream is + // not a harmless extra buffer: the driver must apply what it is given, every + // residual dequantises to zero, and the picture drifts to flat prediction. + // (M5's review caught exactly this on the DXVA rung, where the buffer was + // unconditional. The conversion answers `None` here so the rung cannot.) + let iq = converted.iq_matrix.as_ref().map(as_ptr); + let slice_ptrs: Vec<(*const c_void, usize)> = converted.slices.iter().map(as_ptr).collect(); + submit( + display, + s, + target, + as_ptr(&converted.pic_params), + iq, + &slice_ptrs, + &converted.slice_data, + au, + )?; + + let frame = finish( + display, + s, + &plan.dpb.outputs, + &plan.dpb.removed, + damaged, + plan.picture.is_idr, + colour_of(&plan.picture.colour), + &mut self.recovery_request, + &self.release_tx, + )?; + Ok((frame, damaged)) + } +} + +impl Drop for NativeVaapiDecoder { + fn drop(&mut self) { + if self.stale_releases > 0 { + // Not an error — a renegotiated session's frames come home to a pool + // that no longer exists — but a count worth seeing, because the only + // other thing that produces it is release bookkeeping gone wrong. + tracing::debug!( + count = self.stale_releases, + "native VAAPI: releases for retired surface pools" + ); + } + if let Some(s) = self.session.take() { + s.destroy(&self.display); + } + } +} + +// --------------------------------------------------------------------------- +// Shared decode plumbing (free functions: the two codec arms differ only in the +// types their conversions produce, and splitting the borrow of `self` here is +// what lets one implementation serve both) +// --------------------------------------------------------------------------- + +/// Apply the consumer's finished-with tokens to the pool's holds. +/// +/// A free function rather than a method so the rule can be tested without a +/// `Display`: it is pure bookkeeping, and the one thing it must get right — refusing +/// a token from a pool that no longer exists — is precisely what a test can check and +/// hardware cannot. +fn drain_releases_into( + rx: &mpsc::Receiver, + mut session: Option<&mut Session>, + stale: &mut u64, +) { + while let Ok(token) = rx.try_recv() { + let Some(s) = session.as_deref_mut() else { + *stale += 1; + continue; + }; + if token.generation != s.generation { + // A retired pool's surface. Its whole session is gone, so there is + // nothing to free — and freeing that index in the CURRENT pool would hand + // a live surface to the decoder as a spare. + *stale += 1; + continue; + } + match s.held.get_mut(token.surface) { + Some(h) => *h = false, + None => *stale += 1, + } + } +} + +/// A live struct as a `(pointer, size)` for `vaCreateBuffer`, which COPIES. +fn as_ptr(value: &T) -> (*const c_void, usize) { + ((value as *const T).cast::(), size_of::()) +} + +/// H.273 code points straight off the picture's ACTIVE SPS/VUI — per frame, never +/// latched, because the Windows host switches an HDR desktop to PQ/BT.2020 IN-BAND +/// with a new SPS while the Welcome still says SDR. +fn colour_of(c: &pf_vaadec::ColourDescription) -> ColorDesc { + ColorDesc { + primaries: c.colour_primaries, + transfer: c.transfer_characteristics, + matrix: c.matrix_coefficients, + full_range: c.video_full_range, + } +} + +/// Derive the session shape, refusing what the hand-off cannot express. +fn shape_of( + coded_width: u32, + coded_height: u32, + crop: pf_vaadec::DisplayCrop, + max_dpb_frames: usize, + chroma_format_idc: u8, + bit_depth: u8, +) -> Result { + // A non-zero conformance-window ORIGIN would have to shift every plane's offset, + // and nothing downstream carries one: the dmabuf planes are handed over with the + // driver's own offsets and the consumer samples from (0,0). Refused rather than + // cropped from the wrong corner — the same latent gap M5 flagged on the D3D11 + // rung, closed here instead of inherited. Our hosts emit origin (0,0); a stream + // that does not simply falls through to the next rung. + if crop.x != 0 || crop.y != 0 { + bail!( + "conformance window at ({}, {}) — this rung hands the surface over \ + uncropped and cannot express a non-zero origin", + crop.x, + crop.y + ); + } + Ok(StreamShape { + coded_width, + coded_height, + display_width: crop.width, + display_height: crop.height, + max_dpb_frames, + chroma_format_idc, + bit_depth, + }) +} + +/// The session for this shape, rebuilt whole if the stream renegotiated. +fn ensure_session<'a>( + d: &Display, + slot: &'a mut Option, + codec: pf_vaadec::Codec, + shape: StreamShape, + generation: &mut u64, +) -> Result<&'a mut Session> { + if slot.as_ref().is_some_and(|s| s.shape == shape) { + return Ok(slot.as_mut().expect("just matched")); + } + if let Some(old) = slot.take() { + tracing::info!( + from = ?old.shape, + to = ?shape, + "native VAAPI stream renegotiated — rebuilding the session" + ); + // Dropped BEFORE the replacement is built so the old pool's video memory is + // released first; a 4K pool is on the order of a hundred megabytes. + // + // Surfaces the CONSUMER still holds are destroyed here too, and that is + // sound: an exported PRIME fd holds its own reference on the underlying + // buffer object, and the presenter dup'd every fd it imported. The pixels + // outlive the VASurface — which is the whole mechanism zero-copy rests on. + old.destroy(d); + } + // A NEW generation, always — this is what makes those outstanding frames safe to + // let go of. Their release tokens name surface indices in a pool that no longer + // exists, and applying one to the new pool would mark a live surface free and + // hand it to the decoder as a spare. The bump is here, at the one place a pool is + // ever replaced, rather than at the call sites. + *generation += 1; + let mut built = Session::build(d, codec, shape)?; + built.generation = *generation; + Ok(slot.insert(built)) +} + +/// Record which surface holds the picture just planned. +/// +/// The slot bindings are re-derived from the ledger FIRST — the conversion has +/// already applied this AU's removals, so a slot the planner released binds nothing +/// — and only then is the setup picture bound, by asking the ledger where it landed. +/// Asking rather than assuming is what handles the one awkward case: a non-reference +/// picture with no free frame buffer is stored and evicted inside a single plan, so +/// it holds NO slot when the conversion returns. Its surface is kept out of the free +/// list by `pending` instead, until it has been output. +fn bind_setup(s: &mut Session, stored: Option, surface: usize) { + s.sync_slot_bindings(); + if let Some(id) = stored { + if let Some(slot) = s.slots.slot_of(id) { + s.slot_surface[usize::from(slot)] = Some(surface); + } + s.pending.push((id, surface)); + } +} + +/// One picture's buffers, in the order libavcodec's VAAPI path submits them: the +/// parameter buffers in one `vaRenderPicture`, then the interleaved +/// slice-parameter/slice-data pairs in another. Matching the path drivers are +/// validated against is worth more than any tidier arrangement. +#[allow(clippy::too_many_arguments)] +fn submit( + d: &Display, + s: &Session, + target: VaSurfaceId, + pic: (*const c_void, usize), + iq: Option<(*const c_void, usize)>, + slices: &[(*const c_void, usize)], + slice_data: &[std::ops::Range], + au: &[u8], +) -> Result<()> { + if slices.len() != slice_data.len() { + bail!( + "{} slice records for {} data ranges", + slices.len(), + slice_data.len() + ); + } + let mut params: Vec = Vec::with_capacity(2); + let mut slice_buffers: Vec = Vec::with_capacity(slices.len() * 2); + // A picture that was BEGUN must be ended even if a step in between failed, or + // the context stays mid-picture and every later `vaBeginPicture` fails on a + // stream that was otherwise recoverable. libavcodec's VAAPI path has the same + // `fail_with_picture` label for the same reason. + let mut begun = false; + // Every buffer created below must be destroyed whatever happens next — libva + // does not reclaim them at `vaEndPicture` (see `Display::destroy_buffers`). + let result = (|| -> Result<()> { + params.push( + d.create_buffer( + s.context, + pf_vaadec::va::VA_PICTURE_PARAMETER_BUFFER_TYPE, + pic.1, + pic.0, + ) + .context("picture parameter buffer")?, + ); + if let Some((ptr, size)) = iq { + params.push( + d.create_buffer( + s.context, + pf_vaadec::va::VA_IQ_MATRIX_BUFFER_TYPE, + size, + ptr, + ) + .context("IQ matrix buffer")?, + ); + } + for (n, ((ptr, size), range)) in slices.iter().zip(slice_data).enumerate() { + let data = au.get(range.clone()).ok_or_else(|| { + anyhow!( + "slice {n}: range {range:?} is outside a {}-byte access unit", + au.len() + ) + })?; + slice_buffers.push( + d.create_buffer( + s.context, + pf_vaadec::va::VA_SLICE_PARAMETER_BUFFER_TYPE, + *size, + *ptr, + ) + .with_context(|| format!("slice {n} parameter buffer"))?, + ); + slice_buffers.push( + d.create_buffer( + s.context, + pf_vaadec::va::VA_SLICE_DATA_BUFFER_TYPE, + data.len(), + data.as_ptr().cast::(), + ) + .with_context(|| format!("slice {n} data buffer"))?, + ); + } + + // SAFETY: a live display, context and target surface; both buffer arrays are + // locals that outlive their calls and their counts match their lengths. + unsafe { + d.va.check( + "vaBeginPicture", + (d.va.begin_picture)(d.display, s.context, target), + )?; + begun = true; + d.va.check( + "vaRenderPicture(parameters)", + (d.va.render_picture)( + d.display, + s.context, + params.as_mut_ptr(), + params.len() as c_int, + ), + )?; + d.va.check( + "vaRenderPicture(slices)", + (d.va.render_picture)( + d.display, + s.context, + slice_buffers.as_mut_ptr(), + slice_buffers.len() as c_int, + ), + )?; + begun = false; + d.va.check("vaEndPicture", (d.va.end_picture)(d.display, s.context))?; + } + Ok(()) + })(); + if begun { + // SAFETY: a live display and context with a picture open; the status is + // deliberately discarded — the real failure is `result`, and reporting this + // one would replace the cause with its consequence. + unsafe { (d.va.end_picture)(d.display, s.context) }; + } + d.destroy_buffers(¶ms); + d.destroy_buffers(&slice_buffers); + result +} + +/// Turn this AU's OUTPUT list into at most one shipped frame. +/// +/// Display order, not decode order. `plan.dpb.outputs` is what the planner says is +/// ready to be shown and in what order, and the surface for each is looked up by +/// `PicId` — so a reordering stream presents correctly rather than in the order the +/// pictures happened to decode. (The native D3D11VA rung does present in decode +/// order; that is a known finding on a rung that blits its output away, and there +/// was no reason to inherit it here where the display-order queue costs a lookup.) +/// +/// Newest wins, which is the same rule the FFmpeg VAAPI rung applies inside its +/// receive loop: on a live stream a picture already superseded is not worth a frame +/// interval. Superseded outputs are released rather than exported. +/// +/// The retirement rule is `pf_vkdecode`'s `settle_dpb`, reimplemented here over this +/// rung's flat pending list rather than reasoned out again, because both halves of it +/// are easy to get wrong: +/// +/// * **`removed` retires a pending picture too.** A picture can leave the DPB without +/// ever being output (`no_output_of_prior_pics` at an IDR is the everyday case), and +/// a pending list that only shrinks on OUTPUT keeps its surface off the free list +/// for the rest of the session — a slow, silent walk into pool exhaustion. +/// * **An output naming no pending picture is a TRACE, not an error.** Ids planned +/// before this decoder existed, or dropped across a session rebuild, are +/// display-order gaps. +#[allow(clippy::too_many_arguments)] +fn finish( + d: &Display, + s: &mut Session, + outputs: &[u64], + removed: &[u64], + damaged: bool, + keyframe: bool, + color: ColorDesc, + recovery_request: &mut bool, + tx: &mpsc::Sender, +) -> Result> { + // A concealed picture is not shown: it was decoded from a substitute reference, + // so shipping it paints the substitution on screen. Nothing this AU output is + // shown, the pump is asked to re-anchor, and the caller records the damage. + let shown = if damaged { + None + } else { + outputs.last().copied() + }; + // OUTPUTS FIRST, and the shown one is taken out before anything else runs. + // A picture is normally output and removed by the SAME access unit — that is + // what bumping is — so retiring `removed` before claiming the frame would + // discard the very picture about to be displayed, on essentially every AU. + let claimed = shown.and_then(|id| { + let found = s.pending.iter().position(|(pid, _)| *pid == id); + if found.is_none() { + tracing::trace!(id, "output id without a pending picture"); + } + found.map(|index| s.pending.remove(index).1) + }); + for id in outputs { + if Some(*id) != shown { + s.pending.retain(|(pid, _)| pid != id); + } + } + // Whatever left the DPB is retired from the pending list whether or not it was + // ever output. Its SURFACE only becomes free if nothing else holds it — a + // reference still bound to a slot, or a frame the consumer has, stays put. + for id in removed { + s.pending.retain(|(pid, _)| pid != id); + } + if damaged { + *recovery_request = true; + return Ok(None); + } + let Some(surface_index) = claimed else { + return Ok(None); + }; + let surface = s.surfaces[surface_index]; + + // OWNED from here. `export` wraps the descriptor's fds the moment the call + // succeeds, so every refusal below closes them by dropping rather than by + // remembering to — an earlier draft leaked one fd per refused frame. + let (exported, fds) = export(d, surface)?; + if exported.fourcc != s.fourcc { + // The pool was created with an explicit pixel format; a surface exporting a + // different one means the driver silently substituted, and the consumer + // would import the wrong layout. + bail!( + "surface exported fourcc {:#010x}, the pool was created as {:#010x}", + exported.fourcc, + s.fourcc + ); + } + if exported.planes.len() < 2 { + bail!( + "a two-plane surface exported {} plane(s) — the chroma is missing", + exported.planes.len() + ); + } + + s.held[surface_index] = true; + let planes = exported + .planes + .iter() + .map(|p| DmabufPlane { + fd: p.fd, + offset: p.offset, + stride: p.stride, + }) + .collect(); + Ok(Some(DmabufFrame { + // The DISPLAY region. The surface is allocated at the coded size and is + // taller/wider than the picture; handing over the coded size would show the + // codec's granule padding. + width: s.shape.display_width, + height: s.shape.display_height, + fourcc: exported.fourcc, + modifier: exported.modifier, + planes, + color, + keyframe, + guard: DrmFrameGuard::NativeVa(VaFrameGuard { + _fds: fds, + tx: tx.clone(), + release: VaRelease { + surface: surface_index, + generation: s.generation, + }, + }), + })) +} + +/// Wait for the decode and export the surface as DRM-PRIME dmabufs. +/// +/// The `vaSyncSurface` is what makes the hand-off safe: VAAPI exposes no fence to +/// the importer, so the accepted contract on this path — and what libavcodec's own +/// VAAPI→DRM_PRIME mapping does — is to sync before the fds leave. It is a blocking +/// wait on the pump thread, which is worth naming: at 60 fps against decodes of a +/// millisecond or two it is slack, and the alternative is handing the presenter a +/// surface the GPU has not finished writing. +/// +/// Returns the flattened surface AND the fds it owns, together — so that from the +/// instant the export succeeds those fds are RAII-owned and every later refusal +/// closes them by dropping. `ExportedSurface::planes` still carries the raw fds, +/// borrowed from these: several planes routinely name one object, and each object's +/// fd must be closed exactly once. +fn export(d: &Display, surface: VaSurfaceId) -> Result<(pf_vaadec::ExportedSurface, Vec)> { + // SAFETY: a live display and a surface from its own pool. + d.va.check("vaSyncSurface", unsafe { + (d.va.sync_surface)(d.display, surface) + })?; + + let mut desc = pf_vaadec::VaDrmPrimeSurfaceDescriptor::zeroed(); + // SAFETY: a live display and surface; `desc` is a local of exactly the layout + // `VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2` writes (measured by + // `pf-vaadec/layout-probe.c` and compile-asserted), and it outlives the call. + d.va.check("vaExportSurfaceHandle", unsafe { + (d.va.export_surface_handle)( + d.display, + surface, + VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2, + pf_vaadec::VA_EXPORT_SURFACE_SEPARATE_LAYERS | pf_vaadec::VA_EXPORT_SURFACE_READ_ONLY, + (&mut desc as *mut pf_vaadec::VaDrmPrimeSurfaceDescriptor).cast::(), + ) + })?; + + match pf_vaadec::flatten(&desc) { + Ok(exported) => { + let fds = exported + .object_fds + .iter() + // SAFETY: each fd came out of a successful `vaExportSurfaceHandle` + // and is owned by this process exactly once. `flatten` lists one per + // OBJECT, so no fd is wrapped twice even where planes share it. + .map(|fd| unsafe { OwnedFd::from_raw_fd(*fd) }) + .collect(); + Ok((exported, fds)) + } + Err(e) => { + // The export SUCCEEDED, so its fds belong to this process even though the + // descriptor cannot be read as a surface. Every writable slot is swept + // rather than the first `num_objects` — a refusal for a bogus + // `num_objects` is exactly the case where that count cannot be trusted to + // bound anything. + for object in &desc.objects { + if object.fd >= 0 { + // SAFETY: an fd this process owns from the successful export; + // wrapping it in an `OwnedFd` that immediately drops closes it once. + drop(unsafe { OwnedFd::from_raw_fd(object.fd) }); + } + } + Err(anyhow!("{e}")) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A session whose libva handles are never used — every field exercised below is + /// plain bookkeeping, which is exactly why this rule can be tested at all without + /// a GPU. The pool is deliberately small so an exhausted free list is reachable. + fn session(surfaces: usize, slots: usize) -> Session { + Session { + shape: StreamShape { + coded_width: 64, + coded_height: 64, + display_width: 64, + display_height: 64, + max_dpb_frames: slots - 1, + chroma_format_idc: 1, + bit_depth: 8, + }, + config: VA_INVALID_ID, + context: VA_INVALID_ID, + surfaces: (0..surfaces as u32).map(|i| 0x100 + i).collect(), + held: vec![false; surfaces], + slot_surface: vec![None; slots], + pending: Vec::new(), + slots: pf_vaadec::SlotMap::new(slots - 1), + fourcc: pf_vaadec::VA_FOURCC_NV12, + generation: 1, + } + } + + /// The whole rule, in one test: a surface is free only when NOTHING claims it, + /// and the three claims end at different moments. + #[test] + fn a_surface_is_free_only_when_no_slot_no_output_and_no_consumer_claims_it() { + let mut s = session(4, 3); + assert_eq!( + s.free_surface(), + Some(0), + "a fresh pool starts at the front" + ); + + // 0: a live DPB reference. 1: decoded, still owing an output. 2: on screen. + s.slot_surface[0] = Some(0); + s.pending.push((7, 1)); + s.held[2] = true; + assert_eq!( + s.free_surface(), + Some(3), + "the first three are each claimed a different way" + ); + + // Losing ONE claim is not enough when another still stands. + s.held[3] = true; + s.slot_surface[1] = Some(1); + assert_eq!( + s.free_surface(), + None, + "an exhausted pool must say so rather than hand out a claimed surface" + ); + + // Surface 1 is both slot-bound and pending: releasing only the output keeps + // it out, and only when the slot goes too does it come back. + s.pending.clear(); + assert_eq!(s.free_surface(), None); + s.slot_surface[1] = None; + assert_eq!(s.free_surface(), Some(1)); + } + + /// The consumer's release is what ends the third claim — and it must be matched + /// to the generation that issued it, or a renegotiation hands a live surface out. + #[test] + fn a_release_from_a_retired_pool_never_frees_a_surface_in_the_new_one() { + let mut s = session(4, 3); + s.held[2] = true; + let (tx, rx) = mpsc::channel(); + let mut stale = 0u64; + + // A token from the pool that was retired before this one. Surface index 2 + // exists in BOTH pools, which is what makes this dangerous: the index is + // valid, and only the generation says it means a different surface. + tx.send(VaRelease { + surface: 2, + generation: 0, + }) + .expect("the receiver is alive"); + drain_releases_into(&rx, Some(&mut s), &mut stale); + assert!( + s.held[2], + "a stale generation must not clear a hold in the CURRENT pool" + ); + assert_eq!(stale, 1, "and it must be counted, not silent"); + + // The matching generation does free it. + tx.send(VaRelease { + surface: 2, + generation: 1, + }) + .expect("the receiver is alive"); + drain_releases_into(&rx, Some(&mut s), &mut stale); + assert!(!s.held[2]); + assert_eq!(stale, 1); + + // An index the pool does not have is counted, never a panic. + tx.send(VaRelease { + surface: 99, + generation: 1, + }) + .expect("the receiver is alive"); + drain_releases_into(&rx, Some(&mut s), &mut stale); + assert_eq!(stale, 2); + } + + /// The bindings follow the ledger: a slot the planner released binds nothing, + /// and the surface it held is only free if nothing else claims it. + #[test] + fn syncing_bindings_drops_the_slots_the_ledger_no_longer_holds() { + let mut s = session(4, 3); + s.slot_surface[0] = Some(0); + s.slot_surface[1] = Some(1); + s.slots.assign(11).expect("a free slot"); + s.sync_slot_bindings(); + assert_eq!( + s.slot_surface, + vec![Some(0), None, None], + "slot 0 is held by picture 11; slot 1's picture is gone" + ); + } + + /// A conformance window with a non-zero ORIGIN is refused, not cropped from the + /// wrong corner: nothing downstream carries an origin. + #[test] + fn a_non_zero_crop_origin_is_refused() { + let ok = shape_of( + 1920, + 1088, + pf_vaadec::DisplayCrop { + x: 0, + y: 0, + width: 1920, + height: 1080, + }, + 4, + 1, + 8, + ) + .expect("the ordinary 1088-coded 1080 picture"); + assert_eq!((ok.display_width, ok.display_height), (1920, 1080)); + assert_eq!((ok.coded_width, ok.coded_height), (1920, 1088)); + + assert!(shape_of( + 1920, + 1088, + pf_vaadec::DisplayCrop { + x: 8, + y: 0, + width: 1912, + height: 1080, + }, + 4, + 1, + 8, + ) + .is_err()); + } + + /// A shape this rung cannot decode is refused BEFORE libva is even loaded. + /// + /// The ordering is the point, not the refusal. M3 WP-2's review caught the + /// opposite arrangement on the Vulkan rung: a backend that accepts a session and + /// then refuses its first access unit has already cost the ladder its + /// fall-through — the refusal arrives as a decode error, burns the demotion + /// streak, and lands the session several rungs lower than it would have been. + /// Asserting on the MESSAGE is what pins the order: this test passes on a machine + /// with libva and on one without, and only stays passing while the profile probe + /// comes first. + #[test] + fn a_shape_with_no_profile_is_refused_before_libva_is_loaded() { + let e = NativeVaapiDecoder::new( + pf_vaadec::Codec::H264, + StreamFormat { + chroma_format_idc: 3, + bit_depth: 8, + }, + ) + .err() + .expect("4:4:4 H.264 has no VAAPI profile in this rung's envelope"); + let text = format!("{e:#}"); + assert!( + text.contains("profile"), + "the refusal must name the stream shape, not whatever libva said: {text}" + ); + } + + /// On-glass probe: resolve every entry point against the REAL libva on this + /// machine, then say what each render node does. + /// + /// This is the one thing no gate can check. `dlsym` takes a STRING: a mistyped + /// entry point compiles, clippies and unit-tests perfectly and fails only on a + /// machine with libva — so the 19 names are worth exercising once against a real + /// runtime, and this test is how. It is also the honest report of a box's VAAPI + /// situation: a node that will not initialise is a legitimate outcome (NVIDIA has + /// no usable VAAPI driver), printed rather than failed, because the rung's claim + /// is that such a box REFUSES CLEANLY and lets the ladder fall through. + /// + /// `cargo test -p pf-client-core --lib probe_this_machines_libva -- --ignored --nocapture` + #[test] + #[ignore = "needs a machine with a libva runtime"] + fn probe_this_machines_libva() { + let va = match Libva::load() { + Ok(va) => { + eprintln!("libva: every entry point resolved"); + va + } + Err(e) => { + eprintln!("libva: NOT LOADED — {e:#}"); + eprintln!("(this is the clean-refusal path; the ladder falls through here)"); + return; + } + }; + + let mut nodes: Vec = std::fs::read_dir("/dev/dri") + .expect("/dev/dri") + .filter_map(|e| e.ok()) + .map(|e| e.path()) + .filter(|p| { + p.file_name() + .and_then(|n| n.to_str()) + .is_some_and(|n| n.starts_with("renderD")) + }) + .collect(); + nodes.sort(); + eprintln!("render nodes: {nodes:?}"); + + let mut opened = 0; + for node in &nodes { + let path = node.to_string_lossy().into_owned(); + match Display::probe(&va, &path) { + Ok((display, fd, version)) => { + opened += 1; + eprintln!(" {path}: VA-API {}.{}", version.0, version.1); + let d = Display { + va: Libva::load().expect("libva loaded once already"), + display, + node: Some(fd), + path: path.clone(), + version, + }; + for (name, profile) in [ + ("H.264 High", pf_vaadec::config::VA_PROFILE_H264_HIGH), + ("HEVC Main", pf_vaadec::config::VA_PROFILE_HEVC_MAIN), + ("HEVC Main 10", pf_vaadec::config::VA_PROFILE_HEVC_MAIN10), + ] { + match d.require_entrypoint(profile) { + Ok(()) => eprintln!(" {name}: VLD decode"), + Err(e) => eprintln!(" {name}: no ({e})"), + } + } + } + Err(e) => eprintln!(" {path}: {e:#}"), + } + } + eprintln!( + "{opened} of {} node(s) initialised a VAAPI display", + nodes.len() + ); + } +} diff --git a/crates/pf-client-core/src/video_vulkan.rs b/crates/pf-client-core/src/video_vulkan.rs index 3b667483..98fc8e62 100644 --- a/crates/pf-client-core/src/video_vulkan.rs +++ b/crates/pf-client-core/src/video_vulkan.rs @@ -424,7 +424,7 @@ impl VulkanDecoder { coded_height: ((*fc).height.max((*self.frame).height)) as u32, color: ColorDesc::from_raw(self.frame), keyframe: frame_is_keyframe(self.frame), - guard: DrmFrameGuard(clone), + guard: DrmFrameGuard::Av(clone), }) } } diff --git a/crates/pf-presenter/src/run.rs b/crates/pf-presenter/src/run.rs index 718151a9..ebb56d0d 100644 --- a/crates/pf-presenter/src/run.rs +++ b/crates/pf-presenter/src/run.rs @@ -1635,8 +1635,14 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result st.hdr_untonemapped = true; presenter.present(&window, FrameInput::Cpu(&c), overlay_frame.as_ref())? } + // Both VAAPI rungs — libavcodec's and M6's native one — hand over + // the same thing: dmabuf fds plus a plane layout, with the guard + // opaque either way. One arm, therefore, rather than a second copy + // of the import and its failure-streak demotion. They stay separate + // VARIANTS so that everywhere it MATTERS which rung decoded (the + // stats tag) the compiler asks; here it genuinely does not. #[cfg(target_os = "linux")] - DecodedImage::Dmabuf(d) + DecodedImage::Dmabuf(d) | DecodedImage::NativeDmabuf(d) if presenter.supports_dmabuf() && !st.dmabuf_demoted => { st.hdr = d.color.is_pq(); @@ -1673,7 +1679,7 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result } } #[cfg(target_os = "linux")] - DecodedImage::Dmabuf(_) => { + DecodedImage::Dmabuf(_) | DecodedImage::NativeDmabuf(_) => { // No import extensions on this device (or already demoted) — the // pump rebuilds the decoder as software; frames flow again soon. if !st.dmabuf_demoted { diff --git a/crates/pf-vaadec/layout-probe.c b/crates/pf-vaadec/layout-probe.c index 6bff7b91..27357f75 100644 --- a/crates/pf-vaadec/layout-probe.c +++ b/crates/pf-vaadec/layout-probe.c @@ -24,6 +24,7 @@ #include #include #include +#include #define S(t) printf("size %-34s %zu align %zu\n", #t, sizeof(t), _Alignof(t)) #define O(t, f) printf("off %-20s %-28s %zu\n", #t, #f, offsetof(t, f)) @@ -221,5 +222,67 @@ int main(void) { } printf("VA_PADDING_LOW=%d VA_PADDING_MEDIUM=%d\n", VA_PADDING_LOW, VA_PADDING_MEDIUM); + + /* + * The export descriptor. This one is not a buffer we FILL — it is a struct the + * driver WRITES, so a wrong layout is read as plausible garbage (an fd from the + * middle of a pitch, a plane count from a modifier's high word) rather than + * refused. It carries fixed-size arrays whose bounds the flattening walk trusts, + * which is exactly the shape that turned into the green-screen bug once already. + */ + S(VADRMPRIMESurfaceDescriptor); + O(VADRMPRIMESurfaceDescriptor, fourcc); + O(VADRMPRIMESurfaceDescriptor, width); + O(VADRMPRIMESurfaceDescriptor, height); + O(VADRMPRIMESurfaceDescriptor, num_objects); + O(VADRMPRIMESurfaceDescriptor, objects); + O(VADRMPRIMESurfaceDescriptor, num_layers); + O(VADRMPRIMESurfaceDescriptor, layers); + printf("count VADRMPRIMESurfaceDescriptor objects %zu\n", + sizeof(((VADRMPRIMESurfaceDescriptor *)0)->objects) / + sizeof(((VADRMPRIMESurfaceDescriptor *)0)->objects[0])); + printf("count VADRMPRIMESurfaceDescriptor layers %zu\n", + sizeof(((VADRMPRIMESurfaceDescriptor *)0)->layers) / + sizeof(((VADRMPRIMESurfaceDescriptor *)0)->layers[0])); + printf("count layer.object_index %zu\n", + sizeof(((VADRMPRIMESurfaceDescriptor *)0)->layers[0].object_index) / + sizeof(((VADRMPRIMESurfaceDescriptor *)0)->layers[0].object_index[0])); + + /* + * The enumerators the runtime calls pass by value. Printed rather than + * transcribed because two of them are the exact pair this program has already + * been warned about: VASliceParameterBufferType and VASliceDataBufferType are + * 4 and 5, not the 3 and 4 that counting the enum from the top suggests. + */ + printf("enum VAEntrypointVLD %d\n", VAEntrypointVLD); + printf("enum VAConfigAttribRTFormat %d\n", VAConfigAttribRTFormat); + printf("enum VAPictureParameterBufferType %d\n", VAPictureParameterBufferType); + printf("enum VAIQMatrixBufferType %d\n", VAIQMatrixBufferType); + printf("enum VASliceParameterBufferType %d\n", VASliceParameterBufferType); + printf("enum VASliceDataBufferType %d\n", VASliceDataBufferType); + printf("enum VA_EXPORT_SURFACE_READ_ONLY 0x%04x\n", VA_EXPORT_SURFACE_READ_ONLY); + printf("enum VA_EXPORT_SURFACE_SEPARATE_LAYERS 0x%04x\n", + VA_EXPORT_SURFACE_SEPARATE_LAYERS); + printf("enum VA_SURFACE_ATTRIB_SETTABLE 0x%04x\n", VA_SURFACE_ATTRIB_SETTABLE); + printf("enum VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2 0x%08x\n", + VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2); + printf("enum VASurfaceAttribPixelFormat %d\n", VASurfaceAttribPixelFormat); + printf("enum VAGenericValueTypeInteger %d\n", VAGenericValueTypeInteger); + printf("enum VA_STATUS_SUCCESS %d\n", VA_STATUS_SUCCESS); + printf("enum VA_INVALID_ID 0x%08x\n", VA_INVALID_ID); + printf("enum VA_FOURCC_NV12 0x%08x\n", VA_FOURCC_NV12); + printf("enum VA_FOURCC_P010 0x%08x\n", VA_FOURCC_P010); + printf("enum VA_PROGRESSIVE 0x%04x\n", VA_PROGRESSIVE); + + S(VASurfaceAttrib); + O(VASurfaceAttrib, type); + O(VASurfaceAttrib, flags); + O(VASurfaceAttrib, value); + S(VAGenericValue); + O(VAGenericValue, type); + O(VAGenericValue, value); + S(VAConfigAttrib); + O(VAConfigAttrib, type); + O(VAConfigAttrib, value); return 0; } diff --git a/crates/pf-vaadec/src/config.rs b/crates/pf-vaadec/src/config.rs index e13eec94..7aa09678 100644 --- a/crates/pf-vaadec/src/config.rs +++ b/crates/pf-vaadec/src/config.rs @@ -112,11 +112,25 @@ pub fn rt_format(chroma_format_idc: u8, depth: u8) -> Result { } } -/// Headroom over the DPB for pictures the presenter still holds. +/// Headroom over the DPB for pictures the CONSUMER still holds. /// -/// A surface handed to the compositor is not free to decode into, and a pool sized -/// exactly to the DPB stalls the decoder behind the display. -pub const PRESENTER_HEADROOM: usize = 4; +/// A surface handed to the presenter is not free to decode into — that is what +/// zero-copy costs — and a pool sized exactly to the DPB stalls the decoder behind +/// the display. +/// +/// **8, matching `pf_vkdecode::images::HOLD_HEADROOM`, and for its measurement**: +/// the real client pipeline holds roughly four to seven frames at steady state (two +/// bounded(2) channels, the frame store's 1..=3 preroll, the in-flight present and +/// the retired-frame slot), so eight leaves a frame of slack and a consumer holding +/// more than that has earned an honest "pool exhausted" rather than a silent stall. +/// The number was 4 when this module was written against no consumer; the native +/// Vulkan rung had already measured the pipeline by then, and 4 would have run the +/// pool dry on an ordinary stream. +/// +/// (The FFmpeg VAAPI rung asks libavcodec for `extra_hw_frames = 4` and survives on +/// it, but its pool is not this pool: `av_hwframe_get_buffer` BLOCKS until a surface +/// frees, so its headroom buys latency where ours buys correctness.) +pub const PRESENTER_HEADROOM: usize = 8; /// How many decode surfaces a session allocates: the DPB, plus the picture being /// decoded, plus [`PRESENTER_HEADROOM`]. @@ -168,7 +182,19 @@ mod tests { #[test] fn the_surface_pool_covers_dpb_plus_current_plus_headroom() { - assert_eq!(surface_count(4), 9); - assert_eq!(surface_count(16), 21); + assert_eq!(surface_count(4), 4 + 1 + PRESENTER_HEADROOM); + assert_eq!(surface_count(16), 16 + 1 + PRESENTER_HEADROOM); + } + + /// The headroom must cover what the client pipeline actually holds, which the + /// native Vulkan rung measured before this crate existed. Pinning it to that + /// crate's constant means a future re-measurement moves both rungs together + /// instead of leaving this one quietly short. + #[test] + fn the_headroom_matches_the_pipeline_depth_the_vulkan_rung_measured() { + assert_eq!( + PRESENTER_HEADROOM, + pf_vkdecode::images::HOLD_HEADROOM as usize + ); } } diff --git a/crates/pf-vaadec/src/drm.rs b/crates/pf-vaadec/src/drm.rs new file mode 100644 index 00000000..3f83d6ae --- /dev/null +++ b/crates/pf-vaadec/src/drm.rs @@ -0,0 +1,466 @@ +//! The export descriptor — `vaExportSurfaceHandle`'s answer — and the walk that +//! turns it into the plane list a dmabuf import consumes. +//! +//! This is the one structure in the rung that the DRIVER writes and we read. Every +//! other buffer here is one we fill, where a wrong field is at worst refused; a +//! misread descriptor is plausible garbage — an fd taken from the middle of a +//! pitch, a plane count read out of a modifier's high word — and it imports +//! successfully into a texture of nonsense. So the layout is measured by +//! `layout-probe.c` like everything else, and the walk lives here, pure, where +//! macOS and the container run its tests. +//! +//! # The bug this walk exists to not repeat +//! +//! With `VA_EXPORT_SURFACE_SEPARATE_LAYERS` an NV12 surface comes back as **two +//! layers** — an `R8` luma layer and a `GR88` chroma layer, one plane each — not as +//! one two-plane `NV12` layer. Taking `layers[0]` and calling it the surface is how +//! this project once painted the screen green: the importer saw a single-plane R8 +//! texture and the chroma was simply gone. Hence [`flatten`]: every plane of every +//! layer, in declared order, and the surface's format comes from the descriptor's +//! own top-level `fourcc` rather than from any layer's component format. +//! +//! (The alternative, `VA_EXPORT_SURFACE_COMPOSED_LAYERS`, asks the driver for one +//! layer describing the whole surface. It is not universally implemented, and the +//! separate-layers form is what the FFmpeg VAAPI path this rung replaces has always +//! used — so it is the form the fleet's drivers are exercised on.) + +use std::os::raw::c_int; + +/// `VA_EXPORT_SURFACE_READ_ONLY` — the decoder keeps writing this surface's future +/// siblings; the consumer only samples. +pub const VA_EXPORT_SURFACE_READ_ONLY: u32 = 0x0001; + +/// `VA_EXPORT_SURFACE_SEPARATE_LAYERS` — one layer per plane (module docs). +pub const VA_EXPORT_SURFACE_SEPARATE_LAYERS: u32 = 0x0004; + +/// `VA_FOURCC_NV12` — identical to `DRM_FORMAT_NV12`; the two namespaces agree on +/// the packed-fourcc value, which is why the descriptor's `fourcc` can be handed +/// to a DRM importer unchanged. +pub const VA_FOURCC_NV12: u32 = 0x3231_564e; + +/// `VA_FOURCC_P010` — identical to `DRM_FORMAT_P010`. +pub const VA_FOURCC_P010: u32 = 0x3031_3050; + +/// Fixed array bounds in the descriptor, measured (`layout-probe.c`). +pub const MAX_OBJECTS: usize = 4; +pub const MAX_LAYERS: usize = 4; +pub const MAX_PLANES_PER_LAYER: usize = 4; + +/// One buffer object backing the surface: an fd we OWN and must close. +#[repr(C)] +#[derive(Debug, Clone, Copy)] +pub struct VaDrmPrimeObject { + /// DRM PRIME fd. `c_int` because that is what the header says; the caller + /// wraps it in an `OwnedFd` the moment the export succeeds. + pub fd: c_int, + pub size: u32, + pub drm_format_modifier: u64, +} + +/// One layer: under `SEPARATE_LAYERS` this is a single plane with its own +/// component format. +#[repr(C)] +#[derive(Debug, Clone, Copy)] +pub struct VaDrmPrimeLayer { + /// The LAYER's DRM format (`R8`, `GR88`, …) — a component format, never the + /// surface's. See the module docs. + pub drm_format: u32, + pub num_planes: u32, + pub object_index: [u32; MAX_PLANES_PER_LAYER], + pub offset: [u32; MAX_PLANES_PER_LAYER], + pub pitch: [u32; MAX_PLANES_PER_LAYER], +} + +/// `VADRMPRIMESurfaceDescriptor` (`va_drmcommon.h`). +#[repr(C)] +#[derive(Debug, Clone, Copy)] +pub struct VaDrmPrimeSurfaceDescriptor { + /// The SURFACE's fourcc (`VA_FOURCC_NV12`, `VA_FOURCC_P010`, …) — the combined + /// format, and the one a DRM importer wants. + pub fourcc: u32, + pub width: u32, + pub height: u32, + pub num_objects: u32, + pub objects: [VaDrmPrimeObject; MAX_OBJECTS], + pub num_layers: u32, + pub layers: [VaDrmPrimeLayer; MAX_LAYERS], +} + +impl VaDrmPrimeSurfaceDescriptor { + /// A zeroed descriptor for the driver to fill. + /// + /// Zero is not a valid `num_objects`/`num_layers`, so a driver that returns + /// success without writing anything is caught by [`flatten`] rather than read + /// as a surface with no planes. + pub fn zeroed() -> Self { + Self { + fourcc: 0, + width: 0, + height: 0, + num_objects: 0, + objects: [VaDrmPrimeObject { + fd: -1, + size: 0, + drm_format_modifier: 0, + }; MAX_OBJECTS], + num_layers: 0, + layers: [VaDrmPrimeLayer { + drm_format: 0, + num_planes: 0, + object_index: [0; MAX_PLANES_PER_LAYER], + offset: [0; MAX_PLANES_PER_LAYER], + pitch: [0; MAX_PLANES_PER_LAYER], + }; MAX_LAYERS], + } + } +} + +/// One plane of the flattened surface. `fd` is BORROWED from the descriptor's +/// object list — several planes routinely name the same object — so the caller +/// owns the objects and the planes reference them. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ExportedPlane { + pub fd: c_int, + pub offset: u32, + pub stride: u32, +} + +/// The flattened surface: what an importer needs, in plane order. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ExportedSurface { + /// The combined DRM fourcc, from the descriptor's own top-level field. + pub fourcc: u32, + pub width: u32, + pub height: u32, + /// The tiling modifier. Every object must agree on it — see [`flatten`]. + pub modifier: u64, + /// Every plane of every layer, in declared order. + pub planes: Vec, + /// The fds the caller OWNS and must close, one per object. + pub object_fds: Vec, +} + +/// Why a descriptor cannot be read as a surface. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ExportError { + /// Zero (a driver that "succeeded" without writing) or more than the arrays hold. + ObjectCount(u32), + LayerCount(u32), + PlaneCount { + layer: usize, + planes: u32, + }, + /// A plane named an object outside `num_objects` — reading it would take an fd + /// from uninitialised descriptor memory. + ObjectIndex { + layer: usize, + plane: usize, + index: u32, + }, + /// An object came back without a usable fd. + BadFd { + object: usize, + fd: c_int, + }, + /// The objects disagree on the tiling modifier. A dmabuf import takes ONE + /// modifier for the whole image, so importing plane 1 under plane 0's tiling + /// would decode the chroma as if it were laid out some other way. Every fleet + /// driver puts the whole surface in one BO; a driver that does not needs code + /// that does not exist yet, and must say so rather than guess. + MixedModifiers { + first: u64, + other: u64, + }, +} + +impl std::fmt::Display for ExportError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + ExportError::ObjectCount(n) => { + write!( + f, + "descriptor declares {n} objects (want 1..={MAX_OBJECTS})" + ) + } + ExportError::LayerCount(n) => { + write!(f, "descriptor declares {n} layers (want 1..={MAX_LAYERS})") + } + ExportError::PlaneCount { layer, planes } => write!( + f, + "layer {layer} declares {planes} planes (want 1..={MAX_PLANES_PER_LAYER})" + ), + ExportError::ObjectIndex { + layer, + plane, + index, + } => write!( + f, + "layer {layer} plane {plane} names object {index}, which the descriptor \ + does not have" + ), + ExportError::BadFd { object, fd } => { + write!(f, "object {object} exported fd {fd}") + } + ExportError::MixedModifiers { first, other } => write!( + f, + "the surface's objects disagree on tiling ({first:#018x} vs {other:#018x}) — \ + a single-modifier import cannot express it" + ), + } + } +} + +impl std::error::Error for ExportError {} + +/// Flatten a descriptor into an importable surface: **every plane of every layer, +/// in declared order** (module docs). +/// +/// Validates before it walks, so a malformed descriptor is a typed refusal and +/// never an out-of-bounds read of the fixed arrays. The caller owns +/// [`ExportedSurface::object_fds`] on success; on failure it owns the descriptor's +/// fds and must close them itself — this function takes no ownership either way, +/// because it cannot know whether the export call succeeded. +pub fn flatten(desc: &VaDrmPrimeSurfaceDescriptor) -> Result { + let objects = desc.num_objects as usize; + if objects == 0 || objects > MAX_OBJECTS { + return Err(ExportError::ObjectCount(desc.num_objects)); + } + let layers = desc.num_layers as usize; + if layers == 0 || layers > MAX_LAYERS { + return Err(ExportError::LayerCount(desc.num_layers)); + } + for (i, o) in desc.objects[..objects].iter().enumerate() { + if o.fd < 0 { + return Err(ExportError::BadFd { + object: i, + fd: o.fd, + }); + } + } + let modifier = desc.objects[0].drm_format_modifier; + if let Some(o) = desc.objects[1..objects] + .iter() + .find(|o| o.drm_format_modifier != modifier) + { + return Err(ExportError::MixedModifiers { + first: modifier, + other: o.drm_format_modifier, + }); + } + + let mut planes = Vec::with_capacity(layers * 2); + for (l, layer) in desc.layers[..layers].iter().enumerate() { + let n = layer.num_planes as usize; + if n == 0 || n > MAX_PLANES_PER_LAYER { + return Err(ExportError::PlaneCount { + layer: l, + planes: layer.num_planes, + }); + } + for p in 0..n { + let index = layer.object_index[p]; + if index as usize >= objects { + return Err(ExportError::ObjectIndex { + layer: l, + plane: p, + index, + }); + } + planes.push(ExportedPlane { + fd: desc.objects[index as usize].fd, + offset: layer.offset[p], + stride: layer.pitch[p], + }); + } + } + + Ok(ExportedSurface { + fourcc: desc.fourcc, + width: desc.width, + height: desc.height, + modifier, + planes, + object_fds: desc.objects[..objects].iter().map(|o| o.fd).collect(), + }) +} + +// Measured by `layout-probe.c` against libva 2.23.0 headers, not transcribed. +const _: () = { + use std::mem::align_of; + use std::mem::offset_of; + use std::mem::size_of; + assert!(size_of::() == 16); + assert!(size_of::() == 56); + assert!(size_of::() == 312); + assert!(align_of::() == 8); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, fourcc) == 0); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, width) == 4); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, height) == 8); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, num_objects) == 12); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, objects) == 16); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, num_layers) == 80); + assert!(offset_of!(VaDrmPrimeSurfaceDescriptor, layers) == 84); +}; + +#[cfg(test)] +mod tests { + use super::*; + + /// `DRM_FORMAT_R8` / `DRM_FORMAT_GR88` — the component formats a driver reports + /// per layer for NV12 under `SEPARATE_LAYERS`. Present only to build the + /// realistic fixture; nothing in the walk reads them, which is the point. + const DRM_FORMAT_R8: u32 = 0x2038_5220; + const DRM_FORMAT_GR88: u32 = 0x3838_5247; + const MOD: u64 = 0x0200_0000_0180_1002; + + /// What radeonsi/iHD actually hand back for an NV12 decode surface: ONE object, + /// TWO layers of one plane each, chroma at a non-zero offset in the same buffer. + fn nv12_two_layers() -> VaDrmPrimeSurfaceDescriptor { + let mut d = VaDrmPrimeSurfaceDescriptor::zeroed(); + d.fourcc = VA_FOURCC_NV12; + d.width = 1920; + d.height = 1080; + d.num_objects = 1; + d.objects[0] = VaDrmPrimeObject { + fd: 7, + size: 1920 * 1088 * 3 / 2, + drm_format_modifier: MOD, + }; + d.num_layers = 2; + d.layers[0] = VaDrmPrimeLayer { + drm_format: DRM_FORMAT_R8, + num_planes: 1, + object_index: [0; 4], + offset: [0; 4], + pitch: [1920, 0, 0, 0], + }; + d.layers[1] = VaDrmPrimeLayer { + drm_format: DRM_FORMAT_GR88, + num_planes: 1, + object_index: [0; 4], + offset: [1920 * 1088, 0, 0, 0], + pitch: [1920, 0, 0, 0], + }; + d + } + + #[test] + fn both_layers_become_planes_and_the_surface_keeps_its_own_fourcc() { + let out = flatten(&nv12_two_layers()).expect("a well-formed NV12 export"); + // The green-screen regression, as an assertion: two planes, not one. + assert_eq!(out.planes.len(), 2, "chroma was dropped"); + assert_eq!( + out.fourcc, VA_FOURCC_NV12, + "the surface fourcc must come from the descriptor, not from layers[0].drm_format \ + ({DRM_FORMAT_R8:#010x})" + ); + assert_ne!(out.fourcc, DRM_FORMAT_R8); + assert_eq!(out.planes[0].offset, 0); + assert_eq!(out.planes[1].offset, 1920 * 1088); + // Both planes live in the SAME object, so both carry the same fd — and the + // caller must close it exactly once. + assert_eq!(out.planes[0].fd, 7); + assert_eq!(out.planes[1].fd, 7); + assert_eq!(out.object_fds, vec![7]); + assert_eq!(out.modifier, MOD); + } + + #[test] + fn a_multi_plane_layer_flattens_in_declared_order() { + // The COMPOSED-ish shape: one layer that declares both planes itself. The + // walk must handle it without caring which shape the driver chose. + let mut d = VaDrmPrimeSurfaceDescriptor::zeroed(); + d.fourcc = VA_FOURCC_P010; + d.num_objects = 2; + d.objects[0] = VaDrmPrimeObject { + fd: 11, + size: 64, + drm_format_modifier: MOD, + }; + d.objects[1] = VaDrmPrimeObject { + fd: 12, + size: 32, + drm_format_modifier: MOD, + }; + d.num_layers = 1; + d.layers[0] = VaDrmPrimeLayer { + drm_format: VA_FOURCC_P010, + num_planes: 2, + object_index: [0, 1, 0, 0], + offset: [0, 0, 0, 0], + pitch: [3840, 3840, 0, 0], + }; + let out = flatten(&d).expect("a well-formed two-object export"); + assert_eq!(out.planes.len(), 2); + assert_eq!(out.planes[0].fd, 11); + assert_eq!(out.planes[1].fd, 12); + assert_eq!(out.object_fds, vec![11, 12], "both objects must be closed"); + } + + #[test] + fn a_driver_that_wrote_nothing_is_refused_rather_than_read() { + let d = VaDrmPrimeSurfaceDescriptor::zeroed(); + assert_eq!(flatten(&d), Err(ExportError::ObjectCount(0))); + } + + #[test] + fn counts_past_the_arrays_are_refused_before_the_walk() { + let mut d = nv12_two_layers(); + d.num_objects = 5; + assert_eq!(flatten(&d), Err(ExportError::ObjectCount(5))); + let mut d = nv12_two_layers(); + d.num_layers = 9; + assert_eq!(flatten(&d), Err(ExportError::LayerCount(9))); + let mut d = nv12_two_layers(); + d.layers[1].num_planes = 5; + assert_eq!( + flatten(&d), + Err(ExportError::PlaneCount { + layer: 1, + planes: 5 + }) + ); + } + + #[test] + fn a_plane_naming_an_object_that_does_not_exist_is_refused() { + // `num_objects` is 1, so object_index 1 addresses descriptor memory the + // driver never wrote — an fd of -1 or worse, a stale one. + let mut d = nv12_two_layers(); + d.layers[1].object_index[0] = 1; + assert_eq!( + flatten(&d), + Err(ExportError::ObjectIndex { + layer: 1, + plane: 0, + index: 1 + }) + ); + } + + #[test] + fn objects_that_disagree_on_tiling_are_refused_not_averaged() { + let mut d = nv12_two_layers(); + d.num_objects = 2; + d.objects[1] = VaDrmPrimeObject { + fd: 8, + size: 16, + drm_format_modifier: 0, + }; + d.layers[1].object_index[0] = 1; + assert_eq!( + flatten(&d), + Err(ExportError::MixedModifiers { + first: MOD, + other: 0 + }) + ); + } + + #[test] + fn an_object_without_an_fd_is_refused() { + let mut d = nv12_two_layers(); + d.objects[0].fd = -1; + assert_eq!(flatten(&d), Err(ExportError::BadFd { object: 0, fd: -1 })); + } +} diff --git a/crates/pf-vaadec/src/lib.rs b/crates/pf-vaadec/src/lib.rs index 5858889e..e57ec530 100644 --- a/crates/pf-vaadec/src/lib.rs +++ b/crates/pf-vaadec/src/lib.rs @@ -15,9 +15,10 @@ //! //! # Status //! -//! **H.264 conversion complete; no libva calls yet.** What remains for the rung is -//! the Linux-only plumbing (config/context/surface creation, `vaRenderPicture`, -//! sync, dmabuf export) and the H.265 twin of [`pic`]. +//! **Both codecs converted, and the rung is wired.** `pf-client-core`'s +//! `video_vaapi_native` dlopens libva and drives these buffers; this crate holds +//! everything decidable without a device — including [`drm`], the export +//! descriptor the driver writes back and the plane walk that reads it. //! //! Four things this crate settled that a reader would otherwise have to re-derive: //! @@ -56,6 +57,7 @@ //! a parameter and stay pure. pub mod config; +pub mod drm; pub mod pic; pub mod pic_h265; pub mod va; @@ -69,6 +71,8 @@ pub use pf_vkdecode::SlotMap; /// every type it touches through `pf_vaadec` — the same courtesy `pf-dxvadec` does /// for the Windows layer. pub use pf_bitstream::h264::AuPlan; +pub use pf_bitstream::h264::ColourDescription; +pub use pf_bitstream::h264::DisplayCrop; pub use pf_bitstream::h264::H264Planner; pub use pf_bitstream::h264::PlanError; pub use pf_bitstream::h264::PlanWarning; @@ -81,6 +85,16 @@ pub use pf_bitstream::h265::PlanWarning as PlanWarningH265; pub use pf_vkdecode::is_integrity_warning; pub use pf_vkdecode::is_integrity_warning_h265; +pub use drm::flatten; +pub use drm::ExportError; +pub use drm::ExportedPlane; +pub use drm::ExportedSurface; +pub use drm::VaDrmPrimeSurfaceDescriptor; +pub use drm::VA_EXPORT_SURFACE_READ_ONLY; +pub use drm::VA_EXPORT_SURFACE_SEPARATE_LAYERS; +pub use drm::VA_FOURCC_NV12; +pub use drm::VA_FOURCC_P010; + pub use config::profile_for; pub use config::rt_format; pub use config::surface_count; diff --git a/crates/pf-vaadec/src/pic.rs b/crates/pf-vaadec/src/pic.rs index ba656e1c..a66d3825 100644 --- a/crates/pf-vaadec/src/pic.rs +++ b/crates/pf-vaadec/src/pic.rs @@ -237,8 +237,26 @@ fn surface_of(slots: &SlotMap, surfaces: &[u32], id: PicId) -> Result<(u8, u32), /// /// `au` is the access unit the plan was built from — needed because the slice data /// buffer must start at the NAL header byte, and the start-code prefix is three or -/// four bytes depending on the encoder. `surfaces` maps DPB slot to `VASurfaceID`; -/// this crate never allocates one. +/// four bytes depending on the encoder. `surfaces` maps DPB slot to `VASurfaceID` +/// for the pictures the DPB already holds; this crate never allocates one. +/// +/// # Why the decode target is a parameter and not `surfaces[setup_slot]` +/// +/// The caller binds the target surface, at activation time, exactly as +/// `pf-vkdecode` binds a pool image when a DPB slot is activated. A slot ledger +/// is not a surface allocator: [`SlotMap::assign`] takes the lowest free slot, and +/// a slot freed by this AU's own removals is free by the time the setup picture +/// takes it — measured at **225 of the vendored vector's 250 access units** +/// (`the_setup_picture_routinely_inherits_a_just_freed_slot`). Reading the target +/// out of a slot-indexed table would therefore decode, on nine frames in ten, +/// into the surface holding the picture that was just displayed — which the +/// consumer may still be sampling. Zero-copy means the decoder cannot have that +/// surface back until the consumer says so, and only the caller knows. +/// +/// `setup_surface` must be free in that sense: bound to no live picture and held +/// by no consumer. After a successful call the caller binds it to +/// [`DecodePlanVa::setup_slot`], so later access units resolve references to this +/// picture through `surfaces`. /// /// Nothing mutates `slots` until every fallible step has passed. pub fn plan_to_va( @@ -246,6 +264,7 @@ pub fn plan_to_va( au: &[u8], slots: &mut SlotMap, surfaces: &[u32], + setup_surface: u32, ) -> Result { if plan.slices.is_empty() { return Err(PlanToVaError::NoSlices); @@ -271,6 +290,16 @@ pub fn plan_to_va( capacity: slots.capacity(), }); } + // The caller binds `setup_surface` to the returned slot, so a table that cannot + // express every slot is caught HERE — before anything mutates the ledger — + // rather than as an out-of-range bind after a successful conversion. Checked + // against the capacity, not the chosen slot, precisely so it stays a pre-check. + if surfaces.len() < slots.capacity() { + return Err(PlanToVaError::SurfaceOutOfRange { + slot: (slots.capacity() - 1) as u8, + surfaces: surfaces.len(), + }); + } // Height is expressed in FRAME macroblocks, so the map-units count doubles for a // non-frame-only SPS — unreachable inside pf-bitstream's progressive envelope, @@ -415,16 +444,9 @@ pub fn plan_to_va( if setup_evicted { slots.release(setup_id); } - let curr_surface = - *surfaces - .get(usize::from(setup_slot)) - .ok_or(PlanToVaError::SurfaceOutOfRange { - slot: setup_slot, - surfaces: surfaces.len(), - })?; let curr_pic = VaPictureH264 { - picture_id: curr_surface, + picture_id: setup_surface, // For the current picture this is `frame_num`, not a long-term index. frame_idx: u32::from(pic.frame_num), flags: if pic.is_reference { @@ -511,6 +533,14 @@ pub fn plan_to_va( #[cfg(test)] mod tests { use super::*; + use crate::va::VA_INVALID_SURFACE; + + /// Surface ids for the walks below: `SURFACE_BASE + access-unit index`, so every + /// picture gets its own and none is ever reused. Well away from slot indices, so + /// a mix-up shows as a value rather than a plausible off-by-one — and unique, so + /// a stale or aliased reference cannot hide behind a surface that happens to be + /// right again. + const SURFACE_BASE: u32 = 0x9000; use crate::va::VA_PICTURE_H264_INVALID; /// The vendored conformance vector every other rung's parity legs decode: 250 @@ -567,10 +597,13 @@ mod tests { assert_eq!(aus.len(), 250, "the vendored vector is 250 access units"); let mut planner = H264Planner::new(); - // One surface per slot, with ids that are distinguishable from slot indices - // so a mix-up shows up as a value, not as an off-by-one that still looks - // plausible. - let surfaces: Vec = (0..32u32).map(|i| 0x9000 + i).collect(); + // The caller's binding, modelled the way the rung does it: every picture is + // given its OWN never-reused surface id, and the slot table is updated after + // the conversion returns. Ids start well away from slot indices so a mix-up + // shows up as a value rather than as a plausible-looking off-by-one, and + // never reusing one means a stale or aliased reference cannot hide behind a + // surface that happens to be right again. + let mut surfaces: Vec = Vec::new(); let mut slots: Option = None; let mut converted = 0usize; let mut saw_multi_slice = false; @@ -581,8 +614,11 @@ mod tests { .plan_au(au) .unwrap_or_else(|e| panic!("AU {index}: the clean vector must plan, got {e:?}")); let map = slots.get_or_insert_with(|| SlotMap::new(plan.picture.max_dpb_frames)); - let out = plan_to_va(&plan, au, map, &surfaces) + surfaces.resize(map.capacity(), VA_INVALID_SURFACE); + let setup_surface = SURFACE_BASE + index as u32; + let out = plan_to_va(&plan, au, map, &surfaces, setup_surface) .unwrap_or_else(|e| panic!("AU {index}: conversion failed: {e}")); + surfaces[usize::from(out.setup_slot)] = setup_surface; assert_eq!( out.slices.len(), @@ -657,6 +693,86 @@ mod tests { ); } + /// The decode target must never be a surface this same access unit READS. + /// + /// This is the question a slot ledger cannot answer, and it is why the caller + /// binds the setup surface instead of the conversion reading one out of a + /// slot-indexed table. + /// + /// `SlotMap::assign` takes the LOWEST free slot, and a slot freed by this + /// access unit's own removals is free by the time the setup picture is + /// assigned. Measured on the vendored vector, that is not an edge case: the + /// setup picture inherits a just-freed slot on **225 of 250** access units. + /// A surface bound BY SLOT would therefore decode, on nine frames in ten, + /// into the surface still holding the picture that was just displayed — which + /// under zero-copy the consumer may still be sampling. Hence the pool model + /// this crate's callers use, and hence `setup_surface`. + /// + /// The second half of the test is the reassurance that comes with it: given + /// the caller's contract (a surface bound to no live picture), the decode + /// target is never a surface the same access unit READS. That is checked + /// against both readable sets, which are not the same snapshot — `dpb_refs` is + /// taken after this AU's marking process, the per-slice lists before it. + #[test] + fn the_setup_picture_routinely_inherits_a_just_freed_slot() { + use pf_bitstream::h264::H264Planner; + + let aus = split_aus(TEST_25FPS_H264); + let mut planner = H264Planner::new(); + let mut surfaces: Vec = Vec::new(); + let mut slots: Option = None; + let mut collisions = 0usize; + let mut first: Option = None; + let mut inherited = 0usize; + + for (index, au) in aus.iter().enumerate() { + let plan = planner.plan_au(au).expect("the clean vector plans"); + let map = slots.get_or_insert_with(|| SlotMap::new(plan.picture.max_dpb_frames)); + surfaces.resize(map.capacity(), VA_INVALID_SURFACE); + // Which slots this AU's own removals will free — read BEFORE the + // conversion applies them, because afterwards the ledger has forgotten. + let freed: Vec = plan + .dpb + .removed + .iter() + .filter_map(|id| map.slot_of(*id)) + .collect(); + let setup_surface = SURFACE_BASE + index as u32; + let out = plan_to_va(&plan, au, map, &surfaces, setup_surface) + .expect("the clean vector converts"); + surfaces[usize::from(out.setup_slot)] = setup_surface; + if freed.contains(&out.setup_slot) { + inherited += 1; + } + let curr = out.pic_params.curr_pic.picture_id; + let names = + |e: &VaPictureH264| e.flags & VA_PICTURE_H264_INVALID == 0 && e.picture_id == curr; + let read_by_this_au = out.pic_params.reference_frames.iter().any(names) + || out.slices.iter().any(|s| { + s.ref_pic_list0.iter().any(names) || s.ref_pic_list1.iter().any(names) + }); + if read_by_this_au { + collisions += 1; + first.get_or_insert(index); + } + } + // The measurement this design rests on. A floor rather than the exact + // count, so a planner change that shifts it by a frame does not fail — + // but one that made slot reuse RARE would, and would mean the doc above + // has stopped being true. + assert!( + inherited > 200, + "the setup picture inherited a just-freed slot on only {inherited} of 250 access \ + units — the reason `setup_surface` is a parameter no longer holds, and the \ + documentation that cites it needs re-measuring" + ); + assert_eq!( + collisions, 0, + "the decode target collided with a picture this access unit reads, on \ + {collisions} of 250 (first at AU {first:?})" + ); + } + #[test] fn start_code_len_reads_both_prefix_forms() { assert_eq!(start_code_len(&[0, 0, 1, 0x65]), Some(3)); diff --git a/crates/pf-vaadec/src/pic_h265.rs b/crates/pf-vaadec/src/pic_h265.rs index 5f75ad4f..1ff59c41 100644 --- a/crates/pf-vaadec/src/pic_h265.rs +++ b/crates/pf-vaadec/src/pic_h265.rs @@ -144,12 +144,15 @@ impl std::fmt::Display for PlanToVaH265Error { impl std::error::Error for PlanToVaH265Error {} /// Convert one planned HEVC access unit. See [`crate::pic::plan_to_va`] for the -/// parameter contract — `au` and `surfaces` mean the same things. +/// parameter contract — `au`, `surfaces` and `setup_surface` mean the same things, +/// including the reason the decode target is bound by the caller rather than read +/// out of a slot-indexed table. pub fn plan_to_va_h265( plan: &AuPlanH265, au: &[u8], slots: &mut SlotMap, surfaces: &[u32], + setup_surface: u32, ) -> Result { if plan.slices.is_empty() { return Err(PlanToVaH265Error::NoSlices); @@ -169,6 +172,14 @@ pub fn plan_to_va_h265( capacity: slots.capacity(), }); } + // See the H.264 twin: a pre-check, so the caller's post-call bind of + // `setup_surface` to the returned slot is always in range. + if surfaces.len() < slots.capacity() { + return Err(PlanToVaH265Error::SurfaceOutOfRange { + slot: (slots.capacity() - 1) as u8, + surfaces: surfaces.len(), + }); + } if plan.dpb_refs.len() > REFERENCE_FRAMES_LEN_H265 { return Err(PlanToVaH265Error::TooManyReferences(plan.dpb_refs.len())); } @@ -358,17 +369,9 @@ pub fn plan_to_va_h265( if setup_evicted { slots.release(setup_id); } - let curr_surface = - *surfaces - .get(usize::from(setup_slot)) - .ok_or(PlanToVaH265Error::SurfaceOutOfRange { - slot: setup_slot, - surfaces: surfaces.len(), - })?; - let pic_params = VaPictureParameterBufferHEVC { curr_pic: VaPictureHEVC { - picture_id: curr_surface, + picture_id: setup_surface, pic_order_cnt: pic.pic_order_cnt, flags: 0, va_reserved: [0; 4], @@ -533,6 +536,9 @@ mod tests { use crate::va_h265::REF_PIC_LIST_UNUSED; use crate::va_h265::VA_PICTURE_HEVC_INVALID; + /// `SURFACE_BASE + access-unit index` — see the H.264 twin's constant. + const SURFACE_BASE: u32 = 0xa000; + const TEST_25FPS_H265: &[u8] = include_bytes!( "../../pf-bitstream/vendor/cros-codecs/src/codec/h265/test_data/test-25fps.h265" ); @@ -576,7 +582,9 @@ mod tests { assert_eq!(aus.len(), expect_aus, "{label}: access-unit count"); let mut planner = H265Planner::new(); - let surfaces: Vec = (0..32u32).map(|i| 0xa000 + i).collect(); + // The caller's binding model — see the H.264 twin: one never-reused surface + // id per picture, bound to its slot after the conversion returns. + let mut surfaces: Vec = Vec::new(); let mut slots: Option = None; let mut saw_rps_flags = false; let mut saw_list_entries = false; @@ -586,8 +594,11 @@ mod tests { .plan_au(au) .unwrap_or_else(|e| panic!("{label} AU {index}: must plan, got {e:?}")); let map = slots.get_or_insert_with(|| SlotMap::new(plan.picture.max_dpb_frames)); - let out = plan_to_va_h265(&plan, au, map, &surfaces) + surfaces.resize(map.capacity(), crate::va::VA_INVALID_SURFACE); + let setup_surface = SURFACE_BASE + index as u32; + let out = plan_to_va_h265(&plan, au, map, &surfaces, setup_surface) .unwrap_or_else(|e| panic!("{label} AU {index}: conversion failed: {e}")); + surfaces[usize::from(out.setup_slot)] = setup_surface; assert_eq!(out.slices.len(), plan.slices.len()); for (n, (rec, range)) in out.slices.iter().zip(&out.slice_data).enumerate() { diff --git a/crates/pf-vaadec/src/va.rs b/crates/pf-vaadec/src/va.rs index 02a65943..6de946ca 100644 --- a/crates/pf-vaadec/src/va.rs +++ b/crates/pf-vaadec/src/va.rs @@ -48,6 +48,19 @@ /// one vendor and not another, so both are always written together. pub const VA_INVALID_SURFACE: u32 = 0xffff_ffff; +/// `VABufferType` for the four buffers a decode submits, measured off real headers +/// by `layout-probe.c` rather than counted off the enum in the header. +/// +/// ⚠ The last two are the trap: `VASliceParameterBufferType` is **4** and +/// `VASliceDataBufferType` is **5**, not the 3 and 4 that counting from the top +/// gives — `VABitPlaneBufferType` and `VASliceGroupMapBufferType` sit in between +/// for the codecs that need them. Getting these wrong hands the driver a slice as +/// if it were something else, which is not a decode error but a decode of garbage. +pub const VA_PICTURE_PARAMETER_BUFFER_TYPE: u32 = 0; +pub const VA_IQ_MATRIX_BUFFER_TYPE: u32 = 1; +pub const VA_SLICE_PARAMETER_BUFFER_TYPE: u32 = 4; +pub const VA_SLICE_DATA_BUFFER_TYPE: u32 = 5; + /// Flags for [`VaPictureH264::flags`]. pub const VA_PICTURE_H264_INVALID: u32 = 0x0000_0001; pub const VA_PICTURE_H264_TOP_FIELD: u32 = 0x0000_0002;