worktree-native-decode-m0
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 enricobuehler
|
cdd1f3efce |
fix(vkdecode): AV1 is bit-exact — the bug was a use-after-free, not the driver
250/250 frames bit-identical to libavcodec on NVIDIA 610.57.04, and all four other parity legs (H.264, H.265, Main 10, both four-byte-prefix twins) still green. session_av1 built the sequence header, handed pStdSequenceHeader to vkCreateVideoSessionParametersKHR, and dropped the backing the instant the call returned — on the documented assumption that Vulkan copies parameter data before returning. NVIDIA does not. It keeps the pointer and dereferences pColorConfig when a decode is RECORDED. The freed block became our own next allocation, whose bytes read back as mono_chrome = 1, and a monochrome frame skips exactly loop_filter_level[2..3] (AV1 7.14). That is the whole fingerprint two earlier rounds chased: luma bit-exact, chroma off by small amounts, and rewriting the chroma levels in the bitstream changing nothing — the driver read them correctly and then discarded them, because it believed the stream had no chroma. StoredParamsAv1 now holds the parameters object and its Std backing in one value, so an object whose backing is gone is unrepresentable. The road there is worth recording, because two well-evidenced conclusions were wrong before this one was right. A software oracle reproduced the divergence exactly by disabling chroma deblocking, and a GPU probe showed chroma levels [8,12] and [63,63] producing byte-identical output — which looked conclusive and was not. libavcodec's own Vulkan AV1 hwaccel is bit-exact on this same driver, which proved the hardware fine and the defect ours. ffmpeg never hits it: with VK_KHR_video_maintenance2 it uses inline session parameters and never creates a parameters object at all. The proof is direct rather than inferred: a throwaway Vulkan capture layer dumped both submissions and every byte of our AV1 picture info already matched libavcodec's, including the loop filter block; only the session parameters layer differed. Watching the block's address showed correct bytes at create and our next allocation at decode. Ruled out on hardware, so nobody re-tests them: filmGrainSupport, maxCodedExtent, maxDpbSlots/maxActiveReferences, VkVideoDecodeUsageInfoKHR, the tile-start sentinel, the setup slot's SavedOrderHints, a NULL pTimingInfo, and heap luck. Two earlier fixes are confirmed against libavcodec's captured wire bytes and kept: CDEF secondary strengths carry the coded value rather than the spec's in-place fixup, and LoopRestorationSize is log2-based. The refuted driver-ignores-chroma-levels claim is corrected everywhere it was written down, and that probe test now passes and points at the lifetime of everything a submission points at before blaming a vendor. ⚠ Adjacent and NOT fixed: session.rs and session_h265.rs drop their Std backings the same way, and those sets carry embedded pointers too. Both are measured bit-exact on four drivers, so nothing is known to be wrong — but the contract now rests on a driver behaviour measured FALSE for AV1 on a shipping driver. The SAFETY comments asserting it have been corrected; the structure is deliberately untouched pending its own pass. Gates: macOS fmt/clippy/336 tests, container clippy -D warnings, all green; 8/8 gpu_parity and 3/3 gpu_smoke legs verified on the RTX 5070 Ti. |
||
|
enricobuehler
|
96fc3eca10 |
test(vkdecode): the AV1 rung finally has pixels to answer to
A parity and smoke harness for AV1, mirroring the H.264 and H.265 legs that proved those rungs bit-identical to libavcodec on four drivers before either ran on glass. This was the milestone's largest test gap: the adversarial review found four blocking defects in the AV1 conversion — flags unset on 274 frames of 274, a units error in LoopRestorationSize, per-reference info describing the wrong picture, film-grain fields left zero — and every one of them would have shown on frame 1 of a parity run, while clippy and 164 green unit tests said nothing at all. The golden is 250 per-frame SHA-256s in DISPLAY order, not 274. The vector carries 274 coded frames in 250 temporal units; the 24 extras are hidden ALTREFs, decoded and referenced but never shown, and the rung delivers what dpb.outputs names. The count is re-derived from the planner rather than assumed. Cross-checked between ffmpeg 8.1.1 on macOS arm64 and 8.0.1 on Linux x86_64, whose raw outputs are byte-identical — and then against a third party neither build knows about: the vendored vector ships upstream's own per-frame MD5s, and re-running those reproduces all 250. The golden agrees with a decode nobody in this program performed. I reproduced both independently before committing. 8-bit NV12, traced from the sequence header rather than presumed (seq_profile 0, high_bitdepth 0, mono_chrome 0), so the P010 scar does not apply here — and the header says which check to make if a Main 10 golden is ever added. film_grain_params_present is 0, which is load-bearing: grain synthesis is part of the Vulkan decode profile, so this golden is only comparable against a grain-less profile key. Anti-vacuity is the point of the exercise, so it is structural. The golden guard asserts the exact count, that every line is a bare digest, and that all entries are DISTINCT — 250 copies of one digest would let a decoder frozen on a single frame pass parity. The parity body asserts the golden set and the access-unit count before it touches hardware, so an IVF reader returning nothing cannot become "0 frames compared, pass". The agent verified the guards fire by mutating the golden three ways. assert_bit_identical now names the FIRST divergent frame, which is what localises a defect; that improves all six legs, not just AV1. AV1 has no four-byte-start-code twin, deliberately: OBUs are length-delimited, so there is no prefix for a driver to mis-skip. Documented where a reader would otherwise see an omission. Nothing here has run on a GPU. The harness exists precisely so the four review defects can be answered by measurement instead of argument. |
||
|
enricobuehler
|
c91a482b4e |
test(vkdecode): the ten-bit path finally has pixels
Every golden set in this program was eight-bit. So the strongest thing anyone could say about ten-bit decode was that a Main10 session BUILDS and streams clean — which is not the same claim, and is exactly the shape of claim this program has been burned by. A Main10 stream decoding to garbage logs just as cleanly: HEVC Main10 on D3D11VA has no per-picture status query at all, and on the Vulkan side the devices that matter report queryResultStatusSupport=false. The HDR legs were measuring that the pipe ran, not that the pixels were right. So: a Main10 vector and its goldens, and a ten-bit leg that runs them. The vector is 50 frames of 320x240 HEVC Main 10 4:2:0 from libx265 — 48 KB, generated by a command recorded in the golden file's header along with everything else needed to regenerate it. The goldens come from libavcodec's software decoder and were cross-checked between two independent builds on two architectures (ffmpeg 8.1.1 Homebrew/macOS-arm64 and 8.0.1 Ubuntu/x86_64), which agreed on all 50. The goldens are P010, NOT yuv420p10le, and that distinction is the whole reason this could have quietly gone wrong: P010 puts the ten bits in the HIGH bits of each little-endian 16-bit word with the low six zeroed, which is what a D3D11 P010 surface and Vulkan's G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16 both contain. Hashing LSB-aligned samples against MSB-aligned ones would fail on every frame on every driver, for a reason that has nothing to do with decoding. One golden file therefore serves both native rungs. The readback is now depth-aware. Its only eight-bit assumption was the second region's buffer_offset, which is a BYTE offset where the extents around it are TEXELS — that plus the buffer size are the whole change, because bufferRowLength = 0 already packs rows at the copy extent. The expected pool format moved onto the readback so the sizing and the per-frame assertion come from one source; a readback sized for eight bits that then accepted a ten-bit frame would hash half a picture and blame the decoder. A CPU guard asserts the vector really is ten-bit — 4:2:0, both depths minus8 == 2, 320x240, 50 access units, 50 planned outputs. Without it a regenerated eight-bit vector would turn the ten-bit leg into a second run of the eight-bit path wearing a ten-bit name, and it would PASS, because its goldens would have been regenerated alongside it. That guard is not ignored, so it runs on macOS and in the container rather than only on the fleet. Hardware: HEVC Main 10 50/50 bit-identical on NVIDIA 610.43.03 (Linux) and on the Steam Deck's RADV/VanGogh — first run on both, which also confirms the P010/3PACK16 layout match rather than assuming it. The four eight-bit legs are unchanged and still green on both boxes. |
||
|
enricobuehler
|
5d0b269e58 |
test(vkdecode): parity over the start-code form the host actually emits
Both vendored vectors carry three-byte Annex-B start codes throughout. The real host emits four-byte ones on 100% of access units in both codecs — 1514/1514 H.264 and 1133/1133 HEVC, measured off the M0 NVENC corpus through the capture hook's own .idx offsets. So every parity verdict this program has recorded was taken on a prefix form that never ships, and the one form that does ship was exercised by nothing. That gap is not hypothetical. Submitting four-byte start codes to vkCmdDecodeVideoKHR unchanged is exactly what made HEVC unplayable on every driver tested: drivers are validated on the three-byte form, and a fixed +3 + 2 skip into a four-byte-prefixed slice reads a nonsense pps_id — the 115 and 119 both NVIDIAs printed. H.264 was never safe here by structure, only by its vendored encoder's convention, which is why the cure lives in the shared ring layer and why this coverage is generic over both codecs. Each codec's parity body now takes its access units as a parameter and runs twice: once over the vector as it sits, once over the same vector rewritten to four-byte prefixes. Prefix width carries no information, so both runs must reproduce the same goldens — sharing one body is what makes that an equality rather than two assertions that can drift. The rewrite copies nalu.data[nalu.offset..], the same nal_size bytes the parser hands the planner, so trailing_zero_8bits are dropped exactly where the production parser drops them: the only difference between the two streams is the width of every prefix. Two CPU guards keep the new legs from passing vacuously, which is the failure mode they are most exposed to — a rewrite that quietly returned its input would make them trivially green and nothing on the fleet would notice. They assert the original really does carry three-byte prefixes, that the rewritten stream carries none, that the NAL count is preserved exactly, and that the planner still yields 250 pictures. Hardware: all four legs 250/250 bit-identical to libavcodec on two independent driver stacks — AMD VanGogh on RADV/Mesa 26.0-devel (the Steam Deck) and NVIDIA 610.43.03 on Linux. NVIDIA is the family that rejected the four-byte form outright, so it is the meaningful witness for this regression. |
||
|
enricobuehler
|
5c6b09a5c5 |
test(vkdecode): the HEVC GPU legs, which find M3 broken on every driver
M3 was recorded as code complete. Its exit criteria named the HEVC gpu_smoke and gpu_parity legs, and the goldens for them were committed — 250 per-frame NV12 hashes, cross-checked between two independent FFmpeg builds, with a header saying they are "consumed the same way by the HEVC parity test". No such test existed. Both GPU files were H.264 only, with zero references to h265, so nothing had ever decoded a single HEVC frame through this crate on hardware. They exist now, and the first run answered. On AMD Adrenalin 25.10.30.02 (distinct mode, queryResultStatusSupport=false) 247 of 250 frames diverge from libavcodec, and that device's smoke leg PASSES — because smoke only reads the driver's verdict and that driver reports none. That is the Ally X class, reproduced in-house on demand: output that is wrong everywhere the picture is looked at and clean everywhere the decoder is asked. Both NVIDIA drivers reject the stream outright and name the cause themselves, "Invalid PPS/SPS id in slice header (pps_id=119 / 115)" — the identical two values, and the smoke leg dies at the identical AU 9, on a 4090 under 610.88 on Windows and on an RTX 5070 Ti under 610.43.03 on Linux. Same wrong values, same access unit, two GPU generations, two operating systems: deterministic, and therefore ours rather than any driver's. It is not an ordering fault. Five of the divergent hashes appear nowhere in the 250 goldens, so the pixels are wrong rather than correct-but-reordered. Parity dies at frame 1 while smoke dies at AU 9 only because smoke holds four frames before it looks; the first inter-predicted picture is already corrupt. The legs are committed ahead of the fix deliberately. They are the regression test for the defect, they are #[ignore]d so no CI leg changes colour, and the evidence above is worth recording in the order it was obtained. Adding a third and fourth copy of ~150 lines of unsafe Vulkan bring-up was not acceptable, so it moved to tests/common. The two behavioural differences between the callers are now named parameters rather than accidents: the parity legs read back on a graphics queue and require one, while the smoke legs accept a decode-only device and fall back to the decode family — which also decides whether pool images are EXCLUSIVE or CONCURRENT, so it is load-bearing rather than cosmetic. H.264 came through the refactor unchanged, verified two ways: argument-by-argument against the previous file, and on hardware, still 250/250 bit-identical on NVIDIA Windows, AMD Windows and now NVIDIA Linux. The loader is deliberately leaked at teardown. ash::Entry owns the Arc<Library>, so dropping it unloads the Vulkan loader with every ICD and implicit layer; harmless while each binary held one GPU leg, but each now holds two, and the second would re-open a loader the first had torn down. Three guards run without a GPU, because everything above is #[ignore]d: the golden file's count and digest shape, the HEVC access-unit split agreeing with what the CPU planner emits (with iraps == 1 pinning "no CRA anywhere", so a re-synced vector that opens with one fails here rather than as a frame-count mismatch on the fleet), the vector staying Main 4:2:0 8-bit since both legs hard-code that probe, and a refusal to run the smoke legs with PF_VKD_TEST_READBACK set, which would quietly grow the pool a usage flag production never carries. |
||
|
enricobuehler
|
2a57ee36f8 |
feat(client): M4 — the decoder's own verdict reaches the session
This program exists because a field corruption was architecturally undetectable through FFmpeg: no decode-status read, no corrupt-frame flag, errors only as scraped log lines, and no recovery-point signal so intra-refresh healing was invisible. The native decoder has all of those. M4 is where they stop being internal. DecodeHealth counts, per session and without allocating per frame, what the three answers actually are: damaged (the stream arrived incomplete), refused (the rung would not decode it at all) and driver-failed (the hardware says it could not decode what arrived), plus the current and worst concealment run — the figures that separate one bad AU from a stream that never came back. They ride the stats line additively, so an FFmpeg session and a healthy native session emit byte-identical output to today. The status-query capability is reported too: without it a clean report cannot be told from an unmeasured one, which is the whole nb_queries=0 lesson. The headline is local recovery. Until now the pump could only learn that intra-refresh healing finished from wire flags the host sends; absent those it froze until the 500 ms backstop forced an IDR. The parsed recovery-point SEI now feeds the re-anchor gate directly, so a session lifts on the picture that is actually clean. Wire semantics are untouched for every client that never calls it. Detection now asks for recovery instead of erroring — an integrity warning ticking the error streak would demote the native rung on exactly the lossy links it exists to diagnose, where an FFmpeg rung conceals silently and keeps its job. Review round 12 found that trade had removed the escape hatch entirely. Concealment returning Ok(None) reset the demotion streak, and worse: the driver-verdict ledger is only populated when a frame ships, so under continuous concealment no verdict was ever read and the erroring arm could not fire at all. A host framing regression of the 0.23.0 slice-wire class — which does not self-heal, and which a keyframe does not clear — would have frozen indefinitely with no demotion and a clean integrity line, where before it demoted to FFmpeg-Vulkan and showed a picture. Now only an answer that proves the rung works clears the streak: a shipped frame, or a clean no-frame. Concealment neither ticks nor clears, so a lossy link still cannot demote a healthy rung while a driver failure interleaved with concealment reaches the threshold again. Two more honesty defects from the same round. A rung refusing every AU reported no integrity line at all — the founding failure mode, wearing the shape of a clean bill of health; refusals are now counted. And driver-failed could be non-zero on a device that cannot produce driver verdicts, because a degraded timeline read looked the same as one; the attribution is now withheld inside the counter rather than at call sites, so the self-contradictory line is unrepresentable. Local recovery also no longer trusts any recovery-point SEI: only one whose target advances past an outstanding wave counts as a new wave, so an encoder re-announcing the current wave with a decreasing count — legal, and what x264 intra-refresh does — cannot lift the freeze early onto a partially stale picture. Frames buffered across an arm are dropped by decode order for the same reason. Fault injection is a first-class tool now (PUNKTFUNK_AU_FAULT, inert unless set, env read once). Its test replays the vendored vectors through the real planners and asserts a negative the plan assumed away: truncation and bit flips are PROVABLY invisible to the parser — Annex-B carries no NALU length, so a cut slice is just a shorter slice and a flipped payload byte is syntactically perfect. Only dropped AUs are parser-detectable; the rest need the driver verdict, which is why the status query matters. The H.265 leg found a second: three of that vector's faulted AUs are sub-layer non-reference pictures, so dropping them damages nothing and silence is correct — the test asserts both verdicts and guards that neither half goes vacuous. Per-frame decode latency was deliberately NOT built. Polling answers only 'complete by now', and the pump polls once per AU, so every sample would quantise up by as much as a frame interval — 8.3 ms at 120 Hz against decodes of 0.1-2 ms. Sampling faster needs a spin or a second thread on a decoder that is deliberately not Sync. A blocking per-frame wait is the field scar that once capped a stream at 51 fps. The honest sampled stat stands. Also fixed, pre-existing: the re-anchor gate re-armed on every damaged AU, so sustained damage permanently zeroed the mark count — meaning the wire's two-mark rule could never complete on exactly the lossy links it was written for. Field note recorded while wiring this: intra_refresh_recovery is set by exactly one encoder backend (Linux libav-NVENC under PUNKTFUNK_INTRA_REFRESH). AMF and QSV run a wave with no wire mark, and AMF emits no recovery-point SEI either, so AMD/Windows intra-refresh sessions still have no clean recovery point by either route. Gates: fmt clean; container clippy -D warnings zero across pf-client-core + pf-presenter + pf-vkdecode + punktfunk-core; tests 69/131/129/354/41 plus 5 fault-detection green; cargo check --workspace clean. |
||
|
enricobuehler
|
c985438db1 |
test(pf-bitstream): replay real host captures through the planners + HEVC goldens
M0's capture hook has been in since
|
||
|
enricobuehler
|
e6d6498a49 |
test(pf-vkdecode): frame-hash parity vs libavcodec — bit-exact on the whole fleet
WP-D parity A/B. gpu_parity (ignored) decodes the conformance vector, reads every frame back through the presenter's exact contract (wait, layout round-trip, signal-back, release), crops at the copy so pitch can never leak, and compares SHA-256s in display order against goldens from ffmpeg software decode — cross-checked bit-identical between ffmpeg 8.0.1 (linux) and 8.1.1 (macOS), so the reference is the spec, not one build. PF_VKD_TEST_READBACK=1 is the one test-only hook (ORs TRANSFER_SRC into pool usage; production pools stay zero-copy-tight). Fleet verdict: 250/250 frames bit-identical to libavcodec on RADV (Mesa 26.0.3, distinct), AMD proprietary Windows (25.10.30.02, distinct) and NVIDIA Windows (610.88, coincide) — H.264 decode is exactly specified, and the native path meets the spec on every driver and both DPB arrangements. |
||
|
enricobuehler
|
ca92dab6fd |
fix(pf-vkdecode): prefer DEVICE_LOCAL, never require it — NVIDIA runs; both DPB modes hardware-green
Session-memory and image allocation now honor each binding's memoryTypeBits with DEVICE_LOCAL preferred, not required: NVIDIA 610.88 legally places a video-session binding in host-visible-only memory and the hard requirement refused the whole device. The bitstream ring keeps its hard HOST_VISIBLE|COHERENT need. Smoke test gains PF_VKD_SMOKE_VENDOR device pinning + attribution and a final-state print (DPB mode now observed, not inferred). On-glass matrix after this fix (.173, vendor-pinned): NVIDIA 4090 PASSES in COINCIDE mode — the first end-to-end run of the RESULT_STATUS query path, ~44 per-frame driver verdicts on the recording pattern that hangs RADV's VCN — and Adrenalin re-passes in distinct mode unchanged. With RADV's distinct pass, both DPB arrangements and three of four desktop drivers are now hardware-validated; Intel remains a clean caps refusal (no SAMPLED on decode outputs — its rung stays D3D11VA). Gates: fmt clean, clippy -D warnings zero, 45+27+53 green both platforms. |
||
|
enricobuehler
|
6331ae7fd9 |
fix(pf-vkdecode): zero-copy pool model + the two faults the first hardware run found
WP-D leg 1 (.25 RADV, distinct mode) root causes, both real: 1. Output starvation: the fixed 4-deep ring lost to a stream that keeps max_dpb_frames+1 = 8 pictures pending. Zero-copy fix (user requirement, no copies): one picture pool of required_slots + HOLD_HEADROOM(8) images decoupled from DPB slots — a re-activated slot binds a fresh free image, so a delivered picture is never a decode target; the WP-B pin layer became dead and is deleted. Per-image timeline semaphores carry the AVVkFrame contract: decode signals value+1, the presenter waits and signals back, later decodes wait the image's latest value — layout traffic ordered against reference reads with no copy anywhere. 2. RESULT_STATUS queries HANG RADV's VCN firmware (ring timeout, DEVICE_LOST): queryResultStatusSupport=false on the decode family. Queries are now caps-gated; without them poll/wait degrade to timeline-completion verdicts (FFmpeg parity — and the likely reason upstream never wired nb_queries). The Ally-X-class detection runs where drivers advertise the query; .173 probes NVIDIA/Windows-AMD. Also: slice-only bitstream feeding (the field-proven consumer shape), graveyarded pool retirement keyed by release tokens + generation, decode-current-AU-before-status attribution, take_ready drained, H264-bit gating, teardown short-circuit on disconnected channel. On-glass: 48 AUs green on .25 holding 4 frames like the real client. Gates: fmt clean, container clippy -D warnings zero, 27+121+52 green both platforms. |
||
|
enricobuehler
|
540c0d3027 |
feat(pf-vkdecode): the GPU half — session, DPB pools, decode recording, status queries
M2 WP-B. VkVideoSessionKHR lifecycle with drain-before-destroy on parameters recreation, DPB pools in both coincide and distinct modes (caps-derived, usage/flags validated against the driver's format properties), an aligned bitstream ring, vkCmdDecodeVideoKHR recording with one-shot RESET re-armed on failed submits, timeline-semaphore completion, and the per-op RESULT_STATUS query ring — the signal FFmpeg's hwaccel never reads and the reason this program exists. Frame lifetime is two-phase by construction: release_frame pins a delivered frame's slot against reuse, closing the coincide-mode overwrite the adversarial review round proved (a full DPB handed a just-returned frame's image back as the same call's decode target). Nine review findings fixed pre-commit; a counterfactual test pins the collision. Generation-stamped frames, memory-type misses as errors, granularity-aligned extents, level gate. AuPlan now carries its activated SPS/PPS (Rc) so backends never re-parse. GPU smoke test (ignored) decodes 48 AUs past DPB-full with releases — the fleet runs it in WP-D. Gates: fmt clean, clippy -D warnings zero, 45+27+53 tests green on macOS and the linux/amd64 container. |