The grey native-vulkan stream: a host re-anchor claim the client could not check #262

Merged

enricobuehler merged 3 commits from worktree-grey-frame-reanchor-fixes into main 2026-08-16 07:57:21 +00:00

Conversation 0 Commits 3 Files Changed 24

+1613 -113

enricobuehler commented 2026-08-15 23:31:44 +00:00

Owner

Copy Link

Copy Source

Field report (reporter + 1–2 other testers): the native-vulkan HEVC stream on the Linux client goes grey with moving artifacts, reliably triggered by a host context change (starting a game), and recovers on its own in 0.5–2 s. Host is AMD Linux on the Vulkan Video encoder.

What it actually is

Of the three signals that lift the client's post-loss freeze, two are self-evident to the client and one is pure hearsay:

Signal	Scrutiny before this PR
Real IDR	none needed — predicts from nothing
USER_FLAG_RECOVERY_POINT	two marks required (a_single_recovery_mark_does_not_lift)
local recovery-point SEI	paired against the arm (a_recovery_point_from_a_wave_that_predates_the_loss_is_ignored)
USER_FLAG_RECOVERY_ANCHOR	none at all (an_rfi_anchor_lifts_immediately)

The anchor is the host asserting a fact about the client's decoder — the picture I coded this P-frame against is one you still hold, intact — and the gate believed it on the first occurrence. The host derives that claim from slot_wire, which tracks what the client received, not what it managed to decode. Those diverge exactly when the client had to conceal, and then the anchor lifts the freeze onto a picture predicted from damage and leaves it lifted — so the grey plate reaches the screen and keeps reaching it until a later signal re-arms and the 500 ms backstop extracts a real IDR.

The observed 0.5–2 s is the host's own constants: IDR_COOLDOWN_FULL 750 ms, RFI_ECHO_WINDOW 300 ms × RFI_ECHO_MAX_SWALLOWED 2, client REANCHOR_FREEZE_MAX 500 ms.

Ruled out by construction, not by guesswork: the HEVC lane already detects and withholds every missing reference (h265.rs:973/981/1013/1023, the empty-RefPicList0 check at :1249, truncation → integrity.rs → released unshown), and the presenter's only clear colour is black. So the grey picture's references resolve — they are simply the wrong content, which nothing in the bitstream can say.

The fix

Client — corroborate the claim (26d06195). pf-bitstream's planners carry a per-picture clean bit that propagates down the prediction chain (the descendants of a concealed picture raise no warning of their own), surfaced as PicturePlan::references_clean and carried on DecodedVkFrame. The gate gains AnchorEvidence + on_decoded_corroborated and refuses an anchor whose references the client can prove were damaged.

Refusing can only ever hold longer: the freeze stays up, the backstop fires on its original deadline, and the client escalates to the IDR the anchor failed to be. A real IDR still lifts unconditionally — refusing that would turn the fix into a permanent freeze. Lanes with no local parser pass Unavailable; all 29 pre-existing reanchor tests pass untouched.

Host — stop offering it (93c1ed07). Adds Encoder::distrust_references, implemented by the three slot-family backends through their own persistence markers (rfi.rs says explicitly not to harmonize these). Called on the IDR-cooldown coalesce branch, where the client is still reporting damage and nothing is honestly known-good until the in-flight IDR lands.

Deliberately not on the RFI-echo branch while its budget holds — that branch's premise is that the request echoes the loss the RFI just repaired, and distrusting on the first echo would poison the table after every successful recovery, so RFI could never fire twice running. RFI_ECHO_MAX_SWALLOWED is already the hedge: when the client keeps asking past the budget, the anchor demonstrably did not heal it, and the escalation arm withdraws trust then — on evidence rather than suspicion.

Also: the H.264 lane was the only one that failed open

Found on the way, latent rather than field-reported (default codec is HEVC). decoder.rs was alone among the three in decoding anyway when a DPB slot had no bound image — traced and continued where H.265/AV1 return UnboundReferenceSlot; computed reference_count after the held-slot loop, so a dropped reference silently took an unrelated slot's picture in its place; and had no RecoveryLatch, so one failure left the planner DPB, the slot map and the image bindings disagreeing forever. None of it raised a warning, so the frame was shipped, presented, and cleared the demotion streak on its way past. Both latches landed with the HEVC and AV1 decoders in August; H.264 predates them and was never retro-fitted.

Verification

macOS native — pf-bitstream 95 · pf-vkdecode 197 + 5 fault-detection + 11 GPU-parity + 1 smoke (14 GPU tests skipped, no hardware) · punktfunk-core 238 incl. 36 reanchor (29 pre-existing untouched + 7 new) · fmt --all --check clean.

Linux amd64 container, all six steps Finished, zero failures — pf-client-core check --all-targets + 194 tests; pf-vaadec check --all-targets; pf-encode check --all-targets + 50 tests (vulkan-encode), including the pre-existing h265_rps_retains_all_residents, the invariant the Vulkan distrust_references had to not break; punktfunk-host check --all-targets.

The gate earned its keep: it caught a PicturePlanAv1 literal in a cfg(test) fixture reaching the plan types through pf-vaadec's re-exports (89fd4c6f) — invisible to macOS and to any single-crate run.

⚠ Needs the Windows CI leg before merge

AMF and QSV are compiled by nothing in this PR. They are Windows-only behind amf-qsv/qsv, and the CI runner at .133 was unreachable throughout. Both impls were reviewed by hand — the [None; NUM_LTR_SLOTS] / [true; NUM_LTR_SLOTS] idioms match each file's own constructor, field types line up, and the trait signature is proven by the Linux impl compiling — but punktfunk-linux-compile-check-docker records this exact gap as how a red main was pushed on 2026-07-25. Please let the Windows leg go green before merging.

On-glass still owed

Nothing here has been tested on real hardware. The confirming trace, host at debug: vulkan-encode: emitting clean recovery-anchor P-frame immediately before the grey, followed by keyframe request coalesced. If the anchor line is absent when it goes grey, the anchor path is not the carrier and this fix — while independently correct — is not the cure for that instance.

Field report (reporter + 1–2 other testers): the **native-vulkan HEVC stream on the Linux client goes grey with moving artifacts**, reliably triggered by a host context change (starting a game), and **recovers on its own in 0.5–2 s**. Host is AMD Linux on the Vulkan Video encoder. ## What it actually is Of the three signals that lift the client's post-loss freeze, two are self-evident to the client and one is pure hearsay: | Signal | Scrutiny before this PR | |---|---| | Real IDR | none needed — predicts from nothing | | `USER_FLAG_RECOVERY_POINT` | **two** marks required (`a_single_recovery_mark_does_not_lift`) | | local recovery-point SEI | **paired against the arm** (`a_recovery_point_from_a_wave_that_predates_the_loss_is_ignored`) | | `USER_FLAG_RECOVERY_ANCHOR` | **none at all** (`an_rfi_anchor_lifts_immediately`) | The anchor is the host asserting a fact about the *client's* decoder — *the picture I coded this P-frame against is one you still hold, intact* — and the gate believed it on the first occurrence. The host derives that claim from `slot_wire`, which tracks what the client **received**, not what it managed to **decode**. Those diverge exactly when the client had to conceal, and then the anchor lifts the freeze onto a picture predicted from damage **and leaves it lifted** — so the grey plate reaches the screen and keeps reaching it until a later signal re-arms and the 500 ms backstop extracts a real IDR. The observed 0.5–2 s is the host's own constants: `IDR_COOLDOWN_FULL` 750 ms, `RFI_ECHO_WINDOW` 300 ms × `RFI_ECHO_MAX_SWALLOWED` 2, client `REANCHOR_FREEZE_MAX` 500 ms. Ruled out by construction, not by guesswork: the HEVC lane already detects and **withholds** every *missing* reference (`h265.rs:973/981/1013/1023`, the empty-`RefPicList0` check at `:1249`, truncation → `integrity.rs` → released unshown), and the presenter's only clear colour is black. So the grey picture's references *resolve* — they are simply the wrong content, which nothing in the bitstream can say. ## The fix **Client — corroborate the claim** (`26d06195`). pf-bitstream's planners carry a per-picture clean bit that propagates down the prediction chain (the descendants of a concealed picture raise no warning of their own), surfaced as `PicturePlan::references_clean` and carried on `DecodedVkFrame`. The gate gains `AnchorEvidence` + `on_decoded_corroborated` and refuses an anchor whose references the client can prove were damaged. Refusing can only ever hold **longer**: the freeze stays up, the backstop fires on its *original* deadline, and the client escalates to the IDR the anchor failed to be. A real IDR still lifts unconditionally — refusing *that* would turn the fix into a permanent freeze. Lanes with no local parser pass `Unavailable`; **all 29 pre-existing reanchor tests pass untouched**. **Host — stop offering it** (`93c1ed07`). Adds `Encoder::distrust_references`, implemented by the three slot-family backends through their own persistence markers (`rfi.rs` says explicitly not to harmonize these). Called on the IDR-cooldown coalesce branch, where the client is still reporting damage and nothing is honestly known-good until the in-flight IDR lands. Deliberately **not** on the RFI-echo branch while its budget holds — that branch's premise is that the request echoes the loss the RFI just repaired, and distrusting on the first echo would poison the table after *every* successful recovery, so RFI could never fire twice running. `RFI_ECHO_MAX_SWALLOWED` is already the hedge: when the client keeps asking past the budget, the anchor demonstrably did not heal it, and the escalation arm withdraws trust then — on evidence rather than suspicion. ## Also: the H.264 lane was the only one that failed open Found on the way, latent rather than field-reported (default codec is HEVC). `decoder.rs` was alone among the three in decoding anyway when a DPB slot had no bound image — traced and continued where H.265/AV1 return `UnboundReferenceSlot`; computed `reference_count` after the held-slot loop, so a dropped reference silently took an unrelated slot's picture in its place; and had no `RecoveryLatch`, so one failure left the planner DPB, the slot map and the image bindings disagreeing forever. None of it raised a warning, so the frame was shipped, presented, and **cleared the demotion streak** on its way past. Both latches landed with the HEVC and AV1 decoders in August; H.264 predates them and was never retro-fitted. ## Verification **macOS native** — pf-bitstream 95 · pf-vkdecode 197 + 5 fault-detection + 11 GPU-parity + 1 smoke (14 GPU tests skipped, no hardware) · punktfunk-core 238 incl. 36 reanchor (29 pre-existing untouched + 7 new) · `fmt --all --check` clean. **Linux amd64 container**, all six steps `Finished`, zero failures — `pf-client-core` check `--all-targets` + **194 tests**; `pf-vaadec` check `--all-targets`; `pf-encode` check `--all-targets` + **50 tests** (`vulkan-encode`), including the pre-existing `h265_rps_retains_all_residents`, the invariant the Vulkan `distrust_references` had to not break; `punktfunk-host` check `--all-targets`. The gate earned its keep: it caught a `PicturePlanAv1` literal in a `cfg(test)` fixture reaching the plan types through `pf-vaadec`'s re-exports (`89fd4c6f`) — invisible to macOS and to any single-crate run. ## ⚠ Needs the Windows CI leg before merge **AMF and QSV are compiled by nothing in this PR.** They are Windows-only behind `amf-qsv`/`qsv`, and the CI runner at `.133` was unreachable throughout. Both impls were reviewed by hand — the `[None; NUM_LTR_SLOTS]` / `[true; NUM_LTR_SLOTS]` idioms match each file's own constructor, field types line up, and the trait signature is proven by the Linux impl compiling — but `punktfunk-linux-compile-check-docker` records this exact gap as how a red main was pushed on 2026-07-25. Please let the Windows leg go green before merging. ## On-glass still owed Nothing here has been tested on real hardware. The confirming trace, host at debug: `vulkan-encode: emitting clean recovery-anchor P-frame` immediately before the grey, followed by `keyframe request coalesced`. If the anchor line is *absent* when it goes grey, the anchor path is not the carrier and this fix — while independently correct — is not the cure for that instance.

enricobuehler added 3 commits 2026-08-15 23:31:46 +00:00

fix(client): a host recovery anchor lifted the freeze onto a grey picture, unchecked ... 26d06195f5

Field report: the native-vulkan HEVC stream goes grey with moving artifacts after a
host context change (starting a game), and recovers by itself in 0.5-2 s.

Of the three signals that lift the post-loss freeze, two are self-evident to the
client and one is pure hearsay. An IDR predicts from nothing. A recovery mark is
half a re-anchor and the gate says so by requiring two. But USER_FLAG_RECOVERY_ANCHOR
is the HOST asserting a fact about the CLIENT's decoder -- "the picture I coded this
P-frame against is one you still hold, intact" -- and the gate took it on faith, on
the first occurrence, with no scrutiny at all.

The host derives that claim from bookkeeping that tracks what the client RECEIVED,
not what it managed to DECODE. Those diverge exactly when the client had to conceal,
and then the anchor lifts the freeze onto a picture predicted from damage AND LEAVES
IT LIFTED -- so the grey plate reaches the screen and keeps reaching it until some
later signal re-arms and the 500 ms backstop extracts a real IDR. That is the
observed 0.5-2 s, and it is the worst-shaped failure in the module: a re-anchor claim
the client can refute is worse than no claim, because no claim merely holds.

So corroborate it. pf-bitstream's planners now carry a per-picture clean bit --
damage propagates down the prediction chain, because the descendants of a concealed
picture raise no warning of their own -- surfaced as PicturePlan::references_clean
and carried to the consumer on DecodedVkFrame. The gate gains AnchorEvidence and
on_decoded_corroborated, and refuses an anchor whose references the client can prove
were damaged. Refusing can only ever hold LONGER: the freeze stays up, the backstop
fires on its ORIGINAL deadline, and the client escalates to the IDR the anchor failed
to be. Lanes with no local parser pass Unavailable and are bit-for-bit unchanged --
all 29 pre-existing reanchor tests pass untouched.

Also brings the H.264 decoder to parity with its two siblings, found on the way. It
was the only one of the three that failed OPEN: a DPB slot with no bound image was
traced and decoded anyway (H.265 and AV1 return UnboundReferenceSlot), reference_count
was computed after the held-slot loop so a dropped reference silently took an
unrelated slot's picture in its place, and there was no RecoveryLatch, so a failure
left the planner DPB, the slot map and the image bindings disagreeing forever. None
of it raised a warning, so the frame was shipped, presented, and cleared the demotion
streak on its way past. Both latches landed with the HEVC and AV1 decoders in August;
H.264 predates them and was never retro-fitted.

The damage classification moves onto the warning enums so the planner's ledger and
the client's concealment test cannot drift apart -- still exactly one list, now in
the crate that owns the enum, with pf_vkdecode::is_integrity_warning* delegating.
Every copy stays an exhaustive match with no wildcard: a future variant must stop the
compiler, never default to clean.

fix(host): an RFI anchor could be picked over damage the client had already reported ... 93c1ed0723

The slot-family RFI backends choose a recovery anchor over `slot_wire`, which answers
"did the client RECEIVE this frame" when the question is "did the client DECODE it
intact". The taint sweep exists precisely to bridge that gap -- rfi.rs says so -- but
it only ever runs inside invalidate_ref_frames, reachable from exactly ONE of the
client's five damage signals (the frame-index gap). The other four send a plain
keyframe request, which sets force_kf and taints nothing.

That is self-healing while the IDR is actually emitted. It is not when the request is
coalesced away by the 750 ms IDR cooldown: the client's damage then goes unrepaired
AND unrecorded, and those references stay anchor candidates for the next loss -- so
the host serves an anchor over damage the client already told it about, tagged as the
client's definitive clean re-anchor. The client-side half of this fix now refuses such
an anchor; this is the other half, which stops it being offered.

Adds Encoder::distrust_references (defaulted no-op, forwarded through TrackedEncoder
-- unforwarded it would have been a silent no-op for every session), implemented by
the three slot-family backends through their own persistence markers, which rfi.rs
explicitly says not to harmonize: Vulkan Video blanks slot_wire ONLY and leaves
slot_poc naming every resident, or build_h265_rps_s0 stops retaining them and a
conforming decoder evicts pictures the encoder still references -- a separate grey
bug that file already documents. AMF clears its mirror slot; QSV raises ltr_tainted
rather than clearing its mirror, because the RejectedRefList only names Some slots and
a cleared entry would skip the very reference being distrusted.

Called on the IDR-cooldown branch, where the client is still reporting damage and
nothing in the table is honestly known-good until the in-flight IDR lands. Deliberately
NOT on the RFI-echo branch while its budget holds: that branch's premise is that the
request echoes the loss the RFI just repaired, and distrusting on the first echo would
poison the table after EVERY successful recovery, so RFI could never fire twice running
and a sustained-loss session would fall back to the IDR path this block exists to keep
it off. RFI_ECHO_MAX_SWALLOWED is already the hedge for that premise being wrong: when
the client keeps asking past the budget, the anchor demonstrably did not heal it, and
the escalation arm withdraws trust then -- on evidence rather than on suspicion.

Distrust never touches prediction (that runs off slot indices, not the wire domain) and
all three markers self-correct within a few frames, so the suppression is brief by
construction and never spans a session.

fix(client): teach the VAAPI AV1 sizing fixture about the clean bit ... 89fd4c6f87

Adding a public field breaks LITERAL constructors, and the only one outside
pf-bitstream lives in a `cfg(test)` fixture in a crate that consumes the plan types
through pf-vaadec's re-exports — so nothing on the macOS side and no single-crate
test run could see it. The Linux `--all-targets` gate did.

Vacuously `true`: the fixture codes a key frame, which predicts from nothing. It
exists to exercise the sizing path (sequence max vs coded vs render), so the clean
bit is incidental here — but it still has to state the honest value, because `false`
is the answer that withholds a re-anchor.

enricobuehler merged commit fb9f605120 into main 2026-08-16 07:57:21 +00:00

enricobuehler deleted branch worktree-grey-frame-reanchor-fixes 2026-08-16 07:57:32 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX HitFrostbite enricobuehler (Enrico Bühler) kszlim luxus nannku neoney sassycrown

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#262