fix(pyrowave): guard 4:4:4 modes that overflow the rate controller's block index

The vendored rate controller packs its wavelet block index into 16 bits
(RDOperation.block_offset_saving), so a mode whose 32x32-block count exceeds
u16::MAX wraps inside the controller and corrupts the bitstream — ~8K-class
4:4:4 territory. Compute the exact count (`block_count_32x32`, the counting walk
of upstream init_block_meta, pinned against the validated Apple WaveletLayout)
and expose `pyrowave_mode_fits_rdo`; the negotiator downgrades such a session to
4:2:0 before the Welcome (the honest-downgrade channel), and both encoders
refuse outright if one slips through rather than emit a wrapped stream.

Vendor patches: 0002-rdo-saving-clamp (analyze_rate_control.comp clamps the
saving accumulation to the target, same overrun class as 0001; slangmosh.hpp
regenerated), 0003-devel-encode-16bit-read (devel tool y4m 16-bit plane reads;
tool-only, kept so the vendored source stays honest).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 01:16:51 +02:00
parent ac0e73321c
commit 75474dcc90
11 changed files with 5877 additions and 1896 deletions
@@ -132,6 +132,10 @@ void emit_rdo_operations()
else if (gl_SubgroupInvocationID < 16)
{
uint saving = shared_rate_cost[gl_SubgroupInvocationID - 1] - shared_rate_cost[gl_SubgroupInvocationID];
// PUNKTFUNK PATCH 0002: RDOperation packs saving into 16 bits; clamp the bucket
// accounting to match, else resolve over-credits applied ops and the produced
// bitstream can overshoot the hard rate target (a buffer-overrun class).
saving = min(saving, 65535u);
if (saving != 0)
{