fix(pyrowave): guard 4:4:4 modes that overflow the rate controller's block index

The vendored rate controller packs its wavelet block index into 16 bits
(RDOperation.block_offset_saving), so a mode whose 32x32-block count exceeds
u16::MAX wraps inside the controller and corrupts the bitstream — ~8K-class
4:4:4 territory. Compute the exact count (`block_count_32x32`, the counting walk
of upstream init_block_meta, pinned against the validated Apple WaveletLayout)
and expose `pyrowave_mode_fits_rdo`; the negotiator downgrades such a session to
4:2:0 before the Welcome (the honest-downgrade channel), and both encoders
refuse outright if one slips through rather than emit a wrapped stream.

Vendor patches: 0002-rdo-saving-clamp (analyze_rate_control.comp clamps the
saving accumulation to the target, same overrun class as 0001; slangmosh.hpp
regenerated), 0003-devel-encode-16bit-read (devel tool y4m 16-bit plane reads;
tool-only, kept so the vendored source stays honest).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 01:16:51 +02:00
parent ac0e73321c
commit 75474dcc90
11 changed files with 5877 additions and 1896 deletions
@@ -277,6 +277,22 @@ pub(super) async fn negotiate(
} else {
crate::encode::ChromaFormat::Yuv420
};
// PyroWave-only mode-size gate: the vendored rate controller packs its block index into
// 16 bits (pyrowave-sys patches/0002 note), which ≈8K-class 4:4:4 overflows — downgrade
// to 4:2:0 BEFORE the Welcome (the honest-downgrade channel), like every gate above.
let chroma = if codec == crate::encode::Codec::PyroWave
&& chroma.is_444()
&& !crate::encode::pyrowave_mode_fits_rdo(hello.mode.width, hello.mode.height, true)
{
tracing::warn!(
mode = %format_args!("{}x{}", hello.mode.width, hello.mode.height),
"PyroWave 4:4:4 at this mode exceeds the rate controller's block-index range — \
negotiating 4:2:0"
);
crate::encode::ChromaFormat::Yuv420
} else {
chroma
};
tracing::info!(
chroma = ?chroma,
host_wants_444,