Every AV1 session died at initialize_encoder with NV_ENC_ERR_INVALID_PARAM
("rejected even at the floor bitrate") while negotiation and the session plan
looked healthy. Live-bisected on the RTX 4090 box against a 10-bit macOS AV1
session; three independent poisons, all HEVC-isms applied across the codec
union:
- Unconditional `hevcConfig.tier = 1`: NVENC's AV1 encoder is Main-tier only,
and the same write scribbled HEVC offsets into h264Config too. Tier/level
are now per-codec; AV1 deliberately writes NEITHER — its preset defaults are
the only accepted values, and `level = 0` is NOT autoselect on AV1 (that's
Level 2.0 in the AV1 enum — it pins the stream to the smallest level and
rejects any real mode). Explicit `idrPeriod = INFINITE` is likewise rejected;
with PTD enabled the driver follows the infinite gopLength, matching the
proven HEVC pattern of never touching idrPeriod.
- The 10-bit branch was gated on bit depth only, never codec: it stamped
NV_ENC_HEVC_PROFILE_MAIN10_GUID + hevcConfig bitfields onto AV1 (the killer
for HDR/10-bit clients). AV1's Main profile covers 10-bit natively; it now
sets output/input depth on its own config.
- The split-encode fallback only retried FORCED modes, so an AUTO-mode
rejection masqueraded as a bitrate cap and failed the whole clamp search.
Any non-disabled mode now retries once with split disabled. (Also corrects
the split-mode log legend: NV_ENC_SPLIT_DISABLE_MODE is 15, not 0.)
The AV1 RFI DPB write (maxNumRefFramesInDPB = 5) survives the bisect and
stays. Verified end to end: 10-bit AV1, macOS client → RTX 4090 host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
enricobuehler
968a9c66c1