fix(encode/nvenc): the host advertises what the driver lists, not a superset

Every NVIDIA host advertised a static H.264|HEVC|AV1 superset, so a 1st-gen
Maxwell (GTX 960M, no HEVC/AV1 encode) offered HEVC — a client that believed
it got ~15 s of blank video and a disconnect instead of a stream. Both OSes
now ask the driver itself (nvEncGetEncodeGUIDs) on one throwaway direct-SDK
session: Linux on the shared CUDA context, Windows on the selected render
adapter, wired into host_wire_caps AND the GameStream serverinfo mask (which
had been left on the superset for NVIDIA on both OSes). Fails open — an
unanswerable probe keeps the historical superset, so it can only ever narrow
the advertisement to codecs the GPU really encodes.

The HEVC 4:4:4 answer rides the same session on Linux instead of opening a
libav hevc_nvenc FREXT probe: that open is the prime suspect for the field
bug where one probe wedges NVENC process-wide (NV_ENC_ERR_INVALID_VERSION on
every later session until a host restart), and the direct backend re-checks
the same caps bit at session open anyway. The ffmpeg probe remains only for
hosts that really stream over libav (PUNKTFUNK_NVENC_DIRECT=0 or a build
without the nvenc feature), where ffmpeg's NVENC client runs regardless. The
10-bit probe deliberately stays libav — Linux HDR rides the libav P010 path.

On-hardware: .136 (RTX 5070 Ti) 14/14 nvenc tests in one process incl. the
probe followed by real sessions and dirty teardown; .173 (Windows RTX) probe
+ 47 release lib tests. The Windows probe test documents the pre-existing
MSVC debug-link failure (LNK2019 via the sdk crate's unused lazy loader) —
run it with --release, the same reason windows-host.yml gates with clippy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 0346ec8090568eb499e8cb7d735305b28471185e)
This commit is contained in:
2026-07-28 17:01:59 +02:00
parent 560e663aef
commit 188f55d3b1
5 changed files with 428 additions and 48 deletions
+6
View File
@@ -984,6 +984,12 @@ impl Drop for QuietLibavLog {
/// takes for a live 4:4:4 stream — and reports whether it succeeded. HEVC-only; the result is cached
/// by the caller ([`crate::can_encode_444`]). A GPU/driver/ffmpeg without RExt 4:4:4 fails
/// the open here, so the host resolves the session to 4:2:0 before the Welcome (honest downgrade).
///
/// ⚠️ Only consulted when libav will really serve the session (`PUNKTFUNK_NVENC_DIRECT=0`, or a
/// build without `--features nvenc`). A direct-SDK host answers from the driver's caps bit instead
/// (`nvenc_cuda::probe_support`) — running THIS probe there mixes ffmpeg's NVENC client into a
/// direct-SDK process, which is the LOG-3 field bug: one successful `hevc_nvenc` FREXT open+close
/// wedged every later NVENC open process-wide (`NV_ENC_ERR_INVALID_VERSION`) until a host restart.
pub fn probe_can_encode_444(codec: Codec) -> bool {
if codec != Codec::H265 {
return false;