Files
punktfunk/crates/pf-encode
enricobuehler 5b3ea6e8db fix(pf-encode): one NVENC open failure could kill every session on the box
`punktfunk-host` died twice on home-nobara-1 with the same stack:

    __strlen_evex <- av_vbprintf <- format_line <- av_log_default_callback
      <- ff_cuda_check <- ff_nvenc_encode_init <- avcodec_open2
      <- NvencEncoder::open <- NvencEncoder::reset <- virtual_stream

once as an outright SIGSEGV mid-session, and once as a thread wedged in that
stack so the service never answered SIGTERM and systemd escalated to SIGABRT
("State 'stop-sigterm' timed out. Aborting."). Both times a client's session was
rebuilding its encoder. The blast radius is the whole host process — every other
client's session goes with it.

The fault is in libav, not here. `ff_cuda_check` logs the failing CUDA call as
`"%s failed -> %s: %s"` using an `err_name`/`err_string` pair the error lookup
does not always fill, and glibc then walks whatever was on the stack. We cannot
patch the distro's FFmpeg, so the fix denies it the chance to format: the guard
already used by the 4:4:4 probe drops the level to AV_LOG_FATAL across the open,
and `av_log_default_callback` returns on the level check before `format_line` —
these messages are AV_LOG_ERROR. The failure is not swallowed; it still comes
back as `Err(e)` and is reported with our own context, which now says the libav
text was deliberately silenced so nobody hunts for a message that will not come.

Scoped to the `open_with` call ALONE. The ENOSYS arm immediately below recurses
into `Self::open`, and `QuietLibavLog` holds a non-reentrant global mutex —
wrapping the whole `match` would have deadlocked the intra-refresh retry.

Verified on home-nobara-1 (fc44, libavcodec 62). With CUDA made unavailable so
the open fails inside the CUDA layer, the old binary prints

    [hevc_nvenc @ ..] cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable
    device is detected

— that line IS `ff_cuda_check` formatting the two `%s` — and the fixed binary
does not; both exit 1 with our error instead. A successful open is unaffected on
both the direct-SDK and the libav paths (90/90 frames, identical output size).

What this does NOT claim: the uninitialized-pointer condition itself was not
reproduced on demand — it depends on the CUDA error lookup failing to fill the
strings, and in the forced case above it filled them fine. What is demonstrated
is that the formatting call which faulted is no longer reached during the open.
2026-08-10 07:46:47 +02:00
..