punktfunk/design/zerocopy-hardening-handoff.md

Zero-copy capture hardening — issue handoff

Status: FIXED + validated (2026-07-06). The fix is implemented and on-glass validated — see zerocopy-worker-isolation.md: the GPU import (tiled EGL/GL→CUDA and LINEAR Vulkan→CUDA) now runs in a per-capture worker subprocess (CUDA-IPC frame hand-off), so this driver SIGSEGV kills the worker and the host degrades to its capture-loss rebuild; plus in-process teardown-order fixes and a poison/latch path replacing the corrupt tiled→CPU fallback. Validated on the RTX 5070 Ti/GNOME box: worker path streams at p50 1.30 ms, and a kill -9 of the worker mid-stream is survived + recovered (fresh worker in ~185 ms, streaming resumes). The description below is kept as the issue record.

(Original handoff intro:) This document describes a reproduced host SIGSEGV in the Linux zero-copy capture path. It deliberately does not prescribe a fix — the next agent plans the implementation. Everything below is observed fact + root-cause analysis; the "Considerations / open questions" section frames the solution space without committing to one.

This is a pre-existing capture-layer issue, NOT a regression from the gamemode/dedicated-sessions work (design/gamemode-and-dedicated-sessions.md). The crashing code (crates/punktfunk-host/src/linux/zerocopy/{cuda,egl}.rs, capture/linux/pipewire.rs) is untouched by that branch; it merely surfaced during on-glass validation of it.

1. What happened

On .181 (Bazzite F44, RTX 4090, NVIDIA open driver 610.43.02 — see [[gamemode-onglass-181-2026-07-06]]), streaming a game at the client's mode worked smoothly with zero-copy enabled (PUNKTFUNK_ZEROCOPY=1). When the user then switched the box from Steam Game Mode to the KDE/Plasma desktop mid-stream, the host process crashed (SIGSEGV, core dumped) and the client saw "session ended". systemd auto-restarted the host ~3 s later.

The host's own logic on the switch was correct up to the crash — the session watcher detected Gaming → DesktopKde, bumped the session epoch, rebuilt the backend to KWin, brought up the virtual output at 5120×1440@240, set it sole desktop, and began PipeWire capture. The crash came after that, the first time the capture thread mapped a KWin frame into CUDA.

2. The crash (backtrace)

coredumpctl for the host process, crashing thread:

Signal: 11 (SEGV)
#0  libnvidia-eglcore.so.610.43.02          ← SIGSEGV inside the NVIDIA driver
#1  libnvidia-eglcore.so.610.43.02
#2  libnvidia-eglcore.so.610.43.02
#3  libnvidia-eglcore.so.610.43.02
#4  libEGL_nvidia.so.0
#5  libcuda.so.1
#6  libcuda.so.1
#7  cuGraphicsMapResources                   (libcuda.so.1)
#8  punktfunk_host::zerocopy::cuda::RegisteredTexture::copy_mapped_plane
#9  punktfunk_host::zerocopy::egl::EglImporter::import_inner
#10 punktfunk_host::capture::linux::pipewire::pipewire_thread::{{closure}}   (the PipeWire on_process callback)
#11 pipewire::stream::…::on_process
#12 libpipewire-0.3.so.0  (impl_node_process_input → process_node → node_on_fd_events → pw_main_loop_run)

The fault is inside libnvidia-eglcore, reached through cuGraphicsMapResources — i.e. CUDA was asked to map a GL/EGL-imported resource, and the driver dereferenced GPU state that was no longer valid.

3. The code path

This is the tiled-dmabuf zero-copy path used for compositors that hand out tiled buffers (KWin, and the NVIDIA tiled path generally): PipeWire dmabuf → EGL/GL import (EglImporter, crates/punktfunk-host/src/linux/zerocopy/egl.rs) → register as a CUDA graphics resource and map per frame (RegisteredTexture::copy_mapped_plane / copy_mapped_to, crates/punktfunk-host/src/linux/zerocopy/cuda.rs:810-848 and the sibling copy_mapped_plane below it). The crash is at the cuGraphicsMapResources(1, &mut self.resource, …) call (cuda.rs:824-828), driven per frame from the PipeWire on_process callback (capture/linux/pipewire.rs, the closure at backtrace frame #10).

The relevant // SAFETY: proof (cuda.rs:814-823) reasons that self.resource is a valid CUgraphicsResource from a successful register_gl, the GL+CUDA contexts are current, and the map/unmap pair is balanced with the copy synced before unmap. That reasoning is sound for a well-behaved compositor that keeps the imported dmabuf alive across the map — it does not cover the case where the producer invalidates the underlying buffer/texture out from under an in-flight map.

Not affected on this box: the gamescope path (LINEAR dmabuf → Vulkan bridge → CUDA, linux/zerocopy/vulkan.rs) did not crash — game-mode streaming was smooth. And the non-zero-copy SHM/CPU path (PUNKTFUNK_ZEROCOPY=0, the NVENC default) has no EGL/CUDA import at all, so there is nothing for the driver to fault on. So the fault is specific to the EGL/GL→CUDA tiled import, not zero-copy in general.

4. Root cause

cuGraphicsMapResources faulted inside the driver on GPU state that had become invalid — almost certainly because the KWin compositor freed/recycled (or crashed with) the dmabuf that our EglImporter had imported and registered, while our capture thread was mapping it for the next frame. Strong corroborating evidence: in the same 8-second window, the box logged core dumps for plasmashell (×2), Xwayland, gamescope, and mangoapp. So the F44 Game↔Desktop transition on this box is itself highly unstable, and the KWin buffer our zero-copy path held a handle to almost certainly went away mid-map.

Why this is a real host-robustness gap, not just "the box is flaky": a SIGSEGV inside a closed-source driver (libnvidia-eglcore) is not catchable from Rust — it is not a Result, not a Rust panic, not something a catch_unwind can contain. So any time the producer's buffer can vanish between "we hold a CUDA graphics resource for it" and "we map it" (compositor crash, buffer-pool recycle, output/mode teardown, hot-unplug), the driver can take the whole host process down. A capture pipeline that must survive a compositor going away (which the host already tries to do — it has a capture-loss → rebuild path) cannot rely on cuGraphicsMapResources returning an error on a stale resource; the driver may just crash instead.

5. Trigger conditions (what invalidates the imported buffer)

The observed trigger was a compositor crash during a Game→Desktop switch, but the same class of fault can be reached by anything that frees/recycles the imported dmabuf or its GL texture while a map is in flight or a RegisteredTexture still references it:

• compositor crash / restart (observed);
• normal PipeWire buffer-pool recycle / renegotiation (format change, buffer count change) where a registered texture outlives the buffer it wrapped;
• virtual-output teardown / mode change (e.g. the mid-stream Reconfigure, the session-switch rebuild) racing an in-flight map;
• output removal / disconnect. The next agent should establish which of these are actually reachable in the current code (the per-frame registration/lifetime in EglImporter/RegisteredTexture vs. the PipeWire buffer lifecycle) versus only the compositor-crash case.

6. Scope

• Affected: the EGL/GL→CUDA tiled-import path (zerocopy::egl + zerocopy::cuda), driven from capture/linux/pipewire.rs. On NVIDIA this is used for tiled dmabufs (KWin desktop capture is the concrete case here; the NVIDIA tiled path in general).
• Likely also worth auditing (same class): the Vulkan bridge path (zerocopy::vulkan.rs, LINEAR dmabuf → Vulkan → CUDA) — it did not crash here, but it imports external dmabufs into the GPU too and may have the same "producer freed the buffer mid-use" exposure; confirm whether it validates/owns the buffer lifetime differently.
• Not affected: the SHM/CPU capture path (no GPU import).

7. What is NOT the cause (to save the next agent time)

• Not the gamemode/dedicated-sessions branch. That branch's switch logic worked correctly (epoch bump, watcher rebuild to KWin, virtual output up); the crash is downstream in pre-existing capture code it doesn't touch.
• Not a .desktop/KWin authorization problem. The KWin virtual output was created and set sole desktop successfully — auth was fine.
• Not the gamescope "out of buffers" issue from the same validation session (that was a separate gamescope-3.16.19 PipeWire-node limitation on SHM). This is a hard driver SEGV on the GPU-import path.

8. Observed mitigation (already available, not the fix)

Setting PUNKTFUNK_ZEROCOPY=0 (SHM/CPU path — the NVENC default anyway; the box had it forced =1) removes the EGL/CUDA import, so this crash cannot occur and a compositor going away degrades to a graceful capture-loss rebuild. Cost: a CPU copy per frame (higher latency than the zero-copy stream the user measured). This is an operational workaround, not a code fix, and it forfeits zero-copy on the desktop path.

9. Considerations / open questions for the implementation plan (do not treat as a prescription)

These frame the solution space; the next agent decides.

• A driver SIGSEGV is uncatchable in-process. Any design that "handles" this by wrapping the FFI call in error handling will not work — the process is already dead. So the fix has to be about never handing the driver a resource that can be stale, or isolating the GPU-import work so a driver crash doesn't take the streaming host down. Both directions are open:
  • Prevent-the-stale-resource directions to evaluate: strict per-frame import/register/map/unmap lifetime tied to the exact PipeWire buffer being processed (so no RegisteredTexture outlives its buffer); detecting compositor/output teardown and stopping capture before the next map; reconciling the EGL texture / CUDA resource lifetime with PipeWire's buffer-recycle events. Establish whether the current code can ever map a resource whose buffer PipeWire has already recycled/removed.
  • Isolate-the-crash directions to evaluate: whether the GPU import belongs in a separate process (like the Windows two-process/DDA isolation model) so a driver SEGV is contained and the session can rebuild — heavier, but the only thing that truly survives an unpreventable driver fault.
• Per-backend / per-buffer-type routing. The Vulkan-bridge path did not crash; the SHM path is safe. A plan might route tiled dmabufs (KWin) away from the fragile EGL/CUDA path, or only enable the EGL/CUDA path where the producer's buffer lifetime is guaranteed. Decide whether the fix is "harden the EGL/CUDA path" vs. "don't use it for producers that can pull buffers."
• Interaction with the existing capture-loss rebuild. The host already rebuilds on capture loss; the goal is to reach that path on producer teardown instead of the driver crash. Understand why the crash beats the capture-loss detection today (the map happens in the PipeWire on_process callback before any loss is observed).
• Reproducibility. This was observed on a box whose KDE was itself crashing (F44 plasmashell/ Xwayland). To isolate "our zero-copy path is fragile" from "the compositor crashed," reproduce on a stable KDE/NVIDIA box — force a buffer invalidation (output teardown / renegotiation / a scripted compositor restart) mid-capture and confirm the same cuGraphicsMapResources fault without the surrounding compositor chaos. That also tells you whether the non-crash triggers in §5 are real.
• Keep the SAFETY-proof discipline. zerocopy/{cuda,egl,vulkan}.rs are part of the unsafe-audited set (#![deny(clippy::undocumented_unsafe_blocks)], every unsafe carries a // SAFETY:). Any fix updates those proofs to reflect the new lifetime/validity guarantees.

10. Reproduction environment / artifacts

• Box: bazzite@192.168.1.181 (sudo bazzite), Bazzite F44 (bazzite-deck-nvidia:testing), RTX 4090, NVIDIA open 610.43.02. See [[gamemode-onglass-181-2026-07-06]] for deploy/access details.
• Trigger: stream a game-mode session with PUNKTFUNK_ZEROCOPY=1, then switch the box Game Mode → KDE desktop mid-stream (the session watcher rebuilds to KWin → tiled EGL/CUDA capture → crash).
• The coredump was present under coredumpctl on the box at the time of writing (may age out); the backtrace in §2 is captured above.

11. Related

• design/gamemode-and-dedicated-sessions.md (the branch this surfaced under — not the cause).
• design/session-aware-host-followups.md (Game↔Desktop switch behavior; F44 GPU instability #1).
• design/gpu-contention-investigation.md / design/host-latency-plan.md (zero-copy path context).
• crates/punktfunk-host/src/linux/zerocopy/{egl,cuda,vulkan}.rs, capture/linux/pipewire.rs.
• CLAUDE.md: "GPU zero-copy on all paths (tiled dmabuf → EGL/GL → CUDA; LINEAR dmabuf → Vulkan bridge → CUDA)" and the PUNKTFUNK_ZEROCOPY semantics (ON for VAAPI/AMD/Intel with a CPU downgrade; OFF/opt-in for NVENC).