punktfunk/design/zerocopy-hardening-handoff.md

# Zero-copy capture hardening — issue handoff

> **Status: HANDOFF — issue description only (2026-07-06).** This document describes a reproduced
> host **SIGSEGV** in the Linux zero-copy capture path. It deliberately does **not** prescribe a fix —
> the next agent plans the implementation. Everything below is observed fact + root-cause analysis;
> the "Considerations / open questions" section frames the solution space without committing to one.
>
> **This is a pre-existing capture-layer issue, NOT a regression from the gamemode/dedicated-sessions
> work** (`design/gamemode-and-dedicated-sessions.md`). The crashing code
> (`crates/punktfunk-host/src/linux/zerocopy/{cuda,egl}.rs`, `capture/linux/pipewire.rs`) is untouched
> by that branch; it merely surfaced during on-glass validation of it.

## 1. What happened

On **`.181`** (Bazzite F44, RTX 4090, NVIDIA open driver 610.43.02 — see
`[[gamemode-onglass-181-2026-07-06]]`), streaming a game at the client's mode worked smoothly with
**zero-copy enabled** (`PUNKTFUNK_ZEROCOPY=1`). When the user then **switched the box from Steam Game
Mode to the KDE/Plasma desktop mid-stream**, the host process **crashed (SIGSEGV, core dumped)** and
the client saw "session ended". systemd auto-restarted the host ~3 s later.

The host's own logic on the switch was **correct** up to the crash — the session watcher detected
`Gaming → DesktopKde`, bumped the session epoch, rebuilt the backend to KWin, brought up the virtual
output at 5120×1440@240, set it sole desktop, and began PipeWire capture. The crash came **after**
that, the first time the capture thread mapped a KWin frame into CUDA.

## 2. The crash (backtrace)

`coredumpctl` for the host process, crashing thread:

```
Signal: 11 (SEGV)
#0  libnvidia-eglcore.so.610.43.02          ← SIGSEGV inside the NVIDIA driver
#1  libnvidia-eglcore.so.610.43.02
#2  libnvidia-eglcore.so.610.43.02
#3  libnvidia-eglcore.so.610.43.02
#4  libEGL_nvidia.so.0
#5  libcuda.so.1
#6  libcuda.so.1
#7  cuGraphicsMapResources                   (libcuda.so.1)
#8  punktfunk_host::zerocopy::cuda::RegisteredTexture::copy_mapped_plane
#9  punktfunk_host::zerocopy::egl::EglImporter::import_inner
#10 punktfunk_host::capture::linux::pipewire::pipewire_thread::{{closure}}   (the PipeWire on_process callback)
#11 pipewire::stream::…::on_process
#12 libpipewire-0.3.so.0  (impl_node_process_input → process_node → node_on_fd_events → pw_main_loop_run)
```

The fault is **inside `libnvidia-eglcore`, reached through `cuGraphicsMapResources`** — i.e. CUDA was
asked to map a GL/EGL-imported resource, and the driver dereferenced GPU state that was no longer
valid.

## 3. The code path

This is the **tiled-dmabuf zero-copy path** used for compositors that hand out **tiled** buffers
(KWin, and the NVIDIA tiled path generally): PipeWire dmabuf → **EGL/GL import** (`EglImporter`,
`crates/punktfunk-host/src/linux/zerocopy/egl.rs`) → **register as a CUDA graphics resource** and
**map per frame** (`RegisteredTexture::copy_mapped_plane` / `copy_mapped_to`,
`crates/punktfunk-host/src/linux/zerocopy/cuda.rs:810-848` and the sibling `copy_mapped_plane` below
it). The crash is at the `cuGraphicsMapResources(1, &mut self.resource, …)` call
(`cuda.rs:824-828`), driven per frame from the PipeWire `on_process` callback
(`capture/linux/pipewire.rs`, the closure at backtrace frame #10).

The relevant `// SAFETY:` proof (`cuda.rs:814-823`) reasons that `self.resource` is a valid
`CUgraphicsResource` from a successful `register_gl`, the GL+CUDA contexts are current, and the
map/unmap pair is balanced with the copy synced before unmap. **That reasoning is sound for a
well-behaved compositor that keeps the imported dmabuf alive across the map** — it does not cover the
case where the **producer invalidates the underlying buffer/texture out from under an in-flight
map**.

**Not affected on this box:** the **gamescope** path (LINEAR dmabuf → **Vulkan bridge** → CUDA,
`linux/zerocopy/vulkan.rs`) did **not** crash — game-mode streaming was smooth. And the **non-zero-copy
SHM/CPU path** (`PUNKTFUNK_ZEROCOPY=0`, the NVENC default) has no EGL/CUDA import at all, so there is
nothing for the driver to fault on. So the fault is specific to the **EGL/GL→CUDA tiled import**, not
zero-copy in general.

## 4. Root cause

`cuGraphicsMapResources` faulted **inside the driver** on GPU state that had become invalid — almost
certainly because the **KWin compositor freed/recycled (or crashed with) the dmabuf** that our
`EglImporter` had imported and registered, while our capture thread was mapping it for the next frame.
Strong corroborating evidence: in the **same 8-second window**, the box logged core dumps for
`plasmashell` (×2), `Xwayland`, `gamescope`, and `mangoapp`. So the F44 Game↔Desktop transition on
this box is itself highly unstable, and the KWin buffer our zero-copy path held a handle to almost
certainly went away mid-map.

**Why this is a real host-robustness gap, not just "the box is flaky":** a **SIGSEGV inside a
closed-source driver (`libnvidia-eglcore`) is not catchable from Rust** — it is not a `Result`, not a
Rust panic, not something a `catch_unwind` can contain. So *any* time the producer's buffer can vanish
between "we hold a CUDA graphics resource for it" and "we map it" (compositor crash, buffer-pool
recycle, output/mode teardown, hot-unplug), the driver can take the whole host process down. A capture
pipeline that must survive a compositor going away (which the host already tries to do — it has a
capture-loss → rebuild path) cannot rely on `cuGraphicsMapResources` returning an error on a stale
resource; the driver may just crash instead.

## 5. Trigger conditions (what invalidates the imported buffer)

The observed trigger was a **compositor crash during a Game→Desktop switch**, but the same class of
fault can be reached by anything that frees/recycles the imported dmabuf or its GL texture while a map
is in flight or a `RegisteredTexture` still references it:
- compositor crash / restart (observed);
- normal PipeWire buffer-pool recycle / renegotiation (format change, buffer count change) where a
  registered texture outlives the buffer it wrapped;
- virtual-output teardown / mode change (e.g. the mid-stream `Reconfigure`, the session-switch
  rebuild) racing an in-flight map;
- output removal / disconnect.
The next agent should establish **which of these are actually reachable** in the current code (the
per-frame registration/lifetime in `EglImporter`/`RegisteredTexture` vs. the PipeWire buffer
lifecycle) versus only the compositor-crash case.

## 6. Scope

- **Affected:** the EGL/GL→CUDA tiled-import path (`zerocopy::egl` + `zerocopy::cuda`), driven from
  `capture/linux/pipewire.rs`. On NVIDIA this is used for tiled dmabufs (KWin desktop capture is the
  concrete case here; the NVIDIA tiled path in general).
- **Likely also worth auditing (same class):** the **Vulkan bridge** path (`zerocopy::vulkan.rs`, LINEAR
  dmabuf → Vulkan → CUDA) — it did not crash here, but it imports external dmabufs into the GPU too and
  may have the same "producer freed the buffer mid-use" exposure; confirm whether it validates/owns the
  buffer lifetime differently.
- **Not affected:** the SHM/CPU capture path (no GPU import).

## 7. What is NOT the cause (to save the next agent time)

- **Not the gamemode/dedicated-sessions branch.** That branch's switch logic worked correctly
  (epoch bump, watcher rebuild to KWin, virtual output up); the crash is downstream in pre-existing
  capture code it doesn't touch.
- **Not a `.desktop`/KWin authorization problem.** The KWin virtual output was created and set sole
  desktop successfully — auth was fine.
- **Not the gamescope "out of buffers" issue** from the same validation session (that was a separate
  gamescope-3.16.19 PipeWire-node limitation on SHM). This is a hard driver SEGV on the GPU-import path.

## 8. Observed mitigation (already available, not the fix)

Setting **`PUNKTFUNK_ZEROCOPY=0`** (SHM/CPU path — the NVENC default anyway; the box had it forced
`=1`) removes the EGL/CUDA import, so this crash cannot occur and a compositor going away degrades to a
graceful capture-loss rebuild. Cost: a CPU copy per frame (higher latency than the zero-copy stream the
user measured). This is an operational workaround, **not** a code fix, and it forfeits zero-copy on the
desktop path.

## 9. Considerations / open questions for the implementation plan (do not treat as a prescription)

These frame the solution space; the next agent decides.
- **A driver SIGSEGV is uncatchable in-process.** Any design that "handles" this by wrapping the FFI
  call in error handling will not work — the process is already dead. So the fix has to be about
  **never handing the driver a resource that can be stale**, or **isolating the GPU-import work** so a
  driver crash doesn't take the streaming host down. Both directions are open:
  - *Prevent-the-stale-resource* directions to evaluate: strict per-frame import/register/map/unmap
    lifetime tied to the exact PipeWire buffer being processed (so no `RegisteredTexture` outlives its
    buffer); detecting compositor/output teardown and stopping capture **before** the next map;
    reconciling the EGL texture / CUDA resource lifetime with PipeWire's buffer-recycle events.
    Establish whether the current code can ever map a resource whose buffer PipeWire has already
    recycled/removed.
  - *Isolate-the-crash* directions to evaluate: whether the GPU import belongs in a **separate process**
    (like the Windows two-process/DDA isolation model) so a driver SEGV is contained and the session
    can rebuild — heavier, but the only thing that truly survives an unpreventable driver fault.
- **Per-backend / per-buffer-type routing.** The Vulkan-bridge path did not crash; the SHM path is
  safe. A plan might route tiled dmabufs (KWin) away from the fragile EGL/CUDA path, or only enable the
  EGL/CUDA path where the producer's buffer lifetime is guaranteed. Decide whether the fix is
  "harden the EGL/CUDA path" vs. "don't use it for producers that can pull buffers."
- **Interaction with the existing capture-loss rebuild.** The host already rebuilds on capture loss;
  the goal is to reach that path on producer teardown **instead of** the driver crash. Understand why
  the crash beats the capture-loss detection today (the map happens in the PipeWire `on_process`
  callback before any loss is observed).
- **Reproducibility.** This was observed on a box whose KDE was *itself* crashing (F44 plasmashell/
  Xwayland). To isolate "our zero-copy path is fragile" from "the compositor crashed," reproduce on a
  **stable** KDE/NVIDIA box — force a buffer invalidation (output teardown / renegotiation / a scripted
  compositor restart) mid-capture and confirm the same `cuGraphicsMapResources` fault without the
  surrounding compositor chaos. That also tells you whether the non-crash triggers in §5 are real.
- **Keep the SAFETY-proof discipline.** `zerocopy/{cuda,egl,vulkan}.rs` are part of the unsafe-audited
  set (`#![deny(clippy::undocumented_unsafe_blocks)]`, every `unsafe` carries a `// SAFETY:`). Any fix
  updates those proofs to reflect the new lifetime/validity guarantees.

## 10. Reproduction environment / artifacts

- Box: `bazzite@192.168.1.181` (sudo `bazzite`), Bazzite F44 (`bazzite-deck-nvidia:testing`), RTX 4090,
  NVIDIA open 610.43.02. See `[[gamemode-onglass-181-2026-07-06]]` for deploy/access details.
- Trigger: stream a game-mode session with `PUNKTFUNK_ZEROCOPY=1`, then switch the box Game Mode →
  KDE desktop mid-stream (the session watcher rebuilds to KWin → tiled EGL/CUDA capture → crash).
- The coredump was present under `coredumpctl` on the box at the time of writing (may age out); the
  backtrace in §2 is captured above.

## 11. Related

- `design/gamemode-and-dedicated-sessions.md` (the branch this surfaced under — not the cause).
- `design/session-aware-host-followups.md` (Game↔Desktop switch behavior; F44 GPU instability #1).
- `design/gpu-contention-investigation.md` / `design/host-latency-plan.md` (zero-copy path context).
- `crates/punktfunk-host/src/linux/zerocopy/{egl,cuda,vulkan}.rs`, `capture/linux/pipewire.rs`.
- CLAUDE.md: "GPU **zero-copy** on all paths (tiled dmabuf → EGL/GL → CUDA; LINEAR dmabuf → **Vulkan
  bridge** → CUDA)" and the `PUNKTFUNK_ZEROCOPY` semantics (ON for VAAPI/AMD/Intel with a CPU downgrade;
  OFF/opt-in for NVENC).