diff --git a/design/zerocopy-hardening-handoff.md b/design/zerocopy-hardening-handoff.md new file mode 100644 index 0000000..158b59b --- /dev/null +++ b/design/zerocopy-hardening-handoff.md @@ -0,0 +1,188 @@ +# Zero-copy capture hardening — issue handoff + +> **Status: HANDOFF — issue description only (2026-07-06).** This document describes a reproduced +> host **SIGSEGV** in the Linux zero-copy capture path. It deliberately does **not** prescribe a fix — +> the next agent plans the implementation. Everything below is observed fact + root-cause analysis; +> the "Considerations / open questions" section frames the solution space without committing to one. +> +> **This is a pre-existing capture-layer issue, NOT a regression from the gamemode/dedicated-sessions +> work** (`design/gamemode-and-dedicated-sessions.md`). The crashing code +> (`crates/punktfunk-host/src/linux/zerocopy/{cuda,egl}.rs`, `capture/linux/pipewire.rs`) is untouched +> by that branch; it merely surfaced during on-glass validation of it. + +## 1. What happened + +On **`.181`** (Bazzite F44, RTX 4090, NVIDIA open driver 610.43.02 — see +`[[gamemode-onglass-181-2026-07-06]]`), streaming a game at the client's mode worked smoothly with +**zero-copy enabled** (`PUNKTFUNK_ZEROCOPY=1`). When the user then **switched the box from Steam Game +Mode to the KDE/Plasma desktop mid-stream**, the host process **crashed (SIGSEGV, core dumped)** and +the client saw "session ended". systemd auto-restarted the host ~3 s later. + +The host's own logic on the switch was **correct** up to the crash — the session watcher detected +`Gaming → DesktopKde`, bumped the session epoch, rebuilt the backend to KWin, brought up the virtual +output at 5120×1440@240, set it sole desktop, and began PipeWire capture. The crash came **after** +that, the first time the capture thread mapped a KWin frame into CUDA. + +## 2. The crash (backtrace) + +`coredumpctl` for the host process, crashing thread: + +``` +Signal: 11 (SEGV) +#0 libnvidia-eglcore.so.610.43.02 ← SIGSEGV inside the NVIDIA driver +#1 libnvidia-eglcore.so.610.43.02 +#2 libnvidia-eglcore.so.610.43.02 +#3 libnvidia-eglcore.so.610.43.02 +#4 libEGL_nvidia.so.0 +#5 libcuda.so.1 +#6 libcuda.so.1 +#7 cuGraphicsMapResources (libcuda.so.1) +#8 punktfunk_host::zerocopy::cuda::RegisteredTexture::copy_mapped_plane +#9 punktfunk_host::zerocopy::egl::EglImporter::import_inner +#10 punktfunk_host::capture::linux::pipewire::pipewire_thread::{{closure}} (the PipeWire on_process callback) +#11 pipewire::stream::…::on_process +#12 libpipewire-0.3.so.0 (impl_node_process_input → process_node → node_on_fd_events → pw_main_loop_run) +``` + +The fault is **inside `libnvidia-eglcore`, reached through `cuGraphicsMapResources`** — i.e. CUDA was +asked to map a GL/EGL-imported resource, and the driver dereferenced GPU state that was no longer +valid. + +## 3. The code path + +This is the **tiled-dmabuf zero-copy path** used for compositors that hand out **tiled** buffers +(KWin, and the NVIDIA tiled path generally): PipeWire dmabuf → **EGL/GL import** (`EglImporter`, +`crates/punktfunk-host/src/linux/zerocopy/egl.rs`) → **register as a CUDA graphics resource** and +**map per frame** (`RegisteredTexture::copy_mapped_plane` / `copy_mapped_to`, +`crates/punktfunk-host/src/linux/zerocopy/cuda.rs:810-848` and the sibling `copy_mapped_plane` below +it). The crash is at the `cuGraphicsMapResources(1, &mut self.resource, …)` call +(`cuda.rs:824-828`), driven per frame from the PipeWire `on_process` callback +(`capture/linux/pipewire.rs`, the closure at backtrace frame #10). + +The relevant `// SAFETY:` proof (`cuda.rs:814-823`) reasons that `self.resource` is a valid +`CUgraphicsResource` from a successful `register_gl`, the GL+CUDA contexts are current, and the +map/unmap pair is balanced with the copy synced before unmap. **That reasoning is sound for a +well-behaved compositor that keeps the imported dmabuf alive across the map** — it does not cover the +case where the **producer invalidates the underlying buffer/texture out from under an in-flight +map**. + +**Not affected on this box:** the **gamescope** path (LINEAR dmabuf → **Vulkan bridge** → CUDA, +`linux/zerocopy/vulkan.rs`) did **not** crash — game-mode streaming was smooth. And the **non-zero-copy +SHM/CPU path** (`PUNKTFUNK_ZEROCOPY=0`, the NVENC default) has no EGL/CUDA import at all, so there is +nothing for the driver to fault on. So the fault is specific to the **EGL/GL→CUDA tiled import**, not +zero-copy in general. + +## 4. Root cause + +`cuGraphicsMapResources` faulted **inside the driver** on GPU state that had become invalid — almost +certainly because the **KWin compositor freed/recycled (or crashed with) the dmabuf** that our +`EglImporter` had imported and registered, while our capture thread was mapping it for the next frame. +Strong corroborating evidence: in the **same 8-second window**, the box logged core dumps for +`plasmashell` (×2), `Xwayland`, `gamescope`, and `mangoapp`. So the F44 Game↔Desktop transition on +this box is itself highly unstable, and the KWin buffer our zero-copy path held a handle to almost +certainly went away mid-map. + +**Why this is a real host-robustness gap, not just "the box is flaky":** a **SIGSEGV inside a +closed-source driver (`libnvidia-eglcore`) is not catchable from Rust** — it is not a `Result`, not a +Rust panic, not something a `catch_unwind` can contain. So *any* time the producer's buffer can vanish +between "we hold a CUDA graphics resource for it" and "we map it" (compositor crash, buffer-pool +recycle, output/mode teardown, hot-unplug), the driver can take the whole host process down. A capture +pipeline that must survive a compositor going away (which the host already tries to do — it has a +capture-loss → rebuild path) cannot rely on `cuGraphicsMapResources` returning an error on a stale +resource; the driver may just crash instead. + +## 5. Trigger conditions (what invalidates the imported buffer) + +The observed trigger was a **compositor crash during a Game→Desktop switch**, but the same class of +fault can be reached by anything that frees/recycles the imported dmabuf or its GL texture while a map +is in flight or a `RegisteredTexture` still references it: +- compositor crash / restart (observed); +- normal PipeWire buffer-pool recycle / renegotiation (format change, buffer count change) where a + registered texture outlives the buffer it wrapped; +- virtual-output teardown / mode change (e.g. the mid-stream `Reconfigure`, the session-switch + rebuild) racing an in-flight map; +- output removal / disconnect. +The next agent should establish **which of these are actually reachable** in the current code (the +per-frame registration/lifetime in `EglImporter`/`RegisteredTexture` vs. the PipeWire buffer +lifecycle) versus only the compositor-crash case. + +## 6. Scope + +- **Affected:** the EGL/GL→CUDA tiled-import path (`zerocopy::egl` + `zerocopy::cuda`), driven from + `capture/linux/pipewire.rs`. On NVIDIA this is used for tiled dmabufs (KWin desktop capture is the + concrete case here; the NVIDIA tiled path in general). +- **Likely also worth auditing (same class):** the **Vulkan bridge** path (`zerocopy::vulkan.rs`, LINEAR + dmabuf → Vulkan → CUDA) — it did not crash here, but it imports external dmabufs into the GPU too and + may have the same "producer freed the buffer mid-use" exposure; confirm whether it validates/owns the + buffer lifetime differently. +- **Not affected:** the SHM/CPU capture path (no GPU import). + +## 7. What is NOT the cause (to save the next agent time) + +- **Not the gamemode/dedicated-sessions branch.** That branch's switch logic worked correctly + (epoch bump, watcher rebuild to KWin, virtual output up); the crash is downstream in pre-existing + capture code it doesn't touch. +- **Not a `.desktop`/KWin authorization problem.** The KWin virtual output was created and set sole + desktop successfully — auth was fine. +- **Not the gamescope "out of buffers" issue** from the same validation session (that was a separate + gamescope-3.16.19 PipeWire-node limitation on SHM). This is a hard driver SEGV on the GPU-import path. + +## 8. Observed mitigation (already available, not the fix) + +Setting **`PUNKTFUNK_ZEROCOPY=0`** (SHM/CPU path — the NVENC default anyway; the box had it forced +`=1`) removes the EGL/CUDA import, so this crash cannot occur and a compositor going away degrades to a +graceful capture-loss rebuild. Cost: a CPU copy per frame (higher latency than the zero-copy stream the +user measured). This is an operational workaround, **not** a code fix, and it forfeits zero-copy on the +desktop path. + +## 9. Considerations / open questions for the implementation plan (do not treat as a prescription) + +These frame the solution space; the next agent decides. +- **A driver SIGSEGV is uncatchable in-process.** Any design that "handles" this by wrapping the FFI + call in error handling will not work — the process is already dead. So the fix has to be about + **never handing the driver a resource that can be stale**, or **isolating the GPU-import work** so a + driver crash doesn't take the streaming host down. Both directions are open: + - *Prevent-the-stale-resource* directions to evaluate: strict per-frame import/register/map/unmap + lifetime tied to the exact PipeWire buffer being processed (so no `RegisteredTexture` outlives its + buffer); detecting compositor/output teardown and stopping capture **before** the next map; + reconciling the EGL texture / CUDA resource lifetime with PipeWire's buffer-recycle events. + Establish whether the current code can ever map a resource whose buffer PipeWire has already + recycled/removed. + - *Isolate-the-crash* directions to evaluate: whether the GPU import belongs in a **separate process** + (like the Windows two-process/DDA isolation model) so a driver SEGV is contained and the session + can rebuild — heavier, but the only thing that truly survives an unpreventable driver fault. +- **Per-backend / per-buffer-type routing.** The Vulkan-bridge path did not crash; the SHM path is + safe. A plan might route tiled dmabufs (KWin) away from the fragile EGL/CUDA path, or only enable the + EGL/CUDA path where the producer's buffer lifetime is guaranteed. Decide whether the fix is + "harden the EGL/CUDA path" vs. "don't use it for producers that can pull buffers." +- **Interaction with the existing capture-loss rebuild.** The host already rebuilds on capture loss; + the goal is to reach that path on producer teardown **instead of** the driver crash. Understand why + the crash beats the capture-loss detection today (the map happens in the PipeWire `on_process` + callback before any loss is observed). +- **Reproducibility.** This was observed on a box whose KDE was *itself* crashing (F44 plasmashell/ + Xwayland). To isolate "our zero-copy path is fragile" from "the compositor crashed," reproduce on a + **stable** KDE/NVIDIA box — force a buffer invalidation (output teardown / renegotiation / a scripted + compositor restart) mid-capture and confirm the same `cuGraphicsMapResources` fault without the + surrounding compositor chaos. That also tells you whether the non-crash triggers in §5 are real. +- **Keep the SAFETY-proof discipline.** `zerocopy/{cuda,egl,vulkan}.rs` are part of the unsafe-audited + set (`#![deny(clippy::undocumented_unsafe_blocks)]`, every `unsafe` carries a `// SAFETY:`). Any fix + updates those proofs to reflect the new lifetime/validity guarantees. + +## 10. Reproduction environment / artifacts + +- Box: `bazzite@192.168.1.181` (sudo `bazzite`), Bazzite F44 (`bazzite-deck-nvidia:testing`), RTX 4090, + NVIDIA open 610.43.02. See `[[gamemode-onglass-181-2026-07-06]]` for deploy/access details. +- Trigger: stream a game-mode session with `PUNKTFUNK_ZEROCOPY=1`, then switch the box Game Mode → + KDE desktop mid-stream (the session watcher rebuilds to KWin → tiled EGL/CUDA capture → crash). +- The coredump was present under `coredumpctl` on the box at the time of writing (may age out); the + backtrace in §2 is captured above. + +## 11. Related + +- `design/gamemode-and-dedicated-sessions.md` (the branch this surfaced under — not the cause). +- `design/session-aware-host-followups.md` (Game↔Desktop switch behavior; F44 GPU instability #1). +- `design/gpu-contention-investigation.md` / `design/host-latency-plan.md` (zero-copy path context). +- `crates/punktfunk-host/src/linux/zerocopy/{egl,cuda,vulkan}.rs`, `capture/linux/pipewire.rs`. +- CLAUDE.md: "GPU **zero-copy** on all paths (tiled dmabuf → EGL/GL → CUDA; LINEAR dmabuf → **Vulkan + bridge** → CUDA)" and the `PUNKTFUNK_ZEROCOPY` semantics (ON for VAAPI/AMD/Intel with a CPU downgrade; + OFF/opt-in for NVENC).