unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 14 Wiki Activity

docs: zero-copy EGL/CUDA capture-crash hardening handoff

Browse Source 
Describes a reproduced host SIGSEGV in cuGraphicsMapResources (inside
libnvidia-eglcore), reached via zerocopy::cuda / zerocopy::egl on the tiled
EGL/GL->CUDA capture path, when the KWin dmabuf is invalidated mid-map (observed
on .181 during a Game->Desktop switch under zero-copy, with the compositor itself
crashing). Pre-existing capture-layer issue, not the gamemode work. Issue
description + root cause + solution-space considerations only -- the next agent
plans the implementation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Enrico Bühler
2026-07-06 06:26:11 +00:00
parent 04309d0ad9
commit 8ef320662b
1 changed files with 188 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
design/zerocopy-hardening-handoff.md
+188
View File
@@ -0,0 +1,188 @@
# Zero-copy capture hardening — issue handoff
> **Status: HANDOFF — issue description only (2026-07-06).** This document describes a reproduced
> host **SIGSEGV** in the Linux zero-copy capture path. It deliberately does **not** prescribe a fix —
> the next agent plans the implementation. Everything below is observed fact + root-cause analysis;
> the "Considerations / open questions" section frames the solution space without committing to one.
>
> **This is a pre-existing capture-layer issue, NOT a regression from the gamemode/dedicated-sessions
> work** (`design/gamemode-and-dedicated-sessions.md`). The crashing code
> (`crates/punktfunk-host/src/linux/zerocopy/{cuda,egl}.rs`, `capture/linux/pipewire.rs`) is untouched
> by that branch; it merely surfaced during on-glass validation of it.
## 1. What happened
On **`.181`** (Bazzite F44, RTX 4090, NVIDIA open driver 610.43.02 — see
`[[gamemode-onglass-181-2026-07-06]]`), streaming a game at the client's mode worked smoothly with
**zero-copy enabled** (`PUNKTFUNK_ZEROCOPY=1`). When the user then **switched the box from Steam Game
Mode to the KDE/Plasma desktop mid-stream**, the host process **crashed (SIGSEGV, core dumped)** and
the client saw "session ended". systemd auto-restarted the host ~3 s later.
The host's own logic on the switch was **correct** up to the crash — the session watcher detected
`Gaming → DesktopKde`, bumped the session epoch, rebuilt the backend to KWin, brought up the virtual
output at 5120×1440@240, set it sole desktop, and began PipeWire capture. The crash came **after**
that, the first time the capture thread mapped a KWin frame into CUDA.
## 2. The crash (backtrace)
`coredumpctl` for the host process, crashing thread:
```
Signal: 11 (SEGV)
#0 libnvidia-eglcore.so.610.43.02 ← SIGSEGV inside the NVIDIA driver
#1 libnvidia-eglcore.so.610.43.02
#2 libnvidia-eglcore.so.610.43.02
#3 libnvidia-eglcore.so.610.43.02
#4 libEGL_nvidia.so.0
#5 libcuda.so.1
#6 libcuda.so.1
#7 cuGraphicsMapResources (libcuda.so.1)
#8 punktfunk_host::zerocopy::cuda::RegisteredTexture::copy_mapped_plane
#9 punktfunk_host::zerocopy::egl::EglImporter::import_inner
#10 punktfunk_host::capture::linux::pipewire::pipewire_thread::{{closure}} (the PipeWire on_process callback)
#11 pipewire::stream::…::on_process
#12 libpipewire-0.3.so.0 (impl_node_process_input → process_node → node_on_fd_events → pw_main_loop_run)
```
The fault is **inside `libnvidia-eglcore`, reached through `cuGraphicsMapResources`** — i.e. CUDA was
asked to map a GL/EGL-imported resource, and the driver dereferenced GPU state that was no longer
valid.
## 3. The code path
This is the **tiled-dmabuf zero-copy path** used for compositors that hand out **tiled** buffers
(KWin, and the NVIDIA tiled path generally): PipeWire dmabuf → **EGL/GL import** (`EglImporter`,
`crates/punktfunk-host/src/linux/zerocopy/egl.rs`) → **register as a CUDA graphics resource** and
**map per frame** (`RegisteredTexture::copy_mapped_plane` / `copy_mapped_to`,
`crates/punktfunk-host/src/linux/zerocopy/cuda.rs:810-848` and the sibling `copy_mapped_plane` below
it). The crash is at the `cuGraphicsMapResources(1, &mut self.resource, …)` call
(`cuda.rs:824-828`), driven per frame from the PipeWire `on_process` callback
(`capture/linux/pipewire.rs`, the closure at backtrace frame #10).
The relevant `// SAFETY:` proof (`cuda.rs:814-823`) reasons that `self.resource` is a valid
`CUgraphicsResource` from a successful `register_gl`, the GL+CUDA contexts are current, and the
map/unmap pair is balanced with the copy synced before unmap. **That reasoning is sound for a
well-behaved compositor that keeps the imported dmabuf alive across the map** — it does not cover the
case where the **producer invalidates the underlying buffer/texture out from under an in-flight
map**.
**Not affected on this box:** the **gamescope** path (LINEAR dmabuf → **Vulkan bridge** → CUDA,
`linux/zerocopy/vulkan.rs`) did **not** crash — game-mode streaming was smooth. And the **non-zero-copy
SHM/CPU path** (`PUNKTFUNK_ZEROCOPY=0`, the NVENC default) has no EGL/CUDA import at all, so there is
nothing for the driver to fault on. So the fault is specific to the **EGL/GL→CUDA tiled import**, not
zero-copy in general.
## 4. Root cause
`cuGraphicsMapResources` faulted **inside the driver** on GPU state that had become invalid — almost
certainly because the **KWin compositor freed/recycled (or crashed with) the dmabuf** that our
`EglImporter` had imported and registered, while our capture thread was mapping it for the next frame.
Strong corroborating evidence: in the **same 8-second window**, the box logged core dumps for
`plasmashell` (×2), `Xwayland`, `gamescope`, and `mangoapp`. So the F44 Game↔Desktop transition on
this box is itself highly unstable, and the KWin buffer our zero-copy path held a handle to almost
certainly went away mid-map.
**Why this is a real host-robustness gap, not just "the box is flaky":** a **SIGSEGV inside a
closed-source driver (`libnvidia-eglcore`) is not catchable from Rust** — it is not a `Result`, not a
Rust panic, not something a `catch_unwind` can contain. So *any* time the producer's buffer can vanish
between "we hold a CUDA graphics resource for it" and "we map it" (compositor crash, buffer-pool
recycle, output/mode teardown, hot-unplug), the driver can take the whole host process down. A capture
pipeline that must survive a compositor going away (which the host already tries to do — it has a
capture-loss → rebuild path) cannot rely on `cuGraphicsMapResources` returning an error on a stale
resource; the driver may just crash instead.
## 5. Trigger conditions (what invalidates the imported buffer)
The observed trigger was a **compositor crash during a Game→Desktop switch**, but the same class of
fault can be reached by anything that frees/recycles the imported dmabuf or its GL texture while a map
is in flight or a `RegisteredTexture` still references it:
- compositor crash / restart (observed);
- normal PipeWire buffer-pool recycle / renegotiation (format change, buffer count change) where a
registered texture outlives the buffer it wrapped;
- virtual-output teardown / mode change (e.g. the mid-stream `Reconfigure`, the session-switch
rebuild) racing an in-flight map;
- output removal / disconnect.
The next agent should establish **which of these are actually reachable** in the current code (the
per-frame registration/lifetime in `EglImporter`/`RegisteredTexture` vs. the PipeWire buffer
lifecycle) versus only the compositor-crash case.
## 6. Scope
- **Affected:** the EGL/GL→CUDA tiled-import path (`zerocopy::egl` + `zerocopy::cuda`), driven from
`capture/linux/pipewire.rs`. On NVIDIA this is used for tiled dmabufs (KWin desktop capture is the
concrete case here; the NVIDIA tiled path in general).
- **Likely also worth auditing (same class):** the **Vulkan bridge** path (`zerocopy::vulkan.rs`, LINEAR
dmabuf → Vulkan → CUDA) — it did not crash here, but it imports external dmabufs into the GPU too and
may have the same "producer freed the buffer mid-use" exposure; confirm whether it validates/owns the
buffer lifetime differently.
- **Not affected:** the SHM/CPU capture path (no GPU import).
## 7. What is NOT the cause (to save the next agent time)
- **Not the gamemode/dedicated-sessions branch.** That branch's switch logic worked correctly
(epoch bump, watcher rebuild to KWin, virtual output up); the crash is downstream in pre-existing
capture code it doesn't touch.
- **Not a `.desktop`/KWin authorization problem.** The KWin virtual output was created and set sole
desktop successfully — auth was fine.
- **Not the gamescope "out of buffers" issue** from the same validation session (that was a separate
gamescope-3.16.19 PipeWire-node limitation on SHM). This is a hard driver SEGV on the GPU-import path.
## 8. Observed mitigation (already available, not the fix)
Setting **`PUNKTFUNK_ZEROCOPY=0`** (SHM/CPU path — the NVENC default anyway; the box had it forced
`=1`) removes the EGL/CUDA import, so this crash cannot occur and a compositor going away degrades to a
graceful capture-loss rebuild. Cost: a CPU copy per frame (higher latency than the zero-copy stream the
user measured). This is an operational workaround, **not** a code fix, and it forfeits zero-copy on the
desktop path.
## 9. Considerations / open questions for the implementation plan (do not treat as a prescription)
These frame the solution space; the next agent decides.
- **A driver SIGSEGV is uncatchable in-process.** Any design that "handles" this by wrapping the FFI
call in error handling will not work — the process is already dead. So the fix has to be about
**never handing the driver a resource that can be stale**, or **isolating the GPU-import work** so a
driver crash doesn't take the streaming host down. Both directions are open:
- *Prevent-the-stale-resource* directions to evaluate: strict per-frame import/register/map/unmap
lifetime tied to the exact PipeWire buffer being processed (so no `RegisteredTexture` outlives its
buffer); detecting compositor/output teardown and stopping capture **before** the next map;
reconciling the EGL texture / CUDA resource lifetime with PipeWire's buffer-recycle events.
Establish whether the current code can ever map a resource whose buffer PipeWire has already
recycled/removed.
- *Isolate-the-crash* directions to evaluate: whether the GPU import belongs in a **separate process**
(like the Windows two-process/DDA isolation model) so a driver SEGV is contained and the session
can rebuild — heavier, but the only thing that truly survives an unpreventable driver fault.
- **Per-backend / per-buffer-type routing.** The Vulkan-bridge path did not crash; the SHM path is
safe. A plan might route tiled dmabufs (KWin) away from the fragile EGL/CUDA path, or only enable the
EGL/CUDA path where the producer's buffer lifetime is guaranteed. Decide whether the fix is
"harden the EGL/CUDA path" vs. "don't use it for producers that can pull buffers."
- **Interaction with the existing capture-loss rebuild.** The host already rebuilds on capture loss;
the goal is to reach that path on producer teardown **instead of** the driver crash. Understand why
the crash beats the capture-loss detection today (the map happens in the PipeWire `on_process`
callback before any loss is observed).
- **Reproducibility.** This was observed on a box whose KDE was *itself* crashing (F44 plasmashell/
Xwayland). To isolate "our zero-copy path is fragile" from "the compositor crashed," reproduce on a
**stable** KDE/NVIDIA box — force a buffer invalidation (output teardown / renegotiation / a scripted
compositor restart) mid-capture and confirm the same `cuGraphicsMapResources` fault without the
surrounding compositor chaos. That also tells you whether the non-crash triggers in §5 are real.
- **Keep the SAFETY-proof discipline.** `zerocopy/{cuda,egl,vulkan}.rs` are part of the unsafe-audited
set (`#![deny(clippy::undocumented_unsafe_blocks)]`, every `unsafe` carries a `// SAFETY:`). Any fix
updates those proofs to reflect the new lifetime/validity guarantees.
## 10. Reproduction environment / artifacts
- Box: `bazzite@192.168.1.181` (sudo `bazzite`), Bazzite F44 (`bazzite-deck-nvidia:testing`), RTX 4090,
NVIDIA open 610.43.02. See `[[gamemode-onglass-181-2026-07-06]]` for deploy/access details.
- Trigger: stream a game-mode session with `PUNKTFUNK_ZEROCOPY=1`, then switch the box Game Mode →
KDE desktop mid-stream (the session watcher rebuilds to KWin → tiled EGL/CUDA capture → crash).
- The coredump was present under `coredumpctl` on the box at the time of writing (may age out); the
backtrace in §2 is captured above.
## 11. Related
- `design/gamemode-and-dedicated-sessions.md` (the branch this surfaced under — not the cause).
- `design/session-aware-host-followups.md` (Game↔Desktop switch behavior; F44 GPU instability #1).
- `design/gpu-contention-investigation.md` / `design/host-latency-plan.md` (zero-copy path context).
- `crates/punktfunk-host/src/linux/zerocopy/{egl,cuda,vulkan}.rs`, `capture/linux/pipewire.rs`.
- CLAUDE.md: "GPU **zero-copy** on all paths (tiled dmabuf → EGL/GL → CUDA; LINEAR dmabuf → **Vulkan
bridge** → CUDA)" and the `PUNKTFUNK_ZEROCOPY` semantics (ON for VAAPI/AMD/Intel with a CPU downgrade;
OFF/opt-in for NVENC).