punktfunk/design/zerocopy-worker-isolation.md

# Zero-copy capture hardening — GPU-import worker isolation

> **Status: IMPLEMENTED + on-glass validated (2026-07-06).** This is the implementation
> plan + decision record for the crash described in
> [`zerocopy-hardening-handoff.md`](zerocopy-hardening-handoff.md) (host SIGSEGV inside
> `libnvidia-eglcore` via `cuGraphicsMapResources` when the compositor invalidated an imported
> dmabuf mid-map, observed on the Bazzite F44 Game→Desktop switch). Validated on the RTX 5070 Ti /
> GNOME box (.21): the isolated worker carries frames at **p50 1.30 ms** end-to-end (NV12, 1800
> frames 0-mismatched), and a `kill -9` of the worker mid-stream is survived by the host and
> recovered — poison → `capture lost — rebuilding pipeline in place` → a fresh worker in **~185 ms**
> → streaming resumes (2385 frames, 0 mismatched, one 33 ms blip at the rebuild seam). See §6.

## 1. The decision: isolate, don't (only) prevent

The handoff's §9 framed two directions — *prevent the stale resource* vs *isolate the crash*.
The audit (§3 below) shows our per-frame lifetime discipline is already correct: the `EGLImage`
is created and destroyed strictly inside the PipeWire `on_process` callback while the buffer is
held (not requeued), and the CUDA-registered textures are **our own GL render targets**, never
wrappers around producer buffers. The invalidation that crashed the host is **external** —
a compositor crash (or GPU channel wreckage from the surrounding plasmashell/Xwayland core dumps)
yanked the dmabuf's GPU-side state while the driver executed our in-flight GL sampling + CUDA map.
No in-process ordering fix can close that race, and a driver SIGSEGV is not catchable.

So the fix is **process isolation**: the entire `EglImporter` (tiled dmabuf → EGL/GL → CUDA *and*
LINEAR dmabuf → Vulkan bridge → CUDA) moves into a small per-capture **worker subprocess**. If the
driver faults, the *worker* dies; the host observes a dead socket, fails the frame/capture cleanly,
and the existing capture-loss rebuild path (`gamestream/stream.rs`, `punktfunk1.rs`) takes over —
which is exactly what already happens today on the safe SHM path when a compositor goes away.

What is deliberately **not** isolated:

- **SHM/CPU capture** — no GPU import, nothing to contain.
- **VAAPI passthrough** (AMD/Intel) — capture only `dup`s the dmabuf fd; the GPU import happens in
  the encoder (Mesa VA, which reports errors rather than faulting; no observed crashes). Out of
  scope here.
- **NVENC itself** — libavcodec/NVENC surface errors as return codes; if the GPU is globally
  wedged the encoder errors and the session rebuilds. Isolating encode would mean shipping a
  session-wide media-pipeline process, far beyond this fix.

## 2. Architecture

```
host process                                worker process (punktfunk-host zerocopy-worker)
────────────                                ───────────────────────────────────────────────
PipeWire on_process                          EGLDisplay + GL ctx + CUDA ctx + VkBridge
  │ dmabuf fd (held, fence-waited)             │
  ├── IMPORT{key,geometry} + fd ──────────────▶│ eglCreateImage → GL blit/NV12 convert
  │        (SCM_RIGHTS, first sight per key)   │ → cuGraphicsMapResources → copy → unmap
  │                                            │ → pooled CUDA buffer (cuMemAllocPitch)
  │◀────────── FRAME{id [, ipc desc]} ─────────┤    exported ONCE via cuIpcGetMemHandle
  │  host opens the IPC handle once,           │
  │  wraps it as DeviceBuffer                  │
  ▼                                            │
encode thread (NVENC) reads the device ptr    │ keeps the DeviceBuffer in-flight
  │ DeviceBuffer drop                          │
  └── RELEASE{id} ────────────────────────────▶│ returns the buffer to its pool
```

- **Transport**: a `socketpair(AF_UNIX, SOCK_SEQPACKET)` created before spawn; the child end is
  `dup2`'d to fd 3 (`zerocopy-worker --fd 3`). SEQPACKET gives reliable, ordered, message-framed
  delivery; dmabuf fds ride as `SCM_RIGHTS`. Messages are small serde_json bodies (~200 B/frame;
  negligible at 240 fps).
- **Frame data never crosses the socket.** The worker's `BufferPool` allocations are exported once
  each via `cuIpcGetMemHandle`; the host `cuIpcOpenMemHandle`s each exactly once (cached by buffer
  id) and reuses the mapping as the pool recycles. Per frame the reply is just `{id}` — the copy
  was already synced (`copy_blocking`) worker-side before the reply, so the host reads complete
  pixels. The result is the same zero-CPU-touch path as before, plus one socket RTT (~tens of µs).
- **fd caching**: the host keys each PipeWire buffer by its dmabuf `st_ino` (unique per dma-buf
  object) and sends the fd only on first sight; the worker keeps the received dup (tiled: for the
  per-frame `eglCreateImage`; LINEAR: for the Vulkan `src_cache`). A format renegotiation
  (`param_changed`) sends `CLEAR_CACHE`, dropping both sides' caches — this also fixes the
  pre-existing LINEAR-path bug where `VkBridge::src_cache` was keyed by raw fd number and never
  invalidated across pool recycles (§3, trigger b). Cache desync is self-healing: a worker that no
  longer holds a key's fd (LRU eviction) answers `NeedFd` and the host retries once with the fd.
- **Lifetimes**: the worker holds each exported frame as a real `DeviceBuffer` in an in-flight map
  until `RELEASE{id}` arrives, so the existing pool `Arc` machinery keeps device memory alive
  across pool replacement while the host still reads it. Host-side, every remote `DeviceBuffer`
  holds an `Arc` of the client's shared state (socket + IPC-mapping cache), so mappings are closed
  only after the last in-flight frame drops.
- **Worker lifetime**: one worker per capture (per `pipewire_thread`), spawned from
  `/proc/self/exe`. It exits on socket EOF; the host reaps children via a global sweep list (no
  zombies). Host death ⇒ EOF ⇒ worker exit.

### Failure semantics (the point of the exercise)

| event | behavior |
|---|---|
| worker init fails (no GPU, EGL error) | handshake reports `init_err` → capture falls back to the CPU/SHM offer, same as `EglImporter::new()` failure today |
| driver SIGSEGV in the worker (the observed crash) | socket EOF → import fails with a *dead-worker* error → the capturer is **poisoned** → `next_frame`/`try_latest` return an error → the session's capture-loss rebuild runs (new capturer, new worker). **The host process survives.** |
| tiled import fails but worker alive (e.g. `EGL_BAD_MATCH` on one frame) | frame dropped; after 3 consecutive failures the capturer poisons → rebuild. It must **never** fall through to the CPU mmap path — mmap of a *tiled* dmabuf de-pads scrambled bytes (a pre-existing fallback bug; the CPU fallback was only ever correct for LINEAR). |
| LINEAR import fails | unchanged: fall back to the CPU mmap path in-stream (a LINEAR dmabuf is mappable), degraded not dead |
| repeated worker deaths | a process-wide latch (`note_gpu_import_death`, 3 consecutive deaths without a successful import between them) disables the GPU importer for the rest of the process — rebuilds renegotiate the SHM offer. Stops a wedged GPU stack from crash-looping the worker while still streaming (CPU path). A successful import resets the streak. |

### Escape hatch

`PUNKTFUNK_ZEROCOPY_INPROC=1` keeps the importer in-process (the pre-isolation behavior) for
debugging and A/B latency comparison. Default is the worker.

## 3. Audit answers for handoff §5 (which triggers are actually reachable)

- **Compositor crash / restart** — reachable (observed). Contained by the worker.
- **PipeWire buffer-pool recycle / renegotiation**:
  - *Tiled EGL path*: **not reachable in code** — the `EGLImage` lives strictly inside
    `on_process` while the buffer is held; the CUDA registrations wrap our own persistent GL
    textures, not producer buffers.
  - *LINEAR Vulkan path*: **reachable** — `VkBridge::src_cache` keyed by raw fd, never
    invalidated: a pool teardown + fd-number reuse could serve a stale imported buffer (wrong
    frame or driver fault), and old entries leaked. Fixed by st_ino keys + `CLEAR_CACHE` on
    renegotiation + an LRU cap.
- **Virtual-output teardown / mode change racing an in-flight map** — same class as compositor
  crash (external invalidation, another thread); contained by the worker.
- **Output removal** — ditto.

## 4. In-process lifetime fixes (also shipped, they harden the worker itself)

- `Nv12Blit::drop` deleted its GL textures **before** the struct fields dropped, i.e. while
  `y_tex`/`uv_tex` were still CUDA-registered. Now `RegisteredTexture::release()` runs first
  (unregister → delete), removing a driver-state hazard of exactly the class that crashed.
- `GlBlit` had **no** `Drop` — its GL program/VAO/FBO/textures leaked on every size change and on
  importer teardown. Now mirrors `Nv12Blit` (release registrations, then delete GL objects).

## 5. Residual risks, accepted

- A worker death while the encode thread still holds an IPC-mapped frame: the exporting process is
  gone; the host-side mapping stays open until the `DeviceBuffer` drops. CUDA surfaces this as a
  copy error at worst (encode error → session rebuild), not a host fault.
- The VAAPI encoder's in-host VA dmabuf import (Mesa) keeps its current exposure; no NVIDIA-class
  faults observed there.
- `cuIpcOpenMemHandle` requires same-device, different-process — both hold by construction.

## 6. Validation

- **GPU-less (CI / dev VM)**: protocol unit tests (framing, fd round-trip over a socketpair,
  error propagation, dead-worker detection against a mock server, latch behavior); worker-spawn
  failure path (spawning a non-worker exe ⇒ clean fallback).
- **On-glass (NVIDIA RTX 5070 Ti + GNOME/Mutter, .21, 2026-07-06)** — steps 1–2 **PASSED**:
  1. streamed `PUNKTFUNK_ZEROCOPY=1` through the worker (`zerocopy import worker ready` →
     `zero-copy GPU import isolated in a worker process` → `dmabuf imported to CUDA … nv12=true`),
     end-to-end **p50 1.30 ms** (1800 frames, 0 mismatched) — parity with the pre-isolation path;
  2. `kill -9` the worker mid-stream → host **survived**; the next import logged
     `tiled GPU import lost — failing this capture for rebuild … Broken pipe … dead=true`, then
     `capture lost — rebuilding pipeline in place, rebuild=1`, a **fresh worker (new pid) in
     ~185 ms**, and streaming resumed (2385 frames, 0 mismatched; single 33 ms frame at the seam).
     The `worker-ready` count was 2 (original + rebuild), confirming the respawn.
  Still pending: 3. a real compositor kill/restart mid-stream on a KWin box (the exact original
  trigger — a `kill -9` of the worker is a strictly harsher event, so this is corroboration not a
  gap); 4. `nv12-selftest` (in-process path untouched). *Note: on a static virtual desktop the
  dead-worker detection only fires once a new frame triggers an import — realistic (a running game
  produces continuous frames) but it means an idle desktop can sit poisoned-but-quiet briefly.*

## 7. Files

- `crates/punktfunk-host/src/linux/zerocopy/proto.rs` — message types + SEQPACKET/SCM_RIGHTS I/O.
- `crates/punktfunk-host/src/linux/zerocopy/worker.rs` — worker main loop (`zerocopy-worker`),
  backend trait (testable), EGL/CUDA backend.
- `crates/punktfunk-host/src/linux/zerocopy/client.rs` — `RemoteImporter` (spawn, handshake, IPC
  mapping cache, release plumbing, reaping) + the `Importer` enum (Remote | InProc).
- `crates/punktfunk-host/src/linux/zerocopy/cuda.rs` — CUDA IPC entry points; remote-release
  `DeviceBuffer`s.
- `crates/punktfunk-host/src/linux/zerocopy/egl.rs` — teardown-order fixes (§4).
- `crates/punktfunk-host/src/capture/linux/mod.rs` — `Importer` wiring, tiled-failure poisoning,
  death latch, `CLEAR_CACHE` on renegotiation.
- `crates/punktfunk-host/src/main.rs` — the hidden `zerocopy-worker` subcommand.