punktfunk/design/zerocopy-worker-isolation.md

Zero-copy capture hardening — GPU-import worker isolation

Status: IMPLEMENTED + on-glass validated (2026-07-06). This is the implementation plan + decision record for the crash described in zerocopy-hardening-handoff.md (host SIGSEGV inside libnvidia-eglcore via cuGraphicsMapResources when the compositor invalidated an imported dmabuf mid-map, observed on the Bazzite F44 Game→Desktop switch). Validated on the RTX 5070 Ti / GNOME box (.21): the isolated worker carries frames at p50 1.30 ms end-to-end (NV12, 1800 frames 0-mismatched), and a kill -9 of the worker mid-stream is survived by the host and recovered — poison → capture lost — rebuilding pipeline in place → a fresh worker in ~185 ms → streaming resumes (2385 frames, 0 mismatched, one 33 ms blip at the rebuild seam). See §6.

1. The decision: isolate, don't (only) prevent

The handoff's §9 framed two directions — prevent the stale resource vs isolate the crash. The audit (§3 below) shows our per-frame lifetime discipline is already correct: the EGLImage is created and destroyed strictly inside the PipeWire on_process callback while the buffer is held (not requeued), and the CUDA-registered textures are our own GL render targets, never wrappers around producer buffers. The invalidation that crashed the host is external — a compositor crash (or GPU channel wreckage from the surrounding plasmashell/Xwayland core dumps) yanked the dmabuf's GPU-side state while the driver executed our in-flight GL sampling + CUDA map. No in-process ordering fix can close that race, and a driver SIGSEGV is not catchable.

So the fix is process isolation: the entire EglImporter (tiled dmabuf → EGL/GL → CUDA and LINEAR dmabuf → Vulkan bridge → CUDA) moves into a small per-capture worker subprocess. If the driver faults, the worker dies; the host observes a dead socket, fails the frame/capture cleanly, and the existing capture-loss rebuild path (gamestream/stream.rs, punktfunk1.rs) takes over — which is exactly what already happens today on the safe SHM path when a compositor goes away.

What is deliberately not isolated:

• SHM/CPU capture — no GPU import, nothing to contain.
• VAAPI passthrough (AMD/Intel) — capture only dups the dmabuf fd; the GPU import happens in the encoder (Mesa VA, which reports errors rather than faulting; no observed crashes). Out of scope here.
• NVENC itself — libavcodec/NVENC surface errors as return codes; if the GPU is globally wedged the encoder errors and the session rebuilds. Isolating encode would mean shipping a session-wide media-pipeline process, far beyond this fix.

2. Architecture

host process                                worker process (punktfunk-host zerocopy-worker)
────────────                                ───────────────────────────────────────────────
PipeWire on_process                          EGLDisplay + GL ctx + CUDA ctx + VkBridge
  │ dmabuf fd (held, fence-waited)             │
  ├── IMPORT{key,geometry} + fd ──────────────▶│ eglCreateImage → GL blit/NV12 convert
  │        (SCM_RIGHTS, first sight per key)   │ → cuGraphicsMapResources → copy → unmap
  │                                            │ → pooled CUDA buffer (cuMemAllocPitch)
  │◀────────── FRAME{id [, ipc desc]} ─────────┤    exported ONCE via cuIpcGetMemHandle
  │  host opens the IPC handle once,           │
  │  wraps it as DeviceBuffer                  │
  ▼                                            │
encode thread (NVENC) reads the device ptr    │ keeps the DeviceBuffer in-flight
  │ DeviceBuffer drop                          │
  └── RELEASE{id} ────────────────────────────▶│ returns the buffer to its pool

• Transport: a socketpair(AF_UNIX, SOCK_SEQPACKET) created before spawn; the child end is dup2'd to fd 3 (zerocopy-worker --fd 3). SEQPACKET gives reliable, ordered, message-framed delivery; dmabuf fds ride as SCM_RIGHTS. Messages are small serde_json bodies (~200 B/frame; negligible at 240 fps).
• Frame data never crosses the socket. The worker's BufferPool allocations are exported once each via cuIpcGetMemHandle; the host cuIpcOpenMemHandles each exactly once (cached by buffer id) and reuses the mapping as the pool recycles. Per frame the reply is just {id} — the copy was already synced (copy_blocking) worker-side before the reply, so the host reads complete pixels. The result is the same zero-CPU-touch path as before, plus one socket RTT (~tens of µs).
• fd caching: the host keys each PipeWire buffer by its dmabuf st_ino (unique per dma-buf object) and sends the fd only on first sight; the worker keeps the received dup (tiled: for the per-frame eglCreateImage; LINEAR: for the Vulkan src_cache). A format renegotiation (param_changed) sends CLEAR_CACHE, dropping both sides' caches — this also fixes the pre-existing LINEAR-path bug where VkBridge::src_cache was keyed by raw fd number and never invalidated across pool recycles (§3, trigger b). Cache desync is self-healing: a worker that no longer holds a key's fd (LRU eviction) answers NeedFd and the host retries once with the fd.
• Lifetimes: the worker holds each exported frame as a real DeviceBuffer in an in-flight map until RELEASE{id} arrives, so the existing pool Arc machinery keeps device memory alive across pool replacement while the host still reads it. Host-side, every remote DeviceBuffer holds an Arc of the client's shared state (socket + IPC-mapping cache), so mappings are closed only after the last in-flight frame drops.
• Worker lifetime: one worker per capture (per pipewire_thread), spawned from /proc/self/exe. It exits on socket EOF; the host reaps children via a global sweep list (no zombies). Host death ⇒ EOF ⇒ worker exit.

Failure semantics (the point of the exercise)

event	behavior
worker init fails (no GPU, EGL error)	handshake reports init_err → capture falls back to the CPU/SHM offer, same as EglImporter::new() failure today
driver SIGSEGV in the worker (the observed crash)	socket EOF → import fails with a dead-worker error → the capturer is poisoned → next_frame/try_latest return an error → the session's capture-loss rebuild runs (new capturer, new worker). The host process survives.
tiled import fails but worker alive (e.g. EGL_BAD_MATCH on one frame)	frame dropped; after 3 consecutive failures the capturer poisons → rebuild. It must never fall through to the CPU mmap path — mmap of a tiled dmabuf de-pads scrambled bytes (a pre-existing fallback bug; the CPU fallback was only ever correct for LINEAR).
LINEAR import fails	unchanged: fall back to the CPU mmap path in-stream (a LINEAR dmabuf is mappable), degraded not dead
repeated worker deaths	a process-wide latch (note_gpu_import_death, 3 consecutive deaths without a successful import between them) disables the GPU importer for the rest of the process — rebuilds renegotiate the SHM offer. Stops a wedged GPU stack from crash-looping the worker while still streaming (CPU path). A successful import resets the streak.

Escape hatch

PUNKTFUNK_ZEROCOPY_INPROC=1 keeps the importer in-process (the pre-isolation behavior) for debugging and A/B latency comparison. Default is the worker.

3. Audit answers for handoff §5 (which triggers are actually reachable)

• Compositor crash / restart — reachable (observed). Contained by the worker.
• PipeWire buffer-pool recycle / renegotiation:
  • Tiled EGL path: not reachable in code — the EGLImage lives strictly inside on_process while the buffer is held; the CUDA registrations wrap our own persistent GL textures, not producer buffers.
  • LINEAR Vulkan path: reachable — VkBridge::src_cache keyed by raw fd, never invalidated: a pool teardown + fd-number reuse could serve a stale imported buffer (wrong frame or driver fault), and old entries leaked. Fixed by st_ino keys + CLEAR_CACHE on renegotiation + an LRU cap.
• Virtual-output teardown / mode change racing an in-flight map — same class as compositor crash (external invalidation, another thread); contained by the worker.
• Output removal — ditto.

4. In-process lifetime fixes (also shipped, they harden the worker itself)

• Nv12Blit::drop deleted its GL textures before the struct fields dropped, i.e. while y_tex/uv_tex were still CUDA-registered. Now RegisteredTexture::release() runs first (unregister → delete), removing a driver-state hazard of exactly the class that crashed.
• GlBlit had no Drop — its GL program/VAO/FBO/textures leaked on every size change and on importer teardown. Now mirrors Nv12Blit (release registrations, then delete GL objects).

5. Residual risks, accepted

• A worker death while the encode thread still holds an IPC-mapped frame: the exporting process is gone; the host-side mapping stays open until the DeviceBuffer drops. CUDA surfaces this as a copy error at worst (encode error → session rebuild), not a host fault.
• The VAAPI encoder's in-host VA dmabuf import (Mesa) keeps its current exposure; no NVIDIA-class faults observed there.
• cuIpcOpenMemHandle requires same-device, different-process — both hold by construction.

6. Validation

• GPU-less (CI / dev VM): protocol unit tests (framing, fd round-trip over a socketpair, error propagation, dead-worker detection against a mock server, latch behavior); worker-spawn failure path (spawning a non-worker exe ⇒ clean fallback).
• On-glass (NVIDIA RTX 5070 Ti + GNOME/Mutter, .21, 2026-07-06) — steps 1–2 PASSED:
  1. streamed PUNKTFUNK_ZEROCOPY=1 through the worker (zerocopy import worker ready → zero-copy GPU import isolated in a worker process → dmabuf imported to CUDA … nv12=true), end-to-end p50 1.30 ms (1800 frames, 0 mismatched) — parity with the pre-isolation path;
  2. kill -9 the worker mid-stream → host survived; the next import logged tiled GPU import lost — failing this capture for rebuild … Broken pipe … dead=true, then capture lost — rebuilding pipeline in place, rebuild=1, a fresh worker (new pid) in ~185 ms, and streaming resumed (2385 frames, 0 mismatched; single 33 ms frame at the seam). The worker-ready count was 2 (original + rebuild), confirming the respawn. Still pending: 3. a real compositor kill/restart mid-stream on a KWin box (the exact original trigger — a kill -9 of the worker is a strictly harsher event, so this is corroboration not a gap); 4. nv12-selftest (in-process path untouched). Note: on a static virtual desktop the dead-worker detection only fires once a new frame triggers an import — realistic (a running game produces continuous frames) but it means an idle desktop can sit poisoned-but-quiet briefly.

7. Files

• crates/punktfunk-host/src/linux/zerocopy/proto.rs — message types + SEQPACKET/SCM_RIGHTS I/O.
• crates/punktfunk-host/src/linux/zerocopy/worker.rs — worker main loop (zerocopy-worker), backend trait (testable), EGL/CUDA backend.
• crates/punktfunk-host/src/linux/zerocopy/client.rs — RemoteImporter (spawn, handshake, IPC mapping cache, release plumbing, reaping) + the Importer enum (Remote | InProc).
• crates/punktfunk-host/src/linux/zerocopy/cuda.rs — CUDA IPC entry points; remote-release DeviceBuffers.
• crates/punktfunk-host/src/linux/zerocopy/egl.rs — teardown-order fixes (§4).
• crates/punktfunk-host/src/capture/linux/mod.rs — Importer wiring, tiled-failure poisoning, death latch, CLEAR_CACHE on renegotiation.
• crates/punktfunk-host/src/main.rs — the hidden zerocopy-worker subcommand.