refactor(host/W6.2): extract the Linux zero-copy GPU plumbing into the pf-zerocopy leaf crate
linux/zerocopy/* (CUDA context/buffers + EGL/Vulkan dmabuf import + the isolated import worker) and linux/dmabuf_fence.rs move wholesale into crates/pf-zerocopy, so the coming pf-frame vocabulary crate (FramePayload::Cuda owns a DeviceBuffer) and the pf-encode/pf-capture subsystem crates can reach the GPU plumbing without the host orchestrator in between (plan §W6). Content stays Linux-only; the crate compiles to an empty lib elsewhere, so dependents carry a plain dependency. drm_fourcc deliberately does NOT move: it consumes the frame vocabulary (PixelFormat), which sits ABOVE pf-zerocopy — it lives with capture for now and moves into pf-frame next. cuda's ffi re-export bumps pub(crate)->pub (the raw CUdeviceptr vocabulary is consumed across the crate boundary by the encode backends). A crate::zerocopy shim module keeps every existing path valid until capture/encode themselves move out. Verified: Linux clippy -D warnings (pf-zerocopy --all-targets + host nvenc,vulkan-encode,pyrowave --all-targets) + 17/17 pf-zerocopy tests + 321/321 host tests; Windows clippy nvenc,amf-qsv --all-targets Finished exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,93 @@
|
||||
//! Consumer-side implicit-fence wait for dmabuf capture (`DMA_BUF_IOCTL_EXPORT_SYNC_FILE`).
|
||||
//!
|
||||
//! Mutter renders its virtual monitor DIRECTLY into the PipeWire dmabuf and hands the buffer over
|
||||
//! at GPU-submit time. With no fencing the consumer can sample mid-render and encode the buffer's
|
||||
//! *previous* contents — the "stale/old frame" flashing on NVIDIA (KWin/gamescope blit into the
|
||||
//! buffer so they don't hit this). The producer-driven fix is PipeWire explicit sync, but
|
||||
//! Mutter+NVIDIA can't produce a sync_fd (`error alloc buffers` / no cogl sync_fd).
|
||||
//!
|
||||
//! So sync from the *consumer* side instead: a dmabuf carries its in-flight GPU work as an implicit
|
||||
//! fence on its reservation object. `DMA_BUF_IOCTL_EXPORT_SYNC_FILE` snapshots that into a sync_file
|
||||
//! fd we can `poll()` — readable once the producer's writes complete. This makes zero-copy capture
|
||||
//! race-free WITHOUT the producer doing anything, *iff* the driver actually attaches the fence. If it
|
||||
//! attaches none, the export yields an already-signaled sync_file (poll returns immediately) — no
|
||||
//! wait, no harm, and `waited=false` tells us the driver doesn't fence (so zero-copy would still race).
|
||||
|
||||
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use std::os::fd::RawFd;
|
||||
|
||||
// linux/dma-buf.h ioctls on the DMA_BUF_BASE ('b' = 0x62) magic. _IOWR = dir(3)<<30 | size<<16 | base<<8 | nr.
|
||||
const DMA_BUF_BASE: u64 = 0x62;
|
||||
const fn iowr(nr: u32, size: usize) -> u64 {
|
||||
(3u64 << 30) | ((size as u64) << 16) | (DMA_BUF_BASE << 8) | nr as u64
|
||||
}
|
||||
|
||||
#[repr(C)]
|
||||
struct DmaBufExportSyncFile {
|
||||
flags: u32,
|
||||
fd: i32,
|
||||
}
|
||||
|
||||
const DMA_BUF_IOCTL_EXPORT_SYNC_FILE: u64 = iowr(2, std::mem::size_of::<DmaBufExportSyncFile>());
|
||||
/// We will READ the buffer → export the fence(s) we must wait for before reading (the producer's writes).
|
||||
const DMA_BUF_SYNC_READ: u32 = 1 << 0;
|
||||
|
||||
/// Wait until the producer's writes to `dmabuf_fd` complete (or `timeout_ms` elapses). Returns:
|
||||
/// - `Ok(true)` — a render was still in flight and we waited on its fence (the race was real, now closed).
|
||||
/// - `Ok(false)` — no fence / already signaled (the driver attaches no implicit fence; zero-copy can race).
|
||||
/// - `Err` — the ioctl failed (e.g. the kernel/driver lacks `EXPORT_SYNC_FILE`).
|
||||
pub fn wait_read_ready(dmabuf_fd: RawFd, timeout_ms: i32) -> std::io::Result<bool> {
|
||||
let mut req = DmaBufExportSyncFile {
|
||||
flags: DMA_BUF_SYNC_READ,
|
||||
fd: -1,
|
||||
};
|
||||
// SAFETY: `dmabuf_fd` is a live dmabuf fd supplied by the caller (borrowed for this call; we
|
||||
// never close it). `DMA_BUF_IOCTL_EXPORT_SYNC_FILE` encodes `size_of::<DmaBufExportSyncFile>()`
|
||||
// — the exact byte count the kernel copies — and `&mut req` is a live, correctly-sized
|
||||
// `#[repr(C)]` struct the EXPORT_SYNC_FILE ioctl reads (`flags`) and writes (`fd`). `req`
|
||||
// outlives this synchronous call and is not aliased elsewhere.
|
||||
let r = unsafe { libc::ioctl(dmabuf_fd, DMA_BUF_IOCTL_EXPORT_SYNC_FILE, &mut req) };
|
||||
if r < 0 {
|
||||
return Err(std::io::Error::last_os_error());
|
||||
}
|
||||
let sync_fd = req.fd;
|
||||
if sync_fd < 0 {
|
||||
return Ok(false); // no sync_file exported
|
||||
}
|
||||
let mut pfd = libc::pollfd {
|
||||
fd: sync_fd,
|
||||
events: libc::POLLIN,
|
||||
revents: 0,
|
||||
};
|
||||
// Non-blocking probe: not-yet-signaled (poll==0) means the producer is still rendering.
|
||||
// SAFETY: `&mut pfd` points at a single live `libc::pollfd` and `nfds == 1` matches that one
|
||||
// element; `pfd.fd` is `sync_fd`, the sync_file fd just exported (already checked `>= 0`).
|
||||
// `poll` reads `fd`/`events` and writes `revents` for this non-blocking (timeout 0) probe, then
|
||||
// returns — `pfd` outlives the call and aliases nothing.
|
||||
let pending = unsafe { libc::poll(&mut pfd, 1, 0) } == 0;
|
||||
if pending {
|
||||
pfd.revents = 0;
|
||||
// SAFETY: same live single-element `pfd` (its `revents` reset to 0 just above), `nfds == 1`,
|
||||
// and `sync_fd` still open. This blocking `poll` (up to `timeout_ms`) waits for the render
|
||||
// fence to signal; it reads `fd`/`events`, writes `revents`, and returns before `pfd` ends.
|
||||
unsafe { libc::poll(&mut pfd, 1, timeout_ms) }; // block until the render fence signals
|
||||
}
|
||||
// SAFETY: `sync_fd` is the sync_file fd the EXPORT_SYNC_FILE ioctl created and handed us to own;
|
||||
// this point is reached only when `sync_fd >= 0`, this `close` runs exactly once on it, and it is
|
||||
// never used afterward — no double-close or use-after-close.
|
||||
unsafe { libc::close(sync_fd) };
|
||||
Ok(pending)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The ioctl number must match linux/dma-buf.h exactly — it's computed, so lock it down.
|
||||
#[test]
|
||||
fn ioctl_number_matches_dma_buf_h() {
|
||||
assert_eq!(DMA_BUF_IOCTL_EXPORT_SYNC_FILE, 0xC008_6202);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,726 @@
|
||||
//! Host side of the isolated zero-copy GPU import (design:
|
||||
//! [`design/zerocopy-worker-isolation.md`]): spawns the `zerocopy-worker` subprocess, mirrors the
|
||||
//! [`super::egl::EglImporter`] entry points over the [`super::proto`] socket, and materializes
|
||||
//! the worker's pooled CUDA buffers in this process via CUDA IPC (each buffer's handles are
|
||||
//! opened exactly once and reused as the pool recycles). A worker death — the whole point of the
|
||||
//! isolation — surfaces as an `Err` with [`RemoteImporter::dead`] set, never as a host fault.
|
||||
|
||||
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use super::cuda::{self, CUdeviceptr, DeviceBuffer, CU_IPC_HANDLE_SIZE};
|
||||
use super::egl::DmabufPlane;
|
||||
use super::proto::{self, BufferDesc, ImportKind, Reply, Request};
|
||||
use anyhow::{bail, Context, Result};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::fs::File;
|
||||
use std::io;
|
||||
use std::os::fd::{AsFd, AsRawFd, BorrowedFd, OwnedFd};
|
||||
use std::os::unix::process::CommandExt;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Child, Command};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex, OnceLock};
|
||||
use std::time::Duration;
|
||||
|
||||
/// Handshake budget: EGL + CUDA bring-up is ~200 ms; a cold driver load can take seconds.
|
||||
const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(20);
|
||||
/// Per-request budget. An import is a few ms of GPU work; if the worker can't answer in this
|
||||
/// window it is wedged (GPU fault in progress) and gets treated as dead.
|
||||
const REPLY_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
|
||||
/// State shared with in-flight frames: the socket (their release messages) and the CUDA IPC
|
||||
/// mappings (their device pointers). Lives until the LAST in-flight [`DeviceBuffer`] drops, so a
|
||||
/// mapping is never closed under a frame the encoder still reads — and only then does the socket
|
||||
/// close, which is what tells an idle worker to exit.
|
||||
struct Shared {
|
||||
sock: OwnedFd,
|
||||
mappings: Mutex<HashMap<u32, Mapping>>,
|
||||
dead: AtomicBool,
|
||||
}
|
||||
|
||||
/// One pooled worker buffer, opened in this process.
|
||||
#[derive(Clone, Copy)]
|
||||
struct Mapping {
|
||||
y: CUdeviceptr,
|
||||
y_pitch: usize,
|
||||
uv: Option<(CUdeviceptr, usize)>,
|
||||
width: u32,
|
||||
height: u32,
|
||||
}
|
||||
|
||||
impl Drop for Shared {
|
||||
fn drop(&mut self) {
|
||||
// Last reference gone — no DeviceBuffer can still point into these mappings.
|
||||
for (_, m) in self.mappings.lock().unwrap().drain() {
|
||||
cuda::ipc_close(m.y);
|
||||
if let Some((uv, _)) = m.uv {
|
||||
cuda::ipc_close(uv);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Children whose worker hasn't exited yet at `RemoteImporter` drop time (it exits on socket
|
||||
/// EOF, i.e. after the last in-flight frame drops). Swept on every spawn and every drop so
|
||||
/// workers don't linger as zombies for more than one capture generation.
|
||||
static REAPER: Mutex<Vec<Child>> = Mutex::new(Vec::new());
|
||||
|
||||
fn sweep_reaper() {
|
||||
let mut list = REAPER.lock().unwrap();
|
||||
list.retain_mut(|c| !matches!(c.try_wait(), Ok(Some(_))));
|
||||
}
|
||||
|
||||
/// Fd pinned to this process's own executable image, opened (once, lazily) via the
|
||||
/// `/proc/self/exe` magic link. The link names the running image's *inode*, not its path, so it
|
||||
/// resolves even after the installed binary was replaced or deleted — and exec'ing the fd (via
|
||||
/// [`fd_exec_path`]) then still runs byte-for-byte the build this process is. `current_exe()`
|
||||
/// instead readlinks to a path: after a package upgrade under a running host that path is
|
||||
/// "<path> (deleted)" and spawning it fails ENOENT — every capture then silently fell back to
|
||||
/// the CPU copy — and even while the path exists it may hold a newer build whose worker
|
||||
/// protocol mismatches this process.
|
||||
static SELF_EXE: OnceLock<Option<File>> = OnceLock::new();
|
||||
|
||||
fn self_exe() -> Option<BorrowedFd<'static>> {
|
||||
SELF_EXE
|
||||
.get_or_init(|| {
|
||||
let f = match File::open("/proc/self/exe") {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
error = %e,
|
||||
"cannot pin /proc/self/exe — worker spawns use the current_exe() path, \
|
||||
which breaks if this binary is replaced on disk"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
if f.as_raw_fd() != 3 {
|
||||
return Some(f);
|
||||
}
|
||||
// Fd 3 is the slot the spawn hands the worker its socket on (the `dup2` in
|
||||
// `spawn_exe`) — pinned there, the child would clobber it before exec resolves
|
||||
// `/proc/self/fd/3`. Re-number: 3 stays occupied by `f` during the clone, so the
|
||||
// duplicate cannot land on it.
|
||||
match f.try_clone() {
|
||||
Ok(clone) => Some(clone),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "re-numbering the pinned exe fd off fd 3 failed");
|
||||
None
|
||||
}
|
||||
}
|
||||
})
|
||||
.as_ref()
|
||||
.map(|f| f.as_fd())
|
||||
}
|
||||
|
||||
/// `/proc/self/fd/<n>` — an exec'able path to `fd`'s inode. The kernel resolves it at exec time
|
||||
/// inside the forked child, whose fd table is a copy of ours (close-on-exec applies only once
|
||||
/// the exec succeeds), so it names the pinned inode no matter what sits at the file's original
|
||||
/// path by then.
|
||||
fn fd_exec_path(fd: BorrowedFd<'_>) -> PathBuf {
|
||||
PathBuf::from(format!("/proc/self/fd/{}", fd.as_raw_fd()))
|
||||
}
|
||||
|
||||
/// The remote (isolated) importer — one per capture. Method-for-method mirror of the in-process
|
||||
/// [`super::egl::EglImporter`] surface the capture thread uses.
|
||||
pub struct RemoteImporter {
|
||||
shared: Arc<Shared>,
|
||||
child: Option<Child>,
|
||||
/// Reused receive scratch buffer (all replies are read by the single capture thread).
|
||||
rbuf: Vec<u8>,
|
||||
/// Dmabuf keys (`st_ino`) whose fd the worker already holds — the fd is passed only once.
|
||||
sent_keys: HashSet<u64>,
|
||||
}
|
||||
|
||||
impl RemoteImporter {
|
||||
/// Spawn the worker from this host binary and complete the readiness handshake. The worker
|
||||
/// is exec'd through the pinned [`SELF_EXE`] fd, so it is always the exact image this
|
||||
/// process runs — even after the installed binary was replaced mid-flight. An `Err` here
|
||||
/// means "no isolated zero-copy available" — callers fall back to the CPU path, exactly like
|
||||
/// an in-process `EglImporter::new()` failure.
|
||||
pub fn spawn() -> Result<RemoteImporter> {
|
||||
match self_exe() {
|
||||
Some(fd) => Self::spawn_exe(&fd_exec_path(fd)),
|
||||
None => Self::spawn_exe(
|
||||
&std::env::current_exe().context("resolve /proc/self/exe for the worker")?,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// [`Self::spawn`] with an explicit executable (separated for tests).
|
||||
fn spawn_exe(exe: &Path) -> Result<RemoteImporter> {
|
||||
sweep_reaper();
|
||||
let (host_end, worker_end) = proto::socketpair_seqpacket().context("worker socketpair")?;
|
||||
let mut cmd = Command::new(exe);
|
||||
// `exe` is normally an opaque `/proc/self/fd/<n>` — keep `ps` output meaningful.
|
||||
cmd.arg0("punktfunk-host");
|
||||
cmd.arg("zerocopy-worker").arg("--fd").arg("3");
|
||||
let raw = worker_end.as_raw_fd();
|
||||
// SAFETY: `pre_exec` runs between fork and exec, so only async-signal-safe calls are
|
||||
// allowed — `dup2` and `fcntl` both are, and the closure captures only the `Copy` int
|
||||
// `raw` (no allocation, no locks). `dup2(raw, 3)` installs the socket at the fd number
|
||||
// the subcommand expects and clears CLOEXEC on the copy; if the parent's fd already IS 3,
|
||||
// `dup2(3,3)` would preserve CLOEXEC, so that case clears the flag explicitly instead.
|
||||
unsafe {
|
||||
cmd.pre_exec(move || {
|
||||
if raw == 3 {
|
||||
let flags = libc::fcntl(3, libc::F_GETFD);
|
||||
if flags < 0 || libc::fcntl(3, libc::F_SETFD, flags & !libc::FD_CLOEXEC) < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
} else if libc::dup2(raw, 3) < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
Ok(())
|
||||
});
|
||||
}
|
||||
let child = cmd.spawn().context("spawn zerocopy-worker")?;
|
||||
drop(worker_end); // the child holds its own copy now
|
||||
Self::from_socket(host_end, Some(child))
|
||||
}
|
||||
|
||||
/// Complete the handshake on an already-connected socket (the unit tests drive this against
|
||||
/// a mock server thread instead of a real subprocess).
|
||||
fn from_socket(sock: OwnedFd, child: Option<Child>) -> Result<RemoteImporter> {
|
||||
let mut importer = RemoteImporter {
|
||||
shared: Arc::new(Shared {
|
||||
sock,
|
||||
mappings: Mutex::new(HashMap::new()),
|
||||
dead: AtomicBool::new(false),
|
||||
}),
|
||||
child,
|
||||
rbuf: Vec::new(),
|
||||
sent_keys: HashSet::new(),
|
||||
};
|
||||
proto::set_recv_timeout(importer.shared.sock.as_fd(), Some(HANDSHAKE_TIMEOUT))?;
|
||||
let ready = proto::recv::<Reply>(importer.shared.sock.as_fd(), &mut importer.rbuf);
|
||||
proto::set_recv_timeout(importer.shared.sock.as_fd(), Some(REPLY_TIMEOUT))?;
|
||||
match ready {
|
||||
Ok((Reply::Ready { version }, _)) if version == proto::PROTO_VERSION => {
|
||||
tracing::info!(
|
||||
pid = importer.child.as_ref().map(|c| c.id()),
|
||||
"zero-copy GPU import isolated in a worker process"
|
||||
);
|
||||
Ok(importer)
|
||||
}
|
||||
Ok((Reply::Ready { version }, _)) => {
|
||||
importer.mark_dead();
|
||||
bail!(
|
||||
"zerocopy worker protocol mismatch (worker v{version}, host v{})",
|
||||
proto::PROTO_VERSION
|
||||
)
|
||||
}
|
||||
Ok((Reply::InitErr { message }, _)) => {
|
||||
// The worker exits by itself after reporting; not a death, just "no GPU here".
|
||||
bail!("zerocopy worker init failed: {message}")
|
||||
}
|
||||
Ok((other, _)) => {
|
||||
importer.mark_dead();
|
||||
bail!("unexpected zerocopy worker handshake: {other:?}")
|
||||
}
|
||||
Err(e) => {
|
||||
importer.mark_dead();
|
||||
Err(e).context("zerocopy worker handshake (died on startup?)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// True once any exchange failed at the transport level — the worker is gone (or wedged) and
|
||||
/// every further call fails fast. The capture layer poisons its stream on this.
|
||||
pub fn dead(&self) -> bool {
|
||||
self.shared.dead.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
fn mark_dead(&self) {
|
||||
self.shared.dead.store(true, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Mirror of [`super::egl::EglImporter::supported_modifiers`] (worker round-trip; empty on
|
||||
/// any failure, which makes the capture fall back like an importless negotiation).
|
||||
pub fn supported_modifiers(&mut self, fourcc: u32) -> Vec<u64> {
|
||||
if self.dead() {
|
||||
return Vec::new();
|
||||
}
|
||||
if let Err(e) = proto::send(
|
||||
self.shared.sock.as_fd(),
|
||||
&Request::Modifiers { fourcc },
|
||||
None,
|
||||
) {
|
||||
tracing::warn!(error = %e, "zerocopy worker modifier query failed");
|
||||
self.mark_dead();
|
||||
return Vec::new();
|
||||
}
|
||||
match proto::recv::<Reply>(self.shared.sock.as_fd(), &mut self.rbuf) {
|
||||
Ok((Reply::Modifiers { modifiers }, _)) => modifiers,
|
||||
Ok((other, _)) => {
|
||||
tracing::warn!(?other, "unexpected zerocopy worker reply to Modifiers");
|
||||
self.mark_dead();
|
||||
Vec::new()
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "zerocopy worker modifier reply failed");
|
||||
self.mark_dead();
|
||||
Vec::new()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Mirror of [`super::egl::EglImporter::import`] (tiled dmabuf → BGRx CUDA buffer).
|
||||
pub fn import(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> Result<DeviceBuffer> {
|
||||
self.import_impl(plane, ImportKind::Tiled, width, height, fourcc, modifier)
|
||||
}
|
||||
|
||||
/// Mirror of [`super::egl::EglImporter::import_nv12`].
|
||||
pub fn import_nv12(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> Result<DeviceBuffer> {
|
||||
self.import_impl(
|
||||
plane,
|
||||
ImportKind::TiledNv12,
|
||||
width,
|
||||
height,
|
||||
fourcc,
|
||||
modifier,
|
||||
)
|
||||
}
|
||||
|
||||
/// Mirror of [`super::egl::EglImporter::import_yuv444`] (tiled dmabuf → stacked 3-plane
|
||||
/// YUV444 CUDA buffer — the 4:4:4 zero-copy path).
|
||||
pub fn import_yuv444(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> Result<DeviceBuffer> {
|
||||
self.import_impl(plane, ImportKind::Tiled444, width, height, fourcc, modifier)
|
||||
}
|
||||
|
||||
/// Mirror of [`super::egl::EglImporter::import_linear`] (LINEAR dmabuf → Vulkan bridge).
|
||||
pub fn import_linear(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
) -> Result<DeviceBuffer> {
|
||||
self.import_impl(plane, ImportKind::Linear, width, height, 0, None)
|
||||
}
|
||||
|
||||
fn import_impl(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
kind: ImportKind,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> Result<DeviceBuffer> {
|
||||
if self.dead() {
|
||||
bail!("zerocopy worker is dead");
|
||||
}
|
||||
let key = dmabuf_key(plane.fd)?;
|
||||
// One retry: a `NeedFd` reply (the worker's fd cache evicted this key) clears our
|
||||
// "already sent" note so the second attempt carries the fd again.
|
||||
let mut attempts = 0;
|
||||
let reply = loop {
|
||||
attempts += 1;
|
||||
let has_fd = self.sent_keys.insert(key);
|
||||
// SAFETY: `plane.fd` is the dmabuf fd of the PipeWire buffer the capture thread still
|
||||
// holds for this callback (`consume_frame`'s contract), so it is open and stays open
|
||||
// for this synchronous call; the `BorrowedFd` never outlives it (used only for the
|
||||
// `send`).
|
||||
let pass = has_fd.then(|| unsafe { BorrowedFd::borrow_raw(plane.fd) });
|
||||
let req = Request::Import {
|
||||
key,
|
||||
kind,
|
||||
width,
|
||||
height,
|
||||
fourcc,
|
||||
modifier,
|
||||
offset: plane.offset,
|
||||
stride: plane.stride,
|
||||
has_fd,
|
||||
};
|
||||
if let Err(e) = proto::send(self.shared.sock.as_fd(), &req, pass) {
|
||||
self.mark_dead();
|
||||
return Err(e).context("zerocopy worker died (send)");
|
||||
}
|
||||
let reply = match proto::recv::<Reply>(self.shared.sock.as_fd(), &mut self.rbuf) {
|
||||
Ok((reply, _)) => reply,
|
||||
Err(e) => {
|
||||
self.mark_dead();
|
||||
return Err(e).context("zerocopy worker died (no reply)");
|
||||
}
|
||||
};
|
||||
match reply {
|
||||
Reply::NeedFd if attempts == 1 => {
|
||||
self.sent_keys.remove(&key);
|
||||
continue;
|
||||
}
|
||||
Reply::NeedFd => {
|
||||
self.mark_dead();
|
||||
bail!("zerocopy worker still lacks the fd after a resend (desync)");
|
||||
}
|
||||
other => break other,
|
||||
}
|
||||
};
|
||||
match reply {
|
||||
Reply::Frame { id, desc } => {
|
||||
if let Some(desc) = desc {
|
||||
let mapping = open_mapping(&desc).with_context(|| {
|
||||
// An unopenable mapping poisons every future frame in this buffer —
|
||||
// treat it as a dead worker so the capture rebuilds cleanly.
|
||||
self.mark_dead();
|
||||
format!("open CUDA IPC mapping for worker buffer {id}")
|
||||
})?;
|
||||
self.shared.mappings.lock().unwrap().insert(id, mapping);
|
||||
}
|
||||
let m = self
|
||||
.shared
|
||||
.mappings
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get(&id)
|
||||
.copied()
|
||||
.ok_or_else(|| {
|
||||
self.mark_dead();
|
||||
anyhow::anyhow!("worker delivered unknown buffer id {id} (desync)")
|
||||
})?;
|
||||
let shared = self.shared.clone();
|
||||
Ok(DeviceBuffer::remote(
|
||||
m.y,
|
||||
m.y_pitch,
|
||||
m.width,
|
||||
m.height,
|
||||
m.uv,
|
||||
// The wire carries no plane format — the buffer's layout is what WE requested.
|
||||
kind == ImportKind::Tiled444,
|
||||
Box::new(move || {
|
||||
// Fire-and-forget recycle; a dead worker just means EPIPE, ignored. The
|
||||
// captured `shared` Arc is what keeps the mapping + socket alive until
|
||||
// the last frame drops.
|
||||
let _ = proto::send(shared.sock.as_fd(), &Request::Release { id }, None);
|
||||
}),
|
||||
))
|
||||
}
|
||||
Reply::Err { message } => bail!("zerocopy worker import failed: {message}"),
|
||||
other => {
|
||||
self.mark_dead();
|
||||
bail!("unexpected zerocopy worker reply: {other:?}")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The PipeWire stream renegotiated — reset both sides' per-buffer caches.
|
||||
pub fn clear_cache(&mut self) {
|
||||
self.sent_keys.clear();
|
||||
if !self.dead() {
|
||||
if let Err(e) = proto::send(self.shared.sock.as_fd(), &Request::ClearCache, None) {
|
||||
tracing::warn!(error = %e, "zerocopy worker ClearCache failed");
|
||||
self.mark_dead();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for RemoteImporter {
|
||||
fn drop(&mut self) {
|
||||
// The worker exits on socket EOF, which happens when the last `Shared` reference (this
|
||||
// importer, or the final in-flight frame on the encode side) drops. Reap what's already
|
||||
// gone; park the rest for the next sweep.
|
||||
if let Some(mut child) = self.child.take() {
|
||||
if !matches!(child.try_wait(), Ok(Some(_))) {
|
||||
REAPER.lock().unwrap().push(child);
|
||||
}
|
||||
}
|
||||
sweep_reaper();
|
||||
}
|
||||
}
|
||||
|
||||
/// Identity of the dma-buf behind `fd`, stable across frames and across `SCM_RIGHTS` re-numbering:
|
||||
/// every dma-buf gets a unique inode on the kernel's dmabuf pseudo-fs for its lifetime. Used as
|
||||
/// the worker's fd-cache key so the fd itself is only passed once.
|
||||
fn dmabuf_key(fd: i32) -> Result<u64> {
|
||||
// SAFETY: `libc::stat` is plain-old-data for which all-zero is a valid value, so
|
||||
// `mem::zeroed()` is a sound initializer. `fd` is the caller's live dmabuf fd; `fstat` writes
|
||||
// into `&mut st`, a live, correctly-sized stack struct that outlives the synchronous call,
|
||||
// and `st_ino` is read only after the return value is checked.
|
||||
unsafe {
|
||||
let mut st: libc::stat = std::mem::zeroed();
|
||||
if libc::fstat(fd, &mut st) != 0 {
|
||||
bail!("fstat(dmabuf fd): {}", io::Error::last_os_error());
|
||||
}
|
||||
Ok(st.st_ino)
|
||||
}
|
||||
}
|
||||
|
||||
/// Open a worker buffer's CUDA IPC handles in this process.
|
||||
fn open_mapping(desc: &BufferDesc) -> Result<Mapping> {
|
||||
cuda::make_current()?;
|
||||
let y_handle: [u8; CU_IPC_HANDLE_SIZE] = desc
|
||||
.y_handle
|
||||
.as_slice()
|
||||
.try_into()
|
||||
.context("worker sent a malformed Y IPC handle")?;
|
||||
let y = cuda::ipc_open(&y_handle).context("open Y plane IPC handle")?;
|
||||
let uv = match &desc.uv {
|
||||
Some((handle, pitch)) => {
|
||||
let handle: [u8; CU_IPC_HANDLE_SIZE] = handle
|
||||
.as_slice()
|
||||
.try_into()
|
||||
.context("worker sent a malformed UV IPC handle")?;
|
||||
match cuda::ipc_open(&handle) {
|
||||
Ok(ptr) => Some((ptr, *pitch)),
|
||||
Err(e) => {
|
||||
// Don't leak the Y mapping on a half-open failure.
|
||||
cuda::ipc_close(y);
|
||||
return Err(e).context("open UV plane IPC handle");
|
||||
}
|
||||
}
|
||||
}
|
||||
None => None,
|
||||
};
|
||||
Ok(Mapping {
|
||||
y,
|
||||
y_pitch: desc.y_pitch,
|
||||
uv,
|
||||
width: desc.width,
|
||||
height: desc.height,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::thread;
|
||||
|
||||
fn handshake_server(reply: Reply) -> OwnedFd {
|
||||
let (host, worker) = proto::socketpair_seqpacket().unwrap();
|
||||
proto::send(worker.as_fd(), &reply, None).unwrap();
|
||||
// Keep the worker end alive alongside the host end for the test's duration by leaking it
|
||||
// into the reply thread below? Not needed: the handshake reply is already queued in the
|
||||
// socket buffer, so the worker end may drop — recv still delivers queued data first.
|
||||
drop(worker);
|
||||
host
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn handshake_ready_and_version_gate() {
|
||||
let host = handshake_server(Reply::Ready {
|
||||
version: proto::PROTO_VERSION,
|
||||
});
|
||||
let imp = RemoteImporter::from_socket(host, None).unwrap();
|
||||
assert!(!imp.dead());
|
||||
|
||||
let host = handshake_server(Reply::Ready { version: 999 });
|
||||
assert!(RemoteImporter::from_socket(host, None).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn handshake_init_err() {
|
||||
let host = handshake_server(Reply::InitErr {
|
||||
message: "no GPU".into(),
|
||||
});
|
||||
let Err(err) = RemoteImporter::from_socket(host, None) else {
|
||||
panic!("InitErr handshake must fail")
|
||||
};
|
||||
assert!(format!("{err:#}").contains("no GPU"), "{err:#}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn handshake_eof_is_an_error() {
|
||||
let (host, worker) = proto::socketpair_seqpacket().unwrap();
|
||||
drop(worker);
|
||||
assert!(RemoteImporter::from_socket(host, None).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn spawning_a_non_worker_fails_cleanly() {
|
||||
// `true` exits immediately without a handshake → EOF → clean spawn error, the same
|
||||
// fallback path a GPU-less box takes.
|
||||
let Err(err) = RemoteImporter::spawn_exe(Path::new("true")) else {
|
||||
panic!("spawning a non-worker must fail")
|
||||
};
|
||||
assert!(format!("{err:#}").contains("handshake"), "{err:#}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn spawn_execs_the_pinned_self_exe() {
|
||||
// `spawn()` execs this very process's image via the pinned `/proc/self/fd/…` path. Here
|
||||
// that image is the libtest harness, which rejects `--fd` and exits without a handshake
|
||||
// — so a "handshake" error proves the exec itself succeeded (an exec failure would read
|
||||
// "spawn zerocopy-worker" instead).
|
||||
let Err(err) = RemoteImporter::spawn() else {
|
||||
panic!("the test harness is not a worker; spawn must fail at the handshake")
|
||||
};
|
||||
assert!(format!("{err:#}").contains("handshake"), "{err:#}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pinned_fd_exec_survives_on_disk_replacement() {
|
||||
// The 2026-07-10 canary regression: a package upgrade replaced the installed binary and
|
||||
// every worker spawn ENOENT'd (`current_exe()` readlinked to "<path> (deleted)"). The
|
||||
// pinned-fd mechanism must keep exec'ing the original image after the file is gone: pin
|
||||
// a copy of /bin/sh, delete it, then run it through the fd path.
|
||||
let copy = std::env::temp_dir().join(format!("pf-zerocopy-exe-pin-{}", std::process::id()));
|
||||
std::fs::copy("/bin/sh", ©).unwrap();
|
||||
let pinned = File::open(©).unwrap();
|
||||
std::fs::remove_file(©).unwrap();
|
||||
// Retry ETXTBSY: `fs::copy`'s write fd leaks into other tests' concurrently-forked
|
||||
// children until their execs clear it (CLOEXEC applies only at exec), and exec'ing a
|
||||
// file someone holds open for writing is refused. A harness artifact of copy-then-exec,
|
||||
// not the mechanism under test — production pins a read-only fd on a binary nobody
|
||||
// write-opens.
|
||||
let status = loop {
|
||||
match Command::new(fd_exec_path(pinned.as_fd()))
|
||||
.arg("-c")
|
||||
.arg("exit 42")
|
||||
.status()
|
||||
{
|
||||
Err(e) if e.raw_os_error() == Some(libc::ETXTBSY) => {
|
||||
std::thread::sleep(Duration::from_millis(10))
|
||||
}
|
||||
other => break other.expect("exec via /proc/self/fd of a deleted file"),
|
||||
}
|
||||
};
|
||||
assert_eq!(status.code(), Some(42));
|
||||
}
|
||||
|
||||
/// A scripted peer: answers the handshake, then serves canned replies per request.
|
||||
fn scripted_server(replies: Vec<Reply>) -> (RemoteImporter, thread::JoinHandle<Vec<Request>>) {
|
||||
let (host, worker) = proto::socketpair_seqpacket().unwrap();
|
||||
proto::send(
|
||||
worker.as_fd(),
|
||||
&Reply::Ready {
|
||||
version: proto::PROTO_VERSION,
|
||||
},
|
||||
None,
|
||||
)
|
||||
.unwrap();
|
||||
let join = thread::spawn(move || {
|
||||
let mut buf = Vec::new();
|
||||
let mut seen = Vec::new();
|
||||
let mut replies = replies.into_iter();
|
||||
while let Ok((req, _fd)) = proto::recv::<Request>(worker.as_fd(), &mut buf) {
|
||||
let needs_reply = matches!(req, Request::Modifiers { .. } | Request::Import { .. });
|
||||
seen.push(req);
|
||||
if needs_reply {
|
||||
match replies.next() {
|
||||
Some(r) => proto::send(worker.as_fd(), &r, None).unwrap(),
|
||||
None => break, // close → client sees a dead worker
|
||||
}
|
||||
}
|
||||
}
|
||||
seen
|
||||
});
|
||||
let imp = RemoteImporter::from_socket(host, None).unwrap();
|
||||
(imp, join)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn modifiers_round_trip() {
|
||||
let (mut imp, join) = scripted_server(vec![Reply::Modifiers {
|
||||
modifiers: vec![1, 2, 3],
|
||||
}]);
|
||||
assert_eq!(imp.supported_modifiers(0x3432_5258), vec![1, 2, 3]);
|
||||
assert!(!imp.dead());
|
||||
drop(imp);
|
||||
let seen = join.join().unwrap();
|
||||
assert_eq!(
|
||||
seen,
|
||||
vec![Request::Modifiers {
|
||||
fourcc: 0x3432_5258
|
||||
}]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn need_fd_triggers_one_resend_with_the_fd() {
|
||||
let (mut imp, join) = scripted_server(vec![
|
||||
Reply::Err {
|
||||
message: "one".into(),
|
||||
},
|
||||
Reply::NeedFd,
|
||||
Reply::Err {
|
||||
message: "two".into(),
|
||||
},
|
||||
]);
|
||||
let (pr, _pw) = std::io::pipe().unwrap();
|
||||
let plane = DmabufPlane {
|
||||
fd: pr.as_fd().as_raw_fd(),
|
||||
offset: 0,
|
||||
stride: 256,
|
||||
};
|
||||
// First import: first sight of the key → fd rides along; the Err reply keeps the key
|
||||
// marked as sent (the worker cached the fd before failing).
|
||||
assert!(imp.import(&plane, 64, 64, 1, Some(2)).is_err());
|
||||
// Second import: no fd (already sent) → worker answers NeedFd → one retry WITH the fd.
|
||||
assert!(imp.import(&plane, 64, 64, 1, Some(2)).is_err());
|
||||
assert!(!imp.dead(), "NeedFd handling must not mark the worker dead");
|
||||
drop(imp);
|
||||
let fd_flags: Vec<bool> = join
|
||||
.join()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|r| match r {
|
||||
Request::Import { has_fd, .. } => *has_fd,
|
||||
other => panic!("unexpected request {other:?}"),
|
||||
})
|
||||
.collect();
|
||||
assert_eq!(fd_flags, vec![true, false, true]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn import_error_reply_keeps_worker_alive_and_death_is_detected() {
|
||||
let (mut imp, join) = scripted_server(vec![Reply::Err {
|
||||
message: "EGL_BAD_MATCH".into(),
|
||||
}]);
|
||||
// Any pipe works as a stand-in fd for key derivation.
|
||||
let (pr, _pw) = std::io::pipe().unwrap();
|
||||
let plane = DmabufPlane {
|
||||
fd: pr.as_fd().as_raw_fd(),
|
||||
offset: 0,
|
||||
stride: 256,
|
||||
};
|
||||
let Err(err) = imp.import(&plane, 64, 64, 1, Some(2)) else {
|
||||
panic!("scripted Err reply must fail the import")
|
||||
};
|
||||
assert!(format!("{err:#}").contains("EGL_BAD_MATCH"));
|
||||
assert!(!imp.dead(), "an Err reply must not mark the worker dead");
|
||||
|
||||
// The scripted replies are exhausted → the server closes → the next import dies.
|
||||
let Err(err) = imp.import(&plane, 64, 64, 1, Some(2)) else {
|
||||
panic!("a closed worker must fail the import")
|
||||
};
|
||||
assert!(format!("{err:#}").contains("died"), "{err:#}");
|
||||
assert!(imp.dead());
|
||||
drop(imp);
|
||||
let seen = join.join().unwrap();
|
||||
// First import carried the fd (first sight of the key); the retry didn't re-send it.
|
||||
match (&seen[0], &seen[1]) {
|
||||
(
|
||||
Request::Import {
|
||||
has_fd: true,
|
||||
kind: ImportKind::Tiled,
|
||||
..
|
||||
},
|
||||
Request::Import { has_fd: false, .. },
|
||||
) => {}
|
||||
other => panic!("unexpected requests {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,488 @@
|
||||
//! Raw CUDA Driver API FFI (plan §W4, carved out of the zero-copy CUDA facade): the opaque handle
|
||||
//! typedefs + struct/const definitions, the `dlopen`'d `libcuda.so.1` symbol table ([`CudaApi`] +
|
||||
//! [`cuda_api`]), the `unsafe` `cuXxx` wrappers, and the `ck` result check. No higher-level state —
|
||||
//! the shared `CUcontext`, device buffers, GL/dmabuf interop, and cursor blend all live in [`super`]
|
||||
//! and drive this layer.
|
||||
|
||||
#![allow(non_camel_case_types, non_snake_case)]
|
||||
// Every `unsafe` block/impl below carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use anyhow::{bail, Result};
|
||||
use std::os::raw::{c_char, c_int, c_uint, c_void};
|
||||
use std::sync::OnceLock;
|
||||
|
||||
pub type CUresult = c_uint; // CUDA_SUCCESS == 0
|
||||
pub type CUdevice = c_int;
|
||||
pub type CUcontext = *mut c_void; // opaque CUctx_st*
|
||||
pub type CUstream = *mut c_void; // opaque CUstream_st*
|
||||
pub type CUdeviceptr = u64;
|
||||
pub type CUgraphicsResource = *mut c_void;
|
||||
pub type CUarray = *mut c_void;
|
||||
pub type CUexternalMemory = *mut c_void; // opaque CUextMemory_st*
|
||||
pub type CUmodule = *mut c_void; // opaque CUmod_st*
|
||||
pub type CUfunction = *mut c_void; // opaque CUfunc_st*
|
||||
|
||||
/// `CUmemorytype` (cuda.h): HOST=1, DEVICE=2, ARRAY=3, UNIFIED=4.
|
||||
pub const CU_MEMORYTYPE_DEVICE: c_uint = 2;
|
||||
pub const CU_MEMORYTYPE_ARRAY: c_uint = 3;
|
||||
|
||||
/// `CUctx_flags` (cuda.h): block the CPU on an OS primitive while waiting for the GPU instead of
|
||||
/// busy-spinning. On this shared box (compositor + send thread on the same cores) spinning a core
|
||||
/// to detect copy completion steals CPU from the very threads we want scheduled; BLOCKING_SYNC
|
||||
/// frees it. Default (`CU_CTX_SCHED_AUTO=0`) heuristically picks SPIN vs YIELD by core count.
|
||||
pub(crate) const CU_CTX_SCHED_BLOCKING_SYNC: c_uint = 0x04;
|
||||
|
||||
/// `cuStreamCreateWithPriority` flag: don't implicitly synchronize with the legacy NULL stream.
|
||||
pub(crate) const CU_STREAM_NON_BLOCKING: c_uint = 0x01;
|
||||
|
||||
/// `CUDA_MEMCPY2D` (cuda.h, `_v2` ABI). Field order is load-bearing.
|
||||
#[repr(C)]
|
||||
#[derive(Default)]
|
||||
pub struct CUDA_MEMCPY2D {
|
||||
pub srcXInBytes: usize,
|
||||
pub srcY: usize,
|
||||
pub srcMemoryType: c_uint,
|
||||
pub srcHost: *const c_void,
|
||||
pub srcDevice: CUdeviceptr,
|
||||
pub srcArray: CUarray,
|
||||
pub srcPitch: usize,
|
||||
pub dstXInBytes: usize,
|
||||
pub dstY: usize,
|
||||
pub dstMemoryType: c_uint,
|
||||
pub dstHost: *mut c_void,
|
||||
pub dstDevice: CUdeviceptr,
|
||||
pub dstArray: CUarray,
|
||||
pub dstPitch: usize,
|
||||
pub WidthInBytes: usize,
|
||||
pub Height: usize,
|
||||
}
|
||||
|
||||
/// `CUDA_EXTERNAL_MEMORY_HANDLE_DESC` (cuda.h, 64-bit layout). `handle` is a union whose
|
||||
/// largest member is the win32 two-pointer struct (16 bytes, align 8); for the OPAQUE_FD type
|
||||
/// only the first 4 bytes (the `int fd`) are read.
|
||||
#[repr(C)]
|
||||
#[derive(Default)]
|
||||
pub struct CUDA_EXTERNAL_MEMORY_HANDLE_DESC {
|
||||
pub type_: c_uint, // CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD = 1
|
||||
pub(crate) _pad: u32,
|
||||
pub handle: [u64; 2], // union { int fd; {void*,void*} win32; void* nvSciBufObject }
|
||||
pub size: u64,
|
||||
pub flags: c_uint,
|
||||
pub(crate) reserved: [c_uint; 16],
|
||||
pub(crate) _pad2: u32,
|
||||
}
|
||||
|
||||
/// `CUDA_EXTERNAL_MEMORY_BUFFER_DESC` (cuda.h, 64-bit layout).
|
||||
#[repr(C)]
|
||||
#[derive(Default)]
|
||||
pub struct CUDA_EXTERNAL_MEMORY_BUFFER_DESC {
|
||||
pub offset: u64,
|
||||
pub size: u64,
|
||||
pub flags: c_uint,
|
||||
pub(crate) reserved: [c_uint; 16],
|
||||
pub(crate) _pad: u32,
|
||||
}
|
||||
|
||||
pub const CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD: c_uint = 1;
|
||||
|
||||
/// `CUipcMemHandle` (cuda.h): an opaque 64-byte struct identifying a device allocation across
|
||||
/// processes. Produced by `cuIpcGetMemHandle` in the exporting process, consumed by
|
||||
/// `cuIpcOpenMemHandle` in the importer — passed **by value**, matching the C
|
||||
/// `struct { char reserved[64]; }`. Plain bytes — safe to ship over a socket.
|
||||
pub const CU_IPC_HANDLE_SIZE: usize = 64;
|
||||
#[repr(C)]
|
||||
#[derive(Clone, Copy)]
|
||||
pub struct CUipcMemHandle {
|
||||
pub reserved: [u8; CU_IPC_HANDLE_SIZE],
|
||||
}
|
||||
|
||||
/// `CUipcMem_flags`: lazily enable peer access on open (the documented flag for
|
||||
/// `cuIpcOpenMemHandle`; a no-op for a same-device open, which is our only case).
|
||||
pub(crate) const CU_IPC_MEM_LAZY_ENABLE_PEER_ACCESS: c_uint = 0x1;
|
||||
|
||||
/// CUDA Driver API entry points, resolved at runtime from `libcuda.so.1` via `dlopen` rather than
|
||||
/// a link-time `#[link(name = "cuda")]`. This is what lets ONE host binary run on NVIDIA
|
||||
/// (zero-copy via CUDA → NVENC) *and* on AMD/Intel (VAAPI, where the NVIDIA driver — and thus
|
||||
/// `libcuda` — is absent): with a hard link the loader would refuse to start the binary at all.
|
||||
/// Every `cu*` call below goes through a same-named wrapper fn that forwards to this table; when
|
||||
/// the driver isn't present the table is `None` and the wrappers return a non-zero `CUresult`, so
|
||||
/// `context()` fails cleanly and the capturer falls back to the CPU path. The `cuda_api()` loader
|
||||
/// is memoised; the library handle is intentionally leaked (process-lifetime, like the context).
|
||||
pub(crate) struct CudaApi {
|
||||
cuInit: unsafe extern "C" fn(c_uint) -> CUresult,
|
||||
cuDeviceGet: unsafe extern "C" fn(*mut CUdevice, c_int) -> CUresult,
|
||||
cuCtxCreate_v2: unsafe extern "C" fn(*mut CUcontext, c_uint, CUdevice) -> CUresult,
|
||||
cuCtxDestroy_v2: unsafe extern "C" fn(CUcontext) -> CUresult,
|
||||
cuCtxSetCurrent: unsafe extern "C" fn(CUcontext) -> CUresult,
|
||||
cuMemAllocPitch_v2:
|
||||
unsafe extern "C" fn(*mut CUdeviceptr, *mut usize, usize, usize, c_uint) -> CUresult,
|
||||
cuMemFree_v2: unsafe extern "C" fn(CUdeviceptr) -> CUresult,
|
||||
cuMemcpy2DAsync_v2: unsafe extern "C" fn(*const CUDA_MEMCPY2D, CUstream) -> CUresult,
|
||||
cuStreamSynchronize: unsafe extern "C" fn(CUstream) -> CUresult,
|
||||
cuCtxGetStreamPriorityRange: unsafe extern "C" fn(*mut c_int, *mut c_int) -> CUresult,
|
||||
cuStreamCreateWithPriority: unsafe extern "C" fn(*mut CUstream, c_uint, c_int) -> CUresult,
|
||||
cuGraphicsGLRegisterImage:
|
||||
unsafe extern "C" fn(*mut CUgraphicsResource, c_uint, c_uint, c_uint) -> CUresult,
|
||||
cuGraphicsMapResources:
|
||||
unsafe extern "C" fn(c_uint, *mut CUgraphicsResource, *mut c_void) -> CUresult,
|
||||
cuGraphicsUnmapResources:
|
||||
unsafe extern "C" fn(c_uint, *mut CUgraphicsResource, *mut c_void) -> CUresult,
|
||||
cuGraphicsSubResourceGetMappedArray:
|
||||
unsafe extern "C" fn(*mut CUarray, CUgraphicsResource, c_uint, c_uint) -> CUresult,
|
||||
cuGraphicsUnregisterResource: unsafe extern "C" fn(CUgraphicsResource) -> CUresult,
|
||||
cuImportExternalMemory: unsafe extern "C" fn(
|
||||
*mut CUexternalMemory,
|
||||
*const CUDA_EXTERNAL_MEMORY_HANDLE_DESC,
|
||||
) -> CUresult,
|
||||
cuExternalMemoryGetMappedBuffer: unsafe extern "C" fn(
|
||||
*mut CUdeviceptr,
|
||||
CUexternalMemory,
|
||||
*const CUDA_EXTERNAL_MEMORY_BUFFER_DESC,
|
||||
) -> CUresult,
|
||||
cuDestroyExternalMemory: unsafe extern "C" fn(CUexternalMemory) -> CUresult,
|
||||
cuIpcGetMemHandle: unsafe extern "C" fn(*mut CUipcMemHandle, CUdeviceptr) -> CUresult,
|
||||
cuIpcOpenMemHandle: unsafe extern "C" fn(*mut CUdeviceptr, CUipcMemHandle, c_uint) -> CUresult,
|
||||
cuIpcCloseMemHandle: unsafe extern "C" fn(CUdeviceptr) -> CUresult,
|
||||
// Cursor-overlay blend: a linear device alloc + a PTX module with the blend kernels launched
|
||||
// over the cursor's small rectangle (see [`CursorBlend`]).
|
||||
cuMemAlloc_v2: unsafe extern "C" fn(*mut CUdeviceptr, usize) -> CUresult,
|
||||
cuModuleLoadData: unsafe extern "C" fn(*mut CUmodule, *const c_void) -> CUresult,
|
||||
cuModuleUnload: unsafe extern "C" fn(CUmodule) -> CUresult,
|
||||
cuModuleGetFunction: unsafe extern "C" fn(*mut CUfunction, CUmodule, *const c_char) -> CUresult,
|
||||
#[allow(clippy::type_complexity)]
|
||||
cuLaunchKernel: unsafe extern "C" fn(
|
||||
CUfunction,
|
||||
c_uint,
|
||||
c_uint,
|
||||
c_uint,
|
||||
c_uint,
|
||||
c_uint,
|
||||
c_uint,
|
||||
c_uint,
|
||||
CUstream,
|
||||
*mut *mut c_void,
|
||||
*mut *mut c_void,
|
||||
) -> CUresult,
|
||||
}
|
||||
// SAFETY: every field is a bare `extern "C" fn` address into the leaked, process-lifetime
|
||||
// `libcuda` mapping (`cuda_api` `forget`s the `Library`, so it is never unloaded) — an immutable
|
||||
// value with no interior mutability and no thread affinity. Moving the table to another thread
|
||||
// cannot dangle (the code it points at stays mapped) or race (the fields are read-only).
|
||||
unsafe impl Send for CudaApi {}
|
||||
// SAFETY: as above — the table is a set of immutable fn-pointer addresses with no interior
|
||||
// mutability, so concurrent shared reads from multiple threads cannot race; the driver entry
|
||||
// points they address are themselves thread-safe.
|
||||
unsafe impl Sync for CudaApi {}
|
||||
|
||||
/// `CUresult` returned by the wrappers when `libcuda` isn't loaded (no NVIDIA driver). Non-zero so
|
||||
/// the existing `ck()`/`!= 0` checks treat it as an ordinary driver error; distinct from any real
|
||||
/// `CUDA_ERROR_*` (all < 1000). Never produced by the actual driver.
|
||||
pub(crate) const CU_ERROR_NOT_LOADED: CUresult = 999;
|
||||
|
||||
pub(crate) static CUDA_API: OnceLock<Option<CudaApi>> = OnceLock::new();
|
||||
|
||||
/// Resolve `libcuda.so.1` and its symbols once. `None` when the NVIDIA driver isn't installed
|
||||
/// (the expected case on AMD/Intel hosts) — logged at debug, not an error.
|
||||
pub(crate) fn cuda_api() -> Option<&'static CudaApi> {
|
||||
CUDA_API
|
||||
// SAFETY: `Library::new` runs `libcuda.so.1`'s initializers — it is the trusted NVIDIA
|
||||
// driver library, so loading has no unexpected effects; `?`/`None` handle its absence.
|
||||
// Each `lib.get::<T>(name)` asserts the symbol's real ABI equals `T`: every NUL-terminated
|
||||
// name is a documented CUDA Driver API entry point and `T` is the exact
|
||||
// `unsafe extern "C" fn(..)` signature from cuda.h/cudaGL.h (`_v2` for ctx/mem ops). Each
|
||||
// `Symbol` only borrows `lib` until the end of the struct-literal statement; we deref-copy
|
||||
// the raw fn-pointer out first, then `forget(lib)` leaks the mapping so those addresses
|
||||
// stay valid for the whole process. Runs once under the `OnceLock` init — no aliasing.
|
||||
.get_or_init(|| unsafe {
|
||||
let lib = libloading::Library::new("libcuda.so.1")
|
||||
.or_else(|_| libloading::Library::new("libcuda.so"))
|
||||
.map_err(|e| {
|
||||
tracing::debug!(error = %e, "libcuda not loadable — CUDA zero-copy unavailable (expected on AMD/Intel)");
|
||||
})
|
||||
.ok()?;
|
||||
// Resolve all symbols; the field types drive `get`'s inference. `lib` is leaked after
|
||||
// construction so the fn pointers stay valid for the process lifetime (the temporary
|
||||
// `Symbol` borrows end with the struct-literal statement, before the forget).
|
||||
let api = CudaApi {
|
||||
cuInit: *lib.get(b"cuInit\0").ok()?,
|
||||
cuDeviceGet: *lib.get(b"cuDeviceGet\0").ok()?,
|
||||
cuCtxCreate_v2: *lib.get(b"cuCtxCreate_v2\0").ok()?,
|
||||
cuCtxDestroy_v2: *lib.get(b"cuCtxDestroy_v2\0").ok()?,
|
||||
cuCtxSetCurrent: *lib.get(b"cuCtxSetCurrent\0").ok()?,
|
||||
cuMemAllocPitch_v2: *lib.get(b"cuMemAllocPitch_v2\0").ok()?,
|
||||
cuMemFree_v2: *lib.get(b"cuMemFree_v2\0").ok()?,
|
||||
cuMemcpy2DAsync_v2: *lib.get(b"cuMemcpy2DAsync_v2\0").ok()?,
|
||||
cuStreamSynchronize: *lib.get(b"cuStreamSynchronize\0").ok()?,
|
||||
cuCtxGetStreamPriorityRange: *lib.get(b"cuCtxGetStreamPriorityRange\0").ok()?,
|
||||
cuStreamCreateWithPriority: *lib.get(b"cuStreamCreateWithPriority\0").ok()?,
|
||||
cuGraphicsGLRegisterImage: *lib.get(b"cuGraphicsGLRegisterImage\0").ok()?,
|
||||
cuGraphicsMapResources: *lib.get(b"cuGraphicsMapResources\0").ok()?,
|
||||
cuGraphicsUnmapResources: *lib.get(b"cuGraphicsUnmapResources\0").ok()?,
|
||||
cuGraphicsSubResourceGetMappedArray: *lib
|
||||
.get(b"cuGraphicsSubResourceGetMappedArray\0")
|
||||
.ok()?,
|
||||
cuGraphicsUnregisterResource: *lib.get(b"cuGraphicsUnregisterResource\0").ok()?,
|
||||
cuImportExternalMemory: *lib.get(b"cuImportExternalMemory\0").ok()?,
|
||||
cuExternalMemoryGetMappedBuffer: *lib
|
||||
.get(b"cuExternalMemoryGetMappedBuffer\0")
|
||||
.ok()?,
|
||||
cuDestroyExternalMemory: *lib.get(b"cuDestroyExternalMemory\0").ok()?,
|
||||
cuIpcGetMemHandle: *lib.get(b"cuIpcGetMemHandle\0").ok()?,
|
||||
// CUDA 11 renamed the entry point (per-thread-stream ABI split); every modern
|
||||
// driver exports `_v2`, but accept the unsuffixed one too (same signature).
|
||||
cuIpcOpenMemHandle: *lib
|
||||
.get(b"cuIpcOpenMemHandle_v2\0")
|
||||
.or_else(|_| lib.get(b"cuIpcOpenMemHandle\0"))
|
||||
.ok()?,
|
||||
cuIpcCloseMemHandle: *lib.get(b"cuIpcCloseMemHandle\0").ok()?,
|
||||
cuMemAlloc_v2: *lib.get(b"cuMemAlloc_v2\0").ok()?,
|
||||
cuModuleLoadData: *lib.get(b"cuModuleLoadData\0").ok()?,
|
||||
cuModuleUnload: *lib.get(b"cuModuleUnload\0").ok()?,
|
||||
cuModuleGetFunction: *lib.get(b"cuModuleGetFunction\0").ok()?,
|
||||
cuLaunchKernel: *lib.get(b"cuLaunchKernel\0").ok()?,
|
||||
};
|
||||
std::mem::forget(lib); // keep libcuda mapped for the fn pointers' lifetime (process)
|
||||
Some(api)
|
||||
})
|
||||
.as_ref()
|
||||
}
|
||||
|
||||
// Same-named wrappers so the call sites below are unchanged. Each forwards through the dlopen'd
|
||||
// table, or returns `CU_ERROR_NOT_LOADED` when the driver is absent (AMD/Intel) — which the
|
||||
// `CUresult` checks already handle. Only `context()` is reachable before the driver is confirmed
|
||||
// present; every other entry runs after `context()` succeeded, so its wrapper always hits `Some`.
|
||||
pub(crate) unsafe fn cuInit(flags: c_uint) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuInit)(flags),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuDeviceGet(device: *mut CUdevice, ordinal: c_int) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuDeviceGet)(device, ordinal),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuCtxCreate_v2(
|
||||
pctx: *mut CUcontext,
|
||||
flags: c_uint,
|
||||
dev: CUdevice,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuCtxCreate_v2)(pctx, flags, dev),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuCtxDestroy_v2(ctx: CUcontext) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuCtxDestroy_v2)(ctx),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuCtxSetCurrent(ctx: CUcontext) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuCtxSetCurrent)(ctx),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuMemAllocPitch_v2(
|
||||
dptr: *mut CUdeviceptr,
|
||||
pitch: *mut usize,
|
||||
width_bytes: usize,
|
||||
height: usize,
|
||||
element_size: c_uint,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuMemAllocPitch_v2)(dptr, pitch, width_bytes, height, element_size),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuMemFree_v2(dptr: CUdeviceptr) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuMemFree_v2)(dptr),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuMemAlloc_v2(dptr: *mut CUdeviceptr, size: usize) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuMemAlloc_v2)(dptr, size),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuModuleLoadData(m: *mut CUmodule, image: *const c_void) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuModuleLoadData)(m, image),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuModuleUnload(m: CUmodule) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuModuleUnload)(m),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuModuleGetFunction(
|
||||
f: *mut CUfunction,
|
||||
m: CUmodule,
|
||||
name: *const c_char,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuModuleGetFunction)(f, m, name),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(crate) unsafe fn cuLaunchKernel(
|
||||
f: CUfunction,
|
||||
gx: c_uint,
|
||||
gy: c_uint,
|
||||
gz: c_uint,
|
||||
bx: c_uint,
|
||||
by: c_uint,
|
||||
bz: c_uint,
|
||||
shmem: c_uint,
|
||||
stream: CUstream,
|
||||
params: *mut *mut c_void,
|
||||
extra: *mut *mut c_void,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuLaunchKernel)(f, gx, gy, gz, bx, by, bz, shmem, stream, params, extra),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuMemcpy2DAsync_v2(copy: *const CUDA_MEMCPY2D, stream: CUstream) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuMemcpy2DAsync_v2)(copy, stream),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuStreamSynchronize(stream: CUstream) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuStreamSynchronize)(stream),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuCtxGetStreamPriorityRange(
|
||||
least: *mut c_int,
|
||||
greatest: *mut c_int,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuCtxGetStreamPriorityRange)(least, greatest),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuStreamCreateWithPriority(
|
||||
stream: *mut CUstream,
|
||||
flags: c_uint,
|
||||
priority: c_int,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuStreamCreateWithPriority)(stream, flags, priority),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuGraphicsGLRegisterImage(
|
||||
resource: *mut CUgraphicsResource,
|
||||
texture: c_uint,
|
||||
target: c_uint,
|
||||
flags: c_uint,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuGraphicsGLRegisterImage)(resource, texture, target, flags),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuGraphicsMapResources(
|
||||
count: c_uint,
|
||||
resources: *mut CUgraphicsResource,
|
||||
stream: *mut c_void,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuGraphicsMapResources)(count, resources, stream),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuGraphicsUnmapResources(
|
||||
count: c_uint,
|
||||
resources: *mut CUgraphicsResource,
|
||||
stream: *mut c_void,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuGraphicsUnmapResources)(count, resources, stream),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuGraphicsSubResourceGetMappedArray(
|
||||
array: *mut CUarray,
|
||||
resource: CUgraphicsResource,
|
||||
array_index: c_uint,
|
||||
mip_level: c_uint,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuGraphicsSubResourceGetMappedArray)(array, resource, array_index, mip_level),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuGraphicsUnregisterResource(resource: CUgraphicsResource) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuGraphicsUnregisterResource)(resource),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuImportExternalMemory(
|
||||
ext_mem_out: *mut CUexternalMemory,
|
||||
mem_handle_desc: *const CUDA_EXTERNAL_MEMORY_HANDLE_DESC,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuImportExternalMemory)(ext_mem_out, mem_handle_desc),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuExternalMemoryGetMappedBuffer(
|
||||
dev_ptr: *mut CUdeviceptr,
|
||||
ext_mem: CUexternalMemory,
|
||||
buffer_desc: *const CUDA_EXTERNAL_MEMORY_BUFFER_DESC,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuExternalMemoryGetMappedBuffer)(dev_ptr, ext_mem, buffer_desc),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuDestroyExternalMemory(ext_mem: CUexternalMemory) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuDestroyExternalMemory)(ext_mem),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuIpcGetMemHandle(handle: *mut CUipcMemHandle, dptr: CUdeviceptr) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuIpcGetMemHandle)(handle, dptr),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuIpcOpenMemHandle(
|
||||
dptr: *mut CUdeviceptr,
|
||||
handle: CUipcMemHandle,
|
||||
flags: c_uint,
|
||||
) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuIpcOpenMemHandle)(dptr, handle, flags),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn cuIpcCloseMemHandle(dptr: CUdeviceptr) -> CUresult {
|
||||
match cuda_api() {
|
||||
Some(a) => (a.cuIpcCloseMemHandle)(dptr),
|
||||
None => CU_ERROR_NOT_LOADED,
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn ck(r: CUresult, what: &str) -> Result<()> {
|
||||
if r == 0 {
|
||||
Ok(())
|
||||
} else {
|
||||
bail!("CUDA driver error {r} in {what}")
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,191 @@
|
||||
//! GL plumbing for the EGL zero-copy blit (plan §W4, carved out of the EGL facade): the GL enum
|
||||
//! constants, the `#[link]`'d libGL / libgbm entry points, the fullscreen-triangle shader sources
|
||||
//! (BGRA swizzle + the NV12 / YUV444 BT.709 convert passes), and the shader/program compile
|
||||
//! helpers. The de-tiling blit passes and the EGLDisplay importer that drive this all live in
|
||||
//! [`super`].
|
||||
|
||||
#![allow(non_upper_case_globals)]
|
||||
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use anyhow::{bail, ensure, Result};
|
||||
use std::os::raw::{c_int, c_void};
|
||||
|
||||
pub(crate) const GL_TEXTURE_2D: u32 = 0x0DE1;
|
||||
pub(crate) const GL_TEXTURE_MIN_FILTER: u32 = 0x2801;
|
||||
pub(crate) const GL_TEXTURE_MAG_FILTER: u32 = 0x2800;
|
||||
pub(crate) const GL_LINEAR: c_int = 0x2601;
|
||||
pub(crate) const GL_NEAREST: c_int = 0x2600;
|
||||
pub(crate) const GL_RGBA8: u32 = 0x8058;
|
||||
// Single/dual-channel 8-bit formats for the NV12 convert targets: R8 luma (full-res),
|
||||
// RG8 interleaved chroma (half-res). The `_RED`/`_RG` enums are the matching client formats.
|
||||
pub(crate) const GL_R8: u32 = 0x8229;
|
||||
pub(crate) const GL_RG8: u32 = 0x822B;
|
||||
// Client pixel format/type for texture uploads (self-test only): RGBA bytes.
|
||||
pub(crate) const GL_RGBA: u32 = 0x1908;
|
||||
pub(crate) const GL_UNSIGNED_BYTE: u32 = 0x1401;
|
||||
pub(crate) const GL_FRAMEBUFFER: u32 = 0x8D40;
|
||||
pub(crate) const GL_COLOR_ATTACHMENT0: u32 = 0x8CE0;
|
||||
pub(crate) const GL_FRAMEBUFFER_COMPLETE: u32 = 0x8CD5;
|
||||
pub(crate) const GL_TEXTURE0: u32 = 0x84C0;
|
||||
pub(crate) const GL_TRIANGLES: u32 = 0x0004;
|
||||
pub(crate) const GL_VERTEX_SHADER: u32 = 0x8B31;
|
||||
pub(crate) const GL_FRAGMENT_SHADER: u32 = 0x8B30;
|
||||
pub(crate) const GL_COMPILE_STATUS: u32 = 0x8B81;
|
||||
pub(crate) const GL_LINK_STATUS: u32 = 0x8B82;
|
||||
|
||||
// libglvnd's libGL dispatches these to the NVIDIA driver based on the current EGL/GL context.
|
||||
#[link(name = "GL")]
|
||||
extern "C" {
|
||||
pub(crate) fn glGenTextures(n: c_int, textures: *mut u32);
|
||||
pub(crate) fn glBindTexture(target: u32, texture: u32);
|
||||
pub(crate) fn glTexParameteri(target: u32, pname: u32, param: c_int);
|
||||
pub(crate) fn glDeleteTextures(n: c_int, textures: *const u32);
|
||||
pub(crate) fn glTexStorage2D(
|
||||
target: u32,
|
||||
levels: c_int,
|
||||
internalformat: u32,
|
||||
width: c_int,
|
||||
height: c_int,
|
||||
);
|
||||
pub(crate) fn glGetError() -> u32;
|
||||
pub(crate) fn glGenFramebuffers(n: c_int, framebuffers: *mut u32);
|
||||
pub(crate) fn glDeleteFramebuffers(n: c_int, framebuffers: *const u32);
|
||||
pub(crate) fn glBindFramebuffer(target: u32, framebuffer: u32);
|
||||
pub(crate) fn glFramebufferTexture2D(
|
||||
target: u32,
|
||||
attachment: u32,
|
||||
textarget: u32,
|
||||
texture: u32,
|
||||
level: c_int,
|
||||
);
|
||||
pub(crate) fn glCheckFramebufferStatus(target: u32) -> u32;
|
||||
pub(crate) fn glViewport(x: c_int, y: c_int, width: c_int, height: c_int);
|
||||
pub(crate) fn glGenVertexArrays(n: c_int, arrays: *mut u32);
|
||||
pub(crate) fn glDeleteVertexArrays(n: c_int, arrays: *const u32);
|
||||
pub(crate) fn glBindVertexArray(array: u32);
|
||||
pub(crate) fn glDrawArrays(mode: u32, first: c_int, count: c_int);
|
||||
pub(crate) fn glActiveTexture(texture: u32);
|
||||
pub(crate) fn glUseProgram(program: u32);
|
||||
pub(crate) fn glFlush();
|
||||
pub(crate) fn glCreateShader(shader_type: u32) -> u32;
|
||||
pub(crate) fn glShaderSource(
|
||||
shader: u32,
|
||||
count: c_int,
|
||||
string: *const *const i8,
|
||||
length: *const c_int,
|
||||
);
|
||||
pub(crate) fn glCompileShader(shader: u32);
|
||||
pub(crate) fn glGetShaderiv(shader: u32, pname: u32, params: *mut c_int);
|
||||
pub(crate) fn glDeleteShader(shader: u32);
|
||||
pub(crate) fn glCreateProgram() -> u32;
|
||||
pub(crate) fn glAttachShader(program: u32, shader: u32);
|
||||
pub(crate) fn glLinkProgram(program: u32);
|
||||
pub(crate) fn glGetProgramiv(program: u32, pname: u32, params: *mut c_int);
|
||||
pub(crate) fn glGetUniformLocation(program: u32, name: *const i8) -> c_int;
|
||||
pub(crate) fn glUniform1i(location: c_int, v0: c_int);
|
||||
pub(crate) fn glDeleteProgram(program: u32);
|
||||
pub(crate) fn glTexSubImage2D(
|
||||
target: u32,
|
||||
level: c_int,
|
||||
xoffset: c_int,
|
||||
yoffset: c_int,
|
||||
width: c_int,
|
||||
height: c_int,
|
||||
format: u32,
|
||||
type_: u32,
|
||||
pixels: *const c_void,
|
||||
);
|
||||
}
|
||||
|
||||
#[link(name = "gbm")]
|
||||
extern "C" {
|
||||
pub(crate) fn gbm_create_device(fd: c_int) -> *mut c_void;
|
||||
pub(crate) fn gbm_device_destroy(device: *mut c_void);
|
||||
}
|
||||
|
||||
/// `glEGLImageTargetTexture2DOES(target, EGLImage)` — loaded via `eglGetProcAddress`.
|
||||
pub(crate) type EglImageTargetFn = unsafe extern "system" fn(u32, *mut c_void);
|
||||
|
||||
// Fullscreen-triangle blit: sample the dmabuf EGLImage texture and write it (swizzled to BGRA,
|
||||
// to match the BGRx the encoder expects) into a normal GL_RGBA8 texture that CUDA *can* register.
|
||||
pub(crate) const VERT_SRC: &[u8] = b"#version 330 core\nout vec2 v_tex;\nvoid main(){vec2 p=vec2(float((gl_VertexID<<1)&2),float(gl_VertexID&2));v_tex=p;gl_Position=vec4(p*2.0-1.0,0.0,1.0);}\n";
|
||||
pub(crate) const FRAG_SRC: &[u8] = b"#version 330 core\nuniform sampler2D image;\nin vec2 v_tex;\nout vec4 o_color;\nvoid main(){o_color=texture(image,v_tex).bgra;}\n";
|
||||
|
||||
// NV12 BT.709 LIMITED-range convert from full-range RGB in [0,1]. Two passes share `VERT_SRC` and
|
||||
// the same source texture (the de-tiled dmabuf):
|
||||
// Y pass → GL_R8 luma, full-res: Y = (16 + 219·(0.2126R+0.7152G+0.0722B))/255
|
||||
// UV pass → GL_RG8 chroma, half-res (GL_LINEAR averages the 2×2 footprint):
|
||||
// U = (128 + 224·(-0.1146R-0.3854G+0.5000B))/255 → R channel
|
||||
// V = (128 + 224·( 0.5000R-0.4542G-0.0458B))/255 → G channel
|
||||
// RG8's (R=U, G=V) byte order matches NV12's interleaved [U,V]. All outputs clamped to [0,1].
|
||||
// Matches the Windows VideoConverter (BT.709, limited/studio range) so the two hosts look identical.
|
||||
pub(crate) const FRAG_Y_SRC: &[u8] = b"#version 330 core\nuniform sampler2D image;\nin vec2 v_tex;\nout vec4 o_color;\nvoid main(){vec3 c=texture(image,v_tex).rgb;float Y=(16.0+219.0*(0.2126*c.r+0.7152*c.g+0.0722*c.b))/255.0;o_color=vec4(clamp(Y,0.0,1.0),0.0,0.0,1.0);}\n";
|
||||
pub(crate) const FRAG_UV_SRC: &[u8] = b"#version 330 core\nuniform sampler2D image;\nin vec2 v_tex;\nout vec4 o_color;\nvoid main(){vec3 c=texture(image,v_tex).rgb;float U=(128.0+224.0*(-0.1146*c.r-0.3854*c.g+0.5000*c.b))/255.0;float V=(128.0+224.0*(0.5000*c.r-0.4542*c.g-0.0458*c.b))/255.0;o_color=vec4(clamp(U,0.0,1.0),clamp(V,0.0,1.0),0.0,1.0);}\n";
|
||||
|
||||
/// The three planar-YUV444 convert shaders (full-res `R8` target each) — the [`Yuv444Blit`]
|
||||
/// analogue of `FRAG_Y_SRC`/`FRAG_UV_SRC` with NO subsampling (4:4:4 keeps every chroma sample).
|
||||
/// Same BT.709 coefficients; `full_range` flips the quantization from studio (16+219 / 128±112)
|
||||
/// to the full 0..255 swing — the encoder flips the VUI (`PUNKTFUNK_444_FULLRANGE`, read by both
|
||||
/// processes from the same inherited environment) in lockstep, so pixels and signaling agree.
|
||||
pub(crate) fn yuv444_frag_sources(full_range: bool) -> (Vec<u8>, Vec<u8>, Vec<u8>) {
|
||||
let (y_scale, y_off, c_scale) = if full_range {
|
||||
("255.0", "0.0", "255.0")
|
||||
} else {
|
||||
("219.0", "16.0", "224.0")
|
||||
};
|
||||
let head = "#version 330 core\nuniform sampler2D image;\nin vec2 v_tex;\nout vec4 o_color;\nvoid main(){vec3 c=texture(image,v_tex).rgb;";
|
||||
let y = format!(
|
||||
"{head}float Y=({y_off}+{y_scale}*(0.2126*c.r+0.7152*c.g+0.0722*c.b))/255.0;o_color=vec4(clamp(Y,0.0,1.0),0.0,0.0,1.0);}}\n"
|
||||
);
|
||||
let u = format!(
|
||||
"{head}float U=(128.0+{c_scale}*(-0.1146*c.r-0.3854*c.g+0.5000*c.b))/255.0;o_color=vec4(clamp(U,0.0,1.0),0.0,0.0,1.0);}}\n"
|
||||
);
|
||||
let v = format!(
|
||||
"{head}float V=(128.0+{c_scale}*(0.5000*c.r-0.4542*c.g-0.0458*c.b))/255.0;o_color=vec4(clamp(V,0.0,1.0),0.0,0.0,1.0);}}\n"
|
||||
);
|
||||
(y.into_bytes(), u.into_bytes(), v.into_bytes())
|
||||
}
|
||||
|
||||
pub(crate) unsafe fn compile_shader(kind: u32, src: &[u8]) -> Result<u32> {
|
||||
let sh = glCreateShader(kind);
|
||||
ensure!(sh != 0, "glCreateShader failed");
|
||||
let ptr = src.as_ptr() as *const i8;
|
||||
let len = src.len() as c_int;
|
||||
glShaderSource(sh, 1, &ptr, &len);
|
||||
glCompileShader(sh);
|
||||
let mut ok: c_int = 0;
|
||||
glGetShaderiv(sh, GL_COMPILE_STATUS, &mut ok);
|
||||
if ok == 0 {
|
||||
glDeleteShader(sh);
|
||||
bail!("GL shader compile failed");
|
||||
}
|
||||
Ok(sh)
|
||||
}
|
||||
|
||||
/// Compile+link the fullscreen-triangle program with fragment source `frag` and bind its `image`
|
||||
/// sampler to texture unit 0.
|
||||
pub(crate) unsafe fn compile_program_with(frag: &[u8]) -> Result<u32> {
|
||||
let vs = compile_shader(GL_VERTEX_SHADER, VERT_SRC)?;
|
||||
let fs = compile_shader(GL_FRAGMENT_SHADER, frag)?;
|
||||
let prog = glCreateProgram();
|
||||
glAttachShader(prog, vs);
|
||||
glAttachShader(prog, fs);
|
||||
glLinkProgram(prog);
|
||||
glDeleteShader(vs);
|
||||
glDeleteShader(fs);
|
||||
let mut ok: c_int = 0;
|
||||
glGetProgramiv(prog, GL_LINK_STATUS, &mut ok);
|
||||
ensure!(ok != 0, "GL program link failed");
|
||||
glUseProgram(prog);
|
||||
let loc = glGetUniformLocation(prog, c"image".as_ptr());
|
||||
if loc >= 0 {
|
||||
glUniform1i(loc, 0); // sampler -> texture unit 0
|
||||
}
|
||||
glUseProgram(0);
|
||||
Ok(prog)
|
||||
}
|
||||
|
||||
pub(crate) unsafe fn compile_program() -> Result<u32> {
|
||||
compile_program_with(FRAG_SRC)
|
||||
}
|
||||
@@ -0,0 +1,410 @@
|
||||
//! Zero-copy capture→encode (plan §9): the PipeWire dmabuf is imported into CUDA via EGL and
|
||||
//! handed straight to NVENC, eliminating the per-frame CPU copies (at 5K the CPU-copy path
|
||||
//! moves ~3.5 GB/s). On NVENC opt in with `PUNKTFUNK_ZEROCOPY=1` (the CPU-copy path stays that
|
||||
//! backend's default and the runtime fallback: foreign-allocator / no-dmabuf / import failure).
|
||||
//! On the VAAPI (AMD/Intel) backend zero-copy is the **default** — its LINEAR-dmabuf passthrough
|
||||
//! replaces a triple CPU touch (mmap de-pad + swscale CSC + surface upload) — with a one-shot
|
||||
//! downgrade to the CPU path if the compositor never accepts the dmabuf offer.
|
||||
//!
|
||||
//! Pieces: [`cuda`] (driver-API FFI + the shared `CUcontext` + device buffers), [`egl`] (the
|
||||
//! headless EGLDisplay + dmabuf→`EGLImage`→CUDA import). The encoder's CUDA-frame path lives in
|
||||
//! the encode backends (`pf-encode`); the dmabuf negotiation lives in the Linux capturer
|
||||
//! (`pf-capture`).
|
||||
|
||||
pub mod client;
|
||||
pub mod cuda;
|
||||
pub mod egl;
|
||||
pub mod proto;
|
||||
pub mod vulkan;
|
||||
pub mod worker;
|
||||
|
||||
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
|
||||
|
||||
pub use cuda::DeviceBuffer;
|
||||
pub use egl::{DmabufPlane, EglImporter};
|
||||
|
||||
/// Whether a `PUNKTFUNK_*` flag is truthy (`1`/`true`/`yes`/`on`), or `None` when unset.
|
||||
fn flag_opt(name: &str) -> Option<bool> {
|
||||
std::env::var(name)
|
||||
.ok()
|
||||
.map(|v| matches!(v.trim(), "1" | "true" | "yes" | "on"))
|
||||
}
|
||||
|
||||
/// Whether a `PUNKTFUNK_*` flag is truthy (`1`/`true`/`yes`/`on`); unset ⇒ false.
|
||||
fn flag(name: &str) -> bool {
|
||||
flag_opt(name).unwrap_or(false)
|
||||
}
|
||||
|
||||
/// One-shot downgrade latch: a VAAPI-passthrough capture whose dmabuf-only offer never negotiated
|
||||
/// (the compositor can't allocate a LINEAR BGRx dmabuf) flips this, so the encode loop's pipeline
|
||||
/// rebuild lands on the CPU offer instead of failing the same negotiation forever. Only consulted
|
||||
/// when `PUNKTFUNK_ZEROCOPY` is unset — an explicit `=1` keeps forcing the dmabuf offer.
|
||||
static VAAPI_DMABUF_FAILED: AtomicBool = AtomicBool::new(false);
|
||||
|
||||
/// Record that the VAAPI LINEAR-dmabuf offer failed negotiation (see [`VAAPI_DMABUF_FAILED`]).
|
||||
pub fn note_vaapi_dmabuf_failed() {
|
||||
VAAPI_DMABUF_FAILED.store(true, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// True when `PUNKTFUNK_ZEROCOPY` is explicitly truthy — the operator forced the dmabuf offer, so
|
||||
/// a failed negotiation keeps erroring loudly instead of silently downgrading to the CPU path.
|
||||
pub fn vaapi_dmabuf_forced() -> bool {
|
||||
flag_opt("PUNKTFUNK_ZEROCOPY") == Some(true)
|
||||
}
|
||||
|
||||
/// Whether the zero-copy path is on. `PUNKTFUNK_ZEROCOPY` decides when set (truthy = on, else off).
|
||||
/// **Unset defaults ON for both GPU backends** — the stock install gets the GPU dmabuf path, not
|
||||
/// three full-frame CPU touches. This includes NVENC (previously opt-in): the EGL→CUDA (tiled) and
|
||||
/// Vulkan (LINEAR) imports now run in a per-capture worker subprocess
|
||||
/// (`design/zerocopy-worker-isolation.md`), so a driver fault on a producer-invalidated dmabuf kills
|
||||
/// the worker and the host degrades to its capture-loss rebuild instead of dying — the reason the
|
||||
/// NVENC path stayed opt-in is gone. Fallbacks stay in place: VAAPI has a one-shot CPU downgrade if
|
||||
/// the LINEAR-dmabuf offer never negotiates ([`note_vaapi_dmabuf_failed`]); NVENC falls back per
|
||||
/// capture when no importer/importable modifier is available and latches the import off after
|
||||
/// repeated worker deaths. `PUNKTFUNK_ZEROCOPY=0` opts out; `PUNKTFUNK_FORCE_SHM` forces the
|
||||
/// race-free SHM path.
|
||||
pub fn enabled() -> bool {
|
||||
match flag_opt("PUNKTFUNK_ZEROCOPY") {
|
||||
Some(v) => v,
|
||||
None => !VAAPI_DMABUF_FAILED.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the tiled-GL zero-copy path converts to NV12 on the GPU and feeds NVENC native YUV —
|
||||
/// deleting NVENC's internal RGB→YUV CSC, which otherwise runs on the SM/3D engine the game
|
||||
/// saturates (Tier 2A). **Default ON** (validated color-correct on the RTX 5070 Ti via
|
||||
/// `nv12-selftest` + live decode on dev + Bazzite/KWin boxes; latency- and CPU-neutral idle,
|
||||
/// frees SM headroom under load — the same default the Windows host ships). `PUNKTFUNK_NV12=0`
|
||||
/// restores the RGB/BGRx feed. LINEAR (gamescope/Vulkan-bridge) captures are unaffected either way.
|
||||
pub fn nv12_enabled() -> bool {
|
||||
flag_opt("PUNKTFUNK_NV12").unwrap_or(true)
|
||||
}
|
||||
|
||||
/// The GPU importer a capture uses — normally the [`client::RemoteImporter`] worker subprocess
|
||||
/// (design: `design/zerocopy-worker-isolation.md`), so a driver fault on a producer-invalidated
|
||||
/// dmabuf kills the worker instead of the host. `PUNKTFUNK_ZEROCOPY_INPROC=1` keeps the import
|
||||
/// in-process (the pre-isolation behavior) for debugging and A/B latency comparison.
|
||||
pub enum Importer {
|
||||
Remote(client::RemoteImporter),
|
||||
InProc(Box<EglImporter>),
|
||||
}
|
||||
|
||||
impl Importer {
|
||||
/// Build the importer for a capture session, honoring the `PUNKTFUNK_ZEROCOPY_INPROC`
|
||||
/// escape hatch. An `Err` means "no GPU import available" — callers fall back to the CPU path.
|
||||
pub fn new_for_capture() -> anyhow::Result<Importer> {
|
||||
if flag("PUNKTFUNK_ZEROCOPY_INPROC") {
|
||||
tracing::warn!(
|
||||
"PUNKTFUNK_ZEROCOPY_INPROC=1 — GPU import runs IN-PROCESS; a driver fault on a \
|
||||
dying compositor's dmabuf can take the whole host down (debug/A-B use only)"
|
||||
);
|
||||
return Ok(Importer::InProc(Box::new(EglImporter::new()?)));
|
||||
}
|
||||
Ok(Importer::Remote(client::RemoteImporter::spawn()?))
|
||||
}
|
||||
|
||||
pub fn supported_modifiers(&mut self, fourcc: u32) -> Vec<u64> {
|
||||
match self {
|
||||
Importer::Remote(r) => r.supported_modifiers(fourcc),
|
||||
Importer::InProc(i) => i.supported_modifiers(fourcc),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn import(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> anyhow::Result<DeviceBuffer> {
|
||||
match self {
|
||||
Importer::Remote(r) => r.import(plane, width, height, fourcc, modifier),
|
||||
Importer::InProc(i) => i.import(plane, width, height, fourcc, modifier),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn import_nv12(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> anyhow::Result<DeviceBuffer> {
|
||||
match self {
|
||||
Importer::Remote(r) => r.import_nv12(plane, width, height, fourcc, modifier),
|
||||
Importer::InProc(i) => i.import_nv12(plane, width, height, fourcc, modifier),
|
||||
}
|
||||
}
|
||||
|
||||
/// Tiled dmabuf → planar-YUV444 GPU convert → one stacked 3-plane CUDA buffer (the 4:4:4
|
||||
/// zero-copy path).
|
||||
pub fn import_yuv444(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
) -> anyhow::Result<DeviceBuffer> {
|
||||
match self {
|
||||
Importer::Remote(r) => r.import_yuv444(plane, width, height, fourcc, modifier),
|
||||
Importer::InProc(i) => i.import_yuv444(plane, width, height, fourcc, modifier),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn import_linear(
|
||||
&mut self,
|
||||
plane: &DmabufPlane,
|
||||
width: u32,
|
||||
height: u32,
|
||||
) -> anyhow::Result<DeviceBuffer> {
|
||||
match self {
|
||||
Importer::Remote(r) => r.import_linear(plane, width, height),
|
||||
Importer::InProc(i) => i.import_linear(plane, width, height),
|
||||
}
|
||||
}
|
||||
|
||||
/// True once the worker process is gone/wedged (every further call fails fast). Always
|
||||
/// `false` in-process — an in-process driver fault doesn't return.
|
||||
pub fn dead(&self) -> bool {
|
||||
match self {
|
||||
Importer::Remote(r) => r.dead(),
|
||||
Importer::InProc(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// The PipeWire stream renegotiated its format (the buffer pool is replaced) — drop all
|
||||
/// per-buffer caches so a recycled fd number can never resolve to a stale import.
|
||||
pub fn clear_cache(&mut self) {
|
||||
match self {
|
||||
Importer::Remote(r) => r.clear_cache(),
|
||||
Importer::InProc(i) => i.clear_linear_cache(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Consecutive zero-copy worker deaths without a successful import in between. A short streak is
|
||||
/// normal (the observed trigger — a compositor crash — kills the worker once, and the rebuilt
|
||||
/// session's fresh worker succeeds); a sustained streak means the GPU stack itself is wedged and
|
||||
/// respawning would crash-loop, so [`note_gpu_import_death`] latches [`GPU_IMPORT_DISABLED`] and
|
||||
/// every later capture negotiates the safe CPU/SHM path instead.
|
||||
static GPU_IMPORT_DEATH_STREAK: AtomicU32 = AtomicU32::new(0);
|
||||
static GPU_IMPORT_DISABLED: AtomicBool = AtomicBool::new(false);
|
||||
const GPU_IMPORT_DEATH_LATCH: u32 = 3;
|
||||
|
||||
/// Record a worker death (transport-level failure). Latches the process-wide disable after
|
||||
/// [`GPU_IMPORT_DEATH_LATCH`] consecutive deaths.
|
||||
pub fn note_gpu_import_death() {
|
||||
let streak = GPU_IMPORT_DEATH_STREAK.fetch_add(1, Ordering::Relaxed) + 1;
|
||||
if streak >= GPU_IMPORT_DEATH_LATCH && !GPU_IMPORT_DISABLED.swap(true, Ordering::Relaxed) {
|
||||
tracing::error!(
|
||||
streak,
|
||||
"zero-copy GPU import disabled for this host process: the import worker died {streak} \
|
||||
times in a row (GPU/driver stack unstable) — captures fall back to the CPU path"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Record a successful GPU import — resets the death streak (the stack works again).
|
||||
pub fn note_gpu_import_ok() {
|
||||
GPU_IMPORT_DEATH_STREAK.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// True once repeated worker deaths latched the GPU import off (see [`note_gpu_import_death`]).
|
||||
pub fn gpu_import_disabled() -> bool {
|
||||
GPU_IMPORT_DISABLED.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// DRM FourCC for a packed 32-bit format name (little-endian, e.g. `b"XR24"`).
|
||||
const fn fourcc(c: &[u8; 4]) -> u32 {
|
||||
(c[0] as u32) | ((c[1] as u32) << 8) | ((c[2] as u32) << 16) | ((c[3] as u32) << 24)
|
||||
}
|
||||
|
||||
/// Standalone probe (the `zerocopy-probe` subcommand): initialize the EGL importer + CUDA
|
||||
/// context and report, then exercise the production path — spawn the isolated worker (exec'd
|
||||
/// from this binary's pinned exe fd), handshake, and query modifiers. De-risks the
|
||||
/// FFI/linking/GPU-access AND the worker spawn (e.g. the installed binary replaced under a
|
||||
/// running host) without needing a capture session.
|
||||
pub fn probe() -> anyhow::Result<()> {
|
||||
let _importer = EglImporter::new()?;
|
||||
let ctx = cuda::context()?;
|
||||
tracing::info!(cuda_ctx = ?ctx, "zero-copy probe OK — EGL display + CUDA context initialized");
|
||||
let mut worker = client::RemoteImporter::spawn()?;
|
||||
let modifiers = worker.supported_modifiers(fourcc(b"XR24")).len();
|
||||
tracing::info!(
|
||||
modifiers,
|
||||
"zero-copy probe OK — worker spawned, handshake + modifier query"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Reference BT.709 LIMITED-range conversion of one full-range RGB pixel (`u8`) to (Y, U, V) in
|
||||
/// `f64`, matching the GPU shaders in [`egl`]. Y in [16,235], U/V in [16,240].
|
||||
fn bt709_limited(r: u8, g: u8, b: u8) -> (f64, f64, f64) {
|
||||
let (r, g, b) = (r as f64 / 255.0, g as f64 / 255.0, b as f64 / 255.0);
|
||||
let y = 16.0 + 219.0 * (0.2126 * r + 0.7152 * g + 0.0722 * b);
|
||||
let u = 128.0 + 224.0 * (-0.1146 * r - 0.3854 * g + 0.5000 * b);
|
||||
let v = 128.0 + 224.0 * (0.5000 * r - 0.4542 * g - 0.0458 * b);
|
||||
(y, u, v)
|
||||
}
|
||||
|
||||
/// NV12 colour self-test (the `nv12-selftest` subcommand): stand up the EGL/GL + CUDA stack, upload
|
||||
/// a known synthetic RGBA pattern, run the real NV12 convert shaders on the GPU, read the Y and UV
|
||||
/// planes back, and compare against a Rust BT.709 limited-range reference. Validates colour
|
||||
/// correctness on the GPU **without a display** (the project's green-screen bugs came from exactly
|
||||
/// this kind of plane/layout error). PASS if max abs error Y ≤ 2, U/V ≤ 3.
|
||||
pub fn nv12_selftest() -> anyhow::Result<()> {
|
||||
use anyhow::bail;
|
||||
|
||||
// 64x64, even dims. A 4x4 grid of 16x16 flat-colour blocks (so each 2x2 chroma footprint is
|
||||
// uniform → exact chroma comparison) covering the primaries + gray/black/white, then the rest
|
||||
// is a diagonal gradient (every pixel changes — a Y-channel stress that also exercises the
|
||||
// chroma averaging; the gradient blocks are compared on Y only).
|
||||
const W: u32 = 64;
|
||||
const H: u32 = 64;
|
||||
const BLK: u32 = 16;
|
||||
// (name, r, g, b) for the labelled blocks in row-major grid order; the rest fall to gradient.
|
||||
let named: [(&str, u8, u8, u8); 8] = [
|
||||
("red", 255, 0, 0),
|
||||
("green", 0, 255, 0),
|
||||
("blue", 0, 0, 255),
|
||||
("white", 255, 255, 255),
|
||||
("black", 0, 0, 0),
|
||||
("gray128", 128, 128, 128),
|
||||
("yellow", 255, 255, 0),
|
||||
("cyan", 0, 255, 255),
|
||||
];
|
||||
|
||||
// Build the RGBA pattern + a parallel record of each pixel's (r,g,b) and whether it sits in a
|
||||
// flat block (chroma-comparable) or the gradient (Y-only).
|
||||
let mut rgba = vec![0u8; (W * H * 4) as usize];
|
||||
let mut flat = vec![false; (W * H) as usize];
|
||||
let grid_cols = W / BLK; // 4
|
||||
let pixel_rgb = |x: u32, y: u32| -> (u8, u8, u8, bool) {
|
||||
let bx = x / BLK;
|
||||
let by = y / BLK;
|
||||
let idx = (by * grid_cols + bx) as usize;
|
||||
if idx < named.len() {
|
||||
let (_, r, g, b) = named[idx];
|
||||
(r, g, b, true)
|
||||
} else {
|
||||
// Diagonal gradient — distinct per pixel.
|
||||
let r = ((x * 4) & 0xff) as u8;
|
||||
let g = ((y * 4) & 0xff) as u8;
|
||||
let b = (((x + y) * 2) & 0xff) as u8;
|
||||
(r, g, b, false)
|
||||
}
|
||||
};
|
||||
for y in 0..H {
|
||||
for x in 0..W {
|
||||
let (r, g, b, is_flat) = pixel_rgb(x, y);
|
||||
let i = ((y * W + x) * 4) as usize;
|
||||
rgba[i] = r;
|
||||
rgba[i + 1] = g;
|
||||
rgba[i + 2] = b;
|
||||
rgba[i + 3] = 255;
|
||||
flat[(y * W + x) as usize] = is_flat;
|
||||
}
|
||||
}
|
||||
|
||||
// GPU convert.
|
||||
let mut importer = EglImporter::new()?;
|
||||
let nv12 = importer.convert_rgba_for_test(&rgba, W, H)?;
|
||||
let (uv_ptr, uv_pitch) = nv12
|
||||
.uv
|
||||
.ok_or_else(|| anyhow::anyhow!("self-test buffer is not NV12"))?;
|
||||
// Read both planes back to host (tightly packed).
|
||||
let y_host = cuda::read_plane_to_host(nv12.ptr, nv12.pitch, W as usize, H as usize)?;
|
||||
let uv_host = cuda::read_plane_to_host(uv_ptr, uv_pitch, (W as usize / 2) * 2, H as usize / 2)?;
|
||||
|
||||
// Compare Y over every pixel.
|
||||
let mut max_y_err = 0.0f64;
|
||||
for y in 0..H {
|
||||
for x in 0..W {
|
||||
let (r, g, b, _) = pixel_rgb(x, y);
|
||||
let (ref_y, _, _) = bt709_limited(r, g, b);
|
||||
let got = y_host[(y * W + x) as usize] as f64;
|
||||
max_y_err = max_y_err.max((got - ref_y).abs());
|
||||
}
|
||||
}
|
||||
|
||||
// Compare U/V over flat blocks only (each 2x2 footprint is a single colour → exact reference).
|
||||
// Chroma is W/2 × H/2 samples, interleaved [U,V] per sample.
|
||||
let cw = W / 2;
|
||||
let ch = H / 2;
|
||||
let mut max_u_err = 0.0f64;
|
||||
let mut max_v_err = 0.0f64;
|
||||
for cy in 0..ch {
|
||||
for cx in 0..cw {
|
||||
// The 2x2 source footprint of this chroma sample.
|
||||
let (sx, sy) = (cx * 2, cy * 2);
|
||||
// Only compare where all 4 source pixels are flat (uniform colour).
|
||||
let all_flat =
|
||||
(0..2).all(|dy| (0..2).all(|dx| flat[((sy + dy) * W + (sx + dx)) as usize]));
|
||||
if !all_flat {
|
||||
continue;
|
||||
}
|
||||
let (r, g, b, _) = pixel_rgb(sx, sy);
|
||||
let (_, ref_u, ref_v) = bt709_limited(r, g, b);
|
||||
let base = ((cy * cw + cx) * 2) as usize;
|
||||
let got_u = uv_host[base] as f64;
|
||||
let got_v = uv_host[base + 1] as f64;
|
||||
max_u_err = max_u_err.max((got_u - ref_u).abs());
|
||||
max_v_err = max_v_err.max((got_v - ref_v).abs());
|
||||
}
|
||||
}
|
||||
|
||||
// Per-primary actual-vs-expected (block centre for chroma).
|
||||
println!("NV12 self-test ({W}x{H}, BT.709 limited range)");
|
||||
println!(
|
||||
" {:<8} {:>14} {:>14} {:>14}",
|
||||
"color", "Y exp/got", "U exp/got", "V exp/got"
|
||||
);
|
||||
for (idx, (name, r, g, b)) in named.iter().enumerate() {
|
||||
let bx = (idx as u32 % grid_cols) * BLK + BLK / 2;
|
||||
let by = (idx as u32 / grid_cols) * BLK + BLK / 2;
|
||||
let (ey, eu, ev) = bt709_limited(*r, *g, *b);
|
||||
let gy = y_host[(by * W + bx) as usize] as f64;
|
||||
let (ccx, ccy) = (bx / 2, by / 2);
|
||||
let cbase = ((ccy * cw + ccx) * 2) as usize;
|
||||
let gu = uv_host[cbase] as f64;
|
||||
let gv = uv_host[cbase + 1] as f64;
|
||||
println!(
|
||||
" {:<8} {:>6.1}/{:<6.0} {:>6.1}/{:<6.0} {:>6.1}/{:<6.0}",
|
||||
name, ey, gy, eu, gu, ev, gv
|
||||
);
|
||||
}
|
||||
println!(
|
||||
" max abs error: Y={max_y_err:.2} (≤2) U={max_u_err:.2} (≤3) V={max_v_err:.2} (≤3)"
|
||||
);
|
||||
|
||||
if max_y_err <= 2.0 && max_u_err <= 3.0 && max_v_err <= 3.0 {
|
||||
println!("PASS");
|
||||
Ok(())
|
||||
} else {
|
||||
println!("FAIL");
|
||||
bail!("NV12 self-test FAILED (Y={max_y_err:.2} U={max_u_err:.2} V={max_v_err:.2})");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Single test owning the process-global latch statics (they are never reset by design).
|
||||
#[test]
|
||||
fn gpu_import_death_latch() {
|
||||
note_gpu_import_death();
|
||||
note_gpu_import_ok(); // a successful import resets the streak
|
||||
note_gpu_import_death();
|
||||
note_gpu_import_death();
|
||||
assert!(
|
||||
!gpu_import_disabled(),
|
||||
"two consecutive deaths must not latch"
|
||||
);
|
||||
note_gpu_import_death(); // third consecutive death
|
||||
assert!(gpu_import_disabled());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,395 @@
|
||||
//! Wire protocol between the PipeWire capture thread and the isolated zero-copy GPU-import
|
||||
//! worker process (`punktfunk-host zerocopy-worker`; design:
|
||||
//! [`design/zerocopy-worker-isolation.md`]). Transport is a `SOCK_SEQPACKET` unix socketpair —
|
||||
//! reliable, ordered, message-framed (one `sendmsg` = one message) — with dmabuf fds riding as
|
||||
//! `SCM_RIGHTS` control data. Bodies are small serde_json blobs (~200 B/frame); pixels never
|
||||
//! cross the socket (they move GPU-side via CUDA IPC, see [`super::cuda::ipc_export`]).
|
||||
//!
|
||||
//! Zero-length messages are reserved: `recvmsg` returning 0 on a SEQPACKET socket is EOF (the
|
||||
//! peer died/closed), and every serialized message here is non-empty JSON, so the two can't be
|
||||
//! confused.
|
||||
|
||||
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use serde::de::DeserializeOwned;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::io;
|
||||
use std::os::fd::{AsRawFd, BorrowedFd, FromRawFd, OwnedFd};
|
||||
use std::time::Duration;
|
||||
|
||||
/// Bumped on any wire change; the worker echoes it in [`Reply::Ready`] and the host refuses a
|
||||
/// mismatch. Host and worker are the same binary (`/proc/self/exe`), so this only ever trips on
|
||||
/// exotic deployment mistakes (a stale binary re-exec'd across an upgrade).
|
||||
pub const PROTO_VERSION: u32 = 1;
|
||||
|
||||
/// Upper bound for one serialized message (the largest real message — a modifier list — is far
|
||||
/// below this). A message reported truncated at this size is a protocol error.
|
||||
pub const MAX_MSG: usize = 64 * 1024;
|
||||
|
||||
/// How a dmabuf should be imported — mirrors the `EglImporter` entry points.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum ImportKind {
|
||||
/// Tiled dmabuf → EGL/GL de-tile blit → BGRx CUDA buffer.
|
||||
Tiled,
|
||||
/// Tiled dmabuf → EGL/GL NV12 convert → two-plane CUDA buffer (`PUNKTFUNK_NV12`).
|
||||
TiledNv12,
|
||||
/// LINEAR dmabuf → Vulkan bridge → BGRx CUDA buffer (gamescope's only offer).
|
||||
Linear,
|
||||
/// Tiled dmabuf → EGL/GL planar-YUV444 convert → ONE stacked 3-plane CUDA buffer (a 4:4:4
|
||||
/// session). APPENDED last: the worker can outlive a replaced host binary, so the earlier
|
||||
/// variants' wire tags must never shift — an old worker receiving this fails the decode and
|
||||
/// the import-fail machinery handles it like any other worker error.
|
||||
Tiled444,
|
||||
}
|
||||
|
||||
/// host → worker.
|
||||
#[derive(Serialize, Deserialize, Debug, PartialEq)]
|
||||
pub enum Request {
|
||||
/// The EGL-importable DRM modifiers for `fourcc` (startup, before the stream connects —
|
||||
/// the host advertises these to PipeWire).
|
||||
Modifiers { fourcc: u32 },
|
||||
/// Import one frame. `key` identifies the underlying dmabuf across frames (the host uses
|
||||
/// the fd's `st_ino` — unique per dma-buf object); the fd itself rides along as
|
||||
/// `SCM_RIGHTS` only on first sight of `key` (`has_fd`), and the worker keeps its dup.
|
||||
Import {
|
||||
key: u64,
|
||||
kind: ImportKind,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fourcc: u32,
|
||||
modifier: Option<u64>,
|
||||
offset: u32,
|
||||
stride: u32,
|
||||
has_fd: bool,
|
||||
},
|
||||
/// The frame buffer previously delivered as `id` is no longer in use — recycle it into the
|
||||
/// worker's pool. Fire-and-forget (no reply); may be sent from any host thread.
|
||||
Release { id: u32 },
|
||||
/// The PipeWire stream renegotiated its format: the buffer pool is gone, so drop all cached
|
||||
/// per-`key` state (stored fds, Vulkan per-fd imports). Fire-and-forget.
|
||||
ClearCache,
|
||||
}
|
||||
|
||||
/// worker → host.
|
||||
#[derive(Serialize, Deserialize, Debug, PartialEq)]
|
||||
pub enum Reply {
|
||||
/// Sent once at startup after EGL + CUDA came up.
|
||||
Ready {
|
||||
version: u32,
|
||||
},
|
||||
/// Startup failed (no NVIDIA driver, EGL error, …) — the host falls back to the CPU path,
|
||||
/// exactly like an in-process `EglImporter::new()` failure.
|
||||
InitErr {
|
||||
message: String,
|
||||
},
|
||||
Modifiers {
|
||||
modifiers: Vec<u64>,
|
||||
},
|
||||
/// The imported frame is complete (the GPU copy already synced worker-side) in buffer `id`.
|
||||
/// `desc` rides along the first time `id` is ever delivered — the host opens its CUDA IPC
|
||||
/// handles once and caches the mapping for every later frame in the same buffer.
|
||||
Frame {
|
||||
id: u32,
|
||||
desc: Option<BufferDesc>,
|
||||
},
|
||||
/// The worker has no cached fd for the import's `key` (evicted, or the two sides' caches
|
||||
/// diverged) — the host forgets its "already sent" note and retries once WITH the fd.
|
||||
NeedFd,
|
||||
/// This import failed but the worker is alive (e.g. `EGL_BAD_MATCH` on one buffer).
|
||||
Err {
|
||||
message: String,
|
||||
},
|
||||
}
|
||||
|
||||
/// CUDA IPC identity of one pooled device buffer (sent once per buffer, then referenced by id).
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
|
||||
pub struct BufferDesc {
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
/// `cuIpcGetMemHandle` blob for the (Y or BGRx) plane — exactly 64 bytes.
|
||||
pub y_handle: Vec<u8>,
|
||||
pub y_pitch: usize,
|
||||
/// NV12 only: the interleaved chroma plane's `(handle, pitch)`.
|
||||
pub uv: Option<(Vec<u8>, usize)>,
|
||||
}
|
||||
|
||||
/// A CLOEXEC `SOCK_SEQPACKET` socketpair — `(host_end, worker_end)`.
|
||||
pub fn socketpair_seqpacket() -> io::Result<(OwnedFd, OwnedFd)> {
|
||||
let mut fds = [0i32; 2];
|
||||
// SAFETY: `socketpair` writes two fds into `fds`, a live 2-element stack array matching the
|
||||
// API contract; it reads no other Rust memory. The result is checked before the fds are used,
|
||||
// and each returned fd is fresh (owned by no other wrapper), so the two `OwnedFd::from_raw_fd`
|
||||
// each take sole ownership of a distinct, valid descriptor — no alias, no double-close.
|
||||
unsafe {
|
||||
if libc::socketpair(
|
||||
libc::AF_UNIX,
|
||||
libc::SOCK_SEQPACKET | libc::SOCK_CLOEXEC,
|
||||
0,
|
||||
fds.as_mut_ptr(),
|
||||
) != 0
|
||||
{
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
Ok((OwnedFd::from_raw_fd(fds[0]), OwnedFd::from_raw_fd(fds[1])))
|
||||
}
|
||||
}
|
||||
|
||||
/// Set (or clear) the receive timeout: a blocked [`recv`] then fails with
|
||||
/// `ErrorKind::WouldBlock`. Used by the host so a hung worker can't wedge the capture thread.
|
||||
pub fn set_recv_timeout(sock: BorrowedFd, timeout: Option<Duration>) -> io::Result<()> {
|
||||
let tv = match timeout {
|
||||
Some(d) => libc::timeval {
|
||||
tv_sec: d.as_secs() as libc::time_t,
|
||||
tv_usec: d.subsec_micros() as libc::suseconds_t,
|
||||
},
|
||||
None => libc::timeval {
|
||||
tv_sec: 0,
|
||||
tv_usec: 0,
|
||||
},
|
||||
};
|
||||
// SAFETY: `setsockopt(SO_RCVTIMEO)` reads `size_of::<timeval>()` bytes from `&tv`, a live
|
||||
// stack `timeval` that outlives this synchronous call; `sock` is the caller's live socket fd.
|
||||
// Nothing is retained or written through Rust pointers.
|
||||
let r = unsafe {
|
||||
libc::setsockopt(
|
||||
sock.as_raw_fd(),
|
||||
libc::SOL_SOCKET,
|
||||
libc::SO_RCVTIMEO,
|
||||
&tv as *const libc::timeval as *const libc::c_void,
|
||||
std::mem::size_of::<libc::timeval>() as libc::socklen_t,
|
||||
)
|
||||
};
|
||||
if r != 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Send one message (+ optionally one fd as `SCM_RIGHTS`) as a single SEQPACKET datagram.
|
||||
/// Atomic per message, so concurrent senders on the same socket (the capture thread's imports,
|
||||
/// the encode thread's releases) need no lock. `MSG_NOSIGNAL` turns a dead peer into `EPIPE`
|
||||
/// instead of `SIGPIPE`.
|
||||
pub fn send<T: Serialize>(
|
||||
sock: BorrowedFd,
|
||||
msg: &T,
|
||||
pass_fd: Option<BorrowedFd>,
|
||||
) -> io::Result<()> {
|
||||
let body =
|
||||
serde_json::to_vec(msg).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
|
||||
debug_assert!(
|
||||
!body.is_empty(),
|
||||
"zero-length messages are reserved for EOF"
|
||||
);
|
||||
if body.len() > MAX_MSG {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"zerocopy proto message too large",
|
||||
));
|
||||
}
|
||||
let mut iov = libc::iovec {
|
||||
iov_base: body.as_ptr() as *mut libc::c_void,
|
||||
iov_len: body.len(),
|
||||
};
|
||||
// Control buffer for one fd: CMSG_SPACE(4) = 24 bytes on 64-bit; [u64; 4] gives 32 bytes at
|
||||
// the 8-byte alignment `cmsghdr` requires.
|
||||
let mut cmsg_store = [0u64; 4];
|
||||
// SAFETY: `mhdr` is a plain-old-data C struct for which all-zero is a valid value.
|
||||
let mut mhdr: libc::msghdr = unsafe { std::mem::zeroed() };
|
||||
mhdr.msg_iov = &mut iov;
|
||||
mhdr.msg_iovlen = 1;
|
||||
if let Some(fd) = pass_fd {
|
||||
mhdr.msg_control = cmsg_store.as_mut_ptr() as *mut libc::c_void;
|
||||
// SAFETY: `CMSG_SPACE`/`CMSG_LEN` are pure size computations (no memory access).
|
||||
// `CMSG_FIRSTHDR(&mhdr)` returns a pointer into `cmsg_store` (non-null: msg_controllen
|
||||
// ≥ one cmsghdr), which is live, 8-aligned, and large enough (32 ≥ CMSG_SPACE(4) = 24)
|
||||
// for the header fields and the 4-byte fd written via `CMSG_DATA`; `write_unaligned`
|
||||
// handles the data area's byte alignment. All writes stay within `cmsg_store`, which
|
||||
// outlives the synchronous `sendmsg` below.
|
||||
unsafe {
|
||||
mhdr.msg_controllen = libc::CMSG_SPACE(4) as _;
|
||||
let c = libc::CMSG_FIRSTHDR(&mhdr);
|
||||
(*c).cmsg_level = libc::SOL_SOCKET;
|
||||
(*c).cmsg_type = libc::SCM_RIGHTS;
|
||||
(*c).cmsg_len = libc::CMSG_LEN(4) as _;
|
||||
std::ptr::write_unaligned(libc::CMSG_DATA(c) as *mut i32, fd.as_raw_fd());
|
||||
}
|
||||
}
|
||||
// SAFETY: `sock` is the caller's live socket; `mhdr` points at the live `iov` (over `body`,
|
||||
// which outlives the call) and — when an fd is passed — at `cmsg_store` (ditto). `sendmsg`
|
||||
// only reads these buffers. The kernel dups the fd into the message; our `BorrowedFd` stays
|
||||
// owned by the caller.
|
||||
let n = unsafe { libc::sendmsg(sock.as_raw_fd(), &mhdr, libc::MSG_NOSIGNAL) };
|
||||
if n < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if n as usize != body.len() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::WriteZero,
|
||||
"short sendmsg on SEQPACKET socket",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Receive one message (+ up to one `SCM_RIGHTS` fd). `buf` is a caller-owned scratch buffer
|
||||
/// (grown to [`MAX_MSG`] once, then reused frame to frame). Errors:
|
||||
/// `UnexpectedEof` = the peer is gone; `WouldBlock` = the [`set_recv_timeout`] expired.
|
||||
pub fn recv<T: DeserializeOwned>(
|
||||
sock: BorrowedFd,
|
||||
buf: &mut Vec<u8>,
|
||||
) -> io::Result<(T, Option<OwnedFd>)> {
|
||||
buf.resize(MAX_MSG, 0);
|
||||
let mut iov = libc::iovec {
|
||||
iov_base: buf.as_mut_ptr() as *mut libc::c_void,
|
||||
iov_len: buf.len(),
|
||||
};
|
||||
let mut cmsg_store = [0u64; 4];
|
||||
// SAFETY: `mhdr` is a plain-old-data C struct for which all-zero is a valid value.
|
||||
let mut mhdr: libc::msghdr = unsafe { std::mem::zeroed() };
|
||||
mhdr.msg_iov = &mut iov;
|
||||
mhdr.msg_iovlen = 1;
|
||||
mhdr.msg_control = cmsg_store.as_mut_ptr() as *mut libc::c_void;
|
||||
mhdr.msg_controllen = std::mem::size_of_val(&cmsg_store) as _;
|
||||
// SAFETY: `sock` is the caller's live socket. `recvmsg` writes at most `iov_len` bytes into
|
||||
// `buf` (live for the call) and at most `msg_controllen` control bytes into `cmsg_store`
|
||||
// (live, 8-aligned). `MSG_CMSG_CLOEXEC` makes any received fd CLOEXEC atomically.
|
||||
let n = unsafe { libc::recvmsg(sock.as_raw_fd(), &mut mhdr, libc::MSG_CMSG_CLOEXEC) };
|
||||
if n < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if n == 0 {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::UnexpectedEof,
|
||||
"zerocopy proto peer closed",
|
||||
));
|
||||
}
|
||||
// Collect a passed fd (if any) BEFORE any early return below, so it can't leak.
|
||||
let mut got_fd: Option<OwnedFd> = None;
|
||||
// SAFETY: `CMSG_FIRSTHDR`/`CMSG_NXTHDR` walk the control area the kernel just wrote inside
|
||||
// `cmsg_store` (bounded by the updated `mhdr.msg_controllen`), returning either null or a
|
||||
// pointer to a complete `cmsghdr` within it — each dereference reads kernel-initialized
|
||||
// fields in bounds. For an `SCM_RIGHTS` cmsg the data area holds whole `i32` fds; we read the
|
||||
// first via `read_unaligned`. The kernel gave us ownership of that fd (it is a fresh
|
||||
// descriptor in our table), so `OwnedFd::from_raw_fd` takes sole ownership — any previously
|
||||
// collected `got_fd` is dropped (closed) first, so nothing leaks even with multiple cmsgs.
|
||||
unsafe {
|
||||
let mut c = libc::CMSG_FIRSTHDR(&mhdr);
|
||||
while !c.is_null() {
|
||||
if (*c).cmsg_level == libc::SOL_SOCKET && (*c).cmsg_type == libc::SCM_RIGHTS {
|
||||
let fd = std::ptr::read_unaligned(libc::CMSG_DATA(c) as *const i32);
|
||||
if fd >= 0 {
|
||||
got_fd = Some(OwnedFd::from_raw_fd(fd));
|
||||
}
|
||||
}
|
||||
c = libc::CMSG_NXTHDR(&mhdr, c);
|
||||
}
|
||||
}
|
||||
if mhdr.msg_flags & libc::MSG_TRUNC != 0 {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"zerocopy proto message truncated",
|
||||
));
|
||||
}
|
||||
let msg = serde_json::from_slice(&buf[..n as usize])
|
||||
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
|
||||
Ok((msg, got_fd))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::{Read, Write};
|
||||
use std::os::fd::AsFd;
|
||||
|
||||
#[test]
|
||||
fn round_trip_no_fd() {
|
||||
let (a, b) = socketpair_seqpacket().unwrap();
|
||||
let mut buf = Vec::new();
|
||||
let req = Request::Import {
|
||||
key: 0xdead_beef_u64,
|
||||
kind: ImportKind::TiledNv12,
|
||||
width: 5120,
|
||||
height: 1440,
|
||||
fourcc: 0x3432_5258,
|
||||
modifier: Some(0x0300_0000_0000_1234),
|
||||
offset: 0,
|
||||
stride: 5120 * 4,
|
||||
has_fd: false,
|
||||
};
|
||||
send(a.as_fd(), &req, None).unwrap();
|
||||
let (got, fd) = recv::<Request>(b.as_fd(), &mut buf).unwrap();
|
||||
assert_eq!(got, req);
|
||||
assert!(fd.is_none());
|
||||
|
||||
let reply = Reply::Frame {
|
||||
id: 7,
|
||||
desc: Some(BufferDesc {
|
||||
width: 5120,
|
||||
height: 1440,
|
||||
y_handle: vec![1u8; 64],
|
||||
y_pitch: 5632,
|
||||
uv: Some((vec![2u8; 64], 5632)),
|
||||
}),
|
||||
};
|
||||
send(b.as_fd(), &reply, None).unwrap();
|
||||
let (got, fd) = recv::<Reply>(a.as_fd(), &mut buf).unwrap();
|
||||
assert_eq!(got, reply);
|
||||
assert!(fd.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn passes_an_fd() {
|
||||
let (a, b) = socketpair_seqpacket().unwrap();
|
||||
let mut buf = Vec::new();
|
||||
// A pipe stands in for a dmabuf: pass the read end, write through the original write end,
|
||||
// and read the bytes back through the RECEIVED fd.
|
||||
let (mut pr, mut pw) = std::io::pipe().unwrap();
|
||||
send(a.as_fd(), &Request::ClearCache, Some(pr.as_fd())).unwrap();
|
||||
let (got, fd) = recv::<Request>(b.as_fd(), &mut buf).unwrap();
|
||||
assert_eq!(got, Request::ClearCache);
|
||||
let fd = fd.expect("fd should have been passed");
|
||||
pw.write_all(b"hello").unwrap();
|
||||
drop(pw);
|
||||
let mut file = std::fs::File::from(fd);
|
||||
let mut s = String::new();
|
||||
file.read_to_string(&mut s).unwrap();
|
||||
assert_eq!(s, "hello");
|
||||
// The original read end still works independently of the passed dup.
|
||||
let mut nothing = [0u8; 1];
|
||||
assert_eq!(pr.read(&mut nothing).unwrap(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn eof_when_peer_closes() {
|
||||
let (a, b) = socketpair_seqpacket().unwrap();
|
||||
drop(a);
|
||||
let mut buf = Vec::new();
|
||||
let err = recv::<Reply>(b.as_fd(), &mut buf).unwrap_err();
|
||||
assert_eq!(err.kind(), io::ErrorKind::UnexpectedEof);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn send_to_dead_peer_is_epipe_not_sigpipe() {
|
||||
let (a, b) = socketpair_seqpacket().unwrap();
|
||||
drop(b);
|
||||
let err = send(a.as_fd(), &Request::ClearCache, None).unwrap_err();
|
||||
// MSG_NOSIGNAL: a dead peer surfaces as EPIPE (BrokenPipe), never a process-killing signal.
|
||||
assert_eq!(err.kind(), io::ErrorKind::BrokenPipe);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_fires() {
|
||||
let (a, _b) = socketpair_seqpacket().unwrap();
|
||||
set_recv_timeout(a.as_fd(), Some(Duration::from_millis(50))).unwrap();
|
||||
let mut buf = Vec::new();
|
||||
let err = recv::<Reply>(a.as_fd(), &mut buf).unwrap_err();
|
||||
assert!(
|
||||
matches!(
|
||||
err.kind(),
|
||||
io::ErrorKind::WouldBlock | io::ErrorKind::TimedOut
|
||||
),
|
||||
"unexpected error kind: {err:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,423 @@
|
||||
//! Vulkan bridge for LINEAR dmabufs (gamescope's only offer), completing zero-copy where the
|
||||
//! other interops can't: NVIDIA's EGL won't sample LINEAR, and the CUDA driver rejects raw
|
||||
//! dmabuf fds as external memory. Vulkan *does* import dmabufs (`VK_EXT_external_memory_dma_buf`)
|
||||
//! and *does* export `OPAQUE_FD` memory that CUDA officially imports. So:
|
||||
//!
|
||||
//! ```text
|
||||
//! dmabuf fd ──VkImportMemoryFdInfoKHR(DMA_BUF)──▶ VkBuffer (cached per fd)
|
||||
//! │ vkCmdCopyBuffer (GPU, device-local)
|
||||
//! ▼
|
||||
//! exportable VkBuffer ──vkGetMemoryFdKHR(OPAQUE_FD)──▶ cuImportExternalMemory ──▶ CUdeviceptr
|
||||
//! ```
|
||||
//!
|
||||
//! The exportable buffer + its CUDA mapping are created once per resolution; per frame it's one
|
||||
//! GPU buffer copy (fence-waited) and one pitched CUDA copy into the encoder's pooled buffer.
|
||||
//! No CPU ever touches pixels. Imports are cached per fd (PipeWire's buffer pool is stable for
|
||||
//! a stream's life). Falls back cleanly: any init/import error disables the importer and the
|
||||
//! CPU mmap path takes over.
|
||||
|
||||
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use super::cuda::{self, DeviceBuffer};
|
||||
use anyhow::{anyhow, bail, Context as _, Result};
|
||||
use ash::vk;
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Vulkan objects for one imported source dmabuf (cached per fd).
|
||||
struct SrcBuf {
|
||||
buffer: vk::Buffer,
|
||||
memory: vk::DeviceMemory,
|
||||
size: u64,
|
||||
}
|
||||
|
||||
/// The per-resolution destination: exportable Vulkan memory mapped into CUDA.
|
||||
struct DstBuf {
|
||||
buffer: vk::Buffer,
|
||||
memory: vk::DeviceMemory,
|
||||
size: u64,
|
||||
/// CUDA's view of the same memory (owns the exported OPAQUE_FD).
|
||||
cuda: cuda::ExternalDmabuf,
|
||||
}
|
||||
|
||||
pub struct VkBridge {
|
||||
_entry: ash::Entry,
|
||||
instance: ash::Instance,
|
||||
device: ash::Device,
|
||||
ext_fd: ash::khr::external_memory_fd::Device,
|
||||
queue: vk::Queue,
|
||||
cmd_pool: vk::CommandPool,
|
||||
cmd: vk::CommandBuffer,
|
||||
fence: vk::Fence,
|
||||
mem_props: vk::PhysicalDeviceMemoryProperties,
|
||||
src_cache: HashMap<i32, SrcBuf>,
|
||||
dst: Option<DstBuf>,
|
||||
}
|
||||
|
||||
// SAFETY: `VkBridge` owns ash Vulkan handles (instance/device/queue/command pool+buffer/fence), a
|
||||
// CUDA external-memory mapping, and an fd→buffer cache — none `Sync`, and a single queue +
|
||||
// command buffer must be externally synchronized. It is created inside `EglImporter::import_linear`
|
||||
// on the dedicated `punktfunk-pipewire` capture thread and every method (`import_linear`, `Drop`)
|
||||
// runs on that thread; it is never shared via `&` across threads. `Send` asserts only that
|
||||
// transferring ownership is sound (so the bridge can live inside the `Send` `EglImporter`); the live
|
||||
// handles are never touched off-thread, and `Sync` is deliberately NOT implied.
|
||||
unsafe impl Send for VkBridge {}
|
||||
|
||||
impl VkBridge {
|
||||
/// Bring up Vulkan on the NVIDIA GPU with the external-memory extensions.
|
||||
pub fn new() -> Result<VkBridge> {
|
||||
// SAFETY: standard ash bring-up — every call is `unsafe` only because ash cannot statically
|
||||
// verify Vulkan handle/CreateInfo validity. `ash::Entry::load` dlopens a real system
|
||||
// libvulkan. Each `*CreateInfo`/`AllocateInfo` is built by ash's builders from locals (`app`,
|
||||
// `exts`, `prio`, `qci`, and the inline infos) that all live for the duration of the
|
||||
// synchronous `create_*`/`enumerate_*` call that reads them — in particular the
|
||||
// `enabled_extension_names(&exts)` and `queue_priorities(&prio)` borrows outlive their calls.
|
||||
// Every handle passed (`instance`, `phys`, `device`, `qf`, `cmd_pool`) was just created and
|
||||
// checked via `?`/`ok_or_else` in this same function, so no invalid handle is ever used. This
|
||||
// constructor shares nothing across threads.
|
||||
unsafe {
|
||||
let entry = ash::Entry::load().context("load libvulkan")?;
|
||||
let app = vk::ApplicationInfo::default().api_version(vk::API_VERSION_1_1);
|
||||
let instance = entry
|
||||
.create_instance(
|
||||
&vk::InstanceCreateInfo::default().application_info(&app),
|
||||
None,
|
||||
)
|
||||
.context("vkCreateInstance")?;
|
||||
|
||||
// Pick the NVIDIA GPU (matches CUDA device 0 on this single-dGPU host).
|
||||
let phys = instance
|
||||
.enumerate_physical_devices()
|
||||
.context("enumerate GPUs")?
|
||||
.into_iter()
|
||||
.find(|&p| instance.get_physical_device_properties(p).vendor_id == 0x10DE)
|
||||
.ok_or_else(|| anyhow!("no NVIDIA Vulkan device"))?;
|
||||
let mem_props = instance.get_physical_device_memory_properties(phys);
|
||||
|
||||
// Any queue family supporting transfer (graphics/compute imply it).
|
||||
let qf = instance
|
||||
.get_physical_device_queue_family_properties(phys)
|
||||
.iter()
|
||||
.position(|q| {
|
||||
q.queue_flags.intersects(
|
||||
vk::QueueFlags::TRANSFER
|
||||
| vk::QueueFlags::GRAPHICS
|
||||
| vk::QueueFlags::COMPUTE,
|
||||
)
|
||||
})
|
||||
.ok_or_else(|| anyhow!("no transfer-capable queue family"))?
|
||||
as u32;
|
||||
|
||||
let exts = [
|
||||
ash::khr::external_memory_fd::NAME.as_ptr(),
|
||||
ash::ext::external_memory_dma_buf::NAME.as_ptr(),
|
||||
];
|
||||
let prio = [1.0f32];
|
||||
let qci = [vk::DeviceQueueCreateInfo::default()
|
||||
.queue_family_index(qf)
|
||||
.queue_priorities(&prio)];
|
||||
let device = instance
|
||||
.create_device(
|
||||
phys,
|
||||
&vk::DeviceCreateInfo::default()
|
||||
.queue_create_infos(&qci)
|
||||
.enabled_extension_names(&exts),
|
||||
None,
|
||||
)
|
||||
.context("vkCreateDevice (external-memory extensions supported?)")?;
|
||||
let ext_fd = ash::khr::external_memory_fd::Device::new(&instance, &device);
|
||||
let queue = device.get_device_queue(qf, 0);
|
||||
|
||||
let cmd_pool = device
|
||||
.create_command_pool(
|
||||
&vk::CommandPoolCreateInfo::default()
|
||||
.queue_family_index(qf)
|
||||
.flags(vk::CommandPoolCreateFlags::RESET_COMMAND_BUFFER),
|
||||
None,
|
||||
)
|
||||
.context("create command pool")?;
|
||||
let cmd = device
|
||||
.allocate_command_buffers(
|
||||
&vk::CommandBufferAllocateInfo::default()
|
||||
.command_pool(cmd_pool)
|
||||
.level(vk::CommandBufferLevel::PRIMARY)
|
||||
.command_buffer_count(1),
|
||||
)
|
||||
.context("allocate command buffer")?[0];
|
||||
let fence = device
|
||||
.create_fence(&vk::FenceCreateInfo::default(), None)
|
||||
.context("create fence")?;
|
||||
|
||||
tracing::info!("Vulkan bridge ready (dmabuf import → OPAQUE_FD export → CUDA)");
|
||||
Ok(VkBridge {
|
||||
_entry: entry,
|
||||
instance,
|
||||
device,
|
||||
ext_fd,
|
||||
queue,
|
||||
cmd_pool,
|
||||
cmd,
|
||||
fence,
|
||||
mem_props,
|
||||
src_cache: HashMap::new(),
|
||||
dst: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn memory_type(&self, type_bits: u32, flags: vk::MemoryPropertyFlags) -> Result<u32> {
|
||||
(0..self.mem_props.memory_type_count)
|
||||
.find(|&i| {
|
||||
type_bits & (1 << i) != 0
|
||||
&& self.mem_props.memory_types[i as usize]
|
||||
.property_flags
|
||||
.contains(flags)
|
||||
})
|
||||
.ok_or_else(|| anyhow!("no compatible Vulkan memory type"))
|
||||
}
|
||||
|
||||
/// Import `fd` (dup'd internally; Vulkan owns the dup) as a transfer-src buffer of `size`.
|
||||
unsafe fn import_src(&mut self, fd: i32, size: u64) -> Result<()> {
|
||||
let dup = libc::dup(fd);
|
||||
if dup < 0 {
|
||||
bail!("dup(dmabuf fd)");
|
||||
}
|
||||
let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
|
||||
.handle_types(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT);
|
||||
let buffer = self
|
||||
.device
|
||||
.create_buffer(
|
||||
&vk::BufferCreateInfo::default()
|
||||
.size(size)
|
||||
.usage(vk::BufferUsageFlags::TRANSFER_SRC)
|
||||
.push_next(&mut ext_info),
|
||||
None,
|
||||
)
|
||||
.context("create import buffer")?;
|
||||
let mut fd_props = vk::MemoryFdPropertiesKHR::default();
|
||||
self.ext_fd
|
||||
.get_memory_fd_properties(
|
||||
vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT,
|
||||
dup,
|
||||
&mut fd_props,
|
||||
)
|
||||
.context("vkGetMemoryFdPropertiesKHR")?;
|
||||
let reqs = self.device.get_buffer_memory_requirements(buffer);
|
||||
let mem_type = self.memory_type(
|
||||
reqs.memory_type_bits & fd_props.memory_type_bits,
|
||||
vk::MemoryPropertyFlags::empty(),
|
||||
)?;
|
||||
let mut import = vk::ImportMemoryFdInfoKHR::default()
|
||||
.handle_type(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT)
|
||||
.fd(dup); // Vulkan takes ownership of `dup` on success
|
||||
let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer);
|
||||
let memory = self
|
||||
.device
|
||||
.allocate_memory(
|
||||
&vk::MemoryAllocateInfo::default()
|
||||
.allocation_size(reqs.size.max(size))
|
||||
.memory_type_index(mem_type)
|
||||
.push_next(&mut import)
|
||||
.push_next(&mut dedicated),
|
||||
None,
|
||||
)
|
||||
.map_err(|e| {
|
||||
libc::close(dup); // failed import does not consume the fd
|
||||
anyhow!("import dmabuf memory: {e}")
|
||||
})?;
|
||||
self.device
|
||||
.bind_buffer_memory(buffer, memory, 0)
|
||||
.context("bind import memory")?;
|
||||
self.src_cache.insert(
|
||||
fd,
|
||||
SrcBuf {
|
||||
buffer,
|
||||
memory,
|
||||
size,
|
||||
},
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// (Re)create the exportable destination of at least `size` bytes + its CUDA mapping.
|
||||
unsafe fn ensure_dst(&mut self, size: u64) -> Result<()> {
|
||||
if self.dst.as_ref().is_some_and(|d| d.size >= size) {
|
||||
return Ok(());
|
||||
}
|
||||
if let Some(old) = self.dst.take() {
|
||||
self.device.destroy_buffer(old.buffer, None);
|
||||
self.device.free_memory(old.memory, None);
|
||||
// old.cuda drops its mapping with it
|
||||
}
|
||||
let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
|
||||
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
|
||||
let buffer = self
|
||||
.device
|
||||
.create_buffer(
|
||||
&vk::BufferCreateInfo::default()
|
||||
.size(size)
|
||||
.usage(vk::BufferUsageFlags::TRANSFER_DST)
|
||||
.push_next(&mut ext_info),
|
||||
None,
|
||||
)
|
||||
.context("create export buffer")?;
|
||||
let reqs = self.device.get_buffer_memory_requirements(buffer);
|
||||
let mem_type =
|
||||
self.memory_type(reqs.memory_type_bits, vk::MemoryPropertyFlags::DEVICE_LOCAL)?;
|
||||
let mut export = vk::ExportMemoryAllocateInfo::default()
|
||||
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
|
||||
let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer);
|
||||
let memory = self
|
||||
.device
|
||||
.allocate_memory(
|
||||
&vk::MemoryAllocateInfo::default()
|
||||
.allocation_size(reqs.size)
|
||||
.memory_type_index(mem_type)
|
||||
.push_next(&mut export)
|
||||
.push_next(&mut dedicated),
|
||||
None,
|
||||
)
|
||||
.context("allocate exportable memory")?;
|
||||
self.device
|
||||
.bind_buffer_memory(buffer, memory, 0)
|
||||
.context("bind export memory")?;
|
||||
let opaque_fd = self
|
||||
.ext_fd
|
||||
.get_memory_fd(
|
||||
&vk::MemoryGetFdInfoKHR::default()
|
||||
.memory(memory)
|
||||
.handle_type(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD),
|
||||
)
|
||||
.context("vkGetMemoryFdKHR")?;
|
||||
// CUDA imports (and on success owns) the exported fd. Size must match the allocation.
|
||||
let cuda = cuda::ExternalDmabuf::import_owned_fd(opaque_fd, reqs.size)
|
||||
.context("cuImportExternalMemory(OPAQUE_FD from Vulkan)")?;
|
||||
tracing::info!(size, "Vulkan→CUDA exportable staging buffer ready");
|
||||
self.dst = Some(DstBuf {
|
||||
buffer,
|
||||
memory,
|
||||
size: reqs.size,
|
||||
cuda,
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Drop the cached import for `fd` (the PipeWire buffer it wrapped is gone — pool recycle /
|
||||
/// renegotiation — or the caller is about to store a different dmabuf under the same slot).
|
||||
/// Without this the cache could serve a stale imported buffer for a reused fd number, or
|
||||
/// leak an entry per recycled pool buffer.
|
||||
pub fn forget_fd(&mut self, fd: i32) {
|
||||
if let Some(s) = self.src_cache.remove(&fd) {
|
||||
// SAFETY: `s.buffer`/`s.memory` were created by this bridge's `import_src` and are
|
||||
// exclusively owned by the removed cache entry, so each is destroyed exactly once.
|
||||
// No GPU work can still reference them: every `import_linear` fence-waits its copy to
|
||||
// completion before returning, and this runs on the same single owning thread.
|
||||
unsafe {
|
||||
self.device.destroy_buffer(s.buffer, None);
|
||||
self.device.free_memory(s.memory, None);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Bridge one LINEAR dmabuf frame into a pooled CUDA buffer: GPU copy dmabuf→exportable,
|
||||
/// then pitched CUDA copy exportable→`pool` buffer.
|
||||
pub fn import_linear(
|
||||
&mut self,
|
||||
fd: i32,
|
||||
offset: u32,
|
||||
stride: u32,
|
||||
height: u32,
|
||||
pool: &cuda::BufferPool,
|
||||
) -> Result<DeviceBuffer> {
|
||||
// SAFETY: `fd` is the live dmabuf fd handed in by the caller (borrowed; `import_src` dup's it
|
||||
// internally and Vulkan owns the dup). `libc::lseek` only queries the fd's size. The unsafe
|
||||
// `import_src`/`ensure_dst` are called with a valid fd and a checked size. The bounds are
|
||||
// proven: `import_src` asserts `size >= span` (so the cached `src_size >= span`),
|
||||
// `copy_size = src_size.min(span)`, and `ensure_dst(copy_size)` makes `dst` at least
|
||||
// `copy_size` — so the GPU `cmd_copy_buffer` of `copy_size` bytes reads/writes within both
|
||||
// buffers, and the later CUDA pitched copy reading `[offset, span)` from `dst.cuda.ptr` (=
|
||||
// `offset + stride*height = span <= copy_size`) stays inside the freshly-copied region. The
|
||||
// `*Info`/`region`/`cmds`/`submit` are locals that outlive the synchronous calls reading them.
|
||||
// `cmd`/`queue`/`fence` are this bridge's own handles, used on this single thread only. The
|
||||
// host-side `wait_for_fences` fully retires the Vulkan copy BEFORE CUDA reads the shared
|
||||
// memory, so there is no GPU write/read data race. `dst` is an `&self.dst` shared borrow that
|
||||
// does not alias the `&self.device` calls.
|
||||
unsafe {
|
||||
let span = offset as u64 + stride as u64 * height as u64;
|
||||
if !self.src_cache.contains_key(&fd) {
|
||||
let size = libc::lseek(fd, 0, libc::SEEK_END);
|
||||
anyhow::ensure!(size > 0, "lseek(dmabuf)");
|
||||
anyhow::ensure!(size as u64 >= span, "dmabuf smaller than frame span");
|
||||
self.import_src(fd, size as u64)?;
|
||||
}
|
||||
let (src_buffer, src_size) = {
|
||||
let s = &self.src_cache[&fd];
|
||||
(s.buffer, s.size)
|
||||
};
|
||||
let copy_size = src_size.min(span);
|
||||
self.ensure_dst(copy_size)?;
|
||||
let dst = self.dst.as_ref().unwrap();
|
||||
|
||||
// Record + submit the GPU copy, wait on the fence (GPU-GPU, sub-millisecond).
|
||||
self.device
|
||||
.begin_command_buffer(
|
||||
self.cmd,
|
||||
&vk::CommandBufferBeginInfo::default()
|
||||
.flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT),
|
||||
)
|
||||
.context("begin cmd")?;
|
||||
let region = vk::BufferCopy::default().size(copy_size);
|
||||
self.device
|
||||
.cmd_copy_buffer(self.cmd, src_buffer, dst.buffer, &[region]);
|
||||
self.device
|
||||
.end_command_buffer(self.cmd)
|
||||
.context("end cmd")?;
|
||||
let cmds = [self.cmd];
|
||||
let submit = vk::SubmitInfo::default().command_buffers(&cmds);
|
||||
self.device
|
||||
.queue_submit(self.queue, &[submit], self.fence)
|
||||
.context("queue submit")?;
|
||||
self.device
|
||||
.wait_for_fences(&[self.fence], true, 1_000_000_000)
|
||||
.context("fence wait")?;
|
||||
self.device
|
||||
.reset_fences(&[self.fence])
|
||||
.context("reset fence")?;
|
||||
|
||||
// De-stride from the CUDA view of the exportable memory into a pooled buffer.
|
||||
cuda::make_current()?;
|
||||
let out = pool.get()?;
|
||||
cuda::copy_pitched_to_buffer(dst.cuda.ptr + offset as u64, stride as usize, &out)?;
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for VkBridge {
|
||||
fn drop(&mut self) {
|
||||
// SAFETY: runs once when the bridge is dropped on its owning capture thread.
|
||||
// `device_wait_idle` first drains all in-flight GPU work, so no queued command still
|
||||
// references these objects. Every handle freed (the `src_cache` buffers+memories, the `dst`
|
||||
// buffer+memory, `fence`, `cmd_pool`, `device`, `instance`) was created by this `VkBridge`
|
||||
// and owned exclusively by it, so each `destroy_*`/`free_*` runs exactly once with no
|
||||
// double-free, in dependency order (child objects before `device`, `device` before
|
||||
// `instance`). `dst.cuda` is dropped after `free_memory`, which is safe because CUDA holds
|
||||
// its own dup'd OPAQUE_FD reference to the underlying allocation. No other thread touches
|
||||
// these handles.
|
||||
unsafe {
|
||||
let _ = self.device.device_wait_idle();
|
||||
for (_, s) in self.src_cache.drain() {
|
||||
self.device.destroy_buffer(s.buffer, None);
|
||||
self.device.free_memory(s.memory, None);
|
||||
}
|
||||
if let Some(d) = self.dst.take() {
|
||||
self.device.destroy_buffer(d.buffer, None);
|
||||
self.device.free_memory(d.memory, None);
|
||||
}
|
||||
self.device.destroy_fence(self.fence, None);
|
||||
self.device.destroy_command_pool(self.cmd_pool, None);
|
||||
self.device.destroy_device(None);
|
||||
self.instance.destroy_instance(None);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,480 @@
|
||||
//! The isolated zero-copy GPU-import worker (`punktfunk-host zerocopy-worker`; design:
|
||||
//! [`design/zerocopy-worker-isolation.md`]). It owns the fragile driver stack — the headless
|
||||
//! EGLDisplay + GL context, the CUDA context, and the Vulkan bridge — so that a driver fault on a
|
||||
//! producer-invalidated dmabuf (the `cuGraphicsMapResources` SIGSEGV the F44 Game→Desktop switch
|
||||
//! reproduced) kills THIS process, not the streaming host. The host observes the dead socket,
|
||||
//! fails the frame cleanly, and its existing capture-loss rebuild takes over.
|
||||
//!
|
||||
//! One worker serves one capture (spawned per `pipewire_thread`). It exits on socket EOF — which
|
||||
//! only happens after the capturer AND every in-flight frame on the host side are gone, so pooled
|
||||
//! device memory is never freed under a frame the host still reads.
|
||||
|
||||
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
use super::cuda::{self, CUdeviceptr, DeviceBuffer};
|
||||
use super::egl::{DmabufPlane, EglImporter};
|
||||
use super::proto::{self, BufferDesc, ImportKind, Reply, Request};
|
||||
use anyhow::{bail, Context, Result};
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::io;
|
||||
use std::os::fd::{AsFd, AsRawFd, FromRawFd, OwnedFd};
|
||||
|
||||
/// Cap on cached per-key dmabuf fds. PipeWire buffer pools are ≤ ~16 buffers; the cap only
|
||||
/// matters if a misbehaving producer churns buffers without a renegotiation.
|
||||
const FD_CACHE_CAP: usize = 64;
|
||||
|
||||
/// Entry point for the hidden `zerocopy-worker` subcommand. `args` are the subcommand's own
|
||||
/// arguments (`--fd N`, default 3 — the socket end the spawning host `dup2`'d in).
|
||||
pub fn run_from_args(args: &[String]) -> Result<()> {
|
||||
// The host execs this worker through its pinned exe fd (`client::self_exe`), so the kernel
|
||||
// derives our comm from the exec path's basename — a meaningless fd number. Rename so
|
||||
// `top`/`pkill` see the worker.
|
||||
// SAFETY: `PR_SET_NAME` copies at most 16 bytes from the given pointer; the C-string literal
|
||||
// is valid, NUL-terminated, and short enough. No pointer is retained past the call.
|
||||
unsafe {
|
||||
libc::prctl(libc::PR_SET_NAME, c"pf-zerocopy".as_ptr());
|
||||
}
|
||||
let fd: i32 = args
|
||||
.iter()
|
||||
.skip_while(|a| *a != "--fd")
|
||||
.nth(1)
|
||||
.map(|s| s.parse())
|
||||
.transpose()
|
||||
.context("parse --fd")?
|
||||
.unwrap_or(3);
|
||||
// SAFETY: the spawning host `dup2`'d its socketpair end onto exactly this fd number before
|
||||
// exec (the subcommand's contract) and nothing else in this fresh process owns it, so
|
||||
// `OwnedFd` takes sole ownership and closes it exactly once at exit.
|
||||
let sock = unsafe { OwnedFd::from_raw_fd(fd) };
|
||||
run(sock)
|
||||
}
|
||||
|
||||
/// Bring up the GPU stack, report readiness, and serve until the host goes away.
|
||||
fn run(sock: OwnedFd) -> Result<()> {
|
||||
let importer = match EglImporter::new() {
|
||||
Ok(i) => i,
|
||||
Err(e) => {
|
||||
// Init failure is an ANSWER, not a crash: the host falls back to the CPU path,
|
||||
// exactly like an in-process `EglImporter::new()` failure.
|
||||
let _ = proto::send(
|
||||
sock.as_fd(),
|
||||
&Reply::InitErr {
|
||||
message: format!("{e:#}"),
|
||||
},
|
||||
None,
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
proto::send(
|
||||
sock.as_fd(),
|
||||
&Reply::Ready {
|
||||
version: proto::PROTO_VERSION,
|
||||
},
|
||||
None,
|
||||
)
|
||||
.context("send Ready")?;
|
||||
tracing::info!(pid = std::process::id(), "zerocopy import worker ready");
|
||||
let mut backend = EglBackend::new(importer);
|
||||
serve(&sock, &mut backend)
|
||||
}
|
||||
|
||||
/// What [`serve`] needs from an import implementation — split out so the dispatch loop is
|
||||
/// unit-testable without a GPU.
|
||||
pub(crate) trait ImportBackend {
|
||||
fn modifiers(&mut self, fourcc: u32) -> Vec<u64>;
|
||||
/// Answers with [`Reply::Frame`] (buffer id + [`BufferDesc`] iff first delivery of that id),
|
||||
/// [`Reply::NeedFd`] (this side lacks the key's fd — host resends it once), or [`Reply::Err`].
|
||||
fn import(&mut self, req: &ImportReq, fd: Option<OwnedFd>) -> Reply;
|
||||
fn release(&mut self, id: u32);
|
||||
fn clear_cache(&mut self);
|
||||
}
|
||||
|
||||
/// The [`Request::Import`] fields, destructured for [`ImportBackend::import`].
|
||||
pub(crate) struct ImportReq {
|
||||
pub key: u64,
|
||||
pub kind: ImportKind,
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub fourcc: u32,
|
||||
pub modifier: Option<u64>,
|
||||
pub offset: u32,
|
||||
pub stride: u32,
|
||||
pub has_fd: bool,
|
||||
}
|
||||
|
||||
/// The request loop. Returns `Ok(())` on host EOF (normal end-of-life); any other socket error
|
||||
/// propagates (the process exits — the host treats it like a death, which it is).
|
||||
pub(crate) fn serve(sock: &OwnedFd, backend: &mut dyn ImportBackend) -> Result<()> {
|
||||
let mut buf = Vec::new();
|
||||
loop {
|
||||
let (req, fd) = match proto::recv::<Request>(sock.as_fd(), &mut buf) {
|
||||
Ok(v) => v,
|
||||
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => return Ok(()),
|
||||
Err(e) => return Err(e).context("worker recv"),
|
||||
};
|
||||
match req {
|
||||
Request::Modifiers { fourcc } => {
|
||||
let reply = Reply::Modifiers {
|
||||
modifiers: backend.modifiers(fourcc),
|
||||
};
|
||||
if send_or_eof(sock, &reply)? {
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
Request::Import {
|
||||
key,
|
||||
kind,
|
||||
width,
|
||||
height,
|
||||
fourcc,
|
||||
modifier,
|
||||
offset,
|
||||
stride,
|
||||
has_fd,
|
||||
} => {
|
||||
let req = ImportReq {
|
||||
key,
|
||||
kind,
|
||||
width,
|
||||
height,
|
||||
fourcc,
|
||||
modifier,
|
||||
offset,
|
||||
stride,
|
||||
has_fd,
|
||||
};
|
||||
let reply = backend.import(&req, fd);
|
||||
if send_or_eof(sock, &reply)? {
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
Request::Release { id } => backend.release(id),
|
||||
Request::ClearCache => backend.clear_cache(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Send a reply; `Ok(true)` means the host is gone (EPIPE) and the loop should end quietly.
|
||||
fn send_or_eof(sock: &OwnedFd, reply: &Reply) -> Result<bool> {
|
||||
match proto::send(sock.as_fd(), reply, None) {
|
||||
Ok(()) => Ok(false),
|
||||
Err(e) if e.kind() == io::ErrorKind::BrokenPipe => Ok(true),
|
||||
Err(e) => Err(e).context("worker send"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The real backend: the in-process [`EglImporter`] plus the cross-process bookkeeping —
|
||||
/// per-key dmabuf fds, in-flight frames (held until `Release`), and stable buffer ids.
|
||||
struct EglBackend {
|
||||
importer: EglImporter,
|
||||
/// The dmabuf fd for each host key (`st_ino`), kept because the tiled path re-imports the fd
|
||||
/// every frame (`eglCreateImage`) and the LINEAR path caches per fd inside the Vulkan bridge.
|
||||
fds: HashMap<u64, OwnedFd>,
|
||||
/// Insertion order of `fds` keys for the LRU cap.
|
||||
fd_lru: VecDeque<u64>,
|
||||
/// Frames delivered to the host and not yet released — holding the `DeviceBuffer` is what
|
||||
/// keeps its device memory alive (pool `Arc`s) while the host encodes from it.
|
||||
inflight: HashMap<u32, DeviceBuffer>,
|
||||
/// Buffer id per device allocation. Valid only within one pool generation: pools never free
|
||||
/// allocations while alive, so a device VA can't repeat until a size change replaces the pool
|
||||
/// — at which point [`Self::note_dims`] clears this map (ids themselves are never reused;
|
||||
/// `next_id` only counts up).
|
||||
ids: HashMap<CUdeviceptr, u32>,
|
||||
next_id: u32,
|
||||
/// The (kind, width, height) of the last import — a change means the importer replaced its
|
||||
/// pool, invalidating the VA→id map (see [`Self::ids`]).
|
||||
last_shape: Option<(ImportKind, u32, u32)>,
|
||||
}
|
||||
|
||||
impl EglBackend {
|
||||
fn new(importer: EglImporter) -> EglBackend {
|
||||
EglBackend {
|
||||
importer,
|
||||
fds: HashMap::new(),
|
||||
fd_lru: VecDeque::new(),
|
||||
inflight: HashMap::new(),
|
||||
ids: HashMap::new(),
|
||||
next_id: 0,
|
||||
last_shape: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Store (or replace) the cached fd for `key`, evicting beyond the cap. A replaced or
|
||||
/// evicted fd is first forgotten by the Vulkan bridge so its per-fd import can't go stale.
|
||||
fn store_fd(&mut self, key: u64, fd: OwnedFd) {
|
||||
if let Some(old) = self.fds.insert(key, fd) {
|
||||
self.importer.forget_linear_fd(old.as_raw_fd());
|
||||
self.fd_lru.retain(|k| *k != key);
|
||||
}
|
||||
self.fd_lru.push_back(key);
|
||||
while self.fds.len() > FD_CACHE_CAP {
|
||||
let Some(oldest) = self.fd_lru.pop_front() else {
|
||||
break;
|
||||
};
|
||||
if let Some(old) = self.fds.remove(&oldest) {
|
||||
self.importer.forget_linear_fd(old.as_raw_fd());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Clear the VA→id map when the importer is about to replace its per-size pool (see
|
||||
/// [`Self::ids`]).
|
||||
fn note_dims(&mut self, kind: ImportKind, width: u32, height: u32) {
|
||||
if self.last_shape != Some((kind, width, height)) {
|
||||
self.last_shape = Some((kind, width, height));
|
||||
self.ids.clear();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl ImportBackend for EglBackend {
|
||||
fn modifiers(&mut self, fourcc: u32) -> Vec<u64> {
|
||||
self.importer.supported_modifiers(fourcc)
|
||||
}
|
||||
|
||||
fn import(&mut self, req: &ImportReq, fd: Option<OwnedFd>) -> Reply {
|
||||
if let Some(fd) = fd {
|
||||
self.store_fd(req.key, fd);
|
||||
} else if req.has_fd {
|
||||
return Reply::Err {
|
||||
message: "Import said has_fd but no fd arrived".into(),
|
||||
};
|
||||
}
|
||||
let Some(raw) = self.fds.get(&req.key).map(|f| f.as_raw_fd()) else {
|
||||
// We no longer hold this buffer's fd (LRU eviction / cache desync) — ask the host to
|
||||
// resend it rather than failing the frame.
|
||||
return Reply::NeedFd;
|
||||
};
|
||||
match self.import_inner(req, raw) {
|
||||
Ok((id, desc)) => Reply::Frame { id, desc },
|
||||
Err(e) => Reply::Err {
|
||||
message: format!("{e:#}"),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn release(&mut self, id: u32) {
|
||||
if self.inflight.remove(&id).is_none() {
|
||||
tracing::warn!(id, "release for a frame not in flight (host/worker desync)");
|
||||
}
|
||||
}
|
||||
|
||||
fn clear_cache(&mut self) {
|
||||
for (_, fd) in self.fds.drain() {
|
||||
self.importer.forget_linear_fd(fd.as_raw_fd());
|
||||
}
|
||||
self.fd_lru.clear();
|
||||
self.importer.clear_linear_cache();
|
||||
}
|
||||
}
|
||||
|
||||
impl EglBackend {
|
||||
/// The fallible core of [`ImportBackend::import`], once the fd for `req.key` is resolved.
|
||||
fn import_inner(&mut self, req: &ImportReq, raw: i32) -> Result<(u32, Option<BufferDesc>)> {
|
||||
let plane = DmabufPlane {
|
||||
fd: raw,
|
||||
offset: req.offset,
|
||||
stride: req.stride,
|
||||
};
|
||||
self.note_dims(req.kind, req.width, req.height);
|
||||
let buf = match req.kind {
|
||||
ImportKind::Tiled => {
|
||||
self.importer
|
||||
.import(&plane, req.width, req.height, req.fourcc, req.modifier)?
|
||||
}
|
||||
ImportKind::TiledNv12 => self.importer.import_nv12(
|
||||
&plane,
|
||||
req.width,
|
||||
req.height,
|
||||
req.fourcc,
|
||||
req.modifier,
|
||||
)?,
|
||||
ImportKind::Tiled444 => self.importer.import_yuv444(
|
||||
&plane,
|
||||
req.width,
|
||||
req.height,
|
||||
req.fourcc,
|
||||
req.modifier,
|
||||
)?,
|
||||
ImportKind::Linear => self.importer.import_linear(&plane, req.width, req.height)?,
|
||||
};
|
||||
// Assign / look up the buffer's id and export its CUDA IPC identity on first delivery.
|
||||
cuda::make_current()?;
|
||||
let (id, desc) = match self.ids.get(&buf.ptr) {
|
||||
Some(&id) => (id, None),
|
||||
None => {
|
||||
let id = self.next_id;
|
||||
self.next_id = self.next_id.wrapping_add(1);
|
||||
let y_handle = cuda::ipc_export(buf.ptr)?.to_vec();
|
||||
let uv = match buf.uv {
|
||||
Some((uv_ptr, uv_pitch)) => {
|
||||
Some((cuda::ipc_export(uv_ptr)?.to_vec(), uv_pitch))
|
||||
}
|
||||
None => None,
|
||||
};
|
||||
self.ids.insert(buf.ptr, id);
|
||||
(
|
||||
id,
|
||||
Some(BufferDesc {
|
||||
width: buf.width,
|
||||
height: buf.height,
|
||||
y_handle,
|
||||
y_pitch: buf.pitch,
|
||||
uv,
|
||||
}),
|
||||
)
|
||||
}
|
||||
};
|
||||
if self.inflight.insert(id, buf).is_some() {
|
||||
// A pool never hands out a buffer that hasn't been recycled, so a duplicate id means
|
||||
// corrupted bookkeeping — fail the import rather than alias two frames.
|
||||
bail!("buffer id {id} already in flight");
|
||||
}
|
||||
Ok((id, desc))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::mpsc;
|
||||
|
||||
/// Records calls; import behavior is scripted per key.
|
||||
struct MockBackend {
|
||||
calls: mpsc::Sender<String>,
|
||||
next: u32,
|
||||
}
|
||||
|
||||
impl ImportBackend for MockBackend {
|
||||
fn modifiers(&mut self, fourcc: u32) -> Vec<u64> {
|
||||
let _ = self.calls.send(format!("modifiers:{fourcc}"));
|
||||
vec![7, 8, 9]
|
||||
}
|
||||
fn import(&mut self, req: &ImportReq, fd: Option<OwnedFd>) -> Reply {
|
||||
let _ = self.calls.send(format!(
|
||||
"import:key={} kind={:?} fd={}",
|
||||
req.key,
|
||||
req.kind,
|
||||
fd.is_some()
|
||||
));
|
||||
if req.key == 0xbad {
|
||||
return Reply::Err {
|
||||
message: "scripted failure".into(),
|
||||
};
|
||||
}
|
||||
if req.key == 0xfeed && !req.has_fd {
|
||||
return Reply::NeedFd;
|
||||
}
|
||||
let id = self.next;
|
||||
self.next += 1;
|
||||
let desc = (id == 0).then(|| BufferDesc {
|
||||
width: req.width,
|
||||
height: req.height,
|
||||
y_handle: vec![0u8; 64],
|
||||
y_pitch: 256,
|
||||
uv: None,
|
||||
});
|
||||
Reply::Frame { id, desc }
|
||||
}
|
||||
fn release(&mut self, id: u32) {
|
||||
let _ = self.calls.send(format!("release:{id}"));
|
||||
}
|
||||
fn clear_cache(&mut self) {
|
||||
let _ = self.calls.send("clear".into());
|
||||
}
|
||||
}
|
||||
|
||||
fn start_server() -> (
|
||||
OwnedFd,
|
||||
mpsc::Receiver<String>,
|
||||
std::thread::JoinHandle<Result<()>>,
|
||||
) {
|
||||
let (host, worker) = proto::socketpair_seqpacket().unwrap();
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let join = std::thread::spawn(move || {
|
||||
let mut backend = MockBackend { calls: tx, next: 0 };
|
||||
serve(&worker, &mut backend)
|
||||
});
|
||||
(host, rx, join)
|
||||
}
|
||||
|
||||
fn import_req(key: u64, has_fd: bool) -> Request {
|
||||
Request::Import {
|
||||
key,
|
||||
kind: ImportKind::Tiled,
|
||||
width: 64,
|
||||
height: 64,
|
||||
fourcc: 1,
|
||||
modifier: None,
|
||||
offset: 0,
|
||||
stride: 256,
|
||||
has_fd,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_and_eof() {
|
||||
let (host, rx, join) = start_server();
|
||||
let mut buf = Vec::new();
|
||||
|
||||
proto::send(host.as_fd(), &Request::Modifiers { fourcc: 42 }, None).unwrap();
|
||||
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
|
||||
assert_eq!(
|
||||
reply,
|
||||
Reply::Modifiers {
|
||||
modifiers: vec![7, 8, 9]
|
||||
}
|
||||
);
|
||||
|
||||
// First import delivers the desc; the second (same mock id sequence continues) doesn't.
|
||||
proto::send(host.as_fd(), &import_req(1, false), None).unwrap();
|
||||
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
|
||||
match reply {
|
||||
Reply::Frame {
|
||||
id: 0,
|
||||
desc: Some(_),
|
||||
} => {}
|
||||
other => panic!("unexpected reply {other:?}"),
|
||||
}
|
||||
proto::send(host.as_fd(), &import_req(1, false), None).unwrap();
|
||||
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
|
||||
assert_eq!(reply, Reply::Frame { id: 1, desc: None });
|
||||
|
||||
// A missing worker-side fd is a NeedFd reply (host resends), not a failure.
|
||||
proto::send(host.as_fd(), &import_req(0xfeed, false), None).unwrap();
|
||||
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
|
||||
assert_eq!(reply, Reply::NeedFd);
|
||||
|
||||
// A failed import is an Err reply, not a dead worker.
|
||||
proto::send(host.as_fd(), &import_req(0xbad, false), None).unwrap();
|
||||
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
|
||||
match reply {
|
||||
Reply::Err { message } => assert!(message.contains("scripted failure")),
|
||||
other => panic!("unexpected reply {other:?}"),
|
||||
}
|
||||
|
||||
// Fire-and-forget ops reach the backend without replies.
|
||||
proto::send(host.as_fd(), &Request::Release { id: 0 }, None).unwrap();
|
||||
proto::send(host.as_fd(), &Request::ClearCache, None).unwrap();
|
||||
|
||||
// Closing the host end terminates serve() cleanly.
|
||||
drop(host);
|
||||
join.join().unwrap().unwrap();
|
||||
|
||||
let calls: Vec<String> = rx.iter().collect();
|
||||
assert_eq!(
|
||||
calls,
|
||||
vec![
|
||||
"modifiers:42",
|
||||
"import:key=1 kind=Tiled fd=false",
|
||||
"import:key=1 kind=Tiled fd=false",
|
||||
"import:key=65261 kind=Tiled fd=false", // 0xfeed
|
||||
"import:key=2989 kind=Tiled fd=false", // 0xbad
|
||||
"release:0",
|
||||
"clear",
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
//! Linux GPU zero-copy plumbing (plan §9), extracted from the host's `linux/zerocopy/` module
|
||||
//! tree (plan §W6): the shared CUDA context + device buffers, the EGL/Vulkan dmabuf importers,
|
||||
//! the isolated import-worker subprocess (client/proto/worker), the zero-copy policy latches, and
|
||||
//! the dmabuf implicit-fence wait. Everything is Linux-only; on other targets this crate compiles
|
||||
//! to an empty lib so dependents can carry a plain (non-target-gated) dependency.
|
||||
//!
|
||||
//! The `PixelFormat → DRM FourCC` mapping (`drm_fourcc`) deliberately does NOT live here — it
|
||||
//! consumes the shared frame vocabulary, which sits ABOVE this crate (this crate provides the
|
||||
//! `DeviceBuffer` that vocabulary's `FramePayload::Cuda` owns).
|
||||
|
||||
// Unsafe-proof program: every `unsafe {}` / `unsafe impl` must carry a `// SAFETY:` proof. Each
|
||||
// file keeps its own `#![deny(...)]` too; this crate-root deny is the catch-all gate.
|
||||
#![deny(clippy::undocumented_unsafe_blocks)]
|
||||
|
||||
/// Wait for a dmabuf's implicit read-ready fence (`DMA_BUF_IOCTL_EXPORT_SYNC_FILE` + poll).
|
||||
#[cfg(target_os = "linux")]
|
||||
pub mod dmabuf_fence;
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
mod imp;
|
||||
#[cfg(target_os = "linux")]
|
||||
pub use imp::*;
|
||||
Reference in New Issue
Block a user