unom/punktfunk
6
0
Fork 0
Code Issues 2 Pull Requests 2 Actions 32 Packages Projects Releases 39 Wiki Activity

fix(zerocopy): unsafe fn bodies join the unsafe-proof program

Browse Source 
#![deny(clippy::undocumented_unsafe_blocks)] never inspected the body of an
unsafe fn — operations there are not "unsafe blocks" — so roughly 20
functions' worth of raw GL/CUDA/Vulkan driver calls sat outside the
invariant the crate advertises. Concretely, that blind spot is why the
constructor-leak and teardown-ordering shapes fixed earlier in this series
could ship without ever prompting a reviewer.

#![deny(unsafe_op_in_unsafe_fn)] now closes the gap: every unsafe fn body
is an explicit unsafe block carrying a SAFETY comment that names the
caller contract it relies on (the dlopen'd CUDA wrapper table gets a
uniform forward-to-live-table proof). Mechanical; no behavior change.

(V2 from design/pf-zerocopy-sweep-handoff.md.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
enricobuehler
2026-07-28 12:59:15 +02:00
co-authored by Claude Fable 5
parent c677732c60
commit fa083f50d3
8 changed files with 862 additions and 682 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/pf-zerocopy/src/dmabuf_fence.rs
+1 -1
View File
@@ -11,7 +11,7 @@
//! fd we can `poll()` — readable once the producer's writes complete. This makes zero-copy capture //! fd we can `poll()` — readable once the producer's writes complete. This makes zero-copy capture
//! race-free WITHOUT the producer doing anything, *iff* the driver actually attaches the fence. If it //! race-free WITHOUT the producer doing anything, *iff* the driver actually attaches the fence. If it
//! attaches none, the export yields an already-signaled sync_file (poll returns immediately) — no //! attaches none, the export yields an already-signaled sync_file (poll returns immediately) — no
//! wait, no harm, and [`WaitOutcome::NoFence`] tells us the driver doesn't fence (so zero-copy //! wait, no harm, and `WaitOutcome::NoFence` tells us the driver doesn't fence (so zero-copy
//! would still race). //! would still race).
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program). // Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
crates/pf-zerocopy/src/imp/cuda.rs
+42 -22
View File
@@ -249,9 +249,14 @@ fn copy_stream() -> CUstream {
/// (the source dmabuf is safe to recycle once this returns), but the wait is scoped to our own /// (the source dmabuf is safe to recycle once this returns), but the wait is scoped to our own
/// stream and the copy carries the high priority hint. /// stream and the copy carries the high priority hint.
unsafe fn copy_blocking(copy: &CUDA_MEMCPY2D, what: &str) -> Result<()> { unsafe fn copy_blocking(copy: &CUDA_MEMCPY2D, what: &str) -> Result<()> {
let stream = copy_stream(); // SAFETY: caller contract: the shared context is current and `copy` describes live, in-bounds
ck(cuMemcpy2DAsync_v2(copy, stream), what)?; // device/host memory (each caller carries that proof). Wrapper -> live table; `&copy` outlives
ck(cuStreamSynchronize(stream), "cuStreamSynchronize") // the synchronous call.
unsafe {
let stream = copy_stream();
ck(cuMemcpy2DAsync_v2(copy, stream), what)?;
ck(cuStreamSynchronize(stream), "cuStreamSynchronize")
}
} }
/// Issue `copy` on this thread's priority stream WITHOUT waiting — for stream-ordered consumers /// Issue `copy` on this thread's priority stream WITHOUT waiting — for stream-ordered consumers
@@ -259,7 +264,10 @@ unsafe fn copy_blocking(copy: &CUDA_MEMCPY2D, what: &str) -> Result<()> {
/// stream, not the CPU, orders completion, so the SOURCE must stay valid until the downstream /// stream, not the CPU, orders completion, so the SOURCE must stay valid until the downstream
/// stream work (the encode) has finished. /// stream work (the encode) has finished.
unsafe fn copy_async(copy: &CUDA_MEMCPY2D, what: &str) -> Result<()> { unsafe fn copy_async(copy: &CUDA_MEMCPY2D, what: &str) -> Result<()> {
ck(cuMemcpy2DAsync_v2(copy, copy_stream()), what) // SAFETY: caller contract: the shared context is current and `copy` describes live, in-bounds
// device/host memory that stays valid until the stream work completes (each caller carries that
// proof). Wrapper -> live table.
unsafe { ck(cuMemcpy2DAsync_v2(copy, copy_stream()), what) }
} }
/// Block until everything enqueued on THIS THREAD's copy stream completed — the shared tail of /// Block until everything enqueued on THIS THREAD's copy stream completed — the shared tail of
@@ -268,16 +276,23 @@ unsafe fn copy_async(copy: &CUDA_MEMCPY2D, what: &str) -> Result<()> {
/// game's GPU load each exposed wait eats scheduling latency). The shared context must be /// game's GPU load each exposed wait eats scheduling latency). The shared context must be
/// current. /// current.
unsafe fn sync_copy_stream() -> Result<()> { unsafe fn sync_copy_stream() -> Result<()> {
ck(cuStreamSynchronize(copy_stream()), "cuStreamSynchronize") // SAFETY: caller contract: the shared context is current. Wrapper -> live table; synchronizing
// the dedicated copy stream touches no Rust memory.
unsafe { ck(cuStreamSynchronize(copy_stream()), "cuStreamSynchronize") }
} }
/// `copy_blocking` when `sync`, else `copy_async` — the shared tail of the public `copy_*_to_device` /// `copy_blocking` when `sync`, else `copy_async` — the shared tail of the public `copy_*_to_device`
/// helpers, whose `sync: false` mode carries `copy_async`'s source-lifetime contract. /// helpers, whose `sync: false` mode carries `copy_async`'s source-lifetime contract.
unsafe fn copy_issue(copy: &CUDA_MEMCPY2D, what: &str, sync: bool) -> Result<()> { unsafe fn copy_issue(copy: &CUDA_MEMCPY2D, what: &str, sync: bool) -> Result<()> {
if sync { // SAFETY: caller contract: the shared context is current and `copy` describes live, in-bounds
copy_blocking(copy, what) // memory (each caller carries that proof). The stream handle is the once-created per-process
} else { // copy stream. Wrapper -> live table.
copy_async(copy, what) unsafe {
if sync {
copy_blocking(copy, what)
} else {
copy_async(copy, what)
}
} }
} }
@@ -785,19 +800,24 @@ impl RegisteredTexture {
/// The GL context and the shared CUDA context must both be current on this thread, and /// The GL context and the shared CUDA context must both be current on this thread, and
/// `texture` must be a valid `GL_TEXTURE_2D`. /// `texture` must be a valid `GL_TEXTURE_2D`.
pub unsafe fn register_gl(texture: u32) -> Result<RegisteredTexture> { pub unsafe fn register_gl(texture: u32) -> Result<RegisteredTexture> {
const GL_TEXTURE_2D: c_uint = 0x0DE1; // SAFETY: caller contract: the GL context owning `texture` and the shared CUDA context are
const CU_GRAPHICS_REGISTER_FLAGS_READ_ONLY: c_uint = 0x01; // current on this thread, and `texture` is a live, complete GL texture. The out-param is a
let mut resource: CUgraphicsResource = std::ptr::null_mut(); // live stack local; wrapper -> live table.
ck( unsafe {
cuGraphicsGLRegisterImage( const GL_TEXTURE_2D: c_uint = 0x0DE1;
&mut resource, const CU_GRAPHICS_REGISTER_FLAGS_READ_ONLY: c_uint = 0x01;
texture, let mut resource: CUgraphicsResource = std::ptr::null_mut();
GL_TEXTURE_2D, ck(
CU_GRAPHICS_REGISTER_FLAGS_READ_ONLY, cuGraphicsGLRegisterImage(
), &mut resource,
"cuGraphicsGLRegisterImage", texture,
)?; GL_TEXTURE_2D,
Ok(RegisteredTexture { resource }) CU_GRAPHICS_REGISTER_FLAGS_READ_ONLY,
),
"cuGraphicsGLRegisterImage",
)?;
Ok(RegisteredTexture { resource })
}
} }
/// Map the texture for this frame, copy its (already-linear RGBA8) array into `dst`, then /// Map the texture for this frame, copy its (already-linear RGBA8) array into `dst`, then
crates/pf-zerocopy/src/imp/cuda/ffi.rs
+108 -26
View File
@@ -307,13 +307,19 @@ pub(crate) fn cuda_api() -> Option<&'static CudaApi> {
// present; every other entry runs after `context()` succeeded, so its wrapper always hits `Some`. // present; every other entry runs after `context()` succeeded, so its wrapper always hits `Some`.
pub(crate) unsafe fn cuInit(flags: c_uint) -> CUresult { pub(crate) unsafe fn cuInit(flags: c_uint) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuInit)(flags), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuInit)(flags) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuDeviceGet(device: *mut CUdevice, ordinal: c_int) -> CUresult { pub(crate) unsafe fn cuDeviceGet(device: *mut CUdevice, ordinal: c_int) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuDeviceGet)(device, ordinal), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuDeviceGet)(device, ordinal) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -323,19 +329,28 @@ pub(crate) unsafe fn cuCtxCreate_v2(
dev: CUdevice, dev: CUdevice,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuCtxCreate_v2)(pctx, flags, dev), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuCtxCreate_v2)(pctx, flags, dev) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuCtxDestroy_v2(ctx: CUcontext) -> CUresult { pub(crate) unsafe fn cuCtxDestroy_v2(ctx: CUcontext) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuCtxDestroy_v2)(ctx), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuCtxDestroy_v2)(ctx) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuCtxSetCurrent(ctx: CUcontext) -> CUresult { pub(crate) unsafe fn cuCtxSetCurrent(ctx: CUcontext) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuCtxSetCurrent)(ctx), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuCtxSetCurrent)(ctx) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -347,25 +362,39 @@ pub(crate) unsafe fn cuMemAllocPitch_v2(
element_size: c_uint, element_size: c_uint,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuMemAllocPitch_v2)(dptr, pitch, width_bytes, height, element_size), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe {
(a.cuMemAllocPitch_v2)(dptr, pitch, width_bytes, height, element_size)
},
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuMemFree_v2(dptr: CUdeviceptr) -> CUresult { pub(crate) unsafe fn cuMemFree_v2(dptr: CUdeviceptr) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuMemFree_v2)(dptr), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuMemFree_v2)(dptr) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuMemcpy2DAsync_v2(copy: *const CUDA_MEMCPY2D, stream: CUstream) -> CUresult { pub(crate) unsafe fn cuMemcpy2DAsync_v2(copy: *const CUDA_MEMCPY2D, stream: CUstream) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuMemcpy2DAsync_v2)(copy, stream), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuMemcpy2DAsync_v2)(copy, stream) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuStreamSynchronize(stream: CUstream) -> CUresult { pub(crate) unsafe fn cuStreamSynchronize(stream: CUstream) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuStreamSynchronize)(stream), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuStreamSynchronize)(stream) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -374,7 +403,10 @@ pub(crate) unsafe fn cuCtxGetStreamPriorityRange(
greatest: *mut c_int, greatest: *mut c_int,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuCtxGetStreamPriorityRange)(least, greatest), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuCtxGetStreamPriorityRange)(least, greatest) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -384,7 +416,10 @@ pub(crate) unsafe fn cuStreamCreateWithPriority(
priority: c_int, priority: c_int,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuStreamCreateWithPriority)(stream, flags, priority), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuStreamCreateWithPriority)(stream, flags, priority) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -395,7 +430,10 @@ pub(crate) unsafe fn cuGraphicsGLRegisterImage(
flags: c_uint, flags: c_uint,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuGraphicsGLRegisterImage)(resource, texture, target, flags), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuGraphicsGLRegisterImage)(resource, texture, target, flags) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -405,7 +443,10 @@ pub(crate) unsafe fn cuGraphicsMapResources(
stream: *mut c_void, stream: *mut c_void,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuGraphicsMapResources)(count, resources, stream), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuGraphicsMapResources)(count, resources, stream) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -415,7 +456,10 @@ pub(crate) unsafe fn cuGraphicsUnmapResources(
stream: *mut c_void, stream: *mut c_void,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuGraphicsUnmapResources)(count, resources, stream), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuGraphicsUnmapResources)(count, resources, stream) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -426,13 +470,21 @@ pub(crate) unsafe fn cuGraphicsSubResourceGetMappedArray(
mip_level: c_uint, mip_level: c_uint,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuGraphicsSubResourceGetMappedArray)(array, resource, array_index, mip_level), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe {
(a.cuGraphicsSubResourceGetMappedArray)(array, resource, array_index, mip_level)
},
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuGraphicsUnregisterResource(resource: CUgraphicsResource) -> CUresult { pub(crate) unsafe fn cuGraphicsUnregisterResource(resource: CUgraphicsResource) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuGraphicsUnregisterResource)(resource), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuGraphicsUnregisterResource)(resource) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -441,7 +493,10 @@ pub(crate) unsafe fn cuImportExternalMemory(
mem_handle_desc: *const CUDA_EXTERNAL_MEMORY_HANDLE_DESC, mem_handle_desc: *const CUDA_EXTERNAL_MEMORY_HANDLE_DESC,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuImportExternalMemory)(ext_mem_out, mem_handle_desc), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuImportExternalMemory)(ext_mem_out, mem_handle_desc) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -451,13 +506,19 @@ pub(crate) unsafe fn cuExternalMemoryGetMappedBuffer(
buffer_desc: *const CUDA_EXTERNAL_MEMORY_BUFFER_DESC, buffer_desc: *const CUDA_EXTERNAL_MEMORY_BUFFER_DESC,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuExternalMemoryGetMappedBuffer)(dev_ptr, ext_mem, buffer_desc), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuExternalMemoryGetMappedBuffer)(dev_ptr, ext_mem, buffer_desc) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuDestroyExternalMemory(ext_mem: CUexternalMemory) -> CUresult { pub(crate) unsafe fn cuDestroyExternalMemory(ext_mem: CUexternalMemory) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuDestroyExternalMemory)(ext_mem), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuDestroyExternalMemory)(ext_mem) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -466,13 +527,19 @@ pub(crate) unsafe fn cuImportExternalSemaphore(
sem_handle_desc: *const CUDA_EXTERNAL_SEMAPHORE_HANDLE_DESC, sem_handle_desc: *const CUDA_EXTERNAL_SEMAPHORE_HANDLE_DESC,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuImportExternalSemaphore)(ext_sem_out, sem_handle_desc), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuImportExternalSemaphore)(ext_sem_out, sem_handle_desc) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuDestroyExternalSemaphore(ext_sem: CUexternalSemaphore) -> CUresult { pub(crate) unsafe fn cuDestroyExternalSemaphore(ext_sem: CUexternalSemaphore) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuDestroyExternalSemaphore)(ext_sem), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuDestroyExternalSemaphore)(ext_sem) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -483,7 +550,10 @@ pub(crate) unsafe fn cuSignalExternalSemaphoresAsync(
stream: CUstream, stream: CUstream,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuSignalExternalSemaphoresAsync)(ext_sems, params, count, stream), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuSignalExternalSemaphoresAsync)(ext_sems, params, count, stream) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -494,13 +564,19 @@ pub(crate) unsafe fn cuWaitExternalSemaphoresAsync(
stream: CUstream, stream: CUstream,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuWaitExternalSemaphoresAsync)(ext_sems, params, count, stream), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuWaitExternalSemaphoresAsync)(ext_sems, params, count, stream) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuIpcGetMemHandle(handle: *mut CUipcMemHandle, dptr: CUdeviceptr) -> CUresult { pub(crate) unsafe fn cuIpcGetMemHandle(handle: *mut CUipcMemHandle, dptr: CUdeviceptr) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuIpcGetMemHandle)(handle, dptr), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuIpcGetMemHandle)(handle, dptr) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
@@ -510,13 +586,19 @@ pub(crate) unsafe fn cuIpcOpenMemHandle(
flags: c_uint, flags: c_uint,
) -> CUresult { ) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuIpcOpenMemHandle)(dptr, handle, flags), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuIpcOpenMemHandle)(dptr, handle, flags) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
pub(crate) unsafe fn cuIpcCloseMemHandle(dptr: CUdeviceptr) -> CUresult { pub(crate) unsafe fn cuIpcCloseMemHandle(dptr: CUdeviceptr) -> CUresult {
match cuda_api() { match cuda_api() {
Some(a) => (a.cuIpcCloseMemHandle)(dptr), // SAFETY: forwards this unsafe wrapper's arguments to the live dlopen'd entry point (the
// table is never unloaded — `mem::forget(lib)`); the caller upholds the driver-API contract,
// with the site-specific proof at each call site.
Some(a) => unsafe { (a.cuIpcCloseMemHandle)(dptr) },
None => CU_ERROR_NOT_LOADED, None => CU_ERROR_NOT_LOADED,
} }
} }
crates/pf-zerocopy/src/imp/egl.rs
+293 -261
View File
@@ -179,89 +179,98 @@ struct GlBlit {
impl GlBlit { impl GlBlit {
unsafe fn new(width: u32, height: u32) -> Result<GlBlit> { unsafe fn new(width: u32, height: u32) -> Result<GlBlit> {
// Declared before every GL name so it drops LAST on unwind — after the CUDA registration // SAFETY: caller contract (`import_inner`): the GL context and the shared CUDA context are
// (declared below) has unregistered itself. // current on this thread. Raw GL calls pass live locals whose pointers outlive each
let mut guard = GlNameGuard::default(); // synchronous call; every created name is owned by `guard` until the struct exists.
let program = compile_program()?; unsafe {
guard.programs.push(program); // Declared before every GL name so it drops LAST on unwind — after the CUDA registration
let mut vao = 0u32; // (declared below) has unregistered itself.
glGenVertexArrays(1, &mut vao); // core profile needs a bound VAO for glDrawArrays let mut guard = GlNameGuard::default();
guard.vaos.push(vao); let program = compile_program()?;
let mut fbo = 0u32; guard.programs.push(program);
glGenFramebuffers(1, &mut fbo); let mut vao = 0u32;
guard.fbos.push(fbo); glGenVertexArrays(1, &mut vao); // core profile needs a bound VAO for glDrawArrays
guard.vaos.push(vao);
let mut fbo = 0u32;
glGenFramebuffers(1, &mut fbo);
guard.fbos.push(fbo);
let mut dst_tex = 0u32; let mut dst_tex = 0u32;
glGenTextures(1, &mut dst_tex); glGenTextures(1, &mut dst_tex);
guard.textures.push(dst_tex); guard.textures.push(dst_tex);
glBindTexture(GL_TEXTURE_2D, dst_tex); glBindTexture(GL_TEXTURE_2D, dst_tex);
glTexStorage2D(GL_TEXTURE_2D, 1, GL_RGBA8, width as c_int, height as c_int); glTexStorage2D(GL_TEXTURE_2D, 1, GL_RGBA8, width as c_int, height as c_int);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
let mut src_tex = 0u32; let mut src_tex = 0u32;
glGenTextures(1, &mut src_tex); glGenTextures(1, &mut src_tex);
guard.textures.push(src_tex); guard.textures.push(src_tex);
glBindTexture(GL_TEXTURE_2D, src_tex); glBindTexture(GL_TEXTURE_2D, src_tex);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_2D, 0); glBindTexture(GL_TEXTURE_2D, 0);
glBindFramebuffer(GL_FRAMEBUFFER, fbo); glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glFramebufferTexture2D( glFramebufferTexture2D(
GL_FRAMEBUFFER, GL_FRAMEBUFFER,
GL_COLOR_ATTACHMENT0, GL_COLOR_ATTACHMENT0,
GL_TEXTURE_2D, GL_TEXTURE_2D,
dst_tex, dst_tex,
0, 0,
); );
let status = glCheckFramebufferStatus(GL_FRAMEBUFFER); let status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
glBindFramebuffer(GL_FRAMEBUFFER, 0); glBindFramebuffer(GL_FRAMEBUFFER, 0);
ensure!( ensure!(
status == GL_FRAMEBUFFER_COMPLETE, status == GL_FRAMEBUFFER_COMPLETE,
"blit FBO incomplete ({status:#x})" "blit FBO incomplete ({status:#x})"
); );
// Register the (immutable, reused) destination texture with CUDA once, and stand up the // Register the (immutable, reused) destination texture with CUDA once, and stand up the
// device-buffer pool — both per-resolution, not per-frame. Requires the CUDA context to be // device-buffer pool — both per-resolution, not per-frame. Requires the CUDA context to be
// current (the caller makes it current before constructing the blit). // current (the caller makes it current before constructing the blit).
let registered = cuda::RegisteredTexture::register_gl(dst_tex)?; let registered = cuda::RegisteredTexture::register_gl(dst_tex)?;
let pool = cuda::BufferPool::new(width, height)?; let pool = cuda::BufferPool::new(width, height)?;
guard.defuse(); guard.defuse();
Ok(GlBlit { Ok(GlBlit {
program, program,
vao, vao,
fbo, fbo,
dst_tex, dst_tex,
src_tex, src_tex,
width, width,
height, height,
registered, registered,
pool, pool,
}) })
}
} }
/// Bind `image` to the source texture and render it into `dst_tex`. /// Bind `image` to the source texture and render it into `dst_tex`.
/// ///
/// # Safety: the GL context is current on this thread; `image` is a valid `EGLImage`. /// # Safety: the GL context is current on this thread; `image` is a valid `EGLImage`.
unsafe fn run(&self, egl_image_target: EglImageTargetFn, image: *mut c_void) -> Result<()> { unsafe fn run(&self, egl_image_target: EglImageTargetFn, image: *mut c_void) -> Result<()> {
glBindTexture(GL_TEXTURE_2D, self.src_tex); // SAFETY: caller contract (`# Safety` above): GL context current, `image` a valid EGLImage.
let _ = glGetError(); // Raw GL calls pass names owned by `self`, created on this same context.
egl_image_target(GL_TEXTURE_2D, image); unsafe {
let e = glGetError(); glBindTexture(GL_TEXTURE_2D, self.src_tex);
glBindTexture(GL_TEXTURE_2D, 0); let _ = glGetError();
ensure!(e == 0, "glEGLImageTargetTexture2DOES failed ({e:#x})"); egl_image_target(GL_TEXTURE_2D, image);
let e = glGetError();
glBindTexture(GL_TEXTURE_2D, 0);
ensure!(e == 0, "glEGLImageTargetTexture2DOES failed ({e:#x})");
glBindFramebuffer(GL_FRAMEBUFFER, self.fbo); glBindFramebuffer(GL_FRAMEBUFFER, self.fbo);
glViewport(0, 0, self.width as c_int, self.height as c_int); glViewport(0, 0, self.width as c_int, self.height as c_int);
glUseProgram(self.program); glUseProgram(self.program);
glActiveTexture(GL_TEXTURE0); glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, self.src_tex); glBindTexture(GL_TEXTURE_2D, self.src_tex);
glBindVertexArray(self.vao); glBindVertexArray(self.vao);
glDrawArrays(GL_TRIANGLES, 0, 3); glDrawArrays(GL_TRIANGLES, 0, 3);
glBindVertexArray(0); glBindVertexArray(0);
glBindFramebuffer(GL_FRAMEBUFFER, 0); glBindFramebuffer(GL_FRAMEBUFFER, 0);
glFlush(); // submit GL work before CUDA maps the texture glFlush(); // submit GL work before CUDA maps the texture
Ok(()) Ok(())
}
} }
} }
@@ -319,102 +328,111 @@ struct Nv12Blit {
impl Nv12Blit { impl Nv12Blit {
unsafe fn new(width: u32, height: u32) -> Result<Nv12Blit> { unsafe fn new(width: u32, height: u32) -> Result<Nv12Blit> {
ensure!( // SAFETY: caller contract (`import_inner`): the GL context and the shared CUDA context are
width % 2 == 0 && height % 2 == 0, // current on this thread. Raw GL calls pass live locals whose pointers outlive each
"NV12 convert needs even dimensions (got {width}x{height})" // synchronous call; every created name is owned by `guard` until the struct exists.
); unsafe {
// Declared before every GL name so it drops LAST on unwind — after the CUDA registrations
// (declared below) have unregistered themselves.
let mut guard = GlNameGuard::default();
let y_program = compile_program_with(FRAG_Y_SRC)?;
guard.programs.push(y_program);
let uv_program = compile_program_with(FRAG_UV_SRC)?;
guard.programs.push(uv_program);
let mut vao = 0u32;
glGenVertexArrays(1, &mut vao);
guard.vaos.push(vao);
let mut fbos = [0u32; 2];
glGenFramebuffers(2, fbos.as_mut_ptr());
guard.fbos.extend_from_slice(&fbos);
let (y_fbo, uv_fbo) = (fbos[0], fbos[1]);
// Luma target: GL_R8 at full resolution.
let mut y_tex = 0u32;
glGenTextures(1, &mut y_tex);
guard.textures.push(y_tex);
glBindTexture(GL_TEXTURE_2D, y_tex);
glTexStorage2D(GL_TEXTURE_2D, 1, GL_R8, width as c_int, height as c_int);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
// Chroma target: GL_RG8 at half resolution (R=U, G=V).
let mut uv_tex = 0u32;
glGenTextures(1, &mut uv_tex);
guard.textures.push(uv_tex);
glBindTexture(GL_TEXTURE_2D, uv_tex);
glTexStorage2D(
GL_TEXTURE_2D,
1,
GL_RG8,
(width / 2) as c_int,
(height / 2) as c_int,
);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
// Source: GL_LINEAR so the half-res UV pass averages the 2×2 chroma footprint.
let mut src_tex = 0u32;
glGenTextures(1, &mut src_tex);
guard.textures.push(src_tex);
glBindTexture(GL_TEXTURE_2D, src_tex);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_2D, 0);
for (fbo, tex) in [(y_fbo, y_tex), (uv_fbo, uv_tex)] {
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, tex, 0);
let status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
ensure!( ensure!(
status == GL_FRAMEBUFFER_COMPLETE, width % 2 == 0 && height % 2 == 0,
"NV12 blit FBO incomplete ({status:#x}) — GL_R8/GL_RG8 not renderable?" "NV12 convert needs even dimensions (got {width}x{height})"
); );
// Declared before every GL name so it drops LAST on unwind — after the CUDA registrations
// (declared below) have unregistered themselves.
let mut guard = GlNameGuard::default();
let y_program = compile_program_with(FRAG_Y_SRC)?;
guard.programs.push(y_program);
let uv_program = compile_program_with(FRAG_UV_SRC)?;
guard.programs.push(uv_program);
let mut vao = 0u32;
glGenVertexArrays(1, &mut vao);
guard.vaos.push(vao);
let mut fbos = [0u32; 2];
glGenFramebuffers(2, fbos.as_mut_ptr());
guard.fbos.extend_from_slice(&fbos);
let (y_fbo, uv_fbo) = (fbos[0], fbos[1]);
// Luma target: GL_R8 at full resolution.
let mut y_tex = 0u32;
glGenTextures(1, &mut y_tex);
guard.textures.push(y_tex);
glBindTexture(GL_TEXTURE_2D, y_tex);
glTexStorage2D(GL_TEXTURE_2D, 1, GL_R8, width as c_int, height as c_int);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
// Chroma target: GL_RG8 at half resolution (R=U, G=V).
let mut uv_tex = 0u32;
glGenTextures(1, &mut uv_tex);
guard.textures.push(uv_tex);
glBindTexture(GL_TEXTURE_2D, uv_tex);
glTexStorage2D(
GL_TEXTURE_2D,
1,
GL_RG8,
(width / 2) as c_int,
(height / 2) as c_int,
);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
// Source: GL_LINEAR so the half-res UV pass averages the 2×2 chroma footprint.
let mut src_tex = 0u32;
glGenTextures(1, &mut src_tex);
guard.textures.push(src_tex);
glBindTexture(GL_TEXTURE_2D, src_tex);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_2D, 0);
for (fbo, tex) in [(y_fbo, y_tex), (uv_fbo, uv_tex)] {
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, tex, 0);
let status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
ensure!(
status == GL_FRAMEBUFFER_COMPLETE,
"NV12 blit FBO incomplete ({status:#x}) — GL_R8/GL_RG8 not renderable?"
);
}
// Register both convert targets with CUDA once (per-resolution), + the NV12 two-plane pool.
let y_registered = cuda::RegisteredTexture::register_gl(y_tex)?;
let uv_registered = cuda::RegisteredTexture::register_gl(uv_tex)?;
let pool = cuda::BufferPool::new_nv12(width, height)?;
guard.defuse();
Ok(Nv12Blit {
y_program,
uv_program,
vao,
y_fbo,
uv_fbo,
y_tex,
uv_tex,
src_tex,
width,
height,
y_registered,
uv_registered,
pool,
test_src_storage: false,
})
} }
// Register both convert targets with CUDA once (per-resolution), + the NV12 two-plane pool.
let y_registered = cuda::RegisteredTexture::register_gl(y_tex)?;
let uv_registered = cuda::RegisteredTexture::register_gl(uv_tex)?;
let pool = cuda::BufferPool::new_nv12(width, height)?;
guard.defuse();
Ok(Nv12Blit {
y_program,
uv_program,
vao,
y_fbo,
uv_fbo,
y_tex,
uv_tex,
src_tex,
width,
height,
y_registered,
uv_registered,
pool,
test_src_storage: false,
})
} }
/// Bind `image` to the source texture and run both convert passes into `y_tex`/`uv_tex`. /// Bind `image` to the source texture and run both convert passes into `y_tex`/`uv_tex`.
/// ///
/// # Safety: the GL context is current on this thread; `image` is a valid `EGLImage`. /// # Safety: the GL context is current on this thread; `image` is a valid `EGLImage`.
unsafe fn run(&self, egl_image_target: EglImageTargetFn, image: *mut c_void) -> Result<()> { unsafe fn run(&self, egl_image_target: EglImageTargetFn, image: *mut c_void) -> Result<()> {
glBindTexture(GL_TEXTURE_2D, self.src_tex); // SAFETY: caller contract (`# Safety` above): GL context current, `image` a valid EGLImage.
let _ = glGetError(); // Raw GL calls pass names owned by `self`, created on this same context.
egl_image_target(GL_TEXTURE_2D, image); unsafe {
let e = glGetError(); glBindTexture(GL_TEXTURE_2D, self.src_tex);
glBindTexture(GL_TEXTURE_2D, 0); let _ = glGetError();
ensure!(e == 0, "glEGLImageTargetTexture2DOES failed ({e:#x})"); egl_image_target(GL_TEXTURE_2D, image);
self.run_passes() let e = glGetError();
glBindTexture(GL_TEXTURE_2D, 0);
ensure!(e == 0, "glEGLImageTargetTexture2DOES failed ({e:#x})");
self.run_passes()
}
} }
/// Run the two convert passes from whatever is currently in `src_tex` (caller populated it). /// Run the two convert passes from whatever is currently in `src_tex` (caller populated it).
@@ -422,25 +440,29 @@ impl Nv12Blit {
/// ///
/// # Safety: the GL context is current on this thread. /// # Safety: the GL context is current on this thread.
unsafe fn run_passes(&self) -> Result<()> { unsafe fn run_passes(&self) -> Result<()> {
glActiveTexture(GL_TEXTURE0); // SAFETY: caller contract (`# Safety` above): GL context current. Raw GL calls pass names
glBindVertexArray(self.vao); // owned by `self`, created on this same context.
// Y pass: full-res into the R8 target. unsafe {
glBindFramebuffer(GL_FRAMEBUFFER, self.y_fbo); glActiveTexture(GL_TEXTURE0);
glViewport(0, 0, self.width as c_int, self.height as c_int); glBindVertexArray(self.vao);
glUseProgram(self.y_program); // Y pass: full-res into the R8 target.
glBindTexture(GL_TEXTURE_2D, self.src_tex); glBindFramebuffer(GL_FRAMEBUFFER, self.y_fbo);
glDrawArrays(GL_TRIANGLES, 0, 3); glViewport(0, 0, self.width as c_int, self.height as c_int);
// UV pass: half-res into the RG8 target (GL_LINEAR averages the 2×2). glUseProgram(self.y_program);
glBindFramebuffer(GL_FRAMEBUFFER, self.uv_fbo); glBindTexture(GL_TEXTURE_2D, self.src_tex);
glViewport(0, 0, (self.width / 2) as c_int, (self.height / 2) as c_int); glDrawArrays(GL_TRIANGLES, 0, 3);
glUseProgram(self.uv_program); // UV pass: half-res into the RG8 target (GL_LINEAR averages the 2×2).
glBindTexture(GL_TEXTURE_2D, self.src_tex); glBindFramebuffer(GL_FRAMEBUFFER, self.uv_fbo);
glDrawArrays(GL_TRIANGLES, 0, 3); glViewport(0, 0, (self.width / 2) as c_int, (self.height / 2) as c_int);
glUseProgram(self.uv_program);
glBindTexture(GL_TEXTURE_2D, self.src_tex);
glDrawArrays(GL_TRIANGLES, 0, 3);
glBindVertexArray(0); glBindVertexArray(0);
glBindFramebuffer(GL_FRAMEBUFFER, 0); glBindFramebuffer(GL_FRAMEBUFFER, 0);
glFlush(); // submit GL work before CUDA maps the textures glFlush(); // submit GL work before CUDA maps the textures
Ok(()) Ok(())
}
} }
} }
@@ -498,100 +520,110 @@ struct Yuv444Blit {
impl Yuv444Blit { impl Yuv444Blit {
unsafe fn new(width: u32, height: u32) -> Result<Yuv444Blit> { unsafe fn new(width: u32, height: u32) -> Result<Yuv444Blit> {
ensure!( // SAFETY: caller contract (`import_inner`): the GL context and the shared CUDA context are
width % 2 == 0 && height % 2 == 0, // current on this thread. Raw GL calls pass live locals whose pointers outlive each
"YUV444 convert needs even dimensions (got {width}x{height})" // synchronous call; every created name is owned by `guard` until the struct exists.
); unsafe {
let full_range = std::env::var("PUNKTFUNK_444_FULLRANGE").is_ok_and(|v| v.trim() == "1");
let (y_src, u_src, v_src) = yuv444_frag_sources(full_range);
// Declared before every GL name so it drops LAST on unwind — after the CUDA registrations
// (declared below) have unregistered themselves.
let mut guard = GlNameGuard::default();
let mut programs = [0u32; 3];
for (p, src) in programs.iter_mut().zip([&y_src, &u_src, &v_src]) {
*p = compile_program_with(src)?;
guard.programs.push(*p);
}
let mut vao = 0u32;
glGenVertexArrays(1, &mut vao);
guard.vaos.push(vao);
let mut fbos = [0u32; 3];
glGenFramebuffers(3, fbos.as_mut_ptr());
guard.fbos.extend_from_slice(&fbos);
let mut texs = [0u32; 3];
glGenTextures(3, texs.as_mut_ptr());
guard.textures.extend_from_slice(&texs);
for &tex in &texs {
glBindTexture(GL_TEXTURE_2D, tex);
glTexStorage2D(GL_TEXTURE_2D, 1, GL_R8, width as c_int, height as c_int);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
}
// Source: LINEAR is exact at the 1:1 mapping every pass uses (texel centres), matching
// the Nv12Blit source setup.
let mut src_tex = 0u32;
glGenTextures(1, &mut src_tex);
guard.textures.push(src_tex);
glBindTexture(GL_TEXTURE_2D, src_tex);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_2D, 0);
for (&fbo, &tex) in fbos.iter().zip(&texs) {
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, tex, 0);
let status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
ensure!( ensure!(
status == GL_FRAMEBUFFER_COMPLETE, width % 2 == 0 && height % 2 == 0,
"YUV444 blit FBO incomplete ({status:#x}) — GL_R8 not renderable?" "YUV444 convert needs even dimensions (got {width}x{height})"
); );
let full_range =
std::env::var("PUNKTFUNK_444_FULLRANGE").is_ok_and(|v| v.trim() == "1");
let (y_src, u_src, v_src) = yuv444_frag_sources(full_range);
// Declared before every GL name so it drops LAST on unwind — after the CUDA registrations
// (declared below) have unregistered themselves.
let mut guard = GlNameGuard::default();
let mut programs = [0u32; 3];
for (p, src) in programs.iter_mut().zip([&y_src, &u_src, &v_src]) {
*p = compile_program_with(src)?;
guard.programs.push(*p);
}
let mut vao = 0u32;
glGenVertexArrays(1, &mut vao);
guard.vaos.push(vao);
let mut fbos = [0u32; 3];
glGenFramebuffers(3, fbos.as_mut_ptr());
guard.fbos.extend_from_slice(&fbos);
let mut texs = [0u32; 3];
glGenTextures(3, texs.as_mut_ptr());
guard.textures.extend_from_slice(&texs);
for &tex in &texs {
glBindTexture(GL_TEXTURE_2D, tex);
glTexStorage2D(GL_TEXTURE_2D, 1, GL_R8, width as c_int, height as c_int);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
}
// Source: LINEAR is exact at the 1:1 mapping every pass uses (texel centres), matching
// the Nv12Blit source setup.
let mut src_tex = 0u32;
glGenTextures(1, &mut src_tex);
guard.textures.push(src_tex);
glBindTexture(GL_TEXTURE_2D, src_tex);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_2D, 0);
for (&fbo, &tex) in fbos.iter().zip(&texs) {
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, tex, 0);
let status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
ensure!(
status == GL_FRAMEBUFFER_COMPLETE,
"YUV444 blit FBO incomplete ({status:#x}) — GL_R8 not renderable?"
);
}
let registered = [
cuda::RegisteredTexture::register_gl(texs[0])?,
cuda::RegisteredTexture::register_gl(texs[1])?,
cuda::RegisteredTexture::register_gl(texs[2])?,
];
let pool = cuda::BufferPool::new_yuv444(width, height)?;
guard.defuse();
if full_range {
tracing::info!("YUV444 zero-copy convert: FULL range (PUNKTFUNK_444_FULLRANGE=1)");
}
Ok(Yuv444Blit {
programs,
vao,
fbos,
texs,
src_tex,
width,
height,
registered,
pool,
})
} }
let registered = [
cuda::RegisteredTexture::register_gl(texs[0])?,
cuda::RegisteredTexture::register_gl(texs[1])?,
cuda::RegisteredTexture::register_gl(texs[2])?,
];
let pool = cuda::BufferPool::new_yuv444(width, height)?;
guard.defuse();
if full_range {
tracing::info!("YUV444 zero-copy convert: FULL range (PUNKTFUNK_444_FULLRANGE=1)");
}
Ok(Yuv444Blit {
programs,
vao,
fbos,
texs,
src_tex,
width,
height,
registered,
pool,
})
} }
/// Bind `image` to the source texture and run the three plane passes. /// Bind `image` to the source texture and run the three plane passes.
/// ///
/// # Safety: the GL context is current on this thread; `image` is a valid `EGLImage`. /// # Safety: the GL context is current on this thread; `image` is a valid `EGLImage`.
unsafe fn run(&self, egl_image_target: EglImageTargetFn, image: *mut c_void) -> Result<()> { unsafe fn run(&self, egl_image_target: EglImageTargetFn, image: *mut c_void) -> Result<()> {
glBindTexture(GL_TEXTURE_2D, self.src_tex); // SAFETY: caller contract (`# Safety` above): GL context current, `image` a valid EGLImage.
let _ = glGetError(); // Raw GL calls pass names owned by `self`, created on this same context.
egl_image_target(GL_TEXTURE_2D, image); unsafe {
let e = glGetError();
glBindTexture(GL_TEXTURE_2D, 0);
ensure!(e == 0, "glEGLImageTargetTexture2DOES failed ({e:#x})");
glActiveTexture(GL_TEXTURE0);
glBindVertexArray(self.vao);
for (&fbo, &program) in self.fbos.iter().zip(&self.programs) {
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glViewport(0, 0, self.width as c_int, self.height as c_int);
glUseProgram(program);
glBindTexture(GL_TEXTURE_2D, self.src_tex); glBindTexture(GL_TEXTURE_2D, self.src_tex);
glDrawArrays(GL_TRIANGLES, 0, 3); let _ = glGetError();
egl_image_target(GL_TEXTURE_2D, image);
let e = glGetError();
glBindTexture(GL_TEXTURE_2D, 0);
ensure!(e == 0, "glEGLImageTargetTexture2DOES failed ({e:#x})");
glActiveTexture(GL_TEXTURE0);
glBindVertexArray(self.vao);
for (&fbo, &program) in self.fbos.iter().zip(&self.programs) {
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glViewport(0, 0, self.width as c_int, self.height as c_int);
glUseProgram(program);
glBindTexture(GL_TEXTURE_2D, self.src_tex);
glDrawArrays(GL_TRIANGLES, 0, 3);
}
glBindVertexArray(0);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glFlush(); // submit GL work before CUDA maps the textures
Ok(())
} }
glBindVertexArray(0);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glFlush(); // submit GL work before CUDA maps the textures
Ok(())
} }
} }
crates/pf-zerocopy/src/imp/egl/gl.rs
+53 -42
View File
@@ -148,60 +148,71 @@ pub(crate) fn yuv444_frag_sources(full_range: bool) -> (Vec<u8>, Vec<u8>, Vec<u8
} }
pub(crate) unsafe fn compile_shader(kind: u32, src: &[u8]) -> Result<u32> { pub(crate) unsafe fn compile_shader(kind: u32, src: &[u8]) -> Result<u32> {
let sh = glCreateShader(kind); // SAFETY: caller contract: the GL context is current on this thread. `src` is a live slice; its
ensure!(sh != 0, "glCreateShader failed"); // pointer/length locals outlive the synchronous `glShaderSource`; the shader name is deleted on
let ptr = src.as_ptr() as *const i8; // the compile-failure path.
let len = src.len() as c_int; unsafe {
glShaderSource(sh, 1, &ptr, &len); let sh = glCreateShader(kind);
glCompileShader(sh); ensure!(sh != 0, "glCreateShader failed");
let mut ok: c_int = 0; let ptr = src.as_ptr() as *const i8;
glGetShaderiv(sh, GL_COMPILE_STATUS, &mut ok); let len = src.len() as c_int;
if ok == 0 { glShaderSource(sh, 1, &ptr, &len);
glDeleteShader(sh); glCompileShader(sh);
bail!("GL shader compile failed"); let mut ok: c_int = 0;
glGetShaderiv(sh, GL_COMPILE_STATUS, &mut ok);
if ok == 0 {
glDeleteShader(sh);
bail!("GL shader compile failed");
}
Ok(sh)
} }
Ok(sh)
} }
/// Compile+link the fullscreen-triangle program with fragment source `frag` and bind its `image` /// Compile+link the fullscreen-triangle program with fragment source `frag` and bind its `image`
/// sampler to texture unit 0. /// sampler to texture unit 0.
pub(crate) unsafe fn compile_program_with(frag: &[u8]) -> Result<u32> { pub(crate) unsafe fn compile_program_with(frag: &[u8]) -> Result<u32> {
// Callers retry per frame, so every error path below must delete what it created — a leak // SAFETY: caller contract: the GL context is current on this thread. All names are created here
// here is unbounded, not one-shot. // and deleted on every failure path; `c"image"` is a valid NUL-terminated literal.
let vs = compile_shader(GL_VERTEX_SHADER, VERT_SRC)?; unsafe {
let fs = match compile_shader(GL_FRAGMENT_SHADER, frag) { // Callers retry per frame, so every error path below must delete what it created — a leak
Ok(fs) => fs, // here is unbounded, not one-shot.
Err(e) => { let vs = compile_shader(GL_VERTEX_SHADER, VERT_SRC)?;
let fs = match compile_shader(GL_FRAGMENT_SHADER, frag) {
Ok(fs) => fs,
Err(e) => {
glDeleteShader(vs);
return Err(e);
}
};
let prog = glCreateProgram();
if prog == 0 {
glDeleteShader(vs); glDeleteShader(vs);
return Err(e); glDeleteShader(fs);
bail!("glCreateProgram failed");
} }
}; glAttachShader(prog, vs);
let prog = glCreateProgram(); glAttachShader(prog, fs);
if prog == 0 { glLinkProgram(prog);
glDeleteShader(vs); glDeleteShader(vs);
glDeleteShader(fs); glDeleteShader(fs);
bail!("glCreateProgram failed"); let mut ok: c_int = 0;
glGetProgramiv(prog, GL_LINK_STATUS, &mut ok);
if ok == 0 {
glDeleteProgram(prog);
bail!("GL program link failed");
}
glUseProgram(prog);
let loc = glGetUniformLocation(prog, c"image".as_ptr());
if loc >= 0 {
glUniform1i(loc, 0); // sampler -> texture unit 0
}
glUseProgram(0);
Ok(prog)
} }
glAttachShader(prog, vs);
glAttachShader(prog, fs);
glLinkProgram(prog);
glDeleteShader(vs);
glDeleteShader(fs);
let mut ok: c_int = 0;
glGetProgramiv(prog, GL_LINK_STATUS, &mut ok);
if ok == 0 {
glDeleteProgram(prog);
bail!("GL program link failed");
}
glUseProgram(prog);
let loc = glGetUniformLocation(prog, c"image".as_ptr());
if loc >= 0 {
glUniform1i(loc, 0); // sampler -> texture unit 0
}
glUseProgram(0);
Ok(prog)
} }
pub(crate) unsafe fn compile_program() -> Result<u32> { pub(crate) unsafe fn compile_program() -> Result<u32> {
compile_program_with(FRAG_SRC) // SAFETY: caller contract: the GL context is current on this thread (forwarded to
// `compile_program_with`).
unsafe { compile_program_with(FRAG_SRC) }
} }
crates/pf-zerocopy/src/imp/vkslot.rs
+74 -68
View File
@@ -797,74 +797,80 @@ impl VkSlotBlend {
gx: u32, gx: u32,
gy: u32, gy: u32,
) -> Result<vk::CommandBuffer> { ) -> Result<vk::CommandBuffer> {
let alloc = self // SAFETY: caller contract (`# Safety` above): the slot's previous submission completed, so
.slots // its command buffer is re-recordable. Handles are owned by `self` on this single thread;
.get(id) // `bytes` reborrows `push` (repr(C), size_of::<Push>()) for the synchronous push-constant
.ok_or_else(|| anyhow!("bad slot id {id}"))?; // copy; builder infos are locals outliving each call.
let d = &self.device; unsafe {
let cmd = alloc.cmd; let alloc = self
d.begin_command_buffer( .slots
cmd, .get(id)
&vk::CommandBufferBeginInfo::default() .ok_or_else(|| anyhow!("bad slot id {id}"))?;
.flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT), let d = &self.device;
) let cmd = alloc.cmd;
.context("begin blend cmd")?; d.begin_command_buffer(
// CUDA wrote the frame into this memory outside Vulkan's view — make it visible to cmd,
// the shader (external-memory coherence ceremony; NVIDIA honors this with the &vk::CommandBufferBeginInfo::default()
// fence/semaphore ordering alone, the barrier is the spec-shaped belt-and-braces). .flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT),
let acquire = [vk::MemoryBarrier::default() )
.src_access_mask(vk::AccessFlags::MEMORY_WRITE) .context("begin blend cmd")?;
.dst_access_mask(vk::AccessFlags::SHADER_READ | vk::AccessFlags::SHADER_WRITE)]; // CUDA wrote the frame into this memory outside Vulkan's view — make it visible to
d.cmd_pipeline_barrier( // the shader (external-memory coherence ceremony; NVIDIA honors this with the
cmd, // fence/semaphore ordering alone, the barrier is the spec-shaped belt-and-braces).
vk::PipelineStageFlags::TOP_OF_PIPE, let acquire = [vk::MemoryBarrier::default()
vk::PipelineStageFlags::COMPUTE_SHADER, .src_access_mask(vk::AccessFlags::MEMORY_WRITE)
vk::DependencyFlags::empty(), .dst_access_mask(vk::AccessFlags::SHADER_READ | vk::AccessFlags::SHADER_WRITE)];
&acquire, d.cmd_pipeline_barrier(
&[], cmd,
&[], vk::PipelineStageFlags::TOP_OF_PIPE,
); vk::PipelineStageFlags::COMPUTE_SHADER,
d.cmd_bind_pipeline( vk::DependencyFlags::empty(),
cmd, &acquire,
vk::PipelineBindPoint::COMPUTE, &[],
self.pipelines[fmt.mode() as usize], &[],
); );
d.cmd_bind_descriptor_sets( d.cmd_bind_pipeline(
cmd, cmd,
vk::PipelineBindPoint::COMPUTE, vk::PipelineBindPoint::COMPUTE,
self.pipe_layout, self.pipelines[fmt.mode() as usize],
0, );
&[alloc.desc], d.cmd_bind_descriptor_sets(
&[], cmd,
); vk::PipelineBindPoint::COMPUTE,
let bytes = std::slice::from_raw_parts( self.pipe_layout,
(push as *const Push) as *const u8, 0,
std::mem::size_of::<Push>(), &[alloc.desc],
); &[],
d.cmd_push_constants( );
cmd, let bytes = std::slice::from_raw_parts(
self.pipe_layout, (push as *const Push) as *const u8,
vk::ShaderStageFlags::COMPUTE, std::mem::size_of::<Push>(),
0, );
bytes, d.cmd_push_constants(
); cmd,
d.cmd_dispatch(cmd, gx.max(1), gy.max(1), 1); self.pipe_layout,
// Release the shader's writes so the downstream CUDA/NVENC reads (fence- or vk::ShaderStageFlags::COMPUTE,
// semaphore-ordered) see them. 0,
let release = [vk::MemoryBarrier::default() bytes,
.src_access_mask(vk::AccessFlags::SHADER_WRITE) );
.dst_access_mask(vk::AccessFlags::MEMORY_READ)]; d.cmd_dispatch(cmd, gx.max(1), gy.max(1), 1);
d.cmd_pipeline_barrier( // Release the shader's writes so the downstream CUDA/NVENC reads (fence- or
cmd, // semaphore-ordered) see them.
vk::PipelineStageFlags::COMPUTE_SHADER, let release = [vk::MemoryBarrier::default()
vk::PipelineStageFlags::BOTTOM_OF_PIPE, .src_access_mask(vk::AccessFlags::SHADER_WRITE)
vk::DependencyFlags::empty(), .dst_access_mask(vk::AccessFlags::MEMORY_READ)];
&release, d.cmd_pipeline_barrier(
&[], cmd,
&[], vk::PipelineStageFlags::COMPUTE_SHADER,
); vk::PipelineStageFlags::BOTTOM_OF_PIPE,
d.end_command_buffer(cmd).context("end blend cmd")?; vk::DependencyFlags::empty(),
Ok(cmd) &release,
&[],
&[],
);
d.end_command_buffer(cmd).context("end blend cmd")?;
Ok(cmd)
}
} }
/// Blend the uploaded cursor into `slot` at `(ox, oy)`: record, submit, fence-wait. The /// Blend the uploaded cursor into `slot` at `(ox, oy)`: record, submit, fence-wait. The
crates/pf-zerocopy/src/imp/vulkan.rs
+287 -262
View File
@@ -292,184 +292,197 @@ impl VkBridge {
/// Import `fd` (dup'd internally; Vulkan owns the dup) as a transfer-src buffer of `size`. /// Import `fd` (dup'd internally; Vulkan owns the dup) as a transfer-src buffer of `size`.
unsafe fn import_src(&mut self, fd: i32, size: u64) -> Result<()> { unsafe fn import_src(&mut self, fd: i32, size: u64) -> Result<()> {
use std::os::fd::{AsRawFd, FromRawFd, IntoRawFd, OwnedFd}; // SAFETY: caller contract: single-threaded use of handles this bridge owns; every builder
let dup = libc::dup(fd); // info is built from locals that outlive the synchronous call reading them, and every
if dup < 0 { // fallible step destroys what it created before returning (comments below).
bail!("dup(dmabuf fd)"); unsafe {
} use std::os::fd::{AsRawFd, FromRawFd, IntoRawFd, OwnedFd};
// Own the dup so every early return BEFORE Vulkan consumes it (at `allocate_memory` success) let dup = libc::dup(fd);
// closes it. `SrcBuf` holds raw handles with no Drop and is only populated on the success if dup < 0 {
// path, so each fallible step below must also destroy the buffer it created — otherwise a bail!("dup(dmabuf fd)");
// failed import (which the worker survives and the caller retries every frame) leaks a
// VkBuffer + VkDeviceMemory + fd per frame for the worker's whole lifetime.
let dup = OwnedFd::from_raw_fd(dup);
let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT);
let buffer = self
.device
.create_buffer(
&vk::BufferCreateInfo::default()
.size(size)
// STORAGE so the NV12 compute CSC can read it as an SSBO (T2.5b); harmless
// for the plain copy path.
.usage(
vk::BufferUsageFlags::TRANSFER_SRC | vk::BufferUsageFlags::STORAGE_BUFFER,
)
.push_next(&mut ext_info),
None,
)
.context("create import buffer")?; // `dup` drops → closes on failure
let mut fd_props = vk::MemoryFdPropertiesKHR::default();
if let Err(e) = self.ext_fd.get_memory_fd_properties(
vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT,
dup.as_raw_fd(),
&mut fd_props,
) {
self.device.destroy_buffer(buffer, None);
return Err(e).context("vkGetMemoryFdPropertiesKHR");
}
let reqs = self.device.get_buffer_memory_requirements(buffer);
let mem_type = match self.memory_type(
reqs.memory_type_bits & fd_props.memory_type_bits,
vk::MemoryPropertyFlags::empty(),
) {
Ok(t) => t,
Err(e) => {
self.device.destroy_buffer(buffer, None);
return Err(e);
} }
}; // Own the dup so every early return BEFORE Vulkan consumes it (at `allocate_memory` success)
// Vulkan takes ownership of the fd on a SUCCESSFUL import: hand over the raw fd now, and on // closes it. `SrcBuf` holds raw handles with no Drop and is only populated on the success
// failure close it ourselves (matching the original contract) plus destroy the buffer. // path, so each fallible step below must also destroy the buffer it created — otherwise a
let raw = dup.into_raw_fd(); // failed import (which the worker survives and the caller retries every frame) leaks a
let mut import = vk::ImportMemoryFdInfoKHR::default() // VkBuffer + VkDeviceMemory + fd per frame for the worker's whole lifetime.
.handle_type(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT) let dup = OwnedFd::from_raw_fd(dup);
.fd(raw); let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer); .handle_types(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT);
let memory = match self.device.allocate_memory( let buffer = self
&vk::MemoryAllocateInfo::default() .device
.allocation_size(reqs.size.max(size)) .create_buffer(
.memory_type_index(mem_type) &vk::BufferCreateInfo::default()
.push_next(&mut import) .size(size)
.push_next(&mut dedicated), // STORAGE so the NV12 compute CSC can read it as an SSBO (T2.5b); harmless
None, // for the plain copy path.
) { .usage(
Ok(m) => m, vk::BufferUsageFlags::TRANSFER_SRC
Err(e) => { | vk::BufferUsageFlags::STORAGE_BUFFER,
libc::close(raw); // failed import does not consume the fd )
.push_next(&mut ext_info),
None,
)
.context("create import buffer")?; // `dup` drops → closes on failure
let mut fd_props = vk::MemoryFdPropertiesKHR::default();
if let Err(e) = self.ext_fd.get_memory_fd_properties(
vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT,
dup.as_raw_fd(),
&mut fd_props,
) {
self.device.destroy_buffer(buffer, None); self.device.destroy_buffer(buffer, None);
return Err(anyhow!("import dmabuf memory: {e}")); return Err(e).context("vkGetMemoryFdPropertiesKHR");
} }
}; let reqs = self.device.get_buffer_memory_requirements(buffer);
if let Err(e) = self.device.bind_buffer_memory(buffer, memory, 0) { let mem_type = match self.memory_type(
// `memory` owns the imported fd — freeing it releases the fd too. reqs.memory_type_bits & fd_props.memory_type_bits,
self.device.free_memory(memory, None); vk::MemoryPropertyFlags::empty(),
self.device.destroy_buffer(buffer, None); ) {
return Err(e).context("bind import memory");
}
self.src_cache.insert(
fd,
SrcBuf {
buffer,
memory,
size,
},
);
Ok(())
}
/// (Re)create the exportable destination of at least `size` bytes + its CUDA mapping.
unsafe fn ensure_dst(&mut self, size: u64) -> Result<()> {
if self.dst.as_ref().is_some_and(|d| d.size >= size) {
return Ok(());
}
// Build the replacement FULLY before retiring the old one. Previously the old dst was
// destroyed and `self.dst` nulled up front, so a failed rebuild both dropped the working
// buffer AND leaked every object the partial rebuild created (`buffer`/`memory` are raw ash
// handles with no Drop, and `VkBridge::drop` only frees the live `self.dst`). Now every
// fallible step unwinds locally, and the swap happens only on full success.
let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
let buffer = self
.device
.create_buffer(
&vk::BufferCreateInfo::default()
.size(size)
// STORAGE so the NV12 compute CSC can write it as an SSBO (T2.5b).
.usage(
vk::BufferUsageFlags::TRANSFER_DST | vk::BufferUsageFlags::STORAGE_BUFFER,
)
.push_next(&mut ext_info),
None,
)
.context("create export buffer")?;
let reqs = self.device.get_buffer_memory_requirements(buffer);
let mem_type =
match self.memory_type(reqs.memory_type_bits, vk::MemoryPropertyFlags::DEVICE_LOCAL) {
Ok(t) => t, Ok(t) => t,
Err(e) => { Err(e) => {
self.device.destroy_buffer(buffer, None); self.device.destroy_buffer(buffer, None);
return Err(e); return Err(e);
} }
}; };
let mut export = vk::ExportMemoryAllocateInfo::default() // Vulkan takes ownership of the fd on a SUCCESSFUL import: hand over the raw fd now, and on
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD); // failure close it ourselves (matching the original contract) plus destroy the buffer.
let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer); let raw = dup.into_raw_fd();
let memory = match self.device.allocate_memory( let mut import = vk::ImportMemoryFdInfoKHR::default()
&vk::MemoryAllocateInfo::default() .handle_type(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT)
.allocation_size(reqs.size) .fd(raw);
.memory_type_index(mem_type) let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer);
.push_next(&mut export) let memory = match self.device.allocate_memory(
.push_next(&mut dedicated), &vk::MemoryAllocateInfo::default()
None, .allocation_size(reqs.size.max(size))
) { .memory_type_index(mem_type)
Ok(m) => m, .push_next(&mut import)
Err(e) => { .push_next(&mut dedicated),
self.device.destroy_buffer(buffer, None); None,
return Err(e).context("allocate exportable memory"); ) {
} Ok(m) => m,
}; Err(e) => {
if let Err(e) = self.device.bind_buffer_memory(buffer, memory, 0) { libc::close(raw); // failed import does not consume the fd
self.device.free_memory(memory, None); self.device.destroy_buffer(buffer, None);
self.device.destroy_buffer(buffer, None); return Err(anyhow!("import dmabuf memory: {e}"));
return Err(e).context("bind export memory"); }
} };
let opaque_fd = match self.ext_fd.get_memory_fd( if let Err(e) = self.device.bind_buffer_memory(buffer, memory, 0) {
&vk::MemoryGetFdInfoKHR::default() // `memory` owns the imported fd — freeing it releases the fd too.
.memory(memory)
.handle_type(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD),
) {
Ok(f) => f,
Err(e) => {
self.device.free_memory(memory, None); self.device.free_memory(memory, None);
self.device.destroy_buffer(buffer, None); self.device.destroy_buffer(buffer, None);
return Err(e).context("vkGetMemoryFdKHR"); return Err(e).context("bind import memory");
} }
}; self.src_cache.insert(
// CUDA imports (and on success owns) the exported fd. Size must match the allocation. fd,
// `import_owned_fd` closes `opaque_fd` on its own failure, so only the Vulkan objects unwind. SrcBuf {
let cuda = match cuda::ExternalDmabuf::import_owned_fd(opaque_fd, reqs.size) { buffer,
Ok(c) => c, memory,
Err(e) => { size,
},
);
Ok(())
}
}
/// (Re)create the exportable destination of at least `size` bytes + its CUDA mapping.
unsafe fn ensure_dst(&mut self, size: u64) -> Result<()> {
// SAFETY: caller contract: single-threaded use of handles this bridge owns; every builder
// info is built from locals that outlive the synchronous call reading them; created handles
// are destroyed on the error paths below or owned by `DstBuf`.
unsafe {
if self.dst.as_ref().is_some_and(|d| d.size >= size) {
return Ok(());
}
// Build the replacement FULLY before retiring the old one. Previously the old dst was
// destroyed and `self.dst` nulled up front, so a failed rebuild both dropped the working
// buffer AND leaked every object the partial rebuild created (`buffer`/`memory` are raw ash
// handles with no Drop, and `VkBridge::drop` only frees the live `self.dst`). Now every
// fallible step unwinds locally, and the swap happens only on full success.
let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
let buffer = self
.device
.create_buffer(
&vk::BufferCreateInfo::default()
.size(size)
// STORAGE so the NV12 compute CSC can write it as an SSBO (T2.5b).
.usage(
vk::BufferUsageFlags::TRANSFER_DST
| vk::BufferUsageFlags::STORAGE_BUFFER,
)
.push_next(&mut ext_info),
None,
)
.context("create export buffer")?;
let reqs = self.device.get_buffer_memory_requirements(buffer);
let mem_type = match self
.memory_type(reqs.memory_type_bits, vk::MemoryPropertyFlags::DEVICE_LOCAL)
{
Ok(t) => t,
Err(e) => {
self.device.destroy_buffer(buffer, None);
return Err(e);
}
};
let mut export = vk::ExportMemoryAllocateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer);
let memory = match self.device.allocate_memory(
&vk::MemoryAllocateInfo::default()
.allocation_size(reqs.size)
.memory_type_index(mem_type)
.push_next(&mut export)
.push_next(&mut dedicated),
None,
) {
Ok(m) => m,
Err(e) => {
self.device.destroy_buffer(buffer, None);
return Err(e).context("allocate exportable memory");
}
};
if let Err(e) = self.device.bind_buffer_memory(buffer, memory, 0) {
self.device.free_memory(memory, None); self.device.free_memory(memory, None);
self.device.destroy_buffer(buffer, None); self.device.destroy_buffer(buffer, None);
return Err(e).context("cuImportExternalMemory(OPAQUE_FD from Vulkan)"); return Err(e).context("bind export memory");
} }
}; let opaque_fd = match self.ext_fd.get_memory_fd(
// Full success: retire the previous buffer now, then publish the new one. &vk::MemoryGetFdInfoKHR::default()
if let Some(old) = self.dst.take() { .memory(memory)
self.device.destroy_buffer(old.buffer, None); .handle_type(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD),
self.device.free_memory(old.memory, None); ) {
// old.cuda drops its mapping with it Ok(f) => f,
Err(e) => {
self.device.free_memory(memory, None);
self.device.destroy_buffer(buffer, None);
return Err(e).context("vkGetMemoryFdKHR");
}
};
// CUDA imports (and on success owns) the exported fd. Size must match the allocation.
// `import_owned_fd` closes `opaque_fd` on its own failure, so only the Vulkan objects unwind.
let cuda = match cuda::ExternalDmabuf::import_owned_fd(opaque_fd, reqs.size) {
Ok(c) => c,
Err(e) => {
self.device.free_memory(memory, None);
self.device.destroy_buffer(buffer, None);
return Err(e).context("cuImportExternalMemory(OPAQUE_FD from Vulkan)");
}
};
// Full success: retire the previous buffer now, then publish the new one.
if let Some(old) = self.dst.take() {
self.device.destroy_buffer(old.buffer, None);
self.device.free_memory(old.memory, None);
// old.cuda drops its mapping with it
}
tracing::info!(size, "Vulkan→CUDA exportable staging buffer ready");
self.dst = Some(DstBuf {
buffer,
memory,
size: reqs.size,
cuda,
});
Ok(())
} }
tracing::info!(size, "Vulkan→CUDA exportable staging buffer ready");
self.dst = Some(DstBuf {
buffer,
memory,
size: reqs.size,
cuda,
});
Ok(())
} }
/// Build the RGB→NV12 compute pipeline once (T2.5b): two-SSBO descriptor set + a 28-byte /// Build the RGB→NV12 compute pipeline once (T2.5b): two-SSBO descriptor set + a 28-byte
@@ -477,112 +490,124 @@ impl VkBridge {
/// what was already created (the caller retries per frame, so a leak would be unbounded) and /// what was already created (the caller retries per frame, so a leak would be unbounded) and
/// leaves `self.csc` `None`. /// leaves `self.csc` `None`.
unsafe fn ensure_csc(&mut self) -> Result<()> { unsafe fn ensure_csc(&mut self) -> Result<()> {
if self.csc.is_some() { // SAFETY: caller contract: single-threaded use of handles this bridge owns. `build_csc`
return Ok(()); // fills `csc` front to back; on failure the destroy calls below run in reverse creation
// order, and Vulkan destroys are defined no-ops on the null handles a partial build left.
unsafe {
if self.csc.is_some() {
return Ok(());
}
let mut csc = Csc {
module: vk::ShaderModule::null(),
dset_layout: vk::DescriptorSetLayout::null(),
playout: vk::PipelineLayout::null(),
pipeline: vk::Pipeline::null(),
dpool: vk::DescriptorPool::null(),
dset: vk::DescriptorSet::null(),
};
if let Err(e) = self.build_csc(&mut csc) {
// Reverse creation order; Vulkan destroy calls are defined no-ops on the null
// handles a partial build left behind.
let d = &self.device;
d.destroy_descriptor_pool(csc.dpool, None); // frees `csc.dset` with it
d.destroy_pipeline(csc.pipeline, None);
d.destroy_pipeline_layout(csc.playout, None);
d.destroy_descriptor_set_layout(csc.dset_layout, None);
d.destroy_shader_module(csc.module, None);
return Err(e);
}
self.csc = Some(csc);
tracing::info!(
"Vulkan-bridge NV12 compute CSC ready (LINEAR path feeds NVENC native YUV)"
);
Ok(())
} }
let mut csc = Csc {
module: vk::ShaderModule::null(),
dset_layout: vk::DescriptorSetLayout::null(),
playout: vk::PipelineLayout::null(),
pipeline: vk::Pipeline::null(),
dpool: vk::DescriptorPool::null(),
dset: vk::DescriptorSet::null(),
};
if let Err(e) = self.build_csc(&mut csc) {
// Reverse creation order; Vulkan destroy calls are defined no-ops on the null
// handles a partial build left behind.
let d = &self.device;
d.destroy_descriptor_pool(csc.dpool, None); // frees `csc.dset` with it
d.destroy_pipeline(csc.pipeline, None);
d.destroy_pipeline_layout(csc.playout, None);
d.destroy_descriptor_set_layout(csc.dset_layout, None);
d.destroy_shader_module(csc.module, None);
return Err(e);
}
self.csc = Some(csc);
tracing::info!("Vulkan-bridge NV12 compute CSC ready (LINEAR path feeds NVENC native YUV)");
Ok(())
} }
/// The fallible half of [`ensure_csc`](Self::ensure_csc): fills `csc` front to back so the /// The fallible half of [`ensure_csc`](Self::ensure_csc): fills `csc` front to back so the
/// caller knows exactly what to destroy when a step fails. /// caller knows exactly what to destroy when a step fails.
unsafe fn build_csc(&mut self, csc: &mut Csc) -> Result<()> { unsafe fn build_csc(&mut self, csc: &mut Csc) -> Result<()> {
let words: Vec<u32> = CSC_SPV // SAFETY: caller contract (via `ensure_csc`): single-threaded use of handles this bridge
.chunks_exact(4) // owns; every builder info is built from locals that outlive the synchronous call reading
.map(|c| u32::from_le_bytes(c.try_into().unwrap())) // them; partial results land in `csc` for the caller to destroy on failure.
.collect(); unsafe {
csc.module = self let words: Vec<u32> = CSC_SPV
.device .chunks_exact(4)
.create_shader_module(&vk::ShaderModuleCreateInfo::default().code(&words), None) .map(|c| u32::from_le_bytes(c.try_into().unwrap()))
.context("create CSC shader module")?; .collect();
let bindings = [ csc.module = self
vk::DescriptorSetLayoutBinding::default() .device
.binding(0) .create_shader_module(&vk::ShaderModuleCreateInfo::default().code(&words), None)
.descriptor_type(vk::DescriptorType::STORAGE_BUFFER) .context("create CSC shader module")?;
.descriptor_count(1) let bindings = [
.stage_flags(vk::ShaderStageFlags::COMPUTE), vk::DescriptorSetLayoutBinding::default()
vk::DescriptorSetLayoutBinding::default() .binding(0)
.binding(1) .descriptor_type(vk::DescriptorType::STORAGE_BUFFER)
.descriptor_type(vk::DescriptorType::STORAGE_BUFFER) .descriptor_count(1)
.descriptor_count(1) .stage_flags(vk::ShaderStageFlags::COMPUTE),
.stage_flags(vk::ShaderStageFlags::COMPUTE), vk::DescriptorSetLayoutBinding::default()
]; .binding(1)
csc.dset_layout = self .descriptor_type(vk::DescriptorType::STORAGE_BUFFER)
.device .descriptor_count(1)
.create_descriptor_set_layout( .stage_flags(vk::ShaderStageFlags::COMPUTE),
&vk::DescriptorSetLayoutCreateInfo::default().bindings(&bindings), ];
None, csc.dset_layout = self
) .device
.context("create CSC dset layout")?; .create_descriptor_set_layout(
let pc = [vk::PushConstantRange::default() &vk::DescriptorSetLayoutCreateInfo::default().bindings(&bindings),
.stage_flags(vk::ShaderStageFlags::COMPUTE) None,
.size(28)]; )
let layouts = [csc.dset_layout]; .context("create CSC dset layout")?;
csc.playout = self let pc = [vk::PushConstantRange::default()
.device .stage_flags(vk::ShaderStageFlags::COMPUTE)
.create_pipeline_layout( .size(28)];
&vk::PipelineLayoutCreateInfo::default() let layouts = [csc.dset_layout];
.set_layouts(&layouts) csc.playout = self
.push_constant_ranges(&pc), .device
None, .create_pipeline_layout(
) &vk::PipelineLayoutCreateInfo::default()
.context("create CSC pipeline layout")?; .set_layouts(&layouts)
let entry = c"main"; .push_constant_ranges(&pc),
let stage = vk::PipelineShaderStageCreateInfo::default() None,
.stage(vk::ShaderStageFlags::COMPUTE) )
.module(csc.module) .context("create CSC pipeline layout")?;
.name(entry); let entry = c"main";
csc.pipeline = self let stage = vk::PipelineShaderStageCreateInfo::default()
.device .stage(vk::ShaderStageFlags::COMPUTE)
.create_compute_pipelines( .module(csc.module)
vk::PipelineCache::null(), .name(entry);
&[vk::ComputePipelineCreateInfo::default() csc.pipeline = self
.stage(stage) .device
.layout(csc.playout)], .create_compute_pipelines(
None, vk::PipelineCache::null(),
) &[vk::ComputePipelineCreateInfo::default()
.map_err(|(_, e)| anyhow!("create CSC pipeline: {e}"))?[0]; .stage(stage)
let sizes = [vk::DescriptorPoolSize::default() .layout(csc.playout)],
.ty(vk::DescriptorType::STORAGE_BUFFER) None,
.descriptor_count(2)]; )
csc.dpool = self .map_err(|(_, e)| anyhow!("create CSC pipeline: {e}"))?[0];
.device let sizes = [vk::DescriptorPoolSize::default()
.create_descriptor_pool( .ty(vk::DescriptorType::STORAGE_BUFFER)
&vk::DescriptorPoolCreateInfo::default() .descriptor_count(2)];
.max_sets(1) csc.dpool = self
.pool_sizes(&sizes), .device
None, .create_descriptor_pool(
) &vk::DescriptorPoolCreateInfo::default()
.context("create CSC descriptor pool")?; .max_sets(1)
csc.dset = self .pool_sizes(&sizes),
.device None,
.allocate_descriptor_sets( )
&vk::DescriptorSetAllocateInfo::default() .context("create CSC descriptor pool")?;
.descriptor_pool(csc.dpool) csc.dset = self
.set_layouts(&layouts), .device
) .allocate_descriptor_sets(
.context("allocate CSC descriptor set")?[0]; &vk::DescriptorSetAllocateInfo::default()
Ok(()) .descriptor_pool(csc.dpool)
.set_layouts(&layouts),
)
.context("allocate CSC descriptor set")?[0];
Ok(())
}
} }
/// Bridge one LINEAR dmabuf frame into a pooled NV12 CUDA buffer (latency plan T2.5b): /// Bridge one LINEAR dmabuf frame into a pooled NV12 CUDA buffer (latency plan T2.5b):
crates/pf-zerocopy/src/lib.rs
+4
View File
@@ -10,7 +10,11 @@
// Unsafe-proof program: every `unsafe {}` / `unsafe impl` must carry a `// SAFETY:` proof. Each // Unsafe-proof program: every `unsafe {}` / `unsafe impl` must carry a `// SAFETY:` proof. Each
// file keeps its own `#![deny(...)]` too; this crate-root deny is the catch-all gate. // file keeps its own `#![deny(...)]` too; this crate-root deny is the catch-all gate.
// `unsafe_op_in_unsafe_fn` closes the gap the clippy lint leaves: operations inside an
// `unsafe fn` body are not "unsafe blocks", so without it ~45 functions' worth of raw driver
// calls sat OUTSIDE the invariant this crate advertises.
#![deny(clippy::undocumented_unsafe_blocks)] #![deny(clippy::undocumented_unsafe_blocks)]
#![deny(unsafe_op_in_unsafe_fn)]
/// Wait for a dmabuf's implicit read-ready fence (`DMA_BUF_IOCTL_EXPORT_SYNC_FILE` + poll). /// Wait for a dmabuf's implicit read-ready fence (`DMA_BUF_IOCTL_EXPORT_SYNC_FILE` + poll).
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]