fix(host/capture): mmap the buffer fd ourselves — xdpw MemFd over-reads MAP_BUFFERS
apple / swift (push) Successful in 55s
windows-host / package (push) Successful in 2m28s
android / android (push) Successful in 10m10s
ci / web (push) Successful in 32s
ci / docs-site (push) Successful in 29s
ci / rust (push) Successful in 11m44s
deb / build-publish (push) Successful in 3m7s
decky / build-publish (push) Successful in 34s
ci / bench (push) Successful in 4m44s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 15s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m57s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m51s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 21s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m21s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m8s
docker / deploy-docs (push) Successful in 20s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m54s

The CPU de-pad path trusted PipeWire's MAP_BUFFERS slice (`d.data()`, length =
`data.maxsize`). xdg-desktop-portal-wlr hands MemFd ScreenCast buffers whose
maxsize exceeds the bytes PipeWire actually maps into our process, so reading to
maxsize ran off the end of the mapping and SIGSEGV'd the capture thread —
crashing every CPU-path capture on Sway/wlroots (and thus any non-NVIDIA host,
which has no CUDA zero-copy importer and always falls back to this path).

mmap the fd ourselves, sized to its real length (fstat), for any fd-backed
buffer (MemFd SHM or DmaBuf); fall back to `d.data()` then drop. The existing
`needed > avail` guard now drops cleanly instead of over-reading. This also
subsumes the original "MAP_BUFFERS didn't map a Vulkan dmabuf" fallback.

Verified: fixes real Sway-desktop portal capture -> VAAPI HEVC on a Radeon 780M
(correct image + colours); the NVIDIA zero-copy path (returns before this code)
and the NVIDIA/KWin CPU path (self-mmap, fd_len == maxsize) both still work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-19 21:48:49 +00:00
parent f96e4ec9f8
commit 5e27f65f2e
+48 -12
View File
@@ -845,7 +845,9 @@ mod pipewire {
let d = &mut datas[0];
// CPU path may also receive LINEAR dmabufs (gamescope offers only those once its
// modifier-bearing format pod wins); capture the fd before `data()` borrows `d`.
let dmabuf_fd = (d.type_() == pw::spa::buffer::DataType::DmaBuf).then(|| d.fd());
let data_type = d.type_();
// fd-backed buffer (MemFd SHM, or DmaBuf)? Capture the fd before `data()` borrows `d`.
let raw_fd = d.fd();
let (size, offset, stride) = {
let c = d.chunk();
(
@@ -865,22 +867,43 @@ mod pipewire {
let needed = stride * (h - 1) + row;
// dmabuf chunks commonly report size 0; fall back to the computed span.
let size = if size == 0 { needed } else { size };
// MAP_BUFFERS only maps buffers flagged mappable; Vulkan-exported dmabufs
// (gamescope) usually aren't, so mmap the fd ourselves for the de-pad read.
// For fd-backed buffers (MemFd SHM, DmaBuf) mmap the fd OURSELVES, sized to the fd's real
// length (fstat), rather than trusting PipeWire's MAP_BUFFERS slice: xdg-desktop-portal-wlr
// hands MemFd buffers whose reported `data.maxsize` exceeds the bytes actually mapped into
// our process, so reading to maxsize segfaults (it also covers the original case — MAP_BUFFERS
// not mapping Vulkan dmabufs, e.g. gamescope). The `needed > avail` guard below then drops
// cleanly if the real buffer is genuinely too small. MemPtr buffers (no fd) are same-process —
// trust `d.data()`.
let fd_len = if raw_fd > 0 {
unsafe {
let mut st: libc::stat = std::mem::zeroed();
(libc::fstat(raw_fd as i32, &mut st) == 0 && st.st_size > 0)
.then_some(st.st_size as usize)
}
} else {
None
};
let _mapping; // keeps a manual mmap alive for the copy below
let buf: &[u8] = if let Some(data) = d.data() {
data
} else if let Some(fd) = dmabuf_fd.filter(|&fd| fd > 0) {
match DmabufMap::new(fd, offset + needed) {
// Prefer our own fstat-sized mmap of the fd; fall back to PipeWire's MAP_BUFFERS slice
// (and finally drop) so an fd PipeWire could map but we can't never silently over-reads.
let self_mapped: Option<&[u8]> = if raw_fd > 0 {
let map_len = fd_len.unwrap_or(offset + needed);
match DmabufMap::new(raw_fd as i32, map_len) {
Some(m) => {
_mapping = m;
unsafe { std::slice::from_raw_parts(_mapping.ptr as *const u8, _mapping.len) }
}
None => {
warn_once("mmap(dmabuf) failed — frames dropped");
return;
Some(unsafe {
std::slice::from_raw_parts(_mapping.ptr as *const u8, _mapping.len)
})
}
None => None,
}
} else {
None
};
let buf: &[u8] = if let Some(b) = self_mapped {
b
} else if let Some(data) = d.data() {
data
} else {
warn_once("buffer has no mappable data — frames dropped");
return;
@@ -890,6 +913,19 @@ mod pipewire {
return;
}
let avail = buf.len() - offset;
{
// One-time geometry dump — makes a new compositor/GPU's buffer layout visible in the
// logs (the kind of mismatch that crashed xdpw MemFd capture before the self-mmap fix).
use std::sync::atomic::{AtomicBool, Ordering};
static ONCE: AtomicBool = AtomicBool::new(true);
if ONCE.swap(false, Ordering::Relaxed) {
tracing::info!(
stride, size, offset, buf_len = buf.len(), needed,
data_type = ?data_type, fd_len = ?fd_len, self_mapped = self_mapped.is_some(),
"capture CPU de-pad geometry (first frame)"
);
}
}
if needed > avail || needed > size {
warn_once("buffer smaller than frame span — frames dropped");
return;