punktfunk-encode-worker: GPU priority via a capability-carrying worker, with WP3 on-glass complete #153

Merged
enricobuehler merged 14 commits from worktree-worktree-encode-worker into main 2026-08-10 10:45:26 +00:00
39 changed files with 6113 additions and 585 deletions
+15
View File
@@ -280,6 +280,21 @@ jobs:
done
echo "OK: $(echo "$DEPS" | grep -E '^libav|^libsw' | tr '\n' ' ')"
# 0.26.0-1 setcap'd `cap_sys_nice=ep` on the host from this package's .INSTALL scriptlet and
# killed desktop streaming on every KDE box — with a green board, because nothing here ever
# looked at what the built package would DO. The lesson recorded then was "verify the
# PACKAGE, never the board"; this is that, and pacman is the channel where it matters most,
# since capabilities live in the scriptlet rather than in package metadata.
#
# Host must carry NOTHING, the worker exactly cap_sys_nice=ep. `--self-test` runs first so a
# guard that has quietly lost the ability to fail takes the job down rather than approving a
# release. (Only the host package is checked: the client/web/scripting packages ship neither
# binary and the script skips them by itself.)
- name: Assert the capability matrix (Arch package)
run: |
bash scripts/ci/assert-cap-matrix.sh --self-test
bash scripts/ci/assert-cap-matrix.sh "$GITHUB_WORKSPACE"/dist/punktfunk-host-*.pkg.tar.zst
# The optional HDR gamescope companion (packaging/gamescope) — a separate pkgbase with a
# completely different dependency set, published into the same repo so `pacman -S
# punktfunk-gamescope` is all an Arch/SteamOS box needs for 10-bit BT.2020 PQ.
+18 -1
View File
@@ -310,8 +310,14 @@ jobs:
# with "there is no reactor running, must be called from the context of a Tokio 1.x runtime".
# It WAS listed here, which is why only the .deb shipped a crashing tray while the RPM and
# Arch packages — which already split it — were fine.
#
# punktfunk-encode-worker IS in this invocation: it is the capability-carrying PyroWave
# encode worker that ships next to the host in /usr/bin, and build-deb.sh only builds it
# if the artifact is missing — building it here keeps it on the same sccache pass as the
# host. Unlike the tray it shares the host's dependency graph by design (v1 accepts that
# the worker links the same FFmpeg), so feature unification here is harmless.
cargo build --release --locked --features punktfunk-host/nvenc,punktfunk-host/vulkan-encode \
-p punktfunk-host
-p punktfunk-host -p punktfunk-encode-worker
- name: Build host .deb (FFmpeg bundled)
# BUNDLE_FFMPEG=1 copies the image's /opt/ffmpeg libav* into the package and repoints the
@@ -320,6 +326,17 @@ jobs:
run: |
VERSION="$VERSION" BUNDLE_FFMPEG=1 bash packaging/debian/build-deb.sh
# Read the capability matrix out of the BUILT .deb before it is published. dpkg carries no
# capability metadata — the postinst applies them — so this reads the postinst that will
# actually run on a user's box, plus the payload. 0.26.0-1 granted the host cap_sys_nice=ep
# from exactly that postinst and killed every KDE desktop session while every board stayed
# green: host must carry NOTHING, worker exactly cap_sys_nice=ep. `--self-test` first so a
# guard that can no longer fail takes the job down instead of waving the release through.
- name: Assert the capability matrix (host .deb)
run: |
bash scripts/ci/assert-cap-matrix.sh --self-test
bash scripts/ci/assert-cap-matrix.sh dist/punktfunk-host_*.deb
# punktfunk-gamescope for apt. Same reasoning as the RPM leg in rpm.yml: without a packaged
# build, a Debian/Ubuntu box has no route to the patched gamescope except compiling it, and a
# stock gamescope streams SDR, cursorless, and tells every game its display is 60 Hz.
+32 -1
View File
@@ -103,7 +103,11 @@ jobs:
# gamescope`.) Matches packaging/rpm/punktfunk.spec, which dropped its BuildRequires too.
dnf -y install gtk4-devel libadwaita-devel SDL3-devel
# sysext build (packaging/bazzite/build-sysext.sh): squashfs + SELinux labeling.
dnf -y install squashfs-tools cpio libselinux-utils selinux-policy-targeted
# libcap = setcap/getcap: the sysext is the ONLY place the image can acquire
# cap_sys_nice=ep on punktfunk-encode-worker (a merged /usr is read-only squashfs and no
# scriptlet ever runs), and it is also what the build's host-must-be-uncapped assertion
# and the capability-matrix CI leg read with. Without it the image ships the lever inert.
dnf -y install squashfs-tools cpio libselinux-utils selinux-policy-targeted libcap
# Fedora's own gamescope, for its RUNTIME libraries only — never shipped, never run. The
# sysext folds in our punktfunk-gamescope and verifies it by executing `--version`, and
# on a cache hit (the common case) nothing else in this job would have pulled libavif /
@@ -155,6 +159,20 @@ jobs:
RPM_GPG_PASSPHRASE: ${{ secrets.RPM_GPG_PASSPHRASE }}
run: bash packaging/rpm/sign-rpms.sh
# Read the file-capability matrix out of the BUILT rpm, before anything is signed or
# published. 0.26.0-1 shipped `%caps(cap_sys_nice=ep)` on the host through this very spec —
# on Fedora and, via rpm-ostree layering, on Bazzite — and every board was green while every
# KDE desktop session died in the field. The lesson recorded then was "verify the PACKAGE,
# never the board"; this is that. Host must carry NOTHING; the worker must carry exactly
# cap_sys_nice=ep. `--self-test` first, so a guard that has quietly stopped being able to
# fail takes the job down instead of waving the release through.
- name: Assert the capability matrix (rpm)
run: |
bash scripts/ci/assert-cap-matrix.sh --self-test
# Only the main host package carries binaries; -debuginfo/-debugsource and the
# client/web/scripting subpackages ship neither and are skipped by the script itself.
bash scripts/ci/assert-cap-matrix.sh dist/punktfunk-[0-9]*.rpm
- name: Publish to the Gitea RPM registry
env:
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
@@ -302,6 +320,19 @@ jobs:
dist/punktfunk-web-"${PF_VERSION}-${PF_RELEASE}"*.rpm \
dist/punktfunk-scripting-"${PF_VERSION}-${PF_RELEASE}"*.rpm
# Read the capability matrix back OUT of the image that is about to be published — the one
# channel where getting it wrong is unrepairable, because a merged sysext's /usr is read-only
# squashfs and the only fix is a new image plus a feed republish. 0.26.0-1's Bazzite breakage
# was confirmed exactly this way, after the fact, by mounting the published .raw and running
# getcap on it. Doing it here means the .raw never reaches the feed.
#
# The script proves its own reader first (cap a file, squash it, unsquash it, read it back)
# so a runner that cannot see file capabilities FAILS the leg instead of blessing the image.
- name: Assert the capability matrix (sysext image)
run: |
bash scripts/ci/assert-cap-matrix.sh \
"dist-sysext/punktfunk-${PF_VERSION}-${PF_RELEASE}-x86-64.raw"
# The feed's SHA256SUMS is OpenPGP-signed with the same packages@unom.io key as the RPMs, and
# punktfunk-sysext(8) refuses a feed it can't verify — the checksums alone never proved
# anything, sitting on the same registry as the images they describe.
+78
View File
@@ -12,6 +12,84 @@ with the version table of the release you are moving to, then read **Breaking ch
---
## v0.27.0 — in development
Sections accumulate here as work lands. The version table and the commit count are written at the
version bump, alongside `docs/releases/v0.27.0.md` — see `docs/releases/README.md`.
### `punktfunk-encode-worker` — the GPU-priority capability moves off the host
0.26.0 left the PyroWave priority ladder wired and inert: it needs `CAP_SYS_NICE`, and 0.26.0-1
proved the host can never hold one — see **PyroWave on Linux — Wave 2**, PW1, under v0.26.0 below. A
capability-carrying process cannot be identified by KWin (`cap_ptrace_access_check` refuses
`/proc/<pid>/exe` to a reader whose effective set is not a superset of the target's **permitted**
set), so it never gets `zkde_screencast_unstable_v1` and every KDE desktop session dies. Neither
`prctl(PR_SET_DUMPABLE, 1)` nor systemd `AmbientCapabilities=` nor a NixOS `security.wrappers` entry
changes that — all three land the capability in the same permitted set.
The capability therefore moves to a process that fronts nothing. **`punktfunk-encode-worker`** is a
new workspace member and a new installed binary: it owns the priority-elevated Vulkan device for
PyroWave sessions, receives capture dmabufs over a `SOCK_SEQPACKET` pair from its parent, and returns
compressed access units. It connects to no compositor, no D-Bus and no network, so its
non-dumpability costs nothing and its blast radius is one socket to the host that spawned it.
🛑 **The invariant, for anyone packaging this:** the worker is a **separate file**. Never a hardlink
to `punktfunk-host` and never a subcommand of it — a shared inode shares the file capability, which
silently re-creates 0.26.0-1 on every KDE box. `punktfunk-host` carries no capability, on any
channel, ever.
- **The grants are re-targeted, not re-introduced.** Every channel that granted in 0.26.0-1 grants
again, at the worker: Arch `.install` (`post_install` **and** `post_upgrade` — a replaced binary is
a new inode), RPM `%caps(cap_sys_nice=ep)` in `%files` (never a `%post setcap`; this covers Fedora
and Bazzite layering), the Bazzite sysext staging tree pre-`mksquashfs` (which does record
`security.capability`), the deb `postinst`, the Deck installer, and NixOS
`security.wrappers.punktfunk-encode-worker`. Every #136 host-side removal stays verbatim, including
the sysext's host hard-fail.
- **The sysext assertion is amended, not removed** — host must be empty (hard fail), worker must
carry **exactly** `cap_sys_nice=ep`. A *missing* worker capability is not an error: the grant is
best-effort everywhere.
- **A new release-CI leg asserts the getcap matrix** on the built Arch package, the deb and the
mounted sysext raw. The 0.26.0-1 lesson was "verify the package, never the board"; this is that,
mechanized, and it is what would have caught the original break.
- **On NixOS the env override is load-bearing**, not a convenience: a file capability cannot live on
a read-only store path, so the module wraps the worker and sets `PUNKTFUNK_ENCODE_WORKER` to the
wrapper path in the unit. An ambient grant is fine *here* — the worker is not a KWin client. The
host's `ExecStart` stays on the plain store path (the #136 fix stands).
**Fallback ladder — no rung can kill a negotiated session.** Binary not found → spawn failure →
handshake timeout → protocol or workspace-version mismatch → socket EOF mid-session all fall back to
the **in-process encoder exactly as today**, at default priority, with one warning. Host and worker
are different files now, so the version check is load-bearing rather than decorative; they ship
lockstep in every channel. The in-process path stays compiled and tested — it is the floor, not dead
code. `PYROWAVE_QUEUE_PRIORITY` keeps its 0.26.0 grammar and is now forwarded **explicitly** in the
handshake rather than read from the worker's environment, which is sanitized at spawn; one env var
still means one thing on both platforms.
### Host and client environment variables
- **`PUNKTFUNK_ENCODE_WORKER`** *(new, host, Linux)* — where to find the encode worker. Resolution
order: this variable → alongside `/proc/self/exe``PATH`. `off` forces the in-process encoder,
the debug escape hatch that makes the A/B a one-line change. Load-bearing on NixOS (above).
- **`PYROWAVE_QUEUE_PRIORITY`** *(unchanged grammar, new consumer)* — the *intent*, forwarded to the
worker; the granted class comes back in the handshake and the host logs it centrally, so the
in-process INERT warning does not double-fire. When the worker is uncapped as well — an operator
stripped it, or the filesystem cannot store the capability — the same INERT wording fires, now
naming the worker binary rather than the host.
### Documentation
- `docs-site` **Running as a service → GPU scheduling priority** rewritten around the split: the
worker carries the capability, the host never does, and `setcap` on `punktfunk-host` is called out
as the thing an operator must never do, with the `zkde_screencast_unstable_v1` symptom spelled out
so anyone who already did it can self-diagnose. The anchor is unchanged, so existing links hold.
- `configuration.md` gains the `PUNKTFUNK_ENCODE_WORKER` row and rewrites `PYROWAVE_QUEUE_PRIORITY`
off "the packages deliberately do not grant this".
- The 0.26.0 user-facing notes describe a privilege that is deliberately not granted. That is the
record of what 0.26.0 shipped and is **not** rewritten; the new phrasing — granted to the worker,
never to the host — lives in `docs/releases/v0.27.0.md`.
---
## v0.26.0
52 commits since v0.25.0.
Generated
+11
View File
@@ -3118,6 +3118,8 @@ dependencies = [
"pf-zerocopy",
"punktfunk-core",
"pyrowave-sys",
"serde",
"serde_json",
"tracing",
"tracing-subscriber",
"windows 0.62.2 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -3624,6 +3626,15 @@ dependencies = [
"zeroize",
]
[[package]]
name = "punktfunk-encode-worker"
version = "0.26.0"
dependencies = [
"pf-encode",
"tracing",
"tracing-subscriber",
]
[[package]]
name = "punktfunk-host"
version = "0.26.0"
+3
View File
@@ -4,6 +4,9 @@ members = [
"crates/punktfunk-core",
"crates/punktfunk-host",
"crates/punktfunk-host/vendor/usbip-sim",
# The capability-carrying PyroWave encode worker. A SEPARATE binary by design — never a
# hardlink of, or a subcommand of, punktfunk-host (design/gpu-priority-capability-worker.md).
"crates/punktfunk-encode-worker",
"crates/punktfunk-tray",
"crates/pf-bitstream",
"crates/pf-bitstream/vendor/cros-codecs",
+8
View File
@@ -34,7 +34,15 @@ pf-capture = { path = "../pf-capture" }
# Software H.264 (openh264, BSD-2) — the GPU-less encode path on both platforms.
openh264 = "0.9"
[target.'cfg(target_os = "linux")'.dev-dependencies]
# The encode-worker protocol tests measure what an AU costs as a serde_json body — the reason the
# access units ride a memfd instead (enc/linux/worker.rs).
serde_json = "1"
[target.'cfg(target_os = "linux")'.dependencies]
# The `punktfunk-encode-worker` protocol (enc/linux/worker.rs). The framing is pf-zerocopy's
# `ipc`, which is generic over the serde body; the message enums live here and version separately.
serde = { version = "1", features = ["derive"] }
# libavcodec (NVENC libav + VAAPI backends). `ffmpeg-sys-next` auto-detects the FFmpeg version, so
# this pin tracks the crate's own major (which shadows FFmpeg's): 9 = FFmpeg 9 (libavcodec 63,
# libavutil 61). Arch shipped FFmpeg 9 on 2026-08-08 and every soname moved with it; the packaged
+40 -3
View File
@@ -525,7 +525,32 @@ impl NvencEncoder {
None => {}
}
let enc = match video.open_with(opts) {
// libav's OWN failure path can take the whole host down with it. When NVENC init fails,
// `ff_nvenc_encode_init` calls `ff_cuda_check`, which hands `av_log` an `err_name`/
// `err_string` pair it did not initialize when the CUDA error lookup does not fill them —
// and glibc then walks that pointer in `strlen` inside `av_vbprintf`. Measured twice on
// home-nobara-1 (fc44, libavcodec 62), identical stack both times:
//
// __strlen_evex <- av_vbprintf <- format_line <- av_log_default_callback
// <- ff_cuda_check <- ff_nvenc_encode_init <- avcodec_open2 <- NvencEncoder::open
//
// once as an outright SIGSEGV mid-session, and once as a thread wedged in that stack so
// the service never answered SIGTERM and systemd SIGABRT'd it. Either way one encoder
// open failure kills every session on the box.
//
// We cannot fix the distro's FFmpeg, so deny it the chance to format: these messages are
// AV_LOG_ERROR, and `av_log_default_callback` returns on the level check before
// `format_line` when the level is AV_LOG_FATAL. The failure is NOT swallowed — it comes
// back as `Err(e)` below and is reported with our own context.
//
// Scoped to the call ALONE, deliberately: the ENOSYS arm below recurses into `Self::open`,
// and `QuietLibavLog` takes a non-reentrant global mutex — holding it across the match
// would deadlock the retry.
let opened = {
let _quiet = QuietLibavLog::new();
video.open_with(opts)
};
let enc = match opened {
Ok(enc) => enc,
// The GPU lacks NV_ENC_CAPS_SUPPORT_INTRA_REFRESH — ffmpeg fails the open with
// ENOSYS ("Function not implemented"). Latch it (skip the doomed attempt on later
@@ -560,9 +585,21 @@ impl NvencEncoder {
);
}
Err(e) => {
// libav's own message for this failure was suppressed on purpose (see above), so
// say so — otherwise the next person debugging an NVENC open wonders why the
// journal has our error and none of FFmpeg's. There is no env switch to get it
// back (the guard is unconditional, and it outranks PUNKTFUNK_FFMPEG_DEBUG for the
// duration of the call): to read libav's text, drop the guard in a local build.
// What it costs is one line of the shape
// [hevc_nvenc @ ..] cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable ..
// and the AVERROR itself still travels in `e`.
return Err(e).with_context(|| {
format!("open {name} ({width}x{height}@{fps}, {bitrate_bps} bps)")
})
format!(
"open {name} ({width}x{height}@{fps}, {bitrate_bps} bps) — libav's own \
diagnostic is silenced across this call because its CUDA error formatter \
can fault the process"
)
});
}
};
if intra_refresh {
+52 -2
View File
@@ -194,6 +194,16 @@ pub(crate) struct ProbedSupport {
/// full-chroma 4:4:4 HEVC. `false` when unanswered (fail CLOSED, unlike `codecs`: the honest
/// downgrade is a 4:2:0 session, not a dead one).
pub hevc_444: bool,
/// `NV_ENC_CAPS_SUPPORT_10BIT_ENCODE` per listed codec — HEVC Main10 / AV1 10-bit. Rides this
/// same session for exactly the reason 4:4:4 does: the alternative, `linux::probe_can_encode_10bit`,
/// answers by opening an ffmpeg `hevc_nvenc`, and one ffmpeg NVENC open in a direct-SDK process
/// is the LOG-3 field bug that wedges every later open process-wide with
/// `NV_ENC_ERR_INVALID_VERSION`. That probe was left on ffmpeg when the 4:4:4 one was moved off
/// it, on the reading that "Linux HDR rides the libav P010 path" — no longer true for a CUDA
/// capture, which `open_video` sends to the direct backend (`bit_depth` passes straight through,
/// and `is_ten_bit_input` accepts packed 10-bit RGB). `false` when unanswered — fail CLOSED, an
/// 8-bit session beats a wedged one.
pub ten_bit: crate::CodecSupport,
}
/// The cached [`probe_support_uncached`] answer — one throwaway session per process lifetime.
@@ -234,6 +244,11 @@ fn probe_support_uncached() -> ProbedSupport {
av1: false,
},
hevc_444: false,
ten_bit: crate::CodecSupport {
h264: false,
h265: false,
av1: false,
},
};
let Ok(api) = try_api() else {
return unknown;
@@ -299,6 +314,33 @@ fn probe_support_uncached() -> ProbedSupport {
.is_ok()
&& val != 0;
}
// The 10-bit cap, per codec, on the SAME still-open session — same reason 4:4:4 rides it.
// Only queried against a listed GUID (a cap query for an absent codec is undefined).
let mut ten_bit = crate::CodecSupport {
h264: false,
h265: false,
av1: false,
};
if listed {
for (guid, slot) in [
(nv::NV_ENC_CODEC_HEVC_GUID, &mut ten_bit.h265),
(nv::NV_ENC_CODEC_AV1_GUID, &mut ten_bit.av1),
] {
if !guids.contains(&guid) {
continue;
}
let mut param = nv::NV_ENC_CAPS_PARAM {
version: nv::NV_ENC_CAPS_PARAM_VER,
capsToQuery: nv::NV_ENC_CAPS::NV_ENC_CAPS_SUPPORT_10BIT_ENCODE,
reserved: [0; 62],
};
let mut val: core::ffi::c_int = 0;
*slot = (api.get_encode_caps)(enc, guid, &mut param, &mut val)
.nv_ok()
.is_ok()
&& val != 0;
}
}
let _ = (api.destroy_encoder)(enc);
if !listed {
tracing::warn!(
@@ -313,6 +355,7 @@ fn probe_support_uncached() -> ProbedSupport {
av1: guids.contains(&nv::NV_ENC_CODEC_AV1_GUID),
},
hevc_444,
ten_bit,
}
}
}
@@ -704,8 +747,15 @@ pub struct NvencCudaEncoder {
fps: u32,
bitrate_bps: u64,
buffer_fmt: nv::NV_ENC_BUFFER_FORMAT,
/// Encoded bit depth (8 on Linux until Phase 5.1 lands a P010 capture path). Kept for parity with
/// the Windows Main10 config, which is ported but inert until a 10-bit input exists.
/// Encoded bit depth. **10 is live on Linux** — this used to say "8 until Phase 5.1 lands a P010
/// capture path", which the code has since outrun: the gamescope HDR capture patches offer
/// 10-bit BT.2020/PQ formats, `nvenc_fmt` maps `X2Rgb10`/`X2Bgr10` to `ARGB10`/`ABGR10`, and
/// [`is_ten_bit_input`] flips this field and `hdr` from the negotiated input. A 10-bit frame
/// deliberately takes NEITHER the NV12 nor the YUV444 convert (both compute CSCs write 8-bit
/// planes) and rides packed RGB to the encoder, which does its own BT.2020 CSC —
/// `pf-capture/src/linux/pipewire.rs` owns that gate. So Main10 needed no P010 path to arrive.
/// P010 remains worth having later purely as a perf win (pre-convert so NVENC skips its internal
/// RGB→YUV CSC, as NV12 already does for SDR) — it is not what makes 10-bit work.
bit_depth: u8,
/// Full-chroma 4:4:4 (HEVC Range Extensions) — set when the capturer delivers a planar-YUV444
/// `DeviceBuffer` on an HEVC session and the GPU supports YUV444 encode.
+123 -10
View File
@@ -599,6 +599,15 @@ pub struct PyroWaveEncoder {
/// Session-fixed negotiated chroma: 4:4:4 = full-res RG8 chroma plane + per-pixel CSC
/// (`rgb2yuv444.comp`) + `Chroma444` pyrowave objects.
chroma444: bool,
/// What the global-priority ladder in `open_inner` actually produced, kept so it can be
/// REPORTED rather than only logged. `punktfunk-encode-worker` sends it back to the host in
/// its handshake, which is the process that owns the log pipeline and knows which binary to
/// name — see [`super::worker::PriorityOutcome`].
priority: super::worker::PriorityOutcome,
/// `VkPhysicalDeviceProperties::deviceName` of the device this encoder opened. Sanity for the
/// same handshake: on a multi-GPU host, "which GPU is the worker on" is otherwise invisible
/// from the host process.
device_name: String,
/// Per-frame bitstream budget (hard CBR): `bitrate / (8 * fps)`.
frame_budget: usize,
/// `PUNKTFUNK_PERF`: the synchronous encode's own duration, which is the quantity the
@@ -679,6 +688,18 @@ impl PyroWaveEncoder {
);
}
/// What the global-priority ladder produced for this encoder — the quantity
/// `punktfunk-encode-worker` reports back so the host can log the grant (or the INERT refusal)
/// once, naming the right binary.
pub(crate) fn priority_outcome(&self) -> super::worker::PriorityOutcome {
self.priority
}
/// The Vulkan device this encoder opened on.
pub(crate) fn device_name(&self) -> &str {
&self.device_name
}
pub fn open(
width: u32,
height: u32,
@@ -686,7 +707,54 @@ impl PyroWaveEncoder {
bitrate_bps: u64,
chroma: crate::ChromaFormat,
) -> Result<Self> {
if !chroma.is_444() && (width % 2 != 0 || height % 2 != 0) {
// The in-process path reads the intent from ITS OWN environment, exactly as it always
// has, and owns the INERT warn. (`punktfunk-encode-worker` takes both from its parent —
// see `open_in_worker`.)
let intent = std::env::var("PYROWAVE_QUEUE_PRIORITY").ok();
Self::open_checked(
width,
height,
fps,
bitrate_bps,
chroma.is_444(),
intent.as_deref(),
true,
)
}
/// [`Self::open`] as `punktfunk-encode-worker` runs it.
///
/// Two things differ, and only two — the encoder itself is opened by the identical code path,
/// which is what keeps the worker/in-process A/B honest:
///
/// * `intent` arrives **explicitly** from the host's handshake rather than from this process's
/// environment (which the worker strips of `PYROWAVE_QUEUE_PRIORITY` at startup), so one
/// operator knob cannot come to mean two different things across the process boundary;
/// * the INERT warn is **left to the host**. It is the process with the log pipeline, and its
/// wording has to name the worker binary — the historical text says "CAP_SYS_NICE on the
/// host binary", which after 0.26.0-1 would send an operator to do the one thing that
/// breaks every KDE session.
pub(crate) fn open_in_worker(
width: u32,
height: u32,
fps: u32,
bitrate_bps: u64,
chroma444: bool,
intent: Option<&str>,
) -> Result<Self> {
Self::open_checked(width, height, fps, bitrate_bps, chroma444, intent, false)
}
fn open_checked(
width: u32,
height: u32,
fps: u32,
bitrate_bps: u64,
chroma444: bool,
intent: Option<&str>,
warn_inert: bool,
) -> Result<Self> {
if !chroma444 && (width % 2 != 0 || height % 2 != 0) {
bail!("pyrowave 4:2:0 needs even dimensions (got {width}x{height})");
}
// Checked against the chroma actually being opened, NOT hardcoded 4:4:4. The 4:2:0 block
@@ -697,11 +765,11 @@ impl PyroWaveEncoder {
// (its own bounds `assert` is compiled out by the Release vendored build).
// `validate_dimensions` rejects the impossible-at-any-chroma modes earlier; this is the
// 4:4:4-specific half plus defence in depth for the lab override.
if !crate::pyrowave_mode_fits_rdo(width, height, chroma.is_444()) {
if !crate::pyrowave_mode_fits_rdo(width, height, chroma444) {
bail!(
"pyrowave {} at {width}x{height} exceeds the rate controller's 16-bit block \
index (see pyrowave-sys patches/0002 note) lower the resolution",
if chroma.is_444() { "4:4:4" } else { "4:2:0" }
if chroma444 { "4:4:4" } else { "4:2:0" }
);
}
// SAFETY: `open_inner` only issues Vulkan/pyrowave calls whose preconditions it
@@ -713,12 +781,27 @@ impl PyroWaveEncoder {
height,
fps.max(1),
bitrate_bps.max(1_000_000),
chroma.is_444(),
chroma444,
intent,
warn_inert,
)
}
}
unsafe fn open_inner(w: u32, h: u32, fps: u32, bitrate: u64, chroma444: bool) -> Result<Self> {
/// `intent` is the raw `PYROWAVE_QUEUE_PRIORITY` value (`None` = unset ⇒ the default ladder),
/// resolved by the CALLER: in-process from this process's environment, in the worker from the
/// host's handshake. `warn_inert` decides whether THIS process emits the "every class refused"
/// warning — see [`Self::open_in_worker`].
#[allow(clippy::too_many_arguments)]
unsafe fn open_inner(
w: u32,
h: u32,
fps: u32,
bitrate: u64,
chroma444: bool,
intent: Option<&str>,
warn_inert: bool,
) -> Result<Self> {
let entry = ash::Entry::load().context("load vulkan loader")?;
let mut hold = DeviceHold {
@@ -860,8 +943,7 @@ impl PyroWaveEncoder {
// Granite takes the inherit branch and the patch has never done anything here. This
// is the Linux half. Must be pushed BEFORE the count/as_ptr wiring below, exactly
// like queue_family_foreign above.
let gp_candidates =
queue_priority_candidates(std::env::var("PYROWAVE_QUEUE_PRIORITY").ok().as_deref());
let gp_candidates = queue_priority_candidates(intent);
// Enable whichever alias the driver advertises (KHR = the promoted name), mirroring
// pf-zerocopy's VkBridge probe so the two can never disagree about the spelling.
let gp_ext =
@@ -943,13 +1025,18 @@ impl PyroWaveEncoder {
// created with. (The extension itself stays enabled and that is correct: it
// IS enabled on the device, it just carries no request.)
hold._queue_ci[0].p_next = std::ptr::null();
if !gp_candidates.is_empty() && gp.is_some() {
if !gp_candidates.is_empty() && gp.is_some() && warn_inert {
// MEASURED on .21 (RTX 5070 Ti, NVIDIA 610.43.02, 2026-08-08), and it is
// not a vendor quirk: an unprivileged host is refused EVERY class, and the
// same binary with `cap_sys_nice+ep` is granted REALTIME on the first
// attempt. So this arm is the normal state of a packaged host today, the
// lever is inert until the capability ships, and the message has to say
// which capability rather than leave an operator guessing.
//
// `warn_inert` is false in `punktfunk-encode-worker`: it reports the
// outcome to its parent, which logs the same sentence naming the WORKER
// binary. Sending an operator to `setcap` the host — which this wording
// does — is precisely the 0.26.0-1 incident.
tracing::warn!(
"pyrowave: every global queue priority class was refused — encoding \
at default priority. The GPU-preemption lever is INERT without \
@@ -962,9 +1049,33 @@ impl PyroWaveEncoder {
.context("create device")?
}
};
Ok((pd, family, device, foreign_qfi))
// The ladder's outcome, made reportable. `queue_priority_candidates` only ever yields
// REALTIME or HIGH, so the `Some(_)` arm is exact rather than a fallback (ash models
// the class as a newtype, not a Rust enum, so this cannot be a `match` on constants).
let priority = match chosen {
Some(c) if c == vk::QueueGlobalPriorityKHR::REALTIME => {
super::worker::PriorityOutcome::Granted(super::worker::GrantedClass::Realtime)
}
Some(_) => {
super::worker::PriorityOutcome::Granted(super::worker::GrantedClass::High)
}
// Exactly the condition the INERT warn above fires on: something was asked for,
// the extension was there, and every class came back refused.
None if !gp_candidates.is_empty() && gp.is_some() => {
super::worker::PriorityOutcome::Refused
}
None => super::worker::PriorityOutcome::NotRequested,
};
let device_name = instance
.get_physical_device_properties(pd)
.device_name_as_c_str()
.ok()
.and_then(|s| s.to_str().ok())
.unwrap_or("unknown")
.to_string();
Ok((pd, family, device, foreign_qfi, priority, device_name))
})();
let (pd, family, device, foreign_qfi) = match selected {
let (pd, family, device, foreign_qfi, priority, device_name) = match selected {
Ok(v) => v,
Err(e) => {
instance.destroy_instance(None);
@@ -1013,6 +1124,8 @@ impl PyroWaveEncoder {
height: h,
fps,
chroma444,
priority,
device_name,
frame_budget: budget_for(bitrate, fps),
perf_us: Vec::new(),
perf_logged_at: None,
File diff suppressed because it is too large Load Diff
+971
View File
@@ -0,0 +1,971 @@
//! `punktfunk-encode-worker` — the vocabulary both halves speak, and the worker half itself
//! (design: `design/gpu-priority-capability-worker.md` §3; plan §2/WP1). The host half is
//! [`super::pyrowave_remote`].
//!
//! **Why this process exists at all.** PyroWave encodes on the same shader cores a game
//! saturates, and the only lever that preempts it is an elevated `VK_KHR_global_priority` queue —
//! which the driver grants only to a process holding `CAP_SYS_NICE`. `punktfunk-host` may never
//! hold one: KWin identifies a client by `readlink /proc/<pid>/exe`, the kernel refuses that
//! readlink to a reader whose effective set is not a superset of the target's PERMITTED set, and
//! a capped host is therefore unidentifiable — 0.26.0-1 killed every KDE desktop session that
//! way. So the capability lives here, in a leaf that fronts nothing: no Wayland, no D-Bus, no
//! network, one socket to its parent.
//!
//! 🛑 This worker is a **separate executable file**, never a hardlink of the host and never a
//! subcommand of it (unlike the zerocopy worker, which deliberately re-execs the host image). A
//! shared inode shares the file capability, which silently re-creates 0.26.0-1.
//!
//! ## Shape
//!
//! One worker per PyroWave session, spawned at encoder open on the shared [`ipc`] rails (SEQPACKET
//! framing, `SCM_RIGHTS`, fd-3 inheritance, pinned-exe spawn, the zombie sweep). Strict
//! request/response: **every** host→worker message gets exactly one reply, so the two sides can
//! never desync into "whose turn is it".
//!
//! ## Where the bytes go (and why they are not in the JSON)
//!
//! [`ipc::MAX_MSG`] is 64 KiB and the bodies are serde_json, which renders a `Vec<u8>` as one
//! decimal number per byte. A PyroWave AU is `bitrate / (8 × fps)` — 83 KB at 1080p60/40 Mb/s,
//! ~830 KB at 4K — so an inline `bytes` field is not a slow path, it is *unrepresentable*, and
//! base64 would still need ~17 datagrams and ~0.8 ms of codec per frame against a +1.0 ms
//! whole-IPC-hop budget (plan §4 R1). The AU therefore rides a **memfd** the worker creates once
//! and `pwrite`s at offset 0 every frame; the host `pread`s exactly `len` bytes out of it. The fd
//! crosses once, in [`FromWorker::Ready`]. A memfd grows on write, so there is no capacity
//! negotiation and no regrow protocol — a bitrate retarget is invisible to it.
//!
//! Cursor bitmaps take the same route for the same reason (256×256 RGBA = 256 KiB > `MAX_MSG`),
//! except they are rare enough (only when the pointer *image* changes) that a fresh memfd rides
//! along with the frame instead of a persistent one.
//!
//! Frame pixels never cross at all: the dmabuf fd is passed on first sight of its `key` and the
//! worker caches it, so the steady state passes **zero** descriptors (the PipeWire pool recycles a
//! small buffer set).
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use anyhow::{Context, Result};
use pf_frame::{CapturedFrame, CursorOverlay, DmabufFrame, FramePayload, PixelFormat};
use pf_zerocopy::ipc;
use serde::de::DeserializeOwned;
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, VecDeque};
use std::ffi::CStr;
use std::fs::File;
use std::io;
use std::os::fd::{AsFd, BorrowedFd, FromRawFd, OwnedFd};
use std::os::unix::fs::FileExt;
use std::sync::Arc;
use std::time::{Duration, Instant};
/// Bumped on any wire change. Unlike the zerocopy worker — the same binary as its host by
/// construction — host and worker are **different files** here, so this check is load-bearing:
/// a package that shipped them out of lockstep must degrade to the in-process encoder, never to
/// a dead session.
pub(crate) const PROTO_VERSION: u32 = 1;
/// The workspace version this half was compiled from. A protocol can be unchanged while the
/// *encoder* moves (a vendored-codec bump, a CSC shader change), and the two halves must still be
/// one build — so the handshake compares this too. `env!` resolves at compile time of THIS crate,
/// so a stale worker binary carries its own older string even though both link the same source.
pub(crate) const WORKSPACE_VERSION: &str = env!("CARGO_PKG_VERSION");
/// Cached dmabuf fds. PipeWire pools are ≤ ~16 buffers; the cap only matters if a producer churns
/// buffers without a renegotiation, and an eviction is recoverable ([`FromWorker::NeedFd`]).
const FD_CACHE_CAP: usize = 64;
/// The largest cursor bitmap that can matter: the encoder clamps to a 256×256 RGBA texture
/// (`pyrowave.rs::CURSOR_MAX`), so uploading more would be bytes the blend cannot read.
const CURSOR_UPLOAD_MAX: usize = 256 * 256 * 4;
// ---------------------------------------------------------------------------
// Vocabulary
// ---------------------------------------------------------------------------
/// What the `VK_KHR_global_priority` ladder produced — i.e. whether the capability is doing
/// anything. Reported to the host so exactly ONE process logs it: the worker's own
/// `tracing` goes to inherited stderr, but the host is the process with the log pipeline (the
/// ring the web console serves), and the in-process INERT warn's wording ("CAP_SYS_NICE on the
/// host binary") would now actively mislead — the capability belongs on the worker.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum PriorityOutcome {
/// A class was granted — the lever is live.
Granted(GrantedClass),
/// A class was requested, the extension is there, and every class was refused: the lever is
/// INERT. This is the normal state of an *uncapped* worker.
Refused,
/// Nothing was asked for (`PYROWAVE_QUEUE_PRIORITY=off`) or the device advertises no
/// global-priority extension — not a problem, and never warned about.
NotRequested,
}
/// The granted `VkQueueGlobalPriorityKHR` class, wire-side.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum GrantedClass {
Realtime,
High,
}
/// [`pf_frame::PixelFormat`] on the wire. A hand-written mirror rather than a serde derive on the
/// original: pf-frame carries no serde dependency, and the exhaustive `match` in both directions
/// makes a new capture format a COMPILE error here instead of a silently mis-described frame.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum WireFormat {
Bgrx,
Rgbx,
Bgra,
Rgba,
Rgb,
Bgr,
Rgb10a2,
Nv12,
P010,
Yuv444,
X2Rgb10,
X2Bgr10,
}
impl From<PixelFormat> for WireFormat {
fn from(f: PixelFormat) -> WireFormat {
match f {
PixelFormat::Bgrx => WireFormat::Bgrx,
PixelFormat::Rgbx => WireFormat::Rgbx,
PixelFormat::Bgra => WireFormat::Bgra,
PixelFormat::Rgba => WireFormat::Rgba,
PixelFormat::Rgb => WireFormat::Rgb,
PixelFormat::Bgr => WireFormat::Bgr,
PixelFormat::Rgb10a2 => WireFormat::Rgb10a2,
PixelFormat::Nv12 => WireFormat::Nv12,
PixelFormat::P010 => WireFormat::P010,
PixelFormat::Yuv444 => WireFormat::Yuv444,
PixelFormat::X2Rgb10 => WireFormat::X2Rgb10,
PixelFormat::X2Bgr10 => WireFormat::X2Bgr10,
}
}
}
impl From<WireFormat> for PixelFormat {
fn from(f: WireFormat) -> PixelFormat {
match f {
WireFormat::Bgrx => PixelFormat::Bgrx,
WireFormat::Rgbx => PixelFormat::Rgbx,
WireFormat::Bgra => PixelFormat::Bgra,
WireFormat::Rgba => PixelFormat::Rgba,
WireFormat::Rgb => PixelFormat::Rgb,
WireFormat::Bgr => PixelFormat::Bgr,
WireFormat::Rgb10a2 => PixelFormat::Rgb10a2,
WireFormat::Nv12 => PixelFormat::Nv12,
WireFormat::P010 => PixelFormat::P010,
WireFormat::Yuv444 => PixelFormat::Yuv444,
WireFormat::X2Rgb10 => PixelFormat::X2Rgb10,
WireFormat::X2Bgr10 => PixelFormat::X2Bgr10,
}
}
}
/// [`pf_frame::CursorOverlay`] minus its pixels — cursor-as-metadata, the way the CSC consumes it.
/// `upload` is the pixel channel: `Some(len)` means a fresh memfd carrying `len` bytes of straight
/// -alpha RGBA rides with this frame (the bitmap `serial` changed); `None` means "reuse the bitmap
/// you cached for `serial`", which is every frame of a pointer that is merely moving.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub(crate) struct WireCursor {
pub x: i32,
pub y: i32,
pub w: u32,
pub h: u32,
pub serial: u64,
pub hot_x: u32,
pub hot_y: u32,
pub visible: bool,
pub upload: Option<usize>,
}
/// host → worker. Every variant has exactly one reply.
#[derive(Serialize, Deserialize, Debug, PartialEq)]
pub(crate) enum ToWorker {
/// Open the encoder. Answered with [`FromWorker::Ready`] (which carries the AU memfd) or
/// [`FromWorker::InitErr`].
///
/// `priority_intent` is the raw `PYROWAVE_QUEUE_PRIORITY` value as the HOST resolved it
/// (`None` = unset ⇒ the default REALTIME→HIGH ladder). Forwarded **explicitly** rather than
/// read from the worker's environment, and the worker strips the variable from its own env
/// before opening, so one knob cannot mean two things across the process boundary.
Hello {
proto: u32,
workspace_version: String,
/// The host's `PUNKTFUNK_RENDER_NODE` (`None` = unset). Log-only, exactly as it is
/// in-process: the device selection deliberately ignores every node anchor (see
/// `pyrowave.rs::select_physical_device` — two "fixes" were withdrawn). Carried so a
/// wrong-device field report shows the host's anchor beside the worker's pick.
drm_node: Option<String>,
width: u32,
height: u32,
fps: u32,
bitrate_bps: u64,
chroma444: bool,
priority_intent: Option<String>,
},
/// Encode one frame. The dmabuf fd rides as `SCM_RIGHTS` only on first sight of `key`
/// (`has_fd`); a cursor upload, when present, is the fd AFTER it. Answered with
/// [`FromWorker::Au`], [`FromWorker::NeedFd`] or [`FromWorker::EncodeErr`].
Frame {
key: u64,
has_fd: bool,
fourcc: u32,
modifier: u64,
offset: u32,
stride: u32,
plane1: Option<(u32, u32)>,
width: u32,
height: u32,
pts_ns: u64,
format: WireFormat,
cursor: Option<WireCursor>,
},
/// `Encoder::set_wire_chunking` — the datagram-aligned packetization boundary (plan §4.4).
/// This has to cross: it changes the AU BYTES (the windowed `build_au` framing) and the rate
/// budget, not just how the host hands them out. The streamed-AU *cutting* stays host-side.
SetWireChunking { shard_payload: usize },
/// `Encoder::reconfigure_bitrate` — an in-place rate retarget.
Reconfigure { bitrate_bps: u64 },
/// `Encoder::reset` — the stall watchdog's in-place rebuild, run INSIDE the worker so the
/// priority-elevated device survives it (see [`super::pyrowave_remote::RemotePyroWave::reset`]
/// for why this is a message and not a respawn).
Reset,
}
/// worker → host.
#[derive(Serialize, Deserialize, Debug, PartialEq)]
pub(crate) enum FromWorker {
/// The encoder is open. Carries the AU memfd as its single `SCM_RIGHTS` descriptor.
///
/// ⚠ `proto` and `workspace_version` are the first two fields and must never be renamed: they
/// are how a version-skewed pair diagnoses itself instead of failing obscurely.
Ready {
proto: u32,
workspace_version: String,
priority: PriorityOutcome,
device: String,
/// The chroma the encoder REALLY opened, and whether it blends the cursor — i.e.
/// `EncoderCaps` as only the opened encoder knows it. The proxy must not guess: a
/// hardcoded default mis-reports a 4:4:4 open and fires the session glue's spurious
/// "chroma disagrees with the negotiated Welcome" warn.
chroma444: bool,
blends_cursor: bool,
},
/// The open failed (no Vulkan 1.3 device, missing features, …) — an ANSWER, not a crash: the
/// host falls back to the in-process encoder, which will fail the same way if the cause is
/// real and succeed if the cause was the worker's own environment.
InitErr { message: String },
/// One access unit, complete, at offset 0 of the AU memfd. **Doubles as the buffer-release
/// signal**: it maps 1:1 onto `Encoder::submit`'s lifetime contract (the caller already holds
/// the frame alive until its AU comes back from `poll`), so the host loop needs no change.
Au {
key: u64,
len: usize,
pts_ns: u64,
keyframe: bool,
chunk_aligned: bool,
encode_us: u32,
},
/// No cached fd for this `key` (evicted, or the caches diverged) — the host forgets its
/// "already sent" note and retries the frame once, with the fd.
NeedFd,
/// This frame failed but the worker is alive.
EncodeErr { message: String },
/// Reply to [`ToWorker::SetWireChunking`] / [`ToWorker::Reconfigure`] / [`ToWorker::Reset`].
Ack { ok: bool },
}
// ---------------------------------------------------------------------------
// Framing helpers — EINTR, and the deadline it must not defeat
// ---------------------------------------------------------------------------
/// [`ipc::recv_fds`] that survives a signal.
///
/// With `SO_RCVTIMEO` armed the kernel returns **EINTR**, not `ERESTARTSYS`, so `SA_RESTART` does
/// not save the caller — any signal delivered to a thread blocked in `recv` surfaces as an error.
/// pf-zerocopy's importer maps *any* recv error to "the worker died", which is right for a
/// once-per-capture handshake and wrong for a per-frame AU: one stray signal would drop a healthy
/// session to the in-process fallback. So retry here.
///
/// The retry re-arms with the REMAINING budget rather than the full one — a signal arriving every
/// 100 ms would otherwise reset the clock forever and a real hang would never time out.
/// `budget = None` means "block until the host speaks or closes" (the worker's own serve loop).
pub(crate) fn recv_eintr<T: DeserializeOwned>(
sock: BorrowedFd,
buf: &mut Vec<u8>,
budget: Option<Duration>,
) -> io::Result<(T, Vec<OwnedFd>)> {
let deadline = budget.map(|d| Instant::now() + d);
loop {
if let Some(deadline) = deadline {
let left = deadline.saturating_duration_since(Instant::now());
if left.is_zero() {
return Err(io::Error::new(
io::ErrorKind::TimedOut,
"encode worker did not answer within its budget",
));
}
ipc::set_recv_timeout(sock, Some(left))?;
}
match ipc::recv_fds::<T>(sock, buf) {
Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
other => return other,
}
}
}
/// [`ipc::send_fds`] that survives a signal. A small body on a socket whose peer is actively
/// reading does not block, so this normally retries never; it exists so that "normally" is not
/// load-bearing.
pub(crate) fn send_eintr<T: Serialize>(
sock: BorrowedFd,
msg: &T,
fds: &[BorrowedFd],
) -> io::Result<()> {
loop {
match ipc::send_fds(sock, msg, fds) {
Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
other => return other,
}
}
}
/// An anonymous RAM-backed file for the bulk channels. Grows on `pwrite`, so callers never size it.
fn memfd(name: &CStr) -> io::Result<File> {
// SAFETY: `memfd_create` reads a NUL-terminated name (a live `CStr` for the duration of the
// call) and returns a fresh descriptor or -1; it retains no pointer. The result is checked
// before use, and the returned fd is owned by nobody else, so `File::from_raw_fd` takes sole
// ownership and closes it exactly once.
let fd = unsafe { libc::memfd_create(name.as_ptr(), libc::MFD_CLOEXEC) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
// SAFETY: `fd` is the fresh, valid descriptor just created and checked above.
Ok(unsafe { File::from_raw_fd(fd) })
}
/// Build the memfd carrying one cursor bitmap, clamped to what the blend can actually sample.
pub(crate) fn cursor_upload(rgba: &[u8]) -> io::Result<(File, usize)> {
let n = rgba.len().min(CURSOR_UPLOAD_MAX);
let f = memfd(c"pf-encode-cursor")?;
f.write_all_at(&rgba[..n], 0)?;
Ok((f, n))
}
// ---------------------------------------------------------------------------
// The worker half
// ---------------------------------------------------------------------------
/// `punktfunk-encode-worker` entry point. `args` are the process's own arguments after argv[0]
/// (`--fd N`, default 3 — the socket end the spawning host `dup2`'d in).
pub fn run_from_args(args: &[String]) -> Result<()> {
// Core dumps ON, and FIRST — the opposite of the host's posture, deliberately. `PR_SET_DUMPABLE`
// is cleared by the kernel whenever a process gains a file capability, which also suppresses
// core dumps and makes `/proc/<pid>/environ` unreadable. This process fronts nothing (no
// Wayland, no D-Bus, no network), so nothing is protected by that suppression and a crash in a
// GPU driver is exactly what we want a core for. It does NOT make us identifiable to KWin —
// the 0.26.0-1 matrix measured that dumpable is not the gate, the PERMITTED set is — and it
// does not need to: this process never speaks Wayland.
// SAFETY: `prctl(PR_SET_DUMPABLE, 1)` takes integers by value, touches no Rust memory and
// affects only this process.
unsafe {
libc::prctl(libc::PR_SET_DUMPABLE, 1);
}
// The host execs us via a pinned `/proc/self/fd/<n>`, so the kernel derives our comm from a
// meaningless fd number. Rename so `top`/`pkill`/a coredump path see the worker.
// SAFETY: `PR_SET_NAME` copies at most 16 bytes from the given pointer; the C-string literal is
// valid, NUL-terminated and short enough, and no pointer is retained past the call.
unsafe {
libc::prctl(libc::PR_SET_NAME, c"pf-encode-wk".as_ptr());
}
sanitize_env();
// Real teeth, and the second half of what the capability buys: `setpriority` is a silent no-op
// without `CAP_SYS_NICE`/`RLIMIT_NICE`, which is why the in-host encode thread's nice(-10) has
// never actually applied on a packaged Linux host. Here it applies. The worker is
// single-threaded, so this IS the encode thread.
pf_frame::thread_qos::boost_thread_priority(true);
let fd: i32 = args
.iter()
.skip_while(|a| *a != "--fd")
.nth(1)
.map(|s| s.parse())
.transpose()
.context("parse --fd")?
.unwrap_or(3);
// Refuse anything that cannot be the spawning host's socket: a negative fd is UB inside
// `OwnedFd` (its niche), and 02 would make the worker close one of its own stdio streams on
// exit. Then confirm the number really holds a socket — this binary is installed and runnable
// by hand, and adopting an arbitrary inherited fd would close it behind its real owner.
anyhow::ensure!(fd >= 3, "--fd must be >= 3 (got {fd})");
// SAFETY: `libc::stat` is plain-old-data for which all-zero is a valid value, so `mem::zeroed`
// is a sound initializer; `fstat` writes into the live, correctly-sized `&mut st` and only
// reads `fd`. `st_mode` is read only after the return value is checked.
let is_socket = unsafe {
let mut st: libc::stat = std::mem::zeroed();
libc::fstat(fd, &mut st) == 0 && (st.st_mode & libc::S_IFMT) == libc::S_IFSOCK
};
anyhow::ensure!(
is_socket,
"--fd {fd} is not an open socket (this binary is spawned by punktfunk-host, not run by hand)"
);
// SAFETY: the spawning host `dup2`'d its socketpair end onto exactly this fd number before
// exec (the worker's contract, just verified to be an open socket ≥ 3) and nothing else in
// this fresh process owns it, so `OwnedFd` takes sole ownership and closes it once at exit.
let sock = unsafe { OwnedFd::from_raw_fd(fd) };
run(sock)
}
/// Drop the environment variables this process must not act on.
///
/// Deliberately a DENYLIST, not an allowlist. The obvious "clear everything but a handful of
/// names" is wrong here: the Vulkan loader discovers its ICDs through the environment
/// (`VK_ICD_FILENAMES`/`VK_DRIVER_FILES`/`XDG_DATA_DIRS`), so a strict allowlist would leave the
/// worker with no GPU exactly on NixOS — the one channel where this worker's env override is
/// load-bearing. What must go is the punktfunk state that would make one knob mean two things:
/// the priority intent (it arrives explicitly in `Hello`) and the worker path itself (nothing here
/// spawns a worker, and a stale value in a core dump is just noise).
fn sanitize_env() {
// Single-threaded — this runs before anything in this process creates a thread, which is the
// one situation where mutating the environment is sound (the `getenv` race the house rule
// about `set_var` is about needs a second thread).
for k in ["PYROWAVE_QUEUE_PRIORITY", "PUNKTFUNK_ENCODE_WORKER"] {
std::env::remove_var(k);
}
}
/// Handshake, then serve until the host goes away.
fn run(sock: OwnedFd) -> Result<()> {
let mut buf = Vec::new();
// No timeout on the worker's own receives: the host owns the clock (it arms `SO_RCVTIMEO` on
// its end), and a worker that gave up on its own would look exactly like a crash.
let (hello, _) = recv_eintr::<ToWorker>(sock.as_fd(), &mut buf, None).context("recv Hello")?;
let ToWorker::Hello {
proto,
workspace_version,
drm_node,
width,
height,
fps,
bitrate_bps,
chroma444,
priority_intent,
} = hello
else {
anyhow::bail!("first message was not Hello");
};
if proto != PROTO_VERSION || workspace_version != WORKSPACE_VERSION {
// Answer, don't crash: the host prints one warn naming both builds and encodes in-process.
let _ = send_eintr(
sock.as_fd(),
&FromWorker::InitErr {
message: format!(
"version skew: worker proto {PROTO_VERSION} v{WORKSPACE_VERSION}, \
host proto {proto} v{workspace_version}"
),
},
&[],
);
return Ok(());
}
let enc = match super::pyrowave::PyroWaveEncoder::open_in_worker(
width,
height,
fps,
bitrate_bps,
chroma444,
priority_intent.as_deref(),
) {
Ok(e) => e,
Err(e) => {
let _ = send_eintr(
sock.as_fd(),
&FromWorker::InitErr {
message: format!("{e:#}"),
},
&[],
);
return Ok(());
}
};
let au_buf = memfd(c"pf-encode-au").context("create the AU return buffer")?;
let caps = crate::Encoder::caps(&enc);
let ready = FromWorker::Ready {
proto: PROTO_VERSION,
workspace_version: WORKSPACE_VERSION.to_string(),
priority: enc.priority_outcome(),
device: enc.device_name().to_string(),
chroma444: caps.chroma_444,
blends_cursor: caps.blends_cursor,
};
send_eintr(sock.as_fd(), &ready, &[au_buf.as_fd()]).context("send Ready")?;
tracing::info!(
pid = std::process::id(),
device = %enc.device_name(),
priority = ?enc.priority_outcome(),
host_render_node = ?drm_node,
"punktfunk-encode-worker ready"
);
serve(&sock, enc, &au_buf)
}
/// The request loop. `Ok(())` on host EOF (normal end-of-life — the host dropped its proxy);
/// any other socket error propagates and the process exits, which the host reads as a death,
/// because it is one.
fn serve(sock: &OwnedFd, mut enc: super::pyrowave::PyroWaveEncoder, au_buf: &File) -> Result<()> {
use crate::Encoder as _;
let mut buf = Vec::new();
let mut fds: HashMap<u64, OwnedFd> = HashMap::new();
// Insertion order, for the eviction the cap implies.
let mut fd_order: VecDeque<u64> = VecDeque::new();
// The cursor bitmap the host last uploaded, by `serial` — a moving pointer re-sends only its
// position, exactly like the in-process path re-uses its uploaded texture.
let mut cursor_rgba: Option<(u64, Arc<Vec<u8>>)> = None;
loop {
let (msg, got) = match recv_eintr::<ToWorker>(sock.as_fd(), &mut buf, None) {
Ok(v) => v,
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => return Ok(()),
Err(e) => return Err(e).context("worker recv"),
};
let reply = match msg {
ToWorker::Hello { .. } => FromWorker::EncodeErr {
message: "duplicate Hello".into(),
},
ToWorker::SetWireChunking { shard_payload } => {
enc.set_wire_chunking(shard_payload);
FromWorker::Ack { ok: true }
}
ToWorker::Reconfigure { bitrate_bps } => FromWorker::Ack {
ok: enc.reconfigure_bitrate(bitrate_bps),
},
ToWorker::Reset => FromWorker::Ack { ok: enc.reset() },
ToWorker::Frame {
key,
has_fd,
fourcc,
modifier,
offset,
stride,
plane1,
width,
height,
pts_ns,
format,
cursor,
} => {
// Descriptor order is the sender's: the dmabuf (iff `has_fd`), then the cursor
// upload (iff the bitmap changed). Taken before any early return so an unexpected
// extra descriptor is closed with the `Vec` rather than leaked.
let mut got = got.into_iter();
let dmabuf = if has_fd { got.next() } else { None };
let cursor_fd = cursor.as_ref().and_then(|c| c.upload.map(|_| got.next()));
if let Some(fd) = dmabuf {
if fds.insert(key, fd).is_none() {
fd_order.push_back(key);
}
while fd_order.len() > FD_CACHE_CAP {
if let Some(old) = fd_order.pop_front() {
fds.remove(&old);
}
}
}
match encode_one(
&mut enc,
au_buf,
&fds,
&mut cursor_rgba,
FrameReq {
key,
fourcc,
modifier,
offset,
stride,
plane1,
width,
height,
pts_ns,
format,
cursor,
},
cursor_fd.flatten(),
) {
Ok(reply) => reply,
Err(e) => FromWorker::EncodeErr {
message: format!("{e:#}"),
},
}
}
};
match send_eintr(sock.as_fd(), &reply, &[]) {
Ok(()) => {}
// The host vanished between our recv and our send — the same end-of-life as EOF.
Err(e) if e.kind() == io::ErrorKind::BrokenPipe => return Ok(()),
Err(e) => return Err(e).context("worker send"),
}
}
}
/// [`ToWorker::Frame`] minus the descriptors, so [`encode_one`] takes one argument per concept.
struct FrameReq {
key: u64,
fourcc: u32,
modifier: u64,
offset: u32,
stride: u32,
plane1: Option<(u32, u32)>,
width: u32,
height: u32,
pts_ns: u64,
format: WireFormat,
cursor: Option<WireCursor>,
}
/// Rebuild the `CapturedFrame`, encode it synchronously, and write the AU into `au_buf`.
fn encode_one(
enc: &mut super::pyrowave::PyroWaveEncoder,
au_buf: &File,
fds: &HashMap<u64, OwnedFd>,
cursor_rgba: &mut Option<(u64, Arc<Vec<u8>>)>,
req: FrameReq,
cursor_fd: Option<OwnedFd>,
) -> Result<FromWorker> {
use crate::Encoder as _;
let Some(cached) = fds.get(&req.key) else {
return Ok(FromWorker::NeedFd);
};
// A dup per frame, not a borrow: `DmabufFrame` owns its fd (the encoder's import path dups it
// again for Vulkan and drops the rest), while the cache must keep holding the original so the
// steady state passes no descriptors at all. One `dup`/`close` pair per frame is µs.
let fd = cached.try_clone().context("dup the cached dmabuf fd")?;
let cursor = match req.cursor {
Some(c) => {
match (c.upload, cursor_fd) {
(Some(len), Some(f)) => {
let mut px = vec![0u8; len];
File::from(f)
.read_exact_at(&mut px, 0)
.context("read the cursor upload")?;
*cursor_rgba = Some((c.serial, Arc::new(px)));
}
// An announced upload whose descriptor did not arrive. Rare (it takes a kernel
// refusal of the `SCM_RIGHTS`), and the reason it is an ERROR rather than a
// shrug: the host marks the serial "sent" on a successful AU, so blending
// nothing here would leave the pointer INVISIBLE for the rest of that bitmap's
// life, silently. Failing the frame drops the session onto the in-process
// encoder instead, which is a rung with a warning attached.
(Some(_), None) => anyhow::bail!("cursor upload announced but no descriptor came"),
(None, _) => {}
}
// Likewise a serial we hold no pixels for: the host only omits the upload for a serial
// it has seen acknowledged, so a miss is a desync, not a frame to guess at.
let Some(rgba) = cursor_rgba
.as_ref()
.filter(|(serial, _)| *serial == c.serial)
.map(|(_, px)| px.clone())
else {
anyhow::bail!("no cursor bitmap cached for serial {}", c.serial);
};
Some(CursorOverlay {
x: c.x,
y: c.y,
w: c.w,
h: c.h,
rgba,
serial: c.serial,
hot_x: c.hot_x,
hot_y: c.hot_y,
visible: c.visible,
})
}
None => None,
};
let frame = CapturedFrame {
width: req.width,
height: req.height,
pts_ns: req.pts_ns,
format: req.format.into(),
payload: FramePayload::Dmabuf(DmabufFrame {
fd,
fourcc: req.fourcc,
modifier: req.modifier,
plane1: req.plane1,
offset: req.offset,
stride: req.stride,
}),
cursor,
};
// submit→poll in one breath: this backend's encode is synchronous at depth 1, so the AU is
// ready when `poll` returns and `frame` (with its fd) is alive across both halves — the
// trait's lifetime contract, honored on this side of the socket too.
let t0 = Instant::now();
enc.submit(&frame)?;
let Some(au) = enc.poll()? else {
anyhow::bail!("encoder returned no AU for a submitted frame");
};
let encode_us = t0.elapsed().as_micros() as u32;
au_buf
.write_all_at(&au.data, 0)
.context("write the AU into the return buffer")?;
Ok(FromWorker::Au {
key: req.key,
len: au.data.len(),
pts_ns: au.pts_ns,
keyframe: au.keyframe,
chunk_aligned: au.chunk_aligned,
encode_us,
})
}
#[cfg(test)]
mod tests {
use super::*;
use std::os::fd::AsFd;
fn hello() -> ToWorker {
ToWorker::Hello {
proto: PROTO_VERSION,
workspace_version: WORKSPACE_VERSION.to_string(),
drm_node: Some("/dev/dri/renderD128".into()),
width: 3840,
height: 2160,
fps: 60,
bitrate_bps: 400_000_000,
chroma444: true,
priority_intent: Some("realtime".into()),
}
}
/// The vocabulary survives the wire in both directions, descriptors included. (The framing —
/// EOF, timeouts, the descriptor cap — is pf-zerocopy's `ipc` tests' job; this pins the
/// message types and the fd ORDER the frame path depends on.)
#[test]
fn proto_round_trip_both_directions() {
let (a, b) = ipc::socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
ipc::send(a.as_fd(), &hello(), None).unwrap();
let (got, fds) = ipc::recv_fds::<ToWorker>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, hello());
assert!(fds.is_empty());
let frame = ToWorker::Frame {
key: 0xdead_beef,
has_fd: true,
fourcc: 0x3432_5258,
modifier: 0x0300_0000_0000_1234,
offset: 0,
stride: 3840 * 4,
plane1: None,
width: 3840,
height: 2160,
pts_ns: 1_234_567_890,
format: WireFormat::Bgrx,
cursor: Some(WireCursor {
x: 10,
y: 20,
w: 32,
h: 32,
serial: 7,
hot_x: 1,
hot_y: 2,
visible: true,
upload: Some(32 * 32 * 4),
}),
};
// Two descriptors, in the order the receiver destructures them: dmabuf, then cursor.
let (dma, cur) = (memfd(c"t-dma").unwrap(), memfd(c"t-cur").unwrap());
ipc::send_fds(a.as_fd(), &frame, &[dma.as_fd(), cur.as_fd()]).unwrap();
let (got, fds) = ipc::recv_fds::<ToWorker>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, frame);
assert_eq!(fds.len(), 2);
let ready = FromWorker::Ready {
proto: PROTO_VERSION,
workspace_version: WORKSPACE_VERSION.to_string(),
priority: PriorityOutcome::Granted(GrantedClass::Realtime),
device: "NVIDIA GeForce RTX 5070 Ti".into(),
chroma444: true,
blends_cursor: true,
};
ipc::send(b.as_fd(), &ready, Some(dma.as_fd())).unwrap();
let (got, fd) = ipc::recv::<FromWorker>(a.as_fd(), &mut buf).unwrap();
assert_eq!(got, ready);
assert!(fd.is_some(), "Ready carries the AU return buffer");
for reply in [
FromWorker::Au {
key: 1,
len: 830_000,
pts_ns: 5,
keyframe: true,
chunk_aligned: true,
encode_us: 4400,
},
FromWorker::NeedFd,
FromWorker::Ack { ok: true },
FromWorker::EncodeErr {
message: "boom".into(),
},
] {
ipc::send(b.as_fd(), &reply, None).unwrap();
let (got, _) = ipc::recv::<FromWorker>(a.as_fd(), &mut buf).unwrap();
assert_eq!(got, reply);
}
}
/// An AU never rides in the JSON body, and this is why: the smallest per-frame budget the
/// encoder will ever use is already `MAX_MSG`, and serde_json renders a byte as up to four
/// characters. Pinned as a test so nobody "simplifies" the memfd away.
#[test]
fn an_inline_au_would_not_fit_a_message() {
// 1080p60 at a modest 40 Mb/s — well inside the shipped range.
let au = vec![0xABu8; 40_000_000 / (8 * 60)];
let body = serde_json::to_vec(&au).unwrap();
assert!(
body.len() > ipc::MAX_MSG,
"a {}-byte AU serialized to {} bytes, which would (wrongly) fit MAX_MSG {}",
au.len(),
body.len(),
ipc::MAX_MSG
);
}
/// A body over [`ipc::MAX_MSG`] is refused at the sender rather than truncated on the wire —
/// the property the memfd channel exists to respect.
#[test]
fn oversized_messages_are_refused_not_truncated() {
let (a, _b) = ipc::socketpair_seqpacket().unwrap();
let ToWorker::Hello {
proto,
drm_node,
width,
height,
fps,
bitrate_bps,
chroma444,
priority_intent,
..
} = hello()
else {
unreachable!("hello() builds a Hello");
};
let huge = ToWorker::Hello {
proto,
// Over `MAX_MSG` on its own — enum variants take no functional-update syntax, so the
// rest is destructured above rather than `..hello()`.
workspace_version: "x".repeat(ipc::MAX_MSG),
drm_node,
width,
height,
fps,
bitrate_bps,
chroma444,
priority_intent,
};
let err = ipc::send(a.as_fd(), &huge, None).unwrap_err();
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
}
/// Every `PixelFormat` maps to a wire tag and back unchanged. The `match`es are exhaustive, so
/// a new capture format is a compile error; this catches a mis-typed ARM in either direction.
#[test]
fn pixel_formats_round_trip() {
for f in [
PixelFormat::Bgrx,
PixelFormat::Rgbx,
PixelFormat::Bgra,
PixelFormat::Rgba,
PixelFormat::Rgb,
PixelFormat::Bgr,
PixelFormat::Rgb10a2,
PixelFormat::Nv12,
PixelFormat::P010,
PixelFormat::Yuv444,
PixelFormat::X2Rgb10,
PixelFormat::X2Bgr10,
] {
assert_eq!(PixelFormat::from(WireFormat::from(f)), f);
}
}
/// The bulk channel: a memfd written by one holder of the descriptor is readable at offset 0
/// by another, and it grows on write with no explicit sizing. That is the whole mechanism the
/// AU return depends on.
#[test]
fn memfd_round_trips_bytes_across_a_descriptor() {
let f = memfd(c"pf-encode-test").unwrap();
let au = vec![0x5Au8; 900_000];
f.write_all_at(&au, 0).unwrap();
let (a, b) = ipc::socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
ipc::send(a.as_fd(), &FromWorker::NeedFd, Some(f.as_fd())).unwrap();
let (_, fd) = ipc::recv::<FromWorker>(b.as_fd(), &mut buf).unwrap();
let mut back = vec![0u8; au.len()];
File::from(fd.unwrap()).read_exact_at(&mut back, 0).unwrap();
assert_eq!(back, au);
}
/// A cursor bitmap is clamped to what the 256×256 blend texture can sample — a larger one is
/// truncated, exactly as `prep_cursor`'s `bytes.min(rgba.len())` copy already truncates it.
#[test]
fn cursor_upload_clamps_to_the_blend_texture() {
let (_, n) = cursor_upload(&vec![0u8; CURSOR_UPLOAD_MAX * 4]).unwrap();
assert_eq!(n, CURSOR_UPLOAD_MAX);
let (_, n) = cursor_upload(&vec![0u8; 64 * 64 * 4]).unwrap();
assert_eq!(n, 64 * 64 * 4);
}
/// EINTR must not read as a dead worker. A `SIGURG` (default-ignored, so the test process
/// survives it) delivered to a thread parked in `recv` with `SO_RCVTIMEO` armed returns EINTR
/// — `SA_RESTART` does not apply to a timeout-armed socket — and the retry must swallow it and
/// still deliver the message that arrives afterwards.
#[test]
fn recv_survives_a_signal() {
let (a, b) = ipc::socketpair_seqpacket().unwrap();
let b = std::sync::Arc::new(b);
let waiter = {
let b = b.clone();
std::thread::spawn(move || {
let mut buf = Vec::new();
recv_eintr::<FromWorker>(b.as_fd(), &mut buf, Some(Duration::from_secs(10)))
})
};
// Give the thread time to park in `recvmsg`, then interrupt it repeatedly while the
// message is still not there.
std::thread::sleep(Duration::from_millis(50));
for _ in 0..5 {
// SAFETY: `pthread_kill` takes the live thread's id by value and a signal number;
// SIGURG's default disposition is "ignore", so delivery cannot kill the process.
unsafe {
libc::pthread_kill(
std::os::unix::thread::JoinHandleExt::as_pthread_t(&waiter),
libc::SIGURG,
);
}
std::thread::sleep(Duration::from_millis(10));
}
ipc::send(a.as_fd(), &FromWorker::Ack { ok: true }, None).unwrap();
let (got, _) = waiter.join().unwrap().expect("EINTR must not surface");
assert_eq!(got, FromWorker::Ack { ok: true });
}
/// …and the retry must not defeat the deadline: a socket nobody ever writes to still times
/// out, signals or no signals.
#[test]
fn recv_still_times_out() {
let (a, _b) = ipc::socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
let err = recv_eintr::<FromWorker>(a.as_fd(), &mut buf, Some(Duration::from_millis(80)))
.unwrap_err();
assert!(
matches!(
err.kind(),
io::ErrorKind::WouldBlock | io::ErrorKind::TimedOut
),
"unexpected error kind: {err:?}"
);
}
}
+62 -5
View File
@@ -358,8 +358,17 @@ fn open_video_backend_linux(
if codec == Codec::PyroWave {
#[cfg(feature = "pyrowave")]
{
return pyrowave::PyroWaveEncoder::open(width, height, fps, bitrate_bps, chroma)
.map(|e| (Box::new(e) as Box<dyn Encoder>, "pyrowave"));
// Through the worker seam, not straight at the encoder: PyroWave's GPU-priority lever
// needs CAP_SYS_NICE, which only `punktfunk-encode-worker` may carry. Every rung of
// that ladder ends at this exact in-process open — see `pyrowave_remote`.
return pyrowave_remote::open_preferring_worker(
width,
height,
fps,
bitrate_bps,
chroma,
)
.map(|e| (e, "pyrowave"));
}
#[cfg(not(feature = "pyrowave"))]
anyhow::bail!(
@@ -517,14 +526,16 @@ fn open_video_backend_linux(
// The lab override forces the wavelet stream onto a session negotiated for
// another codec — that session's chroma may be HEVC-4:4:4, which the
// pyrowave encoder doesn't do yet, so pin the override to 4:2:0.
pyrowave::PyroWaveEncoder::open(
// Same worker seam as the negotiated arm above: the lab override is where the
// A/B is measured, so it must not be the one path that skips the worker.
pyrowave_remote::open_preferring_worker(
width,
height,
fps,
bitrate_bps,
ChromaFormat::Yuv420,
)
.map(|e| (Box::new(e) as Box<dyn Encoder>, "pyrowave"))
.map(|e| (e, "pyrowave"))
}
#[cfg(not(feature = "pyrowave"))]
{
@@ -1589,7 +1600,41 @@ pub fn can_encode_10bit(codec: Codec) -> bool {
};
vulkan10 || vaapi::probe_can_encode_10bit(codec)
} else {
linux::probe_can_encode_10bit(codec)
// NVIDIA. Same rule the 4:4:4 arm above already follows, and for the same field
// bug: on a direct-SDK host the answer comes from the driver's own
// `NV_ENC_CAPS_SUPPORT_10BIT_ENCODE` over the direct SDK, NOT from opening an
// ffmpeg `hevc_nvenc`. One ffmpeg NVENC open in a direct-SDK process wedges every
// later open process-wide with `NV_ENC_ERR_INVALID_VERSION` until the host
// restarts (LOG-3). This probe was the LAST ffmpeg-NVENC use left on a default
// host — it was kept on the reading that "Linux HDR rides the libav P010 path",
// which `open_video` contradicts: a CUDA payload goes to the direct backend at
// whatever `bit_depth` was resolved, and `is_ten_bit_input` accepts the packed
// 10-bit RGB (`X2Bgr10`) a gamescope HDR capture negotiates. Reproduced on
// home-nobara-1 2026-08-10: HDR on → probe opens ffmpeg → the live direct-SDK
// session's caps probe returns INVALID_VERSION → repeated in-place rebuilds →
// "encoder did not recover" and the session dies. With the ffmpeg probe out of the
// process the same HDR session streams clean.
//
// Only a host that will REALLY serve the session over libav keeps the ffmpeg probe
// (PUNKTFUNK_NVENC_DIRECT=0, or a build without `--features nvenc`): there it
// validates the actual path, and ffmpeg's NVENC client runs in that process anyway.
#[cfg(feature = "nvenc")]
{
if nvenc_direct_enabled() {
let t = nvenc_cuda::probe_support().ten_bit;
match codec {
Codec::H265 => t.h265,
Codec::Av1 => t.av1,
_ => false,
}
} else {
linux::probe_can_encode_10bit(codec)
}
}
#[cfg(not(feature = "nvenc"))]
{
linux::probe_can_encode_10bit(codec)
}
}
}
#[cfg(target_os = "windows")]
@@ -2035,6 +2080,18 @@ mod vk_util;
#[cfg(all(target_os = "linux", feature = "pyrowave"))]
#[path = "enc/linux/pyrowave.rs"]
mod pyrowave;
// `punktfunk-encode-worker` (design/gpu-priority-capability-worker.md): the capability-carrying
// process that owns the priority-elevated PyroWave device, because `punktfunk-host` may never hold
// a file capability (0.26.0-1 — a capped host is unidentifiable to KWin and loses desktop
// streaming). `worker` is the vocabulary plus the worker's own run loop, and it is `pub` for
// exactly one caller: the ~30-line `main` of the separate `punktfunk-encode-worker` binary.
// `pyrowave_remote` is the host-side proxy and its fallback ladder.
#[cfg(all(target_os = "linux", feature = "pyrowave"))]
#[path = "enc/linux/pyrowave_remote.rs"]
mod pyrowave_remote;
#[cfg(all(target_os = "linux", feature = "pyrowave"))]
#[path = "enc/linux/worker.rs"]
pub mod worker;
// The Windows PyroWave encoder — NV12 zero-copy D3D11→Vulkan via pyrowave's own compat device
// (design/pyrowave-windows-host-zerocopy.md). Same module name as the Linux one (per-platform
// `#[path]`, mutually-exclusive cfg) so `crate::pyrowave::*` is flat on both.
+47 -211
View File
@@ -1,27 +1,27 @@
//! Host side of the isolated zero-copy GPU import (design:
//! `design/zerocopy-worker-isolation.md`): spawns the `zerocopy-worker` subprocess, mirrors the
//! [`super::egl::EglImporter`] entry points over the [`super::proto`] socket, and materializes
//! the worker's pooled CUDA buffers in this process via CUDA IPC (each buffer's handles are
//! opened exactly once and reused as the pool recycles). A worker death — the whole point of the
//! isolation — surfaces as an `Err` with [`RemoteImporter::dead`] set, never as a host fault.
//! `design/zerocopy-worker-isolation.md`): spawns the `zerocopy-worker` subprocess on the shared
//! [`super::ipc`] rails, mirrors the [`super::egl::EglImporter`] entry points over the
//! [`super::proto`] vocabulary, and materializes the worker's pooled CUDA buffers in this process
//! via CUDA IPC (each buffer's handles are opened exactly once and reused as the pool recycles).
//! A worker death — the whole point of the isolation — surfaces as an `Err` with
//! [`RemoteImporter::dead`] set, never as a host fault.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use super::cuda::{self, CUdeviceptr, DeviceBuffer, CU_IPC_HANDLE_SIZE};
use super::egl::DmabufPlane;
use super::proto::{self, BufferDesc, ImportKind, Reply, Request};
use super::ipc;
use super::proto::{BufferDesc, ImportKind, Reply, Request, PROTO_VERSION};
use anyhow::{bail, Context, Result};
use std::collections::{HashMap, HashSet};
use std::fs::File;
use std::io;
use std::os::fd::{AsFd, AsRawFd, BorrowedFd, OwnedFd};
use std::os::unix::process::CommandExt;
use std::path::{Path, PathBuf};
use std::process::{Child, Command};
use std::os::fd::{AsFd, BorrowedFd, OwnedFd};
use std::path::Path;
use std::process::Child;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::{Duration, Instant};
use std::sync::{Arc, Mutex};
use std::time::Duration;
/// Handshake budget: EGL + CUDA bring-up is ~200 ms; a cold driver load can take seconds.
const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(20);
@@ -79,110 +79,6 @@ fn close_mapping(m: &Mapping) {
}
}
/// Children whose worker hasn't exited yet at `RemoteImporter` drop time (it exits on socket
/// EOF, i.e. after the last in-flight frame drops). Swept on every spawn and every drop so
/// workers don't linger as zombies for more than one capture generation.
static REAPER: Mutex<Vec<(Child, Instant)>> = Mutex::new(Vec::new());
/// How long past `REPLY_TIMEOUT` a parked worker may linger before it is force-killed. A worker
/// wedged INSIDE a driver call never observes socket EOF, so `try_wait` alone would keep it (and
/// its CUcontext + BufferPool — order hundreds of MB of VRAM) forever.
const REAPER_KILL_DEADLINE: Duration = Duration::from_secs(20);
fn sweep_reaper() {
// Partition under the lock; kill/reap OUTSIDE it. A worker wedged inside a driver ioctl sits
// in D state and ignores SIGKILL — the old blocking `wait()` under the global mutex would
// then park every later `spawn()` and `drop()` behind a process that may never die.
let mut expired: Vec<Child> = Vec::new();
{
let mut list = REAPER.lock().unwrap();
let now = Instant::now();
let mut i = 0;
while i < list.len() {
if matches!(list[i].0.try_wait(), Ok(Some(_))) {
list.swap_remove(i); // exited on its own → reaped
} else if now.duration_since(list[i].1) > REAPER_KILL_DEADLINE {
expired.push(list.swap_remove(i).0);
} else {
i += 1;
}
}
}
for mut c in expired {
let _ = c.kill();
// Bounded reap (~100 ms of polls): a SIGKILL'd process reaps near-instantly unless it is
// in D state — then park it again (re-killing later is harmless) so a future sweep reaps
// it once the driver unwedges, instead of blocking anyone here forever.
let mut reaped = false;
for _ in 0..10 {
if matches!(c.try_wait(), Ok(Some(_))) {
reaped = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
if !reaped {
tracing::warn!(
pid = c.id(),
"zerocopy worker ignored SIGKILL (likely wedged in a driver call, D state) — \
parked for a later sweep"
);
REAPER.lock().unwrap().push((c, Instant::now()));
}
}
}
/// Fd pinned to this process's own executable image, opened (once, lazily) via the
/// `/proc/self/exe` magic link. The link names the running image's *inode*, not its path, so it
/// resolves even after the installed binary was replaced or deleted — and exec'ing the fd (via
/// [`fd_exec_path`]) then still runs byte-for-byte the build this process is. `current_exe()`
/// instead readlinks to a path: after a package upgrade under a running host that path is
/// "<path> (deleted)" and spawning it fails ENOENT — every capture then silently fell back to
/// the CPU copy — and even while the path exists it may hold a newer build whose worker
/// protocol mismatches this process.
static SELF_EXE: OnceLock<Option<File>> = OnceLock::new();
fn self_exe() -> Option<BorrowedFd<'static>> {
SELF_EXE
.get_or_init(|| {
let f = match File::open("/proc/self/exe") {
Ok(f) => f,
Err(e) => {
tracing::warn!(
error = %e,
"cannot pin /proc/self/exe — worker spawns use the current_exe() path, \
which breaks if this binary is replaced on disk"
);
return None;
}
};
if f.as_raw_fd() != 3 {
return Some(f);
}
// Fd 3 is the slot the spawn hands the worker its socket on (the `dup2` in
// `spawn_exe`) — pinned there, the child would clobber it before exec resolves
// `/proc/self/fd/3`. Re-number: 3 stays occupied by `f` during the clone, so the
// duplicate cannot land on it.
match f.try_clone() {
Ok(clone) => Some(clone),
Err(e) => {
tracing::warn!(error = %e, "re-numbering the pinned exe fd off fd 3 failed");
None
}
}
})
.as_ref()
.map(|f| f.as_fd())
}
/// `/proc/self/fd/<n>` — an exec'able path to `fd`'s inode. The kernel resolves it at exec time
/// inside the forked child, whose fd table is a copy of ours (close-on-exec applies only once
/// the exec succeeds), so it names the pinned inode no matter what sits at the file's original
/// path by then.
fn fd_exec_path(fd: BorrowedFd<'_>) -> PathBuf {
PathBuf::from(format!("/proc/self/fd/{}", fd.as_raw_fd()))
}
/// The remote (isolated) importer — one per capture. Method-for-method mirror of the in-process
/// [`super::egl::EglImporter`] surface the capture thread uses.
pub struct RemoteImporter {
@@ -196,13 +92,17 @@ pub struct RemoteImporter {
impl RemoteImporter {
/// Spawn the worker from this host binary and complete the readiness handshake. The worker
/// is exec'd through the pinned `SELF_EXE` fd, so it is always the exact image this
/// is exec'd through the pinned [`ipc::self_exe`] fd, so it is always the exact image this
/// process runs — even after the installed binary was replaced mid-flight. An `Err` here
/// means "no isolated zero-copy available" — callers fall back to the CPU path, exactly like
/// an in-process `EglImporter::new()` failure.
///
/// Self-exec is right *here* — host and worker are the same build by construction, so the
/// version check is a formality. It is the one thing the capability-carrying encode worker
/// must NOT copy: a shared inode shares the file capability.
pub fn spawn() -> Result<RemoteImporter> {
match self_exe() {
Some(fd) => Self::spawn_exe(&fd_exec_path(fd)),
match ipc::self_exe() {
Some(exe) => Self::spawn_exe(&exe.exec_path()),
None => Self::spawn_exe(
&std::env::current_exe().context("resolve /proc/self/exe for the worker")?,
),
@@ -211,45 +111,10 @@ impl RemoteImporter {
/// [`Self::spawn`] with an explicit executable (separated for tests).
fn spawn_exe(exe: &Path) -> Result<RemoteImporter> {
sweep_reaper();
let (host_end, worker_end) = proto::socketpair_seqpacket().context("worker socketpair")?;
let mut cmd = Command::new(exe);
// `exe` is normally an opaque `/proc/self/fd/<n>` — keep `ps` output meaningful.
cmd.arg0("punktfunk-host");
cmd.arg("zerocopy-worker").arg("--fd").arg("3");
let raw = worker_end.as_raw_fd();
let parent = std::process::id() as libc::pid_t;
// SAFETY: `pre_exec` runs between fork and exec, so only async-signal-safe calls are
// allowed — `prctl`, `getppid`, `dup2` and `fcntl` all are, and the closure captures only
// `Copy` ints (no allocation, no locks; the error paths use `from_raw_os_error`, which
// does not allocate). PR_SET_PDEATHSIG makes the kernel SIGKILL the worker when the host
// dies — without it a crashed host left the worker holding its CUcontext + BufferPool
// (order hundreds of MB of VRAM) indefinitely. The `getppid` check closes the standard
// race: if the host died between fork and the prctl, the signal is never delivered, so
// refuse to exec instead. `dup2(raw, 3)` installs the socket at the fd number the
// subcommand expects and clears CLOEXEC on the copy; if the parent's fd already IS 3,
// `dup2(3,3)` would preserve CLOEXEC, so that case clears the flag explicitly instead.
unsafe {
cmd.pre_exec(move || {
if libc::prctl(libc::PR_SET_PDEATHSIG, libc::SIGKILL) != 0 {
return Err(io::Error::last_os_error());
}
if libc::getppid() != parent {
return Err(io::Error::from_raw_os_error(libc::ESRCH));
}
if raw == 3 {
let flags = libc::fcntl(3, libc::F_GETFD);
if flags < 0 || libc::fcntl(3, libc::F_SETFD, flags & !libc::FD_CLOEXEC) < 0 {
return Err(io::Error::last_os_error());
}
} else if libc::dup2(raw, 3) < 0 {
return Err(io::Error::last_os_error());
}
Ok(())
});
}
let child = cmd.spawn().context("spawn zerocopy-worker")?;
drop(worker_end); // the child holds its own copy now
// `exe` is normally an opaque `/proc/self/fd/<n>` — the argv[0] keeps `ps` meaningful.
let (host_end, child) =
ipc::spawn_worker(exe, "punktfunk-host", &["zerocopy-worker", "--fd", "3"])
.context("spawn zerocopy-worker")?;
Self::from_socket(host_end, Some(child))
}
@@ -266,11 +131,11 @@ impl RemoteImporter {
rbuf: Vec::new(),
sent_keys: HashSet::new(),
};
proto::set_recv_timeout(importer.shared.sock.as_fd(), Some(HANDSHAKE_TIMEOUT))?;
let ready = proto::recv::<Reply>(importer.shared.sock.as_fd(), &mut importer.rbuf);
proto::set_recv_timeout(importer.shared.sock.as_fd(), Some(REPLY_TIMEOUT))?;
ipc::set_recv_timeout(importer.shared.sock.as_fd(), Some(HANDSHAKE_TIMEOUT))?;
let ready = ipc::recv::<Reply>(importer.shared.sock.as_fd(), &mut importer.rbuf);
ipc::set_recv_timeout(importer.shared.sock.as_fd(), Some(REPLY_TIMEOUT))?;
match ready {
Ok((Reply::Ready { version }, _)) if version == proto::PROTO_VERSION => {
Ok((Reply::Ready { version }, _)) if version == PROTO_VERSION => {
tracing::info!(
pid = importer.child.as_ref().map(|c| c.id()),
"zero-copy GPU import isolated in a worker process"
@@ -281,7 +146,7 @@ impl RemoteImporter {
importer.mark_dead();
bail!(
"zerocopy worker protocol mismatch (worker v{version}, host v{})",
proto::PROTO_VERSION
PROTO_VERSION
)
}
Ok((Reply::InitErr { message }, _)) => {
@@ -315,7 +180,7 @@ impl RemoteImporter {
if self.dead() {
return Vec::new();
}
if let Err(e) = proto::send(
if let Err(e) = ipc::send(
self.shared.sock.as_fd(),
&Request::Modifiers { fourcc },
None,
@@ -324,7 +189,7 @@ impl RemoteImporter {
self.mark_dead();
return Vec::new();
}
match proto::recv::<Reply>(self.shared.sock.as_fd(), &mut self.rbuf) {
match ipc::recv::<Reply>(self.shared.sock.as_fd(), &mut self.rbuf) {
Ok((Reply::Modifiers { modifiers }, _)) => modifiers,
Ok((other, _)) => {
tracing::warn!(?other, "unexpected zerocopy worker reply to Modifiers");
@@ -439,11 +304,11 @@ impl RemoteImporter {
stride: plane.stride,
has_fd,
};
if let Err(e) = proto::send(self.shared.sock.as_fd(), &req, pass) {
if let Err(e) = ipc::send(self.shared.sock.as_fd(), &req, pass) {
self.mark_dead();
return Err(e).context("zerocopy worker died (send)");
}
let reply = match proto::recv::<Reply>(self.shared.sock.as_fd(), &mut self.rbuf) {
let reply = match ipc::recv::<Reply>(self.shared.sock.as_fd(), &mut self.rbuf) {
Ok((reply, _)) => reply,
Err(e) => {
self.mark_dead();
@@ -503,7 +368,7 @@ impl RemoteImporter {
// captured `shared` Arc is what keeps the mapping + socket alive until
// the last frame drops. A retired mapping (its generation renegotiated
// away) closes here with its last reference.
let _ = proto::send(shared.sock.as_fd(), &Request::Release { id }, None);
let _ = ipc::send(shared.sock.as_fd(), &Request::Release { id }, None);
let mut g = shared.mappings.lock().unwrap();
if let Some(entry) = g.get_mut(&id) {
entry.refs = entry.refs.saturating_sub(1);
@@ -542,7 +407,7 @@ impl RemoteImporter {
});
}
if !self.dead() {
if let Err(e) = proto::send(self.shared.sock.as_fd(), &Request::ClearCache, None) {
if let Err(e) = ipc::send(self.shared.sock.as_fd(), &Request::ClearCache, None) {
tracing::warn!(error = %e, "zerocopy worker ClearCache failed");
self.mark_dead();
}
@@ -557,10 +422,10 @@ impl Drop for RemoteImporter {
// gone; park the rest for the next sweep.
if let Some(mut child) = self.child.take() {
if !matches!(child.try_wait(), Ok(Some(_))) {
REAPER.lock().unwrap().push((child, Instant::now()));
ipc::park_child(child);
}
}
sweep_reaper();
ipc::sweep_reaper();
}
}
@@ -619,11 +484,12 @@ fn open_mapping(desc: &BufferDesc) -> Result<Mapping> {
#[cfg(test)]
mod tests {
use super::*;
use std::os::fd::AsRawFd;
use std::thread;
fn handshake_server(reply: Reply) -> OwnedFd {
let (host, worker) = proto::socketpair_seqpacket().unwrap();
proto::send(worker.as_fd(), &reply, None).unwrap();
let (host, worker) = ipc::socketpair_seqpacket().unwrap();
ipc::send(worker.as_fd(), &reply, None).unwrap();
// Keep the worker end alive alongside the host end for the test's duration by leaking it
// into the reply thread below? Not needed: the handshake reply is already queued in the
// socket buffer, so the worker end may drop — recv still delivers queued data first.
@@ -634,7 +500,7 @@ mod tests {
#[test]
fn handshake_ready_and_version_gate() {
let host = handshake_server(Reply::Ready {
version: proto::PROTO_VERSION,
version: PROTO_VERSION,
});
let imp = RemoteImporter::from_socket(host, None).unwrap();
assert!(!imp.dead());
@@ -656,7 +522,7 @@ mod tests {
#[test]
fn handshake_eof_is_an_error() {
let (host, worker) = proto::socketpair_seqpacket().unwrap();
let (host, worker) = ipc::socketpair_seqpacket().unwrap();
drop(worker);
assert!(RemoteImporter::from_socket(host, None).is_err());
}
@@ -683,36 +549,6 @@ mod tests {
assert!(format!("{err:#}").contains("handshake"), "{err:#}");
}
#[test]
fn pinned_fd_exec_survives_on_disk_replacement() {
// The 2026-07-10 canary regression: a package upgrade replaced the installed binary and
// every worker spawn ENOENT'd (`current_exe()` readlinked to "<path> (deleted)"). The
// pinned-fd mechanism must keep exec'ing the original image after the file is gone: pin
// a copy of /bin/sh, delete it, then run it through the fd path.
let copy = std::env::temp_dir().join(format!("pf-zerocopy-exe-pin-{}", std::process::id()));
std::fs::copy("/bin/sh", &copy).unwrap();
let pinned = File::open(&copy).unwrap();
std::fs::remove_file(&copy).unwrap();
// Retry ETXTBSY: `fs::copy`'s write fd leaks into other tests' concurrently-forked
// children until their execs clear it (CLOEXEC applies only at exec), and exec'ing a
// file someone holds open for writing is refused. A harness artifact of copy-then-exec,
// not the mechanism under test — production pins a read-only fd on a binary nobody
// write-opens.
let status = loop {
match Command::new(fd_exec_path(pinned.as_fd()))
.arg("-c")
.arg("exit 42")
.status()
{
Err(e) if e.raw_os_error() == Some(libc::ETXTBSY) => {
std::thread::sleep(Duration::from_millis(10))
}
other => break other.expect("exec via /proc/self/fd of a deleted file"),
}
};
assert_eq!(status.code(), Some(42));
}
/// A request as the scripted peer saw it, paired with the identity (`st_ino`) of the
/// descriptor that actually arrived via SCM_RIGHTS — the `has_fd` boolean in the JSON body is
/// a *claim*; the received fd is the mechanism the whole worker design rests on, so tests
@@ -723,11 +559,11 @@ mod tests {
fn scripted_server(
replies: Vec<Reply>,
) -> (RemoteImporter, thread::JoinHandle<Vec<SeenRequest>>) {
let (host, worker) = proto::socketpair_seqpacket().unwrap();
proto::send(
let (host, worker) = ipc::socketpair_seqpacket().unwrap();
ipc::send(
worker.as_fd(),
&Reply::Ready {
version: proto::PROTO_VERSION,
version: PROTO_VERSION,
},
None,
)
@@ -736,7 +572,7 @@ mod tests {
let mut buf = Vec::new();
let mut seen = Vec::new();
let mut replies = replies.into_iter();
while let Ok((req, fd)) = proto::recv::<Request>(worker.as_fd(), &mut buf) {
while let Ok((req, fd)) = ipc::recv::<Request>(worker.as_fd(), &mut buf) {
let needs_reply = matches!(req, Request::Modifiers { .. } | Request::Import { .. });
let ino = fd
.as_ref()
@@ -744,7 +580,7 @@ mod tests {
seen.push((req, ino));
if needs_reply {
match replies.next() {
Some(r) => proto::send(worker.as_fd(), &r, None).unwrap(),
Some(r) => ipc::send(worker.as_fd(), &r, None).unwrap(),
None => break, // close → client sees a dead worker
}
}
+680
View File
@@ -0,0 +1,680 @@
//! Worker-IPC rails, shared by every punktfunk worker subprocess (design:
//! `design/zerocopy-worker-isolation.md`, generalized in
//! `design/gpu-priority-capability-worker-implementation-plan.md` §2/WP0). Two halves, both
//! deliberately free of any worker's *vocabulary* — the message enums live with their worker and
//! stay independently versioned:
//!
//! - **Framing.** A `SOCK_SEQPACKET` unix socketpair — reliable, ordered, message-framed (one
//! `sendmsg` = one message) — carrying serde bodies, with descriptors riding as `SCM_RIGHTS`
//! control data. Zero-length messages are reserved: `recvmsg` returning 0 on a SEQPACKET socket
//! is EOF (the peer died/closed), and a serialized message is never empty, so the two can't be
//! confused.
//! - **Process rails.** Spawn a worker on a *pinned* executable with its socket end on fd 3, kill
//! the host's children with it (`PR_SET_PDEATHSIG`), and reap them without ever blocking a
//! caller behind a process wedged in a driver ioctl.
//!
//! The zerocopy worker execs this process's own image ([`self_exe`]); the encode worker
//! (`design/gpu-priority-capability-worker.md`) is a **separate file** — it must never share an
//! inode with `punktfunk-host`, because a shared inode shares the file capability — so it passes
//! its own resolved path to [`spawn_worker`] instead.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use serde::de::DeserializeOwned;
use serde::Serialize;
use std::fs::File;
use std::io;
use std::os::fd::{AsRawFd, BorrowedFd, FromRawFd, OwnedFd, RawFd};
use std::os::unix::process::CommandExt;
use std::path::{Path, PathBuf};
use std::process::{Child, Command};
use std::sync::{Mutex, OnceLock};
use std::time::{Duration, Instant};
/// Upper bound for one serialized message (the largest real message — a modifier list — is far
/// below this). A message reported truncated at this size is a protocol error.
pub const MAX_MSG: usize = 64 * 1024;
/// Descriptors one message may carry. Four, because a multi-planar dmabuf can hold one fd per
/// plane; the single-fd case ([`send`]/[`recv`]) is the hot path and stays allocation-free.
pub const MAX_FDS: usize = 4;
/// Backing store for the `SCM_RIGHTS` control data, `u64` so it carries the 8-byte alignment
/// `cmsghdr` requires. `MAX_FDS` fds need `CMSG_SPACE(4 * 4) = 32` bytes on 64-bit Linux (a
/// 16-byte `cmsghdr` + 16 bytes of fds); 64 is double that, so the slack absorbs any platform
/// whose header is larger — the `cmsg_store_is_large_enough` test asserts it for real.
type CmsgStore = [u64; 8];
/// Control bytes the kernel may use for `n` descriptors. (`CMSG_SPACE` is pure size arithmetic —
/// the `unsafe` is libc's signature, not a contract.)
fn cmsg_space(n: usize) -> usize {
// SAFETY: `CMSG_SPACE` performs alignment arithmetic on its argument and touches no memory.
unsafe { libc::CMSG_SPACE((n * std::mem::size_of::<RawFd>()) as u32) as usize }
}
/// A CLOEXEC `SOCK_SEQPACKET` socketpair — `(host_end, worker_end)`.
pub fn socketpair_seqpacket() -> io::Result<(OwnedFd, OwnedFd)> {
let mut fds = [0i32; 2];
// SAFETY: `socketpair` writes two fds into `fds`, a live 2-element stack array matching the
// API contract; it reads no other Rust memory. The result is checked before the fds are used,
// and each returned fd is fresh (owned by no other wrapper), so the two `OwnedFd::from_raw_fd`
// each take sole ownership of a distinct, valid descriptor — no alias, no double-close.
unsafe {
if libc::socketpair(
libc::AF_UNIX,
libc::SOCK_SEQPACKET | libc::SOCK_CLOEXEC,
0,
fds.as_mut_ptr(),
) != 0
{
return Err(io::Error::last_os_error());
}
Ok((OwnedFd::from_raw_fd(fds[0]), OwnedFd::from_raw_fd(fds[1])))
}
}
/// Set (or clear) the receive timeout: a blocked [`recv`] then fails with
/// `ErrorKind::WouldBlock`. Used by the host so a hung worker can't wedge the calling thread.
pub fn set_recv_timeout(sock: BorrowedFd, timeout: Option<Duration>) -> io::Result<()> {
let tv = match timeout {
Some(d) => libc::timeval {
tv_sec: d.as_secs() as libc::time_t,
tv_usec: d.subsec_micros() as libc::suseconds_t,
},
None => libc::timeval {
tv_sec: 0,
tv_usec: 0,
},
};
// SAFETY: `setsockopt(SO_RCVTIMEO)` reads `size_of::<timeval>()` bytes from `&tv`, a live
// stack `timeval` that outlives this synchronous call; `sock` is the caller's live socket fd.
// Nothing is retained or written through Rust pointers.
let r = unsafe {
libc::setsockopt(
sock.as_raw_fd(),
libc::SOL_SOCKET,
libc::SO_RCVTIMEO,
&tv as *const libc::timeval as *const libc::c_void,
std::mem::size_of::<libc::timeval>() as libc::socklen_t,
)
};
if r != 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
/// Send one message (+ optionally one fd as `SCM_RIGHTS`) as a single SEQPACKET datagram — the
/// single-descriptor fast path over [`send_fds`] (a borrowed one-element slice; no fd list is
/// built).
pub fn send<T: Serialize>(
sock: BorrowedFd,
msg: &T,
pass_fd: Option<BorrowedFd>,
) -> io::Result<()> {
match pass_fd {
Some(fd) => send_fds(sock, msg, &[fd]),
None => send_fds(sock, msg, &[]),
}
}
/// Send one message plus up to [`MAX_FDS`] descriptors as a single SEQPACKET datagram. Atomic per
/// message, so concurrent senders on the same socket (e.g. the capture thread's imports and the
/// encode thread's releases) need no lock. `MSG_NOSIGNAL` turns a dead peer into `EPIPE` instead
/// of `SIGPIPE`. More than [`MAX_FDS`] is a caller bug, refused like an over-long body rather than
/// panicking.
pub fn send_fds<T: Serialize>(sock: BorrowedFd, msg: &T, fds: &[BorrowedFd]) -> io::Result<()> {
if fds.len() > MAX_FDS {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
format!(
"worker ipc: {} fds in one message (max {MAX_FDS})",
fds.len()
),
));
}
let body =
serde_json::to_vec(msg).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
debug_assert!(
!body.is_empty(),
"zero-length messages are reserved for EOF"
);
if body.len() > MAX_MSG {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"worker ipc message too large",
));
}
let mut iov = libc::iovec {
iov_base: body.as_ptr() as *mut libc::c_void,
iov_len: body.len(),
};
let mut cmsg_store: CmsgStore = [0; 8];
// SAFETY: `mhdr` is a plain-old-data C struct for which all-zero is a valid value.
let mut mhdr: libc::msghdr = unsafe { std::mem::zeroed() };
mhdr.msg_iov = &mut iov;
mhdr.msg_iovlen = 1;
if !fds.is_empty() {
let bytes = (fds.len() * std::mem::size_of::<RawFd>()) as u32;
debug_assert!(cmsg_space(fds.len()) <= std::mem::size_of_val(&cmsg_store));
mhdr.msg_control = cmsg_store.as_mut_ptr() as *mut libc::c_void;
// SAFETY: `CMSG_SPACE`/`CMSG_LEN` are pure size computations (no memory access).
// `CMSG_FIRSTHDR(&mhdr)` returns a pointer into `cmsg_store` (non-null: msg_controllen
// ≥ one cmsghdr), which is live, 8-aligned, and large enough — the store holds
// `CMSG_SPACE(MAX_FDS * 4)` and `fds.len() <= MAX_FDS` was checked above — for the header
// fields plus one 4-byte fd per element written through `CMSG_DATA`; `write_unaligned`
// handles the data area's byte alignment. All writes stay within `cmsg_store`, which
// outlives the synchronous `sendmsg` below.
unsafe {
mhdr.msg_controllen = libc::CMSG_SPACE(bytes) as _;
let c = libc::CMSG_FIRSTHDR(&mhdr);
(*c).cmsg_level = libc::SOL_SOCKET;
(*c).cmsg_type = libc::SCM_RIGHTS;
(*c).cmsg_len = libc::CMSG_LEN(bytes) as _;
let data = libc::CMSG_DATA(c) as *mut RawFd;
for (i, fd) in fds.iter().enumerate() {
std::ptr::write_unaligned(data.add(i), fd.as_raw_fd());
}
}
}
// SAFETY: `sock` is the caller's live socket; `mhdr` points at the live `iov` (over `body`,
// which outlives the call) and — when fds are passed — at `cmsg_store` (ditto). `sendmsg`
// only reads these buffers. The kernel dups the fds into the message; our `BorrowedFd`s stay
// owned by the caller.
let n = unsafe { libc::sendmsg(sock.as_raw_fd(), &mhdr, libc::MSG_NOSIGNAL) };
if n < 0 {
return Err(io::Error::last_os_error());
}
if n as usize != body.len() {
return Err(io::Error::new(
io::ErrorKind::WriteZero,
"short sendmsg on SEQPACKET socket",
));
}
Ok(())
}
/// Receive one message (+ up to one `SCM_RIGHTS` fd) — the single-descriptor fast path over
/// [`recv_fds`]. Any further descriptors the peer attached are dropped, i.e. closed, so a
/// protocol mix-up cannot leak fds into a caller that only knows about one.
pub fn recv<T: DeserializeOwned>(
sock: BorrowedFd,
buf: &mut Vec<u8>,
) -> io::Result<(T, Option<OwnedFd>)> {
let (msg, fds) = recv_fds(sock, buf)?;
Ok((msg, fds.into_iter().next()))
}
/// Receive one message plus its `SCM_RIGHTS` descriptors (up to [`MAX_FDS`], in the order the
/// sender listed them). `buf` is a caller-owned scratch buffer (grown to [`MAX_MSG`] once, then
/// reused message to message); the returned `Vec` does not allocate when no fd arrived, which is
/// the steady state under an fd-identity cache. Errors: `UnexpectedEof` = the peer is gone;
/// `WouldBlock` = the [`set_recv_timeout`] expired; `InvalidData` = a truncated body or more
/// descriptors than [`MAX_FDS`] (the kernel drops the excess and flags `MSG_CTRUNC`).
pub fn recv_fds<T: DeserializeOwned>(
sock: BorrowedFd,
buf: &mut Vec<u8>,
) -> io::Result<(T, Vec<OwnedFd>)> {
buf.resize(MAX_MSG, 0);
let mut iov = libc::iovec {
iov_base: buf.as_mut_ptr() as *mut libc::c_void,
iov_len: buf.len(),
};
let mut cmsg_store: CmsgStore = [0; 8];
// SAFETY: `mhdr` is a plain-old-data C struct for which all-zero is a valid value.
let mut mhdr: libc::msghdr = unsafe { std::mem::zeroed() };
mhdr.msg_iov = &mut iov;
mhdr.msg_iovlen = 1;
mhdr.msg_control = cmsg_store.as_mut_ptr() as *mut libc::c_void;
// Exactly MAX_FDS worth of control space (not the store's full size): a peer that attaches
// more gets its excess dropped by the kernel and the message flagged `MSG_CTRUNC`, which is
// checked below — the cap is enforced by the kernel rather than trusted from the peer.
debug_assert!(cmsg_space(MAX_FDS) <= std::mem::size_of_val(&cmsg_store));
mhdr.msg_controllen = cmsg_space(MAX_FDS) as _;
// SAFETY: `sock` is the caller's live socket. `recvmsg` writes at most `iov_len` bytes into
// `buf` (live for the call) and at most `msg_controllen` control bytes into `cmsg_store`
// (live, 8-aligned, and at least that large — asserted above). `MSG_CMSG_CLOEXEC` makes any
// received fd CLOEXEC atomically.
let n = unsafe { libc::recvmsg(sock.as_raw_fd(), &mut mhdr, libc::MSG_CMSG_CLOEXEC) };
if n < 0 {
return Err(io::Error::last_os_error());
}
if n == 0 {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"worker ipc peer closed",
));
}
// Collect the passed fds (if any) BEFORE any early return below, so they can't leak — an
// error path drops the `Vec`, which closes every one of them.
let mut got: Vec<OwnedFd> = Vec::new();
// SAFETY: `CMSG_FIRSTHDR`/`CMSG_NXTHDR` walk the control area the kernel just wrote inside
// `cmsg_store` (bounded by the updated `mhdr.msg_controllen`), returning either null or a
// pointer to a complete `cmsghdr` within it — each dereference reads kernel-initialized
// fields in bounds. For an `SCM_RIGHTS` cmsg the data area holds whole `RawFd`s, `cmsg_len -
// CMSG_LEN(0)` bytes of them, all inside that same complete cmsg, so every `read_unaligned`
// at index < that count is in bounds. The kernel gave us ownership of each fd (they are fresh
// descriptors in our table), so each `OwnedFd::from_raw_fd` takes sole ownership of a distinct
// descriptor — no alias, no double-close, and nothing dropped on the floor even with multiple
// cmsgs or multiple fds per cmsg.
unsafe {
let mut c = libc::CMSG_FIRSTHDR(&mhdr);
while !c.is_null() {
if (*c).cmsg_level == libc::SOL_SOCKET && (*c).cmsg_type == libc::SCM_RIGHTS {
let payload = ((*c).cmsg_len as usize).saturating_sub(libc::CMSG_LEN(0) as usize);
let data = libc::CMSG_DATA(c) as *const RawFd;
for i in 0..payload / std::mem::size_of::<RawFd>() {
let fd = std::ptr::read_unaligned(data.add(i));
if fd >= 0 {
got.push(OwnedFd::from_raw_fd(fd));
}
}
}
c = libc::CMSG_NXTHDR(&mhdr, c);
}
}
if mhdr.msg_flags & libc::MSG_CTRUNC != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("worker ipc message carried more than {MAX_FDS} descriptors"),
));
}
if mhdr.msg_flags & libc::MSG_TRUNC != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"worker ipc message truncated",
));
}
let msg = serde_json::from_slice(&buf[..n as usize])
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
Ok((msg, got))
}
/// An executable pinned by an open fd, exec'able through [`PinnedExe::exec_path`]. The fd names
/// the file's *inode*, not its path, so it resolves even after the binary was replaced or deleted
/// — and exec'ing it then still runs byte-for-byte the build that was pinned. `current_exe()`
/// instead readlinks to a path: after a package upgrade under a running host that path is
/// `"<path> (deleted)"` and spawning it fails ENOENT — every capture then silently fell back to the
/// CPU copy (the 2026-07-10 canary regression) — and even while the path exists it may hold a
/// newer build whose worker protocol mismatches this process.
pub struct PinnedExe(File);
impl PinnedExe {
/// Pin the executable at `path`.
pub fn open(path: &Path) -> io::Result<PinnedExe> {
let f = File::open(path)?;
if f.as_raw_fd() != 3 {
return Ok(PinnedExe(f));
}
// Fd 3 is the slot the spawn hands the worker its socket on (the `dup2` in
// [`spawn_worker`]) — pinned there, the child would clobber it before exec resolves
// `/proc/self/fd/3`. Re-number: 3 stays occupied by `f` during the clone, so the
// duplicate cannot land on it.
let clone = f.try_clone().map_err(|e| {
io::Error::new(
e.kind(),
format!("re-numbering the pinned exe fd off fd 3 failed: {e}"),
)
})?;
Ok(PinnedExe(clone))
}
/// `/proc/self/fd/<n>` — an exec'able path to the pinned inode. The kernel resolves it at exec
/// time inside the forked child, whose fd table is a copy of ours (close-on-exec applies only
/// once the exec succeeds), so it names the pinned inode no matter what sits at the file's
/// original path by then.
pub fn exec_path(&self) -> PathBuf {
PathBuf::from(format!("/proc/self/fd/{}", self.0.as_raw_fd()))
}
}
/// This process's own executable image, pinned once (lazily) via the `/proc/self/exe` magic link
/// — see [`PinnedExe`] for why the fd and not the path. `None` when it could not be opened, in
/// which case callers fall back to `current_exe()` and inherit that trap.
///
/// Only for a worker that is the same file as its host by construction (the zerocopy worker
/// re-execs this binary). The encode worker must stay a **separate file** — it carries a file
/// capability the host must never have — so it pins its own resolved path with [`PinnedExe::open`].
pub fn self_exe() -> Option<&'static PinnedExe> {
static SELF_EXE: OnceLock<Option<PinnedExe>> = OnceLock::new();
SELF_EXE
.get_or_init(|| match PinnedExe::open(Path::new("/proc/self/exe")) {
Ok(p) => Some(p),
Err(e) => {
tracing::warn!(
error = %e,
"cannot pin /proc/self/exe — worker spawns use the current_exe() path, \
which breaks if this binary is replaced on disk"
);
None
}
})
.as_ref()
}
/// Spawn a worker process on `exe` (normally a [`PinnedExe::exec_path`]) and hand it one end of a
/// fresh SEQPACKET socketpair on **fd 3**; the host end is returned alongside the child.
///
/// `argv0` is what `ps` shows — worth setting, because `exe` is normally an opaque
/// `/proc/self/fd/<n>`. `args` follow it.
pub fn spawn_worker(exe: &Path, argv0: &str, args: &[&str]) -> io::Result<(OwnedFd, Child)> {
sweep_reaper();
let (host_end, worker_end) = socketpair_seqpacket()?;
let mut cmd = Command::new(exe);
cmd.arg0(argv0);
cmd.args(args);
let raw = worker_end.as_raw_fd();
let parent = std::process::id() as libc::pid_t;
// SAFETY: `pre_exec` runs between fork and exec, so only async-signal-safe calls are
// allowed — `prctl`, `getppid`, `dup2` and `fcntl` all are, and the closure captures only
// `Copy` ints (no allocation, no locks; the error paths use `from_raw_os_error`, which
// does not allocate). PR_SET_PDEATHSIG makes the kernel SIGKILL the worker when the host
// dies — without it a crashed host left the worker holding its driver state (order hundreds
// of MB of VRAM) indefinitely. The `getppid` check closes the standard race: if the host died
// between fork and the prctl, the signal is never delivered, so refuse to exec instead.
// `dup2(raw, 3)` installs the socket at the fd number the worker expects and clears CLOEXEC
// on the copy; if the parent's fd already IS 3, `dup2(3,3)` would preserve CLOEXEC, so that
// case clears the flag explicitly instead.
unsafe {
cmd.pre_exec(move || {
if libc::prctl(libc::PR_SET_PDEATHSIG, libc::SIGKILL) != 0 {
return Err(io::Error::last_os_error());
}
if libc::getppid() != parent {
return Err(io::Error::from_raw_os_error(libc::ESRCH));
}
if raw == 3 {
let flags = libc::fcntl(3, libc::F_GETFD);
if flags < 0 || libc::fcntl(3, libc::F_SETFD, flags & !libc::FD_CLOEXEC) < 0 {
return Err(io::Error::last_os_error());
}
} else if libc::dup2(raw, 3) < 0 {
return Err(io::Error::last_os_error());
}
Ok(())
});
}
let child = cmd.spawn()?;
drop(worker_end); // the child holds its own copy now
Ok((host_end, child))
}
/// Children whose worker hasn't exited yet at teardown time (a worker exits on socket EOF, i.e.
/// after the last in-flight frame drops). Swept on every spawn and every drop so workers don't
/// linger as zombies for more than one generation.
static REAPER: Mutex<Vec<(Child, Instant)>> = Mutex::new(Vec::new());
/// How long past the caller's reply timeout a parked worker may linger before it is force-killed.
/// A worker wedged INSIDE a driver call never observes socket EOF, so `try_wait` alone would keep
/// it (and its driver state — order hundreds of MB of VRAM) forever.
const REAPER_KILL_DEADLINE: Duration = Duration::from_secs(20);
/// Hand a still-running worker to the reaper. Call from a `Drop` after the socket is (about to
/// be) closed: the worker sees EOF and exits, and the next [`sweep_reaper`] collects it.
pub fn park_child(child: Child) {
REAPER.lock().unwrap().push((child, Instant::now()));
}
/// Reap exited workers; force-kill the ones parked past the kill deadline (20 s).
pub fn sweep_reaper() {
// Partition under the lock; kill/reap OUTSIDE it. A worker wedged inside a driver ioctl sits
// in D state and ignores SIGKILL — the old blocking `wait()` under the global mutex would
// then park every later spawn and drop behind a process that may never die.
let mut expired: Vec<Child> = Vec::new();
{
let mut list = REAPER.lock().unwrap();
let now = Instant::now();
let mut i = 0;
while i < list.len() {
if matches!(list[i].0.try_wait(), Ok(Some(_))) {
list.swap_remove(i); // exited on its own → reaped
} else if now.duration_since(list[i].1) > REAPER_KILL_DEADLINE {
expired.push(list.swap_remove(i).0);
} else {
i += 1;
}
}
}
for mut c in expired {
let _ = c.kill();
// Bounded reap (~100 ms of polls): a SIGKILL'd process reaps near-instantly unless it is
// in D state — then park it again (re-killing later is harmless) so a future sweep reaps
// it once the driver unwedges, instead of blocking anyone here forever.
let mut reaped = false;
for _ in 0..10 {
if matches!(c.try_wait(), Ok(Some(_))) {
reaped = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
if !reaped {
tracing::warn!(
pid = c.id(),
"worker ignored SIGKILL (likely wedged in a driver call, D state) — \
parked for a later sweep"
);
park_child(c);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use serde::Deserialize;
use std::io::{Read, Write};
use std::os::fd::AsFd;
/// A stand-in payload: the transport is generic over the serde body, and the workers' own
/// message enums live with the workers — nothing here may depend on one.
#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct Msg {
tag: String,
n: u32,
}
fn msg(tag: &str) -> Msg {
Msg {
tag: tag.into(),
n: 7,
}
}
/// A pipe whose read end is passed over the socket, carrying `payload` for the receiver to
/// read back — the only way to prove the *descriptor* crossed, not just the claim that it did.
fn marked_pipe(payload: &[u8]) -> (std::io::PipeReader, std::io::PipeWriter) {
let (pr, mut pw) = std::io::pipe().unwrap();
pw.write_all(payload).unwrap();
(pr, pw)
}
#[test]
fn cmsg_store_is_large_enough() {
assert!(
cmsg_space(MAX_FDS) <= std::mem::size_of::<CmsgStore>(),
"CMSG_SPACE({MAX_FDS} fds) = {} > store {}",
cmsg_space(MAX_FDS),
std::mem::size_of::<CmsgStore>()
);
}
#[test]
fn round_trip_no_fd() {
let (a, b) = socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
send(a.as_fd(), &msg("hello"), None).unwrap();
let (got, fd) = recv::<Msg>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, msg("hello"));
assert!(fd.is_none());
}
#[test]
fn passes_an_fd() {
let (a, b) = socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
// A pipe stands in for a dmabuf: pass the read end, write through the original write end,
// and read the bytes back through the RECEIVED fd.
let (mut pr, mut pw) = std::io::pipe().unwrap();
send(a.as_fd(), &msg("one"), Some(pr.as_fd())).unwrap();
let (got, fd) = recv::<Msg>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, msg("one"));
let fd = fd.expect("fd should have been passed");
pw.write_all(b"hello").unwrap();
drop(pw);
let mut file = File::from(fd);
let mut s = String::new();
file.read_to_string(&mut s).unwrap();
assert_eq!(s, "hello");
// The original read end still works independently of the passed dup.
let mut nothing = [0u8; 1];
assert_eq!(pr.read(&mut nothing).unwrap(), 0);
}
#[test]
fn round_trip_three_fds() {
// The multi-planar case (WP0): one message carrying one fd per plane. Each pipe is
// pre-loaded with a distinct byte, so reading through the RECEIVED descriptors proves
// both that all three crossed and that they arrived in the sender's order.
let (a, b) = socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
let planes = [marked_pipe(b"Y"), marked_pipe(b"U"), marked_pipe(b"V")];
{
let fds: Vec<BorrowedFd> = planes.iter().map(|(pr, _)| pr.as_fd()).collect();
send_fds(a.as_fd(), &msg("planes"), &fds).unwrap();
}
let (got, fds) = recv_fds::<Msg>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, msg("planes"));
assert_eq!(fds.len(), 3, "all three descriptors must cross");
// Drop the write ends so each read sees EOF after its byte.
drop(planes);
let read_back: Vec<String> = fds
.into_iter()
.map(|fd| {
let mut s = String::new();
File::from(fd).read_to_string(&mut s).unwrap();
s
})
.collect();
assert_eq!(read_back, vec!["Y", "U", "V"]);
}
#[test]
fn max_fds_round_trip_and_one_more_is_refused() {
let (a, b) = socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
let pipes: Vec<_> = (0..MAX_FDS + 1).map(|_| std::io::pipe().unwrap()).collect();
let fds: Vec<BorrowedFd> = pipes.iter().map(|(pr, _)| pr.as_fd()).collect();
send_fds(a.as_fd(), &msg("full"), &fds[..MAX_FDS]).unwrap();
let (_, got) = recv_fds::<Msg>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got.len(), MAX_FDS);
// One over the cap is refused at the sender (a caller bug), not panicked on.
let err = send_fds(a.as_fd(), &msg("over"), &fds).unwrap_err();
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
}
#[test]
fn recv_keeps_the_first_fd_and_closes_the_rest() {
// A peer that attaches more descriptors than the single-fd caller expects must not leak
// them: `recv` returns the first and drops (closes) the others.
let (a, b) = socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
let (first, second) = (marked_pipe(b"1"), marked_pipe(b"2"));
send_fds(a.as_fd(), &msg("two"), &[first.0.as_fd(), second.0.as_fd()]).unwrap();
let (_, fd) = recv::<Msg>(b.as_fd(), &mut buf).unwrap();
let fd = fd.expect("the first fd is returned");
drop(first);
let mut s = String::new();
File::from(fd).read_to_string(&mut s).unwrap();
assert_eq!(s, "1");
// The second pipe's receiver-side dup was closed with the returned Vec's tail, so the
// writer sees EPIPE once the local read end goes too.
drop(second.0);
let mut pw = second.1;
assert_eq!(
pw.write_all(b"x").unwrap_err().kind(),
io::ErrorKind::BrokenPipe
);
}
#[test]
fn eof_when_peer_closes() {
let (a, b) = socketpair_seqpacket().unwrap();
drop(a);
let mut buf = Vec::new();
let err = recv::<Msg>(b.as_fd(), &mut buf).unwrap_err();
assert_eq!(err.kind(), io::ErrorKind::UnexpectedEof);
}
#[test]
fn send_to_dead_peer_is_epipe_not_sigpipe() {
let (a, b) = socketpair_seqpacket().unwrap();
drop(b);
let err = send(a.as_fd(), &msg("gone"), None).unwrap_err();
// MSG_NOSIGNAL: a dead peer surfaces as EPIPE (BrokenPipe), never a process-killing signal.
assert_eq!(err.kind(), io::ErrorKind::BrokenPipe);
}
#[test]
fn recv_timeout_fires() {
let (a, _b) = socketpair_seqpacket().unwrap();
set_recv_timeout(a.as_fd(), Some(Duration::from_millis(50))).unwrap();
let mut buf = Vec::new();
let err = recv::<Msg>(a.as_fd(), &mut buf).unwrap_err();
assert!(
matches!(
err.kind(),
io::ErrorKind::WouldBlock | io::ErrorKind::TimedOut
),
"unexpected error kind: {err:?}"
);
}
#[test]
fn pinned_fd_exec_survives_on_disk_replacement() {
// The 2026-07-10 canary regression: a package upgrade replaced the installed binary and
// every worker spawn ENOENT'd (`current_exe()` readlinked to "<path> (deleted)"). The
// pinned-fd mechanism must keep exec'ing the original image after the file is gone: pin
// a copy of /bin/sh, delete it, then run it through the fd path.
let copy = std::env::temp_dir().join(format!("pf-zerocopy-exe-pin-{}", std::process::id()));
std::fs::copy("/bin/sh", &copy).unwrap();
let pinned = PinnedExe::open(&copy).unwrap();
std::fs::remove_file(&copy).unwrap();
// Retry ETXTBSY: `fs::copy`'s write fd leaks into other tests' concurrently-forked
// children until their execs clear it (CLOEXEC applies only at exec), and exec'ing a
// file someone holds open for writing is refused. A harness artifact of copy-then-exec,
// not the mechanism under test — production pins a read-only fd on a binary nobody
// write-opens.
let status = loop {
match Command::new(pinned.exec_path())
.arg("-c")
.arg("exit 42")
.status()
{
Err(e) if e.raw_os_error() == Some(libc::ETXTBSY) => {
std::thread::sleep(Duration::from_millis(10))
}
other => break other.expect("exec via /proc/self/fd of a deleted file"),
}
};
assert_eq!(status.code(), Some(42));
}
#[test]
fn spawn_worker_hands_the_socket_on_fd_3() {
// `sh` echoes back through fd 3 — the inheritance slot every worker reads its socket
// from. Receiving it here proves the dup2 landed and CLOEXEC was cleared on the copy.
let (host, mut child) = spawn_worker(
Path::new("/bin/sh"),
"pf-test-worker",
&["-c", r#"printf '"pong"' >&3"#],
)
.unwrap();
let mut buf = Vec::new();
let (got, fds) = recv_fds::<String>(host.as_fd(), &mut buf).unwrap();
assert_eq!(got, "pong");
assert!(fds.is_empty());
child.wait().unwrap();
}
}
+5
View File
@@ -14,6 +14,11 @@
pub mod client;
pub mod cuda;
pub mod egl;
// Worker-subprocess IPC rails (SEQPACKET framing ± `SCM_RIGHTS`, pinned-exe spawn, reaping),
// generic over the message body so every punktfunk worker shares them — the zerocopy one here,
// the capability-carrying encode worker in `pf-encode`. Vocabulary stays per-worker (`proto` is
// only this one's).
pub mod ipc;
pub mod proto;
pub mod vkslot;
pub mod vulkan;
+16 -263
View File
@@ -1,32 +1,20 @@
//! Wire protocol between the PipeWire capture thread and the isolated zero-copy GPU-import
//! Wire *vocabulary* between the PipeWire capture thread and the isolated zero-copy GPU-import
//! worker process (`punktfunk-host zerocopy-worker`; design:
//! `design/zerocopy-worker-isolation.md`). Transport is a `SOCK_SEQPACKET` unix socketpair —
//! reliable, ordered, message-framed (one `sendmsg` = one message) — with dmabuf fds riding as
//! `SCM_RIGHTS` control data. Bodies are small serde_json blobs (~200 B/frame); pixels never
//! cross the socket (they move GPU-side via CUDA IPC, see [`super::cuda::ipc_export`]).
//! `design/zerocopy-worker-isolation.md`) — the message types and this protocol's version, and
//! nothing else. The transport they ride on ([`super::ipc`]: SEQPACKET framing, `SCM_RIGHTS`,
//! spawn/reap) is shared with the other workers and is deliberately generic over the body type;
//! each worker's vocabulary stays its own and versions independently.
//!
//! Zero-length messages are reserved: `recvmsg` returning 0 on a SEQPACKET socket is EOF (the
//! peer died/closed), and every serialized message here is non-empty JSON, so the two can't be
//! confused.
//! Bodies are small serde_json blobs (~200 B/frame); pixels never cross the socket (they move
//! GPU-side via CUDA IPC, see [`super::cuda::ipc_export`]).
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use serde::de::DeserializeOwned;
use serde::{Deserialize, Serialize};
use std::io;
use std::os::fd::{AsRawFd, BorrowedFd, FromRawFd, OwnedFd};
use std::time::Duration;
/// Bumped on any wire change; the worker echoes it in [`Reply::Ready`] and the host refuses a
/// mismatch. Host and worker are the same binary (`/proc/self/exe`), so this only ever trips on
/// exotic deployment mistakes (a stale binary re-exec'd across an upgrade).
pub const PROTO_VERSION: u32 = 1;
/// Upper bound for one serialized message (the largest real message — a modifier list — is far
/// below this). A message reported truncated at this size is a protocol error.
pub const MAX_MSG: usize = 64 * 1024;
/// How a dmabuf should be imported — mirrors the `EglImporter` entry points.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
pub enum ImportKind {
@@ -118,197 +106,17 @@ pub struct BufferDesc {
pub uv: Option<(Vec<u8>, usize)>,
}
/// A CLOEXEC `SOCK_SEQPACKET` socketpair — `(host_end, worker_end)`.
pub fn socketpair_seqpacket() -> io::Result<(OwnedFd, OwnedFd)> {
let mut fds = [0i32; 2];
// SAFETY: `socketpair` writes two fds into `fds`, a live 2-element stack array matching the
// API contract; it reads no other Rust memory. The result is checked before the fds are used,
// and each returned fd is fresh (owned by no other wrapper), so the two `OwnedFd::from_raw_fd`
// each take sole ownership of a distinct, valid descriptor — no alias, no double-close.
unsafe {
if libc::socketpair(
libc::AF_UNIX,
libc::SOCK_SEQPACKET | libc::SOCK_CLOEXEC,
0,
fds.as_mut_ptr(),
) != 0
{
return Err(io::Error::last_os_error());
}
Ok((OwnedFd::from_raw_fd(fds[0]), OwnedFd::from_raw_fd(fds[1])))
}
}
/// Set (or clear) the receive timeout: a blocked [`recv`] then fails with
/// `ErrorKind::WouldBlock`. Used by the host so a hung worker can't wedge the capture thread.
pub fn set_recv_timeout(sock: BorrowedFd, timeout: Option<Duration>) -> io::Result<()> {
let tv = match timeout {
Some(d) => libc::timeval {
tv_sec: d.as_secs() as libc::time_t,
tv_usec: d.subsec_micros() as libc::suseconds_t,
},
None => libc::timeval {
tv_sec: 0,
tv_usec: 0,
},
};
// SAFETY: `setsockopt(SO_RCVTIMEO)` reads `size_of::<timeval>()` bytes from `&tv`, a live
// stack `timeval` that outlives this synchronous call; `sock` is the caller's live socket fd.
// Nothing is retained or written through Rust pointers.
let r = unsafe {
libc::setsockopt(
sock.as_raw_fd(),
libc::SOL_SOCKET,
libc::SO_RCVTIMEO,
&tv as *const libc::timeval as *const libc::c_void,
std::mem::size_of::<libc::timeval>() as libc::socklen_t,
)
};
if r != 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
/// Send one message (+ optionally one fd as `SCM_RIGHTS`) as a single SEQPACKET datagram.
/// Atomic per message, so concurrent senders on the same socket (the capture thread's imports,
/// the encode thread's releases) need no lock. `MSG_NOSIGNAL` turns a dead peer into `EPIPE`
/// instead of `SIGPIPE`.
pub fn send<T: Serialize>(
sock: BorrowedFd,
msg: &T,
pass_fd: Option<BorrowedFd>,
) -> io::Result<()> {
let body =
serde_json::to_vec(msg).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
debug_assert!(
!body.is_empty(),
"zero-length messages are reserved for EOF"
);
if body.len() > MAX_MSG {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"zerocopy proto message too large",
));
}
let mut iov = libc::iovec {
iov_base: body.as_ptr() as *mut libc::c_void,
iov_len: body.len(),
};
// Control buffer for one fd: CMSG_SPACE(4) = 24 bytes on 64-bit; [u64; 4] gives 32 bytes at
// the 8-byte alignment `cmsghdr` requires.
let mut cmsg_store = [0u64; 4];
// SAFETY: `mhdr` is a plain-old-data C struct for which all-zero is a valid value.
let mut mhdr: libc::msghdr = unsafe { std::mem::zeroed() };
mhdr.msg_iov = &mut iov;
mhdr.msg_iovlen = 1;
if let Some(fd) = pass_fd {
mhdr.msg_control = cmsg_store.as_mut_ptr() as *mut libc::c_void;
// SAFETY: `CMSG_SPACE`/`CMSG_LEN` are pure size computations (no memory access).
// `CMSG_FIRSTHDR(&mhdr)` returns a pointer into `cmsg_store` (non-null: msg_controllen
// ≥ one cmsghdr), which is live, 8-aligned, and large enough (32 ≥ CMSG_SPACE(4) = 24)
// for the header fields and the 4-byte fd written via `CMSG_DATA`; `write_unaligned`
// handles the data area's byte alignment. All writes stay within `cmsg_store`, which
// outlives the synchronous `sendmsg` below.
unsafe {
mhdr.msg_controllen = libc::CMSG_SPACE(4) as _;
let c = libc::CMSG_FIRSTHDR(&mhdr);
(*c).cmsg_level = libc::SOL_SOCKET;
(*c).cmsg_type = libc::SCM_RIGHTS;
(*c).cmsg_len = libc::CMSG_LEN(4) as _;
std::ptr::write_unaligned(libc::CMSG_DATA(c) as *mut i32, fd.as_raw_fd());
}
}
// SAFETY: `sock` is the caller's live socket; `mhdr` points at the live `iov` (over `body`,
// which outlives the call) and — when an fd is passed — at `cmsg_store` (ditto). `sendmsg`
// only reads these buffers. The kernel dups the fd into the message; our `BorrowedFd` stays
// owned by the caller.
let n = unsafe { libc::sendmsg(sock.as_raw_fd(), &mhdr, libc::MSG_NOSIGNAL) };
if n < 0 {
return Err(io::Error::last_os_error());
}
if n as usize != body.len() {
return Err(io::Error::new(
io::ErrorKind::WriteZero,
"short sendmsg on SEQPACKET socket",
));
}
Ok(())
}
/// Receive one message (+ up to one `SCM_RIGHTS` fd). `buf` is a caller-owned scratch buffer
/// (grown to [`MAX_MSG`] once, then reused frame to frame). Errors:
/// `UnexpectedEof` = the peer is gone; `WouldBlock` = the [`set_recv_timeout`] expired.
pub fn recv<T: DeserializeOwned>(
sock: BorrowedFd,
buf: &mut Vec<u8>,
) -> io::Result<(T, Option<OwnedFd>)> {
buf.resize(MAX_MSG, 0);
let mut iov = libc::iovec {
iov_base: buf.as_mut_ptr() as *mut libc::c_void,
iov_len: buf.len(),
};
let mut cmsg_store = [0u64; 4];
// SAFETY: `mhdr` is a plain-old-data C struct for which all-zero is a valid value.
let mut mhdr: libc::msghdr = unsafe { std::mem::zeroed() };
mhdr.msg_iov = &mut iov;
mhdr.msg_iovlen = 1;
mhdr.msg_control = cmsg_store.as_mut_ptr() as *mut libc::c_void;
mhdr.msg_controllen = std::mem::size_of_val(&cmsg_store) as _;
// SAFETY: `sock` is the caller's live socket. `recvmsg` writes at most `iov_len` bytes into
// `buf` (live for the call) and at most `msg_controllen` control bytes into `cmsg_store`
// (live, 8-aligned). `MSG_CMSG_CLOEXEC` makes any received fd CLOEXEC atomically.
let n = unsafe { libc::recvmsg(sock.as_raw_fd(), &mut mhdr, libc::MSG_CMSG_CLOEXEC) };
if n < 0 {
return Err(io::Error::last_os_error());
}
if n == 0 {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"zerocopy proto peer closed",
));
}
// Collect a passed fd (if any) BEFORE any early return below, so it can't leak.
let mut got_fd: Option<OwnedFd> = None;
// SAFETY: `CMSG_FIRSTHDR`/`CMSG_NXTHDR` walk the control area the kernel just wrote inside
// `cmsg_store` (bounded by the updated `mhdr.msg_controllen`), returning either null or a
// pointer to a complete `cmsghdr` within it — each dereference reads kernel-initialized
// fields in bounds. For an `SCM_RIGHTS` cmsg the data area holds whole `i32` fds; we read the
// first via `read_unaligned`. The kernel gave us ownership of that fd (it is a fresh
// descriptor in our table), so `OwnedFd::from_raw_fd` takes sole ownership — any previously
// collected `got_fd` is dropped (closed) first, so nothing leaks even with multiple cmsgs.
unsafe {
let mut c = libc::CMSG_FIRSTHDR(&mhdr);
while !c.is_null() {
if (*c).cmsg_level == libc::SOL_SOCKET && (*c).cmsg_type == libc::SCM_RIGHTS {
let fd = std::ptr::read_unaligned(libc::CMSG_DATA(c) as *const i32);
if fd >= 0 {
got_fd = Some(OwnedFd::from_raw_fd(fd));
}
}
c = libc::CMSG_NXTHDR(&mhdr, c);
}
}
if mhdr.msg_flags & libc::MSG_TRUNC != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"zerocopy proto message truncated",
));
}
let msg = serde_json::from_slice(&buf[..n as usize])
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
Ok((msg, got_fd))
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::{Read, Write};
use crate::imp::ipc;
use std::os::fd::AsFd;
/// The vocabulary survives the wire in both directions. (The framing itself — fds, EOF,
/// timeouts, the descriptor cap — is exercised in [`ipc`]; this only pins the message types.)
#[test]
fn round_trip_no_fd() {
let (a, b) = socketpair_seqpacket().unwrap();
fn round_trip_both_directions() {
let (a, b) = ipc::socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
let req = Request::Import {
key: 0xdead_beef_u64,
@@ -321,8 +129,8 @@ mod tests {
stride: 5120 * 4,
has_fd: false,
};
send(a.as_fd(), &req, None).unwrap();
let (got, fd) = recv::<Request>(b.as_fd(), &mut buf).unwrap();
ipc::send(a.as_fd(), &req, None).unwrap();
let (got, fd) = ipc::recv::<Request>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, req);
assert!(fd.is_none());
@@ -336,64 +144,9 @@ mod tests {
uv: Some((vec![2u8; 64], 5632)),
}),
};
send(b.as_fd(), &reply, None).unwrap();
let (got, fd) = recv::<Reply>(a.as_fd(), &mut buf).unwrap();
ipc::send(b.as_fd(), &reply, None).unwrap();
let (got, fd) = ipc::recv::<Reply>(a.as_fd(), &mut buf).unwrap();
assert_eq!(got, reply);
assert!(fd.is_none());
}
#[test]
fn passes_an_fd() {
let (a, b) = socketpair_seqpacket().unwrap();
let mut buf = Vec::new();
// A pipe stands in for a dmabuf: pass the read end, write through the original write end,
// and read the bytes back through the RECEIVED fd.
let (mut pr, mut pw) = std::io::pipe().unwrap();
send(a.as_fd(), &Request::ClearCache, Some(pr.as_fd())).unwrap();
let (got, fd) = recv::<Request>(b.as_fd(), &mut buf).unwrap();
assert_eq!(got, Request::ClearCache);
let fd = fd.expect("fd should have been passed");
pw.write_all(b"hello").unwrap();
drop(pw);
let mut file = std::fs::File::from(fd);
let mut s = String::new();
file.read_to_string(&mut s).unwrap();
assert_eq!(s, "hello");
// The original read end still works independently of the passed dup.
let mut nothing = [0u8; 1];
assert_eq!(pr.read(&mut nothing).unwrap(), 0);
}
#[test]
fn eof_when_peer_closes() {
let (a, b) = socketpair_seqpacket().unwrap();
drop(a);
let mut buf = Vec::new();
let err = recv::<Reply>(b.as_fd(), &mut buf).unwrap_err();
assert_eq!(err.kind(), io::ErrorKind::UnexpectedEof);
}
#[test]
fn send_to_dead_peer_is_epipe_not_sigpipe() {
let (a, b) = socketpair_seqpacket().unwrap();
drop(b);
let err = send(a.as_fd(), &Request::ClearCache, None).unwrap_err();
// MSG_NOSIGNAL: a dead peer surfaces as EPIPE (BrokenPipe), never a process-killing signal.
assert_eq!(err.kind(), io::ErrorKind::BrokenPipe);
}
#[test]
fn recv_timeout_fires() {
let (a, _b) = socketpair_seqpacket().unwrap();
set_recv_timeout(a.as_fd(), Some(Duration::from_millis(50))).unwrap();
let mut buf = Vec::new();
let err = recv::<Reply>(a.as_fd(), &mut buf).unwrap_err();
assert!(
matches!(
err.kind(),
io::ErrorKind::WouldBlock | io::ErrorKind::TimedOut
),
"unexpected error kind: {err:?}"
);
}
}
+23 -22
View File
@@ -14,7 +14,8 @@
use super::cuda::{self, CUdeviceptr, DeviceBuffer};
use super::egl::{DmabufPlane, EglImporter};
use super::proto::{self, BufferDesc, ImportKind, Reply, Request};
use super::ipc;
use super::proto::{BufferDesc, ImportKind, Reply, Request, PROTO_VERSION};
use anyhow::{bail, Context, Result};
use std::collections::{HashMap, VecDeque};
use std::io;
@@ -27,7 +28,7 @@ const FD_CACHE_CAP: usize = 64;
/// Entry point for the hidden `zerocopy-worker` subcommand. `args` are the subcommand's own
/// arguments (`--fd N`, default 3 — the socket end the spawning host `dup2`'d in).
pub fn run_from_args(args: &[String]) -> Result<()> {
// The host execs this worker through its pinned exe fd (`client::self_exe`), so the kernel
// The host execs this worker through its pinned exe fd (`ipc::self_exe`), so the kernel
// derives our comm from the exec path's basename — a meaningless fd number. Rename so
// `top`/`pkill` see the worker.
// SAFETY: `PR_SET_NAME` copies at most 16 bytes from the given pointer; the C-string literal
@@ -72,7 +73,7 @@ fn run(sock: OwnedFd) -> Result<()> {
Err(e) => {
// Init failure is an ANSWER, not a crash: the host falls back to the CPU path,
// exactly like an in-process `EglImporter::new()` failure.
let _ = proto::send(
let _ = ipc::send(
sock.as_fd(),
&Reply::InitErr {
message: format!("{e:#}"),
@@ -82,10 +83,10 @@ fn run(sock: OwnedFd) -> Result<()> {
return Ok(());
}
};
proto::send(
ipc::send(
sock.as_fd(),
&Reply::Ready {
version: proto::PROTO_VERSION,
version: PROTO_VERSION,
},
None,
)
@@ -124,7 +125,7 @@ pub(crate) struct ImportReq {
pub(crate) fn serve(sock: &OwnedFd, backend: &mut dyn ImportBackend) -> Result<()> {
let mut buf = Vec::new();
loop {
let (req, fd) = match proto::recv::<Request>(sock.as_fd(), &mut buf) {
let (req, fd) = match ipc::recv::<Request>(sock.as_fd(), &mut buf) {
Ok(v) => v,
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => return Ok(()),
Err(e) => return Err(e).context("worker recv"),
@@ -173,7 +174,7 @@ pub(crate) fn serve(sock: &OwnedFd, backend: &mut dyn ImportBackend) -> Result<(
/// Send a reply; `Ok(true)` means the host is gone (EPIPE) and the loop should end quietly.
fn send_or_eof(sock: &OwnedFd, reply: &Reply) -> Result<bool> {
match proto::send(sock.as_fd(), reply, None) {
match ipc::send(sock.as_fd(), reply, None) {
Ok(()) => Ok(false),
Err(e) if e.kind() == io::ErrorKind::BrokenPipe => Ok(true),
Err(e) => Err(e).context("worker send"),
@@ -434,7 +435,7 @@ mod tests {
mpsc::Receiver<String>,
std::thread::JoinHandle<Result<()>>,
) {
let (host, worker) = proto::socketpair_seqpacket().unwrap();
let (host, worker) = ipc::socketpair_seqpacket().unwrap();
let (tx, rx) = mpsc::channel();
let join = std::thread::spawn(move || {
let mut backend = MockBackend { calls: tx, next: 0 };
@@ -462,8 +463,8 @@ mod tests {
let (host, rx, join) = start_server();
let mut buf = Vec::new();
proto::send(host.as_fd(), &Request::Modifiers { fourcc: 42 }, None).unwrap();
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
ipc::send(host.as_fd(), &Request::Modifiers { fourcc: 42 }, None).unwrap();
let (reply, _) = ipc::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
assert_eq!(
reply,
Reply::Modifiers {
@@ -472,8 +473,8 @@ mod tests {
);
// First import delivers the desc; the second (same mock id sequence continues) doesn't.
proto::send(host.as_fd(), &import_req(1, false), None).unwrap();
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
ipc::send(host.as_fd(), &import_req(1, false), None).unwrap();
let (reply, _) = ipc::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
match reply {
Reply::Frame {
id: 0,
@@ -481,8 +482,8 @@ mod tests {
} => {}
other => panic!("unexpected reply {other:?}"),
}
proto::send(host.as_fd(), &import_req(1, false), None).unwrap();
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
ipc::send(host.as_fd(), &import_req(1, false), None).unwrap();
let (reply, _) = ipc::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
assert_eq!(reply, Reply::Frame { id: 1, desc: None });
// The descriptor itself must cross the socket: an import WITH an fd rides SCM_RIGHTS and
@@ -490,26 +491,26 @@ mod tests {
// received fd (e.g. `backend.import(&req, None)`) would only be caught here.
let (pr, _pw) = std::io::pipe().unwrap();
let sent_ino = fd_ino(pr.as_fd().as_raw_fd());
proto::send(host.as_fd(), &import_req(3, true), Some(pr.as_fd())).unwrap();
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
ipc::send(host.as_fd(), &import_req(3, true), Some(pr.as_fd())).unwrap();
let (reply, _) = ipc::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
assert_eq!(reply, Reply::Frame { id: 2, desc: None });
// A missing worker-side fd is a NeedFd reply (host resends), not a failure.
proto::send(host.as_fd(), &import_req(0xfeed, false), None).unwrap();
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
ipc::send(host.as_fd(), &import_req(0xfeed, false), None).unwrap();
let (reply, _) = ipc::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
assert_eq!(reply, Reply::NeedFd);
// A failed import is an Err reply, not a dead worker.
proto::send(host.as_fd(), &import_req(0xbad, false), None).unwrap();
let (reply, _) = proto::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
ipc::send(host.as_fd(), &import_req(0xbad, false), None).unwrap();
let (reply, _) = ipc::recv::<Reply>(host.as_fd(), &mut buf).unwrap();
match reply {
Reply::Err { message } => assert!(message.contains("scripted failure")),
other => panic!("unexpected reply {other:?}"),
}
// Fire-and-forget ops reach the backend without replies.
proto::send(host.as_fd(), &Request::Release { id: 0 }, None).unwrap();
proto::send(host.as_fd(), &Request::ClearCache, None).unwrap();
ipc::send(host.as_fd(), &Request::Release { id: 0 }, None).unwrap();
ipc::send(host.as_fd(), &Request::ClearCache, None).unwrap();
// Closing the host end terminates serve() cleanly.
drop(host);
+39
View File
@@ -0,0 +1,39 @@
# The capability-carrying PyroWave encode worker (design/gpu-priority-capability-worker.md).
#
# 🛑 This is a SEPARATE BINARY on purpose, and it must stay one. It is the only Punktfunk file that
# may carry `cap_sys_nice=ep`; `punktfunk-host` carries no capability on any channel, ever, because
# a capped process cannot be identified by KWin (`cap_ptrace_access_check` refuses
# `/proc/<pid>/exe` to a reader whose effective set is not a superset of the target's PERMITTED
# set) and therefore never gets `zkde_screencast_unstable_v1` — that was 0.26.0-1, and it killed
# every KDE desktop session. A hardlink to the host, or a hidden host subcommand, shares the inode
# and so shares the capability: same incident, silently. Never do either.
#
# All the logic lives in `pf-encode::worker`; this crate is the file, not the code.
[package]
name = "punktfunk-encode-worker"
version.workspace = true
edition = "2021"
rust-version.workspace = true
license = "MIT OR Apache-2.0"
description = "punktfunk PyroWave encode worker: owns the priority-elevated Vulkan device so the host never carries a capability."
publish = false
[[bin]]
name = "punktfunk-encode-worker"
path = "src/main.rs"
[dependencies]
# `features = ["pyrowave"]` unconditionally rather than through a feature of this crate: packaging
# builds this with `cargo build -p punktfunk-encode-worker` and default features, and a worker
# built without the codec would be a binary that hands every session straight back to the
# in-process fallback while still carrying the capability. (No extra cost to the workspace:
# punktfunk-host already has `default = ["pyrowave"]`, so a workspace build resolves it anyway.)
pf-encode = { path = "../pf-encode", features = ["pyrowave"] }
tracing = "0.1"
# The worker's stderr is inherited from the host that spawned it, so its lines land in the same
# journal — but it is a fresh process with no subscriber of its own, and without one the
# device-pick and priority-ladder lines would go nowhere.
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
[lints]
workspace = true
@@ -0,0 +1,43 @@
//! `punktfunk-encode-worker` — spawned by `punktfunk-host` for the duration of one PyroWave
//! session, never run by hand. It reads its socket from the inherited fd 3 and speaks only to the
//! parent that spawned it: no Wayland, no D-Bus, no network, no plugins.
//!
//! Everything it does lives in [`pf_encode::worker`]; this file exists so the capability has a
//! **file of its own** (see this crate's Cargo.toml for why that is not negotiable).
fn main() -> std::process::ExitCode {
// Stderr, inherited from the host, so the worker's lines land in the host's journal next to
// the session that spawned it. `RUST_LOG` is inherited too, so raising the host's level
// raises the worker's.
let filter =
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into());
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_writer(std::io::stderr)
.init();
#[cfg(target_os = "linux")]
{
let args: Vec<String> = std::env::args().skip(1).collect();
match pf_encode::worker::run_from_args(&args) {
Ok(()) => std::process::ExitCode::SUCCESS,
Err(e) => {
// The host reads the dead socket long before it could read this, so the message is
// for a human running `journalctl` — say enough to place the failure.
tracing::error!(error = %format!("{e:#}"), "punktfunk-encode-worker exiting");
std::process::ExitCode::FAILURE
}
}
}
// Linux-only by construction: the worker exists for `VK_KHR_global_priority` under
// `CAP_SYS_NICE`, and the Windows host raises its GPU scheduling priority through WDDM
// instead (`D3DKMTSetProcessSchedulingPriorityClass`). Packaging never installs this
// elsewhere; the arm exists so a workspace build stays green on every platform.
#[cfg(not(target_os = "linux"))]
{
tracing::error!(
"punktfunk-encode-worker is a Linux-only helper and has nothing to do here"
);
std::process::ExitCode::FAILURE
}
}
+14 -3
View File
@@ -1899,10 +1899,21 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
// a stats capture armed after a resize registers the real mode. Seeded with the refresh the
// initial build actually achieved (`interval_hz`), not the request — KWin may cap a virtual
// output at 60 Hz.
// …and seeded with the SIZE the build actually delivered too, not just the refresh. Attach is
// what made the asymmetry visible: `gamescope: box drives a physical display — attaching at its
// own mode (no re-mode)` streams the panel's size, so a client that asked for 5120x1440 is
// really watching 1920x1080 while `/api/v1/local/summary` echoed the request straight back and
// the console reported a resolution nobody was looking at. The rebuild path below already
// publishes `delivered_mode`; bring-up simply never did, so only a mid-stream resize ever
// corrected it. This publishes the stats slot ONLY — it deliberately does not send the client a
// corrective `Reconfigured`, which stays owed exactly where it already was (`adopted_at_bringup`
// above): an ordinary connect's mode came from the Welcome, not from an accept the client has
// acted on, so it is not owed one.
let delivered = delivered_mode(frame.width, frame.height, interval);
let live_mode = Arc::new(AtomicU64::new(pack_mode(
mode.width,
mode.height,
interval_hz(interval),
delivered.width,
delivered.height,
delivered.refresh_hz,
)));
// One-shot force-keyframe flag driven by the management API (`POST /session/idr`, the web-console
// Dashboard's "Request IDR" button) — drained in the encode loop below exactly like a client
+2 -1
View File
@@ -241,7 +241,8 @@ notes for context.
| `PUNKTFUNK_NVENC_SPLIT_ARBITRATE` | `1` | Opt-in: let the host change its split-encode decision **live**, mid-session, as the pixel rate moves, instead of only choosing once at session start. Currently wired on the Linux direct-NVENC path. Only interesting alongside `PUNKTFUNK_SPLIT_ENCODE=auto` at very high pixel rates. |
| `PUNKTFUNK_GPU_PRIORITY_CLASS` | `off` · `normal` · `high` · `realtime` · `auto` | **(Windows)** GPU scheduling priority for capture/encode under a GPU-saturating game. Default `auto` (starts `high`, upgrades to `realtime` when it's safe — e.g. HAGS off); `high` pins the static pre-gate behaviour; `realtime` is the strongest lever but can freeze NVENC on some setups. |
| `PUNKTFUNK_IDD_DEPTH` | `N` (default `2`) | **(Windows)** IDD-push pipeline depth. `1` cuts latency once GPU priority is raised; higher smooths a contended GPU. |
| `PYROWAVE_QUEUE_PRIORITY` | `realtime` *(default)* · `high` · `off` | [PyroWave](/docs/pyrowave) sessions only. PyroWave encodes on the same GPU shader cores a game uses, so a demanding game can starve it and the frame rate drops. This asks the driver to schedule the encode ahead of the game. `realtime` tries the strongest class and falls back to `high`; `high` asks only for the middle one; `off` disables the request. A driver that refuses simply encodes at normal priority — it can never stop a session starting. This needs the `CAP_SYS_NICE` capability, which the packages deliberately do **not** grant: a host holding a capability cannot be identified by KWin and loses desktop streaming entirely (see [Running as a service](/docs/running-as-a-service#gpu-scheduling-priority)). The request is therefore refused on a stock install and the host says so once at session start. Set `off` if you see the desktop stutter while streaming. |
| `PYROWAVE_QUEUE_PRIORITY` | `realtime` *(default)* · `high` · `off` | [PyroWave](/docs/pyrowave) sessions only — the *intent*, forwarded to whichever process does the encode. PyroWave encodes on the same GPU shader cores a game uses, so a demanding game can starve it and the frame rate drops. This asks the driver to schedule the encode ahead of the game. `realtime` tries the strongest class and falls back to `high`; `high` asks only for the middle one; `off` disables the request. A driver that refuses simply encodes at normal priority — it can never stop a session starting. Granting the request needs the `CAP_SYS_NICE` capability, which the Linux packages give to `punktfunk-encode-worker` and **never** to `punktfunk-host` a host holding any capability cannot be identified by KWin and loses desktop streaming entirely. Do not `setcap` the host to "make this work"; see [Running as a service](/docs/running-as-a-service#gpu-scheduling-priority). Set `off` if you see the desktop stutter while streaming. |
| `PUNKTFUNK_ENCODE_WORKER` | path · `off` | Where the host looks for `punktfunk-encode-worker`, the small capability-carrying helper that owns the priority-elevated [PyroWave](/docs/pyrowave) encode (previous row). Unset, the host looks beside its own binary and then on `PATH`, which is right for every package — set it only when the worker lives somewhere unusual. **NixOS needs it and the module sets it for you:** a file capability cannot live on a read-only nix store path, so the worker is exposed through `security.wrappers` and this points the host at that wrapper. `off` forces the encode back into the host process at default priority — a debug escape hatch, not a tuning knob. Every failure short of that is already handled: a missing binary, a worker that will not start, or one that dies mid-session falls back to encoding in-process with one line in the log, and never drops the session. |
## Diagnostics
+4
View File
@@ -140,6 +140,10 @@ The session unit brings up headless KWin; the host unit follows it and starts li
inside the session. It exits 0 only when KWin is up *and* advertising the grant to the host, and
prints the reason when it isn't — after a fresh install that is usually just the missing log out
and back in. On the headless appliance session, also check `kwin_wayland --version` is ≥ 6.5.6.
The other cause has nothing to do with the `.desktop` file: `getcap /usr/bin/punktfunk-host` must
print **nothing at all**. A host binary carrying a Linux capability cannot be identified by KWin
and is never offered the protocol, however correctly its grant is installed — see
[GPU scheduling priority](/docs/running-as-a-service#gpu-scheduling-priority).
- **Black screen / no picture:** confirm you're on a Wayland session (not X11) and, on NVIDIA, that
the GL userspace is installed. More in [Troubleshooting](/docs/troubleshooting).
+43 -23
View File
@@ -207,41 +207,61 @@ a Windows host, run `punktfunk-host service status` from an elevated prompt on t
## GPU scheduling priority
The host binary carries **no Linux capability**, and on a KDE desktop it must not.
The [PyroWave](/docs/pyrowave) codec encodes on the same GPU shader cores your game is using, so a
demanding game can crowd it out and the stream's frame rate drops with it. The fix is to ask the
driver to schedule the encode ahead of the game, and every driver we tested gates that request on
`CAP_SYS_NICE`. Version 0.26.0-1 granted it for that reason — and it broke desktop streaming on
every KDE box, so 0.26.0-2 takes it away again. The other codecs use a separate video engine on the
GPU and were never affected.
driver to schedule that encode ahead of the game, and every driver we tested gates the request on a
single Linux capability, `CAP_SYS_NICE`. The other codecs use a separate video engine on the GPU and
are unaffected either way.
The two cannot coexist. To hand the host its virtual display, KWin first has to work out *which*
program is asking, which it does by reading the connecting process's `/proc/<pid>/exe` and matching
it against the `.desktop` file the packages install. Linux refuses that read for any process holding
a capability the reader does not also hold — and KWin holds none. So a host with `CAP_SYS_NICE` is a
host KWin cannot identify, and every session fails with:
That capability cannot live on the host, so it lives next to it. Every way of installing a Linux
host — apt, dnf, pacman, the Bazzite sysext, the NixOS module, the Steam Deck script — ships a
second, deliberately small program alongside it, **`punktfunk-encode-worker`**, and grants
`cap_sys_nice=ep` to *that*. The host starts one per PyroWave session, hands it the captured frames,
and takes the compressed video back; the worker talks to nothing else, not your desktop and not the
network.
`punktfunk-host` itself carries **no capability, on any channel** — the posture it has had since
0.25.0, and the one KDE needs.
> **Never `setcap` `punktfunk-host`.** Not by hand, not through a systemd `AmbientCapabilities=`
> line, not through a NixOS `security.wrappers` entry. All three put the capability in the same
> place, and all three take KDE desktop streaming away completely. There is no capability the host
> wants: the worker is the thing that needs one, and your packages already gave it one.
Here is why a privileged host is so much worse than a privileged worker. To hand the host its
virtual display, KWin first has to work out *which* program is asking, which it does by reading the
connecting process's `/proc/<pid>/exe` and matching it against the `.desktop` file the packages
install. Linux refuses that read unless the reader already holds every capability the target holds —
and KWin holds none. So a host carrying a capability is a host KWin cannot identify, its restricted
protocols are never offered, and every session fails at capture with:
```
KWin virtual output failed: KWin does not expose zkde_screencast_unstable_v1 to this client
```
which looks exactly like a missing `.desktop` file and cannot be fixed by reinstalling. Moving the
grant into the systemd unit does not help either — same capability, same refused read.
If you are on 0.26.0-1, update. On the Bazzite image the `/usr` is read-only, so the only repair is
the next image (`sudo punktfunk-sysext update`). Elsewhere you can clear it by hand:
which reads exactly like a missing or mis-installed `.desktop` file and survives reinstalling both
ends. Version 0.26.0-1 granted the host the capability for the reason above and shipped precisely
this, on every Linux channel; 0.26.0-2 revoked it everywhere. If you ever see that error, check the
binaries before anything else — the host's own message names the capability when it finds one:
```sh
getcap /usr/bin/punktfunk-host # prints nothing when correct
sudo setcap -r /usr/bin/punktfunk-host # clear it, then restart the host
getcap /usr/bin/punktfunk-host # correct output is nothing at all
getcap /usr/bin/punktfunk-encode-worker # /usr/bin/punktfunk-encode-worker cap_sys_nice=ep
sudo setcap -r /usr/bin/punktfunk-host # clear it, then restart the host
```
Losing the capability costs frame pacing under a GPU-bound game and nothing else — the host asks for
the elevated priority, is refused, and encodes at the normal one. `PYROWAVE_QUEUE_PRIORITY=off`
stops it asking at all. If you stream only with gamescope (Steam Gaming Mode) you can grant the
capability yourself and keep the pacing, at the cost of desktop streaming; gamescope has no such
identity check.
On the Bazzite image `/usr` is read-only, so there is nothing to repair in place — take the next
image (`sudo punktfunk-sysext update`). On NixOS the worker *is* wrapped, because a file capability
cannot live on a read-only store path: the module creates the wrapper and points the host at it, and
the host's own `ExecStart` stays on the plain store path.
**The grant is best-effort, and no session depends on it.** A worker without the capability still
encodes — it asks for the elevated priority, is refused, says so once, and runs at the normal one.
So does a host that cannot find or start a worker at all: it encodes in-process, logs one line, and
streams. The only thing at stake is frame pacing under a GPU-bound game.
`PYROWAVE_QUEUE_PRIORITY=off` stops the host asking for priority, and `PUNKTFUNK_ENCODE_WORKER=off`
keeps the encode in the host process — both on
[Configuration](/docs/configuration#advanced-performance-tuning).
## Stopping and removing
+28
View File
@@ -0,0 +1,28 @@
<!--
Accumulating ahead of the version bump, per docs/releases/README.md: entries are added as work
lands, and the lead-in, the TL;DR and any "Before you update" section are written at the bump,
when the shape of the whole release is known. The angle-bracket lines below are the TEMPLATE.md
placeholders and must be replaced before the tag is pushed. docs/releases/whatsnew/v0.27.0.txt is
still owed and is a hard gate on the android job.
-->
Wire-compatible with 0.26.x — everything you have already paired keeps working, and you can update one side at a time. <one sentence on how older/newer clients negotiate or fall back, so nobody fears updating.>
<Lead-in: one or two sentences naming the headline change of this release — everything above the first `##` is what the Discord #releases embed shows, so make it read as a standalone summary.>
## TL;DR
- **<The one thing you would tell someone in a sentence.>**
- **<Second. Anything that needs the reader to ACT goes here too, and in `## Before you update`.>**
- **<Third. Three to six bullets, one line each. This is what a reader sees instead of giving up.>**
## Improved
- **The wavelet codec can ask your graphics card for priority again — on every desktop this time.** Under a demanding game it encodes on the same graphics cores the game is using and gets crowded out, and asking the driver to put that work first is worth about a third off the worst frames. It needs a system privilege, and 0.26 could not use it: an interim build gave that privilege to the host, KDE stopped recognising the host as a result, and desktop streaming failed outright — so the privilege was taken away everywhere and the switch sat dormant. The encoding now happens in a small separate program installed alongside the host, and the privilege is granted to **that** program rather than to the host. The host still holds nothing at all, so KDE keeps recognising it, and the priority request works on KDE, GNOME, gamescope and everything else alike. Nothing to switch on. There is also nothing to grant by hand — in particular, do not give the host that privilege yourself, which is still the one thing that takes KDE desktop streaming away. Linux hosts, wavelet-codec sessions only.
- **A host that cannot start that helper simply carries on.** If the program is missing, refuses to start, is a mismatched version, or stops mid-session, the host does the encoding itself exactly as it did before, notes one line in the log, and the stream keeps running. The same is true if the privilege was never granted: the request is refused, the encoding runs at normal priority, and the only cost is frame pacing under a game that is already saturating the graphics card.
## For developers
Protocol, ABI, driver and add-on detail — including the version table and any breaking changes — is in [CHANGELOG.md](https://git.unom.io/unom/punktfunk/src/tag/v0.27.0/CHANGELOG.md).
If you package Punktfunk, one thing changes shape: there is a second installed binary, `punktfunk-encode-worker`, and it is the **only** file that may carry `cap_sys_nice=ep`. It must stay a separate file — never a hardlink to `punktfunk-host`, never a subcommand of it — because a shared inode shares the file capability and re-creates the KDE breakage. `punktfunk-host` carries no capability, on any channel, ever.
+13 -1
View File
@@ -106,8 +106,16 @@ build() {
cargo build --release --locked -p punktfunk-client-linux -p punktfunk-client-session \
-p punktfunk-cli -p pf-update
else
# `-p punktfunk-encode-worker`: the capability-carrying PyroWave encode worker, shipped next to
# the host in /usr/bin and setcap'd by punktfunk-host.install. It MUST be its own file — the host
# can never carry a capability (KWin identification; see the scriptlet) — and it must ship in the
# SAME package as the host, because host and worker version-check each other over their socket
# and fall back to the in-process encoder on any mismatch. Co-built here on purpose: v1 accepts
# that the worker links the same FFmpeg the host does (same package, same sonames, no new break
# class), so cargo's feature unification across this one invocation is harmless.
cargo build --release --locked --features punktfunk-host/nvenc,punktfunk-host/vulkan-encode \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-cli \
-p punktfunk-host -p punktfunk-encode-worker \
-p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-cli \
-p pf-update
# The status tray in its OWN cargo invocation — load-bearing, not tidiness. Cargo unifies features
# across everything in one build, so co-building the tray with the host pulls the host's
@@ -183,6 +191,10 @@ package_punktfunk-host() {
local R; R="$(_repo)"; local T="$srcdir/target/release"
install -Dm0755 "$T/punktfunk-host" "$pkgdir/usr/bin/punktfunk-host"
# The PyroWave encode worker — a SEPARATE file in the same bindir (the host resolves it as a
# sibling of /proc/self/exe). punktfunk-host.install setcaps this one, and only this one; the
# host must stay capability-free or KWin cannot identify it and desktop streaming dies.
install -Dm0755 "$T/punktfunk-encode-worker" "$pkgdir/usr/bin/punktfunk-encode-worker"
# /dev/uinput + /dev/uhid -> input group (virtual gamepads + DualSense UHID)
install -Dm0644 "$R/scripts/60-punktfunk.rules" "$pkgdir/usr/lib/udev/rules.d/60-punktfunk.rules"
# Managed gamescope takeover on DM-autologin boxes: root helper + polkit action so the host can
+5 -2
View File
@@ -174,8 +174,11 @@ systemctl --user enable --now punktfunk-host # the user unit is now under /u
The udev rule, sysctl, and systemd **user** unit all live under `/usr/lib`, so the merged sysext
exposes them. `systemd-sysext refresh` re-merges after a reboot. (One HDR nuance of the sysext
path: the image ships gamescope without `CAP_SYS_NICE`, so its frame pacing is marginally worse —
everything works. Note the host binary carries no capability on *either* path, deliberately: one
would make the host unidentifiable to KWin and break desktop streaming, see
everything works. Capabilities inside the image: `punktfunk-host` carries **none**, on *either*
path, deliberately — one would make it unidentifiable to KWin and break desktop streaming;
`punktfunk-encode-worker` carries `cap_sys_nice=ep`, applied by `build-sysext.sh` because pacman
scriptlets never run for a sysext and a merged `/usr` is read-only. Both are asserted at build
time. See
[Running as a service](https://punktfunk.io/docs/running-as-a-service#gpu-scheduling-priority).)
## Steam Deck — the client (what the Decky plugin launches)
+83 -9
View File
@@ -16,15 +16,23 @@
# its `+pfhdr` banner, never trusted by filename. Omit it and the image is exactly what it was —
# the host then stays SDR on that backend, by design.
#
# No CAP_SYS_NICE inside the image, for either binary. ⚠ NOT because capabilities are lost on the
# way in — that was this comment's earlier claim and it is false: mksquashfs records
# security.capability, and the published Bazzite 0.26.0-1 image really did carry `cap_sys_nice=ep`
# on usr/bin/punktfunk-host. It is left out on purpose. A capability on the HOST binary makes it
# unidentifiable to KWin (which resolves a client's /proc/<pid>/exe to match it against a .desktop,
# and cannot read it for a capability-carrying process) and kills every Desktop-mode session — see
# packaging/bazzite/build-sysext.sh, which now hard-fails if one is staged. `punktfunk-gamescope`
# is a compositor, not a KWin client, so it is unaffected by that rule and simply runs without the
# capability here, pacing slightly worse.
# Capabilities in the image: NEVER on usr/bin/punktfunk-host, `cap_sys_nice=ep` on
# usr/bin/punktfunk-encode-worker (best-effort), and none on punktfunk-gamescope.
#
# ⚠ Capabilities are NOT lost on the way in — that was this comment's earlier claim and it is
# false: mksquashfs records security.capability, and the published Bazzite 0.26.0-1 image really
# did carry `cap_sys_nice=ep` on usr/bin/punktfunk-host. The host is left uncapped on purpose. A
# capability on the HOST binary makes it unidentifiable to KWin (which resolves a client's
# /proc/<pid>/exe to match it against a .desktop, and cannot read it for a capability-carrying
# process) and kills every Desktop-mode session.
#
# ⚠ And it is NOT enough to leave it out here: pacman scriptlets never run for a sysext, so the
# `setcap` in punktfunk-host.install cannot reach this image either way. The encode worker is
# therefore capped on the staging tree below — this is the only place a sysext can acquire it — and
# both halves of the matrix are asserted before mksquashfs, exactly as
# packaging/bazzite/build-sysext.sh does. `punktfunk-gamescope` is a compositor, not a KWin client,
# so it is unaffected by the host rule and simply runs without a capability here, pacing slightly
# worse.
set -euo pipefail
GAMESCOPE=""
@@ -80,6 +88,72 @@ ID=_any
ARCHITECTURE=x86-64
EOF
# CAP_SYS_NICE on the encode worker (see the header). A pacman payload carries no capabilities and
# no scriptlet ever runs for a sysext, so without this the SteamOS image ships the lever inert —
# on the box with the smallest GPU shared between game and encode. Needs CAP_SETFCAP, i.e. root or
# fakeroot; a plain-user build simply ships without it, which is a pacing loss and nothing more.
#
# `getcap` on an uncapped file exits 0 and prints nothing, so an empty read is unambiguous; the
# output form differs across libcap versions ("path cap_sys_nice=ep" since ~2.36, "path =
# cap_sys_nice+ep" before), hence the normalizer.
_pf_caps_of() {
local raw; raw="$(getcap "$1" 2>/dev/null || true)"
[ -n "$raw" ] || { printf ''; return 0; }
printf '%s' "${raw#* }" | sed -e 's/^= *//' -e 's/+/=/' -e 's/[[:space:]]*$//'
}
# BEFORE granting: refuse a capability that arrived from somewhere else. The setcap below would
# overwrite it and ship a correct-looking image while the surprise went unreported everywhere else.
# Order matters: assert first, then grant, or the "anything else" arm can never fire.
if command -v getcap >/dev/null 2>&1 && [ -f "$STAGE/usr/bin/punktfunk-encode-worker" ]; then
arrived_caps="$(_pf_caps_of "$STAGE/usr/bin/punktfunk-encode-worker")"
case "$arrived_caps" in
''|cap_sys_nice=ep) : ;;
*)
echo "ERROR: staged usr/bin/punktfunk-encode-worker ARRIVED carrying '$arrived_caps'." >&2
echo " A pacman payload carries no capabilities, so something else granted it — find" >&2
echo " out what, because it is doing the same on the plain package path, unchecked." >&2
exit 1 ;;
esac
fi
if [ -f "$STAGE/usr/bin/punktfunk-encode-worker" ]; then
if setcap 'cap_sys_nice=ep' "$STAGE/usr/bin/punktfunk-encode-worker" 2>/dev/null; then
echo "granted CAP_SYS_NICE to usr/bin/punktfunk-encode-worker (GPU-priority lever active)"
else
echo "WARNING: could not setcap CAP_SYS_NICE on usr/bin/punktfunk-encode-worker (need" >&2
echo " root/CAP_SETFCAP) — the image ships without it and PyroWave encodes at" >&2
echo " default GPU priority." >&2
fi
fi
# Assert the final matrix before it is sealed into a read-only squashfs: host EMPTY (hard fail),
# worker exactly cap_sys_nice=ep or nothing at all (missing is fine — the grant is best-effort).
if command -v getcap >/dev/null 2>&1; then
if [ -f "$STAGE/usr/bin/punktfunk-host" ]; then
staged_caps="$(_pf_caps_of "$STAGE/usr/bin/punktfunk-host")"
if [ -n "$staged_caps" ]; then
echo "ERROR: staged usr/bin/punktfunk-host carries capabilities: $staged_caps" >&2
echo " A capability makes the host unidentifiable to KWin and breaks every Desktop-mode" >&2
echo " session on a merged image, which cannot be repaired on the box (read-only /usr)." >&2
echo " The GPU-priority capability belongs on usr/bin/punktfunk-encode-worker, never here." >&2
exit 1
fi
fi
if [ -f "$STAGE/usr/bin/punktfunk-encode-worker" ]; then
worker_caps="$(_pf_caps_of "$STAGE/usr/bin/punktfunk-encode-worker")"
case "$worker_caps" in
'') echo "note: usr/bin/punktfunk-encode-worker ships uncapped — PyroWave encodes at default GPU priority" ;;
cap_sys_nice=ep) : ;;
*)
echo "ERROR: staged usr/bin/punktfunk-encode-worker carries '$worker_caps'," >&2
echo " expected exactly 'cap_sys_nice=ep' (or nothing at all)." >&2
echo " Refusing to bake an unexpected capability into a read-only image." >&2
exit 1 ;;
esac
fi
fi
OUT="$NAME.raw"
rm -f "$OUT"
mksquashfs "$STAGE" "$OUT" -all-root -noappend -quiet
+34
View File
@@ -50,10 +50,40 @@ _revoke_sched_capability() {
setcap -r usr/bin/punktfunk-host 2>/dev/null || true
}
# CAP_SYS_NICE on the ENCODE WORKER — the same GPU-scheduling grant 0.26.0-1 aimed at the wrong
# binary, now on a binary that can carry it.
#
# punktfunk-encode-worker is a separate executable (never a hardlink or a subcommand of the host —
# a shared inode would share the file capability and silently re-create the breakage above). It is
# spawned per PyroWave session, speaks one socketpair to its parent, and never connects to Wayland,
# D-Bus or the network — so it is not a KWin client, nothing ever resolves its /proc/<pid>/exe, and
# a capability on it is invisible to the identification path that the host must keep clear.
#
# What it buys: PyroWave encodes on the same GPU shader cores the game saturates, and an elevated
# VK_KHR_global_priority queue is the preemption lever for that. Every driver tested (NVIDIA and
# RADV alike) refuses EVERY priority class without CAP_SYS_NICE, so without this line the lever is
# decoration. Measured on .21 (RTX 5070 Ti, GRID 2 loop): encode p99 6.4 -> 4.4 ms.
#
# NARROW: CAP_SYS_NICE permits raising scheduling priority only (nice/ioprio/affinity/RT class). No
# filesystem, network or user-switching privilege, and it is NOT setuid.
#
# BEST-EFFORT, always: an uncapped worker still encodes, at default priority. A box without libcap,
# or a filesystem that cannot store capabilities, must never fail an install over a pacing lever.
#
# Two consequences worth knowing before debugging the WORKER (they do not apply to the host):
# * a file capability makes the process AT_SECURE, so the loader ignores LD_LIBRARY_PATH and
# LD_PRELOAD for it — a library-path shim that rescues the host will NOT reach the worker.
# * core dumps are suppressed for capability-carrying binaries by default (fs.suid_dumpable).
_grant_worker_sched_capability() {
[ -f usr/bin/punktfunk-encode-worker ] || return 0
setcap 'cap_sys_nice=ep' usr/bin/punktfunk-encode-worker 2>/dev/null || true
}
post_install() {
_ensure_update_group
_ensure_punktfunk_group
_revoke_sched_capability
_grant_worker_sched_capability
udevadm control --reload-rules 2>/dev/null || true
udevadm trigger --subsystem-match=misc 2>/dev/null || true
# Apply the UDP socket-buffer tuning now (also auto-applied at boot by systemd-sysctl).
@@ -114,6 +144,10 @@ post_upgrade() {
_ensure_punktfunk_group
# Strip the cap_sys_nice 0.26.0-1 granted: it makes the host unidentifiable to KWin (see above).
_revoke_sched_capability
# And (re-)grant it to the encode worker. On UPGRADE too, and this one is not belt-and-braces:
# pacman writes a REPLACED binary as a new inode, file capabilities live on the inode, so the
# grant is gone after every single upgrade unless it is re-applied here.
_grant_worker_sched_capability
udevadm control --reload-rules 2>/dev/null || true
sysctl -p /usr/lib/sysctl.d/99-punktfunk-net.conf >/dev/null 2>&1 || true
_warn_stale_firewall_ports
+86 -16
View File
@@ -130,9 +130,9 @@ SYSEXT_VERSION_ID=$PF_VR
EXTENSION_RELOAD_MANAGER=1
EOF
# NO CAP_SYS_NICE in the image — and an assertion that none crept back in.
# CAP_SYS_NICE on the ENCODE WORKER, never on the host — and an assertion of BOTH halves.
#
# 0.26.0-1 setcap'd the staged binary here for the GPU-priority lever. mksquashfs records
# 0.26.0-1 setcap'd the staged HOST binary here for the GPU-priority lever. mksquashfs records
# security.capability, so the capability really did ship: verified by mounting the published
# punktfunk-0.26.0-1-x86-64.raw, where `getcap usr/bin/punktfunk-host` reports `cap_sys_nice=ep`.
# That broke desktop streaming on every Bazzite KDE box, field-reported as
@@ -143,22 +143,92 @@ EOF
# /proc/<pid>/exe and matching it against an installed .desktop's Exec= — the image ships
# usr/share/applications/io.unom.Punktfunk.Host.desktop for exactly that. The kernel refuses that
# readlink to any reader whose effective set is not a superset of the target's PERMITTED set
# (cap_ptrace_access_check), and KWin holds no capabilities. So a capability in this image makes the
# host unidentifiable and every Desktop-mode session dies. Full matrix, including why neither
# prctl(PR_SET_DUMPABLE, 1) nor systemd AmbientCapabilities= rescues it, in
# (cap_ptrace_access_check), and KWin holds no capabilities. So a capability on the HOST in this
# image makes it unidentifiable and every Desktop-mode session dies. Full matrix, including why
# neither prctl(PR_SET_DUMPABLE, 1) nor systemd AmbientCapabilities= rescues it, in
# packaging/arch/punktfunk-host.install.
#
# A merged sysext's /usr is a read-only squashfs, so this cannot be repaired on the box — the image
# is the only place it can be got right. Assert it rather than trust it: the RPM payload arrives via
# `rpm2cpio | cpio`, which carries no capabilities today, but the spec is one `%caps()` away from
# changing that and this build would silently bake it in.
if [ -f "$STAGE/usr/bin/punktfunk-host" ] && command -v getcap >/dev/null 2>&1; then
staged_caps="$(getcap "$STAGE/usr/bin/punktfunk-host" 2>/dev/null || true)"
if [ -n "$staged_caps" ]; then
echo "ERROR: staged usr/bin/punktfunk-host carries capabilities: $staged_caps" >&2
echo " A capability makes the host unidentifiable to KWin and breaks every Desktop-mode" >&2
echo " session on a merged image, which cannot be repaired on the box (read-only /usr)." >&2
exit 1
# usr/bin/punktfunk-encode-worker is the OTHER binary: a separate executable (never a hardlink or a
# host subcommand — a shared inode shares the capability and re-creates the above), spawned per
# PyroWave session, speaking one socketpair to its parent and touching neither Wayland nor D-Bus
# nor the network. Nothing resolves ITS /proc/<pid>/exe, so it can carry the capability the lever
# needs. This is the ONLY place the sysext can acquire it: a merged sysext's /usr is a read-only
# squashfs, and it cannot ride in from the RPM either — the spec declares %caps(cap_sys_nice=ep),
# but rpm keeps capabilities in its own header and `rpm2cpio | cpio` carries only the payload, so
# the staged file arrives with none. mksquashfs DOES record security.capability (only
# security.selinux is excluded below), so a setcap on the staging tree is what lands in the image.
#
# Needs CAP_SETFCAP, i.e. root (or fakeroot). A plain-user build simply cannot, and that is NOT
# fatal: an uncapped worker still encodes, at default priority. Warn and carry on rather than fail
# a release over a pacing lever.
#
# `getcap` on a file with no capability exits 0 and prints nothing, so an empty read is unambiguous.
# The output form differs across libcap versions ("path cap_sys_nice=ep" since ~2.36, "path =
# cap_sys_nice+ep" before), hence the normalizer.
_pf_caps_of() {
# -> canonical "cap_sys_nice=ep", or "" when the file carries no capability.
local raw; raw="$(getcap "$1" 2>/dev/null || true)"
[ -n "$raw" ] || { printf ''; return 0; }
printf '%s' "${raw#* }" | sed -e 's/^= *//' -e 's/+/=/' -e 's/[[:space:]]*$//'
}
# BEFORE granting: refuse a capability that arrived from somewhere else. The setcap below would
# overwrite it and ship a correct-looking image while the surprise — a stray %caps() in the spec, a
# payload from an unexpected source — went unreported on every other channel. Order matters: assert
# first, then grant, or the "anything else" arm can never fire.
if command -v getcap >/dev/null 2>&1 && [ -f "$STAGE/usr/bin/punktfunk-encode-worker" ]; then
arrived_caps="$(_pf_caps_of "$STAGE/usr/bin/punktfunk-encode-worker")"
case "$arrived_caps" in
''|cap_sys_nice=ep) : ;;
*)
echo "ERROR: staged usr/bin/punktfunk-encode-worker ARRIVED carrying '$arrived_caps'." >&2
echo " Nothing upstream of this script should grant it anything: rpm keeps capabilities" >&2
echo " in its own header and 'rpm2cpio | cpio' carries only the payload. Find out what" >&2
echo " did — it is granting the same thing on the plain RPM path, unchecked." >&2
exit 1 ;;
esac
fi
if [ -f "$STAGE/usr/bin/punktfunk-encode-worker" ]; then
if setcap 'cap_sys_nice=ep' "$STAGE/usr/bin/punktfunk-encode-worker" 2>/dev/null; then
echo "granted CAP_SYS_NICE to usr/bin/punktfunk-encode-worker (GPU-priority lever active)"
else
echo "WARNING: could not setcap CAP_SYS_NICE on usr/bin/punktfunk-encode-worker (need" >&2
echo " root/CAP_SETFCAP) — the image ships without it and PyroWave encodes at" >&2
echo " default GPU priority." >&2
fi
fi
# Assert the final matrix rather than trust it. A merged sysext's /usr is a read-only squashfs, so
# a bad image cannot be repaired on the box — the image is the only place this can be got right.
#
# host -> MUST be empty. Hard fail. (The RPM payload carries no capabilities today, but the
# spec is one `%caps()` away from changing that and this build would bake it in.)
# worker -> MUST be exactly cap_sys_nice=ep if it carries anything at all. MISSING IS NOT AN
# ERROR (a plain-user build cannot setcap; best-effort by design), but a DIFFERENT or
# WIDER capability is — and a read-only image is not the place to discover it.
if command -v getcap >/dev/null 2>&1; then
if [ -f "$STAGE/usr/bin/punktfunk-host" ]; then
staged_caps="$(_pf_caps_of "$STAGE/usr/bin/punktfunk-host")"
if [ -n "$staged_caps" ]; then
echo "ERROR: staged usr/bin/punktfunk-host carries capabilities: $staged_caps" >&2
echo " A capability makes the host unidentifiable to KWin and breaks every Desktop-mode" >&2
echo " session on a merged image, which cannot be repaired on the box (read-only /usr)." >&2
echo " The GPU-priority capability belongs on usr/bin/punktfunk-encode-worker, never here." >&2
exit 1
fi
fi
if [ -f "$STAGE/usr/bin/punktfunk-encode-worker" ]; then
worker_caps="$(_pf_caps_of "$STAGE/usr/bin/punktfunk-encode-worker")"
case "$worker_caps" in
'') echo "note: usr/bin/punktfunk-encode-worker ships uncapped — PyroWave encodes at default GPU priority" ;;
cap_sys_nice=ep) : ;;
*)
echo "ERROR: staged usr/bin/punktfunk-encode-worker carries '$worker_caps'," >&2
echo " expected exactly 'cap_sys_nice=ep' (or nothing at all)." >&2
echo " Refusing to bake an unexpected capability into a read-only image." >&2
exit 1 ;;
esac
fi
fi
+18
View File
@@ -45,5 +45,23 @@ RUN printf '%s\n' \
# time (host + console run per-user in the graphical session, enabled after first boot with
# `systemctl --user enable --now punktfunk-host punktfunk-web`).
# NO `setcap` here — deliberately, in BOTH directions, and this file must stay that way.
#
# /usr/bin/punktfunk-host must carry NO capability. A capability-carrying process
# cannot have its /proc/<pid>/exe read, so KWin cannot identify
# it, never advertises zkde_screencast_unstable_v1, and every
# KDE desktop session dies. That is the 0.26.0-1 incident; a
# layered/bootc image is as unrepairable in place as a sysext.
# /usr/bin/punktfunk-encode-worker carries cap_sys_nice=ep, declared with %caps in
# packaging/rpm/punktfunk.spec. rpm applies file capabilities
# from package metadata during the dnf5 install above and the
# ostree commit preserves the security.capability xattr — so it
# arrives correctly without anything to do here, and the
# capability matrix is asserted on the .rpm in CI
# (scripts/ci/assert-cap-matrix.sh, .gitea/workflows/rpm.yml).
#
# If a capability is ever wanted in this image, change the SPEC, never this file: a setcap here
# would apply to one channel and drift from the other four.
# bootc image hygiene: the container build must leave a clean ostree commit.
RUN ostree container commit
+54 -3
View File
@@ -38,6 +38,16 @@ if [ ! -x "$BIN" ]; then
echo "==> building $PKG (release)"
PUNKTFUNK_BUILD_VERSION="$VERSION" cargo build --release -p "$PKG" --locked # stamp --version (build.rs)
fi
# The PyroWave encode worker — the capability-carrying half. A SEPARATE executable, never a
# hardlink or a host subcommand: a shared inode would share the file capability and make the host
# unidentifiable to KWin all over again (see the postinst note below). It ships in this same .deb
# because host and worker version-check each other over their socket and fall back to the
# in-process encoder on any mismatch, so they must move in lockstep.
WORKER_BIN="target/release/punktfunk-encode-worker"
if [ ! -x "$WORKER_BIN" ]; then
echo "==> building punktfunk-encode-worker (release)"
PUNKTFUNK_BUILD_VERSION="$VERSION" cargo build --release -p punktfunk-encode-worker --locked
fi
TRAY_BIN="target/release/punktfunk-tray"
# ALWAYS built here, in its OWN cargo invocation — load-bearing, not tidiness, and deliberately not
# skipped when the artifact already exists. Cargo unifies features across everything in one build,
@@ -60,6 +70,9 @@ SHAREDIR="$STAGE/usr/share/$PKG"
# --- file layout (matches the RPM %install) ----------------------------------
install -Dm0755 "$BIN" "$STAGE/usr/bin/$PKG"
# Next to the host in the SAME bindir — the host resolves the worker as a sibling of
# /proc/self/exe. postinst grants this one (and only this one) cap_sys_nice=ep.
install -Dm0755 "$WORKER_BIN" "$STAGE/usr/bin/punktfunk-encode-worker"
# Web-console-triggered updates (host-update-from-web-console.md §7): root helper + its
# oneshot unit + the polkit rule scoping `systemctl start punktfunk-update.service` to the
# (shipped-empty) punktfunk-update group. Opt-in = joining the group; postinst creates it.
@@ -195,13 +208,27 @@ if [ "$BUNDLE_FFMPEG" = "1" ]; then
patchelf --set-rpath '$ORIGIN' "$so"
done
patchelf --force-rpath --set-rpath "\$ORIGIN/../lib/$PKG" "$STAGE/usr/bin/$PKG"
# The encode worker gets an ABSOLUTE rpath, not the $ORIGIN one the host uses — and this is
# load-bearing, not style. postinst grants the worker cap_sys_nice=ep, which makes it AT_SECURE,
# and glibc DROPS any $ORIGIN-expanded RPATH entry for a secure binary unless it normalizes into
# a system-trusted directory (/lib, /usr/lib — /usr/lib/punktfunk-host is not one). So a capped
# worker with `$ORIGIN/../lib/punktfunk-host` would find no libavcodec at all on Ubuntu 24.04 and
# fail to exec — the host would fall back inline (never a dead session, by the ladder's design)
# but the lever would be silently dead on exactly the channel that bundles FFmpeg. An absolute
# DT_RPATH is honoured under AT_SECURE, and because it is DT_RPATH (--force-rpath) it is searched
# transitively, so it also resolves libavutil for the bundled libavcodec — whose own $ORIGIN
# RUNPATH is subject to the same AT_SECURE rule inside this process.
patchelf --force-rpath --set-rpath "/usr/lib/$PKG" "$STAGE/usr/bin/punktfunk-encode-worker"
BUNDLED_LIBS="$(printf '%s ' "$DEST"/*.so.*)"
echo "==> bundled FFmpeg from $FFMPEG_PREFIX into /$LIBDIR_REL"
fi
# --- dependencies ------------------------------------------------------------
# Auto: the binary's directly-linked shared libs (libcuda ignored, see header). In bundle mode the
# bundled .so's are appended so their external deps (libva2/libdrm2/…) are captured too.
# Auto: the binaries' directly-linked shared libs (libcuda ignored, see header). In bundle mode the
# bundled .so's are appended so their external deps (libva2/libdrm2/…) are captured too. The encode
# worker is scanned alongside the host: its link set is a subset today, but it is a shipped
# executable in this package and a future divergence must show up as a Depends, not as a worker
# that silently fails to exec on a fresh install.
SHLIB_TMP="$(mktemp -d)"
mkdir -p "$SHLIB_TMP/debian"
cat > "$SHLIB_TMP/debian/control" <<EOF
@@ -218,7 +245,7 @@ EOF
SHDEPS_RAW="$(
cd "$SHLIB_TMP"
if [ "$BUNDLE_FFMPEG" = "1" ]; then export LD_LIBRARY_PATH="$FFMPEG_PREFIX/lib"; fi
dpkg-shlibdeps -O --ignore-missing-info "$ROOTDIR/$BIN" $BUNDLED_LIBS 2>"$SHLIB_TMP/err" \
dpkg-shlibdeps -O --ignore-missing-info "$ROOTDIR/$BIN" "$ROOTDIR/$WORKER_BIN" $BUNDLED_LIBS 2>"$SHLIB_TMP/err" \
| sed -n 's/^shlibs:Depends=//p'
)" || { echo "dpkg-shlibdeps failed (exit $?):" >&2; sed 's/^/ /' "$SHLIB_TMP/err" >&2; rm -rf "$SHLIB_TMP"; exit 1; }
rm -rf "$SHLIB_TMP"
@@ -311,6 +338,30 @@ if [ "$1" = "configure" ]; then
# postinst runs on upgrade too, so this heals boxes that installed 0.26.0-1. `setcap -r` exits
# non-zero on a file that has no capability, hence the redirect and `|| true`.
setcap -r /usr/bin/punktfunk-host 2>/dev/null || true
# CAP_SYS_NICE on the ENCODE WORKER — the same grant, on the binary that can carry it.
#
# punktfunk-encode-worker is a SEPARATE executable (never a hardlink or a host subcommand: a
# shared inode shares the capability and re-creates the breakage above). It is spawned per
# PyroWave session, speaks one socketpair to its parent, and never connects to Wayland, D-Bus
# or the network — so nothing ever resolves ITS /proc/<pid>/exe and the KWin identification
# path above stays clear.
#
# Why it is worth a capability at all: PyroWave encodes on the GPU shader cores the game
# saturates, and an elevated VK_KHR_global_priority queue is the preemption lever. Every driver
# tested (NVIDIA and RADV) refuses EVERY class without CAP_SYS_NICE. Measured on an RTX 5070
# Ti under load: encode p99 6.4 -> 4.4 ms. Narrow — scheduling priority only, no filesystem,
# network or user-switching privilege, not setuid.
#
# Best-effort, always: an uncapped worker still encodes at default priority, so a box without
# libcap or a filesystem that cannot store capabilities must not fail this install. postinst
# runs on upgrade too, which is what re-applies the grant to the replaced (new-inode) file.
#
# Debugging the WORKER: a capability makes it AT_SECURE — the loader ignores LD_LIBRARY_PATH
# and LD_PRELOAD for it, and core dumps are suppressed. (On a bundled-FFmpeg build the worker
# carries an ABSOLUTE rpath for exactly that reason; see build-deb.sh.)
if [ -x /usr/bin/punktfunk-encode-worker ]; then
setcap 'cap_sys_nice=ep' /usr/bin/punktfunk-encode-worker 2>/dev/null || true
fi
# Pick up the /dev/uinput rule without a reboot (best-effort, no-op in containers).
udevadm control --reload-rules 2>/dev/null || true
udevadm trigger --subsystem-match=misc 2>/dev/null || true
+43 -3
View File
@@ -356,7 +356,8 @@ in
allowedUDPPorts = nativeUDP ++ optionals cfg.host.gamestream gamestreamUDP;
};
# NO CAP_SYS_NICE wrapper here — deliberately. 0.26.0-1 gave the host a
# NO CAP_SYS_NICE wrapper for the HOST — deliberately, and note there is deliberately one for
# the WORKER just below; the difference is the whole point. 0.26.0-1 gave the host a
# `security.wrappers.punktfunk-host` carrying `cap_sys_nice=ep` for the GPU-priority lever,
# and that broke desktop streaming on every KDE box.
#
@@ -373,8 +374,36 @@ in
# ambient-only grant (dumpable=1, CapPrm set) is refused exactly like a file capability. See
# packaging/arch/punktfunk-host.install for the full matrix.
#
# Costs pacing only: pf-zerocopy walks REALTIME -> HIGH -> default when a priority class is
# refused, and pf-frame's thread nice is a best-effort no-op — 0.25.0's behaviour exactly.
# The capability lives on the ENCODE WORKER instead — a different binary, and one nothing
# ever has to identify.
#
# punktfunk-encode-worker is spawned per PyroWave session, speaks one socketpair to its
# parent, and never connects to Wayland, D-Bus or the network. Nothing resolves ITS
# /proc/<pid>/exe, so the ambient grant a NixOS wrapper performs — the very thing that makes
# a wrapper useless for the host — is exactly right here. (It must also stay a SEPARATE file:
# a hardlink or a host subcommand would share the inode, hence the capability, and re-create
# the breakage above on every file-capability channel.)
#
# A file capability cannot live on a store path (read-only, and shared by every generation),
# so `security.wrappers` is the only mechanism NixOS has — which is why the unit below points
# PUNKTFUNK_ENCODE_WORKER at `config.security.wrapperDir` rather than the store path. The
# host's own ExecStart stays on the store path and must never move.
#
# Best-effort by construction: if the wrapper is absent or the operator overrides the env,
# the host falls back to its in-process encoder at default GPU priority — one warn, never a
# dead session. What the capability buys: PyroWave encodes on the GPU shader cores a game
# saturates, and every driver tested (NVIDIA and RADV) refuses EVERY elevated
# VK_KHR_global_priority class without CAP_SYS_NICE. Measured on an RTX 5070 Ti under load:
# encode p99 6.4 -> 4.4 ms.
#
# Narrow: CAP_SYS_NICE permits raising scheduling priority only — no filesystem, network or
# user-switching privilege, and the wrapper is capability-based, NOT setuid.
security.wrappers.punktfunk-encode-worker = {
source = "${cfg.host.package}/bin/punktfunk-encode-worker";
capabilities = "cap_sys_nice=ep";
owner = "root";
group = "root";
};
systemd.user.services.punktfunk-host = {
description = "punktfunk GameStream + punktfunk/1 streaming host";
@@ -393,6 +422,17 @@ in
# The HDR-capable gamescope, if enabled. On PATH rather than pinned through
# PUNKTFUNK_GAMESCOPE_BIN so an operator's own override of that env still wins.
++ optional cfg.host.gamescopeHdr cfg.host.gamescopePackage;
# Point the host at the WRAPPED encode worker (see `security.wrappers` above). The host's
# own resolution order is PUNKTFUNK_ENCODE_WORKER -> alongside /proc/self/exe -> PATH, and
# on NixOS the sibling of the store binary is the UNCAPPED store copy — it would run, and
# be refused every priority class, silently. This env is the whole reason the override
# exists. `config.security.wrapperDir` rather than a hard-coded /run/wrappers/bin so an
# operator who has moved it is still correct.
#
# NixOS renders `Environment=` before `EnvironmentFile=`, so `settings`/`environmentFile`
# can still override this (or set it to `off` to force the in-process encoder) — the same
# "an operator's own override still wins" posture as PUNKTFUNK_GAMESCOPE_BIN above.
environment.PUNKTFUNK_ENCODE_WORKER = "${config.security.wrapperDir}/punktfunk-encode-worker";
serviceConfig = {
# The store path DIRECTLY — not a capability wrapper. /proc/<pid>/exe then resolves to the
# very path packages.nix substituted into io.unom.Punktfunk.Host.desktop's Exec=, which is
+15 -2
View File
@@ -141,9 +141,17 @@ in
commonArgs
// {
pname = "punktfunk-host";
# HOST ONLY — the tray is a separate derivation (see the note above; co-building crashes it).
# HOST + ENCODE WORKER — the tray is a separate derivation (see the note above; co-building
# crashes it), but punktfunk-encode-worker belongs here: it is the capability-carrying half of
# the PyroWave encode path, it shares the host's dependency graph by design, and host and
# worker version-check each other over their socket, so they must be built and shipped
# lockstep. It is a SEPARATE executable, never a hardlink or a host subcommand — on
# file-capability channels a shared inode would share the capability and make the host
# unidentifiable to KWin (see the note in nixos-module.nix). Without `-p` here crane never
# builds it and `$out/bin` simply would not contain it.
cargoExtraArgs =
"--locked -p punktfunk-host " + "--features punktfunk-host/nvenc,punktfunk-host/vulkan-encode";
"--locked -p punktfunk-host -p punktfunk-encode-worker "
+ "--features punktfunk-host/nvenc,punktfunk-host/vulkan-encode";
PUNKTFUNK_BUILD_VERSION = buildVersion;
@@ -203,6 +211,11 @@ in
postFixup = ''
# Only the host dlopens the GPU stack; the tray (its own derivation, copied in above) does not.
addDriverRunpath "$out/bin/punktfunk-host"
# The encode worker owns a Vulkan device of its own (PyroWave encodes through ash, which
# dlopens the loader and the vendor ICD), so it needs the same driver runpath. Without it
# the worker starts and then finds no usable device — the host falls back to the in-process
# encoder, so nothing breaks, but the GPU-priority lever this binary exists for is dead.
addDriverRunpath "$out/bin/punktfunk-encode-worker"
'';
meta = meta // {
+34 -1
View File
@@ -237,9 +237,17 @@ export PUNKTFUNK_BUILD_VERSION="%{version}-%{release}"
# with real RFI (clean P-frame recovery anchor via DPB reference slots; design/linux-vulkan-video-encode.md).
# Pure Rust `ash` (no new lib / no link-time dep); default on for HEVC (PUNKTFUNK_VULKAN_ENCODE=0 opts
# back to libav VAAPI), and a failed open falls back to VAAPI so unsupported devices degrade gracefully.
# -p punktfunk-encode-worker: the capability-carrying PyroWave encode worker, shipped next to the
# host in %%{_bindir} and granted cap_sys_nice=ep via %%caps in %%files. It MUST be a separate file
# (the host can never carry a capability — KWin identification, see the note in %%files), and it
# must ship in the SAME package: host and worker version-check each other over their socket and
# fall back to the in-process encoder on any mismatch. Co-built in this one invocation on purpose —
# v1 accepts that the worker links the same FFmpeg the host does (same package, same sonames, no
# new break class), so cargo's feature unification here is harmless.
%if %{with host}
cargo build --release --locked --features punktfunk-host/nvenc,punktfunk-host/vulkan-encode \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-cli \
-p punktfunk-host -p punktfunk-encode-worker \
-p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-cli \
-p pf-update
%else
# Client-only (aarch64): no host crate, so none of the encode features apply. pf-update still
@@ -282,6 +290,10 @@ fi
%if %{with host}
# Binary
install -Dm0755 target/release/punktfunk-host %{buildroot}%{_bindir}/punktfunk-host
# The PyroWave encode worker — a SEPARATE executable in the same bindir (the host resolves it as a
# sibling of /proc/self/exe). This is the ONLY binary in this package that carries a capability;
# see the %%caps note in %%files.
install -Dm0755 target/release/punktfunk-encode-worker %{buildroot}%{_bindir}/punktfunk-encode-worker
# udev rule — /dev/uinput access for virtual gamepads (input group).
install -Dm0644 scripts/60-punktfunk.rules %{buildroot}%{_udevrulesdir}/60-punktfunk.rules
@@ -499,6 +511,27 @@ install -Dm0644 scripts/punktfunk-scripting.service %{buildroot}%{_userunitdir}/
# rpm applies file capabilities from package metadata, so a package built WITHOUT %caps() installs
# the binary with none and an upgrade from 0.26.0-1 clears it — no scriptlet needed.
%{_bindir}/punktfunk-host
# CAP_SYS_NICE on the ENCODE WORKER — the grant 0.26.0-1 aimed at the wrong binary, on a binary
# that can carry it. punktfunk-encode-worker is a separate executable (never a hardlink or a host
# subcommand: a shared inode would share the capability and re-create the breakage above). It is
# spawned per PyroWave session, speaks one socketpair to its parent and never touches Wayland,
# D-Bus or the network — so nothing ever resolves ITS /proc/<pid>/exe and the KWin identification
# path stays clear.
#
# Declared with %%caps rather than a %%post setcap because that is the rpm-native form: rpm applies
# the capability at install, RESTORES it on upgrade (a replaced file is a new inode), and verifies
# it under `rpm -V`. A scriptlet does none of those. This also covers Bazzite via rpm-ostree
# layering, which honours file capabilities from package metadata.
#
# Why: PyroWave encodes on the GPU shader cores the game saturates, and an elevated
# VK_KHR_global_priority queue is the preemption lever. Every driver tested (NVIDIA and RADV)
# refuses EVERY class without CAP_SYS_NICE. Measured on .21 (RTX 5070 Ti): encode p99 6.4 -> 4.4 ms.
# Narrow — scheduling priority only, no filesystem/network/user-switching privilege, not setuid.
# Best-effort by construction: an uncapped worker still encodes, at default priority.
#
# Debugging the WORKER (not the host): a capability makes it AT_SECURE, so the loader ignores
# LD_LIBRARY_PATH/LD_PRELOAD for it and core dumps are suppressed by default.
%caps(cap_sys_nice=ep) %{_bindir}/punktfunk-encode-worker
%{_bindir}/punktfunk-tray
%{_udevrulesdir}/60-punktfunk.rules
%dir %{_libexecdir}/punktfunk
+357
View File
@@ -0,0 +1,357 @@
#!/usr/bin/env bash
# Assert the file-capability matrix of a BUILT package, not of the source tree.
#
# usr/bin/punktfunk-host MUST carry no capability, ever. -> hard fail
# usr/bin/punktfunk-encode-worker MUST carry exactly cap_sys_nice=ep -> hard fail
#
# WHY THIS EXISTS. 0.26.0-1 shipped `cap_sys_nice=ep` on the host binary through five packaging
# channels at once. KWin identifies a Wayland client by resolving its /proc/<pid>/exe and matching
# it against an installed .desktop's Exec=, and the kernel refuses that readlink to any reader
# whose effective set is not a superset of the target's PERMITTED set (cap_ptrace_access_check).
# KWin holds no capabilities, so a capability-carrying host is unidentifiable, the restricted
# globals are never advertised, and EVERY KDE desktop session dies — presenting as a missing or
# wrong .desktop file. A merged sysext cannot even be repaired on the box (read-only /usr).
#
# Every board in that release was green. The lesson recorded at the time was "verify the PACKAGE,
# never the board"; this script is that, mechanized. It reads what the artifact will actually do on
# a user's machine — the pacman scriptlet, the dpkg postinst, rpm's file-capability metadata, the
# xattrs inside the squashfs — and refuses the release if the matrix is wrong in either direction.
#
# Usage:
# scripts/ci/assert-cap-matrix.sh <artifact> [<artifact> ...]
# scripts/ci/assert-cap-matrix.sh --self-test # red-team the assertions themselves
#
# Artifacts, dispatched by extension:
# *.pkg.tar.zst Arch — the payload listing + the .INSTALL scriptlet (pacman applies caps there,
# not from package metadata, so the scriptlet TEXT is the ground truth)
# *.deb Debian— the payload listing + DEBIAN/postinst (same reason)
# *.rpm RPM — rpm's own file-capability metadata (%caps), which is what rpm applies,
# restores on upgrade and verifies — and what rpm-ostree layers on Bazzite
# *.raw sysext— the squashfs xattrs, read back out of the image that will actually ship
#
# A skipped artifact (no host and no worker inside, e.g. a client-only package) is reported and
# ignored. Anything it cannot READ is a failure, never a pass: a blind check is worse than none,
# which is why the sysext path proves its own reader with a capability round-trip first.
set -euo pipefail
HOST_REL='usr/bin/punktfunk-host'
WORKER_REL='usr/bin/punktfunk-encode-worker'
WANT_WORKER_CAPS='cap_sys_nice=ep'
RC=0
err() { printf '::error::%s\n' "$*" >&2; }
note() { printf '%s\n' "$*"; }
# --- the matrix -------------------------------------------------------------------------------
# Pure function of four already-extracted facts, so it can be (and is, below) unit-tested on any
# box with a bash — including one with no setcap, no rpm and no dpkg.
#
# $1 label human-readable artifact name, for the message
# $2 host_caps canonical capability string on the host binary, "" = none
# $3 worker_caps canonical capability string on the worker binary, "" = none
# $4 worker_present 1 if the artifact ships the worker at all
assert_matrix() {
local label="$1" host_caps="$2" worker_caps="$3" worker_present="$4" rc=0
if [ -n "$host_caps" ]; then
err "$label: $HOST_REL carries '$host_caps' — it must carry NO capability, ever."
err "$label: a capability makes the host unidentifiable to KWin (it cannot readlink"
err "$label: /proc/<pid>/exe of a capability-carrying process), so every KDE desktop session"
err "$label: dies with 'KWin does not expose zkde_screencast_unstable_v1 to this client'."
err "$label: The GPU-priority capability belongs on $WORKER_REL. This is the 0.26.0-1 incident."
rc=1
fi
if [ "$worker_present" != 1 ]; then
err "$label: does not ship $WORKER_REL. Host and worker must move lockstep — they"
err "$label: version-check each other over their socket — and the GPU-priority lever is inert"
err "$label: without the worker."
rc=1
elif [ "$worker_caps" != "$WANT_WORKER_CAPS" ]; then
err "$label: $WORKER_REL carries '${worker_caps:-<none>}', expected exactly '$WANT_WORKER_CAPS'."
if [ -z "$worker_caps" ]; then
err "$label: without it every driver refuses every elevated VK_KHR_global_priority class and"
err "$label: PyroWave encodes at default GPU priority. Granting it needs CAP_SETFCAP at build"
err "$label: or install time — check the scriptlet/%caps/setcap for this channel."
fi
rc=1
fi
if [ "$rc" = 0 ]; then
note "OK $label: host uncapped, worker $WANT_WORKER_CAPS"
fi
return "$rc"
}
# Canonicalize a capability string. getcap has printed two forms over its life
# ("path cap_sys_nice=ep" since libcap ~2.36, "path = cap_sys_nice+ep" before) and rpm renders
# "(none)" for a file with no capability. Everything downstream compares canonical strings.
caps_norm() {
local s="${1:-}"
case "$s" in ''|'(none)'|'<none>') printf ''; return 0 ;; esac
printf '%s' "$s" | sed -e 's/^= *//' -e 's/+/=/g' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
}
# --- scriptlet readers (Arch .INSTALL, dpkg postinst) -----------------------------------------
# pacman and dpkg do NOT carry file capabilities in package metadata: the scriptlet applies them.
# So for those two channels the scriptlet text IS the shipped behaviour, and that is what gets
# read. Comments are stripped first — every one of these files carries a long comment block that
# quotes the very commands being searched for.
#
# Limitation, stated rather than hidden: this reads literal `setcap` invocations. A grant smuggled
# through a shell variable or an eval would not be seen. Nothing in this repo does that, and the
# reviewer-facing rule is simply "spell setcap out".
scriptlet_strip_comments() { sed -e 's/#.*$//'; }
# Any capability GRANT naming the host -> echoed (and therefore fatal). `setcap -r <host>` is the
# removal we ship and carries no `cap_` token, so it is correctly invisible here.
scriptlet_host_grant() {
scriptlet_strip_comments \
| grep -E 'setcap' \
| grep -E 'punktfunk-host' \
| grep -E 'cap_[a-z_]+[=+]' \
| sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
| head -1 || true
}
# The worker grant, echoed as its canonical capability string when present.
scriptlet_worker_grant() {
scriptlet_strip_comments \
| grep -E 'setcap' \
| grep -E 'punktfunk-encode-worker' \
| grep -oE 'cap_[a-z_]+[=+][a-z]+' \
| head -1 || true
}
# --- per-format extractors ---------------------------------------------------------------------
# An artifact whose payload could not be listed must FAIL, never "skip": a reader that silently
# produces nothing would wave through the exact package this script exists to reject.
require_listing() {
local label="$1" list="$2"
if [ -z "$list" ]; then
err "$label: could not list the payload — refusing to report a PASS from an empty read."
return 1
fi
return 0
}
check_arch_pkg() {
local pkg="$1" label; label="$(basename "$pkg")"
local list scriptlet worker_present=0 host_grant worker_grant
list="$(bsdtar -tf "$pkg" 2>/dev/null || tar -tf "$pkg" 2>/dev/null || true)"
require_listing "$label" "$list" || return 1
case "$list" in *"$WORKER_REL"*) worker_present=1 ;; esac
case "$list" in *"$HOST_REL"*) ;; *)
if [ "$worker_present" = 0 ]; then note "-- $label: no host and no worker inside, skipping"; return 0; fi ;;
esac
scriptlet="$(bsdtar -xOf "$pkg" .INSTALL 2>/dev/null || true)"
if [ -z "$scriptlet" ]; then
err "$label: no .INSTALL scriptlet in the package — pacman applies capabilities ONLY from the"
err "$label: scriptlet, so a package without one cannot grant the worker anything."
return 1
fi
host_grant="$(printf '%s\n' "$scriptlet" | scriptlet_host_grant)"
worker_grant="$(printf '%s\n' "$scriptlet" | scriptlet_worker_grant)"
assert_matrix "$label" "$(caps_norm "$host_grant")" "$(caps_norm "$worker_grant")" "$worker_present"
}
check_deb() {
local deb="$1" label; label="$(basename "$deb")"
local list postinst worker_present=0 host_grant worker_grant
if command -v dpkg-deb >/dev/null 2>&1; then
list="$(dpkg-deb -c "$deb" 2>/dev/null || true)"
postinst="$(dpkg-deb --info "$deb" postinst 2>/dev/null || true)"
elif command -v bsdtar >/dev/null 2>&1; then
# dpkg-less fallback: a .deb is an `ar` archive of two tarballs, and libarchive reads both
# layers. (GNU `ar`/`ar p` is NOT used — Apple's ar rewrites the archive and loses members.)
list="$(bsdtar -xOf "$deb" 'data.tar*' 2>/dev/null | bsdtar -tf - 2>/dev/null || true)"
postinst="$(bsdtar -xOf "$deb" 'control.tar*' 2>/dev/null | bsdtar -xOf - './postinst' 'postinst' 2>/dev/null || true)"
else
err "$label: neither dpkg-deb nor bsdtar available — cannot read this package"
return 1
fi
require_listing "$label" "$list" || return 1
case "$list" in *"$WORKER_REL"*) worker_present=1 ;; esac
case "$list" in *"$HOST_REL"*) ;; *)
if [ "$worker_present" = 0 ]; then note "-- $label: no host and no worker inside, skipping"; return 0; fi ;;
esac
if [ -z "$postinst" ]; then
err "$label: no DEBIAN/postinst — dpkg applies capabilities only from the postinst, so this"
err "$label: package cannot grant the worker anything."
return 1
fi
host_grant="$(printf '%s\n' "$postinst" | scriptlet_host_grant)"
worker_grant="$(printf '%s\n' "$postinst" | scriptlet_worker_grant)"
assert_matrix "$label" "$(caps_norm "$host_grant")" "$(caps_norm "$worker_grant")" "$worker_present"
}
check_rpm() {
local rpm_file="$1" label; label="$(basename "$rpm_file")"
local caps_table host_caps worker_caps worker_present=0
command -v rpm >/dev/null 2>&1 || { err "$label: no rpm(8) to read file capabilities with"; return 1; }
# rpm carries capabilities in its own header (%caps) and applies/restores/verifies them itself —
# this is the metadata, i.e. exactly what lands on the box (and what rpm-ostree layers).
caps_table="$(rpm -qp --qf '[%{FILENAMES} %{FILECAPS}\n]' "$rpm_file" 2>/dev/null || true)"
require_listing "$label" "$caps_table" || return 1
case "$caps_table" in *"/$WORKER_REL"*|*"$WORKER_REL"*) worker_present=1 ;; esac
case "$caps_table" in
*"$HOST_REL"*) ;;
*) if [ "$worker_present" = 0 ]; then note "-- $label: no host and no worker inside, skipping"; return 0; fi ;;
esac
host_caps="$(printf '%s\n' "$caps_table" | awk -v p="/$HOST_REL" '$1 == p { $1=""; sub(/^ /,""); print; exit }')"
worker_caps="$(printf '%s\n' "$caps_table" | awk -v p="/$WORKER_REL" '$1 == p { $1=""; sub(/^ /,""); print; exit }')"
assert_matrix "$label" "$(caps_norm "$host_caps")" "$(caps_norm "$worker_caps")" "$worker_present"
}
# Prove the reader is not blind BEFORE trusting an empty read from a squashfs. A check that cannot
# see a capability would pass the exact image it exists to reject, so: stage a file, cap it, squash
# it, unsquash it, read it back. If that round trip loses the capability (no CAP_SETFCAP in the
# container, a filesystem that cannot store security.capability, an unsquashfs without xattr
# support) this returns non-zero and the caller FAILS rather than silently approving.
squashfs_reader_is_honest() {
local probe img out got
probe="$(mktemp -d)"; img="$probe/probe.squashfs"; out="$probe/out"
mkdir -p "$probe/tree"
printf '#!/bin/true\n' > "$probe/tree/capped"; chmod 0755 "$probe/tree/capped"
printf '#!/bin/true\n' > "$probe/tree/plain"; chmod 0755 "$probe/tree/plain"
if ! setcap "$WANT_WORKER_CAPS" "$probe/tree/capped" 2>/dev/null; then
rm -rf "$probe"; return 1
fi
mksquashfs "$probe/tree" "$img" -noappend -quiet >/dev/null 2>&1 || { rm -rf "$probe"; return 1; }
unsquashfs -no-progress -xattrs -d "$out" "$img" >/dev/null 2>&1 || { rm -rf "$probe"; return 1; }
got="$(caps_norm "$(getcap "$out/capped" 2>/dev/null | sed 's/^[^ ]* //')")"
# Positive control AND negative control: it must see the capability that is there, and must not
# invent one that is not.
[ "$got" = "$WANT_WORKER_CAPS" ] || { rm -rf "$probe"; return 1; }
[ -z "$(caps_norm "$(getcap "$out/plain" 2>/dev/null | sed 's/^[^ ]* //')")" ] || { rm -rf "$probe"; return 1; }
rm -rf "$probe"; return 0
}
check_sysext_raw() {
local raw="$1" label; label="$(basename "$raw")"
local tmp list host_caps worker_caps worker_present=0
for t in unsquashfs mksquashfs getcap setcap; do
command -v "$t" >/dev/null 2>&1 || { err "$label: missing $t — cannot read the image's capabilities"; return 1; }
done
if ! squashfs_reader_is_honest; then
err "$label: this runner cannot round-trip a file capability through squashfs (no CAP_SETFCAP,"
err "$label: or an unsquashfs/filesystem without xattr support). Refusing to report a PASS that"
err "$label: would be blind — a guard that cannot fail is not a guard. Run this leg as root on"
err "$label: a filesystem that stores security.capability."
return 1
fi
list="$(unsquashfs -no-progress -l "$raw" 2>/dev/null || true)"
require_listing "$label" "$list" || return 1
case "$list" in *"$WORKER_REL"*) worker_present=1 ;; esac
case "$list" in
*"$HOST_REL"*) ;;
*) if [ "$worker_present" = 0 ]; then note "-- $label: no host and no worker inside, skipping"; return 0; fi ;;
esac
tmp="$(mktemp -d)"
unsquashfs -no-progress -xattrs -d "$tmp/x" "$raw" "$HOST_REL" "$WORKER_REL" >/dev/null 2>&1 || true
host_caps=""; worker_caps=""
[ -f "$tmp/x/$HOST_REL" ] && host_caps="$(getcap "$tmp/x/$HOST_REL" 2>/dev/null | sed 's/^[^ ]* //')"
[ -f "$tmp/x/$WORKER_REL" ] && worker_caps="$(getcap "$tmp/x/$WORKER_REL" 2>/dev/null | sed 's/^[^ ]* //')"
rm -rf "$tmp"
assert_matrix "$label" "$(caps_norm "$host_caps")" "$(caps_norm "$worker_caps")" "$worker_present"
}
# --- self-test ---------------------------------------------------------------------------------
# Red-teams the assertions themselves: every row states the verdict it MUST produce, and the row
# that matters most is the 0.26.0-1 one — a capped host has to come out RED. Pure bash, so it runs
# anywhere (macOS included) with no setcap, rpm or dpkg in sight.
self_test() {
local failures=0
_expect() { # _expect <want:pass|fail> <label> <host> <worker> <present>
local want="$1"; shift
local got=pass
assert_matrix "$@" >/dev/null 2>&1 || got=fail
if [ "$got" = "$want" ]; then
printf ' ok %-42s -> %s\n' "$1" "$got"
else
printf ' FAIL %-42s -> %s (wanted %s)\n' "$1" "$got" "$want"; failures=$((failures + 1))
fi
}
note "matrix:"
_expect pass "clean package" "" "$WANT_WORKER_CAPS" 1
_expect fail "HOST CAPPED (the 0.26.0-1 regression)" "cap_sys_nice=ep" "$WANT_WORKER_CAPS" 1
_expect fail "host capped with something else" "cap_net_admin=ep" "$WANT_WORKER_CAPS" 1
_expect fail "host capped, worker fine but missing" "cap_sys_nice=ep" "" 0
_expect fail "worker absent" "" "" 0
_expect fail "worker present but uncapped" "" "" 1
_expect fail "worker over-granted" "" "cap_sys_admin=ep" 1
_expect fail "worker granted the wrong flags" "" "cap_sys_nice=eip" 1
note "canonicalisation:"
_norm() {
local got; got="$(caps_norm "$1")"
if [ "$got" = "$2" ]; then printf ' ok %-42s -> %s\n' "${1:-<empty>}" "${got:-<empty>}"
else printf ' FAIL %-42s -> %s (wanted %s)\n' "${1:-<empty>}" "${got:-<empty>}" "$2"; failures=$((failures + 1)); fi
}
_norm 'cap_sys_nice=ep' 'cap_sys_nice=ep' # libcap >= ~2.36
_norm '= cap_sys_nice+ep' 'cap_sys_nice=ep' # older libcap
_norm '(none)' '' # rpm, no %caps
_norm '' ''
note "scriptlet reader (the Arch .INSTALL / dpkg postinst channels):"
local _sl
_slcase() { # _slcase <label> <text> <want_host_grant:yes|no> <want_worker_caps>
local label="$1" text="$2" want_host="$3" want_worker="$4" gh gw ok=1
gh="$(printf '%s\n' "$text" | scriptlet_host_grant)"
gw="$(caps_norm "$(printf '%s\n' "$text" | scriptlet_worker_grant)")"
case "$want_host" in
yes) [ -n "$gh" ] || ok=0 ;;
no) [ -z "$gh" ] || ok=0 ;;
esac
[ "$gw" = "$want_worker" ] || ok=0
if [ "$ok" = 1 ]; then printf ' ok %-42s\n' "$label"
else printf ' FAIL %-42s host_grant=%s worker=%s\n' "$label" "${gh:-<none>}" "${gw:-<none>}"; failures=$((failures + 1)); fi
}
_sl='_grant() { setcap '"'"'cap_sys_nice=ep'"'"' usr/bin/punktfunk-encode-worker; }
_revoke() { setcap -r usr/bin/punktfunk-host 2>/dev/null || true; }'
_slcase "what this repo ships" "$_sl" no 'cap_sys_nice=ep'
_sl='# setcap '"'"'cap_sys_nice=ep'"'"' usr/bin/punktfunk-host <- only a comment
setcap '"'"'cap_sys_nice=ep'"'"' usr/bin/punktfunk-encode-worker'
_slcase "a commented-out host grant is not a grant" "$_sl" no 'cap_sys_nice=ep'
_sl='setcap '"'"'cap_sys_nice=ep'"'"' usr/bin/punktfunk-host
setcap '"'"'cap_sys_nice=ep'"'"' usr/bin/punktfunk-encode-worker'
_slcase "0.26.0-1 scriptlet (must be caught)" "$_sl" yes 'cap_sys_nice=ep'
_sl='setcap -r usr/bin/punktfunk-host 2>/dev/null || true'
_slcase "removal only, no worker grant" "$_sl" no ''
if [ "$failures" != 0 ]; then err "self-test: $failures case(s) wrong"; return 1; fi
note "self-test: all cases behaved as specified"
return 0
}
# --- main ---------------------------------------------------------------------------------------
main() {
[ $# -gt 0 ] || { echo "usage: $0 <artifact> [...] | --self-test" >&2; return 2; }
if [ "$1" = "--self-test" ]; then self_test; return $?; fi
local artifact
for artifact in "$@"; do
[ -e "$artifact" ] || { err "no such artifact: $artifact"; RC=1; continue; }
case "$artifact" in
*.pkg.tar.zst|*.pkg.tar.xz) check_arch_pkg "$artifact" || RC=1 ;;
*.deb) check_deb "$artifact" || RC=1 ;;
*.rpm) check_rpm "$artifact" || RC=1 ;;
*.raw) check_sysext_raw "$artifact" || RC=1 ;;
*) err "don't know how to read capabilities out of $artifact"; RC=1 ;;
esac
done
if [ "$RC" != 0 ]; then
err "capability matrix WRONG — refusing to publish. The host must never carry a capability"
err "(0.26.0-1 killed every KDE desktop session that way); the GPU-priority grant belongs on"
err "$WORKER_REL and nowhere else."
fi
return "$RC"
}
# Sourceable: `source scripts/ci/assert-cap-matrix.sh` defines the readers and `assert_matrix`
# without running anything, so they can be driven from a test harness on a box that has none of
# the packaging tools (this is how the scriptlet reader is exercised against the real
# packaging/arch/punktfunk-host.install and the real dpkg postinst).
if [ "${BASH_SOURCE[0]}" = "$0" ]; then
main "$@"
exit $?
fi
+37 -1
View File
@@ -52,6 +52,10 @@ for arg in "$@"; do
done
TARGET_DIR="$SRC/target-steamos"
BIN="$TARGET_DIR/release/punktfunk-host"
# The PyroWave encode worker — a separate executable next to the host, and the ONLY binary this
# installer setcaps. Never a hardlink or a mode of $BIN: a shared inode shares the file capability
# and would void the KWin .desktop grant written below.
WORKER="$TARGET_DIR/release/punktfunk-encode-worker"
CONFIG="$HOME/.config/punktfunk"
UNITS="$HOME/.config/systemd/user"
XRD="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
@@ -141,13 +145,25 @@ log "Building punktfunk-host (release) — first build is slow (~10-15 min)"
# vulkan-encode matches the packaged builds (deb/arch): the raw Vulkan Video HEVC/AV1 backend
# (real RFI loss recovery). Pure-Rust ash — no extra system dep. A featureless hand build would
# silently fall back to libav VAAPI.
#
# punktfunk-encode-worker is built alongside: the capability-carrying PyroWave encode worker, a
# SEPARATE binary that lands next to the host in $TARGET_DIR/release (which is how the host finds
# it — sibling of /proc/self/exe). The sudo block further down setcaps that one and only that one;
# the host must stay capability-free or KWin cannot identify it and Desktop mode dies.
distrobox enter "$BOX" -- bash -lc "
set -e
export PATH=\$HOME/.cargo/bin:\$PATH CARGO_TARGET_DIR='$TARGET_DIR'
cd '$SRC' && cargo build -r -p punktfunk-host --features punktfunk-host/vulkan-encode
cd '$SRC' && cargo build -r -p punktfunk-host -p punktfunk-encode-worker --features punktfunk-host/vulkan-encode
"
[ -x "$BIN" ] || die "build did not produce $BIN"
ok "host binary: $BIN"
# Not fatal if it is missing — an absent worker just means the in-process encoder at default GPU
# priority, which is what every 0.26.x Deck already runs.
if [ -x "$WORKER" ]; then
ok "encode worker: $WORKER"
else
warn "no punktfunk-encode-worker at $WORKER — PyroWave will encode at default GPU priority"
fi
if [ "$WITH_WEB" = 1 ]; then
log "Building the management web console (bun)"
@@ -362,6 +378,26 @@ if [ "$SUDO_OK" = 1 ]; then
if [ -x "$BIN" ]; then
sudo setcap -r "$BIN" 2>/dev/null || true
fi
# CAP_SYS_NICE on the ENCODE WORKER — the same grant, on the binary that can hold it, and the
# Deck is the box that wants it most: one small Van Gogh GPU shared between the game and
# PyroWave's compute-shader encode. punktfunk-encode-worker is a separate executable spawned
# per session that speaks one socketpair to the host and never touches Wayland, D-Bus or the
# network — nothing resolves ITS /proc/<pid>/exe, so the .desktop grant written above stays
# valid. Every driver tested (RADV included) refuses EVERY elevated global-priority class
# without this capability, so without it the lever is decoration.
#
# Like $BIN it lives under $HOME, so it survives a SteamOS A/B update on its own — but a
# REBUILD is a new inode and file capabilities do not follow, so it must be re-applied after
# every rebuild. Re-running this installer does that, and so does update.sh.
#
# Best-effort: a failure just means the encode runs at default priority, as it does today.
if [ -x "$WORKER" ]; then
if sudo setcap 'cap_sys_nice=ep' "$WORKER" 2>/dev/null; then
ok "granted CAP_SYS_NICE to the encode worker (PyroWave can outrank a GPU-bound game)"
else
warn "could not grant CAP_SYS_NICE to $WORKER — PyroWave stays at default GPU priority"
fi
fi
# SteamOS A/B updates rebuild /etc and DROP everything not on Valve's keep list — verified
# live: an OS update stripped the udev rule + vhci autoload + UDP sysctl (gamepads silently
# degrade to Xbox 360, buffers back to 208 KB). The sanctioned fix is a preserve drop-in in
+34 -2
View File
@@ -23,6 +23,11 @@ ensure_group() {
SRC="${PUNKTFUNK_SRC:-$HOME/punktfunk}"
BOX="${PUNKTFUNK_BOX:-pf2}"
TARGET_DIR="$SRC/target-steamos"
BIN="$TARGET_DIR/release/punktfunk-host"
# The PyroWave encode worker — a separate executable next to the host, and the only one this
# script setcaps (see the capability block in the sudo section). Never a hardlink or a mode of
# $BIN: a shared inode shares the file capability and voids the KWin .desktop grant.
WORKER="$TARGET_DIR/release/punktfunk-encode-worker"
[ -d "$SRC/crates/punktfunk-host" ] || die "no punktfunk source at $SRC (set PUNKTFUNK_SRC)"
WEB=0; [ -f "$HOME/.config/systemd/user/punktfunk-web.service" ] && WEB=1
@@ -52,8 +57,10 @@ if [ "${1:-}" = "--pull" ]; then
fi
log "Rebuilding host (release)"
# vulkan-encode matches the packaged builds (deb/arch) — see install.sh.
distrobox enter "$BOX" -- bash -lc "set -e; export PATH=\$HOME/.cargo/bin:\$PATH CARGO_TARGET_DIR='$TARGET_DIR'; cd '$SRC' && cargo build -r -p punktfunk-host --features punktfunk-host/vulkan-encode"
# vulkan-encode matches the packaged builds (deb/arch) — see install.sh. punktfunk-encode-worker
# rides along: host and worker version-check each other over their socket and fall back to the
# in-process encoder on any mismatch, so an update must never move one without the other.
distrobox enter "$BOX" -- bash -lc "set -e; export PATH=\$HOME/.cargo/bin:\$PATH CARGO_TARGET_DIR='$TARGET_DIR'; cd '$SRC' && cargo build -r -p punktfunk-host -p punktfunk-encode-worker --features punktfunk-host/vulkan-encode"
ok "host rebuilt"
if [ "$WEB" = 1 ]; then
log "Rebuilding web console"
@@ -198,6 +205,31 @@ if [ "$SUDO_OK" = 1 ]; then
warn "could not create the 'punktfunk' group — the native Steam Deck pad will not attach."
warn "By hand: sudo groupadd --system punktfunk; sudo usermod -aG punktfunk $USER"
fi
# Capabilities, re-applied because this script just REBUILT both binaries and a rebuilt file is
# a new inode — file capabilities do not follow it.
#
# host -> `setcap -r`. It must carry NO capability, ever: KWin identifies a client by
# resolving /proc/<pid>/exe against a .desktop Exec= (the one refreshed above), and
# the kernel refuses that readlink for a capability-carrying process, so every
# Desktop-mode session dies with "KWin does not expose zkde_screencast_unstable_v1
# to this client". This also heals a Deck that ran 0.26.0-1's installer and has
# only ever updated since — install.sh's removal is not reachable from this path.
# worker -> `cap_sys_nice=ep`. Separate binary, spawned per PyroWave session, no Wayland/
# D-Bus/network, so nothing resolves its /proc/<pid>/exe. Without the capability
# every driver refuses every elevated global-priority class and the lever is inert.
#
# Both best-effort: `setcap -r` exits non-zero on a file that has no capability, and a failed
# grant just means the encode runs at default priority.
if [ -x "$BIN" ]; then
sudo setcap -r "$BIN" 2>/dev/null || true
fi
if [ -x "$WORKER" ]; then
if sudo setcap 'cap_sys_nice=ep' "$WORKER" 2>/dev/null; then
ok "re-granted CAP_SYS_NICE to the encode worker (rebuild = new inode)"
else
warn "could not grant CAP_SYS_NICE to $WORKER — PyroWave stays at default GPU priority"
fi
fi
# Register the tuning on Valve's atomic-update preserve list (see install.sh §4): without
# this, every SteamOS A/B update strips the three files above again (verified live —
# gamepads silently degrade to Xbox 360, UDP buffers back to 208 KB).
+1521
View File
File diff suppressed because it is too large Load Diff