The Linux data-plane renice was a silent no-op on every install — RealtimeKit fallback, audio threads boosted at all, nice-limit headroom on every channel #232

Merged
enricobuehler merged 3 commits from worktree-thread-qos-rtkit into main 2026-08-14 19:15:41 +00:00
13 changed files with 174 additions and 11 deletions
+32
View File
@@ -642,6 +642,38 @@ CONTRIBUTING.md) and nothing in CI enforces it.** Three drifts in two release cy
argument for gating it; until something does, **treat the copy as part of regenerating, not as a
follow-up.**
### Linux — the data-plane threads finally get the priority they ask for (⚠ packager-visible)
**On every Linux host to date, `pf_frame::thread_qos`'s per-thread renice was a silent no-op**
it needs CAP_SYS_NICE or a raised RLIMIT_NICE, no packaging channel granted either, and the host
binary can never carry a file capability (KWin identification, the 0.26.0-1 incident). So the
capture/encode and send threads ran at nice 0, and a CPU-saturating burst on the host — a fresh
game launch's shader-compile storm is the canonical one — descheduled them at will. A 2026-08-14
field log showed the result end to end: 5 ms audio datagrams leaving late enough to stutter, the
client's delay signal rising, and ABR cutting a gigabit-Ethernet session to its 5 Mbps floor with
zero packet loss — while the box carried 708 Mbps cleanly minutes later, once the storm passed.
**The renice now falls back to RealtimeKit** (`MakeThreadHighPriorityWithPID`, one blocking
system-bus call per boosted thread) — the same unprivileged broker PipeWire clients use, present
on effectively every desktop install. No capability enters the host's permitted set, so KWin
identification is untouched. Boxes with neither rtkit nor the new limit keep today's best-effort
no-op, one debug line per thread.
**The audio plane is boosted at all for the first time.** The 5 ms Opus capture→encode→send loop,
the PipeWire capture mainloop thread (its `process` callbacks run there — PipeWire's own
`module-rt` only covers data loops we don't use), and the pad-audio streamer now take the same
boost the video threads always asked for. The audio loop is `critical`: a scheduling stall there
is directly audible where a late video frame is one presentation slip.
**Packagers: a new `user@.service.d` drop-in.** rpm/deb/Arch (and the Bazzite sysext, via the
RPM) now ship `packaging/linux/50-punktfunk-nice.conf`
`/usr/lib/systemd/system/user@.service.d/50-punktfunk-nice.conf` (`LimitNICE=-15`), so the direct
`setpriority()` also works where rtkit isn't running. It raises a session *limit*, from the next
login — nothing is reprioritized by itself. The NixOS module instead sets
`security.rtkit.enable = lib.mkDefault true` (rtkit is not a given there). It remains true that
**no channel may ever grant the host binary a file capability** — this change is the sanctioned
route to the same end.
---
## v0.28.0
Generated
+1
View File
@@ -3115,6 +3115,7 @@ dependencies = [
"punktfunk-core",
"tracing",
"windows 0.62.2 (registry+https://github.com/rust-lang/crates.io-index)",
"zbus",
]
[[package]]
+4
View File
@@ -21,6 +21,10 @@ tracing = "0.1"
# `FramePayload::Cuda` owns a zero-copy `DeviceBuffer`; `libc` for the per-thread `setpriority`.
pf-zerocopy = { path = "../pf-zerocopy" }
libc = "0.2"
# The rtkit fallback in `thread_qos` (one blocking system-bus call per boosted thread). Same zbus
# the host already pulls via ashpd; `tokio` mirrors ashpd's backend choice so this adds the
# `blocking-api` surface without changing the resolved I/O backend, and no default `async-io`.
zbus = { version = "5", default-features = false, features = ["tokio", "blocking-api"] }
[target.'cfg(target_os = "windows")'.dependencies]
# The DXGI capture identity (`WinCaptureTarget`/`D3d11Frame`/`pack_luid`/`make_device`) + the GPU
+61 -8
View File
@@ -44,10 +44,9 @@ pub fn boost_thread_priority(critical: bool) {
// Best-effort nice of the CALLING thread. On Linux `setpriority(PRIO_PROCESS, 0, …)` acts on
// the calling thread (the kernel resolves who==0 to the current task/tid), and both call
// sites run inside their worker thread — so this nices exactly the capture/encode (critical)
// and send (non-critical) threads, nothing else. Silently no-ops without CAP_SYS_NICE / a
// raised RLIMIT_NICE, which is fine. We deliberately do NOT use SCHED_RR/FIFO by default: a
// realtime CPU class can preempt the compositor AND the game's own render thread, adding the
// very frame-time we refuse to add (opt-in only — see PUNKTFUNK_SCHED_RR).
// and send (non-critical) threads, nothing else. We deliberately do NOT use SCHED_RR/FIFO by
// default: a realtime CPU class can preempt the compositor AND the game's own render thread,
// adding the very frame-time we refuse to add (opt-in only — see PUNKTFUNK_SCHED_RR).
let nice = if critical { -10 } else { -5 };
// SAFETY: `setpriority` takes three by-value integers and no pointers, so there is nothing to
// alias or outlive. `PRIO_PROCESS` with `who == 0` targets the calling task on Linux and
@@ -57,10 +56,24 @@ pub fn boost_thread_priority(critical: bool) {
if rc == 0 {
tracing::debug!(critical, nice, "thread nice raised");
} else {
tracing::debug!(
critical,
"setpriority(nice) no-op (needs CAP_SYS_NICE / RLIMIT_NICE)"
);
// The direct call needs CAP_SYS_NICE or a raised RLIMIT_NICE, and the host binary can
// NEVER carry a file capability (a capped process's /proc/<pid>/exe is unreadable to
// KWin, which kills desktop streaming — the 0.26.0-1 field incident). RealtimeKit is
// the sanctioned unprivileged path: the same broker PipeWire's clients use, present on
// effectively every desktop install. Packaging also ships a `user@.service.d`
// LimitNICE drop-in so the direct call works on rtkit-less boxes — but only from the
// next login, and existing installs upgrade the binary alone; rtkit is what fixes the
// installed base. A 2026-08-14 field log showed exactly this rung missing: every
// fresh-launch shader storm descheduled the unprioritized audio/send threads.
match linux_rtkit::make_high_priority(nice) {
Ok(()) => tracing::debug!(critical, nice, "thread nice raised via rtkit"),
Err(e) => tracing::debug!(
critical,
reason = %e,
"setpriority(nice) no-op (needs CAP_SYS_NICE / RLIMIT_NICE, and rtkit \
was unavailable)"
),
}
}
}
#[cfg(not(any(target_os = "windows", target_os = "linux")))]
@@ -68,3 +81,43 @@ pub fn boost_thread_priority(critical: bool) {
let _ = critical;
}
}
/// RealtimeKit fallback for [`boost_thread_priority`]: ask the system-bus broker
/// (`org.freedesktop.RealtimeKit1`) to renice the calling thread when the direct
/// `setpriority` was refused. This is how PulseAudio/PipeWire clients get their boosts on a
/// stock desktop — no capability anywhere, which matters here because a file capability on the
/// host binary breaks KWin's client identification outright.
///
/// Only the high-priority (nice) verb is used, never `MakeThreadRealtime` — the SCHED_RR
/// reservations in [`boost_thread_priority`]'s comment apply to rtkit-granted RR too (and the
/// RT verb additionally demands an RLIMIT_RTTIME we don't set).
#[cfg(target_os = "linux")]
mod linux_rtkit {
/// One-shot blocking D-Bus call. Must be made from a plain worker thread, never from async
/// context — which already holds for every caller: `boost_thread_priority` acts on the
/// calling thread, so it only ever runs inside the dedicated capture/encode/send threads.
/// The connection is per-call rather than cached: this runs at most a handful of times per
/// session (thread starts), and holding a system-bus connection for the session's lifetime
/// to save microseconds at session start is a bad trade against a wedged bus daemon pinning
/// a socket in every session forever.
pub(super) fn make_high_priority(nice: i32) -> Result<(), zbus::Error> {
// SAFETY: `gettid` takes no arguments, touches no memory, and returns the calling
// thread's kernel tid — always valid on Linux.
let tid = unsafe { libc::syscall(libc::SYS_gettid) } as u64;
let pid = u64::from(std::process::id());
let conn = zbus::blocking::Connection::system()?;
// `MakeThreadHighPriorityWithPID(u64 process, u64 thread, i32 priority)` — priority is a
// nice level, floored by rtkit's MinNiceLevel (defaults well below our -10). The WithPID
// variant with our own pid is the explicit spelling of "this thread of this process";
// rtkit still authenticates the caller via the bus, so it grants nothing a plain
// `setpriority` caller couldn't be granted.
conn.call_method(
Some("org.freedesktop.RealtimeKit1"),
"/org/freedesktop/RealtimeKit1",
Some("org.freedesktop.RealtimeKit1"),
"MakeThreadHighPriorityWithPID",
&(pid, tid, nice),
)?;
Ok(())
}
}
@@ -682,6 +682,10 @@ fn pw_thread(
use pw::{properties::properties, spa};
use spa::param::audio::{AudioFormat, AudioInfoRaw};
use spa::pod::Pod;
// The stream's `process` callbacks run ON this mainloop thread (we never hand PipeWire a
// separate data loop), so PipeWire's own client `module-rt` boost of its data loops does not
// cover it — the ~2.7 ms capture quantum lives or dies by this thread's scheduling.
pf_frame::thread_qos::boost_thread_priority(true);
// Setup errors funnel through the ready handshake (mirrors mic_pw_thread's IIFE).
let result = (|| -> Result<()> {
@@ -100,6 +100,11 @@ pub(super) fn audio_thread(
/// pacing exists to prevent — so past this point the debt is forgiven, not repaid.
const PACE_REANCHOR: std::time::Duration = std::time::Duration::from_millis(100);
let want = punktfunk_core::audio::normalize_channels(channels);
// Same boost the video capture/encode loop takes, and this thread needs it MORE: it paces
// 5 ms datagrams, so a scheduling stall here is directly audible where a late video frame
// is one presentation slip. The 2026-08-14 field log's stutter was exactly this thread
// descheduled by fresh-game-launch shader storms — it carried no priority at all.
pf_frame::thread_qos::boost_thread_priority(true);
// Tier and redundancy are ONE decision, budgeted against the session's video bitrate — see
// `handshake::audio_budget`. An unparseable `audio.quality` was already warned about there
// and fell back to the default, so nothing here can silently downgrade someone's audio.
@@ -472,6 +472,9 @@ fn pad_audio_thread<C: crate::audio::AudioCapturer>(
open: impl Fn() -> anyhow::Result<C>,
stop: Arc<AtomicBool>,
) {
// Above-normal like the session send thread — this plane is silence-gated and tiny, but when
// a pad speaker/haptics stream IS live it runs the same ≤10 ms cadence as session audio.
crate::native::boost_thread_priority(false);
let mut lanes = match build_lanes(kinds) {
Ok(l) => l,
Err(e) => {
+8 -1
View File
@@ -189,7 +189,8 @@ package_punktfunk-host() {
'mesa' 'libglvnd' 'libxkbcommon' 'wayland'
'libavcodec.so' 'libavutil.so' 'libavfilter.so' 'libavdevice.so'
'libavformat.so' 'libswscale.so' 'libswresample.so')
optdepends=('pipewire-pulse: PulseAudio-API audio from games/apps (real `pulseaudio` also works)'
optdepends=('rtkit: data-plane thread priority without a relogin (else the shipped LimitNICE drop-in applies from next login)'
'pipewire-pulse: PulseAudio-API audio from games/apps (real `pulseaudio` also works)'
'nvidia-utils: NVENC hardware encode + GPU EGL/CUDA zero-copy (REQUIRED to encode on NVIDIA)'
'gamescope: per-session nested compositor backend (no desktop login needed) — needs >=3.16.22'
'punktfunk-gamescope: HDR (10-bit BT.2020 PQ) streaming on the gamescope backend — attempted by default when installed'
@@ -235,6 +236,12 @@ package_punktfunk-host() {
install -Dm0644 "$R/scripts/punktfunk-modules.conf" "$pkgdir/usr/lib/modules-load.d/punktfunk.conf"
# 32 MB UDP socket buffers (send-side headroom at high bitrate)
install -Dm0644 "$R/scripts/99-punktfunk-net.conf" "$pkgdir/usr/lib/sysctl.d/99-punktfunk-net.conf"
# Nice-limit headroom for the host's data-plane threads: raises the user-session RLIMIT_NICE so
# pf-frame's setpriority() works on boxes without RealtimeKit (with rtkit the host never needs
# it). A limit, not a grant — and NEVER a file capability on the host binary (see the KWin
# identification note in punktfunk-host.install). Applies from the next login.
install -Dm0644 "$R/packaging/linux/50-punktfunk-nice.conf" \
"$pkgdir/usr/lib/systemd/system/user@.service.d/50-punktfunk-nice.conf"
# systemd USER units (the host runs in the graphical session, not as root); repoint ExecStart.
install -Dm0644 "$R/scripts/punktfunk-host.service" "$pkgdir/usr/lib/systemd/user/punktfunk-host.service"
sed -i 's#%h/punktfunk/target/release/punktfunk-host#/usr/bin/punktfunk-host#' \
+6
View File
@@ -92,6 +92,11 @@ install -Dm0644 scripts/punktfunk-modules.conf "$STAGE/usr/lib/modules-load.
# UDP socket-buffer tuning (32 MB) — without it the kernel clamps the host's SO_SNDBUF to ~416 KB
# and high-bitrate frames overflow it (send-side packet loss). systemd-sysctl applies it at boot.
install -Dm0644 scripts/99-punktfunk-net.conf "$STAGE/usr/lib/sysctl.d/99-punktfunk-net.conf"
# Nice-limit headroom for the host's data-plane threads: raises the user-session RLIMIT_NICE so
# pf-frame's setpriority() works on boxes without RealtimeKit (with rtkit the host never needs
# it). A limit, not a grant, and never a file capability on the host binary (KWin identification).
install -Dm0644 packaging/linux/50-punktfunk-nice.conf \
"$STAGE/usr/lib/systemd/system/user@.service.d/50-punktfunk-nice.conf"
install -Dm0644 scripts/punktfunk-host.service "$STAGE/usr/lib/systemd/user/punktfunk-host.service"
# The source unit's ExecStart points at the dev source tree; a packaged install has the binary at
# /usr/bin. Rewrite it so a fresh apt install (no hand-rolled unit) starts the installed binary.
@@ -237,6 +242,7 @@ Source: $PKG
Package: $PKG
Architecture: any
Depends: \${shlibs:Depends}
Recommends: rtkit
EOF
# In bundle mode the libav* live in FFMPEG_PREFIX/lib — not a standard loader path, and the
# target/release binary carries no rpath (only the staged copy does) — so dpkg-shlibdeps can't
+13
View File
@@ -0,0 +1,13 @@
# Punktfunk: let the streaming host's data-plane threads renice themselves.
#
# The host raises its capture/encode/send and audio threads to nice -10/-5 so a CPU-saturating
# game (or its shader-compile storm at launch) cannot deschedule them mid-stream. That call needs
# CAP_SYS_NICE or a raised RLIMIT_NICE — and the host binary must never carry a file capability
# (it would make the process unidentifiable to KWin and kill desktop streaming), so the limit is
# the right lever. Desktops with RealtimeKit don't need this file (the host falls back to rtkit);
# it covers rtkit-less installs, from the next login onward.
#
# This raises only the LIMIT for user sessions. Nothing is reprioritized by itself: a process
# still has to call setpriority(), exactly as before.
[Service]
LimitNICE=-15
+9
View File
@@ -497,6 +497,15 @@ in
group = "root";
};
# CPU-side thread priority for the host's data-plane threads (capture/encode/send and the
# 5 ms audio loop) rides RealtimeKit: the host asks rtkit to renice the thread when a direct
# setpriority() is refused — the same unprivileged broker PipeWire clients use, so nothing
# ever enters the host's permitted set and the KWin identification above stays intact.
# NixOS is the one distro family where rtkit is not a given, hence the default here;
# mkDefault so an operator who runs without rtkit can turn it off (the host then keeps its
# pre-0.29 best-effort no-op behaviour, a pacing cost only).
security.rtkit.enable = mkDefault true;
systemd.user.services.punktfunk-host = {
description = "punktfunk GameStream + punktfunk/1 streaming host";
documentation = [ "https://git.unom.io/unom/punktfunk" ];
+17 -2
View File
@@ -123,6 +123,11 @@ Requires: wireplumber
# made the host uninstallable for anyone running real PulseAudio, which serves those games just
# as well. Fedora installs pipewire-pulseaudio by default, so the default box is unaffected.
Recommends: pipewire-pulseaudio
# The data-plane threads renice themselves through RealtimeKit when the direct setpriority() is
# refused (thread_qos — the host binary can never carry CAP_SYS_NICE, see the %%files note).
# Weak-dep: Fedora desktops ship rtkit anyway, and without it the user@.service.d LimitNICE
# drop-in below still covers the direct path from the next login.
Recommends: rtkit
Requires: opus
Requires: libei
# FFmpeg runtime with NVENC (RPM Fusion). Weak-dep so the package installs even if
@@ -371,6 +376,12 @@ sed -i 's#%h/punktfunk/scripts/headless/run-headless-kde.sh#%{_datadir}/%{name}/
install -Dm0644 packaging/linux/io.unom.Punktfunk.Host.desktop \
%{buildroot}%{_datadir}/applications/io.unom.Punktfunk.Host.desktop
# Scheduling headroom for the host's data-plane threads (see the no-caps note in %%files): raise
# the user-session nice hard limit so pf-frame's setpriority() also works where RealtimeKit isn't
# running. A limit, not a grant — takes effect at the user's next login.
install -Dm0644 packaging/linux/50-punktfunk-nice.conf \
%{buildroot}%{_unitdir}/user@.service.d/50-punktfunk-nice.conf
# Status tray: the per-user SNI icon + its XDG autostart entry (self-gating: --autostart exits
# silently for users who don't run a host) + the hicolor status icons it names.
install -Dm0755 target/release/punktfunk-tray %{buildroot}%{_bindir}/punktfunk-tray
@@ -526,8 +537,10 @@ install -Dm0644 scripts/punktfunk-scripting.service %{buildroot}%{_userunitdir}/
# why neither prctl(PR_SET_DUMPABLE, 1) nor systemd AmbientCapabilities= rescues it.
#
# The cost of not having it is pacing only: pf-zerocopy walks REALTIME -> HIGH -> default when a
# priority class is refused, and pf-frame's thread nice is a best-effort no-op. That is exactly
# how 0.25.0 behaved, which is the behaviour that worked.
# priority class is refused, and pf-frame's thread nice falls back to RealtimeKit (the same
# unprivileged broker PipeWire clients use — no capability enters the permitted set, so the KWin
# identification above is untouched) and to the user@.service.d LimitNICE drop-in shipped below.
# Only on a box with neither does it remain the best-effort no-op 0.25.0 shipped with.
#
# rpm applies file capabilities from package metadata, so a package built WITHOUT %caps() installs
# the binary with none and an upgrade from 0.26.0-1 clears it — no scriptlet needed.
@@ -553,6 +566,8 @@ install -Dm0644 scripts/punktfunk-scripting.service %{buildroot}%{_userunitdir}/
# Debugging the WORKER (not the host): a capability makes it AT_SECURE, so the loader ignores
# LD_LIBRARY_PATH/LD_PRELOAD for it and core dumps are suppressed by default.
%caps(cap_sys_nice=ep) %{_bindir}/punktfunk-encode-worker
%dir %{_unitdir}/user@.service.d
%{_unitdir}/user@.service.d/50-punktfunk-nice.conf
%{_bindir}/punktfunk-tray
%{_udevrulesdir}/60-punktfunk.rules
%dir %{_libexecdir}/punktfunk
+11
View File
@@ -313,6 +313,17 @@ if [ "$SUDO_OK" = 1 ]; then
| sudo tee /etc/sysctl.d/99-punktfunk-net.conf >/dev/null
sudo sysctl -q -p /etc/sysctl.d/99-punktfunk-net.conf >/dev/null
ok "UDP socket buffers raised to 32 MB (persisted)"
# Nice-limit headroom for the host's data-plane threads (audio/send): without it (or rtkit,
# which SteamOS does not guarantee) the per-thread renice silently no-ops and a busy game can
# deschedule the 5 ms audio loop. SteamOS's /usr is read-only, so unlike the packaged installs
# this lands in /etc — same drop-in, same effect, from the next login. NEVER a file capability
# on the host binary (see the setcap note above — KWin identification).
if [ -f "$SRC/packaging/linux/50-punktfunk-nice.conf" ]; then
sudo install -Dm644 "$SRC/packaging/linux/50-punktfunk-nice.conf" \
/etc/systemd/system/user@.service.d/50-punktfunk-nice.conf
sudo systemctl daemon-reload || true
ok "nice-limit drop-in installed (data-plane thread priority; applies from next login)"
fi
if [ -f "$SRC/scripts/60-punktfunk.rules" ]; then
sudo install -m644 "$SRC/scripts/60-punktfunk.rules" /etc/udev/rules.d/60-punktfunk.rules
sudo udevadm control --reload-rules && sudo udevadm trigger || true