PyroWave on Linux: the GPU-priority knob never fired, a dmabuf timeout condemned the host, and the jumbo grow was dead code #132
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: unom/punktfunk#132
Reference in New Issue
Block a user
Wave 2 of the Linux host performance program — the PyroWave packages (PW1–PW8), plus the
instruments that had to exist before any of it could be measured.
Design doc:
punktfunk-planning/design/linux-host-performance-wave2-pyrowave.md(its per-packagestatus blocks are updated with every number below, and are the place to read the detail).
Everything here is default-neutral or default-off except PW1's capability grant and PW3's latch
behaviour. Nothing changes what a client negotiates, and no wire format moves.
What each package did, and what it measured
PW2 — a PyroWave session could drop to CPU capture and log nothing at all.
The CPU-fallback warning was gated on
backend_is_vaapi, which reads the host-global encoderpref. A PyroWave session is negotiated per session, so on an NVIDIA/auto host that gate is false
and the session fell out of every arm of the log chain — paying a full-resolution CPU pixel touch
per frame while logging nothing. A degraded host and a healthy one produced identical logs.
Measured on the exact silent configuration: 0 → 2 lines. All three capture arms now announce
themselves (
cuda-import → nvenc,dmabuf-passthrough → pyrowave,cpu → pyrowave).PW1 — the GPU-priority knob had never done anything on Linux.
patches/0005requests an elevated global-priority queue, but it is gatedif (!inherit_info)andonly Windows leaves that null; Linux passes its own create-infos and takes the inherit branch, so
the block was skipped. Wired natively, with the C patch's grammar and a REALTIME→HIGH→none ladder
that can never fail an open.
Then the measurement changed the package: the lever is inert without
CAP_SYS_NICE, on everyvendor — not the RADV-only downgrade the plan predicted. Without it every class is refused; with
it REALTIME is granted first try. So the capability now ships from all six packaging paths (Arch,
deb, RPM
%caps, NixOSsecurity.wrappers, Deck installer, Bazzite sysext at image-build time).Measured under a GRID 2 benchmark loop at 54–87 % GPU, same binary both arms:
p99 down ~30 %, worst frame roughly halved, for ~0.6 ms on the median — the right trade for a
streaming encoder, where the tail is what becomes a visible hitch. This contradicts patch 0005's
only prior datum (RTX 4090 / Windows / WDDM: "did not reduce the spikes"), and its header now records
both results side by side with an explicit "do not delete this patch — the two stacks disagree".
PW3 — one dmabuf timeout condemned every later capture on the host, forever.
The raw-dmabuf latch had two causes sharing one
AtomicBool: an encoder that cannot import what thecompositor allocates (unrecoverable) and an offer that never negotiated (often just a compositor
mid-restart). One timeout meant CPU capture for the rest of the process — including sessions against
a completely different compositor and node. Now: import failures stay sticky, negotiation timeouts
get one retry, a negotiated capture credits the budget back, and both are keyed to a capture
identity so a new node earns a fresh attempt while the same one keeps its verdict. The session-open
line carries the latch state.
PW4 — retired on evidence, exactly as pre-registered.
The package proposed moving the producer-fence wait off the PipeWire loop thread. It is already
free: no producer in the fleet attaches an implicit fence.
Shipped instead: the histogram (kept — it is how anyone re-checks) and a comment correction. The
100 ms budget stays as a guard for a producer that does fence.
PW5 — stages 1–5, overlap deliberately OFF (
max_inflight= 1).The prerequisite analysis found pyrowave's encoder structurally cannot hold two frames in flight
(single wavelet/scratch buffers, and
Impl::encodeopens by discarding them), so "double patch0004's pool" was the wrong fix — the design is two alternated handles. That carries a landmine:
the 3-bit wire sequence counter is per-encoder, so alternating emits
1,1,2,2,3,3…and the decoder(
restart = diff != 0) reads a repeat as more blocks of the same frame — every second framesilently swallowed.
patches/0007exposes a sequence setter; the gate has a negative controlthat reproduces the failure exactly when the patch is removed.
Landed: pool census, buffer headroom request,
submit/pollsplit, doubled per-slot resources(six single-slot resources, not the two the plan named), second handle. Depth 2 was run on hardware
(bit-identical output vs depth 1) but is not shipped — the capture-side tear-hunt needs a
live client and is not done.
PW6 — PyroWave can stream its AU to the wire. Off by default.
Also establishes that newest-wins draining was never the blocker: host STREAMED_AU chunks still
complete as one
Frame; it is client prefix delivery that the entry-counting drain cannotsurvive. So PW6 buys host send-side pipelining, not decode-while-arriving — the "~7 ms" framing
in the original plan is not reachable as scoped.
Loss behaviour, 20 000 frames per cell at 2 % loss with FEC off: streamed and whole-AU are an
exact tie, zero vanished frames either way. But a probe dropping precisely the final block
separates them completely — whole-AU 200 partials / 0 losses, streamed 0 partials / 200 total
losses. Hence the default stays off. On-glass: busy test cards, luma PSNR 40.16 dB, and
byte-identical AU streams across chunk sizes.
PW7a — the shipped jumbo grow was dead code.
quinn caps MTU discovery at the peer's advertised
max_udp_payload_size, which defaults to 1472,and nothing in the repo had ever touched
EndpointConfig. Proven by A/B: server-only opt-in settlesat 1472 B, both ends opted in reaches 8972 B in 5 ms. Fixed, and the session-start grow is
authorised not by memory but by a live re-proof on the connection being welcomed — quinn's
current_mtuonly rises to a probe size that was acked, so a moved laptop cannot inherit a jumboverdict. ⚠ Jumbo now needs the opt-in on both ends; Android/iOS would need a client setting.
PW8 — closed, premise stale. KWin does not cap virtual outputs at 60 Hz; it creates them at 60
and
kwin.rsalready does what PW8 proposed to investigate. The 60 Hz is the rejection fallback.Instruments added (none existed before)
PUNKTFUNK_PERFencode split (mean/p50/p99/max) — every other backend had one; theencoder this program exists to protect was the one you could not measure.
spike --codec pyrowave+--wire-chunk, andtools/loss-harness.Review notes
one: PW6 guards
poll()while a chunk cursor is open, PW5 moved the fence wait intopoll().Either side alone silently breaks the other; both are kept, guard first.
patches/0007re-applies clean to a pristine tree;git diff crates/pyrowave-sys/vendor/touches only intended files.produce three different AU hashes (the 4:2:0 encoder emits run-varying bytes the decoder ignores).
The real gate is decode identity, and it holds.
docs-site/content/docs/configuration.md+running-as-a-service.md.Gates
Full CI parity on a Linux box, on the merged result, not just per branch:
cargo fmt --check;workspace
clippy -D warnings;pf-encodeclippy withnvenc,vulkan-encode,pyrowaveandwithout; workspace build; workspace tests. All green.
Still owed (all hardware-gated)
worst case — it downclocks to 180 MHz of 3090, so idle timings are ~3× slow, not fast.)
Found while taking PW2's on-glass measurement, and it is what made the measurement possible. `spike` built its capture request from `OutputFormat::resolve`, the constructor shared with the GameStream path, which hard-codes `pyrowave: false` ("GameStream never negotiates PyroWave"). On Linux that flag is not cosmetic: `capture_virtual_output` feeds it to `zero_copy_policy` as `ZeroCopyPolicy::pyrowave_session`, which is what puts the capture on the raw-dmabuf passthrough. So `--codec pyrowave` opened a PyroWave encoder over a capture negotiated for a different consumer, and the only way to exercise the real path was the host-global `PUNKTFUNK_ENCODER=pyrowave` lever. That lever cannot stand in for the per-session flag, which is the part that matters here: it resolves the backend to `Pyrowave`, and `linux_zero_copy_is_vaapi_for` returns true for that — so it ALSO flips `backend_is_vaapi` on. A per-session PyroWave negotiation on an auto/NVENC host, where `backend_is_vaapi` is false, was therefore unreachable from the CLI — and that is exactly the configuration whose CPU downgrade logged nothing at all. The spike now sets the flag from its own codec, the same comparison `session_plan::output_format` makes for a real session. With it, the before/after on .21 is unambiguous: origin/main logs zero capture-path lines on that configuration, this branch logs two (the resolved arm, and the named downgrade with its cause and fix).Wave-2 PW3. The raw-dmabuf passthrough has two very different reasons to switch itself off, and they shared one `AtomicBool`: * the encoder repeatedly failed to import what this compositor allocates — unrecoverable, a driver fact, and the reason this latch was written (it stops the encode-stall recovery rebuilding the same doomed encoder five times and then ending the session, on every connection, forever); * the dmabuf-only capture offer never negotiated — which can simply mean the compositor was mid-restart. Sharing the flag made the second as permanent as the first. One timeout, and EVERY later session on the host captured CPU frames until the process was restarted — including sessions against a completely different compositor and a different node, which had never failed at anything. Nothing said so; the arm line PW2 added would have shown `cpu` with no explanation. Now the two causes have the lifetimes they should have, in a `RawDmabufLatch` that owns both: * Import failures stay sticky. Unchanged threshold (3 consecutive), unchanged hazard coverage. * Negotiation timeouts get a retry budget of 2 — one retry, deliberately small: each failure costs a ~10 s stall, so a larger budget is paid by the user in dead air. One retry survives the mid-restart transient; a compositor that genuinely never accepts keeps the same identity, so it latches on the second try, one extra stall per host lifetime versus the old behaviour. * A capture that negotiates credits the budget back, so an evening of reconnects against a compositor that failed once cannot accumulate its way into a latch. * BOTH are keyed to a capture identity (node id + portal bit). A new node — fresh virtual output, compositor restart, the Bazzite Gaming↔Desktop switch — is a genuinely different question and earns a fresh dmabuf attempt instead of inheriting a verdict about something else. The SAME capture keeps its verdict, which is what preserves the 10 s-stall protection the latch exists for. The session-open line now carries the latch state, so `cpu` is no longer ambiguous between "this host was never going to do dmabuf" and "something failed earlier and we are still living with the verdict" — only the second is a bug worth chasing, and only the second is now visible as one. Atomics rather than a lock because `note_import_ok` is on the per-frame import path; everything else runs at pipeline build or on failure. The state machine is tested against a local instance rather than the process-wide static — seven tests covering both lifetimes, the identity clear, the same-identity hold, the budget credit, and the cause naming. One honest note on the identity: it is the PipeWire node id, not the "(compositor-id, modifier list)" pair the design sketched. Node id is what capture actually has at that point, and it changes on exactly the events that matter here (new virtual output, compositor restart, session switch). Keying on the modifier list too would need the list before the importer is built, which is the wrong order.Wave-2 PW4, step one of one-so-far. The package's own first line is "investigation step first (measure, then decide)", and it is pre-registered to be ABANDONED if the wait's p99 is ~0 — so the instrument ships before the change, not after. A per-session histogram of `wait_read_ready`, taken on the PipeWire loop thread, which is exactly where the wait is expensive: that thread is the compositor's consumer, so time blocked there delays buffer recycling for the NEXT frame. Logged under PUNKTFUNK_PERF at the same cadence and gate the encode backends use for their submit splits, so a perf run reads as one instrument: samples, mean, max, p50/p99 bucket, and the Signaled/NoFence/TimedOut/failed split. Buckets are coarse on purpose (100us -> 10ms, plus overflow). The decision this feeds is binary — a p99 in the first bucket means the wait is already free and PW4 becomes a comment correction; a p99 past 1ms is a real stall against a 16.6ms frame budget. Edges are placed so those two worlds cannot be confused, and anything past the last edge reports as overflow rather than clamping into the top bucket ("worse than 10ms" is a distinct finding). The outcome split sits next to the timings because "the wait is short" and "there is nothing to wait for" are different results with different consequences, and one data point already shows the second: on gamescope/NVIDIA the probe reports NoFence, i.e. that producer attaches no implicit fence at all. 5 tests pin the arithmetic, including that an empty histogram reports "no answer" rather than a decisive-looking zero — the failure mode that would retire the package on no evidence. No behaviour change: the wait still happens where it always did. Gates green at CI parity.Wave-2 PW5 stage 1, and the one stage with no risk at all. The zero-copy capture path dups the dmabuf fd, publishes the frame, and hands the SPA buffer straight back to the producer at `.process` return — while the encode thread has not yet imported it, let alone read it. The code says so itself ("content stability across the brief import/encode window relies on the compositor's buffer-pool depth, like any zero-copy capture"). That depth is therefore load-bearing: it is the ONLY thing standing between us and the producer overwriting a buffer mid-read. And it had never been measured. Not logged, not asserted, not even requested — `build_dmabuf_buffers` set `SPA_PARAM_BUFFERS_dataType` and nothing else, so whatever the producer picked is what we got, silently. This adds the `add_buffer`/`remove_buffer` stream callbacks PipeWire has always offered and logs the count once per distinct depth: `pool_depth`, `high_water`, and the latest-frame-only `drained` count beside it. One line per session on a stable pool (`.process` runs at the capture rate — an unconditional log would be 240 lines a second of the same number), a second line if a renegotiation changes the depth. `high_water` is tracked separately from `live` because a renegotiation frees the pool before re-allocating it: any decision keyed on the live count would read that dip as "the pool shrank". `remove` saturates at zero rather than wrapping, so an unmatched remove cannot report `u32::MAX` buffers. Measurement only — no behaviour change, and no consumer of the number yet. PW5's later stages need it (a deeper encode pipeline widens the overwrite window by a full frame period), but the number is worth having regardless of whether those stages ever land: it is the answer to "is our zero-copy capture actually safe on this compositor", and until now the honest answer was "nobody knows". 3 tests pin the once-per-depth logging, the renegotiation dip, and the saturating remove. Gates green at CI parity.Wave-2 PW5 stage 3. Depth is STILL 1; this is the shape change alone. `encode_frame` recorded CSC+encode, queue-submitted, waited the fence and packetized, all inside `Encoder::submit`. Every other backend in this crate puts the wait on the POLL side. That difference is the whole reason the host loop's cadence folds around this encoder: with the wait inline, `submit` returns only after the GPU is done, so the arrival-anchored floor absorbs the encode only while it stays under 0.9x the frame interval. Split into `submit_frame` (ingest -> CSC -> pyrowave encode -> queue-submit -> return) and `wait_and_packetize` (fence wait -> packetize -> AU), with an `InFlight` deque between them capped by `max_inflight`, which is 1. **One is the only value the resources can support today** — `cmd`, `fence`, `csc_set` and the y/uv images are one each, so a second concurrent frame would record into a PENDING command buffer and storage-write images pyrowave is still sampling. `submit` therefore drains to `max_inflight - 1` before recording, which states that invariant in one place instead of leaving it implicit in "the encode is synchronous". The subtle part is the command-buffer state machine, and it is unchanged: the record-and-submit closure still resets `cmd` on every PRE-submit failure (RECORDING/INVALID/EXECUTABLE, never PENDING), and the fence wait still does NOT reset on failure, because a timeout leaves the buffer PENDING where a reset violates VUID-vkResetCommandBuffer-commandBuffer-00045. What changed is that a failed wait now also leaves the entry IN FLIGHT — which is precisely what tells `reset()` there is live GPU work to re-wait before the pyrowave encoder object may be destroyed. `gpu_pending` is gone; `!inflight.is_empty()` is the same fact, and cannot drift from it. The split opened two windows that did not exist when everything ran inline, both closed here: `reconfigure_bitrate` and `set_wire_chunking` can now land BETWEEN a submit and its poll, so the packetize boundary and the bitstream cap are snapshotted into `InFlight` at submit time. Reading the live fields would have let a mid-flight bitrate drop turn a perfectly good frame into "unexpected packet count", and a mid-flight chunking change into an AU with the wrong `chunk_aligned` flag. `flush()` is no longer a no-op — it drains the in-flight frame, so the trait's poll-until-None contract still returns every AU (the `spike` subcommand and the hardware smoke tests are the real users). The perf instrument still measures submit->AU, stamped at submit and taken when the AU becomes readable, so `92326312`'s numbers stay directly comparable; the log line now carries `depth` and says plainly that above depth 1 the number legitimately grows by about one loop period. VERIFIED ON GLASS (.21, RTX 5070 Ti, GPU idle at 180 MHz of 3090 — so these are slow-clock runs, which is the worst case on this card, not the best): all 6 `#[ignore]`d GPU tests pass — the 4:2:0, 4:4:4 and 24-bpp PSNR smokes, the mode-mismatch refusal, the fd-leak check and the golden dump. Byte-identity, honestly: the AU bytes are NOT reproducible, and were not before this commit either. Three runs of the SAME unmodified binary produced three different `au-dense.bin` hashes (ab7ecaf6 / 8735700e / 933b3d40) — the vendored 4:2:0 encoder emits run-varying bytes that the decoder ignores. So the meaningful gate is DECODE identity, and that holds exactly: every decoded plane (`ref-dense-{y,cb,cr}`, `ref-chunked-{y,cb,cr}`, `ref-dense444-{y,cb,cr}`) is bit-identical between the pre-split base and this commit, across four runs. 4:4:4 AUs are additionally bit-stable and match the checked-in Apple fixture exactly. Gates green at CI parity.Wave-2 PW5 stage 4. Pure capacity — `max_inflight` is STILL 1, nothing overlaps yet. The plan named the y/uv images as the thing to double. Reading the backend found five more, and each is a correctness problem under overlap rather than a performance one: * `csc_set` — ONE descriptor set, rewritten every frame by `bind_rgb`. Updating a set still bound by a PENDING command buffer violates VUID-vkUpdateDescriptorSets-None-03047, and on most drivers that is a wrong picture rather than an error. * `y_img`/`uv_img` — the CSC of N+1 storage-writes exactly the images pyrowave is still sampling for N. The barrier comment ("the previous frame's encode already completed under our synchronous fence") was load-bearing and said so. * `cursor_img` + `cursor_stage` — the struct comment stated the assumption outright: *"Single (not ring) because PyroWave encodes one frame synchronously — no in-flight overlap to race."* * `cmd` + `fence` — you cannot record into a PENDING command buffer at all. * `cpu_img`/`cpu_stage` (software capture / tests) — the host writes staging while the previous frame's copy is still pending. All of it moves into a `Slot`, and the encoder now owns `SLOTS` of them. Two, because Granite caps the overlap at two for us: the pyrowave device defaults to `init_frame_contexts(2)` and `next_frame_context()` — called at the top of every `encode_gpu_synchronous` — waits the context it rotates into. A third slot would need a vendored `init_frame_contexts(3)` that is not exposed. `bitstream` and `import_cache` are deliberately NOT per-slot, and the `Slot` doc says why so a later sweep does not "fix" it: `bitstream` is only touched during packetize, i.e. only on the poll side one frame at a time, and `import_cache` retaining the VkImage/VkDeviceMemory per dmabuf inode is precisely what makes it safe for two slots to sample the same imported buffer. `cpu_expand` is shared for the same reason — it is copied into staging before `submit_frame` returns, so no GPU work ever reads it. Each frame carries its slot index in `InFlight` rather than recomputing it, so `wait_and_packetize` cannot wait the wrong fence — the failure that would look like corruption rather than an error. `reset()` now waits EVERY in-flight fence, not just one, which matters the moment depth rises. WHAT IT COSTS, measured from the driver's own memory requirements rather than estimated (.21, RTX 5070 Ti, and there is now an `#[ignore]`d test that prints it on any GPU): 1080p 4:2:0 3872 KiB per slot 7744 KiB for both 4K 4:2:0 12992 KiB per slot 25984 KiB for both 4K 4:4:4 24992 KiB per slot 49984 KiB for both So the extra slot costs ~3.8 MiB at 1080p and ~24 MiB at 4K 4:4:4 — an order of magnitude under the plan's ~25-35 MB / 100-150 MB estimate, because that estimate included pyrowave's internal wavelet and scratch buffers, which stage 5's second encoder handle will add and this stage does not. Affordable on an iGPU. The open line now logs `slots`, `slot_kib` and `slots_kib` so this is visible per session and not only in a test. VERIFIED ON GLASS (.21, GPU idle at 195 MHz of 3090 — slow-clock, the worst case on this card): all 6 `#[ignore]`d GPU tests pass, and all NINE decoded-plane hashes (`ref-dense-{y,cb,cr}`, `ref-chunked-*`, `ref-dense444-*`) are bit-identical to the pre-PW5 base. Decode identity is the meaningful gate here — the raw AU bytes are not reproducible run-to-run even from an unmodified binary, which stage 3's message documents. Gates green at CI parity.