Audio was late because nothing ever placed it against the picture #101
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: unom/punktfunk#101
Reference in New Issue
Block a user
Field report from a Steam Deck (client
.253← host.21), noticed while testing the new FFmpeg-free decoder: "the audio delay seems way too high."Plan:
punktfunk-planning/design/audio-latency-overhaul.md(planning PR #4).The theory that was wrong, first
The host is a KVM VM, and stock
pipewire.confraisesdefault.clock.min-quantumto 1024 inside a VM. Our capture asks for 5 ms and was silently clamped to 21.3 ms — proven by the host's ownaudio first capture buffer samples=2048. The send loop then had no pacing, so each chunk went out as a 4–5 datagram burst, andJitterTuning::PIPEWIRE's 15 ms prime sits below that burst period: a structural guarantee of underruns, driving the adaptive floor toward its 60 ms ceiling.Every step of that is true. It is not what was heard. The A/B —
pw-metadata -n settings 0 clock.min-quantum 256plus a host restart (required: the capturer is parked and reused across sessions, so metadata alone changes nothing) — producedsamples=512withdelivered_pct=100 dropped_chunks=0, and the verdict on glass was "not really improved at all." ~16 ms of ~90 recovered.Recorded in the design doc so nobody re-runs it. A mechanism you can prove is present is not thereby the one that dominates.
The actual cause
The host stamps
pts_nson every audio datagram.datagram_task.rsdecodes it intoAudioPacket. The client never read it. Video'spts_nsis used end-to-end — the presenter computes a true glass-to-glassdisplayed + clock_offset − pts.grep -riE "av_sync|lip.?sync"overcrates/returned nothing.So audio free-ran at whatever depth its jitter ring settled at, video was presented on an independent path, and nothing ever compared them. The A/V offset was an accident of buffer depths — it moved whenever the ring ratcheted, and it got worse every time video got faster, because a quicker decoder lowers the video leg and leaves audio's exactly where it was. Which is precisely why testing the new decoder surfaced it, and why shaving milliseconds off the audio budget had not helped.
audio-quality-and-latency.mdhad already named this in its own summary back on 2026-08-04 — "thepts_nsthe host stamps on every audio datagram is consumed by no client, so audio can trail video by 50–140 ms" — and deferred it as Phase 5.Architecture: video-master, audio steered
Video is master: in a game streamer the video leg is the input-feel budget and must never be inflated to satisfy the audio clock, while audio tolerates small crossfaded corrections that are inaudible — and
crossfade_dropalready applied them.AvSyncsmooths with an EWMA, ignores what sits inside a deadband no listener can detect, and refuses implausible observations outright rather than clamping them (a wall-clock step must not steer the ring).The invariant to review against: continuity outranks sync.
JitterPolicy::set_sync_targetonly ever takes a request, clamped between the existing underrun-driven floor and the hard cap. A link whose jitter genuinely needs more buffer than the picture is away keeps its buffer, and the residual is reported rather than taken out of the listener's stream.Noneis the default and is bit-identical to today, which is what let the four rings adopt this one at a time.Commits
12a5318AvSync/AudioSyncCell+JitterPolicysync target; Linux + Windows rings; presenter publishes the video leg; HUD; host:pts_nsoff the capture clock (was stamped at encode time, so every frame in a chunk shared a timestamp — a sync loop on that regulates against a fiction) and the send loop now paces70e6b807427010c437692bfed711main(26 commits, incl. the gyro program) + full re-verificationTwo refusals worth reading
Android publishes the raw glass figure, not the floor-shaved one. The HUD subtracts the measured latch p50 from what it displays; feeding the sync loop that number would place audio a whole latch period (~8–21 ms on real panels) early on every device. Shaving is a reporting policy; the ear/eye coincidence is physics. Below API 33 the loop stays inert rather than steering on the buffer-release instant, which targets a future vsync.
vsync.rs's twounnecessary_castlints were not taken.timespec's fields are 32-bit on armv7 and 64-bit on arm64, so the casts are required on one shipping ABI. Following clippy would break the 32-bit build — the ABI for the 32-bit Google TV boxes. This is why the new gate lints both pointer widths; a single-ABI gate would have licensed the break.Verification (re-run on the merged tree)
Main moved 26 commits under this branch and overlapped it on two Apple files, so everything below was re-run after merging rather than trusted from before:
.21): 366 core + 40 presenter tests;clippy -D warningsclean onpunktfunk-{core,host}+pf-{client-core,presenter}:kit:cargoNdkClippygreen on both ABIs — including main's newly merged Android code, which the gate had never seen beforeswift buildclean;swift test232 tests, 5 skipped, 0 failurescargo fmt --all --checkcleanNot verified
None of this has run on glass. The HUD's new
a/vfigure is the instrument; the verdict is the reporter's own Deck ←.21path, with the offset converging toward zero while video latency moves underneath it. Given the disproved theory above, that is deliberately not being called done yet.Escape hatch for bisecting a field regression without a rebuild:
PUNKTFUNK_NO_AV_SYNC=1, plusdebug.punktfunk.no_av_syncon Android (an app launched from the launcher inherits no environment).The host stamps `pts_ns` on every audio datagram and the client decoded it into `AudioPacket` — and then never read it. Video's `pts_ns` is used end to end (the presenter computes a true glass-to-glass `displayed + clock_offset − pts`), so audio free-ran at whatever depth its jitter ring happened to reach, video was presented on an independent path, and nothing ever compared them. The A/V offset was an accident of buffer depths: it moved whenever the ring ratcheted under underrun pressure, and it got WORSE every time video got faster, because a quicker decoder lowers the video leg and leaves audio's exactly where it was. That is what a field report on the Steam Deck heard as "the audio delay is way too high", and it is why shaving milliseconds off the audio budget had not helped. Video is the master. In a game streamer the video leg is the input-feel budget and must never be inflated to satisfy the audio clock, while audio tolerates small crossfaded corrections that are inaudible — and `crossfade_drop` already applies them. So audio moves: audio_e2e = (now + buffered_ahead + clock_offset) − pts_ns av_offset = audio_e2e − video_e2e (> 0 ⇒ audio behind the picture) `AvSync` smooths that with an EWMA, ignores what sits inside a deadband no listener can detect, refuses the implausible outright rather than clamping it (a wall-clock step must not steer the ring), and proposes a depth. Continuity outranks sync, always. `JitterPolicy::set_sync_target` only ever takes a REQUEST, clamped between the existing underrun-driven floor and the hard cap. A link whose jitter genuinely needs more buffer than the picture is away keeps its buffer and the residual is reported — sync can never starve the ring into dropouts. `None` is the default and reproduces the previous behaviour exactly, so the four client rings can adopt this one at a time without diverging. Two upstream defects found on the way, both prerequisites: * The host stamped `pts_ns` at ENCODE time, inside the loop draining an already-accumulated chunk, so every frame of a chunk carried near-identical timestamps describing when we got round to encoding. Harmless while nothing consumed it; a sync loop regulating against it would regulate against a fiction. It now comes off the capture clock. * The host did not pace. One capture callback hands over a whole quantum — 5 ms when the graph honours our ask, 21.3 ms on a VM, where stock PipeWire raises `min-quantum` to 1024 — and the loop drained all of it into back-to-back `send_datagram` calls. The wire carried a 4-5 frame burst then ~21 ms of nothing, and a ring can only absorb that by standing a burst period deep. Frames now leave on the audio clock, which costs no average latency. And the reason none of this was visible: `buffer_ms`/`target_ms` existed only as a `tracing::debug!` line, absent from `Stats`. On a Deck the client runs under Steam's `reaper` with stdout on a pipe nobody can read, so the one number identifying a deep ring was unobtainable on the device reporting the latency. The HUD now carries `audio buffer N ms · a/v ±N ms` — both, because a deep ring on a jittery link is correct and only the offset separates that from audio held late. The host also reports its negotiated quantum against the one it asked for, per capture open rather than once per process. Verified: 364 core + 40 presenter tests on Linux, clippy -D warnings clean on punktfunk-{core,host} + pf-{client-core,presenter}, fmt clean. New tests pin the safety invariant (sync cannot pull the target below the continuity floor on any preset), that `None` leaves the policy bit-identical, and that a device quantum exceeding the hard cap does not panic `Ord::clamp` inside a realtime callback. Android and Apple keep today's behaviour (the `None` default) until their presenters publish a video figure to align against; design/audio-latency- overhaul.md carries the plan.