Audio was late because nothing ever placed it against the picture #101

Merged

enricobuehler merged 5 commits from audio/latency-overhaul into main 2026-08-07 22:32:11 +00:00

Conversation 0 Commits 5 Files Changed 30

+2021 -85

5 Commits

Author	SHA1	Message	Date
enricobuehler	bfed711921	Merge remote-tracking branch 'origin/main' into audio/latency-overhaul	2026-08-08 00:16:02 +02:00
enricobuehler	c43769282a	fix(apple): place audio with the picture instead of wherever the ring settles ... The Apple half of the A/V sync overhaul; the Rust half is 12a53183 and this mirrors its policy rather than re-deriving one. The host stamps `pts_ns` on every audio datagram and the client decoded it into `AudioPCM` — and then never read it. Video's `pts_ns` is used end to end (the end-to-end meter computes a true glass-to-glass `displayed + clockOffset − pts` per presented frame), so audio free-ran at whatever depth its jitter ring happened to reach, video was presented on an independent path, and nothing ever compared them. The A/V offset was an accident of buffer depths: it moved whenever the ring ratcheted under underrun pressure, and it got WORSE every time video got faster, because a quicker decoder lowers the video leg and leaves audio's exactly where it was. Video is the master: audio_e2e = (now + buffered_ahead + clock_offset) − pts_ns av_offset = audio_e2e − video_e2e (> 0 ⇒ audio behind the picture) `AvSync` smooths that with an EWMA, ignores what sits inside a deadband no listener can detect, refuses the implausible outright rather than clamping it (a wall-clock step must not steer the ring), and proposes a depth. Swift refuses one thing Rust does not have to: the arithmetic itself. The Rust controller works in i128, while Swift has no Int128 at this tools version, so the terms are combined with overflow-REPORTING arithmetic instead of the `&-` the latency meters use. That is not defensive padding — `ptsNs = 1 << 63` reads as `Int64.min`, the difference lands on exactly `Int64.min`, and `abs()` of that has no representable result, so checking the overflow flags AFTER the sanity limit does not mis-measure the stream, it aborts the process from the audio drain thread. The guard's short-circuit ordering is what makes the sanity check safe to run at all. Continuity outranks sync, always. `AudioRing.setSyncTarget` only ever takes a REQUEST, clamped between the existing underrun-driven floor and the hard cap. A link whose jitter genuinely needs more buffer than the picture is away keeps its buffer and the residual is reported. `nil` is the default and reproduces the previous behaviour exactly. The clamp raises its ceiling to the floor rather than using it as-is: a device whose callback quantum alone exceeds the hard cap makes floor > cap, and a plain `min(max(s, floor), cap)` would then hand back the CAP — quietly below the continuity floor, inverting the exact ordering this exists to guarantee, on the awkward hardware it exists to survive. (Rust's `Ord::clamp` announces that condition by panicking; Swift would just get it wrong, which is worse.) The reference is the other half, and without it the loop is inert — which is why this was split out rather than shipped alongside the Rust side. `LatencyMeter` now publishes its most recent sample as a LEVEL, so the end-to-end meter the presenter already writes per presented frame becomes the video figure the audio plane reads. Both present paths (arrival and deadline) feed it without either knowing audio exists, and the stage-1 fallback presenter — which stamps no present at all — offers nothing, so the loop correctly declines to correct. The level EXPIRES, unlike the Rust atomic: this client has a backgrounded keep-alive that keeps audio playing and drops video decode entirely, and a reference with no expiry would go on steering the ring against a figure minutes old and frozen. And the reason none of this was visible: `bufferedMS`/`targetMS` existed only in a periodic log line, absent from anything a surface could render. The HUD's detailed tier now carries `audio buffer N ms · a/v ±N ms` and the 1 Hz stats log gains the same pair, appended last so existing parsers are unaffected — both numbers, because a deep ring on a jittery link is correct and only the offset separates that from audio held late. `PUNKTFUNK_NO_AV_SYNC=1` disarms the loop without a rebuild, as on the Rust clients. Verified: swift build + 225 tests (5 skipped) green. Every new gate was proven non-vacuous by planting its own defect and confirming the gate caught it — the continuity invariant, the clamp inversion, the deadband, both refusal paths, the evidence threshold, the sync-pressure relax, the reference's staleness and its survival of a drain, and `setSyncTarget` being live at all rather than dead code, which is how the previous pass in this area shipped a correction that was structurally unreachable with a green test. Two gates came back VACUOUS on the first sweep and are the reason their inputs look so specific: the overflow test was being caught by the sanity limit instead of the overflow guard, and the refused-reference test was being caught by `latestSample`'s own `> 0` check rather than by where the publish sits.	2026-08-08 00:10:48 +02:00
enricobuehler	74270109dd	ci(android): lint the Android target, which nothing had ever done ... `ci.yml` runs `cargo clippy --workspace` on the HOST, where `clients/android/native` and every `#[cfg(target_os = "android")]` module elsewhere compile out, and `android.yml` only ever built. So the Android target was never linted at all — not once. Five lints were sitting in clients/android/native when this was noticed, in code no gate had ever read. The gate is a Gradle task rather than a YAML step because cargo-ndk needs a specific discovery environment (NDK sysroot, SDK cmake 3.22.1 for libopus, `LIBOPUS_STATIC`, Ninja) and duplicating it into the workflow would let the lint drift from the build — a lint that ran against a different toolchain is a lint about a different program. `registerCargoNdkClippy` reuses the build task's environment verbatim via the extracted `cargoNdkEnvironment`, so local and CI runs are the same invocation. It lints BOTH pointer widths, and that is load-bearing rather than thorough: arm64-v8a is 64-bit and armeabi-v7a is 32-bit, so a cast that is redundant on one can be required on the other. Linting only the primary ABI would license "fixes" that break the 32-bit build — the shipping ABI for the many 32-bit Google TV / Android TV boxes this client targets. x86_64 is skipped: it is emulator-only and shares its width with arm64, so it costs lint time for no signal the other two do not already carry. The five resident lints: * `audio.rs` / `mic.rs` `type_complexity` — the open-attempt closures now return named `OpenedPlayback` / `OpenedCapture` aliases. The two tuples are mirror images of each other (playback sends, capture receives), which the aliases now say out loud. * `vsync.rs` ×2 `unnecessary_cast` — **not** taken. `timespec`'s fields are 32-bit on armv7 and 64-bit on arm64, so the casts are REQUIRED on one shipping ABI and redundant on the other; following the suggestion would break the 32-bit build. `i64::from`/`.into()` do not escape it either, they trade `unnecessary_cast` for `useless_conversion` on the 64-bit side. Answered with a documented `#[allow]` at the expression instead of in whichever build breaks first. * `pad_audio.rs` `needless_range_loop` — iterator form, preserving the `channels < 2` no-op the range had. Verified: `:kit:cargoNdkClippy` green on both ABIs, host-lane clippy for the crate still clean, `cargo fmt --all --check` clean. The gate was proven non-vacuous by planting `1i32 as i32` in an android-only module and confirming it fails the task, then reverting.	2026-08-08 00:01:39 +02:00
enricobuehler	70e6b80200	fix(client/android): place audio with the picture on Android too ... The core, Linux, Windows and host halves of the audio latency overhaul landed with Android deliberately left inert: `JitterPolicy`'s sync target defaults to `None`, so this ring kept behaving exactly as it always had. What was missing was not the loop but its REFERENCE — nothing here published where a frame actually reached glass, and a controller with no reference is the mechanism you can prove is present but that cannot act. This wires both halves. The decode thread now reads the host capture `pts_ns` that every `AudioPacket` has always carried and that this client, like every other, dropped on the floor. Against the ring depth (published by the AAudio callback through the shared `AudioSyncCell`) and the video plane's end-to-end figure it computes audio_e2e = (now + buffered_ahead + clock_offset) − pts_ns av_offset = audio_e2e − video_e2e (> 0 ⇒ audio behind the picture) and asks the ring for a depth that closes it. Only ASKS: `set_sync_target` is clamped between the underrun-driven adaptive floor and the hard cap, so a link whose jitter genuinely needs more buffer than the picture is away keeps its buffer and the residual is reported instead of being taken out of the listener's stream. Continuity outranks sync, on this ring as on the others. The reference comes from `DisplayTracker`'s `OnFrameRendered` callback — the one place in the client that knows a frame truly latched — and it is computed ABOVE the HUD gate now. A sync loop that only ran while the overlay was up would be off on exactly the devices that report latency; the stats LOCK stays gated, which is what that early-return was really protecting. Both decode loops feed it, so sync works with "Low-latency mode" off as well. Two deliberate refusals: * The figure is published RAW. The HUD shaves the OS present floor off its shown display/end-to-end numbers — metrics report what Punktfunk controls — but sound has to reach the ear when the light reaches the eye, and a floor-shaved reference would place audio a whole latch period early on every device. * Below API 33 there is no render callback, so there is no confirmed present and the loop stays inert (target `None` ⇒ today's behaviour exactly). The release instant is NOT substituted for it: a release targets a FUTURE vsync and runs a whole latch period (8-21 ms measured) ahead of glass, well outside the loop's deadband — it would place audio early on every frame while looking like it was working. The plane is also no longer invisible. Ring depth and the smoothed offset ride the stats array at 33/34 and the Detailed HUD carries `audio buffer N ms · a/v ±N ms`, the same wording the desktop HUD uses — both numbers, because a deep ring on a jittery link is correct behaviour and only the offset separates that from audio simply held late. The 1 Hz logcat line gains `av_ms` beside its depth, and the depth itself now has ONE publisher: the counter copy is gone in favour of the sync cell both readers already share. The escape hatch is two levers. `PUNKTFUNK_NO_AV_SYNC=1` keeps the contract the desktop clients document, but an app launched from the launcher inherits no environment, so the one a field tester can actually reach is `adb shell setprop debug.punktfunk.no_av_sync 1` — no rebuild, exactly like `debug.punktfunk.presenter`. A loop that steers playback has to be bisectable on the device that reports the regression. Verified: `cargo ndk -t arm64-v8a check` clean; `cargo clippy -p punktfunk-client-android --all-targets -- -D warnings` clean on the host lane CI lints, and the Android target introduces no new findings (5 pre-existing lints in audio/mic/pad_audio/vsync are unchanged — the android-gated modules are never linted by the host workspace); `cargo fmt --all --check` clean; `./gradlew :app:testDebugUnitTest` green. The new HUD test was proven non-vacuous by planting the defect first — dropping the render call fails its three positive assertions and leaves the three absence assertions passing, which is the shape a test that "passes for the wrong reason" would not have. design/audio-latency-overhaul.md W4. Apple (W6) still keeps today's behaviour.	2026-08-07 23:51:15 +02:00
enricobuehler	12a5318397	fix(audio): place audio with the picture instead of wherever the ring settles ... The host stamps `pts_ns` on every audio datagram and the client decoded it into `AudioPacket` — and then never read it. Video's `pts_ns` is used end to end (the presenter computes a true glass-to-glass `displayed + clock_offset − pts`), so audio free-ran at whatever depth its jitter ring happened to reach, video was presented on an independent path, and nothing ever compared them. The A/V offset was an accident of buffer depths: it moved whenever the ring ratcheted under underrun pressure, and it got WORSE every time video got faster, because a quicker decoder lowers the video leg and leaves audio's exactly where it was. That is what a field report on the Steam Deck heard as "the audio delay is way too high", and it is why shaving milliseconds off the audio budget had not helped. Video is the master. In a game streamer the video leg is the input-feel budget and must never be inflated to satisfy the audio clock, while audio tolerates small crossfaded corrections that are inaudible — and `crossfade_drop` already applies them. So audio moves: audio_e2e = (now + buffered_ahead + clock_offset) − pts_ns av_offset = audio_e2e − video_e2e (> 0 ⇒ audio behind the picture) `AvSync` smooths that with an EWMA, ignores what sits inside a deadband no listener can detect, refuses the implausible outright rather than clamping it (a wall-clock step must not steer the ring), and proposes a depth. Continuity outranks sync, always. `JitterPolicy::set_sync_target` only ever takes a REQUEST, clamped between the existing underrun-driven floor and the hard cap. A link whose jitter genuinely needs more buffer than the picture is away keeps its buffer and the residual is reported — sync can never starve the ring into dropouts. `None` is the default and reproduces the previous behaviour exactly, so the four client rings can adopt this one at a time without diverging. Two upstream defects found on the way, both prerequisites: * The host stamped `pts_ns` at ENCODE time, inside the loop draining an already-accumulated chunk, so every frame of a chunk carried near-identical timestamps describing when we got round to encoding. Harmless while nothing consumed it; a sync loop regulating against it would regulate against a fiction. It now comes off the capture clock. * The host did not pace. One capture callback hands over a whole quantum — 5 ms when the graph honours our ask, 21.3 ms on a VM, where stock PipeWire raises `min-quantum` to 1024 — and the loop drained all of it into back-to-back `send_datagram` calls. The wire carried a 4-5 frame burst then ~21 ms of nothing, and a ring can only absorb that by standing a burst period deep. Frames now leave on the audio clock, which costs no average latency. And the reason none of this was visible: `buffer_ms`/`target_ms` existed only as a `tracing::debug!` line, absent from `Stats`. On a Deck the client runs under Steam's `reaper` with stdout on a pipe nobody can read, so the one number identifying a deep ring was unobtainable on the device reporting the latency. The HUD now carries `audio buffer N ms · a/v ±N ms` — both, because a deep ring on a jittery link is correct and only the offset separates that from audio held late. The host also reports its negotiated quantum against the one it asked for, per capture open rather than once per process. Verified: 364 core + 40 presenter tests on Linux, clippy -D warnings clean on punktfunk-{core,host} + pf-{client-core,presenter}, fmt clean. New tests pin the safety invariant (sync cannot pull the target below the continuity floor on any preset), that `None` leaves the policy bit-identical, and that a device quantum exceeding the hard cap does not panic `Ord::clamp` inside a realtime callback. Android and Apple keep today's behaviour (the `None` default) until their presenters publish a video figure to align against; design/audio-latency- overhaul.md carries the plan.	2026-08-07 23:33:45 +02:00