fix(client/android): place audio with the picture on Android too
The core, Linux, Windows and host halves of the audio latency overhaul landed with Android deliberately left inert: `JitterPolicy`'s sync target defaults to `None`, so this ring kept behaving exactly as it always had. What was missing was not the loop but its REFERENCE — nothing here published where a frame actually reached glass, and a controller with no reference is the mechanism you can prove is present but that cannot act. This wires both halves. The decode thread now reads the host capture `pts_ns` that every `AudioPacket` has always carried and that this client, like every other, dropped on the floor. Against the ring depth (published by the AAudio callback through the shared `AudioSyncCell`) and the video plane's end-to-end figure it computes audio_e2e = (now + buffered_ahead + clock_offset) − pts_ns av_offset = audio_e2e − video_e2e (> 0 ⇒ audio behind the picture) and asks the ring for a depth that closes it. Only ASKS: `set_sync_target` is clamped between the underrun-driven adaptive floor and the hard cap, so a link whose jitter genuinely needs more buffer than the picture is away keeps its buffer and the residual is reported instead of being taken out of the listener's stream. Continuity outranks sync, on this ring as on the others. The reference comes from `DisplayTracker`'s `OnFrameRendered` callback — the one place in the client that knows a frame truly latched — and it is computed ABOVE the HUD gate now. A sync loop that only ran while the overlay was up would be off on exactly the devices that report latency; the stats LOCK stays gated, which is what that early-return was really protecting. Both decode loops feed it, so sync works with "Low-latency mode" off as well. Two deliberate refusals: * The figure is published RAW. The HUD shaves the OS present floor off its shown display/end-to-end numbers — metrics report what Punktfunk controls — but sound has to reach the ear when the light reaches the eye, and a floor-shaved reference would place audio a whole latch period early on every device. * Below API 33 there is no render callback, so there is no confirmed present and the loop stays inert (target `None` ⇒ today's behaviour exactly). The release instant is NOT substituted for it: a release targets a FUTURE vsync and runs a whole latch period (8-21 ms measured) ahead of glass, well outside the loop's deadband — it would place audio early on every frame while looking like it was working. The plane is also no longer invisible. Ring depth and the smoothed offset ride the stats array at 33/34 and the Detailed HUD carries `audio buffer N ms · a/v ±N ms`, the same wording the desktop HUD uses — both numbers, because a deep ring on a jittery link is correct behaviour and only the offset separates that from audio simply held late. The 1 Hz logcat line gains `av_ms` beside its depth, and the depth itself now has ONE publisher: the counter copy is gone in favour of the sync cell both readers already share. The escape hatch is two levers. `PUNKTFUNK_NO_AV_SYNC=1` keeps the contract the desktop clients document, but an app launched from the launcher inherits no environment, so the one a field tester can actually reach is `adb shell setprop debug.punktfunk.no_av_sync 1` — no rebuild, exactly like `debug.punktfunk.presenter`. A loop that steers playback has to be bisectable on the device that reports the regression. Verified: `cargo ndk -t arm64-v8a check` clean; `cargo clippy -p punktfunk-client-android --all-targets -- -D warnings` clean on the host lane CI lints, and the Android target introduces no new findings (5 pre-existing lints in audio/mic/pad_audio/vsync are unchanged — the android-gated modules are never linted by the host workspace); `cargo fmt --all --check` clean; `./gradlew :app:testDebugUnitTest` green. The new HUD test was proven non-vacuous by planting the defect first — dropping the render call fails its three positive assertions and leaves the three absence assertions passing, which is the shape a test that "passes for the wrong reason" would not have. design/audio-latency-overhaul.md W4. Apple (W6) still keeps today's behaviour.
This commit is contained in:
@@ -204,7 +204,15 @@ pub(super) fn run_async(
|
||||
// SurfaceFlinger's render timestamp. `render_cb` is the callback's leaked Arc refcount,
|
||||
// reclaimed after the codec is dropped below.
|
||||
let meter = Arc::new(PresentMeter::new());
|
||||
let tracker = DisplayTracker::new(stats.clone(), clock_offset.clone(), meter.clone());
|
||||
// The tracker also publishes each confirmed present's end-to-end into the shared cell the audio
|
||||
// plane steers its jitter ring by (`design/audio-latency-overhaul.md`) — video is the master,
|
||||
// and this is the only point that knows when a frame actually reached glass.
|
||||
let tracker = DisplayTracker::new(
|
||||
stats.clone(),
|
||||
clock_offset.clone(),
|
||||
client.video_e2e_shared(),
|
||||
meter.clone(),
|
||||
);
|
||||
let render_cb = install_render_callback(&codec, &tracker);
|
||||
|
||||
// The timeline presenter (see `presenter.rs`): newest-wins / smoothing store, one-in-flight
|
||||
|
||||
@@ -5,7 +5,7 @@ use ndk::media::media_codec::MediaCodec;
|
||||
use ndk::native_window::NativeWindow;
|
||||
use std::collections::VecDeque;
|
||||
use std::ffi::c_void;
|
||||
use std::sync::atomic::{AtomicI64, Ordering};
|
||||
use std::sync::atomic::{AtomicI64, AtomicU64, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use super::latency::now_realtime_ns;
|
||||
@@ -35,6 +35,16 @@ pub(super) struct DisplayTracker {
|
||||
/// loaded per callback so mid-stream re-syncs apply. Holding the handle (not the client)
|
||||
/// keeps the leaked render-callback refcount from pinning the whole session alive.
|
||||
clock_offset: Arc<AtomicI64>,
|
||||
/// Where the AUDIO plane reads the video leg it has to land with (ns) — `displayed +
|
||||
/// clock_offset − pts`, published on every confirmed present. Written here, read by
|
||||
/// [`crate::audio`]'s sync loop; the two planes never touch each other directly (the presenter
|
||||
/// must not know about audio, and the audio thread cannot see the glass).
|
||||
///
|
||||
/// Published RAW. The HUD shaves the OS present floor off its shown display / end-to-end
|
||||
/// numbers (`StatsOverlay.osFloorMs` — metrics report what Punktfunk controls), but sound has
|
||||
/// to reach the ear when the light reaches the eye, and a floor-shaved reference would place
|
||||
/// audio a whole latch period early on every device. Presentation policy, not physics.
|
||||
video_e2e: Arc<AtomicU64>,
|
||||
/// Always-on latch/display accumulator for the presenter's 1 Hz `pf-present` line —
|
||||
/// independent of the HUD gate, so a HUD-off A/B stays measurable from logcat.
|
||||
meter: Arc<super::presenter::PresentMeter>,
|
||||
@@ -48,11 +58,13 @@ impl DisplayTracker {
|
||||
pub(super) fn new(
|
||||
stats: Arc<crate::stats::VideoStats>,
|
||||
clock_offset: Arc<AtomicI64>,
|
||||
video_e2e: Arc<AtomicU64>,
|
||||
meter: Arc<super::presenter::PresentMeter>,
|
||||
) -> Arc<DisplayTracker> {
|
||||
Arc::new(DisplayTracker {
|
||||
stats,
|
||||
clock_offset,
|
||||
video_e2e,
|
||||
meter,
|
||||
rendered: Mutex::new(VecDeque::new()),
|
||||
})
|
||||
@@ -105,7 +117,14 @@ pub(super) fn install_render_callback(
|
||||
}
|
||||
let sym = libc::dlsym(lib, c"AMediaCodec_setOnFrameRenderedCallback".as_ptr());
|
||||
if sym.is_null() {
|
||||
log::info!("decode: no render callback on this API level (<33) — no display stage");
|
||||
// No confirmed present ⇒ no `display` stage AND no reference for the audio plane's A/V
|
||||
// sync, which then stays inert and leaves the ring exactly as it was. The release
|
||||
// instant is NOT substituted: releases target a future vsync, so it runs a whole latch
|
||||
// period (8-21 ms measured) ahead of glass — well outside the loop's deadband, i.e. it
|
||||
// would place audio early on every frame while looking like it was working.
|
||||
log::info!(
|
||||
"decode: no render callback on this API level (<33) — no display stage, no A/V sync"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
std::mem::transmute::<*mut c_void, SetOnFrameRenderedFn>(sym)
|
||||
@@ -145,8 +164,10 @@ pub(super) unsafe fn release_render_callback(ud: *const DisplayTracker) {
|
||||
/// between the frame rendering and the (batchable) callback delivery — to subtract against the
|
||||
/// receipt/decode stamps and the host capture pts. Records the HUD's `displayed` point:
|
||||
/// `end-to-end` = capture→displayed (skew-corrected) and `display` = decoded→displayed
|
||||
/// (single-clock local). Panic-free by construction (poison-proof lock, saturating math) — an
|
||||
/// unwind out of an `extern "C"` fn would abort the process.
|
||||
/// (single-clock local) — and publishes that end-to-end figure for the audio plane to align
|
||||
/// against, which is the only place in the client that knows when a frame truly reached glass.
|
||||
/// Panic-free by construction (poison-proof lock, saturating math) — an unwind out of an
|
||||
/// `extern "C"` fn would abort the process.
|
||||
unsafe extern "C" fn on_frame_rendered(
|
||||
_codec: *mut ndk_sys::AMediaCodec,
|
||||
userdata: *mut c_void,
|
||||
@@ -186,13 +207,28 @@ unsafe extern "C" fn on_frame_rendered(
|
||||
let latch_us = paired.and_then(|(_, r)| clamp(displayed_ns - r));
|
||||
// Always-on half: the presenter's pf-present line reads these with the HUD off.
|
||||
t.meter.note_latch(latch_us);
|
||||
if !t.stats.enabled() {
|
||||
return; // HUD hidden — skip the skew math + the stats lock
|
||||
}
|
||||
// The glass-to-glass figure, computed ABOVE the HUD gate: the audio plane steers its ring by it
|
||||
// (see `video_e2e`), and a sync loop that only worked while the overlay was up would be off on
|
||||
// the exact devices that report latency — on a Deck-class report the overlay is precisely what
|
||||
// the field cannot reach. The cost is one relaxed load and some integer arithmetic per confirmed
|
||||
// present (≤ the panel rate); the stats LOCK stays behind the gate, which is what that
|
||||
// early-return was really protecting.
|
||||
let e2e_ns =
|
||||
displayed_ns + t.clock_offset.load(Ordering::Relaxed) as i128 - pts_us as i128 * 1000;
|
||||
let e2e_us = (e2e_ns > 0 && e2e_ns < 10_000_000_000).then_some((e2e_ns / 1000) as u64);
|
||||
t.stats.note_displayed(e2e_us, display_us, latch_us);
|
||||
// Same (0, 10 s) clamp as every other e2e sample — a vendor's first render callbacks can carry
|
||||
// a garbage `system_nano`, and here that would step the audio ring rather than just a p95.
|
||||
let e2e_valid = e2e_ns > 0 && e2e_ns < 10_000_000_000;
|
||||
if e2e_valid {
|
||||
t.video_e2e.store(e2e_ns as u64, Ordering::Relaxed);
|
||||
}
|
||||
if !t.stats.enabled() {
|
||||
return; // HUD hidden — skip the stats lock
|
||||
}
|
||||
t.stats.note_displayed(
|
||||
e2e_valid.then_some((e2e_ns / 1000) as u64),
|
||||
display_us,
|
||||
latch_us,
|
||||
);
|
||||
}
|
||||
|
||||
/// React to an output-format change by signalling the stream's HDR dataspace on the Surface (SDR
|
||||
|
||||
@@ -185,9 +185,12 @@ pub(super) fn run_sync(
|
||||
// render = true are parked in the tracker; the OnFrameRendered callback pairs them with
|
||||
// SurfaceFlinger's render timestamp. `render_cb` is the callback's leaked Arc refcount,
|
||||
// reclaimed after the codec is dropped below.
|
||||
// The `video_e2e` cell is the audio plane's alignment reference (see `DisplayTracker`): this
|
||||
// legacy loop feeds it too, so A/V sync works with "Low-latency mode" off as well.
|
||||
let tracker = DisplayTracker::new(
|
||||
stats.clone(),
|
||||
clock_offset.clone(),
|
||||
client.video_e2e_shared(),
|
||||
std::sync::Arc::new(super::presenter::PresentMeter::new()),
|
||||
);
|
||||
let render_cb = install_render_callback(&codec, &tracker);
|
||||
|
||||
Reference in New Issue
Block a user