feat(clients/android): OnFrameRendered display stage — HUD headline becomes capture→displayed
The long-deferred Android display stage (design/stats-unification.md; plan 4.1 of design/client-parity-and-network-resilience.md): AMediaCodec_setOnFrameRenderedCallback (API 26, under the minSdk-28 floor ⇒ hard-linked via ndk-sys) reports SurfaceFlinger's per-frame render timestamp, giving the HUD the spec's `display` = decoded→displayed term and the directly-measured capture→displayed end-to-end headline on both decode loops. Falls back per spec to the v1 capture→decoded endpoint on any window without render callbacks (the platform may drop them under load), and to it permanently if registration is refused. - The render timestamp arrives on CLOCK_MONOTONIC; it's re-based onto CLOCK_REALTIME against monotonic-now at callback time, which also cancels the (batchable) callback delivery lag. - The `ndk` crate exposes neither the callback nor the codec pointer needed to bind it raw, so the workspace pins `ndk` 0.9.0 to a vendored copy (clients/android/native/ vendor/ndk) whose ONLY change makes MediaCodec::as_ptr public — the "as_ptr patch". Workspace-excluded so host builds never compile it; drop when upstream exposes either. - nativeVideoStats grows to 26 doubles (22–25: dispValid, displayP50, e2eDispP50/P95; 0–21 unchanged for older readers); StatsOverlay moves headline endpoint + equation together so the equation always tiles the headline interval. Verified: host cargo check/test/clippy, aarch64-linux-android check/clippy, Kotlin app+kit+tests compile, roborazzi HUD render shows the full 4-term equation. Device verification rides plan 4.2's phone A/B. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -36,6 +36,12 @@ const IN_FLIGHT_CAP: usize = 64;
|
||||
/// this deep is a lost datagram (or an old host that never sends any) and gets evicted.
|
||||
const PENDING_SPLIT_CAP: usize = 256;
|
||||
|
||||
/// Cap on rendered frames parked in [`DisplayTracker`] awaiting their `OnFrameRendered` render
|
||||
/// timestamp: the callback trails its release by at most a vsync or two, so anything this deep
|
||||
/// means the platform stopped delivering render callbacks (allowed under load, per the docs) and
|
||||
/// gets evicted.
|
||||
const RENDERED_CAP: usize = 64;
|
||||
|
||||
/// Whether low-latency mode uses the event-driven async decode loop (default) or the synchronous
|
||||
/// poll loop. Flip to `false` to A/B the two on the HUD (`design/…`); the async loop presents a
|
||||
/// decoded frame the instant it's ready instead of waiting out a poll interval. Only consulted when
|
||||
@@ -208,6 +214,12 @@ fn run_sync(
|
||||
// host-minus-client clock offset (0 if the host didn't answer the skew handshake — then the
|
||||
// HUD flags it "(same-host clock)").
|
||||
let clock_offset = client.clock_offset_ns;
|
||||
// Display stage (spec `display` + the capture→displayed headline): frames released with
|
||||
// render = true are parked in the tracker; the OnFrameRendered callback pairs them with
|
||||
// SurfaceFlinger's render timestamp. `render_cb` is the callback's leaked Arc refcount,
|
||||
// reclaimed after the codec is dropped below.
|
||||
let tracker = DisplayTracker::new(stats.clone(), clock_offset);
|
||||
let render_cb = install_render_callback(&codec, &tracker);
|
||||
// HUD stage split: receipt timestamps keyed by the pts we queue into the codec, so the decoded
|
||||
// point (output-buffer dequeue — MediaCodec round-trips presentationTimeUs) can be paired back
|
||||
// to its receipt for the `decode` stage. Only fed while the HUD is visible.
|
||||
@@ -309,6 +321,7 @@ fn run_sync(
|
||||
&stats,
|
||||
&mut in_flight,
|
||||
clock_offset,
|
||||
&tracker,
|
||||
);
|
||||
rendered += r;
|
||||
discarded += d;
|
||||
@@ -367,6 +380,11 @@ fn run_sync(
|
||||
}
|
||||
|
||||
let _ = codec.stop();
|
||||
drop(codec); // AMediaCodec_delete — after this no render callback can fire
|
||||
if let Some(ud) = render_cb {
|
||||
// SAFETY: the codec was dropped above; this registration's single reclaim.
|
||||
unsafe { release_render_callback(ud) };
|
||||
}
|
||||
log::info!("decode: stopped (fed={fed} rendered={rendered} discarded={discarded})");
|
||||
}
|
||||
|
||||
@@ -380,6 +398,145 @@ fn now_realtime_ns() -> i128 {
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// `CLOCK_MONOTONIC` now in nanoseconds — the base of the `systemNano` render timestamp the
|
||||
/// `OnFrameRendered` callback reports (Android's `System.nanoTime`), read only to re-base that
|
||||
/// stamp onto `CLOCK_REALTIME` (see [`on_frame_rendered`]).
|
||||
fn now_monotonic_ns() -> i128 {
|
||||
let mut ts = libc::timespec {
|
||||
tv_sec: 0,
|
||||
tv_nsec: 0,
|
||||
};
|
||||
// SAFETY: `clock_gettime` with a valid out-pointer is an always-safe syscall.
|
||||
unsafe { libc::clock_gettime(libc::CLOCK_MONOTONIC, &mut ts) };
|
||||
ts.tv_sec as i128 * 1_000_000_000 + ts.tv_nsec as i128
|
||||
}
|
||||
|
||||
/// State shared between the decode loop and the `AMediaCodec` `OnFrameRendered` callback (which
|
||||
/// fires on a codec-internal thread): rendered frames awaiting their render timestamp, so the HUD
|
||||
/// gets the spec's `display` stage (decoded→displayed) and the `capture→displayed` end-to-end
|
||||
/// headline (`design/stats-unification.md` — this replaces Android's v1 `capture→decoded`
|
||||
/// endpoint whenever the platform delivers render callbacks).
|
||||
struct DisplayTracker {
|
||||
stats: Arc<crate::stats::VideoStats>,
|
||||
/// Host-minus-client clock offset (ns) for the skew-corrected end-to-end sample.
|
||||
clock_offset: i64,
|
||||
/// `(pts_us, decoded_real_ns)` of frames released with `render = true`, in release order,
|
||||
/// awaiting their callback. Pushes are HUD-gated by the caller, so this stays empty (and the
|
||||
/// callback early-outs) while the overlay is hidden.
|
||||
rendered: Mutex<VecDeque<(u64, i128)>>,
|
||||
}
|
||||
|
||||
impl DisplayTracker {
|
||||
fn new(stats: Arc<crate::stats::VideoStats>, clock_offset: i64) -> Arc<DisplayTracker> {
|
||||
Arc::new(DisplayTracker {
|
||||
stats,
|
||||
clock_offset,
|
||||
rendered: Mutex::new(VecDeque::new()),
|
||||
})
|
||||
}
|
||||
|
||||
/// Park one just-rendered frame's `(pts, decoded stamp)` for the render callback to pair.
|
||||
/// Caller gates on the HUD being visible.
|
||||
fn note_rendered(&self, pts_us: u64, decoded_ns: i128) {
|
||||
let mut g = self
|
||||
.rendered
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
g.push_back((pts_us, decoded_ns));
|
||||
if g.len() > RENDERED_CAP {
|
||||
g.pop_front(); // render callbacks stopped coming (allowed under load) — evict
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Register [`on_frame_rendered`] on the codec (`AMediaCodec_setOnFrameRenderedCallback`, API 26 —
|
||||
/// under the minSdk-28 floor, so hard-linked via `ndk-sys`; the `ndk` wrapper has no binding, which
|
||||
/// is what the vendored crate's public `as_ptr` patch is for). Returns the userdata pointer holding
|
||||
/// a leaked `Arc<DisplayTracker>` refcount; the caller MUST reclaim it with
|
||||
/// [`release_render_callback`] AFTER dropping the codec (`AMediaCodec_delete` is what guarantees no
|
||||
/// further callback can fire). `None` (nothing to reclaim) if the platform refused — the HUD then
|
||||
/// simply has no `display` stage, exactly the pre-callback behaviour.
|
||||
fn install_render_callback(
|
||||
codec: &MediaCodec,
|
||||
tracker: &Arc<DisplayTracker>,
|
||||
) -> Option<*const DisplayTracker> {
|
||||
let ud = Arc::into_raw(tracker.clone());
|
||||
// SAFETY: `codec.as_ptr()` is the live codec this thread owns; `ud` outlives the registration
|
||||
// (reclaimed only after the codec is deleted, per this function's contract).
|
||||
let status = unsafe {
|
||||
ndk_sys::AMediaCodec_setOnFrameRenderedCallback(
|
||||
codec.as_ptr(),
|
||||
Some(on_frame_rendered),
|
||||
ud as *mut c_void,
|
||||
)
|
||||
};
|
||||
if status == ndk_sys::media_status_t::AMEDIA_OK {
|
||||
Some(ud)
|
||||
} else {
|
||||
log::warn!("decode: setOnFrameRenderedCallback failed ({status:?}) — no display stage");
|
||||
// SAFETY: registration failed, so the codec never took the reference — reclaim it now.
|
||||
unsafe { drop(Arc::from_raw(ud)) };
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Reclaim [`install_render_callback`]'s leaked `Arc` refcount.
|
||||
///
|
||||
/// # Safety
|
||||
/// Call exactly once, and only after the codec the callback was registered on has been dropped —
|
||||
/// deleting the codec stops its internal threads, so no callback can still be running (or run
|
||||
/// later) against this pointer.
|
||||
unsafe fn release_render_callback(ud: *const DisplayTracker) {
|
||||
drop(Arc::from_raw(ud));
|
||||
}
|
||||
|
||||
/// The `AMediaCodecOnFrameRendered` trampoline: fires (possibly batched) on a codec-internal
|
||||
/// thread once per output frame actually placed on the output surface, with SurfaceFlinger's
|
||||
/// render timestamp. That timestamp (`system_nano`) is on `CLOCK_MONOTONIC`, so it is re-based
|
||||
/// onto `CLOCK_REALTIME` here — against monotonic-now at callback time, which also cancels any lag
|
||||
/// between the frame rendering and the (batchable) callback delivery — to subtract against the
|
||||
/// receipt/decode stamps and the host capture pts. Records the HUD's `displayed` point:
|
||||
/// `end-to-end` = capture→displayed (skew-corrected) and `display` = decoded→displayed
|
||||
/// (single-clock local). Panic-free by construction (poison-proof lock, saturating math) — an
|
||||
/// unwind out of an `extern "C"` fn would abort the process.
|
||||
unsafe extern "C" fn on_frame_rendered(
|
||||
_codec: *mut ndk_sys::AMediaCodec,
|
||||
userdata: *mut c_void,
|
||||
media_time_us: i64,
|
||||
system_nano: i64,
|
||||
) {
|
||||
let t = &*(userdata as *const DisplayTracker);
|
||||
if !t.stats.enabled() {
|
||||
return; // HUD hidden — the ring is empty too (pushes are caller-gated)
|
||||
}
|
||||
let displayed_ns = now_realtime_ns() - (now_monotonic_ns() - system_nano as i128);
|
||||
let pts_us = media_time_us.max(0) as u64;
|
||||
// Pair the frame back to its release record, evicting older entries (their callbacks were
|
||||
// dropped by the platform, or the entry predates a HUD toggle) — same monotonic-eviction
|
||||
// discipline as `note_decoded_pts`.
|
||||
let mut decoded_ns = None;
|
||||
{
|
||||
let mut g = t
|
||||
.rendered
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
while let Some(&(p, d)) = g.front() {
|
||||
if p > pts_us {
|
||||
break; // future frame — leave it for its own callback
|
||||
}
|
||||
g.pop_front();
|
||||
if p == pts_us {
|
||||
decoded_ns = Some(d);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
let e2e_ns = displayed_ns + t.clock_offset as i128 - pts_us as i128 * 1000;
|
||||
let e2e_us = (e2e_ns > 0 && e2e_ns < 10_000_000_000).then_some((e2e_ns / 1000) as u64);
|
||||
let display_us = decoded_ns.map(|d| ((displayed_ns - d).max(0) / 1000) as u64);
|
||||
t.stats.note_displayed(e2e_us, display_us);
|
||||
}
|
||||
|
||||
/// The MediaCodec MIME for the codec the host resolved (`Welcome.codec`). Shared by the decode
|
||||
/// thread and `nativeVideoMime` (which tells Kotlin what to rank decoders for). AV1 uses the
|
||||
/// AOSP `video/av01` type; anything not H.264/AV1 is treated as HEVC (every pre-negotiation host
|
||||
@@ -649,6 +806,12 @@ fn run_async(
|
||||
// HUD is visible.
|
||||
let clock_offset = client.clock_offset_ns;
|
||||
let in_flight = Arc::new(Mutex::new(VecDeque::<(u64, i128)>::new()));
|
||||
// Display stage (spec `display` + the capture→displayed headline): the rendered frame is
|
||||
// parked in the tracker at release; the OnFrameRendered callback pairs it with
|
||||
// SurfaceFlinger's render timestamp. `render_cb` is the callback's leaked Arc refcount,
|
||||
// reclaimed after the codec is dropped below.
|
||||
let tracker = DisplayTracker::new(stats.clone(), clock_offset);
|
||||
let render_cb = install_render_callback(&codec, &tracker);
|
||||
|
||||
// Feeder thread: block on the network so this loop doesn't (an AU's arrival becomes an event that
|
||||
// wakes us immediately, with no input-side poll latency). It also records the `received` HUD stat.
|
||||
@@ -744,6 +907,7 @@ fn run_async(
|
||||
&stats,
|
||||
&in_flight,
|
||||
clock_offset,
|
||||
&tracker,
|
||||
&mut rendered,
|
||||
&mut discarded,
|
||||
);
|
||||
@@ -796,6 +960,11 @@ fn run_async(
|
||||
if let Some(j) = feeder {
|
||||
let _ = j.join();
|
||||
}
|
||||
drop(codec); // AMediaCodec_delete — after this no render callback can fire
|
||||
if let Some(ud) = render_cb {
|
||||
// SAFETY: the codec was dropped above; this registration's single reclaim.
|
||||
unsafe { release_render_callback(ud) };
|
||||
}
|
||||
log::info!("decode: stopped (async, fed={fed} rendered={rendered} discarded={discarded})");
|
||||
}
|
||||
|
||||
@@ -938,13 +1107,17 @@ fn feed_ready(
|
||||
/// burst of stale frames on glass is worse than skipping to the freshest (the sync loop's newest-ready
|
||||
/// policy, callback-driven). Every dequeued buffer, rendered or not, is the HUD's `decoded`
|
||||
/// measurement point (it finished decoding either way); samples are recorded in pts order so the
|
||||
/// receipt-map eviction stays monotonic. `ready` is drained.
|
||||
/// receipt-map eviction stays monotonic. The presented frame's `(pts, decoded stamp)` is parked in
|
||||
/// `tracker` for the OnFrameRendered callback — the `display` stage's other endpoint. `ready` is
|
||||
/// drained.
|
||||
#[allow(clippy::too_many_arguments)] // one call site; mirrors the sync loop's drain
|
||||
fn present_ready(
|
||||
codec: &MediaCodec,
|
||||
ready: &mut Vec<OutputReady>,
|
||||
stats: &crate::stats::VideoStats,
|
||||
in_flight: &Mutex<VecDeque<(u64, i128)>>,
|
||||
clock_offset: i64,
|
||||
tracker: &DisplayTracker,
|
||||
rendered: &mut u64,
|
||||
discarded: &mut u64,
|
||||
) {
|
||||
@@ -964,7 +1137,12 @@ fn present_ready(
|
||||
for (i, o) in ready.drain(..).enumerate() {
|
||||
let render = i == last;
|
||||
match codec.release_output_buffer_by_index(o.index, render) {
|
||||
Ok(()) if render => *rendered += 1,
|
||||
Ok(()) if render => {
|
||||
*rendered += 1;
|
||||
if stats.enabled() {
|
||||
tracker.note_rendered(o.pts_us, o.decoded_ns);
|
||||
}
|
||||
}
|
||||
Ok(()) => {
|
||||
*discarded += 1;
|
||||
skipped += 1;
|
||||
@@ -1143,7 +1321,10 @@ fn feed(codec: &MediaCodec, au: &[u8], pts_us: u64) -> bool {
|
||||
/// Each dequeued buffer is also the HUD's `decoded` measurement point (rendered or not — the frame
|
||||
/// finished decoding either way): end-to-end = decoded + clock_offset − capture pts, and the
|
||||
/// `decode` stage pairs the buffer's echoed presentationTimeUs back to the receipt stamp in
|
||||
/// `in_flight` (single-clock local difference, no skew involved).
|
||||
/// `in_flight` (single-clock local difference, no skew involved). The presented frame's
|
||||
/// `(pts, decoded stamp)` is additionally parked in `tracker` for the OnFrameRendered callback —
|
||||
/// the `display` stage's other endpoint.
|
||||
#[allow(clippy::too_many_arguments)] // one call site; mirrors the async loop's present_ready
|
||||
fn drain(
|
||||
codec: &MediaCodec,
|
||||
window: &NativeWindow,
|
||||
@@ -1152,18 +1333,27 @@ fn drain(
|
||||
stats: &crate::stats::VideoStats,
|
||||
in_flight: &mut VecDeque<(u64, i128)>,
|
||||
clock_offset: i64,
|
||||
tracker: &DisplayTracker,
|
||||
) -> (u64, u64) {
|
||||
let mut held = None; // newest ready buffer so far, presented after the loop
|
||||
// Newest ready buffer so far (presented after the loop) with its HUD metadata —
|
||||
// `Some((pts_us, decoded_ns))` only while the HUD is visible (the stamp read is gated).
|
||||
let mut held: Option<(OutputBuffer<'_>, Option<(u64, i128)>)> = None;
|
||||
let mut discarded: u64 = 0;
|
||||
let mut wait = first_wait;
|
||||
loop {
|
||||
match codec.dequeue_output_buffer(wait) {
|
||||
Ok(DequeuedOutputBufferInfoResult::Buffer(buf)) => {
|
||||
wait = Duration::ZERO; // only the first dequeue may block
|
||||
if stats.enabled() {
|
||||
note_decoded(stats, in_flight, clock_offset, &buf);
|
||||
}
|
||||
if let Some(stale) = held.replace(buf) {
|
||||
let meta = if stats.enabled() {
|
||||
// The dequeue IS the sync loop's decoded-availability instant.
|
||||
let pts_us = buf.info().presentation_time_us().max(0) as u64;
|
||||
let decoded_ns = now_realtime_ns();
|
||||
note_decoded_pts(stats, in_flight, clock_offset, pts_us, decoded_ns);
|
||||
Some((pts_us, decoded_ns))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some((stale, _)) = held.replace((buf, meta)) {
|
||||
// A newer frame is ready — drop the held one without rendering.
|
||||
if let Err(e) = codec.release_output_buffer(stale, false) {
|
||||
log::warn!("decode: release_output_buffer(discard): {e}");
|
||||
@@ -1202,41 +1392,30 @@ fn drain(
|
||||
}
|
||||
}
|
||||
}
|
||||
// Present the newest ready frame, if any.
|
||||
// Present the newest ready frame, if any, and park its metadata for the render callback.
|
||||
let mut rendered = 0;
|
||||
if let Some(buf) = held {
|
||||
if let Some((buf, meta)) = held {
|
||||
match codec.release_output_buffer(buf, true) {
|
||||
Ok(()) => rendered = 1,
|
||||
Ok(()) => {
|
||||
rendered = 1;
|
||||
if let Some((pts_us, decoded_ns)) = meta {
|
||||
tracker.note_rendered(pts_us, decoded_ns);
|
||||
}
|
||||
}
|
||||
Err(e) => log::warn!("decode: release_output_buffer: {e}"),
|
||||
}
|
||||
}
|
||||
(rendered, discarded)
|
||||
}
|
||||
|
||||
/// HUD `decoded` point for one dequeued output buffer: build the end-to-end (capture→decoded,
|
||||
/// skew-corrected, clamped to (0, 10 s)) and `decode` (received→decoded, single-clock local, ≥ 0)
|
||||
/// samples and hand them to [`crate::stats::VideoStats::note_decoded`]. The codec echoes the input
|
||||
/// `presentationTimeUs` on the output buffer, which keys the receipt stamp in `in_flight`; entries
|
||||
/// older than the echoed pts are evicted (decode order == input order here — low-latency, no
|
||||
/// HUD `decoded` point for one dequeued output frame, keyed by the echoed `presentationTimeUs`:
|
||||
/// build the end-to-end (capture→decoded, skew-corrected, clamped to (0, 10 s)) and `decode`
|
||||
/// (received→decoded, single-clock local, ≥ 0) samples and hand them to
|
||||
/// [`crate::stats::VideoStats::note_decoded`]. The pts keys the receipt stamp in `in_flight`;
|
||||
/// entries older than it are evicted (decode order == input order here — low-latency, no
|
||||
/// B-frames — so anything before it was dropped inside the codec or stamped before a flush).
|
||||
fn note_decoded(
|
||||
stats: &crate::stats::VideoStats,
|
||||
in_flight: &mut VecDeque<(u64, i128)>,
|
||||
clock_offset: i64,
|
||||
buf: &OutputBuffer<'_>,
|
||||
) {
|
||||
note_decoded_pts(
|
||||
stats,
|
||||
in_flight,
|
||||
clock_offset,
|
||||
buf.info().presentation_time_us().max(0) as u64,
|
||||
now_realtime_ns(), // sync loop: the dequeue IS the availability instant
|
||||
);
|
||||
}
|
||||
|
||||
/// The [`note_decoded`] body keyed by the echoed `presentationTimeUs` directly — the async loop has
|
||||
/// the pts (from the output callback's `BufferInfo`) but no borrowed `OutputBuffer`, so it calls
|
||||
/// this with the `decoded` stamp taken in the output callback itself (the availability instant).
|
||||
/// `decoded_ns` is the availability instant: the dequeue (sync loop) or the output callback's
|
||||
/// stamp (async loop).
|
||||
fn note_decoded_pts(
|
||||
stats: &crate::stats::VideoStats,
|
||||
in_flight: &mut VecDeque<(u64, i128)>,
|
||||
|
||||
@@ -144,11 +144,12 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopVideo(
|
||||
}
|
||||
|
||||
/// `NativeBridge.nativeVideoStats(handle): DoubleArray?` — drain ~1 s of decode stats for the HUD
|
||||
/// (unified stats spec, `design/stats-unification.md`). Returns 22 doubles
|
||||
/// (unified stats spec, `design/stats-unification.md`). Returns 26 doubles
|
||||
/// `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skewCorrected, width, height, refreshHz, framesLost,
|
||||
/// bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms,
|
||||
/// netP50Ms, lostWindow, skippedWindow, fecWindow, framesWindow]`
|
||||
/// (the two flags are 1.0/0.0; indexes 0–17 match the previous 18-double layout — 0–13 the original
|
||||
/// netP50Ms, lostWindow, skippedWindow, fecWindow, framesWindow, dispValid, displayP50Ms,
|
||||
/// e2eDispP50Ms, e2eDispP95Ms]`
|
||||
/// (the flags are 1.0/0.0; indexes 0–21 match the previous 22-double layout — 0–13 the original
|
||||
/// 14-double one with the latency pair re-based to the end-to-end capture→decoded headline, 14/15
|
||||
/// the stage p50s tiling it: `host+network` = capture→received, `decode` = received→decoded; 16/17
|
||||
/// are the Phase-2 split of the `host+network` term from the per-AU 0xCF host timings — `host` =
|
||||
@@ -156,7 +157,11 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopVideo(
|
||||
/// window, i.e. an old host; 18–21 are the spec's per-window line-4 counters — `lost` =
|
||||
/// unrecoverable drops this window, `skipped` = client newest-wins/pacing drops, `fec` = shards
|
||||
/// recovered, `frames` = AUs received, so the HUD can compute `lost/(frames+lost)` — index 9 stays
|
||||
/// the cumulative session total for older readers), or `null` when no decode thread is running.
|
||||
/// the cumulative session total for older readers; 22–25 are the `display` stage from the
|
||||
/// OnFrameRendered render timestamps — when `dispValid` is 1.0 the HUD headline becomes the
|
||||
/// directly-measured capture→displayed pair at 24/25 with `display` = decoded→displayed p50 at 23
|
||||
/// closing the equation, and when 0.0 — no render callback landed this window — it falls back to
|
||||
/// the capture→decoded headline at 2/3), or `null` when no decode thread is running.
|
||||
/// Poll ~1 Hz from the UI; each call
|
||||
/// resets the measurement window. Not android-gated — pure `jni` + connector reads, so it links on
|
||||
/// the host build too (Kotlin only ever calls it on device).
|
||||
@@ -180,7 +185,7 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats(
|
||||
.drain(h.client.frames_dropped(), h.client.fec_recovered_shards());
|
||||
let mode = h.client.mode();
|
||||
let color = h.client.color;
|
||||
let buf: [f64; 22] = [
|
||||
let buf: [f64; 26] = [
|
||||
snap.fps,
|
||||
snap.mbps,
|
||||
snap.e2e_p50_ms,
|
||||
@@ -213,6 +218,14 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats(
|
||||
snap.skipped as f64,
|
||||
snap.fec as f64,
|
||||
snap.frames as f64,
|
||||
// `display` stage (OnFrameRendered render timestamps): validity flag, the
|
||||
// decoded→displayed stage p50, and the directly-measured capture→displayed headline
|
||||
// pair that supersedes 2/3 whenever the flag is set (spec: the equation always tiles
|
||||
// the headline interval, so endpoint and terms move together).
|
||||
if snap.disp_valid { 1.0 } else { 0.0 },
|
||||
snap.display_p50_ms,
|
||||
snap.e2e_disp_p50_ms,
|
||||
snap.e2e_disp_p95_ms,
|
||||
];
|
||||
let arr = match env.new_double_array(buf.len() as jsize) {
|
||||
Ok(a) => a,
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
//! Live decode stats for the on-stream HUD, following the unified stats spec
|
||||
//! (`design/stats-unification.md`): FPS, receive throughput, and the Android v1 stage split —
|
||||
//! headline `end-to-end` = capture→decoded (p50/p95) tiled by `host+network` = capture→received
|
||||
//! and `decode` = received→decoded (stage p50s). When the host emits per-AU 0xCF host timings, the
|
||||
//! `host+network` term further splits into `host` + `network` (Phase 2, `note_host_split`); an old
|
||||
//! host emits none and the combined term stands. The spec's line-4 counters are per-window too:
|
||||
//! (`design/stats-unification.md`): FPS, receive throughput, and the full stage split — headline
|
||||
//! `end-to-end` = capture→displayed (p50/p95, measured directly from the `OnFrameRendered` render
|
||||
//! timestamps — `note_displayed`) tiled by `host+network` = capture→received, `decode` =
|
||||
//! received→decoded, and `display` = decoded→displayed (stage p50s). When the platform delivers no
|
||||
//! render callbacks (allowed under load; `disp_valid` false), the HUD falls back to the v1
|
||||
//! capture→decoded headline without the `display` term. When the host emits per-AU 0xCF host
|
||||
//! timings, the `host+network` term further splits into `host` + `network` (Phase 2,
|
||||
//! `note_host_split`); an old host emits none and the combined term stands. The spec's line-4 counters are per-window too:
|
||||
//! `lost` / `FEC` are windowed here from the connector's cumulative counters (the caller passes the
|
||||
//! current totals into `set_enabled`/`drain`), `skipped` counts the client's own newest-wins drops
|
||||
//! (`note_skipped`). The decode thread is the sole writer
|
||||
@@ -57,6 +60,13 @@ struct Inner {
|
||||
net_us: Vec<u64>,
|
||||
/// `decode` stage = received→decoded samples, in microseconds (client-local, single clock).
|
||||
decode_us: Vec<u64>,
|
||||
/// `display` stage = decoded→displayed samples, in microseconds (client-local, single clock),
|
||||
/// from the `OnFrameRendered` render timestamps. Empty when the platform delivers no render
|
||||
/// callbacks — the HUD then drops the term and the headline endpoint moves back to `decoded`.
|
||||
display_us: Vec<u64>,
|
||||
/// `end-to-end` = capture→displayed samples, µs (skew-corrected) — the spec's headline,
|
||||
/// measured directly (not summed from stages). Empty under the same fallback as `display_us`.
|
||||
e2e_disp_us: Vec<u64>,
|
||||
/// Client-side newest-wins/pacing drops this window (decoded frames released without
|
||||
/// rendering, or parked AUs dropped on overflow) — the spec's `skipped` counter.
|
||||
skipped: u64,
|
||||
@@ -75,12 +85,22 @@ struct Inner {
|
||||
pub struct Snapshot {
|
||||
pub fps: f64,
|
||||
pub mbps: f64,
|
||||
/// Headline `end-to-end` (capture→decoded) percentiles, ms.
|
||||
/// Headline `end-to-end` (capture→decoded) percentiles, ms — the fallback headline when no
|
||||
/// render callback landed this window (`disp_valid` false).
|
||||
pub e2e_p50_ms: f64,
|
||||
pub e2e_p95_ms: f64,
|
||||
/// Stage p50s (ms): `host+network` (capture→received) and `decode` (received→decoded).
|
||||
/// The full headline: `end-to-end` = capture→displayed percentiles, ms, measured directly from
|
||||
/// the render timestamps. Meaningful only when `disp_valid`.
|
||||
pub e2e_disp_p50_ms: f64,
|
||||
pub e2e_disp_p95_ms: f64,
|
||||
/// Stage p50s (ms): `host+network` (capture→received), `decode` (received→decoded), and
|
||||
/// `display` (decoded→displayed; 0.0 when `disp_valid` is false — the HUD drops the term).
|
||||
pub hostnet_p50_ms: f64,
|
||||
pub decode_p50_ms: f64,
|
||||
pub display_p50_ms: f64,
|
||||
/// Whether any capture→displayed sample landed this window — gates the HUD's headline endpoint
|
||||
/// (`capture→displayed` vs the capture→decoded fallback) and the equation's `display` term.
|
||||
pub disp_valid: bool,
|
||||
/// Phase-2 `host` / `network` split p50s (ms) — 0.0 when no 0xCF timing matched this window
|
||||
/// (old host / no samples yet), in which case the HUD keeps the combined `host+network` term.
|
||||
pub host_p50_ms: f64,
|
||||
@@ -122,6 +142,8 @@ impl VideoStats {
|
||||
host_us: Vec::with_capacity(256),
|
||||
net_us: Vec::with_capacity(256),
|
||||
decode_us: Vec::with_capacity(256),
|
||||
display_us: Vec::with_capacity(256),
|
||||
e2e_disp_us: Vec::with_capacity(256),
|
||||
skipped: 0,
|
||||
last_dropped_total: 0,
|
||||
last_fec_total: 0,
|
||||
@@ -157,6 +179,8 @@ impl VideoStats {
|
||||
g.host_us.clear();
|
||||
g.net_us.clear();
|
||||
g.decode_us.clear();
|
||||
g.display_us.clear();
|
||||
g.e2e_disp_us.clear();
|
||||
g.skipped = 0;
|
||||
g.last_dropped_total = dropped_total;
|
||||
g.last_fec_total = fec_total;
|
||||
@@ -273,6 +297,30 @@ impl VideoStats {
|
||||
}
|
||||
}
|
||||
|
||||
/// Record one displayed frame (the `OnFrameRendered` render timestamp, re-based to the
|
||||
/// realtime clock): its capture→displayed `end-to-end` sample and its decoded→displayed
|
||||
/// `display` stage sample (either may be absent — the e2e clamp rejected an out-of-range
|
||||
/// value, or the decoded stamp for this pts was already evicted/pre-HUD). Fired from the
|
||||
/// codec's render-callback thread, not the decode thread — the lock makes that safe.
|
||||
// Driven only by the android-only decode path; unreferenced on the host build — expected.
|
||||
#[cfg_attr(not(target_os = "android"), allow(dead_code))]
|
||||
pub fn note_displayed(&self, e2e_us: Option<u64>, display_us: Option<u64>) {
|
||||
if !self.enabled.load(Ordering::Relaxed) {
|
||||
return; // HUD hidden — skip the lock (the callback already skipped the clock reads)
|
||||
}
|
||||
// Poison-proof for the same reason as `note_received`.
|
||||
let mut g = self
|
||||
.inner
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if let Some(l) = e2e_us {
|
||||
g.e2e_disp_us.push(l);
|
||||
}
|
||||
if let Some(l) = display_us {
|
||||
g.display_us.push(l);
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the window's rates + latency percentiles, then reset for the next window.
|
||||
/// `dropped_total` / `fec_total` are the connector's session-cumulative counters now; the
|
||||
/// snapshot's `lost` / `fec` are their deltas since the last drain (or the enabling show).
|
||||
@@ -291,13 +339,19 @@ impl VideoStats {
|
||||
g.host_us.sort_unstable();
|
||||
g.net_us.sort_unstable();
|
||||
g.decode_us.sort_unstable();
|
||||
g.display_us.sort_unstable();
|
||||
g.e2e_disp_us.sort_unstable();
|
||||
let snap = Snapshot {
|
||||
fps,
|
||||
mbps,
|
||||
e2e_p50_ms: pctl_ms(&g.e2e_us, 0.50),
|
||||
e2e_p95_ms: pctl_ms(&g.e2e_us, 0.95),
|
||||
e2e_disp_p50_ms: pctl_ms(&g.e2e_disp_us, 0.50),
|
||||
e2e_disp_p95_ms: pctl_ms(&g.e2e_disp_us, 0.95),
|
||||
hostnet_p50_ms: pctl_ms(&g.hostnet_us, 0.50),
|
||||
decode_p50_ms: pctl_ms(&g.decode_us, 0.50),
|
||||
display_p50_ms: pctl_ms(&g.display_us, 0.50),
|
||||
disp_valid: !g.e2e_disp_us.is_empty(),
|
||||
host_p50_ms: pctl_ms(&g.host_us, 0.50),
|
||||
net_p50_ms: pctl_ms(&g.net_us, 0.50),
|
||||
lat_valid: !g.e2e_us.is_empty(),
|
||||
@@ -315,6 +369,8 @@ impl VideoStats {
|
||||
g.host_us.clear();
|
||||
g.net_us.clear();
|
||||
g.decode_us.clear();
|
||||
g.display_us.clear();
|
||||
g.e2e_disp_us.clear();
|
||||
g.skipped = 0;
|
||||
g.last_dropped_total = dropped_total;
|
||||
g.last_fec_total = fec_total;
|
||||
|
||||
Reference in New Issue
Block a user