feat(clients/android): OnFrameRendered display stage — HUD headline becomes capture→displayed

The long-deferred Android display stage (design/stats-unification.md; plan 4.1 of
design/client-parity-and-network-resilience.md): AMediaCodec_setOnFrameRenderedCallback
(API 26, under the minSdk-28 floor ⇒ hard-linked via ndk-sys) reports SurfaceFlinger's
per-frame render timestamp, giving the HUD the spec's `display` = decoded→displayed term
and the directly-measured capture→displayed end-to-end headline on both decode loops.
Falls back per spec to the v1 capture→decoded endpoint on any window without render
callbacks (the platform may drop them under load), and to it permanently if registration
is refused.

- The render timestamp arrives on CLOCK_MONOTONIC; it's re-based onto CLOCK_REALTIME
  against monotonic-now at callback time, which also cancels the (batchable) callback
  delivery lag.
- The `ndk` crate exposes neither the callback nor the codec pointer needed to bind it
  raw, so the workspace pins `ndk` 0.9.0 to a vendored copy (clients/android/native/
  vendor/ndk) whose ONLY change makes MediaCodec::as_ptr public — the "as_ptr patch".
  Workspace-excluded so host builds never compile it; drop when upstream exposes either.
- nativeVideoStats grows to 26 doubles (22–25: dispValid, displayP50, e2eDispP50/P95;
  0–21 unchanged for older readers); StatsOverlay moves headline endpoint + equation
  together so the equation always tiles the headline interval.

Verified: host cargo check/test/clippy, aarch64-linux-android check/clippy, Kotlin
app+kit+tests compile, roborazzi HUD render shows the full 4-term equation. Device
verification rides plan 4.2's phone A/B.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 11:35:16 +02:00
parent f508d3213f
commit 13d1aa5738
39 changed files with 10477 additions and 70 deletions
+63 -7
View File
@@ -1,9 +1,12 @@
//! Live decode stats for the on-stream HUD, following the unified stats spec
//! (`design/stats-unification.md`): FPS, receive throughput, and the Android v1 stage split —
//! headline `end-to-end` = capture→decoded (p50/p95) tiled by `host+network` = capture→received
//! and `decode` = received→decoded (stage p50s). When the host emits per-AU 0xCF host timings, the
//! `host+network` term further splits into `host` + `network` (Phase 2, `note_host_split`); an old
//! host emits none and the combined term stands. The spec's line-4 counters are per-window too:
//! (`design/stats-unification.md`): FPS, receive throughput, and the full stage split — headline
//! `end-to-end` = capture→displayed (p50/p95, measured directly from the `OnFrameRendered` render
//! timestamps — `note_displayed`) tiled by `host+network` = capture→received, `decode` =
//! received→decoded, and `display` = decoded→displayed (stage p50s). When the platform delivers no
//! render callbacks (allowed under load; `disp_valid` false), the HUD falls back to the v1
//! capture→decoded headline without the `display` term. When the host emits per-AU 0xCF host
//! timings, the `host+network` term further splits into `host` + `network` (Phase 2,
//! `note_host_split`); an old host emits none and the combined term stands. The spec's line-4 counters are per-window too:
//! `lost` / `FEC` are windowed here from the connector's cumulative counters (the caller passes the
//! current totals into `set_enabled`/`drain`), `skipped` counts the client's own newest-wins drops
//! (`note_skipped`). The decode thread is the sole writer
@@ -57,6 +60,13 @@ struct Inner {
net_us: Vec<u64>,
/// `decode` stage = received→decoded samples, in microseconds (client-local, single clock).
decode_us: Vec<u64>,
/// `display` stage = decoded→displayed samples, in microseconds (client-local, single clock),
/// from the `OnFrameRendered` render timestamps. Empty when the platform delivers no render
/// callbacks — the HUD then drops the term and the headline endpoint moves back to `decoded`.
display_us: Vec<u64>,
/// `end-to-end` = capture→displayed samples, µs (skew-corrected) — the spec's headline,
/// measured directly (not summed from stages). Empty under the same fallback as `display_us`.
e2e_disp_us: Vec<u64>,
/// Client-side newest-wins/pacing drops this window (decoded frames released without
/// rendering, or parked AUs dropped on overflow) — the spec's `skipped` counter.
skipped: u64,
@@ -75,12 +85,22 @@ struct Inner {
pub struct Snapshot {
pub fps: f64,
pub mbps: f64,
/// Headline `end-to-end` (capture→decoded) percentiles, ms.
/// Headline `end-to-end` (capture→decoded) percentiles, ms — the fallback headline when no
/// render callback landed this window (`disp_valid` false).
pub e2e_p50_ms: f64,
pub e2e_p95_ms: f64,
/// Stage p50s (ms): `host+network` (capture→received) and `decode` (received→decoded).
/// The full headline: `end-to-end` = capture→displayed percentiles, ms, measured directly from
/// the render timestamps. Meaningful only when `disp_valid`.
pub e2e_disp_p50_ms: f64,
pub e2e_disp_p95_ms: f64,
/// Stage p50s (ms): `host+network` (capture→received), `decode` (received→decoded), and
/// `display` (decoded→displayed; 0.0 when `disp_valid` is false — the HUD drops the term).
pub hostnet_p50_ms: f64,
pub decode_p50_ms: f64,
pub display_p50_ms: f64,
/// Whether any capture→displayed sample landed this window — gates the HUD's headline endpoint
/// (`capture→displayed` vs the capture→decoded fallback) and the equation's `display` term.
pub disp_valid: bool,
/// Phase-2 `host` / `network` split p50s (ms) — 0.0 when no 0xCF timing matched this window
/// (old host / no samples yet), in which case the HUD keeps the combined `host+network` term.
pub host_p50_ms: f64,
@@ -122,6 +142,8 @@ impl VideoStats {
host_us: Vec::with_capacity(256),
net_us: Vec::with_capacity(256),
decode_us: Vec::with_capacity(256),
display_us: Vec::with_capacity(256),
e2e_disp_us: Vec::with_capacity(256),
skipped: 0,
last_dropped_total: 0,
last_fec_total: 0,
@@ -157,6 +179,8 @@ impl VideoStats {
g.host_us.clear();
g.net_us.clear();
g.decode_us.clear();
g.display_us.clear();
g.e2e_disp_us.clear();
g.skipped = 0;
g.last_dropped_total = dropped_total;
g.last_fec_total = fec_total;
@@ -273,6 +297,30 @@ impl VideoStats {
}
}
/// Record one displayed frame (the `OnFrameRendered` render timestamp, re-based to the
/// realtime clock): its capture→displayed `end-to-end` sample and its decoded→displayed
/// `display` stage sample (either may be absent — the e2e clamp rejected an out-of-range
/// value, or the decoded stamp for this pts was already evicted/pre-HUD). Fired from the
/// codec's render-callback thread, not the decode thread — the lock makes that safe.
// Driven only by the android-only decode path; unreferenced on the host build — expected.
#[cfg_attr(not(target_os = "android"), allow(dead_code))]
pub fn note_displayed(&self, e2e_us: Option<u64>, display_us: Option<u64>) {
if !self.enabled.load(Ordering::Relaxed) {
return; // HUD hidden — skip the lock (the callback already skipped the clock reads)
}
// Poison-proof for the same reason as `note_received`.
let mut g = self
.inner
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some(l) = e2e_us {
g.e2e_disp_us.push(l);
}
if let Some(l) = display_us {
g.display_us.push(l);
}
}
/// Compute the window's rates + latency percentiles, then reset for the next window.
/// `dropped_total` / `fec_total` are the connector's session-cumulative counters now; the
/// snapshot's `lost` / `fec` are their deltas since the last drain (or the enabling show).
@@ -291,13 +339,19 @@ impl VideoStats {
g.host_us.sort_unstable();
g.net_us.sort_unstable();
g.decode_us.sort_unstable();
g.display_us.sort_unstable();
g.e2e_disp_us.sort_unstable();
let snap = Snapshot {
fps,
mbps,
e2e_p50_ms: pctl_ms(&g.e2e_us, 0.50),
e2e_p95_ms: pctl_ms(&g.e2e_us, 0.95),
e2e_disp_p50_ms: pctl_ms(&g.e2e_disp_us, 0.50),
e2e_disp_p95_ms: pctl_ms(&g.e2e_disp_us, 0.95),
hostnet_p50_ms: pctl_ms(&g.hostnet_us, 0.50),
decode_p50_ms: pctl_ms(&g.decode_us, 0.50),
display_p50_ms: pctl_ms(&g.display_us, 0.50),
disp_valid: !g.e2e_disp_us.is_empty(),
host_p50_ms: pctl_ms(&g.host_us, 0.50),
net_p50_ms: pctl_ms(&g.net_us, 0.50),
lat_valid: !g.e2e_us.is_empty(),
@@ -315,6 +369,8 @@ impl VideoStats {
g.host_us.clear();
g.net_us.clear();
g.decode_us.clear();
g.display_us.clear();
g.e2e_disp_us.clear();
g.skipped = 0;
g.last_dropped_total = dropped_total;
g.last_fec_total = fec_total;