fix(client/net): split the receipt stamp from the pull + bleed standing latency
ci / docs-site (push) Successful in 56s
apple / swift (push) Successful in 1m14s
ci / web (push) Successful in 2m30s
decky / build-publish (push) Successful in 19s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 9s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m57s
release / apple (push) Successful in 9m8s
deb / build-publish (push) Successful in 9m33s
arch / build-publish (push) Successful in 13m14s
docker / deploy-docs (push) Successful in 23s
flatpak / build-publish (push) Failing after 8m1s
deb / build-publish-host (push) Successful in 10m24s
android / android (push) Successful in 14m56s
apple / screenshots (push) Successful in 6m43s
ci / rust (push) Successful in 19m29s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 16m35s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m29s
windows-host / package (push) Successful in 15m13s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 5m48s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 6m38s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 7m45s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 8m51s
ci / docs-site (push) Successful in 56s
apple / swift (push) Successful in 1m14s
ci / web (push) Successful in 2m30s
decky / build-publish (push) Successful in 19s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 9s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
ci / bench (push) Successful in 5m57s
release / apple (push) Successful in 9m8s
deb / build-publish (push) Successful in 9m33s
arch / build-publish (push) Successful in 13m14s
docker / deploy-docs (push) Successful in 23s
flatpak / build-publish (push) Failing after 8m1s
deb / build-publish-host (push) Successful in 10m24s
android / android (push) Successful in 14m56s
apple / screenshots (push) Successful in 6m43s
ci / rust (push) Successful in 19m29s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 16m35s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m29s
windows-host / package (push) Successful in 15m13s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 5m48s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 6m38s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 7m45s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 8m51s
The two-pair investigation (wired Mac clients stuck at a rock-steady ~18-19 ms "network" that survived the load ending and cleared only on reconnect) exposed two structural gaps, one of measurement and one of recovery: - Receipt was stamped at the hand-off PULL (Swift nextAU, pf-client-core, Android decode loops), not at reassembly completion — so any client-side standing state between the reassembler and the pull read as NETWORK latency, undiagnosable from the HUD. ABI v9: `PunktfunkFrame`/`Frame` grow `received_ns`, stamped by `Session::poll_frame` as the AU crosses the session boundary. Every embedder now uses the core stamp; the Apple client keeps the pull instant as `AccessUnit.pulledNs` and shows the receipt→pull wait as its own "client queue" term (detailed HUD tier from 2 ms + a `queue_p50` stats-log field). Decode stages keep their pull anchor on all platforms, so no historical stage shifts meaning. - The jump-to-live detectors deliberately ignore anything under 6 queued frames / 400 ms behind — so a small, constant, loss-free elevation (a sub-frame standing backlog, or a stale clock offset after a wall-clock step/slew) is carried for the rest of the session. New third detector (`StandingLatency`, unit-tested ladder): window-MIN one-way delay ≥ 10 ms above the session floor with zero loss for ~4.5 s escalates gently — a free clock re-sync first (an applied re-sync re-bases the floor), then at most 3 flush+keyframe bleeds sharing the jump-to-live cooldown, then a loud disarm naming what it means. Loss windows reset the run: congestion belongs to FEC/ABR, not this detector. Also: mid-stream re-sync apply/discard logs debug→info — they are the forensic trail for the stale-offset case and were invisible in the field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,9 @@
|
||||
//! The client worker: QUIC handshake + control/input/datagram tasks + the blocking data-plane pump.
|
||||
|
||||
use super::frame_channel::{
|
||||
CLOCK_RESYNC_INTERVAL, FLUSH_AFTER, FLUSH_COOLDOWN, FLUSH_LATENCY,
|
||||
NOOP_CLOCK_FLUSHES_TO_DISARM, NOOP_FLUSH_DATAGRAMS, QUEUE_HIGH, QUEUE_LOW, STANDING_TIME,
|
||||
StandingLatAction, StandingLatency, CLOCK_RESYNC_INTERVAL, FLUSH_AFTER, FLUSH_COOLDOWN,
|
||||
FLUSH_LATENCY, NOOP_CLOCK_FLUSHES_TO_DISARM, NOOP_FLUSH_DATAGRAMS, QUEUE_HIGH, QUEUE_LOW,
|
||||
STANDING_TIME,
|
||||
};
|
||||
use super::worker::reject_from_close;
|
||||
use super::*;
|
||||
@@ -514,15 +515,19 @@ pub(super) async fn run_pump(args: WorkerArgs) {
|
||||
// late exactly then) — keep the old estimate and let the next
|
||||
// periodic batch try again.
|
||||
if accept_resync(rtt_ns, clock_rtt_ns.unwrap_or(0)) {
|
||||
clock_offset.store(offset_ns, Ordering::Relaxed);
|
||||
clock_gen.fetch_add(1, Ordering::Relaxed);
|
||||
tracing::debug!(
|
||||
// info, not debug: ≤1/min, and it is THE forensic
|
||||
// trail for a stale-offset (stepped/slewed wall clock)
|
||||
// latency plateau — the 2026-07 two-pair investigation
|
||||
// had to reconstruct this blind.
|
||||
tracing::info!(
|
||||
offset_ns,
|
||||
rtt_us = rtt_ns / 1000,
|
||||
"mid-stream clock re-sync applied"
|
||||
);
|
||||
clock_offset.store(offset_ns, Ordering::Relaxed);
|
||||
clock_gen.fetch_add(1, Ordering::Relaxed);
|
||||
} else {
|
||||
tracing::debug!(
|
||||
tracing::info!(
|
||||
rtt_us = rtt_ns / 1000,
|
||||
"clock re-sync batch discarded — RTT above the \
|
||||
connect-time baseline (congested window)"
|
||||
@@ -729,6 +734,12 @@ pub(super) async fn run_pump(args: WorkerArgs) {
|
||||
let mut clock_detector_armed = true;
|
||||
let mut resync_wanted = false;
|
||||
let mut seen_clock_gen = pump_clock_gen.load(Ordering::Relaxed);
|
||||
// Standing-latency bleed (see StandingLatency): the third detector, for the small,
|
||||
// constant, loss-free OWD elevation the two jump-to-live detectors deliberately
|
||||
// tolerate (< QUEUE_HIGH frames, < FLUSH_LATENCY behind) — a sub-frame standing
|
||||
// backlog, or a stale clock offset after a wall-clock step, either of which otherwise
|
||||
// reads as permanent extra "network" latency for the rest of the session.
|
||||
let mut standing_lat = StandingLatency::new();
|
||||
while !pump_shutdown.load(Ordering::SeqCst) {
|
||||
// The live host↔client offset: re-loaded every iteration so an applied mid-stream
|
||||
// re-sync takes effect on the very next frame's latency math.
|
||||
@@ -740,6 +751,10 @@ pub(super) async fn run_pump(args: WorkerArgs) {
|
||||
seen_clock_gen = gen;
|
||||
stale_since = None;
|
||||
noop_clock_flushes = 0;
|
||||
// Every OWD reading shifted with the offset — the standing-latency floor and
|
||||
// any elevation measured under the old one are meaningless now. If a stale
|
||||
// offset WAS the elevation, this is also the moment it gets fixed.
|
||||
standing_lat.rebase();
|
||||
if !clock_detector_armed {
|
||||
clock_detector_armed = true;
|
||||
tracing::info!(
|
||||
@@ -843,6 +858,51 @@ pub(super) async fn run_pump(args: WorkerArgs) {
|
||||
window_dropped,
|
||||
);
|
||||
let _ = ctrl_tx.try_send(CtrlRequest::Loss(LossReport { loss_ppm }));
|
||||
// Standing-latency bleed: close the detector's window with this report's loss
|
||||
// verdict and run its escalation ladder — re-sync first (free; a stale offset
|
||||
// from a stepped wall clock produces exactly this signature and the applied
|
||||
// re-sync rebases the floor), then a bounded flush+keyframe (drains a real
|
||||
// sub-threshold standing backlog the jump-to-live thresholds tolerate), then a
|
||||
// loud disarm (the path latency itself changed; nothing local fixes that).
|
||||
match standing_lat.on_window(loss_ppm == 0 && window_dropped == 0) {
|
||||
StandingLatAction::None => {}
|
||||
StandingLatAction::Resync { above_ms } => {
|
||||
tracing::info!(
|
||||
above_ms,
|
||||
"standing latency above the session floor with zero loss — \
|
||||
requesting a clock re-sync first (a stale offset reads exactly \
|
||||
like this)"
|
||||
);
|
||||
let _ = ctrl_tx.try_send(CtrlRequest::ClockResync);
|
||||
}
|
||||
StandingLatAction::Bleed { above_ms } => {
|
||||
// Shares the jump-to-live cooldown: an unexecuted bleed simply re-arms
|
||||
// over the next windows (the detector's run rebuilds).
|
||||
if last_flush.is_none_or(|t| t.elapsed() >= FLUSH_COOLDOWN) {
|
||||
last_flush = Some(Instant::now());
|
||||
flush_in_window = true;
|
||||
let flushed = session.flush_backlog().unwrap_or(0);
|
||||
let dropped = frames.clear();
|
||||
let _ = ctrl_tx.try_send(CtrlRequest::Keyframe);
|
||||
standing_lat.bled();
|
||||
tracing::warn!(
|
||||
above_ms,
|
||||
flushed_datagrams = flushed,
|
||||
dropped_frames = dropped,
|
||||
"standing latency survived a clock re-sync — bled the local \
|
||||
backlog (flush + keyframe)"
|
||||
);
|
||||
}
|
||||
}
|
||||
StandingLatAction::Disarm { above_ms } => {
|
||||
tracing::warn!(
|
||||
above_ms,
|
||||
"standing latency persists after a re-sync and every bleed — not \
|
||||
local, not clock; the path latency changed. Leaving it be \
|
||||
(reconnect re-baselines)"
|
||||
);
|
||||
}
|
||||
}
|
||||
// Adaptive bitrate: drain any host ack first (its clamp is authoritative), then
|
||||
// feed the controller this window's congestion signals; a decision becomes a
|
||||
// SetBitrate on the control stream.
|
||||
@@ -981,6 +1041,13 @@ pub(super) async fn run_pump(args: WorkerArgs) {
|
||||
if clock_offset_ns != 0 && lat_ns > 0 {
|
||||
owd_sum_ns += lat_ns;
|
||||
owd_frames += 1;
|
||||
// The standing-latency detector rides the same signal, but off the
|
||||
// window MINIMUM (robust against jitter/burst spikes — a standing
|
||||
// state elevates the floor itself). Same 10 s plausibility clamp as
|
||||
// the hn stats use.
|
||||
if lat_ns < 10_000_000_000 {
|
||||
standing_lat.note_frame(lat_ns);
|
||||
}
|
||||
}
|
||||
if clock_detector_armed
|
||||
&& clock_offset_ns != 0
|
||||
|
||||
Reference in New Issue
Block a user