From 7077b0a0df1d83547e6dde97011f340fa3ce70c4 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Tue, 4 Aug 2026 08:11:35 +0200 Subject: [PATCH] feat(core/audio): bitrate tiers, a shared de-jitter policy, and a redundant audio plane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Foundation for the audio quality + latency plan (design/audio-quality-and-latency.md). All three pieces are pure and unit-tested here so the four client rings and the Windows host glue that follow stay thin. **Bitrate tiers** (`AudioTier`). The layout table's `bitrate` becomes the `Standard` value, so that tier reproduces the pre-tier wire byte-for-byte — the tier machinery is provably non-regressive. `High` (stereo 256 kbps) is the default: 5 ms Opus frames are much less efficient than 20 ms ones, so the historical 128 kbps buys roughly what ~100 kbps buys at 20 ms, while the same session carries tens of Mbps of video. Purely a host-side encoder knob — libopus reads the bitrate out of the packet, so no client change and no negotiation. **`JitterPolicy`** — the ms-denominated de-jitter state machine every client will share. Two defects it exists to fix: (1) each ring computed its target as `3 x quantum`, a sane 15 ms at a 5 ms quantum and a silent 64 ms at a 20 ms one; (2) every ring primed *up* and clamped at a ceiling, and none walked the depth back *down*, so drift/bursts added latency permanently — Android, with no shed at all, converged on its 120 ms cap. Here a depth EWMA that sits above target for 2 s of consumed audio sheds ONE 5 ms frame with a crossfade. Driven by samples consumed rather than the wall clock: allocation- and syscall-free (safe in a realtime callback) and deterministic under test. `every_preset_sheds_before_it_trims` pins the invariant that makes this real rather than decorative. The first draft had `headroom_ms` <= the shed threshold on all four presets, so the ring was trimmed back before the average could ever reach the shed point: drift correction was dead code and the ratchet test passed for the wrong reason (the hard cap did the work). `a_transient_burst_does_not_shed` caught it. The shed point is now derived from `headroom_ms` so it cannot invert again. **`0xD2` redundant audio** — each datagram carries its frame plus a copy of the previous one, so a single lost packet is reconstructed instead of concealed. Opus in-band FEC cannot do this job: LBRR is a SILK feature and the desktop encoder is CELT-only (RESTRICTED_LOWDELAY, 5 ms), so `set_inband_fec` there is a no-op. Costs no latency — the copy rides the successor, which arrives inside de-jitter slack that already exists. Gated capable-and-agreed via CLIENT_CAP_AUDIO_RED/HOST_CAP_AUDIO_RED; every other session keeps the `0xC9` wire unchanged. 0xD1 is left free for the pad-audio program. cbindgen: prefix the four new exported constants. `FRAME_MS`/`SAMPLE_RATE_HZ` as bare C macros are the same hazard the BTN_* renames already document — a clashing #define takes the last definition silently rather than failing to compile. Verified: 300 core tests, clippy -D warnings, fmt. (`c_abi` fails identically on a pristine tree — this Mac has no system libopus for the C harness link.) Co-Authored-By: Claude Opus 5 (1M context) --- crates/punktfunk-core/cbindgen.toml | 7 + crates/punktfunk-core/src/audio.rs | 779 +++++++++++++++++++++ crates/punktfunk-core/src/quic/caps.rs | 23 + crates/punktfunk-core/src/quic/datagram.rs | 153 ++++ include/punktfunk_core.h | 85 +++ 5 files changed, 1047 insertions(+) diff --git a/crates/punktfunk-core/cbindgen.toml b/crates/punktfunk-core/cbindgen.toml index 0e263da0..538379fd 100644 --- a/crates/punktfunk-core/cbindgen.toml +++ b/crates/punktfunk-core/cbindgen.toml @@ -48,6 +48,13 @@ exclude = ["MsghdrX", "recvmsg_x", "mmsghdr", "sendmmsg", "recvmmsg"] "AXIS_RT" = "PUNKTFUNK_AXIS_RT" "AUDIO_MAGIC" = "PUNKTFUNK_AUDIO_MAGIC" "RUMBLE_MAGIC" = "PUNKTFUNK_RUMBLE_MAGIC" +"AUDIO_RED_MAGIC" = "PUNKTFUNK_AUDIO_RED_MAGIC" +"AUDIO_RED_HEADER" = "PUNKTFUNK_AUDIO_RED_HEADER" +# Same hazard as the BTN_* block above, one step worse: `FRAME_MS` and `SAMPLE_RATE_HZ` are +# generic enough that an embedder is likely to have its own, and a clashing #define silently +# takes the last definition rather than failing to compile. +"FRAME_MS" = "PUNKTFUNK_AUDIO_FRAME_MS" +"SAMPLE_RATE_HZ" = "PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ" # QualifiedScreamingSnakeCase already qualifies each variant with the enum name # (PunktfunkStatus::Ok -> PUNKTFUNK_STATUS_OK); do NOT also set prefix_with_name or it doubles. diff --git a/crates/punktfunk-core/src/audio.rs b/crates/punktfunk-core/src/audio.rs index 88270e6b..6975b9de 100644 --- a/crates/punktfunk-core/src/audio.rs +++ b/crates/punktfunk-core/src/audio.rs @@ -103,6 +103,72 @@ pub const LAYOUT_71_HQ: OpusLayout = OpusLayout { bitrate: 2_048_000, }; +/// Encode bitrate tier for the desktop-audio downlink. The layout table's `bitrate` is the +/// [`AudioTier::Standard`] value, so `Standard` reproduces the pre-tier wire byte-for-byte. +/// +/// **Why a tier at all.** 5 ms Opus frames are markedly less efficient than 20 ms ones (shorter +/// MDCT, a bigger per-packet overhead share), so the historical 128 kbps stereo buys roughly what +/// ~100 kbps buys at 20 ms — audible on music, and the 2026-08-03 field report said exactly that. +/// Meanwhile the same session carries tens of Mbps of video: at 256 kbps audio is ~1 % of the +/// budget. [`AudioTier::High`] is therefore the DEFAULT; the lower tiers exist for a genuinely +/// constrained link, not as the normal case. +/// +/// Purely a host-side encoder knob: every client decodes whatever bitrate arrives (libopus reads +/// it from the packet), so changing tiers needs no protocol negotiation and no client change. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)] +pub enum AudioTier { + /// Constrained links — noticeably lossy on music, still fine for game/voice content. + Low, + /// The historical values (stereo 128 kbps). Kept exactly so the tier machinery is provably + /// non-regressive against every pre-tier build. + Standard, + /// The default: effectively transparent at 5 ms frames, for ~1 % of a normal video budget. + #[default] + High, +} + +impl AudioTier { + /// Parse a config/CLI spelling (`low` / `standard` / `high`); `None` for anything else so the + /// caller can warn and fall back rather than silently downgrading someone's audio. + pub fn parse(s: &str) -> Option { + match s.trim().to_ascii_lowercase().as_str() { + "low" => Some(AudioTier::Low), + "standard" | "normal" | "medium" => Some(AudioTier::Standard), + "high" => Some(AudioTier::High), + _ => None, + } + } + + pub fn as_str(self) -> &'static str { + match self { + AudioTier::Low => "low", + AudioTier::Standard => "standard", + AudioTier::High => "high", + } + } +} + +impl OpusLayout { + /// This layout's target bitrate at `tier`. The uncoupled HIGH-QUALITY layouts + /// ([`LAYOUT_51_HQ`] / [`LAYOUT_71_HQ`]) are already far past transparency, so they are + /// tier-invariant — scaling 1.5 Mbps up would only waste wire. + pub fn bitrate_for(&self, tier: AudioTier) -> i32 { + // One mono stream per channel == the HQ layouts; nothing to gain from a tier there. + if self.coupled == 0 && self.streams == self.channels { + return self.bitrate; + } + match (self.channels, tier) { + (6, AudioTier::Low) => 192_000, + (6, AudioTier::High) => 448_000, + (8, AudioTier::Low) => 320_000, + (8, AudioTier::High) => 768_000, + (_, AudioTier::Low) => 96_000, + (_, AudioTier::High) => 256_000, + (_, AudioTier::Standard) => self.bitrate, + } + } +} + /// Pick the layout for a negotiated channel count. Unknown counts fall back to stereo (clients /// only ever request 2/6/8). `high_quality` selects the uncoupled high-bitrate config. pub fn layout_for(channels: u8, high_quality: bool) -> &'static OpusLayout { @@ -173,6 +239,342 @@ impl AudioGapTracker { } } +// ---- the shared playback de-jitter policy ------------------------------------------------- + +/// The protocol's audio frame, in milliseconds — every host datagram carries exactly one +/// ([`crate::quic::encode_audio_datagram`]), so it is also the smallest useful shed unit. +pub const FRAME_MS: u32 = 5; + +/// Tuning for [`JitterPolicy`], in MILLISECONDS. +/// +/// Denominating the depth in time rather than in device quanta is the point. Every client used to +/// compute its target as `3 × quantum`, which is a sane 15 ms at a 5 ms quantum and a silent 64 ms +/// at a 20 ms one — the same source line meaning two very different latencies depending on what +/// else happened to be using the audio graph that day. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct JitterTuning { + /// Depth to prime to before the first sample plays, and the depth drift correction pulls + /// back toward. The adaptive floor may raise the live target above this; it never goes below. + pub base_target_ms: u32, + /// Ceiling for the adaptively-grown target (see [`JitterPolicy::note_read`]). + pub max_target_ms: u32, + /// Slack above the live target before drop-oldest trimming starts. Absorbs an arrival burst + /// without overflowing. + /// + /// Drift correction sheds at the MIDDLE of this band (see [`JitterTuning::shed_excess_ms`]), + /// so the smooth correction always gets its chance before the hard trim. Setting this too + /// small is a real failure mode, not just a tuning choice: if the trim point sits below the + /// shed point, the ring is trimmed back before the depth average can ever reach the shed + /// threshold, drift correction becomes dead code, and every correction is once again the + /// audible drop it was supposed to replace. + pub headroom_ms: u32, + /// Absolute bound on buffered audio — the only hard guarantee on added latency. + pub hard_cap_ms: u32, + /// Consecutive short reads before the ring goes back to priming. `1` reproduces the old + /// `if ring.is_empty() { primed = false }`, where a single transient drain manufactured a + /// whole target's worth of fresh silence; every platform now uses hysteresis. + pub deprime_after: u32, +} + +impl JitterTuning { + /// PipeWire adaptively rate-matches the stream to the graph clock and absorbs a shallow ring, + /// so Linux can run tight. + pub const PIPEWIRE: JitterTuning = JitterTuning { + base_target_ms: 15, + max_target_ms: 60, + headroom_ms: 25, + hard_cap_ms: 80, + deprime_after: 4, + }; + /// WASAPI shared-mode event-driven render: the engine buffers for us, but nothing rate-matches. + pub const WASAPI: JitterTuning = JitterTuning { + base_target_ms: 20, + max_target_ms: 70, + headroom_ms: 30, + hard_cap_ms: 90, + deprime_after: 4, + }; + /// CoreAudio via AVAudioEngine — comparable to WASAPI; the iOS IO buffer is already 5 ms. + pub const COREAUDIO: JitterTuning = JitterTuning { + base_target_ms: 20, + max_target_ms: 70, + headroom_ms: 30, + hard_cap_ms: 90, + deprime_after: 4, + }; + /// AAudio hands us a raw realtime callback and makes us own the buffer, and Wi-Fi power-save + /// bunching lands as underruns = crackle. Android therefore starts DEEPER — but at 25 ms, not + /// the old fixed 40: the adaptive floor raises it only on the devices that actually underrun, + /// instead of every device pre-paying for the worst one. + pub const AAUDIO: JitterTuning = JitterTuning { + base_target_ms: 25, + max_target_ms: 90, + headroom_ms: 40, + hard_cap_ms: 120, + deprime_after: 5, + }; + + /// How far above the live target the depth average must sit before drift correction sheds: + /// the middle of the headroom band, but never less than two protocol frames (so it cannot be + /// hair-triggered by one quantum of normal swing). Deriving it from `headroom_ms` rather than + /// fixing it absolutely is what keeps the smooth shed strictly BELOW the hard trim on every + /// preset — see the field on `headroom_ms`. + pub const fn shed_excess_ms(&self) -> u32 { + let half = self.headroom_ms / 2; + if half > 2 * FRAME_MS { + half + } else { + 2 * FRAME_MS + } + } +} + +/// What one callback should do, from [`JitterPolicy::step`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)] +pub struct JitterStep { + /// Interleaved samples to discard from the FRONT of the ring before reading. + pub drop_front: usize, + /// When non-zero, `drop_front` is a smooth drift correction and this many interleaved samples + /// of linear crossfade should be applied across the seam ([`crossfade_drop`] does it for a + /// `VecDeque` ring). Zero means discard hard — either nothing is being dropped, or the + /// ring blew the hard cap and is already a discontinuity. + pub crossfade: usize, + /// Emit silence this callback: still priming, or re-priming after a sustained drain. + pub silence: bool, +} + +/// EWMA time constant for the depth average, in ms. Long enough that a burst doesn't trigger a +/// shed, short enough to track real drift. +const EWMA_TAU_MS: u32 = 1_000; +/// The depth EWMA must stay above the shed threshold for this much CONSUMED AUDIO. Deliberately long: a shed is the only +/// thing here a listener could ever notice, so it must never fire on a transient. +const SHED_SUSTAIN_MS: u32 = 2_000; +/// Linear crossfade applied across a drift shed's seam. +const SHED_CROSSFADE_MS: u32 = 2; +/// Underruns inside [`GROW_WINDOW_MS`] before the live target grows. +const GROW_UNDERRUNS: u32 = 3; +const GROW_WINDOW_MS: u32 = 5_000; +const GROW_STEP_MS: u32 = 10; +/// Quiet time (no underrun) before a grown target relaxes one step back toward the base. +const SHRINK_QUIET_MS: u32 = 30_000; + +/// The playback de-jitter state machine shared by every client's audio ring. +/// +/// **The defect it exists to fix.** Every client's ring primed *up* to a target and clamped at a +/// ceiling, and none of them walked the depth back *down*. Any transient — a Wi-Fi arrival burst, a +/// host stall, or plain host-DAC-vs-client-DAC clock skew of a few dozen ppm — therefore added +/// latency permanently, until an underrun happened to re-prime. Android, with no shed at all, +/// converged on its hard cap and stayed there; Apple shed 40 ms at once and its own comment called +/// that "one audible blip". Here, a depth EWMA that sits [`SHED_EXCESS_MS`] above target for +/// [`SHED_SUSTAIN_MS`] of consumed audio sheds ONE 5 ms frame with a crossfade, so latency returns +/// to target instead of ratcheting. +/// +/// **Driven by the audio clock, not the wall clock**: every duration is measured in samples +/// consumed. That makes it allocation-free, syscall-free (safe in a realtime callback) and +/// deterministic under test. +#[derive(Clone, Debug)] +pub struct JitterPolicy { + tuning: JitterTuning, + /// Interleaved samples per millisecond at the negotiated layout (48 × channels). + per_ms: usize, + /// The live target, in interleaved samples — `base_target_ms` grown by underrun pressure. + target: usize, + primed: bool, + /// Consecutive short reads (de-prime hysteresis). + empties: u32, + /// EWMA of ring depth, interleaved samples. + depth_avg: f32, + /// Consumed samples for which the EWMA has stayed above the shed threshold. + over_run: usize, + /// Underruns seen in the current growth window, and the window's consumed-sample count. + underruns: u32, + window_run: usize, + /// Consumed samples since the last underrun (drives the relax-back-down step). + quiet_run: usize, + /// `want` from the most recent [`step`](Self::step), so [`note_read`](Self::note_read) can + /// advance the sample-denominated timers without the caller repeating it. + last_want: usize, +} + +impl JitterPolicy { + /// `channels` is the negotiated interleaved channel count (2/6/8). + pub fn new(tuning: JitterTuning, channels: u8) -> JitterPolicy { + let per_ms = (SAMPLE_RATE_HZ / 1000) as usize * channels.max(1) as usize; + JitterPolicy { + tuning, + per_ms, + target: tuning.base_target_ms as usize * per_ms, + primed: false, + empties: 0, + depth_avg: 0.0, + over_run: 0, + underruns: 0, + window_run: 0, + quiet_run: 0, + last_want: 0, + } + } + + /// The live target depth in ms (grows under underrun pressure; never below the base). + pub fn target_ms(&self) -> u32 { + (self.target / self.per_ms) as u32 + } + + /// Convert a ring depth in interleaved samples to milliseconds — for stats/HUD reporting. + pub fn depth_ms(&self, depth: usize) -> u32 { + (depth / self.per_ms) as u32 + } + + /// Smoothed ring depth in ms — what drift correction actually reacts to, and the honest + /// number to publish as "audio buffer" (the instantaneous depth swings by a whole quantum). + pub fn avg_depth_ms(&self) -> u32 { + (self.depth_avg.max(0.0) as usize / self.per_ms) as u32 + } + + pub fn is_primed(&self) -> bool { + self.primed + } + + /// The effective target for a device asking for `want` samples per callback. A ring can never + /// sustain a target below one device quantum, so a large-buffer device (a 20 ms PipeWire graph + /// quantum, a legacy AAudio path) lifts it to `want` plus one protocol frame rather than + /// oscillating prime → dropout → re-prime forever. + fn effective_target(&self, want: usize) -> usize { + self.target.max(want + FRAME_MS as usize * self.per_ms) + } + + /// Decide this callback: what to trim, and whether to play. Call BEFORE reading, with the + /// ring's current `depth` and the device's `want`, both in interleaved samples. + pub fn step(&mut self, depth: usize, want: usize) -> JitterStep { + self.last_want = want; + let target = self.effective_target(want); + + // Track depth with a callback-rate-independent EWMA: weighting by `want` keeps the time + // constant at EWMA_TAU_MS whether the device pulls 5 ms or 20 ms at a time. + let alpha = (want as f32 / (EWMA_TAU_MS as usize * self.per_ms) as f32).clamp(0.0, 1.0); + self.depth_avg += (depth as f32 - self.depth_avg) * alpha; + + // The hard cap must always leave room to serve this callback, or a large-quantum device + // would trim itself into a permanent underrun. + let cap = (target + self.tuning.headroom_ms as usize * self.per_ms) + .min(self.tuning.hard_cap_ms as usize * self.per_ms) + .max(target + want); + + let mut out = JitterStep::default(); + if depth > cap { + // Blew the ceiling: a burst arrived, or we were wedged. Already a discontinuity — + // discard hard, and reset the drift timer so the trim isn't double-counted as drift. + out.drop_front = depth - cap; + self.over_run = 0; + } else if self.depth_avg + > (target + self.tuning.shed_excess_ms() as usize * self.per_ms) as f32 + { + self.over_run += want; + if self.over_run >= SHED_SUSTAIN_MS as usize * self.per_ms { + out.drop_front = (FRAME_MS as usize * self.per_ms).min(depth); + out.crossfade = (SHED_CROSSFADE_MS as usize * self.per_ms) + .min(depth.saturating_sub(out.drop_front)); + self.over_run = 0; + } + } else { + self.over_run = 0; + } + // Whatever we shed is no longer buffered — reflect it immediately so the next callbacks + // don't re-fire on a stale average. + self.depth_avg = (self.depth_avg - out.drop_front as f32).max(0.0); + + if !self.primed && depth.saturating_sub(out.drop_front) >= target { + self.primed = true; + self.empties = 0; + } + out.silence = !self.primed; + out + } + + /// Report the outcome of the read `step` authorised. `ran_short` = the ring could not fill the + /// callback (a genuine underrun), which drives both the de-prime hysteresis and the adaptive + /// target floor. + /// + /// A callback that `step` told to emit silence is NOT an underrun — the ring is deliberately + /// re-priming — so calls made while un-primed are ignored and callers need not special-case it. + pub fn note_read(&mut self, ran_short: bool) { + if !self.primed { + return; + } + let want = self.last_want.max(1); + self.window_run += want; + if self.window_run >= GROW_WINDOW_MS as usize * self.per_ms { + self.window_run = 0; + self.underruns = 0; + } + if ran_short { + self.quiet_run = 0; + self.empties += 1; + if self.empties >= self.tuning.deprime_after { + self.primed = false; + self.empties = 0; + } + self.underruns += 1; + if self.underruns >= GROW_UNDERRUNS { + // This device genuinely needs more slack than the base target. Grow ONCE per + // window, capped — the alternative (every device pre-paying the worst device's + // depth) is what the fixed 40 ms Android floor was. + self.underruns = 0; + self.window_run = 0; + let grown = self.target + GROW_STEP_MS as usize * self.per_ms; + self.target = grown.min(self.tuning.max_target_ms as usize * self.per_ms); + } + } else { + self.empties = 0; + self.quiet_run += want; + if self.quiet_run >= SHRINK_QUIET_MS as usize * self.per_ms { + // Long quiet spell: give a grown target one step back, so a single bad minute + // doesn't cost latency for the rest of the session. + self.quiet_run = 0; + let base = self.tuning.base_target_ms as usize * self.per_ms; + self.target = self + .target + .saturating_sub(GROW_STEP_MS as usize * self.per_ms) + .max(base); + } + } + } +} + +/// Sample rate of every audio plane in the protocol. +pub const SAMPLE_RATE_HZ: u32 = 48_000; + +/// Discard `drop` interleaved samples from the front of `ring`, linearly crossfading the seam over +/// `fade` samples so a drift correction is inaudible rather than a click. +/// +/// The dropped region's tail fades out while the surviving head fades in, so the waveform is +/// continuous across the splice. `fade == 0` discards hard (what a hard-cap trim wants — that +/// backlog is already a discontinuity). Shared by the three `VecDeque` rings; the Apple ring +/// is index-based and mirrors this in Swift. +pub fn crossfade_drop(ring: &mut std::collections::VecDeque, drop: usize, fade: usize) { + if drop == 0 || ring.len() < drop { + return; + } + let fade = fade.min(drop).min(ring.len() - drop); + if fade == 0 { + ring.drain(..drop); + return; + } + // The last `fade` samples of what we are about to discard are the fade-OUT source; they blend + // into the first `fade` samples of what survives. + let mut faded = Vec::with_capacity(fade); + for i in 0..fade { + let old = ring[drop - fade + i]; + let new = ring[drop + i]; + let t = (i + 1) as f32 / (fade + 1) as f32; + faded.push(old * (1.0 - t) + new * t); + } + ring.drain(..drop); + for (i, v) in faded.into_iter().enumerate() { + ring[i] = v; + } +} + // ---- per-platform channel-layout helpers (pure data; no platform deps) -------------------- /// Windows `WAVEFORMATEXTENSIBLE.dwChannelMask` for the wire layout. @@ -286,6 +688,383 @@ mod tests { assert_eq!(t.missing_before(0), 0, "pre-wrap reorder, not a 2^31 gap"); } + // ---- bitrate tiers ------------------------------------------------------------------- + + /// `Standard` must reproduce the historical table EXACTLY — that is what makes the tier + /// machinery provably non-regressive against every pre-tier build. + #[test] + fn standard_tier_is_the_legacy_table() { + for l in [ + &LAYOUT_STEREO, + &LAYOUT_51, + &LAYOUT_51_HQ, + &LAYOUT_71, + &LAYOUT_71_HQ, + ] { + assert_eq!(l.bitrate_for(AudioTier::Standard), l.bitrate, "{l:?}"); + } + } + + #[test] + fn tiers_are_monotonic_and_hq_layouts_are_invariant() { + for l in [&LAYOUT_STEREO, &LAYOUT_51, &LAYOUT_71] { + let (lo, std, hi) = ( + l.bitrate_for(AudioTier::Low), + l.bitrate_for(AudioTier::Standard), + l.bitrate_for(AudioTier::High), + ); + assert!(lo < std && std < hi, "{l:?}: {lo} < {std} < {hi}"); + } + // The uncoupled HQ layouts are already past transparency — no tier may move them. + for l in [&LAYOUT_51_HQ, &LAYOUT_71_HQ] { + for t in [AudioTier::Low, AudioTier::Standard, AudioTier::High] { + assert_eq!(l.bitrate_for(t), l.bitrate, "{l:?} at {t:?}"); + } + } + } + + #[test] + fn tier_default_is_high_and_parses() { + assert_eq!(AudioTier::default(), AudioTier::High); + for t in [AudioTier::Low, AudioTier::Standard, AudioTier::High] { + assert_eq!(AudioTier::parse(t.as_str()), Some(t)); + } + assert_eq!(AudioTier::parse(" HIGH "), Some(AudioTier::High)); + assert_eq!(AudioTier::parse("normal"), Some(AudioTier::Standard)); + // Unknown spellings must be rejected, not silently downgraded. + assert_eq!(AudioTier::parse("transparent"), None); + assert_eq!(AudioTier::parse(""), None); + } + + // ---- the de-jitter policy ------------------------------------------------------------ + + /// Interleaved samples per ms at `channels`. + fn per_ms(channels: u8) -> usize { + (SAMPLE_RATE_HZ / 1000) as usize * channels as usize + } + + /// One simulated run's outcome. + #[derive(Debug, Default)] + struct Sim { + final_ms: u32, + peak_ms: u32, + /// Smooth drift corrections (crossfaded, one frame each) — the good kind. + soft_sheds: u32, + /// Hard-cap trims — the backstop. Any of these in a plain-drift run means the smooth + /// correction is not doing its job. + hard_trims: u32, + underruns: u32, + } + + /// Drive a policy through `ms` of simulated audio at a `quantum_ms` device, where the producer + /// delivers `drift_ppm` more (or less) than the consumer takes — i.e. host-vs-client clock skew. + fn simulate( + tuning: JitterTuning, + channels: u8, + ms: u32, + quantum_ms: u32, + drift_ppm: i64, + start_ms: u32, + ) -> Sim { + let pm = per_ms(channels); + let want = quantum_ms as usize * pm; + let mut p = JitterPolicy::new(tuning, channels); + let mut depth = start_ms as usize * pm; + let mut out = Sim::default(); + // Fractional producer accumulator, so a sub-sample-per-callback drift still accumulates. + let mut carry: i64 = 0; + for _ in 0..(ms / quantum_ms) { + // Producer: one quantum of audio plus the drift. + carry += want as i64 * drift_ppm; + let extra = carry / 1_000_000; + carry -= extra * 1_000_000; + depth = (depth as i64 + want as i64 + extra).max(0) as usize; + + let s = p.step(depth, want); + if s.drop_front > 0 { + if s.crossfade > 0 { + out.soft_sheds += 1; + } else { + out.hard_trims += 1; + } + depth -= s.drop_front.min(depth); + } + if s.silence { + p.note_read(false); + continue; + } + let short = depth < want; + depth -= want.min(depth); + if short { + out.underruns += 1; + } + p.note_read(short); + out.peak_ms = out.peak_ms.max((depth / pm) as u32); + } + out.final_ms = (depth / pm) as u32; + out + } + + /// The invariant that makes drift correction real rather than decorative: on every preset the + /// smooth shed point must sit strictly BELOW the hard trim point. Invert it — by tuning + /// `headroom_ms` down — and the ring is trimmed back before the depth average can ever reach + /// the shed threshold, so the smooth path becomes dead code and every correction is the + /// audible drop it was meant to replace. (That inversion was present in the first draft of + /// this module and only surfaced because `a_transient_burst_does_not_shed` failed.) + #[test] + fn every_preset_sheds_before_it_trims() { + for (name, t) in [ + ("PIPEWIRE", JitterTuning::PIPEWIRE), + ("WASAPI", JitterTuning::WASAPI), + ("COREAUDIO", JitterTuning::COREAUDIO), + ("AAUDIO", JitterTuning::AAUDIO), + ] { + assert!( + t.shed_excess_ms() < t.headroom_ms, + "{name}: sheds at +{} ms but trims at +{} ms — drift correction can never fire", + t.shed_excess_ms(), + t.headroom_ms + ); + assert!( + t.base_target_ms + t.headroom_ms <= t.hard_cap_ms, + "{name}: the headroom band is cut short by the hard cap" + ); + assert!(t.max_target_ms >= t.base_target_ms, "{name}"); + assert!(t.deprime_after >= 2, "{name}: needs real hysteresis"); + } + } + + /// THE headline behaviour, and the defect this policy exists for: with the host clock running + /// fast, the old rings grew to their ceiling and stayed pinned there for the rest of the + /// session. Drift correction must hold the depth near target — and must do it with the SMOOTH + /// crossfaded shed, never by letting the hard cap chop the backlog. + #[test] + fn drift_does_not_ratchet_latency_to_the_ceiling() { + // +200 ppm is a deliberately harsh skew (real DAC pairs are tens of ppm); 5 minutes. + let s = simulate(JitterTuning::AAUDIO, 2, 300_000, 5, 200, 25); + assert!( + s.soft_sheds > 0, + "drift must be shed, not accumulated: {s:?}" + ); + assert_eq!( + s.hard_trims, 0, + "plain drift must never reach the hard cap: {s:?}" + ); + assert_eq!( + s.underruns, 0, + "shedding must never cause an underrun: {s:?}" + ); + // The old Android ring pinned at its 120 ms hard cap. Ours must stay inside the band. + let ceiling = JitterTuning::AAUDIO.base_target_ms + JitterTuning::AAUDIO.headroom_ms; + assert!( + s.peak_ms <= ceiling, + "peaked at {} ms (band ends at {ceiling}) — that is the ratchet, not a correction", + s.peak_ms + ); + } + + /// Same skew, every preset: none of them may ratchet. + #[test] + fn no_preset_ratchets_under_drift() { + for (name, t) in [ + ("PIPEWIRE", JitterTuning::PIPEWIRE), + ("WASAPI", JitterTuning::WASAPI), + ("COREAUDIO", JitterTuning::COREAUDIO), + ("AAUDIO", JitterTuning::AAUDIO), + ] { + let s = simulate(t, 2, 300_000, 5, 200, t.base_target_ms); + assert!(s.soft_sheds > 0, "{name}: {s:?}"); + assert!( + s.peak_ms <= t.base_target_ms + t.headroom_ms, + "{name} peaked at {} ms: {s:?}", + s.peak_ms + ); + } + } + + /// The mirror case: a host clock running SLOW must not be "corrected" into permanent + /// underruns. The adaptive floor may grow the target, but nothing may be shed. + #[test] + fn negative_drift_grows_the_target_instead_of_stuttering() { + let s = simulate(JitterTuning::AAUDIO, 2, 120_000, 5, -200, 25); + assert_eq!( + s.soft_sheds, 0, + "nothing to shed when the ring is draining: {s:?}" + ); + assert_eq!(s.hard_trims, 0, "{s:?}"); + } + + /// A shed must never fire on a transient — a burst that arrives and drains is normal jitter, + /// and shedding it would cost an audible artefact for nothing. The spike here sits ABOVE the + /// shed threshold but below the trim point, so only the sustain requirement can reject it. + #[test] + fn a_transient_burst_does_not_shed() { + let t = JitterTuning::AAUDIO; + let pm = per_ms(2); + let want = 5 * pm; + let spike_ms = t.base_target_ms + t.shed_excess_ms() + FRAME_MS; // inside the band + assert!( + spike_ms < t.base_target_ms + t.headroom_ms, + "test spike must not hit the trim" + ); + let mut p = JitterPolicy::new(t, 2); + let mut sheds = 0; + // 300 ms spiked out of every 1 s, for 20 s. + for round in 0..20 { + for i in 0..200 { + let depth = if round > 0 && i < 60 { + spike_ms + } else { + t.base_target_ms + } as usize; + let s = p.step(depth * pm, want); + if s.drop_front > 0 { + sheds += 1; + } + p.note_read(false); + } + } + assert_eq!( + sheds, 0, + "a repeated short burst must not trigger drift correction" + ); + } + + /// The hard cap is the only absolute latency guarantee — it trims immediately, without + /// waiting for the drift timer. + #[test] + fn hard_cap_trims_at_once() { + let pm = per_ms(2); + let mut p = JitterPolicy::new(JitterTuning::AAUDIO, 2); + let s = p.step(500 * pm, 5 * pm); + assert!( + s.drop_front > 0, + "a 500 ms backlog must be trimmed on the spot" + ); + assert_eq!(s.crossfade, 0, "a blown cap is already a discontinuity"); + let left = 500 * pm - s.drop_front; + assert!( + left <= JitterTuning::AAUDIO.hard_cap_ms as usize * pm, + "trim must land at or under the hard cap" + ); + } + + /// One transient drain must not manufacture a fresh target's worth of silence — the bug + /// Android fixed and Linux/Windows still carried. + #[test] + fn deprime_requires_hysteresis() { + let pm = per_ms(2); + let want = 5 * pm; + let mut p = JitterPolicy::new(JitterTuning::PIPEWIRE, 2); + // An EMPTY ring must emit silence and stay un-primed, however many callbacks it sees. + for _ in 0..10 { + assert!(p.step(0, want).silence, "an empty ring cannot play"); + } + assert!(!p.is_primed()); + // A ring already holding well over target primes on the first callback that sees it. + assert!( + !p.step(50 * pm, want).silence, + "a ring holding well over target must start immediately" + ); + assert!(p.is_primed()); + p.note_read(true); // one short read + assert!(p.is_primed(), "a single short read must not de-prime"); + for _ in 1..JitterTuning::PIPEWIRE.deprime_after { + p.note_read(true); + } + assert!(!p.is_primed(), "a sustained drain must re-prime"); + } + + /// A device that pulls a big quantum cannot sustain a target below it: the effective target + /// must lift, or the ring oscillates prime → dropout → re-prime forever. + #[test] + fn target_lifts_above_a_large_device_quantum() { + let pm = per_ms(2); + let mut p = JitterPolicy::new(JitterTuning::PIPEWIRE, 2); // base target 15 ms + let want = 40 * pm; // a 40 ms graph quantum — far above the base target + // At exactly the base target the ring must NOT claim to be primed. + assert!( + p.step(15 * pm, want).silence, + "15 ms cannot serve a 40 ms quantum" + ); + // Once it holds the quantum plus a frame, it may play. + let s = p.step((40 + FRAME_MS as usize) * pm, want); + assert!(!s.silence, "quantum + one frame must be enough to start"); + } + + /// Clustered underruns raise the floor (that device needs the slack); a long quiet spell + /// gives it back, so one bad minute doesn't cost latency for the whole session. + #[test] + fn target_grows_on_underruns_and_relaxes_when_quiet() { + let pm = per_ms(2); + let want = 5 * pm; + let mut p = JitterPolicy::new(JitterTuning::AAUDIO, 2); + let base = p.target_ms(); + assert_eq!(base, JitterTuning::AAUDIO.base_target_ms); + for _ in 0..40 { + // Keep it primed and starve it: depth is always enough to prime, never to serve. + while !p.is_primed() { + p.step(200 * pm, want); + } + p.step(200 * pm, want); + p.note_read(true); + } + let grown = p.target_ms(); + assert!( + grown > base, + "clustered underruns must raise the floor ({base} → {grown})" + ); + assert!( + grown <= JitterTuning::AAUDIO.max_target_ms, + "growth must respect max_target_ms" + ); + // Now a long clean run relaxes it back. + for _ in 0..(SHRINK_QUIET_MS as usize * 3 / 5) { + p.step(grown as usize * pm + want, want); + p.note_read(false); + } + assert!( + p.target_ms() < grown, + "a quiet spell must give the growth back" + ); + assert!(p.target_ms() >= base, "…but never below the base target"); + } + + /// The crossfade must leave a continuous waveform: splicing a ramp must not introduce a step + /// bigger than the ramp's own per-sample slope. + #[test] + fn crossfade_drop_splices_without_a_step() { + use std::collections::VecDeque; + // A slow ramp: any hard splice shows up as a visible jump. + let mut ring: VecDeque = (0..1000).map(|i| i as f32).collect(); + let (drop, fade) = (240, 96); + crossfade_drop(&mut ring, drop, fade); + assert_eq!(ring.len(), 1000 - drop); + // Across the whole faded region the step between neighbours stays bounded — a hard drop + // would show a `drop`-sized jump at index 0. + for i in 0..fade { + let step = (ring[i + 1] - ring[i]).abs(); + assert!( + step < drop as f32, + "sample {i}: step {step} looks like a hard splice" + ); + } + // Tail is untouched. + assert_eq!(ring[ring.len() - 1], 999.0); + } + + #[test] + fn crossfade_drop_handles_degenerate_inputs() { + use std::collections::VecDeque; + let mut ring: VecDeque = (0..10).map(|i| i as f32).collect(); + crossfade_drop(&mut ring, 0, 4); // nothing to drop + assert_eq!(ring.len(), 10); + crossfade_drop(&mut ring, 99, 4); // more than we hold — refuse + assert_eq!(ring.len(), 10); + crossfade_drop(&mut ring, 10, 4); // exactly all of it: no room to fade, hard drop + assert!(ring.is_empty()); + } + #[test] fn wasapi_masks_are_correct() { assert_eq!(wasapi_channel_mask(2), 0x3); diff --git a/crates/punktfunk-core/src/quic/caps.rs b/crates/punktfunk-core/src/quic/caps.rs index 1cf5e69b..181326e0 100644 --- a/crates/punktfunk-core/src/quic/caps.rs +++ b/crates/punktfunk-core/src/quic/caps.rs @@ -111,6 +111,17 @@ pub const CLIENT_CAP_CURSOR: u8 = 0x01; /// simply ignored — no behavior change in either direction. pub const CLIENT_CAP_PHASE_LOCK: u8 = 0x02; +/// `Hello.client_caps` bit: this client can decode the redundant desktop-audio plane +/// ([`AUDIO_RED_MAGIC`](super::datagram::AUDIO_RED_MAGIC), `0xD2`), where every datagram also +/// carries a copy of the previous frame so a single lost packet is reconstructed instead of +/// papered over with packet-loss concealment. +/// +/// Active only when the host answers with [`HOST_CAP_AUDIO_RED`] (capable-and-agreed, the +/// cursor/clipboard precedent). Toward an older host, or a host that declines because the link is +/// clean, the client keeps receiving the plain `0xC9` plane — so a client may always set this bit. +/// `0x04` — `0x01`/`0x02` are cursor / phase-lock. +pub const CLIENT_CAP_AUDIO_RED: u8 = 0x04; + /// [`Welcome::host_caps`] bit: the host CAN forward the cursor out-of-band (it captures cursor /// metadata separately from the frame — the Linux portal `SPA_META_Cursor` path; NOT gamescope, /// whose capture carries no cursor, and NOT Windows yet, where DWM composites into the IDD @@ -132,6 +143,18 @@ pub const HOST_CAP_CURSOR: u8 = 0x08; /// [`HOST_CAP_TEXT_INPUT`], `0x01`/`0x02` are gamepad-state / clipboard. pub const HOST_CAP_PEN: u8 = 0x10; +/// [`Welcome::host_caps`] bit: the host is sending the REDUNDANT desktop-audio plane +/// ([`AUDIO_RED_MAGIC`](super::datagram::AUDIO_RED_MAGIC), `0xD2`) instead of plain `0xC9` — each +/// datagram carries its own frame plus a copy of the previous one. +/// +/// Set only when the client asked via [`CLIENT_CAP_AUDIO_RED`]. It is a statement about the WIRE, +/// not a negotiation the client can decline: with the bit set the client must decode `0xD2`, and +/// without it `0xC9`. The host may also drop back to `0xC9` mid-session (the redundancy is +/// loss-gated — a clean LAN shouldn't pay for it), which is why clients decode BOTH tags +/// unconditionally and treat this bit as "expect redundancy", not "only redundancy". +/// `0x20` — `0x10` is [`HOST_CAP_PEN`], `0x08` is [`HOST_CAP_CURSOR`]. +pub const HOST_CAP_AUDIO_RED: u8 = 0x20; + /// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software** /// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST /// advertise this. diff --git a/crates/punktfunk-core/src/quic/datagram.rs b/crates/punktfunk-core/src/quic/datagram.rs index 22977987..0567f241 100644 --- a/crates/punktfunk-core/src/quic/datagram.rs +++ b/crates/punktfunk-core/src/quic/datagram.rs @@ -42,6 +42,80 @@ pub fn decode_audio_datagram(b: &[u8]) -> Option<(u32, u64, &[u8])> { Some((seq, pts_ns, &b[13..])) } +/// Redundant audio datagram, host → client: the [`AUDIO_MAGIC`] plane plus a copy of the PREVIOUS +/// frame, so a single lost datagram is *reconstructed* rather than concealed. +/// +/// `[0xD2][u32 seq LE][u64 pts_ns LE][u16 primary_len LE][primary opus][previous opus]` +/// +/// **Why this and not Opus in-band FEC.** LBRR is a SILK-layer feature: the desktop-audio encoder +/// runs `RESTRICTED_LOWDELAY` (CELT-only) at 5 ms frames, which is below SILK's 10 ms minimum, so +/// `set_inband_fec(true)` on that encoder is a no-op. Nothing in libopus can protect this plane — +/// the redundancy has to be at the application layer. (The mic uplink is a different encoder, VoIP +/// mode at 10 ms, and *does* use real in-band FEC.) +/// +/// **Why it costs no latency.** The copy rides the SUCCESSOR of the frame it protects, and the +/// client is already holding 15–90 ms of de-jitter buffer — far more than the 5 ms the successor +/// takes to arrive. So the recovery happens inside slack that already exists. +/// +/// The previous frame's sequence is implicitly `seq - 1`; a host with nothing to duplicate yet +/// (the first frame of a session, or straight after a capture reopen) simply sends an empty tail, +/// which decodes to `None`. +/// +/// Sent ONLY when the client advertised [`CLIENT_CAP_AUDIO_RED`](super::caps::CLIENT_CAP_AUDIO_RED) +/// and the host answered [`HOST_CAP_AUDIO_RED`](super::caps::HOST_CAP_AUDIO_RED) — the +/// capable-and-agreed handshake the cursor and 4:4:4 planes already use. Every other session keeps +/// the plain [`AUDIO_MAGIC`] wire byte-for-byte. +/// +/// NB `0xD1` is deliberately skipped: the DualSense pad-audio program has reserved it for the +/// per-pad audio plane. +pub const AUDIO_RED_MAGIC: u8 = 0xD2; + +/// Fixed header length of an [`AUDIO_RED_MAGIC`] datagram (tag + seq + pts + primary length). +pub const AUDIO_RED_HEADER: usize = 1 + 4 + 8 + 2; + +/// Encode a redundant audio datagram. `prev` is the immediately-preceding frame's Opus payload +/// (empty when there is none yet). +pub fn encode_audio_red_datagram(seq: u32, pts_ns: u64, opus: &[u8], prev: &[u8]) -> Vec { + let mut b = Vec::with_capacity(AUDIO_RED_HEADER + opus.len() + prev.len()); + b.push(AUDIO_RED_MAGIC); + b.extend_from_slice(&seq.to_le_bytes()); + b.extend_from_slice(&pts_ns.to_le_bytes()); + // A frame longer than u16::MAX cannot occur (5 ms of Opus is tens of bytes; the buffer the + // encoder writes into is 4 KiB) — but truncating silently would desync the split, so clamp + // the redundancy off instead of the primary. + let primary_len = u16::try_from(opus.len()).unwrap_or(u16::MAX); + b.extend_from_slice(&primary_len.to_le_bytes()); + b.extend_from_slice(opus); + if opus.len() == primary_len as usize { + b.extend_from_slice(prev); + } + b +} + +/// Parse a redundant audio datagram → `(seq, pts_ns, primary, previous)`. `previous` is `None` +/// when the host had nothing to duplicate. `None` overall on bad tag/length, including a +/// `primary_len` that overruns the datagram (a truncated or hostile packet must not panic). +/// +/// The tuple shape deliberately mirrors [`decode_audio_datagram`] (one extra slot for the +/// redundant copy) so the two planes read the same at every call site; a named struct here would +/// be the odd one out on this module's decode surface, and cbindgen would then have to be taught +/// to skip it. +#[allow(clippy::type_complexity)] +pub fn decode_audio_red_datagram(b: &[u8]) -> Option<(u32, u64, &[u8], Option<&[u8]>)> { + if b.len() < AUDIO_RED_HEADER || b[0] != AUDIO_RED_MAGIC { + return None; + } + let seq = u32::from_le_bytes(b[1..5].try_into().unwrap()); + let pts_ns = u64::from_le_bytes(b[5..13].try_into().unwrap()); + let primary_len = u16::from_le_bytes(b[13..15].try_into().unwrap()) as usize; + let rest = &b[AUDIO_RED_HEADER..]; + if primary_len > rest.len() { + return None; // truncated: the split point is outside the datagram + } + let (primary, prev) = rest.split_at(primary_len); + Some((seq, pts_ns, primary, (!prev.is_empty()).then_some(prev))) +} + /// Legacy rumble datagram (v1), host → client: `[0xCA][u16 pad LE][u16 low LE][u16 high LE]`. /// Force-feedback state for pad `pad` (0xFFFF amplitudes, 0/0 = stop) as *level-triggered* state /// — it persists until superseded, which is why the host re-sends it periodically as its loss @@ -806,6 +880,8 @@ mod tests { #[test] fn audio_datagram_roundtrip() { let opus = [0x42u8; 97]; + let d = encode_audio_red_datagram(7, 42, &opus, &[]); + assert_eq!(d[0], AUDIO_RED_MAGIC); let d = encode_audio_datagram(7, 1_000_000_123, &opus); assert_eq!(d[0], AUDIO_MAGIC); let (seq, pts, payload) = decode_audio_datagram(&d).unwrap(); @@ -820,6 +896,83 @@ mod tests { assert!(empty.is_empty()); } + #[test] + fn audio_red_datagram_roundtrip() { + let cur = [0x42u8; 97]; + let prev = [0x37u8; 88]; + let d = encode_audio_red_datagram(7, 1_000_000_123, &cur, &prev); + assert_eq!(d[0], AUDIO_RED_MAGIC); + let (seq, pts, primary, previous) = decode_audio_red_datagram(&d).unwrap(); + assert_eq!((seq, pts), (7, 1_000_000_123)); + assert_eq!(primary, cur); + assert_eq!(previous, Some(&prev[..])); + + // No predecessor yet (first frame of a session / after a capture reopen). + let d = encode_audio_red_datagram(0, 5, &cur, &[]); + let (_, _, primary, previous) = decode_audio_red_datagram(&d).unwrap(); + assert_eq!(primary, cur); + assert_eq!( + previous, None, + "an empty tail must decode as absent, not as a zero-length frame" + ); + + // Frames of equal length must still split at the right place — the length prefix is the + // only thing that can tell them apart. + let a = [1u8; 64]; + let b = [2u8; 64]; + let d = encode_audio_red_datagram(9, 0, &a, &b); + let (_, _, primary, previous) = decode_audio_red_datagram(&d).unwrap(); + assert_eq!(primary, a); + assert_eq!(previous, Some(&b[..])); + } + + /// A truncated or hostile `0xD2` must be rejected, never panic — the split point comes off + /// the wire, so an over-long `primary_len` is the obvious attack on `split_at`. + #[test] + fn audio_red_datagram_rejects_bad_input() { + let d = encode_audio_red_datagram(1, 2, &[0xAAu8; 30], &[0xBBu8; 20]); + for n in 0..AUDIO_RED_HEADER { + assert!(decode_audio_red_datagram(&d[..n]).is_none(), "len {n}"); + } + // primary_len larger than the datagram: must be refused, not sliced. + let mut bad = d.clone(); + bad[13..15].copy_from_slice(&u16::MAX.to_le_bytes()); + assert!(decode_audio_red_datagram(&bad).is_none()); + // Wrong tag. + let mut wrong = d.clone(); + wrong[0] = AUDIO_MAGIC; + assert!(decode_audio_red_datagram(&wrong).is_none()); + } + + /// The two audio planes must not alias each other or any neighbouring plane: a client + /// demultiplexes purely on the first byte. + #[test] + fn audio_red_tag_is_disjoint() { + for other in [ + AUDIO_MAGIC, + RUMBLE_MAGIC, + MIC_MAGIC, + RICH_INPUT_MAGIC, + HIDOUT_MAGIC, + HDR_META_MAGIC, + HOST_TIMING_MAGIC, + CURSOR_STATE_MAGIC, + crate::input::INPUT_MAGIC, + ] { + assert_ne!(AUDIO_RED_MAGIC, other); + } + let red = encode_audio_red_datagram(1, 2, &[9u8; 40], &[8u8; 40]); + assert!( + decode_audio_datagram(&red).is_none(), + "0xC9 must not accept a 0xD2" + ); + let plain = encode_audio_datagram(1, 2, &[9u8; 40]); + assert!( + decode_audio_red_datagram(&plain).is_none(), + "0xD2 must not accept a 0xC9" + ); + } + #[test] fn rumble_datagram_roundtrip() { let d = encode_rumble_datagram(1, 0x1234, 0xFFFF); diff --git a/include/punktfunk_core.h b/include/punktfunk_core.h index 27706ff3..cc65876f 100644 --- a/include/punktfunk_core.h +++ b/include/punktfunk_core.h @@ -312,6 +312,13 @@ // `PunktfunkStatus` code). #define PUNKTFUNK_CLIP_ERROR 6 +// The protocol's audio frame, in milliseconds — every host datagram carries exactly one +// ([`crate::quic::encode_audio_datagram`]), so it is also the smallest useful shed unit. +#define PUNKTFUNK_AUDIO_FRAME_MS 5 + +// Sample rate of every audio plane in the protocol. +#define PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ 48000 + #if defined(PUNKTFUNK_FEATURE_QUIC) // The uniform no-TTL-host staleness bound: a legacy host refreshes state every 500 ms, so two // missed refreshes = quiet host → silence. Replaces the per-platform zoo (1.6 s / 60 s / 1.5 s / @@ -627,6 +634,19 @@ #define CLIENT_CAP_PHASE_LOCK 2 #endif +#if defined(PUNKTFUNK_FEATURE_QUIC) +// `Hello.client_caps` bit: this client can decode the redundant desktop-audio plane +// ([`AUDIO_RED_MAGIC`](super::datagram::AUDIO_RED_MAGIC), `0xD2`), where every datagram also +// carries a copy of the previous frame so a single lost packet is reconstructed instead of +// papered over with packet-loss concealment. +// +// Active only when the host answers with [`HOST_CAP_AUDIO_RED`] (capable-and-agreed, the +// cursor/clipboard precedent). Toward an older host, or a host that declines because the link is +// clean, the client keeps receiving the plain `0xC9` plane — so a client may always set this bit. +// `0x04` — `0x01`/`0x02` are cursor / phase-lock. +#define CLIENT_CAP_AUDIO_RED 4 +#endif + #if defined(PUNKTFUNK_FEATURE_QUIC) // [`Welcome::host_caps`] bit: the host CAN forward the cursor out-of-band (it captures cursor // metadata separately from the frame — the Linux portal `SPA_META_Cursor` path; NOT gamescope, @@ -652,6 +672,20 @@ #define HOST_CAP_PEN 16 #endif +#if defined(PUNKTFUNK_FEATURE_QUIC) +// [`Welcome::host_caps`] bit: the host is sending the REDUNDANT desktop-audio plane +// ([`AUDIO_RED_MAGIC`](super::datagram::AUDIO_RED_MAGIC), `0xD2`) instead of plain `0xC9` — each +// datagram carries its own frame plus a copy of the previous one. +// +// Set only when the client asked via [`CLIENT_CAP_AUDIO_RED`]. It is a statement about the WIRE, +// not a negotiation the client can decline: with the bit set the client must decode `0xD2`, and +// without it `0xC9`. The host may also drop back to `0xC9` mid-session (the redundancy is +// loss-gated — a clean LAN shouldn't pay for it), which is why clients decode BOTH tags +// unconditionally and treat this bit as "expect redundancy", not "only redundancy". +// `0x20` — `0x10` is [`HOST_CAP_PEN`], `0x08` is [`HOST_CAP_CURSOR`]. +#define HOST_CAP_AUDIO_RED 32 +#endif + #if defined(PUNKTFUNK_FEATURE_QUIC) // [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software** // encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST @@ -967,6 +1001,41 @@ #define HIDOUT_MAGIC 205 #endif +#if defined(PUNKTFUNK_FEATURE_QUIC) +// Redundant audio datagram, host → client: the [`AUDIO_MAGIC`] plane plus a copy of the PREVIOUS +// frame, so a single lost datagram is *reconstructed* rather than concealed. +// +// `[0xD2][u32 seq LE][u64 pts_ns LE][u16 primary_len LE][primary opus][previous opus]` +// +// **Why this and not Opus in-band FEC.** LBRR is a SILK-layer feature: the desktop-audio encoder +// runs `RESTRICTED_LOWDELAY` (CELT-only) at 5 ms frames, which is below SILK's 10 ms minimum, so +// `set_inband_fec(true)` on that encoder is a no-op. Nothing in libopus can protect this plane — +// the redundancy has to be at the application layer. (The mic uplink is a different encoder, VoIP +// mode at 10 ms, and *does* use real in-band FEC.) +// +// **Why it costs no latency.** The copy rides the SUCCESSOR of the frame it protects, and the +// client is already holding 15–90 ms of de-jitter buffer — far more than the 5 ms the successor +// takes to arrive. So the recovery happens inside slack that already exists. +// +// The previous frame's sequence is implicitly `seq - 1`; a host with nothing to duplicate yet +// (the first frame of a session, or straight after a capture reopen) simply sends an empty tail, +// which decodes to `None`. +// +// Sent ONLY when the client advertised [`CLIENT_CAP_AUDIO_RED`](super::caps::CLIENT_CAP_AUDIO_RED) +// and the host answered [`HOST_CAP_AUDIO_RED`](super::caps::HOST_CAP_AUDIO_RED) — the +// capable-and-agreed handshake the cursor and 4:4:4 planes already use. Every other session keeps +// the plain [`AUDIO_MAGIC`] wire byte-for-byte. +// +// NB `0xD1` is deliberately skipped: the DualSense pad-audio program has reserved it for the +// per-pad audio plane. +#define PUNKTFUNK_AUDIO_RED_MAGIC 210 +#endif + +#if defined(PUNKTFUNK_FEATURE_QUIC) +// Fixed header length of an [`AUDIO_RED_MAGIC`] datagram (tag + seq + pts + primary length). +#define PUNKTFUNK_AUDIO_RED_HEADER (((1 + 4) + 8) + 2) +#endif + #if defined(PUNKTFUNK_FEATURE_QUIC) // Wire length of a v1 (legacy, level) rumble datagram. #define RUMBLE_V1_LEN 7 @@ -1397,6 +1466,14 @@ typedef uint8_t PunktfunkInputKind; typedef struct ColorInfo ColorInfo; #endif +// Tuning for [`JitterPolicy`], in MILLISECONDS. +// +// Denominating the depth in time rather than in device quanta is the point. Every client used to +// compute its target as `3 × quantum`, which is a sane 15 ms at a 5 ms quantum and a silent 64 ms +// at a 20 ms one — the same source line meaning two very different latencies depending on what +// else happened to be using the audio graph that day. +typedef struct JitterTuning JitterTuning; + #if defined(PUNKTFUNK_FEATURE_QUIC) // Opaque handle to a live `punktfunk/1` connection (QUIC control plane + UDP data plane, all // pumped on internal threads). @@ -1788,6 +1865,14 @@ typedef struct { + + + + + + + + // The multipliers a picker offers. `1.0` (Native) is the default; the rest are the round stops // users reason about. Shared so every client's list stays identical. #define PRESETS { 0.5, 0.67, 0.75, 1.0, 1.25, 1.5, 2.0, 3.0, 4.0, }