fix(audio): the quality root cause, the latency ratchet, and making the plane observable #33

Merged

enricobuehler merged 7 commits from worktree-audio-quality-latency into main 2026-08-04 16:51:38 +00:00

Conversation 0 Commits 7 Files Changed 24

+2947 -179

7 Commits

Author	SHA1	Message	Date
enricobuehlerandClaude Opus 5	2dfb7791a2	fix(apple): the drift test tripped Swift's static exclusivity check ... CI caught what my local harness could not: reading `huge.count` inside the closure that already holds `huge` exclusively is an exclusivity violation, so PunktfunkKitTests failed to compile. The blind spot is worth recording. I verified `AudioRing` by compiling it against a standalone harness whose bodies were TOP-LEVEL code, where Swift applies DYNAMIC exclusivity — the same statement in a function body gets the static check and is a hard error. A harness that does not share the shape of the thing it stands in for can be green for a reason the real build does not have. The harness now puts every body in a method and compiles with `-enforce-exclusivity=checked`. Length now comes off the buffer pointer (`$0.count`), which is what the closure already owns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 18:14:24 +02:00
enricobuehlerandClaude Opus 5	c6597cbeb5	docs(troubleshooting): the audio quality knobs are a request, not a guarantee ... The page claimed "audio is a fraction of a percent of a stream's bandwidth, so high costs nothing worth counting". At 256 kbps plus redundancy that is 512 kbps — true of a 20 Mbps session, wrong by an order of magnitude on a 5 Mbps one, which is why the budget now exists. Says what actually happens on a narrow link, and points at the log line that reports the settled tier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 17:58:39 +02:00
enricobuehlerandClaude Opus 5	2cfc82e96c	fix(audio): budget the audio plane against the link, and close the review's gaps ... Findings from the post-implementation review of design/audio-quality-and-latency.md. **The bandwidth gap (highest).** Tier `High` (256 kbps) and the redundant `0xD2` plane were added separately, each costed as "~1 % of the video budget", and nobody added them together: 256 kbps sent twice is 512 kbps — ~2.5 % of a 20 Mbps session but ~10 % of a 5 Mbps one. Audio rides QUIC datagrams, OUTSIDE the ABR loop, so ABR could neither see that nor reclaim it; a constrained link quietly handed a tenth of its bandwidth to audio while ABR carefully managed the rest. `plan_audio_budget` now makes tier and redundancy ONE decision against the session's resolved video bitrate, ordered by preference rather than cost — transparent audio beats redundant audio, since the field report was about quality and redundancy only pays under loss, so `High` alone outranks `Standard`+redundancy even though they cost the same. It can lower what the operator asked for, never raise it, and never goes below `Low`: a stream with unintelligible audio is worse than one spending a few percent more. **The Linux host kept the exact defect fixed on Windows.** `let _ = tx.try_send(samples)` — silent, uncounted data loss, where the encoder concatenates across the hole, so every drop is a click AND a permanent shift of everything after it. WP0.2 turned out to be Windows-only and had not said so. Linux now shares `capture_policy::CaptureStats`: drops counted and warned, plus per-window peak/RMS/delivered%. A Linux audio report was until now exactly as un-triageable as the Windows one was on 2026-08-03. **Apple's WP0.3 was half-done** — `bufferedMS` was added and wired to nothing. The drain thread now logs buffer/target/underruns/sheds like the other three, from one locked snapshot so the numbers in a line describe the same instant. Also: the Linux "audio format negotiated" line now says WHICH mode produced it, because that changes what it is worth — in stream-sink mode the host owns the sink so the mix cannot have been narrowed upstream, but in legacy monitor mode a 16 kHz Bluetooth sink would still be reported as a clean 48 kHz through PipeWire's resampler, the same way WASAPI's autoconvert hid it on Windows. Reading the monitored node's own rate needs a registry lookup this stream does not do; recorded as an open gap rather than implied to be covered. Two stale docs: `audio_wasapi.rs` cited `clients/windows/src/audio.rs` (deleted) and still described the pre-shared-policy "prime to ~3 quanta" behaviour. And the Apple ring's `prefill:` parameter, dead since the depth moved into the ring, is gone. Verified: clippy --all-targets -D warnings on Linux (docker) AND Windows (runner .133, forced clean rebuild of punktfunk-host + pf-client-core); core 167 tests; host 57 audio tests on Windows; Android clippy count identical to pristine (6, all documented arm64 artifacts); Apple ring re-simulated. The host suite's `gamestream::stream::tests::sender_delivers_batches` fails under qemu — the recorded environmental flake, unrelated to audio, green on the earlier less-loaded run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 17:58:17 +02:00
enricobuehlerandClaude Opus 5	e9a209ef61	docs(troubleshooting): why streamed audio can sound worse than the host, and the knobs ... WP0.4. The 2026-08-03 reporter had no way to know their desktop mix was being routed through Steam's voice-carrier endpoint, and no documented way to change it — `PUNKTFUNK_HOST_AUDIO` existed only in a module doc comment. Two new sections: what the host actually captures (a render endpoint, not "the sound card"), what the new `engine_hz/engine_ch/engine_bits` log line tells you, and the `PUNKTFUNK_AUDIO_OUTPUT_MODE` / `_QUALITY` / `_REDUNDANCY` knobs — with host_and_client called out as the quickest A/B for the endpoint question; and why audio that lags the picture should now correct itself, plus what to check when it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 09:42:06 +02:00
enricobuehlerandClaude Opus 5	a12f1f092c	feat(clients/audio): one de-jitter policy for all four rings, and lossless single-packet recovery ... Phase 4 + WP3.2 of design/audio-quality-and-latency.md. **The defect.** Every client ring primed *up* to a target and clamped at a ceiling, and none walked the depth back *down*. Any transient — a Wi-Fi arrival burst, a host stall, or plain host-DAC-vs-client-DAC skew of a few dozen ppm — therefore added latency permanently, until an underrun happened to re-prime. Android, with no shed at all, converged on its 120 ms hard cap and stayed there for the rest of the session; that is the "audio latency is too high" report. Apple did shed, 40 ms in one go, which its own comment called "one audible blip". All four now share `punktfunk_core::audio::JitterPolicy`: depths in MILLISECONDS rather than device quanta (`3 x quantum` meant 15 ms at a 5 ms quantum and a silent 64 ms at a 20 ms one), a crossfaded 5 ms shed once the depth average has sat above target for 2 s of consumed audio, and de-prime hysteresis. Linux and Windows had never had that hysteresis — they still carried the `if ring.is_empty()` instant re-prime that Android identified as self-inflicted crackle, where one transient drain manufactured a whole target's worth of silence. Android's floor drops 40 -> 25 ms: the policy grows the target on the devices that actually underrun, instead of every device pre-paying for the worst one. The Windows ring moves from raw bytes to interleaved f32 so it can share the policy and the crossfade helper at all. Apple is the one client where the policy is hand-written in a second language, so it gets its own XCTest (`AudioRingDriftTests`). Verified here by compiling `AudioRing.swift` standalone against a simulation harness — +200 ppm for 5 minutes settles at 30 ms with zero silent callbacks, where the old ring would have ridden its 80 ms high-water mark. **WP3.2 — recovery lives in core, not in the clients.** The rebuilt frame is re-inserted into the demux queue in order, so every embedder (including any C-ABI consumer) gets a complete stream without knowing the `0xD2` plane exists, and their `AudioGapTracker` simply stops seeing the gap. `recovery_and_the_gap_tracker_agree` pins exactly that. For the same reason core advertises CLIENT_CAP_AUDIO_RED itself rather than making four embedders remember to. Verified: clippy --all-targets -D warnings and the full test suites for punktfunk-core, pf-client-core, punktfunk-host, pf-host-config under Linux/docker (163 + 61 tests); punktfunk-client-android `cargo ndk check` for aarch64 with the gate proven non-vacuous by a planted type error, and its 6 clippy findings confirmed IDENTICAL to the pristine file (all are the documented arm64-only artifacts); AudioRing.swift type-checked and simulated on macOS; fmt. The Windows client half (audio_wasapi.rs) is still not compile-verified anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 09:28:01 +02:00
enricobuehlerandClaude Opus 5	3055e29ebb	feat(host/audio): make audio observable, fix the endpoint choice, raise the encode quality ... Phases 0-3 of design/audio-quality-and-latency.md, host side. **WP2.1 — the 2026-08-03 root cause.** The client-only loopback preference took Steam's Streaming *Microphone* render endpoint over real hardware unconditionally, because it is silent on the host. But that endpoint exists to carry remote VOICE, and nothing checked whether it could carry music: on the reporter's box it won all 31 loopback opens across 25 sessions while a clean AMD HD Audio endpoint sat idle, and the whole desktop mix went through it before reaching Opus. A silent sink now has to EARN its preference — if its mix format narrows the mix it drops below real hardware. It is still taken when nothing better exists (narrow audio beats no audio), but flagged so the capture side says why. `plan_with_formats` takes a probe rather than reading WASAPI, so all 26 wiring-plan tests still run on every platform. An unknown format counts as fine, which is asserted: `unknown_formats_reproduce_the_formatless_plan` proves a probe failure can never make the plan worse than it was before formats existed. **WP0.1 — log the endpoint's ACTUAL mix format.** Everything the old log printed ("48 kHz f32 channels=2") was our REQUEST; with `autoconvert` WASAPI converts silently from whatever the endpoint really runs. That is why a 3,600-line log filed over an audio-quality complaint contained nothing that could diagnose it. **WP0.2 — count what we drop.** The capture->encode handoff was a silent lossy `try_send`: a stalled encode thread lost chunks, the encoder concatenated across the hole, and nothing recorded it — a click plus a permanent shift of everything after. Now counted and warned, alongside per-window peak/RMS/delivered% so a quiet host, a broken endpoint and a stream we are damaging ourselves stop looking identical. **WP2.4 — stop the default-device tug-of-war.** In Assert mode the capture is bound to the planned endpoint EXPLICITLY, so a hijacked default changes only where apps render — the old full reopen tore the capture down for nothing. The field log shows the cost: something re-set the default every ~4 s and each round was a teardown, a wiring pass with IPolicyConfig writes, and an audible dropout — seven in sixteen seconds, one ending in a 2 s error backoff. Now: put the default back, keep the stream, and after four rounds in twenty seconds concede for a minute and say so once. **WP1.1/1.2 — encode quality.** Constrained VBR (the hard-CBR comment justifies itself with GameStream's audio FEC, which this plane does not have) and `AudioTier::High` by default: stereo 128 -> 256 kbps, ~1 % of a 20 Mbps session. GameStream's encoder is deliberately untouched — its FEC really does need fixed-size packets. **WP3.1 — redundant `0xD2` plane**, sent when the client asked for it. **WP2.2 — `audio.output_mode`** as a first-class setting (`client_only` / `host_and_client` / `follow_default`), superseding the two undocumented env vars, which stay honoured. The enum lives in pf-host-config, which is deliberately dependency-free, so the tier table stays in core where the codec knowledge is. `capture_policy.rs` is split out for the same reason `wiring_plan.rs` is: both encode field behaviour, so their tests must run on Linux CI, not only on a Windows box. That split immediately earned itself — `capture_stats_separate_silence_from_signal` caught RMS being divided by the FRAME count while summed over interleaved SAMPLES, which inflated it by sqrt(channels) and made a sine report an RMS equal to its own peak. WP4.5 (open the loopback at the minimum device period) is deliberately NOT done: in shared mode `IAudioClient::Initialize` cannot change the engine period at all, so it would be a no-op at best and a new failure path at worst. Recorded in the code. WP2.3 (force the parked endpoint's volume) is deferred — `wasapi` keeps IMMDevice private, so it needs new raw COM on a path this tree cannot compile, let alone test; its diagnostic half ships as the RMS line above. Verified: punktfunk-host + pf-host-config clippy --all-targets -D warnings and the audio test suite under Linux/docker (gate proven non-vacuous with a planted type error); 26 wiring-plan tests standalone; fmt. The Windows-only halves of wasapi_cap.rs and audio_control.rs are NOT compile-verified anywhere yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 09:02:12 +02:00
enricobuehlerandClaude Opus 5	7077b0a0df	feat(core/audio): bitrate tiers, a shared de-jitter policy, and a redundant audio plane ... Foundation for the audio quality + latency plan (design/audio-quality-and-latency.md). All three pieces are pure and unit-tested here so the four client rings and the Windows host glue that follow stay thin. **Bitrate tiers** (`AudioTier`). The layout table's `bitrate` becomes the `Standard` value, so that tier reproduces the pre-tier wire byte-for-byte — the tier machinery is provably non-regressive. `High` (stereo 256 kbps) is the default: 5 ms Opus frames are much less efficient than 20 ms ones, so the historical 128 kbps buys roughly what ~100 kbps buys at 20 ms, while the same session carries tens of Mbps of video. Purely a host-side encoder knob — libopus reads the bitrate out of the packet, so no client change and no negotiation. **`JitterPolicy`** — the ms-denominated de-jitter state machine every client will share. Two defects it exists to fix: (1) each ring computed its target as `3 x quantum`, a sane 15 ms at a 5 ms quantum and a silent 64 ms at a 20 ms one; (2) every ring primed *up* and clamped at a ceiling, and none walked the depth back *down*, so drift/bursts added latency permanently — Android, with no shed at all, converged on its 120 ms cap. Here a depth EWMA that sits above target for 2 s of consumed audio sheds ONE 5 ms frame with a crossfade. Driven by samples consumed rather than the wall clock: allocation- and syscall-free (safe in a realtime callback) and deterministic under test. `every_preset_sheds_before_it_trims` pins the invariant that makes this real rather than decorative. The first draft had `headroom_ms` <= the shed threshold on all four presets, so the ring was trimmed back before the average could ever reach the shed point: drift correction was dead code and the ratchet test passed for the wrong reason (the hard cap did the work). `a_transient_burst_does_not_shed` caught it. The shed point is now derived from `headroom_ms` so it cannot invert again. **`0xD2` redundant audio** — each datagram carries its frame plus a copy of the previous one, so a single lost packet is reconstructed instead of concealed. Opus in-band FEC cannot do this job: LBRR is a SILK feature and the desktop encoder is CELT-only (RESTRICTED_LOWDELAY, 5 ms), so `set_inband_fec` there is a no-op. Costs no latency — the copy rides the successor, which arrives inside de-jitter slack that already exists. Gated capable-and-agreed via CLIENT_CAP_AUDIO_RED/HOST_CAP_AUDIO_RED; every other session keeps the `0xC9` wire unchanged. 0xD1 is left free for the pad-audio program. cbindgen: prefix the four new exported constants. `FRAME_MS`/`SAMPLE_RATE_HZ` as bare C macros are the same hazard the BTN_* renames already document — a clashing #define takes the last definition silently rather than failing to compile. Verified: 300 core tests, clippy -D warnings, fmt. (`c_abi` fails identically on a pristine tree — this Mac has no system libopus for the C harness link.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 08:11:35 +02:00