unom/punktfunk
10
0
Fork 4
Code Issues 15 Pull Requests 1 Actions 1 Packages Projects 1 Releases 60 Wiki Activity

Lossless audio: the 0xD3 PCM plane, 44.1–176.4 kHz, surround, verified on glass (ABI 24) #263

Merged
enricobuehler merged 27 commits from worktree-hires-audio-pcm into main 2026-08-16 11:18:37 +00:00
Conversation 0 Commits 27 Files Changed 74
+11932 -898
74 changed files with 11932 additions and 898 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/android.yml
+24
View File
@@ -222,6 +222,19 @@ jobs:
# screenshot scenes, which are a release-artifact job (android-screenshots.yml, gated to v*
# tags) and have no business adding a minute to every push. The filter is what lets the
# contract gate here without dragging the rest of the app suite in with it.
#
# ⚠ The filter is an ALLOWLIST, so a test class that is not named here does not run — it
# reads as coverage in the tree and gates nothing. `ProfilesTest` and `StatsOverlayAudioTest`
# sat outside it and were only noticed when the hi-res audio work added cases to both; the
# HUD ones had never run in CI at all. Adding a test class to `app/src/test` is therefore
# only half the job: add it here too, or it is decoration.
#
# That audit found NINE more in the same state (gamepad palette/rows/UI, OS icons, render
# scale, safe area, SC2 bluetooth grant, settings scope, speed test) — every one of them
# passing, so nothing was hiding, but none of them gating either. They are all listed now.
# The list is deliberately explicit rather than a package glob: the unfiltered task also
# drags in the ~20 Roborazzi screenshot scenes above, and a glob would quietly re-admit them
# the moment someone added one.
- name: console parity vectors + app-module logic tests
working-directory: clients/android
run: >-
@@ -231,6 +244,17 @@ jobs:
--tests 'io.unom.punktfunk.GamepadSettingsLayoutTest'
--tests 'io.unom.punktfunk.ConsoleSubScreenRowsTest'
--tests 'io.unom.punktfunk.ConsoleSubScreenRoutesTest'
--tests 'io.unom.punktfunk.ProfilesTest'
--tests 'io.unom.punktfunk.StatsOverlayAudioTest'
--tests 'io.unom.punktfunk.GamepadPaletteTest'
--tests 'io.unom.punktfunk.GamepadSettingsRowsTest'
--tests 'io.unom.punktfunk.GamepadUiTest'
--tests 'io.unom.punktfunk.OsIconsTest'
--tests 'io.unom.punktfunk.RenderScaleTest'
--tests 'io.unom.punktfunk.SafeAreaTest'
--tests 'io.unom.punktfunk.Sc2BluetoothGrantTest'
--tests 'io.unom.punktfunk.SettingsScopeTest'
--tests 'io.unom.punktfunk.SpeedTestTest'
--stacktrace
- name: assembleDebug (cargo-ndk → jniLibs → APK)
clients/android/app/src/main/kotlin/io/unom/punktfunk/GamepadSettingsScreen.kt
+13
View File
@@ -917,6 +917,19 @@ internal fun buildSettingsRows(
"The speaker layout requested from the host.",
AUDIO_CHANNEL_OPTIONS, s.audioChannels,
) { update(s.copy(audioChannels = it)) },
// Live at every channel count now. It used to dim on 5.1/7.1 because a lossless surround
// frame did not fit one datagram at the default MTU; the frame ladder is channel-aware, so
// a surround session negotiates a shorter frame instead — only the top of the list fits
// nothing, and that is the host's call to make rather than this row's.
choice(
"audioFormat", GpTab.AUDIO, null, "Audio quality",
"Lossless sends bit-exact PCM instead of compressed audio, on top of the video — " +
"2.3 Mbps at 48 kHz, 4.6 at 96, 8.5 at 176.4. It must be enabled on the host " +
"too, this device's output has to accept the rate, and the link has to fit the " +
"frames; otherwise the session falls back to Standard. The stats overlay says " +
"which.",
AUDIO_FORMAT_OPTIONS, s.audioFormat,
) { update(s.copy(audioFormat = it)) },
toggle(
"mic", GpTab.AUDIO, null, "Microphone",
"Send this device's microphone to the host's virtual mic.",
clients/android/app/src/main/kotlin/io/unom/punktfunk/HostConnect.kt
+15
View File
@@ -41,6 +41,16 @@ suspend fun connectToHost(
val hdrEnabled = settings.hdrEnabled && displaySupportsHdr(context)
// "Automatic" resolves to a concrete pad type from the connected controller's VID/PID.
val gamepadPref = Gamepad.resolvePref(settings.gamepad)
// The requested audio format as the two Hello fields — `0`/`0` when the user chose Standard,
// which is what keeps the lossless capability bit OFF (see `audioFormatWire`).
//
// Sent at every channel count, including surround. This used to be clamped to Opus on 5.1/7.1
// because a lossless surround frame did not fit one QUIC datagram, but the frame ladder is
// channel-aware: a 5.1 session simply negotiates a shorter frame (and pays for it in packet
// rate) and 96/24 5.1 fits nothing and is declined. That is the host's decision to make with the
// connection's real datagram size in hand, not one to pre-empt from here with an MTU this side
// never measured.
val (audioRateHz, audioBits) = settings.audioFormatWire()
return withContext(Dispatchers.IO) {
// Transport-level half of "Low-latency mode (experimental)" (DSCP marking on the media
// sockets) — must be applied before connect, since sockets are tagged at creation.
@@ -75,6 +85,11 @@ suspend fun connectToHost(
hdrEnabled, multiSlice,
frameParts,
settings.audioChannels,
// The audio format this session asks for. Only ever a request: the host's own gate
// may resolve it back to Opus, and the native side downgrades it first if AAudio on
// this device will not open the rate — a rate the wire has committed to cannot be
// rescued afterwards, so the fallback has to happen before the Hello.
audioRateHz, audioBits,
// What this device can decode (H.264|HEVC always, AV1 when a real decoder exists) +
// the soft codec preference (user choice, or the Automatic AV1 rule above) — the
// host resolves the emitted codec from both.
clients/android/app/src/main/kotlin/io/unom/punktfunk/Profiles.kt
+13 -1
View File
@@ -37,6 +37,12 @@ data class SettingsOverlay(
val hdrEnabled: Boolean? = null,
val compositor: Int? = null,
val audioChannels: Int? = null,
/**
* The requested audio format ([AUDIO_FORMAT_OPTIONS]'s stored value). Profileable because it
* is about how a HOST is streamed — a wired desktop can afford lossless, a phone on a hotspot
* cannot — rather than about this device's hardware.
*/
val audioFormat: String? = null,
val micEnabled: Boolean? = null,
val echoCancel: Boolean? = null,
val touchMode: TouchMode? = null,
@@ -73,6 +79,7 @@ data class SettingsOverlay(
hdrEnabled = hdrEnabled ?: base.hdrEnabled,
compositor = compositor ?: base.compositor,
audioChannels = audioChannels ?: base.audioChannels,
audioFormat = audioFormat ?: base.audioFormat,
micEnabled = micEnabled ?: base.micEnabled,
echoCancel = echoCancel ?: base.echoCancel,
touchMode = touchMode ?: base.touchMode,
@@ -110,6 +117,7 @@ data class SettingsOverlay(
hdrEnabled = if (after.hdrEnabled != before.hdrEnabled) after.hdrEnabled else hdrEnabled,
compositor = if (after.compositor != before.compositor) after.compositor else compositor,
audioChannels = if (after.audioChannels != before.audioChannels) after.audioChannels else audioChannels,
audioFormat = if (after.audioFormat != before.audioFormat) after.audioFormat else audioFormat,
micEnabled = if (after.micEnabled != before.micEnabled) after.micEnabled else micEnabled,
echoCancel = if (after.echoCancel != before.echoCancel) after.echoCancel else echoCancel,
touchMode = if (after.touchMode != before.touchMode) after.touchMode else touchMode,
@@ -141,6 +149,7 @@ data class SettingsOverlay(
"hdr_enabled" -> copy(hdrEnabled = null)
"compositor" -> copy(compositor = null)
"audio_channels" -> copy(audioChannels = null)
"audio_format" -> copy(audioFormat = null)
"mic_enabled" -> copy(micEnabled = null)
"echo_cancel" -> copy(echoCancel = null)
"touch_mode" -> copy(touchMode = null)
@@ -167,6 +176,7 @@ data class SettingsOverlay(
if (hdrEnabled != null) add("hdr_enabled")
if (compositor != null) add("compositor")
if (audioChannels != null) add("audio_channels")
if (audioFormat != null) add("audio_format")
if (micEnabled != null) add("mic_enabled")
if (echoCancel != null) add("echo_cancel")
if (touchMode != null) add("touch_mode")
@@ -201,6 +211,7 @@ data class SettingsOverlay(
hdrEnabled?.let { j.put("hdr_enabled", it) }
compositor?.let { j.put("compositor", it) }
audioChannels?.let { j.put("audio_channels", it) }
audioFormat?.let { j.put("audio_format", it) }
micEnabled?.let { j.put("mic_enabled", it) }
echoCancel?.let { j.put("echo_cancel", it) }
touchMode?.let { j.put("touch_mode", it.name) }
@@ -224,7 +235,7 @@ data class SettingsOverlay(
/** Keys this build models; everything else in a stored overlay is carried through. */
private val KNOWN = setOf(
"width", "height", "refresh_hz", "bitrate_kbps", "render_scale", "codec",
"hdr_enabled", "compositor", "audio_channels", "mic_enabled", "echo_cancel",
"hdr_enabled", "compositor", "audio_channels", "audio_format", "mic_enabled", "echo_cancel",
"touch_mode", "mouse_mode", "invert_scroll", "gamepad", "gamepad_forwarding",
"system_buttons", "guide_gesture",
"stats_verbosity",
@@ -241,6 +252,7 @@ data class SettingsOverlay(
hdrEnabled = j.optBooleanOrNull("hdr_enabled"),
compositor = j.optIntOrNull("compositor"),
audioChannels = j.optIntOrNull("audio_channels"),
audioFormat = j.optStringOrNull("audio_format"),
micEnabled = j.optBooleanOrNull("mic_enabled"),
echoCancel = j.optBooleanOrNull("echo_cancel"),
touchMode = j.optStringOrNull("touch_mode")
clients/android/app/src/main/kotlin/io/unom/punktfunk/Settings.kt
+128
View File
@@ -62,6 +62,19 @@ data class Settings(
/** Requested audio channel count: 2 (stereo), 6 (5.1) or 8 (7.1). The host clamps to what it
* can capture; the resolved count drives the decoder + AAudio layout. */
val audioChannels: Int = 2,
/**
* Requested audio format — the cross-client `audio_format` key: [AUDIO_FORMAT_OPUS] (the
* default, and byte-for-byte the session every build before the lossless plane ran) or one of
* the lossless rows in [AUDIO_FORMAT_OPTIONS], which span both rate families.
*
* Off by default and deliberately: lossless takes 2.1–8.5 Mbps off the top of the link,
* OUTSIDE the ABR loop that manages the video budget, against the ~256 kbps Opus it replaces —
* so it has to be asked for at both ends (`PUNKTFUNK_AUDIO_HIRES` is the host's half, also off
* by default). A REQUEST, never a fact: the host runs its gate and may answer Opus anyway, and
* the native side downgrades the rate first if THIS device will not open it. What actually
* happened is on the stats HUD, and in logcat's `audio: plane codec=… rate=…` line.
*/
val audioFormat: String = AUDIO_FORMAT_OPUS,
/** Preferred video codec: `"auto"` (host decides), `"hevc"`, `"h264"`, or `"av1"`. A soft
* preference — the host emits it when it can, else falls back. AMediaCodec decodes whichever
* the host resolves (AV1 is only advertised/offered when the device has a real AV1 decoder). */
@@ -295,6 +308,7 @@ class SettingsStore(context: Context) {
systemButtons = prefs.getString(K_SYSTEM_BUTTONS, "auto") ?: "auto",
guideGesture = prefs.getString(K_GUIDE_GESTURE, "auto") ?: "auto",
audioChannels = prefs.getInt(K_AUDIO_CH, 2),
audioFormat = prefs.getString(K_AUDIO_FORMAT, AUDIO_FORMAT_OPUS) ?: AUDIO_FORMAT_OPUS,
codec = prefs.getString(K_CODEC, "auto") ?: "auto",
micEnabled = prefs.getBoolean(K_MIC, false),
echoCancel = prefs.getBoolean(K_ECHO_CANCEL, true),
@@ -350,6 +364,7 @@ class SettingsStore(context: Context) {
.putString(K_SYSTEM_BUTTONS, s.systemButtons)
.putString(K_GUIDE_GESTURE, s.guideGesture)
.putInt(K_AUDIO_CH, s.audioChannels)
.putString(K_AUDIO_FORMAT, s.audioFormat)
.putString(K_CODEC, s.codec)
.putBoolean(K_MIC, s.micEnabled)
.putBoolean(K_ECHO_CANCEL, s.echoCancel)
@@ -387,6 +402,7 @@ class SettingsStore(context: Context) {
const val K_SYSTEM_BUTTONS = "system_buttons"
const val K_GUIDE_GESTURE = "guide_gesture"
const val K_AUDIO_CH = "audio_channels"
const val K_AUDIO_FORMAT = "audio_format"
const val K_CODEC = "codec"
const val K_MIC = "mic_enabled"
const val K_ECHO_CANCEL = "echo_cancel"
@@ -691,6 +707,118 @@ val AUDIO_CHANNEL_OPTIONS = listOf(
8 to "7.1 Surround",
)
/** Opus 48 kHz — the default, and byte-for-byte the session every earlier build ran. */
const val AUDIO_FORMAT_OPUS = "opus"
/**
* Bit-exact PCM at 44.1 kHz / 24-bit (~2.1 Mbps). The CD family's base rate: what an ordinary
* Windows endpoint or a 44.1 kHz interface reports as its own engine rate, and the request that
* spares such a host a resample it would otherwise do on the way out.
*/
const val AUDIO_FORMAT_LOSSLESS_441 = "lossless441"
/**
* Bit-exact PCM at 48 kHz / 24-bit (~2.3 Mbps). The honest win even without a hi-res interface:
* no lossy stage at all, and no double resample on a host whose engine already runs at 48 kHz.
*/
const val AUDIO_FORMAT_LOSSLESS_48 = "lossless48"
/** Bit-exact PCM at 88.2 kHz / 24-bit (~4.2 Mbps) — 96 kHz's counterpart in the 44.1 family. */
const val AUDIO_FORMAT_LOSSLESS_882 = "lossless882"
/**
* Bit-exact PCM at 96 kHz / 24-bit (~4.6 Mbps), and only real if the host's capture endpoint
* genuinely runs at 96 kHz — the host declines rather than upsampling to meet the request.
*/
const val AUDIO_FORMAT_LOSSLESS_96 = "lossless96"
/**
* Bit-exact PCM at 176.4 kHz / 24-bit — **8.5 Mbps**, and the one row far more likely to be
* declined than granted. Three separate things have to go right: the host's bandwidth gate gives
* audio at most a quarter of the video budget, so the session needs ~34 Mbps of video before it
* will even consider it; a stereo frame only fits a QUIC datagram on the ladder's shortest rung
* (1 ms — a thousand datagrams a second — at ~1 069 B, so the first connection with a smaller
* datagram declines it), and a surround one fits no rung at all; and very few Android outputs will
* open the rate, which the native probe settles before the handshake. Offered because it is
* reachable, not because it is likely — the HUD's `audio lossless …` line is what says which
* happened.
*/
const val AUDIO_FORMAT_LOSSLESS_1764 = "lossless1764"
/**
* (stored value, label) for the requested audio format — the cross-client table, matching the
* Apple client's `AudioFormatChoice` raw values and the desktop `AUDIO_FORMATS` so a profile
* written on any of them is honoured on the others.
*
* ⚠ **The stored values are shared VERBATIM and must never be renamed.** A profile carries the key
* through untouched, so a spelling that differs by one character fails in the worst possible way:
* the profile keeps "working" on the other client and silently inherits its global default
* instead. The naming rule is the kHz figure with the decimal point dropped — `lossless48`,
* `lossless96`, and for the 44.1 family `lossless441` / `lossless882` / `lossless1764`.
*
* **Both rate families are here now.** They were not: every buffer figure in the shared jitter
* policy used to be `ms × perMs` with `perMs` an INTEGER number of samples per millisecond, which
* made 44 100 → 44.1 truncate to 44 — a silent 2.3 % error in every target, every de-prime fuse
* and every reported buffer depth, and the whole reason the 44.1 family was deferred rather than
* refused (design/hi-res-audio.md §4.1). Core now multiplies before it divides, which is exact at
* every rate, so the deferral is lifted.
*
* A row being offered is not a promise it can be delivered: the host's gate, this device's own
* output, and the path MTU each get a veto, and the ones at the top of the list get vetoed often.
* What actually happened is on the HUD.
*
* Lossless at **16**-bit is deliberately absent at every rate: it spends ~1.4–1.5 Mbps to sound
* like the transparent 256 kbps Opus it replaces, and it is the one lossless request whose wire
* parameters are indistinguishable from a legacy one. 24-bit is where the plane earns its
* bandwidth.
*/
val AUDIO_FORMAT_OPTIONS = listOf(
AUDIO_FORMAT_OPUS to "Standard (Opus)",
AUDIO_FORMAT_LOSSLESS_441 to "Lossless 44.1 kHz / 24-bit",
AUDIO_FORMAT_LOSSLESS_48 to "Lossless 48 kHz / 24-bit",
AUDIO_FORMAT_LOSSLESS_882 to "Lossless 88.2 kHz / 24-bit",
AUDIO_FORMAT_LOSSLESS_96 to "Lossless 96 kHz / 24-bit",
AUDIO_FORMAT_LOSSLESS_1764 to "Lossless 176.4 kHz / 24-bit",
)
/**
* The `(rateHz, bits)` pair [audioFormat] asks the host for, in `nativeConnect`'s terms.
*
* ⚠⚠ **Opus is `0`/`0`, the "did not ask" sentinel — NOT `48000`/`16`.** Core sets
* `CLIENT_CAP_AUDIO_HIRES` when either field is non-zero, because it keys on *a format was
* specified* rather than *the format differs from the default*: 48 kHz/16-bit is the cheapest
* lossless rung as well as the legacy pair, so the other rule would make it the one rung nobody
* could ask for. Sending `48000`/`16` for a user who chose Standard therefore advertises the
* capability, and any host with `PUNKTFUNK_AUDIO_HIRES=1` hands that user 1.5 Mbps of lossless PCM
* instead of 256 kbps of Opus. This returned that pair until all four clients were compared.
*
* The zeroes are also what keeps a default `Hello` byte-identical to a pre-lossless one — the wire
* encodes an explicit 48 000/16 the same as absent, and the whole difference is the capability bit.
*
* Deriving the pair FROM the stored format is what stops the two ever disagreeing. An unrecognized
* stored value — a newer build's, or a corrupted pref — resolves to Opus rather than blocking the
* connect.
*
* The rate this returns is only the REQUEST. The native side runs it down a fallback ladder first
* (`session::connect::rate_fallback_ladder`), because AAudio grants an explicitly-asked rate or
* fails the open and never substitutes — so a rate this device cannot play must never reach the
* wire.
*/
fun Settings.audioFormatWire(): Pair<Int, Int> = when (audioFormat) {
AUDIO_FORMAT_LOSSLESS_441 -> 44_100 to 24
AUDIO_FORMAT_LOSSLESS_48 -> 48_000 to 24
AUDIO_FORMAT_LOSSLESS_882 -> 88_200 to 24
AUDIO_FORMAT_LOSSLESS_96 -> 96_000 to 24
AUDIO_FORMAT_LOSSLESS_1764 -> 176_400 to 24
else -> AUDIO_FORMAT_WIRE_UNSPECIFIED
}
/**
* The `(rateHz, bits)` that mean "this session is not asking for the lossless plane" — see
* [audioFormatWire] for why it is a pair of zeroes rather than the legacy 48 000/16.
*/
val AUDIO_FORMAT_WIRE_UNSPECIFIED = 0 to 0
/**
* (stored value, label) for the preferred video codec — the cross-client table (the Rust
* `CODECS`), so a value another client or a profile stored is always representable here.
clients/android/app/src/main/kotlin/io/unom/punktfunk/SettingsScreen.kt
+17
View File
@@ -824,6 +824,23 @@ private fun AudioSettings(s: Settings, update: (Settings) -> Unit, onMicChange:
field = "audio_channels",
caption = "Requested from the host; it downmixes if it has fewer.",
) { ch -> update(s.copy(audioChannels = ch)) }
// Offered at every channel count. It used to be hidden on 5.1/7.1, because a lossless
// surround frame did not fit one QUIC datagram at the default MTU — but the frame ladder is
// channel-aware, so a surround session negotiates a shorter frame instead of being refused,
// and only the top of this list genuinely fits nothing. Which rows a given session can
// actually have depends on the host, this device's output and the path MTU, none of which
// this screen knows; the HUD's `audio lossless …` line is what reports the answer.
SettingDropdown(
label = "Audio format",
options = AUDIO_FORMAT_OPTIONS,
selected = s.audioFormat,
field = "audio_format",
caption = "Lossless sends uncompressed audio on top of the video — 2.3 Mbps at " +
"48 kHz, 4.6 at 96, 8.5 at 176.4 — and the top rates are often declined, " +
"surround especially. The host has its own switch and both must be on; " +
"otherwise the session stays on Opus, which is already effectively " +
"transparent. The overlay shows what a session actually got.",
) { f -> update(s.copy(audioFormat = f)) }
ToggleRow(
title = "Microphone",
subtitle = "Feeds this device's microphone to the host",
clients/android/app/src/main/kotlin/io/unom/punktfunk/StatsOverlay.kt
+59 -2
View File
@@ -18,13 +18,13 @@ import kotlin.math.roundToInt
* The live stats overlay — the unified HUD (`design/stats-unification.md`): headline is
* `capture→displayed` tiled by `host+network` + `decode` + `display` when the platform delivered
* OnFrameRendered render callbacks this window (`dispValid`), falling back to the v1
* `capture→decoded` headline without the `display` term when it didn't. Reads the 35-double
* `capture→decoded` headline without the `display` term when it didn't. Reads the 38-double
* layout from [NativeBridge.nativeVideoStats] (that KDoc is the authoritative index list):
* `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skew, w, h, hz, lostTotal, bitDepth, colorPrimaries,
* colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms, netP50Ms, lost, skipped,
* fec, frames, dispValid, displayP50Ms, e2eDispP50Ms, e2eDispP95Ms, paceP50Ms, latchP50Ms,
* presentsWindow, presenterActive, feedP50Ms, codecP50Ms, skippedOverflowWindow, audioBufferMs,
* audioAvOffsetMs]`. Every read
* audioAvOffsetMs, audioCodec, audioRateHz, audioBits]`. Every read
* is length-guarded, so an older native lib simply omits the lines it can't feed.
*
* The shown `display` and `end-to-end` numbers EXCLUDE the OS present floor (see [osFloorMs]) at
@@ -46,6 +46,10 @@ import kotlin.math.roundToInt
* - [StatsVerbosity.DETAILED] — also the decoder label, the video-feed descriptor (10–13), the
* stage equation (14/15, split into `host + network` when the Phase-2 terms at 16/17 are nonzero),
* the excluded-floor line when one was measured, and the audio plane's own latency (33/34).
*
* The RESOLVED audio format (35–37) is the one figure that is not reserved for
* [StatsVerbosity.DETAILED] — it renders from [StatsVerbosity.NORMAL] up, and only on a lossless
* session. See [audioFormatLine]. (Not on COMPACT, which is one line by definition.)
* [StatsVerbosity.OFF] renders nothing. Older native layouts simply omit the lines they lack (the
* counter line falls back to the cumulative `lostTotal` at index 9 on a pre-window lib).
*/
@@ -182,6 +186,15 @@ internal fun StatsOverlay(
if (detailed) {
audioLine(s)?.let { statLine(it, Color.White) }
}
// NOT gated to the detailed tier, unlike the audio latency above it, and deliberately: it
// is the one thing a user who turned lossless on needs to see. The failure it guards
// against (design/hi-res-audio.md §4.3, §10) is a session that costs 2.1–8.5 Mbps and
// delivers ordinary Opus, which is indistinguishable from success without a surface naming
// what the HOST resolved. `null` on the Opus plane every ordinary session runs, so the
// common case gains no line at all — and the top of the format menu is declined often
// enough (176.4 kHz fits only the ladder's shortest 1 ms rung, and hi-res surround fits no
// rung at all) that "the setting says one thing" is not evidence of anything.
audioFormatLine(s)?.let { statLine(it, Color(0xFFB0FFD0)) }
counterLine(s, lost)?.let { statLine(it, Color(0xFFFFB0B0)) }
}
}
@@ -215,6 +228,50 @@ private fun audioLine(s: DoubleArray): String? {
return "audio buffer $bufferMs ms$avTerm"
}
/**
* The RESOLVED audio format from 35–37 — `audio lossless 96 kHz / 24-bit` — or `null` on the Opus
* plane and on an older native layout.
*
* Deliberately silent for Opus rather than printing `audio opus 48 kHz`: that is what every
* session has always been, so a line stating it would be noise on the HUD of every user who never
* touched the setting. The line exists for the opposite case, and it is the only surface that can
* answer it: the format the SETTINGS screen shows is what this device REQUESTED, and the host's
* gate can decline every one of them (its own switch is off by default) — leaving a session that
* looks, sounds and measures exactly like a granted one. The native side can also have downgraded
* the request before the handshake, if this device's output would not open the rate. Both land
* here as the truth.
*
* `codec` is the wire byte: 0 = Opus on `0xC9`, 2 = lossless PCM on `0xD3` (1 is reserved for a
* FLAC that was measured and not taken).
*
* The rate is rendered in kHz to one decimal when it needs one, because half the ladder does: the
* 44.1 kHz family (44 100 / 88 200 / 176 400) does not divide by a thousand, and printing raw Hz
* for it — as this did while the ladder was 48/96 only — put the settings menu's "44.1 kHz" next
* to a HUD saying "44100 Hz" and left the reader to decide whether those were the same session.
* The whole point of this line is that it is comparable at a glance with what was asked for.
*
* Built by integer division rather than `"%.1f".format(…)` deliberately: that formatter renders
* through the default locale and would say "44,1 kHz" on a device set to most of Europe — a
* decimal comma where the settings row it is meant to be compared against has a point. Every rate
* this plane carries is a whole number of hundreds of Hz, so the tenths digit is exact.
*/
private fun audioFormatLine(s: DoubleArray): String? {
if (s.size < 38) return null
if (s[35].roundToInt() != AUDIO_CODEC_PCM_WIRE) return null
val rateHz = s[36].roundToInt()
val bits = s[37].roundToInt()
if (rateHz <= 0 || bits <= 0) return null
val khz = if (rateHz % 1000 == 0) {
"${rateHz / 1000} kHz"
} else {
"${rateHz / 1000}.${rateHz % 1000 / 100} kHz"
}
return "audio lossless $khz / $bits-bit"
}
/** `quic::AUDIO_CODEC_PCM` — the `0xD3` lossless plane's wire byte. */
private const val AUDIO_CODEC_PCM_WIRE = 2
/** One monospace HUD line — the shared type ramp so every tier's rows line up. */
@Composable
private fun statLine(text: String, color: Color) {
clients/android/app/src/test/kotlin/io/unom/punktfunk/ProfilesTest.kt
+120
View File
@@ -50,6 +50,7 @@ class ProfilesTest {
hdrEnabled = false,
compositor = 4,
audioChannels = 6,
audioFormat = AUDIO_FORMAT_LOSSLESS_96,
micEnabled = true,
touchMode = TouchMode.POINTER,
mouseMode = MouseMode.CAPTURE,
@@ -67,6 +68,7 @@ class ProfilesTest {
assertFalse(out.hdrEnabled)
assertEquals(4, out.compositor)
assertEquals(6, out.audioChannels)
assertEquals(AUDIO_FORMAT_LOSSLESS_96, out.audioFormat)
assertTrue(out.micEnabled)
assertEquals(TouchMode.POINTER, out.touchMode)
assertEquals(MouseMode.CAPTURE, out.mouseMode)
@@ -236,6 +238,124 @@ class ProfilesTest {
assertEquals(base, made.first().overrides.apply(base))
}
/**
* The audio-format setting is a STRING, and the two numbers it turns into are what the `Hello`
* carries — get the mapping wrong and the session either spends 8.5 Mbps it was not asked for
* or silently declines to ask for what it was. The Opus row is the load-bearing one: it must
* be the `0`/`0` "did not ask" sentinel, because core sets `CLIENT_CAP_AUDIO_HIRES` on ANY
* non-zero field — see [theOpusSettingDoesNotAdvertiseTheLosslessCapability].
*/
@Test
fun theAudioFormatSettingMapsToTheWireFieldsItClaims() {
assertEquals(
AUDIO_FORMAT_WIRE_UNSPECIFIED,
base.copy(audioFormat = AUDIO_FORMAT_OPUS).audioFormatWire(),
)
// Both rate families. The 44.1 one was deferred only for as long as the shared jitter
// policy divided by 1 000 before it multiplied (44 100 → 44 samples/ms, every buffer
// figure 2.3 % out); core multiplies first now, so these are simply rates.
assertEquals(
44_100 to 24,
base.copy(audioFormat = AUDIO_FORMAT_LOSSLESS_441).audioFormatWire(),
)
assertEquals(
48_000 to 24,
base.copy(audioFormat = AUDIO_FORMAT_LOSSLESS_48).audioFormatWire(),
)
assertEquals(
88_200 to 24,
base.copy(audioFormat = AUDIO_FORMAT_LOSSLESS_882).audioFormatWire(),
)
assertEquals(
96_000 to 24,
base.copy(audioFormat = AUDIO_FORMAT_LOSSLESS_96).audioFormatWire(),
)
assertEquals(
176_400 to 24,
base.copy(audioFormat = AUDIO_FORMAT_LOSSLESS_1764).audioFormatWire(),
)
// The default is the legacy request — a fresh install asks for exactly what it always did.
assertEquals(AUDIO_FORMAT_WIRE_UNSPECIFIED, Settings().audioFormatWire())
// A newer build's value (or a corrupted pref) falls back to Opus rather than reaching the
// host as an unrepresentable rate: a settings string must never be able to block a connect.
assertEquals(
AUDIO_FORMAT_WIRE_UNSPECIFIED,
base.copy(audioFormat = "lossless192").audioFormatWire(),
)
}
/**
* ⚠⚠ **A user who chose Standard (Opus) must not advertise `CLIENT_CAP_AUDIO_HIRES`**, and the
* only thing standing between them and 1.5 Mbps of PCM they did not ask for is that this pair
* is `0`/`0`.
*
* Core's `advertised_client_caps` sets the bit when EITHER field is non-zero — it keys on "a
* format was specified", not on "the format differs from the default", because 48 kHz/16-bit is
* both the legacy pair AND the cheapest lossless rung and the other rule would make that rung
* unrequestable. The host's gate then accepts 48 kHz/16-bit as a perfectly supported format. So
* a client that sends the legacy-looking numbers as its stand-in for "default" opts every one of
* its users in, on every host running `PUNKTFUNK_AUDIO_HIRES=1`, with no surface anywhere saying
* so — a declined session and a silently granted one look identical from the settings screen.
*
* This client did exactly that until the four clients were compared. The rule is restated here
* rather than reached through core because Kotlin cannot call it; core's own tests pin the other
* half.
*/
@Test
fun theOpusSettingDoesNotAdvertiseTheLosslessCapability() {
// Core's rule, verbatim: `audio_rate_hz != 0 || audio_bits != 0`.
fun asksForHiRes(wire: Pair<Int, Int>) = wire.first != 0 || wire.second != 0
assertFalse(asksForHiRes(base.copy(audioFormat = AUDIO_FORMAT_OPUS).audioFormatWire()))
assertFalse(asksForHiRes(Settings().audioFormatWire()))
assertFalse(asksForHiRes(base.copy(audioFormat = "lossless192").audioFormatWire()))
// …and every row that IS a lossless choice must ask, or the setting does nothing at all.
// That asymmetry is the whole contract.
for ((value, _) in AUDIO_FORMAT_OPTIONS.drop(1)) {
assertTrue(value, asksForHiRes(base.copy(audioFormat = value).audioFormatWire()))
}
}
/**
* The stored values are a CROSS-CLIENT contract, shared verbatim with the Apple client's
* `AudioFormatChoice` raw values and the desktop `AUDIO_FORMATS`. A profile carries the key
* through untouched, so a rename here does not break a round trip loudly — it breaks it
* silently, by leaving the other client to fall back to its own global default on a profile
* that looks like it applied. Spelled out as literals rather than referenced through the
* constants, because a test that reads the constant cannot detect the constant changing.
*
* The naming rule for anything added later is the kHz figure with the decimal point dropped.
*/
@Test
fun theStoredAudioFormatValuesAreTheOnesEveryOtherClientStores() {
assertEquals("opus", AUDIO_FORMAT_OPUS)
assertEquals("lossless441", AUDIO_FORMAT_LOSSLESS_441)
assertEquals("lossless48", AUDIO_FORMAT_LOSSLESS_48)
assertEquals("lossless882", AUDIO_FORMAT_LOSSLESS_882)
assertEquals("lossless96", AUDIO_FORMAT_LOSSLESS_96)
assertEquals("lossless1764", AUDIO_FORMAT_LOSSLESS_1764)
// Opus first (the default), then the lossless rows by ascending rate.
assertEquals(
listOf(
AUDIO_FORMAT_OPUS,
AUDIO_FORMAT_LOSSLESS_441,
AUDIO_FORMAT_LOSSLESS_48,
AUDIO_FORMAT_LOSSLESS_882,
AUDIO_FORMAT_LOSSLESS_96,
AUDIO_FORMAT_LOSSLESS_1764,
),
AUDIO_FORMAT_OPTIONS.map { it.first },
)
// Every offered row resolves to a DISTINCT request — a duplicate would be a menu entry the
// wire cannot tell from its neighbour — every lossless one is 24-bit, and exactly one row
// (Opus, the first) is the "did not ask" sentinel.
val wire = AUDIO_FORMAT_OPTIONS.map { base.copy(audioFormat = it.first).audioFormatWire() }
assertEquals(wire.size, wire.toSet().size)
assertTrue(wire.drop(1).all { it.second == 24 })
assertEquals(1, wire.count { it == AUDIO_FORMAT_WIRE_UNSPECIFIED })
assertEquals(AUDIO_FORMAT_WIRE_UNSPECIFIED, wire.first())
}
@Test
fun mintedIdsAreWellFormed() {
val id = newProfileId()
clients/android/app/src/test/kotlin/io/unom/punktfunk/StatsOverlayAudioTest.kt
+78 -3
View File
@@ -30,10 +30,19 @@ class StatsOverlayAudioTest {
val compose = createAndroidComposeRule<ComponentActivity>()
/**
* A plausible 35-double window with the audio gauges dialled in. Everything before 33 is the
* DETAILED-renderable shape the ShotScenes fixture uses; only the last two matter here.
* A plausible 38-double window with the audio gauges dialled in. Everything before 33 is the
* DETAILED-renderable shape the ShotScenes fixture uses; only the tail matters here. The
* format triple (35–37) defaults to an ordinary Opus session, so a test that says nothing
* about it is asserting against the shape every session has always had.
*/
private fun stats(bufferMs: Double, avOffsetMs: Double, size: Int = 35): DoubleArray {
private fun stats(
bufferMs: Double,
avOffsetMs: Double,
size: Int = 38,
codec: Double = 0.0,
rateHz: Double = 48_000.0,
bits: Double = 16.0,
): DoubleArray {
val full = doubleArrayOf(
238.0, 921.4, 1.3, 2.1, 1.0, 1.0, 5120.0, 1440.0, 240.0, 2.0,
10.0, 9.0, 16.0, 1.0, 0.9, 0.4, 0.6, 0.3,
@@ -42,6 +51,7 @@ class StatsOverlayAudioTest {
0.2, 0.3, 236.0, 1.0,
0.1, 0.3, 0.0,
bufferMs, avOffsetMs,
codec, rateHz, bits,
)
return full.copyOf(size)
}
@@ -91,4 +101,69 @@ class StatsOverlayAudioTest {
show(stats(bufferMs = 42.0, avOffsetMs = 18.0, size = 33))
compose.onNodeWithText("audio buffer", substring = true).assertDoesNotExist()
}
/**
* The RESOLVED audio format (35–37) — the surface `design/hi-res-audio.md` §10 requires, and
* the only one that can answer "did lossless actually happen". The settings screen shows what
* this device REQUESTED; the host's gate (its own switch off by default) can decline every
* one of them and the session then looks, sounds and measures exactly like a granted one.
* Codec `2` is the `0xD3` lossless plane.
*/
@Test
fun aLosslessSessionNamesTheFormatItResolved() {
show(stats(bufferMs = 42.0, avOffsetMs = 0.0, codec = 2.0, rateHz = 96_000.0, bits = 24.0))
compose.onNodeWithText("audio lossless 96 kHz / 24-bit").assertExists()
}
/**
* The 44.1 kHz family renders as kHz with a tenth, not as raw Hz.
*
* The rest of the HUD can print whatever reads well; this line cannot, because its entire job
* is to be compared at a glance with the settings row that asked for the format. While the
* ladder was 48/96 only every rate divided by a thousand and the fallback arm was unreachable;
* admitting 44 100 / 88 200 / 176 400 made it the arm half the menu now takes, and "44100 Hz"
* next to a menu saying "44.1 kHz" is one more thing for a reader to have to work out.
*/
@Test
fun aFractionalRateRendersInKilohertzRatherThanRawHertz() {
show(stats(bufferMs = 42.0, avOffsetMs = 0.0, codec = 2.0, rateHz = 44_100.0, bits = 24.0))
compose.onNodeWithText("audio lossless 44.1 kHz / 24-bit").assertExists()
}
/** …and the top of the ladder, which is the row most likely to have been declined outright. */
@Test
fun theTopOfTheLadderNamesItselfExactly() {
show(stats(bufferMs = 42.0, avOffsetMs = 0.0, codec = 2.0, rateHz = 176_400.0, bits = 24.0))
compose.onNodeWithText("audio lossless 176.4 kHz / 24-bit").assertExists()
}
/**
* Shown from NORMAL, unlike every other audio figure: a user who paid 4.6 Mbps for this should
* not have to find the DETAILED tier to learn whether they got it.
*/
@Test
fun theFormatLineIsNotReservedForTheDetailedTier() {
show(
stats(bufferMs = 42.0, avOffsetMs = 0.0, codec = 2.0, rateHz = 48_000.0, bits = 24.0),
verbosity = StatsVerbosity.NORMAL,
)
compose.onNodeWithText("audio lossless 48 kHz / 24-bit").assertExists()
}
/**
* Silent for Opus — which is every session anyone who never touched the setting will ever run,
* so a line stating it would be noise on almost every HUD. Absence IS the ordinary case.
*/
@Test
fun anOpusSessionSaysNothingAboutTheFormat() {
show(stats(bufferMs = 42.0, avOffsetMs = 0.0))
compose.onNodeWithText("audio lossless", substring = true).assertDoesNotExist()
}
/** An older native lib emits 35 doubles; the format line must be omitted, never mis-indexed. */
@Test
fun aPreFormatNativeLayoutOmitsTheFormatLine() {
show(stats(bufferMs = 42.0, avOffsetMs = 0.0, size = 35, codec = 2.0))
compose.onNodeWithText("audio lossless", substring = true).assertDoesNotExist()
}
}
clients/android/app/src/test/kotlin/io/unom/punktfunk/screenshots/ShotScenes.kt
+7 -2
View File
@@ -450,12 +450,12 @@ internal fun StreamScene(verbosity: StatsVerbosity = StatsVerbosity.DETAILED) {
Brush.linearGradient(listOf(Color(0xFF2A1E5C), Color(0xFF0E1B3D), Color(0xFF06122B))),
),
) {
// The full 35-double unified layout — NativeBridge.nativeVideoStats' KDoc is the
// The full 38-double unified layout — NativeBridge.nativeVideoStats' KDoc is the
// authoritative index list: [fps, mbps, e2eP50, e2eP95, latValid, skew, w, h, hz,
// lostTotal, bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc, hostNetP50,
// decodeP50, hostP50, netP50, lost, skipped, fec, frames, dispValid, displayP50,
// e2eDispP50, e2eDispP95, paceP50, latchP50, presents, presenterActive, feedP50, codecP50,
// skippedOverflow, audioBufferMs, audioAvOffsetMs].
// skippedOverflow, audioBufferMs, audioAvOffsetMs, audioCodec, audioRateHz, audioBits].
// 10/9/16/1 = a 10-bit BT.2020 PQ (HDR) 4:2:0 feed so the DETAILED HUD renders its
// video-feed line; the display stage is valid (dispValid 1) so the headline is the
// directly-measured capture→displayed pair, less the excluded OS present floor (the 0.3
@@ -479,6 +479,11 @@ internal fun StreamScene(verbosity: StatsVerbosity = StatsVerbosity.DETAILED) {
// The audio plane: a 28 ms ring placed 4 ms behind the picture — a converged sync
// loop, i.e. inside the deadband it deliberately leaves alone.
28.0, 4.0,
// The resolved audio format: codec 0 = Opus at 48 kHz/16-bit, which is what an
// ordinary session runs and what these shots are of. The HUD's format line only
// renders for the lossless plane (codec 2), so this triple deliberately adds
// nothing to the capture — the scene shows the shape almost every user sees.
0.0, 48_000.0, 16.0,
),
verbosity = verbosity,
decoderLabel = "c2.qti.hevc.decoder · low-latency",
clients/android/kit/src/main/kotlin/io/unom/punktfunk/kit/NativeBridge.kt
+29 -3
View File
@@ -56,6 +56,26 @@ object NativeBridge {
* decode loop then feeds slices with `BUFFER_FLAG_PARTIAL_FRAME` as they arrive). */
framePartsOk: Boolean,
audioChannels: Int,
/** Requested audio sample rate: **`0` (with [audioBits] `0`) for the legacy Opus plane**, or
* any rung of the lossless ladder — `44100`, `48000`, `88200`, `96000`, `176400`, both rate
* families.
*
* ⚠⚠ **`48000`/`16` is NOT "the default" — it is the cheapest lossless rung.** Core sets
* `CLIENT_CAP_AUDIO_HIRES` when either field is non-zero (it keys on "a format was
* specified", so that 48/16 lossless is requestable at all), and the host's gate accepts
* 48 kHz/16-bit as a supported format. Passing it as a stand-in for "unset" opts every
* session into the `0xD3` plane on any host with `PUNKTFUNK_AUDIO_HIRES=1`. Send `0`/`0`.
*
* A request on BOTH counts. The host runs its gate (its own `PUNKTFUNK_AUDIO_HIRES` switch
* among them, plus whether a frame of this format fits one datagram at all) and may answer
* Opus; and the native side first proves THIS device can open the rate — AAudio grants an
* explicit rate or fails the open, and there is no recovery once the wire is negotiated —
* walking a fallback ladder and downgrading the request if it cannot. */
audioRateHz: Int,
/** Requested audio sample depth: `0` alongside a `0` [audioRateHz] for the legacy Opus
* plane, else `16` or `24`. See [audioRateHz] for why `16` is a request rather than a
* default; 24-bit is where lossless earns its bandwidth. */
audioBits: Int,
/** `quic::CODEC_*` bitfield of codecs this device decodes ([VideoDecoders.decodableCodecBits]);
* `0` falls back to H.264|HEVC. The host resolves the emitted codec from this ∩ its GPU. */
videoCodecs: Int,
@@ -275,12 +295,13 @@ object NativeBridge {
/**
* Drain ~1 s of live decode stats for the on-stream HUD, or `null` when no decode thread runs.
* Returns 35 doubles (unified stats spec, `design/stats-unification.md`):
* Returns 38 doubles (unified stats spec, `design/stats-unification.md`):
* `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skewCorrected, width, height, refreshHz, framesLost,
* bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms,
* netP50Ms, lostWindow, skippedWindow, fecWindow, framesWindow, dispValid, displayP50Ms,
* e2eDispP50Ms, e2eDispP95Ms, paceP50Ms, latchP50Ms, presentsWindow, presenterActive,
* feedP50Ms, codecP50Ms, skippedOverflowWindow, audioBufferMs, audioAvOffsetMs]`
* feedP50Ms, codecP50Ms, skippedOverflowWindow, audioBufferMs, audioAvOffsetMs, audioCodec,
* audioRateHz, audioBits]`
* (the flags are 1.0/0.0; indexes 2/3 are the end-to-end capture→decoded headline; 10–13
* describe the negotiated video feed — bit depth 8/10, CICP primaries/transfer, and the HEVC
* chroma_format_idc 1=4:2:0 / 3=4:4:4; 14/15 are the stage p50s tiling the headline —
@@ -299,7 +320,12 @@ object NativeBridge {
* `skipped` (19), i.e. the decoder falling behind rather than benign newest-wins pacing;
* 33/34 are the AUDIO plane — the playback ring's live depth in ms and the A/V sync loop's
* smoothed offset in ms, positive meaning audio plays BEHIND the picture. Those two are live
* gauges, not windowed samples, and the offset reads 0 until the loop has a video reference).
* gauges, not windowed samples, and the offset reads 0 until the loop has a video reference;
* 35–37 are the audio FORMAT the host RESOLVED at the handshake — `audioCodec` 0 = Opus on
* `0xC9`, 2 = lossless PCM on `0xD3` — plus the resolved rate in Hz and depth in bits. Static
* for the session, and separate from 33/34 because they answer a different question: not "how
* late is the audio" but "is this the format the user asked for", which nothing else can tell
* apart — a declined lossless session looks exactly like a granted one from the outside).
* Poll ~1 Hz; each call resets the measurement window.
*/
external fun nativeVideoStats(handle: Long): DoubleArray?
clients/android/native/src/audio.rs
+569 -189
View File
File diff suppressed because it is too large Load Diff
clients/android/native/src/audio_format.rs
+432
View File
@@ -0,0 +1,432 @@
//! The audio format a session RESOLVED, and the millisecond ⇄ interleaved-sample arithmetic every
//! figure the playback plane reports is expressed in.
//!
//! **Why this is its own module, and why it is NOT `#[cfg(target_os = "android")]` like the
//! [`crate::audio`] that owns it.** The conversions below are the part of the plane that was
//! *wrong* — see [`ms_to_samples`] — and the whole class of defect is one that measures cleanly
//! while being off by a fixed percentage. A bug like that is only ever caught by arithmetic tests,
//! and an arithmetic test that can only run on a phone is a test that runs when someone remembers.
//! Nothing here touches AAudio, so nothing here needs a device: it compiles and is tested on the
//! ordinary `cargo test -p punktfunk-client-android --lib` leg, and `:kit:cargoNdkClippy` lints it
//! at both Android widths on top.
use punktfunk_core::audio::pcm;
// Only [`SessionAudio::of`] touches the connector, and only on device — see the `cfg` there.
#[cfg(target_os = "android")]
use punktfunk_core::client::NativeClient;
/// The `0xC9` plane's frame duration: fixed by the protocol at 5 ms (the host's `audio_thread`),
/// not negotiated. Only `0xD3` carries `audio_frame_us`.
pub(crate) const OPUS_FRAME_US: u32 = 5_000;
// ---- ms ⇄ interleaved samples: multiply FIRST, divide LAST ------------------------------------
//
// This mirrors `punktfunk_core::audio`'s own pair, and it mirrors it because the defect it fixes
// was copied from there. Both used to precompute `per_ms = rate_hz / 1000 * channels` and express
// every ms-denominated figure as `ms * per_ms`. **That division happens first**, so 44 100 Hz
// became 44 samples per millisecond and every depth, hard cap and reported `buffer_ms` was 2.3 %
// out — quietly, permanently, and only on the rates the old ladder happened not to offer. 48 000
// and 96 000 were exact by luck: they divide.
//
// Keeping the rate and the channel count as the two numbers they are, and dividing last, is exact
// at every rate on `pcm::rate_is_supported`'s ladder for one integer division per conversion, and
// 48/96 kHz stay bit-identical by construction (`per_sec == 1000 × per_ms` exactly there, so both
// conversions reduce to the expression they replace).
/// Interleaved samples per second at a negotiated layout — the denominator both conversions share.
///
/// `max(1)` on both factors: a degenerate layout must not divide by zero on a realtime thread.
/// [`SessionAudio::of`] already clamps, so this is the belt to that pair of braces.
fn interleaved_per_sec(rate_hz: u32, channels: usize) -> u64 {
let hz = if rate_hz == 0 { 1 } else { rate_hz } as u64;
let ch = if channels == 0 { 1 } else { channels } as u64;
hz * ch
}
/// `ms` milliseconds of audio, in interleaved samples.
///
/// u64 intermediates because the product is large where a `usize` may be 32 bits — and on this
/// client that is not hypothetical: **armeabi-v7a is a shipping ABI** (every 32-bit Google TV /
/// Android TV box), so the same expression runs at both widths. `JitterTuning::AAUDIO`'s hard cap
/// against 176 400 Hz × 8 ch would be fine, but the type is what makes that a fact rather than an
/// audit. Saturating rather than wrapping, because a wrapped window is a *tiny* one — a buffer cap
/// that is instantly exceeded instead of one that is never reached.
fn ms_to_samples(rate_hz: u32, channels: usize, ms: u32) -> usize {
let n = ms as u64 * interleaved_per_sec(rate_hz, channels) / 1000;
if n > u32::MAX as u64 {
u32::MAX as usize
} else {
n as usize
}
}
/// Interleaved samples back to whole milliseconds — the exact inverse of [`ms_to_samples`].
///
/// u128 because `samples` arrives from a ring depth and nothing bounds it: `usize::MAX * 1000`
/// overflows a u64 on the 64-bit ABI.
fn samples_to_ms(rate_hz: u32, channels: usize, samples: usize) -> u32 {
let ms = samples as u128 * 1000 / interleaved_per_sec(rate_hz, channels) as u128;
if ms > u32::MAX as u128 {
u32::MAX
} else {
ms as u32
}
}
/// The audio format this session RESOLVED, read once from the connector and threaded through the
/// whole plane.
///
/// Gathered into one value rather than passed as five parameters because the fields are only
/// meaningful together: a rate without the codec cannot tell a 48 kHz lossless session from a
/// 48 kHz Opus one, and those two agree on every other resolved value.
///
/// ⚠ Everything here is what the HOST resolved, never what this client asked for. A client that
/// requests 96 kHz, is answered 48 kHz and opens at 96 kHz anyway is `design/hi-res-audio.md`
/// §4.3's failure one end further along — a session that audits clean at both ends and plays the
/// wrong content.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) struct SessionAudio {
/// [`punktfunk_core::quic::AUDIO_CODEC_OPUS`] (`0xC9`) or
/// [`punktfunk_core::quic::AUDIO_CODEC_PCM`] (`0xD3`) — what SELECTS the decoder, and the only
/// field that can.
pub(crate) codec: u8,
/// The resolved sample rate: 48 000 on every Opus session, and any rung of
/// [`pcm::rate_is_supported`] on `0xD3` — 44 100 / 48 000 / 88 200 / 96 000 / 176 400. Both
/// families are exact in every conversion this module performs; the 44.1 one was deferred
/// only for as long as the arithmetic above divided before it multiplied.
pub(crate) rate_hz: u32,
/// The resolved sample depth (16 or 24) — the stride `0xD3` payloads are unpacked at.
/// Meaningless on the Opus plane, which decodes to f32 regardless.
pub(crate) bits: u8,
/// The resolved, normalized channel count (2 / 6 / 8).
///
/// ⚠ Not "2" any more on the lossless plane. Surround was excluded from `0xD3` because a 5.1
/// frame did not fit a datagram at the default MTU, but the ladder is channel-aware and the
/// restriction was one host-side condition, not a wire limitation: at the conservative
/// datagram size a 48 kHz/16-bit 5.1 session negotiates 2 ms frames and a 24-bit one 1 ms
/// (a thousand datagrams a second), while 96/24 5.1 still fits nothing and is declined. Every
/// per-frame size below is taken from THIS count for that reason.
pub(crate) channels: usize,
/// How much audio one datagram carries. Negotiated from the path MTU on `0xD3` (at 96 kHz /
/// 24-bit stereo the default MTU ceiling only leaves room for 2 ms, and for 24-bit surround
/// 1 ms), and the Opus plane's fixed 5 ms otherwise — folded to one field here so nothing
/// downstream has to branch to size a buffer.
///
/// ⚠ A **label**, not a duration. It is a whole number of samples per channel only when the
/// rate divides the rung, which the 44.1 kHz family never does: a nominal 5 ms frame at
/// 44 100 Hz carries 220 samples per channel = 4 988 662 ns. Size from [`Self::frame_samples`];
/// time from [`pcm::frame_duration_ns`].
pub(crate) frame_us: u32,
}
impl SessionAudio {
/// Read the whole resolved format off the connector, once, at the top of the plane.
///
/// Android-only, because a [`NativeClient`] only exists once a session has been negotiated on
/// a device — the pure half is [`resolved`](Self::resolved), which is where the clamping lives
/// and what the tests exercise. Left ungated it would be dead code on the host build, and
/// `-D warnings` is a hard gate there.
#[cfg(target_os = "android")]
pub(crate) fn of(client: &NativeClient) -> SessionAudio {
SessionAudio::resolved(
client.audio_codec,
client.audio_sample_rate_hz,
client.audio_bits,
client.audio_channels,
u32::from(client.audio_frame_us),
)
}
/// The `Welcome`'s five audio fields, clamped into something every buffer below can be sized
/// from. **Every value here arrives off the wire**, so each clamp is defending a realtime
/// thread against a host that is old, wrong, or hostile — none of them bite a conforming one.
pub(crate) fn resolved(
codec: u8,
rate_hz: u32,
bits: u8,
channels: u8,
frame_us: u32,
) -> SessionAudio {
let is_pcm = codec == punktfunk_core::quic::AUDIO_CODEC_PCM;
// A zero rate is inexpressible off the wire (`Welcome::decode` folds both absence and a
// literal 0 to the legacy 48 kHz), but it is the denominator of every conversion above,
// so a 0 that ever DID reach here would be a division by zero on the decode thread. One
// clamp, at the one place the value enters this module.
let rate_hz = if rate_hz == 0 {
punktfunk_core::audio::SAMPLE_RATE_HZ
} else {
rate_hz
};
SessionAudio {
codec,
rate_hz,
bits,
channels: punktfunk_core::audio::normalize_channels(channels) as usize,
// Same reasoning as the rate: an old host sends no `audio_frame_us` at all and a
// hostile one could send 0, and this number divides nothing but sizes everything.
//
// Capped at the longest rung of `FRAME_US_LADDER` (which is also the Opus plane's
// 5 ms) because the decode scratch is sized from that rung and clamps its copies to
// it: an unclamped `frame_us` would let a `Welcome` claim frames the scratch cannot
// hold, and the ring would then be reserved for a size the loop can never deliver.
// A conforming host only ever names a rung, so this bites nobody real.
frame_us: match (is_pcm, frame_us) {
(true, us) if us > 0 => us.min(pcm::FRAME_US_LADDER[0]),
// The Opus plane's frames are the protocol's fixed 5 ms (host `audio_thread`).
_ => OPUS_FRAME_US,
},
}
}
/// True when this session runs the lossless `0xD3` plane rather than Opus on `0xC9`.
pub(crate) fn is_pcm(&self) -> bool {
self.codec == punktfunk_core::quic::AUDIO_CODEC_PCM
}
/// `ms` of audio in interleaved samples at this session's layout — see [`ms_to_samples`].
pub(crate) fn ms_samples(&self, ms: u32) -> usize {
ms_to_samples(self.rate_hz, self.channels, ms)
}
/// The inverse, for the depths this plane reports to the HUD — see [`samples_to_ms`].
pub(crate) fn samples_ms(&self, samples: usize) -> u32 {
samples_to_ms(self.rate_hz, self.channels, samples)
}
/// Interleaved samples in ONE frame of this plane — what the ring reserves per queued chunk
/// and what the decode-scratch assertion is written against.
///
/// Taken from [`pcm::samples_per_frame`] rather than re-derived here, because that function is
/// the single source of truth for how long a frame is and the host fills its buffers from it.
/// The two are only interchangeable when the rate divides the rung: **5 ms of audio at
/// 44 100 Hz stereo is 441 interleaved samples, but a 5 ms FRAME carries 440** — 220.5 samples
/// per channel do not exist, so the wire floors. Both the ring reserve and the debug assertion
/// that guards it mean "exactly one packet", and a self-derived answer would describe a packet
/// no host ever sends.
pub(crate) fn frame_samples(&self) -> usize {
pcm::samples_per_frame(self.rate_hz, self.frame_us, self.channels as u8)
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Every rate the lossless plane admits, so a conversion that is only exact on one family can
/// never be pinned by accident.
const RATES: [u32; 5] = [44_100, 48_000, 88_200, 96_000, 176_400];
fn fmt(rate_hz: u32, channels: usize, frame_us: u32) -> SessionAudio {
SessionAudio {
codec: punktfunk_core::quic::AUDIO_CODEC_PCM,
rate_hz,
bits: pcm::BITS_24,
channels,
frame_us,
}
}
/// **The defect, stated as numbers.** `per_ms = rate_hz / 1000 * channels` truncates 44 100 Hz
/// stereo to 88 samples per millisecond where it is really 88.2, so everything the plane sizes
/// or reports in milliseconds came out 2.3 % wrong: the ring's hard cap 2.3 % SHALLOW, and the
/// `buffer_ms` on the HUD 2.3 % DEEP — a plane that measures itself cleanly while being off in
/// both directions at once.
///
/// Paired with [`the_48_khz_family_is_bit_identical_to_the_arithmetic_it_replaced`], which is
/// the other half of the claim and passes under BOTH expressions. Restore
/// `ms × (rate_hz / 1000 × channels)` and exactly one of the two fails; that asymmetry is the
/// whole point, and it is why they are two tests rather than one.
#[test]
fn the_ms_conversions_are_exact_on_the_rates_that_do_not_divide() {
// 44 100 × 2 = 88 200 interleaved samples a second; one second of them is 88 200, not the
// 88 000 an integer samples-per-millisecond would have claimed.
let f = fmt(44_100, 2, 5_000);
assert_eq!(f.ms_samples(1_000), 88_200);
assert_eq!(f.samples_ms(88_200), 1_000);
// …and the truncated pair, spelled out, so the size of the error is on the record rather
// than in a commit message: 88 000 samples and 1 002 ms are what the old code produced.
assert_ne!(f.ms_samples(1_000), 1_000 * 88);
assert_ne!(f.samples_ms(88_200), 88_200 / 88);
// 5.1 at 88 200 Hz — the two axes that used to be folded into one constant, both moving.
let s = fmt(88_200, 6, 2_000);
assert_eq!(s.ms_samples(100), 52_920);
assert_eq!(s.samples_ms(52_920), 100);
// And the top of the ladder, where the truncation is smallest in relative terms and still
// wrong: 176 400 Hz × 8 ch is 1 411 200 samples a second, not 1 408 000.
let top = fmt(176_400, 8, 1_000);
assert_eq!(top.ms_samples(1_000), 1_411_200);
}
/// The other half of the same claim: on 48 000 and 96 000 Hz the new conversions are
/// **bit-identical** to the `ms × (rate_hz / 1000 × channels)` they replaced, at every layout
/// and every figure the tuning names.
///
/// Load-bearing, not decorative. Every session anyone has ever run is on this family, and the
/// value of "we fixed the arithmetic" depends entirely on nobody's ring having moved by a
/// sample while we did it. It holds by construction — `rate × ch` is exactly `1000 × per_ms`
/// where the rate divides — and this is that construction asserted rather than argued.
///
/// It also passes under the OLD expression, which is what makes its partner above a real test:
/// plant `per_ms` back and this one still goes green.
#[test]
fn the_48_khz_family_is_bit_identical_to_the_arithmetic_it_replaced() {
for rate in [48_000u32, 96_000] {
for ch in [2usize, 6, 8] {
let f = fmt(rate, ch, 5_000);
let per_ms = (rate as usize / 1000) * ch;
for ms in [1u32, 2, 12, 25, 47, 120, 1_000] {
assert_eq!(
f.ms_samples(ms),
ms as usize * per_ms,
"{rate} Hz/{ch}ch must be unchanged at {ms} ms"
);
assert_eq!(
f.samples_ms(ms as usize * per_ms),
ms,
"{rate} Hz/{ch}ch must read back unchanged at {ms} ms"
);
}
// A depth that is NOT a whole number of milliseconds truncates the same way it
// always did — the reported `buffer_ms` never rounds up into a figure the ring
// does not hold.
assert_eq!(f.samples_ms(per_ms * 12 + per_ms / 2), 12);
}
}
}
/// The round trip `design/hi-res-audio.md` §4.1 names as the tell that this rework is
/// incomplete: a depth expressed in samples and read back as milliseconds must be the
/// milliseconds it was built from, at every rate on the ladder and every layout the plane can
/// resolve. Core asserts the same property for [`punktfunk_core::audio::JitterPolicy`]; this
/// is the half of it this client owns, since the ring's cap and the HUD's `buffer_ms` are
/// converted here rather than there.
#[test]
fn the_shipping_ladder_round_trips_ms_to_samples_at_every_rate() {
let t = punktfunk_core::audio::JitterTuning::AAUDIO;
for rate in RATES {
for ch in [2usize, 6, 8] {
let f = fmt(rate, ch, 2_000);
// Every ms figure this preset names — each is a threshold something compares a
// sample count against, and a rate that skewed 2.3 % skewed all of them together,
// which is exactly what kept the defect invisible.
for ms in [
t.base_target_ms,
t.max_target_ms,
t.headroom_ms,
t.hard_cap_ms,
t.deprime_ms,
t.plc_max_ms(),
] {
assert_eq!(
f.samples_ms(f.ms_samples(ms)),
ms,
"{rate} Hz/{ch}ch lost {ms} ms on the round trip"
);
}
// The conversion itself, against the arithmetic done the honest way rather than
// against itself: multiply by the rate and the channels, and only THEN divide.
for ms in [1u32, 2, 12, 47, 1_000, 480_000] {
assert_eq!(
f.ms_samples(ms) as u64,
ms as u64 * rate as u64 * ch as u64 / 1000,
"{ms} ms at {rate} Hz/{ch}ch"
);
}
}
}
// ⚠ Exact is not the same as lossless in both directions, and the difference is worth
// stating rather than discovering. A millisecond is 88.2 samples at 44 100 Hz stereo, so
// an ms figure that is not a multiple of 5 has no whole-sample answer at all: 1 ms floors
// to 88 samples, which reads back as 0. That is a floor of at most ONE SAMPLE — against
// the 2.3 % the old arithmetic was out by on EVERY figure, in the same direction,
// permanently. Every threshold `JitterTuning::AAUDIO` names is a multiple of 5, which is
// why the loop above is exact and this note is a note.
let f = fmt(44_100, 2, 5_000);
assert_eq!(f.ms_samples(1), 88); // the true 88.2, floored
assert_eq!(f.samples_ms(88), 0);
assert_eq!(f.ms_samples(15), 1_323, "15 ms of 44.1 kHz stereo");
assert_eq!(15 * (44_100 / 1000) * 2, 1_320, "what it used to compute");
}
/// **A frame is not the milliseconds it is labelled with.** At 44 100 Hz a nominal 5 ms frame
/// carries 220 samples per channel — 440 interleaved — while 5 ms of *audio* is 441, because
/// 220.5 samples per channel do not exist and the wire floors.
///
/// The ring reserve, the decode-scratch assertion and the policy's shed all mean "exactly one
/// packet", so this must come from [`pcm::samples_per_frame`] — the same function the host
/// fills its buffers from — and never from a millisecond count. One sample of disagreement on
/// an interleaved stream walks the channels around each other.
#[test]
fn a_frame_is_the_wires_sample_count_not_the_labels_milliseconds() {
let f = fmt(44_100, 2, 5_000);
assert_eq!(f.frame_samples(), 440, "220 samples per channel, floored");
assert_eq!(f.ms_samples(5), 441, "5 ms of AUDIO is 441 interleaved");
assert_ne!(f.frame_samples(), f.ms_samples(5));
// The real duration of that frame, which is what a `pts_ns` must advance by — 0.23 % short
// of the label it negotiated.
assert_eq!(pcm::frame_duration_ns(440, 44_100, 2), 4_988_662);
// Where the rate divides the rung the two agree, which is why nothing noticed for as long
// as the ladder was 48/96 kHz only.
for rate in [48_000u32, 96_000] {
for ch in [2usize, 6, 8] {
let f = fmt(rate, ch, 5_000);
assert_eq!(f.frame_samples(), f.ms_samples(5), "{rate} Hz/{ch}ch");
}
}
// Surround sizes from the RESOLVED channel count, not from a stereo assumption: a 5.1
// frame is three times a stereo one and the ring is reserved from it.
let stereo = fmt(48_000, 2, 2_000);
let five_one = fmt(48_000, 6, 2_000);
assert_eq!(stereo.frame_samples(), 192);
assert_eq!(five_one.frame_samples(), 576);
}
/// A `Welcome` this client cannot trust must not become a division fault or a buffer sized
/// from garbage on the decode thread. Absence, a literal zero and an over-long frame all have
/// defined answers, and they are the safe ones.
#[test]
fn a_degenerate_welcome_clamps_instead_of_dividing_by_zero() {
const OPUS: u8 = punktfunk_core::quic::AUDIO_CODEC_OPUS;
const PCM: u8 = punktfunk_core::quic::AUDIO_CODEC_PCM;
// The ordinary session: a pre-lossless host sends none of these fields, and every absent
// one has to land on exactly what the plane has always been.
let legacy = SessionAudio::resolved(OPUS, 0, 0, 0, 0);
assert!(!legacy.is_pcm());
assert_eq!(legacy.rate_hz, punktfunk_core::audio::SAMPLE_RATE_HZ);
assert_eq!(legacy.channels, 2);
assert_eq!(legacy.frame_us, OPUS_FRAME_US);
// `audio_frame_us` is a `0xD3` field and must not be honoured on the Opus plane, whose
// frames the protocol fixes at 5 ms — a host that sent one anyway would otherwise resize
// this client's ring for frames it never sends.
assert_eq!(
SessionAudio::resolved(OPUS, 48_000, 16, 2, 2_000).frame_us,
OPUS_FRAME_US
);
// …and on `0xD3` a frame longer than the ladder's top rung is capped there, because the
// decode scratch is sized from that rung and clamps its copies to it.
let overlong = SessionAudio::resolved(PCM, 96_000, 24, 6, 60_000);
assert!(overlong.is_pcm());
assert_eq!(overlong.frame_us, pcm::FRAME_US_LADDER[0]);
assert_eq!(overlong.channels, 6);
// A layout off the wire is normalized rather than trusted: 3 channels is not a layout the
// decoder or AAudio can be opened with.
assert_eq!(
SessionAudio::resolved(PCM, 44_100, 24, 3, 5_000).channels,
2
);
// The conversions still have to survive a 0 that reached them some other way, because they
// run on a realtime-adjacent thread that may not panic.
let broken = fmt(0, 0, 0);
assert_eq!(broken.ms_samples(10), 0);
assert_eq!(broken.samples_ms(480), 480_000);
assert_eq!(broken.frame_samples(), 0);
}
}
clients/android/native/src/lib.rs
+7
View File
@@ -30,6 +30,13 @@ use jni::EnvUnowned;
mod adpf;
#[cfg(target_os = "android")]
mod audio;
// The RESOLVED audio format + its ms ⇄ sample arithmetic, split out of `audio` and — unlike it —
// ungated, because that arithmetic is what a rate the ladder does not divide gets wrong (44 100 Hz
// used to come out 2.3 % off in every direction at once) and it must be provable without a phone.
// Nothing in it touches AAudio. `test`-gated for the host build on top of the Android one so the
// off-device leg still compiles and runs the proof; `audio` is its only non-test user.
#[cfg(any(target_os = "android", test))]
mod audio_format;
#[cfg(target_os = "android")]
mod decode;
// Ungated: pure `mdns-sd` + `jni`, so the browse + its JNI seam link into the host workspace build
clients/android/native/src/session/connect.rs
+316 -8
View File
@@ -102,9 +102,127 @@ fn force_parts_sysprop() -> bool {
false
}
/// The rates this session may ask for when the one the user chose will not open, best first.
///
/// **Down the requested rate's own FAMILY, then the 48 kHz floor.** The two families
/// ([`punktfunk_core::audio::pcm::rate_is_supported`]) are 44.1 / 88.2 / 176.4 kHz and 48 / 96 kHz,
/// and within a family the lower rates are the same material at half the samples — a 176.4 kHz
/// interface that will not open is overwhelmingly likely to be an 88.2 or 44.1 kHz one, and asking
/// there next is asking for the rate the endpoint most plausibly runs at.
///
/// **48 kHz terminates every ladder, including the 44.1 family's**, and that crossing is deliberate
/// rather than an oversight. It is the rate every Android output grants, it is the rate this
/// protocol has always run, and the alternative to a 48 kHz *lossless* session is a 48 kHz *Opus*
/// one — the same rate with a lossy stage added. Nothing is resampled by this decision: the host
/// captures at the rate it answers with, or declines (§8.2/§8.3), so a 44.1 kHz-locked endpoint
/// answers a 48 kHz request with Opus rather than with a quiet upsample.
///
/// The requested rate is the first rung, so an openable rate is asked for unchanged and a default
/// session's ladder is one rung long.
fn rate_fallback_ladder(rate_hz: u32) -> &'static [u32] {
const HZ48: u32 = punktfunk_core::audio::SAMPLE_RATE_HZ;
match rate_hz {
176_400 => &[176_400, 88_200, 44_100, HZ48],
88_200 => &[88_200, 44_100, HZ48],
44_100 => &[44_100, HZ48],
96_000 => &[96_000, HZ48],
// 48 kHz itself, and — via the `rate_is_supported` guard in the caller — nothing else.
_ => &[HZ48],
}
}
/// Resolve the audio format this `Hello` should ASK for, from what Kotlin's setting requested —
/// after proving this device can actually open it.
///
/// This is `design/hi-res-audio.md` §7's rule made mechanical: *"a client that cannot open a
/// 96 kHz output must not set `CLIENT_CAP_AUDIO_HIRES`"*. It has to happen here, before the
/// handshake, because after it there is no recovery: AAudio grants an explicitly-requested rate or
/// fails the open (it never substitutes), the host does not renegotiate the plane mid-session
/// (§6), and the only ways to play a wire of one rate through a stream of another are the wrong
/// speed or a resampler nobody asked for — which §9 forbids in as many words ("say so and fall
/// back, not resample quietly"). So the fall back happens where falling back is still free: in the
/// request.
///
/// **Every rung above the floor is probed, and admitting the 44.1 kHz family made that matter
/// more, not less.** When the ladder was 96 → 48 there was one uncertain rate; now there are four,
/// and their odds are nothing alike — 44 100 Hz is granted by very nearly every Android output,
/// 176 400 Hz by very nearly none, and 88 200 Hz by whatever the HAL happens to think. None of
/// that is inferable from the number, so [`crate::audio::output_rate_is_openable`] opens a stream
/// and reads back what it was granted, once per rung, until one holds.
///
/// Dropping the RATE keeps the depth, so a device that refuses the rate still gets a 24-bit
/// lossless session rather than being pushed all the way back to Opus — the depth is where the
/// plane earns its bandwidth anyway (and it is the half that is audible at all: §12).
///
/// The 48 kHz floor is never probed. It is universally supported, and the DEPTH never reaches
/// AAudio at all (the device is opened as f32 on both planes — see `crate::audio`), so there is
/// nothing about 16-vs-24-bit for a probe to discover. An ordinary session therefore opens no
/// stream here and pays nothing.
///
/// # ⚠⚠ "Not asking" is `(0, 0)`, and it is NOT `(48 000, 16)`
///
/// Core's `advertised_client_caps` sets `CLIENT_CAP_AUDIO_HIRES` when **either field is non-zero**
/// — it keys on *the caller specified a format*, not on *the format differs from the default*, and
/// deliberately: 48 kHz/16-bit is the cheapest lossless rung as well as the legacy pair, so a
/// "differs from the default" rule would make it the one rung on the ladder nobody could ask for.
///
/// So returning the legacy-looking `(48 000, 16)` for a user who chose **Standard (Opus)** does not
/// mean "no request" — it advertises the capability, the host's gate accepts 48 kHz/16-bit as a
/// perfectly supported format, and any host running `PUNKTFUNK_AUDIO_HIRES=1` silently gives that
/// user the lossless `0xD3` plane at 1.5 Mbps in place of 256 kbps of Opus. This returned exactly
/// that pair until it was caught by comparing all four clients; the desktop client and every
/// pre-v24 `punktfunk_connect_ex*` send `(0, 0)`, and so does this now.
///
/// `(0, 0)` is also what keeps the `Hello` byte-identical to a legacy one, because the wire encodes
/// an explicit 48 000/16 the same as absent — the difference lives entirely in the capability bit.
fn resolve_requested_audio_format(rate_hz: u32, bits: u8, channels: u8) -> (u32, u8) {
const HZ48: u32 = punktfunk_core::audio::SAMPLE_RATE_HZ;
/// "This session did not ask for the lossless plane" — see the ⚠⚠ section above for why this
/// is a pair of zeroes and not the legacy 48 000/16.
const UNSPECIFIED: (u32, u8) = (0, 0);
// A format core would not carry — including Kotlin's `0`/`0` for the Opus setting — asks for
// nothing, rather than being an error: the request is a preference, and an unrecognized one
// must not block a connect. The rate set comes from core rather than being re-expressed here,
// so the host's gate and every client's request validation cannot drift apart.
if !punktfunk_core::audio::pcm::depth_is_supported(bits)
|| !punktfunk_core::audio::pcm::rate_is_supported(rate_hz)
{
return UNSPECIFIED;
}
let granted = rate_fallback_ladder(rate_hz)
.iter()
.copied()
// `HZ48` short-circuits the probe rather than being trusted after one: it is the ladder's
// floor, so a probe there could only turn a working session into no lossless session at
// all — and it is the rate a failed probe would have fallen back TO.
.find(|&hz| hz == HZ48 || audio_rate_is_openable(hz, channels))
// Unreachable while every ladder ends at `HZ48`; the belt is here so a future rung added
// above the floor cannot silently produce an unrequestable format.
.unwrap_or(HZ48);
if granted != rate_hz {
log::warn!(
"audio: this device will not open a {rate_hz} Hz output, so the session asks for {granted} Hz / {bits}-bit instead — the wire is only ever offered a format this client has proved it can play"
);
}
(granted, bits)
}
#[cfg(target_os = "android")]
fn audio_rate_is_openable(rate_hz: u32, channels: u8) -> bool {
crate::audio::output_rate_is_openable(rate_hz, channels)
}
/// Off-device (the host `cargo build --workspace` leg, where there is no AAudio at all): nothing
/// can be proved, so nothing is claimed. The caller falls back to the legacy rate, which is the
/// safe answer for a build that never runs on a phone anyway.
#[cfg(not(target_os = "android"))]
fn audio_rate_is_openable(_rate_hz: u32, _channels: u8) -> bool {
false
}
/// `NativeBridge.nativeConnect(host, port, w, h, hz, certPem, keyPem, pinHex, bitrateKbps,
/// compositorPref, gamepadPref, hdrEnabled, audioChannels, preferredCodec, timeoutMs, launch,
/// deviceName): Long`.
/// compositorPref, gamepadPref, hdrEnabled, audioChannels, audioRateHz, audioBits, preferredCodec,
/// timeoutMs, launch, deviceName): Long`.
/// `launch` (empty ⇒ none) is a store-qualified library id to boot straight into a game.
/// `deviceName` (empty ⇒ none) rides the Hello as `name` — what the host's pending-approval list
/// and trust store show for this device (Kotlin passes `Build.MODEL`, its `nativePair` convention).
@@ -113,6 +231,14 @@ fn force_parts_sysprop() -> bool {
/// `bitrateKbps` 0 = host default. `compositorPref`/`gamepadPref` are `CompositorPref`/`GamepadPref`
/// wire bytes (0 = Auto; unknown → Auto). `audioChannels` is the requested surround layout (2/6/8;
/// normalized, anything else → stereo) — the host clamps it and the resolved count drives playback.
/// `audioRateHz`/`audioBits` are the audio FORMAT asked for. **`0`/`0` — and anything unrecognized
/// — is "did not ask", the legacy Opus request every build has made**; any pair core can carry asks
/// the host for the lossless `0xD3` plane, `48000`/`16` INCLUDED (that is the cheapest lossless
/// rung, not a spelling of "default" — see [`resolve_requested_audio_format`], which is where
/// getting this backwards silently upgraded every Opus session). Only a request; the host's gate may
/// answer Opus regardless, and this device may not be able to open the rate at all, which is what
/// [`resolve_requested_audio_format`] settles HERE rather than letting the session negotiate a wire
/// it cannot play.
/// `preferredCodec` is the soft codec preference wire byte (0 = Auto). `timeoutMs` is the handshake
/// budget: the normal path passes a short value, the no-PIN "request access" path a long one (≥ the
/// host's approval-park window) so a slow operator approval lands on this same parked connection
@@ -137,6 +263,8 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
multi_slice_ok: jboolean,
frame_parts_ok: jboolean,
audio_channels: jint,
audio_rate_hz: jint,
audio_bits: jint,
video_codecs: jint,
preferred_codec: jint,
timeout_ms: jint,
@@ -222,7 +350,21 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
height: height as u32,
refresh_hz: refresh_hz as u32,
};
match NativeClient::connect(
// Requested surround layout (2 = stereo / 6 = 5.1 / 8 = 7.1); anything else is stereo. The
// host clamps it and echoes the resolved count in `connector.audio_channels`, which drives the
// decoder + AAudio layout (read in `crate::audio::AudioPlayback::start`).
let audio_channels =
punktfunk_core::audio::normalize_channels(audio_channels.clamp(0, u8::MAX as jint) as u8);
// The audio format, downgraded to something this device has PROVED it can open before the
// `Hello` carries it — see `resolve_requested_audio_format` for why it cannot wait until
// playback. `clamp` first: a negative jint from a corrupted setting must not wrap into a
// plausible rate.
let (audio_rate_hz, audio_bits) = resolve_requested_audio_format(
audio_rate_hz.max(0) as u32,
audio_bits.clamp(0, u8::MAX as jint) as u8,
audio_channels,
);
match NativeClient::connect_with_audio_format(
&host,
port as u16,
mode,
@@ -248,11 +390,15 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
} else {
0
}),
// Requested surround layout (2 = stereo / 6 = 5.1 / 8 = 7.1). The host clamps to what it can
// capture and echoes the resolved count in `connector.audio_channels`, which drives the
// decoder + AAudio layout (read in `crate::audio::AudioPlayback::start`). Anything else
// normalizes to stereo here.
punktfunk_core::audio::normalize_channels(audio_channels.clamp(0, u8::MAX as jint) as u8),
audio_channels,
// The audio format this session ASKS for (resolved above). A non-default pair is what
// makes core set `CLIENT_CAP_AUDIO_HIRES` in the `Hello` — capable AND the user turned it
// on, the `VIDEO_CAP_444` precedent — and it is answered by the host re-formatting the
// wire, so it must never be advertised on a device that cannot open the output. The host
// may still decline; `connector.audio_codec`/`audio_sample_rate_hz`/`audio_bits` are what
// actually happened, and `crate::audio` opens the device from those, never from these.
audio_rate_hz,
audio_bits,
// Codecs this device can decode, ranked on the Kotlin side (`VideoDecoders.decodableCodecBits`:
// H.264 + HEVC always, AV1 when a real `video/av01` decoder exists — AMediaCodec is
// mime-driven, see `codec_mime`). Mask to the known bits and fall back to the pre-AV1
@@ -490,3 +636,165 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativePair<'local
})
.resolve::<LogErrorAndDefault>()
}
#[cfg(test)]
mod tests {
use super::*;
use punktfunk_core::audio::pcm::{BITS_16, BITS_24};
use punktfunk_core::audio::SAMPLE_RATE_HZ;
/// The rule this leg exists to enforce: the `Hello` never asks for an audio format this device
/// has not proved it can open, because after the handshake there is no way back — AAudio grants
/// an explicit rate or fails the open, the host does not renegotiate the plane mid-session, and
/// playing a 96 kHz wire through a 48 kHz stream is not a fallback, it is the wrong audio.
///
/// Off-device (this test's target) `audio_rate_is_openable` answers `false` for everything, so
/// what is pinned here is the DOWNGRADE, which is the half that has to be right: a device that
/// cannot do the rate still gets a lossless session at 48 kHz rather than being pushed all the
/// way back to Opus, and the depth — the thing lossless is actually for — survives.
#[test]
fn an_unopenable_rate_is_downgraded_before_the_hello_and_keeps_its_depth() {
// 48 kHz/16-bit is the cheapest LOSSLESS rung, not a way of spelling "default" — asking
// for it explicitly passes through and probes nothing. What a default session sends is
// `(0, 0)`, pinned in `an_opus_session_asks_for_nothing_and_a_lossless_one_asks_for_
// something`, and conflating the two is what silently upgraded every Opus user.
assert_eq!(
resolve_requested_audio_format(SAMPLE_RATE_HZ, BITS_16, 2),
(SAMPLE_RATE_HZ, BITS_16)
);
// 48 kHz is never probed, so 48/24 lossless survives even where nothing can be opened.
assert_eq!(
resolve_requested_audio_format(SAMPLE_RATE_HZ, BITS_24, 2),
(SAMPLE_RATE_HZ, BITS_24)
);
// Every rung above the floor IS probed, is refused here, and lands on 48 kHz with the
// depth intact — including the whole 44.1 kHz family, which this pass admitted. AAudio
// never substitutes a rate, so a device that would not grant 176 400 Hz and was asked for
// it anyway is silence, not a slower session.
for rate in [44_100u32, 88_200, 96_000, 176_400] {
assert_eq!(
resolve_requested_audio_format(rate, BITS_24, 2),
(SAMPLE_RATE_HZ, BITS_24),
"{rate} Hz should have fallen to 48 kHz and kept 24-bit"
);
}
// Surround asks exactly as stereo does. The lossless plane was stereo-only while a
// surround frame did not fit a datagram; the frame ladder is channel-aware, the host
// decides, and this leg's only job is to prove the OUTPUT opens (at the layout it will
// actually be opened with).
assert_eq!(
resolve_requested_audio_format(SAMPLE_RATE_HZ, BITS_24, 6),
(SAMPLE_RATE_HZ, BITS_24)
);
}
/// A settings string, a profile written by a newer build, or a corrupted preference must never
/// reach the wire as a format the plane cannot carry — and must never block a connect either.
/// Every one resolves to the "did not ask" sentinel, which every host can answer.
///
/// The rate set is `pcm::rate_is_supported`'s, not a second copy of it: 44 100 Hz used to sit
/// in this table because §4.1's integer samples-per-millisecond arithmetic could not express
/// it, and the day that stopped being true a locally re-expressed set would have kept refusing
/// it with a stale reason.
#[test]
fn an_unrepresentable_request_falls_back_instead_of_failing() {
for (rate, bits) in [
(0, 0), // Kotlin's Opus setting, and its "unset"
(22_050, BITS_24), // below the ladder — a rate this protocol never negotiates
(192_000, BITS_24), // out by §3's scope decision, not by any arithmetic
(384_000, BITS_24), // above anything anyone has asked for
(SAMPLE_RATE_HZ, 32), // 32-bit float is deliberately not on the wire
(SAMPLE_RATE_HZ, 8), // not a depth this plane carries
(176_400, 32), // a carried rate cannot rescue an uncarried depth
] {
assert_eq!(
resolve_requested_audio_format(rate, bits, 2),
(0, 0),
"{rate} Hz / {bits}-bit should have asked for nothing at all"
);
}
}
/// ⚠⚠ **The one that decides whether a user who chose Opus is quietly given 1.5 Mbps of PCM.**
///
/// Core's `advertised_client_caps` sets `CLIENT_CAP_AUDIO_HIRES` when **either** field of the
/// pair below is non-zero. It keys on "a format was specified" rather than "the format differs
/// from the default", and deliberately: 48 kHz/16-bit is the legacy pair AND the cheapest
/// lossless rung, so the other rule would make that rung the one nobody could request.
///
/// The consequence is that `(48 000, 16)` is not a way of saying "default" — it is a request,
/// the host's gate accepts it as a supported format, and on any host with
/// `PUNKTFUNK_AUDIO_HIRES=1` the user who chose Standard gets the lossless `0xD3` plane instead
/// of Opus. Nothing surfaces it: the settings screen shows what was asked for, and a granted
/// session and a declined one look identical from there. This function returned that pair until
/// all four clients were compared against each other.
///
/// The rule is restated here rather than called, because it is private to core; core's own
/// tests pin the other half of it. What this test owns is that **this** client never hands it a
/// non-zero pair unless the user asked for one.
#[test]
fn an_opus_session_asks_for_nothing_and_a_lossless_one_asks_for_something() {
// Core's rule, verbatim: `audio_rate_hz != 0 || audio_bits != 0`.
let asks_for_hires = |(rate_hz, bits): (u32, u8)| rate_hz != 0 || bits != 0;
assert!(
!asks_for_hires(resolve_requested_audio_format(0, 0, 2)),
"the default session must not advertise the capability"
);
// Surround changes nothing about it: the plane is negotiated by format, not by layout.
assert!(!asks_for_hires(resolve_requested_audio_format(0, 0, 8)));
// And every rung the settings screen offers does ask — otherwise the setting is inert.
// The 48 kHz rows resolve unprobed; the rest fall to 48 kHz off-device (see above) and
// still ask, because the fallback keeps the DEPTH and a 24-bit request is a real one.
for rate in [44_100u32, SAMPLE_RATE_HZ, 88_200, 96_000, 176_400] {
let wire = resolve_requested_audio_format(rate, BITS_24, 2);
assert!(
asks_for_hires(wire),
"{rate} Hz / 24-bit resolved to {wire:?}, which asks for nothing"
);
assert_eq!(wire.1, BITS_24, "the depth must survive every fallback");
}
}
/// The fallback ladder's shape, which decides what a device that refuses the user's rate is
/// asked for next — and which is the only place this client makes a quality choice on the
/// user's behalf, so it is worth pinning rather than reading.
#[test]
fn the_rate_ladder_descends_its_own_family_and_ends_at_the_48_khz_floor() {
for rate in [44_100u32, SAMPLE_RATE_HZ, 88_200, 96_000, 176_400] {
let ladder = rate_fallback_ladder(rate);
assert_eq!(ladder[0], rate, "{rate} Hz must ask for itself first");
assert_eq!(
ladder.last().copied(),
Some(SAMPLE_RATE_HZ),
"{rate} Hz must end at the floor every Android output grants"
);
for &rung in ladder {
assert!(
punktfunk_core::audio::pcm::rate_is_supported(rung),
"{rung} Hz is on {rate} Hz's ladder but is not a rate the plane carries"
);
}
// Strictly descending, so a fallback is never an upgrade in cost — except for the
// 48 kHz floor itself, which is above 44 100 and is the crossing the ladder makes on
// purpose (see `rate_fallback_ladder`).
for w in ladder.windows(2) {
assert!(
w[1] < w[0] || w[1] == SAMPLE_RATE_HZ,
"{rate} Hz's ladder goes up at {:?}",
w
);
}
}
// The 44.1 family stays in the 44.1 family for as long as it can: a 176.4 kHz endpoint
// that will not open is far likelier to be an 88.2 or 44.1 kHz one than a 96 kHz one.
assert_eq!(
rate_fallback_ladder(176_400),
&[176_400, 88_200, 44_100, SAMPLE_RATE_HZ]
);
assert_eq!(rate_fallback_ladder(96_000), &[96_000, SAMPLE_RATE_HZ]);
// A default session's ladder is one rung, so it opens no probe stream at all.
assert_eq!(rate_fallback_ladder(SAMPLE_RATE_HZ), &[SAMPLE_RATE_HZ]);
}
}
clients/android/native/src/session/planes.rs
+21 -5
View File
@@ -174,12 +174,13 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopVideo(
}
/// `NativeBridge.nativeVideoStats(handle): DoubleArray?` — drain ~1 s of decode stats for the HUD
/// (unified stats spec, `design/stats-unification.md`). Returns 35 doubles
/// (unified stats spec, `design/stats-unification.md`). Returns 38 doubles
/// `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skewCorrected, width, height, refreshHz, framesLost,
/// bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms,
/// netP50Ms, lostWindow, skippedWindow, fecWindow, framesWindow, dispValid, displayP50Ms,
/// e2eDispP50Ms, e2eDispP95Ms, paceP50Ms, latchP50Ms, presentsWindow, presenterActive,
/// feedP50Ms, codecP50Ms, skippedOverflowWindow, audioBufferMs, audioAvOffsetMs]`
/// feedP50Ms, codecP50Ms, skippedOverflowWindow, audioBufferMs, audioAvOffsetMs, audioCodec,
/// audioRateHz, audioBits]`
/// (the flags are 1.0/0.0; indexes 0–21 match the previous 22-double layout — 0–13 the original
/// 14-double one with the latency pair re-based to the end-to-end capture→decoded headline, 14/15
/// the stage p50s tiling it: `host+network` = capture→received, `decode` = received→decoded; 16/17
@@ -202,8 +203,13 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopVideo(
/// parked-AU overflow subset of the window's `skipped` at 19 (decoder fell behind, vs benign
/// newest-wins pacing); 33/34 are the AUDIO plane's latency — the playback ring's live depth in ms
/// and the A/V sync loop's smoothed offset in ms (positive = audio behind the picture) — both live
/// gauges rather than windowed samples, like the cumulative drop total at 9), or `null` when no
/// decode thread is running.
/// gauges rather than windowed samples, like the cumulative drop total at 9; 35–37 are the audio
/// FORMAT the host resolved at the handshake — `audioCodec` (`0` = Opus on `0xC9`, `2` = lossless
/// PCM on `0xD3`), the resolved rate in Hz and the resolved depth in bits. Static for the session,
/// and here because `design/hi-res-audio.md` §10 requires a surface for the RESOLVED format rather
/// than the requested one: a session that spends 4.6 Mbps and a session whose host quietly
/// declined look identical from the outside, which is §4.3's failure wearing a UI hat), or `null`
/// when no decode thread is running.
/// Poll ~1 Hz from the UI; each call
/// resets the measurement window. Not android-gated — pure `jni` + connector reads, so it links on
/// the host build too (Kotlin only ever calls it on device).
@@ -227,7 +233,7 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats<
.drain(h.client.frames_dropped(), h.client.fec_recovered_shards());
let mode = h.client.mode();
let color = h.client.color;
let buf: [f64; 35] = [
let buf: [f64; 38] = [
snap.fps,
snap.mbps,
snap.e2e_p50_ms,
@@ -290,6 +296,16 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats<
// high" report had no instrument behind it at all.
h.client.audio_buffer_ms() as f64,
h.client.audio_av_offset_ms() as f64,
// The audio format the host RESOLVED (`Welcome`), not what this device asked for.
// A lossless session and a session whose host declined lossless are indistinguishable
// from the outside — same picture, same latency figures, one of them quietly spending
// 2.3–4.6 Mbps of the link on nothing — so the HUD has to be able to name which
// (`design/hi-res-audio.md` §10, and §4.3 for why it matters). Static for the session:
// the plane is settled at the handshake and the host never switches it underneath a
// client whose output device is already open.
h.client.audio_codec as f64,
h.client.audio_sample_rate_hz as f64,
h.client.audio_bits as f64,
];
let arr = env.new_double_array(buf.len())?;
arr.set_region(env, 0, &buf)?;
clients/apple/Sources/PunktfunkClient/Session/SessionModel.swift
+64
View File
@@ -179,6 +179,45 @@ final class SessionModel: ObservableObject {
@Published var audioBufferMs = 0
@Published var audioAvOffsetMs = 0
@Published var audioValid = false
/// The audio format the host RESOLVED, for the HUD — `nil` on an ordinary Opus session, where
/// there is nothing to say and a line saying "48 kHz" would be noise.
///
/// The resolved format, emphatically not the requested one. A UI that reads "96 kHz" because
/// the user picked 96 kHz, on a session the host declined, is the exact bug
/// design/hi-res-audio.md §4.3 names wearing a different hat — and it is the one place a user
/// would ever look to check that the bandwidth they are spending is buying anything.
@Published var audioFormatLabel: String?
/// A resolved sample rate as the kHz figure a listener recognises: `48`, `96`, and — since the
/// 44.1 kHz family was admitted — `44.1`, `88.2`, `176.4`.
///
/// ⚠ This exists because `rateHz / 1000` is INTEGER division, and a HUD reading "44 kHz" on a
/// 44 100 Hz session would be the one surface whose whole job is naming what the host resolved,
/// naming it wrong.
///
/// Built from integer parts rather than `String(format: "%.1f", …)` for the reason the Android
/// port records at its own copy of this: that formatter renders through the current locale and
/// would print "44,1 kHz" across most of Europe, a decimal comma facing a settings row that
/// says "44.1 kHz" — and this line exists precisely to be compared at a glance with what was
/// asked for. Interpolating `Int`s is locale-independent, and every rate the plane carries is a
/// whole number of hundreds of hertz, so the tenths digit is exact.
private static func kHzLabel(_ rateHz: UInt32) -> String {
let whole = rateHz / 1000
let tenths = (rateHz % 1000) / 100 // 44 100 → 1, 176 400 → 4; 0 for the 48 kHz family
return tenths == 0 ? "\(whole)" : "\(whole).\(tenths)"
}
/// The resolved speaker layout, spelled the way the settings row spells it. Named on this line
/// because the lossless plane is no longer stereo-only: what a surround lossless session costs
/// is three or four times the stereo figure, so "which layout did I actually get" is now part
/// of "is the bandwidth I am spending buying anything".
private static func layoutLabel(_ channels: UInt8) -> String {
switch channels {
case 6: return "5.1"
case 8: return "7.1"
default: return "stereo"
}
}
/// The floor-shaved values every HUD tier displays (raw − floor, never below 0). Identical
/// to the raw values whenever no floor is measured.
@@ -358,6 +397,22 @@ final class SessionModel: ObservableObject {
rawValue: UInt32(clamping: effective.compositor)) ?? .auto
let bitrateKbps = UInt32(clamping: effective.bitrateKbps)
let audioChannels = UInt8(clamping: effective.audioChannels)
// The audio format this session ASKS for — the user's choice, at every channel count.
//
// This used to be forced to Opus for 5.1/7.1, on the reasoning that a lossless surround
// frame does not fit one datagram. That was a statement about ONE frame length: the ladder
// is sized from `(rate, depth, channels, max_datagram)`, so a surround session negotiates a
// shorter frame rather than failing, and only the top of the rate ladder has no rung that
// fits. Deciding that here, from a rule this side cannot measure, meant a client guess
// standing in for the host's measurement — and guessing "no" costs a session that would
// have worked. The host's gate is the one place that knows the connection's real datagram
// size; asking and being declined is one `Welcome` field, and it is the honest shape.
//
// **The request is never the answer.** `resolvedAudioRateHz`/`resolvedAudioBits`/
// `isLosslessAudio` on the connection are what the host actually granted, SessionAudio
// opens the device from THOSE, and `audioFormatLabel` below reports THOSE.
let audioFormat = effective.audioFormatChoice
let (audioRateHz, audioBits) = audioFormat.wire
let hdrEnabled = effective.hdrEnabled
let preferredCodec = PunktfunkConnection.codecByte(effective.codec)
let pin = host.pinnedSHA256
@@ -452,6 +507,7 @@ final class SessionModel: ObservableObject {
pinSHA256: pin, identity: identity, compositor: compositor,
gamepad: gamepad, bitrateKbps: bitrateKbps, videoCaps: videoCaps,
audioChannels: audioChannels,
audioRateHz: audioRateHz, audioBits: audioBits,
videoCodecs: videoCodecs, preferredCodec: preferredCodec,
clientCaps: clientCaps, launchID: launchID,
// Delegated approval: the host holds this connect open until the operator approves
@@ -820,6 +876,7 @@ final class SessionModel: ObservableObject {
// link may never come up (a non-deadline rung has none at all).
PresentLinkInfo.shared.clear()
audioValid = false
audioFormatLabel = nil
lostFrames = 0
lostPct = 0
mouseCaptured = false
@@ -917,6 +974,13 @@ final class SessionModel: ObservableObject {
// correctly declines to correct.
videoLatency: endToEnd)
self.audio = audio
// Only when the session is genuinely on the lossless plane — the HUD says nothing for an
// ordinary Opus one. Read from the connection's Welcome, so a request the host's gate
// declined shows the fallback it actually landed on rather than what was asked for.
audioFormatLabel = conn.isLosslessAudio
? "lossless \(Self.kHzLabel(conn.resolvedAudioRateHz)) kHz / "
+ "\(conn.resolvedAudioBits)-bit \(Self.layoutLabel(conn.resolvedAudioChannels))"
: nil
// Gamepads: forward every controller GamepadManager selected — each on its own wire pad
// index (a pin forwards only one, Automatic forwards all) — and render the host's feedback
// back to the pad it's addressed to (rumble always; lightbar/player-LEDs/adaptive-triggers
clients/apple/Sources/PunktfunkClient/Session/StreamHUDView.swift
+11
View File
@@ -214,6 +214,17 @@ struct StreamHUDView: View {
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.tertiary)
}
// The RESOLVED audio format, and only when it is worth a line: `audioFormatLabel` is
// nil on the Opus plane every ordinary session runs. Unlike the numbers above it is
// NOT gated to the detailed tier — it is the one thing a user who turned lossless on
// needs to see, because the failure this guards against (design/hi-res-audio.md §4.3)
// is a session that costs the bandwidth and delivers nothing, and that is
// indistinguishable from success without a surface naming what the HOST resolved.
if let format = model.audioFormatLabel {
Text("audio \(format)")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.tertiary)
}
if model.lostFrames > 0 {
// Unrecoverable network drops this window; hidden while the link is clean.
// String(format:) rather than specifier interpolation: the literal % would
clients/apple/Sources/PunktfunkClient/Settings/GamepadSettingsView.swift
+17
View File
@@ -82,6 +82,7 @@ struct GamepadSettingsView: View {
@AppStorage(DefaultsKey.guideGesture) private var guideGesture = "auto"
@AppStorage(DefaultsKey.bitrateKbps) private var bitrateKbps = 0
@AppStorage(DefaultsKey.audioChannels) private var audioChannels = 2
@AppStorage(DefaultsKey.audioFormat) private var audioFormat = AudioFormatChoice.opus.rawValue
@AppStorage(DefaultsKey.hdrEnabled) private var hdrEnabled = true
@AppStorage(DefaultsKey.enable444) private var enable444 = false
@AppStorage(DefaultsKey.codec) private var codec = "auto"
@@ -855,6 +856,22 @@ struct GamepadSettingsView: View {
detail: "The speaker layout requested from the host.",
options: SettingsOptions.audioChannels, current: audioChannels
) { audioChannels = $0 },
// No longer chained to the row above. The lossless plane was stereo-only because a
// surround frame did not fit one datagram; the frame ladder is sized per channel
// count, so 5.1/7.1 negotiate a SHORTER frame instead — a higher packet rate, not an
// impossibility — and only the top of the rate ladder genuinely has nowhere to go.
choiceRow(
id: "audioFormat", tab: .audio, icon: "waveform.badge.magnifyingglass",
label: "Audio quality",
detail: "Lossless sends bit-exact PCM instead of compressed audio — 2.1 to 8.5 "
+ "Mbps on top of the video for stereo, three times that for 5.1 and four for "
+ "7.1. It must be enabled on the host too, the host's own interface has to "
+ "run the rate, and this device's output has to accept it. The top of the "
+ "ladder needs room the network may not have: 176.4 kHz sends a thousand "
+ "packets a second, and surround above 48 kHz does not fit at all. Anything "
+ "missing falls back to Standard.",
options: SettingsOptions.audioFormats, current: audioFormat
) { audioFormat = $0 },
toggleRow(
id: "mic", tab: .audio, icon: "mic", label: "Microphone",
detail: "Send this device's microphone to the host's virtual mic.",
clients/apple/Sources/PunktfunkClient/Settings/SettingsOptions.swift
+26
View File
@@ -25,6 +25,32 @@ enum SettingsOptions {
("7.1 Surround", 8),
]
/// Audio format (`DefaultsKey.audioFormat`) — the `tag` is an `AudioFormatChoice` raw value.
/// Standard is the default; every lossless row is a per-session opt-in that spends real
/// bandwidth outside the ABR loop, and the host has its own switch which is also off by
/// default.
///
/// Ordered by rate rather than by family, because that is the order a listener reads a rate
/// ladder in — the two families interleave (44.1, 48, 88.2, 96, 176.4) and grouping them would
/// put 88.2 above 48.
///
/// **Offered at every channel count**, unlike before. This picker used to be hidden unless the
/// session was stereo, on the grounds that a surround frame does not fit one datagram — but the
/// frame ladder is sized per channel count (`pcm::frame_us_for`), so 5.1 and 7.1 simply
/// negotiate SHORTER frames: 48 kHz/16-bit 5.1 lands around 2 ms and 48/24 shorter still, which
/// is a higher packet rate rather than an impossibility. What genuinely fits no rung at the
/// default MTU is surround above 48 kHz, and the honest treatment of that is the caption plus
/// the host's own decline — not a hidden row, which silently discards a choice the user made
/// and explains nothing.
static let audioFormats: [(label: String, tag: String)] = [
("Standard (Opus)", AudioFormatChoice.opus.rawValue),
("Lossless 44.1 kHz / 24-bit", AudioFormatChoice.lossless441.rawValue),
("Lossless 48 kHz / 24-bit", AudioFormatChoice.lossless48.rawValue),
("Lossless 88.2 kHz / 24-bit", AudioFormatChoice.lossless882.rawValue),
("Lossless 96 kHz / 24-bit", AudioFormatChoice.lossless96.rawValue),
("Lossless 176.4 kHz / 24-bit", AudioFormatChoice.lossless1764.rawValue),
]
/// Virtual-pad types — the `tag` is the wire value (`PunktfunkConnection.GamepadType` raw).
static let padTypes: [(label: String, tag: Int)] = [
("Automatic", 0),
clients/apple/Sources/PunktfunkClient/Settings/SettingsView+Scope.swift
+5
View File
@@ -94,6 +94,10 @@ enum SettingsFields {
.init(name: "audio_channels", key: DefaultsKey.audioChannels,
overlay: \.audioChannels, effective: \.audioChannels)
}
static var audioFormat: SettingsField<String> {
.init(name: "audio_format", key: DefaultsKey.audioFormat,
overlay: \.audioFormat, effective: \.audioFormat)
}
static var micEnabled: SettingsField<Bool> {
.init(name: "mic_enabled", key: DefaultsKey.micEnabled,
overlay: \.micEnabled, effective: \.micEnabled)
@@ -194,6 +198,7 @@ extension SettingsView {
base.enable444 = enable444
base.compositor = compositor
base.audioChannels = audioChannels
base.audioFormat = audioFormat
base.micEnabled = micEnabled
base.echoCancel = echoCancel
base.gamepadType = gamepadType
clients/apple/Sources/PunktfunkClient/Settings/SettingsView+Sections.swift
+90
View File
@@ -575,6 +575,18 @@ extension SettingsView {
}
}
}
// Offered at every channel count. This row used to be hidden unless the session was
// stereo; the frame ladder is channel-aware, so surround negotiates a shorter frame
// rather than being impossible, and the caption is where the cases that genuinely do
// not fit are stated (see `audioFormatCaption`). A hidden row silently discards a
// choice and explains nothing.
described(audioFormatCaption, field: "audio_format") {
Picker("Audio quality", selection: scoped(SettingsFields.audioFormat)) {
ForEach(SettingsOptions.audioFormats, id: \.tag) { option in
Text(option.label).tag(option.tag)
}
}
}
#if os(macOS)
// Which speaker THIS Mac plays through is this device's audio routing (tier G).
if !inProfileScope {
@@ -639,6 +651,84 @@ extension SettingsView {
}
}
/// The SELECTED audio format explained, and honest about the ways it can come to nothing: the
/// host has its own switch (off by default) and its own capture gate, this device's output has
/// to be able to open the rate, and the frame has to fit one QUIC datagram. Every failure
/// resolves back to Opus — a good outcome, but not the one the row's label promises, so the
/// caption says so before the user goes looking.
///
/// ⚠ The rule this caption exists to keep is the design's: **the UI states the RESOLVED
/// format, never the requested one.** Nothing here may read as a guarantee — the HUD's
/// `audioFormatLabel` is built from the connection's `Welcome`, and that is the only place a
/// format is asserted as fact.
private var audioFormatCaption: String {
let choice = AudioFormatChoice(setting: effective.audioFormat)
guard choice != .opus else {
return "Compressed audio at 256 kbps — effectively transparent, and what every "
+ "session used before lossless existed."
}
// Stereo cost at 24-bit, from `pcm::bitrate_kbps`. 5.1 is three times it and 7.1 four, so
// the surround rider below states the multiplier rather than repeating the table.
let head: String
switch choice {
case .opus:
head = "" // unreachable — the guard above returns
case .lossless441:
head = "Bit-exact 44.1 kHz / 24-bit PCM — about 2.1 Mbps on top of the video. The "
+ "rate to pick when the host's own endpoint runs at 44.1 kHz, since it is then "
+ "the one that avoids a resample."
case .lossless48:
head = "Bit-exact 48 kHz / 24-bit PCM — no lossy stage at all. Costs about 2.3 Mbps "
+ "on top of the video, and is the rate a game host's engine usually already runs "
+ "at."
case .lossless882:
head = "Bit-exact 88.2 kHz / 24-bit PCM — about 4.2 Mbps on top of the video, and "
+ "only real if the host's interface genuinely runs at 88.2 kHz."
case .lossless96:
head = "Bit-exact 96 kHz / 24-bit PCM — about 4.6 Mbps on top of the video, and only "
+ "real if the host's interface genuinely runs at 96 kHz."
case .lossless1764:
// The one row that is honest about being mostly unreachable. Offering it is fine;
// presenting it as if it will be granted is not. Three vetoes, and the numbers behind
// them: the host gives audio at most a QUARTER of the video budget (8.5 Mbps of audio
// therefore needs ~34 Mbps of video), a stereo frame fits a datagram only on the
// ladder's shortest 1 ms rung at ~1 069 B, and a surround one fits no rung at all.
//
// Plain prose, no markdown: `described` hands this to `Text(String)`, which does NOT
// parse markup — asterisks would render as asterisks.
head = "Bit-exact 176.4 kHz / 24-bit PCM — 8.5 Mbps on top of the video, and far more "
+ "likely to be declined than granted. The host caps audio at a quarter of the "
+ "video bitrate, so it needs a session of about 34 Mbps or more; and each packet "
+ "has to shrink to 1 ms of audio, a thousand a second, which only fits on a "
+ "network with room to spare."
}
// The host's switch is off by default and its capture gate is about the world, not policy;
// this device's output is the third way a granted rate still is not what reaches a speaker
// (a Bluetooth route has no 96 kHz mode to give, and the session log says so).
let gates = " Needs lossless enabled on the host as well, and an output here that accepts "
+ "the rate; anything missing resolves the session back down and the log says which."
guard effective.audioChannels > 2 else { return head + gates }
// Surround is no longer hidden, so what it costs has to be said out loud. The plane sends
// one frame per datagram and never fragments, so more channels buy a SHORTER frame rather
// than a bigger packet — a packet rate, not an impossibility, right up until no rung on the
// ladder fits at all. Where that line falls, at 24-bit and the default datagram size
// (`pcm::frame_us_for` against ~1 387 B of payload): 44.1 kHz 5.1/7.1 and 48 kHz 7.1 land
// on the 1 ms rung, 48 kHz 5.1 on 1.5 ms, and 88.2 kHz upward fit NOTHING. Two different
// sentences, because "it will cost you" and "it will not happen" are two different things
// to tell someone.
let layout = effective.audioChannels == 6 ? "5.1" : "7.1"
let multiple = effective.audioChannels == 6 ? "three" : "four"
let fitsSurround = choice == .lossless441 || choice == .lossless48
guard fitsSurround else {
return head + gates + " And on \(layout) this rate does not fit an ordinary network's "
+ "packets at any length, so the session resolves back to Standard — 48 kHz or "
+ "44.1 kHz is as far up as surround goes here."
}
return head + gates + " On \(layout) it costs \(multiple) times that, and the host shortens "
+ "each packet to about 1–1.5 ms of audio to keep it inside one datagram — roughly "
+ "650 to 1 000 packets a second."
}
/// Honest about the macOS escape hatch: the voice processor only follows the system
/// default devices, so hand-picked endpoints silently keep the raw path (see
/// SessionAudio's topology note) — better said here than discovered mid-call.
clients/apple/Sources/PunktfunkClient/Settings/SettingsView.swift
+16
View File
@@ -68,6 +68,7 @@ struct SettingsView: View {
@AppStorage(DefaultsKey.micEnabled) var micEnabled = true
@AppStorage(DefaultsKey.echoCancel) var echoCancel = true
@AppStorage(DefaultsKey.audioChannels) var audioChannels = 2
@AppStorage(DefaultsKey.audioFormat) var audioFormat = AudioFormatChoice.opus.rawValue
@AppStorage(DefaultsKey.codec) var codec = "auto"
// The overlay tier's raw string (the pickers tag by rawValue); the absent-key default runs
// the legacy-hudEnabled migration (same pattern as ContentView/StreamCommands).
@@ -475,6 +476,21 @@ struct SettingsView: View {
title: "Audio channels",
options: SettingsOptions.audioChannels,
selection: $audioChannels)
// Offered at every channel count — the lossless plane is no longer stereo-only,
// because the frame ladder is sized per channel count and surround simply
// negotiates a shorter frame (see SettingsOptions.audioFormats).
TVSelectionRow(
title: "Audio quality",
options: SettingsOptions.audioFormats,
selection: $audioFormat)
tvCaption("Lossless sends bit-exact PCM instead of compressed audio — 2.1 to 8.5 "
+ "Mbps on top of the video for stereo, three times that for 5.1 and four for "
+ "7.1. It also has to be enabled on the host, the host's own interface has "
+ "to run the rate, and the Apple TV's output has to accept it. The top of "
+ "the ladder needs room the network may not have: 176.4 kHz sends a thousand "
+ "packets a second, and surround above 48 kHz does not fit at all. Anything "
+ "missing falls back to Standard — the stats overlay shows what the session "
+ "actually got.")
TVSelectionRow(
title: "Auto-wake on connect",
options: [("On", "on"), ("Off", "off")], selection: autoWakeEnabledTag)
clients/apple/Sources/PunktfunkKit/Audio/AudioRing.swift
+346 -82
View File
@@ -1,6 +1,74 @@
import AVFoundation
import os
// MARK: - the ms ⇄ interleaved-sample conversion both the ring and the sync loop run on
//
// **Multiply first, divide last.** This is the whole of design/hi-res-audio.md §4.1, and it is the
// Swift half of the same fix core just took (`punktfunk_core::audio::ms_to_samples`). Both types
// below used to precompute `perMS = (rateHz / 1000) * channels` and express every figure they own
// as `ms * perMS`. That division happens FIRST, so 44 100 Hz became 44 samples per millisecond and
// every depth, target, shed threshold, hard cap, de-prime fuse and reported `bufferedMS`/`targetMS`
// came out **2.3 % low** — quietly, permanently, and in the one subsystem two previous programs
// spent their time making trustworthy. 48 000 and 96 000 were exact only because they happen to
// divide.
//
// Keeping `rateHz` and `channels` as the two numbers they are, and dividing last, is exact at every
// rate on the ladder (`pcm::rate_is_supported`: 44 100 / 48 000 / 88 200 / 96 000 / 176 400). It
// costs one integer division per conversion and buys three more rates.
//
// **Why `Int` is enough here, where core needed an explicit `u64`.** Core's conversions run against
// `usize`, which is 32 bits on some embedder targets, so they widen and saturate by hand. This
// package builds for macOS 14 / iOS 17 / tvOS 17 only (Package.swift) — arm64 and x86_64, where
// `Int` is 64-bit — and the largest product any caller can reach is the longest span this file
// names against the top of the ladder: `syncBackoffMaxMS` (480 000 ms) × 176 400 Hz × 8 ch =
// 6.8 × 10¹¹, forty bits, before the divide brings it back to 6.8 × 10⁸. That is five orders of
// magnitude inside `Int.max`. The samples → ms direction is the one that takes a caller-supplied
// count and is guarded, because Swift TRAPS on overflow rather than wrapping and that trap would
// land in a realtime render callback.
/// Interleaved samples per second at a negotiated layout — the denominator both conversions share.
/// `max(1)` on both: a degenerate layout must not divide by zero in a render callback.
func audioInterleavedPerSec(rateHz: Int, channels: Int) -> Int {
max(rateHz, 1) * max(channels, 1)
}
/// `ms` milliseconds of audio, in interleaved samples. Mirrors `ms_to_samples`.
func audioMsToSamples(rateHz: Int, channels: Int, ms: Int) -> Int {
ms * audioInterleavedPerSec(rateHz: rateHz, channels: channels) / 1_000
}
/// Interleaved samples back to whole milliseconds — the exact inverse of [`audioMsToSamples`], and
/// the reason `depthMS(target)` round-trips to `targetMS` at every rate on the ladder. §4.1 names
/// that round trip as the tell that this rework is incomplete, so it is also the shape of the test
/// that guards it (`testTheShippingRateLadderRoundTripsMsToSamplesExactly`).
///
/// `samples` arrives from a caller and nothing bounds it — `setSyncTarget(Int.max / 2)` is a real
/// call this file's own tests make — and `samples * 1_000` on that would TRAP, taking the process
/// down from wherever it was asked (the drain thread, or the render callback). Core widens to u128
/// for the same reason; Swift has no u128, so the multiply reports its overflow and saturates.
/// Saturating rather than wrapping, because a wrapped duration is a tiny one: a fuse that blows
/// instantly instead of one that never blows.
func audioSamplesToMs(rateHz: Int, channels: Int, samples: Int) -> Int {
let (scaled, overflow) = samples.multipliedReportingOverflow(by: 1_000)
guard !overflow else { return Int.max }
return scaled / audioInterleavedPerSec(rateHz: rateHz, channels: channels)
}
/// Interleaved samples in one `frameUs` frame — **per channel × channels**. Mirrors
/// `punktfunk_core::audio::pcm::samples_per_frame`, which is the single source of truth for how
/// long a frame is: the host fills a buffer of this size and this ring drains one, so the two agree
/// by construction rather than by both re-deriving `rate × µs` and hoping they round the same way.
///
/// ⚠ **Not the same question as "how many samples is `frameUs` of audio", and at 44.1 kHz not the
/// same answer.** The divide is per channel and FLOORS, because 220.5 samples do not exist: 5 ms of
/// 44.1 kHz stereo audio is 441 interleaved samples, but a 5 ms FRAME of it carries 440. Both the
/// shed size and the near-miss margin mean *exactly one packet*, so computing the first where the
/// wire delivers the second would describe a packet that does not exist. Multiply first here too —
/// `rateHz / 1_000_000` is 0 for every rate below a megahertz.
func audioSamplesPerFrame(rateHz: Int, frameUs: Int, channels: Int) -> Int {
(max(rateHz, 1) * max(frameUs, 0) / 1_000_000) * max(channels, 1)
}
/// SPSC-ish jitter ring (interleaved float, `channels` per frame), drain thread → render
/// callback. The unfair lock is held for microseconds; fine at render-callback rates. Priming:
/// reads return silence until enough is buffered (at least the target, and at least one
@@ -8,13 +76,14 @@ import os
/// chronically out-demand the prefill and oscillate prime → dropout → re-prime).
/// All counts stay whole frames (multiples of `channels`), so the interleave can never slip.
///
/// **Drift correction.** Both ends run at 48 kHz but on different crystals, so backlog from a
/// network stall or plain host-vs-DAC skew never drains on its own: without correction one 300 ms
/// hiccup leaves audio 300 ms behind video for the rest of the session. This used to be handled by
/// a `highWater` shed that dropped a whole `2 × prefill` at once — its own comment called that "one
/// audible blip". It is now the same two-stage scheme the Rust clients share
/// (`punktfunk_core::audio::JitterPolicy`): a slow depth average that sits above target for a
/// sustained window sheds ONE 5 ms frame with a crossfade, and the hard cap is only a backstop.
/// **Drift correction.** Both ends run at the same nominal rate but on different crystals, so
/// backlog from a network stall or plain host-vs-DAC skew never drains on its own: without
/// correction one 300 ms hiccup leaves audio 300 ms behind video for the rest of the session. This
/// used to be handled by a `highWater` shed that dropped a whole `2 × prefill` at once — its own
/// comment called that "one audible blip". It is now the same two-stage scheme the Rust clients
/// share (`punktfunk_core::audio::JitterPolicy`): a slow depth average that sits above target for a
/// sustained window sheds exactly ONE audio frame with a crossfade — the session's real frame, see
/// `setFrameUs` — and the hard cap is only a backstop.
///
/// **Adaptive depth.** The target is a floor, not a constant: a NEAR-MISS — a read served with
/// less than one frame left over — grows it a step BEFORE anything was audible, repeated genuine
@@ -57,7 +126,31 @@ final class AudioRing: @unchecked Sendable {
/// Floor in callbacks under `deprimeMS`, so a large-quantum device keeps real hysteresis
/// instead of de-priming on the first short read. Mirrors `MIN_DEPRIME_CALLBACKS`.
private static let minDeprimeCallbacks = 2
/// The protocol's frame: the shed unit, and the slack added over a large device quantum.
/// The protocol's DEFAULT frame — the Opus plane's 5 ms, mirroring `PUNKTFUNK_AUDIO_FRAME_MS`.
///
/// The frame a session actually runs is RESOLVED, not assumed: the lossless plane sizes it to
/// the path MTU because a 5 ms hi-res frame does not fit one QUIC datagram — 4 ms at
/// 48 kHz/24-bit stereo, 2 ms at 96 kHz/24-bit stereo under the default ceiling, and shorter
/// again for surround, whose frame carries three or four times the samples for the same
/// duration (design/hi-res-audio.md §4.2). `setFrameUs` takes that figure from
/// `punktfunk_connection_audio_frame_us`; this constant is what a ring keeps until somebody
/// calls it, which is exactly right for every Opus session and for the tests that pin the
/// pre-hi-res behaviour.
///
/// Still the right unit for `AvSync`'s EWMA weight, which core also leaves on the constant: it
/// is a time constant on a loop with a 100-observation settling gate, so a shorter real frame
/// only makes it settle sooner.
///
/// ⚠ **`DroughtConceal` below is a different story, and it is now BEHIND core.** Core moved its
/// drought policy onto the resolved frame (`DroughtConceal::new_at_frame_us`) after this file
/// was last touched: it charges one `frame_us` per concealed frame and triggers at two of them,
/// where this leg still charges a flat 5 ms. The frame COUNT stays right either way — the drain
/// thread writes one real frame per `conceal()` — so nothing plays wrong; what is wrong is the
/// BUDGET and the REPORT. On a 2 ms lossless frame `plcMaxMS` is spent after two fifths of the
/// wall clock it promises and `plc_ms` over-reports by 2.5×, and a 1 ms surround frame makes
/// that a factor of five. Fixing it means giving this type the frame the same way the ring gets
/// it; it is deliberately NOT part of the rate/surround change, and it is the next thing this
/// file owes core.
static let frameMS = 5
/// Depth average must exceed target by this before drift correction fires — the middle of the
/// headroom band, so the smooth shed always gets its chance BEFORE the hard cap trims.
@@ -83,11 +176,6 @@ final class AudioRing: @unchecked Sendable {
/// a measurement saying the extra depth is costing alignment right now — so a smaller target
/// gets tested sooner. Mirrors `SHRINK_QUIET_SYNC_MS`.
private static let shrinkQuietSyncMS = 5_000
/// Post-read depth below which a served callback counts as a NEAR-MISS: the device got its
/// samples, but with less than one protocol frame left in hand — the same evidence as an
/// underrun, except nobody heard it yet, so the target grows BEFORE the click instead of
/// after the third one. Mirrors `NEAR_MISS_MARGIN_MS`.
private static let nearMissMarginMS = frameMS
/// How long a shrink remains a PROBE, in consumed audio: an underrun or near-miss inside
/// this window means the shrink was wrong, and the previous target is restored at once.
/// Mirrors `SHRINK_PROBE_MS`.
@@ -119,7 +207,7 @@ final class AudioRing: @unchecked Sendable {
private var depthAvg: Double = 0
private var overRun = 0
/// The live target in interleaved samples — `targetMS` grown by underrun pressure
/// (`noteRead`), never below the base. Set in `init` (needs `perMS`).
/// (`noteRead`), never below the base. Set in `init` (needs the rate).
private var targetLive = 0
/// Underruns seen in the current growth window, and the window's consumed-sample count.
private var underrunsInWindow = 0
@@ -135,8 +223,8 @@ final class AudioRing: @unchecked Sendable {
/// `nil` — the default, and what an un-wired session keeps — reproduces the pre-sync
/// behaviour exactly, so this ring could adopt sync without the other three diverging.
private var syncTarget: Int?
/// This read was served with less than `nearMissMarginMS` left over (set in `read`,
/// consumed by `noteRead`).
/// This read was served with less than one frame left over (set in `read`, consumed by
/// `noteRead`).
private var nearMiss = false
/// A near-miss already grew the target this window — one step per window, so a bunching
/// episode (a RUN of consecutive near-misses while the ring refills) buys one measured
@@ -161,17 +249,108 @@ final class AudioRing: @unchecked Sendable {
/// same reason `avOffsetMS` is: the ring cannot compute it, but it is where the numbers a
/// listener's complaint needs can be read under one lock.
private var plcMS = 0
/// The negotiated sample rate and interleaved channel count, kept as the two numbers they are
/// rather than pre-divided into samples-per-millisecond — see the conversion helpers at the top
/// of this file for why that division WAS the defect. Both are clamped to ≥ 1 on the way in.
private let channels: Int
private let perMS: Int
private let rateHz: Int
/// One protocol audio frame in MICROSECONDS — see `setFrameUs`. Guarded by `lock`, like
/// everything else the render callback reads.
private var frameUs = AudioRing.frameMS * 1_000
private let lock = OSAllocatedUnfairLock()
/// `capacity` in samples (interleaved — `channels` per frame, a whole number of frames).
/// The de-jitter depth is the ring's own business (`targetMS`), not a caller's prefill.
init(capacity: Int, channels: Int) {
buf = [Float](repeating: 0, count: capacity)
self.channels = channels
perMS = 48 * channels
targetLive = Self.targetMS * perMS
/// A ring holding `seconds` of audio at the session's negotiated format. The de-jitter depth
/// is the ring's own business (`targetMS`), not a caller's prefill.
///
/// Sized in TIME rather than in a sample count, because the sample count is `rateHz × channels`
/// and the two were indistinguishable while every session ran at 48 kHz: this was
/// `capacity: 48_000 * channels`, where the literal was silently doing double duty as
/// samples-per-second. At 96 kHz that same expression is half a second of ring — no error, no
/// warning, just half the overflow headroom on the plane that needs it most.
///
/// **Every rate on the lossless ladder is exact here** — 44 100 / 48 000 / 88 200 / 96 000 /
/// 176 400 (`pcm::rate_is_supported`). It was not always: this used to precompute an INTEGER
/// `perMS = (rateHz / 1000) * channels` and express every figure it owns — target, EWMA depth,
/// shed threshold, hard cap, de-prime fuse, and the `bufferedMS`/`targetMS` the HUD reports —
/// as `ms * perMS`. That leading division truncated 44 100 Hz to 44 samples/ms and put all of
/// them 2.3 % low, which is the sole reason the 44.1 kHz family was deferred rather than
/// refused (design/hi-res-audio.md §4.1). The conversions now multiply first and divide last,
/// so there is no rate this ring cannot represent. Mirrors `JitterPolicy::new_at_rate`; keep
/// the two in step.
///
/// A `rateHz` or `channels` of zero is clamped to 1 rather than rejected: this is built from
/// wire-supplied values on a path that must not fault in a render callback, and it used to
/// divide by `perMS` with nothing but a `max(1)` at the use sites.
init(seconds: Int, channels: Int, rateHz: Int) {
self.channels = max(channels, 1)
self.rateHz = max(rateHz, 1)
buf = [Float](repeating: 0, count: max(seconds, 1) * self.rateHz * self.channels)
targetLive = msSamples(Self.targetMS)
}
/// `ms` of audio in interleaved samples at this session's layout — see `audioMsToSamples`.
private func msSamples(_ ms: Int) -> Int {
audioMsToSamples(rateHz: rateHz, channels: channels, ms: ms)
}
/// The inverse, for the figures this ring reports — see `audioSamplesToMs`.
private func samplesMs(_ samples: Int) -> Int {
audioSamplesToMs(rateHz: rateHz, channels: channels, samples: samples)
}
/// Tell the ring how long one audio frame actually is, in microseconds
/// (`punktfunk_connection_audio_frame_us`). Mirrors `JitterPolicy::set_frame_us`.
///
/// Two of this ring's decisions are denominated in FRAMES rather than milliseconds — the floor
/// under the effective target (a device quantum plus one frame) and the smooth shed (drop
/// exactly one frame) — and both were written when `frameMS` was the only frame this protocol
/// had. The lossless plane negotiates shorter ones: 4 ms at 48 kHz/24-bit, 2 ms at
/// 96 kHz/24-bit under the default MTU. Left unset, a 96 kHz session would shed two and a half
/// frames at a time and fade across an entire frame.
///
/// A SETTER rather than an initialiser parameter, exactly as core has it: the default keeps
/// every Opus session — and every test in `AudioRingDriftTests`, which pins the pre-hi-res
/// numbers — bit-identical, and a caller that never learned the figure cannot accidentally pass
/// a wrong one. Idempotent, so the engine-rebuild path that reuses a live ring can simply call
/// it again.
///
/// Clamped to ≥ 1 µs so a degenerate value can never make `frameSamples` zero and turn the
/// shed into an infinite no-op.
func setFrameUs(_ us: Int) {
lock.lock()
defer { lock.unlock() }
frameUs = max(us, 1)
}
/// One frame in interleaved samples. Computed in µs so a sub-millisecond frame does not
/// truncate: 2 500 µs at 48 kHz stereo is 240 samples, not the 192 that routing it through
/// integer milliseconds first would give. Mirrors `JitterPolicy::frame_samples`; caller holds
/// the lock.
///
/// Delegated to `audioSamplesPerFrame` — the mirror of core's `pcm::samples_per_frame` — rather
/// than re-derived from this ring's own ms↔sample conversion, because a second derivation is a
/// second rounding. The two are only interchangeable when the rate divides the frame: at
/// 44 100 Hz a 5 ms frame is 220 samples PER CHANNEL, and `msSamples(5)` would say 441 where
/// the wire delivers 440. The shed and the near-miss margin both mean "exactly one packet", so
/// a self-derived answer would put them one sample away from the packet they describe.
private var frameSamples: Int {
max(audioSamplesPerFrame(rateHz: rateHz, frameUs: frameUs, channels: channels), 1)
}
/// The seam crossfade, capped at HALF a frame. `crossfadeMS`'s flat 2 ms is a comfortable slice
/// of a 5 ms Opus frame and the whole of a 2 ms lossless one — and a fade as long as the
/// material it is fading is not a crossfade, it is a wholesale replacement of the seam with a
/// ramp. Mirrors `JitterPolicy::crossfade_samples`; caller holds the lock.
private var crossfadeSamples: Int { min(msSamples(Self.crossfadeMS), frameSamples / 2) }
/// The two frame-denominated quantities, taken under the lock — for `AudioRingDriftTests`,
/// which pins them the same way core's `the_shed_follows_the_negotiated_frame_length` pins its
/// side. Locked, unlike the computed properties it wraps: those are only ever read from paths
/// that already hold it.
var frameGeometry: (frame: Int, crossfade: Int) {
lock.lock()
defer { lock.unlock() }
return (frameSamples, crossfadeSamples)
}
/// Effective target depth in interleaved samples: the (adaptively grown) live target, lifted
@@ -199,9 +378,9 @@ final class AudioRing: @unchecked Sendable {
/// oversized read would otherwise inflate the debt threshold forever and turn the very next
/// late packet into a full re-prime.
private func target(lift quantum: Int) -> Int {
let floor = max(targetLive, quantum + Self.frameMS * perMS)
let floor = max(targetLive, quantum + frameSamples)
guard let want = syncTarget else { return floor }
let cap = max(Self.hardCapMS * perMS, floor)
let cap = max(msSamples(Self.hardCapMS), floor)
return min(max(want, floor), cap)
}
@@ -273,7 +452,7 @@ final class AudioRing: @unchecked Sendable {
// Rust policy's `.max(target + want)`) or a large-quantum device would trim itself into
// a permanent underrun.
let cap = max(
min(target + Self.headroomMS * perMS, Self.hardCapMS * perMS),
min(target + msSamples(Self.headroomMS), msSamples(Self.hardCapMS)),
target + renderQuantum)
if writeIdx - readIdx > cap {
// Crossfaded, like the smooth shed — see `dropFront`. This is the correction a
@@ -293,7 +472,7 @@ final class AudioRing: @unchecked Sendable {
// Depth average, weighted by the callback size so its time constant is independent of the
// device quantum.
let alpha = min(1.0, Double(count) / Double(Self.ewmaTauMS * perMS))
let alpha = min(1.0, Double(count) / Double(msSamples(Self.ewmaTauMS)))
depthAvg += (Double(available) - depthAvg) * alpha
if !primed {
@@ -317,13 +496,13 @@ final class AudioRing: @unchecked Sendable {
// this instant: a single late packet empties the ring for a callback without making it
// hollow, and must keep the consecutive-empties hysteresis. Lifted by THIS callback's
// size, not the high-water quantum — see `target(lift:)`.
hollow = depthAvg + Double(Self.deprimeDebtMS * perMS) < Double(target(lift: count))
hollow = depthAvg + Double(msSamples(Self.deprimeDebtMS)) < Double(target(lift: count))
// Drift correction: shed exactly one frame, crossfaded, once the AVERAGE has sat above
// the threshold for the sustain window. Anything shorter is jitter and must be left alone.
if depthAvg > Double(target + Self.shedExcessMS * perMS) {
if depthAvg > Double(target + msSamples(Self.shedExcessMS)) {
overRun += count
if overRun >= Self.shedSustainMS * perMS {
if overRun >= msSamples(Self.shedSustainMS) {
overRun = 0
shedOneFrame()
shedCount += 1
@@ -344,7 +523,20 @@ final class AudioRing: @unchecked Sendable {
}
// Near-miss: served in full, but with less than one frame left over — the next callback
// starves unless a packet lands within one frame time.
nearMiss = n == count && writeIdx - readIdx < Self.nearMissMarginMS * perMS
//
// Denominated in the RESOLVED frame, not a fixed 5 ms: against a 2 ms lossless frame a
// frozen margin stops meaning "one packet in hand" and starts meaning two and a half, so it
// would grow the target on a ring that was never close to starving — inverting the thing it
// exists to detect. Identical on every Opus session.
//
// (This used to be flagged here as a deliberate divergence, with core still measuring
// against a `NEAR_MISS_MARGIN_MS` constant and a note that core should follow. It has:
// `JitterPolicy::step` now compares against `frame_samples()` too, and its own
// `the_near_miss_margin_is_one_negotiated_frame` pins it. The two policies agree again.)
//
// ⚠ `frameSamples` is the WIRE's frame — floored per channel — not `msSamples(frameMS)`.
// The margin means "exactly one packet", and at 44 100 Hz those two are 440 and 441.
nearMiss = n == count && writeIdx - readIdx < frameSamples
noteRead(ranShort: n < count, count: count)
}
@@ -356,7 +548,7 @@ final class AudioRing: @unchecked Sendable {
/// doesn't cost latency for the rest of the session. Caller holds the lock.
private func noteRead(ranShort: Bool, count: Int) {
windowRun += count
if windowRun >= Self.growWindowMS * perMS {
if windowRun >= msSamples(Self.growWindowMS) {
windowRun = 0
underrunsInWindow = 0
nearMissGrown = false
@@ -375,7 +567,7 @@ final class AudioRing: @unchecked Sendable {
// is no longer at, so growing past the proven target on top would overshoot.
probeRun = 0
targetLive = max(targetLive, probePrevTarget)
syncBackoffRun = syncBackoffLenMS * perMS
syncBackoffRun = msSamples(syncBackoffLenMS)
syncBackoffLenMS = min(syncBackoffLenMS * 2, Self.syncBackoffMaxMS)
restored = true
} else if probeRun == 0 {
@@ -393,7 +585,7 @@ final class AudioRing: @unchecked Sendable {
// Both, because either alone is wrong at one end of the quantum range: time alone is a
// hair trigger on a device whose single quantum already exceeds the window, and a
// callback count alone is the device-dependent fuse this replaced.
let starved = emptyRun >= Self.deprimeMS * perMS
let starved = emptyRun >= msSamples(Self.deprimeMS)
&& emptyReads >= Self.minDeprimeCallbacks
if starved || hollow {
// The starvation hysteresis protects a FULL ring from one late packet.
@@ -411,7 +603,7 @@ final class AudioRing: @unchecked Sendable {
if underrunsInWindow >= Self.growUnderruns {
underrunsInWindow = 0
windowRun = 0
targetLive = min(targetLive + Self.growStepMS * perMS, Self.maxTargetMS * perMS)
targetLive = min(targetLive + msSamples(Self.growStepMS), msSamples(Self.maxTargetMS))
}
} else if nearMiss {
// Came within one frame of an underrun — the same evidence as one, heard by no one.
@@ -425,7 +617,7 @@ final class AudioRing: @unchecked Sendable {
emptyRun = 0
if !nearMissGrown, !restored {
nearMissGrown = true
targetLive = min(targetLive + Self.growStepMS * perMS, Self.maxTargetMS * perMS)
targetLive = min(targetLive + msSamples(Self.growStepMS), msSamples(Self.maxTargetMS))
}
} else {
emptyReads = 0
@@ -439,20 +631,22 @@ final class AudioRing: @unchecked Sendable {
// above), and a failed sync-driven guess is not retried for a backoff.
let syncShrink = syncWantsLess && syncBackoffRun == 0
let quietNeeded = syncShrink ? Self.shrinkQuietSyncMS : Self.shrinkQuietMS
if quietRun >= quietNeeded * perMS {
if quietRun >= msSamples(quietNeeded) {
quietRun = 0
let prev = targetLive
targetLive = max(targetLive - Self.growStepMS * perMS, Self.targetMS * perMS)
targetLive = max(targetLive - msSamples(Self.growStepMS), msSamples(Self.targetMS))
if targetLive < prev {
probeRun = Self.shrinkProbeMS * perMS
probeRun = msSamples(Self.shrinkProbeMS)
probePrevTarget = prev
}
}
}
}
/// Drop one protocol frame from the front — the smooth drift correction.
private func shedOneFrame() { dropFront(Self.frameMS * perMS) }
/// Drop one audio frame from the front — the smooth drift correction. The session's REAL frame
/// (`setFrameUs`), so a lossless session sheds its own 2–4 ms rather than two and a half of
/// them.
private func shedOneFrame() { dropFront(frameSamples) }
/// Drop `drop` interleaved samples from the front, linearly crossfading the seam so the
/// correction is inaudible rather than a click. Mirrors `punktfunk_core::audio::crossfade_drop`;
@@ -463,10 +657,13 @@ final class AudioRing: @unchecked Sendable {
/// ARRIVALS, not the samples either side of the seam, which are ordinary continuous audio. It
/// is also the drop that actually fires here: a bunching Wi-Fi link trims far more often than
/// drift sheds, so the one path left unfaded was the audible one.
///
/// The fade is `crossfadeSamples` — capped at half a frame — then clamped again to what this
/// particular drop can actually spare on either side of the seam.
private func dropFront(_ drop: Int) {
let available = writeIdx - readIdx
guard drop > 0, available > drop else { return }
let fade = min(Self.crossfadeMS * perMS, min(drop, available - drop))
let fade = min(crossfadeSamples, min(drop, available - drop))
let capacity = buf.count
if fade > 0 {
// The tail of what we discard fades out into the head of what survives.
@@ -485,7 +682,7 @@ final class AudioRing: @unchecked Sendable {
var bufferedMS: Int {
lock.lock()
defer { lock.unlock() }
return (writeIdx - readIdx) / max(perMS, 1)
return samplesMs(writeIdx - readIdx)
}
/// One consistent snapshot of the ring's vitals, taken under a single lock so the numbers in
@@ -511,8 +708,8 @@ final class AudioRing: @unchecked Sendable {
lock.lock()
defer { lock.unlock() }
return Stats(
bufferedMS: (writeIdx - readIdx) / max(perMS, 1),
targetMS: target / max(perMS, 1),
bufferedMS: samplesMs(writeIdx - readIdx),
targetMS: samplesMs(target),
underruns: underrunCount,
sheds: shedCount,
avOffsetMS: avOffsetMS,
@@ -564,11 +761,17 @@ struct AvSync {
/// would empty or overfill it outright. Beyond this the loop reports and waits rather than acts.
private static let saneLimitMS = 1_000
/// The protocol's frame, in ms — the EWMA is weighted by it so the time constant means the
/// same thing however often the caller observes.
private static let frameMS = 5
/// same thing however often the caller observes. Shares `AudioRing.frameMS`'s caveat: on the
/// lossless plane the real frame is shorter, so observations arrive more often than this
/// weight assumes and the average settles proportionally faster. It is a time constant on a
/// loop with a 100-observation settling gate, so faster is harmless; see that constant.
private static let frameMS = AudioRing.frameMS
/// Interleaved samples per millisecond at the negotiated layout (48 × channels).
private let perMS: Int
/// The negotiated layout, in the same two numbers `AudioRing` keeps and for the same reason —
/// this type's proposal is denominated in the ring's own units, so the two have to agree about
/// what a millisecond is down to the sample.
private let rateHz: Int
private let channels: Int
/// EWMA of the measured offset in ns. Positive = audio is scheduled to play LATE relative to
/// the picture it belongs with.
private var offsetAvgNs: Double = 0
@@ -576,9 +779,19 @@ struct AvSync {
/// Set once an observation lands outside `saneLimitMS`, for reporting.
private(set) var implausible = false
/// `channels` is the negotiated interleaved channel count (2/6/8).
init(channels: Int) {
perMS = 48 * max(channels, 1)
/// `channels` is the negotiated interleaved channel count (2/6/8), `rateHz` the negotiated
/// sample rate — every rate on the lossless ladder, exactly, for the reason `AudioRing.init`
/// gives: the ms ⇄ sample conversion multiplies before it divides, so the 44.1 kHz family is
/// representable here too and the depth this type proposes lands in the units the ring measures
/// itself in. Mirrors `AvSync::new_at_rate`.
init(channels: Int, rateHz: Int) {
self.rateHz = max(rateHz, 1)
self.channels = max(channels, 1)
}
/// Interleaved samples to whole milliseconds — see `audioSamplesToMs`.
private func samplesMs(_ samples: Int) -> Int {
audioSamplesToMs(rateHz: rateHz, channels: channels, samples: samples)
}
/// One measurement handed to `observe`. Every field is in the units its source already
@@ -617,7 +830,11 @@ struct AvSync {
// When this frame's samples will actually reach the speaker, expressed in the host's
// capture clock — the same clock, and the same shape, as the video figure it is compared
// against.
let bufferedNs = Int64(o.bufferedAhead / max(perMS, 1)) * 1_000_000
// Deliberately still rounded to whole MILLISECONDS rather than converted straight to ns: it
// keeps every 48/96 kHz session bit-identical to the shipped behaviour, and the ≤ 1 ms it
// discards is an order of magnitude inside `deadbandMS`, which is the resolution this loop
// acts on at all. The conversion itself is now exact at every rate.
let bufferedNs = Int64(samplesMs(o.bufferedAhead)) * 1_000_000
// Overflow-reporting arithmetic, NOT the wrapping `&+`/`&-` the meters use. Every term is
// a nanosecond count on the same epoch (~1.8e18), so the DIFFERENCE is tiny while the
// operands sit within a factor of five of `Int64.max` — and a garbage `pts_ns` would wrap
@@ -661,7 +878,14 @@ struct AvSync {
guard settled else { return nil }
let offsetMs = offsetAvgNs / 1_000_000
guard abs(offsetMs) >= Double(Self.deadbandMS) else { return nil }
let delta = Int(offsetMs * Double(perMS))
// One millisecond of samples as a float, so a fractional offset scales smoothly. The
// division is done on the CONSTANT, not on the product: `x * 96000.0 / 1000.0` rounds twice
// and can land one ulp — and so one sample — away from the `x * 96.0` every shipped 48 kHz
// session computes today. This way it is exactly 96.0 / 192.0 there, and correctly 88.2 at
// 44 100 Hz stereo, where the old integer `perMS` said 88 and steered every correction
// 0.23 % short. Mirrors `AvSync::desired_depth`.
let perMs = Double(audioInterleavedPerSec(rateHz: rateHz, channels: channels)) / 1_000
let delta = Int(offsetMs * perMs)
return max(0, currentDepth - delta)
}
}
@@ -681,51 +905,91 @@ struct AvSync {
/// wall-clock in.
///
/// So a drought that is draining the ring gets concealed too, from the same decoder state
/// (`PunktfunkConnection.audioPlc`), for a bounded time. Denominated in TIME, never in frames or
/// callbacks: that is the recorded lesson from the very fuse this protects, where a count gave an
/// iPad a third of a Mac's slack (`AudioRing.deprimeMS`, and
/// `testDeprimeFuseIsADurationNotACallbackCount`).
/// (`PunktfunkConnection.audioPlc`), for a bounded time. The BOUND is denominated in TIME, never in
/// frames or callbacks: that is the recorded lesson from the very fuse this protects, where a count
/// gave an iPad a third of a Mac's slack (`AudioRing.deprimeMS`, and
/// `testDeprimeFuseIsADurationNotACallbackCount`). What it COUNTS is frames — one per synthesized
/// packet, which is what the drain thread actually produces — and the resolved frame length is what
/// converts between the two. Those are the same discipline, not opposite ones: the policy is stated
/// in time and the conversion is exact, instead of a frame being assumed to be 5 ms.
///
/// Time is passed IN, so the policy stays as deterministic as the ring's own.
struct DroughtConceal {
/// A drought must outlast ordinary arrival jitter before anything is synthesized for it: two
/// protocol frames, the same tolerance the host's capture-hole infill uses at the other end.
private static let afterMS = 2 * AudioRing.frameMS
/// …and the ring must actually be running out. A drought a deep ring can cover is not audible,
/// and concealing it would synthesize audio the late packets are about to duplicate — pushing
/// the whole stream later and handing the drift shed a mess to clean up audibly.
private static let floorMS = 2 * AudioRing.frameMS
/// Concealed since the last real packet.
private var concealedMS = 0
/// Frames concealed since the last real packet. Counted in FRAMES rather than milliseconds
/// because that is what the drain thread actually does — one `audioPlc()` frame per `conceal()`
/// that says yes — and because the frame is no longer a fixed 5 ms; see `init(maxMS:frameUs:)`.
private var concealed = 0
private let maxMS: Int
/// Concealed over the session — what the 10 s `plc_ms=` line reports. Concealment must be
/// visible: a policy that quietly papers over a failing link is a policy that hides the bug.
private(set) var totalMS = 0
/// One frame, in MICROSECONDS. Everything time-denominated here derives from it.
private let frameUs: Int
/// Concealed over the session, in FRAMES.
private var total = 0
/// At the protocol's default frame (`AudioRing.frameMS`) — every Opus session, and every test
/// that pins the pre-hi-res numbers.
init(maxMS: Int) {
self.maxMS = maxMS
self.init(maxMS: maxMS, frameUs: AudioRing.frameMS * 1_000)
}
/// At an explicitly negotiated frame length (`punktfunk_connection_audio_frame_us`).
///
/// This type charges one frame per concealed frame and bounds itself in WALL-CLOCK
/// milliseconds, so the two have to agree about how long a frame is. They did not: the frame was
/// assumed to be 5 ms, and on a 2 ms lossless frame that made the `maxMS` budget run out after
/// two fifths of the time it is meant to buy, with the reported `plc_ms` two and a half times
/// too high — and on the 1 ms frame a 5.1 session negotiates, a fifth and five times. The frame
/// COUNT was always right (it charged 5 and divided by 5), which is exactly why this went
/// unnoticed: the load-bearing number was fine and only the two human-facing ones were wrong.
/// Mirrors `DroughtConceal::new_at_frame_us`.
init(maxMS: Int, frameUs: Int) {
self.maxMS = maxMS
self.frameUs = max(frameUs, 1)
}
/// How long a drought must last before it is concealed at all — TWO FRAMES, so an ordinary
/// inter-packet gap is never mistaken for a stall. It was a fixed `2 × frameMS`, which on a 2 ms
/// lossless frame waits five frames instead of two before conceding there is a stall.
///
/// ⚠ In whole milliseconds, because that is the granularity the caller measures the quiet wire
/// at (core compares `Duration`s in µs). Every rung on the ladder is a multiple of 500 µs, so
/// `2 × frameUs` is always a whole number of ms and nothing truncates; the floor of 1 exists
/// only so a degenerate `frameUs` cannot produce a zero-length tolerance, which would conceal
/// ordinary jitter as though it were a stall.
private var afterMS: Int { max(2 * frameUs / 1_000, 1) }
/// Ring depth below which a drought is worth concealing, in ms — also two frames. A drought a
/// deep ring can cover is not audible, and concealing it would synthesize audio the late packets
/// are about to duplicate, pushing the whole stream later and handing the drift shed a mess to
/// clean up audibly. Rounds UP, like core's `div_ceil`, so the floor is never *less* than the
/// two frames it promises.
private var floorMS: Int { (2 * frameUs + 999) / 1_000 }
/// Concealed since the last real packet, in ms — the figure the `maxMS` budget bounds.
private var concealedMS: Int { concealed * frameUs / 1_000 }
/// Concealment over the session, ms — what the 10 s `plc_ms=` line reports. Concealment must be
/// visible: a policy that quietly papers over a failing link is a policy that hides the bug.
var totalMS: Int { total * frameUs / 1_000 }
/// A packet arrived, ending any drought — the next one starts from a full budget.
///
/// The Rust twin also hands back the frames it concealed, for its caller to subtract from the
/// loss concealment the seq path is about to ask for. Here that subtraction is core's, on the
/// far side of the ABI, because that is where the gap tracker lives (see
/// `punktfunk_connection_audio_plc`) — a packet genuinely lost inside a covered drought must
/// not be concealed twice either way.
/// Nothing to divide: the run is a frame count, so ending it is one assignment. The Rust twin
/// hands that count BACK, for its caller to subtract from the loss concealment the seq path is
/// about to ask for. Here that subtraction is core's, on the far side of the ABI, because that
/// is where the gap tracker lives (see `punktfunk_connection_audio_plc`) — a packet genuinely
/// lost inside a covered drought must not be concealed twice either way.
mutating func packet() {
concealedMS = 0
concealed = 0
}
/// Should one more frame be concealed? `depthMS` is the playout ring as the render callback
/// last left it.
mutating func conceal(sinceLastPacketMS: Int, depthMS: Int) -> Bool {
if sinceLastPacketMS < Self.afterMS || depthMS > Self.floorMS || concealedMS >= maxMS {
if sinceLastPacketMS < afterMS || depthMS > floorMS || concealedMS >= maxMS {
return false
}
concealedMS += AudioRing.frameMS
totalMS += AudioRing.frameMS
concealed += 1
total += 1
return true
}
}
clients/apple/Sources/PunktfunkKit/Audio/OpusCodec.swift
+8
View File
@@ -5,6 +5,14 @@
// AVAudioConverter handles both as single-packet AVAudioCompressedBuffers with explicit
// packet descriptions.
//
// ⚠ **The 48 kHz in this file is CORRECT and must not be "fixed" for hi-res.** Opus is 48 kHz by
// construction — it has no other internal rate — which is the whole reason the lossless plane
// exists (design/hi-res-audio.md §2). Nothing on that plane comes through here: `0xD3` frames are
// unpacked in core and arrive as f32 via `nextAudioPcm`, and the only surviving user of this file
// in the app is `OpusEncoder` on the mic UPLINK, which §3 keeps at 48 kHz deliberately (voice,
// 10 ms frames, unchanged). `OpusDecoder` is exercised by the codec tests; the playback path stopped
// using it when decoding moved into core.
//
// Both classes are single-threaded by contract (one per direction, owned by their
// drain/capture pipelines).
clients/apple/Sources/PunktfunkKit/Audio/SessionAudio.swift
+133 -24
View File
@@ -1,15 +1,24 @@
// Session audio, both directions:
//
// host → speaker: a drain thread pulls Opus packets (nextAudio, its own plane in the
// core), decodes via OpusDecoder, and writes PCM into a jitter ring; an
// AVAudioSourceNode pulls from the ring (silence on underrun with re-priming, so a
// network gap costs one dip, not permanent crackle).
// host → speaker: a drain thread pulls audio packets off their own plane in the core, which
// decodes them there (nextAudioPcm) and hands back interleaved f32, and writes that into a
// jitter ring; an AVAudioSourceNode pulls from the ring (silence on underrun with re-priming,
// so a network gap costs one dip, not permanent crackle).
//
// mic → host: a tap on the input node folds the capture to one mono bus (the chosen channel
// of a multi-channel interface, or a sum of all channels), resamples to 48 kHz mono, slices
// 10 ms chunks, Opus-encodes, and sendMic()s each packet — the host feeds them into a
// virtual PipeWire source.
//
// The downlink's FORMAT is negotiated, not assumed. `connection.resolvedAudioRateHz` is 48 kHz
// for every Opus session and every host older than the lossless plane, and any rate on the
// lossless ladder (44 100 / 48 000 / 88 200 / 96 000 / 176 400) on `0xD3` — and it is what the
// ring, the A/V sync loop and the render graph's AVAudioFormat are all built from
// (design/hi-res-audio.md §9). Its CHANNEL count is negotiated the same way and is no longer
// stereo on the lossless plane either: the frame ladder is sized per channel count, so a 5.1/7.1
// lossless session simply arrives on a shorter frame. The UPLINK is deliberately untouched: Opus
// is 48 kHz by construction and the mic carries voice, so §3 excludes it.
//
// Engine topology. With the mic enabled and echo cancellation on (both defaults), BOTH
// directions run on ONE AVAudioEngine with the system voice processor engaged
// (`setVoiceProcessingEnabled`) — AEC needs render and capture on the same unit so it can
@@ -216,11 +225,34 @@ public final class SessionAudio {
#endif
}
/// The rate the samples on the wire are actually at — `Welcome`'s RESOLVED figure, not what
/// this client asked for. 48 000 on every Opus session and every host older than the lossless
/// plane; any rate on the ladder (44 100 / 48 000 / 88 200 / 96 000 / 176 400) on `0xD3`.
/// Everything that turns samples into time — the ring's ms ⇄ sample conversion, the A/V sync
/// loop's, and the `AVAudioFormat` the render graph is built at — is denominated in this,
/// because it is what `nextAudioPcm` hands back.
private var wireRateHz: Int { Int(connection.resolvedAudioRateHz) }
/// How much audio one datagram carries, in MICROSECONDS — `Welcome`'s resolved
/// `audio_frame_us`. 5 000 on every Opus session; on the lossless plane the host sizes it so
/// the payload fits one datagram, which is 4 000 at 48 kHz/24-bit stereo, 2 000 at
/// 96 kHz/24-bit stereo, and shorter again for surround (a 5.1 frame carries three times the
/// samples, so it drops to roughly 1 000–1 500). Microseconds because the ladder has
/// sub-millisecond rungs (`AudioRing.setFrameUs`).
private var wireFrameUs: Int { Int(connection.resolvedAudioFrameUs) }
/// The same figure rounded UP to whole milliseconds, for the one consumer that can only express
/// itself in them: the drain thread's poll timeout. Rounding up rather than down keeps it "at
/// most one frame" — a 2 500 µs frame polls at 3 ms, never 2, so the loop cannot spin a wake-up
/// per frame for nothing. Never below 1.
private var wireFrameMS: Int { max(1, (wireFrameUs + 999) / 1000) }
#if !os(macOS)
/// Route + policy live in the session, not per-engine: stereo playback, mic capture when
/// enabled, Bluetooth allowed. Failure is non-fatal (defaults). Runs on `sessionQueue`.
private func activateAudioSession(micEnabled: Bool) {
let session = AVAudioSession.sharedInstance()
let wanted = Double(wireRateHz)
do {
#if os(iOS)
if micEnabled {
@@ -272,22 +304,32 @@ public final class SessionAudio {
// measures the fuse in ms), but there is still no reason to ask for a quantum
// finer than the packets we send.
try? session.setPreferredIOBufferDuration(0.010)
try? session.setPreferredSampleRate(48_000)
} else {
try session.setCategory(.playback, mode: .default, options: [.mixWithOthers])
}
#else // tvOS — no app-accessible mic
try session.setCategory(.playback, mode: .default, options: [.mixWithOthers])
#endif
// The session's rate, asked for on EVERY branch — the `.playback` ones (mic off, and
// all of tvOS) used to ask for nothing at all, which was invisible while the answer
// was always 48 kHz and is the difference between real and resampled hi-res now. Set
// BEFORE `setActive`: the hardware is configured on activation, and a preference
// expressed after it only takes effect at the next route change.
//
// Best-effort by API contract, and genuinely refused in practice — a Bluetooth route
// has no 96 kHz mode to give (§9's iOS caveat). Which is why nothing downstream reads
// this back as permission: `noteOutputRate` checks what the graph was ACTUALLY built
// on, and the honest statement is made there.
try? session.setPreferredSampleRate(wanted)
try session.setActive(true)
// What we were actually GRANTED, not what we asked for. Both are best-effort, and the
// ring's behaviour depends on the quantum it really gets — without this, a report of
// What we were actually GRANTED, not what we asked for. All three are best-effort, and
// the ring's behaviour depends on the quantum it really gets — without this, a report of
// audio jitter arrives with no way to tell a 10 ms session from a 5 ms or a 23 ms one,
// which is exactly the gap that made the last round of this take a simulation to close.
log.info("""
AVAudioSession active: io_buffer_ms=\
\(session.ioBufferDuration * 1000, format: .fixed(precision: 2)) \
sample_rate=\(Int(session.sampleRate)) \
sample_rate=\(Int(session.sampleRate)) wire_rate=\(Int(wanted)) \
route=\(session.currentRoute.outputs.first?.portType.rawValue ?? "none")
""")
#if os(iOS)
@@ -923,28 +965,45 @@ public final class SessionAudio {
-> (ring: AudioRing, source: AVAudioSourceNode, format: AVAudioFormat)?
{
// Build the playback layout from the host-RESOLVED channel count (never the request):
// 2 = stereo / 6 = 5.1 / 8 = 7.1, canonical wire order FL FR FC LFE RL RR SL SR.
// 2 = stereo / 6 = 5.1 / 8 = 7.1, canonical wire order FL FR FC LFE RL RR SL SR. Same rule
// for the rate — `resolvedAudioRateHz`, never the 96 kHz this client may have asked for.
let channels = Int(connection.resolvedAudioChannels)
// 1 s interleaved capacity, scaled by the channel count. The de-jitter depth itself is
let rateHz = wireRateHz
// One SECOND of interleaved capacity at the session's format. The de-jitter depth itself is
// the ring's own business now (`AudioRing.targetMS`, mirroring `JitterTuning::COREAUDIO`)
// rather than a prefill passed in here.
stateLock.lock()
let ring = self.ring ?? AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = self.ring ?? AudioRing(seconds: 1, channels: channels, rateHz: rateHz)
self.ring = ring
stateLock.unlock()
// The session's REAL frame, which the ring cannot know at construction and must not assume:
// the shed drops exactly one frame and the target floor is a device quantum plus one, so a
// ring left on the 5 ms default sheds two and a half frames at a time on a 96 kHz session
// and fades across a whole one. Idempotent, so the rebuild path that reuses this very ring
// simply sets it again.
ring.setFrameUs(wireFrameUs)
// Engine-native deinterleaved float; the render block deinterleaves from the ring. Surround
// uses an explicit wire-order channel layout; the mixer downmixes to the output device when
// it has fewer speakers (e.g. an iPhone's stereo built-ins). (Explicit if/else rather than
// map/flatMap so it's correct whether the channelLayout initializer is failable or not.)
//
// The rate here describes the SAMPLES, not the hardware: it is the rate `nextAudioPcm`
// hands them back at, and the engine converts from it to whatever the output device runs
// at. Declaring the device's rate instead would play a 96 kHz stream at half speed — which
// is why the honesty check about a device that refused the rate (`noteOutputRate`) reports
// rather than re-formats. Resampling in the mixer is the fallback; claiming hi-res while it
// happens is the thing §9 forbids.
let rate = Double(rateHz)
var format: AVAudioFormat?
if channels == 2 {
format = AVAudioFormat(standardFormatWithSampleRate: 48_000, channels: 2)
format = AVAudioFormat(standardFormatWithSampleRate: rate, channels: 2)
} else if let layout = wireChannelLayout(channels: channels) {
format = AVAudioFormat(standardFormatWithSampleRate: 48_000, channelLayout: layout)
format = AVAudioFormat(standardFormatWithSampleRate: rate, channelLayout: layout)
}
guard let format else {
log.error("could not build \(channels)-channel audio format — audio disabled")
log.error(
"could not build \(channels)-channel \(rateHz) Hz audio format — audio disabled")
return nil
}
let scratch = ScratchBuffer() // block-owned; freed with the closure
@@ -966,6 +1025,35 @@ public final class SessionAudio {
return (ring, source, format)
}
/// Say — out loud, in the log — what rate this engine is REALLY rendering at, and whether it is
/// the one the session negotiated. Call it after `prepare()`, when the output node has settled
/// on the device's format; on iOS/tvOS that follows the AVAudioSession, on macOS the HAL device.
///
/// This is §9's "never claim a rate you did not get", at the client end. The whole hi-res
/// exercise is contingent on the samples reaching a converter-free path, and every layer here
/// will happily hide a failure to do so: `setPreferredSampleRate` is advisory and a Bluetooth
/// route simply has no 96 kHz mode, `AVAudioEngine`'s mixer resamples silently between any two
/// formats, and the stream keeps playing perfectly. The session would then cost 3.2 Mbps,
/// report 96 kHz on the HUD, and carry nothing above 24 kHz — which is precisely the shape of
/// bug design/hi-res-audio.md §4.3 exists to name, wearing the client's hat instead of the
/// host's. Nothing here re-formats the graph (see `makePlaybackChain`): the samples are what
/// they are, the mixer's conversion is the correct fallback, and the only thing missing was
/// somebody saying so.
private func noteOutputRate(_ engine: AVAudioEngine, wireRateHz: Int) {
let deviceRate = Int(engine.outputNode.outputFormat(forBus: 0).sampleRate)
// 0 = the node has no device yet (a start that is about to fail) — nothing to compare.
guard deviceRate > 0 else { return }
guard deviceRate != wireRateHz else {
log.info("audio output opened at \(wireRateHz) Hz — the negotiated rate")
return
}
log.warning("""
audio output is \(deviceRate) Hz but the session negotiated \(wireRateHz) Hz — the \
engine is resampling. Playback is correct; this session is NOT \(wireRateHz) Hz at the \
speaker, whatever the host resolved
""")
}
private func startPlayback(speakerUID: String) {
guard let (ring, source, format) = makePlaybackChain() else { return }
let engine = AVAudioEngine()
@@ -990,6 +1078,7 @@ public final class SessionAudio {
log.error("playback engine failed to start: \(error.localizedDescription)")
return
}
noteOutputRate(engine, wireRateHz: wireRateHz)
stateLock.lock()
if flag.isStopped {
stateLock.unlock()
@@ -1026,15 +1115,27 @@ public final class SessionAudio {
let videoLatency = syncEnabled ? self.videoLatency : nil
if !syncEnabled { log.info("A/V sync disabled by PUNKTFUNK_NO_AV_SYNC") }
let channels = Int(connection.resolvedAudioChannels)
let rateHz = wireRateHz
// Read on the caller's thread, not inside the closure: `self` is deliberately not captured
// by the drain thread (it holds the connection strongly and self not at all — see
// `deinit`), so anything derived from the connection has to be resolved out here.
let frameUs = wireFrameUs
let frameMS = wireFrameMS
let thread = Thread { [connection, flag, drainDone] in
defer { drainDone.signal() }
var drained = 0
var av = AvSync(channels: channels)
var av = AvSync(channels: channels, rateHz: rateHz)
// WP-C1 — the drought half of concealment. Core heals a SEQ GAP, but only when a later
// packet arrives to reveal it; when the wire simply goes quiet nothing arrives to
// reveal anything, and the ring drains into an underrun and a de-prime whose re-prime
// is a longer artifact than the audio that was missing.
var drought = DroughtConceal(maxMS: AudioRing.plcMaxMS)
//
// Given the SESSION's frame, like the ring: this type spends a wall-clock budget one
// frame at a time, and each `conceal()` that says yes costs exactly one `audioPlc()`
// frame below — so if it assumed 5 ms, a 2 ms lossless session would spend the budget
// in two fifths of the time it promises and report `plc_ms` two and a half times too
// high. A 5.1 session, whose frame drops to ~1 ms, would be five times out.
var drought = DroughtConceal(maxMS: AudioRing.plcMaxMS, frameUs: frameUs)
var lastPacketNs = DispatchTime.now().uptimeNanoseconds
// Something has decoded, so there is both state to conceal from and continuity to
// hold. Until then a session whose host never sends audio keeps the long timeout below
@@ -1050,9 +1151,10 @@ public final class SessionAudio {
do {
// Wait at most one frame WHILE there is a stream to protect: the drought
// decision has to be made on the wire's schedule, not whenever the next packet
// happens to turn up.
// happens to turn up. The SESSION's frame, so a lossless plane sending every
// 2 ms is not judged on a 5 ms clock.
pcm = try connection.nextAudioPcm(
timeoutMs: decoded ? UInt32(AudioRing.frameMS) : 100)
timeoutMs: decoded ? UInt32(frameMS) : 100)
} catch {
return false // session closed
}
@@ -1114,17 +1216,20 @@ public final class SessionAudio {
ring.write(base, count: pcm.frameCount * pcm.channels)
}
}
// Periodic vitals (~10 s at the protocol's 5 ms frames). The other three clients
// log buffer depth and underruns; without this an Apple audio report — latency or
// dropout — arrives with no numbers at all, which is the position every platform
// was in before the 2026-08 audio work. `plc_ms` rides along because a healthy
// Periodic vitals (~10 s at the protocol's 5 ms frames; proportionally sooner on a
// lossless plane, whose frames are 2–4 ms). The other three clients log buffer
// depth and underruns; without this an Apple audio report — latency or dropout —
// arrives with no numbers at all, which is the position every platform was in
// before the 2026-08 audio work. `plc_ms` rides along because a healthy
// `underruns` bought with a climbing `plc_ms` is a link in trouble, not a link
// that is fine.
// that is fine. `rate_hz`/`frame_us` lead it so a field log says which plane the
// session was on, and on what frame the shed and target floor were sized, without
// needing the connect lines above it.
drained += 1
if drained % 2_000 == 0 {
let s = ring.stats
log.info(
"audio: buffer_ms=\(s.bufferedMS) target_ms=\(s.targetMS) underruns=\(s.underruns) drift_sheds=\(s.sheds) av_offset_ms=\(s.avOffsetMS) plc_ms=\(s.plcMS)"
"audio: rate_hz=\(rateHz) frame_us=\(frameUs) buffer_ms=\(s.bufferedMS) target_ms=\(s.targetMS) underruns=\(s.underruns) drift_sheds=\(s.sheds) av_offset_ms=\(s.avOffsetMS) plc_ms=\(s.plcMS)"
)
}
return true
@@ -1250,6 +1355,10 @@ public final class SessionAudio {
let muted = micMuted // latched before this engine existed (a mute during the prompt)
stateLock.unlock()
apply(micMuted: muted, capture: nil, combined: engine)
// Worth its own read on this path rather than only the plain one: the voice processor picks
// its OWN formats when it engages (that is why the mic tap reads them after `prepare()`),
// and a VPIO unit is the least likely thing in the graph to have honoured a 96 kHz request.
noteOutputRate(engine, wireRateHz: wireRateHz)
startDrain(into: ring)
log.info("audio engines joined — voice processing (echo cancellation) active")
}
clients/apple/Sources/PunktfunkKit/Connection/PunktfunkConnection.swift
+121 -12
View File
@@ -452,6 +452,56 @@ public final class PunktfunkConnection {
/// PCM from `nextAudioPcm` is interleaved in the canonical wire order FL FR FC LFE RL RR SL SR.
public private(set) var resolvedAudioChannels: UInt8 = 2
/// The sample rate the host RESOLVED for this session: `48000` on every Opus session and every
/// host older than the lossless plane, or the rate a hi-res session actually landed on — which
/// may be LOWER than `audioRateHz` asked for, because the host runs a five-condition gate
/// (design/hi-res-audio.md §8.4) and any failure resolves the session back to Opus at 48 kHz.
///
/// **Open the output device and size the jitter ring from THIS, never from the request.**
/// Opening at 96 kHz because we asked for 96 kHz, when the host answered 48 kHz, is §4.3's
/// failure repeated at the client end: everything audits clean and the content is wrong. The
/// samples `nextAudioPcm` hands back are at this rate, so it is also what every ms↔sample
/// conversion in `AudioRing`/`AvSync` has to be denominated in.
public private(set) var resolvedAudioRateHz: UInt32 = 48_000
/// The sample depth the host resolved: `16` on every Opus session and every older host, `16` or
/// `24` on the lossless plane. Reporting only — `nextAudioPcm` unpacks either depth in core and
/// hands back f32 regardless. It exists so a UI can state the format HONESTLY; a client that
/// says "24-bit" while the host declined is the same class of lie as claiming a rate it did not
/// get. Which PLANE a session runs is `hostCaps & PUNKTFUNK_HOST_CAP_AUDIO_HIRES`, not this:
/// 48 kHz/16-bit reads identically on both.
public private(set) var resolvedAudioBits: UInt8 = 16
/// How much audio one datagram carries, in MICROSECONDS — `5000` on the Opus plane and every
/// host older than the lossless one; on `0xD3` the host picks the longest rung whose payload
/// fits one datagram, which is `4000` at 48 kHz/24-bit stereo and `2000` at 96 kHz/24-bit
/// stereo under the default MTU, and shorter again for surround, whose frame carries three
/// (5.1) or four (7.1) times the samples for the same duration.
///
/// Microseconds, not milliseconds, because the ladder has sub-millisecond rungs and a frame
/// that goes through integer ms truncates: 2 500 µs at 48 kHz stereo is 240 interleaved
/// samples, and 2 ms would make it 192.
///
/// ⚠ It is a LABEL, not a duration, on the 44.1 kHz family: a frame carries a whole number of
/// samples per channel and no rung divides 44 100 Hz, so a nominal 5 ms frame there is really
/// 4 988 662 ns. Size buffers from it (`AudioRing.setFrameUs`, which routes it through the same
/// floor-per-channel rule the host filled the frame with); never advance a clock by it.
///
/// Needed only because this client ports the de-jitter policy into Swift rather than letting
/// core run it. Two of the policy's decisions are denominated in FRAMES, not milliseconds — the
/// smooth shed drops exactly one, and the effective-target floor is a device quantum plus one —
/// so a ring compiled against 5 ms sheds two and a half frames at a time on a 96 kHz session.
/// **Not derivable from `nextAudioPcm`**: concealed frames are prepended into the same buffer,
/// so `frameCount` answers "how many samples did I get", not "how long is one frame".
public private(set) var resolvedAudioFrameUs: UInt16 = UInt16(PUNKTFUNK_AUDIO_FRAME_MS * 1000)
/// True when this session resolved the LOSSLESS `0xD3` plane (`HOST_CAP_AUDIO_HIRES`) rather
/// than Opus — the one honest answer to "is this bit-exact?", which the rate and depth alone
/// cannot give (48 kHz/16-bit is a legal resolution on both planes).
public var isLosslessAudio: Bool {
hostCaps & UInt8(PUNKTFUNK_HOST_CAP_AUDIO_HIRES) != 0
}
/// The video codec the host resolved for this session (`Welcome.codec`, `PUNKTFUNK_CODEC_*`):
/// `2` = HEVC (default / older host), `1` = H.264, `4` = AV1, `8` = PyroWave (only when this
/// client opted in). Build the decoder from THIS. The resolved value honors the client's
@@ -733,6 +783,17 @@ public final class PunktfunkConnection {
/// `bitrateKbps`: requested video encoder bitrate (0 = host default; the host clamps
/// to its supported range). Check `resolvedBitrateKbps` afterwards — a speed test
/// (`startSpeedTest`) is how a client picks an informed value.
///
/// `audioRateHz`/`audioBits`: the audio format to ASK for (48 000/16 = today's Opus plane, the
/// default and the only pair that is byte-for-byte identical on the wire to every session
/// before the lossless plane existed). Anything else asks for the bit-exact `0xD3` plane, whose
/// rate ladder is 44 100 / 48 000 / 88 200 / 96 000 / 176 400 (`pcm::rate_is_supported`), and
/// takes 2.1–8.5 Mbps off the top of the link for stereo — three times that for 5.1, four for
/// 7.1 — outside ABR, so it is a deliberate user opt-in on both ends, never a default.
/// **The request is not the answer**: read `resolvedAudioRateHz`/`resolvedAudioBits`/
/// `resolvedAudioChannels` afterwards and open the output device from those. Asking for
/// something the connection cannot carry is not an error — a format whose frame does not fit
/// one datagram, 176 400/24-bit among them, resolves the session back to Opus 48 kHz.
public init(
host: String, port: UInt16 = 9777,
width: UInt32, height: UInt32, refreshHz: UInt32,
@@ -743,6 +804,8 @@ public final class PunktfunkConnection {
bitrateKbps: UInt32 = 0,
videoCaps: UInt8 = 0,
audioChannels: UInt8 = 2,
audioRateHz: UInt32 = 48_000,
audioBits: UInt8 = 16,
videoCodecs: UInt8 = 0x02, // PUNKTFUNK_CODEC_HEVC — the codecs this client can decode
preferredCodec: UInt8 = 0, // 0 = auto; else PUNKTFUNK_CODEC_* soft preference
clientCaps: UInt8 = 0, // ABI v11: PUNKTFUNK_CLIENT_CAP_CURSOR = render the host cursor locally
@@ -766,26 +829,53 @@ public final class PunktfunkConnection {
// device pending approval reads "This device".
let override = deviceName?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
let label = override.isEmpty ? DeviceName.current : override
// `ex11` only when a NON-DEFAULT audio format is being asked for, and this branch is
// LOAD-BEARING rather than a tidiness preference.
//
// ⚠ It used to be commented as one — "48 000/16 through `ex11` is byte-for-byte identical
// to `ex10`, but staying on the older entry point keeps that identity a property of this
// client" — and the C header now says plainly that it is not identical. `ex10` passes
// `0`/`0`, meaning UNSPECIFIED, and core's capability bit keys on "the caller specified a
// format", not on "the format differs from the default". So an explicit 48 000/16 through
// `ex11` is a genuine request for the cheapest LOSSLESS rung: it sets
// `CLIENT_CAP_AUDIO_HIRES`, and a host with the operator policy on resolves the session
// onto the `0xD3` plane at 1.5 Mbps — for audio indistinguishable from the 256 kbps Opus it
// replaced. That is deliberate in core (48/16 would otherwise be the one rung nobody could
// ask for), which is exactly why deleting this test would opt every ordinary session in.
//
// `AudioFormatChoice.opus.wire` is `(48_000, 16)` for readability, so this comparison — not
// that pair — is what keeps a Standard session on the legacy path.
//
// `ex11` also derives `CLIENT_CAP_AUDIO_HIRES` from the format itself and ORs it into
// `clientCaps`, so the bit and the format it advertises can never disagree; nothing here
// sets it by hand.
let wantsHiRes = audioRateHz != 48_000 || audioBits != 16
handle = host.withCString { cs in
withOptionalCString(identity?.certPEM) { cert in
withOptionalCString(identity?.keyPEM) { key in
withOptionalCString(launchID) { launch in
label.withCString { name in
if let pin = pinSHA256 {
return pin.withUnsafeBytes { p in
punktfunk_connect_ex10(
func dial(_ pin: UnsafePointer<UInt8>?) -> OpaquePointer? {
if wantsHiRes {
return punktfunk_connect_ex11(
cs, port, width, height, refreshHz, compositor.rawValue,
gamepad.rawValue, bitrateKbps, videoCaps, audioChannels,
audioRateHz, audioBits,
videoCodecs, preferredCodec, clientCaps, launch,
p.bindMemory(to: UInt8.self).baseAddress, &observed,
cert, key, name, timeoutMs, &connectStatus)
pin, &observed, cert, key, name, timeoutMs, &connectStatus)
}
return punktfunk_connect_ex10(
cs, port, width, height, refreshHz, compositor.rawValue,
gamepad.rawValue, bitrateKbps, videoCaps, audioChannels,
videoCodecs, preferredCodec, clientCaps, launch,
pin, &observed, cert, key, name, timeoutMs, &connectStatus)
}
if let pin = pinSHA256 {
return pin.withUnsafeBytes { p in
dial(p.bindMemory(to: UInt8.self).baseAddress)
}
}
return punktfunk_connect_ex10(
cs, port, width, height, refreshHz, compositor.rawValue,
gamepad.rawValue, bitrateKbps, videoCaps, audioChannels,
videoCodecs, preferredCodec, clientCaps, launch,
nil, &observed, cert, key, name, timeoutMs, &connectStatus)
return dial(nil)
}
}
}
@@ -828,6 +918,24 @@ public final class PunktfunkConnection {
var ac: UInt8 = 2
_ = punktfunk_connection_audio_channels(handle, &ac)
resolvedAudioChannels = ac
// The format the host RESOLVED, which may be below what `audioRateHz`/`audioBits` asked
// for — the five-condition gate declines to Opus 48 kHz rather than failing the connect.
// The defaults survive a status the accessors never fill (an older core), which is the
// right answer: every session such a core can run IS 48 kHz/16-bit.
var rate: UInt32 = 48_000
_ = punktfunk_connection_audio_sample_rate(handle, &rate)
resolvedAudioRateHz = rate
var bits: UInt8 = 16
_ = punktfunk_connection_audio_bits(handle, &bits)
resolvedAudioBits = bits
// `0` is the accessor's "the host stated nothing" — an older host, or an Opus session that
// never needed to say. Both mean the protocol's default frame, so map it here rather than
// letting a zero reach the ring, where it would be a zero-length shed unit.
var frameUs: UInt16 = 0
_ = punktfunk_connection_audio_frame_us(handle, &frameUs)
resolvedAudioFrameUs = frameUs == 0
? UInt16(PUNKTFUNK_AUDIO_FRAME_MS * 1000)
: frameUs
var codec: UInt8 = 2 // PUNKTFUNK_CODEC_HEVC
_ = punktfunk_connection_codec(handle, &codec)
resolvedCodec = codec
@@ -1095,8 +1203,9 @@ public final class PunktfunkConnection {
}
}
/// One decoded audio frame from `nextAudioPcm`: interleaved 32-bit float at 48 kHz, in the
/// canonical wire channel order FL FR FC LFE RL RR SL SR (the first `channels`).
/// One decoded audio frame from `nextAudioPcm`: interleaved 32-bit float at
/// `resolvedAudioRateHz` — 48 kHz on the Opus plane, any rate on the lossless ladder on `0xD3`
/// — in the canonical wire channel order FL FR FC LFE RL RR SL SR (the first `channels`).
public struct AudioPCM: Sendable {
/// Interleaved f32 samples (`frameCount * channels` long), wire channel order.
public let samples: [Float]
clients/apple/Sources/PunktfunkShared/DefaultsKeys.swift
+12
View File
@@ -62,6 +62,18 @@ public enum DefaultsKey {
/// Requested audio channel count: 2 (stereo), 6 (5.1) or 8 (7.1). The host clamps to what it
/// can capture; the resolved count drives the in-core decode + AVAudioEngine layout.
public static let audioChannels = "punktfunk.audioChannels"
/// Requested audio format — `AudioFormatChoice`'s raw value: `"opus"` (the default, and every
/// session before the lossless plane existed), or one of the lossless rows,
/// `"lossless44_1"` / `"lossless48"` / `"lossless88_2"` / `"lossless96"` / `"lossless176_4"`.
///
/// Off by default and deliberately: lossless takes 2.1–8.5 Mbps off the top of the link for
/// stereo (three times that for 5.1, four for 7.1), OUTSIDE the ABR loop that manages the video
/// budget, against the ~256 kbps Opus it replaces — so it must be asked for on both ends
/// (`PUNKTFUNK_AUDIO_HIRES` is the host's half, also off by default). A REQUEST: the host's
/// five-condition gate may resolve the session back to Opus, and
/// `PunktfunkConnection.resolvedAudioRateHz`/`resolvedAudioBits`/`resolvedAudioChannels` are
/// what actually happened.
public static let audioFormat = "punktfunk.audioFormat"
/// Preferred video codec: `"auto"` (host decides), `"hevc"`, `"h264"`, `"av1"`, or
/// `"pyrowave"` (the opt-in wired-LAN wavelet codec — picking it advertises AND prefers it,
/// and forces the session SDR). A soft preference — the host emits it when it can, else
clients/apple/Sources/PunktfunkShared/EffectiveSettings.swift
+104
View File
@@ -28,6 +28,9 @@ public struct EffectiveSettings: Equatable, Sendable {
public var hdrEnabled = true
public var compositor = 0
public var audioChannels = 2
/// An `AudioFormatChoice` raw value. `"opus"` — the default — is byte-for-byte the session
/// every build before the lossless plane ran.
public var audioFormat = AudioFormatChoice.opus.rawValue
public var micEnabled = true
public var echoCancel = true
public var touchMode = "trackpad"
@@ -95,6 +98,7 @@ public struct EffectiveSettings: Equatable, Sendable {
hdrEnabled = bool(DefaultsKey.hdrEnabled, hdrEnabled)
compositor = int(DefaultsKey.compositor, compositor)
audioChannels = int(DefaultsKey.audioChannels, audioChannels)
audioFormat = str(DefaultsKey.audioFormat, audioFormat)
micEnabled = bool(DefaultsKey.micEnabled, micEnabled)
echoCancel = bool(DefaultsKey.echoCancel, echoCancel)
touchMode = str(DefaultsKey.touchMode, touchMode)
@@ -176,6 +180,7 @@ public struct EffectiveSettings: Equatable, Sendable {
if let v = overlay.hdrEnabled { s.hdrEnabled = v }
if let v = overlay.compositor { s.compositor = v }
if let v = overlay.audioChannels { s.audioChannels = v }
if let v = overlay.audioFormat { s.audioFormat = v }
if let v = overlay.micEnabled { s.micEnabled = v }
if let v = overlay.echoCancel { s.echoCancel = v }
if let v = overlay.touchMode { s.touchMode = v }
@@ -227,6 +232,105 @@ public struct EffectiveSettings: Equatable, Sendable {
}
}
// MARK: - Audio format
/// The audio format a session ASKS the host for (`DefaultsKey.audioFormat`) — one choice rather
/// than a free rate/depth pair, so the states that cannot be asked for are unrepresentable rather
/// than merely validated.
///
/// **The ladder is both rate families**, matching `punktfunk_core::audio::pcm::rate_is_supported`:
/// 44 100 / 48 000 / 88 200 / 96 000 / 176 400 Hz. The 44.1 kHz family used to be absent, and for
/// exactly one reason: every buffer figure in the jitter policy — at both ends and in all four
/// clients — was `ms × perMS` with `perMS` an INTEGER number of samples per millisecond, so
/// 44 100 → 44.1 truncated to 44 and put every target, every de-prime fuse and every reported
/// `buffer_ms` 2.3 % low. That arithmetic now multiplies before it divides (see the conversion
/// helpers at the top of `AudioRing.swift`, and `JitterPolicy::new_at_rate`), which is the whole of
/// what §4.1 deferred the family behind (design/hi-res-audio.md §4.1).
///
/// ⚠ **A rate being representable is not a promise that it will be granted**, and this surface must
/// never read as one. The host runs a five-condition gate and any failure resolves the session back
/// to Opus 48 kHz; on top of that the frame has to FIT one QUIC datagram, which the top of the
/// ladder only barely does — 176 400/24-bit stereo is 8.5 Mbps and fits only the shortest rung
/// (1 ms, ~1 069 B), and surround above 48 kHz fits no rung at all. What a session actually got is
/// `PunktfunkConnection`'s `resolvedAudioRateHz`/`resolvedAudioBits`/`resolvedAudioChannels`/
/// `isLosslessAudio`, which is what the HUD shows.
///
/// Lossless at the DEFAULT 48 kHz/16-bit is deliberately not offered: it spends ~1.5 Mbps to sound
/// like the transparent 256 kbps Opus it replaces, and it is the one lossless request whose wire
/// parameters are indistinguishable from a legacy one (it needs `CLIENT_CAP_AUDIO_HIRES` set by
/// hand — see the C ABI's note on that constant). 24-bit is where the plane earns its bandwidth.
public enum AudioFormatChoice: String, CaseIterable, Sendable {
/// Opus 48 kHz — the default, and byte-for-byte the session every earlier build ran.
case opus
/// Bit-exact PCM at 44.1 kHz / 24-bit. ~2.1 Mbps. The CD family's base rate: what an ordinary
/// Windows endpoint or a 44.1 kHz interface reports as its OWN engine rate, and the request
/// that spares such a host a resample, exactly as `lossless48` does on a 48 kHz one.
case lossless441
/// Bit-exact PCM at 48 kHz / 24-bit. ~2.3 Mbps. The honest win even without a hi-res
/// interface: no lossy stage at all, and no double resample on a 48 kHz host.
case lossless48
/// Bit-exact PCM at 88.2 kHz / 24-bit. ~4.2 Mbps — 96 kHz's counterpart in the 44.1 family, and
/// the one to prefer over it on 44.1-derived material, since doubling is exact.
case lossless882
/// Bit-exact PCM at 96 kHz / 24-bit. ~4.6 Mbps, and only real if the host's capture endpoint
/// genuinely runs at 96 kHz — the host declines rather than upsampling, and this client says so
/// rather than claiming a rate its own output device refused.
case lossless96
/// Bit-exact PCM at 176.4 kHz / 24-bit — 8.5 Mbps, and the one row far more likely to be
/// declined than granted. Three things have to go right: the host's bandwidth gate gives audio
/// at most a quarter of the video budget, so the session needs ~34 Mbps of video before it is
/// even considered; a stereo frame fits a QUIC datagram only on the ladder's shortest rung
/// (1 ms — a thousand datagrams a second — at ~1 069 B, so any connection with a smaller
/// datagram declines it) and a surround one fits no rung at all; and this device's output has
/// to open the rate. Offered because it is reachable, not because it is likely — the HUD's
/// `audio lossless …` line is what says which happened.
case lossless1764
/// The stored raw value, falling back to `.opus` for anything a newer build wrote.
///
/// ⚠ **The raw values are shared VERBATIM with `pf_client_core::session::AUDIO_FORMATS` and the
/// Android client's `AUDIO_FORMAT_*`, and must never be renamed.** One profile catalog
/// round-trips through all four clients, and a spelling that differs by a single character
/// fails in the worst possible way: the key is carried through untouched, so the profile keeps
/// "working" on the other client and silently inherits its global default instead. The naming
/// rule is the kHz figure with the decimal point dropped — `lossless48`, `lossless96`, and for
/// the 44.1 family `lossless441` / `lossless882` / `lossless1764`. Left implicit (case name ==
/// raw value) precisely so the two cannot drift apart here; `SharedFoundationTests` pins the
/// resulting strings against the other clients' tables.
public init(setting: String) {
self = AudioFormatChoice(rawValue: setting) ?? .opus
}
/// The `Hello` fields this choice asks for. Anything other than `48 000`/`16` asks core for the
/// `0xD3` plane and lets it derive `CLIENT_CAP_AUDIO_HIRES` from the format, so the bit and the
/// format can never disagree.
///
/// ⚠ `.opus` reads `(48_000, 16)` here because that is what an Opus session runs at — **not**
/// because that pair is a way to ask for it. Core's hi-res entry point treats an explicit
/// 48 000/16 as a real request for the lossless plane's cheapest rung (the unspecified pair is
/// `0`/`0`, which is what the legacy entry point sends). `PunktfunkConnection.init` is where
/// that distinction is enforced — it compares against this pair and dials the legacy entry
/// point instead. Read that comment before changing either side.
public var wire: (rateHz: UInt32, bits: UInt8) {
switch self {
case .opus: return (48_000, 16)
case .lossless441: return (44_100, 24)
case .lossless48: return (48_000, 24)
case .lossless882: return (88_200, 24)
case .lossless96: return (96_000, 24)
case .lossless1764: return (176_400, 24)
}
}
/// True for the lossless plane — the gate for anything that spends the extra bandwidth.
public var isLossless: Bool { self != .opus }
}
public extension EffectiveSettings {
/// This session's requested format, resolved from the stored string.
var audioFormatChoice: AudioFormatChoice { AudioFormatChoice(setting: audioFormat) }
}
/// What a single connect was told to use, before any store is consulted.
///
/// The third case is why this is an enum rather than an `Optional<StreamProfile>`: "Connect with ▸
clients/apple/Sources/PunktfunkShared/StreamProfile.swift
+14
View File
@@ -104,6 +104,15 @@ public struct SettingsOverlay: Codable, Equatable, Sendable {
public var hdrEnabled: Bool?
public var compositor: Int?
public var audioChannels: Int?
/// An `AudioFormatChoice` raw value — the audio format this profile asks the host for.
/// Profileable because it is about how a HOST is streamed (a wired desktop can afford
/// lossless; a phone on cellular cannot), not about this device's hardware.
///
/// ⚠ The one key here with **no counterpart in the Rust overlay yet**
/// (`pf-client-core::profiles`): Apple is the first client to carry it. `audio_format` is the
/// name the others should adopt, and until they do a profile written here round-trips through
/// their unknown-key carry-through untouched rather than being honoured.
public var audioFormat: String?
public var micEnabled: Bool?
public var echoCancel: Bool?
public var touchMode: String?
@@ -149,6 +158,7 @@ public struct SettingsOverlay: Codable, Equatable, Sendable {
case hdrEnabled = "hdr_enabled"
case compositor
case audioChannels = "audio_channels"
case audioFormat = "audio_format"
case micEnabled = "mic_enabled"
case echoCancel = "echo_cancel"
case touchMode = "touch_mode"
@@ -186,6 +196,7 @@ public struct SettingsOverlay: Codable, Equatable, Sendable {
hdrEnabled = bool(.hdrEnabled)
compositor = int(.compositor)
audioChannels = int(.audioChannels)
audioFormat = str(.audioFormat)
micEnabled = bool(.micEnabled)
echoCancel = bool(.echoCancel)
touchMode = str(.touchMode)
@@ -225,6 +236,7 @@ public struct SettingsOverlay: Codable, Equatable, Sendable {
try c.encodeIfPresent(hdrEnabled, forKey: AnyKey(Key.hdrEnabled.rawValue))
try c.encodeIfPresent(compositor, forKey: AnyKey(Key.compositor.rawValue))
try c.encodeIfPresent(audioChannels, forKey: AnyKey(Key.audioChannels.rawValue))
try c.encodeIfPresent(audioFormat, forKey: AnyKey(Key.audioFormat.rawValue))
try c.encodeIfPresent(micEnabled, forKey: AnyKey(Key.micEnabled.rawValue))
try c.encodeIfPresent(echoCancel, forKey: AnyKey(Key.echoCancel.rawValue))
try c.encodeIfPresent(touchMode, forKey: AnyKey(Key.touchMode.rawValue))
@@ -282,6 +294,7 @@ public enum OverlayField {
case "hdr_enabled": overlay.hdrEnabled = nil
case "compositor": overlay.compositor = nil
case "audio_channels": overlay.audioChannels = nil
case "audio_format": overlay.audioFormat = nil
case "mic_enabled": overlay.micEnabled = nil
case "echo_cancel": overlay.echoCancel = nil
case "touch_mode": overlay.touchMode = nil
@@ -321,6 +334,7 @@ public enum OverlayField {
case "hdr_enabled": return o.hdrEnabled != nil
case "compositor": return o.compositor != nil
case "audio_channels": return o.audioChannels != nil
case "audio_format": return o.audioFormat != nil
case "mic_enabled": return o.micEnabled != nil
case "echo_cancel": return o.echoCancel != nil
case "touch_mode": return o.touchMode != nil
clients/apple/Tests/PunktfunkKitTests/AudioRingDriftTests.swift
+519 -30
View File
@@ -20,7 +20,7 @@ final class AudioRingDriftTests: XCTestCase {
/// Run `ms` of audio through the ring at a `quantumMS` device where the producer delivers
/// `driftPPM` more than the consumer takes. Returns `(final ms, peak ms, silent callbacks)`.
private func simulate(ms: Int, quantumMS: Int, driftPPM: Int) -> (Int, Int, Int) {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = quantumMS * perMS
var scratch = [Float](repeating: 0, count: want)
// Non-zero so a silent callback is distinguishable from real audio.
@@ -81,7 +81,7 @@ final class AudioRingDriftTests: XCTestCase {
/// One transient drain must not manufacture a whole target's worth of fresh silence: the ring
/// de-primes only after a RUN of short reads.
func testSingleShortReadDoesNotDeprime() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
// Prime well past target.
@@ -120,7 +120,7 @@ final class AudioRingDriftTests: XCTestCase {
let quanta = [5, 8, 10, 16, 21]
var deprimedAt: [Int: Int] = [:]
for quantumMS in quanta {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = quantumMS * perMS
var scratch = [Float](repeating: 0, count: want)
// Prime DEEP: the depth average is seeded with the refill, so `hollow` stays false for
@@ -184,7 +184,7 @@ final class AudioRingDriftTests: XCTestCase {
/// runs DEEP: a knife-edge refill (exactly what each read takes) leaves the ring within a
/// frame of empty every callback, which now correctly reads as pressure, not quiet.
func testTargetGrowsOnUnderrunsAndRelaxesWhenQuiet() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
let feed = [Float](repeating: 0.5, count: 60 * perMS)
@@ -232,7 +232,7 @@ final class AudioRingDriftTests: XCTestCase {
/// Growth is capped at `maxTargetMS`, exactly like `JitterPolicy` respects
/// `JitterTuning.max_target_ms`.
func testTargetGrowthRespectsTheCap() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
let feed = [Float](repeating: 0.5, count: 25 * perMS)
@@ -254,7 +254,7 @@ final class AudioRingDriftTests: XCTestCase {
/// forever; the adaptive floor must deepen until the bunching rides through, and the tail of
/// the session must be silence-free.
func testWifiBunchingConvergesToSilenceFree() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
var pending = 0 // ms produced by the host but still "in flight"
@@ -306,8 +306,11 @@ final class AudioRingDriftTests: XCTestCase {
/// Build an observation whose measured offset is exactly `offsetMS` (positive = audio late).
/// Mirrors the Rust `obs` helper: pin now/skew/pts so the only free term is the buffered depth,
/// then choose the video figure so the difference lands where we want it.
private func obs(offsetMS: Int, depth: Int) -> AvSync.Observation {
let bufferedMS = depth / perMS
///
/// `rateHz` must match the `AvSync` under test — the depth→ms conversion here has to be the
/// same one the type does internally, or the observation asks for an offset it isn't building.
private func obs(offsetMS: Int, depth: Int, rateHz: Int = 48_000) -> AvSync.Observation {
let bufferedMS = depth / ((rateHz / 1000) * channels)
let audioE2eMS = bufferedMS + 40 // 40 ms of transport, arbitrary but fixed
let videoE2eMS = audioE2eMS - offsetMS
return AvSync.Observation(
@@ -319,12 +322,16 @@ final class AudioRingDriftTests: XCTestCase {
}
/// Fold `n` identical observations in.
private func settle(_ sync: inout AvSync, offsetMS: Int, depth: Int, count: Int = 100) {
for _ in 0..<count { sync.observe(obs(offsetMS: offsetMS, depth: depth)) }
private func settle(
_ sync: inout AvSync, offsetMS: Int, depth: Int, count: Int = 100, rateHz: Int = 48_000
) {
for _ in 0..<count {
sync.observe(obs(offsetMS: offsetMS, depth: depth, rateHz: rateHz))
}
}
func testAvSyncNeedsEvidenceBeforeActing() {
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
// One sample is never enough — the skew estimate and the video figure both settle after
// connect, and acting on the first would chase the handshake, not the stream.
XCTAssertNil(s.observe(obs(offsetMS: 50, depth: 30 * perMS)))
@@ -338,7 +345,7 @@ final class AudioRingDriftTests: XCTestCase {
/// arrive. This is the state every session starts in, and the one the stage-1 fallback
/// presenter stays in for its whole life.
func testAvSyncWithoutAVideoReferenceNeverActs() {
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
for _ in 0..<500 {
s.observe(AvSync.Observation(
ptsNs: 1_000_000_000, nowLocalNs: 1_040_000_000, clockOffsetNs: 0,
@@ -350,7 +357,7 @@ final class AudioRingDriftTests: XCTestCase {
func testAvSyncAimsShallowerWhenAudioIsLate() {
let depth = 60 * perMS
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
settle(&s, offsetMS: 40, depth: depth, count: 400)
guard let want = s.desiredDepth(currentDepth: depth) else {
return XCTFail("a 40 ms offset is actionable")
@@ -364,7 +371,7 @@ final class AudioRingDriftTests: XCTestCase {
func testAvSyncAimsDeeperWhenAudioIsEarly() {
let depth = 20 * perMS
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
settle(&s, offsetMS: -30, depth: depth, count: 400)
guard let want = s.desiredDepth(currentDepth: depth) else {
return XCTFail("a 30 ms offset is actionable")
@@ -375,7 +382,7 @@ final class AudioRingDriftTests: XCTestCase {
func testAvSyncDeadbandsWhatNoOneCanHear() {
let depth = 30 * perMS
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
settle(&s, offsetMS: 8, depth: depth, count: 400) // inside the 10 ms deadband
XCTAssertNil(
s.desiredDepth(currentDepth: depth),
@@ -388,7 +395,7 @@ final class AudioRingDriftTests: XCTestCase {
/// refused outright and the running average is left untouched.
func testAvSyncRejectsTheImplausibleInsteadOfClampingIt() {
let depth = 30 * perMS
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
settle(&s, offsetMS: 30, depth: depth, count: 400)
let before = s.offsetMS
// Built directly rather than through `obs`: that helper floors the video figure at zero,
@@ -413,7 +420,7 @@ final class AudioRingDriftTests: XCTestCase {
/// stream, it aborts the process, from the audio drain thread. The guard's short-circuit
/// ordering is what makes the sanity check itself safe to run.
func testAvSyncRefusesAnOffsetItCannotEvenCompute() {
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
let wild = AvSync.Observation(
ptsNs: 1 << 63, nowLocalNs: 40_000_000, clockOffsetNs: 0,
bufferedAhead: 0, videoE2eNs: 40_000_000)
@@ -435,13 +442,13 @@ final class AudioRingDriftTests: XCTestCase {
/// effective target. Without this the whole feature could ship as unreachable code with every
/// other test still green — which is exactly how the previous drift correction shipped dead.
func testSyncActuallyMovesTheTarget() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
primeQuantum(ring, quantumMS: 5)
XCTAssertEqual(ring.stats.targetMS, 20, "base target (JitterTuning.COREAUDIO)")
// Audio 30 ms EARLY at a 20 ms depth ⇒ aim 50 ms deep: above the floor, under the 90 ms
// cap, so the ring has no reason to refuse.
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
settle(&s, offsetMS: -30, depth: 20 * perMS, count: 400)
ring.setSyncTarget(s.desiredDepth(currentDepth: 20 * perMS))
XCTAssertEqual(ring.stats.targetMS, 50, "the ring must adopt a legal request")
@@ -456,7 +463,7 @@ final class AudioRingDriftTests: XCTestCase {
/// not just the base, because the floor sync is most likely to argue with is the one a bad link
/// earned.
func testSyncCanNeverStarveTheRing() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
let feed = [Float](repeating: 0.5, count: 25 * perMS)
@@ -497,7 +504,7 @@ final class AudioRingDriftTests: XCTestCase {
/// back the cap — quietly below the floor, inverting the whole ordering — on exactly the
/// awkward hardware this code exists to survive.
func testAHugeDeviceQuantumDoesNotInvertTheClamp() {
let ring = AudioRing(capacity: 48_000 * channels * 2, channels: channels)
let ring = AudioRing(seconds: 2, channels: channels, rateHz: 48_000)
let quantumMS = 500 // absurd, but not a reason to starve the callback
primeQuantum(ring, quantumMS: quantumMS)
ring.setSyncTarget(0)
@@ -550,12 +557,12 @@ final class AudioRingDriftTests: XCTestCase {
return reads
}
let slow = AudioRing(capacity: 48_000 * channels, channels: channels)
let slow = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
grow(slow)
slow.setSyncTarget(nil)
let slowReads = quietToRelax(slow)
let fast = AudioRing(capacity: 48_000 * channels, channels: channels)
let fast = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
grow(fast)
fast.setSyncTarget(perMS) // strictly shallower than the grown target
let fastReads = quietToRelax(fast)
@@ -571,7 +578,7 @@ final class AudioRingDriftTests: XCTestCase {
/// depth every five quiet seconds and paid an audible starvation event each time it was wrong,
/// forever — the 0.25.0 MacBook field report.
func testAFailedShrinkProbeIsUndoneAtOnceAndBacksTheSyncLoopOff() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
let feed = [Float](repeating: 0.5, count: 60 * perMS)
@@ -637,7 +644,7 @@ final class AudioRingDriftTests: XCTestCase {
/// bunching period indefinitely. The average, not the instant, is what separates a hollow ring
/// from one late packet (`testSingleShortReadDoesNotDeprime` pins that side).
func testAHollowRingReprimesOnItsFirstClick() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
let feed = [Float](repeating: 0.5, count: 60 * perMS)
@@ -681,8 +688,8 @@ final class AudioRingDriftTests: XCTestCase {
/// did. `nil` is the default, so this pins the initializer too — and every other test in this
/// file runs without a sync target, which is the real guard that nothing moved underneath them.
func testNoSyncTargetLeavesTheRingExactlyAsItWas() {
let a = AudioRing(capacity: 48_000 * channels, channels: channels)
let b = AudioRing(capacity: 48_000 * channels, channels: channels)
let a = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let b = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
b.setSyncTarget(nil)
let want = 5 * perMS
var sa = [Float](repeating: 0, count: want)
@@ -707,12 +714,12 @@ final class AudioRingDriftTests: XCTestCase {
/// a depth on its own cannot distinguish "deep because the link needs it" from "deep and
/// therefore late". This is the number the HUD and the 1 Hz log line read.
func testAvOffsetIsReportedAlongsideTheDepth() {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
XCTAssertEqual(ring.stats.avOffsetMS, 0, "no evidence yet reads as zero, not as noise")
let feed = [Float](repeating: 0.5, count: 30 * perMS)
feed.withUnsafeBufferPointer { ring.write($0.baseAddress!, count: 30 * perMS) }
var s = AvSync(channels: channels)
var s = AvSync(channels: channels, rateHz: 48_000)
settle(&s, offsetMS: 37, depth: 30 * perMS, count: 400)
ring.noteAvOffset(s.offsetMS)
let stats = ring.stats
@@ -800,7 +807,7 @@ final class AudioRingDriftTests: XCTestCase {
/// Prime, then stall the wire for `ms`, ticking the drain thread's 5 ms loop and the
/// device callback in step. Returns when the first silent callback lands (nil = none).
func stall(ms: Int, concealing: Bool) -> Int? {
let ring = AudioRing(capacity: 48_000 * channels, channels: channels)
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let want = 5 * perMS
var scratch = [Float](repeating: 0, count: want)
let feed = [Float](repeating: 0.5, count: 25 * perMS)
@@ -831,5 +838,487 @@ final class AudioRingDriftTests: XCTestCase {
"a stall inside the budget must not reach the listener at all (unconcealed: silent "
+ "after \(deprimedAt) ms)")
}
/// The drought budget is WALL CLOCK, and it is spent one frame at a time — so the two have to
/// agree about how long a frame is. They did not: a frame was assumed to be 5 ms, so on a 2 ms
/// lossless frame the budget ran out after two fifths of the time it promises and `plc_ms`
/// over-reported by the same factor. A 5.1 session, whose frame drops to about 1 ms, was five
/// times out. Mirrors core's `the_drought_budget_is_spent_at_the_negotiated_frame_length`.
///
/// **The assertion that matters is the FRAME COUNT, not the millisecond total**, for exactly the
/// reason the rate ladder's proof had to be made in samples: the defect charges 5 ms and reports
/// 5 ms, so `totalMS` reads a perfectly correct 120 either way. What it cannot satisfy is the
/// relationship between the two — the frames actually synthesized, times the frame the wire
/// really carried, has to BE the total reported. Under the defect at 2 ms that is 24 × 2 = 48 ms
/// of audio reported as 120.
func testTheDroughtBudgetIsSpentAtTheNegotiatedFrameLength() {
/// Spend the whole budget on a dead wire, returning the frames it bought and what the
/// session total reads.
func spend(frameUs: Int) -> (frames: Int, totalMS: Int) {
var c = DroughtConceal(maxMS: AudioRing.plcMaxMS, frameUs: frameUs)
var frames = 0
// A second of silence and an empty ring: both thresholds are wide open, so the only
// thing that can stop this loop is the budget.
while c.conceal(sinceLastPacketMS: 1_000, depthMS: 0) { frames += 1 }
return (frames, c.totalMS)
}
// The Opus plane, unchanged: 120 ms of 5 ms frames is 24 of them. Every figure here is
// exactly what shipped, which is the bit-identity gate.
XCTAssertEqual(spend(frameUs: 5_000).frames, 24, "120 ms of 5 ms frames")
XCTAssertEqual(spend(frameUs: 5_000).totalMS, AudioRing.plcMaxMS)
// …and the same budget at every shorter frame must buy the same WALL CLOCK, which means
// MORE frames — not the 24 a 5 ms charge would have allowed.
// 4 000 µs — 48 kHz/24-bit stereo at the default MTU
// 2 000 µs — 96 kHz/24-bit stereo
// 1 500 µs — 48 kHz/24-bit 5.1
// 1 000 µs — 48 kHz/24-bit 7.1, and the ladder's shortest rung
for (frameUs, expected) in [(4_000, 30), (2_000, 60), (1_500, 80), (1_000, 120)] {
let (frames, totalMS) = spend(frameUs: frameUs)
XCTAssertEqual(
frames, expected,
"\(frameUs) µs: the budget must buy \(expected) frames, not a 5 ms charge's 24")
// THE assertion: the audio actually synthesized is the audio reported. A flat 5 ms
// charge satisfies the right-hand side and fails this.
XCTAssertEqual(
frames * frameUs / 1_000, totalMS,
"\(frameUs) µs: plc_ms must be the concealment that really happened")
XCTAssertEqual(
totalMS, AudioRing.plcMaxMS,
"\(frameUs) µs: and the budget is the same wall clock at every frame length")
}
// A packet ends the run and hands back a full budget — at the negotiated frame too, so a
// link that stalls once a minute is covered every time and not only the first.
var c = DroughtConceal(maxMS: AudioRing.plcMaxMS, frameUs: 2_000)
while c.conceal(sinceLastPacketMS: 1_000, depthMS: 0) {}
XCTAssertFalse(c.conceal(sinceLastPacketMS: 1_000, depthMS: 0), "budget spent")
c.packet()
XCTAssertTrue(c.conceal(sinceLastPacketMS: 1_000, depthMS: 0), "a full budget again")
XCTAssertEqual(
c.totalMS, AudioRing.plcMaxMS + 2,
"the SESSION total keeps counting, in the frame the wire really carried")
// The convenience initializer IS the default frame — the property that keeps every Opus
// session and the four drought tests above bit-identical.
var byDefault = DroughtConceal(maxMS: AudioRing.plcMaxMS)
var explicit = DroughtConceal(maxMS: AudioRing.plcMaxMS, frameUs: AudioRing.frameMS * 1_000)
for quiet in [0, 5, 9, 10, 50] {
for depth in [0, 4, 10, 11, 40] {
XCTAssertEqual(
byDefault.conceal(sinceLastPacketMS: quiet, depthMS: depth),
explicit.conceal(sinceLastPacketMS: quiet, depthMS: depth),
"quiet=\(quiet) depth=\(depth): the default frame must be 5 ms")
}
}
XCTAssertEqual(byDefault.totalMS, explicit.totalMS)
}
/// Both thresholds — how long a quiet wire must stay quiet, and how empty the ring must be —
/// are TWO FRAMES, so they move with the negotiated frame instead of sitting at a fixed 10 ms.
///
/// At 5 ms they are the 10 ms that shipped. On a 2 ms lossless frame a frozen 10 ms waits five
/// frames before conceding there is a stall, and treats a ring holding five frames as "running
/// out" — both a factor of two and a half away from the "about a couple of packets" the policy
/// is written to mean.
func testTheDroughtThresholdsFollowTheNegotiatedFrame() {
/// Fresh each time: `conceal` mutates on success, and these probe the thresholds, not a run.
func concealsAfter(_ quietMS: Int, frameUs: Int) -> Bool {
var c = DroughtConceal(maxMS: AudioRing.plcMaxMS, frameUs: frameUs)
return c.conceal(sinceLastPacketMS: quietMS, depthMS: 0)
}
func concealsAtDepth(_ depthMS: Int, frameUs: Int) -> Bool {
var c = DroughtConceal(maxMS: AudioRing.plcMaxMS, frameUs: frameUs)
return c.conceal(sinceLastPacketMS: 1_000, depthMS: depthMS)
}
// The Opus plane: two 5 ms frames, exactly the 10 ms that shipped.
XCTAssertFalse(concealsAfter(9, frameUs: 5_000), "9 ms is under two 5 ms frames")
XCTAssertTrue(concealsAfter(10, frameUs: 5_000), "two 5 ms frames is a stall")
XCTAssertFalse(concealsAtDepth(11, frameUs: 5_000), "an 11 ms ring covers it by itself")
XCTAssertTrue(concealsAtDepth(10, frameUs: 5_000), "two frames deep is running out")
// A 2 ms lossless frame: four, not ten. Frozen at 10 ms this would wait five frames.
XCTAssertFalse(concealsAfter(3, frameUs: 2_000), "3 ms is under two 2 ms frames")
XCTAssertTrue(
concealsAfter(4, frameUs: 2_000),
"two 2 ms frames of silence is a stall — a fixed 10 ms would still be waiting")
XCTAssertFalse(concealsAtDepth(5, frameUs: 2_000), "a 5 ms ring covers a 2 ms-frame gap")
XCTAssertTrue(
concealsAtDepth(4, frameUs: 2_000),
"four ms is two 2 ms frames — running out; a fixed 10 ms floor would call a ring "
+ "holding five frames empty and synthesize over audio it is about to duplicate")
// The 1 ms rung a 7.1 lossless session lands on, where the fixed threshold is ten frames.
XCTAssertFalse(concealsAfter(1, frameUs: 1_000))
XCTAssertTrue(concealsAfter(2, frameUs: 1_000))
XCTAssertTrue(concealsAtDepth(2, frameUs: 1_000))
XCTAssertFalse(concealsAtDepth(3, frameUs: 1_000))
// A rung that is not a whole millisecond: 1 500 µs → two frames is exactly 3 ms, and the
// depth floor rounds UP (core's `div_ceil`) so it is never *less* than the two frames it
// promises. A degenerate frame must not produce a zero-length tolerance either, or ordinary
// jitter would be concealed as though it were a stall.
XCTAssertFalse(concealsAfter(2, frameUs: 1_500))
XCTAssertTrue(concealsAfter(3, frameUs: 1_500))
XCTAssertTrue(concealsAtDepth(3, frameUs: 1_500))
XCTAssertFalse(concealsAfter(0, frameUs: 1), "a degenerate frame keeps a 1 ms tolerance")
}
// MARK: - The negotiated rate
/// The rate REACHES the arithmetic — every ms↔sample conversion in the ring, not just its
/// capacity. Pinned because the failure mode is silent in both directions: a ring left at 48
/// while the wire runs at 96 reports (and targets, and sheds at) double the milliseconds it
/// really holds, and a capacity left as the old `48_000 * channels` literal is half a second of
/// ring on the one plane that most needs the overflow headroom. Neither throws, warns, or
/// sounds wrong until a link goes bad.
func testRateDrivesEveryMsConversionAndTheCapacity() {
let fast = AudioRing(seconds: 1, channels: channels, rateHz: 96_000)
// The base target is a TIME (JitterTuning.COREAUDIO's 20 ms) and must read as one at any
// rate — while costing twice the samples at 96 kHz, which is the whole point.
XCTAssertEqual(fast.stats.targetMS, 20, "the target is denominated in ms, not samples")
// 20 ms of 96 kHz audio is 1 920 frames; at 48 kHz the same sample count would read 40 ms.
let ms20 = 20 * 96 * channels
let feed = [Float](repeating: 0.5, count: ms20)
feed.withUnsafeBufferPointer { fast.write($0.baseAddress!, count: ms20) }
XCTAssertEqual(fast.bufferedMS, 20, "depth must be ms at the NEGOTIATED rate")
// Capacity is a second of audio at the NEGOTIATED rate, not a second's worth of the old
// `48_000 * channels` literal. Probed through `write`'s over-capacity guard, which drops a
// too-large write whole rather than wrapping it: one second exactly must be taken, one
// sample more must not. On a ring still sized from the 48 000 literal the first of these
// would be the one silently dropped — which is the half-second-ring defect, expressed as
// something a test can see.
let empty = AudioRing(seconds: 1, channels: channels, rateHz: 96_000)
let overflow = [Float](repeating: 0.5, count: 96_000 * channels + channels)
overflow.withUnsafeBufferPointer { empty.write($0.baseAddress!, count: overflow.count) }
XCTAssertEqual(empty.bufferedMS, 0, "an over-capacity write is dropped, not wrapped")
overflow.withUnsafeBufferPointer {
empty.write($0.baseAddress!, count: 96_000 * channels)
}
XCTAssertGreaterThan(
empty.bufferedMS, 0,
"one second of 96 kHz audio must fit — a ring sized from a 48 000 literal holds half, "
+ "and would have dropped this write entirely")
// And the sync loop agrees about what a millisecond is: a 30 ms correction has to be 30 ms
// of samples in the ring's own units, or the depth it proposes means something else.
var s = AvSync(channels: channels, rateHz: 96_000)
settle(&s, offsetMS: 30, depth: 40 * 96 * channels, count: 400, rateHz: 96_000)
XCTAssertEqual(
s.desiredDepth(currentDepth: 40 * 96 * channels), 10 * 96 * channels,
"audio 30 ms late at a 40 ms depth ⇒ aim 10 ms, in 96 kHz samples")
}
// MARK: - The negotiated frame length
/// The Swift half of core's `the_shed_follows_the_negotiated_frame_length`. Two of this ring's
/// decisions are denominated in FRAMES, not milliseconds — the smooth shed drops exactly one,
/// and the effective-target floor is a device quantum plus one — and both were written when
/// 5 ms was the only frame the protocol had. The lossless plane negotiates 4 ms at 48 kHz/24-bit
/// and 2 ms at 96 kHz/24-bit, so a ring left on the constant sheds two and a half frames at a
/// time and fades across an entire one.
func testFrameGeometryFollowsTheNegotiatedFrameLength() {
// Default: one 5 ms frame, a 2 ms fade — exactly the pre-hi-res numbers, which is what
// keeps every Opus session (and the twenty-nine tests above) bit-identical.
let base = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
XCTAssertEqual(base.frameGeometry.frame, AudioRing.frameMS * perMS)
XCTAssertEqual(base.frameGeometry.crossfade, 2 * perMS)
// A 2 ms lossless frame sheds 2 ms, and the fade is capped at HALF of it rather than
// consuming the whole dropped frame — a fade as long as the material it fades is not a
// crossfade, it is a ramp replacing the seam.
let short = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
short.setFrameUs(2_000)
XCTAssertEqual(short.frameGeometry.frame, 2 * perMS)
XCTAssertEqual(short.frameGeometry.crossfade, perMS, "fade must be half a 2 ms frame")
XCTAssertLessThan(
short.frameGeometry.crossfade, short.frameGeometry.frame,
"a fade as long as the frame is not a crossfade")
// Sub-millisecond precision: 2 500 µs at 48 kHz stereo is 240 interleaved samples, and must
// not truncate to 192 by going through integer milliseconds on the way. This is the whole
// reason the accessor is denominated in µs.
let half = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
half.setFrameUs(2_500)
XCTAssertEqual(half.frameGeometry.frame, 240, "2 500 µs must not truncate to 2 ms")
// At 96 kHz the same 2 ms frame is twice the samples for the same duration.
let hires = AudioRing(seconds: 1, channels: channels, rateHz: 96_000)
hires.setFrameUs(2_000)
XCTAssertEqual(hires.frameGeometry.frame, 2 * 96 * channels)
// A degenerate value must not produce a zero-length frame — the shed would become an
// infinite no-op and the target floor would lose its packet of slack.
let zero = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
zero.setFrameUs(0)
XCTAssertGreaterThanOrEqual(zero.frameGeometry.frame, 1)
}
/// The frame reaches the EFFECTIVE TARGET FLOOR, not just a getter. A large-quantum device
/// cannot sustain a target below its own callback, so the floor is `quantum + one frame` — and
/// on a 2 ms session that packet of slack should be 2 ms, not the 5 a constant would give.
func testTargetFloorCarriesOneNegotiatedFrameOverTheDeviceQuantum() {
func floorMS(frameUs: Int?) -> Int {
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
if let frameUs { ring.setFrameUs(frameUs) }
// One oversized callback is all it takes: `renderQuantum` is a high-water mark, and
// 30 ms exceeds the 20 ms base target so the lift is what decides the floor.
var scratch = [Float](repeating: 0, count: 30 * perMS)
scratch.withUnsafeMutableBufferPointer {
ring.read(into: $0.baseAddress!, count: $0.count)
}
return ring.stats.targetMS
}
XCTAssertEqual(floorMS(frameUs: nil), 35, "30 ms quantum + the default 5 ms frame")
XCTAssertEqual(floorMS(frameUs: 2_000), 32, "30 ms quantum + a 2 ms lossless frame")
XCTAssertEqual(floorMS(frameUs: 4_000), 34, "30 ms quantum + a 4 ms lossless frame")
}
/// The half-frame cap reaches the SAMPLES. Driven through the hard-cap trim rather than the
/// slow drift shed because they share `dropFront`, and the trim is the drop that actually fires
/// in the field (a bunching link trims far more often than it sheds).
///
/// The ring is filled with silence where the trim will cut and full scale after it, so every
/// blended sample is strictly below full scale and the fade length is simply countable.
func testTheSeamCrossfadeIsCappedAtHalfTheNegotiatedFrame() {
func fadeLength(frameUs: Int?) -> Int {
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
if let frameUs { ring.setFrameUs(frameUs) }
// A fresh ring's cap is target(20) + headroom(30) = 50 ms, so 60 ms of audio trims
// exactly 10 ms off the front — comfortably more than any fade under test.
var feed = [Float](repeating: 1, count: 60 * perMS)
for i in 0..<(10 * perMS) { feed[i] = 0 }
feed.withUnsafeBufferPointer { ring.write($0.baseAddress!, count: feed.count) }
// Read one 5 ms callback: small enough to stay primed (the floor needs quantum + a
// frame ≤ the 50 ms banked), large enough to contain any fade under test.
var out = [Float](repeating: -1, count: 5 * perMS)
out.withUnsafeMutableBufferPointer {
ring.read(into: $0.baseAddress!, count: $0.count)
}
return out.prefix { $0 < 1 }.count
}
// Default: the flat 2 ms fade, well under half of a 5 ms frame.
XCTAssertEqual(fadeLength(frameUs: nil), 2 * perMS)
// A 2 ms frame caps the fade at 1 ms — without the cap it would be the whole frame.
XCTAssertEqual(fadeLength(frameUs: 2_000), perMS, "half of a 2 ms frame")
// 4 ms leaves the flat 2 ms fade untouched: half of 4 is exactly 2, so the cap binds
// without shortening it — the boundary worth pinning.
XCTAssertEqual(fadeLength(frameUs: 4_000), 2 * perMS)
}
/// The NEAR-MISS margin follows the frame too — a read that leaves more than one frame in hand
/// is not a near miss and must not grow the target.
///
/// This was once the one place the Swift ring deliberately diverged from core, which measured
/// the margin against a `NEAR_MISS_MARGIN_MS` constant. Core has since followed — its
/// `the_near_miss_margin_is_one_negotiated_frame` pins the same rule — so this now mirrors
/// rather than diverges. A margin frozen at 5 ms against a 2 ms frame stops meaning "one packet
/// in hand" and starts meaning "two and a half", growing the target on a ring that was never
/// close to starving, which is the opposite of what the near-miss exists to detect.
func testNearMissMarginFollowsTheNegotiatedFrame() {
func targetAfterLeaving(_ leftover: Int, frameUs: Int?) -> Int {
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
if let frameUs { ring.setFrameUs(frameUs) }
// Bank 25 ms — over the 20 ms base target, under the 50 ms hard cap, so nothing trims.
let feed = [Float](repeating: 0.5, count: 25 * perMS)
feed.withUnsafeBufferPointer { ring.write($0.baseAddress!, count: feed.count) }
// A 5 ms callback primes the ring and leaves 20 ms — nowhere near any margin.
var prime = [Float](repeating: 0, count: 5 * perMS)
prime.withUnsafeMutableBufferPointer {
ring.read(into: $0.baseAddress!, count: $0.count)
}
// Then serve one FULL read that leaves exactly `leftover` samples in hand.
var out = [Float](repeating: 0, count: 20 * perMS - leftover)
out.withUnsafeMutableBufferPointer {
ring.read(into: $0.baseAddress!, count: $0.count)
}
return ring.stats.targetMS
}
// 300 samples ≈ 3.1 ms: MORE than a 2 ms frame, LESS than the 5 ms constant. With the
// margin tied to the frame this is an ordinary read; tied to the constant it is a near
// miss and buys a 10 ms growth step.
XCTAssertEqual(
targetAfterLeaving(300, frameUs: 2_000), 20,
"3.1 ms left over is more than a 2 ms frame — not a near miss, no growth")
// The same read against the DEFAULT 5 ms frame genuinely is a near miss, which is what
// keeps this test honest: it is not simply asserting that growth never happens.
XCTAssertEqual(
targetAfterLeaving(300, frameUs: nil), 30,
"3.1 ms left over IS inside a 5 ms frame — one growth step")
}
// MARK: - The whole rate ladder (design/hi-res-audio.md §4.1)
/// Every rate the lossless plane carries — `pcm::rate_is_supported`. Both families, so a rate
/// that only one of them divides can never be pinned by accident.
private static let ladder = [44_100, 48_000, 88_200, 96_000, 176_400]
/// §4.1's tripwire, as an assertion rather than a comment — and now the proof that the deferral
/// it guarded is lifted. Mirrors core's `the_shipping_rate_ladder_round_trips_ms_to_samples_exactly`.
///
/// **Why this cannot be asserted in milliseconds.** The obvious shape — write N ms, read back
/// `bufferedMS`, expect N — passes under the defect too, because BOTH directions divided by the
/// same wrong `perMS` and a wrong number used consistently is self-consistent. That is exactly
/// what let a 2.3 % error live in the reported figures unnoticed. The error is only visible in
/// SAMPLES, so this observes the one sample count the ring publishes: the depth its hard-cap
/// trim leaves behind, which is `target + headroom` converted by the ring itself, measured
/// against the same two milliseconds converted here the honest way — multiply first, divide
/// last.
///
/// Plant the defect (`perMS = (rateHz / 1000) * channels`, every figure `ms * perMS`) and this
/// fails at 44 100 / 88 200 / 176 400 while every 48 kHz test in this file stays green — that
/// asymmetry IS the bug, and the reason 48 and 96 kHz shipped first.
func testTheShippingRateLadderRoundTripsMsToSamplesExactly() {
/// `ms` of audio in interleaved samples, computed the way §4.1 says it must be: the whole
/// product first, the divide by 1 000 last. Deliberately NOT `audioMsToSamples` — a test
/// that calls the code under test to compute its own expectation asserts nothing.
func honest(_ ms: Int, _ rateHz: Int, _ channels: Int) -> Int {
ms * rateHz * channels / 1_000
}
for rateHz in Self.ladder {
for channels in [2, 6, 8] {
let ring = AudioRing(seconds: 1, channels: channels, rateHz: rateHz)
// The base target must read back as the preset's 20 ms at every rate…
XCTAssertEqual(
ring.stats.targetMS, 20,
"\(rateHz) Hz / \(channels)ch: the target is denominated in ms, not samples")
// …and the SAMPLES behind it must be the honest ones. 88 200 interleaved samples
// is one second at the shallowest layout on the ladder (44.1 kHz stereo) and a
// whole number of frames at 2/6/8 channels, so one figure over-fills every ring
// here and the hard cap trims each to its own `target + headroom`.
let flood = [Float](repeating: 0.5, count: 88_200)
flood.withUnsafeBufferPointer { ring.write($0.baseAddress!, count: flood.count) }
XCTAssertEqual(
ring.bufferedSamples,
honest(20, rateHz, channels) + honest(30, rateHz, channels),
"\(rateHz) Hz / \(channels)ch: the hard cap sits where ms × rate × ch / 1000 "
+ "puts it — a leading divide by 1 000 truncates 44.1 kHz to 44 samples/ms "
+ "and lands every figure 2.3 % low")
}
}
// The conversion itself, against the same honest arithmetic, over the spans this policy
// actually names — each is a threshold something in `read`/`noteRead` compares a sample
// count against, and a rate that skewed 2.3 % skewed all of them together, which is what
// kept the defect invisible.
for rateHz in Self.ladder {
for channels in [2, 6, 8] {
for ms in [1, 10, 12, 15, 20, 30, 47, 60, 90, 1_000, 5_000, 480_000] {
XCTAssertEqual(
audioMsToSamples(rateHz: rateHz, channels: channels, ms: ms),
honest(ms, rateHz, channels),
"\(ms) ms at \(rateHz) Hz / \(channels)ch")
}
}
}
// The worked example, spelled out, so the 2.3 % is a number rather than an adjective.
XCTAssertEqual(
audioMsToSamples(rateHz: 44_100, channels: 2, ms: 15), 1_323,
"15 ms of 44.1 kHz stereo")
XCTAssertEqual(15 * (44_100 / 1000) * 2, 1_320, "what it used to compute")
// ⚠ Exact is not the same as lossless in both directions, and the difference is worth
// stating rather than discovering. A millisecond is 88.2 samples at 44.1 kHz stereo, so an
// ms figure that is not a multiple of 5 genuinely has no whole-sample answer: 12 ms lands
// on 1 058 samples, which reads back as 11. That is a floor of at most ONE SAMPLE on one
// threshold inside a 25 ms band — as against the 2.3 % the old arithmetic was wrong by on
// EVERY figure, in the same direction, permanently.
XCTAssertEqual(audioMsToSamples(rateHz: 44_100, channels: 2, ms: 12), 1_058) // 1 058.4
XCTAssertEqual(audioSamplesToMs(rateHz: 44_100, channels: 2, samples: 1_058), 11)
// A caller-supplied sample count is not bounded by anything (`setSyncTarget(Int.max / 2)`
// is a call this file makes), and Swift TRAPS on overflow rather than wrapping — so the
// samples → ms direction saturates instead of taking the process down from the render
// callback. Core widens to u128 for the same reason.
XCTAssertEqual(
audioSamplesToMs(rateHz: 48_000, channels: 2, samples: Int.max), Int.max,
"an unbounded sample count must saturate, not trap")
}
/// The ring's idea of a frame must be the WIRE's idea of a frame, at a rate where the two are
/// no longer the same arithmetic. Mirrors core's `the_policys_frame_is_the_wires_frame`.
///
/// A frame carries a whole number of samples PER CHANNEL, so a 5 ms frame at 88.2 kHz stereo is
/// 882 interleaved samples and not the 880 that `frameUs × samples-per-ms` produces. Both the
/// shed size and the near-miss margin mean *exactly one packet*, so a ring that computed its own
/// answer would be describing a packet that does not exist — off by one sample per frame, on an
/// interleaved stream, forever.
func testTheRingsFrameIsTheWiresFrame() {
// `pcm::FRAME_US_LADDER`, longest first — the rungs the host may negotiate.
let rungs = [5_000, 4_000, 3_000, 2_500, 2_000, 1_500, 1_000]
for rateHz in Self.ladder {
for channels in [2, 6, 8] {
for us in rungs {
let ring = AudioRing(seconds: 1, channels: channels, rateHz: rateHz)
ring.setFrameUs(us)
let perChannel = rateHz * us / 1_000_000 // floors — 220.5 samples do not exist
XCTAssertEqual(
ring.frameGeometry.frame, perChannel * channels,
"\(rateHz) Hz / \(channels)ch at \(us) µs")
XCTAssertEqual(
ring.frameGeometry.frame % channels, 0,
"a frame must be whole in every channel or the interleave walks")
}
}
}
// The concrete disagreement this prevents: five milliseconds of 44.1 kHz stereo AUDIO is
// 441 interleaved samples, and a five-millisecond FRAME of it carries 440 — because the
// frame has to be whole in each channel and 220.5 is not a sample count. Two different
// questions, two different answers, and only one of them is the packet.
let cd = AudioRing(seconds: 1, channels: 2, rateHz: 44_100)
cd.setFrameUs(5_000)
XCTAssertEqual(cd.frameGeometry.frame, 440, "220 samples per channel, not 220.5")
XCTAssertEqual(
audioMsToSamples(rateHz: 44_100, channels: 2, ms: 5), 441,
"5 ms of audio, which is not a frame")
}
// MARK: - Surround on the lossless plane
/// The lossless plane was stereo-only because a surround frame did not fit a datagram; the
/// frame ladder is channel-aware, so the restriction is lifted and this ring has to be sized
/// from the RESOLVED channel count rather than from an assumed pair.
///
/// 5.1 at 48 kHz/16-bit negotiates a 2 ms frame at the default MTU and 7.1 goes shorter still,
/// so the two figures that follow the frame — the shed unit and the near-miss margin — are what
/// a surround session most depends on being right.
func testSurroundSizesEverythingFromTheResolvedChannelCount() {
for channels in [6, 8] {
let ring = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let perMS = 48 * channels
// A time is a time whatever the layout costs in samples.
XCTAssertEqual(ring.stats.targetMS, 20, "\(channels)ch: the base target is 20 ms")
// 2 ms — what 48 kHz/16-bit 5.1 resolves to under the default MTU.
ring.setFrameUs(2_000)
XCTAssertEqual(
ring.frameGeometry.frame, 2 * perMS,
"\(channels)ch: the shed unit is one 2 ms surround frame, all channels of it")
XCTAssertEqual(
ring.frameGeometry.crossfade, perMS, "\(channels)ch: fade is half a frame")
// Depth still reads in ms, and a 20 ms write of a 5.1 stream is three times the samples
// a stereo one would be — the whole reason a stereo-shaped ring would have reported a
// third of the depth it really held.
let feed = [Float](repeating: 0.5, count: 20 * perMS)
feed.withUnsafeBufferPointer { ring.write($0.baseAddress!, count: feed.count) }
XCTAssertEqual(ring.bufferedMS, 20, "\(channels)ch: depth is ms at the real layout")
// And one second of capacity is one second of THIS layout — probed through `write`'s
// over-capacity guard, which drops a too-large write whole rather than wrapping it.
let empty = AudioRing(seconds: 1, channels: channels, rateHz: 48_000)
let overflow = [Float](repeating: 0.5, count: 48_000 * channels + channels)
overflow.withUnsafeBufferPointer { empty.write($0.baseAddress!, count: overflow.count) }
XCTAssertEqual(
empty.bufferedMS, 0, "\(channels)ch: an over-capacity write is dropped, not wrapped")
}
}
}
#endif
clients/apple/Tests/PunktfunkKitTests/SharedFoundationTests.swift
+67
View File
@@ -597,4 +597,71 @@ final class SharedFoundationTests: XCTestCase {
defaults.removePersistentDomain(forName: "io.unom.punktfunk.tests.effective")
}
// MARK: - The audio format a profile carries
/// `AudioFormatChoice`'s raw values are a CROSS-CLIENT contract, not an implementation detail:
/// they are what a profile stores, and the desktop clients (`pf_client_core::session::
/// AUDIO_FORMATS`) and Android (`Settings.kt`'s `AUDIO_FORMAT_*`) key the same table off the
/// same strings, so one profile catalog has to round-trip through all four. Renaming one fails
/// in the worst possible way: the key is carried through untouched, so the profile keeps
/// "working" on the other client and silently inherits its global default — the setting does
/// not error, the session just quietly costs less and sounds worse.
///
/// So every value is frozen here character by character, including the naming rule the 44.1 kHz
/// family follows: the kHz figure with the decimal point dropped.
func testAudioFormatRawValuesAreTheCrossClientContract() {
XCTAssertEqual(AudioFormatChoice.opus.rawValue, "opus")
XCTAssertEqual(AudioFormatChoice.lossless48.rawValue, "lossless48")
XCTAssertEqual(AudioFormatChoice.lossless96.rawValue, "lossless96")
XCTAssertEqual(AudioFormatChoice.lossless441.rawValue, "lossless441")
XCTAssertEqual(AudioFormatChoice.lossless882.rawValue, "lossless882")
XCTAssertEqual(AudioFormatChoice.lossless1764.rawValue, "lossless1764")
// Ordered by rate, which is the order the settings row lists them in — and the same order
// as the Android and desktop tables, so the three menus read alike. Pinned as a whole so a
// case added here is a case somebody had to look at `SettingsOptions.audioFormats` for:
// that table is in the app target and cannot be reached from these tests.
XCTAssertEqual(
AudioFormatChoice.allCases.map(\.rawValue),
["opus", "lossless441", "lossless48", "lossless882", "lossless96", "lossless1764"])
}
/// Every lossless row must reach the wire as the rate it names, at 24-bit — and `opus` must
/// stay exactly `48 000`/`16`, which is byte-for-byte a pre-lossless request and is what keeps
/// the default session on the legacy connect path.
///
/// The five rates are `punktfunk_core::audio::pcm::rate_is_supported`. 44.1/88.2/176.4 were
/// absent until the jitter policy stopped dividing by 1 000 before it multiplied
/// (design/hi-res-audio.md §4.1); nothing else ever blocked them.
func testAudioFormatWireMappingCoversBothRateFamilies() {
XCTAssertEqual(AudioFormatChoice.opus.wire.rateHz, 48_000)
XCTAssertEqual(AudioFormatChoice.opus.wire.bits, 16)
XCTAssertFalse(AudioFormatChoice.opus.isLossless)
let expected: [(AudioFormatChoice, UInt32)] = [
(.lossless441, 44_100), (.lossless48, 48_000), (.lossless882, 88_200),
(.lossless96, 96_000), (.lossless1764, 176_400),
]
for (choice, rateHz) in expected {
XCTAssertEqual(choice.wire.rateHz, rateHz, "\(choice.rawValue)")
XCTAssertEqual(choice.wire.bits, 24, "\(choice.rawValue): 24-bit earns the bandwidth")
XCTAssertTrue(choice.isLossless, "\(choice.rawValue)")
}
XCTAssertEqual(
expected.count + 1, AudioFormatChoice.allCases.count,
"a case with no wire pair here would connect at whatever the switch fell through to")
}
/// A stored value this build does not know — a newer build's, or a corrupted pref — resolves to
/// Opus rather than blocking the connect, which is what the desktop `audio_format_wire` and the
/// Android `audioFormatWire` do with the same string.
func testUnknownAudioFormatFallsBackToOpus() {
for stored in ["", "lossless192", "lossless44", "lossless44_1", "LOSSLESS48", "flac"] {
XCTAssertEqual(
AudioFormatChoice(setting: stored), .opus,
"\(stored) must not block the connect")
}
XCTAssertEqual(AudioFormatChoice(setting: "lossless441"), .lossless441)
}
}
clients/linux/src/ui_settings.rs
+61 -1
View File
@@ -16,6 +16,11 @@
use crate::trust::Settings;
use adw::prelude::*;
use pf_client_core::profiles::{ProfilesFile, SettingsOverlay, StreamProfile};
// The audio-format table lives in the session crate, not here, because the same three stored
// values also have to reach the wire — and they are shared verbatim with the Apple and Android
// clients so one profile round-trips. A second copy of the spellings in this file is exactly the
// drift the shared table exists to prevent.
use pf_client_core::session::AUDIO_FORMATS;
use pf_client_core::trust::StatsVerbosity;
use std::cell::{Cell, RefCell};
use std::collections::HashSet;
@@ -153,6 +158,15 @@ mod index {
}
}
/// An unknown stored value (a newer client's row, via a shared profile) reads as row 0 —
/// Opus — which is also what the session resolves it to, so the row and the wire agree.
pub fn audio_format(s: &Settings) -> u32 {
AUDIO_FORMATS
.iter()
.position(|(v, _)| *v == s.audio_format)
.unwrap_or(0) as u32
}
pub fn gamepad(s: &Settings) -> u32 {
GAMEPADS.iter().position(|&g| g == s.gamepad).unwrap_or(0) as u32
}
@@ -632,6 +646,9 @@ fn commit_profile(active: &StreamProfile, touched: &Touched, values: &Settings)
if touched.has("audio_channels") {
o.audio_channels = Some(values.audio_channels);
}
if touched.has("audio_format") {
o.audio_format = Some(values.audio_format.clone());
}
if touched.has("mic_enabled") {
o.mic_enabled = Some(values.mic_enabled);
}
@@ -1432,6 +1449,28 @@ pub fn show_scoped(
"Stereo or surround — the host downmixes if its output has fewer",
&["Stereo", "5.1 Surround", "7.1 Surround"],
);
let audio_format_labels: Vec<&str> = AUDIO_FORMATS.iter().map(|(_, l)| *l).collect();
let audio_format_row = ChoiceRow::new(
&dialog,
inline,
"Audio format",
"Lossless is uncompressed PCM — 2.3–4.6 Mb/s off the top of the link, and the host has \
its own switch",
&audio_format_labels,
);
{
// Stereo-only, and insensitive rather than hidden under surround: a lossless surround
// frame does not fit one QUIC datagram at the default MTU, so the host declines it
// outright (design/hi-res-audio.md §4.2). Greying it keeps the reason visible next to the
// channel row that caused it — a row that vanished would read as a missing feature.
//
// Insensitivity covers the whole row including a profile scope's per-row Reset, exactly
// as the mic-dependent rows above do: an audio_format override can only be reset while
// the channels row says Stereo.
let w = audio_format_row.widget().clone();
w.set_sensitive(surround_row.selected() == 0);
surround_row.connect_changed(move |i| w.set_sensitive(i == 0));
}
let mic_row = adw::SwitchRow::builder()
.title("Stream microphone")
.subtitle("Sends your microphone to the host's virtual mic — Ctrl+Alt+Shift+V mutes it mid-stream")
@@ -1683,6 +1722,12 @@ pub fn show_scoped(
chroma_row.set_active(s.enable_444);
library_row.set_active(s.library_enabled);
surround_row.set_selected(index::surround(s));
audio_format_row.set_selected(index::audio_format(s));
// `set_selected` never fires the changed hook, so mirror the stereo gate here — the same
// rule the smooth-buffer row's visibility follows a few lines down.
audio_format_row
.widget()
.set_sensitive(index::surround(s) == 0);
let codec_i = index::codec(s);
codec_row.set_selected(codec_i);
set_row_subtitle(codec_row.widget(), codec_caption(codec_i));
@@ -1890,6 +1935,12 @@ pub fn show_scoped(
o.audio_channels.is_some(),
index::surround
);
choice!(
audio_format_row,
"audio_format",
o.audio_format.is_some(),
index::audio_format
);
choice!(pad_row, "gamepad", o.gamepad.is_some(), index::gamepad);
choice!(
sysbtn_row,
@@ -2057,8 +2108,10 @@ pub fn show_scoped(
let audio = page("Audio", "audio-volume-high-symbolic");
let audio_group = group("", "Applies from the next session.");
audio_group.add(surround_row.widget());
audio_group.add(audio_format_row.widget());
// The speaker/mic endpoint pickers below are this device's audio routing (tier G) — they
// render only in the defaults scope; the surround + mic-uplink rows above are profileable.
// render only in the defaults scope; the surround/format + mic-uplink rows above are
// profileable.
if let (Some(r), false) = (&speaker_row, profile_mode) {
audio_group.add(r.widget());
@@ -2210,6 +2263,13 @@ pub fn show_scoped(
2 => 8,
_ => 2,
};
// Written back whatever the channel row says. The stored choice is a preference, not
// a live request — clearing it because the user is on 5.1 today would lose it the
// moment they went back to stereo, and the session filters the pair anyway.
s.audio_format = AUDIO_FORMATS
[(audio_format_row.selected() as usize).min(AUDIO_FORMATS.len() - 1)]
.0
.to_string();
s.codec = CODECS[(codec_row.selected() as usize).min(CODECS.len() - 1)].to_string();
s.present_priority = PRESENT_PRIORITIES
[(present_row.selected() as usize).min(PRESENT_PRIORITIES.len() - 1)]
clients/probe/src/main.rs
+123 -8
View File
@@ -106,6 +106,19 @@ struct Args {
/// multistream-decodes the host's frames and asserts the per-channel sample count, so it's the
/// headless validator for the surround encode path.
audio_channels: u8,
/// `--audio-format opus|lossless48|lossless96|<rate>/<bits>` — what to ask the host for on the
/// audio plane. `opus` (default) sends the UNSPECIFIED sentinel and keeps the legacy `0xC9`
/// wire; anything else asks for the lossless `0xD3` plane at that rate and depth. The rate set
/// is [`punktfunk_core::audio::pcm::rate_is_supported`] — the probe does not restate it.
audio_format: Option<(u32, u8)>,
/// `--audio-out FILE` — write the DECODED audio to `FILE` as raw interleaved `f32` little-endian
/// at the resolved rate and channel count, for offline analysis.
///
/// This is what makes `design/hi-res-audio.md` §13.2 a command instead of a listening session:
/// play a >24 kHz tone on the host, run the probe with `--audio-format lossless96 --audio-out`,
/// and look for energy above 24 kHz in the result. A brick wall there means the host's capture
/// resampled and the session is claiming a rate its content does not have.
audio_out: Option<String>,
/// `--codec h264|hevc|av1|auto` — the preferred video codec (soft; the host honors it when it can
/// emit it, else falls back). The probe always advertises it can decode all three; this just sets
/// the preference byte. `auto` (default) = no preference (host decides). `0` = auto.
@@ -288,6 +301,30 @@ fn parse_args() -> Args {
.and_then(|s| s.parse().ok())
.unwrap_or(2),
),
audio_format: match get("--audio-format").unwrap_or("opus") {
"opus" => None,
"lossless48" => Some((punktfunk_core::audio::SAMPLE_RATE_HZ, 24)),
"lossless96" => Some((96_000, 24)),
spec => {
let (r, b) = spec.split_once('/').unwrap_or((spec, "24"));
match (r.parse::<u32>(), b.parse::<u8>()) {
(Ok(r), Ok(b))
if punktfunk_core::audio::pcm::rate_is_supported(r)
&& punktfunk_core::audio::pcm::depth_is_supported(b) =>
{
Some((r, b))
}
_ => {
eprintln!(
"--audio-format: expected opus | lossless48 | lossless96 | \
<rate>/<bits>, got {spec:?}"
);
std::process::exit(2);
}
}
}
},
audio_out: get("--audio-out").map(String::from),
preferred_codec: match get("--codec").unwrap_or("auto") {
"h264" | "avc" => punktfunk_core::quic::CODEC_H264,
"hevc" | "h265" => punktfunk_core::quic::CODEC_HEVC,
@@ -553,15 +590,36 @@ async fn session(args: Args) -> Result<()> {
// it would just strip the pointer from the dumped bitstream. `--cursor-capture`
// advertises it deliberately and then flips the channel to the capture model, so the
// HOST composites and the dump is where the pointer must appear.
client_caps: if args.cursor_capture {
punktfunk_core::quic::CLIENT_CAP_CURSOR
} else {
0
client_caps: {
let mut c = if args.cursor_capture {
punktfunk_core::quic::CLIENT_CAP_CURSOR
} else {
0
};
// ⚠ This `Hello` is built BY HAND, so it never passes through
// `client::advertised_client_caps` — the helper that derives this bit from the
// requested format for the shipping clients. The rate and depth below are
// therefore inert on their own: the host's gate checks the CAPABILITY first, so a
// format without this bit is a request the host is right to ignore. Set both, or
// neither.
if args.audio_format.is_some() {
c |= punktfunk_core::quic::CLIENT_CAP_AUDIO_HIRES;
}
c
},
// Like STREAMED_AU above: the shared-core reassembler pins geometry per-frame, so
// the probe accepts a mid-session shard change (and jumbo growth) up to the
// receive ceiling — and it's exactly the tool to measure both.
max_shard_payload: punktfunk_core::config::max_shard_payload() as u16,
// `0`/`0` is UNSPECIFIED. For the shipping clients that distinction decides the
// capability bit — `advertised_client_caps` sets it when either field is non-zero,
// because it keys on "the caller specified a format", so that 48 kHz/16-bit (the
// cheapest lossless rung) is not the one format nobody can request. This `Hello` is
// hand-built and does not use that helper, so here the two are set together above and
// the sentinel is about honesty rather than mechanism: an inert format on the wire
// invites exactly the misreading that a rate alone asks for something.
audio_rate_hz: args.audio_format.map(|(r, _)| r).unwrap_or(0),
audio_bits: args.audio_format.map(|(_, b)| b).unwrap_or(0),
}
.encode(),
)
@@ -1243,17 +1301,68 @@ async fn session(args: Args) -> Result<()> {
// Build a multistream decoder for the host-RESOLVED layout so the probe actually decodes
// the surround stream (not just counts bytes) — the headless validator for the encode path.
let audio_channels = welcome.audio_channels;
// The RESOLVED format, off the Welcome — never what was asked for. The host may decline
// hi-res for any of the reasons in §8.4 and answer Opus, and a probe that assumed its own
// request would then mis-parse every datagram it was actually sent.
let audio_codec = welcome.audio_codec;
let audio_rate_hz = welcome.audio_rate_hz;
let audio_bits = welcome.audio_bits;
let audio_out_path = args.audio_out.clone();
tokio::spawn(async move {
use std::sync::atomic::Ordering::Relaxed;
let mut hdr_logged = false;
let mut rumble_logged = false;
let lossless = audio_codec == punktfunk_core::quic::AUDIO_CODEC_PCM;
let layout = punktfunk_core::audio::layout_for(audio_channels, false);
let mut audio_dec =
opus::MSDecoder::new(48_000, layout.streams, layout.coupled, layout.mapping).ok();
let mut audio_dec = (!lossless)
.then(|| {
opus::MSDecoder::new(48_000, layout.streams, layout.coupled, layout.mapping)
.ok()
})
.flatten();
let mut pcm = vec![0f32; 5760 * audio_channels as usize];
let mut lossless_pcm: Vec<f32> = Vec::new();
let mut audio_decoded_logged = false;
let mut audio_out = audio_out_path.as_deref().and_then(|p| {
std::fs::File::create(p)
.map(std::io::BufWriter::new)
.map_err(|e| tracing::error!(path = p, error = %e, "cannot open --audio-out"))
.ok()
});
// Raw interleaved f32 LE, so the reader needs no container — the rate and channel
// count are on the stats line and in this log.
let write_pcm = move |samples: &[f32], w: &mut Option<_>| {
if let Some(f) = w.as_mut() {
let mut bytes = Vec::with_capacity(samples.len() * 4);
for s in samples {
bytes.extend_from_slice(&s.to_le_bytes());
}
let _ = std::io::Write::write_all(f, &bytes);
}
};
while let Ok(d) = conn2.read_datagram().await {
if let Some((_, _, opus)) = punktfunk_core::quic::decode_audio_datagram(&d) {
if let Some((_, _, pcm_wire)) =
punktfunk_core::quic::decode_audio_pcm_datagram(&d).filter(|_| lossless)
{
a.fetch_add(1, Relaxed);
ab.fetch_add(pcm_wire.len() as u64, Relaxed);
if punktfunk_core::audio::pcm::to_f32(pcm_wire, audio_bits, &mut lossless_pcm)
.is_some()
{
if !audio_decoded_logged {
audio_decoded_logged = true;
tracing::info!(
channels = audio_channels,
rate_hz = audio_rate_hz,
bits = audio_bits,
samples_per_channel =
lossless_pcm.len() / audio_channels.max(1) as usize,
"audio decoded (lossless PCM, 0xD3)"
);
}
write_pcm(&lossless_pcm, &mut audio_out);
}
} else if let Some((_, _, opus)) = punktfunk_core::quic::decode_audio_datagram(&d) {
a.fetch_add(1, Relaxed);
ab.fetch_add(opus.len() as u64, Relaxed);
// Decode + validate: the per-channel sample count must be a legal Opus frame
@@ -1267,8 +1376,14 @@ async fn session(args: Args) -> Result<()> {
samples_per_channel = samples,
"audio decoded (Opus multistream)"
);
write_pcm(
&pcm[..samples * audio_channels as usize],
&mut audio_out,
);
}
Ok(samples) => {
write_pcm(&pcm[..samples * audio_channels as usize], &mut audio_out)
}
Ok(_) => {}
Err(e) => tracing::debug!(error = %e, "probe audio decode"),
}
}
clients/session/src/main.rs
+5
View File
@@ -390,6 +390,11 @@ mod session_main {
},
bitrate_kbps: settings.bitrate_kbps,
audio_channels: settings.audio_channels,
// The lossless-audio opt-in, AS STORED — the pump is what filters it, because only it
// knows whether this box's output device will open the rate and what the host
// answered. `PUNKTFUNK_AUDIO_HIRES` still overrides it there (a headless box or a
// Gaming-Mode kiosk has no settings UI), which is why nothing is resolved here.
audio_format: settings.audio_format.clone(),
preferred_codec: settings.preferred_codec(),
// Nothing excluded on a fresh dial. Only the run loop's codec-fallback retry
// sets this, and it does so on a CLONE of these params — a Settings-level
clients/windows/src/app/settings.rs
+50
View File
@@ -15,6 +15,11 @@ use super::style::*;
use super::{AppCtx, Screen};
use crate::trust::{KnownHosts, Settings};
use pf_client_core::profiles::{ProfilesFile, StreamProfile};
// The audio-format table lives in the session crate, not here: the same three stored values also
// have to reach the wire, and they are shared verbatim with the Apple and Android clients so one
// profile round-trips. A second copy of the spellings in this file is exactly the drift the
// shared table exists to prevent — which is why this row has no `const` beside AUDIO_CHANNELS.
use pf_client_core::session::AUDIO_FORMATS;
use pf_client_core::trust::StatsVerbosity;
use punktfunk_core::config::GamepadPref;
use std::sync::Arc;
@@ -484,6 +489,7 @@ struct OverrideFlags {
enable_444: bool,
compositor: bool,
audio_channels: bool,
audio_format: bool,
mic_enabled: bool,
echo_cancel: bool,
touch_mode: bool,
@@ -519,6 +525,7 @@ impl OverrideFlags {
enable_444: o.enable_444.is_some(),
compositor: o.compositor.is_some(),
audio_channels: o.audio_channels.is_some(),
audio_format: o.audio_format.is_some(),
mic_enabled: o.mic_enabled.is_some(),
echo_cancel: o.echo_cancel.is_some(),
touch_mode: o.touch_mode.is_some(),
@@ -1052,6 +1059,13 @@ pub(crate) fn settings_page(
let channels_combo = setting_combo(ctx, scope, (rev, set_rev), ac_names, ac_i, |s, i| {
s.audio_channels = AUDIO_CHANNELS[i].0;
});
// The lossless-audio opt-in. An unknown stored value (a newer client's row, arriving through a
// shared profile) shows as Opus — which is what the session resolves it to as well, so the row
// and the wire agree rather than the combo silently rewriting the user's choice on save.
let (af_names, af_i) = presets(AUDIO_FORMATS, |v| *v == s.audio_format);
let format_combo = setting_combo(ctx, scope, (rev, set_rev), af_names, af_i, |s, i| {
s.audio_format = AUDIO_FORMATS[i].0.to_string();
});
let mic_toggle = setting_toggle(ctx, scope, (rev, set_rev), s.mic_enabled, |s, on| {
s.mic_enabled = on
});
@@ -1506,6 +1520,30 @@ pub(crate) fn settings_page(
"The speaker layout requested from the host. It downmixes if its own \
output has fewer channels.",
)),
// Stereo-only, so the row is HIDDEN under 5.1/7.1 rather than offered and
// declined: a lossless surround frame does not fit one QUIC datagram at the
// default MTU, and the host refuses it outright (design/hi-res-audio.md §4.2).
// The stored value survives the hide — it is a preference, not a live request,
// and going back to Stereo brings it back exactly as it was.
//
// ⚠ Hidden means its per-row Reset is unreachable too. That is the same
// trade the mic-dependent rows make, and it is the lesser evil: a visible
// control for a request this session cannot make is the worse lie.
(s.audio_channels == 2).then(|| {
described_overridable(
(rev, set_rev),
scope,
"audio_format",
"Audio format",
over.audio_format,
format_combo,
"Lossless sends uncompressed PCM instead of Opus \u{2014} bit-exact, \
at 2.3\u{2013}4.6 Mb/s taken off the top of the link and outside \
the automatic-bitrate loop. The host has its own switch, off by \
default, and quietly stays on Opus if it can\u{2019}t deliver the \
rate; the stats overlay names what the session actually got.",
)
}),
// The endpoint picks are facts about THIS device's hardware — never
// per profile, like Decoder/GPU.
(!profile_mode)
@@ -2000,6 +2038,18 @@ mod tests {
assert!(f3.echo_cancel);
assert!(!f3.mic_enabled);
// Channels and format are likewise independent — a "lossless on this host" profile that
// leaves the layout following the global is valid, and the two are separate keys in the
// catalog every client shares.
let mut p3b = StreamProfile::new("t3b".to_string());
p3b.overrides = SettingsOverlay {
audio_format: Some(pf_client_core::session::AUDIO_FORMAT_LOSSLESS_96.into()),
..Default::default()
};
let f3b = OverrideFlags::of(Some(&p3b));
assert!(f3b.audio_format);
assert!(!f3b.audio_channels);
// The presentation pair, likewise independent: pinning the intent doesn't claim
// the buffer (a "Smoothness, whatever the global buffer is" profile is valid).
let mut p4 = StreamProfile::new("t4".to_string());
crates/pf-client-core/src/audio.rs
+207 -17
View File
@@ -2,10 +2,15 @@
//! (PipeWire capture → Opus → 0xCB datagrams, the inverse of the host's virtual mic).
//!
//! Playback mirrors the host's virtual-mic producer (`punktfunk-host::audio::linux`) with
//! the same adaptive jitter buffer: the session pump pushes 5 ms Opus-decoded chunks on
//! the network clock; PipeWire pulls whole quanta on the device clock. Prime to ~3
//! quanta before producing, cap the ring so latency stays bounded, re-prime after a real
//! drain.
//! the same adaptive jitter buffer: the session pump pushes one decoded frame per network
//! arrival; PipeWire pulls whole quanta on the device clock. Prime to ~3 quanta before
//! producing, cap the ring so latency stays bounded, re-prime after a real drain.
//!
//! The stream is opened at the format the session NEGOTIATED ([`PlaybackFormat`]), not at a
//! constant: 48 kHz Opus frames of 5 ms on the `0xC9` plane, or 48/96 kHz lossless PCM frames of
//! 1–5 ms on `0xD3` (`design/hi-res-audio.md`). The graph format stays F32LE at every rate and
//! depth — core decodes both planes to f32, and the reason that is deliberate rather than an
//! oversight is argued at the `stride` in the process callback.
use anyhow::{Context, Result};
use punktfunk_core::client::NativeClient;
@@ -13,6 +18,9 @@ use std::collections::VecDeque;
use std::sync::mpsc::{Receiver, SyncSender, TrySendError};
use std::sync::Arc;
/// The protocol's default rate — and, now that playback takes its rate from the `Welcome`
/// ([`PlaybackFormat`]), the MIC uplink's rate and nothing else. Voice is Opus, and libopus is
/// 48 kHz by construction, so the uplink has no reason to move and no way to.
const SAMPLE_RATE: u32 = 48_000;
/// Mic capture is MONO: voice is mono at the source, the host accepts any Opus channel
/// layout (its stereo decoder upmixes), and half the samples halve the encode + wire cost.
@@ -100,6 +108,40 @@ pub fn devices() -> Result<(Vec<AudioDevice>, Vec<AudioDevice>)> {
Ok(result)
}
/// The playback format a session RESOLVED, straight off the `Welcome` — never what the client
/// asked for. Passed as one value rather than three positional `u32`s because all three are `u32`
/// and transposing them would open the device at a plausible-looking wrong format.
///
/// (Declared in both audio backends rather than shared: `audio.rs` and `audio_wasapi.rs` are twins
/// by design — same public surface, picked by `lib.rs`'s `#[path]` — and every other item on that
/// surface is already spelled out in each.)
#[derive(Clone, Copy, Debug)]
pub struct PlaybackFormat {
/// Interleaved channel count (2/6/8), canonical wire order FL FR FC LFE RL RR SL SR.
pub channels: u32,
/// The negotiated sample rate: 48 000 on every Opus session, 48 000 or 96 000 on a lossless
/// one (`design/hi-res-audio.md` §3 — 44.1 kHz and its multiples are deferred, because they
/// truncate `JitterPolicy`'s integer samples-per-millisecond arithmetic).
pub rate_hz: u32,
/// One protocol frame in microseconds: 5 000 on the Opus plane, and whatever the lossless
/// plane negotiated from the path MTU (§4.2 — 4 ms at 48/24, 2 ms at 96/24 by default). It
/// sizes the graph quantum we ask for and the policy's shed/floor arithmetic.
pub frame_us: u32,
}
impl PlaybackFormat {
/// Frames (per channel) in one protocol frame — the graph quantum to ask for. Computed in µs
/// so a sub-millisecond rung does not truncate: 2 500 µs at 48 kHz is 120 frames, not 96.
fn quantum_frames(&self) -> u32 {
((self.rate_hz as u64 * self.frame_us as u64 / 1_000_000) as u32).max(1)
}
/// Frames (per channel) per millisecond — 48 at the protocol default, 96 at 96 kHz.
fn frames_per_ms(&self) -> usize {
(self.rate_hz / 1000).max(1) as usize
}
}
pub struct AudioPlayer {
pcm_tx: SyncSender<Vec<f32>>,
/// Drained chunk Vecs coming back from the PipeWire consumer for reuse (the pool half
@@ -113,11 +155,14 @@ pub struct AudioPlayer {
}
impl AudioPlayer {
/// Spawn the PipeWire playback thread for `channels` (2/6/8, canonical wire order
/// FL FR FC LFE RL RR SL SR). Failure (no PipeWire in the session) is survivable — the
/// caller streams video-only.
pub fn spawn(channels: u32) -> Result<AudioPlayer> {
// 64 × 5 ms = 320 ms of slack between the pump and the PipeWire loop.
/// Spawn the PipeWire playback thread at the session's RESOLVED format. Failure (no PipeWire
/// in the session) is survivable — the caller streams video-only.
pub fn spawn(fmt: PlaybackFormat) -> Result<AudioPlayer> {
// 64 queued chunks of slack between the pump and the PipeWire loop — 320 ms at the Opus
// plane's 5 ms frame, proportionally less on a lossless session's shorter one (128 ms at
// 2 ms), which is still far above anything the de-jitter policy targets. Left as a chunk
// COUNT rather than scaled to the negotiated frame, matching core's own `AUDIO_QUEUE`,
// whose comment records the same trade.
let (pcm_tx, pcm_rx) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
// Return path: the process callback sends each drained Vec back for reuse, so
// steady-state playback stops allocating (~200 chunks/s otherwise). Same capacity
@@ -129,7 +174,7 @@ impl AudioPlayer {
let thread = std::thread::Builder::new()
.name("punktfunk-audio".into())
.spawn(move || {
if let Err(e) = pw_thread(pcm_rx, recycle_tx, quit_rx, channels as usize, sync_cb) {
if let Err(e) = pw_thread(pcm_rx, recycle_tx, quit_rx, fmt, sync_cb) {
tracing::warn!(error = %e, "audio playback thread ended");
}
})
@@ -180,6 +225,25 @@ impl Drop for AudioPlayer {
pub(crate) const TUNING: punktfunk_core::audio::JitterTuning =
punktfunk_core::audio::JitterTuning::PIPEWIRE;
/// Can this client render a `rate_hz` stream? — the gate on advertising `CLIENT_CAP_AUDIO_HIRES`,
/// which means *capable **and** the user turned it on* (`design/hi-res-audio.md` §7).
///
/// **Always true here, and that is a statement about PipeWire, not a shortcut.** A playback stream
/// declares its own format and the graph inserts an adapter to reconcile it with the sink, so this
/// client can always OPEN at the resolved rate and always renders every sample the host sends. The
/// WASAPI twin genuinely can fail this test, because shared-mode autoconvert reconciles in the
/// other direction — silently, against an engine format we do not control.
///
/// What PipeWire does NOT promise is that the SINK runs at that rate: a 96 kHz stream into a
/// 48 kHz sink is resampled in the graph, and the detail above 24 kHz is gone. That is the
/// client-side shape of the monitor-mode blind spot in §4.4, and reading the sink's own rate needs
/// a registry lookup this crate does not do. The stream logs what the graph actually granted (its
/// `param_changed` handler) so the waste is at least visible; the remedy is the user's own audio
/// configuration, exactly as the endpoint rate is on Windows.
pub fn can_render_at(_rate_hz: u32) -> bool {
true
}
/// Producer-side state: incoming decoded PCM and the ring the process callback drains.
struct PlayerData {
rx: Receiver<Vec<f32>>,
@@ -193,6 +257,12 @@ struct PlayerData {
policy: punktfunk_core::audio::JitterPolicy,
/// Interleaved channel count this stream was opened with (2/6/8).
channels: usize,
/// The format this stream was opened at, so the callback can report its quantum in
/// milliseconds and `param_changed` can check what the graph actually granted against it.
fmt: PlaybackFormat,
/// What `param_changed` last saw, so a graph RESUME (which re-announces the same format) is
/// not logged as a format change — the host's virtual sink learned the same lesson.
negotiated: Option<(u32, u32)>,
/// Diagnostics (WP0.3), logged ~every 10 s: the audio plane used to be entirely silent in a
/// client log, so a latency or dropout report had nothing to go on.
underruns: u64,
@@ -206,7 +276,7 @@ fn pw_thread(
pcm_rx: Receiver<Vec<f32>>,
recycle_tx: SyncSender<Vec<f32>>,
quit_rx: pipewire::channel::Receiver<Terminate>,
channels: usize,
fmt: PlaybackFormat,
sync: Arc<punktfunk_core::audio::AudioSyncCell>,
) -> Result<()> {
use pipewire as pw;
@@ -217,6 +287,8 @@ fn pw_thread(
static PW_INIT: std::sync::Once = std::sync::Once::new();
PW_INIT.call_once(pw::init);
let channels = fmt.channels as usize;
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("pw Context")?;
let core = context
@@ -228,15 +300,22 @@ fn pw_thread(
move |_| mainloop.quit()
});
// The `NODE_LATENCY` ask — one protocol frame, so the graph hands us whole frames and the ring
// (and so the latency) stays small. `<frames>/<rate>` is how PipeWire spells a latency and BOTH
// halves move with the session: this was the string literal `"240/48000"`, which on a 96 kHz
// lossless session would have asked for 240 frames = 2.5 ms — a different quantum than the one
// the frame duration was negotiated for, at double the callback rate, for no reason anyone
// intended. Built at run time for exactly that reason; the `properties!` macro takes literals
// only, so it goes in with `insert` (the same shape as the host's sink-name property).
let node_latency = format!("{}/{}", fmt.quantum_frames(), fmt.rate_hz);
let mut props = properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CATEGORY => "Playback",
*pw::keys::MEDIA_ROLE => "Game",
*pw::keys::NODE_NAME => "punktfunk-client",
*pw::keys::NODE_DESCRIPTION => "Punktfunk Stream",
// ~5 ms quantum (one Opus frame) keeps the ring — and so the latency — small.
*pw::keys::NODE_LATENCY => "240/48000",
};
props.insert(*pw::keys::NODE_LATENCY, node_latency.as_str());
// The Settings speaker pick (session main maps `Settings::speaker_device` here);
// unset/empty = PipeWire's default routing.
if let Ok(target) = std::env::var("PUNKTFUNK_AUDIO_SINK") {
@@ -253,8 +332,24 @@ fn pw_thread(
rx: pcm_rx,
recycle: recycle_tx,
ring: VecDeque::new(),
policy: punktfunk_core::audio::JitterPolicy::new(TUNING, channels as u8),
policy: {
// Both at the RESOLVED format: `new_at_rate` denominates every depth/target/shed
// figure — and the `buffer_ms`/`target_ms` this client reports — in the right
// samples-per-millisecond, and `set_frame_us` tells the two frame-denominated
// decisions (the floor under the effective target, and the one-frame smooth shed) how
// long a frame is here. Left at the defaults, a 96 kHz session would shed 2.5 frames
// at a time and crossfade across a whole one.
let mut p = punktfunk_core::audio::JitterPolicy::new_at_rate(
TUNING,
fmt.channels as u8,
fmt.rate_hz,
);
p.set_frame_us(fmt.frame_us);
p
},
channels,
fmt,
negotiated: None,
underruns: 0,
sheds: 0,
callbacks: 0,
@@ -266,6 +361,45 @@ fn pw_thread(
.state_changed(|_s, _ud, old, new| {
tracing::debug!(?old, ?new, "pipewire playback stream state");
})
// What the graph GRANTED, not what we asked for (`design/hi-res-audio.md` §9's "read back
// the actual rate and do not assume"). ⚠ Read it for what it is: this is OUR port's
// format, and a playback stream's adapter converts it to the sink — so a rate that comes
// back changed means the graph refused our ask outright, while a rate that comes back
// UNCHANGED still says nothing about the sink behind it. The sink's own rate is a registry
// lookup this stream does not do (`can_render_at`).
.param_changed(|_s, ud, id, param| {
let Some(param) = param else { return };
if id != pw::spa::param::ParamType::Format.as_raw() {
return;
}
let mut info = AudioInfoRaw::default();
if info.parse(param).is_err() {
return;
}
let now = (info.rate(), info.channels());
// A resume re-announces the format we already had; that is the graph waking us, not
// the stream changing.
if ud.negotiated == Some(now) {
return;
}
ud.negotiated = Some(now);
if now.0 != 0 && now.0 != ud.fmt.rate_hz {
tracing::warn!(
granted_hz = now.0,
resolved_hz = ud.fmt.rate_hz,
"PipeWire granted a different playback rate than the session negotiated — \
audio will be resampled and the A/V-sync depth arithmetic is denominated in \
the negotiated rate"
);
} else {
tracing::info!(
format = ?info.format(),
rate = now.0,
channels = now.1,
"playback format negotiated"
);
}
})
.process(|stream, ud| {
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let Some(mut buffer) = stream.dequeue_buffer() else {
@@ -287,7 +421,14 @@ fn pw_thread(
// rule, sync was FORBIDDEN from draining it. Capacity is only the ceiling;
// `requested == 0` (no adapter suggestion) falls back to it.
let requested = usize::try_from(buffer.requested()).unwrap_or(0);
let stride = 4 * ud.channels; // F32LE interleaved
// F32LE interleaved, at EVERY rate and depth this client plays — deliberately,
// and not an oversight left behind by the lossless plane. Core decodes 16- and
// 24-bit PCM to f32 (`pcm::to_f32`) precisely so one graph format serves both
// planes; carrying S24 to PipeWire instead would rewrite this whole callback (the
// stride, the ring, the crossfade helper, the policy's sample arithmetic) to
// deliver bits that are already exact in the f32 they arrived in. f32 holds all
// 24 bits of mantissa with room to spare, so nothing is lost by the choice.
let stride = 4 * ud.channels;
let datas = buffer.datas_mut();
if datas.is_empty() {
return;
@@ -308,7 +449,11 @@ fn pw_thread(
requested_frames = requested,
capacity_frames = max_frames,
write_frames = want_frames,
write_ms = want_frames / 48,
// From the session's rate, not from 48: a 96 kHz quantum divided by 48
// reads as twice the latency it is, in the one line an on-glass latency
// report is triaged from.
write_ms = want_frames / ud.fmt.frames_per_ms(),
rate_hz = ud.fmt.rate_hz,
"audio playback quantum"
);
}
@@ -385,7 +530,7 @@ fn pw_thread(
let mut info = AudioInfoRaw::new();
info.set_format(AudioFormat::F32LE);
info.set_rate(SAMPLE_RATE);
info.set_rate(fmt.rate_hz);
info.set_channels(channels as u32);
// Channel positions in canonical wire order (FL FR FC LFE RL RR SL SR) so PipeWire routes each
// slot to the matching speaker (and downmixes when the sink has fewer). Identity, no permute.
@@ -682,3 +827,48 @@ fn mic_thread(
tracing::debug!("pipewire mic capture loop exited");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn fmt(rate_hz: u32, frame_us: u32) -> PlaybackFormat {
PlaybackFormat {
channels: 2,
rate_hz,
frame_us,
}
}
/// `NODE_LATENCY` is `<frames>/<rate>` and BOTH halves have to move with the session, or the
/// graph quantum stops being one protocol frame. The 48 kHz/5 ms row is the literal `240/48000`
/// this replaced — it must still come out byte-identical, because every Opus session depends on
/// it and none of them may change.
#[test]
fn the_graph_quantum_is_one_protocol_frame_at_every_rung() {
// (rate, frame_us) → frames per channel.
for (rate, us, want) in [
(48_000, 5_000, 240), // the Opus plane, unchanged
(48_000, 4_000, 192), // 48 kHz / 24-bit lossless at the default MTU
// The fractional-millisecond rung: 2.5 ms is 120 frames at 48 kHz, and computing it
// in whole milliseconds would truncate to 2 ms and ask for 96.
(48_000, 2_500, 120),
(96_000, 5_000, 480),
(96_000, 3_000, 288), // 96 kHz / 16-bit
(96_000, 2_000, 192), // 96 kHz / 24-bit
] {
assert_eq!(fmt(rate, us).quantum_frames(), want, "{rate} Hz / {us} µs");
}
}
/// The one-shot quantum log divides by this, and reading it off a constant is how a 96 kHz
/// session reports twice the latency it actually has in the line a report is triaged from.
#[test]
fn frames_per_ms_follows_the_negotiated_rate() {
assert_eq!(fmt(48_000, 5_000).frames_per_ms(), 48);
assert_eq!(fmt(96_000, 2_000).frames_per_ms(), 96);
// A nonsense rate must not divide by zero in a log line.
assert_eq!(fmt(0, 5_000).frames_per_ms(), 1);
assert_eq!(fmt(0, 0).quantum_frames(), 1);
}
}
crates/pf-client-core/src/audio_wasapi.rs
+194 -24
View File
@@ -7,12 +7,18 @@
//! WinUI shell's own audio path; that shell's built-in streaming path has since been deleted,
//! so this is now the only WASAPI client ring.
//!
//! Playback: the session pump pushes 5 ms Opus-decoded chunks on the network clock; the WASAPI
//! render thread pulls whole event-driven quanta on the device clock. The depth policy between
//! them is the SHARED `punktfunk_core::audio::JitterPolicy` (`JitterTuning::WASAPI`) — target in
//! Playback: the session pump pushes one decoded frame per network arrival; the WASAPI render
//! thread pulls whole event-driven quanta on the device clock. The depth policy between them is
//! the SHARED `punktfunk_core::audio::JitterPolicy` (`JitterTuning::WASAPI`) — target in
//! milliseconds, crossfaded drift correction, de-prime hysteresis — so all four clients behave
//! the same way and none of them can ratchet latency upward.
//!
//! The endpoint is opened at the format the session NEGOTIATED ([`PlaybackFormat`]), not at a
//! constant: 48 kHz Opus frames of 5 ms on the `0xC9` plane, or 48/96 kHz lossless PCM frames of
//! 1–5 ms on `0xD3` (`design/hi-res-audio.md`). ⚠ Shared-mode `autoconvert` means an over-rate
//! stream is DOWNSAMPLED on arrival with no error — see [`can_render_at`], which is what keeps
//! the capability advertisement honest, and the engine-rate reading in the render thread.
//!
//! WASAPI objects are COM-apartment-bound and not `Send`, so they live on a dedicated
//! thread (the same discipline as the host's `wasapi_cap`); only the channels + stop flag
//! + join handle cross the boundary.
@@ -29,6 +35,9 @@ use wasapi::{
WaveFormat,
};
/// The protocol's default rate — and, now that render takes its rate from the `Welcome`
/// ([`PlaybackFormat`]), the MIC uplink's rate and nothing else. Voice is Opus, and libopus is
/// 48 kHz by construction, so the uplink has no reason to move and no way to.
const SAMPLE_RATE: usize = 48_000;
/// Mic capture requests STEREO from WASAPI (autoconvert matrixes any endpoint layout down to
/// it — the proven path; `read_from_device_to_deque` then delivers our requested format) and
@@ -165,6 +174,105 @@ fn pick_device(
.context("default endpoint")
}
/// The playback format a session RESOLVED, straight off the `Welcome` — never what the client
/// asked for. Passed as one value rather than three positional `u32`s because all three are `u32`
/// and transposing them would open the endpoint at a plausible-looking wrong format.
///
/// (Declared in both audio backends rather than shared: `audio.rs` and `audio_wasapi.rs` are twins
/// by design — same public surface, picked by `lib.rs`'s `#[path]` — and every other item on that
/// surface is already spelled out in each.)
#[derive(Clone, Copy, Debug)]
pub struct PlaybackFormat {
/// Interleaved channel count (2/6/8), canonical wire order FL FR FC LFE RL RR SL SR.
pub channels: u32,
/// The negotiated sample rate: 48 000 on every Opus session, 48 000 or 96 000 on a lossless
/// one (`design/hi-res-audio.md` §3 — 44.1 kHz and its multiples are deferred, because they
/// truncate `JitterPolicy`'s integer samples-per-millisecond arithmetic).
pub rate_hz: u32,
/// One protocol frame in microseconds: 5 000 on the Opus plane, and whatever the lossless
/// plane negotiated from the path MTU (§4.2 — 4 ms at 48/24, 2 ms at 96/24 by default). It
/// feeds the policy's shed/floor arithmetic, which is denominated in frames.
pub frame_us: u32,
}
/// The render endpoint's own engine rate, or `None` when nothing readable answered.
///
/// ⚠ **This is the client-side twin of the capture trap in `design/hi-res-audio.md` §4.3, and it
/// is the reason this function exists at all.** The render client below initialises with
/// `autoconvert: true`, and in shared mode the ENGINE's mix format is authoritative: autoconvert
/// exists to reconcile our format with the engine's, in whichever direction is needed. So handing
/// a 48 kHz engine a 96 kHz stream does not fail — it succeeds, returns no error, and plays
/// interpolated-back-down samples, while the session spends 3–4 Mbps carrying detail that is
/// discarded on arrival. Both ends would audit clean and the content would be wrong, which is
/// exactly the shape of bug this project has been burned by before.
///
/// Runs on a short-lived MTA thread, like [`devices`]: the caller is the session pump, whose COM
/// apartment is not ours to claim for the rest of the process.
fn render_engine_rate_hz() -> Option<u32> {
std::thread::Builder::new()
.name("pf-audio-engine".into())
.spawn(|| -> Option<u32> {
if wasapi::initialize_mta().ok().is_err() {
return None;
}
let enumerator = DeviceEnumerator::new().ok()?;
// The endpoint the render thread WILL pick, not the default — a picked USB DAC and
// the system default routinely run at different rates.
let device =
pick_device(&enumerator, &Direction::Render, "PUNKTFUNK_AUDIO_SINK").ok()?;
let client = device.get_iaudioclient().ok()?;
client.get_mixformat().ok().map(|f| f.get_samplespersec())
})
.ok()?
.join()
.ok()?
}
/// Can this client render a `rate_hz` stream? — the gate on advertising `CLIENT_CAP_AUDIO_HIRES`,
/// which means *capable **and** the user turned it on* (`design/hi-res-audio.md` §7). A client
/// that advertised it without being able to render it would spend bandwidth off the top of a link
/// ABR can neither see nor reclaim, to play interpolation.
///
/// Answered from the endpoint's own mix format — never assumed, never padded. An engine below
/// the asked-for rate, or an endpoint that will not say what it runs at, both DECLINE: refusing
/// here costs a hi-res session and can never cost a working 48 kHz one, which is the same trade
/// the host makes at the other end (§8.2). The operator's lever is Windows' own device
/// properties — set the endpoint's rate there and this sees it. Driving the engine format from the
/// client would fight the OS and every other application on the box.
///
/// BLOCKS on COM while it asks (the [`devices`] discipline — a few ms against a healthy audio
/// service), and runs on the connect path, so the early return below matters: only a request
/// ABOVE the legacy rate ever touches the endpoint. Every ordinary session, and every 48 kHz
/// lossless one, answers without opening anything.
pub fn can_render_at(rate_hz: u32) -> bool {
if rate_hz <= SAMPLE_RATE as u32 {
return true; // the baseline claim every session already makes
}
match render_engine_rate_hz() {
Some(hz) if hz >= rate_hz => true,
Some(hz) => {
tracing::warn!(
engine_hz = hz,
requested = rate_hz,
"the render endpoint's audio engine runs below the requested rate — not asking \
for lossless audio, because WASAPI's shared-mode autoconvert would downsample it \
on arrival and the bandwidth would buy nothing (raise the rate in Windows' Sound \
→ Device properties → Advanced to change this)"
);
false
}
None => {
tracing::warn!(
requested = rate_hz,
"the render endpoint would not report its engine mix format — not asking for \
lossless audio, because there is no way to tell whether it would be downsampled \
on arrival"
);
false
}
}
}
pub struct AudioPlayer {
pcm_tx: SyncSender<Vec<f32>>,
/// Drained chunk Vecs coming back from the render thread for reuse (the pool half of
@@ -178,35 +286,47 @@ pub struct AudioPlayer {
}
impl AudioPlayer {
/// Spawn the WASAPI render thread for `channels` (2/6/8, canonical wire order
/// FL FR FC LFE RL RR SL SR). Failure (no render endpoint on this box) is survivable — the
/// caller streams video-only.
pub fn spawn(channels: u32) -> Result<AudioPlayer> {
// 64 × 5 ms = 320 ms of slack between the pump and the WASAPI loop.
/// Spawn the WASAPI render thread at the session's RESOLVED format. Failure (no render
/// endpoint on this box) is survivable — the caller streams video-only.
pub fn spawn(fmt: PlaybackFormat) -> Result<AudioPlayer> {
// 64 queued chunks of slack between the pump and the WASAPI loop — 320 ms at the Opus
// plane's 5 ms frame, proportionally less on a lossless session's shorter one (128 ms at
// 2 ms), which is still far above anything the de-jitter policy targets. Left as a chunk
// COUNT rather than scaled to the negotiated frame, matching core's own `AUDIO_QUEUE`,
// whose comment records the same trade.
let (pcm_tx, pcm_rx) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
// Return path: the render thread sends each drained Vec back for reuse, so
// steady-state playback stops allocating (~200 chunks/s otherwise). Same capacity
// as the data channel; a full pool just drops the Vec (plain deallocation).
let (recycle_tx, recycle_rx) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
let stop = Arc::new(AtomicBool::new(false));
let (ready_tx, ready_rx) = std::sync::mpsc::sync_channel::<Result<()>>(1);
// The engine rate the render thread read, so the line below reports what this stream is
// actually up against rather than a constant. `None` = the endpoint said nothing.
let (ready_tx, ready_rx) = std::sync::mpsc::sync_channel::<Result<Option<u32>>>(1);
let stop_t = stop.clone();
let sync: Arc<punktfunk_core::audio::AudioSyncCell> = Arc::default();
let sync_t = sync.clone();
let thread = std::thread::Builder::new()
.name("punktfunk-audio".into())
.spawn(move || {
if let Err(e) =
render_thread(pcm_rx, recycle_tx, stop_t, ready_tx, channels as u8, sync_t)
{
if let Err(e) = render_thread(pcm_rx, recycle_tx, stop_t, ready_tx, fmt, sync_t) {
tracing::warn!(error = %format!("{e:#}"), "audio playback thread ended");
}
})
.context("spawn audio thread")?;
match ready_rx.recv_timeout(Duration::from_secs(3)) {
Ok(Ok(())) => {
// Default endpoint unless PUNKTFUNK_AUDIO_SINK picked one (logged there).
tracing::info!(channels, "WASAPI render: 48 kHz f32");
Ok(Ok(engine_hz)) => {
// Default endpoint unless PUNKTFUNK_AUDIO_SINK picked one (logged there). Every
// number here is the one this stream really opened with — the line used to read
// "48 kHz f32" from a constant, which on a 96 kHz session would have been the
// label-right/content-wrong shape the whole hi-res design is written against.
tracing::info!(
channels = fmt.channels,
rate_hz = fmt.rate_hz,
frame_us = fmt.frame_us,
engine_hz,
"WASAPI render: 32-bit float"
);
Ok(AudioPlayer {
pcm_tx,
recycle_rx,
@@ -257,8 +377,8 @@ fn render_thread(
pcm_rx: Receiver<Vec<f32>>,
recycle_tx: SyncSender<Vec<f32>>,
stop: Arc<AtomicBool>,
ready: SyncSender<Result<()>>,
channels: u8,
ready: SyncSender<Result<Option<u32>>>,
fmt: PlaybackFormat,
sync: Arc<punktfunk_core::audio::AudioSyncCell>,
) -> Result<()> {
if let Err(e) = wasapi::initialize_mta()
@@ -268,14 +388,55 @@ fn render_thread(
let _ = ready.send(Err(e));
return Ok(());
}
let res = (|| -> Result<()> {
// F32LE interleaved: channels × 4 bytes/sample. Stereo (channels == 2) is byte-identical
let res = (|| -> Result<Option<u32>> {
let channels = fmt.channels.clamp(1, 8) as u8;
// 32-bit float interleaved: channels × 4 bytes/sample, at EVERY rate and depth this client
// plays — deliberately, and not an oversight left behind by the lossless plane. Core
// decodes 16- and 24-bit PCM to f32 (`pcm::to_f32`) precisely so one render format serves
// both planes; asking WASAPI for a 24-bit integer format instead would rewrite this whole
// loop (block align, the ring, the crossfade helper, the policy's sample arithmetic) to
// deliver bits that are already exact in the f32 they arrived in. Stereo is byte-identical
// to the old fixed path (mask 0x3, block align 8).
let block_align = channels as usize * 4;
let enumerator = DeviceEnumerator::new().context("DeviceEnumerator")?;
let device = pick_device(&enumerator, &Direction::Render, "PUNKTFUNK_AUDIO_SINK")
.context("render endpoint")?;
let mut audio_client = device.get_iaudioclient().context("IAudioClient")?;
// The endpoint's ACTUAL engine mix format, read BEFORE we initialise — the client-side
// twin of the capture reading in `design/hi-res-audio.md` §4.3/§8.2, and the one §9 asks
// for by name. `autoconvert` below makes an over-rate stream succeed silently, so without
// this line the session could carry 96 kHz, log 96 kHz, spend the bandwidth, and render
// 48 kHz interpolation with nothing above 24 kHz in it.
//
// This is a REPORT, not a gate: by the time this thread runs the wire format is already
// negotiated and the session is streaming, so declining would only mean silence. The gate
// is `can_render_at`, which runs BEFORE the connect and is what keeps the capability bit
// honest; reaching a mismatch here means the endpoint changed under us (a picked device
// unplugged, a shared-mode rate changed mid-session), which is worth a loud line.
let engine_hz = audio_client
.get_mixformat()
.ok()
.map(|f| f.get_samplespersec())
.filter(|&hz| hz > 0);
if let Some(hz) = engine_hz {
if hz < fmt.rate_hz {
tracing::warn!(
engine_hz = hz,
stream_hz = fmt.rate_hz,
endpoint = %device.get_friendlyname().unwrap_or_default(),
"the render endpoint's audio engine runs BELOW this session's negotiated \
rate — WASAPI's shared-mode autoconvert is downsampling every frame on \
arrival, so the extra bandwidth is being spent for nothing (raise the rate \
in Windows' Sound → Device properties → Advanced, then reconnect)"
);
}
} else if fmt.rate_hz != SAMPLE_RATE as u32 {
tracing::warn!(
stream_hz = fmt.rate_hz,
"the render endpoint would not report its engine mix format — there is no way to \
tell whether this session's audio is being downsampled on arrival"
);
}
// The explicit dwChannelMask is the wire order (FL FR FC LFE RL RR SL SR); 5.1 = 0x3F,
// 7.1 = 0x63F. WASAPI delivers channels in ascending mask-bit order, which equals the wire
// order, so the render mapping is the identity — no permute. `autoconvert` (below) lets the
@@ -284,7 +445,7 @@ fn render_thread(
32,
32,
&SampleType::Float,
SAMPLE_RATE,
fmt.rate_hz as usize,
channels as usize,
Some(punktfunk_core::audio::wasapi_channel_mask(channels)),
);
@@ -302,7 +463,7 @@ fn render_thread(
.get_audiorenderclient()
.context("IAudioRenderClient")?;
audio_client.start_stream().context("start render stream")?;
let _ = ready.send(Ok(()));
let _ = ready.send(Ok(engine_hz));
// De-jitter ring, in interleaved f32 SAMPLES (it used to be raw bytes, which made the
// depth arithmetic byte-vs-sample and kept it from sharing the policy and the crossfade
@@ -312,7 +473,16 @@ fn render_thread(
// returns to target instead of ratcheting, and de-prime hysteresis — the last replacing
// the old `if ring.is_empty()`, where a single transient drain manufactured a whole
// target's worth of fresh silence.
let mut policy = punktfunk_core::audio::JitterPolicy::new(TUNING, channels);
//
// Both at the RESOLVED format: `new_at_rate` denominates every depth/target/shed figure —
// and the `buffer_ms`/`target_ms` this client reports — in the right samples-per-
// millisecond, and `set_frame_us` tells the two frame-denominated decisions (the floor
// under the effective target, and the one-frame smooth shed) how long a frame is here.
// Left at the defaults, a 96 kHz session would shed 2.5 frames at a time and crossfade
// across a whole one.
let mut policy =
punktfunk_core::audio::JitterPolicy::new_at_rate(TUNING, channels, fmt.rate_hz);
policy.set_frame_us(fmt.frame_us);
let mut out = Vec::new(); // per-quantum scratch, reused across iterations
let (mut underruns, mut sheds, mut callbacks) = (0u64, 0u64, 0u64);
@@ -383,12 +553,12 @@ fn render_thread(
.context("write_to_device")?;
}
audio_client.stop_stream().ok();
Ok(())
Ok(engine_hz)
})();
if let Err(ref e) = res {
let _ = ready.send(Err(anyhow!("{e:#}")));
}
res
res.map(|_| ())
}
/// The microphone uplink: capture the default input device, Opus-encode 10 ms mono chunks,
crates/pf-client-core/src/profiles.rs
+63
View File
@@ -59,6 +59,13 @@ pub struct SettingsOverlay {
pub compositor: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub audio_channels: Option<u8>,
/// The requested audio format (`crate::session::AUDIO_FORMATS`' stored value). Profileable
/// because it is about how a HOST is streamed — a wired desktop on the LAN can afford the
/// 2.3–4.6 Mbps lossless takes off the top of the link, the same laptop on a hotel Wi-Fi
/// cannot — rather than about this device's hardware. Apple and Android carry it under the
/// same key, so one catalog covers all four clients.
#[serde(skip_serializing_if = "Option::is_none")]
pub audio_format: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub mic_enabled: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -139,6 +146,9 @@ impl SettingsOverlay {
if let Some(v) = self.audio_channels {
s.audio_channels = v;
}
if let Some(v) = &self.audio_format {
s.audio_format = v.clone();
}
if let Some(v) = self.mic_enabled {
s.mic_enabled = v;
}
@@ -238,6 +248,9 @@ impl SettingsOverlay {
if after.audio_channels != before.audio_channels {
self.audio_channels = Some(after.audio_channels);
}
if after.audio_format != before.audio_format {
self.audio_format = Some(after.audio_format.clone());
}
if after.mic_enabled != before.mic_enabled {
self.mic_enabled = Some(after.mic_enabled);
}
@@ -310,6 +323,7 @@ impl SettingsOverlay {
"enable_444" => self.enable_444 = None,
"compositor" => self.compositor = None,
"audio_channels" => self.audio_channels = None,
"audio_format" => self.audio_format = None,
"mic_enabled" => self.mic_enabled = None,
"echo_cancel" => self.echo_cancel = None,
"touch_mode" => self.touch_mode = None,
@@ -656,6 +670,55 @@ mod tests {
assert!(o.is_empty());
}
/// `audio_format` is a first-class overlay field, not an `extra` passenger: it applies,
/// absorbs, clears, and serialises under the `audio_format` key with the exact raw values the
/// Apple and Android clients write — one catalog has to round-trip through all four.
///
/// The failure this pins is silent. An unmodelled key survives a load→save (that is what
/// `extra` is for), so a profile authored on a phone would keep working on a TV and keep
/// round-tripping through this client while quietly never applying — a "lossless on the
/// living-room host" profile that streams Opus, with nothing anywhere saying so.
#[test]
fn audio_format_is_a_first_class_override() {
let base = Settings::default();
assert_eq!(
base.audio_format,
crate::session::AUDIO_FORMAT_OPUS,
"the setting ships off"
);
let mut o = SettingsOverlay::default();
let before = o.apply(&base);
let mut after = before.clone();
after.audio_format = crate::session::AUDIO_FORMAT_LOSSLESS_96.into();
o.absorb(&before, &after);
assert_eq!(o.audio_format.as_deref(), Some("lossless96"));
assert_eq!(o.apply(&base).audio_format, "lossless96");
assert!(
o.extra.is_empty(),
"modelled fields must never land in the passthrough"
);
// Serialised under the shared key, and read back from a foreign client's file — the two
// spellings a phone/TV profile can hold.
let text = serde_json::to_string(&o).unwrap();
assert!(text.contains("\"audio_format\":\"lossless96\""), "{text}");
let from_android: SettingsOverlay =
serde_json::from_str(r#"{"audio_channels":2,"audio_format":"lossless48"}"#).unwrap();
assert_eq!(from_android.audio_format.as_deref(), Some("lossless48"));
assert!(from_android.extra.is_empty());
assert_eq!(from_android.apply(&base).audio_format, "lossless48");
assert!(o.clear("audio_format"));
assert_eq!(o.audio_format, None);
assert!(o.is_empty());
// Back to inheriting the global — not a remembered "lossless96".
assert_eq!(
o.apply(&base).audio_format,
crate::session::AUDIO_FORMAT_OPUS
);
}
/// The presentation cluster is first-class, not `extra` passengers: it applies,
/// absorbs, clears, and serialises under the exact keys the Apple client already
/// writes (`present_priority`/`smooth_buffer`/`vsync`/`allow_vrr`) — one catalog
crates/pf-client-core/src/session.rs
+720 -72
View File
@@ -31,6 +31,19 @@ pub struct SessionParams {
pub bitrate_kbps: u32,
/// Requested audio channel count (2/6/8); the host echoes the resolved value.
pub audio_channels: u8,
/// The requested audio format — a stored [`AUDIO_FORMATS`] value
/// ([`crate::trust::Settings::audio_format`]), `"opus"` for every ordinary session.
///
/// A REQUEST, never a fact. It is filtered here against what this box can play and what
/// [`audio_channels`](Self::audio_channels) resolved to, then the HOST runs its own
/// five-condition gate and may answer Opus anyway — read `NativeClient::audio_codec` /
/// `_sample_rate_hz` / `_bits` for what actually happened. A `String` rather than an enum
/// because it comes straight out of a settings file a newer client may have written; an
/// unknown value resolves to Opus rather than refusing a connect over a dropdown.
///
/// `PUNKTFUNK_AUDIO_HIRES` overrides it — see `requested_audio_format` (not linked: it is
/// private, and a public item may not link into the crate's internals).
pub audio_format: String,
/// The user's preferred video codec (a `quic::CODEC_*` bit, `0` = auto). Soft — the host honors
/// it when it can emit it, else falls back; the resolved codec drives the decoder.
pub preferred_codec: u8,
@@ -224,6 +237,24 @@ pub struct Stats {
/// overhaul is judged by — an absolute buffer depth cannot distinguish "deep because the link
/// needs it" from "deep and therefore late".
pub audio_av_offset_ms: i32,
/// The host RESOLVED the lossless `0xD3` PCM plane for this session (`AUDIO_CODEC_PCM`);
/// false on the Opus plane every ordinary session runs.
///
/// The RESOLVED format, emphatically not the requested one — the whole reason it is published.
/// The Settings screen shows what this device ASKED for, and the host's five-condition gate
/// (`design/hi-res-audio.md` §8.4, and its own switch is off by default) can decline every one
/// of them, leaving a session that looks, sounds and measures exactly like a granted one. An
/// OSD reading "lossless" on a session the host refused is §4.3's bug wearing a different hat,
/// and this is the only surface that can answer it.
pub audio_lossless: bool,
/// The RESOLVED sample rate (Hz) and sample depth (bits) of the audio plane — what the decoder
/// and the output device were actually built from, straight off the Welcome.
///
/// `0` means the host said nothing, which an old host always does; a renderer must treat that
/// as "no reading" rather than as a rate (`spawn_audio` folds it to the legacy 48 kHz for its
/// own arithmetic, but the OSD has nothing honest to print).
pub audio_rate_hz: u32,
pub audio_bits: u8,
/// The decode path frames actually took this window (`"vaapi"`/`"software"`, empty
/// until the first frame) — the OSD's trailing tag; tracks a mid-session fallback.
pub decoder: &'static str,
@@ -470,42 +501,310 @@ pub fn now_ns() -> u64 {
.unwrap_or(0)
}
/// Opus decoder for the audio plane: a plain stereo decoder (the validated path) or a multistream
/// decoder for 5.1/7.1, both behind one `decode_float`. Built from the host-RESOLVED channel count
/// via the shared layout table.
enum AudioDec {
/// The session's audio decoder — the `0xC9` Opus plane or the `0xD3` lossless PCM one, behind one
/// pair of methods so the pull loop below is plane-agnostic.
///
/// Which plane runs is decided ONCE, from `Welcome::audio_codec`, and never changes mid-session:
/// the output device is open at a fixed format, so a switch would mean a re-open (design
/// `hi-res-audio.md` §6). Nothing per-packet says which plane a datagram came from — the two share
/// a header by design — so this type is the only thing that knows.
///
/// Both arms hand back INTERLEAVED sample counts (libopus counts per channel, `pcm::to_f32` counts
/// interleaved); unifying on interleaved here is what lets the loop size its pushes, its
/// concealment and its ring reporting from one number.
struct AudioDec {
/// The host-RESOLVED channel count, needed to turn libopus's per-channel counts into
/// interleaved ones. The PCM arm needs it only through the frame sizing the caller does.
channels: usize,
kind: DecKind,
}
enum DecKind {
/// Plain stereo libopus — the validated path.
Stereo(opus::Decoder),
/// Multistream libopus for 5.1/7.1, built from the shared layout table.
Surround(opus::MSDecoder),
/// The lossless plane: no codec and no decoder state, only the negotiated depth the wire bytes
/// are unpacked at — plus the concealer, because **a lossless format has no PLC to borrow**
/// (§4.5). libopus can synthesise a missing frame from its own internal state; there is
/// nothing in a raw PCM frame to synthesise a successor from, so `PcmConceal` repeats and
/// fades instead.
Pcm {
bits: u8,
conceal: punktfunk_core::audio::pcm::PcmConceal,
},
}
impl AudioDec {
fn new(channels: u8) -> Result<AudioDec, opus::Error> {
if channels == 2 {
Ok(AudioDec::Stereo(opus::Decoder::new(
48_000,
opus::Channels::Stereo,
)?))
/// Build the decoder for the plane the host RESOLVED — `codec`/`rate_hz`/`bits` all come off
/// the `Welcome`, never off what this client asked for.
fn new(codec: u8, channels: u8, rate_hz: u32, bits: u8) -> Result<AudioDec, opus::Error> {
let ch = channels.max(1) as usize;
// A lossless session never reaches libopus, so it never has to justify its rate to it:
// libopus accepts 8/12/16/24/48 kHz and nothing else, which is the whole reason the
// hi-res ladder needed a second plane rather than a parameter (§2).
if codec == punktfunk_core::quic::AUDIO_CODEC_PCM {
// The depth is the STRIDE the wire is unpacked at, so a value the plane does not
// define is not a cosmetic problem: core reads anything that is not 16 as 24, and a
// mismatched stride desyncs every sample after the first. Say so rather than play
// noise — the session still runs, because refusing would mean silence and the
// negotiation should never produce this in the first place.
if !punktfunk_core::audio::pcm::depth_is_supported(bits) {
tracing::warn!(
bits,
"the host resolved a lossless depth this plane does not define — unpacking \
as 24-bit, which will be wrong if it meant anything else"
);
}
return Ok(AudioDec {
channels: ch,
kind: DecKind::Pcm {
bits,
conceal: punktfunk_core::audio::pcm::PcmConceal::new(),
},
});
}
// The Opus plane is 48 kHz by construction, and `Welcome::audio_rate_hz` says so for every
// Opus session. Taking it from the Welcome anyway (rather than repeating the literal
// twice, as this did) means the decoder cannot disagree with the ring and the A/V-sync
// loop about what a millisecond is.
let kind = if channels == 2 {
DecKind::Stereo(opus::Decoder::new(rate_hz, opus::Channels::Stereo)?)
} else {
let l = punktfunk_core::audio::layout_for(channels, false);
Ok(AudioDec::Surround(opus::MSDecoder::new(
48_000, l.streams, l.coupled, l.mapping,
)?))
DecKind::Surround(opus::MSDecoder::new(
rate_hz, l.streams, l.coupled, l.mapping,
)?)
};
Ok(AudioDec { channels: ch, kind })
}
/// Decode one arrived frame into `out`, returning its INTERLEAVED sample count (the caller
/// reads `out[..n]`).
///
/// `out` is the caller's scratch. The Opus arms decode into it as a fixed slice, so it must
/// already be long enough for the biggest frame the plane can carry; the PCM arm hands the Vec
/// to `pcm::to_f32`, which clears and grows it to the frame's true length — so a malformed
/// oversized datagram cannot overrun it there.
fn decode(&mut self, input: &[u8], out: &mut Vec<f32>) -> Option<usize> {
let channels = self.channels;
match &mut self.kind {
DecKind::Stereo(d) => d.decode_float(input, out, false).ok().map(|n| n * channels),
DecKind::Surround(d) => d.decode_float(input, out, false).ok().map(|n| n * channels),
DecKind::Pcm { bits, conceal } => {
// `None` here is a truncated datagram — a partial sample at the end would desync
// every sample after it, so core rejects it outright rather than decoding a
// shifted frame. Treated as a lost frame by the caller, which is what it is.
let n = punktfunk_core::audio::pcm::to_f32(input, *bits, out)?;
conceal.accept(&out[..n]);
Some(n)
}
}
}
fn decode_float(
&mut self,
input: &[u8],
out: &mut [f32],
fec: bool,
) -> Result<usize, opus::Error> {
match self {
AudioDec::Stereo(d) => d.decode_float(input, out, fec),
AudioDec::Surround(d) => d.decode_float(input, out, fec),
/// Synthesise one frame for a datagram that never arrived, into `out`; `Some(n)` = `out[..n]`
/// is playable, `None` = nothing could be built (no frame has decoded yet) and the caller
/// should let the ring re-prime.
///
/// `interleaved` is the last good frame's length — the unit both planes conceal in. The Opus
/// arm needs it because libopus PLC synthesises exactly the slice it is handed; the PCM arm
/// ignores it, because `PcmConceal` already holds the frame it is repeating.
fn conceal(&mut self, interleaved: usize, out: &mut Vec<f32>) -> Option<usize> {
let channels = self.channels;
match &mut self.kind {
// `PcmConceal` already holds the frame it repeats, so it needs no size hint — and it
// reports `false` when nothing has arrived yet to repeat from. The receiver runs
// before the argument, so the length read here is the concealed frame's, not the
// previous call's.
DecKind::Pcm { conceal, .. } => conceal.conceal(out).then_some(out.len()),
libopus => {
// libopus PLC synthesises exactly the slice it is handed; before anything has
// decoded there is no frame length to ask it for.
let plc = interleaved.min(out.len());
if plc == 0 {
return None;
}
let per_ch = match libopus {
DecKind::Stereo(d) => d.decode_float(&[], &mut out[..plc], false).ok()?,
DecKind::Surround(d) => d.decode_float(&[], &mut out[..plc], false).ok()?,
DecKind::Pcm { .. } => unreachable!("the PCM arm matched above"),
};
Some(per_ch * channels)
}
}
}
}
/// The `audio_format` setting's stored value for the Opus plane — the default, and byte for byte
/// the session every build before the lossless plane ran.
pub const AUDIO_FORMAT_OPUS: &str = "opus";
/// Bit-exact PCM at 48 kHz / 24-bit (~2.3 Mbps). The honest win even without a hi-res interface:
/// no lossy stage at all, and no double resample on a host whose engine already runs at 48 kHz.
pub const AUDIO_FORMAT_LOSSLESS_48: &str = "lossless48";
/// Bit-exact PCM at 96 kHz / 24-bit (~4.6 Mbps), and only real if the host's capture endpoint
/// genuinely runs at 96 kHz — the host declines rather than upsampling to meet the request.
pub const AUDIO_FORMAT_LOSSLESS_96: &str = "lossless96";
/// `(stored value, label)` for the requested audio format — the cross-client table both desktop
/// settings UIs render, so the two shells can never drift from each other or from the wire.
///
/// ⚠ **The stored values are shared VERBATIM with the Apple client's `AudioFormatChoice` raw
/// values and the Android client's `AUDIO_FORMAT_*`.** One profile catalog round-trips through all
/// four clients (`profiles.rs`), and a spelling that differs by a single character fails in the
/// worst possible way: the key is carried through untouched, so a profile written on a phone would
/// keep working on a TV and silently inherit the global default here. Change these only in lockstep
/// with `clients/apple/Sources/PunktfunkShared/EffectiveSettings.swift` and
/// `clients/android/app/src/main/kotlin/io/unom/punktfunk/Settings.kt`.
///
/// **The ladder is 48/96 kHz only, and that is arithmetic rather than bandwidth.** Core's jitter
/// policy sizes every buffer as `ms × samples-per-ms` with an INTEGER per-ms: 48 000 → 48 and
/// 96 000 → 96 are exact, 44 100 → 44.1 truncates to 44 — a silent 2.3 % error in every target and
/// every reported depth. 44.1 kHz and its multiples are deferred behind reworking that arithmetic
/// (`design/hi-res-audio.md` §4.1), and the host would decline them regardless.
///
/// Lossless at 48 kHz / **16**-bit is deliberately absent from the menu even though the env
/// override below can still ask for it: it spends ~1.5 Mbps to sound like the transparent 256 kbps
/// Opus it replaces. 24-bit is where the plane earns its bandwidth.
pub const AUDIO_FORMATS: &[(&str, &str)] = &[
(AUDIO_FORMAT_OPUS, "Standard (Opus)"),
(AUDIO_FORMAT_LOSSLESS_48, "Lossless 48 kHz / 24-bit"),
(AUDIO_FORMAT_LOSSLESS_96, "Lossless 96 kHz / 24-bit"),
];
/// The `(rate_hz, bits)` a stored [`AUDIO_FORMATS`] value asks the host for; `None` = the Opus
/// plane, which the caller must turn into the unspecified `0`/`0` pair on the wire rather than an
/// explicit 48 000/16 — core reads any non-zero pair as "this client is asking for the lossless
/// plane", so a literal legacy pair would advertise hi-res on every ordinary session.
///
/// An unrecognized value — a newer client's row, or a corrupted settings file — resolves to Opus
/// rather than blocking the connect, matching what the Apple and Android ports do with the same
/// string. Deriving the pair FROM the stored value is what stops the menu row and the format ever
/// disagreeing.
pub fn audio_format_wire(setting: &str) -> Option<(u32, u8)> {
use punktfunk_core::audio::pcm::BITS_24;
match setting {
AUDIO_FORMAT_LOSSLESS_48 => Some((48_000, BITS_24)),
AUDIO_FORMAT_LOSSLESS_96 => Some((96_000, BITS_24)),
_ => None,
}
}
/// The lossless format this client ASKS the host for — `Some((rate_hz, bits))` when it is on,
/// `None` for the legacy Opus plane.
///
/// Two inputs, and **the environment wins**: `PUNKTFUNK_AUDIO_HIRES` overrides `setting` (the
/// user's stored [`AUDIO_FORMATS`] choice, already resolved through any settings profile).
///
/// That direction, not the reverse, for two reasons. It is how this crate treats every other
/// `PUNKTFUNK_*` lever — `PUNKTFUNK_DECODER` beats `Settings::decoder`, `PUNKTFUNK_NO_AEC` beats
/// `echo_cancel`, `PUNKTFUNK_CLIENT_PEAK_NITS` beats the panel's own volume — and a lever that
/// lost to whatever a stale profile happened to hold would be useless for the thing operators
/// actually use it for (A/B-ing one session against a field report). And the surfaces do not
/// overlap: a headless box, a Gaming-Mode kiosk and the CI probe have no settings UI at all, which
/// is exactly why the var is documented for operators rather than being removed here.
///
/// Env grammar: `1`/`true`/`on`/`yes` → 96 kHz / 24-bit (the flagship rung); `96000` → that rate at
/// 24-bit; `<48000|96000>/<16|24>` → an explicit pair, `48000/16` included — that is the cheapest
/// lossless rung and it is genuinely reachable, see [`AUDIO_FORMAT_UNSPECIFIED`], though no menu
/// row offers it. `0`/`off`/`false`/`no` force the Opus plane even when the setting asks for
/// lossless: an override that could only ever turn the feature ON would be half a lever.
///
/// ⚠ An UNSET var and an UNPARSEABLE one are not the same thing, and neither is "off". Unset means
/// the operator said nothing, so the setting decides. A typo is not an instruction either — it is
/// warned about and then IGNORED, so the setting still decides; the alternative (the pre-settings
/// behaviour, where garbage meant off) would silently defeat a switch the user had turned on in the
/// UI, which is the worse of the two failures now that there IS a UI.
fn requested_audio_format(setting: &str) -> Option<(u32, u8)> {
resolve_audio_format(
std::env::var("PUNKTFUNK_AUDIO_HIRES").ok().as_deref(),
setting,
)
}
/// The precedence half of [`requested_audio_format`], split out so the env-beats-setting rule is
/// testable without mutating the process environment — the same reason [`parse_audio_format`] is
/// its own function.
fn resolve_audio_format(env: Option<&str>, setting: &str) -> Option<(u32, u8)> {
let Some(raw) = env else {
return audio_format_wire(setting);
};
match parse_audio_format(raw) {
AudioRequest::Legacy => None,
AudioRequest::Hires(rate, bits) => Some((rate, bits)),
AudioRequest::Unsupported => {
// Loud, because the user set a lever and is not getting it — the same reason the
// 4:4:4 fallback in `clients/session` shouts. What it falls back TO is the Settings
// choice, which is why the message names it rather than promising Opus.
tracing::warn!(
value = %raw,
setting,
"PUNKTFUNK_AUDIO_HIRES is not a format this client can ask for — use 1, \
96000, 0, or <48000|96000>/<16|24>; ignoring it and using the audio-format \
setting instead"
);
audio_format_wire(setting)
}
}
}
/// `Hello`'s "I did not ask" for the audio format pair, which keeps the `Hello` byte-identical to
/// every pre-hi-res build's.
///
/// ⚠ **Not an explicit 48 000/16.** Core keys `CLIENT_CAP_AUDIO_HIRES` on *a format was specified*
/// rather than on *it differs from the default* — because 48 kHz/16-bit is BOTH the default and
/// the cheapest lossless rung, so a "differs from the default" rule would make it the one format
/// on the ladder nobody could ask for. `0`/`0` is what separates "not asking" from "asking for
/// 48/16 lossless", and passing an explicit 48 000/16 here would advertise hi-res on every
/// ordinary session (`design/hi-res-audio.md` §7, and `client::advertised_client_caps`).
const AUDIO_FORMAT_UNSPECIFIED: (u32, u8) = (0, 0);
/// What `PUNKTFUNK_AUDIO_HIRES` was set to, as the three answers that matter.
#[derive(Debug, PartialEq, Eq)]
enum AudioRequest {
/// Unset or deliberately off — today's Opus plane, and no capability bit.
Legacy,
/// A rung the lossless plane can carry.
Hires(u32, u8),
/// Set to something this client cannot ask for at all.
Unsupported,
}
/// The parse half of [`requested_audio_format`], split out so it is testable without touching the
/// process environment.
fn parse_audio_format(raw: &str) -> AudioRequest {
use punktfunk_core::audio::pcm::{BITS_16, BITS_24};
let v = raw.trim().to_ascii_lowercase();
match v.as_str() {
"" | "0" | "off" | "false" | "no" => return AudioRequest::Legacy,
// 24-bit is the rung the plane earns its bandwidth at: 16-bit PCM would spend 1.5 Mbps to
// sound like transparent 256 kbps Opus, so it is not what a bare "on" should mean.
"1" | "on" | "true" | "yes" => return AudioRequest::Hires(96_000, BITS_24),
_ => {}
}
// `<rate>` or `<rate>/<bits>`. Both halves are checked against what the plane can actually
// carry rather than passed through: 44.1 kHz and its multiples are absent from the ladder
// ON PURPOSE (they truncate `JitterPolicy`'s integer samples-per-millisecond arithmetic, §4.1),
// and the host would decline them anyway — refusing here says so where the user can see it.
//
// 48 000/16 IS accepted — it is the cheapest lossless rung (1.5 Mbps against Opus's 256), and
// core keys the capability bit on *a format was specified* rather than on *it differs from the
// default* precisely so that this rung stays askable. Which is why the caller must send the
// unspecified `0`/`0` when nobody asked, rather than an explicit 48 000/16.
let (rate_s, bits_s) = v.split_once('/').unwrap_or((v.as_str(), "24"));
match rate_s
.trim()
.parse::<u32>()
.ok()
.zip(bits_s.trim().parse::<u8>().ok())
.filter(|&(r, b)| matches!(r, 48_000 | 96_000) && matches!(b, BITS_16 | BITS_24))
{
Some((r, b)) => AudioRequest::Hires(r, b),
None => AudioRequest::Unsupported,
}
}
fn pump(
params: SessionParams,
ev_tx: async_channel::Sender<SessionEvent>,
@@ -550,7 +849,39 @@ fn pump(
"retrying with reduced decode caps"
);
}
let connector = match NativeClient::connect(
// The lossless audio plane's client-side opt-in, filtered by what this box can genuinely
// PLAY. `CLIENT_CAP_AUDIO_HIRES` means *capable **and** the user turned it on* — a client
// that advertised it without being able to render it would spend 1.5–4.6 Mbps, taken off the
// top of a link ABR can neither see nor reclaim, to play interpolation (`hi-res-audio.md` §7).
// `Some` past this block is exactly "ask", and asking IS what sets the bit — core derives it
// from the format pair being specified at all (see `AUDIO_FORMAT_UNSPECIFIED`).
let hires = requested_audio_format(&params.audio_format).filter(|&(rate, _)| {
if params.audio_channels != 2 {
// §4.2: a hi-res surround frame does not fit one datagram at the default MTU and this
// plane is never fragmented, so the host's gate declines it. Saying so here, where the
// user's two settings are both visible, beats a decline logged on the other machine.
//
// ⚠ Not redundant with the settings UIs hiding the picker under surround. The two
// fields are INDEPENDENT overlay keys: a profile can pin `audio_format` while the
// global — or another profile — moves `audio_channels` to 5.1, and the env override
// below answers to no UI at all. This is the one place both are known.
tracing::warn!(
channels = params.audio_channels,
"lossless audio is stereo-only — a surround frame does not fit one QUIC \
datagram; asking for the default Opus plane instead"
);
return false;
}
// `can_render_at` says WHY when it declines — it is the one that read the device.
audio::can_render_at(rate)
});
if let Some((rate, bits)) = hires {
tracing::info!(rate, bits, "asking the host for the lossless audio plane");
}
// This pair IS the request: core derives `CLIENT_CAP_AUDIO_HIRES` from it being specified at
// all, so `None` must reach the wire as unspecified rather than as an explicit 48 000/16.
let (audio_rate_hz, audio_bits) = hires.unwrap_or(AUDIO_FORMAT_UNSPECIFIED);
let connector = match NativeClient::connect_with_audio_format(
&params.host,
params.port,
params.mode,
@@ -559,6 +890,8 @@ fn pump(
params.bitrate_kbps,
params.video_caps,
params.audio_channels,
audio_rate_hz,
audio_bits,
// The codecs OUR rungs speak (`video::decodable_codecs`), plus CODEC_PYROWAVE when
// the presenter device passed the probe, minus whatever a previous attempt proved
// undecodable end to end.
@@ -577,6 +910,10 @@ fn pump(
punktfunk_core::quic::CLIENT_CAP_PHASE_LOCK
} else {
0
// AUDIO_HIRES is NOT set here: core derives it from the `audio_rate_hz`/`audio_bits`
// pair above being specified at all, which is the one rule that keeps 48 kHz/16-bit
// lossless askable. Setting it here as well would be a second copy of that rule, and
// setting it WITHOUT a format would advertise a request the host can only decline.
// PAD_AUDIO: the embedder can render per-pad DualSense haptics/speaker (see above).
}) | (if pad_audio_on {
punktfunk_core::quic::CLIENT_CAP_PAD_AUDIO
@@ -1569,6 +1906,11 @@ fn pump(
mic_dropped,
audio_buffer_ms: connector.audio_buffer_ms(),
audio_av_offset_ms: connector.audio_av_offset_ms() as i32,
// Read off the connector, not off `params`: these three are the Welcome's answer,
// and the request lives one struct away precisely so they cannot be confused.
audio_lossless: connector.audio_codec == punktfunk_core::quic::AUDIO_CODEC_PCM,
audio_rate_hz: connector.audio_sample_rate_hz,
audio_bits: connector.audio_bits,
decoder: dec_path,
target_kbps: connector.current_bitrate_kbps(),
auto_rate,
@@ -1668,24 +2010,81 @@ fn codec_fallback_event(
}
}
/// The dedicated audio thread: owns the Opus decoder, the PCM scratch, and the PipeWire
/// The dedicated audio thread: owns the decoder, the sample scratch, and the PipeWire
/// player, and blocks on `next_audio` (the plane's single consumer — packets land every
/// 5 ms). Decoded chunks are pushed in Vecs recycled from the player's pool, so the
/// frame). Decoded chunks are pushed in Vecs recycled from the player's pool, so the
/// steady state allocates nothing. Best-effort like before: any setup failure logs and
/// the session streams video-only. Exits on the stop flag or a closed plane.
fn spawn_audio(
connector: Arc<NativeClient>,
stop: Arc<AtomicBool>,
) -> Option<std::thread::JoinHandle<()>> {
// Decoder + playback are built from the host-RESOLVED channel count (never the
// request), so an older/clamping host that resolves stereo is decoded as stereo.
// Decoder + playback are built from the host-RESOLVED format (never the request), so an
// older/clamping host that resolves stereo Opus at 48 kHz is decoded and played exactly that
// way — and a host that granted less than was asked for is honoured rather than argued with.
// Opening the device from the REQUEST instead is the failure `hi-res-audio.md` §4.3 is written
// around, one end further along.
let channels = connector.audio_channels;
let player = audio::AudioPlayer::spawn(channels as u32)
.map_err(|e| tracing::warn!(error = %e, "audio disabled"))
.ok()?;
let mut dec = AudioDec::new(channels)
.map_err(|e| tracing::warn!(error = %e, "opus decoder failed — audio disabled"))
.ok()?;
// A codec this client does not speak is refused OUT LOUD, not guessed at. `Welcome::decode`
// takes `audio_codec` VERBATIM — it deliberately does not fold an unknown id onto Opus,
// because that is the one field that selects the plane — so this decision lands here, and it
// has exactly two wrong answers: Opus-decoding a `0xD3` payload is noise, and waiting for
// `0xC9` frames that a `0xD3` session never sends is silence with no explanation. (`1` is
// reserved for FLAC and emitted by nothing today; anything else is a future or corrupt wire.)
if !matches!(
connector.audio_codec,
punktfunk_core::quic::AUDIO_CODEC_OPUS | punktfunk_core::quic::AUDIO_CODEC_PCM
) {
tracing::warn!(
codec = connector.audio_codec,
"the host resolved an audio plane this client cannot decode — streaming video-only"
);
return None;
}
let lossless = connector.audio_codec == punktfunk_core::quic::AUDIO_CODEC_PCM;
// A zero rate is inexpressible off the wire — `Welcome::decode` folds both absence and a
// literal `0` to the legacy rate — but everything below divides by it (the ring's ms
// reporting, the graph quantum, the jitter policy), and libopus refuses it outright. Core's
// own C surface makes exactly this argument in `AudioFormat::of`: this is the one value that
// must not depend on a peer's honesty.
let rate_hz = match connector.audio_sample_rate_hz {
0 => punktfunk_core::audio::SAMPLE_RATE_HZ,
hz => hz,
};
// One protocol frame. The Opus plane's is the fixed 5 ms every build has spoken; the lossless
// plane NEGOTIATES it from the path MTU (§4.2) — 4 ms at 48/24, 2 ms at 96/24 under the default
// ceiling — so it must be read, never assumed.
let frame_us = if lossless {
// Floored at the ladder's shortest rung. A host that resolved the plane always states a
// duration; `0` could only come from one that did not, and sizing a graph quantum, a poll
// timeout and a scratch buffer from zero is a worse answer than the shortest real frame.
(connector.audio_frame_us as u32).max(1_000)
} else {
punktfunk_core::audio::FRAME_MS * 1000
};
tracing::info!(
codec = if lossless { "pcm" } else { "opus" },
channels,
rate_hz,
bits = connector.audio_bits,
frame_us,
"negotiated audio format"
);
let player = audio::AudioPlayer::spawn(audio::PlaybackFormat {
channels: channels as u32,
rate_hz,
frame_us,
})
.map_err(|e| tracing::warn!(error = %e, "audio disabled"))
.ok()?;
let mut dec = AudioDec::new(
connector.audio_codec,
channels,
rate_hz,
connector.audio_bits,
)
.map_err(|e| tracing::warn!(error = %e, "opus decoder failed — audio disabled"))
.ok()?;
// A/V sync (audio latency overhaul). This thread is the only place that holds all three
// ingredients at once: the packet's host capture `pts_ns`, the ring depth (via the sync cell)
// and the video plane's end-to-end figure. `pts_ns` was decoded into `AudioPacket` and then
@@ -1703,15 +2102,40 @@ fn spawn_audio(
let video_e2e = connector.video_e2e_shared();
let av_offset_out = connector.audio_av_offset_shared();
let buffer_ms_out = connector.audio_buffer_ms_shared();
// Interleaved samples per ms, to report the ring depth in the unit a human reads.
let per_ms = 48 * channels.max(1) as usize;
// Interleaved samples per ms, to report the ring depth in the unit a human reads. Denominated
// in the RESOLVED rate: 48 × channels at the protocol default, 96 × channels on a 96 kHz
// lossless session — where the old constant would have halved every `buffer_ms` this thread
// publishes, silently, in the direction that looks healthy.
let per_ms = (rate_hz / 1000).max(1) as usize * channels.max(1) as usize;
// Decode scratch, sized for whichever plane this session actually runs — the two have very
// different worst cases, and sizing for the wrong one is either waste or an overrun:
//
// * **Opus (`0xC9`)**: a packet may carry up to 120 ms, which is what the old `5760 × channels`
// was — 120 ms at 48 kHz. Derived from the rate rather than restated as a literal so the
// figure cannot quietly become 60 ms if this plane ever runs anywhere but 48 kHz. This arm
// is a HARD BOUND: libopus decodes into a fixed slice.
// * **PCM (`0xD3`)**: exactly one negotiated frame, 1–5 ms — two orders of magnitude smaller,
// and the only size a `0xD3` datagram can carry (one frame per datagram, never fragmented).
// Here it is only a capacity hint: `pcm::to_f32` grows the Vec itself, so an oversized
// datagram reallocates rather than overruns.
let scratch = if lossless {
punktfunk_core::audio::pcm::samples_per_frame(rate_hz, frame_us, channels)
} else {
120 * per_ms
};
// The pull loop's tick, one protocol frame. 5 ms on the Opus plane; as short as 1 ms on a
// lossless one, where a fixed 5 ms wait would make the drought decision on the wrong schedule
// and let the ring drain two frames between looks. Rounded UP so a sub-millisecond rung can
// never round to a zero-length timeout and spin.
let frame_ms = (frame_us as u64).div_ceil(1000).max(1);
std::thread::Builder::new()
.name("punktfunk-audio-rx".into())
.spawn(move || {
let mut pcm = vec![0f32; 5760 * channels as usize]; // scratch: max Opus frame (120 ms) × channels
let mut pcm = vec![0f32; scratch];
let mut gaps = punktfunk_core::audio::AudioGapTracker::new();
let mut frame_samples = 0usize; // per-channel samples of the last decoded frame — the PLC unit
let mut av = punktfunk_core::audio::AvSync::new(channels);
// Interleaved samples in the last decoded frame — the unit concealment is produced in.
let mut frame_samples = 0usize;
let mut av = punktfunk_core::audio::AvSync::new_at_rate(channels, rate_hz);
if !av_sync_enabled {
tracing::info!("A/V sync disabled by PUNKTFUNK_NO_AV_SYNC");
}
@@ -1719,8 +2143,14 @@ fn spawn_audio(
// but only when a later packet arrives to reveal it; when the wire simply goes quiet
// nothing arrives to reveal anything, and the ring drains into an underrun and a
// de-prime whose re-prime is a longer artifact than the audio that was missing.
let mut drought =
punktfunk_core::audio::DroughtConceal::new(audio::TUNING.plc_max_ms());
// Told the plane's real frame, so its wall-clock fuse and its `plc_ms` are spent at
// the rate this session actually paces. It used to assume 5 ms, which on a 2 ms
// lossless frame blew the fuse after two fifths of the time the tuning intends and
// over-reported concealment by the same factor.
let mut drought = punktfunk_core::audio::DroughtConceal::new_at_frame_us(
audio::TUNING.plc_max_ms(),
frame_us,
);
let mut last_packet = std::time::Instant::now();
while !stop.load(Ordering::SeqCst) {
// Wait at most one frame WHILE there is a stream to protect: the drought decision
@@ -1728,11 +2158,7 @@ fn spawn_audio(
// turn up. Before anything has decoded there is no state to conceal from and
// nothing to conceal for, so a session whose host never sends audio keeps the old
// long timeout rather than waking two hundred times a second to do nothing.
let wait_ms = if frame_samples > 0 {
punktfunk_core::audio::FRAME_MS as u64
} else {
100
};
let wait_ms = if frame_samples > 0 { frame_ms } else { 100 };
match connector.next_audio(Duration::from_millis(wait_ms)) {
Ok(pkt) => {
// Place this frame against the picture it belongs with, BEFORE it is
@@ -1761,49 +2187,54 @@ fn spawn_audio(
// concealing it a second time here would insert samples it never carried
// and push everything after them later.
let already = drought.packet();
// Conceal lost packets (a seq gap) with libopus PLC before decoding the one
// that arrived: empty input synthesizes `frame_samples` of interpolation per
// missing packet — an inaudible fade instead of the click a hard gap makes.
// Conceal lost packets (a seq gap) before decoding the one that arrived.
// Which concealment that is, is the plane's business: libopus PLC
// interpolates from its own decoder state on `0xC9`, and `PcmConceal`
// repeats-and-fades on `0xD3`, because a lossless format has nothing to
// interpolate FROM (§4.5). The gap ARITHMETIC is codec-independent —
// both planes carry one frame per datagram under the same header, which
// is exactly why `AudioGapTracker` needed no second implementation.
for _ in 0..gaps.missing_before(pkt.seq).saturating_sub(already) {
let plc = frame_samples * channels as usize;
if plc == 0 {
break; // no decoded frame yet to size the concealment from
if frame_samples == 0 {
break; // no decoded frame yet to conceal from
}
if let Ok(samples) = dec.decode_float(&[], &mut pcm[..plc], false) {
let mut buf = player.take_buffer();
buf.extend_from_slice(&pcm[..samples * channels as usize]);
player.push(buf);
}
}
match dec.decode_float(&pkt.data, &mut pcm, false) {
// `samples` is per-channel; the interleaved frame is `samples * channels`.
Ok(samples) => {
frame_samples = samples;
let n = samples * channels as usize;
if let Some(n) = dec.conceal(frame_samples, &mut pcm) {
let mut buf = player.take_buffer();
buf.extend_from_slice(&pcm[..n]);
player.push(buf);
}
Err(e) => tracing::debug!(error = %e, "opus decode failed"),
}
match dec.decode(&pkt.data, &mut pcm) {
// Interleaved, on both planes — see `AudioDec::decode`.
Some(n) => {
frame_samples = n;
let mut buf = player.take_buffer();
buf.extend_from_slice(&pcm[..n]);
player.push(buf);
}
// Opus: a corrupt packet. PCM: a datagram that is not a whole number
// of samples at the negotiated depth, which core refuses rather than
// decode as a shifted frame. Either way the frame is lost, and the
// next arrival's seq gap conceals it.
None => tracing::debug!(bytes = pkt.data.len(), "audio decode failed"),
}
}
Err(PunktfunkError::NoFrame) => {
// Nothing on the wire. If the ring is draining with it, conceal from the
// decoder's own state — the same libopus interpolation the loss path uses,
// bounded by this backend's de-prime fuse so a genuinely dead stream is
// not papered over. `frame_samples` is 0 until something has decoded:
// there is no state to extrapolate from before then.
// Nothing on the wire. If the ring is draining with it, conceal with the
// same machinery the loss path uses, bounded by this backend's de-prime
// fuse so a genuinely dead stream is not papered over. `frame_samples` is
// 0 until something has decoded: there is no state to extrapolate from
// before then.
//
// ONE frame per tick, not a burst: this arm fires every `FRAME_MS`, which
// ONE frame per tick, not a burst: this arm fires every frame time, which
// is exactly the rate the callback drains at, so concealment keeps pace
// with playout instead of racing ahead of a depth reading it has already
// invalidated.
let depth_ms = (sync_cell.depth() / per_ms) as u32;
if frame_samples > 0 && drought.conceal(last_packet.elapsed(), depth_ms) {
let plc = frame_samples * channels as usize;
if let Ok(samples) = dec.decode_float(&[], &mut pcm[..plc], false) {
if let Some(n) = dec.conceal(frame_samples, &mut pcm) {
let mut buf = player.take_buffer();
buf.extend_from_slice(&pcm[..samples * channels as usize]);
buf.extend_from_slice(&pcm[..n]);
player.push(buf);
}
sync_cell.publish_plc_ms(drought.total_ms());
@@ -1836,6 +2267,223 @@ fn parse_debug_reconfigure(s: &str) -> Option<(Mode, Duration)> {
mod tests {
use super::*;
/// The opt-in's whole job is to be the difference between `CLIENT_CAP_AUDIO_HIRES` set and
/// unset, so every spelling the doc comment promises has to land on the right side of it.
#[test]
fn the_hires_opt_in_parses_the_spellings_it_documents() {
use punktfunk_core::audio::pcm::{BITS_16, BITS_24};
// Off, in every shape a user might write it.
for off in ["", "0", "off", "false", "no", " Off "] {
assert_eq!(parse_audio_format(off), AudioRequest::Legacy, "{off:?}");
}
// On → the flagship rung. 24-bit, because 16-bit PCM spends 1.5 Mbps to sound like
// transparent Opus.
for on in ["1", "on", "true", "YES"] {
assert_eq!(
parse_audio_format(on),
AudioRequest::Hires(96_000, BITS_24),
"{on:?}"
);
}
assert_eq!(
parse_audio_format("96000"),
AudioRequest::Hires(96_000, BITS_24)
);
assert_eq!(
parse_audio_format("48000/24"),
AudioRequest::Hires(48_000, BITS_24)
);
assert_eq!(
parse_audio_format("96000/16"),
AudioRequest::Hires(96_000, BITS_16)
);
// 48 kHz/16-bit is a REQUEST, not an "off" — the cheapest lossless rung, and the one a
// "differs from the default" rule would have made unaskable. The connect turns this into
// an explicit `48000`/`16` on the wire and `Legacy` into the unspecified `0`/`0`, which is
// the only thing that tells the two apart.
assert_eq!(
parse_audio_format("48000/16"),
AudioRequest::Hires(48_000, BITS_16)
);
assert_ne!(parse_audio_format("48000/16"), AudioRequest::Legacy);
// …and "not asking" must never reach the wire as a format, or every ordinary session
// would advertise hi-res.
assert_eq!(AUDIO_FORMAT_UNSPECIFIED, (0, 0));
}
/// Every rung the plane cannot carry must be REFUSED here, where the user can be told, rather
/// than sent to a host that will decline it silently from the other machine. 44.1 kHz is the
/// one that looks reasonable: it is absent because it truncates `JitterPolicy`'s integer
/// samples-per-millisecond arithmetic (§4.1), not because the wire could not carry it.
#[test]
fn the_hires_opt_in_refuses_what_the_plane_cannot_carry() {
for bad in [
"44100",
"44100/24",
"88200/24",
"192000/24",
"48000/32",
"96000/8",
"96000/",
"/24",
"96 kHz",
"yes please",
"-96000/24",
] {
assert_eq!(
parse_audio_format(bad),
AudioRequest::Unsupported,
"{bad:?} should not parse"
);
}
}
/// The Settings choice's stored values and the pair each asks for. The SPELLINGS are the
/// point: they are shared verbatim with the Apple `AudioFormatChoice` raw values and the
/// Android `AUDIO_FORMAT_*`, so one profile catalog round-trips through all four clients. A
/// typo here fails silently — the key survives a load→save (it lands in `SettingsOverlay`'s
/// `extra`), so the profile keeps working on the other clients and only this one ignores it.
#[test]
fn the_audio_format_setting_speaks_the_cross_client_spellings() {
use punktfunk_core::audio::pcm::BITS_24;
assert_eq!(AUDIO_FORMAT_OPUS, "opus");
assert_eq!(AUDIO_FORMAT_LOSSLESS_48, "lossless48");
assert_eq!(AUDIO_FORMAT_LOSSLESS_96, "lossless96");
// The menu order every client shows, defaulting to the Opus row.
assert_eq!(
AUDIO_FORMATS.iter().map(|(v, _)| *v).collect::<Vec<_>>(),
[
AUDIO_FORMAT_OPUS,
AUDIO_FORMAT_LOSSLESS_48,
AUDIO_FORMAT_LOSSLESS_96
]
);
assert_eq!(audio_format_wire(AUDIO_FORMAT_OPUS), None);
// Both lossless rows are 24-bit: 16-bit PCM would spend 1.5 Mbps to sound like the
// transparent 256 kbps Opus it replaces, which is why no row offers it.
assert_eq!(
audio_format_wire(AUDIO_FORMAT_LOSSLESS_48),
Some((48_000, BITS_24))
);
assert_eq!(
audio_format_wire(AUDIO_FORMAT_LOSSLESS_96),
Some((96_000, BITS_24))
);
// A newer client's row, and a corrupted store: Opus, never a refused connect.
assert_eq!(audio_format_wire("lossless192"), None);
assert_eq!(audio_format_wire(""), None);
}
/// Precedence: `PUNKTFUNK_AUDIO_HIRES` overrides the setting in BOTH directions, an unset var
/// leaves the setting alone, and a typo is ignored rather than being read as "off".
///
/// The last one is the case that changed when the setting arrived. Before it, garbage meant
/// off, which was the honest answer when the var was the only switch there was; now it would
/// silently defeat a choice the user made in the UI, so the var stands down instead.
#[test]
fn the_env_override_beats_the_setting_in_both_directions() {
use punktfunk_core::audio::pcm::{BITS_16, BITS_24};
// Unset: the setting decides, which is the ordinary path on every desktop.
assert_eq!(resolve_audio_format(None, AUDIO_FORMAT_OPUS), None);
assert_eq!(
resolve_audio_format(None, AUDIO_FORMAT_LOSSLESS_96),
Some((96_000, BITS_24))
);
// Set: it wins, including OVER a lossless setting and including turning it off — a lever
// that could only ever switch the feature on would be half a lever.
assert_eq!(
resolve_audio_format(Some("1"), AUDIO_FORMAT_OPUS),
Some((96_000, BITS_24))
);
assert_eq!(
resolve_audio_format(Some("48000/16"), AUDIO_FORMAT_LOSSLESS_96),
Some((48_000, BITS_16)),
"the env rung the menu does not offer is still reachable"
);
for off in ["0", "off", "false"] {
assert_eq!(
resolve_audio_format(Some(off), AUDIO_FORMAT_LOSSLESS_96),
None,
"{off:?} must force Opus over a lossless setting"
);
}
// A typo is not an instruction: warned about and ignored, so the setting still decides.
assert_eq!(
resolve_audio_format(Some("96 kHz"), AUDIO_FORMAT_LOSSLESS_48),
Some((48_000, BITS_24))
);
assert_eq!(resolve_audio_format(Some("44100"), AUDIO_FORMAT_OPUS), None);
}
/// The lossless arm's contract: interleaved counts (not per-channel), concealment that says
/// no before it has anything to repeat, and a truncated datagram refused outright.
///
/// A per-channel/interleaved mix-up here would halve every push into the ring — audible as a
/// permanently starving ring rather than as an obvious failure, which is why it is pinned.
#[test]
fn the_lossless_plane_decodes_and_conceals_in_interleaved_samples() {
use punktfunk_core::audio::pcm;
let mut dec = AudioDec::new(
punktfunk_core::quic::AUDIO_CODEC_PCM,
2,
96_000,
pcm::BITS_24,
)
.expect("the PCM arm builds no codec and cannot fail");
let mut out = Vec::new();
// Nothing has arrived yet: saying so is what makes the caller emit silence and let the
// ring re-prime, instead of playing an uninitialised buffer.
assert_eq!(dec.conceal(384, &mut out), None);
// One 2 ms frame at 96 kHz/24-bit stereo — the rung the default MTU ceiling lands on.
let frame = pcm::samples_per_frame(96_000, 2_000, 2);
assert_eq!(frame, 384, "192 samples per channel, interleaved");
let mut wire = Vec::new();
pcm::from_f32(&vec![0.5f32; frame], pcm::BITS_24, &mut wire);
assert_eq!(
dec.decode(&wire, &mut out),
Some(frame),
"interleaved count"
);
assert!(out[..frame].iter().all(|s| (s - 0.5).abs() < 1e-3));
// …and now there IS something to conceal from — at the frame's own length, whatever hint
// is passed, because `PcmConceal` holds the frame it repeats.
assert_eq!(dec.conceal(0, &mut out), Some(frame));
// A datagram that is not a whole number of samples at the negotiated depth is refused
// rather than decoded as a shifted frame, which would desync every sample after it.
assert_eq!(dec.decode(&wire[..wire.len() - 1], &mut out), None);
}
/// The Opus arm through the same two methods, because they now return INTERLEAVED counts
/// where libopus itself counts per channel — the one place this refactor could have halved a
/// working plane.
#[test]
fn the_opus_plane_reports_interleaved_samples_too() {
let mut enc = opus::Encoder::new(48_000, opus::Channels::Stereo, opus::Application::Audio)
.expect("opus encoder");
let mut packet = [0u8; 4_000];
let silence = [0.0f32; 240 * 2];
let n = enc
.encode_float(&silence, &mut packet)
.expect("encode one 5 ms stereo frame");
let mut dec = AudioDec::new(punktfunk_core::quic::AUDIO_CODEC_OPUS, 2, 48_000, 16)
.expect("opus decoder");
// The pump's scratch: 120 ms — the biggest frame the Opus plane can carry.
let mut out = vec![0f32; 120 * 48 * 2];
assert_eq!(dec.decode(&packet[..n], &mut out), Some(240 * 2));
// PLC is asked for, and answered, in the same unit.
assert_eq!(dec.conceal(240 * 2, &mut out), Some(240 * 2));
// Nothing to size PLC from is a `None`, not a panic on an empty slice.
let mut empty = Vec::new();
assert_eq!(dec.conceal(0, &mut empty), None);
}
#[test]
fn debug_reconfigure_parses_the_documented_shape() {
let (mode, delay) = parse_debug_reconfigure("1280x720@60:5").unwrap();
crates/pf-client-core/src/trust.rs
+34
View File
@@ -1181,6 +1181,32 @@ pub struct Settings {
/// Requested audio channel count: 2 (stereo), 6 (5.1) or 8 (7.1). The host clamps to what it
/// can capture; the resolved count drives the decoder + playback layout.
pub audio_channels: u8,
/// Requested audio format — the cross-client `audio_format` key, whose stored values are shared
/// verbatim with the Apple and Android clients (`crate::session::AUDIO_FORMATS`):
/// [`crate::session::AUDIO_FORMAT_OPUS`] (the default, and byte for byte the session every
/// build before the lossless plane ran), `..._LOSSLESS_48` or `..._LOSSLESS_96`.
///
/// Off by default and deliberately: lossless takes 2.3–4.6 Mbps off the top of the link,
/// OUTSIDE the ABR loop that manages the video budget, against the ~256 kbps Opus it replaces —
/// so it has to be asked for at BOTH ends (`PUNKTFUNK_AUDIO_HIRES` is the host's half, also off
/// by default). A REQUEST, never a fact: the host runs a five-condition gate and may answer
/// Opus anyway, and this client downgrades it further if the output device will not open the
/// rate. What actually happened is the OSD's `audio lossless …` line, and the log's
/// "negotiated audio format".
///
/// Stereo-only: a lossless surround frame does not fit one QUIC datagram at the default MTU
/// and the host declines it (`design/hi-res-audio.md` §4.2). Both desktop settings UIs take
/// the picker away under 5.1/7.1 — GTK greys the row (its per-row profile Reset lives on the
/// row, and an insensitive row is the idiom its mic-dependent rows already use), the WinUI
/// shell drops it from the rendered card (its idiom for a row that does not apply). The
/// session filters the pair AGAIN whatever either UI did, because the two fields are
/// independent profile overrides and can disagree — and the env override answers to no UI.
///
/// A `String`, not an enum, for the same reason [`codec`](Self::codec) is: it is read out of a
/// file a newer client may have written, and an unrecognized value resolves to Opus rather than
/// ending a session over a dropdown. `default` so pre-existing stores load on the Opus plane.
#[serde(default = "default_audio_format")]
pub audio_format: String,
/// Preferred video codec: `"auto"` (host decides), `"hevc"`, `"h264"`, or `"av1"`. A soft
/// preference — the host honors it when it can emit it, else falls back to the best shared codec.
#[serde(default = "default_codec")]
@@ -1333,6 +1359,13 @@ fn default_codec() -> String {
"auto".into()
}
/// The Opus plane — every session before the lossless one existed, and the one a store written by
/// an older client must load as. Named from `session` so the default and the menu's first row can
/// never be two different strings.
fn default_audio_format() -> String {
crate::session::AUDIO_FORMAT_OPUS.into()
}
fn default_auto() -> String {
"auto".into()
}
@@ -1452,6 +1485,7 @@ impl Default for Settings {
mic_enabled: false,
echo_cancel: true,
audio_channels: 2,
audio_format: default_audio_format(),
codec: "auto".into(),
decoder: "auto".into(),
adapter: String::new(),
crates/pf-console-ui/src/screens/settings.rs
+159 -9
View File
@@ -18,6 +18,7 @@ use crate::screens::{Ctx, Outbox, Screen};
use crate::theme::{fg, Fonts, W};
use crate::widgets::{ListMsg, MenuList, RowSpec, TabStrip, TAB_STRIP_H};
use pf_client_core::gamepad::{MenuEvent, MenuPulse};
use pf_client_core::session::{AUDIO_FORMATS, AUDIO_FORMAT_OPUS};
use pf_client_core::trust::{MouseMode, StatsVerbosity, TouchMode};
use skia_safe::{Canvas, Rect};
@@ -44,6 +45,10 @@ enum RowId {
Vsync,
AllowVrr,
Audio,
/// The lossless-PCM opt-in — the cross-client `audio_format` key. Off (Opus) by default, and
/// tied to the channel count above it for a reason that is NOT the one the design doc gives;
/// see the `enabled` note in [`row_spec`].
AudioFormat,
Mic,
EchoCancel,
PadForward,
@@ -99,7 +104,15 @@ const TABS: [(&str, &[RowId]); 7] = [
RowId::AllowVrr,
],
),
("Audio", &[RowId::Audio, RowId::Mic, RowId::EchoCancel]),
(
"Audio",
&[
RowId::Audio,
RowId::AudioFormat,
RowId::Mic,
RowId::EchoCancel,
],
),
(
"Controller",
&[
@@ -541,14 +554,32 @@ fn row_spec(id: RowId, ctx: &Ctx, profiles: &[(String, String)]) -> RowSpec {
_ => {}
}
let s = &ctx.settings;
// Two rows follow a switch a line or two above them: echo cancellation only means
// anything while the mic streams, and the pad rows only while any controller is
// forwarded at all. Both go dim and inert otherwise — the same relationship the desktop
// shells draw by greying a row out, and dimming (not dropping) is what shows the
// relationship. The smoothness buffer used to be listed here too; it is dropped from the
// list instead now — see [`row_applies`] for why that one is different.
// Three rows follow a switch a line or two above them: echo cancellation only means
// anything while the mic streams, the audio format only while the stream is stereo, and
// the pad rows only while any controller is forwarded at all. All go dim and inert
// otherwise — the same relationship the desktop shells draw by greying a row out, and
// dimming (not dropping) is what shows the relationship. The smoothness buffer used to be
// listed here too; it is dropped from the list instead now — see [`row_applies`] for why
// that one is different.
let enabled = match id {
RowId::EchoCancel => s.mic_enabled,
// ⚠ Lossless follows the channel count for a reason that has MOVED, and the old reason
// is still written down in several places that are now wrong (`hi-res-audio.md` §4.2's
// blanket "surround does not fit a datagram", and `trust::Settings::audio_format`'s doc
// quoting it). The HOST's `channels != 2` decline is deleted: `pcm::frame_us_for` is
// channel-aware, so 48 kHz/24-bit 5.1 and 7.1 simply negotiate a SHORTER frame and fit
// an ordinary datagram — only 96 kHz surround has nowhere left on the ladder. The Apple
// and Android console screens dropped their gates on the strength of that, and this row
// would too if it could.
//
// It cannot, yet: `pf_client_core::session` still filters `audio_channels != 2` out of
// the request BEFORE it reaches the wire, so a lossless choice made here under 5.1/7.1
// is never even asked for — the session logs "lossless audio is stereo-only" and runs
// Opus. A live row would be a control that changes nothing, which is exactly the lie
// this screen dims rows to avoid, and it would also disagree with the GTK dialog
// reading the same settings file on this same machine. Delete this arm when that
// client-side filter learns the frame ladder — not before.
RowId::AudioFormat => s.audio_channels == 2,
RowId::Pad | RowId::PadType | RowId::SystemButtons | RowId::GuideGesture => {
s.gamepad_forwarding
}
@@ -639,6 +670,11 @@ fn row_spec(id: RowId, ctx: &Ctx, profiles: &[(String, String)]) -> RowSpec {
.map_or("Stereo", |(_, l)| l)
.into(),
),
RowId::AudioFormat => (
None,
"Audio quality",
audio_format_label(&s.audio_format).into(),
),
RowId::Mic => (None, "Microphone", on_off(s.mic_enabled).into()),
RowId::EchoCancel => (None, "Echo cancellation", on_off(s.echo_cancel).into()),
RowId::PadForward => (
@@ -755,6 +791,11 @@ fn detail(id: RowId) -> &'static str {
a fixed cadence. Applies to fullscreen sessions; harmless on a fixed screen."
}
RowId::Audio => "The speaker layout requested from the host.",
RowId::AudioFormat => {
"Bit-exact PCM instead of Opus — 2.3 Mb/s at 48 kHz, 4.6 at 96, off the top of the \
link. The host has its own switch and stays on Opus if it can't deliver the rate; \
the stats overlay names what the session got. Stereo only."
}
RowId::Mic => {
"Send this device's microphone to the host's virtual mic. \
Ctrl+Alt+Shift+V mutes and unmutes it while streaming."
@@ -836,6 +877,21 @@ fn label_for<'a>(options: &'a [(&str, &'a str)], value: &str) -> &'a str {
.map_or("—", |(_, l)| l)
}
/// The label for a stored `audio_format` value — [`label_for`] with a different miss, on purpose.
///
/// An unrecognized value is not a corrupt one here: the key travels verbatim through a profile
/// catalog shared with the Apple and Android clients, so a rung a NEWER client offers can land in
/// this file. The session resolves anything it doesn't know to Opus
/// (`pf_client_core::session::audio_format_wire`), so the row says Opus too. `label_for`'s "—"
/// would name a format no session on this box will ever run.
fn audio_format_label(value: &str) -> &'static str {
AUDIO_FORMATS
.iter()
.find(|(v, _)| *v == value)
.or_else(|| AUDIO_FORMATS.iter().find(|(v, _)| *v == AUDIO_FORMAT_OPUS))
.map_or("", |(_, l)| *l)
}
/// Step (`wrap=false`, clamped — false = boundary) or cycle (`wrap=true`) a row's
/// value. Toggles read left = off, right = on; a no-op is a boundary.
fn adjust(id: RowId, delta: i32, wrap: bool, ctx: &mut Ctx) -> bool {
@@ -912,6 +968,16 @@ fn adjust(id: RowId, delta: i32, wrap: bool, ctx: &mut Ctx) -> bool {
let cur = AUDIO.iter().position(|(v, _)| *v == s.audio_channels);
step_option(cur, AUDIO.len(), delta, wrap).map(|i| s.audio_channels = AUDIO[i].0)
}
// Inert under surround — a boundary thud, matching what the dimmed row shows. The gate is
// this client's own request filter rather than anything about the plane; the `enabled`
// note in `row_spec` is where that is written down, and where it gets deleted.
RowId::AudioFormat => {
if s.audio_channels == 2 {
step_str(AUDIO_FORMATS, &mut s.audio_format, delta, wrap)
} else {
None
}
}
RowId::Mic => toggle(&mut s.mic_enabled, delta, wrap),
// Inert while the mic is off — a boundary thud, matching what the dimmed row shows.
RowId::EchoCancel => {
@@ -1603,9 +1669,10 @@ mod tests {
seen.push(*id);
}
}
// The pre-tab flat list, plus the palette row this change added.
assert_eq!(seen.len(), 30, "{seen:?}");
// The pre-tab flat list, plus the palette row and the lossless-audio row later passes added.
assert_eq!(seen.len(), 31, "{seen:?}");
assert!(seen.contains(&RowId::Palette));
assert!(seen.contains(&RowId::AudioFormat));
// The catalog rows belong to the trailing tab, which builds them at render time.
assert!(TABS[PROFILES_TAB].1.is_empty());
assert_eq!(TABS[PROFILES_TAB].0, "Profiles");
@@ -1647,6 +1714,89 @@ mod tests {
assert!(fx.nav.is_none() && fx.cmds.is_empty());
}
/// The lossless opt-in: it ships OFF, steps the cross-client table verbatim, sits directly
/// under the channel count, and follows it — dim and inert under 5.1/7.1, because this
/// client's session refuses to ASK for lossless surround (see the `enabled` note in
/// [`row_spec`]; the host's own decline is gone). Every value is asserted against
/// `pf_client_core::session`'s constants rather than restated, so a spelling change there
/// reds this test instead of quietly making the console write a key nobody reads.
#[test]
fn audio_format_ships_off_and_follows_the_channel_count() {
use pf_client_core::session::{AUDIO_FORMAT_LOSSLESS_48, AUDIO_FORMAT_LOSSLESS_96};
let (mut settings, pads) = ctx_parts();
assert_eq!(settings.audio_format, AUDIO_FORMAT_OPUS, "off by default");
assert_eq!(settings.audio_channels, 2, "…and the gate starts open");
let library = crate::library::LibraryShared::default();
let mut ctx = Ctx {
hosts: &[],
library: &library,
settings: &mut settings,
pads: &pads,
deck: false,
device_name: "t",
t: 0.0,
};
let mut s = SettingsScreen::with_profiles(Vec::new());
s.tab = TABS
.iter()
.position(|(name, _)| *name == "Audio")
.expect("the Audio tab");
let audio = s.row_ids(&ctx);
let channels = audio
.iter()
.position(|id| *id == RowId::Audio)
.expect("the channels row");
assert_eq!(
audio.get(channels + 1),
Some(&RowId::AudioFormat),
"the row sits directly under the one that dims it, like every other pair here"
);
// Steps the shared table in order, clamping at both ends…
assert!(
!adjust(RowId::AudioFormat, -1, false, &mut ctx),
"already Opus = thud"
);
assert!(adjust(RowId::AudioFormat, 1, false, &mut ctx));
assert_eq!(ctx.settings.audio_format, AUDIO_FORMAT_LOSSLESS_48);
assert!(adjust(RowId::AudioFormat, 1, false, &mut ctx));
assert_eq!(ctx.settings.audio_format, AUDIO_FORMAT_LOSSLESS_96);
assert!(
!adjust(RowId::AudioFormat, 1, false, &mut ctx),
"last = thud"
);
// …and A wraps home, so every rung is reachable one-handed.
assert!(adjust(RowId::AudioFormat, 1, true, &mut ctx));
assert_eq!(ctx.settings.audio_format, AUDIO_FORMAT_OPUS);
// Surround dims it AND refuses the write. A row that accepted a change the session
// throws away is the same lie as an enabled-looking control.
ctx.settings.audio_format = AUDIO_FORMAT_LOSSLESS_48.into();
ctx.settings.audio_channels = 6;
assert!(!row_spec(RowId::AudioFormat, &ctx, &[]).enabled);
assert!(
!adjust(RowId::AudioFormat, 1, false, &mut ctx),
"surround = thud"
);
assert!(!adjust(RowId::AudioFormat, 1, true, &mut ctx), "A too");
assert_eq!(
ctx.settings.audio_format, AUDIO_FORMAT_LOSSLESS_48,
"and nothing was written — the stored preference survives the gate"
);
// Dimmed, never dropped: it stays visible beside the row that dimmed it.
assert!(s.row_ids(&ctx).contains(&RowId::AudioFormat));
ctx.settings.audio_channels = 2;
assert!(row_spec(RowId::AudioFormat, &ctx, &[]).enabled);
// A rung this build has no row for — a newer client's, arriving through a shared profile
// catalog — reads as the Opus the session will actually run, not as a blank "—".
ctx.settings.audio_format = AUDIO_FORMAT_OPUS.into();
let opus = row_spec(RowId::AudioFormat, &ctx, &[]).value;
assert!(opus.is_some());
ctx.settings.audio_format = "lossless192".into();
assert_eq!(row_spec(RowId::AudioFormat, &ctx, &[]).value, opus);
}
/// The palette row steps the shared `ui_palette` key through the table and wraps on A,
/// like every other choice row.
#[test]
crates/pf-host-config/src/lib.rs
+33
View File
@@ -230,6 +230,36 @@ pub struct HostConfig {
/// (the default) = automatic: sent only to a client that asked for it, and only while the link
/// is actually losing packets.
pub audio_redundancy: Option<bool>,
/// `PUNKTFUNK_AUDIO_HIRES` — host policy gate for the lossless `0xD3` audio plane
/// (44.1/48/88.2/96/176.4 kHz, 16/24-bit PCM, stereo through 7.1;
/// `design/hi-res-audio.md` §10). The rate set lives in
/// [`punktfunk_core::audio::pcm::rate_is_supported`] and the channel count is decided by
/// whether a frame fits a datagram, not by a list — so neither is restated here.
///
/// **Default OFF, and deliberately unlike every other `Option<bool>` knob here** — the use
/// site is `unwrap_or(false)`, not `unwrap_or(true)`. `audio_redundancy` above defaults ON
/// because it costs a few hundred kbps and buys loss resilience on a plane the user already
/// agreed to; hi-res costs **1.4–8.5 Mbps in stereo, up to 33.9 in 7.1** and rides QUIC
/// datagrams OUTSIDE the ABR loop,
/// so it is taken off the top of the link and adaptive bitrate can neither see nor reclaim
/// it (§4.6). That is bandwidth nobody consented to, on a link the host cannot re-negotiate
/// afterwards — so it must be asked for at BOTH ends: the client sets
/// `CLIENT_CAP_AUDIO_HIRES` (its own user-facing toggle, also default off) and the operator
/// sets this.
///
/// `None` (unset) and an explicit off are therefore the same answer at the use site; the
/// tri-state is kept only so a future status/diagnostics reader can tell "the operator turned
/// it off" from "the operator never said". Explicit-off grammar for symmetry with its
/// neighbours.
///
/// ⚠ **The desktop CLIENTS read a variable of this same name with a RICHER grammar** — see
/// `pf_client_core::session`, which takes `1`/`on`, a bare rate such as `96000`, or an explicit
/// `<rate>/<bits>`. A box that is both host and client therefore configures both halves from
/// one environment line, and they compose only because [`env_on`] reads everything that is not
/// `0`/`false`/`off`/`no` as *on*: a client-shaped `96000/24` happens to say *allow* here too.
/// That is an accident that works, not a shared grammar — `1` is the only spelling that means
/// the same thing at both ends, which is why it is the one the docs give.
pub audio_hires: Option<bool>,
/// `PUNKTFUNK_PERF` — per-stage timing instrumentation.
pub perf: bool,
/// `PUNKTFUNK_VIDEO_SOURCE` — GameStream video source select. `virtual` (the default — a
@@ -435,6 +465,9 @@ impl HostConfig {
audio_output_mode: AudioOutputMode::from_env(),
audio_quality: val("PUNKTFUNK_AUDIO_QUALITY").map(|s| s.trim().to_lowercase()),
audio_redundancy: env_on("PUNKTFUNK_AUDIO_REDUNDANCY"),
// Tri-state like its neighbour, but read as `unwrap_or(FALSE)` at the use site —
// see the field doc for why this one knob inverts the house default.
audio_hires: env_on("PUNKTFUNK_AUDIO_HIRES"),
perf: flag("PUNKTFUNK_PERF"),
// Default ON while the interval-stutter field program runs (see the field doc).
stall_probes: env_on("PUNKTFUNK_STALL_PROBES").unwrap_or(true),
crates/pf-presenter/src/run.rs
+63
View File
@@ -3184,6 +3184,26 @@ fn stats_text(
text.push_str(&format!(" · a/v {:+} ms", s.audio_av_offset_ms));
}
}
// The RESOLVED audio format — `audio lossless 96 kHz / 24-bit`, the same wording the Apple and
// Android HUDs use. NOT gated to Detailed, unlike the latency line above it, and deliberately:
// it is the one thing a user who turned lossless on needs to see. The Settings screen shows
// what this device REQUESTED, and the host's five-condition gate can decline every one of them
// (its own switch is off by default) — leaving a session that costs 2.3–4.6 Mbps off the top
// of the link and delivers nothing, indistinguishable from success without this line
// (design/hi-res-audio.md §4.3, §10).
//
// Silent on Opus rather than printing `audio opus 48 kHz`: that is what every session has
// always been, so a line stating it would be noise on the HUD of every user who never touched
// the setting. A zero rate/depth is an old host that reported neither — no reading, so nothing
// to print, rather than a fabricated 48 kHz.
if s.audio_lossless && s.audio_rate_hz > 0 && s.audio_bits > 0 {
let rate = if s.audio_rate_hz % 1000 == 0 {
format!("{} kHz", s.audio_rate_hz / 1000)
} else {
format!("{} Hz", s.audio_rate_hz)
};
text.push_str(&format!("\naudio lossless {rate} / {}-bit", s.audio_bits));
}
// Decode integrity (M4) — the native lane's answer to "was that stream actually
// clean?". Appended LAST and only when it has something to say, which keeps it
// additive for the stdout `stats:` line's parsers (a machine interface: every
@@ -3562,6 +3582,11 @@ mod tests {
mic_dropped: 0,
audio_buffer_ms: 0,
audio_av_offset_ms: 0,
// The Opus plane every ordinary session runs, so the tier texts below stay
// exactly what they were before the lossless line existed.
audio_lossless: false,
audio_rate_hz: 0,
audio_bits: 0,
// The decode-path tag as the session actually spells it since M10 — the
// ladder's rung names (`NativeRung::name`), not the deleted libavcodec
// ones. A fixture carrying a tag no client emits would let this test go on
@@ -3982,6 +4007,44 @@ mod tests {
assert!(text(&s, StatsVerbosity::Detailed).contains("mic 100 f/s · dropped 7"));
}
/// The RESOLVED audio format line: silent on Opus, silent when the host reported no format at
/// all, and — unlike every other audio figure here — visible from Normal up.
///
/// That tier choice is the test's real subject. The line exists because a declined lossless
/// session is indistinguishable from a granted one: it costs the bandwidth either way, and the
/// Settings screen can only show what was ASKED for. Hiding the one honest answer behind the
/// Detailed tier would leave the common case (a user who turned it on and wants to know)
/// looking at a HUD that says nothing.
#[test]
fn stats_text_audio_format_line() {
let (mut s, p) = sample();
let text = |s: &Stats, v| stats_text(v, "m", s, &p, false, false, false, None);
assert!(
!text(&s, StatsVerbosity::Detailed).contains("audio lossless"),
"the Opus plane every ordinary session runs says nothing"
);
s.audio_lossless = true;
s.audio_rate_hz = 96_000;
s.audio_bits = 24;
assert!(text(&s, StatsVerbosity::Normal).contains("\naudio lossless 96 kHz / 24-bit"));
assert!(text(&s, StatsVerbosity::Detailed).contains("\naudio lossless 96 kHz / 24-bit"));
// Compact is one line by definition, and Off renders nothing at all.
assert!(!text(&s, StatsVerbosity::Compact).contains("audio"));
assert!(text(&s, StatsVerbosity::Off).is_empty());
s.audio_rate_hz = 48_000;
assert!(text(&s, StatsVerbosity::Normal).contains("\naudio lossless 48 kHz / 24-bit"));
// An old host reported no format: no reading, so nothing is printed — a fabricated
// "48 kHz" would be the same class of claim the line exists to prevent.
s.audio_rate_hz = 0;
assert!(!text(&s, StatsVerbosity::Detailed).contains("audio lossless"));
s.audio_rate_hz = 96_000;
s.audio_bits = 0;
assert!(!text(&s, StatsVerbosity::Detailed).contains("audio lossless"));
}
/// Compact omits the latency term until the presenter's first e2e window lands.
#[test]
fn compact_waits_for_e2e() {
crates/punktfunk-core/cbindgen.toml
+27
View File
@@ -70,8 +70,35 @@ include = ["PunktfunkEndReason"]
# Same hazard as the BTN_* block above, one step worse: `FRAME_MS` and `SAMPLE_RATE_HZ` are
# generic enough that an embedder is likely to have its own, and a clashing #define silently
# takes the last definition rather than failing to compile.
#
# `PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ` keeps its value AND its meaning now that sessions can resolve
# other rates: it is the DEFAULT/legacy rate — 48 000, what every Opus session runs at, which is
# every session an embedder that does not call `punktfunk_connect_ex11` can ask for. Removing or
# re-pointing it would be a C-ABI break for every ring already sized from it. A HI-RES session
# must instead read `punktfunk_connection_audio_sample_rate()` /
# `punktfunk_connection_audio_bits()` — accessors added in ABI 24 precisely because the structs
# that would otherwise carry the rate are `#[repr(C)]` and allocated by value.
"FRAME_MS" = "PUNKTFUNK_AUDIO_FRAME_MS"
"SAMPLE_RATE_HZ" = "PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ"
# The lossless plane's constants, under the same rule. `BITS_16`/`BITS_24`/`PCM_HEADER_LEN` are
# the worst offenders the header has ever been asked to export — a bare `#define BITS_24` in
# every C embedder's namespace is exactly the silent-last-definition-wins hazard the note above
# describes, and audio code is likelier than most to have its own.
"PCM_HEADER_LEN" = "PUNKTFUNK_AUDIO_PCM_HEADER_LEN"
"FRAME_US_LADDER" = "PUNKTFUNK_AUDIO_FRAME_US_LADDER"
"BITS_16" = "PUNKTFUNK_AUDIO_BITS_16"
"BITS_24" = "PUNKTFUNK_AUDIO_BITS_24"
"AUDIO_PCM_MAGIC" = "PUNKTFUNK_AUDIO_PCM_MAGIC"
"AUDIO_PCM_HEADER" = "PUNKTFUNK_AUDIO_PCM_HEADER"
"AUDIO_CODEC_OPUS" = "PUNKTFUNK_AUDIO_CODEC_OPUS"
"AUDIO_CODEC_FLAC_RESERVED" = "PUNKTFUNK_AUDIO_CODEC_FLAC_RESERVED"
"AUDIO_CODEC_PCM" = "PUNKTFUNK_AUDIO_CODEC_PCM"
# These two land on the SAME names as their `abi.rs` mirrors, which is the existing pattern
# (`HOST_CAP_PEN`, `CLIENT_CAP_CURSOR`): cbindgen emits both, and an identical `#define` twice is
# a benign redefinition in C. One name for one bit is the point — an embedder that reaches for
# either spelling gets the same constant.
"CLIENT_CAP_AUDIO_HIRES" = "PUNKTFUNK_CLIENT_CAP_AUDIO_HIRES"
"HOST_CAP_AUDIO_HIRES" = "PUNKTFUNK_HOST_CAP_AUDIO_HIRES"
# R21: every remaining exported constant, prefixed. cbindgen emits a bare `#define` per
# `pub const`, so without an entry here names as generic as MAX_PADS, TAG_LEN, ABI_VERSION and
crates/punktfunk-core/src/abi.rs
+929 -61
View File
File diff suppressed because it is too large Load Diff
crates/punktfunk-core/src/audio.rs
+682 -85
View File
File diff suppressed because it is too large Load Diff
crates/punktfunk-core/src/audio/pcm.rs
+779
View File
@@ -0,0 +1,779 @@
//! Lossless PCM for the `0xD3` audio plane.
//!
//! The `0xC9` plane carries Opus, which is transparent but lossy and — by construction — 48 kHz
//! only (RFC 6716; `opus_encoder_create` rejects 96 000). This module is the second plane's
//! payload: interleaved little-endian integer samples, no codec, no container.
//!
//! **Why no codec.** The obvious alternative was FLAC, and it was measured against this on the
//! four axes that decide it (`design/hi-res-audio.md` §5):
//!
//! - A datagram that exceeds the path MTU is not sent *at all*, and this plane is never
//! fragmented — so [`frame_us_for`] must size frames from the **worst case**. FLAC's worst
//! case is a VERBATIM subframe: raw samples plus a frame header. So FLAC and PCM get the same
//! negotiated frame duration, the same packet rate, and the same send-buffer sizing. The codec
//! buys *average* bytes on the wire and nothing structural.
//! - The plane rides outside the ABR loop, so it is provisioned for peak, not average — which is
//! the number a codec's typical-case saving does not move.
//! - The host scales f32 capture to 24-bit with no dither, so the low bits of a float game mix
//! are close to incompressible. At 24 bits — the depth that is the entire point of the feature
//! — a lossless coder saves the least.
//! - PCM adds no dependency to the NDK / xcframework / flatpak / MSIX / Arch packaging targets,
//! and no spike gate.
//!
//! Both formats deliver the identical product claim: bit-exact playback with no lossy stage.
//!
//! **On "lossless".** Neither depth is bit-exact against the f32 engine mix it came from — the
//! host quantises once, deliberately and without dither ([`from_f32`]). What this plane
//! guarantees is that nothing is lost *after* that quantisation: [`from_f32`] → [`to_f32`] →
//! [`from_f32`] is the identity, proven by test, so the samples the client renders are the
//! samples the host captured.
/// The `0xD3` datagram's fixed header: tag + `u32` seq + `u64` pts_ns, the same shape as `0xC9`
/// so the gap tracker and the A/V-sync plumbing work unchanged.
/// `quic::datagram` asserts this against its own encoder.
pub const PCM_HEADER_LEN: usize = 1 + 4 + 8;
/// Frame durations the plane may negotiate, longest first.
///
/// Every rung divides the **48 kHz family** into a whole number of samples per channel, so on
/// those rates the host pacer and the client ring carry an exact frame:
///
/// | µs | samples/ch @48 kHz | samples/ch @96 kHz |
/// |---|---|---|
/// | 5000 | 240 | 480 |
/// | 4000 | 192 | 384 |
/// | 3000 | 144 | 288 |
/// | 2500 | 120 | 240 |
/// | 2000 | 96 | 192 |
/// | 1500 | 72 | 144 |
/// | 1000 | 48 | 96 |
///
/// ⚠⚠ **The 44.1 kHz family does not divide, and this doc used to claim every rate did.** A rung
/// lands on a whole sample only when `rate_hz × µs` is a multiple of 1 000 000, which needs a
/// multiple of **10 000 µs** at 44 100 Hz, **5 000 µs** at 88 200 and **2 500 µs** at 176 400. So
/// of the seven rungs, 44 100 has **none**, 88 200 has only 5 000, and 176 400 has 5 000 and
/// 2 500. Every other pairing carries [`samples_per_frame`]'s FLOOR and is therefore *shorter*
/// than the rung it is labelled with: 5 ms at 44 100 Hz is 220 samples per channel — 4 988 662 ns,
/// 0.23 % short.
///
/// That is safe for the two things this ladder decides, and unsafe for a third:
///
/// - **Payload sizing** — a floored frame is *fewer* bytes, so [`frame_us_for`]'s fit against the
/// datagram holds with margin rather than being eroded (the payload must never exceed the
/// datagram; that invariant is absolute and this rounds the right way for it).
/// - **Buffer sizing** — both ends size from [`samples_per_frame`], so they agree by construction.
/// - **⚠ Timing — no.** A rung is a *nominal* length for the wire and the ring, never a duration.
/// Anything advancing a `pts_ns` must use [`frame_duration_ns`] of the frame's real sample
/// count; adding 5 000 µs to a frame that carries 4 988 662 ns runs the clock 0.23 % fast
/// forever, and the A/V sync loop will fight that drift and never win.
pub const FRAME_US_LADDER: [u32; 7] = [5000, 4000, 3000, 2500, 2000, 1500, 1000];
/// Bit depths the plane carries. 32-bit float is deliberately absent: no source produces detail
/// 24 bits does not capture, and it would cost 33 % more for nothing.
pub const BITS_16: u8 = 16;
/// See [`BITS_16`].
pub const BITS_24: u8 = 24;
/// Full-scale magnitude at a given depth. Deliberately **symmetric** — the most-negative code
/// (`-2^(n-1)`) is not used, so [`from_f32`]/[`to_f32`] round-trip exactly in both directions
/// rather than folding one code onto its neighbour. One code out of 16.7 million is not audible;
/// a round trip that is not the identity would make the bit-exactness gate untestable.
const fn full_scale(bits: u8) -> i32 {
match bits {
BITS_16 => 32_767,
_ => 8_388_607,
}
}
/// Bytes each sample occupies on the wire at `bits`.
pub const fn bytes_per_sample(bits: u8) -> usize {
match bits {
BITS_16 => 2,
_ => 3,
}
}
/// Whether `bits` is a depth this plane can carry.
pub const fn depth_is_supported(bits: u8) -> bool {
matches!(bits, BITS_16 | BITS_24)
}
/// Whether `rate_hz` is a sample rate this plane can carry — the single expression of the set, so
/// the host's negotiation gate and the client's request validation cannot drift apart.
///
/// Two families, and both are now exact in every conversion core performs:
///
/// - **48 kHz** — 48 000 / 96 000. What an engine mix runs at, and the only family Opus speaks.
/// - **44.1 kHz** — 44 100 / 88 200 / 176 400. CD-derived material, and what an ordinary Windows
/// endpoint or a 44.1 kHz interface reports as its own engine rate. These were deferred, not
/// refused: [`JitterPolicy`](crate::audio::JitterPolicy) divided by 1 000 before it multiplied,
/// so 44 100 Hz became 44 samples/ms and every depth, target and reported `buffer_ms` came out
/// 2.3 % low (`design/hi-res-audio.md` §4.1). The order of those two operations was the whole
/// blocker; it is fixed, and this is that deferral being lifted.
///
/// ⚠ A supported rate is **not** a promise that any of it is free: the 44.1 family carries a
/// fractional number of samples in most [`FRAME_US_LADDER`] rungs (see there), and 176 400/24-bit
/// stereo costs 8.5 Mbps off the top of a link that ABR can neither see nor reclaim. The host's
/// own gate still decides whether a session can afford it.
///
/// 192 kHz remains absent by the §3 scope decision rather than by any arithmetic — nothing here
/// would object to it. (§3 also words the scope as "≤ 96 kHz", which 176 400 exceeds while §4.1's
/// deferral list names it as a rate blocked *only* by this arithmetic. The two lines disagree;
/// this follows §4.1, which is the one that gives a reason.)
pub const fn rate_is_supported(rate_hz: u32) -> bool {
matches!(rate_hz, 44_100 | 48_000 | 88_200 | 96_000 | 176_400)
}
/// Interleaved samples in one `frame_us` frame — **per channel × channels**.
///
/// **The single source of truth for how long a frame is.** The host fills a buffer of this size
/// and the client's ring drains one, so the two agree *by construction* rather than by both
/// re-deriving `rate × µs` and hoping they round the same way. Keep that property: a second
/// derivation is a second rounding, and a one-sample disagreement on an interleaved stream walks
/// the channels around each other.
///
/// ⚠ **A frame carries a whole number of samples PER CHANNEL, so `frame_us` is a label, not a
/// duration.** The divide is per channel and floors, because 220.5 samples do not exist: at
/// 44 100 Hz a nominal 5 ms frame is 220 samples per channel — [`frame_duration_ns`] of it is
/// 4 988 662 ns, not 5 000 000. Size from this; **time from [`frame_duration_ns`]**, never from
/// `frame_us`.
pub const fn samples_per_frame(rate_hz: u32, frame_us: u32, channels: u8) -> usize {
// Multiply first, divide last (`rate_hz / 1_000_000` is 0 for every rate below a megahertz),
// and in u64 rather than usize: `usize` is 32 bits on some embedder targets, where 176 400 Hz
// against a frame longer than the ladder's own rungs would wrap. Saturating rather than
// wrapping, because a wrapped count is a SMALL one — an under-sized buffer, which is the
// failure that corrupts rather than the one that merely wastes.
let total = (rate_hz as u64 * frame_us as u64 / 1_000_000) * channels as u64;
if total > u32::MAX as u64 {
u32::MAX as usize
} else {
total as usize
}
}
/// How long `samples` interleaved samples really last, in nanoseconds — the inverse of
/// [`samples_per_frame`], and the only figure a `pts_ns` may be advanced by.
///
/// **Why this exists.** A frame's sample count and its nominal `frame_us` stopped being
/// interchangeable the moment the 44.1 kHz family was admitted. A host that ships 220 samples per
/// channel and advances its clock by the 5 000 µs it negotiated is running **0.23 % fast** — 2.3
/// ms of invented time per second — and every downstream measurement agrees with it, because the
/// timestamps are self-consistent and simply wrong. The A/V sync loop then chases a drift that is
/// manufactured at the source and can never be caught. That is the "measures correctly while being
/// wrong" failure this plane's whole design is written against.
///
/// **Feed it a running total, not one frame.** `samples` is exact only when it divides; 220
/// samples at 44 100 Hz is 4 988 662.13… ns and this floors it. Adding a floored per-frame value
/// accumulates < 1 ns per frame (≈ 0.2 µs/s — irrelevant), but the drift-free formulation is to
/// keep the session's cumulative sample count and stamp
/// `pts_ns = base + frame_duration_ns(samples_so_far, …)`, which never accumulates anything at
/// all. Both beat advancing by `frame_us`, which is wrong by four orders of magnitude more.
///
/// `channels` is the interleaved count the samples are counted in, so this is the exact partner of
/// [`samples_per_frame`]: `frame_duration_ns(samples_per_frame(r, us, ch), r, ch) <= us × 1000`,
/// with equality exactly when the rung divides the rate.
pub const fn frame_duration_ns(samples: usize, rate_hz: u32, channels: u8) -> u64 {
// Interleaved samples per second — the denominator. u128 so the numerator below cannot
// overflow for any `usize` a caller can hand us; this is not a hot path (one call per frame,
// on the thread that stamps it) and correctness at the edge is worth more than the cycles.
let per_sec = rate_hz as u128 * channels as u128;
if per_sec == 0 {
return 0; // a degenerate layout has no duration rather than a division fault
}
let ns = samples as u128 * 1_000_000_000 / per_sec;
if ns > u64::MAX as u128 {
u64::MAX
} else {
ns as u64
}
}
/// Wire bytes one `frame_us` frame occupies, excluding [`PCM_HEADER_LEN`].
pub const fn frame_payload_bytes(rate_hz: u32, bits: u8, channels: u8, frame_us: u32) -> usize {
samples_per_frame(rate_hz, frame_us, channels) * bytes_per_sample(bits)
}
/// What the plane costs, in kbps — payload only, the number [`crate::audio::plan_audio_budget`]
/// must be told about rather than left to infer.
///
/// Floors, which on the 44.1 kHz family loses a fraction of a kbps out of 1 411 (44 100/16-bit is
/// 1 411.2). That is deliberately not worth a rational type: the figure exists to be compared
/// against a link allowance measured in megabits, and rounding it *down* keeps a borderline
/// session from being declined over 0.2 kbps it would in fact have had.
pub const fn bitrate_kbps(rate_hz: u32, bits: u8, channels: u8) -> u32 {
(rate_hz as u64 * bits as u64 * channels as u64 / 1000) as u32
}
/// The longest [`FRAME_US_LADDER`] rung whose frame fits one datagram of `max_datagram` bytes,
/// or `None` if even the shortest does not.
///
/// **Sized from the raw frame, never from a coded estimate.** A datagram larger than the path
/// MTU is not sent at all and this plane is never fragmented, so the only safe input to this
/// decision is the size the payload is *guaranteed* not to exceed. For PCM that is exactly the
/// raw size; for any lossless codec added later it is the raw size plus a small header (a FLAC
/// VERBATIM frame), so this bound holds for both and the two would negotiate the same duration.
///
/// The caller must not ask before QUIC MTU discovery has settled, or it will size against the
/// conservative initial value and spend the rest of the session on shorter frames than the path
/// can carry (`design/hi-res-audio.md` §4.2).
///
/// **Still exact on a rate the ladder does not divide.** The fit is measured through
/// [`samples_per_frame`], which floors, so a 44.1-family frame is *at most* the size the rung's
/// nominal duration implies and usually one sample per channel less. The rounding therefore runs
/// toward the datagram fitting, never away from it — which is the only direction that matters
/// here, because an oversized datagram is not sent at all.
pub fn frame_us_for(rate_hz: u32, bits: u8, channels: u8, max_datagram: usize) -> Option<u32> {
let budget = max_datagram.checked_sub(PCM_HEADER_LEN)?;
FRAME_US_LADDER
.iter()
.copied()
.find(|&us| frame_payload_bytes(rate_hz, bits, channels, us) <= budget)
}
/// Quantise one interleaved f32 frame onto the wire, appending to `out`.
///
/// Scale-and-clamp with **no dither**: the source is a game mix that was already quantised
/// upstream, and dithering it would add noise while destroying the bit-exactness this plane
/// exists to provide.
pub fn from_f32(samples: &[f32], bits: u8, out: &mut Vec<u8>) {
let fs = full_scale(bits);
let scale = fs as f32;
out.reserve(samples.len() * bytes_per_sample(bits));
if bits == BITS_16 {
for &s in samples {
let v = (s * scale).round().clamp(-scale, scale) as i32;
out.extend_from_slice(&(v as i16).to_le_bytes());
}
} else {
for &s in samples {
let v = (s * scale).round().clamp(-scale, scale) as i32;
let b = v.to_le_bytes();
out.extend_from_slice(&b[..3]);
}
}
}
/// Reverse of [`from_f32`]: decode `bytes` into `out`, returning the interleaved sample count.
/// `None` if `bytes` is not a whole number of samples at `bits`.
pub fn to_f32(bytes: &[u8], bits: u8, out: &mut Vec<f32>) -> Option<usize> {
let step = bytes_per_sample(bits);
if bytes.len() % step != 0 {
return None;
}
let inv = 1.0 / full_scale(bits) as f32;
out.clear();
out.reserve(bytes.len() / step);
if bits == BITS_16 {
for c in bytes.chunks_exact(2) {
out.push(i16::from_le_bytes([c[0], c[1]]) as f32 * inv);
}
} else {
for c in bytes.chunks_exact(3) {
// Sign-extend 24 bits into an i32 by placing the sample in the TOP three bytes and
// arithmetic-shifting back down.
let v = i32::from_le_bytes([0, c[0], c[1], c[2]]) >> 8;
out.push(v as f32 * inv);
}
}
Some(out.len())
}
/// Packet-loss concealment for a plane that has none.
///
/// [`crate::audio::AudioGapTracker`] feeds libopus PLC on the `0xC9` plane, so a lost datagram
/// interpolates instead of clicking. **A lossless format cannot do that** — there is nothing in
/// a raw frame from which to synthesise its successor. This is the replacement, and it is the
/// least-proven part of the plane (`design/hi-res-audio.md` §4.5): its tuning wants a
/// loss-injection listen, not just the unit tests below.
///
/// - **One frame lost** → repeat the previous frame with a raised-cosine fade. A frame is short
/// enough that repetition reads as continuity rather than as the pitch artefact a longer
/// repeat would produce.
/// - **Two or more** → fade to silence across the gap and back in on recovery. A clean dropout
/// beats a warble.
#[derive(Debug, Default, Clone)]
pub struct PcmConceal {
/// The last good frame, interleaved f32 — the material every concealed frame is built from.
prev: Vec<f32>,
/// Consecutive frames concealed since the last real one.
run: u32,
}
impl PcmConceal {
pub fn new() -> PcmConceal {
PcmConceal::default()
}
/// Remember a frame that really arrived, and end any concealment run.
pub fn accept(&mut self, frame: &[f32]) {
self.prev.clear();
self.prev.extend_from_slice(frame);
self.run = 0;
}
/// Frames concealed since the last real one — for stats, and for the caller's own cap.
pub fn run(&self) -> u32 {
self.run
}
/// Produce one concealed frame into `out`, or `false` when there is nothing to build from
/// (no frame has arrived yet) — in which case the caller should emit silence and let the
/// ring re-prime.
pub fn conceal(&mut self, out: &mut Vec<f32>) -> bool {
if self.prev.is_empty() {
return false;
}
self.run = self.run.saturating_add(1);
out.clear();
out.extend_from_slice(&self.prev);
let n = out.len();
match self.run {
// First loss: hand back the previous frame, faded out across its tail so a repeated
// waveform does not step at the splice.
1 => raised_cosine_tail(out, n),
// Sustained loss: decay toward silence rather than looping a fragment.
r => {
let g = 0.5f32.powi(r.min(8) as i32 - 1);
for s in out.iter_mut() {
*s *= g;
}
raised_cosine_tail(out, n);
}
}
// The faded frame becomes the source for the next one, so a run decays monotonically
// instead of restarting from the last loud frame every time.
self.prev.clear();
self.prev.extend_from_slice(out);
true
}
}
/// Apply a raised-cosine fade to the final `n` samples in place, so a spliced or repeated frame
/// meets what follows it without a step.
fn raised_cosine_tail(buf: &mut [f32], n: usize) {
let n = n.min(buf.len());
if n == 0 {
return;
}
let start = buf.len() - n;
for (i, s) in buf[start..].iter_mut().enumerate() {
let t = (i as f32 + 0.5) / n as f32;
*s *= 0.5 * (1.0 + (std::f32::consts::PI * t).cos());
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Energy at one frequency, by the Goertzel algorithm — a single-bin DFT, which is all this
/// needs and costs no dependency. Returns the magnitude relative to a full-scale sine, so 1.0
/// is "the whole signal is this tone" and 0.0 is "nothing here".
fn tone_energy(samples: &[f32], rate_hz: u32, freq_hz: f32) -> f32 {
let n = samples.len();
let k = (n as f32 * freq_hz / rate_hz as f32).round();
let w = 2.0 * std::f32::consts::PI * k / n as f32;
let coeff = 2.0 * w.cos();
let (mut s1, mut s2) = (0.0f32, 0.0f32);
for &x in samples {
let s0 = x + coeff * s1 - s2;
s2 = s1;
s1 = s0;
}
let power = s1 * s1 + s2 * s2 - coeff * s1 * s2;
2.0 * power.max(0.0).sqrt() / n as f32
}
/// **The claim hi-res makes, tested rather than assumed.** A tone above 24 kHz cannot exist on
/// the Opus plane — Opus is 48 kHz by construction, so anything above Nyquist is gone before
/// the encoder sees it, and that is the entire reason this second plane exists. So the plane
/// has to be shown to carry one.
///
/// This is the SOFTWARE half of `design/hi-res-audio.md` §13.2. The full check is "play an
/// ultrasonic tone on the host and confirm it arrives", and its other half — that the host's
/// CAPTURE did not silently resample on the way in — cannot be tested here, because that is
/// WASAPI autoconvert and PipeWire's resampler, which need a host and an interface. What this
/// proves is the part that is ours: once a 30 kHz tone is in the pipeline, the `0xD3` payload
/// carries it out intact.
///
/// A brick wall at 24 kHz in the on-glass spectrum therefore indicts the capture path, not the
/// transport — this test is what makes that inference sound.
#[test]
fn a_tone_above_the_opus_ceiling_survives_the_plane() {
// 30 kHz: comfortably above the 24 kHz Nyquist limit of the Opus plane, and inside what
// a 96 kHz session can represent (Nyquist 48 kHz).
const TONE_HZ: f32 = 30_000.0;
for rate in [96_000u32, 176_400] {
let n = rate as usize / 10; // 100 ms, plenty of bins at 30 kHz
let src: Vec<f32> = (0..n)
.map(|i| {
(2.0 * std::f32::consts::PI * TONE_HZ * i as f32 / rate as f32).sin() * 0.5
})
.collect();
let before = tone_energy(&src, rate, TONE_HZ);
assert!(
before > 0.45,
"{rate} Hz: the source tone is not there ({before})"
);
// The detector has to DISCRIMINATE, or every assertion below is vacuous: a bin that
// reads high everywhere would "prove" the tone survived a pipeline that deleted it.
// 12 kHz is silent in this signal and must read as such.
let absent = tone_energy(&src, rate, 12_000.0);
assert!(
absent < 0.01,
"{rate} Hz: the tone detector reads {absent} where there is no tone, so it \
cannot tell survival from loss"
);
let mut wire = Vec::new();
from_f32(&src, BITS_24, &mut wire);
let mut out = Vec::new();
to_f32(&wire, BITS_24, &mut out).expect("whole samples");
let after = tone_energy(&out, rate, TONE_HZ);
assert!(
(after - before).abs() < 0.001,
"{rate} Hz: 30 kHz tone lost {:.4} of its energy crossing the plane \
(before {before:.4}, after {after:.4})",
before - after
);
// And it is not merely *present* — 24-bit quantisation is far below anything audible,
// so the reconstruction must be sample-accurate, not just spectrally similar.
let worst = src
.iter()
.zip(&out)
.map(|(a, b)| (a - b).abs())
.fold(0.0f32, f32::max);
assert!(
worst < 1.0 / full_scale(BITS_24) as f32,
"{rate} Hz: worst sample error {worst} exceeds one 24-bit code"
);
}
}
/// The claim the whole plane exists to make. Every representable code at both depths must
/// survive wire → f32 → wire unchanged; anything less and "lossless" is marketing.
#[test]
fn every_code_round_trips_bit_exactly() {
for bits in [BITS_16, BITS_24] {
let fs = full_scale(bits);
// The endpoints, zero, and a deterministic sweep across the range.
let mut codes: Vec<i32> = vec![0, 1, -1, fs, -fs, fs - 1, -fs + 1];
let stride = (fs / 4096).max(1);
codes.extend((-fs..=fs).step_by(stride as usize));
let mut wire = Vec::new();
for &c in &codes {
if bits == BITS_16 {
wire.extend_from_slice(&(c as i16).to_le_bytes());
} else {
wire.extend_from_slice(&c.to_le_bytes()[..3]);
}
}
let mut floats = Vec::new();
let n = to_f32(&wire, bits, &mut floats).expect("whole samples");
assert_eq!(n, codes.len());
let mut back = Vec::new();
from_f32(&floats, bits, &mut back);
assert_eq!(
back, wire,
"{bits}-bit PCM must survive wire → f32 → wire unchanged"
);
}
}
/// Sign extension is the one place a 24-bit unpack goes quietly wrong — a missing shift
/// turns every negative sample into a large positive one, which sounds like loud noise
/// rather than like a bug.
#[test]
fn twenty_four_bit_negatives_sign_extend() {
let mut wire = Vec::new();
from_f32(&[-0.5, 0.5, -1.0, 1.0], BITS_24, &mut wire);
let mut out = Vec::new();
to_f32(&wire, BITS_24, &mut out).expect("whole samples");
assert!(out[0] < -0.4 && out[0] > -0.6, "got {}", out[0]);
assert!(out[1] > 0.4 && out[1] < 0.6, "got {}", out[1]);
assert!((out[2] + 1.0).abs() < 1e-6, "got {}", out[2]);
assert!((out[3] - 1.0).abs() < 1e-6, "got {}", out[3]);
}
/// Out-of-range input must clamp, not wrap. A wrapped sample is full-scale noise of the
/// opposite sign — the loudest possible artefact from the quietest possible mistake.
#[test]
fn out_of_range_input_clamps_instead_of_wrapping() {
for bits in [BITS_16, BITS_24] {
let mut wire = Vec::new();
from_f32(
&[9.0, -9.0, f32::INFINITY, f32::NEG_INFINITY],
bits,
&mut wire,
);
let mut out = Vec::new();
to_f32(&wire, bits, &mut out).expect("whole samples");
for v in &out {
assert!((-1.0..=1.0).contains(v), "{bits}-bit clamp failed: {v}");
}
assert!(out[0] > 0.99 && out[1] < -0.99);
assert!(out[2] > 0.99 && out[3] < -0.99);
}
}
/// Every rate the plane admits, for the loops below — both families, so a rung that only
/// divides one of them can never be pinned by accident.
const RATES: [u32; 5] = [44_100, 48_000, 88_200, 96_000, 176_400];
/// The ladder must never hand back a frame that does not fit, and must take the longest one
/// that does. This is the check that keeps the plane off the fragmentation path.
#[test]
fn the_frame_ladder_never_exceeds_the_datagram() {
for rate in RATES {
for bits in [BITS_16, BITS_24] {
for budget in [600usize, 900, 1200, 1387, 1400, 1440, 9000] {
let chosen = frame_us_for(rate, bits, 2, budget);
if let Some(us) = chosen {
let bytes = frame_payload_bytes(rate, bits, 2, us);
assert!(
bytes + PCM_HEADER_LEN <= budget,
"{rate}/{bits} at {budget} B chose {us} µs = {bytes} B + header"
);
}
// …and nothing longer would have fitted — including the case where NOTHING
// did. `None` is a real answer, not a hole in the test: 176 400/24-bit needs
// 1 069 B for even a 1 ms frame, so a small datagram declines it exactly the
// way it declines hi-res surround (§4.2), and the caller falls back to Opus.
let longer_than = chosen.unwrap_or(0);
for &longer in FRAME_US_LADDER.iter().take_while(|&&x| x > longer_than) {
assert!(
frame_payload_bytes(rate, bits, 2, longer) + PCM_HEADER_LEN > budget,
"{rate}/{bits} at {budget} B should have chosen {longer} µs"
);
}
}
}
}
// The decline above, pinned rather than merely tolerated.
assert_eq!(frame_us_for(176_400, BITS_24, 2, 1_000), None);
assert_eq!(frame_us_for(176_400, BITS_24, 2, 1_400), Some(1000));
}
/// The concrete ladder the default 1472-byte MTU ceiling produces. Pinned because these are
/// the numbers the design argues from, and a silent change to any of them changes the
/// plane's packet rate.
#[test]
fn the_default_mtu_yields_the_documented_ladder() {
// Conservative usable datagram at the 1472-byte discovery ceiling.
let d = 1400;
assert_eq!(frame_us_for(48_000, BITS_16, 2, d), Some(5000));
assert_eq!(frame_us_for(48_000, BITS_24, 2, d), Some(4000));
assert_eq!(frame_us_for(96_000, BITS_16, 2, d), Some(3000));
assert_eq!(frame_us_for(96_000, BITS_24, 2, d), Some(2000));
// The doc's original 2.5 ms at 96/24 does NOT fit: 240 × 2 × 3 = 1440 B of payload
// against a ~1387 B budget. Sizing from a coded estimate is what hid that.
assert!(frame_payload_bytes(96_000, BITS_24, 2, 2500) + PCM_HEADER_LEN > d);
}
/// Every rung divides the **48 kHz family** into whole samples — that much of the original
/// claim is true and load-bearing, because it is what lets a 48/96 kHz session treat its rung
/// as a duration.
#[test]
fn every_ladder_rung_is_whole_samples_at_the_48k_family() {
for &us in &FRAME_US_LADDER {
for rate in [48_000u64, 96_000] {
assert_eq!(
rate * us as u64 % 1_000_000,
0,
"{us} µs is not a whole number of samples at {rate} Hz"
);
}
}
}
/// …and the rest of that claim was false the moment the 44.1 kHz family was admitted, so this
/// pins what is ACTUALLY true instead: a fractional rung carries the floor, which is short of
/// its label and never over it.
///
/// Short is the safe direction for both things the ladder decides — the payload stays inside
/// the datagram, and the ring is sized from the same floor at both ends — and it is precisely
/// the wrong direction for a clock, which is why `frame_duration_ns` exists and why a `pts_ns`
/// must never advance by the rung.
#[test]
fn a_fractional_rung_carries_the_floor_and_is_short_of_its_label() {
// The rate must land on a whole sample per channel; a whole INTERLEAVED count is not
// enough (5 ms at 44 100 Hz stereo is 441 interleaved samples — but 220.5 per channel).
let divides = |rate: u64, us: u64| rate * us % 1_000_000 == 0;
let mut fractional = 0;
for &us in &FRAME_US_LADDER {
for rate in RATES {
for ch in [2u8, 6] {
let n = samples_per_frame(rate, us, ch);
assert_eq!(
n % ch as usize,
0,
"{n} samples is not whole frames at {ch}ch"
);
let real_ns = frame_duration_ns(n, rate, ch);
let nominal_ns = us as u64 * 1_000;
assert!(
real_ns <= nominal_ns,
"{rate} Hz/{ch}ch at {us} µs carries {real_ns} ns — LONGER than its label, \
so the payload could outgrow the datagram it was sized against"
);
if divides(rate as u64, us as u64) {
assert_eq!(real_ns, nominal_ns, "{rate} Hz at {us} µs must be exact");
} else {
fractional += 1;
// Under one sample per channel short — the floor, not a lost frame.
assert!(
nominal_ns - real_ns < frame_duration_ns(ch as usize, rate, ch) + 1
);
}
}
}
}
// 44 100 divides no rung, 88 200 divides only 5 000 µs, 176 400 only 5 000 and 2 500 —
// 7 + 6 + 5 = 18 fractional pairings, at two channel counts.
assert_eq!(
fractional, 36,
"the fractional set is not what the doc claims"
);
// The headline number the FRAME_US_LADDER doc quotes.
assert_eq!(samples_per_frame(44_100, 5000, 2), 440, "220 per channel");
assert_eq!(frame_duration_ns(440, 44_100, 2), 4_988_662);
}
/// A `pts_ns` advanced by the negotiated `frame_us` instead of by the frame's real duration
/// runs measurably fast, and this is the size of it: 0.23 % at 44 100 Hz, which is 2.3 ms of
/// invented time per second — a drift the A/V sync loop would fight forever and never win.
///
/// Stated as a test because the failure is silent at the source: the timestamps stay
/// self-consistent, every stat agrees with them, and only the picture drifts.
#[test]
fn advancing_a_clock_by_the_nominal_frame_would_drift() {
let (rate, us, ch) = (44_100u32, 5000u32, 2u8);
let n = samples_per_frame(rate, us, ch);
// One second of frames, by each clock.
let frames = 1_000_000 / us as u64;
let nominal_ns = frames * us as u64 * 1_000;
let real_ns = frames * frame_duration_ns(n, rate, ch);
let fast_ppm = (nominal_ns - real_ns) * 1_000_000 / real_ns;
assert!(
(2_200..2_400).contains(&fast_ppm),
"the nominal clock runs {fast_ppm} ppm fast — expected ~2 268 (0.23 %)"
);
// …and the drift-free formulation: stamping from the session's RUNNING sample total does
// not accumulate the per-frame floor at all. Over the same second that is 26 ns of
// difference against the summed frames — both are fine, and the point is the order of
// magnitude either beats the 2.3 ms the nominal clock invented.
let exact_ns = frame_duration_ns(frames as usize * n, rate, ch);
assert!(
exact_ns >= real_ns && exact_ns - real_ns < frames,
"summing floored frames must stay within 1 ns per frame of the running total \
({exact_ns} vs {real_ns} over {frames} frames)"
);
// On a rate the rung divides, the two clocks are the same clock — which is why this went
// unnoticed for as long as the ladder was 48/96 only.
let n48 = samples_per_frame(48_000, us, ch);
assert_eq!(frame_duration_ns(n48, 48_000, ch), us as u64 * 1_000);
}
/// The rate set the plane admits, and the shape of the two families. A rate added here without
/// the `JitterPolicy` arithmetic to carry it is the §4.1 defect coming back.
#[test]
fn the_supported_rate_set_is_both_families() {
for rate in RATES {
assert!(rate_is_supported(rate), "{rate} Hz must be carried");
}
// 192 kHz is out by the §3 scope decision; the rest are simply not audio rates this
// protocol negotiates. `0` matters most: it is the wire's "absent" value.
for rate in [
0u32, 8_000, 16_000, 22_050, 32_000, 64_000, 192_000, 384_000,
] {
assert!(!rate_is_supported(rate), "{rate} Hz must not be offered");
}
}
/// The costs the §8.4 gate and `plan_audio_budget` reason about.
#[test]
fn the_plane_costs_what_the_design_says() {
assert_eq!(bitrate_kbps(48_000, BITS_16, 2), 1_536);
assert_eq!(bitrate_kbps(48_000, BITS_24, 2), 2_304);
assert_eq!(bitrate_kbps(96_000, BITS_16, 2), 3_072);
assert_eq!(bitrate_kbps(96_000, BITS_24, 2), 4_608);
// The 44.1 family, floored — and the top of the ladder, which is 8.5 Mbps taken off the
// top of a link ABR cannot reclaim. Worth seeing written down before anyone offers it.
assert_eq!(bitrate_kbps(44_100, BITS_16, 2), 1_411); // 1 411.2, floored
assert_eq!(bitrate_kbps(44_100, BITS_24, 2), 2_116); // 2 116.8, floored
assert_eq!(bitrate_kbps(88_200, BITS_24, 2), 4_233); // 4 233.6, floored
assert_eq!(bitrate_kbps(176_400, BITS_24, 2), 8_467); // 8 467.2, floored
}
/// A truncated datagram must be rejected outright rather than decoded as a shifted frame —
/// half a sample at the end would desync every sample after it.
#[test]
fn a_partial_sample_is_rejected() {
let mut out = Vec::new();
assert!(to_f32(&[0, 0, 0], BITS_16, &mut out).is_none());
assert!(to_f32(&[0, 0], BITS_24, &mut out).is_none());
assert_eq!(to_f32(&[], BITS_24, &mut out), Some(0));
}
/// Concealment with nothing to conceal from must say so, so the caller emits silence and
/// lets the ring re-prime instead of playing an uninitialised buffer.
#[test]
fn concealment_needs_a_frame_to_build_from() {
let mut c = PcmConceal::new();
let mut out = Vec::new();
assert!(!c.conceal(&mut out), "nothing to repeat yet");
c.accept(&[0.5; 240]);
assert!(c.conceal(&mut out));
assert_eq!(out.len(), 240);
}
/// A sustained gap must decay toward silence rather than loop a fragment, and must never
/// grow louder than the audio it is standing in for.
#[test]
fn a_sustained_gap_decays_to_silence() {
let mut c = PcmConceal::new();
c.accept(&[1.0; 128]);
let mut out = Vec::new();
let mut peaks = Vec::new();
for _ in 0..8 {
assert!(c.conceal(&mut out));
peaks.push(out.iter().fold(0f32, |m, s| m.max(s.abs())));
}
for w in peaks.windows(2) {
assert!(w[1] <= w[0] + 1e-6, "concealment grew louder: {peaks:?}");
}
assert!(peaks[0] <= 1.0, "never louder than the source: {peaks:?}");
assert!(
*peaks.last().unwrap() < 0.05,
"should have faded out: {peaks:?}"
);
assert_eq!(c.run(), 8);
// A real frame ends the run.
c.accept(&[0.25; 128]);
assert_eq!(c.run(), 0);
}
/// The fade must actually reach (near) zero at the splice point, which is the whole reason
/// it exists — a repeat that ends mid-waveform steps audibly into whatever follows.
#[test]
fn the_fade_lands_on_silence() {
let mut c = PcmConceal::new();
c.accept(&[1.0; 64]);
let mut out = Vec::new();
c.conceal(&mut out);
assert!(out[0] > 0.9, "starts at full level: {}", out[0]);
assert!(
*out.last().unwrap() < 0.01,
"ends at silence: {}",
out.last().unwrap()
);
}
}
crates/punktfunk-core/src/client/control.rs
+17
View File
@@ -67,6 +67,23 @@ pub(crate) struct Negotiated {
pub(crate) chroma_format: u8,
/// Resolved audio channel count (2/6/8) — what the Opus decoders must be built from.
pub(crate) audio_channels: u8,
/// Which audio plane the host resolved ([`crate::quic::Welcome::audio_codec`]):
/// [`crate::quic::AUDIO_CODEC_OPUS`] — Opus on `0xC9`, every legacy session — or
/// [`crate::quic::AUDIO_CODEC_PCM`] — lossless PCM on `0xD3`. It is what SELECTS the
/// decoder, and nothing else can: a 48 kHz/16-bit PCM session and a 48 kHz Opus session
/// are identical in every other resolved field.
pub(crate) audio_codec: u8,
/// Resolved sample rate (Hz) — what the host is actually capturing, which may be lower
/// than the client asked for. The client opens its output device from THIS, never from
/// its request (`design/hi-res-audio.md` §4.3).
pub(crate) audio_rate_hz: u32,
/// Resolved sample depth (16 or 24) — the stride `0xD3` payloads are unpacked at. Reading
/// a 24-bit payload at 2 bytes per sample is not silence, it is noise.
pub(crate) audio_bits: u8,
/// Resolved `0xD3` frame duration (µs); `0` on an Opus session, whose frames are the
/// `0xC9` plane's fixed 5 ms. Negotiated from the path MTU, never assumed — see
/// [`crate::quic::Welcome::audio_frame_us`].
pub(crate) audio_frame_us: u16,
/// The single codec the host will emit (`quic::CODEC_*`).
pub(crate) codec: u8,
/// The host capability bitfield ([`crate::quic::Welcome::host_caps`]):
crates/punktfunk-core/src/client/mod.rs
+258 -1
View File
@@ -426,12 +426,71 @@ pub struct NativeClient {
/// host that omits it (→ `2`) yields working stereo. The `0xC9` audio frames are encoded with the
/// matching layout.
pub audio_channels: u8,
/// Which audio plane the host resolved for this session ([`Welcome::audio_codec`]):
/// [`quic::AUDIO_CODEC_OPUS`] — Opus frames on `0xC9` (with `0xD2` redundancy when
/// negotiated), the default and what every older host yields — or [`quic::AUDIO_CODEC_PCM`]
/// — lossless interleaved PCM on `0xD3` ([`crate::audio::pcm`]).
///
/// This is the field that SELECTS the decoder, and nothing else can: a 48 kHz/16-bit lossless
/// session and a 48 kHz Opus session agree on every other resolved value. A session runs one
/// plane or the other for its whole life — the host never switches mid-session, because the
/// client's output device is open at a fixed format.
pub audio_codec: u8,
/// The sample rate the host resolved ([`Welcome::audio_rate_hz`]) — `48_000` for every Opus
/// session and for an older host, or the rate a hi-res session actually landed on, which may
/// be lower than the client asked for.
///
/// ⚠ **Open the output device from THIS, never from the request.** A client that asks for
/// 96 kHz, is answered 48 kHz, and opens at 96 kHz anyway is the exact failure
/// `design/hi-res-audio.md` §4.3 is written around, one end further along.
pub audio_sample_rate_hz: u32,
/// The sample depth the host resolved ([`Welcome::audio_bits`]) — 16 or 24. Meaningful only
/// on the `0xD3` plane, where it is the stride payloads are unpacked at; `16` on every Opus
/// session (whose samples reach the embedder as f32 regardless).
pub audio_bits: u8,
/// How much audio one `0xD3` datagram carries, in microseconds ([`Welcome::audio_frame_us`]);
/// `0` on an Opus session, whose frames are the `0xC9` plane's fixed 5 ms.
///
/// Negotiated from the path MTU rather than assumed, so it must not be hardcoded — at
/// 96 kHz/24-bit the default MTU ceiling only leaves room for 2 ms frames. The C surface
/// exposes the same figure as [`crate::abi::punktfunk_connection_audio_frame_us`].
///
/// ⚠ **Nominal, not a duration.** A frame carries a whole number of samples per channel, and
/// the 44.1 kHz family divides no rung of the ladder — a 5 ms frame at 44 100 Hz is 220
/// samples per channel, 4 988 662 ns. Size rings from this (that is what it is for) and take
/// timing from [`crate::audio::pcm::frame_duration_ns`] of the real sample count; advancing a
/// clock by this figure invents 2.3 ms per second, forever.
pub audio_frame_us: u16,
/// The video codec the host resolved and will emit ([`Welcome::codec`]) — [`quic::CODEC_H264`],
/// [`quic::CODEC_HEVC`] (default / older host), or [`quic::CODEC_AV1`]. The client builds its
/// decoder from THIS, never assuming HEVC.
pub codec: u8,
}
impl NativeClient {
/// What the audio plane costs, in kbps — the figure a stats line or HUD should show so a user
/// who turned the lossless plane on can see what it took (`design/hi-res-audio.md` §4.6).
///
/// `Some` only for the lossless plane, where the answer is **exact rather than measured**:
/// PCM is constant-bitrate by construction, so `rate × depth × channels` IS the wire rate and
/// a byte counter would only add sampling noise to a number already known precisely. `None`
/// for Opus, which is VBR and whose ladder position is chosen host-side by
/// [`crate::audio::plan_audio_budget`] — the client has no honest figure to report, and
/// inventing one from a short window would read as jitter.
///
/// Payload only: the 13-byte per-datagram header and QUIC's own framing are not counted, on
/// the grounds that the same is true of every other bitrate this project quotes.
pub fn audio_kbps(&self) -> Option<u32> {
(self.audio_codec == crate::quic::AUDIO_CODEC_PCM).then(|| {
crate::audio::pcm::bitrate_kbps(
self.audio_sample_rate_hz,
self.audio_bits,
self.audio_channels,
)
})
}
}
/// Pin the calling thread to the user-interactive QoS class on Apple targets.
///
/// The Apple client drains every plane on `.userInteractive` Thread s (video pump, audio,
@@ -563,6 +622,46 @@ fn os_hostname() -> Option<String> {
None
}
/// The `client_caps` a session actually advertises: what the embedder passed, plus the two bits
/// core decides for itself. A named function rather than an expression inside `connect` because
/// this one line decides whether a client asks a host for 1.5–4.6 Mbps of extra bandwidth, and
/// the rule deserves to be pinnable by a test.
///
/// - [`quic::CLIENT_CAP_AUDIO_RED`] is set **always**. Redundancy is a pure "I can decode it": the
/// recovery happens inside core's own datagram demux and re-inserts the rebuilt frame into the
/// same queue, so every embedder benefits without knowing the plane exists and none of them can
/// forget to opt in. It costs ~1 %, and the host still decides whether to spend it.
/// - [`quic::CLIENT_CAP_AUDIO_HIRES`] is set **only when the caller asked for a non-default
/// format**. It means *capable **and** the user turned it on* (the `VIDEO_CAP_444` precedent),
/// it costs 1.5–4.6 Mbps taken off the top of a link ABR can neither see nor reclaim, and it is
/// answered by the host re-formatting the wire — so a client that advertised it without being
/// able to open the output would spend that bandwidth to play nothing. Only the embedder knows
/// whether its device can open at the format, so only the embedder can ask, and asking IS
/// calling [`NativeClient::connect_with_audio_format`] with a non-default one.
///
/// The derived bit is OR'd into the caller's rather than replacing it, which is what leaves the
/// 48 kHz/16-bit lossless request expressible at all: those parameters are indistinguishable from
/// the legacy ones, so an embedder that genuinely wants that (rare — 24-bit is where the plane
/// earns its bandwidth) sets the bit itself and is not overridden.
fn advertised_client_caps(client_caps: u8, audio_rate_hz: u32, audio_bits: u8) -> u8 {
// The bit means "the caller SPECIFIED a format", not "the format differs from the default".
//
// Those two rules agree everywhere except one place, and that place matters: 48 kHz/16-bit is
// the cheapest lossless rung (1.5 Mbps against Opus's 256 kbps) and is also the default, so a
// "differs from the default" rule makes it the one format on the ladder that cannot be asked
// for. `0` is the unspecified value — [`NativeClient::connect`] passes it, and the wire encodes
// an explicit 48 000/16 identically to absent — so keying on "non-zero" separates *asking for
// 48/16 lossless* from *not asking at all* without costing a wire byte.
let hires = audio_rate_hz != 0 || audio_bits != 0;
client_caps
| crate::quic::CLIENT_CAP_AUDIO_RED
| if hires {
crate::quic::CLIENT_CAP_AUDIO_HIRES
} else {
0
}
}
impl NativeClient {
/// Connect to a `punktfunk/1` host and start the session at (up to) `mode`. Blocks until the
/// handshake completes or `timeout` elapses.
@@ -574,6 +673,13 @@ impl NativeClient {
/// `identity`: this client's persistent self-signed identity (PEM cert + PKCS#8 key,
/// see [`endpoint::generate_identity`]), presented via TLS client auth so a host can
/// recognize a paired client. `None` = anonymous (rejected by hosts requiring pairing).
///
/// Requests the legacy audio format — Opus at 48 kHz / 16-bit, the plane every build has
/// spoken — so the `Hello` this produces is byte-identical to the pre-hi-res one. An embedder
/// whose user turned the lossless plane on calls
/// [`connect_with_audio_format`](Self::connect_with_audio_format) instead; this stays as it
/// is so that every existing caller (four clients, the CLI, the host's own integration tests)
/// keeps compiling and keeps behaving identically.
#[allow(clippy::too_many_arguments)]
pub fn connect(
host: &str,
@@ -619,6 +725,91 @@ impl NativeClient {
pin: Option<[u8; 32]>,
identity: Option<(String, String)>,
timeout: Duration,
) -> Result<NativeClient> {
Self::connect_with_audio_format(
host,
port,
mode,
compositor,
gamepad,
bitrate_kbps,
video_caps,
audio_channels,
// 0/0 = UNSPECIFIED, which is what keeps this path's `Hello` byte-identical to the
// pre-hi-res one. Passing an explicit 48 000/16 here would read as "asked for the
// cheapest lossless rung" under the rule in `advertised_client_caps`.
0,
0,
video_codecs,
preferred_codec,
display_hdr,
client_caps,
frame_parts,
launch,
name,
pin,
identity,
timeout,
)
}
/// [`connect`](Self::connect), plus the audio format this client is **asking** for:
/// `audio_rate_hz` (any rate [`crate::audio::pcm::rate_is_supported`] admits — 48 000, 96 000,
/// and the 44.1 kHz family 44 100 / 88 200 / 176 400) and `audio_bits` (16 or 24).
///
/// Everything else is identical. What the pair actually does is decide whether the `Hello`
/// carries [`quic::CLIENT_CAP_AUDIO_HIRES`], and the rule is deliberately narrow:
///
/// **The bit is set exactly when the caller SPECIFIES a format at all** (either argument
/// non-zero; `0` means unspecified, which is what [`connect`](Self::connect) passes).
/// Deliberately not "differs from 48 kHz/16-bit": that rule would make the cheapest lossless
/// rung — 48 kHz/16-bit, 1.5 Mbps — the one format on the ladder nobody could request.
/// It is NOT set unconditionally, and that is the whole difference between it and
/// [`quic::CLIENT_CAP_AUDIO_RED`] — which core ORs in for every session below, because
/// redundancy is a pure "I can decode it" that costs ~1 % and is recovered inside core where
/// no embedder can forget to opt in. Hi-res is the opposite on both counts: it means *capable
/// **and** the user turned it on* (the `VIDEO_CAP_444` precedent), it costs 1.5–4.6 Mbps taken
/// off the top of a link ABR can neither see nor reclaim, and it is answered by the host
/// re-formatting the wire — so a client that advertised it without being able to open the
/// output would spend that bandwidth to play nothing. Only the embedder knows whether its
/// device can open at the format, so only the embedder can ask, and asking is exactly what
/// calling this function with a non-default format IS.
///
/// Two consequences worth stating rather than discovering:
///
/// - **48 kHz/16-bit lossless is not reachable through this parameter pair** — that request is
/// byte-identical to the legacy one, so it stays Opus. Ask for 48 kHz/**24**-bit to get
/// lossless at the default rate (the depth is where lossless earns its keep anyway, and
/// 16-bit PCM would spend 1.5 Mbps to sound like transparent 256 kbps Opus). An embedder
/// that genuinely wants 48/16 on the `0xD3` plane can still set
/// [`quic::CLIENT_CAP_AUDIO_HIRES`] in `client_caps` itself — the bit derived here is OR'd
/// into what the caller passed, never substituted for it.
/// - The host may still answer Opus. It resolves the five-condition gate in
/// `design/hi-res-audio.md` §8.4 and a decline is not a failure; read
/// [`audio_codec`](Self::audio_codec) / [`audio_sample_rate_hz`](Self::audio_sample_rate_hz)
/// / [`audio_bits`](Self::audio_bits) afterwards and open the device from those.
#[allow(clippy::too_many_arguments)]
pub fn connect_with_audio_format(
host: &str,
port: u16,
mode: Mode,
compositor: CompositorPref,
gamepad: GamepadPref,
bitrate_kbps: u32,
video_caps: u8,
audio_channels: u8,
audio_rate_hz: u32,
audio_bits: u8,
video_codecs: u8,
preferred_codec: u8,
display_hdr: Option<HdrMeta>,
client_caps: u8,
frame_parts: bool,
launch: Option<String>,
name: Option<String>,
pin: Option<[u8; 32]>,
identity: Option<(String, String)>,
timeout: Duration,
) -> Result<NativeClient> {
let frame_chan = Arc::new(FrameChannel::new());
let (audio_tx, audio_rx) = std::sync::mpsc::sync_channel::<AudioPacket>(AUDIO_QUEUE);
@@ -716,6 +907,8 @@ impl NativeClient {
bitrate_kbps,
video_caps,
audio_channels,
audio_rate_hz,
audio_bits,
video_codecs,
preferred_codec,
display_hdr,
@@ -725,7 +918,10 @@ impl NativeClient {
// embedder benefits without knowing the plane exists — and none of them can
// forget to opt in. The bit is a pure "I can decode it"; the host still
// decides whether to spend the extra ~1 %.
client_caps: client_caps | crate::quic::CLIENT_CAP_AUDIO_RED,
//
// CLIENT_CAP_AUDIO_HIRES deliberately does NOT join it unconditionally — see
// `advertised_client_caps` for the rule and why the two bits differ.
client_caps: advertised_client_caps(client_caps, audio_rate_hz, audio_bits),
frame_parts,
launch,
name,
@@ -850,6 +1046,10 @@ impl NativeClient {
color: negotiated.color,
chroma_format: negotiated.chroma_format,
audio_channels: negotiated.audio_channels,
audio_codec: negotiated.audio_codec,
audio_sample_rate_hz: negotiated.audio_rate_hz,
audio_bits: negotiated.audio_bits,
audio_frame_us: negotiated.audio_frame_us,
codec: negotiated.codec,
})
}
@@ -1721,3 +1921,60 @@ mod host_port_tests {
.is_ok());
}
}
#[cfg(test)]
mod client_caps_tests {
use super::advertised_client_caps;
use crate::audio::pcm::{BITS_16, BITS_24};
use crate::audio::SAMPLE_RATE_HZ;
use crate::quic::{CLIENT_CAP_AUDIO_HIRES, CLIENT_CAP_AUDIO_RED, CLIENT_CAP_CURSOR};
/// The one line that decides whether a client asks a host to spend 1.5–4.6 Mbps on audio.
/// Redundancy is unconditional (core recovers it, so nobody can forget to opt in); hi-res is
/// not, because it means "capable AND turned on" and only a non-default request expresses
/// that. A regression here is silent in every test that does not look for it: the session
/// still works, it just costs several megabits nobody asked for.
#[test]
fn hires_is_advertised_only_when_the_caller_specified_a_format() {
// The legacy request is UNSPECIFIED (0/0): redundancy on, hi-res off, the embedder's own
// bits untouched. This is what `connect` and every pre-v24 C entry point pass.
let legacy = advertised_client_caps(CLIENT_CAP_CURSOR, 0, 0);
assert_eq!(legacy & CLIENT_CAP_AUDIO_RED, CLIENT_CAP_AUDIO_RED);
assert_eq!(legacy & CLIENT_CAP_AUDIO_HIRES, 0);
assert_eq!(legacy & CLIENT_CAP_CURSOR, CLIENT_CAP_CURSOR);
// …and with no embedder bits at all, which is what every `connect` caller produces.
assert_eq!(advertised_client_caps(0, 0, 0), CLIENT_CAP_AUDIO_RED);
// The rung this rule exists for: 48 kHz/16-bit is the DEFAULT and also the cheapest
// lossless format. Asking for it explicitly must be a request, or it is the one point on
// the ladder no caller can reach.
assert_eq!(
advertised_client_caps(0, SAMPLE_RATE_HZ, BITS_16) & CLIENT_CAP_AUDIO_HIRES,
CLIENT_CAP_AUDIO_HIRES,
"explicit 48 kHz/16-bit is a lossless request, not a legacy one"
);
// Specifying either half alone is still a request.
for (rate, bits) in [
(SAMPLE_RATE_HZ, BITS_24),
(96_000, BITS_16),
(96_000, BITS_24),
(0, BITS_24),
(96_000, 0),
] {
let caps = advertised_client_caps(0, rate, bits);
assert_eq!(
caps & CLIENT_CAP_AUDIO_HIRES,
CLIENT_CAP_AUDIO_HIRES,
"{rate} Hz / {bits}-bit must ask for the lossless plane"
);
assert_eq!(caps & CLIENT_CAP_AUDIO_RED, CLIENT_CAP_AUDIO_RED);
}
// The escape hatch: 48 kHz/16-bit is indistinguishable from a legacy request, so an
// embedder that wants lossless AT the default format sets the bit itself — and is not
// overridden, because the derived bit is OR'd in rather than substituted.
let explicit = advertised_client_caps(CLIENT_CAP_AUDIO_HIRES, SAMPLE_RATE_HZ, BITS_16);
assert_eq!(explicit & CLIENT_CAP_AUDIO_HIRES, CLIENT_CAP_AUDIO_HIRES);
}
}
crates/punktfunk-core/src/client/planes.rs
+21 -2
View File
@@ -2,6 +2,14 @@
/// Audio packets buffered for the embedder: 64 × 5 ms = 320 ms of slack. A lagging
/// embedder drops the newest packet (the audio renderer conceals the gap).
///
/// Counted in PACKETS, not milliseconds, and the lossless `0xD3` plane shares it — so on a
/// session whose negotiated frame is shorter than 5 ms the same 64 entries are proportionally
/// less time (128 ms at the 2 ms frame 96 kHz/24-bit lands on at the default MTU). Left as a
/// packet count deliberately: 128 ms is still far above the 15–90 ms the de-jitter policy
/// targets, and a depth that changed with the negotiated format would make the overflow
/// behaviour session-dependent for no measured gain. Worth knowing before anyone reads "320 ms"
/// as a guarantee.
pub(crate) const AUDIO_QUEUE: usize = 64;
/// Rumble updates buffered for the embedder. Overflow drops the NEWEST update (same
@@ -56,11 +64,22 @@ pub(crate) const CURSOR_SHAPE_QUEUE: usize = 8;
/// newest (try_send), healed by the very next frame's datagram.
pub(crate) const CURSOR_STATE_QUEUE: usize = 8;
/// One Opus packet from the host's audio datagram stream (48 kHz stereo, 5 ms frames).
/// One packet from the host's audio datagram stream — an Opus frame off `0xC9`/`0xD2`
/// (48 kHz, 5 ms) or one lossless PCM frame off `0xD3`, at the negotiated rate/depth and one
/// rung of [`crate::audio::pcm::FRAME_US_LADDER`] long.
///
/// The two planes share this type and the queue that carries it because they share a header:
/// `seq` and `pts_ns` mean the same thing on both. What they do NOT share is how `data` is read,
/// and nothing per-packet says which — the session's
/// [`NativeClient::audio_codec`](crate::client::NativeClient::audio_codec) does, once, for the
/// whole session.
#[derive(Clone, Debug)]
pub struct AudioPacket {
pub seq: u32,
pub pts_ns: u64,
/// The raw Opus payload — feed it to an Opus decoder as one frame.
/// The frame's payload: a raw Opus packet to hand a decoder as one frame, or — on an
/// [`AUDIO_CODEC_PCM`](crate::quic::AUDIO_CODEC_PCM) session — interleaved little-endian
/// integer samples to unpack with [`crate::audio::pcm::to_f32`]. Empty is a DTX silence
/// marker on the Opus plane.
pub data: Vec<u8>,
}
crates/punktfunk-core/src/client/pump/datagram_task.rs
+105
View File
@@ -122,6 +122,28 @@ pub(super) async fn run(
let _ = pad_audio_tx.try_send(f);
}
}
// The lossless plane feeds the SAME queue as `0xC9`, deliberately: the header is
// identical by design, so seq/pts (and therefore the gap tracker, the de-jitter
// policy and A/V sync) mean exactly what they mean on the Opus plane, and only the
// payload format differs. Keeping one queue means the whole downstream pipeline —
// `AUDIO_QUEUE`, `next_audio`, the in-core decode in `abi.rs` — is unchanged, and the
// format that tells a consumer how to read `data` is the session-wide
// `Welcome::audio_codec` rather than anything per-packet.
//
// A session runs one plane or the other for its whole life, so the two arms can never
// interleave into that queue. `AudioRedRecovery` is not involved: `0xD2` redundancy is
// undefined for this plane and never sent with it (it would double a bitrate that is
// already the largest on the connection), so a lost datagram here is concealed by
// `pcm::PcmConceal` at the decode site instead of reconstructed here.
Some(&crate::quic::AUDIO_PCM_MAGIC) => {
if let Some((seq, pts_ns, pcm)) = crate::quic::decode_audio_pcm_datagram(&d) {
let _ = audio_tx.try_send(AudioPacket {
seq,
pts_ns,
data: pcm.to_vec(),
});
}
}
Some(&crate::quic::HDR_META_MAGIC) => {
if let Some(m) = crate::quic::decode_hdr_meta_datagram(&d) {
let _ = hdr_meta_tx.try_send(m);
@@ -146,3 +168,86 @@ pub(super) async fn run(
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// A `0xD3` datagram must land in the SAME queue `0xC9` feeds, with its sequence and
/// presentation time intact and its payload byte-for-byte what the host put on the wire.
///
/// Driven through the REAL demux loop over a real QUIC connection rather than by calling the
/// decoder directly, because the decoder is not what this arm adds: the arm is a tag, a sink
/// and an absence (no `AudioRedRecovery`), and only the loop can be wrong about those. The
/// endpoint pair is the one `endpoint`'s own MTU measurement uses; a single datagram over
/// loopback costs milliseconds.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn a_lossless_datagram_reaches_the_audio_sink() {
let server = crate::quic::endpoint::server("127.0.0.1:0".parse().unwrap()).unwrap();
let addr = server.local_addr().unwrap();
let client = crate::quic::endpoint::client_insecure().unwrap();
let accept = tokio::spawn(async move {
let incoming = server.accept().await.expect("incoming");
(server, incoming.await.expect("host side connects"))
});
let client_conn = client.connect(addr, "punktfunk").unwrap().await.unwrap();
let (_server_ep, host_conn) = accept.await.unwrap();
// Every plane's sink, so nothing the loop touches is a closed channel — the receivers
// must outlive the task or `try_send` would fail for a reason the test does not intend.
let (audio_tx, audio_rx) = std::sync::mpsc::sync_channel::<AudioPacket>(8);
let (rumble_tx, _rumble_rx) = std::sync::mpsc::sync_channel::<RumbleUpdate>(8);
let (hidout_tx, _hidout_rx) = std::sync::mpsc::sync_channel(8);
let (pad_audio_tx, _pad_audio_rx) = std::sync::mpsc::sync_channel(8);
let (hdr_meta_tx, _hdr_meta_rx) = std::sync::mpsc::sync_channel(8);
let (host_timing_tx, _host_timing_rx) = std::sync::mpsc::sync_channel(8);
let (cursor_state_tx, _cursor_state_rx) = std::sync::mpsc::sync_channel(8);
let rumble_feed =
super::super::rumble::RumbleFeed(Arc::new(super::super::rumble::RumbleShared::new()));
tokio::spawn(run(
client_conn,
audio_tx,
rumble_tx,
rumble_feed,
hidout_tx,
pad_audio_tx,
hdr_meta_tx,
host_timing_tx,
Arc::new(Mutex::new(
super::super::frame_channel::EncodeLatAcc::default(),
)),
cursor_state_tx,
));
// One frame of 48 kHz/24-bit stereo, sized the way the host is REQUIRED to size it: from
// the connection's own `max_datagram_size`, because this plane is never fragmented and an
// oversized datagram is not sent at all. Hardcoding 5 ms here fails outright — 1440 B of
// payload does not fit before MTU discovery settles, which is exactly the trap §4.2
// warns the host about, reproduced by accident on the first attempt at this test.
let bits = crate::audio::pcm::BITS_24;
let max_dg = host_conn
.max_datagram_size()
.expect("datagrams are enabled");
let frame_us =
crate::audio::pcm::frame_us_for(crate::audio::SAMPLE_RATE_HZ, bits, 2, max_dg)
.expect("some rung of the ladder fits");
let n = crate::audio::pcm::samples_per_frame(crate::audio::SAMPLE_RATE_HZ, frame_us, 2);
let samples: Vec<f32> = (0..n).map(|i| (i as f32 * 0.01).sin() * 0.8).collect();
let mut wire = Vec::new();
crate::audio::pcm::from_f32(&samples, bits, &mut wire);
assert_eq!(wire.len(), n * 3);
host_conn
.send_datagram(crate::quic::encode_audio_pcm_datagram(7, 1_234_567, &wire).into())
.expect("datagram fits the path");
let got = tokio::task::spawn_blocking(move || {
audio_rx.recv_timeout(std::time::Duration::from_secs(5))
})
.await
.unwrap()
.expect("the 0xD3 arm must feed the audio sink");
assert_eq!(got.seq, 7);
assert_eq!(got.pts_ns, 1_234_567);
assert_eq!(got.data, wire, "the payload must cross unmodified");
}
}
crates/punktfunk-core/src/client/pump/handshake.rs
+20
View File
@@ -162,6 +162,15 @@ pub(super) async fn connect_and_handshake(args: &WorkerArgs) -> Result<Handshake
// (design/shard-payload-reneg.md W0.3 — the host only renegotiates, and only
// grows to jumbo, when this advertises it).
max_shard_payload: crate::config::max_shard_payload() as u16,
// The audio format this client is ASKING for. At the legacy 48 kHz / 16-bit pair
// (what every `connect` caller gets) the encoder omits both fields and the Hello
// stays byte-identical to the pre-hi-res wire form. Anything else came from
// `connect_with_audio_format`, which is also what set CLIENT_CAP_AUDIO_HIRES in
// `client_caps` — capable AND turned on, the VIDEO_CAP_444 precedent. The two
// travel together on purpose: the bit is the opt-in and these are only its
// parameters, so neither is meaningful without the other.
audio_rate_hz: args.audio_rate_hz,
audio_bits: args.audio_bits,
}
.encode(),
)
@@ -252,6 +261,17 @@ pub(super) async fn connect_and_handshake(args: &WorkerArgs) -> Result<Handshake
color: welcome.color,
chroma_format: welcome.chroma_format,
audio_channels: welcome.audio_channels,
// The RESOLVED audio format, taken straight off the Welcome and never reconciled
// with what we asked for: the host may legitimately answer lower (or plain Opus)
// and its answer is the only authority — the rule at both ends is "never claim a
// rate you did not get" (`design/hi-res-audio.md` §4.3/§9). A default Opus
// session decodes these as `AUDIO_CODEC_OPUS` / 48 000 / 16 / 0, which is exactly
// what an older host that omits the whole tail also yields, so the legacy path
// reaches the embedder unchanged.
audio_codec: welcome.audio_codec,
audio_rate_hz: welcome.audio_rate_hz,
audio_bits: welcome.audio_bits,
audio_frame_us: welcome.audio_frame_us,
codec: welcome.codec,
shard_payload: welcome.shard_payload,
host_caps: welcome.host_caps,
crates/punktfunk-core/src/client/worker.rs
+7
View File
@@ -19,6 +19,13 @@ pub(crate) struct WorkerArgs {
pub(crate) bitrate_kbps: u32,
pub(crate) video_caps: u8,
pub(crate) audio_channels: u8,
/// The sample rate/depth this client is ASKING for ([`crate::quic::Hello::audio_rate_hz`] /
/// [`audio_bits`](crate::quic::Hello::audio_bits)) — a request, never a fact. Anything other
/// than the legacy 48 kHz/16-bit pair also sets [`crate::quic::CLIENT_CAP_AUDIO_HIRES`] (see
/// [`NativeClient::connect_with_audio_format`]); the host resolves both and answers in its
/// `Welcome`, which is what the client must actually open its device from.
pub(crate) audio_rate_hz: u32,
pub(crate) audio_bits: u8,
pub(crate) video_codecs: u8,
pub(crate) preferred_codec: u8,
pub(crate) display_hdr: Option<HdrMeta>,
crates/punktfunk-core/src/lib.rs
+27 -1
View File
@@ -220,7 +220,33 @@ pub use stats::Stats;
/// and `pts_ns` of `0` — concealed audio was never on the wire and must not reach an A/V-sync
/// observation. Additive and client-local: nothing new is sent or parsed, so [`WIRE_VERSION`] is
/// unchanged.
pub const ABI_VERSION: u32 = 23;
/// v24: the lossless audio plane's client surface (`design/hi-res-audio.md` §7) —
/// `punktfunk_connect_ex11` asks for a sample rate and depth (whatever
/// `audio::pcm::rate_is_supported` admits — 48/96 kHz plus the 44.1 kHz family — and 16/24-bit;
/// the accepted rates grew after v24 shipped, which is not an ABI change: no symbol, signature or
/// struct moved, an older header stays correct, and a host that cannot carry a rate declines it to
/// Opus exactly as it always has. Anything but
/// the legacy pair also sets `CLIENT_CAP_AUDIO_HIRES`), and `punktfunk_connection_audio_sample_rate`
/// / `punktfunk_connection_audio_bits` report what the host actually RESOLVED — which may be
/// lower, because the host runs a five-condition gate and every decline lands back on Opus at
/// 48 kHz. `punktfunk_connection_next_audio_pcm` decodes both planes behind the same call, using
/// `pcm::PcmConceal` for gaps on the lossless one (libopus PLC extrapolates from a decoder's
/// model of the signal, and a raw frame has none).
///
/// ADDED, not widened, and this time the distinction has teeth: the natural place for a rate is a
/// field on `PunktfunkAudioPcm`, which is `#[repr(C)]` with no `struct_size` guard and is
/// allocated BY VALUE by every C embedder — growing it would change its layout under all of them
/// at once. `PunktfunkStats` is in the same position. So the format is read through accessors, the
/// same rule v18 set with `next_rumble_cmd2`, and `PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ` keeps its value
/// and its meaning as the DEFAULT/legacy rate — a ring sized from it stays correct for every
/// session that resolves to Opus, which is every session an ABI-23 embedder can ask for. An
/// embedder that adopts none of this behaves exactly as before.
///
/// Client-local in the C sense but NOT wire-free in the usual one: the `Hello`/`Welcome` fields
/// this reads and writes landed with the plane itself, appended behind the existing trailing-field
/// discipline (old peers skip them in both directions, and a legacy request encodes byte-identical
/// to the pre-hi-res messages), so [`WIRE_VERSION`] is still unchanged.
pub const ABI_VERSION: u32 = 24;
/// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
/// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
crates/punktfunk-core/src/quic/caps.rs
+41
View File
@@ -175,6 +175,41 @@ pub const HOST_CAP_AUDIO_RED: u8 = 0x20;
/// `0x01`/`0x02` are gamepad-state / clipboard.
pub const HOST_CAP_PAD_AUDIO: u8 = 0x40;
/// [`Hello::client_caps`] bit: the client can play the LOSSLESS audio plane
/// ([`AUDIO_PCM_MAGIC`](super::datagram::AUDIO_PCM_MAGIC), `0xD3`) at the rate and depth it asked
/// for in [`Hello::audio_rate_hz`](super::handshake::Hello::audio_rate_hz) /
/// [`audio_bits`](super::handshake::Hello::audio_bits).
///
/// **Capable AND the user turned it on** — the [`VIDEO_CAP_444`] precedent, not a bare capability.
/// This plane costs 1.5–4.6 Mbps against Opus's 256 kbps and is taken off the top of the link
/// (audio rides datagrams outside the ABR loop, so ABR can neither see it nor reclaim it), so it
/// must be asked for on both ends. A client that cannot open an output at the format it is
/// requesting must not set this bit.
///
/// `0x10` — `0x08` is [`CLIENT_CAP_PAD_AUDIO`], `0x04` is [`CLIENT_CAP_AUDIO_RED`], `0x02` is
/// [`CLIENT_CAP_PHASE_LOCK`], `0x01` is [`CLIENT_CAP_CURSOR`]. `0x20`/`0x40`/`0x80` remain free.
pub const CLIENT_CAP_AUDIO_HIRES: u8 = 0x10;
/// [`Welcome::host_caps`] bit: the host resolved the session onto the lossless audio plane
/// ([`AUDIO_PCM_MAGIC`](super::datagram::AUDIO_PCM_MAGIC), `0xD3`). Like [`HOST_CAP_AUDIO_RED`]
/// this is a statement about the WIRE rather than an offer: with the bit set the client decodes
/// `0xD3` and MUST open its device from the resolved
/// [`Welcome::audio_rate_hz`](super::handshake::Welcome::audio_rate_hz) /
/// [`audio_bits`](super::handshake::Welcome::audio_bits) /
/// [`audio_frame_us`](super::handshake::Welcome::audio_frame_us), never from what it asked for.
///
/// Unlike `0xD2`, the host does NOT drop back mid-session: the client's device is open at a fixed
/// format, so a change would mean a re-open. The plane is resolved once, at handshake, by the
/// five-condition gate in `design/hi-res-audio.md` §8.4, and every decline resolves to Opus
/// 48 kHz with a logged reason.
///
/// ⚠ `0x80` is the **LAST free `host_caps` bit**. The next host capability needs a second byte
/// and an ABI bump — the same wall [`VIDEO_CAP_MULTI_SLICE`] already hit on `video_caps`.
/// `0x40` is [`HOST_CAP_PAD_AUDIO`], `0x20` is [`HOST_CAP_AUDIO_RED`], `0x10` is
/// [`HOST_CAP_PEN`], `0x08` is [`HOST_CAP_CURSOR`], `0x04` is [`HOST_CAP_TEXT_INPUT`],
/// `0x01`/`0x02` are gamepad-state / clipboard.
pub const HOST_CAP_AUDIO_HIRES: u8 = 0x80;
/// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software**
/// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST
/// advertise this.
@@ -306,6 +341,8 @@ impl Default for ColorInfo {
#[cfg(test)]
mod tests {
use crate::audio::pcm::BITS_16;
use crate::audio::SAMPLE_RATE_HZ;
use crate::config::{CompositorPref, FecConfig, FecScheme, GamepadPref, Mode};
use crate::quic::*;
@@ -345,6 +382,10 @@ mod tests {
expires_in_secs: 0,
cipher: 0,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
};
let got = Welcome::decode(&w.encode()).unwrap();
assert_eq!(got.host_caps & HOST_CAP_CLIPBOARD, HOST_CAP_CLIPBOARD);
crates/punktfunk-core/src/quic/datagram.rs
+103 -1
View File
@@ -119,6 +119,50 @@ pub fn decode_audio_red_datagram(b: &[u8]) -> Option<(u32, u64, &[u8], Option<&[
Some((seq, pts_ns, primary, (!prev.is_empty()).then_some(prev)))
}
/// Lossless PCM audio, host → client: `[0xD3][u32 seq LE][u64 pts_ns LE][interleaved LE samples]`.
///
/// **Deliberately the same header as [`AUDIO_MAGIC`]**, so
/// [`AudioGapTracker`](crate::audio::AudioGapTracker) and the pts / A-V-sync plumbing work
/// unchanged and the only new logic on this plane is the payload format and its concealment
/// ([`crate::audio::pcm`]).
///
/// A session runs `0xC9`/`0xD2` **or** `0xD3`, never both, and never switches mid-session: the
/// client's output device is open at a fixed rate and depth, so a change means a re-open. If the
/// capture dies and comes back at a different format the host ends the audio plane rather than
/// changing tags underneath a client that cannot follow.
///
/// One frame per datagram, `audio_frame_us` long, **never fragmented** — the frame duration is
/// chosen at session start by [`crate::audio::pcm::frame_us_for`] so the payload cannot exceed
/// the path MTU. Redundancy ([`AUDIO_RED_MAGIC`]) is not defined for this plane and is never
/// sent with it: it would double a bitrate that is already the largest on the connection, and
/// `plan_audio_budget`'s ladder would never choose it.
pub const AUDIO_PCM_MAGIC: u8 = 0xD3;
/// Fixed header length of an [`AUDIO_PCM_MAGIC`] datagram — identical to the [`AUDIO_MAGIC`]
/// header by design.
pub const AUDIO_PCM_HEADER: usize = crate::audio::pcm::PCM_HEADER_LEN;
/// Encode one lossless PCM frame. `pcm` is already-quantised interleaved little-endian samples
/// at the negotiated depth ([`crate::audio::pcm::from_f32`]).
pub fn encode_audio_pcm_datagram(seq: u32, pts_ns: u64, pcm: &[u8]) -> Vec<u8> {
let mut b = Vec::with_capacity(AUDIO_PCM_HEADER + pcm.len());
b.push(AUDIO_PCM_MAGIC);
b.extend_from_slice(&seq.to_le_bytes());
b.extend_from_slice(&pts_ns.to_le_bytes());
b.extend_from_slice(pcm);
b
}
/// Parse a lossless PCM datagram → `(seq, pts_ns, payload)`. `None` on bad tag/length.
pub fn decode_audio_pcm_datagram(b: &[u8]) -> Option<(u32, u64, &[u8])> {
if b.len() < AUDIO_PCM_HEADER || b[0] != AUDIO_PCM_MAGIC {
return None;
}
let seq = u32::from_le_bytes(b[1..5].try_into().unwrap());
let pts_ns = u64::from_le_bytes(b[5..13].try_into().unwrap());
Some((seq, pts_ns, &b[AUDIO_PCM_HEADER..]))
}
/// Legacy rumble datagram (v1), host → client: `[0xCA][u16 pad LE][u16 low LE][u16 high LE]`.
/// Force-feedback state for pad `pad` (0xFFFF amplitudes, 0/0 = stop) as *level-triggered* state
/// — it persists until superseded, which is why the host re-sends it periodically as its loss
@@ -1132,7 +1176,60 @@ mod tests {
assert!(decode_audio_red_datagram(&wrong).is_none());
}
/// The two audio planes must not alias each other or any neighbouring plane: a client
/// The lossless plane round-trips, keeps the `0xC9` header shape so the gap tracker and A/V
/// sync need no new code, and refuses a foreign tag.
#[test]
fn audio_pcm_datagram_roundtrip() {
let payload: Vec<u8> = (0..1152u32).map(|i| (i % 251) as u8).collect();
let d = encode_audio_pcm_datagram(7, 1_234_567_890, &payload);
assert_eq!(d[0], AUDIO_PCM_MAGIC);
assert_eq!(d.len(), AUDIO_PCM_HEADER + payload.len());
// Same header shape as 0xC9 — seq and pts sit at the same offsets, which is what lets
// the gap tracker and the pts plumbing work unchanged.
assert_eq!(AUDIO_PCM_HEADER, 13);
let (seq, pts, out) = decode_audio_pcm_datagram(&d).expect("decode");
assert_eq!((seq, pts), (7, 1_234_567_890));
assert_eq!(out, &payload[..]);
// An empty payload is structurally legal (nothing to say), and a short buffer is not.
assert!(decode_audio_pcm_datagram(&encode_audio_pcm_datagram(1, 2, &[])).is_some());
assert!(decode_audio_pcm_datagram(&d[..AUDIO_PCM_HEADER - 1]).is_none());
let mut wrong = d.clone();
wrong[0] = AUDIO_MAGIC;
assert!(decode_audio_pcm_datagram(&wrong).is_none());
}
/// The whole point of the frame ladder: whatever it picks must survive the encoder and land
/// inside the datagram budget it was given. Cheap to state, and it is the invariant that
/// keeps this plane off any fragmentation path.
#[test]
fn a_ladder_sized_frame_fits_the_datagram_it_was_sized_for() {
use crate::audio::pcm;
// Both rate families — the 44.1 kHz one carries a FRACTIONAL number of samples in most
// rungs, so its frame is the floor and the fit has margin rather than being eroded.
for rate in [44_100u32, 48_000, 88_200, 96_000, 176_400] {
for bits in [pcm::BITS_16, pcm::BITS_24] {
for budget in [900usize, 1200, 1400] {
let Some(us) = pcm::frame_us_for(rate, bits, 2, budget) else {
// 176 400/24-bit needs 1 069 B for even a 1 ms frame; a budget that small
// declines the plane outright, exactly as it declines hi-res surround.
continue;
};
let samples = pcm::samples_per_frame(rate, us, 2);
let mut wire = Vec::new();
pcm::from_f32(&vec![0.25f32; samples], bits, &mut wire);
let d = encode_audio_pcm_datagram(0, 0, &wire);
assert!(
d.len() <= budget,
"{rate}/{bits} at {budget} B produced a {} B datagram",
d.len()
);
}
}
}
}
/// The three audio planes must not alias each other or any neighbouring plane: a client
/// demultiplexes purely on the first byte.
#[test]
fn audio_red_tag_is_disjoint() {
@@ -1145,6 +1242,7 @@ mod tests {
HDR_META_MAGIC,
HOST_TIMING_MAGIC,
CURSOR_STATE_MAGIC,
AUDIO_PCM_MAGIC,
crate::input::INPUT_MAGIC,
] {
assert_ne!(AUDIO_RED_MAGIC, other);
@@ -1155,6 +1253,10 @@ mod tests {
"0xC9 must not accept a 0xD2"
);
let plain = encode_audio_datagram(1, 2, &[9u8; 40]);
assert!(
decode_audio_pcm_datagram(&plain).is_none(),
"0xD3 must not accept a 0xC9"
);
assert!(
decode_audio_red_datagram(&plain).is_none(),
"0xD2 must not accept a 0xC9"
crates/punktfunk-core/src/quic/endpoint.rs
+22
View File
@@ -46,6 +46,28 @@ fn stream_transport_idle(idle: std::time::Duration) -> Arc<quinn::TransportConfi
// standing delay that never drains while video stays live. Capping the buffer makes the
// plane latest-wins at the source — ~200 ms of stereo Opus (proportionally less at
// surround bitrates), so sustained congestion costs concealable drops, never lag.
//
// THE LOSSLESS PLANE SHARES THIS BUFFER, and its frames are ~10× larger: a 48 kHz/24-bit
// stereo frame is 1 152 B against Opus's ~100 B, so 4 KiB is ~3.5 PCM frames (~14 ms) where
// it is ~40 Opus frames (~200 ms). That is deliberately NOT raised, for two reasons.
//
// (a) It stays faithful to the rule above. Fewer frames of slack is still "shed oldest under
// congestion", and one shed PCM frame is exactly what `audio::pcm::PcmConceal` exists to
// cover. Trading it for lag would invert the property this cap was introduced to get.
// (b) A blanket raise would REGRESS the Opus plane. Sizing for six PCM frames (~16 KiB) would
// make an Opus session's worst-case backlog ~800 ms — four times what this line was
// written to prevent — and the two planes cannot be sized separately here, because
// `TransportConfig` is built before the handshake resolves which plane the session runs.
// Sizing it per-session means building the transport config after negotiation, which is a
// larger change than the hi-res programme needed.
//
// ⚠ AND THE EVICTION IS INVISIBLE. `Connection::send_datagram` calls `send(data, drop=true)`,
// which silently evicts the oldest queued datagrams at the cap and returns `Ok(())` — "buffer
// full" is not one of `SendDatagramError`'s four variants (`UnsupportedByPeer`, `Disabled`,
// `TooLarge`, `ConnectionLost`). So no caller can count these drops today; the host's audio
// egress counters cover `TooLarge` only. Observing them needs `datagram_send_buffer_space()`
// before the send, or `send_datagram_wait`. Worth knowing before anyone reads a clean drop
// counter as proof the plane is not shedding.
t.datagram_send_buffer_size(4 * 1024);
// MTU discovery probes up to EXACTLY the sealed size of a full IPv4 video datagram (1472)
// instead of quinn's stock 1452. Two reasons: (a) on a clean 1500-MTU path QUIC gets the
crates/punktfunk-core/src/quic/handshake.rs
+706 -44
View File
@@ -90,8 +90,9 @@ pub struct Hello {
/// disambiguated by REMAINING LENGTH at decode: fewer than `HDR_META_BODY_LEN` bytes after
/// `preferred_codec` ⇒ no HDR block, the tail bytes are the post-HDR fields directly. This
/// caps everything after `display_hdr` at `HDR_META_BODY_LEN − 1` bytes total — document any
/// future field here and mind the budget (`client_caps` 1 + `max_shard_payload` 2 = 3 of the
/// 27 spent). Omitted when zero and by older clients (→ `0`).
/// future field here and mind the budget (`client_caps` 1 + `max_shard_payload` 2 +
/// `audio_rate_hz` 4 + `audio_bits` 1 = **8 of the 27 spent, 19 free**). Omitted when zero
/// and by older clients (→ `0`).
pub client_caps: u8,
/// The largest video shard payload this client's receive path accepts — sealed datagrams for
/// shards up to this size fit its transport buffers ([`crate::config::max_shard_payload`]).
@@ -103,6 +104,37 @@ pub struct Hello {
/// = legacy: the host must not change the sealed geometry mid-session, and never above
/// the `Welcome` value).
pub max_shard_payload: u16,
/// The sample rate this client is **asking** the host to capture and send at — `48_000` (the
/// legacy rate every build speaks) or any other rate
/// [`pcm::rate_is_supported`](crate::audio::pcm::rate_is_supported) admits: `96_000`, and the
/// 44.1 kHz family `44_100` / `88_200` / `176_400` (`design/hi-res-audio.md` §3/§4.1 — the
/// family was deferred only until [`JitterPolicy`](crate::audio::JitterPolicy) stopped
/// dividing by 1 000 before it multiplied, which it no longer does).
///
/// A request, never a fact. The host resolves it against what its capture path can *genuinely*
/// deliver — never by padding an upsampled stream, which is the trap §4.3 exists to prevent —
/// and states the resolved value in [`Welcome::audio_rate_hz`]. **The client must open its
/// output device from the `Welcome`, not from this.** Meaningless unless the client also set
/// [`CLIENT_CAP_AUDIO_HIRES`]: the bit is the opt-in, this is only the parameter.
///
/// Appended after `max_shard_payload` as 4 trailing LE bytes (forcing every earlier
/// placeholder). `0` on the wire and absence both decode to the legacy
/// [`SAMPLE_RATE_HZ`](crate::audio::SAMPLE_RATE_HZ), so an older client always reads as a
/// plain 48 kHz request and this field costs the legacy wire form nothing.
pub audio_rate_hz: u32,
/// The sample depth this client is **asking** for — [`BITS_16`](crate::audio::pcm::BITS_16)
/// or [`BITS_24`](crate::audio::pcm::BITS_24). Same request-not-fact discipline as
/// [`audio_rate_hz`](Self::audio_rate_hz): the host resolves it and answers in
/// [`Welcome::audio_bits`], which is what the client's device must be opened from.
///
/// 24-bit is the depth the feature exists for — it is also the one a lossless coder saves
/// least on, which is why the plane carries raw PCM rather than a codec
/// (`crate::audio::pcm`). Appended after `audio_rate_hz` as a single trailing byte; `0` and
/// absence both decode to [`BITS_16`](crate::audio::pcm::BITS_16). This is the LAST field of
/// the whole message, so it forces every earlier placeholder (including a 4-byte
/// `audio_rate_hz` written as the legacy 48 000) while nothing can force it: a 96 kHz/16-bit
/// request emits the rate and stops there.
pub audio_bits: u8,
}
/// QUIC application error code a punktfunk/1 client closes the control connection with on a
@@ -135,6 +167,35 @@ pub const CIPHER_AES_128_GCM: u8 = 0;
/// [`VIDEO_CAP_CHACHA20`] for clients without hardware AES.
pub const CIPHER_CHACHA20_POLY1305: u8 = 1;
/// [`Welcome::audio_codec`] id: **Opus on the `0xC9` plane** — the legacy default, 48 kHz, what
/// every pre-hi-res build sends and what every declined hi-res negotiation resolves back to
/// (`design/hi-res-audio.md` §8.4: a fallback to today's transparent 256 kbps Opus is not a
/// defeat; silence is the one unacceptable outcome). `0`, so an absent field and an older host
/// both read as Opus and the common Welcome stays byte-identical to the pre-hi-res wire form.
pub const AUDIO_CODEC_OPUS: u8 = 0;
/// [`Welcome::audio_codec`] id **reserved for FLAC** on the `0xD3` plane — deliberately reserved
/// and deliberately **unimplemented**.
///
/// The design doc numbers the audio codecs `0` = Opus, `1` = FLAC, `2` = PCM, and this project has
/// been bitten before by wire ids that drifted from the document that specifies them. The id is
/// therefore burned rather than compacted: `AUDIO_CODEC_PCM` is `2` because the doc says `2`.
///
/// FLAC lost on the merits, and the reasoning is recorded in `crate::audio::pcm`'s module docs so
/// it does not have to be re-derived: the plane is never fragmented, so a frame must be sized from
/// the codec's WORST case (a FLAC VERBATIM subframe — raw samples plus a header), which means FLAC
/// and PCM negotiate the same frame duration, the same packet rate and the same send-buffer
/// sizing. A codec would buy average bytes on a plane that is provisioned for peak, at the cost of
/// a new dependency in the NDK / xcframework / flatpak / MSIX / Arch packaging targets. No host
/// emits this id and no client should accept it; it exists so that a future one could.
pub const AUDIO_CODEC_FLAC_RESERVED: u8 = 1;
/// [`Welcome::audio_codec`] id: **raw interleaved LE PCM on the `0xD3` plane** (`crate::audio::pcm`)
/// — the lossless format this negotiation exists to reach, at the resolved
/// [`Welcome::audio_rate_hz`] / [`audio_bits`](Welcome::audio_bits) /
/// [`audio_frame_us`](Welcome::audio_frame_us).
///
/// `2` rather than `1` because [`AUDIO_CODEC_FLAC_RESERVED`] holds `1` — see there.
pub const AUDIO_CODEC_PCM: u8 = 2;
/// `host → client`: the complete session offer.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Welcome {
@@ -247,6 +308,71 @@ pub struct Welcome {
/// ever observes an all-zero key. Decode rejects `cipher == 1` with fewer than 32 key
/// bytes following.
pub key_chacha: Option<[u8; 32]>,
/// Which audio plane this session runs — [`AUDIO_CODEC_OPUS`] (`0`, the default: Opus at
/// 48 kHz on `0xC9`, with `0xD2` redundancy when negotiated) or [`AUDIO_CODEC_PCM`] (`2`:
/// lossless PCM on [`AUDIO_PCM_MAGIC`](super::datagram::AUDIO_PCM_MAGIC), `0xD3`).
/// [`AUDIO_CODEC_FLAC_RESERVED`] (`1`) is reserved and never emitted.
///
/// A statement about the wire, resolved once at handshake by the five-condition gate in
/// `design/hi-res-audio.md` §8.4, and paired with [`HOST_CAP_AUDIO_HIRES`] in `host_caps`.
/// A session runs `0xC9`/`0xD2` **or** `0xD3`, never both, and the host does not switch
/// mid-session — the client's output device is open at a fixed format, so a change would mean
/// a re-open. Every decline resolves back to Opus 48 kHz and is logged with its reason.
///
/// This is also the field that decides whether the whole audio tail is on the wire at all:
/// anything other than `AUDIO_CODEC_OPUS` forces the four audio fields AND every earlier
/// placeholder (`cipher`, `mgmt_port`, `grants`, `expires_in_secs`) to be emitted, so a plain
/// Opus session's Welcome stays **byte-identical** to the pre-hi-res wire form. See
/// [`Welcome::encode`] — this is the same chain `mgmt_port` and the access advert already
/// extend, one link further, and the reason it must hold is unchanged: a field that slips
/// into offset 68 is read as `cipher` by every shipped client, whose decode is fail-closed.
///
/// Appended after `expires_in_secs`, i.e. at offset **79** in an AES session and **111**
/// behind a ChaCha key. Absent (an older host) → [`AUDIO_CODEC_OPUS`].
pub audio_codec: u8,
/// The **resolved** capture rate — what the host is actually reading off its endpoint, not
/// what [`Hello::audio_rate_hz`] asked for. May be lower than requested; that is the whole
/// point of "the client asks, the host obliges if it can".
///
/// ⚠ **The client opens its output device from THIS, never from its own request.** The
/// failure this prevents is the one `design/hi-res-audio.md` §4.3 is written around: WASAPI's
/// `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM` will happily accept a 96 kHz request against a 48 kHz
/// engine and hand back interpolated samples with no error — a session that spends 3.2 Mbps,
/// reports 96 kHz everywhere, and carries nothing above 24 kHz. The host is required to read
/// its engine's true rate and decline rather than pad, and this field is where that honesty
/// surfaces. A client whose device then opens at some *other* rate must say so and fall back
/// rather than resample quietly (§9) — the rule is the same at both ends: never claim a rate
/// you did not get.
///
/// 4 LE bytes after `audio_codec` (80..84, or 112..116 behind a ChaCha key). Absent, or `0`
/// on the wire, → the legacy [`SAMPLE_RATE_HZ`](crate::audio::SAMPLE_RATE_HZ).
pub audio_rate_hz: u32,
/// The **resolved** sample depth — [`BITS_16`](crate::audio::pcm::BITS_16) or
/// [`BITS_24`](crate::audio::pcm::BITS_24). Same authority as
/// [`audio_rate_hz`](Self::audio_rate_hz): it is what the client unpacks `0xD3` payloads at,
/// and getting it wrong desynchronises every sample after the first (a 24-bit payload read at
/// 2 bytes per sample is not silence, it is noise).
///
/// One byte at 84 (or 116). Absent, `0`, or a depth this plane cannot carry (checked with
/// [`depth_is_supported`](crate::audio::pcm::depth_is_supported)) → `BITS_16` — the
/// `audio_channels` precedent, so a corrupt byte can never build a decoder at a stride the
/// plane does not speak. Only a corrupt or future wire reaches that branch: the Welcome rides
/// the pinned-TLS control stream, and the host resolves this from a two-value set.
pub audio_bits: u8,
/// The **resolved** frame duration in microseconds — how much audio one `0xD3` datagram
/// carries. `0` for an Opus session (whose frame length is the fixed 5 ms of the `0xC9`
/// plane), and one rung of [`FRAME_US_LADDER`](crate::audio::pcm::FRAME_US_LADDER) otherwise.
///
/// ⚠ **Negotiated, never assumed — do not hardcode it.** The host computes it with
/// [`frame_us_for`](crate::audio::pcm::frame_us_for) from `(rate, depth, channels,
/// conn.max_datagram_size())`, because a datagram over the path MTU is not sent *at all* and
/// this plane is never fragmented. At 96 kHz/24-bit the default 1472-byte MTU ceiling only
/// leaves room for 2 ms frames, and the host must not ask before QUIC MTU discovery has
/// settled or it sizes the whole session against the conservative initial value
/// (`design/hi-res-audio.md` §4.2).
///
/// 2 LE bytes at 85..87 (or 117..119) — the last field of the message. Absent → `0`.
pub audio_frame_us: u16,
}
/// `client → host`: data plane is bound, begin streaming.
@@ -296,13 +422,23 @@ impl Hello {
let hdr_present = self.display_hdr.is_some();
let ccaps_present = self.client_caps != 0;
let msp_present = self.max_shard_payload != 0;
// The hi-res audio request (design/hi-res-audio.md §7). BOTH the wire's `0` and the legacy
// value count as "default": decode resolves an absent field to 48 000 / 16 bits, so a
// struct carrying those explicitly must produce the same bytes as one left at zero —
// otherwise "the legacy request" would have two wire forms, only one of them byte-identical
// to the pre-hi-res Hello, and which one you got would depend on how the caller spelled it.
let arate_present =
self.audio_rate_hz != 0 && self.audio_rate_hz != crate::audio::SAMPLE_RATE_HZ;
let abits_present = self.audio_bits != 0 && self.audio_bits != crate::audio::pcm::BITS_16;
let audio_present = arate_present || abits_present;
let need_placeholders = self.video_caps != 0
|| ac_present
|| vcodecs_present
|| pref_present
|| hdr_present
|| ccaps_present
|| msp_present;
|| msp_present
|| audio_present;
match (&self.name, &self.launch) {
(None, None) if !need_placeholders => {}
(name, _) => {
@@ -329,15 +465,22 @@ impl Hello {
|| hdr_present
|| ccaps_present
|| msp_present
|| audio_present
{
b.push(self.audio_channels);
}
// video_codecs: emitted when non-zero OR a later field follows.
if vcodecs_present || pref_present || hdr_present || ccaps_present || msp_present {
if vcodecs_present
|| pref_present
|| hdr_present
|| ccaps_present
|| msp_present
|| audio_present
{
b.push(self.video_codecs);
}
// preferred_codec: emitted when non-zero OR a later field follows.
if pref_present || hdr_present || ccaps_present || msp_present {
if pref_present || hdr_present || ccaps_present || msp_present || audio_present {
b.push(self.preferred_codec);
}
// display_hdr: fixed HDR_META_BODY_LEN-byte HdrMeta body; omitted when `None` even if
@@ -348,13 +491,31 @@ impl Hello {
}
// client_caps: single byte after the (optional) HDR block. Emitted when non-zero OR a
// later field follows.
if ccaps_present || msp_present {
if ccaps_present || msp_present || audio_present {
b.push(self.client_caps);
}
// max_shard_payload: 2 trailing LE bytes after client_caps. Emitted when non-zero.
if msp_present {
// max_shard_payload: 2 trailing LE bytes after client_caps. Emitted when non-zero OR a
// later field follows.
if msp_present || audio_present {
b.extend_from_slice(&self.max_shard_payload.to_le_bytes());
}
// audio_rate_hz: 4 trailing LE bytes. Emitted when non-default OR `audio_bits` follows —
// in which case it goes out as the legacy 48 000 rather than as `self.audio_rate_hz`,
// because a caller that left the rate at the struct's `0` still means "48 kHz" and the
// byte a decoder reads must say the same thing the struct does.
if audio_present {
let rate = if arate_present {
self.audio_rate_hz
} else {
crate::audio::SAMPLE_RATE_HZ
};
b.extend_from_slice(&rate.to_le_bytes());
}
// audio_bits: the last byte of the message, so nothing can force it — emitted only when
// the depth itself is non-default (a 96 kHz/16-bit request stops after the rate).
if abits_present {
b.push(self.audio_bits);
}
b
}
@@ -371,6 +532,20 @@ impl Hello {
let launch_off = 27 + name_len; // launch's length byte
let launch_len = b.get(launch_off).copied().unwrap_or(0) as usize;
let tail = launch_off + 1 + launch_len; // first trailing byte: video_caps
// Where the post-HDR tail starts. `display_hdr` is a FIXED HDR_META_BODY_LEN-byte block
// with no placeholder form, so its presence is decided by REMAINING LENGTH: ≥ that many
// bytes after `preferred_codec` ⇒ the block is there and the post-HDR fields follow it;
// fewer ⇒ there is no block and those bytes ARE the post-HDR fields. Sound only while the
// whole post-HDR tail stays under HDR_META_BODY_LEN bytes — the budget documented on
// [`Hello::client_caps`] (8 of 27 spent). Computed ONCE: every field from `client_caps` on
// reads off it, and four copies of this predicate would be four chances to update three.
let has_hdr = b.len().saturating_sub(tail + 4) >= super::datagram::HDR_META_BODY_LEN;
let post_hdr = if has_hdr {
tail + 4 + super::datagram::HDR_META_BODY_LEN
} else {
tail + 4
};
Ok(Hello {
abi_version: u32at(4),
mode: Mode {
@@ -419,39 +594,46 @@ impl Hello {
// `0` = no preference; the host decides by precedence.
preferred_codec: b.get(tail + 3).copied().unwrap_or(0),
// Optional trailing HdrMeta body (fixed length) — absent on an older client / a
// client without an HDR display → `None` (the host keeps its EDID defaults).
// Presence is decided by REMAINING LENGTH (there is no placeholder form for the
// fixed block): ≥ HDR_META_BODY_LEN bytes after `preferred_codec` ⇒ the block is
// there and post-HDR fields follow it; fewer ⇒ no block, the bytes ARE the post-HDR
// fields. Sound as long as the post-HDR tail stays under HDR_META_BODY_LEN bytes.
display_hdr: (b.len().saturating_sub(tail + 4) >= super::datagram::HDR_META_BODY_LEN)
// client without an HDR display → `None` (the host keeps its EDID defaults). See
// `has_hdr` above for why presence is a length question rather than a flag.
display_hdr: has_hdr
.then(|| {
b.get(tail + 4..tail + 4 + super::datagram::HDR_META_BODY_LEN)
.map(super::datagram::read_hdr_meta_body)
})
.flatten(),
// client_caps: the byte after the HDR block when present, else directly at tail+4.
client_caps: {
let off = if b.len().saturating_sub(tail + 4) >= super::datagram::HDR_META_BODY_LEN
{
tail + 4 + super::datagram::HDR_META_BODY_LEN
} else {
tail + 4
};
b.get(off).copied().unwrap_or(0)
},
// max_shard_payload: 2 LE bytes after client_caps (same post-HDR offset rule).
client_caps: b.get(post_hdr).copied().unwrap_or(0),
// max_shard_payload: 2 LE bytes after client_caps.
// Absent on an older client → 0 = no mid-session renegotiation, no jumbo.
max_shard_payload: {
let off = if b.len().saturating_sub(tail + 4) >= super::datagram::HDR_META_BODY_LEN
{
tail + 4 + super::datagram::HDR_META_BODY_LEN
} else {
tail + 4
};
b.get(off + 1..off + 3)
.map(|s| u16::from_le_bytes(s.try_into().unwrap()))
.unwrap_or(0)
max_shard_payload: b
.get(post_hdr + 1..post_hdr + 3)
.map(|s| u16::from_le_bytes(s.try_into().unwrap()))
.unwrap_or(0),
// The hi-res audio request: 4 LE rate bytes then the depth byte. Absent (an older
// client) or an explicit `0` → the legacy 48 kHz / 16-bit, which is what a client
// that never heard of this feature is asking for. Resolved to the SEMANTIC default
// rather than left as a raw 0 so nothing downstream has to remember that "0 means
// 48 000" — the host's gate and the client's device-open both read a real rate.
//
// Beyond zero the rate is NOT range-checked: the ladder is 48/96 kHz today and grows
// once JitterPolicy's integer samples-per-ms arithmetic is reworked, and quietly
// rewriting a rate we don't recognise into one we do is the "label right, content
// wrong" lie this whole feature is built to avoid. The HOST decides what it can
// honour, and answers in `Welcome::audio_rate_hz`.
audio_rate_hz: b
.get(post_hdr + 3..post_hdr + 7)
.map(|s| u32::from_le_bytes(s.try_into().unwrap()))
.filter(|&hz| hz != 0)
.unwrap_or(crate::audio::SAMPLE_RATE_HZ),
// The depth IS checked, because it is a byte stride rather than a label: a value the
// plane cannot carry would size a `0xD3` unpack wrongly. It can only come off a
// corrupt or future wire (this message rides the pinned-TLS control stream), and a
// request is only a request — falling back to 16 costs a hi-res session, never
// correctness.
audio_bits: match b.get(post_hdr + 7).copied() {
Some(d) if crate::audio::pcm::depth_is_supported(d) => d,
_ => crate::audio::pcm::BITS_16,
},
})
}
@@ -518,20 +700,34 @@ impl Welcome {
// u32s always land at a deterministic offset. A full-control permanent session
// (`GRANT_ALL`, no deadline) is what every absent-field decode yields anyway, so it is
// omitted and the common case stays byte-identical to the pre-grants wire form.
// The hi-res audio block (codec + resolved rate/depth/frame) extends the chain one link
// further still, and its presence test is the CODEC alone: "this session is not plain
// legacy Opus". Deliberately not "any of the four differs from its default" — a hi-res
// session at 48 kHz/16-bit would then be indistinguishable on the wire from an Opus one,
// and the client would open the wrong plane. Conversely an Opus session never emits the
// block whatever the other three say, so today's Welcome stays byte-identical — the
// interop guarantee the cipher byte bought and every link since has had to keep.
let mgmt_present = self.mgmt_port != 0;
let access_present = self.grants != super::access::GRANT_ALL || self.expires_in_secs != 0;
if self.cipher != CIPHER_AES_128_GCM || mgmt_present || access_present {
let audio_present = self.audio_codec != AUDIO_CODEC_OPUS;
if self.cipher != CIPHER_AES_128_GCM || mgmt_present || access_present || audio_present {
b.push(self.cipher);
if let Some(k) = &self.key_chacha {
b.extend_from_slice(k);
}
if mgmt_present || access_present {
if mgmt_present || access_present || audio_present {
b.extend_from_slice(&self.mgmt_port.to_le_bytes());
}
if access_present {
if access_present || audio_present {
b.extend_from_slice(&self.grants.to_le_bytes());
b.extend_from_slice(&self.expires_in_secs.to_le_bytes());
}
if audio_present {
b.push(self.audio_codec);
b.extend_from_slice(&self.audio_rate_hz.to_le_bytes());
b.push(self.audio_bits);
b.extend_from_slice(&self.audio_frame_us.to_le_bytes());
}
}
b
}
@@ -543,13 +739,16 @@ impl Welcome {
// bit_depth[59] color.primaries[60] color.transfer[61] color.matrix[62] color.range[63]
// chroma_format[64] audio_channels[65] codec[66] host_caps[67] cipher[68]
// key_chacha[69..101] mgmt_port[69..71 | 101..103] grants[71..75 | 103..107]
// expires_in_secs[75..79 | 107..111] (everything from compositor on is an
// optional trailing byte; an older host stops earlier; cipher/key_chacha are present only
// when ChaCha was negotiated). `mgmt_port` and the access pair are the fields whose
// offsets are NOT fixed: they follow the cipher block, shifted by 32 when a ChaCha key
// precedes them. Emitting a later field forces every earlier one (see `encode`), so
// "cipher absent" and "mgmt_port present" — or "mgmt_port absent" and "grants present" —
// can never both hold.
// expires_in_secs[75..79 | 107..111] audio_codec[79 | 111] audio_rate_hz[80..84 | 112..116]
// audio_bits[84 | 116] audio_frame_us[85..87 | 117..119] (everything from compositor on is
// an optional trailing byte; an older host stops earlier; cipher/key_chacha are present
// only when ChaCha was negotiated). `mgmt_port`, the access pair and the audio block are
// the fields whose offsets are NOT fixed: they follow the cipher block, shifted by 32 when
// a ChaCha key precedes them — which is why every one of them is read off `mgmt_off`
// rather than a constant, and why the tests cover BOTH ciphers. Emitting a later field
// forces every earlier one (see `encode`), so "cipher absent" and "mgmt_port present" —
// or "grants absent" and "audio_codec present" — can never both hold. Full length: 87
// bytes for a hi-res AES session, 119 behind a ChaCha key.
if b.len() < 53 || &b[0..4] != MAGIC {
return Err(PunktfunkError::InvalidArg("bad Welcome"));
}
@@ -601,6 +800,40 @@ impl Welcome {
.get(grants_off + 4..grants_off + 8)
.map(|s| u32::from_le_bytes(s.try_into().unwrap()))
.unwrap_or(0);
// The audio block trails the access advert — 79 for AES, 111 behind a ChaCha key. Absent
// (an older host, or any session the §8.4 gate resolved back to Opus, which encode omits)
// → Opus at the legacy 48 kHz / 16-bit, frame length not applicable. That is exactly what
// every pre-hi-res host meant, so nothing has to distinguish "old host" from "declined".
let audio_off = grants_off + 8;
// The codec id is taken VERBATIM, unlike the rate and depth below. A value we don't know
// is not something to fold onto a default: `audio_codec` is what selects the plane, and
// silently reporting Opus while the host ships 0xD3 would be silence — the one outcome
// §8.4 calls unacceptable. The client cross-checks this against HOST_CAP_AUDIO_HIRES and
// refuses a plane it cannot play, which is a decision it must make anyway and can make
// loudly. (`1` = FLAC is reserved and emitted by nothing; see AUDIO_CODEC_FLAC_RESERVED.)
let audio_codec = b.get(audio_off).copied().unwrap_or(AUDIO_CODEC_OPUS);
// The resolved rate. Not clamped to the rates we know: the host states what it actually
// opened, and a decoder that "corrected" 44 100 to 48 000 would be manufacturing precisely
// the label-right/content-wrong lie §4.3 exists to prevent. Only `0`/absent is resolved,
// to the legacy rate.
let audio_rate_hz = b
.get(audio_off + 1..audio_off + 5)
.map(|s| u32::from_le_bytes(s.try_into().unwrap()))
.filter(|&hz| hz != 0)
.unwrap_or(crate::audio::SAMPLE_RATE_HZ);
// The resolved depth IS range-checked, because unlike the rate it feeds an unpack stride:
// a depth the plane cannot carry would have the client walking a `0xD3` payload at the
// wrong step and reading noise, so it folds to 16 — the `audio_channels` precedent. Only a
// corrupt or future wire reaches that branch (this rides the pinned-TLS control stream).
let audio_bits = match b.get(audio_off + 5).copied() {
Some(d) if crate::audio::pcm::depth_is_supported(d) => d,
_ => crate::audio::pcm::BITS_16,
};
// The resolved frame duration. `0` = not applicable (an Opus session, or an older host).
let audio_frame_us = b
.get(audio_off + 6..audio_off + 8)
.map(|s| u16::from_le_bytes(s.try_into().unwrap()))
.unwrap_or(0);
Ok(Welcome {
abi_version: u32at(4),
udp_port: u16at(8),
@@ -673,6 +906,10 @@ impl Welcome {
expires_in_secs,
cipher,
key_chacha,
audio_codec,
audio_rate_hz,
audio_bits,
audio_frame_us,
})
}
@@ -725,6 +962,8 @@ impl Start {
#[cfg(test)]
mod tests {
use crate::audio::pcm::{depth_is_supported, frame_us_for, BITS_16, BITS_24};
use crate::audio::SAMPLE_RATE_HZ;
use crate::config::{CompositorPref, FecConfig, FecScheme, GamepadPref, Mode, Role};
use crate::quic::*;
@@ -762,6 +1001,10 @@ mod tests {
expires_in_secs: 0,
cipher: 0,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
};
assert_eq!(Welcome::decode(&w.encode()).unwrap(), w);
@@ -830,6 +1073,10 @@ mod tests {
expires_in_secs: 0,
cipher: CIPHER_AES_128_GCM,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
};
// An AES session's Welcome is byte-identical to the pre-cipher wire form (68 bytes) —
// the old-client × new-host interop guarantee.
@@ -1083,6 +1330,10 @@ mod tests {
expires_in_secs: 0,
cipher: 0,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
}
.encode(),
)
@@ -1113,6 +1364,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
let enc = h.encode();
let dec = Hello::decode(&enc).unwrap();
@@ -1160,6 +1413,10 @@ mod tests {
expires_in_secs: 0,
cipher: 0,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
}
.encode(),
)
@@ -1194,6 +1451,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
assert_eq!(Hello::decode(&h.encode()).unwrap(), h);
let s = Start {
@@ -1226,6 +1485,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
let enc = h.encode();
assert_eq!(enc.len(), 26);
@@ -1274,6 +1535,10 @@ mod tests {
expires_in_secs: 0,
cipher: 0,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
};
let wenc = w.encode();
assert_eq!(wenc.len(), 68); // 60 base + 4 colour + chroma + audio-channels + codec + host-caps
@@ -1348,6 +1613,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
let enc = base.encode();
assert_eq!(
@@ -1401,6 +1668,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
// launch alone (no name): a zero-length name placeholder keeps the offset deterministic.
let with_launch = Hello {
@@ -1462,6 +1731,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
// A real client-panel volume (P3 primaries, 800-nit peak, 0.05-nit floor, 400-nit FALL).
let vol = HdrMeta {
@@ -1531,6 +1802,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
}
.encode();
assert!(PairRequest::decode(&h).is_err(), "abi {abi} parsed as pair");
@@ -1565,6 +1838,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
let vol = HdrMeta {
display_primaries: [[13250, 34500], [7500, 3000], [34000, 16000]],
@@ -1635,6 +1910,8 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
// The advertisement alone: every earlier trailing field is emitted as a placeholder
// so the 2 LE bytes land at a deterministic offset — and the whole thing roundtrips.
@@ -1674,4 +1951,389 @@ mod tests {
}
);
}
/// The audio codec ids are a registry the design doc owns, not a compact enum. Pinned
/// because `1` is deliberately burned on an unimplemented format: if someone later
/// "tidies" PCM down to `1`, every host and client that already shipped `2` reads the
/// wrong plane off the wire, and the failure is silence rather than an error.
#[test]
fn audio_codec_ids_match_the_design_doc_numbering() {
assert_eq!(AUDIO_CODEC_OPUS, 0);
assert_eq!(AUDIO_CODEC_FLAC_RESERVED, 1);
assert_eq!(AUDIO_CODEC_PCM, 2);
// Opus MUST be the zero id — the whole byte-identical-legacy-wire property rests on an
// absent field decoding to it.
assert_eq!(AUDIO_CODEC_OPUS, 0, "absence decodes to Opus");
}
/// The hi-res audio block on `Welcome` (`design/hi-res-audio.md` §4.7, §7).
///
/// ⚠ THE HAZARD THIS PINS: `Welcome`'s tail is CONDITIONAL. `cipher` sits at 68 and the
/// 32-byte ChaCha key at 69..101 are emitted only when ChaCha was negotiated, so every field
/// appended after them lands at two different offsets — 79 under AES, 111 behind a key. A
/// decoder that computed either from a fixed constant, or a test that only covered AES, ships
/// a working stream to everyone except the soft-AES clients (webOS) that asked for ChaCha
/// precisely because they are the slowest devices in the fleet. §4.7 calls this out by name;
/// this test is the answer.
#[test]
fn welcome_hires_audio_wire_under_both_ciphers() {
let base = Welcome {
abi_version: 2,
udp_port: 7000,
mode: Mode {
width: 1920,
height: 1080,
refresh_hz: 60,
},
fec: FecConfig {
scheme: FecScheme::Gf16,
fec_percent: 20,
max_data_per_block: 4096,
},
shard_payload: 1200,
encrypt: true,
key: [7u8; 16],
salt: [9, 8, 7, 6],
frames: 0,
compositor: CompositorPref::Auto,
gamepad: GamepadPref::Auto,
bitrate_kbps: 50_000,
bit_depth: 8,
color: ColorInfo::SDR_BT709,
chroma_format: CHROMA_IDC_420,
audio_channels: 2,
codec: CODEC_HEVC,
host_caps: 0,
mgmt_port: 0,
grants: GRANT_ALL,
expires_in_secs: 0,
cipher: CIPHER_AES_128_GCM,
key_chacha: None,
audio_codec: AUDIO_CODEC_OPUS,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
audio_frame_us: 0,
};
// ── The interop floor: a plain Opus session is byte-identical to the pre-hi-res wire ──
//
// Everything below only holds because the common case pays nothing. 68 bytes is the
// pre-cipher length, and every field appended since (cipher, mgmt_port, the access
// advert, and now these four) has had to keep it.
assert_eq!(base.encode().len(), 68);
assert_eq!(Welcome::decode(&base.encode()).unwrap(), base);
// …and presence is decided by the CODEC ALONE. A rate/depth that differ from the legacy
// values do NOT put the block on the wire: an Opus session has no other format to be in,
// and if the other three fields could force the block, an ordinary Opus Welcome would
// stop being byte-identical the moment anything set them speculatively.
let opus_with_stray_format = Welcome {
audio_rate_hz: 96_000,
audio_bits: BITS_24,
audio_frame_us: 2000,
..base
};
assert_eq!(
opus_with_stray_format.encode().len(),
68,
"only audio_codec puts the block on the wire"
);
// ── The resolved hi-res session ───────────────────────────────────────────────────────
//
// The frame duration comes from the ladder, never a constant: at 96 kHz/24-bit stereo a
// ~1400 B usable datagram only carries 2 ms, and this plane is never fragmented, so a
// hardcoded 2.5 ms would produce datagrams the path silently refuses to send (§4.2).
let frame_us = frame_us_for(96_000, BITS_24, 2, 1400).expect("a rung fits the default MTU");
assert_eq!(
frame_us, 2000,
"the documented rung at the default MTU ceiling"
);
let hires = Welcome {
host_caps: HOST_CAP_AUDIO_HIRES,
audio_codec: AUDIO_CODEC_PCM,
audio_rate_hz: 96_000,
audio_bits: BITS_24,
audio_frame_us: frame_us as u16,
..base
};
// AES: the block lands at 79..87 — cipher(1) + mgmt(2) + grants(4) + expiry(4) past 68.
let enc = hires.encode();
assert_eq!(enc.len(), 87, "68 + cipher 1 + mgmt 2 + access 8 + audio 8");
assert_eq!(enc[68], CIPHER_AES_128_GCM, "forced cipher placeholder");
assert_eq!(&enc[69..71], &0u16.to_le_bytes(), "forced mgmt placeholder");
assert_eq!(&enc[71..75], &GRANT_ALL.to_le_bytes(), "forced grants");
assert_eq!(&enc[75..79], &0u32.to_le_bytes(), "forced expiry");
assert_eq!(enc[79], AUDIO_CODEC_PCM);
assert_eq!(&enc[80..84], &96_000u32.to_le_bytes());
assert_eq!(enc[84], BITS_24);
assert_eq!(&enc[85..87], &(frame_us as u16).to_le_bytes());
assert_eq!(Welcome::decode(&enc).unwrap(), hires);
// ChaCha: the SAME eight bytes, shifted by the 32-byte key — 111..119.
let k32: [u8; 32] = core::array::from_fn(|i| i as u8 + 1);
let cha = Welcome {
cipher: CIPHER_CHACHA20_POLY1305,
key_chacha: Some(k32),
..hires
};
let cenc = cha.encode();
assert_eq!(cenc.len(), 119, "87 + the 32-byte ChaCha key");
assert_eq!(&cenc[101..103], &0u16.to_le_bytes(), "forced mgmt");
assert_eq!(&cenc[103..107], &GRANT_ALL.to_le_bytes(), "forced grants");
assert_eq!(&cenc[107..111], &0u32.to_le_bytes(), "forced expiry");
assert_eq!(cenc[111], AUDIO_CODEC_PCM);
assert_eq!(&cenc[112..116], &96_000u32.to_le_bytes());
assert_eq!(cenc[116], BITS_24);
assert_eq!(&cenc[117..119], &(frame_us as u16).to_le_bytes());
assert_eq!(Welcome::decode(&cenc).unwrap(), cha);
// The block is the same eight bytes in both, at a 32-byte offset — which is the whole
// claim: the decoder derives its position from `cipher`, not from a constant.
assert_eq!(&enc[79..87], &cenc[111..119]);
// ── The forced placeholders really decode as their own absence ────────────────────────
//
// Emitting audio drags four earlier fields onto the wire that the session does not
// otherwise use. They must read exactly as they would have if omitted, or a hi-res
// session would silently acquire a mgmt port / an access mask it never had.
for w in [
Welcome::decode(&enc).unwrap(),
Welcome::decode(&cenc).unwrap(),
] {
assert_eq!(w.mgmt_port, 0, "forced placeholder, not an advertised port");
assert_eq!(w.grants, GRANT_ALL, "forced placeholder, full control");
assert_eq!(w.expires_in_secs, 0, "forced placeholder, permanent");
assert!(depth_is_supported(w.audio_bits));
}
// …and the block composes with real values in those slots rather than only with zeros.
let guest_hires = Welcome {
mgmt_port: 47991,
grants: GRANT_PRESET_CONTROLLER_ONLY,
expires_in_secs: 4 * 3600,
..hires
};
let genc = guest_hires.encode();
assert_eq!(
genc.len(),
87,
"same length — the placeholders were already paid for"
);
assert_eq!(&genc[69..71], &47991u16.to_le_bytes());
assert_eq!(&genc[71..75], &GRANT_PRESET_CONTROLLER_ONLY.to_le_bytes());
assert_eq!(&genc[79..87], &enc[79..87], "the audio block is unmoved");
assert_eq!(Welcome::decode(&genc).unwrap(), guest_hires);
// ── Back-compat: every shorter wire form is a legacy Opus session ─────────────────────
//
// An older host and a host whose §8.4 gate declined produce the same bytes, and must
// therefore decode the same way. Resolving to 48 000/16 rather than to a raw 0 is what
// lets a client open its device straight off the Welcome without knowing the difference.
let mgmt_era = Welcome {
mgmt_port: 47991,
..base
}
.encode();
for old in [&base.encode()[..], &mgmt_era[..], &enc[..79], &cenc[..111]] {
let w = Welcome::decode(old).unwrap();
assert_eq!(w.audio_codec, AUDIO_CODEC_OPUS);
assert_eq!(w.audio_rate_hz, SAMPLE_RATE_HZ);
assert_eq!(w.audio_bits, BITS_16);
assert_eq!(w.audio_frame_us, 0);
}
// An access-era reader consuming exactly the prefix it knows sees the base session, plus
// the HOST_CAP_AUDIO_HIRES bit — which rides the long-standing `host_caps` byte at 67, not
// the appended block. That is the point: nothing BEFORE the block moved, and a cap bit an
// older client doesn't recognise is inert.
assert_eq!(
Welcome::decode(&enc[..79]).unwrap(),
Welcome {
host_caps: HOST_CAP_AUDIO_HIRES,
..base
}
);
// A truncated block is never HALF a rate: cut mid-u32 and the rate reads as the legacy
// value, not as two bytes of 96 000 zero-extended.
let torn = Welcome::decode(&enc[..82]).unwrap();
assert_eq!(torn.audio_codec, AUDIO_CODEC_PCM, "the codec byte survived");
assert_eq!(torn.audio_rate_hz, SAMPLE_RATE_HZ);
assert_eq!(torn.audio_frame_us, 0);
// A depth this plane cannot carry folds to 16 rather than being handed to a client that
// would then walk a 0xD3 payload at the wrong stride. Only a corrupt wire gets here (the
// Welcome rides the pinned-TLS control stream), which is exactly why it is worth pinning.
let mut bad_depth = enc.clone();
bad_depth[84] = 32;
assert_eq!(Welcome::decode(&bad_depth).unwrap().audio_bits, BITS_16);
// The RATE, by contrast, is passed through verbatim. Clamping an unrecognised rate to one
// we know is the §4.3 lie — label right, content wrong — so a rate outside today's ladder
// must reach the client, which refuses it loudly.
let mut odd_rate = enc.clone();
odd_rate[80..84].copy_from_slice(&44_100u32.to_le_bytes());
assert_eq!(Welcome::decode(&odd_rate).unwrap().audio_rate_hz, 44_100);
}
/// The hi-res audio request on `Hello` (`design/hi-res-audio.md` §7) — the placeholder
/// discipline, the post-`display_hdr` byte budget, and the legacy defaults in both
/// directions.
///
/// `Hello` does NOT use `Welcome`'s conditional-tail pattern: every trailing field has a
/// placeholder form, so a present field always lands at a deterministic offset. The one
/// exception is `display_hdr`, a fixed 28-byte block with NO placeholder whose presence is
/// disambiguated by REMAINING LENGTH — which is what caps the whole tail after it at 27
/// bytes and makes the budget below load-bearing rather than bookkeeping.
#[test]
fn hello_hires_audio_request_roundtrip_and_back_compat() {
let base = Hello {
abi_version: 2,
mode: Mode {
width: 1920,
height: 1080,
refresh_hz: 60,
},
compositor: CompositorPref::Auto,
gamepad: GamepadPref::Auto,
bitrate_kbps: 0,
name: None,
launch: None,
video_caps: 0,
audio_channels: 2,
video_codecs: 0,
preferred_codec: 0,
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
audio_rate_hz: SAMPLE_RATE_HZ,
audio_bits: BITS_16,
};
// The legacy request costs nothing: still the 26-byte bitrate-era form.
assert_eq!(base.encode().len(), 26);
assert_eq!(Hello::decode(&base.encode()).unwrap(), base);
// ── The request forces the whole earlier chain ────────────────────────────────────────
//
// 26 bitrate-era bytes + 6 single-byte placeholders (name len, launch len, video_caps,
// audio_channels, video_codecs, preferred_codec) + client_caps 1 + max_shard_payload 2 +
// the 4-byte rate. No HDR block, and no depth byte: 16-bit is the default and nothing
// follows it to force it out.
let rate_only = Hello {
audio_rate_hz: 96_000,
..base.clone()
};
let enc = rate_only.encode();
assert_eq!(enc.len(), 26 + 6 + 1 + 2 + 4);
assert_eq!(&enc[26..28], &[0, 0], "name + launch length placeholders");
assert_eq!(enc[28], 0, "video_caps placeholder");
assert_eq!(enc[29], 2, "audio_channels placeholder = stereo");
assert_eq!(
&enc[30..32],
&[0, 0],
"video_codecs + preferred placeholders"
);
assert_eq!(enc[32], 0, "client_caps placeholder");
assert_eq!(
&enc[33..35],
&0u16.to_le_bytes(),
"max_shard_payload placeholder"
);
assert_eq!(&enc[35..39], &96_000u32.to_le_bytes());
let dec = Hello::decode(&enc).unwrap();
assert_eq!(dec, rate_only);
assert_eq!(
dec.client_caps, 0,
"the forced placeholder reads as absence"
);
assert_eq!(dec.max_shard_payload, 0, "…and so does this one");
assert_eq!(dec.audio_bits, BITS_16, "absent depth → the legacy 16");
// The depth is the LAST field, so it forces the rate out ahead of it — as the legacy
// 48 000, not as the struct's value, which is the same thing said twice. Without that,
// "16-bit at the default rate" and "24-bit at the default rate" would disagree about
// where the depth byte lives.
let bits_only = Hello {
audio_bits: BITS_24,
..base.clone()
};
let benc = bits_only.encode();
assert_eq!(benc.len(), 26 + 6 + 1 + 2 + 4 + 1);
assert_eq!(&benc[35..39], &SAMPLE_RATE_HZ.to_le_bytes(), "forced rate");
assert_eq!(benc[39], BITS_24);
assert_eq!(Hello::decode(&benc).unwrap(), bits_only);
// The real thing a client sends: capable-and-opted-in, both parameters set.
let req = Hello {
client_caps: CLIENT_CAP_AUDIO_HIRES,
audio_rate_hz: 96_000,
audio_bits: BITS_24,
..base.clone()
};
let renc = req.encode();
assert_eq!(Hello::decode(&renc).unwrap(), req);
assert_eq!(renc[32], CLIENT_CAP_AUDIO_HIRES);
// ⚠ And it must NOT be mistaken for an HdrMeta block. The 8-byte tail is under the
// fixed 28-byte block length, so the remaining-length test correctly says "no HDR" —
// the budget below is what keeps that true.
assert_eq!(Hello::decode(&renc).unwrap().display_hdr, None);
// ── The budget the tail is spending (documented on `Hello::client_caps`) ──────────────
//
// Everything after `display_hdr` must stay under HDR_META_BODY_LEN bytes or the
// remaining-length disambiguation stops being sound: a tail that long is indistinguishable
// from an HDR block, and every field in it decodes as garbage. 8 spent, 19 free.
let vol = HdrMeta {
display_primaries: [[13250, 34500], [7500, 3000], [34000, 16000]],
white_point: [15635, 16450],
max_display_mastering_luminance: 8_000_000,
min_display_mastering_luminance: 500,
max_cll: 0,
max_fall: 400,
};
let full = Hello {
display_hdr: Some(vol),
client_caps: CLIENT_CAP_AUDIO_HIRES | CLIENT_CAP_CURSOR,
max_shard_payload: 8908,
audio_rate_hz: 96_000,
audio_bits: BITS_24,
..base.clone()
};
let fenc = full.encode();
let post_hdr = fenc.len() - (26 + 6 + HDR_META_BODY_LEN);
assert_eq!(
post_hdr, 8,
"client_caps 1 + max_shard_payload 2 + audio_rate_hz 4 + audio_bits 1"
);
assert!(
post_hdr < HDR_META_BODY_LEN,
"the post-display_hdr tail must stay under {HDR_META_BODY_LEN} bytes — \
at or past it, a Hello WITHOUT an HDR block is read as one WITH"
);
// …and with the block actually present, every field after it is still found.
assert_eq!(Hello::decode(&fenc).unwrap(), full);
// ── Back-compat, both directions ──────────────────────────────────────────────────────
//
// A pre-hi-res client omits both fields; a pre-hi-res HOST truncates its read before
// them. Either way the request resolves to 48 kHz / 16-bit — what such a peer means.
assert_eq!(
Hello::decode(&base.encode()).unwrap().audio_rate_hz,
SAMPLE_RATE_HZ
);
assert_eq!(Hello::decode(&base.encode()).unwrap().audio_bits, BITS_16);
let pre_audio = Hello::decode(&renc[..35]).unwrap();
assert_eq!(pre_audio.audio_rate_hz, SAMPLE_RATE_HZ);
assert_eq!(pre_audio.audio_bits, BITS_16);
assert_eq!(pre_audio.max_shard_payload, 0);
// A torn rate is never half a rate.
assert_eq!(
Hello::decode(&renc[..37]).unwrap().audio_rate_hz,
SAMPLE_RATE_HZ
);
// An unsupported depth folds to 16 — a request is only a request, so falling back costs
// a hi-res session and never correctness.
let mut bad_depth = renc.clone();
bad_depth[39] = 32;
assert_eq!(Hello::decode(&bad_depth).unwrap().audio_bits, BITS_16);
assert!(depth_is_supported(Hello::decode(&renc).unwrap().audio_bits));
}
}
crates/punktfunk-host/src/audio.rs
+118 -10
View File
@@ -91,6 +91,25 @@ pub trait AudioCapturer: Send {
CHANNELS as u32
}
/// The sample rate this capturer is **actually** delivering — which is not necessarily the
/// one it was asked for (`design/hi-res-audio.md` §8.1).
///
/// The whole hi-res feature turns on this distinction. Both backends can be handed a rate
/// their endpoint does not really run at and will happily resample to it without an error:
/// WASAPI's `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM` reconciles our format with the engine's in
/// whichever direction is needed (§4.3), and PipeWire's resampler does the same in the
/// legacy monitor mode (§4.4). A host that reported its *request* would advertise 96 kHz in
/// the `Welcome`, spend the bandwidth, and deliver interpolated 48 kHz — the same
/// "label right, content wrong" class of bug as the HDR RB-swap, which survived a long time
/// precisely because both ends audited clean.
///
/// So the contract is: report what was granted, and let the caller decline. The default is
/// the legacy rate, which is what every backend that has not been taught to negotiate one
/// genuinely opens at.
fn sample_rate(&self) -> u32 {
SAMPLE_RATE
}
/// Discard any buffered chunks (called when a persistent capturer is reused for a new
/// stream, so the client doesn't hear stale audio captured while idle). On Linux this is
/// also the session-start hook: the stream-sink capturer re-claims the default sink here
@@ -105,28 +124,117 @@ pub trait AudioCapturer: Send {
fn idle(&mut self) {}
}
/// Open a live capturer for system output via PipeWire, asking for `channels` interleaved
/// channels. Default: a host-owned stream sink claimed as the default output (the sink
/// advertises exactly `channels`, so apps can produce real surround); with
/// `PUNKTFUNK_STREAM_SINK=0`, the default sink's monitor, where a sink with fewer channels
/// gets the missing positions filled with silence (zero upmix).
/// What the capture path can honestly promise about its sample rate, answered **before** a
/// capturer exists (`design/hi-res-audio.md` §8.4 condition 4).
///
/// The gate that decides a session's audio plane runs inside the handshake, and the capturer is
/// not opened until the audio thread starts — well after the `Welcome` has already promised the
/// client a rate and the client has opened its output device at it. Discovering the truth there
/// is too late: the only thing the audio thread can then do is end the lossless plane, and
/// §8.4 is explicit that silence is the one unacceptable outcome. So the question has to be
/// answerable from a DEVICE-LEVEL query that costs no stream — which is what each backend
/// answers below.
///
/// Deliberately three-valued rather than a `bool` or an `Option<u32>`: "the host declares the
/// rate" and "the device runs at 48 kHz" are different facts with different consequences, and
/// collapsing either into "unknown" would decline hi-res on the one configuration (§4.4) where
/// it is honest by construction.
///
/// Each variant carries a `dead_code` allow for the platforms that never construct it — the same
/// `cfg_attr` this module already puts on `wiring_plan` and `capture_policy`, and for the same
/// reason: the crate root's blanket `#![allow(dead_code)]` covers it today, and a local marker is
/// what keeps this honest if that scaffold-era allow is ever narrowed. Per-variant rather than on
/// the enum, so a genuinely dead variant added later would still be caught.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum CaptureRate {
/// The HOST declares the rate to the graph and applications render into it natively, so
/// whatever is asked for is what arrives — there is no upstream resampler to be fooled by.
/// Linux stream-sink mode, the default (§4.4).
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
Declared,
/// The device runs at exactly this rate, and asking it for anything else succeeds anyway by
/// resampling — so only a request at or below it is honest; above it we would advertise a
/// rate, spend the bandwidth, and deliver interpolation.
///
/// Both hosts reach this, from different queries and against different resamplers. On
/// Windows it is the endpoint's engine mix format, which `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM`
/// reconciles our request with in whichever direction is needed, with no error (§4.3). On
/// Linux it is the rate of the sink a `PUNKTFUNK_STREAM_SINK=0` monitor capture would follow,
/// read from the PipeWire registry, because the resampler between that node and our stream
/// is just as silent about what it hid (§4.4).
#[cfg_attr(not(any(target_os = "linux", target_os = "windows")), allow(dead_code))]
Engine(u32),
/// Nobody could be made to say, and so the answer is no: a Linux monitor capture whose
/// elected sink is gone, idle or unnameable (§8.3), a Windows probe that could not reach the
/// endpoint, or a platform with no capture backend at all. Hi-res declines.
Unknown,
}
impl CaptureRate {
/// Whether a session capturing at `rate_hz` would GENUINELY be captured at `rate_hz`.
///
/// The pessimistic direction is deliberate everywhere: an unknown answer declines, because
/// the cost of being wrong is a session that says 96 kHz, spends 4.6 Mbps saying it, and
/// carries interpolated 48 kHz — the same "both ends audit clean, the content is wrong"
/// class as the HDR RB-swap, which survived a long time for exactly that reason.
pub fn can_deliver(self, rate_hz: u32) -> bool {
match self {
CaptureRate::Declared => true,
CaptureRate::Engine(hz) => rate_hz <= hz,
CaptureRate::Unknown => false,
}
}
}
/// Ask the capture path what rate it can honestly deliver, WITHOUT opening a capture stream and
/// without changing anything about the box — see [`CaptureRate`].
///
/// Blocking — Windows enumerates endpoints and activates an `IAudioClient` per candidate, and a
/// Linux monitor-mode host runs a bounded PipeWire registry round-trip — so callers on the async
/// path run it off the reactor. Called only when hi-res is actually on the table: an ordinary
/// session must not pay for a feature nobody asked for.
#[cfg(target_os = "linux")]
pub fn open_audio_capture(channels: u32) -> Result<Box<dyn AudioCapturer>> {
linux::PwAudioCapturer::open(channels).map(|c| Box::new(c) as Box<dyn AudioCapturer>)
pub fn probe_capture_rate() -> CaptureRate {
linux::probe_capture_rate()
}
#[cfg(target_os = "windows")]
pub fn open_audio_capture(channels: u32) -> Result<Box<dyn AudioCapturer>> {
pub fn probe_capture_rate() -> CaptureRate {
audio_control::probe_capture_rate()
}
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
pub fn probe_capture_rate() -> CaptureRate {
// No capture backend at all — `open_audio_capture` bails on this target, so there is no
// plane to promise a rate for.
CaptureRate::Unknown
}
/// Open a live capturer for system output via PipeWire, asking for `channels` interleaved
/// channels at `rate_hz`. Default: a host-owned stream sink claimed as the default output (the
/// sink advertises exactly `channels`, so apps can produce real surround); with
/// `PUNKTFUNK_STREAM_SINK=0`, the default sink's monitor, where a sink with fewer channels
/// gets the missing positions filled with silence (zero upmix).
///
/// `rate_hz` is a REQUEST, exactly like `channels`. What the graph actually granted is read
/// back from [`AudioCapturer::sample_rate`] — see that method for why the difference matters.
#[cfg(target_os = "linux")]
pub fn open_audio_capture(channels: u32, rate_hz: u32) -> Result<Box<dyn AudioCapturer>> {
linux::PwAudioCapturer::open(channels, rate_hz).map(|c| Box::new(c) as Box<dyn AudioCapturer>)
}
#[cfg(target_os = "windows")]
pub fn open_audio_capture(channels: u32, rate_hz: u32) -> Result<Box<dyn AudioCapturer>> {
// The capture thread runs the audio wiring plan itself (audio_control::wire_now) before
// resolving its endpoint — a fresh plan per open, because Windows endpoints churn — and
// parks the default playback device on the plan's loopback endpoint (a silent sink by
// default: audio plays on the client only) until the capturer is dropped.
wasapi_cap::WasapiLoopbackCapturer::open(channels)
wasapi_cap::WasapiLoopbackCapturer::open(channels, rate_hz)
.map(|c| Box::new(c) as Box<dyn AudioCapturer>)
}
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
pub fn open_audio_capture(_channels: u32) -> Result<Box<dyn AudioCapturer>> {
pub fn open_audio_capture(_channels: u32, _rate_hz: u32) -> Result<Box<dyn AudioCapturer>> {
anyhow::bail!("audio capture requires Linux + PipeWire or Windows + WASAPI")
}
crates/punktfunk-host/src/audio/capture_policy.rs
+211 -43
View File
@@ -234,10 +234,6 @@ impl CaptureStats {
}
}
/// A departure this far past its slot is a slip worth counting rather than ordinary jitter: one
/// whole protocol frame, so a frame that merely rounds late never scores.
const LATE_DEPARTURE: Duration = Duration::from_millis(FRAME_MS as u64);
/// One reporting window of AUDIO EGRESS vitals (WP-C).
///
/// Capture has been instrumented since WP-A2 and the send path has not, so a field log could show
@@ -249,13 +245,26 @@ const LATE_DEPARTURE: Duration = Duration::from_millis(FRAME_MS as u64);
/// The point of these counters is to be *boring*. If departures are clean while capture reports
/// holes, the pacing rework introduced in v0.25 is acquitted permanently and the search moves
/// upstream for good.
#[derive(Default)]
///
/// **Denominated in the SESSION's frame**, like [`InfillPolicy`] and for the same reason — see
/// [`SendStats::new`].
pub(crate) struct SendStats {
/// One protocol frame of THIS session, and therefore the SLIP THRESHOLD: a departure at least
/// this far past its paced slot is a slip worth counting rather than ordinary jitter, so a
/// frame that merely rounds late never scores.
///
/// Carried rather than read from the Opus [`FRAME_MS`](punktfunk_core::audio::FRAME_MS), which
/// is what this threshold used to be. The RULE was always "one whole protocol frame"; only the
/// arithmetic assumed there was one such thing. On a lossless plane pacing 1 ms frames a
/// session could miss every slot by four whole frames and still report `late=0` — a counter
/// reading clean for a plane that is not, which is worse than no counter at all given why
/// these exist (an unfalsifiable suspect stays on the list forever).
frame: Duration,
pub(crate) sent: u64,
/// Frames synthesized to cover a capture hole. Wire continuity and captured continuity are
/// different claims and a log that conflates them cannot be used to judge either.
pub(crate) infilled: u64,
/// Departures that missed their paced slot by at least [`LATE_DEPARTURE`].
/// Departures that missed their paced slot by at least one whole [`frame`](Self::frame).
pub(crate) late: u64,
/// The worst such miss, µs — kept even when the count is zero, because "never late" and
/// "never late by a whole frame" are different statements.
@@ -270,6 +279,30 @@ pub(crate) struct SendStats {
}
impl SendStats {
/// A fresh window for a session pacing `frame_us`-long frames — `audio_frame_us` from the
/// `Welcome` on the lossless plane, 5 000 on the Opus one. The same value [`InfillPolicy`] and
/// the encode loop's pacer are built from, so all three agree on what a frame is.
///
/// A 5 ms session is bit-identical to the pre-hi-res behaviour: the slip threshold is the 5 ms
/// it always was.
///
/// Replaces `Default`, which is not merely absent but WRONG here: a zero frame makes
/// `late >= self.frame` true for every departure, so a window built by mistake would report
/// 100 % slipped slots. Windows are re-made on every flush, so that would not have been a
/// once-per-session error either.
pub(crate) fn new(frame_us: u32) -> SendStats {
SendStats {
// Same floor, and same reason, as `InfillPolicy::new`'s.
frame: Duration::from_micros(frame_us.max(1) as u64),
sent: 0,
infilled: 0,
late: 0,
max_late_us: 0,
max_spacing_us: 0,
reanchors: 0,
}
}
/// Score one frame leaving the host. `late` is how far past its paced slot it went (zero when
/// the schedule is unanchored), `since_prev` the spacing from the previous departure.
pub(crate) fn observe_departure(
@@ -283,7 +316,8 @@ impl SendStats {
self.infilled += 1;
}
self.max_late_us = self.max_late_us.max(late.as_micros() as u64);
if late >= LATE_DEPARTURE {
// One whole frame of THIS session — inclusive, exactly as the old constant comparison was.
if late >= self.frame {
self.late += 1;
}
if let Some(gap) = since_prev {
@@ -304,17 +338,26 @@ impl SendStats {
}
}
/// How long a capture hole may run before the wire starts covering it. Two protocol frames: long
/// enough that ordinary quantum jitter never trips it, short enough that the client's ring never
/// notices the hole.
pub(crate) const INFILL_AFTER: Duration = Duration::from_millis(2 * FRAME_MS as u64);
/// How much silence one hole may be covered with. Past this the host is not glitching, it is
/// QUIET — a desktop between games is legitimately silent for hours and paying a few kbps to keep
/// saying so is absurd — so the wire stops, which is exactly the behaviour that shipped before.
///
/// Deliberately NOT re-derived from the frame duration, unlike everything else in
/// [`InfillPolicy`]: this is a statement about WALL CLOCK — how long a hole stays worth covering
/// before we admit the desktop is simply quiet — and half a second is half a second whether the
/// plane sends 100 or 500 frames into it. What did have to change is the ACCOUNTING: the budget
/// used to be spent in units of the Opus frame whatever the session's real frame was, so a 1 ms
/// lossless frame burned the whole 500 ms in 100 ms of real time (`design/hi-res-audio.md` §4.2 —
/// the lossless plane's frames are shorter than 5 ms by construction).
pub(crate) const INFILL_MAX: Duration = Duration::from_millis(500);
/// One protocol audio frame — the wire's unit, and the granularity infill works in.
const FRAME_MS: u32 = punktfunk_core::audio::FRAME_MS;
// NB there is no module-scope frame constant here any more, deliberately. Both figures that used
// to be written against `punktfunk_core::audio::FRAME_MS` — the infill threshold and the egress
// slip threshold — are now carried by the policy that uses them ([`InfillPolicy::new`],
// [`SendStats::new`]), because that constant is the OPUS plane's frame and only its frame: the
// lossless `0xD3` plane's is whatever the handshake negotiated. Anything denominated in it up here
// would be measuring one plane with the other's ruler. The tests below keep a local copy to write
// the "a 5 ms session is unchanged" assertions against.
/// What the wire owes for the slot that is due now.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -337,34 +380,75 @@ pub(crate) enum Infill {
/// anchored, so what the listener loses shrinks to exactly the audio that was genuinely missing.
///
/// Time is passed IN, so the policy is pure and its tests run on every platform.
#[derive(Default)]
///
/// **Denominated in the SESSION's frame, not in [`FRAME_MS`]** — see [`InfillPolicy::new`]. Every
/// figure here used to be written against the Opus 5 ms frame, which was correct for as long as
/// 5 ms was the only frame there was.
pub(crate) struct InfillPolicy {
/// One protocol frame of THIS session: the fixed 5 ms of the Opus `0xC9` plane, or the
/// negotiated `audio_frame_us` of the lossless `0xD3` one. The two are not interchangeable —
/// a 96/24 session paces 1 ms frames, so a policy written in 5 ms units would be off by 5×
/// in both directions at once (covering a hole a fifth as long, and spending its budget five
/// times as fast).
frame: Duration,
/// Silence already sent for the open hole. Denominated in TIME rather than in frames or
/// callbacks, which is the recorded lesson from the client's de-prime fuse: a count there made
/// an iPad give up three times sooner than a Mac for no reason anyone intended.
filled_ms: u32,
/// an iPad give up three times sooner than a Mac for no reason anyone intended. Exactly the
/// bug this field's `u32` millisecond predecessor had, one layer down: it counted `FRAME_MS`
/// per *actual* frame, so the same 500 ms budget meant a different amount of real time on
/// every rung of the lossless frame ladder.
filled: Duration,
/// Latched once a hole outlives the budget and the wire falls silent.
broke: bool,
}
impl InfillPolicy {
/// `frame_us` is the session's RESOLVED frame duration: `audio_frame_us` from the `Welcome`
/// on the lossless plane, 5 000 on the Opus one — the same value the encode loop paces and
/// stamps `pts_ns` with, so the wire and this policy cannot disagree about what a frame is.
///
/// A 5 ms session is bit-identical to the pre-hi-res behaviour: [`Self::after`] is the 10 ms
/// it always was, and the budget is still exactly 100 frames of silence.
///
/// The `max(1)` is a floor against a malformed plane, not a real case — the §8.4 gate never
/// resolves a zero-length frame, and the encode loop applies the same floor to its pacer. A
/// zero here would leave [`Self::decide`] unable to spend the budget at all, so a hole would
/// be covered with silence forever.
pub(crate) fn new(frame_us: u32) -> InfillPolicy {
InfillPolicy {
frame: Duration::from_micros(frame_us.max(1) as u64),
filled: Duration::ZERO,
broke: false,
}
}
/// How long a capture hole may run before the wire starts covering it. Two protocol frames:
/// long enough that ordinary quantum jitter never trips it, short enough that the client's
/// ring never notices the hole. Two frames of THIS session — the client's ring is sized in
/// its own frames, so that is what "before it notices" is measured in.
pub(crate) fn after(&self) -> Duration {
self.frame * 2
}
/// Decide the slot that is due now. Call EXACTLY once per due frame — it consumes budget.
pub(crate) fn decide(&mut self, since_last_chunk: Duration) -> Infill {
if since_last_chunk < INFILL_AFTER {
if since_last_chunk < self.after() {
return Infill::Wait;
}
if self.filled_ms as u64 >= INFILL_MAX.as_millis() as u64 {
if self.filled >= INFILL_MAX {
self.broke = true;
return Infill::Quiet;
}
self.filled_ms += FRAME_MS;
// One frame of silence costs one frame of budget — the identity that was missing, and the
// whole of the fix. Nothing else in this file needs to know what a frame is worth.
self.filled += self.frame;
Infill::Silence
}
/// True once the budget is spent, so the caller can go back to blocking for real audio
/// instead of waking every few milliseconds to decide to stay quiet.
pub(crate) fn exhausted(&self) -> bool {
self.filled_ms as u64 >= INFILL_MAX.as_millis() as u64
self.filled >= INFILL_MAX
}
/// A real chunk arrived. Returns whether the hole it closed BROKE continuity — the wire went
@@ -372,7 +456,7 @@ impl InfillPolicy {
/// both describe audio from before a discontinuity, and neither may be spliced onto what
/// comes next.
pub(crate) fn chunk_arrived(&mut self) -> bool {
self.filled_ms = 0;
self.filled = Duration::ZERO;
std::mem::take(&mut self.broke)
}
}
@@ -536,19 +620,29 @@ mod tests {
assert_eq!(vm.gaps, 1);
}
/// One OPUS protocol frame — the `0xC9` plane's fixed 5 ms. Test-scope on purpose: nothing in
/// the policies above is denominated in it any more (see the note by [`INFILL_MAX`]), and the
/// only thing it is still good for is writing the "a 5 ms session is unchanged" assertions.
const FRAME_MS: u32 = punktfunk_core::audio::FRAME_MS;
/// The same, in µs — what [`InfillPolicy::new`] and [`SendStats::new`] are handed for a `0xC9`
/// session.
const OPUS_FRAME_US: u32 = FRAME_MS * 1_000;
/// Drive one hole from the moment it opens until the policy gives up on it, the way the
/// encode loop does: one decision per due frame slot.
fn cover_a_hole(p: &mut InfillPolicy) -> usize {
/// encode loop does: one decision per due frame slot, `frame` apart.
fn cover_a_hole(p: &mut InfillPolicy, frame: Duration) -> usize {
let mut silence = 0usize;
let mut open = INFILL_AFTER;
let mut open = p.after();
loop {
match p.decide(open) {
Infill::Silence => {
silence += 1;
open += Duration::from_millis(FRAME_MS as u64);
open += frame;
}
Infill::Quiet => return silence,
Infill::Wait => unreachable!("the hole is open — {open:?} is past INFILL_AFTER"),
Infill::Wait => {
unreachable!("the hole is open — {open:?} is past the infill threshold")
}
}
assert!(silence < 10_000, "the budget must be finite");
}
@@ -560,15 +654,13 @@ mod tests {
/// forever.
#[test]
fn infill_covers_a_hole_and_then_admits_the_host_is_quiet() {
let mut p = InfillPolicy::default();
let frame = Duration::from_millis(FRAME_MS as u64);
let mut p = InfillPolicy::new(OPUS_FRAME_US);
// Ordinary quantum jitter, not a hole — nothing owed.
assert_eq!(p.decide(Duration::ZERO), Infill::Wait);
assert_eq!(
p.decide(INFILL_AFTER - Duration::from_millis(1)),
Infill::Wait
);
assert_eq!(p.decide(p.after() - Duration::from_millis(1)), Infill::Wait);
let silence = cover_a_hole(&mut p);
let silence = cover_a_hole(&mut p, frame);
assert_eq!(
silence as u64 * FRAME_MS as u64,
INFILL_MAX.as_millis() as u64,
@@ -577,11 +669,59 @@ mod tests {
assert!(p.exhausted(), "…and then stop asking");
}
/// **The gate on re-deriving these policies from `audio_frame_us`.** A 5 ms Opus session must
/// behave exactly as it did when all three figures were written against `FRAME_MS` — same
/// 10 ms infill threshold, same 100 frames of cover, same 5 ms slip threshold — or a change
/// made for the lossless plane has silently retuned the plane every shipping client uses.
#[test]
fn a_five_millisecond_opus_session_is_unchanged() {
let mut p = InfillPolicy::new(OPUS_FRAME_US);
assert_eq!(p.after(), Duration::from_millis(10), "two 5 ms frames");
assert_eq!(
cover_a_hole(&mut p, Duration::from_millis(FRAME_MS as u64)),
100,
"500 ms of budget in 5 ms frames"
);
// …and the egress side's slip threshold, which was the same constant. BOTH sides of the
// boundary, because one alone pins nothing: "4.999 ms is not late" also passes for a
// threshold of infinity, and "5 ms is late" also passes for a threshold of zero.
let mut s = SendStats::new(OPUS_FRAME_US);
s.observe_departure(Duration::from_micros(4_999), None, false);
assert_eq!(s.late, 0, "just under a 5 ms frame is jitter");
s.observe_departure(Duration::from_millis(5), None, false);
assert_eq!(s.late, 1, "one whole 5 ms frame is a slipped slot");
}
/// …and the point of the exercise: every rung of the lossless frame ladder gets the SAME
/// wall-clock budget, rather than the same frame COUNT. Denominated in `FRAME_MS`, a 1 ms
/// frame spent all 500 ms of budget in 100 ms of real time — the wire fell silent five times
/// sooner than the policy says it should, on the one plane that has no PLC to hide it.
#[test]
fn every_lossless_frame_length_gets_the_same_wall_clock_budget() {
for frame_us in punktfunk_core::audio::pcm::FRAME_US_LADDER {
let frame = Duration::from_micros(frame_us as u64);
let mut p = InfillPolicy::new(frame_us);
assert_eq!(p.after(), frame * 2, "{frame_us} µs: two of its own frames");
let silence = cover_a_hole(&mut p, frame);
// Covered to within ONE frame, not exactly: a frame is atomic, so a rung that does
// not divide 500 ms (3 ms → 167 frames = 501 ms) rounds up by less than one frame
// rather than stopping short. The 5 ms Opus rung divides it exactly, which is what
// keeps that plane bit-identical.
let covered = frame * silence as u32;
assert!(
covered >= INFILL_MAX && covered < INFILL_MAX + frame,
"{frame_us} µs covered {silence} frames = {covered:?}, want {INFILL_MAX:?} \
rounded up by under one frame"
);
}
}
/// A stream that is flowing must never synthesize anything — this policy is invisible until
/// something is actually wrong.
#[test]
fn a_flowing_stream_never_infills() {
let mut p = InfillPolicy::default();
let mut p = InfillPolicy::new(OPUS_FRAME_US);
for _ in 0..1_000 {
assert_eq!(
p.decide(Duration::from_millis(FRAME_MS as u64)),
@@ -597,24 +737,25 @@ mod tests {
/// predecessor still describes the frame before this one and the client can keep using it.
#[test]
fn only_an_uncovered_hole_breaks_continuity() {
let mut covered = InfillPolicy::default();
for k in 0..20u64 {
covered.decide(INFILL_AFTER + Duration::from_millis(FRAME_MS as u64 * k));
let frame = Duration::from_millis(FRAME_MS as u64);
let mut covered = InfillPolicy::new(OPUS_FRAME_US);
for k in 0..20u32 {
covered.decide(covered.after() + frame * k);
}
assert!(
!covered.chunk_arrived(),
"a covered hole is continuous — the client heard silence, not a splice"
);
let mut lost = InfillPolicy::default();
cover_a_hole(&mut lost);
let mut lost = InfillPolicy::new(OPUS_FRAME_US);
cover_a_hole(&mut lost, frame);
assert!(
lost.chunk_arrived(),
"past the budget the wire went quiet, so nothing before the hole may be spliced on"
);
// …and the next hole starts from a clean budget rather than an exhausted one.
assert!(!lost.exhausted());
assert_eq!(cover_a_hole(&mut lost) as u64 * FRAME_MS as u64, 500);
assert_eq!(cover_a_hole(&mut lost, frame) as u64 * FRAME_MS as u64, 500);
}
/// A deliberate pause is not a hole. The caller drops its stamp across a state transition, so
@@ -695,7 +836,7 @@ mod tests {
/// produce a line a reader can dismiss at a glance.
#[test]
fn a_healthy_pacer_reports_nothing_alarming() {
let mut s = SendStats::default();
let mut s = SendStats::new(OPUS_FRAME_US);
let frame = Duration::from_millis(FRAME_MS as u64);
for i in 0..200 {
s.observe_departure(Duration::ZERO, (i > 0).then_some(frame), false);
@@ -712,16 +853,43 @@ mod tests {
/// "never late" and "never late by a whole frame" become the same report.
#[test]
fn sub_frame_lateness_is_measured_without_being_counted() {
let mut s = SendStats::default();
let mut s = SendStats::new(OPUS_FRAME_US);
s.observe_departure(Duration::from_micros(3_400), None, false);
assert_eq!(s.late, 0, "3.4 ms has not slipped a whole 5 ms slot");
assert_eq!(s.max_late_ms(), 3, "…and it is still on the record");
}
/// The egress twin of `every_lossless_frame_length_gets_the_same_wall_clock_budget`: a slipped
/// slot is one frame of WHATEVER this session paces.
///
/// Written against the Opus 5 ms constant, a 96/24 session pacing 1 ms frames could miss every
/// single slot by four whole frames and still report `late=0` — the log line's most load-bearing
/// number reading clean for a plane that is not, which is the exact opposite of why [`SendStats`]
/// was added. The lossless plane makes that reachable; nothing about it was reachable before.
#[test]
fn a_slipped_slot_is_one_frame_of_whatever_this_session_paces() {
for frame_us in punktfunk_core::audio::pcm::FRAME_US_LADDER {
let frame = Duration::from_micros(frame_us as u64);
let mut s = SendStats::new(frame_us);
// One microsecond under a frame is jitter; the frame itself is a slip — inclusive,
// exactly as the `>= LATE_DEPARTURE` comparison always was.
s.observe_departure(frame - Duration::from_micros(1), None, false);
assert_eq!(s.late, 0, "{frame_us} µs: sub-frame lateness is jitter");
s.observe_departure(frame, None, false);
assert_eq!(
s.late, 1,
"{frame_us} µs: one whole frame is a slipped slot"
);
// …and both were MEASURED regardless, which is the property that makes "never late"
// and "never late by a whole frame" different reports.
assert_eq!(s.max_late_us, frame.as_micros() as u64);
}
}
/// A slot missed by a whole frame or more is the event the field logs could never show.
#[test]
fn a_slipped_slot_is_counted_and_its_worst_case_kept() {
let mut s = SendStats::default();
let mut s = SendStats::new(OPUS_FRAME_US);
s.observe_departure(Duration::from_millis(6), None, false);
s.observe_departure(
Duration::from_millis(41),
@@ -741,7 +909,7 @@ mod tests {
/// looks perfect on every other counter, and must not be readable as healthy audio.
#[test]
fn synthesized_frames_stay_distinguishable_from_captured_ones() {
let mut s = SendStats::default();
let mut s = SendStats::new(OPUS_FRAME_US);
let frame = Duration::from_millis(FRAME_MS as u64);
for _ in 0..100 {
s.observe_departure(Duration::ZERO, Some(frame), true);
crates/punktfunk-host/src/audio/linux/mod.rs
+148 -23
View File
@@ -27,13 +27,14 @@
//! surround session can replace a stereo capturer without leaking a PipeWire consumer (see
//! CLAUDE.md: a wedged link head-blocks the daemon).
mod monitor_rate;
pub(crate) mod pad_sink;
mod stream_sink;
use super::{AudioCapturer, MicBackendStats, VirtualMic, SAMPLE_RATE};
use anyhow::{anyhow, Context, Result};
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicUsize, Ordering};
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError};
use std::sync::Arc;
use std::thread;
@@ -52,6 +53,58 @@ fn stream_sink_enabled() -> bool {
.unwrap_or(true)
}
/// §8.4 condition 4 on Linux (`design/hi-res-audio.md` §4.4 / §8.3). The two capture modes give
/// structurally different answers, and that difference is the whole content of §4.4:
///
/// * **Stream-sink mode (the default).** We register the `Audio/Sink` node ourselves and
/// [`pw_thread`] declares its format, so applications render into it at that rate natively.
/// The rate we claim is the rate we get, by construction — there is no upstream resampler in
/// the path to lie about it, so the answer is yes for every rate the plane supports, and no
/// probe of any kind is needed to say so.
/// * **`PUNKTFUNK_STREAM_SINK=0` (monitor mode).** We capture somebody else's sink through
/// PipeWire's resampler, which reports a clean rate whatever the node upstream of it really
/// runs at — the same blindness WASAPI's autoconvert has. So the answer cannot come from our
/// own stream; it comes from the MONITORED NODE, read out of the registry by
/// [`monitor_rate::monitored_sink_rate`]. A rate that can be read is an
/// [`Engine`](super::CaptureRate::Engine) answer, exactly as a Windows endpoint's mix format
/// is, and the gate compares the request against it.
///
/// ⚠ **The two failure directions are not symmetric, and the code leans on that.** An unreadable
/// rate — a suspended sink, an unset metadata key, a node that vanished, a graph that did not
/// answer inside the probe's timeout — is [`Unknown`](super::CaptureRate::Unknown), which
/// declines and costs the session nothing but today's excellent Opus 48 kHz. A *guessed* rate
/// that turns out wrong costs a session that advertises 96 kHz, spends 4.6 Mbps on it, and
/// carries interpolated 48 kHz with both ends auditing clean. So this never guesses: there is no
/// "assume the graph default", no reading `EnumFormat` (a capability, not a fact), and no
/// falling back to the rate we asked for.
///
/// Note the asymmetry with Windows on purpose: there *every* answer needs a device query, here
/// only the monitor mode does, because in the default mode the host is the one declaring the
/// format.
pub(super) fn probe_capture_rate() -> super::CaptureRate {
if stream_sink_enabled() {
return super::CaptureRate::Declared;
}
match monitor_rate::monitored_sink_rate() {
Ok(rate_hz) => {
tracing::debug!(
rate_hz,
"hi-res capture-rate probe: the sink this host would monitor runs at this rate"
);
super::CaptureRate::Engine(rate_hz)
}
Err(e) => {
tracing::debug!(
reason = %format!("{e:#}"),
"hi-res capture-rate probe: the monitored sink's own rate is not readable — \
declining hi-res (PUNKTFUNK_STREAM_SINK=0 captures through PipeWire's resampler, \
so the rate our own stream reports proves nothing)"
);
super::CaptureRate::Unknown
}
}
}
pub struct PwAudioCapturer {
chunks: Receiver<Vec<f32>>,
channels: u32,
@@ -73,14 +126,25 @@ pub struct PwAudioCapturer {
/// meaningless. Distinct from `claimed`, which tracks the sink-routing claim and only
/// exists when the stream sink is enabled at all.
active: Arc<AtomicBool>,
/// The rate the graph actually NEGOTIATED, written by the format callback on the PipeWire
/// thread and read back by [`AudioCapturer::sample_rate`].
///
/// Seeded with the rate we asked for, because that is the honest answer until the graph has
/// said otherwise — and in stream-sink mode it is nearly always the final one, since the
/// host owns the sink and declares its format (`design/hi-res-audio.md` §4.4). In legacy
/// monitor mode the value is a weaker claim: it is the rate of the resampled stream we are
/// handed, not of the node upstream of it, which is why the §8.3 gate reads the monitored
/// node's own rate out of the registry ([`monitor_rate`]) rather than trusting this number.
negotiated_rate: Arc<AtomicU32>,
}
impl PwAudioCapturer {
pub fn open(channels: u32) -> Result<PwAudioCapturer> {
pub fn open(channels: u32, rate_hz: u32) -> Result<PwAudioCapturer> {
anyhow::ensure!(
matches!(channels, 1 | 2 | 6 | 8),
"unsupported audio channel count {channels} (want 2, 6 or 8)"
);
anyhow::ensure!(rate_hz > 0, "audio capture rate must be positive");
// Unique per capturer: overlapping instances (mid-session reopen, concurrent sessions)
// must never alias in metadata claims, and a fresh name gets fresh (unity) WirePlumber
// volume state instead of whatever a previous run left behind.
@@ -105,6 +169,8 @@ impl PwAudioCapturer {
// the first chunk.
let active = Arc::new(AtomicBool::new(true));
let thread_active = Arc::clone(&active);
let negotiated_rate = Arc::new(AtomicU32::new(rate_hz));
let thread_rate = Arc::clone(&negotiated_rate);
thread::Builder::new()
.name("punktfunk-pw-audio".into())
.spawn(move || {
@@ -112,9 +178,11 @@ impl PwAudioCapturer {
tx,
quit_rx,
channels,
rate_hz,
thread_sink_name,
ready_tx,
thread_active,
thread_rate,
) {
tracing::error!(error = %format!("{e:#}"), "pipewire audio thread failed");
}
@@ -141,6 +209,7 @@ impl PwAudioCapturer {
sink_name,
claimed,
active,
negotiated_rate,
})
}
}
@@ -179,6 +248,10 @@ impl AudioCapturer for PwAudioCapturer {
self.channels
}
fn sample_rate(&self) -> u32 {
self.negotiated_rate.load(Ordering::Relaxed)
}
fn drain(&mut self) {
while self.chunks.try_recv().is_ok() {}
// A parked capturer being reused = a new session starting: re-claim the default sink
@@ -395,6 +468,19 @@ const MIC_STALE: Duration = Duration::from_secs(1);
/// against the same number the ask used.
const CAPTURE_QUANTUM_FRAMES: u32 = 240;
/// [`CAPTURE_QUANTUM_FRAMES`] restated at `rate_hz` — the same 5 ms of wall time, whatever the
/// rate. A hi-res session captures at 96 kHz (`design/hi-res-audio.md` §3), where asking for a
/// flat 240 frames would silently halve the quantum to 2.5 ms and double the callback rate for
/// no reason anyone intended; the ask is a LATENCY, and latency is what has to stay constant.
///
/// The desktop-capture site is the only one that takes a negotiated rate. The virtual mic
/// (voice, always 48 kHz) and the pad sinks (DualSense hardware is 48 kHz) keep the constant.
fn capture_quantum_frames(rate_hz: u32) -> u32 {
// Integer maths on both shipping rates: 48 000/48 000 × 240 = 240, 96 000/48 000 × 240 = 480.
// `max(1)` only guards a nonsense rate from producing a zero-frame ask.
((CAPTURE_QUANTUM_FRAMES as u64 * rate_hz as u64 / SAMPLE_RATE as u64) as u32).max(1)
}
/// Callbacks that must agree on a new buffer size before it replaces the one gaps are scored
/// against. Three is enough to reject a boundary artefact and still adopt a genuine re-plan
/// within ~15 ms.
@@ -679,13 +765,16 @@ fn mic_pw_thread(
result
}
#[allow(clippy::too_many_arguments)]
fn pw_thread(
tx: std::sync::mpsc::SyncSender<Vec<f32>>,
quit_rx: pipewire::channel::Receiver<Terminate>,
channels: u32,
rate_hz: u32,
sink_name: Option<String>,
ready: std::sync::mpsc::SyncSender<Result<()>>,
active: Arc<AtomicBool>,
negotiated_rate: Arc<AtomicU32>,
) -> Result<()> {
use pipewire as pw;
use pw::{properties::properties, spa};
@@ -739,6 +828,11 @@ fn pw_thread(
// Which source the negotiated format below actually describes — see the note there.
let sink_mode = sink_name.is_some();
// The `NODE_LATENCY` ask, built at run time because the rate is now a session value:
// `<quantum frames>/<rate>` is how PipeWire spells a latency, and both halves move
// together so the ask stays 5 ms at 48 kHz and at 96 kHz alike. Formatted once here
// rather than at each use so the two property arms cannot drift apart.
let node_latency = format!("{}/{}", capture_quantum_frames(rate_hz), rate_hz);
let props = match &sink_name {
// Stream-sink mode: this stream IS the sink (media.class + Direction::Input). Apps
// play into it, PipeWire mixes them, process() receives the mix. Mirrors the
@@ -751,9 +845,6 @@ fn pw_thread(
*pw::keys::MEDIA_CLASS => "Audio/Sink",
*pw::keys::NODE_DESCRIPTION => "Punktfunk Stream Speaker",
*pw::keys::NODE_VIRTUAL => "true",
// Ask for a ~5ms quantum (= one Opus frame) so buffers arrive smoothly
// rather than in bursts the client's jitter buffer would hear as glitching.
*pw::keys::NODE_LATENCY => "240/48000",
// LOW priority — the opposite of the mic's 3000: between sessions the sink
// node stays alive (parked capturer) but must never win WirePlumber's auto
// default election against real hardware; session routing comes from the
@@ -773,16 +864,24 @@ fn pw_thread(
"session.suspend-timeout-seconds" => "0",
};
p.insert(*pw::keys::NODE_NAME, name.as_str());
// Ask for a ~5 ms quantum (= one protocol audio frame) so buffers arrive
// smoothly rather than in bursts the client's jitter buffer would hear as
// glitching. Inserted rather than written in the `properties!` literal because
// the rate is negotiated — same reason as `NODE_NAME` above.
p.insert(*pw::keys::NODE_LATENCY, node_latency.as_str());
p
}
// Legacy: capture the default sink's monitor (system output), not a microphone.
None => properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CATEGORY => "Capture",
*pw::keys::MEDIA_ROLE => "Music",
*pw::keys::STREAM_CAPTURE_SINK => "true",
*pw::keys::NODE_LATENCY => "240/48000",
},
None => {
let mut p = properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CATEGORY => "Capture",
*pw::keys::MEDIA_ROLE => "Music",
*pw::keys::STREAM_CAPTURE_SINK => "true",
};
p.insert(*pw::keys::NODE_LATENCY, node_latency.as_str());
p
}
};
let stream = pw::stream::StreamBox::new(&core, "punktfunk-audio", props)
.context("pw audio Stream")?;
@@ -827,6 +926,12 @@ fn pw_thread(
/// When the stream last left `Streaming`, so the span can be charged to the window
/// that the span itself stretched. `None` while streaming.
paused_since: Option<std::time::Instant>,
/// The rate every frames↔time conversion below is denominated in. A session value
/// now, not the module constant: at 96 kHz a hardcoded 48 000 would report every
/// quantum as twice its real duration and `delivered_pct` as half of what arrived.
rate_hz: u32,
/// Shared with the capturer — see [`PwAudioCapturer::negotiated_rate`].
negotiated_rate: Arc<AtomicU32>,
}
let ud = CapUd {
tx,
@@ -838,11 +943,13 @@ fn pw_thread(
reported_sched: false,
last_cb: None,
quantum: Duration::from_micros(
CAPTURE_QUANTUM_FRAMES as u64 * 1_000_000 / SAMPLE_RATE as u64,
capture_quantum_frames(rate_hz) as u64 * 1_000_000 / rate_hz as u64,
),
negotiated: None,
active,
paused_since: None,
rate_hz,
negotiated_rate,
};
let _listener = stream
.add_local_listener_with_user_data(ud)
@@ -901,15 +1008,27 @@ fn pw_thread(
return;
}
ud.negotiated = Some(now);
// Report what was GRANTED, not what was asked for
// (`design/hi-res-audio.md` §8.1). Everything downstream — the `Welcome`'s
// resolved rate, the encode loop's samples-per-frame, the client's device
// open — has to follow the same number, and this callback is the only place
// the graph ever states it. A rate of `0` means the pod carried none;
// keeping the previous value is right there, because "unstated" is not a
// claim that the rate changed.
if now.1 != 0 {
ud.rate_hz = now.1;
ud.negotiated_rate.store(now.1, Ordering::Relaxed);
}
// `stream_sink` says WHICH source this format describes, and that changes how
// much it is worth. In stream-sink mode the host owns the sink, so this IS the
// format apps render into and the desktop mix cannot have been narrowed before
// we saw it. In LEGACY monitor mode we are capturing someone else's sink
// through PipeWire's resampler: a 16 kHz Bluetooth headset upstream would
// still be reported here as a clean 48 kHz, exactly the way WASAPI's
// autoconvert hid the same thing on Windows (the 2026-08-03 report). Reading
// the monitored node's OWN rate needs a registry lookup this stream does not
// do — recorded as an open gap rather than implied to be covered.
// autoconvert hid the same thing on Windows (the 2026-08-03 report). So this
// line is a fact about OUR stream and never about the content in legacy mode
// — the monitored node's own rate is a registry lookup, and it lives in
// `monitor_rate`, where the hi-res gate reads it before the `Welcome`.
tracing::info!(
format = ?info.format(),
rate = info.rate(),
@@ -994,11 +1113,11 @@ fn pw_thread(
ud.quantum_candidate = None;
// What a gap is measured against from here on — see `CapUd::quantum`.
ud.quantum = Duration::from_micros(
frames as u64 * 1_000_000 / SAMPLE_RATE as u64,
frames as u64 * 1_000_000 / ud.rate_hz.max(1) as u64,
);
let want = CAPTURE_QUANTUM_FRAMES as usize;
let want = capture_quantum_frames(ud.rate_hz) as usize;
let negotiated_ms =
format!("{:.1}", frames as f32 * 1000.0 / SAMPLE_RATE as f32);
format!("{:.1}", frames as f32 * 1000.0 / ud.rate_hz.max(1) as f32);
if was != 0 {
// A mid-open change. Rare, and worth a line of its own: it moves
// the gap threshold under a reader who is comparing windows.
@@ -1062,7 +1181,7 @@ fn pw_thread(
}
if ud.last_stats.elapsed() >= crate::audio::capture_policy::STATS_EVERY {
let (peak_db, rms_db, delivered_pct) =
ud.stats.summary(ud.last_stats.elapsed(), SAMPLE_RATE);
ud.stats.summary(ud.last_stats.elapsed(), ud.rate_hz);
if ud.stats.dropped_chunks > 0 {
tracing::warn!(
dropped_chunks = ud.stats.dropped_chunks,
@@ -1100,12 +1219,18 @@ fn pw_thread(
.register()
.context("register audio stream listener")?;
// Request F32LE, 48 kHz, at the session's channel count with explicit positions. In
// Request F32LE at the session's rate + channel count with explicit positions. In
// legacy mode PipeWire's channel-mixer up/downmixes the sink monitor to this layout;
// in stream-sink mode this IS the sink's advertised layout (apps mix/route to it).
// in stream-sink mode this IS the sink's advertised layout (apps mix/route to it) —
// which is exactly why hi-res is structurally honest there and has to be PROVEN in
// monitor mode (`design/hi-res-audio.md` §4.4): a sink we OWN renders at the rate we
// declare, while a monitor tap is handed a resampled copy that reports a clean rate
// whatever ran upstream, so that configuration's rate comes from the registry
// (`monitor_rate`) and not from here. What was actually granted comes back through
// `param_changed` above.
let mut info = AudioInfoRaw::new();
info.set_format(AudioFormat::F32LE);
info.set_rate(SAMPLE_RATE);
info.set_rate(rate_hz);
info.set_channels(channels);
info.set_position(spa_positions(channels));
let obj = pw::spa::pod::Object {
crates/punktfunk-host/src/audio/linux/monitor_rate.rs
+369
View File
@@ -0,0 +1,369 @@
//! The **monitored node's own rate**, read from the PipeWire registry — the lookup
//! `design/hi-res-audio.md` §4.4 names as the prerequisite for hi-res in
//! `PUNKTFUNK_STREAM_SINK=0` monitor mode, and §8.3 left as a placeholder.
//!
//! **Why the obvious answer is worthless here.** In monitor mode the host records somebody
//! else's sink *through PipeWire's resampler*, which hands us whatever rate we asked for and
//! reports it back cleanly however narrow the thing upstream really is — the same blindness
//! WASAPI's `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM` has on the Windows capture path (§4.3). So the
//! rate our own capture stream negotiates (what `AudioCapturer::sample_rate` reports) cannot
//! answer §8.4's condition 4, and neither can any amount of listening to it. The question is
//! about the node upstream of that resampler, and that is a registry question, not a stream one.
//!
//! **Two facts, two round trips.** WirePlumber's `default` metadata says WHICH node an untargeted
//! `stream.capture.sink=true` stream gets linked to; only the node itself says what rate it runs
//! at, and the rate is not in the registry's announce props — the same trap
//! `pf_client_core::pad_audio::walk_graph` documents for `audio.channels`, where reading the
//! announce subset looked like it worked and returned zeroes on every real machine. So each
//! candidate sink is bound and the elected one is asked for its negotiated `Format`.
//!
//! **Every failure is a decline, and the asymmetry is the point.** Over-claiming costs a session
//! that says 96 kHz, spends 4.6 Mbps saying it, and carries interpolated 48 kHz — "both ends
//! audit clean, the content is wrong", the class of bug this whole feature exists to prevent.
//! Under-claiming costs Opus 48 kHz, which is the excellent thing every session ships today. So
//! a node that is gone, a key that is unset, a format that was never negotiated and a graph that
//! does not answer in time all resolve the same way: [`super::super::CaptureRate::Unknown`], and
//! the gate declines. **Nothing here ever guesses a rate.**
use anyhow::{anyhow, Context, Result};
use std::time::Duration;
/// The `default` metadata key naming the sink WirePlumber has **elected** — the node an
/// untargeted `stream.capture.sink=true` stream is actually linked to, which is exactly what
/// legacy monitor-mode capture is.
///
/// ⚠ Deliberately NOT [`super::stream_sink`]'s `default.configured.audio.sink`: the neighbouring
/// key, on the same object, and the tempting one to reuse. That one is the user's *preference* —
/// unset on a box whose owner never chose an output, and (as that module's crash self-healing
/// note records) perfectly able to name a node that no longer exists. A preference cannot say
/// what the thing we are about to record is running at; only the elected node can.
const DEFAULT_SINK_KEY: &str = "default.audio.sink";
/// How long the whole round trip may take before it gives up and declines.
///
/// This runs inside the handshake, *before* the `Welcome`: a sick-but-connected graph must cost a
/// connecting client a fallback to Opus, never a stall. Shorter than [`super::stream_sink`]'s 5 s
/// claim timeout on purpose — that one has something to lose by giving up early (host apps keep
/// playing to the previous output for the rest of the session), this one has nothing at all.
const PROBE_TIMEOUT: Duration = Duration::from_secs(3);
/// The `node.name` inside a `default` metadata value (`{"name":"alsa_output.…"}`, typed
/// `Spa:String:JSON` — the same shape [`super::stream_sink`] writes).
///
/// Hand-parsed rather than pulled through a JSON dependency: the value is written by WirePlumber
/// from a fixed template, node names are PipeWire identifiers (`alsa_output.pci-0000_00_1f.3…`,
/// `bluez_output.AA_BB…`) which contain neither quotes nor backslashes, and the cost of this
/// being fooled is `None` → decline, never a wrong rate.
fn sink_name_from_json(value: &str) -> Option<String> {
// Every occurrence, not the first: this is an object, and some other member's VALUE reading
// `"name"` must not shift the parse onto it. Only an occurrence followed by `:` is the key.
for (at, key) in value.match_indices("\"name\"") {
let Some(rest) = value[at + key.len()..].trim_start().strip_prefix(':') else {
continue;
};
let Some(quoted) = rest.trim_start().strip_prefix('"') else {
return None; // `null`, a number, an object — anything but a name.
};
let name = quoted.split_once('"')?.0;
return (!name.is_empty()).then(|| name.to_string());
}
None
}
/// The rate the sink we would monitor is genuinely running at, or an error saying why it is not
/// knowable. See the module docs — an error here is a decline, not a fault.
pub(super) fn monitored_sink_rate() -> Result<u32> {
use pipewire as pw;
use pw::spa::param::audio::AudioInfoRaw;
use pw::spa::param::ParamType;
use std::cell::RefCell;
use std::rc::Rc;
pf_capture::pwinit::ensure_init();
let mainloop = pw::main_loop::MainLoopRc::new(None).context("monitor-rate MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("monitor-rate Context")?;
let core = context
.connect_rc(None)
.context("monitor-rate connect (is PipeWire running in this session?)")?;
let registry = core.get_registry_rc().context("monitor-rate registry")?;
/// Round-trip phases: 0 = globals replaying (binds issued from the callback), 1 = the bound
/// objects replaying their own state, 2 = the elected sink's `Format` being enumerated.
struct Op {
metadata: Option<pw::metadata::Metadata>,
md_listener: Option<pw::metadata::MetadataListener>,
/// `node.name` of the elected default sink, from [`DEFAULT_SINK_KEY`].
elected: Option<String>,
/// Every `Audio/Sink…` node, bound as its global was announced and keyed by the
/// announce `node.name`. Bound EAGERLY because a registry global cannot be bound later:
/// the proxy has to be made while its `GlobalObject` is in hand, and which one we want
/// is not known until the metadata replay of the *next* round. A handful of proxies on
/// any real box, and this path only runs when hi-res is on the table.
sinks: Vec<(String, pw::node::Node, pw::node::NodeListener)>,
/// The rate the elected node's negotiated `Format` reported.
rate: Option<u32>,
phase: u8,
expected: Option<pw::spa::utils::result::AsyncSeq>,
outcome: Option<Result<()>>,
}
let op = Rc::new(RefCell::new(Op {
metadata: None,
md_listener: None,
elected: None,
sinks: Vec::new(),
rate: None,
phase: 0,
expected: None,
outcome: None,
}));
let _registry_listener = registry
.add_listener_local()
.global({
let op = op.clone();
let registry = registry.clone();
move |global| {
let Some(props) = global.props else { return };
match global.type_ {
pw::types::ObjectType::Metadata => {
if op.borrow().metadata.is_some()
|| props.get("metadata.name") != Some("default")
{
return;
}
match registry.bind::<pw::metadata::Metadata, _>(global) {
Ok(md) => {
// The server replays existing properties to a fresh bind, which
// is how the elected sink arrives — there is no getter.
let listener = md
.add_listener_local()
.property({
let op = op.clone();
move |subject, key, _type, value| {
if subject == 0 && key == Some(DEFAULT_SINK_KEY) {
op.borrow_mut().elected =
value.and_then(sink_name_from_json);
}
0
}
})
.register();
let mut o = op.borrow_mut();
o.metadata = Some(md);
o.md_listener = Some(listener);
}
Err(e) => {
op.borrow_mut().outcome =
Some(Err(anyhow!("bind default metadata: {e}")));
}
}
}
pw::types::ObjectType::Node => {
// `media.class` and `node.name` ARE in the announce subset; the rate is
// not, which is what binding buys. `Audio/Sink…` with the ellipsis on
// purpose — `Audio/Sink/Internal` nodes exist and can be elected.
if !props
.get("media.class")
.is_some_and(|c| c.starts_with("Audio/Sink"))
{
return;
}
let Some(name) = props.get("node.name") else {
return;
};
let Ok(node) = registry.bind::<pw::node::Node, _>(global) else {
return;
};
let listener = node
.add_listener_local()
.param({
let op = op.clone();
move |_seq, id, _index, _next, param| {
if id != ParamType::Format {
return;
}
let Some(param) = param else { return };
let mut info = AudioInfoRaw::default();
// A rate of `0`, or a pod that is not audio/raw at all (an
// IEC958/DSD passthrough sink), is not an answer — leave it
// unset and let the caller decline. `parse` reading a
// partially-filled struct is why this is checked rather
// than trusted.
if info.parse(param).is_ok() && info.rate() != 0 {
op.borrow_mut().rate.get_or_insert(info.rate());
}
}
})
.register();
op.borrow_mut()
.sinks
.push((name.to_string(), node, listener));
}
_ => {}
}
}
})
.register();
let _core_listener = core
.add_listener_local()
.done({
let op = op.clone();
let core = core.clone();
let mainloop = mainloop.clone();
move |id, seq| {
if id != pw::core::PW_ID_CORE {
return;
}
let mut o = op.borrow_mut();
if o.expected != Some(seq) || o.outcome.is_some() {
return;
}
match o.phase {
0 => {
// All pre-existing globals replayed, and every bind was issued from
// inside that replay — i.e. AFTER this sync was queued, so nothing they
// provoke has arrived yet. That is what the next round is for.
if o.metadata.is_none() {
o.outcome = Some(Err(anyhow!(
"no 'default' metadata object (is WirePlumber running?)"
)));
mainloop.quit();
return;
}
o.phase = 1;
o.expected = core.sync(0).ok();
}
1 => {
// The binds have replayed: the metadata's properties (so the elected
// sink is known) and the nodes' info. Now ask the ONE node that matters
// for the format it actually negotiated.
let Some(elected) = o.elected.clone() else {
o.outcome = Some(Err(anyhow!(
"'{DEFAULT_SINK_KEY}' is unset — no sink has been elected, so \
there is nothing for a monitor capture to follow"
)));
mainloop.quit();
return;
};
// By index, so no borrow of `o.sinks` is alive across the writes below.
let Some(i) = o.sinks.iter().position(|(n, _, _)| *n == elected) else {
o.outcome = Some(Err(anyhow!(
"the elected default sink '{elected}' is not in the graph"
)));
mainloop.quit();
return;
};
// `Format` — the CONFIGURED one — and never `EnumFormat`, which lists
// what the node *could* be asked for. Reading a capability as if it were
// a fact is precisely the guess this feature exists to refuse.
//
// ⚠ On an adapter node — every ALSA sink is one — `Format` is the
// FOLLOWER's, i.e. the device side, while the monitor ports a legacy
// capture taps sit on the graph side. PipeWire opens the device at the
// graph rate whenever the device can do it, so on any ordinary box the
// two are the same number; they diverge only for a device that cannot
// run the graph's rate at all. Reading LOW then is safe (we decline a
// rate the tap could have carried); reading HIGH is the direction that
// would over-claim, and it needs a device that does 96 kHz on a graph
// that will not — which is also the graph most likely to switch up,
// since the capture stream this gate is deciding for asks for 96 kHz.
// The exactly-right answer is the monitor PORT's own format, one more
// registry hop; named here rather than implied to be covered.
o.sinks[i].1.enum_params(0, Some(ParamType::Format), 0, 1);
o.phase = 2;
o.expected = core.sync(0).ok();
}
_ => {
o.outcome = Some(Ok(()));
mainloop.quit();
}
}
}
})
.error({
let op = op.clone();
let mainloop = mainloop.clone();
move |id, _seq, res, message| {
op.borrow_mut().outcome.get_or_insert(Err(anyhow!(
"pipewire core error id={id} res={res}: {message}"
)));
mainloop.quit();
}
})
.register();
let timer = mainloop.loop_().add_timer({
let op = op.clone();
let mainloop = mainloop.clone();
move |_| {
op.borrow_mut()
.outcome
.get_or_insert(Err(anyhow!("registry round-trip timed out")));
mainloop.quit();
}
});
let _ = timer.update_timer(Some(PROBE_TIMEOUT), None);
op.borrow_mut().expected = core.sync(0).ok();
mainloop.run();
let mut o = op.borrow_mut();
match o.outcome.take() {
// A sink with no negotiated format is a SUSPENDED one: PipeWire closed the device
// because nothing was playing, and the rate it will pick when something does is not a
// fact yet. Decline rather than predict it — the cost is Opus for this session.
Some(Ok(())) => o.rate.take().ok_or_else(|| {
anyhow!("the elected default sink has no negotiated format (it is idle/suspended)")
}),
Some(Err(e)) => Err(e),
None => Err(anyhow!("registry loop exited unexpectedly")),
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The compact shape WirePlumber and [`super::super::stream_sink`] both write, plus a spaced
/// one — nothing in the metadata protocol promises the formatting, so neither does this.
#[test]
fn reads_the_node_name_wireplumber_writes() {
assert_eq!(
sink_name_from_json(r#"{"name":"alsa_output.pci-0000_00_1f.3.analog-stereo"}"#),
Some("alsa_output.pci-0000_00_1f.3.analog-stereo".into())
);
assert_eq!(
sink_name_from_json(r#"{ "name": "punktfunk-speaker-4242-0" }"#),
Some("punktfunk-speaker-4242-0".into())
);
}
/// Anything that is not a name is `None` — which is a DECLINE, not a fallback. A parser that
/// returned something plausible here would hand the gate a node to look up and, if it
/// matched, a rate to believe.
#[test]
fn nothing_parseable_is_never_guessed() {
for v in [
"",
"{}",
r#"{"name":}"#,
r#"{"name":""}"#,
r#"{"name":null}"#,
"alsa_output.pci-0000_00_1f.3.analog-stereo",
r#"{"nickname":"alsa_output.x"}"#,
] {
assert_eq!(sink_name_from_json(v), None, "{v:?} must not parse");
}
}
/// The key is matched with its quotes, so a *value* that merely contains the word cannot
/// hijack the read.
#[test]
fn only_the_name_key_is_read() {
assert_eq!(
sink_name_from_json(r#"{"other":"name","name":"alsa_output.x"}"#),
Some("alsa_output.x".into())
);
}
}
crates/punktfunk-host/src/audio/mic_jitter.rs
+17 -4
View File
@@ -19,10 +19,23 @@ pub(super) enum Deliver {
Conceal,
}
/// Most concealment frames one gap may synthesize (~100 ms at the common 20 ms frames) — the
/// client downlink's `AudioGapTracker` cap, scaled to the uplink's bigger frames. libopus PLC
/// fades to silence after a few frames anyway; past the cap the ring's underrun/re-prime path
/// takes over, as before.
/// Most concealment frames one gap may synthesize. libopus PLC fades to silence after a few
/// frames anyway; past the cap the ring's underrun/re-prime path takes over, as before.
///
/// ⚠ **A COUNT, and the downlink this was copied from no longer has one to be scaled against.**
/// The old wording — "the client downlink's `AudioGapTracker` cap, scaled to the uplink's bigger
/// frames" — described a core that capped at a flat ten packets; core now states that bound in
/// TIME (`MAX_CONCEAL_MS` = 50 ms) and derives the packet count from the frame length each session
/// actually resolved, precisely because a fixed count silently retunes when the frame does. Same
/// defect family as the drought fuse and the near-miss margin.
///
/// This constant is not the same number by another name, and never was: five frames is ~100 ms at
/// the uplink's common 20 ms Opus frame — twice the downlink's bound, not a scaling of it. It is
/// left as a count because the uplink cannot be told its frame length: the client encodes it and
/// nothing announces it, so [`MicDejitter`] can only *measure* it, from consecutive frames' pts
/// deltas (`frame_ms`). That measurement is the material a time-stated cap would be derived from
/// if this is ever retuned — and the reason to retune it is a client that drops to 10 ms frames,
/// where the same five frames become 50 ms of cover instead of 100.
const MAX_CONCEAL_FRAMES: u32 = 5;
/// How long one out-of-order frame may wait for its missing predecessor before the gap is
crates/punktfunk-host/src/audio/windows/audio_control.rs
+74
View File
@@ -86,6 +86,80 @@ pub(crate) fn mix_format_of(ep: &Endpoint) -> Option<MixFormat> {
})
}
/// §8.4 condition 4 on Windows (`design/hi-res-audio.md` §4.3 / §8.2) — the ENGINE rate the
/// desktop-audio loopback would really capture at, answered before the `Welcome` and without
/// opening a capture stream.
///
/// The rule §8.2 states is `requested > engine.rate → decline, never pad`, and the number it
/// turns on is [`mix_format_of`]'s: a shared-mode `IAudioClient` opened with
/// `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM` reconciles our request with the engine's mix format in
/// whichever direction is needed, so asking a 48 kHz engine for 96 kHz succeeds, returns no
/// error, and hands back interpolation. An operator who genuinely wants 96 kHz sets the
/// endpoint's own rate in Windows' device properties; the host then sees it here and honours it.
///
/// **Read-only, deliberately.** This runs mid-handshake, before any session decision has been
/// taken, so it must leave the box exactly as it found it: it enumerates, runs the PURE
/// [`plan_with_formats`], and reads one mix format. It is NOT [`wire_now_full`], which parks the
/// operator's default devices through `IPolicyConfig`, mints endpoints and logs a plan — none of
/// which may happen for a session that is about to resolve to Opus anyway.
///
/// ⚠ The plan inputs below MIRROR [`wire_now_full`]'s, and must keep mirroring them: the whole
/// point is to name the endpoint the capture open will pick a moment later, so a divergence here
/// reads the format of a device we do not end up capturing. They are all process-stable env/state
/// reads, which is what makes recomputing them safe. The real probe (not [`wiring_plan::no_formats`])
/// is load-bearing for the same reason — narrowing DEMOTES a candidate below real hardware, so a
/// format-blind plan can name a different endpoint than the session's.
///
/// Every failure is [`CaptureRate::Unknown`](super::CaptureRate::Unknown) — i.e. decline. Unlike
/// the wiring plan, where an unknown format means "assume it is fine", an unknown format here
/// means "we cannot prove the content matches the label", and this feature exists to refuse
/// exactly that.
///
/// Must run on a COM-initialized thread; it initializes MTA itself because its caller is a tokio
/// blocking-pool thread that has no COM state of its own. A repeat init on a reused pool thread
/// returns `S_FALSE`, which is a success — the same pattern every WASAPI worker here uses.
pub(crate) fn probe_capture_rate() -> super::CaptureRate {
if let Err(e) = wasapi::initialize_mta().ok() {
tracing::debug!(error = %e, "hi-res capture-rate probe: CoInitializeEx (MTA) failed");
return super::CaptureRate::Unknown;
}
let renders = list_endpoints(Direction::Render);
let captures = list_endpoints(Direction::Capture);
let want = std::env::var("PUNKTFUNK_MIC_DEVICE")
.ok()
.map(|s| s.to_lowercase());
let pad_ids = pad_render_ids(&renders);
let wiring = plan_with_formats(
&renders,
&captures,
want.as_deref(),
host_audio_requested(),
&mix_format_of,
// Stereo — the only count hi-res carries at all (§3), and the same floor `wire_now_full`
// plans against.
2,
&pad_ids,
&super::minted::minted_ids(),
);
let Some(ep) = wiring.loopback_render else {
tracing::debug!("hi-res capture-rate probe: no desktop-audio loopback endpoint is planned");
return super::CaptureRate::Unknown;
};
// One more activation of the endpoint the plan just chose. `plan_with_formats` asked for this
// format too, but only to rank candidates — it returns the ranking, not the numbers — and
// threading a cache through it to save one `GetMixFormat` on an opt-in path would buy nothing
// but a second way for the two to disagree.
match mix_format_of(&ep) {
Some(f) => super::CaptureRate::Engine(f.rate_hz),
None => {
tracing::debug!(device = %ep.0,
"hi-res capture-rate probe: the planned loopback endpoint would not report its \
mix format");
super::CaptureRate::Unknown
}
}
}
/// `(friendly_name, endpoint_id)` for every ACTIVE endpoint in direction `dir`.
fn list_endpoints(dir: Direction) -> Vec<Endpoint> {
let mut out = Vec::new();
crates/punktfunk-host/src/audio/windows/wasapi_cap.rs
+149 -19
View File
@@ -1,7 +1,9 @@
//! WASAPI loopback capture of the desktop mix (system output) — the Windows analogue of the
//! PipeWire sink-monitor backend. Delivers interleaved f32 PCM at 48 kHz in the requested
//! channel count (stereo / 5.1 / 7.1, canonical wire order FL FR FC LFE RL RR SL SR via the
//! explicit `dwChannelMask`), ready for the Opus path with NO resampling (WASAPI shared-mode
//! PipeWire sink-monitor backend. Delivers interleaved f32 PCM at the requested rate (48 kHz
//! unless a hi-res session negotiated more, and only ever as high as the endpoint's own engine
//! genuinely runs — see [`WasapiLoopbackCapturer::opened_rate`]) in the requested channel count
//! (stereo / 5.1 / 7.1, canonical wire order FL FR FC LFE RL RR SL SR via the explicit
//! `dwChannelMask`), ready for the encode path with NO resampling in Rust (WASAPI shared-mode
//! autoconvert does any SRC + up/downmix to the requested layout). WASAPI objects are
//! COM-apartment-bound and not `Send`, so they live on a dedicated thread (mirrors
//! `linux::PwAudioCapturer`); only the channel + stop flag + join handle are in the struct.
@@ -31,7 +33,7 @@ use super::capture_policy::{CaptureStats, FightDamper, FIGHT_BACKOFF, STATS_EVER
use super::{audio_control, wiring_plan, AudioCapturer, SAMPLE_RATE};
use anyhow::{anyhow, Context, Result};
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError, SyncSender};
use std::sync::{Arc, Mutex};
use std::thread::{self, JoinHandle};
@@ -52,14 +54,25 @@ pub struct WasapiLoopbackCapturer {
/// Linux twin by a 2026-08-13 field log (100 % drop rate across session gaps); the parking
/// call sites are platform-independent, so this half had the same defect.
active: Arc<AtomicBool>,
/// The rate the endpoint was ACTUALLY opened at, written by the capture thread before it
/// reports ready and read back by [`AudioCapturer::sample_rate`].
///
/// This is the whole Windows half of `design/hi-res-audio.md`: in shared mode the engine's
/// mix format is authoritative and `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM` reconciles our
/// request with it silently, in either direction. Asking a 48 kHz engine for 96 kHz
/// therefore SUCCEEDS and hands back interpolated samples (§4.3). So the open declines
/// instead of padding, and states here what it settled for — the caller compares it against
/// what it promised the client and resolves the plane from the answer (§8.2).
opened_rate: Arc<AtomicU32>,
}
impl WasapiLoopbackCapturer {
pub fn open(channels: u32) -> Result<WasapiLoopbackCapturer> {
pub fn open(channels: u32, rate_hz: u32) -> Result<WasapiLoopbackCapturer> {
anyhow::ensure!(
matches!(channels, 2 | 6 | 8),
"WASAPI loopback backend supports 2/6/8 channels (got {channels})"
);
anyhow::ensure!(rate_hz > 0, "audio capture rate must be positive");
let (tx, rx) = sync_channel::<Vec<f32>>(64);
let stop = Arc::new(AtomicBool::new(false));
// Bring-up handshake: report open success/failure before returning, so a missing render
@@ -70,10 +83,22 @@ impl WasapiLoopbackCapturer {
// Opens at session start, so the consumer is live from the first chunk.
let active = Arc::new(AtomicBool::new(true));
let active_t = active.clone();
// Seeded with the request: it is the honest answer until the endpoint has been read, and
// on the overwhelmingly common 48 kHz path it is also the final one.
let opened_rate = Arc::new(AtomicU32::new(rate_hz));
let opened_rate_t = opened_rate.clone();
let join = thread::Builder::new()
.name("punktfunk-wasapi-audio".into())
.spawn(move || {
if let Err(e) = capture_thread(tx, stop_t, ready_tx, channels, active_t) {
if let Err(e) = capture_thread(
tx,
stop_t,
ready_tx,
channels,
rate_hz,
active_t,
opened_rate_t,
) {
tracing::error!(error = %format!("{e:#}"), "wasapi loopback thread failed");
}
})
@@ -82,13 +107,21 @@ impl WasapiLoopbackCapturer {
// driver installs, ~5 s of settling each) before the endpoint exists.
match ready_rx.recv_timeout(Duration::from_secs(30)) {
Ok(Ok(())) => {
tracing::info!(channels, "WASAPI loopback capture: 48 kHz f32");
// The rate the thread SETTLED on, which is not necessarily `rate_hz` — see
// `opened_rate`. Reported here rather than as a fixed "48 kHz" string so a log
// can never say one rate while the stream carries another.
tracing::info!(
channels,
rate_hz = opened_rate.load(Ordering::Relaxed),
"WASAPI loopback capture: f32"
);
Ok(WasapiLoopbackCapturer {
chunks: rx,
channels,
stop,
join: Some(join),
active,
opened_rate,
})
}
Ok(Err(e)) => Err(e),
@@ -131,6 +164,9 @@ impl AudioCapturer for WasapiLoopbackCapturer {
fn channels(&self) -> u32 {
self.channels
}
fn sample_rate(&self) -> u32 {
self.opened_rate.load(Ordering::Relaxed)
}
fn drain(&mut self) {
while self.chunks.try_recv().is_ok() {}
// Ordered AFTER the backlog drain, so the capture thread never counts a drop against a
@@ -201,7 +237,9 @@ fn capture_thread(
stop: Arc<AtomicBool>,
ready: SyncSender<Result<()>>,
channels: u32,
rate_hz: u32,
active: Arc<AtomicBool>,
opened_rate: Arc<AtomicU32>,
) -> Result<()> {
// COM must be initialized on THIS thread (MTA), before any device call.
if let Err(e) = wasapi::initialize_mta()
@@ -227,7 +265,16 @@ fn capture_thread(
// is said once per topology — the field log drowned in 256+ copies of the same line.
let mut unsat_logged: Option<u64> = None;
while !stop.load(Ordering::Relaxed) {
match capture_once(&tx, &stop, &mut ready, channels, mode, &active) {
match capture_once(
&tx,
&stop,
&mut ready,
channels,
rate_hz,
mode,
&active,
&opened_rate,
) {
Ok(Next::Stopped) => break,
Ok(Next::Reopen(m)) => {
mode = m;
@@ -390,13 +437,16 @@ fn default_render(en: &DeviceEnumerator) -> Option<(Device, String)> {
/// One endpoint open + capture loop. Returns how to continue ([`Next`]) or an error (first open:
/// retried [`FIRST_OPEN_ATTEMPTS`] times, then fatal via the `ready` handshake; later: reopen
/// with capped backoff — or, for a typed [`PlanUnsatisfiable`], an endpoint-set wait).
#[allow(clippy::too_many_arguments)]
fn capture_once(
tx: &SyncSender<Vec<f32>>,
stop: &AtomicBool,
ready: &mut Option<SyncSender<Result<()>>>,
channels: u32,
rate_hz: u32,
mode: TargetMode,
active: &AtomicBool,
opened_rate: &AtomicU32,
) -> Result<Next> {
// Interleaved f32: channels * 4 bytes per frame.
let block_align = channels as usize * 4;
@@ -502,7 +552,85 @@ fn capture_once(
};
let mut audio_client = device.get_iaudioclient().context("IAudioClient")?;
// 48 kHz f32 interleaved in the requested channel layout; autoconvert lets WASAPI's
// WP0.1 — the endpoint's ACTUAL engine mix format, read BEFORE we initialize. Everything the
// old log printed ("48 kHz f32 channels=2") was our REQUEST; with `autoconvert` WASAPI
// silently converts from whatever the endpoint really runs, so a voice-carrier endpoint
// narrowing the desktop mix to mono or 24 kHz was invisible in a 3,600-line field log. This
// line is what makes an audio-quality report triageable without a round trip.
let engine = audio_client.get_mixformat().ok();
// …and, since the hi-res plane exists, this reading is no longer merely diagnostic — it
// DECIDES the rate we open at (`design/hi-res-audio.md` §4.3/§8.2).
//
// In shared mode the engine's mix format is authoritative, and
// `AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM` exists to reconcile our format with it in whichever
// direction is needed. So asking a 48 kHz engine for 96 kHz does not fail: it succeeds,
// returns no error, and hands back interpolated samples with nothing above 24 kHz in them —
// and the session would then advertise 96 000 in its `Welcome`, spend 3–4 Mbps, and be
// wrong in exactly the way the HDR RB-swap was wrong (both ends audit clean, the content is
// a lie). NEVER pad: open at the engine's own rate and say so, so the caller can decline
// hi-res rather than ship interpolation labelled as detail.
//
// Only an UPWARD request is refused. Asking for LESS than the engine runs at is an ordinary
// downsample — the legacy 48 kHz behaviour on a 96 kHz endpoint, which is what this host has
// always done and is lossy only in the way every previous release already was.
//
// ⚠⚠ …and the floor is the LEGACY rate, never the engine's — `hz.max(SAMPLE_RATE)`. That
// `max` serves the **OPUS** path and nothing else, and getting it backwards regresses every
// ordinary Windows box. An endpoint configured at 44 100 Hz is an unremarkable Windows
// configuration; libopus accepts 8/12/16/24/48 kHz only (RFC 6716), so the `0xC9` plane is
// 48 kHz by definition, and this host has always asked such an endpoint for 48 kHz and let
// autoconvert upsample. Settling down to 44 100 for an Opus session would hand libopus a rate
// it refuses — breaking the plane that works today to protect one that is not even running.
//
// What the 44.1 kHz family being admitted to the **PCM** plane changes is only which requests
// can reach here: a hi-res session may now legitimately ask for 44 100, and it settles at
// 44 100 rather than being floored to 48 000 — because the `max` is inside the `rate_hz >`
// test, so a request AT or BELOW the floor never trips this arm at all. The two rules read as
// one line and are genuinely two:
//
// - `rate_hz > hz` — the honesty rule, for either plane: never advertise a rate the engine
// cannot produce. This is what declines 96 kHz on a 48 kHz engine.
// - `.max(SAMPLE_RATE)` — the Opus floor, for the `0xC9` plane: a 48 kHz open on a slower
// engine is the legacy upsample every release has already shipped, and stays.
//
// So a HI-RES request loses here only when it is above the engine; the 48 kHz baseline claim
// every session makes is untouched, and a 44.1 kHz hi-res request is not a loss at all.
//
// The operator's lever is Windows' own device properties: set the endpoint to 96 kHz there
// and the host sees it here and honours it. Driving the engine format FROM the host would
// fight the OS and every other application on the box.
let engine_hz = engine.as_ref().map(|f| f.get_samplespersec());
let open_hz = match engine_hz {
Some(hz) if hz > 0 && rate_hz > hz.max(SAMPLE_RATE) => {
let settled = hz.max(SAMPLE_RATE);
tracing::info!(
device = %dev_name,
engine_hz = hz,
requested = rate_hz,
opening_at = settled,
"engine rate is below the requested capture rate — hi-res declined; opening at \
the engine rate rather than letting WASAPI autoconvert upsample it (set this \
endpoint's rate in Windows' device properties to raise it)"
);
settled
}
// No mix format readable: the endpoint is answering nothing about itself, so the only
// defensible rate is the legacy one. Declining here costs a hi-res session and can
// never cost a working 48 kHz one.
None if rate_hz != SAMPLE_RATE => {
tracing::info!(
device = %dev_name,
requested = rate_hz,
"endpoint mix format unreadable — hi-res declined; opening at the legacy rate"
);
SAMPLE_RATE
}
_ => rate_hz,
};
// Published BEFORE the initialize can fail, because it is the answer to "what did we settle
// for" and that answer is already decided. `sample_rate()` reads it back.
opened_rate.store(open_hz, Ordering::Relaxed);
// f32 interleaved at `open_hz` in the requested channel layout; autoconvert lets WASAPI's
// shared-mode SRC match the engine mix format to ours (incl. up/downmix to the requested
// channel count), so we never resample/remix in Rust. The explicit dwChannelMask pins the
// wire order (FL FR FC LFE RL RR SL SR; 7.1 = 0x63F, not 0xFF). Loopback is implied by
@@ -512,16 +640,10 @@ fn capture_once(
32,
32,
&SampleType::Float,
SAMPLE_RATE as usize,
open_hz as usize,
channels as usize,
Some(mask),
);
// WP0.1 — the endpoint's ACTUAL engine mix format, read BEFORE we initialize. Everything the
// old log printed ("48 kHz f32 channels=2") was our REQUEST; with `autoconvert` WASAPI
// silently converts from whatever the endpoint really runs, so a voice-carrier endpoint
// narrowing the desktop mix to mono or 24 kHz was invisible in a 3,600-line field log. This
// line is what makes an audio-quality report triageable without a round trip.
let engine = audio_client.get_mixformat().ok();
// NB the plan's WP4.5 ("open the loopback at the MINIMUM device period, worth ~5–10 ms") is
// deliberately NOT done here, because its premise is wrong: in shared mode
// `IAudioClient::Initialize` cannot change the engine period at all — `hnsBufferDuration` sizes
@@ -551,6 +673,11 @@ fn capture_once(
tracing::info!(device = %dev_name,
follow = matches!(mode, TargetMode::Follow) || keep_default,
last_resort,
// What we asked for, and what we settled on — the two differ only when the engine
// refused an upward request (see the decline above), and a reader has to be able to
// see which happened without inferring it.
requested_hz = rate_hz,
opened_hz = open_hz,
// The endpoint's own format — NOT the one we asked for.
engine_hz = engine.as_ref().map(|f| f.get_samplespersec()),
engine_ch = engine.as_ref().map(|f| f.get_nchannels()),
@@ -654,7 +781,7 @@ fn capture_once(
let lost = info.index.saturating_sub(next_index);
stats.max_gap_us = stats
.max_gap_us
.max(lost.saturating_mul(1_000_000) / SAMPLE_RATE as u64);
.max(lost.saturating_mul(1_000_000) / open_hz.max(1) as u64);
}
next_index = info.index.saturating_add(frames);
last_packet = Some(now);
@@ -698,7 +825,7 @@ fn capture_once(
}
}
if last_stats.elapsed() >= STATS_EVERY {
let (peak_db, rms_db, delivered_pct) = stats.summary(last_stats.elapsed(), SAMPLE_RATE);
let (peak_db, rms_db, delivered_pct) = stats.summary(last_stats.elapsed(), open_hz);
if stats.dropped_chunks > 0 {
tracing::warn!(
device = %dev_name,
@@ -874,7 +1001,7 @@ mod tests {
if std::env::var("PUNKTFUNK_WASAPI_LIVE").is_err() {
return;
}
let mut cap = match WasapiLoopbackCapturer::open(2) {
let mut cap = match WasapiLoopbackCapturer::open(2, SAMPLE_RATE) {
Ok(c) => c,
Err(e) => {
eprintln!("no render endpoint on this box ({e:#}) — skipping");
@@ -882,6 +1009,9 @@ mod tests {
}
};
assert_eq!(cap.channels(), 2);
// Asking for the legacy rate can never be declined — no engine runs below it in a way
// that would make 48 kHz an upward request — so the settled rate must be what we asked.
assert_eq!(cap.sample_rate(), SAMPLE_RATE);
match cap.next_chunk() {
Ok(samples) => assert!(
samples.len() % 2 == 0,
crates/punktfunk-host/src/gamestream/audio.rs
+10 -3
View File
@@ -304,6 +304,11 @@ fn run(
// Reuse the persistent capturer when its channel count still matches (drain stale
// buffered audio); otherwise drop it (clean PipeWire teardown) and open at the new count.
let want = layout_for(&params).channels as u32;
// Always [`SAMPLE_RATE`], and never a negotiated one: this is Moonlight's protocol, whose
// audio stream is Opus at 48 kHz by definition (moonlight-common-c has no rate field to
// carry anything else, and libopus tops out at 48 kHz anyway). The hi-res `0xD3` plane is
// native-only for exactly that reason — `design/hi-res-audio.md` §3 lists this plane as
// permanently out of scope rather than merely deferred.
let mut cap = match audio_cap.lock().unwrap().take() {
Some(mut c) if c.channels() == want => {
c.drain();
@@ -316,9 +321,9 @@ fn run(
"audio capturer channel count changed — reopening"
);
drop(c);
audio::open_audio_capture(want).context("open audio capture")?
audio::open_audio_capture(want, SAMPLE_RATE).context("open audio capture")?
}
None => audio::open_audio_capture(want).context("open audio capture")?,
None => audio::open_audio_capture(want, SAMPLE_RATE).context("open audio capture")?,
};
let result = audio_body(&mut *cap, &sock, gcm_key, rikeyid, params, running, on_lost);
cap.idle(); // parked between sessions — release the routing claim (Linux stream sink)
@@ -724,7 +729,9 @@ mod tests {
#[test]
#[ignore]
fn surround_capture_live() {
let mut cap = crate::audio::open_audio_capture(6).expect("open 6ch capture");
// 48 kHz, like every other GameStream capture site — Moonlight's protocol (see the
// comment at the live open in `audio_stream`).
let mut cap = crate::audio::open_audio_capture(6, SAMPLE_RATE).expect("open 6ch capture");
let layout = &LAYOUT_51;
let mut enc = opus::MSEncoder::new(
SAMPLE_RATE,
crates/punktfunk-host/src/native.rs
+13 -1
View File
@@ -1809,9 +1809,16 @@ async fn serve_session(
welcome.bitrate_kbps,
channels,
);
// …and the resolved audio FORMAT read back the same way, for the same reason. The
// client opens its output device from these four Welcome fields, so the capture rate,
// the samples-per-frame and the wire tag the encode loop uses have to come from the
// identical bytes rather than from a second evaluation of the §8.4 gate — which reads
// process configuration and a live connection property, neither of which is guaranteed
// to answer the same way twice.
let audio_plane = handshake::AudioPlane::from_welcome(&welcome);
std::thread::Builder::new()
.name("punktfunk1-audio".into())
.spawn(move || audio_thread(conn, stop, cap, channels, budget))
.spawn(move || audio_thread(conn, stop, cap, channels, budget, audio_plane))
.map_err(|e| tracing::warn!(error = %e, "audio thread spawn failed — session continues without audio"))
.ok()
} else {
@@ -3328,6 +3335,11 @@ mod tests {
display_hdr: None,
client_caps: 0,
max_shard_payload: 0,
// Legacy audio request — this fixture exercises the per-client access grants, not
// the audio plane, and the defaults keep its Hello byte-identical to a pre-hi-res
// client's.
audio_rate_hz: punktfunk_core::audio::SAMPLE_RATE_HZ,
audio_bits: punktfunk_core::audio::pcm::BITS_16,
};
io::write_msg(&mut send, &hello.encode())
.await
crates/punktfunk-host/src/native/audio.rs
+574 -75
View File
@@ -1,9 +1,19 @@
//! The native audio plane (plan §W1 — carved out of the [`super`] module): desktop capture → Opus
//! (48 kHz, 5 ms, constrained VBR at the configured [`AudioTier`](punktfunk_core::audio::AudioTier))
//! → `AUDIO_MAGIC` QUIC datagrams — or `AUDIO_RED_MAGIC` when the session negotiated redundancy —
//! at the negotiated channel count. The encoder ([`NativeAudioEnc`]) and the capture/encode/send
//! loop ([`audio_thread`]) are gated to linux/windows (libopus + a real capturer); other targets
//! get the stub, so a dev build streams video-only rather than failing to compile.
//! The native audio plane (plan §W1 — carved out of the [`super`] module): desktop capture →
//! either
//!
//! - **Opus** (48 kHz, 5 ms, constrained VBR at the configured
//! [`AudioTier`](punktfunk_core::audio::AudioTier)) → `AUDIO_MAGIC` QUIC datagrams, or
//! `AUDIO_RED_MAGIC` when the session negotiated redundancy — the default and the fallback; or
//! - **lossless PCM** (both rate families — 44.1/48/88.2/96/176.4 kHz — 16/24-bit, any negotiated
//! channel count, a negotiated frame duration) → `AUDIO_PCM_MAGIC` datagrams, when the
//! handshake's §8.4 gate resolved the hi-res plane (`design/hi-res-audio.md`).
//!
//! …at the negotiated channel count. Which plane a session runs is decided ONCE, at handshake,
//! and never switches underneath the client: its output device is open at a fixed rate, so a
//! change would mean a re-open (§6). The encoder ([`NativeAudioEnc`]) and the
//! capture/encode/send loop ([`audio_thread`]) are gated to linux/windows (libopus + a real
//! capturer); other targets get the stub, so a dev build streams video-only rather than failing
//! to compile.
//!
//! Two things here deliberately DIVERGE from the GameStream plane, which used to share this
//! tuning: hard CBR (its audio FEC needs fixed-size packets; this plane has no FEC, so CBR was a
@@ -11,6 +21,90 @@
use super::*;
#[cfg(any(target_os = "linux", target_os = "windows"))]
use punktfunk_core::audio::pcm;
/// The audio plane's wire clock — the `pts_ns` every datagram is stamped with.
///
/// An **anchor plus a running total of interleaved samples**, not an accumulator advanced once per
/// frame. That shape is the whole point:
///
/// ⚠⚠ **The negotiated `frame_us` is a LABEL, not a duration.** A frame carries a whole number of
/// samples PER CHANNEL, and `pcm::samples_per_frame` floors — so a rung is a real duration only
/// when the rate divides it. Every rung divides the 48 kHz family; **none of the seven divides
/// 44 100 Hz**, 88 200 divides only 5 000 µs and 176 400 only 5 000 and 2 500. A "5 ms" frame at
/// 44 100 Hz is 220 samples per channel — `frame_duration_ns` of it is 4 988 662 ns, not
/// 5 000 000. Stamping `pts += frame_us * 1000` therefore runs the clock **2 268 ppm fast**: 2.3
/// ms of invented time every second, for the life of the session.
///
/// ⚠ And it is not self-correcting. [`reanchor`](Self::reanchor) only ever moves the clock
/// FORWARD (a capture arrival must not un-send frames already on the wire), so a clock running
/// slow is pulled up by the next chunk while a clock running fast is never pulled back by
/// anything. The client's A/V sync loop then chases a drift manufactured at the source, and every
/// stat agrees with it, because the timestamps are self-consistent and simply wrong.
///
/// Counting samples cannot drift: the sample count IS the frame. A running total beats even a sum
/// of per-frame `frame_duration_ns` values — that sum accumulates just under 1 ns per frame
/// (~0.2 µs/s, irrelevant next to 2.3 ms/s, but not nothing), while this accumulates zero.
#[cfg(any(target_os = "linux", target_os = "windows"))]
struct PtsClock {
/// Wall-clock nanoseconds the current run of samples is measured from.
base_ns: u64,
/// Interleaved samples charged to the clock since [`base_ns`](Self::base_ns).
samples: usize,
rate_hz: u32,
channels: u8,
/// Interleaved samples in one second — the fold factor [`advance`](Self::advance) uses to keep
/// `samples` bounded. Precomputed: it is a property of the plane, not of a frame.
samples_per_sec: usize,
}
#[cfg(any(target_os = "linux", target_os = "windows"))]
impl PtsClock {
fn new(rate_hz: u32, channels: u8) -> PtsClock {
PtsClock {
base_ns: 0,
samples: 0,
rate_hz,
channels,
samples_per_sec: rate_hz as usize * channels as usize,
}
}
/// The pts of the NEXT frame to leave, real or synthesized.
fn pts_ns(&self) -> u64 {
self.base_ns + pcm::frame_duration_ns(self.samples, self.rate_hz, self.channels)
}
/// Charge one frame of `samples` interleaved samples: it has left the host.
fn advance(&mut self, samples: usize) {
self.samples += samples;
// Fold whole seconds into the base so a long session cannot grow the total without bound
// (`frame_duration_ns` takes a `usize`, which is 32 bits on some targets, and 176 400 Hz
// 7.1 is 1.4 M samples a second). EXACT, so the fold is invisible to `pts_ns`:
// `rate_hz × channels` interleaved samples are 1 000 000 000 ns at every rate this plane
// carries, with no remainder to round away.
if self.samples_per_sec > 0 && self.samples >= self.samples_per_sec {
let secs = self.samples / self.samples_per_sec;
self.base_ns += secs as u64 * 1_000_000_000;
self.samples -= secs * self.samples_per_sec;
}
}
/// Re-anchor on a capture arrival — **forward only**.
///
/// Infilled frames advanced the wire clock while capture was away, and an anchor re-derived
/// from a chunk's arrival can land at or before the last frame already sent; moving back would
/// re-issue a pts the client has already played. Re-anchoring restarts the running total,
/// because the new base already describes everything the old base plus its samples did.
fn reanchor(&mut self, anchor_ns: u64) {
if anchor_ns > self.pts_ns() {
self.base_ns = anchor_ns;
self.samples = 0;
}
}
}
/// Opus encoder for the native audio plane: a plain stereo encoder (the live-validated,
/// byte-identical path) or a libopus *multistream* encoder for 5.1/7.1, both behind one
/// `encode_float`. Surround uses the safe `opus::MSEncoder` (no `audiopus_sys`).
@@ -74,10 +168,14 @@ impl NativeAudioEnc {
}
}
/// The audio thread: desktop capture → Opus (48 kHz, 5 ms, constrained VBR at the configured
/// tier) → `AUDIO_MAGIC` (or `AUDIO_RED_MAGIC`) datagrams, at the negotiated `channels` (2 stereo / 6 = 5.1 / 8 = 7.1,
/// canonical wire order FL FR FC LFE RL RR SL SR). QUIC already encrypts; no extra layer. The
/// capturer comes from (and returns to) the persistent slot — see [`AudioCapSlot`].
/// The audio thread: desktop capture → the session's resolved audio plane (Opus on
/// `AUDIO_MAGIC`/`AUDIO_RED_MAGIC`, or lossless PCM on `AUDIO_PCM_MAGIC`) at the negotiated
/// `channels` (2 stereo / 6 = 5.1 / 8 = 7.1, canonical wire order FL FR FC LFE RL RR SL SR).
/// QUIC already encrypts; no extra layer. The capturer comes from (and returns to) the persistent
/// slot — see [`AudioCapSlot`].
///
/// `plane` is the format the `Welcome` states — read back off it by the caller, never recomputed,
/// so the wire the client was promised and the wire we send cannot disagree.
#[cfg(any(target_os = "linux", target_os = "windows"))]
pub(super) fn audio_thread(
conn: quinn::Connection,
@@ -85,12 +183,10 @@ pub(super) fn audio_thread(
audio_cap: AudioCapSlot,
channels: u8,
budget: punktfunk_core::audio::AudioBudget,
plane: super::handshake::AudioPlane,
) {
use crate::audio::SAMPLE_RATE;
const FRAME_MS: usize = 5;
const SAMPLES_PER_FRAME: usize = SAMPLE_RATE as usize * FRAME_MS / 1000; // 240
/// One protocol frame of wall time — the cadence paced sends aim for.
const FRAME_INTERVAL: std::time::Duration = std::time::Duration::from_millis(FRAME_MS as u64);
/// Ceiling on a single pacing sleep. The capture channel is finite and `next_chunk` has to be
/// serviced; sleeping past a couple of frames would trade a burst on the wire for a drop at
/// the capturer, which is strictly worse (a drop is a click AND a permanent shift).
@@ -100,6 +196,55 @@ pub(super) fn audio_thread(
/// pacing exists to prevent — so past this point the debt is forgiven, not repaid.
const PACE_REANCHOR: std::time::Duration = std::time::Duration::from_millis(100);
let want = punktfunk_core::audio::normalize_channels(channels);
// The three session values that used to be compile-time constants. Every one of them is now
// a property of the negotiated plane, and everything downstream — the pacer, the sample
// clock, the capture open, the scratch buffers — is derived from these rather than from
// `SAMPLE_RATE`/`FRAME_MS` directly.
let pcm_plane = plane.is_pcm();
let rate_hz = if pcm_plane {
plane.rate_hz
} else {
SAMPLE_RATE
};
let bits = plane.bits;
// Opus is the fixed 5 ms of `0xC9`; the PCM plane's duration was negotiated from the
// connection's datagram size. The `max(1)` is a floor against a malformed plane rather than a
// real case — the §8.4 gate never produces a PCM plane with a zero frame — but a zero here
// would divide the pacer by nothing and produce empty frames at infinite rate.
let frame_us: u32 = if pcm_plane {
(plane.frame_us as u32).max(1)
} else {
FRAME_MS as u32 * 1000
};
// Interleaved samples in one protocol frame, derived from the negotiated rate and frame
// duration rather than from a constant. THE single source of truth for how long a frame is —
// the client's ring drains exactly this many, so both ends agree by construction rather than
// by re-deriving `rate × µs` and hoping they round the same way (`pcm::samples_per_frame`).
//
// ⚠ Exact only on the 48 kHz family. Every ladder rung divides 48 000 and 96 000 into whole
// samples per channel; **none of them divides 44 100**, and 88 200/176 400 divide only the
// longest one or two. On those rates this FLOORS, so a frame is up to one sample per channel
// shorter than the rung it is labelled with — safe for sizing (the payload can only shrink
// inside its datagram) and wrong for timing, which is why the pts below is stamped from
// `frame_duration_ns` and not from `frame_us`.
let frame_len = pcm::samples_per_frame(rate_hz, frame_us, want);
// One protocol frame of wall time — the cadence paced sends aim for. A `let`, not a const:
// the PCM plane's frames are shorter than 5 ms whenever the format does not fit a datagram
// at that length (§4.2), and a 96/24 session paces 500 of them a second.
//
// Measured from the frame's REAL sample count for the same reason the pts is, even though the
// consequence here is far smaller: this only decides when the loop next looks for work, and
// every release is additionally gated on `acc` actually holding a frame, so a rung-length
// interval would produce a slot the pacer waits out rather than time it invents. Cosmetic or
// not, two clocks describing the same frame must not disagree — a 0.23 % gap between them is
// exactly the kind of thing a later reader reconciles in the wrong direction. On the 48 kHz
// family this is bit-identical to `from_micros(frame_us)`.
//
// The `max(1)` guards the same malformed-plane case `frame_us` does above: a sub-microsecond
// rung would floor `frame_len` to zero samples and leave the pacer with a zero interval to
// spin on. The §8.4 gate never produces one (the shortest ladder rung is 1 000 µs).
let frame_interval =
std::time::Duration::from_nanos(pcm::frame_duration_ns(frame_len, rate_hz, want).max(1));
// Same boost the video capture/encode loop takes, and this thread needs it MORE: it paces
// 5 ms datagrams, so a scheduling stall here is directly audible where a late video frame
// is one presentation slip. The 2026-08-14 field log's stutter was exactly this thread
@@ -108,24 +253,33 @@ pub(super) fn audio_thread(
// Tier and redundancy are ONE decision, budgeted against the session's video bitrate — see
// `handshake::audio_budget`. An unparseable `audio.quality` was already warned about there
// and fell back to the default, so nothing here can silently downgrade someone's audio.
let (tier, redundancy) = (budget.tier, budget.redundancy);
//
// Redundancy is FORCED off on the lossless plane (§4.5): `0xD2` is not defined for `0xD3`,
// there is no PCM-side decoder that would receive it, and doubling a 1.4–33.9 Mbps plane is
// absurd on its face. The handshake already refuses to grant both bits together, so this is
// a second lock on the same door — cheap, and it means no future change to the budget ladder
// can switch redundancy on behind this branch.
let (tier, redundancy) = (budget.tier, budget.redundancy && !pcm_plane);
// Reuse the cached capturer ONLY when its channel count matches this session's; a stereo
// capturer left by a prior session must not feed a 5.1/7.1 session (the encoder + the client's
// decoder are sized for `want`, so a mismatched capturer would garble/desync the audio).
// Reuse the cached capturer ONLY when its channel count AND rate match this session's; a
// stereo capturer left by a prior session must not feed a 5.1/7.1 session (the encoder + the
// client's decoder are sized for `want`, so a mismatched capturer would garble/desync the
// audio), and a 48 kHz one must not feed a 96 kHz session — the frame arithmetic below is
// denominated in the negotiated rate, so a mismatch there is a pitch shift plus a sample
// clock that drifts against the wire clock at exactly the ratio of the two rates.
// A FAILED first open does not end the session's audio: session start is peak endpoint churn
// on Windows (the virtual-display attach and the wiring plan's own default-device flips race
// the WASAPI activate — 0x80070002 mid-re-registration), so it enters the same
// reopen-with-backoff loop a mid-session capture death does; audio then starts a few seconds
// late instead of never.
let capturer = match audio_cap.lock().unwrap().take() {
Some(mut c) if c.channels() == want as u32 => {
Some(mut c) if c.channels() == want as u32 && c.sample_rate() == rate_hz => {
c.drain(); // discard audio captured between sessions (also re-claims routing)
Some(c)
}
prev => {
drop(prev); // wrong channel count (or none): clean teardown, open fresh at `want`
match crate::audio::open_audio_capture(want as u32) {
drop(prev); // wrong channel count/rate (or none): clean teardown, open fresh
match crate::audio::open_audio_capture(want as u32, rate_hz) {
Ok(c) => Some(c),
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "punktfunk/1 audio failed to open — retrying in the background until it comes up");
@@ -134,19 +288,24 @@ pub(super) fn audio_thread(
}
}
};
let mut enc = match NativeAudioEnc::new(want, tier) {
Ok(e) => e,
Err(e) => {
tracing::warn!(error = %e, "opus encoder init failed — session continues without audio");
if let Some(mut c) = capturer {
c.idle(); // parked, not streaming — release the routing claim
crate::audio::park_audio_capture(&audio_cap, c);
// No Opus encoder at all on the PCM plane — there is nothing for it to do, and building one
// would make a libopus failure able to kill a session that does not use libopus.
let mut enc = if pcm_plane {
None
} else {
match NativeAudioEnc::new(want, tier) {
Ok(e) => Some(e),
Err(e) => {
tracing::warn!(error = %e, "opus encoder init failed — session continues without audio");
if let Some(mut c) = capturer {
c.idle(); // parked, not streaming — release the routing claim
crate::audio::park_audio_capture(&audio_cap, c);
}
return;
}
return;
}
};
let frame_len = SAMPLES_PER_FRAME * want as usize;
// Operator capture gain, soft-limited (`PUNKTFUNK_AUDIO_GAIN`, default 1.0 = untouched). This
// plane had NO gain at all until now, so `PUNKTFUNK_AUDIO_GAIN` silently did nothing on
// punktfunk/1 while working on GameStream — and since WASAPI loopback taps upstream of the
@@ -167,13 +326,28 @@ pub(super) fn audio_thread(
// one buffer covers both without allocating 200 times a second.
let mut frame_buf: Vec<f32> = Vec::with_capacity(frame_len);
// Sized for the largest surround frame (7.1 HQ ≈ 1.3 KB at 5 ms); ample for normal quality.
let mut opus_buf = vec![0u8; 4096];
// Empty on the PCM plane, which has no Opus encoder to write into it.
let mut opus_buf = vec![0u8; if pcm_plane { 0 } else { 4096 }];
// The PCM plane's own scratch. Sized EXACTLY — `frame_payload_bytes` is not an estimate but
// the size every frame has, and the 4 KB Opus guess would be both too small for 96/24 at the
// longest rungs and meaningless as a bound. `from_f32` appends, so this is cleared per frame
// and never reallocates after the first.
let mut pcm_wire: Vec<u8> = if pcm_plane {
Vec::with_capacity(pcm::frame_payload_bytes(rate_hz, bits, want, frame_us))
} else {
Vec::new()
};
let mut seq: u32 = 0;
// W-B1 — whether the wire covers a capture hole with silence, and for how long. See
// [`InfillPolicy`]: before this, a hole meant the loop simply blocked in `next_chunk` and
// NOTHING left the host for its duration, so the client's ring drained → underran →
// de-primed → re-primed, and a 30 ms hole became a much longer audible artifact.
let mut infill = crate::audio::capture_policy::InfillPolicy::default();
//
// Built from THIS session's frame, like the pacer above it. Both of the policy's figures used
// to be written against the Opus 5 ms frame — correct for as long as 5 ms was the only frame
// there was, and off by up to 5× on the lossless plane, whose frames are shorter by
// construction (§4.2).
let mut infill = crate::audio::capture_policy::InfillPolicy::new(frame_us);
let mut last_chunk_at = std::time::Instant::now();
// Nothing may be synthesized before the first real frame: there is no continuity to protect
// yet, and the wire clock has no anchor to continue from.
@@ -215,22 +389,49 @@ pub(super) fn audio_thread(
//
// Seeded rather than left uninitialised now that infilled frames advance it too: it is the
// pts of the NEXT frame to leave, real or synthesized, and every send advances it by one
// frame. `sent_any` is what keeps the seed from ever reaching the wire.
let mut next_pts_ns: u64 = 0;
// frame. `sent_any` is what keeps the seed from ever reaching the wire. See [`PtsClock`] for
// why it counts SAMPLES rather than accumulating the negotiated frame length.
let mut clock = PtsClock::new(rate_hz, want);
let mut pace_due: Option<std::time::Instant> = None;
// WP-C — what the wire actually did, as opposed to what the tap handed us. See [`SendStats`]:
// until this existed the send path was the one stage of the audio pipeline that could not be
// ruled in or out from a field log.
let mut send_stats = crate::audio::capture_policy::SendStats::default();
//
// Built from this session's frame, like the pacer and the infill policy: its `late` counter is
// "missed its slot by a whole protocol frame", and the lossless plane's frame is as short as
// 1 ms. Against the Opus constant this session could miss every slot and still report zero.
let mut send_stats = crate::audio::capture_policy::SendStats::new(frame_us);
let mut last_send_stats = std::time::Instant::now();
let mut last_departure: Option<std::time::Instant> = None;
// §4.8 — datagrams the wire refused. Counted, not merely survived: an uncounted drop is what
// makes a field report un-triageable (the lesson WP0.2 wrote down for the capture side), and
// on the PCM plane there is no PLC to hide one.
let mut oversized_drops: u64 = 0;
// The Opus plane's capture-rate warning fires at most once per session — the condition is
// re-tested a few hundred times a second, and it is a statement about the capturer, not an
// event.
let rate_mismatch_warned = std::sync::Once::new();
if capturer.is_some() {
tracing::info!(
channels = want,
// The plane this session actually runs, stated rather than assumed. The old line
// said "Opus 48 kHz, 5 ms datagrams" as a literal, which was true of every session
// that existed when it was written and is now one of four possible answers.
plane = if pcm_plane { "0xD3 PCM" } else { "0xC9 Opus" },
lossless = pcm_plane,
rate_hz,
bits,
frame_us,
// Meaningful only on the Opus plane — PCM has no tier and no rate control; its cost
// is exactly `rate × bits × channels`.
tier = tier.as_str(),
kbps = budget.kbps,
kbps = if pcm_plane {
pcm::bitrate_kbps(rate_hz, bits, want)
} else {
budget.kbps
},
redundancy,
"punktfunk/1 audio streaming (Opus 48 kHz, 5 ms datagrams)"
"punktfunk/1 audio streaming"
);
}
'session: while !stop.load(Ordering::SeqCst) {
@@ -239,7 +440,7 @@ pub(super) fn audio_thread(
std::thread::sleep(std::time::Duration::from_millis(200));
continue;
}
match crate::audio::open_audio_capture(want as u32) {
match crate::audio::open_audio_capture(want as u32, rate_hz) {
Ok(c) => {
tracing::info!("punktfunk/1 audio capture reopened");
capturer = Some(c);
@@ -258,6 +459,69 @@ pub(super) fn audio_thread(
}
}
}
// ⚠ Never ship a rate we did not get (`design/hi-res-audio.md` §9: "never claim a rate
// you did not get" — the rule is the same at both ends).
//
// **BELT AND BRACES, not the mechanism.** §8.4 condition 4 now asks the capture path what
// rate it can honestly deliver BEFORE the `Welcome` is built
// (`handshake::resolve_audio_plane` + `crate::audio::probe_capture_rate`), so the Windows
// §8.2 decline and the Linux §8.3 one both land as an ordinary "the session uses Opus
// 48 kHz" with a logged reason. This check is no longer how either of those is reached,
// and it is no longer expected to fire at all.
//
// What is left for it is the race the gate structurally cannot close: the probe reads the
// device, and the capture opens some milliseconds later. An endpoint whose format an
// operator changes in between, a hotplug that re-plans onto a different endpoint, or a
// graph that renegotiates mid-session all land here. That is a much smaller and much
// rarer set than "every 96 kHz request on a 48 kHz box", which is what it used to catch.
//
// The ACTION stays the same, because there is still only one correct one. The `Welcome`
// has already promised the client `rate_hz` and its output device is open at exactly
// that; every pts, every frame length and the client's whole de-jitter ring are
// denominated in it. We cannot switch the wire to Opus mid-session (§6 — a session runs
// one plane, and changing it means re-opening the client's device), and sending
// wrongly-clocked samples under the promised label is precisely the "label right, content
// wrong" failure this feature exists to avoid. So the lossless plane ENDS. What changed
// is the message: it no longer tells the operator to go and set a device rate, because
// the gate says that up front now — reaching here means something moved underneath a
// session the gate had already vetted.
//
// The Opus plane only WARNS, and deliberately: it is 48 kHz by definition, every
// capturer has always been asked for 48 kHz, and both backends resample to it rather
// than hand back something else. If one ever did, that has been true (and mis-clocked)
// for the plane's whole life — turning it into a session-ending condition here would
// risk silencing hosts that work today in order to police a case this pass did not
// introduce. Say it out loud instead; making it fatal is a decision for whoever has a
// reproduction.
//
// Checked every iteration rather than once at open, because the capturers do not all know
// their rate at open time: PipeWire's `param_changed` may land a moment after the ready
// handshake, and either backend can renegotiate mid-session. An atomic load a few hundred
// times a second costs nothing next to the encode.
let live_rate = capturer.as_ref().unwrap().sample_rate();
if live_rate != rate_hz {
if pcm_plane {
tracing::warn!(
promised_hz = rate_hz,
capture_hz = live_rate,
"the capture path changed rate under a session the hi-res gate had already \
vetted — ending the lossless audio plane rather than sending samples under \
a label that is not theirs (video continues). Reconnect: the gate re-runs \
against the capture path as it now is, and resolves either the real rate or \
Opus 48 kHz"
);
break 'session;
}
rate_mismatch_warned.call_once(|| {
tracing::warn!(
promised_hz = rate_hz,
capture_hz = live_rate,
"the capture path reports a rate other than the session's — the Opus plane \
continues (it is 48 kHz by definition and the capturer resamples), but the \
sample clock and the wire clock will not agree if this is real"
);
});
}
// Wake on whichever comes first: a capture chunk, or the moment the wire next has
// something to say. Waiting only on capture is what made a hole cost more than the audio
// it swallowed — see [`InfillPolicy`].
@@ -275,8 +539,8 @@ pub(super) fn audio_thread(
// the due time would spin through the window between them.
let ready_at = match pace_due {
Some(due) if acc.len() >= frame_len => due,
Some(due) => due.max(last_chunk_at + crate::audio::capture_policy::INFILL_AFTER),
None => now + FRAME_INTERVAL,
Some(due) => due.max(last_chunk_at + infill.after()),
None => now + frame_interval,
};
let budget = ready_at.saturating_duration_since(now).min(PACE_MAX_SLEEP);
capturer.as_mut().unwrap().next_chunk_within(budget)
@@ -308,12 +572,13 @@ pub(super) fn audio_thread(
let arrival_ns = now_ns();
acc.extend_from_slice(&chunk);
let queued_frames = (acc.len() / want as usize) as u64;
let anchor =
arrival_ns.saturating_sub(queued_frames * 1_000_000_000 / SAMPLE_RATE as u64);
// Never step backwards. Infilled frames advanced the wire clock while capture was
// away, and an anchor re-derived from this chunk's arrival can land at or before the
// last frame we already sent.
next_pts_ns = anchor.max(next_pts_ns);
// The session's rate, not the module constant: at 96 kHz a 48 000 divisor would put
// the anchor twice as far into the past as the queued audio really is, so every pts
// would be early by the whole buffer occupancy and A/V sync would chase it.
let anchor = arrival_ns.saturating_sub(queued_frames * 1_000_000_000 / rate_hz as u64);
// Never step backwards — and see [`PtsClock::reanchor`] for why that asymmetry is
// exactly what makes a fast clock unrecoverable and the sample-exact stamp mandatory.
clock.reanchor(anchor);
}
// Everything the wire owes for the slots that have come due — real or synthesized, one
// schedule, one encoder, one `seq`. A schedule that has fallen more than one frame behind
@@ -354,32 +619,83 @@ pub(super) fn audio_thread(
| crate::audio::capture_policy::Infill::Quiet => break,
}
}
pace_due = Some(pace_due.unwrap_or_else(std::time::Instant::now) + FRAME_INTERVAL);
pace_due = Some(pace_due.unwrap_or_else(std::time::Instant::now) + frame_interval);
if gain != 1.0 {
punktfunk_core::audio::apply_gain(&mut frame_buf, gain);
}
let pts_ns = next_pts_ns;
next_pts_ns += FRAME_MS as u64 * 1_000_000;
match enc.encode_float(&frame_buf, &mut opus_buf) {
Ok(n) => {
let opus = &opus_buf[..n];
let d = if redundancy {
punktfunk_core::quic::encode_audio_red_datagram(
seq,
pts_ns,
opus,
&prev_frame,
)
} else {
punktfunk_core::quic::encode_audio_datagram(seq, pts_ns, opus)
};
if conn.send_datagram(d.into()).is_err() {
break 'session; // connection gone
// W1.1 — the wire clock. ⚠ Charged the frame's REAL sample count, never the negotiated
// `frame_us`, which is a label on the 44.1 kHz family and would run this 2.3 ms/s fast
// forever; see [`PtsClock`]. `frame_buf.len()` rather than `frame_len` because both
// fill paths above produce exactly `frame_len` today and taking the count off the
// buffer we are about to send keeps that an observation rather than an assumption.
let pts_ns = clock.pts_ns();
clock.advance(frame_buf.len());
// Build this frame's datagram. Two planes, ONE send path below: the pacing, the
// telemetry and the send-error handling are properties of the wire, not of the codec,
// and duplicating them per plane is how the two would drift.
//
// `None` = nothing to send for this slot (an Opus encoder error, already counted).
// The PCM path cannot fail: `from_f32` is scale-and-clamp over a buffer of known
// length, with no encoder state to get stuck in.
let datagram: Option<Vec<u8>> = if pcm_plane {
pcm_wire.clear();
pcm::from_f32(&frame_buf, bits, &mut pcm_wire);
Some(punktfunk_core::quic::encode_audio_pcm_datagram(
seq, pts_ns, &pcm_wire,
))
} else {
// Hoisted out of the `match` scrutinee on purpose: the arms below take an
// IMMUTABLE slice of `opus_buf`, and a `&mut opus_buf` sitting in a scrutinee
// temporary is live for the whole match statement. Binding the result first ends
// that borrow at this semicolon and leaves nothing for the reader (or the borrow
// checker) to reason about.
let encoded = enc
.as_mut()
.expect("opus plane has an encoder")
.encode_float(&frame_buf, &mut opus_buf);
match encoded {
Ok(n) => {
let opus = &opus_buf[..n];
let d = if redundancy {
punktfunk_core::quic::encode_audio_red_datagram(
seq,
pts_ns,
opus,
&prev_frame,
)
} else {
punktfunk_core::quic::encode_audio_datagram(seq, pts_ns, opus)
};
if redundancy {
// The predecessor this frame just became. Recorded BEFORE the send
// rather than after it, so the drop arm below can clear it: a frame
// that never left must not be advertised as frame `seq`'s
// predecessor when the client's numbering has already moved past it.
prev_frame.clear();
prev_frame.extend_from_slice(opus);
}
Some(d)
}
if redundancy {
prev_frame.clear();
prev_frame.extend_from_slice(opus);
Err(e) => {
opus_encode_errs += 1;
if opus_encode_errs.is_power_of_two() {
tracing::warn!(
error = %e,
count = opus_encode_errs,
"opus encode failed — dropping audio frame"
);
}
None
}
}
};
let Some(d) = datagram else { continue };
// §4.8 — `SendDatagramError` is FOUR outcomes, and treating all of them as
// "connection gone" was wrong in a way hi-res makes reachable. A single oversized
// frame used to kill audio for the whole remaining session, silently: no log, no
// counter, and the video stream carrying on around it.
match conn.send_datagram(d.into()) {
Ok(()) => {
seq = seq.wrapping_add(1);
// Score the departure against its slot and against the previous one. `now` is
// from the top of this iteration — microseconds earlier and one clock read
@@ -390,19 +706,51 @@ pub(super) fn audio_thread(
infilled,
);
last_departure = Some(now);
// From here there is a continuity worth protecting, and `next_pts_ns` has a
// real anchor to continue from — both preconditions for synthesizing anything.
// From here there is a continuity worth protecting, and `clock` has a real
// anchor to continue from — both preconditions for synthesizing anything.
sent_any = true;
}
Err(e) => {
opus_encode_errs += 1;
if opus_encode_errs.is_power_of_two() {
// The only outcome that really is "the session is over".
Err(quinn::SendDatagramError::ConnectionLost(_)) => break 'session,
// The frame exceeds what this path can carry right now — quinn refuses it
// outright rather than fragmenting. One frame, not the plane: drop it, advance
// `seq` so the client SEES a gap and conceals it rather than silently
// mis-attributing the next frame, and keep going. Warn on powers of two (the
// idiom the encode-error arm above uses) so a persistently oversized format
// says so once, then twice, then four times, instead of 400 times a second.
//
// Persistent rather than transient is the case worth reading for: it means the
// negotiated `audio_frame_us` was sized against a datagram budget this path
// turned out not to have — see the MTU note in `handshake::negotiate`.
Err(quinn::SendDatagramError::TooLarge) => {
oversized_drops += 1;
if oversized_drops.is_power_of_two() {
tracing::warn!(
error = %e,
count = opus_encode_errs,
"opus encode failed — dropping audio frame"
count = oversized_drops,
frame_us,
rate_hz,
bits,
max_datagram = conn.max_datagram_size(),
"audio datagram rejected as too large — dropping the frame and \
continuing (the session's negotiated audio frame does not fit this \
path's datagram size)"
);
}
seq = seq.wrapping_add(1);
prev_frame.clear();
}
// Datagrams are gone for the rest of the connection — the peer never supported
// them, or the transport disabled them. Nothing this loop can do will make the
// next frame land, so end the audio plane cleanly (the capturer is parked below,
// the session keeps streaming video) instead of burning a core paced against a
// wire that cannot take it. Logged once, by construction: this arm breaks.
Err(e) => {
tracing::warn!(
error = %e,
"the QUIC datagram path is unavailable — ending the audio plane for this \
session (video continues)"
);
break 'session;
}
}
}
@@ -417,9 +765,19 @@ pub(super) fn audio_thread(
max_late_ms = send_stats.max_late_ms(),
max_spacing_ms = send_stats.max_spacing_ms(),
reanchors = send_stats.reanchors,
// §4.8 — frames the wire refused as oversized, cumulative for the session
// (unlike the rest of this line, which resets per window: a total is what
// answers "did this ever happen at all", and it is the question a field log
// gets asked). Zero on every healthy session, which is the point — a counter
// that reads zero for a REASON rather than by luck.
oversized_drops,
"audio egress"
);
send_stats = Default::default();
// The frame rides into the fresh window too — `Default` would have reset it to zero,
// which is why `SendStats` no longer has one: a zero frame makes every departure
// "late by a whole frame", and this is the line that would have said so, every 30 s,
// for the rest of the session.
send_stats = crate::audio::capture_policy::SendStats::new(frame_us);
last_send_stats = std::time::Instant::now();
}
}
@@ -441,6 +799,147 @@ pub(super) fn audio_thread(
_audio_cap: AudioCapSlot,
_channels: u8,
_budget: punktfunk_core::audio::AudioBudget,
_plane: super::handshake::AudioPlane,
) {
tracing::warn!("punktfunk/1 audio requires Linux or Windows — session continues without it");
}
#[cfg(all(test, any(target_os = "linux", target_os = "windows")))]
mod tests {
use super::*;
/// The number this whole clock exists for, pinned exactly rather than approximately.
///
/// One second of "5 ms" frames at 44 100 Hz stereo: 200 frames of 440 interleaved samples =
/// 88 000 samples, which is 997 732 426 ns — the second is 0.23 % SHORT of the 200 × 5 ms the
/// rung is labelled with, because 44 100 divides none of the seven ladder rungs and
/// `samples_per_frame` floors 220.5 to 220 per channel.
///
/// The old stamp added `frame_us * 1000` per frame and would land on exactly 1 000 000 000 —
/// 2 267 574 ns of time this host never captured, every second, forever. Planting that error
/// (`base += frame_us * 1000` in place of `advance`) makes the equality below fail by
/// 2 267 574 ns and the ppm assertion report 2 268.
#[test]
fn the_clock_counts_samples_and_not_nominal_frames() {
let (rate, us, ch) = (44_100u32, 5_000u32, 2u8);
let n = pcm::samples_per_frame(rate, us, ch);
assert_eq!(n, 440, "220 samples per channel, not 220.5");
let mut c = PtsClock::new(rate, ch);
let frames = 1_000_000 / us as u64; // 200
for _ in 0..frames {
c.advance(n);
}
let real_ns = c.pts_ns();
assert_eq!(real_ns, 997_732_426, "88 000 samples at 44 100 Hz stereo");
// What the nominal advance would have claimed, and the gap between the two.
let nominal_ns = frames * us as u64 * 1_000;
assert_eq!(nominal_ns - real_ns, 2_267_574, "invented ns per second");
let fast_ppm = (nominal_ns - real_ns) * 1_000_000 / real_ns;
assert_eq!(fast_ppm, 2_272, "the nominal clock runs this many ppm fast");
// ⚠ And the drift is CUMULATIVE, which is what makes it a defect rather than an offset:
// an hour of session is 8.2 seconds of invented time. The re-anchor cannot take any of it
// back (it only moves forward), so the client's A/V sync loop chases it to the end.
let hour = 3_600 * (nominal_ns - real_ns) / 1_000_000;
assert_eq!(hour, 8_163, "ms of drift over an hour");
}
/// Summing floored per-frame durations — the alternative core's doc offers — is *also* fine,
/// and this pins the size of the difference so the choice is on the record rather than
/// re-litigated. Under 1 ns per frame against the running total, four orders of magnitude
/// below what the nominal advance invents.
#[test]
fn a_running_total_beats_summing_floored_frames_by_a_hair() {
let (rate, us, ch) = (44_100u32, 5_000u32, 2u8);
let n = pcm::samples_per_frame(rate, us, ch);
let frames = 200u64;
let mut c = PtsClock::new(rate, ch);
for _ in 0..frames {
c.advance(n);
}
let summed = frames * pcm::frame_duration_ns(n, rate, ch);
let total = c.pts_ns();
assert!(
total >= summed && total - summed < frames,
"{total} vs {summed}"
);
}
/// On the 48 kHz family a rung IS a duration, so the new clock and the old nominal advance are
/// the same clock. This is why the defect went unnoticed while the ladder was 48/96 only — and
/// it is the compatibility claim that matters most: an Opus session's timestamps must not move
/// by a nanosecond.
#[test]
fn the_48k_family_is_bit_identical_to_the_nominal_advance() {
for (rate, ch) in [(48_000u32, 2u8), (48_000, 6), (48_000, 8), (96_000, 2)] {
for us in pcm::FRAME_US_LADDER {
let n = pcm::samples_per_frame(rate, us, ch);
let mut c = PtsClock::new(rate, ch);
for i in 1..=400u64 {
c.advance(n);
assert_eq!(
c.pts_ns(),
i * us as u64 * 1_000,
"{rate} Hz/{ch}ch at {us} µs, frame {i}"
);
}
}
}
}
/// The fold that keeps the running total bounded must be invisible: it moves whole seconds
/// from the sample count into the base, and `rate × channels` samples are exactly 1e9 ns at
/// every rate the plane carries. Run long enough to fold many times, on a rate that divides
/// nothing.
#[test]
fn folding_whole_seconds_does_not_move_the_clock() {
for (rate, ch) in [(44_100u32, 2u8), (176_400, 8), (88_200, 6), (96_000, 2)] {
let n = pcm::samples_per_frame(rate, 1_000, ch);
let mut c = PtsClock::new(rate, ch);
let mut unfolded: u128 = 0;
for _ in 0..5_000 {
c.advance(n);
unfolded += n as u128;
// The clock must always equal the duration of every sample ever charged to it,
// computed in one shot from zero — the property the fold could silently break.
assert!(
c.samples < c.samples_per_sec,
"{rate}/{ch}ch: the total was not folded"
);
assert_eq!(
c.pts_ns(),
(unfolded * 1_000_000_000 / (rate as u128 * ch as u128)) as u64,
"{rate} Hz/{ch}ch after {unfolded} samples"
);
}
}
}
/// The re-anchor is forward-only, and it restarts the running total rather than adding to it.
/// Both halves matter: moving back would re-issue a pts the client has already played, and
/// keeping the count across a re-anchor would charge the same span twice.
#[test]
fn the_reanchor_only_ever_moves_the_clock_forward() {
let (rate, ch) = (44_100u32, 2u8);
let n = pcm::samples_per_frame(rate, 5_000, ch);
let mut c = PtsClock::new(rate, ch);
c.reanchor(1_000_000_000);
assert_eq!(c.pts_ns(), 1_000_000_000);
c.advance(n);
let after = c.pts_ns();
assert_eq!(after, 1_000_000_000 + 4_988_662);
// An anchor behind the wire clock — capture returning after infilled frames advanced it —
// is ignored outright.
c.reanchor(1_000_000_000);
assert_eq!(c.pts_ns(), after, "a late anchor must not rewind the wire");
c.reanchor(after);
assert_eq!(c.pts_ns(), after, "an equal anchor is not a step either");
// Forward is taken, and the total restarts from it rather than compounding.
c.reanchor(after + 1_000_000);
assert_eq!(c.pts_ns(), after + 1_000_000);
c.advance(n);
assert_eq!(c.pts_ns(), after + 1_000_000 + 4_988_662);
}
}
crates/punktfunk-host/src/native/handshake.rs
+904 -6
View File
@@ -81,6 +81,276 @@ pub(super) fn redundancy_offered(client_caps: u8) -> bool {
&& pf_host_config::config().audio_redundancy.unwrap_or(true)
}
/// THE resolved audio plane for a session — the four values the `Welcome` states and the audio
/// thread is built from, produced together by [`resolve_audio_plane`] so they cannot disagree.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) struct AudioPlane {
/// [`AUDIO_CODEC_OPUS`](punktfunk_core::quic::AUDIO_CODEC_OPUS) (the `0xC9` plane) or
/// [`AUDIO_CODEC_PCM`](punktfunk_core::quic::AUDIO_CODEC_PCM) (the lossless `0xD3` plane).
pub codec: u8,
pub rate_hz: u32,
pub bits: u8,
/// Frame duration in µs on the `0xD3` plane; `0` on the Opus plane, whose frame length is
/// the fixed 5 ms of `0xC9`.
pub frame_us: u16,
}
impl AudioPlane {
/// Today's plane, and the answer to every failed gate below: Opus, 48 kHz, 16-bit.
///
/// A fallback to this is NOT a defeat — it is a 256 kbps stereo Opus stream that is
/// effectively transparent on game content (`design/hi-res-audio.md` §12). The only
/// unacceptable outcome is an *unexplained* one, which is why every caller of this in the
/// resolve gate logs its reason.
fn opus() -> AudioPlane {
AudioPlane {
codec: punktfunk_core::quic::AUDIO_CODEC_OPUS,
rate_hz: punktfunk_core::audio::SAMPLE_RATE_HZ,
bits: punktfunk_core::audio::pcm::BITS_16,
frame_us: 0,
}
}
/// Whether this session runs the lossless `0xD3` plane rather than Opus on `0xC9`.
pub fn is_pcm(self) -> bool {
self.codec == punktfunk_core::quic::AUDIO_CODEC_PCM
}
/// Recover the resolved plane from the [`Welcome`] that was actually sent.
///
/// Deliberately read BACK off the wire rather than passed forward from the gate — the same
/// discipline `serve_session` already uses for `audio_channels` and the granted `HOST_CAP_*`
/// bits. The client builds its decoder and opens its output device from these four values,
/// so the encoder has to be built from the identical ones; recomputing them would leave two
/// places that can drift, and a drift here is a session that sounds like noise.
pub(super) fn from_welcome(w: &Welcome) -> AudioPlane {
AudioPlane {
codec: w.audio_codec,
rate_hz: w.audio_rate_hz,
bits: w.audio_bits,
frame_us: w.audio_frame_us,
}
}
}
/// The largest share of the session's VIDEO bitrate the hi-res audio plane may take before the
/// request is declined (§8.4 condition 5).
///
/// Deliberately far above [`plan_audio_budget`](punktfunk_core::audio::plan_audio_budget)'s 5 %
/// ladder, and deliberately a separate number rather than a new rung on it: §4.6 is explicit that
/// hi-res must **never** be selected by that ladder — it is chosen only by an explicit opt-in on
/// both ends, and its cost has to be *visible* rather than smuggled past a budget written for
/// 96–512 kbps of Opus. Two ends asking for it earns a bigger allowance than the automatic
/// ladder's; it does not earn an unbounded one.
///
/// The reason a ceiling is needed at all is that audio rides QUIC datagrams **outside the ABR
/// loop**: whatever this plane takes is taken off the top, and ABR can neither see it nor reclaim
/// it when the link tightens. So this is not "audio gets 25 % and video adapts around it" — it is
/// "video permanently loses 25 % of a number it was already told to fit inside".
///
/// What 25 % buys, against `pcm::bitrate_kbps` **in stereo**: 48/16 (1 536 kbps) needs ≥ 6.1 Mbps
/// of video, 48/24 (2 304) needs ≥ 9.2, 96/16 (3 072) needs ≥ 12.3, 96/24 (4 608) needs ≥ 18.4.
/// A 5 Mbps session affords none of it, which is the §4.6 case ("more than half of a 5 Mbps
/// session") landing where it should.
///
/// ⚠ A 20 Mbps session no longer affords the whole stereo ladder, and this line used to say it
/// did. The 44.1 kHz family being admitted brought 176 400 Hz with it: 176.4/24 is 8 467 kbps and
/// wants ≥ 33.9 Mbps, 176.4/16 (5 644) wants ≥ 22.6. The cheap end moved too — 44.1/16 is
/// 1 411 kbps and needs only ≥ 5.6 Mbps, the first rung a modest link can actually reach.
///
/// Surround multiplies all of that by the channel count, and it is this gate rather than the frame
/// ladder that keeps it honest: 48/24 5.1 is 6 912 kbps and needs ≥ 27.6 Mbps of video, 7.1 is
/// 9 216 and needs ≥ 36.9. Both FIT a datagram comfortably — what they do not fit is an ordinary
/// link, and saying so in bits is the statement that survives a change of MTU.
const HIRES_MAX_VIDEO_SHARE_PCT: u32 = 25;
/// The §8.4 gate: resolve the session's audio plane. Returns [`AudioPlane::opus`] — today's
/// wire, byte for byte — unless **all four** policy conditions hold, and says out loud which one
/// lost.
///
/// 1. `client_asked` — the client set `CLIENT_CAP_AUDIO_HIRES`. Capable **and** the user turned
/// it on, the `VIDEO_CAP_444` precedent: a client that cannot open a 96 kHz output, or whose
/// user never asked, must not set the bit.
/// 2. `operator_allows` — `PUNKTFUNK_AUDIO_HIRES`, default OFF. This spends bandwidth the host's
/// owner did not previously agree to, so it is asked for at both ends.
/// 3. `capture_rate` — the capture path can GENUINELY deliver the requested rate (§8.2 / §8.3).
/// Not "did the open succeed": both backends accept a rate their endpoint does not run at and
/// resample to it without an error, so the question has to be put to the DEVICE before the
/// `Welcome` is built. See [`CaptureRate`](crate::audio::CaptureRate) for what each OS can
/// honestly answer and why an unknown answer declines.
/// 4. The link can afford it — see [`HIRES_MAX_VIDEO_SHARE_PCT`].
///
/// …plus two that are not policies at all. The requested format must be one the plane can carry:
/// a supported depth, and a rate in [`pcm::rate_is_supported`]'s set — **both** families now,
/// 44 100 / 48 000 / 88 200 / 96 000 / 176 400. The 44.1 kHz family was deferred rather than
/// refused (§4.1: `JitterPolicy` divided by 1 000 before it multiplied, so 44 100 Hz became 44
/// samples/ms and everything derived from it came out 2.3 % low); core fixed that arithmetic, so
/// this gate asks core for the set instead of restating it — a second expression of a rate set is
/// a second thing to forget to update, and a host and a client disagreeing about it is a session
/// that negotiates a format one end cannot open.
///
/// And a frame duration must EXIST for the negotiated format at this connection's datagram size;
/// `max_datagram` is `None` when the peer does not do datagrams (in which case there is no audio
/// plane of any kind to argue about). That test is also **where the channel count is decided** —
/// there is no separate stereo-only rule, and there deliberately never was a correct one; see the
/// note at the `frame_us_for` call.
///
/// **Not a downgrade ladder, on purpose.** A client asking for 96/24 on a link that only affords
/// 48/24 is declined rather than quietly handed the cheaper rung. The wire would carry it
/// perfectly well — the client opens its device from the `Welcome`, not from its request — but
/// choosing a *different* quality on the user's behalf is a product decision, and this pass makes
/// only the mechanical one. The log line names the cost, so the operator can see what to change.
///
/// Pure, so the whole gate is unit-testable: the operator policy AND the capture probe are passed
/// in rather than read from the process environment or the audio subsystem.
#[allow(clippy::too_many_arguments)] // one parameter per §8.4 condition; a struct would only rename them
pub(super) fn resolve_audio_plane(
client_asked: bool,
operator_allows: bool,
requested_rate_hz: u32,
requested_bits: u8,
channels: u8,
capture_rate: crate::audio::CaptureRate,
video_kbps: u32,
max_datagram: Option<usize>,
) -> AudioPlane {
use punktfunk_core::audio::pcm;
// Silent unless the client asked. Not logged: an ordinary session with an ordinary client is
// the overwhelming majority, and "this session did not use a feature nobody requested" is
// noise, not diagnosis.
if !client_asked {
return AudioPlane::opus();
}
if !operator_allows {
tracing::info!(
// ⚠ The range is the OPERATOR's decision criterion, so it has to keep up with what
// the plane can now negotiate: 1.4 Mbps at 44.1/16 stereo up to 8.5 at 176.4/24, and
// up to 33.9 for 176.4/24 7.1. It read "1.5–4.6" while the plane was 48/96 stereo.
"hi-res audio requested by the client but PUNKTFUNK_AUDIO_HIRES is not enabled on \
this host — the session uses Opus 48 kHz (the lossless plane costs 1.4–8.5 Mbps in \
stereo, and up to 33.9 in 7.1, off the top of the link — so it is opt-in on both \
ends)"
);
return AudioPlane::opus();
}
// ⚠ No channel-count test here, deliberately. This used to hard-decline `channels != 2`
// BEFORE the frame ladder was consulted, on the strength of §4.2's blanket "surround is out at
// the default MTU" — which is simply not true below 96 kHz: 48/16 5.1 fits a 2 ms frame and
// 48/24 7.1 fits a 1 ms one, well inside an ordinary datagram. An early `!= 2` did not
// *implement* that claim, it OVERRODE the one piece of code that knows the answer.
// `pcm::frame_us_for` is channel-aware and returns `None` when nothing fits, which is both the
// honest decline and the one that stays right when the MTU, the ladder or the depth set moves.
// See the frame-duration gate at the bottom of this function for what surround actually costs.
if !pcm::depth_is_supported(requested_bits) || !pcm::rate_is_supported(requested_rate_hz) {
tracing::info!(
requested_rate_hz,
requested_bits,
"hi-res audio was requested at a format this host does not carry (44 100 / 48 000 / \
88 200 / 96 000 / 176 400 Hz, 16 or 24-bit) — the session uses Opus 48 kHz"
);
return AudioPlane::opus();
}
// §8.4 condition 4, and the one condition that is about the world rather than about policy.
// It is checked HERE, before the `Welcome`, rather than left for the audio thread to discover
// at capture-open: by then the client has been promised a rate and has opened its device at
// it, and the only remaining move is to end the lossless plane — which is the silence outcome
// §8.4 calls the one unacceptable one. Declining here costs the session nothing but Opus.
if !capture_rate.can_deliver(requested_rate_hz) {
tracing::info!(
requested_rate_hz,
requested_bits,
?capture_rate,
"hi-res audio was requested but this host's capture path cannot honestly deliver \
that rate — the session uses Opus 48 kHz. On Windows the endpoint's own engine rate \
is authoritative (autoconvert would silently hand us an upsampled copy), so set the \
rate in that device's Windows properties; on Linux the default stream-sink mode \
delivers any supported rate, while PUNKTFUNK_STREAM_SINK=0 can only offer the rate \
the monitored sink itself runs at — and declines outright when that sink is idle or \
cannot be read"
);
return AudioPlane::opus();
}
let cost_kbps = pcm::bitrate_kbps(requested_rate_hz, requested_bits, channels);
let allowance = video_kbps.saturating_mul(HIRES_MAX_VIDEO_SHARE_PCT) / 100;
if cost_kbps > allowance {
tracing::info!(
requested_rate_hz,
requested_bits,
cost_kbps,
video_kbps,
allowance_kbps = allowance,
max_share_pct = HIRES_MAX_VIDEO_SHARE_PCT,
"hi-res audio would take more of this session's bitrate than it can spare — audio \
rides outside the ABR loop, so its cost comes off the top and ABR can neither see \
nor reclaim it; the session uses Opus 48 kHz"
);
return AudioPlane::opus();
}
let Some(max_datagram) = max_datagram else {
tracing::info!(
"hi-res audio needs QUIC datagrams and this connection reports none available — the \
session uses Opus 48 kHz"
);
return AudioPlane::opus();
};
// THE channel-count decision, and the only one: the ladder is asked whether a frame of this
// format FITS, and `None` is the decline. Channel count enters exactly here, as the multiplier
// it is — a 7.1 frame is four times a stereo one, so it needs a rung four times shorter and
// runs out of ladder four times sooner.
//
// At a 1 400-byte datagram that lands as: 5.1 at 48/16 on 2 ms and 48/24 on 1.5 ms
// (~667 packets/s), 7.1 at 48/16 on 1.5 ms and 48/24 on 1 ms; 16-bit 5.1 still fitting a 1 ms
// frame at 88.2 and 96 kHz; and **nothing surround above 48 kHz in 24-bit, and no 7.1 above
// 48 kHz at all** — 96/24 5.1 is 1 728 B of payload per millisecond, over the datagram before
// the shortest rung is reached. §4.2's blanket "surround is out at the default MTU" is
// therefore wrong for the whole 48 kHz-and-below half of the table.
//
// ⚠ The 44.1 kHz family fits the same rung or a LONGER one than 48 kHz, never a shorter one —
// 5.1/16 takes 2.5 ms where 48 kHz takes 2 — which is counter-intuitive only until you
// remember that a rung is a sample count here: 44 100 Hz simply puts fewer samples in the same
// milliseconds. It is the same floor that makes the rung a label rather than a duration for
// the pts (`audio.rs`), rounding in the safe direction for a payload and the unsafe one for a
// clock. The arithmetic is the authority rather than the prose; the gate tests pin the matrix.
//
// ⚠ What this does NOT police is packet rate. A 1 ms rung is 1 000 datagrams a second on a
// plane that rides outside the ABR loop; the affordability gate above is what keeps that from
// being reached on a link that cannot carry it, and it is stated in bits, not packets.
let Some(frame_us) =
pcm::frame_us_for(requested_rate_hz, requested_bits, channels, max_datagram)
else {
tracing::info!(
requested_rate_hz,
requested_bits,
channels,
max_datagram,
"no hi-res frame duration fits this connection's datagram size — the session uses \
Opus 48 kHz. This plane is never fragmented, so a frame that would not fit one \
datagram is not sent at all; surround and the rates above 96 kHz are what reach \
this, and a jumbo path (PUNKTFUNK_WIRE_MTU) is what would carry them"
);
return AudioPlane::opus();
};
tracing::info!(
rate_hz = requested_rate_hz,
bits = requested_bits,
frame_us,
cost_kbps,
video_kbps,
max_datagram,
// The evidence behind condition 4, not just its verdict: `Declared` and `Engine(96000)`
// are very different grounds for the same "yes", and a field report that claims the rate
// was padded is answered by which of them this session had.
?capture_rate,
"hi-res audio resolved — the session runs the lossless 0xD3 PCM plane"
);
AudioPlane {
codec: punktfunk_core::quic::AUDIO_CODEC_PCM,
rate_hz: requested_rate_hz,
bits: requested_bits,
frame_us: frame_us as u16,
}
}
pub(super) fn cursor_forward(
client_caps: u8,
compositor: Option<crate::vdisplay::Compositor>,
@@ -538,6 +808,80 @@ pub(super) async fn negotiate(
client_wants_chacha,
"session cipher"
);
// The audio plane (design/hi-res-audio.md §8.4): Opus on `0xC9`, or the lossless PCM `0xD3`
// plane when all five conditions hold. Every decline path inside logs its reason — the design
// is explicit that "silence is the one unacceptable outcome", and an unexplained fallback is
// the shape that produces it.
//
// ⚠ `conn.max_datagram_size()` here is quinn's CURRENT value, and QUIC MTU discovery has
// NOT settled at Welcome time — it starts when the handshake completes and needs an acked
// probe per binary-search step, which is exactly why `negotiated_shard_payload` above has to
// *wait* for a jumbo re-proof. §4.2 warns that reading it too early sizes for the
// conservative initial value, and that is what happens here.
//
// It is deliberate, and it is the SAFE direction: the frame duration is a promise made in
// the `Welcome`, the client sizes its ring and opens its device from it, and this plane has
// no mechanism to restate it mid-session (§6 — a session runs one plane at one frame length,
// and switching would mean re-opening the client's device). A frame that fits the initial
// MTU keeps fitting as the MTU grows; a frame sized for a discovered MTU that then turns out
// not to hold would not be sent at all. So the cost of reading early is a slightly higher
// packet rate than the path could carry — never a dropped plane.
//
// TODO(hi-res H5): to spend the discovered MTU instead, the frame duration has to be decided
// AFTER discovery settles, which needs either a `Welcome` sent later than it is today or a
// wire message that restates `audio_frame_us` before the client opens its output. Both are
// wire/sequencing changes well beyond this pass; neither is needed for 48 kHz, where the
// conservative answer already lands on the longest rung.
let hires_asked = hello.client_caps & punktfunk_core::quic::CLIENT_CAP_AUDIO_HIRES != 0;
// A `Hello` that names a format but does not set the capability is CONTRADICTORY, and the two
// halves come from different places in a client — the capability from a settings toggle, the
// rate and depth from whatever that toggle resolved to. Condition 1 below is deliberately not
// logged, because "no capability" is every ordinary session with every shipping client and
// would drown the log. This case is not ordinary: something asked, and is being ignored.
//
// Worth the line because it is the exact shape that cost an on-glass session its first run —
// the host resolved Opus while every visible condition looked satisfiable, and the reason was
// unlogged by design. An embedder hitting this sees nothing at all otherwise.
if !hires_asked && (hello.audio_rate_hz != 0 || hello.audio_bits != 0) {
tracing::warn!(
requested_rate_hz = hello.audio_rate_hz,
requested_bits = hello.audio_bits,
"client sent an audio format but not CLIENT_CAP_AUDIO_HIRES — ignoring it and \
staying on Opus; the capability and the format must be set together"
);
}
let hires_allowed = pf_host_config::config().audio_hires.unwrap_or(false);
// §8.4 condition 4 — what the capture path can HONESTLY deliver, asked of the device rather
// than inferred from a successful open (§4.3/§4.4: both backends resample a rate they cannot
// run at, without an error). Blocking on Windows (an endpoint enumeration plus an
// `IAudioClient` activation per candidate), so it runs off the reactor like the 10-bit and
// 4:4:4 probes above.
//
// Short-circuited behind the two cheap policy conditions, which is the same discipline those
// probes use: an ordinary session — every session with every shipping client today — must not
// pay COM work for a feature nobody asked for. The value is not merely unused in that case
// but unreachable, since the gate returns on condition 1 or 2 before it looks at this one;
// `Unknown` is nonetheless the correct thing to pass, because "we did not ask" and "we asked
// and could not tell" both mean the same thing to the gate: decline.
let capture_rate = if hires_asked && hires_allowed {
tokio::task::spawn_blocking(crate::audio::probe_capture_rate)
.await
.context("audio capture-rate probe task")?
} else {
crate::audio::CaptureRate::Unknown
};
let audio_plane = resolve_audio_plane(
hires_asked,
hires_allowed,
hello.audio_rate_hz,
hello.audio_bits,
audio_channels,
capture_rate,
bitrate_kbps,
conn.max_datagram_size(),
);
let welcome = Welcome {
abi_version: punktfunk_core::WIRE_VERSION,
udp_port,
@@ -643,12 +987,19 @@ pub(super) async fn negotiate(
// Redundant desktop-audio plane (0xD2): the client asked, the operator has not forced
// it off, AND it fits the session's audio budget. Capable-and-agreed like the cursor
// bit — a client that did not ask keeps the plain 0xC9 wire byte-for-byte.
| if audio_budget(
redundancy_offered(hello.client_caps),
bitrate_kbps,
audio_channels,
)
.redundancy
//
// Never alongside the lossless plane: `0xD2` is not defined for `0xD3` and is never
// sent with it (§4.5). Doubling a 1.4–33.9 Mbps plane is absurd on its face, and the
// client has no `0xD2` decoder on the PCM side to receive it — so the two bits are
// mutually exclusive on the wire, stated here rather than left to the audio thread
// to discover.
| if !audio_plane.is_pcm()
&& audio_budget(
redundancy_offered(hello.client_caps),
bitrate_kbps,
audio_channels,
)
.redundancy
{
punktfunk_core::quic::HOST_CAP_AUDIO_RED
} else {
@@ -663,6 +1014,16 @@ pub(super) async fn negotiate(
punktfunk_core::quic::HOST_CAP_PAD_AUDIO
} else {
0
}
// Lossless desktop audio (0xD3 PCM): set ONLY when the §8.4 gate above actually
// resolved to PCM — the bit is a statement about the wire this session will carry,
// not about what the host could do in principle, exactly like HOST_CAP_AUDIO_RED.
// ⚠ 0x80 is the LAST free host_caps bit; the next host capability needs a second
// byte and an ABI bump (§4.7).
| if audio_plane.is_pcm() {
punktfunk_core::quic::HOST_CAP_AUDIO_HIRES
} else {
0
},
// Where this host serves its game library, so the client never has to have seen an mDNS
// advert to find it. `0` on the standalone punktfunk1-host binary (no management API),
@@ -684,6 +1045,18 @@ pub(super) async fn negotiate(
punktfunk_core::quic::CIPHER_AES_128_GCM
},
key_chacha,
// The RESOLVED audio plane, from the §8.4 gate above. Opus at 48 kHz / 16-bit — the
// legacy answer — makes `Welcome::encode` omit all four fields, so an Opus session's
// Welcome stays byte-identical to the pre-hi-res wire form for every client (the interop
// property the cipher byte bought and every appended field since has had to keep).
//
// These are the values the client opens its output device from; it must never open from
// what it ASKED for. `audio_frame_us` is `0` on the Opus plane, whose frame length is the
// fixed 5 ms of 0xC9.
audio_codec: audio_plane.codec,
audio_rate_hz: audio_plane.rate_hz,
audio_bits: audio_plane.bits,
audio_frame_us: audio_plane.frame_us,
};
io::write_msg(send, &welcome.encode()).await?;
bringup.mark("welcome");
@@ -782,3 +1155,528 @@ pub(super) async fn negotiate(
prep,
))
}
#[cfg(test)]
mod tests {
use super::*;
use punktfunk_core::audio::pcm;
/// A usable datagram at the default 1472-byte discovery ceiling, less QUIC header + AEAD
/// tag. The same number `pcm`'s own ladder test argues from.
const DGRAM: usize = 1400;
/// Comfortably above the 25 % allowance for every STEREO rung on the ladder (96/24 needs
/// 18.4).
const FAT_LINK_KBPS: u32 = 40_000;
/// …and enough for every SURROUND rung too — 176.4/24 7.1 is 33 869 kbps and wants ≥ 135 Mbps.
/// Used only where the frame ladder is the thing under test, so a row that should fail on
/// arithmetic cannot fail on bandwidth first and look like a pass for the wrong reason.
const HUGE_LINK_KBPS: u32 = 200_000;
/// A capture path that can carry anything the plane asks for — Linux stream-sink mode, where
/// the host declares the format itself (§4.4). The condition-4 tests vary this; every other
/// test holds it here so it is never the thing that made them pass or fail.
const HONEST_CAPTURE: crate::audio::CaptureRate = crate::audio::CaptureRate::Declared;
/// The happy path, so every decline test below is a difference from something that works.
#[test]
fn all_five_conditions_met_resolves_to_the_lossless_plane() {
for (rate, bits) in [
(48_000u32, pcm::BITS_16),
(48_000, pcm::BITS_24),
(96_000, pcm::BITS_16),
(96_000, pcm::BITS_24),
] {
let p = resolve_audio_plane(
true,
true,
rate,
bits,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert!(p.is_pcm(), "{rate}/{bits} should have resolved to PCM");
assert_eq!(p.rate_hz, rate);
assert_eq!(p.bits, bits);
// The negotiated frame must actually fit, or the datagram is never sent at all.
assert!(
pcm::frame_payload_bytes(rate, bits, 2, p.frame_us as u32) + pcm::PCM_HEADER_LEN
<= DGRAM,
"{rate}/{bits} chose a {} µs frame that does not fit",
p.frame_us
);
}
}
/// §8.4 condition 1 — the client never asked. This is every session with every shipping
/// client today, so it must be the quietest possible path to the legacy wire.
#[test]
fn a_client_that_did_not_ask_gets_opus() {
let p = resolve_audio_plane(
false,
true,
96_000,
pcm::BITS_24,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert_eq!(p, AudioPlane::opus());
}
/// §8.4 condition 2 — the operator's `PUNKTFUNK_AUDIO_HIRES` gate, default OFF. A client
/// asking is not enough on its own: this costs bandwidth the host's owner never agreed to.
#[test]
fn the_operator_gate_alone_can_decline() {
let p = resolve_audio_plane(
true,
false,
48_000,
pcm::BITS_24,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert_eq!(p, AudioPlane::opus());
}
/// …and the default really is off, so an operator who has set nothing gets today's wire.
/// (`config()` reads the process environment once; no test in this crate sets the knob.)
#[test]
fn the_operator_default_is_off() {
assert!(!pf_host_config::config().audio_hires.unwrap_or(false));
}
/// Surround, decided by the FRAME LADDER rather than by a stereo-only rule — the whole point
/// of removing the `channels != 2` decline that used to sit above it.
///
/// ⚠ This runs on [`HUGE_LINK_KBPS`] on purpose: on an ordinary link every declining row here
/// would decline on BANDWIDTH first (96/24 5.1 is 13.8 Mbps and wants a 55 Mbps session), and
/// the test would prove the affordability gate while claiming to prove the ladder. The link is
/// taken out of the argument so the only thing that can move a row is the arithmetic.
///
/// The matrix contradicts the design in both directions, which is why it is written out rather
/// than summarised: §4.2's blanket "surround is out at the default MTU" is false for the whole
/// 48 kHz-and-below half, and "above 48 kHz surround fits nothing" is false for 16-bit 5.1,
/// which still fits a 1 ms frame at 88.2 and 96 kHz.
#[test]
fn surround_is_decided_by_the_frame_ladder() {
// (channels, rate, bits, the rung it must land on — `None` = the honest decline)
let matrix: [(u8, u32, u8, Option<u16>); 20] = [
// 5.1 — and note 44.1 kHz fits a LONGER rung than 48 kHz, not a shorter one: a rung
// is a sample count, and 44 100 Hz puts fewer samples in the same milliseconds.
(6, 44_100, pcm::BITS_16, Some(2500)),
(6, 44_100, pcm::BITS_24, Some(1500)),
(6, 48_000, pcm::BITS_16, Some(2000)),
(6, 48_000, pcm::BITS_24, Some(1500)), // ~667 packets/s
(6, 88_200, pcm::BITS_16, Some(1000)),
(6, 88_200, pcm::BITS_24, None),
(6, 96_000, pcm::BITS_16, Some(1000)),
(6, 96_000, pcm::BITS_24, None), // 1 728 B per ms — over before the shortest rung
(6, 176_400, pcm::BITS_16, None),
(6, 176_400, pcm::BITS_24, None),
// 7.1 — four times a stereo frame, so it runs out of ladder four times sooner, and
// nothing above 48 kHz fits at either depth.
(8, 44_100, pcm::BITS_16, Some(1500)),
(8, 44_100, pcm::BITS_24, Some(1000)),
(8, 48_000, pcm::BITS_16, Some(1500)),
(8, 48_000, pcm::BITS_24, Some(1000)),
(8, 88_200, pcm::BITS_16, None),
(8, 88_200, pcm::BITS_24, None),
(8, 96_000, pcm::BITS_16, None),
(8, 96_000, pcm::BITS_24, None),
(8, 176_400, pcm::BITS_16, None),
(8, 176_400, pcm::BITS_24, None),
];
for (ch, rate, bits, want_us) in matrix {
let p = resolve_audio_plane(
true,
true,
rate,
bits,
ch,
HONEST_CAPTURE,
HUGE_LINK_KBPS,
Some(DGRAM),
);
match want_us {
Some(us) => {
assert!(
p.is_pcm(),
"{ch}ch {rate}/{bits} should have resolved to PCM"
);
assert_eq!(p.frame_us, us, "{ch}ch {rate}/{bits} rung");
assert_eq!(p.rate_hz, rate);
assert_eq!(p.bits, bits);
// Whatever the ladder chose, it has to FIT — a datagram over the path MTU is
// not sent at all, and this plane is never fragmented.
assert!(
pcm::frame_payload_bytes(rate, bits, ch, us as u32) + pcm::PCM_HEADER_LEN
<= DGRAM,
"{ch}ch {rate}/{bits} chose a {us} µs frame that does not fit"
);
}
None => assert_eq!(
p,
AudioPlane::opus(),
"{ch}ch {rate}/{bits} must decline via the ladder, not be carried"
),
}
}
}
/// …and on an ORDINARY link surround declines on bandwidth long before the ladder is reached,
/// which is the outcome a real session sees. Stated separately so the two gates can never be
/// confused for one another: 48/24 5.1 is 6 912 kbps and wants ≥ 27.6 Mbps of video.
#[test]
fn surround_still_needs_a_link_that_can_afford_it() {
assert_eq!(
resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_24,
6,
HONEST_CAPTURE,
20_000,
Some(DGRAM)
),
AudioPlane::opus(),
"5.1 at 48/24 costs 6 912 kbps — more than a 20 Mbps session's 25 % allowance"
);
assert!(resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_24,
6,
HONEST_CAPTURE,
28_000,
Some(DGRAM)
)
.is_pcm());
}
/// The 44.1 kHz family, which core has just made reachable — the deferral in §4.1 was
/// `JitterPolicy` dividing by 1 000 before it multiplied, not anything about the plane. These
/// used to land in `an_unsupported_format_gets_opus`; a host that still refuses them now
/// disagrees with `pcm::rate_is_supported` and with every client that has already shipped the
/// request.
#[test]
fn the_44_1_khz_family_resolves_to_the_lossless_plane() {
for (rate, bits, want_us) in [
(44_100u32, pcm::BITS_16, 5000u16),
(44_100, pcm::BITS_24, 5000),
(88_200, pcm::BITS_16, 3000),
(88_200, pcm::BITS_24, 2500),
(176_400, pcm::BITS_16, 1500),
(176_400, pcm::BITS_24, 1000),
] {
let p = resolve_audio_plane(
true,
true,
rate,
bits,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert!(p.is_pcm(), "{rate}/{bits} should have resolved to PCM");
assert_eq!(p.rate_hz, rate, "the Welcome must state what was ASKED for");
assert_eq!(p.bits, bits);
assert_eq!(p.frame_us, want_us, "{rate}/{bits} rung");
assert!(
pcm::frame_payload_bytes(rate, bits, 2, p.frame_us as u32) + pcm::PCM_HEADER_LEN
<= DGRAM,
"{rate}/{bits} chose a {} µs frame that does not fit",
p.frame_us
);
}
// ⚠ The gate must never round 44 100 to 48 000 to make it fit something. That would be the
// exact "label right, content wrong" lie the feature is built to avoid — and it is now the
// reachable mistake, where before the whole family was simply refused.
let p = resolve_audio_plane(
true,
true,
44_100,
pcm::BITS_24,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert_eq!(p.rate_hz, 44_100);
}
/// A format the plane cannot carry at all — and after the 44.1 kHz family was admitted, that
/// set is only the rates outside BOTH families. 192 kHz is out by the §3 scope decision rather
/// than by any arithmetic; 16 kHz is a narrow voice rate this plane never offers.
#[test]
fn an_unsupported_format_gets_opus() {
for rate in [192_000u32, 16_000] {
let p = resolve_audio_plane(
true,
true,
rate,
pcm::BITS_24,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert_eq!(p, AudioPlane::opus(), "{rate} Hz");
}
// The gate must read the set off core rather than restate it, so the two cannot drift.
for rate in [44_100u32, 48_000, 88_200, 96_000, 176_400] {
assert!(pcm::rate_is_supported(rate), "{rate} Hz");
}
for rate in [0u32, 22_050, 32_000, 192_000] {
assert!(!pcm::rate_is_supported(rate), "{rate} Hz");
}
for bits in [8u8, 20, 32] {
let p = resolve_audio_plane(
true,
true,
48_000,
bits,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(DGRAM),
);
assert_eq!(p, AudioPlane::opus(), "{bits}-bit");
}
}
/// §8.4 condition 4 — the capture path cannot deliver the rate, so the request is declined
/// BEFORE the `Welcome` states one.
///
/// This is the condition the design cares about most (§4.3, §13 item 2): every other gate
/// failing produces a session that is merely not hi-res, whereas this one failing *silently*
/// produces a session that says 96 kHz, spends 4.6 Mbps saying it, and carries interpolated
/// 48 kHz. Both ends would audit clean.
#[test]
fn a_capture_path_that_cannot_deliver_the_rate_gets_opus() {
use crate::audio::CaptureRate;
// Windows, §8.2: the endpoint's engine runs at 48 kHz. `AUTOCONVERTPCM` would accept a
// 96 kHz request and upsample — so 96 declines and 48 is honoured, which is exactly the
// `requested > engine.rate` rule and not a blanket refusal.
let engine_48 = CaptureRate::Engine(48_000);
assert_eq!(
resolve_audio_plane(
true,
true,
96_000,
pcm::BITS_24,
2,
engine_48,
FAT_LINK_KBPS,
Some(DGRAM)
),
AudioPlane::opus(),
"96 kHz on a 48 kHz engine must decline rather than pad"
);
assert!(
resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_24,
2,
engine_48,
FAT_LINK_KBPS,
Some(DGRAM)
)
.is_pcm(),
"48 kHz on a 48 kHz engine is bit-exact and must be honoured"
);
// An engine ABOVE the request is fine: 96 → 48 is a real resample down to a rate that
// genuinely carries every sample the client will be told about. §8.2 declines only when
// the request is higher than the engine.
assert!(resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_16,
2,
CaptureRate::Engine(96_000),
FAT_LINK_KBPS,
Some(DGRAM)
)
.is_pcm());
// A narrow endpoint (a headset's hands-free profile, Steam's voice-carrier sink) cannot
// even do the base rate.
assert_eq!(
resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_16,
2,
CaptureRate::Engine(24_000),
FAT_LINK_KBPS,
Some(DGRAM)
),
AudioPlane::opus()
);
// Unknown — a Linux `PUNKTFUNK_STREAM_SINK=0` monitor capture whose elected sink could
// not be read (§8.3), or a Windows probe that could not reach the endpoint. Declines
// every rung: an unprovable claim is not a claim.
for (rate, bits) in [
(48_000u32, pcm::BITS_16),
(48_000, pcm::BITS_24),
(96_000, pcm::BITS_16),
(96_000, pcm::BITS_24),
] {
assert_eq!(
resolve_audio_plane(
true,
true,
rate,
bits,
2,
CaptureRate::Unknown,
FAT_LINK_KBPS,
Some(DGRAM)
),
AudioPlane::opus(),
"{rate}/{bits} with an unknowable capture rate"
);
}
}
/// The probe's own arithmetic, pinned away from the gate so a future backend answering
/// [`CaptureRate`](crate::audio::CaptureRate) has the contract stated rather than inferred.
#[test]
fn capture_rate_answers_only_what_it_can_prove() {
use crate::audio::CaptureRate;
// The host owns the sink and declares its format — honest by construction at any rate
// the plane supports (§4.4).
assert!(CaptureRate::Declared.can_deliver(48_000));
assert!(CaptureRate::Declared.can_deliver(96_000));
// At-or-below the engine only, and the boundary is inclusive: an engine at exactly the
// requested rate is the *normal* passing case, not an edge to be conservative about.
assert!(CaptureRate::Engine(96_000).can_deliver(96_000));
assert!(CaptureRate::Engine(96_000).can_deliver(48_000));
assert!(!CaptureRate::Engine(48_000).can_deliver(96_000));
assert!(!CaptureRate::Engine(44_100).can_deliver(48_000));
// Never yes without evidence.
assert!(!CaptureRate::Unknown.can_deliver(48_000));
assert!(!CaptureRate::Unknown.can_deliver(96_000));
}
/// §8.4 condition 5 — the link cannot afford it. The plane rides outside the ABR loop, so
/// its cost is off the top and ABR can neither see nor reclaim it (§4.6).
#[test]
fn a_link_that_cannot_afford_it_gets_opus() {
// 5 Mbps affords nothing on the ladder — the §4.6 case, stated as a test.
for (rate, bits) in [
(48_000u32, pcm::BITS_16),
(48_000, pcm::BITS_24),
(96_000, pcm::BITS_16),
(96_000, pcm::BITS_24),
] {
let p = resolve_audio_plane(
true,
true,
rate,
bits,
2,
HONEST_CAPTURE,
5_000,
Some(DGRAM),
);
assert_eq!(p, AudioPlane::opus(), "{rate}/{bits} on a 5 Mbps session");
}
// 10 Mbps affords 48 kHz at either depth and neither 96 kHz rung — and the boundary is
// the one the constant's doc claims, not one a reader has to re-derive.
assert!(resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_24,
2,
HONEST_CAPTURE,
10_000,
Some(DGRAM)
)
.is_pcm());
assert_eq!(
resolve_audio_plane(
true,
true,
96_000,
pcm::BITS_16,
2,
HONEST_CAPTURE,
10_000,
Some(DGRAM)
),
AudioPlane::opus()
);
// A session with no video bitrate at all can never afford it, and must not divide by it.
assert_eq!(
resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_16,
2,
HONEST_CAPTURE,
0,
Some(DGRAM)
),
AudioPlane::opus()
);
}
/// The sixth, structural condition: a frame has to FIT. A peer with no datagram support has
/// no audio plane to negotiate, and a datagram too small for even the shortest rung must
/// fall back rather than emit a frame that would never be sent.
#[test]
fn a_datagram_that_cannot_carry_a_frame_gets_opus() {
assert_eq!(
resolve_audio_plane(
true,
true,
48_000,
pcm::BITS_16,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
None
),
AudioPlane::opus()
);
// 96/24 at the shortest rung (1000 µs) is 96 × 2 × 3 = 576 B + 13 of header.
assert_eq!(
resolve_audio_plane(
true,
true,
96_000,
pcm::BITS_24,
2,
HONEST_CAPTURE,
FAT_LINK_KBPS,
Some(200)
),
AudioPlane::opus()
);
}
/// The Opus fallback must be byte-for-byte today's answer, or an Opus session's `Welcome`
/// stops being byte-identical to the pre-hi-res wire form and every existing client sees a
/// message it has to have been taught to parse.
#[test]
fn the_opus_fallback_is_the_legacy_wire_form() {
let p = AudioPlane::opus();
assert_eq!(p.codec, punktfunk_core::quic::AUDIO_CODEC_OPUS);
assert_eq!(p.rate_hz, punktfunk_core::audio::SAMPLE_RATE_HZ);
assert_eq!(p.bits, pcm::BITS_16);
assert_eq!(p.frame_us, 0);
assert!(!p.is_pcm());
}
}
crates/punktfunk-host/src/native/pad_audio.rs
+47 -7
View File
@@ -462,8 +462,10 @@ fn build_lanes(kinds: u8) -> Result<Vec<Lane>, opus::Error> {
/// The per-pad streaming thread: capture of the pad's audio device (`open` builds the
/// platform's capturer — Windows loopback / Linux minted sink) → framer → per-kind gate/encode
/// → 0xD1 datagrams. Capture death reopens with the session-audio backoff
/// ([`INJECTOR_REOPEN_BACKOFF`], encoders + seq kept); a send error ends the thread (the
/// connection — the session — is gone).
/// ([`INJECTOR_REOPEN_BACKOFF`], encoders + seq kept); a LOST CONNECTION — or a datagram path
/// that has gone away for good — ends the thread, while a single oversized datagram costs only
/// that frame (`design/hi-res-audio.md` §4.8: the four `SendDatagramError` outcomes are not one
/// outcome, and collapsing them silently ended this plane for the rest of a session).
#[cfg(any(target_os = "windows", target_os = "linux"))]
fn pad_audio_thread<C: crate::audio::AudioCapturer>(
conn: quinn::Connection,
@@ -494,6 +496,10 @@ fn pad_audio_thread<C: crate::audio::AudioCapturer>(
// first open ALSO rides this loop, so an open lost to endpoint churn starts late, not never.
let mut capturer: Option<C> = None;
let mut last_failed: Option<std::time::Instant> = None;
// Datagrams the wire refused as oversized (`design/hi-res-audio.md` §4.8). Vanishingly
// unlikely on this plane — a 64 kbps CBR Opus frame at ≤10 ms is ~80 bytes — but it used to
// be indistinguishable from the connection ending, which is the actual defect being fixed.
let mut oversized_drops: u64 = 0;
tracing::info!(
pad,
haptics = kinds & KIND_BIT_HAPTICS != 0,
@@ -533,9 +539,11 @@ fn pad_audio_thread<C: crate::audio::AudioCapturer>(
continue;
}
};
let mut session_gone = false;
// Whether this plane is finished — a lost connection, or a datagram path that will never
// carry anything again. NOT set by an oversized frame, which costs exactly that frame.
let mut end_plane = false;
framer.feed(&chunk, |kind, frame| {
if session_gone {
if end_plane {
return;
}
let Some(lane) = lanes.iter_mut().find(|l| l.kind == kind) else {
@@ -556,8 +564,40 @@ fn pad_audio_thread<C: crate::audio::AudioCapturer>(
pts_ns,
&opus_buf[..n],
);
if conn.send_datagram(d.into()).is_err() {
session_gone = true; // connection gone — the session is over
// The same four-outcome match the session audio plane makes (§4.8): treating
// every `SendDatagramError` as "the connection is gone" turned a single
// refused frame into the silent end of this pad's audio for the rest of the
// session, with nothing in any log to say so.
match conn.send_datagram(d.into()) {
Ok(()) => {}
// The only outcome that really is "the session is over".
Err(quinn::SendDatagramError::ConnectionLost(_)) => end_plane = true,
// One frame, not the plane. Dropped and counted; `seq` was already taken
// from the lane's gate, so the client sees the gap and its concealment
// handles it exactly as it handles a lost packet.
Err(quinn::SendDatagramError::TooLarge) => {
oversized_drops += 1;
if oversized_drops.is_power_of_two() {
tracing::warn!(
pad,
kind,
count = oversized_drops,
opus_bytes = n,
"pad-audio datagram rejected as too large — dropping the \
frame and continuing"
);
}
}
// Datagrams are gone for this connection's lifetime; nothing this thread
// does will make the next frame land. End the pad's audio plane cleanly.
Err(e) => {
tracing::warn!(
pad,
error = %e,
"the QUIC datagram path is unavailable — ending this pad's audio"
);
end_plane = true;
}
}
}
Err(e) => {
@@ -574,7 +614,7 @@ fn pad_audio_thread<C: crate::audio::AudioCapturer>(
}
}
});
if session_gone {
if end_plane {
break 'session;
}
}
docs-site/content/docs/configuration.md
+4 -1
View File
@@ -157,6 +157,7 @@ See your desktop page ([KDE](/docs/kde), [GNOME](/docs/gnome)) for when to set t
|---|---|---|
| `PUNKTFUNK_AUDIO_QUALITY` | `low` · `standard` · `high` *(default `high`)* | Desktop-audio encode quality. `high` (stereo 256 kbps Opus, effectively transparent) costs about 1 % of a normal video bitrate, so there's rarely a reason to go lower. `standard` is exactly the pre-0.25 encoder (stereo 128 kbps) — handy for an A/B comparison; `low` is for genuinely constrained links (noticeably lossy on music, still fine for game audio and voice). A typo warns in the log and keeps `high` rather than silently downgrading. Host-side only — clients play whatever arrives, no client setting involved. |
| `PUNKTFUNK_AUDIO_REDUNDANCY` | `1` · `0` *(default: automatic)* | Send audio packets redundantly so a lossy link doesn't crackle. Leave it unset: the host turns redundancy on by itself, only toward clients that support it and only while the link is actually losing packets. `1` forces it on for the whole session, `0` never sends it. |
| `PUNKTFUNK_AUDIO_HIRES` | `1` · `0` *(default off)* | Allow the **lossless** audio plane — uncompressed 48/96 kHz, 16/24-bit stereo PCM instead of Opus. Off unless you set it, *and* off unless the client asks for it too: both ends have to opt in, because it costs **1.5–4.6 Mbps** where Opus costs 256 kbps, and audio rides QUIC datagrams outside the adaptive-bitrate loop — so that is taken off the top of the link, and ABR can neither see it nor claw it back when the connection tightens. Be clear about what it buys: on game content it is very unlikely to be *audible* (256 kbps Opus is already effectively transparent, and nothing above 24 kHz is hearable at all), so the real win is **bit-exactness** — no lossy stage anywhere, and no resample for a host whose interface genuinely runs at 96 kHz. If any condition fails — the client didn't ask, the session isn't stereo, the capture device can't genuinely deliver the rate, or the link can't spare the bandwidth — the session quietly stays on Opus and the host log names which one lost. ⚠️ **The desktop clients read a variable of this same name with a richer grammar** (see [Client-side](#client-side-native-clients) below), so on a box that is both host and client, one line configures both ends. `1` is the one spelling that means *on* to each: this host gate accepts anything that isn't `0`/`false`/`off`/`no`, so a client-style `96000/24` happens to read as *allow* here too, but only `1` says the same thing on both sides. |
| `PUNKTFUNK_AUDIO_GAIN` | float (default `1.0`) | Gain applied to captured desktop audio — bump it for a quiet source. Applies to **both** the native `punktfunk/1` and Moonlight/GameStream paths. Peaks are rounded off by a soft limiter rather than clipped, so a boost distorts gracefully instead of abruptly; values above `8.0` (+18 dB) are capped, and a non-positive value is ignored. Note this buys **headroom, not loudness** — it cannot make a desktop mix as loud as already-limited streaming-app audio, and pushing it hard to try will audibly squash the signal. On Windows this is the only host-side control that works at all: loopback capture is tapped upstream of the endpoint's master volume, so the speaker slider does not affect what a client receives. |
| `PUNKTFUNK_MIC_DEVICE` | name substring | **(Windows)** Target mic-uplink device by friendly-name substring (first match wins). |
| `PUNKTFUNK_MIC_LEGACY_BUFFER` | `1` | Restore the fixed pre-adaptive mic buffering (a ~48 ms prime and ~120 ms cap on Windows; a buffer scaled to the recording app's audio quantum on Linux) instead of the adaptive per-client jitter target. One-release escape hatch: if the microphone coming out of the host only sounds right *with* this set, that's a bug — please report it. |
@@ -262,7 +263,8 @@ notes for context.
## Client-side (native clients)
A few knobs are read by the native **clients**, not the host:
A few knobs are read by the native **clients**, not the host — with one exception noted in the
table, where client and host read the *same* variable name for their own half of one feature:
| Setting | Values | Meaning |
|---|---|---|
@@ -272,6 +274,7 @@ A few knobs are read by the native **clients**, not the host:
| `PUNKTFUNK_PAD_SPEAKER_PATH` · `PUNKTFUNK_PAD_SPEAKER_VOLUME` | byte, hex or decimal *(default `0x20` / `0x7F`)* | Which output a DualSense sends [controller audio](/docs/controller-audio) to, and how loud. A controller's channel 1 is shared between its headphone jack and its built-in speaker, and it powers up pointing at the jack — so with no headphones plugged in the speaker stays silent however correctly the audio is routed. Punktfunk points it at the speaker when controller-speaker is on. Change these only if your pad's speaker stays quiet; a game that sets its own audio levels still overrides them. |
| `PUNKTFUNK_PAD_AUDIO_PROFILE` | `0` | **(Linux)** Stop the client from switching a wired DualSense's sound card to **Pro Audio** while it streams [controller audio](/docs/controller-audio) to it. The switch exists because a controller's voice coils are channels 3 and 4 of its sound card, and a controller almost never presents four channels on its own — on any other profile the haptics are folded into the speaker pair and felt as nothing. Punktfunk restores the card's profile when the session ends and never saves it. Set this if you'd rather select the card's profile yourself. |
| `PUNKTFUNK_OSD_SCALE` | multiplier, e.g. `1.5` *(default `1`)* | Size of the in-stream overlay — the stats OSD, the capture hint and the start banner. They already follow your display's scaling setting (200 % display → twice the pixels), so set this only to nudge that: bigger for a TV across the room, smaller if your compositor reports an aggressive scale. Clamped to 0.5×–4×, and a line that would run off the screen is shrunk to fit. |
| `PUNKTFUNK_AUDIO_HIRES` | `1`/`on`/`true`/`yes` · `48000` · `96000` · `<rate>/<bits>` · `0`/`off`/`false`/`no` *(unset: the client's stored audio-format choice decides)* | ⚠️ **The same name as the host's policy gate in Audio / microphone above, and a different grammar** — so one line on a box that is both host and client sets both halves. This is the **request** half, and it overrides the client's stored audio-format choice for the run. `1` asks for 96 kHz / 24-bit, the rung the plane earns its bandwidth at. A bare rate — `48000` or `96000` — asks for that rate at 24-bit. `<rate>/<bits>` names both, which is the only way to reach `48000/16`: the cheapest lossless rung (~1.5 Mbps), and one no menu offers, because at 16-bit there is nothing left to *hear* over the 256 kbps Opus it replaces — only bit-exactness. `0` forces Opus even when the stored choice asks for lossless. Anything else is a typo: the client warns and **ignores** it, so the stored choice still decides rather than being silently switched off. And asking is not getting — the host's own gate, stereo, a capture path that genuinely delivers the rate and the link budget all still have to agree, and the client plays whatever the host answers. Linux and Windows clients. |
| `PUNKTFUNK_NO_AEC` | `1` | Turn the microphone's echo cancellation off for this run, whatever **Echo cancellation** says in [client settings](/docs/client-settings#audio). One-way: it can only switch the processing off, never back on, and the setting is the normal way to control it. Linux and Windows clients. |
| `PUNKTFUNK_PRESENT_MODE` | `mailbox` *(default)* · `fifo` · `immediate` · `fifo_relaxed` | How decoded frames meet the display (the Vulkan present mode). The default prefers MAILBOX — tear-free without queueing behind the vertical refresh — and falls back to FIFO (classic vsync) where the driver doesn't offer it. **AMD's Windows driver offers no MAILBOX**, so those clients run FIFO, which adds a standing frame-pacing wait (up to one refresh interval). `immediate` removes that wait but can tear; `fifo_relaxed` only tears when a frame is late. If your latency floor matters more than tearing, try `immediate` and judge by eye. |
| `PUNKTFUNK_PRESENTER` | `arrival` | Turn the frame-pacing engine off for this run: frames present the instant they decode, exactly as they did before the **Prioritize** setting existed. A diagnostic — if a pacing change is suspected of causing judder or added delay, this switches it off without reinstalling anything. Linux and Windows clients. |
docs-site/content/docs/troubleshooting.md
+29
View File
@@ -557,6 +557,35 @@ INFO punktfunk/1 audio streaming … tier=high kbps=512 redundancy=true
`standard` reproduces the pre-0.25 encoder exactly if you want to A/B it.
If what you want is **no lossy stage at all**, there is a third knob — but read what it costs
first:
```ini
PUNKTFUNK_AUDIO_HIRES=1 # allow the lossless PCM audio plane (default off)
```
That replaces Opus with uncompressed 48/96 kHz, 16/24-bit stereo PCM. It has to be turned on at
**both** ends — the client has its own switch, also off by default — because it costs 1.5–4.6 Mbps
against Opus's 256 kbps, and like every other audio setting here that comes off the top of the
link, where adaptive bitrate can neither see it nor reclaim it.
It is also unlikely to fix the problem *this* section is about: a lossless copy of a 24 kHz mono
mix is still a 24 kHz mono mix, so fix the endpoint first. What it buys is bit-exactness rather
than audibly better sound — on game content, 256 kbps Opus is already effectively transparent. On
Windows the host reads the endpoint's own engine rate (the `engine_hz` line above) and refuses to
pad, so 96 kHz means setting that device to 96 kHz in Windows' own sound properties. A Linux host
normally owns the sink applications play into and states its rate to the audio graph itself, so
96 kHz there needs no device configuration at all. Whenever any condition fails — the client didn't
ask, the session isn't stereo, the capture path can't genuinely deliver the rate, or the link can't
spare it — the session quietly stays on Opus and the log says which one lost.
One trap if the box you are editing is *also* a client: the Linux and Windows clients read a
`PUNKTFUNK_AUDIO_HIRES` of their own, with a richer grammar — a bare rate such as `96000`, or an
explicit `96000/24` — so one line in a shared environment sets both halves at once. **`1` is the
value that means *on* to each of them**, which is why the line above is written that way. The
client's spellings are in
[Configuration → Client-side](/docs/configuration#client-side-native-clients).
## Audio lags behind the picture
The client buffers a little audio to absorb network jitter. Since 0.25 that buffer **corrects
include/punktfunk_core.h
+408 -19
View File
@@ -149,7 +149,33 @@
// and `pts_ns` of `0` — concealed audio was never on the wire and must not reach an A/V-sync
// observation. Additive and client-local: nothing new is sent or parsed, so [`WIRE_VERSION`] is
// unchanged.
#define PUNKTFUNK_ABI_VERSION 23
// v24: the lossless audio plane's client surface (`design/hi-res-audio.md` §7) —
// `punktfunk_connect_ex11` asks for a sample rate and depth (whatever
// `audio::pcm::rate_is_supported` admits — 48/96 kHz plus the 44.1 kHz family — and 16/24-bit;
// the accepted rates grew after v24 shipped, which is not an ABI change: no symbol, signature or
// struct moved, an older header stays correct, and a host that cannot carry a rate declines it to
// Opus exactly as it always has. Anything but
// the legacy pair also sets `CLIENT_CAP_AUDIO_HIRES`), and `punktfunk_connection_audio_sample_rate`
// / `punktfunk_connection_audio_bits` report what the host actually RESOLVED — which may be
// lower, because the host runs a five-condition gate and every decline lands back on Opus at
// 48 kHz. `punktfunk_connection_next_audio_pcm` decodes both planes behind the same call, using
// `pcm::PcmConceal` for gaps on the lossless one (libopus PLC extrapolates from a decoder's
// model of the signal, and a raw frame has none).
//
// ADDED, not widened, and this time the distinction has teeth: the natural place for a rate is a
// field on `PunktfunkAudioPcm`, which is `#[repr(C)]` with no `struct_size` guard and is
// allocated BY VALUE by every C embedder — growing it would change its layout under all of them
// at once. `PunktfunkStats` is in the same position. So the format is read through accessors, the
// same rule v18 set with `next_rumble_cmd2`, and `PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ` keeps its value
// and its meaning as the DEFAULT/legacy rate — a ring sized from it stays correct for every
// session that resolves to Opus, which is every session an ABI-23 embedder can ask for. An
// embedder that adopts none of this behaves exactly as before.
//
// Client-local in the C sense but NOT wire-free in the usual one: the `Hello`/`Welcome` fields
// this reads and writes landed with the plane itself, appended behind the existing trailing-field
// discipline (old peers skip them in both directions, and a legacy request encodes byte-identical
// to the pre-hi-res messages), so [`WIRE_VERSION`] is still unchanged.
#define PUNKTFUNK_ABI_VERSION 24
// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
@@ -379,6 +405,19 @@
// asked via [`PUNKTFUNK_CLIENT_CAP_PAD_AUDIO`]. (Mirrors `quic::HOST_CAP_PAD_AUDIO`.)
#define PUNKTFUNK_HOST_CAP_PAD_AUDIO 64
// Host-capability bit in [`punktfunk_connection_host_caps`]: the host resolved this session onto
// the LOSSLESS audio plane (`0xD3`) instead of Opus. Set only when the client asked via
// [`punktfunk_connect_ex11`]; it is a statement about the wire, not an offer to decline — the
// session runs one plane or the other for its whole life.
//
// A C embedder needs it for exactly one thing: telling the two planes apart when it drains raw
// frames through [`punktfunk_connection_next_audio`], because a 48 kHz/16-bit lossless session
// and a 48 kHz Opus session report identical rate, depth and channels. Embedders on
// [`punktfunk_connection_next_audio_pcm`] never need it — core decodes both planes behind it —
// but they should still read [`punktfunk_connection_audio_sample_rate`] to size their ring.
// (Mirrors `quic::HOST_CAP_AUDIO_HIRES`.)
#define PUNKTFUNK_HOST_CAP_AUDIO_HIRES 128
// Pad-audio `kind` ([`punktfunk_connection_next_pad_audio`]): the BACK channel pair — DualSense
// voice-coil haptics, 5 ms Opus frames. (Mirrors `quic::PAD_AUDIO_KIND_HAPTICS`.)
#define PUNKTFUNK_PAD_AUDIO_KIND_HAPTICS 0
@@ -412,6 +451,18 @@
// with [`PUNKTFUNK_HOST_CAP_PAD_AUDIO`]. (Mirrors `quic::CLIENT_CAP_PAD_AUDIO`.)
#define PUNKTFUNK_CLIENT_CAP_PAD_AUDIO 8
// [`punktfunk_connect_ex9`] `client_caps` bit: ask for the LOSSLESS audio plane (`0xD3`).
//
// **Normally you do not set this by hand** — pass a non-default `audio_rate_hz`/`audio_bits` to
// [`punktfunk_connect_ex11`] and core sets it for you, which keeps "the bit" and "the format it
// is asking for" from ever disagreeing. The one case that needs it explicitly is asking for
// lossless at the DEFAULT 48 kHz/16-bit, whose parameters are indistinguishable from a legacy
// request; core ORs the derived bit into what you pass rather than replacing it, so setting it
// here works. Do it only if this embedder can genuinely open a 48 kHz/16-bit output and its user
// asked for lossless — 1.5 Mbps to sound like transparent 256 kbps Opus is a poor trade, and
// 24-bit is where the plane earns its bandwidth. (Mirrors `quic::CLIENT_CAP_AUDIO_HIRES`.)
#define PUNKTFUNK_CLIENT_CAP_AUDIO_HIRES 16
// `*ttl_ms` sentinel written by [`punktfunk_connection_next_rumble2`] for a legacy (v1) rumble
// datagram — an old host that sent no self-termination lease. The client then falls back to its
// own staleness heuristic for that update instead of a host-supplied deadline.
@@ -450,11 +501,35 @@
// The protocol's audio frame, in milliseconds — every host datagram carries exactly one
// ([`crate::quic::encode_audio_datagram`]), so it is also the smallest useful shed unit.
//
// ⚠ **This is the OPUS plane's frame, and the default. It is not the only one.** The lossless
// plane negotiates shorter frames sized to the path MTU ([`pcm::frame_us_for`]): 4 ms at
// 48 kHz/24-bit and 2 ms at 96 kHz/24-bit under the default ceiling. The resolved value rides the
// `Welcome` as `audio_frame_us` and is on [`crate::client::NativeClient::audio_frame_us`].
//
// It is exported to C as `PUNKTFUNK_AUDIO_FRAME_MS` and is **kept at 5 with its meaning
// unchanged**, exactly like `PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ`: embedders size rings from it and
// removing it would be a silent C break. But sizing a ring as *frames × `PUNKTFUNK_AUDIO_FRAME_MS`*
// is wrong by up to 2.5× on a lossless session. An embedder that drains
// [`crate::abi::punktfunk_connection_next_audio_pcm`] is safe without doing anything — that call
// reports each frame's real length in `frame_count`, which is the figure to size from.
#define PUNKTFUNK_AUDIO_FRAME_MS 5
// Sample rate of every audio plane in the protocol.
#define PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ 48000
// The `0xD3` datagram's fixed header: tag + `u32` seq + `u64` pts_ns, the same shape as `0xC9`
// so the gap tracker and the A/V-sync plumbing work unchanged.
// `quic::datagram` asserts this against its own encoder.
#define PUNKTFUNK_AUDIO_PCM_HEADER_LEN ((1 + 4) + 8)
// Bit depths the plane carries. 32-bit float is deliberately absent: no source produces detail
// 24 bits does not capture, and it would cost 33 % more for nothing.
#define PUNKTFUNK_AUDIO_BITS_16 16
// See [`BITS_16`].
#define PUNKTFUNK_AUDIO_BITS_24 24
#if defined(PUNKTFUNK_FEATURE_QUIC)
// The uniform no-TTL-host staleness bound: a legacy host refreshes state every 500 ms, so two
// missed refreshes = quiet host → silence. Replaces the per-platform zoo (1.6 s / 60 s / 1.5 s /
@@ -958,6 +1033,45 @@
#define HOST_CAP_PAD_AUDIO 64
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Hello::client_caps`] bit: the client can play the LOSSLESS audio plane
// ([`AUDIO_PCM_MAGIC`](super::datagram::AUDIO_PCM_MAGIC), `0xD3`) at the rate and depth it asked
// for in [`Hello::audio_rate_hz`](super::handshake::Hello::audio_rate_hz) /
// [`audio_bits`](super::handshake::Hello::audio_bits).
//
// **Capable AND the user turned it on** — the [`VIDEO_CAP_444`] precedent, not a bare capability.
// This plane costs 1.5–4.6 Mbps against Opus's 256 kbps and is taken off the top of the link
// (audio rides datagrams outside the ABR loop, so ABR can neither see it nor reclaim it), so it
// must be asked for on both ends. A client that cannot open an output at the format it is
// requesting must not set this bit.
//
// `0x10` — `0x08` is [`CLIENT_CAP_PAD_AUDIO`], `0x04` is [`CLIENT_CAP_AUDIO_RED`], `0x02` is
// [`CLIENT_CAP_PHASE_LOCK`], `0x01` is [`CLIENT_CAP_CURSOR`]. `0x20`/`0x40`/`0x80` remain free.
#define PUNKTFUNK_CLIENT_CAP_AUDIO_HIRES 16
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Welcome::host_caps`] bit: the host resolved the session onto the lossless audio plane
// ([`AUDIO_PCM_MAGIC`](super::datagram::AUDIO_PCM_MAGIC), `0xD3`). Like [`HOST_CAP_AUDIO_RED`]
// this is a statement about the WIRE rather than an offer: with the bit set the client decodes
// `0xD3` and MUST open its device from the resolved
// [`Welcome::audio_rate_hz`](super::handshake::Welcome::audio_rate_hz) /
// [`audio_bits`](super::handshake::Welcome::audio_bits) /
// [`audio_frame_us`](super::handshake::Welcome::audio_frame_us), never from what it asked for.
//
// Unlike `0xD2`, the host does NOT drop back mid-session: the client's device is open at a fixed
// format, so a change would mean a re-open. The plane is resolved once, at handshake, by the
// five-condition gate in `design/hi-res-audio.md` §8.4, and every decline resolves to Opus
// 48 kHz with a logged reason.
//
// ⚠ `0x80` is the **LAST free `host_caps` bit**. The next host capability needs a second byte
// and an ABI bump — the same wall [`VIDEO_CAP_MULTI_SLICE`] already hit on `video_caps`.
// `0x40` is [`HOST_CAP_PAD_AUDIO`], `0x20` is [`HOST_CAP_AUDIO_RED`], `0x10` is
// [`HOST_CAP_PEN`], `0x08` is [`HOST_CAP_CURSOR`], `0x04` is [`HOST_CAP_TEXT_INPUT`],
// `0x01`/`0x02` are gamepad-state / clipboard.
#define PUNKTFUNK_HOST_CAP_AUDIO_HIRES 128
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software**
// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST
@@ -1336,6 +1450,27 @@
#define PUNKTFUNK_AUDIO_RED_HEADER (((1 + 4) + 8) + 2)
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Lossless PCM audio, host → client: `[0xD3][u32 seq LE][u64 pts_ns LE][interleaved LE samples]`.
//
// **Deliberately the same header as [`AUDIO_MAGIC`]**, so
// [`AudioGapTracker`](crate::audio::AudioGapTracker) and the pts / A-V-sync plumbing work
// unchanged and the only new logic on this plane is the payload format and its concealment
// ([`crate::audio::pcm`]).
//
// A session runs `0xC9`/`0xD2` **or** `0xD3`, never both, and never switches mid-session: the
// client's output device is open at a fixed rate and depth, so a change means a re-open. If the
// capture dies and comes back at a different format the host ends the audio plane rather than
// changing tags underneath a client that cannot follow.
//
// One frame per datagram, `audio_frame_us` long, **never fragmented** — the frame duration is
// chosen at session start by [`crate::audio::pcm::frame_us_for`] so the payload cannot exceed
// the path MTU. Redundancy ([`AUDIO_RED_MAGIC`]) is not defined for this plane and is never
// sent with it: it would double a bitrate that is already the largest on the connection, and
// `plan_audio_budget`'s ladder would never choose it.
#define PUNKTFUNK_AUDIO_PCM_MAGIC 211
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Wire length of a v1 (legacy, level) rumble datagram.
#define PUNKTFUNK_RUMBLE_V1_LEN 7
@@ -1501,6 +1636,43 @@
#define PUNKTFUNK_CIPHER_CHACHA20_POLY1305 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Welcome::audio_codec`] id: **Opus on the `0xC9` plane** — the legacy default, 48 kHz, what
// every pre-hi-res build sends and what every declined hi-res negotiation resolves back to
// (`design/hi-res-audio.md` §8.4: a fallback to today's transparent 256 kbps Opus is not a
// defeat; silence is the one unacceptable outcome). `0`, so an absent field and an older host
// both read as Opus and the common Welcome stays byte-identical to the pre-hi-res wire form.
#define PUNKTFUNK_AUDIO_CODEC_OPUS 0
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Welcome::audio_codec`] id **reserved for FLAC** on the `0xD3` plane — deliberately reserved
// and deliberately **unimplemented**.
//
// The design doc numbers the audio codecs `0` = Opus, `1` = FLAC, `2` = PCM, and this project has
// been bitten before by wire ids that drifted from the document that specifies them. The id is
// therefore burned rather than compacted: `AUDIO_CODEC_PCM` is `2` because the doc says `2`.
//
// FLAC lost on the merits, and the reasoning is recorded in `crate::audio::pcm`'s module docs so
// it does not have to be re-derived: the plane is never fragmented, so a frame must be sized from
// the codec's WORST case (a FLAC VERBATIM subframe — raw samples plus a header), which means FLAC
// and PCM negotiate the same frame duration, the same packet rate and the same send-buffer
// sizing. A codec would buy average bytes on a plane that is provisioned for peak, at the cost of
// a new dependency in the NDK / xcframework / flatpak / MSIX / Arch packaging targets. No host
// emits this id and no client should accept it; it exists so that a future one could.
#define PUNKTFUNK_AUDIO_CODEC_FLAC_RESERVED 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Welcome::audio_codec`] id: **raw interleaved LE PCM on the `0xD3` plane** (`crate::audio::pcm`)
// — the lossless format this negotiation exists to reach, at the resolved
// [`Welcome::audio_rate_hz`] / [`audio_bits`](Welcome::audio_bits) /
// [`audio_frame_us`](Welcome::audio_frame_us).
//
// `2` rather than `1` because [`AUDIO_CODEC_FLAC_RESERVED`] holds `1` — see there.
#define PUNKTFUNK_AUDIO_CODEC_PCM 2
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`PairRequest`].
#define PUNKTFUNK_MSG_PAIR_REQUEST 16
@@ -2007,8 +2179,16 @@ typedef struct {
} PunktfunkStats;
#if defined(PUNKTFUNK_FEATURE_QUIC)
// One Opus audio packet pulled off a `punktfunk/1` connection (48 kHz stereo, 5 ms frames).
// One audio packet pulled off a `punktfunk/1` connection — an Opus frame (48 kHz, 5 ms) on
// every ordinary session, or one lossless PCM frame on a session that resolved the `0xD3` plane.
// `data` borrows connection memory until the next `punktfunk_connection_next_audio` call.
//
// Nothing here says which: the plane is a property of the SESSION, read once via
// `punktfunk_connection_host_caps() & PUNKTFUNK_HOST_CAP_AUDIO_HIRES` (with the format itself
// from [`punktfunk_connection_audio_sample_rate`] / [`punktfunk_connection_audio_bits`]). An
// embedder that never asks for the lossless plane can only ever be handed Opus, so this struct's
// meaning is unchanged for it — and one that does is far better off on
// [`punktfunk_connection_next_audio_pcm`], which decodes both planes in core.
typedef struct {
const uint8_t *data;
uintptr_t len;
@@ -2019,9 +2199,19 @@ typedef struct {
#if defined(PUNKTFUNK_FEATURE_QUIC)
// One decoded audio frame from [`punktfunk_connection_next_audio_pcm`]: interleaved 32-bit
// float PCM at 48 kHz, in the canonical wire channel order `FL FR FC LFE RL RR SL SR` (the
// first `channels` of it). `samples` points at `frame_count * channels` floats and borrows
// float PCM in the canonical wire channel order `FL FR FC LFE RL RR SL SR` (the first
// `channels` of it). `samples` points at `frame_count * channels` floats and borrows
// connection memory **until the next PCM call** on this handle.
//
// **The sample rate is not in this struct and never will be.** It was 48 kHz for every session
// until the lossless plane, and it is still 48 kHz for every Opus one — but a hi-res session
// resolves its own rate and depth, and this is a `#[repr(C)]` type with no `struct_size` guard
// that C embedders allocate BY VALUE. Adding a field would silently change its layout under
// every one of them. So the format is read through
// [`punktfunk_connection_audio_sample_rate`] / [`punktfunk_connection_audio_bits`] instead —
// accessors ADDED, not structs widened, which is the rule ABI 18 set with
// `punktfunk_connection_next_rumble_cmd2`. An embedder that never calls them keeps sizing its
// ring for 48 kHz, which is exactly right for the sessions it can already play.
typedef struct {
// Interleaved f32 samples (wire channel order), `frame_count * channels` long.
const float *samples;
@@ -2308,6 +2498,41 @@ typedef struct {
// Frame durations the plane may negotiate, longest first.
//
// Every rung divides the **48 kHz family** into a whole number of samples per channel, so on
// those rates the host pacer and the client ring carry an exact frame:
//
// | µs | samples/ch @48 kHz | samples/ch @96 kHz |
// |---|---|---|
// | 5000 | 240 | 480 |
// | 4000 | 192 | 384 |
// | 3000 | 144 | 288 |
// | 2500 | 120 | 240 |
// | 2000 | 96 | 192 |
// | 1500 | 72 | 144 |
// | 1000 | 48 | 96 |
//
// ⚠⚠ **The 44.1 kHz family does not divide, and this doc used to claim every rate did.** A rung
// lands on a whole sample only when `rate_hz × µs` is a multiple of 1 000 000, which needs a
// multiple of **10 000 µs** at 44 100 Hz, **5 000 µs** at 88 200 and **2 500 µs** at 176 400. So
// of the seven rungs, 44 100 has **none**, 88 200 has only 5 000, and 176 400 has 5 000 and
// 2 500. Every other pairing carries [`samples_per_frame`]'s FLOOR and is therefore *shorter*
// than the rung it is labelled with: 5 ms at 44 100 Hz is 220 samples per channel — 4 988 662 ns,
// 0.23 % short.
//
// That is safe for the two things this ladder decides, and unsafe for a third:
//
// - **Payload sizing** — a floored frame is *fewer* bytes, so [`frame_us_for`]'s fit against the
// datagram holds with margin rather than being eroded (the payload must never exceed the
// datagram; that invariant is absolute and this rounds the right way for it).
// - **Buffer sizing** — both ends size from [`samples_per_frame`], so they agree by construction.
// - **⚠ Timing — no.** A rung is a *nominal* length for the wire and the ring, never a duration.
// Anything advancing a `pts_ns` must use [`frame_duration_ns`] of the frame's real sample
// count; adding 5 000 µs to a frame that carries 4 988 662 ns runs the clock 0.23 % fast
// forever, and the A/V sync loop will fight that drift and never win.
#define PUNKTFUNK_AUDIO_FRAME_US_LADDER { 5000, 4000, 3000, 2500, 2000, 1500, 1000, }
// What a controller sitting still, face up, actually puts on the wire: **1 g along the UP
// axis** — which is index 1 — and nothing on the other two.
//
@@ -2745,6 +2970,77 @@ PunktfunkConnection *punktfunk_connect_ex10(const char *host,
int32_t *status_out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Like [`punktfunk_connect_ex10`], plus the audio format this client is **asking** for (ABI v24):
// `audio_rate_hz` — `48000`, `96000`, or the 44.1 kHz family `44100` / `88200` / `176400` — and
// `audio_bits` (`16` or `24`).
//
// Passing anything other than `48000`/`16` sets `CLIENT_CAP_AUDIO_HIRES` in the `Hello` and asks
// the host for the LOSSLESS `0xD3` plane — bit-exact PCM instead of Opus. That is an opt-in on
// both ends, and it is meant to be: it costs **1.5–4.6 Mbps** taken off the top of the link
// (audio rides QUIC datagrams outside the ABR loop, so ABR can neither see it nor reclaim it),
// against the ~256 kbps Opus this replaces. Only call it with a non-default format when the
// user turned the feature on AND this embedder can genuinely open an output device at it.
//
// **The request is not the answer.** The host runs a five-condition gate
// (`design/hi-res-audio.md` §8.4 — client asked, operator policy allows, stereo, the capture
// path can *really* deliver the rate, and the link can afford it) and any failure resolves the
// session back to Opus at 48 kHz. That is not an error and the connect still succeeds. Read
// [`punktfunk_connection_audio_sample_rate`] / [`punktfunk_connection_audio_bits`] afterwards
// and open the device from THOSE — opening at what you asked for is
// `design/hi-res-audio.md` §4.3's failure repeated at the client end.
//
// ⚠ **Passing `48000`/`16` is NOT the same as [`punktfunk_connect_ex10`], and an earlier version
// of this comment claimed it was.** `ex10` passes `0`/`0` — *unspecified* — and the capability bit
// keys on "the caller specified a format", not on "the format differs from the default". So an
// explicit 48 kHz/16-bit is a genuine request for the cheapest lossless rung: the `Hello` carries
// [`quic::CLIENT_CAP_AUDIO_HIRES`](crate::quic::CLIENT_CAP_AUDIO_HIRES), the host's gate accepts
// 48/16 as a supported format, and a host with the operator policy on will resolve the session
// onto the lossless plane at 1.5 Mbps.
//
// That is the intended behaviour — 48/16 would otherwise be the one rung on the ladder nobody
// could ask for — but it makes `0`/`0` load-bearing. **An embedder whose user chose "Opus" must
// pass `0`/`0` here, or call [`punktfunk_connect_ex10`].** Forwarding a hardcoded 48 000/16 as a
// stand-in for "default" silently opts every ordinary session into the lossless plane.
//
// The 44.1 kHz family is on the ladder, and was not always: core's de-jitter policy divided the
// rate by 1 000 before it multiplied, which made 44 100 Hz "44 samples per millisecond" and every
// buffer figure 2.3 % low, so §4.1 deferred those rates behind fixing that arithmetic. It is
// fixed. Note what it does NOT buy: at 44 100 Hz **no** rung of the frame ladder is a whole
// number of samples, so [`punktfunk_connection_audio_frame_us`] becomes a nominal length rather
// than a duration. An embedder that advances any clock by it runs 0.23 % fast forever; the honest
// figure is the samples it actually rendered, divided by
// [`punktfunk_connection_audio_sample_rate`].
//
// A NEW symbol, not a widened one — `ex10` keeps its parameter list AND its behaviour.
//
// # Safety
// Same as [`punktfunk_connect_ex10`].
PunktfunkConnection *punktfunk_connect_ex11(const char *host,
uint16_t port,
uint32_t width,
uint32_t height,
uint32_t refresh_hz,
uint32_t compositor,
uint32_t gamepad,
uint32_t bitrate_kbps,
uint8_t video_caps,
uint8_t audio_channels,
uint32_t audio_rate_hz,
uint8_t audio_bits,
uint8_t video_codecs,
uint8_t preferred_codec,
uint8_t client_caps,
const char *launch_id,
const uint8_t *pin_sha256,
uint8_t *observed_sha256_out,
const char *client_cert_pem,
const char *client_key_pem,
const char *device_name,
uint32_t timeout_ms,
int32_t *status_out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Generate a persistent client identity: a self-signed certificate + private key, both
// PEM, NUL-terminated, written into the caller's buffers. Generate ONCE, store both
@@ -2810,11 +3106,12 @@ PunktfunkStatus punktfunk_connection_next_au(PunktfunkConnection *c,
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Pull the next Opus audio packet, waiting up to `timeout_ms`. Returns
// [`PunktfunkStatus::NoFrame`] on timeout and [`PunktfunkStatus::Closed`] once the session ended.
// On `Ok`, `out->data` borrows connection memory **until the next audio call** on this
// handle (independent of the video slot). Drain from a dedicated audio thread — packets
// arrive every 5 ms and the internal queue holds 320 ms.
// Pull the next audio packet (see [`PunktfunkAudioPacket`] for what it holds), waiting up to
// `timeout_ms`. Returns [`PunktfunkStatus::NoFrame`] on timeout and [`PunktfunkStatus::Closed`]
// once the session ended. On `Ok`, `out->data` borrows connection memory **until the next audio
// call** on this handle (independent of the video slot). Drain from a dedicated audio thread —
// Opus packets arrive every 5 ms (lossless ones every 1–5 ms, per the negotiated frame length)
// and the internal queue holds 320 ms.
//
// # Safety
// `c` is a valid connection handle; `out` is writable. At most one thread pulls audio —
@@ -2837,6 +3134,83 @@ PunktfunkStatus punktfunk_connection_next_audio(PunktfunkConnection *c,
PunktfunkStatus punktfunk_connection_audio_channels(PunktfunkConnection *c, uint8_t *out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Read the sample rate the host resolved for this session (from its Welcome): `48000` for every
// Opus session — and for every host older than the lossless plane — or the rate a hi-res session
// actually landed on, which may be LOWER than the client asked for. `*out` is filled when
// non-NULL. Available immediately after a successful connect; it never changes mid-session.
//
// ⚠ **Open the output device from this, not from `PUNKTFUNK_AUDIO_SAMPLE_RATE_HZ`.** That
// compile-time constant keeps its value and its meaning — it is the DEFAULT/legacy rate, and
// every ring sized from it is still correct for every session that resolves to Opus — but on a
// hi-res session it is simply not the rate on the wire. Opening at 96 kHz because you asked for
// 96 kHz, when the host answered 48 kHz, is `design/hi-res-audio.md` §4.3's failure repeated at
// the other end of the link: everything audits clean and the content is wrong.
//
// An ACCESSOR rather than a field on [`PunktfunkAudioPcm`] or `PunktfunkStats`: both are
// `#[repr(C)]` with no `struct_size` guard and are allocated by value by C embedders, so growing
// either would break every one of them at once. Same rule ABI 18 set with
// `punktfunk_connection_next_rumble_cmd2` — added, not widened — so an embedder that never calls
// this behaves exactly as it did before it existed.
//
// # Safety
// `c` is a valid connection handle; `out` is NULL or writable for one `u32`.
PunktfunkStatus punktfunk_connection_audio_sample_rate(PunktfunkConnection *c,
uint32_t *out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Read the sample depth the host resolved for this session (from its Welcome): `16` on every
// Opus session and every older host, `16` or `24` on the lossless plane. `*out` is filled when
// non-NULL. Available immediately after a successful connect; it never changes mid-session.
//
// Only sessions on the lossless plane can report `24`, and only they need it: the depth is the
// stride the `0xD3` payload is unpacked at, and [`punktfunk_connection_next_audio_pcm`] already
// does that unpacking in core — it hands out f32 either way. This is here so an embedder can
// *report* the format honestly (a UI that says "24-bit" while the host declined is the same
// class of lie as claiming a rate you did not get) and so one draining raw frames through
// [`punktfunk_connection_next_audio`] can unpack them itself. Which PLANE a session is on is
// `punktfunk_connection_host_caps() & PUNKTFUNK_HOST_CAP_AUDIO_HIRES`, not this: 48 kHz/16-bit
// reads identically on both.
//
// ADDED, not widened — see [`punktfunk_connection_audio_sample_rate`] for why.
//
// # Safety
// `c` is a valid connection handle; `out` is NULL or writable for one `u8`.
PunktfunkStatus punktfunk_connection_audio_bits(PunktfunkConnection *c, uint8_t *out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// The resolved audio frame length in MICROSECONDS — how much audio one datagram carries.
//
// `PUNKTFUNK_AUDIO_FRAME_MS` is the Opus plane's 5 ms and stays that way, but the lossless plane
// sizes its frame to the path MTU: 4 ms at 48 kHz/24-bit and 2 ms at 96 kHz/24-bit under the
// default ceiling. An embedder that ports the de-jitter policy (rather than draining
// [`punktfunk_connection_next_audio_pcm`] and letting core do it) needs the real figure — the
// shed drops exactly one frame and the target floor is a device quantum plus one frame, so a
// policy compiled against 5 ms sheds 2.5 frames at a time on a 96 kHz session.
//
// Microseconds rather than milliseconds because the ladder has sub-millisecond rungs; `0` means
// the host did not state one, in which case `PUNKTFUNK_AUDIO_FRAME_MS × 1000` is correct.
//
// ⚠⚠ **A NOMINAL length, not a duration, and on the 44.1 kHz family they differ.** A frame
// carries a whole number of samples per channel, and 44 100 Hz divides no rung of the ladder: a
// 5 ms frame there is 220 samples per channel, which is 4 988 662 ns. Sizing a ring from this is
// right (that is what it is for); advancing a **clock** by it is not — it invents 2.3 ms of time
// per second, indefinitely, and every stat downstream will agree with the lie because the
// timestamps stay self-consistent. Derive elapsed time from the samples rendered and
// [`punktfunk_connection_audio_sample_rate`] instead.
//
// **Not derivable from `next_audio_pcm`'s `frame_count`.** That call prepends concealed frames
// into the same buffer, so its count is "how many samples you got", not "how long one frame is".
//
// ADDED, not widened — see [`punktfunk_connection_audio_sample_rate`] for why.
//
// # Safety
// `c` is a valid connection handle; `out` is NULL or writable for one `u16`.
PunktfunkStatus punktfunk_connection_audio_frame_us(PunktfunkConnection *c, uint16_t *out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// WHY this session ended: `*out` receives a [`PunktfunkEndReason`] byte
// (`PUNKTFUNK_END_REASON_*`). The return status reports only whether the handle was usable.
@@ -2869,14 +3243,25 @@ PunktfunkStatus punktfunk_connection_end_reason(PunktfunkConnection *c, uint8_t
// [`punktfunk_connection_next_audio`] on a given connection, from one dedicated audio thread —
// not both (they share the underlying queue).
//
// **Both audio planes come out of this one call.** On a session that resolved the lossless
// `0xD3` plane there is no Opus decoder at all — the samples are unpacked at the negotiated
// depth — but the output is the same interleaved f32 in the same borrowed buffer, so an embedder
// needs no branch. It DOES need [`punktfunk_connection_audio_sample_rate`] to size its ring and
// open its device: `frame_count` is samples per channel, and at 96 kHz they arrive twice as fast.
//
// **Loss concealment**: packets the wire lost (a gap in the sequence, after the redundant-plane
// recovery has had its chance) are synthesized via libopus packet-loss concealment and returned
// IN FRONT of the arriving frame in the same buffer — `out->frame_count` then covers the
// concealed frames plus the real one (`out->seq`/`out->pts_ns` are the real packet's). The
// embedder just writes the whole buffer to its ring, same as any other frame; gaps arrive
// pre-healed, exactly as they do on the clients that decode outside core. That covers a gap a
// LATER packet reveals; when the wire goes quiet instead, see
// [`punktfunk_connection_audio_plc`].
// recovery has had its chance) are synthesized and returned IN FRONT of the arriving frame in
// the same buffer — `out->frame_count` then covers the concealed frames plus the real one
// (`out->seq`/`out->pts_ns` are the real packet's). The embedder just writes the whole buffer to
// its ring, same as any other frame; gaps arrive pre-healed, exactly as they do on the clients
// that decode outside core. That covers a gap a LATER packet reveals; when the wire goes quiet
// instead, see [`punktfunk_connection_audio_plc`].
//
// What synthesizes them differs by plane, and has to: Opus gaps use libopus PLC, which
// extrapolates from the decoder's model of the signal. A lossless frame has no such model — only
// the signal — so `0xD3` gaps are concealed by repeating the last good frame under a raised-cosine
// fade, decaying to silence across a sustained gap (`design/hi-res-audio.md` §4.5). Same shape,
// same buffer, same cap; only the material differs.
//
// # Safety
// `c` is a valid connection handle; `out` is writable. At most one thread pulls audio.
@@ -2904,9 +3289,13 @@ PunktfunkStatus punktfunk_connection_next_audio_pcm(PunktfunkConnection *c,
// to duplicate, pushing the stream permanently later. Core supplies only the mechanism, one frame
// per call, at the cadence the embedder drains at.
//
// Returns [`PunktfunkStatus::NoFrame`] when nothing has decoded yet — PLC extrapolates from the
// last decoded frame, so before there is one there is no state to extrapolate from — and if
// libopus declines to interpolate. Both mean "write nothing this tick", exactly like a timeout.
// Works on both audio planes, using each one's own concealer — libopus PLC on `0xC9`, the
// repeat-and-fade of [`crate::audio::pcm::PcmConceal`] on `0xD3` (a lossless frame carries no
// model of the signal for PLC to extrapolate from; `design/hi-res-audio.md` §4.5).
//
// Returns [`PunktfunkStatus::NoFrame`] when nothing has decoded yet — both concealers build on
// the last decoded frame, so before there is one there is nothing to build from — and if libopus
// declines to interpolate. Both mean "write nothing this tick", exactly like a timeout.
//
// `out->seq` and `out->pts_ns` read 0: this frame was never on the wire, so it has no sequence
// number and no capture instant, and it must never be fed to an A/V-sync observation.