feat(protocol,clients): codec preference negotiation + Linux client decodes per Welcome (Phase 2a)

Adds a client-selectable **preferred codec** and wires the core + ABI + probe + Linux client to
negotiate and decode it. (Windows/Apple/Android follow in 2b.)

**Core:**
- `Hello.preferred_codec` (a single CODEC_* bit, 0 = auto) — a soft hint appended after
  `video_codecs`. `resolve_codec(client, host, preferred)` now honors the preference when the host
  can also emit it, else falls back to precedence (HEVC > AV1 > H.264). Roundtrip + preference tests.
- `NativeClient::connect` takes `video_codecs` + `preferred_codec`; `NativeClient.codec` exposes the
  resolved `Welcome.codec`.
- ABI: `punktfunk_connect_ex7` (adds the two codec params; `ex6` delegates to it advertising
  HEVC-only) + `punktfunk_connection_codec` getter + `PUNKTFUNK_CODEC_{H264,HEVC,AV1}` constants
  (drift-guarded against the wire values). Header regenerated.

**Host:** passes `hello.preferred_codec` into `resolve_codec`.

**probe:** `--codec h264|hevc|av1|auto` sets the preference (still advertises it can decode all
three); the dump extension already follows the resolved codec.

**Linux client:** advertises the codecs FFmpeg can actually decode (`decodable_codecs()`), threads
the user's `codec` setting as the preference, and builds the decoder — both the software and VAAPI
paths, plus the mid-session VAAPI→software demotion — from the negotiated `Welcome.codec` instead of
hardcoding HEVC. New "Video codec" dropdown in Preferences (Automatic/HEVC/H.264/AV1).

Live-validated on the dev box: probe `--codec hevc` against a software (H.264-only) host resolves to
H.264 (graceful soft-preference fallback), no failure. clippy + core (57) + host (133) tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-02 00:13:26 +00:00
parent ffc0b07b46
commit 12843fe253
36 changed files with 529 additions and 144 deletions
+98 -2
View File
@@ -844,12 +844,23 @@ pub const PUNKTFUNK_VIDEO_CAP_HDR: u8 = 0x02;
/// [`punktfunk_connection_chroma_format`] reports the real value. (Mirrors `quic::VIDEO_CAP_444`.)
pub const PUNKTFUNK_VIDEO_CAP_444: u8 = 0x04;
/// Codec bit for [`punktfunk_connect_ex7`] (`video_codecs` / `preferred_codec`) and the value
/// [`punktfunk_connection_codec`] returns: H.264 / AVC. (Mirrors `quic::CODEC_H264`.)
pub const PUNKTFUNK_CODEC_H264: u8 = 0x01;
/// Codec bit: H.265 / HEVC — the default codec. (Mirrors `quic::CODEC_HEVC`.)
pub const PUNKTFUNK_CODEC_HEVC: u8 = 0x02;
/// Codec bit: AV1. (Mirrors `quic::CODEC_AV1`.)
pub const PUNKTFUNK_CODEC_AV1: u8 = 0x04;
// Keep the ABI cap bits in lockstep with the wire constants (compile-time guard against drift).
#[cfg(feature = "quic")]
const _: () = {
assert!(PUNKTFUNK_VIDEO_CAP_10BIT == crate::quic::VIDEO_CAP_10BIT);
assert!(PUNKTFUNK_VIDEO_CAP_HDR == crate::quic::VIDEO_CAP_HDR);
assert!(PUNKTFUNK_VIDEO_CAP_444 == crate::quic::VIDEO_CAP_444);
assert!(PUNKTFUNK_CODEC_H264 == crate::quic::CODEC_H264);
assert!(PUNKTFUNK_CODEC_HEVC == crate::quic::CODEC_HEVC);
assert!(PUNKTFUNK_CODEC_AV1 == crate::quic::CODEC_AV1);
};
// Keep the ABI gamepad constants in lockstep with the wire enum (compile-time guard against drift).
@@ -1160,8 +1171,8 @@ pub unsafe extern "C" fn punktfunk_connect_ex5(
/// Like [`punktfunk_connect_ex5`], but additionally requests the audio channel count:
/// `2` (stereo, the default behaviour of every earlier variant), `6` (5.1) or `8` (7.1). The host
/// clamps the request to what it can actually capture and echoes the resolved count via
/// [`punktfunk_connection_audio_channels`]; the `0xC9` audio frames are Opus-(multi)stream encoded
/// for that layout. A client that wants surround calls this; everything else inherits stereo.
/// [`punktfunk_connection_audio_channels`]. Advertises HEVC-only with no codec preference (call
/// [`punktfunk_connect_ex7`] to negotiate the codec).
///
/// # Safety
/// Same as [`punktfunk_connect`].
@@ -1185,6 +1196,62 @@ pub unsafe extern "C" fn punktfunk_connect_ex6(
client_cert_pem: *const std::os::raw::c_char,
client_key_pem: *const std::os::raw::c_char,
timeout_ms: u32,
) -> *mut PunktfunkConnection {
unsafe {
punktfunk_connect_ex7(
host,
port,
width,
height,
refresh_hz,
compositor,
gamepad,
bitrate_kbps,
video_caps,
audio_channels,
PUNKTFUNK_CODEC_HEVC, // pre-negotiation default: HEVC-only, no preference
0,
launch_id,
pin_sha256,
observed_sha256_out,
client_cert_pem,
client_key_pem,
timeout_ms,
)
}
}
/// Like [`punktfunk_connect_ex6`], but additionally advertises the codecs the client can decode
/// (`video_codecs` — a bitfield of [`PUNKTFUNK_CODEC_H264`] / [`PUNKTFUNK_CODEC_HEVC`] /
/// [`PUNKTFUNK_CODEC_AV1`]) and a soft `preferred_codec` (a single codec bit, `0` = no preference).
/// The host resolves the codec it emits from these (preference honored when it can also produce it,
/// else best shared codec) and reports it via [`punktfunk_connection_codec`]. A client that omits
/// this (calls `ex6`) advertises HEVC-only, no preference — the pre-negotiation behaviour.
///
/// # Safety
/// Same as [`punktfunk_connect`].
#[cfg(feature = "quic")]
#[no_mangle]
#[allow(clippy::too_many_arguments)]
pub unsafe extern "C" fn punktfunk_connect_ex7(
host: *const std::os::raw::c_char,
port: u16,
width: u32,
height: u32,
refresh_hz: u32,
compositor: u32,
gamepad: u32,
bitrate_kbps: u32,
video_caps: u8,
audio_channels: u8,
video_codecs: u8,
preferred_codec: u8,
launch_id: *const std::os::raw::c_char,
pin_sha256: *const u8,
observed_sha256_out: *mut u8,
client_cert_pem: *const std::os::raw::c_char,
client_key_pem: *const std::os::raw::c_char,
timeout_ms: u32,
) -> *mut PunktfunkConnection {
let r = std::panic::catch_unwind(AssertUnwindSafe(|| {
if host.is_null() {
@@ -1235,6 +1302,8 @@ pub unsafe extern "C" fn punktfunk_connect_ex6(
bitrate_kbps,
video_caps,
crate::audio::normalize_channels(audio_channels),
video_codecs,
preferred_codec,
launch,
pin,
identity,
@@ -1763,6 +1832,33 @@ pub unsafe extern "C" fn punktfunk_connection_chroma_format(
})
}
/// Read the video codec the host resolved for this session (from its Welcome): one of
/// [`PUNKTFUNK_CODEC_H264`] / [`PUNKTFUNK_CODEC_HEVC`] / [`PUNKTFUNK_CODEC_AV1`]. The embedder builds
/// its decoder from THIS (never assuming HEVC). `*out` is filled when non-NULL. Available
/// immediately after a successful connect (it doesn't change without a reconfigure). An older host
/// that didn't negotiate a codec reports [`PUNKTFUNK_CODEC_HEVC`].
///
/// # Safety
/// `c` is a valid connection handle; `out` is NULL or writable for one `u8`.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_codec(
c: *mut PunktfunkConnection,
out: *mut u8,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if !out.is_null() {
// SAFETY: `out` is non-null and the caller guarantees it is writable for one `u8`.
unsafe { *out = c.inner.codec };
}
PunktfunkStatus::Ok
})
}
/// Send one input event to the host as a QUIC datagram (non-blocking enqueue).
///
/// # Safety
+27 -6
View File
@@ -41,8 +41,8 @@ enum CtrlRequest {
/// certificate fingerprint, the resolved encoder bitrate (kbps), and the host↔client clock offset
/// (ns, host minus client; 0 = no skew correction / an old host that didn't answer the handshake).
/// The trailing `u8`s are the resolved encode bit depth (8/10), the chroma `chroma_format_idc`
/// (1 = 4:2:0, 3 = 4:4:4), and the resolved audio channel count (2/6/8), with [`ColorInfo`] the
/// resolved colour signalling — all from the [`Welcome`].
/// (1 = 4:2:0, 3 = 4:4:4), the resolved audio channel count (2/6/8), and the resolved video codec
/// (`quic::CODEC_*`), with [`ColorInfo`] the resolved colour signalling — all from the [`Welcome`].
type Negotiated = (
Mode,
CompositorPref,
@@ -54,6 +54,7 @@ type Negotiated = (
ColorInfo,
u8,
u8,
u8,
);
/// Accumulated state of an in-flight / finished speed test. The data-plane pump mirrors the
@@ -216,6 +217,10 @@ pub struct NativeClient {
/// host that omits it (→ `2`) yields working stereo. The `0xC9` audio frames are encoded with the
/// matching layout.
pub audio_channels: u8,
/// The video codec the host resolved and will emit ([`Welcome::codec`]) — [`quic::CODEC_H264`],
/// [`quic::CODEC_HEVC`] (default / older host), or [`quic::CODEC_AV1`]. The client builds its
/// decoder from THIS, never assuming HEVC.
pub codec: u8,
}
/// Pin the calling thread to the user-interactive QoS class on Apple targets.
@@ -263,6 +268,11 @@ impl NativeClient {
// Requested audio channel count (2 = stereo / 6 = 5.1 / 8 = 7.1); the host clamps to what it
// can capture and echoes the result in [`NativeClient::audio_channels`].
audio_channels: u8,
// The codecs this client can decode (bitfield of quic::CODEC_H264 / CODEC_HEVC / CODEC_AV1)
// and the user's soft preference (a single codec bit, 0 = auto). The host resolves the codec
// it emits from these and echoes it in [`NativeClient::codec`].
video_codecs: u8,
preferred_codec: u8,
launch: Option<String>,
pin: Option<[u8; 32]>,
identity: Option<(String, String)>,
@@ -316,6 +326,8 @@ impl NativeClient {
bitrate_kbps,
video_caps,
audio_channels,
video_codecs,
preferred_codec,
launch,
pin,
identity,
@@ -349,6 +361,7 @@ impl NativeClient {
color,
chroma_format,
audio_channels,
codec,
) = match ready_rx.recv_timeout(timeout) {
Ok(Ok(t)) => t,
Ok(Err(e)) => return Err(e),
@@ -382,6 +395,7 @@ impl NativeClient {
color,
chroma_format,
audio_channels,
codec,
})
}
@@ -689,6 +703,8 @@ struct WorkerArgs {
bitrate_kbps: u32,
video_caps: u8,
audio_channels: u8,
video_codecs: u8,
preferred_codec: u8,
launch: Option<String>,
pin: Option<[u8; 32]>,
identity: Option<(String, String)>,
@@ -721,6 +737,8 @@ async fn worker_main(args: WorkerArgs) {
bitrate_kbps,
video_caps,
audio_channels,
video_codecs,
preferred_codec,
launch,
pin,
identity,
@@ -789,10 +807,10 @@ async fn worker_main(args: WorkerArgs) {
video_caps,
// Requested surround channel count; the host echoes the resolved value in Welcome.
audio_channels,
// Phase 1: the embeddable clients decode HEVC (their decoders are still HEVC-wired),
// so advertise HEVC-only until Phase 2 threads real per-client codec caps through the
// connect ABI and switches decoders on `Welcome::codec`.
video_codecs: crate::quic::CODEC_HEVC,
// The codecs this client can decode + its soft preference (0 = auto). The host
// resolves the emitted codec from these and reports it in `Welcome::codec`.
video_codecs,
preferred_codec,
}
.encode(),
)
@@ -866,6 +884,7 @@ async fn worker_main(args: WorkerArgs) {
welcome.color,
welcome.chroma_format,
welcome.audio_channels,
welcome.codec,
))
};
@@ -884,6 +903,7 @@ async fn worker_main(args: WorkerArgs) {
color,
chroma_format,
audio_channels,
codec,
) = match setup.await {
Ok(t) => t,
Err(e) => {
@@ -902,6 +922,7 @@ async fn worker_main(args: WorkerArgs) {
color,
chroma_format,
audio_channels,
codec,
)));
// Input task: embedder events → QUIC datagrams.
+88 -22
View File
@@ -94,6 +94,13 @@ pub struct Hello {
/// clients (decodes to `0`, which [`resolve_codec`] treats as HEVC-only — every pre-negotiation
/// build decoded HEVC).
pub video_codecs: u8,
/// The client's *preferred* codec (a single [`CODEC_H264`] / [`CODEC_HEVC`] / [`CODEC_AV1`] bit),
/// or `0` = no preference (host decides by its own precedence). A **soft** hint: the host emits
/// it when it can also produce it (and the client advertised it in `video_codecs`), else falls
/// back to the best shared codec — see [`resolve_codec`]. Mirrors the [`Hello::compositor`] /
/// [`Hello::gamepad`] preference pattern; the resolved codec is echoed in [`Welcome::codec`].
/// Appended after `video_codecs` as a single trailing byte. Omitted by older clients (→ `0`).
pub preferred_codec: u8,
}
/// [`Hello::video_caps`] bit: the client can decode a 10-bit (Main10) HEVC stream.
@@ -120,12 +127,13 @@ pub const CODEC_AV1: u8 = 0x04;
/// Resolve which single codec the host will emit, from the client's advertised [`Hello::video_codecs`]
/// bitfield (`0` = an older client, treated as HEVC-only) intersected with what the host's chosen
/// encoder can produce (`host_capable`, also a bitfield). Precedence when several are shared:
/// **HEVC > AV1 > H.264** (HEVC is the established, best-tested path; H.264 is the compatibility /
/// software floor). Returns the single-bit codec value, or `None` when the two share nothing — the
/// caller then refuses the session with a clear error rather than emitting a stream the client can't
/// decode.
pub fn resolve_codec(client_codecs: u8, host_capable: u8) -> Option<u8> {
/// encoder can produce (`host_capable`, also a bitfield). `preferred` is the client's soft preference
/// ([`Hello::preferred_codec`], `0` = none): when it's in the shared set it wins; otherwise the tie is
/// broken by **HEVC > AV1 > H.264** (HEVC is the established, best-tested path; H.264 is the
/// compatibility / software floor). Returns the single-bit codec value, or `None` when client and host
/// share nothing — the caller then refuses the session with a clear error rather than emitting a
/// stream the client can't decode.
pub fn resolve_codec(client_codecs: u8, host_capable: u8, preferred: u8) -> Option<u8> {
// An older client (no codec byte) decodes HEVC — the only codec every pre-negotiation build sent.
let client = if client_codecs == 0 {
CODEC_HEVC
@@ -133,6 +141,13 @@ pub fn resolve_codec(client_codecs: u8, host_capable: u8) -> Option<u8> {
client_codecs
};
let shared = client & host_capable;
if shared == 0 {
return None;
}
// Honor the client's preference when the host can also emit it; else fall back to precedence.
if preferred != 0 && shared & preferred != 0 {
return Some(preferred);
}
// Precedence: HEVC > AV1 > H.264.
[CODEC_HEVC, CODEC_AV1, CODEC_H264]
.into_iter()
@@ -716,8 +731,13 @@ impl Hello {
// present (video_caps non-zero / audio_channels not stereo) the name/launch length bytes
// AND the video_caps byte must still be emitted (0 / 0) so the later byte lands at a
// deterministic offset — the same discipline `launch` already imposes on `name`.
// Trailing single-byte fields, in wire order. Each is emitted when it (or ANY later field)
// carries a non-default value, so a present field always lands at a deterministic offset.
let ac_present = self.audio_channels != 2;
let vcodecs_present = self.video_codecs != 0;
let pref_present = self.preferred_codec != 0;
let need_placeholders =
self.video_caps != 0 || self.audio_channels != 2 || self.video_codecs != 0;
self.video_caps != 0 || ac_present || vcodecs_present || pref_present;
match (&self.name, &self.launch) {
(None, None) if !need_placeholders => {}
(name, _) => {
@@ -734,17 +754,21 @@ impl Hello {
}
// video_caps: single trailing byte. Emitted when non-zero OR when a later field follows (so
// that field lands at a deterministic offset right after it).
if self.video_caps != 0 || self.audio_channels != 2 || self.video_codecs != 0 {
if self.video_caps != 0 || ac_present || vcodecs_present || pref_present {
b.push(self.video_caps);
}
// audio_channels: single trailing byte. Emitted when non-stereo OR when video_codecs follows.
if self.audio_channels != 2 || self.video_codecs != 0 {
// audio_channels: emitted when non-stereo OR a later field follows.
if ac_present || vcodecs_present || pref_present {
b.push(self.audio_channels);
}
// video_codecs: single trailing byte. Last field; omitted when `0` (older client → HEVC-only).
if self.video_codecs != 0 {
// video_codecs: emitted when non-zero OR preferred_codec follows.
if vcodecs_present || pref_present {
b.push(self.video_codecs);
}
// preferred_codec: single trailing byte. Last field; omitted when `0` (no preference).
if pref_present {
b.push(self.preferred_codec);
}
b
}
@@ -825,6 +849,15 @@ impl Hello {
let video_caps_off = launch_off + 1 + launch_len;
b.get(video_caps_off + 2).copied().unwrap_or(0)
},
// Optional trailing preferred-codec byte, one past video_codecs. Absent on an older
// client → `0` (no preference; the host decides by precedence).
preferred_codec: {
let name_len = b.get(26).copied().unwrap_or(0) as usize;
let launch_off = 27 + name_len;
let launch_len = b.get(launch_off).copied().unwrap_or(0) as usize;
let video_caps_off = launch_off + 1 + launch_len;
b.get(video_caps_off + 3).copied().unwrap_or(0)
},
})
}
}
@@ -2025,21 +2058,41 @@ mod tests {
#[test]
fn codec_negotiation_and_back_compat() {
// resolve_codec precedence (HEVC > AV1 > H.264) and the no-shared-codec refusal.
// resolve_codec precedence (HEVC > AV1 > H.264), no preference (0).
assert_eq!(
resolve_codec(CODEC_H264 | CODEC_HEVC, CODEC_HEVC | CODEC_AV1),
resolve_codec(CODEC_H264 | CODEC_HEVC, CODEC_HEVC | CODEC_AV1, 0),
Some(CODEC_HEVC)
);
assert_eq!(
resolve_codec(CODEC_H264 | CODEC_AV1, CODEC_AV1 | CODEC_H264),
resolve_codec(CODEC_H264 | CODEC_AV1, CODEC_AV1 | CODEC_H264, 0),
Some(CODEC_AV1)
);
assert_eq!(resolve_codec(CODEC_H264, CODEC_H264), Some(CODEC_H264));
assert_eq!(resolve_codec(CODEC_H264, CODEC_H264, 0), Some(CODEC_H264));
// A software host (H.264 only) + an HEVC-only client share nothing → refuse.
assert_eq!(resolve_codec(CODEC_HEVC, CODEC_H264), None);
assert_eq!(resolve_codec(CODEC_HEVC, CODEC_H264, 0), None);
// An older client (0 = no codec byte) is treated as HEVC-only.
assert_eq!(resolve_codec(0, CODEC_HEVC | CODEC_H264), Some(CODEC_HEVC));
assert_eq!(resolve_codec(0, CODEC_H264), None);
assert_eq!(
resolve_codec(0, CODEC_HEVC | CODEC_H264, 0),
Some(CODEC_HEVC)
);
assert_eq!(resolve_codec(0, CODEC_H264, 0), None);
// Soft preference: honored when the host can also emit it, overriding precedence...
assert_eq!(
resolve_codec(CODEC_H264 | CODEC_HEVC, CODEC_H264 | CODEC_HEVC, CODEC_H264),
Some(CODEC_H264)
);
assert_eq!(
resolve_codec(CODEC_HEVC | CODEC_AV1, CODEC_HEVC | CODEC_AV1, CODEC_AV1),
Some(CODEC_AV1)
);
// ...but falls back to precedence when the preferred codec isn't in the shared set.
assert_eq!(
resolve_codec(CODEC_HEVC | CODEC_H264, CODEC_HEVC | CODEC_H264, CODEC_AV1),
Some(CODEC_HEVC)
);
// A preference the host can't emit still can't rescue a no-shared-codec case.
assert_eq!(resolve_codec(CODEC_HEVC, CODEC_H264, CODEC_HEVC), None);
// A Hello advertising codecs roundtrips, and the wire form of a codec-only Hello decodes on
// a build that ignores the trailing byte (back-compat: extra bytes are skipped).
@@ -2058,15 +2111,23 @@ mod tests {
video_caps: 0,
audio_channels: 2, // stereo — forces the video_caps/audio_channels placeholders
video_codecs: CODEC_H264 | CODEC_HEVC,
preferred_codec: CODEC_H264,
};
let enc = h.encode();
let dec = Hello::decode(&enc).unwrap();
assert_eq!(dec.video_codecs, CODEC_H264 | CODEC_HEVC);
assert_eq!(dec.preferred_codec, CODEC_H264);
// Drop the preferred_codec byte → still decodes, video_codecs intact, preference gone.
let no_pref = &enc[..enc.len() - 1];
assert_eq!(
Hello::decode(&enc).unwrap().video_codecs,
Hello::decode(no_pref).unwrap().video_codecs,
CODEC_H264 | CODEC_HEVC
);
// A pre-codec Hello (no trailing codec byte) decodes to 0 → HEVC-only via resolve_codec.
let legacy = &enc[..enc.len() - 1]; // drop the codec byte (it was the last field)
assert_eq!(Hello::decode(no_pref).unwrap().preferred_codec, 0);
// A pre-codec Hello (no video_codecs/preferred bytes) decodes to 0 → HEVC-only.
let legacy = &enc[..enc.len() - 2];
assert_eq!(Hello::decode(legacy).unwrap().video_codecs, 0);
assert_eq!(Hello::decode(legacy).unwrap().preferred_codec, 0);
// A pre-codec Welcome (no codec byte) decodes to HEVC.
let mut w = Welcome::decode(
@@ -2145,6 +2206,7 @@ mod tests {
video_caps: VIDEO_CAP_10BIT,
audio_channels: 2,
video_codecs: CODEC_H264 | CODEC_HEVC, // exercise the codec bitfield roundtrip
preferred_codec: CODEC_HEVC,
};
assert_eq!(Hello::decode(&h.encode()).unwrap(), h);
let s = Start {
@@ -2226,6 +2288,7 @@ mod tests {
video_caps: 0,
audio_channels: 2,
video_codecs: 0,
preferred_codec: 0,
};
let enc = h.encode();
assert_eq!(enc.len(), 26);
@@ -2332,6 +2395,7 @@ mod tests {
video_caps: 0,
audio_channels: 2,
video_codecs: 0,
preferred_codec: 0,
};
let enc = base.encode();
assert_eq!(
@@ -2381,6 +2445,7 @@ mod tests {
video_caps: 0,
audio_channels: 2,
video_codecs: 0,
preferred_codec: 0,
};
// launch alone (no name): a zero-length name placeholder keeps the offset deterministic.
let with_launch = Hello {
@@ -2589,6 +2654,7 @@ mod tests {
video_caps: 0,
audio_channels: 2,
video_codecs: 0,
preferred_codec: 0,
}
.encode();
assert!(PairRequest::decode(&h).is_err(), "abi {abi} parsed as pair");