feat(codec): make client codec selection real — GPU-aware native advertisement
apple / swift (push) Successful in 1m7s
release / apple (push) Successful in 8m27s
windows-host / package (push) Successful in 7m59s
apple / screenshots (push) Failing after 2m38s
android / android (push) Successful in 4m44s
ci / rust (push) Failing after 46s
arch / build-publish (push) Successful in 5m32s
ci / web (push) Successful in 50s
ci / docs-site (push) Successful in 1m2s
deb / build-publish (push) Successful in 4m44s
ci / bench (push) Successful in 5m7s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
decky / build-publish (push) Successful in 13s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 11m9s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 11m14s
docker / deploy-docs (push) Successful in 13s

The clients' codec pickers sent a preference the host threw away:
host_wire_caps() hardcoded HEVC for every GPU backend, so the native
path never emitted H.264/AV1 regardless of what the user chose.

Host: advertise what the backend actually encodes, mirroring the
GameStream serverinfo logic — software openh264 emits H.264; probed
backends (Linux VAAPI, Windows AMF/QSV) advertise their per-GPU
CodecSupport via the new wire_mask(); NVENC keeps the
Moonlight-validated H.264|HEVC|AV1 static superset; an empty probe
falls back to the superset so auto clients still resolve HEVC. Gate
10-bit to HEVC (like the 4:4:4 gate) now that a client can steer the
codec — Main10 is the only 10-bit encode path. Fix the web-console
stats label hardcoded to "hevc" (new Codec::label(), shared with the
GameStream register_session mapping).

Android: replace the hardcoded H264|HEVC Hello advertisement with a
videoCodecs param fed by VideoDecoders.decodableCodecBits() (AV1 bit
only when a real hardware video/av01 decoder exists — the decode loop
was already mime-driven); offer AV1 in the codec picker on capable
devices.

Decky: add the missing "Video codec" dropdown (auto/hevc/h264/av1) to
the plugin settings, round-tripping the same codec key the flatpak
client reads.

Apple: unchanged by design (AnnexB.swift is NAL-only, AV1 is never
advertised); refresh the stale "hosts don't emit AV1 on the native
path yet" comments here and host-side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-08 16:45:30 +02:00
parent 0290bf7285
commit 838a1239cf
15 changed files with 225 additions and 41 deletions
+119 -6
View File
@@ -77,14 +77,68 @@ impl Codec {
}
/// The `quic` codec bitfield the host can currently **emit** on the punktfunk/1 native path,
/// given the resolved encode backend. The GPU-less software encoder (openh264) produces H.264
/// only; every GPU backend emits HEVC today (per-GPU H.264/AV1 negotiation on the native path is
/// future work — GameStream already negotiates codecs with Moonlight separately). Fed to
/// given the resolved encode backend — the same GPU-aware advertisement GameStream builds for
/// Moonlight ([`crate::gamestream::serverinfo`]), in `quic::CODEC_*` bits. The GPU-less software
/// encoder (openh264) produces H.264 only; the probed backends (Linux VAAPI, Windows AMF/QSV)
/// advertise exactly what the GPU encodes ([`vaapi_codec_support`] / [`windows_codec_support`] —
/// AV1 encode is narrow, an old iGPU might lack HEVC); NVENC keeps the Moonlight-validated
/// static superset. An empty probe means the GPU wasn't usable at probe time (GPU-less CI,
/// wrong-vendor pref), not that it encodes nothing — fall back to the superset so `resolve_codec`
/// still lands on HEVC for an auto client, exactly the pre-probe behaviour. Fed to
/// [`punktfunk_core::quic::resolve_codec`] against the client's advertised codecs.
pub fn host_wire_caps() -> u8 {
match crate::config::config().encoder_pref.as_str() {
"software" | "sw" | "openh264" => punktfunk_core::quic::CODEC_H264,
_ => punktfunk_core::quic::CODEC_HEVC,
/// The static GPU superset (H.264 | HEVC | AV1) — mirrors the GameStream
/// `SERVER_CODEC_MODE_SUPPORT` advertisement for the unprobed backends.
const GPU_SUPERSET: u8 = punktfunk_core::quic::CODEC_H264
| punktfunk_core::quic::CODEC_HEVC
| punktfunk_core::quic::CODEC_AV1;
#[cfg(target_os = "linux")]
{
if matches!(
crate::config::config().encoder_pref.as_str(),
"software" | "sw" | "openh264"
) {
return punktfunk_core::quic::CODEC_H264;
}
if linux_zero_copy_is_vaapi() {
if let Some(m) = vaapi_codec_support().wire_mask() {
return m;
}
}
// NVENC (static superset, like GameStream) — or an empty VAAPI probe (see above).
GPU_SUPERSET
}
#[cfg(target_os = "windows")]
{
if windows_resolved_backend() == WindowsBackend::Software {
return punktfunk_core::quic::CODEC_H264;
}
if windows_backend_is_probed() {
if let Some(m) = windows_codec_support().wire_mask() {
return m;
}
}
// NVENC (static superset, like GameStream) — or an empty AMF/QSV probe (see above).
GPU_SUPERSET
}
// The macOS dev/test host has no GPU encode backend — keep the pre-probe advertisement.
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
{
let _ = GPU_SUPERSET;
match crate::config::config().encoder_pref.as_str() {
"software" | "sw" | "openh264" => punktfunk_core::quic::CODEC_H264,
_ => punktfunk_core::quic::CODEC_HEVC,
}
}
}
/// Lowercase stats/console label (`"h264"` / `"hevc"` / `"av1"`) — the codec string seeded into
/// the web console's session meta ([`crate::stats_recorder::StatsRecorder::register_session`]).
pub fn label(self) -> &'static str {
match self {
Codec::H264 => "h264",
Codec::H265 => "hevc",
Codec::Av1 => "av1",
}
}
@@ -741,6 +795,27 @@ pub struct CodecSupport {
pub av1: bool,
}
#[cfg(any(target_os = "linux", target_os = "windows"))]
impl CodecSupport {
/// The probed codecs as a `quic::CODEC_*` bitfield, or `None` when the probe found nothing —
/// meaning the GPU wasn't usable at probe time (GPU-less CI, a wrong-vendor pref), NOT that it
/// encodes zero codecs; the caller then falls back to the static superset (the native-path
/// analogue of `gamestream::serverinfo::probed_mask`).
pub fn wire_mask(self) -> Option<u8> {
let mut m = 0u8;
if self.h264 {
m |= punktfunk_core::quic::CODEC_H264;
}
if self.h265 {
m |= punktfunk_core::quic::CODEC_HEVC;
}
if self.av1 {
m |= punktfunk_core::quic::CODEC_AV1;
}
(m != 0).then_some(m)
}
}
/// Probe the active Linux GPU backend for its encodable codecs (cached; opens a tiny encoder per
/// codec, once). Only the VAAPI (AMD/Intel) backend is probed — NVENC keeps its Moonlight-validated
/// static advertisement (callers gate on [`linux_zero_copy_is_vaapi`]).
@@ -1042,4 +1117,42 @@ mod tests {
}
}
}
/// The probed-capability → wire-bitfield mapping the native codec advertisement is built from.
#[cfg(any(target_os = "linux", target_os = "windows"))]
#[test]
fn codec_support_wire_mask() {
use punktfunk_core::quic::{CODEC_AV1, CODEC_H264, CODEC_HEVC};
let all = CodecSupport {
h264: true,
h265: true,
av1: true,
};
assert_eq!(all.wire_mask(), Some(CODEC_H264 | CODEC_HEVC | CODEC_AV1));
let hevc_only = CodecSupport {
h264: false,
h265: true,
av1: false,
};
assert_eq!(hevc_only.wire_mask(), Some(CODEC_HEVC));
// An all-false probe means "GPU unusable at probe time", not "zero codecs" — `None` tells
// the caller to advertise the static superset instead of refusing every handshake.
let none = CodecSupport {
h264: false,
h265: false,
av1: false,
};
assert_eq!(none.wire_mask(), None);
}
/// Wire round-trip and the stats label stay in lockstep with the `quic::CODEC_*` bits.
#[test]
fn codec_wire_roundtrip_and_label() {
for c in [Codec::H264, Codec::H265, Codec::Av1] {
assert_eq!(Codec::from_wire(c.to_wire()), c);
}
assert_eq!(Codec::H264.label(), "h264");
assert_eq!(Codec::H265.label(), "hevc");
assert_eq!(Codec::Av1.label(), "av1");
}
}
@@ -663,11 +663,7 @@ fn stream_body(
// Web-console stats accumulation (active when `perf` OR a capture is armed): per-stage vectors
// for p50/p99, the goodput bytes queued to the sender this window, the previous window's
// dropped-frame count for delta computation, and the registration id cached on the first sample.
let codec_name = match cfg.codec {
Codec::H264 => "h264",
Codec::H265 => "hevc",
Codec::Av1 => "av1",
};
let codec_name = cfg.codec.label();
let mut sid: Option<u32> = None;
let (mut v_cap, mut v_enc, mut v_pkt, mut v_send): (Vec<u32>, Vec<u32>, Vec<u32>, Vec<u32>) =
(Vec::new(), Vec::new(), Vec::new(), Vec::new());
+20 -11
View File
@@ -732,9 +732,10 @@ async fn serve_session(
// The pairing gate (require_pairing → paired? else park for delegated approval) ran above,
// before this future, so a client reaching here is paired (or the host is `--open`).
// Codec negotiation: pick the one codec this host will emit (its backend capability ∩ the
// client's advertised codecs). A GPU-less software host emits H.264, so an HEVC-only client
// shares nothing with it → refuse honestly rather than send a stream it can't decode.
// Codec negotiation: pick the one codec this host will emit (its GPU-probed backend
// capability ∩ the client's advertised codecs, honoring the client's soft preference).
// A GPU-less software host emits H.264 only, so an HEVC-only client shares nothing with
// it → refuse honestly rather than send a stream it can't decode.
let host_codecs = crate::encode::Codec::host_wire_caps();
let codec_bit =
punktfunk_core::quic::resolve_codec(hello.video_codecs, host_codecs, hello.preferred_codec)
@@ -888,10 +889,16 @@ async fn serve_session(
// Resolve the encode bit depth: HEVC Main10 only when the client advertised it AND the host
// opted in (PUNKTFUNK_10BIT). A client that can't decode 10-bit (caps bit clear, or an older
// client) always gets the 8-bit stream. PUNKTFUNK_10BIT is the host policy gate until a
// mgmt/console toggle replaces it.
// mgmt/console toggle replaces it. 10-bit is HEVC-only (like the 4:4:4 gate below): now that
// the client can steer the codec to H.264/AV1, a non-HEVC session must stay 8-bit — the
// encoders' 10-bit path is Main10, and AV1 10-bit stays off until live-confirmed (the same
// stance as the GameStream Main10 advertisement).
let host_wants_10bit = crate::config::config().ten_bit;
let client_supports_10bit = hello.video_caps & punktfunk_core::quic::VIDEO_CAP_10BIT != 0;
let bit_depth: u8 = if host_wants_10bit && client_supports_10bit {
let bit_depth: u8 = if host_wants_10bit
&& client_supports_10bit
&& codec == crate::encode::Codec::H265
{
10
} else {
8
@@ -1010,8 +1017,9 @@ async fn serve_session(
// The resolved audio channel count the audio thread will capture + Opus-(multi)stream
// encode (2/6/8). The client builds its decoder from this echoed value.
audio_channels,
// The negotiated codec the encoder will emit (H.264 for a software host, else HEVC). The
// client builds its decoder from this instead of assuming HEVC.
// The negotiated codec the encoder will emit (client preference ∩ GPU capability;
// HEVC-precedence tie-break). The client builds its decoder from this instead of
// assuming HEVC.
codec: codec_bit,
};
io::write_msg(&mut send, &welcome.encode()).await?;
@@ -1027,7 +1035,7 @@ async fn serve_session(
.await
.map_err(|_| anyhow!("handshake timed out after {HANDSHAKE_TIMEOUT:?}"))??;
let (mut ctrl_send, mut ctrl_recv) = (send, recv);
// Negotiated codec (HEVC / H.264), derived from the Welcome. `Copy`, so the control task's
// Negotiated codec (HEVC / H.264 / AV1), derived from the Welcome. `Copy`, so the control task's
// `async move` captures a copy and it stays usable for the data-plane SessionContext below.
let codec = crate::encode::Codec::from_wire(welcome.codec);
let client_udp = std::net::SocketAddr::new(peer.ip(), start.client_udp_port);
@@ -3080,8 +3088,9 @@ struct SessionContext {
bit_depth: u8,
/// Negotiated chroma subsampling (4:2:0, or 4:4:4 when the client + host + GPU all support it).
chroma: crate::encode::ChromaFormat,
/// Negotiated video codec the encoder emits (HEVC by default; H.264 for a software host). Also
/// used to rebuild the encoder at the same codec across a mid-stream mode reconfigure.
/// Negotiated video codec the encoder emits (HEVC by default; H.264 / AV1 when the client
/// prefers one the GPU encodes; H.264 for a software host). Also used to rebuild the encoder
/// at the same codec across a mid-stream mode reconfigure.
codec: crate::encode::Codec,
/// Speed-test burst requests (see [`service_probes`]).
probe_rx: std::sync::mpsc::Receiver<ProbeRequest>,
@@ -3236,7 +3245,7 @@ fn virtual_stream(ctx: SessionContext) -> Result<()> {
width: mode.width,
height: mode.height,
fps: mode.refresh_hz,
codec: "hevc",
codec: plan.codec.label(),
client: client_label,
bitrate_kbps,
};