feat(clients): codec preference on Windows/Apple/Android clients (Phase 2b)
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s

Rounds out codec negotiation across the last three clients — each advertises what it can decode,
builds its decoder from the resolved Welcome.codec, and exposes a "Video codec" preference picker.

**Windows** (Rust, mirrors Linux): `decodable_codecs()` + `ffmpeg_codec_id()`; the D3D11VA and
software FFmpeg decoders (and the mid-session D3D11VA→software demotion) open the negotiated codec
instead of hardcoding HEVC; settings gain a `codec` field + reactor ComboBox; `--codec` CLI flag.

**Apple** (Swift/C-ABI): AnnexB is now codec-aware — a `VideoCodec` enum drives H.264 vs HEVC NAL
parsing / parameter-set extraction (`CMVideoFormatDescriptionCreateFromH264ParameterSets` for H.264,
no VPS) and AVCC repacking; `PunktfunkConnection` advertises H264|HEVC via `punktfunk_connect_ex7`,
reads `resolvedCodec` (`punktfunk_connection_codec`), and threads `videoCodec` into the stage-1/2
pipelines + `VideoDecoder`; SettingsView "Video codec" Picker (auto/HEVC/H.264). AV1 is left out
(hosts don't emit it on the native path, and it's not an AnnexB codec). Test call sites updated.

**Android** (Kotlin + Rust JNI): the JNI `nativeConnect` gains `preferredCodec`; the native decode
loop picks the AMediaCodec MIME (`video/hevc`|`video/avc`) from `connector.codec` and advertises
H264|HEVC; Settings `codec` field + Compose dropdown.

Core/host/probe/Linux clippy + tests green (unchanged from 2a). Windows/Apple/Android compile on
their platform CI (this Linux box can't build them — Windows toolchain / Xcode / the Android NDK's
opus-cmake toolchain). All follow the Linux client's validated pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-02 00:29:38 +00:00
parent 12843fe253
commit 3678c182d5
24 changed files with 328 additions and 74 deletions
+42 -12
View File
@@ -124,17 +124,46 @@ enum Backend {
pub struct Decoder {
backend: Backend,
/// The negotiated codec, so a mid-session D3D11VA→software demotion rebuilds for the same codec.
codec_id: ffmpeg::codec::Id,
}
/// Map a negotiated `quic` codec bit to the FFmpeg decoder id the client opens.
pub fn ffmpeg_codec_id(wire: u8) -> ffmpeg::codec::Id {
match wire {
punktfunk_core::quic::CODEC_H264 => ffmpeg::codec::Id::H264,
punktfunk_core::quic::CODEC_AV1 => ffmpeg::codec::Id::AV1,
_ => ffmpeg::codec::Id::HEVC,
}
}
/// The `quic` codec bitfield this client can decode — whatever FFmpeg has a decoder for (HEVC/H.264
/// always; AV1 when built in). Advertised to the host so it never emits a codec we can't decode.
pub fn decodable_codecs() -> u8 {
let _ = ffmpeg::init();
let mut bits = 0u8;
for (id, bit) in [
(ffmpeg::codec::Id::HEVC, punktfunk_core::quic::CODEC_HEVC),
(ffmpeg::codec::Id::H264, punktfunk_core::quic::CODEC_H264),
(ffmpeg::codec::Id::AV1, punktfunk_core::quic::CODEC_AV1),
] {
if ffmpeg::decoder::find(id).is_some() {
bits |= bit;
}
}
bits
}
impl Decoder {
pub fn new(pref: DecoderPref) -> Result<Decoder> {
pub fn new(pref: DecoderPref, codec_id: ffmpeg::codec::Id) -> Result<Decoder> {
ffmpeg::init().context("ffmpeg init")?;
if pref != DecoderPref::Software {
match D3d11vaDecoder::new() {
match D3d11vaDecoder::new(codec_id) {
Ok(d) => {
tracing::info!("D3D11VA hardware decode active (zero-copy)");
tracing::info!(?codec_id, "D3D11VA hardware decode active (zero-copy)");
return Ok(Decoder {
backend: Backend::D3d11va(d),
codec_id,
});
}
Err(e) => {
@@ -146,7 +175,8 @@ impl Decoder {
}
}
Ok(Decoder {
backend: Backend::Software(SoftwareDecoder::new()?),
backend: Backend::Software(SoftwareDecoder::new(codec_id)?),
codec_id,
})
}
@@ -164,7 +194,7 @@ impl Decoder {
Ok(f) => Ok(f.map(DecodedFrame::Gpu)),
Err(e) => {
tracing::warn!(error = %e, "D3D11VA decode failed — falling back to software");
self.backend = Backend::Software(SoftwareDecoder::new()?);
self.backend = Backend::Software(SoftwareDecoder::new(self.codec_id)?);
Ok(None)
}
},
@@ -183,9 +213,9 @@ struct SoftwareDecoder {
}
impl SoftwareDecoder {
fn new() -> Result<SoftwareDecoder> {
let codec =
ffmpeg::decoder::find(ffmpeg::codec::Id::HEVC).ok_or(anyhow!("no HEVC decoder"))?;
fn new(codec_id: ffmpeg::codec::Id) -> Result<SoftwareDecoder> {
let codec = ffmpeg::decoder::find(codec_id)
.ok_or_else(|| anyhow!("no {codec_id:?} decoder in libavcodec"))?;
let mut ctx = ffmpeg::codec::Context::new_with_codec(codec);
unsafe {
let raw = ctx.as_mut_ptr();
@@ -194,7 +224,7 @@ impl SoftwareDecoder {
(*raw).thread_type = ffmpeg::ffi::FF_THREAD_SLICE;
(*raw).thread_count = 0; // auto
}
let decoder = ctx.decoder().video().context("open HEVC decoder")?;
let decoder = ctx.decoder().video().context("open video decoder")?;
Ok(SoftwareDecoder { decoder, sws: None })
}
@@ -359,7 +389,7 @@ struct D3d11vaDecoder {
unsafe impl Send for D3d11vaDecoder {}
impl D3d11vaDecoder {
fn new() -> Result<D3d11vaDecoder> {
fn new(codec_id: ffmpeg::codec::Id) -> Result<D3d11vaDecoder> {
use ffmpeg::ffi;
let shared = crate::gpu::shared().ok_or_else(|| anyhow!("no shared D3D11 device"))?;
if !shared.hardware {
@@ -387,11 +417,11 @@ impl D3d11vaDecoder {
bail!("av_hwdevice_ctx_init: {}", ffmpeg::Error::from(r));
}
let codec = ffi::avcodec_find_decoder(ffi::AVCodecID::AV_CODEC_ID_HEVC);
let codec = ffi::avcodec_find_decoder(codec_id.into());
if codec.is_null() {
let mut hw = hw_device;
ffi::av_buffer_unref(&mut hw);
bail!("no HEVC decoder");
bail!("no {codec_id:?} decoder");
}
let ctx = ffi::avcodec_alloc_context3(codec);
(*ctx).hw_device_ctx = ffi::av_buffer_ref(hw_device);