feat(clients): codec preference on Windows/Apple/Android clients (Phase 2b)
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s
Rounds out codec negotiation across the last three clients — each advertises what it can decode, builds its decoder from the resolved Welcome.codec, and exposes a "Video codec" preference picker. **Windows** (Rust, mirrors Linux): `decodable_codecs()` + `ffmpeg_codec_id()`; the D3D11VA and software FFmpeg decoders (and the mid-session D3D11VA→software demotion) open the negotiated codec instead of hardcoding HEVC; settings gain a `codec` field + reactor ComboBox; `--codec` CLI flag. **Apple** (Swift/C-ABI): AnnexB is now codec-aware — a `VideoCodec` enum drives H.264 vs HEVC NAL parsing / parameter-set extraction (`CMVideoFormatDescriptionCreateFromH264ParameterSets` for H.264, no VPS) and AVCC repacking; `PunktfunkConnection` advertises H264|HEVC via `punktfunk_connect_ex7`, reads `resolvedCodec` (`punktfunk_connection_codec`), and threads `videoCodec` into the stage-1/2 pipelines + `VideoDecoder`; SettingsView "Video codec" Picker (auto/HEVC/H.264). AV1 is left out (hosts don't emit it on the native path, and it's not an AnnexB codec). Test call sites updated. **Android** (Kotlin + Rust JNI): the JNI `nativeConnect` gains `preferredCodec`; the native decode loop picks the AMediaCodec MIME (`video/hevc`|`video/avc`) from `connector.codec` and advertises H264|HEVC; Settings `codec` field + Compose dropdown. Core/host/probe/Linux clippy + tests green (unchanged from 2a). Windows/Apple/Android compile on their platform CI (this Linux box can't build them — Windows toolchain / Xcode / the Android NDK's opus-cmake toolchain). All follow the Linux client's validated pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -124,17 +124,46 @@ enum Backend {
|
||||
|
||||
pub struct Decoder {
|
||||
backend: Backend,
|
||||
/// The negotiated codec, so a mid-session D3D11VA→software demotion rebuilds for the same codec.
|
||||
codec_id: ffmpeg::codec::Id,
|
||||
}
|
||||
|
||||
/// Map a negotiated `quic` codec bit to the FFmpeg decoder id the client opens.
|
||||
pub fn ffmpeg_codec_id(wire: u8) -> ffmpeg::codec::Id {
|
||||
match wire {
|
||||
punktfunk_core::quic::CODEC_H264 => ffmpeg::codec::Id::H264,
|
||||
punktfunk_core::quic::CODEC_AV1 => ffmpeg::codec::Id::AV1,
|
||||
_ => ffmpeg::codec::Id::HEVC,
|
||||
}
|
||||
}
|
||||
|
||||
/// The `quic` codec bitfield this client can decode — whatever FFmpeg has a decoder for (HEVC/H.264
|
||||
/// always; AV1 when built in). Advertised to the host so it never emits a codec we can't decode.
|
||||
pub fn decodable_codecs() -> u8 {
|
||||
let _ = ffmpeg::init();
|
||||
let mut bits = 0u8;
|
||||
for (id, bit) in [
|
||||
(ffmpeg::codec::Id::HEVC, punktfunk_core::quic::CODEC_HEVC),
|
||||
(ffmpeg::codec::Id::H264, punktfunk_core::quic::CODEC_H264),
|
||||
(ffmpeg::codec::Id::AV1, punktfunk_core::quic::CODEC_AV1),
|
||||
] {
|
||||
if ffmpeg::decoder::find(id).is_some() {
|
||||
bits |= bit;
|
||||
}
|
||||
}
|
||||
bits
|
||||
}
|
||||
|
||||
impl Decoder {
|
||||
pub fn new(pref: DecoderPref) -> Result<Decoder> {
|
||||
pub fn new(pref: DecoderPref, codec_id: ffmpeg::codec::Id) -> Result<Decoder> {
|
||||
ffmpeg::init().context("ffmpeg init")?;
|
||||
if pref != DecoderPref::Software {
|
||||
match D3d11vaDecoder::new() {
|
||||
match D3d11vaDecoder::new(codec_id) {
|
||||
Ok(d) => {
|
||||
tracing::info!("D3D11VA hardware decode active (zero-copy)");
|
||||
tracing::info!(?codec_id, "D3D11VA hardware decode active (zero-copy)");
|
||||
return Ok(Decoder {
|
||||
backend: Backend::D3d11va(d),
|
||||
codec_id,
|
||||
});
|
||||
}
|
||||
Err(e) => {
|
||||
@@ -146,7 +175,8 @@ impl Decoder {
|
||||
}
|
||||
}
|
||||
Ok(Decoder {
|
||||
backend: Backend::Software(SoftwareDecoder::new()?),
|
||||
backend: Backend::Software(SoftwareDecoder::new(codec_id)?),
|
||||
codec_id,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -164,7 +194,7 @@ impl Decoder {
|
||||
Ok(f) => Ok(f.map(DecodedFrame::Gpu)),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "D3D11VA decode failed — falling back to software");
|
||||
self.backend = Backend::Software(SoftwareDecoder::new()?);
|
||||
self.backend = Backend::Software(SoftwareDecoder::new(self.codec_id)?);
|
||||
Ok(None)
|
||||
}
|
||||
},
|
||||
@@ -183,9 +213,9 @@ struct SoftwareDecoder {
|
||||
}
|
||||
|
||||
impl SoftwareDecoder {
|
||||
fn new() -> Result<SoftwareDecoder> {
|
||||
let codec =
|
||||
ffmpeg::decoder::find(ffmpeg::codec::Id::HEVC).ok_or(anyhow!("no HEVC decoder"))?;
|
||||
fn new(codec_id: ffmpeg::codec::Id) -> Result<SoftwareDecoder> {
|
||||
let codec = ffmpeg::decoder::find(codec_id)
|
||||
.ok_or_else(|| anyhow!("no {codec_id:?} decoder in libavcodec"))?;
|
||||
let mut ctx = ffmpeg::codec::Context::new_with_codec(codec);
|
||||
unsafe {
|
||||
let raw = ctx.as_mut_ptr();
|
||||
@@ -194,7 +224,7 @@ impl SoftwareDecoder {
|
||||
(*raw).thread_type = ffmpeg::ffi::FF_THREAD_SLICE;
|
||||
(*raw).thread_count = 0; // auto
|
||||
}
|
||||
let decoder = ctx.decoder().video().context("open HEVC decoder")?;
|
||||
let decoder = ctx.decoder().video().context("open video decoder")?;
|
||||
Ok(SoftwareDecoder { decoder, sws: None })
|
||||
}
|
||||
|
||||
@@ -359,7 +389,7 @@ struct D3d11vaDecoder {
|
||||
unsafe impl Send for D3d11vaDecoder {}
|
||||
|
||||
impl D3d11vaDecoder {
|
||||
fn new() -> Result<D3d11vaDecoder> {
|
||||
fn new(codec_id: ffmpeg::codec::Id) -> Result<D3d11vaDecoder> {
|
||||
use ffmpeg::ffi;
|
||||
let shared = crate::gpu::shared().ok_or_else(|| anyhow!("no shared D3D11 device"))?;
|
||||
if !shared.hardware {
|
||||
@@ -387,11 +417,11 @@ impl D3d11vaDecoder {
|
||||
bail!("av_hwdevice_ctx_init: {}", ffmpeg::Error::from(r));
|
||||
}
|
||||
|
||||
let codec = ffi::avcodec_find_decoder(ffi::AVCodecID::AV_CODEC_ID_HEVC);
|
||||
let codec = ffi::avcodec_find_decoder(codec_id.into());
|
||||
if codec.is_null() {
|
||||
let mut hw = hw_device;
|
||||
ffi::av_buffer_unref(&mut hw);
|
||||
bail!("no HEVC decoder");
|
||||
bail!("no {codec_id:?} decoder");
|
||||
}
|
||||
let ctx = ffi::avcodec_alloc_context3(codec);
|
||||
(*ctx).hw_device_ctx = ffi::av_buffer_ref(hw_device);
|
||||
|
||||
Reference in New Issue
Block a user