feat(clients): codec preference on Windows/Apple/Android clients (Phase 2b)
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s

Rounds out codec negotiation across the last three clients — each advertises what it can decode,
builds its decoder from the resolved Welcome.codec, and exposes a "Video codec" preference picker.

**Windows** (Rust, mirrors Linux): `decodable_codecs()` + `ffmpeg_codec_id()`; the D3D11VA and
software FFmpeg decoders (and the mid-session D3D11VA→software demotion) open the negotiated codec
instead of hardcoding HEVC; settings gain a `codec` field + reactor ComboBox; `--codec` CLI flag.

**Apple** (Swift/C-ABI): AnnexB is now codec-aware — a `VideoCodec` enum drives H.264 vs HEVC NAL
parsing / parameter-set extraction (`CMVideoFormatDescriptionCreateFromH264ParameterSets` for H.264,
no VPS) and AVCC repacking; `PunktfunkConnection` advertises H264|HEVC via `punktfunk_connect_ex7`,
reads `resolvedCodec` (`punktfunk_connection_codec`), and threads `videoCodec` into the stage-1/2
pipelines + `VideoDecoder`; SettingsView "Video codec" Picker (auto/HEVC/H.264). AV1 is left out
(hosts don't emit it on the native path, and it's not an AnnexB codec). Test call sites updated.

**Android** (Kotlin + Rust JNI): the JNI `nativeConnect` gains `preferredCodec`; the native decode
loop picks the AMediaCodec MIME (`video/hevc`|`video/avc`) from `connector.codec` and advertises
H264|HEVC; Settings `codec` field + Compose dropdown.

Core/host/probe/Linux clippy + tests green (unchanged from 2a). Windows/Apple/Android compile on
their platform CI (this Linux box can't build them — Windows toolchain / Xcode / the Android NDK's
opus-cmake toolchain). All follow the Linux client's validated pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-02 00:29:38 +00:00
parent 12843fe253
commit 3678c182d5
24 changed files with 328 additions and 74 deletions
+89 -33
View File
@@ -10,6 +10,20 @@
import CoreMedia
import Foundation
/// The video codec of the host's elementary stream negotiated in the Welcome and read via
/// `punktfunk_connection_codec`. Both are Annex-B with in-band parameter sets on every IDR; they
/// differ only in NAL-header layout and which parameter sets exist (HEVC adds a VPS). AV1 is not an
/// Annex-B/NAL codec and isn't handled here (hosts don't emit it on the native path yet).
public enum VideoCodec: Equatable {
case h264
case hevc
/// Resolve from the wire `Welcome.codec` byte (`PUNKTFUNK_CODEC_*`; unknown HEVC).
public init(wire: UInt8) {
self = wire == 0x01 ? .h264 : .hevc // 0x01 = PUNKTFUNK_CODEC_H264
}
}
public enum AnnexB {
/// Split an Annex-B stream into NAL units (start codes 00 00 01 / 00 00 00 01 stripped).
/// All zeros immediately preceding a start code are dropped: they're either the
@@ -47,40 +61,83 @@ public enum AnnexB {
return (first >> 1) & 0x3F
}
/// Build a format description from an IDR AU's in-band VPS(32)/SPS(33)/PPS(34).
/// Returns nil when the AU carries no parameter sets (non-IDR).
public static func formatDescription(fromIDR au: Data) -> CMVideoFormatDescription? {
/// H.264 NAL unit type (bits 0..4 of the first byte).
public static func h264NalType(_ nal: Data) -> UInt8 {
guard let first = nal.first else { return 0xFF }
return first & 0x1F
}
/// True if this NAL is a parameter set for `codec` (dropped from AVCC; kept for the format desc).
/// HEVC: VPS 32 / SPS 33 / PPS 34. H.264: SPS 7 / PPS 8 (no VPS).
private static func isParameterSet(_ nal: Data, _ codec: VideoCodec) -> Bool {
switch codec {
case .hevc: let t = hevcNalType(nal); return t == 32 || t == 33 || t == 34
case .h264: let t = h264NalType(nal); return t == 7 || t == 8
}
}
/// Build a format description from an IDR AU's in-band parameter sets (HEVC: VPS/SPS/PPS;
/// H.264: SPS/PPS). Returns nil when the AU carries no parameter sets (non-IDR).
public static func formatDescription(fromIDR au: Data, codec: VideoCodec)
-> CMVideoFormatDescription?
{
// Collect the parameter-set NALs in the order VideoToolbox wants them (HEVC: VPS,SPS,PPS;
// H.264: SPS,PPS).
var vps: Data?, sps: Data?, pps: Data?
for nal in nalUnits(in: au) {
switch hevcNalType(nal) {
case 32: vps = nal
case 33: sps = nal
case 34: pps = nal
default: break
switch codec {
case .hevc:
switch hevcNalType(nal) {
case 32: vps = nal
case 33: sps = nal
case 34: pps = nal
default: break
}
case .h264:
switch h264NalType(nal) {
case 7: sps = nal
case 8: pps = nal
default: break
}
}
}
guard let vps, let sps, let pps else { return nil }
guard let sps, let pps else { return nil }
let sets: [Data] = codec == .hevc ? [vps, sps, pps].compactMap { $0 } : [sps, pps]
guard codec == .h264 || sets.count == 3 else { return nil } // HEVC needs the VPS too
var format: CMVideoFormatDescription?
let sets = [vps, sps, pps]
let status: OSStatus = sets[0].withUnsafeBytes { v in
sets[1].withUnsafeBytes { s in
sets[2].withUnsafeBytes { p in
let pointers: [UnsafePointer<UInt8>] = [
v.bindMemory(to: UInt8.self).baseAddress!,
s.bindMemory(to: UInt8.self).baseAddress!,
p.bindMemory(to: UInt8.self).baseAddress!,
]
let sizes = [vps.count, sps.count, pps.count]
return CMVideoFormatDescriptionCreateFromHEVCParameterSets(
allocator: kCFAllocatorDefault,
parameterSetCount: 3,
parameterSetPointers: pointers,
parameterSetSizes: sizes,
nalUnitHeaderLength: 4,
extensions: nil,
formatDescriptionOut: &format)
}
// Pin every parameter set's bytes for the duration of the create call, then hand
// VideoToolbox parallel pointer/size arrays.
var pointers: [UnsafePointer<UInt8>] = []
var sizes: [Int] = []
func withAll(_ i: Int, _ body: () -> Void) {
if i == sets.count { body(); return }
sets[i].withUnsafeBytes { raw in
pointers.append(raw.bindMemory(to: UInt8.self).baseAddress!)
sizes.append(sets[i].count)
withAll(i + 1, body)
}
}
var status: OSStatus = -1
withAll(0) {
switch codec {
case .hevc:
status = CMVideoFormatDescriptionCreateFromHEVCParameterSets(
allocator: kCFAllocatorDefault,
parameterSetCount: pointers.count,
parameterSetPointers: pointers,
parameterSetSizes: sizes,
nalUnitHeaderLength: 4,
extensions: nil,
formatDescriptionOut: &format)
case .h264:
status = CMVideoFormatDescriptionCreateFromH264ParameterSets(
allocator: kCFAllocatorDefault,
parameterSetCount: pointers.count,
parameterSetPointers: pointers,
parameterSetSizes: sizes,
nalUnitHeaderLength: 4,
formatDescriptionOut: &format)
}
}
return status == noErr ? format : nil
@@ -88,11 +145,10 @@ public enum AnnexB {
/// Re-pack an Annex-B AU as AVCC (4-byte big-endian length before each NAL), dropping
/// the parameter-set NALs (they live in the format description).
public static func avcc(from au: Data) -> Data {
public static func avcc(from au: Data, codec: VideoCodec) -> Data {
var out = Data(capacity: au.count + 16)
for nal in nalUnits(in: au) {
let t = hevcNalType(nal)
if t == 32 || t == 33 || t == 34 { continue } // VPS/SPS/PPS
if isParameterSet(nal, codec) { continue }
var len = UInt32(nal.count).bigEndian
withUnsafeBytes(of: &len) { out.append(contentsOf: $0) }
out.append(nal)
@@ -102,9 +158,9 @@ public enum AnnexB {
/// Wrap one AU as a decode-ready CMSampleBuffer.
public static func sampleBuffer(
au: AccessUnit, format: CMVideoFormatDescription
au: AccessUnit, format: CMVideoFormatDescription, codec: VideoCodec
) -> CMSampleBuffer? {
let avccData = avcc(from: au.data)
let avccData = avcc(from: au.data, codec: codec)
var blockBuffer: CMBlockBuffer?
guard CMBlockBufferCreateWithMemoryBlock(
allocator: kCFAllocatorDefault, memoryBlock: nil,