feat(clients): codec preference on Windows/Apple/Android clients (Phase 2b)
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s
Rounds out codec negotiation across the last three clients — each advertises what it can decode, builds its decoder from the resolved Welcome.codec, and exposes a "Video codec" preference picker. **Windows** (Rust, mirrors Linux): `decodable_codecs()` + `ffmpeg_codec_id()`; the D3D11VA and software FFmpeg decoders (and the mid-session D3D11VA→software demotion) open the negotiated codec instead of hardcoding HEVC; settings gain a `codec` field + reactor ComboBox; `--codec` CLI flag. **Apple** (Swift/C-ABI): AnnexB is now codec-aware — a `VideoCodec` enum drives H.264 vs HEVC NAL parsing / parameter-set extraction (`CMVideoFormatDescriptionCreateFromH264ParameterSets` for H.264, no VPS) and AVCC repacking; `PunktfunkConnection` advertises H264|HEVC via `punktfunk_connect_ex7`, reads `resolvedCodec` (`punktfunk_connection_codec`), and threads `videoCodec` into the stage-1/2 pipelines + `VideoDecoder`; SettingsView "Video codec" Picker (auto/HEVC/H.264). AV1 is left out (hosts don't emit it on the native path, and it's not an AnnexB codec). Test call sites updated. **Android** (Kotlin + Rust JNI): the JNI `nativeConnect` gains `preferredCodec`; the native decode loop picks the AMediaCodec MIME (`video/hevc`|`video/avc`) from `connector.codec` and advertises H264|HEVC; Settings `codec` field + Compose dropdown. Core/host/probe/Linux clippy + tests green (unchanged from 2a). Windows/Apple/Android compile on their platform CI (this Linux box can't build them — Windows toolchain / Xcode / the Android NDK's opus-cmake toolchain). All follow the Linux client's validated pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -10,6 +10,20 @@
|
||||
import CoreMedia
|
||||
import Foundation
|
||||
|
||||
/// The video codec of the host's elementary stream — negotiated in the Welcome and read via
|
||||
/// `punktfunk_connection_codec`. Both are Annex-B with in-band parameter sets on every IDR; they
|
||||
/// differ only in NAL-header layout and which parameter sets exist (HEVC adds a VPS). AV1 is not an
|
||||
/// Annex-B/NAL codec and isn't handled here (hosts don't emit it on the native path yet).
|
||||
public enum VideoCodec: Equatable {
|
||||
case h264
|
||||
case hevc
|
||||
|
||||
/// Resolve from the wire `Welcome.codec` byte (`PUNKTFUNK_CODEC_*`; unknown → HEVC).
|
||||
public init(wire: UInt8) {
|
||||
self = wire == 0x01 ? .h264 : .hevc // 0x01 = PUNKTFUNK_CODEC_H264
|
||||
}
|
||||
}
|
||||
|
||||
public enum AnnexB {
|
||||
/// Split an Annex-B stream into NAL units (start codes 00 00 01 / 00 00 00 01 stripped).
|
||||
/// All zeros immediately preceding a start code are dropped: they're either the
|
||||
@@ -47,40 +61,83 @@ public enum AnnexB {
|
||||
return (first >> 1) & 0x3F
|
||||
}
|
||||
|
||||
/// Build a format description from an IDR AU's in-band VPS(32)/SPS(33)/PPS(34).
|
||||
/// Returns nil when the AU carries no parameter sets (non-IDR).
|
||||
public static func formatDescription(fromIDR au: Data) -> CMVideoFormatDescription? {
|
||||
/// H.264 NAL unit type (bits 0..4 of the first byte).
|
||||
public static func h264NalType(_ nal: Data) -> UInt8 {
|
||||
guard let first = nal.first else { return 0xFF }
|
||||
return first & 0x1F
|
||||
}
|
||||
|
||||
/// True if this NAL is a parameter set for `codec` (dropped from AVCC; kept for the format desc).
|
||||
/// HEVC: VPS 32 / SPS 33 / PPS 34. H.264: SPS 7 / PPS 8 (no VPS).
|
||||
private static func isParameterSet(_ nal: Data, _ codec: VideoCodec) -> Bool {
|
||||
switch codec {
|
||||
case .hevc: let t = hevcNalType(nal); return t == 32 || t == 33 || t == 34
|
||||
case .h264: let t = h264NalType(nal); return t == 7 || t == 8
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a format description from an IDR AU's in-band parameter sets (HEVC: VPS/SPS/PPS;
|
||||
/// H.264: SPS/PPS). Returns nil when the AU carries no parameter sets (non-IDR).
|
||||
public static func formatDescription(fromIDR au: Data, codec: VideoCodec)
|
||||
-> CMVideoFormatDescription?
|
||||
{
|
||||
// Collect the parameter-set NALs in the order VideoToolbox wants them (HEVC: VPS,SPS,PPS;
|
||||
// H.264: SPS,PPS).
|
||||
var vps: Data?, sps: Data?, pps: Data?
|
||||
for nal in nalUnits(in: au) {
|
||||
switch hevcNalType(nal) {
|
||||
case 32: vps = nal
|
||||
case 33: sps = nal
|
||||
case 34: pps = nal
|
||||
default: break
|
||||
switch codec {
|
||||
case .hevc:
|
||||
switch hevcNalType(nal) {
|
||||
case 32: vps = nal
|
||||
case 33: sps = nal
|
||||
case 34: pps = nal
|
||||
default: break
|
||||
}
|
||||
case .h264:
|
||||
switch h264NalType(nal) {
|
||||
case 7: sps = nal
|
||||
case 8: pps = nal
|
||||
default: break
|
||||
}
|
||||
}
|
||||
}
|
||||
guard let vps, let sps, let pps else { return nil }
|
||||
guard let sps, let pps else { return nil }
|
||||
let sets: [Data] = codec == .hevc ? [vps, sps, pps].compactMap { $0 } : [sps, pps]
|
||||
guard codec == .h264 || sets.count == 3 else { return nil } // HEVC needs the VPS too
|
||||
|
||||
var format: CMVideoFormatDescription?
|
||||
let sets = [vps, sps, pps]
|
||||
let status: OSStatus = sets[0].withUnsafeBytes { v in
|
||||
sets[1].withUnsafeBytes { s in
|
||||
sets[2].withUnsafeBytes { p in
|
||||
let pointers: [UnsafePointer<UInt8>] = [
|
||||
v.bindMemory(to: UInt8.self).baseAddress!,
|
||||
s.bindMemory(to: UInt8.self).baseAddress!,
|
||||
p.bindMemory(to: UInt8.self).baseAddress!,
|
||||
]
|
||||
let sizes = [vps.count, sps.count, pps.count]
|
||||
return CMVideoFormatDescriptionCreateFromHEVCParameterSets(
|
||||
allocator: kCFAllocatorDefault,
|
||||
parameterSetCount: 3,
|
||||
parameterSetPointers: pointers,
|
||||
parameterSetSizes: sizes,
|
||||
nalUnitHeaderLength: 4,
|
||||
extensions: nil,
|
||||
formatDescriptionOut: &format)
|
||||
}
|
||||
// Pin every parameter set's bytes for the duration of the create call, then hand
|
||||
// VideoToolbox parallel pointer/size arrays.
|
||||
var pointers: [UnsafePointer<UInt8>] = []
|
||||
var sizes: [Int] = []
|
||||
func withAll(_ i: Int, _ body: () -> Void) {
|
||||
if i == sets.count { body(); return }
|
||||
sets[i].withUnsafeBytes { raw in
|
||||
pointers.append(raw.bindMemory(to: UInt8.self).baseAddress!)
|
||||
sizes.append(sets[i].count)
|
||||
withAll(i + 1, body)
|
||||
}
|
||||
}
|
||||
var status: OSStatus = -1
|
||||
withAll(0) {
|
||||
switch codec {
|
||||
case .hevc:
|
||||
status = CMVideoFormatDescriptionCreateFromHEVCParameterSets(
|
||||
allocator: kCFAllocatorDefault,
|
||||
parameterSetCount: pointers.count,
|
||||
parameterSetPointers: pointers,
|
||||
parameterSetSizes: sizes,
|
||||
nalUnitHeaderLength: 4,
|
||||
extensions: nil,
|
||||
formatDescriptionOut: &format)
|
||||
case .h264:
|
||||
status = CMVideoFormatDescriptionCreateFromH264ParameterSets(
|
||||
allocator: kCFAllocatorDefault,
|
||||
parameterSetCount: pointers.count,
|
||||
parameterSetPointers: pointers,
|
||||
parameterSetSizes: sizes,
|
||||
nalUnitHeaderLength: 4,
|
||||
formatDescriptionOut: &format)
|
||||
}
|
||||
}
|
||||
return status == noErr ? format : nil
|
||||
@@ -88,11 +145,10 @@ public enum AnnexB {
|
||||
|
||||
/// Re-pack an Annex-B AU as AVCC (4-byte big-endian length before each NAL), dropping
|
||||
/// the parameter-set NALs (they live in the format description).
|
||||
public static func avcc(from au: Data) -> Data {
|
||||
public static func avcc(from au: Data, codec: VideoCodec) -> Data {
|
||||
var out = Data(capacity: au.count + 16)
|
||||
for nal in nalUnits(in: au) {
|
||||
let t = hevcNalType(nal)
|
||||
if t == 32 || t == 33 || t == 34 { continue } // VPS/SPS/PPS
|
||||
if isParameterSet(nal, codec) { continue }
|
||||
var len = UInt32(nal.count).bigEndian
|
||||
withUnsafeBytes(of: &len) { out.append(contentsOf: $0) }
|
||||
out.append(nal)
|
||||
@@ -102,9 +158,9 @@ public enum AnnexB {
|
||||
|
||||
/// Wrap one AU as a decode-ready CMSampleBuffer.
|
||||
public static func sampleBuffer(
|
||||
au: AccessUnit, format: CMVideoFormatDescription
|
||||
au: AccessUnit, format: CMVideoFormatDescription, codec: VideoCodec
|
||||
) -> CMSampleBuffer? {
|
||||
let avccData = avcc(from: au.data)
|
||||
let avccData = avcc(from: au.data, codec: codec)
|
||||
var blockBuffer: CMBlockBuffer?
|
||||
guard CMBlockBufferCreateWithMemoryBlock(
|
||||
allocator: kCFAllocatorDefault, memoryBlock: nil,
|
||||
|
||||
Reference in New Issue
Block a user