feat(clients/apple): AV1 decode support — OBU plumbing, hardware-gated advertisement
apple / swift (push) Successful in 1m15s
release / apple (push) Successful in 15m35s
apple / screenshots (push) Successful in 4m35s
android / android (push) Successful in 4m17s
arch / build-publish (push) Successful in 5m51s
ci / web (push) Successful in 1m5s
ci / docs-site (push) Successful in 1m22s
ci / rust (push) Successful in 5m14s
deb / build-publish (push) Successful in 3m31s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
ci / bench (push) Successful in 5m0s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 11m12s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 11m20s
docker / deploy-docs (push) Successful in 6s
apple / swift (push) Successful in 1m15s
release / apple (push) Successful in 15m35s
apple / screenshots (push) Successful in 4m35s
android / android (push) Successful in 4m17s
arch / build-publish (push) Successful in 5m51s
ci / web (push) Successful in 1m5s
ci / docs-site (push) Successful in 1m22s
ci / rust (push) Successful in 5m14s
deb / build-publish (push) Successful in 3m31s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
ci / bench (push) Successful in 5m0s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 11m12s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 11m20s
docker / deploy-docs (push) Successful in 6s
The Apple client was HEVC/H.264-only: the receive path spoke Annex-B NALs exclusively, so AV1 was never advertised and the codec picker hid it. Add the OBU flavor of the same plumbing (AV1.swift, sibling of AnnexB.swift): a zero-copy OBU walker, a full spec-5.5.1 sequence-header parser, an av1C CMVideoFormatDescription with colorimetry extensions (so isHDRFormat and the presenter stay codec-agnostic), and an ISOBMFF 'av01' sample repack (temporal delimiter stripped, everything size-fielded, one copy per AU). VideoCodec gains .av1 (wire 0x04); both pumps and VideoDecoder route through dispatching formatDescription(fromKeyframe:)/sampleBuffer(au:) — keyframe gating keys on the in-band sequence header exactly as the NAL codecs key on in-band parameter sets, so loss recovery and mid-session reconfigure work unchanged. AV1 sessions require a hardware decoder (VideoToolbox has no software AV1; same fail-fast policy as 4:4:4), and both the Hello advertisement and the Settings picker are gated on VTIsHardwareDecodeSupported — AV1 only appears on devices that can actually decode it (M3-class Macs, A17 Pro-class iPhones; no Apple TV). Tests: real SVT-AV1 blobs (generation recipe in the file) cover the walk, the parse against an independent reference, av1C bytes, delta-TU gating, repack byte-exactness, and — on AV1 hardware — a real VTDecompressionSession decode through VideoDecoder. Host precedence stays HEVC > AV1 > H.264, so AV1 engages only when explicitly picked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,4 +1,5 @@
|
||||
// Stage-2 presenter, decode half: explicit VideoToolbox decode of the host's HEVC AUs.
|
||||
// Stage-2 presenter, decode half: explicit VideoToolbox decode of the host's AUs (H.264 /
|
||||
// HEVC / AV1 — whatever the Welcome resolved).
|
||||
//
|
||||
// Stage-1 hands compressed samples to AVSampleBufferDisplayLayer, which decodes AND presents
|
||||
// internally with no per-frame callback — so neither decode-completion nor present can be
|
||||
@@ -61,8 +62,8 @@ public final class VideoDecoder: @unchecked Sendable {
|
||||
/// depth / HDR). Read inside `createSessionLocked` under `lock`.
|
||||
private var chroma444 = false
|
||||
|
||||
/// The negotiated codec (`connection.videoCodec`), set once at session start. Drives the AnnexB
|
||||
/// NAL parsing (H.264 vs HEVC parameter sets). Read under `lock`.
|
||||
/// The negotiated codec (`connection.videoCodec`), set once at session start. Drives the
|
||||
/// bitstream framing (H.264/HEVC NAL parsing vs AV1 OBU repack). Read under `lock`.
|
||||
private var codec: VideoCodec = .hevc
|
||||
|
||||
public init(
|
||||
@@ -84,8 +85,8 @@ public final class VideoDecoder: @unchecked Sendable {
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
/// Select the negotiated codec (H.264 vs HEVC). Call once at session start, before decoding,
|
||||
/// from `connection.videoCodec`. Thread-safe.
|
||||
/// Select the negotiated codec (H.264 / HEVC / AV1). Call once at session start, before
|
||||
/// decoding, from `connection.videoCodec`. Thread-safe.
|
||||
public func setCodec(_ c: VideoCodec) {
|
||||
lock.lock()
|
||||
codec = c
|
||||
@@ -93,8 +94,9 @@ public final class VideoDecoder: @unchecked Sendable {
|
||||
}
|
||||
|
||||
/// Submit one AU for asynchronous decode, (re)creating the session if `format` changed. The
|
||||
/// caller resolves `format` from the IDR exactly as stage-1 does (`AnnexB.formatDescription`).
|
||||
/// Returns false if the session couldn't be created or the frame couldn't be submitted.
|
||||
/// caller resolves `format` from the keyframe exactly as stage-1 does
|
||||
/// (`VideoCodec.formatDescription(fromKeyframe:)`). Returns false if the session couldn't be
|
||||
/// created or the frame couldn't be submitted.
|
||||
@discardableResult
|
||||
public func decode(au: AccessUnit, format newFormat: CMVideoFormatDescription) -> Bool {
|
||||
lock.lock()
|
||||
@@ -112,7 +114,7 @@ public final class VideoDecoder: @unchecked Sendable {
|
||||
// invalidate the session between here and DecodeFrame. The VT output callback takes the
|
||||
// ring lock, not this one, so there's no re-entrancy. DecodeFrame is async — non-blocking.
|
||||
guard let session,
|
||||
let sample = AnnexB.sampleBuffer(au: au, format: newFormat, codec: codec)
|
||||
let sample = codec.sampleBuffer(au: au, format: newFormat)
|
||||
else { lock.unlock(); return false }
|
||||
var infoOut = VTDecodeInfoFlags()
|
||||
let status = VTDecompressionSessionDecodeFrame(
|
||||
@@ -199,13 +201,14 @@ public final class VideoDecoder: @unchecked Sendable {
|
||||
var callback = VTDecompressionOutputCallbackRecord(
|
||||
decompressionOutputCallback: decoderOutputCallback,
|
||||
decompressionOutputRefCon: Unmanaged.passUnretained(self).toOpaque())
|
||||
// 4:4:4 sessions REQUIRE a hardware decoder: we only advertise 4:4:4 when the hardware probe
|
||||
// passed, so a hardware-incapable mode (e.g. a resolution past the HW 4:4:4 ceiling) must fail
|
||||
// HERE, synchronously, letting the pump's backstop end the session — rather than silently
|
||||
// falling back to a software 4:4:4 decoder far too slow for a real-time stream. 4:2:0 keeps the
|
||||
// software fallback (nil spec) as a robustness net.
|
||||
// 4:4:4 and AV1 sessions REQUIRE a hardware decoder: both are only advertised when the
|
||||
// hardware gate passed (the 4:4:4 probe / `AV1.hardwareDecodeSupported`), so a
|
||||
// hardware-incapable mode (e.g. a resolution past a HW ceiling) must fail HERE,
|
||||
// synchronously, letting the pump's backstop end the session — rather than silently
|
||||
// falling back to a software decoder far too slow for a real-time stream. 4:2:0
|
||||
// H.264/HEVC keeps the software fallback (nil spec) as a robustness net.
|
||||
let spec: CFDictionary? =
|
||||
chroma444
|
||||
chroma444 || codec == .av1
|
||||
? [kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder: true] as CFDictionary
|
||||
: nil
|
||||
var newSession: VTDecompressionSession?
|
||||
|
||||
Reference in New Issue
Block a user