feat(pyrowave): Apple Metal 4:4:4 + HDR decode, EDR present — self-configured in-band
docker / deploy-docs (push) Successful in 11s
flatpak / build-publish (push) Successful in 6m25s
deb / build-publish (push) Successful in 9m7s
windows-host / package (push) Successful in 16m12s
deb / build-publish-host (push) Successful in 9m31s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 6m41s
ci / rust (push) Successful in 25m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 8m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 20m28s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m59s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 6m45s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 8m2s
apple / swift (push) Successful in 1m15s
release / apple (push) Successful in 9m36s
apple / screenshots (push) Successful in 6m43s
ci / web (push) Successful in 56s
ci / docs-site (push) Successful in 50s
android / android (push) Successful in 13m41s
decky / build-publish (push) Successful in 22s
arch / build-publish (push) Successful in 14m23s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 12s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 57s
ci / bench (push) Successful in 5m58s

Phases 4+5 of design/pyrowave-444-hdr.md. The Metal decoder needs NO new ABI:
every frame's sequence header carries chroma (444) and, since the Phase-3
stamps, the PQ/BT.2020 bits — so the decoder self-configures per session.

Decoder: WaveletLayout grows the 4:4:4 block space (chroma runs the full
pyramid like luma — no level-0 skip, no early half-res emit; the Metal
kernels were already chroma-agnostic, only the dispatch structure changes);
the parser accepts chroma_resolution=444, reads the PQ transfer bit, and
lifts the even-dims rule for 444; the plane ring allocates full-res chroma
and r16Unorm for PQ streams; CSC rows switch to depth-10 MSB-packed.

Presenter: planar HDR passthrough reuses pf_frag_planar on an rgba16Float
drawable (itur_2100_PQ + EDR metadata interpret the samples — same split as
pf_frag/pf_frag_hdr), plus a new pf_frag_planar_tm PQ->SDR tone-map (shared
pqToSdr tail refactored out of pf_frag_hdr_tv) for tvOS-without-headroom AND
macOS WINDOWED sessions, whose IOSurface present path (the DCP-panic
mitigation) is BGRA8-only. SessionModel stops stripping the HDR/10-bit/444
caps on the PyroWave opt-in.

New golden: au-dense444 + upstream's own 4:4:4 reference planes (regenerated
via the extended pyrowave_dump_golden); Metal decode matches at 64-67 dB
(420 fixtures re-verify 77-88 dB). Full Apple suite 157 tests green on a
real M-series GPU. Docs updated: the 8-bit-SDR-only wording is gone, the
Windows host is no longer 'on the roadmap', bpp scaling documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-18 13:21:23 +02:00
parent 188edde2b3
commit 3ff1973d7f
9 changed files with 167 additions and 57 deletions
@@ -264,14 +264,11 @@ final class SessionModel: ObservableObject {
// PyroWave (wired LAN) is a pure opt-in: picking it in the codec setting both // PyroWave (wired LAN) is a pure opt-in: picking it in the codec setting both
// advertises the bit and prefers it the host never auto-selects it, and the // advertises the bit and prefers it the host never auto-selects it, and the
// picker only offers it when the Metal decode probe passed (simdgroup floor A13; // picker only offers it when the Metal decode probe passed (simdgroup floor A13;
// every M-series Mac and the ATV 4K gen 3 pass). The codec is 8-bit 4:2:0 SDR // every M-series Mac and the ATV 4K gen 3 pass). The decoder self-configures from
// BT.709 by contract, so the opt-in also drops the HDR/10-bit/4:4:4 caps for this // the per-frame sequence header (4:2:0/4:4:4, SDR/PQ design/pyrowave-444-hdr.md),
// session HDR sessions stay HEVC/AV1 (plan §4.7). // so the session keeps the user's HDR/10-bit/4:4:4 caps exactly like HEVC/AV1.
if preferredCodec == PunktfunkConnection.codecPyroWave, MetalWaveletDecoder.supported { if preferredCodec == PunktfunkConnection.codecPyroWave, MetalWaveletDecoder.supported {
videoCodecs |= PunktfunkConnection.codecPyroWave videoCodecs |= PunktfunkConnection.codecPyroWave
videoCaps &= ~(PunktfunkConnection.videoCap10Bit
| PunktfunkConnection.videoCapHDR
| PunktfunkConnection.videoCap444)
} }
let result = Result { try PunktfunkConnection( let result = Result { try PunktfunkConnection(
host: host.address, port: host.port, host: host.address, port: host.port,
@@ -196,13 +196,11 @@ fragment float4 pf_frag_hdr(VOut in [[stage_in]],
// display-referred SDR. (When the display IS in an HDR mode — requested per session via // display-referred SDR. (When the display IS in an HDR mode — requested per session via
// AVDisplayManager, see StreamViewIOS — tvOS presents pf_frag_hdr's PQ passthrough instead: // AVDisplayManager, see StreamViewIOS — tvOS presents pf_frag_hdr's PQ passthrough instead:
// in a genuine HDR10 output, PQ passthrough is the correct emission and the TV tone-maps.) // in a genuine HDR10 output, PQ passthrough is the correct emission and the TV tone-maps.)
fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]], // The shared PQ→display-referred-SDR tail (see pf_frag_hdr_tv's rationale above): ST 2084
texture2d<float> lumaTex [[texture(0)]], // EOTF → 203-nit-anchored scene light → BT.2020→709 primaries → extended-Reinhard rolloff →
texture2d<float> chromaTex [[texture(1)]], // BT.709 OETF. Used by the tvOS biplanar tone-map and the planar (PyroWave) tone-map — the
constant CscUniform& csc [[buffer(0)]]) { // latter also on macOS windowed sessions, whose IOSurface present path is BGRA8-only.
// YCbCr → full-range PQ RGB via the per-frame rows (as pf_frag_hdr). static inline float3 pqToSdr(float3 pq) {
float3 pq = sampleRgb(lumaTex, chromaTex, in.uv, csc);
// ST 2084 EOTF: PQ code value → linear light, 1.0 = 10,000 nits.
const float m1 = 2610.0/16384.0; const float m1 = 2610.0/16384.0;
const float m2 = 78.84375; const float m2 = 78.84375;
const float c1 = 3424.0/4096.0; const float c1 = 3424.0/4096.0;
@@ -210,20 +208,49 @@ fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]],
const float c3 = 18.6875; const float c3 = 18.6875;
float3 p = pow(pq, 1.0/m2); float3 p = pow(pq, 1.0/m2);
float3 lin = pow(max(p - c1, 0.0) / (c2 - c3 * p), 1.0/m1); float3 lin = pow(max(p - c1, 0.0) / (c2 - c3 * p), 1.0/m1);
// Scene-referred with diffuse white at 1.0 (the same 203-nit anchor the EDR path uses).
float3 t = lin * (10000.0/203.0); float3 t = lin * (10000.0/203.0);
// BT.2020 → BT.709 primaries while still linear; negatives are out-of-gamut, floor them.
float3 t709 = float3( float3 t709 = float3(
dot(t, float3( 1.6605, -0.5876, -0.0728)), dot(t, float3( 1.6605, -0.5876, -0.0728)),
dot(t, float3(-0.1246, 1.1329, -0.0083)), dot(t, float3(-0.1246, 1.1329, -0.0083)),
dot(t, float3(-0.0182, -0.1006, 1.1187))); dot(t, float3(-0.0182, -0.1006, 1.1187)));
t709 = max(t709, 0.0); t709 = max(t709, 0.0);
// Extended Reinhard: 1.0 stays put, the 1000-nit knee lands at display white, above rolls off.
const float w = 1000.0/203.0; const float w = 1000.0/203.0;
float3 mapped = saturate(t709 * (1.0 + t709 / (w * w)) / (1.0 + t709)); float3 mapped = saturate(t709 * (1.0 + t709 / (w * w)) / (1.0 + t709));
// BT.709 OETF — the same encoding the SDR stream arrives in, so both paths present alike.
float3 e = select(1.099 * pow(mapped, 0.45) - 0.099, 4.5 * mapped, mapped < 0.018); float3 e = select(1.099 * pow(mapped, 0.45) - 0.099, 4.5 * mapped, mapped < 0.018);
return float4(e, 1.0); return e;
}
fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> chromaTex [[texture(1)]],
constant CscUniform& csc [[buffer(0)]]) {
// YCbCr → full-range PQ RGB via the per-frame rows (as pf_frag_hdr), then the tail.
return float4(pqToSdr(sampleRgb(lumaTex, chromaTex, in.uv, csc)), 1.0);
}
// PyroWave planar HDR tone-map: three separate R16 planes (P010-style studio codes; the rows
// fold in depth-10 MSB packing) → PQ RGB → the shared SDR tail. Used when a PQ pyrowave
// stream must land on an 8-bit surface: tvOS without HDR headroom, and macOS WINDOWED sessions
// (the IOSurface present path — the DCP-panic mitigation — is BGRA8). The passthrough planar
// HDR pipeline reuses pf_frag_planar itself on an rgba16Float drawable (identical math — the
// layer's itur_2100_PQ colour space + EDR metadata do the interpretation).
fragment float4 pf_frag_planar_tm(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> cbTex [[texture(1)]],
texture2d<float> crTex [[texture(2)]],
constant CscUniform& csc [[buffer(0)]]) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
#ifdef PF_BILINEAR_LUMA
float lumaY = lumaTex.sample(s, in.uv).r;
#else
float lumaY = catmullRomLuma(lumaTex, s, in.uv);
#endif
float2 cuv = chromaUV(lumaTex, cbTex, in.uv);
float3 yuv = float3(lumaY, cbTex.sample(s, cuv).r, crTex.sample(s, cuv).r);
float3 pq = saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
dot(csc.r1.xyz, yuv) + csc.r1.w,
dot(csc.r2.xyz, yuv) + csc.r2.w));
return float4(pqToSdr(pq), 1.0);
} }
""" """
@@ -287,6 +314,11 @@ public final class MetalVideoPresenter {
private let pipelineHDRToneMap: MTLRenderPipelineState? private let pipelineHDRToneMap: MTLRenderPipelineState?
/// PyroWave's 3-plane SDR path (pf_frag_planar bgra8) see `renderPlanar`. /// PyroWave's 3-plane SDR path (pf_frag_planar bgra8) see `renderPlanar`.
private let pipelinePlanar: MTLRenderPipelineState private let pipelinePlanar: MTLRenderPipelineState
/// PyroWave planar HDR passthrough (pf_frag_planar rgba16Float; the layer's PQ colour
/// space + EDR interpret the samples) and the planar PQSDR tone-map (pf_frag_planar_tm
/// bgra8; tvOS without headroom + macOS windowed IOSurface presents).
private let pipelinePlanarHDR: MTLRenderPipelineState
private let pipelinePlanarToneMap: MTLRenderPipelineState
private var textureCache: CVMetalTextureCache? private var textureCache: CVMetalTextureCache?
/// The PyroWave Metal decoder records on the presenter's device + queue: one device means /// The PyroWave Metal decoder records on the presenter's device + queue: one device means
@@ -337,6 +369,8 @@ public final class MetalVideoPresenter {
let pipelineHDR: MTLRenderPipelineState let pipelineHDR: MTLRenderPipelineState
let pipelineHDRToneMap: MTLRenderPipelineState? let pipelineHDRToneMap: MTLRenderPipelineState?
let pipelinePlanar: MTLRenderPipelineState let pipelinePlanar: MTLRenderPipelineState
let pipelinePlanarHDR: MTLRenderPipelineState
let pipelinePlanarToneMap: MTLRenderPipelineState
do { do {
// DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF // DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF
// (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is // (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is
@@ -374,8 +408,18 @@ public final class MetalVideoPresenter {
let planar = MTLRenderPipelineDescriptor() let planar = MTLRenderPipelineDescriptor()
planar.vertexFunction = vtx planar.vertexFunction = vtx
planar.fragmentFunction = library.makeFunction(name: "pf_frag_planar") planar.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
planar.colorAttachments[0].pixelFormat = .bgra8Unorm // PyroWave is 8-bit SDR planar.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelinePlanar = try device.makeRenderPipelineState(descriptor: planar) pipelinePlanar = try device.makeRenderPipelineState(descriptor: planar)
let planarHdr = MTLRenderPipelineDescriptor()
planarHdr.vertexFunction = vtx
planarHdr.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
planarHdr.colorAttachments[0].pixelFormat = .rgba16Float // PQ passthrough
pipelinePlanarHDR = try device.makeRenderPipelineState(descriptor: planarHdr)
let planarTm = MTLRenderPipelineDescriptor()
planarTm.vertexFunction = vtx
planarTm.fragmentFunction = library.makeFunction(name: "pf_frag_planar_tm")
planarTm.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelinePlanarToneMap = try device.makeRenderPipelineState(descriptor: planarTm)
} catch { } catch {
return nil return nil
} }
@@ -416,6 +460,7 @@ public final class MetalVideoPresenter {
return MetalVideoPresenter( return MetalVideoPresenter(
device: device, queue: queue, pipelineSDR: pipelineSDR, pipelineHDR: pipelineHDR, device: device, queue: queue, pipelineSDR: pipelineSDR, pipelineHDR: pipelineHDR,
pipelineHDRToneMap: pipelineHDRToneMap, pipelinePlanar: pipelinePlanar, pipelineHDRToneMap: pipelineHDRToneMap, pipelinePlanar: pipelinePlanar,
pipelinePlanarHDR: pipelinePlanarHDR, pipelinePlanarToneMap: pipelinePlanarToneMap,
textureCache: textureCache, layer: layer) textureCache: textureCache, layer: layer)
} }
@@ -423,6 +468,8 @@ public final class MetalVideoPresenter {
device: MTLDevice, queue: MTLCommandQueue, pipelineSDR: MTLRenderPipelineState, device: MTLDevice, queue: MTLCommandQueue, pipelineSDR: MTLRenderPipelineState,
pipelineHDR: MTLRenderPipelineState, pipelineHDRToneMap: MTLRenderPipelineState?, pipelineHDR: MTLRenderPipelineState, pipelineHDRToneMap: MTLRenderPipelineState?,
pipelinePlanar: MTLRenderPipelineState, pipelinePlanar: MTLRenderPipelineState,
pipelinePlanarHDR: MTLRenderPipelineState,
pipelinePlanarToneMap: MTLRenderPipelineState,
textureCache: CVMetalTextureCache, layer: CAMetalLayer textureCache: CVMetalTextureCache, layer: CAMetalLayer
) { ) {
self.device = device self.device = device
@@ -431,6 +478,8 @@ public final class MetalVideoPresenter {
self.pipelineHDR = pipelineHDR self.pipelineHDR = pipelineHDR
self.pipelineHDRToneMap = pipelineHDRToneMap self.pipelineHDRToneMap = pipelineHDRToneMap
self.pipelinePlanar = pipelinePlanar self.pipelinePlanar = pipelinePlanar
self.pipelinePlanarHDR = pipelinePlanarHDR
self.pipelinePlanarToneMap = pipelinePlanarToneMap
self.textureCache = textureCache self.textureCache = textureCache
self.layer = layer self.layer = layer
} }
@@ -652,7 +701,13 @@ public final class MetalVideoPresenter {
let surfaceMode = surfacePresentsStaged let surfaceMode = surfacePresentsStaged
#endif #endif
stagingLock.unlock() stagingLock.unlock()
configure(hdr: false) // A PQ (HDR) pyrowave stream drives the same layer/EDR machinery as the biplanar path;
// macOS WINDOWED sessions stay on the SDR layer (the IOSurface path tone-maps in-shader).
#if os(macOS)
configure(hdr: planes.pq && !surfaceMode)
#else
configure(hdr: planes.pq)
#endif
var csc = planes.csc var csc = planes.csc
#if os(macOS) #if os(macOS)
if surfaceMode != surfacePresentsActive { if surfaceMode != surfacePresentsActive {
@@ -672,9 +727,21 @@ public final class MetalVideoPresenter {
planes, targetFromLayout: targetFromLayout, csc: &csc, onPresented: onPresented) planes, targetFromLayout: targetFromLayout, csc: &csc, onPresented: onPresented)
} }
#endif #endif
// PQ passthrough needs the HDR drawable; a PQ frame while the drawable is (still)
// 8-bit tvOS without display headroom, or a not-yet-flipped layer tone-maps
// in-shader instead (the pipeline must match the drawable's pixel format).
#if os(tvOS)
let planarPassthrough = hdrActive && hdrPassthroughActive
#else
let planarPassthrough = hdrActive
#endif
let planarPipeline: MTLRenderPipelineState =
planes.pq
? (planarPassthrough ? pipelinePlanarHDR : pipelinePlanarToneMap)
: pipelinePlanar
return encodePresent( return encodePresent(
decodedSize: CGSize(width: planes.width, height: planes.height), decodedSize: CGSize(width: planes.width, height: planes.height),
targetFromLayout: targetFromLayout, pipeline: pipelinePlanar, targetFromLayout: targetFromLayout, pipeline: planarPipeline,
presentAtMediaTime: presentAtMediaTime, onPresented: onPresented, presentAtMediaTime: presentAtMediaTime, onPresented: onPresented,
// The ring textures stay valid by ring depth; retaining them here also pins the // The ring textures stay valid by ring depth; retaining them here also pins the
// slot's set until the sample completes (mirrors the biplanar keep-alive). // slot's set until the sample completes (mirrors the biplanar keep-alive).
@@ -716,7 +783,7 @@ public final class MetalVideoPresenter {
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else { guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else {
return false return false
} }
encoder.setRenderPipelineState(pipelinePlanar) encoder.setRenderPipelineState(planes.pq ? pipelinePlanarToneMap : pipelinePlanar)
encoder.setFragmentTexture(planes.y, index: 0) encoder.setFragmentTexture(planes.y, index: 0)
encoder.setFragmentTexture(planes.cb, index: 1) encoder.setFragmentTexture(planes.cb, index: 1)
encoder.setFragmentTexture(planes.cr, index: 2) encoder.setFragmentTexture(planes.cr, index: 2)
@@ -47,6 +47,9 @@ struct WaveletLayout {
let width: Int let width: Int
let height: Int let height: Int
/// Full-res chroma (4:4:4): chroma components get the full band set including level 0,
/// exactly like luma upstream `init_block_meta` with `Chroma444`.
let chroma444: Bool
let alignedWidth: Int let alignedWidth: Int
let alignedHeight: Int let alignedHeight: Int
/// blockMeta[component][level][band] = (blockOffset32x32, blockStride32x32); -1 offset = /// blockMeta[component][level][band] = (blockOffset32x32, blockStride32x32); -1 offset =
@@ -59,9 +62,10 @@ struct WaveletLayout {
func levelWidth(_ level: Int) -> Int { (alignedWidth / 2) >> level } func levelWidth(_ level: Int) -> Int { (alignedWidth / 2) >> level }
func levelHeight(_ level: Int) -> Int { (alignedHeight / 2) >> level } func levelHeight(_ level: Int) -> Int { (alignedHeight / 2) >> level }
init(width: Int, height: Int) { init(width: Int, height: Int, chroma444: Bool) {
self.width = width self.width = width
self.height = height self.height = height
self.chroma444 = chroma444
let align = { (v: Int) in let align = { (v: Int) in
max((v + Self.alignment - 1) & ~(Self.alignment - 1), Self.minimumImageSize) max((v + Self.alignment - 1) & ~(Self.alignment - 1), Self.minimumImageSize)
} }
@@ -78,7 +82,7 @@ struct WaveletLayout {
let ah = alignedHeight let ah = alignedHeight
for level in stride(from: Self.decompositionLevels - 1, through: 0, by: -1) { for level in stride(from: Self.decompositionLevels - 1, through: 0, by: -1) {
for component in 0..<3 { for component in 0..<3 {
if level == 0 && component != 0 { continue } // 4:2:0: no top-level chroma if level == 0 && component != 0 && !chroma444 { continue } // 4:2:0: no top-level chroma
for band in (level == Self.decompositionLevels - 1 ? 0 : 1)..<4 { for band in (level == Self.decompositionLevels - 1 ? 0 : 1)..<4 {
let levelW = (aw / 2) >> level let levelW = (aw / 2) >> level
let levelH = (ah / 2) >> level let levelH = (ah / 2) >> level
@@ -108,6 +112,9 @@ struct ParsedWaveletFrame {
var decodedBlocks: Int var decodedBlocks: Int
/// VUI bits from the sequence header (BitstreamSequenceHeader). /// VUI bits from the sequence header (BitstreamSequenceHeader).
var bt2020: Bool var bt2020: Bool
/// PQ transfer HDR session: 16-bit studio-code planes + EDR present (the host stamps
/// this bit iff the session negotiated 10-bit the depth is coupled to the transfer).
var pq: Bool
var fullRange: Bool var fullRange: Bool
/// The frame's YCbCrRGB signal for the presenter's planar CSC. PyroWave today is always /// The frame's YCbCrRGB signal for the presenter's planar CSC. PyroWave today is always
@@ -203,6 +210,7 @@ enum WaveletBitstream {
var totalBlocks = 0 var totalBlocks = 0
var decodedBlocks = 0 var decodedBlocks = 0
var bt2020 = false var bt2020 = false
var pq = false
var fullRange = false var fullRange = false
var sawSOF = false var sawSOF = false
@@ -220,22 +228,28 @@ enum WaveletBitstream {
// siting[31]. // siting[31].
let code = (word1 >> 24) & 0x3 let code = (word1 >> 24) & 0x3
guard code == 0 else { return false } // only START_OF_FRAME is defined guard code == 0 else { return false } // only START_OF_FRAME is defined
let chromaRes = (word1 >> 26) & 1 let chroma444 = (word1 >> 26) & 1 != 0
guard chromaRes == 0 else { return false } // host contract: 4:2:0
let w = Int(word0 & 0x3fff) + 1 let w = Int(word0 & 0x3fff) + 1
let h = Int((word0 >> 14) & 0x3fff) + 1 let h = Int((word0 >> 14) & 0x3fff) + 1
guard w >= 2, h >= 2, w % 2 == 0, h % 2 == 0 else { return false } guard w >= 2, h >= 2, chroma444 || (w % 2 == 0 && h % 2 == 0) else {
return false
}
if sawSOF { if sawSOF {
// One frame, one geometry a second SOF must agree. // One frame, one geometry a second SOF must agree.
guard layout?.width == w, layout?.height == h else { return false } guard layout?.width == w, layout?.height == h,
layout?.chroma444 == chroma444
else { return false }
} else { } else {
sawSOF = true sawSOF = true
let l = WaveletLayout(width: w, height: h) let l = WaveletLayout(width: w, height: h, chroma444: chroma444)
layout = l layout = l
offsets = [UInt32](repeating: .max, count: l.blockCount32) offsets = [UInt32](repeating: .max, count: l.blockCount32)
payload.reserveCapacity(64 * 1024 / 4) payload.reserveCapacity(64 * 1024 / 4)
totalBlocks = Int(word1 & 0xff_ffff) totalBlocks = Int(word1 & 0xff_ffff)
bt2020 = (word1 >> 29) & 1 != 0 bt2020 = (word1 >> 29) & 1 != 0
// transfer_function bit: PQ an HDR session (16-bit studio-code
// planes by the negotiated coupling design/pyrowave-444-hdr.md).
pq = (word1 >> 28) & 1 != 0
fullRange = (word1 >> 30) & 1 == 0 // YCBCR_RANGE_FULL = 0 fullRange = (word1 >> 30) & 1 == 0 // YCBCR_RANGE_FULL = 0
} }
pos += 8 pos += 8
@@ -280,7 +294,7 @@ enum WaveletBitstream {
return ParsedWaveletFrame( return ParsedWaveletFrame(
layout: layout, offsets: offsets, payload: payload, layout: layout, offsets: offsets, payload: payload,
totalBlocks: totalBlocks, decodedBlocks: decodedBlocks, totalBlocks: totalBlocks, decodedBlocks: decodedBlocks,
bt2020: bt2020, fullRange: fullRange) bt2020: bt2020, pq: pq, fullRange: fullRange)
} }
} }
} }
@@ -293,6 +307,8 @@ public struct WaveletPlanes: @unchecked Sendable {
public let cb: MTLTexture public let cb: MTLTexture
public let cr: MTLTexture public let cr: MTLTexture
public let csc: CscUniform public let csc: CscUniform
/// PQ (HDR) stream: the presenter picks the HDR/tone-map planar pipeline + EDR config.
public let pq: Bool
public var width: Int { y.width } public var width: Int { y.width }
public var height: Int { y.height } public var height: Int { y.height }
} }
@@ -351,6 +367,8 @@ public final class MetalWaveletDecoder {
private var slots: [Slot] = [] private var slots: [Slot] = []
private var nextSlot = 0 private var nextSlot = 0
/// The ring's plane format facts (from the last SOF): PQ 16-bit UNORM planes.
private var hdr16 = false
/// The current geometry (from the last SOF that built the resources) the pump reports /// The current geometry (from the last SOF that built the resources) the pump reports
/// decoded-size changes to the resize overlay from this. PUMP THREAD. /// decoded-size changes to the resize overlay from this. PUMP THREAD.
@@ -409,8 +427,9 @@ public final class MetalWaveletDecoder {
au: au, chunkAligned: chunkAligned, windowSize: windowSize) au: au, chunkAligned: chunkAligned, windowSize: windowSize)
else { return false } else { return false }
if layout?.width != frame.layout.width || layout?.height != frame.layout.height { if layout?.width != frame.layout.width || layout?.height != frame.layout.height
guard rebuild(layout: frame.layout) else { return false } || layout?.chroma444 != frame.layout.chroma444 || hdr16 != frame.pq {
guard rebuild(layout: frame.layout, hdr16: frame.pq) else { return false }
} }
guard let layout, !slots.isEmpty else { return false } guard let layout, !slots.isEmpty else { return false }
@@ -450,7 +469,7 @@ public final class MetalWaveletDecoder {
dequant.setBuffer(slot.payload, offset: 0, index: 1) dequant.setBuffer(slot.payload, offset: 0, index: 1)
for level in 0..<WaveletLayout.decompositionLevels { for level in 0..<WaveletLayout.decompositionLevels {
for component in 0..<3 { for component in 0..<3 {
if level == 0 && component != 0 { continue } // 4:2:0 if level == 0 && component != 0 && !layout.chroma444 { continue } // 4:2:0
for band in (level == WaveletLayout.decompositionLevels - 1 ? 0 : 1)..<4 { for band in (level == WaveletLayout.decompositionLevels - 1 ? 0 : 1)..<4 {
let meta = layout.blockMeta[component][level][band] let meta = layout.blockMeta[component][level][band]
let w = layout.levelWidth(level) let w = layout.levelWidth(level)
@@ -489,15 +508,20 @@ public final class MetalWaveletDecoder {
let grid = MTLSize(width: (rx + 15) / 16, height: (ry + 15) / 16, depth: 1) let grid = MTLSize(width: (rx + 15) / 16, height: (ry + 15) / 16, depth: 1)
let group = MTLSize(width: 64, height: 1, depth: 1) let group = MTLSize(width: 64, height: 1, depth: 1)
if inputLevel == 0 { if inputLevel == 0 {
// 4:2:0: the final full-res pass is luma only (chroma finished at level 1). // Final full-res pass: luma only in 4:2:0 (chroma finished at level 1); all
// three components in 4:4:4 (chroma runs the full pyramid like luma).
idwt.setComputePipelineState(idwtShiftPipeline) idwt.setComputePipelineState(idwtShiftPipeline)
idwt.setTexture(coefficients[0][0], index: 0) let components = layout.chroma444 ? 3 : 1
idwt.setTexture(slot.y, index: 1) for component in 0..<components {
idwt.setTexture(coefficients[component][0], index: 0)
let out = component == 0 ? slot.y : (component == 1 ? slot.cb : slot.cr)
idwt.setTexture(out, index: 1)
idwt.dispatchThreadgroups(grid, threadsPerThreadgroup: group) idwt.dispatchThreadgroups(grid, threadsPerThreadgroup: group)
}
} else { } else {
for component in 0..<3 { for component in 0..<3 {
idwt.setTexture(coefficients[component][inputLevel], index: 0) idwt.setTexture(coefficients[component][inputLevel], index: 0)
if component != 0 && inputLevel == 1 { if component != 0 && inputLevel == 1 && !layout.chroma444 {
// 4:2:0 chroma emits its final half-res plane one level early. // 4:2:0 chroma emits its final half-res plane one level early.
idwt.setComputePipelineState(idwtShiftPipeline) idwt.setComputePipelineState(idwtShiftPipeline)
idwt.setTexture(component == 1 ? slot.cb : slot.cr, index: 1) idwt.setTexture(component == 1 ? slot.cb : slot.cr, index: 1)
@@ -513,7 +537,9 @@ public final class MetalWaveletDecoder {
let planes = WaveletPlanes( let planes = WaveletPlanes(
y: slot.y, cb: slot.cb, cr: slot.cr, y: slot.y, cb: slot.cb, cr: slot.cr,
csc: CscRows.rows(frame.cscSignal, depth: 8, msbPacked: false)) csc: CscRows.rows(
frame.cscSignal, depth: frame.pq ? 10 : 8, msbPacked: frame.pq),
pq: frame.pq)
cmd.addCompletedHandler { buffer in cmd.addCompletedHandler { buffer in
completion(buffer.error == nil ? planes : nil) completion(buffer.error == nil ? planes : nil)
} }
@@ -524,9 +550,9 @@ public final class MetalWaveletDecoder {
/// (Re)allocate every size-dependent resource for `layout`'s geometry. Also the mid-stream /// (Re)allocate every size-dependent resource for `layout`'s geometry. Also the mid-stream
/// resize path: a Reconfigure shows up here as new SOF dims. /// resize path: a Reconfigure shows up here as new SOF dims.
private func rebuild(layout newLayout: WaveletLayout) -> Bool { private func rebuild(layout newLayout: WaveletLayout, hdr16 newHdr16: Bool) -> Bool {
waveletLog.info( waveletLog.info(
"pyrowave: building decoder \(newLayout.width)x\(newLayout.height) (aligned \(newLayout.alignedWidth)x\(newLayout.alignedHeight), \(newLayout.blockCount32) blocks)") "pyrowave: building decoder \(newLayout.width)x\(newLayout.height) (aligned \(newLayout.alignedWidth)x\(newLayout.alignedHeight), \(newLayout.blockCount32) blocks, \(newLayout.chroma444 ? "4:4:4" : "4:2:0", privacy: .public)\(newHdr16 ? " HDR16" : "", privacy: .public))")
var coeff: [[MTLTexture]] = [] var coeff: [[MTLTexture]] = []
var lls: [[MTLTexture]] = [] var lls: [[MTLTexture]] = []
for component in 0..<3 { for component in 0..<3 {
@@ -560,19 +586,22 @@ public final class MetalWaveletDecoder {
var newSlots: [Slot] = [] var newSlots: [Slot] = []
for i in 0..<Self.ringDepth { for i in 0..<Self.ringDepth {
let planeFormat: MTLPixelFormat = newHdr16 ? .r16Unorm : .r8Unorm
let plane = { (w: Int, h: Int, name: String) -> MTLTexture? in let plane = { (w: Int, h: Int, name: String) -> MTLTexture? in
let desc = MTLTextureDescriptor.texture2DDescriptor( let desc = MTLTextureDescriptor.texture2DDescriptor(
pixelFormat: .r8Unorm, width: w, height: h, mipmapped: false) pixelFormat: planeFormat, width: w, height: h, mipmapped: false)
desc.usage = [.shaderRead, .shaderWrite] desc.usage = [.shaderRead, .shaderWrite]
desc.storageMode = .private desc.storageMode = .private
let t = self.device.makeTexture(descriptor: desc) let t = self.device.makeTexture(descriptor: desc)
t?.label = name t?.label = name
return t return t
} }
let cw = newLayout.chroma444 ? newLayout.width : newLayout.width / 2
let ch = newLayout.chroma444 ? newLayout.height : newLayout.height / 2
guard guard
let y = plane(newLayout.width, newLayout.height, "pyrowave Y[\(i)]"), let y = plane(newLayout.width, newLayout.height, "pyrowave Y[\(i)]"),
let cb = plane(newLayout.width / 2, newLayout.height / 2, "pyrowave Cb[\(i)]"), let cb = plane(cw, ch, "pyrowave Cb[\(i)]"),
let cr = plane(newLayout.width / 2, newLayout.height / 2, "pyrowave Cr[\(i)]"), let cr = plane(cw, ch, "pyrowave Cr[\(i)]"),
let offsets = device.makeBuffer( let offsets = device.makeBuffer(
length: max(newLayout.blockCount32 * 4, 4), options: .storageModeShared), length: max(newLayout.blockCount32 * 4, 4), options: .storageModeShared),
let payload = device.makeBuffer(length: 64 * 1024, options: .storageModeShared) let payload = device.makeBuffer(length: 64 * 1024, options: .storageModeShared)
@@ -585,6 +614,7 @@ public final class MetalWaveletDecoder {
slots = newSlots slots = newSlots
nextSlot = 0 nextSlot = 0
layout = newLayout layout = newLayout
hdr16 = newHdr16
return true return true
} }
@@ -57,7 +57,7 @@ final class PyroWaveParserTests: XCTestCase {
func testLayoutMatchesUpstreamBlockSpace() { func testLayoutMatchesUpstreamBlockSpace() {
// init_block_meta's walk for 256x144 (aligned 256x160): level extents halve from // init_block_meta's walk for 256x144 (aligned 256x160): level extents halve from
// 128x80; per (comp,level,band) count32 = ceil(ceil(w/8)/4) * ceil(ceil(h/8)/4). // 128x80; per (comp,level,band) count32 = ceil(ceil(w/8)/4) * ceil(ceil(h/8)/4).
let layout = WaveletLayout(width: width, height: height) let layout = WaveletLayout(width: width, height: height, chroma444: false)
XCTAssertEqual(layout.alignedWidth, 256) XCTAssertEqual(layout.alignedWidth, 256)
XCTAssertEqual(layout.alignedHeight, 160) XCTAssertEqual(layout.alignedHeight, 160)
XCTAssertEqual(layout.levelWidth(0), 128) XCTAssertEqual(layout.levelWidth(0), 128)
@@ -85,7 +85,7 @@ final class PyroWaveParserTests: XCTestCase {
} }
func testDenseParseFillsOffsetsAndCountsBlocks() throws { func testDenseParseFillsOffsetsAndCountsBlocks() throws {
let layout = WaveletLayout(width: width, height: height) let layout = WaveletLayout(width: width, height: height, chroma444: false)
var au = sof(totalBlocks: 4) var au = sof(totalBlocks: 4)
au += packet(blockIndex: 0) au += packet(blockIndex: 0)
au += packet(blockIndex: 3) au += packet(blockIndex: 3)
@@ -273,6 +273,17 @@ final class PyroWaveGoldenTests: XCTestCase {
try assertMatchesReference(decoded, prefix: "ref-chunked") try assertMatchesReference(decoded, prefix: "ref-chunked")
} }
/// 4:4:4: the chroma components run the full pyramid like luma (no level-0 skip, no
/// early half-res emit) the layout + dispatch structure Phase 4 added
/// (design/pyrowave-444-hdr.md). The fixture comes from the 4:4:4 host encoder; the
/// reference is upstream's own 4:4:4 decode (full-res chroma planes).
func testDense444GoldenFrame() throws {
try XCTSkipIf(!MetalWaveletDecoder.supported, "no capable Metal device")
let au = try fixture("au-dense444")
let decoded = try decode(au: au, chunkAligned: false, windowSize: 0)
try assertMatchesReference(decoded, prefix: "ref-dense444")
}
/// Phase-4 partial delivery: zero a mid-AU window (a lost shard) the frame must still /// Phase-4 partial delivery: zero a mid-AU window (a lost shard) the frame must still
/// decode (blocks > half) and stay recognizably the same picture (holes reconstruct as /// decode (blocks > half) and stay recognizably the same picture (holes reconstruct as
/// localized blur, not garbage). /// localized blur, not garbage).
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+13 -11
View File
@@ -36,11 +36,14 @@ Bandwidth. At the codec's ~1.6 bits-per-pixel operating point (4:2:0, 60 fps):
| 2560×1440 @ 60 | ≈ 355 Mbps | | 2560×1440 @ 60 | ≈ 355 Mbps |
| 3840×2160 @ 60 | ≈ 800 Mbps | | 3840×2160 @ 60 | ≈ 800 Mbps |
120 Hz doubles these. Gigabit Ethernet tops out around 940 Mbps of payload, so 4K60 wants 120 Hz doubles these, 4:4:4 multiplies them by ~1.6, and an HDR (10-bit) session adds ~15 %.
2.5GbE (or a lower rate). **Do not run this over Wi-Fi** — that's what HEVC/AV1 are for. Gigabit Ethernet tops out around 940 Mbps of payload, so 4K60 wants 2.5GbE (or a lower
rate). **Do not run this over Wi-Fi** — that's what HEVC/AV1 are for.
Also: PyroWave is **8-bit SDR only**. An HDR session stays on HEVC/AV1 regardless of this PyroWave follows the same 4:4:4 and HDR settings as HEVC/AV1: a session negotiates
setting. full-chroma 4:4:4 when your client's 4:4:4 setting is on, and HDR (BT.2020 PQ, carried in
16-bit planes) when the host's display pipeline is HDR — currently the Windows host only
(the Linux host's capture path has no HDR source yet, so Linux-hosted sessions are SDR).
## Turning it on ## Turning it on
@@ -53,10 +56,10 @@ setting.
gamepad console, or launch with `PUNKTFUNK_PREFER_PYROWAVE=1`. gamepad console, or launch with `PUNKTFUNK_PREFER_PYROWAVE=1`.
- Apple (Mac, Apple TV 4K, iPad — wired networking strongly recommended): set - Apple (Mac, Apple TV 4K, iPad — wired networking strongly recommended): set
**Settings → Codec → PyroWave (wired LAN)**. The option appears only on devices whose **Settings → Codec → PyroWave (wired LAN)**. The option appears only on devices whose
GPU passes the decode probe (Apple Silicon and A13-class or newer); picking it forces GPU passes the decode probe (Apple Silicon and A13-class or newer). The decoder follows
the session SDR. the stream: 4:2:0 or 4:4:4, SDR or HDR, per what the session negotiated.
3. Leave the bitrate on Automatic: a PyroWave session pins itself to the ~1.6 bpp rate for 3. Leave the bitrate on Automatic: a PyroWave session pins itself to the ~1.6 bpp rate for
your mode (≈200 Mbps at 1080p60). An explicit bitrate is honored if you set one, but the your mode (≈200 Mbps at 1080p60; ~2.6 bpp for 4:4:4, +15 % for HDR). An explicit bitrate is honored if you set one, but the
adaptive-bitrate controller stays off either way — this codec has no useful low-rate adaptive-bitrate controller stays off either way — this codec has no useful low-rate
regime, so under sustained loss the right move is switching back to HEVC, not degrading. regime, so under sustained loss the right move is switching back to HEVC, not degrading.
The pin follows the resolution: a mid-stream resize (e.g. Match window) re-pins the rate The pin follows the resolution: a mid-stream resize (e.g. Match window) re-pins the rate
@@ -66,7 +69,6 @@ The stats overlay shows `pyrowave` as the decode path when the mode is active.
## Current limits ## Current limits
- Linux host + Linux client (including docked Deck) and Apple clients (native Metal decode - Linux and Windows hosts; Linux clients (including docked Deck) and Apple clients (native
on Mac / Apple TV 4K / iPad) today; the Windows host is tracked on the Metal decode on Mac / Apple TV 4K / iPad) today.
[roadmap](/docs/roadmap). - HDR needs a Windows host (the Linux host's capture path has no HDR source yet).
- 8-bit SDR, 4:2:0 only.