feat(pyrowave): Apple Metal 4:4:4 + HDR decode, EDR present — self-configured in-band
docker / deploy-docs (push) Successful in 11s
flatpak / build-publish (push) Successful in 6m25s
deb / build-publish (push) Successful in 9m7s
windows-host / package (push) Successful in 16m12s
deb / build-publish-host (push) Successful in 9m31s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 6m41s
ci / rust (push) Successful in 25m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 8m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 20m28s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m59s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 6m45s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 8m2s
apple / swift (push) Successful in 1m15s
release / apple (push) Successful in 9m36s
apple / screenshots (push) Successful in 6m43s
ci / web (push) Successful in 56s
ci / docs-site (push) Successful in 50s
android / android (push) Successful in 13m41s
decky / build-publish (push) Successful in 22s
arch / build-publish (push) Successful in 14m23s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 12s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 57s
ci / bench (push) Successful in 5m58s
docker / deploy-docs (push) Successful in 11s
flatpak / build-publish (push) Successful in 6m25s
deb / build-publish (push) Successful in 9m7s
windows-host / package (push) Successful in 16m12s
deb / build-publish-host (push) Successful in 9m31s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 6m41s
ci / rust (push) Successful in 25m11s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 8m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 20m28s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m59s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 6m45s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 8m2s
apple / swift (push) Successful in 1m15s
release / apple (push) Successful in 9m36s
apple / screenshots (push) Successful in 6m43s
ci / web (push) Successful in 56s
ci / docs-site (push) Successful in 50s
android / android (push) Successful in 13m41s
decky / build-publish (push) Successful in 22s
arch / build-publish (push) Successful in 14m23s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 12s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 57s
ci / bench (push) Successful in 5m58s
Phases 4+5 of design/pyrowave-444-hdr.md. The Metal decoder needs NO new ABI: every frame's sequence header carries chroma (444) and, since the Phase-3 stamps, the PQ/BT.2020 bits — so the decoder self-configures per session. Decoder: WaveletLayout grows the 4:4:4 block space (chroma runs the full pyramid like luma — no level-0 skip, no early half-res emit; the Metal kernels were already chroma-agnostic, only the dispatch structure changes); the parser accepts chroma_resolution=444, reads the PQ transfer bit, and lifts the even-dims rule for 444; the plane ring allocates full-res chroma and r16Unorm for PQ streams; CSC rows switch to depth-10 MSB-packed. Presenter: planar HDR passthrough reuses pf_frag_planar on an rgba16Float drawable (itur_2100_PQ + EDR metadata interpret the samples — same split as pf_frag/pf_frag_hdr), plus a new pf_frag_planar_tm PQ->SDR tone-map (shared pqToSdr tail refactored out of pf_frag_hdr_tv) for tvOS-without-headroom AND macOS WINDOWED sessions, whose IOSurface present path (the DCP-panic mitigation) is BGRA8-only. SessionModel stops stripping the HDR/10-bit/444 caps on the PyroWave opt-in. New golden: au-dense444 + upstream's own 4:4:4 reference planes (regenerated via the extended pyrowave_dump_golden); Metal decode matches at 64-67 dB (420 fixtures re-verify 77-88 dB). Full Apple suite 157 tests green on a real M-series GPU. Docs updated: the 8-bit-SDR-only wording is gone, the Windows host is no longer 'on the roadmap', bpp scaling documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -264,14 +264,11 @@ final class SessionModel: ObservableObject {
|
||||
// PyroWave (wired LAN) is a pure opt-in: picking it in the codec setting both
|
||||
// advertises the bit and prefers it — the host never auto-selects it, and the
|
||||
// picker only offers it when the Metal decode probe passed (simdgroup floor ≈ A13;
|
||||
// every M-series Mac and the ATV 4K gen 3 pass). The codec is 8-bit 4:2:0 SDR
|
||||
// BT.709 by contract, so the opt-in also drops the HDR/10-bit/4:4:4 caps for this
|
||||
// session — HDR sessions stay HEVC/AV1 (plan §4.7).
|
||||
// every M-series Mac and the ATV 4K gen 3 pass). The decoder self-configures from
|
||||
// the per-frame sequence header (4:2:0/4:4:4, SDR/PQ — design/pyrowave-444-hdr.md),
|
||||
// so the session keeps the user's HDR/10-bit/4:4:4 caps exactly like HEVC/AV1.
|
||||
if preferredCodec == PunktfunkConnection.codecPyroWave, MetalWaveletDecoder.supported {
|
||||
videoCodecs |= PunktfunkConnection.codecPyroWave
|
||||
videoCaps &= ~(PunktfunkConnection.videoCap10Bit
|
||||
| PunktfunkConnection.videoCapHDR
|
||||
| PunktfunkConnection.videoCap444)
|
||||
}
|
||||
let result = Result { try PunktfunkConnection(
|
||||
host: host.address, port: host.port,
|
||||
|
||||
@@ -196,13 +196,11 @@ fragment float4 pf_frag_hdr(VOut in [[stage_in]],
|
||||
// display-referred SDR. (When the display IS in an HDR mode — requested per session via
|
||||
// AVDisplayManager, see StreamViewIOS — tvOS presents pf_frag_hdr's PQ passthrough instead:
|
||||
// in a genuine HDR10 output, PQ passthrough is the correct emission and the TV tone-maps.)
|
||||
fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]],
|
||||
texture2d<float> lumaTex [[texture(0)]],
|
||||
texture2d<float> chromaTex [[texture(1)]],
|
||||
constant CscUniform& csc [[buffer(0)]]) {
|
||||
// Y′CbCr → full-range PQ R′G′B′ via the per-frame rows (as pf_frag_hdr).
|
||||
float3 pq = sampleRgb(lumaTex, chromaTex, in.uv, csc);
|
||||
// ST 2084 EOTF: PQ code value → linear light, 1.0 = 10,000 nits.
|
||||
// The shared PQ→display-referred-SDR tail (see pf_frag_hdr_tv's rationale above): ST 2084
|
||||
// EOTF → 203-nit-anchored scene light → BT.2020→709 primaries → extended-Reinhard rolloff →
|
||||
// BT.709 OETF. Used by the tvOS biplanar tone-map and the planar (PyroWave) tone-map — the
|
||||
// latter also on macOS windowed sessions, whose IOSurface present path is BGRA8-only.
|
||||
static inline float3 pqToSdr(float3 pq) {
|
||||
const float m1 = 2610.0/16384.0;
|
||||
const float m2 = 78.84375;
|
||||
const float c1 = 3424.0/4096.0;
|
||||
@@ -210,20 +208,49 @@ fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]],
|
||||
const float c3 = 18.6875;
|
||||
float3 p = pow(pq, 1.0/m2);
|
||||
float3 lin = pow(max(p - c1, 0.0) / (c2 - c3 * p), 1.0/m1);
|
||||
// Scene-referred with diffuse white at 1.0 (the same 203-nit anchor the EDR path uses).
|
||||
float3 t = lin * (10000.0/203.0);
|
||||
// BT.2020 → BT.709 primaries while still linear; negatives are out-of-gamut, floor them.
|
||||
float3 t709 = float3(
|
||||
dot(t, float3( 1.6605, -0.5876, -0.0728)),
|
||||
dot(t, float3(-0.1246, 1.1329, -0.0083)),
|
||||
dot(t, float3(-0.0182, -0.1006, 1.1187)));
|
||||
t709 = max(t709, 0.0);
|
||||
// Extended Reinhard: 1.0 stays put, the 1000-nit knee lands at display white, above rolls off.
|
||||
const float w = 1000.0/203.0;
|
||||
float3 mapped = saturate(t709 * (1.0 + t709 / (w * w)) / (1.0 + t709));
|
||||
// BT.709 OETF — the same encoding the SDR stream arrives in, so both paths present alike.
|
||||
float3 e = select(1.099 * pow(mapped, 0.45) - 0.099, 4.5 * mapped, mapped < 0.018);
|
||||
return float4(e, 1.0);
|
||||
return e;
|
||||
}
|
||||
|
||||
fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]],
|
||||
texture2d<float> lumaTex [[texture(0)]],
|
||||
texture2d<float> chromaTex [[texture(1)]],
|
||||
constant CscUniform& csc [[buffer(0)]]) {
|
||||
// Y′CbCr → full-range PQ R′G′B′ via the per-frame rows (as pf_frag_hdr), then the tail.
|
||||
return float4(pqToSdr(sampleRgb(lumaTex, chromaTex, in.uv, csc)), 1.0);
|
||||
}
|
||||
|
||||
// PyroWave planar HDR tone-map: three separate R16 planes (P010-style studio codes; the rows
|
||||
// fold in depth-10 MSB packing) → PQ R′G′B′ → the shared SDR tail. Used when a PQ pyrowave
|
||||
// stream must land on an 8-bit surface: tvOS without HDR headroom, and macOS WINDOWED sessions
|
||||
// (the IOSurface present path — the DCP-panic mitigation — is BGRA8). The passthrough planar
|
||||
// HDR pipeline reuses pf_frag_planar itself on an rgba16Float drawable (identical math — the
|
||||
// layer's itur_2100_PQ colour space + EDR metadata do the interpretation).
|
||||
fragment float4 pf_frag_planar_tm(VOut in [[stage_in]],
|
||||
texture2d<float> lumaTex [[texture(0)]],
|
||||
texture2d<float> cbTex [[texture(1)]],
|
||||
texture2d<float> crTex [[texture(2)]],
|
||||
constant CscUniform& csc [[buffer(0)]]) {
|
||||
constexpr sampler s(filter::linear, address::clamp_to_edge);
|
||||
#ifdef PF_BILINEAR_LUMA
|
||||
float lumaY = lumaTex.sample(s, in.uv).r;
|
||||
#else
|
||||
float lumaY = catmullRomLuma(lumaTex, s, in.uv);
|
||||
#endif
|
||||
float2 cuv = chromaUV(lumaTex, cbTex, in.uv);
|
||||
float3 yuv = float3(lumaY, cbTex.sample(s, cuv).r, crTex.sample(s, cuv).r);
|
||||
float3 pq = saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
|
||||
dot(csc.r1.xyz, yuv) + csc.r1.w,
|
||||
dot(csc.r2.xyz, yuv) + csc.r2.w));
|
||||
return float4(pqToSdr(pq), 1.0);
|
||||
}
|
||||
"""
|
||||
|
||||
@@ -287,6 +314,11 @@ public final class MetalVideoPresenter {
|
||||
private let pipelineHDRToneMap: MTLRenderPipelineState?
|
||||
/// PyroWave's 3-plane SDR path (pf_frag_planar → bgra8) — see `renderPlanar`.
|
||||
private let pipelinePlanar: MTLRenderPipelineState
|
||||
/// PyroWave planar HDR passthrough (pf_frag_planar → rgba16Float; the layer's PQ colour
|
||||
/// space + EDR interpret the samples) and the planar PQ→SDR tone-map (pf_frag_planar_tm →
|
||||
/// bgra8; tvOS without headroom + macOS windowed IOSurface presents).
|
||||
private let pipelinePlanarHDR: MTLRenderPipelineState
|
||||
private let pipelinePlanarToneMap: MTLRenderPipelineState
|
||||
private var textureCache: CVMetalTextureCache?
|
||||
|
||||
/// The PyroWave Metal decoder records on the presenter's device + queue: one device means
|
||||
@@ -337,6 +369,8 @@ public final class MetalVideoPresenter {
|
||||
let pipelineHDR: MTLRenderPipelineState
|
||||
let pipelineHDRToneMap: MTLRenderPipelineState?
|
||||
let pipelinePlanar: MTLRenderPipelineState
|
||||
let pipelinePlanarHDR: MTLRenderPipelineState
|
||||
let pipelinePlanarToneMap: MTLRenderPipelineState
|
||||
do {
|
||||
// DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF
|
||||
// (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is
|
||||
@@ -374,8 +408,18 @@ public final class MetalVideoPresenter {
|
||||
let planar = MTLRenderPipelineDescriptor()
|
||||
planar.vertexFunction = vtx
|
||||
planar.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
|
||||
planar.colorAttachments[0].pixelFormat = .bgra8Unorm // PyroWave is 8-bit SDR
|
||||
planar.colorAttachments[0].pixelFormat = .bgra8Unorm
|
||||
pipelinePlanar = try device.makeRenderPipelineState(descriptor: planar)
|
||||
let planarHdr = MTLRenderPipelineDescriptor()
|
||||
planarHdr.vertexFunction = vtx
|
||||
planarHdr.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
|
||||
planarHdr.colorAttachments[0].pixelFormat = .rgba16Float // PQ passthrough
|
||||
pipelinePlanarHDR = try device.makeRenderPipelineState(descriptor: planarHdr)
|
||||
let planarTm = MTLRenderPipelineDescriptor()
|
||||
planarTm.vertexFunction = vtx
|
||||
planarTm.fragmentFunction = library.makeFunction(name: "pf_frag_planar_tm")
|
||||
planarTm.colorAttachments[0].pixelFormat = .bgra8Unorm
|
||||
pipelinePlanarToneMap = try device.makeRenderPipelineState(descriptor: planarTm)
|
||||
} catch {
|
||||
return nil
|
||||
}
|
||||
@@ -416,6 +460,7 @@ public final class MetalVideoPresenter {
|
||||
return MetalVideoPresenter(
|
||||
device: device, queue: queue, pipelineSDR: pipelineSDR, pipelineHDR: pipelineHDR,
|
||||
pipelineHDRToneMap: pipelineHDRToneMap, pipelinePlanar: pipelinePlanar,
|
||||
pipelinePlanarHDR: pipelinePlanarHDR, pipelinePlanarToneMap: pipelinePlanarToneMap,
|
||||
textureCache: textureCache, layer: layer)
|
||||
}
|
||||
|
||||
@@ -423,6 +468,8 @@ public final class MetalVideoPresenter {
|
||||
device: MTLDevice, queue: MTLCommandQueue, pipelineSDR: MTLRenderPipelineState,
|
||||
pipelineHDR: MTLRenderPipelineState, pipelineHDRToneMap: MTLRenderPipelineState?,
|
||||
pipelinePlanar: MTLRenderPipelineState,
|
||||
pipelinePlanarHDR: MTLRenderPipelineState,
|
||||
pipelinePlanarToneMap: MTLRenderPipelineState,
|
||||
textureCache: CVMetalTextureCache, layer: CAMetalLayer
|
||||
) {
|
||||
self.device = device
|
||||
@@ -431,6 +478,8 @@ public final class MetalVideoPresenter {
|
||||
self.pipelineHDR = pipelineHDR
|
||||
self.pipelineHDRToneMap = pipelineHDRToneMap
|
||||
self.pipelinePlanar = pipelinePlanar
|
||||
self.pipelinePlanarHDR = pipelinePlanarHDR
|
||||
self.pipelinePlanarToneMap = pipelinePlanarToneMap
|
||||
self.textureCache = textureCache
|
||||
self.layer = layer
|
||||
}
|
||||
@@ -652,7 +701,13 @@ public final class MetalVideoPresenter {
|
||||
let surfaceMode = surfacePresentsStaged
|
||||
#endif
|
||||
stagingLock.unlock()
|
||||
configure(hdr: false)
|
||||
// A PQ (HDR) pyrowave stream drives the same layer/EDR machinery as the biplanar path;
|
||||
// macOS WINDOWED sessions stay on the SDR layer (the IOSurface path tone-maps in-shader).
|
||||
#if os(macOS)
|
||||
configure(hdr: planes.pq && !surfaceMode)
|
||||
#else
|
||||
configure(hdr: planes.pq)
|
||||
#endif
|
||||
var csc = planes.csc
|
||||
#if os(macOS)
|
||||
if surfaceMode != surfacePresentsActive {
|
||||
@@ -672,9 +727,21 @@ public final class MetalVideoPresenter {
|
||||
planes, targetFromLayout: targetFromLayout, csc: &csc, onPresented: onPresented)
|
||||
}
|
||||
#endif
|
||||
// PQ passthrough needs the HDR drawable; a PQ frame while the drawable is (still)
|
||||
// 8-bit — tvOS without display headroom, or a not-yet-flipped layer — tone-maps
|
||||
// in-shader instead (the pipeline must match the drawable's pixel format).
|
||||
#if os(tvOS)
|
||||
let planarPassthrough = hdrActive && hdrPassthroughActive
|
||||
#else
|
||||
let planarPassthrough = hdrActive
|
||||
#endif
|
||||
let planarPipeline: MTLRenderPipelineState =
|
||||
planes.pq
|
||||
? (planarPassthrough ? pipelinePlanarHDR : pipelinePlanarToneMap)
|
||||
: pipelinePlanar
|
||||
return encodePresent(
|
||||
decodedSize: CGSize(width: planes.width, height: planes.height),
|
||||
targetFromLayout: targetFromLayout, pipeline: pipelinePlanar,
|
||||
targetFromLayout: targetFromLayout, pipeline: planarPipeline,
|
||||
presentAtMediaTime: presentAtMediaTime, onPresented: onPresented,
|
||||
// The ring textures stay valid by ring depth; retaining them here also pins the
|
||||
// slot's set until the sample completes (mirrors the biplanar keep-alive).
|
||||
@@ -716,7 +783,7 @@ public final class MetalVideoPresenter {
|
||||
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else {
|
||||
return false
|
||||
}
|
||||
encoder.setRenderPipelineState(pipelinePlanar)
|
||||
encoder.setRenderPipelineState(planes.pq ? pipelinePlanarToneMap : pipelinePlanar)
|
||||
encoder.setFragmentTexture(planes.y, index: 0)
|
||||
encoder.setFragmentTexture(planes.cb, index: 1)
|
||||
encoder.setFragmentTexture(planes.cr, index: 2)
|
||||
|
||||
@@ -47,6 +47,9 @@ struct WaveletLayout {
|
||||
|
||||
let width: Int
|
||||
let height: Int
|
||||
/// Full-res chroma (4:4:4): chroma components get the full band set including level 0,
|
||||
/// exactly like luma — upstream `init_block_meta` with `Chroma444`.
|
||||
let chroma444: Bool
|
||||
let alignedWidth: Int
|
||||
let alignedHeight: Int
|
||||
/// blockMeta[component][level][band] = (blockOffset32x32, blockStride32x32); -1 offset =
|
||||
@@ -59,9 +62,10 @@ struct WaveletLayout {
|
||||
func levelWidth(_ level: Int) -> Int { (alignedWidth / 2) >> level }
|
||||
func levelHeight(_ level: Int) -> Int { (alignedHeight / 2) >> level }
|
||||
|
||||
init(width: Int, height: Int) {
|
||||
init(width: Int, height: Int, chroma444: Bool) {
|
||||
self.width = width
|
||||
self.height = height
|
||||
self.chroma444 = chroma444
|
||||
let align = { (v: Int) in
|
||||
max((v + Self.alignment - 1) & ~(Self.alignment - 1), Self.minimumImageSize)
|
||||
}
|
||||
@@ -78,7 +82,7 @@ struct WaveletLayout {
|
||||
let ah = alignedHeight
|
||||
for level in stride(from: Self.decompositionLevels - 1, through: 0, by: -1) {
|
||||
for component in 0..<3 {
|
||||
if level == 0 && component != 0 { continue } // 4:2:0: no top-level chroma
|
||||
if level == 0 && component != 0 && !chroma444 { continue } // 4:2:0: no top-level chroma
|
||||
for band in (level == Self.decompositionLevels - 1 ? 0 : 1)..<4 {
|
||||
let levelW = (aw / 2) >> level
|
||||
let levelH = (ah / 2) >> level
|
||||
@@ -108,6 +112,9 @@ struct ParsedWaveletFrame {
|
||||
var decodedBlocks: Int
|
||||
/// VUI bits from the sequence header (BitstreamSequenceHeader).
|
||||
var bt2020: Bool
|
||||
/// PQ transfer ⇒ HDR session: 16-bit studio-code planes + EDR present (the host stamps
|
||||
/// this bit iff the session negotiated 10-bit — the depth is coupled to the transfer).
|
||||
var pq: Bool
|
||||
var fullRange: Bool
|
||||
|
||||
/// The frame's Y′CbCr→RGB signal for the presenter's planar CSC. PyroWave today is always
|
||||
@@ -203,6 +210,7 @@ enum WaveletBitstream {
|
||||
var totalBlocks = 0
|
||||
var decodedBlocks = 0
|
||||
var bt2020 = false
|
||||
var pq = false
|
||||
var fullRange = false
|
||||
var sawSOF = false
|
||||
|
||||
@@ -220,22 +228,28 @@ enum WaveletBitstream {
|
||||
// siting[31].
|
||||
let code = (word1 >> 24) & 0x3
|
||||
guard code == 0 else { return false } // only START_OF_FRAME is defined
|
||||
let chromaRes = (word1 >> 26) & 1
|
||||
guard chromaRes == 0 else { return false } // host contract: 4:2:0
|
||||
let chroma444 = (word1 >> 26) & 1 != 0
|
||||
let w = Int(word0 & 0x3fff) + 1
|
||||
let h = Int((word0 >> 14) & 0x3fff) + 1
|
||||
guard w >= 2, h >= 2, w % 2 == 0, h % 2 == 0 else { return false }
|
||||
guard w >= 2, h >= 2, chroma444 || (w % 2 == 0 && h % 2 == 0) else {
|
||||
return false
|
||||
}
|
||||
if sawSOF {
|
||||
// One frame, one geometry — a second SOF must agree.
|
||||
guard layout?.width == w, layout?.height == h else { return false }
|
||||
guard layout?.width == w, layout?.height == h,
|
||||
layout?.chroma444 == chroma444
|
||||
else { return false }
|
||||
} else {
|
||||
sawSOF = true
|
||||
let l = WaveletLayout(width: w, height: h)
|
||||
let l = WaveletLayout(width: w, height: h, chroma444: chroma444)
|
||||
layout = l
|
||||
offsets = [UInt32](repeating: .max, count: l.blockCount32)
|
||||
payload.reserveCapacity(64 * 1024 / 4)
|
||||
totalBlocks = Int(word1 & 0xff_ffff)
|
||||
bt2020 = (word1 >> 29) & 1 != 0
|
||||
// transfer_function bit: PQ ⇒ an HDR session (16-bit studio-code
|
||||
// planes by the negotiated coupling — design/pyrowave-444-hdr.md).
|
||||
pq = (word1 >> 28) & 1 != 0
|
||||
fullRange = (word1 >> 30) & 1 == 0 // YCBCR_RANGE_FULL = 0
|
||||
}
|
||||
pos += 8
|
||||
@@ -280,7 +294,7 @@ enum WaveletBitstream {
|
||||
return ParsedWaveletFrame(
|
||||
layout: layout, offsets: offsets, payload: payload,
|
||||
totalBlocks: totalBlocks, decodedBlocks: decodedBlocks,
|
||||
bt2020: bt2020, fullRange: fullRange)
|
||||
bt2020: bt2020, pq: pq, fullRange: fullRange)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -293,6 +307,8 @@ public struct WaveletPlanes: @unchecked Sendable {
|
||||
public let cb: MTLTexture
|
||||
public let cr: MTLTexture
|
||||
public let csc: CscUniform
|
||||
/// PQ (HDR) stream: the presenter picks the HDR/tone-map planar pipeline + EDR config.
|
||||
public let pq: Bool
|
||||
public var width: Int { y.width }
|
||||
public var height: Int { y.height }
|
||||
}
|
||||
@@ -351,6 +367,8 @@ public final class MetalWaveletDecoder {
|
||||
|
||||
private var slots: [Slot] = []
|
||||
private var nextSlot = 0
|
||||
/// The ring's plane format facts (from the last SOF): PQ ⇒ 16-bit UNORM planes.
|
||||
private var hdr16 = false
|
||||
|
||||
/// The current geometry (from the last SOF that built the resources) — the pump reports
|
||||
/// decoded-size changes to the resize overlay from this. PUMP THREAD.
|
||||
@@ -409,8 +427,9 @@ public final class MetalWaveletDecoder {
|
||||
au: au, chunkAligned: chunkAligned, windowSize: windowSize)
|
||||
else { return false }
|
||||
|
||||
if layout?.width != frame.layout.width || layout?.height != frame.layout.height {
|
||||
guard rebuild(layout: frame.layout) else { return false }
|
||||
if layout?.width != frame.layout.width || layout?.height != frame.layout.height
|
||||
|| layout?.chroma444 != frame.layout.chroma444 || hdr16 != frame.pq {
|
||||
guard rebuild(layout: frame.layout, hdr16: frame.pq) else { return false }
|
||||
}
|
||||
guard let layout, !slots.isEmpty else { return false }
|
||||
|
||||
@@ -450,7 +469,7 @@ public final class MetalWaveletDecoder {
|
||||
dequant.setBuffer(slot.payload, offset: 0, index: 1)
|
||||
for level in 0..<WaveletLayout.decompositionLevels {
|
||||
for component in 0..<3 {
|
||||
if level == 0 && component != 0 { continue } // 4:2:0
|
||||
if level == 0 && component != 0 && !layout.chroma444 { continue } // 4:2:0
|
||||
for band in (level == WaveletLayout.decompositionLevels - 1 ? 0 : 1)..<4 {
|
||||
let meta = layout.blockMeta[component][level][band]
|
||||
let w = layout.levelWidth(level)
|
||||
@@ -489,15 +508,20 @@ public final class MetalWaveletDecoder {
|
||||
let grid = MTLSize(width: (rx + 15) / 16, height: (ry + 15) / 16, depth: 1)
|
||||
let group = MTLSize(width: 64, height: 1, depth: 1)
|
||||
if inputLevel == 0 {
|
||||
// 4:2:0: the final full-res pass is luma only (chroma finished at level 1).
|
||||
// Final full-res pass: luma only in 4:2:0 (chroma finished at level 1); all
|
||||
// three components in 4:4:4 (chroma runs the full pyramid like luma).
|
||||
idwt.setComputePipelineState(idwtShiftPipeline)
|
||||
idwt.setTexture(coefficients[0][0], index: 0)
|
||||
idwt.setTexture(slot.y, index: 1)
|
||||
let components = layout.chroma444 ? 3 : 1
|
||||
for component in 0..<components {
|
||||
idwt.setTexture(coefficients[component][0], index: 0)
|
||||
let out = component == 0 ? slot.y : (component == 1 ? slot.cb : slot.cr)
|
||||
idwt.setTexture(out, index: 1)
|
||||
idwt.dispatchThreadgroups(grid, threadsPerThreadgroup: group)
|
||||
}
|
||||
} else {
|
||||
for component in 0..<3 {
|
||||
idwt.setTexture(coefficients[component][inputLevel], index: 0)
|
||||
if component != 0 && inputLevel == 1 {
|
||||
if component != 0 && inputLevel == 1 && !layout.chroma444 {
|
||||
// 4:2:0 chroma emits its final half-res plane one level early.
|
||||
idwt.setComputePipelineState(idwtShiftPipeline)
|
||||
idwt.setTexture(component == 1 ? slot.cb : slot.cr, index: 1)
|
||||
@@ -513,7 +537,9 @@ public final class MetalWaveletDecoder {
|
||||
|
||||
let planes = WaveletPlanes(
|
||||
y: slot.y, cb: slot.cb, cr: slot.cr,
|
||||
csc: CscRows.rows(frame.cscSignal, depth: 8, msbPacked: false))
|
||||
csc: CscRows.rows(
|
||||
frame.cscSignal, depth: frame.pq ? 10 : 8, msbPacked: frame.pq),
|
||||
pq: frame.pq)
|
||||
cmd.addCompletedHandler { buffer in
|
||||
completion(buffer.error == nil ? planes : nil)
|
||||
}
|
||||
@@ -524,9 +550,9 @@ public final class MetalWaveletDecoder {
|
||||
|
||||
/// (Re)allocate every size-dependent resource for `layout`'s geometry. Also the mid-stream
|
||||
/// resize path: a Reconfigure shows up here as new SOF dims.
|
||||
private func rebuild(layout newLayout: WaveletLayout) -> Bool {
|
||||
private func rebuild(layout newLayout: WaveletLayout, hdr16 newHdr16: Bool) -> Bool {
|
||||
waveletLog.info(
|
||||
"pyrowave: building decoder \(newLayout.width)x\(newLayout.height) (aligned \(newLayout.alignedWidth)x\(newLayout.alignedHeight), \(newLayout.blockCount32) blocks)")
|
||||
"pyrowave: building decoder \(newLayout.width)x\(newLayout.height) (aligned \(newLayout.alignedWidth)x\(newLayout.alignedHeight), \(newLayout.blockCount32) blocks, \(newLayout.chroma444 ? "4:4:4" : "4:2:0", privacy: .public)\(newHdr16 ? " HDR16" : "", privacy: .public))")
|
||||
var coeff: [[MTLTexture]] = []
|
||||
var lls: [[MTLTexture]] = []
|
||||
for component in 0..<3 {
|
||||
@@ -560,19 +586,22 @@ public final class MetalWaveletDecoder {
|
||||
|
||||
var newSlots: [Slot] = []
|
||||
for i in 0..<Self.ringDepth {
|
||||
let planeFormat: MTLPixelFormat = newHdr16 ? .r16Unorm : .r8Unorm
|
||||
let plane = { (w: Int, h: Int, name: String) -> MTLTexture? in
|
||||
let desc = MTLTextureDescriptor.texture2DDescriptor(
|
||||
pixelFormat: .r8Unorm, width: w, height: h, mipmapped: false)
|
||||
pixelFormat: planeFormat, width: w, height: h, mipmapped: false)
|
||||
desc.usage = [.shaderRead, .shaderWrite]
|
||||
desc.storageMode = .private
|
||||
let t = self.device.makeTexture(descriptor: desc)
|
||||
t?.label = name
|
||||
return t
|
||||
}
|
||||
let cw = newLayout.chroma444 ? newLayout.width : newLayout.width / 2
|
||||
let ch = newLayout.chroma444 ? newLayout.height : newLayout.height / 2
|
||||
guard
|
||||
let y = plane(newLayout.width, newLayout.height, "pyrowave Y[\(i)]"),
|
||||
let cb = plane(newLayout.width / 2, newLayout.height / 2, "pyrowave Cb[\(i)]"),
|
||||
let cr = plane(newLayout.width / 2, newLayout.height / 2, "pyrowave Cr[\(i)]"),
|
||||
let cb = plane(cw, ch, "pyrowave Cb[\(i)]"),
|
||||
let cr = plane(cw, ch, "pyrowave Cr[\(i)]"),
|
||||
let offsets = device.makeBuffer(
|
||||
length: max(newLayout.blockCount32 * 4, 4), options: .storageModeShared),
|
||||
let payload = device.makeBuffer(length: 64 * 1024, options: .storageModeShared)
|
||||
@@ -585,6 +614,7 @@ public final class MetalWaveletDecoder {
|
||||
slots = newSlots
|
||||
nextSlot = 0
|
||||
layout = newLayout
|
||||
hdr16 = newHdr16
|
||||
return true
|
||||
}
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ final class PyroWaveParserTests: XCTestCase {
|
||||
func testLayoutMatchesUpstreamBlockSpace() {
|
||||
// init_block_meta's walk for 256x144 (aligned 256x160): level extents halve from
|
||||
// 128x80; per (comp,level,band) count32 = ceil(ceil(w/8)/4) * ceil(ceil(h/8)/4).
|
||||
let layout = WaveletLayout(width: width, height: height)
|
||||
let layout = WaveletLayout(width: width, height: height, chroma444: false)
|
||||
XCTAssertEqual(layout.alignedWidth, 256)
|
||||
XCTAssertEqual(layout.alignedHeight, 160)
|
||||
XCTAssertEqual(layout.levelWidth(0), 128)
|
||||
@@ -85,7 +85,7 @@ final class PyroWaveParserTests: XCTestCase {
|
||||
}
|
||||
|
||||
func testDenseParseFillsOffsetsAndCountsBlocks() throws {
|
||||
let layout = WaveletLayout(width: width, height: height)
|
||||
let layout = WaveletLayout(width: width, height: height, chroma444: false)
|
||||
var au = sof(totalBlocks: 4)
|
||||
au += packet(blockIndex: 0)
|
||||
au += packet(blockIndex: 3)
|
||||
@@ -273,6 +273,17 @@ final class PyroWaveGoldenTests: XCTestCase {
|
||||
try assertMatchesReference(decoded, prefix: "ref-chunked")
|
||||
}
|
||||
|
||||
/// 4:4:4: the chroma components run the full pyramid like luma (no level-0 skip, no
|
||||
/// early half-res emit) — the layout + dispatch structure Phase 4 added
|
||||
/// (design/pyrowave-444-hdr.md). The fixture comes from the 4:4:4 host encoder; the
|
||||
/// reference is upstream's own 4:4:4 decode (full-res chroma planes).
|
||||
func testDense444GoldenFrame() throws {
|
||||
try XCTSkipIf(!MetalWaveletDecoder.supported, "no capable Metal device")
|
||||
let au = try fixture("au-dense444")
|
||||
let decoded = try decode(au: au, chunkAligned: false, windowSize: 0)
|
||||
try assertMatchesReference(decoded, prefix: "ref-dense444")
|
||||
}
|
||||
|
||||
/// Phase-4 partial delivery: zero a mid-AU window (a lost shard) — the frame must still
|
||||
/// decode (blocks > half) and stay recognizably the same picture (holes reconstruct as
|
||||
/// localized blur, not garbage).
|
||||
|
||||
Binary file not shown.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -36,11 +36,14 @@ Bandwidth. At the codec's ~1.6 bits-per-pixel operating point (4:2:0, 60 fps):
|
||||
| 2560×1440 @ 60 | ≈ 355 Mbps |
|
||||
| 3840×2160 @ 60 | ≈ 800 Mbps |
|
||||
|
||||
120 Hz doubles these. Gigabit Ethernet tops out around 940 Mbps of payload, so 4K60 wants
|
||||
2.5GbE (or a lower rate). **Do not run this over Wi-Fi** — that's what HEVC/AV1 are for.
|
||||
120 Hz doubles these, 4:4:4 multiplies them by ~1.6, and an HDR (10-bit) session adds ~15 %.
|
||||
Gigabit Ethernet tops out around 940 Mbps of payload, so 4K60 wants 2.5GbE (or a lower
|
||||
rate). **Do not run this over Wi-Fi** — that's what HEVC/AV1 are for.
|
||||
|
||||
Also: PyroWave is **8-bit SDR only**. An HDR session stays on HEVC/AV1 regardless of this
|
||||
setting.
|
||||
PyroWave follows the same 4:4:4 and HDR settings as HEVC/AV1: a session negotiates
|
||||
full-chroma 4:4:4 when your client's 4:4:4 setting is on, and HDR (BT.2020 PQ, carried in
|
||||
16-bit planes) when the host's display pipeline is HDR — currently the Windows host only
|
||||
(the Linux host's capture path has no HDR source yet, so Linux-hosted sessions are SDR).
|
||||
|
||||
## Turning it on
|
||||
|
||||
@@ -53,10 +56,10 @@ setting.
|
||||
gamepad console, or launch with `PUNKTFUNK_PREFER_PYROWAVE=1`.
|
||||
- Apple (Mac, Apple TV 4K, iPad — wired networking strongly recommended): set
|
||||
**Settings → Codec → PyroWave (wired LAN)**. The option appears only on devices whose
|
||||
GPU passes the decode probe (Apple Silicon and A13-class or newer); picking it forces
|
||||
the session SDR.
|
||||
GPU passes the decode probe (Apple Silicon and A13-class or newer). The decoder follows
|
||||
the stream: 4:2:0 or 4:4:4, SDR or HDR, per what the session negotiated.
|
||||
3. Leave the bitrate on Automatic: a PyroWave session pins itself to the ~1.6 bpp rate for
|
||||
your mode (≈200 Mbps at 1080p60). An explicit bitrate is honored if you set one, but the
|
||||
your mode (≈200 Mbps at 1080p60; ~2.6 bpp for 4:4:4, +15 % for HDR). An explicit bitrate is honored if you set one, but the
|
||||
adaptive-bitrate controller stays off either way — this codec has no useful low-rate
|
||||
regime, so under sustained loss the right move is switching back to HEVC, not degrading.
|
||||
The pin follows the resolution: a mid-stream resize (e.g. Match window) re-pins the rate
|
||||
@@ -66,7 +69,6 @@ The stats overlay shows `pyrowave` as the decode path when the mode is active.
|
||||
|
||||
## Current limits
|
||||
|
||||
- Linux host + Linux client (including docked Deck) and Apple clients (native Metal decode
|
||||
on Mac / Apple TV 4K / iPad) today; the Windows host is tracked on the
|
||||
[roadmap](/docs/roadmap).
|
||||
- 8-bit SDR, 4:2:0 only.
|
||||
- Linux and Windows hosts; Linux clients (including docked Deck) and Apple clients (native
|
||||
Metal decode on Mac / Apple TV 4K / iPad) today.
|
||||
- HDR needs a Windows host (the Linux host's capture path has no HDR source yet).
|
||||
|
||||
Reference in New Issue
Block a user