feat(apple): PyroWave Phase 5 — native Metal decode on Mac / Apple TV / iPad (§4.7)
ci / web (push) Successful in 1m3s
ci / docs-site (push) Successful in 58s
decky / build-publish (push) Successful in 16s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 43s
ci / bench (push) Successful in 5m21s
arch / build-publish (push) Successful in 10m58s
docker / deploy-docs (push) Successful in 28s
android / android (push) Successful in 13m23s
deb / build-publish (push) Successful in 12m58s
ci / rust (push) Successful in 18m5s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m31s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m56s
flatpak / build-publish (push) Successful in 5m53s
windows-host / package (push) Successful in 14m37s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m9s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m15s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m12s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 6m6s
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
release / apple (push) Has been cancelled

The Apple client now decodes PyroWave natively on the presenter's own MTLDevice —
no MoltenVK, no upstream C++ in the app. Completes and wires up the decoder whose
early working-tree snapshot rode along in 9127c346:

- MetalWaveletShaders.swift: wavelet_dequant + idwt hand-ported from the vendored
  GLSL (STORAGE_MODE 0 only; subgroup scans → 32-wide simdgroups; DCShift spec
  constant → function constant; precision-1 split: fp16 levels 0-1 / fp32 2-4).
- MetalWaveletDecoder.swift: Swift reimplementation of push_packet/decode_packet
  incl. the Phase-4 chunk-aligned window walk (FRAG chains, zeroed missing shards,
  the >half-blocks partial rule), init_block_meta's block-index space, and the
  42-dequant + 13-idwt dispatch structure with encoder-boundary barriers. SOF-dims
  changes rebuild the size-dependent resources, which is also the mid-stream
  resize path. Ring of 4 output plane sets on the presenter's queue.
- Presenter: pf_frag_planar (3xR8, the planar_csc.frag twin) + renderPlanar with
  a shared present tail; ReadyFrame carries an image enum (.video | .planar).
- Stage2Pipeline: a dedicated PyroWave pump — no VideoToolbox machinery, no
  keyframe/re-anchor recovery (all-intra; partials render as localized blur by
  design), newest-frame-index staleness guard for late partials.
- Opt-in: "PyroWave (wired LAN)" codec entry (probe-gated, ≈A13 floor via a real
  kernel-compile probe), selecting it advertises + prefers the codec and forces
  the session SDR (HDR/10-bit/4:4:4 caps dropped, plan contract).
- Core ABI: punktfunk_connection_shard_payload() — the Welcome's negotiated shard
  payload, needed by native decoders to walk chunk-aligned AUs.
- Validation: golden fixtures generated by the host encoder + upstream's own
  decoder (pyrowave_dump_golden, RTX 5070 Ti); the Metal decode PSNR-matches at
  77-88 dB across all planes for dense AND chunk-aligned AUs, and a hole-punched
  partial still decodes. Parser unit tests cover the window walk, FRAG chains,
  broken chains, the half-blocks gate, and the block-index layout.

Tests: apple 134 green (mac; iOS/tvOS build), host 312 w/ pyrowave on .21,
core 148 w/ quic; clippy/fmt clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-15 12:16:43 +02:00
parent a70811043e
commit 739a5f76bf
28 changed files with 826 additions and 81 deletions
@@ -149,6 +149,28 @@ fragment float4 pf_frag(VOut in [[stage_in]],
return float4(sampleRgb(lumaTex, chromaTex, in.uv, csc), 1.0);
}
// PyroWave planar SDR: three separate R8 planes (Y full-res, Cb/Cr half-res 4:2:0) from the
// Metal wavelet decoder — the Metal twin of pf-presenter's planar_csc.frag. Same bicubic luma
// and left-cosited chroma correction as the biplanar path (chromaUV self-disables at 4:4:4).
fragment float4 pf_frag_planar(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> cbTex [[texture(1)]],
texture2d<float> crTex [[texture(2)]],
constant CscUniform& csc [[buffer(0)]]) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
#ifdef PF_BILINEAR_LUMA
float lumaY = lumaTex.sample(s, in.uv).r;
#else
float lumaY = catmullRomLuma(lumaTex, s, in.uv);
#endif
float2 cuv = chromaUV(lumaTex, cbTex, in.uv);
float3 yuv = float3(lumaY, cbTex.sample(s, cuv).r, crTex.sample(s, cuv).r);
float3 rgb = saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
dot(csc.r1.xyz, yuv) + csc.r1.w,
dot(csc.r2.xyz, yuv) + csc.r2.w));
return float4(rgb, 1.0);
}
// HDR: 10-bit P010 / 4:4:4 (BT.2020, PQ-encoded YCbCr) → full-range PQ RGB, output as-is —
// the CAMetalLayer's itur_2100_PQ colour space + edrMetadata tell the compositor the samples are
// PQ, so it does the PQ→display tone-map. No EOTF here. The rows fold in the exact 10-bit
@@ -215,8 +237,16 @@ public final class MetalVideoPresenter {
/// tvOS only: the in-shader PQSDR tone-map fallback (pf_frag_hdr_tv bgra8), used whenever
/// the display is composited without HDR headroom see `setDisplayHeadroom`. nil elsewhere.
private let pipelineHDRToneMap: MTLRenderPipelineState?
/// PyroWave's 3-plane SDR path (pf_frag_planar bgra8) see `renderPlanar`.
private let pipelinePlanar: MTLRenderPipelineState
private var textureCache: CVMetalTextureCache?
/// The PyroWave Metal decoder records on the presenter's device + queue: one device means
/// decode, CSC and present share textures with zero interop, and one queue means Metal's
/// hazard tracking orders a ring-slot rewrite after the render still sampling it.
var metalDevice: MTLDevice { device }
var metalQueue: MTLCommandQueue { queue }
/// Current layer configuration switched in `configure(hdr:)` when a frame's HDR-ness differs.
/// Render-thread confined once the pipeline runs (Stage2Pipeline.start's one pre-thread
/// `configure` call is ordered before the thread starts, so it doesn't race).
@@ -258,6 +288,7 @@ public final class MetalVideoPresenter {
let pipelineSDR: MTLRenderPipelineState
let pipelineHDR: MTLRenderPipelineState
let pipelineHDRToneMap: MTLRenderPipelineState?
let pipelinePlanar: MTLRenderPipelineState
do {
// DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF
// (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is
@@ -292,6 +323,11 @@ public final class MetalVideoPresenter {
#else
pipelineHDRToneMap = nil
#endif
let planar = MTLRenderPipelineDescriptor()
planar.vertexFunction = vtx
planar.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
planar.colorAttachments[0].pixelFormat = .bgra8Unorm // PyroWave is 8-bit SDR
pipelinePlanar = try device.makeRenderPipelineState(descriptor: planar)
} catch {
return nil
}
@@ -331,12 +367,14 @@ public final class MetalVideoPresenter {
return MetalVideoPresenter(
device: device, queue: queue, pipelineSDR: pipelineSDR, pipelineHDR: pipelineHDR,
pipelineHDRToneMap: pipelineHDRToneMap, textureCache: textureCache, layer: layer)
pipelineHDRToneMap: pipelineHDRToneMap, pipelinePlanar: pipelinePlanar,
textureCache: textureCache, layer: layer)
}
private init(
device: MTLDevice, queue: MTLCommandQueue, pipelineSDR: MTLRenderPipelineState,
pipelineHDR: MTLRenderPipelineState, pipelineHDRToneMap: MTLRenderPipelineState?,
pipelinePlanar: MTLRenderPipelineState,
textureCache: CVMetalTextureCache, layer: CAMetalLayer
) {
self.device = device
@@ -344,6 +382,7 @@ public final class MetalVideoPresenter {
self.pipelineSDR = pipelineSDR
self.pipelineHDR = pipelineHDR
self.pipelineHDRToneMap = pipelineHDRToneMap
self.pipelinePlanar = pipelinePlanar
self.textureCache = textureCache
self.layer = layer
}
@@ -514,6 +553,67 @@ public final class MetalVideoPresenter {
pixelBuffer, plane: 1, format: tenBit ? .rg16Unorm : .rg8Unorm, cache: textureCache)
else { return false }
#if os(tvOS)
// HDR splits by the display's headroom (kept in step with the layer by `configure` above):
// PQ passthrough into an HDR-composited display, the tone-map shader otherwise.
let hdrPipeline = hdrPassthroughActive ? pipelineHDR : (pipelineHDRToneMap ?? pipelineHDR)
let pipeline = hdrActive ? hdrPipeline : pipelineSDR
#else
let pipeline = hdrActive ? pipelineHDR : pipelineSDR
#endif
let decodedSize = CGSize(
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
return encodePresent(
decodedSize: decodedSize, targetFromLayout: targetFromLayout, pipeline: pipeline,
presentAtMediaTime: presentAtMediaTime, onPresented: onPresented,
// Hold the CVMetalTextures + source pixel buffer (its IOSurface) alive until the GPU
// finishes sampling releasing them at scope exit could free the backing mid-read.
keepAlive: [luma, chroma, pixelBuffer]
) { encoder in
encoder.setFragmentTexture(CVMetalTextureGetTexture(luma), index: 0)
encoder.setFragmentTexture(CVMetalTextureGetTexture(chroma), index: 1)
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
}
}
/// Draw one PyroWave planar frame (three R8 planes off the Metal wavelet decoder) and
/// present it. RENDER THREAD, same contract as `render` PyroWave is 8-bit SDR, so the
/// layer always takes the plain SDR config, and the CSC rows arrive precomputed from the
/// stream's own sequence-header signaling (no CVPixelBuffer to inspect).
@discardableResult
func renderPlanar(
_ planes: WaveletPlanes,
presentAtMediaTime: CFTimeInterval? = nil,
onPresented: ((Int64?) -> Void)? = nil
) -> Bool {
stagingLock.lock()
let targetFromLayout = drawableTarget
stagingLock.unlock()
configure(hdr: false)
var csc = planes.csc
return encodePresent(
decodedSize: CGSize(width: planes.width, height: planes.height),
targetFromLayout: targetFromLayout, pipeline: pipelinePlanar,
presentAtMediaTime: presentAtMediaTime, onPresented: onPresented,
// The ring textures stay valid by ring depth; retaining them here also pins the
// slot's set until the sample completes (mirrors the biplanar keep-alive).
keepAlive: [planes.y, planes.cb, planes.cr]
) { encoder in
encoder.setFragmentTexture(planes.y, index: 0)
encoder.setFragmentTexture(planes.cb, index: 1)
encoder.setFragmentTexture(planes.cr, index: 2)
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
}
}
/// The shared present tail of `render`/`renderPlanar`: size the drawable, encode one
/// fullscreen triangle with `pipeline` (`bind` supplies the fragment resources), schedule
/// the present and the on-glass callback.
private func encodePresent(
decodedSize: CGSize, targetFromLayout: CGSize, pipeline: MTLRenderPipelineState,
presentAtMediaTime: CFTimeInterval?, onPresented: ((Int64?) -> Void)?,
keepAlive: [Any], bind: (MTLRenderCommandEncoder) -> Void
) -> Bool {
// Size the drawable to the LAYER's pixels (its laid-out frame × contentsScale, pushed here by
// SessionPresenter.layout via `setDrawableTarget` not read off the layer, whose geometry the
// main thread owns) so the Catmull-Rom shader performs the decodedon-screen scale in one pass:
@@ -522,8 +622,6 @@ public final class MetalVideoPresenter {
// Before the first layout (zero target) fall back to the decoded size. drawableSize does NOT
// track bounds (defaults to 0), so set it BEFORE nextDrawable; re-set only on a change
// (layout / Reconfigure / HDR flip and every frame of a live resize, which is fine).
let decodedSize = CGSize(
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
let targetSize = (targetFromLayout.width > 0 && targetFromLayout.height > 0)
? targetFromLayout : decodedSize
if layer.drawableSize != targetSize { layer.drawableSize = targetSize }
@@ -542,17 +640,8 @@ public final class MetalVideoPresenter {
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else {
return false
}
#if os(tvOS)
// HDR splits by the display's headroom (kept in step with the layer by `configure` above):
// PQ passthrough into an HDR-composited display, the tone-map shader otherwise.
let hdrPipeline = hdrPassthroughActive ? pipelineHDR : (pipelineHDRToneMap ?? pipelineHDR)
encoder.setRenderPipelineState(hdrActive ? hdrPipeline : pipelineSDR)
#else
encoder.setRenderPipelineState(hdrActive ? pipelineHDR : pipelineSDR)
#endif
encoder.setFragmentTexture(CVMetalTextureGetTexture(luma), index: 0)
encoder.setFragmentTexture(CVMetalTextureGetTexture(chroma), index: 1)
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
encoder.setRenderPipelineState(pipeline)
bind(encoder)
encoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
encoder.endEncoding()
if let onPresented {
@@ -580,9 +669,8 @@ public final class MetalVideoPresenter {
} else {
commandBuffer.present(drawable)
}
// Hold the CVMetalTextures + source pixel buffer (its IOSurface) alive until the GPU finishes
// sampling releasing them at scope exit could free the backing mid-read.
commandBuffer.addCompletedHandler { _ in _ = (luma, chroma, pixelBuffer) }
// Keep the bound sources alive until the GPU finishes sampling (see the callers).
commandBuffer.addCompletedHandler { _ in _ = keepAlive }
commandBuffer.commit()
return true
}