Files
punktfunk/clients/apple/Sources/PunktfunkKit/Video/MetalVideoPresenter.swift
T
enricobuehler 0e60e58cab
ci / docs-site (push) Successful in 52s
ci / web (push) Successful in 56s
decky / build-publish (push) Successful in 21s
apple / swift (push) Successful in 1m24s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 11s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 10s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 11s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 9s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 12s
ci / bench (push) Successful in 5m53s
release / apple (push) Successful in 9m19s
deb / build-publish (push) Successful in 12m27s
arch / build-publish (push) Successful in 12m52s
docker / deploy-docs (push) Successful in 26s
deb / build-publish-host (push) Successful in 13m18s
apple / screenshots (push) Successful in 6m20s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m51s
android / android (push) Successful in 18m40s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m15s
ci / rust (push) Successful in 27m38s
perf(apple): windowed macOS presents — off-main transactional commit, surface prototype, safe-present setting
Task 1 (latency): the transactional present now commits from the RENDER
thread inside an explicit CATransaction + flush() instead of hopping to
main. The present harness (2026-07-21, 240 Hz Studio, full-size window)
measured the main hop batching presents at runloop-iteration rate when
main is busy — an active implicit transaction NESTS the explicit one —
which is the field's presents=55 @ fps=240 / display_p50 18.6 ms.
Off-main commits measured immune to main churn: glass p50 ~10 ms,
cadence a clean 4.17 ms at 240. flush() is load-bearing: without it the
render thread's own implicit transaction (drawableSize/colour writes)
swallows the explicit commit and NOTHING reaches glass — the harness
reproduced the exact freeze (every present dropped). Legacy main-hop
kept as PUNKTFUNK_TXN_PRESENT=main for field A/B. New pf-windowed
Console line (subsystem io.unom.punktfunk, category presenter)
decomposes the issue side per second.

Task 1 lever D: the f407f418 IOSurface contents-swap path is
resurrected format-aware as WindowedPresentMode.surface — bgra8 SDR /
rgba16Float+PQ-tagged HDR pool, swap committed off-main from the
completion handler; EDR anchored by the metal layer underneath.
Env-only prototype (PUNKTFUNK_WINDOWED_PRESENT=surface) until the HDR
composite is eyeballed on glass.

Task 2 (setting): punktfunk.windowedSafePresent (default ON =
transactional mitigation; OFF = the fast async path whose panic the
saga is about — the caption says so in plain words). Rows in the touch
Settings (Presentation) and gamepad settings, macOS-only.
PUNKTFUNK_WINDOWED_PRESENT=async|transaction|surface overrides for dev
A/B. maximumDrawableCount stays 3 — the harness measured 2 starving the
render loop (172/s) and worsening glass p50.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 15:58:47 +02:00

1206 lines
68 KiB
Swift
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// Stage-2 presenter, present half: draw a decoded NV12 / P010 / 4:4:4 CVPixelBuffer into a CAMetalLayer
// drawable with a YCbCr→RGB shader. The hosting view's CADisplayLink still paces the pipeline once per
// vsync, but the actual `render` runs on Stage2Pipeline's dedicated RENDER THREAD (the link tick just
// signals it), so `nextDrawable()`'s blocking never lands on the main thread. See docs
// apple-stage2-presenter.md.
//
// Threading: created during view setup (main); `render`/`configure` run on the render thread — the
// layer's drawable/format/colour mutations all happen there (CAMetalLayer is designed for dedicated
// render threads; only the layer's GEOMETRY — frame/contentsScale — is touched from main, in
// SessionPresenter.layout, which also pushes the resulting pixel size here via `setDrawableTarget`
// so the render thread never reads layer geometry cross-thread). `setHdrMeta` (pump thread) and
// `setDrawableTarget` (main) only write lock-guarded staging state the render thread drains.
#if canImport(Metal) && canImport(QuartzCore)
import CoreGraphics
import CoreVideo
#if os(macOS)
import IOSurface
#endif
import Metal
import QuartzCore
import os
private let presenterLog = Logger(subsystem: "io.unom.punktfunk", category: "presenter")
#if os(macOS)
/// HOW a windowed (composited) macOS session pushes finished frames to glass — the DCP
/// "mismatched swapID's" kernel-panic saga's mechanism picker. Fullscreen always presents
/// `async` (direct-scanout promotion, lowest latency, no panic reports there); the windowed
/// mechanism is resolved per session by SessionPresenter (user setting +
/// PUNKTFUNK_WINDOWED_PRESENT env override) and routed here via `setWindowedPresent`.
///
/// - `async`: the CAMetalLayer image queue (`commandBuffer.present`) — the fastest composited
/// path and the PANIC TRIGGER on high-refresh displays (the out-of-band swaps race
/// WindowServer's compositor; it survived glass pacing and every codec).
/// - `transaction`: `CAMetalLayer.presentsWithTransaction` — the swap commits WITH the layer
/// tree, in lockstep with the compositor (Apple's documented remedy; validated no-panic on
/// the 240 Hz repro machine). The present is committed from the RENDER thread inside an
/// explicit CATransaction + flush — see `encodePresent` for why that beats the original
/// main-thread hop.
/// - `surface`: no image queue at all — render into a pooled IOSurface and swap it into a plain
/// CALayer's `contents` (the f407f418 PyroWave mitigation, resurrected format-aware:
/// rgba16Float + PQ tagging keeps HDR). WindowServer treats it as ordinary layer damage on
/// its own composite cadence. PROTOTYPE: whether the compositor honors PQ/EDR for plain-layer
/// IOSurface contents still needs an on-glass eyeball — the metal layer stays underneath with
/// `wantsExtendedDynamicRangeContent` as the EDR anchor.
enum WindowedPresentMode: String, Sendable {
case async
case transaction
case surface
}
#endif
/// HDR reference white (BT.2408 "HDR Reference White"): the absolute luminance, in nits, that the
/// PQ signal's diffuse white sits at. Passed to `CAEDRMetadata.hdr10(opticalOutputScale:)`, it anchors
/// 203-nit diffuse white at EDR 1.0 (the display's SDR-white level) and lets the system tone-map the
/// brighter highlights into the panel's headroom. This is the missing anchor that made the old HDR path
/// render "way too bright" (no `edrMetadata` → no reference-white anchoring); a LARGER value renders
/// dimmer. Matches the host's standard PQ reference white.
private let hdrReferenceWhiteNits: Float = 203.0
/// PUNKTFUNK_SDR_COLORSPACE=srgb — A/B hatch for the SDR layer's colour tag. Today the SDR layer
/// ships with `colorspace = nil`, which on macOS means NO colour matching: the BT.709/sRGB-encoded
/// stream is displayed with the panel's native primaries — mild oversaturation on every P3 Mac.
/// `srgb` tags the layer so CoreAnimation colour-matches it into the panel's gamut (the strictly
/// correct rendering). Kept OFF by default until the on-glass A/B confirms it (the nil path is the
/// long-proven look, and some users may prefer the vivid rendition); flip the default once verified.
private let sdrColorspaceOverride: CGColorSpace? = {
guard ProcessInfo.processInfo.environment["PUNKTFUNK_SDR_COLORSPACE"] == "srgb" else {
return nil
}
return CGColorSpace(name: CGColorSpace.sRGB)
}()
/// Runtime-compiled (no metallib build step needed in SwiftPM): a fullscreen triangle and YCbCr→RGB
/// fragment shaders whose conversion arrives as three constant rows computed per frame on the CPU
/// (`CscRows` — the Swift port of pf-client-core's `csc_rows`, from the decoded buffer's actual
/// signaling). One set of coefficients honors BT.601/709/2020 × full/limited × 8/10-bit instead of
/// the old hardcoded BT.709/BT.2020 matrices — a BT.601-signaled stream (a Linux host's RGB-input
/// NVENC) used to render with BT.709 coefficients, a constant hue error. uv.y is flipped (1 - p.y)
/// so the top-left-origin texture presents upright (NDC y is up). The HDR shader outputs PQ-encoded
/// RGB as-is — the CAMetalLayer's `itur_2100_PQ` colour space + `edrMetadata` tell the system
/// compositor the samples are PQ and how to tone-map them (no EOTF here, matching the host's
/// BT.2020 PQ emission).
private let shaderSource = """
#include <metal_stdlib>
using namespace metal;
struct VOut { float4 pos [[position]]; float2 uv; };
// The CPU-computed CSC rows (CscRows.swift, layout-matched): rgb[i] = dot(ri.xyz, yuv) + ri.w.
// Range expansion, the matrix, and the 10-bit MSB-packing factor are all folded in.
struct CscUniform { float4 r0; float4 r1; float4 r2; };
vertex VOut pf_vtx(uint vid [[vertex_id]]) {
float2 p = float2(float((vid << 1) & 2), float(vid & 2));
VOut o;
o.pos = float4(p * 2.0 - 1.0, 0.0, 1.0);
o.uv = float2(p.x, 1.0 - p.y);
return o;
}
// Bicubic (Catmull-Rom) sampling of the single-channel luma plane. The drawable is sized to the
// LAYER's pixels (see `render`), so this kernel performs the decoded→on-screen scale: when the
// window/view is bigger than the host's fixed mode a bilinear upscale looks soft; Catmull-Rom
// keeps edges crisp — matching AVSampleBufferDisplayLayer's (stage-1) scaler — and reduces to the
// exact texel at 1:1, so a native-resolution present stays pixel-exact.
// Nine bilinear taps (TheRealMJP's optimisation of the 16-tap kernel); `s` MUST be a linear
// sampler. Luma carries the perceived detail, so only it gets bicubic; chroma stays bilinear.
float catmullRomLuma(texture2d<float> tex, sampler s, float2 uv) {
float2 texSize = float2(tex.get_width(), tex.get_height());
float2 samplePos = uv * texSize;
float2 tc1 = floor(samplePos - 0.5) + 0.5;
float2 f = samplePos - tc1;
float2 w0 = f * (-0.5 + f * (1.0 - 0.5 * f));
float2 w1 = 1.0 + f * f * (-2.5 + 1.5 * f);
float2 w2 = f * (0.5 + f * (2.0 - 1.5 * f));
float2 w3 = f * f * (-0.5 + 0.5 * f);
float2 w12 = w1 + w2;
float2 off12 = w2 / w12;
float2 tc0 = (tc1 - 1.0) / texSize;
float2 tc3 = (tc1 + 2.0) / texSize;
float2 tc12 = (tc1 + off12) / texSize;
float r = 0.0;
r += tex.sample(s, float2(tc0.x, tc0.y)).r * (w0.x * w0.y);
r += tex.sample(s, float2(tc12.x, tc0.y)).r * (w12.x * w0.y);
r += tex.sample(s, float2(tc3.x, tc0.y)).r * (w3.x * w0.y);
r += tex.sample(s, float2(tc0.x, tc12.y)).r * (w0.x * w12.y);
r += tex.sample(s, float2(tc12.x, tc12.y)).r * (w12.x * w12.y);
r += tex.sample(s, float2(tc3.x, tc12.y)).r * (w3.x * w12.y);
r += tex.sample(s, float2(tc0.x, tc3.y)).r * (w0.x * w3.y);
r += tex.sample(s, float2(tc12.x, tc3.y)).r * (w12.x * w3.y);
r += tex.sample(s, float2(tc3.x, tc3.y)).r * (w3.x * w3.y);
return r;
}
// 4:2:0 chroma is left-cosited horizontally (H.273 chroma_loc type 0 — the MPEG convention the
// host encodes and VideoToolbox decodes as-is), but sampling the half-res plane at the luma UV
// assumes CENTER siting — a ~0.5-luma-px rightward chroma shift on hard colored edges. Offset the
// sample by +0.25 chroma texels to re-align (libplacebo/mpv's correction). Vertical siting for
// type 0 is centered, which plain sampling already matches. A full-size 4:4:4 plane has no
// subsampling to correct — the offset self-disables when the plane widths match.
float2 chromaUV(texture2d<float> lumaTex, texture2d<float> chromaTex, float2 uv) {
if (chromaTex.get_width() < lumaTex.get_width()) {
uv.x += 0.25 / float(chromaTex.get_width());
}
return uv;
}
// The shared sample + row-multiply: YCbCr (bicubic luma, siting-corrected bilinear chroma) →
// RGB via the per-frame rows. A full-size 4:4:4 chroma plane needs no change vs 4:2:0 (the siting
// offset self-disables). What the result MEANS depends on the stream: an SDR frame's rows yield
// gamma-encoded RGB, an HDR frame's rows yield PQ-encoded RGB — the fragment variants below
// differ only in what they do next.
float3 sampleRgb(texture2d<float> lumaTex, texture2d<float> chromaTex, float2 uv,
constant CscUniform& csc) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
#ifdef PF_BILINEAR_LUMA
// DEBUG (PUNKTFUNK_BILINEAR_LUMA=1): plain bilinear luma — Catmull-Rom OFF. A/B lever to see if
// the bicubic overshoot contributes to edge fringing. NOTE: at a true 1:1 present both paths
// reduce to the identity texel, so if this toggle VISIBLY changes the picture, the present is
// NOT 1:1 (there's a resample); if it looks identical, the fringing is upstream (codec/source/OS).
float lumaY = lumaTex.sample(s, uv).r;
#else
float lumaY = catmullRomLuma(lumaTex, s, uv);
#endif
float3 yuv = float3(lumaY,
chromaTex.sample(s, chromaUV(lumaTex, chromaTex, uv)).rg);
return saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
dot(csc.r1.xyz, yuv) + csc.r1.w,
dot(csc.r2.xyz, yuv) + csc.r2.w));
}
// SDR: 8-bit NV12 / 4:4:4 → full-range RGB, transfer left baked (shown as-is, the proven SDR
// layer config).
fragment float4 pf_frag(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> chromaTex [[texture(1)]],
constant CscUniform& csc [[buffer(0)]]) {
return float4(sampleRgb(lumaTex, chromaTex, in.uv, csc), 1.0);
}
// PyroWave planar SDR: three separate R8 planes (Y full-res, Cb/Cr half-res 4:2:0) from the
// Metal wavelet decoder — the Metal twin of pf-presenter's planar_csc.frag. Same bicubic luma
// and left-cosited chroma correction as the biplanar path (chromaUV self-disables at 4:4:4).
fragment float4 pf_frag_planar(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> cbTex [[texture(1)]],
texture2d<float> crTex [[texture(2)]],
constant CscUniform& csc [[buffer(0)]]) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
#ifdef PF_BILINEAR_LUMA
float lumaY = lumaTex.sample(s, in.uv).r;
#else
float lumaY = catmullRomLuma(lumaTex, s, in.uv);
#endif
float2 cuv = chromaUV(lumaTex, cbTex, in.uv);
float3 yuv = float3(lumaY, cbTex.sample(s, cuv).r, crTex.sample(s, cuv).r);
float3 rgb = saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
dot(csc.r1.xyz, yuv) + csc.r1.w,
dot(csc.r2.xyz, yuv) + csc.r2.w));
return float4(rgb, 1.0);
}
// HDR: 10-bit P010 / 4:4:4 (BT.2020, PQ-encoded YCbCr) → full-range PQ RGB, output as-is —
// the CAMetalLayer's itur_2100_PQ colour space + edrMetadata tell the compositor the samples are
// PQ, so it does the PQ→display tone-map. No EOTF here. The rows fold in the exact 10-bit
// MSB-packing factor (the old hardcoded shader carried a documented ~0.1% /1024-vs-/1023 error).
fragment float4 pf_frag_hdr(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> chromaTex [[texture(1)]],
constant CscUniform& csc [[buffer(0)]]) {
return float4(sampleRgb(lumaTex, chromaTex, in.uv, csc), 1.0);
}
// HDR on tvOS when the display is composited WITHOUT HDR headroom (SDR output mode, or the user
// disabled Match Dynamic Range): no Metal EDR API exists there (CAEDRMetadata /
// wantsExtendedDynamicRangeContent are API_UNAVAILABLE(tvos)), and a bare PQ colour-space tag
// composites UNtone-mapped — the CAMetalLayer header says so outright — which showed as a badly
// overblown picture on Apple TV. So this variant finishes the job in-shader: PQ EOTF → linear
// light, 203-nit reference white (BT.2408) anchored at display white, extended-Reinhard highlight
// rolloff with a 1000-nit knee, BT.2020→BT.709 primaries, BT.709 OETF — into the proven SDR layer
// config. The 10-bit BT.2020 stream keeps its full decode depth; only the final presentation is
// display-referred SDR. (When the display IS in an HDR mode — requested per session via
// AVDisplayManager, see StreamViewIOS — tvOS presents pf_frag_hdr's PQ passthrough instead:
// in a genuine HDR10 output, PQ passthrough is the correct emission and the TV tone-maps.)
// The shared PQ→display-referred-SDR tail (see pf_frag_hdr_tv's rationale above): ST 2084
// EOTF → 203-nit-anchored scene light → BT.2020→709 primaries → extended-Reinhard rolloff →
// BT.709 OETF. Used by the tvOS biplanar tone-map and the tvOS planar (PyroWave) tone-map (the
// no-HDR-headroom fallback). macOS keeps real HDR windowed now — see `WindowedPresentMode`.
static inline float3 pqToSdr(float3 pq) {
const float m1 = 2610.0/16384.0;
const float m2 = 78.84375;
const float c1 = 3424.0/4096.0;
const float c2 = 18.8515625;
const float c3 = 18.6875;
float3 p = pow(pq, 1.0/m2);
float3 lin = pow(max(p - c1, 0.0) / (c2 - c3 * p), 1.0/m1);
float3 t = lin * (10000.0/203.0);
float3 t709 = float3(
dot(t, float3( 1.6605, -0.5876, -0.0728)),
dot(t, float3(-0.1246, 1.1329, -0.0083)),
dot(t, float3(-0.0182, -0.1006, 1.1187)));
t709 = max(t709, 0.0);
const float w = 1000.0/203.0;
float3 mapped = saturate(t709 * (1.0 + t709 / (w * w)) / (1.0 + t709));
float3 e = select(1.099 * pow(mapped, 0.45) - 0.099, 4.5 * mapped, mapped < 0.018);
return e;
}
fragment float4 pf_frag_hdr_tv(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> chromaTex [[texture(1)]],
constant CscUniform& csc [[buffer(0)]]) {
// YCbCr → full-range PQ RGB via the per-frame rows (as pf_frag_hdr), then the tail.
return float4(pqToSdr(sampleRgb(lumaTex, chromaTex, in.uv, csc)), 1.0);
}
// PyroWave planar HDR tone-map: three separate R16 planes (P010-style studio codes; the rows
// fold in depth-10 MSB packing) → PQ RGB → the shared SDR tail. Used when a PQ pyrowave
// stream must land on an 8-bit surface: tvOS without HDR headroom. The passthrough planar
// HDR pipeline reuses pf_frag_planar itself on an rgba16Float drawable (identical math — the
// layer's itur_2100_PQ colour space + EDR metadata do the interpretation).
fragment float4 pf_frag_planar_tm(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> cbTex [[texture(1)]],
texture2d<float> crTex [[texture(2)]],
constant CscUniform& csc [[buffer(0)]]) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
#ifdef PF_BILINEAR_LUMA
float lumaY = lumaTex.sample(s, in.uv).r;
#else
float lumaY = catmullRomLuma(lumaTex, s, in.uv);
#endif
float2 cuv = chromaUV(lumaTex, cbTex, in.uv);
float3 yuv = float3(lumaY, cbTex.sample(s, cuv).r, crTex.sample(s, cuv).r);
float3 pq = saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
dot(csc.r1.xyz, yuv) + csc.r1.w,
dot(csc.r2.xyz, yuv) + csc.r2.w));
return float4(pqToSdr(pq), 1.0);
}
"""
public final class MetalVideoPresenter {
/// The layer the hosting view installs (as a sublayer) and sizes to its bounds.
public let layer: CAMetalLayer
#if os(macOS)
/// WINDOWED-mode present coordination — the macOS DCP KERNEL PANIC mitigation.
///
/// The panic ("mismatched swapID's" @UnifiedPipeline.cpp, WindowServer dies, machine reboots):
/// the CAMetalLayer's ASYNCHRONOUS image queue (`commandBuffer.present(drawable)` — an
/// out-of-band flip, mandatory with `displaySyncEnabled=false`) diverges from WindowServer's
/// compositor on a high-refresh COMPOSITED (windowed) session — the compositor's notion of the
/// current swap and the layer's queued swap disagree, and the DCP asserts. It survived glass
/// pacing: a fully serialized one-in-flight present stream still panicked a 240 Hz Mac Studio
/// (2026-07-18, PyroWave), and a windowed HEVC session panicked the same machine 2026-07-21 —
/// so it is the async image queue itself, at any pacing or codec, not a present rate.
///
/// The fix keeps the full render path (rgba16Float / PQ / EDR — real HDR is preserved) and
/// only changes HOW the drawable is presented: `CAMetalLayer.presentsWithTransaction`. With it
/// set, we don't hand the drawable to the command buffer; we commit, wait until scheduled, then
/// call `drawable.present()` INSIDE a CATransaction — the present is enrolled in Core
/// Animation's transaction and committed together with the layer tree, so the swap stays in
/// lockstep with the compositor instead of racing it (Apple's documented remedy for Metal
/// presentation drifting out of sync with CA). Fullscreen keeps the async path (direct-scanout
/// promotion, lowest latency, no compositor and no panic reports there).
///
/// 2026-07-21 latency rework: the mitigation MECHANISM is now a three-way pick
/// (`WindowedPresentMode`) and the transactional present commits from the RENDER thread —
/// see `encodePresent`. Staged under `stagingLock` (main pushes it via
/// `setComposited`→`setWindowedPresent`); the render thread drains it and toggles the layer
/// property + present style. `Active` is the render-thread copy so the layer property flips
/// exactly once per mode change.
private var windowedPresentStaged: WindowedPresentMode = .async
private var windowedPresentActive: WindowedPresentMode = .async
/// PUNKTFUNK_TXN_PRESENT=main — the ORIGINAL transactional present (commit →
/// waitUntilScheduled → hop to the MAIN thread and present inside its CATransaction), kept
/// as a field A/B lever. The default is the render-thread commit: the present harness
/// (2026-07-21, this saga) measured the main hop landing a runloop turn late on a busy main
/// thread, and an ACTIVE implicit transaction there NESTS the explicit one — presents batch
/// at runloop-iteration rate (the field's presents=55 @ fps=240, display_p50 18.6 ms).
/// Off-main commits measured immune to main-thread churn (~10 ms glass p50 at 240 Hz
/// full-size vs 14+ ms under a churned main hop).
private let txnPresentOnMain =
ProcessInfo.processInfo.environment["PUNKTFUNK_TXN_PRESENT"] == "main"
/// The WINDOWED-mode `surface` present target: a plain CALayer sized like `layer` (installed
/// as a sibling ABOVE it by SessionPresenter), fed IOSurfaces via `contents` inside explicit
/// CATransactions. Transparent (nil contents) whenever surface mode is off, so the metal
/// layer below shows through. See `WindowedPresentMode.surface`.
let surfaceLayer: CALayer = {
let l = CALayer()
l.contentsGravity = .resize // frame is already aspect-fit + pixel-snapped by layout
l.isOpaque = true
l.actions = ["contents": NSNull(), "bounds": NSNull(), "position": NSNull()]
return l
}()
/// One IOSurface-backed render target of the windowed surface-present pool. All pool state
/// is RENDER-THREAD confined; only the immutable surface refs cross threads (contents swap).
private struct SurfaceSlot {
let surface: IOSurfaceRef
let texture: MTLTexture
/// Monotonic use stamp — the reuse picker takes the least-recently-rendered free slot.
var seq: UInt64 = 0
}
private var surfacePool: [SurfaceSlot] = []
private var surfacePoolSize: CGSize = .zero
private var surfacePoolHDR = false
private var surfaceSeq: UInt64 = 0
/// Index of the slot most recently handed to the layer — never rewritten next, even if its
/// use count already dropped (the compositor may still be scanning out the previous frame).
private var lastHandedOff: Int?
/// Once-per-second decomposition of the ACTIVE windowed present path (the field-diagnosis
/// half of the DCP-latency work): scheduled/completed wait + commit/flush cost per present,
/// and how many presents/swaps were issued. The pf-present line shows the GLASS side
/// (latchMs / dropped); this shows the ISSUE side. Logged via `presenterLog` only while a
/// windowed mechanism is active (zero cost fullscreen). Lock-guarded: transaction mode
/// records from the render thread, surface mode from Metal completion threads.
private final class WindowedPresentDiag: @unchecked Sendable {
private let lock = NSLock()
private var presents = 0
private var schedMs: [Double] = []
private var commitMs: [Double] = []
private var last = CACurrentMediaTime()
func record(schedMs sched: Double, commitMs commit: Double, mode: WindowedPresentMode) {
lock.lock()
presents += 1
schedMs.append(sched)
commitMs.append(commit)
let now = CACurrentMediaTime()
guard now - last >= 1 else {
lock.unlock()
return
}
last = now
let sSched = schedMs.sorted()
let sCommit = commitMs.sorted()
let line = String(
format: "pf-windowed mode=%@ presents=%d schedMs p50=%.2f max=%.2f "
+ "commitMs p50=%.2f max=%.2f",
mode.rawValue, presents, sSched[sSched.count / 2], sSched.last ?? 0,
sCommit[sCommit.count / 2], sCommit.last ?? 0)
presents = 0
schedMs.removeAll(keepingCapacity: true)
commitMs.removeAll(keepingCapacity: true)
lock.unlock()
presenterLog.info("\(line, privacy: .public)")
}
}
private let windowedDiag = WindowedPresentDiag()
#endif
private let device: MTLDevice
private let queue: MTLCommandQueue
/// SDR (BT.709 8-bit → bgra8) and HDR (BT.2020 PQ 10-bit → rgba16Float) pipelines. Selected per
/// frame in `render`; the layer is reconfigured to match when the session flips (HDR toggle).
private let pipelineSDR: MTLRenderPipelineState
private let pipelineHDR: MTLRenderPipelineState
/// tvOS only: the in-shader PQ→SDR tone-map fallback (pf_frag_hdr_tv → bgra8), used whenever
/// the display is composited without HDR headroom — see `setDisplayHeadroom`. nil elsewhere.
private let pipelineHDRToneMap: MTLRenderPipelineState?
/// PyroWave's 3-plane SDR path (pf_frag_planar → bgra8) — see `renderPlanar`.
private let pipelinePlanar: MTLRenderPipelineState
/// PyroWave planar HDR passthrough (pf_frag_planar → rgba16Float; the layer's PQ colour
/// space + EDR interpret the samples) and the planar PQ→SDR tone-map (pf_frag_planar_tm →
/// bgra8; tvOS without HDR headroom).
private let pipelinePlanarHDR: MTLRenderPipelineState
private let pipelinePlanarToneMap: MTLRenderPipelineState
private var textureCache: CVMetalTextureCache?
/// The PyroWave Metal decoder records on the presenter's device + queue: one device means
/// decode, CSC and present share textures with zero interop, and one queue means Metal's
/// hazard tracking orders a ring-slot rewrite after the render still sampling it.
var metalDevice: MTLDevice { device }
var metalQueue: MTLCommandQueue { queue }
/// Current layer configuration — switched in `configure(hdr:)` when a frame's HDR-ness differs.
/// Render-thread confined once the pipeline runs (Stage2Pipeline.start's one pre-thread
/// `configure` call is ordered before the thread starts, so it doesn't race).
private var hdrActive = false
/// tvOS only: whether HDR frames currently present as PQ PASSTHROUGH (display has HDR headroom
/// — its own tone-map applies) vs the in-shader tone-map fallback. Render-thread confined;
/// derived from the staged display headroom at the top of every `render`.
private var hdrPassthroughActive = false
/// Last HDR mastering grade received via `setHdrMeta` (the host's 0xCE). Cached so a mid-session
/// SDR→HDR flip's `configureColor` re-applies the real grade instead of clobbering it back to the
/// bare reference-white anchor (an out-of-order race otherwise: `setHdrMeta` and the flip both write
/// `edrMetadata`). Render-thread confined (drained from `pendingHdrMeta` at the top of `render`).
private var lastHdrMeta: PunktfunkConnection.HdrMeta?
/// Cross-thread staging, all under `stagingLock`: the pump thread parks a fresh 0xCE grade in
/// `pendingHdrMeta` and the main thread parks the layout-derived drawable pixel size in
/// `drawableTarget`; the render thread drains both at the top of `render`, so every layer
/// format/colour mutation stays on the one thread that also calls `nextDrawable()`.
private let stagingLock = NSLock()
private var pendingHdrMeta: PunktfunkConnection.HdrMeta?
private var drawableTarget: CGSize = .zero
/// tvOS: the display's current EDR headroom (UIScreen.currentEDRHeadroom), pushed from the
/// main thread (SessionPresenter.layout + the mode-switch observers). > 1 ⇒ the display is
/// composited with HDR headroom, so HDR frames present as PQ passthrough; otherwise the
/// in-shader tone-map keeps the picture from blowing out. 1 (the default) is the safe start.
private var stagedDisplayHeadroom: CGFloat = 1.0
#if DEBUG
/// Last logged "decoded→drawable" signature, so the diagnostic logs only on a size/HDR change.
private var lastSizeSig = ""
#endif
/// nil if Metal is unavailable (no GPU / a headless CI) or a shader fails to compile — the caller
/// falls back to stage-1.
public static func make() -> MetalVideoPresenter? {
guard let device = MTLCreateSystemDefaultDevice(),
let queue = device.makeCommandQueue()
else { return nil }
let pipelineSDR: MTLRenderPipelineState
let pipelineHDR: MTLRenderPipelineState
let pipelineHDRToneMap: MTLRenderPipelineState?
let pipelinePlanar: MTLRenderPipelineState
let pipelinePlanarHDR: MTLRenderPipelineState
let pipelinePlanarToneMap: MTLRenderPipelineState
do {
// DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF
// (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is
// the normal bicubic path. Read at presenter creation — set it in the environment and
// relaunch to flip; the log line confirms which path built.
let bilinearLuma = ProcessInfo.processInfo.environment["PUNKTFUNK_BILINEAR_LUMA"] == "1"
let source = (bilinearLuma ? "#define PF_BILINEAR_LUMA 1\n" : "") + shaderSource
if bilinearLuma {
presenterLog.info("stage2: PUNKTFUNK_BILINEAR_LUMA=1 — Catmull-Rom luma DISABLED (bilinear)")
}
let library = try device.makeLibrary(source: source, options: nil)
let vtx = library.makeFunction(name: "pf_vtx")
let sdr = MTLRenderPipelineDescriptor()
sdr.vertexFunction = vtx
sdr.fragmentFunction = library.makeFunction(name: "pf_frag")
sdr.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelineSDR = try device.makeRenderPipelineState(descriptor: sdr)
let hdr = MTLRenderPipelineDescriptor()
hdr.vertexFunction = vtx
hdr.fragmentFunction = library.makeFunction(name: "pf_frag_hdr")
hdr.colorAttachments[0].pixelFormat = .rgba16Float // PQ passthrough
pipelineHDR = try device.makeRenderPipelineState(descriptor: hdr)
#if os(tvOS)
// tvOS carries BOTH HDR pipelines: PQ passthrough when the display is composited
// with HDR headroom, the in-shader tone-map (→ the 8-bit SDR config) when it isn't.
// See setDisplayHeadroom / configureColor.
let tm = MTLRenderPipelineDescriptor()
tm.vertexFunction = vtx
tm.fragmentFunction = library.makeFunction(name: "pf_frag_hdr_tv")
tm.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelineHDRToneMap = try device.makeRenderPipelineState(descriptor: tm)
#else
pipelineHDRToneMap = nil
#endif
let planar = MTLRenderPipelineDescriptor()
planar.vertexFunction = vtx
planar.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
planar.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelinePlanar = try device.makeRenderPipelineState(descriptor: planar)
let planarHdr = MTLRenderPipelineDescriptor()
planarHdr.vertexFunction = vtx
planarHdr.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
planarHdr.colorAttachments[0].pixelFormat = .rgba16Float // PQ passthrough
pipelinePlanarHDR = try device.makeRenderPipelineState(descriptor: planarHdr)
let planarTm = MTLRenderPipelineDescriptor()
planarTm.vertexFunction = vtx
planarTm.fragmentFunction = library.makeFunction(name: "pf_frag_planar_tm")
planarTm.colorAttachments[0].pixelFormat = .bgra8Unorm
pipelinePlanarToneMap = try device.makeRenderPipelineState(descriptor: planarTm)
} catch {
return nil
}
var cache: CVMetalTextureCache?
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, device, nil, &cache)
guard let textureCache = cache else { return nil }
let layer = CAMetalLayer()
layer.device = device
layer.pixelFormat = .bgra8Unorm
layer.framebufferOnly = true
layer.isOpaque = true
#if os(macOS)
// displaySyncEnabled MUST stay false on macOS. It has flip-flopped, so the full history:
// sync ON was tried twice and starves the drawable pool both times — on macOS 26 a synced
// present only reaches glass when the WindowServer composites the window, and its FramePacing
// path does not treat our out-of-band image-queue presents as damage, so with a static scene
// the ONLY recurring damage is the 1 Hz stats HUD update: presents queue, all drawables stay
// held, `nextDrawable()` sleeps (sampled: ~70% of the render thread inside
// CAMetalLayerPrivateNextDrawableLocked → usleep), and the stream turns into a ~1 fps
// slideshow with normal-LOOKING stats (each rare frame is fresh, newest-wins ring). The
// 94fb7d1b fullscreen judder was the same starvation biting the then-main-thread render.
// With sync OFF the flip is immediate; the vsync alignment that sync was supposed to give
// (the HUD-off direct-scanout pacing fix) comes from scheduling the present at the display
// link's target time instead (`present(at:)` — see `render`).
layer.displaySyncEnabled = false
#endif
// The drawable is rendered at the LAYER's pixel size (set per-frame in `render`), so the
// shader — not the compositor — performs the decoded→on-screen scale (bicubic luma; the
// compositor's contentsGravity path is plain bilinear). The gravity stays aspect-fit as a
// transient fallback: during a live resize the compositor may composite a drawable from
// the previous layout before the next render catches up.
layer.contentsGravity = .resizeAspect
// Triple-buffer: more in-flight drawables before `nextDrawable()` (called on the display-link /
// MAIN thread) has to block waiting for one to free.
layer.maximumDrawableCount = 3
return MetalVideoPresenter(
device: device, queue: queue, pipelineSDR: pipelineSDR, pipelineHDR: pipelineHDR,
pipelineHDRToneMap: pipelineHDRToneMap, pipelinePlanar: pipelinePlanar,
pipelinePlanarHDR: pipelinePlanarHDR, pipelinePlanarToneMap: pipelinePlanarToneMap,
textureCache: textureCache, layer: layer)
}
private init(
device: MTLDevice, queue: MTLCommandQueue, pipelineSDR: MTLRenderPipelineState,
pipelineHDR: MTLRenderPipelineState, pipelineHDRToneMap: MTLRenderPipelineState?,
pipelinePlanar: MTLRenderPipelineState,
pipelinePlanarHDR: MTLRenderPipelineState,
pipelinePlanarToneMap: MTLRenderPipelineState,
textureCache: CVMetalTextureCache, layer: CAMetalLayer
) {
self.device = device
self.queue = queue
self.pipelineSDR = pipelineSDR
self.pipelineHDR = pipelineHDR
self.pipelineHDRToneMap = pipelineHDRToneMap
self.pipelinePlanar = pipelinePlanar
self.pipelinePlanarHDR = pipelinePlanarHDR
self.pipelinePlanarToneMap = pipelinePlanarToneMap
self.textureCache = textureCache
self.layer = layer
}
/// Configure the layer + active pipeline for an SDR or HDR session. Called once at session start
/// (main, before the render thread exists) and again per-frame from `render` on the RENDER THREAD
/// (idempotent — the guard makes a same-state call a no-op), so a mid-session HDR toggle (the host
/// re-inits its encoder; the decoded `frame.isHDR` flips) reconfigures here automatically. HDR uses
/// an rgba16Float drawable + BT.2020 PQ colour space + EDR with a 203-nit reference-white anchor;
/// SDR uses the plain 8-bit sRGB path.
public func configure(hdr: Bool) {
#if os(tvOS)
// Reconfigure on an HDR flip AND on a passthrough↔tone-map flip: the display's headroom
// changes when the AVDisplayManager mode switch (requested at session start) completes —
// typically a second or two into the session.
stagingLock.lock()
let passthrough = stagedDisplayHeadroom > 1.0
stagingLock.unlock()
guard hdr != hdrActive || (hdr && passthrough != hdrPassthroughActive) else { return }
hdrActive = hdr
hdrPassthroughActive = passthrough
#else
guard hdr != hdrActive else { return }
hdrActive = hdr
#endif
configureColor(hdr: hdr)
}
/// tvOS: park the display's current EDR headroom (a MAIN-thread `UIScreen` read — pushed by
/// SessionPresenter.layout and the stream view's mode-switch observers). > 1 flips HDR frames
/// to PQ passthrough (the display's own tone-map applies); ≤ 1 keeps the in-shader tone-map.
/// Applied by the render thread on the next frame, like every other staged value here.
public func setDisplayHeadroom(_ headroom: CGFloat) {
stagingLock.lock()
stagedDisplayHeadroom = headroom
stagingLock.unlock()
}
/// Set the layer's pixel format + colour config for SDR or HDR. MAIN THREAD ONLY. EDR is requested
/// on macOS + iOS (the old `#if os(macOS)` guard left iOS EDR half-engaged). tvOS has NO EDR API
/// (`wantsExtendedDynamicRangeContent`/`edrMetadata`/`CAEDRMetadata` are all unavailable there) —
/// and a bare PQ colour-space tag composites UNtone-mapped (the "overblown HDR" Apple TV report),
/// so tvOS instead tone-maps PQ→SDR in the shader (pf_frag_hdr_tv) and keeps the SDR layer config.
private func configureColor(hdr: Bool) {
if hdr {
#if os(tvOS)
if hdrPassthroughActive {
// Display composited WITH HDR headroom (the session's AVDisplayManager request
// landed): emit PQ passthrough — in a real HDR10 output that's the correct
// emission, and the TV applies its own tone-map.
layer.pixelFormat = .rgba16Float
layer.colorspace = CGColorSpace(name: CGColorSpace.itur_2100_PQ)
} else {
// SDR-composited display: PQ would render untone-mapped (blown out) — the
// pf_frag_hdr_tv shader tone-maps to SDR instead.
layer.pixelFormat = .bgra8Unorm
layer.colorspace = nil
}
#else
layer.pixelFormat = .rgba16Float
layer.colorspace = CGColorSpace(name: CGColorSpace.itur_2100_PQ)
layer.wantsExtendedDynamicRangeContent = true
// Anchor reference white. Re-apply the real grade if one already arrived (0xCE before the
// flip); otherwise the bare 203-nit anchor. Without this anchor the PQ signal is too bright.
layer.edrMetadata = makeEDR(lastHdrMeta)
#endif
} else {
// SDR: gamma-encoded BT.709 [0,1] in an 8-bit drawable. Default: nil colorspace = NO
// colour matching on macOS (the panel's native primaries — the long-proven look,
// slightly oversaturated on P3 panels); PUNKTFUNK_SDR_COLORSPACE=srgb tags the layer
// for correct colour matching instead (A/B pending — see sdrColorspaceOverride).
layer.pixelFormat = .bgra8Unorm
layer.colorspace = sdrColorspaceOverride
#if !os(tvOS)
layer.wantsExtendedDynamicRangeContent = false
layer.edrMetadata = nil
#endif
}
}
#if !os(tvOS)
private func makeEDR(_ meta: PunktfunkConnection.HdrMeta?) -> CAEDRMetadata {
CAEDRMetadata.hdr10(
displayInfo: meta?.masteringDisplayColorVolume(),
contentInfo: meta?.contentLightLevelInfo(),
opticalOutputScale: hdrReferenceWhiteNits)
}
#endif
/// Update the HDR mastering metadata (drained from the host's 0xCE datagram) to refine the system
/// tone-map from the real grade. Called from the PUMP thread — the grade is only PARKED here (lock-
/// guarded); the render thread applies it at the top of the next `render`, keeping every layer
/// colour mutation on the one thread that also vends drawables.
public func setHdrMeta(_ meta: PunktfunkConnection.HdrMeta) {
stagingLock.lock()
pendingHdrMeta = meta
stagingLock.unlock()
}
/// Park the drawable pixel size the shader should render at: the metal layer's laid-out frame ×
/// contentsScale, both owned by the MAIN thread (SessionPresenter.layout pushes it on every layout/
/// backing change). The render thread reads this instead of the layer's geometry so it never
/// touches main-owned CALayer state. Zero until the first layout → `render` falls back to the
/// decoded frame size.
public func setDrawableTarget(_ size: CGSize) {
stagingLock.lock()
drawableTarget = size
stagingLock.unlock()
}
#if os(macOS)
/// Park the windowed present mechanism (MAIN thread — the hosting view pushes its window
/// state on every layout; SessionPresenter resolves the mechanism per session). `.async` =
/// FULLSCREEN (or the user opted out of the mitigation): the image queue. `.transaction` /
/// `.surface` = COMPOSITED (windowed) mitigation mechanisms — see `WindowedPresentMode`.
/// Applied by the render thread on the next frame, like every other staged value here.
func setWindowedPresent(_ mode: WindowedPresentMode) {
stagingLock.lock()
windowedPresentStaged = mode
stagingLock.unlock()
}
#endif
/// Deadline pacing only, RENDER THREAD: reconcile the layer with a decoded frame BEFORE a
/// drawable exists. The link vends from the layer's CURRENT config, and the layer starts
/// with `drawableSize` 0 (it never tracks bounds once set explicitly, and the sublayer's
/// frame isn't even laid out when the link spins up) — so leaving all reconciliation to the
/// render path (which needs a frame AND a vended drawable) deadlocks at session start:
/// every vend fails allocation at 0×0, the stash stays empty, no pair ever completes, and
/// the size is never set. The 2026-07-19 iPad black screen ("[CAMetalLayer nextDrawable]
/// returning nil because allocation failed" every refresh). Called on EVERY frame arrival:
/// drains the same staging the render path drains (both are idempotent about it) and
/// applies size + HDR config, so the next vend always matches the frame about to present —
/// this also makes a mid-session HDR flip cost at most one skipped vend instead of waiting
/// for a paired present to retag the layer.
func reconcileLayer(decodedSize: CGSize, isHDR: Bool) {
stagingLock.lock()
let targetFromLayout = drawableTarget
let newHdrMeta = pendingHdrMeta
pendingHdrMeta = nil
stagingLock.unlock()
configure(hdr: isHDR)
if let newHdrMeta {
self.lastHdrMeta = newHdrMeta
#if !os(tvOS)
if hdrActive { layer.edrMetadata = makeEDR(newHdrMeta) }
#endif
}
let targetSize = (targetFromLayout.width > 0 && targetFromLayout.height > 0)
? targetFromLayout : decodedSize
if layer.drawableSize != targetSize { layer.drawableSize = targetSize }
}
/// Draw one decoded frame to the next drawable and present it. RENDER THREAD (Stage2Pipeline's;
/// `nextDrawable()` may block up to a frame — that wait belongs here, never on main). `isHDR`
/// selects the 10-bit BT.2020 PQ path vs the 8-bit BT.709 path and is reconciled with the
/// layer config via `configure`. Returns true on success; false when there's no drawable yet, a
/// texture couldn't be made, or Metal errored — the caller then doesn't stamp a present (and can
/// requeue the frame). `onPresented` fires once the drawable actually reached glass, with the
/// `CLOCK_REALTIME` instant from the drawable's `presentedTime` — or nil when the system reports
/// none (a dropped drawable). It runs on a Metal callback thread; keep the handler thread-safe.
///
/// `presentAtMediaTime` (a `CACurrentMediaTime`-basis host time — the display link's
/// `targetTimestamp`) schedules the flip ON the vsync instead of "as soon as the GPU finishes":
/// with the layer's own sync disabled (mandatory on macOS — see init) an immediate present hits
/// glass mid-refresh whenever the layer is direct-scanout promoted (fullscreen, no HUD), which
/// is the "frametimes are off with the stats HUD closed" report. nil presents immediately
/// (`PUNKTFUNK_PRESENT_MODE=immediate` — the pre-fix behavior, kept as a diagnostic A/B).
///
/// `into drawable` (deadline pacing) supplies the CAMetalDisplayLink-vended drawable to
/// render into instead of calling `nextDrawable()` — see `encodePresent` for the format
/// guard that skips a vend the layer's config outran.
@discardableResult
public func render(
_ pixelBuffer: CVPixelBuffer, isHDR: Bool = false,
presentAtMediaTime: CFTimeInterval? = nil,
into drawable: CAMetalDrawable? = nil,
onPresented: ((Int64?) -> Void)? = nil
) -> Bool {
// Drain the cross-thread staging (see `stagingLock`): the layout-derived drawable size and
// any freshly-arrived HDR grade, both applied from this thread.
stagingLock.lock()
let targetFromLayout = drawableTarget
let newHdrMeta = pendingHdrMeta
pendingHdrMeta = nil
stagingLock.unlock()
// Reconcile the layer with the decoded frame's HDR-ness (handles a mid-session SDR↔HDR flip).
configure(hdr: isHDR)
if let newHdrMeta {
self.lastHdrMeta = newHdrMeta
// tvOS has no edrMetadata — the cached grade is still kept (a later HDR flip's
// configureColor is where it matters there). macOS/iOS refine the live tone-map now.
#if !os(tvOS)
if hdrActive { layer.edrMetadata = makeEDR(newHdrMeta) }
#endif
}
// P010/x444 store 10-bit luma/chroma in 16-bit samples → R16/RG16; NV12/444v is 8-bit → R8/RG8.
// Derived from the actual decoded buffer so a 4:4:4 (full chroma plane) frame just works.
let pf = CVPixelBufferGetPixelFormatType(pixelBuffer)
let tenBit =
pf == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange
|| pf == kCVPixelFormatType_420YpCbCr10BiPlanarFullRange
|| pf == kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange
|| pf == kCVPixelFormatType_444YpCbCr10BiPlanarFullRange
// The frame's YCbCr→RGB rows, from its ACTUAL signaling (buffer attachments + pixel
// format) — a BT.601-signaled stream gets 601 coefficients, full-range gets full-range
// expansion; recomputed per frame because the host can flip colour in-band (SDR↔HDR).
var csc = CscRows.rows(
CscRows.signal(of: pixelBuffer), depth: tenBit ? 10 : 8, msbPacked: tenBit)
guard let textureCache,
let luma = makeTexture(
pixelBuffer, plane: 0, format: tenBit ? .r16Unorm : .r8Unorm, cache: textureCache),
let chroma = makeTexture(
pixelBuffer, plane: 1, format: tenBit ? .rg16Unorm : .rg8Unorm, cache: textureCache)
else { return false }
#if os(tvOS)
// HDR splits by the display's headroom (kept in step with the layer by `configure` above):
// PQ passthrough into an HDR-composited display, the tone-map shader otherwise.
let hdrPipeline = hdrPassthroughActive ? pipelineHDR : (pipelineHDRToneMap ?? pipelineHDR)
let pipeline = hdrActive ? hdrPipeline : pipelineSDR
#else
let pipeline = hdrActive ? pipelineHDR : pipelineSDR
#endif
let decodedSize = CGSize(
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
return encodePresent(
decodedSize: decodedSize, targetFromLayout: targetFromLayout, pipeline: pipeline,
presentAtMediaTime: presentAtMediaTime, providedDrawable: drawable,
onPresented: onPresented,
// Hold the CVMetalTextures + source pixel buffer (its IOSurface) alive until the GPU
// finishes sampling — releasing them at scope exit could free the backing mid-read.
keepAlive: [luma, chroma, pixelBuffer]
) { encoder in
encoder.setFragmentTexture(CVMetalTextureGetTexture(luma), index: 0)
encoder.setFragmentTexture(CVMetalTextureGetTexture(chroma), index: 1)
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
}
}
/// Draw one PyroWave planar frame (three R8 planes off the Metal wavelet decoder) and
/// present it. RENDER THREAD, same contract as `render` — PyroWave is 8-bit SDR, so the
/// layer always takes the plain SDR config, and the CSC rows arrive precomputed from the
/// stream's own sequence-header signaling (no CVPixelBuffer to inspect).
@discardableResult
func renderPlanar(
_ planes: WaveletPlanes,
presentAtMediaTime: CFTimeInterval? = nil,
into drawable: CAMetalDrawable? = nil,
onPresented: ((Int64?) -> Void)? = nil
) -> Bool {
stagingLock.lock()
let targetFromLayout = drawableTarget
stagingLock.unlock()
// A PQ (HDR) pyrowave stream drives the same layer/EDR machinery as the biplanar path —
// including macOS windowed sessions, which keep real HDR (the DCP mitigation is the
// transactional present in `encodePresent`, not a colour downgrade).
configure(hdr: planes.pq)
var csc = planes.csc
// PQ passthrough needs the HDR drawable; a PQ frame while the drawable is (still)
// 8-bit — tvOS without display headroom, or a not-yet-flipped layer — tone-maps
// in-shader instead (the pipeline must match the drawable's pixel format).
#if os(tvOS)
let planarPassthrough = hdrActive && hdrPassthroughActive
#else
let planarPassthrough = hdrActive
#endif
let planarPipeline: MTLRenderPipelineState =
planes.pq
? (planarPassthrough ? pipelinePlanarHDR : pipelinePlanarToneMap)
: pipelinePlanar
return encodePresent(
decodedSize: CGSize(width: planes.width, height: planes.height),
targetFromLayout: targetFromLayout, pipeline: planarPipeline,
presentAtMediaTime: presentAtMediaTime, providedDrawable: drawable,
onPresented: onPresented,
// The ring textures stay valid by ring depth; retaining them here also pins the
// slot's set until the sample completes (mirrors the biplanar keep-alive).
keepAlive: [planes.y, planes.cb, planes.cr]
) { encoder in
encoder.setFragmentTexture(planes.y, index: 0)
encoder.setFragmentTexture(planes.cb, index: 1)
encoder.setFragmentTexture(planes.cr, index: 2)
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
}
}
/// The shared present tail of `render`/`renderPlanar`: size the drawable, encode one
/// fullscreen triangle with `pipeline` (`bind` supplies the fragment resources), schedule
/// the present and the on-glass callback.
///
/// `providedDrawable` (deadline pacing) is the CAMetalDisplayLink-vended drawable to render
/// into instead of `nextDrawable()`. It was vended against the layer's config at vend time,
/// so after a mid-session reconfigure (HDR flip: `configure` above already retagged the
/// layer) its pixel format can lag the pipeline's attachment format — encoding would be a
/// Metal validation failure. The guard returns false instead: the drawable drops back to
/// the pool, the caller re-rings the frame, and the link's next vend carries the new format.
private func encodePresent(
decodedSize: CGSize, targetFromLayout: CGSize, pipeline: MTLRenderPipelineState,
presentAtMediaTime: CFTimeInterval?, providedDrawable: CAMetalDrawable? = nil,
onPresented: ((Int64?) -> Void)?,
keepAlive: [Any], bind: (MTLRenderCommandEncoder) -> Void
) -> Bool {
// Size the drawable to the LAYER's pixels (its laid-out frame × contentsScale, pushed here by
// SessionPresenter.layout via `setDrawableTarget` — not read off the layer, whose geometry the
// main thread owns) so the Catmull-Rom shader performs the decoded→on-screen scale in one pass:
// a native-mode session stays exactly 1:1 (the kernel reduces to the identity texel), and a
// window bigger than the host's mode gets bicubic luma instead of the compositor's bilinear.
// Before the first layout (zero target) fall back to the decoded size. drawableSize does NOT
// track bounds (defaults to 0), so set it BEFORE nextDrawable; re-set only on a change
// (layout / Reconfigure / HDR flip — and every frame of a live resize, which is fine).
let targetSize = (targetFromLayout.width > 0 && targetFromLayout.height > 0)
? targetFromLayout : decodedSize
// Under a provided (link-vended) drawable this sizes the NEXT vend — the one in hand
// keeps its size, and a live-resize transient composites via contentsGravity as ever.
if layer.drawableSize != targetSize { layer.drawableSize = targetSize }
#if DEBUG
logSizeIfChanged(decoded: decodedSize, drawable: targetSize)
#endif
#if os(macOS)
// Windowed (composited) → the DCP swapID-panic mitigation mechanism (see
// `WindowedPresentMode`). Toggle the layer property BEFORE vending a drawable so the
// vend matches how it will be presented; drained here on the render thread, flipped
// exactly once per mode change.
stagingLock.lock()
let windowedMode = windowedPresentStaged
stagingLock.unlock()
if windowedMode != windowedPresentActive {
windowedPresentActive = windowedMode
layer.presentsWithTransaction = windowedMode == .transaction
if windowedMode != .surface, !surfacePool.isEmpty {
// Leaving surface mode (fullscreen entry / mechanism A/B): drop the pool — at 5K
// it holds >100 MB, and re-entering rebuilds it in one frame. SessionPresenter
// clears the surface layer's contents on main.
surfacePool.removeAll()
surfacePoolSize = .zero
lastHandedOff = nil
}
presenterLog.info(
"stage2: windowed present mode \(windowedMode.rawValue, privacy: .public) (DCP swapID-panic mitigation)")
}
if windowedMode == .surface {
// No image queue at all: render into a pooled IOSurface and swap it into the
// sibling layer's contents. The drawable/queue tail below never runs.
return encodeToSurface(
targetSize: targetSize, pipeline: pipeline, onPresented: onPresented,
keepAlive: keepAlive, bind: bind)
}
#endif
if let providedDrawable,
providedDrawable.texture.pixelFormat != layer.pixelFormat {
return false // config outran the vend (HDR flip) — next vend has the new format
}
guard let drawable = providedDrawable ?? layer.nextDrawable(),
let commandBuffer = queue.makeCommandBuffer()
else { return false }
let pass = MTLRenderPassDescriptor()
pass.colorAttachments[0].texture = drawable.texture
pass.colorAttachments[0].loadAction = .clear
pass.colorAttachments[0].clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 1)
pass.colorAttachments[0].storeAction = .store
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else {
return false
}
encoder.setRenderPipelineState(pipeline)
bind(encoder)
encoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
encoder.endEncoding()
if let onPresented {
#if targetEnvironment(simulator)
// The simulator SDK exposes neither addPresentedHandler nor presentedTime — report
// nil so the caller stamps with its display-link estimate (the pre-presentedTime
// behavior; simulator numbers are indicative only anyway).
onPresented(nil)
#else
// Registered BEFORE present. presentedTime is CACurrentMediaTime-based; 0 means the
// system never put this drawable on glass (dropped) — report nil, the caller falls
// back to its display-link estimate.
drawable.addPresentedHandler { d in
onPresented(
d.presentedTime > 0
? Stage2Pipeline.realtimeNs(forDisplayLinkTimestamp: d.presentedTime)
: nil)
}
#endif
}
// Keep the bound sources alive until the GPU finishes sampling (see the callers).
commandBuffer.addCompletedHandler { _ in _ = keepAlive }
#if os(macOS)
if windowedPresentActive == .transaction {
// Windowed DCP mitigation: present the drawable THROUGH a Core Animation transaction
// (`presentsWithTransaction`, set above) instead of the async image queue, so the swap
// commits with the layer tree and stays in lockstep with the compositor (no out-of-band
// flip to race WindowServer's swaps). Wait until the GPU work is scheduled (contents
// will be ready — p50 ~0.1 ms), then present inside an EXPLICIT CATransaction ON THIS
// RENDER THREAD and `flush()`. `presentAtMediaTime` does not apply — the transaction
// paces.
//
// Threading history, because BOTH failure modes shipped or nearly shipped:
// • A bare `present()` from this thread (no transaction) never flushes — nothing
// commits a runloop-less thread's implicit transaction, so drawables are never
// released; after maximumDrawableCount vends `nextDrawable()` blocks forever and
// the stream FREEZES (the fullscreen→windowed switch did exactly this).
// • The explicit begin/commit alone is NOT enough either: this thread has an ACTIVE
// implicit transaction (the layer mutations above — drawableSize/colour — created
// it), so the explicit transaction NESTS inside it and its commit defers to the
// implicit one that never comes. The harness reproduced the exact freeze: every
// present reported presentedTime=0, nothing reached glass. `CATransaction.flush()`
// pushes the implicit transaction (present included) to the render server NOW.
// • The original fix hopped to MAIN and presented there — correct, but slow in the
// field (presents=55 @ fps=240, display_p50 18.6 ms on the 240 Hz Studio): each
// present lands a runloop turn late, and main's own implicit transaction batches
// enrolled presents at runloop-iteration rate. Kept as PUNKTFUNK_TXN_PRESENT=main.
// The off-main commit measured immune to main-thread churn in the harness
// (2026-07-21: glass p50 ~10 ms at 240 Hz full-size, cadence a clean 4.17 ms).
commandBuffer.commit()
let schedStart = CACurrentMediaTime()
commandBuffer.waitUntilScheduled()
let schedMs = (CACurrentMediaTime() - schedStart) * 1000
let commitStart = CACurrentMediaTime()
if txnPresentOnMain {
let presentedDrawable = drawable
DispatchQueue.main.async {
CATransaction.begin()
CATransaction.setDisableActions(true)
presentedDrawable.present()
CATransaction.commit()
}
} else {
CATransaction.begin()
CATransaction.setDisableActions(true)
drawable.present()
CATransaction.commit()
CATransaction.flush()
}
windowedDiag.record(
schedMs: schedMs, commitMs: (CACurrentMediaTime() - commitStart) * 1000,
mode: .transaction)
return true
}
#endif
// Scheduled on the vsync when the pipeline gave us the link's target (see the doc comment);
// immediate otherwise. A target already in the past presents immediately — same thing.
if let presentAtMediaTime {
commandBuffer.present(drawable, atTime: presentAtMediaTime)
} else {
commandBuffer.present(drawable)
}
commandBuffer.commit()
return true
}
#if os(macOS)
/// The WINDOWED `surface` present tail (see `WindowedPresentMode.surface`): render with the
/// same per-frame pipeline into a pooled IOSurface and hand it to `surfaceLayer.contents`
/// from the command buffer's COMPLETION handler, inside an explicit CATransaction + flush
/// (the same off-main commit discipline as the transactional present — an ordinary
/// damaged-layer update on WindowServer's own composite cadence, no image queue anywhere).
/// RENDER THREAD. `onPresented` is stamped right after the contents swap commits — the
/// closest observable analogue of "reached glass" here (the composite follows within a
/// refresh, so the display-stage meters read slightly OPTIMISTIC in this mode).
///
/// The pool tracks `hdrActive`: bgra8 for SDR, rgba16Float tagged BT.2100 PQ for HDR —
/// `configure` already ran, so the caller's `pipeline` attachment format always matches.
/// HDR OPEN RISK (why this whole mode is a prototype): whether the compositor honors the
/// PQ tag + EDR for plain-CALayer IOSurface contents needs an on-glass eyeball; the metal
/// layer underneath keeps `wantsExtendedDynamicRangeContent` as the EDR anchor (the harness
/// measured the display's EDR headroom engaging with this arrangement).
private func encodeToSurface(
targetSize: CGSize, pipeline: MTLRenderPipelineState,
onPresented: ((Int64?) -> Void)?,
keepAlive: [Any], bind: (MTLRenderCommandEncoder) -> Void
) -> Bool {
ensureSurfacePool(size: targetSize, hdr: hdrActive)
guard let slotIndex = takeSurfaceSlot(),
let commandBuffer = queue.makeCommandBuffer()
else { return false }
let slot = surfacePool[slotIndex]
let pass = MTLRenderPassDescriptor()
pass.colorAttachments[0].texture = slot.texture
pass.colorAttachments[0].loadAction = .clear
pass.colorAttachments[0].clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 1)
pass.colorAttachments[0].storeAction = .store
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else {
return false
}
encoder.setRenderPipelineState(pipeline)
bind(encoder)
encoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
encoder.endEncoding()
let surface = slot.surface
let surfaceLayer = surfaceLayer // captured directly — the handler must not retain self
let diag = windowedDiag
let commitStamp = CACurrentMediaTime()
commandBuffer.addCompletedHandler { _ in
_ = keepAlive // sources pinned until the GPU finished sampling
let completedAt = CACurrentMediaTime()
// Swap on THIS Metal completion thread: explicit transaction + flush, so the commit
// reaches the render server now, independent of main (completion handlers for one
// queue fire in execution order, so swaps can't reorder).
CATransaction.begin()
CATransaction.setDisableActions(true)
surfaceLayer.contents = surface
CATransaction.commit()
CATransaction.flush()
diag.record(
schedMs: (completedAt - commitStamp) * 1000,
commitMs: (CACurrentMediaTime() - completedAt) * 1000, mode: .surface)
onPresented?(Stage2Pipeline.realtimeNs(forDisplayLinkTimestamp: CACurrentMediaTime()))
}
commandBuffer.commit()
lastHandedOff = slotIndex
return true
}
/// (Re)build the pool at `size`/`hdr` — 4 IOSurface render targets (one on glass, one
/// committed in CA, one rendering, one spare). RENDER THREAD. A failed allocation leaves the
/// pool empty; the caller returns false and the ring's putBack + display-link retry take
/// over.
private func ensureSurfacePool(size: CGSize, hdr: Bool) {
guard size != surfacePoolSize || hdr != surfacePoolHDR else { return }
surfacePool.removeAll()
surfacePoolSize = size
surfacePoolHDR = hdr
lastHandedOff = nil
let w = Int(size.width)
let h = Int(size.height)
guard w > 0, h > 0 else { return }
// rgba16Float (8 B/px) carries the PQ-encoded HDR samples; bgra8 the SDR ones. 256-byte
// row alignment satisfies both IOSurface and Metal linear-texture rules.
let bytesPerElement = hdr ? 8 : 4
let bytesPerRow = ((w * bytesPerElement) + 255) & ~255
let props: [String: Any] = [
kIOSurfaceWidth as String: w,
kIOSurfaceHeight as String: h,
kIOSurfaceBytesPerElement as String: bytesPerElement,
kIOSurfaceBytesPerRow as String: bytesPerRow,
kIOSurfacePixelFormat as String: hdr
? kCVPixelFormatType_64RGBAHalf : kCVPixelFormatType_32BGRA,
]
let desc = MTLTextureDescriptor.texture2DDescriptor(
pixelFormat: hdr ? .rgba16Float : .bgra8Unorm, width: w, height: h, mipmapped: false)
desc.usage = [.renderTarget]
desc.storageMode = .shared
for _ in 0..<4 {
guard let surface = IOSurfaceCreate(props as CFDictionary),
let texture = device.makeTexture(descriptor: desc, iosurface: surface, plane: 0)
else {
surfacePool.removeAll()
return
}
if hdr, let name = CGColorSpace(name: CGColorSpace.itur_2100_PQ)?.name {
// Tag the surface BT.2100 PQ so the compositor interprets the half-float
// samples as PQ-encoded HDR (the CALayer-contents analogue of the metal
// layer's colorspace).
IOSurfaceSetValue(surface, "IOSurfaceColorSpace" as CFString, name)
}
surfacePool.append(SurfaceSlot(surface: surface, texture: texture))
}
// The EDR request rides the SURFACE layer too (its contents are what composite); the
// metal layer underneath keeps its own from configureColor as the anchor. Layer flags
// are committed by the next swap's transaction flush.
surfaceLayer.wantsExtendedDynamicRangeContent = hdr
}
/// Pick the slot to render into: never the one just handed to the layer (the compositor may
/// still scan it), prefer surfaces the window server isn't holding (`IOSurfaceIsInUse`), and
/// among those the least recently rendered. Falls back to the LRU busy slot rather than
/// stalling — a visible glitch at worst, never a queue-up. RENDER THREAD.
private func takeSurfaceSlot() -> Int? {
guard !surfacePool.isEmpty else { return nil }
var free: Int?
var busy: Int?
for i in surfacePool.indices where i != lastHandedOff {
if !IOSurfaceIsInUse(surfacePool[i].surface) {
if free == nil || surfacePool[i].seq < surfacePool[free!].seq { free = i }
} else {
if busy == nil || surfacePool[i].seq < surfacePool[busy!].seq { busy = i }
}
}
guard let pick = free ?? busy else { return nil }
surfaceSeq += 1
surfacePool[pick].seq = surfaceSeq
return pick
}
#endif
/// Returns the CVMetalTexture (not just its MTLTexture) so the caller can keep it alive past the
/// draw — the MTLTexture is only valid while its CVMetalTexture is retained.
private func makeTexture(
_ pixelBuffer: CVPixelBuffer, plane: Int, format: MTLPixelFormat, cache: CVMetalTextureCache
) -> CVMetalTexture? {
let w = CVPixelBufferGetWidthOfPlane(pixelBuffer, plane)
let h = CVPixelBufferGetHeightOfPlane(pixelBuffer, plane)
var cvTexture: CVMetalTexture?
let status = CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault, cache, pixelBuffer, nil, format, w, h, plane, &cvTexture)
guard status == kCVReturnSuccess, let cvTexture,
CVMetalTextureGetTexture(cvTexture) != nil
else { return nil }
return cvTexture
}
#if DEBUG
private func logSizeIfChanged(decoded: CGSize, drawable: CGSize) {
let sig = "\(Int(decoded.width))x\(Int(decoded.height))\(Int(drawable.width))x\(Int(drawable.height))|hdr\(hdrActive ? 1 : 0)"
if sig != lastSizeSig {
lastSizeSig = sig
// Explicit verdict: is the shader presenting 1:1 (decoded == drawable) or resampling? The
// scale ratio makes a residual match-window mismatch obvious. If this says 1:1 but the
// picture is still soft, the resample is downstream of us (macOS compositor — a scaled
// display mode, or a fractional-pixel window position), not the shader.
let sx = decoded.width > 0 ? drawable.width / decoded.width : 0
let sy = decoded.height > 0 ? drawable.height / decoded.height : 0
let verdict = decoded == drawable
? "1:1 (no resample)"
: String(format: "RESAMPLE scale=%.4fx%.4f", sx, sy)
let msg =
"stage2: decoded \(Int(decoded.width))x\(Int(decoded.height)) → drawable \(Int(drawable.width))x\(Int(drawable.height)) [\(verdict)] hdr=\(hdrActive)"
presenterLog.info("\(msg, privacy: .public)")
}
}
#endif
}
#endif