feat(clients/apple): cross-client shortcuts + start banner, opt-in V-Sync, presenter rework
apple / swift (push) Failing after 28s
release / apple (push) Failing after 22s
apple / screenshots (push) Has been skipped
android / android (push) Has been cancelled
arch / build-publish (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
apple / swift (push) Failing after 28s
release / apple (push) Failing after 22s
apple / screenshots (push) Has been skipped
android / android (push) Has been cancelled
arch / build-publish (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
Align the macOS/iPad Stream menu on the cross-client Ctrl+Alt+Shift set the Windows and Linux clients reserve — Release Mouse (⌃⌥⇧Q), Disconnect (⌃⌥⇧D), HUD toggle (⌃⌥⇧S) — with ⌘⎋ kept as the macOS/iPad capture toggle, and surface them on a 6-second banner at stream start. Add an opt-in V-Sync present mode (punktfunk.vsync, default OFF = lowest-latency immediate present; PUNKTFUNK_PRESENT_MODE overrides for A/B), with the presenter reworked to a frame-arrival-triggered render thread across Stage2Pipeline / MetalVideoPresenter / SessionPresenter, plus the windowed title-bar safe-area handling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,12 +1,15 @@
|
||||
// Stage-2 presenter, present half: draw a decoded NV12 / P010 / 4:4:4 CVPixelBuffer into a CAMetalLayer
|
||||
// drawable with a Y′CbCr→RGB shader. The hosting view's CADisplayLink drives `render` once per vsync
|
||||
// (via Stage2Pipeline.renderTick) with the target present time, so a present can be stamped and the
|
||||
// present tail hand-paced. See docs apple-stage2-presenter.md.
|
||||
// drawable with a Y′CbCr→RGB shader. The hosting view's CADisplayLink still paces the pipeline once per
|
||||
// vsync, but the actual `render` runs on Stage2Pipeline's dedicated RENDER THREAD (the link tick just
|
||||
// signals it), so `nextDrawable()`'s blocking never lands on the main thread. See docs
|
||||
// apple-stage2-presenter.md.
|
||||
//
|
||||
// Main-thread only: created during view setup, `render`/`configure` called from the view's CADisplayLink
|
||||
// (which fires on the main runloop). The Metal objects + texture cache are touched only here. The one
|
||||
// exception is `setHdrMeta`, called from the pump thread — it hops the layer write to main so every
|
||||
// CALayer mutation stays on one thread.
|
||||
// Threading: created during view setup (main); `render`/`configure` run on the render thread — the
|
||||
// layer's drawable/format/colour mutations all happen there (CAMetalLayer is designed for dedicated
|
||||
// render threads; only the layer's GEOMETRY — frame/contentsScale — is touched from main, in
|
||||
// SessionPresenter.layout, which also pushes the resulting pixel size here via `setDrawableTarget`
|
||||
// so the render thread never reads layer geometry cross-thread). `setHdrMeta` (pump thread) and
|
||||
// `setDrawableTarget` (main) only write lock-guarded staging state the render thread drains.
|
||||
|
||||
#if canImport(Metal) && canImport(QuartzCore)
|
||||
import CoreGraphics
|
||||
@@ -144,14 +147,23 @@ public final class MetalVideoPresenter {
|
||||
private var textureCache: CVMetalTextureCache?
|
||||
|
||||
/// Current layer configuration — switched in `configure(hdr:)` when a frame's HDR-ness differs.
|
||||
/// Main-thread only (read + written from `render`/`configure`, all on the display-link runloop).
|
||||
/// Render-thread confined once the pipeline runs (Stage2Pipeline.start's one pre-thread
|
||||
/// `configure` call is ordered before the thread starts, so it doesn't race).
|
||||
private var hdrActive = false
|
||||
/// Last HDR mastering grade received via `setHdrMeta` (the host's 0xCE). Cached so a mid-session
|
||||
/// SDR→HDR flip's `configureColor` re-applies the real grade instead of clobbering it back to the
|
||||
/// bare reference-white anchor (an out-of-order race otherwise: `setHdrMeta` and the flip both write
|
||||
/// `edrMetadata`). Main-thread only.
|
||||
/// `edrMetadata`). Render-thread confined (drained from `pendingHdrMeta` at the top of `render`).
|
||||
private var lastHdrMeta: PunktfunkConnection.HdrMeta?
|
||||
|
||||
/// Cross-thread staging, all under `stagingLock`: the pump thread parks a fresh 0xCE grade in
|
||||
/// `pendingHdrMeta` and the main thread parks the layout-derived drawable pixel size in
|
||||
/// `drawableTarget`; the render thread drains both at the top of `render`, so every layer
|
||||
/// format/colour mutation stays on the one thread that also calls `nextDrawable()`.
|
||||
private let stagingLock = NSLock()
|
||||
private var pendingHdrMeta: PunktfunkConnection.HdrMeta?
|
||||
private var drawableTarget: CGSize = .zero
|
||||
|
||||
#if DEBUG
|
||||
/// Last logged "decoded→drawable" signature, so the diagnostic logs only on a size/HDR change.
|
||||
private var lastSizeSig = ""
|
||||
@@ -191,12 +203,18 @@ public final class MetalVideoPresenter {
|
||||
layer.framebufferOnly = true
|
||||
layer.isOpaque = true
|
||||
#if os(macOS)
|
||||
// The display link already paces exactly one present per vsync. Leaving the layer's own vsync
|
||||
// wait on means `commandBuffer.present` ALSO blocks for the hardware vsync, so `nextDrawable()`
|
||||
// stalls the MAIN thread until a drawable frees — windowed, the WindowServer's looser
|
||||
// compositing hides it; FULLSCREEN's tighter path serializes the main thread to the display and
|
||||
// the stall surfaces as bad judder. Disabling the layer-level sync lets present return promptly
|
||||
// (the display link is the pacing source) — the fix for the fullscreen stutter. macOS-only.
|
||||
// displaySyncEnabled MUST stay false on macOS. It has flip-flopped, so the full history:
|
||||
// sync ON was tried twice and starves the drawable pool both times — on macOS 26 a synced
|
||||
// present only reaches glass when the WindowServer composites the window, and its FramePacing
|
||||
// path does not treat our out-of-band image-queue presents as damage, so with a static scene
|
||||
// the ONLY recurring damage is the 1 Hz stats HUD update: presents queue, all drawables stay
|
||||
// held, `nextDrawable()` sleeps (sampled: ~70% of the render thread inside
|
||||
// CAMetalLayerPrivateNextDrawableLocked → usleep), and the stream turns into a ~1 fps
|
||||
// slideshow with normal-LOOKING stats (each rare frame is fresh, newest-wins ring). The
|
||||
// 94fb7d1b fullscreen judder was the same starvation biting the then-main-thread render.
|
||||
// With sync OFF the flip is immediate; the vsync alignment that sync was supposed to give
|
||||
// (the HUD-off direct-scanout pacing fix) comes from scheduling the present at the display
|
||||
// link's target time instead (`present(at:)` — see `render`).
|
||||
layer.displaySyncEnabled = false
|
||||
#endif
|
||||
// The drawable is rendered at the LAYER's pixel size (set per-frame in `render`), so the
|
||||
@@ -226,11 +244,12 @@ public final class MetalVideoPresenter {
|
||||
self.layer = layer
|
||||
}
|
||||
|
||||
/// Configure the layer + active pipeline for an SDR or HDR session. MAIN THREAD ONLY. Called once at
|
||||
/// session start and again per-frame from `render` (idempotent — the guard makes a same-state call a
|
||||
/// no-op), so a mid-session HDR toggle (the host re-inits its encoder; the decoded `frame.isHDR`
|
||||
/// flips) reconfigures here automatically. HDR uses an rgba16Float drawable + BT.2020 PQ colour space
|
||||
/// + EDR with a 203-nit reference-white anchor; SDR uses the plain 8-bit sRGB path.
|
||||
/// Configure the layer + active pipeline for an SDR or HDR session. Called once at session start
|
||||
/// (main, before the render thread exists) and again per-frame from `render` on the RENDER THREAD
|
||||
/// (idempotent — the guard makes a same-state call a no-op), so a mid-session HDR toggle (the host
|
||||
/// re-inits its encoder; the decoded `frame.isHDR` flips) reconfigures here automatically. HDR uses
|
||||
/// an rgba16Float drawable + BT.2020 PQ colour space + EDR with a 203-nit reference-white anchor;
|
||||
/// SDR uses the plain 8-bit sRGB path.
|
||||
public func configure(hdr: Bool) {
|
||||
guard hdr != hdrActive else { return }
|
||||
hdrActive = hdr
|
||||
@@ -273,36 +292,65 @@ public final class MetalVideoPresenter {
|
||||
#endif
|
||||
|
||||
/// Update the HDR mastering metadata (drained from the host's 0xCE datagram) to refine the system
|
||||
/// tone-map from the real grade. Called from the PUMP thread, so the layer write is hopped to MAIN
|
||||
/// (every CALayer mutation stays on one thread). The grade is cached so a later SDR→HDR
|
||||
/// `configureColor` re-applies it; the `edrMetadata` write is gated on `hdrActive` (setting it on an
|
||||
/// SDR layer is harmless but pointless, and the flip will apply it anyway).
|
||||
/// tone-map from the real grade. Called from the PUMP thread — the grade is only PARKED here (lock-
|
||||
/// guarded); the render thread applies it at the top of the next `render`, keeping every layer
|
||||
/// colour mutation on the one thread that also vends drawables.
|
||||
public func setHdrMeta(_ meta: PunktfunkConnection.HdrMeta) {
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
guard let self else { return }
|
||||
self.lastHdrMeta = meta
|
||||
// tvOS has no edrMetadata — the cached grade is still kept above (harmless), it just can't
|
||||
// be applied to the layer there. macOS/iOS refine the system tone-map from the real grade.
|
||||
#if !os(tvOS)
|
||||
if self.hdrActive { self.layer.edrMetadata = self.makeEDR(meta) }
|
||||
#endif
|
||||
}
|
||||
stagingLock.lock()
|
||||
pendingHdrMeta = meta
|
||||
stagingLock.unlock()
|
||||
}
|
||||
|
||||
/// Draw one decoded frame to the next drawable and present it. MAIN THREAD (the display link).
|
||||
/// `isHDR` selects the 10-bit BT.2020 PQ path vs the 8-bit BT.709 path and is reconciled with the
|
||||
/// Park the drawable pixel size the shader should render at: the metal layer's laid-out frame ×
|
||||
/// contentsScale, both owned by the MAIN thread (SessionPresenter.layout pushes it on every layout/
|
||||
/// backing change). The render thread reads this instead of the layer's geometry so it never
|
||||
/// touches main-owned CALayer state. Zero until the first layout → `render` falls back to the
|
||||
/// decoded frame size.
|
||||
public func setDrawableTarget(_ size: CGSize) {
|
||||
stagingLock.lock()
|
||||
drawableTarget = size
|
||||
stagingLock.unlock()
|
||||
}
|
||||
|
||||
/// Draw one decoded frame to the next drawable and present it. RENDER THREAD (Stage2Pipeline's;
|
||||
/// `nextDrawable()` may block up to a frame — that wait belongs here, never on main). `isHDR`
|
||||
/// selects the 10-bit BT.2020 PQ path vs the 8-bit BT.709 path and is reconciled with the
|
||||
/// layer config via `configure`. Returns true on success; false when there's no drawable yet, a
|
||||
/// texture couldn't be made, or Metal errored — the caller then doesn't stamp a present (and can
|
||||
/// requeue the frame). `onPresented` fires once the drawable actually reached glass, with the
|
||||
/// `CLOCK_REALTIME` instant from the drawable's `presentedTime` — or nil when the system reports
|
||||
/// none (a dropped drawable). It runs on a Metal callback thread; keep the handler thread-safe.
|
||||
///
|
||||
/// `presentAtMediaTime` (a `CACurrentMediaTime`-basis host time — the display link's
|
||||
/// `targetTimestamp`) schedules the flip ON the vsync instead of "as soon as the GPU finishes":
|
||||
/// with the layer's own sync disabled (mandatory on macOS — see init) an immediate present hits
|
||||
/// glass mid-refresh whenever the layer is direct-scanout promoted (fullscreen, no HUD), which
|
||||
/// is the "frametimes are off with the stats HUD closed" report. nil presents immediately
|
||||
/// (`PUNKTFUNK_PRESENT_MODE=immediate` — the pre-fix behavior, kept as a diagnostic A/B).
|
||||
@discardableResult
|
||||
public func render(
|
||||
_ pixelBuffer: CVPixelBuffer, isHDR: Bool = false,
|
||||
presentAtMediaTime: CFTimeInterval? = nil,
|
||||
onPresented: ((Int64?) -> Void)? = nil
|
||||
) -> Bool {
|
||||
// Drain the cross-thread staging (see `stagingLock`): the layout-derived drawable size and
|
||||
// any freshly-arrived HDR grade, both applied from this thread.
|
||||
stagingLock.lock()
|
||||
let targetFromLayout = drawableTarget
|
||||
let newHdrMeta = pendingHdrMeta
|
||||
pendingHdrMeta = nil
|
||||
stagingLock.unlock()
|
||||
|
||||
// Reconcile the layer with the decoded frame's HDR-ness (handles a mid-session SDR↔HDR flip).
|
||||
configure(hdr: isHDR)
|
||||
if let newHdrMeta {
|
||||
self.lastHdrMeta = newHdrMeta
|
||||
// tvOS has no edrMetadata — the cached grade is still kept (a later HDR flip's
|
||||
// configureColor is where it matters there). macOS/iOS refine the live tone-map now.
|
||||
#if !os(tvOS)
|
||||
if hdrActive { layer.edrMetadata = makeEDR(newHdrMeta) }
|
||||
#endif
|
||||
}
|
||||
|
||||
// P010/x444 store 10-bit luma/chroma in 16-bit samples → R16/RG16; NV12/444v is 8-bit → R8/RG8.
|
||||
// Derived from the actual decoded buffer so a 4:4:4 (full chroma plane) frame just works.
|
||||
@@ -319,22 +367,18 @@ public final class MetalVideoPresenter {
|
||||
pixelBuffer, plane: 1, format: tenBit ? .rg16Unorm : .rg8Unorm, cache: textureCache)
|
||||
else { return false }
|
||||
|
||||
// Size the drawable to the LAYER's pixels (bounds × contentsScale, both set by the hosting
|
||||
// view's layout) so the Catmull-Rom shader performs the decoded→on-screen scale in one pass:
|
||||
// Size the drawable to the LAYER's pixels (its laid-out frame × contentsScale, pushed here by
|
||||
// SessionPresenter.layout via `setDrawableTarget` — not read off the layer, whose geometry the
|
||||
// main thread owns) so the Catmull-Rom shader performs the decoded→on-screen scale in one pass:
|
||||
// a native-mode session stays exactly 1:1 (the kernel reduces to the identity texel), and a
|
||||
// window bigger than the host's mode gets bicubic luma instead of the compositor's bilinear.
|
||||
// Before the first layout (empty bounds) fall back to the decoded size. drawableSize does NOT
|
||||
// Before the first layout (zero target) fall back to the decoded size. drawableSize does NOT
|
||||
// track bounds (defaults to 0), so set it BEFORE nextDrawable; re-set only on a change
|
||||
// (layout / Reconfigure / HDR flip — and every frame of a live resize, which is fine).
|
||||
let decodedSize = CGSize(
|
||||
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
|
||||
let scale = layer.contentsScale
|
||||
let boundsSize = layer.bounds.size
|
||||
let targetSize = (boundsSize.width > 0 && boundsSize.height > 0)
|
||||
? CGSize(
|
||||
width: (boundsSize.width * scale).rounded(),
|
||||
height: (boundsSize.height * scale).rounded())
|
||||
: decodedSize
|
||||
let targetSize = (targetFromLayout.width > 0 && targetFromLayout.height > 0)
|
||||
? targetFromLayout : decodedSize
|
||||
if layer.drawableSize != targetSize { layer.drawableSize = targetSize }
|
||||
#if DEBUG
|
||||
logSizeIfChanged(decoded: decodedSize, drawable: targetSize)
|
||||
@@ -374,7 +418,13 @@ public final class MetalVideoPresenter {
|
||||
}
|
||||
#endif
|
||||
}
|
||||
commandBuffer.present(drawable) // present at the next vsync — lowest latency
|
||||
// Scheduled on the vsync when the pipeline gave us the link's target (see the doc comment);
|
||||
// immediate otherwise. A target already in the past presents immediately — same thing.
|
||||
if let presentAtMediaTime {
|
||||
commandBuffer.present(drawable, atTime: presentAtMediaTime)
|
||||
} else {
|
||||
commandBuffer.present(drawable)
|
||||
}
|
||||
// Hold the CVMetalTextures + source pixel buffer (its IOSurface) alive until the GPU finishes
|
||||
// sampling — releasing them at scope exit could free the backing mid-read.
|
||||
commandBuffer.addCompletedHandler { _ in _ = (luma, chroma, pixelBuffer) }
|
||||
|
||||
@@ -68,10 +68,13 @@ final class SessionPresenter {
|
||||
baseLayer.addSublayer(metal)
|
||||
metalLayer = metal
|
||||
stage2 = pipeline
|
||||
// The link is the vsync CLOCK + putBack-retry nudge, not the presentation trigger
|
||||
// (frame arrival is — see Stage2Pipeline's header). timestamp→targetTimestamp is the
|
||||
// link's own report of the current refresh period (tracks VRR rate changes).
|
||||
let proxy = DisplayLinkProxy { [weak self] link in
|
||||
self?.stage2?.renderTick(
|
||||
targetPresentNs: Stage2Pipeline.realtimeNs(
|
||||
forDisplayLinkTimestamp: link.targetTimestamp))
|
||||
targetMediaTime: link.targetTimestamp,
|
||||
period: link.targetTimestamp - link.timestamp)
|
||||
}
|
||||
let link = makeDisplayLink(proxy, #selector(DisplayLinkProxy.tick(_:)))
|
||||
link.add(to: .main, forMode: .common)
|
||||
@@ -127,6 +130,11 @@ final class SessionPresenter {
|
||||
metalLayer.contentsScale = contentsScale
|
||||
metalLayer.frame = fit
|
||||
CATransaction.commit()
|
||||
// Hand the resulting pixel size to the render thread (it must not read layer geometry
|
||||
// cross-thread) — this is what the presenter sizes its drawable to.
|
||||
stage2?.setDrawableTarget(CGSize(
|
||||
width: (fit.width * contentsScale).rounded(),
|
||||
height: (fit.height * contentsScale).rounded()))
|
||||
}
|
||||
|
||||
/// Stop the active pump/pipeline (≤ one poll timeout; stage-2 joins its pump) and detach the
|
||||
|
||||
@@ -1,25 +1,60 @@
|
||||
// Stage-2 presenter orchestrator: a pump thread pulls AUs → VideoDecoder; the decoder's async output
|
||||
// drops the newest decoded frame into a 1-slot ring; the hosting view's display link calls `renderTick`
|
||||
// once per vsync to draw + present the newest ready frame and stamp the unified latency stages
|
||||
// (end-to-end capture→on-glass, plus the decode and display stage terms —
|
||||
// design/stats-unification.md). Mirrors StreamPump's lifecycle (one per start; cancel is permanent).
|
||||
// Stage-2 presenter orchestrator. GOAL ARCHITECTURE (the result of the 2026-07 pacing saga —
|
||||
// read this before touching presentation):
|
||||
//
|
||||
// Threading: the pump runs on its own thread; the decoder callback on a VT thread; `renderTick` +
|
||||
// `start`/`stop` on the MAIN thread (the view's CADisplayLink fires there). Only the ring (lock-guarded)
|
||||
// and the decoder/presenter (internally locked / main-hopped) cross threads.
|
||||
// net pump ──► VideoDecoder (VT async) ──► newest-wins 1-slot ring ──► RENDER THREAD ──► CAMetalLayer
|
||||
//
|
||||
// • The render thread is woken by FRAME ARRIVAL (the decoder callback signals it), never gated on
|
||||
// the display link: on macOS the WindowServer's damage tracking / FramePacing does not count our
|
||||
// out-of-band presents, so anything display-link-gated stalls exactly when the rest of the screen
|
||||
// goes quiet (adaptive-sync displays idle the link down). A decoded frame is always presented
|
||||
// promptly. The display link remains only as (a) a vsync CLOCK (phase + period, for the opt-in
|
||||
// V-Sync policy below), (b) a retry tick for a frame that couldn't get a drawable (`putBack`),
|
||||
// and (c) the iOS ProMotion rate hint.
|
||||
// • The layer's own displaySyncEnabled stays FALSE on macOS — synced presents starve the drawable
|
||||
// pool outright (see MetalVideoPresenter's init for the post-mortem).
|
||||
// • Present policy is a USER SETTING (DefaultsKey.vsync; PUNKTFUNK_PRESENT_MODE=immediate|vsync
|
||||
// overrides it for A/B), resolved once per session in start():
|
||||
// – V-Sync OFF (default): present immediately — lowest latency, the long-proven behavior.
|
||||
// – V-Sync ON: present(at: next vsync) predicted from the link's last phase/period, at most one
|
||||
// period ahead by construction, falling back to immediate when the link data is stale — a
|
||||
// schedule can never sit far in the future holding drawables hostage.
|
||||
// • Rendering lives on its own thread so any `nextDrawable()` wait lands off-main (input, SwiftUI).
|
||||
//
|
||||
// The render thread also stamps the unified latency stages (end-to-end capture→on-glass + decode and
|
||||
// display stage terms — design/stats-unification.md). Mirrors StreamPump's lifecycle (one per start;
|
||||
// cancel is permanent). PUNKTFUNK_PRESENT_DEBUG=1 prints per-second pacing stats (see
|
||||
// PresentDebugStats).
|
||||
//
|
||||
// Threading: the pump runs on its own thread; the decoder callback on a VT thread; the render loop on
|
||||
// the render thread; `renderTick` + `start`/`stop` on the MAIN thread (the view's CADisplayLink fires
|
||||
// there). Only the ring (lock-guarded), the vsync clock (lock-guarded), and the decoder/presenter
|
||||
// (internally locked / staged) cross threads.
|
||||
|
||||
#if canImport(Metal) && canImport(QuartzCore)
|
||||
import AVFoundation
|
||||
import Foundation
|
||||
import QuartzCore
|
||||
|
||||
/// PUNKTFUNK_PRESENT_DEBUG=1: the render thread prints a once-per-second line with the decode
|
||||
/// (ring-submit) rate, present rate, failed/empty wakes and the slowest render call — for
|
||||
/// diagnosing pacing regressions without instruments. Plain print: the unbundled CLI client's
|
||||
/// stdout is the cheapest reliable capture channel.
|
||||
let presentDebug = ProcessInfo.processInfo.environment["PUNKTFUNK_PRESENT_DEBUG"] == "1"
|
||||
|
||||
/// Newest-ready 1-slot ring: the decoder overwrites (drops the older undisplayed frame — lowest
|
||||
/// latency, no smoothing buffer), the display link takes-and-clears. Sendable; lock-guarded.
|
||||
private final class ReadyRing: @unchecked Sendable {
|
||||
private let lock = NSLock()
|
||||
private var frame: ReadyFrame?
|
||||
/// Ring submissions since the last `drainSubmitted` — the decode rate for the
|
||||
/// PUNKTFUNK_PRESENT_DEBUG stat line.
|
||||
private var submitted = 0
|
||||
func submit(_ f: ReadyFrame) {
|
||||
lock.lock(); frame = f; lock.unlock()
|
||||
lock.lock(); frame = f; submitted += 1; lock.unlock()
|
||||
}
|
||||
func drainSubmitted() -> Int {
|
||||
lock.lock(); defer { lock.unlock() }
|
||||
let n = submitted; submitted = 0; return n
|
||||
}
|
||||
func take() -> ReadyFrame? {
|
||||
lock.lock(); defer { lock.unlock() }
|
||||
@@ -37,6 +72,86 @@ private final class ReadyRing: @unchecked Sendable {
|
||||
}
|
||||
}
|
||||
|
||||
/// The display's vsync grid as last reported by the display link (target timestamp + period,
|
||||
/// `CACurrentMediaTime` basis), written on main by `renderTick`, read by the render thread to
|
||||
/// schedule V-Sync-mode presents. A shared box (like `ReadyRing`) so neither thread captures the
|
||||
/// pipeline itself. Sendable; lock-guarded.
|
||||
private final class VsyncClock: @unchecked Sendable {
|
||||
private let lock = NSLock()
|
||||
private var target: CFTimeInterval = 0
|
||||
private var period: CFTimeInterval = 0
|
||||
|
||||
func set(target t: CFTimeInterval, period p: CFTimeInterval) {
|
||||
lock.lock(); target = t; period = p; lock.unlock()
|
||||
}
|
||||
|
||||
/// The next vsync at or after `now`, extrapolated from the last reported phase/period — by
|
||||
/// construction less than one period ahead, so a scheduled present can never sit far in the
|
||||
/// future holding its drawable. nil (⇒ present immediately) when the link has reported nothing
|
||||
/// yet, its period is nonsense, or its data is STALE (an idle/suspended link on an
|
||||
/// adaptive-sync display — exactly the case where scheduling onto its grid stalls the stream).
|
||||
func nextVsync(after now: CFTimeInterval) -> CFTimeInterval? {
|
||||
lock.lock(); defer { lock.unlock() }
|
||||
guard period > 0.0005, target > 0, now - target < 0.25 else { return nil }
|
||||
if target >= now { return target }
|
||||
return target + ceil((now - target) / period) * period
|
||||
}
|
||||
}
|
||||
|
||||
/// PUNKTFUNK_PRESENT_DEBUG=1 aggregation: one printed line per second from the render thread with
|
||||
/// the decode rate, render outcomes, the slowest render call (≈ nextDrawable wait) and the deltas
|
||||
/// between system-reported on-glass times (vsync-aligned presents show clean refresh-period
|
||||
/// multiples; immediate flips scatter). Lock-guarded — `presented` lands on a Metal callback thread.
|
||||
private final class PresentDebugStats: @unchecked Sendable {
|
||||
private let lock = NSLock()
|
||||
private var last = CACurrentMediaTime()
|
||||
private var ok = 0, failed = 0, empty = 0, dropped = 0
|
||||
private var maxRenderMs = 0.0
|
||||
private var lastGlassNs: Int64 = 0
|
||||
private var glassDeltasMs: [Double] = []
|
||||
|
||||
func emptyWake() { lock.lock(); empty += 1; lock.unlock() }
|
||||
|
||||
func renderReturned(ok rendered: Bool, tookMs: Double) {
|
||||
lock.lock()
|
||||
if rendered { ok += 1 } else { failed += 1 }
|
||||
maxRenderMs = max(maxRenderMs, tookMs)
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
func presented(atNs: Int64?) {
|
||||
lock.lock()
|
||||
if let atNs {
|
||||
if lastGlassNs > 0 { glassDeltasMs.append(Double(atNs - lastGlassNs) / 1e6) }
|
||||
lastGlassNs = atNs
|
||||
} else {
|
||||
dropped += 1
|
||||
}
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
func flushIfDue(ring: ReadyRing) {
|
||||
lock.lock()
|
||||
let now = CACurrentMediaTime()
|
||||
guard now - last >= 1 else { lock.unlock(); return }
|
||||
last = now
|
||||
let decoded = ring.drainSubmitted()
|
||||
let deltas = glassDeltasMs.sorted()
|
||||
let p50 = deltas.isEmpty ? 0 : deltas[deltas.count / 2]
|
||||
let dMax = deltas.last ?? 0
|
||||
let line = String(
|
||||
format: "pf-present decoded=%d ok=%d fail=%d empty=%d dropped=%d "
|
||||
+ "maxRenderMs=%.1f glassDeltaMs p50=%.2f max=%.2f n=%d",
|
||||
decoded, ok, failed, empty, dropped, maxRenderMs, p50, dMax, deltas.count)
|
||||
ok = 0; failed = 0; empty = 0; dropped = 0
|
||||
maxRenderMs = 0
|
||||
glassDeltasMs.removeAll(keepingCapacity: true)
|
||||
lock.unlock()
|
||||
print(line)
|
||||
fflush(stdout) // stdout is a pipe when captured — flush per line or nothing shows
|
||||
}
|
||||
}
|
||||
|
||||
public final class Stage2Pipeline {
|
||||
private let ring = ReadyRing()
|
||||
private let presenter: MetalVideoPresenter
|
||||
@@ -54,6 +169,19 @@ public final class Stage2Pipeline {
|
||||
private let pumpStopped = DispatchSemaphore(value: 0)
|
||||
private var pumpJoinable = false
|
||||
|
||||
/// Render-thread plumbing. `renderSignal` wakes the render thread — signalled by the DECODER
|
||||
/// callback on every frame (the primary trigger: presentation must never be gated on the
|
||||
/// display link, see the header) and by each display-link tick (the `putBack` retry + the
|
||||
/// vsync-clock refresh). Signals coalesce harmlessly (an extra wake finds an empty ring and
|
||||
/// goes back to sleep). `vsyncClock` is the link's last phase/period for V-Sync-mode
|
||||
/// scheduling. Lock-guarded boxes — the render thread, like the pump thread, must not capture
|
||||
/// `self`, or a missed stop() would leak a spinning pipeline. `renderStopped`/`renderJoinable`
|
||||
/// mirror the pump's bounded join.
|
||||
private let renderSignal = DispatchSemaphore(value: 0)
|
||||
private let vsyncClock = VsyncClock()
|
||||
private let renderStopped = DispatchSemaphore(value: 0)
|
||||
private var renderJoinable = false
|
||||
|
||||
/// The Metal layer the hosting view installs + sizes.
|
||||
public var layer: CAMetalLayer { presenter.layer }
|
||||
|
||||
@@ -74,6 +202,7 @@ public final class Stage2Pipeline {
|
||||
self.displayMeter = displayMeter
|
||||
let ring = ring
|
||||
let recovery = recovery
|
||||
let renderSignal = renderSignal
|
||||
self.decoder = VideoDecoder(
|
||||
onDecoded: { frame in
|
||||
// Decode stage = received→decoded, both client CLOCK_REALTIME (offset 0 — no
|
||||
@@ -82,6 +211,8 @@ public final class Stage2Pipeline {
|
||||
decodeMeter?.record(
|
||||
ptsNs: UInt64(frame.receivedNs), atNs: frame.decodedNs, offsetNs: 0)
|
||||
ring.submit(frame)
|
||||
// FRAME ARRIVAL is the render trigger (never the display link — see the header).
|
||||
renderSignal.signal()
|
||||
},
|
||||
// Async decode failure (a bad P-frame referencing a lost/corrupt IDR): the pump resets to
|
||||
// re-gate on the next IDR, and we ask the host to send one now (infinite GOP — it wouldn't
|
||||
@@ -176,34 +307,89 @@ public final class Stage2Pipeline {
|
||||
thread.qualityOfService = .userInteractive
|
||||
pumpJoinable = true
|
||||
thread.start()
|
||||
}
|
||||
|
||||
/// MAIN thread, once per vsync. Present the newest ready frame (if any). The latency stamps
|
||||
/// use the drawable's ACTUAL on-glass instant (`addPresentedHandler`/`presentedTime` — the
|
||||
/// handler fires on a Metal callback thread; the meters are thread-safe), falling back to
|
||||
/// `targetPresentNs` — the display link's target present instant, already converted to
|
||||
/// `CLOCK_REALTIME` (see `realtimeNs(forDisplayLinkTimestamp:)`) — when the system reports
|
||||
/// no presented time (a dropped drawable). A frame that could not be rendered (no drawable
|
||||
/// yet) goes back into the ring so the next tick retries it.
|
||||
public func renderTick(targetPresentNs: Int64) {
|
||||
guard let frame = ring.take() else { return }
|
||||
let offsetNs = offsetNs
|
||||
// The render thread: one present per display-link signal. It owns every layer format/colour/
|
||||
// drawable interaction (see MetalVideoPresenter's threading notes); with displaySyncEnabled on,
|
||||
// nextDrawable's up-to-a-frame wait lands here instead of on main. The 100 ms timed wait is
|
||||
// only the stop-flag poll for a session whose link stopped ticking.
|
||||
let ring = ring
|
||||
let endToEndMeter = endToEndMeter
|
||||
let displayMeter = displayMeter
|
||||
let rendered = presenter.render(frame.pixelBuffer, isHDR: frame.isHDR) { presentedNs in
|
||||
let atNs = presentedNs ?? targetPresentNs
|
||||
// End-to-end = capture→on-glass, measured directly (skew-corrected via the
|
||||
// connect-time clock offset) — the HUD headline.
|
||||
endToEndMeter?.record(ptsNs: frame.ptsNs, atNs: atNs, offsetNs: offsetNs)
|
||||
// Display stage = decoded → on-glass. Both instants are client CLOCK_REALTIME,
|
||||
// so no skew offset applies.
|
||||
displayMeter?.record(ptsNs: UInt64(frame.decodedNs), atNs: atNs, offsetNs: 0)
|
||||
let offsetNs = offsetNs
|
||||
let renderSignal = renderSignal
|
||||
let renderStopped = renderStopped
|
||||
// Present policy — the user's V-Sync setting (default OFF = immediate, the long-proven
|
||||
// lowest-latency behavior); PUNKTFUNK_PRESENT_MODE=immediate|vsync overrides it for A/B.
|
||||
// Resolved once per session.
|
||||
let presentMode = ProcessInfo.processInfo.environment["PUNKTFUNK_PRESENT_MODE"]
|
||||
let vsyncEnabled = presentMode == "vsync"
|
||||
|| (presentMode != "immediate"
|
||||
&& UserDefaults.standard.bool(forKey: DefaultsKey.vsync))
|
||||
let debugStats = presentDebug ? PresentDebugStats() : nil
|
||||
let vsyncClock = vsyncClock
|
||||
let renderThread = Thread {
|
||||
defer { renderStopped.signal() }
|
||||
while !token.isStopped {
|
||||
if renderSignal.wait(timeout: .now() + .milliseconds(100)) == .timedOut {
|
||||
debugStats?.flushIfDue(ring: ring)
|
||||
continue
|
||||
}
|
||||
guard !token.isStopped, let frame = ring.take() else {
|
||||
debugStats?.emptyWake()
|
||||
debugStats?.flushIfDue(ring: ring)
|
||||
continue
|
||||
}
|
||||
// V-Sync ON: flip on the next predicted vsync (< one period out, stale link ⇒
|
||||
// immediate — see VsyncClock). OFF: flip as soon as the GPU finishes.
|
||||
let presentAt = vsyncEnabled
|
||||
? vsyncClock.nextVsync(after: CACurrentMediaTime()) : nil
|
||||
let renderStarted = CACurrentMediaTime()
|
||||
let rendered = presenter.render(
|
||||
frame.pixelBuffer, isHDR: frame.isHDR, presentAtMediaTime: presentAt
|
||||
) { presentedNs in
|
||||
// Fallback stamp for a dropped drawable (no system presentedTime): "now" on
|
||||
// the Metal callback, converted to the CLOCK_REALTIME the meters live in.
|
||||
let atNs = presentedNs
|
||||
?? Stage2Pipeline.realtimeNs(forDisplayLinkTimestamp: CACurrentMediaTime())
|
||||
// End-to-end = capture→on-glass, measured directly (skew-corrected via the
|
||||
// connect-time clock offset) — the HUD headline.
|
||||
endToEndMeter?.record(ptsNs: frame.ptsNs, atNs: atNs, offsetNs: offsetNs)
|
||||
// Display stage = decoded → on-glass. Both instants are client CLOCK_REALTIME,
|
||||
// so no skew offset applies.
|
||||
displayMeter?.record(ptsNs: UInt64(frame.decodedNs), atNs: atNs, offsetNs: 0)
|
||||
debugStats?.presented(atNs: presentedNs)
|
||||
}
|
||||
debugStats?.renderReturned(
|
||||
ok: rendered, tookMs: (CACurrentMediaTime() - renderStarted) * 1000)
|
||||
if !rendered { ring.putBack(frame) }
|
||||
debugStats?.flushIfDue(ring: ring)
|
||||
}
|
||||
}
|
||||
if !rendered { ring.putBack(frame) }
|
||||
renderThread.name = "punktfunk-stage2-render"
|
||||
renderThread.qualityOfService = .userInteractive
|
||||
renderJoinable = true
|
||||
renderThread.start()
|
||||
}
|
||||
|
||||
/// Stop the pump (≤ one poll timeout) and drop the decode session. MAIN THREAD; idempotent. Does not
|
||||
/// close the connection. A restart needs a fresh Stage2Pipeline (the stop is permanent).
|
||||
/// MAIN thread, once per display-link tick: refresh the vsync clock (V-Sync-mode scheduling)
|
||||
/// and nudge the render thread. The nudge is NOT the presentation trigger — frame arrival is
|
||||
/// (see the header) — it only retries a frame a transient `nextDrawable` failure put back into
|
||||
/// the ring, which matters under the host's infinite GOP where a static scene sends no
|
||||
/// replacement frame.
|
||||
public func renderTick(targetMediaTime: CFTimeInterval, period: CFTimeInterval) {
|
||||
vsyncClock.set(target: targetMediaTime, period: period)
|
||||
renderSignal.signal()
|
||||
}
|
||||
|
||||
/// Forward the layout-derived drawable pixel size to the presenter (MAIN thread — see
|
||||
/// `MetalVideoPresenter.setDrawableTarget`).
|
||||
public func setDrawableTarget(_ size: CGSize) {
|
||||
presenter.setDrawableTarget(size)
|
||||
}
|
||||
|
||||
/// Stop the pump + render thread (≤ one poll timeout each) and drop the decode session. MAIN
|
||||
/// THREAD; idempotent. Does not close the connection. A restart needs a fresh Stage2Pipeline
|
||||
/// (the stop is permanent).
|
||||
public func stop() {
|
||||
token.stop()
|
||||
// Join the pump (bounded: ≤ one nextAU poll + an in-flight decode) before resetting the decoder,
|
||||
@@ -213,11 +399,22 @@ public final class Stage2Pipeline {
|
||||
pumpJoinable = false
|
||||
_ = pumpStopped.wait(timeout: .now() + 0.5)
|
||||
}
|
||||
// Wake + join the render thread (bounded: it may sit in `nextDrawable` for up to ~a frame; a
|
||||
// timed-out join is fine — the loop exits at its next stop-flag check, and a final present on
|
||||
// the detached layer is harmless).
|
||||
if renderJoinable {
|
||||
renderJoinable = false
|
||||
renderSignal.signal()
|
||||
_ = renderStopped.wait(timeout: .now() + 0.5)
|
||||
}
|
||||
decoder.reset()
|
||||
recovery.bind(nil) // stop requesting keyframes once the session is torn down
|
||||
}
|
||||
|
||||
deinit { token.stop() }
|
||||
deinit {
|
||||
token.stop()
|
||||
renderSignal.signal() // wake the render thread so it can observe the stop and exit
|
||||
}
|
||||
|
||||
/// Convert a `CADisplayLink.targetTimestamp` (CACurrentMediaTime basis) to a `CLOCK_REALTIME`
|
||||
/// nanosecond instant — the present clock the AU pts + skew offset live in. Projects to the target
|
||||
|
||||
Reference in New Issue
Block a user