feat(apple): stage-2 default + pixel-perfect, decode robustness, UI/rumble polish
apple / swift (push) Successful in 1m4s
android / android (push) Successful in 4m33s
ci / rust (push) Successful in 5m4s
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 3m12s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 4s
release / apple (push) Successful in 8m30s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 19s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
ci / bench (push) Successful in 4m45s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m48s
apple / screenshots (push) Successful in 5m43s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m24s
docker / deploy-docs (push) Successful in 19s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m46s
apple / swift (push) Successful in 1m4s
android / android (push) Successful in 4m33s
ci / rust (push) Successful in 5m4s
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 3m12s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 4s
release / apple (push) Successful in 8m30s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 19s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
ci / bench (push) Successful in 4m45s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m48s
apple / screenshots (push) Successful in 5m43s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m24s
docker / deploy-docs (push) Successful in 19s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m46s
Stream reliability - Default to the stage-2 presenter (VTDecompressionSession + CAMetalLayer): it detects and recovers a wedged decoder, where stage-1's AVSampleBufferDisplayLayer freezes hard on a lost HEVC reference frame with no app-side recovery (confirmed Apple limitation). Stage 1 is now a DEBUG-only presenter toggle, plus the automatic no-Metal fallback. - Stage-2 pixel-perfect: render the drawable at the decoded size (shader stays 1:1 = identity) and let the layer's contentsGravity scale via the system compositor — the same path stage-1's videoGravity used — instead of scaling in-shader. - Loss recovery in both pumps is now a persistent awaitingIDR want, retried until an IDR actually lands, so a keyframe request swallowed by the throttle can't strand a frozen frame; 100 ms keyframe throttle to match the Android path. - Fix "Publishing changes from within view updates": defer the HostStore writes out of the .onChange(of: model.phase) callback. - Move AVAudioSession setActive/setCategory off the main thread (async on a shared serial queue) to stop the UI-stall warning. Controllers - Rumble: capped-exponential backoff when the gamecontrollerd.haptics XPC breaks (-4811) so a transient server interruption self-heals instead of cascading; playsHapticsOnly so a controller engine doesn't join the always-active streaming audio session. - Host cards: iPad pointer "magnet" hover effect; iPhone press scale + light haptic. UI / design - Ship Geist (SIL OFL 1.1) as the app font (bundled OTFs + registration), with the license surfaced in Acknowledgements. - Restructure iOS/iPadOS Settings into a category NavigationSplitView; resolution wheel with custom-resolution entry; 10-bit HDR toggle in Display. - Industrial host-card redesign (left-aligned, bold, brand monogram tiles). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
// Geist — the punktfunk brand typeface (the same family the website ships). Bundled as static
|
||||
// OTF weights in this kit's resources and registered with Core Text at first use, so it works
|
||||
// identically in the Xcode app and the `swift run` dev shell (Bundle.module resolves to the
|
||||
// package resource bundle in both). Geist Sans carries titles/UI; Geist Mono carries the technical
|
||||
// readouts — host addresses, status labels, the stream-stats HUD — for the industrial look.
|
||||
//
|
||||
// Licensed under the SIL Open Font License 1.1 (Resources/Fonts/Geist-OFL.txt).
|
||||
|
||||
import CoreText
|
||||
import SwiftUI
|
||||
#if canImport(UIKit)
|
||||
import UIKit
|
||||
#elseif canImport(AppKit)
|
||||
import AppKit
|
||||
#endif
|
||||
|
||||
public enum BrandFont {
|
||||
public enum Weight {
|
||||
case regular, medium, semibold, bold
|
||||
}
|
||||
|
||||
/// PostScript names of the bundled faces (verified from each OTF's name table). Geist Sans only
|
||||
/// — Geist Mono is intentionally not shipped; the app's typeface is Geist Sans throughout.
|
||||
private static let sansFaces = ["Geist-Regular", "Geist-Medium", "Geist-SemiBold", "Geist-Bold"]
|
||||
|
||||
/// Registered exactly once per process — a static `let` initializer is run lazily and is
|
||||
/// guaranteed thread-safe + run-at-most-once by the runtime.
|
||||
private static let registered: Void = {
|
||||
for face in sansFaces {
|
||||
guard let url = Bundle.module.url(
|
||||
forResource: face, withExtension: "otf", subdirectory: "Fonts") else {
|
||||
#if DEBUG
|
||||
print("BrandFont: bundled face \(face).otf not found — text will fall back to system")
|
||||
#endif
|
||||
continue
|
||||
}
|
||||
var error: Unmanaged<CFError>?
|
||||
if !CTFontManagerRegisterFontsForURL(url as CFURL, .process, &error) {
|
||||
#if DEBUG
|
||||
let message = error?.takeRetainedValue().localizedDescription ?? "unknown error"
|
||||
print("BrandFont: failed to register \(face): \(message)")
|
||||
#endif
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
/// Force registration before the first `Font.custom` lookup. Cheap to call repeatedly.
|
||||
public static func registerIfNeeded() { _ = registered }
|
||||
|
||||
fileprivate static func sansFace(_ weight: Weight) -> String {
|
||||
switch weight {
|
||||
case .regular: return "Geist-Regular"
|
||||
case .medium: return "Geist-Medium"
|
||||
case .semibold: return "Geist-SemiBold"
|
||||
case .bold: return "Geist-Bold"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public extension Color {
|
||||
/// The punktfunk brand purple (the app-icon lens / website `--brand`). Defined explicitly,
|
||||
/// independent of the asset-catalog accent — `Color.accentColor` resolution is environment- and
|
||||
/// timing-sensitive (it can fall back to system blue), and the brand mark must never drift.
|
||||
/// Light: #6656F2, Dark: #8678F5 (the lighter violet reads better on dark surfaces).
|
||||
static let brand: Color = {
|
||||
#if canImport(UIKit)
|
||||
return Color(UIColor { traits in
|
||||
traits.userInterfaceStyle == .dark
|
||||
? UIColor(red: 0x86 / 255, green: 0x78 / 255, blue: 0xF5 / 255, alpha: 1)
|
||||
: UIColor(red: 0x66 / 255, green: 0x56 / 255, blue: 0xF2 / 255, alpha: 1)
|
||||
})
|
||||
#elseif canImport(AppKit)
|
||||
return Color(NSColor(name: nil) { appearance in
|
||||
appearance.bestMatch(from: [.aqua, .darkAqua]) == .darkAqua
|
||||
? NSColor(red: 0x86 / 255, green: 0x78 / 255, blue: 0xF5 / 255, alpha: 1)
|
||||
: NSColor(red: 0x66 / 255, green: 0x56 / 255, blue: 0xF2 / 255, alpha: 1)
|
||||
})
|
||||
#else
|
||||
// Non-Apple fallback: the light brand value, so all branches agree on a canonical color.
|
||||
return Color(red: 0x66 / 255, green: 0x56 / 255, blue: 0xF2 / 255)
|
||||
#endif
|
||||
}()
|
||||
}
|
||||
|
||||
public extension Font {
|
||||
/// Geist Sans at an explicit point size, scaling with Dynamic Type relative to `textStyle`.
|
||||
static func geist(
|
||||
_ size: CGFloat, _ weight: BrandFont.Weight = .regular,
|
||||
relativeTo textStyle: TextStyle = .body
|
||||
) -> Font {
|
||||
BrandFont.registerIfNeeded()
|
||||
return .custom(BrandFont.sansFace(weight), size: size, relativeTo: textStyle)
|
||||
}
|
||||
|
||||
/// Geist Sans at a FIXED point size that does not scale with Dynamic Type — for glyphs pinned
|
||||
/// inside a fixed-size container (e.g. the monogram tile), where a scaled letter would overflow.
|
||||
static func geistFixed(_ size: CGFloat, _ weight: BrandFont.Weight = .regular) -> Font {
|
||||
BrandFont.registerIfNeeded()
|
||||
return .custom(BrandFont.sansFace(weight), fixedSize: size)
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,9 @@ public enum DefaultsKey {
|
||||
public static let speakerUID = "punktfunk.speakerUID"
|
||||
public static let micUID = "punktfunk.micUID"
|
||||
public static let presenter = "punktfunk.presenter"
|
||||
/// Request a 10-bit BT.2020 PQ (HDR10) stream. On by default; only takes effect when the host
|
||||
/// has HDR content AND this display supports HDR — otherwise the stream stays 8-bit SDR.
|
||||
public static let hdrEnabled = "punktfunk.hdrEnabled"
|
||||
public static let hosts = "punktfunk.hosts"
|
||||
/// Client-side cursor mode: "auto" (shown only in gamescope sessions), "always", "never".
|
||||
public static let cursorMode = "punktfunk.cursorMode"
|
||||
|
||||
@@ -68,6 +68,14 @@ private final class RumbleRenderer: @unchecked Sendable {
|
||||
private var broken = false
|
||||
/// Last logged active/silent state — for a one-line transition log, not per-event spam.
|
||||
private var wasActive = false
|
||||
// Backoff after an engine failure. A broken `gamecontrollerd.haptics` XPC connection (CoreHaptics
|
||||
// -4811 "server connection broke") fails EVERY rebuild until the service relaunches — and that
|
||||
// break fires neither stoppedHandler nor resetHandler, so without a cooldown the next rumble
|
||||
// update immediately rebuilds into the same dead connection, flooding the log and never
|
||||
// recovering. Delay the next setup() — growing 0.5→1→2→4 s on repeated failure — and clear it
|
||||
// the moment a player runs cleanly (or the controller changes).
|
||||
private var retryAfter = Date.distantPast
|
||||
private var consecutiveFailures = 0
|
||||
|
||||
/// CHHapticEvent sharpness = actuator frequency. A DualSense's voice-coil motors need a
|
||||
/// defined frequency to move at all — an intensity-only event (no sharpness) left them
|
||||
@@ -91,6 +99,8 @@ private final class RumbleRenderer: @unchecked Sendable {
|
||||
self.closeHID()
|
||||
self.controller = c
|
||||
self.broken = false
|
||||
self.consecutiveFailures = 0
|
||||
self.retryAfter = .distantPast
|
||||
_ = self.openHIDIfDualSense(c)
|
||||
onBackend?(self.backendNote(for: c))
|
||||
}
|
||||
@@ -108,7 +118,7 @@ private final class RumbleRenderer: @unchecked Sendable {
|
||||
// other pad (and for a DualSense whose HID device could not be opened).
|
||||
if self.hidRumble(low: lowAmp, high: highAmp) { return }
|
||||
guard !self.broken else { return }
|
||||
if active, self.low == nil, self.high == nil {
|
||||
if active, self.low == nil, self.high == nil, Date() >= self.retryAfter {
|
||||
self.setup()
|
||||
}
|
||||
let ok: Bool
|
||||
@@ -124,8 +134,15 @@ private final class RumbleRenderer: @unchecked Sendable {
|
||||
}
|
||||
// Rebuild on the next nonzero amplitude if an engine errored — and tear down OUTSIDE
|
||||
// the `inout` accesses above, so teardown() never mutates a motor that a `drive` call
|
||||
// still holds an exclusive reference to.
|
||||
if !ok { self.teardown() }
|
||||
// still holds an exclusive reference to. Back off so a broken XPC isn't re-hit every
|
||||
// update; once a player is actually running the path has recovered, so clear the backoff.
|
||||
if !ok {
|
||||
self.teardown()
|
||||
self.scheduleRetryBackoff()
|
||||
} else if self.low?.player != nil || self.high?.player != nil {
|
||||
self.consecutiveFailures = 0
|
||||
self.retryAfter = .distantPast
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -157,14 +174,29 @@ private final class RumbleRenderer: @unchecked Sendable {
|
||||
low = makeMotor(haptics, .default)
|
||||
}
|
||||
if low == nil, high == nil {
|
||||
// Haptics present but no engine could be built right now (server busy / a transient
|
||||
// error). Do NOT latch broken — the next nonzero amplitude retries setup().
|
||||
log.warning("rumble: haptics present but engine setup failed — will retry on next rumble")
|
||||
// Haptics present but no engine could be built right now (server busy / XPC broken). Do
|
||||
// NOT latch broken — back off and the next nonzero amplitude past the cooldown retries.
|
||||
log.warning("rumble: haptics present but engine setup failed — backing off, will retry")
|
||||
scheduleRetryBackoff()
|
||||
}
|
||||
}
|
||||
|
||||
/// Push the next engine-build attempt out after a failure (capped exponential backoff), so a
|
||||
/// broken `gamecontrollerd.haptics` connection gets time to relaunch instead of being re-hit on
|
||||
/// every rumble update.
|
||||
private func scheduleRetryBackoff() {
|
||||
consecutiveFailures += 1
|
||||
let shift = min(consecutiveFailures - 1, 4)
|
||||
retryAfter = Date().addingTimeInterval(min(0.5 * Double(1 << shift), 4))
|
||||
}
|
||||
|
||||
private func makeMotor(_ haptics: GCDeviceHaptics, _ locality: GCHapticsLocality) -> Motor? {
|
||||
guard let engine = haptics.createEngine(withLocality: locality) else { return nil }
|
||||
// A controller's motors carry no audio, so keep this engine OUT of the app's audio session
|
||||
// (the default is to join it). Streaming keeps an AVAudioSession active the whole time;
|
||||
// letting a haptics-only engine join it is a needless coupling that can get its
|
||||
// gamecontrollerd XPC connection interrupted (the repeated -4811 server-connection breaks).
|
||||
engine.playsHapticsOnly = true
|
||||
// The haptic server can stop or reset the engine out from under us — app backgrounding, an
|
||||
// audio-session interruption (a call, Siri, another audio app), or a server crash. Left
|
||||
// unhandled the players go dead and every later rumble throws, latching rumble off for the
|
||||
|
||||
@@ -27,6 +27,17 @@ public enum Licenses {
|
||||
+ apache
|
||||
}
|
||||
|
||||
/// The bundled brand typeface (Geist Sans + Geist Mono) — SIL Open Font License 1.1. The
|
||||
/// license file ships alongside the OTFs in `Resources/Fonts/`, satisfying the OFL's
|
||||
/// distribution requirement; this surfaces it in the Acknowledgements screen too.
|
||||
public static var fontLicense: String {
|
||||
guard let url = Bundle.module.url(
|
||||
forResource: "Geist-OFL", withExtension: "txt", subdirectory: "Fonts"),
|
||||
let text = try? String(contentsOf: url, encoding: .utf8)
|
||||
else { return "" }
|
||||
return text
|
||||
}
|
||||
|
||||
/// Third-party software notices for the linked Rust crates (generated by
|
||||
/// `scripts/gen-third-party-notices.sh`).
|
||||
public static var thirdPartyNotices: String {
|
||||
|
||||
@@ -11,6 +11,9 @@ import CoreGraphics
|
||||
import CoreVideo
|
||||
import Metal
|
||||
import QuartzCore
|
||||
import os
|
||||
|
||||
private let presenterLog = Logger(subsystem: "io.unom.punktfunk", category: "presenter")
|
||||
|
||||
/// Runtime-compiled (no metallib build step needed in SwiftPM): a fullscreen triangle and a
|
||||
/// BT.709 limited-range NV12→RGB fragment shader. uv.y is flipped (1 - p.y) so the top-left-
|
||||
@@ -30,11 +33,44 @@ vertex VOut pf_vtx(uint vid [[vertex_id]]) {
|
||||
return o;
|
||||
}
|
||||
|
||||
// Bicubic (Catmull-Rom) sampling of the single-channel luma plane. When the drawable is larger
|
||||
// than the decoded frame (a window/view bigger than the host's fixed mode), a bilinear upscale
|
||||
// looks soft; Catmull-Rom keeps edges crisp — matching AVSampleBufferDisplayLayer's (stage-1)
|
||||
// scaler — and reduces to the exact texel at 1:1, so a native-resolution present stays pixel-exact.
|
||||
// Nine bilinear taps (TheRealMJP's optimisation of the 16-tap kernel); `s` MUST be a linear
|
||||
// sampler. Luma carries the perceived detail, so only it gets bicubic; chroma stays bilinear.
|
||||
float catmullRomLuma(texture2d<float> tex, sampler s, float2 uv) {
|
||||
float2 texSize = float2(tex.get_width(), tex.get_height());
|
||||
float2 samplePos = uv * texSize;
|
||||
float2 tc1 = floor(samplePos - 0.5) + 0.5;
|
||||
float2 f = samplePos - tc1;
|
||||
float2 w0 = f * (-0.5 + f * (1.0 - 0.5 * f));
|
||||
float2 w1 = 1.0 + f * f * (-2.5 + 1.5 * f);
|
||||
float2 w2 = f * (0.5 + f * (2.0 - 1.5 * f));
|
||||
float2 w3 = f * f * (-0.5 + 0.5 * f);
|
||||
float2 w12 = w1 + w2;
|
||||
float2 off12 = w2 / w12;
|
||||
float2 tc0 = (tc1 - 1.0) / texSize;
|
||||
float2 tc3 = (tc1 + 2.0) / texSize;
|
||||
float2 tc12 = (tc1 + off12) / texSize;
|
||||
float r = 0.0;
|
||||
r += tex.sample(s, float2(tc0.x, tc0.y)).r * (w0.x * w0.y);
|
||||
r += tex.sample(s, float2(tc12.x, tc0.y)).r * (w12.x * w0.y);
|
||||
r += tex.sample(s, float2(tc3.x, tc0.y)).r * (w3.x * w0.y);
|
||||
r += tex.sample(s, float2(tc0.x, tc12.y)).r * (w0.x * w12.y);
|
||||
r += tex.sample(s, float2(tc12.x, tc12.y)).r * (w12.x * w12.y);
|
||||
r += tex.sample(s, float2(tc3.x, tc12.y)).r * (w3.x * w12.y);
|
||||
r += tex.sample(s, float2(tc0.x, tc3.y)).r * (w0.x * w3.y);
|
||||
r += tex.sample(s, float2(tc12.x, tc3.y)).r * (w12.x * w3.y);
|
||||
r += tex.sample(s, float2(tc3.x, tc3.y)).r * (w3.x * w3.y);
|
||||
return r;
|
||||
}
|
||||
|
||||
fragment float4 pf_frag(VOut in [[stage_in]],
|
||||
texture2d<float> lumaTex [[texture(0)]],
|
||||
texture2d<float> chromaTex [[texture(1)]]) {
|
||||
constexpr sampler s(filter::linear, address::clamp_to_edge);
|
||||
float y = lumaTex.sample(s, in.uv).r;
|
||||
float y = catmullRomLuma(lumaTex, s, in.uv);
|
||||
float2 c = chromaTex.sample(s, in.uv).rg;
|
||||
// BT.709, 8-bit limited (video) range → full-range RGB.
|
||||
y = (y - 16.0/255.0) * (255.0/219.0);
|
||||
@@ -55,7 +91,7 @@ fragment float4 pf_frag_hdr(VOut in [[stage_in]],
|
||||
texture2d<float> lumaTex [[texture(0)]],
|
||||
texture2d<float> chromaTex [[texture(1)]]) {
|
||||
constexpr sampler s(filter::linear, address::clamp_to_edge);
|
||||
float y = lumaTex.sample(s, in.uv).r;
|
||||
float y = catmullRomLuma(lumaTex, s, in.uv);
|
||||
float2 c = chromaTex.sample(s, in.uv).rg;
|
||||
// BT.2020 10-bit limited (video) range → full-range PQ R'G'B'.
|
||||
y = (y - 64.0/1023.0) * (1023.0/876.0);
|
||||
@@ -81,6 +117,11 @@ public final class MetalVideoPresenter {
|
||||
private var textureCache: CVMetalTextureCache?
|
||||
/// Current layer configuration — switched lazily in `configure(hdr:)` when a frame's mode differs.
|
||||
private var hdrActive = false
|
||||
#if DEBUG
|
||||
/// Last logged "decoded→drawable" signature, so the diagnostic logs only when a size changes
|
||||
/// (on first frame, a resize, or a host Reconfigure) instead of every frame.
|
||||
private var lastSizeSig = ""
|
||||
#endif
|
||||
|
||||
/// nil if Metal is unavailable (no GPU / a headless CI) — the caller falls back to stage-1.
|
||||
public init?() {
|
||||
@@ -113,6 +154,12 @@ public final class MetalVideoPresenter {
|
||||
layer.pixelFormat = .bgra8Unorm
|
||||
layer.framebufferOnly = true
|
||||
layer.isOpaque = true
|
||||
// Render the drawable at the DECODED frame's resolution (set per-frame in `render`) and let
|
||||
// the system compositor scale it to the layer's bounds — the same `.resizeAspect` path
|
||||
// stage-1's AVSampleBufferDisplayLayer (videoGravity) uses, so stage-2 matches its sharpness.
|
||||
// A native-resolution present is then pixel-exact (1:1, no shader scaling), and any display
|
||||
// scaling uses the system's high-quality scaler rather than the in-shader bicubic.
|
||||
layer.contentsGravity = .resizeAspect
|
||||
// Triple-buffer: more in-flight drawables before `nextDrawable()` (called on the
|
||||
// display-link / MAIN thread) has to block waiting for one to free.
|
||||
layer.maximumDrawableCount = 3
|
||||
@@ -129,12 +176,6 @@ public final class MetalVideoPresenter {
|
||||
self.layer = layer
|
||||
}
|
||||
|
||||
/// Track the stream mode (the host can Reconfigure mid-stream). Size is in pixels.
|
||||
public func setDrawableSize(_ size: CGSize) {
|
||||
guard size.width > 0, size.height > 0 else { return }
|
||||
if layer.drawableSize != size { layer.drawableSize = size }
|
||||
}
|
||||
|
||||
/// Reconfigure the layer for SDR or HDR when the stream mode flips (HDR toggle). HDR uses an
|
||||
/// rgba16Float drawable + a BT.2020 PQ colour space + EDR, so the compositor PQ-maps to the
|
||||
/// display; SDR uses the plain 8-bit sRGB path. Main-thread only (called from `render`).
|
||||
@@ -171,13 +212,33 @@ public final class MetalVideoPresenter {
|
||||
let chroma = makeTexture(pixelBuffer, plane: 1, format: chromaFmt, cache: textureCache)
|
||||
else { return false }
|
||||
|
||||
// The hosting view owns drawableSize (aspect-fit to its bounds); skip until it's laid
|
||||
// out. The fullscreen triangle scales the decoded texture to fill the drawable.
|
||||
guard layer.drawableSize.width > 0, layer.drawableSize.height > 0,
|
||||
let drawable = layer.nextDrawable(),
|
||||
// Size the drawable to the decoded frame so the fullscreen triangle samples the texture 1:1
|
||||
// (pixel-exact); the layer's contentsGravity then scales it to the on-screen bounds via the
|
||||
// system compositor (matching stage-1). Re-set only on a change (first frame / Reconfigure).
|
||||
let decodedSize = CGSize(
|
||||
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
|
||||
if layer.drawableSize != decodedSize { layer.drawableSize = decodedSize }
|
||||
guard let drawable = layer.nextDrawable(),
|
||||
let commandBuffer = queue.makeCommandBuffer()
|
||||
else { return false }
|
||||
|
||||
#if DEBUG
|
||||
// Diagnose sharpness: decoded should equal the drawable (the shader is 1:1); the layer's
|
||||
// bounds may differ (the system scales). Logged only when a size changes.
|
||||
let decodedW = Int(decodedSize.width)
|
||||
let decodedH = Int(decodedSize.height)
|
||||
let sig = "\(decodedW)x\(decodedH)|\(Int(layer.drawableSize.width))x\(Int(layer.drawableSize.height))"
|
||||
if sig != lastSizeSig {
|
||||
lastSizeSig = sig
|
||||
let msg = "stage2: decoded \(decodedW)x\(decodedH) → drawable "
|
||||
+ "\(Int(layer.drawableSize.width))x\(Int(layer.drawableSize.height)) "
|
||||
+ "(texture \(drawable.texture.width)x\(drawable.texture.height), "
|
||||
+ "contentsScale \(layer.contentsScale), "
|
||||
+ "layerBounds \(Int(layer.bounds.width))x\(Int(layer.bounds.height)))"
|
||||
presenterLog.info("\(msg, privacy: .public)")
|
||||
}
|
||||
#endif
|
||||
|
||||
let pass = MTLRenderPassDescriptor()
|
||||
pass.colorAttachments[0].texture = drawable.texture
|
||||
pass.colorAttachments[0].loadAction = .clear
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,93 @@
|
||||
Copyright 2024 The Geist Project Authors (https://github.com/vercel/geist-font)
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
https://openfontlicense.org
|
||||
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
Binary file not shown.
Binary file not shown.
@@ -177,6 +177,16 @@ public final class SessionAudio {
|
||||
private var playbackEngine: AVAudioEngine?
|
||||
private var captureEngine: AVAudioEngine?
|
||||
private var drainStarted = false
|
||||
#if !os(macOS)
|
||||
/// AVAudioSession `setCategory`/`setActive` are synchronous and block on the audio server, so
|
||||
/// they must not run on the main thread (UI stall — AVFoundation warns about it). PROCESS-WIDE
|
||||
/// (static) so every SessionAudio shares one serial queue: the AVAudioSession is a process
|
||||
/// singleton, and across a reconnect the old session's deactivate must be ordered before the
|
||||
/// new session's activate (a per-instance queue would let them race and leave the new session's
|
||||
/// audio deactivated). stop() enqueues its deactivate promptly so it lands before the next
|
||||
/// session's activate.
|
||||
private static let sessionQueue = DispatchQueue(label: "io.unom.punktfunk.audio.session")
|
||||
#endif
|
||||
|
||||
public init(connection: PunktfunkConnection) {
|
||||
self.connection = connection
|
||||
@@ -189,37 +199,60 @@ public final class SessionAudio {
|
||||
flag.stop()
|
||||
}
|
||||
|
||||
/// Start playback (and, if enabled+authorized, the mic uplink). Empty UIDs = system
|
||||
/// default device; on iOS the UIDs are ignored entirely (routes are
|
||||
/// AVAudioSession-managed). Main thread (engine setup); returns after the engines
|
||||
/// start — the mic may start slightly later if the permission prompt is pending.
|
||||
/// Start playback (and, if enabled+authorized, the mic uplink). Empty UIDs = system default
|
||||
/// device; on iOS the UIDs are ignored entirely (routes are AVAudioSession-managed). On macOS
|
||||
/// the engines start synchronously on the caller's (main) thread. On iOS/tvOS start() is
|
||||
/// ASYNCHRONOUS: it activates the AVAudioSession off the main thread, then starts the engines on
|
||||
/// a later main-queue hop (gated by `!flag.isStopped`) — so playback is live shortly after, not
|
||||
/// on return. The mic may start later still if the permission prompt is pending.
|
||||
public func start(speakerUID: String, micUID: String, micEnabled: Bool) {
|
||||
#if os(iOS)
|
||||
// Route + policy live in the session, not per-engine: stereo playback, mic
|
||||
// capture when enabled, Bluetooth allowed. Failure is non-fatal (defaults).
|
||||
#if os(macOS)
|
||||
// No AVAudioSession on macOS — start the engines directly (caller's thread, as before).
|
||||
startEngines(speakerUID: speakerUID, micUID: micUID, micEnabled: micEnabled)
|
||||
#else
|
||||
// Configure + activate the session OFF the main thread (it blocks on the audio server),
|
||||
// then start the engines back on the main thread once it's active — engine routing/format
|
||||
// depend on the active session. A stop() racing in between is caught by the flag guard.
|
||||
Self.sessionQueue.async { [weak self] in
|
||||
guard let self else { return }
|
||||
self.activateAudioSession(micEnabled: micEnabled)
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
guard let self, !self.flag.isStopped else { return }
|
||||
self.startEngines(speakerUID: speakerUID, micUID: micUID, micEnabled: micEnabled)
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
#if !os(macOS)
|
||||
/// Route + policy live in the session, not per-engine: stereo playback, mic capture when
|
||||
/// enabled, Bluetooth allowed. Failure is non-fatal (defaults). Runs on `sessionQueue`.
|
||||
private func activateAudioSession(micEnabled: Bool) {
|
||||
let session = AVAudioSession.sharedInstance()
|
||||
do {
|
||||
#if os(iOS)
|
||||
if micEnabled {
|
||||
// .defaultToSpeaker: .playAndRecord otherwise routes to the iPhone
|
||||
// EARPIECE; only affects the built-in route (headphones/BT still win).
|
||||
// .defaultToSpeaker: .playAndRecord otherwise routes to the iPhone EARPIECE; only
|
||||
// affects the built-in route (headphones/BT still win).
|
||||
try session.setCategory(
|
||||
.playAndRecord, mode: .default,
|
||||
options: [.allowBluetoothA2DP, .defaultToSpeaker])
|
||||
} else {
|
||||
try session.setCategory(.playback, mode: .default)
|
||||
}
|
||||
#else // tvOS — no app-accessible mic
|
||||
try session.setCategory(.playback, mode: .default)
|
||||
#endif
|
||||
try session.setActive(true)
|
||||
} catch {
|
||||
log.warning("AVAudioSession setup failed: \(error.localizedDescription)")
|
||||
}
|
||||
#elseif os(tvOS)
|
||||
do {
|
||||
try AVAudioSession.sharedInstance().setCategory(.playback, mode: .default)
|
||||
try AVAudioSession.sharedInstance().setActive(true)
|
||||
} catch {
|
||||
log.warning("AVAudioSession setup failed: \(error.localizedDescription)")
|
||||
}
|
||||
#endif
|
||||
}
|
||||
#endif
|
||||
|
||||
/// Build + start the playback engine (and the mic uplink when enabled + authorized). Main
|
||||
/// thread (engine setup); on iOS/tvOS the session is already active by the time this runs.
|
||||
private func startEngines(speakerUID: String, micUID: String, micEnabled: Bool) {
|
||||
startPlayback(speakerUID: speakerUID)
|
||||
#if os(tvOS)
|
||||
// No app-accessible microphone input on tvOS — playback only.
|
||||
@@ -258,19 +291,24 @@ public final class SessionAudio {
|
||||
capture.stop()
|
||||
}
|
||||
playback?.stop()
|
||||
#if !os(macOS)
|
||||
// Release the session so audio we interrupted (Music, podcasts) gets its resume cue. Like
|
||||
// activation, setActive is synchronous/blocking — run it on the shared serial session queue
|
||||
// (off the main thread). Enqueued HERE — engines already stopped, and BEFORE the drain wait
|
||||
// below — so across a reconnect it lands ahead of the next session's activate on the shared
|
||||
// queue (otherwise a deferred deactivate could deactivate the new session). Fire-and-forget.
|
||||
Self.sessionQueue.async {
|
||||
do {
|
||||
try AVAudioSession.sharedInstance().setActive(
|
||||
false, options: .notifyOthersOnDeactivation)
|
||||
} catch {
|
||||
log.warning("AVAudioSession deactivation failed: \(error.localizedDescription)")
|
||||
}
|
||||
}
|
||||
#endif
|
||||
if wasDraining {
|
||||
_ = drainDone.wait(timeout: .now() + .milliseconds(400))
|
||||
}
|
||||
#if !os(macOS)
|
||||
// Release the session so audio we interrupted (Music, podcasts) gets its
|
||||
// resume cue.
|
||||
do {
|
||||
try AVAudioSession.sharedInstance().setActive(
|
||||
false, options: .notifyOthersOnDeactivation)
|
||||
} catch {
|
||||
log.warning("AVAudioSession deactivation failed: \(error.localizedDescription)")
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
// MARK: - Playback (host → speaker)
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
// capture→present. Mirrors StreamPump's lifecycle (one per start; cancel is permanent).
|
||||
//
|
||||
// Threading: the pump runs on its own thread; the decoder callback on a VT thread; `renderTick`
|
||||
// + `setDrawableSize` + `start`/`stop` on the MAIN thread (the view's CADisplayLink fires there).
|
||||
// + `start`/`stop` on the MAIN thread (the view's CADisplayLink fires there).
|
||||
// Only the ring + decoder cross threads and both are internally locked.
|
||||
|
||||
#if canImport(Metal) && canImport(QuartzCore)
|
||||
@@ -60,7 +60,7 @@ private final class KeyframeRecovery: @unchecked Sendable {
|
||||
func request() {
|
||||
lock.lock()
|
||||
let now = DispatchTime.now().uptimeNanoseconds
|
||||
let due = lastNs == 0 || now &- lastNs > 250_000_000 // ≥ 250 ms since the last request
|
||||
let due = lastNs == 0 || now &- lastNs > 100_000_000 // ≥ 100 ms since the last request (matches Android)
|
||||
if due { lastNs = now }
|
||||
let conn = due ? connection : nil
|
||||
lock.unlock()
|
||||
@@ -114,20 +114,24 @@ public final class Stage2Pipeline {
|
||||
let thread = Thread {
|
||||
var format: CMVideoFormatDescription?
|
||||
var lastFramesDropped = connection.framesDropped()
|
||||
// Persistent recovery WANT, not a one-shot edge (see StreamPump for the full rationale):
|
||||
// the old code advanced lastFramesDropped on the same edge it called recovery.request(),
|
||||
// so a request swallowed by the throttle (the lost recovery IDR being pruned within the
|
||||
// window) was never re-sent and the picture stayed frozen. Keep asking until an IDR lands.
|
||||
var awaitingIDR = false
|
||||
while token.isLive {
|
||||
do {
|
||||
// Loss recovery (the primary recovery path). The reassembler drops unrecoverable
|
||||
// AUs (framesDropped) and the decoder then conceals the reference-missing delta
|
||||
// frames that follow — often rendering them WITHOUT an error callback — so the
|
||||
// onDecodeError trigger rarely fires after a real network blip. Ask the host for
|
||||
// a fresh IDR whenever the drop count climbs (throttled in KeyframeRecovery).
|
||||
// Polled every iteration so a total-loss drought recovers the moment packets
|
||||
// resume and the reassembler counts the gap.
|
||||
// Loss recovery (the primary path). The reassembler drops unrecoverable AUs
|
||||
// (framesDropped) and the decoder conceals the reference-missing deltas that
|
||||
// follow — often WITHOUT an error callback — so key off the drop count climbing,
|
||||
// then keep asking (awaitingIDR) until a fresh IDR re-anchors decode. Polled every
|
||||
// iteration so a total-loss drought recovers the moment packets resume.
|
||||
let dropped = connection.framesDropped()
|
||||
if dropped > lastFramesDropped {
|
||||
lastFramesDropped = dropped
|
||||
recovery.request()
|
||||
awaitingIDR = true
|
||||
}
|
||||
if awaitingIDR { recovery.request() }
|
||||
// Drain any HDR mastering-metadata update (0xCE) and hand it to the decoder, which
|
||||
// attaches it to subsequent HDR frames. Non-blocking; only HDR sessions emit these.
|
||||
if connection.isHDR, let meta = try? connection.nextHdrMeta(timeoutMs: 0) {
|
||||
@@ -136,15 +140,16 @@ public final class Stage2Pipeline {
|
||||
guard let au = try connection.nextAU(timeoutMs: 100) else { continue }
|
||||
onFrame?(au)
|
||||
if let f = AnnexB.formatDescription(fromIDR: au.data) {
|
||||
format = f // refreshed on every IDR (mode changes included)
|
||||
format = f // refreshed on every IDR (mode changes included)
|
||||
awaitingIDR = false // a fresh IDR re-anchored decode — recovery complete
|
||||
}
|
||||
guard let f = format, token.isLive else { continue }
|
||||
if !decoder.decode(au: au, format: f) {
|
||||
// Submit/decoder error: drop the session and re-gate on the next IDR's
|
||||
// in-band parameter sets (a delta frame can't recover) — stage-1's policy
|
||||
// — and ask the host for that IDR now (infinite GOP; throttled).
|
||||
// in-band parameter sets (a delta frame can't recover) — stage-1's policy —
|
||||
// and keep asking for that IDR (infinite GOP) until one re-anchors decode.
|
||||
decoder.reset()
|
||||
recovery.request()
|
||||
awaitingIDR = true
|
||||
}
|
||||
} catch {
|
||||
if token.isLive { onSessionEnd?() }
|
||||
@@ -166,11 +171,6 @@ public final class Stage2Pipeline {
|
||||
presentMeter.record(ptsNs: frame.ptsNs, atNs: targetPresentNs, offsetNs: offsetNs)
|
||||
}
|
||||
|
||||
/// MAIN thread. Keep the drawable matched to the negotiated mode (host can Reconfigure).
|
||||
public func setDrawableSize(_ size: CGSize) {
|
||||
presenter.setDrawableSize(size)
|
||||
}
|
||||
|
||||
/// Stop the pump (≤ one poll timeout) and drop the decode session. Does not close the
|
||||
/// connection. A restart needs a fresh Stage2Pipeline (cancel is permanent).
|
||||
public func stop() {
|
||||
|
||||
@@ -6,6 +6,9 @@
|
||||
|
||||
import AVFoundation
|
||||
import Foundation
|
||||
import os
|
||||
|
||||
private let pumpLog = Logger(subsystem: "io.unom.punktfunk", category: "video")
|
||||
|
||||
/// Cancellation handle owned by exactly one pump thread — a restart hands the old pump
|
||||
/// its own token, so it can never be revived by a newer start().
|
||||
@@ -47,44 +50,74 @@ final class StreamPump {
|
||||
var format: CMVideoFormatDescription?
|
||||
var lastKeyframeRequest = Date.distantPast
|
||||
var lastFramesDropped = connection.framesDropped()
|
||||
// Coalesced host keyframe request: the decode stays wedged for several frames until
|
||||
// the IDR lands, so requesting on every frame would flood the control stream.
|
||||
// Recovery is a persistent WANT, not a one-shot edge: set it on detected loss (or a
|
||||
// decoder reset), retry the throttled request EVERY iteration, and clear it only when a
|
||||
// fresh IDR actually re-anchors decode. The old code advanced `lastFramesDropped` on the
|
||||
// same edge it fired the throttled request — so a request swallowed by the throttle (a
|
||||
// second drop within the window, e.g. the lost recovery IDR itself being pruned) was
|
||||
// never re-sent: the counter went flat, the climb never re-fired, and the picture stayed
|
||||
// frozen for good while audio kept playing. The iPhone's lossy Wi-Fi hits this where the
|
||||
// Mac's Ethernet never does.
|
||||
var awaitingIDR = false
|
||||
var awaitingSince = Date.distantPast // when the current recovery began (for the resume log)
|
||||
var wasFailed = false
|
||||
// Coalesced host keyframe request. 100 ms throttle (matches the working Android path):
|
||||
// fast enough that a lost recovery IDR is re-requested promptly, bounded so a sustained
|
||||
// freeze can't flood the control stream.
|
||||
func requestKeyframeThrottled() {
|
||||
let now = Date()
|
||||
if now.timeIntervalSince(lastKeyframeRequest) > 0.25 {
|
||||
if now.timeIntervalSince(lastKeyframeRequest) > 0.1 {
|
||||
connection.requestKeyframe()
|
||||
lastKeyframeRequest = now
|
||||
}
|
||||
}
|
||||
while token.isLive {
|
||||
do {
|
||||
// Loss recovery (the primary recovery path). Under the host's infinite GOP the
|
||||
// only recovery keyframe is one we request. The reassembler drops unrecoverable
|
||||
// AUs (framesDropped); the decoder then *conceals* the reference-missing delta
|
||||
// frames that follow — a frozen / garbage picture, WITHOUT flipping the layer to
|
||||
// .failed — so the .failed check below rarely fires after a real network blip.
|
||||
// Ask the host for a fresh IDR whenever the drop count climbs. Polled every
|
||||
// iteration (not just per AU) so a total-loss drought still recovers the moment
|
||||
// packets resume and the reassembler counts the gap.
|
||||
// Loss recovery (the primary path). Under the host's infinite GOP the only
|
||||
// recovery keyframe is one we request. The reassembler drops unrecoverable AUs
|
||||
// (framesDropped); the decoder then *conceals* the reference-missing deltas — a
|
||||
// frozen / garbage picture that never flips the layer to .failed — so key off the
|
||||
// drop count climbing, then keep asking (awaitingIDR) until an IDR lands. Polled
|
||||
// every iteration so a total-loss drought still recovers when packets resume.
|
||||
let dropped = connection.framesDropped()
|
||||
if dropped > lastFramesDropped {
|
||||
// Log only on the false→true transition (once per recovery cycle), not per
|
||||
// dropped AU, so heavy loss doesn't spam the log.
|
||||
if !awaitingIDR {
|
||||
awaitingSince = Date()
|
||||
pumpLog.notice(
|
||||
"video: unrecoverable drop (framesDropped=\(dropped, privacy: .public)) — requesting recovery IDR")
|
||||
}
|
||||
lastFramesDropped = dropped
|
||||
requestKeyframeThrottled()
|
||||
awaitingIDR = true
|
||||
}
|
||||
if awaitingIDR { requestKeyframeThrottled() }
|
||||
|
||||
guard let au = try connection.nextAU(timeoutMs: 100) else { continue }
|
||||
onFrame?(au)
|
||||
if let f = AnnexB.formatDescription(fromIDR: au.data) {
|
||||
format = f // refreshed on every IDR (mode changes included)
|
||||
let idrFormat = AnnexB.formatDescription(fromIDR: au.data)
|
||||
if let f = idrFormat {
|
||||
format = f // refreshed on every IDR (mode changes included)
|
||||
if awaitingIDR {
|
||||
let ms = Int(Date().timeIntervalSince(awaitingSince) * 1000)
|
||||
pumpLog.notice("video: recovery IDR received — resumed after \(ms, privacy: .public) ms")
|
||||
}
|
||||
awaitingIDR = false // a fresh IDR re-anchored decode — recovery complete
|
||||
}
|
||||
if layer.status == .failed {
|
||||
let failed = layer.status == .failed
|
||||
if failed {
|
||||
// Decode wedged hard (the cold-first-connect case — a lost/corrupt opening
|
||||
// IDR): flush and re-gate on the next in-band parameter sets (resuming with
|
||||
// a delta frame can't recover), AND ask the host for a fresh IDR. Throttled:
|
||||
// the layer stays .failed across several polls until the IDR lands.
|
||||
// IDR): flush and, unless THIS AU is the recovering IDR (re-anchored above),
|
||||
// re-gate on the next in-band parameter sets and keep asking — enqueuing a
|
||||
// delta into a failed layer can't recover it.
|
||||
if !wasFailed { pumpLog.warning("video: display layer .failed — flushing + re-anchoring") }
|
||||
layer.flush()
|
||||
format = AnnexB.formatDescription(fromIDR: au.data)
|
||||
requestKeyframeThrottled()
|
||||
if idrFormat == nil {
|
||||
format = nil
|
||||
awaitingIDR = true
|
||||
}
|
||||
}
|
||||
wasFailed = failed
|
||||
guard let f = format,
|
||||
let sample = AnnexB.sampleBuffer(au: au, format: f),
|
||||
token.isLive // don't enqueue a stale frame after a restart
|
||||
|
||||
@@ -245,6 +245,15 @@ public final class StreamLayerView: NSView {
|
||||
layoutMetalLayer() // keep the stage-2 sublayer aspect-fit to the view
|
||||
}
|
||||
|
||||
public override func setFrameSize(_ newSize: NSSize) {
|
||||
super.setFrameSize(newSize)
|
||||
// `layout()` isn't guaranteed on a manual-frame (no-Auto-Layout) live resize, so the
|
||||
// stage-2 metal sublayer's drawableSize could stay at the old size while the view grows —
|
||||
// the compositor then upscales a too-small drawable and the video turns blocky. Resize the
|
||||
// drawable here too so it always tracks the window's pixel size (no stale upscale).
|
||||
layoutMetalLayer()
|
||||
}
|
||||
|
||||
// MARK: - Capture state machine
|
||||
|
||||
/// Clicking into the video engages capture; that click is local (engagement), so
|
||||
@@ -549,10 +558,17 @@ public final class StreamLayerView: NSView {
|
||||
cursorVisible = false
|
||||
_ = connection.resolvedCompositor // (was: Auto → gamescope; kept to document intent)
|
||||
|
||||
// Presenter choice — default stage-1 (the known-good AVSampleBufferDisplayLayer). Stage-2
|
||||
// (`punktfunk.presenter == "stage2"`) takes explicit VTDecompressionSession decode + a
|
||||
// CAMetalLayer/display-link present; it falls back here if Metal can't be set up.
|
||||
if UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage2",
|
||||
// Presenter choice — stage-2 is the DEFAULT (explicit VTDecompressionSession decode + a
|
||||
// CAMetalLayer/display-link present): it can detect + recover a wedged decoder where
|
||||
// stage-1's AVSampleBufferDisplayLayer freezes hard on a lost HEVC reference. Stage-1 is
|
||||
// reachable only via the DEBUG presenter toggle; release always takes stage-2 (the stage-1
|
||||
// pump below stays the automatic fallback if Metal is missing).
|
||||
#if DEBUG
|
||||
let forceStage1 = UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage1"
|
||||
#else
|
||||
let forceStage1 = false
|
||||
#endif
|
||||
if !forceStage1,
|
||||
let meter = presentMeter,
|
||||
let pipeline = Stage2Pipeline(presentMeter: meter) {
|
||||
startStage2(pipeline, connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
|
||||
@@ -593,9 +609,11 @@ public final class StreamLayerView: NSView {
|
||||
targetPresentNs: Stage2Pipeline.realtimeNs(forDisplayLinkTimestamp: link.targetTimestamp))
|
||||
}
|
||||
|
||||
/// Aspect-fit the metal sublayer in the view (the host streams at the client's native mode,
|
||||
/// so this is usually the full bounds; it letterboxes a resized window). drawableSize is the
|
||||
/// layer's pixel size — the fullscreen-triangle shader scales the decoded texture to fill it.
|
||||
/// Position the metal sublayer aspect-fit in the view (the host streams at the client's native
|
||||
/// mode, so this is usually the full bounds; it letterboxes a resized window). Only the layer
|
||||
/// FRAME is set here — the presenter sizes the drawable to the decoded frame and the layer's
|
||||
/// contentsGravity (.resizeAspect) scales it to this frame via the system compositor, so a
|
||||
/// resized window rescales through the system's filter (matching stage-1) instead of the shader.
|
||||
private func layoutMetalLayer() {
|
||||
guard let metalLayer, let connection else { return }
|
||||
let mode = connection.currentMode()
|
||||
@@ -604,14 +622,12 @@ public final class StreamLayerView: NSView {
|
||||
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
|
||||
insideRect: bounds)
|
||||
: bounds
|
||||
let scale = window?.backingScaleFactor ?? 1
|
||||
// No implicit resize animation; refresh contentsScale on a retina↔non-retina move.
|
||||
CATransaction.begin()
|
||||
CATransaction.setDisableActions(true)
|
||||
metalLayer.contentsScale = scale
|
||||
metalLayer.contentsScale = window?.backingScaleFactor ?? 1
|
||||
metalLayer.frame = fit
|
||||
CATransaction.commit()
|
||||
stage2?.setDrawableSize(CGSize(width: fit.width * scale, height: fit.height * scale))
|
||||
}
|
||||
|
||||
public override func viewDidChangeBackingProperties() {
|
||||
|
||||
@@ -136,6 +136,13 @@ public final class StreamViewController: UIViewController {
|
||||
|
||||
public override func loadView() {
|
||||
view = StreamLayerUIView()
|
||||
// Re-size the stage-2 drawable if the display scale changes without a bounds change (e.g.
|
||||
// moving to an external display at a different scale) — the iOS analogue of macOS's
|
||||
// viewDidChangeBackingProperties relayout. The handler takes the VC as its argument, so it
|
||||
// doesn't capture self (no retain cycle with the registration).
|
||||
registerForTraitChanges([UITraitDisplayScale.self]) { (vc: StreamViewController, _) in
|
||||
vc.layoutMetalLayer()
|
||||
}
|
||||
#if os(iOS)
|
||||
// Hide the iPadOS cursor while it hovers the video: the host renders its own
|
||||
// cursor from our deltas, so the local one only diverges from it. This hides the
|
||||
@@ -219,10 +226,17 @@ public final class StreamViewController: UIViewController {
|
||||
inputCapture = capture
|
||||
#endif
|
||||
|
||||
// Presenter choice — default stage-1 (the known-good AVSampleBufferDisplayLayer). Stage-2
|
||||
// (`punktfunk.presenter == "stage2"`) takes VTDecompressionSession decode + a
|
||||
// CAMetalLayer/display-link present; falls back here if Metal can't be set up.
|
||||
if UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage2",
|
||||
// Presenter choice — stage-2 is the DEFAULT (VTDecompressionSession decode + a
|
||||
// CAMetalLayer/display-link present): it can detect + recover a wedged decoder, where
|
||||
// stage-1's AVSampleBufferDisplayLayer freezes hard on a lost HEVC reference frame with no
|
||||
// way to recover. Stage-1 is reachable only via the DEBUG presenter toggle; release always
|
||||
// takes stage-2 (the stage-1 pump below stays the automatic fallback if Metal is missing).
|
||||
#if DEBUG
|
||||
let forceStage1 = UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage1"
|
||||
#else
|
||||
let forceStage1 = false
|
||||
#endif
|
||||
if !forceStage1,
|
||||
let meter = presentMeter,
|
||||
let pipeline = Stage2Pipeline(presentMeter: meter) {
|
||||
startStage2(pipeline, connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
|
||||
@@ -300,8 +314,8 @@ public final class StreamViewController: UIViewController {
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)?, onSessionEnd: (@Sendable () -> Void)?
|
||||
) {
|
||||
let metal = pipeline.layer
|
||||
metal.contentsScale = streamView.contentScaleFactor
|
||||
// Composites OVER the idle (un-enqueued in stage-2) AVSampleBufferDisplayLayer base.
|
||||
// (contentsScale + frame + drawableSize are all set by layoutMetalLayer() just below.)
|
||||
streamView.layer.addSublayer(metal)
|
||||
metalLayer = metal
|
||||
stage2 = pipeline
|
||||
@@ -325,9 +339,20 @@ public final class StreamViewController: UIViewController {
|
||||
layoutMetalLayer()
|
||||
}
|
||||
|
||||
/// Aspect-fit the metal sublayer in the view (the host streams at the client's native mode,
|
||||
/// so this is usually the full bounds). drawableSize is the layer's pixel size; the shader's
|
||||
/// fullscreen triangle scales the decoded texture to fill it.
|
||||
/// The display scale to render the metal drawable at. `traitCollection.displayScale` is the
|
||||
/// canonical render scale and is reliable once the controller is in the hierarchy;
|
||||
/// `view.contentScaleFactor` can read 1.0 before the view attaches to a window/screen, which
|
||||
/// would size the drawable at point resolution → a pixelated, upscaled mess. Falls back to the
|
||||
/// main screen scale if the trait is still unspecified.
|
||||
private var renderScale: CGFloat {
|
||||
let s = traitCollection.displayScale
|
||||
return s > 0 ? s : UIScreen.main.scale
|
||||
}
|
||||
|
||||
/// Position the metal sublayer aspect-fit in the view (the host streams at the client's native
|
||||
/// mode, so this is usually the full bounds). Only the layer FRAME is set here — the presenter
|
||||
/// sizes the drawable to the decoded frame and the layer's contentsGravity (.resizeAspect)
|
||||
/// scales it to this frame via the system compositor (matching stage-1's videoGravity).
|
||||
private func layoutMetalLayer() {
|
||||
guard let metalLayer, let connection else { return }
|
||||
let mode = connection.currentMode()
|
||||
@@ -337,13 +362,11 @@ public final class StreamViewController: UIViewController {
|
||||
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
|
||||
insideRect: bounds)
|
||||
: bounds
|
||||
let scale = streamView.contentScaleFactor
|
||||
CATransaction.begin()
|
||||
CATransaction.setDisableActions(true) // don't animate the resize
|
||||
metalLayer.contentsScale = scale
|
||||
metalLayer.contentsScale = renderScale
|
||||
metalLayer.frame = fit
|
||||
CATransaction.commit()
|
||||
stage2?.setDrawableSize(CGSize(width: fit.width * scale, height: fit.height * scale))
|
||||
}
|
||||
|
||||
private func teardownStage2() {
|
||||
|
||||
Reference in New Issue
Block a user