feat(apple): stage-2 default + pixel-perfect, decode robustness, UI/rumble polish
apple / swift (push) Successful in 1m4s
android / android (push) Successful in 4m33s
ci / rust (push) Successful in 5m4s
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 3m12s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 4s
release / apple (push) Successful in 8m30s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 19s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
ci / bench (push) Successful in 4m45s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m48s
apple / screenshots (push) Successful in 5m43s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m24s
docker / deploy-docs (push) Successful in 19s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m46s

Stream reliability
- Default to the stage-2 presenter (VTDecompressionSession + CAMetalLayer): it detects
  and recovers a wedged decoder, where stage-1's AVSampleBufferDisplayLayer freezes hard
  on a lost HEVC reference frame with no app-side recovery (confirmed Apple limitation).
  Stage 1 is now a DEBUG-only presenter toggle, plus the automatic no-Metal fallback.
- Stage-2 pixel-perfect: render the drawable at the decoded size (shader stays 1:1 =
  identity) and let the layer's contentsGravity scale via the system compositor — the
  same path stage-1's videoGravity used — instead of scaling in-shader.
- Loss recovery in both pumps is now a persistent awaitingIDR want, retried until an IDR
  actually lands, so a keyframe request swallowed by the throttle can't strand a frozen
  frame; 100 ms keyframe throttle to match the Android path.
- Fix "Publishing changes from within view updates": defer the HostStore writes out of
  the .onChange(of: model.phase) callback.
- Move AVAudioSession setActive/setCategory off the main thread (async on a shared serial
  queue) to stop the UI-stall warning.

Controllers
- Rumble: capped-exponential backoff when the gamecontrollerd.haptics XPC breaks (-4811)
  so a transient server interruption self-heals instead of cascading; playsHapticsOnly so
  a controller engine doesn't join the always-active streaming audio session.
- Host cards: iPad pointer "magnet" hover effect; iPhone press scale + light haptic.

UI / design
- Ship Geist (SIL OFL 1.1) as the app font (bundled OTFs + registration), with the
  license surfaced in Acknowledgements.
- Restructure iOS/iPadOS Settings into a category NavigationSplitView; resolution wheel
  with custom-resolution entry; 10-bit HDR toggle in Display.
- Industrial host-card redesign (left-aligned, bold, brand monogram tiles).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-29 20:25:43 +02:00
parent 46b9aa8cf0
commit 4e00037a89
31 changed files with 1221 additions and 297 deletions
@@ -0,0 +1,101 @@
// Geist the punktfunk brand typeface (the same family the website ships). Bundled as static
// OTF weights in this kit's resources and registered with Core Text at first use, so it works
// identically in the Xcode app and the `swift run` dev shell (Bundle.module resolves to the
// package resource bundle in both). Geist Sans carries titles/UI; Geist Mono carries the technical
// readouts host addresses, status labels, the stream-stats HUD for the industrial look.
//
// Licensed under the SIL Open Font License 1.1 (Resources/Fonts/Geist-OFL.txt).
import CoreText
import SwiftUI
#if canImport(UIKit)
import UIKit
#elseif canImport(AppKit)
import AppKit
#endif
public enum BrandFont {
public enum Weight {
case regular, medium, semibold, bold
}
/// PostScript names of the bundled faces (verified from each OTF's name table). Geist Sans only
/// Geist Mono is intentionally not shipped; the app's typeface is Geist Sans throughout.
private static let sansFaces = ["Geist-Regular", "Geist-Medium", "Geist-SemiBold", "Geist-Bold"]
/// Registered exactly once per process a static `let` initializer is run lazily and is
/// guaranteed thread-safe + run-at-most-once by the runtime.
private static let registered: Void = {
for face in sansFaces {
guard let url = Bundle.module.url(
forResource: face, withExtension: "otf", subdirectory: "Fonts") else {
#if DEBUG
print("BrandFont: bundled face \(face).otf not found — text will fall back to system")
#endif
continue
}
var error: Unmanaged<CFError>?
if !CTFontManagerRegisterFontsForURL(url as CFURL, .process, &error) {
#if DEBUG
let message = error?.takeRetainedValue().localizedDescription ?? "unknown error"
print("BrandFont: failed to register \(face): \(message)")
#endif
}
}
}()
/// Force registration before the first `Font.custom` lookup. Cheap to call repeatedly.
public static func registerIfNeeded() { _ = registered }
fileprivate static func sansFace(_ weight: Weight) -> String {
switch weight {
case .regular: return "Geist-Regular"
case .medium: return "Geist-Medium"
case .semibold: return "Geist-SemiBold"
case .bold: return "Geist-Bold"
}
}
}
public extension Color {
/// The punktfunk brand purple (the app-icon lens / website `--brand`). Defined explicitly,
/// independent of the asset-catalog accent `Color.accentColor` resolution is environment- and
/// timing-sensitive (it can fall back to system blue), and the brand mark must never drift.
/// Light: #6656F2, Dark: #8678F5 (the lighter violet reads better on dark surfaces).
static let brand: Color = {
#if canImport(UIKit)
return Color(UIColor { traits in
traits.userInterfaceStyle == .dark
? UIColor(red: 0x86 / 255, green: 0x78 / 255, blue: 0xF5 / 255, alpha: 1)
: UIColor(red: 0x66 / 255, green: 0x56 / 255, blue: 0xF2 / 255, alpha: 1)
})
#elseif canImport(AppKit)
return Color(NSColor(name: nil) { appearance in
appearance.bestMatch(from: [.aqua, .darkAqua]) == .darkAqua
? NSColor(red: 0x86 / 255, green: 0x78 / 255, blue: 0xF5 / 255, alpha: 1)
: NSColor(red: 0x66 / 255, green: 0x56 / 255, blue: 0xF2 / 255, alpha: 1)
})
#else
// Non-Apple fallback: the light brand value, so all branches agree on a canonical color.
return Color(red: 0x66 / 255, green: 0x56 / 255, blue: 0xF2 / 255)
#endif
}()
}
public extension Font {
/// Geist Sans at an explicit point size, scaling with Dynamic Type relative to `textStyle`.
static func geist(
_ size: CGFloat, _ weight: BrandFont.Weight = .regular,
relativeTo textStyle: TextStyle = .body
) -> Font {
BrandFont.registerIfNeeded()
return .custom(BrandFont.sansFace(weight), size: size, relativeTo: textStyle)
}
/// Geist Sans at a FIXED point size that does not scale with Dynamic Type for glyphs pinned
/// inside a fixed-size container (e.g. the monogram tile), where a scaled letter would overflow.
static func geistFixed(_ size: CGFloat, _ weight: BrandFont.Weight = .regular) -> Font {
BrandFont.registerIfNeeded()
return .custom(BrandFont.sansFace(weight), fixedSize: size)
}
}
@@ -22,6 +22,9 @@ public enum DefaultsKey {
public static let speakerUID = "punktfunk.speakerUID"
public static let micUID = "punktfunk.micUID"
public static let presenter = "punktfunk.presenter"
/// Request a 10-bit BT.2020 PQ (HDR10) stream. On by default; only takes effect when the host
/// has HDR content AND this display supports HDR otherwise the stream stays 8-bit SDR.
public static let hdrEnabled = "punktfunk.hdrEnabled"
public static let hosts = "punktfunk.hosts"
/// Client-side cursor mode: "auto" (shown only in gamescope sessions), "always", "never".
public static let cursorMode = "punktfunk.cursorMode"
@@ -68,6 +68,14 @@ private final class RumbleRenderer: @unchecked Sendable {
private var broken = false
/// Last logged active/silent state for a one-line transition log, not per-event spam.
private var wasActive = false
// Backoff after an engine failure. A broken `gamecontrollerd.haptics` XPC connection (CoreHaptics
// -4811 "server connection broke") fails EVERY rebuild until the service relaunches and that
// break fires neither stoppedHandler nor resetHandler, so without a cooldown the next rumble
// update immediately rebuilds into the same dead connection, flooding the log and never
// recovering. Delay the next setup() growing 0.5124 s on repeated failure and clear it
// the moment a player runs cleanly (or the controller changes).
private var retryAfter = Date.distantPast
private var consecutiveFailures = 0
/// CHHapticEvent sharpness = actuator frequency. A DualSense's voice-coil motors need a
/// defined frequency to move at all an intensity-only event (no sharpness) left them
@@ -91,6 +99,8 @@ private final class RumbleRenderer: @unchecked Sendable {
self.closeHID()
self.controller = c
self.broken = false
self.consecutiveFailures = 0
self.retryAfter = .distantPast
_ = self.openHIDIfDualSense(c)
onBackend?(self.backendNote(for: c))
}
@@ -108,7 +118,7 @@ private final class RumbleRenderer: @unchecked Sendable {
// other pad (and for a DualSense whose HID device could not be opened).
if self.hidRumble(low: lowAmp, high: highAmp) { return }
guard !self.broken else { return }
if active, self.low == nil, self.high == nil {
if active, self.low == nil, self.high == nil, Date() >= self.retryAfter {
self.setup()
}
let ok: Bool
@@ -124,8 +134,15 @@ private final class RumbleRenderer: @unchecked Sendable {
}
// Rebuild on the next nonzero amplitude if an engine errored and tear down OUTSIDE
// the `inout` accesses above, so teardown() never mutates a motor that a `drive` call
// still holds an exclusive reference to.
if !ok { self.teardown() }
// still holds an exclusive reference to. Back off so a broken XPC isn't re-hit every
// update; once a player is actually running the path has recovered, so clear the backoff.
if !ok {
self.teardown()
self.scheduleRetryBackoff()
} else if self.low?.player != nil || self.high?.player != nil {
self.consecutiveFailures = 0
self.retryAfter = .distantPast
}
}
}
@@ -157,14 +174,29 @@ private final class RumbleRenderer: @unchecked Sendable {
low = makeMotor(haptics, .default)
}
if low == nil, high == nil {
// Haptics present but no engine could be built right now (server busy / a transient
// error). Do NOT latch broken the next nonzero amplitude retries setup().
log.warning("rumble: haptics present but engine setup failed — will retry on next rumble")
// Haptics present but no engine could be built right now (server busy / XPC broken). Do
// NOT latch broken back off and the next nonzero amplitude past the cooldown retries.
log.warning("rumble: haptics present but engine setup failed — backing off, will retry")
scheduleRetryBackoff()
}
}
/// Push the next engine-build attempt out after a failure (capped exponential backoff), so a
/// broken `gamecontrollerd.haptics` connection gets time to relaunch instead of being re-hit on
/// every rumble update.
private func scheduleRetryBackoff() {
consecutiveFailures += 1
let shift = min(consecutiveFailures - 1, 4)
retryAfter = Date().addingTimeInterval(min(0.5 * Double(1 << shift), 4))
}
private func makeMotor(_ haptics: GCDeviceHaptics, _ locality: GCHapticsLocality) -> Motor? {
guard let engine = haptics.createEngine(withLocality: locality) else { return nil }
// A controller's motors carry no audio, so keep this engine OUT of the app's audio session
// (the default is to join it). Streaming keeps an AVAudioSession active the whole time;
// letting a haptics-only engine join it is a needless coupling that can get its
// gamecontrollerd XPC connection interrupted (the repeated -4811 server-connection breaks).
engine.playsHapticsOnly = true
// The haptic server can stop or reset the engine out from under us app backgrounding, an
// audio-session interruption (a call, Siri, another audio app), or a server crash. Left
// unhandled the players go dead and every later rumble throws, latching rumble off for the
@@ -27,6 +27,17 @@ public enum Licenses {
+ apache
}
/// The bundled brand typeface (Geist Sans + Geist Mono) SIL Open Font License 1.1. The
/// license file ships alongside the OTFs in `Resources/Fonts/`, satisfying the OFL's
/// distribution requirement; this surfaces it in the Acknowledgements screen too.
public static var fontLicense: String {
guard let url = Bundle.module.url(
forResource: "Geist-OFL", withExtension: "txt", subdirectory: "Fonts"),
let text = try? String(contentsOf: url, encoding: .utf8)
else { return "" }
return text
}
/// Third-party software notices for the linked Rust crates (generated by
/// `scripts/gen-third-party-notices.sh`).
public static var thirdPartyNotices: String {
@@ -11,6 +11,9 @@ import CoreGraphics
import CoreVideo
import Metal
import QuartzCore
import os
private let presenterLog = Logger(subsystem: "io.unom.punktfunk", category: "presenter")
/// Runtime-compiled (no metallib build step needed in SwiftPM): a fullscreen triangle and a
/// BT.709 limited-range NV12RGB fragment shader. uv.y is flipped (1 - p.y) so the top-left-
@@ -30,11 +33,44 @@ vertex VOut pf_vtx(uint vid [[vertex_id]]) {
return o;
}
// Bicubic (Catmull-Rom) sampling of the single-channel luma plane. When the drawable is larger
// than the decoded frame (a window/view bigger than the host's fixed mode), a bilinear upscale
// looks soft; Catmull-Rom keeps edges crisp — matching AVSampleBufferDisplayLayer's (stage-1)
// scaler — and reduces to the exact texel at 1:1, so a native-resolution present stays pixel-exact.
// Nine bilinear taps (TheRealMJP's optimisation of the 16-tap kernel); `s` MUST be a linear
// sampler. Luma carries the perceived detail, so only it gets bicubic; chroma stays bilinear.
float catmullRomLuma(texture2d<float> tex, sampler s, float2 uv) {
float2 texSize = float2(tex.get_width(), tex.get_height());
float2 samplePos = uv * texSize;
float2 tc1 = floor(samplePos - 0.5) + 0.5;
float2 f = samplePos - tc1;
float2 w0 = f * (-0.5 + f * (1.0 - 0.5 * f));
float2 w1 = 1.0 + f * f * (-2.5 + 1.5 * f);
float2 w2 = f * (0.5 + f * (2.0 - 1.5 * f));
float2 w3 = f * f * (-0.5 + 0.5 * f);
float2 w12 = w1 + w2;
float2 off12 = w2 / w12;
float2 tc0 = (tc1 - 1.0) / texSize;
float2 tc3 = (tc1 + 2.0) / texSize;
float2 tc12 = (tc1 + off12) / texSize;
float r = 0.0;
r += tex.sample(s, float2(tc0.x, tc0.y)).r * (w0.x * w0.y);
r += tex.sample(s, float2(tc12.x, tc0.y)).r * (w12.x * w0.y);
r += tex.sample(s, float2(tc3.x, tc0.y)).r * (w3.x * w0.y);
r += tex.sample(s, float2(tc0.x, tc12.y)).r * (w0.x * w12.y);
r += tex.sample(s, float2(tc12.x, tc12.y)).r * (w12.x * w12.y);
r += tex.sample(s, float2(tc3.x, tc12.y)).r * (w3.x * w12.y);
r += tex.sample(s, float2(tc0.x, tc3.y)).r * (w0.x * w3.y);
r += tex.sample(s, float2(tc12.x, tc3.y)).r * (w12.x * w3.y);
r += tex.sample(s, float2(tc3.x, tc3.y)).r * (w3.x * w3.y);
return r;
}
fragment float4 pf_frag(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> chromaTex [[texture(1)]]) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
float y = lumaTex.sample(s, in.uv).r;
float y = catmullRomLuma(lumaTex, s, in.uv);
float2 c = chromaTex.sample(s, in.uv).rg;
// BT.709, 8-bit limited (video) range → full-range RGB.
y = (y - 16.0/255.0) * (255.0/219.0);
@@ -55,7 +91,7 @@ fragment float4 pf_frag_hdr(VOut in [[stage_in]],
texture2d<float> lumaTex [[texture(0)]],
texture2d<float> chromaTex [[texture(1)]]) {
constexpr sampler s(filter::linear, address::clamp_to_edge);
float y = lumaTex.sample(s, in.uv).r;
float y = catmullRomLuma(lumaTex, s, in.uv);
float2 c = chromaTex.sample(s, in.uv).rg;
// BT.2020 10-bit limited (video) range → full-range PQ R'G'B'.
y = (y - 64.0/1023.0) * (1023.0/876.0);
@@ -81,6 +117,11 @@ public final class MetalVideoPresenter {
private var textureCache: CVMetalTextureCache?
/// Current layer configuration switched lazily in `configure(hdr:)` when a frame's mode differs.
private var hdrActive = false
#if DEBUG
/// Last logged "decodeddrawable" signature, so the diagnostic logs only when a size changes
/// (on first frame, a resize, or a host Reconfigure) instead of every frame.
private var lastSizeSig = ""
#endif
/// nil if Metal is unavailable (no GPU / a headless CI) the caller falls back to stage-1.
public init?() {
@@ -113,6 +154,12 @@ public final class MetalVideoPresenter {
layer.pixelFormat = .bgra8Unorm
layer.framebufferOnly = true
layer.isOpaque = true
// Render the drawable at the DECODED frame's resolution (set per-frame in `render`) and let
// the system compositor scale it to the layer's bounds the same `.resizeAspect` path
// stage-1's AVSampleBufferDisplayLayer (videoGravity) uses, so stage-2 matches its sharpness.
// A native-resolution present is then pixel-exact (1:1, no shader scaling), and any display
// scaling uses the system's high-quality scaler rather than the in-shader bicubic.
layer.contentsGravity = .resizeAspect
// Triple-buffer: more in-flight drawables before `nextDrawable()` (called on the
// display-link / MAIN thread) has to block waiting for one to free.
layer.maximumDrawableCount = 3
@@ -129,12 +176,6 @@ public final class MetalVideoPresenter {
self.layer = layer
}
/// Track the stream mode (the host can Reconfigure mid-stream). Size is in pixels.
public func setDrawableSize(_ size: CGSize) {
guard size.width > 0, size.height > 0 else { return }
if layer.drawableSize != size { layer.drawableSize = size }
}
/// Reconfigure the layer for SDR or HDR when the stream mode flips (HDR toggle). HDR uses an
/// rgba16Float drawable + a BT.2020 PQ colour space + EDR, so the compositor PQ-maps to the
/// display; SDR uses the plain 8-bit sRGB path. Main-thread only (called from `render`).
@@ -171,13 +212,33 @@ public final class MetalVideoPresenter {
let chroma = makeTexture(pixelBuffer, plane: 1, format: chromaFmt, cache: textureCache)
else { return false }
// The hosting view owns drawableSize (aspect-fit to its bounds); skip until it's laid
// out. The fullscreen triangle scales the decoded texture to fill the drawable.
guard layer.drawableSize.width > 0, layer.drawableSize.height > 0,
let drawable = layer.nextDrawable(),
// Size the drawable to the decoded frame so the fullscreen triangle samples the texture 1:1
// (pixel-exact); the layer's contentsGravity then scales it to the on-screen bounds via the
// system compositor (matching stage-1). Re-set only on a change (first frame / Reconfigure).
let decodedSize = CGSize(
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
if layer.drawableSize != decodedSize { layer.drawableSize = decodedSize }
guard let drawable = layer.nextDrawable(),
let commandBuffer = queue.makeCommandBuffer()
else { return false }
#if DEBUG
// Diagnose sharpness: decoded should equal the drawable (the shader is 1:1); the layer's
// bounds may differ (the system scales). Logged only when a size changes.
let decodedW = Int(decodedSize.width)
let decodedH = Int(decodedSize.height)
let sig = "\(decodedW)x\(decodedH)|\(Int(layer.drawableSize.width))x\(Int(layer.drawableSize.height))"
if sig != lastSizeSig {
lastSizeSig = sig
let msg = "stage2: decoded \(decodedW)x\(decodedH) → drawable "
+ "\(Int(layer.drawableSize.width))x\(Int(layer.drawableSize.height)) "
+ "(texture \(drawable.texture.width)x\(drawable.texture.height), "
+ "contentsScale \(layer.contentsScale), "
+ "layerBounds \(Int(layer.bounds.width))x\(Int(layer.bounds.height)))"
presenterLog.info("\(msg, privacy: .public)")
}
#endif
let pass = MTLRenderPassDescriptor()
pass.colorAttachments[0].texture = drawable.texture
pass.colorAttachments[0].loadAction = .clear
@@ -0,0 +1,93 @@
Copyright 2024 The Geist Project Authors (https://github.com/vercel/geist-font)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://openfontlicense.org
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
@@ -177,6 +177,16 @@ public final class SessionAudio {
private var playbackEngine: AVAudioEngine?
private var captureEngine: AVAudioEngine?
private var drainStarted = false
#if !os(macOS)
/// AVAudioSession `setCategory`/`setActive` are synchronous and block on the audio server, so
/// they must not run on the main thread (UI stall AVFoundation warns about it). PROCESS-WIDE
/// (static) so every SessionAudio shares one serial queue: the AVAudioSession is a process
/// singleton, and across a reconnect the old session's deactivate must be ordered before the
/// new session's activate (a per-instance queue would let them race and leave the new session's
/// audio deactivated). stop() enqueues its deactivate promptly so it lands before the next
/// session's activate.
private static let sessionQueue = DispatchQueue(label: "io.unom.punktfunk.audio.session")
#endif
public init(connection: PunktfunkConnection) {
self.connection = connection
@@ -189,37 +199,60 @@ public final class SessionAudio {
flag.stop()
}
/// Start playback (and, if enabled+authorized, the mic uplink). Empty UIDs = system
/// default device; on iOS the UIDs are ignored entirely (routes are
/// AVAudioSession-managed). Main thread (engine setup); returns after the engines
/// start the mic may start slightly later if the permission prompt is pending.
/// Start playback (and, if enabled+authorized, the mic uplink). Empty UIDs = system default
/// device; on iOS the UIDs are ignored entirely (routes are AVAudioSession-managed). On macOS
/// the engines start synchronously on the caller's (main) thread. On iOS/tvOS start() is
/// ASYNCHRONOUS: it activates the AVAudioSession off the main thread, then starts the engines on
/// a later main-queue hop (gated by `!flag.isStopped`) so playback is live shortly after, not
/// on return. The mic may start later still if the permission prompt is pending.
public func start(speakerUID: String, micUID: String, micEnabled: Bool) {
#if os(iOS)
// Route + policy live in the session, not per-engine: stereo playback, mic
// capture when enabled, Bluetooth allowed. Failure is non-fatal (defaults).
#if os(macOS)
// No AVAudioSession on macOS start the engines directly (caller's thread, as before).
startEngines(speakerUID: speakerUID, micUID: micUID, micEnabled: micEnabled)
#else
// Configure + activate the session OFF the main thread (it blocks on the audio server),
// then start the engines back on the main thread once it's active engine routing/format
// depend on the active session. A stop() racing in between is caught by the flag guard.
Self.sessionQueue.async { [weak self] in
guard let self else { return }
self.activateAudioSession(micEnabled: micEnabled)
DispatchQueue.main.async { [weak self] in
guard let self, !self.flag.isStopped else { return }
self.startEngines(speakerUID: speakerUID, micUID: micUID, micEnabled: micEnabled)
}
}
#endif
}
#if !os(macOS)
/// Route + policy live in the session, not per-engine: stereo playback, mic capture when
/// enabled, Bluetooth allowed. Failure is non-fatal (defaults). Runs on `sessionQueue`.
private func activateAudioSession(micEnabled: Bool) {
let session = AVAudioSession.sharedInstance()
do {
#if os(iOS)
if micEnabled {
// .defaultToSpeaker: .playAndRecord otherwise routes to the iPhone
// EARPIECE; only affects the built-in route (headphones/BT still win).
// .defaultToSpeaker: .playAndRecord otherwise routes to the iPhone EARPIECE; only
// affects the built-in route (headphones/BT still win).
try session.setCategory(
.playAndRecord, mode: .default,
options: [.allowBluetoothA2DP, .defaultToSpeaker])
} else {
try session.setCategory(.playback, mode: .default)
}
#else // tvOS no app-accessible mic
try session.setCategory(.playback, mode: .default)
#endif
try session.setActive(true)
} catch {
log.warning("AVAudioSession setup failed: \(error.localizedDescription)")
}
#elseif os(tvOS)
do {
try AVAudioSession.sharedInstance().setCategory(.playback, mode: .default)
try AVAudioSession.sharedInstance().setActive(true)
} catch {
log.warning("AVAudioSession setup failed: \(error.localizedDescription)")
}
#endif
}
#endif
/// Build + start the playback engine (and the mic uplink when enabled + authorized). Main
/// thread (engine setup); on iOS/tvOS the session is already active by the time this runs.
private func startEngines(speakerUID: String, micUID: String, micEnabled: Bool) {
startPlayback(speakerUID: speakerUID)
#if os(tvOS)
// No app-accessible microphone input on tvOS playback only.
@@ -258,19 +291,24 @@ public final class SessionAudio {
capture.stop()
}
playback?.stop()
#if !os(macOS)
// Release the session so audio we interrupted (Music, podcasts) gets its resume cue. Like
// activation, setActive is synchronous/blocking run it on the shared serial session queue
// (off the main thread). Enqueued HERE engines already stopped, and BEFORE the drain wait
// below so across a reconnect it lands ahead of the next session's activate on the shared
// queue (otherwise a deferred deactivate could deactivate the new session). Fire-and-forget.
Self.sessionQueue.async {
do {
try AVAudioSession.sharedInstance().setActive(
false, options: .notifyOthersOnDeactivation)
} catch {
log.warning("AVAudioSession deactivation failed: \(error.localizedDescription)")
}
}
#endif
if wasDraining {
_ = drainDone.wait(timeout: .now() + .milliseconds(400))
}
#if !os(macOS)
// Release the session so audio we interrupted (Music, podcasts) gets its
// resume cue.
do {
try AVAudioSession.sharedInstance().setActive(
false, options: .notifyOthersOnDeactivation)
} catch {
log.warning("AVAudioSession deactivation failed: \(error.localizedDescription)")
}
#endif
}
// MARK: - Playback (host speaker)
@@ -4,7 +4,7 @@
// capturepresent. Mirrors StreamPump's lifecycle (one per start; cancel is permanent).
//
// Threading: the pump runs on its own thread; the decoder callback on a VT thread; `renderTick`
// + `setDrawableSize` + `start`/`stop` on the MAIN thread (the view's CADisplayLink fires there).
// + `start`/`stop` on the MAIN thread (the view's CADisplayLink fires there).
// Only the ring + decoder cross threads and both are internally locked.
#if canImport(Metal) && canImport(QuartzCore)
@@ -60,7 +60,7 @@ private final class KeyframeRecovery: @unchecked Sendable {
func request() {
lock.lock()
let now = DispatchTime.now().uptimeNanoseconds
let due = lastNs == 0 || now &- lastNs > 250_000_000 // 250 ms since the last request
let due = lastNs == 0 || now &- lastNs > 100_000_000 // 100 ms since the last request (matches Android)
if due { lastNs = now }
let conn = due ? connection : nil
lock.unlock()
@@ -114,20 +114,24 @@ public final class Stage2Pipeline {
let thread = Thread {
var format: CMVideoFormatDescription?
var lastFramesDropped = connection.framesDropped()
// Persistent recovery WANT, not a one-shot edge (see StreamPump for the full rationale):
// the old code advanced lastFramesDropped on the same edge it called recovery.request(),
// so a request swallowed by the throttle (the lost recovery IDR being pruned within the
// window) was never re-sent and the picture stayed frozen. Keep asking until an IDR lands.
var awaitingIDR = false
while token.isLive {
do {
// Loss recovery (the primary recovery path). The reassembler drops unrecoverable
// AUs (framesDropped) and the decoder then conceals the reference-missing delta
// frames that follow often rendering them WITHOUT an error callback so the
// onDecodeError trigger rarely fires after a real network blip. Ask the host for
// a fresh IDR whenever the drop count climbs (throttled in KeyframeRecovery).
// Polled every iteration so a total-loss drought recovers the moment packets
// resume and the reassembler counts the gap.
// Loss recovery (the primary path). The reassembler drops unrecoverable AUs
// (framesDropped) and the decoder conceals the reference-missing deltas that
// follow often WITHOUT an error callback so key off the drop count climbing,
// then keep asking (awaitingIDR) until a fresh IDR re-anchors decode. Polled every
// iteration so a total-loss drought recovers the moment packets resume.
let dropped = connection.framesDropped()
if dropped > lastFramesDropped {
lastFramesDropped = dropped
recovery.request()
awaitingIDR = true
}
if awaitingIDR { recovery.request() }
// Drain any HDR mastering-metadata update (0xCE) and hand it to the decoder, which
// attaches it to subsequent HDR frames. Non-blocking; only HDR sessions emit these.
if connection.isHDR, let meta = try? connection.nextHdrMeta(timeoutMs: 0) {
@@ -136,15 +140,16 @@ public final class Stage2Pipeline {
guard let au = try connection.nextAU(timeoutMs: 100) else { continue }
onFrame?(au)
if let f = AnnexB.formatDescription(fromIDR: au.data) {
format = f // refreshed on every IDR (mode changes included)
format = f // refreshed on every IDR (mode changes included)
awaitingIDR = false // a fresh IDR re-anchored decode recovery complete
}
guard let f = format, token.isLive else { continue }
if !decoder.decode(au: au, format: f) {
// Submit/decoder error: drop the session and re-gate on the next IDR's
// in-band parameter sets (a delta frame can't recover) stage-1's policy
// and ask the host for that IDR now (infinite GOP; throttled).
// in-band parameter sets (a delta frame can't recover) stage-1's policy
// and keep asking for that IDR (infinite GOP) until one re-anchors decode.
decoder.reset()
recovery.request()
awaitingIDR = true
}
} catch {
if token.isLive { onSessionEnd?() }
@@ -166,11 +171,6 @@ public final class Stage2Pipeline {
presentMeter.record(ptsNs: frame.ptsNs, atNs: targetPresentNs, offsetNs: offsetNs)
}
/// MAIN thread. Keep the drawable matched to the negotiated mode (host can Reconfigure).
public func setDrawableSize(_ size: CGSize) {
presenter.setDrawableSize(size)
}
/// Stop the pump ( one poll timeout) and drop the decode session. Does not close the
/// connection. A restart needs a fresh Stage2Pipeline (cancel is permanent).
public func stop() {
@@ -6,6 +6,9 @@
import AVFoundation
import Foundation
import os
private let pumpLog = Logger(subsystem: "io.unom.punktfunk", category: "video")
/// Cancellation handle owned by exactly one pump thread a restart hands the old pump
/// its own token, so it can never be revived by a newer start().
@@ -47,44 +50,74 @@ final class StreamPump {
var format: CMVideoFormatDescription?
var lastKeyframeRequest = Date.distantPast
var lastFramesDropped = connection.framesDropped()
// Coalesced host keyframe request: the decode stays wedged for several frames until
// the IDR lands, so requesting on every frame would flood the control stream.
// Recovery is a persistent WANT, not a one-shot edge: set it on detected loss (or a
// decoder reset), retry the throttled request EVERY iteration, and clear it only when a
// fresh IDR actually re-anchors decode. The old code advanced `lastFramesDropped` on the
// same edge it fired the throttled request so a request swallowed by the throttle (a
// second drop within the window, e.g. the lost recovery IDR itself being pruned) was
// never re-sent: the counter went flat, the climb never re-fired, and the picture stayed
// frozen for good while audio kept playing. The iPhone's lossy Wi-Fi hits this where the
// Mac's Ethernet never does.
var awaitingIDR = false
var awaitingSince = Date.distantPast // when the current recovery began (for the resume log)
var wasFailed = false
// Coalesced host keyframe request. 100 ms throttle (matches the working Android path):
// fast enough that a lost recovery IDR is re-requested promptly, bounded so a sustained
// freeze can't flood the control stream.
func requestKeyframeThrottled() {
let now = Date()
if now.timeIntervalSince(lastKeyframeRequest) > 0.25 {
if now.timeIntervalSince(lastKeyframeRequest) > 0.1 {
connection.requestKeyframe()
lastKeyframeRequest = now
}
}
while token.isLive {
do {
// Loss recovery (the primary recovery path). Under the host's infinite GOP the
// only recovery keyframe is one we request. The reassembler drops unrecoverable
// AUs (framesDropped); the decoder then *conceals* the reference-missing delta
// frames that follow a frozen / garbage picture, WITHOUT flipping the layer to
// .failed so the .failed check below rarely fires after a real network blip.
// Ask the host for a fresh IDR whenever the drop count climbs. Polled every
// iteration (not just per AU) so a total-loss drought still recovers the moment
// packets resume and the reassembler counts the gap.
// Loss recovery (the primary path). Under the host's infinite GOP the only
// recovery keyframe is one we request. The reassembler drops unrecoverable AUs
// (framesDropped); the decoder then *conceals* the reference-missing deltas a
// frozen / garbage picture that never flips the layer to .failed so key off the
// drop count climbing, then keep asking (awaitingIDR) until an IDR lands. Polled
// every iteration so a total-loss drought still recovers when packets resume.
let dropped = connection.framesDropped()
if dropped > lastFramesDropped {
// Log only on the falsetrue transition (once per recovery cycle), not per
// dropped AU, so heavy loss doesn't spam the log.
if !awaitingIDR {
awaitingSince = Date()
pumpLog.notice(
"video: unrecoverable drop (framesDropped=\(dropped, privacy: .public)) — requesting recovery IDR")
}
lastFramesDropped = dropped
requestKeyframeThrottled()
awaitingIDR = true
}
if awaitingIDR { requestKeyframeThrottled() }
guard let au = try connection.nextAU(timeoutMs: 100) else { continue }
onFrame?(au)
if let f = AnnexB.formatDescription(fromIDR: au.data) {
format = f // refreshed on every IDR (mode changes included)
let idrFormat = AnnexB.formatDescription(fromIDR: au.data)
if let f = idrFormat {
format = f // refreshed on every IDR (mode changes included)
if awaitingIDR {
let ms = Int(Date().timeIntervalSince(awaitingSince) * 1000)
pumpLog.notice("video: recovery IDR received — resumed after \(ms, privacy: .public) ms")
}
awaitingIDR = false // a fresh IDR re-anchored decode recovery complete
}
if layer.status == .failed {
let failed = layer.status == .failed
if failed {
// Decode wedged hard (the cold-first-connect case a lost/corrupt opening
// IDR): flush and re-gate on the next in-band parameter sets (resuming with
// a delta frame can't recover), AND ask the host for a fresh IDR. Throttled:
// the layer stays .failed across several polls until the IDR lands.
// IDR): flush and, unless THIS AU is the recovering IDR (re-anchored above),
// re-gate on the next in-band parameter sets and keep asking enqueuing a
// delta into a failed layer can't recover it.
if !wasFailed { pumpLog.warning("video: display layer .failed — flushing + re-anchoring") }
layer.flush()
format = AnnexB.formatDescription(fromIDR: au.data)
requestKeyframeThrottled()
if idrFormat == nil {
format = nil
awaitingIDR = true
}
}
wasFailed = failed
guard let f = format,
let sample = AnnexB.sampleBuffer(au: au, format: f),
token.isLive // don't enqueue a stale frame after a restart
@@ -245,6 +245,15 @@ public final class StreamLayerView: NSView {
layoutMetalLayer() // keep the stage-2 sublayer aspect-fit to the view
}
public override func setFrameSize(_ newSize: NSSize) {
super.setFrameSize(newSize)
// `layout()` isn't guaranteed on a manual-frame (no-Auto-Layout) live resize, so the
// stage-2 metal sublayer's drawableSize could stay at the old size while the view grows
// the compositor then upscales a too-small drawable and the video turns blocky. Resize the
// drawable here too so it always tracks the window's pixel size (no stale upscale).
layoutMetalLayer()
}
// MARK: - Capture state machine
/// Clicking into the video engages capture; that click is local (engagement), so
@@ -549,10 +558,17 @@ public final class StreamLayerView: NSView {
cursorVisible = false
_ = connection.resolvedCompositor // (was: Auto gamescope; kept to document intent)
// Presenter choice default stage-1 (the known-good AVSampleBufferDisplayLayer). Stage-2
// (`punktfunk.presenter == "stage2"`) takes explicit VTDecompressionSession decode + a
// CAMetalLayer/display-link present; it falls back here if Metal can't be set up.
if UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage2",
// Presenter choice stage-2 is the DEFAULT (explicit VTDecompressionSession decode + a
// CAMetalLayer/display-link present): it can detect + recover a wedged decoder where
// stage-1's AVSampleBufferDisplayLayer freezes hard on a lost HEVC reference. Stage-1 is
// reachable only via the DEBUG presenter toggle; release always takes stage-2 (the stage-1
// pump below stays the automatic fallback if Metal is missing).
#if DEBUG
let forceStage1 = UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage1"
#else
let forceStage1 = false
#endif
if !forceStage1,
let meter = presentMeter,
let pipeline = Stage2Pipeline(presentMeter: meter) {
startStage2(pipeline, connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
@@ -593,9 +609,11 @@ public final class StreamLayerView: NSView {
targetPresentNs: Stage2Pipeline.realtimeNs(forDisplayLinkTimestamp: link.targetTimestamp))
}
/// Aspect-fit the metal sublayer in the view (the host streams at the client's native mode,
/// so this is usually the full bounds; it letterboxes a resized window). drawableSize is the
/// layer's pixel size the fullscreen-triangle shader scales the decoded texture to fill it.
/// Position the metal sublayer aspect-fit in the view (the host streams at the client's native
/// mode, so this is usually the full bounds; it letterboxes a resized window). Only the layer
/// FRAME is set here the presenter sizes the drawable to the decoded frame and the layer's
/// contentsGravity (.resizeAspect) scales it to this frame via the system compositor, so a
/// resized window rescales through the system's filter (matching stage-1) instead of the shader.
private func layoutMetalLayer() {
guard let metalLayer, let connection else { return }
let mode = connection.currentMode()
@@ -604,14 +622,12 @@ public final class StreamLayerView: NSView {
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
insideRect: bounds)
: bounds
let scale = window?.backingScaleFactor ?? 1
// No implicit resize animation; refresh contentsScale on a retinanon-retina move.
CATransaction.begin()
CATransaction.setDisableActions(true)
metalLayer.contentsScale = scale
metalLayer.contentsScale = window?.backingScaleFactor ?? 1
metalLayer.frame = fit
CATransaction.commit()
stage2?.setDrawableSize(CGSize(width: fit.width * scale, height: fit.height * scale))
}
public override func viewDidChangeBackingProperties() {
@@ -136,6 +136,13 @@ public final class StreamViewController: UIViewController {
public override func loadView() {
view = StreamLayerUIView()
// Re-size the stage-2 drawable if the display scale changes without a bounds change (e.g.
// moving to an external display at a different scale) the iOS analogue of macOS's
// viewDidChangeBackingProperties relayout. The handler takes the VC as its argument, so it
// doesn't capture self (no retain cycle with the registration).
registerForTraitChanges([UITraitDisplayScale.self]) { (vc: StreamViewController, _) in
vc.layoutMetalLayer()
}
#if os(iOS)
// Hide the iPadOS cursor while it hovers the video: the host renders its own
// cursor from our deltas, so the local one only diverges from it. This hides the
@@ -219,10 +226,17 @@ public final class StreamViewController: UIViewController {
inputCapture = capture
#endif
// Presenter choice default stage-1 (the known-good AVSampleBufferDisplayLayer). Stage-2
// (`punktfunk.presenter == "stage2"`) takes VTDecompressionSession decode + a
// CAMetalLayer/display-link present; falls back here if Metal can't be set up.
if UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage2",
// Presenter choice stage-2 is the DEFAULT (VTDecompressionSession decode + a
// CAMetalLayer/display-link present): it can detect + recover a wedged decoder, where
// stage-1's AVSampleBufferDisplayLayer freezes hard on a lost HEVC reference frame with no
// way to recover. Stage-1 is reachable only via the DEBUG presenter toggle; release always
// takes stage-2 (the stage-1 pump below stays the automatic fallback if Metal is missing).
#if DEBUG
let forceStage1 = UserDefaults.standard.string(forKey: DefaultsKey.presenter) == "stage1"
#else
let forceStage1 = false
#endif
if !forceStage1,
let meter = presentMeter,
let pipeline = Stage2Pipeline(presentMeter: meter) {
startStage2(pipeline, connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
@@ -300,8 +314,8 @@ public final class StreamViewController: UIViewController {
onFrame: (@Sendable (AccessUnit) -> Void)?, onSessionEnd: (@Sendable () -> Void)?
) {
let metal = pipeline.layer
metal.contentsScale = streamView.contentScaleFactor
// Composites OVER the idle (un-enqueued in stage-2) AVSampleBufferDisplayLayer base.
// (contentsScale + frame + drawableSize are all set by layoutMetalLayer() just below.)
streamView.layer.addSublayer(metal)
metalLayer = metal
stage2 = pipeline
@@ -325,9 +339,20 @@ public final class StreamViewController: UIViewController {
layoutMetalLayer()
}
/// Aspect-fit the metal sublayer in the view (the host streams at the client's native mode,
/// so this is usually the full bounds). drawableSize is the layer's pixel size; the shader's
/// fullscreen triangle scales the decoded texture to fill it.
/// The display scale to render the metal drawable at. `traitCollection.displayScale` is the
/// canonical render scale and is reliable once the controller is in the hierarchy;
/// `view.contentScaleFactor` can read 1.0 before the view attaches to a window/screen, which
/// would size the drawable at point resolution a pixelated, upscaled mess. Falls back to the
/// main screen scale if the trait is still unspecified.
private var renderScale: CGFloat {
let s = traitCollection.displayScale
return s > 0 ? s : UIScreen.main.scale
}
/// Position the metal sublayer aspect-fit in the view (the host streams at the client's native
/// mode, so this is usually the full bounds). Only the layer FRAME is set here the presenter
/// sizes the drawable to the decoded frame and the layer's contentsGravity (.resizeAspect)
/// scales it to this frame via the system compositor (matching stage-1's videoGravity).
private func layoutMetalLayer() {
guard let metalLayer, let connection else { return }
let mode = connection.currentMode()
@@ -337,13 +362,11 @@ public final class StreamViewController: UIViewController {
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
insideRect: bounds)
: bounds
let scale = streamView.contentScaleFactor
CATransaction.begin()
CATransaction.setDisableActions(true) // don't animate the resize
metalLayer.contentsScale = scale
metalLayer.contentsScale = renderScale
metalLayer.frame = fit
CATransaction.commit()
stage2?.setDrawableSize(CGSize(width: fit.width * scale, height: fit.height * scale))
}
private func teardownStage2() {