feat(apple): PyroWave Phase 5 — native Metal decode on Mac / Apple TV / iPad (§4.7)
ci / web (push) Successful in 1m3s
ci / docs-site (push) Successful in 58s
decky / build-publish (push) Successful in 16s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 43s
ci / bench (push) Successful in 5m21s
arch / build-publish (push) Successful in 10m58s
docker / deploy-docs (push) Successful in 28s
android / android (push) Successful in 13m23s
deb / build-publish (push) Successful in 12m58s
ci / rust (push) Successful in 18m5s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m31s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m56s
flatpak / build-publish (push) Successful in 5m53s
windows-host / package (push) Successful in 14m37s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m9s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m15s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m12s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 6m6s
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
release / apple (push) Has been cancelled
ci / web (push) Successful in 1m3s
ci / docs-site (push) Successful in 58s
decky / build-publish (push) Successful in 16s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 43s
ci / bench (push) Successful in 5m21s
arch / build-publish (push) Successful in 10m58s
docker / deploy-docs (push) Successful in 28s
android / android (push) Successful in 13m23s
deb / build-publish (push) Successful in 12m58s
ci / rust (push) Successful in 18m5s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m31s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m56s
flatpak / build-publish (push) Successful in 5m53s
windows-host / package (push) Successful in 14m37s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m9s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m15s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m12s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 6m6s
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
release / apple (push) Has been cancelled
The Apple client now decodes PyroWave natively on the presenter's own MTLDevice —
no MoltenVK, no upstream C++ in the app. Completes and wires up the decoder whose
early working-tree snapshot rode along in 9127c346:
- MetalWaveletShaders.swift: wavelet_dequant + idwt hand-ported from the vendored
GLSL (STORAGE_MODE 0 only; subgroup scans → 32-wide simdgroups; DCShift spec
constant → function constant; precision-1 split: fp16 levels 0-1 / fp32 2-4).
- MetalWaveletDecoder.swift: Swift reimplementation of push_packet/decode_packet
incl. the Phase-4 chunk-aligned window walk (FRAG chains, zeroed missing shards,
the >half-blocks partial rule), init_block_meta's block-index space, and the
42-dequant + 13-idwt dispatch structure with encoder-boundary barriers. SOF-dims
changes rebuild the size-dependent resources, which is also the mid-stream
resize path. Ring of 4 output plane sets on the presenter's queue.
- Presenter: pf_frag_planar (3xR8, the planar_csc.frag twin) + renderPlanar with
a shared present tail; ReadyFrame carries an image enum (.video | .planar).
- Stage2Pipeline: a dedicated PyroWave pump — no VideoToolbox machinery, no
keyframe/re-anchor recovery (all-intra; partials render as localized blur by
design), newest-frame-index staleness guard for late partials.
- Opt-in: "PyroWave (wired LAN)" codec entry (probe-gated, ≈A13 floor via a real
kernel-compile probe), selecting it advertises + prefers the codec and forces
the session SDR (HDR/10-bit/4:4:4 caps dropped, plan contract).
- Core ABI: punktfunk_connection_shard_payload() — the Welcome's negotiated shard
payload, needed by native decoders to walk chunk-aligned AUs.
- Validation: golden fixtures generated by the host encoder + upstream's own
decoder (pyrowave_dump_golden, RTX 5070 Ti); the Metal decode PSNR-matches at
77-88 dB across all planes for dense AND chunk-aligned AUs, and a hole-punched
partial still decodes. Parser unit tests cover the window walk, FRAG chains,
broken chains, the half-blocks gate, and the block-index layout.
Tests: apple 134 green (mac; iOS/tvOS build), host 312 w/ pyrowave on .21,
core 148 w/ quic; clippy/fmt clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -42,6 +42,13 @@ let package = Package(
|
||||
.executableTarget(name: "PunktfunkClient", dependencies: ["PunktfunkKit"]),
|
||||
// PunktfunkCore is a direct dep too so the wire tests can name the C ABI's
|
||||
// `PunktfunkInputEvent` / `PUNKTFUNK_INPUT_KIND_*` when asserting the gamepad byte layout.
|
||||
.testTarget(name: "PunktfunkKitTests", dependencies: ["PunktfunkKit", "PunktfunkCore"]),
|
||||
.testTarget(
|
||||
name: "PunktfunkKitTests", dependencies: ["PunktfunkKit", "PunktfunkCore"],
|
||||
resources: [
|
||||
// PyroWave golden fixtures: host-encoded AUs + upstream-decoded reference
|
||||
// planes (regenerate with punktfunk-host's `pyrowave_dump_golden` on a
|
||||
// Vulkan box — see PyroWaveDecoderTests.swift).
|
||||
.copy("PyroWaveFixtures")
|
||||
]),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -46,6 +46,7 @@ struct ContentView: View {
|
||||
case "h264": return PunktfunkConnection.codecH264
|
||||
case "hevc": return PunktfunkConnection.codecHEVC
|
||||
case "av1": return PunktfunkConnection.codecAV1
|
||||
case "pyrowave": return PunktfunkConnection.codecPyroWave
|
||||
default: return 0
|
||||
}
|
||||
}
|
||||
|
||||
@@ -239,6 +239,18 @@ final class SessionModel: ObservableObject {
|
||||
// from these + the soft `preferredCodec`; `resolvedCodec` reflects what it chose.
|
||||
var videoCodecs = PunktfunkConnection.codecH264 | PunktfunkConnection.codecHEVC
|
||||
if AV1.hardwareDecodeSupported { videoCodecs |= PunktfunkConnection.codecAV1 }
|
||||
// PyroWave (wired LAN) is a pure opt-in: picking it in the codec setting both
|
||||
// advertises the bit and prefers it — the host never auto-selects it, and the
|
||||
// picker only offers it when the Metal decode probe passed (simdgroup floor ≈ A13;
|
||||
// every M-series Mac and the ATV 4K gen 3 pass). The codec is 8-bit 4:2:0 SDR
|
||||
// BT.709 by contract, so the opt-in also drops the HDR/10-bit/4:4:4 caps for this
|
||||
// session — HDR sessions stay HEVC/AV1 (plan §4.7).
|
||||
if preferredCodec == PunktfunkConnection.codecPyroWave, MetalWaveletDecoder.supported {
|
||||
videoCodecs |= PunktfunkConnection.codecPyroWave
|
||||
videoCaps &= ~(PunktfunkConnection.videoCap10Bit
|
||||
| PunktfunkConnection.videoCapHDR
|
||||
| PunktfunkConnection.videoCap444)
|
||||
}
|
||||
let result = Result { try PunktfunkConnection(
|
||||
host: host.address, port: host.port,
|
||||
width: width, height: height, refreshHz: hz,
|
||||
|
||||
@@ -79,6 +79,13 @@ enum SettingsOptions {
|
||||
if AV1.hardwareDecodeSupported {
|
||||
options.insert(("AV1", "av1"), at: 2)
|
||||
}
|
||||
// PyroWave is the opt-in wired-LAN low-latency codec (100–400 Mbps all-intra wavelet,
|
||||
// 8-bit SDR): selecting it advertises + prefers it for the session. Offered only when
|
||||
// the Metal decode probe passes (same gate SessionModel advertises by) — elsewhere the
|
||||
// host could never emit it.
|
||||
if MetalWaveletDecoder.supported {
|
||||
options.append(("PyroWave (wired LAN)", "pyrowave"))
|
||||
}
|
||||
return options
|
||||
}()
|
||||
|
||||
|
||||
@@ -337,9 +337,15 @@ public final class PunktfunkConnection {
|
||||
public private(set) var resolvedAudioChannels: UInt8 = 2
|
||||
|
||||
/// The video codec the host resolved for this session (`Welcome.codec`, `PUNKTFUNK_CODEC_*`):
|
||||
/// `2` = HEVC (default / older host), `1` = H.264, `4` = AV1. Build the decoder from THIS. The
|
||||
/// resolved value honors the client's `preferredCodec` when the host could emit it.
|
||||
/// `2` = HEVC (default / older host), `1` = H.264, `4` = AV1, `8` = PyroWave (only when this
|
||||
/// client opted in). Build the decoder from THIS. The resolved value honors the client's
|
||||
/// `preferredCodec` when the host could emit it.
|
||||
public private(set) var resolvedCodec: UInt8 = 2 // PUNKTFUNK_CODEC_HEVC
|
||||
|
||||
/// The session's negotiated wire shard payload (`Welcome.shard_payload`, bytes) — the
|
||||
/// parse-window size for `USER_FLAG_CHUNK_ALIGNED` PyroWave AUs (plan §4.4). Other codecs
|
||||
/// never need it.
|
||||
public private(set) var shardPayload: UInt32 = 1408
|
||||
/// The resolved codec as a `VideoCodec` (H.264 / HEVC / AV1) — drives the bitstream framing
|
||||
/// (Annex-B NAL parsing vs the AV1 OBU repack).
|
||||
public var videoCodec: VideoCodec { VideoCodec(wire: resolvedCodec) }
|
||||
@@ -452,6 +458,9 @@ public final class PunktfunkConnection {
|
||||
var codec: UInt8 = 2 // PUNKTFUNK_CODEC_HEVC
|
||||
_ = punktfunk_connection_codec(handle, &codec)
|
||||
resolvedCodec = codec
|
||||
var shard: UInt32 = 1408
|
||||
_ = punktfunk_connection_shard_payload(handle, &shard)
|
||||
shardPayload = shard
|
||||
}
|
||||
|
||||
/// A bandwidth speed-test measurement (see `startSpeedTest`). Partial until `done`.
|
||||
@@ -790,6 +799,15 @@ public final class PunktfunkConnection {
|
||||
public static let codecH264: UInt8 = UInt8(PUNKTFUNK_CODEC_H264)
|
||||
public static let codecHEVC: UInt8 = UInt8(PUNKTFUNK_CODEC_HEVC)
|
||||
public static let codecAV1: UInt8 = UInt8(PUNKTFUNK_CODEC_AV1)
|
||||
/// PyroWave (opt-in wired-LAN wavelet codec, 8-bit SDR): the host only ever resolves it
|
||||
/// when the client both advertises the bit AND names it `preferredCodec` — never
|
||||
/// auto-selected. Decoded by the Metal wavelet decoder, not VideoToolbox.
|
||||
public static let codecPyroWave: UInt8 = UInt8(PUNKTFUNK_CODEC_PYROWAVE)
|
||||
|
||||
/// `AccessUnit.flags` bit: the AU is shard-aligned self-delimiting chunks (the wire's
|
||||
/// `USER_FLAG_CHUNK_ALIGNED`, PyroWave datagram-aligned mode §4.4) — walk it
|
||||
/// window-by-window at `shardPayload`. (The C `#define` doesn't import into Swift.)
|
||||
public static let userFlagChunkAligned: UInt32 = 64
|
||||
|
||||
/// Static HDR mastering metadata (SMPTE ST.2086 + content light level) the host sent for an HDR
|
||||
/// session. Mirrors the wire/ABI `PunktfunkHdrMeta`; primaries are in ST.2086 **G, B, R** order,
|
||||
|
||||
@@ -27,8 +27,10 @@ public enum DefaultsKey {
|
||||
/// Requested audio channel count: 2 (stereo), 6 (5.1) or 8 (7.1). The host clamps to what it
|
||||
/// can capture; the resolved count drives the in-core decode + AVAudioEngine layout.
|
||||
public static let audioChannels = "punktfunk.audioChannels"
|
||||
/// Preferred video codec: `"auto"` (host decides), `"hevc"`, or `"h264"`. A soft preference —
|
||||
/// the host emits it when it can, else falls back. Drives the decoder via `Welcome.codec`.
|
||||
/// Preferred video codec: `"auto"` (host decides), `"hevc"`, `"h264"`, `"av1"`, or
|
||||
/// `"pyrowave"` (the opt-in wired-LAN wavelet codec — picking it advertises AND prefers it,
|
||||
/// and forces the session SDR). A soft preference — the host emits it when it can, else
|
||||
/// falls back. Drives the decoder via `Welcome.codec`.
|
||||
public static let codec = "punktfunk.codec"
|
||||
public static let micEnabled = "punktfunk.micEnabled"
|
||||
public static let speakerUID = "punktfunk.speakerUID"
|
||||
|
||||
@@ -543,19 +543,24 @@ public enum AV1 {
|
||||
|
||||
extension VideoCodec {
|
||||
/// Codec-dispatching format-description refresh: the AV1 path keys on an in-band sequence
|
||||
/// header, the NAL codecs on in-band parameter sets — one call site in each pump.
|
||||
/// header, the NAL codecs on in-band parameter sets — one call site in each pump. PyroWave
|
||||
/// has no CoreMedia representation at all (its pump feeds the Metal wavelet decoder raw).
|
||||
public func formatDescription(fromKeyframe au: Data) -> CMVideoFormatDescription? {
|
||||
self == .av1
|
||||
? AV1.formatDescription(fromKeyframe: au)
|
||||
: AnnexB.formatDescription(fromIDR: au, codec: self)
|
||||
switch self {
|
||||
case .av1: return AV1.formatDescription(fromKeyframe: au)
|
||||
case .pyrowave: return nil
|
||||
default: return AnnexB.formatDescription(fromIDR: au, codec: self)
|
||||
}
|
||||
}
|
||||
|
||||
/// Codec-dispatching sample wrap (see `formatDescription(fromKeyframe:)`).
|
||||
public func sampleBuffer(
|
||||
au: AccessUnit, format: CMVideoFormatDescription
|
||||
) -> CMSampleBuffer? {
|
||||
self == .av1
|
||||
? AV1.sampleBuffer(au: au, format: format)
|
||||
: AnnexB.sampleBuffer(au: au, format: format, codec: self)
|
||||
switch self {
|
||||
case .av1: return AV1.sampleBuffer(au: au, format: format)
|
||||
case .pyrowave: return nil
|
||||
default: return AnnexB.sampleBuffer(au: au, format: format, codec: self)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,12 +26,18 @@ public enum VideoCodec: Equatable {
|
||||
case h264
|
||||
case hevc
|
||||
case av1
|
||||
/// PyroWave wavelet (opt-in wired-LAN low-latency codec): not a NAL/OBU codec and not
|
||||
/// VideoToolbox-decoded at all — the Metal wavelet decoder consumes the raw AUs
|
||||
/// (Stage2Pipeline's PyroWave pump). Only ever resolved when this client both advertised
|
||||
/// and preferred it.
|
||||
case pyrowave
|
||||
|
||||
/// Resolve from the wire `Welcome.codec` byte (`PUNKTFUNK_CODEC_*`; unknown → HEVC).
|
||||
public init(wire: UInt8) {
|
||||
switch wire {
|
||||
case 0x01: self = .h264 // PUNKTFUNK_CODEC_H264
|
||||
case 0x04: self = .av1 // PUNKTFUNK_CODEC_AV1
|
||||
case 0x08: self = .pyrowave // PUNKTFUNK_CODEC_PYROWAVE
|
||||
default: self = .hevc // PUNKTFUNK_CODEC_HEVC — the default / older-host codec
|
||||
}
|
||||
}
|
||||
@@ -147,8 +153,8 @@ public enum AnnexB {
|
||||
sets = [vps, sps, pps]
|
||||
case .h264:
|
||||
sets = [sps, pps]
|
||||
case .av1:
|
||||
return nil // OBU stream, no parameter-set NALs — handled in AV1.swift, never here
|
||||
case .av1, .pyrowave:
|
||||
return nil // no parameter-set NALs — dispatched in AV1.swift, never reaches here
|
||||
}
|
||||
|
||||
var format: CMVideoFormatDescription?
|
||||
@@ -184,8 +190,8 @@ public enum AnnexB {
|
||||
parameterSetSizes: sizes,
|
||||
nalUnitHeaderLength: 4,
|
||||
formatDescriptionOut: &format)
|
||||
case .av1:
|
||||
break // unreachable — the .av1 arm above already returned
|
||||
case .av1, .pyrowave:
|
||||
break // unreachable — the arm above already returned
|
||||
}
|
||||
}
|
||||
return status == noErr ? format : nil
|
||||
|
||||
@@ -149,6 +149,28 @@ fragment float4 pf_frag(VOut in [[stage_in]],
|
||||
return float4(sampleRgb(lumaTex, chromaTex, in.uv, csc), 1.0);
|
||||
}
|
||||
|
||||
// PyroWave planar SDR: three separate R8 planes (Y full-res, Cb/Cr half-res 4:2:0) from the
|
||||
// Metal wavelet decoder — the Metal twin of pf-presenter's planar_csc.frag. Same bicubic luma
|
||||
// and left-cosited chroma correction as the biplanar path (chromaUV self-disables at 4:4:4).
|
||||
fragment float4 pf_frag_planar(VOut in [[stage_in]],
|
||||
texture2d<float> lumaTex [[texture(0)]],
|
||||
texture2d<float> cbTex [[texture(1)]],
|
||||
texture2d<float> crTex [[texture(2)]],
|
||||
constant CscUniform& csc [[buffer(0)]]) {
|
||||
constexpr sampler s(filter::linear, address::clamp_to_edge);
|
||||
#ifdef PF_BILINEAR_LUMA
|
||||
float lumaY = lumaTex.sample(s, in.uv).r;
|
||||
#else
|
||||
float lumaY = catmullRomLuma(lumaTex, s, in.uv);
|
||||
#endif
|
||||
float2 cuv = chromaUV(lumaTex, cbTex, in.uv);
|
||||
float3 yuv = float3(lumaY, cbTex.sample(s, cuv).r, crTex.sample(s, cuv).r);
|
||||
float3 rgb = saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
|
||||
dot(csc.r1.xyz, yuv) + csc.r1.w,
|
||||
dot(csc.r2.xyz, yuv) + csc.r2.w));
|
||||
return float4(rgb, 1.0);
|
||||
}
|
||||
|
||||
// HDR: 10-bit P010 / 4:4:4 (BT.2020, PQ-encoded Y′CbCr) → full-range PQ R′G′B′, output as-is —
|
||||
// the CAMetalLayer's itur_2100_PQ colour space + edrMetadata tell the compositor the samples are
|
||||
// PQ, so it does the PQ→display tone-map. No EOTF here. The rows fold in the exact 10-bit
|
||||
@@ -215,8 +237,16 @@ public final class MetalVideoPresenter {
|
||||
/// tvOS only: the in-shader PQ→SDR tone-map fallback (pf_frag_hdr_tv → bgra8), used whenever
|
||||
/// the display is composited without HDR headroom — see `setDisplayHeadroom`. nil elsewhere.
|
||||
private let pipelineHDRToneMap: MTLRenderPipelineState?
|
||||
/// PyroWave's 3-plane SDR path (pf_frag_planar → bgra8) — see `renderPlanar`.
|
||||
private let pipelinePlanar: MTLRenderPipelineState
|
||||
private var textureCache: CVMetalTextureCache?
|
||||
|
||||
/// The PyroWave Metal decoder records on the presenter's device + queue: one device means
|
||||
/// decode, CSC and present share textures with zero interop, and one queue means Metal's
|
||||
/// hazard tracking orders a ring-slot rewrite after the render still sampling it.
|
||||
var metalDevice: MTLDevice { device }
|
||||
var metalQueue: MTLCommandQueue { queue }
|
||||
|
||||
/// Current layer configuration — switched in `configure(hdr:)` when a frame's HDR-ness differs.
|
||||
/// Render-thread confined once the pipeline runs (Stage2Pipeline.start's one pre-thread
|
||||
/// `configure` call is ordered before the thread starts, so it doesn't race).
|
||||
@@ -258,6 +288,7 @@ public final class MetalVideoPresenter {
|
||||
let pipelineSDR: MTLRenderPipelineState
|
||||
let pipelineHDR: MTLRenderPipelineState
|
||||
let pipelineHDRToneMap: MTLRenderPipelineState?
|
||||
let pipelinePlanar: MTLRenderPipelineState
|
||||
do {
|
||||
// DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF
|
||||
// (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is
|
||||
@@ -292,6 +323,11 @@ public final class MetalVideoPresenter {
|
||||
#else
|
||||
pipelineHDRToneMap = nil
|
||||
#endif
|
||||
let planar = MTLRenderPipelineDescriptor()
|
||||
planar.vertexFunction = vtx
|
||||
planar.fragmentFunction = library.makeFunction(name: "pf_frag_planar")
|
||||
planar.colorAttachments[0].pixelFormat = .bgra8Unorm // PyroWave is 8-bit SDR
|
||||
pipelinePlanar = try device.makeRenderPipelineState(descriptor: planar)
|
||||
} catch {
|
||||
return nil
|
||||
}
|
||||
@@ -331,12 +367,14 @@ public final class MetalVideoPresenter {
|
||||
|
||||
return MetalVideoPresenter(
|
||||
device: device, queue: queue, pipelineSDR: pipelineSDR, pipelineHDR: pipelineHDR,
|
||||
pipelineHDRToneMap: pipelineHDRToneMap, textureCache: textureCache, layer: layer)
|
||||
pipelineHDRToneMap: pipelineHDRToneMap, pipelinePlanar: pipelinePlanar,
|
||||
textureCache: textureCache, layer: layer)
|
||||
}
|
||||
|
||||
private init(
|
||||
device: MTLDevice, queue: MTLCommandQueue, pipelineSDR: MTLRenderPipelineState,
|
||||
pipelineHDR: MTLRenderPipelineState, pipelineHDRToneMap: MTLRenderPipelineState?,
|
||||
pipelinePlanar: MTLRenderPipelineState,
|
||||
textureCache: CVMetalTextureCache, layer: CAMetalLayer
|
||||
) {
|
||||
self.device = device
|
||||
@@ -344,6 +382,7 @@ public final class MetalVideoPresenter {
|
||||
self.pipelineSDR = pipelineSDR
|
||||
self.pipelineHDR = pipelineHDR
|
||||
self.pipelineHDRToneMap = pipelineHDRToneMap
|
||||
self.pipelinePlanar = pipelinePlanar
|
||||
self.textureCache = textureCache
|
||||
self.layer = layer
|
||||
}
|
||||
@@ -514,6 +553,67 @@ public final class MetalVideoPresenter {
|
||||
pixelBuffer, plane: 1, format: tenBit ? .rg16Unorm : .rg8Unorm, cache: textureCache)
|
||||
else { return false }
|
||||
|
||||
#if os(tvOS)
|
||||
// HDR splits by the display's headroom (kept in step with the layer by `configure` above):
|
||||
// PQ passthrough into an HDR-composited display, the tone-map shader otherwise.
|
||||
let hdrPipeline = hdrPassthroughActive ? pipelineHDR : (pipelineHDRToneMap ?? pipelineHDR)
|
||||
let pipeline = hdrActive ? hdrPipeline : pipelineSDR
|
||||
#else
|
||||
let pipeline = hdrActive ? pipelineHDR : pipelineSDR
|
||||
#endif
|
||||
let decodedSize = CGSize(
|
||||
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
|
||||
return encodePresent(
|
||||
decodedSize: decodedSize, targetFromLayout: targetFromLayout, pipeline: pipeline,
|
||||
presentAtMediaTime: presentAtMediaTime, onPresented: onPresented,
|
||||
// Hold the CVMetalTextures + source pixel buffer (its IOSurface) alive until the GPU
|
||||
// finishes sampling — releasing them at scope exit could free the backing mid-read.
|
||||
keepAlive: [luma, chroma, pixelBuffer]
|
||||
) { encoder in
|
||||
encoder.setFragmentTexture(CVMetalTextureGetTexture(luma), index: 0)
|
||||
encoder.setFragmentTexture(CVMetalTextureGetTexture(chroma), index: 1)
|
||||
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
|
||||
}
|
||||
}
|
||||
|
||||
/// Draw one PyroWave planar frame (three R8 planes off the Metal wavelet decoder) and
|
||||
/// present it. RENDER THREAD, same contract as `render` — PyroWave is 8-bit SDR, so the
|
||||
/// layer always takes the plain SDR config, and the CSC rows arrive precomputed from the
|
||||
/// stream's own sequence-header signaling (no CVPixelBuffer to inspect).
|
||||
@discardableResult
|
||||
func renderPlanar(
|
||||
_ planes: WaveletPlanes,
|
||||
presentAtMediaTime: CFTimeInterval? = nil,
|
||||
onPresented: ((Int64?) -> Void)? = nil
|
||||
) -> Bool {
|
||||
stagingLock.lock()
|
||||
let targetFromLayout = drawableTarget
|
||||
stagingLock.unlock()
|
||||
configure(hdr: false)
|
||||
var csc = planes.csc
|
||||
return encodePresent(
|
||||
decodedSize: CGSize(width: planes.width, height: planes.height),
|
||||
targetFromLayout: targetFromLayout, pipeline: pipelinePlanar,
|
||||
presentAtMediaTime: presentAtMediaTime, onPresented: onPresented,
|
||||
// The ring textures stay valid by ring depth; retaining them here also pins the
|
||||
// slot's set until the sample completes (mirrors the biplanar keep-alive).
|
||||
keepAlive: [planes.y, planes.cb, planes.cr]
|
||||
) { encoder in
|
||||
encoder.setFragmentTexture(planes.y, index: 0)
|
||||
encoder.setFragmentTexture(planes.cb, index: 1)
|
||||
encoder.setFragmentTexture(planes.cr, index: 2)
|
||||
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
|
||||
}
|
||||
}
|
||||
|
||||
/// The shared present tail of `render`/`renderPlanar`: size the drawable, encode one
|
||||
/// fullscreen triangle with `pipeline` (`bind` supplies the fragment resources), schedule
|
||||
/// the present and the on-glass callback.
|
||||
private func encodePresent(
|
||||
decodedSize: CGSize, targetFromLayout: CGSize, pipeline: MTLRenderPipelineState,
|
||||
presentAtMediaTime: CFTimeInterval?, onPresented: ((Int64?) -> Void)?,
|
||||
keepAlive: [Any], bind: (MTLRenderCommandEncoder) -> Void
|
||||
) -> Bool {
|
||||
// Size the drawable to the LAYER's pixels (its laid-out frame × contentsScale, pushed here by
|
||||
// SessionPresenter.layout via `setDrawableTarget` — not read off the layer, whose geometry the
|
||||
// main thread owns) so the Catmull-Rom shader performs the decoded→on-screen scale in one pass:
|
||||
@@ -522,8 +622,6 @@ public final class MetalVideoPresenter {
|
||||
// Before the first layout (zero target) fall back to the decoded size. drawableSize does NOT
|
||||
// track bounds (defaults to 0), so set it BEFORE nextDrawable; re-set only on a change
|
||||
// (layout / Reconfigure / HDR flip — and every frame of a live resize, which is fine).
|
||||
let decodedSize = CGSize(
|
||||
width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
|
||||
let targetSize = (targetFromLayout.width > 0 && targetFromLayout.height > 0)
|
||||
? targetFromLayout : decodedSize
|
||||
if layer.drawableSize != targetSize { layer.drawableSize = targetSize }
|
||||
@@ -542,17 +640,8 @@ public final class MetalVideoPresenter {
|
||||
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: pass) else {
|
||||
return false
|
||||
}
|
||||
#if os(tvOS)
|
||||
// HDR splits by the display's headroom (kept in step with the layer by `configure` above):
|
||||
// PQ passthrough into an HDR-composited display, the tone-map shader otherwise.
|
||||
let hdrPipeline = hdrPassthroughActive ? pipelineHDR : (pipelineHDRToneMap ?? pipelineHDR)
|
||||
encoder.setRenderPipelineState(hdrActive ? hdrPipeline : pipelineSDR)
|
||||
#else
|
||||
encoder.setRenderPipelineState(hdrActive ? pipelineHDR : pipelineSDR)
|
||||
#endif
|
||||
encoder.setFragmentTexture(CVMetalTextureGetTexture(luma), index: 0)
|
||||
encoder.setFragmentTexture(CVMetalTextureGetTexture(chroma), index: 1)
|
||||
encoder.setFragmentBytes(&csc, length: MemoryLayout<CscUniform>.stride, index: 0)
|
||||
encoder.setRenderPipelineState(pipeline)
|
||||
bind(encoder)
|
||||
encoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
|
||||
encoder.endEncoding()
|
||||
if let onPresented {
|
||||
@@ -580,9 +669,8 @@ public final class MetalVideoPresenter {
|
||||
} else {
|
||||
commandBuffer.present(drawable)
|
||||
}
|
||||
// Hold the CVMetalTextures + source pixel buffer (its IOSurface) alive until the GPU finishes
|
||||
// sampling — releasing them at scope exit could free the backing mid-read.
|
||||
commandBuffer.addCompletedHandler { _ in _ = (luma, chroma, pixelBuffer) }
|
||||
// Keep the bound sources alive until the GPU finishes sampling (see the callers).
|
||||
commandBuffer.addCompletedHandler { _ in _ = keepAlive }
|
||||
commandBuffer.commit()
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -183,18 +183,12 @@ enum WaveletBitstream {
|
||||
}
|
||||
return state.pushPackets(UnsafeBufferPointer(start: base, count: count))
|
||||
}
|
||||
guard ok, var frame = state.finish() else { return nil }
|
||||
guard ok, let frame = state.finish() else { return nil }
|
||||
// Upstream decode_is_ready(allow_partial=true): with no SOF the frame is undecodable;
|
||||
// at half the blocks or fewer it is presumed garbage.
|
||||
guard frame.totalBlocks > 0, frame.decodedBlocks > frame.totalBlocks / 2 else {
|
||||
return nil
|
||||
}
|
||||
// The dequant kernel indexes the offset table by the LAYOUT's block space; the wire's
|
||||
// total_blocks only counts blocks the encoder emitted. They agree for a full-coverage
|
||||
// frame, but size the table by the layout.
|
||||
if frame.offsets.count != frame.layout.blockCount32 {
|
||||
frame.offsets = Array(frame.offsets.prefix(frame.layout.blockCount32))
|
||||
}
|
||||
return frame
|
||||
}
|
||||
|
||||
@@ -293,17 +287,17 @@ enum WaveletBitstream {
|
||||
|
||||
/// One decoded frame's output planes, handed to the presenter's planar render path. The
|
||||
/// textures belong to the decoder's ring — ring depth (4) plus same-queue hazard tracking keep
|
||||
/// them valid while referenced.
|
||||
struct WaveletPlanes {
|
||||
let y: MTLTexture
|
||||
let cb: MTLTexture
|
||||
let cr: MTLTexture
|
||||
let csc: CscUniform
|
||||
var width: Int { y.width }
|
||||
var height: Int { y.height }
|
||||
/// them valid while referenced. Public because it rides inside `ReadyImage`.
|
||||
public struct WaveletPlanes: @unchecked Sendable {
|
||||
public let y: MTLTexture
|
||||
public let cb: MTLTexture
|
||||
public let cr: MTLTexture
|
||||
public let csc: CscUniform
|
||||
public var width: Int { y.width }
|
||||
public var height: Int { y.height }
|
||||
}
|
||||
|
||||
final class MetalWaveletDecoder {
|
||||
public final class MetalWaveletDecoder {
|
||||
/// Matches the Vulkan client's ring: deep enough that a slot is never rewritten while the
|
||||
/// presenter still samples it in practice; same-queue hazard tracking is the hard backstop.
|
||||
private static let ringDepth = 4
|
||||
@@ -312,7 +306,7 @@ final class MetalWaveletDecoder {
|
||||
/// dequant kernel needs simdgroup prefix sums with its 16 header lanes inside one
|
||||
/// simdgroup, so compile the real kernels once and check the pipeline facts. Apple6 (A13)
|
||||
/// and every Mac2 device pass the family check; the compile probe is authoritative.
|
||||
static let supported: Bool = {
|
||||
public static let supported: Bool = {
|
||||
guard let device = MTLCreateSystemDefaultDevice() else { return false }
|
||||
guard device.supportsFamily(.apple6) || device.supportsFamily(.mac2) else { return false }
|
||||
do {
|
||||
@@ -358,6 +352,12 @@ final class MetalWaveletDecoder {
|
||||
private var slots: [Slot] = []
|
||||
private var nextSlot = 0
|
||||
|
||||
/// The current geometry (from the last SOF that built the resources) — the pump reports
|
||||
/// decoded-size changes to the resize overlay from this. PUMP THREAD.
|
||||
var decodedSize: (width: Int, height: Int)? {
|
||||
layout.map { ($0.width, $0.height) }
|
||||
}
|
||||
|
||||
/// The pump thread owns `decode`; everything mutable is confined to it.
|
||||
init?(device: MTLDevice, queue: MTLCommandQueue) {
|
||||
self.device = device
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
#if canImport(Metal) && canImport(QuartzCore)
|
||||
import AVFoundation
|
||||
import Foundation
|
||||
import Metal
|
||||
import QuartzCore
|
||||
|
||||
/// PUNKTFUNK_PRESENT_DEBUG=1: the render thread prints a once-per-second line with the decode
|
||||
@@ -257,6 +258,7 @@ public final class Stage2Pipeline {
|
||||
/// the pipeline's lifetime; SessionPresenter resolves it per session (see PresentPacing).
|
||||
private let pacing: PresentPacing
|
||||
private let endToEndMeter: LatencyMeter?
|
||||
private let decodeMeter: LatencyMeter?
|
||||
private let displayMeter: LatencyMeter?
|
||||
private let recovery = KeyframeRecovery()
|
||||
/// Post-loss freeze-until-reanchor gate (shared core policy via the C ABI). Created here seeded 0;
|
||||
@@ -306,6 +308,7 @@ public final class Stage2Pipeline {
|
||||
self.presenter = presenter
|
||||
self.pacing = pacing
|
||||
self.endToEndMeter = endToEndMeter
|
||||
self.decodeMeter = decodeMeter
|
||||
self.displayMeter = displayMeter
|
||||
let ring = ring
|
||||
let recovery = recovery
|
||||
@@ -362,7 +365,21 @@ public final class Stage2Pipeline {
|
||||
let presenter = presenter
|
||||
let pumpStopped = pumpStopped
|
||||
let reanchorGate = gate
|
||||
let thread = Thread {
|
||||
// PyroWave rides a different decode half: no CMFormatDescription/VideoToolbox machinery
|
||||
// (a wavelet AU has no parameter sets), no keyframe recovery or re-anchor freeze (the
|
||||
// stream is all-intra and Phase 4's partial delivery WANTS lossy frames on glass as
|
||||
// localized blur, not a freeze). The ready ring, render thread, pacing and meters are
|
||||
// shared unchanged.
|
||||
let thread: Thread
|
||||
if connection.videoCodec == .pyrowave {
|
||||
thread = Self.makePyroWavePump(
|
||||
connection: connection, token: token, pumpStopped: pumpStopped,
|
||||
ring: ring, renderSignal: renderSignal,
|
||||
device: presenter.metalDevice, queue: presenter.metalQueue,
|
||||
decodeMeter: decodeMeter,
|
||||
onFrame: onFrame, onSessionEnd: onSessionEnd, onDecodedSize: onDecodedSize)
|
||||
} else {
|
||||
thread = Thread {
|
||||
defer { pumpStopped.signal() } // let stop() join the pump (bounded) before decoder.reset()
|
||||
var format: CMVideoFormatDescription?
|
||||
// Report coded dims to the resize overlay only on a CHANGE (new-mode IDR), not per
|
||||
@@ -446,6 +463,7 @@ public final class Stage2Pipeline {
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
thread.name = "punktfunk-stage2-pump"
|
||||
thread.qualityOfService = .userInteractive
|
||||
pumpJoinable = true
|
||||
@@ -504,9 +522,7 @@ public final class Stage2Pipeline {
|
||||
let presentAt = vsyncEnabled
|
||||
? vsyncClock.nextVsync(after: CACurrentMediaTime()) : nil
|
||||
let renderStarted = CACurrentMediaTime()
|
||||
let rendered = presenter.render(
|
||||
frame.pixelBuffer, isHDR: frame.isHDR, presentAtMediaTime: presentAt
|
||||
) { presentedNs in
|
||||
let onGlass: (Int64?) -> Void = { presentedNs in
|
||||
// Stage-3: the flip reached glass (or was dropped) — free the present slot,
|
||||
// then re-signal so the freshest waiting ring frame goes out immediately.
|
||||
if let gate {
|
||||
@@ -525,6 +541,18 @@ public final class Stage2Pipeline {
|
||||
displayMeter?.record(ptsNs: UInt64(frame.decodedNs), atNs: atNs, offsetNs: 0)
|
||||
debugStats?.presented(atNs: presentedNs)
|
||||
}
|
||||
// One present tail, two decode sources: the VideoToolbox biplanar buffer or the
|
||||
// PyroWave Metal planes — the ring, pacing and meters are agnostic to which.
|
||||
let rendered: Bool
|
||||
switch frame.image {
|
||||
case .video(let pixelBuffer, let isHDR):
|
||||
rendered = presenter.render(
|
||||
pixelBuffer, isHDR: isHDR, presentAtMediaTime: presentAt,
|
||||
onPresented: onGlass)
|
||||
case .planar(let planes):
|
||||
rendered = presenter.renderPlanar(
|
||||
planes, presentAtMediaTime: presentAt, onPresented: onGlass)
|
||||
}
|
||||
debugStats?.renderReturned(
|
||||
ok: rendered, tookMs: (CACurrentMediaTime() - renderStarted) * 1000)
|
||||
if !rendered {
|
||||
@@ -592,6 +620,93 @@ public final class Stage2Pipeline {
|
||||
renderSignal.signal() // wake the render thread so it can observe the stop and exit
|
||||
}
|
||||
|
||||
/// The PyroWave pump: AUs go straight into the Metal wavelet decoder (no VideoToolbox, no
|
||||
/// format descriptions), decoded planes ride the same ready ring / render thread. All-intra
|
||||
/// stream, so none of the VT pump's recovery machinery applies: keyframe/RFI requests are
|
||||
/// silenced host-side for this codec, and a lossy (partial-delivery) frame is MEANT to
|
||||
/// present as localized blur — never a freeze. Static + capture-by-parameter for the same
|
||||
/// reason the VT pump avoids capturing `self` (a missed stop must not leak a live pipeline).
|
||||
private static func makePyroWavePump(
|
||||
connection: PunktfunkConnection, token: StopFlag, pumpStopped: DispatchSemaphore,
|
||||
ring: ReadyRing, renderSignal: DispatchSemaphore,
|
||||
device: MTLDevice, queue: MTLCommandQueue,
|
||||
decodeMeter: LatencyMeter?,
|
||||
onFrame: (@Sendable (AccessUnit) -> Void)?,
|
||||
onSessionEnd: (@Sendable () -> Void)?,
|
||||
onDecodedSize: (@Sendable (Int, Int) -> Void)?
|
||||
) -> Thread {
|
||||
// The chunk-aligned parse window = the session's negotiated shard payload (Welcome);
|
||||
// the 64-byte floor mirrors the Rust client's guard against a nonsense value.
|
||||
let windowSize = max(64, Int(connection.shardPayload))
|
||||
return Thread {
|
||||
defer { pumpStopped.signal() }
|
||||
// Compiles the two compute kernels on the session's first frames' thread — ~tens of
|
||||
// ms, once per session. Failure = this device can't run the negotiated codec (the
|
||||
// advertisement probe should have prevented this); end the session cleanly.
|
||||
guard let decoder = MetalWaveletDecoder(device: device, queue: queue) else {
|
||||
if !token.isStopped { onSessionEnd?() }
|
||||
return
|
||||
}
|
||||
// Newest decoded frame index — a late partial (the reassembler's 30 ms fuse can
|
||||
// deliver one behind a newer complete frame) must not travel back in time.
|
||||
var newestIndex: UInt32?
|
||||
var lastDims: (w: Int, h: Int)?
|
||||
var alive = true
|
||||
while alive, !token.isStopped {
|
||||
alive = autoreleasepool { () -> Bool in
|
||||
do {
|
||||
guard let au = try connection.nextAU(timeoutMs: 100) else { return true }
|
||||
onFrame?(au)
|
||||
if let newest = newestIndex,
|
||||
Int32(bitPattern: au.frameIndex &- newest) <= 0 {
|
||||
return true // stale (or duplicate) frame — skip
|
||||
}
|
||||
guard !token.isStopped else { return true }
|
||||
let chunkAligned =
|
||||
au.flags & PunktfunkConnection.userFlagChunkAligned != 0
|
||||
let ptsNs = au.ptsNs
|
||||
let receivedNs = au.receivedNs
|
||||
let flags = au.flags
|
||||
let submitted = decoder.decode(
|
||||
au: au.data, chunkAligned: chunkAligned, windowSize: windowSize
|
||||
) { planes in
|
||||
// Metal completed-handler thread — stamp + enqueue, don't block
|
||||
// (the exact contract of the VT output callback).
|
||||
guard let planes else { return }
|
||||
var ts = timespec()
|
||||
clock_gettime(CLOCK_REALTIME, &ts)
|
||||
let decodedNs =
|
||||
Int64(ts.tv_sec) * 1_000_000_000 + Int64(ts.tv_nsec)
|
||||
decodeMeter?.record(
|
||||
ptsNs: UInt64(receivedNs), atNs: decodedNs, offsetNs: 0)
|
||||
ring.submit(
|
||||
ReadyFrame(
|
||||
ptsNs: ptsNs, receivedNs: receivedNs, decodedNs: decodedNs,
|
||||
image: .planar(planes), flags: flags))
|
||||
renderSignal.signal()
|
||||
}
|
||||
if submitted {
|
||||
newestIndex = au.frameIndex
|
||||
// Decoded-size changes come from the SOF dims (this is also how a
|
||||
// mid-stream Reconfigure lands here) — report like the VT pump.
|
||||
if let size = decoder.decodedSize,
|
||||
lastDims?.w != size.width || lastDims?.h != size.height {
|
||||
lastDims = (size.width, size.height)
|
||||
onDecodedSize?(size.width, size.height)
|
||||
}
|
||||
}
|
||||
// A dropped AU (malformed / SOF lost / too few blocks) is just skipped:
|
||||
// every PyroWave frame is independently decodable, the next one heals.
|
||||
return true
|
||||
} catch {
|
||||
if !token.isStopped { onSessionEnd?() }
|
||||
return false // session closed
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert a `CADisplayLink.targetTimestamp` (CACurrentMediaTime basis) to a `CLOCK_REALTIME`
|
||||
/// nanosecond instant — the present clock the AU pts + skew offset live in. Projects to the target
|
||||
/// present time (when the frame is actually on glass), not the moment we drew.
|
||||
|
||||
@@ -12,7 +12,23 @@ import CoreVideo
|
||||
import Foundation
|
||||
import VideoToolbox
|
||||
|
||||
/// One decoded frame waiting to be presented. Owns a retained `CVPixelBuffer` until shown.
|
||||
/// A decoded frame's pixels — which present path they take. VideoToolbox codecs deliver a
|
||||
/// biplanar `CVPixelBuffer` (NV12/P010/444v/x444); the PyroWave Metal decoder delivers three
|
||||
/// separate R8 plane textures straight off its compute pass (there is no CVPixelBuffer — the
|
||||
/// planes never leave the GPU).
|
||||
public enum ReadyImage: @unchecked Sendable {
|
||||
/// 8-bit NV12 / 4:4:4 biplanar (SDR) or 10-bit P010 / x444 (HDR), Metal-compatible.
|
||||
/// `isHDR` = the stream is BT.2020 PQ and the presenter must configure EDR output.
|
||||
case video(CVPixelBuffer, isHDR: Bool)
|
||||
#if canImport(Metal)
|
||||
/// PyroWave planar output (Y full-res + Cb/Cr half-res, 8-bit SDR) with its precomputed
|
||||
/// CSC rows — presented by `MetalVideoPresenter.renderPlanar`.
|
||||
case planar(WaveletPlanes)
|
||||
#endif
|
||||
}
|
||||
|
||||
/// One decoded frame waiting to be presented. Owns its image (a retained `CVPixelBuffer`, or
|
||||
/// the PyroWave ring textures) until shown.
|
||||
public struct ReadyFrame: @unchecked Sendable {
|
||||
/// Host capture clock (the AU's pts), in nanoseconds.
|
||||
public let ptsNs: UInt64
|
||||
@@ -22,15 +38,26 @@ public struct ReadyFrame: @unchecked Sendable {
|
||||
public let receivedNs: Int64
|
||||
/// Client `CLOCK_REALTIME` instant decode completed, in nanoseconds.
|
||||
public let decodedNs: Int64
|
||||
/// The decoded image — 8-bit NV12 biplanar (SDR) or 10-bit P010 biplanar (HDR), Metal-compatible.
|
||||
public let pixelBuffer: CVPixelBuffer
|
||||
/// True when the stream is HDR (BT.2020 PQ): the buffer is 10-bit P010 and the presenter must
|
||||
/// configure EDR + BT.2020 PQ output. Derived from the decoded buffer's pixel format.
|
||||
public let isHDR: Bool
|
||||
/// The decoded image and which present path it takes.
|
||||
public let image: ReadyImage
|
||||
/// The AU's wire `user_flags` (`AccessUnit.flags`), threaded through the decode via the frame
|
||||
/// context so the re-anchor gate can classify this decoded frame (IDR / RFI anchor / recovery
|
||||
/// mark) at present time — the async decode callback has no other access to it. 0 when unknown.
|
||||
public let flags: UInt32
|
||||
|
||||
/// The VideoToolbox path's buffer; nil for a PyroWave planar frame. (Kept as the accessor
|
||||
/// the decode round-trip tests assert against.)
|
||||
public var pixelBuffer: CVPixelBuffer? {
|
||||
if case .video(let buffer, _) = image { return buffer }
|
||||
return nil
|
||||
}
|
||||
|
||||
/// Whether this frame presents on the HDR path. PyroWave planar frames are 8-bit SDR by
|
||||
/// contract.
|
||||
public var isHDR: Bool {
|
||||
if case .video(_, let hdr) = image { return hdr }
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-frame context threaded through the VideoToolbox frame refcon: the AU's receipt instant (for
|
||||
@@ -286,6 +313,6 @@ public final class VideoDecoder: @unchecked Sendable {
|
||||
onDecoded(
|
||||
ReadyFrame(
|
||||
ptsNs: ptsNs, receivedNs: receivedNs, decodedNs: decodedNs,
|
||||
pixelBuffer: imageBuffer, isHDR: isHDR, flags: flags))
|
||||
image: .video(imageBuffer, isHDR: isHDR), flags: flags))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -237,10 +237,11 @@ final class AV1Tests: XCTestCase {
|
||||
let ready = try XCTUnwrap(frame)
|
||||
XCTAssertEqual(ready.ptsNs, 42_000_000)
|
||||
XCTAssertFalse(ready.isHDR)
|
||||
XCTAssertEqual(CVPixelBufferGetWidth(ready.pixelBuffer), 320)
|
||||
XCTAssertEqual(CVPixelBufferGetHeight(ready.pixelBuffer), 180)
|
||||
let buffer = try XCTUnwrap(ready.pixelBuffer, "a VT decode delivers a .video frame")
|
||||
XCTAssertEqual(CVPixelBufferGetWidth(buffer), 320)
|
||||
XCTAssertEqual(CVPixelBufferGetHeight(buffer), 180)
|
||||
XCTAssertEqual(
|
||||
CVPixelBufferGetPixelFormatType(ready.pixelBuffer),
|
||||
CVPixelBufferGetPixelFormatType(buffer),
|
||||
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, "SDR AV1 must decode to NV12")
|
||||
decoder.reset()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,292 @@
|
||||
// PyroWave Metal decoder tests — two layers:
|
||||
//
|
||||
// 1. Bitstream/window-walk parser tests (pure CPU): hand-crafted packet streams assert the
|
||||
// exact wire semantics of pyrowave_decoder.cpp's push_packet walk + the Phase-4
|
||||
// chunk-aligned framing (4-byte window prefix, FRAG chains, zeroed missing shards).
|
||||
//
|
||||
// 2. Golden-frame PSNR tests (Metal GPU): host-encoded fixtures (crates/punktfunk-host
|
||||
// encode/linux/pyrowave.rs `pyrowave_dump_golden`, run on a Vulkan box) decoded by the
|
||||
// Metal port and PSNR-matched against upstream's own decoder output. Float wavelet math is
|
||||
// not bit-exact across implementations (upstream ships precision variants), so the gate is
|
||||
// PSNR, not equality. This is the §4.7 validation oracle for the hand-ported kernels —
|
||||
// the gather/mirror addressing in idwt is the spot most likely to drift.
|
||||
|
||||
#if canImport(Metal)
|
||||
import Metal
|
||||
import XCTest
|
||||
|
||||
@testable import PunktfunkKit
|
||||
|
||||
final class PyroWaveParserTests: XCTestCase {
|
||||
// 256x144 → aligned 256x160; block space identical to the committed fixtures.
|
||||
private let width = 256
|
||||
private let height = 144
|
||||
|
||||
/// A BitstreamSequenceHeader (START_OF_FRAME) for `width`x`height`, 4:2:0 BT.709 limited.
|
||||
private func sof(totalBlocks: Int, sequence: UInt32 = 1) -> [UInt8] {
|
||||
let word0 =
|
||||
UInt32(width - 1) | (UInt32(height - 1) << 14) | (sequence << 28) | (1 << 31)
|
||||
// code=0 (SOF), chroma=0 (420), primaries/trc/matrix=0 (BT.709), range=1 (LIMITED),
|
||||
// siting=0.
|
||||
let word1 = UInt32(totalBlocks) | (1 << 30)
|
||||
return le32(word0) + le32(word1)
|
||||
}
|
||||
|
||||
/// A minimal coefficient packet: ballot=0 (all 8x8 blocks empty — legal and decodable),
|
||||
/// payload_words=2 (header only).
|
||||
private func packet(blockIndex: Int, sequence: UInt32 = 1) -> [UInt8] {
|
||||
let word0 = UInt32(0) | (2 << 16) | (sequence << 28)
|
||||
let word1 = UInt32(0) | (UInt32(blockIndex) << 8)
|
||||
return le32(word0) + le32(word1)
|
||||
}
|
||||
|
||||
private func le32(_ v: UInt32) -> [UInt8] {
|
||||
[UInt8(v & 0xff), UInt8((v >> 8) & 0xff), UInt8((v >> 16) & 0xff), UInt8(v >> 24)]
|
||||
}
|
||||
|
||||
/// Wrap bodies into `windowSize`-sized windows with the 4-byte used/kind prefix.
|
||||
private func window(_ body: [UInt8], kind: UInt16, size: Int) -> [UInt8] {
|
||||
precondition(body.count + 4 <= size)
|
||||
var out = [UInt8(body.count & 0xff), UInt8(body.count >> 8)]
|
||||
out += [UInt8(kind & 0xff), UInt8(kind >> 8)]
|
||||
out += body
|
||||
out += [UInt8](repeating: 0, count: size - out.count)
|
||||
return out
|
||||
}
|
||||
|
||||
func testLayoutMatchesUpstreamBlockSpace() {
|
||||
// init_block_meta's walk for 256x144 (aligned 256x160): level extents halve from
|
||||
// 128x80; per (comp,level,band) count32 = ceil(ceil(w/8)/4) * ceil(ceil(h/8)/4).
|
||||
let layout = WaveletLayout(width: width, height: height)
|
||||
XCTAssertEqual(layout.alignedWidth, 256)
|
||||
XCTAssertEqual(layout.alignedHeight, 160)
|
||||
XCTAssertEqual(layout.levelWidth(0), 128)
|
||||
XCTAssertEqual(layout.levelHeight(0), 80)
|
||||
XCTAssertEqual(layout.levelWidth(4), 8)
|
||||
XCTAssertEqual(layout.levelHeight(4), 5)
|
||||
// Hand-summed: L4 (8x5 → 1 block) × 3 comps × 4 bands = 12; L3 (16x10 → 1) × 9 = 9;
|
||||
// L2 (32x20 → 1) × 9 = 9; L1 (64x40 → 2x2=4... ) — trust the invariant instead:
|
||||
// every band's count is ceil(w8/4)*ceil(h8/4) and the total is their sum.
|
||||
var expected = 0
|
||||
for level in stride(from: 4, through: 0, by: -1) {
|
||||
let w8 = (layout.levelWidth(level) + 7) / 8
|
||||
let h8 = (layout.levelHeight(level) + 7) / 8
|
||||
let per = ((w8 + 3) / 4) * ((h8 + 3) / 4)
|
||||
for component in 0..<3 {
|
||||
if level == 0 && component != 0 { continue }
|
||||
expected += per * (level == 4 ? 4 : 3)
|
||||
}
|
||||
}
|
||||
XCTAssertEqual(layout.blockCount32, expected)
|
||||
// The finest luma level's stride is its 32-block row width.
|
||||
XCTAssertEqual(layout.blockMeta[0][0][1].stride, (128 + 31) / 32)
|
||||
// Level-0 chroma is not coded in 4:2:0.
|
||||
XCTAssertEqual(layout.blockMeta[1][0][1].offset, -1)
|
||||
}
|
||||
|
||||
func testDenseParseFillsOffsetsAndCountsBlocks() throws {
|
||||
let layout = WaveletLayout(width: width, height: height)
|
||||
var au = sof(totalBlocks: 4)
|
||||
au += packet(blockIndex: 0)
|
||||
au += packet(blockIndex: 3)
|
||||
au += packet(blockIndex: 3) // duplicate — first wins, not double-counted
|
||||
au += packet(blockIndex: layout.blockCount32 - 1)
|
||||
let frame = try XCTUnwrap(
|
||||
WaveletBitstream.parse(au: Data(au), chunkAligned: false, windowSize: 0))
|
||||
XCTAssertEqual(frame.layout.width, width)
|
||||
XCTAssertEqual(frame.totalBlocks, 4)
|
||||
XCTAssertEqual(frame.decodedBlocks, 3)
|
||||
XCTAssertEqual(frame.offsets[0], 0)
|
||||
XCTAssertEqual(frame.offsets[3], 2) // u32 words: each header-only packet is 2 words
|
||||
XCTAssertEqual(frame.offsets[1], UInt32.max)
|
||||
XCTAssertEqual(frame.payload.count, 6)
|
||||
XCTAssertFalse(frame.bt2020)
|
||||
XCTAssertFalse(frame.fullRange) // range bit 1 = LIMITED
|
||||
}
|
||||
|
||||
func testHalfOrFewerBlocksIsDropped() {
|
||||
var au = sof(totalBlocks: 4)
|
||||
au += packet(blockIndex: 0)
|
||||
au += packet(blockIndex: 1)
|
||||
// 2 of 4 decoded = exactly half — upstream requires MORE than half.
|
||||
XCTAssertNil(WaveletBitstream.parse(au: Data(au), chunkAligned: false, windowSize: 0))
|
||||
}
|
||||
|
||||
func testMissingSOFIsDropped() {
|
||||
let au = packet(blockIndex: 0) + packet(blockIndex: 1)
|
||||
XCTAssertNil(WaveletBitstream.parse(au: Data(au), chunkAligned: false, windowSize: 0))
|
||||
}
|
||||
|
||||
func testTruncatedPacketIsRejected() {
|
||||
var au = sof(totalBlocks: 1)
|
||||
// Claims 4 payload words but only the 8-byte header follows.
|
||||
let word0 = UInt32(0) | (4 << 16) | (1 << 28)
|
||||
au += le32(word0) + le32(0)
|
||||
XCTAssertNil(WaveletBitstream.parse(au: Data(au), chunkAligned: false, windowSize: 0))
|
||||
}
|
||||
|
||||
func testWindowWalkPackedFragAndMissingShard() throws {
|
||||
let size = 64
|
||||
// Window 1: SOF + one packet, PACKED. Window 2: a FRAG chain carrying one packet split
|
||||
// across two windows. Window 3: all zeros (a lost shard of a partial frame). Window 4:
|
||||
// a PACKED packet — the chain break must not eat it.
|
||||
let fragPacket = packet(blockIndex: 2)
|
||||
var au = window(sof(totalBlocks: 3) + packet(blockIndex: 0), kind: 0, size: size)
|
||||
au += window(Array(fragPacket[0..<5]), kind: 1, size: size)
|
||||
au += window(Array(fragPacket[5...]), kind: 3, size: size)
|
||||
au += [UInt8](repeating: 0, count: size) // missing shard
|
||||
au += window(packet(blockIndex: 1), kind: 0, size: size)
|
||||
let frame = try XCTUnwrap(
|
||||
WaveletBitstream.parse(au: Data(au), chunkAligned: true, windowSize: size))
|
||||
XCTAssertEqual(frame.decodedBlocks, 3)
|
||||
XCTAssertEqual(frame.offsets[0], 0)
|
||||
XCTAssertEqual(frame.offsets[2], 2)
|
||||
XCTAssertEqual(frame.offsets[1], 4)
|
||||
}
|
||||
|
||||
func testBrokenFragChainIsDiscarded() throws {
|
||||
let size = 64
|
||||
let fragPacket = packet(blockIndex: 2)
|
||||
var au = window(sof(totalBlocks: 1) + packet(blockIndex: 0), kind: 0, size: size)
|
||||
au += window(Array(fragPacket[0..<5]), kind: 1, size: size)
|
||||
au += [UInt8](repeating: 0, count: size) // the chain's middle shard was lost
|
||||
au += window(Array(fragPacket[5...]), kind: 3, size: size) // dangling LAST — dropped
|
||||
let frame = try XCTUnwrap(
|
||||
WaveletBitstream.parse(au: Data(au), chunkAligned: true, windowSize: size))
|
||||
XCTAssertEqual(frame.decodedBlocks, 1)
|
||||
XCTAssertEqual(frame.offsets[2], UInt32.max)
|
||||
}
|
||||
}
|
||||
|
||||
/// Golden-frame decode against the committed host-encoder fixtures. Skipped when the machine
|
||||
/// has no Metal device (headless CI) — everywhere else this is the hand-ported kernels' guard.
|
||||
final class PyroWaveGoldenTests: XCTestCase {
|
||||
private static let fixtureDir = "PyroWaveFixtures"
|
||||
|
||||
private func fixture(_ name: String) throws -> Data {
|
||||
let url = try XCTUnwrap(
|
||||
Bundle.module.url(
|
||||
forResource: name, withExtension: "bin", subdirectory: Self.fixtureDir),
|
||||
"missing fixture \(name).bin — regenerate with pyrowave_dump_golden")
|
||||
return try Data(contentsOf: url)
|
||||
}
|
||||
|
||||
/// Completion box — the decode callback lands on a Metal thread.
|
||||
private final class ResultBox: @unchecked Sendable {
|
||||
let lock = NSLock()
|
||||
var planes: WaveletPlanes?
|
||||
}
|
||||
|
||||
/// Decode `au` synchronously and read all three planes back to CPU bytes.
|
||||
private func decode(
|
||||
au: Data, chunkAligned: Bool, windowSize: Int
|
||||
) throws -> (y: [UInt8], cb: [UInt8], cr: [UInt8]) {
|
||||
let device = try XCTUnwrap(MTLCreateSystemDefaultDevice())
|
||||
let queue = try XCTUnwrap(device.makeCommandQueue())
|
||||
let decoder = try XCTUnwrap(MetalWaveletDecoder(device: device, queue: queue))
|
||||
let done = expectation(description: "decode completes")
|
||||
let box = ResultBox()
|
||||
let submitted = decoder.decode(
|
||||
au: au, chunkAligned: chunkAligned, windowSize: windowSize
|
||||
) { planes in
|
||||
box.lock.lock()
|
||||
box.planes = planes
|
||||
box.lock.unlock()
|
||||
done.fulfill()
|
||||
}
|
||||
XCTAssertTrue(submitted, "the fixture AU must parse")
|
||||
wait(for: [done], timeout: 10)
|
||||
box.lock.lock()
|
||||
let result = box.planes
|
||||
box.lock.unlock()
|
||||
let planes = try XCTUnwrap(result, "the GPU pass must complete without error")
|
||||
return (
|
||||
try readback(planes.y, device: device, queue: queue),
|
||||
try readback(planes.cb, device: device, queue: queue),
|
||||
try readback(planes.cr, device: device, queue: queue)
|
||||
)
|
||||
}
|
||||
|
||||
private func readback(
|
||||
_ texture: MTLTexture, device: MTLDevice, queue: MTLCommandQueue
|
||||
) throws -> [UInt8] {
|
||||
let bytesPerRow = texture.width
|
||||
let length = bytesPerRow * texture.height
|
||||
let buffer = try XCTUnwrap(device.makeBuffer(length: length, options: .storageModeShared))
|
||||
let cmd = try XCTUnwrap(queue.makeCommandBuffer())
|
||||
let blit = try XCTUnwrap(cmd.makeBlitCommandEncoder())
|
||||
blit.copy(
|
||||
from: texture, sourceSlice: 0, sourceLevel: 0,
|
||||
sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0),
|
||||
sourceSize: MTLSize(width: texture.width, height: texture.height, depth: 1),
|
||||
to: buffer, destinationOffset: 0, destinationBytesPerRow: bytesPerRow,
|
||||
destinationBytesPerImage: length)
|
||||
blit.endEncoding()
|
||||
cmd.commit()
|
||||
cmd.waitUntilCompleted()
|
||||
return [UInt8](UnsafeRawBufferPointer(start: buffer.contents(), count: length))
|
||||
}
|
||||
|
||||
private func psnr(_ a: [UInt8], _ b: [UInt8]) -> Double {
|
||||
precondition(a.count == b.count)
|
||||
var sse = 0.0
|
||||
for i in 0..<a.count {
|
||||
let d = Double(a[i]) - Double(b[i])
|
||||
sse += d * d
|
||||
}
|
||||
if sse == 0 { return .infinity }
|
||||
let mse = sse / Double(a.count)
|
||||
return 10 * log10(255.0 * 255.0 / mse)
|
||||
}
|
||||
|
||||
private func assertMatchesReference(
|
||||
_ decoded: (y: [UInt8], cb: [UInt8], cr: [UInt8]), prefix: String,
|
||||
file: StaticString = #filePath, line: UInt = #line
|
||||
) throws {
|
||||
for (name, plane, ref) in [
|
||||
("y", decoded.y, try fixture("\(prefix)-y")),
|
||||
("cb", decoded.cb, try fixture("\(prefix)-cb")),
|
||||
("cr", decoded.cr, try fixture("\(prefix)-cr")),
|
||||
] {
|
||||
XCTAssertEqual(plane.count, ref.count, file: file, line: line)
|
||||
let db = psnr(plane, [UInt8](ref))
|
||||
print("pyrowave golden \(prefix) \(name): \(db) dB")
|
||||
// The Metal port and upstream's decoder run the same math at the same precision
|
||||
// tier; residual differences are float rounding + the gather/mirror edge handling.
|
||||
// Well-matched ports measure ≫50 dB; 45 catches a real divergence long before it
|
||||
// is visible.
|
||||
XCTAssertGreaterThan(db, 45.0, "plane PSNR \(db) dB", file: file, line: line)
|
||||
}
|
||||
}
|
||||
|
||||
func testDenseGoldenFrame() throws {
|
||||
try XCTSkipIf(!MetalWaveletDecoder.supported, "no capable Metal device")
|
||||
let au = try fixture("au-dense")
|
||||
let decoded = try decode(au: au, chunkAligned: false, windowSize: 0)
|
||||
try assertMatchesReference(decoded, prefix: "ref-dense")
|
||||
}
|
||||
|
||||
func testChunkAlignedGoldenFrame() throws {
|
||||
try XCTSkipIf(!MetalWaveletDecoder.supported, "no capable Metal device")
|
||||
let au = try fixture("au-chunked")
|
||||
let decoded = try decode(au: au, chunkAligned: true, windowSize: 1408)
|
||||
try assertMatchesReference(decoded, prefix: "ref-chunked")
|
||||
}
|
||||
|
||||
/// Phase-4 partial delivery: zero a mid-AU window (a lost shard) — the frame must still
|
||||
/// decode (blocks > half) and stay recognizably the same picture (holes reconstruct as
|
||||
/// localized blur, not garbage).
|
||||
func testPartialFrameStillDecodes() throws {
|
||||
try XCTSkipIf(!MetalWaveletDecoder.supported, "no capable Metal device")
|
||||
var au = try fixture("au-chunked")
|
||||
let windows = au.count / 1408
|
||||
try XCTSkipIf(windows < 3, "fixture too small to punch a hole in")
|
||||
let hole = (windows / 2) * 1408
|
||||
au.replaceSubrange(hole..<(hole + 1408), with: [UInt8](repeating: 0, count: 1408))
|
||||
let decoded = try decode(au: au, chunkAligned: true, windowSize: 1408)
|
||||
let ref = try fixture("ref-chunked-y")
|
||||
let db = psnr(decoded.y, [UInt8](ref))
|
||||
XCTAssertGreaterThan(db, 25.0, "lossy frame should still resemble the source (\(db) dB)")
|
||||
}
|
||||
}
|
||||
#endif
|
||||
Binary file not shown.
Binary file not shown.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -47,18 +47,21 @@ final class Stage444Tests: XCTestCase {
|
||||
box.lock.lock(); let frame = box.frame; let error = box.error; box.lock.unlock()
|
||||
XCTAssertNil(error.map { "decode error \($0)" })
|
||||
let ready = try XCTUnwrap(frame, "a 4:4:4 ReadyFrame must be delivered")
|
||||
XCTAssertEqual(CVPixelBufferGetWidth(ready.pixelBuffer), 256)
|
||||
XCTAssertEqual(CVPixelBufferGetHeight(ready.pixelBuffer), 256)
|
||||
let pf = CVPixelBufferGetPixelFormatType(ready.pixelBuffer)
|
||||
guard case .video(let buffer, let isHDR) = ready.image else {
|
||||
return XCTFail("a VideoToolbox decode must deliver a .video frame")
|
||||
}
|
||||
XCTAssertEqual(CVPixelBufferGetWidth(buffer), 256)
|
||||
XCTAssertEqual(CVPixelBufferGetHeight(buffer), 256)
|
||||
let pf = CVPixelBufferGetPixelFormatType(buffer)
|
||||
XCTAssertTrue(
|
||||
pf == kCVPixelFormatType_444YpCbCr8BiPlanarVideoRange
|
||||
|| pf == kCVPixelFormatType_444YpCbCr8BiPlanarFullRange,
|
||||
"expected a biplanar 4:4:4 8-bit buffer, got \(fourCCString(pf))")
|
||||
XCTAssertFalse(ready.isHDR, "an 8-bit BT.709 4:4:4 stream is SDR")
|
||||
XCTAssertFalse(isHDR, "an 8-bit BT.709 4:4:4 stream is SDR")
|
||||
// The chroma plane (plane 1) must be FULL resolution for 4:4:4 (vs half for 4:2:0) — this is
|
||||
// what lets the unchanged shader sample chroma at the luma UV.
|
||||
XCTAssertEqual(CVPixelBufferGetWidthOfPlane(ready.pixelBuffer, 1), 256)
|
||||
XCTAssertEqual(CVPixelBufferGetHeightOfPlane(ready.pixelBuffer, 1), 256)
|
||||
XCTAssertEqual(CVPixelBufferGetWidthOfPlane(buffer, 1), 256)
|
||||
XCTAssertEqual(CVPixelBufferGetHeightOfPlane(buffer, 1), 256)
|
||||
}
|
||||
|
||||
private func fourCCString(_ t: OSType) -> String {
|
||||
|
||||
@@ -99,8 +99,9 @@ final class VideoToolboxRoundTripTests: XCTestCase {
|
||||
box.lock.unlock()
|
||||
XCTAssertNil(error.map { "decode error \($0)" })
|
||||
let ready = try XCTUnwrap(frame, "the async output callback must deliver a ReadyFrame")
|
||||
XCTAssertEqual(CVPixelBufferGetWidth(ready.pixelBuffer), width)
|
||||
XCTAssertEqual(CVPixelBufferGetHeight(ready.pixelBuffer), height)
|
||||
let buffer = try XCTUnwrap(ready.pixelBuffer, "a VT decode delivers a .video frame")
|
||||
XCTAssertEqual(CVPixelBufferGetWidth(buffer), width)
|
||||
XCTAssertEqual(CVPixelBufferGetHeight(buffer), height)
|
||||
XCTAssertEqual(ready.ptsNs, 42_000_000, "pts round-trips through the decoder")
|
||||
XCTAssertEqual(
|
||||
ready.receivedNs, 41_000_000, "receivedNs round-trips through the frame refcon")
|
||||
|
||||
@@ -2230,6 +2230,33 @@ pub unsafe extern "C" fn punktfunk_connection_codec(
|
||||
})
|
||||
}
|
||||
|
||||
/// Read the session's negotiated wire shard payload (the `Welcome`'s value, bytes). This is the
|
||||
/// parse-window size of a [`USER_FLAG_CHUNK_ALIGNED`] AU (PyroWave datagram-aligned mode,
|
||||
/// design/pyrowave-codec-plan.md §4.4): every `shard_payload`-sized window of the frame buffer
|
||||
/// starts a fresh self-delimiting chunk. Clients that decode PyroWave natively (the Apple Metal
|
||||
/// port) need it to walk those AUs; other codecs never need this.
|
||||
///
|
||||
/// # Safety
|
||||
/// `c` is a valid connection handle; `out` is NULL or writable for one `u32`.
|
||||
#[cfg(feature = "quic")]
|
||||
#[no_mangle]
|
||||
pub unsafe extern "C" fn punktfunk_connection_shard_payload(
|
||||
c: *mut PunktfunkConnection,
|
||||
out: *mut u32,
|
||||
) -> PunktfunkStatus {
|
||||
guard(|| {
|
||||
let c = match unsafe { c.as_ref() } {
|
||||
Some(c) => c,
|
||||
None => return PunktfunkStatus::NullPointer,
|
||||
};
|
||||
if !out.is_null() {
|
||||
// SAFETY: `out` is non-null and the caller guarantees it is writable for one `u32`.
|
||||
unsafe { *out = u32::from(c.inner.shard_payload) };
|
||||
}
|
||||
PunktfunkStatus::Ok
|
||||
})
|
||||
}
|
||||
|
||||
/// Send one input event to the host as a QUIC datagram (non-blocking enqueue).
|
||||
///
|
||||
/// # Safety
|
||||
|
||||
@@ -1130,10 +1130,10 @@ mod tests {
|
||||
)
|
||||
}
|
||||
|
||||
/// Decode an AU with a standalone pyrowave CPU decoder and return plane means (Y, Cb, Cr).
|
||||
/// This is the Phase-1 "golden frames" oracle: the host-encoded bitstream must round-trip
|
||||
/// through upstream's own decoder to the CSC's expected values.
|
||||
unsafe fn decode_plane_means(w: u32, h: u32, au: &[u8]) -> (f64, f64, f64) {
|
||||
/// Decode an AU with a standalone pyrowave decoder and return the full YUV420P planes.
|
||||
/// This is the golden oracle for both the Phase-1 smoke check (plane means) and the Apple
|
||||
/// Metal port's committed PSNR fixtures (`pyrowave_dump_golden`).
|
||||
unsafe fn decode_planes(w: u32, h: u32, au: &[u8]) -> (Vec<u8>, Vec<u8>, Vec<u8>) {
|
||||
let mut dev: pw::pyrowave_device = std::ptr::null_mut();
|
||||
assert_eq!(
|
||||
pw::pyrowave_create_default_device(&mut dev),
|
||||
@@ -1177,7 +1177,13 @@ mod tests {
|
||||
);
|
||||
pw::pyrowave_decoder_destroy(dec);
|
||||
pw::pyrowave_device_destroy(dev);
|
||||
(y, cb, cr)
|
||||
}
|
||||
|
||||
/// Plane means of an upstream-decoded AU — the Phase-1 smoke assertion.
|
||||
unsafe fn decode_plane_means(w: u32, h: u32, au: &[u8]) -> (f64, f64, f64) {
|
||||
// SAFETY: forwarded — same contract as the caller.
|
||||
let (y, cb, cr) = unsafe { decode_planes(w, h, au) };
|
||||
let mean = |v: &[u8]| v.iter().map(|&x| x as f64).sum::<f64>() / v.len() as f64;
|
||||
(mean(&y), mean(&cb), mean(&cr))
|
||||
}
|
||||
@@ -1304,4 +1310,107 @@ mod tests {
|
||||
.expect("submit after reset");
|
||||
assert!(enc.poll().expect("poll").is_some());
|
||||
}
|
||||
|
||||
/// A deterministic busy BGRA test card (gradients + checker + LCG noise) — flat fills
|
||||
/// exercise almost none of the entropy decoder, this hits every subband.
|
||||
fn test_card(w: u32, h: u32, seed: u32) -> CapturedFrame {
|
||||
let mut rng = seed | 1;
|
||||
let mut buf = vec![0u8; (w * h * 4) as usize];
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
rng = rng.wrapping_mul(1664525).wrapping_add(1013904223);
|
||||
let i = ((y * w + x) * 4) as usize;
|
||||
let checker = if (x / 16 + y / 16) % 2 == 0 { 48 } else { 0 };
|
||||
let noise = (rng >> 24) as u8 / 8;
|
||||
buf[i] = ((x * 255 / w) as u8).saturating_add(noise); // B
|
||||
buf[i + 1] = ((y * 255 / h) as u8).saturating_add(checker); // G
|
||||
buf[i + 2] = (((x + y) * 255 / (w + h)) as u8).saturating_add(noise); // R
|
||||
buf[i + 3] = 255;
|
||||
}
|
||||
}
|
||||
CapturedFrame {
|
||||
width: w,
|
||||
height: h,
|
||||
pts_ns: seed as u64 * 16_666_667,
|
||||
format: PixelFormat::Bgrx,
|
||||
payload: FramePayload::Cpu(buf),
|
||||
}
|
||||
}
|
||||
|
||||
/// Dump the Apple Metal port's golden fixtures (plan §4.7): host-encoded AUs (dense AND
|
||||
/// chunk-aligned) plus upstream's own decode of each as raw YUV420P planes. The Swift test
|
||||
/// (PyroWaveGoldenTests.swift) PSNR-matches the Metal decode against these — float wavelet
|
||||
/// math is not bit-exact across implementations, upstream itself ships precision variants.
|
||||
/// `#[ignore]`d GPU test; regenerate on a Vulkan 1.3 host:
|
||||
/// cargo test -p punktfunk-host --features pyrowave --no-run
|
||||
/// PYROWAVE_GOLDEN_DIR=/tmp/golden <bin> --ignored --nocapture pyrowave_dump_golden
|
||||
/// then copy the files into clients/apple/Tests/PunktfunkKitTests/PyroWaveFixtures/.
|
||||
#[test]
|
||||
#[ignore = "fixture generator — needs a real Vulkan 1.3 compute device"]
|
||||
fn pyrowave_dump_golden() {
|
||||
let dir = match std::env::var("PYROWAVE_GOLDEN_DIR") {
|
||||
Ok(d) => std::path::PathBuf::from(d),
|
||||
Err(_) => {
|
||||
eprintln!("PYROWAVE_GOLDEN_DIR not set — skipping dump");
|
||||
return;
|
||||
}
|
||||
};
|
||||
std::fs::create_dir_all(&dir).expect("create golden dir");
|
||||
|
||||
// Odd-block geometry on purpose: 256 aligns clean, 144 → aligned 160 exercises the
|
||||
// block-grid overhang. ~1.6 bpp at 60 fps.
|
||||
let (w, h) = (256u32, 144u32);
|
||||
let mut enc = PyroWaveEncoder::open(w, h, 60, 4_000_000).expect("open");
|
||||
|
||||
let dump = |name: &str, bytes: &[u8]| {
|
||||
std::fs::write(dir.join(name), bytes).expect("write fixture");
|
||||
eprintln!("wrote {name}: {} bytes", bytes.len());
|
||||
};
|
||||
|
||||
// Dense AU + upstream-decoded reference planes.
|
||||
enc.submit(&test_card(w, h, 7)).expect("submit");
|
||||
let au = enc.poll().expect("poll").expect("AU");
|
||||
assert!(!au.chunk_aligned);
|
||||
dump("au-dense.bin", &au.data);
|
||||
// SAFETY: test-only FFI with locally-owned buffers.
|
||||
let (y, cb, cr) = unsafe { decode_planes(w, h, &au.data) };
|
||||
dump("ref-dense-y.bin", &y);
|
||||
dump("ref-dense-cb.bin", &cb);
|
||||
dump("ref-dense-cr.bin", &cr);
|
||||
|
||||
// Chunk-aligned AU of a DIFFERENT frame (its own reference): the Swift window walk +
|
||||
// FRAG reassembly must reproduce the packet stream.
|
||||
enc.set_wire_chunking(1408);
|
||||
enc.submit(&test_card(w, h, 11)).expect("chunked submit");
|
||||
let au = enc.poll().expect("poll").expect("chunked AU");
|
||||
assert!(au.chunk_aligned);
|
||||
assert_eq!(au.data.len() % 1408, 0);
|
||||
dump("au-chunked.bin", &au.data);
|
||||
// SAFETY: test-only FFI with locally-owned buffers.
|
||||
let (y, cb, cr) = unsafe {
|
||||
// Feed upstream through the same framed walk the clients use.
|
||||
let mut stream = Vec::new();
|
||||
let mut frag: Vec<u8> = Vec::new();
|
||||
for win in au.data.chunks(1408) {
|
||||
let used = u16::from_le_bytes([win[0], win[1]]) as usize;
|
||||
let kind = u16::from_le_bytes([win[2], win[3]]);
|
||||
let body = &win[4..4 + used];
|
||||
match kind {
|
||||
0 => stream.extend_from_slice(body),
|
||||
1 => frag = body.to_vec(),
|
||||
2 => frag.extend_from_slice(body),
|
||||
3 => {
|
||||
frag.extend_from_slice(body);
|
||||
stream.extend_from_slice(&frag);
|
||||
frag.clear();
|
||||
}
|
||||
k => panic!("unknown window kind {k}"),
|
||||
}
|
||||
}
|
||||
decode_planes(w, h, &stream)
|
||||
};
|
||||
dump("ref-chunked-y.bin", &y);
|
||||
dump("ref-chunked-cb.bin", &cb);
|
||||
dump("ref-chunked-cr.bin", &cr);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,9 +47,14 @@ setting.
|
||||
1. **Host** (Linux): build/install a host with the `pyrowave` feature. On an NVIDIA host the
|
||||
capture path additionally needs `PUNKTFUNK_ENCODER=pyrowave` in `host.env` for the codec to
|
||||
be advertised; AMD/Intel hosts advertise it automatically when the feature is present.
|
||||
2. **Client** (Linux session client with the `pyrowave` feature): set **Settings → Video
|
||||
codec → PyroWave (wired LAN)** in the gamepad console, or launch with
|
||||
2. **Client**:
|
||||
- Linux session client (with the `pyrowave` feature): set **Settings → Video codec →
|
||||
PyroWave (wired LAN)** in the gamepad console, or launch with
|
||||
`PUNKTFUNK_PREFER_PYROWAVE=1`.
|
||||
- Apple (Mac, Apple TV 4K, iPad — wired networking strongly recommended): set
|
||||
**Settings → Codec → PyroWave (wired LAN)**. The option appears only on devices whose
|
||||
GPU passes the decode probe (Apple Silicon and A13-class or newer); picking it forces
|
||||
the session SDR.
|
||||
3. Leave the bitrate on Automatic: a PyroWave session pins itself to the ~1.6 bpp rate for
|
||||
your mode (≈200 Mbps at 1080p60). An explicit bitrate is honored if you set one, but the
|
||||
adaptive-bitrate controller stays off either way — this codec has no useful low-rate
|
||||
@@ -61,6 +66,7 @@ The stats overlay shows `pyrowave` as the decode path when the mode is active.
|
||||
|
||||
## Current limits
|
||||
|
||||
- Linux host + Linux client (including docked Deck) today; the Windows host and the Apple
|
||||
native port are tracked on the [roadmap](/docs/roadmap).
|
||||
- Linux host + Linux client (including docked Deck) and Apple clients (native Metal decode
|
||||
on Mac / Apple TV 4K / iPad) today; the Windows host is tracked on the
|
||||
[roadmap](/docs/roadmap).
|
||||
- 8-bit SDR, 4:2:0 only.
|
||||
|
||||
@@ -1684,6 +1684,18 @@ PunktfunkStatus punktfunk_connection_codec(PunktfunkConnection *c,
|
||||
uint8_t *out);
|
||||
#endif
|
||||
|
||||
#if defined(PUNKTFUNK_FEATURE_QUIC)
|
||||
// Read the session's negotiated wire shard payload (the `Welcome`'s value, bytes). This is the
|
||||
// parse-window size of a [`USER_FLAG_CHUNK_ALIGNED`] AU (PyroWave datagram-aligned mode,
|
||||
// design/pyrowave-codec-plan.md §4.4): every `shard_payload`-sized window of the frame buffer
|
||||
// starts a fresh self-delimiting chunk. Clients that decode PyroWave natively (the Apple Metal
|
||||
// port) need it to walk those AUs; other codecs never need this.
|
||||
//
|
||||
// # Safety
|
||||
// `c` is a valid connection handle; `out` is NULL or writable for one `u32`.
|
||||
PunktfunkStatus punktfunk_connection_shard_payload(PunktfunkConnection *c, uint32_t *out);
|
||||
#endif
|
||||
|
||||
#if defined(PUNKTFUNK_FEATURE_QUIC)
|
||||
// Send one input event to the host as a QUIC datagram (non-blocking enqueue).
|
||||
//
|
||||
|
||||
Reference in New Issue
Block a user