enricobuehler
d5e23146c0
The ladder's last rung no longer runs FFmpeg. H.264 decodes through openh264, AV1 through rav1d, and HEVC is refused outright: no permissively licensed software HEVC decoder exists, so an HEVC session that exhausts its hardware rungs now tears down and re-dials advertising HEVC-less caps, and the host picks H.264. The plan calls that a first-class path; it is one. swscale is deleted, and with it the BT.601 default that its correction code existed to undo. Colour on the H.264 lane now comes from the same pf-bitstream planner every hardware rung submits from — openh264 reports no VUI at all — and AV1's comes per-picture from the sequence header. One colour source, one CSC: the old default is unrepresentable rather than merely fixed. Frames reach the presenter as three tightly-packed planes through the planar CSC pass, which had to be un-gated from the pyrowave feature and its device probe, since the last rung must exist on devices that failed that probe. rav1d rather than the dav1d crate, deliberately and against the plan's literal wording: dav1d-sys is system-deps-only, so it would add a system library and a .pc file to every client package — in the milestone family whose excision checklist exists to delete exactly that. rav1d is the same decoder, same licence, statically linked. The cost is honest: no-asm builds on both decoders, and software throughput is still unmeasured. The colour test is the milestone's exit criterion, so it is built to fail. Three fixtures, and a mutation check: hardcoding the swscale default turns the red bar to [255,24,0], and swapping Cb/Cr turns red to blue — a silent error no metadata assertion could catch. Review then disproved the range half of it numerically: with eight saturated bars, decoding the full-range fixture with the wrong range gives max error ZERO, because a mismatch only pushes values outside [0,1] where the shader clamps. A mid-tone was added; the wrong range now costs 11, well past the tolerance. The exit criterion I set was satisfiable by a test that proved nothing. Two blocking defects, both emergent rather than local. Software AV1 on a 10-bit stream never reached its typed refusal: rav1d is built 8-bit-only and returns ENOPROTOOPT, which the send loop turned into a generic error, so the pump's typed downcast missed and every AU failed identically — a permanent freeze on precisely the shipping case, since AV1 is advertised only where hardware AV1 exists and hardware AV1 plus HDR is Main 10. The shape is now read from the sequence header before any byte reaches the decoder, exactly as the H.264 leg reads the active SPS. And the new Reconnecting phase was the first state that is not streaming, not connecting, and still holding a live stream — which opened all three guards that had made a second launch impossible. Pressing A assigned over `stream` where every other site shuts down first, and StreamState has no Drop, so the old pump was detached: a second live session still submitting to a Vulkan device that gets destroyed underneath it. Nothing about the reconnect was wrong in isolation; the defect lived between a new state and three guards nobody re-examined. Start is now defensive and the retry raises the connecting modal, so the UI matches the state and B can cancel. Also closed: retry_caps was computed, tested and never applied, so a shape refusal could end a session reporting no codec available while a working retry existed; the retry inherited force_software sticky-true, landing an HEVC→H.264 fallback on software H.264 with working hardware H.264; it re-dialled with a stale mode; the CPU present arm had no survivable-failure handling where the pyrowave arm — same pass — has it; HEVC is no longer advertised when the decoder is pinned to software; and the software rung now feeds the recovery-point SEI it already had in hand to the re-anchor gate. ⚠ Two host-side gaps found while tracing, neither in scope here: Hello::launch is NOT idempotent (gog:/custom: targets spawn a second copy on a retry; the field is kept verbatim because dropping it orphans the gamescope display whose reuse key includes the command), and a reconnected session can never adopt a game predating its own launch stamp, so it has no game-exit detection. ⚠ OWED: the on-glass software run. ~200 lines of new Vulkan on a path that only runs because the GPU already failed, and no driver has seen it. The review's minimum check is sync validation enabled, a non-multiple-of-16 mode, a mid-session resize and demotion, and both colour matrices. Gates: container clippy -D warnings over four crates, 236 tests, workspace check. pf-vkdecode and pf-bitstream are byte-for-byte untouched, so the hardware rungs' 250/250 stands.
punktfunk — Apple client (macOS · iOS · iPadOS · tvOS)
The native Apple app for streaming a punktfunk host to your Mac, iPhone, iPad, or Apple TV. A SwiftUI app that finds hosts on your network, pairs with a PIN, and streams at your display's own resolution and refresh rate — with VideoToolbox hardware decode and full controller support.
All the networking and protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM,
Opus audio, cert pinning — lives in the shared Rust punktfunk-core (statically linked as
PunktfunkCore.xcframework). This package is the Swift shell: decode, present, input, and UI.
Features
- Hardware decode — VideoToolbox H.264/HEVC (plus AV1 on devices with an AV1 hardware
decoder — M3-class Macs, A17 Pro-class iPhones), with a low-latency stage-2 presenter
(
VTDecompressionSession→CAMetalLayer, presented off aCADisplayLink, ~11 ms p50) as the default and anAVSampleBufferDisplayLayerfallback. - HDR & 4:4:4 — PQ passthrough with a correct reference-white anchor, mid-session SDR↔HDR reconfiguration, and hardware-probed 4:4:4 support.
- Your display's native mode — the host builds a virtual output at exactly your WxH@Hz; mid-stream resize renegotiates without reconnecting.
- Audio both ways — Opus playback (CoreAudio, no bundled libopus) with a jitter ring, plus mic uplink; speaker/mic selectable in Settings.
- Full controller support — one selected controller forwarded as pad 0, including DualSense feedback (rumble → CoreHaptics, lightbar, player LEDs, adaptive triggers) and touchpad/motion. The virtual pad type auto-resolves from your physical controller.
- Mouse & keyboard —
GCMouse/GCKeyboardcapture with click-to-capture and a ⌃⌥⇧Q release (the cross-client Ctrl+Alt+Shift+Q; ⌘⎋ still works as the macOS/iPad toggle), plus iPad pointer lock and touch input. - Find hosts automatically — mDNS discovery (
NWBrowserover_punktfunk._udp); first connect does a one-time SPAKE2 PIN pairing (or TOFU on trusted LANs), then reconnects on a pinned, Keychain-stored identity. - Tune the stream — a fps / Mb·s / latency HUD (skew-corrected across machines), a bitrate control, a per-host network speed test with a recommended bitrate, and a host-compositor picker.
Runs from one shared codebase across macOS, iOS, iPadOS, and tvOS.
Get it
Install from the App Store / TestFlight, or build from source below. Per-device install steps and the pairing walkthrough: docs.punktfunk.unom.io/docs/install-client.
Build / run / test (on a Mac)
Requires Xcode 26.5 / Swift 6.3. First build the Rust core into an xcframework, then build the app:
rustup target add aarch64-apple-darwin x86_64-apple-darwin
bash scripts/build-xcframework.sh # → clients/apple/PunktfunkCore.xcframework
# BUILD_IOS=1 also builds the iOS slices (add the ios rustup targets)
# BUILD_TVOS=1 also builds tvOS (tier-3 targets, built from source — see below)
cd clients/apple
open Punktfunk.xcodeproj # the real app: ⌘R builds + runs Punktfunk.app
swift run PunktfunkClient # or the unbundled dev shell (CLI)
swift build && swift test # unit + loopback/remote tests (self-skip w/o a host)
tvOS slices are tier-3 Rust targets, built from source:
rustup toolchain install nightly && rustup component add rust-src --toolchain nightly.
Test against a host
# full loopback proof — builds punktfunk-host (synthetic source, runs on macOS) and streams
# byte-verified frames into the Swift client, incl. the PIN pairing ceremony:
bash test-loopback.sh
# against a real Linux host on the LAN (see the repo README "Running on this box"):
PUNKTFUNK_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests # headless
PUNKTFUNK_AUTOCONNECT=<box-ip> PUNKTFUNK_MODE=1280x720x60 swift run PunktfunkClient # on glass
Project layout
PunktfunkKit(library) — the reusable pieces:PunktfunkConnection— the wrapper over the C ABI (thread-safeclose(), per-plane locks, pinning + TOFU).AnnexB/StreamView/VideoDecoder/MetalVideoPresenter— format handling, the stage-1 (AVSampleBufferDisplayLayer) and stage-2 (VTDecompressionSession→CAMetalLayer) presenters.InputCapture—GCMouse/GCKeyboard→ host VK/mouse, with fractional-delta accumulation.GamepadManager/GamepadCapture/GamepadFeedback/DualSenseTriggerEffect— controller discovery + selection, capture (buttons/axes/touchpad/motion), and host-feedback rendering.HostDiscovery—NWBrowserover_punktfunk._udp.
PunktfunkClient(the app) — hosts grid with an On this network section, add-host sheet, the two trust flows (TOFU prompt + SPAKE2PairSheet), the stream view with the HUD, a tabbed Settings pane (General / Display / Audio / Controllers / Advanced), and the network speed test. A Scene-level Stream menu carries the cross-client shortcut set: Release Mouse (⌃⌥⇧Q), Disconnect (⌃⌥⇧D) and the HUD toggle (⌃⌥⇧S) — the same Ctrl+Alt+Shift combos the Windows and Linux clients reserve, also shown on a 6-second banner at stream start. On iOS/iPadOS and macOS a connected controller swaps the whole home for the gamepad UI (Home/Gamepad*,Settings/GamepadSettingsView): a console-style host carousel (A connect · Y library · X settings), a controller-navigable settings screen, an add-host flow with an on-screen controller keyboard (no touch required anywhere), and the coverflow library browser — all driven by the sharedGamepadMenuInputpoller +GamepadCarousel/GamepadMenuListfocus machinery, with dual-channel haptics (device Taptic + controllerMenuHaptics), over an animated "aurora" backdrop (GamepadScreenBackground— TimelineView-driven drifting color blobs; deliberately pure SwiftUI, since a .metal library only reliably bundles in one of the two build systems these sources compile under). macOS presents the settings/add-host screens as sheets (nofullScreenCoverthere);PUNKTFUNK_FORCE_GAMEPAD_UI=1forces the mode without a physical pad (dev/screenshots).- Tests (
swift test) — Annex-B units, a real-codec VideoToolbox round trip, DualSense trigger-effect and gamepad-wire conversions, loopback integration against real local hosts, and the remote first-light test.
Notes for contributors
- Xcode project (
Punktfunk.xcodeproj) wraps the same sources as theswift runshell (a synchronized folder — no duplication). The macOS target is App-Sandboxed (needsnetwork.server— the raw-UDP plane and quinn bothbind()); iOS/tvOS use the shared entitlements file (keepapp-sandboxout of it). Verify withcodesign -d --entitlements :- <built .app>. - Decode flow: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, and recovery keyframes re-send them — refresh the format description on every IDR; there is no out-of-band extradata, ever.
- ABI threading: one video pump thread per connection, one optional audio drain thread, and one
optional feedback drain thread (rumble + HID-output).
send()is enqueue-only and safe alongside all of them. The wrapper's per-plane locks makeclose()safe from anywhere. - DualSense motion scale (
GamepadWire) is derived from hid-playstation's math, not yet live-verified — if gyro/accel feel wrong in a game, correct sign/scale there andevtestthe host's virtual pad. - App Store screenshots are automated —
tools/screenshots.sh allrenders the real UI at the required pixel sizes via a DEBUG-only shot mode; theappleCI workflow captures the iOS sizes on every main push. See the script header for details. The script'sSCENESarray is the listing set, in listing order; override it (SCENES="06-gamepad-home 10-edithost" tools/screenshots.sh ios) to capture any of the other scenes inShotScenes.all. Mock data — hosts, adverts, profiles — is seeded inShotMockso a capture is byte-for-byte deterministic and never browses the real LAN (a stranger's hostname reached the live listing that way once). - Deeper design notes live in the internal planning repo (punktfunk-planning:
apple-stage2-presenter.md).
Related
- Documentation — quick start, pairing, troubleshooting
- Project README — the host, the other clients, and how it all fits together