punktfunk/clients/apple/README.md

punktfunk — Apple client (macOS · iOS · iPadOS · tvOS)

The native Apple app for streaming a punktfunk host to your Mac, iPhone, iPad, or Apple TV. A SwiftUI app that finds hosts on your network, pairs with a PIN, and streams at your display's own resolution and refresh rate — with VideoToolbox hardware decode and full controller support.

All the networking and protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM, Opus audio, cert pinning — lives in the shared Rust punktfunk-core (statically linked as PunktfunkCore.xcframework). This package is the Swift shell: decode, present, input, and UI.

Features

• Hardware decode — VideoToolbox H.264/HEVC (plus AV1 on devices with an AV1 hardware decoder — M3-class Macs, A17 Pro-class iPhones), with a low-latency stage-2 presenter (VTDecompressionSession → CAMetalLayer, presented off a CADisplayLink, ~11 ms p50) as the default and an AVSampleBufferDisplayLayer fallback.
• HDR & 4:4:4 — PQ passthrough with a correct reference-white anchor, mid-session SDR↔HDR reconfiguration, and hardware-probed 4:4:4 support.
• Your display's native mode — the host builds a virtual output at exactly your WxH@Hz; mid-stream resize renegotiates without reconnecting.
• Audio both ways — Opus playback (CoreAudio, no bundled libopus) with a jitter ring, plus mic uplink; speaker/mic selectable in Settings.
• Full controller support — one selected controller forwarded as pad 0, including DualSense feedback (rumble → CoreHaptics, lightbar, player LEDs, adaptive triggers) and touchpad/motion. The virtual pad type auto-resolves from your physical controller.
• Mouse & keyboard — GCMouse/GCKeyboard capture with click-to-capture and a ⌃⌥⇧Q release (the cross-client Ctrl+Alt+Shift+Q; ⌘⎋ still works as the macOS/iPad toggle), plus iPad pointer lock and touch input.
• Find hosts automatically — mDNS discovery (NWBrowser over _punktfunk._udp); first connect does a one-time SPAKE2 PIN pairing (or TOFU on trusted LANs), then reconnects on a pinned, Keychain-stored identity.
• Tune the stream — a fps / Mb·s / latency HUD (skew-corrected across machines), a bitrate control, a per-host network speed test with a recommended bitrate, and a host-compositor picker.

Runs from one shared codebase across macOS, iOS, iPadOS, and tvOS.

Get it

Install from the App Store / TestFlight, or build from source below. Per-device install steps and the pairing walkthrough: docs.punktfunk.unom.io/docs/install-client.

Build / run / test (on a Mac)

Requires Xcode 26.5 / Swift 6.3. First build the Rust core into an xcframework, then build the app:

rustup target add aarch64-apple-darwin x86_64-apple-darwin
bash scripts/build-xcframework.sh            # → clients/apple/PunktfunkCore.xcframework
#   BUILD_IOS=1  also builds the iOS slices  (add the ios rustup targets)
#   BUILD_TVOS=1 also builds tvOS            (tier-3 targets, built from source — see below)

cd clients/apple
open Punktfunk.xcodeproj                      # the real app: ⌘R builds + runs Punktfunk.app
swift run PunktfunkClient                     # or the unbundled dev shell (CLI)
swift build && swift test                     # unit + loopback/remote tests (self-skip w/o a host)

tvOS slices are tier-3 Rust targets, built from source: rustup toolchain install nightly && rustup component add rust-src --toolchain nightly.

Test against a host

# full loopback proof — builds punktfunk-host (synthetic source, runs on macOS) and streams
# byte-verified frames into the Swift client, incl. the PIN pairing ceremony:
bash test-loopback.sh

# against a real Linux host on the LAN (see the repo README "Running on this box"):
PUNKTFUNK_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests            # headless
PUNKTFUNK_AUTOCONNECT=<box-ip> PUNKTFUNK_MODE=1280x720x60 swift run PunktfunkClient # on glass

Project layout

• PunktfunkKit (library) — the reusable pieces:
  • PunktfunkConnection — the wrapper over the C ABI (thread-safe close(), per-plane locks, pinning + TOFU).
  • AnnexB / StreamView / VideoDecoder / MetalVideoPresenter — format handling, the stage-1 (AVSampleBufferDisplayLayer) and stage-2 (VTDecompressionSession → CAMetalLayer) presenters.
  • InputCapture — GCMouse/GCKeyboard → host VK/mouse, with fractional-delta accumulation.
  • GamepadManager / GamepadCapture / GamepadFeedback / DualSenseTriggerEffect — controller discovery + selection, capture (buttons/axes/touchpad/motion), and host-feedback rendering.
  • HostDiscovery — NWBrowser over _punktfunk._udp.
• PunktfunkClient (the app) — hosts grid with an On this network section, add-host sheet, the two trust flows (TOFU prompt + SPAKE2 PairSheet), the stream view with the HUD, a tabbed Settings pane (General / Display / Audio / Controllers / Advanced), and the network speed test. A Scene-level Stream menu carries the cross-client shortcut set: Release Mouse (⌃⌥⇧Q), Disconnect (⌃⌥⇧D) and the HUD toggle (⌃⌥⇧S) — the same Ctrl+Alt+Shift combos the Windows and Linux clients reserve, also shown on a 6-second banner at stream start. On iOS/iPadOS and macOS a connected controller swaps the whole home for the gamepad UI (Home/Gamepad*, Settings/GamepadSettingsView): a console-style host carousel (A connect · Y library · X settings), a controller-navigable settings screen, an add-host flow with an on-screen controller keyboard (no touch required anywhere), and the coverflow library browser — all driven by the shared GamepadMenuInput poller + GamepadCarousel/GamepadMenuList focus machinery, with dual-channel haptics (device Taptic + controller MenuHaptics), over an animated "aurora" backdrop (GamepadScreenBackground — TimelineView-driven drifting color blobs; deliberately pure SwiftUI, since a .metal library only reliably bundles in one of the two build systems these sources compile under). macOS presents the settings/add-host screens as sheets (no fullScreenCover there); PUNKTFUNK_FORCE_GAMEPAD_UI=1 forces the mode without a physical pad (dev/screenshots).
• Tests (swift test) — Annex-B units, a real-codec VideoToolbox round trip, DualSense trigger-effect and gamepad-wire conversions, loopback integration against real local hosts, and the remote first-light test.

Notes for contributors

• Xcode project (Punktfunk.xcodeproj) wraps the same sources as the swift run shell (a synchronized folder — no duplication). The macOS target is App-Sandboxed (needs network.server — the raw-UDP plane and quinn both bind()); iOS/tvOS use the shared entitlements file (keep app-sandbox out of it). Verify with codesign -d --entitlements :- <built .app>.
• Decode flow: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, and recovery keyframes re-send them — refresh the format description on every IDR; there is no out-of-band extradata, ever.
• ABI threading: one video pump thread per connection, one optional audio drain thread, and one optional feedback drain thread (rumble + HID-output). send() is enqueue-only and safe alongside all of them. The wrapper's per-plane locks make close() safe from anywhere.
• DualSense motion scale (GamepadWire) is derived from hid-playstation's math, not yet live-verified — if gyro/accel feel wrong in a game, correct sign/scale there and evtest the host's virtual pad.
• App Store screenshots are automated — tools/screenshots.sh all renders the real UI at the required pixel sizes via a DEBUG-only shot mode; the apple CI workflow captures the iOS sizes on every main push. See the script header for details.
• Deeper design notes live in the internal planning repo (punktfunk-planning: apple-stage2-presenter.md).

Related

• Documentation — quick start, pairing, troubleshooting
• Project README — the host, the other clients, and how it all fits together