Files

T

enricobuehler 133e25849d feat(apple): gamepad UI v2 — controller settings + add host, aurora, macOS

Sources reorganized (client: Home/Session/Settings/Stores/Support/Trust; kit:
Audio/Connection/Gamepad/Input/Support/Video/Views) with the big files split
along the same seams.

The gamepad mode is couch-complete, and now on macOS too (the living-room
Mac case), not just iOS/iPadOS:

- GamepadSettingsView: a console-style, fully controller-navigable settings
  screen (X from the launcher) — up/down moves focus, left/right steps values
  (clamped, boundary thud), A cycles/toggles, B closes; the focused row shows a
  one-line description. Backed by GamepadMenuList, the vertical sibling of
  GamepadCarousel, and SettingsOptions — the option lists hoisted out of
  SettingsView statics and shared by the touch, tvOS and gamepad settings.
- GamepadAddHostView + GamepadKeyboard: register a host end to end with a pad
  — field rows open an on-screen controller keyboard (dpad grid, A types,
  X backspaces, B done); the launcher carousel ends in an Add Host tile, so
  the dead-end "add one with touch first" empty state is gone.
- Launcher polish: contextual hint bar with the pad's real button glyphs,
  controller name + battery chip, one shared console chrome.
- GamepadScreenBackground: an animated aurora (TimelineView-driven drifting
  blobs in the brand's violet family, breathing radii, slow hue shift,
  legibility scrim; freezes under Reduce Motion). Pure SwiftUI on purpose — a
  .metal library only bundles reliably in one of the two build systems (SPM vs
  the xcodeproj's synced folders) these sources compile under.
- macOS port: settings/add-host/library present as sized sheets (a macOS sheet
  takes its content's IDEAL size, and the GeometryReader-driven screens
  collapsed to nothing), NSScreen-based mode lists, scroll indicators .never
  (the "always show scroll bars" setting overrides .hidden), tray scrims so
  scrolled rows dim under the pinned title/hints, extra title clearance, and a
  PUNKTFUNK_FORCE_GAMEPAD_UI=1 dev hook — launcher/settings/add-host/keyboard/
  library render-verified live on a real Mac + LAN hosts.
- GamepadMenuInput: X button support, and (re)start now snapshots held buttons
  so a controller handoff press never fires twice (the B that closed the
  keyboard no longer also cancels the screen underneath).
- Cleanups: one "Connection failed" alert in ContentView instead of one per
  home screen; HostDiscovery.advertises/unsaved shared by both home screens.
- host: can_encode_444 stub for the non-Linux/Windows host build (the macOS
  synthetic-source loopback used by the Swift tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-02 11:24:44 +02:00

7.8 KiB

Raw Blame History

punktfunk — Apple client (macOS · iOS · iPadOS · tvOS)

The native Apple app for streaming a punktfunk host to your Mac, iPhone, iPad, or Apple TV. A SwiftUI app that finds hosts on your network, pairs with a PIN, and streams at your display's own resolution and refresh rate — with VideoToolbox hardware decode and full controller support.

All the networking and protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM, Opus audio, cert pinning — lives in the shared Rust punktfunk-core (statically linked as PunktfunkCore.xcframework). This package is the Swift shell: decode, present, input, and UI.

Features

Hardware decode — VideoToolbox HEVC, with a low-latency stage-2 presenter (VTDecompressionSession → CAMetalLayer, presented off a CADisplayLink, ~11 ms p50) as the default and an AVSampleBufferDisplayLayer fallback.
HDR & 4:4:4 — PQ passthrough with a correct reference-white anchor, mid-session SDR↔HDR reconfiguration, and hardware-probed 4:4:4 support.
Your display's native mode — the host builds a virtual output at exactly your WxH@Hz; mid-stream resize renegotiates without reconnecting.
Audio both ways — Opus playback (CoreAudio, no bundled libopus) with a jitter ring, plus mic uplink; speaker/mic selectable in Settings.
Full controller support — one selected controller forwarded as pad 0, including DualSense feedback (rumble → CoreHaptics, lightbar, player LEDs, adaptive triggers) and touchpad/motion. The virtual pad type auto-resolves from your physical controller.
Mouse & keyboard — GCMouse/GCKeyboard capture with click-to-capture and a ⌘⎋ release, plus iPad pointer lock and touch input.
Find hosts automatically — mDNS discovery (NWBrowser over _punktfunk._udp); first connect does a one-time SPAKE2 PIN pairing (or TOFU on trusted LANs), then reconnects on a pinned, Keychain-stored identity.
Tune the stream — a fps / Mb·s / latency HUD (skew-corrected across machines), a bitrate control, a per-host network speed test with a recommended bitrate, and a host-compositor picker.

Runs from one shared codebase across macOS, iOS, iPadOS, and tvOS.

Get it

Install from the App Store / TestFlight, or build from source below. Per-device install steps and the pairing walkthrough: docs.punktfunk.unom.io/docs/install-client.

Build / run / test (on a Mac)

Requires Xcode 26.5 / Swift 6.3. First build the Rust core into an xcframework, then build the app:

rustup target add aarch64-apple-darwin x86_64-apple-darwin
bash scripts/build-xcframework.sh            # → clients/apple/PunktfunkCore.xcframework
#   BUILD_IOS=1  also builds the iOS slices  (add the ios rustup targets)
#   BUILD_TVOS=1 also builds tvOS            (tier-3 targets, built from source — see below)

cd clients/apple
open Punktfunk.xcodeproj                      # the real app: ⌘R builds + runs Punktfunk.app
swift run PunktfunkClient                     # or the unbundled dev shell (CLI)
swift build && swift test                     # unit + loopback/remote tests (self-skip w/o a host)

tvOS slices are tier-3 Rust targets, built from source: rustup toolchain install nightly && rustup component add rust-src --toolchain nightly.

Test against a host

# full loopback proof — builds punktfunk-host (synthetic source, runs on macOS) and streams
# byte-verified frames into the Swift client, incl. the PIN pairing ceremony:
bash test-loopback.sh

# against a real Linux host on the LAN (see the repo README "Running on this box"):
PUNKTFUNK_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests            # headless
PUNKTFUNK_AUTOCONNECT=<box-ip> PUNKTFUNK_MODE=1280x720x60 swift run PunktfunkClient # on glass

Project layout

PunktfunkKit (library) — the reusable pieces:
- PunktfunkConnection — the wrapper over the C ABI (thread-safe close(), per-plane locks, pinning + TOFU).
- AnnexB / StreamView / VideoDecoder / MetalVideoPresenter — format handling, the stage-1 (AVSampleBufferDisplayLayer) and stage-2 (VTDecompressionSession → CAMetalLayer) presenters.
- InputCapture — GCMouse/GCKeyboard → host VK/mouse, with fractional-delta accumulation.
- GamepadManager / GamepadCapture / GamepadFeedback / DualSenseTriggerEffect — controller discovery + selection, capture (buttons/axes/touchpad/motion), and host-feedback rendering.
- HostDiscovery — NWBrowser over _punktfunk._udp.
PunktfunkClient (the app) — hosts grid with an On this network section, add-host sheet, the two trust flows (TOFU prompt + SPAKE2 PairSheet), the stream view with the HUD, a tabbed Settings pane (General / Display / Audio / Controllers / Advanced), and the network speed test. A Scene-level Stream menu carries Disconnect (⌘D) and the HUD toggle (⌘⇧S). On iOS/iPadOS and macOS a connected controller swaps the whole home for the gamepad UI (Home/Gamepad*, Settings/GamepadSettingsView): a console-style host carousel (A connect · Y library · X settings), a controller-navigable settings screen, an add-host flow with an on-screen controller keyboard (no touch required anywhere), and the coverflow library browser — all driven by the shared GamepadMenuInput poller + GamepadCarousel/GamepadMenuList focus machinery, with dual-channel haptics (device Taptic + controller MenuHaptics), over an animated "aurora" backdrop (GamepadScreenBackground — TimelineView-driven drifting color blobs; deliberately pure SwiftUI, since a .metal library only reliably bundles in one of the two build systems these sources compile under). macOS presents the settings/add-host screens as sheets (no fullScreenCover there); PUNKTFUNK_FORCE_GAMEPAD_UI=1 forces the mode without a physical pad (dev/screenshots).
Tests (swift test) — Annex-B units, a real-codec VideoToolbox round trip, DualSense trigger-effect and gamepad-wire conversions, loopback integration against real local hosts, and the remote first-light test.

Notes for contributors

Xcode project (Punktfunk.xcodeproj) wraps the same sources as the swift run shell (a synchronized folder — no duplication). The macOS target is App-Sandboxed (needs network.server — the raw-UDP plane and quinn both bind()); iOS/tvOS use the shared entitlements file (keep app-sandbox out of it). Verify with codesign -d --entitlements :- <built .app>.
Decode flow: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, and recovery keyframes re-send them — refresh the format description on every IDR; there is no out-of-band extradata, ever.
ABI threading: one video pump thread per connection, one optional audio drain thread, and one optional feedback drain thread (rumble + HID-output). send() is enqueue-only and safe alongside all of them. The wrapper's per-plane locks make close() safe from anywhere.
DualSense motion scale (GamepadWire) is derived from hid-playstation's math, not yet live-verified — if gyro/accel feel wrong in a game, correct sign/scale there and evtest the host's virtual pad.
App Store screenshots are automated — tools/screenshots.sh all renders the real UI at the required pixel sizes via a DEBUG-only shot mode; the apple CI workflow captures the iOS sizes on every main push. See the script header for details.
Deeper design notes live in design/apple-stage2-presenter.md.

Documentation — quick start, pairing, troubleshooting
Project README — the host, the other clients, and how it all fits together

7.8 KiB Raw Blame History