Files

T

enricobuehler 8f35155c14 fix(apple/shots): the store screenshots show the app as it actually is

Uploading the 0.24.0 set surfaced a pair screen that reads as broken, and a
hero that was never the orientation it claimed.

The capture harness:

- Landscape scenes were captured in PORTRAIT. `IOSOrientationConfigurator`
  asked for the geometry update from `updateUIViewController`, where
  `view.window` is still nil — SwiftUI makes one update pass for a
  `.background` representable, before the hierarchy is in a window, so the
  guard fell through and nothing ever asked again. Both `.landscape` scenes
  (the stream hero, the trust card) shipped as portrait. Now a real
  UIViewController asks from `viewDidAppear` and pins
  `supportedInterfaceOrientations`.

- The shot host applied `.ignoresSafeArea()` to the whole scene, so the
  hero's HUD — resolution, bitrate, the latency breakdown, the entire point
  of that screenshot — sat under the Dynamic Island. Only the black backing
  ignores it now; scenes that want full bleed already ignore it themselves.

- `03-pair` was hand-composed into a ZStack rather than presented. PairSheet
  is a bottom sheet on iOS: its detents and the system's Liquid Glass only
  exist inside a real `.sheet`. Composed, the grouped Form stretched to full
  screen height and the capture was a strip of content over a black void,
  with a DISABLED "Pair & Connect" (empty PIN) and the capture simulator's
  own name — `pf-shot-iphone-6.9` — rendered in as the device name.

- Sheets do not inherit `.environment(\.colorScheme, .dark)` across the
  presentation boundary; they follow the DEVICE. The pairing sheet came out
  light grey over the dark app. The simulator is now set to dark appearance.

- Discovery browsed the live LAN mid-capture, so a bystanding machine's
  hostname went out on the listing and no two runs matched. `HostDiscovery`
  gains a `debugSet` seam (the counterpart to `HostWaker.debugSet`); the
  mock hosts advertise, so cards read ONLINE through the real `advertises`
  path and the reachability probe never touches the network.

- Created simulators were named `pf-shot-<prefix>`, which the reuse regex
  never matches: every run created another simulator and none was reused.
  They are named after the device now — reusable, and not user-visible junk.

Two bugs found on the way, neither screenshot-only:

- HostStore/ProfileStore PERSISTED the harness's mock data. On a dev Mac
  that is the same App-Group suite the real app reads, so running the
  script could replace the tester's saved hosts with "Battlestation" & co.
- GamepadHomeView drew the controller chip as a trailing `.overlay`, which
  reserves no width — on a portrait phone it sat on top of the centred
  "Select a Host". Laid out as a row with a hidden leading mirror.
- The pairing sheet's field prompt said "How the host lists this Mac" on
  iPhone and iPad.

Coverage: the listing set is six scenes in listing order, and is now the
stream, the machines it found, the couch/controller mode, waking a sleeping
host, the quality controls and pairing — the console and wake screens
already existed in `ShotScenes.all` and were simply never captured. Mock
hosts carry OS marks, Wake-on-LAN MACs and profile chips so the grid is
full rather than three offline rows over an empty half-screen. `SCENES=`
overrides the set for the dev scenes.

2026-08-04 21:30:18 +02:00

8.6 KiB

Raw Blame History

punktfunk — Apple client (macOS · iOS · iPadOS · tvOS)

The native Apple app for streaming a punktfunk host to your Mac, iPhone, iPad, or Apple TV. A SwiftUI app that finds hosts on your network, pairs with a PIN, and streams at your display's own resolution and refresh rate — with VideoToolbox hardware decode and full controller support.

All the networking and protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM, Opus audio, cert pinning — lives in the shared Rust punktfunk-core (statically linked as PunktfunkCore.xcframework). This package is the Swift shell: decode, present, input, and UI.

Features

Hardware decode — VideoToolbox H.264/HEVC (plus AV1 on devices with an AV1 hardware decoder — M3-class Macs, A17 Pro-class iPhones), with a low-latency stage-2 presenter (VTDecompressionSession → CAMetalLayer, presented off a CADisplayLink, ~11 ms p50) as the default and an AVSampleBufferDisplayLayer fallback.
HDR & 4:4:4 — PQ passthrough with a correct reference-white anchor, mid-session SDR↔HDR reconfiguration, and hardware-probed 4:4:4 support.
Your display's native mode — the host builds a virtual output at exactly your WxH@Hz; mid-stream resize renegotiates without reconnecting.
Audio both ways — Opus playback (CoreAudio, no bundled libopus) with a jitter ring, plus mic uplink; speaker/mic selectable in Settings.
Full controller support — one selected controller forwarded as pad 0, including DualSense feedback (rumble → CoreHaptics, lightbar, player LEDs, adaptive triggers) and touchpad/motion. The virtual pad type auto-resolves from your physical controller.
Mouse & keyboard — GCMouse/GCKeyboard capture with click-to-capture and a ⌃⌥⇧Q release (the cross-client Ctrl+Alt+Shift+Q; ⌘⎋ still works as the macOS/iPad toggle), plus iPad pointer lock and touch input.
Find hosts automatically — mDNS discovery (NWBrowser over _punktfunk._udp); first connect does a one-time SPAKE2 PIN pairing (or TOFU on trusted LANs), then reconnects on a pinned, Keychain-stored identity.
Tune the stream — a fps / Mb·s / latency HUD (skew-corrected across machines), a bitrate control, a per-host network speed test with a recommended bitrate, and a host-compositor picker.

Runs from one shared codebase across macOS, iOS, iPadOS, and tvOS.

Get it

Install from the App Store / TestFlight, or build from source below. Per-device install steps and the pairing walkthrough: docs.punktfunk.unom.io/docs/install-client.

Build / run / test (on a Mac)

Requires Xcode 26.5 / Swift 6.3. First build the Rust core into an xcframework, then build the app:

rustup target add aarch64-apple-darwin x86_64-apple-darwin
bash scripts/build-xcframework.sh            # → clients/apple/PunktfunkCore.xcframework
#   BUILD_IOS=1  also builds the iOS slices  (add the ios rustup targets)
#   BUILD_TVOS=1 also builds tvOS            (tier-3 targets, built from source — see below)

cd clients/apple
open Punktfunk.xcodeproj                      # the real app: ⌘R builds + runs Punktfunk.app
swift run PunktfunkClient                     # or the unbundled dev shell (CLI)
swift build && swift test                     # unit + loopback/remote tests (self-skip w/o a host)

tvOS slices are tier-3 Rust targets, built from source: rustup toolchain install nightly && rustup component add rust-src --toolchain nightly.

Test against a host

# full loopback proof — builds punktfunk-host (synthetic source, runs on macOS) and streams
# byte-verified frames into the Swift client, incl. the PIN pairing ceremony:
bash test-loopback.sh

# against a real Linux host on the LAN (see the repo README "Running on this box"):
PUNKTFUNK_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests            # headless
PUNKTFUNK_AUTOCONNECT=<box-ip> PUNKTFUNK_MODE=1280x720x60 swift run PunktfunkClient # on glass

Project layout

PunktfunkKit (library) — the reusable pieces:
- PunktfunkConnection — the wrapper over the C ABI (thread-safe close(), per-plane locks, pinning + TOFU).
- AnnexB / StreamView / VideoDecoder / MetalVideoPresenter — format handling, the stage-1 (AVSampleBufferDisplayLayer) and stage-2 (VTDecompressionSession → CAMetalLayer) presenters.
- InputCapture — GCMouse/GCKeyboard → host VK/mouse, with fractional-delta accumulation.
- GamepadManager / GamepadCapture / GamepadFeedback / DualSenseTriggerEffect — controller discovery + selection, capture (buttons/axes/touchpad/motion), and host-feedback rendering.
- HostDiscovery — NWBrowser over _punktfunk._udp.
PunktfunkClient (the app) — hosts grid with an On this network section, add-host sheet, the two trust flows (TOFU prompt + SPAKE2 PairSheet), the stream view with the HUD, a tabbed Settings pane (General / Display / Audio / Controllers / Advanced), and the network speed test. A Scene-level Stream menu carries the cross-client shortcut set: Release Mouse (⌃⌥⇧Q), Disconnect (⌃⌥⇧D) and the HUD toggle (⌃⌥⇧S) — the same Ctrl+Alt+Shift combos the Windows and Linux clients reserve, also shown on a 6-second banner at stream start. On iOS/iPadOS and macOS a connected controller swaps the whole home for the gamepad UI (Home/Gamepad*, Settings/GamepadSettingsView): a console-style host carousel (A connect · Y library · X settings), a controller-navigable settings screen, an add-host flow with an on-screen controller keyboard (no touch required anywhere), and the coverflow library browser — all driven by the shared GamepadMenuInput poller + GamepadCarousel/GamepadMenuList focus machinery, with dual-channel haptics (device Taptic + controller MenuHaptics), over an animated "aurora" backdrop (GamepadScreenBackground — TimelineView-driven drifting color blobs; deliberately pure SwiftUI, since a .metal library only reliably bundles in one of the two build systems these sources compile under). macOS presents the settings/add-host screens as sheets (no fullScreenCover there); PUNKTFUNK_FORCE_GAMEPAD_UI=1 forces the mode without a physical pad (dev/screenshots).
Tests (swift test) — Annex-B units, a real-codec VideoToolbox round trip, DualSense trigger-effect and gamepad-wire conversions, loopback integration against real local hosts, and the remote first-light test.

Notes for contributors

Xcode project (Punktfunk.xcodeproj) wraps the same sources as the swift run shell (a synchronized folder — no duplication). The macOS target is App-Sandboxed (needs network.server — the raw-UDP plane and quinn both bind()); iOS/tvOS use the shared entitlements file (keep app-sandbox out of it). Verify with codesign -d --entitlements :- <built .app>.
Decode flow: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, and recovery keyframes re-send them — refresh the format description on every IDR; there is no out-of-band extradata, ever.
ABI threading: one video pump thread per connection, one optional audio drain thread, and one optional feedback drain thread (rumble + HID-output). send() is enqueue-only and safe alongside all of them. The wrapper's per-plane locks make close() safe from anywhere.
DualSense motion scale (GamepadWire) is derived from hid-playstation's math, not yet live-verified — if gyro/accel feel wrong in a game, correct sign/scale there and evtest the host's virtual pad.
App Store screenshots are automated — tools/screenshots.sh all renders the real UI at the required pixel sizes via a DEBUG-only shot mode; the apple CI workflow captures the iOS sizes on every main push. See the script header for details. The script's SCENES array is the listing set, in listing order; override it (SCENES="06-gamepad-home 10-edithost" tools/screenshots.sh ios) to capture any of the other scenes in ShotScenes.all. Mock data — hosts, adverts, profiles — is seeded in ShotMock so a capture is byte-for-byte deterministic and never browses the real LAN (a stranger's hostname reached the live listing that way once).
Deeper design notes live in the internal planning repo (punktfunk-planning: apple-stage2-presenter.md).

Documentation — quick start, pairing, troubleshooting
Project README — the host, the other clients, and how it all fits together

8.6 KiB Raw Blame History