enricobuehler
0b550dad84
WP8 and WP9 of `punktfunk-planning/design/android-console-ui-visual-refresh.md`, in part. **WP9.3 — shared parity vectors.** The console's background palettes, its settings section names and its screen-transition motion each existed in three hand-written copies (`pf-console-ui`, this client, the Apple client) held together by a comment asking the next person to keep them in step. `clients/shared/console-vectors.json` now holds them, read the way `deeplink-vectors.json` already is: `include_str!` in Rust, a relative path in Kotlin, `#filePath` in Swift — never a copy, because a copy is a fourth contract free to go stale. It carries the DERIVED tables too, the 16-cell mesh and the 4 blob colours per palette, which is the half that reaches the screen and the half Android never checked: `GamepadPaletteTest` only ever measured the `stops` they are computed from. Two drifts it immediately caught, both now closed: * **The easing was the wrong curve.** `ConsoleMotion.EaseOutCubic` shipped as `cubic-bezier(0.215, 0.61, 0.355, 1)` while claiming to be the desktop's `ease_out_cubic`. It is not: that is the Penner/Ceaser table's curve, ~0.80 at the midpoint where `1 − (1−t)³` is 0.875 — visibly slacker over a 260 ms transition. Compose's `Easing` is a plain function, so it now evaluates the real thing analytically rather than approximating it at all. (Apple approximates with a different bezier only because SwiftUI's `timingCurve` cannot take a closure; the vectors sample the curve with a tolerance so all three can meet it.) * **The desktop has a seventh tab.** Input — touch mode, mouse, invert-scroll, shortcuts — with nothing to set on a phone or a TV. `settings.rs` claims in prose that a setting is found under the same word on every client; that was true modulo an omission nobody could see. The vectors model it with `desktop_only` rather than picking a side, so neither client has to be wrong. Rust reads it from three tests placed in the files that own the constants, so nothing had to be made `pub` to be checkable. Verified green under Linux (the crate is `cfg(linux|windows)` throughout — `cargo test` on a Mac compiles nothing and passes vacuously): 77 passed, 0 failed. Android's side gates in CI as a FILTERED task; a plain `:app:testDebugUnitTest` would drag the ~20 Roborazzi screenshot scenes into every push, and those are a release-artifact job. **WP8.1 — a pad route to the stats overlay.** The tier could only be cycled by a three-finger tap, which does not exist on a TV, on a gamepad-only session, or under touch passthrough — while the settings row promised a live cycle. `Select + X` now cycles it, byte-identical to the Apple client's `GamepadWire.back | GamepadWire.x`, implemented as the mic chord's twin in `GamepadRouter` and edge-triggered on the button that completes the mask. The buttons still reach the game, as both existing chords do. `GamepadChordTest` pins eight cases the kit had no cover for at all, including that the three chords intersect only on Select and that none is reachable through another. **WP8.5 — a start-of-stream banner.** The desktop's `skia_overlay` banner, ported with its timing (opaque 5.4 s, then a 0.6 s fade) and its rule of naming only shortcuts that exist: pad chords when a pad is present, the touch gesture when there is a touchscreen and the mode can use it. Nothing `Ctrl+Alt+Shift` is advertised, because Android has none of it. It yields to the motion-unreachable notice rather than stacking with it — that one reports something broken about *this* session. **WP8.6 — the home card says which profile it connects with.** `HomeTile` carried a `pinnedProfileId` the card never drew, so a pinned host+profile card was distinguishable from the host's own only by a subtitle that had been quietly repurposed to hold the profile name. Both now show the address like every other card and wear a tinted profile chip — the touch grid's own convention and the Apple client's, inked from the console palette. Unsaved tiles (discovered, Add Host) take a dashed edge, which is what the other two surfaces already use to say "not yours yet". ⚠ Not a detail panel: the Apple client REMOVED its own and moved the status onto the card, which is where the lock and the online pip already were here. **WP8.7 — accessibility, in part.** The console screens carried three `contentDescription`s and no `semantics`, `Role` or `stateDescription` at all. A settings row now announces once, merged — label, value, and the description that lives in the floating band far from it — with `Role.Switch` and a real toggle state, because a toggle row's on/off string was drawn by nothing at all: the switch replaces the value text, and the switch was two undescribed `Box`es. Decoration is silenced rather than labelled (the chevrons were read aloud as punctuation on every focused row). The hint bar's glyphs, the tab strip and the home tiles are done; `GamepadAddHostScreen` and `LibraryScreen` are not yet.
punktfunk — Apple client (macOS · iOS · iPadOS · tvOS)
The native Apple app for streaming a punktfunk host to your Mac, iPhone, iPad, or Apple TV. A SwiftUI app that finds hosts on your network, pairs with a PIN, and streams at your display's own resolution and refresh rate — with VideoToolbox hardware decode and full controller support.
All the networking and protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM,
Opus audio, cert pinning — lives in the shared Rust punktfunk-core (statically linked as
PunktfunkCore.xcframework). This package is the Swift shell: decode, present, input, and UI.
Features
- Hardware decode — VideoToolbox H.264/HEVC (plus AV1 on devices with an AV1 hardware
decoder — M3-class Macs, A17 Pro-class iPhones), with a low-latency stage-2 presenter
(
VTDecompressionSession→CAMetalLayer, presented off aCADisplayLink, ~11 ms p50) as the default and anAVSampleBufferDisplayLayerfallback. - HDR & 4:4:4 — PQ passthrough with a correct reference-white anchor, mid-session SDR↔HDR reconfiguration, and hardware-probed 4:4:4 support.
- Your display's native mode — the host builds a virtual output at exactly your WxH@Hz; mid-stream resize renegotiates without reconnecting.
- Audio both ways — Opus playback (CoreAudio, no bundled libopus) with a jitter ring, plus mic uplink; speaker/mic selectable in Settings.
- Full controller support — one selected controller forwarded as pad 0, including DualSense feedback (rumble → CoreHaptics, lightbar, player LEDs, adaptive triggers) and touchpad/motion. The virtual pad type auto-resolves from your physical controller.
- Mouse & keyboard —
GCMouse/GCKeyboardcapture with click-to-capture and a ⌃⌥⇧Q release (the cross-client Ctrl+Alt+Shift+Q; ⌘⎋ still works as the macOS/iPad toggle), plus iPad pointer lock and touch input. - Find hosts automatically — mDNS discovery (
NWBrowserover_punktfunk._udp); first connect does a one-time SPAKE2 PIN pairing (or TOFU on trusted LANs), then reconnects on a pinned, Keychain-stored identity. - Tune the stream — a fps / Mb·s / latency HUD (skew-corrected across machines), a bitrate control, a per-host network speed test with a recommended bitrate, and a host-compositor picker.
Runs from one shared codebase across macOS, iOS, iPadOS, and tvOS.
Get it
Install from the App Store / TestFlight, or build from source below. Per-device install steps and the pairing walkthrough: docs.punktfunk.unom.io/docs/install-client.
Build / run / test (on a Mac)
Requires Xcode 26.5 / Swift 6.3. First build the Rust core into an xcframework, then build the app:
rustup target add aarch64-apple-darwin x86_64-apple-darwin
bash scripts/build-xcframework.sh # → clients/apple/PunktfunkCore.xcframework
# BUILD_IOS=1 also builds the iOS slices (add the ios rustup targets)
# BUILD_TVOS=1 also builds tvOS (tier-3 targets, built from source — see below)
cd clients/apple
open Punktfunk.xcodeproj # the real app: ⌘R builds + runs Punktfunk.app
swift run PunktfunkClient # or the unbundled dev shell (CLI)
swift build && swift test # unit + loopback/remote tests (self-skip w/o a host)
tvOS slices are tier-3 Rust targets, built from source:
rustup toolchain install nightly && rustup component add rust-src --toolchain nightly.
Test against a host
# full loopback proof — builds punktfunk-host (synthetic source, runs on macOS) and streams
# byte-verified frames into the Swift client, incl. the PIN pairing ceremony:
bash test-loopback.sh
# against a real Linux host on the LAN (see the repo README "Running on this box"):
PUNKTFUNK_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests # headless
PUNKTFUNK_AUTOCONNECT=<box-ip> PUNKTFUNK_MODE=1280x720x60 swift run PunktfunkClient # on glass
Project layout
PunktfunkKit(library) — the reusable pieces:PunktfunkConnection— the wrapper over the C ABI (thread-safeclose(), per-plane locks, pinning + TOFU).AnnexB/StreamView/VideoDecoder/MetalVideoPresenter— format handling, the stage-1 (AVSampleBufferDisplayLayer) and stage-2 (VTDecompressionSession→CAMetalLayer) presenters.InputCapture—GCMouse/GCKeyboard→ host VK/mouse, with fractional-delta accumulation.GamepadManager/GamepadCapture/GamepadFeedback/DualSenseTriggerEffect— controller discovery + selection, capture (buttons/axes/touchpad/motion), and host-feedback rendering.HostDiscovery—NWBrowserover_punktfunk._udp.
PunktfunkClient(the app) — hosts grid with an On this network section, add-host sheet, the two trust flows (TOFU prompt + SPAKE2PairSheet), the stream view with the HUD, a tabbed Settings pane (General / Display / Audio / Controllers / Advanced), and the network speed test. A Scene-level Stream menu carries the cross-client shortcut set: Release Mouse (⌃⌥⇧Q), Disconnect (⌃⌥⇧D) and the HUD toggle (⌃⌥⇧S) — the same Ctrl+Alt+Shift combos the Windows and Linux clients reserve, also shown on a 6-second banner at stream start. On iOS/iPadOS and macOS a connected controller swaps the whole home for the gamepad UI (Home/Gamepad*,Settings/GamepadSettingsView): a console-style host carousel (A connect · Y library · X settings), a controller-navigable settings screen, an add-host flow with an on-screen controller keyboard (no touch required anywhere), and the coverflow library browser — all driven by the sharedGamepadMenuInputpoller +GamepadCarousel/GamepadMenuListfocus machinery, with dual-channel haptics (device Taptic + controllerMenuHaptics), over an animated "aurora" backdrop (GamepadScreenBackground— TimelineView-driven drifting color blobs; deliberately pure SwiftUI, since a .metal library only reliably bundles in one of the two build systems these sources compile under). macOS presents the settings/add-host screens as sheets (nofullScreenCoverthere);PUNKTFUNK_FORCE_GAMEPAD_UI=1forces the mode without a physical pad (dev/screenshots).- Tests (
swift test) — Annex-B units, a real-codec VideoToolbox round trip, DualSense trigger-effect and gamepad-wire conversions, loopback integration against real local hosts, and the remote first-light test.
Notes for contributors
- Xcode project (
Punktfunk.xcodeproj) wraps the same sources as theswift runshell (a synchronized folder — no duplication). The macOS target is App-Sandboxed (needsnetwork.server— the raw-UDP plane and quinn bothbind()); iOS/tvOS use the shared entitlements file (keepapp-sandboxout of it). Verify withcodesign -d --entitlements :- <built .app>. - Decode flow: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, and recovery keyframes re-send them — refresh the format description on every IDR; there is no out-of-band extradata, ever.
- ABI threading: one video pump thread per connection, one optional audio drain thread, and one
optional feedback drain thread (rumble + HID-output).
send()is enqueue-only and safe alongside all of them. The wrapper's per-plane locks makeclose()safe from anywhere. - DualSense motion scale (
GamepadWire) is derived from hid-playstation's math, not yet live-verified — if gyro/accel feel wrong in a game, correct sign/scale there andevtestthe host's virtual pad. - App Store screenshots are automated —
tools/screenshots.sh allrenders the real UI at the required pixel sizes via a DEBUG-only shot mode; theappleCI workflow captures the iOS sizes on every main push. See the script header for details. The script'sSCENESarray is the listing set, in listing order; override it (SCENES="06-gamepad-home 10-edithost" tools/screenshots.sh ios) to capture any of the other scenes inShotScenes.all. Mock data — hosts, adverts, profiles — is seeded inShotMockso a capture is byte-for-byte deterministic and never browses the real LAN (a stranger's hostname reached the live listing that way once). - Deeper design notes live in the internal planning repo (punktfunk-planning:
apple-stage2-presenter.md).
Related
- Documentation — quick start, pairing, troubleshooting
- Project README — the host, the other clients, and how it all fits together