From d6596ff81b9e7aaeaff267dfbb8951e15ad86506 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Wed, 1 Jul 2026 19:31:06 +0000 Subject: [PATCH] docs: rework client/crate READMEs, add missing ones Rework the client READMEs to be accurate and inviting to first-time visitors, and fill in the gaps where crates and tools had none. - Rewrite clients/{apple,android,decky} READMEs (features-first, trim dense internal narrative; drop the stale "one session at a time" / "renegotiation not implemented" section from the Apple README). - Add READMEs for clients/{linux,windows,probe}, which had none. - Add crate READMEs for punktfunk-host, punktfunk-core, pf-driver-proto. - Add brief READMEs for tools/{loss-harness,latency-probe}. - Fix packaging/README duplicate "Option B" heading (bootc -> Option C). - Fix docs-site/README stale docs/ -> design/ reference. - De-stale packaging/windows/drivers/pf-dualsense README (drop "M0 spike" / external-checkout framing; reflect in-tree workspace + shipped + installer-bundled + multi-pad), keeping the driver-authoring lore. Co-Authored-By: Claude Opus 4.8 --- clients/android/README.md | 113 +++-- clients/apple/README.md | 433 ++++-------------- clients/decky/README.md | 215 +++------ clients/linux/README.md | 77 ++++ clients/probe/README.md | 50 ++ clients/windows/README.md | 71 +++ crates/pf-driver-proto/README.md | 16 + crates/punktfunk-core/README.md | 61 +++ crates/punktfunk-host/README.md | 90 ++++ docs-site/README.md | 6 +- packaging/README.md | 2 +- .../windows/drivers/pf-dualsense/README.md | 44 +- tools/latency-probe/README.md | 15 + tools/loss-harness/README.md | 16 + 14 files changed, 624 insertions(+), 585 deletions(-) create mode 100644 clients/linux/README.md create mode 100644 clients/probe/README.md create mode 100644 clients/windows/README.md create mode 100644 crates/pf-driver-proto/README.md create mode 100644 crates/punktfunk-core/README.md create mode 100644 crates/punktfunk-host/README.md create mode 100644 tools/latency-probe/README.md create mode 100644 tools/loss-harness/README.md diff --git a/clients/android/README.md b/clients/android/README.md index 8f54da5..d1495d7 100644 --- a/clients/android/README.md +++ b/clients/android/README.md @@ -1,83 +1,78 @@ -# punktfunk Android client +# punktfunk — Android client (phone & TV) -Native Android client for **punktfunk/1**, targeting **phone + TV** (Compose, D-pad + touch). +The native **Android** app for streaming a punktfunk host to your phone, tablet, or Android TV. A +Compose app that finds hosts on your network, pairs with a PIN, and streams at the display's own +resolution — with hardware HEVC decode, HDR10, and controller support, built for both touch and the +couch (D-pad / gamepad focus navigation). -## Architecture — Rust-heavy (like the Linux client, not thin-native like Apple) +## Features -Kotlin cannot `import` the cbindgen C header the way Swift can, so a native bridge is unavoidable. -We write it in **Rust** and link `punktfunk-core` directly — so the Android client reuses the Linux +- **Hardware decode** — NDK `AMediaCodec` HEVC → `SurfaceView`, including **HDR10** (Main10 / + BT.2020 PQ), with low-latency tuning and a live stats HUD. +- **Audio both ways** — Opus + Oboe playback with a jitter ring, plus mic uplink to the host. +- **Controller support** — buttons + axes with rumble and HID feedback (lightbar / adaptive + triggers); D-pad / gamepad focus navigation for TV and phone. +- **Find hosts automatically** — native mDNS discovery; first connect does a one-time **SPAKE2 PIN + pairing** (or TOFU on trusted LANs), then reconnects on a Keystore-wrapped, pinned identity. +- **Compose UI** — Connect / Settings / Stream screens with Material You theming. + +Built for `arm64-v8a` + `x86_64`. + +## Get it + +Published to **Google Play (Internal Testing)** — join the beta via the +[Discord](https://discord.gg/kaPNvzMuGU). Per-device setup and pairing: +**[docs.punktfunk.unom.io/docs/install-client](https://docs.punktfunk.unom.io/docs/install-client)**. + +## How it's built — Rust-heavy + +Kotlin can't `import` the cbindgen C header the way Swift can, so a native bridge is unavoidable. We +write it in **Rust** and link `punktfunk-core` directly — so the Android client reuses the Linux client's orchestration (audio jitter ring, VK keymap inverse, latency/skew math, capture state machine, trust logic) instead of re-porting it into Kotlin. | Side | Owns | |------|------| -| **Rust** (`clients/android/native` → `libpunktfunk_android.so`) | the JNI seam, `NativeClient` (QUIC control + UDP data plane), AnnexB→`AMediaCodec` decode, Opus+Oboe audio, VK keymap, latency math, trust/pairing, **mDNS discovery** (`mdns-sd`, the same browse the Linux/Windows clients use) | -| **Kotlin** (`clients/android`) | Compose UI (host grid / settings / stream), `SurfaceView` lifecycle, input capture, the Wi-Fi `MulticastLock` + permission UX, Keystore identity, permissions | +| **Rust** (`native/` → `libpunktfunk_android.so`) | the JNI seam, `NativeClient` (QUIC control + UDP data plane), AnnexB → `AMediaCodec` decode (incl. HDR10), Opus + Oboe audio + mic, controller feedback, latency math, trust/pairing, `mdns-sd` discovery | +| **Kotlin** (`app/`, `kit/`) | Compose UI, `SurfaceView` lifecycle, input capture, the Wi-Fi `MulticastLock` + permission UX, Keystore identity | The single seam is `io.unom.punktfunk.kit.NativeBridge` ⇄ `Java_io_unom_punktfunk_kit_NativeBridge_*`. -## Layout - ``` -clients/android/native/ Rust cdylib (workspace member) — links punktfunk-core directly - src/lib.rs JNI seam (connect/pair, input, plane getters, abi/core version) - src/session.rs session lifecycle + plane pumps - src/decode.rs AnnexB → AMediaCodec HEVC hardware decode → SurfaceView (incl. HDR10) - src/audio.rs · src/mic.rs Opus + Oboe playback / mic uplink (jitter ring) - src/feedback.rs rumble + HID output (lightbar / adaptive triggers) - src/stats.rs live video stats - -clients/android/ Gradle project (this dir) - settings.gradle.kts · build.gradle.kts · gradle.properties · gradlew - app/ :app — Compose UI: Connect / Settings / Stream screens (phone + TV) - kit/ :kit — NativeBridge · discovery (native mdns-sd, polled) · Gamepad · Keymap · - security (Keystore identity + known-host store) · cargo-ndk build +native/ Rust cdylib (workspace member) — links punktfunk-core directly + src/lib.rs JNI seam (connect/pair, input, plane getters, versions) + src/session.rs session lifecycle + plane pumps + src/decode.rs AnnexB → AMediaCodec HEVC hardware decode → SurfaceView (incl. HDR10) + src/audio.rs · src/mic.rs Opus + Oboe playback / mic uplink + src/feedback.rs · src/stats.rs rumble + HID feedback; live video stats +app/ :app — Compose UI: Connect / Settings / Stream (phone + TV) +kit/ :kit — NativeBridge · native mDNS discovery · Gamepad · Keymap · Keystore identity ``` -## Prerequisites - -- Android SDK + **NDK r30** (`30.0.14904198`), `platforms;android-37.0`, `build-tools;37.0.0`, - **`cmake;3.22.1`** (`sdkmanager "cmake;3.22.1"` — the `cmake` crate builds libopus with it) -- **JDK 21** for Gradle/AGP (AGP 9.2 runs on JDK 17–21, *not* a newer default JDK like 25) -- Rust + `rustup target add aarch64-linux-android x86_64-linux-android` + `cargo install cargo-ndk` - -Toolchain pinned: AGP 9.2.0 · Gradle 9.4.1 · Kotlin 2.3.21 · Compose BOM 2026.05.01 · -compileSdk 37 · targetSdk 36 · minSdk 31 · ABIs arm64-v8a + x86_64. - ## Build & run -**Android Studio:** open `clients/android` — it uses its bundled JBR 21 automatically. The -`cargoNdk*` task builds the `.so` as part of the normal build. +**Prerequisites:** Android SDK + **NDK r30** (`30.0.14904198`), `platforms;android-37.0`, +`build-tools;37.0.0`, **`cmake;3.22.1`** (builds libopus); **JDK 21** (AGP 9.2 runs on JDK 17–21, not +a newer default); Rust with `rustup target add aarch64-linux-android x86_64-linux-android` and +`cargo install cargo-ndk`. Toolchain is pinned (AGP 9.2 · Gradle 9.4.1 · Kotlin 2.3.21 · Compose BOM +2026.05.01 · compileSdk 37 · minSdk 31). -**CLI** (point Gradle at a JDK 21 if your machine default is newer, e.g. JDK 25): +**Android Studio:** open `clients/android` — it uses its bundled JBR 21, and the `cargoNdk*` task +builds the `.so` as part of the normal build. + +**CLI** (point Gradle at JDK 21 if your machine default is newer): ```sh -# Adoptium/Temurin 21 (installed by the Android Studio setup, or `brew install temurin@21`): -export JAVA_HOME="$(/usr/libexec/java_home -v 21)" +export JAVA_HOME="$(/usr/libexec/java_home -v 21)" # or your Temurin 21 path cd clients/android -./gradlew :app:assembleDebug # cargo-ndk cross-compiles libpunktfunk_android.so first -./gradlew :app:installDebug # onto a running emulator/device - -# Emulators (created during env setup): emulator -avd pf_phone | emulator -avd pf_tv +./gradlew :app:assembleDebug # cargo-ndk cross-compiles libpunktfunk_android.so first +./gradlew :app:installDebug # onto a running emulator/device +# emulators from env setup: emulator -avd pf_phone | emulator -avd pf_tv ``` -The debug APK lands in `app/build/outputs/apk/debug/`. Launch it, pick a host from the list, pair, -and stream. +The debug APK lands in `app/build/outputs/apk/debug/`. Launch it, pick a host, pair, and stream. -## Status +## Related -A working native client (phone + Android TV), at parity with the Linux and Apple apps for the core -streaming experience: - -- **Video** — `AMediaCodec` hardware HEVC decode → `SurfaceView`, including **HDR10** (Main10 / - BT.2020 PQ), with low-latency decode tuning and a live stats HUD. -- **Audio** — Opus + Oboe playback with a jitter ring, plus mic uplink to the host. -- **Input** — game controllers (buttons + axes) with rumble and HID feedback; D-pad / - game-controller focus navigation for the couch (TV + phone). -- **Discovery & trust** — native `mdns-sd` mDNS host list (polled over JNI; the same browse the - Linux/Windows clients use, not `NsdManager`), SPAKE2 PIN pairing and TOFU, with a - Keystore-wrapped client identity and a known-host store. -- **UI** — Compose host list / settings / stream screens, Material You theming. -- **Shipping** — built for `arm64-v8a` + `x86_64`; published to Google Play (Internal Testing). - -`crates/punktfunk-core` uses the `ring` `rcgen` backend so the client `.so` is aws-lc-free. +- **[Documentation](https://docs.punktfunk.unom.io)** — quick start, pairing, troubleshooting +- **[Project README](../../README.md)** — the host, the other clients, and how it all fits together diff --git a/clients/apple/README.md b/clients/apple/README.md index f8e6b30..f7371dd 100644 --- a/clients/apple/README.md +++ b/clients/apple/README.md @@ -1,364 +1,115 @@ -# punktfunk Apple client (SwiftUI) +# punktfunk — Apple client (macOS · iOS · iPadOS · tvOS) -The native macOS/iOS client for **`punktfunk/1`** (the post-GameStream protocol). All -networking/protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM, -input datagrams, Opus audio, cert pinning — lives in the shared Rust core (statically -linked as `PunktfunkCore.xcframework`); this package is the Swift shell: decode -(VideoToolbox), present (SwiftUI), input capture. +The native **Apple** app for streaming a punktfunk host to your Mac, iPhone, iPad, or Apple TV. A +SwiftUI app that finds hosts on your network, pairs with a PIN, and streams at your display's own +resolution and refresh rate — with VideoToolbox hardware decode and full controller support. -## Status — working client (macOS, with iOS / tvOS in the shared build) +All the networking and protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM, +Opus audio, cert pinning — lives in the shared Rust **`punktfunk-core`** (statically linked as +`PunktfunkCore.xcframework`). This package is the Swift shell: decode, present, input, and UI. -A full streaming client: VideoToolbox HEVC decode, controllers incl. DualSense feedback, host -discovery, PIN pairing, and a network speed test. The lower-latency **stage-2 presenter** -(`VTDecompressionSession` → `CAMetalLayer`) is built and opt-in (Settings → Presenter); see below. +## Features -First light was achieved 2026-06-10 — validated live, Mac ↔ a Linux host over the LAN: gamescope -virtual output → NVENC HEVC → -`punktfunk/1` (GF(2¹⁶) FEC + AES-GCM over UDP, QUIC control) → VideoToolbox → -`AVSampleBufferDisplayLayer` on glass at 1280×720@60, with mouse/keyboard flowing back as -QUIC datagrams into the host's gamescope EIS injector (thousands of events injected during -the session). Headless variant of the same proof: `RemoteFirstLightTests` decoded 60/60 -received AUs spanning 983 ms of host capture clock. +- **Hardware decode** — VideoToolbox HEVC, with a low-latency **stage-2 presenter** + (`VTDecompressionSession` → `CAMetalLayer`, presented off a `CADisplayLink`, ~11 ms p50) as the + default and an `AVSampleBufferDisplayLayer` fallback. +- **HDR & 4:4:4** — PQ passthrough with a correct reference-white anchor, mid-session SDR↔HDR + reconfiguration, and hardware-probed 4:4:4 support. +- **Your display's native mode** — the host builds a virtual output at exactly your WxH@Hz; + mid-stream resize renegotiates without reconnecting. +- **Audio both ways** — Opus playback (CoreAudio, no bundled libopus) with a jitter ring, plus mic + uplink; speaker/mic selectable in Settings. +- **Full controller support** — one selected controller forwarded as pad 0, including **DualSense** + feedback (rumble → CoreHaptics, lightbar, player LEDs, adaptive triggers) and touchpad/motion. The + virtual pad type auto-resolves from your physical controller. +- **Mouse & keyboard** — `GCMouse`/`GCKeyboard` capture with click-to-capture and a ⌘⎋ release, plus + iPad pointer lock and touch input. +- **Find hosts automatically** — mDNS discovery (`NWBrowser` over `_punktfunk._udp`); first connect + does a one-time **SPAKE2 PIN pairing** (or TOFU on trusted LANs), then reconnects on a pinned, + Keychain-stored identity. +- **Tune the stream** — a fps / Mb·s / **latency** HUD (skew-corrected across machines), a bitrate + control, a per-host **network speed test** with a recommended bitrate, and a host-compositor picker. -The connector underneath (`punktfunk_core::client::NativeClient` over the C ABI) carries the -full session: video AUs, **Opus audio** (`nextAudio()`), **rumble** (`nextRumble()`), -**DualSense feedback** (`nextHidOutput()` — lightbar, player LEDs, adaptive-trigger -effects), input incl. gamepads + DualSense touchpad/motion (`sendTouchpad`/`sendMotion`), -and **cert pinning + TOFU** (`pinSHA256:`/`hostFingerprint`) — see -`punktfunk1.rs::tests::c_abi_connection_roundtrip` (three sequential sessions: TOFU, pinned -reconnect, wrong-pin rejection). The host (`punktfunk-host punktfunk1-host`) is a persistent listener: -reconnect at will during development. +Runs from one shared codebase across **macOS, iOS, iPadOS, and tvOS**. -What's here, all compiled and tested on macOS (Xcode 26.5 / Swift 6.3): +## Get it -- **`PunktfunkKit`** (library) - - `PunktfunkConnection.swift` — wrapper over the C ABI. AUs/audio are copied into `Data` - (the C pointer is only valid until the next call of the same kind). `close()` is safe - from any thread: per-plane locks enforce the C contract ("never close with a - `next_au`/`next_audio` in flight") instead of leaving it to callers. Pinning + TOFU - via `pinSHA256:`/`hostFingerprint`. - - `AnnexB.swift` — in-band VPS/SPS/PPS → `CMVideoFormatDescription`; Annex-B → AVCC - `CMSampleBuffer` with `DisplayImmediately` set. - - `StreamView.swift` — SwiftUI `NSViewRepresentable` over `AVSampleBufferDisplayLayer` - (stage-1 presenter: the layer hardware-decodes compressed HEVC itself). One pump - thread per view, token-cancelled so reconnects can't double-pump. - - `InputCapture.swift` — `GCMouse` raw deltas + `GCKeyboard` HID→VK mapping (the host's - `vk_to_evdev` consumes Windows VKs), with fractional-delta accumulation so sub-pixel - motion isn't truncated away. Buttons use GameStream ids (1=left … 5=X2). Scroll is - WHEEL_DELTA(120)-scaled: macOS via the stream view's `scrollWheel` override, iPad via - GCMouse's scroll dpad when pointer-locked and a scroll-only `UIPanGestureRecognizer` - otherwise (trackpad gestures never reach GC's scroll dpad). - - `GamepadManager.swift` — app-lifetime controller discovery + selection (`.shared`): - watches `GCController` connect/disconnect, fingerprints each pad for the Settings UI - (name, capabilities, battery), and selects the ONE controller forwarded to the host - (user pin via "Use controller", else most recently connected extended gamepad). - - `GamepadCapture.swift` — the active controller → wire: snapshot-diff over - `GCExtendedGamepad` into incremental `gamepadButton`/`gamepadAxis` events (pad 0), - plus DualSense touchpad contacts and ~250 Hz motion samples on the rich-input plane - (the GC→DualSense unit conversions live in `GamepadWire`, one place). Held state is - released on the wire on controller switch / app deactivation / stop. - - `GamepadFeedback.swift` + `DualSenseTriggerEffect.swift` — host feedback → the real - controller: one drain thread for `nextRumble()` (→ `CHHapticEngine` per handle - locality) and `nextHidOutput()` (lightbar → `GCDeviceLight`, player LEDs → - `playerIndex`, adaptive-trigger effect blocks → a total, table-driven parser → - `GCDualSenseAdaptiveTrigger`, exact for the 10-zone positional modes). - - `HostDiscovery.swift` — LAN auto-discovery: an `NWBrowser` over `_punktfunk._udp` - (the host's `crate::discovery` mDNS advert), resolving each service to an IP:port via a - throwaway `NWConnection` and parsing the TXT (`fp` advisory cert fingerprint, `pair`, - stable `id`). iOS/tvOS need `NSBonjourServices` (`Config/Info.plist`) or the system - blocks the browse. -- **`PunktfunkClient`** (the app): hosts grid (saved in UserDefaults) with an **On this - network** section listing mDNS-discovered hosts (tap to save + connect, or pair if the - host requires it), "+" toolbar sheet to add hosts manually, stream mode in Settings (⌘,), - two trust flows — the - trust-on-first-use fingerprint prompt over the live-but-blurred stream, and SPAKE2 PIN - pairing (`PairSheet`, from a host card's context menu or the trust prompt; - `ClientIdentityStore` keeps the client identity in the Keychain and presents it on - every connect) — then pinned reconnects, fps/Mb-s HUD + a **capture→client-receipt latency** - line (`LatencyMeter`, p50/p95): the AU `pts_ns` (host capture clock) to the instant the client - received it, **skew-corrected** across machines via `PunktfunkConnection.clockOffsetNs` (the - connect-time wall-clock handshake, `punktfunk_connection_clock_offset_ns`). It excludes the - layer's decode+present (stage-1 `AVSampleBufferDisplayLayer` has no per-frame present callback); - the opt-in **stage-2 presenter** (Settings → Presenter) adds a **capture→present** - (glass-to-glass) line via explicit decode + a Metal/display-link present. Settings also picks the HOST - compositor (KWin/wlroots/Mutter/gamescope, default automatic — the host honors it - only if that backend is available there) and has a **Controllers** section: every - detected controller (capability glyphs, battery, "In use" badge), which one to forward - ("Use controller", default automatic), and the virtual pad type the host creates - ("Controller type": Automatic / Xbox 360 / DualSense — Automatic matches the physical - pad; resolved at connect time, the host pad is fixed per session). Gamepad capture + - feedback run with streaming (`SessionModel` owns them, same trust gate as audio). - Settings also sets the **Bitrate** (Automatic toggle = host default; manual is a - log-scale slider, 2 Mbps – 3 Gbps, snapped to two significant figures — above 1 Gbps - an inline warning says to run a speed test first; tvOS uses a preset picker instead, - Slider doesn't exist there; negotiated via the Hello on every connect), and a host - card's context menu offers **"Test Network Speed…"** (`SpeedTestSheet`): connects, has - the host burst probe filler over the real data plane (up to the host's 3 Gbps probe - ceiling for 2 s, roadmap §9), - shows measured goodput · loss · a recommended bitrate (≈70% of measured), and applies - it in one tap. The streaming **statistics overlay** can be turned off and moved to any - corner (Settings → Display → Statistics, `DefaultsKey.hudEnabled`/`hudPlacement`), and - toggled live with **⌘⇧S** — a Scene-level **"Stream" menu** (`StreamCommands`) that also - carries **Disconnect ⌘D**, so disconnect survives the HUD being hidden (on iOS a small - exit chip appears instead; on tvOS the Siri-Remote Menu button still disconnects). The - macOS Settings window is a **tabbed preferences pane** (General / Display / Audio / - Controllers / Advanced) — the sections are shared with the iOS single-Form layout and the - tvOS pushed-picker layout, defined once each. -- **Tests** (`swift test`): byte-level Annex-B units; a real-codec round trip - (VTCompressionSession-encoded HEVC rebuilt as the host's wire shape → `AnnexB` → - VTDecompressionSession → pixels); table-driven DualSense trigger-effect parsing - (`DualSenseTriggerEffectTests`) and the gamepad wire conversions - (`GamepadWireTests`); loopback integration against real local hosts - (`test-loopback.sh` — stream round trip incl. gamepad/touchpad/motion sends, a - host-scripted feedback burst asserted on the rumble + HID-output planes - (`PUNKTFUNK_TEST_FEEDBACK=1`), the bitrate-negotiation echo and a real 20 Mbps - bandwidth probe, plus the PIN pairing ceremony and the `--require-pairing` gate - against a second, armed host); the remote first-light test above. +Install from the App Store / TestFlight, or build from source below. Per-device install steps and the +pairing walkthrough: +**[docs.punktfunk.unom.io/docs/install-client](https://docs.punktfunk.unom.io/docs/install-client)**. ## Build / run / test (on a Mac) +Requires Xcode 26.5 / Swift 6.3. First build the Rust core into an xcframework, then build the app: + ```sh rustup target add aarch64-apple-darwin x86_64-apple-darwin -bash scripts/build-xcframework.sh # → clients/apple/PunktfunkCore.xcframework -# + BUILD_IOS=1 for the iOS slices (rustup target add aarch64-apple-ios{,-sim} x86_64-apple-ios) -# + BUILD_TVOS=1 for tvOS — TIER-3 Rust targets, built from source: -# rustup toolchain install nightly && rustup component add rust-src --toolchain nightly +bash scripts/build-xcframework.sh # → clients/apple/PunktfunkCore.xcframework +# BUILD_IOS=1 also builds the iOS slices (add the ios rustup targets) +# BUILD_TVOS=1 also builds tvOS (tier-3 targets, built from source — see below) + cd clients/apple -swift build && swift test # loopback/remote tests self-skip without a host -swift run PunktfunkClient # the unbundled dev shell (CLI) -open Punktfunk.xcodeproj # the real app: ⌘R builds + runs Punktfunk.app +open Punktfunk.xcodeproj # the real app: ⌘R builds + runs Punktfunk.app +swift run PunktfunkClient # or the unbundled dev shell (CLI) +swift build && swift test # unit + loopback/remote tests (self-skip w/o a host) +``` -bash test-loopback.sh # full loopback proof: builds punktfunk-host - # (synthetic source — runs on macOS), streams - # byte-verified frames into the Swift client +tvOS slices are tier-3 Rust targets, built from source: +`rustup toolchain install nightly && rustup component add rust-src --toolchain nightly`. -# against the real host (Linux box, see CLAUDE.md "Running on this box") — punktfunk1-host is a -# persistent listener, reconnect at will: -# PUNKTFUNK_COMPOSITOR=gamescope PUNKTFUNK_GAMESCOPE_APP=vkcube PUNKTFUNK_ZEROCOPY=1 \ -# cargo run -rp punktfunk-host -- punktfunk1-host --source virtual --seconds 60 -PUNKTFUNK_REMOTE_HOST= swift test --filter RemoteFirstLightTests # headless -# (+ PUNKTFUNK_REMOTE_PORT / PUNKTFUNK_REMOTE_COMPOSITOR=gamescope|kwin|… / -# PUNKTFUNK_REMOTE_PIN= for the remote pairing test) +### Test against a host + +```sh +# full loopback proof — builds punktfunk-host (synthetic source, runs on macOS) and streams +# byte-verified frames into the Swift client, incl. the PIN pairing ceremony: +bash test-loopback.sh + +# against a real Linux host on the LAN (see the repo README "Running on this box"): +PUNKTFUNK_REMOTE_HOST= swift test --filter RemoteFirstLightTests # headless PUNKTFUNK_AUTOCONNECT= PUNKTFUNK_MODE=1280x720x60 swift run PunktfunkClient # on glass ``` -## Xcode project (`Punktfunk.xcodeproj`) +## Project layout -The app target **Punktfunk** wraps the same sources as the `swift run` shell -(`Sources/PunktfunkClient`, a synchronized folder — no duplication) plus `App/` (asset -catalog) and links `PunktfunkKit` from the local package. Generated Info.plist, ad-hoc -signing, bundle id `io.unom.punktfunk`. Notes: +- **`PunktfunkKit`** (library) — the reusable pieces: + - `PunktfunkConnection` — the wrapper over the C ABI (thread-safe `close()`, per-plane locks, + pinning + TOFU). + - `AnnexB` / `StreamView` / `VideoDecoder` / `MetalVideoPresenter` — format handling, the stage-1 + (`AVSampleBufferDisplayLayer`) and stage-2 (`VTDecompressionSession` → `CAMetalLayer`) presenters. + - `InputCapture` — `GCMouse`/`GCKeyboard` → host VK/mouse, with fractional-delta accumulation. + - `GamepadManager` / `GamepadCapture` / `GamepadFeedback` / `DualSenseTriggerEffect` — controller + discovery + selection, capture (buttons/axes/touchpad/motion), and host-feedback rendering. + - `HostDiscovery` — `NWBrowser` over `_punktfunk._udp`. +- **`PunktfunkClient`** (the app) — hosts grid with an *On this network* section, add-host sheet, + the two trust flows (TOFU prompt + SPAKE2 `PairSheet`), the stream view with the HUD, a + tabbed Settings pane (General / Display / Audio / Controllers / Advanced), and the network speed + test. A Scene-level **Stream** menu carries Disconnect (⌘D) and the HUD toggle (⌘⇧S). +- **Tests** (`swift test`) — Annex-B units, a real-codec VideoToolbox round trip, DualSense + trigger-effect and gamepad-wire conversions, loopback integration against real local hosts, and the + remote first-light test. -- **Entitlements (sandbox)**: the macOS target uses - `Config/Punktfunk-macOS.entitlements`; iOS/tvOS use the shared - `Config/Punktfunk.entitlements`. The macOS app is **App-Sandboxed** (mandatory for the Mac - App Store/TestFlight, and used for the Developer ID DMG too so the local build matches what - ships): `com.apple.security.app-sandbox`, `network.client` + **`network.server`** (the - sandbox gates `bind()`; quinn + the raw-UDP plane both bind, so receive breaks without it), - `device.audio-input` (mic), `device.bluetooth` + `device.usb` (GameController over BT/USB), - and the existing `keychain-access-groups`. `app-sandbox` is macOS-only — keep it OUT of the - shared iOS/tvOS file (it fails upload validation there). Verify a build is sandboxed with - `codesign -d --entitlements :- `. Heads-up: `device.usb` draws some App Review - scrutiny — justify it in the review notes ("reads input from USB game controllers"). -- **App icon**: `App/Assets.xcassets` ships an empty `AppIcon` slot. For an Icon Composer - `.icon`: add the file to the project (target Punktfunk), set it as the App Icon in the - target's General tab, and delete the placeholder `AppIcon.appiconset`. Heads-up: CLI - `actool` (Xcode 26.5) crashed compiling `punktfunk_Logo.icon` — if Xcode does the same, - suspect the icon bundle (it has a duplicate-named layer, "…Layer-3 2.svg"), not the - project. -- **Tests from Xcode**: the package tests run with `swift test`; to get them on ⌘U, add - `PunktfunkKitTests` once via Edit Scheme → Test → + (Xcode persists it into the shared - scheme — a hand-written package-test reference doesn't resolve headlessly). -- `xcodebuild -project Punktfunk.xcodeproj -scheme Punktfunk build` works headlessly; - same for `-scheme Punktfunk-iOS -destination 'generic/platform=iOS Simulator'` (run it - in a simulator via `xcrun simctl install/launch` — `SIMCTL_CHILD_PUNKTFUNK_AUTOCONNECT=…` - passes the dev autoconnect env through). +## Notes for contributors -## App Store screenshots +- **Xcode project** (`Punktfunk.xcodeproj`) wraps the same sources as the `swift run` shell (a + synchronized folder — no duplication). The macOS target is **App-Sandboxed** (needs + `network.server` — the raw-UDP plane and quinn both `bind()`); iOS/tvOS use the shared + entitlements file (keep `app-sandbox` **out** of it). Verify with + `codesign -d --entitlements :- `. +- **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, and recovery + keyframes re-send them — refresh the format description on every IDR; there is no out-of-band + extradata, ever. +- **ABI threading**: one video pump thread per connection, one optional audio drain thread, and one + optional feedback drain thread (rumble + HID-output). `send()` is enqueue-only and safe alongside + all of them. The wrapper's per-plane locks make `close()` safe from anywhere. +- **DualSense motion scale** (`GamepadWire`) is derived from hid-playstation's math, not yet + live-verified — if gyro/accel feel wrong in a game, correct sign/scale there and `evtest` the + host's virtual pad. +- **App Store screenshots** are automated — `tools/screenshots.sh all` renders the real UI at the + required pixel sizes via a DEBUG-only shot mode; the `apple` CI workflow captures the iOS sizes on + every main push. See the script header for details. +- Deeper design notes live in `docs-site/content/docs/apple-stage2-presenter.md`. -Automated, faithful screenshots of the real UI for App Store Connect — one set per platform at -exactly the accepted pixel sizes. Driver: **`tools/screenshots.sh`**. +## Related -```sh -tools/screenshots.sh all # macOS + (if full Xcode) iOS, iPadOS, tvOS → ./screenshots -tools/screenshots.sh macos # just macOS -OUT=~/Desktop/shots tools/screenshots.sh ios ipad tvos -PUNKTFUNK_SHOT_HERO=~/frame.png tools/screenshots.sh ios # real captured frame behind the hero -``` - -How it works: the app has a DEBUG-only **shot mode** (`Sources/PunktfunkClient/Screenshots/`). -Launched with `PUNKTFUNK_SHOT_SCENE=` it renders **one** mock-populated screen full-bleed -(`ScreenshotHostView`) instead of `ContentView`, then the OS screenshots the *real, fully-rendered* -window — `screencapture` on macOS, `xcrun simctl io booted screenshot` on the Simulators. The five -scenes (`ShotScenes.all`): `01-stream` (the stream hero — a synthetic frame + the glass HUD, since -`StreamView` needs a live connection), `02-hosts`, `03-pair`, `04-trust`, `05-settings`. Mock data -is in `ShotMock`; nothing touches a host. - -Output pixels are App Store Connect's required/largest sizes (Apple auto-derives the smaller ones): -`mac` 2880×1800 · `iphone-6.9` 1320×2868 (hero 2868×1320) · `ipad-13` 2064×2752 (hero 2752×2064) · -`appletv` 1920×1080. - -Why not `ImageRenderer` (the obvious offscreen route)? It can't rasterize this app's chrome — -`NavigationStack`, `Form`/`TabView`, and Liquid-Glass/`NSVisualEffect` materials all render black or -SwiftUI's "can't render" placeholder. Capturing the live window/Simulator avoids that entirely. - -Requirements / gotchas: -- **macOS**: only the Swift toolchain is needed, **plus a one-time Screen Recording grant** for - your terminal (System Settings → Privacy & Security → Screen Recording) — without it - `screencapture -l` fails with "could not create image from window". (A no-permission fallback, - `PUNKTFUNK_SHOT_SELFCAPTURE=`, uses `cacheDisplay` — but it omits material blur and can't - read `ScrollView` content, so it's for quick checks, not submission.) -- **iOS/iPadOS/tvOS**: needs **full Xcode** (xcodebuild + Simulators), not just Command Line Tools, - and the matching device Simulators installed (iPhone 16 Pro Max, iPad Pro 13", Apple TV). Run it - on a full-Xcode Mac (e.g. the `macos-arm64` CI mini). -- The hero defaults to a synthetic synthwave frame — set `PUNKTFUNK_SHOT_HERO` to a real captured - frame for a production-quality lead screenshot. - -**CI**: the `apple` workflow's **`screenshots`** job runs on the `macos-arm64` runner on every main -push + manual dispatch (skipped on PRs), and attaches the result as a single zip artifact, -**`punktfunk-appstore-screenshots`** (download it from the run's Artifacts; `upload-artifact@v3` — -Gitea's backend rejects v4). It captures the two **required iOS sizes — iPhone 6.9" + iPad 13"** — -on the Simulator (auto-creating the device if the runner lacks it), and is isolated from the -build/test job so a capture hiccup never reds the build. - -**macOS and tvOS are NOT in CI**, by design: the self-hosted runner is **headless** (no -window-server session), so the macOS window capture can't run there, and tvOS needs the Tier-3 -build-std slice. Generate those on a GUI Mac: `tools/screenshots.sh macos tvos`. (If the runner is -ever switched to a logged-in GUI session, re-adding macOS to the job's capture step is one line.) - -## Notes for whoever picks this up next - -1. **cbindgen import quirk** (the predicted "small compile fixes", now fixed): the - C17-compatible header spells `PunktfunkStatus`/`PunktfunkInputKind` as integer typedefs while - the enum *constants* import into Swift as a distinct same-named type — bridge with - `.rawValue` (see the top of `PunktfunkConnection.swift`). Don't fight the generated header. -2. **ABI contract**: one video pump thread per connection, plus optionally one *separate* - audio drain thread for `nextAudio()` and one feedback drain thread for - `nextRumble()`/`nextHidOutput()` (the core keeps per-plane borrow slots, so the planes - never alias; rumble + HID-output are two planes drained sequentially by the one - feedback thread); `send()` is enqueue-only and safe alongside all of them. The - wrapper's per-plane locks make `close()` safe from anywhere (it waits out in-flight - polls, ≤ their timeouts). -3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band - and recovery keyframes re-send them — "refresh the format description on every IDR" - (what `StreamView` does) is sufficient; there is no out-of-band extradata, ever. -4. **Stage 2 — built, opt-in (`punktfunk.presenter == "stage2"`, default stage 1).** Explicit - `VTDecompressionSession` decode (`VideoDecoder`) → a `CAMetalLayer` + display-link present - (`MetalVideoPresenter`/`Stage2Pipeline`), hosted as a sublayer by the same `StreamView`s with - input capture + HUD unchanged. It adds a **capture→present** (glass-to-glass, modulo the host - render→capture term) HUD line, skew-corrected via `PunktfunkConnection.clockOffsetNs`. The - decode half is unit-tested (`testVideoDecoderAsyncCallbackDeliversPixels`); the Metal present - is display-bound — **validate live** (flip the Settings "Presenter" picker, watch the HUD - number and that the image looks right) before making it the default. 10-bit/HDR + a smoothing - pacer are later. Plan: `docs-site/content/docs/apple-stage2-presenter.md`. -5. **Audio — wired, both directions.** Playback: `SessionAudio` drains `nextAudio()` - on its own thread, decodes through CoreAudio's built-in Opus codec (`OpusCodec.swift` - — kAudioFormatOpus, no bundled libopus; round-trip unit-tested) into a priming - jitter ring feeding an `AVAudioSourceNode`. Mic: a second engine taps the input - device, resamples to 48 kHz stereo, Opus-encodes 20 ms chunks and `sendMic()`s them - (the host's virtual PipeWire source accepts any frame size ≤ 120 ms). Speaker/mic - are chosen in Settings (`AudioDevices.swift` — persisted by UID; "System default" - leaves the engines unpinned so they follow macOS device changes), mic on/off toggle - included; the app asks for mic permission on first use - (NSMicrophoneUsageDescription is in the Xcode target). A/V sync and packet-loss - concealment beyond silence-fill are still open (AudioPacket.seq/ptsNs carry what's - needed). Decode with libopus or `AVAudioConverter`/`kAudioFormatOpus` into an - `AVAudioEngine` source node; conceal gaps (drop/dup) rather than blocking — the Rust - side buffers 320 ms and drops the newest packet when the puller lags. Wall-clock - `ptsNs` shares the host clock with video AUs for A/V sync. Wiring this into - `PunktfunkClient` is the next app-side task. -6. **Gamepads — wired end to end.** Exactly ONE controller (the `GamepadManager` - selection) forwards as pad 0; the host accumulates the incremental events into a - virtual pad whose TYPE the client negotiates in the Hello (`gamepad:` connect - parameter, echoed resolved in `resolvedGamepad` — Automatic resolves from the physical - pad at connect time; host precedence: explicit client choice > host `PUNKTFUNK_GAMEPAD` - env > Xbox 360). A DualSense session carries the full feel: adaptive-trigger blocks - (`DualSenseTriggerEffect.parse` — mode bytes per the community convention - (Nielk1/ds5w/inputtino), total, unknown → `.off`), lightbar, player LEDs, touchpad, - motion. **Motion scale constants** (`GamepadWire.gyroLSBPerRadS` = 20 LSB per deg/s, - `accelLSBPerG` = 10000) are derived from hid-playstation's math over the host's fixed - calibration blob, not yet live-verified — if gyro/accel feel wrong in a real game, - correct sign/scale in `GamepadCapture.forwardMotion`/`GamepadWire` and `evtest` the - host's virtual pad. Twin identical controllers share a fingerprint base, so a manual - pin can swap between them across reconnects (documented in the Settings footer). -7. **Trust — the full ceremony exists now (SPAKE2).** `generateIdentity()` once (persist - both PEMs in the Keychain), then `pair(host:identity:pin:name:)` with the 4-digit PIN - the host prints when it ARMS pairing (`--allow-pairing`/`--require-pairing`; one PIN - per arming window, surfaced in the host's web console — port 3000 → Pairing — and - printed at startup; the user reads it before pairing). Returns the - host's VERIFIED fingerprint; persist it and pass `pinSHA256:` + `identity:` to every - connect. Pairing is a real PAKE: a wrong PIN gets ONE online guess (no offline - dictionary attack), throwing `.wrongPIN`; a wrong-size pin throws `.invalidPin`. `PunktfunkClient` implements both flows: - the TOFU fingerprint sheet keeps working against hosts not running - `--require-pairing`, and the PIN ceremony is wired in — `ClientIdentityStore` - (Keychain) on every connect, `PairSheet` from a host card's context menu or the trust - prompt's "Pair with PIN instead…" (the host's accept loop is sequential, so that path - drops the live session before pairing). With `--require-pairing` the host now - authorizes clients too (the "other direction" is no longer open, opt-in per host); - the whole gate is regression-tested in `testPairingCeremonyAndRequirePairingGate`. -7b. **Resize without reconnect**: `requestMode(width:height:refreshHz:)` mid-stream — - the host rebuilds at the new mode in ~90 ms; the first new-mode AU is an IDR with - fresh parameter sets (the refresh-on-IDR decode flow handles it untouched) and - `currentMode()` reflects the switch. Wire it to window-resize events. -8. **Input capture** (stage 1): capture is a deliberate, reversible STATE owned by - `StreamLayerView`, Moonlight-style. Engaged when the stream starts / trust is - confirmed and when the user clicks into the video (that click is suppressed toward - the host); released by ⌘⎋ (toggles) or focus loss; NEVER engaged by mere app - activation — activating clicks may be title-bar drags or resizes, which used to get - their cursor warped away mid-drag. While captured: the local cursor is hidden + - frozen mid-view (the host renders its own), all input is forwarded, and the view - consumes key events as first responder so unhandled keyDowns don't beep — ⌘-combos - still work locally (⌘D disconnect, ⌘Q) *and* reach the host via GC. While released: - nothing is forwarded (`InputCapture.forwarding` gates the GC handlers; held - keys/buttons are flushed host-side on release so nothing sticks down), the cursor is - free, and the HUD shows "Click the stream to capture input". GC handlers only fire - while the app has focus, and focus loss also auto-releases everything held. One live capture per process (the GC - mouse/keyboard singletons have a single handler slot — ownership is tracked so a stale - capture's stop() can't clobber a newer one). -9. **iOS/iPadOS — ported and first-lit** (iPad simulator ↔ the real host, 60 fps). - `BUILD_IOS=1 bash scripts/build-xcframework.sh` builds device + universal-simulator - slices; the Xcode project has a second target, **Punktfunk-iOS**, sharing the same - synchronized sources. The iOS `StreamView` (StreamViewIOS.swift — same name/signature - as the macOS one, so the SwiftUI shell is identical) hosts the shared `StreamPump` in - a view controller for `prefersPointerLocked`: with a hardware mouse/trackpad that is - the iPadOS cursor capture (system honors it fullscreen-and-frontmost; in Stage - Manager it degrades to absolute-mouse forwarding). Input is routed by kind: DIRECT - fingers / Pencil are touches (each gets a wire touch id, coordinates mapped through the - aspect-fit letterbox into host-mode pixels — surface == host mode, so the host rescale is - the identity), while a mouse/trackpad is a MOUSE — pointer-LOCKED it is GCMouse relative - deltas; unlocked it is absolute moves + buttons + scroll over the UIKit pointer path - (hover + `.indirectPointer` touches), the local cursor staying visible so you can aim. An - indirect pointer is never sent as a touch. Touch is gated on trust (not forwarded under - the TOFU prompt), and returning to the foreground restores the capture you had on leaving. - `InputCapture` is cross-platform (GC works the same on iPadOS; ⌘⎋ is detected from - the HID stream there); audio routes via `AVAudioSession` (the Settings device - pickers are macOS-only). For the iPad-with-external-display setup: the target - enables multiple scenes + indirect input events — on Stage Manager iPads, drag the - punktfunk window onto the external screen and the stream runs there with full - keyboard/mouse/touch. While streaming the session is immersive (edge-to-edge, - status bar + home indicator hidden) and the iPadOS cursor is hidden over the video only - while the scene is actually pointer-LOCKED (`UIPointerInteraction` `.hidden()`); when the - lock isn't held it stays visible and the mouse forwards as an absolute cursor instead; on - iOS first run the stream mode defaults to the device's native screen so the video - fills the display. **tvOS** runs the same app (target **Punktfunk-tvOS**, first-lit - in the Apple TV simulator at 720p60): playback-only audio (no mic on tvOS), - focus-driven UI (`.card` host tiles), no kb/mouse capture yet — input lands with - gamepad support, the natural tvOS input anyway. While streaming there is NO focusable - control (a focusable Disconnect button would let the focus engine eat the controller's A - before the host sees it); the Siri Remote's **Menu** button disconnects (`.onExitCommand`). - Core slices are tier-3 Rust targets (see Build above). Known gaps: true pointer LOCK (`prefersPointerLocked`) isn't - consulted through UIHostingController, so the hidden cursor can still drift onto a - second screen (fixing it means putting the controller into the UIKit presentation - chain); and - AVAudioSession interruptions (calls, Siri) don't auto-restart the audio engines yet - (reconnect recovers). - -## Known limitations of the current host (relevant to client UX) - -- One session **at a time** (the listener is persistent, but a second concurrent client - waits in the accept queue until the current session ends — the virtual output and - encoder are single-tenant). -- Mid-stream renegotiation (resolution change without reconnect) is designed-for but not - implemented (the Welcome is one-shot today). -- Host-side gamepad injection needs `/dev/uinput` access on the box (udev rule from - `design/linux-setup.md`). +- **[Documentation](https://docs.punktfunk.unom.io)** — quick start, pairing, troubleshooting +- **[Project README](../../README.md)** — the host, the other clients, and how it all fits together diff --git a/clients/decky/README.md b/clients/decky/README.md index d7247a5..fa658f3 100644 --- a/clients/decky/README.md +++ b/clients/decky/README.md @@ -1,189 +1,84 @@ -# punktfunk Decky plugin (SteamOS / Steam Deck) +# punktfunk — Steam Deck plugin (Decky) -A **[Decky Loader](https://decky.xyz/)** plugin that adds a **punktfunk** panel to the Steam -Deck's Quick Access Menu (the QAM, opened with the `…` button), so you can launch the -punktfunk streaming client from **Gaming Mode** without dropping to the desktop. +Stream to your **Steam Deck** without ever leaving Gaming Mode. This +**[Decky Loader](https://decky.xyz/)** plugin adds a **punktfunk** panel to the Quick Access Menu +(the `…` button): discover hosts on your network, pair with a PIN, tweak stream settings, and launch +a fullscreen, gamescope-focused stream — all from the couch, gamepad-navigable. -Because Decky plugins run inside Steam's CEF, the panel is built from real Steam UI -primitives (`@decky/ui`: `PanelSection`, `PanelSectionRow`, `ButtonItem`, `Field`, -`Spinner`) — so it looks and feels native to Gaming Mode. - -> **Full Gaming-Mode client.** Discovery, a fullscreen page, in-UI SPAKE2 PIN pairing, -> stream settings, and a stream that actually launches fullscreen under gamescope (via a -> Steam shortcut, MoonDeck-style). The video itself is the existing GTK4 flatpak client -> (`io.unom.Punktfunk`) — the plugin discovers, pairs, configures, and *launches it the -> right way* so gamescope focuses it. The Steam-shortcut launch + pairing need a real Deck -> in Gaming Mode to fully confirm. +The video itself is the native GTK4 Linux client (the `io.unom.Punktfunk` flatpak); the plugin +discovers, pairs, configures, and *launches it the right way* so gamescope fullscreens it — the same +Steam-shortcut trick MoonDeck uses. Because it's built from real Steam UI primitives (`@decky/ui`), +the panel looks and feels native to Gaming Mode. ## What it does -1. **Discover** — browses the LAN over mDNS for punktfunk/1 hosts (`_punktfunk._udp`, - backend `discover()` via `avahi-browse`). Shown in both the QAM panel and a **fullscreen - page** (Decky route `/punktfunk`, via `routerHook.addRoute`). -2. **Pair** — for a `pair=required` host: a gamepad-navigable PIN keypad. The operator arms - pairing on the host (it shows a 4-digit PIN), the user enters it on the Deck, and the - backend runs the SPAKE2 ceremony headlessly via the flatpak client's `--pair` mode - (`pair()`), persisting the host as paired so the stream then connects silently. -3. **Stream** — launches fullscreen in Gaming Mode. The plugin registers ONE hidden - non-Steam shortcut pointing at `bin/punktfunkrun.sh`, passes `PF_HOST` as the shortcut's - Steam launch options, and starts it with `SteamClient.Apps.RunGame` — so gamescope - focuses + fullscreens it. (A flatpak launched directly from the backend is invisible: - gamescope only focuses the process tree Steam launched via `reaper` — gamescope#484.) - The wrapper then execs `flatpak run io.unom.Punktfunk --connect `. -4. **Settings** — resolution / refresh / bitrate / gamepad / mic, written to the client's - `client-gtk-settings.json` (`get_settings`/`set_settings`), which the launched client reads. +1. **Discover** — browses the LAN over mDNS for punktfunk hosts, in both the QAM panel and a + fullscreen page. +2. **Pair** — for a host that requires it, a gamepad-navigable PIN keypad runs the SPAKE2 pairing + ceremony headlessly, then remembers the host so future streams connect silently. +3. **Stream** — launches fullscreen via a hidden Steam shortcut so gamescope focuses it. +4. **Settings** — resolution / refresh / bitrate / gamepad / mic, written to the client's config. -To leave the stream: the in-client controller chord (**L1+R1+Start+Select**) or close the -"game" from the Steam overlay — exiting the client ends the Steam game and returns to -Gaming Mode automatically. - -## Architecture - -| File | Role | -| --- | --- | -| `src/index.tsx` | Frontend: QAM panel + the `/punktfunk` fullscreen page (host list, PIN keypad modal, settings). | -| `src/steam.ts` | Steam-shortcut launch (`AddShortcut` / `SetAppLaunchOptions` / `RunGame`) — the focus-correct stream start. | -| `src/backend.ts` | Typed `callable` bridges to `main.py`. | -| `bin/punktfunkrun.sh` | The launch wrapper the Steam shortcut targets (so the window is focusable). | -| `main.py` | Backend: `discover` / `pair` / `runner_info` / `get_settings` / `set_settings` / `kill_stream` / `check_update`. | -| `plugin.json` | Decky plugin manifest. | -| `update.json` | CI-baked `{channel, manifest}` — where `check_update()` polls (absent on dev builds). | -| `decky.pyi` | Type stub for the injected `decky` module (vendored from the template). | - -### Discovery (`discover()`) - -Shells out to **`avahi-browse -rpt _punktfunk._udp`** (SteamOS and Bazzite ship -`avahi-daemon`; this avoids bundling python-zeroconf): - -- `-r` resolve services, `-p` parseable output, `-t` terminate after the cache dump. -- Resolved records start with `=` and are semicolon-separated: - `=;iface;protocol;name;type;domain;hostname;address;port;txt`. -- The `txt` column is space-separated, quoted `"key=value"` tokens. We read the keys the - host advertises (`crates/punktfunk-host/src/discovery.rs`): `proto`, `fp`, `pair`, `id`. -- Records are deduped on the `id` TXT key (a host re-advertises per interface and across - IPv4/IPv6), preferring the IPv4 address for the user-facing host string. - -### Client launch (`connect()`) - -The client binary `punktfunk-client` is resolved in order: `PATH` → `/usr/bin` → -`/usr/local/bin` → `~/.local/bin` → a `flatpak run io.unom.Punktfunk` fallback. The resolved -argv and a clear `client-not-found` error surface to the UI. The child PID is tracked so -`disconnect()` (and plugin `_unload`) can terminate it. - -> On the **Steam Deck** the client install is the flatpak `io.unom.Punktfunk` -> (`packaging/flatpak/`) — SteamOS `/usr` is read-only and lacks `libadwaita`/`libSDL3`, so -> the flatpak (which bundles them) is the canonical path; the resolver's flatpak fallback -> launches exactly that. - -## Prerequisites - -- **Decky Loader** installed on the Deck (https://decky.xyz/). -- **`punktfunk-client`** (the GTK4/libadwaita Linux client, crate `punktfunk-client-linux`) - installed and runnable on the Deck — via `.deb`/RPM/flatpak, or symlinked into - `~/.local/bin`. -- **avahi** (`avahi-daemon` + `avahi-browse`) for discovery — present on SteamOS/Bazzite. -- A punktfunk/1 host on the LAN (`punktfunk-host serve` or `punktfunk1-host`). - -## Build - -```sh -pnpm install -pnpm build # rollup → dist/index.js -``` - -(`npm install && npm run build` also works.) +To leave a stream: the in-client controller chord (**L1 + R1 + Start + Select**), or close the +"game" from the Steam overlay — either returns you to Gaming Mode. ## Install on the Deck -### Option A — Decky "install from URL" (recommended; published by CI) +You need **[Decky Loader](https://decky.xyz/)** and the **`io.unom.Punktfunk` flatpak** +([`packaging/flatpak`](../../packaging/flatpak/README.md)) installed on the Deck — SteamOS `/usr` is +read-only, so the flatpak (which bundles libadwaita/SDL3) is the canonical client. Discovery uses +`avahi-browse`, which ships on SteamOS/Bazzite. -CI (`.gitea/workflows/decky.yml`) builds the plugin into a store-layout zip and publishes it to -Gitea's **generic package registry** on every push to `main` and on `v*` tags, exposing a stable -URL. In Decky's settings → **Developer Mode** → **Install Plugin from URL**, paste: +**Recommended — install from URL** (published by CI): in Decky → Settings → **Developer Mode** → +**Install Plugin from URL**, paste: ``` https://git.unom.io/api/packages/unom/generic/punktfunk-decky/latest/punktfunk.zip ``` -(or a pinned version: `.../punktfunk-decky//punktfunk.zip`). On tags the same zip is -also attached to the Gitea release. The zip's layout is the store-required one — a single -top-level `punktfunk/` dir holding `plugin.json`, `package.json`, `main.py`, `dist/index.js`, -`README.md`, and `LICENSE`. +(or a pinned `.../punktfunk-decky//punktfunk.zip`). The plugin then **self-updates** without +the Decky store — when a newer build exists, an **Update to vX** button appears and drives Decky +Loader's own (SHA-256-verified) install. -### Option B — manual dev copy (sideload) - -Decky's `~/homebrew/plugins/` is **root-owned** (PluginLoader runs as root and manages it), so a -plain `rsync` into it fails — stage to a writable temp dir, then `sudo`-install and restart the -loader. The two helper scripts do exactly this: +## Build & sideload (development) ```sh cd clients/decky pnpm install +pnpm build # rollup → dist/index.js pnpm run package # → out/punktfunk/ + out/punktfunk-v.zip -DECK=deck@ pnpm run deploy # rsync → /tmp, sudo cp into plugins/, chown root, restart +DECK=deck@ pnpm run deploy # rsync → /tmp, sudo-install into the root-owned plugins dir, restart loader ``` -`deploy.sh` prompts for the Deck's sudo password interactively (via `ssh -t`); set `DECKPASS=…` -to run it non-interactively. Equivalent by hand: +`~/homebrew/plugins/` is root-owned (the loader runs as root), so `deploy.sh` stages to a temp dir +then `sudo`-installs and restarts the loader — set `DECKPASS=…` to run it non-interactively. A loader +restart is required for an out-of-band install to appear. -```sh -cd clients/decky && pnpm build && bash scripts/package.sh -rsync -azp --delete out/punktfunk/ deck@:/tmp/punktfunk/ -ssh -t deck@ 'sudo sh -c "rm -rf ~deck/homebrew/plugins/punktfunk && \ - cp -r /tmp/punktfunk ~deck/homebrew/plugins/punktfunk && \ - chown -R root:root ~deck/homebrew/plugins/punktfunk && systemctl restart plugin_loader"' -``` +## Architecture -A loader restart is required for an out-of-band install to appear. The **punktfunk** panel then -shows up in the Quick Access Menu. +| File | Role | +| --- | --- | +| `src/index.tsx` | Frontend: QAM panel + the `/punktfunk` fullscreen page (host list, PIN keypad, settings). | +| `src/steam.ts` | Steam-shortcut launch (`AddShortcut` / `SetAppLaunchOptions` / `RunGame`) — the focus-correct stream start. | +| `src/backend.ts` | Typed `callable` bridges to `main.py`. | +| `bin/punktfunkrun.sh` | The launch wrapper the Steam shortcut targets (so the window is focusable). | +| `main.py` | Backend: `discover` (via `avahi-browse`) / `pair` / settings / `kill_stream` / `check_update`. | +| `plugin.json` · `update.json` | Decky manifest; CI-baked update channel. | -> The plugin launches the client via the flatpak `io.unom.Punktfunk` (see -> [`../../packaging/flatpak/README.md`](../../packaging/flatpak/README.md)) — install that on -> the Deck too, or the panel's Connect surfaces a `client-not-found` error. - -## Updating (self-update, no store) - -The plugin updates itself without the official Decky store. CI (`decky.yml`) publishes a tiny -per-channel `manifest.json` next to the zip in the Gitea registry: - -```json -{"version":"0.3.123","artifact":".../punktfunk-decky/0.3.123/punktfunk.zip","sha256":"…"} -``` - -and bakes an `update.json` (`{channel, manifest}`) into the plugin so it knows which channel it was -installed from. The backend `check_update()` reads the **installed** version from `package.json` — -the value Decky itself reports (it does **not** read `plugin.json`) — fetches the channel manifest, -and compares. When a newer build exists the frontend shows an **Update to vX** button that drives -Decky Loader's own install RPC: - -```ts -window.DeckyBackend.callable("utilities/install_plugin")(artifact, "punktfunk", version, hash, /*UPDATE=*/2) -``` - -The loader (root) downloads the immutable per-version zip, **SHA-256-verifies** it against `hash`, -replaces `~/homebrew/plugins/punktfunk`, and hot-reloads — the unprivileged backend never writes the -root-owned plugins dir itself. `window.DeckyBackend` / `utilities/install_plugin` are loader -internals (not `@decky/api`), so every access is guarded; missing them, the button falls back to a -toast pointing at **Install Plugin from URL**. - -> CI stamps a **plain numeric** semver per channel (`0.3.` canary, `X.Y.Z` stable) into -> `package.json`. Decky's `compare-versions` orders pre-release identifiers lexically (so `ci10 < ci9`) -> — a `-ciN` suffix would mis-detect updates. - -**Optional — native Updates tab:** Decky's store is single-source (a custom store URL *replaces* the -official catalog), so punktfunk doesn't ship one by default. A user who wants the native update badge -can point Decky → Settings → **Custom store** at a punktfunk-only store JSON — not recommended if you -use other plugins, since it hides the official catalog. +The client binary is resolved `PATH` → `/usr/bin` → `/usr/local/bin` → `~/.local/bin` → a +`flatpak run io.unom.Punktfunk` fallback, so the flatpak install always works. ## Limitations / next steps -- **Needs on-Deck validation in Gaming Mode**: the Steam-shortcut launch (`AddShortcut` / - `RunGame` / the `gameId` encoding) and the headless pairing env are coded to MoonDeck's - proven pattern but verified only at build time here. -- mDNS discovery depends on `avahi-browse`; no manual "add host by IP" entry yet. -- No in-stream overlay (latency/bitrate HUD) inside the plugin — the client owns the session - once launched; leave it with the L1+R1+Start+Select chord. -- Pairing requires the operator to **arm pairing on the host** (so it shows the PIN); the - plugin can't arm it remotely (no host mgmt token on the Deck). -- Settings are written to the flatpak's sandbox config path; if the client ever moves its - config location, that path mapping must follow. +- **Needs on-Deck validation in Gaming Mode** — the Steam-shortcut launch and headless pairing follow + MoonDeck's proven pattern but are verified only at build time here. +- No manual "add host by IP" entry yet (discovery is mDNS-only). +- No in-stream overlay inside the plugin — the client owns the session once launched. +- Pairing needs the operator to **arm pairing on the host** so it shows the PIN; the plugin can't arm + it remotely. + +## Related + +- **[Documentation](https://docs.punktfunk.unom.io/docs/steam-deck)** — Steam Deck setup guide +- **[Linux client](../linux/README.md)** — the app this plugin launches +- **[Project README](../../README.md)** — the host, the other clients, and how it all fits together diff --git a/clients/linux/README.md b/clients/linux/README.md new file mode 100644 index 0000000..9f08ba2 --- /dev/null +++ b/clients/linux/README.md @@ -0,0 +1,77 @@ +# punktfunk — Linux client + +The native **Linux** app for streaming a punktfunk host to your desktop, laptop, or Steam Deck. +It's a clean GTK4/libadwaita app that finds hosts on your network, pairs with a PIN, and puts a +low-latency stream on glass at your display's own resolution and refresh rate. + +Built in Rust, it links the shared **`punktfunk-core`** directly (no C ABI) and speaks the fast +**`punktfunk/1`** protocol — QUIC control plane, GF(2¹⁶) FEC + AES-GCM data plane. + +## Features + +- **Zero-copy hardware decode** — FFmpeg VAAPI decode → DRM-PRIME dmabuf → `GdkDmabufTexture` + (Tier-1 zero-copy on Intel and AMD), with an automatic software-HEVC fallback on NVIDIA or when + VAAPI is unavailable. +- **Your display's native mode** — the host builds a virtual output at exactly your WxH@Hz; no + scaling, no letterboxing. Steady 60 fps at 1080p60, ~6 ms capture→decoded on the LAN. +- **Audio both ways** — PipeWire playback with a jitter ring, plus mic uplink to the host. +- **Full controller support** — SDL3 gamepads with rumble and DualSense fidelity (lightbar, player + LEDs, touchpad, motion, adaptive-trigger replay). Click-to-capture keyboard and mouse, with a + release chord (Ctrl+Alt+Shift+Q) and focus-loss release. +- **Find hosts automatically** — mDNS discovery lists hosts on your LAN; saved hosts persist. + First connect does a one-time **SPAKE2 PIN pairing** (or TOFU on trusted LANs), then reconnects on + a pinned identity. +- **Per-host speed test** to pick a bitrate, plus compositor and mode preferences in Settings. + +## Get it + +Most people should install a package rather than build from source: + +| Distro | Install | +|--------|---------| +| **Flatpak** (any distro, Steam Deck) | `io.unom.Punktfunk` — see [`packaging/flatpak`](../../packaging/flatpak/README.md) | +| **Ubuntu / Debian** (apt) | `sudo apt install punktfunk-client` *(after adding the repo)* | +| **Fedora / Bazzite** (rpm) | `rpm-ostree install punktfunk-client` | +| **Arch** (PKGBUILD) | see [`packaging/arch`](../../packaging/arch/README.md) | + +Per-device install steps and pairing walkthrough: +**[docs.punktfunk.unom.io/docs/install-client](https://docs.punktfunk.unom.io/docs/install-client)**. + +## Build & run from source + +Requires GTK ≥ 4.16, libadwaita ≥ 1.5, FFmpeg 7 or 8 (with VAAPI for hardware decode), PipeWire, +and SDL3 (with hidapi) development packages. + +```sh +# from the repo root +cargo run -p punktfunk-client-linux # launch the app +cargo run -p punktfunk-client-linux -- --discover # list hosts on the LAN, then exit +cargo run -p punktfunk-client-linux -- --connect HOST[:PORT] # skip the host list and connect +``` + +The binary is named **`punktfunk-client`**. Handy flags: `--connect host[:port]` (start a session +immediately — for scripting and the Steam Deck launcher), `--discover [secs]`, and +`--pair --connect host[:port]` (run the pairing ceremony headlessly). Force a decoder with +`PUNKTFUNK_DECODER=software|vaapi`. + +## Layout + +``` +src/ + main.rs · app.rs entry point, GTK application, CLI paths + ui_hosts.rs host list (mDNS + saved), pairing / trust dialogs + ui_settings.rs resolution · refresh · decoder · bitrate · compositor · mic + ui_stream.rs the stream window (GtkGraphicsOffload present) + input capture + session.rs session lifecycle over the NativeClient connector + video.rs FFmpeg VAAPI / software decode → dmabuf / texture + audio.rs PipeWire playback + mic uplink + gamepad.rs · keymap.rs SDL3 controllers + feedback; keyboard VK mapping + trust.rs · discovery.rs persistent identity, TOFU/PIN pairing, mDNS browse +tools/screenshots.sh store screenshot capture +``` + +## Related + +- **[Documentation](https://docs.punktfunk.unom.io)** — quick start, pairing, troubleshooting +- **[Steam Deck plugin](../decky/README.md)** — launches this client fullscreen in Gaming Mode +- **[Project README](../../README.md)** — the host, the other clients, and how it all fits together diff --git a/clients/probe/README.md b/clients/probe/README.md new file mode 100644 index 0000000..611ade5 --- /dev/null +++ b/clients/probe/README.md @@ -0,0 +1,50 @@ +# punktfunk — probe (reference client) + +`punktfunk-probe` is the **headless reference client** for the `punktfunk/1` protocol — a +command-line tool for testing, latency measurement, and validating host behavior. It's not a +streaming app you'd watch on; it connects, exercises a plane, and reports numbers. If you want to +actually stream, use the [Linux](../linux/README.md), [Windows](../windows/README.md), +[Apple](../apple/README.md), or [Android](../android/README.md) clients. + +Because it links the same **`punktfunk-core`** as every other client, it's also the canonical +example of driving the protocol end to end: QUIC control plane, UDP data plane, and the side planes +(input, audio, rumble) over QUIC datagrams. + +## What it does + +- **Receives a real stream**, writes a playable `.h265`, and reports per-frame + **capture→…→reassembled latency** percentiles (the host stamps each frame with its capture clock). +- **Verification mode** against a synthetic host — byte-checks deterministic test frames. +- **Exercises every plane** with scripted test traffic: + `--input-test` (mouse/keyboard), `--mic-test` (a 440 Hz Opus tone up to the host mic), + `--touch-test` (a synthetic finger), `--rich-input-test` (DualSense touchpad + motion, logging the + HID-output feedback that comes back). +- **Trust** — `--pin <64-hex>` pins the host fingerprint; `--pair ` runs the SPAKE2 pairing + ceremony and prints the verified fingerprint to pin from then on. Without a pin it trusts on first + use. +- **Discovery** — `--discover [secs]` browses the LAN for `_punktfunk._udp` hosts and prints each + (name, addr:port, pairing requirement, cert fingerprint), then exits. +- **Negotiation knobs** — `--mode WxHxFPS`, `--remode` (mid-stream mode change), `--bitrate`, + `--audio-channels` (stereo / 5.1 / 7.1), `--compositor`, `--gamepad`, `--launch`, `--speed-test`. + +## Usage + +```sh +# stream 720p120 from a host, save the video, and print latency percentiles: +cargo run -p punktfunk-probe -- --mode 1280x720x120 --connect HOST:PORT --out /tmp/a.h265 + +# list hosts on the LAN: +cargo run -p punktfunk-probe -- --discover + +# pair with a host that requires it (read the PIN off the host), then stream: +cargo run -p punktfunk-probe -- --connect HOST:PORT --pair 1234 +cargo run -p punktfunk-probe -- --connect HOST:PORT --pin <64-hex> --input-test +``` + +Full flag reference is in the module doc-comment at the top of [`src/main.rs`](src/main.rs). + +## Related + +- **[Project README](../../README.md)** — the host, the streaming clients, and the protocol +- **`punktfunk-host punktfunk1-host`** — the persistent native-protocol listener to probe against + (see the "Running on this box" section of the repo README / `CLAUDE.md`) diff --git a/clients/windows/README.md b/clients/windows/README.md new file mode 100644 index 0000000..13f4037 --- /dev/null +++ b/clients/windows/README.md @@ -0,0 +1,71 @@ +# punktfunk — Windows client + +The native **Windows** app for streaming a punktfunk host to your PC. A modern WinUI 3 app that +discovers hosts on your network, pairs with a PIN, and streams at your display's own resolution and +refresh rate — with a hardware-accelerated D3D11 video path and HDR. + +It's **pure Rust**: the UI is WinUI 3 driven through [windows-reactor](https://github.com/microsoft/windows-rs) +(a declarative, React-like framework), and it links the shared **`punktfunk-core`** directly to speak +the fast **`punktfunk/1`** protocol. + +## Features + +- **Hardware decode, GPU present** — FFmpeg HEVC with a **D3D11VA zero-copy path** (decoder and + presenter share one D3D11 device; NV12/P010 textures sampled straight into a `SwapChainPanel` + composition swapchain), with a robust software-decode fallback. +- **HDR10** — advertise 10-bit/HDR, detect PQ in-band, and flip the swapchain to `R10G10B10A2` + + ST.2084 with HDR10 metadata. +- **Your display's native mode** — the host builds a virtual display at exactly your WxH@Hz. +- **Audio both ways** — WASAPI render + mic capture. +- **Full controller support** — SDL3 gamepads with rumble, lightbar, and DualSense feedback. +- **Find hosts automatically** — mDNS discovery lists hosts on your LAN, alongside saved and manual + entries. First connect does a one-time **SPAKE2 PIN pairing** (or TOFU on trusted LANs), then + reconnects on a pinned identity. +- **Polished shell** — host cards, settings (resolution / refresh / decoder / bitrate / HDR / mic), + a status-chip stream HUD, and the full trust surface. Stream input uses Win32 low-level hooks with + a Ctrl+Alt+Shift+Q capture toggle. + +Builds and ships for both **x64** and **ARM64** as a signed **MSIX**. + +## Get it + +Install the signed MSIX from the package registry — see +**[docs.punktfunk.unom.io/docs/install-client](https://docs.punktfunk.unom.io/docs/install-client)**. +A stock [Moonlight](https://moonlight-stream.org/) client also works over GameStream if you prefer. + +## Build from source + +Windows-only (the crate builds as a stub on other platforms so the workspace stays green). You need +the MSVC toolchain, an `FFMPEG_DIR` FFmpeg tree, and CMake (SDL3 builds from source). windows-reactor's +`build.rs` downloads the Windows App SDK NuGets and needs `CARGO_WORKSPACE_DIR` set. + +```sh +cargo build -p punktfunk-client-windows --target x86_64-pc-windows-msvc + +# CLI paths for testing (no window): +punktfunk-client --discover # list hosts on the LAN +punktfunk-client --headless --connect host[:port] [--pin HEX] # connect, count frames, print stats +``` + +> `CARGO_HOME` must be an ASCII path — non-ASCII characters break SDL3's MSVC precompiled-header +> build. Packaging (MSIX manifest, signing) lives in [`packaging/`](packaging/). + +## Layout + +``` +src/ + main.rs · app.rs entry point + CLI paths; WinUI 3 shell (windows-reactor) + present.rs · gpu.rs SwapChainPanel D3D11 composition swapchain; shared D3D11 device + video.rs FFmpeg HEVC decode (D3D11VA zero-copy + software fallback) + audio.rs WASAPI render + mic capture + gamepad.rs SDL3 controllers + rumble/lightbar/DualSense feedback + input.rs Win32 low-level keyboard/mouse hooks → host input + session.rs session lifecycle over the NativeClient connector + trust.rs · discovery.rs persistent identity, TOFU/PIN pairing, mDNS browse +packaging/ MSIX manifest, signing, pack script +``` + +## Related + +- **[Documentation](https://docs.punktfunk.unom.io)** — quick start, pairing, troubleshooting +- **[Project README](../../README.md)** — the host, the other clients, and how it all fits together diff --git a/crates/pf-driver-proto/README.md b/crates/pf-driver-proto/README.md new file mode 100644 index 0000000..a02b271 --- /dev/null +++ b/crates/pf-driver-proto/README.md @@ -0,0 +1,16 @@ +# pf-driver-proto + +The shared **host ↔ driver binary contract** for punktfunk's Windows **pf-vdisplay** virtual display — +the control IOCTLs and the IDD-push frame transport, defined exactly once. + +It's a path dependency of **both** the host workspace ([`crates/punktfunk-host`](../punktfunk-host)) +and the out-of-workspace driver workspace ([`packaging/windows/drivers/`](../../packaging/windows)), +so it must resolve identically from either build graph. That's why it's deliberately self-contained: +`no_std` (+ alloc), platform-neutral (GUID/LUID are plain integers each side converts to its own OS +type), and free of `*.workspace = true` inheritance. + +Defining every wire struct here — with `const` size/offset asserts and `bytemuck` round-trips — turns +host↔driver ABI drift into a **compile error** instead of a silent frame or IOCTL corruption. + +See the crate root ([`src/`](src/)) for the wire types; the Windows virtual-display design is in +[`design/windows-virtual-display-rust-port.md`](../../design/windows-virtual-display-rust-port.md). diff --git a/crates/punktfunk-core/README.md b/crates/punktfunk-core/README.md new file mode 100644 index 0000000..59a57e3 --- /dev/null +++ b/crates/punktfunk-core/README.md @@ -0,0 +1,61 @@ +# punktfunk-core + +The **shared protocol core** — the one place where punktfunk's transport, forward error correction, +and crypto live. It's linked into the [host](../punktfunk-host/README.md) and every native client, so +there's exactly one implementation of the wire format everywhere. + +Written in Rust with **no async on the per-frame path** (native threads only). It exposes both a +normal Rust API and a **stable, versioned C ABI**, so the Swift and Kotlin clients — and any C +embedder — link the same code as the Rust ones. + +## What's in here + +- **Transport & session** (`session.rs`, `transport/`, `packet.rs`) — the `punktfunk/1` data plane + over raw UDP: packetization, reassembly (with attacker-bounded limits), pacing, and socket tuning. +- **FEC** (`fec/`) — the wall-breaker. Two codes: + - **GF(2⁸)** classic Reed–Solomon with the *Cauchy* generator matrix — byte-identical to the + `nanors` library Moonlight uses, so our parity is decodable by a stock Moonlight client. + - **GF(2¹⁶) Leopard-RS** (SIMD, O(n log n)) — up to 65535 shards/block, which removes the ~1 Gbps + FEC ceiling. `punktfunk/1` negotiates this one. +- **Crypto** (`crypto.rs`) — AES-128-GCM session encryption with per-direction nonce salts and + sequence-as-AAD; SPAKE2 PIN pairing lives behind the `quic` feature. +- **QUIC control plane** (`quic.rs`, `client.rs`, feature `quic`) — the Hello/Welcome/Start handshake, + cert pinning/TOFU, reverse audio, and the embeddable `NativeClient` connector. This is the **only** + place `tokio`/`quinn` are allowed; the feature is **off by default** so the core stays runtime-free. +- **C ABI** (`abi.rs`) — the versioned surface (`punktfunk_abi_version()`, `PunktfunkConfig` carrying + its own `struct_size`) that generates [`include/punktfunk_core.h`](../../include/punktfunk_core.h) + via cbindgen at build time. + +## Build outputs + +The crate builds three ways at once (`crate-type = ["lib", "cdylib", "staticlib"]`): + +| Output | Used by | +|--------|---------| +| `lib` (rlib) | the host, probe, and tools link it as a normal Rust crate | +| `cdylib` (`.so`/`.dylib`) | the Swift / Kotlin clients via the C ABI | +| `staticlib` (`.a`) | the C test harness and static embedding | + +## Test + +```sh +cargo test -p punktfunk-core # unit + proptest + loopback +cargo run -p loss-harness # FEC loss-resilience sweep (no network needed) +bash crates/punktfunk-core/tests/c/run.sh # standalone C-ABI link + round-trip proof +``` + +## Design invariants (do not regress) + +- **One core, linked everywhere** — protocol/FEC/crypto live only here, behind the stable C ABI. +- **No async on the hot path** — the per-frame pipeline is native threads only; `quic` (tokio/quinn) + is control-plane only, feature-gated, off by default. +- **Security hardening stays intact** — the reassembler bounds attacker-controlled fields before + allocating; AES-GCM keeps per-direction nonce salts + seq-as-AAD; the ABI checks `struct_size`. + Regression tests exist — keep them green. + +## Related + +- **[`punktfunk-host`](../punktfunk-host/README.md)** — the streaming host built on this core +- **[Clients](../../clients/)** — the apps that link this core over the C ABI (or directly, in Rust) +- **[`design/implementation-plan.md`](../../design/implementation-plan.md)** — why GF(2¹⁶) FEC, the + latency budget, and the architecture thesis diff --git a/crates/punktfunk-host/README.md b/crates/punktfunk-host/README.md new file mode 100644 index 0000000..5336dd7 --- /dev/null +++ b/crates/punktfunk-host/README.md @@ -0,0 +1,90 @@ +# punktfunk-host + +The **streaming host** — the program you run on the machine whose desktop or games you want to +stream. For each client that connects, it spins up a **virtual display sized to that device**, +captures it on the GPU, encodes with hardware NVENC/VAAPI/AMF/QSV, and sends it out over a +low-latency transport — no physical monitor, no letterboxing, no rearranging your real screens. + +It speaks two protocols from **one process**: + +- **GameStream** — so any [Moonlight](https://moonlight-stream.org/) / Artemis client works day one. +- **`punktfunk/1`** — punktfunk's own faster protocol (QUIC control plane, GF(2¹⁶) FEC + AES-GCM data + plane) that the native clients use. + +Runs on **Linux** (the primary, most battle-tested path) and **Windows** (x64). The shared protocol, +FEC, and crypto live in [`punktfunk-core`](../punktfunk-core/README.md); this crate is everything +platform-facing around it. + +## What it does + +- **Per-client virtual displays at the exact WxH@Hz.** Linux uses per-compositor backends — **KWin**, + **gamescope**, **Mutter**, and **Sway/wlroots**; Windows uses its own all-Rust IddCx virtual display, + even on the secure desktop (UAC / lock screen). +- **GPU zero-copy capture → encode.** dmabuf → CUDA/Vulkan → NVENC on Linux; DXGI/WGC → GPU encode on + Windows. Encoders auto-select by GPU vendor: **NVENC** (NVIDIA), **VAAPI** (Linux AMD/Intel), + **AMF/QSV** (Windows AMD/Intel), or software H.264 as a floor. HDR/10-bit and HEVC 4:4:4 supported. +- **Input injection.** Mouse/keyboard (libei / gamescope EIS / wlr / Windows SendInput) and virtual + **gamepads** — Xbox 360/One, DualSense, DualShock 4 — with rumble and HID feedback back-channels. +- **Audio both ways.** Opus audio host→client, plus a virtual microphone the client can talk into. +- **Trust & discovery.** A persistent host identity, **SPAKE2 PIN pairing** (default) or TOFU, and + mDNS auto-advertisement so clients find the host without typing an IP. +- **Management API + web console.** A REST API (`mgmt.rs`, OpenAPI at + [`api/openapi.json`](../../api/openapi.json)) drives status, paired devices, and on-demand pairing; + the browser UI is in [`web/`](../../web/README.md). + +## Run it + +`punktfunk-host serve` runs inside your desktop session. Bare `serve` is the **secure native-only +default** (`punktfunk/1` + the management API); add `--gamestream` on a trusted LAN to also accept +stock Moonlight clients. + +```sh +# Linux, from the repo root (see the repo README "Running on this box" for the headless recipe): +cargo run -rp punktfunk-host -- serve # native-only (secure default) +cargo run -rp punktfunk-host -- serve --gamestream # + Moonlight compatibility +``` + +Then pair from the web console (`https://:3000`) or the client app. + +Most people should install a **package** rather than run from source — see +[`packaging/`](../../packaging/README.md) (apt · rpm/COPR/bootc · Arch/sysext · Windows installer) and +the per-platform guides at **[docs.punktfunk.unom.io/docs/install](https://docs.punktfunk.unom.io/docs/install)**. + +### Subcommands + +| Command | Purpose | +|---------|---------| +| `serve` | The host (native `punktfunk/1` + mgmt API; `--gamestream` adds Moonlight). | +| `punktfunk1-host` | Standalone native-protocol listener for testing/measurement (`--source virtual`, `--max-sessions`). | +| `openapi` | Print the management-API OpenAPI spec (regenerates `api/openapi.json`). | +| `library` | Inspect the multi-store game library. | +| `service` · `driver` · `web` | Windows: SCM service, driver install, bundled web console. | +| `*-test` / `*-selftest` / `*-probe` | Diagnostics (input, zero-copy, HDR, compositor, gamepads). | + +`--help` lists them all. + +## Layout + +``` +src/ + main.rs CLI + subcommand dispatch + config.rs · session_plan.rs · session_tuning.rs · pipeline.rs session setup + the frame pipeline + vdisplay/ per-compositor virtual outputs (kwin · gamescope · mutter · wlroots) + capture/ · capture.rs screen/dmabuf capture (+ Windows DXGI/WGC) + encode/ · encode.rs per-GPU encoders (nvenc · vaapi · ffmpeg_win (AMF/QSV) · sw) + zerocopy/ dmabuf → CUDA → NVENC bridges (EGL/GL tiled, Vulkan LINEAR) + inject/ · inject.rs input backends (libei · wlr · uinput gamepads · UHID DualSense/DS4) + audio/ · audio.rs Opus out + virtual mic (PipeWire / WASAPI) + gamestream/ Moonlight compat: nvhttp · pairing · rtsp · control · stream · gamepad · apps + punktfunk1.rs the native punktfunk/1 host (QUIC control + native-thread UDP data plane) + mgmt.rs · native_pairing.rs · stats_recorder.rs management API, pairing, perf capture + hdr.rs · library.rs HDR metadata; multi-store game library + linux/ · windows/ platform-confined backends +``` + +## Related + +- **[`punktfunk-core`](../punktfunk-core/README.md)** — the shared protocol · FEC · crypto core +- **[Clients](../../clients/)** — the apps that connect (Apple · Linux · Windows · Android · probe) +- **[Packaging](../../packaging/README.md)** & **[docs](https://docs.punktfunk.unom.io)** — install & operate +- **[`design/`](../../design/README.md)** — architecture rationale and deep-dive plans diff --git a/docs-site/README.md b/docs-site/README.md index 1b38dd9..08a2a32 100644 --- a/docs-site/README.md +++ b/docs-site/README.md @@ -3,9 +3,9 @@ The punktfunk documentation site: [Fumadocs](https://fumadocs.dev) on [TanStack Start](https://tanstack.com/start) (Vite + Nitro/bun preset). -Content lives in [`content/docs/`](content/docs) as `.md`/`.mdx`. Several pages are imported -verbatim from the repo's `docs/` design notes (with added frontmatter); edit those there or -here as the docs site becomes the source of truth. +Content lives in [`content/docs/`](content/docs) as `.md`/`.mdx`. This site is the source of truth +for the **user-facing** guides; repo-internal design rationale lives in +[`../design/`](../design/README.md). ## API reference diff --git a/packaging/README.md b/packaging/README.md index 92b84b8..0a74402 100644 --- a/packaging/README.md +++ b/packaging/README.md @@ -73,7 +73,7 @@ rpm-ostree install punktfunk && systemctl reboot systemctl reboot ``` -## Option B — bootc (image-based, atomic) +## Option C — bootc (image-based, atomic) Layer punktfunk into a Bazzite image once, then rebase any number of hosts onto it — no per-host drift. See `bootc/Containerfile`: diff --git a/packaging/windows/drivers/pf-dualsense/README.md b/packaging/windows/drivers/pf-dualsense/README.md index 2e727d9..dcc8fdb 100644 --- a/packaging/windows/drivers/pf-dualsense/README.md +++ b/packaging/windows/drivers/pf-dualsense/README.md @@ -1,26 +1,27 @@ -# pf-dualsense — virtual DualSense UMDF2 HID minidriver (M0 spike) +# pf-dualsense — virtual DualSense UMDF2 HID minidriver A self-authored **Rust UMDF2 HID minidriver** that presents a virtual Sony **DualSense** (VID `054C` / PID `0CE6`) to Windows, so games drive adaptive triggers / lightbar / rumble — -capabilities ViGEm structurally cannot deliver. This is the M0 feasibility spike for rich -controller support in the punktfunk Windows host. +capabilities ViGEm structurally cannot deliver. It's how the punktfunk Windows host gives a client's +DualSense a near-native feel with **no external gamepad dependencies** (no ViGEmBus). -## Status (2026-06-21) +Shipping: the driver is one member of the in-tree driver workspace +([`packaging/windows/drivers/`](../../README.md)), built from source in CI, and bundled + +`pnputil`-installed by the Windows host [installer](../../README.md). The host feeds it over a shared +memory channel from `crates/punktfunk-host/src/inject/dualsense_windows.rs`. The same UMDF driver also +serves the **DualShock 4** identity per a `device_type` byte the host stamps. -**Load + recognition: DONE.** A self-signed build **loads under Secure Boot ON** and enumerates as a -genuine DualSense HID game controller (`Status: OK`, VID `054C`, 273-byte DualSense report descriptor, -PID `0CE6` via `GET_DEVICE_ATTRIBUTES`). Validated live on the RTX box (`192.168.1.173`, Win11 25H2). +This README captures the driver-authoring lore — the bugs and the signing recipe that make a +self-signed UMDF HID driver actually load. The authoritative build/sign/package flow (CI + Inno Setup) +lives in the [Windows host packaging README](../../README.md). -**Remaining:** the real-game `0x02` adaptive-trigger gate (Cyberpunk 2077 on the interactive desktop → -confirm `[pf-ds] *** OUTPUT ...` in the driver log), then wire into the host (M1+). +## Build workspace -## This is a reference snapshot - -The crate's `Cargo.toml` uses path-deps into `microsoft/windows-drivers-rs` -(`../../crates/wdk{,-sys,-build}`), so it builds **inside a `windows-drivers-rs` checkout's -`examples/` dir**, not standalone in this repo. On the dev box it lives at -`C:\Users\Public\m0\windows-drivers-rs\examples\pf-dualsense`. These files are checked in for -version control / portability of the spike. +This crate builds as a member of the [`packaging/windows/drivers/`](../../drivers) workspace, which +uses the published **crates.io `wdk`/`wdk-sys`/`wdk-build`** (0.4/0.5) — not the old dev-box +`windows-drivers-rs` path-deps. It's a separate cargo workspace from the main tree because driver +crates are cdylibs built with the WDK toolchain on Windows only; it path-deps the shared ABI crate +[`crates/pf-driver-proto`](../../../../crates/pf-driver-proto/README.md). ## Build / sign / install recipe (the one that actually loads) @@ -76,9 +77,10 @@ silently breaks them: zero requests → `EvtIoDeviceControl` never fires → no HID handshake → ~5 s timeout → `CM_PROB_FAILED_START`. Set to `u32::MAX`. -## Known limitations +## Notes -- Uses **statics, not per-device WDF contexts** → only one device instance per WUDFHost works. - Multi-instance needs proper device contexts. -- Port of the WDK `vhidmini2` UMDF2 sample; DualSense identity + 273-byte descriptor + feature blobs - `0x05`/`0x09`/`0x20` from `crates/punktfunk-host/src/inject/dualsense.rs`. +- **Multi-pad** works via `UmdfHostProcessSharing=ProcessSharingDisabled` — each pad gets its own + WUDFHost (so the per-instance statics don't collide), and the driver reads its pad index from the + device Location (`WdfDeviceAllocAndQueryProperty`) to map its own `*-shm-` channel. +- Port of the WDK `vhidmini2` UMDF2 sample; the DualSense identity + 273-byte descriptor + feature + blobs `0x05`/`0x09`/`0x20` come from `crates/punktfunk-host/src/inject/dualsense.rs`. diff --git a/tools/latency-probe/README.md b/tools/latency-probe/README.md new file mode 100644 index 0000000..599f037 --- /dev/null +++ b/tools/latency-probe/README.md @@ -0,0 +1,15 @@ +# latency-probe + +A **glass-to-glass latency** measurement tool (design/implementation-plan §10): it renders a +timestamp/QR on the host, reads it back off the client's capture (or a photodiode, for true photons), +and tracks p50/p99 — so latency regressions are quantifiable rather than felt. + +> **Status: scaffold.** The measurement harness is stubbed out and not yet wired to a live pipeline. +> For latency numbers today, use the per-frame **capture→…→reassembled** percentiles the +> [`probe`](../../clients/probe/README.md) client reports over a real `punktfunk/1` session. + +```sh +cargo run -p latency-probe # from the repo root +``` + +Companion to [`loss-harness`](../loss-harness/README.md) (FEC loss sweep). diff --git a/tools/loss-harness/README.md b/tools/loss-harness/README.md new file mode 100644 index 0000000..b06297e --- /dev/null +++ b/tools/loss-harness/README.md @@ -0,0 +1,16 @@ +# loss-harness + +A **FEC loss-resilience sweep** for [`punktfunk-core`](../../crates/punktfunk-core/README.md). It +drives access units through the in-process loopback at increasing packet-loss rates — for **both** FEC +schemes (GF(2⁸) and GF(2¹⁶)) — and reports how many frames survive. + +It's a pure-software stand-in for `tc netem`: no network, no root, runs anywhere `punktfunk-core` +builds. Use it to sanity-check the FEC before reaching for the real `punktfunk/1` harness (which adds +`tc netem` jitter/reorder on the UDP path). + +```sh +cargo run -p loss-harness # from the repo root +``` + +Part of the measurement tooling (design/implementation-plan §10), alongside +[`latency-probe`](../latency-probe/README.md).