feat(protocol,clients): codec preference negotiation + Linux client decodes per Welcome (Phase 2a)
Adds a client-selectable **preferred codec** and wires the core + ABI + probe + Linux client to
negotiate and decode it. (Windows/Apple/Android follow in 2b.)
**Core:**
- `Hello.preferred_codec` (a single CODEC_* bit, 0 = auto) — a soft hint appended after
`video_codecs`. `resolve_codec(client, host, preferred)` now honors the preference when the host
can also emit it, else falls back to precedence (HEVC > AV1 > H.264). Roundtrip + preference tests.
- `NativeClient::connect` takes `video_codecs` + `preferred_codec`; `NativeClient.codec` exposes the
resolved `Welcome.codec`.
- ABI: `punktfunk_connect_ex7` (adds the two codec params; `ex6` delegates to it advertising
HEVC-only) + `punktfunk_connection_codec` getter + `PUNKTFUNK_CODEC_{H264,HEVC,AV1}` constants
(drift-guarded against the wire values). Header regenerated.
**Host:** passes `hello.preferred_codec` into `resolve_codec`.
**probe:** `--codec h264|hevc|av1|auto` sets the preference (still advertises it can decode all
three); the dump extension already follows the resolved codec.
**Linux client:** advertises the codecs FFmpeg can actually decode (`decodable_codecs()`), threads
the user's `codec` setting as the preference, and builds the decoder — both the software and VAAPI
paths, plus the mid-session VAAPI→software demotion — from the negotiated `Welcome.codec` instead of
hardcoding HEVC. New "Video codec" dropdown in Preferences (Automatic/HEVC/H.264/AV1).
Live-validated on the dev box: probe `--codec hevc` against a software (H.264-only) host resolves to
H.264 (graceful soft-preference fallback), no failure. clippy + core (57) + host (133) tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -38,7 +38,7 @@ On Linux the host **rewrites `WAYLAND_DISPLAY` / `XDG_CURRENT_DESKTOP` / `XDG_RU
|
||||
| `PUNKTFUNK_VIDEO_SOURCE` | `virtual` · `portal` | `virtual` creates a per-client display at the client's exact mode (the normal choice). `portal` captures an existing monitor instead. |
|
||||
| `PUNKTFUNK_ZEROCOPY` | `1` · `0` | GPU zero-copy capture→encode (dmabuf → CUDA → NVENC, or D3D11 on Windows). Leave on; it falls back to a CPU path automatically. |
|
||||
| `PUNKTFUNK_INPUT_BACKEND` | `libei` · `gamescope` · `wlr` · `uinput` | How input is injected. `libei` for GNOME/KDE, `gamescope` for Bazzite/gamescope, `wlr` for Sway/wlroots. Auto-detected with the compositor. |
|
||||
| `PUNKTFUNK_ENCODER` | `auto` · `nvenc` · `vaapi` (Linux) · `amf` · `qsv` · `sw` (Windows) | Encoder backend. `auto` (default) detects the GPU vendor: NVIDIA→NVENC, AMD→VAAPI/AMF, Intel→VAAPI/QSV, else software. |
|
||||
| `PUNKTFUNK_ENCODER` | `auto` · `nvenc` · `vaapi` (Linux) · `amf` · `qsv` (Windows) · `software` | Encoder backend. `auto` (default) detects the GPU vendor: NVIDIA→NVENC, AMD→VAAPI/AMF, Intel→VAAPI/QSV. `software` (aliases `sw`/`openh264`) is the GPU-less H.264 path on both platforms — on Windows `auto` falls back to it when no GPU is found; on Linux it is **explicit-only** (`auto` never picks it). |
|
||||
| `PUNKTFUNK_RENDER_NODE` | path | Linux DRM render node for zero-copy (default `/dev/dri/renderD128`). Set on multi-GPU boxes to pick the right GPU. |
|
||||
|
||||
Resolution and refresh are **not** set here — **the client chooses them.** When a device connects,
|
||||
@@ -76,6 +76,7 @@ picture.
|
||||
| `PUNKTFUNK_10BIT` | `1` | HEVC Main10 / HDR. Honored only when the client also advertises 10-bit. **Windows host only** (the Linux host stays 8-bit). |
|
||||
| `PUNKTFUNK_444` | `1` | Full-chroma HEVC 4:4:4 (Range Extensions) — sharper text/desktop, no chroma loss. **punktfunk/1 native only** (Moonlight stays 4:2:0), HEVC-only, honored only when the client advertises 4:4:4 **and** the GPU supports it (probed; NVENC is the validated path — VAAPI/AMF/QSV decline). Independent of 10-bit. |
|
||||
| `PUNKTFUNK_DSCP` | `1` | Opt-in DSCP / `SO_PRIORITY` QoS tagging on the media sockets. No-op on the wire on Windows without a qWAVE policy. |
|
||||
| `PUNKTFUNK_OH264_THREADS` / `PUNKTFUNK_OH264_GOP` | `N` | Software (openh264) encoder tuning: encode threads (default 2 — latency over throughput) and GOP length (default 0 = encoder-auto). Only relevant with `PUNKTFUNK_ENCODER=software`. |
|
||||
|
||||
## Gamepads
|
||||
|
||||
@@ -151,13 +152,14 @@ good value:
|
||||
|
||||
## Multiple devices at once
|
||||
|
||||
Today the native `punktfunk/1` host (`serve`) streams **one session at a time** — additional clients
|
||||
wait in the accept queue until the active session ends. Each session gets its own virtual display at
|
||||
the client's exact resolution; concurrent native sessions are on the roadmap. (`punktfunk1-host`, the
|
||||
standalone test host, has a `--max-concurrent N` knob — see the [Host CLI](/docs/host-cli) reference —
|
||||
but `serve` does not take that flag.)
|
||||
The native `punktfunk/1` host (`serve`) streams up to **4 sessions at once** by default (an encoder
|
||||
bound); further clients wait in the accept queue until a slot frees up. Each session gets its own
|
||||
virtual display at the client's exact resolution, sharing the host's input/audio/mic services. The
|
||||
limit isn't settable from `serve`'s command line yet — `punktfunk1-host`, the standalone test host,
|
||||
exposes it as `--max-concurrent N` (see the [Host CLI](/docs/host-cli) reference).
|
||||
|
||||
## Codec and FEC
|
||||
|
||||
- The host encodes **HEVC (H.265)** by default; **AV1** is available for clients that support it.
|
||||
- Client and host **negotiate the codec**: **HEVC (H.265)** by default, **AV1** for clients that
|
||||
support it, and **H.264** when the session runs on the GPU-less software encoder.
|
||||
- The native protocol adds forward error correction for lossy links — see `PUNKTFUNK_FEC_PCT` above.
|
||||
|
||||
@@ -67,8 +67,8 @@ punktfunk-host punktfunk1-host --source virtual
|
||||
| `--require-pairing` | Only serve paired devices (implies `--allow-pairing`). |
|
||||
|
||||
`--max-concurrent`, `--allow-pairing`, and `--require-pairing` are **`punktfunk1-host`-only** — `serve` does not
|
||||
accept them. On `serve` you arm pairing from the web console instead, and concurrency is not
|
||||
yet capped from the command line.
|
||||
accept them. On `serve` you arm pairing from the web console instead, and concurrency is fixed at
|
||||
the built-in default (4 sessions) rather than settable from the command line.
|
||||
|
||||
Both `serve` and `punktfunk1-host` advertise the host on the network so clients can discover it. List
|
||||
hosts from another machine with `punktfunk-probe --discover`.
|
||||
|
||||
@@ -28,6 +28,7 @@
|
||||
"forgot-password",
|
||||
"---Project---",
|
||||
"roadmap",
|
||||
"status",
|
||||
"channels",
|
||||
"---Reference---",
|
||||
"[API Reference](/api)"
|
||||
|
||||
@@ -5,7 +5,8 @@ description: What you need to run a punktfunk host — GPU, driver, desktop, and
|
||||
|
||||
## Supported setups
|
||||
|
||||
A punktfunk host runs primarily on a Linux machine with an NVIDIA GPU (a native
|
||||
A punktfunk host runs primarily on a Linux machine with a dedicated GPU — NVIDIA (NVENC) is the
|
||||
most-exercised path, and AMD/Intel GPUs work via VAAPI (a native
|
||||
[Windows host](/docs/windows-host) is also available — see below). These are the Linux desktop
|
||||
environments it supports today, each with its own guide:
|
||||
|
||||
@@ -32,6 +33,10 @@ listed, the host still needs one of these compositor backends to create a virtua
|
||||
not just `nvidia-utils` — without it the compositor can't initialise the GPU and capture fails. Each
|
||||
setup guide installs the right package (e.g. `libnvidia-gl-<version>` on Ubuntu).
|
||||
- **`nvidia-drm modeset=1`** must be enabled (Wayland on NVIDIA needs it). The setup guides cover this.
|
||||
- **AMD / Intel GPUs** encode via **VAAPI** instead (install `mesa-va-drivers` or
|
||||
`intel-media-driver`; validated live on AMD RDNA3). The NVIDIA-specific notes above don't apply
|
||||
there. A GPU-less software H.264 encoder also exists (`PUNKTFUNK_ENCODER=software`), meant as a
|
||||
fallback rather than a daily driver.
|
||||
|
||||
> Consumer GeForce cards historically cap the number of **concurrent** NVENC sessions (a few at once);
|
||||
> workstation cards don't. This only matters if you stream to many devices simultaneously.
|
||||
|
||||
@@ -52,8 +52,8 @@ see [Status & Progress](/docs/status).
|
||||
|
||||
- **Windows client on-glass validation.** The hardware (D3D11VA) decode, HDR present, and GUI are
|
||||
built and ship as a signed MSIX — they just need verification on real GPU hardware.
|
||||
- **Apple stage-2 presenter as the default.** The lower-latency `VTDecompressionSession` →
|
||||
`CAMetalLayer` path is live behind an opt-in flag and graduating to the default.
|
||||
- **Apple presenter polish.** The lower-latency `VTDecompressionSession` → `CAMetalLayer` stage-2
|
||||
path is now the default; HDR brightness and 4:4:4 still need on-glass validation.
|
||||
- **Web console parity.** Surfacing the speed test and bitrate picker the apps already have.
|
||||
- **Windows host hardening.** Broader real-world testing — especially on-glass validation of the
|
||||
AMD (AMF) and Intel (QSV) encode paths, which are CI-green but newer than NVENC.
|
||||
|
||||
@@ -15,7 +15,7 @@ A high-level view of where punktfunk stands. The ordered plan of work is on the
|
||||
| **GameStream host** (Moonlight-compatible) | ✅ working end-to-end; HDR/surround-audio polish open |
|
||||
| **Native protocol** — `punktfunk/1` (QUIC control + UDP data, GF(2¹⁶) Leopard FEC + AES-GCM) | ✅ full session planes, validated live |
|
||||
| **Windows host** (x64) | 🟡 implemented & shipping as a signed installer; NVIDIA/AMD/Intel encode, newer than the Linux host |
|
||||
| **macOS / iOS / iPadOS / tvOS client** | ✅ full client; on-glass stage-2 presenter behind an opt-in flag, becoming the default |
|
||||
| **macOS / iOS / iPadOS / tvOS client** | ✅ full client; on-glass-validated stage-2 presenter is the default |
|
||||
| **Linux client** (`punktfunk-client`, GTK4/libadwaita) | ✅ full client; VAAPI zero-copy decode + software fallback |
|
||||
| **Windows client** (`punktfunk-client`, WinUI 3) | ✅ stage 1 complete; ships as signed MSIX; on-glass hardware validation pending |
|
||||
| **Android client** (phone + Android TV) | ✅ full client; hardware HEVC decode + HDR10 |
|
||||
@@ -35,7 +35,7 @@ host is newer than the Linux host.)
|
||||
**gamescope**, **Mutter**, and **Sway/wlroots**.
|
||||
- **Zero-copy GPU pipeline.** Captured frames stay on the GPU (dmabuf → CUDA → NVENC) with
|
||||
automatic split-encode at very high resolutions. Stable 240 fps at 5120×1440 has been
|
||||
measured.
|
||||
measured. A GPU-less software H.264 encoder exists as an explicit fallback.
|
||||
- **HDR (10-bit), on the Windows host.** An HDR Windows desktop is captured and encoded as HEVC
|
||||
Main10 (BT.2020 PQ) to HDR-capable clients (Windows, Android). Linux hosts stream 8-bit for now —
|
||||
HDR there is blocked upstream at the compositor.
|
||||
@@ -55,7 +55,7 @@ host is newer than the Linux host.)
|
||||
|
||||
| Client | Highlights |
|
||||
|---|---|
|
||||
| **macOS / iOS / iPadOS / tvOS** | VideoToolbox HEVC decode, GameController capture, full DualSense feedback, mDNS discovery, PIN pairing + TOFU, network speed test, latency HUD. Stage-2 presenter (`VTDecompressionSession` → `CAMetalLayer`, ~11 ms p50 capture→present) is built and validated on glass behind an opt-in flag, becoming the default. Ships as one universal TestFlight build / App Store listing. |
|
||||
| **macOS / iOS / iPadOS / tvOS** | VideoToolbox HEVC decode, GameController capture, full DualSense feedback, mDNS discovery, PIN pairing + TOFU, network speed test, latency HUD. Stage-2 presenter (`VTDecompressionSession` → `CAMetalLayer`, ~11 ms p50 capture→present) is validated on glass and is the default (stage 1 remains the fallback when Metal is unavailable). Ships as one universal TestFlight build / App Store listing. |
|
||||
| **Linux** (`punktfunk-client`) | GTK4/libadwaita. FFmpeg decode with VAAPI → DRM-PRIME dmabuf zero-copy (Intel/AMD; software fallback on NVIDIA), PipeWire audio + mic, SDL3 gamepads incl. DualSense, mDNS discovery, PIN pairing + TOFU, speed test. Ships as Flatpak, apt, rpm, and Arch packages. |
|
||||
| **Windows** (`punktfunk-client`) | WinUI 3. D3D11VA zero-copy decode, HDR10, WASAPI audio + mic, SDL3 gamepads incl. DualSense, mDNS discovery, and the full PIN/TOFU trust surface are all implemented. Ships as a signed MSIX (x86_64 + ARM64). **Stage 1 complete; D3D11VA decode, HDR present, and the GUI are pending on-glass validation on real GPU hardware.** |
|
||||
| **Android** (phone + Android TV) | Kotlin app with a Rust core over JNI. NDK `AMediaCodec` hardware HEVC decode + HDR10 (Main10/BT.2020 PQ), Opus/Oboe audio + mic, gamepad input with rumble/HID feedback, mDNS discovery, PIN pairing + TOFU (Keystore identity), live stats HUD, and D-pad/controller focus navigation for TV. Ships to the Google Play Internal Testing track. |
|
||||
@@ -113,7 +113,6 @@ See the [Roadmap](/docs/roadmap) for the ordered list. Near-term:
|
||||
|
||||
- **True glass-to-glass latency** — combine the client present-stamp (decode → present)
|
||||
with the host render → capture term for a complete end-to-end number.
|
||||
- **Make the Apple stage-2 presenter the default** after a few more resolution/HDR checks.
|
||||
- **On-glass validation of the Windows client** (D3D11VA decode, HDR present, GUI) on real
|
||||
GPU hardware.
|
||||
- **gamescope multi-user isolation** — per-session input/audio so concurrent sessions can
|
||||
|
||||
@@ -96,5 +96,3 @@ The check follows the [channel](/docs/channels) you installed from: a plugin ins
|
||||
|
||||
The plugin source lives in
|
||||
[`clients/decky`](https://git.unom.io/unom/punktfunk/src/branch/main/clients/decky/README.md).
|
||||
</content>
|
||||
</invoke>
|
||||
|
||||
@@ -69,9 +69,9 @@ Then log out and back in. On other distros this is `sudo usermod -aG input $USER
|
||||
clients' [speed test](/docs/configuration#bitrate) picks a safe value; with Moonlight, set it
|
||||
manually.
|
||||
- Prefer a **wired** connection or 5 GHz Wi-Fi between host and client.
|
||||
- Streaming to **many devices at once** shares the GPU encoder. The production host
|
||||
(`serve`) handles one native session at a time, with extra clients queued; heavy load is
|
||||
usually bitrate-bound, so lower the bitrate first.
|
||||
- Streaming to **many devices at once** shares the GPU encoder. The host serves several
|
||||
concurrent native sessions (up to 4 by default); heavy load is usually bitrate-bound, so
|
||||
lower the bitrate first.
|
||||
|
||||
## Still stuck?
|
||||
|
||||
|
||||
Reference in New Issue
Block a user