enricobuehler
12843fe253
Adds a client-selectable **preferred codec** and wires the core + ABI + probe + Linux client to
negotiate and decode it. (Windows/Apple/Android follow in 2b.)
**Core:**
- `Hello.preferred_codec` (a single CODEC_* bit, 0 = auto) — a soft hint appended after
`video_codecs`. `resolve_codec(client, host, preferred)` now honors the preference when the host
can also emit it, else falls back to precedence (HEVC > AV1 > H.264). Roundtrip + preference tests.
- `NativeClient::connect` takes `video_codecs` + `preferred_codec`; `NativeClient.codec` exposes the
resolved `Welcome.codec`.
- ABI: `punktfunk_connect_ex7` (adds the two codec params; `ex6` delegates to it advertising
HEVC-only) + `punktfunk_connection_codec` getter + `PUNKTFUNK_CODEC_{H264,HEVC,AV1}` constants
(drift-guarded against the wire values). Header regenerated.
**Host:** passes `hello.preferred_codec` into `resolve_codec`.
**probe:** `--codec h264|hevc|av1|auto` sets the preference (still advertises it can decode all
three); the dump extension already follows the resolved codec.
**Linux client:** advertises the codecs FFmpeg can actually decode (`decodable_codecs()`), threads
the user's `codec` setting as the preference, and builds the decoder — both the software and VAAPI
paths, plus the mid-session VAAPI→software demotion — from the negotiated `Welcome.codec` instead of
hardcoding HEVC. New "Video codec" dropdown in Preferences (Automatic/HEVC/H.264/AV1).
Live-validated on the dev box: probe `--codec hevc` against a software (H.264-only) host resolves to
H.264 (graceful soft-preference fallback), no failure. clippy + core (57) + host (133) tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
107 lines
6.7 KiB
Markdown
107 lines
6.7 KiB
Markdown
---
|
||
title: "Roadmap"
|
||
description: "What's shipped, what's in progress, and what's next for punktfunk."
|
||
---
|
||
|
||
A quick map of where punktfunk is today and where it's heading. For the detailed, dated changelog,
|
||
see [Status & Progress](/docs/status).
|
||
|
||
**Legend:** ✅ shipped · 🟡 in progress · 🔭 planned · ⛔ blocked upstream
|
||
|
||
## At a glance
|
||
|
||
| Area | |
|
||
|---|---|
|
||
| Protocol core — FEC · crypto · C ABI | ✅ |
|
||
| GameStream host (works with Moonlight) | ✅ |
|
||
| Native `punktfunk/1` protocol | ✅ |
|
||
| Linux host (KWin · GNOME · gamescope · Sway) | ✅ |
|
||
| Windows host (NVIDIA) | ✅ beta |
|
||
| Apple client (macOS · iOS · iPadOS · tvOS) | ✅ |
|
||
| Linux client (GTK4) | ✅ |
|
||
| Android client (phone · TV) | ✅ |
|
||
| Windows client | 🟡 |
|
||
| Web console + pairing | ✅ |
|
||
| Concurrent sessions (shared desktop) | ✅ |
|
||
| Network speed test + bitrate | ✅ |
|
||
| HDR / 10-bit streaming | ✅ Windows host · ⛔ Linux host |
|
||
| Sub-frame pipelining (latency) | 🔭 |
|
||
|
||
## ✅ Shipped
|
||
|
||
- **The host, two ways.** The lower-latency native [`punktfunk/1`](/docs/how-it-works) protocol (QUIC
|
||
control + UDP data with GF(2¹⁶) Leopard FEC + AES-GCM) — the secure default — and, opt-in via
|
||
`serve --gamestream`, a GameStream host any [Moonlight](/docs/moonlight) client can use. Both run
|
||
from one process.
|
||
- **Native-resolution virtual displays** on Linux across KWin, GNOME/Mutter, gamescope, and
|
||
Sway/wlroots, with a fully zero-copy GPU path to NVENC (stable 240 fps at 5120×1440).
|
||
- **A native Windows host** (x64; NVIDIA/AMD/Intel encode) — a signed installer with secure-desktop capture and a
|
||
bundled virtual-display driver, and the only host that can stream **HDR** (10-bit BT.2020 PQ,
|
||
captured from an HDR Windows desktop and encoded as HEVC Main10). See
|
||
[Windows Host](/docs/windows-host). *(Beta — newer than the Linux host.)*
|
||
- **Clients on every platform** — native apps for **Apple** (macOS, iOS, iPadOS, tvOS), **Linux**,
|
||
**Android** (phone + TV), and **Windows**, each with hardware decode, controllers including
|
||
DualSense, audio + mic, and automatic host discovery. See [Clients](/docs/clients).
|
||
- **Secure by default** — SPAKE2 PIN pairing with pinned reconnects, one-click delegated approval from
|
||
the web console, and mDNS LAN auto-discovery.
|
||
- **Tuned for latency** — concurrent sessions (stream one desktop to several devices at once),
|
||
mid-stream resolution renegotiation, a cross-machine clock-skew handshake, a 1 Gbps+ data plane, and
|
||
an in-app network speed test that informs the bitrate picker.
|
||
|
||
## 🟡 In progress
|
||
|
||
- **Windows client on-glass validation.** The hardware (D3D11VA) decode, HDR present, and GUI are
|
||
built and ship as a signed MSIX — they just need verification on real GPU hardware.
|
||
- **Apple presenter polish.** The lower-latency `VTDecompressionSession` → `CAMetalLayer` stage-2
|
||
path is now the default; HDR brightness and 4:4:4 still need on-glass validation.
|
||
- **Web console parity.** Surfacing the speed test and bitrate picker the apps already have.
|
||
- **Windows host hardening.** Broader real-world testing — especially on-glass validation of the
|
||
AMD (AMF) and Intel (QSV) encode paths, which are CI-green but newer than NVENC.
|
||
|
||
## 🔭 Planned
|
||
|
||
- **Magic multi-user support.** Map a connecting client to a real user account on the host and log
|
||
them in automatically. The client picks an identity — e.g. an **Apple TV profile** — which maps to
|
||
an available host profile (likely behind a second per-user auth layer); on connect to an idle host,
|
||
the user lands in *their own* desktop/session, signed in, without touching the host. Turns one box
|
||
into a personal, profile-aware streaming target for a household.
|
||
- **Surround & spatial audio.** Today every path is stereo end to end (the Linux host already encodes
|
||
5.1/7.1 via multistream Opus, but no client renders it yet). Near term: carry a **7.1 channel bed** —
|
||
multichannel capture, surround Opus on every host, and clients that render more than two channels
|
||
(macOS can then spatialize the bed to head-tracked AirPods audio). The moonshot is **object-based
|
||
spatial audio**: native-Windows Atmos/DTS:X object capture is blocked by a closed renderer API, but
|
||
**Proton already routes game audio objects through Wine's `ISpatialAudioClient`** — where it currently
|
||
*discards* the dynamic/height objects and their 3D positions. Finishing that rendering would give
|
||
every Proton gamer real spatial sound, and tunnelling the objects + positions to the client would let
|
||
punktfunk spatialize them *on the client* — head-tracked remote Atmos-style audio that no streaming
|
||
stack does today.
|
||
- **Sub-frame pipelining.** Overlap encode and transmit within a single frame (a direct NVENC slice
|
||
path) — the next big latency lever at high resolutions.
|
||
- **True glass-to-glass latency** measured end to end (capture → on-screen present).
|
||
- **gamescope multi-user isolation.** Per-session input and audio so concurrent clients are fully
|
||
independent desktops (the shared-desktop case already works).
|
||
- **Peer-approved pairing.** Approve a new device from an already-paired device's own app.
|
||
- **WAN / anywhere access.** Reach a host beyond the LAN — NAT traversal (ICE/STUN/TURN) plus a
|
||
self-hostable relay for when no direct path can be punched, and **QUIC connection migration** so a
|
||
client roams between networks (Wi-Fi ↔ cellular) without dropping the session. The control plane is
|
||
already QUIC over UDP, so this is mostly signalling and relay rather than a protocol change.
|
||
- **VRR / adaptive-sync passthrough.** Variable refresh end to end — the host renders at a variable
|
||
rate and the client presents with tearing-control/VRR instead of a fixed cadence, for tear- and
|
||
judder-free gaming. Builds on the client's presentation-feedback path and the per-session virtual
|
||
outputs.
|
||
- **Desktop quality-of-life.** The essentials that make remote *work* pleasant, each a new side plane
|
||
over the existing QUIC datagram channel: bidirectional rich **clipboard sync** (text and images),
|
||
**multi-monitor streaming** (present the host's several outputs as separate client windows), and
|
||
**virtual-webcam redirection** (the client's camera shows up as a webcam on the host, so video calls
|
||
run on the remote machine).
|
||
|
||
## ⛔ Parked / blocked
|
||
|
||
- **HDR / 10-bit on the *Linux* host.** HDR streaming already works from a
|
||
[Windows host](/docs/windows-host) to an HDR-capable client (Windows, Android). On Linux it's
|
||
blocked upstream — no shipping compositor emits a 10-bit/HDR capture stream yet — and ready the
|
||
moment one does.
|
||
- **Advanced DualSense voice-coil haptics.** Scoped and shelved (it rides the controller's USB audio
|
||
interface, with near-zero game support on Linux). Adaptive triggers, rumble, and the lightbar
|
||
already ship.
|