feat(host): HDR Vulkan layer so Vulkan games get HDR on the virtual display
ci / web (push) Failing after 22s
windows-host / package (push) Failing after 4m16s
ci / rust (push) Failing after 4m56s
ci / docs-site (push) Successful in 1m7s
android / android (push) Successful in 9m19s
ci / bench (push) Successful in 4m47s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Failing after 3s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
docker / deploy-docs (push) Has been skipped
deb / build-publish (push) Failing after 6m29s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 7m4s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 7m17s
apple / swift (push) Successful in 1m13s
apple / screenshots (push) Successful in 5m27s
ci / web (push) Failing after 22s
windows-host / package (push) Failing after 4m16s
ci / rust (push) Failing after 4m56s
ci / docs-site (push) Successful in 1m7s
android / android (push) Successful in 9m19s
ci / bench (push) Successful in 4m47s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Failing after 3s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
docker / deploy-docs (push) Has been skipped
deb / build-publish (push) Failing after 6m29s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 7m4s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 7m17s
apple / swift (push) Successful in 1m13s
apple / screenshots (push) Successful in 5m27s
NVIDIA/AMD Vulkan ICDs refuse to *advertise* an HDR color space for a surface on an
IddCx indirect/virtual display, so Vulkan games (Doom: The Dark Ages, id Tech, Indiana
Jones, …) report "device does not support HDR" — even though Windows HDR, DWM compose,
and the client PQ stream all work, and the ICD happily *accepts + presents* a forced HDR
swapchain there. The whole gap is enumeration; the community (Apollo/Sunshine/VDD) wrote
this off as kernel-side / unfixable.
Add VK_LAYER_PUNKTFUNK_hdr_inject (packaging/windows/pf-vkhdr-layer/): a standalone
cdylib Vulkan implicit layer that appends {A2B10G10R10, HDR10_ST2084} + {RGBA16F, scRGB}
to vkGetPhysicalDeviceSurfaceFormats[2]KHR (no need to hook vkCreateSwapchainKHR — the
ICD doesn't validate the color space there). Self-gated on the surface monitor's actual
advanced-color state (DisplayConfig GET_ADVANCED_COLOR_INFO), so it is a complete no-op
on SDR sessions and real monitors (dedup). Always-on (registry-discovered) so it works
regardless of how a game is launched — env-scoping silently fails for already-running
Steam. Escape hatches: DISABLE_PF_VKHDR, PF_VKHDR_EXCLUDE, and a built-in kernel-anti-
cheat denylist.
The installer builds/signs/stages it and registers it under
HKLM64\SOFTWARE\Khronos\Vulkan\ImplicitLayers (opt-out "Install the HDR Vulkan layer"
task); windows-host CI fmt+clippy-gates it (msvc-only FFI).
Live-validated on the RTX box: Doom: The Dark Ages enables HDR over the pf-vdisplay
virtual display.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,170 @@
|
||||
# Windows native client — bootstrap handoff
|
||||
|
||||
A handoff for an agent picking up the **native Windows punktfunk/1 client**. The host side is done
|
||||
and live-validated on a real RTX 4090; the client is the remaining piece. This doc is the concrete
|
||||
starting point: the locked decisions, the reference code to port, the stack swaps, the dev loop, and
|
||||
the gotchas. Read it top to bottom, then start at **Phase 1** (de-risk Reactor first).
|
||||
|
||||
## Status — WinUI 3 client landed (2026-06-15)
|
||||
|
||||
The client is implemented in `clients/windows` (binary `punktfunk-client`) and is
|
||||
**build + clippy + fmt green on `x86_64-pc-windows-msvc` and `aarch64-pc-windows-msvc`** (the ARM64
|
||||
target cross-compiled off the one x64 runner — see `windows.yml`; signed MSIX for both arches via
|
||||
`windows-msix.yml`). It is the **WinUI 3** client this doc planned: native chrome (host list,
|
||||
settings, in-app SPAKE2 PIN pairing) + the video on a **`SwapChainPanel`**, all in pure Rust.
|
||||
|
||||
- **Reactor is viable after all — it is what we use.** The locked decision held. windows-rs
|
||||
[PR #4499](https://github.com/microsoft/windows-rs/pull/4499) (merged 2026-06-01) added a
|
||||
`SwapChainPanel` widget to **`windows-reactor`** with `set_swap_chain` over
|
||||
`CreateSwapChainForComposition` — so a DXGI presenter *can* be hosted. (An earlier read that Reactor
|
||||
had no swapchain hatch was wrong/stale.) The UI is a declarative React-like tree
|
||||
(`App::new().render(app)`, `use_state`/`use_resource`/`use_effect` hooks, `list_view`/`text_box`/
|
||||
`combo_box`/`content_dialog`/`button`/`ToggleSwitch`); the video page is `swap_chain_panel()
|
||||
.on_ready(|p| p.set_swap_chain(&sc))` driven by `on_rendering`. **`present.rs`** owns the D3D11
|
||||
composition swapchain (WARP fallback, runtime shaders, Contain-fit) — the same renderer, bound to
|
||||
the panel instead of an HWND.
|
||||
- **windows-reactor is unpublished** (`version 0.0.0`) and fast-moving — depend on it as a **git dep
|
||||
pinned to a commit** (`b4129fcc`), and pin the `windows` crate to the **same commit** so the
|
||||
`IDXGISwapChain1` you pass to `set_swap_chain` satisfies reactor's `windows_core::Interface`. Its
|
||||
`build.rs` downloads the Windows App SDK NuGets (Foundation/Interactive/Runtime) and stages the
|
||||
bootstrap DLL + `resources.pri` next to the exe; it **`.unwrap()`s `CARGO_WORKSPACE_DIR`**, so set
|
||||
it in the build env (`CARGO_WORKSPACE_DIR=C:\Users\Public\punktfunk`). It writes `/temp` + `/winmd`
|
||||
to the workspace root (gitignored). The App SDK runtime must be installed to *run*.
|
||||
- **Stream input is Win32 low-level hooks**, not XAML: reactor exposes only keyboard *accelerators* +
|
||||
pointer *button-state* (no raw key-down/up, no pointer position, no wheel), insufficient for a game
|
||||
stream. `input.rs` installs `WH_KEYBOARD_LL`/`WH_MOUSE_LL` on the stream page (uninstalled on exit),
|
||||
maps the pointer through the window client rect, sends native VK + abs mouse + wheel, with a
|
||||
Ctrl+Alt+Shift+Q capture toggle. (A future alternative: generate `Microsoft.UI.Xaml.UIElement`
|
||||
bindings from the staged winmd and subscribe to `KeyDown`/`PointerMoved` — scoped to the panel.)
|
||||
- **Build gotcha:** `CARGO_HOME` must be on an **ASCII path** (`C:\Users\Public\.cargo`). SDL3's
|
||||
`build-from-source` PCH embeds the registry source path; the `ü` in the dev box's username makes
|
||||
MSVC fail (`MSB8084` / `C4828`).
|
||||
- **Still pending:** **on-glass validation** — the dev VM is headless / SSH Session 0, so the WinUI
|
||||
window can't show there; validate over RDP or on the RTX box. Then **D3D11VA hardware decode** +
|
||||
**10-bit/HDR present**, RAWINPUT relative-mouse pointer-lock, and a per-host speed test in the UI.
|
||||
|
||||
## What we're building
|
||||
|
||||
A native Windows client that connects to a punktfunk/1 host (`serve` / `punktfunk1-host`), decodes
|
||||
HEVC, presents it low-latency, plays Opus audio, and captures local mouse/keyboard/gamepad to send
|
||||
back — i.e. the Windows analogue of the **GTK4 Linux client** (`clients/linux`),
|
||||
which is the architectural template. The Windows client is close to a 1:1 port of the Linux client
|
||||
with the platform layers swapped.
|
||||
|
||||
## Locked decisions (from the Windows-host/client plan, `docs/windows-host.md` + project memory)
|
||||
|
||||
- **Pure Rust.** `windows-rs` + **Windows App SDK "Reactor"** (WinUI 3 from Rust, merged windows-rs
|
||||
PR #4479). No C++/C#. De-risk Reactor + `SwapChainPanel` FIRST — it's the only novel/uncertain
|
||||
piece; everything else is a known-good port.
|
||||
- **Links `punktfunk-core` directly** (Cargo path dep, `features = ["quic"]`) — **no C ABI**, exactly
|
||||
like the GTK client. `NativeClient` is already `Sync` (mutexed plane receivers), so it drops into a
|
||||
UI app cleanly. The C ABI (`punktfunk_connect` + `next_au`/`next_audio`/`next_rumble`/`next_hidout`/
|
||||
`send_input`/`send_rich_input`) is the *Apple* path; the native Rust clients call
|
||||
`crates/punktfunk-core/src/client.rs` (`NativeClient`) methods directly.
|
||||
- **Video widget = WinUI 3 `SwapChainPanel`** (built-in), fed a D3D11 swapchain via
|
||||
`ISwapChainPanelNative::SetSwapChain`.
|
||||
- **Decode = FFmpeg-next + D3D11VA** (HEVC; **Main10** for 10-bit/HDR — see below).
|
||||
- **Audio playback = WASAPI render** + Opus decode (`opus` crate, vendors libopus via cmake; set
|
||||
`CMAKE_POLICY_VERSION_MINIMUM=3.5`).
|
||||
- **Input capture→send**: the client captures LOCAL input and sends it. Mouse (abs + relative) +
|
||||
keyboard via the **inverse VK table** (port `keymap.rs`); gamepad via **SDL3** (already a workspace
|
||||
dep, cross-platform) → `NativeClient::send_input`/`send_rich_input`. (`SendInput`/`ViGEm` are
|
||||
HOST-side injection — not used by the client.)
|
||||
- **Discovery = `mdns-sd`** (cross-platform, browses `_punktfunk._udp`).
|
||||
- **Trust = shared client identity + SPAKE2 PIN pairing + TOFU** (port `trust.rs`; same identity
|
||||
files/logic as the other native clients).
|
||||
|
||||
## The reference: `clients/linux/src/`
|
||||
|
||||
Port these files (near 1:1; only the platform layers change):
|
||||
|
||||
| Linux file | Role | Windows swap |
|
||||
|---|---|---|
|
||||
| `main.rs` / `app.rs` | app shell, lifecycle | WinUI 3 `App`/`Window` via Reactor |
|
||||
| `ui_hosts.rs` | host list / connect screen | WinUI 3 page |
|
||||
| `ui_settings.rs` | settings | WinUI 3 page |
|
||||
| `ui_stream.rs` | the streaming view | WinUI 3 page hosting `SwapChainPanel` |
|
||||
| `video.rs` | FFmpeg decode + present | FFmpeg **D3D11VA** → D3D11 swapchain in `SwapChainPanel` |
|
||||
| `audio.rs` | Opus decode + playback | **WASAPI render** (was PipeWire) |
|
||||
| `session.rs` | `NativeClient` connect + plane pumps | **reuse almost verbatim** (core is cross-platform) |
|
||||
| `trust.rs` | identity, PIN, TOFU | **reuse almost verbatim** |
|
||||
| `discovery.rs` | mDNS browse | **reuse verbatim** (`mdns-sd`) |
|
||||
| `keymap.rs` | inverse VK table | reuse; Windows VK is the native source so this is *simpler* |
|
||||
| `gamepad.rs` | SDL3 pad capture + rumble/feedback | **reuse almost verbatim** (SDL3 is cross-platform) |
|
||||
|
||||
`session.rs`, `trust.rs`, `discovery.rs`, `keymap.rs`, `gamepad.rs` are mostly platform-neutral
|
||||
(they touch `punktfunk-core` + SDL3 + mdns, all cross-platform) — expect to reuse them with minimal
|
||||
changes. The real work is `video.rs` (D3D11VA + swapchain), `audio.rs` (WASAPI), and the WinUI shell.
|
||||
|
||||
## 10-bit + HDR (NEW — landed this session, the client MUST handle it)
|
||||
|
||||
The host now negotiates and emits **HEVC Main10 + BT.2020 PQ HDR10** when the captured desktop is
|
||||
HDR (and 10-bit SDR Main10 when negotiated). The Apple client already does the matching present; the
|
||||
Windows client should mirror it:
|
||||
|
||||
- **Advertise caps** in the `Hello`: `video_caps = VIDEO_CAP_10BIT | VIDEO_CAP_HDR`
|
||||
(`crates/punktfunk-core/src/quic.rs`). The host enables 10-bit only if the client advertised it.
|
||||
(The native-client connector in `client.rs` currently hardcodes `video_caps: 0` with a TODO —
|
||||
thread the real caps through when you wire decode; or detect HDR purely in-band, see next.)
|
||||
- **Detect HDR in-band** from the HEVC VUI (transfer characteristics = SMPTE ST 2084 / PQ), exactly
|
||||
like the Apple client's `VideoDecoder.isHDRFormat` (`clients/apple/Sources/PunktfunkKit/`). This
|
||||
handles a mid-session HDR toggle without renegotiation. `Welcome.bit_depth` (8/10) is also available.
|
||||
- **Decode** Main10 → **P010** (10-bit) via D3D11VA.
|
||||
- **Present HDR**: swapchain in `DXGI_FORMAT_R10G10B10A2_UNORM` (or `R16G16B16A16_FLOAT`),
|
||||
`IDXGISwapChain3::SetColorSpace1(DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020)` +
|
||||
`SetHDRMetaData` for HDR10; the host's stream is BT.2020 PQ, so present PQ. For SDR, the existing
|
||||
`DXGI_FORMAT_B8G8R8A8_UNORM` + BT.709 path. (The host-side HDR conversion math is in
|
||||
`crates/punktfunk-host/src/capture/dxgi.rs` `HDR_PS`/`HdrConverter` if you need the inverse.)
|
||||
|
||||
## Dev boxes
|
||||
|
||||
- **No-GPU dev box (UI + connect + software decode):** `ssh "Enrico Bühler"@192.168.1.57` — Win11 Pro
|
||||
25H2 (build 26200), QEMU Q35, 8 vCPU/12 GB, **no working GPU** (so no NVENC, no D3D11VA hardware
|
||||
decode — use FFmpeg software decode here; this box is for UI/connect/protocol work). Has Rust 1.96
|
||||
MSVC, VS 2026 + VC tools + Win SDK, Win App Runtime 2.2, SudoVDA + Parsec VDD.
|
||||
- **Real-GPU box (HDR / hardware decode / end-to-end):** `ssh "Enrico Bühler"@192.168.1.174` — Win11,
|
||||
RTX 4090, runs the host. Use it to test the client against a live HDR host.
|
||||
|
||||
### Dev-loop gotchas (both boxes)
|
||||
|
||||
- **Build under an ASCII path** (`C:\Users\Public\…`). The username "Enrico Bühler" has a `ü` → MSVC
|
||||
`LNK1201` PDB-write failure under `~/Developer`.
|
||||
- **Toolchain gaps:** `winget install NASM.NASM Kitware.CMake LLVM.LLVM` (aws-lc-rs on the quic path,
|
||||
ffmpeg-sys needs libclang).
|
||||
- **`CMAKE_POLICY_VERSION_MINIMUM=3.5`** in the build env (CMake 4 rejects libopus's old minimum).
|
||||
- **File transfer = `sftp`** (scp is broken under the PowerShell DefaultShell):
|
||||
`printf 'put %s /C:/Users/Public/REL/PATH\n' LOCAL | sftp -b - "Enrico Bühler@192.168.1.57"` —
|
||||
note the **leading slash** `/C:/…`. Let the VM regenerate its own `Cargo.lock` (don't transfer it).
|
||||
- **Windows clippy is stricter** than Linux CI and `cfg(windows)` code is excluded from Linux CI →
|
||||
run `cargo clippy -p punktfunk-client-windows -- -D warnings` ON THE VM before committing.
|
||||
- Work on `main`; fetch+merge `origin/main` before pushing.
|
||||
|
||||
## Suggested phased plan
|
||||
|
||||
1. **De-risk Reactor (do this first).** A windows-rs Reactor (WinUI 3) hello-world that hosts a
|
||||
`SwapChainPanel` and presents a cleared D3D11 swapchain into it. Confirm the windows-rs Reactor
|
||||
version/API (PR #4479) and `ISwapChainPanelNative::SetSwapChain` interop. If Reactor proves too
|
||||
raw, the fallback is `winit` + a child HWND swapchain, but try Reactor first per the decision.
|
||||
2. **Crate scaffold.** `clients/windows`, `[target.'cfg(windows)'.dependencies]`:
|
||||
`punktfunk-core { path, features=["quic"] }`, `windows`, the Reactor crate, `ffmpeg-next`, `opus`,
|
||||
`sdl3`, `mdns-sd`, `anyhow`, `tracing`. Mirror `clients/linux/Cargo.toml`.
|
||||
3. **Connect + control plane.** Port `session.rs` + `trust.rs`; validate headless against the 4090
|
||||
box (`punktfunk1-host`/`serve`) — handshake, PIN/TOFU, plane counters — before any UI/decode.
|
||||
4. **Decode + present.** FFmpeg D3D11VA → `SwapChainPanel`. SDR (8-bit BGRA) first, then **P010 +
|
||||
HDR colorspace** (see the HDR section).
|
||||
5. **Audio.** WASAPI render + Opus decode (port `audio.rs`).
|
||||
6. **Input.** Mouse + keyboard capture→send (port `keymap.rs`), gamepad via SDL3 (port `gamepad.rs`),
|
||||
feedback from `next_rumble`/`next_hidout`.
|
||||
7. **Discovery + UI.** Port `discovery.rs` + `ui_hosts.rs` + `ui_settings.rs` to WinUI pages.
|
||||
|
||||
## Key references
|
||||
|
||||
- **Template:** `clients/linux/src/*` (the client to port).
|
||||
- **Apple HDR present** (the pattern to mirror): `clients/apple/Sources/PunktfunkKit/{VideoDecoder,
|
||||
MetalVideoPresenter,Stage2Pipeline}.swift` — in-band PQ detection, P010 decode, EDR present.
|
||||
- **Core client API:** `crates/punktfunk-core/src/client.rs` (`NativeClient`).
|
||||
- **Protocol:** `crates/punktfunk-core/src/quic.rs` (`Hello.video_caps`, `Welcome.bit_depth`,
|
||||
`VIDEO_CAP_10BIT`/`VIDEO_CAP_HDR`).
|
||||
- **Full Windows plan + SudoVDA/host details:** `docs/windows-host.md`.
|
||||
- **Host HDR conversion (for the inverse math):** `crates/punktfunk-host/src/capture/dxgi.rs`
|
||||
(`HDR_PS`, `HdrConverter`) + `crates/punktfunk-host/src/encode/nvenc.rs` (BT.2020/PQ VUI).
|
||||
Reference in New Issue
Block a user