enricobuehler
7b99b41ede
Much of design/ described work that has since shipped. Trim each doc to
its durable rationale + still-open items (the code is the source of truth
for shipped detail; git history holds the full originals).
- Shipped plans -> status stubs: stats-capture, gamestream-host-plan,
apple-stage2-presenter, windows-service.
- Trimmed completed-out / open-kept: implementation-plan, hdr-pipeline,
host-latency, gpu-contention (fixed stale status table), game-library,
linux-setup (fixed m0->spike + stale zero-copy claim),
session-aware-host-followups, windows-client-bootstrap,
windows-dualsense-{scoping,game-detection}, windows-virtual-display,
security-review (per-finding status table; #12 still open),
apollo-comparison (shipped backlog collapsed to one-liners).
- Windows-host cluster consolidated: windows-host.md -> redirect into
windows-host-rewrite.md (whose stale scorecard is corrected -- goal1 is
merged, M4 done); windows-secure-desktop.md archived (now a fallback
behind IDD-push primary).
- Kept evergreen: ci.md, gamescope-multiuser.md, windows-build-and-packaging.md.
- New design/README.md: per-doc status table + consolidated open-items
roll-up so nothing is tracked in only one buried doc.
- Repoint 5 code comments to the archived secure-desktop doc path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
107 lines
7.7 KiB
Markdown
107 lines
7.7 KiB
Markdown
# Windows native client — bootstrap handoff
|
|
|
|
> **Status:** SHIPPED — `clients/windows` (binary `punktfunk-client`), WinUI 3 via `windows-reactor`;
|
|
> commits `0a3b92d..0cc36fa`. Build + clippy + fmt green on `x86_64-pc-windows-msvc` and
|
|
> `aarch64-pc-windows-msvc` (ARM64 cross-compiled off the one x64 runner; signed MSIX for both arches
|
|
> via `windows-msix.yml`). This doc is trimmed to design rationale, the HDR reference, hard-won
|
|
> gotchas, and open items. The shipped source under `clients/windows/src/` is the truth.
|
|
|
|
The native Windows punktfunk/1 client connects to a host (`serve` / `punktfunk1-host`), decodes HEVC,
|
|
presents it low-latency on a `SwapChainPanel`, plays Opus audio, and captures local
|
|
mouse/keyboard/gamepad to send back — the Windows analogue of the GTK4 Linux client
|
|
(`clients/linux`), which was the architectural template. The locked decisions below are the durable
|
|
"why"; the HDR section is the evergreen present reference.
|
|
|
|
## Locked decisions (the "why")
|
|
|
|
- **Pure Rust.** `windows-rs` + **Windows App SDK "Reactor"** (WinUI 3 from Rust, merged windows-rs
|
|
PR #4479). No C++/C#. Reactor + `SwapChainPanel` was the only novel/uncertain piece and was
|
|
de-risked first; everything else is a known-good port of the Linux client.
|
|
- Reactor is viable: windows-rs [PR #4499](https://github.com/microsoft/windows-rs/pull/4499)
|
|
(merged 2026-06-01) added a `SwapChainPanel` widget to `windows-reactor` with `set_swap_chain`
|
|
over `CreateSwapChainForComposition`, so a DXGI presenter *can* be hosted. (An earlier read that
|
|
Reactor had no swapchain hatch was wrong/stale.) The UI is a declarative React-like tree
|
|
(`App::new().render(app)`, `use_state`/`use_resource`/`use_effect` hooks); the video page is
|
|
`swap_chain_panel().on_ready(|p| p.set_swap_chain(&sc))` driven by `on_rendering`.
|
|
- **Links `punktfunk-core` directly** (Cargo path dep, `features = ["quic"]`) — **no C ABI**, exactly
|
|
like the GTK client, *unlike* the Apple path. `NativeClient` is already `Sync` (mutexed plane
|
|
receivers), so it drops into a UI app cleanly. The C ABI (`punktfunk_connect` + `next_au`/
|
|
`next_audio`/`next_rumble`/`next_hidout`/`send_input`/`send_rich_input`) is the *Apple* path; the
|
|
native Rust clients call `crates/punktfunk-core/src/client.rs` (`NativeClient`) methods directly.
|
|
- **Video widget = WinUI 3 `SwapChainPanel`** (built-in), fed a D3D11 swapchain via
|
|
`ISwapChainPanelNative::SetSwapChain`. `present.rs` owns the D3D11 composition swapchain (WARP
|
|
fallback, runtime shaders, Contain-fit) — the same renderer, bound to the panel instead of an HWND.
|
|
- **Decode = FFmpeg-next + D3D11VA** (HEVC; **Main10** for 10-bit/HDR — see below).
|
|
- **Audio playback = WASAPI render** + Opus decode (`opus` crate, vendors libopus via cmake).
|
|
- **Input capture→send**: the client captures LOCAL input and sends it. Mouse (abs + relative) +
|
|
keyboard via the **inverse VK table** (Windows VK is the native source, so simpler than Linux);
|
|
gamepad via **SDL3** (already a workspace dep, cross-platform) → `NativeClient::send_input`/
|
|
`send_rich_input`. (`SendInput`/`ViGEm` are HOST-side injection — not used by the client.)
|
|
- **Stream input is Win32 low-level hooks**, not XAML: reactor exposes only keyboard *accelerators*
|
|
+ pointer *button-state* (no raw key-down/up, no pointer position, no wheel), insufficient for a
|
|
game stream. `input.rs` installs `WH_KEYBOARD_LL`/`WH_MOUSE_LL` on the stream page (uninstalled
|
|
on exit), maps the pointer through the window client rect, sends native VK + abs mouse + wheel,
|
|
with a Ctrl+Alt+Shift+Q capture toggle. (A future alternative: generate
|
|
`Microsoft.UI.Xaml.UIElement` bindings from the staged winmd and subscribe to `KeyDown`/
|
|
`PointerMoved` — scoped to the panel.)
|
|
- **Discovery = `mdns-sd`** (cross-platform, browses `_punktfunk._udp`).
|
|
- **Trust = shared client identity + SPAKE2 PIN pairing + TOFU** (`trust.rs`; same identity
|
|
files/logic as the other native clients).
|
|
|
|
## 10-bit + HDR (the present reference)
|
|
|
|
The host negotiates and emits **HEVC Main10 + BT.2020 PQ HDR10** when the captured desktop is HDR
|
|
(and 10-bit SDR Main10 when negotiated). The Windows client mirrors the Apple present:
|
|
|
|
- **Advertise caps** in the `Hello`: `video_caps = VIDEO_CAP_10BIT | VIDEO_CAP_HDR`
|
|
(`crates/punktfunk-core/src/quic.rs`). The host enables 10-bit only if the client advertised it.
|
|
- **Detect HDR in-band** from the HEVC VUI (transfer characteristics = SMPTE ST 2084 / PQ), exactly
|
|
like the Apple client's `VideoDecoder.isHDRFormat` (`clients/apple/Sources/PunktfunkKit/`). This
|
|
handles a mid-session HDR toggle without renegotiation. `Welcome.bit_depth` (8/10) is also
|
|
available.
|
|
- **Decode** Main10 → **P010** (10-bit) via D3D11VA.
|
|
- **Present HDR**: swapchain in `DXGI_FORMAT_R10G10B10A2_UNORM` (or `R16G16B16A16_FLOAT`),
|
|
`IDXGISwapChain3::SetColorSpace1(DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020)` + `SetHDRMetaData`
|
|
for HDR10; the host's stream is BT.2020 PQ, so present PQ. For SDR, the existing
|
|
`DXGI_FORMAT_B8G8R8A8_UNORM` + BT.709 path. (The host-side HDR conversion math is in
|
|
`crates/punktfunk-host/src/capture/dxgi.rs` `HDR_PS`/`HdrConverter` if you need the inverse.)
|
|
|
|
## Build gotchas (hard-won)
|
|
|
|
- **`CARGO_HOME` must be an ASCII path** (`C:\Users\Public\.cargo`). SDL3's `build-from-source` PCH
|
|
embeds the registry source path; the `ü` in the dev box's username makes MSVC fail (`MSB8084` /
|
|
`C4828`). Build under an ASCII path generally (the same `ü` triggers `LNK1201` PDB-write failures
|
|
under `~/Developer`).
|
|
- **`CMAKE_POLICY_VERSION_MINIMUM=3.5`** in the build env (CMake 4 rejects libopus's old minimum).
|
|
- **Toolchain:** `winget install NASM.NASM Kitware.CMake LLVM.LLVM` (NASM for aws-lc-rs on the quic
|
|
path; libclang/LLVM for ffmpeg-sys).
|
|
- **windows-reactor is unpublished** (`version 0.0.0`) and fast-moving — depend on it as a **git dep
|
|
pinned to a commit** (`b4129fcc`), and pin the `windows` crate to the **same commit** so the
|
|
`IDXGISwapChain1` you pass to `set_swap_chain` satisfies reactor's `windows_core::Interface`. Its
|
|
`build.rs` downloads the Windows App SDK NuGets (Foundation/Interactive/Runtime), stages the
|
|
bootstrap DLL + `resources.pri` next to the exe, and **`.unwrap()`s `CARGO_WORKSPACE_DIR`** — set
|
|
it in the build env (`CARGO_WORKSPACE_DIR=C:\Users\Public\punktfunk`). It writes `/temp` + `/winmd`
|
|
to the workspace root (gitignored). The App SDK runtime must be installed to *run*.
|
|
- **Windows clippy is stricter** than Linux CI and `cfg(windows)` code is excluded from Linux CI →
|
|
run `cargo clippy -p punktfunk-client-windows -- -D warnings` ON the Windows box before committing.
|
|
|
|
## Open items
|
|
|
|
- **On-glass validation on the RTX box** of **D3D11VA hardware decode** + **10-bit/HDR present** +
|
|
the **WinUI GUI** — the dev VM is headless / SSH Session 0 / WARP, so the WinUI window can't show
|
|
there and there is no hardware decode. Validate over RDP or on the RTX box against a live HDR host.
|
|
- **RAWINPUT relative-mouse pointer-lock** for the stream view.
|
|
- **Per-host speed-test widget** in the UI.
|
|
|
|
## Key references
|
|
|
|
- **Full Windows plan + SudoVDA/host details:** `design/windows-host.md`.
|
|
- **Template ported from:** `clients/linux/src/*`.
|
|
- **Apple HDR present** (the pattern mirrored): `clients/apple/Sources/PunktfunkKit/{VideoDecoder,
|
|
MetalVideoPresenter,Stage2Pipeline}.swift` — in-band PQ detection, P010 decode, EDR present.
|
|
- **Core client API:** `crates/punktfunk-core/src/client.rs` (`NativeClient`).
|
|
- **Protocol:** `crates/punktfunk-core/src/quic.rs` (`Hello.video_caps`, `Welcome.bit_depth`,
|
|
`VIDEO_CAP_10BIT`/`VIDEO_CAP_HDR`).
|
|
- **Host HDR conversion (inverse math):** `crates/punktfunk-host/src/capture/dxgi.rs` (`HDR_PS`,
|
|
`HdrConverter`) + `crates/punktfunk-host/src/encode/nvenc.rs` (BT.2020/PQ VUI).
|