unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 9 Packages Projects Releases 9 Wiki Activity
Files
c21549c1364a29137aa9f878e8d0a04d74f693e6
punktfunk/design/README.md
T
enricobuehler 12843fe253 feat(protocol,clients): codec preference negotiation + Linux client decodes per Welcome (Phase 2a) 
Adds a client-selectable **preferred codec** and wires the core + ABI + probe + Linux client to
negotiate and decode it. (Windows/Apple/Android follow in 2b.)

**Core:**
- `Hello.preferred_codec` (a single CODEC_* bit, 0 = auto) — a soft hint appended after
  `video_codecs`. `resolve_codec(client, host, preferred)` now honors the preference when the host
  can also emit it, else falls back to precedence (HEVC > AV1 > H.264). Roundtrip + preference tests.
- `NativeClient::connect` takes `video_codecs` + `preferred_codec`; `NativeClient.codec` exposes the
  resolved `Welcome.codec`.
- ABI: `punktfunk_connect_ex7` (adds the two codec params; `ex6` delegates to it advertising
  HEVC-only) + `punktfunk_connection_codec` getter + `PUNKTFUNK_CODEC_{H264,HEVC,AV1}` constants
  (drift-guarded against the wire values). Header regenerated.

**Host:** passes `hello.preferred_codec` into `resolve_codec`.

**probe:** `--codec h264|hevc|av1|auto` sets the preference (still advertises it can decode all
three); the dump extension already follows the resolved codec.

**Linux client:** advertises the codecs FFmpeg can actually decode (`decodable_codecs()`), threads
the user's `codec` setting as the preference, and builds the decoder — both the software and VAAPI
paths, plus the mid-session VAAPI→software demotion — from the negotiated `Welcome.codec` instead of
hardcoding HEVC. New "Video codec" dropdown in Preferences (Automatic/HEVC/H.264/AV1).

Live-validated on the dev box: probe `--codec hevc` against a software (H.264-only) host resolves to
H.264 (graceful soft-preference fallback), no failure. clippy + core (57) + host (133) tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 00:13:26 +00:00

96 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# design/ — design notes & deep-dive plans
Repo-internal design docs: architecture rationale, investigations, and the *why* behind decisions that
the code and [`../CLAUDE.md`](../CLAUDE.md) don't capture. **Authoritative current status lives in
[`../CLAUDE.md`](../CLAUDE.md)** ("Where the work stands" / "What's left"); the user-facing guides live in
`docs-site/`. These docs are kept trimmed: once work ships, the redundant implementation detail is dropped
(the code is the source of truth) and only the durable rationale + still-open items remain. Git history
holds the full originals.
## Index
| Doc | What it is | Status |
|-----|-----------|--------|
| [`implementation-plan.md`](implementation-plan.md) | Master design thesis (why GF(2¹⁶) FEC + Linux virtual displays; three-phase de-risking), architecture invariants, latency budget, risk register | **Design reference** — §0–7,9 kept; milestones → CLAUDE.md |
| [`apollo-comparison.md`](apollo-comparison.md) | Apollo↔punktfunk architecture map + file index + ~63-item transferable-improvement backlog (Windows-host focus) | **Reference + open backlog** — ~⅓ shipped (collapsed); rest open |
| [`security-review.md`](security-review.md) | Whole-project security audit (2026-06-21), 12 findings | **Audit trail** — 11 fixed/inherent; **#12 open** |
| [`security-review-2026-06-28.md`](security-review-2026-06-28.md) | Host-scoped follow-up audit (18 attack surfaces; 15 confirmed + 9 partial) | **Audit trail** — fixes landed; deferred/accepted items recorded |
| [`ci.md`](ci.md) | CI/CD architecture: Gitea workflows, runners, release model, signing | **Evergreen reference** |
| [`linux-setup.md`](linux-setup.md) | Linux host bring-up (NVIDIA/headless) + troubleshooting | **Setup guide** (evergreen) |
| [`gamestream-host-plan.md`](gamestream-host-plan.md) | GameStream/Moonlight-compat host (P1.1–P1.6) | **Shipped** — stub + the 2 deferral decisions |
| [`stats-capture-plan.md`](stats-capture-plan.md) | Web-console performance capture | **Shipped** — stub |
| [`session-aware-host-followups.md`](session-aware-host-followups.md) | Session-aware host known limitations | **Open items** — #2/#3 shipped; #1,#4–8 parked |
| [`gamescope-multiuser.md`](gamescope-multiuser.md) | Per-session gamescope isolation (the 4 plumbing items) | **Superseded** — absorbed by `multi-user-profiles.md` |
| [`multi-user-profiles.md`](multi-user-profiles.md) | Multi-user / profiles end to end: map a client to a real host OS user account (own isolated desktop), web-console config, per-profile passcode | **Design, schema-of-record** — not yet implemented |
| [`host-latency-plan.md`](host-latency-plan.md) | Latency under GPU contention — 4-tier plan | **Partly shipped** — superseded by ↓; diagnostics + open tiers kept |
| [`gpu-contention-investigation.md`](gpu-contention-investigation.md) | GPU-contention root-cause + ranked levers (supersedes ↑) | **Active plan** — §5.A shipped; §5.B/C/E/F/G open |
| [`hdr-pipeline-plan.md`](hdr-pipeline-plan.md) | Glass-to-glass HDR | **Steps 0–3 shipped**; Step 4 (Linux) open |
| [`windows-host-rewrite.md`](windows-host-rewrite.md) | **Windows host — the single architecture/status/reference doc** (validated invariants, ops, open work) | **Active reference** |
| [`windows-build-and-packaging.md`](windows-build-and-packaging.md) | How the Windows host is built, signed, packaged (drivers-from-source, Inno, CI) | **Evergreen reference** |
| [`windows-service.md`](windows-service.md) | SYSTEM SCM service + secure-desktop deployment model | **Shipped** — stub + graceful-stop open item |
| [`windows-host.md`](windows-host.md) | (original 2026-06 plan) | **Redirect** → `windows-host-rewrite.md` |
| [`windows-virtual-display-rust-port.md`](windows-virtual-display-rust-port.md) | pf-vdisplay IddCx port + the "IDD-push is impossible on bare metal" finding | **Shipped** — P2 do-not-retry record kept |
| [`windows-dualsense-scoping.md`](windows-dualsense-scoping.md) | Virtual DualSense (UMDF2) decision + M0 bug lessons | **Shipped (M0–M4)** — public signing open |
| [`windows-dualsense-game-detection.md`](windows-dualsense-game-detection.md) | Native game-detection fix (SwDeviceCreate identity) | **Shipped** — on-glass test + GameInput open |
| [`windows-client-bootstrap.md`](windows-client-bootstrap.md) | Windows client architecture record + HDR guide + build gotchas | **Shipped** — on-glass validation open |
| [`apple-stage2-presenter.md`](apple-stage2-presenter.md) | Apple stage-2 (VTDecompressionSession + CAMetalLayer) presenter | **Shipped (opt-in)** — make-default + iOS open |
| [`game-library-stores.md`](game-library-stores.md) | Multi-store game library | **Phases 1–4 shipped** — 6 providers + 8 Qs open |
| [`dualsense-haptics.md`](dualsense-haptics.md) | DualSense advanced-haptics feasibility | **HID shipped**; audio haptics deferred (3 walls) |
| [`steam-controller-deck-support.md`](steam-controller-deck-support.md) | Rich Steam Controller / Steam Deck **input fidelity** (paddles · trackpads · gyro → virtual `hid-steam`) | **M0–M6 green** — full pipeline + fallback + conflict gate built; validation items open |
| [`steam-deck-passthrough-plan.md`](steam-deck-passthrough-plan.md) | Shippable virtual Deck on any Linux host (usbip/`vhci_hcd` + raw_gadget transport ladder) + client leave-shortcuts | **Built, CI-green** — on-glass validation open |
| [`controller-only-mode.md`](controller-only-mode.md) | Controller-only **session shape** — Deck/desktop as a remote gamepad, no video/audio (complements ↑) | **Design** — not yet implemented |
| [`archive/windows-secure-desktop.md`](archive/windows-secure-desktop.md) | Two-process WGC secure-desktop design | **Archived** — shipped but now a fallback (IDD-push primary) |
Plus `research/gamestream-protocol-research.json` — raw Moonlight/GameStream wire reference (data, not prose).
## Consolidated open items
Still-open work scattered across the docs above, rolled up by theme so nothing is tracked in only one
buried doc. CLAUDE.md "What's left" is the headline list; this is the design-level detail. (→ names the
owning doc.)
**Latency / performance**
- Sub-frame pipelining — overlap encode+transmit within a frame; needs a direct NVENC SDK wrapper (~2–4 ms). → `implementation-plan`, `gamestream-host-plan`
- GPU-contention levers: correct async NVENC pipeline, auto-gated REALTIME GPU priority, clock/P-state pinning, frame-source escape (swapchain-hook/NvFBC/compose-flip), iGPU encode offload, PERF uniq-vs-fps instrumentation. → `gpu-contention-investigation` (§5.B/C/E/F/G), `host-latency-plan` (Tiers 1A/1B/3B/3C/3D/4)
- Apple stage-2 as default (after resolution/HDR checks) + smoothing/pacing policy + glass-to-glass numbers via `tools/latency-probe`. → `apple-stage2-presenter`
**HDR**
- Linux 10-bit HDR (Step 4): 8-bit→Main10 shim, true 10-bit PipeWire capture (blocked upstream — gamescope #2126), Linux-client P010 + GTK color management. → `hdr-pipeline-plan`
- GameStream HDR/10-bit (capture + metadata plumbing). → `gamestream-host-plan`
- Open Qs: MaxCLL source, GameStream SS_HDR_METADATA vs deliberate SDR, HLG sources, mid-session SDR-downgrade + SDR-for-SDR-client validation. → `hdr-pipeline-plan`
**Clients**
- Windows client on-glass validation (D3D11VA decode + HDR present + GUI on the RTX box) + RAWINPUT relative-mouse pointer-lock + per-host speed-test UI. → `windows-client-bootstrap`, `implementation-plan`
- iOS/iPadOS/tvOS stage-2 presenter variants. → `apple-stage2-presenter`, `implementation-plan`
- Android real-device validation (gamepad rumble/lightbar/DualSense, HDR10). → `implementation-plan`
**Windows host**
- Graceful stop signal — host is killed via TerminateProcess (skips RAII teardown → a stale virtual monitor can linger). → `windows-service`
- pf-vdisplay slot-reclaim on-glass reconnect-storm A/B; M4 driver-unification source-build validation; P2/P3 cleanup (D1-host lints, M6 scaffolding, M5 reshape WGC/DDA onto session/pipeline). → `windows-host-rewrite`
- Session-aware follow-ups: F44 gamescope teardown GPU-context corruption (#1, SIGKILL hypothesis); mid-stream-switch input-loss window; NVENC InitializeEncoder noise at 5K@240; NVENC HEVC ~800 Mbps cap (prefer AV1 above it); restore-guard/keep-warm coupling; Feature B (`PUNKTFUNK_SESSION_WATCH`) opt-in → default. → `session-aware-host-followups`
- Apollo backlog (~63 open) — highest-value: #9 Windows app launch (CreateProcessAsUserW), #7/#18 WASAPI device-loss recovery, #3 per-frame `IDXGIFactory::IsCurrent()`, #15 watchdog escalation, #14/#30/#56 abs-mouse through the real output rect, #10/#20/#32/#33 tray + browser-UI + in-binary service install + logs endpoint, #67/#68 frame pacing. → `apollo-comparison`
**Windows gamepads**
- DualSense public-distribution signing (EV cert + Microsoft Partner Center attestation — blocks public release); GameInput detection (reads VID/PID 0x0000 — may need a rank-3 KMDF USB-emulating bus driver); HidHide integration; minimum-OS / UMDFVERSION targeting; on-glass Cyberpunk glyph test. → `windows-dualsense-scoping`, `windows-dualsense-game-detection`
**GameStream**
- AV1 + surround 5.1/7.1 live stock-Moonlight confirmation (incl. FEC-under-loss); reconnect-at-new-mode robustness. → `gamestream-host-plan`, `implementation-plan`
**Game library**
- 6 remaining providers (Desktop/Flatpak, itch.io, Ubisoft Connect, Amazon Games, Battle.net, EA app); the `/library/art/<entryId>/<slot>` mgmt endpoint; refactor `library.rs` into a `library/` dir; 8 open design questions; optional SteamGridDB v2 enrichment. → `game-library-stores`
**Controllers / input**
- Steam Controller / Steam Deck remaining validation: Moonlight paddle regression + a live SDL-game consume test. → `steam-controller-deck-support`
- usbip virtual-Deck on-glass validation (non-SteamOS promote-in-game + Deck raw_gadget regression). → `steam-deck-passthrough-plan`
- Controller-only session shape (Deck/desktop as a remote gamepad, no video/audio) — `session_flags`/`SESSION_INPUT_ONLY` protocol bit + host skip-data-plane branch + client controller-only path. → `controller-only-mode`
**Multi-user / sessions**
- Magic multi-user / profiles — the full client→host-OS-user design is written (schema-of-record, absorbs the gamescope isolation plumbing); implementation not started. → `multi-user-profiles`
**Security**
- **#12** — scope `NODE_TLS_REJECT_UNAUTHORIZED` to a per-request pinned agent (needs `bun add undici`); latent-only today, but **must fix before the web app gains any off-loopback server-side TLS**. → `security-review`
**Deferred / do-not-retry records** (kept so the dead ends aren't re-explored)
- DualSense audio-driven haptics — deferred until all 3 GO conditions are met. → `dualsense-haptics`
- IDD-push direct frame-push on bare-metal console capture — architecturally impossible (no presentation consumer for the swapchain). → `windows-virtual-display-rust-port`
Reference in New Issue View Git Blame Copy Permalink