unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 9 Wiki Activity
Files
fd1086074b417db7dd4d5b3fac4f8eff6eb4dd26
punktfunk/design/README.md
T
enricobuehler 12843fe253 feat(protocol,clients): codec preference negotiation + Linux client decodes per Welcome (Phase 2a) 
Adds a client-selectable **preferred codec** and wires the core + ABI + probe + Linux client to
negotiate and decode it. (Windows/Apple/Android follow in 2b.)

**Core:**
- `Hello.preferred_codec` (a single CODEC_* bit, 0 = auto) — a soft hint appended after
  `video_codecs`. `resolve_codec(client, host, preferred)` now honors the preference when the host
  can also emit it, else falls back to precedence (HEVC > AV1 > H.264). Roundtrip + preference tests.
- `NativeClient::connect` takes `video_codecs` + `preferred_codec`; `NativeClient.codec` exposes the
  resolved `Welcome.codec`.
- ABI: `punktfunk_connect_ex7` (adds the two codec params; `ex6` delegates to it advertising
  HEVC-only) + `punktfunk_connection_codec` getter + `PUNKTFUNK_CODEC_{H264,HEVC,AV1}` constants
  (drift-guarded against the wire values). Header regenerated.

**Host:** passes `hello.preferred_codec` into `resolve_codec`.

**probe:** `--codec h264|hevc|av1|auto` sets the preference (still advertises it can decode all
three); the dump extension already follows the resolved codec.

**Linux client:** advertises the codecs FFmpeg can actually decode (`decodable_codecs()`), threads
the user's `codec` setting as the preference, and builds the decoder — both the software and VAAPI
paths, plus the mid-session VAAPI→software demotion — from the negotiated `Welcome.codec` instead of
hardcoding HEVC. New "Video codec" dropdown in Preferences (Automatic/HEVC/H.264/AV1).

Live-validated on the dev box: probe `--codec hevc` against a software (H.264-only) host resolves to
H.264 (graceful soft-preference fallback), no failure. clippy + core (57) + host (133) tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 00:13:26 +00:00

11 KiB
Raw Blame History

design/ — design notes & deep-dive plans

Repo-internal design docs: architecture rationale, investigations, and the why behind decisions that the code and ../CLAUDE.md don't capture. Authoritative current status lives in ../CLAUDE.md ("Where the work stands" / "What's left"); the user-facing guides live in docs-site/. These docs are kept trimmed: once work ships, the redundant implementation detail is dropped (the code is the source of truth) and only the durable rationale + still-open items remain. Git history holds the full originals.

Index

Doc What it is Status
implementation-plan.md Master design thesis (why GF(2¹⁶) FEC + Linux virtual displays; three-phase de-risking), architecture invariants, latency budget, risk register Design reference — §0–7,9 kept; milestones → CLAUDE.md
apollo-comparison.md Apollo↔punktfunk architecture map + file index + ~63-item transferable-improvement backlog (Windows-host focus) Reference + open backlog — ~⅓ shipped (collapsed); rest open
security-review.md Whole-project security audit (2026-06-21), 12 findings Audit trail — 11 fixed/inherent; #12 open
security-review-2026-06-28.md Host-scoped follow-up audit (18 attack surfaces; 15 confirmed + 9 partial) Audit trail — fixes landed; deferred/accepted items recorded
ci.md CI/CD architecture: Gitea workflows, runners, release model, signing Evergreen reference
linux-setup.md Linux host bring-up (NVIDIA/headless) + troubleshooting Setup guide (evergreen)
gamestream-host-plan.md GameStream/Moonlight-compat host (P1.1–P1.6) Shipped — stub + the 2 deferral decisions
stats-capture-plan.md Web-console performance capture Shipped — stub
session-aware-host-followups.md Session-aware host known limitations Open items — #2/#3 shipped; #1,#4–8 parked
gamescope-multiuser.md Per-session gamescope isolation (the 4 plumbing items) Superseded — absorbed by multi-user-profiles.md
multi-user-profiles.md Multi-user / profiles end to end: map a client to a real host OS user account (own isolated desktop), web-console config, per-profile passcode Design, schema-of-record — not yet implemented
host-latency-plan.md Latency under GPU contention — 4-tier plan Partly shipped — superseded by ↓; diagnostics + open tiers kept
gpu-contention-investigation.md GPU-contention root-cause + ranked levers (supersedes ↑) Active plan — §5.A shipped; §5.B/C/E/F/G open
hdr-pipeline-plan.md Glass-to-glass HDR Steps 0–3 shipped; Step 4 (Linux) open
windows-host-rewrite.md Windows host — the single architecture/status/reference doc (validated invariants, ops, open work) Active reference
windows-build-and-packaging.md How the Windows host is built, signed, packaged (drivers-from-source, Inno, CI) Evergreen reference
windows-service.md SYSTEM SCM service + secure-desktop deployment model Shipped — stub + graceful-stop open item
windows-host.md (original 2026-06 plan) Redirect → windows-host-rewrite.md
windows-virtual-display-rust-port.md pf-vdisplay IddCx port + the "IDD-push is impossible on bare metal" finding Shipped — P2 do-not-retry record kept
windows-dualsense-scoping.md Virtual DualSense (UMDF2) decision + M0 bug lessons Shipped (M0–M4) — public signing open
windows-dualsense-game-detection.md Native game-detection fix (SwDeviceCreate identity) Shipped — on-glass test + GameInput open
windows-client-bootstrap.md Windows client architecture record + HDR guide + build gotchas Shipped — on-glass validation open
apple-stage2-presenter.md Apple stage-2 (VTDecompressionSession + CAMetalLayer) presenter Shipped (opt-in) — make-default + iOS open
game-library-stores.md Multi-store game library Phases 1–4 shipped — 6 providers + 8 Qs open
dualsense-haptics.md DualSense advanced-haptics feasibility HID shipped; audio haptics deferred (3 walls)
steam-controller-deck-support.md Rich Steam Controller / Steam Deck input fidelity (paddles · trackpads · gyro → virtual hid-steam) M0–M6 green — full pipeline + fallback + conflict gate built; validation items open
steam-deck-passthrough-plan.md Shippable virtual Deck on any Linux host (usbip/vhci_hcd + raw_gadget transport ladder) + client leave-shortcuts Built, CI-green — on-glass validation open
controller-only-mode.md Controller-only session shape — Deck/desktop as a remote gamepad, no video/audio (complements ↑) Design — not yet implemented
archive/windows-secure-desktop.md Two-process WGC secure-desktop design Archived — shipped but now a fallback (IDD-push primary)

Plus research/gamestream-protocol-research.json — raw Moonlight/GameStream wire reference (data, not prose).

Consolidated open items

Still-open work scattered across the docs above, rolled up by theme so nothing is tracked in only one buried doc. CLAUDE.md "What's left" is the headline list; this is the design-level detail. (→ names the owning doc.)

Latency / performance

  • Sub-frame pipelining — overlap encode+transmit within a frame; needs a direct NVENC SDK wrapper (~2–4 ms). → implementation-plan, gamestream-host-plan
  • GPU-contention levers: correct async NVENC pipeline, auto-gated REALTIME GPU priority, clock/P-state pinning, frame-source escape (swapchain-hook/NvFBC/compose-flip), iGPU encode offload, PERF uniq-vs-fps instrumentation. → gpu-contention-investigation (§5.B/C/E/F/G), host-latency-plan (Tiers 1A/1B/3B/3C/3D/4)
  • Apple stage-2 as default (after resolution/HDR checks) + smoothing/pacing policy + glass-to-glass numbers via tools/latency-probe. → apple-stage2-presenter

HDR

  • Linux 10-bit HDR (Step 4): 8-bit→Main10 shim, true 10-bit PipeWire capture (blocked upstream — gamescope #2126), Linux-client P010 + GTK color management. → hdr-pipeline-plan
  • GameStream HDR/10-bit (capture + metadata plumbing). → gamestream-host-plan
  • Open Qs: MaxCLL source, GameStream SS_HDR_METADATA vs deliberate SDR, HLG sources, mid-session SDR-downgrade + SDR-for-SDR-client validation. → hdr-pipeline-plan

Clients

  • Windows client on-glass validation (D3D11VA decode + HDR present + GUI on the RTX box) + RAWINPUT relative-mouse pointer-lock + per-host speed-test UI. → windows-client-bootstrap, implementation-plan
  • iOS/iPadOS/tvOS stage-2 presenter variants. → apple-stage2-presenter, implementation-plan
  • Android real-device validation (gamepad rumble/lightbar/DualSense, HDR10). → implementation-plan

Windows host

  • Graceful stop signal — host is killed via TerminateProcess (skips RAII teardown → a stale virtual monitor can linger). → windows-service
  • pf-vdisplay slot-reclaim on-glass reconnect-storm A/B; M4 driver-unification source-build validation; P2/P3 cleanup (D1-host lints, M6 scaffolding, M5 reshape WGC/DDA onto session/pipeline). → windows-host-rewrite
  • Session-aware follow-ups: F44 gamescope teardown GPU-context corruption (#1, SIGKILL hypothesis); mid-stream-switch input-loss window; NVENC InitializeEncoder noise at 5K@240; NVENC HEVC ~800 Mbps cap (prefer AV1 above it); restore-guard/keep-warm coupling; Feature B (PUNKTFUNK_SESSION_WATCH) opt-in → default. → session-aware-host-followups
  • Apollo backlog (~63 open) — highest-value: #9 Windows app launch (CreateProcessAsUserW), #7/#18 WASAPI device-loss recovery, #3 per-frame IDXGIFactory::IsCurrent(), #15 watchdog escalation, #14/#30/#56 abs-mouse through the real output rect, #10/#20/#32/#33 tray + browser-UI + in-binary service install + logs endpoint, #67/#68 frame pacing. → apollo-comparison

Windows gamepads

  • DualSense public-distribution signing (EV cert + Microsoft Partner Center attestation — blocks public release); GameInput detection (reads VID/PID 0x0000 — may need a rank-3 KMDF USB-emulating bus driver); HidHide integration; minimum-OS / UMDFVERSION targeting; on-glass Cyberpunk glyph test. → windows-dualsense-scoping, windows-dualsense-game-detection

GameStream

  • AV1 + surround 5.1/7.1 live stock-Moonlight confirmation (incl. FEC-under-loss); reconnect-at-new-mode robustness. → gamestream-host-plan, implementation-plan

Game library

  • 6 remaining providers (Desktop/Flatpak, itch.io, Ubisoft Connect, Amazon Games, Battle.net, EA app); the /library/art/<entryId>/<slot> mgmt endpoint; refactor library.rs into a library/ dir; 8 open design questions; optional SteamGridDB v2 enrichment. → game-library-stores

Controllers / input

  • Steam Controller / Steam Deck remaining validation: Moonlight paddle regression + a live SDL-game consume test. → steam-controller-deck-support
  • usbip virtual-Deck on-glass validation (non-SteamOS promote-in-game + Deck raw_gadget regression). → steam-deck-passthrough-plan
  • Controller-only session shape (Deck/desktop as a remote gamepad, no video/audio) — session_flags/SESSION_INPUT_ONLY protocol bit + host skip-data-plane branch + client controller-only path. → controller-only-mode

Multi-user / sessions

  • Magic multi-user / profiles — the full client→host-OS-user design is written (schema-of-record, absorbs the gamescope isolation plumbing); implementation not started. → multi-user-profiles

Security

  • #12 — scope NODE_TLS_REJECT_UNAUTHORIZED to a per-request pinned agent (needs bun add undici); latent-only today, but must fix before the web app gains any off-loopback server-side TLS. → security-review

Deferred / do-not-retry records (kept so the dead ends aren't re-explored)

  • DualSense audio-driven haptics — deferred until all 3 GO conditions are met. → dualsense-haptics
  • IDD-push direct frame-push on bare-metal console capture — architecturally impossible (no presentation consumer for the swapchain). → windows-virtual-display-rust-port
Reference in New Issue View Git Blame Copy Permalink