enricobuehler
520d7342dd
ci / rust (push) Has been cancelled
m3-host is now a real host, not a one-shot demo. Everything validated live on this box (two back-to-back sessions, pinned + TOFU, ~200 audio pkts/s, p50 0.84 ms at 720p60). lumen-core: - quic.rs: QUIC-datagram side planes demuxed by first byte — Opus audio 0xC9 ([magic][u32 seq][u64 pts_ns][opus], host→client) and rumble 0xCA ([magic][pad][low][high]). - Trust: endpoint::server_with_identity (persistent PEM identity) and endpoint::client_pinned — SHA-256 cert-fingerprint pinning with TOFU (observed fingerprint reported back for persisting). The verifier checks the TLS 1.3 CertificateVerify signature for real (an MITM replaying the host's public cert without its key is rejected; cert pinning alone would not prove key possession). - client.rs: NativeClient gains pin + host_fingerprint, audio/rumble receivers (next_audio / next_rumble); pull methods take &self so the C ABI's per-plane threads never alias a &mut (per-plane mutexed borrow slots in abi.rs). - abi.rs: lumen_connect(pin_sha256, observed_sha256_out) + lumen_connection_next_audio / next_rumble. input.rs: documented gamepad wire contract (GameStream buttonFlags bits, XInput axis conventions, +y = up) — exported as LUMEN_BTN_*/LUMEN_AXIS_* (bare BTN_* collides with <linux/input-event-codes.h> at different values). lumen-host (m3): - Persistent accept loop: sessions back to back on one endpoint (--max-sessions, 0 = forever); per-session failures log and the loop keeps serving; 10 s handshake deadline so a silent client can't wedge the sequential accept queue; teardown on every exit path (stop flag → conn.close → join audio+input threads). - Audio plane: desktop PipeWire capture → Opus 48 kHz stereo 5 ms CBR → datagrams; ONE capturer reused across sessions via an AudioCapSlot (PipeWire streams have no cheap teardown — per-session opens would leak a thread + core connection + live node each). - Gamepad routing: incremental GamepadButton/GamepadAxis datagrams accumulate into per-pad state feeding the uinput xpad manager; force feedback returns as rumble datagrams, with current state re-sent every 500 ms (idempotent-state healing for the lossy channel). QUIC endpoint serves the persistent ~/.config/lumen identity and logs the pinnable fingerprint. lumen-client-rs: --pin (malformed values abort — never silently downgrade to TOFU), TOFU fingerprint logging, audio/rumble datagram counters, gamepad events in --input-test. clients/apple: scaffold synced — pinSHA256/hostFingerprint (wrong-size pin throws, fail-closed), nextAudio/nextRumble, gamepad event constructors; README handoff updated (persistent listener, audio decode notes, trust UX). Adversarially reviewed (5-dimension multi-agent pass over the diff, 2-skeptic verification): fixed the MITM signature-check gap, a Y-axis contract inversion, header macro collisions, ABI aliasing UB, the PipeWire per-session leak, the missing handshake deadline, fail-open pin parsing, and teardown-on-error paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
155 lines
9.5 KiB
Markdown
155 lines
9.5 KiB
Markdown
# CLAUDE.md — lumen
|
||
|
||
Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protocol core
|
||
(`lumen-core`) exposed over a C ABI and native clients per platform. Full design:
|
||
[`docs/implementation-plan.md`](docs/implementation-plan.md). Status table: `README.md`.
|
||
|
||
## Where the work stands
|
||
|
||
- **M1 (`lumen-core` + C ABI): complete and hardened.** FEC recovery, loopback-under-loss,
|
||
proptests, C ABI harness all green; 13 adversarial-review findings fixed +
|
||
regression-tested (`a913042`).
|
||
- **M2 (GameStream host): working end-to-end with a stock Moonlight client.** Validated live
|
||
on this box: pairing (persists across restarts), serverinfo/applist (app catalog from
|
||
`~/.config/lumen/apps.json` → each entry picks a compositor + nested command), RTSP, ENet
|
||
control, audio, and video at the **client's native resolution and refresh** — the host
|
||
creates a per-session virtual output via per-compositor `VirtualDisplay` backends:
|
||
**KWin** (`zkde_screencast stream_virtual_output`, needs KWin ≥ 6.5.6 headless; >60 Hz via
|
||
custom modes), **gamescope** (spawned headless at WxH@Hz, its PipeWire node captured, needs
|
||
gamescope ≥ 3.16.22 — older deadlocks on PipeWire ≥ 1.6), **Mutter** (D-Bus
|
||
`RecordVirtual`; implemented, live validation pending a gnome-shell install).
|
||
Performance work landed and measured: GPU **zero-copy** on all paths (tiled dmabuf →
|
||
EGL/GL → CUDA; LINEAR dmabuf → **Vulkan bridge** → CUDA → NVENC), auto 2-way NVENC
|
||
split-encode above ~1 Gpix/s (5K@240), infinite GOP + RFI keyframes (killed the periodic
|
||
freeze), encode|send thread split with `sendmmsg` batching. Stable 240 fps at 5120×1440.
|
||
Input: mouse/keyboard (libei via RemoteDesktop portal on KWin/GNOME, gamescope's own EIS
|
||
socket, wlr protocols on Sway) and **gamepads** (uinput X-Box-360 pads + rumble
|
||
back-channel; live validation pending the udev rule below). Management REST API +
|
||
checked-in OpenAPI doc (`mgmt.rs`).
|
||
- **M3 (`lumen/1`, the native protocol): full session planes, validated live.** QUIC
|
||
control plane (`lumen-core` `quic` feature: Hello{mode}/Welcome{full Config}/Start), data
|
||
plane = the hardened M1 `Session` over raw UDP with **GF(2¹⁶) Leopard FEC + AES-GCM**
|
||
(inexpressible in GameStream), host creates the native virtual output at the client's
|
||
requested mode. `m3-host` is a **persistent listener** (sessions back to back;
|
||
`--max-sessions`). QUIC datagrams carry the side planes, demuxed by first byte: input
|
||
0xC8 (incl. **gamepads** — incremental events accumulated into the uinput xpad), **Opus
|
||
audio** 0xC9 (48 kHz stereo, 5 ms, host→client), **rumble** 0xCA (host→client). **Trust:**
|
||
host serves its persistent identity (`~/.config/lumen/cert.pem`, shared with GameStream
|
||
pairing) and logs the SHA-256 fingerprint; clients pin it (TOFU on first connect —
|
||
`endpoint::client_pinned`). Measured on-box at 720p120: 1680/1680 frames, **p50 0.83 ms**
|
||
capture→…→reassembled; audio measured live (~200 pkts/s). `lumen-client-rs` is the
|
||
working reference client (`--pin`, datagram counters, `--input-test` incl. gamepad).
|
||
The embeddable connector (`NativeClient`) exposes it all over the C ABI: `lumen_connect`
|
||
(pin/TOFU) + `next_au`/`next_audio`/`next_rumble`/`send_input`.
|
||
|
||
## What's left
|
||
|
||
1. **M4 — client decode + present**: the SwiftUI client is scaffolded and handed off —
|
||
the lumen/1 connector is in the C ABI (`lumen_connect` & co., ABI-roundtrip-tested) with
|
||
an xcframework build script + LumenKit Swift package; **see
|
||
[`clients/apple/README.md`](clients/apple/README.md) for the Mac-side pickup**. Then
|
||
glass-to-glass numbers via `tools/latency-probe` (scaffold). The Linux reference client
|
||
(`lumen-client-rs`) gets VAAPI + wgpu on the same connector later.
|
||
2. **Sub-frame pipelining**: overlap encode and transmit within a frame. Requires a direct
|
||
NVENC SDK wrapper (libavcodec only emits whole AUs) — the next big latency lever (~2–4 ms
|
||
at high res).
|
||
3. **lumen/1 protocol growth**: a PIN-style pairing ceremony on top of fingerprint pinning,
|
||
mid-stream mode renegotiation (the Welcome is one-shot today), concurrent sessions
|
||
(today: one at a time, extras wait in the accept queue).
|
||
4. **M2 polish**: wlroots/Sway `VirtualDisplay` backend (deferred; swaymsg `create_output`),
|
||
GNOME live validation, gamepad live validation (blocked on the udev rule below),
|
||
HDR/10-bit/AV1 negotiation, surround audio, reconnect-at-new-mode robustness.
|
||
5. **Native clients** (`clients/{apple,android}` scaffolds) consuming `lumen_core.h`.
|
||
6. **This box, one-time setup still pending**: `sudo cp scripts/60-lumen.rules
|
||
/etc/udev/rules.d/` + user into `input` group (gamepads); `sudo ninja -C
|
||
/tmp/gamescope-src/build install` (the fixed gamescope ≥ 3.16.22 — until then use
|
||
`PATH=/tmp/gamescope-src/build/src:$PATH`); `apt install gnome-shell` (Mutter validation).
|
||
|
||
## Build / test / run
|
||
|
||
```sh
|
||
cargo build --workspace # green on Linux and macOS
|
||
cargo test --workspace # unit + loopback + proptest + C ABI harness (~97 tests)
|
||
cargo clippy --workspace --all-targets -- -D warnings
|
||
cargo fmt --all --check
|
||
|
||
cargo run -p loss-harness # FEC loss-resilience sweep (no network needed)
|
||
bash crates/lumen-core/tests/c/run.sh # standalone C-ABI link + round-trip proof
|
||
```
|
||
|
||
Generated artifacts are **checked in** and CI fails on drift: `include/lumen_core.h`
|
||
(cbindgen from `lumen-core/src/abi.rs`) and `docs/api/openapi.json` (regenerate with
|
||
`cargo run -p lumen-host -- openapi > docs/api/openapi.json`; spec lives in `mgmt.rs`).
|
||
|
||
## Layout
|
||
|
||
```
|
||
crates/lumen-core/ protocol · FEC · crypto · quic (lumen/1 control plane, feature-gated)
|
||
crates/lumen-host/
|
||
gamestream/ Moonlight compat: nvhttp · pairing · rtsp · control · stream · gamepad · apps
|
||
vdisplay/{kwin,gamescope,mutter}.rs per-compositor client-sized virtual outputs
|
||
zerocopy/{egl,cuda,vulkan}.rs dmabuf → CUDA → NVENC (tiled via EGL/GL, LINEAR via Vulkan)
|
||
inject/{libei,wlr,gamepad}.rs input backends (+ uinput virtual gamepads)
|
||
capture.rs · encode.rs · audio.rs · m0.rs · m3.rs · mgmt.rs
|
||
crates/lumen-client-rs/ lumen/1 reference client (M3 headless; M4 adds decode+present)
|
||
tools/{loss-harness,latency-probe}/ measurement (plan §10)
|
||
scripts/ 60-lumen.rules · lumen-host.service · host.env.example · headless/
|
||
include/lumen_core.h generated C header
|
||
```
|
||
|
||
## Design invariants — do not regress
|
||
|
||
- **One core, linked everywhere.** Protocol/FEC/crypto live only in `lumen-core`, behind a
|
||
stable, versioned C ABI. `tokio`/`quinn` exist only behind the `quic` feature (control
|
||
plane); **no async on the per-frame path** — native threads only.
|
||
- **Native client resolution, no scaling.** A session gets a virtual output at exactly the
|
||
client's WxH@Hz via the `VirtualDisplay` trait (`create(mode) → VirtualOutput { node_id,
|
||
remote_fd, preferred_mode, keepalive }`, RAII teardown). There is no cross-compositor
|
||
protocol for this — each compositor keeps its own backend.
|
||
- **FEC is the wall-breaker.** GF(2⁸) (≤255 shards/block, Moonlight-compatible) and GF(2¹⁶)
|
||
Leopard (≤65535 shards/block) — lumen/1 negotiates the latter, removing the ~1 Gbps
|
||
ceiling.
|
||
- **M1 security hardening stays intact**: reassembler bounds attacker-controlled fields
|
||
before allocating (`ReassemblerLimits`); AES-GCM per-direction nonce salts + seq-as-AAD;
|
||
ABI `struct_size` checks. Regression tests exist — keep them green.
|
||
- **PipeWire consumer discipline**: our capture streams set `node.dont-reconnect` and tear
|
||
down promptly on negotiation timeout — one wedged link head-blocks the daemon's shared
|
||
work queue system-wide.
|
||
|
||
## Running on this box
|
||
|
||
Headless QEMU VM (Ubuntu 26.04, kernel 7.0), passthrough RTX 5070 Ti (driver 595 **open**
|
||
module — a kernel update silently drops it; reinstall `nvidia-driver-595-open`), no KMS
|
||
scanout → KWin `--drm` impossible; everything renders offscreen via `renderD128`.
|
||
|
||
```sh
|
||
# compositor session (shell 1, or the systemd unit in scripts/):
|
||
XDG_RUNTIME_DIR=/run/user/1000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus \
|
||
XDG_CURRENT_DESKTOP=KDE KWIN_WAYLAND_NO_PERMISSION_CHECKS=1 \
|
||
kwin_wayland --virtual --width 1920 --height 1080 --no-lockscreen --socket wayland-kde \
|
||
--exit-with-session wev
|
||
|
||
# host (shell 2; gamescope entries need the PATH override until ninja install):
|
||
WAYLAND_DISPLAY=wayland-kde XDG_CURRENT_DESKTOP=KDE LUMEN_VIDEO_SOURCE=virtual \
|
||
LUMEN_ZEROCOPY=1 PATH=/tmp/gamescope-src/build/src:$PATH cargo run -rp lumen-host -- serve
|
||
|
||
# lumen/1 native loopback test (no Moonlight needed; same env as serve, listener persists
|
||
# across sessions — bound it with --max-sessions):
|
||
cargo run -rp lumen-host -- m3-host --source virtual --seconds 10 --max-sessions 1
|
||
cargo run -rp lumen-client-rs -- --mode 1280x720x120 --out /tmp/a.h265 --input-test # + --pin HEX
|
||
```
|
||
|
||
Pinned crate facts: `ashpd` 0.13 + `pipewire` 0.9 (must match ashpd's) + `ffmpeg-next` 8.x
|
||
(system FFmpeg 8 / libavcodec 62). Env knobs: `LUMEN_VIDEO_SOURCE=virtual|portal`,
|
||
`LUMEN_COMPOSITOR=kwin|gamescope|mutter`, `LUMEN_ZEROCOPY=1`, `LUMEN_GAMESCOPE_APP=...`,
|
||
`LUMEN_INPUT_BACKEND=...`, `LUMEN_PERF=1` (per-stage timing), `LUMEN_VIDEO_DROP=N` (FEC
|
||
test), `LUMEN_FEC_PCT=N`.
|
||
|
||
## Conventions
|
||
|
||
- Rust 2021, `rustfmt` + `clippy -D warnings` clean before commit.
|
||
- Match the surrounding code's comment density and naming.
|
||
- Commit messages end with the Co-Authored-By trailer (see `git log`).
|
||
- `pkill` caution on this box: match exact comm names (`pkill -x gamescope-wl`,
|
||
`pkill -x lumen-host`) — `pkill -f` self-matches the invoking shell.
|