feat: M3 — full lumen/1 session planes: audio, gamepads+rumble, pinned trust, persistent listener
ci / rust (push) Has been cancelled

m3-host is now a real host, not a one-shot demo. Everything validated live on this box
(two back-to-back sessions, pinned + TOFU, ~200 audio pkts/s, p50 0.84 ms at 720p60).

lumen-core:
- quic.rs: QUIC-datagram side planes demuxed by first byte — Opus audio 0xC9
  ([magic][u32 seq][u64 pts_ns][opus], host→client) and rumble 0xCA ([magic][pad][low][high]).
- Trust: endpoint::server_with_identity (persistent PEM identity) and
  endpoint::client_pinned — SHA-256 cert-fingerprint pinning with TOFU (observed
  fingerprint reported back for persisting). The verifier checks the TLS 1.3
  CertificateVerify signature for real (an MITM replaying the host's public cert without
  its key is rejected; cert pinning alone would not prove key possession).
- client.rs: NativeClient gains pin + host_fingerprint, audio/rumble receivers
  (next_audio / next_rumble); pull methods take &self so the C ABI's per-plane threads
  never alias a &mut (per-plane mutexed borrow slots in abi.rs).
- abi.rs: lumen_connect(pin_sha256, observed_sha256_out) + lumen_connection_next_audio /
  next_rumble. input.rs: documented gamepad wire contract (GameStream buttonFlags bits,
  XInput axis conventions, +y = up) — exported as LUMEN_BTN_*/LUMEN_AXIS_* (bare BTN_*
  collides with <linux/input-event-codes.h> at different values).

lumen-host (m3):
- Persistent accept loop: sessions back to back on one endpoint (--max-sessions, 0 =
  forever); per-session failures log and the loop keeps serving; 10 s handshake deadline
  so a silent client can't wedge the sequential accept queue; teardown on every exit path
  (stop flag → conn.close → join audio+input threads).
- Audio plane: desktop PipeWire capture → Opus 48 kHz stereo 5 ms CBR → datagrams; ONE
  capturer reused across sessions via an AudioCapSlot (PipeWire streams have no cheap
  teardown — per-session opens would leak a thread + core connection + live node each).
- Gamepad routing: incremental GamepadButton/GamepadAxis datagrams accumulate into
  per-pad state feeding the uinput xpad manager; force feedback returns as rumble
  datagrams, with current state re-sent every 500 ms (idempotent-state healing for the
  lossy channel). QUIC endpoint serves the persistent ~/.config/lumen identity and logs
  the pinnable fingerprint.

lumen-client-rs: --pin (malformed values abort — never silently downgrade to TOFU),
TOFU fingerprint logging, audio/rumble datagram counters, gamepad events in --input-test.

clients/apple: scaffold synced — pinSHA256/hostFingerprint (wrong-size pin throws,
fail-closed), nextAudio/nextRumble, gamepad event constructors; README handoff updated
(persistent listener, audio decode notes, trust UX).

Adversarially reviewed (5-dimension multi-agent pass over the diff, 2-skeptic
verification): fixed the MITM signature-check gap, a Y-axis contract inversion, header
macro collisions, ABI aliasing UB, the PipeWire per-session leak, the missing handshake
deadline, fail-open pin parsing, and teardown-on-error paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-10 12:26:18 +00:00
parent 3ea096ace9
commit 520d7342dd
14 changed files with 1508 additions and 242 deletions
+25 -14
View File
@@ -26,13 +26,21 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc
socket, wlr protocols on Sway) and **gamepads** (uinput X-Box-360 pads + rumble
back-channel; live validation pending the udev rule below). Management REST API +
checked-in OpenAPI doc (`mgmt.rs`).
- **M3 (`lumen/1`, the native protocol): seeded and validated.** QUIC control plane
(`lumen-core` `quic` feature: Hello{mode}/Welcome{full Config}/Start), data plane = the
hardened M1 `Session` over raw UDP with **GF(2¹⁶) Leopard FEC + AES-GCM** (inexpressible
in GameStream), input over **QUIC datagrams**, host creates the native virtual output at
the client's requested mode. Measured on-box at 720p120: 1680/1680 frames, **p50 0.83 ms**
capture→encode→FEC→crypto→UDP→reassembled. `lumen-client-rs` is a working (headless)
reference client. Trust is seed-stage (self-signed / accept-any).
- **M3 (`lumen/1`, the native protocol): full session planes, validated live.** QUIC
control plane (`lumen-core` `quic` feature: Hello{mode}/Welcome{full Config}/Start), data
plane = the hardened M1 `Session` over raw UDP with **GF(2¹⁶) Leopard FEC + AES-GCM**
(inexpressible in GameStream), host creates the native virtual output at the client's
requested mode. `m3-host` is a **persistent listener** (sessions back to back;
`--max-sessions`). QUIC datagrams carry the side planes, demuxed by first byte: input
0xC8 (incl. **gamepads** — incremental events accumulated into the uinput xpad), **Opus
audio** 0xC9 (48 kHz stereo, 5 ms, host→client), **rumble** 0xCA (host→client). **Trust:**
host serves its persistent identity (`~/.config/lumen/cert.pem`, shared with GameStream
pairing) and logs the SHA-256 fingerprint; clients pin it (TOFU on first connect —
`endpoint::client_pinned`). Measured on-box at 720p120: 1680/1680 frames, **p50 0.83 ms**
capture→…→reassembled; audio measured live (~200 pkts/s). `lumen-client-rs` is the
working reference client (`--pin`, datagram counters, `--input-test` incl. gamepad).
The embeddable connector (`NativeClient`) exposes it all over the C ABI: `lumen_connect`
(pin/TOFU) + `next_au`/`next_audio`/`next_rumble`/`send_input`.
## What's left
@@ -45,10 +53,12 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc
2. **Sub-frame pipelining**: overlap encode and transmit within a frame. Requires a direct
NVENC SDK wrapper (libavcodec only emits whole AUs) — the next big latency lever (~24 ms
at high res).
3. **lumen/1 trust model**: pairing + certificate pinning to replace accept-any.
3. **lumen/1 protocol growth**: a PIN-style pairing ceremony on top of fingerprint pinning,
mid-stream mode renegotiation (the Welcome is one-shot today), concurrent sessions
(today: one at a time, extras wait in the accept queue).
4. **M2 polish**: wlroots/Sway `VirtualDisplay` backend (deferred; swaymsg `create_output`),
GNOME live validation, gamepad live validation, HDR/10-bit/AV1 negotiation, surround
audio, reconnect-at-new-mode robustness.
GNOME live validation, gamepad live validation (blocked on the udev rule below),
HDR/10-bit/AV1 negotiation, surround audio, reconnect-at-new-mode robustness.
5. **Native clients** (`clients/{apple,android}` scaffolds) consuming `lumen_core.h`.
6. **This box, one-time setup still pending**: `sudo cp scripts/60-lumen.rules
/etc/udev/rules.d/` + user into `input` group (gamepads); `sudo ninja -C
@@ -59,7 +69,7 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc
```sh
cargo build --workspace # green on Linux and macOS
cargo test --workspace # unit + loopback + proptest + C ABI harness (~92 tests)
cargo test --workspace # unit + loopback + proptest + C ABI harness (~97 tests)
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all --check
@@ -123,9 +133,10 @@ kwin_wayland --virtual --width 1920 --height 1080 --no-lockscreen --socket wayla
WAYLAND_DISPLAY=wayland-kde XDG_CURRENT_DESKTOP=KDE LUMEN_VIDEO_SOURCE=virtual \
LUMEN_ZEROCOPY=1 PATH=/tmp/gamescope-src/build/src:$PATH cargo run -rp lumen-host -- serve
# lumen/1 native loopback test (no Moonlight needed):
cargo run -rp lumen-host -- m3-host --source virtual --seconds 10 # + LUMEN_COMPOSITOR=gamescope etc.
cargo run -rp lumen-client-rs -- --mode 1280x720x120 --out /tmp/a.h265 --input-test
# lumen/1 native loopback test (no Moonlight needed; same env as serve, listener persists
# across sessions — bound it with --max-sessions):
cargo run -rp lumen-host -- m3-host --source virtual --seconds 10 --max-sessions 1
cargo run -rp lumen-client-rs -- --mode 1280x720x120 --out /tmp/a.h265 --input-test # + --pin HEX
```
Pinned crate facts: `ashpd` 0.13 + `pipewire` 0.9 (must match ashpd's) + `ffmpeg-next` 8.x