enricobuehler
ab6dda2e5f
M0 (lumen-host) — verified on NVIDIA RTX 5070 Ti / Ubuntu 25.10: headless wlroots → xdg ScreenCast portal → PipeWire → NVENC HEVC → playable file, with each access unit round-tripped through a lumen_core host↔client Session (FEC + packetize + reassemble), 0 mismatches. - capture.rs: SyntheticCapturer + portal capture (ashpd 0.13 + pipewire 0.9), format-aware - encode/linux.rs: NVENC via ffmpeg-next 7 (BGRx/RGB → rgb0, no host-side swscale) - m0.rs: capture→encode→file + lumen-core loopback verification M2 P1 (lumen-host gamestream/) — a stock Moonlight client pairs + launches, verified live: - mDNS _nvstream._tcp + nvhttp /serverinfo (HTTP 47989, mutual-TLS HTTPS 47984) - 4-phase pairing: PIN→AES-128-ECB / SHA-256 / RSA-PKCS1v15 / X.509, custom rustls ClientCertVerifier for the mutual-TLS pairchallenge - /applist, /launch (rikey/rikeyid/mode), hand-rolled RTSP (OPTIONS/DESCRIBE/SETUP×3/ ANNOUNCE/PLAY, one-request-per-TCP-connection per moonlight-common-c's read-to-EOF) - video.rs: GameStream RTP + NV_VIDEO_PACKET wire packetizer, data-shards-only (0% FEC, clean-LAN), unit-tested (single/multi-block) Docs: docs/m2-plan.md (phased plan) + docs/research/ (ground-truth protocol spec). Bootstrap/setup updated for the verified path (libnvidia-gl, render/video groups, GPU EGL, pipewire 0.9). Workspace clippy-clean, tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
92 lines
6.0 KiB
Markdown
92 lines
6.0 KiB
Markdown
# M2 — P1 host: stream to a stock Moonlight client
|
|
|
|
The shippable milestone (plan §8). A stock Moonlight/Artemis client discovers this host,
|
|
pairs, launches, and gets video (then input, then audio) on a client-sized virtual display.
|
|
Ground-truth protocol reference: [`research/gamestream-protocol-research.json`](research/gamestream-protocol-research.json)
|
|
(distilled from Sunshine + moonlight-common-c source; cite those for byte-level detail).
|
|
|
|
## Architecture (respects the "one core" invariant)
|
|
|
|
- **lumen-core** gains a **P1 GameStream wire codec** (`ProtocolPhase::P1GameStream`, the
|
|
hook already exists): the exact RTP+`NV_VIDEO_PACKET` framing, the GameStream FEC shard
|
|
layout, and the video/audio AES-GCM/CBC paths. Hot path, native threads, **no async**.
|
|
Kept beside lumen's native internal format (P2), selected by phase.
|
|
- **lumen-host** gains the **control plane** (tokio/axum OK — I/O-bound, not the hot path):
|
|
mDNS discovery, nvhttp serverinfo + the 4-phase pairing, the RTSP handshake, the ENet
|
|
control stream + input injection, the virtual-display lifecycle, and Opus audio encode.
|
|
|
|
## Port map (base 47989; Moonlight derives all by offset)
|
|
|
|
| Port | Proto | Role |
|
|
|---|---|---|
|
|
| 47989 | TCP | HTTP nvhttp (unpaired: /serverinfo, /pair PIN flow) |
|
|
| 47984 | TCP | HTTPS nvhttp (paired; **client-cert pinned**) — /launch, /resume, … |
|
|
| 48010 | TCP | RTSP (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY) |
|
|
| 47998 | UDP | Video RTP (+ RS-FEC, optional AES-GCM) |
|
|
| 47999 | UDP | Audio RTP (Opus, RS-FEC 4+2, optional **AES-CBC**) |
|
|
| 48000 | UDP | ENet control stream (AES-GCM) + remote input |
|
|
| 5353 | UDP | mDNS `_nvstream._tcp.local` advertisement |
|
|
|
|
## Key wire facts (the non-obvious ones)
|
|
|
|
- **Video datagram** = `RTP_PACKET(12, BIG-endian)` + `reserved[4]` + `NV_VIDEO_PACKET(16,
|
|
LITTLE-endian)` + payload. Endianness differs *within the same packet*. `header=0x80|0x10`.
|
|
- `fecInfo` (u32 LE) = `(dataShards<<22)|(fecIndex<<12)|(fecPercentage<<4)`; parityShards is
|
|
**recomputed** by the client as `ceil(dataShards*pct/100)` — must match exactly.
|
|
- `multiFecBlocks` = `(blockIdx<<4)|((nBlocks-1)<<6)`; **≤4 FEC blocks/frame**, ≤255 shards/block.
|
|
- Each frame's bitstream is prefixed with an 8-byte `video_short_frame_header_t`
|
|
(`headerType=0x01`, `frameType` 2=IDR, `lastPayloadLen`) before striping into shards.
|
|
- Shard size = `packetSize + 16`. Data shards first, then parity, over a contiguous RTP
|
|
sequence range. Last data shard zero-padded.
|
|
- **Video crypto** (when `SS_ENC_VIDEO` negotiated): AES-128-GCM, key = raw 16-byte RIKEY
|
|
(from `/launch?rikey=`), IV = `counter_le[8]||0,0,0||'V'(0x56)`, **NO AAD**, 32-byte
|
|
`ENC_VIDEO_HEADER{iv[12],frameNumber,tag[16]}` prefix; **FEC first, then encrypt per shard**.
|
|
- **Pairing**: PIN key = `SHA-256(salt[16] || ascii_pin)[..16]`; AES-128-**ECB** (no padding)
|
|
for the challenge blocks; SHA-256 rolling hashes; RSA-SHA256 signatures over X.509 certs;
|
|
the client cert is pinned for subsequent HTTPS. 4 phases over `/pair?phrase=…`.
|
|
- **RTSP** `Session: DEADBEEFCAFE;timeout = 90` (literal), `Transport: server_port=<p>`,
|
|
`streamid=video/0/0` / `control/13/0`. ANNOUNCE carries the negotiated config
|
|
(`x-nv-video[0].*`, `x-nv-vqos[0].*`) → maps to `lumen_core::Config`.
|
|
|
|
## The two highest interop risks (validate EARLY)
|
|
|
|
1. **RS-FEC matrix compatibility.** Sunshine + Moonlight both use **nanors** (GF(2⁸), poly
|
|
0x11d, Vandermonde systematic). lumen-core uses `reed-solomon-erasure` (Cauchy) — parity
|
|
bytes likely **don't match**, so Moonlight silently fails to recover any frame with a lost
|
|
data shard. Mitigation: **on a clean LAN with no loss the client never runs RS decode**, so
|
|
defer this — get a frame decoded first, then FFI/port nanors for loss recovery.
|
|
2. **Crypto layout.** lumen's `SessionCrypto` (salt + seq-as-AAD) is wire-incompatible. P1
|
|
needs a separate GameStream GCM path. Mitigation: **video encryption is negotiated and
|
|
usually off on LAN** — implement plaintext video first, add GCM later.
|
|
|
|
## Phasing (each phase independently testable with a real Moonlight client)
|
|
|
|
- **P1.1 — Discovery + serverinfo + pairing.** mDNS `_nvstream._tcp`, HTTP/HTTPS nvhttp,
|
|
`/serverinfo` XML, the 4-phase pairing + cert pinning. *Acceptance: Moonlight discovers,
|
|
pairs (PIN), and shows the host as ready.* ← first slice.
|
|
- **P1.2 — Launch + RTSP + virtual display.** `/launch` (parse rikey/rikeyid/mode), the RTSP
|
|
handshake, negotiate `Config`, create a wlroots virtual output sized to the client.
|
|
*Acceptance: Moonlight completes RTSP and the host stands up the UDP streams.*
|
|
- **P1.3 — Video (lumen-core P1 codec), plaintext, clean-LAN.** RTP+NV framing + FEC shard
|
|
layout in lumen-core; wire M0's NVENC AUs → UDP 47998. *Acceptance: Moonlight DISPLAYS video.*
|
|
- **P1.4 — Control + input.** ENet (`rusty_enet`) control stream; decode input → `inject.rs`
|
|
(uinput/reis); request-IDR → force NVENC keyframe. *Acceptance: mouse/keyboard work.*
|
|
- **P1.5 — Robustness: FEC recovery + encryption.** nanors-exact FEC; per-shard AES-GCM.
|
|
*Acceptance: stable under `tc netem` loss; encrypted streams.*
|
|
- **P1.6 — Audio + polish.** Opus + audio RTP/FEC/CBC (UDP 47999); disconnect teardown; KWin
|
|
backend for the user's KDE box. *Acceptance: full game stream with sound — the M2 goal.*
|
|
|
|
## Crates (verified available)
|
|
|
|
`mdns-sd` 0.20 (discovery) · `axum` 0.8 + `rustls` + `tokio-rustls` (nvhttp/HTTPS, custom
|
|
`ClientCertVerifier` for pinning) · `rcgen` 0.14 + `x509-parser` 0.18 + `rsa`/`sha2`/`aes`/
|
|
`ecb` (pairing crypto) · hand-rolled RTSP over `tokio::net::TcpListener` · `rusty_enet` 0.4
|
|
(control) · `opus` 0.3 (audio) · `reis` 0.6 + `input-linux` (input) · `aes-gcm` (already in
|
|
core) for the P1 video/control GCM path; nanors (FFI/port) for FEC recovery in P1.5.
|
|
|
|
## Testing note
|
|
|
|
The host is headless; end-to-end needs a **stock Moonlight client on the LAN** pointed at
|
|
this box (manual "add host" by IP works without mDNS). P1.1 is testable with `curl` against
|
|
`/serverinfo` + the Moonlight pair flow; P1.3+ needs a client that can display.
|