feat: M0 capture→encode pipeline + M2 GameStream host (pairing, RTSP, video)
M0 (lumen-host) — verified on NVIDIA RTX 5070 Ti / Ubuntu 25.10: headless wlroots → xdg ScreenCast portal → PipeWire → NVENC HEVC → playable file, with each access unit round-tripped through a lumen_core host↔client Session (FEC + packetize + reassemble), 0 mismatches. - capture.rs: SyntheticCapturer + portal capture (ashpd 0.13 + pipewire 0.9), format-aware - encode/linux.rs: NVENC via ffmpeg-next 7 (BGRx/RGB → rgb0, no host-side swscale) - m0.rs: capture→encode→file + lumen-core loopback verification M2 P1 (lumen-host gamestream/) — a stock Moonlight client pairs + launches, verified live: - mDNS _nvstream._tcp + nvhttp /serverinfo (HTTP 47989, mutual-TLS HTTPS 47984) - 4-phase pairing: PIN→AES-128-ECB / SHA-256 / RSA-PKCS1v15 / X.509, custom rustls ClientCertVerifier for the mutual-TLS pairchallenge - /applist, /launch (rikey/rikeyid/mode), hand-rolled RTSP (OPTIONS/DESCRIBE/SETUP×3/ ANNOUNCE/PLAY, one-request-per-TCP-connection per moonlight-common-c's read-to-EOF) - video.rs: GameStream RTP + NV_VIDEO_PACKET wire packetizer, data-shards-only (0% FEC, clean-LAN), unit-tested (single/multi-block) Docs: docs/m2-plan.md (phased plan) + docs/research/ (ground-truth protocol spec). Bootstrap/setup updated for the verified path (libnvidia-gl, render/video groups, GPU EGL, pipewire 0.9). Workspace clippy-clean, tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -10,10 +10,15 @@ Low-latency desktop streaming stack, Linux-first, with a shared Rust protocol co
|
||||
suite passes (FEC recovery, loopback-under-loss, proptests, a C ABI harness). It was put
|
||||
through an adversarial review and 13 verified findings were fixed + regression-tested
|
||||
(commit `a913042`).
|
||||
- **The host backends are `#[cfg(target_os = "linux")]` stubs.** They compile everywhere
|
||||
but `bail!` until implemented. This is the next work (**M0**, then **M2**) and needs a
|
||||
real Linux GPU + Wayland stack — which is why this repo is being moved to the NVIDIA
|
||||
Ubuntu VM.
|
||||
- **M0 (the pipeline spike) is done and verified** on the NVIDIA box (Ubuntu 25.10, RTX
|
||||
5070 Ti, driver 595): `lumen-host m0` captures a headless wlroots output via the
|
||||
ScreenCast portal + PipeWire, NVENC-encodes it, writes a playable H.265 file, and
|
||||
round-trips every access unit through a `lumen_core` host→client session (0 mismatches).
|
||||
See [`docs/linux-setup.md`](docs/linux-setup.md); the code is in
|
||||
`crates/lumen-host/src/{m0,capture,encode}.rs` (+ `capture/linux.rs`, `encode/linux.rs`).
|
||||
- **The remaining host backends are `#[cfg(target_os = "linux")]` stubs** — KWin/Mutter
|
||||
virtual displays (`vdisplay.rs`), libei/uinput input (`inject.rs`), web/pairing
|
||||
(`web.rs`). They compile everywhere but `bail!` until implemented. This is **M2**.
|
||||
|
||||
## Build / test / run
|
||||
|
||||
@@ -56,19 +61,26 @@ include/lumen_core.h generated C header
|
||||
(`crypto.rs`); the ABI enforces `struct_size` and range-checks inputs. There are
|
||||
regression tests for these — keep them green.
|
||||
|
||||
## Next: M0 (the pipeline spike) on this VM
|
||||
## Running the M0 spike on this box
|
||||
|
||||
**Start here on the NVIDIA Ubuntu VM:** [`docs/linux-setup.md`](docs/linux-setup.md), then
|
||||
run `bash scripts/bootstrap-ubuntu.sh` (verifies NVIDIA/NVENC, installs the Rust/PipeWire/
|
||||
wlroots/FFmpeg-dev deps) and bring up headless Sway with `scripts/headless/`.
|
||||
[`docs/linux-setup.md`](docs/linux-setup.md) is the reference. One-time: `bash
|
||||
scripts/bootstrap-ubuntu.sh` (verifies NVIDIA/NVENC, installs deps incl. `libnvidia-gl`,
|
||||
adds the `render`/`video` groups — re-login after). Then per run: `bash
|
||||
scripts/headless/run-headless-sway.sh` (shell 1) and `bash
|
||||
scripts/headless/prepare-session.sh` (shell 2), then `cargo run -p lumen-host -- m0
|
||||
--source portal --out /tmp/lumen-m0.h265`. `--source synthetic` needs no capture session.
|
||||
|
||||
Per plan §8/§12: drive a headless Sway/wlroots output → capture via PipeWire (ScreenCast
|
||||
portal, `ashpd` 0.13 + `pipewire` 0.10) → encode with NVENC (`ffmpeg-next` 7.x,
|
||||
`hevc_nvenc`) → write a playable H.265 file. Then wire that pipeline into a `lumen-core`
|
||||
host `Session` (M2). The module seams exist in
|
||||
`crates/lumen-host/src/{vdisplay,capture,encode,inject,pipeline}.rs`. **Budget for the
|
||||
CPU-copy fallback first** — dmabuf→NVENC zero-copy import is unreliable across NVIDIA
|
||||
driver versions (plan §9 risk); the setup doc covers it.
|
||||
M0 uses the **CPU-copy capture path** (portal → PipeWire shm, packed `RGB` on wlroots →
|
||||
NVENC `rgb0`); dmabuf→NVENC zero-copy is deferred (plan §9). Pinned crate facts (the setup
|
||||
doc has the why): `ashpd` **0.13** (`screencast` feature, options-struct API, multi-thread
|
||||
tokio runtime) + `pipewire` **0.9** (must match ashpd's; not 0.10) + `ffmpeg-next` **7.x**.
|
||||
|
||||
## Next: M2 — P1 host to a stock Moonlight client
|
||||
|
||||
Wire M0's capture→encode pipeline (`m0.rs` / `pipeline.rs`) into a streaming host: KWin
|
||||
virtual output (`vdisplay.rs`, study KRdp), `serverinfo`/pairing/RTSP (`web.rs`) enough for
|
||||
a real Moonlight client, input via reis/uinput (`inject.rs`). The module seams exist and
|
||||
`bail!` today.
|
||||
|
||||
## Conventions
|
||||
|
||||
|
||||
Generated
+1808
-17
File diff suppressed because it is too large
Load Diff
@@ -13,16 +13,20 @@ negotiated extension. See [`docs/implementation-plan.md`](docs/implementation-pl
|
||||
| Milestone | State |
|
||||
|-----------|-------|
|
||||
| **M1 — `lumen-core` + C ABI** | ✅ done & tested (FEC, packetization, crypto, session, `lumen_core.h`) |
|
||||
| M0 — pipeline spike (wlroots→PipeWire→encode→file) | ⬜ needs Linux GPU |
|
||||
| M2 — P1 host → stock Moonlight | ⬜ scaffolded (`lumen-host`) |
|
||||
| **M0 — pipeline spike** (wlroots→PipeWire→NVENC→file→`lumen-core`) | ✅ done & verified on NVIDIA (RTX 5070 Ti / driver 595) |
|
||||
| M2 — P1 host → stock Moonlight | 🟡 capture+encode landed in M0; pairing/RTSP/vdisplay pending |
|
||||
| M3 — measurement harness | 🟡 `tools/loss-harness` runs; `latency-probe` scaffolded |
|
||||
| M4 — P2 transport + Rust client | 🟡 GF(2¹⁶) core done; `lumen-client-rs` scaffolded |
|
||||
| M5 — Apple client | ⬜ scaffolded (`clients/apple`) |
|
||||
|
||||
`lumen-core` is complete and verified: it builds and its full test suite (FEC recovery,
|
||||
loopback round-trip under loss, property tests, and a **C ABI harness**) passes on
|
||||
macOS/aarch64. The Linux host backends (PipeWire, VAAPI/NVENC, KWin, libei) are
|
||||
`#[cfg(target_os = "linux")]` seams — defined and compiling, implementations pending.
|
||||
macOS/aarch64. **M0 is done:** `lumen-host` captures a headless wlroots output via the
|
||||
ScreenCast portal + PipeWire, encodes it with NVENC, writes a playable H.265 file, and
|
||||
round-trips every access unit through a `lumen_core` host→client session (see
|
||||
`docs/linux-setup.md`). The remaining Linux host backends (KWin/Mutter virtual displays,
|
||||
libei input, web/pairing) are `#[cfg(target_os = "linux")]` seams — defined and compiling,
|
||||
implementations pending (M2).
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
@@ -13,15 +13,30 @@ lumen-core = { path = "../lumen-core" }
|
||||
anyhow = "1"
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
axum = "0.8"
|
||||
mdns-sd = "0.20"
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
rsa = "0.9"
|
||||
sha2 = { version = "0.10", features = ["oid"] }
|
||||
aes = "0.8"
|
||||
rand = "0.8"
|
||||
hex = "0.4"
|
||||
rcgen = { version = "0.13", default-features = false, features = ["aws_lc_rs", "pem"] }
|
||||
x509-parser = "0.16"
|
||||
axum-server = { version = "0.7", features = ["tls-rustls"] }
|
||||
rustls = "0.23"
|
||||
rustls-pemfile = "2"
|
||||
|
||||
# Linux backends are wired in M0/M2. They live behind `#[cfg(target_os = "linux")]`
|
||||
# so the workspace stays green on macOS; the dep list (per plan §4) is:
|
||||
#
|
||||
# [target.'cfg(target_os = "linux")'.dependencies]
|
||||
# pipewire = "..." # ScreenCast portal stream -> dmabuf (capture)
|
||||
# ashpd = "..." # xdg-desktop-portal: ScreenCast, RemoteDesktop
|
||||
# zbus = "..." # DBus: KWin/Mutter virtual-output creation
|
||||
# ffmpeg-next / rsmpeg # VAAPI / NVENC encode, dmabuf import
|
||||
# reis = "..." # libei input injection (Wayland-native)
|
||||
# input-linux = "..." # uinput fallback
|
||||
# axum + tokio # web config / pairing API
|
||||
[target.'cfg(target_os = "linux")'.dependencies]
|
||||
# `screencast` gates the ScreenCast portal module; `tokio` is the default runtime.
|
||||
# `open_pipe_wire_remote` is unconditional, so ashpd's own `pipewire` feature is not
|
||||
# needed — we drive PipeWire with the `pipewire` crate below.
|
||||
ashpd = { version = "0.13", features = ["screencast"] }
|
||||
ffmpeg-next = "7"
|
||||
libc = "0.2"
|
||||
# Must match the pipewire crate ashpd 0.13 links (libspa/pipewire-sys `links` key is
|
||||
# unique per build), i.e. 0.9 — NOT the 0.10 the setup doc mentions.
|
||||
pipewire = "0.9"
|
||||
# ashpd 0.13 uses the tokio runtime; a current-thread runtime drives the one-time
|
||||
# portal handshake (control plane — never the per-frame path).
|
||||
tokio = { version = "1", features = ["rt", "rt-multi-thread", "net", "time"] }
|
||||
|
||||
@@ -1,15 +1,48 @@
|
||||
//! Frame capture (plan §7). On Linux: a PipeWire ScreenCast portal stream delivering
|
||||
//! dmabuf frames with no copy to the CPU. The encoder imports the dmabuf directly.
|
||||
//! Frame capture (plan §7). On Linux: a PipeWire ScreenCast portal stream. M0 uses the
|
||||
//! CPU-copy fallback (the portal delivers a CPU buffer; the encoder uploads it to the GPU
|
||||
//! internally). Zero-copy dmabuf→NVENC import is deferred (plan §9 risk).
|
||||
|
||||
use anyhow::Result;
|
||||
|
||||
/// A captured frame. For zero-copy the real type wraps a dmabuf fd + modifier; the CPU
|
||||
/// buffer is only a fallback path (plan §9 risk: per-GPU dmabuf import quirks).
|
||||
/// Packed pixel layout of a [`CapturedFrame`]. The ScreenCast portal negotiates the
|
||||
/// format; on wlroots it is commonly packed `RGB` (3 bytes/pixel). The encoder maps these
|
||||
/// to an NVENC-accepted input format (`rgb0`/`bgr0`/`rgba`/`bgra`), expanding 3→4 bytes
|
||||
/// where needed — no host-side colour conversion.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum PixelFormat {
|
||||
/// `[B,G,R,x]`, 4 bpp.
|
||||
Bgrx,
|
||||
/// `[R,G,B,x]`, 4 bpp.
|
||||
Rgbx,
|
||||
/// `[B,G,R,A]`, 4 bpp.
|
||||
Bgra,
|
||||
/// `[R,G,B,A]`, 4 bpp.
|
||||
Rgba,
|
||||
/// `[R,G,B]`, 3 bpp.
|
||||
Rgb,
|
||||
/// `[B,G,R]`, 3 bpp.
|
||||
Bgr,
|
||||
}
|
||||
|
||||
impl PixelFormat {
|
||||
pub fn bytes_per_pixel(self) -> usize {
|
||||
match self {
|
||||
PixelFormat::Rgb | PixelFormat::Bgr => 3,
|
||||
_ => 4,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A captured frame. For zero-copy the real type would wrap a dmabuf fd + modifier; the
|
||||
/// CPU buffer is the M0 fallback path (plan §9 risk: per-GPU dmabuf import quirks).
|
||||
pub struct CapturedFrame {
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub pts_ns: u64,
|
||||
/// Fallback CPU pixels (empty when a dmabuf is used).
|
||||
/// Pixel layout of `cpu_bytes`.
|
||||
pub format: PixelFormat,
|
||||
/// Tightly-packed pixels in `format`, `width * height * format.bytes_per_pixel()`
|
||||
/// bytes (no row padding).
|
||||
pub cpu_bytes: Vec<u8>,
|
||||
}
|
||||
|
||||
@@ -19,14 +52,86 @@ pub trait Capturer: Send {
|
||||
fn next_frame(&mut self) -> Result<CapturedFrame>;
|
||||
}
|
||||
|
||||
/// Open a capturer for a PipeWire node id (from the ScreenCast portal).
|
||||
pub fn open_pipewire(_node_id: u32) -> Result<Box<dyn Capturer>> {
|
||||
/// A deterministic moving test pattern (BGRx). Lets M0 exercise the encode → file →
|
||||
/// `lumen_core` path with no live capture session, and produces obviously non-static
|
||||
/// content (a sweeping bar + animated gradient) so the encoded output is verifiable.
|
||||
pub struct SyntheticCapturer {
|
||||
width: u32,
|
||||
height: u32,
|
||||
fps: u32,
|
||||
frame_idx: u64,
|
||||
buf: Vec<u8>,
|
||||
}
|
||||
|
||||
impl SyntheticCapturer {
|
||||
const BPP: usize = 4; // emits BGRx
|
||||
|
||||
pub fn new(width: u32, height: u32, fps: u32) -> Self {
|
||||
assert!(width > 0 && height > 0 && fps > 0);
|
||||
let buf = vec![0u8; width as usize * height as usize * Self::BPP];
|
||||
SyntheticCapturer {
|
||||
width,
|
||||
height,
|
||||
fps,
|
||||
frame_idx: 0,
|
||||
buf,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Capturer for SyntheticCapturer {
|
||||
fn next_frame(&mut self) -> Result<CapturedFrame> {
|
||||
let w = self.width as usize;
|
||||
let h = self.height as usize;
|
||||
let bpp = Self::BPP;
|
||||
let t = self.frame_idx;
|
||||
// A vertical bar sweeps left→right once every ~2s; the background is a gradient
|
||||
// whose phase advances each frame, so every pixel changes frame-to-frame.
|
||||
let bar_x = ((t * w as u64) / (self.fps as u64 * 2)) % w as u64;
|
||||
let phase = (t % 256) as usize;
|
||||
for y in 0..h {
|
||||
let row = y * w * bpp;
|
||||
for x in 0..w {
|
||||
let i = row + x * bpp;
|
||||
let on_bar = (x as u64).abs_diff(bar_x) < 8;
|
||||
// BGRx byte order: [B, G, R, x]
|
||||
self.buf[i] = if on_bar {
|
||||
255
|
||||
} else {
|
||||
((x + phase) & 0xff) as u8
|
||||
};
|
||||
self.buf[i + 1] = if on_bar {
|
||||
255
|
||||
} else {
|
||||
((y + phase) & 0xff) as u8
|
||||
};
|
||||
self.buf[i + 2] = if on_bar { 255 } else { ((x + y) & 0xff) as u8 };
|
||||
self.buf[i + 3] = 0;
|
||||
}
|
||||
}
|
||||
let pts_ns = self.frame_idx * 1_000_000_000 / self.fps as u64;
|
||||
self.frame_idx += 1;
|
||||
Ok(CapturedFrame {
|
||||
width: self.width,
|
||||
height: self.height,
|
||||
pts_ns,
|
||||
format: PixelFormat::Bgrx,
|
||||
cpu_bytes: self.buf.clone(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Open a live capturer for a client-sized monitor via the xdg ScreenCast portal
|
||||
/// (`ashpd`) → PipeWire (`pipewire`). Implemented in the `linux` submodule.
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
anyhow::bail!("pipewire capture not yet implemented (M0)")
|
||||
pub fn open_portal_monitor() -> Result<Box<dyn Capturer>> {
|
||||
linux::PortalCapturer::open().map(|c| Box::new(c) as Box<dyn Capturer>)
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
{
|
||||
anyhow::bail!("capture requires Linux + PipeWire")
|
||||
}
|
||||
pub fn open_portal_monitor() -> Result<Box<dyn Capturer>> {
|
||||
anyhow::bail!("portal capture requires Linux (xdg-desktop-portal + PipeWire)")
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
mod linux;
|
||||
|
||||
@@ -0,0 +1,407 @@
|
||||
//! Live capture: xdg ScreenCast portal (`ashpd`) → PipeWire (`pipewire`), CPU-copy path.
|
||||
//!
|
||||
//! Two dedicated threads, because both stacks are tied to their thread:
|
||||
//! * **portal thread** drives the async ashpd handshake on a multi-thread tokio runtime
|
||||
//! (control plane — never the per-frame path), then parks on a pending future so the
|
||||
//! `proxy` + its zbus connection stay alive (the cast is torn down when that connection
|
||||
//! drops; ashpd's `Session` has no `Drop`);
|
||||
//! * **pipewire thread** owns the (`!Send`) MainLoop/Stream and pumps frames.
|
||||
//!
|
||||
//! The portal hands the PipeWire remote fd + node id to the pipewire thread; decoded BGRx
|
||||
//! frames leave the pipewire thread over a bounded channel. The authoritative frame size
|
||||
//! comes from the negotiated PipeWire format, not the portal's size hint.
|
||||
//!
|
||||
//! Cleanup note (M0): the two threads are detached and torn down at process exit. A
|
||||
//! graceful stop (pipewire `channel` quit + Session close) belongs with the M2 session
|
||||
//! lifecycle.
|
||||
|
||||
use super::{CapturedFrame, Capturer, PixelFormat};
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use std::os::fd::OwnedFd;
|
||||
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError};
|
||||
use std::thread;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Live monitor capturer backed by the portal + PipeWire threads.
|
||||
pub struct PortalCapturer {
|
||||
frames: Receiver<CapturedFrame>,
|
||||
}
|
||||
|
||||
impl PortalCapturer {
|
||||
pub fn open() -> Result<PortalCapturer> {
|
||||
// Portal handshake (async) on its own thread; hands back the PW fd + node id.
|
||||
let (setup_tx, setup_rx) = std::sync::mpsc::channel::<Result<(OwnedFd, u32), String>>();
|
||||
thread::Builder::new()
|
||||
.name("lumen-portal".into())
|
||||
.spawn(move || portal_thread(setup_tx))
|
||||
.context("spawn portal thread")?;
|
||||
|
||||
let (fd, node_id) = match setup_rx.recv_timeout(Duration::from_secs(20)) {
|
||||
Ok(Ok(v)) => v,
|
||||
Ok(Err(e)) => return Err(anyhow!("ScreenCast portal setup failed: {e}")),
|
||||
Err(_) => return Err(anyhow!("timed out waiting for the ScreenCast portal")),
|
||||
};
|
||||
tracing::info!(
|
||||
node_id,
|
||||
"ScreenCast portal session started; connecting PipeWire"
|
||||
);
|
||||
|
||||
// Frames flow from the pipewire thread over a small bounded channel.
|
||||
let (frame_tx, frame_rx) = sync_channel::<CapturedFrame>(8);
|
||||
thread::Builder::new()
|
||||
.name("lumen-pipewire".into())
|
||||
.spawn(move || {
|
||||
if let Err(e) = pipewire::pipewire_thread(fd, node_id, frame_tx) {
|
||||
tracing::error!(error = %format!("{e:#}"), "pipewire capture thread failed");
|
||||
}
|
||||
})
|
||||
.context("spawn pipewire thread")?;
|
||||
|
||||
Ok(PortalCapturer { frames: frame_rx })
|
||||
}
|
||||
}
|
||||
|
||||
impl Capturer for PortalCapturer {
|
||||
fn next_frame(&mut self) -> Result<CapturedFrame> {
|
||||
// First frame can lag behind format negotiation; later frames arrive at ~fps.
|
||||
match self.frames.recv_timeout(Duration::from_secs(10)) {
|
||||
Ok(frame) => Ok(frame),
|
||||
Err(RecvTimeoutError::Timeout) => Err(anyhow!("no PipeWire frame within 10s")),
|
||||
Err(RecvTimeoutError::Disconnected) => Err(anyhow!("PipeWire capture thread ended")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The portal handshake: connect ScreenCast, select a single monitor, start, open the
|
||||
/// PipeWire remote, hand the fd + node id back, then keep the session alive.
|
||||
fn portal_thread(setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String>>) {
|
||||
use ashpd::desktop::screencast::{CursorMode, Screencast, SelectSourcesOptions, SourceType};
|
||||
use ashpd::desktop::PersistMode;
|
||||
use ashpd::enumflags2::BitFlags;
|
||||
|
||||
// Multi-thread runtime: the zbus connection's background reader must be pumped
|
||||
// continuously across the create_session → select_sources → start handshake, or the
|
||||
// portal reports "Invalid session". (A current-thread runtime starves it.)
|
||||
let rt = match tokio::runtime::Builder::new_multi_thread()
|
||||
.worker_threads(2)
|
||||
.enable_all()
|
||||
.build()
|
||||
{
|
||||
Ok(rt) => rt,
|
||||
Err(e) => {
|
||||
let _ = setup_tx.send(Err(format!("build tokio runtime: {e}")));
|
||||
return;
|
||||
}
|
||||
};
|
||||
let err_tx = setup_tx.clone();
|
||||
|
||||
rt.block_on(async move {
|
||||
let result: Result<()> = async {
|
||||
let proxy = Screencast::new()
|
||||
.await
|
||||
.context("connect ScreenCast portal")?;
|
||||
let session = proxy
|
||||
.create_session(Default::default())
|
||||
.await
|
||||
.context("create_session")?;
|
||||
proxy
|
||||
.select_sources(
|
||||
&session,
|
||||
SelectSourcesOptions::default()
|
||||
.set_cursor_mode(CursorMode::Hidden)
|
||||
// Only MONITOR is offered by the wlroots backend
|
||||
// (AvailableSourceTypes=1); requesting unsupported types
|
||||
// invalidates the session.
|
||||
.set_sources(BitFlags::from_flag(SourceType::Monitor))
|
||||
.set_multiple(false)
|
||||
.set_persist_mode(PersistMode::DoNot),
|
||||
)
|
||||
.await
|
||||
.context("select_sources")?
|
||||
.response()
|
||||
.context("select_sources rejected (unsupported source type / cursor mode?)")?;
|
||||
let streams = proxy
|
||||
.start(&session, None, Default::default())
|
||||
.await
|
||||
.context("start cast")?
|
||||
.response()
|
||||
.context("start response (chooser cancelled? portal misconfigured?)")?;
|
||||
let stream = streams
|
||||
.streams()
|
||||
.first()
|
||||
.context("portal returned no streams")?
|
||||
.clone();
|
||||
let node_id = stream.pipe_wire_node_id();
|
||||
let fd = proxy
|
||||
.open_pipe_wire_remote(&session, Default::default())
|
||||
.await
|
||||
.context("open_pipe_wire_remote")?;
|
||||
|
||||
setup_tx
|
||||
.send(Ok((fd, node_id)))
|
||||
.map_err(|_| anyhow!("capturer dropped before setup completed"))?;
|
||||
|
||||
// Keep `proxy` + `session` (and the underlying zbus connection) alive for the
|
||||
// capture; the cast is torn down when the connection drops (ashpd's `Session`
|
||||
// has no `Drop`), which here happens at process exit.
|
||||
let _keep_alive = (&proxy, &session);
|
||||
std::future::pending::<()>().await;
|
||||
Ok(())
|
||||
}
|
||||
.await;
|
||||
|
||||
if let Err(e) = result {
|
||||
let _ = err_tx.send(Err(format!("{e:#}")));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
mod pipewire {
|
||||
//! The PipeWire consumer, confined to its own thread (the PW types are `!Send`).
|
||||
|
||||
use super::{CapturedFrame, PixelFormat};
|
||||
use anyhow::{Context, Result};
|
||||
use pipewire as pw;
|
||||
use pw::{properties::properties, spa};
|
||||
use std::os::fd::OwnedFd;
|
||||
use std::sync::mpsc::SyncSender;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use spa::param::video::{VideoFormat, VideoInfoRaw};
|
||||
use spa::pod::Pod;
|
||||
|
||||
/// Map a negotiated SPA video format to a layout the encoder can consume. Returns
|
||||
/// `None` for formats we don't handle (the frame is then skipped).
|
||||
fn map_format(f: VideoFormat) -> Option<PixelFormat> {
|
||||
Some(match f {
|
||||
VideoFormat::BGRx => PixelFormat::Bgrx,
|
||||
VideoFormat::RGBx => PixelFormat::Rgbx,
|
||||
VideoFormat::BGRA => PixelFormat::Bgra,
|
||||
VideoFormat::RGBA => PixelFormat::Rgba,
|
||||
VideoFormat::RGB => PixelFormat::Rgb,
|
||||
VideoFormat::BGR => PixelFormat::Bgr,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
struct UserData {
|
||||
info: VideoInfoRaw,
|
||||
/// Negotiated layout (`None` until param_changed, or if unsupported).
|
||||
format: Option<PixelFormat>,
|
||||
tx: SyncSender<CapturedFrame>,
|
||||
}
|
||||
|
||||
pub fn pipewire_thread(fd: OwnedFd, node_id: u32, tx: SyncSender<CapturedFrame>) -> Result<()> {
|
||||
pw::init();
|
||||
|
||||
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw MainLoop")?;
|
||||
let context = pw::context::ContextRc::new(&mainloop, None).context("pw Context")?;
|
||||
let core = context
|
||||
.connect_fd_rc(fd, None)
|
||||
.context("pw connect_fd (portal remote)")?;
|
||||
|
||||
let data = UserData {
|
||||
info: VideoInfoRaw::default(),
|
||||
format: None,
|
||||
tx,
|
||||
};
|
||||
|
||||
let stream = pw::stream::StreamBox::new(
|
||||
&core,
|
||||
"lumen-screencast",
|
||||
properties! {
|
||||
*pw::keys::MEDIA_TYPE => "Video",
|
||||
*pw::keys::MEDIA_CATEGORY => "Capture",
|
||||
*pw::keys::MEDIA_ROLE => "Screen",
|
||||
},
|
||||
)
|
||||
.context("pw Stream")?;
|
||||
|
||||
let _listener = stream
|
||||
.add_local_listener_with_user_data(data)
|
||||
.state_changed(|_stream, _ud, old, new| {
|
||||
tracing::info!(?old, ?new, "pipewire stream state");
|
||||
})
|
||||
.param_changed(|_stream, ud, id, param| {
|
||||
let Some(param) = param else { return };
|
||||
if id != pw::spa::param::ParamType::Format.as_raw() {
|
||||
return;
|
||||
}
|
||||
let Ok((media_type, media_subtype)) =
|
||||
pw::spa::param::format_utils::parse_format(param)
|
||||
else {
|
||||
return;
|
||||
};
|
||||
if media_type != pw::spa::param::format::MediaType::Video
|
||||
|| media_subtype != pw::spa::param::format::MediaSubtype::Raw
|
||||
{
|
||||
return;
|
||||
}
|
||||
if ud.info.parse(param).is_ok() {
|
||||
let sz = ud.info.size();
|
||||
ud.format = map_format(ud.info.format());
|
||||
tracing::info!(
|
||||
width = sz.width,
|
||||
height = sz.height,
|
||||
spa_format = ?ud.info.format(),
|
||||
mapped = ?ud.format,
|
||||
"pipewire format negotiated"
|
||||
);
|
||||
if ud.format.is_none() {
|
||||
tracing::error!(
|
||||
spa_format = ?ud.info.format(),
|
||||
"negotiated a pixel format the encoder cannot consume — frames will be skipped"
|
||||
);
|
||||
}
|
||||
}
|
||||
})
|
||||
.process(|stream, ud| {
|
||||
// PipeWire dispatches this from a C trampoline with no catch_unwind; a
|
||||
// panic crossing that FFI boundary would abort the whole host. Contain it.
|
||||
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
|
||||
let Some(mut buffer) = stream.dequeue_buffer() else {
|
||||
return;
|
||||
};
|
||||
let datas = buffer.datas_mut();
|
||||
if datas.is_empty() {
|
||||
return;
|
||||
}
|
||||
let sz = ud.info.size();
|
||||
let (w, h) = (sz.width as usize, sz.height as usize);
|
||||
if w == 0 || h == 0 {
|
||||
return; // format not negotiated yet
|
||||
}
|
||||
let d = &mut datas[0];
|
||||
let (size, offset, stride) = {
|
||||
let c = d.chunk();
|
||||
(
|
||||
c.size() as usize,
|
||||
c.offset() as usize,
|
||||
c.stride().max(0) as usize,
|
||||
)
|
||||
};
|
||||
let Some(fmt) = ud.format else { return }; // unsupported/not negotiated
|
||||
let bpp = fmt.bytes_per_pixel();
|
||||
let row = w * bpp;
|
||||
let stride = if stride == 0 { row } else { stride };
|
||||
let Some(buf) = d.data() else { return };
|
||||
// Need stride*(h-1)+row valid bytes within [offset, offset+size).
|
||||
if stride < row || offset > buf.len() {
|
||||
return;
|
||||
}
|
||||
let avail = buf.len() - offset;
|
||||
let needed = stride * (h - 1) + row;
|
||||
if needed > avail || needed > size {
|
||||
return;
|
||||
}
|
||||
let region = &buf[offset..offset + size.min(avail)];
|
||||
// De-pad into a tightly-packed buffer (chunk stride may exceed w*bpp).
|
||||
let mut tight = vec![0u8; row * h];
|
||||
for y in 0..h {
|
||||
tight[y * row..y * row + row]
|
||||
.copy_from_slice(®ion[y * stride..y * stride + row]);
|
||||
}
|
||||
let pts_ns = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos() as u64)
|
||||
.unwrap_or(0);
|
||||
let frame = CapturedFrame {
|
||||
width: w as u32,
|
||||
height: h as u32,
|
||||
pts_ns,
|
||||
format: fmt,
|
||||
cpu_bytes: tight,
|
||||
};
|
||||
// Drop if the encoder is behind — never block the pipewire loop.
|
||||
let _ = ud.tx.try_send(frame);
|
||||
}));
|
||||
if outcome.is_err() {
|
||||
tracing::error!("panic in pipewire process callback — frame dropped");
|
||||
}
|
||||
})
|
||||
.register()
|
||||
.context("register stream listener")?;
|
||||
|
||||
// Request raw video in any encoder-mappable layout, any size/framerate.
|
||||
let obj = pw::spa::pod::object!(
|
||||
pw::spa::utils::SpaTypes::ObjectParamFormat,
|
||||
pw::spa::param::ParamType::EnumFormat,
|
||||
pw::spa::pod::property!(
|
||||
pw::spa::param::format::FormatProperties::MediaType,
|
||||
Id,
|
||||
pw::spa::param::format::MediaType::Video
|
||||
),
|
||||
pw::spa::pod::property!(
|
||||
pw::spa::param::format::FormatProperties::MediaSubtype,
|
||||
Id,
|
||||
pw::spa::param::format::MediaSubtype::Raw
|
||||
),
|
||||
// Offer the layouts the encoder can map to an NVENC input format. wlroots
|
||||
// commonly fixates packed RGB (3 bpp); other compositors offer 4 bpp. Only
|
||||
// these are requested, so negotiation fails loudly rather than handing us a
|
||||
// format we'd misinterpret.
|
||||
pw::spa::pod::property!(
|
||||
pw::spa::param::format::FormatProperties::VideoFormat,
|
||||
Choice,
|
||||
Enum,
|
||||
Id,
|
||||
VideoFormat::RGB,
|
||||
VideoFormat::RGB,
|
||||
VideoFormat::BGR,
|
||||
VideoFormat::RGBx,
|
||||
VideoFormat::BGRx,
|
||||
VideoFormat::RGBA,
|
||||
VideoFormat::BGRA,
|
||||
),
|
||||
pw::spa::pod::property!(
|
||||
pw::spa::param::format::FormatProperties::VideoSize,
|
||||
Choice,
|
||||
Range,
|
||||
Rectangle,
|
||||
pw::spa::utils::Rectangle {
|
||||
width: 1920,
|
||||
height: 1080
|
||||
},
|
||||
pw::spa::utils::Rectangle {
|
||||
width: 1,
|
||||
height: 1
|
||||
},
|
||||
pw::spa::utils::Rectangle {
|
||||
width: 8192,
|
||||
height: 8192
|
||||
}
|
||||
),
|
||||
pw::spa::pod::property!(
|
||||
pw::spa::param::format::FormatProperties::VideoFramerate,
|
||||
Choice,
|
||||
Range,
|
||||
Fraction,
|
||||
pw::spa::utils::Fraction { num: 60, denom: 1 },
|
||||
pw::spa::utils::Fraction { num: 0, denom: 1 },
|
||||
pw::spa::utils::Fraction { num: 240, denom: 1 }
|
||||
),
|
||||
);
|
||||
|
||||
let values: Vec<u8> = pw::spa::pod::serialize::PodSerializer::serialize(
|
||||
std::io::Cursor::new(Vec::new()),
|
||||
&pw::spa::pod::Value::Object(obj),
|
||||
)
|
||||
.context("serialize format pod")?
|
||||
.0
|
||||
.into_inner();
|
||||
let mut params = [Pod::from_bytes(&values).context("pod from bytes")?];
|
||||
|
||||
stream
|
||||
.connect(
|
||||
spa::utils::Direction::Input,
|
||||
Some(node_id),
|
||||
pw::stream::StreamFlags::AUTOCONNECT | pw::stream::StreamFlags::MAP_BUFFERS,
|
||||
&mut params,
|
||||
)
|
||||
.context("pw stream connect")?;
|
||||
|
||||
// Blocks this thread, pumping frame callbacks until process exit.
|
||||
mainloop.run();
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,15 @@
|
||||
//! Hardware video encode (plan §7). Binds FFmpeg (VAAPI / NVENC); never rewrites codecs.
|
||||
//! Low-latency preset, lookahead off, dmabuf import for zero-copy from [`crate::capture`].
|
||||
//! Hardware video encode (plan §7). Binds FFmpeg (NVENC); never rewrites codecs.
|
||||
//! Low-latency preset, B-frames off. M0 feeds BGRx CPU frames directly — `*_nvenc`
|
||||
//! accepts `bgr0` input and converts to YUV on the GPU, so no host-side swscale is
|
||||
//! needed (dmabuf zero-copy import is deferred; plan §9).
|
||||
|
||||
use crate::capture::CapturedFrame;
|
||||
use crate::capture::{CapturedFrame, PixelFormat};
|
||||
use anyhow::Result;
|
||||
|
||||
/// An encoded access unit (one NAL/AU) to hand to `lumen_core` for FEC + packetization.
|
||||
/// `data` is in-band Annex-B (the encoder is opened without a global header), so each
|
||||
/// keyframe carries its own VPS/SPS/PPS — the bytes are both a playable elementary
|
||||
/// stream and a self-contained AU for the wire.
|
||||
pub struct EncodedFrame {
|
||||
pub data: Vec<u8>,
|
||||
pub pts_ns: u64,
|
||||
@@ -20,21 +25,50 @@ pub enum Codec {
|
||||
Av1,
|
||||
}
|
||||
|
||||
impl Codec {
|
||||
/// The FFmpeg NVENC encoder name (selected by name, not codec id — the latter would
|
||||
/// pick the software encoder).
|
||||
pub fn nvenc_name(self) -> &'static str {
|
||||
match self {
|
||||
Codec::H264 => "h264_nvenc",
|
||||
Codec::H265 => "hevc_nvenc",
|
||||
Codec::Av1 => "av1_nvenc",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A hardware encoder. One per session; runs on the encode thread.
|
||||
pub trait Encoder: Send {
|
||||
fn submit(&mut self, frame: &CapturedFrame) -> Result<()>;
|
||||
/// Pull the next encoded AU if one is ready.
|
||||
fn poll(&mut self) -> Result<Option<EncodedFrame>>;
|
||||
/// Signal end-of-stream. After this, drain the remaining AUs with [`poll`](Self::poll)
|
||||
/// until it returns `None` — NVENC buffers frames internally even at `delay=0`.
|
||||
fn flush(&mut self) -> Result<()>;
|
||||
}
|
||||
|
||||
/// Open an encoder. `bitrate_bps` and `codec` come from session negotiation.
|
||||
pub fn open(_codec: Codec, _bitrate_bps: u64) -> Result<Box<dyn Encoder>> {
|
||||
/// Open an NVENC encoder for packed RGB/BGR CPU frames of the given `format` and mode.
|
||||
/// `format`, `bitrate_bps`, `codec`, and the mode come from session negotiation; M0 takes
|
||||
/// them from the first captured frame.
|
||||
pub fn open_video(
|
||||
codec: Codec,
|
||||
format: PixelFormat,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fps: u32,
|
||||
bitrate_bps: u64,
|
||||
) -> Result<Box<dyn Encoder>> {
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
anyhow::bail!("VAAPI/NVENC encode not yet implemented (M0)")
|
||||
let enc = linux::NvencEncoder::open(codec, format, width, height, fps, bitrate_bps)?;
|
||||
Ok(Box::new(enc) as Box<dyn Encoder>)
|
||||
}
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
{
|
||||
anyhow::bail!("encode requires Linux (VAAPI/NVENC via FFmpeg)")
|
||||
let _ = (codec, format, width, height, fps, bitrate_bps);
|
||||
anyhow::bail!("NVENC encode requires Linux (FFmpeg + NVIDIA driver)")
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
mod linux;
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
//! NVENC encoder via `ffmpeg-next` (binds the system FFmpeg 7.x / libavcodec 61).
|
||||
//!
|
||||
//! Input is a packed RGB/BGR CPU frame; `*_nvenc` accepts `rgb0`/`bgr0`/`rgba`/`bgra`
|
||||
//! directly and does the RGB→YUV conversion on the GPU, so the host stays off the
|
||||
//! colour-conversion path. The portal commonly negotiates packed 24-bit `RGB`, which NVENC
|
||||
//! does *not* accept — we expand it to `rgb0` (one padding byte/pixel, no colour math).
|
||||
//! The encoder is opened *without* a global header so VPS/SPS/PPS are emitted in-band on
|
||||
//! every IDR — the output is both a playable raw Annex-B stream and self-contained AUs.
|
||||
|
||||
use super::{Codec, EncodedFrame, Encoder};
|
||||
use crate::capture::{CapturedFrame, PixelFormat};
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use ffmpeg::format::Pixel;
|
||||
use ffmpeg::util::frame::Video as VideoFrame;
|
||||
use ffmpeg::{codec, encoder, Dictionary, Packet, Rational};
|
||||
use ffmpeg_next as ffmpeg;
|
||||
|
||||
/// Map a captured layout to the NVENC input pixel format, and whether a 3→4 byte expand is
|
||||
/// needed (packed RGB/BGR have no padding byte; the NVENC `*0` formats do).
|
||||
fn nvenc_input(format: PixelFormat) -> (Pixel, bool) {
|
||||
match format {
|
||||
PixelFormat::Bgrx => (Pixel::BGRZ, false), // bgr0
|
||||
PixelFormat::Rgbx => (Pixel::RGBZ, false), // rgb0
|
||||
PixelFormat::Bgra => (Pixel::BGRA, false),
|
||||
PixelFormat::Rgba => (Pixel::RGBA, false),
|
||||
PixelFormat::Rgb => (Pixel::RGBZ, true), // RGB -> rgb0
|
||||
PixelFormat::Bgr => (Pixel::BGRZ, true), // BGR -> bgr0
|
||||
}
|
||||
}
|
||||
|
||||
pub struct NvencEncoder {
|
||||
enc: encoder::video::Encoder,
|
||||
/// Reusable 4-bpp input frame in `nvenc_pixel` (its plane stride may exceed width*4).
|
||||
/// Mutating it in place across frames is sound only because the encoder is opened with
|
||||
/// `delay=0`/`bf=0`/`max_b_frames=0` and the caller drains `poll()` after each `submit`,
|
||||
/// so libavcodec holds no reference to the previous frame's buffer when we overwrite it.
|
||||
frame: VideoFrame,
|
||||
src_format: PixelFormat,
|
||||
expand: bool,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fps: u32,
|
||||
/// Monotonic presentation index, in `1/fps` time-base units.
|
||||
frame_idx: i64,
|
||||
}
|
||||
|
||||
impl NvencEncoder {
|
||||
pub fn open(
|
||||
codec: Codec,
|
||||
format: PixelFormat,
|
||||
width: u32,
|
||||
height: u32,
|
||||
fps: u32,
|
||||
bitrate_bps: u64,
|
||||
) -> Result<Self> {
|
||||
ffmpeg::init().context("ffmpeg init")?;
|
||||
let name = codec.nvenc_name();
|
||||
let av_codec = encoder::find_by_name(name)
|
||||
.ok_or_else(|| anyhow!("{name} not built into libavcodec"))?;
|
||||
let (nvenc_pixel, expand) = nvenc_input(format);
|
||||
|
||||
let mut video = codec::context::Context::new_with_codec(av_codec)
|
||||
.encoder()
|
||||
.video()
|
||||
.context("alloc video encoder")?;
|
||||
video.set_width(width);
|
||||
video.set_height(height);
|
||||
video.set_format(nvenc_pixel); // NVENC converts RGB→YUV internally
|
||||
video.set_time_base(Rational(1, fps as i32));
|
||||
video.set_frame_rate(Some(Rational(fps as i32, 1)));
|
||||
video.set_bit_rate(bitrate_bps as usize);
|
||||
video.set_max_bit_rate(bitrate_bps as usize);
|
||||
video.set_gop(fps.saturating_mul(2).max(1)); // ~2s keyframe interval
|
||||
video.set_max_b_frames(0);
|
||||
|
||||
// Low-latency NVENC tuning (plan §7 / linux-setup doc).
|
||||
let mut opts = Dictionary::new();
|
||||
opts.set("preset", "p1"); // fastest
|
||||
opts.set("tune", "ull"); // ultra-low-latency
|
||||
opts.set("rc", "cbr");
|
||||
opts.set("bf", "0");
|
||||
opts.set("delay", "0");
|
||||
|
||||
let enc = video
|
||||
.open_with(opts)
|
||||
.with_context(|| format!("open {name} ({width}x{height}@{fps}, {bitrate_bps} bps)"))?;
|
||||
|
||||
let frame = VideoFrame::new(nvenc_pixel, width, height);
|
||||
Ok(NvencEncoder {
|
||||
enc,
|
||||
frame,
|
||||
src_format: format,
|
||||
expand,
|
||||
width,
|
||||
height,
|
||||
fps,
|
||||
frame_idx: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl Encoder for NvencEncoder {
|
||||
fn submit(&mut self, captured: &CapturedFrame) -> Result<()> {
|
||||
anyhow::ensure!(
|
||||
captured.width == self.width && captured.height == self.height,
|
||||
"captured frame {}x{} != encoder {}x{}",
|
||||
captured.width,
|
||||
captured.height,
|
||||
self.width,
|
||||
self.height
|
||||
);
|
||||
anyhow::ensure!(
|
||||
captured.format == self.src_format,
|
||||
"captured format {:?} != encoder source {:?}",
|
||||
captured.format,
|
||||
self.src_format
|
||||
);
|
||||
let w = self.width as usize;
|
||||
let h = self.height as usize;
|
||||
let src_bpp = self.src_format.bytes_per_pixel();
|
||||
let src_row = w * src_bpp;
|
||||
anyhow::ensure!(
|
||||
captured.cpu_bytes.len() >= src_row * h,
|
||||
"captured buffer {} bytes < required {}",
|
||||
captured.cpu_bytes.len(),
|
||||
src_row * h
|
||||
);
|
||||
|
||||
let stride = self.frame.stride(0); // dst is 4-bpp, aligned
|
||||
let dst = self.frame.data_mut(0);
|
||||
if self.expand {
|
||||
// packed 3-bpp RGB/BGR → 4-bpp *0 (copy 3 bytes, zero the pad byte)
|
||||
for y in 0..h {
|
||||
let s = &captured.cpu_bytes[y * src_row..y * src_row + src_row];
|
||||
let drow = &mut dst[y * stride..y * stride + w * 4];
|
||||
for x in 0..w {
|
||||
drow[x * 4..x * 4 + 3].copy_from_slice(&s[x * 3..x * 3 + 3]);
|
||||
drow[x * 4 + 3] = 0;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// 4-bpp → 4-bpp, honoring the (possibly larger) dst stride
|
||||
for y in 0..h {
|
||||
dst[y * stride..y * stride + src_row]
|
||||
.copy_from_slice(&captured.cpu_bytes[y * src_row..y * src_row + src_row]);
|
||||
}
|
||||
}
|
||||
self.frame.set_pts(Some(self.frame_idx));
|
||||
self.frame_idx += 1;
|
||||
self.enc.send_frame(&self.frame).context("send_frame")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn poll(&mut self) -> Result<Option<EncodedFrame>> {
|
||||
let mut pkt = Packet::empty();
|
||||
match self.enc.receive_packet(&mut pkt) {
|
||||
Ok(()) => {
|
||||
let data = pkt.data().map(|d| d.to_vec()).unwrap_or_default();
|
||||
let pts = pkt.pts().unwrap_or(0).max(0) as u64;
|
||||
let pts_ns = pts * 1_000_000_000 / self.fps as u64;
|
||||
Ok(Some(EncodedFrame {
|
||||
data,
|
||||
pts_ns,
|
||||
keyframe: pkt.is_key(),
|
||||
}))
|
||||
}
|
||||
// No packet ready yet (need another input frame).
|
||||
Err(ffmpeg::Error::Other { errno })
|
||||
if errno == ffmpeg::util::error::EAGAIN
|
||||
|| errno == ffmpeg::util::error::EWOULDBLOCK =>
|
||||
{
|
||||
Ok(None)
|
||||
}
|
||||
// Fully drained after flush().
|
||||
Err(ffmpeg::Error::Eof) => Ok(None),
|
||||
Err(e) => Err(e).context("receive_packet"),
|
||||
}
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> Result<()> {
|
||||
self.enc.send_eof().context("send_eof")?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
//! The host's self-signed RSA-2048 identity: the cert returned to clients as `plaincert`
|
||||
//! during pairing AND presented as the TLS server cert on 47984 (Moonlight pins it). The
|
||||
//! cert's own X.509 signature bytes are an input to the pairing hashes, so we extract them.
|
||||
|
||||
use super::config_dir;
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use rsa::pkcs1v15::SigningKey;
|
||||
use rsa::pkcs8::DecodePrivateKey;
|
||||
use rsa::RsaPrivateKey;
|
||||
use sha2::Sha256;
|
||||
use std::fs;
|
||||
|
||||
pub struct ServerIdentity {
|
||||
/// PEM of the cert (returned hex-encoded as `plaincert`; also the TLS server cert).
|
||||
pub cert_pem: String,
|
||||
/// PKCS#8 PEM of the private key (TLS server key).
|
||||
pub key_pem: String,
|
||||
/// The cert's X.509 `signatureValue` bytes — bound into the pairing challenge hashes.
|
||||
pub signature: Vec<u8>,
|
||||
/// RSA-PKCS1v15-SHA256 signer over the host key (the pairing `sign256`).
|
||||
pub signing_key: SigningKey<Sha256>,
|
||||
}
|
||||
|
||||
impl ServerIdentity {
|
||||
pub fn load_or_create() -> Result<ServerIdentity> {
|
||||
let dir = config_dir();
|
||||
let cert_path = dir.join("cert.pem");
|
||||
let key_path = dir.join("key.pem");
|
||||
let (cert_pem, key_pem) = match (
|
||||
fs::read_to_string(&cert_path),
|
||||
fs::read_to_string(&key_path),
|
||||
) {
|
||||
(Ok(c), Ok(k)) if !c.trim().is_empty() && !k.trim().is_empty() => (c, k),
|
||||
_ => {
|
||||
let (c, k) = generate()?;
|
||||
fs::create_dir_all(&dir).ok();
|
||||
fs::write(&cert_path, &c)
|
||||
.with_context(|| format!("write {}", cert_path.display()))?;
|
||||
fs::write(&key_path, &k)
|
||||
.with_context(|| format!("write {}", key_path.display()))?;
|
||||
tracing::info!(path = %cert_path.display(), "generated lumen host certificate (RSA-2048)");
|
||||
(c, k)
|
||||
}
|
||||
};
|
||||
let priv_key = RsaPrivateKey::from_pkcs8_pem(&key_pem).context("parse host private key")?;
|
||||
let signing_key = SigningKey::<Sha256>::new(priv_key);
|
||||
let signature = cert_signature(&cert_pem)?;
|
||||
Ok(ServerIdentity {
|
||||
cert_pem,
|
||||
key_pem,
|
||||
signature,
|
||||
signing_key,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn generate() -> Result<(String, String)> {
|
||||
let key = rcgen::KeyPair::generate_for(&rcgen::PKCS_RSA_SHA256).context("rcgen RSA keygen")?;
|
||||
let mut params = rcgen::CertificateParams::new(Vec::<String>::new()).context("cert params")?;
|
||||
params
|
||||
.distinguished_name
|
||||
.push(rcgen::DnType::CommonName, "lumen");
|
||||
params.not_before = rcgen::date_time_ymd(2020, 1, 1);
|
||||
params.not_after = rcgen::date_time_ymd(2040, 1, 1);
|
||||
let cert = params.self_signed(&key).context("self-sign cert")?;
|
||||
Ok((cert.pem(), key.serialize_pem()))
|
||||
}
|
||||
|
||||
/// Extract the X.509 `signatureValue` bytes from a cert PEM.
|
||||
fn cert_signature(cert_pem: &str) -> Result<Vec<u8>> {
|
||||
let (_, pem) = x509_parser::pem::parse_x509_pem(cert_pem.as_bytes())
|
||||
.map_err(|e| anyhow!("parse cert pem: {e}"))?;
|
||||
let x509 = pem.parse_x509().context("parse x509")?;
|
||||
Ok(x509.signature_value.data.to_vec())
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
//! Pairing crypto primitives (control plane only — distinct from `lumen_core`'s AES-GCM
|
||||
//! data-plane sealing). GameStream pairing uses: AES-128-**ECB** with **no padding**,
|
||||
//! SHA-256 (host appversion major ≥ 7), and RSA-PKCS1v15-SHA256 signatures. See the
|
||||
//! `serverinfo + pairing` section of `docs/research/gamestream-protocol-research.json`.
|
||||
|
||||
use aes::cipher::generic_array::GenericArray;
|
||||
use aes::cipher::{BlockDecrypt, BlockEncrypt, KeyInit};
|
||||
use aes::Aes128;
|
||||
use rand::RngCore;
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
/// `n` cryptographically-random bytes.
|
||||
pub fn random<const N: usize>() -> [u8; N] {
|
||||
let mut b = [0u8; N];
|
||||
rand::thread_rng().fill_bytes(&mut b);
|
||||
b
|
||||
}
|
||||
|
||||
/// SHA-256 over the concatenation of `parts`.
|
||||
pub fn sha256(parts: &[&[u8]]) -> [u8; 32] {
|
||||
let mut h = Sha256::new();
|
||||
for p in parts {
|
||||
h.update(p);
|
||||
}
|
||||
h.finalize().into()
|
||||
}
|
||||
|
||||
/// The PIN-derived AES-128 key: `SHA-256(salt || pin)[..16]` (salt first, PIN as ASCII).
|
||||
pub fn pin_key(salt: &[u8; 16], pin: &str) -> [u8; 16] {
|
||||
let d = sha256(&[salt, pin.as_bytes()]);
|
||||
let mut k = [0u8; 16];
|
||||
k.copy_from_slice(&d[..16]);
|
||||
k
|
||||
}
|
||||
|
||||
/// AES-128-ECB encrypt, no padding: input is zero-extended to a 16-byte multiple.
|
||||
pub fn ecb_encrypt(key: &[u8; 16], data: &[u8]) -> Vec<u8> {
|
||||
let cipher = Aes128::new(GenericArray::from_slice(key));
|
||||
let mut out = data.to_vec();
|
||||
let rem = out.len() % 16;
|
||||
if rem != 0 {
|
||||
out.resize(out.len() + (16 - rem), 0);
|
||||
}
|
||||
for chunk in out.chunks_mut(16) {
|
||||
cipher.encrypt_block(GenericArray::from_mut_slice(chunk));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// AES-128-ECB decrypt, no padding: trailing bytes past the last whole block are ignored.
|
||||
pub fn ecb_decrypt(key: &[u8; 16], data: &[u8]) -> Vec<u8> {
|
||||
let cipher = Aes128::new(GenericArray::from_slice(key));
|
||||
let mut out = Vec::with_capacity(data.len());
|
||||
for chunk in data.chunks_exact(16) {
|
||||
let mut block = *GenericArray::from_slice(chunk);
|
||||
cipher.decrypt_block(&mut block);
|
||||
out.extend_from_slice(&block);
|
||||
}
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
//! mDNS advertisement of `_nvstream._tcp.local.` so Moonlight auto-discovers the host.
|
||||
//! (Manual "add host by IP" also works as a fallback, which is what we test with first.)
|
||||
|
||||
use super::Host;
|
||||
use anyhow::{Context, Result};
|
||||
use mdns_sd::{ServiceDaemon, ServiceInfo};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Holds the mDNS daemon; dropping it unregisters the service.
|
||||
pub struct Advert {
|
||||
_daemon: ServiceDaemon,
|
||||
}
|
||||
|
||||
pub fn advertise(host: &Host) -> Result<Advert> {
|
||||
let daemon = ServiceDaemon::new().context("create mDNS daemon")?;
|
||||
let host_name = format!("{}.local.", host.hostname);
|
||||
// No TXT records are required for Moonlight discovery; it resolves the A record and then
|
||||
// GETs /serverinfo for capabilities.
|
||||
let props: HashMap<String, String> = HashMap::new();
|
||||
let service = ServiceInfo::new(
|
||||
"_nvstream._tcp.local.",
|
||||
&host.hostname,
|
||||
&host_name,
|
||||
host.local_ip,
|
||||
host.http_port,
|
||||
props,
|
||||
)
|
||||
.context("build mDNS ServiceInfo")?;
|
||||
daemon.register(service).context("register mDNS service")?;
|
||||
tracing::info!(
|
||||
service = "_nvstream._tcp",
|
||||
port = host.http_port,
|
||||
host = %host_name,
|
||||
"mDNS advertising"
|
||||
);
|
||||
Ok(Advert { _daemon: daemon })
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
//! GameStream (P1) control plane — what a stock Moonlight/Artemis client talks to around
|
||||
//! the media streams: mDNS discovery, the nvhttp serverinfo + pairing HTTP(S) API, RTSP,
|
||||
//! and the ENet control stream. `tokio`/`axum` live here (control plane, I/O-bound — never
|
||||
//! the per-frame hot path; that is `lumen_core`'s P1 wire codec). See `docs/m2-plan.md`.
|
||||
//!
|
||||
//! Status: P1.1 — mDNS `_nvstream._tcp` advertisement + `/serverinfo`. Pairing, RTSP, and
|
||||
//! the media streams follow (see the M2 task list / plan).
|
||||
|
||||
mod cert;
|
||||
mod crypto;
|
||||
mod mdns;
|
||||
mod nvhttp;
|
||||
mod pairing;
|
||||
mod rtsp;
|
||||
mod serverinfo;
|
||||
mod tls;
|
||||
mod video;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use std::net::{IpAddr, Ipv4Addr, UdpSocket};
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// nvhttp ports (Moonlight derives all stream ports by offset from the HTTP base 47989).
|
||||
pub const HTTP_PORT: u16 = 47989;
|
||||
pub const HTTPS_PORT: u16 = 47984;
|
||||
pub const RTSP_PORT: u16 = 48010;
|
||||
pub const VIDEO_PORT: u16 = 47998;
|
||||
pub const CONTROL_PORT: u16 = 47999;
|
||||
pub const AUDIO_PORT: u16 = 48000;
|
||||
|
||||
/// Advertised host version. Major ≥ 7 tells Moonlight to use SHA-256 for pairing.
|
||||
pub const APP_VERSION: &str = "7.1.431.-1";
|
||||
pub const GFE_VERSION: &str = "3.23.0.74";
|
||||
/// Codec support bitmask: 3=H264, 259=+HEVC, 3843=+AV1 (we encode HEVC/H264/AV1 via NVENC).
|
||||
pub const SERVER_CODEC_MODE_SUPPORT: u32 = 3843;
|
||||
|
||||
/// Stable host identity + advertised capabilities, shared across control-plane handlers.
|
||||
pub struct Host {
|
||||
pub hostname: String,
|
||||
/// Stable per-host id (persisted), echoed in serverinfo + matched on pairing.
|
||||
pub uniqueid: String,
|
||||
pub local_ip: IpAddr,
|
||||
pub http_port: u16,
|
||||
pub https_port: u16,
|
||||
// Pairing state (server cert, paired client certs) lands in the next P1.1 slice.
|
||||
}
|
||||
|
||||
impl Host {
|
||||
pub fn detect() -> Result<Host> {
|
||||
Ok(Host {
|
||||
hostname: hostname_string(),
|
||||
uniqueid: load_or_create_uniqueid()?,
|
||||
local_ip: primary_local_ip().unwrap_or(IpAddr::V4(Ipv4Addr::LOCALHOST)),
|
||||
http_port: HTTP_PORT,
|
||||
https_port: HTTPS_PORT,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// The stream parameters a client passes at `/launch`, shared with the RTSP + media stages.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub struct LaunchSession {
|
||||
/// AES-128 key for the RTSP/control/video/audio planes (from `rikey`).
|
||||
pub gcm_key: [u8; 16],
|
||||
/// `rikeyid` — seeds the per-stream GCM IVs.
|
||||
pub rikeyid: i32,
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub fps: u32,
|
||||
}
|
||||
|
||||
/// Shared control-plane state used as the axum app state.
|
||||
pub struct AppState {
|
||||
pub host: Host,
|
||||
pub identity: cert::ServerIdentity,
|
||||
pub pairing: pairing::Pairing,
|
||||
/// Pinned (paired) client certificate DERs — the post-pair allow-list.
|
||||
pub paired: std::sync::Mutex<Vec<Vec<u8>>>,
|
||||
/// The active launch session (set by `/launch`, consumed by RTSP/media).
|
||||
pub launch: std::sync::Mutex<Option<LaunchSession>>,
|
||||
}
|
||||
|
||||
/// Run the GameStream control plane (blocks): mDNS advertisement + the nvhttp servers.
|
||||
pub fn serve() -> Result<()> {
|
||||
let host = Host::detect()?;
|
||||
let identity = cert::ServerIdentity::load_or_create().context("host certificate")?;
|
||||
let state = Arc::new(AppState {
|
||||
host,
|
||||
identity,
|
||||
pairing: pairing::Pairing::new(),
|
||||
paired: std::sync::Mutex::new(Vec::new()),
|
||||
launch: std::sync::Mutex::new(None),
|
||||
});
|
||||
tracing::info!(
|
||||
hostname = %state.host.hostname,
|
||||
uniqueid = %state.host.uniqueid,
|
||||
ip = %state.host.local_ip,
|
||||
"lumen GameStream host (P1.1: serverinfo + pairing + mDNS)"
|
||||
);
|
||||
let rt = tokio::runtime::Runtime::new().context("build tokio runtime")?;
|
||||
rt.block_on(async move {
|
||||
// rustls needs a process-wide crypto provider before any TLS config is built.
|
||||
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
|
||||
let _advert = mdns::advertise(&state.host).context("mDNS advertise")?;
|
||||
rtsp::spawn(state.clone()).context("start RTSP server")?;
|
||||
nvhttp::run(state).await
|
||||
})
|
||||
}
|
||||
|
||||
/// `~/.config/lumen`, created on demand — host identity + (later) pairing state live here.
|
||||
fn config_dir() -> PathBuf {
|
||||
let base = std::env::var_os("XDG_CONFIG_HOME")
|
||||
.map(PathBuf::from)
|
||||
.or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".config")))
|
||||
.unwrap_or_else(|| PathBuf::from("."));
|
||||
base.join("lumen")
|
||||
}
|
||||
|
||||
fn hostname_string() -> String {
|
||||
std::fs::read_to_string("/proc/sys/kernel/hostname")
|
||||
.ok()
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or_else(|| "lumen-host".to_string())
|
||||
}
|
||||
|
||||
/// Load the persisted host uniqueid, or mint one (from the kernel UUID source) and store it.
|
||||
fn load_or_create_uniqueid() -> Result<String> {
|
||||
let path = config_dir().join("uniqueid");
|
||||
if let Ok(s) = std::fs::read_to_string(&path) {
|
||||
let t = s.trim();
|
||||
if !t.is_empty() {
|
||||
return Ok(t.to_string());
|
||||
}
|
||||
}
|
||||
let id = std::fs::read_to_string("/proc/sys/kernel/random/uuid")
|
||||
.map(|u| u.trim().replace('-', ""))
|
||||
.unwrap_or_else(|_| format!("{:016x}{:016x}", std::process::id(), HTTP_PORT));
|
||||
std::fs::create_dir_all(config_dir()).ok();
|
||||
std::fs::write(&path, &id).with_context(|| format!("write {}", path.display()))?;
|
||||
Ok(id)
|
||||
}
|
||||
|
||||
/// Best-effort primary LAN IP: open a UDP socket "toward" a public address and read the
|
||||
/// local address the OS would route through. No packets are actually sent.
|
||||
fn primary_local_ip() -> Option<IpAddr> {
|
||||
let sock = UdpSocket::bind("0.0.0.0:0").ok()?;
|
||||
sock.connect("8.8.8.8:80").ok()?;
|
||||
sock.local_addr().ok().map(|a| a.ip())
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
//! The nvhttp servers: plain HTTP on 47989 and mutual-TLS on 47984. Serves `/serverinfo`,
|
||||
//! the `/pair` flow, `/applist`, and `/launch`/`/resume`/`/cancel`, plus a lumen-only
|
||||
//! `/pin` endpoint to deliver the Moonlight-displayed PIN. Over HTTPS the client is
|
||||
//! mutual-TLS-authenticated, so `/serverinfo` reports `PairStatus=1` there.
|
||||
|
||||
use super::{serverinfo, AppState, LaunchSession, HTTPS_PORT, HTTP_PORT, RTSP_PORT};
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use axum::{
|
||||
extract::{Query, State},
|
||||
http::header,
|
||||
response::IntoResponse,
|
||||
routing::get,
|
||||
Extension, Router,
|
||||
};
|
||||
use std::collections::HashMap;
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Which listener a request arrived on — HTTPS means a mutual-TLS-authenticated client.
|
||||
#[derive(Clone, Copy)]
|
||||
struct Https(bool);
|
||||
|
||||
pub async fn run(state: Arc<AppState>) -> Result<()> {
|
||||
// Mutual-TLS: request + verify the client cert (Moonlight presents one for the
|
||||
// post-pairing pairchallenge + all post-pair endpoints).
|
||||
let tls = axum_server::tls_rustls::RustlsConfig::from_config(super::tls::server_config(
|
||||
&state.identity.cert_pem,
|
||||
&state.identity.key_pem,
|
||||
)?);
|
||||
|
||||
let http_addr = SocketAddr::from(([0, 0, 0, 0], HTTP_PORT));
|
||||
let https_addr = SocketAddr::from(([0, 0, 0, 0], HTTPS_PORT));
|
||||
tracing::info!(%http_addr, %https_addr, "nvhttp listening (serverinfo + pair + launch)");
|
||||
|
||||
let http = axum_server::bind(http_addr).serve(router(state.clone(), false).into_make_service());
|
||||
let https =
|
||||
axum_server::bind_rustls(https_addr, tls).serve(router(state, true).into_make_service());
|
||||
tokio::try_join!(async { http.await.context("nvhttp HTTP server") }, async {
|
||||
https.await.context("nvhttp HTTPS server")
|
||||
},)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn router(state: Arc<AppState>, https: bool) -> Router {
|
||||
Router::new()
|
||||
.route("/serverinfo", get(h_serverinfo))
|
||||
.route("/pair", get(h_pair))
|
||||
.route("/pin", get(h_pin))
|
||||
.route("/applist", get(h_applist))
|
||||
.route("/launch", get(h_launch))
|
||||
.route("/resume", get(h_resume))
|
||||
.route("/cancel", get(h_cancel))
|
||||
.layer(Extension(Https(https)))
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
fn xml(body: String) -> impl IntoResponse {
|
||||
([(header::CONTENT_TYPE, "application/xml")], body)
|
||||
}
|
||||
|
||||
async fn h_serverinfo(
|
||||
State(st): State<Arc<AppState>>,
|
||||
Extension(Https(https)): Extension<Https>,
|
||||
) -> impl IntoResponse {
|
||||
// Over the mutual-TLS port the peer is an authenticated (paired) client → PairStatus=1.
|
||||
xml(serverinfo::serverinfo_xml(&st.host, https))
|
||||
}
|
||||
|
||||
async fn h_pin(
|
||||
State(st): State<Arc<AppState>>,
|
||||
Query(q): Query<HashMap<String, String>>,
|
||||
) -> impl IntoResponse {
|
||||
match q.get("pin").filter(|p| !p.is_empty()) {
|
||||
Some(pin) => {
|
||||
st.pairing.pin.submit(pin.clone());
|
||||
"PIN accepted\n".to_string()
|
||||
}
|
||||
None => "usage: GET /pin?pin=NNNN\n".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn h_applist(State(_st): State<Arc<AppState>>) -> impl IntoResponse {
|
||||
// One app for now: the headless desktop (the wlroots virtual output).
|
||||
xml("<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"200\">\n<App>\n<IsHdrSupported>0</IsHdrSupported>\n<AppTitle>Desktop</AppTitle>\n<ID>1</ID>\n</App>\n</root>\n".to_string())
|
||||
}
|
||||
|
||||
async fn h_launch(
|
||||
State(st): State<Arc<AppState>>,
|
||||
Query(q): Query<HashMap<String, String>>,
|
||||
) -> impl IntoResponse {
|
||||
match launch(&st, &q) {
|
||||
Ok(session) => {
|
||||
*st.launch.lock().unwrap() = Some(session);
|
||||
tracing::info!(
|
||||
w = session.width,
|
||||
h = session.height,
|
||||
fps = session.fps,
|
||||
rikeyid = session.rikeyid,
|
||||
"launch — session created; RTSP at rtsp://{}:{RTSP_PORT}",
|
||||
st.host.local_ip
|
||||
);
|
||||
xml(session_url_xml(&st, "gamesession"))
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "launch failed");
|
||||
xml(error_xml())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn h_resume(State(st): State<Arc<AppState>>) -> impl IntoResponse {
|
||||
if st.launch.lock().unwrap().is_some() {
|
||||
xml(session_url_xml(&st, "resume"))
|
||||
} else {
|
||||
xml(error_xml())
|
||||
}
|
||||
}
|
||||
|
||||
async fn h_cancel(State(st): State<Arc<AppState>>) -> impl IntoResponse {
|
||||
*st.launch.lock().unwrap() = None;
|
||||
tracing::info!("cancel — launch session cleared");
|
||||
xml("<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"200\"><cancel>1</cancel></root>\n".to_string())
|
||||
}
|
||||
|
||||
/// Parse the `/launch` query (rikey/rikeyid/mode) into a [`LaunchSession`].
|
||||
fn launch(_st: &AppState, q: &HashMap<String, String>) -> Result<LaunchSession> {
|
||||
let rikey = q.get("rikey").ok_or_else(|| anyhow!("missing rikey"))?;
|
||||
let key_bytes = hex::decode(rikey).context("rikey hex")?;
|
||||
if key_bytes.len() < 16 {
|
||||
return Err(anyhow!("rikey too short"));
|
||||
}
|
||||
let mut gcm_key = [0u8; 16];
|
||||
gcm_key.copy_from_slice(&key_bytes[..16]);
|
||||
// rikeyid is a signed 32-bit int (negative values wrap to a big-endian u32 IV later).
|
||||
let rikeyid: i32 = q.get("rikeyid").and_then(|s| s.parse().ok()).unwrap_or(0);
|
||||
let (width, height, fps) = q
|
||||
.get("mode")
|
||||
.and_then(|m| parse_mode(m))
|
||||
.unwrap_or((1920, 1080, 60));
|
||||
Ok(LaunchSession {
|
||||
gcm_key,
|
||||
rikeyid,
|
||||
width,
|
||||
height,
|
||||
fps,
|
||||
})
|
||||
}
|
||||
|
||||
/// `"1920x1080x60"` → `(1920, 1080, 60)`.
|
||||
fn parse_mode(mode: &str) -> Option<(u32, u32, u32)> {
|
||||
let mut it = mode.split('x');
|
||||
let w = it.next()?.parse().ok()?;
|
||||
let h = it.next()?.parse().ok()?;
|
||||
let fps = it.next()?.parse().ok()?;
|
||||
Some((w, h, fps))
|
||||
}
|
||||
|
||||
fn session_url_xml(st: &AppState, tag: &str) -> String {
|
||||
format!(
|
||||
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"200\">\n<sessionUrl0>rtsp://{}:{RTSP_PORT}</sessionUrl0>\n<{tag}>1</{tag}>\n</root>\n",
|
||||
st.host.local_ip
|
||||
)
|
||||
}
|
||||
|
||||
async fn h_pair(
|
||||
State(st): State<Arc<AppState>>,
|
||||
Query(q): Query<HashMap<String, String>>,
|
||||
) -> impl IntoResponse {
|
||||
let uniqueid = q.get("uniqueid").cloned().unwrap_or_default();
|
||||
let phrase = q.get("phrase").map(String::as_str);
|
||||
|
||||
let step = phrase
|
||||
.filter(|p| *p == "getservercert" || *p == "pairchallenge")
|
||||
.or_else(|| {
|
||||
[
|
||||
"clientchallenge",
|
||||
"serverchallengeresp",
|
||||
"clientpairingsecret",
|
||||
]
|
||||
.into_iter()
|
||||
.find(|k| q.contains_key(*k))
|
||||
})
|
||||
.unwrap_or("?");
|
||||
tracing::info!(uniqueid, step, "pair request");
|
||||
|
||||
let result = if phrase == Some("getservercert") {
|
||||
match (q.get("salt"), q.get("clientcert")) {
|
||||
(Some(salt), Some(cc)) => {
|
||||
st.pairing
|
||||
.getservercert(&st.identity, &uniqueid, salt, cc)
|
||||
.await
|
||||
}
|
||||
_ => Ok(pair_error_xml()),
|
||||
}
|
||||
} else if phrase == Some("pairchallenge") {
|
||||
// Reached only over the TLS port with the pinned host cert; the handshake is the
|
||||
// proof, so acknowledge success.
|
||||
Ok(paired_ok_xml())
|
||||
} else if let Some(v) = q.get("clientchallenge") {
|
||||
st.pairing.clientchallenge(&st.identity, &uniqueid, v)
|
||||
} else if let Some(v) = q.get("serverchallengeresp") {
|
||||
st.pairing.serverchallengeresp(&st.identity, &uniqueid, v)
|
||||
} else if let Some(v) = q.get("clientpairingsecret") {
|
||||
st.pairing.clientpairingsecret(&uniqueid, v, &st.paired)
|
||||
} else {
|
||||
Ok(pair_error_xml())
|
||||
};
|
||||
|
||||
let body = result.unwrap_or_else(|e| {
|
||||
tracing::warn!(error = %format!("{e:#}"), uniqueid, "pair handler error");
|
||||
pair_error_xml()
|
||||
});
|
||||
xml(body)
|
||||
}
|
||||
|
||||
fn paired_ok_xml() -> String {
|
||||
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"200\"><paired>1</paired></root>\n"
|
||||
.to_string()
|
||||
}
|
||||
|
||||
fn pair_error_xml() -> String {
|
||||
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"200\"><paired>0</paired></root>\n"
|
||||
.to_string()
|
||||
}
|
||||
|
||||
fn error_xml() -> String {
|
||||
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"400\"></root>\n".to_string()
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
//! The 4-phase GameStream pairing state machine (over HTTP), keyed by `uniqueid`. Proves
|
||||
//! both sides know the PIN (via the SHA-256(salt||pin) AES-ECB key) and own their certs
|
||||
//! (RSA signatures), then pins the client cert. The final `pairchallenge` happens over
|
||||
//! HTTPS (handled in `nvhttp`). Byte-exact spec: `docs/research/…-research.json`.
|
||||
|
||||
use super::cert::ServerIdentity;
|
||||
use super::crypto;
|
||||
use anyhow::{anyhow, bail, Context, Result};
|
||||
use rsa::pkcs1v15::{Signature, VerifyingKey};
|
||||
use rsa::pkcs8::DecodePublicKey;
|
||||
use rsa::signature::{SignatureEncoding, Signer, Verifier};
|
||||
use rsa::RsaPublicKey;
|
||||
use sha2::Sha256;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Mutex;
|
||||
use std::time::Duration;
|
||||
use tokio::sync::Notify;
|
||||
|
||||
/// Out-of-band PIN delivery. Moonlight generates + displays a PIN; the user submits it
|
||||
/// (here via `GET /pin?pin=NNNN`). `getservercert` parks until a PIN arrives.
|
||||
pub struct PinGate {
|
||||
pin: Mutex<Option<String>>,
|
||||
notify: Notify,
|
||||
}
|
||||
|
||||
impl PinGate {
|
||||
fn new() -> Self {
|
||||
PinGate {
|
||||
pin: Mutex::new(None),
|
||||
notify: Notify::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn submit(&self, pin: String) {
|
||||
*self.pin.lock().unwrap() = Some(pin);
|
||||
self.notify.notify_waiters();
|
||||
}
|
||||
|
||||
async fn take(&self, timeout: Duration) -> Option<String> {
|
||||
let deadline = tokio::time::Instant::now() + timeout;
|
||||
loop {
|
||||
if let Some(p) = self.pin.lock().unwrap().take() {
|
||||
return Some(p);
|
||||
}
|
||||
if tokio::time::timeout_at(deadline, self.notify.notified())
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
return None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-client pairing session carried across the 4 separate HTTP GETs.
|
||||
struct Session {
|
||||
aes_key: [u8; 16],
|
||||
client_cert_der: Vec<u8>,
|
||||
client_cert_sig: Vec<u8>,
|
||||
client_pubkey: RsaPublicKey,
|
||||
serversecret: [u8; 16],
|
||||
server_challenge: [u8; 16],
|
||||
/// The client's phase-3 hash, recomputed + checked in phase 4.
|
||||
client_hash: Vec<u8>,
|
||||
}
|
||||
|
||||
pub struct Pairing {
|
||||
sessions: Mutex<HashMap<String, Session>>,
|
||||
pub pin: PinGate,
|
||||
}
|
||||
|
||||
impl Pairing {
|
||||
pub fn new() -> Self {
|
||||
Pairing {
|
||||
sessions: Mutex::new(HashMap::new()),
|
||||
pin: PinGate::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Phase 1: store the client cert, await the PIN, derive the AES key, return our cert.
|
||||
pub async fn getservercert(
|
||||
&self,
|
||||
id: &ServerIdentity,
|
||||
uniqueid: &str,
|
||||
salt_hex: &str,
|
||||
clientcert_hex: &str,
|
||||
) -> Result<String> {
|
||||
let salt_bytes = hex::decode(salt_hex).context("salt hex")?;
|
||||
if salt_bytes.len() < 16 {
|
||||
bail!("salt too short");
|
||||
}
|
||||
let mut salt = [0u8; 16];
|
||||
salt.copy_from_slice(&salt_bytes[..16]);
|
||||
let pem_bytes = hex::decode(clientcert_hex).context("clientcert hex")?;
|
||||
let (der, sig, pubkey) = parse_client_cert(&pem_bytes)?;
|
||||
|
||||
tracing::info!(
|
||||
uniqueid,
|
||||
"pairing phase 1 (getservercert) — awaiting PIN: submit `GET /pin?pin=NNNN`"
|
||||
);
|
||||
let pin = self
|
||||
.pin
|
||||
.take(Duration::from_secs(300))
|
||||
.await
|
||||
.ok_or_else(|| anyhow!("no PIN submitted within 300s"))?;
|
||||
let aes_key = crypto::pin_key(&salt, &pin);
|
||||
|
||||
self.sessions.lock().unwrap().insert(
|
||||
uniqueid.to_string(),
|
||||
Session {
|
||||
aes_key,
|
||||
client_cert_der: der,
|
||||
client_cert_sig: sig,
|
||||
client_pubkey: pubkey,
|
||||
serversecret: [0; 16],
|
||||
server_challenge: [0; 16],
|
||||
client_hash: Vec::new(),
|
||||
},
|
||||
);
|
||||
tracing::info!(
|
||||
uniqueid,
|
||||
"pairing phase 1 — PIN accepted, returning host cert"
|
||||
);
|
||||
let inner = format!(
|
||||
"<plaincert>{}</plaincert>",
|
||||
hex::encode(id.cert_pem.as_bytes())
|
||||
);
|
||||
Ok(paired_xml(&inner, true))
|
||||
}
|
||||
|
||||
/// Phase 2: decrypt the client challenge, return our hash + server challenge.
|
||||
pub fn clientchallenge(
|
||||
&self,
|
||||
id: &ServerIdentity,
|
||||
uniqueid: &str,
|
||||
hexv: &str,
|
||||
) -> Result<String> {
|
||||
let mut map = self.sessions.lock().unwrap();
|
||||
let s = map
|
||||
.get_mut(uniqueid)
|
||||
.ok_or_else(|| anyhow!("no pairing session"))?;
|
||||
let enc = hex::decode(hexv).context("clientchallenge hex")?;
|
||||
let client_challenge = crypto::ecb_decrypt(&s.aes_key, &enc);
|
||||
if client_challenge.len() < 16 {
|
||||
bail!("short client challenge");
|
||||
}
|
||||
s.serversecret = crypto::random();
|
||||
s.server_challenge = crypto::random();
|
||||
let server_hash =
|
||||
crypto::sha256(&[&client_challenge[..16], &id.signature, &s.serversecret]);
|
||||
let mut plain = Vec::with_capacity(48);
|
||||
plain.extend_from_slice(&server_hash);
|
||||
plain.extend_from_slice(&s.server_challenge);
|
||||
let resp = crypto::ecb_encrypt(&s.aes_key, &plain);
|
||||
let inner = format!(
|
||||
"<challengeresponse>{}</challengeresponse>",
|
||||
hex::encode(resp)
|
||||
);
|
||||
Ok(paired_xml(&inner, true))
|
||||
}
|
||||
|
||||
/// Phase 3: store the client's hash, return our RSA-signed serversecret.
|
||||
pub fn serverchallengeresp(
|
||||
&self,
|
||||
id: &ServerIdentity,
|
||||
uniqueid: &str,
|
||||
hexv: &str,
|
||||
) -> Result<String> {
|
||||
let mut map = self.sessions.lock().unwrap();
|
||||
let s = map
|
||||
.get_mut(uniqueid)
|
||||
.ok_or_else(|| anyhow!("no pairing session"))?;
|
||||
let enc = hex::decode(hexv).context("serverchallengeresp hex")?;
|
||||
let client_hash = crypto::ecb_decrypt(&s.aes_key, &enc);
|
||||
if client_hash.len() < 32 {
|
||||
bail!("short challenge response");
|
||||
}
|
||||
s.client_hash = client_hash[..32].to_vec();
|
||||
let sig: Signature = id.signing_key.sign(&s.serversecret);
|
||||
let mut secret = Vec::with_capacity(16 + 256);
|
||||
secret.extend_from_slice(&s.serversecret);
|
||||
secret.extend_from_slice(&sig.to_vec());
|
||||
let inner = format!("<pairingsecret>{}</pairingsecret>", hex::encode(secret));
|
||||
Ok(paired_xml(&inner, true))
|
||||
}
|
||||
|
||||
/// Phase 4: verify the client knew the PIN (hash match) and owns its cert (RSA verify);
|
||||
/// on success, pin the client cert.
|
||||
pub fn clientpairingsecret(
|
||||
&self,
|
||||
uniqueid: &str,
|
||||
hexv: &str,
|
||||
paired_store: &Mutex<Vec<Vec<u8>>>,
|
||||
) -> Result<String> {
|
||||
let mut map = self.sessions.lock().unwrap();
|
||||
let s = map
|
||||
.get_mut(uniqueid)
|
||||
.ok_or_else(|| anyhow!("no pairing session"))?;
|
||||
let data = hex::decode(hexv).context("clientpairingsecret hex")?;
|
||||
if data.len() < 16 {
|
||||
bail!("short pairing secret");
|
||||
}
|
||||
let client_secret = &data[..16];
|
||||
let client_sig = &data[16..];
|
||||
let expected = crypto::sha256(&[&s.server_challenge, &s.client_cert_sig, client_secret]);
|
||||
let hash_ok = expected[..] == s.client_hash[..];
|
||||
let sig_ok = verify256(&s.client_pubkey, client_secret, client_sig).is_ok();
|
||||
if hash_ok && sig_ok {
|
||||
paired_store.lock().unwrap().push(s.client_cert_der.clone());
|
||||
tracing::info!(uniqueid, "pairing phase 4 — SUCCESS, client cert pinned");
|
||||
Ok(paired_xml("", true))
|
||||
} else {
|
||||
tracing::warn!(
|
||||
uniqueid,
|
||||
hash_ok,
|
||||
sig_ok,
|
||||
"pairing phase 4 — FAILED (PIN/cert)"
|
||||
);
|
||||
map.remove(uniqueid);
|
||||
Ok(paired_xml("", false))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn verify256(pubkey: &RsaPublicKey, msg: &[u8], sig: &[u8]) -> Result<()> {
|
||||
let vk = VerifyingKey::<Sha256>::new(pubkey.clone());
|
||||
let signature = Signature::try_from(sig).context("parse client signature")?;
|
||||
vk.verify(msg, &signature)
|
||||
.context("verify client signature")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn parse_client_cert(pem_bytes: &[u8]) -> Result<(Vec<u8>, Vec<u8>, RsaPublicKey)> {
|
||||
let (_, pem) =
|
||||
x509_parser::pem::parse_x509_pem(pem_bytes).map_err(|e| anyhow!("client cert pem: {e}"))?;
|
||||
let der = pem.contents.clone();
|
||||
let x509 = pem.parse_x509().context("parse client x509")?;
|
||||
let sig = x509.signature_value.data.to_vec();
|
||||
let pubkey =
|
||||
RsaPublicKey::from_public_key_der(x509.public_key().raw).context("client rsa pubkey")?;
|
||||
Ok((der, sig, pubkey))
|
||||
}
|
||||
|
||||
/// `<root status_code="200"><paired>0|1</paired> inner </root>`.
|
||||
fn paired_xml(inner: &str, paired: bool) -> String {
|
||||
format!(
|
||||
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<root status_code=\"200\">\n<paired>{}</paired>\n{}</root>\n",
|
||||
u8::from(paired),
|
||||
inner
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
//! The GameStream RTSP handshake (TCP 48010). Hand-rolled because GameStream's RTSP is
|
||||
//! non-standard (streamid= targets, the literal `DEADBEEFCAFE` session, the X-SS-* headers)
|
||||
//! and off-the-shelf RTSP crates assume standard semantics. Sequence Moonlight drives:
|
||||
//! OPTIONS → DESCRIBE → SETUP(audio/video/control) → ANNOUNCE → PLAY. ANNOUNCE carries the
|
||||
//! negotiated stream config; PLAY is where the media stages start (P1.3+).
|
||||
//!
|
||||
//! Runs on its own native thread (control-plane setup, not the per-frame hot path), one
|
||||
//! thread per connection. Plaintext only for now (encryption is negotiated; P1.5).
|
||||
|
||||
use super::{AppState, AUDIO_PORT, CONTROL_PORT, RTSP_PORT, VIDEO_PORT};
|
||||
use anyhow::{Context, Result};
|
||||
use std::collections::HashMap;
|
||||
use std::io::{Read, Write};
|
||||
use std::net::{TcpListener, TcpStream};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Opaque per-session payload the client echoes as its first UDP datagram (port-learning).
|
||||
const PING_PAYLOAD: &str = "0011223344556677";
|
||||
|
||||
/// Bind 48010 and accept RTSP connections on a dedicated thread.
|
||||
pub fn spawn(state: Arc<AppState>) -> Result<()> {
|
||||
let listener = TcpListener::bind(("0.0.0.0", RTSP_PORT))
|
||||
.with_context(|| format!("bind RTSP {RTSP_PORT}"))?;
|
||||
tracing::info!(port = RTSP_PORT, "RTSP listening");
|
||||
std::thread::Builder::new()
|
||||
.name("lumen-rtsp".into())
|
||||
.spawn(move || {
|
||||
for conn in listener.incoming() {
|
||||
match conn {
|
||||
Ok(stream) => {
|
||||
let st = state.clone();
|
||||
std::thread::spawn(move || {
|
||||
if let Err(e) = handle_conn(stream, st) {
|
||||
tracing::warn!(error = %format!("{e:#}"), "RTSP connection ended");
|
||||
}
|
||||
});
|
||||
}
|
||||
Err(e) => tracing::warn!(error = %e, "RTSP accept failed"),
|
||||
}
|
||||
}
|
||||
})
|
||||
.context("spawn RTSP thread")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
struct Request {
|
||||
method: String,
|
||||
uri: String,
|
||||
cseq: String,
|
||||
head: String,
|
||||
body: String,
|
||||
}
|
||||
|
||||
fn handle_conn(mut stream: TcpStream, state: Arc<AppState>) -> Result<()> {
|
||||
let peer = stream.peer_addr().ok();
|
||||
let mut buf: Vec<u8> = Vec::new();
|
||||
// GameStream RTSP is one request per TCP connection: moonlight-common-c reads the
|
||||
// response until EOF, so we answer one message and close the connection (which signals
|
||||
// the end of the response). Session state lives in `AppState`, not the connection.
|
||||
if let Some(req) = read_message(&mut stream, &mut buf)? {
|
||||
tracing::info!(
|
||||
method = %req.method, cseq = %req.cseq,
|
||||
"RTSP {} | {}", req.head.replace("\r\n", " | "),
|
||||
if req.body.is_empty() { String::new() } else { format!("body: {}", req.body.replace("\r\n", " | ")) }
|
||||
);
|
||||
let resp = handle_request(&req, &state);
|
||||
stream.write_all(resp.as_bytes()).context("RTSP write")?;
|
||||
stream.flush().ok();
|
||||
// Close (FIN after the flushed response) so the client detects end-of-response.
|
||||
let _ = stream.shutdown(std::net::Shutdown::Both);
|
||||
}
|
||||
let _ = peer;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Read one complete RTSP message (headers + any Content-Length body) from the stream,
|
||||
/// buffering across reads and leaving any pipelined remainder in `buf`.
|
||||
fn read_message(stream: &mut TcpStream, buf: &mut Vec<u8>) -> Result<Option<Request>> {
|
||||
loop {
|
||||
if let Some(end) = find_subslice(buf, b"\r\n\r\n") {
|
||||
let head = std::str::from_utf8(&buf[..end]).context("RTSP header utf8")?;
|
||||
let content_len = header_value(head, "content-length")
|
||||
.and_then(|v| v.trim().parse::<usize>().ok())
|
||||
.unwrap_or(0);
|
||||
let total = end + 4 + content_len;
|
||||
if buf.len() < total {
|
||||
// headers complete but body still arriving — read more
|
||||
} else {
|
||||
let head = head.to_string();
|
||||
let body = String::from_utf8_lossy(&buf[end + 4..total]).into_owned();
|
||||
buf.drain(..total);
|
||||
return Ok(Some(parse_request(&head, body)));
|
||||
}
|
||||
}
|
||||
let mut tmp = [0u8; 8192];
|
||||
let n = stream.read(&mut tmp).context("RTSP read")?;
|
||||
if n == 0 {
|
||||
return Ok(None); // peer closed
|
||||
}
|
||||
buf.extend_from_slice(&tmp[..n]);
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_request(head: &str, body: String) -> Request {
|
||||
let mut lines = head.split("\r\n");
|
||||
let request_line = lines.next().unwrap_or("");
|
||||
let mut parts = request_line.split_whitespace();
|
||||
let method = parts.next().unwrap_or("").to_string();
|
||||
let uri = parts.next().unwrap_or("").to_string();
|
||||
let cseq = header_value(head, "cseq").unwrap_or("0").trim().to_string();
|
||||
Request {
|
||||
method,
|
||||
uri,
|
||||
cseq,
|
||||
head: head.to_string(),
|
||||
body,
|
||||
}
|
||||
}
|
||||
|
||||
fn handle_request(req: &Request, state: &AppState) -> String {
|
||||
match req.method.as_str() {
|
||||
"OPTIONS" => response(
|
||||
&req.cseq,
|
||||
&[("Public", "OPTIONS DESCRIBE SETUP ANNOUNCE PLAY TEARDOWN")],
|
||||
None,
|
||||
),
|
||||
"DESCRIBE" => response(
|
||||
&req.cseq,
|
||||
&[("Content-Type", "application/sdp")],
|
||||
Some(&describe_sdp()),
|
||||
),
|
||||
"SETUP" => {
|
||||
let (port, extra_key) = match stream_type(&req.uri) {
|
||||
Some("audio") => (AUDIO_PORT, "X-SS-Ping-Payload"),
|
||||
Some("video") => (VIDEO_PORT, "X-SS-Ping-Payload"),
|
||||
Some("control") => (CONTROL_PORT, "X-SS-Connect-Data"),
|
||||
_ => return response_status("404 Not Found", &req.cseq, &[], None),
|
||||
};
|
||||
let transport = format!("server_port={port}");
|
||||
response(
|
||||
&req.cseq,
|
||||
&[
|
||||
("Session", "DEADBEEFCAFE;timeout = 90"),
|
||||
("Transport", &transport),
|
||||
(extra_key, PING_PAYLOAD),
|
||||
],
|
||||
None,
|
||||
)
|
||||
}
|
||||
"ANNOUNCE" => {
|
||||
let cfg = parse_announce(&req.body);
|
||||
tracing::info!(
|
||||
width = cfg
|
||||
.get("x-nv-video[0].clientViewportWd")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
height = cfg
|
||||
.get("x-nv-video[0].clientViewportHt")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
fps = cfg
|
||||
.get("x-nv-video[0].maxFPS")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
bitrate_kbps = cfg
|
||||
.get("x-nv-vqos[0].bw.maximumBitrateKbps")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
packet_size = cfg
|
||||
.get("x-nv-video[0].packetSize")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
codec = cfg
|
||||
.get("x-nv-vqos[0].bitStreamFormat")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
fec_pct = cfg
|
||||
.get("x-nv-vqos[0].fec.repairPercent")
|
||||
.map(String::as_str)
|
||||
.unwrap_or("?"),
|
||||
"RTSP ANNOUNCE — negotiated stream config"
|
||||
);
|
||||
// TODO(P1.3): map `cfg` → lumen_core::Config and stash it for the media stages.
|
||||
let _ = state;
|
||||
response(&req.cseq, &[], None)
|
||||
}
|
||||
"PLAY" => {
|
||||
tracing::info!("RTSP PLAY — client ready; media streams would start here (P1.3)");
|
||||
response(&req.cseq, &[("Session", "DEADBEEFCAFE;timeout = 90")], None)
|
||||
}
|
||||
"TEARDOWN" => response(&req.cseq, &[], None),
|
||||
other => {
|
||||
tracing::warn!(method = other, "RTSP unsupported method");
|
||||
response_status("501 Not Implemented", &req.cseq, &[], None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Host capability SDP returned by DESCRIBE. Advertises HEVC + AV1 and no encryption
|
||||
/// (plaintext streams for now; P1.5 adds the negotiated AES paths).
|
||||
fn describe_sdp() -> String {
|
||||
// Line-oriented a=key:value, matching what moonlight-common-c scans for.
|
||||
[
|
||||
"a=x-ss-general.featureFlags:0",
|
||||
"a=x-ss-general.encryptionSupported:0",
|
||||
"a=x-ss-general.encryptionRequested:0",
|
||||
"sprop-parameter-sets=AAAAAU", // HEVC capability indicator
|
||||
"a=rtpmap:98 AV1/90000", // AV1 capability indicator
|
||||
"",
|
||||
]
|
||||
.join("\r\n")
|
||||
}
|
||||
|
||||
/// Parse an ANNOUNCE SDP body's `a=key:value` lines into a map.
|
||||
fn parse_announce(body: &str) -> HashMap<String, String> {
|
||||
let mut map = HashMap::new();
|
||||
for line in body.lines() {
|
||||
if let Some(rest) = line.strip_prefix("a=") {
|
||||
if let Some((k, v)) = rest.split_once(':') {
|
||||
map.insert(k.to_string(), v.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
map
|
||||
}
|
||||
|
||||
/// Extract the stream type from a SETUP URI like `…/streamid=video/0/0`.
|
||||
fn stream_type(uri: &str) -> Option<&str> {
|
||||
let after = uri.split("streamid=").nth(1)?;
|
||||
let token = after.split('/').next()?;
|
||||
match token {
|
||||
"audio" | "video" | "control" => Some(token),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn response(cseq: &str, headers: &[(&str, &str)], body: Option<&str>) -> String {
|
||||
response_status("200 OK", cseq, headers, body)
|
||||
}
|
||||
|
||||
fn response_status(
|
||||
status: &str,
|
||||
cseq: &str,
|
||||
headers: &[(&str, &str)],
|
||||
body: Option<&str>,
|
||||
) -> String {
|
||||
let body = body.unwrap_or("");
|
||||
let mut out = format!("RTSP/1.0 {status}\r\nCSeq: {cseq}\r\n");
|
||||
for (k, v) in headers {
|
||||
out.push_str(&format!("{k}: {v}\r\n"));
|
||||
}
|
||||
out.push_str(&format!("Content-Length: {}\r\n\r\n", body.len()));
|
||||
out.push_str(body);
|
||||
out
|
||||
}
|
||||
|
||||
fn find_subslice(hay: &[u8], needle: &[u8]) -> Option<usize> {
|
||||
hay.windows(needle.len()).position(|w| w == needle)
|
||||
}
|
||||
|
||||
fn header_value<'a>(head: &'a str, key_lower: &str) -> Option<&'a str> {
|
||||
head.split("\r\n").find_map(|line| {
|
||||
let (k, v) = line.split_once(':')?;
|
||||
(k.trim().eq_ignore_ascii_case(key_lower)).then(|| v.trim_start())
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
//! The `/serverinfo` capability/status XML Moonlight GETs before pairing and each launch.
|
||||
|
||||
use super::{Host, APP_VERSION, GFE_VERSION, SERVER_CODEC_MODE_SUPPORT};
|
||||
|
||||
/// Build the `<root status_code="200">…</root>` serverinfo document. `https` selects the
|
||||
/// paired-HTTPS variant (real MAC). Element names are case-sensitive and match what
|
||||
/// moonlight-common-c parses.
|
||||
pub fn serverinfo_xml(host: &Host, https: bool) -> String {
|
||||
// MAC is hidden over plain HTTP; PairStatus reflects the pairing store once the HTTPS
|
||||
// path carries per-client identity (a hardening follow-up — 0 for now).
|
||||
let mac = if https {
|
||||
"01:02:03:04:05:06"
|
||||
} else {
|
||||
"00:00:00:00:00:00"
|
||||
};
|
||||
// Over the mutual-TLS HTTPS port the peer is an authenticated (paired) client.
|
||||
let pair_status = u8::from(https);
|
||||
format!(
|
||||
r#"<?xml version="1.0" encoding="utf-8"?>
|
||||
<root status_code="200">
|
||||
<hostname>{hostname}</hostname>
|
||||
<appversion>{APP_VERSION}</appversion>
|
||||
<GfeVersion>{GFE_VERSION}</GfeVersion>
|
||||
<uniqueid>{uniqueid}</uniqueid>
|
||||
<HttpsPort>{https_port}</HttpsPort>
|
||||
<ExternalPort>{http_port}</ExternalPort>
|
||||
<MaxLumaPixelsHEVC>1869449984</MaxLumaPixelsHEVC>
|
||||
<mac>{mac}</mac>
|
||||
<LocalIP>{local_ip}</LocalIP>
|
||||
<ServerCodecModeSupport>{SERVER_CODEC_MODE_SUPPORT}</ServerCodecModeSupport>
|
||||
<PairStatus>{pair_status}</PairStatus>
|
||||
<currentgame>0</currentgame>
|
||||
<state>SUNSHINE_SERVER_FREE</state>
|
||||
</root>
|
||||
"#,
|
||||
hostname = host.hostname,
|
||||
uniqueid = host.uniqueid,
|
||||
https_port = host.https_port,
|
||||
http_port = host.http_port,
|
||||
local_ip = host.local_ip,
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
//! TLS for the HTTPS nvhttp port (47984). Moonlight does **mutual TLS** — it presents its
|
||||
//! client cert and expects the server to request one — so a plain server-auth config makes
|
||||
//! the post-pairing `pairchallenge` fail. This config requests the client cert and verifies
|
||||
//! the client owns its key, but (for now) accepts any well-formed cert; enforcing the
|
||||
//! paired allow-list (rejecting unpaired clients on /launch) is a follow-up hardening step.
|
||||
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use rustls::client::danger::HandshakeSignatureValid;
|
||||
use rustls::crypto::{verify_tls12_signature, verify_tls13_signature, CryptoProvider};
|
||||
use rustls::pki_types::{CertificateDer, UnixTime};
|
||||
use rustls::server::danger::{ClientCertVerified, ClientCertVerifier};
|
||||
use rustls::{DigitallySignedStruct, DistinguishedName, ServerConfig, SignatureScheme};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Requests + signature-checks the client cert but accepts any (the pairing handshake is
|
||||
/// the real proof). Pinning to the paired set is a hardening follow-up.
|
||||
#[derive(Debug)]
|
||||
struct AcceptAnyClientCert {
|
||||
provider: Arc<CryptoProvider>,
|
||||
}
|
||||
|
||||
impl ClientCertVerifier for AcceptAnyClientCert {
|
||||
fn offer_client_auth(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn client_auth_mandatory(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn root_hint_subjects(&self) -> &[DistinguishedName] {
|
||||
&[]
|
||||
}
|
||||
|
||||
fn verify_client_cert(
|
||||
&self,
|
||||
_end_entity: &CertificateDer,
|
||||
_intermediates: &[CertificateDer],
|
||||
_now: UnixTime,
|
||||
) -> Result<ClientCertVerified, rustls::Error> {
|
||||
Ok(ClientCertVerified::assertion())
|
||||
}
|
||||
|
||||
fn verify_tls12_signature(
|
||||
&self,
|
||||
message: &[u8],
|
||||
cert: &CertificateDer,
|
||||
dss: &DigitallySignedStruct,
|
||||
) -> Result<HandshakeSignatureValid, rustls::Error> {
|
||||
verify_tls12_signature(
|
||||
message,
|
||||
cert,
|
||||
dss,
|
||||
&self.provider.signature_verification_algorithms,
|
||||
)
|
||||
}
|
||||
|
||||
fn verify_tls13_signature(
|
||||
&self,
|
||||
message: &[u8],
|
||||
cert: &CertificateDer,
|
||||
dss: &DigitallySignedStruct,
|
||||
) -> Result<HandshakeSignatureValid, rustls::Error> {
|
||||
verify_tls13_signature(
|
||||
message,
|
||||
cert,
|
||||
dss,
|
||||
&self.provider.signature_verification_algorithms,
|
||||
)
|
||||
}
|
||||
|
||||
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
|
||||
self.provider
|
||||
.signature_verification_algorithms
|
||||
.supported_schemes()
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a mutual-TLS `ServerConfig` presenting the host cert/key.
|
||||
pub fn server_config(cert_pem: &str, key_pem: &str) -> Result<Arc<ServerConfig>> {
|
||||
let provider = Arc::new(rustls::crypto::aws_lc_rs::default_provider());
|
||||
let certs = rustls_pemfile::certs(&mut cert_pem.as_bytes())
|
||||
.collect::<std::result::Result<Vec<_>, _>>()
|
||||
.context("parse host cert PEM")?;
|
||||
let key = rustls_pemfile::private_key(&mut key_pem.as_bytes())
|
||||
.context("parse host key PEM")?
|
||||
.ok_or_else(|| anyhow!("no private key in host key PEM"))?;
|
||||
|
||||
let verifier = Arc::new(AcceptAnyClientCert {
|
||||
provider: provider.clone(),
|
||||
});
|
||||
let config = ServerConfig::builder_with_provider(provider)
|
||||
.with_safe_default_protocol_versions()
|
||||
.context("rustls protocol versions")?
|
||||
.with_client_cert_verifier(verifier)
|
||||
.with_single_cert(certs, key)
|
||||
.context("rustls server cert")?;
|
||||
Ok(Arc::new(config))
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
//! GameStream video wire packetization: an encoded access unit → UDP datagrams a stock
|
||||
//! Moonlight client decodes. Each datagram is
|
||||
//! `RTP_PACKET(12, big-endian) + reserved[4] + NV_VIDEO_PACKET(16, little-endian) + payload`
|
||||
//! and the frame's bitstream is prefixed with an 8-byte `video_short_frame_header_t`, then
|
||||
//! striped into ≤4 FEC blocks of ≤255 data shards. Byte-exact spec:
|
||||
//! `docs/research/gamestream-protocol-research.json` (video plane).
|
||||
//!
|
||||
//! P1.3 sends **data shards only** (`fecPercentage = 0`): on a clean LAN the client has
|
||||
//! every data shard and never runs Reed–Solomon recovery, so we get a decodable frame
|
||||
//! without matching Moonlight's `nanors` parity matrix (that interop work is P1.5). Plaintext
|
||||
//! only (encryption negotiated off for now). This lives in lumen-host for fast iteration;
|
||||
//! the wire codec moves into lumen-core (the P1 wire mode) once proven.
|
||||
|
||||
/// RTP `header` byte: version 2 (0x80) | extension (0x10) — Moonlight keys on the extension.
|
||||
const RTP_HEADER_BYTE: u8 = 0x80 | 0x10;
|
||||
const FLAG_PIC: u8 = 0x1;
|
||||
const FLAG_EOF: u8 = 0x2;
|
||||
const FLAG_SOF: u8 = 0x4;
|
||||
const MULTI_FEC_FLAGS: u8 = 0x10;
|
||||
const MAX_DATA_SHARDS_PER_BLOCK: usize = 255;
|
||||
const MAX_FEC_BLOCKS: usize = 4;
|
||||
/// Per-shard header: RTP(12) + reserved(4) + NV_VIDEO_PACKET(16).
|
||||
const SHARD_HEADER: usize = 32;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum FrameType {
|
||||
Idr,
|
||||
P,
|
||||
}
|
||||
|
||||
/// Splits encoded access units into GameStream video datagrams.
|
||||
pub struct VideoPacketizer {
|
||||
/// Negotiated `packetSize` (ANNOUNCE `x-nv-video[0].packetSize`).
|
||||
packet_size: usize,
|
||||
/// Per-shard payload bytes = `blocksize - SHARD_HEADER`, `blocksize = packetSize + 16`.
|
||||
payload_per_shard: usize,
|
||||
frame_index: u32,
|
||||
/// Monotonic per-stream packet counter (the RTP sequence / streamPacketIndex source).
|
||||
seq: u32,
|
||||
}
|
||||
|
||||
impl VideoPacketizer {
|
||||
pub fn new(packet_size: usize) -> Self {
|
||||
VideoPacketizer {
|
||||
packet_size,
|
||||
payload_per_shard: packet_size + 16 - SHARD_HEADER,
|
||||
frame_index: 0,
|
||||
seq: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Packetize one encoded AU into wire datagrams (ready for UDP send).
|
||||
pub fn packetize(
|
||||
&mut self,
|
||||
au: &[u8],
|
||||
frame_type: FrameType,
|
||||
timestamp_90k: u32,
|
||||
) -> Vec<Vec<u8>> {
|
||||
let frame_index = self.frame_index;
|
||||
self.frame_index = self.frame_index.wrapping_add(1);
|
||||
let pps = self.payload_per_shard;
|
||||
|
||||
// frame payload = 8-byte short frame header + the AU bitstream.
|
||||
let total_len = 8 + au.len();
|
||||
let last_payload_len = match total_len % pps {
|
||||
0 => pps,
|
||||
r => r,
|
||||
};
|
||||
let mut fp = Vec::with_capacity(total_len);
|
||||
fp.extend_from_slice(&short_frame_header(frame_type, last_payload_len as u16));
|
||||
fp.extend_from_slice(au);
|
||||
|
||||
let total_data = total_len.div_ceil(pps).max(1);
|
||||
let n_blocks = total_data
|
||||
.div_ceil(MAX_DATA_SHARDS_PER_BLOCK)
|
||||
.clamp(1, MAX_FEC_BLOCKS);
|
||||
let per_block = total_data.div_ceil(n_blocks);
|
||||
|
||||
let mut packets = Vec::with_capacity(total_data);
|
||||
for b in 0..n_blocks {
|
||||
let first = b * per_block;
|
||||
let last = ((b + 1) * per_block).min(total_data);
|
||||
if first >= last {
|
||||
break;
|
||||
}
|
||||
let block_data_count = last - first;
|
||||
for (fec_index, shard) in (first..last).enumerate() {
|
||||
let start = shard * pps;
|
||||
let end = (start + pps).min(fp.len());
|
||||
let mut payload = vec![0u8; pps]; // last shard zero-padded
|
||||
payload[..end - start].copy_from_slice(&fp[start..end]);
|
||||
|
||||
let mut flags = FLAG_PIC;
|
||||
if shard == 0 {
|
||||
flags |= FLAG_SOF;
|
||||
}
|
||||
if shard == total_data - 1 {
|
||||
flags |= FLAG_EOF;
|
||||
}
|
||||
let multi_fec_blocks = ((b as u8) << 4) | (((n_blocks - 1) as u8) << 6);
|
||||
// fecInfo: dataShards<<22 | fecIndex<<12 | fecPercentage<<4 (pct = 0).
|
||||
let fec_info: u32 = ((block_data_count as u32) << 22) | ((fec_index as u32) << 12);
|
||||
let seq = self.seq;
|
||||
self.seq = self.seq.wrapping_add(1);
|
||||
|
||||
packets.push(build_packet(
|
||||
seq,
|
||||
timestamp_90k,
|
||||
frame_index,
|
||||
flags,
|
||||
multi_fec_blocks,
|
||||
fec_info,
|
||||
&payload,
|
||||
));
|
||||
}
|
||||
}
|
||||
packets
|
||||
}
|
||||
}
|
||||
|
||||
/// 8-byte `video_short_frame_header_t` (little-endian), prefixed to the AU bitstream.
|
||||
fn short_frame_header(frame_type: FrameType, last_payload_len: u16) -> [u8; 8] {
|
||||
let mut h = [0u8; 8];
|
||||
h[0] = 0x01; // headerType
|
||||
h[1..3].copy_from_slice(&0u16.to_le_bytes()); // frame_processing_latency
|
||||
h[3] = match frame_type {
|
||||
FrameType::Idr => 2,
|
||||
FrameType::P => 1,
|
||||
};
|
||||
h[4..6].copy_from_slice(&last_payload_len.to_le_bytes());
|
||||
// h[6..8] unknown = 0
|
||||
h
|
||||
}
|
||||
|
||||
/// Build one wire datagram: RTP(BE) + reserved + NV_VIDEO_PACKET(LE) + payload.
|
||||
fn build_packet(
|
||||
seq: u32,
|
||||
timestamp_90k: u32,
|
||||
frame_index: u32,
|
||||
flags: u8,
|
||||
multi_fec_blocks: u8,
|
||||
fec_info: u32,
|
||||
payload: &[u8],
|
||||
) -> Vec<u8> {
|
||||
let mut p = Vec::with_capacity(SHARD_HEADER + payload.len());
|
||||
// --- RTP_PACKET (12 bytes, big-endian) ---
|
||||
p.push(RTP_HEADER_BYTE); // header
|
||||
p.push(0); // packetType (unused for video)
|
||||
p.extend_from_slice(&(seq as u16).to_be_bytes()); // sequenceNumber
|
||||
p.extend_from_slice(×tamp_90k.to_be_bytes()); // timestamp (90 kHz)
|
||||
p.extend_from_slice(&0u32.to_be_bytes()); // ssrc
|
||||
// --- reserved[4] ---
|
||||
p.extend_from_slice(&[0u8; 4]);
|
||||
// --- NV_VIDEO_PACKET (16 bytes, little-endian) ---
|
||||
p.extend_from_slice(&(seq << 8).to_le_bytes()); // streamPacketIndex (low byte 0)
|
||||
p.extend_from_slice(&frame_index.to_le_bytes()); // frameIndex
|
||||
p.push(flags);
|
||||
p.push(0); // extraFlags
|
||||
p.push(MULTI_FEC_FLAGS);
|
||||
p.push(multi_fec_blocks);
|
||||
p.extend_from_slice(&fec_info.to_le_bytes()); // fecInfo
|
||||
// --- payload ---
|
||||
p.extend_from_slice(payload);
|
||||
p
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn single_block_layout() {
|
||||
let mut pk = VideoPacketizer::new(1392); // payload_per_shard = 1392+16-32 = 1376
|
||||
assert_eq!(pk.payload_per_shard, 1376);
|
||||
let au = vec![0xABu8; 4000]; // 8+4000 = 4008 → ceil(4008/1376) = 3 data shards
|
||||
let pkts = pk.packetize(&au, FrameType::Idr, 90_000);
|
||||
assert_eq!(pkts.len(), 3);
|
||||
// Every datagram is SHARD_HEADER + payload_per_shard.
|
||||
for p in &pkts {
|
||||
assert_eq!(p.len(), SHARD_HEADER + 1376);
|
||||
assert_eq!(p[0], 0x90); // RTP header byte
|
||||
}
|
||||
// First packet: SOF set, fecIndex 0, frameIndex 0.
|
||||
let first = &pkts[0];
|
||||
assert_eq!(first[24] & FLAG_SOF, FLAG_SOF);
|
||||
assert_eq!(first[24] & FLAG_PIC, FLAG_PIC);
|
||||
let frame_index = u32::from_le_bytes(first[20..24].try_into().unwrap());
|
||||
assert_eq!(frame_index, 0);
|
||||
let fec_info = u32::from_le_bytes(first[28..32].try_into().unwrap());
|
||||
assert_eq!(fec_info >> 22, 3); // dataShards = 3
|
||||
assert_eq!((fec_info >> 12) & 0x3ff, 0); // fecIndex 0
|
||||
// Last packet: EOF set, fecIndex 2.
|
||||
let last = &pkts[2];
|
||||
assert_eq!(last[24] & FLAG_EOF, FLAG_EOF);
|
||||
let fec_info_last = u32::from_le_bytes(last[28..32].try_into().unwrap());
|
||||
assert_eq!((fec_info_last >> 12) & 0x3ff, 2);
|
||||
// RTP sequence numbers are 0,1,2.
|
||||
for (i, p) in pkts.iter().enumerate() {
|
||||
assert_eq!(u16::from_be_bytes(p[2..4].try_into().unwrap()), i as u16);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multi_block_split() {
|
||||
let mut pk = VideoPacketizer::new(1392);
|
||||
// Need > 255 data shards → multi-block. 255*1376 ≈ 351 KB; use 600 KB.
|
||||
let au = vec![0u8; 600_000];
|
||||
let pkts = pk.packetize(&au, FrameType::P, 0);
|
||||
let total = (8 + au.len()).div_ceil(1376);
|
||||
assert_eq!(pkts.len(), total);
|
||||
// n_blocks = ceil(total/255), clamped to 4; check multiFecBlocks lastBlock nibble.
|
||||
let n_blocks = total.div_ceil(255).clamp(1, 4);
|
||||
let last_block = ((pkts.last().unwrap()[27]) >> 6) & 0x3;
|
||||
assert_eq!(last_block as usize, n_blocks - 1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
//! M0 — the pipeline spike (plan §8): capture → NVENC encode → playable file, with the
|
||||
//! encoded access units also fed through a `lumen_core` host→client `Session` over an
|
||||
//! in-process loopback to prove the core's FEC + packetize + reassemble path on real
|
||||
//! encoder output.
|
||||
//!
|
||||
//! This is the spike runner, not the M2 hot path: it drives the stages on one thread (the
|
||||
//! per-stage-thread pipeline with bounded channels is [`crate::pipeline`]). Source is
|
||||
//! either a synthetic BGRx test pattern (no capture session needed) or the live xdg
|
||||
//! ScreenCast portal monitor.
|
||||
|
||||
use crate::capture::{self, Capturer, SyntheticCapturer};
|
||||
use crate::encode::{self, Codec, EncodedFrame, Encoder};
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use lumen_core::packet::{FLAG_PIC, FLAG_SOF};
|
||||
use lumen_core::{Config, Role, Session};
|
||||
use std::fs::File;
|
||||
use std::io::{BufWriter, Write};
|
||||
use std::path::PathBuf;
|
||||
use std::time::Instant;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Source {
|
||||
/// Deterministic moving BGRx test pattern — no capture session required.
|
||||
Synthetic,
|
||||
/// Live monitor via the xdg ScreenCast portal + PipeWire.
|
||||
Portal,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Options {
|
||||
pub source: Source,
|
||||
/// Synthetic-only; the portal source uses the PipeWire-negotiated size.
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub fps: u32,
|
||||
pub seconds: u32,
|
||||
pub codec: Codec,
|
||||
pub bitrate_bps: u64,
|
||||
/// Raw Annex-B elementary-stream sink (`.h265`/`.h264`/`.ivf-less .obu`); playable.
|
||||
pub out: PathBuf,
|
||||
/// Also round-trip every AU through a `lumen_core` host→client loopback and verify.
|
||||
pub loopback: bool,
|
||||
}
|
||||
|
||||
pub fn run(opts: Options) -> Result<()> {
|
||||
let mut capturer: Box<dyn Capturer> = match opts.source {
|
||||
Source::Synthetic => {
|
||||
tracing::info!(
|
||||
width = opts.width,
|
||||
height = opts.height,
|
||||
fps = opts.fps,
|
||||
"M0 source: synthetic BGRx test pattern"
|
||||
);
|
||||
Box::new(SyntheticCapturer::new(opts.width, opts.height, opts.fps))
|
||||
}
|
||||
Source::Portal => {
|
||||
tracing::info!("M0 source: xdg ScreenCast portal (live monitor)");
|
||||
capture::open_portal_monitor().context("open portal capturer")?
|
||||
}
|
||||
};
|
||||
|
||||
// The first frame establishes the authoritative dimensions (the portal's negotiated
|
||||
// size, or the synthetic size) used to configure the encoder.
|
||||
let first = capturer.next_frame().context("capture first frame")?;
|
||||
let (w, h) = (first.width, first.height);
|
||||
tracing::info!(
|
||||
width = w,
|
||||
height = h,
|
||||
format = ?first.format,
|
||||
codec = ?opts.codec,
|
||||
bitrate_bps = opts.bitrate_bps,
|
||||
"opening NVENC encoder"
|
||||
);
|
||||
let mut encoder =
|
||||
encode::open_video(opts.codec, first.format, w, h, opts.fps, opts.bitrate_bps)
|
||||
.context("open encoder")?;
|
||||
|
||||
let mut sink = BufWriter::new(
|
||||
File::create(&opts.out).with_context(|| format!("create {}", opts.out.display()))?,
|
||||
);
|
||||
|
||||
let mut lb = if opts.loopback {
|
||||
Some(Loopback::new().context("build lumen-core loopback")?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let target_frames = (opts.seconds as u64) * (opts.fps as u64);
|
||||
let started = Instant::now();
|
||||
let mut stats = Stats::default();
|
||||
|
||||
let mut frame = first;
|
||||
loop {
|
||||
encoder.submit(&frame).context("encoder submit")?;
|
||||
stats.submitted += 1;
|
||||
drain_encoder(encoder.as_mut(), &mut sink, lb.as_mut(), &mut stats)?;
|
||||
if stats.submitted >= target_frames {
|
||||
break;
|
||||
}
|
||||
frame = capturer.next_frame().context("capture frame")?;
|
||||
}
|
||||
|
||||
// NVENC buffers frames internally even at delay=0 — flush and drain the tail.
|
||||
encoder.flush().context("encoder flush")?;
|
||||
drain_encoder(encoder.as_mut(), &mut sink, lb.as_mut(), &mut stats)?;
|
||||
sink.flush().context("flush output file")?;
|
||||
|
||||
let elapsed = started.elapsed().as_secs_f64();
|
||||
tracing::info!(
|
||||
submitted = stats.submitted,
|
||||
encoded = stats.encoded,
|
||||
keyframes = stats.keyframes,
|
||||
bytes_out = stats.bytes_out,
|
||||
out = %opts.out.display(),
|
||||
elapsed_s = format!("{elapsed:.2}"),
|
||||
encode_fps = format!("{:.1}", stats.encoded as f64 / elapsed.max(1e-9)),
|
||||
"M0 capture→encode→file complete"
|
||||
);
|
||||
|
||||
if let Some(lb) = lb {
|
||||
lb.report();
|
||||
if lb.mismatches > 0 || lb.recovered != lb.submitted {
|
||||
return Err(anyhow!(
|
||||
"lumen-core loopback verification FAILED: {} mismatches, {}/{} AUs recovered",
|
||||
lb.mismatches,
|
||||
lb.recovered,
|
||||
lb.submitted
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Stats {
|
||||
submitted: u64,
|
||||
encoded: u64,
|
||||
keyframes: u64,
|
||||
bytes_out: u64,
|
||||
}
|
||||
|
||||
fn drain_encoder(
|
||||
encoder: &mut dyn Encoder,
|
||||
sink: &mut impl Write,
|
||||
mut lb: Option<&mut Loopback>,
|
||||
stats: &mut Stats,
|
||||
) -> Result<()> {
|
||||
while let Some(au) = encoder.poll().context("encoder poll")? {
|
||||
sink.write_all(&au.data).context("write AU to file")?;
|
||||
stats.encoded += 1;
|
||||
stats.bytes_out += au.data.len() as u64;
|
||||
if au.keyframe {
|
||||
stats.keyframes += 1;
|
||||
}
|
||||
if let Some(lb) = lb.as_deref_mut() {
|
||||
lb.submit(&au)?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A host↔client `lumen_core` pair over a lossless in-process loopback. Each encoded AU is
|
||||
/// FEC-protected, packetized, sent, then reassembled on the client and byte-compared to the
|
||||
/// original — exercising the core on real encoder output (the M0 "feed into a Session" goal).
|
||||
struct Loopback {
|
||||
host: Session,
|
||||
client: Session,
|
||||
submitted: u64,
|
||||
recovered: u64,
|
||||
mismatches: u64,
|
||||
bytes: u64,
|
||||
}
|
||||
|
||||
impl Loopback {
|
||||
fn new() -> Result<Loopback> {
|
||||
let (host_tx, client_tx) = lumen_core::transport::loopback_pair(0, 0);
|
||||
let host = Session::new(Config::p1_defaults(Role::Host), Box::new(host_tx))
|
||||
.map_err(|e| anyhow!("host session: {e:?}"))?;
|
||||
let client = Session::new(Config::p1_defaults(Role::Client), Box::new(client_tx))
|
||||
.map_err(|e| anyhow!("client session: {e:?}"))?;
|
||||
Ok(Loopback {
|
||||
host,
|
||||
client,
|
||||
submitted: 0,
|
||||
recovered: 0,
|
||||
mismatches: 0,
|
||||
bytes: 0,
|
||||
})
|
||||
}
|
||||
|
||||
fn submit(&mut self, au: &EncodedFrame) -> Result<()> {
|
||||
let mut flags = FLAG_PIC as u32;
|
||||
if au.keyframe {
|
||||
flags |= FLAG_SOF as u32;
|
||||
}
|
||||
self.host
|
||||
.submit_frame(&au.data, au.pts_ns, flags)
|
||||
.map_err(|e| anyhow!("host submit_frame: {e:?}"))?;
|
||||
self.submitted += 1;
|
||||
self.bytes += au.data.len() as u64;
|
||||
|
||||
// Lossless + in-order loopback: each submit yields exactly the AU just sent.
|
||||
loop {
|
||||
match self.client.poll_frame() {
|
||||
Ok(frame) => {
|
||||
self.recovered += 1;
|
||||
if frame.data != au.data {
|
||||
self.mismatches += 1;
|
||||
tracing::warn!(
|
||||
recovered = self.recovered,
|
||||
got = frame.data.len(),
|
||||
expected = au.data.len(),
|
||||
"loopback AU mismatch"
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(lumen_core::LumenError::NoFrame) => break,
|
||||
Err(e) => return Err(anyhow!("client poll_frame: {e:?}")),
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn report(&self) {
|
||||
tracing::info!(
|
||||
submitted = self.submitted,
|
||||
recovered = self.recovered,
|
||||
mismatches = self.mismatches,
|
||||
bytes = self.bytes,
|
||||
"lumen-core loopback: AUs FEC-packetized → sent → reassembled & verified"
|
||||
);
|
||||
}
|
||||
}
|
||||
+148
-29
@@ -6,20 +6,26 @@
|
||||
//! `#[cfg(target_os = "linux")]`; the crate compiles everywhere so the workspace builds
|
||||
//! on non-Linux dev machines — it just can't run the pipeline there.
|
||||
//!
|
||||
//! Status: scaffold. M0 wires capture→encode→file; M2 wires the full P1 host that a
|
||||
//! stock Moonlight client connects to.
|
||||
//! Status: M0. The `m0` subcommand runs the capture→encode→file pipeline spike and feeds
|
||||
//! the encoded AUs through a `lumen_core` loopback. M2 wires the full P1 host that a stock
|
||||
//! Moonlight client connects to.
|
||||
|
||||
// Scaffold: trait methods and config paths are defined ahead of their backends.
|
||||
#![allow(dead_code)]
|
||||
|
||||
mod capture;
|
||||
mod encode;
|
||||
mod gamestream;
|
||||
mod inject;
|
||||
mod m0;
|
||||
mod pipeline;
|
||||
mod vdisplay;
|
||||
mod web;
|
||||
|
||||
use vdisplay::{Compositor, Mode};
|
||||
use anyhow::{bail, Result};
|
||||
use encode::Codec;
|
||||
use m0::{Options, Source};
|
||||
use std::path::PathBuf;
|
||||
|
||||
fn main() {
|
||||
tracing_subscriber::fmt()
|
||||
@@ -28,34 +34,147 @@ fn main() {
|
||||
)
|
||||
.init();
|
||||
|
||||
tracing::info!(
|
||||
"lumen-host scaffold (lumen_core ABI v{})",
|
||||
lumen_core::ABI_VERSION
|
||||
);
|
||||
if let Err(e) = real_main() {
|
||||
tracing::error!("{e:#}");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// The intended startup sequence (each step is a separate, pluggable subsystem):
|
||||
// 1. negotiate mode + codec + FEC scheme over the control plane (web::WebConfig)
|
||||
// 2. vdisplay::open(compositor).create(mode) -> client-sized virtual output
|
||||
// 3. capture::open_pipewire(node) ; encode::open(codec, bitrate)
|
||||
// 4. build a lumen_core::Session (host role) over a UDP transport to the client
|
||||
// 5. loop pipeline::pump_once(..) until disconnect, then destroy the output
|
||||
let target_mode = Mode {
|
||||
width: 2560,
|
||||
height: 1440,
|
||||
refresh_hz: 240,
|
||||
fn real_main() -> Result<()> {
|
||||
tracing::info!("lumen-host (lumen_core ABI v{})", lumen_core::ABI_VERSION);
|
||||
|
||||
let args: Vec<String> = std::env::args().skip(1).collect();
|
||||
match args.first().map(String::as_str) {
|
||||
// M2 GameStream host control plane (P1.1: mDNS + serverinfo).
|
||||
Some("serve") => gamestream::serve(),
|
||||
// M0 pipeline spike.
|
||||
Some("m0") => m0::run(parse_m0(&args[1..])?),
|
||||
Some("-h") | Some("--help") | Some("help") | None => {
|
||||
print_usage();
|
||||
Ok(())
|
||||
}
|
||||
// Bare flags (no subcommand) default to the m0 spike for back-compat.
|
||||
Some(_) => m0::run(parse_m0(&args)?),
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_m0(args: &[String]) -> Result<Options> {
|
||||
let mut source = Source::Portal;
|
||||
let mut width = 1920u32;
|
||||
let mut height = 1080u32;
|
||||
let mut fps = 60u32;
|
||||
let mut seconds = 5u32;
|
||||
let mut codec = Codec::H265;
|
||||
let mut bitrate_mbps = 20u64;
|
||||
let mut out: Option<PathBuf> = None;
|
||||
let mut loopback = true;
|
||||
|
||||
let mut i = 0;
|
||||
while i < args.len() {
|
||||
let arg = args[i].as_str();
|
||||
let mut next = || {
|
||||
i += 1;
|
||||
args.get(i)
|
||||
.cloned()
|
||||
.ok_or_else(|| anyhow::anyhow!("missing value for {arg}"))
|
||||
};
|
||||
let compositor = Compositor::Kwin; // MVP target
|
||||
match arg {
|
||||
"--source" => {
|
||||
source = match next()?.as_str() {
|
||||
"synthetic" => Source::Synthetic,
|
||||
"portal" => Source::Portal,
|
||||
other => bail!("unknown --source '{other}' (synthetic|portal)"),
|
||||
}
|
||||
}
|
||||
"--width" => {
|
||||
width = next()?
|
||||
.parse()
|
||||
.map_err(|_| anyhow::anyhow!("bad --width"))?
|
||||
}
|
||||
"--height" => {
|
||||
height = next()?
|
||||
.parse()
|
||||
.map_err(|_| anyhow::anyhow!("bad --height"))?
|
||||
}
|
||||
"--fps" => fps = next()?.parse().map_err(|_| anyhow::anyhow!("bad --fps"))?,
|
||||
"--seconds" => {
|
||||
seconds = next()?
|
||||
.parse()
|
||||
.map_err(|_| anyhow::anyhow!("bad --seconds"))?
|
||||
}
|
||||
"--codec" => {
|
||||
codec = match next()?.as_str() {
|
||||
"h264" => Codec::H264,
|
||||
"h265" | "hevc" => Codec::H265,
|
||||
"av1" => Codec::Av1,
|
||||
other => bail!("unknown --codec '{other}' (h264|h265|av1)"),
|
||||
}
|
||||
}
|
||||
"--bitrate" => {
|
||||
bitrate_mbps = next()?
|
||||
.parse()
|
||||
.map_err(|_| anyhow::anyhow!("bad --bitrate (Mbps)"))?
|
||||
}
|
||||
"--out" => out = Some(PathBuf::from(next()?)),
|
||||
"--no-loopback" => loopback = false,
|
||||
"-h" | "--help" => {
|
||||
print_usage();
|
||||
std::process::exit(0);
|
||||
}
|
||||
other => bail!("unknown argument '{other}' (try --help)"),
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
|
||||
if cfg!(target_os = "linux") {
|
||||
tracing::info!(
|
||||
?compositor,
|
||||
?target_mode,
|
||||
"would create a virtual output and start streaming (backends pending M0/M2)"
|
||||
);
|
||||
} else {
|
||||
tracing::warn!(
|
||||
"this is a Linux host; on {} only the shared lumen_core builds and is testable",
|
||||
std::env::consts::OS
|
||||
if fps == 0 || width == 0 || height == 0 || seconds == 0 {
|
||||
bail!("--fps/--width/--height/--seconds must be > 0");
|
||||
}
|
||||
|
||||
let out = out.unwrap_or_else(|| {
|
||||
let ext = match codec {
|
||||
Codec::H264 => "h264",
|
||||
Codec::H265 => "h265",
|
||||
Codec::Av1 => "obu",
|
||||
};
|
||||
PathBuf::from(format!("/tmp/lumen-m0.{ext}"))
|
||||
});
|
||||
|
||||
Ok(Options {
|
||||
source,
|
||||
width,
|
||||
height,
|
||||
fps,
|
||||
seconds,
|
||||
codec,
|
||||
bitrate_bps: bitrate_mbps.saturating_mul(1_000_000),
|
||||
out,
|
||||
loopback,
|
||||
})
|
||||
}
|
||||
|
||||
fn print_usage() {
|
||||
eprintln!(
|
||||
"lumen-host — Linux streaming host
|
||||
|
||||
USAGE:
|
||||
lumen-host serve GameStream host control plane (M2: mDNS + serverinfo …)
|
||||
lumen-host m0 [OPTIONS] M0 capture→encode→file pipeline spike
|
||||
|
||||
OPTIONS:
|
||||
--source <synthetic|portal> frame source (default: portal)
|
||||
--seconds <N> capture duration in seconds (default: 5)
|
||||
--fps <N> target frame rate (default: 60)
|
||||
--codec <h264|h265|av1> NVENC codec (default: h265)
|
||||
--bitrate <MBPS> target bitrate in Mbps (default: 20)
|
||||
--width <W> --height <H> synthetic source size (default: 1920x1080)
|
||||
--out <PATH> raw Annex-B output (default: /tmp/lumen-m0.<ext>)
|
||||
--no-loopback skip the lumen_core round-trip verification
|
||||
-h, --help this help
|
||||
|
||||
NOTES:
|
||||
'portal' needs headless Sway + xdg-desktop-portal-wlr running in this session
|
||||
(see docs/linux-setup.md). 'synthetic' needs no capture session and always runs.
|
||||
Encoded AUs are written to a playable file AND (unless --no-loopback) fed through a
|
||||
lumen_core host→client loopback that reassembles and byte-verifies each one."
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+61
-12
@@ -38,6 +38,20 @@ cat /sys/module/nvidia_drm/parameters/modeset # must print Y after reboot
|
||||
```
|
||||
|
||||
- Driver **≥ 535** is the floor for headless wlroots (EGL/dmabuf); 550+ recommended.
|
||||
- **Install the NVIDIA GL/EGL userspace, not just `nvidia-utils`:**
|
||||
`sudo apt install libnvidia-gl-<NNN>` (matching the driver, e.g. `libnvidia-gl-595`).
|
||||
`nvidia-utils-NNN` ships nvidia-smi + NVENC but **not** `libEGL_nvidia.so.0` or the GLVND
|
||||
vendor JSON (`/usr/share/glvnd/egl_vendor.d/10_nvidia.json`). Without them libglvnd falls
|
||||
back to Mesa, wlroots can't init EGL on the GPU and drops to the **pixman** software
|
||||
renderer — and the ScreenCast portal then fails to negotiate a buffer format
|
||||
(`unable to receive a valid format from wlr_screencopy`). Verify after install:
|
||||
`ls /usr/share/glvnd/egl_vendor.d/10_nvidia.json && ldconfig -p | grep libEGL_nvidia`.
|
||||
A correct GPU Sway logs `EGL vendor: NVIDIA` and a list of DMA-BUF formats.
|
||||
- **Join the `render` + `video` groups:** `sudo usermod -aG render,video $USER`, then
|
||||
**re-login** (group changes only apply to new logins). wlroots opens
|
||||
`/dev/dri/renderD128` (group `render`) and `/dev/dri/card*` (group `video`), both 0660;
|
||||
without membership Sway aborts with `Permission denied`. (`scripts/headless/*.sh` bridge a
|
||||
not-yet-re-logged-in shell with `sg render`, but re-login is the clean fix.)
|
||||
- A **headless VM GPU exposes no DRM connectors** — that's expected. We don't use the DRM
|
||||
backend; `WLR_BACKENDS=headless` renders to an offscreen GBM/EGL surface and creates a
|
||||
virtual `HEADLESS-1` output. Use the render node `/dev/dri/renderD128`.
|
||||
@@ -49,20 +63,27 @@ cat /sys/module/nvidia_drm/parameters/modeset # must print Y after reboot
|
||||
## 3. Bring up the headless compositor + prove capture→NVENC
|
||||
|
||||
```sh
|
||||
# shell 1 — start headless Sway (prints WAYLAND_DISPLAY, default wayland-1)
|
||||
bash scripts/headless/run-headless-sway.sh
|
||||
# shell 1 — start headless GPU Sway on the shared user bus (blocks; -d for debug log)
|
||||
bash scripts/headless/run-headless-sway.sh # success logs "EGL vendor: NVIDIA"
|
||||
|
||||
# shell 2 — same user
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u) WAYLAND_DISPLAY=wayland-1
|
||||
swaymsg -t get_outputs # confirm HEADLESS-1
|
||||
swaymsg output HEADLESS-1 resolution 2560x1440@60Hz # set your client size
|
||||
# shell 2 — same user: set the client mode, import the portal env, write the env file
|
||||
bash scripts/headless/prepare-session.sh 2560x1440@60Hz
|
||||
source /tmp/lumen-sway-env.sh
|
||||
swaymsg -t get_outputs # confirm HEADLESS-1 active
|
||||
swaymsg exec foot # optional: animated content to capture
|
||||
bash scripts/headless/capture-smoke-test.sh # wf-recorder (wlr-screencopy) -> hevc_nvenc
|
||||
ffprobe /tmp/lumen-headless-test.mkv # confirm a real H.265 stream
|
||||
```
|
||||
|
||||
`wf-recorder` uses `wlr-screencopy` directly (no portal/D-Bus) — the fastest way to
|
||||
de-risk the GPU encode path. **Note:** screencopy encodes straight to a file and *cannot*
|
||||
feed PipeWire; the real integration uses the ScreenCast portal (see M0).
|
||||
feed PipeWire; the real integration uses the ScreenCast portal (see M0). If shell 1 logged
|
||||
a Mesa/EGL fallback (or Sway dropped to pixman) instead of `EGL vendor: NVIDIA`, install the
|
||||
NVIDIA GL userspace (§2) — the portal cannot capture a pixman output.
|
||||
|
||||
**An idle headless output produces no frames** (its frame clock is driven by damage); give
|
||||
it a real refresh mode (`prepare-session.sh` does) *and* run something animated
|
||||
(`swaymsg exec foot`) or the capture will be ~1 frame.
|
||||
|
||||
The wlroots-on-NVIDIA env workarounds (`WLR_RENDERER=gles2`, `WLR_NO_HARDWARE_CURSORS=1`,
|
||||
`GBM_BACKEND=nvidia-drm`, `sway --unsupported-gpu`, …) live in
|
||||
@@ -74,13 +95,41 @@ Goal (plan §8): headless output → PipeWire ScreenCast → NVENC → a playabl
|
||||
the encoded access units into a `lumen_core::Session` (host role). The module seams exist
|
||||
in `crates/lumen-host/src/{vdisplay,capture,encode,inject,pipeline}.rs`.
|
||||
|
||||
**Status: implemented and verified end-to-end** in `crates/lumen-host` (`m0.rs`,
|
||||
`capture/linux.rs`, `encode/linux.rs`). After the §3 bring-up:
|
||||
|
||||
```sh
|
||||
source /tmp/lumen-sway-env.sh
|
||||
swaymsg exec foot # animated content
|
||||
# Live portal capture → NVENC HEVC → playable file, with each AU also round-tripped
|
||||
# through a lumen_core host→client Session (FEC + packetize + reassemble) and verified:
|
||||
cargo run -p lumen-host -- m0 --source portal --seconds 5 --out /tmp/lumen-m0.h265
|
||||
ffprobe /tmp/lumen-m0.h265
|
||||
# No capture session needed (encode + core only): --source synthetic
|
||||
```
|
||||
|
||||
Verified result: `1920x1080` HEVC, ~300 frames in 5s, `lumen-core loopback … 0 mismatches`.
|
||||
The portal negotiates packed **`RGB` (24-bit, 3 bpp)** on wlroots; the encoder expands it to
|
||||
`rgb0` (one pad byte/pixel, no colour math) since NVENC accepts `rgb0`/`bgr0` but not
|
||||
`rgb24`. dmabuf zero-copy import is still deferred (plan §9) — this is the CPU-copy path.
|
||||
|
||||
Crate choices, verified current:
|
||||
- **Capture (portal path):** [`ashpd`](https://docs.rs/ashpd) **0.13** with the
|
||||
`screencast` + `pipewire` features → `ScreenCast::create_session` → `select_sources`
|
||||
(`Monitor`) → `start` → `pipe_wire_node_id()` + `open_pipe_wire_remote()`; pull frames
|
||||
with [`pipewire`](https://docs.rs/pipewire) **0.10**. (crates.io's "newest" field shows
|
||||
0.9 for ashpd — ignore it, pin `0.13`.) Set `XDG_CURRENT_DESKTOP=sway` so the wlr
|
||||
portal backend is chosen.
|
||||
`screencast` feature (the `pipewire` feature is *not* needed — `open_pipe_wire_remote`
|
||||
is unconditional). Flow (0.13 API, verified against the vendored source): `Screencast::new`
|
||||
→ `create_session(Default)` → `select_sources(&session, SelectSourcesOptions::default()
|
||||
.set_sources(BitFlags::from_flag(SourceType::Monitor))…)` → `start(&session, None,
|
||||
Default)` → `.response()?` → `Stream::pipe_wire_node_id()` + `open_pipe_wire_remote()`.
|
||||
Note 0.13 takes **options structs**, not the old positional args, and defaults to the
|
||||
**tokio** runtime — drive the handshake on a *multi-thread* tokio runtime (a
|
||||
current-thread one starves zbus's reader and the portal reports "Invalid session").
|
||||
Pull frames with [`pipewire`](https://docs.rs/pipewire) **0.9** — it must match the
|
||||
pipewire crate ashpd 0.13 links (the `pipewire-sys` `links` key is unique per build, so
|
||||
`0.10` fails to resolve). 0.9 uses `MainLoopRc`/`ContextRc::connect_fd_rc(OwnedFd)`/
|
||||
`StreamBox`. Only request `SourceType::Monitor` — the wlr backend's
|
||||
`AvailableSourceTypes` is `1` (Monitor only); asking for `Window`/`Virtual` invalidates
|
||||
the session. Set `XDG_CURRENT_DESKTOP=sway` so the wlr portal backend is chosen, and
|
||||
import it into the portal's environment (see "Portal bring-up" below).
|
||||
- **Encode:** [`ffmpeg-next`](https://crates.io/crates/ffmpeg-next) **7.x** (binds the
|
||||
system FFmpeg 6.1.1 via pkg-config; needs `clang`/`libclang`). Select the encoder by
|
||||
name — `encoder::find_by_name("hevc_nvenc")`, *not* by codec id (that's the SW encoder).
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# M2 — P1 host: stream to a stock Moonlight client
|
||||
|
||||
The shippable milestone (plan §8). A stock Moonlight/Artemis client discovers this host,
|
||||
pairs, launches, and gets video (then input, then audio) on a client-sized virtual display.
|
||||
Ground-truth protocol reference: [`research/gamestream-protocol-research.json`](research/gamestream-protocol-research.json)
|
||||
(distilled from Sunshine + moonlight-common-c source; cite those for byte-level detail).
|
||||
|
||||
## Architecture (respects the "one core" invariant)
|
||||
|
||||
- **lumen-core** gains a **P1 GameStream wire codec** (`ProtocolPhase::P1GameStream`, the
|
||||
hook already exists): the exact RTP+`NV_VIDEO_PACKET` framing, the GameStream FEC shard
|
||||
layout, and the video/audio AES-GCM/CBC paths. Hot path, native threads, **no async**.
|
||||
Kept beside lumen's native internal format (P2), selected by phase.
|
||||
- **lumen-host** gains the **control plane** (tokio/axum OK — I/O-bound, not the hot path):
|
||||
mDNS discovery, nvhttp serverinfo + the 4-phase pairing, the RTSP handshake, the ENet
|
||||
control stream + input injection, the virtual-display lifecycle, and Opus audio encode.
|
||||
|
||||
## Port map (base 47989; Moonlight derives all by offset)
|
||||
|
||||
| Port | Proto | Role |
|
||||
|---|---|---|
|
||||
| 47989 | TCP | HTTP nvhttp (unpaired: /serverinfo, /pair PIN flow) |
|
||||
| 47984 | TCP | HTTPS nvhttp (paired; **client-cert pinned**) — /launch, /resume, … |
|
||||
| 48010 | TCP | RTSP (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY) |
|
||||
| 47998 | UDP | Video RTP (+ RS-FEC, optional AES-GCM) |
|
||||
| 47999 | UDP | Audio RTP (Opus, RS-FEC 4+2, optional **AES-CBC**) |
|
||||
| 48000 | UDP | ENet control stream (AES-GCM) + remote input |
|
||||
| 5353 | UDP | mDNS `_nvstream._tcp.local` advertisement |
|
||||
|
||||
## Key wire facts (the non-obvious ones)
|
||||
|
||||
- **Video datagram** = `RTP_PACKET(12, BIG-endian)` + `reserved[4]` + `NV_VIDEO_PACKET(16,
|
||||
LITTLE-endian)` + payload. Endianness differs *within the same packet*. `header=0x80|0x10`.
|
||||
- `fecInfo` (u32 LE) = `(dataShards<<22)|(fecIndex<<12)|(fecPercentage<<4)`; parityShards is
|
||||
**recomputed** by the client as `ceil(dataShards*pct/100)` — must match exactly.
|
||||
- `multiFecBlocks` = `(blockIdx<<4)|((nBlocks-1)<<6)`; **≤4 FEC blocks/frame**, ≤255 shards/block.
|
||||
- Each frame's bitstream is prefixed with an 8-byte `video_short_frame_header_t`
|
||||
(`headerType=0x01`, `frameType` 2=IDR, `lastPayloadLen`) before striping into shards.
|
||||
- Shard size = `packetSize + 16`. Data shards first, then parity, over a contiguous RTP
|
||||
sequence range. Last data shard zero-padded.
|
||||
- **Video crypto** (when `SS_ENC_VIDEO` negotiated): AES-128-GCM, key = raw 16-byte RIKEY
|
||||
(from `/launch?rikey=`), IV = `counter_le[8]||0,0,0||'V'(0x56)`, **NO AAD**, 32-byte
|
||||
`ENC_VIDEO_HEADER{iv[12],frameNumber,tag[16]}` prefix; **FEC first, then encrypt per shard**.
|
||||
- **Pairing**: PIN key = `SHA-256(salt[16] || ascii_pin)[..16]`; AES-128-**ECB** (no padding)
|
||||
for the challenge blocks; SHA-256 rolling hashes; RSA-SHA256 signatures over X.509 certs;
|
||||
the client cert is pinned for subsequent HTTPS. 4 phases over `/pair?phrase=…`.
|
||||
- **RTSP** `Session: DEADBEEFCAFE;timeout = 90` (literal), `Transport: server_port=<p>`,
|
||||
`streamid=video/0/0` / `control/13/0`. ANNOUNCE carries the negotiated config
|
||||
(`x-nv-video[0].*`, `x-nv-vqos[0].*`) → maps to `lumen_core::Config`.
|
||||
|
||||
## The two highest interop risks (validate EARLY)
|
||||
|
||||
1. **RS-FEC matrix compatibility.** Sunshine + Moonlight both use **nanors** (GF(2⁸), poly
|
||||
0x11d, Vandermonde systematic). lumen-core uses `reed-solomon-erasure` (Cauchy) — parity
|
||||
bytes likely **don't match**, so Moonlight silently fails to recover any frame with a lost
|
||||
data shard. Mitigation: **on a clean LAN with no loss the client never runs RS decode**, so
|
||||
defer this — get a frame decoded first, then FFI/port nanors for loss recovery.
|
||||
2. **Crypto layout.** lumen's `SessionCrypto` (salt + seq-as-AAD) is wire-incompatible. P1
|
||||
needs a separate GameStream GCM path. Mitigation: **video encryption is negotiated and
|
||||
usually off on LAN** — implement plaintext video first, add GCM later.
|
||||
|
||||
## Phasing (each phase independently testable with a real Moonlight client)
|
||||
|
||||
- **P1.1 — Discovery + serverinfo + pairing.** mDNS `_nvstream._tcp`, HTTP/HTTPS nvhttp,
|
||||
`/serverinfo` XML, the 4-phase pairing + cert pinning. *Acceptance: Moonlight discovers,
|
||||
pairs (PIN), and shows the host as ready.* ← first slice.
|
||||
- **P1.2 — Launch + RTSP + virtual display.** `/launch` (parse rikey/rikeyid/mode), the RTSP
|
||||
handshake, negotiate `Config`, create a wlroots virtual output sized to the client.
|
||||
*Acceptance: Moonlight completes RTSP and the host stands up the UDP streams.*
|
||||
- **P1.3 — Video (lumen-core P1 codec), plaintext, clean-LAN.** RTP+NV framing + FEC shard
|
||||
layout in lumen-core; wire M0's NVENC AUs → UDP 47998. *Acceptance: Moonlight DISPLAYS video.*
|
||||
- **P1.4 — Control + input.** ENet (`rusty_enet`) control stream; decode input → `inject.rs`
|
||||
(uinput/reis); request-IDR → force NVENC keyframe. *Acceptance: mouse/keyboard work.*
|
||||
- **P1.5 — Robustness: FEC recovery + encryption.** nanors-exact FEC; per-shard AES-GCM.
|
||||
*Acceptance: stable under `tc netem` loss; encrypted streams.*
|
||||
- **P1.6 — Audio + polish.** Opus + audio RTP/FEC/CBC (UDP 47999); disconnect teardown; KWin
|
||||
backend for the user's KDE box. *Acceptance: full game stream with sound — the M2 goal.*
|
||||
|
||||
## Crates (verified available)
|
||||
|
||||
`mdns-sd` 0.20 (discovery) · `axum` 0.8 + `rustls` + `tokio-rustls` (nvhttp/HTTPS, custom
|
||||
`ClientCertVerifier` for pinning) · `rcgen` 0.14 + `x509-parser` 0.18 + `rsa`/`sha2`/`aes`/
|
||||
`ecb` (pairing crypto) · hand-rolled RTSP over `tokio::net::TcpListener` · `rusty_enet` 0.4
|
||||
(control) · `opus` 0.3 (audio) · `reis` 0.6 + `input-linux` (input) · `aes-gcm` (already in
|
||||
core) for the P1 video/control GCM path; nanors (FFI/port) for FEC recovery in P1.5.
|
||||
|
||||
## Testing note
|
||||
|
||||
The host is headless; end-to-end needs a **stock Moonlight client on the LAN** pointed at
|
||||
this box (manual "add host" by IP works without mDNS). P1.1 is testable with `curl` against
|
||||
`/serverinfo` + the Moonlight pair flow; P1.3+ needs a client that can display.
|
||||
@@ -0,0 +1,448 @@
|
||||
{
|
||||
"summary": "Extract the exact GameStream/Moonlight P1 host protocol from Sunshine + moonlight-common-c",
|
||||
"agentCount": 6,
|
||||
"logs": [
|
||||
"[research:control-input] failed: API Error: The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()"
|
||||
],
|
||||
"result": [
|
||||
{
|
||||
"area": "GameStream HTTP serverinfo + pairing handshake (host side, what stock Moonlight expects)",
|
||||
"summary": "A GameStream host runs two HTTP servers from the same NvHTTP code: plain HTTP on port 47989 (insecure, unauthenticated) and HTTPS with mutual TLS on 47984. Ports are derived from a base port (config default 47989) plus a signed offset: PORT_HTTP=0 -> 47989, PORT_HTTPS=-5 -> 47984. Moonlight first GETs /serverinfo over HTTP (before pairing) to read an XML document of capabilities and the host's pairing/running state; key fields it parses are hostname, appversion (its major version selects the pairing hash: >=7 -> SHA-256, else SHA-1; Sunshine advertises \"7.1.431.-1\"), GfeVersion, uniqueid, HttpsPort, ExternalPort, mac, MaxLumaPixelsHEVC, ServerCodecModeSupport (a bitmask: 3=H264-only, 259=+HEVC, 3843=+AV1), PairStatus, currentgame and state. Pairing is a 4-phase challenge/response over /pair driven entirely by repeated HTTP GETs with a `phrase` query param: getservercert, clientchallenge, serverchallengeresp, clientpairingsecret, followed by a final pairchallenge over HTTPS. The shared secret is an AES-128 key = SHA(salt(16) || PIN-as-utf8) truncated to 16 bytes; salt is client-generated random 16 bytes sent hex-encoded in phase 1. All pairing AES uses AES-128 in ECB mode with NO padding (inputs zero-extended to a 16-byte multiple, so a 32-byte SHA-256 hash is exactly two blocks). Each side proves it knows the PIN by exchanging encrypted random challenges and verifying SHA hashes that bind both X.509 cert signatures and a per-side 16-byte secret; the secrets are additionally RSA-SHA256-signed by each side's cert key and verified. On success the host stores the client's self-signed X.509 cert in an allow-list; thereafter every HTTPS request requires that exact client cert (cert pinning via a custom OpenSSL verify callback comparing the presented cert's PEM against stored authorized clients). All XML responses are an `<root status_code=\"200\">...</root>` tree; pairing replies carry `paired` (1/0) plus the phase-specific element (plaincert, challengeresponse, pairingsecret).",
|
||||
"ports": [
|
||||
"47989/tcp = HTTP (insecure NvHTTP), offset PORT_HTTP=0 from base port",
|
||||
"47984/tcp = HTTPS (mutual-TLS NvHTTP), offset PORT_HTTPS=-5 from base port",
|
||||
"Base/config port default = 47989; map_port(p) = config.port + p (uint16, warns if <1024 or >65535)",
|
||||
"Related stream ports (offsets, not part of this area but advertised/used post-launch): 48010/tcp RTSP (+21), 47998/udp video (+9), 47999/udp control (+10), 48000/udp audio (+11), 48002/udp mic (+13)",
|
||||
"Moonlight resolves stream ports via LiGetPortFromPortFlagIndex; TCP set {47984,47989,48010}, UDP set {47998,47999,48000,48010}"
|
||||
],
|
||||
"wire_formats": [
|
||||
{
|
||||
"name": "/serverinfo XML response",
|
||||
"layout": "<root status_code=\"200\">\n <hostname>...</hostname>\n <appversion>7.1.431.-1</appversion> (VERSION; major>=7 => client uses SHA-256)\n <GfeVersion>3.23.0.74</GfeVersion> (GFE_VERSION)\n <uniqueid>...</uniqueid> (host unique id)\n <HttpsPort>47984</HttpsPort> (net::map_port(PORT_HTTPS))\n <ExternalPort>47989</ExternalPort> (net::map_port(PORT_HTTP))\n <MaxLumaPixelsHEVC>1869449984</MaxLumaPixelsHEVC> (or \"0\" if HEVC disabled)\n <mac>aa:bb:cc:dd:ee:ff</mac> (real MAC on HTTPS; \"00:00:00:00:00:00\" on plain HTTP)\n <LocalIP>...</LocalIP>\n <ServerCodecModeSupport>3843</ServerCodecModeSupport> (bitmask; 3=H264 only,259=+HEVC,3843=+AV1)\n <ExternalIP>...</ExternalIP> (conditional)\n <PairStatus>1</PairStatus> (1 if request carried a known uniqueid over HTTPS, else 0)\n <currentgame>0</currentgame> (0 if idle, else running app id)\n <state>SUNSHINE_SERVER_FREE</state> (or SUNSHINE_SERVER_BUSY; GFE uses MJOLNIR_IDLE/_SERVER_BUSY)\n</root>",
|
||||
"notes": "Served on BOTH HTTP(47989) and HTTPS(47984). Plain-HTTP version hides mac and forces PairStatus=0. Element NAMES are case-sensitive and Moonlight requires hostname, appversion, PairStatus, currentgame, state, and at least one port. appversion MAJOR number is the SHA-1-vs-SHA-256 switch."
|
||||
},
|
||||
{
|
||||
"name": "/pair phase 1 request+response (getservercert)",
|
||||
"layout": "GET /pair?uniqueid=<id>&uuid=<uuid>&devicename=<name>&updateState=1&phrase=getservercert&salt=<32 hex chars = 16 bytes>&clientcert=<hex(PEM bytes)>\nResponse: <root status_code=\"200\"><paired>1</paired><plaincert>hex(server X.509 PEM)</plaincert></root>",
|
||||
"notes": "salt is client-random 16 bytes, sent as 32 hex chars. Server validates salt length >=32 hex chars, takes first 16 bytes, computes AES key = SHA256(salt||PIN)[..16]. clientcert is the client's self-signed cert (kept for later signature checks and TLS pinning). If PIN not yet entered, server may stall here until the user enters it."
|
||||
},
|
||||
{
|
||||
"name": "/pair phase 2 request+response (clientchallenge)",
|
||||
"layout": "GET /pair?uniqueid=<id>&clientchallenge=<hex(ECB-encrypt(randomChallenge[16]))>\nResponse: <root status_code=\"200\"><paired>1</paired><challengeresponse>hex(ECB-encrypt(hash[H] || serverChallenge[16]))</challengeresponse></root>",
|
||||
"notes": "Client sends AES-ECB(16-byte random). Server decrypts -> clientChallenge; computes hash = SHA( clientChallenge || serverCertSignature || serversecret[16 random] ); generates serverChallenge[16 random]; returns ECB(hash || serverChallenge). H = 32 (SHA-256) or 20 (SHA-1). With no padding, the ECB input is hash(32)+serverChallenge(16)=48 bytes = 3 blocks for SHA-256."
|
||||
},
|
||||
{
|
||||
"name": "/pair phase 3 request+response (serverchallengeresp)",
|
||||
"layout": "GET /pair?uniqueid=<id>&serverchallengeresp=<hex(ECB-encrypt(challengeRespHash[H]))>\nResponse: <root status_code=\"200\"><pairingsecret>hex( serversecret[16] || RSA-SHA256-sign(serversecret) )</pairingsecret><paired>1</paired></root>",
|
||||
"notes": "Client decrypts phase-2 challengeresponse into [serverResponseHash(H) || serverChallenge(16)], generates clientSecret[16 random], computes challengeRespHash = SHA( serverChallenge || clientCert.signature || clientSecret ), sends ECB(challengeRespHash). Server stores decrypted value as clienthash for the phase-4 check, then returns its serversecret plus that secret signed by the server cert's private key (sign256)."
|
||||
},
|
||||
{
|
||||
"name": "/pair phase 4 request+response (clientpairingsecret)",
|
||||
"layout": "GET /pair?uniqueid=<id>&clientpairingsecret=<hex( clientSecret[16] || RSA-SHA256-sign(clientSecret) )>\nResponse: <root status_code=\"200\"><paired>1 or 0</paired></root>",
|
||||
"notes": "Server splits into secret(16)+signature(rest). Builds data = serverChallenge || clientCert.signature || clientSecret, hashes it, compares to the clienthash stored in phase 3 (proves client knew PIN). Also verify256(clientCert, clientSecret, signature) (proves client owns its cert key). Both must pass; on success the client cert is added to the authorized allow-list. paired=0 means PIN/cert mismatch."
|
||||
},
|
||||
{
|
||||
"name": "/pair final pairchallenge (HTTPS)",
|
||||
"layout": "GET /pair?uniqueid=<id>&phrase=pairchallenge (over HTTPS:47984, presenting the now-trusted client cert)\nResponse: <root status_code=\"200\"><paired>1</paired></root>",
|
||||
"notes": "Moonlight calls executePairingChallenge() after phase 4; it must succeed over the mutual-TLS connection using the freshly-paired client cert, confirming the cert pinning round-trips. If this fails Moonlight calls /unpair and reports FAILED."
|
||||
},
|
||||
{
|
||||
"name": "Other HTTPS endpoints (params only, post-pair)",
|
||||
"layout": "/applist -> XML list of <App><IsHdrSupported><AppTitle><ID>; /appasset?appid=&assetidx=&assettype= -> PNG; /launch?uniqueid=&appid=&mode=WxHxFPS&additionalStates=&sops=&rikey=<hex AES key>&rikeyid=<int>&localAudioPlayMode=&surroundAudioInfo=&hdrMode=&corever=; /resume?rikey=&rikeyid=&surroundAudioInfo=; /cancel (no params)",
|
||||
"notes": "Out of this area's depth but listed for completeness. rikey/rikeyid carry the AES-128 RTSP/stream key (16-byte key as hex, plus a 4-byte key id) used to seal the control/RTSP plane; mode is 'WIDTHxHEIGHTxFPS'. /launch and /resume require an already-paired (pinned) HTTPS connection."
|
||||
}
|
||||
],
|
||||
"flow": [
|
||||
"0. Moonlight GET http://host:47989/serverinfo -> parses appversion major (>=7 => SHA-256 hash, else SHA-1), HttpsPort, PairStatus, state.",
|
||||
"1. Client generates salt=random16 and a self-signed RSA-2048 X.509 cert. GET /pair?...&phrase=getservercert&salt=hex(salt)&clientcert=hex(certPEM). User enters PIN on host. Both sides compute aesKey = SHA(salt || pinUTF8)[0..16].",
|
||||
"2. Server replies plaincert=hex(serverCertPEM), paired=1. Client stores server cert.",
|
||||
"3. Client: randomChallenge=random16; GET ...&clientchallenge=hex(AES_ECB_enc(randomChallenge, aesKey)).",
|
||||
"4. Server: decrypt -> clientChallenge; serversecret=random16; respHash=SHA(clientChallenge || serverCert.signature || serversecret); serverChallenge=random16; reply challengeresponse=hex(AES_ECB_enc(respHash || serverChallenge)).",
|
||||
"5. Client: decrypt challengeresponse -> [serverRespHash(H) || serverChallenge(16)]; clientSecret=random16; challengeRespHash=SHA(serverChallenge || clientCert.signature || clientSecret); GET ...&serverchallengeresp=hex(AES_ECB_enc(challengeRespHash)).",
|
||||
"6. Server: decrypt -> store as clienthash; reply pairingsecret=hex(serversecret || RSA_SHA256_sign(serversecret, serverPrivKey)), paired=1.",
|
||||
"7. Client: split pairingsecret into [serversecret(16) || serverSig]; verify expected = SHA(randomChallenge || serverCert.signature || serversecret) (sanity) and RSA-verify serverSig over serversecret with server cert. GET ...&clientpairingsecret=hex(clientSecret || RSA_SHA256_sign(clientSecret, clientPrivKey)).",
|
||||
"8. Server: split into [clientSecret(16) || clientSig]; recompute SHA(serverChallenge || clientCert.signature || clientSecret) and compare to stored clienthash; RSA-verify clientSig over clientSecret with clientCert. Both pass => add clientCert to authorized list; reply paired=1 (else paired=0).",
|
||||
"9. Client: GET https://host:47984/pair?...&phrase=pairchallenge over mutual-TLS presenting its now-trusted cert; expects paired=1. Pairing complete -> PairState.PAIRED.",
|
||||
"10. On any mismatch the client GET /unpair and returns FAILED / PIN_WRONG."
|
||||
],
|
||||
"crypto": "PIN-derived key: aesKey = HASH(salt[16] || PIN_utf8)[0..16], where HASH = SHA-256 if server appversion major >=7 (Sunshine: 7.1.431.-1) else SHA-1. Salt = client random 16 bytes. PIN is the 4-digit code shown/entered by user; concatenation is salt FIRST then pin. Pairing cipher: AES-128 in ECB mode, NO padding / padding DISABLED (Sunshine ecb_t(key,false); Moonlight AESLightEngine block loop). Inputs are zero-extended to a 16-byte multiple before encryption (a 32-byte SHA-256 hash = 2 blocks; respHash(32)+serverChallenge(16)=48=3 blocks). Per-side proofs: serverHash = SHA(clientChallenge || serverCert.signature || serversecret16); clientHash = SHA(serverChallenge || clientCert.signature || clientSecret16) (cert.signature = the DER signature bytes of the self-signed X.509). Identity binding: each side RSA-signs its own 16-byte secret with its cert's private key using RSA-PKCS1 over SHA-256 (sign256/verify256), other side verifies. Certs: self-signed RSA-2048, SHA-256 signed, ~20-year validity. Result of pairing is NOT a streaming key — it only establishes mutual TLS trust (pinned certs). The actual AES-128 STREAM key is delivered separately at /launch as rikey (16-byte hex) + rikeyid; that is where lumen-core's existing AES-128-GCM session crypto plugs in. IMPORTANT: pairing AES-ECB-no-pad is distinct from and unrelated to lumen-core's AES-128-GCM session sealing.",
|
||||
"rust_options": "HTTP/HTTPS control plane belongs in crates/lumen-host/src/web.rs (the existing stub explicitly permits tokio/axum here, off the hot path). Use axum or hyper for the two servers. TLS with mutual auth + custom cert pinning: use rustls via axum-server/tokio-rustls with a custom ClientCertVerifier (rustls::server::danger::ClientCertVerifier) that accepts any well-formed cert at handshake time and then matches the presented leaf DER/PEM against the paired allow-list (mirror Sunshine's verify callback), OR use openssl/openssl crate to match Sunshine 1:1. XML: build/parse with quick-xml or xml-rs (or just format! the small fixed templates and a tiny extractor). Crypto: aes crate (already a dep transitively) in ECB mode via the `ecb` crate with NoPadding (Aes128 + ecb::Decryptor/Encryptor, manual block handling) — note RustCrypto deprecates ECB so call the block cipher directly (aes::Aes128 + cipher::BlockEncrypt/BlockDecrypt over 16-byte chunks). Hashing: sha2 (SHA-256) and sha1 crates. X.509 self-signed cert generation + RSA-SHA256 sign/verify: rcgen for cert gen, rsa + sha2 for PKCS1v15 sign/verify, x509-parser or x509-cert to extract the cert's signature bytes (cert.getSignature() equivalent) and to do TLS-trust comparison. Persist authorized client certs (PEM) in a small JSON/sled store. Run all of this on tokio in web.rs; keep it fully separate from the native-thread per-frame pipeline.",
|
||||
"reuse_from_lumen": "REUSE little of lumen-core's crypto here — its crypto.rs is AES-128-GCM session sealing (nonce = salt||seq, seq as AAD) for the VIDEO/INPUT plane, which corresponds to the post-/launch rikey stream key, NOT the pairing handshake. Pairing needs AES-128-ECB-no-pad + SHA-256/SHA-1 + RSA, none of which exist in lumen yet and must be newly built (best placed in lumen-host, not lumen-core, since it is control-plane only). The natural seam is the existing web.rs stub (WebConfig::run) whose TODO already says 'GameStream serverinfo, pairing handshake, RTSP SETUP' — implement the two HTTP servers and the 4-phase /pair state machine there. lumen-core's AES-128-GCM SessionCrypto IS reusable downstream: once paired and /launch hands over rikey (16-byte AES key) + rikeyid, feed that key into lumen-core's Session/SessionCrypto for the encrypted video/control planes. The internal 40-byte packet format is unrelated to this HTTP/pairing area. So: new pairing crypto + axum servers in lumen-host/web.rs (control), reuse lumen-core GCM for the data plane post-launch.",
|
||||
"gotchas": [
|
||||
"appversion MAJOR number is load-bearing: it silently switches the client's pairing hash. Advertise major >=7 (e.g. \"7.1.431.-1\") to get SHA-256; advertise <7 and the client uses SHA-1 with 20-byte hashes (changes all the ECB block counts). Mismatch => silent pairing failure.",
|
||||
"AES-ECB has NO padding and NO IV. Do not use a library that auto-pads (PKCS7) — Sunshine passes ecb_t(key,false). Inputs are zero-extended to 16-byte multiples; a 32-byte SHA-256 hash is exactly 2 blocks but respHash(32)+serverChallenge(16)=48 must be encrypted as one 48-byte buffer.",
|
||||
"The hash inputs use cert.getSignature() = the X.509 DER SIGNATURE bytes (the signatureValue), NOT the cert body, NOT a hash of the cert. Getting this field wrong is the most common pairing bug.",
|
||||
"Salt+PIN order is salt FIRST then PIN (UTF-8 ascii digits). PIN is the literal 4-char string, not parsed as an integer.",
|
||||
"Pairing does NOT yield the stream key. The streaming AES key arrives later as /launch?rikey=<hex>&rikeyid=<n>. Don't conflate the PIN-derived ECB key with the GCM stream key.",
|
||||
"Two distinct servers: /serverinfo and /pair answer on BOTH 47989 (HTTP) and 47984 (HTTPS); /applist,/appasset,/launch,/resume,/cancel are HTTPS-ONLY and require the pinned client cert. The plain-HTTP /serverinfo must zero out mac and force PairStatus=0.",
|
||||
"TLS pinning is custom: standard CA validation is bypassed; the verify callback accepts the handshake then checks the presented leaf cert's PEM against the stored authorized-client list. Client certs are self-signed, so a normal rustls verifier would reject them — you must supply a permissive ClientCertVerifier + post-hoc allow-list match.",
|
||||
"All responses wrap in <root status_code=\"200\"> ... </root>; the status_code attribute (and HTTP 200) both matter. paired=1/0 appears in every pairing reply and the client checks it as a string \"1\".",
|
||||
"RSA signatures are PKCS#1 v1.5 over SHA-256 (sign256/verify256). Use the cert's RSA key; ECDSA certs would change this.",
|
||||
"ServerCodecModeSupport is a bitmask, advertise the decimal: 3 (H264 only), 259 (H264+HEVC), 3843 (H264+HEVC+AV1); flags SCM_H264=0x1, SCM_HEVC=0x100, SCM_HEVC_MAIN10=0x200, SCM_AV1_MAIN8=0x10000, SCM_AV1_MAIN10=0x20000.",
|
||||
"If the user hasn't entered the PIN yet, the host stalls the phase-1 response until it's entered (or until clientchallenge); design the state machine to await PIN input keyed by uniqueid.",
|
||||
"Each pairing phase validates the previous phase happened for that uniqueid; the host keeps per-client pairing session state (last_phase, cipher_key, serversecret, serverChallenge, clienthash) across the 4 separate HTTP GETs."
|
||||
],
|
||||
"sources": [
|
||||
"Sunshine src/nvhttp.cpp: serverinfo() (root.hostname/appversion/GfeVersion/uniqueid/HttpsPort/ExternalPort/MaxLumaPixelsHEVC/mac/LocalIP/ServerCodecModeSupport/PairStatus/currentgame/state; codec_mode_flags ORing SCM_*; MaxLumaPixelsHEVC=\"1869449984\"; state SUNSHINE_SERVER_FREE/BUSY) and pair() 4-phase state machine (getservercert/clientchallenge/serverchallengeresp/clientpairingsecret; ecb_t(*cipher_key,false); crypto::hash; sign256; verify256; add_authorized_client) and the https_server.verify pinning callback",
|
||||
"Sunshine src/nvhttp.h: VERSION=\"7.1.431.-1\", GFE_VERSION=\"3.23.0.74\", PORT_HTTP=0, PORT_HTTPS=-5",
|
||||
"Sunshine src/crypto.cpp: gen_aes_key (salt||pin -> SHA-256 -> first 16 bytes), hash() = EVP_sha256, sign256/verify256 = EVP_sha256, ecb_t/gcm_t/cbc_t (EVP_aes_128_ecb/gcm/cbc, EVP_CIPHER_CTX_set_padding), AES-128 throughout",
|
||||
"Sunshine src/network.cpp: map_port(p) = config::sunshine.port + p",
|
||||
"moonlight-android PairingManager.java: serverMajorVersion>=7 -> Sha256PairingHash else Sha1PairingHash; saltPin (salt then pin utf-8); generateAesKey = copyOf(hash,16); encryptAes/decryptAes AESLightEngine ECB; performBlockCipher 16-byte block loop with zero-pad blockRoundedSize; phase byte orders (clientchallenge=enc(random16); challengeRespHash=hash(serverChallenge||cert.signature||clientSecret); clientPairingSecret=clientSecret||signData(clientSecret)); Sha256PairingHash.getHashLength=32 / Sha1=20; signData=SHA256withRSA; verifySignature; PairState{NOT_PAIRED,PAIRED,PIN_WRONG,FAILED,ALREADY_IN_PROGRESS}; executePairingChallenge final pairchallenge",
|
||||
"moonlight-common-c src/Limelight.h: SCM_H264=0x1, SCM_HEVC=0x100, SCM_HEVC_MAIN10=0x200, SCM_AV1_MAIN8=0x10000, SCM_AV1_MAIN10=0x20000, SCM_*_444 flags; VIDEO_FORMAT_* constants; default ports 47984/47989/48010 tcp, 47998/47999/48000/48010 udp",
|
||||
"DeepWiki LizardByte/Sunshine NVHTTP page: ServerCodecModeSupport decimal values 3 / 259 / 3843",
|
||||
"lumen repo (local): crates/lumen-host/src/web.rs (WebConfig stub, control-plane seam) and crates/lumen-core/src/crypto.rs (AES-128-GCM SessionCrypto = data-plane, not pairing)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"area": "RTSP handshake + SDP + stream config negotiation (GameStream / Sunshine ↔ Moonlight)",
|
||||
"summary": "GameStream negotiation is two phases. Phase 1 is an HTTPS GET to /launch (or /resume) on port 47984/47989 where the client passes the session parameters as URL query args — most importantly rikey (a 32-hex-char = 16-byte AES-128 key) and rikeyid (a signed 32-bit int). The host derives the per-stream AES-GCM key directly from rikey and a 16-byte IV from rikeyid as a big-endian uint32 left-padded into a 16-byte buffer (Sunshine make_launch_session). Phase 2 is a GameStream-flavored RTSP/1.0 exchange over TCP on port 48010 (RTSP_SETUP_PORT = base 47989 + 21). The sequence is OPTIONS → DESCRIBE → SETUP(audio) → SETUP(video) → SETUP(control) → ANNOUNCE → PLAY, each carrying a CSeq header. DESCRIBE returns an SDP-ish body of a= attributes advertising host capabilities (x-ss-general.featureFlags, encryptionSupported/Requested, refPicInvalidation, AV1 rtpmap, Opus surround-params). SETUP returns Session: DEADBEEFCAFE;timeout = 90 plus Transport: server_port=<port> and an X-SS-Connect-Data (control) or X-SS-Ping-Payload (A/V) header. ANNOUNCE is where the client sends the actual negotiated stream config as an SDP body of x-nv-video[0].*, x-nv-vqos[0].*, x-nv-general.*, x-nv-audio.*, x-nv-aqos.*, x-ss-video[0].*, x-ml-* attributes (resolution, fps, bitrate, packetSize, fecPercentage, codec/bitStreamFormat, HDR, surround, encryptionEnabled). PLAY simply ACKs and the host begins sending RTP. When encryption is negotiated the RTSP messages themselves are wrapped in an encrypted framing (typeAndLength MSB-flagged + sequenceNumber + 16-byte GCM tag) keyed by the same gcm_key with IV bytes 10/11 = direction+'R'.",
|
||||
"ports": [
|
||||
"47984 = HTTPS control (PORT_HTTPS, base+5) — serves /launch, /resume, /serverinfo, /pair over TLS",
|
||||
"47989 = HTTP control (base port; PORT_HTTP) — unauthenticated /serverinfo etc.",
|
||||
"48010 TCP = RTSP_SETUP_PORT (base 47989 + 21) — the RTSP handshake; this is the task's stated port",
|
||||
"47998 UDP = VIDEO_STREAM_PORT (base + 9) — RTP video, returned in SETUP video Transport server_port",
|
||||
"47999 UDP = CONTROL_PORT (base + 10) — ENet/control + remote input, returned in SETUP control",
|
||||
"48000 UDP = AUDIO_STREAM_PORT (base + 11) — RTP audio, returned in SETUP audio",
|
||||
"All ports are base+offset via net::map_port(); base is configurable (default 47989). Moonlight overrides VideoPortNumber/AudioPortNumber/ControlPortNumber from the SETUP Transport server_port= field, with fallbacks video=47998, audio=48000, control=47999"
|
||||
],
|
||||
"wire_formats": [
|
||||
{
|
||||
"name": "RTSP request line + headers (plaintext)",
|
||||
"layout": "ASCII text, CRLF line endings. Request line: <METHOD> <target> RTSP/1.0\\r\\n. Methods: OPTIONS, DESCRIBE, SETUP, ANNOUNCE, PLAY. Common headers: CSeq: <n>\\r\\n (monotonic, set by client currentSeqNumber), X-GS-ClientVersion: <n> (AppVersionQuad[0] map 3→10,4→11,5→12,7→14), Host: <addr> (TCP only). Header block terminated by \\r\\n\\r\\n. Body (if any) preceded by Content-type: application/sdp and Content-length: <bytes>.",
|
||||
"notes": "Parsed by moonlight-common-c parseRtspMessage(), which Sunshine vendors in. NOT 100% standard RTSP — targets use streamid= scheme and there is an alternate ENet 'rtspru://' transport for ancient GFE (<v404); modern Moonlight uses raw TCP RTSP/1.0."
|
||||
},
|
||||
{
|
||||
"name": "SETUP target (request URI)",
|
||||
"layout": "Modern (GFE≥5): streamid=audio/0/0 , streamid=video/0/0 , streamid=control/13/0 (control id is 13 for GFE≥7.1.431, else 1). Legacy (GFE<5): streamid=audio , streamid=video. Sunshine parses by finding '=' then '/': type = audio|video|control.",
|
||||
"notes": "Sunshine maps audio→AUDIO_STREAM_PORT(48000), video→VIDEO_STREAM_PORT(47998), control→CONTROL_PORT(47999); unknown type → 404."
|
||||
},
|
||||
{
|
||||
"name": "SETUP response headers",
|
||||
"layout": "CSeq: <echo>\\r\\nSession: DEADBEEFCAFE;timeout = 90\\r\\nTransport: server_port=<port>\\r\\n then ONE of: X-SS-Connect-Data: <control_connect_data> (for control stream) OR X-SS-Ping-Payload: <av_ping_payload> (for audio/video). 200 OK.",
|
||||
"notes": "Session string is the literal constant DEADBEEFCAFE;timeout = 90 (note the spaces around '='). server_port is the UDP port the client must send/recv on. X-SS-Ping-Payload is the per-session magic the client must send as the first UDP datagram to A/V ports so the host learns the client's source port; X-SS-Connect-Data likewise for control."
|
||||
},
|
||||
{
|
||||
"name": "DESCRIBE response body (host capabilities SDP)",
|
||||
"layout": "Newline-joined a= lines (Sunshine cmd_describe builds via stringstream, each << std::endl):\\n a=x-ss-general.featureFlags:<platf::get_capabilities() uint32>\\n a=x-ss-general.encryptionSupported:<flags>\\n a=x-ss-general.encryptionRequested:<flags>\\n a=x-nv-video[0].refPicInvalidation:1 (only if encoder supports RFI)\\n sprop-parameter-sets=AAAAAU (emitted unless HEVC-only forced; HEVC indicator)\\n a=rtpmap:98 AV1/90000 (emitted unless AV1 forced off)\\n a=fmtp:97 surround-params=<channelCount><streams><coupledStreams><mapping digits...> (one per audio::stream_configs entry; 5.1/7.1 rotate mapping)\\n a=rtpmap audio is implied via fmtp:97 / fmtp:96.",
|
||||
"notes": "encryptionSupported default = SS_ENC_CONTROL_V2|SS_ENC_AUDIO (=0x05); adds SS_ENC_VIDEO(0x02) unless encryption mode NEVER. encryptionRequested default = SS_ENC_CONTROL_V2 (=0x01); adds VIDEO|AUDIO if mode MANDATORY. Moonlight scans this body for 'AV1/90000', 'sprop-parameter-sets=AAAAAU' (HEVC), refPicInvalidation, the x-ss-general.* flags, and the fmtp:97 surround-params (channelCount, streamCount, coupledStreams, channel mapping for Opus multistream)."
|
||||
},
|
||||
{
|
||||
"name": "ANNOUNCE request body (client→host negotiated config SDP)",
|
||||
"layout": "Body: v=0\\r\\no=android 0 <sessver> IN <af> <addr>\\r\\ns=NVIDIA Streaming Client\\r\\n then a=<name>:<value> lines, then t=0 0\\r\\nm=video <port> \\r\\n. Sunshine cmd_announce splits on lines, takes s= as client name and a=name:value into a map. Keys parsed into stream::config_t: x-nv-video[0].clientViewportWd (width), x-nv-video[0].clientViewportHt (height), x-nv-video[0].maxFPS (fps int), x-nv-video[0].clientRefreshRateX100 (fps*100), x-nv-video[0].packetSize (MTU payload), x-nv-video[0].videoEncoderSlicesPerFrame (slices), x-nv-video[0].maxNumReferenceFrames, x-nv-video[0].encoderCscMode, x-nv-video[0].dynamicRangeMode (HDR 0/1), x-nv-vqos[0].bitStreamFormat (codec: 0=H264,1=HEVC,2=AV1), x-nv-vqos[0].bw.maximumBitrateKbps (bitrate), x-nv-vqos[0].fec.minRequiredFecPackets, x-nv-vqos[0].qosTrafficType, x-nv-audio.surround.numChannels, x-nv-audio.surround.channelMask, x-nv-audio.surround.AudioQuality, x-nv-aqos.packetDuration, x-nv-aqos.qosTrafficType, x-nv-general.useReliableUdp (controlProtocolType, 13 or 1), x-nv-general.featureFlags (bit 0x20 ⇒ enable SS_ENC_AUDIO), x-ml-general.featureFlags, x-ml-video.configuredBitrateKbps, x-ss-general.encryptionEnabled (the actual negotiated enc bitmask), x-ss-video[0].chromaSamplingType (0=4:2:0,1=4:4:4), x-ss-video[0].intraRefresh.",
|
||||
"notes": "Sunshine fills defaults via try_emplace before reading: encoderCscMode=0, bitStreamFormat=0, dynamicRangeMode=0, packetDuration=5, useReliableUdp=1, fec.minRequiredFecPackets=0, x-nv-general.featureFlags=135, x-ml-general.featureFlags=0, vqos qosTrafficType=5, aqos qosTrafficType=4, configuredBitrateKbps=0, encryptionEnabled=0, chromaSamplingType=0, intraRefresh=0, clientRefreshRateX100=0. Missing REQUIRED keys (width/height/fps/packetSize/bitrate/channels) throw std::out_of_range → 400 BAD REQUEST. Moonlight's SdpGenerator.c additionally emits but Sunshine ignores: rateControlMode=4, timeoutLengthMs=7000, framesWithInvalidRefThreshold=0, fec.enable=1, fec.repairPercent (5 or 20), fec.minRequiredFecPackets=2, bllFec.enable=0, videoQualityScoreUpdateTime=5000, bw.minimumBitrateKbps, initialBitrateKbps, x-nv-clientSupportHevc, surround.enable, enableRecoveryMode=0, x-nv-ri.useControlChannel=1."
|
||||
},
|
||||
{
|
||||
"name": "Encrypted RTSP framing (when SS_ENC_CONTROL_V2 negotiated)",
|
||||
"layout": "struct encrypted_rtsp_header_t { uint32_t typeAndLength; /*big-endian; MSB ENCRYPTED_MESSAGE_TYPE_BIT=0x80000000 set, low bits = payload length*/ uint32_t sequenceNumber; /*big-endian, monotonic*/ uint8_t tag[16]; /*AES-128-GCM auth tag*/ }; followed by ciphertext payload (the plaintext RTSP message).",
|
||||
"notes": "AES-GCM IV (12 bytes, NIST SP800-38D 8.2.1): bytes[0..4]=sequenceNumber big-endian, byte[10]='C'(client-originated) or 'H'(host-originated), byte[11]='R' (RTSP). Decrypt input is tag||ciphertext. Plaintext RTSP (no enc) is delimited by \\r\\n\\r\\n."
|
||||
},
|
||||
{
|
||||
"name": "/launch HTTPS query (where RIKEY/RIKEYID arrive)",
|
||||
"layout": "GET https://<host>:47984/launch?uniqueid=...&appid=<id>&mode=<W>x<H>x<fps>&additionalStates=...&sops=<0|1>&rikey=<32 hex chars>&rikeyid=<int32>&localAudioPlayMode=<0|1>&surroundAudioInfo=<packed>&remoteControllersBitmap=...&gcmap=...&hdrMode=<0|1>&clientHdrCapabilities=...&corever=<n>. /resume is the same minus appid/mode.",
|
||||
"notes": "Sunshine requires rikey, rikeyid, localAudioPlayMode, appid present or 400. rikey = util::from_hex_vec(rikey,true) → gcm_key (16 bytes). iv = 16-byte buffer with big-endian uint32(rikeyid) in the FIRST 4 bytes, rest zero. mode split on 'x' → width/height/fps. surroundAudioInfo default 196610 (=0x30002 ⇒ 2-channel mask 0x3 high16=channels). corever decides whether encrypted RTSP is used. The launch response gives root.sessionUrl0 = <rtsp_url_scheme><addr>:48010 telling Moonlight where to open RTSP."
|
||||
}
|
||||
],
|
||||
"flow": [
|
||||
"0. (HTTPS, prior to RTSP) Client GETs /serverinfo, then /launch?...&rikey=<hex16>&rikeyid=<int>&mode=WxHxF&... on port 47984. Host stores gcm_key=hex(rikey), iv=BE32(rikeyid)||zeros, parses mode/sops/surroundAudioInfo/hdrMode, allocates the launch_session, and responds with root.sessionUrl0=rtsp[enc]://<addr>:48010.",
|
||||
"1. Client opens TCP to 48010 and sends OPTIONS <rtspTargetUrl> RTSP/1.0 with CSeq:1, X-GS-ClientVersion. Host replies 200 OK echoing CSeq (cmd_option).",
|
||||
"2. Client sends DESCRIBE <rtspTargetUrl> RTSP/1.0 (CSeq:2, Accept/If-Modified-Since). Host replies 200 OK with the a= capability body (featureFlags, encryptionSupported/Requested, refPicInvalidation, AV1 rtpmap, surround-params).",
|
||||
"3. Client sends SETUP streamid=audio/0/0 (CSeq:3). Host replies Session: DEADBEEFCAFE;timeout = 90, Transport: server_port=48000, X-SS-Ping-Payload:<...>.",
|
||||
"4. Client sends SETUP streamid=video/0/0 (CSeq:4). Host replies Transport: server_port=47998, X-SS-Ping-Payload:<...>.",
|
||||
"5. Client sends SETUP streamid=control/13/0 (CSeq:5). Host replies Transport: server_port=47999, X-SS-Connect-Data:<...>. (Moonlight latches these server_port values into Video/Audio/ControlPortNumber.)",
|
||||
"6. Client sends ANNOUNCE (CSeq:6) with Content-type: application/sdp and the full x-nv-*/x-ss-*/x-ml-* config body (resolution, fps, bitrate, packetSize, fecPercent, bitStreamFormat codec, dynamicRangeMode HDR, surround, encryptionEnabled). Host parses into config_t; missing required key → 400. 200 OK on success.",
|
||||
"7. Client sends PLAY (CSeq:7; single PLAY '/' for GFE≥7.1.431, else per-stream). Host replies 200 OK and the RTP video/audio + control/input flows begin on the UDP ports. First UDP datagram from client on each A/V port carries the X-SS-Ping-Payload so the host learns the client source port (NAT punch / port-learn)."
|
||||
],
|
||||
"crypto": "RIKEY/RIKEYID origin: the Moonlight client app generates remoteInputAesKey[16] and remoteInputAesIv[16] (STREAM_CONFIGURATION in Limelight.h) before connecting; rikey = hex(remoteInputAesKey), rikeyid = a 32-bit int. They are delivered to the host NOT in RTSP but in the HTTPS /launch query string. Sunshine make_launch_session: gcm_key = from_hex_vec(rikey) (16 bytes, the AES-128-GCM key shared by video/audio/control/input/RTSP ciphers); iv (16 bytes) = big-endian uint32(rikeyid) in bytes[0..4], zero-padded. AES-128-GCM is used everywhere. Per-stream 12-byte GCM nonce construction (Sunshine stream.cpp / rtsp.cpp): VIDEO (host→client): bytes[0..]=gcm_iv_counter (LE incrementing), byte[11]='V'. CONTROL host→client: 12-byte iv = seq(LE) || byte[10]='H', byte[11]='C'. CONTROL client→host: seq(LE) || byte[10]='C', byte[11]='C'. RTSP (encrypted handshake): seq(BE in bytes[0..4]) || byte[10]='C'(client)|'H'(host), byte[11]='R'. Encryption is negotiated, not key-exchanged: DESCRIBE advertises x-ss-general.encryptionSupported (default SS_ENC_CONTROL_V2|SS_ENC_AUDIO=0x05, +SS_ENC_VIDEO=0x02 if allowed) and encryptionRequested (default SS_ENC_CONTROL_V2=0x01, +VIDEO|AUDIO if MANDATORY); the client echoes the chosen bitmask in ANNOUNCE x-ss-general.encryptionEnabled, and x-nv-general.featureFlags bit 0x20 forces SS_ENC_AUDIO on. Flags: SS_ENC_CONTROL_V2=0x01, SS_ENC_VIDEO=0x02, SS_ENC_AUDIO=0x04. Codec tag bitStreamFormat: 0=H264,1=HEVC,2=AV1 (client capability bits VIDEO_FORMAT_H264=0x0001,H265=0x0100,H265_MAIN10=0x0200,AV1_MAIN8=0x1000,AV1_MAIN10=0x2000).",
|
||||
"rust_options": "For RTSP on 48010: a tiny synchronous TCP server using std::net::TcpListener on a native thread (NO tokio — keep off the hot path, consistent with lumen's no-async-on-hot-path invariant). Parse RTSP/1.0 manually: read until \\\\r\\\\n\\\\r\\\\n, split request line + headers, read Content-length body for ANNOUNCE. There is no need for a crate; a hand-rolled parser mirroring Sunshine's parseRtspMessage is simplest and avoids pulling RTSP libs that assume standard semantics (the streamid= targets and DEADBEEFCAFE session break them). For SDP build/parse, just format!/split on lines — it is line-oriented a=key:value. For the /launch HTTPS endpoint, reuse the existing crates/lumen-host/src/web.rs seam; a small hyper or tiny_http + rustls TLS server (control plane only, async OK here since it is not the hot path — matches the 'quic feature gated' precedent). For encrypted-RTSP framing use the aes-gcm crate already in lumen-core. Suggested new types: an RtspServer in lumen-host that produces a lumen_core::Config from the ANNOUNCE map. Hex decode rikey with the `hex` crate (or from_str_radix). Big-endian rikeyid → IV with u32::to_be_bytes.",
|
||||
"reuse_from_lumen": "REUSE: lumen-core/src/crypto.rs SessionCrypto already implements AES-128-GCM with per-direction salting and seq-as-AAD — but NOTE it is NOT byte-compatible with GameStream's nonce layout. lumen uses nonce = salt(4) || seq(8, BE) with a direction bit folded into salt[0], whereas GameStream uses iv = seq(LE or BE per stream) with literal direction/stream marker bytes at [10]/[11] ('V', 'H'/'C'+'C', 'C'/'H'+'R'). To talk to stock Moonlight you must add a GameStream-exact nonce mode (new constructor or a feature) rather than reuse the existing salt scheme verbatim. The Aes128Gcm cipher init and seal/open plumbing are reusable. REUSE: lumen-core Config/FecConfig — fec_percent maps to GameStream's repairPercent and the recovery_for() ceil(k*pct/100) already matches GameStream's FEC math; map ANNOUNCE packetSize→shard_payload, maximumBitrateKbps→bitrate, fec.minRequiredFecPackets→minRequiredFecPackets. FecScheme::Gf8 is the GameStream-compatible field. BUILD NEW: the entire RTSP/SDP/launch negotiation layer (lumen's internal 40-byte packet format and Config are not wire-exact to RTP/RTSP); the RTSP server, SDP describe/announce codec, the /launch query parser that produces gcm_key+iv from rikey/rikeyid, and the GameStream RTP video/audio packetization + RTPFEC are all new (separate areas). The Session in lumen-core can consume the negotiated Config but its on-wire packet header must be swapped for GameStream RTP for Moonlight compat.",
|
||||
"gotchas": [
|
||||
"lumen-core's AES-GCM nonce layout is NOT GameStream-compatible (salt+BE-seq vs literal 'V'/'C'/'R' marker bytes + LE/BE seq). A stock Moonlight will fail auth unless you implement the exact per-stream IV construction. This is the single biggest bridging hazard.",
|
||||
"rikeyid is parsed as a SIGNED int then cast to big-endian uint32 in Sunshine (util::from_view → int → endian::big<uint32_t>). Negative rikeyid values wrap; match the signed-int→BE-u32 path exactly.",
|
||||
"The IV from /launch (BE32(rikeyid)||zeros, 16 bytes) is the *base*; the actual per-packet 12-byte GCM nonce is rebuilt per stream with seq + marker bytes — do not just use the 16-byte launch IV directly.",
|
||||
"Session header value is the literal 'DEADBEEFCAFE;timeout = 90' WITH spaces around '='. Moonlight is lenient but match it.",
|
||||
"SETUP Transport must be exactly 'server_port=<port>' (Moonlight greps for 'server_port='); it also expects X-SS-Ping-Payload (A/V) / X-SS-Connect-Data (control) headers and will send that payload as the first UDP datagram for port-learning — the host must accept it.",
|
||||
"ANNOUNCE keys are case-sensitive and bracketed exactly 'x-nv-video[0].' — the [0] index is literal. Missing a REQUIRED key (width/height/fps/packetSize/bitrate/channels) yields 400; supply the same try_emplace defaults Sunshine uses or Moonlight builds may omit them.",
|
||||
"Codec is bitStreamFormat (0/1/2), but capability advertisement in DESCRIBE uses sprop-parameter-sets=AAAAAU (HEVC marker) and a=rtpmap:98 AV1/90000 — Moonlight infers codec support from those strings, so emit/omit them to steer codec.",
|
||||
"fec.repairPercent (GFE<7.1.431) vs fec.minRequiredFecPackets (GFE≥7.1.431) — newer clients send the latter; Sunshine reads minRequiredFecPackets and defaults repairPercent handling. Handle both.",
|
||||
"Stream targets differ by GFE version: modern 'streamid=video/0/0' and 'streamid=control/13/0', legacy 'streamid=video'. Parse by splitting on '=' and '/', taking the type token, like Sunshine.",
|
||||
"There is an alternate ENet 'rtspru://' RTSP transport for very old GFE (<v404); stock modern Moonlight uses raw TCP RTSP/1.0 on 48010, so TCP-only is sufficient for current clients (note legacy ENet path as unsupported).",
|
||||
"Encrypted-RTSP is only used when corever/encryption is negotiated; you must support BOTH plaintext (\\r\\n\\r\\n delimited) and the encrypted_rtsp_header_t framing depending on the negotiated SS_ENC_CONTROL_V2 flag."
|
||||
],
|
||||
"sources": [
|
||||
"Sunshine src/rtsp.cpp — cmd_option, cmd_describe (a=x-ss-general.featureFlags/encryptionSupported/encryptionRequested, refPicInvalidation, sprop-parameter-sets=AAAAAU, rtpmap:98 AV1/90000, fmtp:97 surround-params), cmd_setup (Session DEADBEEFCAFE;timeout = 90, Transport server_port=, X-SS-Connect-Data/X-SS-Ping-Payload, audio/video/control→ports), cmd_announce (full x-nv-*/x-ss-*/x-ml-* config_t parse + try_emplace defaults), cmd_play, encrypted_rtsp_header_t + IV construction",
|
||||
"Sunshine src/rtsp.h — RTSP_SETUP_PORT = 21 (offset), launch_session_t (gcm_key, iv, rtsp_cipher)",
|
||||
"Sunshine src/nvhttp.cpp — make_launch_session: rikey hex→gcm_key, rikeyid→BE uint32 IV[0..4]; /launch & /resume required args (rikey,rikeyid,localAudioPlayMode,appid); mode WxHxF, sops, surroundAudioInfo default 196610, hdrMode, corever; sessionUrl0=rtsp://addr:48010; serverinfo HttpsPort/ExternalPort",
|
||||
"Sunshine src/stream.cpp — VIDEO/AUDIO/CONTROL port usage via net::map_port; per-stream GCM IV (iv[11]='V' video, [10]='H'/'C'+[11]='C' control); SS_ENC_CONTROL_V2/VIDEO/AUDIO flag usage",
|
||||
"moonlight-common-c src/RtspConnection.c — performRtspHandshake sequence (OPTIONS,DESCRIBE,SETUP audio/video/control,ANNOUNCE,PLAY), streamid=audio/0/0|video/0/0|control/13/0 (or /1/0), Transport server_port fallbacks (video 47998,audio 48000,control 47999), RTSP/1.0, X-GS-ClientVersion mapping, useEnet/rtspru legacy path, encrypted rtspenc:// framing",
|
||||
"moonlight-common-c src/SdpGenerator.c — full ANNOUNCE SDP: v=0/o=android/s=NVIDIA Streaming Client, x-nv-video[0].clientViewportWd/Ht/maxFPS/packetSize/rateControlMode=4/timeoutLengthMs=7000/videoEncoderSlicesPerFrame/dynamicRangeMode/maxNumReferenceFrames/clientRefreshRateX100/encoderCscMode, x-nv-vqos[0].bitStreamFormat/bw.max-min/fec.enable=1/fec.minRequiredFecPackets=2/fec.repairPercent/bllFec.enable=0/qosTrafficType, x-nv-audio.surround.*, x-nv-aqos.*, x-nv-general.useReliableUdp=13/featureFlags, x-ss-video[0].chromaSamplingType, x-ml-* attributes",
|
||||
"moonlight-common-c src/Limelight.h — ENCFLG_*, STREAM_CFG_*, STREAM_CONFIGURATION.remoteInputAesKey[16]/remoteInputAesIv[16], VIDEO_FORMAT_H264 0x0001/H265 0x0100/H265_MAIN10 0x0200/AV1_MAIN8 0x1000/AV1_MAIN10 0x2000, COLORSPACE_*, COLOR_RANGE_*, CAPABILITY_*",
|
||||
"moonlight-common-c src/Limelight-internal.h — SS_ENC_CONTROL_V2 0x01, SS_ENC_VIDEO 0x02, SS_ENC_AUDIO 0x04; RtspPortNumber/Control/Audio/VideoPortNumber externs",
|
||||
"moonlight-common-c src/Connection.c — port number init (set from RTSP SETUP), resolveHostName base 47984",
|
||||
"DeepWiki LizardByte/Sunshine port management — base 47989 with offsets: video 47998 (+9), control 47999 (+10), audio 48000 (+11), RTSP 48010 (+21) via net::map_port (1024-65535 validated)",
|
||||
"lumen-core src/crypto.rs and src/config.rs (read directly) — existing AES-128-GCM SessionCrypto (salt+BE-seq nonce, NOT GameStream-exact) and Config/FecConfig for reuse assessment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"area": "GameStream video stream wire format + FEC + encryption (host→client video plane)",
|
||||
"summary": "Each video UDP datagram is one FEC shard: a 12-byte RTP_PACKET, then 4 reserved bytes, then a 16-byte NV_VIDEO_PACKET header, then the shard payload. RTP scalar fields are BIG-endian (network order); NV_VIDEO_PACKET scalar fields (streamPacketIndex, frameIndex, fecInfo) are LITTLE-endian. packetType is unused for video on the wire (Sunshine never sets it for video; client reads dataOffset from FLAG_EXTENSION instead). The frame's encoded bitstream is prefixed with an 8-byte video_short_frame_header_t (frame type, last-payload-len, processing latency), then split into one of 1..4 FEC blocks, each block striped into fixed-size data shards of blocksize = packetSize + MAX_RTP_HEADER_SIZE(16). Reed-Solomon GF(2^8) parity (via the nanors library, used identically by both Sunshine host and Moonlight client) is computed per block: parity_shards = ceil(data_shards * fecPercentage / 100). The shard's RS index within its block is derived purely from (sequenceNumber - blockLowestSequenceNumber); data shards occupy indices [0, data_shards), parity [data_shards, data_shards+parity). fecInfo packs fecIndex(10b)<<12 | dataShards(10b)<<22 | fecPercentage(8b)<<4. multiFecBlocks packs currentBlock(2b)<<4 | (lastBlock=blocks-1)(2b)<<6; multiFecFlags is always 0x10. When video encryption is negotiated (SS_ENC_VIDEO), each fully-formed shard (the whole blocksize buffer: RTP+reserved+NV_VIDEO_PACKET+payload) is AES-128-GCM encrypted IN PLACE *after* FEC encoding, and a 32-byte ENC_VIDEO_HEADER (iv[12], frameNumber[4 LE], tag[16]) is prepended as a wire prefix; there is NO AAD. The client strips/decrypts the prefix first, then runs FEC reconstruction over the decrypted shards. The AES key is the 16-byte RIKEY (StreamConfig.remoteInputAesKey / Sunshine launch_session.gcm_key), shared with the control stream; the IV is a deterministic 64-bit little-endian counter in iv[0..8], zero in iv[8..11], and 'V' (0x56) in iv[11].",
|
||||
"ports": [
|
||||
"Video RTP/UDP: client connects to host UDP port 47998 (RTP_VIDEO_PORT, base 47998). Host receives the client's first packet on this port to learn the client's source address/port, then sends all video shards back to that endpoint.",
|
||||
"(context) Audio RTP/UDP 47998+1=47999, Control/ENet UDP 47999+1 / 47999, RTSP TCP 48010 — not part of this video-plane spec but relevant to the session bring-up that supplies RIKEY/packetSize/fecPercentage."
|
||||
],
|
||||
"wire_formats": [
|
||||
{
|
||||
"name": "RTP_PACKET (12 bytes, fields BIG-endian)",
|
||||
"layout": "offset 0: uint8 header; offset 1: uint8 packetType; offset 2: uint16 sequenceNumber (BE); offset 4: uint32 timestamp (BE, 90kHz clock); offset 8: uint32 ssrc (BE). Host sets header = 0x80 | FLAG_EXTENSION (0x10) = 0x90. packetType is NOT set by Sunshine for video (left as whatever / 0); the client does not read it for video — it detects the extension purely via (header & FLAG_EXTENSION). sequenceNumber = BE16(lowseq + shardIndexInBlock); timestamp = BE32(round to 90kHz of (frame_timestamp - video_epoch)); ssrc copied through (host leaves it, client preserves it during recovery).",
|
||||
"notes": "From moonlight Video.h _RTP_PACKET and Sunshine stream.cpp lines ~1490-1493. FLAG_EXTENSION=0x10 (Video.h). FIXED_RTP_HEADER_SIZE=12, MAX_RTP_HEADER_SIZE=16. The extra 4 bytes (the 'reserved') exist because of the extension flag."
|
||||
},
|
||||
{
|
||||
"name": "reserved[4] (4 bytes)",
|
||||
"layout": "offset 12: 4 opaque bytes (Sunshine: `char reserved[4];` inside video_packet_raw_t). Client computes dataOffset = sizeof(RTP_PACKET)=12, then += 4 because (header & FLAG_EXTENSION), giving dataOffset=16 before NV_VIDEO_PACKET. Contents are not interpreted.",
|
||||
"notes": "RtpVideoQueue.c RtpvAddPacket: `int dataOffset = sizeof(*packet); if (header & FLAG_EXTENSION) dataOffset += 4; // 2 additional fields`. So the NV header always starts at byte 16."
|
||||
},
|
||||
{
|
||||
"name": "NV_VIDEO_PACKET (16 bytes, scalar fields LITTLE-endian)",
|
||||
"layout": "offset 16: uint32 streamPacketIndex (LE); offset 20: uint32 frameIndex (LE); offset 24: uint8 flags; offset 25: uint8 extraFlags; offset 26: uint8 multiFecFlags; offset 27: uint8 multiFecBlocks; offset 28: uint32 fecInfo (LE). Host: streamPacketIndex = ((uint32)(lowseq + x)) << 8 (low byte zero); frameIndex = frame index counter; flags = FLAG_CONTAINS_PIC_DATA(0x1) [| FLAG_SOF(0x4) if x==0] [| FLAG_EOF(0x2) if x==lastDataPkt]; multiFecFlags = 0x10; multiFecBlocks = (blockIndex<<4) | ((fec_blocks_needed-1)<<6); extraFlags 0 (NV_VIDEO_PACKET_EXTRA_FLAG_LTR_FRAME=0x1).",
|
||||
"notes": "From Video.h _NV_VIDEO_PACKET and stream.cpp ~1434-1495. Client byteswaps streamPacketIndex/frameIndex/fecInfo with LE32 in RtpVideoQueue.c lines 564-566. Depacketizer does streamPacketIndex >>= 8; &= 0xFFFFFF (24-bit stream index). Flags/extraFlags/multiFecFlags/multiFecBlocks are single bytes (no swap)."
|
||||
},
|
||||
{
|
||||
"name": "fecInfo bit packing (uint32, LE on wire)",
|
||||
"layout": "fecInfo = (fecIndex << 12) | (dataShards << 22) | (fecPercentage << 4). Decode (RtpVideoQueue.c): fecIndex = (fecInfo & 0x3FF000) >> 12 (10 bits, the shard's RS index within its block); dataShards = (fecInfo & 0xFFC00000) >> 22 (10 bits); fecPercentage = (fecInfo & 0xFF0) >> 4 (8 bits). Bits 0..3 unused. parityShards is NOT transmitted — client recomputes: parityShards = (dataShards * fecPercentage + 99) / 100.",
|
||||
"notes": "stream.cpp ~1485-1488 set side; RtpVideoQueue.c lines 583, 703-705 decode side. fecIndex tops out at 1023 (>=1024 packets/block = unrecoverable, logged as error). dataShards <= 255 (DATA_SHARDS_MAX)."
|
||||
},
|
||||
{
|
||||
"name": "multiFecBlocks / multiFecFlags bit packing (uint8 each)",
|
||||
"layout": "multiFecFlags = 0x10 (constant; marks multi-FEC protocol). multiFecBlocks = (currentBlockIndex << 4) | ((totalBlocks-1) << 6). Decode: currentBlock = (multiFecBlocks >> 4) & 0x3; lastBlock = (multiFecBlocks >> 6) & 0x3; totalBlocks = lastBlock+1 (1..4). Legacy/non-multiFec servers: client forces multiFecFlags=0x10, multiFecBlocks=0x00.",
|
||||
"notes": "stream.cpp 1438-1439; RtpVideoQueue.c 584, 709. Only 2 bits each for current and last block → max 4 FEC blocks per frame (MAX_FEC_BLOCKS=4)."
|
||||
},
|
||||
{
|
||||
"name": "video_short_frame_header_t (8 bytes, prepended to the frame bitstream, LITTLE-endian)",
|
||||
"layout": "offset 0: uint8 headerType (always 0x01); offset 1: uint16 frame_processing_latency (LE, 1/10 ms, Sunshine ext, 0 if N/A); offset 3: uint8 frameType (1=normal P, 2=IDR, 4=P w/ intra-refresh, 5=P after ref-frame-invalidation); offset 4: uint16 lastPayloadLen (LE, length of final packet's real payload, for codecs like AV1 that can't tolerate zero padding); offset 6: uint8 unknown[2]. This 8-byte header precedes the actual H.264/HEVC/AV1 access unit, and the concatenation is what gets striped into shards.",
|
||||
"notes": "stream.cpp video_short_frame_header_t (static_assert ==8). lastPayloadLen = (payloadSize + 8) % (packetSize - sizeof(NV_VIDEO_PACKET)); if 0 → set to (packetSize - sizeof(NV_VIDEO_PACKET)). frameType 2 == IDR is how the client detects keyframes."
|
||||
},
|
||||
{
|
||||
"name": "ENC_VIDEO_HEADER / video_packet_enc_prefix_t (32 bytes) — present only when SS_ENC_VIDEO",
|
||||
"layout": "offset 0: uint8 iv[12]; offset 12: uint32 frameNumber (LE); offset 16: uint8 tag[16]. This is a WIRE PREFIX that sits in front of the (encrypted) shard, NOT inside the FEC blocksize. On-wire encrypted packet = ENC_VIDEO_HEADER (32B) || ciphertext(blocksize bytes). ciphertext is the AES-128-GCM encryption of the entire plaintext shard (RTP+reserved+NV_VIDEO_PACKET+payload = blocksize). tag is the 16-byte GCM tag. iv is the literal 12-byte nonce used (sent so the client doesn't have to reconstruct it). frameNumber = packet->frame_index().",
|
||||
"notes": "Video.h _ENC_VIDEO_HEADER and stream.cpp video_packet_enc_prefix_t + encrypt call ~1498-1515. Header must be a multiple of 16 bytes so the FEC blocksize stays a multiple of 16 (comment in Video.h). When encryption is on, the SDP negotiation subtracts sizeof(ENC_VIDEO_HEADER) from packetSize (SdpGenerator.c line 325)."
|
||||
},
|
||||
{
|
||||
"name": "Full on-wire datagram (one shard)",
|
||||
"layout": "UNENCRYPTED: [RTP_PACKET 12B][reserved 4B][NV_VIDEO_PACKET 16B][payload up to (packetSize - sizeof(NV_VIDEO_PACKET))]. Total header before payload = 32B = sizeof(video_packet_raw_t). ENCRYPTED: [ENC_VIDEO_HEADER 32B] || AES-GCM-ciphertext( the entire 32B+payload plaintext = blocksize ). The plaintext that gets encrypted INCLUDES the RTP and NV headers; the only cleartext fields on an encrypted packet are the 32-byte ENC_VIDEO_HEADER prefix.",
|
||||
"notes": "Client recv buffer sizing (VideoStream.c 96-99): decryptedSize = packetSize + MAX_RTP_HEADER_SIZE(16) = blocksize; receiveSize = decryptedSize + (encrypted ? 32 : 0). So blocksize == packetSize + 16. After decrypt, client byteswaps RTP seq/timestamp/ssrc to host order, then calls RtpvAddPacket."
|
||||
}
|
||||
],
|
||||
"flow": [
|
||||
"Host has an encoded access unit (HEVC/H264/AV1) for frame N from NVENC. It computes frame_index N (monotonic) and builds the 8-byte video_short_frame_header_t (headerType=0x01, frameType per IDR/P, lastPayloadLen, latency).",
|
||||
"Host prepends sizeof(video_packet_raw_t)=32 bytes of header space per shard via concat_and_insert with payload_blocksize = blocksize - 32, where blocksize = packetSize + 16. This produces the striped payload buffer with room for per-shard headers.",
|
||||
"Host decides FEC block count: max_data_shards_per_fec_block = (255*100)/(100+fecPercentage); fec_blocks_needed = ceil(payload / (max_data_shards*blocksize)), capped at 4. If >4 needed, FEC is disabled for that frame (fecPercentage=0). Each block aligned to blocksize: aligned_size = roundup(payload/blocks, blocksize).",
|
||||
"For each FEC block: host fills each data shard's RTP+NV headers (frameIndex, streamPacketIndex=(lowseq+x)<<8, multiFecFlags=0x10, multiFecBlocks, flags incl SOF on x==0 and EOF on last data packet).",
|
||||
"Host calls fec::encode(block, blocksize, fecPercentage, minRequiredFecPackets, prefixsize): data_shards = ceil(blockBytes/blocksize) (last data shard zero-padded), parity_shards = ceil(data_shards*fecPercentage/100) (raised to minRequiredFecPackets if below), then nanors reed_solomon_new(data_shards, parity_shards) + reed_solomon_encode over all shards at blocksize.",
|
||||
"Host stamps every shard (data AND parity) with fecInfo = (x<<12 | data_shards<<22 | percentage<<4), rtp.header=0x80|0x10, rtp.sequenceNumber=BE16(lowseq+x), rtp.timestamp=BE32(timestamp), multiFecBlocks, frameIndex.",
|
||||
"If video encryption: for each shard, build iv = (gcm_iv_counter as 8 LE bytes)||0,0,0 with iv[11]='V'; increment counter; AES-128-GCM encrypt the whole blocksize buffer in place (no AAD), writing tag into the prefix; set prefix.frameNumber=frame_index, prefix.iv=iv. Wire packet = prefix(32B)||ciphertext(blocksize).",
|
||||
"Host sends all data then parity shards as UDP datagrams (paced ~80% of 1Gbps, batched up to 64 packets) to the client's video endpoint; lowseq advances by total shards across all blocks of the frame.",
|
||||
"Client recv: if encrypted, read prefix, drop early if prefix.frameNumber < currentFrameNumber, else AES-128-GCM decrypt ciphertext into buffer using iv+tag from prefix (auth-fail → drop). Then byteswap RTP seq/timestamp/ssrc to host order.",
|
||||
"Client RtpvAddPacket: parse NV header (LE32 swaps), derive fecIndex, currentBlock, dataShards, fecPercentage, recompute parityShards; set bufferLowestSequenceNumber = seq - fecIndex; place shard at RS index = seq - bufferLowestSequenceNumber.",
|
||||
"When a block has >= dataShards shards (data+parity), client runs nanors reed_solomon_decode(rs, packets, marks, totalPackets, blocksize) to recover any missing data shards (missing slots zero-filled, marks[]=1 for missing). Recovered data shards get synthetic RTP headers (seq/header/timestamp/ssrc copied from a present packet).",
|
||||
"Client advances through blocks (currentBlock 0..lastBlock); once all blocks' data shards are present/recovered it strips the 32-byte video_packet_raw_t header off each data shard, concatenates payloads in sequence order, parses the 8-byte short frame header, and hands the reassembled access unit to the depacketizer/decoder."
|
||||
],
|
||||
"crypto": "\"Cipher: AES-128-GCM (EVP_aes_128_gcm). Key: the 16-byte RIKEY — Sunshine `launch_session.gcm_key`, Moonlight `StreamConfig.remoteInputAesKey[16]` — established during RTSP/pairing; the SAME key is used for the control stream and (if enabled) audio. There is no separate video key and no key derivation: the raw 16-byte RIKEY is used directly. IV/nonce: 12 bytes, constructed deterministically (NIST SP 800-38D 8.2.1): iv[0..8] = a 64-bit per-session counter (session->video.gcm_iv_counter, starts at 0) copied in NATIVE byte order (little-endian on x86), iv[8..11] = 0 except iv[11] = 'V' (0x56, the video-stream fixed field). The counter increments once per shard. The full 12-byte IV is transmitted in the ENC_VIDEO_HEADER so the client uses it verbatim (it does not reconstruct it). Tag: 16 bytes, standard GCM tag, transmitted in ENC_VIDEO_HEADER. AAD / associated data: NONE — Sunshine's gcm_t::encrypt for video calls EVP_EncryptUpdate only with plaintext (no AAD update), and Moonlight's PltDecryptMessage passes no AAD argument. (NOTE: this differs from lumen-core/crypto.rs which uses seq-as-AAD and a per-direction salt; GameStream video does neither.) Order: FEC FIRST, THEN ENCRYPT — encryption is applied per-shard after RS parity is computed, over the entire blocksize shard buffer (RTP+NV+payload), so the client must DECRYPT each shard before it can run FEC reconstruction. Encrypted plaintext length == blocksize; ciphertext length == blocksize (GCM is a stream cipher, no expansion); on the wire the packet grows only by the 32-byte prefix.\"",
|
||||
"rust_options": "\"FEC math: lumen already has Gf8Coder over `reed-solomon-erasure` (galois_8). CRITICAL RISK: this is NOT guaranteed byte-compatible with nanors. Both Sunshine and Moonlight use nanors (Sunshine's rswrapper.h is 'a drop-in replacement for nanors rs.h', DATA_SHARDS_MAX=255), which uses a specific GF(2^8) field (primitive poly 0x11d) and a Vandermonde-derived generator matrix with a particular systematic encoding. `reed-solomon-erasure` uses Cauchy matrices by default and may produce DIFFERENT parity bytes — meaning Moonlight would FAIL to recover frames where any data shard is lost. RECOMMENDED: vendor/FFI the actual nanors C library (it is tiny, MIT, header+rs.c+oblas) and call reed_solomon_new/encode through a thin Rust FFI, OR port nanors' matrix construction exactly into a new gf8 backend. Do NOT assume reed-solomon-erasure interop without a byte-for-byte test against nanors output. (For lumen-to-lumen P2 traffic, keep the existing coders; for GameStream-client compat, use nanors.) Crypto: use `aes-gcm` (already a dep) but build a NEW path that (a) takes the raw RIKEY as the key, (b) builds the 12-byte IV as counter_le[8]||0||0||0||'V', (c) uses NO AAD, (d) does NOT use the per-direction salt logic in SessionCrypto. The cleanest approach is a small standalone `Aes128Gcm` call rather than reusing SessionCrypto (whose nonce/AAD scheme is incompatible). Byte layout: define `#[repr(C, packed)]` structs RtpPacket, NvVideoPacket, EncVideoHeader and use explicit `to_be_bytes`/`to_le_bytes` per field (RTP=BE, NV=LE) — do not rely on struct memory layout for endianness.\"",
|
||||
"reuse_from_lumen": "\"REUSE the GF(2^8) concept/structure but NOT necessarily the implementation: lumen's `ErasureCoder` trait and `Gf8Coder` (reed-solomon-erasure) give the right data||parity systematic layout and the 255-shard ceiling, but parity bytes likely won't match nanors — so for client-facing GameStream compat add a `nanors`-backed coder (FFI or exact port) behind the same trait. The trait's reconstruct(data_count, recovery_count, received: indices 0..K originals, K..K+M recovery) maps cleanly onto Moonlight's layout (data shards at RS index 0..dataShards, parity after) so the adapter just needs to map RTP-seq→shard-index. REUSE aes-gcm crate but NOT SessionCrypto (its salt+seq-AAD scheme is wire-incompatible with GameStream video which uses no AAD and a counter||'V' IV). REUSE lumen's UDP transport for sending datagrams. Do NOT reuse lumen's internal 40-byte packet format — GameStream needs the exact 12+4+16 header + optional 32-byte enc prefix. NEW work: a `gamestream` wire module in lumen-host (NOT lumen-core, to keep lumen-core's clean internal protocol) that (1) builds RTP_PACKET/NV_VIDEO_PACKET/ENC_VIDEO_HEADER bytes, (2) implements the frame→FEC-block split (max 4 blocks, the 255/(1+F) shard math), (3) drives a nanors coder, (4) does the per-shard counter-IV AES-128-GCM-no-AAD encrypt, (5) paces/batches sends. Best location: a new file like crates/lumen-host/src/gamestream/video.rs (host side) with the nanors FFI either in lumen-host or a small new crate; lumen-core stays the lumen-native protocol and only its aes-gcm + the gf8 *math* are conceptually shared. The 'adapter' lives at the lumen-host pipeline seam: take the NVENC access unit + frame metadata from encode.rs and emit GameStream datagrams instead of (or alongside) lumen-native packets.\"",
|
||||
"gotchas": [
|
||||
"ENDIANNESS SPLIT: RTP_PACKET fields are BIG-endian, NV_VIDEO_PACKET fields are LITTLE-endian, within the SAME packet. Easy to get wrong. RTP: BE16/BE32; NV streamPacketIndex/frameIndex/fecInfo: LE32.",
|
||||
"ENCRYPT-AFTER-FEC, not before: the GCM-encrypted region is the WHOLE shard (RTP+NV+payload) and the client must decrypt each shard before FEC. The 32-byte ENC_VIDEO_HEADER is a wire PREFIX outside the FEC blocksize, not part of the protected data. If you FEC after encrypt or include the prefix in the FEC math, recovery breaks.",
|
||||
"NO AAD on video GCM — unlike lumen-core's SessionCrypto which authenticates the sequence number as AAD. Using SessionCrypto verbatim will fail Moonlight's tag check.",
|
||||
"IV counter byte order: Sunshine copies the 64-bit counter with std::copy_n in NATIVE order (little-endian on the x86 build), so iv[0..8] is the counter LE; iv[11]='V'(0x56), iv[8..11]=0. The client uses the transmitted iv verbatim, so as long as you SEND the iv you used, internal byte order is self-consistent — but match LE to mirror Sunshine exactly and to keep nonces unique.",
|
||||
"FEC parity matrix must match nanors EXACTLY. reed-solomon-erasure (lumen's current backend) is likely NOT byte-compatible (Cauchy vs nanors Vandermonde). Without a byte-for-byte match, Moonlight silently fails to recover any frame with a lost data shard. Validate against real nanors output or FFI nanors.",
|
||||
"streamPacketIndex is (lowseq+x)<<8 with low byte zero; client does >>=8 then &0xFFFFFF → a 24-bit stream-wide packet index, distinct from the 16-bit RTP sequenceNumber. Both must be set consistently or the depacketizer's continuity check (FLAG_SOF / streamPacketIndex == lastPacketInStream+1) rejects the frame.",
|
||||
"Max 4 FEC blocks per frame (2-bit fields). Max 1024 packets per block (10-bit fecIndex). Max 255 shards/block (GF(2^8)). data_shards per block = 255*100/(100+fecPercentage). Exceeding these → FEC disabled or unrecoverable frame.",
|
||||
"Last data shard is zero-padded to blocksize before RS encode; lastPayloadLen in the short frame header tells the client the real length of the final packet's payload (needed for AV1). Padding must be zeros so RS math and the client's memset-padding agree.",
|
||||
"fecPercentage and parityShards: the host transmits dataShards and fecPercentage in fecInfo but NOT parityShards; the client recomputes parityShards = (dataShards*fecPercentage+99)/100. Use the IDENTICAL rounding (ceil) or shard indices misalign. Sunshine may also bump fecPercentage up to satisfy minRequiredFecPackets — recompute percentage = 100*parity/data in that case and stamp the bumped value into fecInfo.",
|
||||
"blocksize = packetSize + MAX_RTP_HEADER_SIZE(16). When encryption is enabled the SDP negotiation REDUCES packetSize by sizeof(ENC_VIDEO_HEADER)=32 first, so the encrypted-shard plaintext stays the original size. Get this off-by-32 right or buffers mismatch.",
|
||||
"packetType in RTP_PACKET is effectively unused for video (Sunshine doesn't set it; client ignores it for video, keying on FLAG_EXTENSION instead). Do not rely on a video packetType constant like 97 (that's the AUDIO packetType; audio FEC is 127)."
|
||||
],
|
||||
"sources": [
|
||||
"Sunshine src/stream.cpp — packetTypes[] (audio 97 / audio-fec 127, control types); struct video_short_frame_header_t (8B), video_packet_raw_t (RTP + reserved[4] + NV_VIDEO_PACKET), video_packet_enc_prefix_t (iv[12]/frameNumber/tag[16]); fec::encode() (data/parity shard math, nanors reed_solomon_new/encode, zero-pad last data shard); videoBroadcastThread() (frame->FEC-block split, fecInfo/multiFecBlocks/streamPacketIndex/flags packing, per-shard AES-GCM encrypt with counter||'V' IV, lines ~1434-1515); session cipher init from launch_session.gcm_key with SS_ENC_VIDEO (lines ~2032-2040).",
|
||||
"Sunshine src/crypto.cpp — gcm_t::encrypt (EVP_aes_128_gcm, no AAD update, EVP_CTRL_GCM_GET_TAG tag_size=16), init_encrypt_gcm (EVP_CTRL_GCM_SET_IVLEN to iv size 12).",
|
||||
"Sunshine src/rswrapper.h — 'drop-in replacement for nanors rs.h', #define DATA_SHARDS_MAX 255, reed_solomon_new/encode/decode signatures.",
|
||||
"Sunshine src/rtsp.cpp — RIKEY/gcm IV context (12-byte IV, deterministic construction comment) confirming RIKEY is the GCM key.",
|
||||
"moonlight-common-c src/Video.h — _RTP_PACKET (12B), _NV_VIDEO_PACKET (16B: streamPacketIndex/frameIndex/flags/extraFlags/multiFecFlags/multiFecBlocks/fecInfo), _ENC_VIDEO_HEADER (iv[12]/frameNumber/tag[16]), FLAG_CONTAINS_PIC_DATA=0x1/FLAG_EOF=0x2/FLAG_SOF=0x4, FLAG_EXTENSION=0x10, FIXED_RTP_HEADER_SIZE=12, MAX_RTP_HEADER_SIZE=16, SS_FRAME_FEC_STATUS.",
|
||||
"moonlight-common-c src/RtpVideoQueue.c — RtpvAddPacket(): dataOffset=12(+4 for extension)=16; LE32 swaps of streamPacketIndex/frameIndex/fecInfo; fecIndex=(fecInfo&0x3FF000)>>12, dataShards=(fecInfo&0xFFC00000)>>22, fecPercentage=(fecInfo&0xFF0)>>4; currentBlock=(multiFecBlocks>>4)&0x3, lastBlock=(multiFecBlocks>>6)&0x3; parityShards=(dataShards*pct+99)/100; bufferLowestSequenceNumber=seq-fecIndex; reconstructFecBlock() nanors reed_solomon_new/decode over totalPackets at blocksize=packetSize+16, RS index = seq-bufferLowestSequenceNumber, recovered shards get synthetic RTP headers.",
|
||||
"moonlight-common-c src/VideoStream.c — VideoReceiveThreadProc(): receiveSize=packetSize+16(+32 if SS_ENC_VIDEO); per-packet AES-GCM decrypt via PltDecryptMessage(ALGORITHM_AES_GCM, key=remoteInputAesKey[16], iv=encHeader->iv[12], tag=encHeader->tag[16], ciphertext after 32B header, NO AAD); early-drop if encHeader->frameNumber < currentFrameNumber; then BE16/BE32 swap of RTP seq/timestamp/ssrc before RtpvAddPacket.",
|
||||
"moonlight-common-c src/VideoDepacketizer.c — processRtpPayload/reassembleFrame: streamPacketIndex >>= 8 & 0xFFFFFF (24-bit), SOF/EOF continuity, frameType for IDR detection.",
|
||||
"moonlight-common-c src/PlatformCrypto.h — ALGORITHM_AES_GCM=2, PltDecryptMessage/PltEncryptMessage signatures (key,iv,tag,input,output — no AAD parameter).",
|
||||
"moonlight-common-c src/Limelight.h — ENCFLG_VIDEO=0x2, ENCFLG_AUDIO=0x1, remoteInputAesKey[16] (the RIKEY).",
|
||||
"moonlight-common-c src/SdpGenerator.c — SS_ENC_VIDEO negotiation; when enabled StreamConfig.packetSize -= sizeof(ENC_VIDEO_HEADER) (32).",
|
||||
"moonlight-common-c .gitmodules + repo tree — nanors at /nanors (rs.c, rs.h, deps/obl GF(2^8) tables), confirming both host(Sunshine via rswrapper) and client use nanors.",
|
||||
"lumen local: crates/lumen-core/src/crypto.rs (SessionCrypto: salt+seq-AAD scheme — incompatible with GameStream video), crates/lumen-core/src/fec/{mod.rs,gf8.rs} (ErasureCoder trait + reed-solomon-erasure galois_8 — needs nanors compat verification)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"area": "GameStream audio stream — UDP RTP transport, Opus multistream config, Reed-Solomon FEC (4+2 over GF(2^8)), AES-CBC encryption",
|
||||
"summary": "The audio stream is a one-way UDP RTP flow from host to client on the \"audio\" port (base+10; with the default GameStream base port 47989 that is 47999 — the well-known \"port 48000\" in the task refers to the default-numbered GameStream audio slot, but the offset is +10 / control is +11; both Sunshine and moonlight derive ports as base+offset). The host sends Opus-encoded 48 kHz audio in RTP packets with a fixed 12-byte RTP header where packetType=97 for data and 127 for FEC. Audio is grouped into fixed FEC blocks of 4 data shards + 2 parity shards (RTPA_DATA_SHARDS=4, RTPA_FEC_SHARDS=2) using Reed-Solomon over GF(2^8); critically Nvidia/Sunshine use a HARDCODED parity matrix that differs from a generic RS implementation (moonlight-common-c overrides its matrix with the bytes {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} from OpenFEC to match the wire). Each Opus frame is one data shard; after every 4th data packet (sequenceNumber % 4 == 0 marks block start, (seq+1)%4==0 triggers encode) two FEC packets are emitted carrying an 12-byte AUDIO_FEC_HEADER after the RTP header. Opus is configured by the host: stereo = 2ch/1 stream/1 coupled, 5.1 = 6ch/4 streams/2 coupled (or 6/0 high quality), 7.1 = 8ch/5 streams/3 coupled (or 8/0 high quality), with channel mappings matching SMPTE/Vorbis order (FL,FR,FC,LFE,RL,RR,SL,SR). samplesPerFrame = 48 * AudioPacketDuration where AudioPacketDuration defaults to 5 ms (240 samples/channel) for lowest latency, optionally 10 ms (480 samples). Audio payload (when SS_ENC_AUDIO is enabled) is encrypted with AES-128-CBC (NOT GCM like video/control), keyed by the same session GCM/RI key, with a per-packet 16-byte IV whose first 4 bytes are big-endian (avRiKeyId + sequenceNumber) and the rest zero; payload is PKCS7-padded. The FEC parity is computed over the encrypted+padded shard bytes, all shards padded to the same block size. This differs from video FEC which uses larger dynamic block sizes, multi-FEC blocks, and a different packet header; audio FEC is a tiny fixed 4+2 layout.",
|
||||
"ports": [
|
||||
"Audio RTP (UDP, host→client): base_port + 10. With default GameStream base 47989 → 47999. The task's 'port 48000' corresponds to the canonical GameStream audio slot; verify against the actual base. Client binds and sends pings to this port (moonlight: SET_PORT(&saddr, AudioPortNumber), AudioPortNumber parsed from RTSP SETUP).",
|
||||
"Video RTP: base_port + 9 (47998 default)",
|
||||
"Control (ENet): base_port + 11 (48000 default)",
|
||||
"RTSP setup: base_port + 21 (48010 default)",
|
||||
"Audio port is dynamically negotiated via RTSP SETUP (moonlight reads AudioPortNumber); do not hardcode — but offset is +10 in Sunshine net::map_port(AUDIO_STREAM_PORT)"
|
||||
],
|
||||
"wire_formats": [
|
||||
{
|
||||
"name": "RTP_PACKET (12-byte audio RTP header)",
|
||||
"layout": "Offset 0: header (uint8, set to 0x80 = RTP version 2, no padding/ext/CSRC). Offset 1: packetType (uint8): 97 (RTP_PAYLOAD_TYPE_AUDIO) for data, 127 (RTP_PAYLOAD_TYPE_FEC) for FEC. Offset 2: sequenceNumber (uint16, big-endian on wire, host-order after parse). Offset 4: timestamp (uint32, big-endian). Offset 8: ssrc (uint32, big-endian; Sunshine sets ssrc=0). Total 12 bytes, then payload.",
|
||||
"notes": "Sunshine: audio_packet.rtp.header=0x80; .packetType=97; .ssrc=0; .sequenceNumber=big(seq); .timestamp=big(ts). seq increments by 1 per packet; timestamp += packetDuration (ms units, i.e. 5 or 10) each packet. Moonlight checks rtp->packetType==97 for data."
|
||||
},
|
||||
{
|
||||
"name": "AUDIO_FEC_HEADER (12 bytes, follows RTP header in FEC packets)",
|
||||
"layout": "Offset 0 (after the 12-byte RTP header, i.e. file offset 12): fecShardIndex (uint8) = 0 or 1 (which of the 2 parity shards). Offset 1: payloadType (uint8) = 97 (the data payload type being protected). Offset 2: baseSequenceNumber (uint16, big-endian) = seq of first data packet in the block (block start). Offset 4: baseTimestamp (uint32, big-endian) = timestamp of first data packet in block. Offset 8: ssrc (uint32, big-endian). Total 12 bytes, then the parity shard bytes (length = blockSize).",
|
||||
"notes": "moonlight struct order: {uint8 fecShardIndex; uint8 payloadType; uint16 baseSequenceNumber; uint32 baseTimestamp; uint32 ssrc;}. Sunshine audio_fec_packet_t = {RTP_PACKET rtp; AUDIO_FEC_HEADER fecHeader;}. FEC packet rtp.packetType=127, rtp.sequenceNumber=big(baseSeq + x + 1) for shard x, fecHeader.payloadType=97, fecHeader.fecShardIndex=x."
|
||||
},
|
||||
{
|
||||
"name": "FEC block (logical grouping)",
|
||||
"layout": "4 data packets (RTPA_DATA_SHARDS) + 2 FEC packets (RTPA_FEC_SHARDS) = 6 total shards (RTPA_TOTAL_SHARDS). Block starts when sequenceNumber % 4 == 0. All shards (data payload + parity) padded to common blockSize for RS. Block can be recovered if any 4 of the 6 shards arrive.",
|
||||
"notes": "Fixed small block: 4+2. moonlight RTPA_FEC_BLOCK holds dataPackets[4], fecPackets[2], marks[6]. Boundary: 'FEC blocks must start on a RTPA_DATA_SHARDS boundary.' This is unlike video which uses variable block sizes and multi-FEC blocks."
|
||||
},
|
||||
{
|
||||
"name": "Encrypted audio payload (AES-128-CBC)",
|
||||
"layout": "When SS_ENC_AUDIO flag set: payload = AES-128-CBC(opus_frame_PKCS7_padded). IV = 16 bytes: bytes[0..4] = big-endian uint32 (avRiKeyId + sequenceNumber), bytes[4..16] = 0. Key = session RI/GCM key (16 bytes, from remoteInputAesKey). Block size rounded via round_to_pkcs7_padded; max_block_size = round_to_pkcs7_padded(2048).",
|
||||
"notes": "CBC, not GCM. No auth tag appended (unlike video/control GCM). moonlight: ivSeq = BE32(avRiKeyId + rtp->sequenceNumber); memcpy(iv,&ivSeq,4); decrypts then strips PKCS7. FEC parity computed over the ENCRYPTED+padded shard."
|
||||
}
|
||||
],
|
||||
"flow": [
|
||||
"RTSP SETUP negotiates audio: client/host agree on audio config (channels via AUDIO_CONFIGURATION mask), AudioPacketDuration (5 ms default, 10 ms fallback), and the audio UDP port (AudioPortNumber). avRiKeyId and the AES key/IV come from the launch/resume request (remoteInputAesKey/Iv).",
|
||||
"Client opens UDP socket to host audio port and begins sending periodic ping packets every 500 ms to punch NAT and tell host where to send (legacy ping = ASCII 'PING' {0x50,0x49,0x4E,0x47}; modern = AudioPingPayload/SS_PING with sequence). Host learns client addr from first ping.",
|
||||
"Host captures PCM, encodes with opus_multistream_encode_float at 48 kHz into samplesPerFrame-sized frames (240 samples/ch at 5 ms).",
|
||||
"Host builds RTP data packet: header=0x80, packetType=97, seq++ (big-endian), timestamp += packetDuration (big-endian), ssrc=0. If SS_ENC_AUDIO: AES-128-CBC encrypt the PKCS7-padded Opus frame with IV=BE32(avRiKeyId+seq)||0. Send to client.",
|
||||
"FEC accumulation: when seq % 4 == 0, record block baseSequenceNumber/baseTimestamp. The 4 (possibly encrypted) data shard payloads are placed in shards_p[seq % 4].",
|
||||
"When (seq+1) % 4 == 0 (i.e. after the 4th data packet): reed_solomon_encode(rs, shards, RTPA_TOTAL_SHARDS=6, blockSize) generates 2 parity shards. Host sends 2 FEC packets: rtp.packetType=127, rtp.sequenceNumber=big(baseSeq + x + 1), fecHeader.fecShardIndex=x (0,1), fecHeader.payloadType=97, baseSequenceNumber/baseTimestamp/ssrc set.",
|
||||
"Client (RtpAudioQueue) groups incoming packets by block (base seq aligned to 4). If ≥4 of 6 shards present and any data missing, reed_solomon_decode(rs, shards, marks, 6, blockSize) recovers them — using the hardcoded Nvidia parity matrix {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c}.",
|
||||
"Client decrypts recovered/received data shards (AES-128-CBC, strip PKCS7), reorders by sequence, and feeds frames to opus_multistream_decoder_create/decode_float."
|
||||
],
|
||||
"crypto": "Cipher: AES-128-CBC (NOT GCM — video & control use GCM, audio uses CBC). Key: 16 bytes = the session AES key (Sunshine launch_session.gcm_key / moonlight StreamConfig.remoteInputAesKey), same key family used for the input/RI channel. IV: 16 bytes, per-packet — first 4 bytes = big-endian uint32 of (avRiKeyId + sequenceNumber), remaining 12 bytes = 0x00. avRiKeyId is a per-session 32-bit value from the RTSP/launch negotiation (session->audio.avRiKeyId). Padding: PKCS7 to AES block (16-byte) multiple; round_to_pkcs7_padded, max_block_size = round_to_pkcs7_padded(2048). No GMAC/auth tag is appended to audio packets. Encryption is gated by the SS_ENC_AUDIO encryption flag (config.encryptionFlagsEnabled & SS_ENC_AUDIO) — if disabled, raw Opus frame is sent. FEC parity is computed over the post-encryption, post-padding shard bytes so recovery yields ciphertext that is then decrypted.",
|
||||
"rust_options": "Opus encode: use the `audiopus` crate or `opus` (libopus bindings) — both expose multistream via opus_multistream_encoder; if missing, FFI to libopus opus_multistream_encoder_create/opus_multistream_encode_float directly. Configure sampleRate=48000, the streams/coupledStreams and mapping per the negotiated AUDIO_CONFIGURATION; frame size = 48*packetDuration samples/ch. AES-128-CBC: use the `aes` + `cbc` crates (cbc::Encryptor<aes::Aes128>) with manual PKCS7 (`block-padding`/`Pkcs7`) — build the 16-byte IV as BE32(avRiKeyId+seq) || [0u8;12]. Reed-Solomon: do NOT use a generic RS matrix; the wire requires Nvidia's specific parity matrix. The `reed-solomon-erasure` crate computes its own (Vandermonde/Cauchy) matrix that will NOT match — either (a) port moonlight's approach: take the rs lib's encode path but inject the OpenFEC parity matrix bytes {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} for the 4+2 case, or (b) hand-roll a tiny GF(2^8) 4-data/2-parity encoder/decoder using that exact 2x4 parity matrix (8 bytes = 2 parity rows × 4 data cols). lumen's existing `reed-solomon` GF(2^8) code can be reused ONLY if it lets you supply a custom generator/parity matrix; otherwise add a dedicated audio-FEC path. Big-endian field writes: use `byteorder`/`to_be_bytes`. UDP: std::net::UdpSocket on a native thread (no async, matching lumen's hot-path rule).",
|
||||
"reuse_from_lumen": "REUSE: lumen-core's AES-128 primitive (crypto.rs) underlies CBC, but the MODE differs — lumen uses AES-128-GCM with per-direction nonce salts + seq-as-AAD; GameStream audio needs AES-128-CBC with the BE32(avRiKeyId+seq) IV and PKCS7, no AAD/tag. So add a CBC path; do not reuse the GCM nonce/AAD scheme for audio. lumen's GF(2^8) Reed-Solomon (reed-solomon crate) is the right field but the MATRIX is wrong for the wire — must supply Nvidia's hardcoded {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} parity matrix or hand-roll the 4+2 encoder; lumen's internal 40-byte packet format and its FEC block sizing are NOT wire-compatible and cannot be reused for the on-wire audio packets. lumen's UDP transport + native-thread pacing model is reusable as plumbing. NEW: 12-byte RTP header serializer, 12-byte AUDIO_FEC_HEADER, the fixed 4+2 audio FEC block state machine (block starts at seq%4==0, encode at (seq+1)%4==0), the Opus multistream encoder integration, the 500 ms ping listener, and the AES-CBC+PKCS7 audio path. These are GameStream-specific and don't exist in lumen-core.",
|
||||
"gotchas": [
|
||||
"AES-CBC, not GCM. Audio is the one stream using CBC; reusing lumen's GCM code verbatim will break interop. No auth tag is on the wire for audio.",
|
||||
"IV is only 4 meaningful bytes: BE32(avRiKeyId + sequenceNumber) then 12 zero bytes. The addition wraps as uint32. avRiKeyId is per-session from RTSP launch.",
|
||||
"The Reed-Solomon parity matrix MUST be Nvidia's hardcoded one. moonlight explicitly notes 'the RS parity matrix computed by our RS implementation doesn't match the one Nvidia uses' and overrides it with the 8 OpenFEC bytes {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c}. A stock reed-solomon-erasure encoder will produce parity the client cannot decode.",
|
||||
"FEC is a FIXED 4+2 block (RTPA_DATA_SHARDS=4, RTPA_FEC_SHARDS=2), unlike video which uses dynamic/large blocks and multi-FEC. Block boundaries must align to seq%4==0 or the client's queue logic rejects them.",
|
||||
"FEC packets carry their OWN incrementing rtp.sequenceNumber = baseSeq + shardIndex + 1, distinct from data packet seq space conceptually but in the same 16-bit counter — get the +x+1 right.",
|
||||
"FEC parity is computed AFTER encryption+padding (over ciphertext shards). All shards must be padded to the same blockSize before encode, and parity packets carry blockSize bytes.",
|
||||
"Port offset is +10 (Sunshine), but the actual audio port is negotiated in RTSP SETUP (AudioPortNumber). Don't hardcode 48000 — 48000 is the CONTROL port (+11) under the default base; audio is +10 (47999 default). Confirm against your base port.",
|
||||
"timestamp increments by packetDuration in ms units (5 or 10), not by sample count — Sunshine: timestamp += packetDuration.",
|
||||
"samplesPerFrame = 48 * AudioPacketDuration → 240 samples/ch at 5 ms default, 480 at 10 ms. Opus must be configured to encode exactly this frame size.",
|
||||
"Surround Opus uses multistream with specific stream/coupled counts (5.1: 4 streams/2 coupled normal or 6/0 high; 7.1: 5/3 normal or 8/0 high) — wrong stream layout makes the client's multistream decoder produce garbage. Channel mapping is FL,FR,FC,LFE,RL,RR,SL,SR (indices 0..7).",
|
||||
"rtp.header byte is 0x80 (RTP v2); ssrc=0 in Sunshine. Match these or some clients may drop packets.",
|
||||
"Client sends 500 ms pings to the audio port; host must read pings to discover the client's UDP source address before sending audio (one-way send relies on the learned addr)."
|
||||
],
|
||||
"sources": [
|
||||
"moonlight-common-c/src/AudioStream.c — packetType==97 check, AES-CBC IV ivSeq=BE32(avRiKeyId+rtp->sequenceNumber), samplesPerFrame=48*AudioPacketDuration, chosenConfig=High/NormalQualityOpusConfig, SET_PORT(&saddr, AudioPortNumber), 500ms ping, MAX_PACKET_SIZE 1400, QUEUED_AUDIO_PACKET",
|
||||
"moonlight-common-c/src/RtpAudioQueue.c — reed_solomon_new(RTPA_DATA_SHARDS, RTPA_FEC_SHARDS), hardcoded OpenFEC parity matrix {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} ('doesn't match the one Nvidia uses'), reed_solomon_decode(rs, shards, marks, RTPA_TOTAL_SHARDS, blockSize), FEC block boundary on RTPA_DATA_SHARDS",
|
||||
"moonlight-common-c/src/RtpAudioQueue.h — RTPA_DATA_SHARDS=4, RTPA_FEC_SHARDS=2, RTPA_TOTAL_SHARDS=6, AUDIO_FEC_HEADER {uint8 fecShardIndex; uint8 payloadType; uint16 baseSequenceNumber; uint32 baseTimestamp; uint32 ssrc;}, RTPA_FEC_BLOCK, RTP_AUDIO_QUEUE",
|
||||
"moonlight-common-c/src/Limelight.h — MAKE_AUDIO_CONFIGURATION, AUDIO_CONFIGURATION_STEREO/51/71, CHANNEL_COUNT_FROM_AUDIO_CONFIGURATION, OPUS_MULTISTREAM_CONFIGURATION {sampleRate,channelCount,streams,coupledStreams,samplesPerFrame,mapping[8]}, remoteInputAesKey[16]/remoteInputAesIv[16]",
|
||||
"moonlight-common-c/src/Limelight-internal.h — extern NormalQualityOpusConfig/HighQualityOpusConfig, extern int AudioPacketDuration, extern SS_PING AudioPingPayload",
|
||||
"moonlight-common-c/src/SdpGenerator.c (via search) — AudioPacketDuration default 5 ms ('Use 5 ms packets by default for lowest latency'), 10 ms fallback",
|
||||
"Sunshine/src/audio.cpp — SAMPLE_RATE=48000, opus_stream_config_t stream_configs[]: STEREO 2ch/1str/1coupled/96000bps, HIGH_STEREO 2/1/1/512000, SURROUND51 6/4/2/256000, HIGH_SURROUND51 6/6/0/1536000, SURROUND71 8/5/3/450000, HIGH_SURROUND71 8/8/0/2048000; opus_multistream_encode_float; buffer 1400",
|
||||
"Sunshine/src/audio.h — opus_stream_config_t {int32 sampleRate;int channelCount;int streams;int coupledStreams;const uint8* mapping;int bitrate;}, stream_params_t, config_t {packetDuration,channels,mask,...}, enum stream_config_e {STEREO,HIGH_STEREO,SURROUND51,HIGH_SURROUND51,SURROUND71,HIGH_SURROUND71,MAX_STREAM_CONFIG}",
|
||||
"Sunshine/src/stream.cpp — audioBroadcastThread: audio_packet.rtp.header=0x80/.packetType=97/.ssrc=0/.sequenceNumber=big(seq)/.timestamp=big(ts); seq++; timestamp+=packetDuration; IV=BE32(avRiKeyId+sequenceNumber); cbc cipher; if seq%4==0 set fecHeader.baseSequenceNumber/baseTimestamp; if (seq+1)%4==0 reed_solomon_encode(rs,shards,RTPA_TOTAL_SHARDS,bytes); fec_packet.rtp.packetType=127/.sequenceNumber=big(seq+x+1); fecHeader.fecShardIndex=x/.payloadType=97; net::map_port(AUDIO_STREAM_PORT); cbc_t{gcm_key,true}; round_to_pkcs7_padded(2048)",
|
||||
"Sunshine/src/platform/common.h — speaker enum {FRONT_LEFT,FRONT_RIGHT,FRONT_CENTER,LOW_FREQUENCY,BACK_LEFT,BACK_RIGHT,SIDE_LEFT,SIDE_RIGHT}; map_stereo={FL,FR}; map_surround51={FL,FR,FC,LFE,BL,BR}; map_surround71={FL,FR,FC,LFE,BL,BR,SL,SR}",
|
||||
"DeepWiki LizardByte/Sunshine Network Configuration — base port 47989, offsets: HTTP+0, HTTPS+1, Video+9 (47998), Audio+10 (47999), Control+11 (48000), RTSP+21 (48010); net::map_port validates 1024-65535",
|
||||
"Sunshine/src/network.h — uint16_t map_port(int port) (maps offset onto base port)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"area": "GameStream wire-format gap analysis + architecture recommendation for lumen-host (P1 / M2)",
|
||||
"summary": "lumen-core today speaks an INTERNAL protocol that is structurally similar to GameStream but byte-incompatible on every wire surface, so a stock Moonlight client cannot connect to it as-is. Differences: (1) lumen prefixes each shard with a 40-byte little-endian `PacketHeader` and no RTP layer; GameStream uses a 12-byte big-endian RTP header + 4 reserved bytes + a 16-byte `NV_VIDEO_PACKET` (28 bytes total) carrying frameIndex/streamPacketIndex/flags and the FEC params bit-packed into a single `fecInfo` u32 and `multiFecBlocks` u8. (2) lumen's RS-FEC interleaves data+recovery shards within one block keyed by `shard_index`; GameStream packs ALL data shards first then ALL parity shards across a contiguous RTP sequence range, derives (data,parity,fecIndex,pct) from `fecInfo`, splits a frame into up to 4 FEC blocks via `multiFecBlocks`, and the data shards must be the literal RTP-framed bytes of the H.264/HEVC NAL slices (the depacketizer concatenates payloads to rebuild Annex-B). (3) lumen seals the whole 40-byte+payload packet under AES-128-GCM with an 8-byte seq prefix and seq-as-AAD; GameStream encrypts only the post-RTP payload, prefixing a `video_packet_enc_prefix_t {iv[12]; u32 frameNumber; u8 tag[16]}` where the IV is an 8-byte little-endian per-stream counter with iv[11]='V'. The RS math itself is identical (ceil(k*pct/100), GF(2^8), <=255 shards) so lumen's `reed-solomon` GF(2^8) coder CAN produce Moonlight-recoverable parity, but ONLY if lumen abandons its own shard layout and emits shards in GameStream's data-then-parity contiguous order with GameStream's exact shard size (packetSize + 4 reserved + RTP). Beyond video, GameStream needs an entire control plane lumen has not started: HTTPS:47984/HTTP:47989 nvhttp pairing (PIN->AES-128 via SHA-256(salt||pin)[..16], ECB challenge exchange, RSA-signed client cert), an RTSP:48010 handshake (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY) carrying SDP `x-nv-*` params, an ENet control stream (UDP 48000) with its own AES-128-GCM framing and opcodes (request-IDR, loss-stats, ping, HDR, termination, rumble), an AES-CBC audio stream (UDP 47999), and mDNS `_nvstream._tcp` advertisement. Recommendation: put the GameStream video/FEC/crypto wire codec as a P1 \"wire mode\" INSIDE lumen-core (the invariant says protocol logic lives in the core), but keep the stateful control plane (nvhttp/RTSP/ENet/pairing/mDNS) in lumen-host as a tokio control-plane adapter that calls into core codec functions, because that machinery is I/O-bound, async, and not part of the hot path.",
|
||||
"ports": [
|
||||
"TCP 47984 — HTTPS nvhttp (paired control: /serverinfo, /pair, /applist, /launch, /resume, /cancel). Client-cert pinned to the paired client.",
|
||||
"TCP 47989 — HTTP nvhttp (unpaired: /serverinfo unauthenticated, /pair PIN flow).",
|
||||
"TCP 48010 — RTSP setup (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY). Plaintext over TCP, or encrypted_rtsp_header_t {u32 typeAndLength MSB=0x80000000; u32 seq; u8 tag[16]} when encryption negotiated.",
|
||||
"UDP 47998 — Video RTP stream (NV_VIDEO_PACKET + RS-FEC). ML_PORT_INDEX_UDP_47998=8.",
|
||||
"UDP 47999 — Audio RTP stream (Opus, AES-CBC, RS-FEC). ML_PORT_INDEX_UDP_47999=9.",
|
||||
"UDP 48000 — ENet control stream (reliable, AES-128-GCM, opcodes). ML_PORT_INDEX_UDP_48000=10.",
|
||||
"UDP/mDNS 5353 — _nvstream._tcp.local advertisement so Moonlight auto-discovers the host.",
|
||||
"Note: Moonlight derives all of these by offset from the HTTP base port (default 47989); changing the base shifts the whole set. lumen-host must advertise the actual HttpsPort/ExternalPort in serverinfo XML."
|
||||
],
|
||||
"wire_formats": [
|
||||
{
|
||||
"name": "DELTA: video packet header (lumen vs GameStream)",
|
||||
"layout": "lumen PacketHeader = 40 bytes, little-endian, repr(C): pts_ns u64, frame_index u32, stream_seq u32, frame_bytes u32, user_flags u32, block_index u16, block_count u16, data_shards u16, recovery_shards u16, shard_index u16, shard_bytes u16, magic u8(0xC9), version u8, fec_scheme u8, flags u8. || GameStream on-wire = RTP_PACKET(12, big-endian: u8 header, u8 packetType, u16 sequenceNumber, u32 timestamp, u32 ssrc) + char reserved[4] + NV_VIDEO_PACKET(16, little-endian: u32 streamPacketIndex@0, u32 frameIndex@4, u8 flags@8, u8 extraFlags@9 (NV_VIDEO_PACKET_EXTRA_FLAG_LTR_FRAME=0x1), u8 multiFecFlags@10, u8 multiFecBlocks@11, u32 fecInfo@12) = 28 bytes before payload.",
|
||||
"notes": "DELTA: drop pts_ns/frame_bytes/shard_bytes from the wire (GameStream carries none of these per-packet); add the RTP header + reserved[4]; replace explicit u16 FEC fields with the bit-packed fecInfo+multiFecBlocks. flags map 1:1 (FLAG_CONTAINS_PIC_DATA=0x1==lumen FLAG_PIC, FLAG_EOF=0x2, FLAG_SOF=0x4). frameIndex == lumen frame_index; streamPacketIndex == per-stream packet counter (NOT lumen stream_seq which is per-AU)."
|
||||
},
|
||||
{
|
||||
"name": "fecInfo bit-packing (GameStream, exact)",
|
||||
"layout": "fecInfo (u32, little-endian field) = (dataShards << 22) | (fecIndex << 12) | (fecPercentage << 4). Decode masks (moonlight-common-c): dataShards=(fecInfo & 0xFFC00000)>>22 (bits 22-31, 10 bits, <=1023 but RS caps at 255); fecIndex=(fecInfo & 0x3FF000)>>12 (bits 12-21, the shard's index within its block); fecPercentage=(fecInfo & 0xFF0)>>4 (bits 4-11). parityShards = (dataShards*fecPercentage + 99)/100 (ceiling). bits 0-3 unused.",
|
||||
"notes": "This is IDENTICAL math to lumen's FecConfig::recovery_for (ceil(k*pct/100)). lumen already computes data_shards/recovery_shards as explicit u16; the only delta is packing them into this bitfield and emitting fecIndex as the contiguous index across [0..data) then [data..data+parity)."
|
||||
},
|
||||
{
|
||||
"name": "multiFecBlocks bit-packing (GameStream, exact)",
|
||||
"layout": "multiFecBlocks (u8) = (blockIndex << 4) | ((fec_blocks_needed - 1) << 6). Decode: fecCurrentBlockNumber=(multiFecBlocks>>4)&0x3; lastBlockNumber=(multiFecBlocks>>6)&0x3. Max 4 FEC blocks per frame (2 bits each).",
|
||||
"notes": "DELTA vs lumen: lumen uses u16 block_index/block_count with no 4-block ceiling. For P1 wire mode, max_data_per_block must be chosen so a frame needs <=4 blocks AND each block <=255 total shards. lumen's p1_defaults (max_data_per_block=200, 15% FEC -> 230 total) already respects 255; just cap blocks at 4 for GameStream mode."
|
||||
},
|
||||
{
|
||||
"name": "RS-FEC shard arrangement (the recoverability question)",
|
||||
"layout": "GameStream: within one FEC block, RTP sequence numbers are contiguous: data shards occupy [bufferLowestSequenceNumber .. bufferFirstParitySequenceNumber-1], parity shards immediately follow. totalPackets = highest-lowest+1 = dataShards+parityShards. Each shard is exactly receiveSize = packetSize + MAX_RTP_HEADER_SIZE bytes, the last data shard zero-padded to receiveSize. Decode: rs=reed_solomon_new(dataShards, parityShards); reed_solomon_decode(rs, packets[], marks[], totalPackets, receiveSize) with marks[i]=1 for missing. Data shards = the RTP-framed bytes of the video payload concatenated; depacketizer strips RTP+NV header and concatenates payloads to rebuild the Annex-B AU.",
|
||||
"notes": "lumen TODAY: shard_index addresses data [0..K) then recovery [K..K+M) within a block, reconstruct() takes received[] of length K+M with None=lost — STRUCTURALLY THE SAME ordering as GameStream's data-then-parity. VERDICT: lumen's reed-solomon GF(2^8) coder CAN produce moonlight-recoverable shards, because both use the same Vandermonde/Cauchy RS over GF(2^8) with data-first layout. BUT the byte CONTENT of each data shard must be GameStream's RTP-framed packet bytes (not lumen's 40-byte-header packets), and the shard size must be packetSize+RTP, and the parity must be computed over those exact bytes. CAVEAT (unverified at byte level): the specific RS library Moonlight uses (reed-solomon-new / Fec.c, a CM256/Plank-style Cauchy matrix) may use a different generator matrix than the Rust `reed-solomon` crate; parity bytes are only interoperable if the matrices match. This MUST be validated against real Moonlight before trusting it — if matrices differ, lumen must port/match Moonlight's Fec.c matrix exactly (this is the single highest-risk interop item)."
|
||||
},
|
||||
{
|
||||
"name": "video AES-GCM crypto (DELTA)",
|
||||
"layout": "GameStream video_packet_enc_prefix_t = { u8 iv[12]; u32 frameNumber; u8 tag[16] } prepended to the encrypted payload. IV = 8-byte little-endian per-stream gcm_iv_counter in iv[0..8], iv[11]='V' (0x56), iv[8..11]=0; counter increments per packet. Cipher = AES-128-GCM, key = the GCM key from /launch (riKey). Only the post-RTP/post-NV payload is encrypted; RTP+NV header stay in clear. video_short_frame_header_t (8 bytes, inside the encrypted payload, first packet of frame) = { u8 headerType=0x01; le_u16 frame_processing_latency; u8 frameType (1=P,2=IDR,4=intra-refresh,5=after-ref-invalidation); le_u16 lastPayloadLen; u8 unknown[2] }.",
|
||||
"notes": "DELTA vs lumen crypto.rs: lumen seals the ENTIRE packet (header+payload) and uses a 4-byte salt + 8-byte big-endian seq nonce with seq as AAD, prefixing an 8-byte seq. GameStream encrypts only payload, uses 8-byte LE counter + 'V' marker (NO AAD), and the prefix carries iv+frameNumber+tag explicitly. lumen's per-direction salt-bit trick is a lumen invention not on the GameStream wire. For P1 wire mode the core needs a SEPARATE gcm path matching this prefix exactly."
|
||||
},
|
||||
{
|
||||
"name": "ENet control crypto + opcodes (new in host)",
|
||||
"layout": "Encrypted control: NVCTL_ENCRYPTED_PACKET_HEADER { le_u16 encryptedHeaderType=0x0001; le_u16 length; u32 seq } then [16-byte AES-GCM tag][encrypted V2 header + payload]. Cipher AES-128-GCM (Sunshine SS_ENC_CONTROL_V2): 12-byte LE IV = seq in bytes 0-3, bytes 10-11='CC'. Plain header V2 = { u16 type; u16 payloadLength }. Opcodes (Gen7 plain): 0x0305 Start A, 0x0307 Start B, 0x0301 invalidate-ref-frames, 0x0201 loss-stats, 0x0206 input, 0x010b rumble, 0x0100 termination, 0x010e HDR, 0x0302 request-IDR (encrypted gen). Periodic ping {le_u16 len=4; le_u32 ts}.",
|
||||
"notes": "Entirely absent from lumen. Belongs in lumen-host (ENet via a Rust ENet crate); the AES-128-GCM seal/open of the control payload can reuse a core crypto primitive but the framing is host-side."
|
||||
},
|
||||
{
|
||||
"name": "audio packet (new in host)",
|
||||
"layout": "audio_packet_t = RTP_PACKET (12) + Opus payload. audio_fec_packet_t = RTP_PACKET + AUDIO_FEC_HEADER. Encryption = AES-128-CBC (NOT GCM); IV = big-endian u32(avRiKeyId + sequenceNumber) where avRiKeyId = first 4 bytes of the launch IV. Fixed RTPA_DATA_SHARDS / RTPA_FEC_SHARDS RS-FEC.",
|
||||
"notes": "Audio is AES-CBC, different from video GCM — a separate codec path. Lower priority for M2 (can stream video-only first; Moonlight tolerates audio coming up after video)."
|
||||
}
|
||||
],
|
||||
"flow": [
|
||||
"PHASE A (core, low risk): Add a P1 'gamestream wire mode' to lumen-core alongside the internal format. New module crates/lumen-core/src/protocol/gamestream.rs implementing (a) RTP+reserved+NV_VIDEO_PACKET serialize/parse with exact bit-packing, (b) a GameStream-layout FEC packetizer/reassembler that emits data-then-parity contiguous RTP shards at packetSize+RTP shard size, (c) the video_packet_enc_prefix_t AES-128-GCM path. Gate behind ProtocolPhase::P1GameStream (already exists). Keep lumen's internal 40-byte format for P2.",
|
||||
"PHASE B (validate the FEC matrix — HIGHEST RISK, do early): Before building any host networking, prove byte-for-byte that lumen's reed-solomon GF(2^8) parity matches Moonlight's expectation. Capture real Sunshine video packets (or vendor moonlight-common-c's Fec.c into a test) and assert lumen-encoded parity is decodable by Moonlight's RS and vice versa. If the generator matrices differ, port Moonlight's Cauchy matrix into lumen's gf8 coder. This gates everything: if shards aren't interoperable, P1 is dead.",
|
||||
"PHASE C (host control plane, in lumen-host): Implement nvhttp on TCP 47989 (HTTP) + 47984 (HTTPS): /serverinfo XML (appversion, GfeVersion, uniqueid, HttpsPort, ExternalPort, mac, MaxLumaPixelsHEVC, ServerCodecModeSupport, currentgame, PairStatus, sessionUrl0), the /pair PIN state machine (getservercert -> clientchallenge -> serverchallengeresp -> clientpairingsecret) with PIN-AES = SHA-256(salt||pin)[..16], AES-128-ECB challenge, SHA-256, X.509 + RSA sign/verify. Persist the paired client cert; pin it for HTTPS client-cert auth.",
|
||||
"PHASE D (RTSP on TCP 48010): OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY. DESCRIBE returns SDP with x-nv-video[0].* , x-nv-vqos[0].fec.* , x-ss-general.* attributes. SETUP returns server_port= per stream. ANNOUNCE parses client's packetSize, fec.minRequiredFecPackets, maximumBitrateKbps, videoEncoderSlicesPerFrame — feed these into the lumen-core Config (shard_payload=packetSize, fec_percent, etc).",
|
||||
"PHASE E (data plane wiring): On PLAY, bind UDP 47998 (video), spawn the M0 capture->NVENC pipeline, and drive lumen-core's P1 packetizer to that socket. Bind UDP 48000 ENet control (request-IDR -> force NVENC keyframe; loss-stats -> adjust; termination). Audio (UDP 47999, AES-CBC) and full input can follow.",
|
||||
"PHASE F (discovery + display): mDNS-advertise _nvstream._tcp. On RTSP SETUP/PLAY, create the wlroots virtual output sized to the negotiated WxH@fps, point M0 capture at it, tear down on RTSP TEARDOWN / ENet termination."
|
||||
],
|
||||
"crypto": "VIDEO (P1 wire): AES-128-GCM, key=riKey from /launch (16 bytes). Per-packet prefix video_packet_enc_prefix_t{iv[12],u32 frameNumber,u8 tag[16]}; IV = 8-byte LE per-stream counter in iv[0..8], iv[11]='V'(0x56), no AAD. Only payload encrypted. ||| CONTROL: AES-128-GCM (Sunshine SS_ENC_CONTROL_V2), 12-byte LE IV = seq[0..4], iv[10..12]='CC', 16-byte tag, NVCTL_ENCRYPTED_PACKET_HEADER prefix. ||| AUDIO: AES-128-CBC, IV = BE u32(avRiKeyId + seq), avRiKeyId = first 4 bytes of launch IV. ||| PAIRING (nvhttp): PIN-derived key = first 16 bytes of SHA-256(salt(16) || ascii-pin(4)); AES-128-ECB for the challenge/response blocks; SHA-256 for the rolling hashes; RSA (server key + client cert) for signing/verifying the pairing secret; X.509 certs exchanged (server cert returned in getservercert, client cert pinned for HTTPS). ||| DELTA vs lumen crypto.rs: lumen uses AES-128-GCM but with a 4-byte random salt + 8-byte BE seq nonce and seq-as-AAD, sealing the WHOLE packet and prefixing 8-byte seq — none of these match GameStream's iv/marker/prefix/no-AAD scheme. lumen has NO ECB/CBC, NO RSA/X.509, NO PIN-KDF. So: keep lumen's GCM for P2; add a distinct gamestream-gcm path for P1; add ECB+CBC+RSA+X.509+SHA-256-KDF in the host pairing layer (rustls/aws-lc-rs/rsa/x509 crates).",
|
||||
"rust_options": "FEC: KEEP the existing `reed-solomon` GF(2^8) coder in lumen-core for math, but it MUST be validated byte-compatible with Moonlight's Fec.c (CM256/Plank Cauchy matrix) — if not, port that matrix. (reed-solomon-simd is GF(2^16), P2 only, NOT moonlight-compatible.) ||| ENet control: `rusty_enet` (pure-Rust ENet 1.3.x, no_std-friendly, actively maintained) — speaks the exact ENet wire protocol Moonlight expects; alternative is FFI to libenet via `enet-sys`. ||| RTSP: NO good off-the-shelf server crate handles GameStream's non-standard interleaved/encrypted RTSP — hand-roll a minimal parser over a tokio TcpListener (it's ~6 verbs); `httparse`-style manual parsing. Do NOT pull a full RTSP stack. ||| HTTPS with pinned client-cert: `axum`/`hyper` + `rustls` (ServerConfig with a custom `ClientCertVerifier` that checks the cert against the paired set) + `tokio-rustls`; or `actix-web` with rustls. The plan already commits to axum+tokio for the control plane. ||| X.509 gen: `rcgen` (generate the self-signed server cert + key on first run); parse/verify client certs with `x509-parser` + `rsa` + `sha2`. PIN-KDF and ECB/CBC/GCM via `aes`, `aes-gcm`, `cbc`, `ecb` (RustCrypto) or `aws-lc-rs`/`openssl`. ||| mDNS: `mdns-sd` (pure-Rust, registers `_nvstream._tcp.local` with TXT records) or `zeroconf` (FFI to Avahi). `mdns-sd` preferred (no daemon dependency). ||| Opus audio: `audiopus`/`opus` crate if/when audio is implemented.",
|
||||
"reuse_from_lumen": "REUSE: (1) lumen-core's GF(2^8) `reed-solomon` coder and its data-then-parity reconstruct() contract — same ordering as GameStream; (2) FecConfig::recovery_for ceil(k*pct/100) — IDENTICAL to Moonlight's parity math; (3) the ReassemblerLimits bounds-before-allocate hardening pattern — reuse the same discipline when parsing attacker-controlled NV_VIDEO_PACKET fields; (4) aes-gcm dependency and crypto.rs structure (the GCM primitive itself, even though nonce/prefix scheme differs); (5) ProtocolPhase::P1GameStream / FecScheme::Gf8 enums already exist as the negotiation hook; (6) lumen-host M0's capture->NVENC pipeline produces exactly the HEVC/H264 Annex-B AUs that become GameStream video payload; (7) the Packetizer/Reassembler split is the right shape — add a parallel GameStream packetizer/reassembler beside them. ||| MUST BUILD NEW: the RTP+NV_VIDEO_PACKET (de)serialization with bit-packed fecInfo/multiFecBlocks; the GameStream-layout shard emitter (contiguous data-then-parity, packetSize+RTP shard size, no 40-byte lumen header); the video_packet_enc_prefix_t GCM path (iv counter + 'V', payload-only, no AAD); the ENTIRE control plane (nvhttp pairing, RTSP, ENet control, mDNS, X.509/RSA, ECB/CBC); audio AES-CBC path. ||| CANNOT REUSE on the wire: lumen's 40-byte PacketHeader, its 8-byte-seq GCM framing, its per-direction salt bit — all are lumen-internal inventions absent from GameStream.",
|
||||
"gotchas": [
|
||||
"RS GENERATOR MATRIX is the #1 interop risk: same GF(2^8) RS and same data-first ordering does NOT guarantee byte-compatible parity. Moonlight's Fec.c uses a specific Cauchy/Vandermonde matrix; the Rust `reed-solomon` crate may differ. Validate against real Moonlight FIRST (Phase B) or all of P1 fails silently as 'unrecoverable loss'.",
|
||||
"GameStream RTP header is BIG-endian; lumen's PacketHeader is little-endian. NV_VIDEO_PACKET itself is little-endian. Don't conflate them.",
|
||||
"streamPacketIndex is a per-STREAM monotonic packet counter (the RTP-ish sequence), NOT lumen's per-AU stream_seq. frameIndex is the per-frame counter. Two different counters.",
|
||||
"multiFecBlocks caps a frame at 4 FEC blocks (2 bits). Combined with the 255-shard GF(2^8) cap, large frames at high res can overflow — this IS the 1 Gbps wall the plan describes. P1 must keep frames within 4 blocks x 255 shards; reduce via slicesPerFrame / bitrate from ANNOUNCE.",
|
||||
"Video GCM uses NO AAD and an 8-byte LE counter + 'V' marker; lumen's GCM uses seq-as-AAD + per-direction salt. A naive reuse of lumen's seal() will produce undecryptable-by-Moonlight packets. Build the gamestream-gcm path separately.",
|
||||
"Encryption is NEGOTIATED (ENCFLG_VIDEO=0x2, ENCFLG_AUDIO=0x1 in serverinfo/SETUP). Many Moonlight setups stream video in the CLEAR on LAN — implement plaintext video first, add GCM second; serverinfo's encryptionSupported/Requested controls this.",
|
||||
"Audio is AES-CBC not GCM, with a BE counter IV — a third distinct crypto scheme. Easy to get wrong if you assume GCM everywhere.",
|
||||
"Pairing PIN key = SHA-256(salt||pin)[..16] where pin is the 4 ASCII digits and salt is 16 raw bytes from the client — order and encoding matter exactly. ECB (not CBC) for the challenge blocks.",
|
||||
"The shard size is packetSize + RTP_HEADER (not just packetSize). lumen's shard_payload must be set to the negotiated packetSize and the shard the core FEC-protects must include the RTP/NV framing bytes, else the depacketizer mis-aligns.",
|
||||
"HTTPS endpoint pins the CLIENT cert obtained during pairing; a stock TLS server that accepts any cert will let unpaired clients in. Use a custom rustls ClientCertVerifier.",
|
||||
"mDNS service name must be exactly _nvstream._tcp with the right TXT records or Moonlight won't auto-discover (manual IP add still works as a fallback for testing)."
|
||||
],
|
||||
"sources": [
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/packet.rs (PacketHeader 40-byte layout, Packetizer/Reassembler, ReassemblerLimits hardening, FLAG_* constants)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/crypto.rs (SessionCrypto AES-128-GCM, 4-byte salt + 8-byte BE seq nonce, seq-as-AAD, per-direction salt bit)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/config.rs (FecConfig::recovery_for ceil(k*pct/100), FecScheme::max_total_shards Gf8=255, ProtocolPhase::P1GameStream, p1_defaults)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/session.rs (seal_for_wire 8-byte seq prefix, submit_frame/poll_frame hot path)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/fec/mod.rs (ErasureCoder trait, data-then-parity reconstruct contract, GF(2^8) Gf8Coder)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-host/src/{web.rs,vdisplay.rs,inject.rs,pipeline.rs,m0.rs} (control-plane stub, VirtualDisplay trait + wlroots/kwin/mutter stubs, M0 capture->NVENC->AU pipeline + lumen-core loopback)",
|
||||
"/home/enricobuehler/lumen/docs/implementation-plan.md sections 3,5,6,8 (P1/P2/P3 strategy, C ABI, virtual-display orchestration, milestones M0/M2)",
|
||||
"moonlight-common-c/src/Video.h (NV_VIDEO_PACKET 16-byte struct, RTP_PACKET 12-byte struct, FLAG_CONTAINS_PIC_DATA/EOF/SOF, FIXED_RTP_HEADER_SIZE)",
|
||||
"moonlight-common-c/src/RtpVideoQueue.c (fecInfo masks 0xFFC00000>>22 / 0x3FF000>>12 / 0xFF0>>4, parity=(data*pct+99)/100, reed_solomon_new/reed_solomon_decode, receiveSize=packetSize+MAX_RTP_HEADER_SIZE, contiguous data-then-parity sequence range, multiFecBlocks>>4&0x3 / >>6&0x3)",
|
||||
"moonlight-common-c/src/ControlStream.c (ENet channels, opcodes 0x0305/0x0307/0x0301/0x0201/0x010b/0x0100/0x010e/0x0302, NVCTL_ENCRYPTED_PACKET_HEADER, AES-128-GCM control IV seq+'CC', ping/loss-stats/HDR/rumble/termination layouts)",
|
||||
"moonlight-common-c/src/Limelight.h (ML_PORT_INDEX/FLAG constants 47984/47989/48010/47998/47999/48000, ENCFLG_AUDIO=0x1/VIDEO=0x2, VIDEO_FORMAT_* codec masks, STREAM_CONFIGURATION fields incl remoteInputAesKey/Iv, packetSize)",
|
||||
"Sunshine/src/stream.cpp (video_packet_raw_t = RTP+reserved[4]+NV_VIDEO_PACKET, fecInfo send packing x<<12|data<<22|pct<<4, multiFecBlocks (block<<4)|((n-1)<<6), video_short_frame_header_t, video_packet_enc_prefix_t iv[12]+frameNumber+tag[16], IV 8-byte counter + iv[11]='V', audio AES-CBC IV=BE(avRiKeyId+seq), CONTROL/VIDEO/AUDIO_STREAM_PORT via map_port)",
|
||||
"Sunshine/src/rtsp.cpp (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY flow, server_port= response, SDP x-nv-video/x-nv-vqos/x-ss-general attributes, encrypted_rtsp_header_t MSB 0x80000000, RTSP_SETUP_PORT default TCP 48010, ANNOUNCE carries packetSize/fec/bitrate/slicesPerFrame)",
|
||||
"Sunshine/src/nvhttp.cpp (endpoints /serverinfo /pair /applist /launch /resume /cancel, HTTPS 47984 / HTTP 47989, pairing state machine getservercert/clientchallenge/serverchallengeresp/clientpairingsecret, serverinfo XML fields)",
|
||||
"Sunshine/src/crypto.cpp (gen_aes_key = SHA-256(salt||pin) truncated to 16 bytes, AES-128 ECB/GCM/CBC modes)",
|
||||
"Moonlight/Sunshine port documentation (TCP 47984/47989/48010, UDP 47998-48000/48010) — moonlight-stream wiki and portforward.com (port roles confirmation)"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -25,9 +25,10 @@ fi
|
||||
CODENAME="$(. /etc/os-release 2>/dev/null && echo "${VERSION_CODENAME:-unknown}")"
|
||||
case "$CODENAME" in
|
||||
noble) ok "Ubuntu 24.04 (noble) — the recommended target" ;;
|
||||
questing) ok "Ubuntu 25.10 (questing) — newer than the tested 24.04; M0 verified here (Sway 1.10, FFmpeg 7.1)" ;;
|
||||
jammy) warn "Ubuntu 22.04 (jammy): Sway 1.7 / FFmpeg 4.4 are too old for the M0 path. \
|
||||
Strongly prefer 24.04, or build Sway/wlroots + FFmpeg 7.x from source here." ;;
|
||||
*) warn "Unrecognized release '$CODENAME' — proceeding, but package names are tuned for noble." ;;
|
||||
Strongly prefer 24.04+, or build Sway/wlroots + FFmpeg 7.x from source here." ;;
|
||||
*) warn "Unrecognized release '$CODENAME' — proceeding, but package names are tuned for noble/questing." ;;
|
||||
esac
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo"
|
||||
|
||||
@@ -92,7 +93,50 @@ apt_install "Wayland dev" libwayland-dev wayland-protocols wayland-utils
|
||||
apt_install "DRM/EGL/GBM/VA" libdrm-dev libgbm-dev libgbm1 libegl-dev libegl1 libgles-dev mesa-common-dev libva-dev
|
||||
apt_install "capture + dbus" wf-recorder grim dbus-user-session drm-info mesa-utils
|
||||
apt_try "NVIDIA EGL platform (multiverse)" libnvidia-egl-wayland1 libnvidia-egl-gbm1
|
||||
apt_try "libei (noble only; reis is pure-Rust so optional)" libei-dev
|
||||
apt_try "libei (reis is pure-Rust so optional)" libei-dev
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
log "NVIDIA GL/EGL userspace (headless GPU Wayland needs this — nvidia-utils alone is NOT enough)"
|
||||
# ---------------------------------------------------------------------------
|
||||
# nvidia-utils-NNN ships nvidia-smi + NVENC (libnvidia-encode) but NOT the GL/EGL libs.
|
||||
# Without libnvidia-gl-NNN there is no libEGL_nvidia.so.0 and no GLVND vendor JSON
|
||||
# (/usr/share/glvnd/egl_vendor.d/10_nvidia.json), so libglvnd falls back to Mesa, wlroots
|
||||
# can't init EGL on the NVIDIA GPU, Sway is forced to the pixman software renderer, and the
|
||||
# ScreenCast portal then can't negotiate a dmabuf buffer format (capture fails). Install the
|
||||
# GL package matching the running driver.
|
||||
if have nvidia-smi; then
|
||||
if [ -e /usr/share/glvnd/egl_vendor.d/10_nvidia.json ] && ldconfig -p 2>/dev/null | grep -qi 'libEGL_nvidia'; then
|
||||
ok "NVIDIA GL/EGL userspace present (libEGL_nvidia + 10_nvidia.json)"
|
||||
elif [ -n "${DRV:-}" ]; then
|
||||
apt_try "NVIDIA GL/EGL userspace (libnvidia-gl-$DRV)" "libnvidia-gl-$DRV"
|
||||
[ -e /usr/share/glvnd/egl_vendor.d/10_nvidia.json ] \
|
||||
&& ok "10_nvidia.json now present — GPU EGL should work" \
|
||||
|| warn "10_nvidia.json still missing — install the libnvidia-gl package matching driver $DRV by hand."
|
||||
else
|
||||
warn "Couldn't determine the driver branch; install libnvidia-gl-<NNN> matching 'nvidia-smi' by hand."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
log "GPU device group membership (render + video) — required to open /dev/dri/*"
|
||||
# ---------------------------------------------------------------------------
|
||||
# wlroots opens the render node (/dev/dri/renderD128, group 'render') and the DRM card
|
||||
# (/dev/dri/card*, group 'video'); both are 0660. Without membership Sway aborts with
|
||||
# 'Permission denied' on the render node.
|
||||
TARGET_USER="${SUDO_USER:-$USER}"
|
||||
NEED_GROUPS=""
|
||||
for g in render video; do
|
||||
id -nG "$TARGET_USER" 2>/dev/null | tr ' ' '\n' | grep -qx "$g" || NEED_GROUPS="$NEED_GROUPS $g"
|
||||
done
|
||||
if [ -n "$NEED_GROUPS" ]; then
|
||||
warn "$TARGET_USER is not in:$NEED_GROUPS — adding (takes effect on next LOGIN; re-login or reboot):"
|
||||
for g in $NEED_GROUPS; do echo " $SUDO usermod -aG $g $TARGET_USER"; done
|
||||
$SUDO usermod -aG "$(echo "$NEED_GROUPS" | tr ' ' ',' | sed 's/^,//')" "$TARGET_USER" 2>/dev/null \
|
||||
&& ok "added; LOG OUT AND BACK IN (or reboot) for it to apply" \
|
||||
|| warn "could not usermod automatically — run the commands above."
|
||||
else
|
||||
ok "$TARGET_USER already in render + video"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
log "FFmpeg dev headers (gated — must NOT clobber your custom NVENC build)"
|
||||
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run AFTER headless Sway is up (run-headless-sway.sh), from a second shell on the same
|
||||
# user. It: (1) points this shell at the running Sway, (2) gives HEADLESS-1 a real refresh
|
||||
# clock (an idle/0 mHz output produces no frames), (3) imports the env the ScreenCast portal
|
||||
# needs to find Sway and pick the wlr backend, and (4) writes /tmp/lumen-sway-env.sh so
|
||||
# other shells (e.g. `cargo run -p lumen-host`) can `source` it.
|
||||
#
|
||||
# Usage: bash scripts/headless/prepare-session.sh [WxH@RHz] (default 1920x1080@60Hz)
|
||||
set -euo pipefail
|
||||
|
||||
export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
|
||||
export WAYLAND_DISPLAY="${WAYLAND_DISPLAY:-wayland-1}"
|
||||
export XDG_CURRENT_DESKTOP=sway
|
||||
export DBUS_SESSION_BUS_ADDRESS="${DBUS_SESSION_BUS_ADDRESS:-unix:path=$XDG_RUNTIME_DIR/bus}"
|
||||
SWAYSOCK="$(ls -t "$XDG_RUNTIME_DIR"/sway-ipc.*.sock 2>/dev/null | head -1)"
|
||||
export SWAYSOCK
|
||||
MODE="${1:-1920x1080@60Hz}"
|
||||
|
||||
if ! swaymsg -t get_outputs >/dev/null 2>&1; then
|
||||
echo "Sway IPC not reachable ($SWAYSOCK) — is run-headless-sway.sh running?" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# A real refresh clock so the compositor actually drives frames (and screencopy/PipeWire
|
||||
# get content). Without on-screen motion the screen is static; launch something animated
|
||||
# (e.g. `swaymsg exec foot`) to exercise the encoder.
|
||||
swaymsg output HEADLESS-1 mode "$MODE" >/dev/null
|
||||
echo "HEADLESS-1 set to $MODE"
|
||||
|
||||
# The portal (xdg-desktop-portal-wlr) is D-Bus/systemd activated and must inherit these to
|
||||
# find the compositor and select the wlr ScreenCast backend.
|
||||
systemctl --user import-environment WAYLAND_DISPLAY XDG_CURRENT_DESKTOP SWAYSOCK XDG_RUNTIME_DIR 2>/dev/null || true
|
||||
dbus-update-activation-environment --systemd WAYLAND_DISPLAY XDG_CURRENT_DESKTOP SWAYSOCK 2>/dev/null || true
|
||||
|
||||
cat > /tmp/lumen-sway-env.sh <<EOF
|
||||
export XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR
|
||||
export WAYLAND_DISPLAY=$WAYLAND_DISPLAY
|
||||
export XDG_CURRENT_DESKTOP=sway
|
||||
export DBUS_SESSION_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS
|
||||
export SWAYSOCK=$SWAYSOCK
|
||||
EOF
|
||||
|
||||
cat <<EOF
|
||||
session ready. From any shell on this user:
|
||||
source /tmp/lumen-sway-env.sh
|
||||
swaymsg exec foot # optional: animated on-screen content
|
||||
cargo run -p lumen-host -- m0 --source portal --seconds 5 --out /tmp/lumen-m0.h265
|
||||
ffprobe /tmp/lumen-m0.h265
|
||||
EOF
|
||||
@@ -1,20 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch headless Sway on the NVIDIA VM with a private D-Bus session so the ScreenCast
|
||||
# portal activates. Prints the WAYLAND_DISPLAY (default wayland-1) to use from other shells.
|
||||
# Launch headless Sway on the NVIDIA box for the lumen M0 capture spike.
|
||||
#
|
||||
# Prereq: scripts/bootstrap-ubuntu.sh has run and nvidia-drm.modeset=Y (reboot if not).
|
||||
# Runs on the user's *shared* session bus (NOT a private dbus-run-session) so that the
|
||||
# ScreenCast portal (xdg-desktop-portal-wlr) and the lumen host share one bus. After this
|
||||
# is up, run `prepare-session.sh` from a second shell to set the mode + portal env.
|
||||
#
|
||||
# Prereqs (see docs/linux-setup.md / scripts/bootstrap-ubuntu.sh):
|
||||
# - nvidia-drm.modeset=Y
|
||||
# - the NVIDIA GL/EGL userspace (libnvidia-gl-NNN) — provides libEGL_nvidia + the GLVND
|
||||
# vendor JSON; without it wlroots can't init EGL on the GPU and falls back to pixman,
|
||||
# which the ScreenCast portal cannot capture.
|
||||
# - membership in the `render` + `video` groups (re-login after `usermod -aG`).
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$HERE/env.sh"
|
||||
export DBUS_SESSION_BUS_ADDRESS="${DBUS_SESSION_BUS_ADDRESS:-unix:path=$XDG_RUNTIME_DIR/bus}"
|
||||
mkdir -p "$XDG_RUNTIME_DIR" 2>/dev/null || true
|
||||
chmod 700 "$XDG_RUNTIME_DIR" 2>/dev/null || true
|
||||
|
||||
echo "starting headless Sway (renderer=$WLR_RENDERER). From another shell on this user:"
|
||||
echo " export XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR WAYLAND_DISPLAY=wayland-1"
|
||||
echo " swaymsg -t get_outputs # expect HEADLESS-1"
|
||||
echo "starting headless Sway (renderer=$WLR_RENDERER) on bus $DBUS_SESSION_BUS_ADDRESS"
|
||||
echo "from another shell on this user: bash $HERE/prepare-session.sh"
|
||||
echo
|
||||
|
||||
# --unsupported-gpu is mandatory on Sway 1.9 with the proprietary NVIDIA driver.
|
||||
exec dbus-run-session -- sway --unsupported-gpu
|
||||
# wlroots opens the render node (/dev/dri/renderD128, group 'render'); NVIDIA's EGL uses it.
|
||||
# If this login predates `usermod -aG render,video` (groups not yet active), bridge with
|
||||
# `sg render` for this launch. The clean fix is to re-login so the groups apply natively.
|
||||
if id -nG | tr ' ' '\n' | grep -qx render; then
|
||||
exec sway --unsupported-gpu
|
||||
else
|
||||
echo "note: 'render' group not active in this login — bridging via 'sg render' (re-login to avoid)"
|
||||
exec sg render -c 'sway --unsupported-gpu'
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user