Full project rename, decided 2026-06-10: - Crates/binaries: punktfunk-core / punktfunk-host / punktfunk-client-rs. - C ABI: punktfunk_* symbols, Punktfunk* types, include/punktfunk_core.h, PUNKTFUNK_FEATURE_QUIC guard (header regenerated; cbindgen renames updated, incl. PUNKTFUNK_BTN_*/PUNKTFUNK_AXIS_* wire constants). - Protocol: punktfunk/1 — control-plane magic LMN1 → PKF1, nonce salt lmn1 → pkf1. WIRE BREAK: clients must be rebuilt from this revision. - Env knobs: PUNKTFUNK_VIDEO_SOURCE / PUNKTFUNK_COMPOSITOR / PUNKTFUNK_ZEROCOPY / …. - Host config dir: ~/.config/punktfunk (the box's dir was migrated in place — the persistent identity is unchanged, pinned fingerprints stay valid). - Swift package: PunktfunkKit + PunktfunkCore.xcframework + PunktfunkConnection (Sources/PunktfunkClient app + tests renamed with it); build-xcframework.sh updated. - scripts/: 60-punktfunk.rules, punktfunk-host.service; OpenAPI doc regenerated. Also: scripts/headless/run-headless-kde.sh — full headless Plasma bringup. Root cause of "desktop but no apps/settings" over the stream: plasmashell launched without XDG_MENU_PREFIX=plasma-, so the launcher resolved a nonexistent applications.menu and rendered an empty menu. The script sets the complete KDE session env (menu prefix, KDE_FULL_SESSION, session version) and rebuilds ksycoca before starting plasmashell. Gate: 97/97 tests, clippy -D warnings (both feature sets), fmt, C-ABI harness PASS, zero lumen references left outside .git. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "lumen management API",
|
||||
"description": "Control-plane API for managing a lumen streaming host: host capabilities, runtime status, paired clients, the pairing PIN flow, and session control. Authentication: HTTP bearer token, enforced on every route except `/api/v1/health` when the host is started with a management token (mandatory for non-loopback binds).",
|
||||
"title": "punktfunk management API",
|
||||
"description": "Control-plane API for managing a punktfunk streaming host: host capabilities, runtime status, paired clients, the pairing PIN flow, and session control. Authentication: HTTP bearer token, enforced on every route except `/api/v1/health` when the host is started with a management token (mandatory for non-loopback binds).",
|
||||
"contact": {
|
||||
"name": "unom"
|
||||
},
|
||||
@@ -393,7 +393,7 @@
|
||||
"abi_version": {
|
||||
"type": "integer",
|
||||
"format": "int32",
|
||||
"description": "`lumen-core` C ABI version.",
|
||||
"description": "`punktfunk-core` C ABI version.",
|
||||
"minimum": 0
|
||||
},
|
||||
"status": {
|
||||
@@ -403,7 +403,7 @@
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "`lumen-host` crate version."
|
||||
"description": "`punktfunk-host` crate version."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -425,7 +425,7 @@
|
||||
"abi_version": {
|
||||
"type": "integer",
|
||||
"format": "int32",
|
||||
"description": "`lumen-core` C ABI version.",
|
||||
"description": "`punktfunk-core` C ABI version.",
|
||||
"minimum": 0
|
||||
},
|
||||
"app_version": {
|
||||
@@ -459,7 +459,7 @@
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "`lumen-host` crate version."
|
||||
"description": "`punktfunk-host` crate version."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+31
-31
@@ -1,8 +1,8 @@
|
||||
# lumen — Implementation Plan
|
||||
# punktfunk — Implementation Plan
|
||||
|
||||
*A ground-up low-latency desktop streaming stack, built Linux-first, with a shared Rust protocol core and native clients per platform.*
|
||||
|
||||
> `lumen` is a placeholder codename — rename freely. It fits the lowercase house style (`unom`, `played`, `remplir`) and reads as "glass-to-glass light," which is the whole point.
|
||||
> `punktfunk` is a placeholder codename — rename freely. It fits the lowercase house style (`unom`, `played`, `remplir`) and reads as "glass-to-glass light," which is the whole point.
|
||||
|
||||
---
|
||||
|
||||
@@ -45,14 +45,14 @@ flowchart TD
|
||||
VD --> CAP --> ENC
|
||||
ENC --> COREH
|
||||
IN_H["Input injector<br/>(libei / uinput)"]
|
||||
COREH["lumen-core (C ABI)<br/>protocol · FEC · pacing · crypto"]
|
||||
COREH["punktfunk-core (C ABI)<br/>protocol · FEC · pacing · crypto"]
|
||||
COREH --> IN_H
|
||||
end
|
||||
|
||||
COREH <-->|"UDP+FEC video / QUIC control+audio"| COREC
|
||||
|
||||
subgraph Client["Client (Rust / Swift / Kotlin)"]
|
||||
COREC["lumen-core (same crate, C ABI)"]
|
||||
COREC["punktfunk-core (same crate, C ABI)"]
|
||||
DEC["Decoder<br/>(VideoToolbox / NVDEC / VAAPI)"]
|
||||
PRES["Present + frame pacing"]
|
||||
INP["Input capture"]
|
||||
@@ -61,7 +61,7 @@ flowchart TD
|
||||
end
|
||||
```
|
||||
|
||||
**The load-bearing decision:** `lumen-core` is one crate, compiled once, linked by every host and client through a C ABI. Protocol logic, FEC, packet pacing, jitter buffering, pairing, and crypto live there and exist exactly once. Platform code (capture, encode, decode, present, input, UI) lives outside the core and is written in whatever language suits the platform.
|
||||
**The load-bearing decision:** `punktfunk-core` is one crate, compiled once, linked by every host and client through a C ABI. Protocol logic, FEC, packet pacing, jitter buffering, pairing, and crypto live there and exist exactly once. Platform code (capture, encode, decode, present, input, UI) lives outside the core and is written in whatever language suits the platform.
|
||||
|
||||
---
|
||||
|
||||
@@ -70,8 +70,8 @@ flowchart TD
|
||||
| Phase | Protocol | Clients that work | Bitrate ceiling | Purpose |
|
||||
|------|----------|-------------------|-----------------|---------|
|
||||
| **P1** | GameStream-compatible (existing Moonlight wire format) | All existing Moonlight/Artemis clients | ~1 Gbps (legacy GF(2⁸) FEC) | Ship the Linux virtual-display win with zero client work |
|
||||
| **P2** | `lumen/1` negotiated extension: GF(2¹⁶) FEC, multi-block framing, optional QUIC control | lumen clients only; falls back to P1 for others | Multi-Gbps | Break the wall; introduce native clients |
|
||||
| **P3** | `lumen/1` as primary; GameStream kept as compat shim | lumen everywhere, Moonlight as fallback | Multi-Gbps | Full control of features (mic passthrough, per-client identity, HDR signalling) |
|
||||
| **P2** | `punktfunk/1` negotiated extension: GF(2¹⁶) FEC, multi-block framing, optional QUIC control | punktfunk clients only; falls back to P1 for others | Multi-Gbps | Break the wall; introduce native clients |
|
||||
| **P3** | `punktfunk/1` as primary; GameStream kept as compat shim | punktfunk everywhere, Moonlight as fallback | Multi-Gbps | Full control of features (mic passthrough, per-client identity, HDR signalling) |
|
||||
|
||||
Negotiation: extend the `serverinfo`/RTSP `SETUP` handshake with a capability flag. Old clients never see the flag and get P1 behavior. This is how Apollo/Artemis diverge cleanly, and it keeps you compatible while you build.
|
||||
|
||||
@@ -91,7 +91,7 @@ Negotiation: extend the `serverinfo`/RTSP `SETUP` handshake with a capability fl
|
||||
| QUIC (control/audio) | `quinn` | Datagram ext for audio; reliable streams for control |
|
||||
| TLS / crypto | `rustls` + `ring` (or `aws-lc-rs`) | Pairing, session keys (AES-GCM to match GameStream in P1) |
|
||||
| Serialization | `zerocopy` / `bytes` | Wire structs `#[repr(C)]`, zero-copy parse |
|
||||
| C header gen | `cbindgen` | Generates `lumen_core.h` from the ABI module |
|
||||
| C header gen | `cbindgen` | Generates `punktfunk_core.h` from the ABI module |
|
||||
| Error/log | `tracing` | Structured; feature-gate off the hot path |
|
||||
|
||||
### Linux host dependencies
|
||||
@@ -107,7 +107,7 @@ Negotiation: extend the `serverinfo`/RTSP `SETUP` handshake with a capability fl
|
||||
|
||||
### Apple client (P2+)
|
||||
|
||||
Swift + VideoToolbox (decode) + Metal (present) + SwiftUI. Imports `lumen_core.h` directly via a module map — no glue layer.
|
||||
Swift + VideoToolbox (decode) + Metal (present) + SwiftUI. Imports `punktfunk_core.h` directly via a module map — no glue layer.
|
||||
|
||||
### Ruled out
|
||||
|
||||
@@ -123,32 +123,32 @@ Swift + VideoToolbox (decode) + Metal (present) + SwiftUI. Imports `lumen_core.h
|
||||
Design it on day one; retrofitting an ABI is painful.
|
||||
|
||||
**Principles**
|
||||
- Opaque handles only across the boundary: `LumenSession*`, never Rust types.
|
||||
- Opaque handles only across the boundary: `PunktfunkSession*`, never Rust types.
|
||||
- All cross-boundary structs are `#[repr(C)]`; primitives + pointer/len pairs for buffers.
|
||||
- Async events via registered C callbacks (`fn ptr` + `void* userdata`).
|
||||
- Explicit, documented ownership: who frees what, when. Provide `lumen_*_free` for every allocation that crosses out.
|
||||
- Versioned ABI: `uint32_t lumen_abi_version(void)` + a `LumenConfig` struct whose first field is its own size for forward-compat.
|
||||
- Explicit, documented ownership: who frees what, when. Provide `punktfunk_*_free` for every allocation that crosses out.
|
||||
- Versioned ABI: `uint32_t punktfunk_abi_version(void)` + a `PunktfunkConfig` struct whose first field is its own size for forward-compat.
|
||||
|
||||
**Minimal surface (sketch)**
|
||||
|
||||
```c
|
||||
// lifecycle
|
||||
LumenSession* lumen_session_new(const LumenConfig* cfg);
|
||||
void lumen_session_free(LumenSession*);
|
||||
PunktfunkSession* punktfunk_session_new(const PunktfunkConfig* cfg);
|
||||
void punktfunk_session_free(PunktfunkSession*);
|
||||
|
||||
// host: feed an encoded access unit (the core does FEC + packetize + pace + send)
|
||||
int lumen_host_submit_frame(LumenSession*, const uint8_t* data, size_t len,
|
||||
uint64_t pts_ns, LumenFrameFlags flags);
|
||||
int punktfunk_host_submit_frame(PunktfunkSession*, const uint8_t* data, size_t len,
|
||||
uint64_t pts_ns, PunktfunkFrameFlags flags);
|
||||
|
||||
// client: pull a reassembled, FEC-recovered access unit ready to decode
|
||||
int lumen_client_poll_frame(LumenSession*, LumenFrame* out /*borrowed until next poll*/);
|
||||
int punktfunk_client_poll_frame(PunktfunkSession*, PunktfunkFrame* out /*borrowed until next poll*/);
|
||||
|
||||
// input (both directions): client captures, host receives via callback
|
||||
int lumen_send_input(LumenSession*, const LumenInputEvent*);
|
||||
void lumen_set_input_callback(LumenSession*, LumenInputCb, void* user);
|
||||
int punktfunk_send_input(PunktfunkSession*, const PunktfunkInputEvent*);
|
||||
void punktfunk_set_input_callback(PunktfunkSession*, PunktfunkInputCb, void* user);
|
||||
|
||||
// stats for the frame-pacing/quality logic and the web UI
|
||||
void lumen_get_stats(LumenSession*, LumenStats* out);
|
||||
void punktfunk_get_stats(PunktfunkSession*, PunktfunkStats* out);
|
||||
```
|
||||
|
||||
Keep it this small. Everything platform-specific (how you got the encoded bytes, how you decode them) stays on the platform side.
|
||||
@@ -215,15 +215,15 @@ Sizing is rough and relative (Spike / S / M / L) for a focused solo dev; treat a
|
||||
|
||||
**M0 — Pipeline spike (S).** wlroots headless output → PipeWire capture → VAAPI/NVENC encode → dump H.265 to a file that plays. *Acceptance:* a valid encoded file from a virtual output, no streaming yet. Proves the Linux capture+encode chain end-to-end.
|
||||
|
||||
**M1 — `lumen-core` skeleton + C ABI (M).** Session lifecycle, GameStream-compatible packetization and GF(2⁸) FEC (P1), AES-GCM, `cbindgen` header, a tiny C test harness. *Acceptance:* core links from C; round-trips packets in a loopback test with simulated loss.
|
||||
**M1 — `punktfunk-core` skeleton + C ABI (M).** Session lifecycle, GameStream-compatible packetization and GF(2⁸) FEC (P1), AES-GCM, `cbindgen` header, a tiny C test harness. *Acceptance:* core links from C; round-trips packets in a loopback test with simulated loss.
|
||||
|
||||
**M2 — P1 host: stream to stock Moonlight (L).** Wire M0's pipeline into the core; implement `serverinfo`/pairing/RTSP enough for a real Moonlight client to connect, with a KWin virtual output created on connect and destroyed on disconnect. Input via `reis`/uinput. *Acceptance:* **you play a game on your KDE box streamed to a stock Moonlight client on a virtual display, no dummy plug, no kernel args.** This is the shippable milestone and the project's reason to exist.
|
||||
|
||||
**M3 — Measurement harness (S).** Glass-to-glass latency measurement (on-screen QR/timestamp or photodiode), packet-loss injection, frame-pacing and stall metrics surfaced in the web UI. *Acceptance:* you can quantify a regression. Build this before optimizing anything.
|
||||
|
||||
**M4 — P2 transport: break the wall (L).** Add `lumen/1` negotiation; swap to `reed-solomon-simd` GF(2¹⁶) with multi-block per-frame framing; optional QUIC control/audio. Write a minimal **Rust** reference client (decode via VAAPI, present via wgpu/Vulkan) to exercise it. *Acceptance:* a stable stream above 1.4 Gbps at 5120×1440@240 with loss recovery working; latency unchanged vs. M2.
|
||||
**M4 — P2 transport: break the wall (L).** Add `punktfunk/1` negotiation; swap to `reed-solomon-simd` GF(2¹⁶) with multi-block per-frame framing; optional QUIC control/audio. Write a minimal **Rust** reference client (decode via VAAPI, present via wgpu/Vulkan) to exercise it. *Acceptance:* a stable stream above 1.4 Gbps at 5120×1440@240 with loss recovery working; latency unchanged vs. M2.
|
||||
|
||||
**M5 — Apple client (L).** Swift + VideoToolbox + Metal + SwiftUI, linking `lumen-core` via the C header. *Acceptance:* the Mac Studio plays a stream at native resolution/refresh.
|
||||
**M5 — Apple client (L).** Swift + VideoToolbox + Metal + SwiftUI, linking `punktfunk-core` via the C header. *Acceptance:* the Mac Studio plays a stream at native resolution/refresh.
|
||||
|
||||
**M6 — Feature surface (M, ongoing).** Mic passthrough as a proper encrypted, per-client reverse audio stream (the thing the upstream PR got wrong); HDR signalling; per-client identity/permissions; pause/resume. *Acceptance:* feature parity with Apollo on the items you care about, plus mic done right.
|
||||
|
||||
@@ -257,26 +257,26 @@ Sizing is rough and relative (Spike / S / M / L) for a focused solo dev; treat a
|
||||
## 11. Repo / workspace structure
|
||||
|
||||
```
|
||||
lumen/
|
||||
punktfunk/
|
||||
├── Cargo.toml # workspace
|
||||
├── crates/
|
||||
│ ├── lumen-core/ # protocol, FEC, pacing, crypto — C ABI (cdylib + staticlib)
|
||||
│ ├── punktfunk-core/ # protocol, FEC, pacing, crypto — C ABI (cdylib + staticlib)
|
||||
│ │ ├── src/abi.rs # #[no_mangle] extern "C" surface
|
||||
│ │ ├── src/fec.rs # GF(2^16) blocking over reed-solomon-simd
|
||||
│ │ ├── src/transport/ # udp+fec video, quinn control/audio
|
||||
│ │ ├── src/protocol/ # gamestream-compat (P1) + lumen/1 (P2)
|
||||
│ │ ├── src/protocol/ # gamestream-compat (P1) + punktfunk/1 (P2)
|
||||
│ │ └── cbindgen.toml
|
||||
│ ├── lumen-host/ # Linux host binary
|
||||
│ ├── punktfunk-host/ # Linux host binary
|
||||
│ │ ├── src/capture/ # pipewire / portal
|
||||
│ │ ├── src/encode/ # ffmpeg vaapi/nvenc
|
||||
│ │ ├── src/vdisplay/ # trait + kwin/wlroots/mutter impls
|
||||
│ │ ├── src/input/ # reis + uinput
|
||||
│ │ └── src/web/ # axum config/pairing API
|
||||
│ └── lumen-client-rs/ # reference Rust client (M4)
|
||||
│ └── punktfunk-client-rs/ # reference Rust client (M4)
|
||||
├── clients/
|
||||
│ ├── apple/ # Swift package, imports lumen_core.h (M5)
|
||||
│ ├── apple/ # Swift package, imports punktfunk_core.h (M5)
|
||||
│ └── android/ # Kotlin + JNI (later)
|
||||
├── include/ # generated lumen_core.h
|
||||
├── include/ # generated punktfunk_core.h
|
||||
└── tools/
|
||||
├── latency-probe/
|
||||
└── loss-harness/
|
||||
@@ -286,7 +286,7 @@ lumen/
|
||||
|
||||
## 12. Immediate next actions (first week)
|
||||
|
||||
1. **Stand up the workspace** with `lumen-core` (empty ABI + `cbindgen`) and `lumen-host` skeletons; CI on your Gitea (you already have BuildKit pipelines).
|
||||
1. **Stand up the workspace** with `punktfunk-core` (empty ABI + `cbindgen`) and `punktfunk-host` skeletons; CI on your Gitea (you already have BuildKit pipelines).
|
||||
2. **M0 spike on wlroots:** headless output → PipeWire capture → NVENC/VAAPI encode → playable file. This validates the riskiest *pipeline* assumptions in days, on your real GPU.
|
||||
3. **Read KRdp's source** for how KDE creates virtual outputs and casts them — it's the closest existing reference for the KWin path you'll need in M2.
|
||||
4. **Decide P1 protocol depth:** confirm exactly which `serverinfo`/RTSP/pairing messages a current Moonlight client requires for a successful connect, so M2's compat surface is scoped precisely (this is also the question to take back to the dev who mentioned the 1G limit).
|
||||
|
||||
+15
-15
@@ -1,8 +1,8 @@
|
||||
# Linux host setup — NVIDIA GPU VM (M0/M2)
|
||||
|
||||
How to bring up the build environment for the lumen Linux host on an NVIDIA-GPU Ubuntu VM
|
||||
and run the **M0** capture→encode spike. `lumen-core` already builds and is tested
|
||||
cross-platform; this is about the platform backends in `crates/lumen-host`.
|
||||
How to bring up the build environment for the punktfunk Linux host on an NVIDIA-GPU Ubuntu VM
|
||||
and run the **M0** capture→encode spike. `punktfunk-core` already builds and is tested
|
||||
cross-platform; this is about the platform backends in `crates/punktfunk-host`.
|
||||
|
||||
> Target **Ubuntu 24.04 (noble)**: Sway 1.9, FFmpeg 6.1.1, xdg-desktop-portal 1.18.
|
||||
> 22.04 (jammy) ships Sway 1.7 / FFmpeg 4.4 — too old for this path; build from source or
|
||||
@@ -11,7 +11,7 @@ cross-platform; this is about the platform backends in `crates/lumen-host`.
|
||||
## 1. Bootstrap
|
||||
|
||||
```sh
|
||||
git clone git@git.unom.io:unom/lumen.git && cd lumen && git checkout m1-lumen-core
|
||||
git clone git@git.unom.io:unom/punktfunk.git && cd punktfunk && git checkout m1-punktfunk-core
|
||||
bash scripts/bootstrap-ubuntu.sh
|
||||
```
|
||||
|
||||
@@ -68,11 +68,11 @@ bash scripts/headless/run-headless-sway.sh # success logs "EGL vendor: NV
|
||||
|
||||
# shell 2 — same user: set the client mode, import the portal env, write the env file
|
||||
bash scripts/headless/prepare-session.sh 2560x1440@60Hz
|
||||
source /tmp/lumen-sway-env.sh
|
||||
source /tmp/punktfunk-sway-env.sh
|
||||
swaymsg -t get_outputs # confirm HEADLESS-1 active
|
||||
swaymsg exec foot # optional: animated content to capture
|
||||
bash scripts/headless/capture-smoke-test.sh # wf-recorder (wlr-screencopy) -> hevc_nvenc
|
||||
ffprobe /tmp/lumen-headless-test.mkv # confirm a real H.265 stream
|
||||
ffprobe /tmp/punktfunk-headless-test.mkv # confirm a real H.265 stream
|
||||
```
|
||||
|
||||
`wf-recorder` uses `wlr-screencopy` directly (no portal/D-Bus) — the fastest way to
|
||||
@@ -89,26 +89,26 @@ The wlroots-on-NVIDIA env workarounds (`WLR_RENDERER=gles2`, `WLR_NO_HARDWARE_CU
|
||||
`GBM_BACKEND=nvidia-drm`, `sway --unsupported-gpu`, …) live in
|
||||
`scripts/headless/env.sh` — `source` it before launching anything Wayland.
|
||||
|
||||
## 4. M0 proper — wire it into `lumen-core`
|
||||
## 4. M0 proper — wire it into `punktfunk-core`
|
||||
|
||||
Goal (plan §8): headless output → PipeWire ScreenCast → NVENC → a playable file, then feed
|
||||
the encoded access units into a `lumen_core::Session` (host role). The module seams exist
|
||||
in `crates/lumen-host/src/{vdisplay,capture,encode,inject,pipeline}.rs`.
|
||||
the encoded access units into a `punktfunk_core::Session` (host role). The module seams exist
|
||||
in `crates/punktfunk-host/src/{vdisplay,capture,encode,inject,pipeline}.rs`.
|
||||
|
||||
**Status: implemented and verified end-to-end** in `crates/lumen-host` (`m0.rs`,
|
||||
**Status: implemented and verified end-to-end** in `crates/punktfunk-host` (`m0.rs`,
|
||||
`capture/linux.rs`, `encode/linux.rs`). After the §3 bring-up:
|
||||
|
||||
```sh
|
||||
source /tmp/lumen-sway-env.sh
|
||||
source /tmp/punktfunk-sway-env.sh
|
||||
swaymsg exec foot # animated content
|
||||
# Live portal capture → NVENC HEVC → playable file, with each AU also round-tripped
|
||||
# through a lumen_core host→client Session (FEC + packetize + reassemble) and verified:
|
||||
cargo run -p lumen-host -- m0 --source portal --seconds 5 --out /tmp/lumen-m0.h265
|
||||
ffprobe /tmp/lumen-m0.h265
|
||||
# through a punktfunk_core host→client Session (FEC + packetize + reassemble) and verified:
|
||||
cargo run -p punktfunk-host -- m0 --source portal --seconds 5 --out /tmp/punktfunk-m0.h265
|
||||
ffprobe /tmp/punktfunk-m0.h265
|
||||
# No capture session needed (encode + core only): --source synthetic
|
||||
```
|
||||
|
||||
Verified result: `1920x1080` HEVC, ~300 frames in 5s, `lumen-core loopback … 0 mismatches`.
|
||||
Verified result: `1920x1080` HEVC, ~300 frames in 5s, `punktfunk-core loopback … 0 mismatches`.
|
||||
The portal negotiates packed **`RGB` (24-bit, 3 bpp)** on wlroots; the encoder expands it to
|
||||
`rgb0` (one pad byte/pixel, no colour math) since NVENC accepts `rgb0`/`bgr0` but not
|
||||
`rgb24`. dmabuf zero-copy import is still deferred (plan §9) — this is the CPU-copy path.
|
||||
|
||||
+8
-8
@@ -7,11 +7,11 @@ Ground-truth protocol reference: [`research/gamestream-protocol-research.json`](
|
||||
|
||||
## Architecture (respects the "one core" invariant)
|
||||
|
||||
- **lumen-core** gains a **P1 GameStream wire codec** (`ProtocolPhase::P1GameStream`, the
|
||||
- **punktfunk-core** gains a **P1 GameStream wire codec** (`ProtocolPhase::P1GameStream`, the
|
||||
hook already exists): the exact RTP+`NV_VIDEO_PACKET` framing, the GameStream FEC shard
|
||||
layout, and the video/audio AES-GCM/CBC paths. Hot path, native threads, **no async**.
|
||||
Kept beside lumen's native internal format (P2), selected by phase.
|
||||
- **lumen-host** gains the **control plane** (tokio/axum OK — I/O-bound, not the hot path):
|
||||
Kept beside punktfunk's native internal format (P2), selected by phase.
|
||||
- **punktfunk-host** gains the **control plane** (tokio/axum OK — I/O-bound, not the hot path):
|
||||
mDNS discovery, nvhttp serverinfo + the 4-phase pairing, the RTSP handshake, the ENet
|
||||
control stream + input injection, the virtual-display lifecycle, and Opus audio encode.
|
||||
|
||||
@@ -46,16 +46,16 @@ Ground-truth protocol reference: [`research/gamestream-protocol-research.json`](
|
||||
the client cert is pinned for subsequent HTTPS. 4 phases over `/pair?phrase=…`.
|
||||
- **RTSP** `Session: DEADBEEFCAFE;timeout = 90` (literal), `Transport: server_port=<p>`,
|
||||
`streamid=video/0/0` / `control/13/0`. ANNOUNCE carries the negotiated config
|
||||
(`x-nv-video[0].*`, `x-nv-vqos[0].*`) → maps to `lumen_core::Config`.
|
||||
(`x-nv-video[0].*`, `x-nv-vqos[0].*`) → maps to `punktfunk_core::Config`.
|
||||
|
||||
## The two highest interop risks (validate EARLY)
|
||||
|
||||
1. **RS-FEC matrix compatibility.** Sunshine + Moonlight both use **nanors** (GF(2⁸), poly
|
||||
0x11d, Vandermonde systematic). lumen-core uses `reed-solomon-erasure` (Cauchy) — parity
|
||||
0x11d, Vandermonde systematic). punktfunk-core uses `reed-solomon-erasure` (Cauchy) — parity
|
||||
bytes likely **don't match**, so Moonlight silently fails to recover any frame with a lost
|
||||
data shard. Mitigation: **on a clean LAN with no loss the client never runs RS decode**, so
|
||||
defer this — get a frame decoded first, then FFI/port nanors for loss recovery.
|
||||
2. **Crypto layout.** lumen's `SessionCrypto` (salt + seq-as-AAD) is wire-incompatible. P1
|
||||
2. **Crypto layout.** punktfunk's `SessionCrypto` (salt + seq-as-AAD) is wire-incompatible. P1
|
||||
needs a separate GameStream GCM path. Mitigation: **video encryption is negotiated and
|
||||
usually off on LAN** — implement plaintext video first, add GCM later.
|
||||
|
||||
@@ -67,8 +67,8 @@ Ground-truth protocol reference: [`research/gamestream-protocol-research.json`](
|
||||
- **P1.2 — Launch + RTSP + virtual display.** `/launch` (parse rikey/rikeyid/mode), the RTSP
|
||||
handshake, negotiate `Config`, create a wlroots virtual output sized to the client.
|
||||
*Acceptance: Moonlight completes RTSP and the host stands up the UDP streams.*
|
||||
- **P1.3 — Video (lumen-core P1 codec), plaintext, clean-LAN.** RTP+NV framing + FEC shard
|
||||
layout in lumen-core; wire M0's NVENC AUs → UDP 47998. *Acceptance: Moonlight DISPLAYS video.*
|
||||
- **P1.3 — Video (punktfunk-core P1 codec), plaintext, clean-LAN.** RTP+NV framing + FEC shard
|
||||
layout in punktfunk-core; wire M0's NVENC AUs → UDP 47998. *Acceptance: Moonlight DISPLAYS video.*
|
||||
- **P1.4 — Control + input.** ENet (`rusty_enet`) control stream; decode input → `inject.rs`
|
||||
(uinput/reis); request-IDR → force NVENC keyframe. *Acceptance: mouse/keyboard work.*
|
||||
- **P1.5 — Robustness: FEC recovery + encryption.** nanors-exact FEC; per-shard AES-GCM.
|
||||
|
||||
@@ -65,9 +65,9 @@
|
||||
"9. Client: GET https://host:47984/pair?...&phrase=pairchallenge over mutual-TLS presenting its now-trusted cert; expects paired=1. Pairing complete -> PairState.PAIRED.",
|
||||
"10. On any mismatch the client GET /unpair and returns FAILED / PIN_WRONG."
|
||||
],
|
||||
"crypto": "PIN-derived key: aesKey = HASH(salt[16] || PIN_utf8)[0..16], where HASH = SHA-256 if server appversion major >=7 (Sunshine: 7.1.431.-1) else SHA-1. Salt = client random 16 bytes. PIN is the 4-digit code shown/entered by user; concatenation is salt FIRST then pin. Pairing cipher: AES-128 in ECB mode, NO padding / padding DISABLED (Sunshine ecb_t(key,false); Moonlight AESLightEngine block loop). Inputs are zero-extended to a 16-byte multiple before encryption (a 32-byte SHA-256 hash = 2 blocks; respHash(32)+serverChallenge(16)=48=3 blocks). Per-side proofs: serverHash = SHA(clientChallenge || serverCert.signature || serversecret16); clientHash = SHA(serverChallenge || clientCert.signature || clientSecret16) (cert.signature = the DER signature bytes of the self-signed X.509). Identity binding: each side RSA-signs its own 16-byte secret with its cert's private key using RSA-PKCS1 over SHA-256 (sign256/verify256), other side verifies. Certs: self-signed RSA-2048, SHA-256 signed, ~20-year validity. Result of pairing is NOT a streaming key — it only establishes mutual TLS trust (pinned certs). The actual AES-128 STREAM key is delivered separately at /launch as rikey (16-byte hex) + rikeyid; that is where lumen-core's existing AES-128-GCM session crypto plugs in. IMPORTANT: pairing AES-ECB-no-pad is distinct from and unrelated to lumen-core's AES-128-GCM session sealing.",
|
||||
"rust_options": "HTTP/HTTPS control plane belongs in crates/lumen-host/src/web.rs (the existing stub explicitly permits tokio/axum here, off the hot path). Use axum or hyper for the two servers. TLS with mutual auth + custom cert pinning: use rustls via axum-server/tokio-rustls with a custom ClientCertVerifier (rustls::server::danger::ClientCertVerifier) that accepts any well-formed cert at handshake time and then matches the presented leaf DER/PEM against the paired allow-list (mirror Sunshine's verify callback), OR use openssl/openssl crate to match Sunshine 1:1. XML: build/parse with quick-xml or xml-rs (or just format! the small fixed templates and a tiny extractor). Crypto: aes crate (already a dep transitively) in ECB mode via the `ecb` crate with NoPadding (Aes128 + ecb::Decryptor/Encryptor, manual block handling) — note RustCrypto deprecates ECB so call the block cipher directly (aes::Aes128 + cipher::BlockEncrypt/BlockDecrypt over 16-byte chunks). Hashing: sha2 (SHA-256) and sha1 crates. X.509 self-signed cert generation + RSA-SHA256 sign/verify: rcgen for cert gen, rsa + sha2 for PKCS1v15 sign/verify, x509-parser or x509-cert to extract the cert's signature bytes (cert.getSignature() equivalent) and to do TLS-trust comparison. Persist authorized client certs (PEM) in a small JSON/sled store. Run all of this on tokio in web.rs; keep it fully separate from the native-thread per-frame pipeline.",
|
||||
"reuse_from_lumen": "REUSE little of lumen-core's crypto here — its crypto.rs is AES-128-GCM session sealing (nonce = salt||seq, seq as AAD) for the VIDEO/INPUT plane, which corresponds to the post-/launch rikey stream key, NOT the pairing handshake. Pairing needs AES-128-ECB-no-pad + SHA-256/SHA-1 + RSA, none of which exist in lumen yet and must be newly built (best placed in lumen-host, not lumen-core, since it is control-plane only). The natural seam is the existing web.rs stub (WebConfig::run) whose TODO already says 'GameStream serverinfo, pairing handshake, RTSP SETUP' — implement the two HTTP servers and the 4-phase /pair state machine there. lumen-core's AES-128-GCM SessionCrypto IS reusable downstream: once paired and /launch hands over rikey (16-byte AES key) + rikeyid, feed that key into lumen-core's Session/SessionCrypto for the encrypted video/control planes. The internal 40-byte packet format is unrelated to this HTTP/pairing area. So: new pairing crypto + axum servers in lumen-host/web.rs (control), reuse lumen-core GCM for the data plane post-launch.",
|
||||
"crypto": "PIN-derived key: aesKey = HASH(salt[16] || PIN_utf8)[0..16], where HASH = SHA-256 if server appversion major >=7 (Sunshine: 7.1.431.-1) else SHA-1. Salt = client random 16 bytes. PIN is the 4-digit code shown/entered by user; concatenation is salt FIRST then pin. Pairing cipher: AES-128 in ECB mode, NO padding / padding DISABLED (Sunshine ecb_t(key,false); Moonlight AESLightEngine block loop). Inputs are zero-extended to a 16-byte multiple before encryption (a 32-byte SHA-256 hash = 2 blocks; respHash(32)+serverChallenge(16)=48=3 blocks). Per-side proofs: serverHash = SHA(clientChallenge || serverCert.signature || serversecret16); clientHash = SHA(serverChallenge || clientCert.signature || clientSecret16) (cert.signature = the DER signature bytes of the self-signed X.509). Identity binding: each side RSA-signs its own 16-byte secret with its cert's private key using RSA-PKCS1 over SHA-256 (sign256/verify256), other side verifies. Certs: self-signed RSA-2048, SHA-256 signed, ~20-year validity. Result of pairing is NOT a streaming key — it only establishes mutual TLS trust (pinned certs). The actual AES-128 STREAM key is delivered separately at /launch as rikey (16-byte hex) + rikeyid; that is where punktfunk-core's existing AES-128-GCM session crypto plugs in. IMPORTANT: pairing AES-ECB-no-pad is distinct from and unrelated to punktfunk-core's AES-128-GCM session sealing.",
|
||||
"rust_options": "HTTP/HTTPS control plane belongs in crates/punktfunk-host/src/web.rs (the existing stub explicitly permits tokio/axum here, off the hot path). Use axum or hyper for the two servers. TLS with mutual auth + custom cert pinning: use rustls via axum-server/tokio-rustls with a custom ClientCertVerifier (rustls::server::danger::ClientCertVerifier) that accepts any well-formed cert at handshake time and then matches the presented leaf DER/PEM against the paired allow-list (mirror Sunshine's verify callback), OR use openssl/openssl crate to match Sunshine 1:1. XML: build/parse with quick-xml or xml-rs (or just format! the small fixed templates and a tiny extractor). Crypto: aes crate (already a dep transitively) in ECB mode via the `ecb` crate with NoPadding (Aes128 + ecb::Decryptor/Encryptor, manual block handling) — note RustCrypto deprecates ECB so call the block cipher directly (aes::Aes128 + cipher::BlockEncrypt/BlockDecrypt over 16-byte chunks). Hashing: sha2 (SHA-256) and sha1 crates. X.509 self-signed cert generation + RSA-SHA256 sign/verify: rcgen for cert gen, rsa + sha2 for PKCS1v15 sign/verify, x509-parser or x509-cert to extract the cert's signature bytes (cert.getSignature() equivalent) and to do TLS-trust comparison. Persist authorized client certs (PEM) in a small JSON/sled store. Run all of this on tokio in web.rs; keep it fully separate from the native-thread per-frame pipeline.",
|
||||
"reuse_from_punktfunk": "REUSE little of punktfunk-core's crypto here — its crypto.rs is AES-128-GCM session sealing (nonce = salt||seq, seq as AAD) for the VIDEO/INPUT plane, which corresponds to the post-/launch rikey stream key, NOT the pairing handshake. Pairing needs AES-128-ECB-no-pad + SHA-256/SHA-1 + RSA, none of which exist in punktfunk yet and must be newly built (best placed in punktfunk-host, not punktfunk-core, since it is control-plane only). The natural seam is the existing web.rs stub (WebConfig::run) whose TODO already says 'GameStream serverinfo, pairing handshake, RTSP SETUP' — implement the two HTTP servers and the 4-phase /pair state machine there. punktfunk-core's AES-128-GCM SessionCrypto IS reusable downstream: once paired and /launch hands over rikey (16-byte AES key) + rikeyid, feed that key into punktfunk-core's Session/SessionCrypto for the encrypted video/control planes. The internal 40-byte packet format is unrelated to this HTTP/pairing area. So: new pairing crypto + axum servers in punktfunk-host/web.rs (control), reuse punktfunk-core GCM for the data plane post-launch.",
|
||||
"gotchas": [
|
||||
"appversion MAJOR number is load-bearing: it silently switches the client's pairing hash. Advertise major >=7 (e.g. \"7.1.431.-1\") to get SHA-256; advertise <7 and the client uses SHA-1 with 20-byte hashes (changes all the ECB block counts). Mismatch => silent pairing failure.",
|
||||
"AES-ECB has NO padding and NO IV. Do not use a library that auto-pads (PKCS7) — Sunshine passes ecb_t(key,false). Inputs are zero-extended to 16-byte multiples; a 32-byte SHA-256 hash is exactly 2 blocks but respHash(32)+serverChallenge(16)=48 must be encrypted as one 48-byte buffer.",
|
||||
@@ -90,7 +90,7 @@
|
||||
"moonlight-android PairingManager.java: serverMajorVersion>=7 -> Sha256PairingHash else Sha1PairingHash; saltPin (salt then pin utf-8); generateAesKey = copyOf(hash,16); encryptAes/decryptAes AESLightEngine ECB; performBlockCipher 16-byte block loop with zero-pad blockRoundedSize; phase byte orders (clientchallenge=enc(random16); challengeRespHash=hash(serverChallenge||cert.signature||clientSecret); clientPairingSecret=clientSecret||signData(clientSecret)); Sha256PairingHash.getHashLength=32 / Sha1=20; signData=SHA256withRSA; verifySignature; PairState{NOT_PAIRED,PAIRED,PIN_WRONG,FAILED,ALREADY_IN_PROGRESS}; executePairingChallenge final pairchallenge",
|
||||
"moonlight-common-c src/Limelight.h: SCM_H264=0x1, SCM_HEVC=0x100, SCM_HEVC_MAIN10=0x200, SCM_AV1_MAIN8=0x10000, SCM_AV1_MAIN10=0x20000, SCM_*_444 flags; VIDEO_FORMAT_* constants; default ports 47984/47989/48010 tcp, 47998/47999/48000/48010 udp",
|
||||
"DeepWiki LizardByte/Sunshine NVHTTP page: ServerCodecModeSupport decimal values 3 / 259 / 3843",
|
||||
"lumen repo (local): crates/lumen-host/src/web.rs (WebConfig stub, control-plane seam) and crates/lumen-core/src/crypto.rs (AES-128-GCM SessionCrypto = data-plane, not pairing)"
|
||||
"punktfunk repo (local): crates/punktfunk-host/src/web.rs (WebConfig stub, control-plane seam) and crates/punktfunk-core/src/crypto.rs (AES-128-GCM SessionCrypto = data-plane, not pairing)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -153,10 +153,10 @@
|
||||
"7. Client sends PLAY (CSeq:7; single PLAY '/' for GFE≥7.1.431, else per-stream). Host replies 200 OK and the RTP video/audio + control/input flows begin on the UDP ports. First UDP datagram from client on each A/V port carries the X-SS-Ping-Payload so the host learns the client source port (NAT punch / port-learn)."
|
||||
],
|
||||
"crypto": "RIKEY/RIKEYID origin: the Moonlight client app generates remoteInputAesKey[16] and remoteInputAesIv[16] (STREAM_CONFIGURATION in Limelight.h) before connecting; rikey = hex(remoteInputAesKey), rikeyid = a 32-bit int. They are delivered to the host NOT in RTSP but in the HTTPS /launch query string. Sunshine make_launch_session: gcm_key = from_hex_vec(rikey) (16 bytes, the AES-128-GCM key shared by video/audio/control/input/RTSP ciphers); iv (16 bytes) = big-endian uint32(rikeyid) in bytes[0..4], zero-padded. AES-128-GCM is used everywhere. Per-stream 12-byte GCM nonce construction (Sunshine stream.cpp / rtsp.cpp): VIDEO (host→client): bytes[0..]=gcm_iv_counter (LE incrementing), byte[11]='V'. CONTROL host→client: 12-byte iv = seq(LE) || byte[10]='H', byte[11]='C'. CONTROL client→host: seq(LE) || byte[10]='C', byte[11]='C'. RTSP (encrypted handshake): seq(BE in bytes[0..4]) || byte[10]='C'(client)|'H'(host), byte[11]='R'. Encryption is negotiated, not key-exchanged: DESCRIBE advertises x-ss-general.encryptionSupported (default SS_ENC_CONTROL_V2|SS_ENC_AUDIO=0x05, +SS_ENC_VIDEO=0x02 if allowed) and encryptionRequested (default SS_ENC_CONTROL_V2=0x01, +VIDEO|AUDIO if MANDATORY); the client echoes the chosen bitmask in ANNOUNCE x-ss-general.encryptionEnabled, and x-nv-general.featureFlags bit 0x20 forces SS_ENC_AUDIO on. Flags: SS_ENC_CONTROL_V2=0x01, SS_ENC_VIDEO=0x02, SS_ENC_AUDIO=0x04. Codec tag bitStreamFormat: 0=H264,1=HEVC,2=AV1 (client capability bits VIDEO_FORMAT_H264=0x0001,H265=0x0100,H265_MAIN10=0x0200,AV1_MAIN8=0x1000,AV1_MAIN10=0x2000).",
|
||||
"rust_options": "For RTSP on 48010: a tiny synchronous TCP server using std::net::TcpListener on a native thread (NO tokio — keep off the hot path, consistent with lumen's no-async-on-hot-path invariant). Parse RTSP/1.0 manually: read until \\\\r\\\\n\\\\r\\\\n, split request line + headers, read Content-length body for ANNOUNCE. There is no need for a crate; a hand-rolled parser mirroring Sunshine's parseRtspMessage is simplest and avoids pulling RTSP libs that assume standard semantics (the streamid= targets and DEADBEEFCAFE session break them). For SDP build/parse, just format!/split on lines — it is line-oriented a=key:value. For the /launch HTTPS endpoint, reuse the existing crates/lumen-host/src/web.rs seam; a small hyper or tiny_http + rustls TLS server (control plane only, async OK here since it is not the hot path — matches the 'quic feature gated' precedent). For encrypted-RTSP framing use the aes-gcm crate already in lumen-core. Suggested new types: an RtspServer in lumen-host that produces a lumen_core::Config from the ANNOUNCE map. Hex decode rikey with the `hex` crate (or from_str_radix). Big-endian rikeyid → IV with u32::to_be_bytes.",
|
||||
"reuse_from_lumen": "REUSE: lumen-core/src/crypto.rs SessionCrypto already implements AES-128-GCM with per-direction salting and seq-as-AAD — but NOTE it is NOT byte-compatible with GameStream's nonce layout. lumen uses nonce = salt(4) || seq(8, BE) with a direction bit folded into salt[0], whereas GameStream uses iv = seq(LE or BE per stream) with literal direction/stream marker bytes at [10]/[11] ('V', 'H'/'C'+'C', 'C'/'H'+'R'). To talk to stock Moonlight you must add a GameStream-exact nonce mode (new constructor or a feature) rather than reuse the existing salt scheme verbatim. The Aes128Gcm cipher init and seal/open plumbing are reusable. REUSE: lumen-core Config/FecConfig — fec_percent maps to GameStream's repairPercent and the recovery_for() ceil(k*pct/100) already matches GameStream's FEC math; map ANNOUNCE packetSize→shard_payload, maximumBitrateKbps→bitrate, fec.minRequiredFecPackets→minRequiredFecPackets. FecScheme::Gf8 is the GameStream-compatible field. BUILD NEW: the entire RTSP/SDP/launch negotiation layer (lumen's internal 40-byte packet format and Config are not wire-exact to RTP/RTSP); the RTSP server, SDP describe/announce codec, the /launch query parser that produces gcm_key+iv from rikey/rikeyid, and the GameStream RTP video/audio packetization + RTPFEC are all new (separate areas). The Session in lumen-core can consume the negotiated Config but its on-wire packet header must be swapped for GameStream RTP for Moonlight compat.",
|
||||
"rust_options": "For RTSP on 48010: a tiny synchronous TCP server using std::net::TcpListener on a native thread (NO tokio — keep off the hot path, consistent with punktfunk's no-async-on-hot-path invariant). Parse RTSP/1.0 manually: read until \\\\r\\\\n\\\\r\\\\n, split request line + headers, read Content-length body for ANNOUNCE. There is no need for a crate; a hand-rolled parser mirroring Sunshine's parseRtspMessage is simplest and avoids pulling RTSP libs that assume standard semantics (the streamid= targets and DEADBEEFCAFE session break them). For SDP build/parse, just format!/split on lines — it is line-oriented a=key:value. For the /launch HTTPS endpoint, reuse the existing crates/punktfunk-host/src/web.rs seam; a small hyper or tiny_http + rustls TLS server (control plane only, async OK here since it is not the hot path — matches the 'quic feature gated' precedent). For encrypted-RTSP framing use the aes-gcm crate already in punktfunk-core. Suggested new types: an RtspServer in punktfunk-host that produces a punktfunk_core::Config from the ANNOUNCE map. Hex decode rikey with the `hex` crate (or from_str_radix). Big-endian rikeyid → IV with u32::to_be_bytes.",
|
||||
"reuse_from_punktfunk": "REUSE: punktfunk-core/src/crypto.rs SessionCrypto already implements AES-128-GCM with per-direction salting and seq-as-AAD — but NOTE it is NOT byte-compatible with GameStream's nonce layout. punktfunk uses nonce = salt(4) || seq(8, BE) with a direction bit folded into salt[0], whereas GameStream uses iv = seq(LE or BE per stream) with literal direction/stream marker bytes at [10]/[11] ('V', 'H'/'C'+'C', 'C'/'H'+'R'). To talk to stock Moonlight you must add a GameStream-exact nonce mode (new constructor or a feature) rather than reuse the existing salt scheme verbatim. The Aes128Gcm cipher init and seal/open plumbing are reusable. REUSE: punktfunk-core Config/FecConfig — fec_percent maps to GameStream's repairPercent and the recovery_for() ceil(k*pct/100) already matches GameStream's FEC math; map ANNOUNCE packetSize→shard_payload, maximumBitrateKbps→bitrate, fec.minRequiredFecPackets→minRequiredFecPackets. FecScheme::Gf8 is the GameStream-compatible field. BUILD NEW: the entire RTSP/SDP/launch negotiation layer (punktfunk's internal 40-byte packet format and Config are not wire-exact to RTP/RTSP); the RTSP server, SDP describe/announce codec, the /launch query parser that produces gcm_key+iv from rikey/rikeyid, and the GameStream RTP video/audio packetization + RTPFEC are all new (separate areas). The Session in punktfunk-core can consume the negotiated Config but its on-wire packet header must be swapped for GameStream RTP for Moonlight compat.",
|
||||
"gotchas": [
|
||||
"lumen-core's AES-GCM nonce layout is NOT GameStream-compatible (salt+BE-seq vs literal 'V'/'C'/'R' marker bytes + LE/BE seq). A stock Moonlight will fail auth unless you implement the exact per-stream IV construction. This is the single biggest bridging hazard.",
|
||||
"punktfunk-core's AES-GCM nonce layout is NOT GameStream-compatible (salt+BE-seq vs literal 'V'/'C'/'R' marker bytes + LE/BE seq). A stock Moonlight will fail auth unless you implement the exact per-stream IV construction. This is the single biggest bridging hazard.",
|
||||
"rikeyid is parsed as a SIGNED int then cast to big-endian uint32 in Sunshine (util::from_view → int → endian::big<uint32_t>). Negative rikeyid values wrap; match the signed-int→BE-u32 path exactly.",
|
||||
"The IV from /launch (BE32(rikeyid)||zeros, 16 bytes) is the *base*; the actual per-packet 12-byte GCM nonce is rebuilt per stream with seq + marker bytes — do not just use the 16-byte launch IV directly.",
|
||||
"Session header value is the literal 'DEADBEEFCAFE;timeout = 90' WITH spaces around '='. Moonlight is lenient but match it.",
|
||||
@@ -179,7 +179,7 @@
|
||||
"moonlight-common-c src/Limelight-internal.h — SS_ENC_CONTROL_V2 0x01, SS_ENC_VIDEO 0x02, SS_ENC_AUDIO 0x04; RtspPortNumber/Control/Audio/VideoPortNumber externs",
|
||||
"moonlight-common-c src/Connection.c — port number init (set from RTSP SETUP), resolveHostName base 47984",
|
||||
"DeepWiki LizardByte/Sunshine port management — base 47989 with offsets: video 47998 (+9), control 47999 (+10), audio 48000 (+11), RTSP 48010 (+21) via net::map_port (1024-65535 validated)",
|
||||
"lumen-core src/crypto.rs and src/config.rs (read directly) — existing AES-128-GCM SessionCrypto (salt+BE-seq nonce, NOT GameStream-exact) and Config/FecConfig for reuse assessment"
|
||||
"punktfunk-core src/crypto.rs and src/config.rs (read directly) — existing AES-128-GCM SessionCrypto (salt+BE-seq nonce, NOT GameStream-exact) and Config/FecConfig for reuse assessment"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -245,15 +245,15 @@
|
||||
"When a block has >= dataShards shards (data+parity), client runs nanors reed_solomon_decode(rs, packets, marks, totalPackets, blocksize) to recover any missing data shards (missing slots zero-filled, marks[]=1 for missing). Recovered data shards get synthetic RTP headers (seq/header/timestamp/ssrc copied from a present packet).",
|
||||
"Client advances through blocks (currentBlock 0..lastBlock); once all blocks' data shards are present/recovered it strips the 32-byte video_packet_raw_t header off each data shard, concatenates payloads in sequence order, parses the 8-byte short frame header, and hands the reassembled access unit to the depacketizer/decoder."
|
||||
],
|
||||
"crypto": "\"Cipher: AES-128-GCM (EVP_aes_128_gcm). Key: the 16-byte RIKEY — Sunshine `launch_session.gcm_key`, Moonlight `StreamConfig.remoteInputAesKey[16]` — established during RTSP/pairing; the SAME key is used for the control stream and (if enabled) audio. There is no separate video key and no key derivation: the raw 16-byte RIKEY is used directly. IV/nonce: 12 bytes, constructed deterministically (NIST SP 800-38D 8.2.1): iv[0..8] = a 64-bit per-session counter (session->video.gcm_iv_counter, starts at 0) copied in NATIVE byte order (little-endian on x86), iv[8..11] = 0 except iv[11] = 'V' (0x56, the video-stream fixed field). The counter increments once per shard. The full 12-byte IV is transmitted in the ENC_VIDEO_HEADER so the client uses it verbatim (it does not reconstruct it). Tag: 16 bytes, standard GCM tag, transmitted in ENC_VIDEO_HEADER. AAD / associated data: NONE — Sunshine's gcm_t::encrypt for video calls EVP_EncryptUpdate only with plaintext (no AAD update), and Moonlight's PltDecryptMessage passes no AAD argument. (NOTE: this differs from lumen-core/crypto.rs which uses seq-as-AAD and a per-direction salt; GameStream video does neither.) Order: FEC FIRST, THEN ENCRYPT — encryption is applied per-shard after RS parity is computed, over the entire blocksize shard buffer (RTP+NV+payload), so the client must DECRYPT each shard before it can run FEC reconstruction. Encrypted plaintext length == blocksize; ciphertext length == blocksize (GCM is a stream cipher, no expansion); on the wire the packet grows only by the 32-byte prefix.\"",
|
||||
"rust_options": "\"FEC math: lumen already has Gf8Coder over `reed-solomon-erasure` (galois_8). CRITICAL RISK: this is NOT guaranteed byte-compatible with nanors. Both Sunshine and Moonlight use nanors (Sunshine's rswrapper.h is 'a drop-in replacement for nanors rs.h', DATA_SHARDS_MAX=255), which uses a specific GF(2^8) field (primitive poly 0x11d) and a Vandermonde-derived generator matrix with a particular systematic encoding. `reed-solomon-erasure` uses Cauchy matrices by default and may produce DIFFERENT parity bytes — meaning Moonlight would FAIL to recover frames where any data shard is lost. RECOMMENDED: vendor/FFI the actual nanors C library (it is tiny, MIT, header+rs.c+oblas) and call reed_solomon_new/encode through a thin Rust FFI, OR port nanors' matrix construction exactly into a new gf8 backend. Do NOT assume reed-solomon-erasure interop without a byte-for-byte test against nanors output. (For lumen-to-lumen P2 traffic, keep the existing coders; for GameStream-client compat, use nanors.) Crypto: use `aes-gcm` (already a dep) but build a NEW path that (a) takes the raw RIKEY as the key, (b) builds the 12-byte IV as counter_le[8]||0||0||0||'V', (c) uses NO AAD, (d) does NOT use the per-direction salt logic in SessionCrypto. The cleanest approach is a small standalone `Aes128Gcm` call rather than reusing SessionCrypto (whose nonce/AAD scheme is incompatible). Byte layout: define `#[repr(C, packed)]` structs RtpPacket, NvVideoPacket, EncVideoHeader and use explicit `to_be_bytes`/`to_le_bytes` per field (RTP=BE, NV=LE) — do not rely on struct memory layout for endianness.\"",
|
||||
"reuse_from_lumen": "\"REUSE the GF(2^8) concept/structure but NOT necessarily the implementation: lumen's `ErasureCoder` trait and `Gf8Coder` (reed-solomon-erasure) give the right data||parity systematic layout and the 255-shard ceiling, but parity bytes likely won't match nanors — so for client-facing GameStream compat add a `nanors`-backed coder (FFI or exact port) behind the same trait. The trait's reconstruct(data_count, recovery_count, received: indices 0..K originals, K..K+M recovery) maps cleanly onto Moonlight's layout (data shards at RS index 0..dataShards, parity after) so the adapter just needs to map RTP-seq→shard-index. REUSE aes-gcm crate but NOT SessionCrypto (its salt+seq-AAD scheme is wire-incompatible with GameStream video which uses no AAD and a counter||'V' IV). REUSE lumen's UDP transport for sending datagrams. Do NOT reuse lumen's internal 40-byte packet format — GameStream needs the exact 12+4+16 header + optional 32-byte enc prefix. NEW work: a `gamestream` wire module in lumen-host (NOT lumen-core, to keep lumen-core's clean internal protocol) that (1) builds RTP_PACKET/NV_VIDEO_PACKET/ENC_VIDEO_HEADER bytes, (2) implements the frame→FEC-block split (max 4 blocks, the 255/(1+F) shard math), (3) drives a nanors coder, (4) does the per-shard counter-IV AES-128-GCM-no-AAD encrypt, (5) paces/batches sends. Best location: a new file like crates/lumen-host/src/gamestream/video.rs (host side) with the nanors FFI either in lumen-host or a small new crate; lumen-core stays the lumen-native protocol and only its aes-gcm + the gf8 *math* are conceptually shared. The 'adapter' lives at the lumen-host pipeline seam: take the NVENC access unit + frame metadata from encode.rs and emit GameStream datagrams instead of (or alongside) lumen-native packets.\"",
|
||||
"crypto": "\"Cipher: AES-128-GCM (EVP_aes_128_gcm). Key: the 16-byte RIKEY — Sunshine `launch_session.gcm_key`, Moonlight `StreamConfig.remoteInputAesKey[16]` — established during RTSP/pairing; the SAME key is used for the control stream and (if enabled) audio. There is no separate video key and no key derivation: the raw 16-byte RIKEY is used directly. IV/nonce: 12 bytes, constructed deterministically (NIST SP 800-38D 8.2.1): iv[0..8] = a 64-bit per-session counter (session->video.gcm_iv_counter, starts at 0) copied in NATIVE byte order (little-endian on x86), iv[8..11] = 0 except iv[11] = 'V' (0x56, the video-stream fixed field). The counter increments once per shard. The full 12-byte IV is transmitted in the ENC_VIDEO_HEADER so the client uses it verbatim (it does not reconstruct it). Tag: 16 bytes, standard GCM tag, transmitted in ENC_VIDEO_HEADER. AAD / associated data: NONE — Sunshine's gcm_t::encrypt for video calls EVP_EncryptUpdate only with plaintext (no AAD update), and Moonlight's PltDecryptMessage passes no AAD argument. (NOTE: this differs from punktfunk-core/crypto.rs which uses seq-as-AAD and a per-direction salt; GameStream video does neither.) Order: FEC FIRST, THEN ENCRYPT — encryption is applied per-shard after RS parity is computed, over the entire blocksize shard buffer (RTP+NV+payload), so the client must DECRYPT each shard before it can run FEC reconstruction. Encrypted plaintext length == blocksize; ciphertext length == blocksize (GCM is a stream cipher, no expansion); on the wire the packet grows only by the 32-byte prefix.\"",
|
||||
"rust_options": "\"FEC math: punktfunk already has Gf8Coder over `reed-solomon-erasure` (galois_8). CRITICAL RISK: this is NOT guaranteed byte-compatible with nanors. Both Sunshine and Moonlight use nanors (Sunshine's rswrapper.h is 'a drop-in replacement for nanors rs.h', DATA_SHARDS_MAX=255), which uses a specific GF(2^8) field (primitive poly 0x11d) and a Vandermonde-derived generator matrix with a particular systematic encoding. `reed-solomon-erasure` uses Cauchy matrices by default and may produce DIFFERENT parity bytes — meaning Moonlight would FAIL to recover frames where any data shard is lost. RECOMMENDED: vendor/FFI the actual nanors C library (it is tiny, MIT, header+rs.c+oblas) and call reed_solomon_new/encode through a thin Rust FFI, OR port nanors' matrix construction exactly into a new gf8 backend. Do NOT assume reed-solomon-erasure interop without a byte-for-byte test against nanors output. (For punktfunk-to-punktfunk P2 traffic, keep the existing coders; for GameStream-client compat, use nanors.) Crypto: use `aes-gcm` (already a dep) but build a NEW path that (a) takes the raw RIKEY as the key, (b) builds the 12-byte IV as counter_le[8]||0||0||0||'V', (c) uses NO AAD, (d) does NOT use the per-direction salt logic in SessionCrypto. The cleanest approach is a small standalone `Aes128Gcm` call rather than reusing SessionCrypto (whose nonce/AAD scheme is incompatible). Byte layout: define `#[repr(C, packed)]` structs RtpPacket, NvVideoPacket, EncVideoHeader and use explicit `to_be_bytes`/`to_le_bytes` per field (RTP=BE, NV=LE) — do not rely on struct memory layout for endianness.\"",
|
||||
"reuse_from_punktfunk": "\"REUSE the GF(2^8) concept/structure but NOT necessarily the implementation: punktfunk's `ErasureCoder` trait and `Gf8Coder` (reed-solomon-erasure) give the right data||parity systematic layout and the 255-shard ceiling, but parity bytes likely won't match nanors — so for client-facing GameStream compat add a `nanors`-backed coder (FFI or exact port) behind the same trait. The trait's reconstruct(data_count, recovery_count, received: indices 0..K originals, K..K+M recovery) maps cleanly onto Moonlight's layout (data shards at RS index 0..dataShards, parity after) so the adapter just needs to map RTP-seq→shard-index. REUSE aes-gcm crate but NOT SessionCrypto (its salt+seq-AAD scheme is wire-incompatible with GameStream video which uses no AAD and a counter||'V' IV). REUSE punktfunk's UDP transport for sending datagrams. Do NOT reuse punktfunk's internal 40-byte packet format — GameStream needs the exact 12+4+16 header + optional 32-byte enc prefix. NEW work: a `gamestream` wire module in punktfunk-host (NOT punktfunk-core, to keep punktfunk-core's clean internal protocol) that (1) builds RTP_PACKET/NV_VIDEO_PACKET/ENC_VIDEO_HEADER bytes, (2) implements the frame→FEC-block split (max 4 blocks, the 255/(1+F) shard math), (3) drives a nanors coder, (4) does the per-shard counter-IV AES-128-GCM-no-AAD encrypt, (5) paces/batches sends. Best location: a new file like crates/punktfunk-host/src/gamestream/video.rs (host side) with the nanors FFI either in punktfunk-host or a small new crate; punktfunk-core stays the punktfunk-native protocol and only its aes-gcm + the gf8 *math* are conceptually shared. The 'adapter' lives at the punktfunk-host pipeline seam: take the NVENC access unit + frame metadata from encode.rs and emit GameStream datagrams instead of (or alongside) punktfunk-native packets.\"",
|
||||
"gotchas": [
|
||||
"ENDIANNESS SPLIT: RTP_PACKET fields are BIG-endian, NV_VIDEO_PACKET fields are LITTLE-endian, within the SAME packet. Easy to get wrong. RTP: BE16/BE32; NV streamPacketIndex/frameIndex/fecInfo: LE32.",
|
||||
"ENCRYPT-AFTER-FEC, not before: the GCM-encrypted region is the WHOLE shard (RTP+NV+payload) and the client must decrypt each shard before FEC. The 32-byte ENC_VIDEO_HEADER is a wire PREFIX outside the FEC blocksize, not part of the protected data. If you FEC after encrypt or include the prefix in the FEC math, recovery breaks.",
|
||||
"NO AAD on video GCM — unlike lumen-core's SessionCrypto which authenticates the sequence number as AAD. Using SessionCrypto verbatim will fail Moonlight's tag check.",
|
||||
"NO AAD on video GCM — unlike punktfunk-core's SessionCrypto which authenticates the sequence number as AAD. Using SessionCrypto verbatim will fail Moonlight's tag check.",
|
||||
"IV counter byte order: Sunshine copies the 64-bit counter with std::copy_n in NATIVE order (little-endian on the x86 build), so iv[0..8] is the counter LE; iv[11]='V'(0x56), iv[8..11]=0. The client uses the transmitted iv verbatim, so as long as you SEND the iv you used, internal byte order is self-consistent — but match LE to mirror Sunshine exactly and to keep nonces unique.",
|
||||
"FEC parity matrix must match nanors EXACTLY. reed-solomon-erasure (lumen's current backend) is likely NOT byte-compatible (Cauchy vs nanors Vandermonde). Without a byte-for-byte match, Moonlight silently fails to recover any frame with a lost data shard. Validate against real nanors output or FFI nanors.",
|
||||
"FEC parity matrix must match nanors EXACTLY. reed-solomon-erasure (punktfunk's current backend) is likely NOT byte-compatible (Cauchy vs nanors Vandermonde). Without a byte-for-byte match, Moonlight silently fails to recover any frame with a lost data shard. Validate against real nanors output or FFI nanors.",
|
||||
"streamPacketIndex is (lowseq+x)<<8 with low byte zero; client does >>=8 then &0xFFFFFF → a 24-bit stream-wide packet index, distinct from the 16-bit RTP sequenceNumber. Both must be set consistently or the depacketizer's continuity check (FLAG_SOF / streamPacketIndex == lastPacketInStream+1) rejects the frame.",
|
||||
"Max 4 FEC blocks per frame (2-bit fields). Max 1024 packets per block (10-bit fecIndex). Max 255 shards/block (GF(2^8)). data_shards per block = 255*100/(100+fecPercentage). Exceeding these → FEC disabled or unrecoverable frame.",
|
||||
"Last data shard is zero-padded to blocksize before RS encode; lastPayloadLen in the short frame header tells the client the real length of the final packet's payload (needed for AV1). Padding must be zeros so RS math and the client's memset-padding agree.",
|
||||
@@ -274,7 +274,7 @@
|
||||
"moonlight-common-c src/Limelight.h — ENCFLG_VIDEO=0x2, ENCFLG_AUDIO=0x1, remoteInputAesKey[16] (the RIKEY).",
|
||||
"moonlight-common-c src/SdpGenerator.c — SS_ENC_VIDEO negotiation; when enabled StreamConfig.packetSize -= sizeof(ENC_VIDEO_HEADER) (32).",
|
||||
"moonlight-common-c .gitmodules + repo tree — nanors at /nanors (rs.c, rs.h, deps/obl GF(2^8) tables), confirming both host(Sunshine via rswrapper) and client use nanors.",
|
||||
"lumen local: crates/lumen-core/src/crypto.rs (SessionCrypto: salt+seq-AAD scheme — incompatible with GameStream video), crates/lumen-core/src/fec/{mod.rs,gf8.rs} (ErasureCoder trait + reed-solomon-erasure galois_8 — needs nanors compat verification)."
|
||||
"punktfunk local: crates/punktfunk-core/src/crypto.rs (SessionCrypto: salt+seq-AAD scheme — incompatible with GameStream video), crates/punktfunk-core/src/fec/{mod.rs,gf8.rs} (ErasureCoder trait + reed-solomon-erasure galois_8 — needs nanors compat verification)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -320,10 +320,10 @@
|
||||
"Client decrypts recovered/received data shards (AES-128-CBC, strip PKCS7), reorders by sequence, and feeds frames to opus_multistream_decoder_create/decode_float."
|
||||
],
|
||||
"crypto": "Cipher: AES-128-CBC (NOT GCM — video & control use GCM, audio uses CBC). Key: 16 bytes = the session AES key (Sunshine launch_session.gcm_key / moonlight StreamConfig.remoteInputAesKey), same key family used for the input/RI channel. IV: 16 bytes, per-packet — first 4 bytes = big-endian uint32 of (avRiKeyId + sequenceNumber), remaining 12 bytes = 0x00. avRiKeyId is a per-session 32-bit value from the RTSP/launch negotiation (session->audio.avRiKeyId). Padding: PKCS7 to AES block (16-byte) multiple; round_to_pkcs7_padded, max_block_size = round_to_pkcs7_padded(2048). No GMAC/auth tag is appended to audio packets. Encryption is gated by the SS_ENC_AUDIO encryption flag (config.encryptionFlagsEnabled & SS_ENC_AUDIO) — if disabled, raw Opus frame is sent. FEC parity is computed over the post-encryption, post-padding shard bytes so recovery yields ciphertext that is then decrypted.",
|
||||
"rust_options": "Opus encode: use the `audiopus` crate or `opus` (libopus bindings) — both expose multistream via opus_multistream_encoder; if missing, FFI to libopus opus_multistream_encoder_create/opus_multistream_encode_float directly. Configure sampleRate=48000, the streams/coupledStreams and mapping per the negotiated AUDIO_CONFIGURATION; frame size = 48*packetDuration samples/ch. AES-128-CBC: use the `aes` + `cbc` crates (cbc::Encryptor<aes::Aes128>) with manual PKCS7 (`block-padding`/`Pkcs7`) — build the 16-byte IV as BE32(avRiKeyId+seq) || [0u8;12]. Reed-Solomon: do NOT use a generic RS matrix; the wire requires Nvidia's specific parity matrix. The `reed-solomon-erasure` crate computes its own (Vandermonde/Cauchy) matrix that will NOT match — either (a) port moonlight's approach: take the rs lib's encode path but inject the OpenFEC parity matrix bytes {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} for the 4+2 case, or (b) hand-roll a tiny GF(2^8) 4-data/2-parity encoder/decoder using that exact 2x4 parity matrix (8 bytes = 2 parity rows × 4 data cols). lumen's existing `reed-solomon` GF(2^8) code can be reused ONLY if it lets you supply a custom generator/parity matrix; otherwise add a dedicated audio-FEC path. Big-endian field writes: use `byteorder`/`to_be_bytes`. UDP: std::net::UdpSocket on a native thread (no async, matching lumen's hot-path rule).",
|
||||
"reuse_from_lumen": "REUSE: lumen-core's AES-128 primitive (crypto.rs) underlies CBC, but the MODE differs — lumen uses AES-128-GCM with per-direction nonce salts + seq-as-AAD; GameStream audio needs AES-128-CBC with the BE32(avRiKeyId+seq) IV and PKCS7, no AAD/tag. So add a CBC path; do not reuse the GCM nonce/AAD scheme for audio. lumen's GF(2^8) Reed-Solomon (reed-solomon crate) is the right field but the MATRIX is wrong for the wire — must supply Nvidia's hardcoded {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} parity matrix or hand-roll the 4+2 encoder; lumen's internal 40-byte packet format and its FEC block sizing are NOT wire-compatible and cannot be reused for the on-wire audio packets. lumen's UDP transport + native-thread pacing model is reusable as plumbing. NEW: 12-byte RTP header serializer, 12-byte AUDIO_FEC_HEADER, the fixed 4+2 audio FEC block state machine (block starts at seq%4==0, encode at (seq+1)%4==0), the Opus multistream encoder integration, the 500 ms ping listener, and the AES-CBC+PKCS7 audio path. These are GameStream-specific and don't exist in lumen-core.",
|
||||
"rust_options": "Opus encode: use the `audiopus` crate or `opus` (libopus bindings) — both expose multistream via opus_multistream_encoder; if missing, FFI to libopus opus_multistream_encoder_create/opus_multistream_encode_float directly. Configure sampleRate=48000, the streams/coupledStreams and mapping per the negotiated AUDIO_CONFIGURATION; frame size = 48*packetDuration samples/ch. AES-128-CBC: use the `aes` + `cbc` crates (cbc::Encryptor<aes::Aes128>) with manual PKCS7 (`block-padding`/`Pkcs7`) — build the 16-byte IV as BE32(avRiKeyId+seq) || [0u8;12]. Reed-Solomon: do NOT use a generic RS matrix; the wire requires Nvidia's specific parity matrix. The `reed-solomon-erasure` crate computes its own (Vandermonde/Cauchy) matrix that will NOT match — either (a) port moonlight's approach: take the rs lib's encode path but inject the OpenFEC parity matrix bytes {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} for the 4+2 case, or (b) hand-roll a tiny GF(2^8) 4-data/2-parity encoder/decoder using that exact 2x4 parity matrix (8 bytes = 2 parity rows × 4 data cols). punktfunk's existing `reed-solomon` GF(2^8) code can be reused ONLY if it lets you supply a custom generator/parity matrix; otherwise add a dedicated audio-FEC path. Big-endian field writes: use `byteorder`/`to_be_bytes`. UDP: std::net::UdpSocket on a native thread (no async, matching punktfunk's hot-path rule).",
|
||||
"reuse_from_punktfunk": "REUSE: punktfunk-core's AES-128 primitive (crypto.rs) underlies CBC, but the MODE differs — punktfunk uses AES-128-GCM with per-direction nonce salts + seq-as-AAD; GameStream audio needs AES-128-CBC with the BE32(avRiKeyId+seq) IV and PKCS7, no AAD/tag. So add a CBC path; do not reuse the GCM nonce/AAD scheme for audio. punktfunk's GF(2^8) Reed-Solomon (reed-solomon crate) is the right field but the MATRIX is wrong for the wire — must supply Nvidia's hardcoded {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c} parity matrix or hand-roll the 4+2 encoder; punktfunk's internal 40-byte packet format and its FEC block sizing are NOT wire-compatible and cannot be reused for the on-wire audio packets. punktfunk's UDP transport + native-thread pacing model is reusable as plumbing. NEW: 12-byte RTP header serializer, 12-byte AUDIO_FEC_HEADER, the fixed 4+2 audio FEC block state machine (block starts at seq%4==0, encode at (seq+1)%4==0), the Opus multistream encoder integration, the 500 ms ping listener, and the AES-CBC+PKCS7 audio path. These are GameStream-specific and don't exist in punktfunk-core.",
|
||||
"gotchas": [
|
||||
"AES-CBC, not GCM. Audio is the one stream using CBC; reusing lumen's GCM code verbatim will break interop. No auth tag is on the wire for audio.",
|
||||
"AES-CBC, not GCM. Audio is the one stream using CBC; reusing punktfunk's GCM code verbatim will break interop. No auth tag is on the wire for audio.",
|
||||
"IV is only 4 meaningful bytes: BE32(avRiKeyId + sequenceNumber) then 12 zero bytes. The addition wraps as uint32. avRiKeyId is per-session from RTSP launch.",
|
||||
"The Reed-Solomon parity matrix MUST be Nvidia's hardcoded one. moonlight explicitly notes 'the RS parity matrix computed by our RS implementation doesn't match the one Nvidia uses' and overrides it with the 8 OpenFEC bytes {0x77,0x40,0x38,0x0e,0xc7,0xa7,0x0d,0x6c}. A stock reed-solomon-erasure encoder will produce parity the client cannot decode.",
|
||||
"FEC is a FIXED 4+2 block (RTPA_DATA_SHARDS=4, RTPA_FEC_SHARDS=2), unlike video which uses dynamic/large blocks and multi-FEC. Block boundaries must align to seq%4==0 or the client's queue logic rejects them.",
|
||||
@@ -352,8 +352,8 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"area": "GameStream wire-format gap analysis + architecture recommendation for lumen-host (P1 / M2)",
|
||||
"summary": "lumen-core today speaks an INTERNAL protocol that is structurally similar to GameStream but byte-incompatible on every wire surface, so a stock Moonlight client cannot connect to it as-is. Differences: (1) lumen prefixes each shard with a 40-byte little-endian `PacketHeader` and no RTP layer; GameStream uses a 12-byte big-endian RTP header + 4 reserved bytes + a 16-byte `NV_VIDEO_PACKET` (28 bytes total) carrying frameIndex/streamPacketIndex/flags and the FEC params bit-packed into a single `fecInfo` u32 and `multiFecBlocks` u8. (2) lumen's RS-FEC interleaves data+recovery shards within one block keyed by `shard_index`; GameStream packs ALL data shards first then ALL parity shards across a contiguous RTP sequence range, derives (data,parity,fecIndex,pct) from `fecInfo`, splits a frame into up to 4 FEC blocks via `multiFecBlocks`, and the data shards must be the literal RTP-framed bytes of the H.264/HEVC NAL slices (the depacketizer concatenates payloads to rebuild Annex-B). (3) lumen seals the whole 40-byte+payload packet under AES-128-GCM with an 8-byte seq prefix and seq-as-AAD; GameStream encrypts only the post-RTP payload, prefixing a `video_packet_enc_prefix_t {iv[12]; u32 frameNumber; u8 tag[16]}` where the IV is an 8-byte little-endian per-stream counter with iv[11]='V'. The RS math itself is identical (ceil(k*pct/100), GF(2^8), <=255 shards) so lumen's `reed-solomon` GF(2^8) coder CAN produce Moonlight-recoverable parity, but ONLY if lumen abandons its own shard layout and emits shards in GameStream's data-then-parity contiguous order with GameStream's exact shard size (packetSize + 4 reserved + RTP). Beyond video, GameStream needs an entire control plane lumen has not started: HTTPS:47984/HTTP:47989 nvhttp pairing (PIN->AES-128 via SHA-256(salt||pin)[..16], ECB challenge exchange, RSA-signed client cert), an RTSP:48010 handshake (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY) carrying SDP `x-nv-*` params, an ENet control stream (UDP 48000) with its own AES-128-GCM framing and opcodes (request-IDR, loss-stats, ping, HDR, termination, rumble), an AES-CBC audio stream (UDP 47999), and mDNS `_nvstream._tcp` advertisement. Recommendation: put the GameStream video/FEC/crypto wire codec as a P1 \"wire mode\" INSIDE lumen-core (the invariant says protocol logic lives in the core), but keep the stateful control plane (nvhttp/RTSP/ENet/pairing/mDNS) in lumen-host as a tokio control-plane adapter that calls into core codec functions, because that machinery is I/O-bound, async, and not part of the hot path.",
|
||||
"area": "GameStream wire-format gap analysis + architecture recommendation for punktfunk-host (P1 / M2)",
|
||||
"summary": "punktfunk-core today speaks an INTERNAL protocol that is structurally similar to GameStream but byte-incompatible on every wire surface, so a stock Moonlight client cannot connect to it as-is. Differences: (1) punktfunk prefixes each shard with a 40-byte little-endian `PacketHeader` and no RTP layer; GameStream uses a 12-byte big-endian RTP header + 4 reserved bytes + a 16-byte `NV_VIDEO_PACKET` (28 bytes total) carrying frameIndex/streamPacketIndex/flags and the FEC params bit-packed into a single `fecInfo` u32 and `multiFecBlocks` u8. (2) punktfunk's RS-FEC interleaves data+recovery shards within one block keyed by `shard_index`; GameStream packs ALL data shards first then ALL parity shards across a contiguous RTP sequence range, derives (data,parity,fecIndex,pct) from `fecInfo`, splits a frame into up to 4 FEC blocks via `multiFecBlocks`, and the data shards must be the literal RTP-framed bytes of the H.264/HEVC NAL slices (the depacketizer concatenates payloads to rebuild Annex-B). (3) punktfunk seals the whole 40-byte+payload packet under AES-128-GCM with an 8-byte seq prefix and seq-as-AAD; GameStream encrypts only the post-RTP payload, prefixing a `video_packet_enc_prefix_t {iv[12]; u32 frameNumber; u8 tag[16]}` where the IV is an 8-byte little-endian per-stream counter with iv[11]='V'. The RS math itself is identical (ceil(k*pct/100), GF(2^8), <=255 shards) so punktfunk's `reed-solomon` GF(2^8) coder CAN produce Moonlight-recoverable parity, but ONLY if punktfunk abandons its own shard layout and emits shards in GameStream's data-then-parity contiguous order with GameStream's exact shard size (packetSize + 4 reserved + RTP). Beyond video, GameStream needs an entire control plane punktfunk has not started: HTTPS:47984/HTTP:47989 nvhttp pairing (PIN->AES-128 via SHA-256(salt||pin)[..16], ECB challenge exchange, RSA-signed client cert), an RTSP:48010 handshake (OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY) carrying SDP `x-nv-*` params, an ENet control stream (UDP 48000) with its own AES-128-GCM framing and opcodes (request-IDR, loss-stats, ping, HDR, termination, rumble), an AES-CBC audio stream (UDP 47999), and mDNS `_nvstream._tcp` advertisement. Recommendation: put the GameStream video/FEC/crypto wire codec as a P1 \"wire mode\" INSIDE punktfunk-core (the invariant says protocol logic lives in the core), but keep the stateful control plane (nvhttp/RTSP/ENet/pairing/mDNS) in punktfunk-host as a tokio control-plane adapter that calls into core codec functions, because that machinery is I/O-bound, async, and not part of the hot path.",
|
||||
"ports": [
|
||||
"TCP 47984 — HTTPS nvhttp (paired control: /serverinfo, /pair, /applist, /launch, /resume, /cancel). Client-cert pinned to the paired client.",
|
||||
"TCP 47989 — HTTP nvhttp (unpaired: /serverinfo unauthenticated, /pair PIN flow).",
|
||||
@@ -362,38 +362,38 @@
|
||||
"UDP 47999 — Audio RTP stream (Opus, AES-CBC, RS-FEC). ML_PORT_INDEX_UDP_47999=9.",
|
||||
"UDP 48000 — ENet control stream (reliable, AES-128-GCM, opcodes). ML_PORT_INDEX_UDP_48000=10.",
|
||||
"UDP/mDNS 5353 — _nvstream._tcp.local advertisement so Moonlight auto-discovers the host.",
|
||||
"Note: Moonlight derives all of these by offset from the HTTP base port (default 47989); changing the base shifts the whole set. lumen-host must advertise the actual HttpsPort/ExternalPort in serverinfo XML."
|
||||
"Note: Moonlight derives all of these by offset from the HTTP base port (default 47989); changing the base shifts the whole set. punktfunk-host must advertise the actual HttpsPort/ExternalPort in serverinfo XML."
|
||||
],
|
||||
"wire_formats": [
|
||||
{
|
||||
"name": "DELTA: video packet header (lumen vs GameStream)",
|
||||
"layout": "lumen PacketHeader = 40 bytes, little-endian, repr(C): pts_ns u64, frame_index u32, stream_seq u32, frame_bytes u32, user_flags u32, block_index u16, block_count u16, data_shards u16, recovery_shards u16, shard_index u16, shard_bytes u16, magic u8(0xC9), version u8, fec_scheme u8, flags u8. || GameStream on-wire = RTP_PACKET(12, big-endian: u8 header, u8 packetType, u16 sequenceNumber, u32 timestamp, u32 ssrc) + char reserved[4] + NV_VIDEO_PACKET(16, little-endian: u32 streamPacketIndex@0, u32 frameIndex@4, u8 flags@8, u8 extraFlags@9 (NV_VIDEO_PACKET_EXTRA_FLAG_LTR_FRAME=0x1), u8 multiFecFlags@10, u8 multiFecBlocks@11, u32 fecInfo@12) = 28 bytes before payload.",
|
||||
"notes": "DELTA: drop pts_ns/frame_bytes/shard_bytes from the wire (GameStream carries none of these per-packet); add the RTP header + reserved[4]; replace explicit u16 FEC fields with the bit-packed fecInfo+multiFecBlocks. flags map 1:1 (FLAG_CONTAINS_PIC_DATA=0x1==lumen FLAG_PIC, FLAG_EOF=0x2, FLAG_SOF=0x4). frameIndex == lumen frame_index; streamPacketIndex == per-stream packet counter (NOT lumen stream_seq which is per-AU)."
|
||||
"name": "DELTA: video packet header (punktfunk vs GameStream)",
|
||||
"layout": "punktfunk PacketHeader = 40 bytes, little-endian, repr(C): pts_ns u64, frame_index u32, stream_seq u32, frame_bytes u32, user_flags u32, block_index u16, block_count u16, data_shards u16, recovery_shards u16, shard_index u16, shard_bytes u16, magic u8(0xC9), version u8, fec_scheme u8, flags u8. || GameStream on-wire = RTP_PACKET(12, big-endian: u8 header, u8 packetType, u16 sequenceNumber, u32 timestamp, u32 ssrc) + char reserved[4] + NV_VIDEO_PACKET(16, little-endian: u32 streamPacketIndex@0, u32 frameIndex@4, u8 flags@8, u8 extraFlags@9 (NV_VIDEO_PACKET_EXTRA_FLAG_LTR_FRAME=0x1), u8 multiFecFlags@10, u8 multiFecBlocks@11, u32 fecInfo@12) = 28 bytes before payload.",
|
||||
"notes": "DELTA: drop pts_ns/frame_bytes/shard_bytes from the wire (GameStream carries none of these per-packet); add the RTP header + reserved[4]; replace explicit u16 FEC fields with the bit-packed fecInfo+multiFecBlocks. flags map 1:1 (FLAG_CONTAINS_PIC_DATA=0x1==punktfunk FLAG_PIC, FLAG_EOF=0x2, FLAG_SOF=0x4). frameIndex == punktfunk frame_index; streamPacketIndex == per-stream packet counter (NOT punktfunk stream_seq which is per-AU)."
|
||||
},
|
||||
{
|
||||
"name": "fecInfo bit-packing (GameStream, exact)",
|
||||
"layout": "fecInfo (u32, little-endian field) = (dataShards << 22) | (fecIndex << 12) | (fecPercentage << 4). Decode masks (moonlight-common-c): dataShards=(fecInfo & 0xFFC00000)>>22 (bits 22-31, 10 bits, <=1023 but RS caps at 255); fecIndex=(fecInfo & 0x3FF000)>>12 (bits 12-21, the shard's index within its block); fecPercentage=(fecInfo & 0xFF0)>>4 (bits 4-11). parityShards = (dataShards*fecPercentage + 99)/100 (ceiling). bits 0-3 unused.",
|
||||
"notes": "This is IDENTICAL math to lumen's FecConfig::recovery_for (ceil(k*pct/100)). lumen already computes data_shards/recovery_shards as explicit u16; the only delta is packing them into this bitfield and emitting fecIndex as the contiguous index across [0..data) then [data..data+parity)."
|
||||
"notes": "This is IDENTICAL math to punktfunk's FecConfig::recovery_for (ceil(k*pct/100)). punktfunk already computes data_shards/recovery_shards as explicit u16; the only delta is packing them into this bitfield and emitting fecIndex as the contiguous index across [0..data) then [data..data+parity)."
|
||||
},
|
||||
{
|
||||
"name": "multiFecBlocks bit-packing (GameStream, exact)",
|
||||
"layout": "multiFecBlocks (u8) = (blockIndex << 4) | ((fec_blocks_needed - 1) << 6). Decode: fecCurrentBlockNumber=(multiFecBlocks>>4)&0x3; lastBlockNumber=(multiFecBlocks>>6)&0x3. Max 4 FEC blocks per frame (2 bits each).",
|
||||
"notes": "DELTA vs lumen: lumen uses u16 block_index/block_count with no 4-block ceiling. For P1 wire mode, max_data_per_block must be chosen so a frame needs <=4 blocks AND each block <=255 total shards. lumen's p1_defaults (max_data_per_block=200, 15% FEC -> 230 total) already respects 255; just cap blocks at 4 for GameStream mode."
|
||||
"notes": "DELTA vs punktfunk: punktfunk uses u16 block_index/block_count with no 4-block ceiling. For P1 wire mode, max_data_per_block must be chosen so a frame needs <=4 blocks AND each block <=255 total shards. punktfunk's p1_defaults (max_data_per_block=200, 15% FEC -> 230 total) already respects 255; just cap blocks at 4 for GameStream mode."
|
||||
},
|
||||
{
|
||||
"name": "RS-FEC shard arrangement (the recoverability question)",
|
||||
"layout": "GameStream: within one FEC block, RTP sequence numbers are contiguous: data shards occupy [bufferLowestSequenceNumber .. bufferFirstParitySequenceNumber-1], parity shards immediately follow. totalPackets = highest-lowest+1 = dataShards+parityShards. Each shard is exactly receiveSize = packetSize + MAX_RTP_HEADER_SIZE bytes, the last data shard zero-padded to receiveSize. Decode: rs=reed_solomon_new(dataShards, parityShards); reed_solomon_decode(rs, packets[], marks[], totalPackets, receiveSize) with marks[i]=1 for missing. Data shards = the RTP-framed bytes of the video payload concatenated; depacketizer strips RTP+NV header and concatenates payloads to rebuild the Annex-B AU.",
|
||||
"notes": "lumen TODAY: shard_index addresses data [0..K) then recovery [K..K+M) within a block, reconstruct() takes received[] of length K+M with None=lost — STRUCTURALLY THE SAME ordering as GameStream's data-then-parity. VERDICT: lumen's reed-solomon GF(2^8) coder CAN produce moonlight-recoverable shards, because both use the same Vandermonde/Cauchy RS over GF(2^8) with data-first layout. BUT the byte CONTENT of each data shard must be GameStream's RTP-framed packet bytes (not lumen's 40-byte-header packets), and the shard size must be packetSize+RTP, and the parity must be computed over those exact bytes. CAVEAT (unverified at byte level): the specific RS library Moonlight uses (reed-solomon-new / Fec.c, a CM256/Plank-style Cauchy matrix) may use a different generator matrix than the Rust `reed-solomon` crate; parity bytes are only interoperable if the matrices match. This MUST be validated against real Moonlight before trusting it — if matrices differ, lumen must port/match Moonlight's Fec.c matrix exactly (this is the single highest-risk interop item)."
|
||||
"notes": "punktfunk TODAY: shard_index addresses data [0..K) then recovery [K..K+M) within a block, reconstruct() takes received[] of length K+M with None=lost — STRUCTURALLY THE SAME ordering as GameStream's data-then-parity. VERDICT: punktfunk's reed-solomon GF(2^8) coder CAN produce moonlight-recoverable shards, because both use the same Vandermonde/Cauchy RS over GF(2^8) with data-first layout. BUT the byte CONTENT of each data shard must be GameStream's RTP-framed packet bytes (not punktfunk's 40-byte-header packets), and the shard size must be packetSize+RTP, and the parity must be computed over those exact bytes. CAVEAT (unverified at byte level): the specific RS library Moonlight uses (reed-solomon-new / Fec.c, a CM256/Plank-style Cauchy matrix) may use a different generator matrix than the Rust `reed-solomon` crate; parity bytes are only interoperable if the matrices match. This MUST be validated against real Moonlight before trusting it — if matrices differ, punktfunk must port/match Moonlight's Fec.c matrix exactly (this is the single highest-risk interop item)."
|
||||
},
|
||||
{
|
||||
"name": "video AES-GCM crypto (DELTA)",
|
||||
"layout": "GameStream video_packet_enc_prefix_t = { u8 iv[12]; u32 frameNumber; u8 tag[16] } prepended to the encrypted payload. IV = 8-byte little-endian per-stream gcm_iv_counter in iv[0..8], iv[11]='V' (0x56), iv[8..11]=0; counter increments per packet. Cipher = AES-128-GCM, key = the GCM key from /launch (riKey). Only the post-RTP/post-NV payload is encrypted; RTP+NV header stay in clear. video_short_frame_header_t (8 bytes, inside the encrypted payload, first packet of frame) = { u8 headerType=0x01; le_u16 frame_processing_latency; u8 frameType (1=P,2=IDR,4=intra-refresh,5=after-ref-invalidation); le_u16 lastPayloadLen; u8 unknown[2] }.",
|
||||
"notes": "DELTA vs lumen crypto.rs: lumen seals the ENTIRE packet (header+payload) and uses a 4-byte salt + 8-byte big-endian seq nonce with seq as AAD, prefixing an 8-byte seq. GameStream encrypts only payload, uses 8-byte LE counter + 'V' marker (NO AAD), and the prefix carries iv+frameNumber+tag explicitly. lumen's per-direction salt-bit trick is a lumen invention not on the GameStream wire. For P1 wire mode the core needs a SEPARATE gcm path matching this prefix exactly."
|
||||
"notes": "DELTA vs punktfunk crypto.rs: punktfunk seals the ENTIRE packet (header+payload) and uses a 4-byte salt + 8-byte big-endian seq nonce with seq as AAD, prefixing an 8-byte seq. GameStream encrypts only payload, uses 8-byte LE counter + 'V' marker (NO AAD), and the prefix carries iv+frameNumber+tag explicitly. punktfunk's per-direction salt-bit trick is a punktfunk invention not on the GameStream wire. For P1 wire mode the core needs a SEPARATE gcm path matching this prefix exactly."
|
||||
},
|
||||
{
|
||||
"name": "ENet control crypto + opcodes (new in host)",
|
||||
"layout": "Encrypted control: NVCTL_ENCRYPTED_PACKET_HEADER { le_u16 encryptedHeaderType=0x0001; le_u16 length; u32 seq } then [16-byte AES-GCM tag][encrypted V2 header + payload]. Cipher AES-128-GCM (Sunshine SS_ENC_CONTROL_V2): 12-byte LE IV = seq in bytes 0-3, bytes 10-11='CC'. Plain header V2 = { u16 type; u16 payloadLength }. Opcodes (Gen7 plain): 0x0305 Start A, 0x0307 Start B, 0x0301 invalidate-ref-frames, 0x0201 loss-stats, 0x0206 input, 0x010b rumble, 0x0100 termination, 0x010e HDR, 0x0302 request-IDR (encrypted gen). Periodic ping {le_u16 len=4; le_u32 ts}.",
|
||||
"notes": "Entirely absent from lumen. Belongs in lumen-host (ENet via a Rust ENet crate); the AES-128-GCM seal/open of the control payload can reuse a core crypto primitive but the framing is host-side."
|
||||
"notes": "Entirely absent from punktfunk. Belongs in punktfunk-host (ENet via a Rust ENet crate); the AES-128-GCM seal/open of the control payload can reuse a core crypto primitive but the framing is host-side."
|
||||
},
|
||||
{
|
||||
"name": "audio packet (new in host)",
|
||||
@@ -402,37 +402,37 @@
|
||||
}
|
||||
],
|
||||
"flow": [
|
||||
"PHASE A (core, low risk): Add a P1 'gamestream wire mode' to lumen-core alongside the internal format. New module crates/lumen-core/src/protocol/gamestream.rs implementing (a) RTP+reserved+NV_VIDEO_PACKET serialize/parse with exact bit-packing, (b) a GameStream-layout FEC packetizer/reassembler that emits data-then-parity contiguous RTP shards at packetSize+RTP shard size, (c) the video_packet_enc_prefix_t AES-128-GCM path. Gate behind ProtocolPhase::P1GameStream (already exists). Keep lumen's internal 40-byte format for P2.",
|
||||
"PHASE B (validate the FEC matrix — HIGHEST RISK, do early): Before building any host networking, prove byte-for-byte that lumen's reed-solomon GF(2^8) parity matches Moonlight's expectation. Capture real Sunshine video packets (or vendor moonlight-common-c's Fec.c into a test) and assert lumen-encoded parity is decodable by Moonlight's RS and vice versa. If the generator matrices differ, port Moonlight's Cauchy matrix into lumen's gf8 coder. This gates everything: if shards aren't interoperable, P1 is dead.",
|
||||
"PHASE C (host control plane, in lumen-host): Implement nvhttp on TCP 47989 (HTTP) + 47984 (HTTPS): /serverinfo XML (appversion, GfeVersion, uniqueid, HttpsPort, ExternalPort, mac, MaxLumaPixelsHEVC, ServerCodecModeSupport, currentgame, PairStatus, sessionUrl0), the /pair PIN state machine (getservercert -> clientchallenge -> serverchallengeresp -> clientpairingsecret) with PIN-AES = SHA-256(salt||pin)[..16], AES-128-ECB challenge, SHA-256, X.509 + RSA sign/verify. Persist the paired client cert; pin it for HTTPS client-cert auth.",
|
||||
"PHASE D (RTSP on TCP 48010): OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY. DESCRIBE returns SDP with x-nv-video[0].* , x-nv-vqos[0].fec.* , x-ss-general.* attributes. SETUP returns server_port= per stream. ANNOUNCE parses client's packetSize, fec.minRequiredFecPackets, maximumBitrateKbps, videoEncoderSlicesPerFrame — feed these into the lumen-core Config (shard_payload=packetSize, fec_percent, etc).",
|
||||
"PHASE E (data plane wiring): On PLAY, bind UDP 47998 (video), spawn the M0 capture->NVENC pipeline, and drive lumen-core's P1 packetizer to that socket. Bind UDP 48000 ENet control (request-IDR -> force NVENC keyframe; loss-stats -> adjust; termination). Audio (UDP 47999, AES-CBC) and full input can follow.",
|
||||
"PHASE A (core, low risk): Add a P1 'gamestream wire mode' to punktfunk-core alongside the internal format. New module crates/punktfunk-core/src/protocol/gamestream.rs implementing (a) RTP+reserved+NV_VIDEO_PACKET serialize/parse with exact bit-packing, (b) a GameStream-layout FEC packetizer/reassembler that emits data-then-parity contiguous RTP shards at packetSize+RTP shard size, (c) the video_packet_enc_prefix_t AES-128-GCM path. Gate behind ProtocolPhase::P1GameStream (already exists). Keep punktfunk's internal 40-byte format for P2.",
|
||||
"PHASE B (validate the FEC matrix — HIGHEST RISK, do early): Before building any host networking, prove byte-for-byte that punktfunk's reed-solomon GF(2^8) parity matches Moonlight's expectation. Capture real Sunshine video packets (or vendor moonlight-common-c's Fec.c into a test) and assert punktfunk-encoded parity is decodable by Moonlight's RS and vice versa. If the generator matrices differ, port Moonlight's Cauchy matrix into punktfunk's gf8 coder. This gates everything: if shards aren't interoperable, P1 is dead.",
|
||||
"PHASE C (host control plane, in punktfunk-host): Implement nvhttp on TCP 47989 (HTTP) + 47984 (HTTPS): /serverinfo XML (appversion, GfeVersion, uniqueid, HttpsPort, ExternalPort, mac, MaxLumaPixelsHEVC, ServerCodecModeSupport, currentgame, PairStatus, sessionUrl0), the /pair PIN state machine (getservercert -> clientchallenge -> serverchallengeresp -> clientpairingsecret) with PIN-AES = SHA-256(salt||pin)[..16], AES-128-ECB challenge, SHA-256, X.509 + RSA sign/verify. Persist the paired client cert; pin it for HTTPS client-cert auth.",
|
||||
"PHASE D (RTSP on TCP 48010): OPTIONS/DESCRIBE/SETUP/ANNOUNCE/PLAY. DESCRIBE returns SDP with x-nv-video[0].* , x-nv-vqos[0].fec.* , x-ss-general.* attributes. SETUP returns server_port= per stream. ANNOUNCE parses client's packetSize, fec.minRequiredFecPackets, maximumBitrateKbps, videoEncoderSlicesPerFrame — feed these into the punktfunk-core Config (shard_payload=packetSize, fec_percent, etc).",
|
||||
"PHASE E (data plane wiring): On PLAY, bind UDP 47998 (video), spawn the M0 capture->NVENC pipeline, and drive punktfunk-core's P1 packetizer to that socket. Bind UDP 48000 ENet control (request-IDR -> force NVENC keyframe; loss-stats -> adjust; termination). Audio (UDP 47999, AES-CBC) and full input can follow.",
|
||||
"PHASE F (discovery + display): mDNS-advertise _nvstream._tcp. On RTSP SETUP/PLAY, create the wlroots virtual output sized to the negotiated WxH@fps, point M0 capture at it, tear down on RTSP TEARDOWN / ENet termination."
|
||||
],
|
||||
"crypto": "VIDEO (P1 wire): AES-128-GCM, key=riKey from /launch (16 bytes). Per-packet prefix video_packet_enc_prefix_t{iv[12],u32 frameNumber,u8 tag[16]}; IV = 8-byte LE per-stream counter in iv[0..8], iv[11]='V'(0x56), no AAD. Only payload encrypted. ||| CONTROL: AES-128-GCM (Sunshine SS_ENC_CONTROL_V2), 12-byte LE IV = seq[0..4], iv[10..12]='CC', 16-byte tag, NVCTL_ENCRYPTED_PACKET_HEADER prefix. ||| AUDIO: AES-128-CBC, IV = BE u32(avRiKeyId + seq), avRiKeyId = first 4 bytes of launch IV. ||| PAIRING (nvhttp): PIN-derived key = first 16 bytes of SHA-256(salt(16) || ascii-pin(4)); AES-128-ECB for the challenge/response blocks; SHA-256 for the rolling hashes; RSA (server key + client cert) for signing/verifying the pairing secret; X.509 certs exchanged (server cert returned in getservercert, client cert pinned for HTTPS). ||| DELTA vs lumen crypto.rs: lumen uses AES-128-GCM but with a 4-byte random salt + 8-byte BE seq nonce and seq-as-AAD, sealing the WHOLE packet and prefixing 8-byte seq — none of these match GameStream's iv/marker/prefix/no-AAD scheme. lumen has NO ECB/CBC, NO RSA/X.509, NO PIN-KDF. So: keep lumen's GCM for P2; add a distinct gamestream-gcm path for P1; add ECB+CBC+RSA+X.509+SHA-256-KDF in the host pairing layer (rustls/aws-lc-rs/rsa/x509 crates).",
|
||||
"rust_options": "FEC: KEEP the existing `reed-solomon` GF(2^8) coder in lumen-core for math, but it MUST be validated byte-compatible with Moonlight's Fec.c (CM256/Plank Cauchy matrix) — if not, port that matrix. (reed-solomon-simd is GF(2^16), P2 only, NOT moonlight-compatible.) ||| ENet control: `rusty_enet` (pure-Rust ENet 1.3.x, no_std-friendly, actively maintained) — speaks the exact ENet wire protocol Moonlight expects; alternative is FFI to libenet via `enet-sys`. ||| RTSP: NO good off-the-shelf server crate handles GameStream's non-standard interleaved/encrypted RTSP — hand-roll a minimal parser over a tokio TcpListener (it's ~6 verbs); `httparse`-style manual parsing. Do NOT pull a full RTSP stack. ||| HTTPS with pinned client-cert: `axum`/`hyper` + `rustls` (ServerConfig with a custom `ClientCertVerifier` that checks the cert against the paired set) + `tokio-rustls`; or `actix-web` with rustls. The plan already commits to axum+tokio for the control plane. ||| X.509 gen: `rcgen` (generate the self-signed server cert + key on first run); parse/verify client certs with `x509-parser` + `rsa` + `sha2`. PIN-KDF and ECB/CBC/GCM via `aes`, `aes-gcm`, `cbc`, `ecb` (RustCrypto) or `aws-lc-rs`/`openssl`. ||| mDNS: `mdns-sd` (pure-Rust, registers `_nvstream._tcp.local` with TXT records) or `zeroconf` (FFI to Avahi). `mdns-sd` preferred (no daemon dependency). ||| Opus audio: `audiopus`/`opus` crate if/when audio is implemented.",
|
||||
"reuse_from_lumen": "REUSE: (1) lumen-core's GF(2^8) `reed-solomon` coder and its data-then-parity reconstruct() contract — same ordering as GameStream; (2) FecConfig::recovery_for ceil(k*pct/100) — IDENTICAL to Moonlight's parity math; (3) the ReassemblerLimits bounds-before-allocate hardening pattern — reuse the same discipline when parsing attacker-controlled NV_VIDEO_PACKET fields; (4) aes-gcm dependency and crypto.rs structure (the GCM primitive itself, even though nonce/prefix scheme differs); (5) ProtocolPhase::P1GameStream / FecScheme::Gf8 enums already exist as the negotiation hook; (6) lumen-host M0's capture->NVENC pipeline produces exactly the HEVC/H264 Annex-B AUs that become GameStream video payload; (7) the Packetizer/Reassembler split is the right shape — add a parallel GameStream packetizer/reassembler beside them. ||| MUST BUILD NEW: the RTP+NV_VIDEO_PACKET (de)serialization with bit-packed fecInfo/multiFecBlocks; the GameStream-layout shard emitter (contiguous data-then-parity, packetSize+RTP shard size, no 40-byte lumen header); the video_packet_enc_prefix_t GCM path (iv counter + 'V', payload-only, no AAD); the ENTIRE control plane (nvhttp pairing, RTSP, ENet control, mDNS, X.509/RSA, ECB/CBC); audio AES-CBC path. ||| CANNOT REUSE on the wire: lumen's 40-byte PacketHeader, its 8-byte-seq GCM framing, its per-direction salt bit — all are lumen-internal inventions absent from GameStream.",
|
||||
"crypto": "VIDEO (P1 wire): AES-128-GCM, key=riKey from /launch (16 bytes). Per-packet prefix video_packet_enc_prefix_t{iv[12],u32 frameNumber,u8 tag[16]}; IV = 8-byte LE per-stream counter in iv[0..8], iv[11]='V'(0x56), no AAD. Only payload encrypted. ||| CONTROL: AES-128-GCM (Sunshine SS_ENC_CONTROL_V2), 12-byte LE IV = seq[0..4], iv[10..12]='CC', 16-byte tag, NVCTL_ENCRYPTED_PACKET_HEADER prefix. ||| AUDIO: AES-128-CBC, IV = BE u32(avRiKeyId + seq), avRiKeyId = first 4 bytes of launch IV. ||| PAIRING (nvhttp): PIN-derived key = first 16 bytes of SHA-256(salt(16) || ascii-pin(4)); AES-128-ECB for the challenge/response blocks; SHA-256 for the rolling hashes; RSA (server key + client cert) for signing/verifying the pairing secret; X.509 certs exchanged (server cert returned in getservercert, client cert pinned for HTTPS). ||| DELTA vs punktfunk crypto.rs: punktfunk uses AES-128-GCM but with a 4-byte random salt + 8-byte BE seq nonce and seq-as-AAD, sealing the WHOLE packet and prefixing 8-byte seq — none of these match GameStream's iv/marker/prefix/no-AAD scheme. punktfunk has NO ECB/CBC, NO RSA/X.509, NO PIN-KDF. So: keep punktfunk's GCM for P2; add a distinct gamestream-gcm path for P1; add ECB+CBC+RSA+X.509+SHA-256-KDF in the host pairing layer (rustls/aws-lc-rs/rsa/x509 crates).",
|
||||
"rust_options": "FEC: KEEP the existing `reed-solomon` GF(2^8) coder in punktfunk-core for math, but it MUST be validated byte-compatible with Moonlight's Fec.c (CM256/Plank Cauchy matrix) — if not, port that matrix. (reed-solomon-simd is GF(2^16), P2 only, NOT moonlight-compatible.) ||| ENet control: `rusty_enet` (pure-Rust ENet 1.3.x, no_std-friendly, actively maintained) — speaks the exact ENet wire protocol Moonlight expects; alternative is FFI to libenet via `enet-sys`. ||| RTSP: NO good off-the-shelf server crate handles GameStream's non-standard interleaved/encrypted RTSP — hand-roll a minimal parser over a tokio TcpListener (it's ~6 verbs); `httparse`-style manual parsing. Do NOT pull a full RTSP stack. ||| HTTPS with pinned client-cert: `axum`/`hyper` + `rustls` (ServerConfig with a custom `ClientCertVerifier` that checks the cert against the paired set) + `tokio-rustls`; or `actix-web` with rustls. The plan already commits to axum+tokio for the control plane. ||| X.509 gen: `rcgen` (generate the self-signed server cert + key on first run); parse/verify client certs with `x509-parser` + `rsa` + `sha2`. PIN-KDF and ECB/CBC/GCM via `aes`, `aes-gcm`, `cbc`, `ecb` (RustCrypto) or `aws-lc-rs`/`openssl`. ||| mDNS: `mdns-sd` (pure-Rust, registers `_nvstream._tcp.local` with TXT records) or `zeroconf` (FFI to Avahi). `mdns-sd` preferred (no daemon dependency). ||| Opus audio: `audiopus`/`opus` crate if/when audio is implemented.",
|
||||
"reuse_from_punktfunk": "REUSE: (1) punktfunk-core's GF(2^8) `reed-solomon` coder and its data-then-parity reconstruct() contract — same ordering as GameStream; (2) FecConfig::recovery_for ceil(k*pct/100) — IDENTICAL to Moonlight's parity math; (3) the ReassemblerLimits bounds-before-allocate hardening pattern — reuse the same discipline when parsing attacker-controlled NV_VIDEO_PACKET fields; (4) aes-gcm dependency and crypto.rs structure (the GCM primitive itself, even though nonce/prefix scheme differs); (5) ProtocolPhase::P1GameStream / FecScheme::Gf8 enums already exist as the negotiation hook; (6) punktfunk-host M0's capture->NVENC pipeline produces exactly the HEVC/H264 Annex-B AUs that become GameStream video payload; (7) the Packetizer/Reassembler split is the right shape — add a parallel GameStream packetizer/reassembler beside them. ||| MUST BUILD NEW: the RTP+NV_VIDEO_PACKET (de)serialization with bit-packed fecInfo/multiFecBlocks; the GameStream-layout shard emitter (contiguous data-then-parity, packetSize+RTP shard size, no 40-byte punktfunk header); the video_packet_enc_prefix_t GCM path (iv counter + 'V', payload-only, no AAD); the ENTIRE control plane (nvhttp pairing, RTSP, ENet control, mDNS, X.509/RSA, ECB/CBC); audio AES-CBC path. ||| CANNOT REUSE on the wire: punktfunk's 40-byte PacketHeader, its 8-byte-seq GCM framing, its per-direction salt bit — all are punktfunk-internal inventions absent from GameStream.",
|
||||
"gotchas": [
|
||||
"RS GENERATOR MATRIX is the #1 interop risk: same GF(2^8) RS and same data-first ordering does NOT guarantee byte-compatible parity. Moonlight's Fec.c uses a specific Cauchy/Vandermonde matrix; the Rust `reed-solomon` crate may differ. Validate against real Moonlight FIRST (Phase B) or all of P1 fails silently as 'unrecoverable loss'.",
|
||||
"GameStream RTP header is BIG-endian; lumen's PacketHeader is little-endian. NV_VIDEO_PACKET itself is little-endian. Don't conflate them.",
|
||||
"streamPacketIndex is a per-STREAM monotonic packet counter (the RTP-ish sequence), NOT lumen's per-AU stream_seq. frameIndex is the per-frame counter. Two different counters.",
|
||||
"GameStream RTP header is BIG-endian; punktfunk's PacketHeader is little-endian. NV_VIDEO_PACKET itself is little-endian. Don't conflate them.",
|
||||
"streamPacketIndex is a per-STREAM monotonic packet counter (the RTP-ish sequence), NOT punktfunk's per-AU stream_seq. frameIndex is the per-frame counter. Two different counters.",
|
||||
"multiFecBlocks caps a frame at 4 FEC blocks (2 bits). Combined with the 255-shard GF(2^8) cap, large frames at high res can overflow — this IS the 1 Gbps wall the plan describes. P1 must keep frames within 4 blocks x 255 shards; reduce via slicesPerFrame / bitrate from ANNOUNCE.",
|
||||
"Video GCM uses NO AAD and an 8-byte LE counter + 'V' marker; lumen's GCM uses seq-as-AAD + per-direction salt. A naive reuse of lumen's seal() will produce undecryptable-by-Moonlight packets. Build the gamestream-gcm path separately.",
|
||||
"Video GCM uses NO AAD and an 8-byte LE counter + 'V' marker; punktfunk's GCM uses seq-as-AAD + per-direction salt. A naive reuse of punktfunk's seal() will produce undecryptable-by-Moonlight packets. Build the gamestream-gcm path separately.",
|
||||
"Encryption is NEGOTIATED (ENCFLG_VIDEO=0x2, ENCFLG_AUDIO=0x1 in serverinfo/SETUP). Many Moonlight setups stream video in the CLEAR on LAN — implement plaintext video first, add GCM second; serverinfo's encryptionSupported/Requested controls this.",
|
||||
"Audio is AES-CBC not GCM, with a BE counter IV — a third distinct crypto scheme. Easy to get wrong if you assume GCM everywhere.",
|
||||
"Pairing PIN key = SHA-256(salt||pin)[..16] where pin is the 4 ASCII digits and salt is 16 raw bytes from the client — order and encoding matter exactly. ECB (not CBC) for the challenge blocks.",
|
||||
"The shard size is packetSize + RTP_HEADER (not just packetSize). lumen's shard_payload must be set to the negotiated packetSize and the shard the core FEC-protects must include the RTP/NV framing bytes, else the depacketizer mis-aligns.",
|
||||
"The shard size is packetSize + RTP_HEADER (not just packetSize). punktfunk's shard_payload must be set to the negotiated packetSize and the shard the core FEC-protects must include the RTP/NV framing bytes, else the depacketizer mis-aligns.",
|
||||
"HTTPS endpoint pins the CLIENT cert obtained during pairing; a stock TLS server that accepts any cert will let unpaired clients in. Use a custom rustls ClientCertVerifier.",
|
||||
"mDNS service name must be exactly _nvstream._tcp with the right TXT records or Moonlight won't auto-discover (manual IP add still works as a fallback for testing)."
|
||||
],
|
||||
"sources": [
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/packet.rs (PacketHeader 40-byte layout, Packetizer/Reassembler, ReassemblerLimits hardening, FLAG_* constants)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/crypto.rs (SessionCrypto AES-128-GCM, 4-byte salt + 8-byte BE seq nonce, seq-as-AAD, per-direction salt bit)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/config.rs (FecConfig::recovery_for ceil(k*pct/100), FecScheme::max_total_shards Gf8=255, ProtocolPhase::P1GameStream, p1_defaults)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/session.rs (seal_for_wire 8-byte seq prefix, submit_frame/poll_frame hot path)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-core/src/fec/mod.rs (ErasureCoder trait, data-then-parity reconstruct contract, GF(2^8) Gf8Coder)",
|
||||
"/home/enricobuehler/lumen/crates/lumen-host/src/{web.rs,vdisplay.rs,inject.rs,pipeline.rs,m0.rs} (control-plane stub, VirtualDisplay trait + wlroots/kwin/mutter stubs, M0 capture->NVENC->AU pipeline + lumen-core loopback)",
|
||||
"/home/enricobuehler/lumen/docs/implementation-plan.md sections 3,5,6,8 (P1/P2/P3 strategy, C ABI, virtual-display orchestration, milestones M0/M2)",
|
||||
"/home/enricobuehler/punktfunk/crates/punktfunk-core/src/packet.rs (PacketHeader 40-byte layout, Packetizer/Reassembler, ReassemblerLimits hardening, FLAG_* constants)",
|
||||
"/home/enricobuehler/punktfunk/crates/punktfunk-core/src/crypto.rs (SessionCrypto AES-128-GCM, 4-byte salt + 8-byte BE seq nonce, seq-as-AAD, per-direction salt bit)",
|
||||
"/home/enricobuehler/punktfunk/crates/punktfunk-core/src/config.rs (FecConfig::recovery_for ceil(k*pct/100), FecScheme::max_total_shards Gf8=255, ProtocolPhase::P1GameStream, p1_defaults)",
|
||||
"/home/enricobuehler/punktfunk/crates/punktfunk-core/src/session.rs (seal_for_wire 8-byte seq prefix, submit_frame/poll_frame hot path)",
|
||||
"/home/enricobuehler/punktfunk/crates/punktfunk-core/src/fec/mod.rs (ErasureCoder trait, data-then-parity reconstruct contract, GF(2^8) Gf8Coder)",
|
||||
"/home/enricobuehler/punktfunk/crates/punktfunk-host/src/{web.rs,vdisplay.rs,inject.rs,pipeline.rs,m0.rs} (control-plane stub, VirtualDisplay trait + wlroots/kwin/mutter stubs, M0 capture->NVENC->AU pipeline + punktfunk-core loopback)",
|
||||
"/home/enricobuehler/punktfunk/docs/implementation-plan.md sections 3,5,6,8 (P1/P2/P3 strategy, C ABI, virtual-display orchestration, milestones M0/M2)",
|
||||
"moonlight-common-c/src/Video.h (NV_VIDEO_PACKET 16-byte struct, RTP_PACKET 12-byte struct, FLAG_CONTAINS_PIC_DATA/EOF/SOF, FIXED_RTP_HEADER_SIZE)",
|
||||
"moonlight-common-c/src/RtpVideoQueue.c (fecInfo masks 0xFFC00000>>22 / 0x3FF000>>12 / 0xFF0>>4, parity=(data*pct+99)/100, reed_solomon_new/reed_solomon_decode, receiveSize=packetSize+MAX_RTP_HEADER_SIZE, contiguous data-then-parity sequence range, multiFecBlocks>>4&0x3 / >>6&0x3)",
|
||||
"moonlight-common-c/src/ControlStream.c (ENet channels, opcodes 0x0305/0x0307/0x0301/0x0201/0x010b/0x0100/0x010e/0x0302, NVCTL_ENCRYPTED_PACKET_HEADER, AES-128-GCM control IV seq+'CC', ping/loss-stats/HDR/rumble/termination layouts)",
|
||||
|
||||
Reference in New Issue
Block a user