A stock Moonlight client now gets video + full input + AUDIO from the
from-scratch GameStream host (verified live end-to-end on a macOS client).
Audio (audio.rs, audio/linux.rs, gamestream/audio.rs):
- Capture the default PipeWire sink's monitor (system output) as interleaved
f32 stereo @ 48kHz via stream.capture.sink, on its own thread.
- Opus-encode 5ms/240-sample stereo frames (RESTRICTED_LOWDELAY, CBR) and send
as GameStream RTP audio: 12-byte BE RTP_PACKET (packetType 97, seq+1/pkt,
timestamp += packetDuration, ssrc 0) on UDP 48000, after learning the client
endpoint from its port-learning ping.
- Encrypt the Opus payload with AES-128-CBC (PKCS7), key = launch rikey, IV =
BE32(rikeyid + seq) in [0..4]. Like the control stream, modern Moonlight
always decrypts audio regardless of the negotiated flags — plaintext makes it
log "Failed to decrypt audio packet" and play silence (diagnosed from the
client log). RTP header stays in the clear. Scheme cross-checked against
Sunshine stream.cpp/crypto.cpp + moonlight AudioStream.c.
- Pace each frame to its 5ms slot (PipeWire delivers ~1024-frame buffers) to
avoid bursts the client's jitter buffer hears as glitches. LUMEN_AUDIO_GAIN
applies optional linear gain for quiet sources.
- DESCRIBE SDP advertises the stereo Opus config (a=fmtp:97 surround-params).
Video (stream.rs): pace at a steady ≤60fps, re-encoding the last captured frame
when the compositor produces none. wlroots only emits on damage, so a static or
slow-updating desktop previously starved the client into a "network too slow"
abort; an unchanged frame costs a near-empty P-frame. Adds a non-blocking
Capturer::try_latest (portal drains to the freshest queued frame).
Misc: serialize pipewire init across the video + audio capture threads
(pwinit.rs, std::sync::Once) to avoid a concurrent pw_init race. Deps: opus,
cbc; libopus-dev in bootstrap-ubuntu.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stock Moonlight client can now drive the headless Sway desktop: mouse
movement, buttons, scroll, and keyboard all inject through the streamed
session (verified live end-to-end — typing, clicking, window management).
Control stream (gamestream/control.rs):
- Moonlight encrypts the ENet control stream with AES-128-GCM even though we
negotiate no media encryption (it detects our Sunshine `state` and turns it
on). Decrypt per-packet under the /launch `rikey`.
- The exact GCM scheme is auto-detected on the first authenticating packet
(nonce construction × key byte-order × tag position × AAD) since GCM gives no
partial credit. Our client uses the legacy 16-byte nonce (`iv[0]=seq&0xff`)
because we advertise no encryption; the 12-byte SS_ENC_CONTROL_V2 nonce is
also supported. Key/IV/tag layout cross-checked against Sunshine stream.cpp +
crypto.cpp and moonlight-common-c ControlStream.c.
Input decode (gamestream/input.rs):
- Decrypted control messages (`[u16 type][u16 len][NV_INPUT packet]`, type
0x0206) decode into lumen_core::input::InputEvent: relative/abs mouse, buttons,
vert/horiz scroll, keyboard down/up. Struct layout from moonlight Input.h
(size BE, magic LE, body BE; keyCode LE masked to the low-byte VK), dispatch
per Sunshine input.cpp (Gen5+). Unit-tested against real captured bytes.
Injection (inject.rs):
- WlrootsInjector: connects to Sway as a Wayland client and injects via the
wlroots virtual-pointer + virtual-keyboard protocols (uinput is invisible to a
compositor running WLR_LIBINPUT_NO_DEVICES=1). Uploads an evdev/US xkb keymap,
tracks modifier state, and maps Windows VK → Linux evdev (full table).
Deps: aes-gcm, wayland-client, wayland-protocols-{wlr,misc}, xkbcommon (+
libxkbcommon-dev in bootstrap-ubuntu.sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire M0's portal desktop capture into the GameStream video plane: with
LUMEN_VIDEO_SOURCE=portal the stream captures the headless wlroots desktop
(PipeWire RGB) instead of the synthetic pattern, opens NVENC from the first
captured frame's format/size, and streams it. Verified live: a stock Moonlight
client shows the real 5120×1440 desktop at ~42 fps (release build).
- capture.rs: FastSyntheticCapturer (cheap fill pattern, real-time at 5K) so both
sources share the Capturer trait
- stream.rs: source select (portal | synthetic), encoder opened from the first
frame, wall-clock 90 kHz RTP timestamps (correct under a variable capture rate)
Note: the CPU-copy RGB→rgb0 path caps ~42 fps at 5K (single-threaded); dmabuf
zero-copy is the deferred optimization (plan §9).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stock Moonlight client now decodes H.265 from the lumen host end-to-end
(verified at 5120×1440@120 on RTX 5070 Ti):
- control.rs: ENet control host on UDP 47999 (rusty_enet). Moonlight starts the
control stream before video (STAGE_CONTROL_STREAM_START precedes _VIDEO_), so it
must be up first — this was the blocker behind the earlier "error 35".
- stream.rs: video data plane — on RTSP PLAY, learn the client endpoint from its
ping, NVENC-encode at the negotiated mode, packetize (GameStream RTP/NV/FEC),
send over UDP 47998; stops when the client disconnects.
- rtsp.rs: ANNOUNCE → StreamConfig (resolution/fps/packetSize/bitrate/codec), PLAY
starts the stream, TEARDOWN stops it; PairStatus=1 over the mutual-TLS port.
P1.3 uses a synthetic test pattern + data-shards-only FEC (clean-LAN). Next: real
portal desktop capture, input injection (decode control → uinput), nanors-exact FEC,
encryption, audio.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prepares the move to the NVIDIA-GPU Ubuntu VM where M0/M2 run (macOS can't drive the
Wayland/GPU stack). The repo carries the context, since Claude Code sessions are
machine-local and don't transfer.
- CLAUDE.md: project state + design invariants + don't-regress security notes. Auto-loads
every session, so a fresh session on the VM continues from here.
- scripts/bootstrap-ubuntu.sh: verifies the (already-installed) NVIDIA/NVENC stack,
installs rustup + PipeWire/portal/wlroots/Sway + DRM/EGL/GBM/VA dev deps; GATES the
FFmpeg -dev headers so apt can't clobber a custom NVENC build; checks nvidia-drm.modeset.
- scripts/headless/: headless-Sway + xdg-desktop-portal-wlr config templates, the
NVIDIA-wlroots env workarounds, run-headless-sway.sh, and a wf-recorder->hevc_nvenc
capture smoke test (proves capture->NVENC with no Rust).
- docs/linux-setup.md: M0 walkthrough + verified gotchas (modeset, headless backend,
vGPU NVENC licensing, dmabuf->NVENC CPU-copy fallback, FFmpeg-dev gate, crate versions).
Ubuntu 24.04 package names/versions verified against the live archive; scripts pass
shellcheck and `bash -n`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler