On RemoteDesktop-capable desktops (KWin, GNOME), select the ScreenCast source on
a session created via the RemoteDesktop portal and start it through RemoteDesktop,
so a single grant — pre-authorized headlessly via the `kde-authorized` permission,
exactly like the libei input path — also covers screen capture. Standalone
ScreenCast has no such bypass and would raise an un-clickable dialog on a headless
box. wlroots/Sway has no RemoteDesktop portal, so it keeps the plain ScreenCast
session; the choice keys off inject::default_backend(). The PipeWire consumer is
unchanged — the anchored session yields the same fd + node id.
Validated on headless KWin (Plasma 6.4): the portal grants the session with no
dialog and PipeWire negotiates the format (1920x1080 BGRx, Streaming). Frame
delivery on KWin still pends dmabuf import — KWin hands GPU dmabuf buffers and the
M0 consumer is CPU-copy/shm only (plan §9, zero-copy) — so it's the next step;
the CPU-copy path remains correct on wlroots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a second input-injection backend that works on compositors implementing
the org.freedesktop.portal.RemoteDesktop interface (KWin, GNOME/Mutter), where
the wlroots virtual-input protocols are absent. Uses ashpd 0.13 to open a
RemoteDesktop session + EIS fd and reis 0.6.1 to drive it as an EI sender:
bind pointer/keyboard/scroll/button capabilities and, per device,
start_emulating → emit → frame. Runs on a dedicated thread with its own tokio
runtime (the portal session + EIS connection must stay alive and the event
stream must be polled continuously); open() returns immediately so a slow or
denied portal can never freeze the ENet control thread, with events enqueued
over an unbounded channel until devices resume.
Backend now auto-selects per session (inject::default_backend): wlr on Sway,
libei on KDE/GNOME; LUMEN_INPUT_BACKEND overrides. Refactor inject.rs into the
inject/{wlr,libei}.rs layout matching the capture/encode convention. Keyboard
codes are evdev (the same space our VK→evdev table produces) and the compositor
supplies the keymap, so no keymap upload and no modifier serialization — pressing
the modifier keys Moonlight sends is enough.
Add a `lumen-host input-test` subcommand that injects a scripted mouse+keyboard
pattern through the session backend, so input injection can be validated without
a Moonlight client.
Live-validated on headless KWin (Plasma 6.4): mouse motion, left click, and the
'A' key inject correctly and are delivered to the focused client.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Disconnect/reconnect now works reliably. Previously each stream spawned its own
portal+PipeWire (and PipeWire audio) capture threads and never stopped them, so a
reconnect opened a SECOND screencast session that conflicted with the leaked
first one ("no PipeWire frame within 10s" → black screen on reconnect).
- The screen capturer and audio capturer are now persistent, held in AppState and
reused across streams (created on the first stream). One screencast session for
the host's lifetime → no conflict, and instant reconnect (no re-handshake).
Verified live: 3 stream cycles, 1 create + 2 "reusing capturer", clean every time.
- Capturer::set_active gates the (5K, ~1.3 GB/s) de-pad copy to active streams, so
the persistent video capturer is nearly free while idle between streams.
- AudioCapturer::drain discards buffered chunks on reuse so the client never hears
stale audio captured while idle.
- stream.rs / gamestream/audio.rs split into a borrow-the-capturer wrapper + the
encode/send body, so the capturer is always returned to its slot on exit.
This holds whether the client reconnects via /resume (Moonlight's "running →
play/continue") or a fresh /launch — both re-run RTSP PLAY → a new stream cycle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Graceful FEC behavior on a lossy link: at a realistic 2% packet loss the stream
is now steady 0% (was spiking 40-60%). Verified live.
- IDR/RFI handling: the control thread recognizes the client's recovery requests
(0x0301 invalidate-reference-frames, 0x0302 request-IDR, 0x0305) and sets a
shared force_idr flag; the video thread forces an NVENC keyframe on the next
frame (Encoder::request_keyframe → input frame pict_type = I). Without this, a
frame that exceeds the FEC budget broke the reference chain until the next GOP
IDR (~2s), cascading to most of the stream being undecodable.
- Min-parity floor: honor the client's x-nv-vqos[0].fec.minRequiredFecPackets
(it asks for 2). Small P-frames previously got m=ceil(k*20/100)=1 parity — a
single loss broke them; flooring m>=2 (capped so k+m<=255, wire pct recomputed)
protects them. This is what turned the 2% spikes into steady 0%.
- Send pacing: spread each frame's packets evenly across the frame interval
instead of blasting them at line rate (a real link drops microbursts), matching
Sunshine's rate-controlled sends; sub-500us sleeps skipped (unreliable).
Note: sustained ~8% uniform loss still degrades — that exceeds 20% FEC for
reference-frame video and real Sunshine degrades there too; real networks are
<1% or bursty, which this now handles cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Moonlight now reconstructs lost video shards from our parity (verified live:
under induced packet loss the picture recovers cleanly instead of failing with
"network connection too bad"; 0% added loss in normal operation).
The decisive finding: Moonlight's nanors uses a CAUCHY generator matrix
(M[j][i] = inv[(m+i)^j], GF(2^8) poly 0x1d), while reed-solomon-erasure is
Vandermonde — so its parity was NOT Moonlight-decodable, despite the old
gf8.rs comment claiming equivalence.
lumen-core:
- Swap the GF(2^8) backend from reed-solomon-erasure to a vendored fec-rs
(vendor/fec-rs, BSD-2), which builds the byte-identical Cauchy matrix. Pure
Rust, no FFI — keeps the "one core" hot path. This makes both lumen's own
protocol and the GameStream parity nanors-compatible.
- Lock it with a regression test against real nanors vectors
(k=4,m=2 [10,20,30,40] -> parity [136,0]) + an independent matrix-derived
cross-check + an erase/recover round-trip. Existing FEC/loopback tests stay
green, so lumen's own protocol is unaffected.
lumen-host video.rs:
- Generate m = ceil(k*pct/100) parity shards per FEC block via Gf8Coder; stamp
fecInfo with the recomputed wire pct (100*m/k) so the client derives the same
count; cap per-block data to 255*100/(100+pct) so k+m <= 255.
- CRITICAL byte-exactness: RS runs over the whole `blocksize` shard (Moonlight
decodes packetSize+16 bytes from the datagram start and PACKET_RECOVERY_FAILUREs
on a bad reconstructed `flags` byte). So the NV header fields RS must reproduce
(streamPacketIndex/frameIndex/flags/multiFec*) are written into data shards
BEFORE encode, and only the transport fields (RTP header/seq/timestamp +
fecInfo) are stamped AFTER — leaving the flags byte RS-covered. Matches
Sunshine stream.cpp. Unit-tested incl. flags recovery.
- fec_percentage wired from stream.rs (Sunshine default 20, LUMEN_FEC_PCT
override; 0 = data-only). LUMEN_VIDEO_DROP injects loss to test recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stock Moonlight client now gets video + full input + AUDIO from the
from-scratch GameStream host (verified live end-to-end on a macOS client).
Audio (audio.rs, audio/linux.rs, gamestream/audio.rs):
- Capture the default PipeWire sink's monitor (system output) as interleaved
f32 stereo @ 48kHz via stream.capture.sink, on its own thread.
- Opus-encode 5ms/240-sample stereo frames (RESTRICTED_LOWDELAY, CBR) and send
as GameStream RTP audio: 12-byte BE RTP_PACKET (packetType 97, seq+1/pkt,
timestamp += packetDuration, ssrc 0) on UDP 48000, after learning the client
endpoint from its port-learning ping.
- Encrypt the Opus payload with AES-128-CBC (PKCS7), key = launch rikey, IV =
BE32(rikeyid + seq) in [0..4]. Like the control stream, modern Moonlight
always decrypts audio regardless of the negotiated flags — plaintext makes it
log "Failed to decrypt audio packet" and play silence (diagnosed from the
client log). RTP header stays in the clear. Scheme cross-checked against
Sunshine stream.cpp/crypto.cpp + moonlight AudioStream.c.
- Pace each frame to its 5ms slot (PipeWire delivers ~1024-frame buffers) to
avoid bursts the client's jitter buffer hears as glitches. LUMEN_AUDIO_GAIN
applies optional linear gain for quiet sources.
- DESCRIBE SDP advertises the stereo Opus config (a=fmtp:97 surround-params).
Video (stream.rs): pace at a steady ≤60fps, re-encoding the last captured frame
when the compositor produces none. wlroots only emits on damage, so a static or
slow-updating desktop previously starved the client into a "network too slow"
abort; an unchanged frame costs a near-empty P-frame. Adds a non-blocking
Capturer::try_latest (portal drains to the freshest queued frame).
Misc: serialize pipewire init across the video + audio capture threads
(pwinit.rs, std::sync::Once) to avoid a concurrent pw_init race. Deps: opus,
cbc; libopus-dev in bootstrap-ubuntu.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stock Moonlight client can now drive the headless Sway desktop: mouse
movement, buttons, scroll, and keyboard all inject through the streamed
session (verified live end-to-end — typing, clicking, window management).
Control stream (gamestream/control.rs):
- Moonlight encrypts the ENet control stream with AES-128-GCM even though we
negotiate no media encryption (it detects our Sunshine `state` and turns it
on). Decrypt per-packet under the /launch `rikey`.
- The exact GCM scheme is auto-detected on the first authenticating packet
(nonce construction × key byte-order × tag position × AAD) since GCM gives no
partial credit. Our client uses the legacy 16-byte nonce (`iv[0]=seq&0xff`)
because we advertise no encryption; the 12-byte SS_ENC_CONTROL_V2 nonce is
also supported. Key/IV/tag layout cross-checked against Sunshine stream.cpp +
crypto.cpp and moonlight-common-c ControlStream.c.
Input decode (gamestream/input.rs):
- Decrypted control messages (`[u16 type][u16 len][NV_INPUT packet]`, type
0x0206) decode into lumen_core::input::InputEvent: relative/abs mouse, buttons,
vert/horiz scroll, keyboard down/up. Struct layout from moonlight Input.h
(size BE, magic LE, body BE; keyCode LE masked to the low-byte VK), dispatch
per Sunshine input.cpp (Gen5+). Unit-tested against real captured bytes.
Injection (inject.rs):
- WlrootsInjector: connects to Sway as a Wayland client and injects via the
wlroots virtual-pointer + virtual-keyboard protocols (uinput is invisible to a
compositor running WLR_LIBINPUT_NO_DEVICES=1). Uploads an evdev/US xkb keymap,
tracks modifier state, and maps Windows VK → Linux evdev (full table).
Deps: aes-gcm, wayland-client, wayland-protocols-{wlr,misc}, xkbcommon (+
libxkbcommon-dev in bootstrap-ubuntu.sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire M0's portal desktop capture into the GameStream video plane: with
LUMEN_VIDEO_SOURCE=portal the stream captures the headless wlroots desktop
(PipeWire RGB) instead of the synthetic pattern, opens NVENC from the first
captured frame's format/size, and streams it. Verified live: a stock Moonlight
client shows the real 5120×1440 desktop at ~42 fps (release build).
- capture.rs: FastSyntheticCapturer (cheap fill pattern, real-time at 5K) so both
sources share the Capturer trait
- stream.rs: source select (portal | synthetic), encoder opened from the first
frame, wall-clock 90 kHz RTP timestamps (correct under a variable capture rate)
Note: the CPU-copy RGB→rgb0 path caps ~42 fps at 5K (single-threaded); dmabuf
zero-copy is the deferred optimization (plan §9).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stock Moonlight client now decodes H.265 from the lumen host end-to-end
(verified at 5120×1440@120 on RTX 5070 Ti):
- control.rs: ENet control host on UDP 47999 (rusty_enet). Moonlight starts the
control stream before video (STAGE_CONTROL_STREAM_START precedes _VIDEO_), so it
must be up first — this was the blocker behind the earlier "error 35".
- stream.rs: video data plane — on RTSP PLAY, learn the client endpoint from its
ping, NVENC-encode at the negotiated mode, packetize (GameStream RTP/NV/FEC),
send over UDP 47998; stops when the client disconnects.
- rtsp.rs: ANNOUNCE → StreamConfig (resolution/fps/packetSize/bitrate/codec), PLAY
starts the stream, TEARDOWN stops it; PairStatus=1 over the mutual-TLS port.
P1.3 uses a synthetic test pattern + data-shards-only FEC (clean-LAN). Next: real
portal desktop capture, input injection (decode control → uinput), nanors-exact FEC,
encryption, audio.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prepares the move to the NVIDIA-GPU Ubuntu VM where M0/M2 run (macOS can't drive the
Wayland/GPU stack). The repo carries the context, since Claude Code sessions are
machine-local and don't transfer.
- CLAUDE.md: project state + design invariants + don't-regress security notes. Auto-loads
every session, so a fresh session on the VM continues from here.
- scripts/bootstrap-ubuntu.sh: verifies the (already-installed) NVIDIA/NVENC stack,
installs rustup + PipeWire/portal/wlroots/Sway + DRM/EGL/GBM/VA dev deps; GATES the
FFmpeg -dev headers so apt can't clobber a custom NVENC build; checks nvidia-drm.modeset.
- scripts/headless/: headless-Sway + xdg-desktop-portal-wlr config templates, the
NVIDIA-wlroots env workarounds, run-headless-sway.sh, and a wf-recorder->hevc_nvenc
capture smoke test (proves capture->NVENC with no Rust).
- docs/linux-setup.md: M0 walkthrough + verified gotchas (modeset, headless backend,
vGPU NVENC licensing, dmabuf->NVENC CPU-copy fallback, FFmpeg-dev gate, crate versions).
Ubuntu 24.04 package names/versions verified against the live archive; scripts pass
shellcheck and `bash -n`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler