feat: M2 P1.6 — audio (Opus + AES-CBC) and steady-rate video pacing

A stock Moonlight client now gets video + full input + AUDIO from the
from-scratch GameStream host (verified live end-to-end on a macOS client).

Audio (audio.rs, audio/linux.rs, gamestream/audio.rs):
- Capture the default PipeWire sink's monitor (system output) as interleaved
  f32 stereo @ 48kHz via stream.capture.sink, on its own thread.
- Opus-encode 5ms/240-sample stereo frames (RESTRICTED_LOWDELAY, CBR) and send
  as GameStream RTP audio: 12-byte BE RTP_PACKET (packetType 97, seq+1/pkt,
  timestamp += packetDuration, ssrc 0) on UDP 48000, after learning the client
  endpoint from its port-learning ping.
- Encrypt the Opus payload with AES-128-CBC (PKCS7), key = launch rikey, IV =
  BE32(rikeyid + seq) in [0..4]. Like the control stream, modern Moonlight
  always decrypts audio regardless of the negotiated flags — plaintext makes it
  log "Failed to decrypt audio packet" and play silence (diagnosed from the
  client log). RTP header stays in the clear. Scheme cross-checked against
  Sunshine stream.cpp/crypto.cpp + moonlight AudioStream.c.
- Pace each frame to its 5ms slot (PipeWire delivers ~1024-frame buffers) to
  avoid bursts the client's jitter buffer hears as glitches. LUMEN_AUDIO_GAIN
  applies optional linear gain for quiet sources.
- DESCRIBE SDP advertises the stereo Opus config (a=fmtp:97 surround-params).

Video (stream.rs): pace at a steady ≤60fps, re-encoding the last captured frame
when the compositor produces none. wlroots only emits on damage, so a static or
slow-updating desktop previously starved the client into a "network too slow"
abort; an unchanged frame costs a near-empty P-frame. Adds a non-blocking
Capturer::try_latest (portal drains to the freshest queued frame).

Misc: serialize pipewire init across the video + audio capture threads
(pwinit.rs, std::sync::Once) to avoid a concurrent pw_init race. Deps: opus,
cbc; libopus-dev in bootstrap-ubuntu.sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-09 10:39:22 +00:00
parent 4c2c41acba
commit 278a6330de
13 changed files with 486 additions and 16 deletions
+18 -2
View File
@@ -18,7 +18,7 @@
use super::{CapturedFrame, Capturer, PixelFormat};
use anyhow::{anyhow, Context, Result};
use std::os::fd::OwnedFd;
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError};
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError, TryRecvError};
use std::thread;
use std::time::Duration;
@@ -70,6 +70,22 @@ impl Capturer for PortalCapturer {
Err(RecvTimeoutError::Disconnected) => Err(anyhow!("PipeWire capture thread ended")),
}
}
fn try_latest(&mut self) -> Result<Option<CapturedFrame>> {
// Drain to the newest queued frame without blocking; `None` means the compositor
// hasn't produced a new frame since last call (static/idle desktop).
let mut latest = None;
loop {
match self.frames.try_recv() {
Ok(frame) => latest = Some(frame),
Err(TryRecvError::Empty) => break,
Err(TryRecvError::Disconnected) => {
return Err(anyhow!("PipeWire capture thread ended"))
}
}
}
Ok(latest)
}
}
/// The portal handshake: connect ScreenCast, select a single monitor, start, open the
@@ -192,7 +208,7 @@ mod pipewire {
}
pub fn pipewire_thread(fd: OwnedFd, node_id: u32, tx: SyncSender<CapturedFrame>) -> Result<()> {
pw::init();
crate::pwinit::ensure_init();
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("pw Context")?;