feat: M2 P1.6 — audio (Opus + AES-CBC) and steady-rate video pacing
A stock Moonlight client now gets video + full input + AUDIO from the from-scratch GameStream host (verified live end-to-end on a macOS client). Audio (audio.rs, audio/linux.rs, gamestream/audio.rs): - Capture the default PipeWire sink's monitor (system output) as interleaved f32 stereo @ 48kHz via stream.capture.sink, on its own thread. - Opus-encode 5ms/240-sample stereo frames (RESTRICTED_LOWDELAY, CBR) and send as GameStream RTP audio: 12-byte BE RTP_PACKET (packetType 97, seq+1/pkt, timestamp += packetDuration, ssrc 0) on UDP 48000, after learning the client endpoint from its port-learning ping. - Encrypt the Opus payload with AES-128-CBC (PKCS7), key = launch rikey, IV = BE32(rikeyid + seq) in [0..4]. Like the control stream, modern Moonlight always decrypts audio regardless of the negotiated flags — plaintext makes it log "Failed to decrypt audio packet" and play silence (diagnosed from the client log). RTP header stays in the clear. Scheme cross-checked against Sunshine stream.cpp/crypto.cpp + moonlight AudioStream.c. - Pace each frame to its 5ms slot (PipeWire delivers ~1024-frame buffers) to avoid bursts the client's jitter buffer hears as glitches. LUMEN_AUDIO_GAIN applies optional linear gain for quiet sources. - DESCRIBE SDP advertises the stereo Opus config (a=fmtp:97 surround-params). Video (stream.rs): pace at a steady ≤60fps, re-encoding the last captured frame when the compositor produces none. wlroots only emits on damage, so a static or slow-updating desktop previously starved the client into a "network too slow" abort; an unchanged frame costs a near-empty P-frame. Adds a non-blocking Capturer::try_latest (portal drains to the freshest queued frame). Misc: serialize pipewire init across the video + audio capture threads (pwinit.rs, std::sync::Once) to avoid a concurrent pw_init race. Deps: opus, cbc; libopus-dev in bootstrap-ubuntu.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
//! Desktop audio capture for the GameStream audio stream. On Linux: a PipeWire stream that
|
||||
//! records the default sink's monitor (i.e. everything playing out of the system), delivered
|
||||
//! as interleaved `f32` stereo PCM at 48 kHz. The audio data plane (`gamestream::audio`)
|
||||
//! reframes this into fixed Opus frames, encodes, and sends it.
|
||||
|
||||
use anyhow::Result;
|
||||
|
||||
/// Opus/GameStream audio is 48 kHz stereo.
|
||||
pub const SAMPLE_RATE: u32 = 48_000;
|
||||
pub const CHANNELS: usize = 2;
|
||||
|
||||
/// Produces interleaved `f32` stereo PCM (L,R,L,R,…) at [`SAMPLE_RATE`]. Lives on its own
|
||||
/// thread; never blocks the capture loop (drops if the consumer falls behind).
|
||||
pub trait AudioCapturer: Send {
|
||||
/// Block until the next chunk of interleaved samples is available (variable size). The
|
||||
/// caller reframes into fixed Opus frames.
|
||||
fn next_chunk(&mut self) -> Result<Vec<f32>>;
|
||||
}
|
||||
|
||||
/// Open a live capturer for the default sink monitor (system output) via PipeWire.
|
||||
#[cfg(target_os = "linux")]
|
||||
pub fn open_audio_capture() -> Result<Box<dyn AudioCapturer>> {
|
||||
linux::PwAudioCapturer::open().map(|c| Box::new(c) as Box<dyn AudioCapturer>)
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
pub fn open_audio_capture() -> Result<Box<dyn AudioCapturer>> {
|
||||
anyhow::bail!("audio capture requires Linux + PipeWire")
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
mod linux;
|
||||
Reference in New Issue
Block a user