refactor: drop milestone names + consolidate clients; loss-recovery & rumble fixes
apple / swift (push) Failing after 40s
audit / cargo-audit (push) Failing after 1m12s
windows-msix / package (push) Successful in 1m37s
windows / build (push) Successful in 1m14s
android / android (push) Successful in 4m48s
ci / web (push) Successful in 27s
ci / rust (push) Successful in 4m21s
ci / docs-site (push) Successful in 31s
ci / bench (push) Successful in 4m39s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 19s
deb / build-publish (push) Successful in 6m3s
flatpak / build-publish (push) Successful in 4m13s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m15s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m16s
docker / deploy-docs (push) Successful in 18s

Two bodies of work in one commit (the rename moved files the fixes also touched).

Naming/structure cleanup (pre-launch):
- Host modules m3.rs->punktfunk1.rs, m0.rs->spike.rs; CLI m3-host->punktfunk1-host,
  m0->spike; bare `punktfunk-host` now prints help. Types M3Options/M3Source->
  Punktfunk1Options/Punktfunk1Source.
- Clients consolidated out of crates/ into clients/: punktfunk-client-rs->
  clients/probe (crate punktfunk-probe), client-linux->clients/linux,
  client-windows->clients/windows, punktfunk-android->clients/android/native
  (crate punktfunk-client-android; kept [lib] name=punktfunk_android so the JNI
  contract is unchanged). crates/ now holds only core + host.
- Milestone codes M0-M4 purged from code/CLI/CLAUDE.md/README/docs/docs-site,
  kept only in docs/implementation-plan.md. docs/m2-plan.md->
  docs/gamestream-host-plan.md. CI/gradle/flatpak paths updated.

Client loss-recovery (video froze and never recovered after a brief drop):
- Export punktfunk_connection_frames_dropped through the C ABI (the core already
  tracked it for the client keyframe-recovery loop; it was never reachable from
  the ABI clients). Regenerated punktfunk_core.h.
- Apple (StreamPump + Stage2Pipeline) and Android (decode.rs) now poll
  frames_dropped and request a keyframe when it climbs -- the same loss-driven
  recovery Linux/Windows already had. Under infinite GOP the decoder silently
  conceals reference-missing frames, so the decode-error trigger rarely fires.

Apple rumble robustness (worked then went spotty -- DualSense + Xbox):
- Add CHHapticEngine stopped/reset handlers (rebuild on app background / audio
  interruption / server reset) and drop the permanent `broken` latch on a
  transient drive failure; latch only when the controller truly has no haptics.
- Surface swallowed SDL set_rumble errors on Linux/Windows + diagnostic logging.

Verified: cargo build/clippy/fmt --workspace, C-ABI harness, header drift.
Not runnable on this box (verify in CI): Gitea workflows, gradle/Android,
flatpak, Swift/decky.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-18 21:03:55 +00:00
parent 1faa6c6ad4
commit 9c8fa9340c
110 changed files with 534 additions and 341 deletions
+40
View File
@@ -0,0 +1,40 @@
[package]
name = "punktfunk-client-linux"
description = "Native Linux punktfunk/1 client — GTK4/libadwaita shell, FFmpeg decode, PipeWire audio, SDL3 gamepads"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[[bin]]
name = "punktfunk-client"
path = "src/main.rs"
# Everything is Linux-gated so `cargo build --workspace` stays green on macOS (the Mac
# client lives in clients/apple); on other platforms this builds as a stub binary.
[target.'cfg(target_os = "linux")'.dependencies]
punktfunk-core = { path = "../../crates/punktfunk-core", features = ["quic"] }
# UI shell. GraphicsOffload needs GTK ≥ 4.14; black-background ≥ 4.16. AlertDialog/
# PreferencesDialog need libadwaita ≥ 1.5.
gtk = { package = "gtk4", version = "0.11", features = ["v4_16"] }
adw = { package = "libadwaita", version = "0.9", features = ["v1_5"] }
async-channel = "2"
# Video decode (same FFmpeg pin as the host) and audio.
ffmpeg-next = "8"
opus = "0.3"
pipewire = "0.9"
# Gamepads: capture + feedback (full DualSense fidelity — touchpad/motion/triggers/LEDs
# need the hidapi driver).
sdl3 = { version = "0.18", features = ["hidapi"] }
mdns-sd = "0.20"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
anyhow = "1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+579
View File
@@ -0,0 +1,579 @@
//! The application shell: window, navigation, trust dialogs, session lifecycle.
use crate::session::{SessionEvent, SessionParams};
use crate::trust::{KnownHost, KnownHosts, Settings};
use crate::ui_hosts::ConnectRequest;
use adw::prelude::*;
use gtk::{gdk, glib};
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::{CompositorPref, GamepadPref};
use std::cell::RefCell;
use std::rc::Rc;
const APP_ID: &str = "io.unom.Punktfunk";
struct App {
window: adw::ApplicationWindow,
nav: adw::NavigationView,
toasts: adw::ToastOverlay,
settings: Rc<RefCell<Settings>>,
identity: (String, String),
/// App-lifetime SDL gamepad service: Settings list + per-session capture/feedback.
gamepad: crate::gamepad::GamepadService,
/// One session at a time — ignore connects while one is starting/running.
busy: std::cell::Cell<bool>,
}
impl App {
fn toast(&self, msg: &str) {
self.toasts.add_toast(adw::Toast::new(msg));
}
}
pub fn run() -> glib::ExitCode {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()),
)
.init();
// Headless pairing path (no GTK window): `--pair <PIN> --connect host[:port] [--name N]`.
// Used by the Decky plugin (a GTK dialog can't pop under gamescope) and for scripting.
if let Some(pin) = arg_value("--pair") {
return headless_pair(&pin);
}
let app = adw::Application::builder().application_id(APP_ID).build();
app.connect_activate(build_ui);
// GTK doesn't see our argv (`--connect` is handled in `build_ui`); an empty argv also
// keeps GApplication from rejecting unknown options.
app.run_with_args(&[] as &[&str])
}
/// The value following `flag` in argv, if present (`--flag value`).
fn arg_value(flag: &str) -> Option<String> {
std::env::args()
.skip_while(|a| a != flag)
.nth(1)
.filter(|v| !v.starts_with("--"))
}
/// Run the SPAKE2 PIN ceremony without a GTK window and persist the verified host to the
/// known-hosts store as paired, so a later `--connect` connects silently. Same identity
/// store the streaming path uses (same binary), so pairing here makes the stream work.
/// Prints a one-line `paired <addr>:<port> fp=<hex>` on success; exits non-zero on failure.
fn headless_pair(pin: &str) -> glib::ExitCode {
let Some(target) = arg_value("--connect") else {
eprintln!("--pair requires --connect host[:port]");
return glib::ExitCode::FAILURE;
};
let (addr, port) = match target.rsplit_once(':') {
Some((a, p)) => (a.to_string(), p.parse().unwrap_or(9777)),
None => (target.clone(), 9777),
};
// The label the HOST stores this client under (its paired-devices list).
let name = arg_value("--name").unwrap_or_else(|| "Steam Deck".to_string());
let identity = match crate::trust::load_or_create_identity() {
Ok(i) => i,
Err(e) => {
eprintln!("client identity: {e:#}");
return glib::ExitCode::FAILURE;
}
};
match NativeClient::pair(
&addr,
port,
(&identity.0, &identity.1),
pin.trim(),
&name,
std::time::Duration::from_secs(90),
) {
Ok(fp) => {
let fp_hex = crate::trust::hex(&fp);
let mut known = KnownHosts::load();
known.upsert(KnownHost {
name: arg_value("--host-label").unwrap_or_else(|| addr.clone()),
addr: addr.clone(),
port,
fp_hex: fp_hex.clone(),
paired: true,
});
let _ = known.save();
println!("paired {addr}:{port} fp={fp_hex}");
glib::ExitCode::SUCCESS
}
Err(e) => {
eprintln!("pairing failed: {e:?} (wrong PIN, or pairing not armed on the host?)");
glib::ExitCode::FAILURE
}
}
}
/// `--connect host[:port]` — skip the hosts page and start a session immediately
/// (scripting + headless testing). Trust follows the same rules as a manual entry: a host
/// already pinned at this address connects silently on its stored pin; an unknown host is
/// routed to the PIN ceremony (never a silent TOFU connect — `fp_hex`/`pair_optional` are
/// unset, so `initiate_connect`'s manual arm mandates pairing).
fn cli_connect_request() -> Option<ConnectRequest> {
let args: Vec<String> = std::env::args().collect();
let target = args
.iter()
.skip_while(|a| *a != "--connect")
.nth(1)?
.clone();
let (addr, port) = match target.rsplit_once(':') {
Some((a, p)) => (a.to_string(), p.parse().ok()?),
None => (target.clone(), 9777),
};
Some(ConnectRequest {
name: addr.clone(),
addr,
port,
fp_hex: None,
pair_optional: false,
})
}
fn build_ui(gtk_app: &adw::Application) {
let identity = match crate::trust::load_or_create_identity() {
Ok(i) => i,
Err(e) => {
tracing::error!("client identity: {e:#}");
std::process::exit(1);
}
};
let nav = adw::NavigationView::new();
let toasts = adw::ToastOverlay::new();
toasts.set_child(Some(&nav));
let window = adw::ApplicationWindow::builder()
.application(gtk_app)
.title("Punktfunk")
.default_width(1100)
.default_height(720)
.content(&toasts)
.build();
let app = Rc::new(App {
window: window.clone(),
nav: nav.clone(),
toasts,
settings: Rc::new(RefCell::new(Settings::load())),
identity,
gamepad: crate::gamepad::GamepadService::start(),
busy: std::cell::Cell::new(false),
});
let hosts_page = crate::ui_hosts::new(
{
let app = app.clone();
Rc::new(move |req| initiate_connect(app.clone(), req))
},
{
let app = app.clone();
Rc::new(move || {
crate::ui_settings::show(&app.window, app.settings.clone(), &app.gamepad)
})
},
{
let app = app.clone();
Rc::new(move |req| speed_test(app.clone(), req))
},
);
nav.add(&hosts_page);
window.present();
if let Some(req) = cli_connect_request() {
initiate_connect(app, req);
}
}
/// The trust gate in front of every connect. The host is the policy authority (it
/// advertises `pair=optional` only when it accepts unpaired clients); the client renders
/// its trust UI from that:
/// 1. PINNED RECONNECT — a host already pinned to this exact fingerprint connects silently.
/// 2. FINGERPRINT CHANGED — a host we know at this address but whose fingerprint no longer
/// matches is the impostor signal: force re-pairing via the PIN ceremony, regardless of
/// the advertised policy.
/// 3. NEW host — TOFU is offered only when the host advertised `pair=optional` (rule 3a);
/// otherwise (pair=required, unknown/empty policy, or a manual entry) PIN pairing is
/// mandatory (rule 3b).
///
/// A new host is never auto-connected without a stored pin or an explicit trust decision.
fn initiate_connect(app: Rc<App>, req: ConnectRequest) {
if app.busy.get() {
return;
}
let known = KnownHosts::load();
match &req.fp_hex {
Some(fp_hex) => {
if known.find_by_fp(fp_hex).is_some() {
// Rule 1: pinned fingerprint matches — silent connect.
start_session(app, req.clone(), crate::trust::parse_hex32(fp_hex));
} else if known.find_by_addr(&req.addr, req.port).is_some() {
// Rule 2: we trust a host at this address but the fingerprint changed —
// the impostor signal. Re-pair via the PIN ceremony (no TOFU shortcut).
app.toast("Host fingerprint changed — re-pair with a PIN to continue");
pin_dialog(app, req);
} else if req.pair_optional {
// Rule 3a: the host opted into reduced-security TOFU; offer it alongside PIN.
tofu_dialog(app, req);
} else {
// Rule 3b: pair=required or unknown policy — PIN pairing is mandatory.
pin_dialog(app, req);
}
}
None => {
// Manual entry (no advertised fingerprint). A known address connects silently
// on its stored pin (rule 1); an unknown one must pair — never silent TOFU.
match known
.find_by_addr(&req.addr, req.port)
.and_then(|k| crate::trust::parse_hex32(&k.fp_hex))
{
Some(pin) => start_session(app, req, Some(pin)),
None => pin_dialog(app, req), // rule 3b
}
}
}
}
/// First contact with a discovered host: show the advertised fingerprint and let the user
/// trust it (TOFU), run the PIN ceremony instead, or walk away.
fn tofu_dialog(app: Rc<App>, req: ConnectRequest) {
let fp = req.fp_hex.clone().unwrap_or_default();
let dialog = adw::AlertDialog::new(
Some("New Host"),
Some(&format!(
"{} at {}:{}\n\nCertificate fingerprint:\n{}\n\nPairing with a PIN verifies it; \
trusting accepts it as-is.",
req.name, req.addr, req.port, fp
)),
);
dialog.add_responses(&[
("cancel", "Cancel"),
("pair", "Pair with PIN…"),
("trust", "Trust & Connect"),
]);
dialog.set_response_appearance("trust", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("trust"));
dialog.set_close_response("cancel");
let parent = app.window.clone();
dialog.connect_response(None, move |_, response| match response {
"trust" => {
let mut known = KnownHosts::load();
known.upsert(KnownHost {
name: req.name.clone(),
addr: req.addr.clone(),
port: req.port,
fp_hex: fp.clone(),
paired: false,
});
let _ = known.save();
start_session(app.clone(), req.clone(), crate::trust::parse_hex32(&fp));
}
"pair" => pin_dialog(app.clone(), req.clone()),
_ => {}
});
dialog.present(Some(&parent));
}
/// The SPAKE2 ceremony: the host is armed and displays a 4-digit PIN; proving knowledge
/// of it pins the host's certificate (and registers ours) with no offline-guessable
/// transcript.
fn pin_dialog(app: Rc<App>, req: ConnectRequest) {
let entry = gtk::Entry::builder()
.input_purpose(gtk::InputPurpose::Digits)
.placeholder_text("4-digit PIN shown by the host")
.activates_default(true)
.build();
let dialog = adw::AlertDialog::new(
Some("Pair with PIN"),
Some(&format!(
"Arm pairing on {} (console or web UI), then enter the PIN it displays.",
req.name
)),
);
dialog.set_extra_child(Some(&entry));
dialog.add_responses(&[("cancel", "Cancel"), ("pair", "Pair")]);
dialog.set_response_appearance("pair", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("pair"));
dialog.set_close_response("cancel");
let parent = app.window.clone();
dialog.connect_response(Some("pair"), move |_, _| {
let pin = entry.text().to_string();
let app = app.clone();
let req = req.clone();
let identity = app.identity.clone();
let (tx, rx) = async_channel::bounded::<Result<[u8; 32], String>>(1);
let (host, port, name) = (req.addr.clone(), req.port, glib::host_name().to_string());
std::thread::spawn(move || {
let result = NativeClient::pair(
&host,
port,
(&identity.0, &identity.1),
pin.trim(),
&name,
std::time::Duration::from_secs(90),
)
.map_err(|e| format!("Pairing failed: {e:?} (wrong PIN, or pairing not armed?)"));
let _ = tx.send_blocking(result);
});
glib::spawn_future_local(async move {
match rx.recv().await {
Ok(Ok(fp)) => {
let fp_hex = crate::trust::hex(&fp);
let mut known = KnownHosts::load();
known.upsert(KnownHost {
name: req.name.clone(),
addr: req.addr.clone(),
port: req.port,
fp_hex,
paired: true,
});
let _ = known.save();
app.toast("Paired — connecting…");
start_session(app.clone(), req, Some(fp));
}
Ok(Err(msg)) => app.toast(&msg),
Err(_) => {}
}
});
});
dialog.present(Some(&parent));
}
/// Measure the path to a host over the real data plane (Swift's "Test Network Speed…"):
/// connect, have the host burst probe filler for 2 s up to its 3 Gbps ceiling, report
/// goodput · loss · a recommended bitrate (≈70 % of measured), and apply it in one tap.
fn speed_test(app: Rc<App>, req: ConnectRequest) {
if app.busy.replace(true) {
return;
}
let pin = req.fp_hex.as_deref().and_then(crate::trust::parse_hex32);
let status = gtk::Label::new(Some("Connecting…"));
let dialog = adw::AlertDialog::new(Some("Network Speed Test"), Some(&req.name));
dialog.set_extra_child(Some(&status));
dialog.add_responses(&[("close", "Close"), ("apply", "Apply")]);
dialog.set_response_enabled("apply", false);
dialog.set_close_response("close");
dialog.present(Some(&app.window));
let (tx, rx) =
async_channel::bounded::<Result<punktfunk_core::client::ProbeOutcome, String>>(1);
let identity = app.identity.clone();
let (host, port) = (req.addr.clone(), req.port);
std::thread::spawn(move || {
let result = (|| {
let c = NativeClient::connect(
&host,
port,
punktfunk_core::config::Mode {
width: 1280,
height: 720,
refresh_hz: 60,
},
CompositorPref::Auto,
GamepadPref::Auto,
0, // bitrate_kbps (host default)
0, // video_caps: the Linux client has no 10-bit/HDR present path yet
None, // launch: speed-test probe connect, no game
pin,
Some(identity),
std::time::Duration::from_secs(15),
)
.map_err(|e| format!("connect: {e:?}"))?;
c.request_probe(3_000_000, 2_000)
.map_err(|e| format!("probe: {e:?}"))?;
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
loop {
std::thread::sleep(std::time::Duration::from_millis(250));
let r = c.probe_result();
if r.done {
// Let the last UDP shards land before tearing down.
std::thread::sleep(std::time::Duration::from_millis(400));
return Ok(c.probe_result());
}
if std::time::Instant::now() > deadline {
return Err("probe timed out".to_string());
}
}
})();
let _ = tx.send_blocking(result);
});
glib::spawn_future_local(async move {
let outcome = rx.recv().await;
app.busy.set(false);
match outcome {
Ok(Ok(r)) => {
let mbps = f64::from(r.throughput_kbps) / 1000.0;
let recommended_kbps = r.throughput_kbps / 10 * 7;
status.set_text(&format!(
"{mbps:.0} Mbit/s measured · {:.1} % loss\nRecommended bitrate: {:.0} Mbit/s",
r.loss_pct,
f64::from(recommended_kbps) / 1000.0,
));
dialog.set_response_enabled("apply", true);
dialog.set_response_appearance("apply", adw::ResponseAppearance::Suggested);
let settings = app.settings.clone();
let toasts = app.toasts.clone();
dialog.connect_response(Some("apply"), move |_, _| {
let mut s = settings.borrow_mut();
s.bitrate_kbps = recommended_kbps;
s.save();
toasts.add_toast(adw::Toast::new(&format!(
"Bitrate set to {:.0} Mbit/s",
f64::from(recommended_kbps) / 1000.0
)));
});
}
Ok(Err(msg)) => status.set_text(&msg),
Err(_) => {}
}
});
}
/// The mode to request: explicit settings, with `0` fields resolved to the native
/// size/refresh of the monitor the window currently occupies (mirrors the Swift client's
/// native-display default).
fn resolve_mode(app: &App) -> punktfunk_core::config::Mode {
let s = app.settings.borrow();
let mut mode = punktfunk_core::config::Mode {
width: s.width,
height: s.height,
refresh_hz: s.refresh_hz,
};
if mode.width == 0 || mode.refresh_hz == 0 {
let monitor = app
.window
.surface()
.zip(gdk::Display::default())
.and_then(|(surf, d)| d.monitor_at_surface(&surf));
if let Some(m) = monitor {
let geo = m.geometry();
let scale = m.scale_factor().max(1);
if mode.width == 0 {
mode.width = (geo.width() * scale) as u32;
mode.height = (geo.height() * scale) as u32;
}
if mode.refresh_hz == 0 {
mode.refresh_hz = ((m.refresh_rate() + 500) / 1000).max(30) as u32;
}
}
}
// No monitor info (early call, odd compositor) — a sane floor.
if mode.width == 0 {
(mode.width, mode.height) = (1920, 1080);
}
if mode.refresh_hz == 0 {
mode.refresh_hz = 60;
}
mode
}
fn start_session(app: Rc<App>, req: ConnectRequest, pin: Option<[u8; 32]>) {
if app.busy.replace(true) {
return;
}
let mode = resolve_mode(&app);
let s = app.settings.borrow();
let params = SessionParams {
host: req.addr.clone(),
port: req.port,
mode,
compositor: CompositorPref::from_name(&s.compositor).unwrap_or(CompositorPref::Auto),
// "Automatic" matches the physical pad (Swift parity); an explicit choice wins.
gamepad: match GamepadPref::from_name(&s.gamepad) {
Some(GamepadPref::Auto) | None => app.gamepad.auto_pref(),
Some(explicit) => explicit,
},
bitrate_kbps: s.bitrate_kbps,
mic_enabled: s.mic_enabled,
pin,
identity: app.identity.clone(),
};
let inhibit = s.inhibit_shortcuts;
drop(s);
let tofu = pin.is_none();
let mut handle = crate::session::start(params);
let frames = std::mem::replace(&mut handle.frames, async_channel::bounded(1).1);
glib::spawn_future_local(async move {
let mut frames = Some(frames);
let mut page: Option<crate::ui_stream::StreamPage> = None;
while let Ok(event) = handle.events.recv().await {
match event {
SessionEvent::Connected {
connector,
mode,
fingerprint,
} => {
// A TOFU connect just observed the real fingerprint — pin it from now on.
if tofu {
let fp_hex = crate::trust::hex(&fingerprint);
let mut known = KnownHosts::load();
known.upsert(KnownHost {
name: req.name.clone(),
addr: req.addr.clone(),
port: req.port,
fp_hex: fp_hex.clone(),
paired: false,
});
let _ = known.save();
app.toast(&format!(
"Trusted on first use — fingerprint {}",
&fp_hex[..16]
));
}
tracing::debug!(?mode, "connected — pushing stream page");
let title = format!(
"{} · {}×{}@{}",
req.name, mode.width, mode.height, mode.refresh_hz
);
app.gamepad.attach(connector.clone());
let p = crate::ui_stream::new(
&app.window,
connector,
frames.take().expect("Connected delivered once"),
app.gamepad.escape_events(),
handle.stop.clone(),
inhibit,
&title,
);
app.nav.push(&p.page);
page = Some(p);
}
SessionEvent::Stats(s) => {
if let Some(p) = &page {
p.update_stats(s);
}
}
SessionEvent::Failed {
msg,
trust_rejected,
} => {
tracing::warn!(%msg, trust_rejected, "connect failed");
app.busy.set(false);
// A pinned connect rejected on trust grounds means the host's cert no
// longer matches the stored pin (rotated cert or impostor) — route to
// the PIN ceremony to re-establish trust rather than dead-ending.
if trust_rejected && !tofu {
app.toast("Host fingerprint changed — re-pair with a PIN to continue");
pin_dialog(app.clone(), req.clone());
} else {
app.toast(&msg);
}
break;
}
SessionEvent::Ended(err) => {
app.gamepad.detach();
app.nav.pop_to_tag("hosts");
if let Some(e) = err {
app.toast(&e);
}
app.busy.set(false);
break;
}
}
}
});
}
+382
View File
@@ -0,0 +1,382 @@
//! Audio: playback (decoded PCM → a PipeWire playback stream) and the microphone uplink
//! (PipeWire capture → Opus → 0xCB datagrams, the inverse of the host's virtual mic).
//!
//! Playback mirrors the host's virtual-mic producer (`punktfunk-host::audio::linux`) with
//! the same adaptive jitter buffer: the session pump pushes 5 ms Opus-decoded chunks on
//! the network clock; PipeWire pulls whole quanta on the device clock. Prime to ~3
//! quanta before producing, cap the ring so latency stays bounded, re-prime after a real
//! drain.
use anyhow::{Context, Result};
use punktfunk_core::client::NativeClient;
use std::collections::VecDeque;
use std::sync::mpsc::{Receiver, SyncSender, TrySendError};
use std::sync::Arc;
const SAMPLE_RATE: u32 = 48_000;
const CHANNELS: usize = 2;
/// Mic frames are 20 ms (960 samples/channel) — any size ≤ 120 ms is fine host-side.
const MIC_FRAME: usize = 960;
struct Terminate;
pub struct AudioPlayer {
pcm_tx: SyncSender<Vec<f32>>,
quit_tx: pipewire::channel::Sender<Terminate>,
thread: Option<std::thread::JoinHandle<()>>,
}
impl AudioPlayer {
/// Spawn the PipeWire playback thread. Failure (no PipeWire in the session) is
/// survivable — the caller streams video-only.
pub fn spawn() -> Result<AudioPlayer> {
// 64 × 5 ms = 320 ms of slack between the pump and the PipeWire loop.
let (pcm_tx, pcm_rx) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
let (quit_tx, quit_rx) = pipewire::channel::channel::<Terminate>();
let thread = std::thread::Builder::new()
.name("punktfunk-audio".into())
.spawn(move || {
if let Err(e) = pw_thread(pcm_rx, quit_rx) {
tracing::warn!(error = %e, "audio playback thread ended");
}
})
.context("spawn audio thread")?;
Ok(AudioPlayer {
pcm_tx,
quit_tx,
thread: Some(thread),
})
}
/// Queue one interleaved-stereo f32 chunk. Drops the chunk if the PipeWire side is
/// wedged (the renderer conceals the gap; never block the session pump).
pub fn push(&self, pcm: Vec<f32>) {
if let Err(TrySendError::Disconnected(_)) = self.pcm_tx.try_send(pcm) {
// Thread already dead — Drop will reap it; nothing to do per-chunk.
}
}
}
impl Drop for AudioPlayer {
fn drop(&mut self) {
let _ = self.quit_tx.send(Terminate);
if let Some(t) = self.thread.take() {
let _ = t.join();
}
}
}
/// Producer-side state: incoming decoded PCM and the ring the process callback drains.
struct PlayerData {
rx: Receiver<Vec<f32>>,
ring: VecDeque<f32>,
primed: bool,
}
fn pw_thread(
pcm_rx: Receiver<Vec<f32>>,
quit_rx: pipewire::channel::Receiver<Terminate>,
) -> Result<()> {
use pipewire as pw;
use pw::{properties::properties, spa};
use spa::param::audio::{AudioFormat, AudioInfoRaw};
use spa::pod::Pod;
static PW_INIT: std::sync::Once = std::sync::Once::new();
PW_INIT.call_once(pw::init);
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("pw Context")?;
let core = context
.connect_rc(None)
.context("pw connect (is PipeWire running in this session?)")?;
let _quit_guard = quit_rx.attach(mainloop.loop_(), {
let mainloop = mainloop.clone();
move |_| mainloop.quit()
});
let stream = pw::stream::StreamBox::new(
&core,
"punktfunk-client",
properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CATEGORY => "Playback",
*pw::keys::MEDIA_ROLE => "Game",
*pw::keys::NODE_NAME => "punktfunk-client",
*pw::keys::NODE_DESCRIPTION => "Punktfunk Stream",
// ~5 ms quantum (one Opus frame) keeps the ring — and so the latency — small.
*pw::keys::NODE_LATENCY => "240/48000",
},
)
.context("pw Stream")?;
let ud = PlayerData {
rx: pcm_rx,
ring: VecDeque::new(),
primed: false,
};
let _listener = stream
.add_local_listener_with_user_data(ud)
.state_changed(|_s, _ud, old, new| {
tracing::debug!(?old, ?new, "pipewire playback stream state");
})
.process(|stream, ud| {
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let Some(mut buffer) = stream.dequeue_buffer() else {
return;
};
while let Ok(chunk) = ud.rx.try_recv() {
ud.ring.extend(chunk);
}
let stride = 4 * CHANNELS; // F32LE interleaved
let datas = buffer.datas_mut();
if datas.is_empty() {
return;
}
let data = &mut datas[0];
let want_frames = data.data().map(|s| s.len() / stride).unwrap_or(0);
let want = want_frames * CHANNELS;
// Adaptive jitter buffer (same shape as the host's virtual mic): prime to
// ~3 quanta, cap at ~1 quantum of slack beyond that, re-prime after a
// genuine drain.
let target = (3 * want).clamp(720 * CHANNELS, 9600 * CHANNELS);
while ud.ring.len() > target.max(want) + want {
ud.ring.pop_front();
}
if !ud.primed && ud.ring.len() >= target {
ud.primed = true;
}
let n_frames = if let Some(slice) = data.data() {
for k in 0..want {
let s = if ud.primed {
ud.ring.pop_front().unwrap_or(0.0)
} else {
0.0
};
let off = k * 4;
slice[off..off + 4].copy_from_slice(&s.to_le_bytes());
}
want_frames
} else {
0
};
if ud.ring.is_empty() {
ud.primed = false;
}
let chunk = data.chunk_mut();
*chunk.offset_mut() = 0;
*chunk.stride_mut() = stride as _;
*chunk.size_mut() = (stride * n_frames) as _;
}));
if outcome.is_err() {
tracing::error!("panic in pipewire playback callback");
}
})
.register()
.context("register playback listener")?;
let mut info = AudioInfoRaw::new();
info.set_format(AudioFormat::F32LE);
info.set_rate(SAMPLE_RATE);
info.set_channels(CHANNELS as u32);
let obj = pw::spa::pod::Object {
type_: pw::spa::utils::SpaTypes::ObjectParamFormat.as_raw(),
id: pw::spa::param::ParamType::EnumFormat.as_raw(),
properties: info.into(),
};
let values: Vec<u8> = pw::spa::pod::serialize::PodSerializer::serialize(
std::io::Cursor::new(Vec::new()),
&pw::spa::pod::Value::Object(obj),
)
.context("serialize format pod")?
.0
.into_inner();
let mut params = [Pod::from_bytes(&values).context("pod from bytes")?];
stream
.connect(
spa::utils::Direction::Output,
None,
pw::stream::StreamFlags::AUTOCONNECT | pw::stream::StreamFlags::MAP_BUFFERS,
&mut params,
)
.context("pw stream connect")?;
mainloop.run();
tracing::debug!("pipewire playback loop exited");
Ok(())
}
/// The microphone uplink: capture the default input device, Opus-encode 20 ms chunks,
/// ship them as 0xCB datagrams into the host's virtual PipeWire source.
pub struct MicStreamer {
quit_tx: pipewire::channel::Sender<Terminate>,
thread: Option<std::thread::JoinHandle<()>>,
}
impl MicStreamer {
pub fn spawn(connector: Arc<NativeClient>) -> Result<MicStreamer> {
let (quit_tx, quit_rx) = pipewire::channel::channel::<Terminate>();
let thread = std::thread::Builder::new()
.name("punktfunk-mic".into())
.spawn(move || {
if let Err(e) = mic_thread(&connector, quit_rx) {
tracing::warn!(error = %e, "mic uplink thread ended");
}
})
.context("spawn mic thread")?;
Ok(MicStreamer {
quit_tx,
thread: Some(thread),
})
}
}
impl Drop for MicStreamer {
fn drop(&mut self) {
let _ = self.quit_tx.send(Terminate);
if let Some(t) = self.thread.take() {
let _ = t.join();
}
}
}
/// Capture-side state: accumulated PCM and the Opus encoder (encoding a 20 ms frame is
/// ~100 µs — fine inside the process callback).
struct MicData {
connector: Arc<NativeClient>,
ring: VecDeque<f32>,
encoder: opus::Encoder,
seq: u32,
out: Vec<u8>,
}
fn mic_thread(
connector: &Arc<NativeClient>,
quit_rx: pipewire::channel::Receiver<Terminate>,
) -> Result<()> {
use pipewire as pw;
use pw::{properties::properties, spa};
use spa::param::audio::{AudioFormat, AudioInfoRaw};
use spa::pod::Pod;
static PW_INIT: std::sync::Once = std::sync::Once::new();
PW_INIT.call_once(pw::init);
let mut encoder =
opus::Encoder::new(SAMPLE_RATE, opus::Channels::Stereo, opus::Application::Voip)
.map_err(|e| anyhow::anyhow!("opus encoder: {e}"))?;
let _ = encoder.set_bitrate(opus::Bitrate::Bits(64_000));
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw mic MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("pw mic Context")?;
let core = context
.connect_rc(None)
.context("pw mic connect (is PipeWire running in this session?)")?;
let _quit_guard = quit_rx.attach(mainloop.loop_(), {
let mainloop = mainloop.clone();
move |_| mainloop.quit()
});
let stream = pw::stream::StreamBox::new(
&core,
"punktfunk-mic-capture",
properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CATEGORY => "Capture",
*pw::keys::MEDIA_ROLE => "Communication",
*pw::keys::NODE_NAME => "punktfunk-mic-capture",
*pw::keys::NODE_DESCRIPTION => "Punktfunk Microphone",
},
)
.context("pw mic Stream")?;
let ud = MicData {
connector: connector.clone(),
ring: VecDeque::new(),
encoder,
seq: 0,
out: vec![0u8; 4000],
};
let _listener = stream
.add_local_listener_with_user_data(ud)
.state_changed(|_s, _ud, old, new| {
tracing::debug!(?old, ?new, "pipewire mic capture stream state");
})
.process(|stream, ud| {
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let Some(mut buffer) = stream.dequeue_buffer() else {
return;
};
let datas = buffer.datas_mut();
if datas.is_empty() {
return;
}
let data = &mut datas[0];
let n = data.chunk().size() as usize;
if let Some(slice) = data.data() {
for s in slice[..n.min(slice.len())].chunks_exact(4) {
ud.ring
.push_back(f32::from_le_bytes([s[0], s[1], s[2], s[3]]));
}
}
// Ship every complete 20 ms stereo frame.
while ud.ring.len() >= MIC_FRAME * CHANNELS {
let pcm: Vec<f32> = ud.ring.drain(..MIC_FRAME * CHANNELS).collect();
match ud.encoder.encode_float(&pcm, &mut ud.out) {
Ok(len) => {
let pts = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0);
let _ = ud.connector.send_mic(ud.seq, pts, ud.out[..len].to_vec());
ud.seq = ud.seq.wrapping_add(1);
}
Err(e) => tracing::debug!(error = %e, "opus mic encode"),
}
}
}));
if outcome.is_err() {
tracing::error!("panic in pipewire mic callback");
}
})
.register()
.context("register mic listener")?;
let mut info = AudioInfoRaw::new();
info.set_format(AudioFormat::F32LE);
info.set_rate(SAMPLE_RATE);
info.set_channels(CHANNELS as u32);
let obj = pw::spa::pod::Object {
type_: pw::spa::utils::SpaTypes::ObjectParamFormat.as_raw(),
id: pw::spa::param::ParamType::EnumFormat.as_raw(),
properties: info.into(),
};
let values: Vec<u8> = pw::spa::pod::serialize::PodSerializer::serialize(
std::io::Cursor::new(Vec::new()),
&pw::spa::pod::Value::Object(obj),
)
.context("serialize mic format pod")?
.0
.into_inner();
let mut params = [Pod::from_bytes(&values).context("mic pod from bytes")?];
stream
.connect(
spa::utils::Direction::Input,
None,
pw::stream::StreamFlags::AUTOCONNECT | pw::stream::StreamFlags::MAP_BUFFERS,
&mut params,
)
.context("pw mic stream connect")?;
mainloop.run();
tracing::debug!("pipewire mic capture loop exited");
Ok(())
}
+76
View File
@@ -0,0 +1,76 @@
//! LAN host discovery: browse the host's mDNS advert (`_punktfunk._udp`, TXT keys
//! `fp`/`pair`/`id` — see the host crate's `discovery.rs`) on a worker thread and stream
//! results to the UI.
use mdns_sd::{ServiceDaemon, ServiceEvent};
#[derive(Clone, Debug)]
pub struct DiscoveredHost {
/// Stable row key: the advertised host id, falling back to the mDNS fullname.
pub key: String,
pub name: String,
pub addr: String,
pub port: u16,
/// Host certificate fingerprint to pin (lowercase hex), empty if not advertised.
pub fp_hex: String,
/// Pairing requirement: `"required"` or `"optional"`.
pub pair: String,
}
/// Browse continuously for the app's lifetime. The thread exits when the receiver is
/// dropped (the send fails) or the daemon dies.
pub fn browse() -> async_channel::Receiver<DiscoveredHost> {
let (tx, rx) = async_channel::unbounded();
std::thread::Builder::new()
.name("punktfunk-mdns".into())
.spawn(move || {
let daemon = match ServiceDaemon::new() {
Ok(d) => d,
Err(e) => {
tracing::warn!(error = %e, "mDNS daemon failed — discovery disabled");
return;
}
};
let receiver = match daemon.browse("_punktfunk._udp.local.") {
Ok(r) => r,
Err(e) => {
tracing::warn!(error = %e, "mDNS browse failed — discovery disabled");
return;
}
};
while let Ok(event) = receiver.recv() {
if let ServiceEvent::ServiceResolved(info) = event {
let props = info.get_properties();
let val = |k: &str| props.get_property_val_str(k).unwrap_or("").to_string();
let Some(addr) = info.get_addresses().iter().next().map(|a| a.to_string())
else {
continue;
};
let id = val("id");
let host = DiscoveredHost {
key: if id.is_empty() {
info.get_fullname().to_string()
} else {
id
},
name: info
.get_fullname()
.split('.')
.next()
.unwrap_or("?")
.to_string(),
addr,
port: info.get_port(),
fp_hex: val("fp"),
pair: val("pair"),
};
if tx.send_blocking(host).is_err() {
break; // UI gone — stop browsing
}
}
}
let _ = daemon.shutdown();
})
.expect("spawn mdns thread");
rx
}
+587
View File
@@ -0,0 +1,587 @@
//! App-lifetime gamepad service over SDL3 (mirrors the Swift client's `GamepadManager` +
//! `GamepadCapture`/`GamepadFeedback`).
//!
//! One worker thread owns SDL for the process lifetime: it tracks connected pads for the
//! Settings UI, selects the ONE controller forwarded as pad 0 (user pin, else the most
//! recently connected), and — while a session is attached — forwards buttons/axes,
//! DualSense touchpad contacts and motion samples (0xCC), and renders feedback: rumble on
//! every pad, lightbar via SDL, and on a real DualSense the raw effects packet
//! (adaptive-trigger blocks replayed verbatim, player LEDs). Held state is zeroed on the
//! wire when the active pad switches or the session detaches, so nothing sticks down.
//!
//! This thread is also the single consumer of the rumble and HID-output pull planes.
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::GamepadPref;
use punktfunk_core::input::{gamepad as wire, InputEvent, InputKind};
use punktfunk_core::quic::{HidOutput, RichInput};
use std::collections::HashMap;
use std::sync::mpsc::{Receiver, Sender};
use std::sync::{Arc, Mutex};
use std::time::Duration;
/// Motion scale constants, shared convention with the Swift client (`GamepadWire`):
/// derived from hid-playstation's math over the host's fixed calibration blob. SDL hands
/// us gyro in rad/s and accel in m/s²; the DualSense report wants raw LSBs.
const GYRO_LSB_PER_RAD_S: f32 = 20.0 * 180.0 / std::f32::consts::PI;
const ACCEL_LSB_PER_G: f32 = 10_000.0;
const G: f32 = 9.80665;
/// The controller "escape" chord (Moonlight convention): L1 + R1 + Start + Select held
/// together. Intercepted by the client to leave fullscreen + release input capture — the
/// Deck has no F11 key and fullscreen hides the window chrome, so with a controller this
/// is the only way out. Four simultaneous buttons that no game uses as a deliberate
/// combo, so it can't be triggered by normal play. Still forwarded to the host (the user
/// is leaving anyway); we only also raise the escape signal.
const ESCAPE_CHORD: [u32; 4] = [wire::BTN_LB, wire::BTN_RB, wire::BTN_START, wire::BTN_BACK];
#[derive(Clone, Debug)]
pub struct PadInfo {
pub id: u32,
pub name: String,
pub is_dualsense: bool,
}
enum Ctl {
Attach(Arc<NativeClient>),
Detach,
Pin(Option<u32>),
}
#[derive(Clone)]
pub struct GamepadService {
pads: Arc<Mutex<Vec<PadInfo>>>,
active: Arc<Mutex<Option<PadInfo>>>,
pinned: Arc<Mutex<Option<u32>>>,
ctl: Sender<Ctl>,
/// Fires once per press of the [`ESCAPE_CHORD`]; the stream page consumes it to leave
/// fullscreen + release capture.
escape_rx: async_channel::Receiver<()>,
}
impl GamepadService {
pub fn start() -> GamepadService {
let pads = Arc::new(Mutex::new(Vec::new()));
let active = Arc::new(Mutex::new(None));
let pinned = Arc::new(Mutex::new(None));
let (ctl, ctl_rx) = std::sync::mpsc::channel();
let (escape_tx, escape_rx) = async_channel::unbounded();
let (p, a, pin) = (pads.clone(), active.clone(), pinned.clone());
if let Err(e) = std::thread::Builder::new()
.name("punktfunk-gamepad".into())
.spawn(move || {
if let Err(e) = run(&p, &a, &pin, &ctl_rx, &escape_tx) {
tracing::warn!(error = %e, "gamepad service ended — pads disabled");
}
})
{
tracing::warn!(error = %e, "gamepad service failed to start");
}
GamepadService {
pads,
active,
pinned,
ctl,
escape_rx,
}
}
/// A receiver that yields one `()` each time the controller escape chord is pressed.
/// A fresh clone per call (shared mpmc channel); the stream page spawns a future on it.
pub fn escape_events(&self) -> async_channel::Receiver<()> {
self.escape_rx.clone()
}
pub fn pads(&self) -> Vec<PadInfo> {
self.pads.lock().unwrap().clone()
}
pub fn active(&self) -> Option<PadInfo> {
self.active.lock().unwrap().clone()
}
pub fn pinned(&self) -> Option<u32> {
*self.pinned.lock().unwrap()
}
pub fn set_pinned(&self, id: Option<u32>) {
let _ = self.ctl.send(Ctl::Pin(id));
}
pub fn attach(&self, connector: Arc<NativeClient>) {
let _ = self.ctl.send(Ctl::Attach(connector));
}
pub fn detach(&self) {
let _ = self.ctl.send(Ctl::Detach);
}
/// What "Automatic" resolves to right now — the virtual pad matching the physical one
/// (Swift parity); no pad connected leaves the host's own default.
pub fn auto_pref(&self) -> GamepadPref {
match self.active() {
Some(p) if p.is_dualsense => GamepadPref::DualSense,
Some(_) => GamepadPref::Xbox360,
None => GamepadPref::Auto,
}
}
}
fn send(connector: &NativeClient, kind: InputKind, code: u32, x: i32) {
let _ = connector.send_input(&InputEvent {
kind,
_pad: [0; 3],
code,
x,
y: 0,
flags: 0, // pad index 0 — single-pad model
});
}
fn button_bit(b: sdl3::gamepad::Button) -> Option<u32> {
use sdl3::gamepad::Button;
Some(match b {
Button::South => wire::BTN_A,
Button::East => wire::BTN_B,
Button::West => wire::BTN_X,
Button::North => wire::BTN_Y,
Button::Back => wire::BTN_BACK,
Button::Start => wire::BTN_START,
Button::Guide => wire::BTN_GUIDE,
Button::LeftStick => wire::BTN_LS_CLICK,
Button::RightStick => wire::BTN_RS_CLICK,
Button::LeftShoulder => wire::BTN_LB,
Button::RightShoulder => wire::BTN_RB,
Button::DPadUp => wire::BTN_DPAD_UP,
Button::DPadDown => wire::BTN_DPAD_DOWN,
Button::DPadLeft => wire::BTN_DPAD_LEFT,
Button::DPadRight => wire::BTN_DPAD_RIGHT,
Button::Touchpad => wire::BTN_TOUCHPAD,
_ => return None,
})
}
/// SDL axis → (wire axis id, wire value). SDL sticks are +y = down; the wire (XInput
/// convention) is +y = up. SDL triggers span 0..32767; the wire wants 0..255.
fn axis_value(axis: sdl3::gamepad::Axis, v: i16) -> (u32, i32) {
use sdl3::gamepad::Axis;
match axis {
Axis::LeftX => (wire::AXIS_LS_X, v as i32),
Axis::LeftY => (wire::AXIS_LS_Y, -(v as i32).max(-32767)),
Axis::RightX => (wire::AXIS_RS_X, v as i32),
Axis::RightY => (wire::AXIS_RS_Y, -(v as i32).max(-32767)),
Axis::TriggerLeft => (wire::AXIS_LT, (v as i32).clamp(0, 32767) >> 7),
Axis::TriggerRight => (wire::AXIS_RT, (v as i32).clamp(0, 32767) >> 7),
}
}
/// The DualSense effects packet (SDL `DS5EffectsState_t`, 47 bytes) — the same layout the
/// host parses off its virtual pad; the wire's 11-byte trigger blocks drop in verbatim.
/// Enable bits select only the fields each update touches, so rumble (driven separately
/// through SDL) and untouched fields keep their state.
#[derive(Default)]
struct Ds5Feedback;
impl Ds5Feedback {
const RIGHT_TRIGGER: usize = 10;
const LEFT_TRIGGER: usize = 21;
const PAD_LIGHTS: usize = 43;
const LED_RGB: usize = 44;
fn trigger_packet(which: u8, effect: &[u8]) -> [u8; 47] {
let mut p = [0u8; 47];
let (flag, off) = if which == 1 {
(0x04, Self::RIGHT_TRIGGER)
} else {
(0x08, Self::LEFT_TRIGGER)
};
p[0] = flag;
let n = effect.len().min(11);
p[off..off + n].copy_from_slice(&effect[..n]);
p
}
fn lightbar_packet(r: u8, g: u8, b: u8) -> [u8; 47] {
let mut p = [0u8; 47];
p[1] = 0x04; // lightbar enable
p[Self::LED_RGB] = r;
p[Self::LED_RGB + 1] = g;
p[Self::LED_RGB + 2] = b;
p
}
fn player_packet(bits: u8) -> [u8; 47] {
let mut p = [0u8; 47];
p[1] = 0x10; // player-LED enable
p[Self::PAD_LIGHTS] = bits & 0x1F;
p
}
}
struct Worker {
subsystem: sdl3::GamepadSubsystem,
opened: HashMap<u32, sdl3::gamepad::Gamepad>,
/// Connection order; the most recently connected is the auto selection.
order: Vec<u32>,
pinned: Option<u32>,
attached: Option<Arc<NativeClient>>,
/// Wire state of the active pad — zeroed on the wire at switch/detach.
last_axis: [i32; 6],
held_buttons: Vec<u32>,
last_accel: [i16; 3],
/// Raises the UI escape signal; the escape chord fires it once per press.
escape_tx: async_channel::Sender<()>,
/// The escape chord is fully held — latched so it fires once, not every poll.
chord_armed: bool,
}
impl Worker {
fn active_id(&self) -> Option<u32> {
self.pinned
.filter(|id| self.opened.contains_key(id))
.or_else(|| self.order.last().copied())
}
fn pad_info(&self, id: u32) -> Option<PadInfo> {
let pad = self.opened.get(&id)?;
Some(PadInfo {
id,
name: pad.name().unwrap_or_else(|| "Controller".into()),
is_dualsense: matches!(
self.subsystem
.type_for_id(sdl3::sys::joystick::SDL_JoystickID(id)),
sdl3::gamepad::GamepadType::PS5
),
})
}
/// Zero everything the host believes is held — on pad switch and detach.
fn flush_held(&mut self) {
if let Some(c) = &self.attached {
for b in self.held_buttons.drain(..) {
send(c, InputKind::GamepadButton, b, 0);
}
for (id, v) in self.last_axis.iter_mut().enumerate() {
if *v != 0 && *v != i32::MIN {
send(c, InputKind::GamepadAxis, id as u32, 0);
}
*v = i32::MIN;
}
} else {
self.held_buttons.clear();
self.last_axis = [i32::MIN; 6];
}
}
/// Raise the UI escape signal when the [`ESCAPE_CHORD`] just completed (latched so it
/// fires once per press). Called after each button-down updates `held_buttons`.
fn maybe_fire_escape(&mut self) {
if self.chord_armed {
return;
}
if ESCAPE_CHORD.iter().all(|b| self.held_buttons.contains(b)) {
self.chord_armed = true;
let _ = self.escape_tx.try_send(());
tracing::info!("gamepad escape chord (L1+R1+Start+Select) — leaving fullscreen");
}
}
/// Re-arm once the chord is broken (any of its buttons released).
fn rearm_escape(&mut self) {
if self.chord_armed && !ESCAPE_CHORD.iter().all(|b| self.held_buttons.contains(b)) {
self.chord_armed = false;
}
}
/// Sensors stream only while a session wants them (they cost USB/BT bandwidth).
fn set_sensors(&mut self, enabled: bool) {
let Some(id) = self.active_id() else { return };
if let Some(pad) = self.opened.get_mut(&id) {
use sdl3::sensor::SensorType;
for s in [SensorType::Gyroscope, SensorType::Accelerometer] {
if unsafe { pad.has_sensor(s) } {
let _ = pad.sensor_set_enabled(s, enabled);
}
}
}
}
}
#[allow(clippy::too_many_lines)]
fn run(
pads_out: &Mutex<Vec<PadInfo>>,
active_out: &Mutex<Option<PadInfo>>,
pinned_out: &Mutex<Option<u32>>,
ctl: &Receiver<Ctl>,
escape_tx: &async_channel::Sender<()>,
) -> Result<(), String> {
// Off-main-thread + no video subsystem: keep SDL away from signals, poll pads on its
// own thread.
sdl3::hint::set("SDL_NO_SIGNAL_HANDLERS", "1");
sdl3::hint::set("SDL_JOYSTICK_THREAD", "1");
let sdl = sdl3::init().map_err(|e| e.to_string())?;
let subsystem = sdl.gamepad().map_err(|e| e.to_string())?;
let mut pump = sdl.event_pump().map_err(|e| e.to_string())?;
let mut w = Worker {
subsystem,
opened: HashMap::new(),
order: Vec::new(),
pinned: None,
attached: None,
last_axis: [i32::MIN; 6],
held_buttons: Vec::new(),
last_accel: [0; 3],
escape_tx: escape_tx.clone(),
chord_armed: false,
};
let publish = |w: &Worker| {
let mut list: Vec<PadInfo> = w.order.iter().filter_map(|&id| w.pad_info(id)).collect();
list.reverse(); // most recent first — the Settings list order
*pads_out.lock().unwrap() = list;
*active_out.lock().unwrap() = w.active_id().and_then(|id| w.pad_info(id));
*pinned_out.lock().unwrap() = w.pinned;
};
loop {
// Control plane from the UI thread.
loop {
match ctl.try_recv() {
Ok(Ctl::Attach(c)) => {
w.attached = Some(c);
w.last_axis = [i32::MIN; 6];
w.set_sensors(true);
}
Ok(Ctl::Detach) => {
w.flush_held();
w.set_sensors(false);
w.attached = None;
}
Ok(Ctl::Pin(id)) => {
let before = w.active_id();
w.pinned = id;
if w.active_id() != before {
w.flush_held();
if w.attached.is_some() {
w.set_sensors(true);
}
}
publish(&w);
}
Err(std::sync::mpsc::TryRecvError::Empty) => break,
Err(std::sync::mpsc::TryRecvError::Disconnected) => return Ok(()), // app gone
}
}
while let Some(event) = pump.poll_event() {
use sdl3::event::Event;
let active = w.active_id();
match event {
Event::ControllerDeviceAdded { which, .. } => {
if !w.opened.contains_key(&which) {
match w.subsystem.open(sdl3::sys::joystick::SDL_JoystickID(which)) {
Ok(pad) => {
tracing::info!(
name = pad.name().unwrap_or_default(),
"gamepad attached"
);
w.opened.insert(which, pad);
w.order.push(which);
if w.attached.is_some() && w.active_id() == Some(which) {
w.set_sensors(true);
}
publish(&w);
}
Err(e) => tracing::warn!(error = %e, "gamepad open failed"),
}
}
}
Event::ControllerDeviceRemoved { which, .. } => {
if w.opened.remove(&which).is_some() {
w.order.retain(|&id| id != which);
if active == Some(which) {
w.flush_held();
}
tracing::info!("gamepad detached");
publish(&w);
}
}
Event::ControllerButtonDown { which, button, .. }
if active == Some(which) && w.attached.is_some() =>
{
if let Some(bit) = button_bit(button) {
w.held_buttons.push(bit);
send(
w.attached.as_ref().unwrap(),
InputKind::GamepadButton,
bit,
1,
);
w.maybe_fire_escape();
}
}
Event::ControllerButtonUp { which, button, .. }
if active == Some(which) && w.attached.is_some() =>
{
if let Some(bit) = button_bit(button) {
w.held_buttons.retain(|&b| b != bit);
send(
w.attached.as_ref().unwrap(),
InputKind::GamepadButton,
bit,
0,
);
w.rearm_escape();
}
}
Event::ControllerAxisMotion {
which, axis, value, ..
} if active == Some(which) && w.attached.is_some() => {
let (id, v) = axis_value(axis, value);
if w.last_axis[id as usize] != v {
w.last_axis[id as usize] = v;
send(w.attached.as_ref().unwrap(), InputKind::GamepadAxis, id, v);
}
}
// DualSense touchpad → the rich-input plane, normalized 0..=65535.
Event::ControllerTouchpadDown {
which,
finger,
x,
y,
..
}
| Event::ControllerTouchpadMotion {
which,
finger,
x,
y,
..
} if active == Some(which) && w.attached.is_some() => {
let _ = w
.attached
.as_ref()
.unwrap()
.send_rich_input(RichInput::Touchpad {
pad: 0,
finger: finger as u8,
active: true,
x: (x.clamp(0.0, 1.0) * 65535.0) as u16,
y: (y.clamp(0.0, 1.0) * 65535.0) as u16,
});
}
Event::ControllerTouchpadUp {
which,
finger,
x,
y,
..
} if active == Some(which) && w.attached.is_some() => {
let _ = w
.attached
.as_ref()
.unwrap()
.send_rich_input(RichInput::Touchpad {
pad: 0,
finger: finger as u8,
active: false,
x: (x.clamp(0.0, 1.0) * 65535.0) as u16,
y: (y.clamp(0.0, 1.0) * 65535.0) as u16,
});
}
// Motion: accel events update the cache; each gyro event ships a sample
// (the DualSense reports both at ~250 Hz). Scale convention shared with
// the Swift client — sign/scale derived, not yet live-verified.
Event::ControllerSensorUpdated {
which,
sensor,
data,
..
} if active == Some(which) && w.attached.is_some() => {
use sdl3::sensor::SensorType;
match sensor {
SensorType::Accelerometer => {
for (i, v) in data.iter().enumerate() {
w.last_accel[i] =
(v / G * ACCEL_LSB_PER_G).clamp(-32768.0, 32767.0) as i16;
}
}
SensorType::Gyroscope => {
let mut gyro = [0i16; 3];
for (i, v) in data.iter().enumerate() {
gyro[i] = (v * GYRO_LSB_PER_RAD_S).clamp(-32768.0, 32767.0) as i16;
}
let _ =
w.attached
.as_ref()
.unwrap()
.send_rich_input(RichInput::Motion {
pad: 0,
gyro,
accel: w.last_accel,
});
}
_ => {}
}
}
_ => {}
}
}
// Feedback planes (this thread is their single consumer). The host re-sends
// rumble state periodically, so a generous duration with refresh-on-update is
// safe — a dropped stop heals within ~500 ms.
if let Some(connector) = w.attached.clone() {
while let Ok((pad, low, high)) = connector.next_rumble(Duration::ZERO) {
if pad == 0 {
if let Some(p) = w.active_id().and_then(|id| w.opened.get_mut(&id)) {
// Surface a failed SDL rumble write: a swallowed error here (DualSense not in
// the right HIDAPI mode, etc.) reads exactly like "rumble doesn't work". The
// host logs the send side on 0xCA, so the two together pinpoint host-game vs
// client-render.
if let Err(e) = p.set_rumble(low, high, 5_000) {
tracing::warn!(low, high, error = %e, "rumble: SDL set_rumble failed");
} else {
tracing::debug!(low, high, "rumble: rendered");
}
} else {
tracing::debug!(low, high, "rumble: received but no active pad to render");
}
}
}
while let Ok(hid) = connector.next_hidout(Duration::ZERO) {
let Some(id) = w.active_id() else { continue };
let is_ds = w.pad_info(id).is_some_and(|p| p.is_dualsense);
let Some(pad) = w.opened.get_mut(&id) else {
continue;
};
match hid {
HidOutput::Led { pad: 0, r, g, b } if is_ds => {
let _ = pad.send_effect(&Ds5Feedback::lightbar_packet(r, g, b));
}
HidOutput::Led { pad: 0, r, g, b } => {
let _ = pad.set_led(r, g, b);
}
HidOutput::PlayerLeds { pad: 0, bits } if is_ds => {
let _ = pad.send_effect(&Ds5Feedback::player_packet(bits));
}
HidOutput::Trigger {
pad: 0,
which,
ref effect,
} if is_ds => {
let _ = pad.send_effect(&Ds5Feedback::trigger_packet(which, effect));
}
_ => {}
}
}
}
std::thread::sleep(Duration::from_millis(if w.attached.is_some() {
2
} else {
30
}));
}
}
+203
View File
@@ -0,0 +1,203 @@
//! Local key/button codes → the punktfunk input wire contract.
//!
//! The wire carries Windows Virtual-Key codes (the GameStream convention; the host maps
//! them back with `inject::vk_to_evdev`). GTK hands us the hardware keycode, which on
//! Wayland (and X11) is the evdev code + 8 — so this table is the exact inverse of the
//! host's, keyed on evdev codes. Layout-independent by construction: positional keys map
//! positionally, exactly what a game expects.
/// Map a Linux evdev key code to the Windows VK code the host expects. `None` = a key the
/// wire contract doesn't cover (media keys etc.) — drop it rather than guess.
pub fn evdev_to_vk(evdev: u16) -> Option<u8> {
Some(match evdev {
// --- Navigation / editing / whitespace ---
14 => 0x08, // KEY_BACKSPACE -> VK_BACK
15 => 0x09, // KEY_TAB -> VK_TAB
28 => 0x0D, // KEY_ENTER -> VK_RETURN
119 => 0x13, // KEY_PAUSE -> VK_PAUSE
58 => 0x14, // KEY_CAPSLOCK -> VK_CAPITAL
1 => 0x1B, // KEY_ESC -> VK_ESCAPE
57 => 0x20, // KEY_SPACE -> VK_SPACE
104 => 0x21, // KEY_PAGEUP -> VK_PRIOR
109 => 0x22, // KEY_PAGEDOWN -> VK_NEXT
107 => 0x23, // KEY_END -> VK_END
102 => 0x24, // KEY_HOME -> VK_HOME
105 => 0x25, // KEY_LEFT -> VK_LEFT
103 => 0x26, // KEY_UP -> VK_UP
106 => 0x27, // KEY_RIGHT -> VK_RIGHT
108 => 0x28, // KEY_DOWN -> VK_DOWN
99 => 0x2C, // KEY_SYSRQ -> VK_SNAPSHOT
110 => 0x2D, // KEY_INSERT -> VK_INSERT
111 => 0x2E, // KEY_DELETE -> VK_DELETE
// --- Digit row (KEY_1..KEY_9 are 2..10, KEY_0 is 11) ---
11 => 0x30,
2 => 0x31,
3 => 0x32,
4 => 0x33,
5 => 0x34,
6 => 0x35,
7 => 0x36,
8 => 0x37,
9 => 0x38,
10 => 0x39,
// --- Letters (evdev order is QWERTY rows, not alphabetical) ---
30 => 0x41, // A
48 => 0x42, // B
46 => 0x43, // C
32 => 0x44, // D
18 => 0x45, // E
33 => 0x46, // F
34 => 0x47, // G
35 => 0x48, // H
23 => 0x49, // I
36 => 0x4A, // J
37 => 0x4B, // K
38 => 0x4C, // L
50 => 0x4D, // M
49 => 0x4E, // N
24 => 0x4F, // O
25 => 0x50, // P
16 => 0x51, // Q
19 => 0x52, // R
31 => 0x53, // S
20 => 0x54, // T
22 => 0x55, // U
47 => 0x56, // V
17 => 0x57, // W
45 => 0x58, // X
21 => 0x59, // Y
44 => 0x5A, // Z
// --- Meta / context-menu ---
125 => 0x5B, // KEY_LEFTMETA -> VK_LWIN
126 => 0x5C, // KEY_RIGHTMETA -> VK_RWIN
127 => 0x5D, // KEY_COMPOSE -> VK_APPS
// --- Numpad ---
82 => 0x60, // KP0
79 => 0x61,
80 => 0x62,
81 => 0x63,
75 => 0x64,
76 => 0x65,
77 => 0x66,
71 => 0x67,
72 => 0x68,
73 => 0x69, // KP9
55 => 0x6A, // KEY_KPASTERISK -> VK_MULTIPLY
78 => 0x6B, // KEY_KPPLUS -> VK_ADD
96 => 0x6C, // KEY_KPENTER -> VK_SEPARATOR
74 => 0x6D, // KEY_KPMINUS -> VK_SUBTRACT
83 => 0x6E, // KEY_KPDOT -> VK_DECIMAL
98 => 0x6F, // KEY_KPSLASH -> VK_DIVIDE
// --- Function keys ---
59 => 0x70, // F1
60 => 0x71,
61 => 0x72,
62 => 0x73,
63 => 0x74,
64 => 0x75,
65 => 0x76,
66 => 0x77,
67 => 0x78,
68 => 0x79, // F10
87 => 0x7A, // F11
88 => 0x7B, // F12
// --- Locks ---
69 => 0x90, // KEY_NUMLOCK -> VK_NUMLOCK
70 => 0x91, // KEY_SCROLLLOCK -> VK_SCROLL
// --- Left/right modifiers (specific VKs; the host maps both generics here too) ---
42 => 0xA0, // KEY_LEFTSHIFT -> VK_LSHIFT
54 => 0xA1, // KEY_RIGHTSHIFT -> VK_RSHIFT
29 => 0xA2, // KEY_LEFTCTRL -> VK_LCONTROL
97 => 0xA3, // KEY_RIGHTCTRL -> VK_RCONTROL
56 => 0xA4, // KEY_LEFTALT -> VK_LMENU
100 => 0xA5, // KEY_RIGHTALT -> VK_RMENU
// --- OEM punctuation (US-layout positions) ---
39 => 0xBA, // KEY_SEMICOLON -> VK_OEM_1
13 => 0xBB, // KEY_EQUAL -> VK_OEM_PLUS
51 => 0xBC, // KEY_COMMA -> VK_OEM_COMMA
12 => 0xBD, // KEY_MINUS -> VK_OEM_MINUS
52 => 0xBE, // KEY_DOT -> VK_OEM_PERIOD
53 => 0xBF, // KEY_SLASH -> VK_OEM_2
41 => 0xC0, // KEY_GRAVE -> VK_OEM_3
26 => 0xDB, // KEY_LEFTBRACE -> VK_OEM_4
43 => 0xDC, // KEY_BACKSLASH -> VK_OEM_5
27 => 0xDD, // KEY_RIGHTBRACE -> VK_OEM_6
40 => 0xDE, // KEY_APOSTROPHE -> VK_OEM_7
86 => 0xE2, // KEY_102ND -> VK_OEM_102
_ => return None,
})
}
/// Map a GTK/GDK mouse button number to the GameStream button id the wire expects
/// (1=left, 2=middle, 3=right, 4=X1, 5=X2). GDK reports back/forward as 8/9.
pub fn gdk_button_to_gs(button: u32) -> Option<u32> {
Some(match button {
1 => 1,
2 => 2,
3 => 3,
8 => 4,
9 => 5,
_ => return None,
})
}
#[cfg(test)]
mod tests {
use super::*;
/// The table must be the exact inverse of the host's `vk_to_evdev` for every key the
/// host knows (modulo the generic-modifier VKs, which collapse onto the same evdev
/// codes as the specific left-hand ones).
#[test]
fn roundtrips_through_the_host_table() {
// Mirror of the host's table (inject::vk_to_evdev), generic modifiers excluded.
let host_pairs: &[(u8, u16)] = &[
(0x08, 14),
(0x09, 15),
(0x0D, 28),
(0x13, 119),
(0x14, 58),
(0x1B, 1),
(0x20, 57),
(0x21, 104),
(0x22, 109),
(0x23, 107),
(0x24, 102),
(0x25, 105),
(0x26, 103),
(0x27, 106),
(0x28, 108),
(0x2C, 99),
(0x2D, 110),
(0x2E, 111),
(0x30, 11),
(0x31, 2),
(0x39, 10),
(0x41, 30),
(0x5A, 44),
(0x5B, 125),
(0x60, 82),
(0x69, 73),
(0x70, 59),
(0x7B, 88),
(0x90, 69),
(0xA0, 42),
(0xA5, 100),
(0xBA, 39),
(0xE2, 86),
];
for &(vk, evdev) in host_pairs {
assert_eq!(evdev_to_vk(evdev), Some(vk), "evdev {evdev}");
}
assert_eq!(evdev_to_vk(113), None); // KEY_MUTE — not in the wire contract
}
}
+42
View File
@@ -0,0 +1,42 @@
//! `punktfunk-client` — the native Linux punktfunk/1 client (design: Option A, 2026-06-12).
//!
//! GTK4/libadwaita shell · `NativeClient` linked as a crate (no C ABI) · FFmpeg decode →
//! `GtkGraphicsOffload` present · PipeWire audio · SDL3 gamepads. The trust surface
//! mirrors the Apple client: persistent identity, TOFU prompt with the host fingerprint,
//! SPAKE2 PIN pairing.
#[cfg(target_os = "linux")]
mod app;
#[cfg(target_os = "linux")]
mod audio;
#[cfg(target_os = "linux")]
mod discovery;
#[cfg(target_os = "linux")]
mod gamepad;
#[cfg(target_os = "linux")]
mod keymap;
#[cfg(target_os = "linux")]
mod session;
#[cfg(target_os = "linux")]
mod trust;
#[cfg(target_os = "linux")]
mod ui_hosts;
#[cfg(target_os = "linux")]
mod ui_settings;
#[cfg(target_os = "linux")]
mod ui_stream;
#[cfg(target_os = "linux")]
mod video;
#[cfg(target_os = "linux")]
fn main() -> gtk::glib::ExitCode {
app::run()
}
/// GTK4/PipeWire/SDL3 are Linux turf; this stub keeps `cargo build --workspace` green on
/// macOS (the Mac client lives in clients/apple).
#[cfg(not(target_os = "linux"))]
fn main() {
eprintln!("punktfunk-client is Linux-only — the macOS client lives in clients/apple");
std::process::exit(2);
}
+265
View File
@@ -0,0 +1,265 @@
//! Session controller: one worker thread runs connect → pump (video pull + decode, audio
//! pull + Opus decode, stats), feeding the GTK main loop over channels. The UI keeps the
//! `Arc<NativeClient>` from the `Connected` event for direct input sends (no extra hop on
//! the input path) — `NativeClient` is `Sync`, planes stay one-consumer-per-thread:
//! video+audio here, rumble+hidout on the gamepad thread.
use crate::audio;
use crate::video::{DecodedFrame, Decoder};
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::{CompositorPref, GamepadPref, Mode};
use punktfunk_core::PunktfunkError;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
pub struct SessionParams {
pub host: String,
pub port: u16,
pub mode: Mode,
pub compositor: CompositorPref,
pub gamepad: GamepadPref,
pub bitrate_kbps: u32,
/// Stream the default microphone to the host's virtual mic source.
pub mic_enabled: bool,
/// Pinned host fingerprint; `None` = trust on first use (caller persists the observed one).
pub pin: Option<[u8; 32]>,
pub identity: (String, String),
}
#[derive(Clone, Copy, Default)]
pub struct Stats {
pub fps: f32,
pub mbps: f32,
pub decode_ms: f32,
/// Median capture→decoded latency over the last window (host-clock corrected).
pub latency_ms: f32,
}
pub enum SessionEvent {
Connected {
connector: Arc<NativeClient>,
mode: Mode,
fingerprint: [u8; 32],
},
/// `trust_rejected` is set when the connect failed the TLS trust check (a `Crypto`
/// error): for a pinned connect this is the fingerprint-changed signal, so the UI can
/// offer a re-pair (PIN) path rather than a dead-end error.
Failed {
msg: String,
trust_rejected: bool,
},
Ended(Option<String>),
Stats(Stats),
}
pub struct SessionHandle {
pub events: async_channel::Receiver<SessionEvent>,
pub frames: async_channel::Receiver<DecodedFrame>,
pub stop: Arc<AtomicBool>,
}
pub fn start(params: SessionParams) -> SessionHandle {
let (ev_tx, ev_rx) = async_channel::unbounded();
// Tiny frame queue, newest wins: force_send displaces the oldest when the UI lags.
let (frame_tx, frame_rx) = async_channel::bounded(2);
let stop = Arc::new(AtomicBool::new(false));
let stop_w = stop.clone();
std::thread::Builder::new()
.name("punktfunk-session".into())
.spawn(move || pump(params, ev_tx, frame_tx, stop_w))
.expect("spawn session thread");
SessionHandle {
events: ev_rx,
frames: frame_rx,
stop,
}
}
fn now_ns() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0)
}
fn pump(
params: SessionParams,
ev_tx: async_channel::Sender<SessionEvent>,
frame_tx: async_channel::Sender<DecodedFrame>,
stop: Arc<AtomicBool>,
) {
let connector = match NativeClient::connect(
&params.host,
params.port,
params.mode,
params.compositor,
params.gamepad,
params.bitrate_kbps,
0, // video_caps: the Linux client has no 10-bit/HDR present path yet
None, // launch: the Linux client has no library picker yet
params.pin,
Some(params.identity),
Duration::from_secs(15),
) {
Ok(c) => Arc::new(c),
Err(e) => {
let trust_rejected = matches!(e, PunktfunkError::Crypto);
let msg = match e {
PunktfunkError::Crypto => {
"Host identity rejected — wrong fingerprint, or the host requires pairing"
.to_string()
}
PunktfunkError::Timeout => "Connection timed out".to_string(),
other => format!("Connect failed: {other:?}"),
};
let _ = ev_tx.send_blocking(SessionEvent::Failed {
msg,
trust_rejected,
});
return;
}
};
let _ = ev_tx.send_blocking(SessionEvent::Connected {
connector: connector.clone(),
mode: connector.mode(),
fingerprint: connector.host_fingerprint,
});
let mut decoder = match Decoder::new() {
Ok(d) => d,
Err(e) => {
let _ = ev_tx.send_blocking(SessionEvent::Ended(Some(format!("video decoder: {e}"))));
return;
}
};
// Audio is best-effort: a session without it still streams. Gamepads are the
// app-lifetime service's job (the UI attaches it on Connected).
let player = audio::AudioPlayer::spawn()
.map_err(|e| tracing::warn!(error = %e, "audio disabled"))
.ok();
let mut opus_dec = opus::Decoder::new(48_000, opus::Channels::Stereo)
.map_err(|e| tracing::warn!(error = %e, "opus decoder failed — audio disabled"))
.ok();
let _mic = params
.mic_enabled
.then(|| {
audio::MicStreamer::spawn(connector.clone())
.map_err(|e| tracing::warn!(error = %e, "mic uplink disabled"))
.ok()
})
.flatten();
let clock_offset = connector.clock_offset_ns;
let mut total_frames = 0u64;
let mut window_start = Instant::now();
let mut frames_n = 0u32;
let mut bytes_n = 0u64;
let mut decode_us_sum = 0u64;
let mut lat_us: Vec<u64> = Vec::with_capacity(256);
let mut pcm = vec![0f32; 5760 * 2]; // decode scratch: max Opus frame (120 ms stereo)
// Loss recovery: watch the host→client unrecoverable-drop count and ask for an IDR when it climbs.
let mut last_dropped = connector.frames_dropped();
let mut last_kf_req: Option<Instant> = None;
let end: Option<String> = loop {
if stop.load(Ordering::SeqCst) {
break None;
}
match connector.next_frame(Duration::from_millis(4)) {
Ok(frame) => {
let t0 = Instant::now();
match decoder.decode(&frame.data) {
Ok(Some(decoded)) => {
total_frames += 1;
if total_frames == 1 {
let (w, h, path) = match &decoded {
DecodedFrame::Cpu(c) => (c.width, c.height, "software"),
DecodedFrame::Dmabuf(d) => (d.width, d.height, "vaapi-dmabuf"),
};
tracing::info!(width = w, height = h, path, "first frame decoded");
}
// Latency: our wall clock expressed in the host's capture clock,
// minus the host-stamped capture pts (same math as client-rs).
let lat = (now_ns() as i128 + clock_offset as i128 - frame.pts_ns as i128)
.max(0) as u64;
if lat > 0 && lat < 10_000_000_000 {
lat_us.push(lat / 1000);
}
decode_us_sum += t0.elapsed().as_micros() as u64;
frames_n += 1;
bytes_n += frame.data.len() as u64;
let _ = frame_tx.force_send(decoded);
}
Ok(None) => {}
// Survivable (loss until the next IDR/RFI recovery) — keep feeding.
Err(e) => tracing::debug!(error = %e, "decode error (recovering)"),
}
}
Err(PunktfunkError::NoFrame) => {}
Err(PunktfunkError::Closed) => break Some("Host ended the session".to_string()),
Err(e) => break Some(format!("session: {e:?}")),
}
// Loss recovery: under infinite GOP the only recovery keyframe is one we request. The
// reassembler drops unrecoverable AUs (frames_dropped); the decoder then conceals the
// reference-missing delta frames that follow and returns Ok, so keying off a decode error
// rarely fires. Request an IDR when the drop count climbs, throttled — the decode stays
// wedged for several frames until the IDR lands, so requesting every frame would flood.
let dropped = connector.frames_dropped();
if dropped > last_dropped {
last_dropped = dropped;
let now = Instant::now();
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!(dropped, "requested keyframe (loss recovery)");
}
}
// Drain audio between frames (packets land every 5 ms; the queue holds 320 ms).
while let Ok(pkt) = connector.next_audio(Duration::ZERO) {
if let (Some(player), Some(dec)) = (&player, opus_dec.as_mut()) {
match dec.decode_float(&pkt.data, &mut pcm, false) {
Ok(samples) => player.push(pcm[..samples * 2].to_vec()),
Err(e) => tracing::debug!(error = %e, "opus decode"),
}
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32();
lat_us.sort_unstable();
let p50 = lat_us.get(lat_us.len() / 2).copied().unwrap_or(0);
tracing::debug!(
fps = frames_n,
lat_p50_us = p50,
total_frames,
"stream window"
);
let _ = ev_tx.try_send(SessionEvent::Stats(Stats {
fps: frames_n as f32 / secs,
mbps: bytes_n as f32 * 8.0 / 1e6 / secs,
decode_ms: if frames_n > 0 {
decode_us_sum as f32 / frames_n as f32 / 1000.0
} else {
0.0
},
latency_ms: p50 as f32 / 1000.0,
}));
window_start = Instant::now();
frames_n = 0;
bytes_n = 0;
decode_us_sum = 0;
lat_us.clear();
}
};
tracing::info!(
total_frames,
reason = end.as_deref().unwrap_or("user"),
"session ended"
);
stop.store(true, Ordering::SeqCst);
let _ = ev_tx.send_blocking(SessionEvent::Ended(end));
}
+164
View File
@@ -0,0 +1,164 @@
//! Client identity, the known-hosts (pinned fingerprint) store, and app settings.
//!
//! The identity shares `~/.config/punktfunk/client-{cert,key}.pem` with `punktfunk-probe`
//! so a box pairs once whichever client it uses.
use anyhow::{anyhow, Context, Result};
use punktfunk_core::quic::endpoint;
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
pub fn config_dir() -> Result<PathBuf> {
let home = std::env::var("HOME").context("HOME unset")?;
Ok(PathBuf::from(home).join(".config/punktfunk"))
}
/// This client's persistent identity, generated on first use — presented on every connect
/// so hosts can recognize it once paired.
pub fn load_or_create_identity() -> Result<(String, String)> {
let dir = config_dir()?;
let (cp, kp) = (dir.join("client-cert.pem"), dir.join("client-key.pem"));
if let (Ok(c), Ok(k)) = (std::fs::read_to_string(&cp), std::fs::read_to_string(&kp)) {
return Ok((c, k));
}
let (c, k) = endpoint::generate_identity().map_err(|e| anyhow!("generate identity: {e}"))?;
std::fs::create_dir_all(&dir)?;
std::fs::write(&cp, &c)?;
std::fs::write(&kp, &k)?;
tracing::info!(cert = %cp.display(), "generated client identity");
Ok((c, k))
}
pub fn hex(fp: &[u8; 32]) -> String {
fp.iter().map(|b| format!("{b:02x}")).collect()
}
pub fn parse_hex32(s: &str) -> Option<[u8; 32]> {
if s.len() != 64 {
return None;
}
let mut out = [0u8; 32];
for (i, b) in out.iter_mut().enumerate() {
*b = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
}
Some(out)
}
/// One trusted host: its pinned certificate fingerprint plus how we got there (TOFU or a
/// PIN ceremony) and where we last reached it.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct KnownHost {
pub name: String,
pub addr: String,
pub port: u16,
/// SHA-256 of the host certificate, lowercase hex — the pin for every later connect.
pub fp_hex: String,
/// True if trust came from the SPAKE2 PIN ceremony (vs. trust-on-first-use).
pub paired: bool,
}
#[derive(Default, Serialize, Deserialize)]
pub struct KnownHosts {
pub hosts: Vec<KnownHost>,
}
impl KnownHosts {
fn path() -> Result<PathBuf> {
Ok(config_dir()?.join("client-known-hosts.json"))
}
pub fn load() -> KnownHosts {
Self::path()
.and_then(|p| Ok(std::fs::read_to_string(p)?))
.ok()
.and_then(|s| serde_json::from_str(&s).ok())
.unwrap_or_default()
}
pub fn save(&self) -> Result<()> {
let p = Self::path()?;
std::fs::create_dir_all(p.parent().unwrap())?;
std::fs::write(&p, serde_json::to_string_pretty(self)?)?;
Ok(())
}
pub fn find_by_fp(&self, fp_hex: &str) -> Option<&KnownHost> {
self.hosts.iter().find(|h| h.fp_hex == fp_hex)
}
pub fn find_by_addr(&self, addr: &str, port: u16) -> Option<&KnownHost> {
self.hosts.iter().find(|h| h.addr == addr && h.port == port)
}
/// Insert or refresh an entry, keyed by fingerprint. `paired` only ever upgrades
/// (a later TOFU connect must not demote a PIN-paired host).
pub fn upsert(&mut self, entry: KnownHost) {
if let Some(h) = self.hosts.iter_mut().find(|h| h.fp_hex == entry.fp_hex) {
h.name = entry.name;
h.addr = entry.addr;
h.port = entry.port;
h.paired |= entry.paired;
} else {
self.hosts.push(entry);
}
}
}
/// App settings, persisted as JSON. Stringly-typed gamepad/compositor prefs so the file
/// stays readable; parsed with `*Pref::from_name` at connect time.
#[derive(Clone, Serialize, Deserialize)]
#[serde(default)]
pub struct Settings {
/// Stream mode; `0` = the native size/refresh of the monitor the window is on,
/// resolved at connect time.
pub width: u32,
pub height: u32,
pub refresh_hz: u32,
/// Requested encoder bitrate (kbps); 0 = host default.
pub bitrate_kbps: u32,
pub gamepad: String,
/// Which host compositor backend to request (advisory; the host falls back to
/// auto-detect when unavailable).
pub compositor: String,
/// Grab compositor shortcuts (Alt+Tab, Super…) while input is captured.
pub inhibit_shortcuts: bool,
/// Stream the default microphone to the host's virtual mic source.
pub mic_enabled: bool,
}
impl Default for Settings {
fn default() -> Self {
Settings {
width: 0,
height: 0,
refresh_hz: 0,
bitrate_kbps: 0,
gamepad: "auto".into(),
compositor: "auto".into(),
inhibit_shortcuts: true,
mic_enabled: false,
}
}
}
impl Settings {
fn path() -> Result<PathBuf> {
Ok(config_dir()?.join("client-gtk-settings.json"))
}
pub fn load() -> Settings {
Self::path()
.and_then(|p| Ok(std::fs::read_to_string(p)?))
.ok()
.and_then(|s| serde_json::from_str(&s).ok())
.unwrap_or_default()
}
pub fn save(&self) {
let Ok(p) = Self::path() else { return };
let _ = std::fs::create_dir_all(p.parent().unwrap());
if let Ok(s) = serde_json::to_string_pretty(self) {
let _ = std::fs::write(&p, s);
}
}
}
+247
View File
@@ -0,0 +1,247 @@
//! The hosts page: saved (trusted) hosts, live mDNS discovery, manual connect entry.
use crate::discovery::{self, DiscoveredHost};
use crate::trust::KnownHosts;
use adw::prelude::*;
use gtk::glib;
use std::cell::RefCell;
use std::collections::HashMap;
use std::rc::Rc;
/// What the user asked to connect to. `fp_hex` comes from the mDNS TXT record when the
/// host was discovered (drives the trust decision *before* connecting); manual entries have
/// none. `pair_optional` is true ONLY when a discovered host advertised `pair=optional`,
/// which is the sole case in which the reduced-security TOFU path may be offered — every
/// other case (pair=required, unknown/empty policy, manual entry) mandates PIN pairing.
#[derive(Clone, Debug)]
pub struct ConnectRequest {
pub name: String,
pub addr: String,
pub port: u16,
pub fp_hex: Option<String>,
pub pair_optional: bool,
}
pub fn new(
on_connect: Rc<dyn Fn(ConnectRequest)>,
on_settings: Rc<dyn Fn()>,
on_speed_test: Rc<dyn Fn(ConnectRequest)>,
) -> adw::NavigationPage {
let list = gtk::ListBox::new();
list.add_css_class("boxed-list");
list.set_selection_mode(gtk::SelectionMode::None);
let placeholder = gtk::Label::new(Some("Searching the LAN for hosts…"));
placeholder.add_css_class("dim-label");
placeholder.set_margin_top(24);
placeholder.set_margin_bottom(24);
list.set_placeholder(Some(&placeholder));
// key → (row, latest advert); the activation closure looks the advert up by key so
// re-adverts (new address, pairing flipped) take effect without rebuilding rows.
type Rows = Rc<RefCell<HashMap<String, (adw::ActionRow, DiscoveredHost)>>>;
let rows: Rows = Rc::new(RefCell::new(HashMap::new()));
{
let rx = discovery::browse();
let rows = rows.clone();
let list = list.downgrade();
let on_connect = on_connect.clone();
glib::spawn_future_local(async move {
while let Ok(host) = rx.recv().await {
let Some(list) = list.upgrade() else { break };
let mut map = rows.borrow_mut();
let subtitle = format!(
"{}:{} · pairing {}",
host.addr,
host.port,
if host.pair.is_empty() {
"optional"
} else {
&host.pair
}
);
if let Some((row, stored)) = map.get_mut(&host.key) {
row.set_title(&host.name);
row.set_subtitle(&subtitle);
*stored = host;
} else {
let row = adw::ActionRow::builder()
.title(&host.name)
.subtitle(&subtitle)
.activatable(true)
.build();
row.add_suffix(&gtk::Image::from_icon_name("go-next-symbolic"));
{
let rows = rows.clone();
let key = host.key.clone();
let on_connect = on_connect.clone();
row.connect_activated(move |_| {
if let Some((_, h)) = rows.borrow().get(&key) {
on_connect(ConnectRequest {
name: h.name.clone(),
addr: h.addr.clone(),
port: h.port,
fp_hex: (!h.fp_hex.is_empty()).then(|| h.fp_hex.clone()),
// TOFU is offered only when the host explicitly opts in
// with pair=optional; required/empty means mandatory PIN.
pair_optional: h.pair == "optional",
});
}
});
}
list.append(&row);
map.insert(host.key.clone(), (row, host));
}
}
});
}
// Manual connect: host:port (punktfunk/1 default port 9777).
let manual = adw::EntryRow::builder().title("host:port").build();
let connect_btn = gtk::Button::with_label("Connect");
connect_btn.set_valign(gtk::Align::Center);
connect_btn.add_css_class("suggested-action");
manual.add_suffix(&connect_btn);
let submit = {
let manual = manual.clone();
let on_connect = on_connect.clone();
move || {
let text = manual.text().to_string();
let text = text.trim();
if text.is_empty() {
return;
}
let (addr, port) = match text.rsplit_once(':') {
Some((a, p)) => match p.parse::<u16>() {
Ok(port) => (a.to_string(), port),
Err(_) => return,
},
None => (text.to_string(), 9777),
};
on_connect(ConnectRequest {
name: addr.clone(),
addr,
port,
fp_hex: None,
// Manual entry carries no advertised policy — never eligible for TOFU.
pair_optional: false,
});
}
};
{
let submit = submit.clone();
connect_btn.connect_clicked(move |_| submit());
}
manual.connect_entry_activated(move |_| submit());
let manual_list = gtk::ListBox::new();
manual_list.add_css_class("boxed-list");
manual_list.set_selection_mode(gtk::SelectionMode::None);
manual_list.append(&manual);
// Saved (trusted/paired) hosts — reachable even when mDNS isn't. Rebuilt every time
// the page is shown, so fresh TOFU/pairing entries appear on return.
let saved_label = gtk::Label::new(Some("Saved hosts"));
saved_label.add_css_class("heading");
saved_label.set_halign(gtk::Align::Start);
let saved_list = gtk::ListBox::new();
saved_list.add_css_class("boxed-list");
saved_list.set_selection_mode(gtk::SelectionMode::None);
let rebuild_saved = {
let saved_list = saved_list.clone();
let saved_label = saved_label.clone();
let on_connect = on_connect.clone();
let on_speed_test = on_speed_test.clone();
move || {
saved_list.remove_all();
let known = KnownHosts::load();
saved_label.set_visible(!known.hosts.is_empty());
saved_list.set_visible(!known.hosts.is_empty());
for k in &known.hosts {
let row = adw::ActionRow::builder()
.title(&k.name)
.subtitle(format!(
"{}:{}{}",
k.addr,
k.port,
if k.paired {
" · paired"
} else {
" · trusted"
}
))
.activatable(true)
.build();
let req = ConnectRequest {
name: k.name.clone(),
addr: k.addr.clone(),
port: k.port,
fp_hex: Some(k.fp_hex.clone()),
// Saved host: its fp is already pinned, so this routes to a silent
// pinned connect; TOFU eligibility is irrelevant.
pair_optional: false,
};
let speed_btn = gtk::Button::from_icon_name("network-transmit-receive-symbolic");
speed_btn.set_tooltip_text(Some("Test network speed"));
speed_btn.set_valign(gtk::Align::Center);
speed_btn.add_css_class("flat");
{
let on_speed_test = on_speed_test.clone();
let req = req.clone();
speed_btn.connect_clicked(move |_| on_speed_test(req.clone()));
}
row.add_suffix(&speed_btn);
row.add_suffix(&gtk::Image::from_icon_name("go-next-symbolic"));
let on_connect = on_connect.clone();
row.connect_activated(move |_| on_connect(req.clone()));
saved_list.append(&row);
}
}
};
rebuild_saved();
let content = gtk::Box::new(gtk::Orientation::Vertical, 18);
content.set_margin_top(24);
content.set_margin_bottom(24);
content.set_margin_start(12);
content.set_margin_end(12);
content.append(&saved_label);
content.append(&saved_list);
let discovered_label = gtk::Label::new(Some("Hosts on this network"));
discovered_label.add_css_class("heading");
discovered_label.set_halign(gtk::Align::Start);
content.append(&discovered_label);
content.append(&list);
let manual_label = gtk::Label::new(Some("Manual connection"));
manual_label.add_css_class("heading");
manual_label.set_halign(gtk::Align::Start);
content.append(&manual_label);
content.append(&manual_list);
let clamp = adw::Clamp::builder()
.maximum_size(560)
.child(&content)
.build();
let scrolled = gtk::ScrolledWindow::builder()
.hscrollbar_policy(gtk::PolicyType::Never)
.child(&clamp)
.build();
let header = adw::HeaderBar::new();
let settings_btn = gtk::Button::from_icon_name("preferences-system-symbolic");
settings_btn.set_tooltip_text(Some("Preferences"));
settings_btn.connect_clicked(move |_| on_settings());
header.pack_end(&settings_btn);
let toolbar = adw::ToolbarView::new();
toolbar.add_top_bar(&header);
toolbar.set_content(Some(&scrolled));
let page = adw::NavigationPage::builder()
.title("Punktfunk")
.tag("hosts")
.child(&toolbar)
.build();
page.connect_shown(move |_| rebuild_saved());
page
}
+189
View File
@@ -0,0 +1,189 @@
//! Preferences dialog: stream mode, bitrate, host compositor, gamepad type, microphone,
//! capture behavior. Written back to disk when the dialog closes.
use crate::trust::Settings;
use adw::prelude::*;
use std::cell::RefCell;
use std::rc::Rc;
/// `(0, 0)` = the native size of the monitor the window is on, resolved at connect.
const RESOLUTIONS: &[(u32, u32)] = &[
(0, 0),
(1280, 720),
(1920, 1080),
(2560, 1440),
(3840, 2160),
];
/// `0` = the monitor's native refresh, resolved at connect.
const REFRESH: &[u32] = &[0, 30, 60, 90, 120, 144, 165, 240];
const GAMEPADS: &[&str] = &["auto", "xbox360", "dualsense"];
const COMPOSITORS: &[&str] = &["auto", "kwin", "wlroots", "mutter", "gamescope"];
pub fn show(
parent: &impl IsA<gtk::Widget>,
settings: Rc<RefCell<Settings>>,
gamepads: &crate::gamepad::GamepadService,
) {
let page = adw::PreferencesPage::new();
let stream = adw::PreferencesGroup::builder().title("Stream").build();
let res_names: Vec<String> = RESOLUTIONS
.iter()
.map(|&(w, h)| {
if w == 0 {
"Native display".to_string()
} else {
format!("{w} × {h}")
}
})
.collect();
let res_row = adw::ComboRow::builder()
.title("Resolution")
.subtitle("The host creates a virtual output at exactly this size")
.model(&gtk::StringList::new(
&res_names.iter().map(String::as_str).collect::<Vec<_>>(),
))
.build();
let hz_names: Vec<String> = REFRESH
.iter()
.map(|&r| {
if r == 0 {
"Native".to_string()
} else {
format!("{r} Hz")
}
})
.collect();
let hz_row = adw::ComboRow::builder()
.title("Refresh rate")
.model(&gtk::StringList::new(
&hz_names.iter().map(String::as_str).collect::<Vec<_>>(),
))
.build();
let bitrate_row = adw::SpinRow::with_range(0.0, 3000.0, 5.0);
bitrate_row.set_title("Bitrate");
bitrate_row.set_subtitle("Mbit/s · 0 = host default · run a speed test before going high");
let compositor_row = adw::ComboRow::builder()
.title("Host compositor")
.subtitle("Advisory — the host falls back to auto-detect when unavailable")
.model(&gtk::StringList::new(&[
"Automatic",
"KWin",
"wlroots (Sway/Hyprland)",
"Mutter (GNOME)",
"gamescope",
]))
.build();
stream.add(&res_row);
stream.add(&hz_row);
stream.add(&bitrate_row);
stream.add(&compositor_row);
let input = adw::PreferencesGroup::builder().title("Input").build();
// Which physical controller forwards as pad 0: automatic = the most recently
// connected; pinning survives until the app exits (Swift parity).
let pads = gamepads.pads();
let mut pad_names = vec!["Automatic (most recent)".to_string()];
pad_names.extend(pads.iter().map(|p| {
if p.is_dualsense {
format!("{} · DualSense", p.name)
} else {
p.name.clone()
}
}));
let forward_row = adw::ComboRow::builder()
.title("Forwarded controller")
.subtitle(if pads.is_empty() {
"No controllers detected"
} else {
"Exactly one controller is forwarded to the host"
})
.model(&gtk::StringList::new(
&pad_names.iter().map(String::as_str).collect::<Vec<_>>(),
))
.build();
let pinned_i = gamepads
.pinned()
.and_then(|id| pads.iter().position(|p| p.id == id))
.map_or(0, |i| i + 1);
forward_row.set_selected(pinned_i as u32);
{
let svc = gamepads.clone();
let ids: Vec<u32> = pads.iter().map(|p| p.id).collect();
forward_row.connect_selected_notify(move |row| {
let sel = row.selected() as usize;
svc.set_pinned(if sel == 0 {
None
} else {
ids.get(sel - 1).copied()
});
});
}
let pad_row = adw::ComboRow::builder()
.title("Gamepad type")
.subtitle("The virtual pad the host creates — Automatic matches the physical pad")
.model(&gtk::StringList::new(&[
"Automatic",
"Xbox 360",
"DualSense",
]))
.build();
let inhibit_row = adw::SwitchRow::builder()
.title("Capture system shortcuts")
.subtitle("Forward Alt+Tab, Super, … to the host while input is captured")
.build();
input.add(&forward_row);
input.add(&pad_row);
input.add(&inhibit_row);
let audio = adw::PreferencesGroup::builder().title("Audio").build();
let mic_row = adw::SwitchRow::builder()
.title("Stream microphone")
.subtitle("Send the default input device to the host's virtual microphone")
.build();
audio.add(&mic_row);
page.add(&stream);
page.add(&input);
page.add(&audio);
// Seed from the current settings.
{
let s = settings.borrow();
let res_i = RESOLUTIONS
.iter()
.position(|&(w, h)| w == s.width && h == s.height)
.unwrap_or(0);
res_row.set_selected(res_i as u32);
let hz_i = REFRESH.iter().position(|&r| r == s.refresh_hz).unwrap_or(0);
hz_row.set_selected(hz_i as u32);
bitrate_row.set_value(f64::from(s.bitrate_kbps) / 1000.0);
let pad_i = GAMEPADS.iter().position(|&g| g == s.gamepad).unwrap_or(0);
pad_row.set_selected(pad_i as u32);
let comp_i = COMPOSITORS
.iter()
.position(|&c| c == s.compositor)
.unwrap_or(0);
compositor_row.set_selected(comp_i as u32);
inhibit_row.set_active(s.inhibit_shortcuts);
mic_row.set_active(s.mic_enabled);
}
let dialog = adw::PreferencesDialog::new();
dialog.set_title("Preferences");
dialog.add(&page);
dialog.connect_closed(move |_| {
let mut s = settings.borrow_mut();
let (w, h) = RESOLUTIONS[(res_row.selected() as usize).min(RESOLUTIONS.len() - 1)];
(s.width, s.height) = (w, h);
s.refresh_hz = REFRESH[(hz_row.selected() as usize).min(REFRESH.len() - 1)];
s.bitrate_kbps = (bitrate_row.value() * 1000.0) as u32;
s.gamepad = GAMEPADS[(pad_row.selected() as usize).min(GAMEPADS.len() - 1)].to_string();
s.compositor = COMPOSITORS[(compositor_row.selected() as usize).min(COMPOSITORS.len() - 1)]
.to_string();
s.inhibit_shortcuts = inhibit_row.is_active();
s.mic_enabled = mic_row.is_active();
s.save();
});
dialog.present(Some(parent));
}
+469
View File
@@ -0,0 +1,469 @@
//! The stream page: decoded frames into a `GtkGraphicsOffload`-wrapped picture, local
//! input captured and forwarded on the wire contract.
//!
//! Input capture is a deliberate, reversible STATE (Moonlight-style, mirroring the Swift
//! client): engaged when the stream starts and when the user clicks into the video (that
//! click is suppressed toward the host); released by Ctrl+Alt+Shift+Q (toggles) or focus
//! loss — held keys/buttons are flushed host-side on release so nothing sticks down.
//! While captured the local cursor is hidden (the host renders its own) and compositor
//! shortcuts are inhibited (configurable); while released nothing is forwarded and the
//! HUD says how to recapture.
//!
//! Keys are hardware keycodes (evdev + 8 on Wayland) → VK via `keymap`, layout-
//! independent. Mouse is absolute (`MouseMoveAbs` scaled into the negotiated mode through
//! the letterbox transform, surface size packed in `flags`) — pointer-lock relative
//! capture is the stage-2 presenter's job. F11 toggles fullscreen locally.
use crate::keymap;
use crate::session::Stats;
use crate::video::DecodedFrame;
use adw::prelude::*;
use gtk::{gdk, glib};
use punktfunk_core::client::NativeClient;
use punktfunk_core::input::{InputEvent, InputKind};
use std::cell::{Cell, RefCell};
use std::collections::HashSet;
use std::rc::Rc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
pub struct StreamPage {
pub page: adw::NavigationPage,
stats_label: gtk::Label,
}
impl StreamPage {
pub fn update_stats(&self, s: Stats) {
self.stats_label.set_text(&format!(
"{:.0} fps · {:.1} Mbit/s · dec {:.1} ms · lat {:.1} ms",
s.fps, s.mbps, s.decode_ms, s.latency_ms
));
}
}
fn send(connector: &NativeClient, kind: InputKind, code: u32, x: i32, y: i32, flags: u32) {
let _ = connector.send_input(&InputEvent {
kind,
_pad: [0; 3],
code,
x,
y,
flags,
});
}
/// Forward an absolute pointer position: widget coordinates → video pixels through the
/// Contain-fit letterbox. `flags` packs the coordinate-space size (`(w << 16) | h`, the
/// same contract as touch) — the host normalizes against it before mapping into the EIS
/// region; without it the event is dropped.
fn send_abs(widget: &impl IsA<gtk::Widget>, connector: &NativeClient, x: f64, y: f64) {
let w = widget.as_ref();
let mode = connector.mode();
let (ww, wh) = (w.width().max(1) as f64, w.height().max(1) as f64);
let (vw, vh) = (mode.width.max(1) as f64, mode.height.max(1) as f64);
let scale = (ww / vw).min(wh / vh);
let (ox, oy) = ((ww - vw * scale) / 2.0, (wh - vh * scale) / 2.0);
let px = (((x - ox) / scale).round()).clamp(0.0, vw - 1.0) as i32;
let py = (((y - oy) / scale).round()).clamp(0.0, vh - 1.0) as i32;
let flags = (mode.width << 16) | (mode.height & 0xffff);
send(connector, InputKind::MouseMoveAbs, 0, px, py, flags);
}
/// The capture state shared by every input controller on the page.
struct Capture {
connector: Arc<NativeClient>,
window: adw::ApplicationWindow,
overlay: gtk::Overlay,
hint: gtk::Label,
inhibit_shortcuts: bool,
captured: Cell<bool>,
/// VKs / GameStream button ids currently held — flushed up on release.
held_keys: RefCell<HashSet<u8>>,
held_buttons: RefCell<HashSet<u32>>,
}
impl Capture {
fn engage(&self) {
if self.captured.replace(true) {
return;
}
self.overlay
.set_cursor(gdk::Cursor::from_name("none", None).as_ref());
self.hint.set_visible(false);
if self.inhibit_shortcuts {
if let Some(tl) = self
.window
.surface()
.and_then(|s| s.downcast::<gdk::Toplevel>().ok())
{
tl.inhibit_system_shortcuts(None::<&gdk::Event>);
}
}
}
fn release(&self) {
if !self.captured.replace(false) {
return;
}
self.overlay.set_cursor(None);
self.hint.set_visible(true);
if let Some(tl) = self
.window
.surface()
.and_then(|s| s.downcast::<gdk::Toplevel>().ok())
{
tl.restore_system_shortcuts();
}
// Flush everything held so nothing sticks down on the host.
for vk in self.held_keys.borrow_mut().drain() {
send(&self.connector, InputKind::KeyUp, vk as u32, 0, 0, 0);
}
for b in self.held_buttons.borrow_mut().drain() {
send(&self.connector, InputKind::MouseButtonUp, b, 0, 0, 0);
}
}
}
#[allow(clippy::too_many_lines)]
pub fn new(
window: &adw::ApplicationWindow,
connector: Arc<NativeClient>,
frames: async_channel::Receiver<DecodedFrame>,
escape_rx: async_channel::Receiver<()>,
stop: Arc<AtomicBool>,
inhibit_shortcuts: bool,
title: &str,
) -> StreamPage {
let picture = gtk::Picture::new();
picture.set_content_fit(gtk::ContentFit::Contain);
// The offload path: with a dmabuf-backed texture (stage 1.5) this becomes a
// subsurface the compositor can scan out directly; with memory textures it is a
// no-op wrapper. Black letterboxing keeps fullscreen scanout-eligible.
let offload = gtk::GraphicsOffload::new(Some(&picture));
offload.set_black_background(true);
let stats_label = gtk::Label::new(None);
stats_label.add_css_class("osd");
stats_label.add_css_class("numeric");
stats_label.set_halign(gtk::Align::Start);
stats_label.set_valign(gtk::Align::Start);
stats_label.set_margin_start(12);
stats_label.set_margin_top(12);
let hint = gtk::Label::new(Some(
"Click the stream to capture input · Ctrl+Alt+Shift+Q releases",
));
hint.add_css_class("osd");
hint.set_halign(gtk::Align::Center);
hint.set_valign(gtk::Align::End);
hint.set_margin_bottom(24);
hint.set_visible(false);
// Flashed when entering fullscreen — the only exit affordances once the header bar is
// hidden (F11 on a keyboard; the L1+R1+Start+Select chord on a controller, which is the
// only way out on a Steam Deck).
let fs_hint = gtk::Label::new(Some("F11 · L1 + R1 + Start + Select — exit fullscreen"));
fs_hint.add_css_class("osd");
fs_hint.set_halign(gtk::Align::Center);
fs_hint.set_valign(gtk::Align::Start);
fs_hint.set_margin_top(12);
fs_hint.set_visible(false);
let overlay = gtk::Overlay::new();
overlay.set_child(Some(&offload));
overlay.add_overlay(&stats_label);
overlay.add_overlay(&hint);
overlay.add_overlay(&fs_hint);
overlay.set_focusable(true);
let capture = Rc::new(Capture {
connector: connector.clone(),
window: window.clone(),
overlay: overlay.clone(),
hint: hint.clone(),
inhibit_shortcuts,
captured: Cell::new(false),
held_keys: RefCell::new(HashSet::new()),
held_buttons: RefCell::new(HashSet::new()),
});
let header = adw::HeaderBar::new();
let fullscreen_btn = gtk::Button::from_icon_name("view-fullscreen-symbolic");
fullscreen_btn.set_tooltip_text(Some("Fullscreen (F11)"));
{
let window = window.clone();
fullscreen_btn.connect_clicked(move |_| {
if window.is_fullscreen() {
window.unfullscreen();
} else {
window.fullscreen();
}
});
}
header.pack_end(&fullscreen_btn);
let toolbar = adw::ToolbarView::new();
toolbar.add_top_bar(&header);
toolbar.set_content(Some(&overlay));
// Fullscreen = the stream and nothing else. (Window handlers are disconnected when
// the page dies — the window outlives every session.)
let fs_handler = {
let toolbar = toolbar.clone();
let fs_hint = fs_hint.clone();
window.connect_fullscreened_notify(move |w| {
let fs = w.is_fullscreen();
toolbar.set_reveal_top_bars(!fs);
if fs {
fs_hint.set_visible(true);
let fs_hint = fs_hint.clone();
glib::timeout_add_seconds_local_once(4, move || fs_hint.set_visible(false));
} else {
fs_hint.set_visible(false);
}
})
};
let page = adw::NavigationPage::builder()
.title(title)
.tag("stream")
.child(&toolbar)
.build();
// --- Frame consumer: newest texture wins, set on the GTK frame clock's cadence. ---
{
let picture = picture.downgrade();
// The host encodes BT.709 limited-range; without an explicit color state GDK
// would convert NV12 dmabufs with the (BT.601) dmabuf default.
let rec709 = {
let cicp = gdk::CicpParams::new();
cicp.set_color_primaries(1);
cicp.set_transfer_function(1);
cicp.set_matrix_coefficients(1);
cicp.set_range(gdk::CicpRange::Narrow);
cicp.build_color_state().ok()
};
glib::spawn_future_local(async move {
while let Ok(f) = frames.recv().await {
let Some(picture) = picture.upgrade() else {
break;
};
match f {
DecodedFrame::Cpu(c) => {
let bytes = glib::Bytes::from_owned(c.rgba);
let tex = gdk::MemoryTexture::new(
c.width as i32,
c.height as i32,
gdk::MemoryFormat::R8g8b8a8,
&bytes,
c.stride,
);
picture.set_paintable(Some(&tex));
}
DecodedFrame::Dmabuf(d) => {
let mut b = gdk::DmabufTextureBuilder::new()
.set_display(&picture.display())
.set_width(d.width)
.set_height(d.height)
.set_fourcc(d.fourcc)
.set_modifier(d.modifier)
.set_n_planes(d.planes.len() as u32)
.set_color_state(rec709.as_ref());
for (i, p) in d.planes.iter().enumerate() {
b = unsafe { b.set_fd(i as u32, p.fd) }
.set_offset(i as u32, p.offset)
.set_stride(i as u32, p.stride);
}
let guard = d.guard;
// GDK runs the release func whether the import succeeds or not.
match unsafe { b.build_with_release_func(move || drop(guard)) } {
Ok(tex) => picture.set_paintable(Some(&tex)),
Err(e) => {
// Import rejected (format/modifier) — surfaces once per
// session in practice; the stream continues on the next
// frame, and PUNKTFUNK_DECODER=software is the escape.
tracing::warn!(error = %e, "dmabuf texture import failed");
}
}
}
}
}
});
}
// --- Keyboard ---
{
let key = gtk::EventControllerKey::new();
key.set_propagation_phase(gtk::PropagationPhase::Capture);
let cap = capture.clone();
let window_k = window.clone();
key.connect_key_pressed(move |_, keyval, keycode, state| {
let chord = gdk::ModifierType::CONTROL_MASK
| gdk::ModifierType::ALT_MASK
| gdk::ModifierType::SHIFT_MASK;
if state.contains(chord) && keyval.to_lower() == gdk::Key::q {
if cap.captured.get() {
cap.release();
} else {
cap.engage();
}
return glib::Propagation::Stop;
}
if keyval == gdk::Key::F11 {
if window_k.is_fullscreen() {
window_k.unfullscreen();
} else {
window_k.fullscreen();
}
return glib::Propagation::Stop;
}
if !cap.captured.get() {
return glib::Propagation::Proceed;
}
if let Some(vk) = keycode
.checked_sub(8)
.and_then(|c| keymap::evdev_to_vk(c as u16))
{
cap.held_keys.borrow_mut().insert(vk);
send(&cap.connector, InputKind::KeyDown, vk as u32, 0, 0, 0);
}
glib::Propagation::Stop
});
let cap = capture.clone();
key.connect_key_released(move |_, _keyval, keycode, _state| {
if let Some(vk) = keycode
.checked_sub(8)
.and_then(|c| keymap::evdev_to_vk(c as u16))
{
// Flush-on-release may have beaten us to it — only forward if still held.
if cap.held_keys.borrow_mut().remove(&vk) {
send(&cap.connector, InputKind::KeyUp, vk as u32, 0, 0, 0);
}
}
});
overlay.add_controller(key);
}
// --- Mouse: absolute motion, buttons, wheel — forwarded only while captured ---
{
let motion = gtk::EventControllerMotion::new();
let cap = capture.clone();
motion.connect_motion(move |_, x, y| {
if cap.captured.get() {
send_abs(&cap.overlay, &cap.connector, x, y);
}
});
overlay.add_controller(motion);
}
{
let click = gtk::GestureClick::builder().button(0).build();
let cap = capture.clone();
click.connect_pressed(move |g, _n, x, y| {
cap.overlay.grab_focus();
if !cap.captured.get() {
cap.engage(); // the engaging click is suppressed toward the host
return;
}
send_abs(&cap.overlay, &cap.connector, x, y);
if let Some(gs) = keymap::gdk_button_to_gs(g.current_button()) {
cap.held_buttons.borrow_mut().insert(gs);
send(&cap.connector, InputKind::MouseButtonDown, gs, 0, 0, 0);
}
});
let cap = capture.clone();
click.connect_released(move |g, _n, _x, _y| {
if let Some(gs) = keymap::gdk_button_to_gs(g.current_button()) {
if cap.held_buttons.borrow_mut().remove(&gs) {
send(&cap.connector, InputKind::MouseButtonUp, gs, 0, 0, 0);
}
}
});
overlay.add_controller(click);
}
{
let scroll = gtk::EventControllerScroll::new(gtk::EventControllerScrollFlags::BOTH_AXES);
let cap = capture.clone();
scroll.connect_scroll(move |_, dx, dy| {
if !cap.captured.get() {
return glib::Propagation::Proceed;
}
// The wire carries WHEEL_DELTA(120) units, positive = up / right; GTK's dy is
// positive = down. Smooth fractions survive — libei's discrete scroll is
// 120-based too.
let vy = (-dy * 120.0) as i32;
if vy != 0 {
send(&cap.connector, InputKind::MouseScroll, 0, vy, 0, 0);
}
let vx = (dx * 120.0) as i32;
if vx != 0 {
send(&cap.connector, InputKind::MouseScroll, 1, vx, 0, 0);
}
glib::Propagation::Stop
});
overlay.add_controller(scroll);
}
// --- Capture lifecycle ---
{
// Engaged when the stream starts (trust is already confirmed by then).
let cap = capture.clone();
overlay.connect_map(move |w| {
w.grab_focus();
cap.engage();
});
}
// Focus loss releases (Alt-Tab away, another window) — Swift does the same.
let active_handler = {
let cap = capture.clone();
window.connect_is_active_notify(move |w| {
if !w.is_active() {
cap.release();
}
})
};
{
let cap = capture.clone();
overlay.connect_unmap(move |_| cap.release());
}
// Controller escape chord (gamepad service) → leave fullscreen + release capture. The
// chord is the only fullscreen exit a controller has (no F11 key; fullscreen hides the
// chrome). Aborted on page-hidden so a stale future can't act on the shared window.
let escape_future = {
let window = window.clone();
let cap = capture.clone();
glib::spawn_future_local(async move {
while escape_rx.recv().await.is_ok() {
if window.is_fullscreen() {
window.unfullscreen();
}
cap.release();
}
})
};
// The page's `hidden` fires once navigation away completes (back button, pop on
// session end) — NOT on the transient unmap/map cycle a NavigationView push performs.
{
let window = window.clone();
let stop_h = stop.clone();
let handlers = RefCell::new(Some((fs_handler, active_handler)));
let escape_future = RefCell::new(Some(escape_future));
page.connect_hidden(move |_| {
tracing::debug!("stream page hidden — ending session");
if let Some((fs, active)) = handlers.borrow_mut().take() {
window.disconnect(fs);
window.disconnect(active);
}
if let Some(f) = escape_future.borrow_mut().take() {
f.abort();
}
if window.is_fullscreen() {
window.unfullscreen();
}
stop_h.store(true, Ordering::SeqCst);
});
}
StreamPage { page, stats_label }
}
+443
View File
@@ -0,0 +1,443 @@
//! Video decode: reassembled HEVC access units → frames for the GTK presenter.
//!
//! Two backends, picked at session start (override: `PUNKTFUNK_DECODER=software|vaapi`):
//!
//! * **VAAPI** (Intel/AMD): libavcodec hwaccel decodes on the GPU; each frame is mapped
//! to a DRM-PRIME dmabuf (`av_hwframe_map`, zero copy) and handed to the UI as fds +
//! plane layout for `GdkDmabufTextureBuilder` — inside `GtkGraphicsOffload` that is the
//! decoder-to-subsurface path, direct-scanout eligible when fullscreen. NVIDIA boxes
//! have no usable VAAPI (nvidia-vaapi-driver is broken for this — Moonlight blacklists
//! it); device creation fails there and the software path takes over. A mid-session
//! VAAPI error also falls back — the host's IDR/RFI recovery resynchronizes.
//! * **Software**: libavcodec on the CPU + swscale to RGBA (`GdkMemoryTexture` upload).
//! Slice threading only — frame threading would add a frame of latency per thread.
//!
//! Both run `AV_CODEC_FLAG_LOW_DELAY`; the host encodes zero-reorder streams (no
//! B-frames, in-band parameter sets on every IDR), so decode is strictly one-in/one-out.
use anyhow::{anyhow, bail, Context as _, Result};
use ffmpeg::format::Pixel;
use ffmpeg::software::scaling;
use ffmpeg::util::frame::Video as AvFrame;
use ffmpeg_next as ffmpeg;
use std::os::fd::RawFd;
use std::ptr;
pub enum DecodedFrame {
Cpu(CpuFrame),
Dmabuf(DmabufFrame),
}
/// RGBA pixels for `GdkMemoryTexture` (which takes a stride).
pub struct CpuFrame {
pub width: u32,
pub height: u32,
/// RGBA row stride in bytes (≥ width*4 — swscale pads rows for SIMD).
pub stride: usize,
pub rgba: Vec<u8>,
}
/// A decoded frame still on the GPU: dmabuf fds + plane layout for
/// `GdkDmabufTextureBuilder`. The fds belong to `guard`'s mapped DRM frame — they stay
/// valid until the guard drops (the texture's release func).
pub struct DmabufFrame {
pub width: u32,
pub height: u32,
/// Combined DRM fourcc of the whole surface (NV12 for 8-bit VAAPI output), derived
/// from the decoder's software format — NOT the per-plane component formats.
pub fourcc: u32,
pub modifier: u64,
pub planes: Vec<DmabufPlane>,
pub guard: DrmFrameGuard,
}
pub struct DmabufPlane {
pub fd: RawFd,
pub offset: u32,
pub stride: u32,
}
/// Owns the mapped DRM-PRIME `AVFrame` (which in turn references the VAAPI surface).
/// Dropping it releases the surface back to the decoder pool and closes the fds.
pub struct DrmFrameGuard(*mut ffmpeg::ffi::AVFrame);
// An AVFrame is plain refcounted data; freeing it from the GTK main thread is fine.
unsafe impl Send for DrmFrameGuard {}
impl Drop for DrmFrameGuard {
fn drop(&mut self) {
unsafe { ffmpeg::ffi::av_frame_free(&mut self.0) };
}
}
enum Backend {
Vaapi(VaapiDecoder),
Software(SoftwareDecoder),
}
pub struct Decoder {
backend: Backend,
}
impl Decoder {
pub fn new() -> Result<Decoder> {
ffmpeg::init().context("ffmpeg init")?;
let choice = std::env::var("PUNKTFUNK_DECODER").unwrap_or_default();
if choice != "software" {
match VaapiDecoder::new() {
Ok(v) => {
tracing::info!("VAAPI hardware decode active (zero-copy dmabuf)");
return Ok(Decoder {
backend: Backend::Vaapi(v),
});
}
Err(e) => {
if choice == "vaapi" {
return Err(e.context("PUNKTFUNK_DECODER=vaapi but VAAPI failed"));
}
tracing::info!(reason = %e, "VAAPI unavailable — software decode");
}
}
}
Ok(Decoder {
backend: Backend::Software(SoftwareDecoder::new()?),
})
}
/// Feed one access unit; returns the decoded frame (the host's streams are
/// one-in/one-out). A software decode error after packet loss is survivable — log
/// upstream and keep feeding. A VAAPI error demotes to software for the rest of the
/// session (broken driver, e.g. nvidia-vaapi-driver) — the next IDR resynchronizes.
pub fn decode(&mut self, au: &[u8]) -> Result<Option<DecodedFrame>> {
match &mut self.backend {
Backend::Vaapi(v) => match v.decode(au) {
Ok(f) => Ok(f.map(DecodedFrame::Dmabuf)),
Err(e) => {
tracing::warn!(error = %e, "VAAPI decode failed — falling back to software");
self.backend = Backend::Software(SoftwareDecoder::new()?);
Ok(None)
}
},
Backend::Software(s) => Ok(s.decode(au)?.map(DecodedFrame::Cpu)),
}
}
}
// --- software backend ---------------------------------------------------------------
struct SoftwareDecoder {
decoder: ffmpeg::decoder::Video,
/// Rebuilt whenever the decoded format/size changes (mid-stream `Reconfigure`).
sws: Option<(scaling::Context, Pixel, u32, u32)>,
}
impl SoftwareDecoder {
fn new() -> Result<SoftwareDecoder> {
let codec =
ffmpeg::decoder::find(ffmpeg::codec::Id::HEVC).ok_or(anyhow!("no HEVC decoder"))?;
let mut ctx = ffmpeg::codec::Context::new_with_codec(codec);
unsafe {
let raw = ctx.as_mut_ptr();
(*raw).flags |= ffmpeg::ffi::AV_CODEC_FLAG_LOW_DELAY as i32;
// Slice threading adds no frame delay (frame threading adds thread_count-1).
(*raw).thread_type = ffmpeg::ffi::FF_THREAD_SLICE;
(*raw).thread_count = 0; // auto
}
let decoder = ctx.decoder().video().context("open HEVC decoder")?;
Ok(SoftwareDecoder { decoder, sws: None })
}
fn decode(&mut self, au: &[u8]) -> Result<Option<CpuFrame>> {
let packet = ffmpeg::Packet::copy(au);
self.decoder
.send_packet(&packet)
.map_err(|e| anyhow!("send_packet: {e}"))?;
let mut frame = AvFrame::empty();
let mut out = None;
while self.decoder.receive_frame(&mut frame).is_ok() {
out = Some(self.convert_rgba(&frame)?);
}
Ok(out)
}
fn convert_rgba(&mut self, frame: &AvFrame) -> Result<CpuFrame> {
let (fmt, w, h) = (frame.format(), frame.width(), frame.height());
let rebuild =
!matches!(&self.sws, Some((_, f, sw, sh)) if *f == fmt && *sw == w && *sh == h);
if rebuild {
let ctx = scaling::Context::get(fmt, w, h, Pixel::RGBA, w, h, scaling::Flags::POINT)
.context("swscale context")?;
self.sws = Some((ctx, fmt, w, h));
}
let (sws, ..) = self.sws.as_mut().unwrap();
let mut rgba = AvFrame::empty();
sws.run(frame, &mut rgba).map_err(|e| anyhow!("sws: {e}"))?;
Ok(CpuFrame {
width: w,
height: h,
stride: rgba.stride(0),
rgba: rgba.data(0).to_vec(),
})
}
}
// --- VAAPI backend --------------------------------------------------------------------
//
// Raw FFI: ffmpeg-next has no hwaccel wrappers. All pointers are owned here and freed in
// Drop; decoded surfaces transfer out through DrmFrameGuard.
const AVERROR_EAGAIN: i32 = -11; // -EAGAIN; Linux-only crate
fn averr(what: &str, code: i32) -> anyhow::Error {
anyhow!("{what}: {}", ffmpeg::Error::from(code))
}
/// libavcodec offers the formats it can decode into; pick the VAAPI hw surface. Falling
/// back to the first (software) entry would silently decode on the CPU *and* break our
/// dmabuf mapping — return NONE instead so the error surfaces and the session demotes
/// to the software backend explicitly.
unsafe extern "C" fn pick_vaapi(
_ctx: *mut ffmpeg::ffi::AVCodecContext,
mut list: *const ffmpeg::ffi::AVPixelFormat,
) -> ffmpeg::ffi::AVPixelFormat {
unsafe {
while *list != ffmpeg::ffi::AVPixelFormat::AV_PIX_FMT_NONE {
if *list == ffmpeg::ffi::AVPixelFormat::AV_PIX_FMT_VAAPI {
return ffmpeg::ffi::AVPixelFormat::AV_PIX_FMT_VAAPI;
}
list = list.add(1);
}
}
ffmpeg::ffi::AVPixelFormat::AV_PIX_FMT_NONE
}
struct VaapiDecoder {
ctx: *mut ffmpeg::ffi::AVCodecContext,
hw_device: *mut ffmpeg::ffi::AVBufferRef,
packet: *mut ffmpeg::ffi::AVPacket,
frame: *mut ffmpeg::ffi::AVFrame,
}
// Single-owner pointers, only touched from the session pump thread.
unsafe impl Send for VaapiDecoder {}
impl VaapiDecoder {
fn new() -> Result<VaapiDecoder> {
use ffmpeg::ffi;
unsafe {
let mut hw_device: *mut ffi::AVBufferRef = ptr::null_mut();
let r = ffi::av_hwdevice_ctx_create(
&mut hw_device,
ffi::AVHWDeviceType::AV_HWDEVICE_TYPE_VAAPI,
ptr::null(),
ptr::null_mut(),
0,
);
if r < 0 {
bail!("no VAAPI device ({})", ffmpeg::Error::from(r));
}
let codec = ffi::avcodec_find_decoder(ffi::AVCodecID::AV_CODEC_ID_HEVC);
if codec.is_null() {
ffi::av_buffer_unref(&mut hw_device);
bail!("no HEVC decoder");
}
let ctx = ffi::avcodec_alloc_context3(codec);
(*ctx).hw_device_ctx = ffi::av_buffer_ref(hw_device);
(*ctx).get_format = Some(pick_vaapi);
(*ctx).flags |= ffi::AV_CODEC_FLAG_LOW_DELAY as i32;
(*ctx).thread_count = 1; // hwaccel: threads only add latency
let r = ffi::avcodec_open2(ctx, codec, ptr::null_mut());
if r < 0 {
let mut ctx = ctx;
ffi::avcodec_free_context(&mut ctx);
let mut hw_device = hw_device;
ffi::av_buffer_unref(&mut hw_device);
bail!("avcodec_open2: {}", ffmpeg::Error::from(r));
}
Ok(VaapiDecoder {
ctx,
hw_device,
packet: ffi::av_packet_alloc(),
frame: ffi::av_frame_alloc(),
})
}
}
fn decode(&mut self, au: &[u8]) -> Result<Option<DmabufFrame>> {
use ffmpeg::ffi;
unsafe {
let r = ffi::av_new_packet(self.packet, au.len() as i32);
if r < 0 {
return Err(averr("av_new_packet", r));
}
ptr::copy_nonoverlapping(au.as_ptr(), (*self.packet).data, au.len());
let r = ffi::avcodec_send_packet(self.ctx, self.packet);
ffi::av_packet_unref(self.packet);
if r < 0 {
return Err(averr("send_packet", r));
}
let mut out = None;
loop {
let r = ffi::avcodec_receive_frame(self.ctx, self.frame);
if r == AVERROR_EAGAIN {
break;
}
if r < 0 {
return Err(averr("receive_frame", r));
}
out = Some(self.map_dmabuf()?); // newest wins; older guards drop here
ffi::av_frame_unref(self.frame);
}
Ok(out)
}
}
/// Map the VAAPI surface to DRM PRIME (zero copy) and lift the descriptor into a
/// `DmabufFrame`. The mapped frame keeps the surface alive via its buffer refs.
///
/// FFmpeg's VAAPI export uses `VA_EXPORT_SURFACE_SEPARATE_LAYERS`, so an NV12 surface
/// comes back as TWO layers (`R8` luma + `GR88` chroma), each one plane — NOT a single
/// `NV12` layer. The previous code took `layers[0]` only: GTK then saw an `R8`
/// single-plane texture with the chroma dropped, painting the screen green. The fix:
/// derive the COMBINED fourcc from the decoder's software pixel format (NV12 →
/// `DRM_FORMAT_NV12`) and flatten every plane across every layer in order (Y then UV).
unsafe fn map_dmabuf(&mut self) -> Result<DmabufFrame> {
use ffmpeg::ffi;
unsafe {
if (*self.frame).format != ffi::AVPixelFormat::AV_PIX_FMT_VAAPI as i32 {
bail!("decoder returned a software frame (no VAAPI surface)");
}
// The real pixel layout lives on the hardware frames context, not the
// DRM-PRIME layer formats (those are the per-plane R8/GR88 component formats).
let sw_format = {
let hwfc = (*self.frame).hw_frames_ctx;
if hwfc.is_null() {
bail!("VAAPI frame without a hardware frames context");
}
(*((*hwfc).data as *const ffi::AVHWFramesContext)).sw_format
};
let fourcc = drm_fourcc_for(sw_format)
.ok_or_else(|| anyhow!("unsupported VAAPI output format {sw_format:?}"))?;
let drm = ffi::av_frame_alloc();
(*drm).format = ffi::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as i32;
let r = ffi::av_hwframe_map(drm, self.frame, ffi::AV_HWFRAME_MAP_READ as i32);
if r < 0 {
let mut drm = drm;
ffi::av_frame_free(&mut drm);
return Err(averr("av_hwframe_map", r));
}
let desc = (*drm).data[0] as *const ffi::AVDRMFrameDescriptor;
let guard = DrmFrameGuard(drm);
let d = &*desc;
if d.nb_layers < 1 || d.nb_objects < 1 {
bail!("DRM descriptor without layers/objects");
}
// Flatten planes across ALL layers, in declared order — the combined fourcc's
// plane order (Y, then UV for NV12) matches the layer order FFmpeg emits.
let mut planes = Vec::new();
for layer in &d.layers[..d.nb_layers as usize] {
for p in &layer.planes[..layer.nb_planes as usize] {
let obj = &d.objects[p.object_index as usize];
planes.push(DmabufPlane {
fd: obj.fd,
offset: p.offset as u32,
stride: p.pitch as u32,
});
}
}
// The whole surface shares one tiling modifier (one BO on radeonsi); GTK takes
// a single modifier for the texture.
let modifier = d.objects[0].format_modifier;
log_descriptor_once(d, sw_format, fourcc, modifier);
Ok(DmabufFrame {
width: (*self.frame).width as u32,
height: (*self.frame).height as u32,
fourcc,
modifier,
planes,
guard,
})
}
}
}
/// `fourcc(a,b,c,d)` — the DRM FourCC packing (little-endian, `a | b<<8 | c<<16 | d<<24`).
const fn fourcc(a: u8, b: u8, c: u8, d: u8) -> u32 {
(a as u32) | ((b as u32) << 8) | ((c as u32) << 16) | ((d as u32) << 24)
}
/// The combined DRM FourCC for a decoder software pixel format. The host streams 8-bit
/// 4:2:0 (NV12); P010 is here for the eventual 10-bit/HDR path.
fn drm_fourcc_for(sw: ffmpeg_next::ffi::AVPixelFormat) -> Option<u32> {
use ffmpeg_next::ffi::AVPixelFormat::*;
Some(match sw {
AV_PIX_FMT_NV12 => fourcc(b'N', b'V', b'1', b'2'),
AV_PIX_FMT_P010LE => fourcc(b'P', b'0', b'1', b'0'),
_ => return None,
})
}
/// One-time dump of the DRM descriptor layout (objects, layers, planes, modifier) — so a
/// new client/driver combination's real layout is visible in the logs without a debugger.
fn log_descriptor_once(
d: &ffmpeg_next::ffi::AVDRMFrameDescriptor,
sw: ffmpeg_next::ffi::AVPixelFormat,
fourcc: u32,
modifier: u64,
) {
use std::sync::atomic::{AtomicBool, Ordering};
static ONCE: AtomicBool = AtomicBool::new(true);
if !ONCE.swap(false, Ordering::Relaxed) {
return;
}
let layers: Vec<(u32, i32)> = d.layers[..d.nb_layers.max(0) as usize]
.iter()
.map(|l| (l.format, l.nb_planes))
.collect();
tracing::info!(
sw_format = ?sw,
chosen_fourcc = format_args!("{:#010x}", fourcc),
nb_objects = d.nb_objects,
nb_layers = d.nb_layers,
?layers,
modifier = format_args!("{:#018x}", modifier),
"VAAPI dmabuf descriptor layout (first frame)"
);
}
impl Drop for VaapiDecoder {
fn drop(&mut self) {
use ffmpeg::ffi;
unsafe {
ffi::av_packet_free(&mut self.packet);
ffi::av_frame_free(&mut self.frame);
ffi::avcodec_free_context(&mut self.ctx);
ffi::av_buffer_unref(&mut self.hw_device);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Lock the DRM FourCC magic numbers against typos — these are the exact values
/// `<drm_fourcc.h>` defines, and a wrong one is what painted the Steam Deck green.
#[test]
fn drm_fourcc_constants() {
assert_eq!(fourcc(b'N', b'V', b'1', b'2'), 0x3231_564e);
assert_eq!(fourcc(b'P', b'0', b'1', b'0'), 0x3031_3050);
assert_eq!(
drm_fourcc_for(ffmpeg::ffi::AVPixelFormat::AV_PIX_FMT_NV12),
Some(0x3231_564e)
);
assert_eq!(
drm_fourcc_for(ffmpeg::ffi::AVPixelFormat::AV_PIX_FMT_RGBA),
None
);
}
}