feat(console): full gamepad shell — hosts, PIN pairing, settings, OSK, screen transitions
apple / swift (push) Successful in 1m6s
android / android (push) Has been cancelled
apple / screenshots (push) Has been cancelled
arch / build-publish (push) Has been cancelled
audit / bun-audit (push) Has been cancelled
audit / cargo-audit (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
deb / build-publish (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
release / apple (push) Has been cancelled
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
windows-host / package (push) Has been cancelled
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Has been cancelled
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Has been cancelled
windows / build (aarch64-pc-windows-msvc) (push) Has been cancelled
windows / build (x86_64-pc-windows-msvc) (push) Has been cancelled
decky / build-publish (push) Successful in 25s
flatpak / build-publish (push) Successful in 4m31s

The Skia console UI grows from the single library coverflow into a complete
couch shell (Apple gamepad-UI parity), so a Deck/gamescope session is
self-sufficient end to end:

- Home: host carousel (saved + discovered + Add Host tile) with presence
  pips, paired locks, wake & connect; A/Y/X/B per the Apple launcher.
- In-console SPAKE2 PIN pairing (previously needed the Decky plugin or a
  desktop), add-host with a controller keyboard, couch settings over the
  shared Settings store, wake-on-LAN overlay with retry, cancelable dials.
- Shell chrome: screen stack with push/pop entrance/exit choreography
  (slide + fade + recede; aurora↔form backdrop crossfade), per-pad button
  glyphs (PS shapes vs ABXY, keycaps with no pad), menu haptics, toasts,
  embedded Geist typography, in-stream start banner.
- Steam Deck: our OSK never draws — fields start SDL text input (the new
  Overlay::text_input_active hook) and Steam's keyboard types; a hint chip
  points at STEAM + X. Other Linux gets the full gamepad keyboard tray.
- punktfunk-session: bare --browse opens Home; --browse host opens that
  host's library with Home behind it (Decky launches keep working, B now
  pops instead of quitting). Service threads run discovery, 10 s probes,
  pairing, wake loops, fetches, and known-hosts persistence.
- Presenter contract: Launch actions carry the host, CancelConnect never
  engages a won race, pad kind/list ride FrameCtx, menu events flow while
  dialing so B can cancel.

Every screen renders to PNG on CPU raster for review
(PF_CONSOLE_DUMP=<dir> cargo test -p pf-console-ui -- --ignored dump).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 08:46:20 +02:00
parent c3fa6c1514
commit 180ac3aa61
31 changed files with 6406 additions and 1032 deletions
+168 -56
View File
@@ -2,9 +2,15 @@
//! two offscreen render-target surfaces (the presenter runs one frame in flight and may
//! still be sampling the previous image while we render the next), damage-driven
//! redraws (content/size change only — an unchanged OSD costs zero GPU work per frame).
//!
//! Two personas over one `Overlay`: the CONSOLE (the full shell — home, library,
//! settings, pairing — always dirty, the aurora animates) and the stream chrome (stats
//! OSD, capture hint, the auto-fading start banner).
use crate::library::LibraryShared;
use crate::library_ui::{build_fonts, Fonts, LibraryUi};
use crate::model::{ConsoleBus, ConsoleShared, HostRow};
use crate::screens::Screen;
use crate::shell::{ConsoleOptions, Shell};
use crate::theme::{match_first_family, Fonts};
use anyhow::{anyhow, Context as _, Result};
use ash::vk as avk;
use ash::vk::Handle as _;
@@ -15,11 +21,16 @@ use pf_presenter::overlay::{
use skia_safe::gpu::vk as skvk;
use skia_safe::gpu::{self, DirectContext, SurfaceOrigin};
use skia_safe::{Canvas, Color4f, Font, FontMgr, Paint, Point, RRect, Rect, Surface};
use std::time::Instant;
/// Skia's GPU resource budget — the OSD/HUD need a few MB; the console library will
/// revisit (the plan budgets 64 MB on Deck-class shared memory).
/// Skia's GPU resource budget — poster art plus a few screen layers; 64 MB fits
/// Deck-class shared memory.
const RESOURCE_CACHE_BYTES: usize = 64 << 20;
/// How long the start-of-stream banner lingers (fading through the tail).
const BANNER_S: f64 = 6.0;
const BANNER_FADE_S: f64 = 0.6;
/// One offscreen target: the Skia surface + the raw Vulkan handles the presenter
/// samples. The image is Skia-owned (freed with the surface); the view is ours.
struct Slot {
@@ -37,6 +48,24 @@ struct Drawn {
height: u32,
stats: Option<String>,
hint: Option<String>,
/// The start banner's alpha, quantized — a fade step is a redraw, steady is not.
banner_step: u8,
}
/// Where the console starts (the session binary's `--browse` forms).
pub enum ConsoleEntry {
/// The host list (bare `--browse`).
Home,
/// Home with this host's library already pushed (`--browse host` — the Decky
/// per-host launch; B backs out to Home).
Library(HostRow),
}
/// The binary's ends of the console: models to write, commands to serve.
pub struct ConsoleHandles {
pub console: ConsoleShared,
pub library: crate::library::LibraryShared,
pub bus: ConsoleBus,
}
pub struct SkiaOverlay {
@@ -47,10 +76,15 @@ pub struct SkiaOverlay {
/// Which slot the LAST returned frame lives in — the next render takes the other.
current: usize,
drawn: Drawn,
/// The stats OSD's monospace face (system-resolved; the console uses Geist).
font: Option<Font>,
/// The console library (`--browse`) — `None` for a plain `--connect` session.
library: Option<LibraryUi>,
fonts: Option<Fonts>,
/// The console shell (`--browse`) — `None` for a plain `--connect` session.
shell: Option<Shell>,
/// When the current stream started presenting — drives the start banner.
streaming_since: Option<Instant>,
/// The banner's words (set per stream from the active-pad state).
banner_text: Option<String>,
}
struct Gpu {
@@ -72,22 +106,44 @@ impl SkiaOverlay {
current: 0,
drawn: Drawn::default(),
font: None,
library: None,
fonts: None,
shell: None,
streaming_since: None,
banner_text: None,
}
}
/// The `--browse` overlay: the console library between streams, stream chrome
/// during them. Returns the shared model the binary's fetch threads write into.
pub fn with_library(host_label: String) -> Result<(SkiaOverlay, LibraryShared)> {
let shared = LibraryShared::default();
/// The `--browse` overlay: the full console shell between streams, stream chrome
/// during them. Returns the binary's handles (models + command bus).
pub fn console(
opts: ConsoleOptions,
entry: ConsoleEntry,
) -> Result<(SkiaOverlay, ConsoleHandles)> {
let console = ConsoleShared::default();
let library = crate::library::LibraryShared::default();
let bus = ConsoleBus::default();
let stack = match entry {
ConsoleEntry::Home => vec![Screen::Home(crate::screens::home::HomeScreen::new())],
ConsoleEntry::Library(host) => vec![
Screen::Home(crate::screens::home::HomeScreen::new()),
Screen::Library(crate::screens::library::LibraryScreen::new(&host)),
],
};
let shell = Shell::new(console.clone(), library.clone(), bus.clone(), opts, stack)?;
let mut o = SkiaOverlay::new();
o.library = Some(LibraryUi::new(shared.clone(), host_label)?);
Ok((o, shared))
o.shell = Some(shell);
Ok((
o,
ConsoleHandles {
console,
library,
bus,
},
))
}
fn library_visible(&self) -> bool {
self.library.as_ref().is_some_and(|l| !l.in_stream)
fn console_visible(&self) -> bool {
self.shell.as_ref().is_some_and(|s| !s.in_stream)
}
}
@@ -146,14 +202,14 @@ impl Overlay for SkiaOverlay {
.ok_or_else(|| anyhow!("Skia DirectContext over the shared device"))?;
context.set_resource_cache_limit(RESOURCE_CACHE_BYTES);
let typeface = crate::library_ui::match_first_family(
let typeface = match_first_family(
&FontMgr::new(),
&["monospace", "Consolas", "Cascadia Mono", "Courier New"],
skia_safe::FontStyle::normal(),
)
.context("no monospace typeface (fontconfig alias or system family)")?;
self.font = Some(Font::new(typeface, 14.0));
self.fonts = Some(build_fonts()?);
self.fonts = Some(crate::theme::build_fonts()?);
self.gpu = Some(Gpu {
device: shared.device.clone(),
@@ -167,77 +223,93 @@ impl Overlay for SkiaOverlay {
}
fn handle_event(&mut self, event: &sdl3::event::Event) -> bool {
// The library's keyboard fallback (arrows/Enter/Esc) — only while it's on
// screen, and never for chord-modified keys (those stay the run loop's).
if self.library_visible() {
if let sdl3::event::Event::KeyDown {
// Keyboard + text into the console while it's on screen — never for
// chord-modified keys (those stay the run loop's), never during a stream.
if !self.console_visible() {
return false;
}
let Some(shell) = &mut self.shell else {
return false;
};
match event {
sdl3::event::Event::KeyDown {
scancode: Some(sc),
keymod,
repeat,
..
} = event
{
} => {
use sdl3::keyboard::Mod;
if !keymod.intersects(Mod::LCTRLMOD | Mod::RCTRLMOD | Mod::LALTMOD | Mod::RALTMOD) {
return self.library.as_mut().is_some_and(|l| l.key(*sc, *repeat));
if keymod.intersects(Mod::LCTRLMOD | Mod::RCTRLMOD | Mod::LALTMOD | Mod::RALTMOD) {
return false;
}
shell.key(*sc, *repeat)
}
sdl3::event::Event::TextInput { text, .. } => {
shell.text_input(text);
true
}
_ => false,
}
false
}
fn handle_menu(&mut self, event: MenuEvent) -> Option<MenuPulse> {
if self.library_visible() {
self.library.as_mut().and_then(|l| l.menu(event))
if self.console_visible() {
self.shell.as_mut().and_then(|s| s.handle_menu(event))
} else {
None
}
}
fn take_action(&mut self) -> Option<OverlayAction> {
self.library.as_mut().and_then(|l| l.take_action())
self.shell.as_mut().and_then(|s| s.take_action())
}
fn text_input_active(&self) -> bool {
self.console_visible() && self.shell.as_ref().is_some_and(Shell::editing)
}
fn session_phase(&mut self, phase: SessionPhase) {
let Some(lib) = &mut self.library else { return };
let Some(shell) = &mut self.shell else { return };
match phase {
SessionPhase::Connecting => lib.set_connecting(true),
SessionPhase::Connecting => {} // the shell raised the Launch; already showing
SessionPhase::Streaming => {
lib.in_stream = true;
lib.set_connecting(false);
shell.session_streaming();
self.streaming_since = Some(Instant::now());
}
SessionPhase::Failed(msg) => lib.session_error(msg),
SessionPhase::Ended(None) => {
lib.in_stream = false;
lib.set_connecting(false);
SessionPhase::Failed(msg) => shell.session_failed(msg),
SessionPhase::Ended(reason) => {
shell.session_ended(reason);
self.streaming_since = None;
}
SessionPhase::Ended(Some(reason)) => lib.session_error(reason),
}
}
fn frame(&mut self, ctx: &FrameCtx) -> Result<Option<OverlayFrame>> {
// The console library: full-screen, opaque, and always dirty (the aurora
// animates every frame — the GPU port's whole point).
if self.library_visible() {
// The console: full-screen, opaque, and always dirty (the aurora animates
// every frame — the GPU port's whole point).
if self.console_visible() {
let next = 1 - self.current;
self.ensure_slot(next, ctx.width, ctx.height)?;
let Self {
gpu,
slots,
library,
shell,
fonts,
..
} = self;
let gpu = gpu.as_mut().expect("init ran");
let slot = slots[next].as_mut().expect("just ensured");
let lib = library.as_mut().expect("library_visible");
let fonts = fonts.as_mut().expect("init ran");
fonts.chip_text = Some(ctx.pad.map_or(
"No controller — keyboard works too".to_string(),
str::to_owned,
));
lib.sync();
lib.render(slot.surface.canvas(), ctx.width, ctx.height, fonts);
let shell = shell.as_mut().expect("console_visible");
let fonts = fonts.as_ref().expect("init ran");
shell.render(
slot.surface.canvas(),
ctx.width,
ctx.height,
fonts,
ctx.pad,
ctx.pad_pref,
ctx.pads,
);
gpu.context.flush_surface_with_texture_state(
&mut slot.surface,
&gpu::FlushInfo::default(),
@@ -258,7 +330,10 @@ impl Overlay for SkiaOverlay {
}));
}
if ctx.stats.is_none() && ctx.hint.is_none() {
// --- Stream chrome: stats OSD + capture hint + the start banner ---------------
let banner_alpha = self.banner_alpha(ctx);
let banner_step = (banner_alpha * 32.0).round() as u8;
if ctx.stats.is_none() && ctx.hint.is_none() && banner_step == 0 {
self.drawn = Drawn::default(); // forget content so re-show re-renders
return Ok(None);
}
@@ -267,6 +342,7 @@ impl Overlay for SkiaOverlay {
height: ctx.height,
stats: ctx.stats.map(str::to_owned),
hint: ctx.hint.map(str::to_owned),
banner_step,
};
if want == self.drawn {
// Unchanged — hand the presenter the already-rendered image.
@@ -292,7 +368,20 @@ impl Overlay for SkiaOverlay {
draw_osd_panel(canvas, font, stats, 12.0, 12.0);
}
if let Some(hint) = &want.hint {
draw_hint_pill(canvas, font, hint, ctx.width, ctx.height);
draw_hint_pill(canvas, font, hint, ctx.width, ctx.height, 1.0);
} else if banner_step > 0 {
// The start banner: the leave/stats shortcuts, fading out on its own —
// discoverable without the stats overlay, gone before it annoys.
if let Some(text) = &self.banner_text {
draw_hint_pill(
canvas,
font,
text,
ctx.width,
ctx.height,
banner_alpha as f32,
);
}
}
// Flush on the shared queue, ending in SHADER_READ_ONLY on our family — the
@@ -309,6 +398,7 @@ impl Overlay for SkiaOverlay {
self.current = next;
self.drawn = want;
let slot = self.slots[next].as_ref().expect("just rendered");
Ok(Some(OverlayFrame {
image: slot.image,
view: slot.view,
@@ -319,6 +409,28 @@ impl Overlay for SkiaOverlay {
}
impl SkiaOverlay {
/// The start banner's current alpha (1 → 0 across the fade tail), refreshing its
/// words while fully visible so a pad hot-plug updates the leave hint.
fn banner_alpha(&mut self, ctx: &FrameCtx) -> f64 {
let Some(since) = self.streaming_since else {
self.banner_text = None;
return 0.0;
};
let age = since.elapsed().as_secs_f64();
if age >= BANNER_S {
self.streaming_since = None;
self.banner_text = None;
return 0.0;
}
self.banner_text = Some(if ctx.pad.is_some() {
"Hold L1 + R1 + Start + Select to leave · Ctrl+Alt+Shift+S stats".to_string()
} else {
"Ctrl+Alt+Shift+Q releases input · Ctrl+Alt+Shift+D disconnects · Ctrl+Alt+Shift+S stats"
.to_string()
});
((BANNER_S - age) / BANNER_FADE_S).min(1.0)
}
/// Make `slots[i]` a render target of exactly `width`×`height` (rebuilt on resize).
fn ensure_slot(&mut self, i: usize, width: u32, height: u32) -> Result<()> {
if self.slots[i]
@@ -414,8 +526,8 @@ fn draw_osd_panel(canvas: &Canvas, font: &Font, text: &str, x: f32, y: f32) {
}
}
/// The capture hint: a centered pill near the bottom edge (the GTK hint's position).
fn draw_hint_pill(canvas: &Canvas, font: &Font, text: &str, width: u32, height: u32) {
/// The capture hint / start banner: a centered pill near the bottom edge.
fn draw_hint_pill(canvas: &Canvas, font: &Font, text: &str, width: u32, height: u32, alpha: f32) {
let (_, metrics) = font.metrics();
let line_h = metrics.descent - metrics.ascent;
let text_w = font.measure_str(text, None).0;
@@ -426,12 +538,12 @@ fn draw_hint_pill(canvas: &Canvas, font: &Font, text: &str, width: u32, height:
let y = height as f32 - h - 24.0;
canvas.draw_rrect(
RRect::new_rect_xy(Rect::from_xywh(x, y, w, h), h / 2.0, h / 2.0),
&Paint::new(Color4f::new(0.0, 0.0, 0.0, 0.62), None),
&Paint::new(Color4f::new(0.0, 0.0, 0.0, 0.62 * alpha), None),
);
canvas.draw_str(
text,
Point::new(x + pad_x, y + pad_y - metrics.ascent),
font,
&Paint::new(Color4f::new(1.0, 1.0, 1.0, 0.92), None),
&Paint::new(Color4f::new(1.0, 1.0, 1.0, 0.92 * alpha), None),
);
}