feat(touch): cross-client touch-input modes on Linux + Windows
ci / web (push) Successful in 43s
ci / rust (push) Failing after 53s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
decky / build-publish (push) Successful in 16s
ci / docs-site (push) Successful in 1m5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m41s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m53s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 52s
apple / swift (push) Successful in 4m41s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 54s
ci / bench (push) Successful in 6m30s
docker / deploy-docs (push) Successful in 23s
flatpak / build-publish (push) Failing after 8m9s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
android / android (push) Has been cancelled
apple / screenshots (push) Has been cancelled
arch / build-publish (push) Has been cancelled
deb / build-publish (push) Has been cancelled

Bring the SDL presenter (Linux/Deck + Windows) to parity with the Android and
Apple clients: a persisted TouchMode selects how a touchscreen drives the host —

  * Trackpad (default): relative cursor with pointer ballistics + the shared
    gesture vocabulary (tap = left click, two-finger tap = right click,
    two-finger drag = scroll, tap-then-drag = held left drag, three-finger tap =
    cycle the stats overlay).
  * Direct pointer: the cursor jumps to and follows the finger (absolute).
  * Touch passthrough: every finger is a real host touchscreen contact.

Previously the presenter had no finger handling, so SDL synthesized mouse events
from touch and — under the stream's relative-mouse lock — walked the host cursor
into the corner (the reported Deck bug). SDL touch->mouse synthesis is now off;
DIRECT touchscreens route through a new incremental gesture engine (a port of
Android TouchInput.kt / Apple TouchMouse.swift), while INDIRECT trackpads keep
driving the mouse. Fingers map through the aspect-fit letterbox onto the content
rect.

TouchMode lives in the shared trust::Settings (default trackpad, so passthrough
is opt-in like the other clients); the GTK and WinUI settings screens both gained
a "Touch input" picker. Gesture engine, letterbox mapping, and settings
back-compat are unit-tested (28 tests green); clippy -D warnings clean; full
Linux client + session build verified on-host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-13 23:51:29 +02:00
parent 94802795e7
commit f88d0ae4dc
9 changed files with 938 additions and 22 deletions
+80
View File
@@ -343,6 +343,53 @@ impl StatsVerbosity {
}
}
/// How a touchscreen's fingers drive the host — the cross-client touch-input model (Android
/// `TouchMode`, Apple `TouchInputMode`). Stored stringly in [`Settings::touch_mode`] so the
/// file stays readable; parsed with [`TouchMode::from_name`].
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum TouchMode {
/// Relative cursor like a laptop touchpad: the cursor stays put on touch-down and moves
/// by the finger's delta (with mild acceleration), tap to click. The default — a cursor
/// is the universally workable model on a screen the host isn't sized for.
Trackpad,
/// Direct pointing: the cursor jumps to the finger and follows it (absolute).
Pointer,
/// Real multi-touch passthrough: every finger is a host touchscreen contact, no gesture
/// interpretation — only helps hosts/apps that actually understand touch.
Touch,
}
impl TouchMode {
/// Cycle/picker order (also the settings pickers' option order).
pub const ALL: [TouchMode; 3] = [TouchMode::Trackpad, TouchMode::Pointer, TouchMode::Touch];
/// Parse the persisted name, defaulting to `Trackpad` for unset/unknown values.
pub fn from_name(s: &str) -> TouchMode {
match s {
"pointer" => TouchMode::Pointer,
"touch" => TouchMode::Touch,
_ => TouchMode::Trackpad,
}
}
/// The persisted name (the inverse of [`from_name`](Self::from_name)).
pub fn as_name(self) -> &'static str {
match self {
TouchMode::Trackpad => "trackpad",
TouchMode::Pointer => "pointer",
TouchMode::Touch => "touch",
}
}
pub fn label(self) -> &'static str {
match self {
TouchMode::Trackpad => "Trackpad",
TouchMode::Pointer => "Direct pointer",
TouchMode::Touch => "Touch passthrough",
}
}
}
/// App settings, persisted as JSON. Stringly-typed gamepad/compositor prefs so the file
/// stays readable; parsed with `*Pref::from_name` at connect time.
#[derive(Clone, Serialize, Deserialize)]
@@ -363,6 +410,12 @@ pub struct Settings {
/// Which host compositor backend to request (advisory; the host falls back to
/// auto-detect when unavailable).
pub compositor: String,
/// How a touchscreen's fingers drive the host (Deck/tablet): a [`TouchMode`] name —
/// `"trackpad"` (default), `"pointer"`, or `"touch"`. Read at connect via
/// [`Settings::touch_mode`]; irrelevant on a mouse-only client. `default` so pre-existing
/// stores load as trackpad.
#[serde(default = "default_touch_mode")]
pub touch_mode: String,
/// Grab compositor shortcuts (Alt+Tab, Super…) while input is captured.
pub inhibit_shortcuts: bool,
/// Stream the default microphone to the host's virtual mic source.
@@ -425,6 +478,10 @@ fn default_codec() -> String {
"auto".into()
}
fn default_touch_mode() -> String {
"trackpad".into()
}
fn default_true() -> bool {
true
}
@@ -447,6 +504,11 @@ impl Settings {
self.show_stats = v != StatsVerbosity::Off;
}
/// The touch-input model for this session (parsed from the stored name).
pub fn touch_mode(&self) -> TouchMode {
TouchMode::from_name(&self.touch_mode)
}
/// The `codec` setting as a `quic::CODEC_*` preference bit (`0` = auto).
pub fn preferred_codec(&self) -> u8 {
match self.codec.as_str() {
@@ -468,6 +530,7 @@ impl Default for Settings {
gamepad: "auto".into(),
forward_pad: String::new(),
compositor: "auto".into(),
touch_mode: "trackpad".into(),
inhibit_shortcuts: true,
mic_enabled: false,
audio_channels: 2,
@@ -519,6 +582,23 @@ impl Settings {
mod tests {
use super::*;
/// A settings file predating the touch-input model loads as `trackpad` (the shipped
/// default), and the name round-trips through the enum both ways.
#[test]
fn settings_touch_mode_defaults_trackpad() {
let old = r#"{"width":1280,"height":720,"gamepad":"auto","compositor":"auto"}"#;
let s: Settings = serde_json::from_str(old).unwrap();
assert_eq!(s.touch_mode, "trackpad");
assert_eq!(s.touch_mode(), TouchMode::Trackpad);
// Explicit values parse; an unknown name falls back to trackpad.
assert_eq!(TouchMode::from_name("pointer"), TouchMode::Pointer);
assert_eq!(TouchMode::from_name("touch"), TouchMode::Touch);
assert_eq!(TouchMode::from_name("bogus"), TouchMode::Trackpad);
for m in TouchMode::ALL {
assert_eq!(TouchMode::from_name(m.as_name()), m);
}
}
/// A pre-`forward_pad` settings file (≤ 0.5.0) loads with the pin on automatic.
#[test]
fn settings_forward_pad_defaults_empty() {
+163 -2
View File
@@ -16,11 +16,21 @@
//! otherwise send a datagram per event).
use crate::keymap_sdl;
use crate::touch::{Abs, Act, Gestures};
use pf_client_core::trust::TouchMode;
use punktfunk_core::client::NativeClient;
use punktfunk_core::input::{InputEvent, InputKind};
use std::collections::HashSet;
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
/// Which transition a forwarded touchscreen finger is (SDL delivers one finger per event).
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum FingerPhase {
Down,
Move,
Up,
}
pub struct Capture {
connector: Arc<NativeClient>,
captured: bool,
@@ -34,6 +44,16 @@ pub struct Capture {
/// Fractional wheel remainder per axis (x, y) in 120-unit WHEEL_DELTA space —
/// precision surfaces deliver sub-unit deltas; truncating each event drops the tail.
scroll_acc: (f64, f64),
/// Active touchscreen contacts: SDL finger id → the small wire touch id (slot) we
/// forward it under. SDL finger ids are opaque and large; the host wants compact,
/// per-contact-unique ids reusable after up (input.rs::TouchDown). Slots are freed on
/// up and flushed up on release so no contact stays pressed on the host. Only used in
/// [`TouchMode::Touch`]; the other modes drive `gestures` instead.
touch_slots: HashMap<u64, u32>,
/// The touchscreen input model for this session, and — for trackpad/pointer — the
/// gesture state machine finger events feed.
touch_mode: TouchMode,
gestures: Gestures,
}
fn send(connector: &NativeClient, kind: InputKind, code: u32, x: i32, y: i32, flags: u32) {
@@ -48,7 +68,7 @@ fn send(connector: &NativeClient, kind: InputKind, code: u32, x: i32, y: i32, fl
}
impl Capture {
pub fn new(connector: Arc<NativeClient>) -> Capture {
pub fn new(connector: Arc<NativeClient>, touch_mode: TouchMode) -> Capture {
Capture {
connector,
captured: false,
@@ -57,6 +77,9 @@ impl Capture {
held_buttons: HashSet::new(),
pending_rel: (0, 0),
scroll_acc: (0.0, 0.0),
touch_slots: HashMap::new(),
touch_mode,
gestures: Gestures::new(touch_mode == TouchMode::Trackpad),
}
}
@@ -93,6 +116,12 @@ impl Capture {
for b in self.held_buttons.drain() {
send(&self.connector, InputKind::MouseButtonUp, b, 0, 0, 0);
}
for slot in self.touch_slots.drain().map(|(_, slot)| slot) {
send(&self.connector, InputKind::TouchUp, slot, 0, 0, 0);
}
// The gesture engine's held left button (a tap-drag in progress) rides in
// `held_buttons` above, so it was just flushed — here we only forget its state.
self.gestures.reset();
true
}
@@ -180,4 +209,136 @@ impl Capture {
}
self.scroll_acc = (ax, ay);
}
/// The compact wire touch id for an SDL finger — its existing slot, or the lowest free
/// one (contacts are few, so a linear scan is nothing). Held until the finger lifts.
fn touch_slot(&mut self, finger_id: u64) -> u32 {
if let Some(&slot) = self.touch_slots.get(&finger_id) {
return slot;
}
let used: HashSet<u32> = self.touch_slots.values().copied().collect();
let slot = (0u32..).find(|s| !used.contains(s)).unwrap_or(0);
self.touch_slots.insert(finger_id, slot);
slot
}
/// Touch flags pack the client surface size the coordinates are relative to, so the
/// host can rescale into its output — identical layout to Android's nativeSendTouch.
fn touch_flags(w: u32, h: u32) -> u32 {
((w & 0xffff) << 16) | (h & 0xffff)
}
/// A new touchscreen contact — `x`/`y` are absolute in the `w`×`h` content surface.
/// Ignored unless captured (the stream owns the glass; the menu is gamepad-driven).
pub fn on_touch_down(&mut self, finger_id: u64, x: i32, y: i32, w: u32, h: u32) {
if !self.captured {
return;
}
let slot = self.touch_slot(finger_id);
send(
&self.connector,
InputKind::TouchDown,
slot,
x,
y,
Self::touch_flags(w, h),
);
}
/// A contact moved. Only forwarded for a finger we already sent a down for — a move
/// with no live slot (capture engaged mid-touch) would have no matching host contact.
pub fn on_touch_move(&mut self, finger_id: u64, x: i32, y: i32, w: u32, h: u32) {
if !self.captured {
return;
}
if let Some(&slot) = self.touch_slots.get(&finger_id) {
send(
&self.connector,
InputKind::TouchMove,
slot,
x,
y,
Self::touch_flags(w, h),
);
}
}
/// A contact lifted — release its slot and the host contact. Forwarded even when not
/// captured: a `release()` may have already flushed it (then the slot is gone and this
/// no-ops), but a stray up must never strand a pressed contact on the host.
pub fn on_touch_up(&mut self, finger_id: u64) {
if let Some(slot) = self.touch_slots.remove(&finger_id) {
send(&self.connector, InputKind::TouchUp, slot, 0, 0, 0);
}
}
/// Route one forwarded touchscreen finger by the session's touch model. `wx`/`wy` are
/// physical window pixels (the trackpad ballistics + gesture geometry); `abs` is the same
/// finger mapped into the letterboxed content rect (pointer moves + raw passthrough). In
/// `Touch` mode fingers go on the wire as real contacts; in `Trackpad`/`Pointer` they
/// drive the gesture engine. Returns true when a three-finger tap asks to cycle the stats
/// overlay — the only signal the run loop must act on.
pub fn dispatch_finger(
&mut self,
phase: FingerPhase,
id: u64,
wx: f32,
wy: f32,
abs: Abs,
t_ms: f64,
) -> bool {
match self.touch_mode {
TouchMode::Touch => {
match phase {
FingerPhase::Down => self.on_touch_down(id, abs.x, abs.y, abs.w, abs.h),
FingerPhase::Move => self.on_touch_move(id, abs.x, abs.y, abs.w, abs.h),
FingerPhase::Up => self.on_touch_up(id),
}
false
}
TouchMode::Trackpad | TouchMode::Pointer => {
// Down/Move only while captured (the stream owns the glass); an Up always runs
// so a lift can conclude a gesture / release a held drag even if capture just
// dropped (focus loss mid-touch).
if !self.captured && phase != FingerPhase::Up {
return false;
}
let acts = match phase {
FingerPhase::Down => self.gestures.down(id, wx, wy, abs, t_ms),
FingerPhase::Move => self.gestures.motion(id, wx, wy, abs, t_ms),
FingerPhase::Up => self.gestures.up(id, t_ms),
};
let mut cycle_stats = false;
for act in acts {
cycle_stats |= self.apply_touch_act(act);
}
cycle_stats
}
}
}
/// Send one gesture [`Act`] on the wire, tracking button holds in `held_buttons` so a
/// capture release flushes them (a tap-drag's left button never sticks down). Returns
/// true for [`Act::CycleStats`], which is a run-loop signal, not a wire event.
fn apply_touch_act(&mut self, act: Act) -> bool {
match act {
Act::CycleStats => return true,
Act::Button { gs, down } => {
if down {
self.flush_motion(); // the press lands where the cursor now is
self.held_buttons.insert(gs);
send(&self.connector, InputKind::MouseButtonDown, gs, 0, 0, 0);
} else if self.held_buttons.remove(&gs) {
self.flush_motion();
send(&self.connector, InputKind::MouseButtonUp, gs, 0, 0, 0);
}
}
other => {
if let Some((kind, code, x, y, flags)) = other.wire() {
send(&self.connector, kind, code, x, y, flags);
}
}
}
false
}
}
+2
View File
@@ -29,6 +29,8 @@ pub mod overlay;
#[cfg(any(target_os = "linux", windows))]
mod run;
#[cfg(any(target_os = "linux", windows))]
pub mod touch;
#[cfg(any(target_os = "linux", windows))]
pub mod vk;
#[cfg(windows)]
mod win32;
+192 -20
View File
@@ -13,13 +13,14 @@
//! the stdout line always carries the full Detailed text so parsers see a stable
//! shape). Logs go to stderr (the binary configures tracing so).
use crate::input::Capture;
use crate::input::{Capture, FingerPhase};
use crate::overlay::{FrameCtx, Overlay, OverlayAction, OverlayFrame, SessionPhase};
use crate::touch::Abs;
use crate::vk::{FrameInput, Presenter};
use anyhow::{Context as _, Result};
use pf_client_core::gamepad::GamepadService;
use pf_client_core::session::{self, SessionEvent, SessionHandle, SessionParams, Stats};
use pf_client_core::trust::StatsVerbosity;
use pf_client_core::trust::{StatsVerbosity, TouchMode};
use pf_client_core::video::VulkanDecodeDevice;
use pf_client_core::video::{DecodedFrame, DecodedImage};
use punktfunk_core::client::NativeClient;
@@ -43,6 +44,10 @@ pub struct SessionOpts {
/// Stats overlay tier at start — gates the OSD panel AND the stdout `stats:` lines
/// (Ctrl+Alt+Shift+S cycles Off → Compact → Normal → Detailed live).
pub stats_verbosity: StatsVerbosity,
/// Touchscreen input model (Deck/tablet): `Trackpad` (relative cursor + gestures),
/// `Pointer` (absolute cursor), or `Touch` (real multi-touch passthrough). Latched per
/// session — a mouse-only client leaves this at the default and never sees a finger.
pub touch_mode: TouchMode,
/// Emit the `{"ready":true}` stdout line after the first presented frame.
pub json_status: bool,
/// Called once on `Connected` with the host's fingerprint (trust persistence is the
@@ -204,6 +209,11 @@ struct StreamState {
/// Resize-in-progress overlay (scrim + spinner) — armed by [`resize_tick`] when it
/// requests a switch, cleared when a decoded frame reaches the target (or on timeout).
resize_overlay: ResizeIndicator,
/// The last presented frame's video dimensions — the source rect touch passthrough
/// maps a finger into (the video is letterboxed within the window, so a finger's
/// window-normalized position must be re-based onto the content rect). `None` until
/// the first frame; touches before then have nothing to map onto and are dropped.
last_video: Option<(u32, u32)>,
}
impl StreamState {
@@ -253,6 +263,7 @@ impl StreamState {
resize_requested: None,
shown_mode: None,
resize_overlay: ResizeIndicator::default(),
last_video: None,
}
}
@@ -299,6 +310,13 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
#[cfg(windows)]
crate::win32::set_app_user_model_id();
sdl3::hint::set("SDL_JOYSTICK_THREAD", "1");
// A touchscreen (the Deck's glass) is forwarded as REAL touch passthrough below — so
// suppress SDL's default synthesis of mouse events from touch. Left on, every touch
// ALSO warps a synthetic mouse to the touch point, which under the stream's relative
// mouse lock becomes a large positive delta that walks the host cursor into the
// bottom-right corner (the reported bug). The menu/library is keyboard+gamepad-driven
// and consumes no mouse, so nothing wanted these synthetic events anyway.
sdl3::hint::set("SDL_TOUCH_MOUSE_EVENTS", "0");
let sdl = sdl3::init().context("SDL init")?;
let video = sdl.video().context("SDL video")?;
let events = sdl.event().context("SDL events")?;
@@ -512,24 +530,8 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
continue;
}
if chord && sc == Scancode::S {
stats_verbosity = stats_verbosity.next();
bump_stats_tier(&mut stats_verbosity, &mut stream, &presenter);
tracing::info!(tier = ?stats_verbosity, "chord: stats verbosity");
// Re-render the OSD from the last window immediately — waiting
// for the next Stats event would lag the keypress by up to 1 s.
if let Some(st) = &mut stream {
let text = match &st.last_stats {
Some(s) => stats_text(
stats_verbosity,
&st.mode_line,
s,
&st.presented,
st.hdr,
presenter.hdr_active(),
),
None => String::new(),
};
st.osd_text = text;
}
continue;
}
// F11 or Alt+Enter (some keyboards' Fn layer sends a media key for
@@ -581,6 +583,54 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
cap.on_wheel(x, y);
}
}
// Touchscreen fingers (the Deck's glass) → the session's touch model
// (Trackpad/Pointer mouse, or real Touch passthrough), routed by `Capture`.
// `x`/`y` are window-normalized (0..1); the dispatcher gets physical window
// pixels AND the letterbox mapping. Only DIRECT devices (touchscreens) — an
// INDIRECT trackpad drives the mouse and must not be mistaken for one. A
// three-finger tap returns `cycle` → bump the stats tier, same as Ctrl+⌥+⇧+S.
Event::FingerDown {
touch_id,
finger_id,
x,
y,
timestamp,
..
} => {
if is_direct_touch(touch_id)
&& dispatch_finger(FingerPhase::Down, &window, &mut stream, finger_id, x, y, timestamp)
{
bump_stats_tier(&mut stats_verbosity, &mut stream, &presenter);
}
}
Event::FingerMotion {
touch_id,
finger_id,
x,
y,
timestamp,
..
} => {
if is_direct_touch(touch_id)
&& dispatch_finger(FingerPhase::Move, &window, &mut stream, finger_id, x, y, timestamp)
{
bump_stats_tier(&mut stats_verbosity, &mut stream, &presenter);
}
}
Event::FingerUp {
touch_id,
finger_id,
x,
y,
timestamp,
..
} => {
if is_direct_touch(touch_id)
&& dispatch_finger(FingerPhase::Up, &window, &mut stream, finger_id, x, y, timestamp)
{
bump_stats_tier(&mut stats_verbosity, &mut stream, &presenter);
}
}
// The wake forwarder's FrameWake (and any other user event): pure
// wake-up — the frame drain below runs this iteration either way.
Event::User { .. } => {}
@@ -713,7 +763,7 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
.ok();
gamepad.attach(c.clone());
st.clock_offset = Some(c.clock_offset_shared());
let mut cap = Capture::new(c.clone());
let mut cap = Capture::new(c.clone(), opts.touch_mode);
cap.engage(); // capture engages when the stream starts (ui_stream parity)
apply_capture(&mut window, &mouse, true);
st.capture = Some(cap);
@@ -887,6 +937,7 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
// picture is here — lift the scrim. A no-op unless a switch is in flight.
let (fw, fh) = f.image.dimensions();
st.resize_overlay.decoded(fw, fh);
st.last_video = Some((fw, fh)); // touch passthrough's source rect
let DecodedFrame {
pts_ns,
decoded_ns,
@@ -1291,6 +1342,94 @@ fn apply_capture(window: &mut sdl3::video::Window, mouse: &sdl3::mouse::MouseUti
window.set_keyboard_grab(on);
}
/// Is this SDL touch device a real touchscreen (DIRECT, window-relative coordinates)?
/// Trackpads report INDIRECT and drive the mouse — their finger events must not be
/// forwarded as touch passthrough. An unknown/invalid id (INVALID) reads as not-direct.
fn is_direct_touch(touch_id: u64) -> bool {
use sdl3::sys::touch::{SDL_GetTouchDeviceType, SDL_TouchDeviceType, SDL_TouchID};
unsafe { SDL_GetTouchDeviceType(SDL_TouchID(touch_id)) == SDL_TouchDeviceType::DIRECT }
}
/// Route one SDL touchscreen finger into the active session's [`Capture`] per the touch
/// model. SDL delivers window-normalized `x`/`y` (0..1) and a nanosecond `timestamp`; the
/// dispatcher hands `Capture` physical window pixels (trackpad ballistics + gesture geometry)
/// AND the finger mapped into the letterboxed content rect (pointer moves + raw passthrough).
/// Returns whether a three-finger tap asked to cycle the stats tier. Down/Move before the
/// first decoded frame have nothing to map onto and are dropped; an Up always dispatches so a
/// lift can release a held contact/drag.
fn dispatch_finger(
phase: FingerPhase,
window: &sdl3::video::Window,
stream: &mut Option<StreamState>,
finger_id: u64,
x: f32,
y: f32,
timestamp: u64,
) -> bool {
let Some(st) = stream.as_mut() else {
return false;
};
let (pw, ph) = window.size_in_pixels();
let (wx, wy) = (x * pw as f32, y * ph as f32);
let abs = match st.last_video {
Some(video) => {
let (ax, ay, aw, ah) = finger_to_content((pw, ph), video, x, y);
Abs { x: ax, y: ay, w: aw, h: ah }
}
None if phase == FingerPhase::Up => Abs { x: 0, y: 0, w: 0, h: 0 },
None => return false,
};
let Some(cap) = st.capture.as_mut() else {
return false;
};
cap.dispatch_finger(phase, finger_id, wx, wy, abs, timestamp as f64 / 1_000_000.0)
}
/// Advance the stats-overlay tier and re-render the OSD immediately from the last window
/// (waiting for the next Stats event would lag the trigger by up to 1 s). Shared by the
/// Ctrl+Alt+Shift+S chord and the three-finger touch tap.
fn bump_stats_tier(
verbosity: &mut StatsVerbosity,
stream: &mut Option<StreamState>,
presenter: &Presenter,
) {
*verbosity = verbosity.next();
if let Some(st) = stream {
st.osd_text = match &st.last_stats {
Some(s) => stats_text(
*verbosity,
&st.mode_line,
s,
&st.presented,
st.hdr,
presenter.hdr_active(),
),
None => String::new(),
};
}
}
/// The pure Contain-fit mapping (window pixels in, content pixels out) — split out so the
/// letterbox math is testable without a live SDL window. Mirrors
/// [`vk::letterbox`]; a finger in the letterbox bars clamps to the nearest content edge.
fn finger_to_content(
surface: (u32, u32),
video: (u32, u32),
x: f32,
y: f32,
) -> (i32, i32, u32, u32) {
let (pw, ph) = (f64::from(surface.0), f64::from(surface.1));
let (vw, vh) = video;
let scale = (pw / f64::from(vw.max(1))).min(ph / f64::from(vh.max(1)));
let dw = (f64::from(vw) * scale).max(1.0);
let dh = (f64::from(vh) * scale).max(1.0);
let ox = (pw - dw) / 2.0;
let oy = (ph - dh) / 2.0;
let cx = ((f64::from(x) * pw - ox) / dw).clamp(0.0, 1.0) * dw;
let cy = ((f64::from(y) * ph - oy) / dh).clamp(0.0, 1.0) * dh;
(cx.round() as i32, cy.round() as i32, dw as u32, dh as u32)
}
/// The presenter's share of the unified stats window — folded into each printed line.
#[derive(Default)]
struct PresentedWindow {
@@ -1614,4 +1753,37 @@ mod tests {
"120 fps · 24 Mb/s"
);
}
#[test]
fn finger_maps_across_a_perfectly_filled_surface() {
// Video exactly fills the window (no letterbox): normalized finger → content
// corners/center map straight through, and the surface size is the video size.
let video = (1920, 1080);
assert_eq!(finger_to_content((1920, 1080), video, 0.0, 0.0), (0, 0, 1920, 1080));
assert_eq!(
finger_to_content((1920, 1080), video, 1.0, 1.0),
(1920, 1080, 1920, 1080)
);
assert_eq!(
finger_to_content((1920, 1080), video, 0.5, 0.5),
(960, 540, 1920, 1080)
);
}
#[test]
fn finger_rebases_onto_the_letterboxed_content_rect() {
// 16:9 video in the Deck's 16:10 glass (1280×800) letterboxes: content is
// 1280×720, centered with 40px bars top/bottom. A finger at the window's vertical
// center is the content's vertical center; a finger inside the top bar clamps to
// the content's top edge (not a negative coordinate).
let surface = (1280, 800);
let video = (1920, 1080);
let (_, cy, w, h) = finger_to_content(surface, video, 0.5, 0.5);
assert_eq!((w, h), (1280, 720));
assert_eq!(cy, 360);
// y=0.01 → window pixel 8, above the 40px bar → clamps to content top (0).
assert_eq!(finger_to_content(surface, video, 0.5, 0.01), (640, 0, 1280, 720));
// Bottom-right corner of the video content.
assert_eq!(finger_to_content(surface, video, 1.0, 1.0), (1280, 720, 1280, 720));
}
}
+463
View File
@@ -0,0 +1,463 @@
//! Touchscreen fingers → host mouse for the `trackpad`/`pointer` touch-input models — an
//! incremental port of the Android client's gesture engine (clients/android
//! `TouchInput.kt`) and its Apple twin (`TouchMouse.swift`) so all three touch clients feel
//! identical. The third model, `touch`, never reaches here: those fingers go on the wire as
//! real multi-touch contacts (`Capture::on_touch_*`).
//!
//! Two mouse models share one gesture vocabulary:
//! * **trackpad** (default): the cursor STAYS PUT on touch-down and moves by the finger's
//! relative delta with mild acceleration — swipe to nudge, lift and re-swipe to walk it
//! across, tap to click where it is. What makes a cursor reachable on a small screen.
//! * **pointer**: the cursor jumps to the finger and follows it (absolute moves through the
//! aspect-fit letterbox) — direct pointing.
//!
//! Shared gestures: tap = left click · two-finger tap = right click · two-finger drag =
//! scroll · tap-then-press-and-drag = held left drag · three-finger tap = cycle the stats
//! overlay tier.
//!
//! Unlike the Android/Apple hosts (which hand the engine a whole event's worth of changed
//! touches at once), SDL delivers ONE finger transition per event, so this is a strictly
//! incremental state machine: it keeps every live finger's position and recomputes the
//! centroid itself. Positions are in physical window pixels (the caller multiplies SDL's
//! normalized 0..1 finger coordinates by the window's pixel size) so the pixel-based
//! ballistics constants port from Android 1:1; timestamps are milliseconds.
use punktfunk_core::input::InputKind;
use std::collections::HashMap;
// Gesture/ballistics tuning (physical px / ms), matching the Android reference exactly.
/// Movement under this (px) still counts as a tap, not a drag.
const TAP_SLOP: f32 = 12.0;
/// A new touch this soon (ms) after a tap, near it, starts a held left-button drag.
const TAP_DRAG_MS: f64 = 250.0;
/// Two-finger pan distance (px) per 120-unit wheel notch (smaller = faster scroll).
const SCROLL_DIV: f32 = 4.0;
/// Base finger-px → host-px gain (~1:1, never twitchy).
const POINTER_SENS: f32 = 1.3;
/// Above `ACCEL_SPEED_FLOOR` px/ms the gain ramps by `ACCEL_GAIN` per px/ms, capped at
/// `ACCEL_MAX` so a fast swipe can't fling the cursor uncontrollably.
const ACCEL_GAIN: f32 = 0.6;
const ACCEL_SPEED_FLOOR: f32 = 0.3;
const ACCEL_MAX: f32 = 3.0;
/// GameStream mouse button ids.
const BTN_LEFT: u32 = 1;
const BTN_RIGHT: u32 = 3;
/// A finger's position in the letterboxed video content rect (absolute host pixels + the
/// content surface size) — what `pointer` mode's absolute moves carry. Mirrors the
/// `MouseMoveAbs` packing the host rescales into its output.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Abs {
pub x: i32,
pub y: i32,
pub w: u32,
pub h: u32,
}
/// A wire intent the engine emits; the owner ([`Capture`](crate::input::Capture)) translates
/// each into an actual `send_input`, and folds [`CycleStats`](Act::CycleStats) back to the
/// run loop.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Act {
/// Relative cursor motion (`MouseMove`).
MoveRel { dx: i32, dy: i32 },
/// Absolute cursor position through the letterbox (`MouseMoveAbs`).
MoveAbs(Abs),
/// A mouse button transition (`gs` = GameStream id; `down` = press/release).
Button { gs: u32, down: bool },
/// A wheel step: `axis` 0 = vertical, 1 = horizontal; `delta` in WHEEL(120) units.
Scroll { axis: u32, delta: i32 },
/// Three-finger tap: cycle the stats-overlay verbosity tier (the run loop owns it).
CycleStats,
}
impl Act {
/// The `(InputKind, code, x, y, flags)` this intent sends. `Button`/`CycleStats` don't map
/// to a single motion send, so callers special-case them; this covers the motion/scroll
/// intents shared with the raw pointer path.
pub fn wire(self) -> Option<(InputKind, u32, i32, i32, u32)> {
match self {
Act::MoveRel { dx, dy } => Some((InputKind::MouseMove, 0, dx, dy, 0)),
Act::MoveAbs(a) => Some((
InputKind::MouseMoveAbs,
0,
a.x,
a.y,
((a.w & 0xffff) << 16) | (a.h & 0xffff),
)),
Act::Scroll { axis, delta } => Some((InputKind::MouseScroll, axis, delta, 0, 0)),
Act::Button { .. } | Act::CycleStats => None,
}
}
}
/// The trackpad/pointer gesture state machine. One per session; `trackpad` picks the model
/// (false = pointer). Fed only DIRECT touchscreen fingers.
pub struct Gestures {
trackpad: bool,
/// Live fingers → current window-pixel position (the centroid needs every finger, but a
/// move event only carries the one that changed).
positions: HashMap<u64, (f32, f32)>,
/// A gesture is in flight (≥ 1 finger down since the first touch).
active: bool,
start: (f32, f32),
max_fingers: usize,
moved: bool,
scrolling: bool,
/// Finger count the scroll centroid is anchored at — re-anchor on a count change so a
/// 2→3 transition isn't read as a scroll notch.
scroll_count: usize,
scroll_anchor: (f32, f32),
/// A tap-then-press-and-drag is holding the left button down for this whole gesture.
drag_held: bool,
// Trackpad relative-motion state: the tracked finger, its last position/time, and the
// sub-pixel remainder so a slow drag isn't lost to integer truncation.
track_id: Option<u64>,
prev: (f32, f32),
prev_t: f64,
carry: (f32, f32),
// Tap-drag arming: a quick tap leaves a window in which the next nearby touch drags.
last_tap_up: f64,
last_tap_pt: (f32, f32),
}
impl Gestures {
pub fn new(trackpad: bool) -> Gestures {
Gestures {
trackpad,
positions: HashMap::new(),
active: false,
start: (0.0, 0.0),
max_fingers: 0,
moved: false,
scrolling: false,
scroll_count: 0,
scroll_anchor: (0.0, 0.0),
drag_held: false,
track_id: None,
prev: (0.0, 0.0),
prev_t: 0.0,
carry: (0.0, 0.0),
last_tap_up: 0.0,
last_tap_pt: (0.0, 0.0),
}
}
/// A finger touched down. `abs` is its letterbox mapping (pointer mode jumps the cursor
/// there on the first finger). `t` is milliseconds.
pub fn down(&mut self, id: u64, wx: f32, wy: f32, abs: Abs, t: f64) -> Vec<Act> {
let mut acts = Vec::new();
let first = self.positions.is_empty() && !self.active;
self.positions.insert(id, (wx, wy));
if first {
self.active = true;
self.start = (wx, wy);
self.max_fingers = 0;
self.moved = false;
self.scrolling = false;
self.scroll_count = 0;
// A touch landing just after a quick tap nearby = tap-and-drag.
self.drag_held = t - self.last_tap_up < TAP_DRAG_MS
&& (wx - self.last_tap_pt.0).abs() < TAP_SLOP
&& (wy - self.last_tap_pt.1).abs() < TAP_SLOP;
self.last_tap_up = 0.0; // consume the arming either way
if !self.trackpad {
acts.push(Act::MoveAbs(abs)); // pointer: place the cursor before any press
}
if self.drag_held {
acts.push(Act::Button { gs: BTN_LEFT, down: true });
}
self.track_id = Some(id);
self.prev = (wx, wy);
self.prev_t = t;
self.carry = (0.0, 0.0);
}
self.max_fingers = self.max_fingers.max(self.positions.len());
acts
}
/// A finger moved.
pub fn motion(&mut self, id: u64, wx: f32, wy: f32, abs: Abs, t: f64) -> Vec<Act> {
if !self.active || !self.positions.contains_key(&id) {
return Vec::new();
}
self.positions.insert(id, (wx, wy));
if self.positions.len() >= 2 {
self.scroll_by_centroid()
} else if !self.scrolling {
// One finger, and the gesture never became a scroll (dropping back from two
// fingers to one must not jerk the cursor).
self.single_finger(id, wx, wy, abs, t)
} else {
Vec::new()
}
}
/// A finger lifted. Only when the LAST finger lifts does the gesture conclude (into a
/// click / drag-end / stats cycle). `t` is the up-time in milliseconds.
pub fn up(&mut self, id: u64, t: f64) -> Vec<Act> {
let mut acts = Vec::new();
self.positions.remove(&id);
if self.track_id == Some(id) {
self.track_id = None;
}
if !self.positions.is_empty() || !self.active {
return acts; // other fingers still down (or no live gesture)
}
self.active = false;
if self.drag_held {
self.drag_held = false;
acts.push(Act::Button { gs: BTN_LEFT, down: false }); // end the held drag
} else if !self.moved {
match self.max_fingers {
n if n >= 3 => acts.push(Act::CycleStats),
2 => {
acts.push(Act::Button { gs: BTN_RIGHT, down: true });
acts.push(Act::Button { gs: BTN_RIGHT, down: false });
}
_ => {
acts.push(Act::Button { gs: BTN_LEFT, down: true });
acts.push(Act::Button { gs: BTN_LEFT, down: false });
self.last_tap_up = t; // arm tap-drag
self.last_tap_pt = self.start;
}
}
}
acts
}
/// Forget all in-flight gesture state (capture release / session teardown). Any left
/// button the engine is holding is released by the owner's held-button flush, so this
/// only clears state — it never re-emits wire events.
pub fn reset(&mut self) {
self.positions.clear();
self.track_id = None;
self.active = false;
self.scrolling = false;
self.moved = false;
self.drag_held = false;
self.last_tap_up = 0.0;
}
/// Two (or more) fingers → scroll by the centroid delta; never move the cursor. Fires a
/// notch per `SCROLL_DIV` px of pan and re-anchors on fire; finger up scrolls up, finger
/// right scrolls right (the host WHEEL(120) convention).
fn scroll_by_centroid(&mut self) -> Vec<Act> {
let mut acts = Vec::new();
let n = self.positions.len() as f32;
let (mut sx, mut sy) = (0.0f32, 0.0f32);
for &(px, py) in self.positions.values() {
sx += px;
sy += py;
}
let (cx, cy) = (sx / n, sy / n);
// (Re-)anchor on scroll start AND whenever the finger count changes.
if !self.scrolling || self.positions.len() != self.scroll_count {
self.scrolling = true;
self.scroll_count = self.positions.len();
self.scroll_anchor = (cx, cy);
}
let notches_y = ((self.scroll_anchor.1 - cy) / SCROLL_DIV) as i32;
let notches_x = ((cx - self.scroll_anchor.0) / SCROLL_DIV) as i32;
if notches_y != 0 {
acts.push(Act::Scroll { axis: 0, delta: notches_y * 120 });
self.scroll_anchor.1 = cy;
self.moved = true;
}
if notches_x != 0 {
acts.push(Act::Scroll { axis: 1, delta: notches_x * 120 });
self.scroll_anchor.0 = cx;
self.moved = true;
}
acts
}
/// One finger, not scrolling: trackpad relative ballistics, or pointer absolute follow.
fn single_finger(&mut self, id: u64, wx: f32, wy: f32, abs: Abs, t: f64) -> Vec<Act> {
let mut acts = Vec::new();
if (wx - self.start.0).abs() > TAP_SLOP || (wy - self.start.1).abs() > TAP_SLOP {
self.moved = true;
}
if !self.trackpad {
acts.push(Act::MoveAbs(abs)); // the cursor follows the finger
return acts;
}
// Re-anchor (zero delta this frame) if the tracked finger changed, so lifting one of
// several fingers never jumps the cursor.
if self.track_id != Some(id) {
self.track_id = Some(id);
self.prev = (wx, wy);
self.prev_t = t;
return acts;
}
let dx = wx - self.prev.0;
let dy = wy - self.prev.1;
let dt_ms = (t - self.prev_t).max(1.0) as f32;
self.prev = (wx, wy);
self.prev_t = t;
let speed = dx.hypot(dy) / dt_ms; // finger px per ms
let accel = (1.0 + ACCEL_GAIN * (speed - ACCEL_SPEED_FLOOR).max(0.0)).min(ACCEL_MAX);
let gain = POINTER_SENS * accel;
self.carry.0 += dx * gain;
self.carry.1 += dy * gain;
let out_x = self.carry.0 as i32; // truncates toward zero → remainder kept with sign
let out_y = self.carry.1 as i32;
if out_x != 0 || out_y != 0 {
acts.push(Act::MoveRel { dx: out_x, dy: out_y });
self.carry.0 -= out_x as f32;
self.carry.1 -= out_y as f32;
}
acts
}
}
#[cfg(test)]
mod tests {
use super::*;
const ABS: Abs = Abs { x: 100, y: 200, w: 1280, h: 720 };
fn abs_at(x: i32, y: i32) -> Abs {
Abs { x, y, w: 1280, h: 720 }
}
#[test]
fn trackpad_tap_is_a_left_click_with_no_motion() {
let mut g = Gestures::new(true);
let mut acts = g.down(1, 50.0, 50.0, ABS, 0.0);
acts.extend(g.up(1, 40.0));
// A trackpad tap places no cursor and moves nothing — just a click.
assert_eq!(
acts,
vec![
Act::Button { gs: BTN_LEFT, down: true },
Act::Button { gs: BTN_LEFT, down: false },
]
);
}
#[test]
fn pointer_tap_places_the_cursor_then_clicks() {
let mut g = Gestures::new(false);
let mut acts = g.down(1, 50.0, 50.0, abs_at(640, 360), 0.0);
acts.extend(g.up(1, 40.0));
assert_eq!(
acts,
vec![
Act::MoveAbs(abs_at(640, 360)),
Act::Button { gs: BTN_LEFT, down: true },
Act::Button { gs: BTN_LEFT, down: false },
]
);
}
#[test]
fn two_finger_tap_is_a_right_click() {
let mut g = Gestures::new(true);
let mut acts = g.down(1, 50.0, 50.0, ABS, 0.0);
acts.extend(g.down(2, 80.0, 52.0, ABS, 5.0));
acts.extend(g.up(1, 40.0));
acts.extend(g.up(2, 42.0));
assert_eq!(
acts,
vec![
Act::Button { gs: BTN_RIGHT, down: true },
Act::Button { gs: BTN_RIGHT, down: false },
]
);
}
#[test]
fn three_finger_tap_cycles_stats() {
let mut g = Gestures::new(true);
let mut acts = g.down(1, 50.0, 50.0, ABS, 0.0);
acts.extend(g.down(2, 80.0, 50.0, ABS, 2.0));
acts.extend(g.down(3, 110.0, 50.0, ABS, 4.0));
acts.extend(g.up(1, 40.0));
acts.extend(g.up(2, 41.0));
acts.extend(g.up(3, 42.0));
assert_eq!(acts, vec![Act::CycleStats]);
}
#[test]
fn trackpad_drag_emits_relative_motion() {
let mut g = Gestures::new(true);
assert!(g.down(1, 100.0, 100.0, ABS, 0.0).is_empty());
// A big move over 16 ms — relative, with acceleration, so it should exceed 1:1.
let acts = g.motion(1, 140.0, 100.0, ABS, 16.0);
match acts.as_slice() {
[Act::MoveRel { dx, dy }] => {
assert!(*dx >= 40, "expected accelerated dx ≥ raw 40, got {dx}");
assert_eq!(*dy, 0);
}
other => panic!("expected one MoveRel, got {other:?}"),
}
// The gesture moved, so the lift is not a tap (no click).
assert!(g.up(1, 32.0).is_empty());
}
#[test]
fn pointer_motion_follows_the_finger_absolutely() {
let mut g = Gestures::new(false);
let _ = g.down(1, 100.0, 100.0, abs_at(300, 300), 0.0);
let acts = g.motion(1, 140.0, 120.0, abs_at(360, 340), 16.0);
assert_eq!(acts, vec![Act::MoveAbs(abs_at(360, 340))]);
}
#[test]
fn two_finger_pan_scrolls_by_the_centroid() {
let mut g = Gestures::new(true);
let _ = g.down(1, 100.0, 200.0, ABS, 0.0);
let _ = g.down(2, 120.0, 200.0, ABS, 2.0);
// Both fingers slide up 40 px → the centroid rises 40 px → +ve (finger-up) notches.
let a1 = g.motion(1, 100.0, 160.0, ABS, 10.0);
let a2 = g.motion(2, 120.0, 160.0, ABS, 12.0);
let scrolls: Vec<_> = a1.into_iter().chain(a2).collect();
assert!(
scrolls.iter().any(|a| matches!(a, Act::Scroll { axis: 0, delta } if *delta > 0)),
"expected an upward vertical scroll, got {scrolls:?}"
);
}
#[test]
fn tap_then_press_drag_holds_the_left_button() {
let mut g = Gestures::new(true);
// Tap at (50,50), lifting at t=10.
let _ = g.down(1, 50.0, 50.0, ABS, 0.0);
let click = g.up(1, 10.0);
assert_eq!(
click,
vec![
Act::Button { gs: BTN_LEFT, down: true },
Act::Button { gs: BTN_LEFT, down: false },
]
);
// A new touch nearby within the window arms a held drag: button down on touch, and
// the whole gesture holds it until the lift.
let down2 = g.down(2, 52.0, 51.0, ABS, 120.0);
assert_eq!(down2, vec![Act::Button { gs: BTN_LEFT, down: true }]);
let _ = g.motion(2, 90.0, 51.0, ABS, 140.0); // drag
let end = g.up(2, 160.0);
assert_eq!(end, vec![Act::Button { gs: BTN_LEFT, down: false }]);
}
#[test]
fn reset_clears_a_drag_without_re_emitting() {
let mut g = Gestures::new(true);
let _ = g.down(1, 50.0, 50.0, ABS, 0.0);
let _ = g.up(1, 5.0); // arm
let _ = g.down(2, 51.0, 50.0, ABS, 50.0); // drag begins (left held)
g.reset();
// After a reset a fresh tap is an ordinary click (no stuck drag state).
let mut acts = g.down(3, 400.0, 400.0, ABS, 500.0);
acts.extend(g.up(3, 510.0));
assert_eq!(
acts,
vec![
Act::Button { gs: BTN_LEFT, down: true },
Act::Button { gs: BTN_LEFT, down: false },
]
);
}
}