feat(core+host+client): cursor channel — remote-desktop sweep M2a+M2b
ci / web (pull_request) Successful in 1m8s
ci / docs-site (pull_request) Successful in 1m10s
apple / swift (pull_request) Successful in 1m23s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Failing after 6m44s
ci / bench (pull_request) Successful in 7m24s
android / android (pull_request) Failing after 7m47s
ci / rust (pull_request) Failing after 8m36s
windows / build (x86_64-pc-windows-msvc) (pull_request) Failing after 4m36s

The host cursor stops riding the video and becomes a real OS cursor on
the client (the Parsec/RDP model): pointer feel no longer pays the
capture→encode→network→decode→present round trip.

Wire (M2a):
- Hello grows a client_caps trailing byte (CLIENT_CAP_CURSOR) after the
  fixed display_hdr block — presence disambiguated by remaining length,
  which caps the post-HDR tail at 27 bytes (documented); Welcome answers
  HOST_CAP_CURSOR (capable-and-asked, the 444/clipboard precedent).
- CursorShape (0x50, control stream): serial + dims + hotspot + straight
  RGBA, ≤120px/side so the u16 frame always fits (128² would overshoot);
  client caches by serial — re-showing a known shape costs 14 bytes, not
  a bitmap (RDP pointer-cache for free).
- CursorState (0xD0 datagram): serial + visible/relative_hint flags +
  position, sent once per encode-loop tick — latest-wins, self-healing
  under loss, no refresh timer. relative_hint is reserved for M3.
- Client core: two new planes (control-task + datagram-task arms) →
  next_cursor_shape/next_cursor_state; connect() grows client_caps
  (C ABI passes 0 until the v11 cursor poll fns exist).

Host (M2b, Linux portal only):
- handshake::cursor_forward is THE predicate (client asked ∧ Linux ∧
  compositor ≠ gamescope) — Welcome bit and session wiring both read it.
- SessionPlan.cursor_blend goes false for a forwarding session; the
  encode loop ticks a CursorForwarder every iteration: shape-serial diff
  → control-task bridge (mirrors probe_result), state datagram → conn.
- CursorOverlay/capture CursorState carry the hotspot through
  (nearest-neighbor downscale backstop for XL cursors, unit-tested).

Presenter:
- CursorChannel drains both planes per loop iteration; shapes become
  SDL color cursors (from_surface + hotspot), applied while the desktop
  mouse model is engaged; visibility follows the host; capture/released
  hands back the system cursor. Sessions advertise the cap when they
  START in desktop mode.

Verified on .21: fmt + clippy -D warnings (7 crates) + tests green
(core 218 incl. new wire roundtrips, host 245 incl. e2e + forwarder
downscale tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 01:09:51 +02:00
parent 12df1388de
commit 1efb8aef74
25 changed files with 885 additions and 13 deletions
+119
View File
@@ -0,0 +1,119 @@
//! Client-side cursor rendering (design/remote-desktop-sweep.md M2): the host forwards the
//! pointer's SHAPE (reliable control stream, cached by serial) and per-frame STATE (lossy
//! `0xD0` — position/visibility), and WE draw it as a real OS cursor — pointer feel stops
//! paying the video round-trip (the Parsec/RDP model). Active only when the session
//! negotiated it (`HOST_CAP_CURSOR` in the Welcome — the host stopped compositing then) and
//! only applied while the DESKTOP mouse model is engaged: under capture the pointer is
//! relative-locked (SDL hides it) and games draw their own cursor in-frame.
use punktfunk_core::client::NativeClient;
use punktfunk_core::quic::{CursorState, HOST_CAP_CURSOR};
use sdl3::mouse::{Cursor, MouseUtil, SystemCursor};
use std::collections::HashMap;
use std::time::Duration;
/// Shape serials cached at most — cursors cycle through a handful of shapes (arrow, I-beam,
/// resize…); a runaway host can't grow the map past this (the cache resets, shapes re-arrive
/// on the reliable stream via the serial-miss path).
const SHAPE_CACHE_MAX: usize = 64;
pub struct CursorChannel {
/// The Welcome carried `HOST_CAP_CURSOR` — the host forwards instead of compositing.
negotiated: bool,
/// Serial → built OS cursor. An SDL `Cursor` must outlive its `set()`, so the cache owns
/// every shape ever applied this session (bounded by [`SHAPE_CACHE_MAX`]).
shapes: HashMap<u32, Cursor>,
/// The serial currently installed via `Cursor::set` (`None` = default/system cursor).
installed: Option<u32>,
/// Latest `0xD0` state (latest-wins across a drained batch).
state: Option<CursorState>,
}
impl CursorChannel {
pub fn new(connector: &NativeClient) -> CursorChannel {
let negotiated = connector.host_caps() & HOST_CAP_CURSOR != 0;
if negotiated {
tracing::info!("cursor channel negotiated — host cursor renders locally");
}
CursorChannel {
negotiated,
shapes: HashMap::new(),
installed: None,
state: None,
}
}
/// Whether the host forwards the cursor this session (it no longer composites one).
pub fn negotiated(&self) -> bool {
self.negotiated
}
/// Drain the two planes and apply the newest state — once per run-loop iteration.
/// `desktop_active` = the desktop mouse model is engaged (captured + desktop): only then
/// do we own the local cursor's shape/visibility; under capture SDL's relative mode owns
/// it, and released the system cursor must look normal.
pub fn pump(&mut self, connector: &NativeClient, mouse: &MouseUtil, desktop_active: bool) {
if !self.negotiated {
return;
}
while let Ok(shape) = connector.next_cursor_shape(Duration::ZERO) {
if self.shapes.len() >= SHAPE_CACHE_MAX {
// Degenerate host: reset — live shapes re-install via the serial-miss path.
self.shapes.clear();
self.installed = None;
}
let mut data = shape.rgba;
let built = sdl3::surface::Surface::from_data(
&mut data,
shape.w as u32,
shape.h as u32,
shape.w as u32 * 4,
sdl3::pixels::PixelFormat::RGBA32,
)
.map_err(|e| e.to_string())
.and_then(|surf| {
Cursor::from_surface(&surf, shape.hot_x as i32, shape.hot_y as i32)
.map_err(|e| e.to_string())
});
match built {
Ok(cursor) => {
// A re-sent serial replaces its entry; force re-install if it's current.
if self.installed == Some(shape.serial) {
self.installed = None;
}
self.shapes.insert(shape.serial, cursor);
}
Err(e) => tracing::warn!(error = %e, w = shape.w, h = shape.h,
"cursor shape rejected by SDL — keeping the previous cursor"),
}
}
while let Ok(st) = connector.next_cursor_state(Duration::ZERO) {
self.state = Some(st); // latest wins
}
if !desktop_active {
// Capture mode / released: hand the cursor back to the system default so a
// released pointer over the window doesn't wear the host's shape.
if self.installed.take().is_some() {
Cursor::from_system(SystemCursor::Arrow)
.map(|c| c.set())
.ok();
}
return;
}
let Some(st) = self.state else { return };
if st.visible() && self.installed != Some(st.serial) {
if let Some(cursor) = self.shapes.get(&st.serial) {
cursor.set();
self.installed = Some(st.serial);
}
// Serial miss: the (reliable) shape hasn't landed yet — keep the previous
// cursor for the RTT rather than flashing default.
}
// Visibility follows the host (a host app hid its pointer ⇒ ours hides too). Queried,
// not shadowed, so apply_capture's own show/hide calls can never desync us.
if mouse.is_cursor_showing() != st.visible() {
mouse.show_cursor(st.visible());
}
}
}
+2
View File
@@ -16,6 +16,8 @@
#[cfg(any(target_os = "linux", windows))]
pub mod csc;
#[cfg(any(target_os = "linux", windows))]
pub mod cursor;
#[cfg(windows)]
pub mod d3d11;
#[cfg(target_os = "linux")]
+16
View File
@@ -233,6 +233,9 @@ struct StreamState {
/// window-normalized position must be re-based onto the content rect). `None` until
/// the first frame; touches before then have nothing to map onto and are dropped.
last_video: Option<(u32, u32)>,
/// Client-side cursor rendering (M2 cursor channel) — created with the connector; inert
/// when the host didn't negotiate the channel.
cursor_chan: Option<crate::cursor::CursorChannel>,
}
impl StreamState {
@@ -263,6 +266,7 @@ impl StreamState {
frames: wake_rx,
connector: None,
capture: None,
cursor_chan: None,
force_software,
canceled: false,
ready_announced: false,
@@ -744,6 +748,17 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
if let Some(cap) = stream.as_mut().and_then(|s| s.capture.as_mut()) {
cap.flush_motion();
}
// Cursor channel (M2): drain forwarded shape/state and drive the local OS cursor —
// only meaningful in the desktop mouse model (capture's relative lock hides it).
if let Some(st) = stream.as_mut() {
if let (Some(chan), Some(c)) = (st.cursor_chan.as_mut(), st.connector.as_ref()) {
let desktop_active = st
.capture
.as_ref()
.is_some_and(|cap| cap.captured() && cap.desktop());
chan.pump(c, &mouse, desktop_active);
}
}
// Text input follows the overlay's editing state (edge-triggered).
let want_text = overlay.as_ref().is_some_and(|o| o.text_input_active());
@@ -888,6 +903,7 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
cap.engage(); // capture engages when the stream starts (ui_stream parity)
apply_capture(&mut window, &mouse, true, cap.desktop());
st.capture = Some(cap);
st.cursor_chan = Some(crate::cursor::CursorChannel::new(&c));
st.connector = Some(c);
if let Some(f) = opts.on_connected.as_mut() {
f(fingerprint);