feat(core+host+client): cursor channel — remote-desktop sweep M2a+M2b
ci / web (pull_request) Successful in 1m8s
ci / docs-site (pull_request) Successful in 1m10s
apple / swift (pull_request) Successful in 1m23s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Failing after 6m44s
ci / bench (pull_request) Successful in 7m24s
android / android (pull_request) Failing after 7m47s
ci / rust (pull_request) Failing after 8m36s
windows / build (x86_64-pc-windows-msvc) (pull_request) Failing after 4m36s
ci / web (pull_request) Successful in 1m8s
ci / docs-site (pull_request) Successful in 1m10s
apple / swift (pull_request) Successful in 1m23s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Failing after 6m44s
ci / bench (pull_request) Successful in 7m24s
android / android (pull_request) Failing after 7m47s
ci / rust (pull_request) Failing after 8m36s
windows / build (x86_64-pc-windows-msvc) (pull_request) Failing after 4m36s
The host cursor stops riding the video and becomes a real OS cursor on the client (the Parsec/RDP model): pointer feel no longer pays the capture→encode→network→decode→present round trip. Wire (M2a): - Hello grows a client_caps trailing byte (CLIENT_CAP_CURSOR) after the fixed display_hdr block — presence disambiguated by remaining length, which caps the post-HDR tail at 27 bytes (documented); Welcome answers HOST_CAP_CURSOR (capable-and-asked, the 444/clipboard precedent). - CursorShape (0x50, control stream): serial + dims + hotspot + straight RGBA, ≤120px/side so the u16 frame always fits (128² would overshoot); client caches by serial — re-showing a known shape costs 14 bytes, not a bitmap (RDP pointer-cache for free). - CursorState (0xD0 datagram): serial + visible/relative_hint flags + position, sent once per encode-loop tick — latest-wins, self-healing under loss, no refresh timer. relative_hint is reserved for M3. - Client core: two new planes (control-task + datagram-task arms) → next_cursor_shape/next_cursor_state; connect() grows client_caps (C ABI passes 0 until the v11 cursor poll fns exist). Host (M2b, Linux portal only): - handshake::cursor_forward is THE predicate (client asked ∧ Linux ∧ compositor ≠ gamescope) — Welcome bit and session wiring both read it. - SessionPlan.cursor_blend goes false for a forwarding session; the encode loop ticks a CursorForwarder every iteration: shape-serial diff → control-task bridge (mirrors probe_result), state datagram → conn. - CursorOverlay/capture CursorState carry the hotspot through (nearest-neighbor downscale backstop for XL cursors, unit-tested). Presenter: - CursorChannel drains both planes per loop iteration; shapes become SDL color cursors (from_surface + hotspot), applied while the desktop mouse model is engaged; visibility follows the host; capture/released hands back the system cursor. Sessions advertise the cap when they START in desktop mode. Verified on .21: fmt + clippy -D warnings (7 crates) + tests green (core 218 incl. new wire roundtrips, host 245 incl. e2e + forwarder downscale tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
//! Client-side cursor rendering (design/remote-desktop-sweep.md M2): the host forwards the
|
||||
//! pointer's SHAPE (reliable control stream, cached by serial) and per-frame STATE (lossy
|
||||
//! `0xD0` — position/visibility), and WE draw it as a real OS cursor — pointer feel stops
|
||||
//! paying the video round-trip (the Parsec/RDP model). Active only when the session
|
||||
//! negotiated it (`HOST_CAP_CURSOR` in the Welcome — the host stopped compositing then) and
|
||||
//! only applied while the DESKTOP mouse model is engaged: under capture the pointer is
|
||||
//! relative-locked (SDL hides it) and games draw their own cursor in-frame.
|
||||
|
||||
use punktfunk_core::client::NativeClient;
|
||||
use punktfunk_core::quic::{CursorState, HOST_CAP_CURSOR};
|
||||
use sdl3::mouse::{Cursor, MouseUtil, SystemCursor};
|
||||
use std::collections::HashMap;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Shape serials cached at most — cursors cycle through a handful of shapes (arrow, I-beam,
|
||||
/// resize…); a runaway host can't grow the map past this (the cache resets, shapes re-arrive
|
||||
/// on the reliable stream via the serial-miss path).
|
||||
const SHAPE_CACHE_MAX: usize = 64;
|
||||
|
||||
pub struct CursorChannel {
|
||||
/// The Welcome carried `HOST_CAP_CURSOR` — the host forwards instead of compositing.
|
||||
negotiated: bool,
|
||||
/// Serial → built OS cursor. An SDL `Cursor` must outlive its `set()`, so the cache owns
|
||||
/// every shape ever applied this session (bounded by [`SHAPE_CACHE_MAX`]).
|
||||
shapes: HashMap<u32, Cursor>,
|
||||
/// The serial currently installed via `Cursor::set` (`None` = default/system cursor).
|
||||
installed: Option<u32>,
|
||||
/// Latest `0xD0` state (latest-wins across a drained batch).
|
||||
state: Option<CursorState>,
|
||||
}
|
||||
|
||||
impl CursorChannel {
|
||||
pub fn new(connector: &NativeClient) -> CursorChannel {
|
||||
let negotiated = connector.host_caps() & HOST_CAP_CURSOR != 0;
|
||||
if negotiated {
|
||||
tracing::info!("cursor channel negotiated — host cursor renders locally");
|
||||
}
|
||||
CursorChannel {
|
||||
negotiated,
|
||||
shapes: HashMap::new(),
|
||||
installed: None,
|
||||
state: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the host forwards the cursor this session (it no longer composites one).
|
||||
pub fn negotiated(&self) -> bool {
|
||||
self.negotiated
|
||||
}
|
||||
|
||||
/// Drain the two planes and apply the newest state — once per run-loop iteration.
|
||||
/// `desktop_active` = the desktop mouse model is engaged (captured + desktop): only then
|
||||
/// do we own the local cursor's shape/visibility; under capture SDL's relative mode owns
|
||||
/// it, and released the system cursor must look normal.
|
||||
pub fn pump(&mut self, connector: &NativeClient, mouse: &MouseUtil, desktop_active: bool) {
|
||||
if !self.negotiated {
|
||||
return;
|
||||
}
|
||||
while let Ok(shape) = connector.next_cursor_shape(Duration::ZERO) {
|
||||
if self.shapes.len() >= SHAPE_CACHE_MAX {
|
||||
// Degenerate host: reset — live shapes re-install via the serial-miss path.
|
||||
self.shapes.clear();
|
||||
self.installed = None;
|
||||
}
|
||||
let mut data = shape.rgba;
|
||||
let built = sdl3::surface::Surface::from_data(
|
||||
&mut data,
|
||||
shape.w as u32,
|
||||
shape.h as u32,
|
||||
shape.w as u32 * 4,
|
||||
sdl3::pixels::PixelFormat::RGBA32,
|
||||
)
|
||||
.map_err(|e| e.to_string())
|
||||
.and_then(|surf| {
|
||||
Cursor::from_surface(&surf, shape.hot_x as i32, shape.hot_y as i32)
|
||||
.map_err(|e| e.to_string())
|
||||
});
|
||||
match built {
|
||||
Ok(cursor) => {
|
||||
// A re-sent serial replaces its entry; force re-install if it's current.
|
||||
if self.installed == Some(shape.serial) {
|
||||
self.installed = None;
|
||||
}
|
||||
self.shapes.insert(shape.serial, cursor);
|
||||
}
|
||||
Err(e) => tracing::warn!(error = %e, w = shape.w, h = shape.h,
|
||||
"cursor shape rejected by SDL — keeping the previous cursor"),
|
||||
}
|
||||
}
|
||||
while let Ok(st) = connector.next_cursor_state(Duration::ZERO) {
|
||||
self.state = Some(st); // latest wins
|
||||
}
|
||||
|
||||
if !desktop_active {
|
||||
// Capture mode / released: hand the cursor back to the system default so a
|
||||
// released pointer over the window doesn't wear the host's shape.
|
||||
if self.installed.take().is_some() {
|
||||
Cursor::from_system(SystemCursor::Arrow)
|
||||
.map(|c| c.set())
|
||||
.ok();
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(st) = self.state else { return };
|
||||
if st.visible() && self.installed != Some(st.serial) {
|
||||
if let Some(cursor) = self.shapes.get(&st.serial) {
|
||||
cursor.set();
|
||||
self.installed = Some(st.serial);
|
||||
}
|
||||
// Serial miss: the (reliable) shape hasn't landed yet — keep the previous
|
||||
// cursor for the RTT rather than flashing default.
|
||||
}
|
||||
// Visibility follows the host (a host app hid its pointer ⇒ ours hides too). Queried,
|
||||
// not shadowed, so apply_capture's own show/hide calls can never desync us.
|
||||
if mouse.is_cursor_showing() != st.visible() {
|
||||
mouse.show_cursor(st.visible());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,8 @@
|
||||
|
||||
#[cfg(any(target_os = "linux", windows))]
|
||||
pub mod csc;
|
||||
#[cfg(any(target_os = "linux", windows))]
|
||||
pub mod cursor;
|
||||
#[cfg(windows)]
|
||||
pub mod d3d11;
|
||||
#[cfg(target_os = "linux")]
|
||||
|
||||
@@ -233,6 +233,9 @@ struct StreamState {
|
||||
/// window-normalized position must be re-based onto the content rect). `None` until
|
||||
/// the first frame; touches before then have nothing to map onto and are dropped.
|
||||
last_video: Option<(u32, u32)>,
|
||||
/// Client-side cursor rendering (M2 cursor channel) — created with the connector; inert
|
||||
/// when the host didn't negotiate the channel.
|
||||
cursor_chan: Option<crate::cursor::CursorChannel>,
|
||||
}
|
||||
|
||||
impl StreamState {
|
||||
@@ -263,6 +266,7 @@ impl StreamState {
|
||||
frames: wake_rx,
|
||||
connector: None,
|
||||
capture: None,
|
||||
cursor_chan: None,
|
||||
force_software,
|
||||
canceled: false,
|
||||
ready_announced: false,
|
||||
@@ -744,6 +748,17 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
|
||||
if let Some(cap) = stream.as_mut().and_then(|s| s.capture.as_mut()) {
|
||||
cap.flush_motion();
|
||||
}
|
||||
// Cursor channel (M2): drain forwarded shape/state and drive the local OS cursor —
|
||||
// only meaningful in the desktop mouse model (capture's relative lock hides it).
|
||||
if let Some(st) = stream.as_mut() {
|
||||
if let (Some(chan), Some(c)) = (st.cursor_chan.as_mut(), st.connector.as_ref()) {
|
||||
let desktop_active = st
|
||||
.capture
|
||||
.as_ref()
|
||||
.is_some_and(|cap| cap.captured() && cap.desktop());
|
||||
chan.pump(c, &mouse, desktop_active);
|
||||
}
|
||||
}
|
||||
|
||||
// Text input follows the overlay's editing state (edge-triggered).
|
||||
let want_text = overlay.as_ref().is_some_and(|o| o.text_input_active());
|
||||
@@ -888,6 +903,7 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
|
||||
cap.engage(); // capture engages when the stream starts (ui_stream parity)
|
||||
apply_capture(&mut window, &mouse, true, cap.desktop());
|
||||
st.capture = Some(cap);
|
||||
st.cursor_chan = Some(crate::cursor::CursorChannel::new(&c));
|
||||
st.connector = Some(c);
|
||||
if let Some(f) = opts.on_connected.as_mut() {
|
||||
f(fingerprint);
|
||||
|
||||
Reference in New Issue
Block a user