feat(core+host+client): cursor channel — remote-desktop sweep M2a+M2b
ci / web (pull_request) Successful in 1m8s
ci / docs-site (pull_request) Successful in 1m10s
apple / swift (pull_request) Successful in 1m23s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Failing after 6m44s
ci / bench (pull_request) Successful in 7m24s
android / android (pull_request) Failing after 7m47s
ci / rust (pull_request) Failing after 8m36s
windows / build (x86_64-pc-windows-msvc) (pull_request) Failing after 4m36s

The host cursor stops riding the video and becomes a real OS cursor on
the client (the Parsec/RDP model): pointer feel no longer pays the
capture→encode→network→decode→present round trip.

Wire (M2a):
- Hello grows a client_caps trailing byte (CLIENT_CAP_CURSOR) after the
  fixed display_hdr block — presence disambiguated by remaining length,
  which caps the post-HDR tail at 27 bytes (documented); Welcome answers
  HOST_CAP_CURSOR (capable-and-asked, the 444/clipboard precedent).
- CursorShape (0x50, control stream): serial + dims + hotspot + straight
  RGBA, ≤120px/side so the u16 frame always fits (128² would overshoot);
  client caches by serial — re-showing a known shape costs 14 bytes, not
  a bitmap (RDP pointer-cache for free).
- CursorState (0xD0 datagram): serial + visible/relative_hint flags +
  position, sent once per encode-loop tick — latest-wins, self-healing
  under loss, no refresh timer. relative_hint is reserved for M3.
- Client core: two new planes (control-task + datagram-task arms) →
  next_cursor_shape/next_cursor_state; connect() grows client_caps
  (C ABI passes 0 until the v11 cursor poll fns exist).

Host (M2b, Linux portal only):
- handshake::cursor_forward is THE predicate (client asked ∧ Linux ∧
  compositor ≠ gamescope) — Welcome bit and session wiring both read it.
- SessionPlan.cursor_blend goes false for a forwarding session; the
  encode loop ticks a CursorForwarder every iteration: shape-serial diff
  → control-task bridge (mirrors probe_result), state datagram → conn.
- CursorOverlay/capture CursorState carry the hotspot through
  (nearest-neighbor downscale backstop for XL cursors, unit-tested).

Presenter:
- CursorChannel drains both planes per loop iteration; shapes become
  SDL color cursors (from_surface + hotspot), applied while the desktop
  mouse model is engaged; visibility follows the host; capture/released
  hands back the system cursor. Sessions advertise the cap when they
  START in desktop mode.

Verified on .21: fmt + clippy -D warnings (7 crates) + tests green
(core 218 incl. new wire roundtrips, host 245 incl. e2e + forwarder
downscale tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 01:09:51 +02:00
parent 12df1388de
commit 1efb8aef74
25 changed files with 885 additions and 13 deletions
@@ -0,0 +1,144 @@
//! Cursor-forward channel, host side (design/remote-desktop-sweep.md M2).
//!
//! When the session negotiated the cursor channel (client `CLIENT_CAP_CURSOR` met our
//! `HOST_CAP_CURSOR`), the encoder stops blending the pointer into the video
//! (`SessionPlan::cursor_blend = false`) and the encode loop forwards it out-of-band instead:
//! the SHAPE (bitmap + hotspot, rare) rides the reliable control stream via the control-task
//! bridge, per-tick STATE (position/visibility, 14 B) rides a lossy `0xD0` datagram — resent
//! every iteration so loss self-heals with no refresh timer.
use punktfunk_core::quic::{
encode_cursor_state_datagram, CursorShape, CursorState, CURSOR_SHAPE_MAX_SIDE, CURSOR_VISIBLE,
};
/// Per-session forward state, owned by the encode loop (the thread that binds frames).
pub(super) struct CursorForwarder {
/// Serial of the last shape handed to the control-task bridge (`None` = none yet).
sent_serial: Option<u64>,
/// Last visible pointer position (hotspot point, frame px) — held across hidden spans so
/// a hide still states WHERE the pointer was (the M3 reappear position).
last_pos: (i32, i32),
}
impl CursorForwarder {
pub(super) fn new() -> CursorForwarder {
CursorForwarder {
sent_serial: None,
last_pos: (0, 0),
}
}
/// Called once per encode-loop iteration with the bound frame's overlay (also on repeat
/// iterations — the state datagram is the plane's loss heal, so it goes out every tick).
/// `None` overlay = hidden pointer (or no bitmap yet): state only, `visible` clear.
pub(super) fn tick(
&mut self,
cursor: Option<&pf_frame::CursorOverlay>,
conn: &quinn::Connection,
shape_tx: &tokio::sync::mpsc::UnboundedSender<CursorShape>,
) {
let flags = match cursor {
Some(ov) => {
if self.sent_serial != Some(ov.serial) {
if let Some(shape) = shape_from_overlay(ov) {
// Bridge full ⇒ control task gone ⇒ session is tearing down anyway.
let _ = shape_tx.send(shape);
self.sent_serial = Some(ov.serial);
}
}
self.last_pos = (ov.x + ov.hot_x as i32, ov.y + ov.hot_y as i32);
CURSOR_VISIBLE
}
None => 0,
};
let state = CursorState {
serial: self.sent_serial.unwrap_or(0) as u32,
flags,
x: self.last_pos.0,
y: self.last_pos.1,
};
let _ = conn.send_datagram(encode_cursor_state_datagram(&state).into());
}
}
/// Build the wire shape from a capture overlay, integer-downscaling (nearest-neighbor) anything
/// over [`CURSOR_SHAPE_MAX_SIDE`] so the message always fits the u16-length control frame.
/// Real cursors are far under the cap — the scale path is a correctness backstop for XL
/// accessibility cursors, not a quality path. `None` on a malformed overlay (short buffer).
fn shape_from_overlay(ov: &pf_frame::CursorOverlay) -> Option<CursorShape> {
let px = (ov.w as usize).checked_mul(ov.h as usize)?.checked_mul(4)?;
if ov.w == 0 || ov.h == 0 || ov.rgba.len() < px {
return None;
}
let max = CURSOR_SHAPE_MAX_SIDE as u32;
let f = ov.w.max(ov.h).div_ceil(max).max(1);
let (w, h) = (ov.w.div_ceil(f), ov.h.div_ceil(f));
let rgba = if f == 1 {
ov.rgba.as_ref().clone()
} else {
let mut out = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
let (sx, sy) = ((x * f).min(ov.w - 1), (y * f).min(ov.h - 1));
let o = ((sy * ov.w + sx) * 4) as usize;
out.extend_from_slice(&ov.rgba[o..o + 4]);
}
}
out
};
Some(CursorShape {
serial: ov.serial as u32,
w: w as u16,
h: h as u16,
hot_x: (ov.hot_x / f).min(w - 1) as u16,
hot_y: (ov.hot_y / f).min(h - 1) as u16,
rgba,
})
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
fn overlay(w: u32, h: u32, hot: (u32, u32)) -> pf_frame::CursorOverlay {
pf_frame::CursorOverlay {
x: 10,
y: 20,
w,
h,
rgba: Arc::new((0..w * h * 4).map(|i| i as u8).collect()),
serial: 3,
hot_x: hot.0,
hot_y: hot.1,
}
}
#[test]
fn small_shape_passes_through() {
let s = shape_from_overlay(&overlay(32, 32, (4, 5))).unwrap();
assert_eq!((s.w, s.h, s.hot_x, s.hot_y, s.serial), (32, 32, 4, 5, 3));
assert_eq!(s.rgba.len(), 32 * 32 * 4);
// Encodes within the u16 control-frame cap.
assert!(s.encode().len() <= u16::MAX as usize);
}
#[test]
fn oversize_shape_downscales_with_hotspot() {
// 256² → f = ceil(256/120) = 3 → 86² (256.div_ceil(3)), hotspot scales with it.
let s = shape_from_overlay(&overlay(256, 256, (255, 0))).unwrap();
assert!(s.w <= CURSOR_SHAPE_MAX_SIDE && s.h <= CURSOR_SHAPE_MAX_SIDE);
assert_eq!(s.rgba.len(), s.w as usize * s.h as usize * 4);
assert!(s.hot_x < s.w && s.hot_y < s.h);
assert!(s.encode().len() <= u16::MAX as usize);
// The scaled message must decode (dims within the cap).
assert_eq!(CursorShape::decode(&s.encode()).unwrap(), s);
}
#[test]
fn short_buffer_rejected() {
let mut ov = overlay(8, 8, (0, 0));
ov.rgba = Arc::new(vec![0; 8]);
assert!(shape_from_overlay(&ov).is_none());
}
}