feat(core+host+client): cursor channel — remote-desktop sweep M2a+M2b
ci / web (pull_request) Successful in 1m8s
ci / docs-site (pull_request) Successful in 1m10s
apple / swift (pull_request) Successful in 1m23s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Failing after 6m44s
ci / bench (pull_request) Successful in 7m24s
android / android (pull_request) Failing after 7m47s
ci / rust (pull_request) Failing after 8m36s
windows / build (x86_64-pc-windows-msvc) (pull_request) Failing after 4m36s
ci / web (pull_request) Successful in 1m8s
ci / docs-site (pull_request) Successful in 1m10s
apple / swift (pull_request) Successful in 1m23s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Failing after 6m44s
ci / bench (pull_request) Successful in 7m24s
android / android (pull_request) Failing after 7m47s
ci / rust (pull_request) Failing after 8m36s
windows / build (x86_64-pc-windows-msvc) (pull_request) Failing after 4m36s
The host cursor stops riding the video and becomes a real OS cursor on the client (the Parsec/RDP model): pointer feel no longer pays the capture→encode→network→decode→present round trip. Wire (M2a): - Hello grows a client_caps trailing byte (CLIENT_CAP_CURSOR) after the fixed display_hdr block — presence disambiguated by remaining length, which caps the post-HDR tail at 27 bytes (documented); Welcome answers HOST_CAP_CURSOR (capable-and-asked, the 444/clipboard precedent). - CursorShape (0x50, control stream): serial + dims + hotspot + straight RGBA, ≤120px/side so the u16 frame always fits (128² would overshoot); client caches by serial — re-showing a known shape costs 14 bytes, not a bitmap (RDP pointer-cache for free). - CursorState (0xD0 datagram): serial + visible/relative_hint flags + position, sent once per encode-loop tick — latest-wins, self-healing under loss, no refresh timer. relative_hint is reserved for M3. - Client core: two new planes (control-task + datagram-task arms) → next_cursor_shape/next_cursor_state; connect() grows client_caps (C ABI passes 0 until the v11 cursor poll fns exist). Host (M2b, Linux portal only): - handshake::cursor_forward is THE predicate (client asked ∧ Linux ∧ compositor ≠ gamescope) — Welcome bit and session wiring both read it. - SessionPlan.cursor_blend goes false for a forwarding session; the encode loop ticks a CursorForwarder every iteration: shape-serial diff → control-task bridge (mirrors probe_result), state datagram → conn. - CursorOverlay/capture CursorState carry the hotspot through (nearest-neighbor downscale backstop for XL cursors, unit-tested). Presenter: - CursorChannel drains both planes per loop iteration; shapes become SDL color cursors (from_surface + hotspot), applied while the desktop mouse model is engaged; visibility follows the host; capture/released hands back the system cursor. Sessions advertise the cap when they START in desktop mode. Verified on .21: fmt + clippy -D warnings (7 crates) + tests green (core 218 incl. new wire roundtrips, host 245 incl. e2e + forwarder downscale tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -76,6 +76,8 @@ mod handshake;
|
||||
/// The mid-stream control task (plan §W1); `serve_session` spawns `control::run` after the
|
||||
/// handshake to multiplex renegotiation / speed-test control messages onto the data-plane channels.
|
||||
mod control;
|
||||
/// Cursor-forward channel (M2): the encode loop's shape/state emission.
|
||||
mod cursor_fwd;
|
||||
|
||||
/// The capture→encode→send data plane (plan §W1); `serve_session` dispatches the synthetic or
|
||||
/// virtual source here (`synthetic_stream` / `virtual_stream`) and hands the latter a
|
||||
@@ -943,6 +945,15 @@ async fn serve_session(
|
||||
// accepted ack as "the active mode is now X" and fixes itself; old clients just log it.
|
||||
let (reconfig_result_tx, reconfig_result_rx) =
|
||||
tokio::sync::mpsc::unbounded_channel::<Reconfigured>();
|
||||
// Cursor-forward bridge (M2): the encode loop diffs each frame's cursor serial and hands
|
||||
// changed SHAPES here; the control task (the control stream's sole writer) sends them.
|
||||
// Same shape as `probe_result_tx`. Wired even when the channel wasn't negotiated — it
|
||||
// just never fires then.
|
||||
let (cursor_shape_tx, cursor_shape_rx) =
|
||||
tokio::sync::mpsc::unbounded_channel::<punktfunk_core::quic::CursorShape>();
|
||||
// Negotiated cursor forwarding: MUST match the HOST_CAP_CURSOR bit the Welcome advertised
|
||||
// (handshake::cursor_forward is the single predicate both read).
|
||||
let cursor_forward = handshake::cursor_forward(hello.client_caps, compositor);
|
||||
// Adaptive FEC: the control task maps each client LossReport to a recovery percent and publishes
|
||||
// it here; the data-plane send loop reads + applies it per frame. Disabled (pinned) when
|
||||
// PUNKTFUNK_FEC_PCT is set. Seeded with the session's starting FEC so it's a no-op until a report.
|
||||
@@ -978,6 +989,7 @@ async fn serve_session(
|
||||
probe_tx,
|
||||
probe_result_rx,
|
||||
reconfig_result_rx,
|
||||
cursor_shape_rx,
|
||||
clip_enabled,
|
||||
clip,
|
||||
));
|
||||
@@ -1363,6 +1375,8 @@ async fn serve_session(
|
||||
fec_target: fec_target_dp,
|
||||
conn: conn_stream,
|
||||
timing_conn,
|
||||
cursor_forward,
|
||||
cursor_shape_tx,
|
||||
probe_seq,
|
||||
streamed_au,
|
||||
stats: stats_dp,
|
||||
@@ -2029,6 +2043,7 @@ mod tests {
|
||||
0, // video_codecs (HEVC-only)
|
||||
0, // preferred_codec
|
||||
None, // display_hdr
|
||||
0, // client_caps
|
||||
None, // launch
|
||||
None, // pin (TOFU)
|
||||
None, // identity (host doesn't require pairing)
|
||||
@@ -2199,6 +2214,7 @@ mod tests {
|
||||
0, // video_codecs (0 → HEVC-only)
|
||||
0, // preferred_codec (auto)
|
||||
None, // display_hdr
|
||||
0, // client_caps
|
||||
None, // launch
|
||||
None, // pin: TOFU — the operator's approval (not a PIN) authorizes this client
|
||||
Some((cert, key)),
|
||||
@@ -2266,6 +2282,7 @@ mod tests {
|
||||
0, // video_codecs
|
||||
0, // preferred_codec
|
||||
None, // display_hdr
|
||||
0, // client_caps
|
||||
None, // launch
|
||||
None,
|
||||
None,
|
||||
@@ -2295,6 +2312,7 @@ mod tests {
|
||||
0, // video_codecs
|
||||
0, // preferred_codec
|
||||
None, // display_hdr
|
||||
0, // client_caps
|
||||
None, // launch
|
||||
Some(host_fp),
|
||||
Some((cert.clone(), key.clone())),
|
||||
|
||||
@@ -30,6 +30,7 @@ pub(super) async fn run(
|
||||
probe_tx: std::sync::mpsc::Sender<ProbeRequest>,
|
||||
mut probe_result_rx: tokio::sync::mpsc::UnboundedReceiver<ProbeResult>,
|
||||
mut reconfig_result_rx: tokio::sync::mpsc::UnboundedReceiver<Reconfigured>,
|
||||
mut cursor_shape_rx: tokio::sync::mpsc::UnboundedReceiver<punktfunk_core::quic::CursorShape>,
|
||||
clip_enabled: Arc<AtomicBool>,
|
||||
clip: pf_clipboard::ClipCoord,
|
||||
) {
|
||||
@@ -262,6 +263,15 @@ pub(super) async fn run(
|
||||
break;
|
||||
}
|
||||
}
|
||||
shape = cursor_shape_rx.recv() => {
|
||||
// Cursor-forward bridge (M2): the encode loop diffed a new pointer bitmap.
|
||||
// Rare (shape changes are human-paced); ≤ ~58 KiB fits the u16 frame by
|
||||
// construction (cursor_fwd downscales).
|
||||
let Some(shape) = shape else { break }; // data plane gone
|
||||
if io::write_msg(&mut ctrl_send, &shape.encode()).await.is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
offer = clip_offer_rx.recv(), if !clip_offer_closed => {
|
||||
// Host copied → the coordinator minted a `ClipOffer`; forward it to the client
|
||||
// (only while sync is on — a race with a just-received disable would otherwise
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
//! Cursor-forward channel, host side (design/remote-desktop-sweep.md M2).
|
||||
//!
|
||||
//! When the session negotiated the cursor channel (client `CLIENT_CAP_CURSOR` met our
|
||||
//! `HOST_CAP_CURSOR`), the encoder stops blending the pointer into the video
|
||||
//! (`SessionPlan::cursor_blend = false`) and the encode loop forwards it out-of-band instead:
|
||||
//! the SHAPE (bitmap + hotspot, rare) rides the reliable control stream via the control-task
|
||||
//! bridge, per-tick STATE (position/visibility, 14 B) rides a lossy `0xD0` datagram — resent
|
||||
//! every iteration so loss self-heals with no refresh timer.
|
||||
|
||||
use punktfunk_core::quic::{
|
||||
encode_cursor_state_datagram, CursorShape, CursorState, CURSOR_SHAPE_MAX_SIDE, CURSOR_VISIBLE,
|
||||
};
|
||||
|
||||
/// Per-session forward state, owned by the encode loop (the thread that binds frames).
|
||||
pub(super) struct CursorForwarder {
|
||||
/// Serial of the last shape handed to the control-task bridge (`None` = none yet).
|
||||
sent_serial: Option<u64>,
|
||||
/// Last visible pointer position (hotspot point, frame px) — held across hidden spans so
|
||||
/// a hide still states WHERE the pointer was (the M3 reappear position).
|
||||
last_pos: (i32, i32),
|
||||
}
|
||||
|
||||
impl CursorForwarder {
|
||||
pub(super) fn new() -> CursorForwarder {
|
||||
CursorForwarder {
|
||||
sent_serial: None,
|
||||
last_pos: (0, 0),
|
||||
}
|
||||
}
|
||||
|
||||
/// Called once per encode-loop iteration with the bound frame's overlay (also on repeat
|
||||
/// iterations — the state datagram is the plane's loss heal, so it goes out every tick).
|
||||
/// `None` overlay = hidden pointer (or no bitmap yet): state only, `visible` clear.
|
||||
pub(super) fn tick(
|
||||
&mut self,
|
||||
cursor: Option<&pf_frame::CursorOverlay>,
|
||||
conn: &quinn::Connection,
|
||||
shape_tx: &tokio::sync::mpsc::UnboundedSender<CursorShape>,
|
||||
) {
|
||||
let flags = match cursor {
|
||||
Some(ov) => {
|
||||
if self.sent_serial != Some(ov.serial) {
|
||||
if let Some(shape) = shape_from_overlay(ov) {
|
||||
// Bridge full ⇒ control task gone ⇒ session is tearing down anyway.
|
||||
let _ = shape_tx.send(shape);
|
||||
self.sent_serial = Some(ov.serial);
|
||||
}
|
||||
}
|
||||
self.last_pos = (ov.x + ov.hot_x as i32, ov.y + ov.hot_y as i32);
|
||||
CURSOR_VISIBLE
|
||||
}
|
||||
None => 0,
|
||||
};
|
||||
let state = CursorState {
|
||||
serial: self.sent_serial.unwrap_or(0) as u32,
|
||||
flags,
|
||||
x: self.last_pos.0,
|
||||
y: self.last_pos.1,
|
||||
};
|
||||
let _ = conn.send_datagram(encode_cursor_state_datagram(&state).into());
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the wire shape from a capture overlay, integer-downscaling (nearest-neighbor) anything
|
||||
/// over [`CURSOR_SHAPE_MAX_SIDE`] so the message always fits the u16-length control frame.
|
||||
/// Real cursors are far under the cap — the scale path is a correctness backstop for XL
|
||||
/// accessibility cursors, not a quality path. `None` on a malformed overlay (short buffer).
|
||||
fn shape_from_overlay(ov: &pf_frame::CursorOverlay) -> Option<CursorShape> {
|
||||
let px = (ov.w as usize).checked_mul(ov.h as usize)?.checked_mul(4)?;
|
||||
if ov.w == 0 || ov.h == 0 || ov.rgba.len() < px {
|
||||
return None;
|
||||
}
|
||||
let max = CURSOR_SHAPE_MAX_SIDE as u32;
|
||||
let f = ov.w.max(ov.h).div_ceil(max).max(1);
|
||||
let (w, h) = (ov.w.div_ceil(f), ov.h.div_ceil(f));
|
||||
let rgba = if f == 1 {
|
||||
ov.rgba.as_ref().clone()
|
||||
} else {
|
||||
let mut out = Vec::with_capacity((w * h * 4) as usize);
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
let (sx, sy) = ((x * f).min(ov.w - 1), (y * f).min(ov.h - 1));
|
||||
let o = ((sy * ov.w + sx) * 4) as usize;
|
||||
out.extend_from_slice(&ov.rgba[o..o + 4]);
|
||||
}
|
||||
}
|
||||
out
|
||||
};
|
||||
Some(CursorShape {
|
||||
serial: ov.serial as u32,
|
||||
w: w as u16,
|
||||
h: h as u16,
|
||||
hot_x: (ov.hot_x / f).min(w - 1) as u16,
|
||||
hot_y: (ov.hot_y / f).min(h - 1) as u16,
|
||||
rgba,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
|
||||
fn overlay(w: u32, h: u32, hot: (u32, u32)) -> pf_frame::CursorOverlay {
|
||||
pf_frame::CursorOverlay {
|
||||
x: 10,
|
||||
y: 20,
|
||||
w,
|
||||
h,
|
||||
rgba: Arc::new((0..w * h * 4).map(|i| i as u8).collect()),
|
||||
serial: 3,
|
||||
hot_x: hot.0,
|
||||
hot_y: hot.1,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn small_shape_passes_through() {
|
||||
let s = shape_from_overlay(&overlay(32, 32, (4, 5))).unwrap();
|
||||
assert_eq!((s.w, s.h, s.hot_x, s.hot_y, s.serial), (32, 32, 4, 5, 3));
|
||||
assert_eq!(s.rgba.len(), 32 * 32 * 4);
|
||||
// Encodes within the u16 control-frame cap.
|
||||
assert!(s.encode().len() <= u16::MAX as usize);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn oversize_shape_downscales_with_hotspot() {
|
||||
// 256² → f = ceil(256/120) = 3 → 86² (256.div_ceil(3)), hotspot scales with it.
|
||||
let s = shape_from_overlay(&overlay(256, 256, (255, 0))).unwrap();
|
||||
assert!(s.w <= CURSOR_SHAPE_MAX_SIDE && s.h <= CURSOR_SHAPE_MAX_SIDE);
|
||||
assert_eq!(s.rgba.len(), s.w as usize * s.h as usize * 4);
|
||||
assert!(s.hot_x < s.w && s.hot_y < s.h);
|
||||
assert!(s.encode().len() <= u16::MAX as usize);
|
||||
// The scaled message must decode (dims within the cap).
|
||||
assert_eq!(CursorShape::decode(&s.encode()).unwrap(), s);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn short_buffer_rejected() {
|
||||
let mut ov = overlay(8, 8, (0, 0));
|
||||
ov.rgba = Arc::new(vec![0; 8]);
|
||||
assert!(shape_from_overlay(&ov).is_none());
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,22 @@
|
||||
|
||||
use super::*;
|
||||
|
||||
/// Whether this session forwards the cursor out-of-band (design/remote-desktop-sweep.md M2):
|
||||
/// the client asked ([`CLIENT_CAP_CURSOR`](punktfunk_core::quic::CLIENT_CAP_CURSOR)) AND the
|
||||
/// capture path can deliver cursor metadata separately from the frame — today that is the
|
||||
/// Linux portal `SPA_META_Cursor` path only: not gamescope (its capture paints no cursor at
|
||||
/// all), not Windows (DWM composites into the IDD frame — M2c). THE single predicate: the
|
||||
/// Welcome's `HOST_CAP_CURSOR` bit and the session's forwarding/blend-off wiring both read it,
|
||||
/// so they can never disagree.
|
||||
pub(super) fn cursor_forward(
|
||||
client_caps: u8,
|
||||
compositor: Option<crate::vdisplay::Compositor>,
|
||||
) -> bool {
|
||||
cfg!(target_os = "linux")
|
||||
&& client_caps & punktfunk_core::quic::CLIENT_CAP_CURSOR != 0
|
||||
&& compositor.is_some_and(|c| c != crate::vdisplay::Compositor::Gamescope)
|
||||
}
|
||||
|
||||
/// Run the Hello→Welcome→Start negotiation. Borrows the control streams (the caller keeps them for
|
||||
/// mid-stream renegotiation afterwards). `first` is the already-read first control message.
|
||||
#[allow(clippy::type_complexity, clippy::too_many_arguments)]
|
||||
@@ -444,6 +460,14 @@ pub(super) async fn negotiate(
|
||||
punktfunk_core::quic::HOST_CAP_CLIPBOARD
|
||||
} else {
|
||||
0
|
||||
}
|
||||
// Cursor channel granted (client asked + this capture path can deliver cursor
|
||||
// metadata out of the frame) — the client turns its local renderer on ONLY when
|
||||
// it sees this bit, and serve_session wires forwarding from the same predicate.
|
||||
| if cursor_forward(hello.client_caps, compositor) {
|
||||
punktfunk_core::quic::HOST_CAP_CURSOR
|
||||
} else {
|
||||
0
|
||||
},
|
||||
// The negotiated session AEAD (resolved above) + its 32-byte key toward a ChaCha
|
||||
// client; toward everyone else cipher 0 keeps the Welcome byte-identical to the
|
||||
|
||||
@@ -938,6 +938,14 @@ pub(super) struct SessionContext {
|
||||
/// thread emits one 0xCF datagram per AU (capture→sent µs) on it, so the client can split its
|
||||
/// `host+network` latency stage. `None` = older client, no emission.
|
||||
pub(super) timing_conn: Option<quinn::Connection>,
|
||||
/// The session negotiated the cursor channel (design/remote-desktop-sweep.md M2 —
|
||||
/// `handshake::cursor_forward`): the encoder does NOT blend the pointer into the video;
|
||||
/// the encode loop forwards shape (via `cursor_shape_tx`) + per-tick `0xD0` state instead.
|
||||
pub(super) cursor_forward: bool,
|
||||
/// SHAPE bridge to the control task (the control stream's sole writer) — mirrors
|
||||
/// `probe_result_tx`. Inert when `cursor_forward` is false.
|
||||
pub(super) cursor_shape_tx:
|
||||
tokio::sync::mpsc::UnboundedSender<punktfunk_core::quic::CursorShape>,
|
||||
/// The client advertised [`punktfunk_core::quic::VIDEO_CAP_PROBE_SEQ`]: speed-test bursts may
|
||||
/// run mid-session in the probe index space (its reassembler keeps a separate probe window).
|
||||
/// `false` = older client whose single-window reassembler would drop probe-space frames as
|
||||
@@ -987,7 +995,10 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
ctx.bit_depth,
|
||||
ctx.chroma,
|
||||
ctx.codec,
|
||||
ctx.compositor != pf_vdisplay::Compositor::Gamescope,
|
||||
// Blend the pointer into the video only where the capture HAS one (not gamescope) AND
|
||||
// the client is not drawing it locally (the M2 cursor channel — blending too would
|
||||
// show it twice).
|
||||
ctx.compositor != pf_vdisplay::Compositor::Gamescope && !ctx.cursor_forward,
|
||||
);
|
||||
// PyroWave rides the datagram-aligned wire mode (§4.4): every encoder this session opens
|
||||
// packetizes at the negotiated shard payload, so a lost datagram costs blocks, not frames.
|
||||
@@ -1021,6 +1032,8 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
fec_target,
|
||||
conn,
|
||||
timing_conn,
|
||||
cursor_forward,
|
||||
cursor_shape_tx,
|
||||
probe_seq,
|
||||
streamed_au,
|
||||
stats,
|
||||
@@ -1034,6 +1047,13 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
// reverts to whole-AU sends without touching the encoder's slicing knobs). The third gate —
|
||||
// whether the ENCODER actually chunks — is dynamic (`supports_chunked_poll`, per AU).
|
||||
let streamed_wire = streamed_au && std::env::var("PUNKTFUNK_STREAMED_AU").as_deref() != Ok("0");
|
||||
// Cursor-forward state (M2): shape-serial diffing + the per-tick 0xD0 state send. The
|
||||
// encoder was told not to blend (SessionPlan above), so from the first frame the client's
|
||||
// locally-drawn cursor is the only one.
|
||||
let mut cursor_fwd = cursor_forward.then(super::cursor_fwd::CursorForwarder::new);
|
||||
if cursor_forward {
|
||||
tracing::info!("cursor channel negotiated — forwarding shape/state, encoder blend off");
|
||||
}
|
||||
if streamed_wire {
|
||||
tracing::info!(
|
||||
"client accepts streamed AUs (VIDEO_CAP_STREAMED_AU) — chunked encoder output \
|
||||
@@ -1985,6 +2005,12 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
);
|
||||
}
|
||||
}
|
||||
// Cursor channel (M2): every iteration — new frame OR repeat — states the pointer
|
||||
// (self-healing under datagram loss) and forwards a changed shape via the control
|
||||
// bridge. `frame` is the newest bound frame either way.
|
||||
if let Some(fwd) = cursor_fwd.as_mut() {
|
||||
fwd.tick(frame.cursor.as_ref(), &conn, &cursor_shape_tx);
|
||||
}
|
||||
if perf && diag_at.elapsed() >= std::time::Duration::from_secs(2) {
|
||||
let secs = diag_at.elapsed().as_secs_f64();
|
||||
tracing::info!(
|
||||
|
||||
Reference in New Issue
Block a user