feat(core+host+client): cursor channel — remote-desktop sweep M2a+M2b
The host cursor stops riding the video and becomes a real OS cursor on the client (the Parsec/RDP model): pointer feel no longer pays the capture→encode→network→decode→present round trip. Wire (M2a): - Hello grows a client_caps trailing byte (CLIENT_CAP_CURSOR) after the fixed display_hdr block — presence disambiguated by remaining length, which caps the post-HDR tail at 27 bytes (documented); Welcome answers HOST_CAP_CURSOR (capable-and-asked, the 444/clipboard precedent). - CursorShape (0x50, control stream): serial + dims + hotspot + straight RGBA, ≤120px/side so the u16 frame always fits (128² would overshoot); client caches by serial — re-showing a known shape costs 14 bytes, not a bitmap (RDP pointer-cache for free). - CursorState (0xD0 datagram): serial + visible/relative_hint flags + position, sent once per encode-loop tick — latest-wins, self-healing under loss, no refresh timer. relative_hint is reserved for M3. - Client core: two new planes (control-task + datagram-task arms) → next_cursor_shape/next_cursor_state; connect() grows client_caps (C ABI passes 0 until the v11 cursor poll fns exist). Host (M2b, Linux portal only): - handshake::cursor_forward is THE predicate (client asked ∧ Linux ∧ compositor ≠ gamescope) — Welcome bit and session wiring both read it. - SessionPlan.cursor_blend goes false for a forwarding session; the encode loop ticks a CursorForwarder every iteration: shape-serial diff → control-task bridge (mirrors probe_result), state datagram → conn. - CursorOverlay/capture CursorState carry the hotspot through (nearest-neighbor downscale backstop for XL cursors, unit-tested). Presenter: - CursorChannel drains both planes per loop iteration; shapes become SDL color cursors (from_surface + hotspot), applied while the desktop mouse model is engaged; visibility follows the host; capture/released hands back the system cursor. Sessions advertise the cap when they START in desktop mode. Verified on .21: fmt + clippy -D warnings (7 crates) + tests green (core 218 incl. new wire roundtrips, host 245 incl. e2e + forwarder downscale tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -938,6 +938,14 @@ pub(super) struct SessionContext {
|
||||
/// thread emits one 0xCF datagram per AU (capture→sent µs) on it, so the client can split its
|
||||
/// `host+network` latency stage. `None` = older client, no emission.
|
||||
pub(super) timing_conn: Option<quinn::Connection>,
|
||||
/// The session negotiated the cursor channel (design/remote-desktop-sweep.md M2 —
|
||||
/// `handshake::cursor_forward`): the encoder does NOT blend the pointer into the video;
|
||||
/// the encode loop forwards shape (via `cursor_shape_tx`) + per-tick `0xD0` state instead.
|
||||
pub(super) cursor_forward: bool,
|
||||
/// SHAPE bridge to the control task (the control stream's sole writer) — mirrors
|
||||
/// `probe_result_tx`. Inert when `cursor_forward` is false.
|
||||
pub(super) cursor_shape_tx:
|
||||
tokio::sync::mpsc::UnboundedSender<punktfunk_core::quic::CursorShape>,
|
||||
/// The client advertised [`punktfunk_core::quic::VIDEO_CAP_PROBE_SEQ`]: speed-test bursts may
|
||||
/// run mid-session in the probe index space (its reassembler keeps a separate probe window).
|
||||
/// `false` = older client whose single-window reassembler would drop probe-space frames as
|
||||
@@ -987,7 +995,10 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
ctx.bit_depth,
|
||||
ctx.chroma,
|
||||
ctx.codec,
|
||||
ctx.compositor != pf_vdisplay::Compositor::Gamescope,
|
||||
// Blend the pointer into the video only where the capture HAS one (not gamescope) AND
|
||||
// the client is not drawing it locally (the M2 cursor channel — blending too would
|
||||
// show it twice).
|
||||
ctx.compositor != pf_vdisplay::Compositor::Gamescope && !ctx.cursor_forward,
|
||||
);
|
||||
// PyroWave rides the datagram-aligned wire mode (§4.4): every encoder this session opens
|
||||
// packetizes at the negotiated shard payload, so a lost datagram costs blocks, not frames.
|
||||
@@ -1021,6 +1032,8 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
fec_target,
|
||||
conn,
|
||||
timing_conn,
|
||||
cursor_forward,
|
||||
cursor_shape_tx,
|
||||
probe_seq,
|
||||
streamed_au,
|
||||
stats,
|
||||
@@ -1034,6 +1047,13 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
// reverts to whole-AU sends without touching the encoder's slicing knobs). The third gate —
|
||||
// whether the ENCODER actually chunks — is dynamic (`supports_chunked_poll`, per AU).
|
||||
let streamed_wire = streamed_au && std::env::var("PUNKTFUNK_STREAMED_AU").as_deref() != Ok("0");
|
||||
// Cursor-forward state (M2): shape-serial diffing + the per-tick 0xD0 state send. The
|
||||
// encoder was told not to blend (SessionPlan above), so from the first frame the client's
|
||||
// locally-drawn cursor is the only one.
|
||||
let mut cursor_fwd = cursor_forward.then(super::cursor_fwd::CursorForwarder::new);
|
||||
if cursor_forward {
|
||||
tracing::info!("cursor channel negotiated — forwarding shape/state, encoder blend off");
|
||||
}
|
||||
if streamed_wire {
|
||||
tracing::info!(
|
||||
"client accepts streamed AUs (VIDEO_CAP_STREAMED_AU) — chunked encoder output \
|
||||
@@ -1985,6 +2005,12 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
);
|
||||
}
|
||||
}
|
||||
// Cursor channel (M2): every iteration — new frame OR repeat — states the pointer
|
||||
// (self-healing under datagram loss) and forwards a changed shape via the control
|
||||
// bridge. `frame` is the newest bound frame either way.
|
||||
if let Some(fwd) = cursor_fwd.as_mut() {
|
||||
fwd.tick(frame.cursor.as_ref(), &conn, &cursor_shape_tx);
|
||||
}
|
||||
if perf && diag_at.elapsed() >= std::time::Duration::from_secs(2) {
|
||||
let secs = diag_at.elapsed().as_secs_f64();
|
||||
tracing::info!(
|
||||
|
||||
Reference in New Issue
Block a user