feat(core+host+driver+client): mid-stream cursor-render flip — host composites in the capture model
The cursor channel excluded the host pointer from the video for the whole session, but the client only draws it under the DESKTOP mouse model — flipping ⌃⌥⇧M to capture mid-stream left the session pointer-blind (user-found). The fix makes render ownership LIVE state: - wire: CursorRenderMode 0x51 (client→host, control stream) — client_draws: true = desktop model (host excludes + forwards), false = capture model / released (host composites, full fidelity — DWM on Windows incl. real XOR inversion, encoder blend on Linux). Sessions start client_draws=true (the pre-message behavior). - host: control task stores the flag; the encode loop edge-detects it — forwarding + frame.cursor strip while the client draws, quiet forwarder + overlay-into-blend while composited. SessionPlan now grants blend CAPABILITY wherever the capture has a pointer (dropping the !cursor_forward gate) so the composite side needs no encoder rebuild; per-tick frame.cursor decides what is drawn. - Windows: Capturer::set_cursor_forward → retained IOCTL_SET_CURSOR_FORWARD sender (proto v6) → driver (un)declares the IddCx hardware cursor on the LIVE monitor. Un-declare re-issues the setup with EMPTY caps (candidate mechanism — the DDI has no documented un-setup); resetup-on-commit is skipped while off, so a mode commit's software-cursor default is the fallback path. Failure against a pre-v6 driver logs and keeps declared-at-ADD behavior. - SDL client: one edge-detected reconciler at the cursor pump site — desktop-active = client draws; capture model or released = host composites. Covers the chord, the M3 auto-flip, and engage/release. - C ABI v12: punktfunk_connection_set_cursor_render (additive; wire version unchanged — pre-§8 hosts ignore the unknown message type). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -174,6 +174,26 @@ pub fn capture_virtual_output(
|
||||
},
|
||||
) as pf_capture::CursorChannelSender
|
||||
});
|
||||
// The mid-stream cursor-render flip (proto v6) — retained by the capturer so the client's
|
||||
// mouse-model chords can (un)declare the driver's hardware cursor live. Same facade contract
|
||||
// as `cursor_sender`.
|
||||
let cursor_forward_sender: Option<pf_capture::CursorForwardSender> =
|
||||
want.hw_cursor.then(|| {
|
||||
std::sync::Arc::new(
|
||||
move |req: &pf_driver_proto::control::SetCursorForwardRequest| {
|
||||
// SAFETY: `control_raw` is the pf-vdisplay control handle resolved above; it is
|
||||
// never closed for the process lifetime (`send_cursor_forward`'s precondition).
|
||||
unsafe {
|
||||
crate::vdisplay::driver::send_cursor_forward(
|
||||
windows::Win32::Foundation::HANDLE(
|
||||
control_raw as *mut core::ffi::c_void,
|
||||
),
|
||||
req,
|
||||
)
|
||||
}
|
||||
},
|
||||
) as pf_capture::CursorForwardSender
|
||||
});
|
||||
pf_capture::open_idd_push(
|
||||
target,
|
||||
pref,
|
||||
@@ -183,6 +203,7 @@ pub fn capture_virtual_output(
|
||||
keep,
|
||||
sender,
|
||||
cursor_sender,
|
||||
cursor_forward_sender,
|
||||
)
|
||||
.map_err(|(e, _keep)| e.context("IDD-push capture open (no fallback)"))
|
||||
}
|
||||
|
||||
@@ -974,6 +974,12 @@ async fn serve_session(
|
||||
// Negotiated cursor forwarding: MUST match the HOST_CAP_CURSOR bit the Welcome advertised
|
||||
// (handshake::cursor_forward is the single predicate both read).
|
||||
let cursor_forward = handshake::cursor_forward(hello.client_caps, compositor);
|
||||
// Who renders the pointer RIGHT NOW (client `CursorRenderMode`, flipped live by the mouse-
|
||||
// model chord): `true` = client draws (exclude + forward), `false` = host composites (the
|
||||
// capture model). Starts true — the pre-message behavior for cap sessions. Control task
|
||||
// writes, data-plane loop edge-detects.
|
||||
let cursor_client_draws = Arc::new(AtomicBool::new(true));
|
||||
let cursor_client_draws_dp = cursor_client_draws.clone();
|
||||
// Adaptive FEC: the control task maps each client LossReport to a recovery percent and publishes
|
||||
// it here; the data-plane send loop reads + applies it per frame. Disabled (pinned) when
|
||||
// PUNKTFUNK_FEC_PCT is set. Seeded with the session's starting FEC so it's a no-op until a report.
|
||||
@@ -1010,6 +1016,7 @@ async fn serve_session(
|
||||
probe_result_rx,
|
||||
reconfig_result_rx,
|
||||
cursor_shape_rx,
|
||||
cursor_client_draws,
|
||||
clip_enabled,
|
||||
clip,
|
||||
));
|
||||
@@ -1397,6 +1404,7 @@ async fn serve_session(
|
||||
timing_conn,
|
||||
cursor_forward,
|
||||
cursor_shape_tx,
|
||||
cursor_client_draws: cursor_client_draws_dp,
|
||||
probe_seq,
|
||||
streamed_au,
|
||||
stats: stats_dp,
|
||||
|
||||
@@ -31,6 +31,7 @@ pub(super) async fn run(
|
||||
mut probe_result_rx: tokio::sync::mpsc::UnboundedReceiver<ProbeResult>,
|
||||
mut reconfig_result_rx: tokio::sync::mpsc::UnboundedReceiver<Reconfigured>,
|
||||
mut cursor_shape_rx: tokio::sync::mpsc::UnboundedReceiver<punktfunk_core::quic::CursorShape>,
|
||||
cursor_client_draws: Arc<AtomicBool>,
|
||||
clip_enabled: Arc<AtomicBool>,
|
||||
clip: pf_clipboard::ClipCoord,
|
||||
) {
|
||||
@@ -198,6 +199,16 @@ pub(super) async fn run(
|
||||
if io::write_msg(&mut ctrl_send, &echo.encode()).await.is_err() {
|
||||
break;
|
||||
}
|
||||
} else if let Ok(m) = punktfunk_core::quic::CursorRenderMode::decode(&msg) {
|
||||
// Who renders the pointer (design/remote-desktop-sweep.md §8): the client's
|
||||
// mouse-model flip. Latest-wins into the shared flag; the data-plane loop
|
||||
// edge-detects it per tick (forward+exclude vs composite). Inert for
|
||||
// sessions that never negotiated the cursor cap.
|
||||
cursor_client_draws.store(m.client_draws, Ordering::Relaxed);
|
||||
tracing::info!(
|
||||
client_draws = m.client_draws,
|
||||
"cursor render mode set by client"
|
||||
);
|
||||
} else if let Ok(ctl) = ClipControl::decode(&msg) {
|
||||
// Shared clipboard enable/disable (design/clipboard-and-file-transfer.md
|
||||
// §3.1). Reply with the resolved state; the operator policy is authoritative
|
||||
|
||||
@@ -939,9 +939,14 @@ pub(super) struct SessionContext {
|
||||
/// `host+network` latency stage. `None` = older client, no emission.
|
||||
pub(super) timing_conn: Option<quinn::Connection>,
|
||||
/// The session negotiated the cursor channel (design/remote-desktop-sweep.md M2 —
|
||||
/// `handshake::cursor_forward`): the encoder does NOT blend the pointer into the video;
|
||||
/// the encode loop forwards shape (via `cursor_shape_tx`) + per-tick `0xD0` state instead.
|
||||
/// `handshake::cursor_forward`): the encode loop forwards shape (via `cursor_shape_tx`)
|
||||
/// + per-tick `0xD0` state while the client draws the pointer locally.
|
||||
pub(super) cursor_forward: bool,
|
||||
/// LIVE render split for cap sessions (client `CursorRenderMode`, §8 mid-stream flip):
|
||||
/// `true` = client draws (exclude from video + forward), `false` = host composites (the
|
||||
/// capture mouse model — DWM on Windows, encoder blend on Linux). Control task writes;
|
||||
/// the encode loop edge-detects per tick. Always `true` for non-cap sessions (inert).
|
||||
pub(super) cursor_client_draws: Arc<AtomicBool>,
|
||||
/// SHAPE bridge to the control task (the control stream's sole writer) — mirrors
|
||||
/// `probe_result_tx`. Inert when `cursor_forward` is false.
|
||||
pub(super) cursor_shape_tx:
|
||||
@@ -995,10 +1000,12 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
ctx.bit_depth,
|
||||
ctx.chroma,
|
||||
ctx.codec,
|
||||
// Blend the pointer into the video only where the capture HAS one (not gamescope) AND
|
||||
// the client is not drawing it locally (the M2 cursor channel — blending too would
|
||||
// show it twice).
|
||||
ctx.compositor != pf_vdisplay::Compositor::Gamescope && !ctx.cursor_forward,
|
||||
// Blend CAPABILITY wherever the capture HAS a pointer (not gamescope) — including
|
||||
// cursor-forward sessions: the client can flip to the capture mouse model mid-stream
|
||||
// (`CursorRenderMode`), and the composite side of that flip must not need an encoder
|
||||
// rebuild. WHETHER a frame's pointer is drawn is per-tick: the encode loop strips
|
||||
// `frame.cursor` while the client draws locally (see the forwarder tick).
|
||||
ctx.compositor != pf_vdisplay::Compositor::Gamescope,
|
||||
ctx.cursor_forward,
|
||||
);
|
||||
// PyroWave rides the datagram-aligned wire mode (§4.4): every encoder this session opens
|
||||
@@ -1035,6 +1042,7 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
timing_conn,
|
||||
cursor_forward,
|
||||
cursor_shape_tx,
|
||||
cursor_client_draws,
|
||||
probe_seq,
|
||||
streamed_au,
|
||||
stats,
|
||||
@@ -1052,6 +1060,9 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
// encoder was told not to blend (SessionPlan above), so from the first frame the client's
|
||||
// locally-drawn cursor is the only one.
|
||||
let mut cursor_fwd = cursor_forward.then(super::cursor_fwd::CursorForwarder::new);
|
||||
// Edge detector for the live render flip (`cursor_client_draws`) — starts true (the
|
||||
// channel's initial state), so the first composite request triggers the capturer hook.
|
||||
let mut cursor_client_drew = true;
|
||||
if cursor_forward {
|
||||
tracing::info!("cursor channel negotiated — forwarding shape/state, encoder blend off");
|
||||
}
|
||||
@@ -2009,21 +2020,46 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
|
||||
);
|
||||
}
|
||||
}
|
||||
// Cursor channel (M2): every iteration — new frame OR repeat — states the pointer
|
||||
// (self-healing under datagram loss) and forwards a changed shape via the control
|
||||
// bridge. A hidden-but-known pointer (overlay with `visible: false`) is the M3
|
||||
// relative-mode hint. The capturer's LIVE cursor (the Windows hardware-cursor channel,
|
||||
// where pointer-only moves produce no frame) outranks the frame-attached overlay
|
||||
// (the Linux portal path); `frame` is the newest bound frame either way.
|
||||
// Cursor channel (M2 + the §8 mid-stream render flip). While the CLIENT draws the
|
||||
// pointer (the desktop mouse model): every iteration — new frame OR repeat — states
|
||||
// the pointer (self-healing under datagram loss) and forwards a changed shape via the
|
||||
// control bridge; `frame.cursor` is stripped so no blend path double-draws it. While
|
||||
// the HOST composites (the capture model, `CursorRenderMode { client_draws: false }`):
|
||||
// the forwarder goes quiet and `frame.cursor` rides into the encoder blend (Linux —
|
||||
// on Windows the flip re-enables DWM composition via the capturer hook below, and
|
||||
// frames never carry an overlay). A hidden-but-known pointer (overlay with
|
||||
// `visible: false`) is the M3 relative-mode hint. The capturer's LIVE cursor (the
|
||||
// Windows GDI-poller channel, where pointer-only moves produce no frame) outranks the
|
||||
// frame-attached overlay (the Linux portal path).
|
||||
if let Some(fwd) = cursor_fwd.as_mut() {
|
||||
let live = capturer.cursor();
|
||||
fwd.tick(
|
||||
live.as_ref().or(frame.cursor.as_ref()),
|
||||
&conn,
|
||||
&cursor_shape_tx,
|
||||
);
|
||||
let client_draws = cursor_client_draws.load(Ordering::Relaxed);
|
||||
if client_draws != cursor_client_drew {
|
||||
cursor_client_drew = client_draws;
|
||||
// Windows IDD: (un)declare the driver's hardware cursor so DWM excludes vs
|
||||
// composites; no-op on every other capturer.
|
||||
capturer.set_cursor_forward(client_draws);
|
||||
tracing::info!(
|
||||
client_draws,
|
||||
"cursor render mode flipped ({})",
|
||||
if client_draws {
|
||||
"client draws — exclude + forward"
|
||||
} else {
|
||||
"host composites"
|
||||
}
|
||||
);
|
||||
}
|
||||
if client_draws {
|
||||
let live = capturer.cursor();
|
||||
fwd.tick(
|
||||
live.as_ref().or(frame.cursor.as_ref()),
|
||||
&conn,
|
||||
&cursor_shape_tx,
|
||||
);
|
||||
// The client draws the pointer — a blend-capable encoder must not also draw it.
|
||||
frame.cursor = None;
|
||||
}
|
||||
}
|
||||
// The overlay now surfaces hidden pointers too (for the hint above) — strip them
|
||||
// The overlay surfaces hidden pointers too (for the hint above) — strip them
|
||||
// HERE, after forwarding, so no blend path ever draws an invisible cursor.
|
||||
if frame.cursor.as_ref().is_some_and(|c| !c.visible) {
|
||||
frame.cursor = None;
|
||||
@@ -2703,7 +2739,9 @@ pub(super) fn prepare_display(
|
||||
bit_depth,
|
||||
chroma,
|
||||
codec,
|
||||
compositor != pf_vdisplay::Compositor::Gamescope && !cursor_forward,
|
||||
// Blend capability regardless of cursor_forward — must MATCH virtual_stream's resolve
|
||||
// (the mid-stream `CursorRenderMode` flip strips/keeps `frame.cursor` per tick).
|
||||
compositor != pf_vdisplay::Compositor::Gamescope,
|
||||
cursor_forward,
|
||||
);
|
||||
if codec == crate::encode::Codec::PyroWave {
|
||||
|
||||
Reference in New Issue
Block a user