fix(host/encode): negotiate the cursor around what the encoder can blend

EncoderCaps::blends_cursor's contract said the HOST must fall back to
capturer-side compositing when a cursor-as-metadata session lands on an
encoder that can't composite — but that host half was never built: open_video
warned and the session streamed WITHOUT a pointer (confirmed on the VAAPI
dmabuf and libav-NVENC CUDA paths; latent on vulkan RGB-direct/native-NV12).
The negotiation is now caps-aware, ahead of capture, on both planes:

* pf-encode grows cursor_blend_capable() — the pre-open dispatch mirror
  (sibling of linux_native_nv12_ok) answering whether the resolved backend
  composites frame.cursor; its pure core is test-pinned arm by arm.
* Native plane: handshake::cursor_forward grants the cursor channel only
  where the resolved backend can blend (the capture-mouse flip makes the host
  draw the pointer on demand); denied sessions keep the pre-channel path —
  the compositor EMBEDS the pointer, never cursorless, never doubled. The
  Welcome's HOST_CAP_CURSOR bit is computed once and read back at both
  session-wiring sites instead of recomputed. SessionPlan::output_format
  additionally keeps every cursor-blend session off producer-native NV12 (the
  arm with no CSC to fold a cursor into), and vulkan RGB-direct now yields to
  a cursor-blend session even when pinned (EFC cannot composite; the open
  logs the override). Windows plans cursor_blend=false via the new shared
  cursor_blend_for() rule — the IDD capturer composites the pointer itself,
  and asking the encoder anyway fired the blends-cursor warn spuriously on
  every cursor-channel session.
* GameStream plane: the hardcoded cursor_blend=true is gone. The portal
  source asks for cursor-as-metadata only when the resolved backend blends,
  otherwise negotiates an Embedded pointer (choose_cursor_mode's new ladder);
  the capturer pool now also keys on that mode. The virtual-output source
  passes false — its capture embeds the pointer where it can.

The per-arm warns in vulkan_video (RGB-direct, native-NV12) are now
structurally unreachable and removed. open_video's post-open check stays as
the single backstop for what planning cannot see: a Vulkan-open falling back
to VAAPI mid-session, and the gamescope residual (no embedded mode exists
there, so a never-blending backend — H.264-on-AMD VAAPI, software — still
streams cursorless; fixing that needs a compositing stage, deliberately not
built in this pass). Zero-copy is preserved throughout — every fallback is a
capture-negotiation change, never a readback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 10:50:33 +02:00
co-authored by Claude Fable 5
parent f495b201e1
commit fc335b39e9
12 changed files with 405 additions and 109 deletions
+17 -3
View File
@@ -66,8 +66,14 @@ fn zero_copy_policy(
/// Open a live capturer for a client-sized monitor via the xdg ScreenCast portal. `want_hdr`
/// offers the GNOME 50+ 10-bit PQ/BT.2020 formats (pass it only when the session negotiated HDR
/// AND the mirrored monitor is in HDR mode — see [`pf_capture::gnome_hdr_monitor_active`]).
/// `want_metadata_cursor` asks for cursor-as-metadata — pass it only when the session's encode
/// backend composites `CapturedFrame::cursor` (`encode::cursor_blend_capable`); otherwise the
/// portal embeds the pointer, so no backend × cursor-mode combination streams cursorless.
#[cfg(target_os = "linux")]
pub fn open_portal_monitor(want_hdr: bool) -> Result<Box<dyn Capturer>> {
pub fn open_portal_monitor(
want_hdr: bool,
want_metadata_cursor: bool,
) -> Result<Box<dyn Capturer>> {
// On RemoteDesktop-capable desktops (KWin/GNOME) anchor ScreenCast to a RemoteDesktop
// session so it inherits that grant headlessly; wlroots/Sway has no RemoteDesktop portal,
// so use a plain ScreenCast session there.
@@ -76,11 +82,19 @@ pub fn open_portal_monitor(want_hdr: bool) -> Result<Box<dyn Capturer>> {
// passthrough is virtual-output-only; the global encoder-pref lever still applies inside.
// Native NV12 stays off too: the mirror path doesn't resolve the codec here, and the desktop
// compositors it mirrors (GNOME/KWin) don't produce NV12 anyway.
pf_capture::open_portal_monitor(anchored, want_hdr, zero_copy_policy(false, false))
pf_capture::open_portal_monitor(
anchored,
want_hdr,
want_metadata_cursor,
zero_copy_policy(false, false),
)
}
#[cfg(not(target_os = "linux"))]
pub fn open_portal_monitor(_want_hdr: bool) -> Result<Box<dyn Capturer>> {
pub fn open_portal_monitor(
_want_hdr: bool,
_want_metadata_cursor: bool,
) -> Result<Box<dyn Capturer>> {
anyhow::bail!("portal capture requires Linux (xdg-desktop-portal + PipeWire)")
}
+55 -14
View File
@@ -33,10 +33,14 @@ pub struct StreamConfig {
pub hdr: bool,
}
/// A pooled capturer plus the two PipeWire-negotiation-time properties reuse must match on —
/// its HDR-ness and its metadata-cursor mode; a mismatch on either needs a fresh screencast
/// session (see `AppState::video_cap`).
pub type PooledCapturer = (Box<dyn Capturer>, bool, bool);
/// Slot for the persistent screen capturer, shared with the control plane and reused across
/// streams so a reconnect doesn't open a second (conflicting) screencast session. The `bool` is
/// the pooled capturer's HDR-ness (see `AppState::video_cap`).
pub type CapturerSlot = Arc<std::sync::Mutex<Option<(Box<dyn Capturer>, bool)>>>;
/// streams so a reconnect doesn't open a second (conflicting) screencast session.
pub type CapturerSlot = Arc<std::sync::Mutex<Option<PooledCapturer>>>;
/// A pending client reference-frame-invalidation range (lost `firstFrame..=lastFrame`), set by the
/// control plane and drained by the video thread (see [`AppState::rfi_range`](super::AppState)).
@@ -136,7 +140,7 @@ fn run(
running: &Arc<AtomicBool>,
force_idr: &AtomicBool,
rfi_range: &std::sync::Mutex<Option<(i64, i64)>>,
video_cap: &std::sync::Mutex<Option<(Box<dyn Capturer>, bool)>>,
video_cap: &std::sync::Mutex<Option<PooledCapturer>>,
// Shared stats recorder for the web-console capture/graph. Threaded into `stream_body` (the
// encode loop); per-frame sample emission is wired by a later pass.
stats: &Arc<crate::stats_recorder::StatsRecorder>,
@@ -250,6 +254,12 @@ fn run(
return stream_body(
&mut capturer,
Some(&rebuild),
// The virtual-output source never selects cursor-as-metadata (`set_hw_cursor` is
// never called → the compositor EMBEDS the pointer where it can), so the encoder
// is handed nothing to composite. gamescope remains the pointerless residual —
// its capture carries no cursor either way (the native plane's XFixes source is
// not wired on this plane).
false,
&sock,
cfg,
running,
@@ -266,13 +276,34 @@ fn run(
// pooled capturer's HDR-ness matching this stream's negotiated `cfg.hdr` — the depth is a
// PipeWire-negotiation-time property of the screencast session, so an HDR↔SDR change needs a
// fresh session (same pattern as the audio capturer's channel-count gate).
// Cursor-as-metadata only where the encode backend this session resolves to composites
// `frame.cursor` (the caps-aware negotiation — mirror of the native plane's); otherwise ask
// the portal to EMBED the pointer so no backend × cursor-mode combination streams
// cursorless. Synthetic frames carry no pointer either way.
let metadata_cursor = {
#[cfg(target_os = "linux")]
{
// Same CUDA-payload prediction SessionPlan/`handshake::cursor_forward` make:
// the NVIDIA resolution plus the zero-copy master switch.
let cuda_planned =
!crate::encode::linux_zero_copy_is_vaapi() && crate::zerocopy::enabled();
crate::encode::cursor_blend_capable(cfg.codec, cuda_planned, cfg.hdr)
}
#[cfg(not(target_os = "linux"))]
false
};
let pooled = match video_cap.lock().unwrap().take() {
Some((c, was_hdr)) if was_hdr == cfg.hdr => Some(c),
Some((c, was_hdr)) => {
Some((c, was_hdr, was_meta)) if was_hdr == cfg.hdr && was_meta == metadata_cursor => {
Some(c)
}
Some((c, was_hdr, was_meta)) => {
tracing::info!(
was_hdr,
want_hdr = cfg.hdr,
"video source: pooled capturer depth mismatch — opening a fresh screencast session"
was_metadata_cursor = was_meta,
want_metadata_cursor = metadata_cursor,
"video source: pooled capturer depth/cursor-mode mismatch — opening a fresh \
screencast session"
);
drop(c);
None
@@ -285,8 +316,13 @@ fn run(
c
}
None if pf_host_config::config().video_source.as_deref() == Some("portal") => {
tracing::info!(hdr = cfg.hdr, "video source: portal desktop capture");
capture::open_portal_monitor(cfg.hdr).context("open portal capturer")?
tracing::info!(
hdr = cfg.hdr,
metadata_cursor,
"video source: portal desktop capture"
);
capture::open_portal_monitor(cfg.hdr, metadata_cursor)
.context("open portal capturer")?
}
None => {
tracing::info!("video source: synthetic test pattern");
@@ -298,6 +334,7 @@ fn run(
let result = stream_body(
&mut capturer,
None,
metadata_cursor,
&sock,
cfg,
running,
@@ -308,7 +345,7 @@ fn run(
on_lost,
);
capturer.set_active(false);
*video_cap.lock().unwrap() = Some((capturer, cfg.hdr));
*video_cap.lock().unwrap() = Some((capturer, cfg.hdr, metadata_cursor));
result
}
@@ -646,6 +683,10 @@ fn stream_body(
// Re-open the video source on capture loss (virtual-display path → follow a Desktop<->Game switch);
// `None` for the portal/synthetic source, which has nothing to re-detect (propagate the error).
rebuild: Option<&dyn Fn() -> Result<Box<dyn Capturer>>>,
// The capture hands the encoder cursor bitmaps to composite (cursor-as-metadata negotiated
// because the resolved backend blends — see the callers). `false` = the pointer is embedded
// in the pixels (or absent), so the encoder is asked to composite nothing.
cursor_blend: bool,
sock: &UdpSocket,
cfg: StreamConfig,
running: &Arc<AtomicBool>,
@@ -681,9 +722,9 @@ fn stream_body(
// GameStream/Moonlight stays 4:2:0 — stock Moonlight clients can't decode 4:4:4, and the
// Windows IDD-push capturer can't yet deliver full-chroma frames. 4:4:4 is punktfunk/1-native only.
encode::ChromaFormat::Yuv420,
// Desktop monitor capture negotiates cursor-as-metadata where available — the encoder
// may be handed cursor bitmaps to composite.
true,
// True only when THIS session's capture negotiated cursor-as-metadata — which the
// callers grant only where the resolved backend composites (`cursor_blend_capable`).
cursor_blend,
)
.context("open video encoder for stream")?;
// Tell the encoder how deep the capturer lets it pipeline. Without this an in-place backend
@@ -855,7 +896,7 @@ fn stream_body(
frame.is_cuda(),
gs_bit_depth(frame.format),
encode::ChromaFormat::Yuv420, // GameStream stays 4:2:0
true, // metadata-cursor capture — see the first open
cursor_blend, // same capture cursor mode — see the first open
)
.context("reopen encoder after rebuild")?;
// A rebuilt encoder starts unconfigured — same reason as the first open above.
+4 -3
View File
@@ -1012,9 +1012,10 @@ async fn serve_session(
// just never fires then.
let (cursor_shape_tx, cursor_shape_rx) =
tokio::sync::mpsc::unbounded_channel::<punktfunk_core::quic::CursorShape>();
// Negotiated cursor forwarding: MUST match the HOST_CAP_CURSOR bit the Welcome advertised
// (handshake::cursor_forward is the single predicate both read).
let cursor_forward = handshake::cursor_forward(hello.client_caps, compositor);
// Negotiated cursor forwarding: the HOST_CAP_CURSOR bit the Welcome advertised, read back
// rather than recomputed (`handshake::cursor_forward` computed it once, with the encoder
// blend-capability gate — re-running it here could drift, and would re-probe).
let cursor_forward = welcome.host_caps & punktfunk_core::quic::HOST_CAP_CURSOR != 0;
// Who renders the pointer RIGHT NOW (client `CursorRenderMode`, flipped live by the mouse-
// model chord): `true` = client draws (exclude + forward), `false` = host composites (the
// capture model). Starts true — the pre-message behavior for cap sessions. Control task
+28 -12
View File
@@ -12,31 +12,46 @@ use super::*;
/// the client asked ([`CLIENT_CAP_CURSOR`](punktfunk_core::quic::CLIENT_CAP_CURSOR)) AND the
/// capture path can deliver cursor metadata separately from the frame — the Linux portal
/// `SPA_META_Cursor` path (not gamescope, whose capture paints no cursor at all), or Windows
/// with a proto-v5 pf-vdisplay driver (the IddCx hardware-cursor channel, M2c). THE single
/// predicate: the Welcome's `HOST_CAP_CURSOR` bit and the session's forwarding/blend-off
/// wiring both read it, so they can never disagree.
/// with a proto-v5 pf-vdisplay driver (the IddCx hardware-cursor channel, M2c) — AND, on
/// Linux, the encode backend this session resolves to can composite the pointer on demand
/// (`encode::cursor_blend_capable`): the channel's capture-mouse flip (`CursorRenderMode`,
/// `client_draws = false`) makes the HOST draw the pointer, and on Linux the encoder is that
/// compositing stage — granting the channel over a backend that can't blend (libav
/// VAAPI/NVENC, software) shipped a cursorless stream on every capture-mode flip. Denied, the
/// session keeps the pre-channel path: the compositor EMBEDS the pointer and the client never
/// draws — never cursorless, never doubled. THE single predicate: the Welcome's
/// `HOST_CAP_CURSOR` bit is computed from it, and the session wiring reads that bit back.
pub(super) fn cursor_forward(
client_caps: u8,
compositor: Option<crate::vdisplay::Compositor>,
codec: crate::encode::Codec,
bit_depth: u8,
) -> bool {
if client_caps & punktfunk_core::quic::CLIENT_CAP_CURSOR == 0 {
return false;
}
#[cfg(target_os = "linux")]
{
// CUDA-payload prediction — the same one `SessionPlan` makes: the NVIDIA resolution
// plus the zero-copy master switch. It decides direct-SDK NVENC (blends) vs libav
// NVENC (doesn't) inside the capability mirror.
let cuda_planned = !crate::encode::linux_zero_copy_is_vaapi() && crate::zerocopy::enabled();
compositor.is_some_and(|c| c != crate::vdisplay::Compositor::Gamescope)
&& crate::encode::cursor_blend_capable(codec, cuda_planned, bit_depth == 10)
}
#[cfg(target_os = "windows")]
{
// Windows (M2c): the pf-vdisplay driver must speak the v5 hardware-cursor channel —
// DWM composites the pointer into the IDD frame otherwise, and forwarding a second
// copy would double it. The probe latches by opening the control device once.
let _ = compositor;
// copy would double it. The probe latches by opening the control device once. The
// encoder is deliberately NOT consulted: the IDD capturer itself composites on the
// capture-mouse flip (`set_cursor_forward`), so no Windows encode backend blends.
let _ = (compositor, codec, bit_depth);
crate::vdisplay::manager::hw_cursor_capable()
}
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
{
let _ = compositor;
let _ = (compositor, codec, bit_depth);
false
}
}
@@ -488,9 +503,10 @@ pub(super) async fn negotiate(
0
}
// Cursor channel granted (client asked + this capture path can deliver cursor
// metadata out of the frame) — the client turns its local renderer on ONLY when
// it sees this bit, and serve_session wires forwarding from the same predicate.
| if cursor_forward(hello.client_caps, compositor) {
// metadata out of the frame + the resolved encoder can composite on the
// capture-mouse flip) — the client turns its local renderer on ONLY when it sees
// this bit, and serve_session wires forwarding by reading the bit back.
| if cursor_forward(hello.client_caps, compositor, codec, bit_depth) {
punktfunk_core::quic::HOST_CAP_CURSOR
} else {
0
@@ -536,9 +552,9 @@ pub(super) async fn negotiate(
let (ctx_tx, ctx_rx) = std::sync::mpsc::sync_channel::<SessionContext>(1);
let client_identity = endpoint::peer_fingerprint(conn);
let client_hdr = hello.display_hdr;
// Same predicate the Welcome's HOST_CAP_CURSOR bit used — the prepared display and
// the session wiring must agree with what we just advertised.
let cursor_fw = cursor_forward(hello.client_caps, Some(comp));
// The bit the Welcome just advertised — read back rather than recomputed, so the
// prepared display and the session wiring cannot disagree with it.
let cursor_fw = welcome.host_caps & punktfunk_core::quic::HOST_CAP_CURSOR != 0;
let (mode, shard_payload) = (hello.mode, welcome.shard_payload);
let trace = bringup.clone();
std::thread::Builder::new()
+15 -5
View File
@@ -1022,8 +1022,13 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
// pointer compositor-EMBEDDED (`vd.set_hw_cursor(false)` → no cursor metadata, nothing to
// blend), keeping the zero-cost pre-channel path. gamescope is the exception (Phase C):
// it can't embed the pointer, so the host ALWAYS composites the XFixes-sourced cursor —
// the blend must be built for every gamescope session.
ctx.compositor == pf_vdisplay::Compositor::Gamescope || ctx.cursor_forward,
// the blend must be built for every gamescope session. (`cursor_forward` is already
// blend-gated: `handshake::cursor_forward` grants the channel only where
// `encode::cursor_blend_capable` says the resolved backend composites.)
crate::session_plan::cursor_blend_for(
ctx.cursor_forward,
ctx.compositor == pf_vdisplay::Compositor::Gamescope,
),
ctx.cursor_forward,
);
// gamescope: the XFixes cursor source feeds the always-on composite (Phase C). Set after
@@ -2098,8 +2103,10 @@ pub(super) fn virtual_stream(ctx: SessionContext, prepared: Option<PreparedDispl
// capture-mode channel); a switch AWAY restores the prior
// gating. `plan` is `Copy` — this is the value the rebuild
// (and its `build_pipeline` attach) reads.
plan.cursor_blend = plan.cursor_forward
|| c == crate::vdisplay::Compositor::Gamescope;
plan.cursor_blend = crate::session_plan::cursor_blend_for(
plan.cursor_forward,
c == crate::vdisplay::Compositor::Gamescope,
);
plan.gamescope_cursor =
c == crate::vdisplay::Compositor::Gamescope;
gamescope_composite =
@@ -2987,7 +2994,10 @@ pub(super) fn prepare_display(
// non-gamescope sessions get the pointer compositor-EMBEDDED, nothing to blend; the
// mid-stream `CursorRenderMode` flip strips/keeps `frame.cursor` per tick for channel
// sessions). gamescope (Phase C) can't embed → always composites the XFixes cursor.
compositor == pf_vdisplay::Compositor::Gamescope || cursor_forward,
crate::session_plan::cursor_blend_for(
cursor_forward,
compositor == pf_vdisplay::Compositor::Gamescope,
),
cursor_forward,
);
plan.gamescope_cursor = compositor == pf_vdisplay::Compositor::Gamescope;
+35 -9
View File
@@ -104,9 +104,13 @@ pub struct SessionPlan {
/// AUs stay shard-aligned across mode/bitrate/stall rebuilds. `None` for the H.26x codecs.
pub wire_chunk: Option<usize>,
/// The session may hand the encoder cursor bitmaps to composite (cursor-as-metadata
/// captures — every non-gamescope compositor; gamescope embeds the pointer itself).
/// Encoders whose fast path cannot blend (the Vulkan EFC RGB-direct source) stay on their
/// blending path when this is set, so the pointer never silently vanishes from the stream.
/// captures). Set via [`cursor_blend_for`] — the single platform rule — so it is `true` only
/// where the ENCODER is the compositing stage (Linux cursor-forward and gamescope sessions);
/// Windows is always `false` (the IDD capturer composites the pointer itself). Encoders
/// whose fast path cannot blend (the Vulkan EFC RGB-direct source, native NV12) stay off
/// those shapes when this is set — see [`Self::output_format`] and
/// `encode::cursor_blend_capable`, the pre-open mirror that gates the cursor channel — so
/// the pointer never silently vanishes from the stream.
pub cursor_blend: bool,
/// The session negotiated the cursor-forward channel (M2/M2c): the client draws the pointer
/// locally, so `cursor_blend` is off AND (on Windows) the capturer sets the driver's
@@ -198,13 +202,15 @@ impl SessionPlan {
// Producer-native NV12 (gamescope) is consumable only by the Linux Vulkan Video
// backend — resolved HERE from the plan's codec so the capturer never reaches back
// into encode (the same one-way edge as `gpu` above). BUT the native-NV12 encode path
// has no CSC stage to fold the cursor into (it assumes gamescope embeds its pointer,
// which it does NOT into the PipeWire node) — so a gamescope-cursor session (Phase C)
// must capture RGB instead, routing to the compute-CSC / VkSlotBlend blend that draws
// `frame.cursor`. Costs the RGB→NV12 CSC we'd otherwise skip; the native-NV12 cursor
// blend is the perf-preserving follow-up.
// has no CSC stage to fold the cursor into — so ANY cursor-compositing session
// (gamescope Phase C, whose XFixes pointer is absent from the PipeWire node, AND a
// cursor-forward session, whose capture-mouse flip needs the host composite on
// demand) must capture RGB instead, routing to the compute-CSC / VkSlotBlend blend
// that draws `frame.cursor`. Costs the RGB→NV12 CSC we'd otherwise skip; the
// native-NV12 cursor blend is the perf-preserving follow-up. (`cursor_blend`
// subsumes `gamescope_cursor` — see [`cursor_blend_for`].)
#[cfg(target_os = "linux")]
nv12_native: crate::encode::linux_native_nv12_ok(self.codec) && !self.gamescope_cursor,
nv12_native: crate::encode::linux_native_nv12_ok(self.codec) && !self.cursor_blend,
#[cfg(not(target_os = "linux"))]
nv12_native: false,
}
@@ -217,6 +223,26 @@ pub(crate) fn resolve_topology() -> SessionTopology {
SessionTopology::SingleProcess
}
/// THE rule for [`SessionPlan::cursor_blend`], shared by every resolve caller (initial plan and
/// the mid-stream compositor re-gate) so they can't drift:
/// * **Linux**: the encoder is the compositing stage — blend for a cursor-forward session (the
/// capture-mouse flip needs the host composite on demand) and for gamescope (its capture
/// carries no pointer at all; the XFixes-sourced cursor must be drawn into the video).
/// * **Windows**: never — the IDD capturer composites the pointer itself (`cursor_blend.rs` /
/// DWM), and no Windows encode backend reads `frame.cursor`. Asking the encoder anyway made
/// `open_video`'s blends-cursor backstop fire spuriously on every cursor-channel session.
pub(crate) fn cursor_blend_for(cursor_forward: bool, gamescope: bool) -> bool {
#[cfg(target_os = "windows")]
{
let _ = (cursor_forward, gamescope);
false
}
#[cfg(not(target_os = "windows"))]
{
cursor_forward || gamescope
}
}
#[cfg(target_os = "windows")]
fn resolve_encoder() -> EncoderBackend {
match crate::encode::windows_resolved_backend() {
+3 -1
View File
@@ -94,7 +94,9 @@ pub fn run(opts: Options) -> Result<()> {
want_hdr,
"spike source: xdg ScreenCast portal (live monitor)"
);
capture::open_portal_monitor(want_hdr).context("open portal capturer")?
// Embedded cursor: the spike passes `cursor_blend = false` to its encoder open, so
// a metadata pointer would be composited by nothing.
capture::open_portal_monitor(want_hdr, false).context("open portal capturer")?
}
Source::KwinVirtual => {
let compositor = crate::vdisplay::detect().unwrap_or(crate::vdisplay::Compositor::Kwin);