fix(host/encode): negotiate the cursor around what the encoder can blend
EncoderCaps::blends_cursor's contract said the HOST must fall back to capturer-side compositing when a cursor-as-metadata session lands on an encoder that can't composite — but that host half was never built: open_video warned and the session streamed WITHOUT a pointer (confirmed on the VAAPI dmabuf and libav-NVENC CUDA paths; latent on vulkan RGB-direct/native-NV12). The negotiation is now caps-aware, ahead of capture, on both planes: * pf-encode grows cursor_blend_capable() — the pre-open dispatch mirror (sibling of linux_native_nv12_ok) answering whether the resolved backend composites frame.cursor; its pure core is test-pinned arm by arm. * Native plane: handshake::cursor_forward grants the cursor channel only where the resolved backend can blend (the capture-mouse flip makes the host draw the pointer on demand); denied sessions keep the pre-channel path — the compositor EMBEDS the pointer, never cursorless, never doubled. The Welcome's HOST_CAP_CURSOR bit is computed once and read back at both session-wiring sites instead of recomputed. SessionPlan::output_format additionally keeps every cursor-blend session off producer-native NV12 (the arm with no CSC to fold a cursor into), and vulkan RGB-direct now yields to a cursor-blend session even when pinned (EFC cannot composite; the open logs the override). Windows plans cursor_blend=false via the new shared cursor_blend_for() rule — the IDD capturer composites the pointer itself, and asking the encoder anyway fired the blends-cursor warn spuriously on every cursor-channel session. * GameStream plane: the hardcoded cursor_blend=true is gone. The portal source asks for cursor-as-metadata only when the resolved backend blends, otherwise negotiates an Embedded pointer (choose_cursor_mode's new ladder); the capturer pool now also keys on that mode. The virtual-output source passes false — its capture embeds the pointer where it can. The per-arm warns in vulkan_video (RGB-direct, native-NV12) are now structurally unreachable and removed. open_video's post-open check stays as the single backstop for what planning cannot see: a Vulkan-open falling back to VAAPI mid-session, and the gamescope residual (no embedded mode exists there, so a never-blending backend — H.264-on-AMD VAAPI, software — still streams cursorless; fixing that needs a compositing stage, deliberately not built in this pass). Zero-copy is preserved throughout — every fallback is a capture-negotiation change, never a readback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -104,9 +104,13 @@ pub struct SessionPlan {
|
||||
/// AUs stay shard-aligned across mode/bitrate/stall rebuilds. `None` for the H.26x codecs.
|
||||
pub wire_chunk: Option<usize>,
|
||||
/// The session may hand the encoder cursor bitmaps to composite (cursor-as-metadata
|
||||
/// captures — every non-gamescope compositor; gamescope embeds the pointer itself).
|
||||
/// Encoders whose fast path cannot blend (the Vulkan EFC RGB-direct source) stay on their
|
||||
/// blending path when this is set, so the pointer never silently vanishes from the stream.
|
||||
/// captures). Set via [`cursor_blend_for`] — the single platform rule — so it is `true` only
|
||||
/// where the ENCODER is the compositing stage (Linux cursor-forward and gamescope sessions);
|
||||
/// Windows is always `false` (the IDD capturer composites the pointer itself). Encoders
|
||||
/// whose fast path cannot blend (the Vulkan EFC RGB-direct source, native NV12) stay off
|
||||
/// those shapes when this is set — see [`Self::output_format`] and
|
||||
/// `encode::cursor_blend_capable`, the pre-open mirror that gates the cursor channel — so
|
||||
/// the pointer never silently vanishes from the stream.
|
||||
pub cursor_blend: bool,
|
||||
/// The session negotiated the cursor-forward channel (M2/M2c): the client draws the pointer
|
||||
/// locally, so `cursor_blend` is off AND (on Windows) the capturer sets the driver's
|
||||
@@ -198,13 +202,15 @@ impl SessionPlan {
|
||||
// Producer-native NV12 (gamescope) is consumable only by the Linux Vulkan Video
|
||||
// backend — resolved HERE from the plan's codec so the capturer never reaches back
|
||||
// into encode (the same one-way edge as `gpu` above). BUT the native-NV12 encode path
|
||||
// has no CSC stage to fold the cursor into (it assumes gamescope embeds its pointer,
|
||||
// which it does NOT into the PipeWire node) — so a gamescope-cursor session (Phase C)
|
||||
// must capture RGB instead, routing to the compute-CSC / VkSlotBlend blend that draws
|
||||
// `frame.cursor`. Costs the RGB→NV12 CSC we'd otherwise skip; the native-NV12 cursor
|
||||
// blend is the perf-preserving follow-up.
|
||||
// has no CSC stage to fold the cursor into — so ANY cursor-compositing session
|
||||
// (gamescope Phase C, whose XFixes pointer is absent from the PipeWire node, AND a
|
||||
// cursor-forward session, whose capture-mouse flip needs the host composite on
|
||||
// demand) must capture RGB instead, routing to the compute-CSC / VkSlotBlend blend
|
||||
// that draws `frame.cursor`. Costs the RGB→NV12 CSC we'd otherwise skip; the
|
||||
// native-NV12 cursor blend is the perf-preserving follow-up. (`cursor_blend`
|
||||
// subsumes `gamescope_cursor` — see [`cursor_blend_for`].)
|
||||
#[cfg(target_os = "linux")]
|
||||
nv12_native: crate::encode::linux_native_nv12_ok(self.codec) && !self.gamescope_cursor,
|
||||
nv12_native: crate::encode::linux_native_nv12_ok(self.codec) && !self.cursor_blend,
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
nv12_native: false,
|
||||
}
|
||||
@@ -217,6 +223,26 @@ pub(crate) fn resolve_topology() -> SessionTopology {
|
||||
SessionTopology::SingleProcess
|
||||
}
|
||||
|
||||
/// THE rule for [`SessionPlan::cursor_blend`], shared by every resolve caller (initial plan and
|
||||
/// the mid-stream compositor re-gate) so they can't drift:
|
||||
/// * **Linux**: the encoder is the compositing stage — blend for a cursor-forward session (the
|
||||
/// capture-mouse flip needs the host composite on demand) and for gamescope (its capture
|
||||
/// carries no pointer at all; the XFixes-sourced cursor must be drawn into the video).
|
||||
/// * **Windows**: never — the IDD capturer composites the pointer itself (`cursor_blend.rs` /
|
||||
/// DWM), and no Windows encode backend reads `frame.cursor`. Asking the encoder anyway made
|
||||
/// `open_video`'s blends-cursor backstop fire spuriously on every cursor-channel session.
|
||||
pub(crate) fn cursor_blend_for(cursor_forward: bool, gamescope: bool) -> bool {
|
||||
#[cfg(target_os = "windows")]
|
||||
{
|
||||
let _ = (cursor_forward, gamescope);
|
||||
false
|
||||
}
|
||||
#[cfg(not(target_os = "windows"))]
|
||||
{
|
||||
cursor_forward || gamescope
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "windows")]
|
||||
fn resolve_encoder() -> EncoderBackend {
|
||||
match crate::encode::windows_resolved_backend() {
|
||||
|
||||
Reference in New Issue
Block a user