fix(host/encode): negotiate the cursor around what the encoder can blend

EncoderCaps::blends_cursor's contract said the HOST must fall back to
capturer-side compositing when a cursor-as-metadata session lands on an
encoder that can't composite — but that host half was never built: open_video
warned and the session streamed WITHOUT a pointer (confirmed on the VAAPI
dmabuf and libav-NVENC CUDA paths; latent on vulkan RGB-direct/native-NV12).
The negotiation is now caps-aware, ahead of capture, on both planes:

* pf-encode grows cursor_blend_capable() — the pre-open dispatch mirror
  (sibling of linux_native_nv12_ok) answering whether the resolved backend
  composites frame.cursor; its pure core is test-pinned arm by arm.
* Native plane: handshake::cursor_forward grants the cursor channel only
  where the resolved backend can blend (the capture-mouse flip makes the host
  draw the pointer on demand); denied sessions keep the pre-channel path —
  the compositor EMBEDS the pointer, never cursorless, never doubled. The
  Welcome's HOST_CAP_CURSOR bit is computed once and read back at both
  session-wiring sites instead of recomputed. SessionPlan::output_format
  additionally keeps every cursor-blend session off producer-native NV12 (the
  arm with no CSC to fold a cursor into), and vulkan RGB-direct now yields to
  a cursor-blend session even when pinned (EFC cannot composite; the open
  logs the override). Windows plans cursor_blend=false via the new shared
  cursor_blend_for() rule — the IDD capturer composites the pointer itself,
  and asking the encoder anyway fired the blends-cursor warn spuriously on
  every cursor-channel session.
* GameStream plane: the hardcoded cursor_blend=true is gone. The portal
  source asks for cursor-as-metadata only when the resolved backend blends,
  otherwise negotiates an Embedded pointer (choose_cursor_mode's new ladder);
  the capturer pool now also keys on that mode. The virtual-output source
  passes false — its capture embeds the pointer where it can.

The per-arm warns in vulkan_video (RGB-direct, native-NV12) are now
structurally unreachable and removed. open_video's post-open check stays as
the single backstop for what planning cannot see: a Vulkan-open falling back
to VAAPI mid-session, and the gamescope residual (no embedded mode exists
there, so a never-blending backend — H.264-on-AMD VAAPI, software — still
streams cursorless; fixing that needs a compositing stage, deliberately not
built in this pass). Zero-copy is preserved throughout — every fallback is a
capture-negotiation change, never a readback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

crates/pf-encode/src/enc/codec.rs

@@ -307,8 +307,11 @@ pub struct EncoderCaps {
     ///
     /// This makes the answer queryable instead of assumed. It is deliberately a plain fact about the
     /// encoder, not a policy: what to DO when a session wants blending and the backend cannot is the
-    /// host's call, since only the host can re-plan capture (fall back to capturer-side compositing).
-    /// `open_video` can only warn, which it does.
+    /// host's call, since only the host can re-plan capture. That call is wired now — the
+    /// negotiation consults the pre-open mirror ([`cursor_blend_capable`](crate::cursor_blend_capable))
+    /// to gate the cursor channel and to keep capture on embedded-cursor / CSC-capable shapes for
+    /// any backend that can't blend; `open_video`'s post-open check remains as the backstop for
+    /// open-time fallbacks the plan can't see.
     pub blends_cursor: bool,
 }
 

crates/pf-encode/src/enc/linux/vulkan_video.rs

@@ -76,9 +76,11 @@ fn quality_request() -> u32 {
 /// (design/vulkan-rgb-direct-encode.md): the captured RGB frame is the encode source and the
 /// VCN EFC front-end does the 709-narrow CSC inline — no compute CSC, no plane copies, one
 /// queue submit per frame (unaligned modes go through the padded-copy staging blit). B2
-/// default: ON wherever the probe passes, EXCEPT sessions that may need the CSC's cursor
-/// blend (see [`VulkanVideoEncoder::open`]). `=0` disables outright; `=1` forces it even on
-/// cursor-blend sessions (the pointer will be missing from the stream); unset = the default.
+/// default: ON wherever the probe passes, EXCEPT sessions that need the CSC's cursor blend
+/// (see [`VulkanVideoEncoder::open`]). `=0` disables outright; `=1` forces it on non-cursor
+/// sessions; unset = the default. A cursor-blend session IGNORES `=1` — EFC cannot composite
+/// the pointer, and the caps-aware negotiation promised the client a composited one, so the
+/// pointer outranks the lab pin (the open logs the override).
 ///
 /// Parses like every sibling knob (`matches!(v.trim(), …)`): anything unrecognised — including
 /// an empty value and a value that is only whitespace — falls back to the default rather than
@@ -614,10 +616,6 @@ pub struct VulkanVideoEncoder {
     /// GPU reset (those paths keep their aligned-size sources/staging).
     native_nv12: bool,
 
-    /// One-shot warning latch: a cursor bitmap arrived on an RGB-direct or native-NV12 session
-    /// (neither has a compositing stage — the cursor will be missing from the stream until the
-    /// CSC path is used).
-    warned_cursor: bool,
     /// A [`reconfigure_bitrate`](Encoder::reconfigure_bitrate) rate not yet installed in the video
     /// session. The next `record_submit` emits an `ENCODE_RATE_CONTROL` control command carrying it
     /// (mid-stream) or folds it into the first frame's RESET+RC install, then promotes it into
@@ -671,7 +669,16 @@ impl VulkanVideoEncoder {
         cursor_blend: bool,
     ) -> Result<Self> {
         let native_nv12 = format == PixelFormat::Nv12;
-        let want_rgb = !native_nv12 && rgb_request().unwrap_or(!cursor_blend);
+        // A cursor-blend session must keep the compute-CSC path — the only arm with the cursor
+        // blend — so it outranks an explicit RGB-direct pin (EFC cannot composite; the
+        // negotiation promised the client a composited pointer).
+        if cursor_blend && rgb_request() == Some(true) {
+            tracing::info!(
+                "PUNKTFUNK_VULKAN_RGB_DIRECT=1 ignored for this session — it composites the \
+                 pointer, which the EFC front-end cannot; using the compute-CSC path"
+            );
+        }
+        let want_rgb = !native_nv12 && !cursor_blend && rgb_request().unwrap_or(true);
         Self::open_opts_inner(
             codec,
             width,
@@ -1448,7 +1455,6 @@ impl VulkanVideoEncoder {
             cpu_expand: Vec::new(),
             rgb: rgb_cfg,
             native_nv12,
-            warned_cursor: false,
             pending_bitrate: None,
             width: w,
             height: h,
@@ -2621,17 +2627,10 @@ impl VulkanVideoEncoder {
                 d.modifier
             );
         }
-        // No compositing stage exists here (like RGB-direct/EFC): gamescope embeds its pointer
-        // in the produced pixels, but any other NV12 producer's metadata cursor would be lost —
-        // say so once instead of silently.
-        if frame.cursor.is_some() && !self.warned_cursor {
-            self.warned_cursor = true;
-            tracing::warn!(
-                "cursor bitmap on a native-NV12 session — nothing composites it; the cursor \
-                 will be missing from the stream (unset PUNKTFUNK_PIPEWIRE_NV12 for \
-                 metadata-cursor captures)"
-            );
-        }
+        // No compositing stage exists here (like RGB-direct/EFC) — and none is needed: the
+        // session plan negotiates native NV12 only for a non-cursor-blend session
+        // (`SessionPlan::output_format` gates `nv12_native` on `!cursor_blend`), so no cursor
+        // bitmap ever reaches this arm. Gamescope embeds its pointer in the produced pixels.
         let dev = self.device.clone();
         let cmd = self.frames[slot].cmd;
         let fence = self.frames[slot].fence;
@@ -2691,16 +2690,9 @@ impl VulkanVideoEncoder {
         let query_pool = self.frames[slot].query_pool;
         let bs_buf = self.frames[slot].bs_buf;
         let ts_pool = self.frames[slot].ts_pool;
-        // EFC cannot composite the cursor bitmap the metadata-cursor captures hand us — say so
-        // once instead of silently losing the pointer (gamescope, the flagship, embeds it).
-        if frame.cursor.is_some() && !self.warned_cursor {
-            self.warned_cursor = true;
-            tracing::warn!(
-                "cursor bitmap on an RGB-direct session — EFC cannot composite it; the cursor \
-                 will be missing from the stream (unset PUNKTFUNK_VULKAN_RGB_DIRECT for \
-                 metadata-cursor captures)"
-            );
-        }
+        // EFC cannot composite a cursor bitmap — and never has to: `open` refuses the RGB-direct
+        // shape for a cursor-blend session (the pin override above), so no cursor bitmap ever
+        // reaches this arm. Gamescope, the flagship, embeds its pointer in the produced pixels.
         let padded = self.rgb.as_ref().is_some_and(|r| r.padded);
         // Only the padded Dmabuf arm below records timestamps (via `record_pad_blit`); the
         // CPU-upload arm records its own command buffer and writes none. Default to "not written"

crates/pf-encode/src/lib.rs

@@ -203,14 +203,15 @@ pub fn open_video(
         }
     };
     // The session asked for a composited pointer; say so loudly if the backend that actually opened
-    // cannot deliver one. `cursor_blend` was a REQUEST with no answer for most of this crate's life
-    // (`let _ = cursor_blend;` below), and the result was a stream with no mouse cursor and nothing
-    // in the logs — confirmed on the VAAPI dmabuf path and the libav-NVENC CUDA path.
-    //
-    // A warning is deliberately all this does. `open_video` cannot re-plan capture, so refusing here
-    // would only trade a missing pointer for a dead session; the host owns `plan.cursor_blend` and is
-    // the only layer that can fall back to capturer-side compositing. This makes the condition
-    // visible and queryable (`EncoderCaps::blends_cursor`) so that decision can be made upstream.
+    // cannot deliver one. Since the negotiation became caps-aware ([`cursor_blend_capable`] gates
+    // the cursor channel, and the session plan keeps cursor sessions off the native-NV12/RGB-direct
+    // shapes), no PLANNED path reaches this: capture negotiates embedded-cursor mode wherever the
+    // resolved backend can't blend. What remains reachable is the open-time divergence the plan
+    // cannot see — a Vulkan Video open failing back to VAAPI mid-`open_amd_intel`, and the
+    // gamescope residual (gamescope has no embedded mode, so a never-blending backend there —
+    // H.264→VAAPI, software — still streams cursorless). This is the backstop that keeps those
+    // honest in the logs; `open_video` cannot re-plan capture, so a warning is deliberately all
+    // it does.
     if cursor_blend && !inner.caps().blends_cursor {
         tracing::warn!(
             backend,
@@ -989,6 +990,87 @@ pub fn linux_native_nv12_ok(codec: Codec) -> bool {
     }
 }
 
+/// Whether the encode backend this session will resolve to composites [`CapturedFrame::cursor`]
+/// ([`EncoderCaps::blends_cursor`]) — answered BEFORE capture opens, so the host plans cursor
+/// delivery honestly instead of discovering a cursorless stream after the fact (the
+/// `blends_cursor` audit finding): a blend-capable backend takes cursor-as-metadata capture
+/// (pointer-free frames + host composite on demand — the cursor channel's contract); for
+/// anything else the host must have the compositor EMBED the pointer. The sibling verdict of
+/// [`linux_native_nv12_ok`], threaded into the same negotiation.
+///
+/// `cuda_planned` is the caller's prediction of a CUDA capture payload (NVIDIA + zero-copy — the
+/// prediction `SessionPlan` already makes); `ten_bit` the negotiated depth. Both shift the
+/// dispatch: a CPU payload keeps NVIDIA on libav NVENC (no blend), and a 10-bit HDR session
+/// skips Vulkan Video for libav VAAPI's P010/Main10 wiring (no blend).
+#[cfg(target_os = "linux")]
+pub fn cursor_blend_capable(codec: Codec, cuda_planned: bool, ten_bit: bool) -> bool {
+    // A negotiated PyroWave session routes to that backend before the pref is consulted
+    // (`open_video_backend_linux`), and its wavelet CSC composites the metadata cursor.
+    if codec == Codec::PyroWave {
+        return true;
+    }
+    let direct_nvenc = {
+        #[cfg(feature = "nvenc")]
+        {
+            nvenc_direct_enabled()
+        }
+        #[cfg(not(feature = "nvenc"))]
+        {
+            false
+        }
+    };
+    let vulkan_csc = {
+        // The compute-CSC arm — the one that blends. Eligibility mirrors `open_amd_intel`;
+        // the device probe runs last (it opens a Vulkan instance, cached per GPU+codec).
+        #[cfg(feature = "vulkan-encode")]
+        {
+            matches!(codec, Codec::H265 | Codec::Av1)
+                && vulkan_encode_enabled()
+                && vulkan_encode_available(codec)
+        }
+        #[cfg(not(feature = "vulkan-encode"))]
+        {
+            false
+        }
+    };
+    let backend = resolve_linux_backend(
+        pf_host_config::config().encoder_pref.as_str(),
+        linux_auto_is_vaapi,
+        cuda_planned,
+    );
+    cursor_blend_capable_for(backend, cuda_planned, ten_bit, direct_nvenc, vulkan_csc)
+}
+
+/// The dispatch-mirroring core of [`cursor_blend_capable`], device-free for the unit tests.
+/// `direct_nvenc` = the direct-SDK NVENC path is compiled in and enabled; `vulkan_csc` = the
+/// Vulkan Video compute-CSC arm (the one that blends) is compiled in, enabled, and
+/// device-supported for the session's codec.
+#[cfg(target_os = "linux")]
+fn cursor_blend_capable_for(
+    backend: Option<LinuxBackend>,
+    cuda_planned: bool,
+    ten_bit: bool,
+    direct_nvenc: bool,
+    vulkan_csc: bool,
+) -> bool {
+    match backend {
+        // The wavelet CSC composites the metadata cursor (`linux/pyrowave.rs`).
+        Some(LinuxBackend::Pyrowave) => true,
+        // Only the direct-SDK arm blends (VkSlotBlend), and it only takes CUDA payloads —
+        // a CPU-payload session stays on libav NVENC, which cannot blend.
+        Some(LinuxBackend::Nvenc) => cuda_planned && direct_nvenc,
+        // The Vulkan Video compute-CSC path blends; a 10-bit HDR session skips it for libav
+        // VAAPI (no blend). The session plan keeps a cursor-blend session off the native-NV12
+        // and RGB-direct shapes (`SessionPlan::output_format` / `VulkanVideoEncoder::open`),
+        // so CSC eligibility IS the answer.
+        Some(LinuxBackend::AmdIntel) | Some(LinuxBackend::Vulkan) => !ten_bit && vulkan_csc,
+        // CPU frames: the capturer composites the metadata cursor inline before the encoder
+        // runs, but the ENCODER blends nothing — the cursor channel's on-demand composite
+        // contract can't be honored. Report the encoder's truth.
+        Some(LinuxBackend::Software) | None => false,
+    }
+}
+
 /// Can this GPU + driver actually open a Vulkan Video **encode** session for `codec`? Cached per
 /// (selected GPU, codec) — the [`can_encode_10bit`] idiom, with the probe run outside the lock.
 ///
@@ -1754,6 +1836,72 @@ mod tests {
         assert_eq!(none.wire_mask(), None);
     }
 
+    /// The cursor-blend capability mirror, arm by arm — the table the caps-aware negotiation
+    /// (cursor channel grant, metadata-vs-embedded capture) stands on. Each row names the arm's
+    /// blending stage or the reason there is none.
+    #[cfg(target_os = "linux")]
+    #[test]
+    fn cursor_blend_capability_mirrors_the_dispatch() {
+        use LinuxBackend::*;
+        // PyroWave: the wavelet CSC composites, always.
+        assert!(cursor_blend_capable_for(
+            Some(Pyrowave),
+            false,
+            false,
+            false,
+            false
+        ));
+        // NVIDIA: only the direct-SDK arm blends (VkSlotBlend), and only for CUDA payloads.
+        assert!(cursor_blend_capable_for(
+            Some(Nvenc),
+            true,
+            false,
+            true,
+            false
+        ));
+        assert!(
+            !cursor_blend_capable_for(Some(Nvenc), false, false, true, false),
+            "a CPU payload stays on libav NVENC, which cannot blend"
+        );
+        assert!(
+            !cursor_blend_capable_for(Some(Nvenc), true, false, false, false),
+            "PUNKTFUNK_NVENC_DIRECT=0 (or a build without the feature) is the libav path"
+        );
+        // AMD/Intel: the Vulkan Video compute-CSC arm blends; VAAPI never does.
+        assert!(cursor_blend_capable_for(
+            Some(AmdIntel),
+            false,
+            false,
+            false,
+            true
+        ));
+        assert!(
+            !cursor_blend_capable_for(Some(AmdIntel), false, false, false, false),
+            "no eligible Vulkan CSC arm (H.264, PUNKTFUNK_VULKAN_ENCODE=0, unsupported \
+             device) resolves to libav VAAPI, which cannot blend"
+        );
+        assert!(
+            !cursor_blend_capable_for(Some(AmdIntel), false, true, false, true),
+            "a 10-bit HDR session skips Vulkan Video for VAAPI's P010 wiring — no blend"
+        );
+        assert!(cursor_blend_capable_for(
+            Some(Vulkan),
+            false,
+            false,
+            false,
+            true
+        ));
+        // Software / unknown pref: CPU frames; the encoder blends nothing.
+        assert!(!cursor_blend_capable_for(
+            Some(Software),
+            false,
+            false,
+            true,
+            true
+        ));
+        assert!(!cursor_blend_capable_for(None, false, false, true, true));
+    }
+
     /// WP7.7 guard (the cheap half): every `Encoder` trait method must be explicitly forwarded by
     /// `TrackedEncoder`. A defaulted trait method that isn't forwarded silently no-ops through the
     /// wrapper — the trap has bitten three times (`set_wire_chunking`'s §4.4 chunking probe,