fix(host/encode): negotiate the cursor around what the encoder can blend

EncoderCaps::blends_cursor's contract said the HOST must fall back to
capturer-side compositing when a cursor-as-metadata session lands on an
encoder that can't composite — but that host half was never built: open_video
warned and the session streamed WITHOUT a pointer (confirmed on the VAAPI
dmabuf and libav-NVENC CUDA paths; latent on vulkan RGB-direct/native-NV12).
The negotiation is now caps-aware, ahead of capture, on both planes:

* pf-encode grows cursor_blend_capable() — the pre-open dispatch mirror
  (sibling of linux_native_nv12_ok) answering whether the resolved backend
  composites frame.cursor; its pure core is test-pinned arm by arm.
* Native plane: handshake::cursor_forward grants the cursor channel only
  where the resolved backend can blend (the capture-mouse flip makes the host
  draw the pointer on demand); denied sessions keep the pre-channel path —
  the compositor EMBEDS the pointer, never cursorless, never doubled. The
  Welcome's HOST_CAP_CURSOR bit is computed once and read back at both
  session-wiring sites instead of recomputed. SessionPlan::output_format
  additionally keeps every cursor-blend session off producer-native NV12 (the
  arm with no CSC to fold a cursor into), and vulkan RGB-direct now yields to
  a cursor-blend session even when pinned (EFC cannot composite; the open
  logs the override). Windows plans cursor_blend=false via the new shared
  cursor_blend_for() rule — the IDD capturer composites the pointer itself,
  and asking the encoder anyway fired the blends-cursor warn spuriously on
  every cursor-channel session.
* GameStream plane: the hardcoded cursor_blend=true is gone. The portal
  source asks for cursor-as-metadata only when the resolved backend blends,
  otherwise negotiates an Embedded pointer (choose_cursor_mode's new ladder);
  the capturer pool now also keys on that mode. The virtual-output source
  passes false — its capture embeds the pointer where it can.

The per-arm warns in vulkan_video (RGB-direct, native-NV12) are now
structurally unreachable and removed. open_video's post-open check stays as
the single backstop for what planning cannot see: a Vulkan-open falling back
to VAAPI mid-session, and the gamescope residual (no embedded mode exists
there, so a never-blending backend — H.264-on-AMD VAAPI, software — still
streams cursorless; fixing that needs a compositing stage, deliberately not
built in this pass). Zero-copy is preserved throughout — every fallback is a
capture-negotiation change, never a readback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 10:50:33 +02:00
co-authored by Claude Fable 5
parent f495b201e1
commit fc335b39e9
12 changed files with 405 additions and 109 deletions
+52 -19
View File
@@ -105,16 +105,21 @@ impl PortalCapturer {
/// RemoteDesktop grant and never raises a separate ScreenCast dialog; `false` uses a plain
/// ScreenCast session (wlroots, which has no RemoteDesktop portal). `want_hdr` offers the
/// GNOME 50+ HDR formats (10-bit PQ/BT.2020, dmabuf-only) instead of the SDR set.
pub fn open(anchored: bool, want_hdr: bool, policy: ZeroCopyPolicy) -> Result<PortalCapturer> {
pub fn open(
anchored: bool,
want_hdr: bool,
want_metadata_cursor: bool,
policy: ZeroCopyPolicy,
) -> Result<PortalCapturer> {
// Portal handshake (async) on its own thread; hands back the PW fd + node id.
let (setup_tx, setup_rx) = std::sync::mpsc::channel::<Result<(OwnedFd, u32), String>>();
thread::Builder::new()
.name("punktfunk-portal".into())
.spawn(move || {
if anchored {
portal_thread_remote_desktop(setup_tx)
portal_thread_remote_desktop(setup_tx, want_metadata_cursor)
} else {
portal_thread(setup_tx)
portal_thread(setup_tx, want_metadata_cursor)
}
})
.context("spawn portal thread")?;
@@ -647,19 +652,23 @@ pub fn gnome_hdr_monitor_active() -> bool {
}
}
/// Pick the ScreenCast cursor mode from what the backend advertises (`AvailableCursorModes`),
/// preferring **cursor-as-metadata**: the compositor keeps its cheap hardware cursor plane and
/// ships the pointer as PipeWire `SPA_META_Cursor` metadata (position + an occasional bitmap),
/// which the consumer composites itself. That avoids forcing the producer to burn the cursor into
/// every frame — the `Embedded` mode — which on gamescope would defeat its HW cursor plane. Falls
/// back to `Embedded`, then `Hidden`, and (if the property query fails, e.g. an older portal)
/// keeps the prior `Embedded` behavior so the cursor is never silently lost.
/// Pick the ScreenCast cursor mode from what the backend advertises (`AvailableCursorModes`).
/// With `want_metadata` the ladder prefers **cursor-as-metadata**: the compositor keeps its cheap
/// hardware cursor plane and ships the pointer as PipeWire `SPA_META_Cursor` metadata (position +
/// an occasional bitmap), which the consumer composites itself — avoiding the producer burning the
/// cursor into every frame (`Embedded`), which on gamescope would defeat its HW cursor plane.
/// Without it — the session's encode path has no compositing stage for a metadata cursor
/// (`pf-encode`'s `cursor_blend_capable` said the resolved backend can't blend) — the ladder
/// prefers `Embedded`, so the pointer is in the pixels instead of in metadata nothing would draw.
/// Both ladders fall through to the other mode, then `Hidden`; a failed property query (an older
/// portal) keeps the prior `Embedded` behavior so the cursor is never silently lost.
async fn choose_cursor_mode(
proxy: &ashpd::desktop::screencast::Screencast,
want_metadata: bool,
) -> ashpd::desktop::screencast::CursorMode {
use ashpd::desktop::screencast::CursorMode;
match proxy.available_cursor_modes().await {
Ok(avail) if avail.contains(CursorMode::Metadata) => {
Ok(avail) if want_metadata && avail.contains(CursorMode::Metadata) => {
tracing::info!(
?avail,
"ScreenCast: requesting cursor-as-metadata (SPA_META_Cursor)"
@@ -667,12 +676,30 @@ async fn choose_cursor_mode(
CursorMode::Metadata
}
Ok(avail) if avail.contains(CursorMode::Embedded) => {
tracing::info!(
?avail,
"ScreenCast: cursor metadata unavailable — requesting Embedded cursor"
);
if want_metadata {
tracing::info!(
?avail,
"ScreenCast: cursor metadata unavailable — requesting Embedded cursor"
);
} else {
tracing::info!(
?avail,
"ScreenCast: requesting Embedded cursor (this session's encoder does not \
composite a metadata cursor)"
);
}
CursorMode::Embedded
}
Ok(avail) if avail.contains(CursorMode::Metadata) => {
// Embedded wanted but not offered. Metadata still beats Hidden: the CPU capture
// path composites `SPA_META_Cursor` inline, so part of the matrix keeps a pointer.
tracing::warn!(
?avail,
"ScreenCast: Embedded cursor not advertised — requesting cursor-as-metadata \
(only CPU-path frames will composite it)"
);
CursorMode::Metadata
}
Ok(avail) => {
tracing::warn!(
?avail,
@@ -692,7 +719,10 @@ async fn choose_cursor_mode(
/// The portal handshake: connect ScreenCast, select a single monitor, start, open the
/// PipeWire remote, hand the fd + node id back, then keep the session alive.
fn portal_thread(setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String>>) {
fn portal_thread(
setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String>>,
want_metadata_cursor: bool,
) {
use ashpd::desktop::screencast::{Screencast, SelectSourcesOptions, SourceType};
use ashpd::desktop::PersistMode;
use ashpd::enumflags2::BitFlags;
@@ -722,7 +752,7 @@ fn portal_thread(setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String
.create_session(Default::default())
.await
.context("create_session")?;
let cursor_mode = choose_cursor_mode(&proxy).await;
let cursor_mode = choose_cursor_mode(&proxy, want_metadata_cursor).await;
proxy
.select_sources(
&session,
@@ -781,7 +811,10 @@ fn portal_thread(setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String
/// path — also covers screen capture, with no separate ScreenCast dialog (which has no such
/// bypass). Yields the same PipeWire fd + node id as the standalone path; the consumer is
/// identical.
fn portal_thread_remote_desktop(setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String>>) {
fn portal_thread_remote_desktop(
setup_tx: std::sync::mpsc::Sender<Result<(OwnedFd, u32), String>>,
want_metadata_cursor: bool,
) {
use ashpd::desktop::remote_desktop::{DeviceType, RemoteDesktop, SelectDevicesOptions};
use ashpd::desktop::screencast::{Screencast, SelectSourcesOptions, SourceType};
use ashpd::desktop::PersistMode;
@@ -826,7 +859,7 @@ fn portal_thread_remote_desktop(setup_tx: std::sync::mpsc::Sender<Result<(OwnedF
.context("select_devices")?
.response()
.context("select_devices rejected")?;
let cursor_mode = choose_cursor_mode(&screencast).await;
let cursor_mode = choose_cursor_mode(&screencast, want_metadata_cursor).await;
screencast
.select_sources(
&session,