fix(host/encode): negotiate the cursor around what the encoder can blend
EncoderCaps::blends_cursor's contract said the HOST must fall back to capturer-side compositing when a cursor-as-metadata session lands on an encoder that can't composite — but that host half was never built: open_video warned and the session streamed WITHOUT a pointer (confirmed on the VAAPI dmabuf and libav-NVENC CUDA paths; latent on vulkan RGB-direct/native-NV12). The negotiation is now caps-aware, ahead of capture, on both planes: * pf-encode grows cursor_blend_capable() — the pre-open dispatch mirror (sibling of linux_native_nv12_ok) answering whether the resolved backend composites frame.cursor; its pure core is test-pinned arm by arm. * Native plane: handshake::cursor_forward grants the cursor channel only where the resolved backend can blend (the capture-mouse flip makes the host draw the pointer on demand); denied sessions keep the pre-channel path — the compositor EMBEDS the pointer, never cursorless, never doubled. The Welcome's HOST_CAP_CURSOR bit is computed once and read back at both session-wiring sites instead of recomputed. SessionPlan::output_format additionally keeps every cursor-blend session off producer-native NV12 (the arm with no CSC to fold a cursor into), and vulkan RGB-direct now yields to a cursor-blend session even when pinned (EFC cannot composite; the open logs the override). Windows plans cursor_blend=false via the new shared cursor_blend_for() rule — the IDD capturer composites the pointer itself, and asking the encoder anyway fired the blends-cursor warn spuriously on every cursor-channel session. * GameStream plane: the hardcoded cursor_blend=true is gone. The portal source asks for cursor-as-metadata only when the resolved backend blends, otherwise negotiates an Embedded pointer (choose_cursor_mode's new ladder); the capturer pool now also keys on that mode. The virtual-output source passes false — its capture embeds the pointer where it can. The per-arm warns in vulkan_video (RGB-direct, native-NV12) are now structurally unreachable and removed. open_video's post-open check stays as the single backstop for what planning cannot see: a Vulkan-open falling back to VAAPI mid-session, and the gamescope residual (no embedded mode exists there, so a never-blending backend — H.264-on-AMD VAAPI, software — still streams cursorless; fixing that needs a compositing stage, deliberately not built in this pass). Zero-copy is preserved throughout — every fallback is a capture-negotiation change, never a readback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
+156
-8
@@ -203,14 +203,15 @@ pub fn open_video(
|
||||
}
|
||||
};
|
||||
// The session asked for a composited pointer; say so loudly if the backend that actually opened
|
||||
// cannot deliver one. `cursor_blend` was a REQUEST with no answer for most of this crate's life
|
||||
// (`let _ = cursor_blend;` below), and the result was a stream with no mouse cursor and nothing
|
||||
// in the logs — confirmed on the VAAPI dmabuf path and the libav-NVENC CUDA path.
|
||||
//
|
||||
// A warning is deliberately all this does. `open_video` cannot re-plan capture, so refusing here
|
||||
// would only trade a missing pointer for a dead session; the host owns `plan.cursor_blend` and is
|
||||
// the only layer that can fall back to capturer-side compositing. This makes the condition
|
||||
// visible and queryable (`EncoderCaps::blends_cursor`) so that decision can be made upstream.
|
||||
// cannot deliver one. Since the negotiation became caps-aware ([`cursor_blend_capable`] gates
|
||||
// the cursor channel, and the session plan keeps cursor sessions off the native-NV12/RGB-direct
|
||||
// shapes), no PLANNED path reaches this: capture negotiates embedded-cursor mode wherever the
|
||||
// resolved backend can't blend. What remains reachable is the open-time divergence the plan
|
||||
// cannot see — a Vulkan Video open failing back to VAAPI mid-`open_amd_intel`, and the
|
||||
// gamescope residual (gamescope has no embedded mode, so a never-blending backend there —
|
||||
// H.264→VAAPI, software — still streams cursorless). This is the backstop that keeps those
|
||||
// honest in the logs; `open_video` cannot re-plan capture, so a warning is deliberately all
|
||||
// it does.
|
||||
if cursor_blend && !inner.caps().blends_cursor {
|
||||
tracing::warn!(
|
||||
backend,
|
||||
@@ -989,6 +990,87 @@ pub fn linux_native_nv12_ok(codec: Codec) -> bool {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the encode backend this session will resolve to composites [`CapturedFrame::cursor`]
|
||||
/// ([`EncoderCaps::blends_cursor`]) — answered BEFORE capture opens, so the host plans cursor
|
||||
/// delivery honestly instead of discovering a cursorless stream after the fact (the
|
||||
/// `blends_cursor` audit finding): a blend-capable backend takes cursor-as-metadata capture
|
||||
/// (pointer-free frames + host composite on demand — the cursor channel's contract); for
|
||||
/// anything else the host must have the compositor EMBED the pointer. The sibling verdict of
|
||||
/// [`linux_native_nv12_ok`], threaded into the same negotiation.
|
||||
///
|
||||
/// `cuda_planned` is the caller's prediction of a CUDA capture payload (NVIDIA + zero-copy — the
|
||||
/// prediction `SessionPlan` already makes); `ten_bit` the negotiated depth. Both shift the
|
||||
/// dispatch: a CPU payload keeps NVIDIA on libav NVENC (no blend), and a 10-bit HDR session
|
||||
/// skips Vulkan Video for libav VAAPI's P010/Main10 wiring (no blend).
|
||||
#[cfg(target_os = "linux")]
|
||||
pub fn cursor_blend_capable(codec: Codec, cuda_planned: bool, ten_bit: bool) -> bool {
|
||||
// A negotiated PyroWave session routes to that backend before the pref is consulted
|
||||
// (`open_video_backend_linux`), and its wavelet CSC composites the metadata cursor.
|
||||
if codec == Codec::PyroWave {
|
||||
return true;
|
||||
}
|
||||
let direct_nvenc = {
|
||||
#[cfg(feature = "nvenc")]
|
||||
{
|
||||
nvenc_direct_enabled()
|
||||
}
|
||||
#[cfg(not(feature = "nvenc"))]
|
||||
{
|
||||
false
|
||||
}
|
||||
};
|
||||
let vulkan_csc = {
|
||||
// The compute-CSC arm — the one that blends. Eligibility mirrors `open_amd_intel`;
|
||||
// the device probe runs last (it opens a Vulkan instance, cached per GPU+codec).
|
||||
#[cfg(feature = "vulkan-encode")]
|
||||
{
|
||||
matches!(codec, Codec::H265 | Codec::Av1)
|
||||
&& vulkan_encode_enabled()
|
||||
&& vulkan_encode_available(codec)
|
||||
}
|
||||
#[cfg(not(feature = "vulkan-encode"))]
|
||||
{
|
||||
false
|
||||
}
|
||||
};
|
||||
let backend = resolve_linux_backend(
|
||||
pf_host_config::config().encoder_pref.as_str(),
|
||||
linux_auto_is_vaapi,
|
||||
cuda_planned,
|
||||
);
|
||||
cursor_blend_capable_for(backend, cuda_planned, ten_bit, direct_nvenc, vulkan_csc)
|
||||
}
|
||||
|
||||
/// The dispatch-mirroring core of [`cursor_blend_capable`], device-free for the unit tests.
|
||||
/// `direct_nvenc` = the direct-SDK NVENC path is compiled in and enabled; `vulkan_csc` = the
|
||||
/// Vulkan Video compute-CSC arm (the one that blends) is compiled in, enabled, and
|
||||
/// device-supported for the session's codec.
|
||||
#[cfg(target_os = "linux")]
|
||||
fn cursor_blend_capable_for(
|
||||
backend: Option<LinuxBackend>,
|
||||
cuda_planned: bool,
|
||||
ten_bit: bool,
|
||||
direct_nvenc: bool,
|
||||
vulkan_csc: bool,
|
||||
) -> bool {
|
||||
match backend {
|
||||
// The wavelet CSC composites the metadata cursor (`linux/pyrowave.rs`).
|
||||
Some(LinuxBackend::Pyrowave) => true,
|
||||
// Only the direct-SDK arm blends (VkSlotBlend), and it only takes CUDA payloads —
|
||||
// a CPU-payload session stays on libav NVENC, which cannot blend.
|
||||
Some(LinuxBackend::Nvenc) => cuda_planned && direct_nvenc,
|
||||
// The Vulkan Video compute-CSC path blends; a 10-bit HDR session skips it for libav
|
||||
// VAAPI (no blend). The session plan keeps a cursor-blend session off the native-NV12
|
||||
// and RGB-direct shapes (`SessionPlan::output_format` / `VulkanVideoEncoder::open`),
|
||||
// so CSC eligibility IS the answer.
|
||||
Some(LinuxBackend::AmdIntel) | Some(LinuxBackend::Vulkan) => !ten_bit && vulkan_csc,
|
||||
// CPU frames: the capturer composites the metadata cursor inline before the encoder
|
||||
// runs, but the ENCODER blends nothing — the cursor channel's on-demand composite
|
||||
// contract can't be honored. Report the encoder's truth.
|
||||
Some(LinuxBackend::Software) | None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Can this GPU + driver actually open a Vulkan Video **encode** session for `codec`? Cached per
|
||||
/// (selected GPU, codec) — the [`can_encode_10bit`] idiom, with the probe run outside the lock.
|
||||
///
|
||||
@@ -1754,6 +1836,72 @@ mod tests {
|
||||
assert_eq!(none.wire_mask(), None);
|
||||
}
|
||||
|
||||
/// The cursor-blend capability mirror, arm by arm — the table the caps-aware negotiation
|
||||
/// (cursor channel grant, metadata-vs-embedded capture) stands on. Each row names the arm's
|
||||
/// blending stage or the reason there is none.
|
||||
#[cfg(target_os = "linux")]
|
||||
#[test]
|
||||
fn cursor_blend_capability_mirrors_the_dispatch() {
|
||||
use LinuxBackend::*;
|
||||
// PyroWave: the wavelet CSC composites, always.
|
||||
assert!(cursor_blend_capable_for(
|
||||
Some(Pyrowave),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false
|
||||
));
|
||||
// NVIDIA: only the direct-SDK arm blends (VkSlotBlend), and only for CUDA payloads.
|
||||
assert!(cursor_blend_capable_for(
|
||||
Some(Nvenc),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
false
|
||||
));
|
||||
assert!(
|
||||
!cursor_blend_capable_for(Some(Nvenc), false, false, true, false),
|
||||
"a CPU payload stays on libav NVENC, which cannot blend"
|
||||
);
|
||||
assert!(
|
||||
!cursor_blend_capable_for(Some(Nvenc), true, false, false, false),
|
||||
"PUNKTFUNK_NVENC_DIRECT=0 (or a build without the feature) is the libav path"
|
||||
);
|
||||
// AMD/Intel: the Vulkan Video compute-CSC arm blends; VAAPI never does.
|
||||
assert!(cursor_blend_capable_for(
|
||||
Some(AmdIntel),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
));
|
||||
assert!(
|
||||
!cursor_blend_capable_for(Some(AmdIntel), false, false, false, false),
|
||||
"no eligible Vulkan CSC arm (H.264, PUNKTFUNK_VULKAN_ENCODE=0, unsupported \
|
||||
device) resolves to libav VAAPI, which cannot blend"
|
||||
);
|
||||
assert!(
|
||||
!cursor_blend_capable_for(Some(AmdIntel), false, true, false, true),
|
||||
"a 10-bit HDR session skips Vulkan Video for VAAPI's P010 wiring — no blend"
|
||||
);
|
||||
assert!(cursor_blend_capable_for(
|
||||
Some(Vulkan),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
));
|
||||
// Software / unknown pref: CPU frames; the encoder blends nothing.
|
||||
assert!(!cursor_blend_capable_for(
|
||||
Some(Software),
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
));
|
||||
assert!(!cursor_blend_capable_for(None, false, false, true, true));
|
||||
}
|
||||
|
||||
/// WP7.7 guard (the cheap half): every `Encoder` trait method must be explicitly forwarded by
|
||||
/// `TrackedEncoder`. A defaulted trait method that isn't forwarded silently no-ops through the
|
||||
/// wrapper — the trap has bitten three times (`set_wire_chunking`'s §4.4 chunking probe,
|
||||
|
||||
Reference in New Issue
Block a user