feat(video): Vulkan Video decode on the presenter's device (NVIDIA hw decode)
FFmpeg's Vulkan Video decoder now runs on the PRESENTER's own VkDevice — the decoded VkImage feeds the existing CICP CSC pass directly: zero copy, no interop, and NVIDIA gets hardware decode for the first time (its VAAPI is unusable by design). One decode architecture for every vendor going forward; VAAPI-dmabuf and software remain the fallbacks (auto: vulkan → vaapi → software; PUNKTFUNK_DECODER=vulkan pins it). Presenter: instance 1.3; probes VK_KHR_video_queue/decode_queue + codec extensions, a VIDEO_DECODE queue family (+ its codec caps via QueueFamilyVideoPropertiesKHR), and the samplerYcbcrConversion/ timelineSemaphore/synchronization2 features — all enabled at device creation when present, exported as a VulkanDecodeDevice handle bundle. Decoder: AVVulkanDeviceContext built over those handles via pf-ffvk (features chain, extension lists, deprecated queue indices + the qf[] map); get_format supplies OUR frames context with MUTABLE_FORMAT so the presenter's per-plane views are legal; output is DecodedImage::VkFrame carrying AVVkFrame/frames-ctx pointers plus the lock fns. Present: R8+R8G8 plane views over the multiplanar image, the live sync state read under the AVVulkanFramesContext lock, a timeline-semaphore wait(sem_value)/signal(sem_value+1) folded into the submit, layout/ queue-family/sem_value written back per FFmpeg's contract, and the frame guard parked in the retire queue until the fence. CSC pass + video framebuffer are now unconditional (NVIDIA has no dmabuf-import path). Verified on the RTX 5070 Ti: device creates with decode_qf=3, caps=DECODE_H264|H265|AV1|VP9; swapchain unaffected. Live stream validation next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,6 +35,9 @@ pub struct SessionParams {
|
||||
/// Library id for the host to launch this session (`"steam:570"`, from the library
|
||||
/// page); `None` = plain desktop session.
|
||||
pub launch: Option<String>,
|
||||
/// The presenter's shared Vulkan device, when its stack can run FFmpeg's Vulkan
|
||||
/// Video decoder (decode lands as VkImages the presenter samples directly).
|
||||
pub vulkan: Option<crate::video::VulkanDecodeDevice>,
|
||||
/// Pinned host fingerprint; `None` = trust on first use (caller persists the observed one).
|
||||
pub pin: Option<[u8; 32]>,
|
||||
pub identity: (String, String),
|
||||
@@ -241,7 +244,7 @@ fn pump(
|
||||
welcome_codec = connector.codec,
|
||||
"negotiated video codec"
|
||||
);
|
||||
let mut decoder = match Decoder::new(codec_id, ¶ms.decoder) {
|
||||
let mut decoder = match Decoder::new(codec_id, ¶ms.decoder, params.vulkan.as_ref()) {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
let _ = ev_tx.send_blocking(SessionEvent::Ended(Some(format!("video decoder: {e}"))));
|
||||
@@ -314,11 +317,13 @@ fn pump(
|
||||
dec_path = match &image {
|
||||
DecodedImage::Cpu(_) => "software",
|
||||
DecodedImage::Dmabuf(_) => "vaapi",
|
||||
DecodedImage::VkFrame(_) => "vulkan",
|
||||
};
|
||||
if total_frames == 1 {
|
||||
let (w, h, path) = match &image {
|
||||
DecodedImage::Cpu(c) => (c.width, c.height, "software"),
|
||||
DecodedImage::Dmabuf(d) => (d.width, d.height, "vaapi-dmabuf"),
|
||||
DecodedImage::VkFrame(v) => (v.width, v.height, "vulkan-video"),
|
||||
};
|
||||
tracing::info!(width = w, height = h, path, "first frame decoded");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user