feat(hdr): GNOME 50 HDR screencast capture + Linux Main10 encode

GNOME 50 (Mutter MR 4928, PipeWire >= 1.6) added HDR screen sharing for
monitor streams: 10-bit PQ formats (xRGB_210LE/xBGR_210LE) with MANDATORY
BT.2020 + SMPTE-2084 colorimetry props, advertised while the mirrored
monitor is in BT.2100 colour mode. Wire the Linux host into it end-to-end
on the GameStream desktop-mirror path (PUNKTFUNK_VIDEO_SOURCE=portal):

* pf-frame: PixelFormat::X2Rgb10/X2Bgr10 (DRM XR30/XB30; X2Bgr10 is the
  Windows Rgb10a2 layout) + fourccs.
* pf-capture: want_hdr portal offer — HDR-only LINEAR-dmabuf pods with
  MANDATORY PQ/BT.2020 props (SHM excluded: Mutter's SHM record path
  paints 8-bit ARGB32 regardless of format; tiled excluded: the EGL
  de-tile blit is 8-bit RGBA8), negotiated-colorimetry parse, generic
  HDR10 hdr_meta(), packed-10-bit CPU cursor blend, a process-wide SDR
  downgrade latch on negotiation timeout, and a DisplayConfig BT.2100
  colour-mode probe (gnome_hdr_monitor_active).
* pf-encode: libav NVENC X2RGB10->P010 swscale (BT.2020 limited) ->
  HEVC Main10 / 10-bit AV1 with PQ VUI; VAAPI 10-bit on both paths (CPU
  P010 upload + dmabuf XR30 scale_vaapi p010/bt2020); can_encode_10bit
  now probes for real on Linux; 10-bit sessions route around the
  8-bit-only Vulkan-video/direct-NVENC backends.
* GameStream: host_hdr_capable() Linux arm, live monitor-HDR check at
  RTSP honor time, capturer-pool reuse keyed on HDR-ness, gs_bit_depth
  covers the new formats. New `punktfunk-host hdr-probe` diagnostic and
  a PUNKTFUNK_SPIKE_HDR spike lever.
* Native plane stays honestly 8-bit via capturer_supports_hdr(): Mutter
  RecordVirtual streams are SDR-only upstream (GNOME 50 and 51-dev), so
  virtual-display sources cannot deliver HDR yet.

Validated on the RTX 5070 Ti (GNOME 50.3 / PipeWire 1.6.8): the Main10
probes pass and the ignored nvenc_hdr10_smoke GPU test emits an IDR that
ffprobe reads as Main 10 / yuv420p10le / bt2020nc / smpte2084 / limited.
Live HDR capture negotiation still needs an HDR monitor on glass; VAAPI
10-bit needs the AMD box.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

crates/punktfunk-host/src/gamestream/mod.rs

@@ -50,21 +50,45 @@ pub const SCM_AV1_MAIN10: u32 = 0x0002_0000;
 /// The **SDR baseline** codec mask: H.264, HEVC Main, AV1 Main 8-bit (= 65793). HEVC Main10 (HDR) is
 /// layered on top of this at runtime by `serverinfo::codec_mode_support` when — and only when — the
 /// host can actually deliver it ([`host_hdr_capable`]); it is never a static claim, because a non-HDR
-/// host (Linux, or a Windows host without the `PUNKTFUNK_10BIT` opt-in) must not invite a client into
-/// an HDR mode it can't produce. (The previous placeholder 3843 = 0xF03 wrongly claimed HEVC Main10 +
+/// host (a host without the `PUNKTFUNK_10BIT` opt-in, or a Linux host whose video source / encoder
+/// can't do Main10) must not invite a client into an HDR mode it can't produce. (The previous placeholder 3843 = 0xF03 wrongly claimed HEVC Main10 +
 /// 4:4:4 and *no* AV1.) 4:4:4 stays off entirely on GameStream: stock Moonlight is 4:2:0 —
 /// full-chroma is a punktfunk/1-native negotiation only (`crate::capture::capturer_supports_444`).
 pub const SERVER_CODEC_MODE_SUPPORT: u32 = SCM_H264 | SCM_HEVC | SCM_AV1_MAIN8;
 
 /// Whether this host can deliver an **HDR** (HEVC Main10 / BT.2020 PQ) GameStream — the single gate
-/// for advertising [`SCM_HEVC_MAIN10`] in serverinfo and `IsHdrSupported` per app, and for honoring a
-/// client's `dynamicRangeMode` request. HDR capture+encode is **Windows-only** (the Linux host is
-/// 8-bit, blocked upstream) and behind the operator's `PUNKTFUNK_10BIT` opt-in — the same policy gate
-/// the native punktfunk/1 plane honors. When this is true the IDD-push capturer streams HEVC Main10 PQ
-/// whenever the desktop is HDR, and a client HDR request makes the GameStream video path proactively
-/// enable advanced color on the per-session virtual display so PQ flows even from an SDR desktop.
+/// for advertising [`SCM_HEVC_MAIN10`] in serverinfo and `IsHdrSupported` per app, and (together
+/// with the live capture-side check at RTSP time) for honoring a client's `dynamicRangeMode`
+/// request. Behind the operator's `PUNKTFUNK_10BIT` opt-in — the same policy gate the native
+/// punktfunk/1 plane honors — on both OSes.
+///
+/// **Windows**: the IDD-push capturer streams HEVC Main10 PQ whenever the desktop is HDR, and a
+/// client HDR request proactively enables advanced color on the per-session virtual display so PQ
+/// flows even from an SDR desktop.
+///
+/// **Linux**: the GNOME 50+ portal **monitor mirror** (`video_source=portal`) can negotiate the
+/// 10-bit PQ formats while the mirrored monitor is in HDR mode, and the NVENC/VAAPI encoders have
+/// a probed Main10 path ([`crate::encode::can_encode_10bit`]). The virtual-output source stays SDR
+/// (Mutter's RecordVirtual streams are 8-bit-only upstream), so this is `false` for it. Whether
+/// the monitor is ACTUALLY in HDR mode right now is checked live at RTSP honor time
+/// ([`pf_capture::gnome_hdr_monitor_active`]) — this fn is the static serverinfo capability.
 pub fn host_hdr_capable() -> bool {
-    cfg!(target_os = "windows") && pf_host_config::config().ten_bit
+    if !pf_host_config::config().ten_bit {
+        return false;
+    }
+    #[cfg(target_os = "windows")]
+    {
+        true
+    }
+    #[cfg(target_os = "linux")]
+    {
+        pf_host_config::config().video_source.as_deref() == Some("portal")
+            && crate::encode::can_encode_10bit(crate::encode::Codec::H265)
+    }
+    #[cfg(not(any(target_os = "windows", target_os = "linux")))]
+    {
+        false
+    }
 }
 
 /// Stable host identity + advertised capabilities, shared across control-plane handlers.
@@ -141,8 +165,11 @@ pub struct AppState {
     pub rfi_range: std::sync::Arc<std::sync::Mutex<Option<(i64, i64)>>>,
     /// Persistent screen capturer, reused across streams so reconnects don't spawn a second
     /// (conflicting) screencast session. The video thread borrows it for the stream's duration
-    /// and returns it; `set_active` gates its cost while idle.
-    pub video_cap: std::sync::Arc<std::sync::Mutex<Option<Box<dyn crate::capture::Capturer>>>>,
+    /// and returns it; `set_active` gates its cost while idle. The slot's `bool` records whether
+    /// it was opened with the HDR (10-bit PQ) offer — a stream whose negotiated `hdr` differs
+    /// drops the pooled capturer and opens a fresh screencast session at the right depth
+    /// (mirroring the audio capturer's channel-count reuse gate).
+    pub video_cap: stream::CapturerSlot,
     /// Persistent audio capturer, reused across streams when the channel count still matches
     /// (avoids a PipeWire stream setup per reconnect); drained on reuse so no stale audio is
     /// sent, dropped + reopened when a session negotiates a different channel count.

crates/punktfunk-host/src/gamestream/rtsp.rs

@@ -396,18 +396,30 @@ fn stream_config(map: &HashMap<String, String>) -> Option<StreamConfig> {
         _ => Codec::H264,
     };
     // 10-bit/HDR request (Moonlight sets `dynamicRangeMode != 0` only when it both saw our Main10 SCM
-    // bit AND the user enabled HDR). Honor it only when the host can actually deliver Main10 (Windows +
-    // PUNKTFUNK_10BIT, `host_hdr_capable`); when honored, the video path proactively enables advanced
-    // color on the virtual display so a PQ stream flows even from an SDR desktop. A request we can't
-    // honor degrades to 8-bit SDR (and a desktop that is ALREADY HDR still streams PQ regardless, since
-    // the IDD-push capturer follows the display).
+    // bit AND the user enabled HDR). Honor it only when the host can actually deliver Main10
+    // (`host_hdr_capable` — Windows IDD-push, or the Linux GNOME 50+ portal mirror). On Windows,
+    // when honored, the video path proactively enables advanced color on the virtual display so a
+    // PQ stream flows even from an SDR desktop. On Linux the portal can only deliver PQ while the
+    // MIRRORED monitor is in HDR mode, so additionally probe the live colour mode here (one D-Bus
+    // round-trip, sync RTSP thread) — an SDR desktop honestly degrades to 8-bit SDR up front
+    // instead of running the capture negotiation into its timeout. A request we can't honor
+    // degrades to 8-bit SDR (and a Windows desktop that is ALREADY HDR still streams PQ
+    // regardless, since the IDD-push capturer follows the display).
     let hdr_requested = parse_u("x-nv-video[0].dynamicRangeMode").unwrap_or(0) != 0;
-    let hdr = hdr_requested && crate::gamestream::host_hdr_capable();
+    let mut hdr = hdr_requested && crate::gamestream::host_hdr_capable();
     if hdr_requested && !hdr {
         tracing::warn!(
             "client requested HDR (dynamicRangeMode != 0) but host is not HDR-capable — streaming 8-bit SDR"
         );
     }
+    #[cfg(target_os = "linux")]
+    if hdr && !pf_capture::gnome_hdr_monitor_active() {
+        tracing::warn!(
+            "client requested HDR but no monitor is in BT.2100 (HDR) colour mode — enable HDR in \
+             GNOME Settings → Displays (GNOME 50+) to stream it; streaming 8-bit SDR"
+        );
+        hdr = false;
+    }
     // The client's requested CSC (moonlight-common-c SdpGenerator.c: `encoderCscMode =
     // (colorspace << 1) | fullRange` — colorspace 0=Rec601, 1=Rec709, 2=Rec2020). Moonlight
     // renderers configure their YUV→RGB from this REQUESTED value (not the bitstream VUI), so a

crates/punktfunk-host/src/gamestream/stream.rs

@@ -34,8 +34,9 @@ pub struct StreamConfig {
 }
 
 /// Slot for the persistent screen capturer, shared with the control plane and reused across
-/// streams so a reconnect doesn't open a second (conflicting) screencast session.
-pub type CapturerSlot = Arc<std::sync::Mutex<Option<Box<dyn Capturer>>>>;
+/// streams so a reconnect doesn't open a second (conflicting) screencast session. The `bool` is
+/// the pooled capturer's HDR-ness (see `AppState::video_cap`).
+pub type CapturerSlot = Arc<std::sync::Mutex<Option<(Box<dyn Capturer>, bool)>>>;
 
 /// A pending client reference-frame-invalidation range (lost `firstFrame..=lastFrame`), set by the
 /// control plane and drained by the video thread (see [`AppState::rfi_range`](super::AppState)).
@@ -120,7 +121,7 @@ fn run(
     running: &Arc<AtomicBool>,
     force_idr: &AtomicBool,
     rfi_range: &std::sync::Mutex<Option<(i64, i64)>>,
-    video_cap: &std::sync::Mutex<Option<Box<dyn Capturer>>>,
+    video_cap: &std::sync::Mutex<Option<(Box<dyn Capturer>, bool)>>,
     // Shared stats recorder for the web-console capture/graph. Threaded into `stream_body` (the
     // encode loop); per-frame sample emission is wired by a later pass.
     stats: &Arc<crate::stats_recorder::StatsRecorder>,
@@ -243,15 +244,31 @@ fn run(
     }
 
     // Reuse the persistent capturer (one screencast session → clean reconnect); create it on
-    // the first stream. Borrow it for this stream and return it on exit.
-    let mut capturer: Box<dyn Capturer> = match video_cap.lock().unwrap().take() {
+    // the first stream. Borrow it for this stream and return it on exit. Reuse is gated on the
+    // pooled capturer's HDR-ness matching this stream's negotiated `cfg.hdr` — the depth is a
+    // PipeWire-negotiation-time property of the screencast session, so an HDR↔SDR change needs a
+    // fresh session (same pattern as the audio capturer's channel-count gate).
+    let pooled = match video_cap.lock().unwrap().take() {
+        Some((c, was_hdr)) if was_hdr == cfg.hdr => Some(c),
+        Some((c, was_hdr)) => {
+            tracing::info!(
+                was_hdr,
+                want_hdr = cfg.hdr,
+                "video source: pooled capturer depth mismatch — opening a fresh screencast session"
+            );
+            drop(c);
+            None
+        }
+        None => None,
+    };
+    let mut capturer: Box<dyn Capturer> = match pooled {
         Some(c) => {
             tracing::info!("video source: reusing capturer");
             c
         }
         None if pf_host_config::config().video_source.as_deref() == Some("portal") => {
-            tracing::info!("video source: portal desktop capture");
-            capture::open_portal_monitor().context("open portal capturer")?
+            tracing::info!(hdr = cfg.hdr, "video source: portal desktop capture");
+            capture::open_portal_monitor(cfg.hdr).context("open portal capturer")?
         }
         None => {
             tracing::info!("video source: synthetic test pattern");
@@ -272,7 +289,7 @@ fn run(
         &client_label,
     );
     capturer.set_active(false);
-    *video_cap.lock().unwrap() = Some(capturer);
+    *video_cap.lock().unwrap() = Some((capturer, cfg.hdr));
     result
 }
 
@@ -380,7 +397,9 @@ fn open_gs_virtual_source(
     // HDR: pass the negotiated `cfg.hdr` (client asked for HDR AND the host can deliver it). On the
     // Windows IDD-push path this proactively enables advanced color on the virtual display so a Main10
     // PQ stream flows even from an SDR desktop; an already-HDR desktop streams PQ regardless (the
-    // capturer follows the display). No-op on Linux (8-bit, and `cfg.hdr` is always false there).
+    // capturer follows the display). No-op on Linux: virtual-output capture is SDR-only upstream
+    // (Mutter RecordVirtual), and `host_hdr_capable` therefore keeps `cfg.hdr` false for this
+    // source — the Linux HDR path is the portal monitor mirror (`video_source=portal`).
     let capturer = capture::capture_virtual_output(
         vout,
         capture::OutputFormat::resolve(cfg.hdr, crate::encode::resolved_backend_is_gpu()),
@@ -399,7 +418,11 @@ fn open_gs_virtual_source(
 fn gs_bit_depth(format: crate::capture::PixelFormat) -> u8 {
     use crate::capture::PixelFormat;
     match format {
-        PixelFormat::P010 | PixelFormat::Rgb10a2 => 10,
+        // Windows IDD-push HDR formats, and the Linux GNOME 50+ portal HDR formats.
+        PixelFormat::P010
+        | PixelFormat::Rgb10a2
+        | PixelFormat::X2Rgb10
+        | PixelFormat::X2Bgr10 => 10,
         _ => 8,
     }
 }