fix(kwin): install >60Hz custom modes in-process too — KWin path is now kscreen-free

Follow-up to the previous commit: the only KWin topology op still shelling out to
kscreen-doctor was the >60 Hz custom-mode INSTALL (`set_custom_refresh`). Port it
to `kde_output_management_v2` so the whole KWin path is kscreen-free on modern KWin.

`kwin_output_mgmt::set_custom_mode` builds a one-entry `kde_mode_list_v2`
(set_resolution / set_refresh_rate / set_reduced_blanking=full / add_mode), applies
it via `kde_output_configuration_v2.set_custom_modes` (since 18), waits for KWin to
generate the mode (its CVT generator may align the width down — matched with the
same height-exact / width-within-8 / refresh-within-1Hz gate as before), then
selects it — which changes the output size and drives the sacrificial-birth stream
renegotiation. `kwin::create`'s want_high branch tries this first and only resolves
a kscreen id + calls `set_custom_refresh` if it returns None (pre-6.6 KWin without
set_custom_modes, or the compositor not answering) — so a >60 Hz session on a box
where kscreen-doctor wedges no longer eats a 5 s resolve timeout either.

Bonus: `set_custom_modes` REPLACES the custom list, so reconnects are idempotent —
no more one-custom-mode-per-connect growth of the user's display list.

Verified: cargo test -p pf-vdisplay (73 pass), clippy -D warnings clean, fmt clean.
The set_custom_modes round-trip is proven live on KWin 6.6.4: apply=applied, and a
1648x928@75 request generates 1648x928@74.901 (CVT fractional refresh, caught by the
1 Hz tolerance). The mode SELECT + full >60 Hz stream is still owed on-glass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

crates/pf-vdisplay/src/vdisplay/linux/kwin.rs

@@ -264,12 +264,12 @@ impl VirtualDisplay for KwinDisplay {
         // on virtual outputs since KWin 6.6) then changes the SIZE, and the first buffers recorded
         // after the consumer connects trigger KWin's resize → a renegotiation to that mode. The
         // capturer holds frames until that lands (`expect_exact_dims`), so the pipeline never
-        // builds against the birth mode. First cut shells out to kscreen-doctor; the in-process
-        // kde_output_management_v2 client is a follow-up. `set_custom_refresh` reads back what
-        // KWin *actually* gave — both the rate (so the encoder paces to the real source) and the
-        // size, which KWin's CVT generator may have aligned down (see `CVT_H_GRANULARITY`). At
-        // ≤60 Hz there's nothing to install — the output is born at the real size and 60 Hz is the
-        // offer anyway.
+        // builds against the birth mode. The install/select runs in-process over
+        // kde_output_management_v2 (`kwin_output_mgmt::set_custom_mode`), with kscreen-doctor
+        // (`set_custom_refresh`) as the fallback; either reads back what KWin *actually* gave — both
+        // the rate (so the encoder paces to the real source) and the size, which KWin's CVT
+        // generator may have aligned down (see `CVT_H_GRANULARITY`). At ≤60 Hz there's nothing to
+        // install — the output is born at the real size and 60 Hz is the offer anyway.
         let want_high = mode.refresh_hz > 60;
         let birth_h = if want_high { height + 16 } else { height };
         let (mut node_id, mut stop) = spawn_vout(width, birth_h)?;
@@ -283,8 +283,8 @@ impl VirtualDisplay for KwinDisplay {
         );
         // Topology + positioning address OUR output by its kde_output_management UUID (resolved
         // in-process in `apply_topology`, supersede-robust) — no early kscreen-doctor resolve, so
-        // the 60 Hz path never shells out. Only the >60 Hz custom-mode install below still uses
-        // kscreen-doctor (its in-process port — set_custom_modes/add_cvt — is a follow-up).
+        // the path never shells out. `Virtual-<name>` is the name KWin exposes our output as.
+        let our_prefix = format!("Virtual-{name}");
         let mut expect_exact_dims = false;
         // The size the output actually ENDS UP at — the request, unless KWin's CVT generator had to
         // shrink the width to the cell grain (see `CVT_H_GRANULARITY`). Reported as the output's
@@ -292,15 +292,27 @@ impl VirtualDisplay for KwinDisplay {
         // encoder opens against, so a CVT-aligned mode flows end-to-end instead of starving.
         let mut final_dims = (width, height);
         let achieved_hz = if want_high {
-            // ⚠️ ADDRESS BY NUMERIC KSCREEN ID, NEVER BY NAME: a supersede (mode switch) creates the
-            // replacement output — SAME per-slot name, deliberately, for KWin's per-name config
-            // persistence — while the superseded sibling is still alive (create-before-drop). A
-            // name-addressed kscreen command would hit the FIRST match = the OLD output. Resolve OUR
-            // output's kscreen id (managed-prefix name AND current mode == the birth size; newest id
-            // wins) for the custom-mode install, and keep it as the kscreen-doctor position fallback.
+            // >60 Hz needs the real high-refresh custom mode installed + selected (sacrificial-birth,
+            // see above). In-process over kde_output_management_v2 first (no kscreen-doctor); fall
+            // back to the kscreen-doctor shell-out on pre-6.6 KWin (no `set_custom_modes`) or if the
+            // compositor doesn't answer in budget.
+            let active = crate::kwin_output_mgmt::set_custom_mode(
+                &our_prefix,
+                width,
+                birth_h,
+                width,
+                height,
+                mode.refresh_hz,
+            )
+            .or_else(|| {
+                // ⚠️ ADDRESS BY NUMERIC KSCREEN ID, NEVER BY NAME: a supersede reuses the per-slot
+                // name while the superseded sibling is still alive, so a name-addressed kscreen
+                // command hits the OLD output. Resolve our kscreen id for the shell-out install +
+                // keep it as apply_position's kscreen fallback.
                 let addr = resolve_kscreen_addr(&name, width, birth_h);
                 self.last_name = Some(addr.clone());
-            let active = set_custom_refresh(width, height, mode.refresh_hz, &addr);
+                set_custom_refresh(width, height, mode.refresh_hz, &addr)
+            });
             // Accept only an active mode that IS our custom one: the exact requested height, and a
             // width at or just below the request (a CVT alignment). That also proves the output
             // left the sacrificial birth size, so the recording stream will renegotiate to it.
@@ -349,7 +361,6 @@ impl VirtualDisplay for KwinDisplay {
         // bootstrap output. Applied over kde_output_management_v2 in-process (immune to a wedged
         // kscreen-doctor backend; see `apply_topology`), with a kscreen-doctor fallback. `disabled`
         // is the physical/bootstrap outputs, each `(name, "WxH@Hz")`, to restore on teardown.
-        let our_prefix = format!("Virtual-{name}");
         let disabled = self.apply_topology(&name, &our_prefix, final_dims);
         // A plain managed name is enough for apply_position's kscreen-doctor fallback when the
         // in-process UUID path isn't set (single-output sessions are unambiguous; a supersede uses

crates/pf-vdisplay/src/vdisplay/linux/kwin_output_mgmt.rs

@@ -70,6 +70,7 @@ pub mod management {
 
 use device::kde_output_device_mode_v2::{Event as ModeEvent, KdeOutputDeviceModeV2 as DeviceMode};
 use device::kde_output_device_v2::{Event as DeviceEvent, KdeOutputDeviceV2 as OutputDevice};
+use management::kde_mode_list_v2::KdeModeListV2 as ModeList;
 use management::kde_output_configuration_v2::{
     Event as ConfigEvent, KdeOutputConfigurationV2 as OutputConfig,
 };
@@ -93,6 +94,11 @@ const OP_BUDGET: Duration = Duration::from_secs(3);
 /// Poll slice while waiting on the Wayland fd (matches the keepalive loop's cadence in `kwin.rs`).
 const POLL_MS: i32 = 100;
 
+/// KWin's CVT generator aligns a custom mode's width DOWN to a multiple of this (libxcvt's cell
+/// grain), so the mode it builds for a `set_custom_modes` request may be a few px narrower than
+/// asked — matches `kwin::CVT_H_GRANULARITY`. Used when matching the generated mode back.
+const CVT_H_GRANULARITY: u32 = 8;
+
 /// Which topology to apply once our output is resolved.
 #[derive(Clone, Copy, PartialEq, Eq)]
 pub(crate) enum TopologyKind {
@@ -200,6 +206,19 @@ impl Dispatch<OutputManagement, ()> for State {
     }
 }
 
+// A client-built custom-mode list has no events; it just needs a Dispatch impl to be created.
+impl Dispatch<ModeList, ()> for State {
+    fn event(
+        _: &mut Self,
+        _: &ModeList,
+        _: management::kde_mode_list_v2::Event,
+        _: &(),
+        _: &Connection,
+        _: &QueueHandle<Self>,
+    ) {
+    }
+}
+
 /// The device's UserData is its global `name` number.
 impl Dispatch<OutputDevice, u32> for State {
     fn event(
@@ -395,6 +414,16 @@ impl Session {
             .create_configuration(&qh, ())
     }
 
+    /// A fresh (empty) `kde_mode_list_v2` on this connection, for a `set_custom_modes` request.
+    fn new_mode_list(&self) -> ModeList {
+        let qh = self.queue.handle();
+        self.state
+            .management
+            .as_ref()
+            .unwrap()
+            .create_mode_list(&qh, ())
+    }
+
     /// `apply()` the config and pump until `applied`/`failed` or the deadline. Returns the verdict
     /// (`true` applied, `false` failed/timeout).
     fn apply(&mut self, config: &OutputConfig, deadline: Instant) -> bool {
@@ -545,6 +574,139 @@ pub(crate) fn apply_topology(
     }
 }
 
+/// Install + select a `want_w`×`want_h`@`want_hz` custom mode on the just-created virtual output
+/// (name starts with `our_prefix`, currently at its sacrificial birth size `birth_w`×`birth_h`) —
+/// entirely over `kde_output_management_v2`, the in-process replacement for the `kscreen-doctor`
+/// `addCustomMode` + `mode` shell-out (`set_custom_refresh`).
+///
+/// `set_custom_modes` hands KWin a one-entry mode list; KWin generates the CVT timing (so the width
+/// may align DOWN — see [`CVT_H_GRANULARITY`]) and adds the mode. We then SELECT it, which changes
+/// the output's size and triggers the screencast stream's renegotiation to the real refresh (the
+/// sacrificial-birth mechanism in `kwin::create`). Returns the ACTIVE mode read back after selection
+/// (Hz rounded), or `None` if management is absent, the output/generated mode never appeared, or an
+/// apply didn't confirm — the caller then falls back to `kscreen-doctor`. `set_custom_modes` REPLACES
+/// the custom list (idempotent across reconnects — no per-connect list growth), and it is `since 18`,
+/// so pre-6.6 KWin without it simply takes the `None` → kscreen-doctor path.
+pub(crate) fn set_custom_mode(
+    our_prefix: &str,
+    birth_w: u32,
+    birth_h: u32,
+    want_w: u32,
+    want_h: u32,
+    want_hz: u32,
+) -> Option<(u32, u32, u32)> {
+    let mut sess = Session::open()?;
+    let deadline = Instant::now() + OP_BUDGET;
+
+    // `set_custom_modes` is `since 18`; calling it on an older bound management object is a protocol
+    // error, so bail to the kscreen-doctor fallback there (pre-6.6 KWin). Our bound version is
+    // `min(advertised, MGMT_MAX)`.
+    if sess.state.mgmt_name_version.map(|(_, v)| v).unwrap_or(0) < 18 {
+        return None;
+    }
+
+    // Resolve our output at its birth size (newest global wins a supersede).
+    let our_proxy = sess
+        .state
+        .devices
+        .values()
+        .filter(|d| {
+            d.name.as_deref().is_some_and(|n| n.starts_with(our_prefix))
+                && sess.current_dims(d).map(|(w, h, _)| (w, h)) == Some((birth_w, birth_h))
+        })
+        .max_by_key(|d| d.global)
+        .and_then(|d| d.proxy.clone())?;
+    let our_key = our_proxy.id();
+
+    let want_mhz = want_hz.saturating_mul(1000);
+    // A generated mode IS our custom one iff: exact height, width at/just-below the request (a CVT
+    // alignment), and refresh within 1 Hz — which excludes the sacrificial 60 Hz birth mode.
+    let mode_matches = move |st: &State, mid: &ObjectId| -> bool {
+        st.mode_dims.get(mid).is_some_and(|&(w, h, mhz)| {
+            h == want_h
+                && w <= want_w
+                && want_w - w < CVT_H_GRANULARITY
+                && (mhz as i64 - want_mhz as i64).abs() <= 1000
+        })
+    };
+
+    // Build a one-entry custom-mode list (full blanking, like kscreen-doctor's `.full`) and install it.
+    let mode_list = sess.new_mode_list();
+    mode_list.set_resolution(want_w, want_h);
+    mode_list.set_refresh_rate(want_mhz);
+    mode_list.set_reduced_blanking(0);
+    mode_list.add_mode();
+    let config = sess.new_config();
+    config.set_custom_modes(&our_proxy, &mode_list);
+    let installed = sess.apply(&config, deadline);
+    config.destroy();
+    mode_list.destroy();
+    if !installed {
+        return None;
+    }
+
+    // Wait for KWin to generate the mode and advertise it on the output.
+    let found = |st: &State| -> bool {
+        st.devices
+            .get(&our_key)
+            .is_some_and(|d| d.modes.iter().any(|(mid, _)| mode_matches(st, mid)))
+    };
+    if !sess.pump_until(deadline, found) {
+        tracing::warn!(
+            want_w,
+            want_h,
+            want_hz,
+            "KWin output management: generated custom mode never appeared — kscreen-doctor fallback"
+        );
+        return None;
+    }
+
+    // Grab the generated mode's proxy, then select it (this is what changes the size).
+    let mode_proxy = {
+        let dev = sess.state.devices.get(&our_key)?;
+        dev.modes
+            .iter()
+            .find(|(mid, _)| mode_matches(&sess.state, mid))
+            .map(|(_, p)| p.clone())?
+    };
+    let config = sess.new_config();
+    config.mode(&our_proxy, &mode_proxy);
+    let selected = sess.apply(&config, deadline);
+    config.destroy();
+    if !selected {
+        return None;
+    }
+
+    // Read back the active mode after selection — the size that really landed paces the encoder.
+    let want_dims = sess
+        .state
+        .mode_dims
+        .get(&mode_proxy.id())
+        .map(|&(w, h, _)| (w, h));
+    let landed = |st: &State| -> bool {
+        st.devices
+            .get(&our_key)
+            .and_then(|d| d.current_mode.as_ref())
+            .and_then(|mid| st.mode_dims.get(mid))
+            .map(|&(w, h, _)| (w, h))
+            == want_dims
+    };
+    sess.pump_until(deadline, landed);
+    let dev = sess.state.devices.get(&our_key)?;
+    let (cw, ch, cmhz) = sess.current_dims(dev)?;
+    let hz = ((cmhz as f64) / 1000.0).round() as u32;
+    tracing::info!(
+        want_w,
+        want_h,
+        want_hz,
+        active_w = cw,
+        active_h = ch,
+        active_hz = hz,
+        "KWin output management: custom mode installed + selected (in-process)"
+    );
+    Some((cw, ch, hz.max(1)))
+}
+
 /// Re-enable outputs by name at their captured `WxH@Hz` modes (teardown), in-process. Returns
 /// `true` if the config applied; `false` (compositor unresponsive / management absent) tells the
 /// caller to fall back to `kscreen-doctor`.