Compare commits

..
Author SHA1 Message Date
enricobuehlerandClaude Fable 5 12df1388de feat(apple/input): desktop (absolute) mouse mode on macOS — remote-desktop sweep M1
ci / web (pull_request) Successful in 53s
ci / docs-site (pull_request) Successful in 53s
apple / swift (pull_request) Successful in 1m21s
apple / screenshots (pull_request) Has been skipped
windows / build (aarch64-pc-windows-msvc) (pull_request) Successful in 4m15s
ci / bench (pull_request) Successful in 5m35s
windows / build (x86_64-pc-windows-msvc) (pull_request) Successful in 1m40s
android / android (pull_request) Successful in 11m14s
ci / rust (pull_request) Successful in 21m49s
Folds the parked client-side-cursor machinery (cursorMode auto/always/
never, hidden while disabled) into the cross-client mouse model:
MouseInputMode capture|desktop under DefaultsKey.mouseMode, picked in
Settings ▸ Keyboard & mouse (macOS), resolved at session start and
gated off on gamescope hosts (relative-only EIS).

Desktop model = the un-neutered absolute path with the SDL cursor
policy: pointer never disassociated (enters/leaves the stream freely),
monitor forwards letterboxed absolute positions, and the local cursor
hides only while over the view via an invisible-cursor rect (the
host's composited cursor is the one you see; AppKit manages the rect,
so no hide/unhide balancing). ⌘⇧C becomes ⌃⌥⇧M — the same chord as
the SDL clients — flipping the model live with an atomic
release/re-engage.

Verified: swift build + full test suites green (macOS arm64).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 00:03:19 +02:00
enricobuehlerandClaude Fable 5 7295ae70f9 feat(client/input): desktop (absolute) mouse mode — remote-desktop sweep M1
New physical-mouse model beside capture: Desktop leaves the pointer
uncaptured and sends absolute positions through the letterbox
(MouseMoveAbs — every host injector already consumes it). Capture
stays the default and the game model.

- pf-client-core: MouseMode (capture|desktop) persisted in Settings,
  default capture so existing stores are unchanged.
- pf-presenter: Capture grows a desktop model — latest-wins pending
  abs position coalesced per loop iteration (same 1000 Hz discipline
  as relative), flushed before clicks/keys/wheel so they land where
  the cursor is; Ctrl+Alt+Shift+M flips the model live; local cursor
  stays hidden over the window (the host's composited cursor is the
  one you see until the M2 cursor channel); Windows keyboard grab
  only engages for capture (a desktop stream is something you
  Alt-Tab away from).
- gamescope gating: its EIS is relative-only, so desktop mode is
  pinned off there (resolved_compositor), with a log note.
- Settings surfaces: GTK row (dynamic caption), WinUI combo,
  console-UI row + step test, capture-hint line.

Verified: fmt + clippy -D warnings + tests (33/40/19) on Linux .21;
clippy -D warnings for all five crates incl. punktfunk-client-windows
native on .173.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 23:52:16 +02:00
154 changed files with 2091 additions and 14793 deletions
+2 -12
View File
@@ -46,23 +46,13 @@ jobs:
# SHA-pinned: this workflow's release job carries the signing keystore + Play service-account
# secrets, so a moved tag on a third-party action could exfiltrate them. v3 = 9fc6c4e.
uses: android-actions/setup-android@9fc6c4e9069bf8d3d10b2204b1fb8f6ef7065407 # v3
with:
# Only platform-tools — NOT the action's default legacy `tools`, whose dependency chain
# drags in the ~250 MB emulator nobody here runs (instrumentation tests are deferred).
# That download was the single flakiest piece of this job: the shared runner fleet drops
# packets under parallel-job load and sdkmanager's streamed unzip turns a truncated
# stream into "Error on ZipFile unknown archive" (observed 2026-07-22, twice).
packages: platform-tools
- name: NDK r30 + platform 36 + build-tools + CMake (libopus cross-build)
# cmake;3.22.1 installs cmake + ninja under $ANDROID_SDK/cmake/3.22.1/bin — the exact path
# kit/build.gradle.kts prepends to PATH for cargo-ndk's audiopus_sys (libopus) CMake build.
# Note: platforms;android-37 is sometimes missing from standard channels; AGP will
# Note: platforms;android-37 is sometimes missing from standard channels; AGP will
# auto-download it if needed during the build.
# retry.sh: sdkmanager is a single-shot multi-hundred-MB fetch, exactly the class the
# helper exists for (fleet-load packet drops truncate the stream mid-unzip); a failed
# attempt leaves no partial package behind, so a plain re-invoke is safe.
run: bash scripts/ci/retry.sh 4 sdkmanager "platform-tools" "platforms;android-36" "build-tools;37.0.0" "ndk;30.0.14904198" "cmake;3.22.1"
run: sdkmanager "platform-tools" "platforms;android-36" "build-tools;37.0.0" "ndk;30.0.14904198" "cmake;3.22.1"
- name: Caches (cargo + gradle)
uses: actions/cache@v4
-39
View File
@@ -1,39 +0,0 @@
# Announce a stable release to the Discord #releases channel.
#
# This is the deliberate "go" step for a release. Release notes live in the repo at
# docs/releases/<tag>.md and are seeded into the Gitea release body at creation by the build
# workflows (scripts/ci/gitea-release.sh), so the release is never noteless. Once every
# platform's CI is green for a tag, dispatch this workflow with that tag: it re-asserts the notes
# file over the live release and posts a formatted embed to #releases.
#
# Manual on purpose — pressing "go" is the quality gate that says "all platforms built, notes are
# final, tell the community." It is NOT wired to the tag push, so a half-built or failed release
# is never announced. Stable-only: a -rc/pre-release tag is refused unless allow_prerelease=true.
#
# Requires the repo secret DISCORD_RELEASE_WEBHOOK (the #releases channel webhook URL); GITEA auth
# reuses REGISTRY_TOKEN like the other release workflows.
name: announce
on:
workflow_dispatch:
inputs:
tag:
description: "Release tag to announce (e.g. v0.18.0)"
required: true
allow_prerelease:
description: "Announce even if the tag is a pre-release (-rc)"
required: false
default: "false"
jobs:
announce:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Post release announcement to Discord
env:
GITEA_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
DISCORD_RELEASE_WEBHOOK: ${{ secrets.DISCORD_RELEASE_WEBHOOK }}
ALLOW_PRERELEASE: ${{ inputs.allow_prerelease }}
run: bash scripts/ci/discord-announce.sh "${{ inputs.tag }}"
+25 -12
View File
@@ -73,20 +73,33 @@ jobs:
# sufficient — the Tooling step's dnf install pulls a systemd package upgrade whose RPM
# trigger re-runs authselect and regenerates this file, undoing the fix. It's reapplied
# there, right before the first `flatpak` network call.
- name: Fix container DNS (drop nss-resolve)
- name: Fix container DNS (drop nss-resolve, resolve over TCP)
run: |
sed -i 's/resolve \[!UNAVAIL=return\] //' /etc/nsswitch.conf
# History: this step used to ALSO force glibc onto TCP DNS (`options use-vc`) because
# the runner fleet's Docker embedded resolver dropped UDP lookups under parallel-job
# load (investigated 2026-07-11; v0.15.0/v0.16.0 each burned retry.sh's whole budget).
# That root cause is now fixed at the infra level (2026-07-22): the runner host runs a
# local dnsmasq cache on the docker bridge and daemon.json points every job container
# at it, so lookups terminate on-box instead of crossing the saturated uplink — the
# UDP path is reliable again. The TCP path through the same chain proved FLAKY under
# fleet concurrency (flatpak remote-add failed 10/10 with instant NXDOMAIN while dnf
# in the same container resolved fine), so `use-vc` flipped from mitigation to sole
# cause of this leg's failures — removed. retry.sh (10×) stays as the backstop for
# genuine upstream blips.
# Resolve over TCP instead of UDP. The documented root cause of the flathub
# bootstrap failures (investigated 2026-07-11, see the Tooling step) is this box's
# Docker embedded resolver at 127.0.0.11 DROPPING UDP lookups while the shared
# runner fleet is saturated — a datagram nobody retransmits, so the lookup just
# times out. The answer then was to widen retry.sh's budget to 10 attempts (~9 min),
# which is enough to outlast a main push's ~8-workflow fan-out but NOT a TAG push's
# 13: v0.15.0 (twice) and v0.16.0 each burned all 10 attempts and failed the job,
# each needing a manual re-run.
#
# `use-vc` makes glibc use TCP, where the kernel retransmits and the query cannot be
# silently lost under load. Same resolver, same search path — only the transport
# changes, so internal names (git.unom.io) resolve exactly as before; deliberately
# NO extra nameservers, which would risk answering an internal name from a public
# resolver. Docker's embedded DNS serves TCP on 127.0.0.11:53 as well as UDP.
# retry.sh stays as the backstop for genuine upstream blips.
#
# Non-fatal: Docker bind-mounts /etc/resolv.conf and can present it read-only, and a
# DNS tuning that cannot be applied must not be what fails the release build — that
# would trade an occasional re-run for a hard stop. Falling back to UDP just restores
# today's behaviour, which retry.sh already covers.
if ! grep -q '^options .*use-vc' /etc/resolv.conf 2>/dev/null; then
echo 'options use-vc timeout:3 attempts:3' >> /etc/resolv.conf \
|| echo "::warning::could not set use-vc (read-only resolv.conf?); staying on UDP"
fi
cat /etc/resolv.conf || true
# fedora:43 has no node, but actions/checkout (a JS action) needs it. A plain `run:` step
-70
View File
@@ -393,76 +393,6 @@ jobs:
-authenticationKeyID "${{ secrets.ASC_API_KEY_ID }}" \
-authenticationKeyIssuerID "${{ secrets.ASC_API_ISSUER_ID }}"
- name: iOS — export .ipa (Gitea release + run artifact)
# The TestFlight step above uploads straight to App Store Connect (destination=upload) and
# leaves NO .ipa on disk. Re-export the SAME archive with destination=export to get an
# App Store distribution-signed .ipa for the Gitea release + the run artifacts. Same gate as
# that archive; a warn+skip (never fails the best-effort iOS leg) if the archive is absent,
# e.g. a workflow_dispatch with testflight=false. NOTE: an App Store-signed .ipa installs
# only via TestFlight/App Store, not by direct sideload — it's a release/archival artifact.
if: gitea.event_name != 'workflow_dispatch' || inputs.testflight == 'true'
id: ios_ipa
run: |
ARCHIVE="$RUNNER_TEMP/Punktfunk-ios.xcarchive"
if [ ! -d "$ARCHIVE" ]; then
echo "::warning::iOS archive not found — skipping .ipa export"
exit 0
fi
PROFILE="Punktfunk iOS App Store Distribution"
WIDGET_PROFILE="Punktfunk iOS Widgets App Store Distribution"
# destination=export writes the .ipa to -exportPath; otherwise identical manual signing to
# the upload plist (both profiles, Apple Distribution). No ASC key needed — no network.
cat > "$RUNNER_TEMP/export-appstore-ipa.plist" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>method</key><string>app-store-connect</string>
<key>destination</key><string>export</string>
<key>teamID</key><string>$TEAM_ID</string>
<key>signingStyle</key><string>manual</string>
<key>signingCertificate</key><string>Apple Distribution</string>
<key>provisioningProfiles</key>
<dict>
<key>io.unom.punktfunk</key><string>$PROFILE</string>
<key>io.unom.punktfunk.widgets</key><string>$WIDGET_PROFILE</string>
</dict>
</dict>
</plist>
EOF
DEVELOPER_DIR="$XCODE_DEV_DIR" xcodebuild -exportArchive \
-archivePath "$ARCHIVE" \
-exportOptionsPlist "$RUNNER_TEMP/export-appstore-ipa.plist" \
-exportPath "$RUNNER_TEMP/export-ipa"
SRC=$(ls "$RUNNER_TEMP/export-ipa/"*.ipa 2>/dev/null | head -1)
[ -n "$SRC" ] || { echo "::warning::no .ipa was produced by export"; exit 0; }
mkdir -p "$GITHUB_WORKSPACE/dist"
IPA="$GITHUB_WORKSPACE/dist/Punktfunk-$VERSION.ipa"
mv "$SRC" "$IPA"
echo "IPA=$IPA" >> "$GITHUB_ENV"
echo "ipa=dist/Punktfunk-$VERSION.ipa" >> "$GITHUB_OUTPUT"
echo "exported $IPA"
- name: Attach .ipa to the workflow run
if: steps.ios_ipa.outputs.ipa != ''
# v3, not v4: Gitea's artifact backend identifies as GHES, which upload-artifact@v4 refuses
# (same reason as android.yml / apple.yml). Download is a zip of the .ipa.
uses: actions/upload-artifact@v3
with:
name: punktfunk-ios-ipa
path: ${{ steps.ios_ipa.outputs.ipa }}
if-no-files-found: warn
retention-days: 30
- name: Attach .ipa to the Gitea release (stable tags only)
if: startsWith(gitea.ref, 'refs/tags/v') && steps.ios_ipa.outputs.ipa != ''
env:
GITEA_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
run: |
. scripts/ci/gitea-release.sh
RID=$(ensure_release "$GITHUB_REF_NAME" "$GITHUB_REF_NAME" auto)
upsert_asset "$RID" "$IPA" "Punktfunk-$VERSION.ipa"
- name: tvOS — archive + upload to TestFlight
# Canary + stable, the same track as iOS/macOS — the tvOS xcframework slice is now built
# on every apple push (above), so this matches the iOS step's gate exactly.
Generated
+27 -55
View File
@@ -1459,16 +1459,6 @@ dependencies = [
"version_check",
]
[[package]]
name = "gethostname"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bd49230192a3797a9a4d6abe9b3eed6f7fa4c8a8a4947977c6f80025f92cbd8"
dependencies = [
"rustix",
"windows-link 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "getrandom"
version = "0.2.17"
@@ -2194,7 +2184,7 @@ dependencies = [
[[package]]
name = "latency-probe"
version = "0.19.0"
version = "0.17.2"
[[package]]
name = "lazy_static"
@@ -2299,7 +2289,7 @@ dependencies = [
[[package]]
name = "libvpl-sys"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"bindgen",
"cmake",
@@ -2334,7 +2324,7 @@ checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "loss-harness"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"punktfunk-core",
]
@@ -2823,7 +2813,7 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pf-capture"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ashpd",
@@ -2839,12 +2829,11 @@ dependencies = [
"tokio",
"tracing",
"windows 0.62.2 (registry+https://github.com/rust-lang/crates.io-index)",
"x11rb",
]
[[package]]
name = "pf-client-core"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ash",
@@ -2868,7 +2857,7 @@ dependencies = [
[[package]]
name = "pf-clipboard"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ashpd",
@@ -2886,7 +2875,7 @@ dependencies = [
[[package]]
name = "pf-console-ui"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ash",
@@ -2907,7 +2896,7 @@ dependencies = [
[[package]]
name = "pf-encode"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ash",
@@ -2931,7 +2920,7 @@ dependencies = [
[[package]]
name = "pf-ffvk"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"ash",
"bindgen",
@@ -2940,7 +2929,7 @@ dependencies = [
[[package]]
name = "pf-frame"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"libc",
@@ -2952,7 +2941,7 @@ dependencies = [
[[package]]
name = "pf-gpu"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"pf-host-config",
@@ -2966,11 +2955,11 @@ dependencies = [
[[package]]
name = "pf-host-config"
version = "0.19.0"
version = "0.17.2"
[[package]]
name = "pf-inject"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ashpd",
@@ -2998,14 +2987,14 @@ dependencies = [
[[package]]
name = "pf-paths"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"tracing",
]
[[package]]
name = "pf-presenter"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ash",
@@ -3020,7 +3009,7 @@ dependencies = [
[[package]]
name = "pf-vdisplay"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ashpd",
@@ -3050,7 +3039,7 @@ dependencies = [
[[package]]
name = "pf-win-display"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"pf-paths",
@@ -3062,7 +3051,7 @@ dependencies = [
[[package]]
name = "pf-zerocopy"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ash",
@@ -3269,7 +3258,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-android"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"android_logger",
"jni",
@@ -3285,7 +3274,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-linux"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"async-channel",
@@ -3301,7 +3290,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-session"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"pf-client-core",
@@ -3316,7 +3305,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-windows"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"async-channel",
"ffmpeg-next",
@@ -3335,7 +3324,7 @@ dependencies = [
[[package]]
name = "punktfunk-core"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"aes-gcm",
"bytes",
@@ -3367,7 +3356,7 @@ dependencies = [
[[package]]
name = "punktfunk-host"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"aes",
"aes-gcm",
@@ -3451,7 +3440,7 @@ dependencies = [
[[package]]
name = "punktfunk-probe"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"mdns-sd",
@@ -3465,7 +3454,7 @@ dependencies = [
[[package]]
name = "punktfunk-tray"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"anyhow",
"ksni",
@@ -3488,7 +3477,7 @@ checksum = "d55d956fa96f5ec02be2e13af0e20391a5aa83d6a074e3ad368959d0fab299ea"
[[package]]
name = "pyrowave-sys"
version = "0.19.0"
version = "0.17.2"
dependencies = [
"bindgen",
"cmake",
@@ -5897,23 +5886,6 @@ version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "x11rb"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9993aa5be5a26815fe2c3eacfc1fde061fc1a1f094bf1ad2a18bf9c495dd7414"
dependencies = [
"gethostname",
"rustix",
"x11rb-protocol",
]
[[package]]
name = "x11rb-protocol"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ea6fc2961e4ef194dcbfe56bb845534d0dc8098940c7e5c012a258bfec6701bd"
[[package]]
name = "x509-parser"
version = "0.16.0"
+1 -1
View File
@@ -48,7 +48,7 @@ exclude = [
ndk = { path = "clients/android/native/vendor/ndk" }
[workspace.package]
version = "0.19.0"
version = "0.17.2"
edition = "2021"
rust-version = "1.82"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -10,7 +10,7 @@
"name": "MIT OR Apache-2.0",
"identifier": "MIT OR Apache-2.0"
},
"version": "0.18.0"
"version": "0.17.2"
},
"paths": {
"/api/v1/clients": {
@@ -1,107 +0,0 @@
package io.unom.punktfunk
import android.content.ClipData
import android.content.ClipboardManager
import android.content.Context
import android.os.Handler
import android.os.Looper
import io.unom.punktfunk.kit.NativeBridge
/**
* Text clipboard sync for the active session (the desktop-client model, text-only v1):
* * **Device → host**: a local copy (the primary-clip listener, plus one probe at start) is
* announced as a lazy offer — the text crosses only when the host actually pastes (a
* `fetch:` event, answered with the clipboard's current content).
* * **Host → device**: a host copy arrives as an `offer:` event and is fetched eagerly into
* the system clipboard (Android apps can't lazily materialize a paste from the network
* without a content-provider round-trip that isn't worth it here).
*
* Loop guard: text set from a host fetch is remembered ([lastFromHost]) so the resulting
* primary-clip-changed callback doesn't bounce it straight back as a new offer. Clipboard reads
* happen while the stream is foreground (Android only allows focused-app reads). The native
* events are drained on a dedicated thread and applied on the main thread; [stop] joins it.
*/
class ClipboardSync(
private val context: Context,
private val handle: Long,
) {
private val main = Handler(Looper.getMainLooper())
private val cm = context.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager
@Volatile private var running = true
private var seq = 0
private var lastOffered: String? = null
private var lastFromHost: String? = null
private var pendingFetch = -1
private var thread: Thread? = null
private val clipListener = ClipboardManager.OnPrimaryClipChangedListener { offerLocal() }
fun start() {
NativeBridge.nativeClipControl(handle, true)
cm.addPrimaryClipChangedListener(clipListener)
thread = Thread({ pollLoop() }, "pf-clipboard").also { it.start() }
offerLocal() // whatever is already on the clipboard is pasteable host-side right away
}
fun stop() {
running = false
cm.removePrimaryClipChangedListener(clipListener)
thread?.join(600) // one poll timeout (250 ms) + slack
thread = null
}
/** Announce the current local text (if it's new and not an echo of a host copy). */
private fun offerLocal() {
if (!running) return
val text = currentClipText() ?: return
if (text == lastOffered || text == lastFromHost) return
lastOffered = text
seq += 1
NativeBridge.nativeClipOfferText(handle, seq)
}
private fun currentClipText(): String? = runCatching {
cm.primaryClip?.takeIf { it.itemCount > 0 }?.getItemAt(0)
?.coerceToText(context)?.toString()?.takeIf { it.isNotEmpty() }
}.getOrNull()
private fun pollLoop() {
while (running) {
val ev = NativeBridge.nativeNextClip(handle) ?: continue
if (ev == "closed") return
main.post { handleEvent(ev) }
}
}
private fun handleEvent(ev: String) {
if (!running) return
val parts = ev.split(":", limit = 3)
when (parts[0]) {
"offer" -> {
val offerSeq = parts.getOrNull(1)?.toIntOrNull() ?: return
if (parts.getOrNull(2) == "1") {
pendingFetch = NativeBridge.nativeClipFetchText(handle, offerSeq)
}
}
"fetch" -> {
val req = parts.getOrNull(1)?.toIntOrNull() ?: return
val text = currentClipText()
if (text != null) {
NativeBridge.nativeClipServeText(handle, req, text)
} else {
NativeBridge.nativeClipCancel(handle, req)
}
}
"data" -> {
val xfer = parts.getOrNull(1)?.toIntOrNull() ?: return
if (xfer != pendingFetch) return // stale/unknown transfer
pendingFetch = -1
val text = parts.getOrNull(2)?.takeIf { it.isNotEmpty() } ?: return
lastFromHost = text
runCatching { cm.setPrimaryClip(ClipData.newPlainText("Punktfunk", text)) }
}
// "state"/"cancel"/"error": nothing to drive in the text-only v1.
}
}
}
@@ -54,21 +54,6 @@ class MainActivity : ComponentActivity() {
var padKeyProbe: ((KeyEvent) -> Boolean)? = null
var padMotionProbe: ((MotionEvent) -> Boolean)? = null
/**
* Physical-mouse forwarder for the active session (built/released by StreamScreen, like
* [gamepadRouter]): uncaptured hover/click/wheel forwards as absolute cursor input, captured
* ([android.view.View.requestPointerCapture]) raw deltas as relative mouse-look. The dispatch
* overrides below route every SOURCE_MOUSE event here while streaming. Null while not streaming.
*/
var mouseForwarder: MouseForwarder? = null
/**
* TV remote-as-pointer for the active session (StreamScreen builds it on TV devices only):
* hold SELECT to toggle, then the D-pad glides the host cursor. Consulted first for
* non-gamepad keys while streaming. Null while not streaming or not a TV.
*/
var remotePointer: RemotePointer? = null
/**
* Set by [StreamScreen] to its disconnect action. The emergency-exit chord (below) invokes it so a
* couch user with no keyboard/Back can always leave a stream.
@@ -339,37 +324,9 @@ class MainActivity : ComponentActivity() {
return true // consumed
}
}
// TV remote-as-pointer sees non-gamepad keys first (SELECT long-press toggles it;
// while active it owns the D-pad/SELECT/PLAY-PAUSE/BACK).
if (!event.isFromSource(InputDevice.SOURCE_GAMEPAD)) {
remotePointer?.let { if (it.onKey(event)) return true }
}
// Ctrl+Alt+Shift+Q — the cross-client pointer-capture toggle chord. Swallow both
// edges of the Q (the modifiers already went over the wire, exactly like desktop).
if (event.keyCode == KeyEvent.KEYCODE_Q &&
event.isCtrlPressed && event.isAltPressed && event.isShiftPressed
) {
if (event.action == KeyEvent.ACTION_DOWN && event.repeatCount == 0) {
mouseForwarder?.toggleCapture()
}
return true
}
when (event.keyCode) {
// A mouse's back/forward buttons already go over the wire as X1/X2 via their
// BUTTON_* motion edges — but Android ALSO delivers them as key events: the input
// reader synthesizes KEYCODE_BACK/FORWARD (stamped SOURCE_MOUSE) unconditionally,
// and a view-level FALLBACK BACK appears when the BUTTON_* press goes unconsumed.
// Swallow every such duplicate or it doubles as Android navigation and yanks the
// user out of the stream. A remote/keyboard BACK is never mouse-sourced, so it
// still falls through to the BackHandler and exits.
KeyEvent.KEYCODE_BACK, KeyEvent.KEYCODE_FORWARD ->
if (event.isFromSource(InputDevice.SOURCE_MOUSE) ||
event.flags and KeyEvent.FLAG_FALLBACK != 0
) {
return true
}
// Leave these to the system even while streaming.
// (BACK above → BackHandler leaves the stream.)
KeyEvent.KEYCODE_BACK, // → BackHandler leaves the stream
KeyEvent.KEYCODE_VOLUME_UP,
KeyEvent.KEYCODE_VOLUME_DOWN,
KeyEvent.KEYCODE_VOLUME_MUTE,
@@ -437,10 +394,6 @@ class MainActivity : ComponentActivity() {
override fun dispatchGenericMotionEvent(event: MotionEvent): Boolean {
if (streamHandle != 0L) {
if (gamepadRouter?.onMotion(event) == true) return true
// Physical mouse (uncaptured): hover motion, wheel, button edges.
if (event.isFromSource(InputDevice.SOURCE_MOUSE)) {
mouseForwarder?.let { if (it.onGenericMotion(event)) return true }
}
return super.dispatchGenericMotionEvent(event)
}
// The Controllers debug screen sees pad motion before the stick→D-pad synthesis below.
@@ -478,24 +431,6 @@ class MainActivity : ComponentActivity() {
return super.dispatchGenericMotionEvent(event)
}
/**
* Mouse clicks/drags ride the TOUCH stream (the pointer is "down"). While streaming they
* belong to the mouse forwarder, never to the Compose touch-gesture layer — a physical
* mouse click must be a real click at the cursor, not a synthesized trackpad tap.
*/
override fun dispatchTouchEvent(ev: MotionEvent): Boolean {
if (streamHandle != 0L && ev.isFromSource(InputDevice.SOURCE_MOUSE)) {
mouseForwarder?.let { if (it.onTouchEvent(ev)) return true }
}
return super.dispatchTouchEvent(ev)
}
/** The OS is the source of truth for pointer capture (it releases on focus loss). */
override fun onPointerCaptureChanged(hasCapture: Boolean) {
super.onPointerCaptureChanged(hasCapture)
mouseForwarder?.onCaptureChanged(hasCapture)
}
/** Keys that drive the console UI — D-pad + face buttons; used to classify the last input source. */
private fun isConsoleNavKey(kc: Int): Boolean = when (kc) {
KeyEvent.KEYCODE_DPAD_UP, KeyEvent.KEYCODE_DPAD_DOWN, KeyEvent.KEYCODE_DPAD_LEFT,
@@ -1,206 +0,0 @@
package io.unom.punktfunk
import android.view.InputDevice
import android.view.MotionEvent
import io.unom.punktfunk.kit.NativeBridge
import kotlin.math.roundToInt
/** True when any connected input device is a pointer (USB/BT mouse, or a touchpad driving one). */
fun hasPhysicalMouse(): Boolean = InputDevice.getDeviceIds().any { id ->
InputDevice.getDevice(id)?.supportsSource(InputDevice.SOURCE_MOUSE) == true
}
/**
* Physical mouse → wire, in two modes (the iPadOS/desktop model):
* * **uncaptured** (default): hover/drag positions forward as absolute cursor moves
* (`MouseMoveAbs`, host-normalized against the window size) — desktop-style pointing. The
* local cursor is hidden over the stream (StreamScreen sets a TYPE_NULL pointer icon); the
* host's own cursor, composited into the video, is the one you see.
* * **captured**: the OS pointer is grabbed ([android.view.View.requestPointerCapture]) and raw
* relative deltas forward as `MouseMove` — FPS mouse-look. Engaged at stream start / by
* clicking into the stream when the "Capture pointer for games" setting is on, and toggled
* any time by Ctrl+Alt+Shift+Q (the cross-client chord). Focus loss releases it (the OS
* guarantees that); a click re-engages.
*
* Buttons ride [MotionEvent.ACTION_BUTTON_PRESS]/RELEASE edges (left/middle/right/back/forward →
* wire 1/2/3/4/5), the wheel rides [MotionEvent.ACTION_SCROLL] with fractional accumulation so
* high-resolution wheels don't lose sub-notch travel. Held buttons are tracked and flushed on
* capture loss / stream exit so nothing sticks on the host. Events reach this class from
* MainActivity's dispatch overrides (uncaptured) and the capture view's captured-pointer listener.
*/
class MouseForwarder(
private val handle: Long,
private val invertScroll: Boolean,
private val captureWanted: Boolean,
private val surfaceSize: () -> Pair<Int, Int>,
) {
/** Capture plumbing, owned by StreamScreen (the focusable capture view). */
var onRequestCapture: (() -> Unit)? = null
var onReleaseCapture: (() -> Unit)? = null
/** Live capture state, updated from [android.app.Activity.onPointerCaptureChanged]. */
var captured = false
private set
/** Chord-released: no auto re-engage (start / click) until the user opts back in. */
private var userReleased = false
private val heldButtons = mutableSetOf<Int>()
private var scrollAccV = 0f
private var scrollAccH = 0f
private var moveAccX = 0f
private var moveAccY = 0f
/** Uncaptured mouse events on the TOUCH stream (position while a button is down). */
fun onTouchEvent(ev: MotionEvent): Boolean {
when (ev.actionMasked) {
MotionEvent.ACTION_DOWN -> {
if (captureWanted && !captured && !userReleased) {
// The engaging click: grab the pointer and swallow the click (desktop
// parity — the click that captures never reaches the host). The paired
// BUTTON_RELEASE is dropped by the held-set guard in [button].
onRequestCapture?.invoke()
return true
}
sendAbs(ev)
}
MotionEvent.ACTION_MOVE -> sendAbs(ev)
// Button edges are documented on the generic stream, but be robust to either.
MotionEvent.ACTION_BUTTON_PRESS -> button(ev.actionButton, true)
MotionEvent.ACTION_BUTTON_RELEASE -> button(ev.actionButton, false)
}
return true
}
/** Uncaptured mouse events on the GENERIC stream (hover motion, wheel, button edges). */
fun onGenericMotion(ev: MotionEvent): Boolean {
when (ev.actionMasked) {
MotionEvent.ACTION_HOVER_MOVE -> sendAbs(ev)
MotionEvent.ACTION_SCROLL -> wheel(ev)
MotionEvent.ACTION_BUTTON_PRESS -> button(ev.actionButton, true)
MotionEvent.ACTION_BUTTON_RELEASE -> button(ev.actionButton, false)
MotionEvent.ACTION_HOVER_ENTER, MotionEvent.ACTION_HOVER_EXIT -> {}
else -> return false
}
return true
}
/**
* Captured-pointer events (the view holds [android.view.View.requestPointerCapture]): x/y ARE
* the relative deltas ([InputDevice.SOURCE_MOUSE_RELATIVE]), batched samples included. A
* captured touchpad reports absolute finger coordinates instead — not handled (the touch
* gesture layer is the touchpad story); returning false leaves those to the framework.
*/
fun onCapturedPointer(ev: MotionEvent): Boolean {
if (!ev.isFromSource(InputDevice.SOURCE_MOUSE_RELATIVE)) return false
when (ev.actionMasked) {
MotionEvent.ACTION_MOVE -> {
var dx = 0f
var dy = 0f
for (i in 0 until ev.historySize) {
dx += ev.getHistoricalX(i)
dy += ev.getHistoricalY(i)
}
dx += ev.x
dy += ev.y
moveAccX += dx
moveAccY += dy
val ox = moveAccX.toInt() // truncate toward zero — sub-pixel remainder kept w/ sign
val oy = moveAccY.toInt()
if (ox != 0 || oy != 0) {
NativeBridge.nativeSendPointerMove(handle, ox, oy)
moveAccX -= ox
moveAccY -= oy
}
}
MotionEvent.ACTION_BUTTON_PRESS -> button(ev.actionButton, true)
MotionEvent.ACTION_BUTTON_RELEASE -> button(ev.actionButton, false)
MotionEvent.ACTION_SCROLL -> wheel(ev)
}
return true
}
/** Ctrl+Alt+Shift+Q: release the grab, or (re-)engage it — works even when auto-capture is off. */
fun toggleCapture() {
if (captured) {
userReleased = true
onReleaseCapture?.invoke()
} else {
userReleased = false
onRequestCapture?.invoke()
}
}
/** Auto-engage at stream start (setting on + a mouse actually present). */
fun engageFromStart() {
if (captureWanted && !captured && !userReleased && hasPhysicalMouse()) {
onRequestCapture?.invoke()
}
}
/** From [android.app.Activity.onPointerCaptureChanged] — the OS is the source of truth. */
fun onCaptureChanged(has: Boolean) {
captured = has
// Losing the grab (focus loss, chord) must not leave buttons held on the host.
if (!has) flushButtons()
}
/** Stream teardown: lift anything held and let the grab go. */
fun release() {
flushButtons()
if (captured) onReleaseCapture?.invoke()
}
private fun sendAbs(ev: MotionEvent) {
val (w, h) = surfaceSize()
if (w <= 0 || h <= 0) return
NativeBridge.nativeSendPointerAbs(
handle,
ev.x.roundToInt().coerceIn(0, w - 1),
ev.y.roundToInt().coerceIn(0, h - 1),
w,
h,
)
}
private fun wheel(ev: MotionEvent) {
val dir = if (invertScroll) -1f else 1f
// Android: AXIS_VSCROLL + = up/away, AXIS_HSCROLL + = right — the wire's convention too.
scrollAccV += ev.getAxisValue(MotionEvent.AXIS_VSCROLL) * 120f * dir
scrollAccH += ev.getAxisValue(MotionEvent.AXIS_HSCROLL) * 120f * dir
val v = scrollAccV.toInt()
if (v != 0) {
NativeBridge.nativeSendScroll(handle, 0, v)
scrollAccV -= v
}
val h = scrollAccH.toInt()
if (h != 0) {
NativeBridge.nativeSendScroll(handle, 1, h)
scrollAccH -= h
}
}
private fun button(actionButton: Int, down: Boolean) {
val b = when (actionButton) {
MotionEvent.BUTTON_PRIMARY -> 1
MotionEvent.BUTTON_TERTIARY -> 2
MotionEvent.BUTTON_SECONDARY -> 3
MotionEvent.BUTTON_BACK -> 4
MotionEvent.BUTTON_FORWARD -> 5
else -> return
}
if (down) {
heldButtons.add(b)
NativeBridge.nativeSendPointerButton(handle, b, true)
} else if (heldButtons.remove(b)) {
// Only release what we pressed — drops the release of a swallowed engaging click
// and anything that raced a capture transition.
NativeBridge.nativeSendPointerButton(handle, b, false)
}
}
private fun flushButtons() {
heldButtons.forEach { NativeBridge.nativeSendPointerButton(handle, it, false) }
heldButtons.clear()
}
}
@@ -1,193 +0,0 @@
package io.unom.punktfunk
import android.os.Handler
import android.os.Looper
import android.view.Choreographer
import android.view.KeyEvent
import io.unom.punktfunk.kit.NativeBridge
import kotlin.math.hypot
// Hold this long on SELECT (pointer-mode toggle) / PLAY-PAUSE (keyboard toggle) for the long-press
// action instead of the tap action.
private const val LONG_PRESS_MS = 800L
// D-pad glide ballistics, in screen-widths per second: start slow enough to hit a close button,
// ramp over RAMP_S seconds of continuous hold so crossing the desktop doesn't take all day.
private const val SPEED_MIN = 0.14f
private const val SPEED_MAX = 0.70f
private const val RAMP_S = 1.2f
/**
* Android TV remote as a pointer — the Android analogue of the Apple client's Siri-remote pointer,
* adapted for D-pad-only remotes (most Android TV remotes have no touch surface). For the
* "TV as a desktop client" use case, where a plain remote is often the only thing in hand.
*
* While streaming on a TV, **hold SELECT ≈ 0.8 s** to toggle pointer mode. While active:
* * D-pad (held) glides the host cursor with ramping acceleration (relative `MouseMove`,
* Choreographer-paced, diagonal-normalized);
* * SELECT tap = left click; PLAY/PAUSE tap = right click (Siri-remote parity);
* * PLAY/PAUSE held = toggle the on-screen keyboard; BACK = leave pointer mode
* (a second BACK then leaves the stream as usual).
* While inactive, everything except the SELECT long-press passes through untouched (D-pad =
* arrow keys, SELECT tap = Enter — synthesized on release, since the down was held back to
* disambiguate the long-press).
*
* Only consulted for non-gamepad key events on TV devices (MainActivity gates the calls); all
* state lives on the main thread.
*/
class RemotePointer(
private val handle: Long,
private val surfaceWidth: () -> Int,
private val onActiveChanged: (Boolean) -> Unit,
private val onKeyboardToggle: () -> Unit,
) {
var active = false
private set
private val handler = Handler(Looper.getMainLooper())
private val held = mutableSetOf<Int>() // D-pad keycodes currently down
private var moveAccX = 0f
private var moveAccY = 0f
private var lastFrameNs = 0L
private var rampSec = 0f
private var tickerRunning = false
private var centerLongFired = false
private var playLongFired = false
private val centerLong = Runnable {
centerLongFired = true
toggle()
}
private val playLong = Runnable {
playLongFired = true
onKeyboardToggle()
}
private val frame = object : Choreographer.FrameCallback {
override fun doFrame(nowNs: Long) {
if (!tickerRunning) return
if (held.isEmpty() || !active) {
tickerRunning = false
return
}
val dt = if (lastFrameNs == 0L) {
1f / 60f
} else {
((nowNs - lastFrameNs) / 1e9f).coerceIn(0.001f, 0.1f)
}
lastFrameNs = nowNs
rampSec += dt
var vx = 0f
var vy = 0f
if (KeyEvent.KEYCODE_DPAD_LEFT in held) vx -= 1f
if (KeyEvent.KEYCODE_DPAD_RIGHT in held) vx += 1f
if (KeyEvent.KEYCODE_DPAD_UP in held) vy -= 1f
if (KeyEvent.KEYCODE_DPAD_DOWN in held) vy += 1f
val mag = hypot(vx, vy)
if (mag > 0f) {
val w = surfaceWidth().coerceAtLeast(640)
val speed = w * (SPEED_MIN + (SPEED_MAX - SPEED_MIN) * (rampSec / RAMP_S).coerceAtMost(1f))
moveAccX += vx / mag * speed * dt
moveAccY += vy / mag * speed * dt
val ox = moveAccX.toInt() // truncate toward zero — sub-pixel remainder kept
val oy = moveAccY.toInt()
if (ox != 0 || oy != 0) {
NativeBridge.nativeSendPointerMove(handle, ox, oy)
moveAccX -= ox
moveAccY -= oy
}
}
Choreographer.getInstance().postFrameCallback(this)
}
}
/** One remote key event; true = consumed. Ignore key repeats — the ticker owns motion. */
fun onKey(event: KeyEvent): Boolean {
val down = event.action == KeyEvent.ACTION_DOWN
when (event.keyCode) {
KeyEvent.KEYCODE_DPAD_CENTER -> {
if (down) {
if (event.repeatCount == 0) {
centerLongFired = false
handler.postDelayed(centerLong, LONG_PRESS_MS)
}
} else {
handler.removeCallbacks(centerLong)
if (!centerLongFired) {
if (active) {
click(1)
} else {
// The down was held back to disambiguate the long-press, so the
// normal path never saw it — synthesize the Enter here instead.
NativeBridge.nativeSendKey(handle, 0x0D, true, 0)
NativeBridge.nativeSendKey(handle, 0x0D, false, 0)
}
}
}
return true
}
KeyEvent.KEYCODE_DPAD_UP, KeyEvent.KEYCODE_DPAD_DOWN,
KeyEvent.KEYCODE_DPAD_LEFT, KeyEvent.KEYCODE_DPAD_RIGHT,
-> {
if (!active) return false
if (down) {
if (held.add(event.keyCode) && held.size == 1) startTicker()
} else {
held.remove(event.keyCode)
}
return true
}
KeyEvent.KEYCODE_MEDIA_PLAY_PAUSE -> {
if (!active) return false // inactive: the media-key VK path owns it
if (down) {
if (event.repeatCount == 0) {
playLongFired = false
handler.postDelayed(playLong, LONG_PRESS_MS)
}
} else {
handler.removeCallbacks(playLong)
if (!playLongFired) click(3)
}
return true
}
KeyEvent.KEYCODE_BACK -> {
if (!active) return false
if (!down) toggle() // leave pointer mode; the next BACK leaves the stream
return true
}
else -> return false
}
}
/** Stream teardown: stop timers/ticker; nothing wire-held to flush (clicks are edges). */
fun release() {
handler.removeCallbacks(centerLong)
handler.removeCallbacks(playLong)
active = false
held.clear()
tickerRunning = false
}
private fun toggle() {
active = !active
if (!active) {
held.clear()
tickerRunning = false
}
onActiveChanged(active)
}
private fun startTicker() {
rampSec = 0f
lastFrameNs = 0L
if (!tickerRunning) {
tickerRunning = true
Choreographer.getInstance().postFrameCallback(frame)
}
}
private fun click(button: Int) {
NativeBridge.nativeSendPointerButton(handle, button, true)
NativeBridge.nativeSendPointerButton(handle, button, false)
}
}
@@ -109,27 +109,6 @@ data class Settings(
* setup where the OS-level pad (lizard mode) is preferred.
*/
val sc2Capture: Boolean = true,
/**
* Lock a physical mouse to the stream ([android.view.View.requestPointerCapture]) and forward
* raw relative motion — FPS mouse-look, the iPad "Capture pointer for games" twin. Engages at
* stream start and on a click into the stream; Ctrl+Alt+Shift+Q toggles it live (the chord
* works even with this off). Off (default): a mouse points absolutely, desktop-style.
*/
val pointerCapture: Boolean = false,
/**
* Flip scroll direction — the mouse wheel and the two-finger touch scroll both. Parity with
* the Apple/GTK clients' "Invert scroll direction".
*/
val invertScroll: Boolean = false,
/**
* Sync text copied on this device to the host and vice versa while streaming (the desktop
* clients' shared clipboard, text-only here). Only effective when the host advertises the
* clipboard capability; the protocol is opt-in per session either way.
*/
val clipboardSync: Boolean = true,
)
/** [Settings.touchMode] values; persisted by name. */
@@ -193,9 +172,6 @@ class SettingsStore(context: Context) {
autoWakeEnabled = prefs.getBoolean(K_AUTO_WAKE, true),
rumbleOnPhone = prefs.getBoolean(K_RUMBLE_ON_PHONE, false),
sc2Capture = prefs.getBoolean(K_SC2_CAPTURE, true),
pointerCapture = prefs.getBoolean(K_POINTER_CAPTURE, false),
invertScroll = prefs.getBoolean(K_INVERT_SCROLL, false),
clipboardSync = prefs.getBoolean(K_CLIPBOARD_SYNC, true),
)
fun save(s: Settings) {
@@ -219,9 +195,6 @@ class SettingsStore(context: Context) {
.putBoolean(K_AUTO_WAKE, s.autoWakeEnabled)
.putBoolean(K_RUMBLE_ON_PHONE, s.rumbleOnPhone)
.putBoolean(K_SC2_CAPTURE, s.sc2Capture)
.putBoolean(K_POINTER_CAPTURE, s.pointerCapture)
.putBoolean(K_INVERT_SCROLL, s.invertScroll)
.putBoolean(K_CLIPBOARD_SYNC, s.clipboardSync)
.apply()
}
@@ -260,9 +233,6 @@ class SettingsStore(context: Context) {
const val K_AUTO_WAKE = "auto_wake_enabled"
const val K_RUMBLE_ON_PHONE = "rumble_on_phone"
const val K_SC2_CAPTURE = "sc2_capture"
const val K_POINTER_CAPTURE = "pointer_capture"
const val K_INVERT_SCROLL = "invert_scroll"
const val K_CLIPBOARD_SYNC = "clipboard_sync"
/** Legacy Boolean the enum replaced — read once as the migration default, never written. */
const val K_TRACKPAD = "trackpad_mode"
@@ -412,27 +412,6 @@ private fun ControlsSettings(s: Settings, update: (Settings) -> Unit, onOpenCont
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
ToggleRow(
title = "Capture pointer for games",
subtitle = "Lock a connected mouse to the stream and send raw relative motion " +
"(mouse-look). Ctrl+Alt+Shift+Q toggles it live; click the stream to re-capture. " +
"Off: the mouse points at the desktop directly",
checked = s.pointerCapture,
onCheckedChange = { on -> update(s.copy(pointerCapture = on)) },
)
ToggleRow(
title = "Invert scroll direction",
subtitle = "Flip the mouse wheel and two-finger touch scrolling",
checked = s.invertScroll,
onCheckedChange = { on -> update(s.copy(invertScroll = on)) },
)
ToggleRow(
title = "Shared clipboard",
subtitle = "Text copied here pastes on the host and vice versa (hosts with " +
"clipboard sharing enabled)",
checked = s.clipboardSync,
onCheckedChange = { on -> update(s.copy(clipboardSync = on)) },
)
}
SettingsCard {
SettingDropdown(
@@ -13,7 +13,6 @@ import android.net.wifi.WifiManager
import android.os.Build
import android.text.InputType
import android.util.Log
import android.view.KeyEvent
import android.view.SurfaceHolder
import android.view.SurfaceView
import android.view.View
@@ -177,14 +176,6 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
// "hold to quit" hint overlay. Set from the router's onExitArmed (main thread).
var exitArming by remember { mutableStateOf(false) }
// True while the TV remote is acting as a pointer (hold SELECT toggles) — drives the mode hint.
var remotePointerOn by remember { mutableStateOf(false) }
// Focus anchor the soft keyboard is summoned onto AND the pointer-capture grab target (a grab
// needs a focusable view; captured-pointer events land on it). Declared before the effect
// below so the capture callbacks can reach the view once it exists.
var keyCapture by remember { mutableStateOf<KeyCaptureView?>(null) }
DisposableEffect(handle) {
window?.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
wifiLocks.forEach { lock ->
@@ -230,54 +221,6 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
// Show a "hold to quit" hint the moment the chord completes (the router debounces the actual
// exit); it clears when the buttons release early or the hold elapses. Runs on the main thread.
router.onExitArmed = { armed -> exitArming = armed }
// Physical mouse: uncaptured hover/click/wheel forwards as absolute pointing; captured
// (setting or the Ctrl+Alt+Shift+Q chord) raw deltas forward as relative mouse-look.
// The local cursor is hidden over the stream — the host's own cursor, composited into
// the video, is the one the user sees (twin of the desktop clients' hidden cursor).
val decor = window?.decorView
val priorPointerIcon = decor?.pointerIcon
decor?.pointerIcon = android.view.PointerIcon.getSystemIcon(
context,
android.view.PointerIcon.TYPE_NULL,
)
val mouse = MouseForwarder(
handle,
invertScroll = initialSettings.invertScroll,
captureWanted = initialSettings.pointerCapture,
surfaceSize = { (decor?.width ?: 0) to (decor?.height ?: 0) },
)
mouse.onRequestCapture = {
// The grab needs the (focusable) capture view: focus it, then ask. Posted so a
// request racing view attach/focus settles on the next frame.
keyCapture?.let { v ->
v.post {
v.requestFocus()
v.requestPointerCapture()
}
}
}
mouse.onReleaseCapture = { keyCapture?.releasePointerCapture() }
activity?.mouseForwarder = mouse
// TV remote-as-pointer: hold SELECT ≈ 0.8 s to toggle; the D-pad then glides the host
// cursor (see RemotePointer). TV only — a phone's remote-less keys stay on the VK path.
val remote = if (isTv) {
RemotePointer(
handle,
surfaceWidth = { decor?.width ?: 1920 },
onActiveChanged = { on -> remotePointerOn = on },
onKeyboardToggle = { keyCapture?.let { it.setImeVisible(!it.imeShown) } },
)
} else {
null
}
activity?.remotePointer = remote
// Shared clipboard (text v1): only when the user setting is on AND the host has a
// working clipboard service. Protocol-level opt-in + the poll thread live in the sync.
val clip = if (initialSettings.clipboardSync && NativeBridge.nativeClipSupported(handle)) {
ClipboardSync(context, handle).also { it.start() }
} else {
null
}
activity?.setConsoleHighRefreshRate(false) // let the decoder's setFrameRate pick the panel rate
// Host→client feedback (rumble + DualSense lightbar/LEDs), routed to each controller by pad
// index via the router; poll threads stopped + joined before the router is released and the
@@ -343,7 +286,6 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
}
onDispose {
closed.set(true) // from here the handle gets freed; surfaceDestroyed must not touch it
clip?.stop() // stop + join the clipboard poll thread BEFORE the handle is freed
feedback.onHidRaw = null
feedback.stop() // stop + join the poll threads BEFORE the router is released / handle freed
sc2UsbReceiver?.let { runCatching { context.unregisterReceiver(it) } }
@@ -351,12 +293,6 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
router.onExitArmed = null // don't poke Compose state from release()'s disarm while tearing down
router.release() // flush every slot (nothing sticks host-side) + drop the hot-plug listener
activity?.gamepadRouter = null
// Mouse/remote-pointer teardown: lift held buttons, drop the grab, restore the cursor.
mouse.release()
activity?.mouseForwarder = null
remote?.release()
activity?.remotePointer = null
decor?.pointerIcon = priorPointerIcon
activity?.streamHandle = 0L
activity?.requestStreamExit = null
// Back in the menus: the SC2 (if present) resumes driving the console UI.
@@ -384,12 +320,8 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
// Back gesture = a deliberate exit → signal the quit so the host tears down now (no linger).
BackHandler { NativeBridge.nativeDisconnectQuit(handle); onDisconnect() }
// Auto-engage pointer capture at stream start (setting on + a mouse actually present).
// Delayed a beat: the grab needs window focus and the capture view attached.
LaunchedEffect(handle) {
delay(400)
activity?.mouseForwarder?.engageFromStart()
}
// Focus anchor the three-finger keyboard swipe summons the IME onto (see KeyCaptureView).
var keyCapture by remember { mutableStateOf<KeyCaptureView?>(null) }
Box(modifier = Modifier.fillMaxSize()) {
AndroidView(
@@ -447,49 +379,23 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
if (exitArming) {
ExitChordHint(Modifier.align(Alignment.TopCenter).padding(top = 16.dp))
}
// Remote-pointer mode hint — the remote's keys are remapped while it's on, so say so.
if (remotePointerOn) {
RemotePointerHint(Modifier.align(Alignment.TopCenter).padding(top = 16.dp))
}
// Invisible 1-px focus anchor for the host-typing soft keyboard (three-finger swipe up
// in the mouse modes) AND the pointer-capture grab target — it never draws or takes
// touches, it just owns IME focus and receives captured-pointer events.
// Invisible 1-px focus anchor for the host-typing soft keyboard (three-finger swipe
// up in the mouse modes) — it never draws or takes touches, it just owns IME focus.
AndroidView(
modifier = Modifier.size(1.dp),
factory = { ctx ->
KeyCaptureView(ctx).also { v ->
keyCapture = v
// Real IME text path when the host types committed text (see KeyCaptureView).
v.textHandle =
if (NativeBridge.nativeTextInputSupported(handle)) handle else 0L
v.setOnCapturedPointerListener { _, ev ->
(ctx as? MainActivity)?.mouseForwarder?.onCapturedPointer(ev) ?: false
}
}
},
factory = { ctx -> KeyCaptureView(ctx).also { keyCapture = it } },
)
// Touch input per the Settings model: trackpad/direct-pointer mouse (the shared gesture
// vocabulary) or real multi-touch passthrough — see TouchInput.kt. Passthrough gets no
// keyboard gesture: its fingers belong to the host verbatim (a swipe there may BE a
// host-OS gesture), so intercepting three fingers would corrupt real multi-touch.
// Stylus lane (design/pen-tablet-input.md §7): against a HOST_CAP_PEN host a stylus
// splits out of BOTH touch models onto the pen plane; its heartbeat coroutine keeps a
// stationary held stroke alive (and its cancellation lifts everything on teardown).
val stylus = remember(handle) {
if (NativeBridge.nativeHostSupportsPen(handle)) StylusStream(handle) else null
}
if (stylus != null) {
LaunchedEffect(stylus) { stylus.heartbeatLoop() }
}
Box(
Modifier.fillMaxSize().pointerInput(handle, touchMode) {
when (touchMode) {
TouchMode.TOUCH -> streamTouchPassthrough(handle, stylus)
TouchMode.TOUCH -> streamTouchPassthrough(handle)
else -> streamTouchInput(
handle,
stylus,
trackpad = touchMode == TouchMode.TRACKPAD,
invertScroll = initialSettings.invertScroll,
onCycleStats = { statsVerbosity = statsVerbosity.next() },
onKeyboard = { show -> keyCapture?.setImeVisible(show) },
)
@@ -517,35 +423,14 @@ private fun ExitChordHint(modifier: Modifier = Modifier) {
)
}
/**
* The remote-pointer mode cue: while active the remote's keys are remapped (D-pad glides the host
* cursor, SELECT clicks), so the overlay both confirms the toggle and teaches the vocabulary.
*/
@Composable
private fun RemotePointerHint(modifier: Modifier = Modifier) {
Text(
"Remote pointer — SELECT click · play/pause right-click · hold SELECT to exit",
modifier = modifier
.background(Color.Black.copy(alpha = 0.55f), RoundedCornerShape(8.dp))
.padding(horizontal = 14.dp, vertical = 8.dp),
color = Color.White,
fontSize = 15.sp,
)
}
/**
* Invisible focus anchor for typing on the host: the three-finger swipe summons the device IME
* onto this view. Two IME models, picked by the host's capabilities:
* * **Text path** ([textHandle] set — the host advertised `HOST_CAP_TEXT_INPUT`): a real
* editable [HostTextConnection], so the IME gives autocorrect, gesture typing, non-Latin
* composition and emoji, all mirrored to the host as committed text + diffs.
* * **Fallback** (older host): `TYPE_NULL` puts the IME in "dumb keyboard" mode — raw
* [KeyEvent]s flow through `MainActivity.dispatchKeyEvent` → `Keymap.toVk` → the host, the
* exact path a hardware keyboard takes (with the IME-shift wrap documented there).
*
* Doubles as the pointer-capture grab target: a grab needs a focusable view, and captured-pointer
* events are delivered to it (routed to [MouseForwarder.onCapturedPointer] via the listener the
* stream screen installs).
* onto this view. `TYPE_NULL` puts the IME in "dumb keyboard" mode — it delivers raw [KeyEvent]s
* (no composing text, no autocorrect), which flow through `MainActivity.dispatchKeyEvent` →
* `Keymap.toVk` → the host, the exact path a hardware keyboard takes. Text an IME insists on
* committing instead still arrives: the non-editable [BaseInputConnection] synthesizes KeyEvents
* for it via `KeyCharacterMap` (with Shift carried as meta state — see the IME-shift wrap in
* `MainActivity.dispatchKeyEvent`).
*/
private class KeyCaptureView(context: Context) : View(context) {
init {
@@ -553,171 +438,22 @@ private class KeyCaptureView(context: Context) : View(context) {
isFocusableInTouchMode = true
}
/** The session handle when the host types committed text; `0` = VK-only fallback. */
var textHandle: Long = 0L
override fun onCheckIsTextEditor(): Boolean = true
/** Whether [setImeVisible] last showed the IME — for toggle-style callers (remote pointer). */
var imeShown = false
private set
override fun onCheckIsTextEditor(): Boolean = imeShown
override fun onCreateInputConnection(outAttrs: EditorInfo): InputConnection? {
// Only an editor while the user has SUMMONED the keyboard (gesture / remote toggle).
// This view holds focus for the whole stream (it's the capture anchor), and with an
// always-live editable connection the IME counts input as active on it — TV IMEs then
// pop their UI the moment a PHYSICAL keyboard key arrives. With no connection, hardware
// typing stays on the raw dispatchKeyEvent → Keymap → wire path and no keyboard appears.
if (!imeShown) return null
outAttrs.imeOptions = EditorInfo.IME_FLAG_NO_EXTRACT_UI or
EditorInfo.IME_FLAG_NO_FULLSCREEN or EditorInfo.IME_FLAG_NO_ENTER_ACTION
return if (textHandle != 0L) {
outAttrs.inputType = InputType.TYPE_CLASS_TEXT or
InputType.TYPE_TEXT_FLAG_AUTO_CORRECT or InputType.TYPE_TEXT_FLAG_MULTI_LINE
HostTextConnection(this, textHandle)
} else {
outAttrs.inputType = InputType.TYPE_NULL
BaseInputConnection(this, false)
}
override fun onCreateInputConnection(outAttrs: EditorInfo): InputConnection {
outAttrs.inputType = InputType.TYPE_NULL
outAttrs.imeOptions = EditorInfo.IME_FLAG_NO_EXTRACT_UI or EditorInfo.IME_FLAG_NO_FULLSCREEN
return BaseInputConnection(this, false)
}
fun setImeVisible(show: Boolean) {
val imm = context.getSystemService(Context.INPUT_METHOD_SERVICE) as? InputMethodManager
?: return
imeShown = show
if (show) {
requestFocus()
// The view may already be focused from a null-connection state — restart so the
// framework re-queries onCreateInputConnection with the gate now open.
imm.restartInput(this)
imm.showSoftInput(this, 0)
} else {
imm.hideSoftInputFromWindow(windowToken, 0)
imm.restartInput(this) // gate closed — drop the editable connection
}
}
/**
* BACK while the summoned keyboard is up: the IME consumes it pre-IME to dismiss itself, so
* [setImeVisible] never hears about it — sync the gate here or a stale `imeShown` leaves the
* editable connection live and physical typing re-pops the keyboard.
*/
override fun onKeyPreIme(keyCode: Int, event: KeyEvent): Boolean {
if (keyCode == KeyEvent.KEYCODE_BACK && imeShown && event.action == KeyEvent.ACTION_UP) {
imeShown = false
(context.getSystemService(Context.INPUT_METHOD_SERVICE) as? InputMethodManager)
?.restartInput(this)
}
return super.onKeyPreIme(keyCode, event)
}
}
/**
* IME → host text bridge (the `HOST_CAP_TEXT_INPUT` path): a real **editable** connection, so
* the IME runs its full machinery (autocorrect, gesture typing, non-Latin composition), mirrored
* to the host as it happens. The one piece of host-side state tracked is *what the host currently
* shows of the active composition* ([sentComposition]): composing updates send a common-prefix
* diff (backspaces + the new suffix) so corrections materialize live on the host; a commit
* settles it. [setComposingRegion] adopts already-committed text as the active composition
* (autocorrect-revert / backspace-into-word flows), so the next update diffs against it instead
* of retyping. Newlines become Enter taps; [deleteSurroundingText] becomes Backspace/Delete taps.
*
* Known approximation: diff lengths are counted in Unicode scalars, assuming one host Backspace
* deletes one scalar — true for the composition text IMEs actually produce (emoji and other
* multi-unit graphemes commit directly rather than composing).
*/
private class HostTextConnection(
view: KeyCaptureView,
private val handle: Long,
) : BaseInputConnection(view, true) {
/** What the host currently shows of the active composition ("" = none). */
private var sentComposition = ""
override fun commitText(text: CharSequence, newCursorPosition: Int): Boolean {
retype(text.toString())
sentComposition = ""
val ok = super.commitText(text, newCursorPosition)
trimEditable()
return ok
}
override fun setComposingText(text: CharSequence, newCursorPosition: Int): Boolean {
retype(text.toString())
return super.setComposingText(text, newCursorPosition)
}
override fun finishComposingText(): Boolean {
// The composition text stands as committed — the host already shows it verbatim.
sentComposition = ""
return super.finishComposingText()
}
override fun setComposingRegion(start: Int, end: Int): Boolean {
val e = editable
if (e != null) {
val a = start.coerceIn(0, e.length)
val b = end.coerceIn(0, e.length)
sentComposition = e.subSequence(minOf(a, b), maxOf(a, b)).toString()
}
return super.setComposingRegion(start, end)
}
override fun deleteSurroundingText(beforeLength: Int, afterLength: Int): Boolean {
repeat(beforeLength.coerceIn(0, MAX_TAPS)) { tapVk(VK_BACK) }
repeat(afterLength.coerceIn(0, MAX_TAPS)) { tapVk(VK_DELETE) }
return super.deleteSurroundingText(beforeLength, afterLength)
}
override fun performEditorAction(actionCode: Int): Boolean {
tapVk(VK_RETURN)
return true
}
/** Replace the host's view of the composition with [text] via a common-prefix diff. */
private fun retype(text: String) {
var common = sentComposition.commonPrefixWith(text)
// Never split a surrogate pair mid-diff — back off to the pair boundary.
if (common.isNotEmpty() && common.last().isHighSurrogate()) {
common = common.dropLast(1)
}
val stale = sentComposition.substring(common.length)
repeat(stale.codePointCount(0, stale.length).coerceAtMost(MAX_TAPS)) { tapVk(VK_BACK) }
sendText(text.substring(common.length))
sentComposition = text
}
/** Forward literal text, turning newlines into Enter taps (control chars never ride text). */
private fun sendText(s: String) {
var chunk = StringBuilder()
for (ch in s) {
if (ch == '\n') {
if (chunk.isNotEmpty()) {
NativeBridge.nativeSendText(handle, chunk.toString())
chunk = StringBuilder()
}
tapVk(VK_RETURN)
} else {
chunk.append(ch)
}
}
if (chunk.isNotEmpty()) NativeBridge.nativeSendText(handle, chunk.toString())
}
private fun tapVk(vk: Int) {
NativeBridge.nativeSendKey(handle, vk, true, 0)
NativeBridge.nativeSendKey(handle, vk, false, 0)
}
/** Bound the mirror buffer: once nothing is composing, old text serves no purpose. */
private fun trimEditable() {
val e = editable ?: return
if (getComposingSpanStart(e) == -1 && e.length > 4000) e.clear()
}
private companion object {
const val VK_BACK = 0x08
const val VK_RETURN = 0x0D
const val VK_DELETE = 0x2E
const val MAX_TAPS = 256
}
}
@@ -1,195 +0,0 @@
package io.unom.punktfunk
import android.view.MotionEvent
import androidx.compose.ui.ExperimentalComposeUiApi
import androidx.compose.ui.input.pointer.PointerEvent
import androidx.compose.ui.input.pointer.PointerType
import androidx.compose.ui.unit.IntSize
import io.unom.punktfunk.kit.NativeBridge
import kotlinx.coroutines.delay
// Wire PEN_* state bits (punktfunk_core::quic::pen; mirrored, asserted by the Rust shim's docs).
private const val PEN_IN_RANGE = 1f
private const val PEN_TOUCHING = 2f
private const val PEN_BARREL1 = 4f
private const val PEN_BARREL2 = 8f
private const val STRIDE = 10
private const val MAX_SAMPLES = 8
/**
* Android stylus → the state-full pen plane (design/pen-tablet-input.md §7): pressure, tilt
* (`AXIS_TILT`, radians from the surface normal), azimuth (`AXIS_ORIENTATION` — Android's 0 =
* "pointed away from the user" IS the wire's north, no offset needed), hover with
* `AXIS_DISTANCE`, the eraser tool, both stylus barrel buttons, and historical (coalesced)
* samples batched oldest-first for full capture-rate fidelity. Android has no barrel-roll
* axis — roll stays unknown on this client.
*
* Both touch loops call [intercept] first; stylus/eraser pointers are consumed here (against a
* pen-capable host) and never reach the finger paths, independent of the touch-input mode.
* [heartbeatLoop] implements the ≤100 ms keepalive wire contract: a stationary held stylus is
* silent in Android's input pipeline, and the host force-releases a stroke after 200 ms
* without samples.
*/
internal class StylusStream(private val handle: Long) {
private var inRange = false
private var touching = false
private var sawHover = false
private val last = FloatArray(STRIDE)
private val batch = FloatArray(MAX_SAMPLES * STRIDE)
init {
idle(last)
}
/**
* Consume the event's stylus pointers into pen samples. Returns true when this event
* carried any (the caller's finger/gesture handling must then skip those changes).
*/
@OptIn(ExperimentalComposeUiApi::class)
fun intercept(ev: PointerEvent, size: IntSize): Boolean {
val stylusChanges = ev.changes.filter {
it.type == PointerType.Stylus || it.type == PointerType.Eraser
}
if (stylusChanges.isEmpty()) return false
stylusChanges.forEach { it.consume() }
val me = ev.motionEvent ?: return true
if (size.width <= 0 || size.height <= 0) return true
// At most one stylus exists — find its pointer index by tool type.
val idx = (0 until me.pointerCount).firstOrNull {
me.getToolType(it) == MotionEvent.TOOL_TYPE_STYLUS ||
me.getToolType(it) == MotionEvent.TOOL_TYPE_ERASER
} ?: return true
when (me.actionMasked) {
MotionEvent.ACTION_DOWN, MotionEvent.ACTION_POINTER_DOWN,
MotionEvent.ACTION_MOVE,
-> {
touching = true
inRange = true
emitSamples(me, idx, size)
}
MotionEvent.ACTION_HOVER_ENTER, MotionEvent.ACTION_HOVER_MOVE -> {
sawHover = true
inRange = true
touching = false
emitSamples(me, idx, size)
}
MotionEvent.ACTION_UP, MotionEvent.ACTION_POINTER_UP -> {
touching = false
// Hover-capable hardware keeps proximity (HOVER_EXIT owns the leave);
// anything else leaves range on lift — the host never parks a phantom pen.
inRange = sawHover
emitSamples(me, idx, size)
}
MotionEvent.ACTION_HOVER_EXIT, MotionEvent.ACTION_CANCEL -> release()
else -> {}
}
return true
}
/** Session/composition teardown: leave range so the host lifts anything still inked. */
fun reset() {
if (inRange || touching) release()
sawHover = false
}
/** The ≤100 ms keepalive (80 ms leaves headroom for one lost datagram). Runs until
* cancelled; resends the last state-full sample while the pen is in range. */
suspend fun heartbeatLoop() {
try {
while (true) {
delay(80)
if (inRange || touching) {
last[9] = 0f // dt
NativeBridge.nativeSendPen(handle, last, 1)
}
}
} finally {
reset()
}
}
private fun release() {
touching = false
inRange = false
last[0] = 0f // state: out of range
last[4] = 0f // pressure
NativeBridge.nativeSendPen(handle, last, 1)
}
/** Historical (coalesced) samples oldest-first, then the current one — a single batch. */
private fun emitSamples(me: MotionEvent, idx: Int, size: IntSize) {
val history = minOf(me.historySize, MAX_SAMPLES - 1)
var count = 0
var prevT = if (history > 0) me.getHistoricalEventTime(0) else me.eventTime
for (h in (me.historySize - history) until me.historySize) {
val t = me.getHistoricalEventTime(h)
fill(
batch, count * STRIDE, size,
x = me.getHistoricalX(idx, h), y = me.getHistoricalY(idx, h),
pressure = me.getHistoricalPressure(idx, h),
tiltRad = me.getHistoricalAxisValue(MotionEvent.AXIS_TILT, idx, h),
orientRad = me.getHistoricalAxisValue(MotionEvent.AXIS_ORIENTATION, idx, h),
distance = me.getHistoricalAxisValue(MotionEvent.AXIS_DISTANCE, idx, h),
buttons = me.buttonState, tool = me.getToolType(idx),
dtUs = ((t - prevT) * 1000).coerceIn(0, 65535).toFloat(),
)
prevT = t
count++
}
fill(
batch, count * STRIDE, size,
x = me.getX(idx), y = me.getY(idx), pressure = me.getPressure(idx),
tiltRad = me.getAxisValue(MotionEvent.AXIS_TILT, idx),
orientRad = me.getAxisValue(MotionEvent.AXIS_ORIENTATION, idx),
distance = me.getAxisValue(MotionEvent.AXIS_DISTANCE, idx),
buttons = me.buttonState, tool = me.getToolType(idx),
dtUs = ((me.eventTime - prevT) * 1000).coerceIn(0, 65535).toFloat(),
)
count++
batch.copyInto(last, 0, (count - 1) * STRIDE, count * STRIDE)
NativeBridge.nativeSendPen(handle, batch, count)
}
private fun fill(
out: FloatArray,
off: Int,
size: IntSize,
x: Float,
y: Float,
pressure: Float,
tiltRad: Float,
orientRad: Float,
distance: Float,
buttons: Int,
tool: Int,
dtUs: Float,
) {
var state = 0f
if (inRange || touching) state += PEN_IN_RANGE
if (touching) state += PEN_TOUCHING
if (buttons and MotionEvent.BUTTON_STYLUS_PRIMARY != 0) state += PEN_BARREL1
if (buttons and MotionEvent.BUTTON_STYLUS_SECONDARY != 0) state += PEN_BARREL2
out[off + 0] = state
out[off + 1] = if (tool == MotionEvent.TOOL_TYPE_ERASER) 1f else 0f
out[off + 2] = (x / (size.width - 1).coerceAtLeast(1)).coerceIn(0f, 1f)
out[off + 3] = (y / (size.height - 1).coerceAtLeast(1)).coerceIn(0f, 1f)
out[off + 4] = if (touching) pressure.coerceIn(0f, 1f) else 0f
// AXIS_DISTANCE units are device-arbitrary; 0..1 covers real hardware, and 0 while
// hovering legitimately means "at the hover floor".
out[off + 5] = if (touching) 0f else distance.coerceIn(0f, 1f)
out[off + 6] = Math.toDegrees(tiltRad.toDouble()).toFloat().coerceIn(0f, 90f)
// AXIS_ORIENTATION: 0 = pointed away from the user (= wire north), clockwise, −π..π.
out[off + 7] = ((Math.toDegrees(orientRad.toDouble()) + 360.0) % 360.0).toFloat()
out[off + 8] = -1f // no barrel-roll axis on Android
out[off + 9] = dtUs
}
private fun idle(out: FloatArray) {
out.fill(0f)
out[5] = -1f // distance unknown
out[6] = -1f // tilt unknown
out[7] = -1f // azimuth unknown
out[8] = -1f // roll unknown
}
}
@@ -1,11 +1,9 @@
package io.unom.punktfunk
import androidx.compose.foundation.gestures.awaitEachGesture
import androidx.compose.ui.input.pointer.AwaitPointerEventScope
import androidx.compose.foundation.gestures.awaitFirstDown
import androidx.compose.ui.input.pointer.PointerId
import androidx.compose.ui.input.pointer.PointerInputChange
import androidx.compose.ui.input.pointer.PointerInputScope
import androidx.compose.ui.input.pointer.PointerType
import androidx.compose.ui.input.pointer.changedToDownIgnoreConsumed
import androidx.compose.ui.input.pointer.changedToUpIgnoreConsumed
import androidx.compose.ui.input.pointer.positionChanged
@@ -58,26 +56,7 @@ private const val ACCEL_MAX = 3.0f
* normalizes and maps into the output). On teardown (stream leaves composition) every still-held
* contact is lifted so nothing stays stuck on the host.
*/
/** Whether this change belongs to the stylus lane (only when a pen-capable host is live). */
private fun isStylus(c: PointerInputChange, stylus: StylusStream?): Boolean =
stylus != null && (c.type == PointerType.Stylus || c.type == PointerType.Eraser)
/** [awaitFirstDown] with the stylus lane split out: pen events feed [stylus] and never start a
* mouse/touch gesture. Toward a pen-less host ([stylus] == null) a stylus stays a finger. */
private suspend fun AwaitPointerEventScope.awaitFirstFingerDown(
stylus: StylusStream?,
): PointerInputChange {
while (true) {
val ev = awaitPointerEvent()
stylus?.intercept(ev, size)
val down = ev.changes.firstOrNull {
it.changedToDownIgnoreConsumed() && !isStylus(it, stylus)
}
if (down != null) return down
}
}
internal suspend fun PointerInputScope.streamTouchPassthrough(handle: Long, stylus: StylusStream?) {
internal suspend fun PointerInputScope.streamTouchPassthrough(handle: Long) {
val ids = mutableMapOf<PointerId, Int>()
fun alloc(p: PointerId): Int {
var id = 0
@@ -89,12 +68,10 @@ internal suspend fun PointerInputScope.streamTouchPassthrough(handle: Long, styl
awaitPointerEventScope {
while (true) {
val ev = awaitPointerEvent()
stylus?.intercept(ev, size)
val sw = size.width
val sh = size.height
if (sw <= 0 || sh <= 0) continue
for (c in ev.changes) {
if (isStylus(c, stylus)) continue // the pen plane owns it
val x = c.position.x.roundToInt().coerceIn(0, sw - 1)
val y = c.position.y.roundToInt().coerceIn(0, sh - 1)
when {
@@ -121,13 +98,10 @@ internal suspend fun PointerInputScope.streamTouchPassthrough(handle: Long, styl
internal suspend fun PointerInputScope.streamTouchInput(
handle: Long,
stylus: StylusStream?,
trackpad: Boolean,
invertScroll: Boolean,
onCycleStats: () -> Unit,
onKeyboard: (show: Boolean) -> Unit,
) {
val scrollDir = if (invertScroll) -1 else 1
var lastTapUp = 0L
var lastTapX = 0f
var lastTapY = 0f
@@ -144,7 +118,7 @@ internal suspend fun PointerInputScope.streamTouchInput(
)
}
awaitEachGesture {
val down = awaitFirstFingerDown(stylus)
val down = awaitFirstDown(requireUnconsumed = false)
val startX = down.position.x
val startY = down.position.y
// A touch landing just after a quick tap nearby = tap-and-drag: hold the left
@@ -181,8 +155,7 @@ internal suspend fun PointerInputScope.streamTouchInput(
while (true) {
val ev = awaitPointerEvent()
stylus?.intercept(ev, size)
val pressed = ev.changes.filter { it.pressed && !isStylus(it, stylus) }
val pressed = ev.changes.filter { it.pressed }
if (pressed.isEmpty()) {
upTime = ev.changes.firstOrNull()?.uptimeMillis ?: upTime
break
@@ -211,12 +184,12 @@ internal suspend fun PointerInputScope.streamTouchInput(
val sy = ((prevCy - cy) / SCROLL_DIV).toInt() // finger up → wheel up
val sx = ((cx - prevCx) / SCROLL_DIV).toInt()
if (sy != 0) {
NativeBridge.nativeSendScroll(handle, 0, sy * 120 * scrollDir)
NativeBridge.nativeSendScroll(handle, 0, sy * 120)
prevCy = cy
moved = true
}
if (sx != 0) {
NativeBridge.nativeSendScroll(handle, 1, sx * 120 * scrollDir)
NativeBridge.nativeSendScroll(handle, 1, sx * 120)
prevCx = cx
moved = true
}
@@ -106,17 +106,6 @@ object Keymap {
KeyEvent.KEYCODE_DPAD_UP -> 0x26
KeyEvent.KEYCODE_DPAD_RIGHT -> 0x27
KeyEvent.KEYCODE_DPAD_DOWN -> 0x28
// TV-remote SELECT = Enter (a gamepad's press routes via SOURCE_GAMEPAD before this).
KeyEvent.KEYCODE_DPAD_CENTER -> 0x0D
// Consumer/media keys — forwarded to the host while streaming (volume stays local:
// MainActivity's pass-through list wins before the map is consulted).
KeyEvent.KEYCODE_MEDIA_PLAY_PAUSE,
KeyEvent.KEYCODE_MEDIA_PLAY,
KeyEvent.KEYCODE_MEDIA_PAUSE -> 0xB3 // VK_MEDIA_PLAY_PAUSE
KeyEvent.KEYCODE_MEDIA_NEXT -> 0xB0 // VK_MEDIA_NEXT_TRACK
KeyEvent.KEYCODE_MEDIA_PREVIOUS -> 0xB1 // VK_MEDIA_PREV_TRACK
KeyEvent.KEYCODE_MEDIA_STOP -> 0xB2 // VK_MEDIA_STOP
// Modifiers (L/R-specific VKs; the host folds the generic ones onto the left variant)
KeyEvent.KEYCODE_SHIFT_LEFT -> 0xA0
@@ -287,66 +287,6 @@ object NativeBridge {
/** One key transition. vk: Windows VK (0 = dropped by Rust). mods: VK modifier mask (0 for now). */
external fun nativeSendKey(handle: Long, vk: Int, down: Boolean, mods: Int)
/**
* Whether the host advertised full-fidelity stylus injection (`HOST_CAP_PEN`) — the gate
* for splitting stylus pointers out of the touch path onto the pen plane. False on `0`.
*/
external fun nativeHostSupportsPen(handle: Long): Boolean
/**
* One stylus batch of STATE-FULL samples (the pen plane; design/pen-tablet-input.md §7):
* [count] × 10 floats, oldest first — `[state, tool, x, y, pressure, distance, tilt_deg,
* azimuth_deg, roll_deg, dt_us]`. `state` = the wire in-range/touching/barrel bits; `tool`
* 0=pen 1=eraser; x/y/pressure/distance normalized 0..1; distance/tilt/azimuth/roll < 0 =
* unknown. Send only when [nativeHostSupportsPen]; repeat the last sample ≤100 ms while the
* pen is in range (the host force-releases a silent stroke after 200 ms).
*/
external fun nativeSendPen(handle: Long, samples: FloatArray, count: Int)
/**
* Whether the host advertised committed-text injection (`HOST_CAP_TEXT_INPUT`) — its inject
* backend can type Unicode text directly. Picks the real IME `InputConnection` (autocorrect,
* gesture typing, non-Latin scripts) over the TYPE_NULL raw-key fallback. False on `0`.
*/
external fun nativeTextInputSupported(handle: Long): Boolean
/**
* Committed IME text → one `TextInput` wire event per Unicode scalar, in order. Control
* characters are skipped natively (Enter/Backspace ride [nativeSendKey]). Only meaningful
* when [nativeTextInputSupported] returned true — older hosts ignore the events.
*/
external fun nativeSendText(handle: Long, text: String)
// ---- Shared clipboard (text v1): Kotlin drives ClipboardManager, Rust the protocol ----
// Opt-in per session (nativeClipControl). Local copies are announced as lazy offers; bytes
// cross only when the host pastes (a "fetch:" event answered by nativeClipServeText). Host
// copies arrive as "offer:" events, fetched eagerly into the system clipboard.
/** Whether the host advertised a working shared-clipboard service (HOST_CAP_CLIPBOARD). */
external fun nativeClipSupported(handle: Long): Boolean
/** Session-level clipboard opt-in/out; nothing happens until enabled=true crosses. */
external fun nativeClipControl(handle: Long, enabled: Boolean)
/** Announce "this device's clipboard now holds text". [seq]: monotonic, newest wins. */
external fun nativeClipOfferText(handle: Long, seq: Int)
/** Pull the text of the host's offer [seq] → transfer id echoed on "data:"/"error:", or -1. */
external fun nativeClipFetchText(handle: Long, seq: Int): Int
/** Answer a "fetch:" event with the clipboard's current text (the host is pasting). */
external fun nativeClipServeText(handle: Long, reqId: Int, text: String)
/** Abort a clipboard transfer by id (either direction). */
external fun nativeClipCancel(handle: Long, id: Int)
/**
* Block ≤250 ms for the next clipboard event, as a compact string: `state:<0|1>` ·
* `offer:<seq>:<hasText>` · `fetch:<reqId>` · `data:<xferId>:<text>` · `cancel:<id>` ·
* `error:<id>:<code>` · `closed` (session gone) — null on timeout. Dedicated poll thread.
*/
external fun nativeNextClip(handle: Long): String?
// ---- Gamepad: each controller forwarded on its own wire pad index (0..15, low byte of flags) ----
// The pad index is assigned per Android device by GamepadRouter; a single controller lands on 0,
// so its wire is byte-identical to the old single-pad path. The core folds the per-transition
@@ -1,182 +0,0 @@
//! Shared-clipboard plane (text-only v1): Kotlin drives the Android `ClipboardManager`, these
//! shims drive [`punktfunk_core::client::NativeClient`]'s clipboard surface.
//!
//! Model (mirrors the desktop clients): opt-in via `nativeClipControl(true)`; local copies are
//! announced lazily as format-list offers (`nativeClipOfferText`) and the bytes cross only when
//! the host pastes (a `fetch` event answered by `nativeClipServeText`); a host copy arrives as an
//! `offer` event, which the Kotlin side fetches eagerly (Android's clipboard has no lazy provider
//! path worth the complexity) and lands in the system clipboard on the `data` event.
//!
//! Events cross to Kotlin as compact strings from the blocking `nativeNextClip` poll (drained on
//! a dedicated thread, same pattern as `nativeNextRumble`):
//! `state:<0|1>` · `offer:<seq>:<has_text 0|1>` · `fetch:<req_id>` · `data:<xfer_id>:<text>` ·
//! `cancel:<id>` · `error:<id>:<code>` · `closed` — null on a poll timeout. Non-text fetch
//! requests are cancelled natively (only text is ever offered, so they shouldn't occur).
use std::time::Duration;
use jni::objects::{JObject, JString};
use jni::sys::{jboolean, jint, jlong, jstring};
use jni::JNIEnv;
use punktfunk_core::clipboard::ClipEventCore;
use punktfunk_core::error::PunktfunkError;
use punktfunk_core::quic::{ClipKind, CLIP_FILE_INDEX_NONE, HOST_CAP_CLIPBOARD};
use super::SessionHandle;
/// The portable wire MIME both ends map to their platform text type.
const TEXT_MIME: &str = "text/plain;charset=utf-8";
/// Deref the opaque handle (`0` → `None`).
///
/// SAFETY: live handle per the nativeConnect/nativeClose contract; every method used is `&self`
/// on the `Sync` connector.
fn client(handle: jlong) -> Option<&'static SessionHandle> {
if handle == 0 {
return None;
}
// SAFETY: see the function docs — the Kotlin side guarantees the handle outlives the call.
Some(unsafe { &*(handle as *const SessionHandle) })
}
/// `NativeBridge.nativeClipSupported(handle)` — the host advertised `HOST_CAP_CLIPBOARD`.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeClipSupported(
_env: JNIEnv,
_this: JObject,
handle: jlong,
) -> jboolean {
client(handle).map_or(0, |h| {
u8::from(h.client.host_caps() & HOST_CAP_CLIPBOARD != 0)
})
}
/// `NativeBridge.nativeClipControl(handle, enabled)` — session-level opt-in/out. Nothing
/// clipboard-related happens on either side until an `enabled: true` crosses.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeClipControl(
_env: JNIEnv,
_this: JObject,
handle: jlong,
enabled: jboolean,
) {
if let Some(h) = client(handle) {
let _ = h.client.clip_control(enabled != 0, 0);
}
}
/// `NativeBridge.nativeClipOfferText(handle, seq)` — announce "the Android clipboard now holds
/// text" (format list only; bytes cross when the host fetches). `seq` is Kotlin's monotonic
/// counter, newest wins.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeClipOfferText(
_env: JNIEnv,
_this: JObject,
handle: jlong,
seq: jint,
) {
if let Some(h) = client(handle) {
let _ = h.client.clip_offer(
seq as u32,
vec![ClipKind {
mime: TEXT_MIME.into(),
size_hint: 0,
}],
);
}
}
/// `NativeBridge.nativeClipFetchText(handle, seq)` — pull the text of the host's offer `seq`.
/// Returns the transfer id echoed on the matching `data:`/`error:` event, or 1.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeClipFetchText(
_env: JNIEnv,
_this: JObject,
handle: jlong,
seq: jint,
) -> jint {
client(handle)
.and_then(|h| {
h.client
.clip_fetch(seq as u32, TEXT_MIME.into(), CLIP_FILE_INDEX_NONE)
.ok()
})
.map_or(-1, |xfer| xfer as jint)
}
/// `NativeBridge.nativeClipServeText(handle, reqId, text)` — answer a `fetch:` event with the
/// clipboard's current text (the host is pasting our offer).
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeClipServeText(
mut env: JNIEnv,
_this: JObject,
handle: jlong,
req_id: jint,
text: JString,
) {
let Some(h) = client(handle) else { return };
let Ok(s) = env.get_string(&text) else {
let _ = h.client.clip_cancel(req_id as u32);
return;
};
let _ = h
.client
.clip_serve(req_id as u32, String::from(s).into_bytes(), true);
}
/// `NativeBridge.nativeClipCancel(handle, id)` — abort a transfer (either direction).
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeClipCancel(
_env: JNIEnv,
_this: JObject,
handle: jlong,
id: jint,
) {
if let Some(h) = client(handle) {
let _ = h.client.clip_cancel(id as u32);
}
}
/// `NativeBridge.nativeNextClip(handle)` — block ≤250 ms for the next clipboard event, encoded
/// as a compact string (module docs); null on timeout, `"closed"` once the session is gone.
/// Call from a dedicated poll thread.
///
/// Text payloads ride `data:<xfer_id>:<text>` decoded lossily — safe because the phase-0
/// clipboard task delivers a whole payload in ONE event (`last = true`), so a chunk boundary
/// can never split a UTF-8 sequence.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextClip(
env: JNIEnv,
_this: JObject,
handle: jlong,
) -> jstring {
let Some(h) = client(handle) else {
return std::ptr::null_mut();
};
let msg = match h.client.next_clip(Duration::from_millis(250)) {
Ok(ClipEventCore::State { enabled, .. }) => format!("state:{}", u8::from(enabled)),
Ok(ClipEventCore::RemoteOffer { seq, kinds }) => {
let has_text = kinds.iter().any(|k| k.mime.starts_with("text/plain"));
format!("offer:{seq}:{}", u8::from(has_text))
}
Ok(ClipEventCore::FetchRequest { req_id, mime, .. }) => {
if mime.starts_with("text/plain") {
format!("fetch:{req_id}")
} else {
// We only ever offer text; cancel anything else rather than stall the host.
let _ = h.client.clip_cancel(req_id);
return std::ptr::null_mut();
}
}
Ok(ClipEventCore::Data { xfer_id, bytes, .. }) => {
format!("data:{xfer_id}:{}", String::from_utf8_lossy(&bytes))
}
Ok(ClipEventCore::Cancelled { id }) => format!("cancel:{id}"),
Ok(ClipEventCore::Error { id, code }) => format!("error:{id}:{code}"),
Err(PunktfunkError::NoFrame) => return std::ptr::null_mut(),
Err(_) => "closed".into(),
};
env.new_string(msg)
.map(|s| s.into_raw())
.unwrap_or(std::ptr::null_mut())
}
@@ -201,9 +201,6 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
// No display-volume forwarding from Android yet (the panel tone-maps PQ itself via the
// Surface dataspace + static metadata) — the host keeps its virtual-display EDID defaults.
None,
// No non-video caps: this client does not render the host cursor locally (no shape/state
// planes in the jni surface), so advertising CLIENT_CAP_CURSOR would stream cursor-less.
0,
launch, // a store-qualified library id to boot into a game, or None for the desktop
pin, // Some → Crypto on host-fp mismatch
identity, // owned (cert, key) PEM, or None (anonymous)
+2 -131
View File
@@ -6,14 +6,11 @@
//! conventions: buttons 1=left/2=middle/3=right/4=X1/5=X2; scroll axis 0=vertical/1=horizontal,
//! signed 120-unit delta, +=up/right; keys are Windows VK (mapped from KEYCODE_* on the Kotlin side).
use jni::objects::{JByteBuffer, JFloatArray, JObject, JString};
use jni::objects::{JByteBuffer, JObject};
use jni::sys::{jboolean, jint, jlong};
use jni::JNIEnv;
use punktfunk_core::input::{InputEvent, InputKind};
use punktfunk_core::quic::{
PenSample, PenTool, RichInput, HID_REPORT_MAX, HOST_CAP_PEN, HOST_CAP_TEXT_INPUT,
PEN_ANGLE_UNKNOWN, PEN_BATCH_MAX, PEN_DISTANCE_UNKNOWN, PEN_TILT_UNKNOWN,
};
use punktfunk_core::quic::{RichInput, HID_REPORT_MAX};
use super::SessionHandle;
@@ -148,132 +145,6 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendKey(
send_event(handle, kind, vk as u32, 0, 0, mods as u32);
}
/// `NativeBridge.nativeTextInputSupported(handle)` — whether the host advertised
/// `HOST_CAP_TEXT_INPUT` (its inject backend types committed text), so the Kotlin side can pick
/// the real IME `InputConnection` over the TYPE_NULL raw-key fallback. `0` handle → false.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeTextInputSupported(
_env: JNIEnv,
_this: JObject,
handle: jlong,
) -> jboolean {
if handle == 0 {
return 0;
}
// SAFETY: live handle per the nativeConnect/nativeClose contract; host_caps is &self.
let h = unsafe { &*(handle as *const SessionHandle) };
u8::from(h.client.host_caps() & HOST_CAP_TEXT_INPUT != 0)
}
/// `NativeBridge.nativeHostSupportsPen(handle)` — the host advertised `HOST_CAP_PEN`, so the
/// Kotlin side splits stylus pointers out of the touch path onto the pen plane
/// (design/pen-tablet-input.md §7). `0` handle → false.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeHostSupportsPen(
_env: JNIEnv,
_this: JObject,
handle: jlong,
) -> jboolean {
if handle == 0 {
return 0;
}
// SAFETY: live handle per the nativeConnect/nativeClose contract; host_caps is &self.
let h = unsafe { &*(handle as *const SessionHandle) };
u8::from(h.client.host_caps() & HOST_CAP_PEN != 0)
}
/// Floats per sample in the `nativeSendPen` flat array.
const PEN_JNI_STRIDE: usize = 10;
/// `NativeBridge.nativeSendPen(handle, samples, count)` — one stylus batch of STATE-FULL
/// samples, `count` × [`PEN_JNI_STRIDE`] floats, oldest first:
/// `[state, tool, x, y, pressure, distance, tilt_deg, azimuth_deg, roll_deg, dt_us]`.
/// `state` = the wire `PEN_*` bits; `tool` 0=pen 1=eraser; `x`/`y`/`pressure`/`distance`
/// normalized 0..1; `distance`/`tilt_deg`/`azimuth_deg`/`roll_deg` < 0 = unknown. Call only
/// against a [`nativeHostSupportsPen`] host; the client heartbeats the last sample ≤100 ms
/// while in range (Kotlin side — see `StylusStream`).
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendPen(
env: JNIEnv,
_this: JObject,
handle: jlong,
samples: JFloatArray,
count: jint,
) {
if handle == 0 || count <= 0 {
return;
}
let count = (count as usize).min(PEN_BATCH_MAX);
let mut buf = [0f32; PEN_BATCH_MAX * PEN_JNI_STRIDE];
let flat = &mut buf[..count * PEN_JNI_STRIDE];
if env.get_float_array_region(&samples, 0, flat).is_err() {
return; // short array — a bridge bug, never worth a crash on the input path
}
let mut batch = [PenSample::default(); PEN_BATCH_MAX];
for (slot, s) in batch.iter_mut().zip(flat.chunks_exact(PEN_JNI_STRIDE)) {
if !s[2].is_finite() || !s[3].is_finite() {
return; // never forward a NaN coordinate
}
*slot = PenSample {
state: s[0] as u8,
tool: if s[1] as u8 == 1 {
PenTool::Eraser
} else {
PenTool::Pen
},
x: s[2].clamp(0.0, 1.0),
y: s[3].clamp(0.0, 1.0),
pressure: (s[4].clamp(0.0, 1.0) * 65535.0) as u16,
distance: if s[5] < 0.0 {
PEN_DISTANCE_UNKNOWN
} else {
(s[5].clamp(0.0, 1.0) * 65534.0) as u16
},
tilt_deg: if s[6] < 0.0 {
PEN_TILT_UNKNOWN
} else {
(s[6].clamp(0.0, 90.0)) as u8
},
azimuth_deg: if s[7] < 0.0 {
PEN_ANGLE_UNKNOWN
} else {
(s[7] as u16) % 360
},
roll_deg: if s[8] < 0.0 {
PEN_ANGLE_UNKNOWN
} else {
(s[8] as u16) % 360
},
dt_us: s[9].clamp(0.0, 65535.0) as u16,
};
}
// SAFETY: live handle per the nativeConnect/nativeClose contract; send_pen is &self.
let h = unsafe { &*(handle as *const SessionHandle) };
let _ = h.client.send_pen(&batch[..count]);
}
/// `NativeBridge.nativeSendText(handle, text)` — committed IME text, one `TextInput` event per
/// Unicode scalar (`code` = the scalar; multi-char commits are consecutive events in order).
/// Control characters are skipped — Enter/Backspace/Tab ride the VK key path. Call only when
/// [`Java_io_unom_punktfunk_kit_NativeBridge_nativeTextInputSupported`] returned true.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendText(
mut env: JNIEnv,
_this: JObject,
handle: jlong,
text: JString,
) {
if handle == 0 {
return;
}
let Ok(s) = env.get_string(&text) else {
return;
};
for ch in String::from(s).chars().filter(|c| !c.is_control()) {
send_event(handle, InputKind::TextInput, ch as u32, 0, 0, 0);
}
}
// ---- Gamepad: Kotlin captures (KeyEvent/MotionEvent) → NativeClient::send_input ---------------
// Multi-pad model: each physical controller is forwarded on its own wire pad index (0..15), carried
// in the low byte of `flags` on every per-pad event — the Kotlin side (`GamepadRouter`) assigns a
@@ -17,7 +17,6 @@
//! TODO(M4 Android stage 1): client→host DualSense rich input (`send_rich_input`), mode
//! renegotiation. Port the remaining orchestration from `clients/linux`.
mod clipboard;
mod connect;
mod input;
mod planes;
@@ -302,26 +302,13 @@ final class SessionModel: ObservableObject {
if preferredCodec == PunktfunkConnection.codecPyroWave, MetalWaveletDecoder.supported {
videoCodecs |= PunktfunkConnection.codecPyroWave
}
// Cursor channel (remote-desktop-sweep M2, macOS): sessions STARTING in the desktop
// mouse model advertise local cursor rendering the host then stops compositing
// the pointer and forwards shape/state, which StreamView draws as the real
// NSCursor. Capture-mode sessions keep today's composited pointer.
#if os(macOS)
let clientCaps: UInt8 =
(MouseInputMode(
rawValue: UserDefaults.standard.string(forKey: DefaultsKey.mouseMode) ?? "")
?? .capture) == .desktop ? 0x01 : 0
#else
let clientCaps: UInt8 = 0
#endif
let result = Result { try PunktfunkConnection(
host: host.address, port: host.port,
width: width, height: height, refreshHz: hz,
pinSHA256: pin, identity: identity, compositor: compositor,
gamepad: gamepad, bitrateKbps: bitrateKbps, videoCaps: videoCaps,
audioChannels: audioChannels,
videoCodecs: videoCodecs, preferredCodec: preferredCodec,
clientCaps: clientCaps, launchID: launchID,
videoCodecs: videoCodecs, preferredCodec: preferredCodec, launchID: launchID,
// Delegated approval: the host holds this connect open until the operator approves
// it (~180 s) outwait that window so a slow approval still lands here. Normal
// connects keep the snappy default.
@@ -221,9 +221,6 @@ public final class PunktfunkConnection {
/// core). The clip *sends* (`clipControl`/`clipOffer`/`clipServe`) share this lock too:
/// they're quick non-blocking enqueues, and a single lock keeps close() ordering simple.
private let clipboardLock = NSLock()
/// Serializes the (single) cursor pull thread against close() both cursor planes are
/// drained by ONE thread, so one lock covers them.
private let cursorLock = NSLock()
/// Negotiated session mode (host-confirmed).
public private(set) var width: UInt32 = 0
@@ -384,106 +381,6 @@ public final class PunktfunkConnection {
public var hostSupportsClipboard: Bool {
hostCaps & UInt8(PUNKTFUNK_HOST_CAP_CLIPBOARD) != 0
}
/// The host answered `HOST_CAP_CURSOR`: it stopped compositing the pointer and forwards
/// shape/state on the cursor planes the client MUST draw the cursor locally.
/// `0x08` the bit moved when `HOST_CAP_TEXT_INPUT` claimed `0x04` on main; testing the
/// old bit would mistake a text-input-capable host (e.g. Windows) for a cursor grant.
public var hostSupportsCursor: Bool {
hostCaps & 0x08 != 0
}
/// The host injects full-fidelity stylus input (`HOST_CAP_PEN`) the gate for splitting
/// Apple Pencil out of the touch path onto the pen plane (``sendPen(_:)``).
public var hostSupportsPen: Bool {
hostCaps & UInt8(PUNKTFUNK_HOST_CAP_PEN) != 0
}
/// One forwarded host-cursor shape (the cursor channel, ABI v11): straight-alpha RGBA,
/// `rgba.count == width * height * 4`, hotspot within the bitmap. Cache by `serial`
/// states reference shapes by it and a re-shown serial never resends pixels.
public struct CursorShapeEvent: Sendable {
public let serial: UInt32
public let width: Int
public let height: Int
public let hotX: Int
public let hotY: Int
public let rgba: Data
}
/// Per-host-tick cursor state: position (host video px, the pointer/hotspot point),
/// visibility, and the host-driven relative-mode hint (an app grabbed/hid the pointer
/// run captured relative; clear absolute, reappearing at `x`/`y`). Latest-wins.
public struct CursorStateEvent: Sendable {
public let serial: UInt32
public let visible: Bool
public let relativeHint: Bool
public let x: Int32
public let y: Int32
}
/// Pull the next forwarded cursor SHAPE (nil = timeout). Only a session connected with
/// `clientCaps` cursor bit against a `hostSupportsCursor` host receives any. Drain shape
/// AND state from ONE dedicated cursor thread (they share a lock).
public func nextCursorShape(timeoutMs: UInt32 = 0) throws -> CursorShapeEvent? {
cursorLock.lock()
defer { cursorLock.unlock() }
guard let h = liveHandle() else { throw PunktfunkClientError.closed }
var out = PunktfunkCursorShape()
let rc = punktfunk_connection_next_cursor_shape(h, &out, timeoutMs)
switch rc {
case statusOK:
// Copy out of the ABI borrow (valid until the next shape call) immediately.
let bytes = out.rgba.map { Data(bytes: $0, count: Int(out.len)) } ?? Data()
return CursorShapeEvent(
serial: out.serial, width: Int(out.w), height: Int(out.h),
hotX: Int(out.hot_x), hotY: Int(out.hot_y), rgba: bytes)
case statusNoFrame:
return nil
case statusClosed:
throw PunktfunkClientError.closed
default:
throw PunktfunkClientError.status(rc)
}
}
/// Pull the next cursor STATE (nil = timeout). Latest-wins drain the queue and apply
/// only the newest. Same thread + gate as [`nextCursorShape`].
public func nextCursorState(timeoutMs: UInt32 = 0) throws -> CursorStateEvent? {
cursorLock.lock()
defer { cursorLock.unlock() }
guard let h = liveHandle() else { throw PunktfunkClientError.closed }
var out = PunktfunkCursorState()
let rc = punktfunk_connection_next_cursor_state(h, &out, timeoutMs)
switch rc {
case statusOK:
return CursorStateEvent(
serial: out.serial,
visible: out.flags & 0x01 != 0,
relativeHint: out.flags & 0x02 != 0,
x: out.x, y: out.y)
case statusNoFrame:
return nil
case statusClosed:
throw PunktfunkClientError.closed
default:
throw PunktfunkClientError.status(rc)
}
}
/// Tell the host who renders the pointer (the §8 mid-stream mouse-model flip, ABI v12):
/// `clientDraws = true` this client draws it locally (the desktop mouse model; the host
/// excludes the pointer from the video and forwards shape/state); `false` the host
/// composites it into the video (the capture model, full fidelity). Idempotent,
/// latest-wins; harmless against hosts without the cursor cap. Fire-and-forget errors
/// are swallowed (a closed session is the only failure and it moots the flip).
public func setCursorRender(clientDraws: Bool) {
cursorLock.lock()
defer { cursorLock.unlock() }
guard let h = liveHandle() else { return }
_ = punktfunk_connection_set_cursor_render(h, clientDraws)
}
/// The resolved codec as a `VideoCodec` (H.264 / HEVC / AV1) drives the bitstream framing
/// (Annex-B NAL parsing vs the AV1 OBU repack).
public var videoCodec: VideoCodec { VideoCodec(wire: resolvedCodec) }
@@ -520,7 +417,6 @@ public final class PunktfunkConnection {
audioChannels: UInt8 = 2,
videoCodecs: UInt8 = 0x02, // PUNKTFUNK_CODEC_HEVC the codecs this client can decode
preferredCodec: UInt8 = 0, // 0 = auto; else PUNKTFUNK_CODEC_* soft preference
clientCaps: UInt8 = 0, // ABI v11: PUNKTFUNK_CLIENT_CAP_CURSOR = render the host cursor locally
launchID: String? = nil,
timeoutMs: UInt32 = 10_000
) throws {
@@ -540,18 +436,18 @@ public final class PunktfunkConnection {
withOptionalCString(launchID) { launch in
if let pin = pinSHA256 {
return pin.withUnsafeBytes { p in
punktfunk_connect_ex9(
punktfunk_connect_ex8(
cs, port, width, height, refreshHz, compositor.rawValue,
gamepad.rawValue, bitrateKbps, videoCaps, audioChannels,
videoCodecs, preferredCodec, clientCaps, launch,
videoCodecs, preferredCodec, launch,
p.bindMemory(to: UInt8.self).baseAddress, &observed,
cert, key, timeoutMs, &connectStatus)
}
}
return punktfunk_connect_ex9(
return punktfunk_connect_ex8(
cs, port, width, height, refreshHz, compositor.rawValue,
gamepad.rawValue, bitrateKbps, videoCaps, audioChannels,
videoCodecs, preferredCodec, clientCaps, launch,
videoCodecs, preferredCodec, launch,
nil, &observed, cert, key, timeoutMs, &connectStatus)
}
}
@@ -1141,19 +1037,6 @@ public final class PunktfunkConnection {
_ = punktfunk_connection_send_input(h, &ev)
}
/// Send one stylus sample batch ( `PUNKTFUNK_PEN_BATCH_MAX`, oldest first) on the pen
/// plane. Gate on ``hostSupportsPen`` the core refuses toward a host without the cap.
/// Thread-safe; silently dropped after close (input is lossy by design).
public func sendPen(_ samples: [PunktfunkPenSample]) {
guard !samples.isEmpty else { return }
abiLock.lock()
defer { abiLock.unlock() }
guard let h = handle, !closeRequested else { return }
samples.withUnsafeBufferPointer { buf in
_ = punktfunk_connection_send_pen(h, buf.baseAddress, UInt32(buf.count))
}
}
/// Signal a **deliberate** user-initiated quit before ``close()``: the connection closes with
/// `QUIT_CLOSE_CODE` (81) so the host tears the session down immediately instead of holding the
/// keep-alive linger for a reconnect. Call only from an explicit "Disconnect" action NOT from a
@@ -1176,12 +1059,10 @@ public final class PunktfunkConnection {
feedbackLock.lock()
statsLock.lock()
clipboardLock.lock()
cursorLock.lock()
abiLock.lock()
let h = handle
handle = nil
abiLock.unlock()
cursorLock.unlock()
clipboardLock.unlock()
statsLock.unlock()
feedbackLock.unlock()
@@ -1,268 +0,0 @@
// Apple Pencil state-full wire pen samples (design/pen-tablet-input.md §7).
//
// Every sample carries the COMPLETE pen state (in-range/touching/buttons + all axes) the
// host diffs consecutive samples and synthesizes down/up/button transitions itself, so a lost
// datagram self-heals and this file never sends edge events. Three sources feed one stream:
// UITouch contacts (with coalesced samples for full 240 Hz fidelity), the hover gesture
// (zOffset > 0 distinguishes a hovering Pencil from a trackpad pointer), and
// UIPencilInteraction (squeeze held barrel 1, double-tap a momentary barrel 2
// Apple Pencil has no hardware eraser end or barrel buttons, so these mappings are how
// host-side apps get their stylus button/eraser affordances).
//
// HEARTBEAT (wire contract see `PunktfunkPenSample` in punktfunk_core.h): while the pen is
// in range or touching, the last sample repeats every 100 ms even when nothing changed.
// UIKit is silent for a stationary Pencil, and the host force-releases the stroke after
// 200 ms without samples (its dead-client failsafe) the timer keeps a held stroke alive.
#if os(iOS)
import PunktfunkCore
import UIKit
final class PencilStream: NSObject, UIPencilInteractionDelegate {
enum Phase { case down, move, up, cancel }
/// One assembled batch ( `PUNKTFUNK_PEN_BATCH_MAX` samples) ready for the connection.
var send: (([PunktfunkPenSample]) -> Void)?
/// View-space point normalized [0,1] video coordinates (the letterbox mapping the
/// touch path already uses). nil until a mode is negotiated samples are dropped then.
var videoNorm: ((CGPoint) -> (Float, Float)?)?
private var inRange = false
private var touching = false
/// Squeeze held (mapped to wire BARREL1).
private var squeezeHeld = false
/// Whether this device/Pencil pair has demonstrated hover decides what a lift means:
/// hover-capable hardware keeps proximity (the hover recognizer owns the exit), anything
/// else leaves range on lift so the host never parks a phantom hovering pen.
private var sawHover = false
/// A hover gesture is live right now (routes ended-state hover callbacks to us even when
/// the recognizer's final zOffset reads 0).
private(set) var hoverActive = false
private var last = PencilStream.idleSample()
private var heartbeat: Timer?
// MARK: - Contact path (UITouch, `.pencil` only)
func touches(_ touches: Set<UITouch>, event: UIEvent?, phase: Phase, in view: UIView) {
// At most one Pencil exists; a set with several is UIKit batching phases of the same
// stylus the last one carries the freshest state.
guard let touch = touches.max(by: { $0.timestamp < $1.timestamp }) else { return }
switch phase {
case .down, .move:
touching = true
inRange = true
// Coalesced samples restore the Pencil's full capture rate (UIKit delivers at
// display cadence); oldest first, `dt_us` preserving their spacing.
let raw = event?.coalescedTouches(for: touch) ?? [touch]
var batch: [PunktfunkPenSample] = []
var prevTs: TimeInterval?
for t in raw.suffix(Int(PUNKTFUNK_PEN_BATCH_MAX)) {
guard let s = contactSample(t, in: view, prevTs: prevTs) else { continue }
prevTs = t.timestamp
batch.append(s)
}
emit(batch)
case .up:
touching = false
// Hover-capable hardware: lift back to hover, the recognizer exits range later.
// Otherwise a lift IS the range exit (mirror of the host's GameStream heuristic).
inRange = sawHover
var s = last
s.pressure = 0
s.state = stateBits()
if let posSample = contactSample(touch, in: view, prevTs: nil) {
s.x = posSample.x
s.y = posSample.y
}
emit([s])
case .cancel:
release()
}
}
// MARK: - Hover path (forwarded from the view's hover recognizer)
/// Returns whether the event was consumed as Pencil hover; `false` hands it back to the
/// pointer path. A hovering Pencil reports `zOffset > 0`; trackpad/mouse hover is 0.
func maybeHover(_ r: UIHoverGestureRecognizer, in view: UIView) -> Bool {
switch r.state {
case .began, .changed:
guard r.zOffset > 0 || hoverActive else { return false }
hoverActive = true
sawHover = true
inRange = true
touching = false
guard let (x, y) = videoNorm?(r.location(in: view)) else { return true }
var s = PencilStream.idleSample()
s.state = stateBits()
s.x = x
s.y = y
s.distance = UInt16((r.zOffset.clamped(to: 0...1) * 65534).rounded())
s.tilt_deg = Self.tiltDeg(altitude: r.altitudeAngle)
s.azimuth_deg = Self.azimuthDeg(r.azimuthAngle(in: view))
if #available(iOS 17.5, *) { s.roll_deg = Self.rollDeg(r.rollAngle) }
emit([s])
return true
case .ended, .cancelled, .failed:
guard hoverActive else { return false }
hoverActive = false
if !touching { release() }
return true
default:
return hoverActive
}
}
// MARK: - UIPencilInteractionDelegate (squeeze barrel 1 held, tap barrel 2 click)
@available(iOS 17.5, *)
func pencilInteraction(
_ interaction: UIPencilInteraction, didReceiveSqueeze squeeze: UIPencilInteraction.Squeeze
) {
switch squeeze.phase {
case .began:
squeezeHeld = true
case .ended, .cancelled:
squeezeHeld = false
default:
return
}
guard inRange || touching else { return }
var s = last
s.state = stateBits()
emit([s])
}
func pencilInteractionDidTap(_ interaction: UIPencilInteraction) {
guard inRange || touching else { return }
// A momentary barrel-2 click: press + release as two state-full samples in ONE batch
// the host's tracker emits the button press and release in order.
var press = last
press.state = stateBits() | UInt8(PUNKTFUNK_PEN_BARREL2)
var releaseS = last
releaseS.state = stateBits()
emit([press, releaseS])
}
// MARK: - Lifecycle
/// Session stop / view teardown: leave range so the host lifts anything held.
func reset() {
if inRange || touching { release() }
sawHover = false
hoverActive = false
squeezeHeld = false
}
private func release() {
touching = false
inRange = false
var s = last
s.pressure = 0
s.state = 0
emit([s])
}
// MARK: - Sample assembly
private func contactSample(
_ t: UITouch, in view: UIView, prevTs: TimeInterval?
) -> PunktfunkPenSample? {
guard let (x, y) = videoNorm?(t.location(in: view)) else { return nil }
var s = PencilStream.idleSample()
s.state = stateBits()
s.x = x
s.y = y
// maximumPossibleForce is 0 until the system knows the stylus full force then
// (binary-stylus semantics, matching the host's unknown-pressure rule).
let maxForce = t.maximumPossibleForce
s.pressure =
maxForce > 0
? UInt16((Double(t.force / maxForce).clamped(to: 0...1) * 65535).rounded())
: UInt16.max
s.distance = 0
s.tilt_deg = Self.tiltDeg(altitude: t.altitudeAngle)
s.azimuth_deg = Self.azimuthDeg(t.azimuthAngle(in: view))
if #available(iOS 17.5, *) { s.roll_deg = Self.rollDeg(t.rollAngle) }
if let prevTs {
s.dt_us = UInt16(((t.timestamp - prevTs) * 1_000_000).clamped(to: 0...65535))
}
return s
}
private func stateBits() -> UInt8 {
var bits: UInt8 = 0
if inRange || touching { bits |= UInt8(PUNKTFUNK_PEN_IN_RANGE) }
if touching { bits |= UInt8(PUNKTFUNK_PEN_TOUCHING) }
if squeezeHeld { bits |= UInt8(PUNKTFUNK_PEN_BARREL1) }
return bits
}
private func emit(_ batch: [PunktfunkPenSample]) {
guard !batch.isEmpty else { return }
last = batch[batch.count - 1]
last.dt_us = 0
send?(batch)
armHeartbeat()
}
/// The 100 ms keepalive while in range (see the file header). 80 ms leaves headroom
/// under the host's 200 ms failsafe even with one lost datagram.
private func armHeartbeat() {
heartbeat?.invalidate()
guard inRange || touching else {
heartbeat = nil
return
}
heartbeat = Timer.scheduledTimer(withTimeInterval: 0.08, repeats: true) {
[weak self] _ in
guard let self, self.inRange || self.touching else {
self?.heartbeat?.invalidate()
self?.heartbeat = nil
return
}
self.send?([self.last])
}
}
// MARK: - Angle conversions
/// Altitude (π/2 = perpendicular) wire tilt-from-normal in degrees, 0...90.
private static func tiltDeg(altitude: CGFloat) -> UInt8 {
UInt8((90 - altitude * 180 / .pi).rounded().clamped(to: 0...90))
}
/// Apple azimuth (0 along the view's +x axis, clockwise, y-down) wire azimuth
/// (0 = north/up on screen, clockwise): +90° offset.
private static func azimuthDeg(_ apple: CGFloat) -> UInt16 {
let deg = (apple * 180 / .pi + 90).truncatingRemainder(dividingBy: 360)
return UInt16((deg + 360).truncatingRemainder(dividingBy: 360).rounded()) % 360
}
/// Pencil Pro roll (radians, π...π) wire barrel roll 0...359°.
private static func rollDeg(_ roll: CGFloat) -> UInt16 {
let deg = (roll * 180 / .pi).truncatingRemainder(dividingBy: 360)
return UInt16(((deg + 360).truncatingRemainder(dividingBy: 360)).rounded()) % 360
}
/// All-unknown baseline: sentinel angles/distance, tool = pen (the eraser is host-side
/// state driven by the squeeze/tap mappings, not a hardware end).
private static func idleSample() -> PunktfunkPenSample {
PunktfunkPenSample(
x: 0, y: 0, pressure: 0,
distance: UInt16(PUNKTFUNK_PEN_DISTANCE_UNKNOWN),
azimuth_deg: UInt16(PUNKTFUNK_PEN_ANGLE_UNKNOWN),
roll_deg: UInt16(PUNKTFUNK_PEN_ANGLE_UNKNOWN),
dt_us: 0, state: 0,
tool: UInt8(PUNKTFUNK_PEN_TOOL_PEN),
tilt_deg: UInt8(PUNKTFUNK_PEN_TILT_UNKNOWN),
_reserved: (0, 0, 0))
}
}
extension Comparable {
fileprivate func clamped(to range: ClosedRange<Self>) -> Self {
min(max(self, range.lowerBound), range.upperBound)
}
}
#endif
@@ -219,32 +219,6 @@ public final class StreamLayerView: NSView {
/// flipped live by M. A live flip re-engages capture in the new model so
/// disassociation + the abs/rel choice swap atomically. Main-thread only.
private var desktopMouse = false
/// Cursor channel (M2): the host forwards shape/state and WE draw the pointer. Active
/// when the Welcome carried `HOST_CAP_CURSOR` (only sessions that advertised the client
/// cap get it). Shapes cache by serial; state is latest-wins. Main-thread only.
private var cursorChannelActive = false
/// A forwarded host cursor shape, cached RAW (not as a finished `NSCursor`) so the pointer can be
/// (re)built at the CURRENT video-fit scale see `scaledCursor`. The host forwards the bitmap in
/// host FRAMEBUFFER pixels, whose size tracks the host's display scaling (32 px at 100%, 96 px at
/// 300% DPI); scaling by the video fit keeps the pointer sized to the streamed desktop at any host
/// scaling instead of ballooning on a high-DPI host.
private struct HostCursorShape {
let cg: CGImage
let width: Int
let height: Int
let hotX: Int
let hotY: Int
}
private var hostCursors: [UInt32: HostCursorShape] = [:]
private var cursorState: PunktfunkConnection.CursorStateEvent?
/// Last `CursorRenderMode.clientDraws` told to the host (the §8 mid-stream render flip);
/// nil = nothing sent yet. Edge-detected by [`reconcileCursorRender`] from the live mouse
/// model, so the chord, engage/release, and session start all reconcile through one path.
private var sentClientDraws: Bool?
/// M3 hint tracking: edge-triggered so a manual M isn't fought the override latch
/// holds until the HOST's intent next changes.
private var lastHint: Bool?
private var hintOverride = false
/// One-shot auto-engage request (stream start, trust confirmed) attempted as soon
/// as the view is in a window with real bounds, then dropped, so it can never fire
/// surprisingly later (e.g. on a resize).
@@ -482,7 +456,6 @@ public final class StreamLayerView: NSView {
window?.makeFirstResponder(self)
window?.invalidateCursorRects(for: self) // desktop model: hide-over-view engages
notifyCaptureChange(true)
reconcileCursorRender()
}
private func releaseCapture() {
@@ -493,7 +466,6 @@ public final class StreamLayerView: NSView {
captured = false
window?.invalidateCursorRects(for: self)
notifyCaptureChange(false)
reconcileCursorRender() // released the host composites the pointer again
}
/// A fully transparent cursor for the desktop mouse model's hide-over-view rect
@@ -508,179 +480,12 @@ public final class StreamLayerView: NSView {
/// globally via `CursorCapture` (the pointer can't leave the view there).
override public func resetCursorRects() {
if captured && desktopMouse {
// Cursor channel active: wear the HOST's pointer shape (it is no longer in the
// video); hidden host pointer (or no shape yet) = invisible. Without the channel,
// M1 behavior: invisible local cursor, the composited host cursor is the visible one.
if cursorChannelActive, let st = cursorState, st.visible,
let shape = hostCursors[st.serial] {
addCursorRect(bounds, cursor: scaledCursor(shape))
} else {
addCursorRect(bounds, cursor: Self.invisibleCursor)
}
addCursorRect(bounds, cursor: Self.invisibleCursor)
} else {
super.resetCursorRects()
}
}
/// Tell the host who renders the pointer (the §8 mid-stream render flip): we draw it only
/// while the DESKTOP model is engaged (the local OS cursor wears the host shape); under
/// the capture model and while released the host composites it into the video (full
/// fidelity, the pre-channel look). One edge-detected reconciler, called from every
/// transition (chord, engage/release, session start).
private func reconcileCursorRender() {
guard cursorChannelActive, let connection else { return }
let clientDraws = captured && desktopMouse
guard sentClientDraws != clientDraws else { return }
sentClientDraws = clientDraws
connection.setCursorRender(clientDraws: clientDraws)
}
/// Flip the mouse model with the atomic release/re-engage swap; `reappearAt` (host video
/// px the M3 hand-back position) warps the local pointer so leaving relative lands the
/// cursor exactly where the host last had it.
private func setDesktopMouse(_ on: Bool, reappearAt: (x: Int32, y: Int32)?) {
guard desktopMouse != on else { return }
let wasCaptured = captured
if wasCaptured { releaseCapture() }
desktopMouse = on
if wasCaptured { engageCapture(fromClick: false) }
window?.invalidateCursorRects(for: self)
if on, let p = reappearAt, let sp = cgScreenPoint(forHostX: p.x, p.y) {
CGWarpMouseCursorPosition(sp)
}
reconcileCursorRender()
}
/// The single cursor pull thread (both planes share the connection's cursor lock):
/// latest-wins state at a short timeout + a non-blocking shape poll per iteration.
/// Exits when the connection closes; events hop to main where all cursor state lives.
private func startCursorPump(_ connection: PunktfunkConnection) {
let thread = Thread { [weak self] in
while true {
do {
var newest: PunktfunkConnection.CursorStateEvent?
if let st = try connection.nextCursorState(timeoutMs: 100) {
newest = st
while let more = try connection.nextCursorState(timeoutMs: 0) {
newest = more // drain latest wins
}
}
while let shape = try connection.nextCursorShape(timeoutMs: 0) {
DispatchQueue.main.async { self?.applyCursorShape(shape) }
}
if let st = newest {
DispatchQueue.main.async { self?.applyCursorState(st) }
}
} catch {
return // connection closed the session is over
}
if self == nil { return }
}
}
thread.name = "pf-cursor-pump"
thread.start()
}
private func applyCursorShape(_ ev: PunktfunkConnection.CursorShapeEvent) {
guard let shape = Self.makeShape(ev) else {
streamInputLog.warning("cursor shape rejected (\(ev.width)x\(ev.height)) — keeping the previous cursor")
return
}
if hostCursors.count >= 64 { hostCursors.removeAll() } // degenerate host: reset
hostCursors[ev.serial] = shape
if cursorState?.serial == ev.serial {
window?.invalidateCursorRects(for: self)
}
}
private func applyCursorState(_ ev: PunktfunkConnection.CursorStateEvent) {
let prev = cursorState
cursorState = ev
if prev?.visible != ev.visible || prev?.serial != ev.serial {
window?.invalidateCursorRects(for: self)
}
// M3 host-driven auto-flip is DISABLED: `relative_hint` is derived from host cursor
// VISIBILITY, and Windows hides the pointer for ordinary desktop activity (clicking,
// typing) not just when a game grabs it. Acting on those transients flipped
// desktopcapturedesktop, which warped the cursor to view-centre and flushed held
// buttons (a spurious button-up ~200 ms into every press broke window drags). Until
// the host exposes a real pointer-LOCK signal (ClipCursor/raw-input, not visibility),
// the mouse model is user-driven only (M). The hint still rides the wire, unused.
_ = (lastHint, hintOverride)
}
/// Decode a forwarded straight-alpha RGBA shape into a CGImage + hotspot. The on-screen SIZE is
/// NOT baked in here it is applied per-use in `scaledCursor` from the live video-fit scale, so
/// the same shape re-fits across window resizes / retina moves without a re-forward.
private static func makeShape(_ ev: PunktfunkConnection.CursorShapeEvent) -> HostCursorShape? {
let (w, h) = (ev.width, ev.height)
guard w > 0, h > 0, ev.rgba.count >= w * h * 4,
let provider = CGDataProvider(data: ev.rgba as CFData),
let cg = CGImage(
width: w, height: h, bitsPerComponent: 8, bitsPerPixel: 32,
bytesPerRow: w * 4, space: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue),
provider: provider, decode: nil, shouldInterpolate: false,
intent: .defaultIntent)
else { return nil }
return HostCursorShape(
cg: cg, width: w, height: h,
hotX: min(ev.hotX, w - 1), hotY: min(ev.hotY, h - 1))
}
/// Points-per-host-pixel: the exact factor the video frame is aspect-fit into the view (the same
/// `AVMakeRect` fit `hostPoint`/`cgScreenPoint` use). The host forwards the pointer bitmap in host
/// framebuffer pixels the mode we drive is in the client's BACKING pixels, so on retina this is
/// ~1/backingScale and the pointer lands at its TRUE size relative to the streamed desktop
/// (crisp, 1:1 with the video) rather than the 2×-inflated pixel-as-points it used to be. Because
/// the bitmap grows with the host's display scaling (96 px at 300% DPI), scaling by this is what
/// keeps a high-DPI host from forwarding a giant pointer. Falls back to 1 before the first
/// mode/layout.
private func cursorFitScale() -> CGFloat {
guard let connection else { return 1 }
let mode = connection.currentMode()
guard mode.width > 0, mode.height > 0, bounds.width > 0, bounds.height > 0 else { return 1 }
let fit = AVMakeRect(
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)), insideRect: bounds)
guard fit.width > 0 else { return 1 }
return fit.width / CGFloat(mode.width)
}
/// Build the `NSCursor` for a cached shape at the CURRENT video-fit scale (see `cursorFitScale`).
/// Both the image size and the hotspot scale together so the click point stays true.
private func scaledCursor(_ shape: HostCursorShape) -> NSCursor {
let scale = cursorFitScale()
let sw = max(1, (CGFloat(shape.width) * scale).rounded())
let sh = max(1, (CGFloat(shape.height) * scale).rounded())
let image = NSImage(cgImage: shape.cg, size: NSSize(width: sw, height: sh))
let hot = NSPoint(
x: min(CGFloat(shape.hotX) * scale, sw - 1),
y: min(CGFloat(shape.hotY) * scale, sh - 1))
return NSCursor(image: image, hotSpot: hot)
}
/// Host video px CG GLOBAL screen coordinates (top-left origin, the
/// `CGWarpMouseCursorPosition` convention `CursorCapture` established) through the
/// aspect-fit letterbox the inverse direction of `hostPoint(from:)`.
private func cgScreenPoint(forHostX hx: Int32, _ hy: Int32) -> CGPoint? {
guard let connection, let window else { return nil }
let mode = connection.currentMode()
guard mode.width > 0, mode.height > 0 else { return nil }
let fit = AVMakeRect(
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
insideRect: bounds)
guard fit.width > 0, fit.height > 0 else { return nil }
let u = (CGFloat(hx) / CGFloat(mode.width)).clamped(to: 0...1)
let v = (CGFloat(hy) / CGFloat(mode.height)).clamped(to: 0...1)
let videoMinYTop = bounds.height - fit.maxY
let pTop = CGPoint(x: fit.minX + u * fit.width, y: videoMinYTop + v * fit.height)
let inView = CGPoint(x: pTop.x, y: bounds.height - pTop.y)
let inWindow = convert(inView, to: nil)
let onScreen = window.convertPoint(toScreen: inWindow)
let primaryHeight = NSScreen.screens.first?.frame.height ?? 0
return CGPoint(x: onScreen.x, y: primaryHeight - onScreen.y)
}
/// A single local monitor for motion + buttons, installed only while captured. A local
/// monitor is more robust than view overrides for relative motion: it sidesteps the
/// `window.acceptsMouseMovedEvents`/tracking-area/responder-chain requirements, and
@@ -842,10 +647,12 @@ public final class StreamLayerView: NSView {
streamInputLog.info("mouse-mode chord ignored: gamescope host is relative-only")
return
}
// A manual flip outranks the standing host hint until the hint next CHANGES.
self.hintOverride = true
self.setDesktopMouse(!self.desktopMouse, reappearAt: nil)
streamInputLog.info("chord: mouse mode \(self.desktopMouse ? "desktop" : "capture", privacy: .public)")
let wasCaptured = self.captured
if wasCaptured { self.releaseCapture() }
self.desktopMouse.toggle()
if wasCaptured { self.engageCapture(fromClick: false) }
self.window?.invalidateCursorRects(for: self)
streamInputLog.info("mouse mode: \(self.desktopMouse ? "desktop" : "capture", privacy: .public)")
}
// The cross-client combos (Q/D/S Ctrl+Alt+Shift on the other clients), delivered by
// the monitor only while captured; the same key-window ownership rule as throughout.
@@ -885,14 +692,6 @@ public final class StreamLayerView: NSView {
if mode == .desktop && !absOK {
streamInputLog.info("desktop mouse mode unavailable on a gamescope host (relative-only) — using capture")
}
// Cursor channel (M2): the host stopped compositing the pointer drain its shape/
// state planes and draw the pointer as the real NSCursor (plus the M3 auto-flip).
if connection.hostSupportsCursor {
cursorChannelActive = true
streamInputLog.info("cursor channel negotiated — host cursor renders locally")
startCursorPump(connection)
reconcileCursorRender() // initial render mode (a capture-model start composites)
}
// Presenter choice + lifecycle live in SessionPresenter (shared with iOS/tvOS): stage-2
// (explicit VTDecompressionSession decode + a CAMetalLayer/display-link present) by
@@ -952,11 +751,6 @@ public final class StreamLayerView: NSView {
matchFollower?.noteSize(
widthPx: Int(px.width.rounded()), heightPx: Int(px.height.rounded()))
}
// The video-fit scale just changed (resize / retina move); rebuild the worn host pointer at
// the new scale so it tracks the video instead of freezing at its build-time size.
if captured, desktopMouse, cursorChannelActive {
window?.invalidateCursorRects(for: self)
}
}
public override func viewDidChangeBackingProperties() {
@@ -987,14 +781,6 @@ public final class StreamLayerView: NSView {
matchFollower = nil
lastDecodedContentSize = nil // the next session re-derives it from its first frame
connection = nil
// Cursor-channel state is per-session: without this reset a next session against a
// host WITHOUT the cap would wear this session's stale shapes (`cursorChannelActive`
// stayed latched true across sessions).
cursorChannelActive = false
cursorState = nil
hostCursors.removeAll()
sentClientDraws = nil
window?.invalidateCursorRects(for: self)
}
deinit {
@@ -333,13 +333,6 @@ public final class StreamViewController: StreamViewControllerBase {
guard self?.captureEnabled == true else { return }
connection?.send(event)
}
// Apple Pencil the stylus plane, only against a pen-capable host (elsewhere the
// Pencil stays a finger, exactly as before). Same trust gate as touch.
streamView.penEnabled = connection.hostSupportsPen
streamView.onPenBatch = { [weak self, weak connection] batch in
guard self?.captureEnabled == true else { return }
connection?.sendPen(batch)
}
// Indirect pointer (mouse/trackpad) WITHOUT a lock absolute cursor + buttons + scroll.
// While the scene is pointer-LOCKED the GCMouse path owns motion AND buttons AND scroll, so
// the whole UIKit indirect path is gated off here (`gcMouseForwarding`). The trackpad and a
@@ -506,8 +499,6 @@ public final class StreamViewController: StreamViewControllerBase {
// onTouchEvent can still deliver the button-up.
streamView.resetTouchInput()
streamView.onTouchEvent = nil
streamView.onPenBatch = nil // after reset the pen's leave-range sample rides it
streamView.penEnabled = false
streamView.onPointerMoveAbs = nil
streamView.onPointerButton = nil
streamView.onScroll = nil
@@ -701,12 +692,6 @@ final class StreamLayerUIView: UIView {
/// Direct fingers / Pencil wire events: real touches in passthrough mode, or the
/// touch-driven mouse events (`TouchMouse`) in the trackpad/pointer modes.
var onTouchEvent: ((PunktfunkInputEvent) -> Void)?
/// Apple Pencil state-full pen sample batches (the stylus plane). Active only while
/// `penEnabled`; without it the Pencil stays on the finger path exactly as before.
var onPenBatch: (([PunktfunkPenSample]) -> Void)?
/// The host advertised `HOST_CAP_PEN`, so Pencil input splits out of the finger path onto
/// the pen plane independent of the touch-input mode (drawing must not depend on it).
var penEnabled = false
/// Indirect pointer (mouse/trackpad with no lock) absolute cursor moves.
var onPointerMoveAbs: ((HostPoint) -> Void)?
/// Indirect-pointer buttons (GameStream ids: 1=left 3=right); `down` = press.
@@ -730,21 +715,10 @@ final class StreamLayerUIView: UIView {
/// The finger route latched at gesture start a Settings change mid-gesture applies to
/// the NEXT touch, so one gesture never splits across input models.
private var fingerRoute: TouchInputMode?
/// The Apple Pencil pipeline (contacts + hover + squeeze/tap pen samples).
private lazy var pencil: PencilStream = {
let stream = PencilStream()
stream.send = { [weak self] batch in self?.onPenBatch?(batch) }
stream.videoNorm = { [weak self] point in
guard let h = self?.hostPoint(from: point) else { return nil }
return (Float(h.x) / Float(max(h.w - 1, 1)), Float(h.y) / Float(max(h.h - 1, 1)))
}
return stream
}()
/// Release anything the touch-driven mouse holds and forget gesture state session stop.
func resetTouchInput() {
touchMouse.reset()
pencil.reset() // leaves range the host lifts anything still inked
fingerRoute = nil
setSoftKeyboardVisible(false) // a stream that's gone takes its keyboard with it
}
@@ -781,11 +755,6 @@ final class StreamLayerUIView: UIView {
scrollPan.allowedScrollTypesMask = .all
scrollPan.allowedTouchTypes = []
addGestureRecognizer(scrollPan)
// Pencil squeeze / double-tap the pen plane's barrel buttons (no-op while
// `penEnabled` is false PencilStream ignores interactions out of range).
let pencilInteraction = UIPencilInteraction()
pencilInteraction.delegate = pencil
addInteraction(pencilInteraction)
#endif
backgroundColor = .black
}
@@ -810,32 +779,17 @@ final class StreamLayerUIView: UIView {
private enum TouchKind { case down, move, up, cancel }
/// Split a touch batch by kind: an INDIRECT POINTER (mouse/trackpad with no lock) drives
/// the host cursor as an absolute mouse; a Pencil goes to the pen plane when the host
/// supports it; everything else (direct finger and the Pencil toward a pen-less host)
/// is a host touch. Mixed batches are possible, so partition rather than branch on the
/// first touch.
/// the host cursor as an absolute mouse; everything else (direct finger, Pencil) is a host
/// touch. Mixed batches are possible, so partition rather than branch on the first touch.
private func route(_ touches: Set<UITouch>, event: UIEvent?, kind: TouchKind) {
var fingers: Set<UITouch> = []
var pencilTouches: Set<UITouch> = []
for touch in touches {
if touch.type == .indirectPointer {
handleIndirectPointer(touch, event: event, kind: kind)
} else if penEnabled, touch.type == .pencil {
pencilTouches.insert(touch)
} else {
fingers.insert(touch)
}
}
if !pencilTouches.isEmpty {
let phase: PencilStream.Phase =
switch kind {
case .down: .down
case .move: .move
case .up: .up
case .cancel: .cancel
}
pencil.touches(pencilTouches, event: event, phase: phase, in: self)
}
if !fingers.isEmpty { forwardFingers(fingers, kind: kind) }
}
@@ -907,11 +861,8 @@ final class StreamLayerUIView: UIView {
}
}
/// Button-less mouse/trackpad movement (no lock) absolute cursor move unless it is a
/// hovering PENCIL (`zOffset > 0`) on a pen-capable host, which becomes in-range pen
/// samples (hover preview with distance/tilt/azimuth) instead of a cursor move.
/// Button-less mouse/trackpad movement (no lock) absolute cursor move.
@objc private func handleHover(_ recognizer: UIHoverGestureRecognizer) {
if penEnabled, pencil.maybeHover(recognizer, in: self) { return }
switch recognizer.state {
case .began, .changed:
if let h = hostPoint(from: recognizer.location(in: self)) { onPointerMoveAbs?(h) }
+3 -33
View File
@@ -10,17 +10,9 @@ import {
showModal,
staticClasses,
} from "@decky/ui";
import { definePlugin, routerHook, toaster } from "@decky/api";
import { definePlugin, routerHook } from "@decky/api";
import { FC } from "react";
import {
FaDownload,
FaLock,
FaLockOpen,
FaPlay,
FaPlus,
FaSyncAlt,
FaTv,
} from "react-icons/fa";
import { FaDownload, FaLock, FaLockOpen, FaPlay, FaSyncAlt, FaTv } from "react-icons/fa";
import { PluginErrorBoundary } from "./boundary";
import {
applyUpdate,
@@ -39,19 +31,7 @@ import {
import { streamPin } from "./library";
import { PunktfunkRoute, ROUTE } from "./page";
import { PairModal } from "./pair";
import { ensureGamepadUiShortcut, recreateShortcuts } from "./steam";
// Recovery action for "the Punktfunk library entry vanished" — recreates the visible shortcut.
// Deleting the shortcut (optionally + reinstalling the plugin) leaves a stale appId in Steam's
// CEF localStorage that self-heal fixes on the next mount, but this gives an in-session button
// that works even without a reload. Always ends in a toast so the tap has feedback.
async function recreatePunktfunkShortcut(): Promise<void> {
const appId = await recreateShortcuts();
toaster.toast({
title: "Punktfunk",
body: appId != null ? "Shortcut restored to your library" : "Couldn't create the shortcut",
});
}
import { ensureGamepadUiShortcut } from "./steam";
// ----------------------------------------------------------------------------------------
// QAM panel — quick status + entry into the full page + one-tap stream for known hosts
@@ -210,16 +190,6 @@ const QamPanel: FC = () => {
{checking ? "Checking…" : "Check for updates"}
</ButtonItem>
</PanelSectionRow>
<PanelSectionRow>
<ButtonItem
layout="below"
description="Missing the Punktfunk entry in your library? This puts it back."
onClick={() => void recreatePunktfunkShortcut()}
>
<FaPlus style={{ marginRight: "0.5em" }} />
Recreate library shortcut
</ButtonItem>
</PanelSectionRow>
</PanelSection>
</>
);
+2 -54
View File
@@ -55,29 +55,6 @@ declare const collectionStore:
| { SetAppsAsHidden?: (appIds: number[], hidden: boolean) => void }
| undefined;
// SteamUI's appStore indexes every registered app/shortcut by appId; a remembered appId whose
// overview is gone was deleted out from under us (the user removed the library entry). We must
// verify this because the remembered appId lives in Steam's CEF localStorage — which survives a
// plugin UNINSTALL/REINSTALL — so a manually-deleted shortcut otherwise leaves a dangling appId
// that the reuse path below silently repoints (SetShortcut* on a dead id is a no-op), and the
// entry never comes back.
declare const appStore:
| { GetAppOverviewByAppID?: (appId: number) => unknown | null }
| undefined;
/** True if a remembered appId still maps to a live Steam shortcut. When appStore is unavailable
* we can't tell, so assume it exists better to keep reusing than risk a duplicate library
* entry from a false "missing". A confident null means the shortcut was deleted recreate. */
function shortcutStillExists(appId: number): boolean {
try {
const get = appStore?.GetAppOverviewByAppID;
if (!get) return true; // no way to verify — preserve the reuse path
return get(appId) != null;
} catch {
return true;
}
}
/** Set a shortcut's library visibility (best-effort, deferred the overview registers a moment
* after AddShortcut). Hides the stateful stream shortcut; keeps the gamepad-UI one visible. */
function setShortcutHidden(appId: number, hidden: boolean): void {
@@ -222,10 +199,8 @@ async function ensureStreamShortcut(): Promise<{ appId: number; runner: string }
const startDir = info.runner.replace(/\/[^/]*$/, ""); // the plugin's bin/ dir
void ensureControllerConfig(); // fire-and-forget — never blocks the launch
// Reuse the remembered shortcut only if it still exists — a stale appId (shortcut deleted, key
// outlived it across a reinstall) must fall through to AddShortcut, not be silently repointed.
const remembered = recall(STORAGE_KEY_STREAM);
if (remembered != null && shortcutStillExists(remembered)) {
if (remembered != null) {
SteamClient.Apps.SetShortcutExe(remembered, SHELL);
SteamClient.Apps.SetShortcutStartDir(remembered, startDir);
SteamClient.Apps.SetShortcutName(remembered, SHORTCUT_NAME);
@@ -261,11 +236,8 @@ export async function ensureGamepadUiShortcut(): Promise<number | null> {
// home). %command% expands to the shortcut exe (/bin/sh); the wrapper rides behind as an arg.
const launchOpts = `PF_BROWSE=1 %command% "${info.runner}"`;
// Reuse the remembered entry only if it still exists; a stale appId (deleted shortcut whose
// localStorage key survived a plugin reinstall) falls through to AddShortcut so the visible
// library entry actually comes back instead of repointing a dead id.
let appId = recall(STORAGE_KEY_UI);
if (appId != null && shortcutStillExists(appId)) {
if (appId != null) {
SteamClient.Apps.SetShortcutExe(appId, SHELL);
SteamClient.Apps.SetShortcutStartDir(appId, startDir);
SteamClient.Apps.SetShortcutName(appId, SHORTCUT_NAME);
@@ -284,30 +256,6 @@ export async function ensureGamepadUiShortcut(): Promise<number | null> {
}
}
/**
* Force the visible "Punktfunk" library entry back into existence the recovery button for
* "my shortcut disappeared". Drops any remembered appId that no longer maps to a live shortcut
* (so it can't shadow a fresh AddShortcut), then re-ensures. Safe to press anytime: a shortcut
* that still exists is left in place (no duplicate); a missing one is recreated. Covers the case
* self-heal-on-mount can't deleting the shortcut WITHOUT reinstalling (no mount no ensure).
* Returns the (new or existing) visible appId, or null on failure.
*/
export async function recreateShortcuts(): Promise<number | null> {
for (const key of [STORAGE_KEY_STREAM, STORAGE_KEY_UI]) {
const id = recall(key);
if (id != null && !shortcutStillExists(id)) {
try {
localStorage.removeItem(artKey(id)); // stale art marker for the dead appId
localStorage.removeItem(key);
} catch {
/* ignore */
}
}
}
// Recreate the visible entry now; the hidden stream shortcut re-registers lazily on next launch.
return ensureGamepadUiShortcut();
}
/** Launch the stateless gamepad-UI shortcut (console home) from the plugin, e.g. a QAM button. */
export async function launchGamepadUi(): Promise<void> {
const appId = await ensureGamepadUiShortcut();
-1
View File
@@ -547,7 +547,6 @@ impl AppModel {
crate::video::decodable_codecs(), // codecs (unused by the probe, but honest)
0, // preferred_codec: no preference
None, // display_hdr: probe connect, nothing presents
0, // client_caps: probe connect, nothing renders a cursor
None, // launch: probe connect, no game
pin,
Some(identity),
-3
View File
@@ -523,9 +523,6 @@ async fn session(args: Args) -> Result<()> {
// writes it into the virtual display's EDID (CTA HDR block), so the EDID-forwarding
// path can be validated headlessly (check the host's monitor caps / ADD log line).
display_hdr: punktfunk_core::client::display_hdr_env_override(),
// No CLIENT_CAP_CURSOR: this headless tool renders nothing — advertising it would
// just strip the pointer from the dumped bitstream.
client_caps: 0,
}
.encode(),
)
-5
View File
@@ -172,11 +172,6 @@ mod session_main {
// defaults for Linux clients; `PUNKTFUNK_CLIENT_PEAK_NITS` (read in the session
// pump) pins one manually.
display_hdr: None,
// The presenter renders the host cursor locally in desktop mouse mode (M2 cursor
// channel); capture-mode sessions keep the composited cursor, so only advertise
// when the session STARTS in desktop mode. The host gates further (Linux portal
// compositors only).
cursor_forward: settings.mouse_mode() == trust::MouseMode::Desktop,
mic_enabled: settings.mic_enabled,
clipboard,
// The Settings preference (auto → VAAPI where it exists; the presenter
-1
View File
@@ -56,7 +56,6 @@ pub fn run_speed_probe(
decodable_codecs(),
0, // preferred_codec: no preference
None, // display_hdr: probe connect, nothing presents
0, // client_caps: probe connect, nothing renders a cursor
None, // launch: no game
pin,
Some(identity),
-8
View File
@@ -30,12 +30,6 @@ pipewire = "0.9"
libc = "0.2"
# ashpd 0.13 uses the tokio runtime for the one-time portal handshake (control plane).
tokio = { version = "1", features = ["rt", "rt-multi-thread", "net", "time"] }
# XFixes cursor source for gamescope (remote-desktop-sweep Phase C): gamescope paints no
# `SPA_META_Cursor`, so the pointer never reaches the PipeWire node. We read the shape/hotspot/
# visibility from gamescope's nested Xwayland via XFixes instead and feed the existing cursor slot.
# `RustConnection` is the pure-Rust default (no libxcb link → no new C dependency on the host); the
# `xfixes` feature (auto-pulls `render` + `shape`) is what exposes GetCursorImage/SelectCursorInput.
x11rb = { version = "0.13", default-features = false, features = ["xfixes"] }
[target.'cfg(target_os = "windows")'.dependencies]
# The host<->driver wire contract for the sealed frame channel (control IOCTL structs + frame header).
@@ -49,9 +43,7 @@ windows = { version = "0.62", features = [
"Win32_Graphics_Direct3D_Fxc",
"Win32_Graphics_Dxgi",
"Win32_Graphics_Dxgi_Common",
"Win32_Graphics_Gdi",
"Win32_System_LibraryLoader",
"Win32_System_StationsAndDesktops",
"Win32_System_Memory",
"Win32_System_Threading",
"Win32_UI_HiDpi",
-54
View File
@@ -69,34 +69,6 @@ pub trait Capturer: Send {
/// SDR / a backend that doesn't expose it (the default — Linux capture has no HDR path yet).
/// The stream loop forwards this to the encoder (in-band SEI) and the client (`0xCE` datagram),
/// so the two stay a single source of truth. May change mid-session if the source is regraded.
/// The capture source's LIVE cursor state, when it arrives out-of-band from the frames
/// (the Windows IddCx hardware-cursor channel). Polled by the encode loop every tick and
/// preferred over `CapturedFrame::cursor` — with a hardware cursor, pointer-only moves
/// produce NO new frame, so the frame-attached overlay would go stale on a static desktop.
/// Default `None`: the Linux portal path attaches its cursor to frames instead.
fn cursor(&mut self) -> Option<pf_frame::CursorOverlay> {
None
}
/// LIVE cursor-render flip for a cursor-forward session (design/remote-desktop-sweep.md §8):
/// `on = true` — the client draws the pointer, keep it OUT of the video; `on = false` —
/// the capture mouse model, the pointer must be IN the video again. The Windows IDD
/// capturer implements the composite side ITSELF (slot-copy + alpha-blended quad from the
/// GDI poller) — a declared IddCx hardware cursor is irrevocable, so DWM can never be
/// handed the job back. Called every encode tick (implementations cache; steady state is
/// one compare). Default no-op: the Linux portal never bakes the pointer into frames —
/// the encode loop blends its overlay instead.
fn set_cursor_forward(&mut self, _on: bool) {}
/// Attach a gamescope cursor source (remote-desktop-sweep Phase C). gamescope paints no
/// `SPA_META_Cursor`, so [`cursor`](Self::cursor)'s slot stays empty — this hands the Linux
/// portal capturer gamescope's nested Xwayland `(DISPLAY, XAUTHORITY)` targets (it may run
/// several — one per `--xwayland-count`) so it reads the pointer shape/position over X11
/// (XFixes + QueryPointer), following whichever display is focused, and publishes it into that
/// same slot. Called once, after the capturer is built, only for gamescope sessions. Default
/// no-op: every non-gamescope capturer already has a cursor source.
fn attach_gamescope_cursor(&mut self, _targets: Vec<(String, Option<String>)>) {}
fn hdr_meta(&self) -> Option<punktfunk_core::quic::HdrMeta> {
None
}
@@ -395,26 +367,6 @@ pub type FrameChannelSender = std::sync::Arc<
dyn Fn(&pf_driver_proto::control::SetFrameChannelRequest) -> Result<()> + Send + Sync,
>;
/// Delivery closure for the v5 hardware-cursor channel (`IOCTL_SET_CURSOR_CHANNEL`) — same
/// facade contract as [`FrameChannelSender`]. `Some` also OPTS THE SESSION IN: the capturer
/// creates + delivers the cursor section only when the host hands it a sender (the negotiated
/// cursor-forward sessions), and the driver only declares the hardware cursor once that
/// delivery lands — so a plain session keeps DWM's composited pointer untouched.
#[cfg(target_os = "windows")]
pub type CursorChannelSender = std::sync::Arc<
dyn Fn(&pf_driver_proto::control::SetCursorChannelRequest) -> Result<()> + Send + Sync,
>;
/// The mid-stream cursor-render flip (`IOCTL_SET_CURSOR_FORWARD`, proto v6) as a host-facade
/// closure — same contract as [`CursorChannelSender`]. `bool` = declare the IddCx hardware
/// cursor (`true`) or stand it down (`false`; the host facade additionally forces the same-mode
/// re-commit that actualises the OS's software-cursor default). The capturer drives this from
/// its secure-desktop watch: UAC/Winlogon render only through the software-cursor path, so a
/// path pinned to the hardware cursor never presents them (the 0.18.0 secure-desktop
/// regression).
#[cfg(target_os = "windows")]
pub type CursorForwardSender = std::sync::Arc<dyn Fn(bool) -> Result<()> + Send + Sync>;
// One-time PipeWire library init, shared by the video (portal) and audio capture threads.
#[cfg(target_os = "linux")]
pub mod pwinit;
@@ -472,7 +424,6 @@ pub fn open_virtual_output(
allow_zerocopy: bool,
want_444: bool,
policy: ZeroCopyPolicy,
expect_exact_dims: bool,
) -> Result<Box<dyn Capturer>> {
linux::PortalCapturer::from_virtual_output(
remote_fd,
@@ -482,7 +433,6 @@ pub fn open_virtual_output(
allow_zerocopy,
want_444,
policy,
expect_exact_dims,
)
.map(|c| Box::new(c) as Box<dyn Capturer>)
}
@@ -500,8 +450,6 @@ pub fn open_idd_push(
pyrowave: bool,
keepalive: Box<dyn Send>,
sender: FrameChannelSender,
cursor_sender: Option<CursorChannelSender>,
cursor_forward: Option<CursorForwardSender>,
) -> std::result::Result<Box<dyn Capturer>, (anyhow::Error, Box<dyn Send>)> {
idd_push::IddPushCapturer::open(
target,
@@ -511,8 +459,6 @@ pub fn open_idd_push(
pyrowave,
keepalive,
sender,
cursor_sender,
cursor_forward,
)
.map(|c| Box::new(c) as Box<dyn Capturer>)
}
+12 -193
View File
@@ -22,10 +22,6 @@
use super::{CapturedFrame, Capturer, DmabufFrame, FramePayload, PixelFormat, ZeroCopyPolicy};
use anyhow::{anyhow, Context, Result};
// gamescope cursor source (remote-desktop-sweep Phase C) — feeds `cursor_live` from XFixes when
// the PipeWire node carries no `SPA_META_Cursor` (gamescope's does not).
mod xfixes_cursor;
use std::os::fd::OwnedFd;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError, TryRecvError};
@@ -61,12 +57,6 @@ pub struct PortalCapturer {
/// renegotiation before declaring the source lost. Cleared whenever a frame arrives or the stream
/// is `Streaming`.
stall_since: Option<std::time::Instant>,
/// The LIVE cursor overlay, published by the PipeWire thread from every buffer's
/// `SPA_META_Cursor` — including the cursor-only "corrupted" buffers that never become
/// frames. [`Capturer::cursor`] serves it so the encode loop's forwarder tracks pointer-only
/// motion on a static desktop; the frame-attached overlay alone goes stale between damage
/// frames (the same gap the Windows IddCx channel fills, and why the tick prefers LIVE).
cursor_live: Arc<std::sync::Mutex<Option<pf_frame::CursorOverlay>>>,
/// True when this capture runs the VAAPI dmabuf passthrough (a LINEAR-dmabuf-only offer). If
/// that offer never negotiates, [`next_frame`](Capturer::next_frame)'s timeout branch latches
/// the process-wide downgrade ([`pf_zerocopy::note_vaapi_dmabuf_failed`]) so the pipeline
@@ -92,12 +82,6 @@ pub struct PortalCapturer {
/// is, releasing the compositor-side output via the keepalive's own `Drop`. `None` for the
/// portal source (its session ends with the portal thread's zbus connection).
_keepalive: Option<Box<dyn Send>>,
/// The gamescope XFixes cursor reader (remote-desktop-sweep Phase C), when this capturer
/// serves a gamescope node. `Some` after
/// [`attach_gamescope_cursor`](Capturer::attach_gamescope_cursor); its `Drop` stops the reader
/// thread, so it lives exactly as long as the capturer. `None` on the portal path (its cursor
/// comes from `SPA_META_Cursor`).
_gs_cursor: Option<xfixes_cursor::XFixesCursorSource>,
}
impl PortalCapturer {
@@ -130,17 +114,10 @@ impl PortalCapturer {
"ScreenCast portal session started; connecting PipeWire"
);
// This portal path (GameStream / monitor capture) is always 4:2:0, so allow zero-copy as before.
Ok(spawn_pipewire(
Some(fd),
node_id,
None,
true,
false,
want_hdr,
policy,
false,
)?
.into_capturer(node_id, None))
Ok(
spawn_pipewire(Some(fd), node_id, None, true, false, want_hdr, policy)?
.into_capturer(node_id, None),
)
}
/// Build a capturer from an already-created virtual output's PipeWire node. The host facade
@@ -160,13 +137,11 @@ impl PortalCapturer {
allow_zerocopy: bool,
want_444: bool,
policy: ZeroCopyPolicy,
expect_exact_dims: bool,
) -> Result<PortalCapturer> {
tracing::info!(
node_id,
allow_zerocopy,
want_444,
expect_exact_dims,
"connecting PipeWire to virtual output"
);
// Virtual outputs are SDR-only upstream (Mutter's RecordVirtual streams advertise 8-bit
@@ -179,7 +154,6 @@ impl PortalCapturer {
want_444,
false,
policy,
expect_exact_dims,
)?
.into_capturer(node_id, Some(keepalive)))
}
@@ -202,8 +176,6 @@ struct PwHandles {
hdr_offer: bool,
/// See [`PortalCapturer::hdr_negotiated`].
hdr_negotiated: Arc<AtomicBool>,
/// See [`PortalCapturer::cursor_live`].
cursor_live: Arc<std::sync::Mutex<Option<pf_frame::CursorOverlay>>>,
quit: ::pipewire::channel::Sender<()>,
join: thread::JoinHandle<()>,
}
@@ -223,12 +195,10 @@ impl PwHandles {
vaapi_dmabuf: self.vaapi_dmabuf,
hdr_offer: self.hdr_offer,
hdr_negotiated: self.hdr_negotiated,
cursor_live: self.cursor_live,
node_id,
quit: Some(self.quit),
join: Some(self.join),
_keepalive: keepalive,
_gs_cursor: None,
}
}
}
@@ -236,7 +206,6 @@ impl PwHandles {
/// Spawn the PipeWire consumer thread for `node_id` (fd `Some` = portal remote, `None` =
/// default daemon) and return its [`PwHandles`]. `preferred` seeds the format negotiation's
/// default size/framerate — for Mutter virtual monitors this is what actually sizes the monitor.
#[allow(clippy::too_many_arguments)]
fn spawn_pipewire(
fd: Option<OwnedFd>,
node_id: u32,
@@ -255,12 +224,6 @@ fn spawn_pipewire(
// Encode-backend facts resolved by the facade (never re-derived here) — the one-way
// capture→encode edge (plan §W6).
policy: ZeroCopyPolicy,
// The producer's FIRST negotiation is for a sacrificial mode and a renegotiation to
// `preferred`'s dims is guaranteed to follow (KWin virtual outputs — see kwin.rs `create`):
// skip whole buffers until the negotiated size matches, so the pipeline never builds against
// the doomed birth mode. `false` everywhere else (Mutter SIZES the monitor from negotiation,
// gamescope fixates its own — gating those would starve legitimate first frames).
expect_exact_dims: bool,
) -> Result<PwHandles> {
// Frames flow from the pipewire thread over a small bounded channel.
let (frame_tx, frame_rx) = sync_channel::<CapturedFrame>(8);
@@ -274,8 +237,6 @@ fn spawn_pipewire(
let broken_cb = broken.clone();
let hdr_negotiated = Arc::new(AtomicBool::new(false));
let hdr_negotiated_cb = hdr_negotiated.clone();
let cursor_live = Arc::new(std::sync::Mutex::new(None::<pf_frame::CursorOverlay>));
let cursor_live_cb = cursor_live.clone();
// pipewire's own cross-thread channel: the receiver attaches to the loop and quits it; the
// sender lives on the capturer and fires in its `Drop`. Absolute `::pipewire` path — the
// inner `mod pipewire` shadows the crate name at this scope.
@@ -311,14 +272,12 @@ fn spawn_pipewire(
streaming_cb,
broken_cb,
hdr_negotiated_cb,
cursor_live_cb,
zerocopy,
want_444,
want_hdr,
preferred,
quit_rx,
policy,
expect_exact_dims,
) {
tracing::error!(error = %format!("{e:#}"), "pipewire capture thread failed");
}
@@ -333,7 +292,6 @@ fn spawn_pipewire(
vaapi_dmabuf,
hdr_offer: want_hdr,
hdr_negotiated,
cursor_live,
quit: quit_tx,
join,
})
@@ -344,24 +302,6 @@ impl Capturer for PortalCapturer {
self.frame_within(Duration::from_secs(10))
}
fn cursor(&mut self) -> Option<pf_frame::CursorOverlay> {
// The PipeWire thread's live cursor slot (fed by every buffer's meta, frames or not) —
// lets the forwarder track pointer-only motion on a static desktop. See `cursor_live`.
// On a gamescope node the meta never arrives; the XFixes source (attached below) fills
// the same slot instead.
self.cursor_live.lock().ok().and_then(|slot| slot.clone())
}
fn attach_gamescope_cursor(&mut self, targets: Vec<(String, Option<String>)>) {
// gamescope paints no `SPA_META_Cursor`, so `cursor_live` would stay empty. Spawn the
// XFixes reader to publish gamescope's pointer into that SAME slot — `cursor()` above then
// serves it and the encode loop composites it, exactly like the portal path. It connects
// to every nested Xwayland and follows the focused one's pointer. A failure (no Xwayland /
// no XFixes) logs and leaves the slot empty = today's cursorless gamescope.
self._gs_cursor =
xfixes_cursor::XFixesCursorSource::spawn(targets, Arc::clone(&self.cursor_live));
}
fn next_frame_within(&mut self, budget: Duration) -> Result<CapturedFrame> {
self.frame_within(budget)
}
@@ -928,21 +868,13 @@ mod pipewire {
/// Bumps whenever the bitmap (`rgba`/`bw`/`bh`) changes — stable across position-only moves,
/// so the GPU encoder re-uploads its cursor texture only on change.
serial: u64,
/// The compositor-reported hotspot — carried on the overlay for the cursor-forward
/// channel (the blend path uses the pre-adjusted `x`/`y` and never reads it).
hot_x: i32,
hot_y: i32,
}
impl CursorState {
/// A shareable overlay for the encode/forward paths, or `None` before the first bitmap
/// arrived. A HIDDEN pointer still yields `Some` (with `visible: false`): the
/// cursor-forward channel needs "known but hidden" — an app grabbed the pointer, the
/// client's relative-mode hint (M3) — which is a different fact from "no cursor yet".
/// The encode loop strips invisible overlays before any blend path sees the frame.
/// Cheap: clones an `Arc` + a few scalars.
/// A shareable overlay for the GPU encode paths (blended at encode time), or `None` when
/// there is nothing to draw. Cheap: clones an `Arc` + a few scalars.
fn overlay(&self) -> Option<pf_frame::CursorOverlay> {
if self.rgba.is_empty() {
if !self.visible || self.rgba.is_empty() {
return None;
}
Some(pf_frame::CursorOverlay {
@@ -952,9 +884,6 @@ mod pipewire {
h: self.bh,
rgba: self.rgba.clone(),
serial: self.serial,
hot_x: self.hot_x.max(0) as u32,
hot_y: self.hot_y.max(0) as u32,
visible: self.visible,
})
}
}
@@ -1005,21 +934,6 @@ mod pipewire {
dbg_log_n: u64,
/// Cursor-as-metadata state, composited into the CPU de-pad path (see `consume_frame`).
cursor: CursorState,
/// LIVE overlay slot shared with [`super::PortalCapturer::cursor_live`] — refreshed after
/// every `update_cursor_meta`, including from cursor-only buffers that never become frames.
cursor_live: Arc<std::sync::Mutex<Option<pf_frame::CursorOverlay>>>,
/// `Some((w, h))` while the producer's negotiated size is a sacrificial birth mode and a
/// renegotiation to these dims is guaranteed (KWin virtual outputs — kwin.rs `create`):
/// `.process` skips whole buffers until the negotiated size matches, then clears this
/// (self-disarming — later legitimate resizes are unaffected). `None` = no gating.
expect_dims: Option<(u32, u32)>,
/// Buffers skipped by the `expect_dims` gate (rate-limits its log).
gate_skips: u64,
/// When the gate first held a buffer — after [`GATE_DEADLINE`] with no renegotiation the
/// gate disarms and accepts what the producer serves (degraded dims beat a session wedged
/// into the first-frame-timeout retry loop; the promised renegotiation normally lands
/// within a frame or two).
gate_since: Option<std::time::Instant>,
}
/// Consecutive tiled-import failures (worker alive, e.g. a per-buffer `EGL_BAD_MATCH`) before
@@ -1398,17 +1312,10 @@ mod pipewire {
value: pw::spa::pod::Value::Choice(pw::spa::pod::ChoiceValue::Int(
pw::spa::utils::Choice(
pw::spa::utils::ChoiceFlags::empty(),
// The max must cover the producer's offer or the Meta param silently
// fails to negotiate and NO buffer ever carries the meta region:
// Mutter offers a FIXED `SPA_POD_Int(CURSOR_META_SIZE(384, 384))`
// (meta-screen-cast-stream-src.c, GNOME 50) — a 256² max made the
// intersection empty, which cost the whole Linux cursor channel
// on-glass. 1024² is headroom, not an allocation: the negotiated
// region follows the producer's value.
pw::spa::utils::ChoiceEnum::Range {
default: meta_size(64, 64),
min: meta_size(1, 1),
max: meta_size(1024, 1024),
max: meta_size(256, 256),
},
),
)),
@@ -1466,19 +1373,13 @@ mod pipewire {
)
};
if id == 0 {
// SPA contract: id 0 = "no cursor information", NOT "cursor hidden". Mutter only
// REWRITES a buffer's meta region when the cursor changed, so recycled buffers
// between damage frames carry a stale id-0 meta — treating that as hidden flickered
// the cursor off between hovers (on-glass round 5). Keep the last-known state; a
// pointer that really left/hid simply stops producing updates. (The M3 hidden hint
// loses its Mutter signal — Windows has its own CURSOR_SUPPRESSED source.)
// Compositor reports no visible pointer (e.g. a game grabbed/hid it).
cursor.visible = false;
return;
}
cursor.visible = true;
cursor.x = pos_x - hot_x;
cursor.y = pos_y - hot_y;
cursor.hot_x = hot_x;
cursor.hot_y = hot_y;
if bmp_off == 0 {
// Position-only update — keep the cached bitmap.
return;
@@ -1504,9 +1405,8 @@ mod pipewire {
(*bmp).offset as usize,
)
};
// Ignore empty or implausibly large bitmaps (the meta-size request covers <= 1024×1024;
// real cursors are ≤96px — the cursor channel downscales >120px for the wire anyway).
if bw == 0 || bh == 0 || bw > 1024 || bh > 1024 {
// Ignore empty or implausibly large bitmaps (we requested <= 256×256).
if bw == 0 || bh == 0 || bw > 256 || bh > 256 {
return;
}
let row = bw as usize * 4;
@@ -2106,9 +2006,6 @@ mod pipewire {
streaming: Arc<AtomicBool>,
broken: Arc<AtomicBool>,
hdr_negotiated: Arc<AtomicBool>,
// LIVE cursor publisher (see `PortalCapturer::cursor_live`): refreshed from every
// dequeued buffer's cursor meta, frames or not.
cursor_live: Arc<std::sync::Mutex<Option<pf_frame::CursorOverlay>>>,
zerocopy: bool,
// 4:4:4 session: tiled dmabufs take the worker's planar-YUV444 GPU convert.
want_444: bool,
@@ -2120,9 +2017,6 @@ mod pipewire {
// Encode-backend facts resolved by the facade (never re-derived here) — the one-way
// capture→encode edge (plan §W6).
policy: ZeroCopyPolicy,
// See `spawn_pipewire`: the first negotiation is for a sacrificial mode; hold frames
// until the producer renegotiates to `preferred`'s dims.
expect_exact_dims: bool,
) -> Result<()> {
crate::pwinit::ensure_init();
@@ -2307,14 +2201,6 @@ mod pipewire {
linear_nv12_failed: false,
dbg_log_n: 0,
cursor: CursorState::default(),
cursor_live,
expect_dims: if expect_exact_dims {
preferred.map(|(w, h, _)| (w, h))
} else {
None
},
gate_skips: 0,
gate_since: None,
};
let stream = pw::stream::StreamBox::new(
@@ -2426,60 +2312,6 @@ mod pipewire {
newest = next;
drained += 1;
}
// Sacrificial-mode gate (kwin.rs `create`): until the producer renegotiates to the
// expected dims, every buffer — frame AND cursor meta, whose positions are in the
// doomed mode's space — belongs to the birth mode; consuming one would build the
// pipeline at the wrong size. Self-disarms on the first matching negotiation, or
// after `GATE_DEADLINE` without one — degraded dims beat wedging the session into
// the first-frame-timeout retry loop when the promised renegotiation never comes.
if let Some((ew, eh)) = ud.expect_dims {
/// The renegotiation normally lands within a frame or two of recording; well
/// past that, the producer is not going to deliver it (the on-glass case: the
/// real mode never actually applied) — stop starving the pipeline.
const GATE_DEADLINE: std::time::Duration = std::time::Duration::from_secs(3);
let sz = ud.info.size();
if sz.width == ew && sz.height == eh {
tracing::info!(
skipped = ud.gate_skips,
width = ew,
height = eh,
"producer renegotiated to the expected mode — frames flow"
);
ud.expect_dims = None;
} else if ud
.gate_since
.get_or_insert_with(std::time::Instant::now)
.elapsed()
> GATE_DEADLINE
{
tracing::warn!(
negotiated_w = sz.width,
negotiated_h = sz.height,
expected_w = ew,
expected_h = eh,
skipped = ud.gate_skips,
"producer never renegotiated to the expected mode — accepting its \
dims (session runs degraded rather than wedged)"
);
ud.expect_dims = None;
} else {
ud.gate_skips += 1;
if ud.gate_skips == 1 || ud.gate_skips.is_power_of_two() {
tracing::info!(
negotiated_w = sz.width,
negotiated_h = sz.height,
expected_w = ew,
expected_h = eh,
n = ud.gate_skips,
"holding frames until the producer renegotiates to the expected mode"
);
}
// SAFETY: `newest` was dequeued from this stream and not yet requeued;
// requeued exactly once here, then never touched (mirrors the null path).
unsafe { stream.queue_raw_buffer(newest) };
return;
}
}
// PipeWire dispatches from a C trampoline with no catch_unwind; a panic crossing that FFI
// boundary would abort the whole host. Contain the inspect/consume work — the only Rust
// code here that can panic — and requeue `newest` unconditionally after it.
@@ -2493,19 +2325,6 @@ mod pipewire {
// pointer-only movements as metadata-only "corrupted" buffers we drop for their
// frame, but their cursor meta is fresh and must still move our overlay.
update_cursor_meta(&mut ud.cursor, spa_buf);
// Publish the LIVE overlay (frames or not) so the encode loop's forwarder
// tracks pointer-only motion on a static desktop — the frame-attached overlay
// alone stales between damage frames. ONLY when we actually have one: a
// gamescope node carries no `SPA_META_Cursor`, so `overlay()` is always `None`
// here, and writing that would clobber — at frame rate — the `Some` the
// attached XFixes source publishes into this SAME slot, strobing the
// composited pointer on/off. Portal cursors are `None` only before the first
// bitmap (nothing to drop), and a HIDDEN pointer is still `Some(visible:false)`.
if let Some(overlay) = ud.cursor.overlay() {
if let Ok(mut slot) = ud.cursor_live.lock() {
*slot = Some(overlay);
}
}
// Inspect the newest buffer's header + first chunk for the diagnostic and the
// CORRUPTED skip. SPA_META_Header is optional — `hdr` may be null.
@@ -1,366 +0,0 @@
//! XFixes cursor source for the gamescope capture path (remote-desktop-sweep Phase C).
//!
//! gamescope draws the pointer on a DRM hardware-cursor plane and its `paint_pipewire()`
//! deliberately excludes the cursor from the frame it feeds its built-in PipeWire node — so
//! `SPA_META_Cursor` never arrives and the ordinary [`cursor_live`](super::PortalCapturer) slot
//! stays empty (a KWin/GNOME session gets its cursor from that meta; gamescope can't embed one
//! either, its `set_hw_cursor` is inert). We instead read the pointer from gamescope's nested
//! Xwayland via X11 — the trick Sunshine uses — and publish a [`CursorOverlay`] into that same
//! slot, so the encoder blend composites the pointer into the video exactly like the portal path.
//!
//! **Multiple Xwaylands.** gamescope runs one Xwayland per `--xwayland-count` (Steam Gaming Mode
//! uses 2: one for Big Picture, one for the game). The pointer lives on whichever is FOCUSED — an
//! inactive display's pointer is frozen. So the source connects to ALL of them and each tick
//! follows the one whose pointer actually moved (gamescope routes input to the focused surface, so
//! exactly one moves at a time). It reads that display's shape too, since each Xwayland has its own
//! current cursor. This is why a single-display read froze the pointer the moment a game on the
//! OTHER Xwayland took focus.
//!
//! Two X sources per display, split by cost (Sunshine's split):
//! * **Position** — core `QueryPointer` on the root, polled fast. Cheap (a few-byte reply, no
//! bitmap), so it can out-pace the stream fps and keep the composited pointer smooth. It also
//! doubles as the focus signal (the display whose pointer moves is the active one).
//! * **Shape / hotspot / visibility** — `XFixesGetCursorImage`, refreshed only after an XFixes
//! `CursorNotify` (a real cursor change). A game hiding the pointer IS a cursor change → the
//! image comes back fully transparent → `visible: false`, which the encode loop strips before
//! any blend path draws it (so a grabbed pointer shows nothing, matching native gamescope).
use std::sync::{
atomic::{AtomicBool, Ordering},
Arc, Mutex,
};
use std::time::Duration;
use pf_frame::CursorOverlay;
use x11rb::connection::Connection;
use x11rb::errors::ReplyError;
use x11rb::protocol::xfixes::{self, ConnectionExt as _, GetCursorImageReply};
use x11rb::protocol::xproto::{ConnectionExt as _, QueryPointerReply, Window};
use x11rb::rust_connection::RustConnection;
/// Serializes the brief `XAUTHORITY` env swap around a connect (the var is process-global). Only
/// ever contended if two gamescope sessions start at once — rare, and the swap is microseconds.
static XAUTH_LOCK: Mutex<()> = Mutex::new(());
/// Position out-paces the stream fps (`POLL`); shape rides `CursorNotify` events drained each tick.
/// 4 ms ≈ 250 Hz matches the Windows GDI poller — the polled position IS the composited position
/// and must out-run a 240 fps session or the pointer stutters.
const POLL: Duration = Duration::from_millis(4);
/// A running XFixes cursor reader. Dropping it stops the worker thread and joins it, releasing the
/// X connections — so it lives exactly as long as the capturer that owns it.
pub(super) struct XFixesCursorSource {
stop: Arc<AtomicBool>,
join: Option<std::thread::JoinHandle<()>>,
}
impl XFixesCursorSource {
/// Connect to every gamescope nested Xwayland in `targets` (`(DISPLAY, XAUTHORITY)`) and start
/// publishing cursor overlays into `slot`, following the focused display's pointer. Returns
/// `None` — and logs — if NONE can be used (no X connection / no XFixes), so the caller
/// degrades to no gamescope cursor (today's behaviour) instead of failing the session.
pub(super) fn spawn(
targets: Vec<(String, Option<String>)>,
slot: Arc<Mutex<Option<CursorOverlay>>>,
) -> Option<Self> {
// Connect on the caller's thread so failures degrade cleanly and the displays are validated
// before we commit a thread.
let mut displays = Vec::new();
for (dpy, xauth) in targets {
match connect(&dpy, xauth.as_deref()) {
Ok((conn, root)) => displays.push(XDisplay::new(dpy, conn, root)),
Err(e) => tracing::warn!(
dpy = %dpy,
error = %e,
"gamescope cursor: skipping a nested Xwayland we can't use"
),
}
}
if displays.is_empty() {
tracing::warn!(
"gamescope cursor: no usable nested Xwayland — no in-video pointer this session \
(falls back to today's cursorless gamescope stream)"
);
return None;
}
let names: Vec<&str> = displays.iter().map(|d| d.name.as_str()).collect();
tracing::info!(
displays = ?names,
"gamescope cursor: XFixes source live — following the focused Xwayland's pointer"
);
let stop = Arc::new(AtomicBool::new(false));
let stop_worker = Arc::clone(&stop);
let join = std::thread::Builder::new()
.name("pf-gs-cursor".into())
.spawn(move || run(displays, slot, stop_worker))
.ok()?;
Some(XFixesCursorSource {
stop,
join: Some(join),
})
}
}
impl Drop for XFixesCursorSource {
fn drop(&mut self) {
self.stop.store(true, Ordering::Relaxed);
if let Some(j) = self.join.take() {
let _ = j.join();
}
}
}
/// Open the X connection, negotiate XFixes, and select cursor-change events — returning the
/// connection + root window. `RustConnection` reads `XAUTHORITY` from the env at connect time only,
/// so set it under the lock (the host isn't a gamescope child), connect, then restore.
fn connect(dpy: &str, xauthority: Option<&str>) -> Result<(RustConnection, Window), String> {
let (conn, screen_num) = {
let _g = XAUTH_LOCK.lock().unwrap_or_else(|e| e.into_inner());
let prev = std::env::var_os("XAUTHORITY");
if let Some(x) = xauthority {
std::env::set_var("XAUTHORITY", x);
}
let out = RustConnection::connect(Some(dpy));
match (&prev, xauthority) {
(Some(p), _) => std::env::set_var("XAUTHORITY", p),
(None, Some(_)) => std::env::remove_var("XAUTHORITY"),
(None, None) => {}
}
out.map_err(|e| format!("connect: {e}"))?
};
// XFixes ≥ 1 gives GetCursorImage / SelectCursorInput; ask for a modern minor, take what we get.
conn.xfixes_query_version(5, 0)
.map_err(ReplyError::from)
.and_then(|c| c.reply())
.map_err(|e| format!("XFixes unavailable: {e}"))?;
let root = conn
.setup()
.roots
.get(screen_num)
.ok_or_else(|| format!("no X screen {screen_num}"))?
.root;
// Wake the worker's event drain whenever the cursor shape changes (incl. hide/show).
conn.xfixes_select_cursor_input(root, xfixes::CursorNotifyMask::DISPLAY_CURSOR)
.map_err(ReplyError::from)
.and_then(|c| c.check())
.map_err(|e| format!("SelectCursorInput: {e}"))?;
let _ = conn.flush();
Ok((conn, root))
}
/// One gamescope Xwayland the source tracks.
struct XDisplay {
name: String,
conn: RustConnection,
root: Window,
/// Last polled pointer position — a change since the previous tick marks this display FOCUSED.
last_pos: Option<(i32, i32)>,
/// Cached cursor shape, refreshed only after this display's XFixes `CursorNotify`.
shape: Shape,
/// A `CursorNotify` (or first read) is pending — fetch the shape when this display is active.
need_shape: bool,
/// The X connection died (game/Xwayland exited) — skip it.
dead: bool,
}
impl XDisplay {
fn new(name: String, conn: RustConnection, root: Window) -> Self {
XDisplay {
name,
conn,
root,
last_pos: None,
shape: Shape::default(),
need_shape: true,
dead: false,
}
}
}
/// Cached cursor shape for one display.
#[derive(Default)]
struct Shape {
/// Straight-alpha RGBA (`w*h*4`, bytes R,G,B,A); empty before the first image arrives.
rgba: Arc<Vec<u8>>,
w: u32,
h: u32,
hot_x: u32,
hot_y: u32,
/// XFixes' own per-display cursor serial — bumps on every shape change.
serial: u64,
/// A hidden pointer arrives as an all-transparent image; kept so a position-only tick preserves
/// the last known visibility.
visible: bool,
}
fn run(
mut displays: Vec<XDisplay>,
slot: Arc<Mutex<Option<CursorOverlay>>>,
stop: Arc<AtomicBool>,
) {
let mut active = 0usize;
// The overlay serial must bump whenever the DRAWN cursor changes — either the active display's
// shape OR which display is active (per-display XFixes serials aren't comparable across
// displays, so switching could reuse a number and the encoder would keep the old texture).
let mut out_serial = 0u64;
let mut last_key = (usize::MAX, u64::MAX);
let mut warned_image = false;
while !stop.load(Ordering::Relaxed) {
// 1) Poll every display's pointer; note which moved since last tick (the focus signal).
let mut active_moved = false;
let mut other_moved: Option<usize> = None;
for (i, d) in displays.iter_mut().enumerate() {
if d.dead {
continue;
}
// Drain pending events; the only ones selected are CursorNotify, so ANY event means
// "re-read this display's shape". poll_for_event never blocks.
loop {
match d.conn.poll_for_event() {
Ok(Some(_)) => d.need_shape = true,
Ok(None) => break,
Err(_) => {
d.dead = true;
break;
}
}
}
match fetch_pointer(&d.conn, d.root) {
Ok(p) if p.same_screen => {
let pos = (i32::from(p.root_x), i32::from(p.root_y));
let moved = d.last_pos.is_some_and(|lp| lp != pos);
d.last_pos = Some(pos);
if moved {
if i == active {
active_moved = true;
} else if other_moved.is_none() {
other_moved = Some(i);
}
}
}
Ok(_) => {} // pointer on another screen — keep the last position.
Err(_) => d.dead = true,
}
}
// 2) Switch focus: sticky to the active display while it moves (no flapping); otherwise
// follow another display that moved. If the active one died, fall to any live display.
if !active_moved {
if let Some(j) = other_moved {
active = j;
}
}
if displays.get(active).is_none_or(|d| d.dead) {
match displays.iter().position(|d| !d.dead) {
Some(k) => active = k,
None => {
std::thread::sleep(POLL); // all connections dead — idle until Drop.
continue;
}
}
}
// 3) Fetch the active display's shape if a CursorNotify (or a focus switch) left it stale.
if displays[active].need_shape {
match fetch_cursor_image(&displays[active].conn) {
Ok(img) => {
update_shape(&mut displays[active].shape, &img);
displays[active].need_shape = false;
}
Err(e) => {
if !warned_image {
warned_image = true;
tracing::warn!(error = %e, "gamescope cursor: GetCursorImage failed — retrying");
}
}
}
}
// 4) Publish the ACTIVE display's pointer + shape (or clear the slot when it has no cursor
// of its own — so a focus switch never leaves the other display's stale pointer showing).
let d = &displays[active];
let overlay = match (d.last_pos, d.shape.rgba.is_empty()) {
(Some((px, py)), false) => {
let key = (active, d.shape.serial);
if key != last_key {
out_serial += 1;
last_key = key;
}
Some(CursorOverlay {
// Top-left = pointer position hotspot (the overlay contract).
x: px - d.shape.hot_x as i32,
y: py - d.shape.hot_y as i32,
w: d.shape.w,
h: d.shape.h,
rgba: Arc::clone(&d.shape.rgba),
serial: out_serial,
hot_x: d.shape.hot_x,
hot_y: d.shape.hot_y,
visible: d.shape.visible,
})
}
_ => None,
};
if let Ok(mut s) = slot.lock() {
*s = overlay;
}
std::thread::sleep(POLL);
}
}
/// Update `shape` from a fresh `GetCursorImage` reply. A hidden pointer (all-transparent) keeps the
/// last bitmap (instant re-show) but flips visibility; the serial still bumps so the change shows.
fn update_shape(shape: &mut Shape, img: &GetCursorImageReply) {
let visible =
img.width > 0 && img.height > 0 && img.cursor_image.iter().any(|&p| (p >> 24) & 0xff != 0);
if visible {
shape.rgba = Arc::new(argb_premul_to_straight_rgba(&img.cursor_image));
shape.w = u32::from(img.width);
shape.h = u32::from(img.height);
shape.hot_x = u32::from(img.xhot);
shape.hot_y = u32::from(img.yhot);
}
shape.visible = visible;
shape.serial = u64::from(img.cursor_serial);
}
/// One request+reply — x11rb splits errors (the request is `ConnectionError`, `reply()` is
/// `ReplyError` which is `From<ConnectionError>`), so the request `?` converts into the reply error.
fn fetch_cursor_image(conn: &RustConnection) -> Result<GetCursorImageReply, ReplyError> {
conn.xfixes_get_cursor_image()?.reply()
}
fn fetch_pointer(conn: &RustConnection, root: Window) -> Result<QueryPointerReply, ReplyError> {
conn.query_pointer(root)?.reply()
}
/// XFixes cursor pixels are packed `0xAARRGGBB` with **premultiplied** alpha (the Xrender / Xcursor
/// convention). The overlay + both blend paths want **straight** alpha RGBA (R,G,B,A bytes), like
/// the `SPA_META_Cursor` path — so un-premultiply here. (If on-glass shows over-bright fringes the
/// source wasn't premultiplied after all; drop the divide.)
fn argb_premul_to_straight_rgba(argb: &[u32]) -> Vec<u8> {
let mut out = Vec::with_capacity(argb.len() * 4);
for &px in argb {
let a = (px >> 24) & 0xff;
let r = (px >> 16) & 0xff;
let g = (px >> 8) & 0xff;
let b = px & 0xff;
let (r, g, b) = match a {
0 => (0, 0, 0),
255 => (r, g, b),
a => (
((r * 255 + a / 2) / a).min(255),
((g * 255 + a / 2) / a).min(255),
((b * 255 + a / 2) / a).min(255),
),
};
out.extend_from_slice(&[r as u8, g as u8, b as u8, a as u8]);
}
out
}
+2 -2
View File
@@ -161,7 +161,7 @@ pub fn install_gpu_pref_hook() {
});
}
pub(crate) unsafe fn compile_shader(src: &str, entry: PCSTR, target: PCSTR) -> Result<Vec<u8>> {
unsafe fn compile_shader(src: &str, entry: PCSTR, target: PCSTR) -> Result<Vec<u8>> {
let mut blob: Option<ID3DBlob> = None;
let mut errs: Option<ID3DBlob> = None;
let r = D3DCompile(
@@ -194,7 +194,7 @@ pub(crate) unsafe fn compile_shader(src: &str, entry: PCSTR, target: PCSTR) -> R
}
/// Fullscreen-triangle vertex shader for the HDR conversion pass (3 verts, no input layout).
pub(crate) const HDR_VS: &str = r"
const HDR_VS: &str = r"
struct VOut { float4 pos : SV_POSITION; float2 uv : TEXCOORD0; };
VOut main(uint vid : SV_VertexID) {
float2 uv = float2((vid << 1) & 2, vid & 2);
+22 -494
View File
@@ -365,12 +365,6 @@ pub unsafe fn verify_is_wudfhost(process: HANDLE, wudf_pid: u32, what: &str) ->
#[path = "idd_push/channel.rs"]
mod channel;
#[path = "idd_push/cursor.rs"]
mod cursor;
#[path = "idd_push/cursor_blend.rs"]
mod cursor_blend;
#[path = "idd_push/cursor_poll.rs"]
mod cursor_poll;
#[path = "idd_push/descriptor.rs"]
mod descriptor;
#[path = "idd_push/stall.rs"]
@@ -392,64 +386,6 @@ pub struct IddPushCapturer {
/// The sealed channel's handle-duplication broker (WUDFHost process + control device); used at open
/// and again on every ring recreate to deliver fresh duplicates.
broker: ChannelBroker,
/// The v5 hardware-cursor channel's host end (`Some` = delivered; the driver declared the
/// hardware cursor and seqlock-publishes into it). Survives ring recreates — the section is
/// independent of the frame ring's generation. With the channel delivered, the driver's
/// hardware cursor keeps DWM from compositing ANY cursor into the frame; the SHAPE now comes
/// from [`cursor_poll::CursorPoller`] (the IddCx query is alpha-only — see cursor_poll.rs),
/// and this shm read is the fallback if that poller dies.
cursor_shared: Option<cursor::CursorShared>,
/// The GDI cursor-shape poller (design §8): the overlay source while alive. `Some` when
/// `cursor_shared` is (both ride the negotiated cursor channel + successful delivery) — or
/// when `composite_forced` (no channel, but the target's sticky declare needs a blend source).
cursor_poll: Option<cursor_poll::CursorPoller>,
/// Retained delivery sender (`IOCTL_SET_CURSOR_CHANNEL`) for RE-delivery: a driver-side
/// monitor re-arrival (match-window re-arrival resize, a sibling session recreating the
/// shared slot) destroys the driver's cursor worker — the section here survives, so the
/// channel is re-delivered on ring recreates.
cursor_sender: Option<crate::CursorChannelSender>,
/// The cursor-render flip sender (`IOCTL_SET_CURSOR_FORWARD`) — the secure-desktop guard's
/// actuator. UAC/Winlogon render only through the OS's software-cursor path (its default on
/// every mode commit); with our hardware cursor declared (and re-declared on every
/// swap-chain assign) that path never comes back, and the secure desktop never presents —
/// the stream freezes on the last normal-desktop frame for the whole UAC/lock interaction.
/// [`Self::poll_secure_desktop`] flips the declare off/on at the secure-desktop edges.
cursor_forward: Option<crate::CursorForwardSender>,
/// The secure-desktop guard's edge state: `true` = the poller reports a secure input
/// desktop and the declare is currently stood down.
secure_active: bool,
/// The CAPTURE mouse model is active — the HOST composites the pointer into the frame
/// (see cursor_blend.rs for why DWM cannot: a declared IddCx hardware cursor is forever).
composite_cursor: bool,
/// This session never negotiated the cursor channel but its target carries an IRREVOCABLE
/// hardware-cursor declare from an earlier session (`WinCaptureTarget::cursor_excluded`,
/// §8.6): DWM delivers pointer-free frames and no client draws the cursor, so the ONLY path
/// to a visible pointer is compositing here. Pins `composite_cursor` on — nothing may turn
/// it off (there is no channel to hand the pointer to).
composite_forced: bool,
/// The cursor-quad blend pass (lazy; per capture device). `None` after a build failure —
/// composite mode then degrades to pointer-less frames (warned once).
cursor_blend: Option<cursor_blend::CursorBlendPass>,
cursor_blend_failed: bool,
/// The frame-sized blend scratch (slot copy + cursor quad): texture + SRV + (w, h, fmt)
/// it was built for — rebuilt when the ring geometry changes.
blend_scratch: Option<(
ID3D11Texture2D,
ID3D11ShaderResourceView,
u32,
u32,
DXGI_FORMAT,
)>,
/// The (serial, x, y, visible) of the LAST blended pointer — the composite-regen change
/// key: pointer-only motion produces no driver publish (the declared hardware cursor
/// doesn't dirty frames), so `try_consume` regenerates from the last slot when this moves.
last_blend_key: Option<(u64, i32, i32, bool)>,
/// The ring slot of the last FRESH publish — the regen source.
last_slot: Option<usize>,
/// The target's SDR-white scale (vs 80 nits) for HDR cursor compositing — refreshed on
/// each blend-scratch rebuild (first use + ring geometry changes). 2.5 ≈ the Windows
/// SDR-brightness default; without it the composited cursor renders visibly dark on HDR.
sdr_white_scale: f32,
width: u32,
height: u32,
slots: Vec<HostSlot>,
@@ -675,22 +611,11 @@ impl IddPushCapturer {
pyrowave: bool,
keepalive: Box<dyn Send>,
sender: crate::FrameChannelSender,
cursor_sender: Option<crate::CursorChannelSender>,
cursor_forward: Option<crate::CursorForwardSender>,
) -> std::result::Result<Self, (anyhow::Error, Box<dyn Send>)> {
// The stall-attribution listener (idempotent): started with the first IDD-push capturer so
// the stall log can correlate DWM holes with OS display events for the session's lifetime.
pf_win_display::display_events::spawn_once();
match Self::open_inner(
target,
preferred,
client_10bit,
want_444,
pyrowave,
sender,
cursor_sender,
cursor_forward,
) {
match Self::open_inner(target, preferred, client_10bit, want_444, pyrowave, sender) {
Ok(mut me) => {
me._keepalive = keepalive;
Ok(me)
@@ -707,8 +632,6 @@ impl IddPushCapturer {
want_444: bool,
pyrowave: bool,
sender: crate::FrameChannelSender,
cursor_sender: Option<crate::CursorChannelSender>,
cursor_forward: Option<crate::CursorForwardSender>,
) -> Result<Self> {
// The ring MUST live on the adapter the driver's swap-chain renders on. Primary: the
// selected render GPU — the same pick SET_RENDER_ADAPTER pinned the driver to at monitor
@@ -731,8 +654,6 @@ impl IddPushCapturer {
pyrowave,
luid,
sender.clone(),
cursor_sender.clone(),
cursor_forward.clone(),
) {
Ok(me) => Ok(me),
Err(e) => {
@@ -766,8 +687,6 @@ impl IddPushCapturer {
pyrowave,
drv,
sender,
cursor_sender,
cursor_forward,
)
.context("IDD-push rebind to the driver's reported render adapter")
}
@@ -783,8 +702,6 @@ impl IddPushCapturer {
pyrowave: bool,
luid: LUID,
sender: crate::FrameChannelSender,
cursor_sender: Option<crate::CursorChannelSender>,
cursor_forward: Option<crate::CursorForwardSender>,
) -> Result<Self> {
let (pw, ph, _hz) = preferred
.context("IDD push needs the negotiated mode (WxH) to size the shared ring")?;
@@ -1018,61 +935,6 @@ impl IddPushCapturer {
)
.context("deliver IDD-push frame channel to the driver")?;
// v5 hardware-cursor channel (M2c): create + deliver the CursorShm section. Failure
// is NON-fatal — the driver never declares the hardware cursor without this delivery,
// so the session degrades to today's composited pointer (and the forwarder simply
// never sees a live overlay).
let cursor_shared = cursor_sender.as_ref().and_then(|send_cursor| {
match cursor::CursorShared::create(target.target_id) {
Ok(cs) => {
// Deliver via the shared helper (also used for RE-delivery after a
// driver-side monitor re-arrival destroyed the worker).
deliver_cursor_channel(&broker, target.target_id, &cs, send_cursor)
.then_some(cs)
}
Err(e) => {
tracing::warn!(
"cursor section creation failed (composited cursor stays): {e:#}"
);
None
}
}
});
// No channel this session, but the target's sticky declare (an EARLIER session's —
// irrevocable, §8.6) keeps DWM's frames pointer-free with no client drawing either:
// the only visible pointer is the one composited here, so force composite mode on.
let composite_forced = target.cursor_excluded && cursor_sender.is_none();
if composite_forced {
tracing::info!(
target_id = target.target_id,
"target carries an irrevocable hardware-cursor declare from an earlier \
desktop-mode session and this session has no cursor channel the host \
composites the pointer into frames (forced, for the session's life)"
);
}
// The GDI shape poller rides the SAME gate as the delivered channel: with the driver's
// hardware cursor keeping the frame cursor-free, the poller supplies the full-fidelity
// shape (masked/monochrome included — the IddCx query can't; see cursor_poll.rs).
// Forced-composite sessions need it too — it is their only shape/position source.
let cursor_poll = (cursor_shared.is_some() || composite_forced).then(|| {
// Safety of the CCD call: read-only QueryDisplayConfig over owned locals (same
// call CursorShared::create makes) — already inside open_on's unsafe region.
let rect = pf_win_display::win_display::source_desktop_rect(target.target_id)
.unwrap_or((0, 0, i32::MAX, i32::MAX));
cursor_poll::CursorPoller::spawn(target.target_id, rect)
});
// Heal the driver's persisted cursor-forward state: a session that died on the
// secure desktop (client drops at the lock screen — the common case) leaves the
// per-target desired state `false`, and the NEXT session's channel delivery would
// adopt UNdeclared (the exact cross-session composite trap of §8.6). A fresh
// session always starts declared; the secure-desktop guard re-disables if the
// secure desktop is (still) up, via its first `poll_secure_desktop` edge.
if let (Some(_), Some(fwd)) = (cursor_shared.as_ref(), cursor_forward.as_ref()) {
if let Err(e) = fwd(true) {
tracing::debug!("cursor-forward reset at open failed (pre-v6 driver?): {e:#}");
}
}
tracing::info!(
target_id = target.target_id,
wudf_pid = target.wudf_pid,
@@ -1131,19 +993,6 @@ impl IddPushCapturer {
last_seq: 0,
last_present: None,
status_logged: false,
cursor_shared,
cursor_poll,
cursor_sender,
cursor_forward,
secure_active: false,
composite_cursor: composite_forced,
composite_forced,
cursor_blend: None,
cursor_blend_failed: false,
blend_scratch: None,
last_blend_key: None,
last_slot: None,
sdr_white_scale: 1.0,
// Held from BEFORE the first-frame gate (the display must not idle off while we
// wait for the first compose) until the capturer drops with the session.
_display_wake: pf_frame::session_tuning::DisplayWakeRequest::new(),
@@ -1488,14 +1337,6 @@ impl IddPushCapturer {
"IDD push: frame-channel re-delivery failed after ring recreate"
);
}
// Ring recreates ride display churn that can also have re-arrived the MONITOR driver-side
// (destroying its cursor worker with it) — re-deliver the surviving cursor section so the
// hardware-cursor declaration follows the CURRENT monitor generation.
if let (Some(cs), Some(send)) = (self.cursor_shared.as_ref(), self.cursor_sender.as_ref()) {
let _ = deliver_cursor_channel(&self.broker, self.target_id, cs, send);
}
self.blend_scratch = None; // ring geometry/format changed — rebuild at next blend
self.last_slot = None; // old-ring slot indices are meaningless now
self.last_seq = 0;
self.out_ring.clear(); // the output format changed → rebuild lazily at the new format
self.video_conv = None; // converters are sized + HDR-specific → rebuild at the new mode
@@ -1781,201 +1622,8 @@ impl IddPushCapturer {
Ok(Some((self.pyro_fence_handle, value)))
}
/// The (serial, x, y, visible) of the CURRENT polled cursor — the composite-regen change
/// key. `None` while the poller has no shape yet (or isn't running).
fn cursor_blend_key(&self) -> Option<(u64, i32, i32, bool)> {
self.cursor_poll
.as_ref()
.and_then(|p| p.read())
.map(|o| (o.serial, o.x, o.y, o.visible))
}
/// Composite the pointer for this convert: ensure the frame-sized blend scratch, copy the
/// slot into it, and alpha-blend the GDI poller's shape at its polled position. Returns the
/// scratch (texture + SRV) the conversion should read INSTEAD of the slot; `None` degrades
/// to the pointer-less slot (scratch/pass creation failed — warned once). A hidden pointer
/// blends nothing (the plain copy is the correct frame).
///
/// # Safety
/// D3D11 calls on the owning capture/encode thread's device + immediate context, called
/// while holding the slot's keyed mutex (the copy reads the slot).
unsafe fn prepare_blend_scratch(
&mut self,
slot_tex: &ID3D11Texture2D,
) -> Option<(ID3D11Texture2D, ID3D11ShaderResourceView)> {
let fmt = self.ring_format();
// (Re)build the scratch at the current ring geometry.
let stale = self
.blend_scratch
.as_ref()
.is_none_or(|(_, _, w, h, f)| (*w, *h, *f) != (self.width, self.height, fmt));
if stale {
self.blend_scratch = None;
let desc = D3D11_TEXTURE2D_DESC {
Width: self.width,
Height: self.height,
MipLevels: 1,
ArraySize: 1,
Format: fmt,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: D3D11_USAGE_DEFAULT,
BindFlags: (D3D11_BIND_RENDER_TARGET.0 | D3D11_BIND_SHADER_RESOURCE.0) as u32,
..Default::default()
};
let mut tex: Option<ID3D11Texture2D> = None;
let built = self
.device
.CreateTexture2D(&desc, None, Some(&mut tex))
.ok()
.and(tex)
.and_then(|t| {
let mut srv: Option<ID3D11ShaderResourceView> = None;
self.device
.CreateShaderResourceView(&t, None, Some(&mut srv))
.ok()
.and(srv)
.map(|v| (t, v))
});
match built {
Some((t, v)) => {
self.blend_scratch = Some((t, v, self.width, self.height, fmt));
if self.display_hdr {
// Where DWM places SDR white on this HDR desktop — the composited
// cursor must match or it reads dark (~2.5x at the Windows default).
// Queried only here: scratch rebuilds are rare, and the CCD query
// contends on the display-config lock, which must stay OFF the
// per-frame path.
// Safety: read-only CCD query over owned locals (within unsafe fn).
let queried =
pf_win_display::win_display::sdr_white_level_scale(self.target_id);
self.sdr_white_scale = queried.unwrap_or(self.sdr_white_scale);
tracing::info!(
target_id = self.target_id,
queried = ?queried,
applied = self.sdr_white_scale,
"cursor composite: HDR SDR-white scale (1.0 = 80 nits; None = \
query failed keeping the prior value)"
);
}
}
None => {
if !self.cursor_blend_failed {
self.cursor_blend_failed = true;
tracing::warn!(
"cursor blend scratch creation failed — capture-model frames stay \
pointer-less this session"
);
}
return None;
}
}
}
let (tex, srv, ..) = self.blend_scratch.as_ref().expect("just ensured");
let (tex, srv) = (tex.clone(), srv.clone());
self.context.CopyResource(&tex, slot_tex);
// Blend the pointer (visible shapes only; hidden = the copy alone is the frame).
let overlay = self.cursor_poll.as_ref().and_then(|p| p.read());
self.last_blend_key = overlay.as_ref().map(|o| (o.serial, o.x, o.y, o.visible));
if let Some(ov) = overlay.filter(|o| o.visible) {
if self.cursor_blend.is_none() && !self.cursor_blend_failed {
match cursor_blend::CursorBlendPass::new(&self.device) {
Ok(p) => self.cursor_blend = Some(p),
Err(e) => {
self.cursor_blend_failed = true;
tracing::warn!(
"cursor blend pass build failed — capture-model frames stay \
pointer-less this session: {e:#}"
);
}
}
}
if let Some(pass) = self.cursor_blend.as_mut() {
// FP16 ring = scRGB linear composition (HDR): linearize the sRGB shape and
// scale it to the target's SDR white so it matches the desktop around it.
let scale = if self.display_hdr {
self.sdr_white_scale
} else {
0.0
};
if let Err(e) = pass.blend(&self.device, &self.context, &tex, &ov, scale) {
if !self.cursor_blend_failed {
self.cursor_blend_failed = true;
tracing::warn!("cursor blend draw failed — pointer-less frames: {e:#}");
}
}
}
}
Some((tex, srv))
}
/// The secure-desktop guard (the 0.18.0 UAC/Winlogon regression). UAC consent and Winlogon
/// live on the SECURE desktop, which the OS renders through the software-cursor path — its
/// per-mode-commit default. With this session's IddCx hardware cursor declared (and
/// re-declared by the driver on every swap-chain assign), that path never materialises, the
/// secure desktop never presents into our swap-chain, and the stream freezes on the last
/// normal-desktop frame for the entire UAC/lock interaction. On the poller's secure edge:
/// stand the declare down (`SET_CURSOR_FORWARD` off — the driver stops its per-assign
/// re-declare — plus the host facade's forced same-mode re-commit that actualises the
/// software cursor); on dismissal, re-declare. Runs on the capture/encode thread every tick
/// (it must keep running while frames are stalled — that is exactly the state it exits).
fn poll_secure_desktop(&mut self) {
let Some(fwd) = self.cursor_forward.as_ref() else {
return;
};
// Sessions with a declare possibly in play: the channel session that declared it, and
// the forced-composite session whose (reused) driver monitor may still run an earlier
// session's cursor worker. A plain session on a clean target has no poller — no guard.
if self.cursor_shared.is_none() && !self.composite_forced {
return;
}
let secure = self
.cursor_poll
.as_ref()
.is_some_and(|p| p.secure_desktop());
if secure == self.secure_active {
return;
}
self.secure_active = secure;
if secure {
tracing::info!(
target_id = self.target_id,
"secure desktop (UAC/Winlogon) active — standing the IddCx hardware-cursor \
declare down so the OS software-cursor path can render it"
);
if let Err(e) = fwd(false) {
tracing::warn!(
"secure-desktop cursor-forward stand-down failed (secure content may stay \
invisible this session): {e:#}"
);
}
} else {
tracing::info!(
target_id = self.target_id,
"secure desktop dismissed — restoring the cursor render model"
);
// Re-declare only for the session that RUNS the cursor channel; a forced-composite
// session never wanted the declare (leaving the driver's desired state off also
// stops a reused worker's per-assign re-declares for good — the next channel
// session's open-time reset re-arms it).
if self.cursor_shared.is_some() {
if let Err(e) = fwd(true) {
tracing::warn!(
"secure-desktop cursor-forward re-enable failed (client-drawn cursor \
may double with a composited one): {e:#}"
);
}
}
}
}
fn try_consume(&mut self) -> Result<Option<CapturedFrame>> {
self.log_driver_status_once();
// The secure-desktop guard first: while UAC/Winlogon is up there may be NO fresh frames
// at all — this edge is what brings them back.
self.poll_secure_desktop();
// Follow the display: a "Use HDR" flip recreates the ring at the matching format.
self.poll_display_hdr();
// Recover-or-drop (GB1): if a descriptor change triggered a recreate but no fresh frame has resumed
@@ -2034,25 +1682,9 @@ impl IddPushCapturer {
return Ok(None);
}
let seq = u64::from(tok.seq);
let mut slot = tok.slot as usize;
let fresh = seq != self.last_seq && slot < self.slots.len();
let mut regen = false;
if !fresh {
// Composite cursor model: pointer-only motion produces NO new publish (the declared
// hardware cursor never dirties the frame), so a static desktop would freeze the
// blended pointer. Regenerate from the LAST slot whenever the polled cursor state
// changed — the re-converted out-ring frame carries the pointer's new position.
let moved = self.composite_cursor
&& self.last_slot.is_some()
&& self.cursor_blend_key() != self.last_blend_key;
if !moved {
return Ok(None);
}
slot = self.last_slot.expect("checked above");
if slot >= self.slots.len() {
return Ok(None); // ring shrank across a recreate — wait for a fresh publish
}
regen = true;
let slot = tok.slot as usize;
if seq == self.last_seq || slot >= self.slots.len() {
return Ok(None);
}
// Build the ring + converter BEFORE acquiring the slot so nothing between Acquire and Release
// can `?`-return and leak the keyed-mutex lock (which would stall the driver on that slot).
@@ -2086,31 +1718,18 @@ impl IddPushCapturer {
// Hold the slot's keyed mutex only across the convert/copy into the host out-ring (NOT across the
// ~3 ms encode — NVENC reads the host out-ring slot, not the keyed-mutex slot), so the driver gets
// the slot back immediately and the encode of the PREVIOUS frame overlaps this convert.
// Clone the slot's COM interfaces (an AddRef each) so the guard borrows LOCALS, leaving
// `self` free for the composite blend prep inside the lock.
let (slot_tex, slot_srv, slot_mutex) = {
let s = &self.slots[slot];
(s.tex.clone(), s.srv.clone(), s.mutex.clone())
};
let s = &self.slots[slot];
// Acquire the slot's keyed mutex via a RAII guard, scoped to JUST the convert/copy below so it
// releases at the same point as the old hand-written `ReleaseSync` (the driver gets the slot back
// immediately, NOT held across the rest of `try_consume`) — but now leak-proof on any early return.
{
let Some(_lock) = KeyedMutexGuard::acquire(&slot_mutex, 0, 8) else {
let Some(_lock) = KeyedMutexGuard::acquire(&s.mutex, 0, 8) else {
return Ok(None);
};
// SAFETY: convert on the owning (encode) thread's immediate context, holding the slot lock.
// A `?` here is leak-safe: `_lock` (the KeyedMutexGuard) drops on the early return, releasing
// the slot back to the driver.
unsafe {
// Composite cursor model: divert the convert input through the blend scratch —
// a slot copy with the pointer quad alpha-blended on top. `None` = compositing
// off or degraded (the conversion then reads the slot as always).
let blended = if self.composite_cursor {
self.prepare_blend_scratch(&slot_tex)
} else {
None
};
if self.pyrowave {
// PyroWave: ring slot SRV (BGRA for SDR, scRGB FP16 for HDR) → the two separate
// plane textures via the mode-aware CSC; the shared fence signalled just after
@@ -2118,17 +1737,22 @@ impl IddPushCapturer {
// convert. The composition format is pinned to the negotiated depth.
let (_, y_rtv, _, cbcr_rtv) = pyro_slot.as_ref().expect("pyro slot");
if let Some(conv) = self.pyro_conv.as_ref() {
let src = blended.as_ref().map(|(_, srv)| srv).unwrap_or(&slot_srv);
conv.convert(&self.context, src, y_rtv, cbcr_rtv, self.width, self.height)?;
conv.convert(
&self.context,
&s.srv,
y_rtv,
cbcr_rtv,
self.width,
self.height,
)?;
}
} else if self.display_hdr {
// HDR: FP16 slot SRV → P010 (BT.2020 PQ) via the shader; NVENC takes native P010.
if let Some(conv) = self.hdr_p010_conv.as_ref() {
let src = blended.as_ref().map(|(_, srv)| srv).unwrap_or(&slot_srv);
conv.convert(
&self.device,
&self.context,
src,
&s.srv,
out.as_ref().expect("out ring"),
self.width,
self.height,
@@ -2138,14 +1762,12 @@ impl IddPushCapturer {
// SDR 4:4:4: pass the BGRA slot through untouched — NVENC ingests full-chroma
// RGB and CSCs to YUV 4:4:4 itself (per the always-written BT.709 VUI). Plain
// copy-engine move; the slot releases back to the driver immediately.
let src = blended.as_ref().map(|(t, _)| t).unwrap_or(&slot_tex);
self.context
.CopyResource(out.as_ref().expect("out ring"), src);
.CopyResource(out.as_ref().expect("out ring"), &s.tex);
} else {
// SDR: BGRA slot → NV12 on the VIDEO engine; NVENC takes native NV12, no SM-side CSC.
if let Some(conv) = self.video_conv.as_ref() {
let src = blended.as_ref().map(|(t, _)| t).unwrap_or(&slot_tex);
conv.convert(src, out.as_ref().expect("out ring"))?;
conv.convert(&s.tex, out.as_ref().expect("out ring"))?;
}
}
}
@@ -2153,20 +1775,13 @@ impl IddPushCapturer {
}
self.out_idx = (i + 1) % ring_len;
self.last_seq = seq;
if fresh {
self.last_slot = Some(slot);
}
if let Some((y, _, cbcr, _)) = pyro_slot.as_ref() {
self.pyro_last = Some((y.clone(), cbcr.clone()));
} else {
self.last_present = Some((out.as_ref().expect("out ring").clone(), pf));
}
let now = Instant::now();
if regen {
// A regen re-encodes OLD desktop content at a new pointer position — it is not a
// fresh driver frame; feeding the freshness/stall bookkeeping would mask a dead
// driver and pollute stall attribution.
} else if self.recovering_since.take().is_some() {
if self.recovering_since.take().is_some() {
// A fresh frame resumed → recovered. The recovery gap is self-inflicted (ring
// recreate, already logged by the recreate path) — reset the stall watch so it
// doesn't read as a DWM stall.
@@ -2238,12 +1853,10 @@ impl IddPushCapturer {
}
}
}
if !regen {
self.last_fresh = now; // feeds the driver-death watch
}
// Build the frame. For PyroWave the encode input is the Y plane
// (`texture`) + the CbCr plane & fence in `pyro`; signal the shared fence
// after the convert above. SAFETY: on the owning capture/encode thread.
self.last_fresh = now; // feeds the driver-death watch
// Build the frame. For PyroWave the encode input is the Y plane
// (`texture`) + the CbCr plane & fence in `pyro`; signal the shared fence
// after the convert above. SAFETY: on the owning capture/encode thread.
let (texture, pyro) = if let Some((y, _, cbcr, _)) = pyro_slot {
// SAFETY: on the owning capture/encode thread holding the immediate context.
let (fence_handle, fence_value) =
@@ -2368,82 +1981,7 @@ impl std::fmt::Display for AttachTexFail {
impl std::error::Error for AttachTexFail {}
/// Duplicate `cs`'s section into the driver's WUDFHost and send `IOCTL_SET_CURSOR_CHANNEL`.
/// `true` = the driver adopted it (worker declared per its `cursor_forward_on` state). Shared by
/// the open-time delivery and every RE-delivery (ring recreate / flip NOT_FOUND) — the request is
/// idempotent driver-side (a replaced worker is stopped + joined).
fn deliver_cursor_channel(
broker: &ChannelBroker,
target_id: u32,
cs: &cursor::CursorShared,
send_cursor: &crate::CursorChannelSender,
) -> bool {
// SAFETY: `cs.section_handle()` borrows the section mapping `cs` owns (live across this
// synchronous call); the broker's WUDFHost process handle is live for the broker's lifetime.
let value = match unsafe { broker.dup_into_public(cs.section_handle()) } {
Ok(v) => v,
Err(e) => {
tracing::warn!("cursor section duplication failed (composited cursor stays): {e:#}");
return false;
}
};
let req = pf_driver_proto::control::SetCursorChannelRequest {
target_id,
_pad: 0,
header_handle: value,
};
match send_cursor(&req) {
Ok(()) => {
tracing::info!(
target_id,
"IDD push(host): cursor channel delivered — driver declares the hardware cursor"
);
true
}
Err(e) => {
broker.close_remote_public(value);
tracing::warn!("cursor channel delivery failed (composited cursor stays): {e:#}");
false
}
}
}
impl Capturer for IddPushCapturer {
fn cursor(&mut self) -> Option<pf_frame::CursorOverlay> {
// A LIVE poller is the sole source — even while it still reports `None` (pre-first-shape):
// falling back to the shm mid-session would interleave two serial namespaces and poison
// the client's shape cache. The shm read only serves a poller that failed to start/died.
if let Some(p) = &self.cursor_poll {
if p.alive() {
return p.read();
}
}
self.cursor_shared.as_mut().and_then(|c| c.read())
}
fn set_cursor_forward(&mut self, on: bool) {
// The composite (capture) model is implemented HOST-side: the driver's hardware cursor
// stays declared for the session's whole life — the only dependable state (there is NO
// working un-declare; see cursor_blend.rs) — keeping every frame pointer-free, and the
// capturer blends the GDI poller's shape into the frame itself. No driver round-trip.
// `composite_forced` (a channel-less session on a sticky-declared target) is pinned ON:
// with no client drawing, un-compositing would erase the pointer entirely.
let composite = (!on && self.cursor_shared.is_some()) || self.composite_forced;
if self.composite_cursor != composite {
self.composite_cursor = composite;
self.last_blend_key = None; // regenerate immediately at the current pointer state
tracing::info!(
composite,
"cursor render model: host compositing {}",
if composite {
"ON (capture model — blending the pointer into frames)"
} else {
"OFF (client draws locally)"
}
);
}
}
fn next_frame(&mut self) -> Result<CapturedFrame> {
let deadline = Instant::now() + Duration::from_secs(20);
loop {
@@ -2577,16 +2115,6 @@ fn warn_444_hdr_downgrade_once() {
impl Drop for IddPushCapturer {
fn drop(&mut self) {
// A channel session ending while the secure-desktop guard is engaged must not leave the
// driver's per-target desired state off — the next session's channel delivery would
// adopt UNdeclared and silently run the composite model (§8.6's cross-session trap).
// The open-time reset also covers this (host-crash case); this is the orderly-teardown
// belt.
if self.secure_active && self.cursor_shared.is_some() {
if let Some(fwd) = self.cursor_forward.as_ref() {
let _ = fwd(true);
}
}
self.slots.clear();
// The shared header section (`MappedSection`), the frame-ready `event` (`OwnedHandle`) and the
// broker's WUDFHost process handle free themselves via RAII (unmap view, then close handle) —
@@ -105,22 +105,6 @@ impl ChannelBroker {
Ok(out.0 as usize as u64)
}
/// Duplicate the cursor section into WUDFHost (v5 cursor channel) with the same
/// least-privilege section rights as the frame header. Thin `pub(super)` face over
/// [`dup_into`](Self::dup_into) for the cursor-delivery path in `open_on`.
///
/// # Safety
/// `h` must be a live handle of the current process.
pub(super) unsafe fn dup_into_public(&self, h: HANDLE) -> Result<u64> {
// SAFETY: forwarded contract — `h` is live per this fn's own contract.
unsafe { self.dup_into(h, Some(SECTION_MAP_RW)) }
}
/// [`close_remote`](Self::close_remote) for the cursor-delivery failure path.
pub(super) fn close_remote_public(&self, value: u64) {
self.close_remote(value);
}
/// Close a handle VALUE inside the WUDFHost table (the failure-path reaper): `DUPLICATE_CLOSE_SOURCE`
/// with no target closes the source handle regardless of the (ignored) result.
fn close_remote(&self, value: u64) {
@@ -1,194 +0,0 @@
//! Host side of the v5 hardware-cursor channel (remote-desktop-sweep M2c): the capturer creates
//! an unnamed [`CursorShm`] section, delivers it to the pf-vdisplay driver (which declares an
//! IddCx hardware cursor — DWM then EXCLUDES the pointer from the frames we consume), and reads
//! the driver's seqlock publishes here at encode-tick pace, converting them into the same
//! [`pf_frame::CursorOverlay`] the Linux portal path produces — everything downstream (the
//! cursor forwarder, the wire, the client renderer) is shared.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it.
#![deny(clippy::undocumented_unsafe_blocks)]
use super::*;
use pf_driver_proto::cursor::{
CursorShm, CURSOR_MAGIC, CURSOR_SHAPE_BYTES, CURSOR_SHAPE_MAX, CURSOR_SHAPE_OFFSET,
CURSOR_SHM_SIZE, CURSOR_TYPE_MASKED_COLOR,
};
use std::sync::atomic::AtomicU32;
/// The host end of one monitor's cursor channel: the section (we created it — the mapping stays
/// valid for the capturer's life) plus the reader's conversion cache.
pub(super) struct CursorShared {
section: MappedSection,
/// The monitor's desktop origin — IddCx reports positions in DESKTOP coordinates; the
/// overlay wants frame-relative. Fetched at attach (the virtual monitor's placement is
/// stable for the session; a topology change recreates the pipeline anyway).
origin: (i32, i32),
/// Conversion cache: the last `shape_id` whose pixels were converted, and the result.
/// Position-only updates (the common case) reuse it — a refcount bump, no pixel work.
cached_id: u32,
cached: Option<ConvertedShape>,
}
struct ConvertedShape {
rgba: std::sync::Arc<Vec<u8>>,
w: u32,
h: u32,
hot_x: u32,
hot_y: u32,
}
impl CursorShared {
/// Create + initialize the section (magic stamped, seq even/zero). The returned handle is
/// the section itself (owned by `self`); the caller duplicates it into the WUDFHost.
pub(super) fn create(target_id: u32) -> Result<CursorShared> {
// SAFETY: plain FFI. Unnamed pagefile-backed section, host-lifetime owned; the view is
// mapped once and unmapped never (the capturer's life = the session's life).
let section = unsafe {
let map = CreateFileMappingW(
INVALID_HANDLE_VALUE,
None,
PAGE_READWRITE,
0,
CURSOR_SHM_SIZE as u32,
PCWSTR::null(),
)
.context("CreateFileMapping(cursor)")?;
let map = OwnedHandle::from_raw_handle(map.0 as _);
let view = MapViewOfFile(
HANDLE(map.as_raw_handle()),
FILE_MAP_ALL_ACCESS,
0,
0,
CURSOR_SHM_SIZE,
);
if view.Value.is_null() {
bail!("MapViewOfFile failed for the cursor section");
}
let shm = view.Value.cast::<CursorShm>();
std::ptr::write_bytes(view.Value.cast::<u8>(), 0, CURSOR_SHM_SIZE);
// Magic LAST-ish (the driver validates it at adopt; seq 0 = even = consistent).
std::sync::atomic::fence(Ordering::Release);
(*shm).magic = CURSOR_MAGIC;
MappedSection { handle: map, view }
};
// Desktop origin of this monitor's source — for the desktop→frame coordinate shift.
// SAFETY: `source_desktop_rect` only runs the CCD QueryDisplayConfig FFI over owned
// locals (same call the compose-kick path makes).
let rect = unsafe { pf_win_display::win_display::source_desktop_rect(target_id) };
let origin = rect.map(|(x, y, _w, _h)| (x, y)).unwrap_or((0, 0));
Ok(CursorShared {
section,
origin,
cached_id: 0,
cached: None,
})
}
/// The section handle for the broker's duplication into the WUDFHost.
pub(super) fn section_handle(&self) -> HANDLE {
HANDLE(self.section.handle.as_raw_handle())
}
/// Seqlock-read the driver's latest publish → a frame-relative [`pf_frame::CursorOverlay`].
/// `None` until the first publish lands (or while the pointer has never been seen). A hidden
/// pointer returns `Some` with `visible: false` — the forwarder turns that into the client's
/// relative-mode hint, exactly like the Linux path.
pub(super) fn read(&mut self) -> Option<pf_frame::CursorOverlay> {
let shm = self.section.ptr::<CursorShm>();
// SAFETY: the view spans CURSOR_SHM_SIZE for self's lifetime; seq is 4-aligned in the
// fixed layout (offset 4).
let seq = unsafe { &*std::ptr::addr_of!((*shm).seq).cast::<AtomicU32>() };
for _ in 0..64 {
let s1 = seq.load(Ordering::Acquire);
if s1 == 0 {
return None; // no publish yet
}
if s1 & 1 != 0 {
std::hint::spin_loop();
continue; // writer mid-update
}
// SAFETY: header reads within the mapped view; consistency is validated by the
// seq re-check below (a torn read is discarded and retried).
let hdr = unsafe { std::ptr::read_volatile(shm) };
// Shape pixels: convert only when the OS minted a new shape id.
if hdr.visible != 0 && hdr.shape_id != self.cached_id {
let rows = hdr.height.min(CURSOR_SHAPE_MAX) as usize;
let width = hdr.width.min(CURSOR_SHAPE_MAX) as usize;
let pitch = (hdr.pitch as usize).min(CURSOR_SHAPE_BYTES / rows.max(1));
let mut raw = vec![0u8; rows * pitch];
// SAFETY: the shape region spans CURSOR_SHAPE_BYTES from CURSOR_SHAPE_OFFSET
// inside the mapped view; `rows * pitch` is clamped to it above.
unsafe {
std::ptr::copy_nonoverlapping(
self.section.ptr::<u8>().add(CURSOR_SHAPE_OFFSET),
raw.as_mut_ptr(),
rows * pitch,
);
}
// Discard the copy if the writer raced us mid-shape (seq moved) — retry.
if seq.load(Ordering::Acquire) != s1 {
continue;
}
self.cached = Some(convert_shape(&hdr, &raw, width, rows, pitch));
self.cached_id = hdr.shape_id;
} else if seq.load(Ordering::Acquire) != s1 {
continue;
}
let shape = self.cached.as_ref()?;
return Some(pf_frame::CursorOverlay {
x: hdr.x - self.origin.0,
y: hdr.y - self.origin.1,
w: shape.w,
h: shape.h,
rgba: shape.rgba.clone(),
serial: u64::from(hdr.shape_id),
hot_x: shape.hot_x,
hot_y: shape.hot_y,
visible: hdr.visible != 0,
});
}
None // persistent tearing (writer wedged mid-seq) — skip this tick
}
}
/// Convert the OS's 32-bpp pitch-strided shape rows into the overlay's packed straight RGBA.
/// ALPHA cursors are BGRA with straight per-pixel alpha (swap R↔B). MASKED_COLOR approximates:
/// alpha 0x00 = opaque color pixel; 0xFF = an XOR pixel we cannot honor client-side — rendered
/// as a translucent mid-gray so inversion cursors stay visible instead of vanishing.
fn convert_shape(
hdr: &CursorShm,
raw: &[u8],
width: usize,
rows: usize,
pitch: usize,
) -> ConvertedShape {
let masked = hdr.cursor_type == CURSOR_TYPE_MASKED_COLOR;
let mut rgba = Vec::with_capacity(width * rows * 4);
for y in 0..rows {
let row = &raw[y * pitch..];
for x in 0..width {
let o = x * 4;
if o + 4 > row.len() {
rgba.extend_from_slice(&[0, 0, 0, 0]);
continue;
}
let (b, g, r, a) = (row[o], row[o + 1], row[o + 2], row[o + 3]);
if masked {
if a == 0 {
rgba.extend_from_slice(&[r, g, b, 0xFF]);
} else {
rgba.extend_from_slice(&[0x80, 0x80, 0x80, 0xB4]);
}
} else {
rgba.extend_from_slice(&[r, g, b, a]);
}
}
}
ConvertedShape {
rgba: std::sync::Arc::new(rgba),
w: width as u32,
h: rows as u32,
hot_x: hdr.hot_x.min(width.saturating_sub(1) as u32),
hot_y: hdr.hot_y.min(rows.saturating_sub(1) as u32),
}
}
@@ -1,216 +0,0 @@
//! Host-side cursor compositing for the CAPTURE mouse model (design/remote-desktop-sweep.md §8).
//!
//! Why the host draws it: once a monitor has ever declared an IddCx hardware cursor, DWM will
//! not composite the software cursor back into its frames — there is no un-declare DDI (the
//! empty-caps re-setup is rejected `STATUS_INVALID_PARAMETER`), and a successful same-mode
//! re-commit with the driver's re-declare provably suppressed still leaves the pointer excluded
//! (all observed on-glass, 26100). So the driver keeps its hardware cursor declared for the
//! session's whole life — the state that works — and when the client flips to the capture model
//! the HOST composites the pointer into the frame itself: a slot→scratch copy plus one
//! alpha-blended quad (the GDI poller's full-fidelity shape at its polled position), entirely
//! GPU-side on the capture device, before the normal conversion runs from the scratch.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use super::*;
use windows::core::s;
use windows::Win32::Graphics::Direct3D::D3D_PRIMITIVE_TOPOLOGY_TRIANGLELIST;
use windows::Win32::Graphics::Direct3D11::{
ID3D11BlendState, ID3D11Buffer, ID3D11PixelShader, ID3D11SamplerState, ID3D11VertexShader,
D3D11_BIND_CONSTANT_BUFFER, D3D11_BLEND_DESC, D3D11_BLEND_INV_SRC_ALPHA, D3D11_BLEND_ONE,
D3D11_BLEND_OP_ADD, D3D11_BLEND_SRC_ALPHA, D3D11_BUFFER_DESC, D3D11_COMPARISON_NEVER,
D3D11_CPU_ACCESS_WRITE, D3D11_FILTER_MIN_MAG_MIP_LINEAR, D3D11_MAPPED_SUBRESOURCE,
D3D11_MAP_WRITE_DISCARD, D3D11_RENDER_TARGET_BLEND_DESC, D3D11_SAMPLER_DESC,
D3D11_SUBRESOURCE_DATA, D3D11_TEXTURE_ADDRESS_CLAMP, D3D11_USAGE_DYNAMIC, D3D11_VIEWPORT,
};
use windows::Win32::Graphics::Dxgi::Common::DXGI_FORMAT_R8G8B8A8_UNORM;
/// Straight-alpha sample of the cursor bitmap. `linear_scale` = 0 passes sRGB through (SDR
/// ring); non-zero linearizes sRGB→scRGB AND multiplies by the target's SDR-white scale
/// (`sdr_white_level_scale` — 1.0 would put cursor-white at 80 nits, visibly DARKER than the
/// surrounding SDR desktop content DWM composes at the user's SDR-brightness setting).
const CURSOR_PS: &str = r"
Texture2D<float4> tx : register(t0);
SamplerState sm : register(s0);
cbuffer C : register(b0) { float linear_scale; float3 pad; };
float4 main(float4 pos : SV_POSITION, float2 uv : TEXCOORD0) : SV_Target {
float4 c = tx.Sample(sm, uv);
if (linear_scale != 0.0) {
c.rgb = pow(abs(c.rgb), 2.2) * linear_scale;
}
return c;
}
";
/// The cursor-quad blend pass + its shape-texture cache. One per capturer (device-scoped).
pub(super) struct CursorBlendPass {
vs: ID3D11VertexShader,
ps: ID3D11PixelShader,
sampler: ID3D11SamplerState,
blend: ID3D11BlendState,
cbuf: ID3D11Buffer,
cbuf_scale: Option<f32>,
/// The uploaded shape (serial-keyed): SRV + dims in host pixels.
shape: Option<(u64, ID3D11ShaderResourceView, u32, u32)>,
}
impl CursorBlendPass {
pub(super) unsafe fn new(device: &ID3D11Device) -> Result<Self> {
let vsb = crate::dxgi::compile_shader(crate::dxgi::HDR_VS, s!("main"), s!("vs_5_0"))?;
let psb = crate::dxgi::compile_shader(CURSOR_PS, s!("main"), s!("ps_5_0"))?;
let mut vs = None;
device.CreateVertexShader(&vsb, None, Some(&mut vs))?;
let mut ps = None;
device.CreatePixelShader(&psb, None, Some(&mut ps))?;
let sd = D3D11_SAMPLER_DESC {
// LINEAR: the quad is drawn 1:1 in frame pixels, so this only matters at the
// half-texel edges; linear keeps them soft instead of ringing.
Filter: D3D11_FILTER_MIN_MAG_MIP_LINEAR,
AddressU: D3D11_TEXTURE_ADDRESS_CLAMP,
AddressV: D3D11_TEXTURE_ADDRESS_CLAMP,
AddressW: D3D11_TEXTURE_ADDRESS_CLAMP,
ComparisonFunc: D3D11_COMPARISON_NEVER,
MaxLOD: f32::MAX,
..Default::default()
};
let mut sampler = None;
device.CreateSamplerState(&sd, Some(&mut sampler))?;
// Straight-alpha over: dst.rgb = src.rgb*a + dst.rgb*(1-a); keep dst alpha.
let mut bd = D3D11_BLEND_DESC::default();
bd.RenderTarget[0] = D3D11_RENDER_TARGET_BLEND_DESC {
BlendEnable: true.into(),
SrcBlend: D3D11_BLEND_SRC_ALPHA,
DestBlend: D3D11_BLEND_INV_SRC_ALPHA,
BlendOp: D3D11_BLEND_OP_ADD,
SrcBlendAlpha: D3D11_BLEND_ONE,
DestBlendAlpha: D3D11_BLEND_ONE,
BlendOpAlpha: D3D11_BLEND_OP_ADD,
RenderTargetWriteMask: 0x0F,
};
let mut blend = None;
device.CreateBlendState(&bd, Some(&mut blend))?;
let cbd = D3D11_BUFFER_DESC {
ByteWidth: 16, // float to_linear + float3 pad
Usage: D3D11_USAGE_DYNAMIC,
BindFlags: D3D11_BIND_CONSTANT_BUFFER.0 as u32,
CPUAccessFlags: D3D11_CPU_ACCESS_WRITE.0 as u32,
..Default::default()
};
let mut cbuf = None;
device.CreateBuffer(&cbd, None, Some(&mut cbuf))?;
Ok(Self {
vs: vs.context("cursor blend vs")?,
ps: ps.context("cursor blend ps")?,
sampler: sampler.context("cursor blend sampler")?,
blend: blend.context("cursor blend state")?,
cbuf: cbuf.context("cursor blend cbuf")?,
cbuf_scale: None,
shape: None,
})
}
/// Upload `ov`'s bitmap if its serial is new; reuse the cached SRV otherwise.
unsafe fn ensure_shape(
&mut self,
device: &ID3D11Device,
ov: &pf_frame::CursorOverlay,
) -> Result<()> {
if self.shape.as_ref().is_some_and(|(s, ..)| *s == ov.serial) {
return Ok(());
}
if ov.rgba.len() < (ov.w as usize) * (ov.h as usize) * 4 || ov.w == 0 || ov.h == 0 {
bail!("malformed cursor overlay ({}x{})", ov.w, ov.h);
}
let desc = D3D11_TEXTURE2D_DESC {
Width: ov.w,
Height: ov.h,
MipLevels: 1,
ArraySize: 1,
Format: DXGI_FORMAT_R8G8B8A8_UNORM,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: D3D11_USAGE_DEFAULT,
BindFlags: D3D11_BIND_SHADER_RESOURCE.0 as u32,
..Default::default()
};
let init = D3D11_SUBRESOURCE_DATA {
pSysMem: ov.rgba.as_ptr().cast(),
SysMemPitch: ov.w * 4,
SysMemSlicePitch: 0,
};
let mut tex: Option<ID3D11Texture2D> = None;
device
.CreateTexture2D(&desc, Some(&init), Some(&mut tex))
.context("CreateTexture2D(cursor shape)")?;
let tex = tex.context("null cursor shape texture")?;
let mut srv: Option<ID3D11ShaderResourceView> = None;
device
.CreateShaderResourceView(&tex, None, Some(&mut srv))
.context("CreateShaderResourceView(cursor shape)")?;
self.shape = Some((ov.serial, srv.context("null cursor shape srv")?, ov.w, ov.h));
Ok(())
}
/// Alpha-blend `ov` onto `dst` (frame-sized, RENDER_TARGET-capable — the blend scratch).
/// `linear_scale`: 0 = SDR passthrough; non-zero = the frame is FP16 scRGB (HDR
/// composition) — linearize and scale to the target's SDR white. The quad is placed purely
/// via the viewport (the fullscreen-triangle VS fills whatever viewport is set), clipped by
/// the target automatically.
pub(super) unsafe fn blend(
&mut self,
device: &ID3D11Device,
ctx: &ID3D11DeviceContext,
dst: &ID3D11Texture2D,
ov: &pf_frame::CursorOverlay,
linear_scale: f32,
) -> Result<()> {
self.ensure_shape(device, ov)?;
let (_, srv, w, h) = self.shape.as_ref().expect("shape just ensured");
if self.cbuf_scale != Some(linear_scale) {
let cb: [f32; 4] = [linear_scale, 0.0, 0.0, 0.0];
let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
if ctx
.Map(&self.cbuf, 0, D3D11_MAP_WRITE_DISCARD, 0, Some(&mut mapped))
.is_ok()
{
std::ptr::copy_nonoverlapping(cb.as_ptr(), mapped.pData as *mut f32, cb.len());
ctx.Unmap(&self.cbuf, 0);
}
self.cbuf_scale = Some(linear_scale);
}
let mut rtv: Option<ID3D11RenderTargetView> = None;
device
.CreateRenderTargetView(dst, None, Some(&mut rtv))
.context("CreateRenderTargetView(cursor blend scratch)")?;
let rtv = rtv.context("null cursor blend rtv")?;
ctx.OMSetRenderTargets(Some(&[Some(rtv)]), None);
ctx.OMSetBlendState(&self.blend, None, 0xffff_ffff);
ctx.VSSetShader(&self.vs, None);
ctx.PSSetShader(&self.ps, None);
ctx.PSSetShaderResources(0, Some(&[Some(srv.clone())]));
ctx.PSSetSamplers(0, Some(&[Some(self.sampler.clone())]));
ctx.PSSetConstantBuffers(0, Some(&[Some(self.cbuf.clone())]));
ctx.IASetInputLayout(None);
ctx.IASetPrimitiveTopology(D3D_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
// Placement IS the viewport: the VS fills it, the OS clips it to the target.
let vp = D3D11_VIEWPORT {
TopLeftX: ov.x as f32,
TopLeftY: ov.y as f32,
Width: *w as f32,
Height: *h as f32,
MinDepth: 0.0,
MaxDepth: 1.0,
};
ctx.RSSetViewports(Some(&[vp]));
ctx.Draw(3, 0);
// Unbind so the scratch can be bound as a conversion INPUT without a hazard warning.
ctx.OMSetRenderTargets(None, None);
let none_srv: [Option<ID3D11ShaderResourceView>; 1] = [None];
ctx.PSSetShaderResources(0, Some(&none_srv));
Ok(())
}
}
@@ -1,508 +0,0 @@
//! GDI cursor poller — the Windows cursor-SHAPE source for the cursor-forward channel
//! (design/remote-desktop-sweep.md §8, the M2c redesign).
//!
//! Why not the IddCx hardware-cursor query (the v5 `CursorShm` path, now the fallback): it is
//! alpha-only BY DESIGN — `IDDCX_CURSOR_SHAPE_TYPE` has no monochrome value, the OS pre-converts
//! monochrome to masked-color (IDDCX_CURSOR_CAPS docs), and masked-color delivery is dead code on
//! modern builds (proven on-glass at every `ColorXorCursorSupport` level; no public evidence of a
//! MASKED_COLOR delivery anywhere). The driver keeps its hardware cursor declared at XOR FULL
//! purely so DWM EXCLUDES every cursor type from the IDD frame.
//!
//! Why not DXGI Desktop Duplication `GetFramePointerShape`: its `PointerPosition.Visible` goes
//! stale when the cursor moves only via injected input on current Win11 (Sunshine #5293 — exactly
//! a Punktfunk session's topology), it burns one of the session's four duplication slots, and its
//! per-output metadata on IDD monitors has conflicting field reports. The GDI path below is the
//! metadata-forwarding-remote-desktop pattern (RustDesk, WebRTC/Chrome Remote Desktop, OBS): the
//! cursor is per-session global state in win32k, readable cross-process, and `CURSOR_SHOWING` is
//! the logical visibility — immune to all of the above.
//!
//! Works because the capture host runs as SYSTEM *inside the interactive session* on
//! `winsta0\default` (the service supervisor retargets the token — `windows/service.rs`
//! `spawn_host`), so the poller thread sees the session's cursor directly; no helper process.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use super::*;
use windows::Win32::Graphics::Gdi::{
DeleteObject, GetDC, GetDIBits, GetObjectW, ReleaseDC, BITMAP, BITMAPINFO, BITMAPINFOHEADER,
BI_RGB, DIB_RGB_COLORS, HBITMAP, HDC,
};
use windows::Win32::System::StationsAndDesktops::{
CloseDesktop, GetUserObjectInformationW, OpenInputDesktop, SetThreadDesktop,
DESKTOP_ACCESS_FLAGS, DESKTOP_CONTROL_FLAGS, HDESK, UOI_NAME,
};
use windows::Win32::UI::HiDpi::{
SetThreadDpiAwarenessContext, DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2,
};
use windows::Win32::UI::WindowsAndMessaging::{
CopyIcon, DestroyIcon, GetCursorInfo, GetIconInfo, CURSORINFO, HICON, ICONINFO,
};
/// `CURSORINFO.flags` bits (WindowsAndMessaging): the pointer is logically shown /
/// touch-or-pen-suppressed. Named locally so the visibility rule below reads as the docs do.
const CURSOR_SHOWING: u32 = 0x1;
const CURSOR_SUPPRESSED: u32 = 0x2;
/// A converted shape: the cache the per-tick overlay is assembled from. `rgba` is `Arc` so the
/// slot publish (and every downstream frame attach) is a refcount bump.
struct Shape {
rgba: std::sync::Arc<Vec<u8>>,
w: u32,
h: u32,
hot_x: u32,
hot_y: u32,
serial: u64,
}
/// Off-thread GDI cursor poller. Samples `GetCursorInfo` at ~60 Hz, rasterises the `HCURSOR` only
/// when its handle value changes, and publishes a ready [`pf_frame::CursorOverlay`] snapshot; the
/// capture thread's per-tick cost is one uncontended mutex read + an `Arc` clone
/// (same split as [`DescriptorPoller`], and for the same reason: user32/gdi32 calls have no place
/// on the capture/encode thread).
pub(super) struct CursorPoller {
slot: Arc<Mutex<Option<pf_frame::CursorOverlay>>>,
stop: Arc<AtomicBool>,
/// The input desktop is a SECURE desktop (Winlogon — UAC consent / lock / logon). Classified
/// on every reattach; the capturer polls it to stand the IddCx hardware-cursor declare down
/// while the secure desktop needs the software-cursor path to render (see
/// `IddPushCapturer::poll_secure_desktop`).
secure: Arc<AtomicBool>,
thread: Option<std::thread::JoinHandle<()>>,
}
impl CursorPoller {
/// ~250 Hz: the polled position is ALSO the composite-blend position (capture model), so
/// it must out-pace the fastest session — at 16 ms a 240 fps stream re-used a stale
/// position for ~4 consecutive frames and the composited pointer visibly stuttered
/// against the video. A tick is one `GetCursorInfo` syscall (rasterisation only on shape
/// change), so 250 Hz is still negligible CPU.
const INTERVAL: Duration = Duration::from_millis(4);
/// Unconditional input-desktop reattach cadence — catches secure-desktop (UAC/lock) switches
/// without a failure signal (`GetCursorInfo` on a stale desktop *succeeds* with stale data).
/// 250 ms, not the original 2 s: the reattach now also feeds [`Self::secure_desktop`], which
/// gates when the secure desktop becomes VISIBLE in the stream (the hardware-cursor
/// stand-down) — a 2 s freeze at every UAC prompt is user-visible, ~4 `OpenInputDesktop`
/// syscalls/s are not.
const REATTACH: Duration = Duration::from_millis(250);
/// Spawn the poller for the virtual display `target_id`. `rect` = the target's desktop rect
/// (`source_desktop_rect` order: x, y, w, h) — cursor positions are desktop-global; the
/// overlay wants frame-relative, and a pointer outside the rect reports `visible: false`
/// (per-output semantics, matching the driver shm path and the Linux portal).
pub(super) fn spawn(target_id: u32, rect: (i32, i32, i32, i32)) -> Self {
let slot: Arc<Mutex<Option<pf_frame::CursorOverlay>>> = Arc::new(Mutex::new(None));
let stop = Arc::new(AtomicBool::new(false));
let secure = Arc::new(AtomicBool::new(false));
let (slot_t, stop_t, secure_t) = (slot.clone(), stop.clone(), secure.clone());
let thread = std::thread::Builder::new()
.name("pf-cursor-poll".into())
.spawn(move || run(target_id, rect, &slot_t, &stop_t, &secure_t))
.ok();
if thread.is_none() {
tracing::warn!("cursor poller thread spawn failed — cursor falls back to driver shm");
}
Self {
slot,
stop,
secure,
thread,
}
}
/// The latest overlay snapshot (`None` until the first successful shape rasterisation).
pub(super) fn read(&self) -> Option<pf_frame::CursorOverlay> {
self.slot.lock().unwrap_or_else(|p| p.into_inner()).clone()
}
/// Whether the input desktop is currently a SECURE desktop (UAC consent / Winlogon lock or
/// logon). Latched by the poll thread on its reattach cadence (≤ [`Self::REATTACH`] stale).
pub(super) fn secure_desktop(&self) -> bool {
self.secure.load(Ordering::Relaxed)
}
/// Whether the worker thread is (still) alive — `false` degrades the capturer to the shm read.
pub(super) fn alive(&self) -> bool {
self.thread.as_ref().is_some_and(|t| !t.is_finished())
}
}
impl Drop for CursorPoller {
fn drop(&mut self) {
self.stop.store(true, Ordering::Relaxed);
if let Some(t) = self.thread.take() {
let _ = t.join(); // worker sleeps ≤ INTERVAL — a bounded join
}
}
}
/// The poll loop. Owns the thread's input-desktop binding and the shape cache.
fn run(
target_id: u32,
rect: (i32, i32, i32, i32),
slot: &Mutex<Option<pf_frame::CursorOverlay>>,
stop: &AtomicBool,
secure: &AtomicBool,
) {
// Physical-pixel coordinates on this thread regardless of the process's DPI awareness:
// `rect` comes from CCD (always physical), and a DPI-virtualized `GetCursorInfo` position
// would land in the wrong frame pixel on any scaled display. Thread-scoped, so the rest of
// the host is untouched.
// SAFETY: takes and returns only a by-value context handle; affects this thread only.
let _ = unsafe { SetThreadDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2) };
let mut desktop = DesktopBinding::default();
// best-effort: already on winsta0\default if this fails
publish_secure(secure, desktop.reattach());
let mut last_attach = Instant::now();
let mut shape: Option<Shape> = None;
let mut cached_handle: isize = 0;
let mut failed_handle: isize = 0; // don't re-rasterise a failing handle every tick
let mut serial: u64 = 0;
let mut logged_live = false;
while !stop.load(Ordering::Relaxed) {
std::thread::sleep(CursorPoller::INTERVAL);
if last_attach.elapsed() >= CursorPoller::REATTACH {
last_attach = Instant::now();
publish_secure(secure, desktop.reattach());
}
let mut ci = CURSORINFO {
cbSize: std::mem::size_of::<CURSORINFO>() as u32,
..Default::default()
};
// SAFETY: `ci` is a live, correctly-sized out-param for this synchronous call; no pointer
// escapes it.
if unsafe { GetCursorInfo(&mut ci) }.is_err() {
// Desktop went away under us (secure-desktop switch mid-call) — rebind and retry
// next tick; the slot keeps its last snapshot meanwhile.
publish_secure(secure, desktop.reattach());
last_attach = Instant::now();
continue;
}
let flags = ci.flags.0;
let showing = flags & CURSOR_SHOWING != 0 && flags & CURSOR_SUPPRESSED == 0;
// Rasterise on handle change only (position-only ticks are a header update). Hidden
// cursors keep the cached shape — the forwarder's hidden-but-known contract needs the
// bitmap to have been seen. v1: animated cursors publish their first frame (the OBS
// behavior); frame cycling via DrawIconEx istep is a known follow-up.
let handle = ci.hCursor.0 as isize;
if showing && handle != 0 && handle != cached_handle && handle != failed_handle {
match rasterize(ci.hCursor) {
Some((rgba, w, h, hot_x, hot_y)) => {
serial += 1;
shape = Some(Shape {
rgba: std::sync::Arc::new(rgba),
w,
h,
hot_x,
hot_y,
serial,
});
cached_handle = handle;
failed_handle = 0;
if !logged_live {
logged_live = true;
tracing::info!(
target_id,
"cursor poller live — GDI shape source publishing (serial 1: {w}x{h})"
);
}
}
None => {
// The owning app may have destroyed the cursor mid-read; keep the previous
// shape and don't hammer this handle again until it changes.
failed_handle = handle;
}
}
}
let overlay = shape.as_ref().map(|s| {
let (px, py) = (ci.ptScreenPos.x - rect.0, ci.ptScreenPos.y - rect.1);
let in_rect = px >= 0 && py >= 0 && px < rect.2 && py < rect.3;
pf_frame::CursorOverlay {
// Overlay x/y = bitmap top-left (reported position hotspot), frame pixels.
x: px - s.hot_x as i32,
y: py - s.hot_y as i32,
w: s.w,
h: s.h,
rgba: s.rgba.clone(),
serial: s.serial,
hot_x: s.hot_x,
hot_y: s.hot_y,
visible: showing && in_rect,
}
});
*slot.lock().unwrap_or_else(|p| p.into_inner()) = overlay;
}
}
/// Store a reattach's secure-desktop verdict (`None` = classification unavailable — keep the
/// previous state rather than flapping the capturer's hardware-cursor stand-down).
fn publish_secure(secure: &AtomicBool, verdict: Option<bool>) {
if let Some(s) = verdict {
secure.store(s, Ordering::Relaxed);
}
}
/// The thread's owned input-desktop handle — the [`SendInputInjector`] reattach model
/// (`pf-inject` sendinput.rs): keep the current binding, swap on demand, close exactly once.
#[derive(Default)]
struct DesktopBinding(Option<HDESK>);
impl DesktopBinding {
/// Rebind to the CURRENT input desktop. Returns whether that desktop is a SECURE one
/// (`UOI_NAME` != "Default": "Winlogon" during UAC consent / lock / logon) — `None` when the
/// input desktop could not be opened, in which case the binding (and the caller's secure
/// state) stays put.
fn reattach(&mut self) -> Option<bool> {
const GENERIC_ALL: u32 = 0x1000_0000;
// SAFETY: `OpenInputDesktop`/`SetThreadDesktop`/`CloseDesktop` take only by-value args.
// `OpenInputDesktop` yields an owned `HDESK` only on `Ok`; it is either installed (and the
// previously-owned handle closed exactly once) or closed on failure — no handle is leaked
// or used after close. `SetThreadDesktop` rebinds only this calling thread (which owns
// no windows/hooks, so the rebind cannot fail on that account).
unsafe {
match OpenInputDesktop(
DESKTOP_CONTROL_FLAGS(0),
false,
DESKTOP_ACCESS_FLAGS(GENERIC_ALL),
) {
Ok(h) => {
let secure = desktop_is_secure(h);
if SetThreadDesktop(h).is_ok() {
if let Some(old) = self.0.replace(h) {
let _ = CloseDesktop(old);
}
} else {
let _ = CloseDesktop(h);
}
Some(secure)
}
Err(_) => None, // not privileged for this desktop; stay put
}
}
}
}
/// `UOI_NAME` of `h` != "Default" — i.e. the input desktop is Winlogon (UAC consent / lock /
/// logon) or a screen-saver desktop, both of which need the OS's software-cursor render path.
/// Unnameable desktops read as NOT secure: the only in-contract failure is a too-small buffer,
/// and misreading secure-as-normal merely keeps today's behavior for a beat.
fn desktop_is_secure(h: HDESK) -> bool {
let mut name = [0u16; 64]; // "Default"/"Winlogon"/"Screen-saver" all fit with room to spare
let mut needed = 0u32;
// SAFETY: `h` is the live desktop handle the caller just opened; `name`/`needed` are live
// out-params sized exactly as passed; the call writes at most `nlength` bytes.
let ok = unsafe {
GetUserObjectInformationW(
windows::Win32::Foundation::HANDLE(h.0),
UOI_NAME,
Some(name.as_mut_ptr().cast()),
(name.len() * 2) as u32,
Some(&mut needed),
)
};
if ok.is_err() {
return false;
}
let len = name.iter().position(|&c| c == 0).unwrap_or(name.len());
let name = String::from_utf16_lossy(&name[..len]);
!name.eq_ignore_ascii_case("Default")
}
impl Drop for DesktopBinding {
fn drop(&mut self) {
if let Some(h) = self.0.take() {
// SAFETY: `h` is our owned desktop handle, closed exactly once here.
let _ = unsafe { CloseDesktop(h) };
}
}
}
/// Rasterise `hcursor` to straight-alpha RGBA: `(rgba, w, h, hot_x, hot_y)`. `None` on any
/// failure (caller keeps the previous shape).
fn rasterize(hcursor: windows::Win32::UI::WindowsAndMessaging::HCURSOR) -> RasterOut {
// CopyIcon first: the owning process can destroy its HCURSOR between GetCursorInfo and the
// reads below; the copy is ours (the OBS/WebRTC guard).
// SAFETY: `HICON(hcursor.0)` reinterprets the cursor handle as an icon handle (cursors ARE
// icons in user32); CopyIcon yields an owned HICON we destroy below.
let Ok(icon) = (unsafe { CopyIcon(HICON(hcursor.0)) }) else {
return None;
};
let mut ii = ICONINFO::default();
// SAFETY: `ii` is a live out-param. On Ok it hands us COPIES of the mask/color bitmaps —
// both deleted below (GDI-handle leak otherwise).
let got = unsafe { GetIconInfo(icon, &mut ii) };
let out = if got.is_ok() { convert(&ii) } else { None };
// SAFETY: deleting the two bitmap copies GetIconInfo returned (null-safe: DeleteObject on a
// null HGDIOBJ fails harmlessly) and the icon copy — each exactly once.
unsafe {
let _ = DeleteObject(ii.hbmColor.into());
let _ = DeleteObject(ii.hbmMask.into());
let _ = DestroyIcon(icon);
}
out.map(|(rgba, w, h)| {
let hot_x = ii.xHotspot.min(w.saturating_sub(1));
let hot_y = ii.yHotspot.min(h.saturating_sub(1));
(rgba, w, h, hot_x, hot_y)
})
}
type RasterOut = Option<(Vec<u8>, u32, u32, u32, u32)>;
/// Convert the ICONINFO bitmaps to straight RGBA. Two families:
/// - color (`hbmColor` set): 32bpp BGRA; if the alpha channel is entirely empty (old-style
/// cursors) the AND mask supplies it (mask bit 1 = transparent).
/// - monochrome (`hbmColor` null): `hbmMask` is DOUBLE height — AND plane over XOR plane, the
/// WebRTC truth table: (0,0) black, (0,1) white, (1,0) transparent, (1,1) invert. Invert
/// pixels — unrepresentable in straight alpha — become opaque black with a white outline
/// grown into adjacent transparency (the WebRTC approximation; keeps the I-beam legible on
/// any background, which the old translucent-gray stand-in did not).
fn convert(ii: &ICONINFO) -> Option<(Vec<u8>, u32, u32)> {
// SAFETY: GetDC(None) yields the screen DC, released below on every path; it is only used
// as the GetDIBits reference DC.
let dc = unsafe { GetDC(None) };
let result = (|| {
if !ii.hbmColor.is_invalid() {
let color = read_bitmap_32(dc, ii.hbmColor)?;
let (w, h) = (color.w as u32, color.h as u32);
let mut rgba = bgra_to_rgba(&color.bgra);
if rgba.chunks_exact(4).all(|p| p[3] == 0) {
// Alpha-less color cursor: transparency lives in the AND mask.
let mask = read_bitmap_32(dc, ii.hbmMask)?;
if mask.w != color.w || mask.h < color.h {
return None;
}
for (px, m) in rgba.chunks_exact_mut(4).zip(mask.bgra.chunks_exact(4)) {
px[3] = if m[0] != 0 { 0 } else { 0xFF }; // mask white (AND=1) = transparent
}
}
Some((rgba, w, h))
} else {
let mask = read_bitmap_32(dc, ii.hbmMask)?;
if mask.h < 2 || mask.h % 2 != 0 {
return None;
}
let (w, h) = (mask.w as usize, (mask.h / 2) as usize);
let row = w * 4;
let (and_plane, xor_plane) = mask.bgra.split_at(h * row);
let mut rgba = vec![0u8; w * h * 4];
let mut invert = vec![false; w * h];
for i in 0..w * h {
let (a, x) = (and_plane[i * 4] != 0, xor_plane[i * 4] != 0);
let px = &mut rgba[i * 4..i * 4 + 4];
match (a, x) {
(false, false) => px.copy_from_slice(&[0, 0, 0, 0xFF]),
(false, true) => px.copy_from_slice(&[0xFF, 0xFF, 0xFF, 0xFF]),
(true, false) => {} // transparent (already zeroed)
(true, true) => {
px.copy_from_slice(&[0, 0, 0, 0xFF]);
invert[i] = true;
}
}
}
// White outline around invert regions so the (now black) shape survives dark
// backgrounds: any transparent 8-neighbor of an invert pixel turns opaque white.
for y in 0..h as i32 {
for x in 0..w as i32 {
if !invert[(y * w as i32 + x) as usize] {
continue;
}
for (dx, dy) in NEIGHBORS {
let (nx, ny) = (x + dx, y + dy);
if nx < 0 || ny < 0 || nx >= w as i32 || ny >= h as i32 {
continue;
}
let o = (ny * w as i32 + nx) as usize * 4;
if rgba[o + 3] == 0 {
rgba[o..o + 4].copy_from_slice(&[0xFF, 0xFF, 0xFF, 0xFF]);
}
}
}
}
Some((rgba, w as u32, h as u32))
}
})();
// SAFETY: releasing the screen DC obtained above, exactly once.
unsafe {
ReleaseDC(None, dc);
}
result
}
const NEIGHBORS: [(i32, i32); 8] = [
(-1, -1),
(0, -1),
(1, -1),
(-1, 0),
(1, 0),
(-1, 1),
(0, 1),
(1, 1),
];
struct RawBitmap {
w: i32,
h: i32,
/// 32bpp top-down BGRA rows, `w*h*4` (monochrome sources arrive expanded: 0x00/0xFF channels).
bgra: Vec<u8>,
}
/// Read any GDI bitmap as 32bpp top-down via `GetDIBits` (which performs the 1bpp→32bpp
/// expansion for the mask planes).
fn read_bitmap_32(dc: HDC, hbm: HBITMAP) -> Option<RawBitmap> {
let mut bm = BITMAP::default();
// SAFETY: `bm` is a live out-param sized exactly as passed; GetObjectW only writes into it.
let n = unsafe {
GetObjectW(
hbm.into(),
std::mem::size_of::<BITMAP>() as i32,
Some((&mut bm as *mut BITMAP).cast()),
)
};
if n == 0 || bm.bmWidth <= 0 || bm.bmHeight <= 0 || bm.bmWidth > 512 || bm.bmHeight > 1024 {
return None; // 512/1024: sanity caps (256² is the wire max; XL accessibility ≤ that)
}
let (w, h) = (bm.bmWidth, bm.bmHeight);
let mut info = BITMAPINFO {
bmiHeader: BITMAPINFOHEADER {
biSize: std::mem::size_of::<BITMAPINFOHEADER>() as u32,
biWidth: w,
biHeight: -h, // top-down
biPlanes: 1,
biBitCount: 32,
biCompression: BI_RGB.0,
..Default::default()
},
..Default::default()
};
let mut buf = vec![0u8; (w as usize) * (h as usize) * 4];
// SAFETY: `buf` spans exactly `h` rows of `w` 32bpp pixels as described by `info`; both are
// live locals for this synchronous call, `hbm` is a live bitmap not selected into any DC
// (fresh GetIconInfo copies).
let rows = unsafe {
GetDIBits(
dc,
hbm,
0,
h as u32,
Some(buf.as_mut_ptr().cast()),
&mut info,
DIB_RGB_COLORS,
)
};
(rows != 0).then_some(RawBitmap { w, h, bgra: buf })
}
fn bgra_to_rgba(bgra: &[u8]) -> Vec<u8> {
let mut out = bgra.to_vec();
for px in out.chunks_exact_mut(4) {
px.swap(0, 2);
}
out
}
+4 -10
View File
@@ -876,16 +876,10 @@ impl Worker {
);
return;
};
let pref = match self.pad_info(id) {
// Steam Input's virtual pad standing in front of the Deck's built-in controls (the
// only-pad-forwarded case, [`Self::forwarded_ids`]): declare the DECK kind, not the
// wrapper's Xbox 360 identity. [`Self::auto_pref`] already resolves the SESSION
// default this way, but a current host honors the per-pad arrival over the session
// default — so without this the host builds an X-Box 360 pad on a real Deck.
Some(p) if p.steam_virtual && is_steam_deck() => GamepadPref::SteamDeck,
Some(p) => p.pref,
None => GamepadPref::Xbox360,
};
let pref = self
.pad_info(id)
.map(|p| p.pref)
.unwrap_or(GamepadPref::Xbox360);
match self.subsystem.open(sdl3::sys::joystick::SDL_JoystickID(id)) {
Ok(pad) => {
let mut slot = Slot::new(id, index, pref, pad);
-12
View File
@@ -44,13 +44,6 @@ pub struct SessionParams {
/// Share the clipboard with this host (the per-host `KnownHost::clipboard_sync`). The
/// bridge additionally needs the host to advertise `HOST_CAP_CLIPBOARD`.
pub clipboard: bool,
/// Advertise `quic::CLIENT_CAP_CURSOR`: this embedder renders the host cursor locally
/// (the presenter's cursor channel, design/remote-desktop-sweep.md M2), so the host may
/// stop compositing the pointer into the video. Only set when the embedder actually
/// draws it (the SDL presenter in desktop mouse mode) — a session that advertises it
/// without rendering streams with NO visible cursor. The host answers `HOST_CAP_CURSOR`
/// when its capture can forward (Linux portal, not gamescope/Windows).
pub cursor_forward: bool,
/// Video decoder preference (Settings; `PUNKTFUNK_DECODER` overrides — see
/// `video::Decoder::new`).
pub decoder: String,
@@ -262,11 +255,6 @@ fn pump(
// This display's HDR volume → the host's virtual-display EDID. The env hatch wins so an
// A/B run can pin an exact peak (PUNKTFUNK_CLIENT_PEAK_NITS=600).
punktfunk_core::client::display_hdr_env_override().or(params.display_hdr),
if params.cursor_forward {
punktfunk_core::quic::CLIENT_CAP_CURSOR
} else {
0
},
params.launch.clone(),
params.pin,
Some(params.identity),
-5
View File
@@ -325,11 +325,6 @@ pub fn connect_reject_message(reason: punktfunk_core::reject::RejectReason) -> S
"Client and host versions don't match — update both to the same release.".into()
}
R::Busy => "The host is busy with another session.".into(),
R::SetupFailed => {
"The host accepted the connection but couldn't start the stream — the host's log \
(web console Log) has the cause."
.into()
}
}
}
+9 -213
View File
@@ -66,31 +66,7 @@ pub const fn interface_guid_fields() -> (u32, u16, u16, [u8; 8]) {
/// too ([`MIN_DRIVER_PROTOCOL_VERSION`]) and simply falls back to the re-arrival resize against it;
/// a v4 driver serving an older (v3-asserting) host fails that host's strict handshake — ship
/// driver+host together, as ever.
/// v5: ADDITIVE — the IddCx HARDWARE CURSOR channel (remote-desktop-sweep M2c):
/// [`control::AddRequest::hw_cursor`] (the former tail `_reserved` — same size, same offsets)
/// asks the driver to declare a hardware cursor for this monitor (DWM then EXCLUDES the pointer
/// from the desktop frame and delivers shape/position out-of-band), and
/// [`control::IOCTL_SET_CURSOR_CHANNEL`] delivers the host-created [`cursor::CursorShm`] section
/// the driver's cursor thread seqlock-publishes into. Nothing existing changed; the host gates
/// the feature on the handshake-reported version (`>= 5`) and keeps composited-cursor behavior
/// against older drivers.
/// v6: ADDITIVE — [`control::IOCTL_SET_CURSOR_FORWARD`] (the mid-stream cursor-render flip,
/// remote-desktop-sweep §8): the client's mouse-model flip (un)declares the hardware cursor on a
/// LIVE monitor, so the capture mouse model gets DWM's composited pointer back (full fidelity)
/// and the desktop model gets exclusion + forwarding. Nothing existing changed; against a v5
/// driver the unknown IOCTL fails and the host logs + keeps the declared-at-ADD behavior.
/// v6 tail ext (no bump, the `AddRequest` luminance-tail discipline):
/// [`control::AddReply::cursor_excluded`] — the driver reports whether its ADAPTER already
/// carries a hardware-cursor declare from an earlier session. A declare is IRREVOCABLE
/// (remote-desktop-sweep §8.6, proven on-glass) and its exclusion reaches EVERY later monitor of
/// the adapter, not just the declaring target (on-glass 2026-07-23: a declare on one target left
/// a different client's fresh target cursor-less): DWM never composites the software cursor back
/// into any of the adapter's frames until the adapter resets. The host uses the flag to
/// self-composite the pointer (GDI poller + blend) in sessions that never negotiate the
/// cursor channel — without it those sessions are silently cursor-less. Both skews degrade
/// cleanly: an old driver writes only the 20-byte reply prefix (host reads `0` = unknown/clean),
/// an old host retrieves a 20-byte buffer (driver writes just the prefix).
pub const PROTOCOL_VERSION: u32 = 6;
pub const PROTOCOL_VERSION: u32 = 4;
/// The OLDEST driver protocol this host still drives (v4 is additive over v3 — see the v4 note on
/// [`PROTOCOL_VERSION`]): a v3 driver lacks only `IOCTL_UPDATE_MODES`, which the host gates on the
@@ -134,22 +110,6 @@ pub mod control {
/// identity (saved per-monitor DPI) and the driver's swap-chain/stash machinery survive. A v3
/// driver fails this unknown IOCTL → the host falls back to the re-arrival resize.
pub const IOCTL_UPDATE_MODES: u32 = ctl_code(0x907);
/// Deliver a monitor's hardware-cursor channel (v5): the handle VALUE of the unnamed
/// [`cursor::CursorShm`](crate::cursor) file mapping the host duplicated into the WUDFHost
/// (same delivery model as [`IOCTL_SET_FRAME_CHANNEL`], no event — the host polls the
/// seqlock at its encode-tick pace). Sent once after ADD, only for a monitor whose
/// [`AddRequest::hw_cursor`] was set. The driver maps it, calls
/// `IddCxMonitorSetupHardwareCursor`, and starts its cursor-query thread. Input
/// [`SetCursorChannelRequest`].
pub const IOCTL_SET_CURSOR_CHANNEL: u32 = ctl_code(0x908);
/// Flip a LIVE monitor's hardware-cursor declaration (v6, the mid-stream cursor-render
/// flip): `enable = 1` re-declares (`IddCxMonitorSetupHardwareCursor` — DWM excludes the
/// pointer, the query/shape machinery resumes), `enable = 0` un-declares (the driver stops
/// re-declaring on mode commits and asks the OS to revert to the software cursor — DWM
/// composites the pointer into the frame, the pre-channel behavior the capture mouse
/// model wants). Only meaningful for a monitor whose cursor channel was delivered. Input
/// [`SetCursorForwardRequest`].
pub const IOCTL_SET_CURSOR_FORWARD: u32 = ctl_code(0x909);
/// `IOCTL_ADD` input. A monotonic `session_id` keys the monitor (the host's refcount manager owns
/// collision safety — no more SudoVDA's 16-byte GUID + pid-mangling). The driver advertises this
@@ -187,13 +147,9 @@ pub mod control {
/// The client display's min luminance in MILLI-nits (0.001 cd/m² — the CTA min-luminance
/// range lives well below 1 nit) → Desired Content Min Luminance. `0` = unknown.
pub min_luminance_millinits: u32,
/// Non-zero = declare an IddCx HARDWARE CURSOR for this monitor (v5, remote-desktop-sweep
/// M2c): DWM stops compositing the pointer into the frame and the driver publishes
/// shape/position into the [`cursor::CursorShm`](crate::cursor) section delivered by
/// [`IOCTL_SET_CURSOR_CHANNEL`]. Byte-compatible with the old tail `_reserved` (offset 36):
/// an un-upgraded driver ignores it (cursor stays composited — the host already gates on
/// the handshake version, this is defense in depth), an un-upgraded host sends `0` (off).
pub hw_cursor: u32,
/// Pads the `u64`-aligned struct to a multiple of 8 (Pod forbids implicit tail padding);
/// free expansion room for the next appended field.
pub _reserved: u32,
}
/// [`AddRequest`]'s size before the client-HDR luminance tail — the prefix an un-upgraded
@@ -218,22 +174,8 @@ pub mod control {
/// `DuplicateHandle`, then [`IOCTL_SET_FRAME_CHANNEL`]). Reported per-ADD, not per-open, so a
/// WUDFHost restart between sessions can never leave the host duplicating into a dead process.
pub wudf_pid: u32,
/// Non-zero = the ADAPTER already carries an IRREVOCABLE hardware-cursor declare from an
/// earlier session (remote-desktop-sweep §8.6; reach is adapter-wide, not per-target —
/// on-glass 2026-07-23): DWM excludes the pointer from every frame on every monitor until
/// the adapter resets, and a session without the cursor channel must
/// self-composite (GDI poller + blend) or stream a cursor-less desktop. Appended after
/// [`ADD_REPLY_LEGACY_SIZE`] under the same dual-size discipline as the `AddRequest`
/// luminance tail: an un-upgraded driver writes only the legacy prefix (the host's
/// zero-initialized buffer then reads `0` = unknown/clean), an un-upgraded host retrieves a
/// legacy-size buffer (the driver writes just the prefix).
pub cursor_excluded: u32,
}
/// [`AddReply`]'s size before the `cursor_excluded` tail — the prefix an un-upgraded driver
/// writes and an un-upgraded host retrieves (see the field docs).
pub const ADD_REPLY_LEGACY_SIZE: usize = 20;
/// `IOCTL_REMOVE` input.
#[repr(C)]
#[derive(Clone, Copy, Pod, Zeroable, Debug, PartialEq, Eq)]
@@ -311,29 +253,6 @@ pub mod control {
/// at the compile-time maximum; `ring_len` says how many entries are live).
pub const RING_LEN_USIZE: usize = RING_LEN as usize;
/// `IOCTL_SET_CURSOR_CHANNEL` input (v5): the hardware-cursor section for one monitor.
#[repr(C)]
#[derive(Clone, Copy, Pod, Zeroable, Debug, PartialEq, Eq)]
pub struct SetCursorChannelRequest {
/// The OS target id from [`AddReply`] — which monitor this channel belongs to.
pub target_id: u32,
pub _pad: u32,
/// The [`cursor::CursorShm`](crate::cursor) file-mapping handle VALUE, already duplicated
/// into the driver's WUDFHost process ([`AddReply::wudf_pid`]).
pub header_handle: u64,
}
/// `IOCTL_SET_CURSOR_FORWARD` input (v6): the mid-stream cursor-render flip for one monitor.
#[repr(C)]
#[derive(Clone, Copy, Pod, Zeroable, Debug, PartialEq, Eq)]
pub struct SetCursorForwardRequest {
/// The OS target id from [`AddReply`] — which monitor to flip.
pub target_id: u32,
/// `1` = declare the hardware cursor (exclude + forward), `0` = un-declare (DWM
/// composites — the capture mouse model).
pub enable: u32,
}
// Layout is load-bearing across the process boundary — pin it. (bytemuck's Pod derive already
// rejects any internal padding; these assert the externally-visible sizes too.) The `offset_of!`
// asserts additionally catch a SAME-SIZE field reorder, which the size+Pod checks alone miss.
@@ -350,18 +269,13 @@ pub mod control {
assert!(offset_of!(AddRequest, max_luminance_nits) == ADD_REQUEST_LEGACY_SIZE);
assert!(offset_of!(AddRequest, max_frame_avg_nits) == 28);
assert!(offset_of!(AddRequest, min_luminance_millinits) == 32);
// v5: the former tail `_reserved` — same offset, same total size (rename-only).
assert!(offset_of!(AddRequest, hw_cursor) == 36);
assert!(size_of::<AddRequest>() == 40);
assert!(size_of::<AddReply>() == 24);
assert!(size_of::<AddReply>() == 20);
assert!(offset_of!(AddReply, adapter_luid_low) == 0);
assert!(offset_of!(AddReply, adapter_luid_high) == 4);
assert!(offset_of!(AddReply, target_id) == 8);
assert!(offset_of!(AddReply, resolved_monitor_id) == 12);
assert!(offset_of!(AddReply, wudf_pid) == 16);
// The cursor-excluded tail starts exactly at the legacy boundary (prefix-compat).
assert!(offset_of!(AddReply, cursor_excluded) == ADD_REPLY_LEGACY_SIZE);
assert!(size_of::<SetFrameChannelRequest>() == 32 + 8 * RING_LEN_USIZE);
assert!(offset_of!(SetFrameChannelRequest, target_id) == 0);
@@ -374,13 +288,6 @@ pub mod control {
assert!(size_of::<RemoveRequest>() == 8);
assert!(offset_of!(RemoveRequest, session_id) == 0);
assert!(size_of::<SetCursorChannelRequest>() == 16);
assert!(offset_of!(SetCursorChannelRequest, target_id) == 0);
assert!(offset_of!(SetCursorChannelRequest, header_handle) == 8);
assert!(size_of::<SetCursorForwardRequest>() == 8);
assert!(offset_of!(SetCursorForwardRequest, target_id) == 0);
assert!(offset_of!(SetCursorForwardRequest, enable) == 4);
assert!(size_of::<UpdateModesRequest>() == 24);
assert!(offset_of!(UpdateModesRequest, session_id) == 0);
assert!(offset_of!(UpdateModesRequest, width) == 8);
@@ -1053,84 +960,6 @@ pub mod mouse {
};
}
/// The v5 hardware-cursor channel (remote-desktop-sweep M2c): one unnamed file mapping per
/// monitor, host-created, delivered by handle value ([`control::IOCTL_SET_CURSOR_CHANNEL`]).
/// The DRIVER's cursor thread (woken by its IddCx `hNewCursorDataAvailable` event) seqlock-writes
/// shape + position + visibility; the HOST reads at its encode-tick pace — no event crosses the
/// boundary. Writer: bump [`CursorShm::seq`] to ODD, write fields (+ shape bytes when the OS said
/// the shape changed), bump to EVEN. Reader: read seq (retry while odd), copy, re-read seq —
/// unchanged ⇒ consistent snapshot. Position-only updates never touch the shape bytes, so a
/// reader that skips unchanged `shape_id`s never copies torn pixels.
pub mod cursor {
use bytemuck::{Pod, Zeroable};
/// First field of [`CursorShm`] — `b"PFCU"` little-endian; anything else = not attached yet.
pub const CURSOR_MAGIC: u32 = u32::from_le_bytes(*b"PFCU");
/// Max cursor side (px) the driver declares to the OS (`IDDCX_CURSOR_CAPS::MaxX/MaxY`) and
/// the section's shape buffer is sized for. Windows XL accessibility cursors top out here;
/// the host's wire forwarder downscales to its own cap anyway.
pub const CURSOR_SHAPE_MAX: u32 = 256;
/// Shape-buffer bytes: 32-bpp at the declared max.
pub const CURSOR_SHAPE_BYTES: usize = (CURSOR_SHAPE_MAX * CURSOR_SHAPE_MAX * 4) as usize;
/// Byte offset of the shape pixels inside the section (one cache-line-ish header).
pub const CURSOR_SHAPE_OFFSET: usize = 64;
/// Total section size.
pub const CURSOR_SHM_SIZE: usize = CURSOR_SHAPE_OFFSET + CURSOR_SHAPE_BYTES;
/// `IDDCX_CURSOR_SHAPE_TYPE` values mirrored for the host (the driver writes the OS value
/// verbatim into [`CursorShm::cursor_type`]).
pub const CURSOR_TYPE_MASKED_COLOR: u32 = 1;
pub const CURSOR_TYPE_ALPHA: u32 = 2;
/// The section header (the shape pixels follow at [`CURSOR_SHAPE_OFFSET`]). `x`/`y` are the
/// shape's TOP-LEFT in desktop coordinates (the IddCx `IDARG_OUT_QUERY_HWCURSOR::X/Y`
/// convention — position hotspot, can be negative); `shape_id` is the OS's per-set counter
/// (bumps on every shape set, the overlay serial); pixels are the OS's 32-bpp rows at
/// `pitch` bytes (BGRA for ALPHA; color+mask for MASKED_COLOR — the host converts).
#[repr(C)]
#[derive(Clone, Copy, Pod, Zeroable, Debug, PartialEq, Eq)]
pub struct CursorShm {
pub magic: u32,
/// Seqlock: odd = writer mid-update.
pub seq: u32,
pub visible: u32,
pub cursor_type: u32,
pub x: i32,
pub y: i32,
pub shape_id: u32,
pub width: u32,
pub height: u32,
pub pitch: u32,
pub hot_x: u32,
pub hot_y: u32,
/// Reserved expansion room up to [`CURSOR_SHAPE_OFFSET`].
pub _reserved: [u32; 4],
}
// Layout is load-bearing across the process boundary — pin it.
const _: () = {
use core::mem::{offset_of, size_of};
assert!(size_of::<CursorShm>() == 64);
assert!(size_of::<CursorShm>() <= CURSOR_SHAPE_OFFSET);
assert!(offset_of!(CursorShm, magic) == 0);
assert!(offset_of!(CursorShm, seq) == 4);
assert!(offset_of!(CursorShm, visible) == 8);
assert!(offset_of!(CursorShm, cursor_type) == 12);
assert!(offset_of!(CursorShm, x) == 16);
assert!(offset_of!(CursorShm, y) == 20);
assert!(offset_of!(CursorShm, shape_id) == 24);
assert!(offset_of!(CursorShm, width) == 28);
assert!(offset_of!(CursorShm, height) == 32);
assert!(offset_of!(CursorShm, pitch) == 36);
assert!(offset_of!(CursorShm, hot_x) == 40);
assert!(offset_of!(CursorShm, hot_y) == 44);
};
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1250,7 +1079,7 @@ mod tests {
max_luminance_nits: 800,
max_frame_avg_nits: 400,
min_luminance_millinits: 50, // 0.05 nits
hw_cursor: 1,
_reserved: 0,
};
let bytes = bytemuck::bytes_of(&req);
assert_eq!(bytes.len(), 40);
@@ -1280,19 +1109,14 @@ mod tests {
target_id: 262,
resolved_monitor_id: 7,
wudf_pid: 4242,
cursor_excluded: 1,
};
let rbytes = bytemuck::bytes_of(&reply);
assert_eq!(rbytes.len(), 24);
assert_eq!(rbytes.len(), 20);
assert_eq!(*bytemuck::from_bytes::<control::AddReply>(rbytes), reply);
// resolved_monitor_id occupies the old `_reserved` slot at offset 12 — byte-compatible.
assert_eq!(rbytes[12..16], 7u32.to_le_bytes());
// The v2 duplication-target pid trails at offset 16.
assert_eq!(rbytes[16..20], 4242u32.to_le_bytes());
// The cursor-excluded tail rides after the legacy boundary; an un-upgraded driver writes
// only the prefix, so a zero-filled tail reads as "unknown/clean" (see the field docs).
assert_eq!(rbytes[20..24], 1u32.to_le_bytes());
assert_eq!(control::ADD_REPLY_LEGACY_SIZE, 20);
}
#[test]
@@ -1337,39 +1161,11 @@ mod tests {
req
);
assert_eq!(bytes[8..12], 2560u32.to_le_bytes());
// The compat window: v4v6 are additive over v3, so the host floor stays at 3.
assert_eq!(PROTOCOL_VERSION, 6);
// The compat window: v4 is additive over v3, so the host floor stays one below.
assert_eq!(PROTOCOL_VERSION, 4);
assert_eq!(MIN_DRIVER_PROTOCOL_VERSION, 3);
}
#[test]
fn cursor_shm_layout_is_pinned() {
use cursor::*;
// The header must leave the shape offset intact whatever grows inside `_reserved`.
assert_eq!(core::mem::size_of::<CursorShm>(), 64);
assert_eq!(CURSOR_SHM_SIZE, 64 + 256 * 256 * 4);
assert_eq!(CURSOR_MAGIC, u32::from_le_bytes(*b"PFCU"));
// Seqlock snapshot discipline survives a bytemuck roundtrip.
let hdr = CursorShm {
magic: CURSOR_MAGIC,
seq: 2,
visible: 1,
cursor_type: CURSOR_TYPE_ALPHA,
x: -3,
y: 7,
shape_id: 42,
width: 32,
height: 32,
pitch: 128,
hot_x: 4,
hot_y: 5,
_reserved: [0; 4],
};
let bytes = bytemuck::bytes_of(&hdr);
assert_eq!(*bytemuck::from_bytes::<CursorShm>(bytes), hdr);
assert_eq!(bytes[16..20], (-3i32).to_le_bytes());
}
#[test]
fn gamepad_names_and_magics_are_stable() {
assert_eq!(gamepad::xusb_boot_name(0), "Global\\pfxusb-boot-0");
@@ -0,0 +1,105 @@
// Cursor-overlay blend kernels for the CUDA/NVENC path (cursor-as-metadata). The cursor bitmap is
// straight-alpha RGBA, row-packed (stride = curW*4). Blended into the encoder-OWNED NVENC input
// surface — never the compositor's dmabuf. One thread per cursor pixel (ARGB / YUV444) or per 2x2
// chroma block (NV12). Coefficients are BT.709 limited, matching rgb2yuv.comp so the cursor colour
// matches the rest of the frame regardless of which zero-copy backend encodes it.
//
// Build (regenerate cursor_blend.ptx after editing):
// nvcc -ptx -arch=compute_75 cursor_blend.cu -o cursor_blend.ptx
// PTX is JIT'd by the driver forward to the actual GPU, so a compute_75 (Turing) baseline runs on
// every Turing-or-newer NVENC GPU. (CUDA 13's nvcc no longer targets pre-Turing archs.)
typedef unsigned char u8;
__device__ __forceinline__ u8 blend8(int dst, int src, int a) {
return (u8)((src * a + dst * (255 - a)) / 255);
}
// Packed 4-byte surface. NVENC's ARGB format stores bytes B,G,R,A in memory; the cursor is R,G,B,A.
extern "C" __global__ void blend_argb(
u8* surf, int pitch, int surfW, int surfH,
const u8* cur, int curW, int curH, int ox, int oy)
{
int cx = blockIdx.x * blockDim.x + threadIdx.x;
int cy = blockIdx.y * blockDim.y + threadIdx.y;
if (cx >= curW || cy >= curH) return;
int px = ox + cx, py = oy + cy;
if (px < 0 || py < 0 || px >= surfW || py >= surfH) return;
const u8* s = cur + (size_t)(cy * curW + cx) * 4;
int a = s[3];
if (a == 0) return;
u8* d = surf + (size_t)py * pitch + (size_t)px * 4;
d[0] = blend8(d[0], s[2], a); // B <- cursor B
d[1] = blend8(d[1], s[1], a); // G <- cursor G
d[2] = blend8(d[2], s[0], a); // R <- cursor R
}
// Planar YUV444: three full-res planes stacked at base, base+plane, base+2*plane (plane=pitch*surfH).
extern "C" __global__ void blend_yuv444(
u8* base, int pitch, int surfW, int surfH,
const u8* cur, int curW, int curH, int ox, int oy)
{
int cx = blockIdx.x * blockDim.x + threadIdx.x;
int cy = blockIdx.y * blockDim.y + threadIdx.y;
if (cx >= curW || cy >= curH) return;
int px = ox + cx, py = oy + cy;
if (px < 0 || py < 0 || px >= surfW || py >= surfH) return;
const u8* s = cur + (size_t)(cy * curW + cx) * 4;
int a = s[3];
if (a == 0) return;
float R = s[0], G = s[1], B = s[2];
int Y = (int)(16.0f + 0.1826f * R + 0.6142f * G + 0.0620f * B + 0.5f);
int U = (int)(128.0f - 0.1006f * R - 0.3386f * G + 0.4392f * B + 0.5f);
int V = (int)(128.0f + 0.4392f * R - 0.3989f * G - 0.0403f * B + 0.5f);
size_t plane = (size_t)pitch * surfH;
u8* yp = base + (size_t)py * pitch + px;
u8* up = base + plane + (size_t)py * pitch + px;
u8* vp = base + 2 * plane + (size_t)py * pitch + px;
*yp = blend8(*yp, Y, a);
*up = blend8(*up, U, a);
*vp = blend8(*vp, V, a);
}
// NV12: full-res Y plane + interleaved half-res UV plane. One thread per 2x2 luma block; each blends
// up to four Y samples and one (alpha-weighted) UV sample.
extern "C" __global__ void blend_nv12(
u8* yb, int yPitch, u8* uvb, int uvPitch, int surfW, int surfH,
const u8* cur, int curW, int curH, int ox, int oy)
{
int bx = blockIdx.x * blockDim.x + threadIdx.x;
int by = blockIdx.y * blockDim.y + threadIdx.y;
int base_cx = bx * 2, base_cy = by * 2;
if (base_cx >= curW || base_cy >= curH) return;
float ua = 0.0f, va = 0.0f, wa = 0.0f;
int cnt = 0;
for (int j = 0; j < 2; j++) {
for (int i = 0; i < 2; i++) {
int cx = base_cx + i, cy = base_cy + j;
if (cx >= curW || cy >= curH) continue;
int px = ox + cx, py = oy + cy;
if (px < 0 || py < 0 || px >= surfW || py >= surfH) continue;
const u8* s = cur + (size_t)(cy * curW + cx) * 4;
int a = s[3];
if (a == 0) continue;
float R = s[0], G = s[1], B = s[2];
int Y = (int)(16.0f + 0.1826f * R + 0.6142f * G + 0.0620f * B + 0.5f);
u8* yp = yb + (size_t)py * yPitch + px;
*yp = blend8(*yp, Y, a);
ua += (128.0f - 0.1006f * R - 0.3386f * G + 0.4392f * B) * a;
va += (128.0f + 0.4392f * R - 0.3989f * G - 0.0403f * B) * a;
wa += a;
cnt++;
}
}
if (wa <= 0.0f || cnt == 0) return;
// The chroma sample covering this block's top-left surface pixel.
int uvx = (ox + base_cx) / 2;
int uvy = (oy + base_cy) / 2;
if (uvx < 0 || uvy < 0 || uvx * 2 >= surfW || uvy * 2 >= surfH) return;
int U = (int)(ua / wa + 0.5f);
int V = (int)(va / wa + 0.5f);
int amean = (int)(wa / cnt + 0.5f);
u8* uv = uvb + (size_t)uvy * uvPitch + (size_t)uvx * 2;
uv[0] = blend8(uv[0], U, amean);
uv[1] = blend8(uv[1], V, amean);
}
@@ -0,0 +1,576 @@
//
// Generated by NVIDIA NVVM Compiler
//
// Compiler Build ID: CL-38244171
// Cuda compilation tools, release 13.3, V13.3.73
// Based on NVVM 7.0.1
//
.version 9.3
.target sm_75
.address_size 64
// .globl blend_argb
.visible .entry blend_argb(
.param .u64 blend_argb_param_0,
.param .u32 blend_argb_param_1,
.param .u32 blend_argb_param_2,
.param .u32 blend_argb_param_3,
.param .u64 blend_argb_param_4,
.param .u32 blend_argb_param_5,
.param .u32 blend_argb_param_6,
.param .u32 blend_argb_param_7,
.param .u32 blend_argb_param_8
)
{
.reg .pred %p<10>;
.reg .b16 %rs<2>;
.reg .b32 %r<34>;
.reg .b64 %rd<17>;
ld.param.u64 %rd2, [blend_argb_param_0];
ld.param.u32 %r5, [blend_argb_param_1];
ld.param.u32 %r6, [blend_argb_param_2];
ld.param.u32 %r7, [blend_argb_param_3];
ld.param.u64 %rd3, [blend_argb_param_4];
ld.param.u32 %r8, [blend_argb_param_5];
ld.param.u32 %r11, [blend_argb_param_6];
ld.param.u32 %r9, [blend_argb_param_7];
ld.param.u32 %r10, [blend_argb_param_8];
mov.u32 %r12, %ntid.x;
mov.u32 %r13, %ctaid.x;
mov.u32 %r14, %tid.x;
mad.lo.s32 %r1, %r13, %r12, %r14;
mov.u32 %r15, %ntid.y;
mov.u32 %r16, %ctaid.y;
mov.u32 %r17, %tid.y;
mad.lo.s32 %r2, %r16, %r15, %r17;
setp.ge.s32 %p1, %r1, %r8;
setp.ge.s32 %p2, %r2, %r11;
or.pred %p3, %p1, %p2;
@%p3 bra $L__BB0_4;
add.s32 %r3, %r1, %r9;
add.s32 %r4, %r2, %r10;
or.b32 %r18, %r4, %r3;
setp.lt.s32 %p4, %r18, 0;
setp.ge.s32 %p5, %r3, %r6;
or.pred %p6, %p5, %p4;
setp.ge.s32 %p7, %r4, %r7;
or.pred %p8, %p7, %p6;
@%p8 bra $L__BB0_4;
mad.lo.s32 %r19, %r2, %r8, %r1;
mul.wide.s32 %rd4, %r19, 4;
cvta.to.global.u64 %rd5, %rd3;
add.s64 %rd1, %rd5, %rd4;
ld.global.u8 %rs1, [%rd1+3];
setp.eq.s16 %p9, %rs1, 0;
@%p9 bra $L__BB0_4;
cvt.u32.u16 %r20, %rs1;
mul.wide.s32 %rd6, %r4, %r5;
mul.wide.s32 %rd7, %r3, 4;
add.s64 %rd8, %rd6, %rd7;
cvta.to.global.u64 %rd9, %rd2;
add.s64 %rd10, %rd9, %rd8;
ld.global.u8 %r21, [%rd10];
ld.global.u8 %r22, [%rd1+2];
xor.b32 %r23, %r20, 255;
mul.lo.s32 %r24, %r23, %r21;
mad.lo.s32 %r25, %r22, %r20, %r24;
mul.wide.u32 %rd11, %r25, -2139062143;
shr.u64 %rd12, %rd11, 39;
st.global.u8 [%rd10], %rd12;
ld.global.u8 %r26, [%rd10+1];
ld.global.u8 %r27, [%rd1+1];
mul.lo.s32 %r28, %r23, %r26;
mad.lo.s32 %r29, %r27, %r20, %r28;
mul.wide.u32 %rd13, %r29, -2139062143;
shr.u64 %rd14, %rd13, 39;
st.global.u8 [%rd10+1], %rd14;
ld.global.u8 %r30, [%rd10+2];
ld.global.u8 %r31, [%rd1];
mul.lo.s32 %r32, %r23, %r30;
mad.lo.s32 %r33, %r31, %r20, %r32;
mul.wide.u32 %rd15, %r33, -2139062143;
shr.u64 %rd16, %rd15, 39;
st.global.u8 [%rd10+2], %rd16;
$L__BB0_4:
ret;
}
// .globl blend_yuv444
.visible .entry blend_yuv444(
.param .u64 blend_yuv444_param_0,
.param .u32 blend_yuv444_param_1,
.param .u32 blend_yuv444_param_2,
.param .u32 blend_yuv444_param_3,
.param .u64 blend_yuv444_param_4,
.param .u32 blend_yuv444_param_5,
.param .u32 blend_yuv444_param_6,
.param .u32 blend_yuv444_param_7,
.param .u32 blend_yuv444_param_8
)
{
.reg .pred %p<10>;
.reg .b16 %rs<5>;
.reg .f32 %f<16>;
.reg .b32 %r<49>;
.reg .b64 %rd<14>;
ld.param.u64 %rd2, [blend_yuv444_param_0];
ld.param.u32 %r5, [blend_yuv444_param_1];
ld.param.u32 %r6, [blend_yuv444_param_2];
ld.param.u32 %r7, [blend_yuv444_param_3];
ld.param.u64 %rd3, [blend_yuv444_param_4];
ld.param.u32 %r8, [blend_yuv444_param_5];
ld.param.u32 %r11, [blend_yuv444_param_6];
ld.param.u32 %r9, [blend_yuv444_param_7];
ld.param.u32 %r10, [blend_yuv444_param_8];
mov.u32 %r12, %ntid.x;
mov.u32 %r13, %ctaid.x;
mov.u32 %r14, %tid.x;
mad.lo.s32 %r1, %r13, %r12, %r14;
mov.u32 %r15, %ntid.y;
mov.u32 %r16, %ctaid.y;
mov.u32 %r17, %tid.y;
mad.lo.s32 %r2, %r16, %r15, %r17;
setp.ge.s32 %p1, %r1, %r8;
setp.ge.s32 %p2, %r2, %r11;
or.pred %p3, %p1, %p2;
@%p3 bra $L__BB1_4;
add.s32 %r3, %r1, %r9;
add.s32 %r4, %r2, %r10;
or.b32 %r18, %r4, %r3;
setp.lt.s32 %p4, %r18, 0;
setp.ge.s32 %p5, %r3, %r6;
or.pred %p6, %p5, %p4;
setp.ge.s32 %p7, %r4, %r7;
or.pred %p8, %p7, %p6;
@%p8 bra $L__BB1_4;
mad.lo.s32 %r19, %r2, %r8, %r1;
mul.wide.s32 %rd4, %r19, 4;
cvta.to.global.u64 %rd5, %rd3;
add.s64 %rd1, %rd5, %rd4;
ld.global.u8 %rs1, [%rd1+3];
setp.eq.s16 %p9, %rs1, 0;
@%p9 bra $L__BB1_4;
cvt.u32.u16 %r20, %rs1;
ld.global.u8 %rs2, [%rd1];
cvt.rn.f32.u16 %f1, %rs2;
ld.global.u8 %rs3, [%rd1+1];
cvt.rn.f32.u16 %f2, %rs3;
ld.global.u8 %rs4, [%rd1+2];
cvt.rn.f32.u16 %f3, %rs4;
fma.rn.f32 %f4, %f1, 0f3E3AFB7F, 0f41800000;
fma.rn.f32 %f5, %f2, 0f3F1D3C36, %f4;
fma.rn.f32 %f6, %f3, 0f3D7DF3B6, %f5;
add.f32 %f7, %f6, 0f3F000000;
cvt.rzi.s32.f32 %r21, %f7;
fma.rn.f32 %f8, %f1, 0fBDCE075F, 0f43000000;
fma.rn.f32 %f9, %f2, 0fBEAD5CFB, %f8;
fma.rn.f32 %f10, %f3, 0f3EE0DED3, %f9;
add.f32 %f11, %f10, 0f3F000000;
cvt.rzi.s32.f32 %r22, %f11;
fma.rn.f32 %f12, %f1, 0f3EE0DED3, 0f43000000;
fma.rn.f32 %f13, %f2, 0fBECC3C9F, %f12;
fma.rn.f32 %f14, %f3, 0fBD25119D, %f13;
add.f32 %f15, %f14, 0f3F000000;
cvt.rzi.s32.f32 %r23, %f15;
mul.wide.s32 %rd6, %r4, %r5;
cvt.s64.s32 %rd7, %r3;
add.s64 %rd8, %rd6, %rd7;
cvta.to.global.u64 %rd9, %rd2;
add.s64 %rd10, %rd9, %rd8;
ld.global.u8 %r24, [%rd10];
mul.lo.s32 %r25, %r21, %r20;
xor.b32 %r26, %r20, 255;
mad.lo.s32 %r27, %r26, %r24, %r25;
mul.hi.s32 %r28, %r27, -2139062143;
add.s32 %r29, %r28, %r27;
shr.u32 %r30, %r29, 31;
shr.u32 %r31, %r29, 7;
add.s32 %r32, %r31, %r30;
st.global.u8 [%rd10], %r32;
mul.wide.s32 %rd11, %r7, %r5;
add.s64 %rd12, %rd10, %rd11;
ld.global.u8 %r33, [%rd12];
mul.lo.s32 %r34, %r22, %r20;
mad.lo.s32 %r35, %r26, %r33, %r34;
mul.hi.s32 %r36, %r35, -2139062143;
add.s32 %r37, %r36, %r35;
shr.u32 %r38, %r37, 31;
shr.u32 %r39, %r37, 7;
add.s32 %r40, %r39, %r38;
st.global.u8 [%rd12], %r40;
add.s64 %rd13, %rd12, %rd11;
ld.global.u8 %r41, [%rd13];
mul.lo.s32 %r42, %r23, %r20;
mad.lo.s32 %r43, %r26, %r41, %r42;
mul.hi.s32 %r44, %r43, -2139062143;
add.s32 %r45, %r44, %r43;
shr.u32 %r46, %r45, 31;
shr.u32 %r47, %r45, 7;
add.s32 %r48, %r47, %r46;
st.global.u8 [%rd13], %r48;
$L__BB1_4:
ret;
}
// .globl blend_nv12
.visible .entry blend_nv12(
.param .u64 blend_nv12_param_0,
.param .u32 blend_nv12_param_1,
.param .u64 blend_nv12_param_2,
.param .u32 blend_nv12_param_3,
.param .u32 blend_nv12_param_4,
.param .u32 blend_nv12_param_5,
.param .u64 blend_nv12_param_6,
.param .u32 blend_nv12_param_7,
.param .u32 blend_nv12_param_8,
.param .u32 blend_nv12_param_9,
.param .u32 blend_nv12_param_10
)
{
.reg .pred %p<43>;
.reg .b16 %rs<17>;
.reg .f32 %f<108>;
.reg .b32 %r<123>;
.reg .b64 %rd<35>;
ld.param.u64 %rd11, [blend_nv12_param_0];
ld.param.u32 %r21, [blend_nv12_param_1];
ld.param.u64 %rd10, [blend_nv12_param_2];
ld.param.u32 %r22, [blend_nv12_param_3];
ld.param.u32 %r23, [blend_nv12_param_4];
ld.param.u32 %r24, [blend_nv12_param_5];
ld.param.u64 %rd12, [blend_nv12_param_6];
ld.param.u32 %r25, [blend_nv12_param_7];
ld.param.u32 %r26, [blend_nv12_param_8];
ld.param.u32 %r27, [blend_nv12_param_9];
ld.param.u32 %r28, [blend_nv12_param_10];
cvta.to.global.u64 %rd1, %rd11;
cvta.to.global.u64 %rd2, %rd12;
mov.u32 %r29, %ntid.x;
mov.u32 %r30, %ctaid.x;
mov.u32 %r31, %tid.x;
mad.lo.s32 %r32, %r30, %r29, %r31;
mov.u32 %r33, %ntid.y;
mov.u32 %r34, %ctaid.y;
mov.u32 %r35, %tid.y;
mad.lo.s32 %r36, %r34, %r33, %r35;
shl.b32 %r1, %r32, 1;
shl.b32 %r2, %r36, 1;
setp.ge.s32 %p1, %r1, %r25;
setp.ge.s32 %p2, %r2, %r26;
or.pred %p3, %p1, %p2;
mov.f32 %f102, 0f00000000;
mov.f32 %f103, 0f00000000;
mov.f32 %f104, 0f00000000;
@%p3 bra $L__BB2_19;
cvt.s64.s32 %rd3, %r21;
add.s32 %r3, %r2, %r28;
setp.ge.s32 %p4, %r3, %r24;
mul.lo.s32 %r4, %r2, %r25;
mul.wide.s32 %rd4, %r3, %r21;
add.s32 %r5, %r1, %r27;
or.b32 %r38, %r5, %r3;
setp.lt.s32 %p5, %r38, 0;
mov.u32 %r121, 0;
setp.ge.s32 %p6, %r5, %r23;
or.pred %p7, %p6, %p5;
or.pred %p8, %p4, %p7;
@%p8 bra $L__BB2_4;
add.s32 %r40, %r1, %r4;
mul.wide.s32 %rd13, %r40, 4;
add.s64 %rd5, %rd2, %rd13;
ld.global.u8 %rs1, [%rd5+3];
setp.eq.s16 %p9, %rs1, 0;
@%p9 bra $L__BB2_4;
cvt.u32.u16 %r42, %rs1;
ld.global.u8 %rs5, [%rd5];
cvt.rn.f32.u16 %f31, %rs5;
ld.global.u8 %rs6, [%rd5+1];
cvt.rn.f32.u16 %f32, %rs6;
ld.global.u8 %rs7, [%rd5+2];
cvt.rn.f32.u16 %f33, %rs7;
fma.rn.f32 %f34, %f31, 0f3E3AFB7F, 0f41800000;
fma.rn.f32 %f35, %f32, 0f3F1D3C36, %f34;
fma.rn.f32 %f36, %f33, 0f3D7DF3B6, %f35;
add.f32 %f37, %f36, 0f3F000000;
cvt.rzi.s32.f32 %r43, %f37;
cvt.s64.s32 %rd14, %r5;
add.s64 %rd15, %rd4, %rd14;
add.s64 %rd16, %rd1, %rd15;
ld.global.u8 %r44, [%rd16];
mul.lo.s32 %r45, %r43, %r42;
xor.b32 %r46, %r42, 255;
mad.lo.s32 %r47, %r46, %r44, %r45;
mul.hi.s32 %r48, %r47, -2139062143;
add.s32 %r49, %r48, %r47;
shr.u32 %r50, %r49, 31;
shr.u32 %r51, %r49, 7;
add.s32 %r52, %r51, %r50;
st.global.u8 [%rd16], %r52;
fma.rn.f32 %f38, %f31, 0fBDCE075F, 0f43000000;
fma.rn.f32 %f39, %f32, 0fBEAD5CFB, %f38;
fma.rn.f32 %f40, %f33, 0f3EE0DED3, %f39;
cvt.rn.f32.u16 %f104, %rs1;
fma.rn.f32 %f102, %f40, %f104, 0f00000000;
fma.rn.f32 %f41, %f31, 0f3EE0DED3, 0f43000000;
fma.rn.f32 %f42, %f32, 0fBECC3C9F, %f41;
fma.rn.f32 %f43, %f33, 0fBD25119D, %f42;
fma.rn.f32 %f103, %f43, %f104, 0f00000000;
mov.u32 %r121, 1;
$L__BB2_4:
add.s32 %r7, %r1, 1;
setp.ge.s32 %p10, %r7, %r25;
@%p10 bra $L__BB2_8;
add.s32 %r8, %r7, %r27;
or.b32 %r53, %r8, %r3;
setp.lt.s32 %p12, %r53, 0;
setp.ge.s32 %p13, %r8, %r23;
or.pred %p14, %p13, %p12;
or.pred %p15, %p4, %p14;
@%p15 bra $L__BB2_8;
add.s32 %r54, %r7, %r4;
mul.wide.s32 %rd17, %r54, 4;
add.s64 %rd6, %rd2, %rd17;
ld.global.u8 %rs2, [%rd6+3];
setp.eq.s16 %p16, %rs2, 0;
@%p16 bra $L__BB2_8;
cvt.u32.u16 %r55, %rs2;
ld.global.u8 %rs8, [%rd6];
cvt.rn.f32.u16 %f44, %rs8;
ld.global.u8 %rs9, [%rd6+1];
cvt.rn.f32.u16 %f45, %rs9;
ld.global.u8 %rs10, [%rd6+2];
cvt.rn.f32.u16 %f46, %rs10;
fma.rn.f32 %f47, %f44, 0f3E3AFB7F, 0f41800000;
fma.rn.f32 %f48, %f45, 0f3F1D3C36, %f47;
fma.rn.f32 %f49, %f46, 0f3D7DF3B6, %f48;
add.f32 %f50, %f49, 0f3F000000;
cvt.rzi.s32.f32 %r56, %f50;
cvt.s64.s32 %rd18, %r8;
add.s64 %rd19, %rd4, %rd18;
add.s64 %rd20, %rd1, %rd19;
ld.global.u8 %r57, [%rd20];
mul.lo.s32 %r58, %r56, %r55;
xor.b32 %r59, %r55, 255;
mad.lo.s32 %r60, %r59, %r57, %r58;
mul.hi.s32 %r61, %r60, -2139062143;
add.s32 %r62, %r61, %r60;
shr.u32 %r63, %r62, 31;
shr.u32 %r64, %r62, 7;
add.s32 %r65, %r64, %r63;
st.global.u8 [%rd20], %r65;
fma.rn.f32 %f51, %f44, 0fBDCE075F, 0f43000000;
fma.rn.f32 %f52, %f45, 0fBEAD5CFB, %f51;
fma.rn.f32 %f53, %f46, 0f3EE0DED3, %f52;
cvt.rn.f32.u16 %f54, %rs2;
fma.rn.f32 %f102, %f53, %f54, %f102;
fma.rn.f32 %f55, %f44, 0f3EE0DED3, 0f43000000;
fma.rn.f32 %f56, %f45, 0fBECC3C9F, %f55;
fma.rn.f32 %f57, %f46, 0fBD25119D, %f56;
fma.rn.f32 %f103, %f57, %f54, %f103;
add.f32 %f104, %f104, %f54;
add.s32 %r121, %r121, 1;
$L__BB2_8:
add.s32 %r11, %r2, 1;
setp.ge.s32 %p17, %r11, %r26;
add.s32 %r12, %r11, %r28;
add.s32 %r13, %r4, %r25;
cvt.s64.s32 %rd21, %r12;
mul.lo.s64 %rd7, %rd21, %rd3;
@%p17 bra $L__BB2_12;
setp.ge.s32 %p18, %r12, %r24;
or.b32 %r66, %r5, %r12;
setp.lt.s32 %p19, %r66, 0;
or.pred %p21, %p6, %p19;
or.pred %p22, %p18, %p21;
@%p22 bra $L__BB2_12;
add.s32 %r67, %r1, %r13;
mul.wide.s32 %rd22, %r67, 4;
add.s64 %rd8, %rd2, %rd22;
ld.global.u8 %rs3, [%rd8+3];
setp.eq.s16 %p23, %rs3, 0;
@%p23 bra $L__BB2_12;
cvt.u32.u16 %r68, %rs3;
ld.global.u8 %rs11, [%rd8];
cvt.rn.f32.u16 %f58, %rs11;
ld.global.u8 %rs12, [%rd8+1];
cvt.rn.f32.u16 %f59, %rs12;
ld.global.u8 %rs13, [%rd8+2];
cvt.rn.f32.u16 %f60, %rs13;
fma.rn.f32 %f61, %f58, 0f3E3AFB7F, 0f41800000;
fma.rn.f32 %f62, %f59, 0f3F1D3C36, %f61;
fma.rn.f32 %f63, %f60, 0f3D7DF3B6, %f62;
add.f32 %f64, %f63, 0f3F000000;
cvt.rzi.s32.f32 %r69, %f64;
cvt.s64.s32 %rd23, %r5;
add.s64 %rd24, %rd7, %rd23;
add.s64 %rd25, %rd1, %rd24;
ld.global.u8 %r70, [%rd25];
mul.lo.s32 %r71, %r69, %r68;
xor.b32 %r72, %r68, 255;
mad.lo.s32 %r73, %r72, %r70, %r71;
mul.hi.s32 %r74, %r73, -2139062143;
add.s32 %r75, %r74, %r73;
shr.u32 %r76, %r75, 31;
shr.u32 %r77, %r75, 7;
add.s32 %r78, %r77, %r76;
st.global.u8 [%rd25], %r78;
fma.rn.f32 %f65, %f58, 0fBDCE075F, 0f43000000;
fma.rn.f32 %f66, %f59, 0fBEAD5CFB, %f65;
fma.rn.f32 %f67, %f60, 0f3EE0DED3, %f66;
cvt.rn.f32.u16 %f68, %rs3;
fma.rn.f32 %f102, %f67, %f68, %f102;
fma.rn.f32 %f69, %f58, 0f3EE0DED3, 0f43000000;
fma.rn.f32 %f70, %f59, 0fBECC3C9F, %f69;
fma.rn.f32 %f71, %f60, 0fBD25119D, %f70;
fma.rn.f32 %f103, %f71, %f68, %f103;
add.f32 %f104, %f104, %f68;
add.s32 %r121, %r121, 1;
$L__BB2_12:
or.pred %p26, %p17, %p10;
@%p26 bra $L__BB2_16;
setp.ge.s32 %p27, %r12, %r24;
add.s32 %r16, %r7, %r27;
or.b32 %r79, %r16, %r12;
setp.lt.s32 %p28, %r79, 0;
setp.ge.s32 %p29, %r16, %r23;
or.pred %p30, %p29, %p28;
or.pred %p31, %p27, %p30;
@%p31 bra $L__BB2_16;
add.s32 %r80, %r7, %r13;
mul.wide.s32 %rd26, %r80, 4;
add.s64 %rd9, %rd2, %rd26;
ld.global.u8 %rs4, [%rd9+3];
setp.eq.s16 %p32, %rs4, 0;
@%p32 bra $L__BB2_16;
cvt.u32.u16 %r81, %rs4;
ld.global.u8 %rs14, [%rd9];
cvt.rn.f32.u16 %f72, %rs14;
ld.global.u8 %rs15, [%rd9+1];
cvt.rn.f32.u16 %f73, %rs15;
ld.global.u8 %rs16, [%rd9+2];
cvt.rn.f32.u16 %f74, %rs16;
fma.rn.f32 %f75, %f72, 0f3E3AFB7F, 0f41800000;
fma.rn.f32 %f76, %f73, 0f3F1D3C36, %f75;
fma.rn.f32 %f77, %f74, 0f3D7DF3B6, %f76;
add.f32 %f78, %f77, 0f3F000000;
cvt.rzi.s32.f32 %r82, %f78;
cvt.s64.s32 %rd27, %r16;
add.s64 %rd28, %rd7, %rd27;
add.s64 %rd29, %rd1, %rd28;
ld.global.u8 %r83, [%rd29];
mul.lo.s32 %r84, %r82, %r81;
xor.b32 %r85, %r81, 255;
mad.lo.s32 %r86, %r85, %r83, %r84;
mul.hi.s32 %r87, %r86, -2139062143;
add.s32 %r88, %r87, %r86;
shr.u32 %r89, %r88, 31;
shr.u32 %r90, %r88, 7;
add.s32 %r91, %r90, %r89;
st.global.u8 [%rd29], %r91;
fma.rn.f32 %f79, %f72, 0fBDCE075F, 0f43000000;
fma.rn.f32 %f80, %f73, 0fBEAD5CFB, %f79;
fma.rn.f32 %f81, %f74, 0f3EE0DED3, %f80;
cvt.rn.f32.u16 %f82, %rs4;
fma.rn.f32 %f102, %f81, %f82, %f102;
fma.rn.f32 %f83, %f72, 0f3EE0DED3, 0f43000000;
fma.rn.f32 %f84, %f73, 0fBECC3C9F, %f83;
fma.rn.f32 %f85, %f74, 0fBD25119D, %f84;
fma.rn.f32 %f103, %f85, %f82, %f103;
add.f32 %f104, %f104, %f82;
add.s32 %r121, %r121, 1;
$L__BB2_16:
setp.eq.s32 %p33, %r121, 0;
setp.le.f32 %p34, %f104, 0f00000000;
or.pred %p35, %p34, %p33;
@%p35 bra $L__BB2_19;
shr.u32 %r92, %r5, 31;
add.s32 %r93, %r5, %r92;
shr.s32 %r19, %r93, 1;
setp.lt.s32 %p36, %r3, -1;
setp.lt.s32 %p37, %r5, -1;
or.pred %p38, %p37, %p36;
and.b32 %r94, %r93, -2;
setp.ge.s32 %p39, %r94, %r23;
or.pred %p40, %p38, %p39;
shr.u32 %r95, %r3, 31;
add.s32 %r96, %r3, %r95;
shr.s32 %r20, %r96, 1;
and.b32 %r97, %r96, -2;
setp.ge.s32 %p41, %r97, %r24;
or.pred %p42, %p40, %p41;
@%p42 bra $L__BB2_19;
div.rn.f32 %f86, %f102, %f104;
add.f32 %f87, %f86, 0f3F000000;
cvt.rzi.s32.f32 %r98, %f87;
div.rn.f32 %f88, %f103, %f104;
add.f32 %f89, %f88, 0f3F000000;
cvt.rzi.s32.f32 %r99, %f89;
cvt.rn.f32.s32 %f90, %r121;
div.rn.f32 %f91, %f104, %f90;
add.f32 %f92, %f91, 0f3F000000;
cvt.rzi.s32.f32 %r100, %f92;
mul.wide.s32 %rd30, %r20, %r22;
mul.wide.s32 %rd31, %r19, 2;
add.s64 %rd32, %rd30, %rd31;
cvta.to.global.u64 %rd33, %rd10;
add.s64 %rd34, %rd33, %rd32;
ld.global.u8 %r101, [%rd34];
mul.lo.s32 %r102, %r100, %r98;
mov.u32 %r103, 255;
sub.s32 %r104, %r103, %r100;
mad.lo.s32 %r105, %r104, %r101, %r102;
mul.hi.s32 %r106, %r105, -2139062143;
add.s32 %r107, %r106, %r105;
shr.u32 %r108, %r107, 31;
shr.u32 %r109, %r107, 7;
add.s32 %r110, %r109, %r108;
st.global.u8 [%rd34], %r110;
ld.global.u8 %r111, [%rd34+1];
mul.lo.s32 %r112, %r100, %r99;
mad.lo.s32 %r113, %r104, %r111, %r112;
mul.hi.s32 %r114, %r113, -2139062143;
add.s32 %r115, %r114, %r113;
shr.u32 %r116, %r115, 31;
shr.u32 %r117, %r115, 7;
add.s32 %r118, %r117, %r116;
st.global.u8 [%rd34+1], %r118;
$L__BB2_19:
ret;
}
+98 -224
View File
@@ -69,7 +69,6 @@ use super::{AuChunk, ChromaFormat, Codec, EncodedFrame, Encoder, EncoderCaps};
use anyhow::{anyhow, bail, Context, Result};
use pf_frame::{CapturedFrame, FramePayload};
use pf_zerocopy::cuda::{self, InputSurface};
use pf_zerocopy::vkslot::{SlotFormat, VkSlotBlend, VkSlotRef};
use std::collections::VecDeque;
use std::ffi::c_void;
use std::ptr;
@@ -77,6 +76,12 @@ use std::sync::mpsc;
use nvidia_video_codec_sdk::sys::nvEncodeAPI as nv;
/// Prebuilt PTX for the cursor-overlay blend kernels (cursor-as-metadata). Source is
/// `cursor_blend.cu` beside this file; regenerate with
/// `nvcc -ptx -arch=compute_75 cursor_blend.cu -o cursor_blend.ptx` after editing. JIT'd by the
/// driver, so it runs on any Turing-or-newer GPU.
const CURSOR_PTX: &[u8] = include_bytes!("cursor_blend.ptx");
// ---------------------------------------------------------------------------------------------
// Runtime-loaded NVENC entry table (Linux). Same shape as the Windows backend's `EncodeApi`, minus
// the async-event entry points (Windows-only). Resolved once from `libnvidia-encode.so.1` — the two
@@ -423,54 +428,13 @@ fn buffer_format(buf: &cuda::DeviceBuffer) -> nv::NV_ENC_BUFFER_FORMAT {
}
}
/// One encoder-owned input surface + its NVENC registration. The surface is copied into each
/// One encoder-owned CUDA input surface + its NVENC registration. The surface is copied into each
/// use (device→device) and the registration is created once at session init, unregistered at teardown.
struct RingSlot {
surface: SlotSurface,
surface: InputSurface,
reg: nv::NV_ENC_REGISTERED_PTR,
}
/// The ring slot's backing allocation: Vulkan external memory CUDA-imported (the normal case —
/// blendable by the SPIR-V cursor pass, see `vkslot.rs`) or a plain pitched CUDA allocation (the
/// fallback when Vulkan bring-up fails: sessions still encode, composite mode just has no
/// cursor). Both present the same `(ptr, pitch, height)` NVENC-registration vocabulary.
enum SlotSurface {
Cuda(InputSurface),
/// Backing objects live in the encoder's [`VkSlotBlend`] (freed by its `free_slots`); the
/// ref itself is Copy and carries the registered geometry.
Vk(VkSlotRef),
}
/// The [`SlotFormat`] for an NVENC buffer format (the ring-build + blend vocabulary).
fn slot_fmt_of(fmt: nv::NV_ENC_BUFFER_FORMAT) -> SlotFormat {
match fmt {
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_YUV444 => SlotFormat::Yuv444,
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_NV12 => SlotFormat::Nv12,
_ => SlotFormat::Argb,
}
}
impl SlotSurface {
fn ptr(&self) -> pf_zerocopy::cuda::CUdeviceptr {
match self {
SlotSurface::Cuda(s) => s.ptr,
SlotSurface::Vk(r) => r.ptr,
}
}
fn pitch(&self) -> usize {
match self {
SlotSurface::Cuda(s) => s.pitch,
SlotSurface::Vk(r) => r.pitch,
}
}
fn height(&self) -> u32 {
match self {
SlotSurface::Cuda(s) => s.height,
SlotSurface::Vk(r) => r.height,
}
}
}
/// `doNotWait` sampling cadence inside [`Encoder::poll_chunk`] — the probe measured ~200 µs
/// between slice completions on the 5070 Ti, so 50 µs keeps the added per-chunk delivery delay
/// well under one slice time without hammering the driver.
@@ -566,23 +530,16 @@ pub struct NvencCudaEncoder {
split_mode: u32,
/// The last reference-frame range we invalidated — dedupes repeated RFI requests for one loss.
last_rfi_range: Option<(i64, i64)>,
/// Cursor-as-metadata GPU blend: the Vulkan device + SPIR-V compute pass the ring's
/// external-memory slots are allocated through (`vkslot.rs`) — the driver-portable
/// replacement for the retired PTX kernels. Brought up once at session init (`cursor_tried`
/// stops re-attempts); `None` = bring-up failed, the ring fell back to plain CUDA
/// allocations and composite mode degrades to no cursor. `cursor_serial` tracks the
/// uploaded bitmap.
vk_blend: Option<VkSlotBlend>,
/// The session may hand this encoder cursor overlays (`SessionPlan.cursor_blend` — only
/// cursor-channel sessions since Phase B). Off = skip the Vulkan bring-up entirely and ring
/// on plain CUDA surfaces: embedded-pointer sessions never carry an overlay, so they pay
/// zero blend cost, per-session or per-frame.
blend_wanted: bool,
/// Cursor-as-metadata GPU blend (loaded lazily on the first frame that carries a cursor, once the
/// CUDA context is current). `None` until then or if the module load fails; `cursor_tried` stops
/// re-attempting a failed load every frame. `cursor_serial` tracks the uploaded bitmap.
cursor: Option<cuda::CursorBlend>,
cursor_tried: bool,
cursor_serial: u64,
/// Suppress-until-success latch for the per-frame blend warn: a persistent failure sits in
/// the submit() hot path, so warn once per failure streak (reset on success) rather than on
/// every cursor-bearing frame, which would evict the log ring.
/// Suppress-until-success latches for the per-frame cursor upload/blend warns: a persistent
/// failure sits in the submit() hot path, so warn once per failure streak (reset on success)
/// rather than on every cursor-bearing frame, which would evict the log ring.
cursor_upload_warned: bool,
cursor_blend_warned: bool,
/// One-shot latch for [`diagnose_failed_open`](Self::diagnose_failed_open) so a rebuild-retry
/// burst (the session loop's bounded encoder resets) logs the diagnosis once, not per attempt.
@@ -650,7 +607,6 @@ impl NvencCudaEncoder {
_cuda: bool,
bit_depth: u8,
chroma: ChromaFormat,
cursor_blend: bool,
) -> Result<Self> {
// The runtime `.so` load is the real "is NVENC possible here" gate: fail the open with a
// clear reason instead of an opaque session error on the first frame.
@@ -689,10 +645,10 @@ impl NvencCudaEncoder {
frame_idx: 0,
force_kf: false,
pending_anchor: false,
vk_blend: None,
blend_wanted: cursor_blend,
cursor: None,
cursor_tried: false,
cursor_serial: u64::MAX,
cursor_upload_warned: false,
cursor_blend_warned: false,
diagnosed: false,
inited: false,
@@ -783,12 +739,7 @@ impl NvencCudaEncoder {
// (the forfeit contract), and the next session re-latches the arming at init.
self.subframe_chunks = false;
self.chunk = None;
self.ring.clear(); // drops the CUDA InputSurfaces; Vk slots are freed just below
if let Some(vk) = &mut self.vk_blend {
// The Vulkan-backed slots' memory (and its CUDA mapping) — the device itself stays
// up for the next session's ring (`cursor_tried` keeps bring-up one-shot).
vk.free_slots();
}
self.ring.clear(); // drops the InputSurfaces, freeing their CUDA allocations
self.bitstreams.clear();
self.pending.clear();
self.encoder = ptr::null_mut();
@@ -1177,102 +1128,36 @@ impl NvencCudaEncoder {
// Encoder-owned input-surface ring: allocate + register POOL surfaces in the negotiated
// format. Registered once here, mapped per submit, unregistered at teardown.
// Preferred backing = Vulkan external memory CUDA-imported (`vkslot.rs`), so the
// SPIR-V cursor blend can composite into the very bytes NVENC encodes; any bring-up
// or per-slot failure falls back to plain pitched CUDA allocations (sessions always
// encode — composite mode just loses the cursor, warned below).
if !self.cursor_tried && self.blend_wanted {
self.cursor_tried = true;
match VkSlotBlend::new() {
Ok(v) => self.vk_blend = Some(v),
Err(e) => tracing::warn!(
error = %format!("{e:#}"),
"NVENC (Linux): Vulkan slot-blend bring-up failed — plain CUDA input \
surfaces, cursor compositing unavailable"
),
}
}
let slot_fmt = slot_fmt_of(self.buffer_fmt);
// Two attempts: the full ring on Vulkan slots, else (any failure) the full ring on
// plain CUDA — never a mixed ring (it would blend on some slots only: a flickering
// cursor) and never a short one.
'ring: for use_vk in [self.vk_blend.is_some(), false] {
if !use_vk && self.vk_blend.is_some() {
// Second attempt: retire the Vulkan side wholesale first.
for s in self.ring.drain(..) {
let _ = (api().unregister_resource)(self.encoder, s.reg);
for _ in 0..POOL {
let surface = match self.buffer_fmt {
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_YUV444 => {
InputSurface::alloc_yuv444(self.width, self.height)
}
if let Some(vk) = &mut self.vk_blend {
vk.free_slots();
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_NV12 => {
InputSurface::alloc_nv12(self.width, self.height)
}
self.vk_blend = None;
_ => InputSurface::alloc_rgb(self.width, self.height),
}
for _ in 0..POOL {
let surface = if use_vk {
let vk = self.vk_blend.as_mut().expect("use_vk implies Some");
match vk.alloc_slot(slot_fmt, self.width, self.height) {
Ok(r) => SlotSurface::Vk(r),
Err(e) => {
tracing::warn!(
error = %format!("{e:#}"),
"NVENC (Linux): Vulkan slot alloc failed — rebuilding the \
ring on plain CUDA surfaces (cursor compositing \
unavailable)"
);
continue 'ring;
}
}
} else {
SlotSurface::Cuda(
match self.buffer_fmt {
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_YUV444 => {
InputSurface::alloc_yuv444(self.width, self.height)
}
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_NV12 => {
InputSurface::alloc_nv12(self.width, self.height)
}
_ => InputSurface::alloc_rgb(self.width, self.height),
}
.context("alloc NVENC input surface")?,
)
};
let mut rr = nv::NV_ENC_REGISTER_RESOURCE {
version: nv::NV_ENC_REGISTER_RESOURCE_VER,
resourceType:
nv::NV_ENC_INPUT_RESOURCE_TYPE::NV_ENC_INPUT_RESOURCE_TYPE_CUDADEVICEPTR,
width: self.width,
height: self.height,
pitch: surface.pitch() as u32,
resourceToRegister: surface.ptr() as *mut c_void,
bufferFormat: self.buffer_fmt,
bufferUsage: nv::NV_ENC_BUFFER_USAGE::NV_ENC_INPUT_IMAGE,
..Default::default()
};
match (api().register_resource)(self.encoder, &mut rr).nv_ok() {
Ok(()) => {}
Err(e) if use_vk => {
// NVENC refusing the imported pointer is a Vulkan-side condition
// too — same wholesale fallback.
tracing::warn!(
error = ?e,
"NVENC (Linux): registering a Vulkan-imported slot failed — \
rebuilding the ring on plain CUDA surfaces"
);
continue 'ring;
}
Err(e) => {
return Err(nvenc_status::call_err(
"register_resource (CUDADEVICEPTR)",
e,
))
}
}
self.ring.push(RingSlot {
surface,
reg: rr.registeredResource,
});
}
break 'ring; // full ring built
.context("alloc NVENC input surface")?;
let mut rr = nv::NV_ENC_REGISTER_RESOURCE {
version: nv::NV_ENC_REGISTER_RESOURCE_VER,
resourceType:
nv::NV_ENC_INPUT_RESOURCE_TYPE::NV_ENC_INPUT_RESOURCE_TYPE_CUDADEVICEPTR,
width: self.width,
height: self.height,
pitch: surface.pitch as u32,
resourceToRegister: surface.ptr as *mut c_void,
bufferFormat: self.buffer_fmt,
bufferUsage: nv::NV_ENC_BUFFER_USAGE::NV_ENC_INPUT_IMAGE,
..Default::default()
};
(api().register_resource)(self.encoder, &mut rr)
.nv_ok()
.map_err(|e| nvenc_status::call_err("register_resource (CUDADEVICEPTR)", e))?;
self.ring.push(RingSlot {
surface,
reg: rr.registeredResource,
});
}
self.inited = true;
@@ -1357,9 +1242,9 @@ impl NvencCudaEncoder {
/// IO-stream binding (stream-ordered submit — see the gate in [`Encoder::submit`]).
fn copy_into_slot(&self, buf: &cuda::DeviceBuffer, slot: usize, sync: bool) -> Result<()> {
let s = &self.ring[slot].surface;
let base = s.ptr();
let pitch = s.pitch();
let hh = s.height() as u64;
let base = s.ptr;
let pitch = s.pitch;
let hh = s.height as u64;
match self.buffer_fmt {
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_YUV444 => {
if !buf.yuv444 {
@@ -1511,83 +1396,72 @@ impl Encoder for NvencCudaEncoder {
// `async_rt` must be absent too: in two-thread mode the frame may be recycled right after
// submit returns while the stream still holds its copy (belt-and-braces — an escalated
// session was rebuilt without the binding, so `stream_ordered` is false there anyway).
// Cursor-bearing frames additionally force the CPU-synced path: the Vulkan blend sits
// between the CUDA copy and the encode, and its cross-API ordering is fence/CPU-
// established, not stream-ordered. Frames without a cursor (games hide it; client-draws
// sessions strip it) keep the stream-ordered fast path untouched.
let ordered = self.stream_ordered
&& self.async_rt.is_none()
&& self.pending.is_empty()
&& captured.cursor.is_none();
let ordered = self.stream_ordered && self.async_rt.is_none() && self.pending.is_empty();
let t0 = std::time::Instant::now();
// Copy the captured buffer into this slot's input surface before encoding it.
self.copy_into_slot(buf, slot, !ordered)?;
let t_copy = t0.elapsed();
// Cursor-as-metadata: blend the overlay into this slot's OWNED input surface via the
// SPIR-V compute pass (a dispatch over the cursor's rect — never the compositor's
// dmabuf). Cursor-bearing frames forced `ordered = false` above, so the CUDA copy has
// completed before the Vulkan dispatch and the fence-waited dispatch completes before
// the encode below — the cross-API ordering is CPU-established. Any failure degrades to
// no cursor, never a dropped frame.
// Cursor-as-metadata: blend the overlay into this slot's OWNED input surface (a tiny kernel
// over the cursor's rect — never the compositor's dmabuf). The PTX module loads lazily on the
// first cursor frame now that the CUDA context is current; any failure degrades to no cursor,
// never a dropped frame.
if let Some(ov) = &captured.cursor {
if let (Some(vk), SlotSurface::Vk(vref)) =
(self.vk_blend.as_mut(), &self.ring[slot].surface)
{
if self.cursor_serial != ov.serial {
vk.upload_cursor(ov.rgba.as_slice(), ov.w, ov.h);
self.cursor_serial = ov.serial;
if !self.cursor_tried {
self.cursor_tried = true;
match cuda::CursorBlend::new(CURSOR_PTX) {
Ok(cb) => self.cursor = Some(cb),
Err(e) => tracing::warn!(
error = %format!("{e:#}"),
"NVENC (Linux): cursor blend module load failed — cursor not composited"
),
}
// surfW = content width; the blend derives plane strides from the slot's luma
// height. Cursor pixels past the content land in cropped padding rows — harmless.
let r = vk.blend_ref(
vref,
slot_fmt_of(self.buffer_fmt),
self.width,
ov.w,
ov.h,
ov.x,
ov.y,
);
}
if let Some(cb) = &self.cursor {
if self.cursor_serial != ov.serial {
match cb.upload(ov.rgba.as_slice(), ov.w, ov.h) {
Ok(()) => {
self.cursor_serial = ov.serial;
self.cursor_upload_warned = false;
}
Err(e) => {
if !self.cursor_upload_warned {
self.cursor_upload_warned = true;
tracing::warn!(
error = %format!("{e:#}"),
serial = ov.serial,
"NVENC (Linux): cursor upload failed — cursor not composited"
);
}
}
}
}
let s = &self.ring[slot].surface;
// surfW = content width; surfH = the surface's allocated height (matches
// `copy_into_slot`'s plane math). Cursor pixels past the content are in cropped
// padding rows — harmless.
let (w, h) = (self.width, s.height);
let r = match self.buffer_fmt {
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_YUV444 => {
cb.blend_yuv444(s.ptr, s.pitch, w, h, ov.w, ov.h, ov.x, ov.y, !ordered)
}
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_NV12 => {
cb.blend_nv12(s.ptr, s.pitch, w, h, ov.w, ov.h, ov.x, ov.y, !ordered)
}
_ => cb.blend_argb(s.ptr, s.pitch, w, h, ov.w, ov.h, ov.x, ov.y, !ordered),
};
if let Err(e) = r {
if !self.cursor_blend_warned {
self.cursor_blend_warned = true;
tracing::warn!(
error = %format!("{e:#}"),
"NVENC (Linux): cursor blend dispatch failed — cursor not composited"
"NVENC (Linux): cursor blend launch failed — cursor not composited"
);
}
} else {
self.cursor_blend_warned = false;
// TEMP KWin composite probe (DROP BEFORE MERGE): prove the blend dispatches
// and with what geometry.
{
use std::sync::atomic::{AtomicU64, Ordering as ProbeOrd};
static PROBE_BLEND: AtomicU64 = AtomicU64::new(0);
let n = PROBE_BLEND.fetch_add(1, ProbeOrd::Relaxed) + 1;
if n == 1 || n % 512 == 0 {
tracing::info!(
n,
fmt = ?self.buffer_fmt,
ov_x = ov.x,
ov_y = ov.y,
ov_w = ov.w,
ov_h = ov.h,
visible = ov.visible,
"cursor blend probe: vulkan dispatch submitted"
);
}
}
}
} else if !self.cursor_blend_warned {
self.cursor_blend_warned = true;
tracing::warn!(
blend_wanted = self.blend_wanted,
"NVENC (Linux): cursor overlay present but no Vulkan blend (bring-up failed, \
or a non-blend session unexpectedly carried an overlay) cursor not \
composited"
);
}
}
@@ -1635,7 +1509,7 @@ impl Encoder for NvencCudaEncoder {
version: nv::NV_ENC_PIC_PARAMS_VER,
inputWidth: self.width,
inputHeight: self.height,
inputPitch: self.ring[slot].surface.pitch() as u32,
inputPitch: self.ring[slot].surface.pitch as u32,
inputBuffer: mp.mappedResource,
bufferFmt: mp.mappedBufferFmt,
outputBitstream: self.bitstreams[slot],
+6 -11
View File
@@ -270,7 +270,7 @@ fn open_video_backend(
chroma: ChromaFormat,
cursor_blend: bool,
) -> Result<(Box<dyn Encoder>, &'static str)> {
let _ = cursor_blend; // consumed only by the Linux vulkan-encode + direct-NVENC arms below
let _ = cursor_blend; // consumed only by the Linux vulkan-encode arm below
validate_dimensions(codec, width, height)?;
// Refresh/fps must be positive and sane: fps feeds the encoder time_base (`Rational(1, fps)`)
// and the pts→ns conversion (`pts * 1e9 / fps`), so 0 builds a 1/0 rational / divides by zero.
@@ -394,7 +394,6 @@ fn open_video_backend(
cuda,
bit_depth,
chroma,
cursor_blend,
)
.map(|e| (e, "nvenc"))
};
@@ -730,15 +729,12 @@ fn open_nvenc_probed(
cuda: bool,
bit_depth: u8,
chroma: ChromaFormat,
cursor_blend: bool,
) -> Result<Box<dyn Encoder>> {
#[cfg(not(feature = "nvenc"))]
let _ = cursor_blend; // consumed by the direct-SDK arm below
// Direct-SDK NVENC (design/linux-direct-nvenc.md): the DEFAULT on NVIDIA, and only for a CUDA
// capture payload (it registers CUDADEVICEPTR inputs — a CPU/dmabuf frame can't feed it, so those
// keep the libav path; and `cuda` is false on AMD/Intel, so they stay on VAAPI). Set
// PUNKTFUNK_NVENC_DIRECT=0 to fall back to libav. It self-clamps the bitrate internally (its own
// level-ceiling binary search at session open), so it skips the probe-loop stepping below.
// Direct-SDK NVENC (design/linux-direct-nvenc.md): the DEFAULT on NVIDIA, and only for a CUDA
// capture payload (it registers CUDADEVICEPTR inputs — a CPU/dmabuf frame can't feed it, so those
// keep the libav path; and `cuda` is false on AMD/Intel, so they stay on VAAPI). Set
// PUNKTFUNK_NVENC_DIRECT=0 to fall back to libav. It self-clamps the bitrate internally (its own
// level-ceiling binary search at session open), so it skips the probe-loop stepping below.
#[cfg(feature = "nvenc")]
if cuda && nvenc_direct_enabled() {
tracing::info!(
@@ -755,7 +751,6 @@ fn open_nvenc_probed(
cuda,
bit_depth,
chroma,
cursor_blend,
)?) as Box<dyn Encoder>);
}
// The silent-degrade trap: a build without `--features nvenc` compiles the direct-SDK
-7
View File
@@ -32,13 +32,6 @@ pub struct WinCaptureTarget {
/// duplicates the sealed frame channel's handles INTO (`idd_push::ChannelBroker`). `0` = unknown
/// (a pre-v2 pairing can't occur — the version handshake is hard — so this only guards misuse).
pub wudf_pid: u32,
/// The ADD reply flagged the ADAPTER as carrying an IRREVOCABLE IddCx hardware-cursor declare
/// from an earlier session (remote-desktop-sweep §8.6; reach is adapter-wide, not per-target —
/// on-glass 2026-07-23, a GameStream session's fresh target streamed cursor-less): DWM
/// excludes the pointer from its frames until adapter reset, so a session WITHOUT the cursor
/// channel must self-composite (the IDD-push capturer's forced-composite gate) or the
/// streamed desktop has no cursor at all.
pub cursor_excluded: bool,
}
/// The PyroWave (Windows) zero-copy sharing payload attached to a captured frame: the SECOND plane
-18
View File
@@ -152,12 +152,6 @@ pub struct OutputFormat {
/// codec's backend, and the fallback family) would misread the two-plane buffer as packed
/// RGB. Always `false` on Windows (the IDD-push capturer owns its own formats).
pub nv12_native: bool,
/// The session negotiated the cursor-forward channel (remote-desktop-sweep M2c): on Windows
/// the IDD-push capturer creates + delivers the driver's hardware-cursor section, so DWM
/// stops compositing the pointer into the frames and the capturer surfaces it via
/// `Capturer::cursor()` instead. Ignored on Linux (the portal's `SPA_META_Cursor` already
/// separates the pointer; the session plan's `cursor_blend` gate handles the rest).
pub hw_cursor: bool,
}
impl OutputFormat {
@@ -175,8 +169,6 @@ impl OutputFormat {
chroma_444: false,
// GameStream never negotiates PyroWave (native punktfunk/1 only).
pyrowave: false,
// GameStream/spike sessions never negotiate the cursor channel.
hw_cursor: false,
// Conservative: the GameStream + spike paths don't resolve the codec here, and a
// Moonlight client may negotiate H264 (whose VAAPI backend can't ingest NV12) — so
// they never prefer the producer-native NV12 pod. The punktfunk/1 plane opts in via
@@ -204,16 +196,6 @@ pub struct CursorOverlay {
pub rgba: std::sync::Arc<Vec<u8>>,
/// Bumps whenever `rgba`/`w`/`h` change; stable across position-only moves.
pub serial: u64,
/// Hotspot (the pixel that IS the pointer position) within `w`×`h`. The blend paths ignore
/// it (`x`/`y` are already hotspot-adjusted); the cursor-forward channel ships it to the
/// client so a locally-drawn OS cursor points with the right pixel.
pub hot_x: u32,
pub hot_y: u32,
/// Compositor-reported pointer visibility. `false` = an app on the host grabbed/hid the
/// pointer — the cursor-forward channel turns that into the client's relative-mode hint
/// (remote-desktop-sweep M3). The encode loop STRIPS invisible overlays before the frame
/// reaches any blend path, so encoders may keep treating `Some` as "draw it".
pub visible: bool,
}
/// A captured frame. [`format`](Self::format)/dimensions describe the pixels regardless of
-5
View File
@@ -61,10 +61,5 @@ windows = { version = "0.62", features = [
"Win32_System_StationsAndDesktops",
"Win32_System_Threading",
"Win32_UI_Input_KeyboardAndMouse",
# Synthetic pointer devices (PT_PEN / PT_TOUCH) — the pen/touch injectors. The device
# create/destroy + POINTER_TYPE_INFO live under Controls in this metadata layout;
# InjectSyntheticPointerInput + POINTER_*_INFO under Input_Pointer.
"Win32_UI_Controls",
"Win32_UI_Input_Pointer",
"Win32_UI_WindowsAndMessaging",
] }
-6
View File
@@ -36,12 +36,6 @@ pub fn vk_to_evdev(vk: u8) -> Option<u16> {
0x2D => Some(110), // VK_INSERT -> KEY_INSERT
0x2E => Some(111), // VK_DELETE -> KEY_DELETE
// --- Consumer/media keys (Android TV remotes, keyboard media rows) ---
0xB0 => Some(163), // VK_MEDIA_NEXT_TRACK -> KEY_NEXTSONG
0xB1 => Some(165), // VK_MEDIA_PREV_TRACK -> KEY_PREVIOUSSONG
0xB2 => Some(166), // VK_MEDIA_STOP -> KEY_STOPCD
0xB3 => Some(164), // VK_MEDIA_PLAY_PAUSE -> KEY_PLAYPAUSE
// --- Generic modifiers ---
0x10 => Some(42), // VK_SHIFT -> KEY_LEFTSHIFT
0x11 => Some(29), // VK_CONTROL -> KEY_LEFTCTRL
@@ -424,9 +424,6 @@ impl InputInjector for KwinFakeInjector {
self.fake.touch_up(event.code);
self.fake.touch_frame();
}
// fake_input can only press host-layout keycodes — no committed-text path (the
// HOST_CAP_TEXT_INPUT cap is not advertised on this backend).
InputKind::TextInput => {}
// Gamepads are injected through uinput, not the compositor.
InputKind::GamepadState
| InputKind::GamepadButton
+30 -205
View File
@@ -109,7 +109,7 @@ async fn session_main(mut rx: UnboundedReceiver<InputEvent>, source: EiSource) {
// Keep `_rd`/`_session` bound for the whole loop — dropping the portal session closes the
// EIS connection. Bound the setup so a headless approval dialog (un-bypassed grant) can't
// hang the worker forever.
let (_keepalive, context, mut events, output_hint) = match tokio::time::timeout(
let (_keepalive, context, mut events) = match tokio::time::timeout(
Duration::from_secs(30),
connect(source),
)
@@ -130,7 +130,6 @@ async fn session_main(mut rx: UnboundedReceiver<InputEvent>, source: EiSource) {
tracing::info!("libei: EIS connected — awaiting devices");
let mut state = EiState::new();
state.output_hint = output_hint;
// Watchdog: a healthy EIS server adds + resumes an input device within a beat of the handshake.
// If none has resumed by this deadline, the connection is dead-on-arrival (stale/half-ready
// gamescope socket the handshake passed but no real server is behind) — exit so the next
@@ -178,29 +177,21 @@ type Connected = (
Box<dyn Send>,
ei::Context,
reis::tokio::EiConvertEventStream,
// The compositor's output size ("WxH" relay-file hint) — the scale target for absolute
// coordinates when the EIS advertises only a degenerate region (gamescope). `None` for
// the portal/Mutter paths, whose regions are real.
Option<(u32, u32)>,
);
/// Reach an EIS server per `source` and run the EI sender handshake.
async fn connect(source: EiSource) -> Result<Connected> {
let (keepalive, stream, output_hint): (Box<dyn Send>, UnixStream, Option<(u32, u32)>) =
match source {
EiSource::Portal => {
let (rd, session, fd) = connect_portal().await?;
(Box::new((rd, session)), UnixStream::from(fd), None)
}
EiSource::MutterEis => {
let (keepalive, fd) = connect_mutter().await?;
(keepalive, UnixStream::from(fd), None)
}
EiSource::SocketPathFile(file) => {
let (stream, hint) = connect_socket_file(&file).await?;
(Box::new(()), stream, hint)
}
};
let (keepalive, stream): (Box<dyn Send>, UnixStream) = match source {
EiSource::Portal => {
let (rd, session, fd) = connect_portal().await?;
(Box::new((rd, session)), UnixStream::from(fd))
}
EiSource::MutterEis => {
let (keepalive, fd) = connect_mutter().await?;
(keepalive, UnixStream::from(fd))
}
EiSource::SocketPathFile(file) => (Box::new(()), connect_socket_file(&file).await?),
};
let context = ei::Context::new(stream).map_err(|e| anyhow!("reis EI context: {e}"))?;
// Bound the handshake. `UnixStream::connect` to a socket *file* succeeds the moment the path
// exists, but a stale/half-ready gamescope (its socket created early in startup, or left behind
@@ -215,7 +206,7 @@ async fn connect(source: EiSource) -> Result<Connected> {
anyhow!("EI handshake timed out (EIS server not responding — stale/half-ready socket?)")
})?
.map_err(|e| anyhow!("EI handshake: {e}"))?;
Ok((keepalive, context, events, output_hint))
Ok((keepalive, context, events))
}
/// Open a RemoteDesktop portal session (pointer + keyboard) and obtain the EIS socket fd.
@@ -303,10 +294,8 @@ async fn connect_mutter() -> Result<(Box<dyn Send>, std::os::fd::OwnedFd)> {
/// Poll `file` for the EIS socket path (the gamescope backend relays `LIBEI_SOCKET` there once
/// the nested app launches), then connect. A bare name is resolved against `XDG_RUNTIME_DIR`,
/// mirroring libei's own `LIBEI_SOCKET` semantics. Line 2 of the relay file, when present,
/// carries the compositor's output size as `WxH` — returned as the absolute-coordinate scale
/// hint (gamescope's EIS region is degenerate, so the geometry can't come from the protocol).
async fn connect_socket_file(file: &std::path::Path) -> Result<(UnixStream, Option<(u32, u32)>)> {
/// mirroring libei's own `LIBEI_SOCKET` semantics.
async fn connect_socket_file(file: &std::path::Path) -> Result<UnixStream> {
// The relay file is rewritten each session with the CURRENT gamescope's `LIBEI_SOCKET`, and the
// socket may not be `listen()`ing the instant its name appears — or the file may briefly still
// hold a prior, now-dead session's name (the host-lifetime injector reconnecting between
@@ -330,12 +319,7 @@ async fn connect_socket_file(file: &std::path::Path) -> Result<(UnixStream, Opti
));
}
if let Ok(s) = std::fs::read_to_string(file) {
let mut file_lines = s.lines();
let name = file_lines.next().unwrap_or("").trim();
let hint = file_lines.next().and_then(|l| {
let (w, h) = l.trim().split_once('x')?;
Some((w.parse::<u32>().ok()?, h.parse::<u32>().ok()?))
});
let name = s.trim();
if !name.is_empty() {
let full = if name.starts_with('/') {
std::path::PathBuf::from(name)
@@ -350,7 +334,7 @@ async fn connect_socket_file(file: &std::path::Path) -> Result<(UnixStream, Opti
logged = name.to_string();
}
match UnixStream::connect(&full) {
Ok(stream) => return Ok((stream, hint)),
Ok(stream) => return Ok(stream),
// Refused = socket file exists but no listener yet (or a dead session);
// NotFound = path not created yet. Both heal once the live gamescope's EIS is
// up — retry. Anything else (e.g. permission) is a real failure.
@@ -374,25 +358,6 @@ async fn connect_socket_file(file: &std::path::Path) -> Result<(UnixStream, Opti
}
/// One EI device and its emulation state.
/// Pick the region to map absolute coordinates into: the one whose logical size matches the
/// streamed mode (the session's virtual output). The device advertises one region per logical
/// monitor, and blindly taking `first()` next to a physical monitor put the pointer — and every
/// click — on whichever output the compositor happened to announce first (on-glass: GNOME with a
/// dummy HDMI beside the virtual primary; the seat cursor never entered the streamed monitor, so
/// neither embedded nor metadata cursor capture could see it). Size is the only key available
/// today: regions carry no output name, and matching the screencast stream's `mapping_id` needs
/// the stream id plumbed across crates (follow-up). Two same-sized monitors stay ambiguous.
fn region_for_mode(
regions: &[reis::event::Region],
w: f32,
h: f32,
) -> Option<&reis::event::Region> {
regions
.iter()
.find(|r| r.width as f32 == w && r.height as f32 == h)
.or_else(|| regions.first())
}
struct DeviceSlot {
device: reis::event::Device,
/// The device is resumed (allowed to emit). Devices arrive paused and may pause again.
@@ -421,22 +386,6 @@ struct EiState {
held_keys: Vec<u32>,
held_buttons: Vec<u32>,
held_touches: Vec<u32>,
/// The touch id currently degraded to the absolute pointer ([`EiState::degrade_touch`]) —
/// the "primary finger" while the EIS has no touchscreen device. `None` between touches.
degraded_touch: Option<u32>,
/// The compositor's output size (relay-file "WxH" hint) — scale target for absolute
/// coordinates when the device's region is degenerate/absent (gamescope). Without it the
/// fallback is raw client pixels, correct only when the stream runs at the output's size.
output_hint: Option<(u32, u32)>,
}
/// Is this EIS region a plausible OUTPUT geometry — something to map normalized coordinates
/// into? gamescope advertises a degenerate `(0,0,INT32_MAX,INT32_MAX)` "everything" region on
/// its virtual input device, meaning "absolute coordinates are raw"; normalizing into it
/// explodes a center tap to x≈1e9, which the compositor clamps to the far corner. 16384
/// comfortably covers real multi-monitor layouts while rejecting the sentinel.
fn sane_region(r: &reis::event::Region) -> bool {
r.width > 0 && r.height > 0 && r.width <= 16_384 && r.height <= 16_384
}
/// Stable small index per [`InputKind`] for the `seen_kinds` bitmask.
@@ -457,7 +406,6 @@ fn kind_bit(kind: InputKind) -> u32 {
InputKind::GamepadState => 12,
InputKind::GamepadRemove => 13,
InputKind::GamepadArrival => 14,
InputKind::TextInput => 15,
};
1 << i
}
@@ -474,8 +422,6 @@ impl EiState {
held_keys: Vec::new(),
held_buttons: Vec::new(),
held_touches: Vec::new(),
degraded_touch: None,
output_hint: None,
}
}
@@ -484,10 +430,6 @@ impl EiState {
/// normal [`EiState::inject`] path so the compositor sees proper key-up / button-up /
/// touch-up frames before the devices disappear.
fn release_all(&mut self, ctx: &ei::Context) {
// A degraded touch is held as a synthesized left button (in `held_buttons`), so the
// loop below releases it — but the primary-finger latch must clear too, or the NEXT
// session's first TouchDown reads as a second finger and is ignored.
self.degraded_touch = None;
let (keys, buttons, touches) = (
std::mem::take(&mut self.held_keys),
std::mem::take(&mut self.held_buttons),
@@ -564,14 +506,6 @@ impl EiState {
keyboard = dev.has_capability(DeviceCapability::Keyboard),
button = dev.has_capability(DeviceCapability::Button),
scroll = dev.has_capability(DeviceCapability::Scroll),
// One region per logical monitor — which one absolute coords map into is
// resolved per event (`region_for_mode`); log them so a mis-mapped pointer
// (cursor/clicks on the wrong output) is diagnosable from the journal.
regions = ?dev
.regions()
.iter()
.map(|r| format!("{}x{}+{}+{}", r.width, r.height, r.x, r.y))
.collect::<Vec<_>>(),
"libei: device RESUMED (now emittable)"
);
}
@@ -603,85 +537,8 @@ impl EiState {
}
}
/// Degrade touch to a single-finger ABSOLUTE POINTER on compositors whose EIS never
/// creates a touchscreen device (gamescope's "Gamescope Virtual Input" advertises
/// pointer/pointer_abs/button but no touch — observed live; headless KWin the same).
/// Down = abs-move + left press, move = abs-move, up = left release — synthesized through
/// the normal [`EiState::inject`] Mouse* paths so region mapping, held-state tracking and
/// [`EiState::release_all`] all apply. Only the FIRST finger drives the pointer; later
/// fingers are ignored (a pointer has no second contact — a pinch degrades to a drag).
fn degrade_touch(&mut self, ev: &InputEvent, ctx: &ei::Context) {
const GS_BUTTON_LEFT: u32 = 1;
match ev.kind {
InputKind::TouchDown => {
if self.degraded_touch.is_some() {
return; // secondary finger — single-pointer degradation
}
self.degraded_touch = Some(ev.code);
static NOTED: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(false);
if !NOTED.swap(true, std::sync::atomic::Ordering::Relaxed) {
tracing::info!(
"compositor's EIS has no touchscreen device — degrading touch to a \
single-finger absolute pointer (tap = left click; multi-touch \
gestures unavailable)"
);
}
self.inject(
&InputEvent {
kind: InputKind::MouseMoveAbs,
..*ev
},
ctx,
);
self.inject(
&InputEvent {
kind: InputKind::MouseButtonDown,
code: GS_BUTTON_LEFT,
..*ev
},
ctx,
);
}
InputKind::TouchMove if self.degraded_touch == Some(ev.code) => {
self.inject(
&InputEvent {
kind: InputKind::MouseMoveAbs,
..*ev
},
ctx,
);
}
InputKind::TouchUp if self.degraded_touch == Some(ev.code) => {
self.degraded_touch = None;
self.inject(
&InputEvent {
kind: InputKind::MouseButtonUp,
code: GS_BUTTON_LEFT,
..*ev
},
ctx,
);
}
_ => {}
}
}
/// Translate and emit one client input event, committing it as a single `frame`.
fn inject(&mut self, ev: &InputEvent, ctx: &ei::Context) {
// No ei_touchscreen device but an absolute pointer exists → degrade rather than drop
// (the game-mode "touch simply does not work" trap). Checked per event, not latched:
// a touchscreen device appearing later (compositor restart, capability change) takes
// over seamlessly on the next touch.
if matches!(
ev.kind,
InputKind::TouchDown | InputKind::TouchMove | InputKind::TouchUp
) && self.device_for(DeviceCapability::Touch).is_none()
&& self.device_for(DeviceCapability::PointerAbsolute).is_some()
{
self.degrade_touch(ev, ctx);
return;
}
let cap = match ev.kind {
InputKind::MouseMove => DeviceCapability::Pointer,
InputKind::MouseMoveAbs => DeviceCapability::PointerAbsolute,
@@ -696,9 +553,6 @@ impl EiState {
| InputKind::GamepadAxis
| InputKind::GamepadRemove
| InputKind::GamepadArrival => return, // uinput path (later)
// libei presses keycodes against the server's negotiated keymap — no committed-text
// path (the HOST_CAP_TEXT_INPUT cap is not advertised on this backend).
InputKind::TextInput => return,
};
self.injected += 1;
let n = self.injected;
@@ -751,33 +605,16 @@ impl EiState {
InputKind::MouseMoveAbs => {
let w = ((ev.flags >> 16) & 0xffff) as f32;
let h = (ev.flags & 0xffff) as f32;
match slot.interface::<ei::PointerAbsolute>() {
Some(p) if w > 0.0 && h > 0.0 => {
// Map the normalized client position into the region matching the streamed
// output's mode (`region_for_mode` picks the right one on a multi-monitor
// EIS). gamescope's "Gamescope Virtual Input" advertises a degenerate
// (0,0,INT32_MAX,INT32_MAX) region meaning "coordinates are raw" —
// `sane_region` rejects it (normalizing into it explodes a center tap to
// x≈1e9, clamped to the far corner), so a non-matching / insane region
// falls to the output hint (correct across a resolution mismatch), then
// raw client pixels as the last resort.
match (
slot.interface::<ei::PointerAbsolute>(),
slot.regions().first(),
) {
(Some(p), Some(region)) if w > 0.0 && h > 0.0 => {
// Map the normalized client position into the device's first region.
let nx = (ev.x as f32 / w).clamp(0.0, 1.0);
let ny = (ev.y as f32 / h).clamp(0.0, 1.0);
let (x, y) = match region_for_mode(slot.regions(), w, h)
.filter(|r| sane_region(r))
{
Some(region) => (
region.x as f32 + nx * region.width as f32,
region.y as f32 + ny * region.height as f32,
),
// Degenerate/absent region: scale into the relay-file output hint
// (correct even when the client streams at a different resolution
// than the session runs); raw client pixels as the last resort.
None => match self.output_hint {
Some((ow, oh)) => (nx * ow as f32, ny * oh as f32),
None => (ev.x as f32, ev.y as f32),
},
};
let x = region.x as f32 + nx * region.width as f32;
let y = region.y as f32 + ny * region.height as f32;
p.motion_absolute(x, y);
}
_ => emitted = false,
@@ -843,23 +680,12 @@ impl EiState {
InputKind::TouchDown | InputKind::TouchMove => {
let w = ((ev.flags >> 16) & 0xffff) as f32;
let h = (ev.flags & 0xffff) as f32;
match slot.interface::<ei::Touchscreen>() {
Some(t) if w > 0.0 && h > 0.0 => {
match (slot.interface::<ei::Touchscreen>(), slot.regions().first()) {
(Some(t), Some(region)) if w > 0.0 && h > 0.0 => {
let nx = (ev.x as f32 / w).clamp(0.0, 1.0);
let ny = (ev.y as f32 / h).clamp(0.0, 1.0);
// Same region-selection + degenerate fallback ladder as MouseMoveAbs.
let (x, y) = match region_for_mode(slot.regions(), w, h)
.filter(|r| sane_region(r))
{
Some(region) => (
region.x as f32 + nx * region.width as f32,
region.y as f32 + ny * region.height as f32,
),
None => match self.output_hint {
Some((ow, oh)) => (nx * ow as f32, ny * oh as f32),
None => (ev.x as f32, ev.y as f32),
},
};
let x = region.x as f32 + nx * region.width as f32;
let y = region.y as f32 + ny * region.height as f32;
if ev.kind == InputKind::TouchDown {
t.down(ev.code, x, y);
} else {
@@ -877,8 +703,7 @@ impl EiState {
| InputKind::GamepadButton
| InputKind::GamepadAxis
| InputKind::GamepadRemove
| InputKind::GamepadArrival
| InputKind::TextInput => emitted = false,
| InputKind::GamepadArrival => emitted = false,
}
if emitted {
-396
View File
@@ -1,396 +0,0 @@
//! Virtual tablet ("Punktfunk Pen"): a uinput stylus device carrying the pen plane's full
//! fidelity — pressure, tilt, barrel roll, hover distance, eraser, barrel buttons
//! (design/pen-tablet-input.md §5).
//!
//! Deliberately a **uinput device, not a compositor-protocol citizen**: no virtual-tablet
//! protocol exists in EI, the RemoteDesktop portal, KWin `fake_input`, or wlroots — while
//! every compositor consumes evdev tablets via libinput and forwards them to apps over
//! `zwp_tablet_v2` with nothing to configure. udev's `input_id` builtin classifies the device
//! from its capabilities (`BTN_TOOL_PEN` + `ABS_X/Y` ⇒ `ID_INPUT_TABLET`), so Krita/GIMP/
//! Xournal++ see a real pen. Output mapping is the compositor's own tablet-mapping default
//! (single output ⇒ correct; multi-monitor pinning is the documented follow-up, which is why
//! the device carries a stable, distinctive identity to key rules on).
//!
//! The consumer feeds it [`PenTransition`]s straight from the core's
//! [`PenTracker`](punktfunk_core::quic::PenTracker); this file only translates transitions to
//! evdev events and groups them into SYN frames so proximity-enter carries its position in the
//! same frame (libinput would otherwise report an entry at a stale point).
//!
//! ioctl numbers/struct layouts mirror `gamepad.rs` (verified against the same kernel
//! generation); each backend file stays self-contained by convention.
use anyhow::{bail, Result};
use punktfunk_core::quic::{PenSample, PenTool, PenTransition, PEN_BARREL1, PEN_BARREL2};
use std::os::fd::{AsRawFd, OwnedFd};
// ioctls (x86_64).
const UI_DEV_CREATE: libc::c_ulong = 0x5501;
const UI_DEV_DESTROY: libc::c_ulong = 0x5502;
const UI_DEV_SETUP: libc::c_ulong = 0x405c_5503;
const UI_ABS_SETUP: libc::c_ulong = 0x401c_5504;
const UI_SET_EVBIT: libc::c_ulong = 0x4004_5564;
const UI_SET_KEYBIT: libc::c_ulong = 0x4004_5565;
const UI_SET_PROPBIT: libc::c_ulong = 0x4004_556e;
// input-event-codes.h subset.
const EV_SYN: u16 = 0x00;
const EV_KEY: u16 = 0x01;
const EV_ABS: u16 = 0x03;
const SYN_REPORT: u16 = 0;
const ABS_X: u16 = 0x00;
const ABS_Y: u16 = 0x01;
/// Barrel roll rides ABS_Z (the Wacom Art-Pen rotation convention); libinput normalizes the
/// declared min..max onto its 0..360° rotation axis.
const ABS_Z: u16 = 0x02;
const ABS_PRESSURE: u16 = 0x18;
const ABS_DISTANCE: u16 = 0x19;
const ABS_TILT_X: u16 = 0x1a;
const ABS_TILT_Y: u16 = 0x1b;
const BTN_TOOL_PEN: u16 = 0x140;
const BTN_TOOL_RUBBER: u16 = 0x141;
const BTN_TOUCH: u16 = 0x14a;
const BTN_STYLUS: u16 = 0x14b;
const BTN_STYLUS2: u16 = 0x14c;
/// The pen writes on the display it is mapped to (a "screen tablet"), not a desk pad —
/// libinput then maps the full ABS range onto the output rect, exactly the wire's normalized
/// coordinate contract.
const INPUT_PROP_DIRECT: libc::c_int = 0x01;
/// Full-scale wire pressure (u16) → the declared 0..4095 axis.
const PRESSURE_SHIFT: u32 = 4;
/// Wire hover distance (u16, 0xFFFF = unknown) → the declared 0..1023 axis.
const DISTANCE_SHIFT: u32 = 6;
const ABS_RANGE: f32 = 65535.0;
#[repr(C)]
struct InputId {
bustype: u16,
vendor: u16,
product: u16,
version: u16,
}
#[repr(C)]
struct UinputSetup {
id: InputId,
name: [u8; 80],
ff_effects_max: u32,
}
#[repr(C)]
#[derive(Default, Clone, Copy)]
struct AbsInfo {
value: i32,
minimum: i32,
maximum: i32,
fuzz: i32,
flat: i32,
resolution: i32,
}
#[repr(C)]
struct UinputAbsSetup {
code: u16,
_pad: u16,
absinfo: AbsInfo,
}
#[repr(C)]
#[derive(Clone, Copy)]
struct InputEventRaw {
time: libc::timeval,
type_: u16,
code: u16,
value: i32,
}
fn ioctl_int(fd: i32, req: libc::c_ulong, arg: libc::c_int, what: &str) -> Result<()> {
// SAFETY: every caller passes a UI_SET_*/UI_DEV_* request whose argument the kernel reads
// as a plain int; `fd` is a live uinput fd owned by the caller. No memory is handed over.
if unsafe { libc::ioctl(fd, req, arg) } < 0 {
bail!("{what}: {}", std::io::Error::last_os_error());
}
Ok(())
}
fn ioctl_ptr<T>(fd: i32, req: libc::c_ulong, arg: *mut T, what: &str) -> Result<()> {
// SAFETY: every caller passes a pointer to a live, initialized `#[repr(C)]` struct matching
// the request's expected layout (UI_DEV_SETUP/UI_ABS_SETUP); the kernel reads it during the
// call and retains nothing.
if unsafe { libc::ioctl(fd, req, arg) } < 0 {
bail!("{what}: {}", std::io::Error::last_os_error());
}
Ok(())
}
/// The active tool's evdev key, one in proximity at a time (Wacom semantics — the core's
/// [`PenTracker`](punktfunk_core::quic::PenTracker) already re-enters proximity on a tool
/// switch, so this only tracks which key to release on `ProximityOut`).
fn tool_key(tool: PenTool) -> u16 {
match tool {
PenTool::Eraser => BTN_TOOL_RUBBER,
// Unknown = a newer client's future tool — nearest ink-capable behavior is the pen.
PenTool::Pen | PenTool::Unknown => BTN_TOOL_PEN,
}
}
/// One per-session virtual tablet. Created lazily on the first pen batch (a session that never
/// draws never creates a device), destroyed with the session (Drop → `UI_DEV_DESTROY`).
pub struct VirtualPen {
fd: OwnedFd,
/// The tool key currently held in proximity (release target for `ProximityOut`).
tool: u16,
/// Whether the current SYN frame already carries a Motion — the frame-split trigger for
/// consecutive samples in one batch.
frame_has_motion: bool,
/// Whether the current SYN frame has any unflushed events.
frame_dirty: bool,
}
impl VirtualPen {
pub fn create() -> Result<VirtualPen> {
use std::os::fd::FromRawFd;
// SAFETY: `c"/dev/uinput"` is a 'static NUL-terminated C string literal; `open` reads it
// as a path, returns a fresh fd (or -1) and retains nothing.
let raw = unsafe {
libc::open(
c"/dev/uinput".as_ptr(),
libc::O_RDWR | libc::O_NONBLOCK | libc::O_CLOEXEC,
)
};
if raw < 0 {
bail!(
"open /dev/uinput: {} (install the udev rule granting the 'input' group access \
see scripts/60-punktfunk.rules and add the user to the 'input' group)",
std::io::Error::last_os_error()
);
}
// SAFETY: `raw >= 0` here, a freshly-opened fd owned nowhere else; `OwnedFd` becomes the
// unique owner and closes it exactly once on drop.
let fd = unsafe { OwnedFd::from_raw_fd(raw) };
ioctl_int(raw, UI_SET_EVBIT, EV_KEY as i32, "UI_SET_EVBIT(EV_KEY)")?;
ioctl_int(raw, UI_SET_EVBIT, EV_ABS as i32, "UI_SET_EVBIT(EV_ABS)")?;
for key in [
BTN_TOOL_PEN,
BTN_TOOL_RUBBER,
BTN_TOUCH,
BTN_STYLUS,
BTN_STYLUS2,
] {
ioctl_int(raw, UI_SET_KEYBIT, key as i32, "UI_SET_KEYBIT")?;
}
ioctl_int(
raw,
UI_SET_PROPBIT,
INPUT_PROP_DIRECT,
"UI_SET_PROPBIT(DIRECT)",
)?;
// Position spans the full u16 range; `resolution` (units/mm) only feeds libinput's mm
// math (nothing pen-relevant), but tablets without one trip its missing-resolution
// fixup — 100 declares a plausible ~655 mm drawing surface.
let pos = AbsInfo {
minimum: 0,
maximum: 65535,
resolution: 100,
..Default::default()
};
// Tilt in degrees from vertical, per evdev convention; resolution = units/radian (57
// ⇔ 1 unit = 1°, what the Wacom driver declares).
let tilt = AbsInfo {
minimum: -90,
maximum: 90,
resolution: 57,
..Default::default()
};
for (code, info) in [
(ABS_X, pos),
(ABS_Y, pos),
(
ABS_PRESSURE,
AbsInfo {
minimum: 0,
maximum: 4095,
..Default::default()
},
),
(
ABS_DISTANCE,
AbsInfo {
minimum: 0,
maximum: 1023,
..Default::default()
},
),
(ABS_TILT_X, tilt),
(ABS_TILT_Y, tilt),
(
// Barrel roll: libinput maps the declared range linearly onto 0..360°.
ABS_Z,
AbsInfo {
minimum: 0,
maximum: 359,
..Default::default()
},
),
] {
let mut a = UinputAbsSetup {
code,
_pad: 0,
absinfo: info,
};
ioctl_ptr(raw, UI_ABS_SETUP, &mut a, "UI_ABS_SETUP")?;
}
// A stable, distinctive identity (pid.codes open-source VID) so compositor
// tablet-mapping rules — GNOME's per-`vendor:product` gsettings path, sway
// `map_to_output` — can target exactly this device.
let mut setup = UinputSetup {
id: InputId {
bustype: 0x0006, // BUS_VIRTUAL
vendor: 0x1209,
product: 0x5046, // "PF"
version: 1,
},
name: [0; 80],
ff_effects_max: 0,
};
let name = b"Punktfunk Pen";
setup.name[..name.len()].copy_from_slice(name);
ioctl_ptr(raw, UI_DEV_SETUP, &mut setup, "UI_DEV_SETUP")?;
ioctl_int(raw, UI_DEV_CREATE, 0, "UI_DEV_CREATE")?;
tracing::info!("virtual tablet created (Punktfunk Pen, uinput)");
Ok(VirtualPen {
fd,
tool: BTN_TOOL_PEN,
frame_has_motion: false,
frame_dirty: false,
})
}
fn emit(&self, type_: u16, code: u16, value: i32) {
let ev = InputEventRaw {
time: libc::timeval {
tv_sec: 0,
tv_usec: 0,
},
type_,
code,
value,
};
// SAFETY: `ev` is a live local `#[repr(C)]` all-integer struct (no padding: timeval=16 +
// u16 + u16 + i32 = 24), so every byte is initialized; the slice spans exactly `ev`'s
// bytes and is used immediately below with no concurrent mutation.
let bytes = unsafe {
std::slice::from_raw_parts(
&ev as *const _ as *const u8,
std::mem::size_of::<InputEventRaw>(),
)
};
// Best-effort like the gamepad path: a full kernel queue drops the event; pen samples
// are state-full, so the next frame re-syncs axes (and the tracker re-syncs state).
// SAFETY: `self.fd` stays open for the synchronous call; `write` only reads
// `bytes.len()` bytes from the still-live local and retains nothing.
let _ = unsafe {
libc::write(
self.fd.as_raw_fd(),
bytes.as_ptr() as *const libc::c_void,
bytes.len(),
)
};
}
fn flush(&mut self) {
if self.frame_dirty {
self.emit(EV_SYN, SYN_REPORT, 0);
self.frame_dirty = false;
self.frame_has_motion = false;
}
}
fn motion(&mut self, s: &PenSample) {
self.emit(EV_ABS, ABS_X, (s.x * ABS_RANGE) as i32);
self.emit(EV_ABS, ABS_Y, (s.y * ABS_RANGE) as i32);
self.emit(EV_ABS, ABS_PRESSURE, (s.pressure >> PRESSURE_SHIFT) as i32);
if s.distance != punktfunk_core::quic::PEN_DISTANCE_UNKNOWN {
self.emit(EV_ABS, ABS_DISTANCE, (s.distance >> DISTANCE_SHIFT) as i32);
}
// Polar → tiltX/tiltY needs both angles; azimuth clockwise from north, so east (90°)
// tilts +X and south (180°, toward the user) tilts +Y — the evdev/W3C signs.
if s.tilt_deg != punktfunk_core::quic::PEN_TILT_UNKNOWN
&& s.azimuth_deg != punktfunk_core::quic::PEN_ANGLE_UNKNOWN
{
let az = (s.azimuth_deg as f32).to_radians();
let tilt = s.tilt_deg as f32;
self.emit(EV_ABS, ABS_TILT_X, (tilt * az.sin()).round() as i32);
self.emit(EV_ABS, ABS_TILT_Y, (-tilt * az.cos()).round() as i32);
}
if s.roll_deg != punktfunk_core::quic::PEN_ANGLE_UNKNOWN {
self.emit(EV_ABS, ABS_Z, (s.roll_deg % 360) as i32);
}
self.frame_dirty = true;
self.frame_has_motion = true;
}
/// Apply one decoded batch's transitions (the core tracker's output, in its documented
/// order), grouping them into SYN frames: a frame closes before a `ProximityIn` (an entry
/// is a new instant — and must carry its own position, not inherit the stale frame) and
/// before a second `Motion` (consecutive samples are consecutive instants), plus a final
/// close. So `[ProxIn, Motion, TipDown]` lands as ONE frame — libinput reports the entry
/// already at the right point with contact — while a drag batch's `[Motion, Motion]`
/// stays two.
pub fn apply_batch(&mut self, transitions: &[PenTransition]) {
for t in transitions {
match t {
PenTransition::ProximityIn { tool } => {
self.flush();
self.tool = tool_key(*tool);
self.emit(EV_KEY, self.tool, 1);
self.frame_dirty = true;
}
PenTransition::Motion { sample } => {
if self.frame_has_motion {
self.flush();
}
self.motion(sample);
}
PenTransition::TipDown => {
self.emit(EV_KEY, BTN_TOUCH, 1);
self.frame_dirty = true;
}
PenTransition::ButtonsChanged { pressed, released } => {
for (bit, key) in [(PEN_BARREL1, BTN_STYLUS), (PEN_BARREL2, BTN_STYLUS2)] {
if pressed & bit != 0 {
self.emit(EV_KEY, key, 1);
self.frame_dirty = true;
}
if released & bit != 0 {
self.emit(EV_KEY, key, 0);
self.frame_dirty = true;
}
}
}
PenTransition::TipUp => {
self.emit(EV_KEY, BTN_TOUCH, 0);
self.emit(EV_ABS, ABS_PRESSURE, 0);
self.frame_dirty = true;
}
PenTransition::ProximityOut => {
self.emit(EV_KEY, self.tool, 0);
self.frame_dirty = true;
}
}
}
self.flush();
}
}
impl Drop for VirtualPen {
fn drop(&mut self) {
// SAFETY: `self.fd` is still open (OwnedFd closes only after this body returns);
// UI_DEV_DESTROY takes no pointer argument. Errors are moot on teardown.
let _ = unsafe { libc::ioctl(self.fd.as_raw_fd(), UI_DEV_DESTROY, 0) };
}
}
-96
View File
@@ -98,26 +98,9 @@ pub struct WlrootsInjector {
keyboard: ZwpVirtualKeyboardV1,
xkb_state: xkb::State,
_keymap_file: std::fs::File, // keep the memfd alive for the compositor's mmap
/// Dedicated committed-text device ([`InputKind::TextInput`]), created on first use.
text: Option<TextKeyboard>,
start: Instant,
}
/// Cap on distinct characters the dynamic text keymap holds before it restarts from scratch
/// (keycodes grow upward from 9; xkb tops out at 255, so stay well under).
const TEXT_KEYMAP_MAX: usize = 200;
/// The dedicated **text** virtual keyboard: types committed IME text (`InputKind::TextInput`,
/// one Unicode scalar per event) by growing a keymap of Unicode keysyms on demand and pressing
/// the character's keycode — the `wtype` model. A separate `zwp_virtual_keyboard` so keymap
/// re-uploads never disturb the main device's layout/modifier state that VK key events ride on.
struct TextKeyboard {
keyboard: ZwpVirtualKeyboardV1,
/// Characters in keycode order: `chars[i]` types on wire keycode `i + 1` (xkb `i + 9`).
chars: Vec<char>,
_keymap_file: Option<std::fs::File>, // keep the memfd alive for the compositor's mmap
}
impl WlrootsInjector {
pub fn open() -> Result<Self> {
let conn = Connection::connect_to_env()
@@ -188,7 +171,6 @@ impl WlrootsInjector {
keyboard,
xkb_state,
_keymap_file: file,
text: None,
start: Instant::now(),
})
}
@@ -197,54 +179,6 @@ impl WlrootsInjector {
self.start.elapsed().as_millis() as u32
}
/// Type one committed-text Unicode scalar on the dedicated text device (created lazily),
/// growing its keymap when the character is new. Control characters are dropped — Enter,
/// Backspace and Tab ride the VK key-event path.
fn type_text(&mut self, cp: u32) -> Result<()> {
let Some(ch) = char::from_u32(cp) else {
return Ok(()); // lone surrogate / out of range
};
if ch.is_control() {
return Ok(());
}
if self.text.is_none() {
let (Some(mgr), Some(seat)) =
(self.globals.keyboard_mgr.clone(), self.globals.seat.clone())
else {
return Ok(());
};
let kb = mgr.create_virtual_keyboard(&seat, &self.queue.handle(), ());
self.text = Some(TextKeyboard {
keyboard: kb,
chars: Vec::new(),
_keymap_file: None,
});
}
let t = self.now_ms();
let text = self.text.as_mut().expect("created above");
let code = match text.chars.iter().position(|&c| c == ch) {
Some(i) => (i + 1) as u32,
None => {
if text.chars.len() >= TEXT_KEYMAP_MAX {
text.chars.clear(); // restart the map; old codes are re-assigned lazily
}
text.chars.push(ch);
let keymap_str = text_keymap(&text.chars);
let file = memfd_with(&keymap_str)?;
text.keyboard.keymap(
1, /* XKB_V1 */
file.as_fd(),
keymap_str.len() as u32 + 1,
);
text._keymap_file = Some(file);
text.chars.len() as u32
}
};
text.keyboard.key(t, code, 1);
text.keyboard.key(t, code, 0);
Ok(())
}
/// Update xkb state for a key and tell the compositor the resulting modifier mask.
fn send_modifiers(&mut self, evdev: u16, down: bool) {
let kc = xkb::Keycode::new(evdev as u32 + 8); // evdev -> xkb keycode
@@ -320,9 +254,6 @@ impl InputInjector for WlrootsInjector {
tracing::debug!(vk = event.code, "unmapped VK keycode — dropped");
}
}
InputKind::TextInput => {
self.type_text(event.code)?;
}
InputKind::GamepadState
| InputKind::GamepadButton
| InputKind::GamepadAxis
@@ -340,33 +271,6 @@ impl InputInjector for WlrootsInjector {
}
}
/// Build a minimal xkb keymap whose keycode `i + 9` (wire code `i + 1`) types `chars[i]`, using
/// Unicode keysym names (`U<hex>` — xkbcommon resolves them for any scalar, emoji included).
/// Types/compat `include "complete"` mirrors `wtype`'s generated keymap — proven on wlroots
/// compositors, and the system XKB data is present (the main keymap compiled from it in `open`).
fn text_keymap(chars: &[char]) -> String {
use std::fmt::Write as _;
let mut keycodes = String::new();
let mut symbols = String::new();
for (i, ch) in chars.iter().enumerate() {
let _ = writeln!(keycodes, " <T{i}> = {};", i + 9);
let _ = writeln!(symbols, " key <T{i}> {{ [ U{:04X} ] }};", *ch as u32);
}
format!(
"xkb_keymap {{\n\
xkb_keycodes \"punktfunk-text\" {{\n\
minimum = 8;\n\
maximum = {};\n\
{keycodes}\
}};\n\
xkb_types \"punktfunk-text\" {{ include \"complete\" }};\n\
xkb_compatibility \"punktfunk-text\" {{ include \"complete\" }};\n\
xkb_symbols \"punktfunk-text\" {{\n{symbols} }};\n\
}};\n",
chars.len() + 9,
)
}
/// Create an anonymous in-memory file holding `s` + a trailing NUL (for the keymap fd).
fn memfd_with(s: &str) -> Result<std::fs::File> {
let name = b"punktfunk-keymap\0";
@@ -1,556 +0,0 @@
//! Windows synthetic-pointer injection (design/pen-tablet-input.md §6): a per-session `PT_PEN`
//! device carrying the pen plane's full fidelity — pressure (rescaled to Windows' 0..1024),
//! polar tilt → tiltX/tiltY, barrel roll on `rotation` (0..359 — Windows Ink renders Pencil
//! Pro roll natively), barrel button, eraser (`PEN_FLAG_INVERTED`/`ERASER`), hover — plus a
//! `PT_TOUCH` device that closes the long-standing SendInput touch no-op for wire touches.
//!
//! Both follow Apollo's proven recipe (`design/apollo-comparison.md`): synthetic pointer state
//! goes STALE if not re-injected (~100 ms), so each device runs a small refresh thread that
//! re-asserts the last frame every [`REFRESH_MS`] while the pen is in range / contacts are
//! held — a stationary stylus must not hover-out (and a held finger must not auto-lift) just
//! because no new samples arrived. `CreateSyntheticPointerDevice` needs Win10 1809+;
//! [`crate::pen_supported`] probes it, so older hosts simply never advertise pen.
//!
//! Frame grouping mirrors the Linux uinput backend: a proximity-enter is injected together
//! with its position (never at a stale point), tip edges get their own DOWN/UP frames, and a
//! range-leave is a final frame without `INRANGE`.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it.
#![deny(clippy::undocumented_unsafe_blocks)]
use anyhow::{Context, Result};
use punktfunk_core::input::{InputEvent, InputKind};
use punktfunk_core::quic::{
PenSample, PenTool, PenTransition, PEN_ANGLE_UNKNOWN, PEN_BARREL1, PEN_TILT_UNKNOWN,
};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use windows::Win32::Foundation::POINT;
use windows::Win32::UI::Controls::{
CreateSyntheticPointerDevice, DestroySyntheticPointerDevice, HSYNTHETICPOINTERDEVICE,
POINTER_FEEDBACK_DEFAULT, POINTER_TYPE_INFO, POINTER_TYPE_INFO_0,
};
use windows::Win32::UI::Input::Pointer::{
InjectSyntheticPointerInput, POINTER_FLAGS, POINTER_FLAG_DOWN, POINTER_FLAG_FIRSTBUTTON,
POINTER_FLAG_INCONTACT, POINTER_FLAG_INRANGE, POINTER_FLAG_NEW, POINTER_FLAG_UP,
POINTER_FLAG_UPDATE, POINTER_INFO, POINTER_PEN_INFO, POINTER_TOUCH_INFO,
};
use windows::Win32::UI::WindowsAndMessaging::{
GetSystemMetrics, PEN_FLAG_BARREL, PEN_FLAG_ERASER, PEN_FLAG_INVERTED, PEN_FLAG_NONE,
PEN_MASK_PRESSURE, PEN_MASK_ROTATION, PEN_MASK_TILT_X, PEN_MASK_TILT_Y, PT_PEN, PT_TOUCH,
SM_CXVIRTUALSCREEN, SM_CYVIRTUALSCREEN, SM_XVIRTUALSCREEN, SM_YVIRTUALSCREEN, TOUCH_FLAG_NONE,
TOUCH_MASK_NONE,
};
/// Re-inject cadence while state is held (in-range pen / touching contacts). Apollo uses
/// 50 ms against the ~100 ms staleness window; 40 ms keeps two refreshes inside it.
const REFRESH_MS: u64 = 40;
/// Windows pen pressure is 0..1024 (vs the wire's full-scale u16).
const WIN_PEN_PRESSURE_MAX: u32 = 1024;
/// Map a normalized [0,1] coordinate pair onto virtual-desktop pixels — the same surface the
/// SendInput absolute mouse targets, so pen, touch, and pointer all land identically.
fn to_screen(x: f32, y: f32) -> POINT {
// SAFETY: `GetSystemMetrics` takes a constant index and reads global metrics; no memory in.
let (vx, vy, vw, vh) = unsafe {
(
GetSystemMetrics(SM_XVIRTUALSCREEN),
GetSystemMetrics(SM_YVIRTUALSCREEN),
GetSystemMetrics(SM_CXVIRTUALSCREEN).max(1),
GetSystemMetrics(SM_CYVIRTUALSCREEN).max(1),
)
};
POINT {
x: vx + (x.clamp(0.0, 1.0) * (vw - 1) as f32) as i32,
y: vy + (y.clamp(0.0, 1.0) * (vh - 1) as f32) as i32,
}
}
/// An owned `HSYNTHETICPOINTERDEVICE` (destroyed exactly once on drop).
struct Device(HSYNTHETICPOINTERDEVICE);
// SAFETY: the handle is a plain kernel object identifier with no thread affinity —
// `InjectSyntheticPointerInput`/`DestroySyntheticPointerDevice` are documented callable from
// any thread; ownership transfer/sharing does not alias memory.
unsafe impl Send for Device {}
impl Drop for Device {
fn drop(&mut self) {
// SAFETY: `self.0` is the device this wrapper uniquely owns; destroyed once here.
unsafe { DestroySyntheticPointerDevice(self.0) };
}
}
/// Probe: can this Windows build create a synthetic pen device (Win10 1809+)?
pub fn synthetic_pen_available() -> bool {
// SAFETY: FFI create with by-value args; on success the returned handle is destroyed
// immediately by the `Device` wrapper, exactly once.
match unsafe { CreateSyntheticPointerDevice(PT_PEN, 1, POINTER_FEEDBACK_DEFAULT) } {
Ok(h) => {
drop(Device(h));
true
}
Err(_) => false,
}
}
/// The tracked pen state a refresh tick re-asserts.
#[derive(Default)]
struct PenState {
in_range: bool,
touching: bool,
barrel: bool,
eraser: bool,
x: f32,
y: f32,
pressure: u16,
tilt_deg: u8,
azimuth_deg: u16,
roll_deg: u16,
}
struct PenShared {
dev: Device,
state: PenState,
/// First injection failure logs at WARN (see `TouchShared::fail_warned`).
fail_warned: bool,
}
/// One per-session virtual pen (the Windows sibling of the Linux uinput tablet — same
/// [`PenTransition`] consumer API). Wire barrel button 2 has no Windows pen equivalent
/// (one barrel + eraser is the platform model) and is ignored here.
pub struct VirtualPen {
shared: Arc<Mutex<PenShared>>,
stop: Arc<AtomicBool>,
refresher: Option<std::thread::JoinHandle<()>>,
// Batch-local frame grouping (single-threaded within apply_batch).
edge_down: bool,
edge_up: bool,
is_new: bool,
frame_dirty: bool,
frame_has_motion: bool,
}
impl VirtualPen {
pub fn create() -> Result<VirtualPen> {
// SAFETY: FFI create with by-value args; the handle's sole owner becomes `Device`.
let dev = unsafe { CreateSyntheticPointerDevice(PT_PEN, 1, POINTER_FEEDBACK_DEFAULT) }
.context("CreateSyntheticPointerDevice(PT_PEN) — needs Windows 10 1809+")?;
let shared = Arc::new(Mutex::new(PenShared {
dev: Device(dev),
state: PenState::default(),
fail_warned: false,
}));
let stop = Arc::new(AtomicBool::new(false));
// The staleness guard: re-assert the last frame while in range so a stationary pen
// (native plane: between 100 ms heartbeats; GameStream: indefinitely) never hovers out.
let refresher = {
let shared = Arc::clone(&shared);
let stop = Arc::clone(&stop);
std::thread::Builder::new()
.name("pf-pen-refresh".into())
.spawn(move || {
while !stop.load(Ordering::Relaxed) {
std::thread::sleep(std::time::Duration::from_millis(REFRESH_MS));
let s = &mut *shared.lock().unwrap();
if s.state.in_range {
inject_pen(s, POINTER_FLAG_UPDATE, false);
}
}
})
.context("spawn pen refresh thread")?
};
tracing::info!("virtual pen created (Windows synthetic pointer, PT_PEN)");
Ok(VirtualPen {
shared,
stop,
refresher: Some(refresher),
edge_down: false,
edge_up: false,
is_new: false,
frame_dirty: false,
frame_has_motion: false,
})
}
/// Apply one batch of tracker transitions — same grouping contract as the Linux backend:
/// `[ProxIn, Motion, TipDown]` is ONE frame (entry lands at its position, in contact),
/// consecutive `Motion`s split, tip edges own their frames, range-leave is a final
/// no-`INRANGE` frame.
pub fn apply_batch(&mut self, transitions: &[PenTransition]) {
for t in transitions {
match t {
PenTransition::ProximityIn { tool } => {
self.flush();
let mut s = self.shared.lock().unwrap();
s.state.in_range = true;
s.state.eraser = *tool == PenTool::Eraser;
drop(s);
self.is_new = true;
self.frame_dirty = true;
}
PenTransition::Motion { sample } => {
if self.frame_has_motion {
self.flush();
}
self.set_axes(sample);
self.frame_dirty = true;
self.frame_has_motion = true;
}
PenTransition::TipDown => {
self.shared.lock().unwrap().state.touching = true;
self.edge_down = true;
self.frame_dirty = true;
}
PenTransition::ButtonsChanged { pressed, released } => {
let mut s = self.shared.lock().unwrap();
if pressed & PEN_BARREL1 != 0 {
s.state.barrel = true;
}
if released & PEN_BARREL1 != 0 {
s.state.barrel = false;
}
drop(s);
self.frame_dirty = true;
}
PenTransition::TipUp => {
self.shared.lock().unwrap().state.touching = false;
self.edge_up = true;
self.frame_dirty = true;
self.flush(); // UP owns its frame (still INRANGE)
}
PenTransition::ProximityOut => {
self.shared.lock().unwrap().state.in_range = false;
self.frame_dirty = true;
self.flush(); // final frame without INRANGE
}
}
}
self.flush();
}
fn set_axes(&mut self, s: &PenSample) {
let mut sh = self.shared.lock().unwrap();
sh.state.x = s.x;
sh.state.y = s.y;
sh.state.pressure = s.pressure;
sh.state.tilt_deg = s.tilt_deg;
sh.state.azimuth_deg = s.azimuth_deg;
sh.state.roll_deg = s.roll_deg;
}
fn flush(&mut self) {
if !self.frame_dirty {
return;
}
let edge = if self.edge_down {
POINTER_FLAG_DOWN
} else if self.edge_up {
POINTER_FLAG_UP
} else {
POINTER_FLAG_UPDATE
};
inject_pen(&mut self.shared.lock().unwrap(), edge, self.is_new);
self.edge_down = false;
self.edge_up = false;
self.is_new = false;
self.frame_dirty = false;
self.frame_has_motion = false;
}
}
impl Drop for VirtualPen {
fn drop(&mut self) {
self.stop.store(true, Ordering::Relaxed);
if let Some(h) = self.refresher.take() {
let _ = h.join();
}
// The device itself dies with `shared` (Device::drop) — Windows releases any held
// in-range/contact state when the synthetic device is destroyed.
}
}
/// Build + inject one pen frame from tracked state. `edge` is DOWN/UP/UPDATE;
/// `INRANGE`/`INCONTACT` derive from the state itself.
fn inject_pen(sh: &mut PenShared, edge: POINTER_FLAGS, is_new: bool) {
let st = &sh.state;
let mut flags = edge;
if st.in_range {
flags |= POINTER_FLAG_INRANGE;
}
if st.touching {
flags |= POINTER_FLAG_INCONTACT | POINTER_FLAG_FIRSTBUTTON;
}
if is_new {
flags |= POINTER_FLAG_NEW;
}
let mut pen_flags = PEN_FLAG_NONE;
if st.barrel {
pen_flags |= PEN_FLAG_BARREL;
}
if st.eraser {
pen_flags |= PEN_FLAG_INVERTED;
if st.touching {
pen_flags |= PEN_FLAG_ERASER;
}
}
let mut pen_mask = PEN_MASK_PRESSURE;
// Contact needs a nonzero pressure to ink; hover reports 0 (Windows convention).
let pressure = if st.touching {
((st.pressure as u32 * WIN_PEN_PRESSURE_MAX) / u16::MAX as u32).max(1)
} else {
0
};
let (mut tilt_x, mut tilt_y) = (0i32, 0i32);
if st.tilt_deg != PEN_TILT_UNKNOWN && st.azimuth_deg != PEN_ANGLE_UNKNOWN {
let az = (st.azimuth_deg as f32).to_radians();
let tilt = st.tilt_deg as f32;
tilt_x = (tilt * az.sin()).round() as i32;
tilt_y = (-tilt * az.cos()).round() as i32;
pen_mask |= PEN_MASK_TILT_X | PEN_MASK_TILT_Y;
}
let mut rotation = 0u32;
if st.roll_deg != PEN_ANGLE_UNKNOWN {
rotation = (st.roll_deg % 360) as u32;
pen_mask |= PEN_MASK_ROTATION;
}
let pt = to_screen(st.x, st.y);
let info = POINTER_TYPE_INFO {
r#type: PT_PEN,
Anonymous: POINTER_TYPE_INFO_0 {
penInfo: POINTER_PEN_INFO {
pointerInfo: POINTER_INFO {
pointerType: PT_PEN,
pointerId: 0,
pointerFlags: flags,
ptPixelLocation: pt,
ptPixelLocationRaw: pt,
..Default::default()
},
penFlags: pen_flags,
penMask: pen_mask,
pressure,
rotation,
tiltX: tilt_x,
tiltY: tilt_y,
},
},
};
// SAFETY: `sh.dev.0` is the live device this wrapper owns; the one-element array is a live
// stack value the call only reads. Best-effort like every injector write — a transient
// failure (desktop switch) is healed by the next refresh tick re-asserting state.
if let Err(e) = unsafe { InjectSyntheticPointerInput(sh.dev.0, &[info]) } {
if !sh.fail_warned {
sh.fail_warned = true;
tracing::warn!(error = %e, "pen inject failed");
} else {
tracing::trace!(error = %e, "pen inject failed (transient)");
}
} else {
sh.fail_warned = false;
}
}
/// One live wire-touch contact. `slot` is the SMALL, DENSE pointer id handed to Windows —
/// synthetic-pointer injection rejects arbitrary large ids, and clients (Moonlight's
/// `pointerId` especially) send exactly those, so wire ids compact into the lowest free slot
/// for the contact's lifetime (Apollo's slot-contiguity rule; the on-glass symptom of passing
/// wire ids through was pen working while every touch silently failed to inject).
#[derive(Clone, Copy)]
struct Contact {
id: u32,
slot: u32,
x: f32,
y: f32,
}
/// Lowest slot not held by a live contact.
fn free_slot(contacts: &[Contact]) -> u32 {
let mut slot = 0u32;
while contacts.iter().any(|c| c.slot == slot) {
slot += 1;
}
slot
}
/// Windows can inject at most this many simultaneous synthetic touch contacts.
const MAX_CONTACTS: usize = 10;
struct TouchShared {
dev: Device,
contacts: Vec<Contact>,
/// First injection failure logs at WARN (an on-glass "touch does nothing" must be
/// visible in the host log); repeats stay at trace.
fail_warned: bool,
}
/// The `PT_TOUCH` device servicing wire `TouchDown/Move/Up` events — closes the SendInput
/// touch no-op. Contacts are keyed by the wire's finger id; every frame re-injects the FULL
/// active set (the synthetic-pointer contract), and a refresh thread re-asserts held contacts
/// against the ~100 ms staleness auto-lift.
pub struct SyntheticTouch {
shared: Arc<Mutex<TouchShared>>,
stop: Arc<AtomicBool>,
refresher: Option<std::thread::JoinHandle<()>>,
}
impl SyntheticTouch {
pub fn create() -> Result<SyntheticTouch> {
// SAFETY: FFI create with by-value args; the handle's sole owner becomes `Device`.
let dev = unsafe {
CreateSyntheticPointerDevice(PT_TOUCH, MAX_CONTACTS as u32, POINTER_FEEDBACK_DEFAULT)
}
.context("CreateSyntheticPointerDevice(PT_TOUCH) — needs Windows 10 1809+")?;
let shared = Arc::new(Mutex::new(TouchShared {
dev: Device(dev),
contacts: Vec::new(),
fail_warned: false,
}));
let stop = Arc::new(AtomicBool::new(false));
let refresher = {
let shared = Arc::clone(&shared);
let stop = Arc::clone(&stop);
std::thread::Builder::new()
.name("pf-touch-refresh".into())
.spawn(move || {
while !stop.load(Ordering::Relaxed) {
std::thread::sleep(std::time::Duration::from_millis(REFRESH_MS));
let s = &mut *shared.lock().unwrap();
if !s.contacts.is_empty() {
inject_touch_frame(s, None);
}
}
})
.context("spawn touch refresh thread")?
};
tracing::info!("virtual touchscreen created (Windows synthetic pointer, PT_TOUCH)");
Ok(SyntheticTouch {
shared,
stop,
refresher: Some(refresher),
})
}
/// Apply one wire touch event (`code` = finger id, pixel x/y against the
/// `flags = (w << 16) | h` reference extent, exactly like `MouseMoveAbs`).
pub fn apply(&mut self, ev: &InputEvent) {
let (w, h) = ((ev.flags >> 16) as f32, (ev.flags & 0xFFFF) as f32);
if (w < 1.0 || h < 1.0) && ev.kind != InputKind::TouchUp {
return; // the documented zero-extent drop, as for MouseMoveAbs
}
let (x, y) = (ev.x as f32 / w.max(1.0), ev.y as f32 / h.max(1.0));
let s = &mut *self.shared.lock().unwrap();
match ev.kind {
InputKind::TouchDown => {
match s.contacts.iter().position(|c| c.id == ev.code) {
Some(i) => (s.contacts[i].x, s.contacts[i].y) = (x, y),
None if s.contacts.len() < MAX_CONTACTS => {
let slot = free_slot(&s.contacts);
s.contacts.push(Contact {
id: ev.code,
slot,
x,
y,
})
}
None => return, // beyond the platform max — drop, never evict a live finger
}
inject_touch_frame(s, Some((ev.code, POINTER_FLAG_DOWN)));
}
InputKind::TouchMove => {
match s.contacts.iter().position(|c| c.id == ev.code) {
Some(i) => {
(s.contacts[i].x, s.contacts[i].y) = (x, y);
inject_touch_frame(s, None);
}
// A move for an unknown id (its DOWN was dropped/lost): synthesize the
// contact so the stroke self-heals, like the pen plane does.
None if s.contacts.len() < MAX_CONTACTS => {
let slot = free_slot(&s.contacts);
s.contacts.push(Contact {
id: ev.code,
slot,
x,
y,
});
inject_touch_frame(s, Some((ev.code, POINTER_FLAG_DOWN)));
}
None => {}
}
}
InputKind::TouchUp => {
let Some(idx) = s.contacts.iter().position(|c| c.id == ev.code) else {
return;
};
// The UP frame still carries the lifting contact (with UP flags), then it
// leaves the active set.
inject_touch_frame(s, Some((ev.code, POINTER_FLAG_UP)));
s.contacts.remove(idx);
}
_ => {}
}
}
}
impl Drop for SyntheticTouch {
fn drop(&mut self) {
self.stop.store(true, Ordering::Relaxed);
if let Some(h) = self.refresher.take() {
let _ = h.join();
}
}
}
/// Inject the full active-contact frame; `edge` marks one contact's DOWN/UP transition (by
/// WIRE id — everyone else is a held UPDATE). Windows sees the compacted `slot` ids only.
fn inject_touch_frame(sh: &mut TouchShared, edge: Option<(u32, POINTER_FLAGS)>) {
let contacts = &sh.contacts;
if contacts.is_empty() {
return;
}
let mut frame: Vec<POINTER_TYPE_INFO> = Vec::with_capacity(contacts.len());
for c in contacts {
let mut flags = POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT | POINTER_FLAG_UPDATE;
if let Some((id, e)) = edge {
if id == c.id {
if e == POINTER_FLAG_DOWN {
flags = POINTER_FLAG_INRANGE | POINTER_FLAG_INCONTACT | POINTER_FLAG_DOWN;
} else if e == POINTER_FLAG_UP {
flags = POINTER_FLAG_UP; // contact + range end together
}
}
}
let pt = to_screen(c.x, c.y);
frame.push(POINTER_TYPE_INFO {
r#type: PT_TOUCH,
Anonymous: POINTER_TYPE_INFO_0 {
touchInfo: POINTER_TOUCH_INFO {
pointerInfo: POINTER_INFO {
pointerType: PT_TOUCH,
pointerId: c.slot,
pointerFlags: flags,
ptPixelLocation: pt,
ptPixelLocationRaw: pt,
..Default::default()
},
touchFlags: TOUCH_FLAG_NONE,
touchMask: TOUCH_MASK_NONE,
..Default::default()
},
},
});
}
// SAFETY: `sh.dev.0` is the live owned device; `frame` is a live Vec the call only reads.
// Best-effort — a transient failure heals on the next event/refresh re-assertion.
if let Err(e) = unsafe { InjectSyntheticPointerInput(sh.dev.0, &frame) } {
if !sh.fail_warned {
sh.fail_warned = true;
tracing::warn!(error = %e, contacts = frame.len(), "touch inject failed");
} else {
tracing::trace!(error = %e, "touch inject failed (transient)");
}
} else {
sh.fail_warned = false;
}
}
@@ -27,10 +27,10 @@ use windows::Win32::System::StationsAndDesktops::{
use windows::Win32::UI::Input::KeyboardAndMouse::{
GetKeyboardLayout, MapVirtualKeyExW, SendInput, HKL, INPUT, INPUT_0, INPUT_KEYBOARD,
INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY, KEYEVENTF_KEYUP, KEYEVENTF_SCANCODE,
KEYEVENTF_UNICODE, MAPVK_VK_TO_VSC_EX, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_HWHEEL,
MOUSEEVENTF_LEFTDOWN, MOUSEEVENTF_LEFTUP, MOUSEEVENTF_MIDDLEDOWN, MOUSEEVENTF_MIDDLEUP,
MOUSEEVENTF_MOVE, MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_VIRTUALDESK,
MOUSEEVENTF_WHEEL, MOUSEEVENTF_XDOWN, MOUSEEVENTF_XUP, MOUSEINPUT, VIRTUAL_KEY,
MAPVK_VK_TO_VSC_EX, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_HWHEEL, MOUSEEVENTF_LEFTDOWN,
MOUSEEVENTF_LEFTUP, MOUSEEVENTF_MIDDLEDOWN, MOUSEEVENTF_MIDDLEUP, MOUSEEVENTF_MOVE,
MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_VIRTUALDESK, MOUSEEVENTF_WHEEL,
MOUSEEVENTF_XDOWN, MOUSEEVENTF_XUP, MOUSEINPUT, VIRTUAL_KEY,
};
use windows::Win32::UI::WindowsAndMessaging::{
GetForegroundWindow, GetSystemMetrics, GetWindowThreadProcessId, SM_CXVIRTUALSCREEN,
@@ -46,11 +46,6 @@ const XBUTTON2: u32 = 0x0002;
pub struct SendInputInjector {
desktop: Option<HDESK>,
/// PT_TOUCH synthetic device, created on the first wire-touch event (a session that never
/// touches never creates one). `None` after a failed create (pre-1809) — touch then stays
/// the historical no-op.
touch: Option<crate::pen::SyntheticTouch>,
touch_failed: bool,
}
// SAFETY: `SendInputInjector` holds only an `Option<HDESK>` (a desktop handle). The host creates
@@ -63,11 +58,7 @@ unsafe impl Send for SendInputInjector {}
impl SendInputInjector {
pub fn open() -> Result<Self> {
let mut me = Self {
desktop: None,
touch: None,
touch_failed: false,
};
let mut me = Self { desktop: None };
me.reattach_input_desktop(); // best-effort
tracing::info!("SendInput injector ready (Win32 KeyboardAndMouse)");
Ok(me)
@@ -306,60 +297,15 @@ impl InputInjector for SendInputInjector {
};
self.send(&[key(ki)])
}
InputKind::TextInput => {
// Committed IME text: one Unicode scalar per event, injected as
// `KEYEVENTF_UNICODE` packets (wScan = UTF-16 unit, no scancode/layout involved
// — the receiving app gets the character verbatim via WM_CHAR). An astral-plane
// scalar (emoji) is its surrogate pair, each unit down+up in order — exactly how
// Windows expects supplementary characters from unicode injection.
let Some(ch) = char::from_u32(event.code) else {
return Ok(()); // lone surrogate / out of range — drop
};
if ch.is_control() {
return Ok(()); // control chars ride the VK path (Enter/Backspace/Tab)
}
let mut units = [0u16; 2];
let mut inputs: Vec<INPUT> = Vec::with_capacity(4);
for &unit in ch.encode_utf16(&mut units).iter() {
for flags in [KEYEVENTF_UNICODE, KEYEVENTF_UNICODE | KEYEVENTF_KEYUP] {
inputs.push(key(KEYBDINPUT {
wVk: VIRTUAL_KEY(0),
wScan: unit,
dwFlags: flags,
time: 0,
dwExtraInfo: 0,
}));
}
}
self.send(&inputs)
}
// Gamepad goes through the XUSB backend.
// Gamepad goes through the XUSB backend. Touch: no SendInput equivalent -> no-op.
InputKind::GamepadButton
| InputKind::GamepadAxis
| InputKind::GamepadState
| InputKind::GamepadRemove
| InputKind::GamepadArrival => Ok(()),
// Wire touch → the PT_TOUCH synthetic pointer device (design/pen-tablet-input.md
// §6; closes the historical SendInput no-op). Lazily created — a session that never
// touches never creates one; a pre-1809 create failure latches back to the no-op.
InputKind::TouchDown | InputKind::TouchMove | InputKind::TouchUp => {
if self.touch.is_none() && !self.touch_failed {
match crate::pen::SyntheticTouch::create() {
Ok(t) => self.touch = Some(t),
Err(e) => {
self.touch_failed = true;
tracing::warn!(
error = %format!("{e:#}"),
"touch: synthetic pointer unavailable — wire touch stays a no-op"
);
}
}
}
if let Some(t) = self.touch.as_mut() {
t.apply(event);
}
Ok(())
}
| InputKind::GamepadArrival
| InputKind::TouchDown
| InputKind::TouchMove
| InputKind::TouchUp => Ok(()),
}
}
}
-91
View File
@@ -174,73 +174,6 @@ pub fn default_backend() -> Backend {
Backend::Unsupported
}
/// Whether the session's inject backend can type **committed text**
/// ([`InputKind::TextInput`] — see `HOST_CAP_TEXT_INPUT`): Windows always (`KEYEVENTF_UNICODE`);
/// Linux only on the wlroots backend (a dedicated virtual keyboard with a dynamically-grown
/// Unicode keymap) — KWin fake-input/libei/gamescope can only press keycodes of the host layout.
/// Consulted at Welcome time to advertise the cap; a mid-session backend switch away from a
/// capable one just degrades to dropped text events (input is lossy by design).
#[cfg(target_os = "windows")]
pub fn text_input_supported() -> bool {
true
}
/// See the Windows variant: Linux types text only through the wlroots virtual-keyboard backend.
#[cfg(target_os = "linux")]
pub fn text_input_supported() -> bool {
matches!(default_backend(), Backend::WlrVirtual)
}
/// No injector ⇒ no text.
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
pub fn text_input_supported() -> bool {
false
}
/// Whether this host can inject full-fidelity stylus input (`HOST_CAP_PEN` —
/// design/pen-tablet-input.md): Linux only, via the [`pen::VirtualPen`] uinput tablet, so the
/// probe is "can we open /dev/uinput" (the same permission the virtual gamepads need) plus the
/// `PUNKTFUNK_PEN=0` operator kill-switch. Consulted at Welcome time; clients without the bit
/// keep folding pen into touch/pointer. Windows PT_PEN synthetic pointers are the design's P3.
#[cfg(target_os = "linux")]
pub fn pen_supported() -> bool {
if std::env::var("PUNKTFUNK_PEN").as_deref() == Ok("0") {
return false;
}
// SAFETY: 'static NUL-terminated path literal; `open` returns a fresh fd (or -1) and
// retains nothing.
let fd = unsafe {
libc::open(
c"/dev/uinput".as_ptr(),
libc::O_RDWR | libc::O_NONBLOCK | libc::O_CLOEXEC,
)
};
if fd < 0 {
return false;
}
// SAFETY: `fd >= 0` is the fd opened above, owned by no one else; closed exactly once here.
unsafe { libc::close(fd) };
true
}
/// Windows: pen (and touch) inject via synthetic pointer devices — available on Win10 1809+,
/// probed by actually creating (and immediately destroying) a PT_PEN device. Same
/// `PUNKTFUNK_PEN=0` kill-switch as Linux. The probe result also stands in for PT_TOUCH
/// (both APIs arrived together in 1809).
#[cfg(target_os = "windows")]
pub fn pen_supported() -> bool {
if std::env::var("PUNKTFUNK_PEN").as_deref() == Ok("0") {
return false;
}
pen::synthetic_pen_available()
}
/// See the Linux/Windows variants — no pen injection elsewhere.
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
pub fn pen_supported() -> bool {
false
}
#[path = "inject/service.rs"]
mod service;
pub use service::InjectorService;
@@ -406,30 +339,6 @@ pub mod gamepad {
pub fn pump_rumble(&mut self, _send: impl FnMut(u16, u16, u16)) {}
}
}
/// Linux: the "Punktfunk Pen" uinput virtual tablet (design/pen-tablet-input.md §5) — the
/// per-session stylus device the native pen plane injects through.
#[cfg(target_os = "linux")]
#[path = "inject/linux/pen.rs"]
pub mod pen;
/// Windows: PT_PEN/PT_TOUCH synthetic pointer devices (design/pen-tablet-input.md §6).
/// `pen::VirtualPen` here is the PT_PEN device; `pen::SyntheticTouch` backs the SendInput
/// injector's wire-touch path.
#[cfg(target_os = "windows")]
#[path = "inject/windows/pointer_windows.rs"]
pub mod pen;
/// Stub — pen injection needs the Linux uinput tablet or Windows synthetic pointers;
/// `pen_supported()` is false here, so no host advertises the cap and no batches arrive.
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
pub mod pen {
use anyhow::{bail, Result};
pub struct VirtualPen;
impl VirtualPen {
pub fn create() -> Result<VirtualPen> {
bail!("no pen injection backend on this platform")
}
pub fn apply_batch(&mut self, _transitions: &[punktfunk_core::quic::PenTransition]) {}
}
}
#[cfg(target_os = "linux")]
#[path = "inject/linux/kwin_fake_input.rs"]
mod kwin_fake_input;
-232
View File
@@ -1,232 +0,0 @@
//! Client-side cursor rendering (design/remote-desktop-sweep.md M2): the host forwards the
//! pointer's SHAPE (reliable control stream, cached by serial) and per-frame STATE (lossy
//! `0xD0` — position/visibility), and WE draw it as a real OS cursor — pointer feel stops
//! paying the video round-trip (the Parsec/RDP model). Active only when the session
//! negotiated it (`HOST_CAP_CURSOR` in the Welcome — the host stopped compositing then) and
//! only applied while the DESKTOP mouse model is engaged: under capture the pointer is
//! relative-locked (SDL hides it) and games draw their own cursor in-frame.
//!
//! The host sends the bitmap in host-FRAMEBUFFER pixels, whose size tracks the host virtual
//! display's DPI scaling (32 px at 100%, 96 px at 300%). Drawn 1:1 it balloons on a high-DPI
//! host; instead we scale it by the SAME aspect-fit factor the video is drawn at
//! (`min(window_px/mode)`), so the pointer stays sized to the streamed desktop at any host
//! scaling. SDL cursors are fixed-size from their surface (no draw-time scaling), so we cache
//! shapes RAW and resample per install — rebuilding when the serial OR the fit changes.
use punktfunk_core::client::NativeClient;
use punktfunk_core::quic::{CursorState, HOST_CAP_CURSOR};
use sdl3::mouse::{Cursor, MouseUtil, SystemCursor};
use sdl3::pixels::PixelFormat;
use sdl3::surface::Surface;
use std::collections::HashMap;
use std::time::Duration;
/// Shape serials cached at most — cursors cycle through a handful of shapes (arrow, I-beam,
/// resize…); a runaway host can't grow the map past this (the cache resets, shapes re-arrive
/// on the reliable stream via the serial-miss path).
const SHAPE_CACHE_MAX: usize = 64;
/// A forwarded cursor shape held RAW (host-framebuffer-pixel bytes + hotspot), so it can be
/// rebuilt into a scaled OS cursor whenever the video-fit changes (a window resize). Caching a
/// fixed-size `Cursor` instead would freeze the pointer at its build-time size.
struct RawShape {
rgba: Vec<u8>,
w: u32,
h: u32,
hot_x: u32,
hot_y: u32,
}
pub struct CursorChannel {
/// The Welcome carried `HOST_CAP_CURSOR` — the host forwards instead of compositing.
negotiated: bool,
/// Serial → raw forwarded shape (bounded by [`SHAPE_CACHE_MAX`]).
shapes: HashMap<u32, RawShape>,
/// The serial + fit scale the currently-installed OS cursor was built at (`None` =
/// default/system cursor). A change in EITHER forces a rebuild.
installed: Option<(u32, f32)>,
/// Keeps the installed `Cursor` alive — SDL requires it to outlive its `set()`.
installed_cursor: Option<Cursor>,
/// Latest `0xD0` state (latest-wins across a drained batch).
state: Option<CursorState>,
}
impl CursorChannel {
pub fn new(connector: &NativeClient) -> CursorChannel {
let negotiated = connector.host_caps() & HOST_CAP_CURSOR != 0;
if negotiated {
tracing::info!("cursor channel negotiated — host cursor renders locally");
}
CursorChannel {
negotiated,
shapes: HashMap::new(),
installed: None,
installed_cursor: None,
state: None,
}
}
/// Whether the host forwards the cursor this session (it no longer composites one).
pub fn negotiated(&self) -> bool {
self.negotiated
}
/// The latest drained `0xD0` state — the run loop reads `relative_hint` off it for the
/// M3 host-driven mode flip (and `x`/`y` as the reappear position when leaving relative).
pub fn state(&self) -> Option<CursorState> {
self.state
}
/// Drain the two planes and apply the newest state — once per run-loop iteration.
/// `desktop_active` = the desktop mouse model is engaged (captured + desktop): only then
/// do we own the local cursor's shape/visibility; under capture SDL's relative mode owns
/// it, and released the system cursor must look normal. `fit_scale` is host-framebuffer
/// pixels → window pixels (the aspect-fit factor the video is drawn at); the shape is
/// resampled by it so the pointer matches the streamed desktop at any host DPI.
pub fn pump(
&mut self,
connector: &NativeClient,
mouse: &MouseUtil,
desktop_active: bool,
fit_scale: f32,
) {
if !self.negotiated {
return;
}
while let Ok(shape) = connector.next_cursor_shape(Duration::ZERO) {
if self.shapes.len() >= SHAPE_CACHE_MAX {
// Degenerate host: reset — live shapes re-install via the serial-miss path.
self.shapes.clear();
self.installed = None;
}
let (w, h) = (shape.w as u32, shape.h as u32);
if w == 0 || h == 0 || shape.rgba.len() < (w * h * 4) as usize {
tracing::warn!(w, h, "cursor shape malformed — ignored");
continue;
}
// A re-sent serial replaces its entry; force re-install if it's current.
if matches!(self.installed, Some((s, _)) if s == shape.serial) {
self.installed = None;
}
self.shapes.insert(
shape.serial,
RawShape {
rgba: shape.rgba,
w,
h,
hot_x: shape.hot_x as u32,
hot_y: shape.hot_y as u32,
},
);
}
while let Ok(st) = connector.next_cursor_state(Duration::ZERO) {
self.state = Some(st); // latest wins
}
if !desktop_active {
// Capture mode / released: hand the cursor back to the system default so a
// released pointer over the window doesn't wear the host's shape.
if self.installed.take().is_some() {
if let Ok(c) = Cursor::from_system(SystemCursor::Arrow) {
c.set();
self.installed_cursor = Some(c); // keep it alive past set()
}
}
return;
}
let Some(st) = self.state else { return };
if st.visible() && self.installed != Some((st.serial, fit_scale)) {
if let Some(shape) = self.shapes.get(&st.serial) {
match build_scaled_cursor(shape, fit_scale) {
Ok(cursor) => {
cursor.set();
self.installed = Some((st.serial, fit_scale));
self.installed_cursor = Some(cursor); // outlive set()
}
Err(e) => tracing::warn!(error = %e, w = shape.w, h = shape.h,
"cursor shape rejected by SDL — keeping the previous cursor"),
}
}
// Serial miss: the (reliable) shape hasn't landed yet — keep the previous
// cursor for the RTT rather than flashing default.
}
// Visibility follows the host (a host app hid its pointer ⇒ ours hides too). Queried,
// not shadowed, so apply_capture's own show/hide calls can never desync us.
if mouse.is_cursor_showing() != st.visible() {
mouse.show_cursor(st.visible());
}
}
}
/// Resample a raw shape by `fit_scale` and build an SDL color cursor from it. The hotspot scales
/// with the bitmap so the click point stays true. `fit_scale <= 0` (or a degenerate result) is
/// clamped so we always hand SDL a ≥1×1 surface.
fn build_scaled_cursor(shape: &RawShape, fit_scale: f32) -> Result<Cursor, String> {
let scale = if fit_scale.is_finite() && fit_scale > 0.0 {
fit_scale
} else {
1.0
};
let dw = ((shape.w as f32 * scale).round() as u32).max(1);
let dh = ((shape.h as f32 * scale).round() as u32).max(1);
let hot_x = ((shape.hot_x as f32 * scale).round() as u32).min(dw - 1) as i32;
let hot_y = ((shape.hot_y as f32 * scale).round() as u32).min(dh - 1) as i32;
if dw == shape.w && dh == shape.h {
// 1:1 fit (mode == window) — no resample, feed the bytes straight through.
let mut data = shape.rgba.clone();
let surf = Surface::from_data(
&mut data,
shape.w,
shape.h,
shape.w * 4,
PixelFormat::RGBA32,
)
.map_err(|e| e.to_string())?;
return Cursor::from_surface(&surf, hot_x, hot_y).map_err(|e| e.to_string());
}
let mut scaled = resample_rgba(&shape.rgba, shape.w, shape.h, dw, dh);
let surf = Surface::from_data(&mut scaled, dw, dh, dw * 4, PixelFormat::RGBA32)
.map_err(|e| e.to_string())?;
Cursor::from_surface(&surf, hot_x, hot_y).map_err(|e| e.to_string())
}
/// Area-average resample of a straight-alpha RGBA bitmap `(sw×sh) → (dw×dh)`. Averaging is done on
/// PREMULTIPLIED colour (weighting each source texel by its alpha) so transparent-pixel colour
/// can't bleed into the fringe, then un-premultiplied back to straight alpha. Handles both down-
/// and up-scale; for a cursor the common case is a downscale (high-DPI host → smaller pointer).
fn resample_rgba(src: &[u8], sw: u32, sh: u32, dw: u32, dh: u32) -> Vec<u8> {
let mut out = vec![0u8; (dw * dh * 4) as usize];
let fx = sw as f32 / dw as f32;
let fy = sh as f32 / dh as f32;
for dy in 0..dh {
let sy0 = (dy as f32 * fy).floor() as u32;
let sy1 = (((dy + 1) as f32 * fy).ceil() as u32).clamp(sy0 + 1, sh);
for dx in 0..dw {
let sx0 = (dx as f32 * fx).floor() as u32;
let sx1 = (((dx + 1) as f32 * fx).ceil() as u32).clamp(sx0 + 1, sw);
let (mut r, mut g, mut b, mut a_sum, mut n) = (0f32, 0f32, 0f32, 0f32, 0f32);
for sy in sy0..sy1 {
for sx in sx0..sx1 {
let i = ((sy * sw + sx) * 4) as usize;
let a = src[i + 3] as f32 / 255.0;
r += src[i] as f32 * a;
g += src[i + 1] as f32 * a;
b += src[i + 2] as f32 * a;
a_sum += a;
n += 1.0;
}
}
let di = ((dy * dw + dx) * 4) as usize;
if a_sum > 0.0 {
out[di] = (r / a_sum).round().clamp(0.0, 255.0) as u8;
out[di + 1] = (g / a_sum).round().clamp(0.0, 255.0) as u8;
out[di + 2] = (b / a_sum).round().clamp(0.0, 255.0) as u8;
out[di + 3] = (a_sum / n * 255.0).round().clamp(0.0, 255.0) as u8;
}
// else fully transparent — already zero-filled.
}
}
out
}
-14
View File
@@ -133,20 +133,6 @@ impl Capture {
Some(self.desktop)
}
/// Set the mouse model directly (the M3 host-driven flip — `relative_hint` says a host
/// app grabbed/hid the pointer, so run relative; hint clear = back to absolute). Same
/// gating and motion hygiene as [`toggle_desktop`](Self::toggle_desktop); returns whether
/// the model actually changed.
pub fn set_desktop(&mut self, on: bool) -> bool {
if !self.abs_ok || self.desktop == on {
return false;
}
self.desktop = on;
self.pending_rel = (0, 0);
self.pending_abs = None;
true
}
/// Whether a regained focus should re-engage: yes unless the user released
/// deliberately (the chord keeps its meaning across an Alt-Tab).
pub fn should_reengage(&self) -> bool {
-2
View File
@@ -16,8 +16,6 @@
#[cfg(any(target_os = "linux", windows))]
pub mod csc;
#[cfg(any(target_os = "linux", windows))]
pub mod cursor;
#[cfg(windows)]
pub mod d3d11;
#[cfg(target_os = "linux")]
+10 -156
View File
@@ -233,19 +233,6 @@ struct StreamState {
/// window-normalized position must be re-based onto the content rect). `None` until
/// the first frame; touches before then have nothing to map onto and are dropped.
last_video: Option<(u32, u32)>,
/// Client-side cursor rendering (M2 cursor channel) — created with the connector; inert
/// when the host didn't negotiate the channel.
cursor_chan: Option<crate::cursor::CursorChannel>,
/// Last observed `relative_hint` (M3): the auto-flip fires on CHANGES only, so it never
/// fights a user who chorded away from the hinted model.
last_hint: Option<bool>,
/// The user flipped the model manually (⌃⌥⇧M) — the standing hint stops driving until
/// the HOST's intent next changes (a fresh hint edge clears this and applies).
hint_override: bool,
/// Last `CursorRenderMode.client_draws` told to the host (§8 mid-stream render flip);
/// `None` = nothing sent yet. Edge-detected each iteration from the live mouse model, so
/// the chord, the M3 auto-flip, and engage/release all reconcile through one path.
sent_client_draws: Option<bool>,
}
impl StreamState {
@@ -276,10 +263,6 @@ impl StreamState {
frames: wake_rx,
connector: None,
capture: None,
cursor_chan: None,
last_hint: None,
hint_override: false,
sent_client_draws: None,
force_software,
canceled: false,
ready_announced: false,
@@ -571,27 +554,18 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
// Mouse model flip (capture ⇄ desktop) — applies immediately when
// engaged; a released stream just changes what the next engage does.
if chord && sc == Scancode::M {
if let Some(st) = stream.as_mut() {
let mut flipped = false;
if let Some(cap) = st.capture.as_mut() {
match cap.toggle_desktop() {
Some(desktop) => {
if cap.captured() {
apply_capture(&mut window, &mouse, true, desktop);
}
flipped = true;
tracing::info!(desktop, "chord: mouse mode");
if let Some(cap) = stream.as_mut().and_then(|s| s.capture.as_mut()) {
match cap.toggle_desktop() {
Some(desktop) => {
if cap.captured() {
apply_capture(&mut window, &mouse, true, desktop);
}
None => tracing::info!(
"chord: mouse mode — host has no absolute pointer \
(gamescope), staying captured"
),
tracing::info!(desktop, "chord: mouse mode");
}
}
// A manual flip outranks the standing hint until the host's
// intent next CHANGES (M3 — the hint edge clears this).
if flipped {
st.hint_override = true;
None => tracing::info!(
"chord: mouse mode — host has no absolute pointer \
(gamescope), staying captured"
),
}
}
continue;
@@ -770,75 +744,6 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
if let Some(cap) = stream.as_mut().and_then(|s| s.capture.as_mut()) {
cap.flush_motion();
}
// Cursor channel (M2): drain forwarded shape/state and drive the local OS cursor —
// only meaningful in the desktop mouse model (capture's relative lock hides it).
if let Some(st) = stream.as_mut() {
// Host-framebuffer px → window px: the aspect-fit factor the video is drawn at
// (same `min(surface/content)` as `finger_to_content`). The forwarded pointer is
// resampled by it so a high-DPI host's oversized bitmap lands sized to the streamed
// desktop rather than ballooning. 1:1 until the first frame gives `last_video`.
let fit_scale = st.last_video.map_or(1.0, |(vw, vh)| {
let (pw, ph) = window.size_in_pixels();
(pw as f32 / vw.max(1) as f32).min(ph as f32 / vh.max(1) as f32)
});
if let (Some(chan), Some(c)) = (st.cursor_chan.as_mut(), st.connector.as_ref()) {
let desktop_active = st
.capture
.as_ref()
.is_some_and(|cap| cap.captured() && cap.desktop());
chan.pump(c, &mouse, desktop_active, fit_scale);
// §8 mid-stream render flip: tell the host who renders the pointer whenever
// the local model changes. Desktop-active = we draw it (host excludes +
// forwards); anything else — the capture model OR a released pointer — the
// host composites it into the video (full fidelity, the pre-channel look).
// One edge-detected reconciler covers the chord, the M3 auto-flip, and
// engage/release alike.
if chan.negotiated() && st.sent_client_draws != Some(desktop_active) {
st.sent_client_draws = Some(desktop_active);
let _ = c.set_cursor_render(desktop_active);
}
}
// M3 — host-driven mode flip: `relative_hint` set = a host app grabbed/hid the
// pointer (run captured relative, like a game expects); clear = the desktop is
// back (return to absolute, local cursor reappearing at the host's position).
// Edge-triggered so a user's manual chord isn't fought: the override latch
// holds until the HOST's intent next changes.
let hint_state = st.cursor_chan.as_ref().and_then(|ch| ch.state());
if let Some(hs) = hint_state {
let hint = hs.relative_hint();
if st.last_hint != Some(hint) {
st.last_hint = Some(hint);
st.hint_override = false;
}
if !st.hint_override {
let video = st.last_video;
if let Some(cap) = st.capture.as_mut() {
// Desired model: hint ⇒ capture (desktop off); clear ⇒ desktop on.
if cap.captured() && cap.set_desktop(!hint) {
apply_capture(&mut window, &mouse, true, cap.desktop());
if cap.desktop() {
// Reappear where the host last had the pointer, so the
// hand-back is seamless (Parsec's positionX/Y idea).
if let Some(video) = video {
let (wx, wy) = content_to_window(
window.size(),
window.size_in_pixels(),
video,
hs.x,
hs.y,
);
mouse.warp_mouse_in_window(&window, wx, wy);
}
}
tracing::info!(
desktop = cap.desktop(),
"host cursor hint: mouse model flipped"
);
}
}
}
}
}
// Text input follows the overlay's editing state (edge-triggered).
let want_text = overlay.as_ref().is_some_and(|o| o.text_input_active());
@@ -983,7 +888,6 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
cap.engage(); // capture engages when the stream starts (ui_stream parity)
apply_capture(&mut window, &mouse, true, cap.desktop());
st.capture = Some(cap);
st.cursor_chan = Some(crate::cursor::CursorChannel::new(&c));
st.connector = Some(c);
if let Some(f) = opts.on_connected.as_mut() {
f(fingerprint);
@@ -1762,32 +1666,6 @@ fn finger_to_content(
(cx.round() as i32, cy.round() as i32, dw as u32, dh as u32)
}
/// The inverse direction of [`finger_to_content`] for the M3 reappear warp: a HOST-frame pixel
/// (`video` space — what `CursorState` carries) → LOGICAL window coordinates (what
/// `warp_mouse_in_window` takes). Maps through the aspect-fit letterbox (physical), then
/// physical → logical via the window's pixel density; out-of-range host coords clamp into the
/// content rect so the warp always lands on the video.
fn content_to_window(
logical: (u32, u32),
surface: (u32, u32),
video: (u32, u32),
x: i32,
y: i32,
) -> (f32, f32) {
let (pw, ph) = (f64::from(surface.0), f64::from(surface.1));
let (vw, vh) = (f64::from(video.0.max(1)), f64::from(video.1.max(1)));
let scale = (pw / vw).min(ph / vh);
let (dw, dh) = ((vw * scale).max(1.0), (vh * scale).max(1.0));
let ox = (pw - dw) / 2.0;
let oy = (ph - dh) / 2.0;
let px = ox + (f64::from(x).clamp(0.0, vw - 1.0)) * scale;
let py = oy + (f64::from(y).clamp(0.0, vh - 1.0)) * scale;
// Physical → logical (HiDPI): the ratio of the window's logical size to its pixel size.
let lx = px * f64::from(logical.0) / pw.max(1.0);
let ly = py * f64::from(logical.1) / ph.max(1.0);
(lx as f32, ly as f32)
}
/// The presenter's share of the unified stats window — folded into each printed line.
#[derive(Default)]
struct PresentedWindow {
@@ -1885,30 +1763,6 @@ fn stats_text(
mod tests {
use super::*;
#[test]
fn content_to_window_inverts_the_letterbox() {
// 1920×1080 video letterboxed in a 1600×1200 (4:3) window at 2× HiDPI: pillarless
// top/bottom bars — scale = 1600/1920, dh = 900, oy = 150 (physical).
let logical = (800u32, 600u32);
let surface = (1600u32, 1200u32);
let video = (1920u32, 1080u32);
// The host-frame center must land at the logical window center.
let (wx, wy) = content_to_window(logical, surface, video, 960, 540);
assert!((wx - 400.0).abs() < 1.0, "wx = {wx}");
assert!((wy - 300.0).abs() < 1.0, "wy = {wy}");
// Roundtrip through the forward mapping: normalized window pos → the same host
// content-rect pixel (finger_to_content returns content-RECT coords, i.e. the
// host pixel scaled by the letterbox factor).
let (nx, ny) = (wx / logical.0 as f32, wy / logical.1 as f32);
let (cx, cy, dw, dh) = finger_to_content(surface, video, nx, ny);
assert_eq!((dw, dh), (1600, 900));
assert!((cx - 800).abs() <= 1, "cx = {cx}"); // 960 * (1600/1920)
assert!((cy - 450).abs() <= 1, "cy = {cy}"); // 540 * ( 900/1080)
// Out-of-range host coords clamp into the video, never the bars.
let (_, wy_clamped) = content_to_window(logical, surface, video, 0, 10_000);
assert!(wy_clamped <= 300.0 + 225.0 + 1.0, "wy = {wy_clamped}"); // ≤ bottom of content
}
#[test]
fn resize_decision_follows_the_d2_discipline() {
let t0 = Instant::now();
+4 -5
View File
@@ -72,14 +72,13 @@ pub use session::{session_epoch, try_recover_session};
#[path = "vdisplay/routing.rs"]
pub(crate) mod routing;
pub use routing::{
apply_input_env, managed_session_available, restore_managed_session,
restore_takeover_on_startup, start_restore_worker, wants_dedicated_game_session,
apply_input_env, restore_managed_session, restore_takeover_on_startup, start_restore_worker,
wants_dedicated_game_session,
};
#[cfg(target_os = "linux")]
pub use routing::{
cancel_pending_tv_restore, dedicated_game_exited, gamescope_xwayland_cursor_targets,
launch_into_gamescope_session, launch_is_nested, steam_appid_from_launch,
watch_steam_game_exit,
cancel_pending_tv_restore, dedicated_game_exited, launch_into_gamescope_session,
launch_is_nested, steam_appid_from_launch, watch_steam_game_exit,
};
/// Compositors punktfunk knows how to drive (plan §6).
@@ -70,12 +70,6 @@ pub struct VirtualOutput {
/// Linux-only (the keep-alive pool is Linux).
#[cfg(target_os = "linux")]
pub pool_gen: Option<u64>,
/// The backend created the output at a SACRIFICIAL mode and the producer will renegotiate the
/// live stream to `preferred_mode`'s dims (KWin's screencast only rebuilds its format offer —
/// and thus its refresh cap — on a size change while recording; see kwin.rs `create`). The
/// capturer must hold frames until that renegotiation lands. Linux-only.
#[cfg(target_os = "linux")]
pub expect_exact_dims: bool,
}
impl VirtualOutput {
@@ -99,8 +93,6 @@ impl VirtualOutput {
reused_gen: None,
#[cfg(target_os = "linux")]
pool_gen: None,
#[cfg(target_os = "linux")]
expect_exact_dims: false,
}
}
}
@@ -142,25 +134,6 @@ pub trait VirtualDisplay: Send {
/// the backend's default EDID. Default: no-op — only the Windows pf-vdisplay backend can mint
/// per-monitor EDIDs today (the Linux compositors' virtual outputs take no EDID from us).
fn set_client_hdr(&mut self, _hdr: Option<punktfunk_core::quic::HdrMeta>) {}
/// Ask the backend for an OUT-OF-BAND cursor on the created output (the cursor channel):
/// the compositor/OS stops compositing the pointer into captured frames and the capture
/// layer surfaces shape/position separately. Carried on the backend instance; set once
/// before [`create`](Self::create) (both session paths pass `cursor_forward`). Off = the
/// compositor EMBEDS the pointer into frames — zero host-side cursor work, the pre-channel
/// path — which is what every session without the negotiated cursor cap gets (Moonlight /
/// GameStream / legacy clients / capture-mode starts), mirroring the Windows no-regression
/// gate. Implementations: Windows pf-vdisplay (IddCx hardware cursor, driver proto v5);
/// KWin (zkde `pointer` metadata vs embedded); Mutter (`cursor-mode` metadata vs embedded);
/// wlroots/hyprland (portal `CursorMode`). Default: no-op (gamescope has no cursor either
/// way — see the Phase C source).
fn set_hw_cursor(&mut self, _on: bool) {}
/// The out-of-band-cursor request currently set (see [`set_hw_cursor`](Self::set_hw_cursor)).
/// The registry includes it in the keep-alive REUSE key: a kept embedded-pointer display can
/// never serve a cursor-channel session (its stream has no cursor metadata to forward) nor
/// vice versa (the pointer would be missing from frames).
fn hw_cursor(&self) -> bool {
false
}
/// The stable identity slot the backend resolved for the most recent [`create`](Self::create) —
/// the per-client id the identity policy assigned (`Some`), or `None` for shared/anonymous. The
/// registry reads it right after `create` to key the display's group **arrangement** (manual
@@ -253,7 +253,6 @@ impl VirtualDisplay for GamescopeDisplay {
ownership: DisplayOwnership::External,
reused_gen: None,
pool_gen: None,
expect_exact_dims: false,
});
}
check_gamescope_version(); // diagnostic only — warns on known-deadlock-prone versions
@@ -410,7 +409,6 @@ fn managed_output(node_id: u32, mode: Mode) -> VirtualOutput {
ownership: DisplayOwnership::SessionManaged,
reused_gen: None,
pool_gen: None,
expect_exact_dims: false,
}
}
@@ -508,7 +506,7 @@ pub fn launch_into_session(cmd: &str) -> Result<std::process::Child> {
let mut c = Command::new("sh");
c.arg("-c").arg(cmd);
match discover_session_display_env() {
Some((x11, wayland, _xauth)) => {
Some((x11, wayland)) => {
tracing::info!(
command = %cmd,
x11_display = x11.as_deref().unwrap_or("-"),
@@ -533,71 +531,11 @@ pub fn launch_into_session(cmd: &str) -> Result<std::process::Child> {
.context("spawn launch command into gamescope session")
}
/// EVERY nested Xwayland the running gamescope session exposes, as `(DISPLAY, XAUTHORITY)` pairs
/// for the XFixes cursor source (remote-desktop-sweep Phase C). gamescope can run several
/// (`--xwayland-count N` — Steam Gaming Mode uses 2: one for Big Picture, one for the game), and
/// the pointer lives on whichever is FOCUSED — so the source connects to all and follows the one
/// whose pointer moves. The host is not a gamescope child, so gamescope's auth cookie rides along
/// when a process exposes it. Empty when no gamescope session is running / none exposes a `DISPLAY`.
#[cfg(target_os = "linux")]
pub(crate) fn xwayland_cursor_targets() -> Vec<(String, Option<String>)> {
// SAFETY: `getuid()` is a parameterless POSIX call that always succeeds and touches no memory.
let uid = unsafe { libc::getuid() };
let mut out: Vec<(String, Option<String>)> = Vec::new();
let Ok(entries) = std::fs::read_dir("/proc") else {
return out;
};
for e in entries.flatten() {
let name = e.file_name();
let Some(pid_str) = name.to_str() else {
continue;
};
if !pid_str.bytes().all(|b| b.is_ascii_digit()) {
continue;
}
let Ok(md) = std::fs::metadata(e.path()) else {
continue;
};
use std::os::unix::fs::MetadataExt;
if md.uid() != uid {
continue;
}
let Ok(raw) = std::fs::read(e.path().join("environ")) else {
continue;
};
let (mut display, mut is_gamescope, mut xauth) = (None, false, None);
for kv in raw.split(|&b| b == 0) {
let kv = String::from_utf8_lossy(kv);
if kv.starts_with("GAMESCOPE_WAYLAND_DISPLAY=") {
is_gamescope = true;
} else if let Some(v) = kv.strip_prefix("DISPLAY=") {
if !v.is_empty() {
display = Some(v.to_string());
}
} else if let Some(v) = kv.strip_prefix("XAUTHORITY=") {
if !v.is_empty() {
xauth = Some(v.to_string());
}
}
}
if let (true, Some(d)) = (is_gamescope, display) {
// Distinct DISPLAY only; prefer the first non-empty XAUTHORITY seen for it.
match out.iter_mut().find(|(dd, _)| *dd == d) {
Some((_, xa)) if xa.is_none() => *xa = xauth,
Some(_) => {}
None => out.push((d, xauth)),
}
}
}
out
}
/// Find the live gamescope session's `(DISPLAY, WAYLAND_DISPLAY, XAUTHORITY)` by scanning same-uid
/// processes for one whose environment carries `GAMESCOPE_WAYLAND_DISPLAY` (gamescope sets it for
/// everything it runs — Steam, the game, our own nested `sh`). The Wayland value returned is that
/// gamescope socket; `DISPLAY` is the nested Xwayland; `XAUTHORITY` is its auth file (for X
/// clients that aren't gamescope children). Any one can be individually absent.
fn discover_session_display_env() -> Option<(Option<String>, Option<String>, Option<String>)> {
/// Find the live gamescope session's `(DISPLAY, WAYLAND_DISPLAY)` by scanning same-uid processes
/// for one whose environment carries `GAMESCOPE_WAYLAND_DISPLAY` (gamescope sets it for everything
/// it runs — Steam, the game, our own nested `sh`). The Wayland value returned is that gamescope
/// socket; `DISPLAY` is the nested Xwayland. Either can be individually absent.
fn discover_session_display_env() -> Option<(Option<String>, Option<String>)> {
// SAFETY: `getuid()` is a parameterless POSIX call that always succeeds and touches no memory.
let uid = unsafe { libc::getuid() };
for e in std::fs::read_dir("/proc").ok()?.flatten() {
@@ -620,7 +558,6 @@ fn discover_session_display_env() -> Option<(Option<String>, Option<String>, Opt
};
let mut display = None;
let mut gs_wayland = None;
let mut xauth = None;
for kv in raw.split(|&b| b == 0) {
let kv = String::from_utf8_lossy(kv);
if let Some(v) = kv.strip_prefix("GAMESCOPE_WAYLAND_DISPLAY=") {
@@ -631,15 +568,11 @@ fn discover_session_display_env() -> Option<(Option<String>, Option<String>, Opt
if !v.is_empty() {
display = Some(v.to_string());
}
} else if let Some(v) = kv.strip_prefix("XAUTHORITY=") {
if !v.is_empty() {
xauth = Some(v.to_string());
}
}
}
// Only a process INSIDE a gamescope session (it has the marker var) is a valid source.
if gs_wayland.is_some() {
return Some((display, gs_wayland, xauth));
return Some((display, gs_wayland));
}
}
None
@@ -1183,26 +1116,6 @@ pub fn schedule_restore_tv_session() {
}
}
/// Does any DRM connector report a physically `connected` display? Scans
/// `/sys/class/drm/*/status` — only connector nodes (`card0-eDP-1`, `card0-HDMI-A-1`, …) have a
/// `status` file, so the bare `cardN` device dirs and `renderD*` nodes filter themselves out. A
/// headless box (VM, panel-less mini PC) has none — in which case a "restore to the physical
/// panel" can only fail, gamescope having no output to drive. Errors (no DRM at all, sysfs
/// unreadable) read as headless: the safe direction is keeping the working session.
fn physical_display_connected() -> bool {
connected_connector_under(std::path::Path::new("/sys/class/drm"))
}
/// [`physical_display_connected`] against an arbitrary sysfs root (the unit-testable core).
fn connected_connector_under(base: &std::path::Path) -> bool {
let Ok(entries) = std::fs::read_dir(base) else {
return false;
};
entries.flatten().any(|e| {
std::fs::read_to_string(e.path().join("status")).is_ok_and(|s| s.trim() == "connected")
})
}
/// Tear down our host-managed session (freeing Steam) and restart the autologin gaming session(s)
/// we stopped on connect — so the TV returns to gaming mode when no one is streaming. Invoked by
/// [`start_restore_worker`] once the debounce deadline passes; takes the stopped-unit list so a
@@ -1214,19 +1127,6 @@ fn do_restore_tv_session() {
{
let mut took = STEAMOS_TOOK_OVER.lock().unwrap_or_else(|e| e.into_inner());
if *took {
// A box with no physically connected display (a VM, a panel-less mini PC) has no
// "physical gaming session" to restore TO: removing the drop-in and restarting the
// target just crash-loops gamescope (no output to drive) and strands every later
// connect on "no usable compositor". Keep the headless session — and the takeover
// state, so a same-mode reconnect reuses it warm — instead. Checked at restore time
// (not connect time) so plugging a panel in later restores normally.
if !physical_display_connected() {
tracing::info!(
"gamescope (SteamOS): no physical display connected — keeping the headless \
session (nothing to restore to)"
);
return;
}
*took = false;
clear_takeover(); // A3: takeover undone — drop the persisted crash-restore marker
*MANAGED_SESSION.lock().unwrap_or_else(|e| e.into_inner()) = None;
@@ -1326,31 +1226,15 @@ pub fn start_restore_worker() -> std::sync::Arc<()> {
/// session). Shared by the attach and host-managed-session paths.
fn point_injector_at_eis() {
match find_gamescope_eis_socket() {
Some(sock) => {
// Relay format: line 1 = socket, optional line 2 = the session's CURRENT output
// size as "WxH". gamescope's EIS advertises only a degenerate INT32_MAX region, so
// the injector can't learn the output geometry from the protocol — the hint lets
// it scale normalized client positions correctly even when the client streams at
// a different resolution than the session runs (foreign attach, supersample).
let size = current_gamescope_output_size();
let body = match size {
Some((w, h)) => format!("{sock}\n{w}x{h}"),
None => sock.clone(),
};
match std::fs::write(ei_socket_file(), body) {
Ok(()) => {
tracing::info!(
socket = %sock,
output = ?size,
"gamescope: pointed injector at the session's EIS socket"
)
}
Err(e) => tracing::warn!(
error = %e,
"gamescope: could not write the EIS relay file — input may not reach the session"
),
Some(sock) => match std::fs::write(ei_socket_file(), &sock) {
Ok(()) => {
tracing::info!(socket = %sock, "gamescope: pointed injector at the session's EIS socket")
}
}
Err(e) => tracing::warn!(
error = %e,
"gamescope: could not write the EIS relay file — input may not reach the session"
),
},
None => tracing::warn!(
"gamescope: no connectable gamescope EIS socket found — input won't reach the session"
),
@@ -1640,35 +1524,7 @@ impl Drop for GamescopeProc {
#[cfg(test)]
mod tests {
use super::{
cgroup_is_punktfunk_owned, connected_connector_under, is_steam_launch,
shape_dedicated_command,
};
#[test]
fn connector_status_scan() {
let base = std::env::temp_dir().join(format!("pf-drm-scan-{}", std::process::id()));
let mk = |name: &str, status: Option<&str>| {
let dir = base.join(name);
std::fs::create_dir_all(&dir).unwrap();
if let Some(s) = status {
std::fs::write(dir.join("status"), s).unwrap();
}
};
// Headless layout: device + render nodes only (no status files) → not connected.
mk("card0", None);
mk("renderD128", None);
assert!(!connected_connector_under(&base));
// Connectors present but nothing plugged in → still not connected.
mk("card0-HDMI-A-1", Some("disconnected\n"));
assert!(!connected_connector_under(&base));
// A live panel → connected.
mk("card0-eDP-1", Some("connected\n"));
assert!(connected_connector_under(&base));
// A missing base dir (no DRM at all) reads as headless.
assert!(!connected_connector_under(&base.join("nope")));
std::fs::remove_dir_all(&base).unwrap();
}
use super::{cgroup_is_punktfunk_owned, is_steam_launch, shape_dedicated_command};
#[test]
fn steam_launch_detection() {
@@ -92,19 +92,11 @@ fn next_output_name() -> String {
/// The Hyprland virtual-display driver. Stateless — each [`create`](VirtualDisplay::create) adds one
/// named headless output and spins up a portal thread owning the cast on it.
pub struct HyprlandDisplay {
/// Out-of-band cursor request (`set_hw_cursor`, the negotiated cursor channel): portal
/// `CursorMode::Metadata` — shapes/positions ride `SPA_META_Cursor` for the channel + the
/// composite blend. Off (every non-channel session): `Embedded` — the compositor paints the
/// pointer into frames, zero host-side cursor work (the pre-channel default this backend
/// always had). ⚠️ Metadata is UNTESTED on-glass for this backend (Phase B wired it so the
/// channel isn't silently dead here; KWin/Mutter are the validated legs).
hw_cursor: bool,
}
pub struct HyprlandDisplay;
impl HyprlandDisplay {
pub fn new() -> Result<Self> {
Ok(HyprlandDisplay { hw_cursor: false })
Ok(HyprlandDisplay)
}
}
@@ -149,14 +141,6 @@ impl VirtualDisplay for HyprlandDisplay {
"hyprland"
}
fn set_hw_cursor(&mut self, on: bool) {
self.hw_cursor = on;
}
fn hw_cursor(&self) -> bool {
self.hw_cursor
}
fn create(&mut self, mode: Mode) -> Result<VirtualOutput> {
// Log the permission-system caveat once per process (silent black frames otherwise).
preflight_once();
@@ -183,10 +167,9 @@ impl VirtualDisplay for HyprlandDisplay {
let (setup_tx, setup_rx) = std::sync::mpsc::channel::<Result<(OwnedFd, u32), String>>();
let stop = Arc::new(AtomicBool::new(false));
let stop_thread = stop.clone();
let hw_cursor = self.hw_cursor;
thread::Builder::new()
.name("punktfunk-hypr-vout".into())
.spawn(move || portal_thread(setup_tx, stop_thread, hw_cursor))
.spawn(move || portal_thread(setup_tx, stop_thread))
.context("spawn hyprland portal thread")?;
let (fd, node_id) = match setup_rx.recv_timeout(Duration::from_secs(20)) {
@@ -216,7 +199,6 @@ impl VirtualDisplay for HyprlandDisplay {
ownership: DisplayOwnership::Owned,
reused_gen: None,
pool_gen: None,
expect_exact_dims: false,
})
}
}
@@ -508,17 +490,7 @@ fn ensure_xdph_config() -> Result<()> {
/// stopped (the zbus connection is the cast's lifetime). xdph answers source selection via our
/// custom picker, no dialog. (Kept separate from wlroots' copy so each wlr-family backend stays
/// self-owned per D1; unify if they ever diverge no further.)
fn portal_thread(
setup_tx: Sender<Result<(OwnedFd, u32), String>>,
stop: Arc<AtomicBool>,
hw_cursor: bool,
) {
// Portal cursor mode per the session's channel negotiation (see the struct doc).
let cursor_mode = if hw_cursor {
CursorMode::Metadata
} else {
CursorMode::Embedded
};
fn portal_thread(setup_tx: Sender<Result<(OwnedFd, u32), String>>, stop: Arc<AtomicBool>) {
use ashpd::desktop::screencast::{CursorMode, Screencast, SelectSourcesOptions, SourceType};
use ashpd::desktop::PersistMode;
use ashpd::enumflags2::BitFlags;
@@ -551,7 +523,7 @@ fn portal_thread(
.select_sources(
&session,
SelectSourcesOptions::default()
.set_cursor_mode(cursor_mode)
.set_cursor_mode(CursorMode::Embedded)
// xdph offers MONITOR; the custom picker selects our output.
.set_sources(BitFlags::from_flag(SourceType::Monitor))
.set_multiple(false)
+89 -257
View File
@@ -56,13 +56,8 @@ use zkde::zkde_screencast_stream_unstable_v1::{
};
use zkde::zkde_screencast_unstable_v1::ZkdeScreencastUnstableV1 as Screencast;
/// `pointer` attachment modes (the protocol enum), chosen per session by `set_hw_cursor`
/// (Phase B — the Windows no-regression gate mirrored): a CURSOR-CHANNEL session gets METADATA
/// (`SPA_META_Cursor` on the stream — shapes forwarded to the client, the composite flip blends
/// host-side; embedded would leave both with nothing, the round-1 mutter trap), every other
/// session gets EMBEDDED — KWin composites the pointer into frames itself, zero host-side
/// cursor work, the pre-channel path Moonlight/legacy clients always had.
const POINTER_METADATA: u32 = 4;
/// `pointer` attachment mode (the protocol enum): render the cursor into the stream so the
/// remote sees it move with injected input.
const POINTER_EMBEDDED: u32 = 2;
/// The name we give the created output; KWin exposes it to output-management as `Virtual-<name>`.
@@ -84,19 +79,14 @@ pub struct KwinDisplay {
/// `None` for shared/anonymous) — reported to the registry via [`last_identity_slot`] so it can key
/// the group arrangement + `/display/state` slot to the same id this backend named the output with.
last_slot: Option<u32>,
/// The RESOLVED kscreen address of the last `create`'s output — the numeric kscreen output id
/// when [`resolve_kscreen_addr`] found it, else the `Virtual-<name>` fallback — so
/// [`apply_position`](VirtualDisplay::apply_position) addresses OUR output even while a
/// superseded same-name sibling is still alive.
/// The base output name the last `create` used (`punktfunk` / `punktfunk-<id>`) — so
/// [`apply_position`](VirtualDisplay::apply_position) can address the KWin output `Virtual-<name>`.
last_name: Option<String>,
/// The topology-restore action the last `create` prepared (re-enable the outputs an `exclusive`
/// topology disabled), pending pickup by the registry via [`take_topology_restore`] — so the
/// physical is re-enabled only when the display GROUP's last member drops (§6.1), not this session's.
/// A backstop [`Drop`] runs it if the registry never took it (so a physical is never left dark).
pending_restore: Option<Box<dyn FnOnce() + Send>>,
/// Out-of-band cursor request (`set_hw_cursor`, i.e. the session negotiated the cursor
/// channel): METADATA pointer mode at creation; off = EMBEDDED (see the consts above).
hw_cursor: bool,
}
impl Drop for KwinDisplay {
@@ -133,22 +123,12 @@ impl VirtualDisplay for KwinDisplay {
self.pending_restore.take()
}
fn set_hw_cursor(&mut self, on: bool) {
self.hw_cursor = on;
}
fn hw_cursor(&self) -> bool {
self.hw_cursor
}
fn apply_position(&mut self, x: i32, y: i32) {
// `last_name` holds the RESOLVED kscreen address (numeric output id, or the
// `Virtual-<name>` fallback) — never re-derive from the name: during a supersede two
// outputs share it and the command would hit the old one (see `create`).
let Some(output) = self.last_name.clone() else {
let Some(name) = self.last_name.clone() else {
return;
};
// kscreen-doctor position syntax: `output.<name-or-id>.position.<x>,<y>`.
let output = format!("Virtual-{name}");
// kscreen-doctor position syntax: `output.<name>.position.<x>,<y>`.
let ok = std::process::Command::new("kscreen-doctor")
.arg(format!("output.{output}.position.{x},{y}"))
.status()
@@ -178,100 +158,32 @@ impl VirtualDisplay for KwinDisplay {
None => VOUT_NAME.to_string(),
};
self.last_name = Some(name.clone()); // for apply_position (registry-driven §6.2 layout)
let (setup_tx, setup_rx) = std::sync::mpsc::channel::<Result<u32, String>>();
let stop = Arc::new(AtomicBool::new(false));
let stop_thread = stop.clone();
let (width, height) = (mode.width, mode.height);
let pointer_mode = if self.hw_cursor {
POINTER_METADATA
} else {
POINTER_EMBEDDED
let name_thread = name.clone();
thread::Builder::new()
.name("punktfunk-kwin-vout".into())
.spawn(move || virtual_output_thread(width, height, name_thread, setup_tx, stop_thread))
.context("spawn KWin virtual-output thread")?;
let node_id = match setup_rx.recv_timeout(Duration::from_secs(20)) {
Ok(Ok(v)) => v,
Ok(Err(e)) => bail!("KWin virtual output failed: {e}"),
Err(_) => bail!("timed out creating the KWin virtual output"),
};
let spawn_vout = |w: u32, h: u32| -> Result<(u32, Arc<AtomicBool>)> {
let (setup_tx, setup_rx) = std::sync::mpsc::channel::<Result<u32, String>>();
let stop = Arc::new(AtomicBool::new(false));
let stop_thread = stop.clone();
let name_thread = name.clone();
thread::Builder::new()
.name("punktfunk-kwin-vout".into())
.spawn(move || {
virtual_output_thread(w, h, name_thread, pointer_mode, setup_tx, stop_thread)
})
.context("spawn KWin virtual-output thread")?;
match setup_rx.recv_timeout(Duration::from_secs(20)) {
Ok(Ok(v)) => Ok((v, stop)),
Ok(Err(e)) => bail!("KWin virtual output failed: {e}"),
Err(_) => bail!("timed out creating the KWin virtual output"),
}
};
// KWin creates virtual outputs at a hardcoded 60 Hz, `stream_virtual_output` has no
// refresh argument — and its screencast stream builds its PipeWire format offer, INCLUDING
// the `maxFramerate` cap it actively throttles delivery to, ONCE at stream creation
// (screencaststream.cpp `buildFormats`, verified against the live offer with `pw-dump`:
// the offer stays `max=60` after a kscreen mode change, and consumers connecting later
// still negotiate against it). The ONLY path that rebuilds the offer is the stream's own
// resize handling: when the source's texture size changes while recording, KWin re-runs
// `buildFormats` — picking up the output's CURRENT refresh — and renegotiates the live
// stream via `pw_stream_update_params`. So above 60 Hz the output is born at a
// SACRIFICIAL height: installing + selecting the real `WxH@hz` custom mode (supported on
// virtual outputs since KWin 6.6) then changes the SIZE, and the first buffers recorded
// after the consumer connects trigger KWin's resize → a renegotiation to `WxH@hz`. The
// capturer holds frames until that lands (`expect_exact_dims`), so the pipeline never
// builds against the birth mode. First cut shells out to kscreen-doctor; the in-process
// kde_output_management_v2 client is a follow-up. `set_custom_refresh` reads back what
// KWin *actually* gave so the encoder paces to the real source rate. At ≤60 Hz there's
// nothing to install — the output is born at the real size and 60 Hz is the offer anyway.
let want_high = mode.refresh_hz > 60;
let birth_h = if want_high { height + 16 } else { height };
let (mut node_id, mut stop) = spawn_vout(width, birth_h)?;
tracing::info!(
node_id,
width,
height,
birth_h,
embedded_pointer = !self.hw_cursor,
"KWin virtual output ready"
);
// ⚠️ ADDRESS BY NUMERIC KSCREEN ID, NEVER BY NAME: a supersede (mode switch) creates the
// replacement output — SAME per-slot name, deliberately, for KWin's per-name config
// persistence — while the superseded sibling is still alive (create-before-drop). Every
// name-addressed kscreen command then hits the FIRST match = the OLD output: on-glass this
// resized the LIVE session's display out from under it (wrong-res/black), read back the
// OLD output as "custom mode applied", made the OLD output primary, and positioned it —
// while the new output never left its birth mode and the capturer's dims gate starved.
// Resolve OUR output's kscreen id once (match: managed-prefix name AND current mode ==
// the just-created birth size; newest id wins), and use it for every kscreen operation.
let mut addr = resolve_kscreen_addr(&name, width, birth_h);
self.last_name = Some(addr.clone()); // for apply_position (registry-driven §6.2 layout)
let mut expect_exact_dims = false;
let achieved_hz = if want_high {
let (achieved, size_applied) =
set_custom_refresh(width, height, mode.refresh_hz, &addr);
if size_applied {
// Real mode selected: the recording stream will renegotiate to it (see above).
expect_exact_dims = true;
achieved
} else {
// Custom-mode install/select rejected (pre-6.6 KWin / stale kscreen-doctor): the
// output is STUCK at the sacrificial birth size — unusable. Recreate plain at the
// real size (the pre-sacrifice behavior: correct size, KWin's native 60 Hz).
tracing::warn!(
"KWin rejected the custom mode — recreating the virtual output at the real \
size (60 Hz ceiling on this KWin)"
);
stop.store(true, Ordering::Relaxed);
// Let KWin retire the doomed output before re-using its name.
std::thread::sleep(Duration::from_millis(300));
let (nid, st) = spawn_vout(width, height)?;
node_id = nid;
stop = st;
addr = resolve_kscreen_addr(&name, width, height);
self.last_name = Some(addr.clone());
tracing::info!(
node_id,
width,
height,
"KWin virtual output ready (fallback)"
);
60
}
tracing::info!(node_id, width, height, "KWin virtual output ready");
// KWin creates virtual outputs at a hardcoded 60 Hz and `stream_virtual_output` has no
// refresh argument, so above 60 Hz we install + select a custom mode (supported on virtual
// outputs since KWin 6.6) before capture connects PipeWire, so the stream negotiates at the
// higher rate. First cut shells out to kscreen-doctor; the in-process
// kde_output_management_v2 client is a follow-up. `set_custom_refresh` reads back and
// returns what KWin *actually* achieved so the encoder paces to the real source rate (a
// rejected custom mode leaves the output at 60 Hz). At ≤60 Hz there's nothing to install —
// the source runs 60 Hz and the encoder downsamples — so carry the requested rate through.
let achieved_hz = if mode.refresh_hz > 60 {
set_custom_refresh(width, height, mode.refresh_hz, &name)
} else {
mode.refresh_hz
};
@@ -282,9 +194,9 @@ impl VirtualDisplay for KwinDisplay {
// bootstrap output. Read from the policy (replacing the PUNKTFUNK_KWIN_VIRTUAL_PRIMARY boolean).
use crate::policy::Topology;
let disabled = match crate::effective_topology() {
Topology::Exclusive => apply_virtual_primary(&addr),
Topology::Exclusive => apply_virtual_primary(&name),
Topology::Primary => {
apply_virtual_primary_only(&addr);
apply_virtual_primary_only(&name);
Vec::new() // nothing disabled → nothing to restore
}
Topology::Extend | Topology::Auto => Vec::new(),
@@ -300,13 +212,11 @@ impl VirtualDisplay for KwinDisplay {
});
// Layout position (§6.2) is applied by the registry via `apply_position` right after create
// (it owns the display group, so it computes auto-row / manual placement over the whole group).
let mut out = VirtualOutput::owned(
Ok(VirtualOutput::owned(
node_id,
Some((mode.width, mode.height, achieved_hz)),
Box::new(StopGuard { stop }),
);
out.expect_exact_dims = expect_exact_dims;
Ok(out)
))
}
}
@@ -344,88 +254,14 @@ fn reenable_outputs(outputs: &[(String, String)]) {
tracing::info!(reenabled = ?outputs, "KWin: restored the physical/bootstrap outputs at their captured modes (group empty)");
}
/// Resolve the kscreen address of the virtual output the host JUST created: the managed-prefix
/// name alone is ambiguous during a supersede (the replacement deliberately reuses the per-slot
/// name while the superseded sibling is still alive), so match on the birth mode's size too —
/// only the just-created output sits at the sacrificial `(w, h)` — and prefer the HIGHEST output
/// id (the newest) if several match. Returns the numeric id as a string (kscreen-doctor accepts
/// `output.<id>.…`), falling back to the ambiguous `Virtual-<name>` if the output hasn't reached
/// kscreen's model yet after a few tries (single-output sessions are unambiguous anyway).
fn resolve_kscreen_addr(name: &str, w: u32, h: u32) -> String {
let fallback = format!("Virtual-{name}");
for attempt in 0..3 {
if attempt > 0 {
std::thread::sleep(Duration::from_millis(150));
}
let Some(doc) = kscreen_json() else { continue };
let Some(outputs) = doc.get("outputs").and_then(|o| o.as_array()) else {
continue;
};
let best = outputs
.iter()
.filter(|o| {
o.get("name")
.and_then(|n| n.as_str())
.is_some_and(|n| n.starts_with(&fallback))
&& output_active_size(o) == Some((w, h))
})
.filter_map(|o| o.get("id").and_then(|i| i.as_u64()))
.max();
if let Some(id) = best {
tracing::info!(id, name, w, h, "KWin: resolved the new output's kscreen id");
return id.to_string();
}
}
tracing::warn!(
name,
w,
h,
"KWin: could not resolve the new output's kscreen id — falling back to name addressing \
(ambiguous during a mode-switch supersede)"
);
fallback
}
/// `kscreen-doctor -j` parsed, `None` on any failure.
fn kscreen_json() -> Option<serde_json::Value> {
let out = std::process::Command::new("kscreen-doctor")
.arg("-j")
.output()
.ok()?;
serde_json::from_slice(&out.stdout).ok()
}
/// The `(width, height)` of an output's CURRENT mode from its `kscreen-doctor -j` entry.
fn output_active_size(o: &serde_json::Value) -> Option<(u32, u32)> {
let as_id = |v: &serde_json::Value| -> Option<String> {
v.as_str()
.map(|s| s.to_string())
.or_else(|| v.as_u64().map(|n| n.to_string()))
};
let current = o.get("currentModeId").and_then(as_id)?;
let mode = o
.get("modes")?
.as_array()?
.iter()
.find(|m| m.get("id").and_then(as_id).as_deref() == Some(current.as_str()))?;
let size = mode.get("size")?;
Some((
size.get("width").and_then(|v| v.as_u64())? as u32,
size.get("height").and_then(|v| v.as_u64())? as u32,
))
}
/// Best-effort: install + select the `width`x`height`@`hz` custom mode on the just-created
/// virtual output via `kscreen-doctor` (`output` is the RESOLVED kscreen address — numeric id or
/// name, see [`resolve_kscreen_addr`] — refresh given in mHz), then **read back the active mode**
/// and return `(achieved_hz, size_applied)`. The apply command can report success yet leave the
/// output on its old mode (rejected), and a silent rate mismatch surfaces downstream as judder /
/// duplicated frames — so the caller paces the encoder to the *achieved* rate, not the requested
/// one. `size_applied` tells the sacrificial-birth caller (see `create`) whether the SIZE half of
/// the mode actually landed — that, not the refresh, is what triggers KWin's stream
/// renegotiation.
fn set_custom_refresh(width: u32, height: u32, hz: u32, output: &str) -> (u32, bool) {
let output = output.to_string();
/// Best-effort: raise the just-created virtual output's refresh above KWin's default 60 Hz by
/// installing + selecting a custom mode via `kscreen-doctor` (the output is `Virtual-<VOUT_NAME>`,
/// refresh given in mHz), then **read back the active mode** and return the refresh KWin actually
/// gave us. The apply command can report success yet leave the output at 60 Hz (mode rejected),
/// and a silent rate mismatch surfaces downstream as judder / duplicated frames so the caller
/// paces the encoder to the *achieved* rate, not the requested one.
fn set_custom_refresh(width: u32, height: u32, hz: u32, name: &str) -> u32 {
let output = format!("Virtual-{name}");
let mhz = hz.saturating_mul(1000);
let run = |arg: String| {
std::process::Command::new("kscreen-doctor")
@@ -439,29 +275,26 @@ fn set_custom_refresh(width: u32, height: u32, hz: u32, output: &str) -> (u32, b
"output.{output}.addCustomMode.{width}.{height}.{mhz}.full"
));
let applied = run(format!("output.{output}.mode.{width}x{height}@{hz}"));
match read_active_mode(&output) {
Some((w, h, achieved)) => {
let size_applied = (w, h) == (width, height);
if achieved >= hz && size_applied {
tracing::info!(
output,
requested = hz,
achieved,
"KWin virtual output: custom refresh applied"
);
} else {
tracing::warn!(
output,
requested = hz,
achieved,
active_w = w,
active_h = h,
applied,
"KWin virtual output mode below requested — pacing the encoder to the \
achieved rate (custom-mode install rejected? is kscreen-doctor up to date?)"
);
}
(achieved.max(1), size_applied)
match read_active_refresh(&output) {
Some(achieved) if achieved >= hz => {
tracing::info!(
output,
requested = hz,
achieved,
"KWin virtual output: custom refresh applied"
);
achieved
}
Some(achieved) => {
tracing::warn!(
output,
requested = hz,
achieved,
applied,
"KWin virtual output refresh below requested — pacing the encoder to the achieved \
rate (custom-mode install rejected? is kscreen-doctor up to date?)"
);
achieved.max(1)
}
None => {
tracing::warn!(
@@ -471,37 +304,38 @@ fn set_custom_refresh(width: u32, height: u32, hz: u32, output: &str) -> (u32, b
"could not read back KWin virtual output refresh — assuming 60 Hz (is \
kscreen-doctor installed?)"
);
(60, false)
60
}
}
}
/// Read the active mode (`(width, height, refresh_hz)`, Hz rounded) of `output` — a RESOLVED
/// kscreen address (numeric id or name, see [`resolve_kscreen_addr`]) — from `kscreen-doctor -j`.
/// `None` if the tool, the output, or its current mode can't be found. Mode/output ids come
/// through as either JSON strings or numbers depending on the KWin version, so both are accepted.
fn read_active_mode(output: &str) -> Option<(u32, u32, u32)> {
let doc = kscreen_json()?;
/// Read the active refresh (Hz, rounded) of `output` from `kscreen-doctor -j`. `None` if the
/// tool, the output, or its current mode can't be found. Mode/output ids come through as either
/// JSON strings or numbers depending on the KWin version, so both are accepted.
fn read_active_refresh(output: &str) -> Option<u32> {
let out = std::process::Command::new("kscreen-doctor")
.arg("-j")
.output()
.ok()?;
let doc: serde_json::Value = serde_json::from_slice(&out.stdout).ok()?;
let as_id = |v: &serde_json::Value| -> Option<String> {
v.as_str()
.map(|s| s.to_string())
.or_else(|| v.as_u64().map(|n| n.to_string()))
};
let o = doc.get("outputs")?.as_array()?.iter().find(|o| {
o.get("name").and_then(|n| n.as_str()) == Some(output)
|| o.get("id").and_then(as_id).as_deref() == Some(output)
})?;
let o = doc
.get("outputs")?
.as_array()?
.iter()
.find(|o| o.get("name").and_then(|n| n.as_str()) == Some(output))?;
let current = o.get("currentModeId").and_then(as_id)?;
let mode = o
.get("modes")?
.as_array()?
.iter()
.find(|m| m.get("id").and_then(as_id).as_deref() == Some(current.as_str()))?;
let size = mode.get("size")?;
let w = size.get("width").and_then(|v| v.as_u64())? as u32;
let h = size.get("height").and_then(|v| v.as_u64())? as u32;
let hz = mode.get("refreshRate").and_then(|r| r.as_f64())?;
Some((w, h, hz.round() as u32))
Some(hz.round() as u32)
}
/// The prefix EVERY managed KWin output shares — Stage 3 names them `punktfunk` / `punktfunk-<id>`,
@@ -540,7 +374,7 @@ fn output_current_mode_spec(o: &serde_json::Value) -> Option<String> {
/// bare re-enable drops a 120 Hz panel to KWin's default ~60 Hz).
/// **Group-aware (§6.1):** excludes the WHOLE managed family (the [`MANAGED_PREFIX`]), not just this
/// session's own output — so a 2nd `exclusive` session (with a distinct per-slot name) never disables
/// the 1st session's live output. Parsed from `kscreen-doctor -j` (same source as [`read_active_mode`]).
/// the 1st session's live output. Parsed from `kscreen-doctor -j` (same source as [`read_active_refresh`]).
fn other_enabled_outputs() -> Vec<(String, String)> {
let out = match std::process::Command::new("kscreen-doctor")
.arg("-j")
@@ -603,12 +437,12 @@ fn a_managed_output_is_primary() -> bool {
.unwrap_or(false)
}
/// Set our output primary and disable the bootstrap output(s) so the managed group becomes
/// the sole desktop (KWin re-homes plasmashell + windows onto it). `ours` is the RESOLVED kscreen
/// address (numeric id or name, see [`resolve_kscreen_addr`]). Returns the disabled outputs for
/// Set `Virtual-punktfunk` primary and disable the bootstrap output(s) so the managed group becomes
/// the sole desktop (KWin re-homes plasmashell + windows onto it). Returns the disabled outputs for
/// the keepalive to re-enable on teardown. Best-effort: on failure, streaming continues (just possibly
/// showing only the wallpaper) rather than failing the session.
fn apply_virtual_primary(ours: &str) -> Vec<(String, String)> {
fn apply_virtual_primary(name: &str) -> Vec<(String, String)> {
let ours = format!("Virtual-{name}");
let kscreen = |args: &[String]| {
std::process::Command::new("kscreen-doctor")
.args(args)
@@ -646,7 +480,8 @@ fn apply_virtual_primary(ours: &str) -> Vec<(String, String)> {
/// **Primary** (Stage 2): make the streamed output the primary but KEEP the other outputs enabled
/// (don't disable the bootstrap/physical) — so the shell re-homes onto the streamed surface while a
/// physical screen stays usable. Nothing to restore on teardown (we disabled nothing).
fn apply_virtual_primary_only(ours: &str) {
fn apply_virtual_primary_only(name: &str) {
let ours = format!("Virtual-{name}");
let ok = std::process::Command::new("kscreen-doctor")
.arg(format!("output.{ours}.primary"))
.status()
@@ -743,11 +578,10 @@ fn virtual_output_thread(
width: u32,
height: u32,
name: String,
pointer_mode: u32,
setup_tx: Sender<Result<u32, String>>,
stop: Arc<AtomicBool>,
) {
if let Err(e) = run(width, height, &name, pointer_mode, &setup_tx, &stop) {
if let Err(e) = run(width, height, &name, &setup_tx, &stop) {
// If we never delivered a node id, report the failure to the waiting opener.
let _ = setup_tx.send(Err(format!("{e:#}")));
}
@@ -788,7 +622,6 @@ fn run(
width: u32,
height: u32,
name: &str,
pointer_mode: u32,
setup_tx: &Sender<Result<u32, String>>,
stop: &AtomicBool,
) -> Result<()> {
@@ -809,14 +642,13 @@ fn run(
)
})?;
// Create the virtual output sized to the client; the pointer rides as stream metadata
// (cursor-channel session) or KWin embeds it into frames (everyone else — see the consts).
// Create the virtual output sized to the client, cursor composited into the stream.
let stream = screencast.stream_virtual_output(
name.to_string(),
width as i32,
height as i32,
1.0, // scale (logical == physical)
pointer_mode,
POINTER_EMBEDDED,
&qh,
(),
);
@@ -46,17 +46,7 @@ const BUS_DC: &str = "org.gnome.Mutter.DisplayConfig";
/// e.g. when our virtual output is torn down — so we never persist a layout to monitors.xml).
const APPLY_TEMPORARY: u32 = 1;
/// Mutter cursor mode: ship the pointer as `SPA_META_Cursor` metadata instead of burning it into
/// the frames. The capturer always negotiates the meta (pf-capture `meta_param`) and the encoder
/// blend composites it for sessions where the client does not draw the cursor itself — while a
/// cursor-forwarding session strips the overlay and sends shape/state over the cursor channel.
/// Embedded mode would leave BOTH paths blind: no metadata means nothing to forward AND nothing
/// to blend, and Mutter's own embedded painting is what the pre-channel path relied on.
const CURSOR_METADATA: u32 = 2;
/// `cursor-mode` embedded (mutter enum 1): Mutter composites the pointer into frames itself —
/// zero host-side cursor work, the pre-channel path. Chosen for every session WITHOUT the
/// negotiated cursor channel (`set_hw_cursor` off — Phase B, the Windows no-regression gate
/// mirrored); metadata stays the cursor-channel sessions' mode (shapes forwarded / host blend).
/// Mutter cursor mode: render the cursor into the stream (matches the KWin/gamescope backends).
const CURSOR_EMBEDDED: u32 = 1;
/// Serializes, process-wide, every Mutter operation that adds/removes a virtual monitor or applies
@@ -79,9 +69,6 @@ pub struct MutterDisplay {
/// sole-monitor config (which would disable the first session's virtual). Defaults true (a lone
/// session establishes topology as before).
first_in_group: bool,
/// Out-of-band cursor request (`set_hw_cursor`): metadata cursor-mode at creation; off =
/// embedded (see [`CURSOR_EMBEDDED`]).
hw_cursor: bool,
/// The connecting client's cert fingerprint (set before [`create`](VirtualDisplay::create)) —
/// keys the per-client persisted **scale** (GNOME can't persist it itself: Mutter mints a fresh
/// EDID serial per `RecordVirtual` monitor, so `monitors.xml` never rematches; see
@@ -97,7 +84,6 @@ impl MutterDisplay {
pub fn new() -> Result<Self> {
Ok(MutterDisplay {
first_in_group: true,
hw_cursor: false,
client_fp: None,
last_slot: None,
})
@@ -126,14 +112,6 @@ impl VirtualDisplay for MutterDisplay {
self.client_fp = fingerprint;
}
fn set_hw_cursor(&mut self, on: bool) {
self.hw_cursor = on;
}
fn hw_cursor(&self) -> bool {
self.hw_cursor
}
fn last_identity_slot(&self) -> Option<u32> {
self.last_slot
}
@@ -163,7 +141,6 @@ impl VirtualDisplay for MutterDisplay {
let stop = Arc::new(AtomicBool::new(false));
let stop_thread = stop.clone();
let first_in_group = self.first_in_group;
let hw_cursor = self.hw_cursor;
thread::Builder::new()
.name("punktfunk-mutter-vout".into())
.spawn(move || {
@@ -172,7 +149,6 @@ impl VirtualDisplay for MutterDisplay {
stop_thread,
mode,
first_in_group,
hw_cursor,
scale_key,
remembered_scale,
)
@@ -227,7 +203,6 @@ fn session_thread(
stop: Arc<AtomicBool>,
mode: Mode,
first_in_group: bool,
hw_cursor: bool,
scale_key: String,
remembered_scale: Option<f64>,
) {
@@ -287,7 +262,7 @@ fn session_thread(
}
};
let session = match connect(mode, hw_cursor, remembered_scale).await {
let session = match connect(mode, remembered_scale).await {
Ok(s) => s,
Err(e) => {
let _ = setup_tx.send(Err(format!("{e:#}")));
@@ -388,11 +363,7 @@ struct MutterSession {
/// desktop scale, passed as the virtual mode's `preferred-scale` so Mutter creates the monitor
/// already scaled (Mutter ≥ 48; older Mutter ignores unknown mode keys) — this covers the
/// `extend` topology, where we never issue our own ApplyMonitorsConfig.
async fn connect(
mode: Mode,
hw_cursor: bool,
preferred_scale: Option<f64>,
) -> Result<MutterSession> {
async fn connect(mode: Mode, preferred_scale: Option<f64>) -> Result<MutterSession> {
let conn = zbus::Connection::session()
.await
.context("connect session D-Bus")?;
@@ -453,14 +424,7 @@ async fn connect(
// once gated behind PUNKTFUNK_MUTTER_VIRTUAL_REFRESH; the stop-screencast-before-any-monitor-
// reconfig teardown below fixed the crash, so pinning the client's refresh is now the default.)
let mut rec: HashMap<&str, Value> = HashMap::new();
rec.insert(
"cursor-mode",
Value::from(if hw_cursor {
CURSOR_METADATA
} else {
CURSOR_EMBEDDED
}),
);
rec.insert("cursor-mode", Value::from(CURSOR_EMBEDDED));
if mode.refresh_hz > 60 || preferred_scale.is_some() {
let mut vmode: HashMap<&str, Value> = HashMap::new();
vmode.insert("size", Value::from((mode.width, mode.height)));
@@ -54,19 +54,11 @@ chooser_cmd=cat {} 2>/dev/null || echo 'Monitor: HEADLESS-1'\n",
/// The wlroots/Sway virtual-display driver. Stateless — each [`create`](VirtualDisplay::create)
/// adds one headless output and spins up a portal thread owning the cast on it.
pub struct WlrootsDisplay {
/// Out-of-band cursor request (`set_hw_cursor`, the negotiated cursor channel): portal
/// `CursorMode::Metadata` — shapes/positions ride `SPA_META_Cursor` for the channel + the
/// composite blend. Off (every non-channel session): `Embedded` — the compositor paints the
/// pointer into frames, zero host-side cursor work (the pre-channel default this backend
/// always had). ⚠️ Metadata is UNTESTED on-glass for this backend (Phase B wired it so the
/// channel isn't silently dead here; KWin/Mutter are the validated legs).
hw_cursor: bool,
}
pub struct WlrootsDisplay;
impl WlrootsDisplay {
pub fn new() -> Result<Self> {
Ok(WlrootsDisplay { hw_cursor: false })
Ok(WlrootsDisplay)
}
}
@@ -81,14 +73,6 @@ impl VirtualDisplay for WlrootsDisplay {
"wlroots"
}
fn set_hw_cursor(&mut self, on: bool) {
self.hw_cursor = on;
}
fn hw_cursor(&self) -> bool {
self.hw_cursor
}
fn create(&mut self, mode: Mode) -> Result<VirtualOutput> {
let before = output_names()
.context("swaymsg get_outputs (is the host inside the sway session env — SWAYSOCK?)")?;
@@ -120,10 +104,9 @@ impl VirtualDisplay for WlrootsDisplay {
let (setup_tx, setup_rx) = std::sync::mpsc::channel::<Result<(OwnedFd, u32), String>>();
let stop = Arc::new(AtomicBool::new(false));
let stop_thread = stop.clone();
let hw_cursor = self.hw_cursor;
thread::Builder::new()
.name("punktfunk-wlr-vout".into())
.spawn(move || portal_thread(setup_tx, stop_thread, hw_cursor))
.spawn(move || portal_thread(setup_tx, stop_thread))
.context("spawn wlroots portal thread")?;
let (fd, node_id) = match setup_rx.recv_timeout(Duration::from_secs(20)) {
@@ -153,7 +136,6 @@ impl VirtualDisplay for WlrootsDisplay {
ownership: DisplayOwnership::Owned,
reused_gen: None,
pool_gen: None,
expect_exact_dims: false,
})
}
}
@@ -272,17 +254,7 @@ fn ensure_xdpw_config() -> Result<()> {
/// The ScreenCast portal handshake (same shape as the capture module's portal thread, but it
/// reports the fd + node id and parks until stopped — the zbus connection is the cast's
/// lifetime). xdpw answers the source selection via the chooser, no dialog.
fn portal_thread(
setup_tx: Sender<Result<(OwnedFd, u32), String>>,
stop: Arc<AtomicBool>,
hw_cursor: bool,
) {
// Portal cursor mode per the session's channel negotiation (see the struct doc).
let cursor_mode = if hw_cursor {
CursorMode::Metadata
} else {
CursorMode::Embedded
};
fn portal_thread(setup_tx: Sender<Result<(OwnedFd, u32), String>>, stop: Arc<AtomicBool>) {
use ashpd::desktop::screencast::{CursorMode, Screencast, SelectSourcesOptions, SourceType};
use ashpd::desktop::PersistMode;
use ashpd::enumflags2::BitFlags;
@@ -315,7 +287,7 @@ fn portal_thread(
.select_sources(
&session,
SelectSourcesOptions::default()
.set_cursor_mode(cursor_mode)
.set_cursor_mode(CursorMode::Embedded)
// xdpw offers MONITOR only; the chooser picks our output.
.set_sources(BitFlags::from_flag(SourceType::Monitor))
.set_multiple(false)
+5 -39
View File
@@ -84,28 +84,20 @@ fn topology_str() -> String {
/// `quit` is the session's deliberate-quit flag: when the session ends with it set (the client closed
/// with the quit application code — a user "stop", not a network drop), the display is torn down
/// **immediately**, skipping the keep-alive linger. A bare disconnect leaves it `false` → normal linger.
///
/// `supersedes`: the pool gen of a display this acquire REPLACES (a mid-stream mode switch creates
/// the new display before retiring the old — create-before-drop). The replacement inherits group
/// topology ownership: without this, the dying predecessor counts as a live sibling and the new
/// display "extends" behind it, losing a Primary/Exclusive topology on every resize. `None`
/// everywhere else.
pub fn acquire(
vd: &mut Box<dyn super::VirtualDisplay>,
mode: super::Mode,
quit: std::sync::Arc<std::sync::atomic::AtomicBool>,
supersedes: Option<u64>,
) -> Result<super::VirtualOutput> {
let backend = vd.name();
#[cfg(target_os = "linux")]
let out = linux::acquire(vd, mode, quit, supersedes);
let out = linux::acquire(vd, mode, quit);
#[cfg(not(target_os = "linux"))]
let out = {
// Windows leases in the manager (its own linger); its deliberate-quit skip is wired through
// `VirtualDisplay::set_quit_flag` on the backend instance (set by the session before any
// `create`, so the retry-hold lease gets it too) — not through this parameter. The
// supersede handoff is Linux-pool-only too (the manager resizes in place).
let _ = (quit, supersedes);
// `create`, so the retry-hold lease gets it too) — not through this parameter.
let _ = quit;
vd.create(mode)
};
if out.is_ok() {
@@ -268,11 +260,6 @@ mod linux {
/// Generation stamp: a [`DisplayLease`] only releases if its gen still matches (a stale lease
/// — its entry was reused + re-stamped — is a no-op).
gen: u64,
/// The out-of-band-cursor mode this display was CREATED with (Phase B): metadata-pointer
/// (cursor-channel session) vs compositor-embedded. Reuse requires an exact match — a kept
/// embedded display has no cursor metadata for a channel session to forward, and a kept
/// metadata display would leave a channel-less session with no pointer in its frames.
hw_cursor: bool,
}
/// A per-group topology-restore action (see [`Entry::topology_restore`]).
@@ -434,7 +421,6 @@ mod linux {
vd: &mut Box<dyn VirtualDisplay>,
mode: Mode,
quit: Arc<AtomicBool>,
supersedes: Option<u64>,
) -> Result<VirtualOutput> {
ensure_timer();
let backend = vd.name();
@@ -478,7 +464,6 @@ mod linux {
) && e.backend == backend
&& e.mode == mode
&& e.launch == launch
&& e.hw_cursor == vd.hw_cursor()
&& epoch_matches(e.backend, e.epoch, cur_epoch)
})
.map(|e| (e.gen, e.node_id))
@@ -550,19 +535,9 @@ mod linux {
// §6.1) — so a topology-establishing backend (Mutter exclusive) extends into an already-exclusive
// desktop rather than re-clobbering the first session's virtual. Best-effort (a concurrent create
// is a narrow race); single-session is always `first == true` → today's behavior.
// Siblings that don't demote the newcomer:
// * the display this acquire SUPERSEDES (mode switch, create-before-drop) — still Active
// here because the old lease drops only after the new pipeline is up, but it's leaving,
// and deferring to it loses the group's Primary/Exclusive topology on every resize;
// * kept (Lingering/Pinned) entries — no session owns them, so there is no live desktop to
// clobber; a new session next to an unclaimed leftover should still establish topology.
let first_in_group = {
let es = r.entries.lock().unwrap();
!es.iter().any(|e| {
e.backend == backend
&& Some(e.gen) != supersedes
&& matches!(e.life, lifecycle::State::Active { .. })
})
!es.iter().any(|e| e.backend == backend)
};
vd.set_first_in_group(first_in_group);
@@ -590,11 +565,6 @@ mod linux {
let node_id = real.node_id;
let preferred_mode = real.preferred_mode;
// Fresh creates only: the backend may have birthed the output at a sacrificial mode whose
// stream must renegotiate before frames count (KWin >60 Hz — see backend.rs). A REUSED kept
// display already renegotiated in its prior session (the producer's rebuilt offer persists
// across consumer reconnects), so the reuse path above correctly leaves the flag off.
let expect_exact_dims = real.expect_exact_dims;
// The backend's topology-restore action (KWin `exclusive` → re-enable the disabled physicals),
// lifted into the group so it runs once when the group's last member drops (§6.1), not at this
// session's teardown. `None` for non-exclusive / non-first / backends whose topology auto-reverts.
@@ -614,7 +584,6 @@ mod linux {
launch: launch.clone(),
epoch: cur_epoch,
gen,
hw_cursor: vd.hw_cursor(),
};
// Compute this new display's position in its group (design §6.2) BEFORE pushing, then push
@@ -659,9 +628,7 @@ mod linux {
if (position.x, position.y) != (0, 0) {
vd.apply_position(position.x, position.y);
}
let mut out = output_for(node_id, preferred_mode, gen, quit, false);
out.expect_exact_dims = expect_exact_dims;
Ok(out)
Ok(output_for(node_id, preferred_mode, gen, quit, false))
}
/// The linger a releasing session actually gets. A deliberate quit (`force_immediate` — the
@@ -1036,7 +1003,6 @@ mod linux {
launch: None,
epoch: 0,
gen,
hw_cursor: false,
}
}
@@ -162,15 +162,6 @@ pub fn launch_into_gamescope_session(cmd: &str) -> Result<std::process::Child> {
gamescope::launch_into_session(cmd)
}
/// Every nested Xwayland `(DISPLAY, XAUTHORITY)` of the running gamescope session for the XFixes
/// cursor source (remote-desktop-sweep Phase C) — gamescope can run several, and the pointer is on
/// whichever is focused. Empty when no gamescope session is running / it exposes no Xwayland (the
/// host then leaves gamescope cursorless, today's behaviour).
#[cfg(target_os = "linux")]
pub fn gamescope_xwayland_cursor_targets() -> Vec<(String, Option<String>)> {
gamescope::xwayland_cursor_targets()
}
/// B2: has a **dedicated** gamescope game session's game exited (its `node_id` doesn't reappear within a
/// short window after capture loss)? The dedicated-spawn session ends cleanly on `true` instead of the
/// capture-loss rebuild. Scoped to the session's OWN node so a coexisting gamescope doesn't mask the
@@ -216,20 +207,6 @@ pub fn cancel_pending_tv_restore() {
#[cfg(not(target_os = "linux"))]
pub fn cancel_pending_tv_restore() {}
/// Can the MANAGED gamescope path stand a session up from nothing on this box (SteamOS's
/// `gamescope-session` launcher or Bazzite's `gamescope-session-plus` present)? Lets the connect
/// path route a "no live graphical session" box to the gamescope takeover — which rebuilds the
/// session at the client's mode — instead of failing the connect. Always `false` off Linux.
#[cfg(target_os = "linux")]
pub fn managed_session_available() -> bool {
gamescope::managed_session_available()
}
#[cfg(not(target_os = "linux"))]
pub fn managed_session_available() -> bool {
false
}
/// Call when a client session ends: if the host-managed gamescope path took over a box's autologin
/// gaming session (stopped its single-instance Steam to stream at the client's mode), **schedule** a
/// debounced restore so the TV returns to gaming mode — unless a client reconnects within the window
@@ -71,15 +71,6 @@ struct Monitor {
/// selectable). The pin is never re-issued on reuse, so this is what the driver still renders
/// on — [`warn_if_pick_moved`] compares the CURRENT pick against it.
render_pin: Option<LUID>,
/// This monitor was ADDed with the v5 hardware-cursor flag (already driver-proto-gated) —
/// preserved across a re-arrival resize so the recreated monitor keeps the cursor channel.
hw_cursor: bool,
/// The ADD reply flagged this monitor's OS target as carrying an IRREVOCABLE hardware-cursor
/// declare from an earlier session (§8.6): DWM excludes the pointer from its frames forever
/// (the sticky state survives monitor REMOVE→ADD via the stable per-client target id), so a
/// session WITHOUT the cursor channel must self-composite — carried into
/// [`WinCaptureTarget`] for the IDD-push capturer's forced-composite gate.
cursor_excluded: bool,
/// The driver's WUDFHost pid (from the ADD reply) — carried into [`WinCaptureTarget`] so the
/// IDD-push capturer knows where to duplicate the sealed frame channel's handles. The SAME
/// process for every parallel monitor (one devnode → one WUDFHost hosts all publishers), which
@@ -107,7 +98,6 @@ impl Monitor {
gdi_name: n,
target_id: self.target_id,
wudf_pid: self.wudf_pid,
cursor_excluded: self.cursor_excluded,
})
}
}
@@ -277,44 +267,10 @@ pub fn vdm() -> &'static VirtualDisplayManager {
/// for the process lifetime — a dead one is RETIRED (kept alive, see [`DeviceSlot`]), so a stale copy
/// can only fail IOCTLs, never dangle. `None` before the first backend open — impossible for a
/// capturer, which only exists on a monitor the manager created.
/// Can this host's pf-vdisplay driver run the v5 hardware-cursor channel? Reads the
/// handshake-latched protocol version, opening the control device once if no session has
/// opened it yet this service run (the same open every session performs anyway) — so the
/// Welcome-time capability decision never guesses. `false` when the driver is missing/stale.
///
/// The FIRST session's Welcome precedes any backend construction (`vdisplay::open` runs at
/// display prep, after the Welcome), so this must not assume an initialised manager —
/// `init` is idempotent and constructing the driver facade is free (on-glass finding: the
/// `vdm()` expect panicked the very first handshake of a fresh service).
pub fn hw_cursor_capable() -> bool {
let m = init(Box::new(crate::driver::PfVdisplayDriver));
let v = m.driver_proto.load(Ordering::Relaxed);
if v != 0 {
return v >= 5;
}
let _ = m.ensure_device();
m.driver_proto.load(Ordering::Relaxed) >= 5
}
pub fn control_device_handle() -> Option<HANDLE> {
VDM.get().and_then(VirtualDisplayManager::device_handle)
}
/// Re-commit the CURRENT display config under the manager `state` lock (the sole-topology-mutator
/// contract of [`force_mode_reenumeration`]). The secure-desktop guard's actuator: the OS only
/// reverts a path to its software-cursor default ON a mode commit, so standing the hardware-cursor
/// declare down (`IOCTL_SET_CURSOR_FORWARD` off) needs this nudge for UAC/Winlogon to actually
/// render. `false` (no-op) before the first backend open — no monitors exist to re-commit for.
pub fn force_recommit() -> bool {
let Some(m) = VDM.get() else {
return false;
};
let _guard = m.state.lock().unwrap();
// SAFETY: `force_mode_reenumeration`'s contract is "call under the manager `state` lock";
// held above. The call reads + re-applies the current CCD config over owned locals.
unsafe { pf_win_display::win_display::force_mode_reenumeration() }
}
/// Best-effort "is this WUDFHost pid still alive?" — the monitor-liveness probe for the JOIN path.
/// `OpenProcess` failing (pid reaped) or the process being signaled ⇒ dead. Pid reuse could
/// theoretically alias a fresh process and read "alive"; the joining session then just retries into
@@ -434,7 +390,6 @@ impl VirtualDisplayManager {
mode: Mode,
client_fp: Option<[u8; 32]>,
client_hdr: Option<punktfunk_core::quic::HdrMeta>,
hw_cursor: bool,
quit: Option<Arc<AtomicBool>>,
) -> Result<VirtualOutput> {
// Console-session guard: a host outside the ACTIVE console session cannot drive the display
@@ -667,9 +622,7 @@ impl VirtualDisplayManager {
// SAFETY: `create_monitor` requires `dev` to be a valid control handle; `dev` is the handle
// `ensure_device()` returned above (cached handles are never closed — a dead one is retired,
// kept alive; see `DeviceSlot`), and we hold the `state` lock.
let mon = match unsafe {
self.create_monitor(dev, mode, slot, client_hdr, hw_cursor, &mut inner)
} {
let mon = match unsafe { self.create_monitor(dev, mode, slot, client_hdr, &mut inner) } {
// The cached device died under us (driver upgrade / WUDFHost restart, detected only
// now — e.g. the host sat idle past the pinger-less window). Retire it, reopen, and
// retry ONCE so the reconnect-after-driver-restart succeeds first try instead of
@@ -682,7 +635,7 @@ impl VirtualDisplayManager {
);
// SAFETY: as above — `dev` is the handle the reopening `ensure_device` just
// returned, and the `state` lock is still held.
unsafe { self.create_monitor(dev, mode, slot, client_hdr, hw_cursor, &mut inner)? }
unsafe { self.create_monitor(dev, mode, slot, client_hdr, &mut inner)? }
}
r => r?,
};
@@ -908,7 +861,6 @@ impl VirtualDisplayManager {
mode: Mode,
slot: u32,
client_hdr: Option<punktfunk_core::quic::HdrMeta>,
hw_cursor: bool,
inner: &mut MgrInner,
) -> Result<Monitor> {
// The slot id doubles as the driver-preferred monitor id (EDID serial / ConnectorIndex), so
@@ -916,17 +868,13 @@ impl VirtualDisplayManager {
// `0` (anonymous) = the driver auto-allocates the lowest-free id.
let preferred_id = slot;
let render_pin = resolve_render_pin();
// Hardware cursor only against a driver that implements the v5 channel: an older driver
// ignores the AddRequest field anyway (composited cursor), but gating here keeps the
// capture layer from creating + delivering a section nobody will ever publish into.
let hw_cursor = hw_cursor && self.driver_proto.load(Ordering::Relaxed) >= 5;
// SAFETY: `create_monitor`'s own `# Safety` contract guarantees `dev` is the live control
// handle; we forward it unchanged to `add_monitor`, whose precondition is exactly that.
// `render_pin` is an `Option<LUID>` by value (plain `Copy`), so no borrowed memory
// crosses the call.
let added = unsafe {
self.driver
.add_monitor(dev, mode, render_pin, preferred_id, client_hdr, hw_cursor)?
.add_monitor(dev, mode, render_pin, preferred_id, client_hdr)?
};
// Mandatory keepalive: ping inside the watchdog window or the driver tears all displays down.
@@ -1113,8 +1061,6 @@ impl VirtualDisplayManager {
resolved_monitor_id: added.resolved_monitor_id,
position: (0, 0),
gen: self.gen.fetch_add(1, Ordering::Relaxed),
hw_cursor,
cursor_excluded: added.cursor_excluded,
})
}
@@ -1285,7 +1231,7 @@ impl VirtualDisplayManager {
// values passed by value — no borrow crosses the call.
let added = unsafe {
self.driver
.add_monitor(dev, mode, render_pin, slot, client_hdr, old.hw_cursor)
.add_monitor(dev, mode, render_pin, slot, client_hdr)
.context("re-arrival ADD at the new mode")?
};
self.ensure_pinger();
@@ -1337,10 +1283,6 @@ impl VirtualDisplayManager {
resolved_monitor_id: added.resolved_monitor_id,
position: old.position,
gen: old.gen,
hw_cursor: old.hw_cursor,
// Fresh from THIS reply, not `old`: the driver's per-target declare registry is the
// ground truth (this session may itself have declared since the original ADD).
cursor_excluded: added.cursor_excluded,
})
}
@@ -22,11 +22,6 @@ pub(crate) struct AddedMonitor {
pub luid: LUID,
pub wudf_pid: u32,
pub resolved_monitor_id: u32,
/// The driver reports the OS target already carries an IRREVOCABLE hardware-cursor declare
/// from an earlier session (`AddReply::cursor_excluded`, remote-desktop-sweep §8.6): DWM
/// excludes the pointer from this target's frames forever, so a session without the cursor
/// channel must self-composite (GDI poller + blend) or stream a cursor-less desktop.
pub cursor_excluded: bool,
}
/// The backend-specific IOCTL surface — the *only* thing that differs between SudoVDA and pf-vdisplay.
@@ -61,7 +56,6 @@ pub(crate) trait VdisplayDriver: Send + Sync {
render_luid: Option<LUID>,
preferred_monitor_id: u32,
client_hdr: Option<punktfunk_core::quic::HdrMeta>,
hw_cursor: bool,
) -> Result<AddedMonitor>;
/// Refresh the LIVE monitor `key`'s advertised mode list to lead with `mode` (the in-place
/// mid-stream resize, latency plan P2 — pf-vdisplay `IOCTL_UPDATE_MODES`, driver protocol v4).
@@ -234,55 +234,6 @@ pub unsafe fn send_frame_channel(dev: HANDLE, req: &control::SetFrameChannelRequ
.context("pf-vdisplay SET_FRAME_CHANNEL")
}
/// Deliver a monitor's hardware-cursor section (`IOCTL_SET_CURSOR_CHANNEL`, proto v5) — the
/// cursor sibling of [`send_frame_channel`], same delivery/ownership contract.
///
/// # Safety
/// `dev` must be a live pf-vdisplay control handle (see [`super::manager::control_device_handle`]).
pub unsafe fn send_cursor_channel(
dev: HANDLE,
req: &control::SetCursorChannelRequest,
) -> Result<()> {
let mut none: [u8; 0] = [];
// SAFETY: per this fn's contract `dev` is the live control handle; `bytes_of(req)` borrows the
// caller's request across this synchronous call; no output buffer.
unsafe {
ioctl(
dev,
control::IOCTL_SET_CURSOR_CHANNEL,
bytemuck::bytes_of(req),
&mut none,
)
}
.map(|_| ())
.context("pf-vdisplay SET_CURSOR_CHANNEL")
}
/// Flip a LIVE monitor's hardware-cursor declaration (`IOCTL_SET_CURSOR_FORWARD`, proto v6) —
/// the mid-stream cursor-render flip. Fails against a pre-v6 driver (unknown IOCTL); callers
/// log and keep the declared-at-ADD behavior.
///
/// # Safety
/// `dev` must be a live pf-vdisplay control handle (see [`super::manager::control_device_handle`]).
pub unsafe fn send_cursor_forward(
dev: HANDLE,
req: &control::SetCursorForwardRequest,
) -> Result<()> {
let mut none: [u8; 0] = [];
// SAFETY: per this fn's contract `dev` is the live control handle; `bytes_of(req)` borrows the
// caller's request across this synchronous call; no output buffer.
unsafe {
ioctl(
dev,
control::IOCTL_SET_CURSOR_FORWARD,
bytemuck::bytes_of(req),
&mut none,
)
}
.map(|_| ())
.context("pf-vdisplay SET_CURSOR_FORWARD")
}
/// RAII over a SetupAPI device-info list: every exit path of [`open_device`] destroys it (the error
/// paths used to leak one `HDEVINFO` per failed open — and a driverless / mid-upgrade box probes
/// repeatedly).
@@ -509,7 +460,6 @@ impl VdisplayDriver for PfVdisplayDriver {
render_luid: Option<LUID>,
preferred_monitor_id: u32,
client_hdr: Option<punktfunk_core::quic::HdrMeta>,
hw_cursor: bool,
) -> Result<AddedMonitor> {
let session_id = next_session_id();
// The client display's volume rides into the monitor's EDID CTA HDR block; all-zero =
@@ -535,11 +485,7 @@ impl VdisplayDriver for PfVdisplayDriver {
max_luminance_nits,
max_frame_avg_nits,
min_luminance_millinits,
// v5 cursor channel: the driver declares an IddCx hardware cursor for this monitor
// (DWM stops compositing the pointer into the frame); the capture layer delivers the
// CursorShm section right after its ring. Zero toward older drivers is harmless —
// the host only sets this when the handshake-reported proto is >= 5.
hw_cursor: hw_cursor as u32,
_reserved: 0,
};
// SET_RENDER_ADAPTER (opt-in; pf-vdisplay IMPLEMENTS it). Non-fatal on failure: the driver reports
// its real render LUID in the shared header, so the host binds correctly even if this is ignored.
@@ -605,13 +551,10 @@ impl VdisplayDriver for PfVdisplayDriver {
})?;
// Fail closed on a short reply — `target_id`/`wudf_pid`/`luid` below feed OpenProcess + the
// WUDFHost verification, so don't decode a partially-written (zeroed) reply as authoritative.
// The LEGACY size, not the full struct: an un-upgraded driver writes only the prefix before
// the `cursor_excluded` tail; `out` is zero-initialized, so the missing tail reads `0`
// (= unknown/clean — exactly what a driver that can't track declares should report).
if (n as usize) < control::ADD_REPLY_LEGACY_SIZE {
if (n as usize) < size_of::<control::AddReply>() {
anyhow::bail!(
"pf-vdisplay ADD returned {n} bytes, expected at least {}",
control::ADD_REPLY_LEGACY_SIZE
"pf-vdisplay ADD returned {n} bytes, expected {}",
size_of::<control::AddReply>()
);
}
// `pod_read_unaligned` (NOT `from_bytes`): `out` is a stack `[u8; N]` with no guaranteed 4-byte
@@ -626,7 +569,6 @@ impl VdisplayDriver for PfVdisplayDriver {
target_id = reply.target_id,
adapter_luid = %format_args!("{:#x}", luid.LowPart),
wudf_pid = reply.wudf_pid,
cursor_excluded = reply.cursor_excluded != 0,
"pf-vdisplay monitor created {}x{}@{}",
mode.width,
mode.height,
@@ -663,7 +605,6 @@ impl VdisplayDriver for PfVdisplayDriver {
luid,
wudf_pid: reply.wudf_pid,
resolved_monitor_id: reply.resolved_monitor_id,
cursor_excluded: reply.cursor_excluded != 0,
})
}
@@ -741,10 +682,6 @@ pub struct PfVdisplayDisplay {
/// freshly created monitor's EDID advertises this volume so host apps tone-map to the client's
/// real panel.
client_hdr: Option<punktfunk_core::quic::HdrMeta>,
/// Declare an IddCx hardware cursor on the created monitor (the M2c cursor channel). Set by
/// [`set_hw_cursor`](VirtualDisplay::set_hw_cursor) before `create`; only honored when the
/// driver handshake reported proto >= 5.
hw_cursor: bool,
/// The session's deliberate-quit flag (`None` = no signal → the linger policy applies). Set by
/// [`set_quit_flag`](VirtualDisplay::set_quit_flag) before `create`; rides into every lease this
/// backend mints so a user "stop" tears the monitor down immediately instead of lingering.
@@ -757,7 +694,6 @@ impl PfVdisplayDisplay {
Ok(Self {
client_fp: None,
client_hdr: None,
hw_cursor: false,
quit: None,
})
}
@@ -776,26 +712,12 @@ impl VirtualDisplay for PfVdisplayDisplay {
self.client_hdr = hdr;
}
fn set_hw_cursor(&mut self, on: bool) {
self.hw_cursor = on;
}
fn hw_cursor(&self) -> bool {
self.hw_cursor
}
fn set_quit_flag(&mut self, quit: std::sync::Arc<std::sync::atomic::AtomicBool>) {
self.quit = Some(quit);
}
fn create(&mut self, mode: Mode) -> Result<VirtualOutput> {
super::manager::vdm().acquire(
mode,
self.client_fp,
self.client_hdr,
self.hw_cursor,
self.quit.clone(),
)
super::manager::vdm().acquire(mode, self.client_fp, self.client_hdr, self.quit.clone())
}
}
@@ -882,12 +804,17 @@ mod tests {
// Live-run diagnostics: surface the manager/backend tracing (activation ladder, settle
// waits, UPDATE_MODES) on stdout — a bare test harness has no subscriber, which made the
// first on-glass run blind.
// (tracing-subscriber is not a dep of this crate — run the host binary for traced runs.)
let _ = tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "debug".into()),
)
.try_init();
// Context probe: can this process see the CCD active-path set at all? (`None` = the query
// itself fails in this session/window-station — the whole ladder would be blind, and a
// "monitor never activated" verdict would be an artifact of the test context.)
// SAFETY: CCD query over an owned empty slice (test-only diagnostics).
let active0 = unsafe { pf_win_display::win_display::count_other_active(&[]) };
let active0 = unsafe { crate::win_display::count_other_active(&[]) };
println!("spike: CCD active paths visible before create: {active0:?}");
let mut vd = PfVdisplayDisplay::new().expect("open pf-vdisplay");
let first = vd
@@ -920,7 +847,7 @@ mod tests {
.target_id;
let in_place = t1 == t2;
// SAFETY: CCD query over a Copy target id (test-only diagnostics).
let active = unsafe { pf_win_display::win_display::active_resolution(t2) };
let active = unsafe { crate::win_display::active_resolution(t2) };
println!(
"in-place resize spike: in_place={in_place} (target {t1} -> {t2}) took {resize_ms} ms, \
active resolution now {active:?}"
+17 -153
View File
@@ -23,10 +23,10 @@ use windows::core::PCWSTR;
use windows::Win32::Devices::Display::{
DisplayConfigGetDeviceInfo, DisplayConfigSetDeviceInfo, GetDisplayConfigBufferSizes,
QueryDisplayConfig, SetDisplayConfig, DISPLAYCONFIG_DEVICE_INFO_GET_ADVANCED_COLOR_INFO,
DISPLAYCONFIG_DEVICE_INFO_GET_SDR_WHITE_LEVEL, DISPLAYCONFIG_DEVICE_INFO_GET_SOURCE_NAME,
DISPLAYCONFIG_DEVICE_INFO_GET_TARGET_NAME, DISPLAYCONFIG_DEVICE_INFO_SET_ADVANCED_COLOR_STATE,
DISPLAYCONFIG_GET_ADVANCED_COLOR_INFO, DISPLAYCONFIG_MODE_INFO,
DISPLAYCONFIG_MODE_INFO_TYPE_SOURCE, DISPLAYCONFIG_OUTPUT_TECHNOLOGY_COMPONENT_VIDEO,
DISPLAYCONFIG_DEVICE_INFO_GET_SOURCE_NAME, DISPLAYCONFIG_DEVICE_INFO_GET_TARGET_NAME,
DISPLAYCONFIG_DEVICE_INFO_SET_ADVANCED_COLOR_STATE, DISPLAYCONFIG_GET_ADVANCED_COLOR_INFO,
DISPLAYCONFIG_MODE_INFO, DISPLAYCONFIG_MODE_INFO_TYPE_SOURCE,
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_COMPONENT_VIDEO,
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_COMPOSITE_VIDEO,
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_DISPLAYPORT_EMBEDDED,
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_DISPLAYPORT_EXTERNAL, DISPLAYCONFIG_OUTPUT_TECHNOLOGY_DVI,
@@ -35,11 +35,10 @@ use windows::Win32::Devices::Display::{
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_SDI, DISPLAYCONFIG_OUTPUT_TECHNOLOGY_SDTVDONGLE,
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_SVIDEO, DISPLAYCONFIG_OUTPUT_TECHNOLOGY_UDI_EMBEDDED,
DISPLAYCONFIG_OUTPUT_TECHNOLOGY_UDI_EXTERNAL, DISPLAYCONFIG_PATH_INFO,
DISPLAYCONFIG_SDR_WHITE_LEVEL, DISPLAYCONFIG_SET_ADVANCED_COLOR_STATE,
DISPLAYCONFIG_SOURCE_DEVICE_NAME, DISPLAYCONFIG_TARGET_DEVICE_NAME,
DISPLAYCONFIG_VIDEO_OUTPUT_TECHNOLOGY, QDC_ALL_PATHS, QDC_ONLY_ACTIVE_PATHS, SDC_ALLOW_CHANGES,
SDC_APPLY, SDC_FORCE_MODE_ENUMERATION, SDC_SAVE_TO_DATABASE, SDC_TOPOLOGY_EXTEND,
SDC_USE_SUPPLIED_DISPLAY_CONFIG,
DISPLAYCONFIG_SET_ADVANCED_COLOR_STATE, DISPLAYCONFIG_SOURCE_DEVICE_NAME,
DISPLAYCONFIG_TARGET_DEVICE_NAME, DISPLAYCONFIG_VIDEO_OUTPUT_TECHNOLOGY, QDC_ALL_PATHS,
QDC_ONLY_ACTIVE_PATHS, SDC_ALLOW_CHANGES, SDC_APPLY, SDC_FORCE_MODE_ENUMERATION,
SDC_SAVE_TO_DATABASE, SDC_TOPOLOGY_EXTEND, SDC_USE_SUPPLIED_DISPLAY_CONFIG,
};
use windows::Win32::Foundation::POINTL;
use windows::Win32::Graphics::Gdi::{
@@ -482,52 +481,6 @@ pub unsafe fn advanced_color_enabled(target_id: u32) -> Option<bool> {
None
}
/// The target's SDR white level as a SCALE relative to 80 nits (`1.0` = 80 nits): where DWM
/// places SDR-white when composing SDR content onto this HDR desktop. An SDR-authored overlay
/// (the composited cursor) must be multiplied by this in scRGB space or it renders visibly
/// darker than the surrounding SDR desktop content (the Windows "SDR content brightness"
/// slider default alone is ~2.5x). `None` = query failed / target not active (callers keep
/// their last value or 1.0).
///
/// # Safety
/// Runs the read-only CCD query FFI over owned locals (same shape as [`advanced_color_enabled`]).
pub unsafe fn sdr_white_level_scale(target_id: u32) -> Option<f32> {
let mut np = 0u32;
let mut nm = 0u32;
if GetDisplayConfigBufferSizes(QDC_ONLY_ACTIVE_PATHS, &mut np, &mut nm).is_err() {
return None;
}
let mut paths = vec![DISPLAYCONFIG_PATH_INFO::default(); np as usize];
let mut modes = vec![DISPLAYCONFIG_MODE_INFO::default(); nm as usize];
if QueryDisplayConfig(
QDC_ONLY_ACTIVE_PATHS,
&mut np,
paths.as_mut_ptr(),
&mut nm,
modes.as_mut_ptr(),
None,
)
.is_err()
{
return None;
}
for p in paths.iter().take(np as usize) {
if p.targetInfo.id == target_id {
let mut info = DISPLAYCONFIG_SDR_WHITE_LEVEL::default();
info.header.r#type = DISPLAYCONFIG_DEVICE_INFO_GET_SDR_WHITE_LEVEL;
info.header.size = size_of::<DISPLAYCONFIG_SDR_WHITE_LEVEL>() as u32;
info.header.adapterId = p.targetInfo.adapterId;
info.header.id = p.targetInfo.id;
if DisplayConfigGetDeviceInfo(&mut info.header) == 0 && info.SDRWhiteLevel > 0 {
// Contract: SDRWhiteLevel/1000 * 80 = nits, i.e. the /1000 IS the 80-nit scale.
return Some(info.SDRWhiteLevel as f32 / 1000.0);
}
return None;
}
}
None
}
/// Force the freshly-added virtual monitor to the client's exact `WxH@Hz`. The ADD IOCTL only
/// ADVERTISES the mode; Windows otherwise activates an IDD target at a 1280x720 default, so the
/// ACTIVE mode (what DXGI Desktop Duplication captures) must be set explicitly. CDS_TEST first so a
@@ -908,18 +861,7 @@ pub unsafe fn isolate_displays_ccd(keep_target_ids: &[u32]) -> Option<SavedConfi
continue;
}
if p.flags & DISPLAYCONFIG_PATH_ACTIVE != 0 {
// Mark the path inactive AND unpin its modes: per the SetDisplayConfig
// contract a path being turned OFF needs BOTH mode indexes marked invalid,
// and leaving them referencing the queried mode entries gets the whole
// supplied config rejected with 0x57 ERROR_INVALID_PARAMETER on some
// driver/topology combinations (field-reported: exclusive mode left the
// physical panel lit, every retry failing 0x57). Writing the all-ones
// sentinel to the whole union is also correct under the virtual-mode-aware
// interpretation (cloneGroupId/sourceModeInfoIdx both become their 0xffff
// INVALID values).
p.flags &= !DISPLAYCONFIG_PATH_ACTIVE;
p.sourceInfo.Anonymous.modeInfoIdx = DISPLAYCONFIG_PATH_MODE_IDX_INVALID;
p.targetInfo.Anonymous.modeInfoIdx = DISPLAYCONFIG_PATH_MODE_IDX_INVALID;
p.flags &= !DISPLAYCONFIG_PATH_ACTIVE; // mark this path inactive
others += 1;
}
}
@@ -928,35 +870,14 @@ pub unsafe fn isolate_displays_ccd(keep_target_ids: &[u32]) -> Option<SavedConfi
// never calls ASSIGN_SWAPCHAIN, so the driver receives no frames. SDC_FORCE_MODE_ENUMERATION
// forces the re-commit; SAVE_TO_DATABASE only in the sole-path case (matches prior behavior —
// don't permanently rewrite the user's multi-display layout; the teardown restore handles it).
let rc = if others > 0 && attempt >= 2 {
// ESCALATION (attempt 2+): supply ONLY the keep paths. Field-reported (AMD +
// pf-vdisplay): carrying the doomed path in the array — inactive, modes unpinned —
// gets the whole config rejected 0x57 on EVERY retry, so the loop alone never
// converged; the same host applies the keep-only shape rc=0 whenever the topology
// database has already made the virtual display sole. The final attempt also drops
// SDC_FORCE_MODE_ENUMERATION in case the driver rejects it combined with a real
// topology change — an actual path removal drives COMMIT_MODES on its own, so the
// re-commit rationale doesn't need the flag here.
let (kp, km) = keep_only_supplied(&paths, &modes);
let mut esc = SDC_APPLY | SDC_USE_SUPPLIED_DISPLAY_CONFIG | SDC_ALLOW_CHANGES;
if attempt < 4 {
esc |= SDC_FORCE_MODE_ENUMERATION;
}
tracing::info!(
"display isolate (CCD): escalating to a keep-only supplied config (attempt {attempt}/4, paths {}→{}, modes {}→{})",
paths.len(), kp.len(), modes.len(), km.len()
);
SetDisplayConfig(Some(kp.as_slice()), Some(km.as_slice()), esc)
} else {
let mut flags = SDC_APPLY
| SDC_USE_SUPPLIED_DISPLAY_CONFIG
| SDC_ALLOW_CHANGES
| SDC_FORCE_MODE_ENUMERATION;
if others == 0 {
flags |= SDC_SAVE_TO_DATABASE;
}
SetDisplayConfig(Some(paths.as_slice()), Some(modes.as_slice()), flags)
};
let mut flags = SDC_APPLY
| SDC_USE_SUPPLIED_DISPLAY_CONFIG
| SDC_ALLOW_CHANGES
| SDC_FORCE_MODE_ENUMERATION;
if others == 0 {
flags |= SDC_SAVE_TO_DATABASE;
}
let rc = SetDisplayConfig(Some(paths.as_slice()), Some(modes.as_slice()), flags);
// A failed apply must be VISIBLE even when the verification below passes vacuously (nothing
// else was active to deactivate — the lid-closed laptop case, where the success INFO used to
// swallow rc=0x5): the re-commit above is load-bearing (COMMIT_MODES → ASSIGN_SWAPCHAIN),
@@ -983,63 +904,6 @@ pub unsafe fn isolate_displays_ccd(keep_target_ids: &[u32]) -> Option<SavedConfi
Some(saved)
}
/// Build the ESCALATED supplied config for [`isolate_displays_ccd`]: ONLY the paths still flagged
/// ACTIVE (the keep set — the caller already cleared ACTIVE on every doomed path), with the mode
/// table rebuilt to just the entries those paths reference (indexes remapped). Docs-wise the
/// dropped inactive entries were declared ignored anyway ("Only the paths within this array that
/// have the DISPLAYCONFIG_PATH_ACTIVE flag set are set"), so this shape asks for the identical
/// topology — minus the array contents some driver/OS validation combos reject with 0x57.
unsafe fn keep_only_supplied(
paths: &[DISPLAYCONFIG_PATH_INFO],
modes: &[DISPLAYCONFIG_MODE_INFO],
) -> (Vec<DISPLAYCONFIG_PATH_INFO>, Vec<DISPLAYCONFIG_MODE_INFO>) {
let mut out_paths = Vec::new();
let mut out_modes = Vec::new();
// old mode index → new. Shared entries dedup through here: a clone-style pair references ONE
// source mode, and the docs require each source/target mode to appear in the table only once.
let mut remap = std::collections::HashMap::new();
for p in paths {
if p.flags & DISPLAYCONFIG_PATH_ACTIVE == 0 {
continue;
}
let mut q = *p;
q.sourceInfo.Anonymous.modeInfoIdx = remap_mode_idx(
q.sourceInfo.Anonymous.modeInfoIdx,
modes,
&mut out_modes,
&mut remap,
);
q.targetInfo.Anonymous.modeInfoIdx = remap_mode_idx(
q.targetInfo.Anonymous.modeInfoIdx,
modes,
&mut out_modes,
&mut remap,
);
out_paths.push(q);
}
(out_paths, out_modes)
}
/// Move `modes[old]` into `out` (once — `remap` dedups) and return its new index. INVALID and
/// out-of-range indexes stay INVALID — `SDC_ALLOW_CHANGES` lets best-mode logic fill the gap.
fn remap_mode_idx(
old: u32,
modes: &[DISPLAYCONFIG_MODE_INFO],
out: &mut Vec<DISPLAYCONFIG_MODE_INFO>,
remap: &mut std::collections::HashMap<u32, u32>,
) -> u32 {
if old == DISPLAYCONFIG_PATH_MODE_IDX_INVALID {
return old;
}
let Some(m) = modes.get(old as usize) else {
return DISPLAYCONFIG_PATH_MODE_IDX_INVALID;
};
*remap.entry(old).or_insert_with(|| {
out.push(*m);
(out.len() - 1) as u32
})
}
/// The desktop-space rectangle `(x, y, w, h)` of `target_id`'s SOURCE — where this display's
/// region lives in the desktop coordinate space. `None` while the target isn't an active path.
/// Used by the IDD-push compose kick to dirty THE TARGET display: with parallel displays the
+255 -6
View File
@@ -7,12 +7,8 @@
//! and ffmpeg's `hevc_nvenc` (encode thread) — each thread makes it current before use;
//! * device memory: pitched allocations, the reusable `BufferPool`/`DeviceBuffer`, IPC
//! export/import, host readback, and the plane copies;
//! * GL / external-memory interop (`RegisteredTexture`, `ExternalDmabuf`).
//!
//! (The CUDA cursor-blend PTX kernel that used to live here is retired: vendored PTX is JIT'd
//! against the driver's ISA ceiling and silently dies on older drivers. The NVENC cursor blend
//! is now the SPIR-V compute pass in [`super::vkslot`], dispatched over Vulkan-allocated,
//! CUDA-imported input slots.)
//! * GL / external-memory interop (`RegisteredTexture`, `ExternalDmabuf`); and
//! * the CUDA cursor-blend kernel (`CursorBlend`).
//!
//! (We use GL interop, not EGL interop: `cuGraphicsEGLRegisterImage` is Tegra-only on the desktop
//! driver — see [`super::egl`].)
@@ -22,6 +18,7 @@
#![deny(clippy::undocumented_unsafe_blocks)]
use anyhow::{bail, Result};
use std::ffi::CStr;
use std::os::raw::{c_uint, c_void};
use std::sync::{Arc, Mutex, OnceLock};
@@ -283,6 +280,258 @@ pub fn copy_stream_handle() -> *mut c_void {
/// Max cursor-overlay bitmap edge (px) uploaded to the device blend buffer — matches the Vulkan path.
pub const CURSOR_MAX: u32 = 256;
/// GPU cursor-overlay compositor for the NVENC path (cursor-as-metadata): loads the `cursor_blend`
/// PTX module once and blends a straight-alpha RGBA cursor into an encoder-OWNED NVENC input surface
/// (ARGB / NV12 / YUV444) with a small kernel launched over the cursor's rectangle — no full-frame
/// pass, and the compositor's dmabuf is never touched. The cursor bitmap lives in a device buffer
/// re-uploaded only when it changes. Requires `context()` to have succeeded (driver present).
pub struct CursorBlend {
module: CUmodule,
f_argb: CUfunction,
f_nv12: CUfunction,
f_yuv444: CUfunction,
cur_buf: CUdeviceptr, // device RGBA staging (CURSOR_MAX²·4, tight rows)
}
// SAFETY: process-lifetime driver handles used only from the encode thread with the shared context
// current — like [`DeviceBuffer`], moving the struct between threads cannot dangle or race.
unsafe impl Send for CursorBlend {}
impl CursorBlend {
/// Load the embedded PTX image and resolve the three blend kernels + a device cursor buffer.
pub fn new(ptx: &[u8]) -> Result<CursorBlend> {
// cuModuleLoadData reads a PTX image as a NUL-terminated string; the embedded .ptx is not,
// so append a terminator.
let mut image = ptx.to_vec();
image.push(0);
let mut module: CUmodule = std::ptr::null_mut();
// SAFETY: `&mut module` is a live out-param the driver fills; `image` is a NUL-terminated PTX
// byte image that outlives the synchronous load. `ck` bails on error before `module` is used.
unsafe {
ck(
cuModuleLoadData(&mut module, image.as_ptr() as *const c_void),
"cuModuleLoadData(cursor_blend)",
)?;
}
let getf = |name: &CStr| -> Result<CUfunction> {
let mut f: CUfunction = std::ptr::null_mut();
// SAFETY: `module` loaded above; each name is a valid NUL-terminated symbol present in
// the module (verified in the .ptx `.entry` list); `&mut f` is a live out-param.
unsafe {
ck(
cuModuleGetFunction(&mut f, module, name.as_ptr()),
"cuModuleGetFunction",
)?;
}
Ok(f)
};
let f_argb = getf(c"blend_argb")?;
let f_nv12 = getf(c"blend_nv12")?;
let f_yuv444 = getf(c"blend_yuv444")?;
let mut cur_buf: CUdeviceptr = 0;
// SAFETY: `&mut cur_buf` is a live out-param; the size fits the CURSOR_MAX² RGBA buffer.
unsafe {
ck(
cuMemAlloc_v2(&mut cur_buf, (CURSOR_MAX * CURSOR_MAX * 4) as usize),
"cuMemAlloc(cursor)",
)?;
}
Ok(CursorBlend {
module,
f_argb,
f_nv12,
f_yuv444,
cur_buf,
})
}
/// Upload the cursor RGBA (`cw*ch*4`, tight rows) into the device blend buffer. Call only when
/// the bitmap changes; position moves are just kernel args.
pub fn upload(&self, rgba: &[u8], cw: u32, ch: u32) -> Result<()> {
let cw = cw.min(CURSOR_MAX);
let ch = ch.min(CURSOR_MAX);
let row = cw as usize * 4;
let copy = CUDA_MEMCPY2D {
srcMemoryType: 1, // HOST
srcHost: rgba.as_ptr() as *const c_void,
srcPitch: row,
dstMemoryType: CU_MEMORYTYPE_DEVICE,
dstDevice: self.cur_buf,
dstPitch: row,
WidthInBytes: row,
Height: ch as usize,
..Default::default()
};
// SAFETY: HOST→DEVICE 2D copy of `row*ch` bytes; `rgba` covers at least that (caller passes
// `cw*ch*4`), `cur_buf` is the CURSOR_MAX²·4 device alloc (row ≤ CURSOR_MAX·4, ch ≤ CURSOR_MAX).
// Synchronous via `copy_blocking`. Requires the context current (caller's contract).
unsafe { copy_blocking(&copy, "cursor HtoD") }
}
/// Blend into a packed 4-byte (NVENC ARGB) owned surface at `(ox,oy)`.
#[allow(clippy::too_many_arguments)] // surface geometry + cursor size + offset — a struct would just be unpacked at the call
pub fn blend_argb(
&self,
surf: CUdeviceptr,
pitch: usize,
w: u32,
h: u32,
cw: u32,
ch: u32,
ox: i32,
oy: i32,
sync: bool,
) -> Result<()> {
let (mut a_surf, mut a_cur) = (surf, self.cur_buf);
let (mut a_pitch, mut a_w, mut a_h) = (pitch as i32, w as i32, h as i32);
let (mut a_cw, mut a_ch) = (cw.min(CURSOR_MAX) as i32, ch.min(CURSOR_MAX) as i32);
let (mut a_ox, mut a_oy) = (ox, oy);
let mut args: [*mut c_void; 9] = [
&mut a_surf as *mut _ as *mut c_void,
&mut a_pitch as *mut _ as *mut c_void,
&mut a_w as *mut _ as *mut c_void,
&mut a_h as *mut _ as *mut c_void,
&mut a_cur as *mut _ as *mut c_void,
&mut a_cw as *mut _ as *mut c_void,
&mut a_ch as *mut _ as *mut c_void,
&mut a_ox as *mut _ as *mut c_void,
&mut a_oy as *mut _ as *mut c_void,
];
self.launch(self.f_argb, a_cw as u32, a_ch as u32, &mut args, sync)
}
/// Blend into an owned planar YUV444 surface (3 stacked full-res planes) at `(ox,oy)`.
#[allow(clippy::too_many_arguments)] // surface geometry + cursor size + offset — a struct would just be unpacked at the call
pub fn blend_yuv444(
&self,
base: CUdeviceptr,
pitch: usize,
w: u32,
h: u32,
cw: u32,
ch: u32,
ox: i32,
oy: i32,
sync: bool,
) -> Result<()> {
let (mut a_base, mut a_cur) = (base, self.cur_buf);
let (mut a_pitch, mut a_w, mut a_h) = (pitch as i32, w as i32, h as i32);
let (mut a_cw, mut a_ch) = (cw.min(CURSOR_MAX) as i32, ch.min(CURSOR_MAX) as i32);
let (mut a_ox, mut a_oy) = (ox, oy);
let mut args: [*mut c_void; 9] = [
&mut a_base as *mut _ as *mut c_void,
&mut a_pitch as *mut _ as *mut c_void,
&mut a_w as *mut _ as *mut c_void,
&mut a_h as *mut _ as *mut c_void,
&mut a_cur as *mut _ as *mut c_void,
&mut a_cw as *mut _ as *mut c_void,
&mut a_ch as *mut _ as *mut c_void,
&mut a_ox as *mut _ as *mut c_void,
&mut a_oy as *mut _ as *mut c_void,
];
self.launch(self.f_yuv444, a_cw as u32, a_ch as u32, &mut args, sync)
}
/// Blend into an owned NV12 surface (Y plane at `base`, interleaved UV at `base + pitch*h`).
#[allow(clippy::too_many_arguments)] // surface geometry + cursor size + offset — a struct would just be unpacked at the call
pub fn blend_nv12(
&self,
base: CUdeviceptr,
pitch: usize,
w: u32,
h: u32,
cw: u32,
ch: u32,
ox: i32,
oy: i32,
sync: bool,
) -> Result<()> {
let (mut a_yb, mut a_uvb, mut a_cur) = (base, base + pitch as u64 * h as u64, self.cur_buf);
let (mut a_yp, mut a_uvp) = (pitch as i32, pitch as i32);
let (mut a_w, mut a_h) = (w as i32, h as i32);
let (mut a_cw, mut a_ch) = (cw.min(CURSOR_MAX) as i32, ch.min(CURSOR_MAX) as i32);
let (mut a_ox, mut a_oy) = (ox, oy);
let mut args: [*mut c_void; 11] = [
&mut a_yb as *mut _ as *mut c_void,
&mut a_yp as *mut _ as *mut c_void,
&mut a_uvb as *mut _ as *mut c_void,
&mut a_uvp as *mut _ as *mut c_void,
&mut a_w as *mut _ as *mut c_void,
&mut a_h as *mut _ as *mut c_void,
&mut a_cur as *mut _ as *mut c_void,
&mut a_cw as *mut _ as *mut c_void,
&mut a_ch as *mut _ as *mut c_void,
&mut a_ox as *mut _ as *mut c_void,
&mut a_oy as *mut _ as *mut c_void,
];
// One thread per 2x2 luma block → grid over ceil(cw/2) × ceil(ch/2).
self.launch(
self.f_nv12,
(a_cw as u32).div_ceil(2),
(a_ch as u32).div_ceil(2),
&mut args,
sync,
)
}
/// Launch `f` over a `work_w × work_h` grid (16×16 blocks) on the copy stream; `sync` waits
/// for it, `!sync` leaves completion to the stream (stream-ordered consumers only — the
/// kernel PARAMETERS are copied at launch time, so the arg locals need not outlive the call).
fn launch(
&self,
f: CUfunction,
work_w: u32,
work_h: u32,
args: &mut [*mut c_void],
sync: bool,
) -> Result<()> {
if work_w == 0 || work_h == 0 {
return Ok(());
}
const B: u32 = 16;
let stream = copy_stream();
// SAFETY: `f` is a resolved kernel from our loaded module; `args` holds pointers to live
// locals whose types match the kernel's C parameters (per the call site above) — CUDA
// copies the parameter values during `cuLaunchKernel` itself, so they need not outlive
// the call. Grid/block dims are non-zero. Launched on the copy stream (ordered after the
// input-surface copy issued on the same stream); `sync` waits, `!sync` leaves ordering to
// the stream (the NVENC IO-stream binding). Requires the context current.
unsafe {
ck(
cuLaunchKernel(
f,
work_w.div_ceil(B),
work_h.div_ceil(B),
1,
B,
B,
1,
0,
stream,
args.as_mut_ptr(),
std::ptr::null_mut(),
),
"cuLaunchKernel(cursor)",
)?;
if sync {
ck(cuStreamSynchronize(stream), "cuStreamSynchronize(cursor)")?;
}
Ok(())
}
}
}
impl Drop for CursorBlend {
fn drop(&mut self) {
// SAFETY: `cur_buf`/`module` are our own handles, freed exactly once here; the context is
// current on the encode thread that drops the encoder. Errors are ignored on teardown.
unsafe {
let _ = cuMemFree_v2(self.cur_buf);
let _ = cuModuleUnload(self.module);
}
}
}
/// Allocate one pitched device buffer for `width`x`height` 4-byte pixels; returns `(ptr, pitch)`.
fn alloc_pitched(width: u32, height: u32) -> Result<(CUdeviceptr, usize)> {
let mut ptr: CUdeviceptr = 0;
+75 -1
View File
@@ -9,7 +9,7 @@
#![deny(clippy::undocumented_unsafe_blocks)]
use anyhow::{bail, Result};
use std::os::raw::{c_int, c_uint, c_void};
use std::os::raw::{c_char, c_int, c_uint, c_void};
use std::sync::OnceLock;
pub type CUresult = c_uint; // CUDA_SUCCESS == 0
@@ -20,6 +20,8 @@ pub type CUdeviceptr = u64;
pub type CUgraphicsResource = *mut c_void;
pub type CUarray = *mut c_void;
pub type CUexternalMemory = *mut c_void; // opaque CUextMemory_st*
pub type CUmodule = *mut c_void; // opaque CUmod_st*
pub type CUfunction = *mut c_void; // opaque CUfunc_st*
/// `CUmemorytype` (cuda.h): HOST=1, DEVICE=2, ARRAY=3, UNIFIED=4.
pub const CU_MEMORYTYPE_DEVICE: c_uint = 2;
@@ -142,6 +144,26 @@ pub(crate) struct CudaApi {
cuIpcGetMemHandle: unsafe extern "C" fn(*mut CUipcMemHandle, CUdeviceptr) -> CUresult,
cuIpcOpenMemHandle: unsafe extern "C" fn(*mut CUdeviceptr, CUipcMemHandle, c_uint) -> CUresult,
cuIpcCloseMemHandle: unsafe extern "C" fn(CUdeviceptr) -> CUresult,
// Cursor-overlay blend: a linear device alloc + a PTX module with the blend kernels launched
// over the cursor's small rectangle (see [`CursorBlend`]).
cuMemAlloc_v2: unsafe extern "C" fn(*mut CUdeviceptr, usize) -> CUresult,
cuModuleLoadData: unsafe extern "C" fn(*mut CUmodule, *const c_void) -> CUresult,
cuModuleUnload: unsafe extern "C" fn(CUmodule) -> CUresult,
cuModuleGetFunction: unsafe extern "C" fn(*mut CUfunction, CUmodule, *const c_char) -> CUresult,
#[allow(clippy::type_complexity)]
cuLaunchKernel: unsafe extern "C" fn(
CUfunction,
c_uint,
c_uint,
c_uint,
c_uint,
c_uint,
c_uint,
c_uint,
CUstream,
*mut *mut c_void,
*mut *mut c_void,
) -> CUresult,
}
// SAFETY: every field is a bare `extern "C" fn` address into the leaked, process-lifetime
// `libcuda` mapping (`cuda_api` `forget`s the `Library`, so it is never unloaded) — an immutable
@@ -214,6 +236,11 @@ pub(crate) fn cuda_api() -> Option<&'static CudaApi> {
.or_else(|_| lib.get(b"cuIpcOpenMemHandle\0"))
.ok()?,
cuIpcCloseMemHandle: *lib.get(b"cuIpcCloseMemHandle\0").ok()?,
cuMemAlloc_v2: *lib.get(b"cuMemAlloc_v2\0").ok()?,
cuModuleLoadData: *lib.get(b"cuModuleLoadData\0").ok()?,
cuModuleUnload: *lib.get(b"cuModuleUnload\0").ok()?,
cuModuleGetFunction: *lib.get(b"cuModuleGetFunction\0").ok()?,
cuLaunchKernel: *lib.get(b"cuLaunchKernel\0").ok()?,
};
std::mem::forget(lib); // keep libcuda mapped for the fn pointers' lifetime (process)
Some(api)
@@ -277,6 +304,53 @@ pub(crate) unsafe fn cuMemFree_v2(dptr: CUdeviceptr) -> CUresult {
None => CU_ERROR_NOT_LOADED,
}
}
pub(crate) unsafe fn cuMemAlloc_v2(dptr: *mut CUdeviceptr, size: usize) -> CUresult {
match cuda_api() {
Some(a) => (a.cuMemAlloc_v2)(dptr, size),
None => CU_ERROR_NOT_LOADED,
}
}
pub(crate) unsafe fn cuModuleLoadData(m: *mut CUmodule, image: *const c_void) -> CUresult {
match cuda_api() {
Some(a) => (a.cuModuleLoadData)(m, image),
None => CU_ERROR_NOT_LOADED,
}
}
pub(crate) unsafe fn cuModuleUnload(m: CUmodule) -> CUresult {
match cuda_api() {
Some(a) => (a.cuModuleUnload)(m),
None => CU_ERROR_NOT_LOADED,
}
}
pub(crate) unsafe fn cuModuleGetFunction(
f: *mut CUfunction,
m: CUmodule,
name: *const c_char,
) -> CUresult {
match cuda_api() {
Some(a) => (a.cuModuleGetFunction)(f, m, name),
None => CU_ERROR_NOT_LOADED,
}
}
#[allow(clippy::too_many_arguments)]
pub(crate) unsafe fn cuLaunchKernel(
f: CUfunction,
gx: c_uint,
gy: c_uint,
gz: c_uint,
bx: c_uint,
by: c_uint,
bz: c_uint,
shmem: c_uint,
stream: CUstream,
params: *mut *mut c_void,
extra: *mut *mut c_void,
) -> CUresult {
match cuda_api() {
Some(a) => (a.cuLaunchKernel)(f, gx, gy, gz, bx, by, bz, shmem, stream, params, extra),
None => CU_ERROR_NOT_LOADED,
}
}
pub(crate) unsafe fn cuMemcpy2DAsync_v2(copy: *const CUDA_MEMCPY2D, stream: CUstream) -> CUresult {
match cuda_api() {
Some(a) => (a.cuMemcpy2DAsync_v2)(copy, stream),
@@ -1,170 +0,0 @@
#version 450
// Cursor-overlay blend for the direct-SDK NVENC path (cursor-as-metadata), dispatched over the
// cursor's rectangle only — the Vulkan replacement for the retired cursor_blend.cu PTX kernels
// (PTX is JIT'd against the driver's ISA ceiling, so a vendored blob silently dies on older
// drivers: CUDA errors 222/218 on-glass; SPIR-V has no such coupling). The NVENC input surface is
// Vulkan-allocated, CUDA-imported external memory (see vkslot.rs), so this shader writes the very
// bytes NVENC encodes.
//
// MODE (spec constant): 0 = packed 4-byte ARGB (NVENC byte order B,G,R,A), 1 = NV12 (Y plane +
// interleaved half-res UV at row surfH), 2 = planar YUV444 (3 full-res planes stacked at
// pitch*surfH). BT.709 limited-range coefficients — identical to rgb2nv12_buf.comp and the
// retired .cu, so the cursor colour matches the frame regardless of backend.
//
// The surface SSBO is uint[] (no 8-bit storage dependency — maximum driver reach): every
// invocation exclusively owns the 32-bit words it read-modify-writes. ARGB: one invocation per
// cursor pixel = one word. NV12/YUV444: one invocation per WORD-ALIGNED 4-px luma span (per two
// rows for NV12, whose 2 chroma bytes-pairs land in one exclusive word). Spans are aligned to the
// SURFACE, not the cursor, so neighbouring invocations never share a word even at odd `ox`.
//
// Rebuild: glslc cursor_blend.comp -o cursor_blend.spv (vendored beside this file)
layout(local_size_x = 8, local_size_y = 8, local_size_z = 1) in;
layout(constant_id = 0) const uint MODE = 0;
layout(std430, binding = 0) buffer Surf { uint surf[]; };
layout(std430, binding = 1) readonly buffer Cur { uint cur[]; };
layout(push_constant) uniform Push {
uint pitch; // surface row stride, bytes (4-aligned by construction)
uint surfW; // content width, px
uint surfH; // luma rows (plane stride multiplier)
uint curW; // cursor bitmap width, px
uint curH; // cursor bitmap height, px
int ox; // cursor top-left on the surface, px (may be negative)
int oy;
} pc;
// Cursor texel (straight-alpha RGBA, tight rows) or (0,0,0,0) outside the bitmap.
uvec4 cursor_px(int cx, int cy) {
if (cx < 0 || cy < 0 || cx >= int(pc.curW) || cy >= int(pc.curH)) return uvec4(0);
uint w = cur[uint(cy) * pc.curW + uint(cx)];
return uvec4(w & 0xFFu, (w >> 8) & 0xFFu, (w >> 16) & 0xFFu, (w >> 24) & 0xFFu); // R,G,B,A
}
uint blend8(uint dst, uint src, uint a) {
return (src * a + dst * (255u - a)) / 255u;
}
// BT.709 limited RGB→Y/U/V (matches the retired .cu / rgb2nv12_buf.comp).
uint y_of(uvec4 s) {
return uint(clamp(16.0 + 0.1826 * float(s.r) + 0.6142 * float(s.g) + 0.0620 * float(s.b) + 0.5, 0.0, 255.0));
}
float u_of(uvec4 s) { return 128.0 - 0.1006 * float(s.r) - 0.3386 * float(s.g) + 0.4392 * float(s.b); }
float v_of(uvec4 s) { return 128.0 + 0.4392 * float(s.r) - 0.3989 * float(s.g) - 0.0403 * float(s.b); }
// Read-modify-write one byte lane of a word index.
void rmw_byte(uint word_idx, uint lane, uint val8, uint a) {
uint w = surf[word_idx];
uint shift = lane * 8u;
uint d = (w >> shift) & 0xFFu;
uint b = blend8(d, val8, a);
surf[word_idx] = (w & ~(0xFFu << shift)) | (b << shift);
}
void main() {
if (MODE == 0u) {
// ARGB: one invocation per cursor pixel; each surface pixel is one exclusive word.
int cx = int(gl_GlobalInvocationID.x);
int cy = int(gl_GlobalInvocationID.y);
if (cx >= int(pc.curW) || cy >= int(pc.curH)) return;
int px = pc.ox + cx, py = pc.oy + cy;
if (px < 0 || py < 0 || px >= int(pc.surfW) || py >= int(pc.surfH)) return;
uvec4 s = cursor_px(cx, cy);
if (s.a == 0u) return;
uint idx = (uint(py) * pc.pitch + uint(px) * 4u) / 4u;
uint w = surf[idx];
uint b = blend8(w & 0xFFu, s.b, s.a); // B lane
uint g = blend8((w >> 8) & 0xFFu, s.g, s.a); // G lane
uint r = blend8((w >> 16) & 0xFFu, s.r, s.a); // R lane
surf[idx] = (w & 0xFF000000u) | (r << 16) | (g << 8) | b;
return;
}
// NV12 / YUV444: one invocation per SURFACE-word-aligned 4-px luma span. Span origin:
// x0 = floor(ox/4)*4 + span*4 (surface px), rows walk the cursor rect.
int span = int(gl_GlobalInvocationID.x);
int row = int(gl_GlobalInvocationID.y);
int x0 = (pc.ox >> 2) << 2; // word-aligned start at/left-of ox (ox may be negative)
int px0 = x0 + span * 4;
if (MODE == 2u) {
// YUV444: rows walk cursor rows one at a time.
if (row >= int(pc.curH)) return;
int py = pc.oy + row;
if (py < 0 || py >= int(pc.surfH)) return;
uint plane = pc.pitch * pc.surfH;
for (int i = 0; i < 4; i++) {
int px = px0 + i;
int cx = px - pc.ox;
if (px < 0 || px >= int(pc.surfW)) continue;
uvec4 s = cursor_px(cx, row);
if (s.a == 0u) continue;
uint off = uint(py) * pc.pitch + uint(px);
uint U = uint(clamp(u_of(s) + 0.5, 0.0, 255.0));
uint V = uint(clamp(v_of(s) + 0.5, 0.0, 255.0));
rmw_byte(off / 4u, off % 4u, y_of(s), s.a);
rmw_byte((plane + off) / 4u, (plane + off) % 4u, U, s.a);
rmw_byte((2u * plane + off) / 4u, (2u * plane + off) % 4u, V, s.a);
}
return;
}
// NV12: rows walk 2-row luma blocks (row = block row). The span's 4 luma px × 2 rows are
// exclusive words; its 2 chroma samples (4 bytes) are one exclusive word.
int base_cy = row * 2;
if (base_cy >= int(pc.curH)) return;
// Luma: 4 px × 2 rows.
for (int j = 0; j < 2; j++) {
int cy = base_cy + j;
int py = pc.oy + cy;
if (cy >= int(pc.curH) || py < 0 || py >= int(pc.surfH)) continue;
for (int i = 0; i < 4; i++) {
int px = px0 + i;
int cx = px - pc.ox;
if (px < 0 || px >= int(pc.surfW)) continue;
uvec4 s = cursor_px(cx, cy);
if (s.a == 0u) continue;
uint off = uint(py) * pc.pitch + uint(px);
rmw_byte(off / 4u, off % 4u, y_of(s), s.a);
}
}
// Chroma: two UV samples covering the span's 2x2 blocks, alpha-weighted like the .cu kernel.
// The UV plane starts at row surfH; sample (uvx, uvy) lives at uv_base + uvy*pitch + uvx*2.
// Guard: only spans whose px0 is 4-aligned own their chroma word (px0 is by construction).
int py_top = pc.oy + base_cy;
int uvy = py_top >> 1;
if (py_top < 0 || uvy < 0 || uvy * 2 >= int(pc.surfH)) return;
uint uv_base = pc.pitch * pc.surfH;
for (int hf = 0; hf < 2; hf++) {
// Each hf = one 2x2 luma block = one UV sample (2 bytes).
int bx = px0 + hf * 2;
if (bx < 0 || bx >= int(pc.surfW)) continue;
int uvx = bx >> 1;
float ua = 0.0, va = 0.0, wa = 0.0;
int cnt = 0;
for (int j = 0; j < 2; j++) {
for (int i = 0; i < 2; i++) {
int px = bx + i;
int py = py_top + j;
int cx = px - pc.ox;
int cy = base_cy + j;
if (px < 0 || py < 0 || px >= int(pc.surfW) || py >= int(pc.surfH)) continue;
uvec4 s = cursor_px(cx, cy);
if (s.a == 0u) continue;
ua += u_of(s) * float(s.a);
va += v_of(s) * float(s.a);
wa += float(s.a);
cnt++;
}
}
if (wa <= 0.0 || cnt == 0) continue;
uint U = uint(clamp(ua / wa + 0.5, 0.0, 255.0));
uint V = uint(clamp(va / wa + 0.5, 0.0, 255.0));
uint amean = uint(clamp(wa / float(cnt) + 0.5, 0.0, 255.0));
uint off = uv_base + uint(uvy) * pc.pitch + uint(uvx) * 2u;
rmw_byte(off / 4u, off % 4u, U, amean);
rmw_byte((off + 1u) / 4u, (off + 1u) % 4u, V, amean);
}
}
Binary file not shown.
-1
View File
@@ -15,7 +15,6 @@ pub mod client;
pub mod cuda;
pub mod egl;
pub mod proto;
pub mod vkslot;
pub mod vulkan;
pub mod worker;
-710
View File
@@ -1,710 +0,0 @@
//! Vulkan-allocated NVENC input slots + the Vulkan compute cursor blend — the driver-portable
//! replacement for the retired `cursor_blend.cu` PTX kernels (design: remote-desktop-sweep §8,
//! Phase A). A vendored PTX blob is JIT'd against the driver's ISA ceiling, so it silently dies
//! on drivers older than the generating toolkit (CUDA errors 222/218 on-glass — the KWin leg's
//! invisible composite cursor); SPIR-V has no such coupling.
//!
//! ```text
//! exportable VkBuffer ──vkGetMemoryFdKHR(OPAQUE_FD)──▶ cuImportExternalMemory ──▶ CUdeviceptr
//! ▲ │ NVENC registers + encodes
//! └── cursor_blend.comp dispatch (cursor rect only) ◀───┘ CUDA copies frames in
//! ```
//!
//! The direct-SDK NVENC encoder allocates its input ring through [`VkSlotBlend::alloc_slot`]
//! instead of `cuMemAllocPitch`: same contiguous layouts (`InputSurface` docs), but the memory is
//! Vulkan external memory both APIs address. Per cursor-bearing frame the encoder CPU-syncs its
//! CUDA copy, then [`VkSlotBlend::blend`] dispatches the compute blend over the cursor's
//! rectangle and fence-waits — the same coherence ceremony [`super::vulkan::VkBridge`] ships for
//! its CSC (fence-ordered cross-API access on NVIDIA, no queue-family transfer needed). Frames
//! without a cursor never touch Vulkan, keeping the stream-ordered fast path intact.
//!
//! Falls back cleanly: if bring-up fails the encoder allocates plain CUDA surfaces and composite
//! mode degrades to no cursor (warned once) — never a failed session.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use super::cuda::{self, CUdeviceptr};
use anyhow::{anyhow, Context as _, Result};
use ash::vk;
/// Max cursor-overlay bitmap edge (px) — matches [`cuda::CURSOR_MAX`] and the capture-side clamp.
pub const CURSOR_MAX: u32 = cuda::CURSOR_MAX;
/// The vendored SPIR-V for `cursor_blend.comp` (beside this file; rebuild with
/// `glslc cursor_blend.comp -o cursor_blend.spv`).
const CURSOR_SPV: &[u8] = include_bytes!("cursor_blend.spv");
/// NVENC input-surface layout — selects the spec-constant `MODE` pipeline and the allocation
/// arithmetic (mirroring `InputSurface`'s contiguous layouts).
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum SlotFormat {
/// Packed 4-byte ARGB (NVENC byte order B,G,R,A): `pitch × height`.
Argb,
/// NV12: Y rows `[0, H)` + interleaved UV rows `[H, 3H/2)` under one pitch.
Nv12,
/// Planar YUV444: three full-res planes stacked at `pitch × height` intervals.
Yuv444,
}
impl SlotFormat {
fn mode(self) -> u32 {
match self {
SlotFormat::Argb => 0,
SlotFormat::Nv12 => 1,
SlotFormat::Yuv444 => 2,
}
}
fn row_bytes(self, width: u32) -> u64 {
match self {
SlotFormat::Argb => width as u64 * 4,
SlotFormat::Nv12 | SlotFormat::Yuv444 => width as u64,
}
}
fn rows(self, height: u32) -> u64 {
match self {
SlotFormat::Argb => height as u64,
SlotFormat::Nv12 => height as u64 + (height as u64 / 2).max(1),
SlotFormat::Yuv444 => height as u64 * 3,
}
}
}
/// What the encoder holds per ring slot: the CUDA view it registers with NVENC plus the id it
/// hands back to [`VkSlotBlend::blend`]. The backing Vulkan objects + CUDA mapping live in the
/// [`VkSlotBlend`] (freed by [`free_slots`](VkSlotBlend::free_slots) / drop), so this is Copy —
/// the encoder's ring keeps its existing shape.
#[derive(Clone, Copy)]
pub struct VkSlotRef {
/// Device pointer NVENC registers (CUDA's mapping of the Vulkan memory).
pub ptr: CUdeviceptr,
/// Row stride in bytes (ours: row bytes rounded up to 256).
pub pitch: usize,
/// Luma rows (the plane-stride multiplier, as in `InputSurface`).
pub height: u32,
/// Index into the blend's slot table.
pub id: usize,
}
/// One allocated slot's backing objects, freed together in reverse order (CUDA mapping first).
struct SlotAlloc {
buffer: vk::Buffer,
memory: vk::DeviceMemory,
/// CUDA's import of the exported OPAQUE_FD — must drop BEFORE the Vulkan memory is freed.
cuda: cuda::ExternalDmabuf,
size: u64,
}
/// 28-byte push-constant block matching `cursor_blend.comp`'s `Push`.
#[repr(C)]
struct Push {
pitch: u32,
surf_w: u32,
surf_h: u32,
cur_w: u32,
cur_h: u32,
ox: i32,
oy: i32,
}
pub struct VkSlotBlend {
_entry: ash::Entry,
instance: ash::Instance,
device: ash::Device,
ext_fd: ash::khr::external_memory_fd::Device,
queue: vk::Queue,
cmd_pool: vk::CommandPool,
cmd: vk::CommandBuffer,
fence: vk::Fence,
mem_props: vk::PhysicalDeviceMemoryProperties,
shader: vk::ShaderModule,
desc_layout: vk::DescriptorSetLayout,
pipe_layout: vk::PipelineLayout,
desc_pool: vk::DescriptorPool,
desc_set: vk::DescriptorSet,
/// One pipeline per [`SlotFormat`], indexed by `mode()` (spec constant).
pipelines: [vk::Pipeline; 3],
/// Host-visible cursor bitmap staging (CURSOR_MAX²·4, tight rows), persistently mapped.
cur_buf: vk::Buffer,
cur_mem: vk::DeviceMemory,
cur_map: *mut u8,
slots: Vec<SlotAlloc>,
}
// SAFETY: raw Vulkan handles + a persistently-mapped pointer, all uniquely owned by this struct
// and destroyed exactly once in `Drop`; used from the encoder thread but moved with it. `Send`
// only (not `Sync`), matching the single-thread use — transferring opaque handles cannot dangle.
unsafe impl Send for VkSlotBlend {}
impl VkSlotBlend {
/// Bring up the device + blend pipelines. Requires the CUDA shared context (the encoder's) to
/// be established; picks the NVIDIA physical device (the NVENC path is NVIDIA by definition).
pub fn new() -> Result<VkSlotBlend> {
// SAFETY: standard ash bring-up, same shape as `VkBridge::new` — every call is `unsafe`
// only because ash cannot statically verify handle/CreateInfo validity. Every
// `*CreateInfo`/`AllocateInfo` is built from locals that live for the duration of the
// synchronous call reading them; every handle passed was created and `?`-checked in this
// same function. Shares nothing across threads.
unsafe {
let entry = ash::Entry::load().context("load libvulkan")?;
let app = vk::ApplicationInfo::default().api_version(vk::API_VERSION_1_1);
let instance = entry
.create_instance(
&vk::InstanceCreateInfo::default().application_info(&app),
None,
)
.context("vkCreateInstance")?;
let phys = match instance
.enumerate_physical_devices()
.context("enumerate GPUs")?
.into_iter()
.find(|&p| instance.get_physical_device_properties(p).vendor_id == 0x10DE)
{
Some(p) => p,
None => {
instance.destroy_instance(None);
return Err(anyhow!("no NVIDIA Vulkan device"));
}
};
let mem_props = instance.get_physical_device_memory_properties(phys);
let qf = match instance
.get_physical_device_queue_family_properties(phys)
.iter()
.position(|q| q.queue_flags.contains(vk::QueueFlags::COMPUTE))
{
Some(i) => i as u32,
None => {
instance.destroy_instance(None);
return Err(anyhow!("no compute-capable queue family"));
}
};
let prio = [1.0f32];
let qci = [vk::DeviceQueueCreateInfo::default()
.queue_family_index(qf)
.queue_priorities(&prio)];
let exts = [ash::khr::external_memory_fd::NAME.as_ptr()];
let device = match instance.create_device(
phys,
&vk::DeviceCreateInfo::default()
.queue_create_infos(&qci)
.enabled_extension_names(&exts),
None,
) {
Ok(d) => d,
Err(e) => {
instance.destroy_instance(None);
return Err(e).context("vkCreateDevice (external_memory_fd supported?)");
}
};
// From here teardown-on-error goes through `destroy_partial`, which tolerates null
// handles — build everything into an incrementally-filled struct.
let ext_fd = ash::khr::external_memory_fd::Device::new(&instance, &device);
let queue = device.get_device_queue(qf, 0);
let mut me = VkSlotBlend {
_entry: entry,
instance,
device,
ext_fd,
queue,
cmd_pool: vk::CommandPool::null(),
cmd: vk::CommandBuffer::null(),
fence: vk::Fence::null(),
mem_props,
shader: vk::ShaderModule::null(),
desc_layout: vk::DescriptorSetLayout::null(),
pipe_layout: vk::PipelineLayout::null(),
desc_pool: vk::DescriptorPool::null(),
desc_set: vk::DescriptorSet::null(),
pipelines: [vk::Pipeline::null(); 3],
cur_buf: vk::Buffer::null(),
cur_mem: vk::DeviceMemory::null(),
cur_map: std::ptr::null_mut(),
slots: Vec::new(),
};
me.init_objects(qf).inspect_err(|_| {
// `Drop` runs the same teardown and tolerates the nulls left by a partial init.
})?;
tracing::info!(
"Vulkan slot blend ready (exportable NVENC inputs + SPIR-V cursor blend)"
);
Ok(me)
}
}
/// The non-device objects: command machinery, cursor staging, descriptor + pipelines.
fn init_objects(&mut self, qf: u32) -> Result<()> {
// SAFETY: same contract as `new` — ash calls on the live `self.device` with builder infos
// from locals outliving each synchronous call; created handles are stored into `self`
// immediately so the caller's `Drop` frees them on any later failure.
unsafe {
let d = &self.device;
self.cmd_pool = d
.create_command_pool(
&vk::CommandPoolCreateInfo::default()
.queue_family_index(qf)
.flags(vk::CommandPoolCreateFlags::RESET_COMMAND_BUFFER),
None,
)
.context("create command pool")?;
self.cmd = d
.allocate_command_buffers(
&vk::CommandBufferAllocateInfo::default()
.command_pool(self.cmd_pool)
.level(vk::CommandBufferLevel::PRIMARY)
.command_buffer_count(1),
)
.context("allocate command buffer")?[0];
self.fence = d
.create_fence(&vk::FenceCreateInfo::default(), None)
.context("create fence")?;
// Cursor staging: host-visible+coherent SSBO, persistently mapped.
let cur_size = (CURSOR_MAX * CURSOR_MAX * 4) as u64;
self.cur_buf = d
.create_buffer(
&vk::BufferCreateInfo::default()
.size(cur_size)
.usage(vk::BufferUsageFlags::STORAGE_BUFFER),
None,
)
.context("create cursor buffer")?;
let reqs = d.get_buffer_memory_requirements(self.cur_buf);
let mem_type = self
.memory_type(
reqs.memory_type_bits,
vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT,
)
.context("cursor buffer memory type")?;
self.cur_mem = d
.allocate_memory(
&vk::MemoryAllocateInfo::default()
.allocation_size(reqs.size)
.memory_type_index(mem_type),
None,
)
.context("allocate cursor memory")?;
d.bind_buffer_memory(self.cur_buf, self.cur_mem, 0)
.context("bind cursor memory")?;
self.cur_map = d
.map_memory(self.cur_mem, 0, cur_size, vk::MemoryMapFlags::empty())
.context("map cursor memory")? as *mut u8;
// Descriptor set: binding 0 = surface SSBO (rebound per blend), 1 = cursor SSBO.
let bindings = [
vk::DescriptorSetLayoutBinding::default()
.binding(0)
.descriptor_type(vk::DescriptorType::STORAGE_BUFFER)
.descriptor_count(1)
.stage_flags(vk::ShaderStageFlags::COMPUTE),
vk::DescriptorSetLayoutBinding::default()
.binding(1)
.descriptor_type(vk::DescriptorType::STORAGE_BUFFER)
.descriptor_count(1)
.stage_flags(vk::ShaderStageFlags::COMPUTE),
];
self.desc_layout = d
.create_descriptor_set_layout(
&vk::DescriptorSetLayoutCreateInfo::default().bindings(&bindings),
None,
)
.context("create descriptor layout")?;
let pc = [vk::PushConstantRange::default()
.stage_flags(vk::ShaderStageFlags::COMPUTE)
.size(std::mem::size_of::<Push>() as u32)];
let dl = [self.desc_layout];
self.pipe_layout = d
.create_pipeline_layout(
&vk::PipelineLayoutCreateInfo::default()
.set_layouts(&dl)
.push_constant_ranges(&pc),
None,
)
.context("create pipeline layout")?;
let pool_sizes = [vk::DescriptorPoolSize::default()
.ty(vk::DescriptorType::STORAGE_BUFFER)
.descriptor_count(2)];
self.desc_pool = d
.create_descriptor_pool(
&vk::DescriptorPoolCreateInfo::default()
.max_sets(1)
.pool_sizes(&pool_sizes),
None,
)
.context("create descriptor pool")?;
let dls = [self.desc_layout];
self.desc_set = d
.allocate_descriptor_sets(
&vk::DescriptorSetAllocateInfo::default()
.descriptor_pool(self.desc_pool)
.set_layouts(&dls),
)
.context("allocate descriptor set")?[0];
// The shader + one pipeline per MODE (spec constant 0).
if CURSOR_SPV.len() % 4 != 0 {
anyhow::bail!("cursor_blend.spv is not word-aligned");
}
let words: Vec<u32> = CURSOR_SPV
.chunks_exact(4)
.map(|c| u32::from_le_bytes([c[0], c[1], c[2], c[3]]))
.collect();
self.shader = d
.create_shader_module(&vk::ShaderModuleCreateInfo::default().code(&words), None)
.context("create blend shader module")?;
for mode in 0u32..3 {
let entries = [vk::SpecializationMapEntry::default()
.constant_id(0)
.offset(0)
.size(4)];
let data = mode.to_le_bytes();
let spec = vk::SpecializationInfo::default()
.map_entries(&entries)
.data(&data);
let stage = vk::PipelineShaderStageCreateInfo::default()
.stage(vk::ShaderStageFlags::COMPUTE)
.module(self.shader)
.name(c"main")
.specialization_info(&spec);
let info = [vk::ComputePipelineCreateInfo::default()
.stage(stage)
.layout(self.pipe_layout)];
let p = d
.create_compute_pipelines(vk::PipelineCache::null(), &info, None)
.map_err(|(_, e)| e)
.context("create blend pipeline")?[0];
self.pipelines[mode as usize] = p;
}
}
Ok(())
}
fn memory_type(&self, type_bits: u32, flags: vk::MemoryPropertyFlags) -> Result<u32> {
(0..self.mem_props.memory_type_count)
.find(|&i| {
type_bits & (1 << i) != 0
&& self.mem_props.memory_types[i as usize]
.property_flags
.contains(flags)
})
.ok_or_else(|| anyhow!("no memory type for flags {flags:?}"))
}
/// Allocate one NVENC input slot as exportable Vulkan memory mapped into CUDA. Layout matches
/// `InputSurface` (contiguous planes under one pitch); pitch = row bytes rounded to 256.
pub fn alloc_slot(&mut self, fmt: SlotFormat, width: u32, height: u32) -> Result<VkSlotRef> {
let pitch = (fmt.row_bytes(width) + 255) & !255;
let size = pitch * fmt.rows(height);
// SAFETY: exportable-buffer allocation, the exact `VkBridge::ensure_dst` incantation:
// `ExternalMemoryBufferCreateInfo`/`ExportMemoryAllocateInfo` declare OPAQUE_FD,
// `MemoryDedicatedAllocateInfo` ties the memory to the buffer; every info is a local
// outliving its synchronous call and every failure path destroys the objects created so
// far exactly once. `get_memory_fd` hands us an fd that `import_owned_fd` either adopts
// (driver owns it) or closes on failure.
unsafe {
let d = &self.device;
let mut ext_info = vk::ExternalMemoryBufferCreateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
let buffer = d
.create_buffer(
&vk::BufferCreateInfo::default()
.size(size)
.usage(vk::BufferUsageFlags::STORAGE_BUFFER)
.push_next(&mut ext_info),
None,
)
.context("create slot buffer")?;
let reqs = d.get_buffer_memory_requirements(buffer);
let mem_type = match self
.memory_type(reqs.memory_type_bits, vk::MemoryPropertyFlags::DEVICE_LOCAL)
{
Ok(t) => t,
Err(e) => {
d.destroy_buffer(buffer, None);
return Err(e);
}
};
let mut export = vk::ExportMemoryAllocateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD);
let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().buffer(buffer);
let memory = match d.allocate_memory(
&vk::MemoryAllocateInfo::default()
.allocation_size(reqs.size)
.memory_type_index(mem_type)
.push_next(&mut export)
.push_next(&mut dedicated),
None,
) {
Ok(m) => m,
Err(e) => {
d.destroy_buffer(buffer, None);
return Err(e).context("allocate exportable slot memory");
}
};
if let Err(e) = d.bind_buffer_memory(buffer, memory, 0) {
d.free_memory(memory, None);
d.destroy_buffer(buffer, None);
return Err(e).context("bind slot memory");
}
let fd = match self.ext_fd.get_memory_fd(
&vk::MemoryGetFdInfoKHR::default()
.memory(memory)
.handle_type(vk::ExternalMemoryHandleTypeFlags::OPAQUE_FD),
) {
Ok(f) => f,
Err(e) => {
d.free_memory(memory, None);
d.destroy_buffer(buffer, None);
return Err(e).context("vkGetMemoryFdKHR(slot)");
}
};
let ext = match cuda::ExternalDmabuf::import_owned_fd(fd, reqs.size) {
Ok(c) => c,
Err(e) => {
d.free_memory(memory, None);
d.destroy_buffer(buffer, None);
return Err(e).context("cuImportExternalMemory(slot OPAQUE_FD)");
}
};
let r = VkSlotRef {
ptr: ext.ptr,
pitch: pitch as usize,
height,
id: self.slots.len(),
};
self.slots.push(SlotAlloc {
buffer,
memory,
cuda: ext,
size,
});
Ok(r)
}
}
/// Free every allocated slot (encoder teardown, alongside its ring clear). CUDA mappings drop
/// first (field order in [`SlotAlloc`] frees `cuda` via its own `Drop` before we free the VK
/// objects explicitly here).
pub fn free_slots(&mut self) {
for s in self.slots.drain(..) {
drop(s.cuda); // CUDA's view of the memory goes first
// SAFETY: `buffer`/`memory` were created in `alloc_slot`, are uniquely owned by the
// drained `SlotAlloc`, and are destroyed exactly once here. No queue work can be
// in flight: every `blend` fence-waits before returning.
unsafe {
self.device.destroy_buffer(s.buffer, None);
self.device.free_memory(s.memory, None);
}
}
}
/// Upload the cursor RGBA (`cw*ch*4`, tight rows) into the mapped staging buffer. Call only
/// when the bitmap changes; position moves are push constants.
pub fn upload_cursor(&mut self, rgba: &[u8], cw: u32, ch: u32) {
let cw = cw.min(CURSOR_MAX);
let ch = ch.min(CURSOR_MAX);
let len = (cw * ch * 4) as usize;
let len = len.min(rgba.len());
// SAFETY: `cur_map` is the live persistent mapping of the CURSOR_MAX²·4 host-coherent
// allocation (created in `init_objects`, unmapped only in `Drop`); `len` is clamped to
// both the source slice and the buffer capacity. No blend is in flight (every `blend`
// fence-waits before returning), so no GPU read races this host write.
unsafe {
std::ptr::copy_nonoverlapping(rgba.as_ptr(), self.cur_map, len);
}
}
/// Blend the uploaded cursor into `slot` at `(ox, oy)`: record, submit, fence-wait. The
/// caller has CPU-synced its CUDA frame copy first; the fence wait makes the shader's writes
/// visible to the subsequent NVENC encode (the `VkBridge` precedent: fence-ordered cross-API
/// access, no queue-family transfer — NVIDIA-only path).
#[allow(clippy::too_many_arguments)] // surface geometry + cursor rect — unpacked kernel args
pub fn blend_ref(
&mut self,
slot: &VkSlotRef,
fmt: SlotFormat,
surf_w: u32,
cw: u32,
ch: u32,
ox: i32,
oy: i32,
) -> Result<()> {
let alloc = self
.slots
.get(slot.id)
.ok_or_else(|| anyhow!("bad slot id {}", slot.id))?;
let cw = cw.min(CURSOR_MAX);
let ch = ch.min(CURSOR_MAX);
if cw == 0 || ch == 0 {
return Ok(());
}
let push = Push {
pitch: slot.pitch as u32,
surf_w,
surf_h: slot.height,
cur_w: cw,
cur_h: ch,
ox,
oy,
};
// Dispatch geometry (must match cursor_blend.comp): ARGB = per cursor px; NV12/YUV444 =
// per word-aligned 4-px span × (2-row blocks | rows).
let (gx, gy) = match fmt {
SlotFormat::Argb => (cw.div_ceil(8), ch.div_ceil(8)),
_ => {
let x0 = (ox >> 2) << 2;
let spans = ((ox + cw as i32) - x0 + 3).div_euclid(4).max(1) as u32;
let rows = match fmt {
SlotFormat::Nv12 => ch.div_ceil(2),
_ => ch,
};
(spans.div_ceil(8), rows.div_ceil(8))
}
};
// SAFETY: single-threaded record/submit/wait on handles this struct owns. The descriptor
// update is safe because no prior submission is in flight (every blend fence-waits and
// the fence is reset before reuse). Buffer infos and barrier structs are locals outliving
// their synchronous calls. The dispatch's shader accesses stay in-bounds by the shader's
// own guards (surfW/surfH/curW/curH from `push`) against the slot allocation sized in
// `alloc_slot` for exactly that geometry.
unsafe {
let d = &self.device;
let surf_info = [vk::DescriptorBufferInfo::default()
.buffer(alloc.buffer)
.offset(0)
.range(alloc.size)];
let cur_info = [vk::DescriptorBufferInfo::default()
.buffer(self.cur_buf)
.offset(0)
.range((CURSOR_MAX * CURSOR_MAX * 4) as u64)];
let writes = [
vk::WriteDescriptorSet::default()
.dst_set(self.desc_set)
.dst_binding(0)
.descriptor_type(vk::DescriptorType::STORAGE_BUFFER)
.buffer_info(&surf_info),
vk::WriteDescriptorSet::default()
.dst_set(self.desc_set)
.dst_binding(1)
.descriptor_type(vk::DescriptorType::STORAGE_BUFFER)
.buffer_info(&cur_info),
];
d.update_descriptor_sets(&writes, &[]);
d.begin_command_buffer(
self.cmd,
&vk::CommandBufferBeginInfo::default()
.flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT),
)
.context("begin blend cmd")?;
// CUDA wrote the frame into this memory outside Vulkan's view — make it visible to
// the shader (external-memory coherence ceremony; NVIDIA honors this with the fence
// ordering alone, the barrier is the spec-shaped belt-and-braces).
let acquire = [vk::MemoryBarrier::default()
.src_access_mask(vk::AccessFlags::MEMORY_WRITE)
.dst_access_mask(vk::AccessFlags::SHADER_READ | vk::AccessFlags::SHADER_WRITE)];
d.cmd_pipeline_barrier(
self.cmd,
vk::PipelineStageFlags::TOP_OF_PIPE,
vk::PipelineStageFlags::COMPUTE_SHADER,
vk::DependencyFlags::empty(),
&acquire,
&[],
&[],
);
d.cmd_bind_pipeline(
self.cmd,
vk::PipelineBindPoint::COMPUTE,
self.pipelines[fmt.mode() as usize],
);
d.cmd_bind_descriptor_sets(
self.cmd,
vk::PipelineBindPoint::COMPUTE,
self.pipe_layout,
0,
&[self.desc_set],
&[],
);
let bytes = std::slice::from_raw_parts(
(&push as *const Push) as *const u8,
std::mem::size_of::<Push>(),
);
d.cmd_push_constants(
self.cmd,
self.pipe_layout,
vk::ShaderStageFlags::COMPUTE,
0,
bytes,
);
d.cmd_dispatch(self.cmd, gx.max(1), gy.max(1), 1);
// Release the shader's writes so the post-fence CUDA/NVENC reads see them.
let release = [vk::MemoryBarrier::default()
.src_access_mask(vk::AccessFlags::SHADER_WRITE)
.dst_access_mask(vk::AccessFlags::MEMORY_READ)];
d.cmd_pipeline_barrier(
self.cmd,
vk::PipelineStageFlags::COMPUTE_SHADER,
vk::PipelineStageFlags::BOTTOM_OF_PIPE,
vk::DependencyFlags::empty(),
&release,
&[],
&[],
);
d.end_command_buffer(self.cmd).context("end blend cmd")?;
let cmds = [self.cmd];
let submit = [vk::SubmitInfo::default().command_buffers(&cmds)];
d.queue_submit(self.queue, &submit, self.fence)
.context("submit blend")?;
let r = d.wait_for_fences(&[self.fence], true, 1_000_000_000);
d.reset_fences(&[self.fence]).ok();
r.context("blend fence wait")?;
}
Ok(())
}
}
impl Drop for VkSlotBlend {
fn drop(&mut self) {
self.free_slots();
// SAFETY: every handle below was created in `new`/`init_objects` (or is null from a
// partial init — Vulkan destroy/free calls are defined no-ops on null handles) and is
// uniquely owned; each is destroyed exactly once here, pipelines/layouts/pools before the
// device, the device before the instance. No work is in flight (`blend` fence-waits).
unsafe {
let d = &self.device;
for p in self.pipelines {
if p != vk::Pipeline::null() {
d.destroy_pipeline(p, None);
}
}
if self.shader != vk::ShaderModule::null() {
d.destroy_shader_module(self.shader, None);
}
if self.desc_pool != vk::DescriptorPool::null() {
d.destroy_descriptor_pool(self.desc_pool, None);
}
if self.pipe_layout != vk::PipelineLayout::null() {
d.destroy_pipeline_layout(self.pipe_layout, None);
}
if self.desc_layout != vk::DescriptorSetLayout::null() {
d.destroy_descriptor_set_layout(self.desc_layout, None);
}
if !self.cur_map.is_null() {
d.unmap_memory(self.cur_mem);
}
if self.cur_buf != vk::Buffer::null() {
d.destroy_buffer(self.cur_buf, None);
}
if self.cur_mem != vk::DeviceMemory::null() {
d.free_memory(self.cur_mem, None);
}
if self.fence != vk::Fence::null() {
d.destroy_fence(self.fence, None);
}
if self.cmd_pool != vk::CommandPool::null() {
d.destroy_command_pool(self.cmd_pool, None);
}
d.destroy_device(None);
self.instance.destroy_instance(None);
}
}
}
+3 -372
View File
@@ -555,9 +555,6 @@ pub struct PunktfunkConnection {
/// (a fetched payload, an offer's format list, or a fetch-request's MIME) —
/// borrow-until-next-call, same contract as `last`.
last_clip: std::sync::Mutex<Option<Vec<u8>>>,
/// The last cursor shape handed out — `next_cursor_shape`'s `rgba` pointer borrows it
/// until the next cursor-shape call (the `last_audio` contract).
last_cursor_shape: std::sync::Mutex<Option<crate::quic::CursorShape>>,
}
/// Lazily-initialized in-core Opus decode state. A coupled-1-stream multistream decoder is
@@ -870,94 +867,6 @@ impl PunktfunkRichInputEx {
}
}
/// [`PunktfunkPenSample::state`] bit: the pen hovers in range (implied by `TOUCHING`).
pub const PUNKTFUNK_PEN_IN_RANGE: u8 = 0x01;
/// [`PunktfunkPenSample::state`] bit: the tip is in contact.
pub const PUNKTFUNK_PEN_TOUCHING: u8 = 0x02;
/// [`PunktfunkPenSample::state`] bit: primary barrel button (or squeeze mapping) held.
pub const PUNKTFUNK_PEN_BARREL1: u8 = 0x04;
/// [`PunktfunkPenSample::state`] bit: secondary barrel button (or double-tap mapping) held.
pub const PUNKTFUNK_PEN_BARREL2: u8 = 0x08;
/// [`PunktfunkPenSample::tool`]: the pen tip.
pub const PUNKTFUNK_PEN_TOOL_PEN: u8 = 0;
/// [`PunktfunkPenSample::tool`]: the eraser (a client-side mode — Apple Pencil has no
/// hardware eraser end; the squeeze/double-tap mapping usually drives this).
pub const PUNKTFUNK_PEN_TOOL_ERASER: u8 = 1;
/// Most samples one [`punktfunk_connection_send_pen`] call accepts (one wire batch).
pub const PUNKTFUNK_PEN_BATCH_MAX: u32 = 8;
/// [`PunktfunkPenSample::tilt_deg`] sentinel: no tilt reading.
pub const PUNKTFUNK_PEN_TILT_UNKNOWN: u8 = 0xFF;
/// [`PunktfunkPenSample::azimuth_deg`] / `roll_deg` sentinel: no reading.
pub const PUNKTFUNK_PEN_ANGLE_UNKNOWN: u16 = 0xFFFF;
/// [`PunktfunkPenSample::distance`] sentinel: no hover-distance reading.
pub const PUNKTFUNK_PEN_DISTANCE_UNKNOWN: u16 = 0xFFFF;
/// One complete stylus state at one instant ([`punktfunk_connection_send_pen`];
/// design/pen-tablet-input.md). STATE-FULL, never an edge event: fill every field on every
/// sample (unknown axes take their `*_UNKNOWN` sentinel) — the host diffs consecutive samples
/// and synthesizes down/up/button transitions itself, which is what makes a lost datagram
/// self-heal. `x`/`y` are normalized `0.0..=1.0` in VIDEO-FRAME space (map your letterbox
/// before filling, exactly like wire touches).
#[cfg(feature = "quic")]
#[repr(C)]
#[derive(Clone, Copy)]
pub struct PunktfunkPenSample {
/// Normalized `0.0..=1.0` across the video frame. Must be finite.
pub x: f32,
/// Normalized `0.0..=1.0` across the video frame. Must be finite.
pub y: f32,
/// Tip force, `0..=65535` full scale (`0` while hovering).
pub pressure: u16,
/// Hover distance `0..=65534` (0 = at the hover floor), or `PUNKTFUNK_PEN_DISTANCE_UNKNOWN`.
pub distance: u16,
/// Tilt azimuth, degrees `0..=359` clockwise from north, or `PUNKTFUNK_PEN_ANGLE_UNKNOWN`.
pub azimuth_deg: u16,
/// Barrel roll (Apple Pencil Pro `rollAngle`), degrees `0..=359`, or
/// `PUNKTFUNK_PEN_ANGLE_UNKNOWN`.
pub roll_deg: u16,
/// µs since the previous sample in the same call (`0` for the first) — the coalesced
/// capture spacing.
pub dt_us: u16,
/// Bitfield of `PUNKTFUNK_PEN_*` state bits. Unknown bits are rejected (`InvalidArg`).
pub state: u8,
/// `PUNKTFUNK_PEN_TOOL_PEN` or `PUNKTFUNK_PEN_TOOL_ERASER`.
pub tool: u8,
/// Tilt from the surface normal, degrees `0..=90`, or `PUNKTFUNK_PEN_TILT_UNKNOWN`.
pub tilt_deg: u8,
/// Set to 0.
pub _reserved: [u8; 3],
}
#[cfg(feature = "quic")]
impl PunktfunkPenSample {
/// `None` = invalid field (non-finite coordinate, unknown state bit, unknown tool) —
/// embedder input is validated strictly, unlike the loss-tolerant wire decode.
fn to_sample(self) -> Option<crate::quic::PenSample> {
use crate::quic as q;
let known = q::PEN_IN_RANGE | q::PEN_TOUCHING | q::PEN_BARREL1 | q::PEN_BARREL2;
if !self.x.is_finite() || !self.y.is_finite() || self.state & !known != 0 {
return None;
}
let tool = match self.tool {
PUNKTFUNK_PEN_TOOL_PEN => q::PenTool::Pen,
PUNKTFUNK_PEN_TOOL_ERASER => q::PenTool::Eraser,
_ => return None,
};
Some(q::PenSample {
state: self.state,
tool,
x: self.x,
y: self.y,
pressure: self.pressure,
distance: self.distance,
tilt_deg: self.tilt_deg,
azimuth_deg: self.azimuth_deg,
roll_deg: self.roll_deg,
dt_us: self.dt_us,
})
}
}
/// Read an optional NUL-terminated UTF-8 string parameter; `Err` = invalid pointer/UTF-8.
#[cfg(feature = "quic")]
unsafe fn opt_cstr<'a>(p: *const std::os::raw::c_char) -> std::result::Result<Option<&'a str>, ()> {
@@ -1076,12 +985,6 @@ pub const PUNKTFUNK_HOST_CAP_GAMEPAD_STATE: u8 = 0x01;
/// Host-capability bit in [`punktfunk_connection_host_caps`]: the host supports the shared
/// clipboard, so a client may offer the toggle. (Mirrors `quic::HOST_CAP_CLIPBOARD`.)
pub const PUNKTFUNK_HOST_CAP_CLIPBOARD: u8 = 0x02;
/// Host-capability bit in [`punktfunk_connection_host_caps`]: the host injects full-fidelity
/// stylus input, so a capable client splits pen contacts out of its touch path and sends them
/// via [`punktfunk_connection_send_pen`]; without the bit that call returns `Unsupported` and
/// the client keeps its pen-as-touch fallback. (Mirrors `quic::HOST_CAP_PEN`;
/// design/pen-tablet-input.md.)
pub const PUNKTFUNK_HOST_CAP_PEN: u8 = 0x10;
// Keep the ABI cap bits in lockstep with the wire constants (compile-time guard against drift).
#[cfg(feature = "quic")]
@@ -1095,15 +998,6 @@ const _: () = {
assert!(PUNKTFUNK_CODEC_PYROWAVE == crate::quic::CODEC_PYROWAVE);
assert!(PUNKTFUNK_HOST_CAP_GAMEPAD_STATE == crate::quic::HOST_CAP_GAMEPAD_STATE);
assert!(PUNKTFUNK_HOST_CAP_CLIPBOARD == crate::quic::HOST_CAP_CLIPBOARD);
assert!(PUNKTFUNK_HOST_CAP_PEN == crate::quic::HOST_CAP_PEN);
assert!(PUNKTFUNK_PEN_IN_RANGE == crate::quic::PEN_IN_RANGE);
assert!(PUNKTFUNK_PEN_TOUCHING == crate::quic::PEN_TOUCHING);
assert!(PUNKTFUNK_PEN_BARREL1 == crate::quic::PEN_BARREL1);
assert!(PUNKTFUNK_PEN_BARREL2 == crate::quic::PEN_BARREL2);
assert!(PUNKTFUNK_PEN_BATCH_MAX as usize == crate::quic::PEN_BATCH_MAX);
assert!(PUNKTFUNK_PEN_TILT_UNKNOWN == crate::quic::PEN_TILT_UNKNOWN);
assert!(PUNKTFUNK_PEN_ANGLE_UNKNOWN == crate::quic::PEN_ANGLE_UNKNOWN);
assert!(PUNKTFUNK_PEN_DISTANCE_UNKNOWN == crate::quic::PEN_DISTANCE_UNKNOWN);
};
// Keep the ABI gamepad constants in lockstep with the wire enum (compile-time guard against drift).
@@ -1506,7 +1400,6 @@ pub unsafe extern "C" fn punktfunk_connect_ex7(
connect_ex_impl(
host,
port,
0, // pre-v11 variant: no client caps
width,
height,
refresh_hz,
@@ -1566,7 +1459,6 @@ pub unsafe extern "C" fn punktfunk_connect_ex8(
connect_ex_impl(
host,
port,
0, // pre-v11 variant: no client caps
width,
height,
refresh_hz,
@@ -1588,70 +1480,6 @@ pub unsafe extern "C" fn punktfunk_connect_ex8(
}
}
/// Like [`punktfunk_connect_ex8`], plus `client_caps` (ABI v11): a bitfield of
/// `PUNKTFUNK_CLIENT_CAP_CURSOR` (0x01). Setting the cursor bit asks the host to STOP
/// compositing the pointer into the video and forward it out-of-band instead — the embedder
/// MUST then drain [`punktfunk_connection_next_cursor_shape`] /
/// [`punktfunk_connection_next_cursor_state`] and draw the pointer itself, or the session has
/// no visible cursor at all. Pass 0 for the composited behavior of every earlier variant.
///
/// # Safety
/// Same as [`punktfunk_connect_ex8`].
#[cfg(feature = "quic")]
#[no_mangle]
#[allow(clippy::too_many_arguments)]
pub unsafe extern "C" fn punktfunk_connect_ex9(
host: *const std::os::raw::c_char,
port: u16,
width: u32,
height: u32,
refresh_hz: u32,
compositor: u32,
gamepad: u32,
bitrate_kbps: u32,
video_caps: u8,
audio_channels: u8,
video_codecs: u8,
preferred_codec: u8,
client_caps: u8,
launch_id: *const std::os::raw::c_char,
pin_sha256: *const u8,
observed_sha256_out: *mut u8,
client_cert_pem: *const std::os::raw::c_char,
client_key_pem: *const std::os::raw::c_char,
timeout_ms: u32,
status_out: *mut i32,
) -> *mut PunktfunkConnection {
unsafe {
connect_ex_impl(
host,
port,
client_caps,
width,
height,
refresh_hz,
compositor,
gamepad,
bitrate_kbps,
video_caps,
audio_channels,
video_codecs,
preferred_codec,
launch_id,
pin_sha256,
observed_sha256_out,
client_cert_pem,
client_key_pem,
timeout_ms,
status_out,
)
}
}
/// [`punktfunk_connect_ex9`] `client_caps` bit: render the host cursor locally (the cursor
/// channel, `design/remote-desktop-sweep.md` M2).
pub const PUNKTFUNK_CLIENT_CAP_CURSOR: u8 = 0x01;
/// Shared body of [`punktfunk_connect_ex7`] / [`punktfunk_connect_ex8`]: `status_out`
/// (nullable) is written on EVERY path — `Ok`, the mapped [`PunktfunkError`],
/// `InvalidArg` for bad arguments, `Panic` if the connect panicked.
@@ -1660,7 +1488,6 @@ pub const PUNKTFUNK_CLIENT_CAP_CURSOR: u8 = 0x01;
unsafe fn connect_ex_impl(
host: *const std::os::raw::c_char,
port: u16,
client_caps: u8,
width: u32,
height: u32,
refresh_hz: u32,
@@ -1747,10 +1574,6 @@ unsafe fn connect_ex_impl(
// themselves (EDR / MediaCodec), so the host's EDID defaults are fine there. An `ex8`
// variant can carry it if a passthrough embedder ever needs it.
None,
// ABI v11 ([`punktfunk_connect_ex9`]): CLIENT_CAP_CURSOR here asks the host to STOP
// compositing the pointer — only an embedder that renders the cursor planes
// ([`punktfunk_connection_next_cursor_shape`]/`_state`) may set it. ex7/ex8 pass 0.
client_caps,
launch,
pin,
identity,
@@ -1770,7 +1593,6 @@ unsafe fn connect_ex_impl(
last_audio: std::sync::Mutex::new(None),
audio_pcm: std::sync::Mutex::new(AudioPcmState::default()),
last_clip: std::sync::Mutex::new(None),
last_cursor_shape: std::sync::Mutex::new(None),
}))
}
Err(e) => {
@@ -2420,151 +2242,6 @@ pub unsafe extern "C" fn punktfunk_connection_next_hdr_meta(
})
}
/// One forwarded host-cursor shape (ABI v11, the cursor channel): straight-alpha RGBA8, no
/// padding, `len == w * h * 4`, hotspot within `w`×`h`. `serial` is the identity
/// [`PunktfunkCursorState`] refers to — cache the built OS cursor by it.
#[repr(C)]
pub struct PunktfunkCursorShape {
pub serial: u32,
pub w: u16,
pub h: u16,
pub hot_x: u16,
pub hot_y: u16,
/// Borrows connection memory until the NEXT cursor-shape call (the audio contract).
pub rgba: *const u8,
pub len: usize,
}
/// Per-frame host-cursor state (ABI v11): position (the pointer/hotspot point in the host
/// video's pixel space), visibility, and the host-driven relative-mode hint. `flags` bit 0 =
/// visible, bit 1 = relative hint (a host app grabbed/hid the pointer — run captured
/// relative; clear = return to absolute, reappearing at `x`/`y`).
#[repr(C)]
pub struct PunktfunkCursorState {
pub serial: u32,
pub flags: u8,
pub x: i32,
pub y: i32,
}
/// Pull the next forwarded cursor SHAPE (sent on pointer-bitmap change over the reliable
/// control stream; only a session connected with `PUNKTFUNK_CLIENT_CAP_CURSOR` against a
/// capable host receives any). On `Ok`, `out->rgba` borrows connection memory until the next
/// cursor-shape call on this handle. Drain from a dedicated thread (one thread per plane).
///
/// # Safety
/// `c` is a valid connection handle; `out` is writable. At most one thread pulls cursor
/// shapes; it may run concurrently with every other plane's puller.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_next_cursor_shape(
c: *mut PunktfunkConnection,
out: *mut PunktfunkCursorShape,
timeout_ms: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if out.is_null() {
return PunktfunkStatus::NullPointer;
}
match c
.inner
.next_cursor_shape(std::time::Duration::from_millis(timeout_ms as u64))
{
Ok(shape) => {
let mut slot = c.last_cursor_shape.lock().unwrap();
*slot = Some(shape);
let sh = slot.as_ref().unwrap();
unsafe {
*out = PunktfunkCursorShape {
serial: sh.serial,
w: sh.w,
h: sh.h,
hot_x: sh.hot_x,
hot_y: sh.hot_y,
rgba: sh.rgba.as_ptr(),
len: sh.rgba.len(),
};
}
PunktfunkStatus::Ok
}
Err(e) => e.status(),
}
})
}
/// Pull the next cursor STATE (a `0xD0` datagram per host encode tick — latest-wins; drain
/// the queue and apply only the newest). Same negotiation gate as
/// [`punktfunk_connection_next_cursor_shape`].
///
/// # Safety
/// `c` is a valid connection handle; `out` is writable. At most one thread pulls cursor
/// state; it may run concurrently with every other plane's puller.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_next_cursor_state(
c: *mut PunktfunkConnection,
out: *mut PunktfunkCursorState,
timeout_ms: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if out.is_null() {
return PunktfunkStatus::NullPointer;
}
match c
.inner
.next_cursor_state(std::time::Duration::from_millis(timeout_ms as u64))
{
Ok(st) => {
unsafe {
*out = PunktfunkCursorState {
serial: st.serial,
flags: st.flags,
x: st.x,
y: st.y,
};
}
PunktfunkStatus::Ok
}
Err(e) => e.status(),
}
})
}
/// Tell the host who renders the pointer (design/remote-desktop-sweep.md §8 — the mid-stream
/// mouse-model flip): `client_draws = true` = this client draws it locally (the desktop mouse
/// model; the host excludes the pointer from the video and forwards shape/state), `false` =
/// the host composites it into the video (the capture model — full fidelity, the pre-channel
/// look). Idempotent, latest-wins; harmless against hosts without the cursor cap (an unknown
/// control message type, ignored). ABI v12.
///
/// # Safety
/// `c` is a valid connection handle.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_set_cursor_render(
c: *mut PunktfunkConnection,
client_draws: bool,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
match c.inner.set_cursor_render(client_draws) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// Pull the next per-AU host timing (0xCF) into `*out`: the host's capture→sent duration for one
/// access unit, correlated to the AU by `pts_ns` (see [`PunktfunkHostTiming`]).
/// [`PunktfunkStatus::NoFrame`] on timeout, [`PunktfunkStatus::Closed`] once the session ended.
@@ -2866,51 +2543,6 @@ pub unsafe extern "C" fn punktfunk_connection_send_rich_input2(
})
}
/// Send one stylus sample batch — `count` (`1..=PUNKTFUNK_PEN_BATCH_MAX`) state-full
/// [`PunktfunkPenSample`]s, oldest first (a capture callback's coalesced samples) — as one
/// `0xCC/0x05` pen datagram (non-blocking enqueue; design/pen-tablet-input.md). Split longer
/// runs into consecutive calls. Gate on `punktfunk_connection_host_caps() &
/// PUNKTFUNK_HOST_CAP_PEN`: toward a host without the bit this returns
/// [`PunktfunkStatus::Unsupported`] — keep the pen-as-touch fallback there.
/// [`PunktfunkStatus::InvalidArg`] on a bad count or a bad sample (non-finite coordinate,
/// unknown state bit / tool).
///
/// # Safety
/// `c` is a valid connection handle; `samples` is null or points to `count` valid
/// [`PunktfunkPenSample`]s.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_send_pen(
c: *mut PunktfunkConnection,
samples: *const PunktfunkPenSample,
count: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if samples.is_null() {
return PunktfunkStatus::NullPointer;
}
if count == 0 || count > PUNKTFUNK_PEN_BATCH_MAX {
return PunktfunkStatus::InvalidArg;
}
let raw = unsafe { std::slice::from_raw_parts(samples, count as usize) };
let mut batch = [crate::quic::PenSample::default(); crate::quic::PEN_BATCH_MAX];
for (slot, s) in batch.iter_mut().zip(raw) {
match s.to_sample() {
Some(v) => *slot = v,
None => return PunktfunkStatus::InvalidArg,
}
}
match c.inner.send_pen(&batch[..count as usize]) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// The currently active session mode — the Welcome's, until an accepted
/// [`punktfunk_connection_request_mode`] switches it. Safe any time after connect.
///
@@ -3125,10 +2757,9 @@ fn build_clip_event(
}
/// The host capability bitfield the session's `Welcome` carried — a bitfield of
/// `PUNKTFUNK_HOST_CAP_GAMEPAD_STATE` / `PUNKTFUNK_HOST_CAP_CLIPBOARD` /
/// `PUNKTFUNK_HOST_CAP_PEN`. A client tests `caps & PUNKTFUNK_HOST_CAP_CLIPBOARD` to decide
/// whether to offer the shared-clipboard toggle, `caps & PUNKTFUNK_HOST_CAP_PEN` before
/// sending stylus batches. Safe any time after connect.
/// `PUNKTFUNK_HOST_CAP_GAMEPAD_STATE` / `PUNKTFUNK_HOST_CAP_CLIPBOARD`. A client tests
/// `caps & PUNKTFUNK_HOST_CAP_CLIPBOARD` to decide whether to offer the shared-clipboard toggle.
/// Safe any time after connect.
///
/// # Safety
/// `c` is a valid connection handle; `caps` is writable (NULL is skipped).
@@ -28,10 +28,6 @@ pub(crate) enum CtrlRequest {
/// Announce that the local clipboard changed — the lazy format-list offer (bytes cross later on
/// a fetch stream). Symmetric message; the host may send one too.
ClipOffer(ClipOffer),
/// Who renders the pointer (cursor-forward sessions): `true` = client draws locally (the
/// desktop mouse model — host excludes + forwards), `false` = host composites into the
/// video (the capture model). Sent on every mouse-model flip; idempotent, latest-wins.
CursorRender(crate::quic::CursorRenderMode),
}
/// What the worker reports to [`NativeClient::connect`] once the handshake lands: the
+7 -107
View File
@@ -20,7 +20,7 @@ use crate::quic::{
RfiRequest, RichInput,
};
use crate::session::Frame;
use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU16, AtomicU32, AtomicU64, Ordering};
use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU32, AtomicU64, Ordering};
use std::sync::mpsc::{Receiver, RecvTimeoutError};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
@@ -42,8 +42,8 @@ pub use self::rumble::{ActuatorQuirks, RumbleCommand};
use self::control::{CtrlRequest, Negotiated};
use self::frame_channel::{DecodeLatAcc, FrameChannel, FramePop};
use self::planes::{
RumbleUpdate, AUDIO_QUEUE, CLIP_EVENT_QUEUE, CURSOR_SHAPE_QUEUE, CURSOR_STATE_QUEUE,
HDR_META_QUEUE, HIDOUT_QUEUE, HOST_TIMING_QUEUE, RUMBLE_QUEUE,
RumbleUpdate, AUDIO_QUEUE, CLIP_EVENT_QUEUE, HDR_META_QUEUE, HIDOUT_QUEUE, HOST_TIMING_QUEUE,
RUMBLE_QUEUE,
};
use self::probe::ProbeState;
use self::pump::run_pump;
@@ -93,22 +93,13 @@ pub struct NativeClient {
/// Inbound per-AU host capture→send timings — 0xCF datagrams (the client always advertises
/// [`quic::VIDEO_CAP_HOST_TIMING`]; an older host simply never sends any).
host_timing: Mutex<Receiver<crate::quic::HostTiming>>,
/// Inbound cursor shapes (control-stream [`crate::quic::CursorShape`]) — only a session
/// that advertised [`quic::CLIENT_CAP_CURSOR`] against a [`quic::HOST_CAP_CURSOR`] host
/// ever receives any.
cursor_shape: Mutex<Receiver<crate::quic::CursorShape>>,
/// Inbound per-frame cursor state — `0xD0` datagrams (same negotiation gate as shapes).
cursor_state: Mutex<Receiver<crate::quic::CursorState>>,
input_tx: tokio::sync::mpsc::UnboundedSender<InputEvent>,
/// Outbound mic frames `(seq, pts_ns, opus)` → encoded as 0xCB datagrams by the worker.
/// Bounded ([`MIC_QUEUE`]): a wedged worker drops fresh frames (logged) instead of queueing
/// audio-latency (and memory) without limit — mic is best-effort end to end.
mic_tx: tokio::sync::mpsc::Sender<(u32, u64, Vec<u8>)>,
/// Outbound 0xCC rich-input plane, PRE-ENCODED datagrams: [`RichInput`] touchpad/motion
/// (encoded in [`NativeClient::send_rich_input`]) and stylus [`crate::quic::PenBatch`]es
/// (encoded in [`NativeClient::send_pen`]) share the channel — the worker's task just
/// forwards bytes, so a new 0xCC kind never touches the pump.
rich_input_tx: tokio::sync::mpsc::UnboundedSender<Vec<u8>>,
/// Outbound rich input (DualSense touchpad / motion) → 0xCC datagrams by the worker.
rich_input_tx: tokio::sync::mpsc::UnboundedSender<RichInput>,
/// Outbound control-stream requests (mode switch, speed test) → the worker's control task.
/// Bounded ([`CTRL_QUEUE`]) — the requests are sparse; a full queue means the control task
/// is wedged/dead, and callers treat it like a closed session.
@@ -124,9 +115,6 @@ pub struct NativeClient {
/// Monotonic id for outbound fetches ([`NativeClient::clip_fetch`]); stays below
/// [`crate::clipboard::INBOUND_REQ_FLAG`] so it never collides with an inbound serve `req_id`.
next_xfer_id: AtomicU32,
/// Wrapping per-connection [`crate::quic::PenBatch::seq`] counter, stamped by
/// [`NativeClient::send_pen`] (the host's reorder gate compares it).
pen_seq: AtomicU16,
/// The host capability bitfield ([`crate::quic::Welcome::host_caps`]) — see
/// [`NativeClient::host_caps`].
pub host_caps: u8,
@@ -328,12 +316,6 @@ impl NativeClient {
// display's EDID so host apps tone-map to the client's real panel; `None` = unknown/SDR
// (the host keeps its built-in EDID defaults). See [`crate::quic::Hello::display_hdr`].
display_hdr: Option<HdrMeta>,
// Non-video client capabilities ([`crate::quic::Hello::client_caps`]) — set
// [`crate::quic::CLIENT_CAP_CURSOR`] ONLY if this embedder actually renders the host
// cursor locally (shape + state planes): the host stops compositing the pointer into
// the video for a session that advertises it, so a non-rendering embedder that sets it
// streams with NO visible cursor at all. `0` = today's composited behavior.
client_caps: u8,
launch: Option<String>,
pin: Option<[u8; 32]>,
identity: Option<(String, String)>,
@@ -350,15 +332,11 @@ impl NativeClient {
std::sync::mpsc::sync_channel::<crate::quic::HostTiming>(HOST_TIMING_QUEUE);
let (input_tx, input_rx) = tokio::sync::mpsc::unbounded_channel::<InputEvent>();
let (mic_tx, mic_rx) = tokio::sync::mpsc::channel::<(u32, u64, Vec<u8>)>(MIC_QUEUE);
let (rich_input_tx, rich_input_rx) = tokio::sync::mpsc::unbounded_channel::<Vec<u8>>();
let (rich_input_tx, rich_input_rx) = tokio::sync::mpsc::unbounded_channel::<RichInput>();
let (ctrl_tx, ctrl_rx) = tokio::sync::mpsc::channel::<CtrlRequest>(CTRL_QUEUE);
let (clip_event_tx, clip_event_rx) =
std::sync::mpsc::sync_channel::<ClipEventCore>(CLIP_EVENT_QUEUE);
let (clip_cmd_tx, clip_cmd_rx) = tokio::sync::mpsc::unbounded_channel::<ClipCommand>();
let (cursor_shape_tx, cursor_shape_rx) =
std::sync::mpsc::sync_channel::<crate::quic::CursorShape>(CURSOR_SHAPE_QUEUE);
let (cursor_state_tx, cursor_state_rx) =
std::sync::mpsc::sync_channel::<crate::quic::CursorState>(CURSOR_STATE_QUEUE);
let (ready_tx, ready_rx) = std::sync::mpsc::channel::<Result<Negotiated>>();
let shutdown = Arc::new(AtomicBool::new(false));
let quit = Arc::new(AtomicBool::new(false));
@@ -412,7 +390,6 @@ impl NativeClient {
video_codecs,
preferred_codec,
display_hdr,
client_caps,
launch,
pin,
identity,
@@ -424,8 +401,6 @@ impl NativeClient {
hidout_tx,
hdr_meta_tx,
host_timing_tx,
cursor_shape_tx,
cursor_state_tx,
input_rx,
mic_rx,
rich_input_rx,
@@ -470,8 +445,6 @@ impl NativeClient {
hidout: Mutex::new(hidout_rx),
hdr_meta: Mutex::new(hdr_meta_rx),
host_timing: Mutex::new(host_timing_rx),
cursor_shape: Mutex::new(cursor_shape_rx),
cursor_state: Mutex::new(cursor_state_rx),
input_tx,
mic_tx,
rich_input_tx,
@@ -479,7 +452,6 @@ impl NativeClient {
clip: Mutex::new(clip_event_rx),
clip_cmd_tx,
next_xfer_id: AtomicU32::new(1),
pen_seq: AtomicU16::new(0),
host_caps: negotiated.host_caps,
probe,
shutdown,
@@ -580,20 +552,6 @@ impl NativeClient {
.map_err(|_| PunktfunkError::Closed)
}
/// Tell the host who renders the pointer (cursor-forward sessions —
/// design/remote-desktop-sweep.md §8): `true` = this client draws it locally (the desktop
/// mouse model; the host excludes the pointer from the video and forwards shape/state),
/// `false` = the host composites it into the video (the capture model — full fidelity,
/// the pre-channel behavior). Call on every mouse-model flip; idempotent, latest-wins,
/// no-op on hosts without [`HOST_CAP_CURSOR`](crate::quic::HOST_CAP_CURSOR).
pub fn set_cursor_render(&self, client_draws: bool) -> Result<()> {
self.ctrl_tx
.try_send(CtrlRequest::CursorRender(crate::quic::CursorRenderMode {
client_draws,
}))
.map_err(|_| PunktfunkError::Closed)
}
/// Ask the host's encoder to emit a fresh IDR keyframe now (client recovery on a stalled
/// decode). Non-blocking, fire-and-forget — the recovered keyframe is the only ack. The
/// caller should throttle (the decode stays wedged across several frames until the IDR
@@ -934,32 +892,6 @@ impl NativeClient {
}
}
/// Pull the next host cursor shape (design/remote-desktop-sweep.md M2): RGBA bitmap +
/// hotspot, sent on pointer-bitmap change over the reliable control stream. The embedder
/// caches by `serial` and builds an OS cursor from it; [`NativeClient::next_cursor_state`]
/// references shapes by serial. Only a session that advertised
/// [`crate::quic::CLIENT_CAP_CURSOR`] against a capable host receives any. Same
/// timeout/closed semantics as [`NativeClient::next_hidout`].
pub fn next_cursor_shape(&self, timeout: Duration) -> Result<crate::quic::CursorShape> {
match self.cursor_shape.lock().unwrap().recv_timeout(timeout) {
Ok(s) => Ok(s),
Err(RecvTimeoutError::Timeout) => Err(PunktfunkError::NoFrame),
Err(RecvTimeoutError::Disconnected) => Err(PunktfunkError::Closed),
}
}
/// Pull the next per-frame cursor state (`0xD0`): position, visibility and the M3
/// relative-mode hint, referencing a shape by serial. Latest-wins — an embedder should
/// drain the queue and apply only the newest. Same negotiation gate and timeout/closed
/// semantics as [`NativeClient::next_cursor_shape`].
pub fn next_cursor_state(&self, timeout: Duration) -> Result<crate::quic::CursorState> {
match self.cursor_state.lock().unwrap().recv_timeout(timeout) {
Ok(s) => Ok(s),
Err(RecvTimeoutError::Timeout) => Err(PunktfunkError::NoFrame),
Err(RecvTimeoutError::Disconnected) => Err(PunktfunkError::Closed),
}
}
/// Pull the next per-AU host timing (0xCF): the host's capture→sent duration for one access
/// unit, correlated to the AU by `pts_ns`. Feeds the unified stats HUD's `host` / `network`
/// split (`network = (received + clock_offset pts) host_us`); a stats consumer should
@@ -1080,39 +1012,7 @@ impl NativeClient {
/// loss like every datagram. No-op unless the host runs the DualSense gamepad backend.
pub fn send_rich_input(&self, rich: RichInput) -> Result<()> {
self.rich_input_tx
.send(rich.encode())
.map_err(|_| PunktfunkError::Closed)
}
/// Queue one stylus sample batch for delivery as a `0xCC/0x05` pen datagram
/// (design/pen-tablet-input.md). `samples` are state-full and oldest-first (a capture
/// callback's coalesced samples), at most [`crate::quic::PEN_BATCH_MAX`] per call — split
/// longer runs into consecutive calls so the stamped wrapping `seq` keeps them ordered.
/// Best-effort like every datagram: a lost batch self-heals on the next one (the samples
/// carry full state, the host diffs — see [`crate::quic::PenTracker`]).
///
/// **Heartbeat contract**: while the pen is in range or touching, repeat the last sample
/// at least every ~100 ms even when nothing changed (capture APIs are silent for a
/// stationary pen) — the host force-releases the stroke after
/// [`crate::quic::PEN_TOUCH_TIMEOUT_MS`] of silence as its dead-client failsafe.
///
/// Requires the host to have advertised [`crate::quic::HOST_CAP_PEN`]; toward an older
/// host this returns `Unsupported` (embedders keep their pen-as-touch fallback instead of
/// spraying 240 Hz datagrams the host drops unread).
pub fn send_pen(&self, samples: &[crate::quic::PenSample]) -> Result<()> {
if self.host_caps & crate::quic::HOST_CAP_PEN == 0 {
return Err(PunktfunkError::Unsupported(
"host did not advertise HOST_CAP_PEN",
));
}
if samples.is_empty() || samples.len() > crate::quic::PEN_BATCH_MAX {
return Err(PunktfunkError::InvalidArg(
"pen batch must hold 1..=PEN_BATCH_MAX samples",
));
}
let seq = self.pen_seq.fetch_add(1, Ordering::Relaxed);
self.rich_input_tx
.send(crate::quic::PenBatch::new(seq, samples).encode())
.send(rich)
.map_err(|_| PunktfunkError::Closed)
}
+4 -16
View File
@@ -86,22 +86,10 @@ impl NativeClient {
// A typed application close from the host (pairing not armed / armed for a
// different device / rate-limited / version mismatch) beats the generic
// transport error the aborted exchange produced — it is the actual answer.
// Same close-vs-stream-error race as the connect handshake: give the
// host's CONNECTION_CLOSE a short grace to be processed before deciding
// the error was plain transport trouble.
Err(e) => {
if conn.close_reason().is_none() {
let _ = tokio::time::timeout(
std::time::Duration::from_millis(300),
conn.closed(),
)
.await;
}
Err(match reject_from_close(&conn) {
Some(r) => PunktfunkError::Rejected(r),
None => e,
})
}
Err(e) => Err(match reject_from_close(&conn) {
Some(r) => PunktfunkError::Rejected(r),
None => e,
}),
ok => ok,
};
// Always tell the host we're done so it never blocks at its read — code 0 on
@@ -35,16 +35,6 @@ pub(crate) const HOST_TIMING_QUEUE: usize = 512;
/// a dropped fetch-request makes the serving stream time out and reset cleanly.
pub(crate) const CLIP_EVENT_QUEUE: usize = 32;
/// Cursor-shape plane depth (control-stream [`crate::quic::CursorShape`], one per pointer-bitmap
/// change — human-paced). Overflow drops the newest (try_send); the next shape change or a
/// serial mismatch against `0xD0` state heals it visually within a shape-change period.
pub(crate) const CURSOR_SHAPE_QUEUE: usize = 8;
/// Cursor-state plane depth (`0xD0`, one datagram per captured frame). Latest-wins state — the
/// embedder drains per present; a tiny ring only bridges scheduling jitter. Overflow drops the
/// newest (try_send), healed by the very next frame's datagram.
pub(crate) const CURSOR_STATE_QUEUE: usize = 8;
/// One Opus packet from the host's audio datagram stream (48 kHz stereo, 5 ms frames).
#[derive(Clone, Debug)]
pub struct AudioPacket {
+3 -8
View File
@@ -51,8 +51,6 @@ pub(super) async fn run_pump(args: WorkerArgs) {
hidout_tx,
hdr_meta_tx,
host_timing_tx,
cursor_shape_tx,
cursor_state_tx,
input_rx,
mut mic_rx,
mut rich_input_rx,
@@ -99,12 +97,11 @@ pub(super) async fn run_pump(args: WorkerArgs) {
}
});
// Rich-input task: pre-encoded 0xCC uplink datagrams (DualSense touchpad / motion, pen
// batches — encoded at the NativeClient surface so new plane kinds never touch the pump).
// Rich-input task: embedder DualSense touchpad / motion → 0xCC uplink datagrams.
let rich_conn = conn.clone();
tokio::spawn(async move {
while let Some(d) = rich_input_rx.recv().await {
let _ = rich_conn.send_datagram(d.into());
while let Some(rich) = rich_input_rx.recv().await {
let _ = rich_conn.send_datagram(rich.encode().into());
}
});
@@ -126,7 +123,6 @@ pub(super) async fn run_pump(args: WorkerArgs) {
clock_offset: clock_offset.clone(),
clock_gen: clock_gen.clone(),
clip_event_tx: clip_event_tx.clone(),
cursor_shape_tx,
}
.run(),
);
@@ -140,7 +136,6 @@ pub(super) async fn run_pump(args: WorkerArgs) {
hidout_tx,
hdr_meta_tx,
host_timing_tx,
cursor_state_tx,
));
// Clipboard task: the fetch-stream accept loop (host pulls what we offered) + outbound fetches
@@ -21,9 +21,6 @@ pub(super) struct ControlTask {
/// Clipboard metadata events (ClipState/ClipOffer) feed the same event plane the
/// clipboard task uses for fetch data.
pub(super) clip_event_tx: std::sync::mpsc::SyncSender<ClipEventCore>,
/// Host cursor shapes ([`CursorShape`], sent on pointer-bitmap change) → the embedder's
/// shape plane ([`NativeClient::next_cursor_shape`]).
pub(super) cursor_shape_tx: std::sync::mpsc::SyncSender<crate::quic::CursorShape>,
}
impl ControlTask {
@@ -39,7 +36,6 @@ impl ControlTask {
clock_offset,
clock_gen,
clip_event_tx,
cursor_shape_tx,
} = self;
// Mid-stream clock re-sync (see [`ClockResync`]): a batch runs every
// CLOCK_RESYNC_INTERVAL and whenever the pump asks (CtrlRequest::ClockResync after
@@ -71,7 +67,6 @@ impl ControlTask {
}
CtrlRequest::ClipControl(c) => c.encode(),
CtrlRequest::ClipOffer(o) => o.encode(),
CtrlRequest::CursorRender(m) => m.encode(),
};
if io::write_msg(&mut ctrl_send, &bytes).await.is_err() {
break;
@@ -172,10 +167,6 @@ impl ControlTask {
seq: offer.seq,
kinds: offer.kinds,
});
} else if let Ok(shape) = crate::quic::CursorShape::decode(&msg) {
// Pointer bitmap changed (cursor channel, only when negotiated). try_send:
// an overflowing ring drops the newest shape — the next change resends.
let _ = cursor_shape_tx.try_send(shape);
} else {
tracing::warn!(
tag = ?msg.first(),
@@ -3,9 +3,6 @@
use super::*;
// One parameter per demuxed plane — grouping them into a struct would just move the field
// list one hop away from the single call site.
#[allow(clippy::too_many_arguments)]
pub(super) async fn run(
conn: quinn::Connection,
audio_tx: std::sync::mpsc::SyncSender<AudioPacket>,
@@ -14,7 +11,6 @@ pub(super) async fn run(
hidout_tx: std::sync::mpsc::SyncSender<crate::quic::HidOutput>,
hdr_meta_tx: std::sync::mpsc::SyncSender<crate::quic::HdrMeta>,
host_timing_tx: std::sync::mpsc::SyncSender<crate::quic::HostTiming>,
cursor_state_tx: std::sync::mpsc::SyncSender<crate::quic::CursorState>,
) {
// Per-pad reorder gate for v2 rumble envelopes (the seq analog of the host's gamepad-state
// gate): a datagram the network reordered must not roll a stopped motor back on. Legacy v1
@@ -77,11 +73,6 @@ pub(super) async fn run(
let _ = host_timing_tx.try_send(t);
}
}
Some(&crate::quic::CURSOR_STATE_MAGIC) => {
if let Some(s) = crate::quic::decode_cursor_state_datagram(&d) {
let _ = cursor_state_tx.try_send(s);
}
}
_ => {} // unknown tag — a newer host; ignore
}
}
@@ -143,10 +143,6 @@ pub(super) async fn connect_and_handshake(args: &WorkerArgs) -> Result<Handshake
// The client display's HDR volume → the host's virtual-display EDID (host apps
// tone-map to the client's real panel). `None` = unknown/SDR.
display_hdr,
// NOT unconditional like HOST_TIMING above: CLIENT_CAP_CURSOR makes the host
// stop compositing the pointer, so only an embedder that actually renders the
// cursor locally may set it (the embedder decides, we pass through).
client_caps: args.client_caps,
}
.encode(),
)
@@ -244,22 +240,9 @@ pub(super) async fn connect_and_handshake(args: &WorkerArgs) -> Result<Handshake
negotiated,
host_caps,
}),
Err(e) => {
// The host's typed close can land a beat AFTER the stream error it caused: the
// stream reset/FIN and the CONNECTION_CLOSE are in flight together, and quinn can
// hand the reader its mid-frame EOF before it processes the close. Give the close a
// short grace to arrive so a host-side setup failure renders as its real reason
// ("the host could not start the stream session") instead of "control stream
// finished mid-frame". No-op when the connection already closed (or never will —
// bounded by the timeout).
if conn.close_reason().is_none() {
let _ = tokio::time::timeout(std::time::Duration::from_millis(300), conn.closed())
.await;
}
Err(match reject_from_close(&conn) {
Some(r) => PunktfunkError::Rejected(r),
None => e,
})
}
Err(e) => Err(match reject_from_close(&conn) {
Some(r) => PunktfunkError::Rejected(r),
None => e,
}),
}
}
+2 -6
View File
@@ -5,7 +5,7 @@ use crate::clipboard::{ClipCommand, ClipEventCore};
use crate::config::{CompositorPref, GamepadPref, Mode};
use crate::error::Result;
use crate::input::InputEvent;
use crate::quic::{HdrMeta, HidOutput};
use crate::quic::{HdrMeta, HidOutput, RichInput};
use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64};
use std::sync::mpsc::SyncSender;
use std::sync::{Arc, Mutex};
@@ -22,7 +22,6 @@ pub(crate) struct WorkerArgs {
pub(crate) video_codecs: u8,
pub(crate) preferred_codec: u8,
pub(crate) display_hdr: Option<HdrMeta>,
pub(crate) client_caps: u8,
pub(crate) launch: Option<String>,
pub(crate) pin: Option<[u8; 32]>,
pub(crate) identity: Option<(String, String)>,
@@ -39,12 +38,9 @@ pub(crate) struct WorkerArgs {
pub(crate) hidout_tx: SyncSender<HidOutput>,
pub(crate) hdr_meta_tx: SyncSender<HdrMeta>,
pub(crate) host_timing_tx: SyncSender<crate::quic::HostTiming>,
pub(crate) cursor_shape_tx: SyncSender<crate::quic::CursorShape>,
pub(crate) cursor_state_tx: SyncSender<crate::quic::CursorState>,
pub(crate) input_rx: tokio::sync::mpsc::UnboundedReceiver<InputEvent>,
pub(crate) mic_rx: tokio::sync::mpsc::Receiver<(u32, u64, Vec<u8>)>,
/// Pre-encoded 0xCC datagrams (rich input AND pen batches — see `NativeClient.rich_input_tx`).
pub(crate) rich_input_rx: tokio::sync::mpsc::UnboundedReceiver<Vec<u8>>,
pub(crate) rich_input_rx: tokio::sync::mpsc::UnboundedReceiver<RichInput>,
pub(crate) ctrl_rx: tokio::sync::mpsc::Receiver<CtrlRequest>,
pub(crate) ctrl_tx: tokio::sync::mpsc::Sender<CtrlRequest>,
/// Inbound clipboard event plane feed — the control task pushes ClipState/ClipOffer, the
-2
View File
@@ -61,7 +61,6 @@ pub enum PunktfunkStatus {
RejectedSuperseded = -26,
RejectedWireVersion = -27,
RejectedBusy = -28,
RejectedSetupFailed = -29,
Panic = -99,
}
@@ -90,7 +89,6 @@ impl PunktfunkError {
R::Superseded => PunktfunkStatus::RejectedSuperseded,
R::WireVersionMismatch => PunktfunkStatus::RejectedWireVersion,
R::Busy => PunktfunkStatus::RejectedBusy,
R::SetupFailed => PunktfunkStatus::RejectedSetupFailed,
}
}
}
+2 -31
View File
@@ -73,17 +73,6 @@ pub enum InputKind {
/// [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); an older host ignores the
/// unknown tag (every pad then uses the session-default kind — the pre-existing behaviour).
GamepadArrival = 14,
/// One Unicode scalar of **committed text** — `code` = the scalar value, everything else 0.
///
/// The IME path: the layout-independent VK key events cannot express text an input method
/// *commits* (autocorrect, gesture typing, non-Latin scripts, emoji), so a capable client
/// sends the committed characters verbatim and the host injects them directly (Windows
/// `KEYEVENTF_UNICODE`; Linux wlroots via a dynamically-grown Unicode keymap on a dedicated
/// virtual keyboard). A multi-character commit is consecutive events in order. Sent only when
/// the host advertised [`HOST_CAP_TEXT_INPUT`](crate::quic::HOST_CAP_TEXT_INPUT) — toward an
/// older host (or one whose inject backend can't type text) clients keep the best-effort VK
/// synthesis, and an older host ignores the unknown tag entirely.
TextInput = 15,
}
/// Pack a [`InputKind::GamepadRemove`] `flags` word (`seq << 24 | pad`) — the same low-byte-pad /
@@ -169,7 +158,6 @@ impl InputKind {
12 => GamepadState,
13 => GamepadRemove,
14 => GamepadArrival,
15 => TextInput,
_ => return None,
})
}
@@ -404,27 +392,10 @@ mod tests {
};
assert_eq!(InputEvent::decode(&e.encode()), Some(e));
}
// GamepadRemove/GamepadArrival/TextInput are valid kinds; 16 (one past them) is not.
// GamepadRemove + GamepadArrival are valid kinds; 15 (one past them) is not.
assert_eq!(InputKind::from_u8(13), Some(InputKind::GamepadRemove));
assert_eq!(InputKind::from_u8(14), Some(InputKind::GamepadArrival));
assert_eq!(InputKind::from_u8(15), Some(InputKind::TextInput));
assert_eq!(InputKind::from_u8(16), None);
}
#[test]
fn text_input_roundtrip() {
// One Unicode scalar per event — BMP and astral (emoji) alike.
for cp in ['a' as u32, 'ß' as u32, '語' as u32, 0x1F600 /* 😀 */] {
let e = InputEvent {
kind: InputKind::TextInput,
_pad: [0; 3],
code: cp,
x: 0,
y: 0,
flags: 0,
};
assert_eq!(InputEvent::decode(&e.encode()), Some(e));
}
assert_eq!(InputKind::from_u8(15), None);
}
#[test]
+1 -8
View File
@@ -102,14 +102,7 @@ pub use stats::Stats;
/// v10: added `punktfunk_connection_clock_offset_now_ns` — the LIVE (mid-stream re-synced)
/// clock offset ongoing latency math must use; the connect-time getter stays frozen by
/// contract. Additive, client-local — no wire change, so [`WIRE_VERSION`] is unchanged.
/// v12: added `punktfunk_connection_set_cursor_render` — the mid-stream cursor-render flip
/// (design/remote-desktop-sweep.md §8): the client's mouse-model chord tells the host who
/// renders the pointer. Additive; rides the existing control stream (a new message TYPE, which
/// pre-§8 hosts ignore), so [`WIRE_VERSION`] is unchanged.
/// v13: added `punktfunk_connection_send_pen` — the stylus wire plane
/// (design/pen-tablet-input.md): a client sends `RICH_PEN` sample batches once the host
/// advertises `HOST_CAP_PEN`. Additive and capability-gated, so [`WIRE_VERSION`] is unchanged.
pub const ABI_VERSION: u32 = 13;
pub const ABI_VERSION: u32 = 10;
/// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
/// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
-41
View File
@@ -71,47 +71,6 @@ pub const HOST_CAP_GAMEPAD_STATE: u8 = 0x01;
/// trailing `host_caps` byte — no wire-layout change.
pub const HOST_CAP_CLIPBOARD: u8 = 0x02;
/// [`Welcome::host_caps`] bit: the host's active inject backend can type **committed text**
/// ([`InputKind::TextInput`](crate::input::InputKind::TextInput) — one Unicode scalar per event):
/// Windows (`KEYEVENTF_UNICODE`) and Linux wlroots (dynamic Unicode keymap on a dedicated virtual
/// keyboard); the KWin/libei/gamescope backends can only press layout keycodes, so those sessions
/// don't set it. A capable client routes its IME's committed text (autocorrect, gesture typing,
/// non-Latin scripts, emoji) through `TextInput` instead of lossy VK synthesis; absent the bit it
/// keeps the VK fallback. Packs into the existing trailing `host_caps` byte — no wire-layout
/// change; an older host ignores the unknown input tag anyway (input is lossy by design).
pub const HOST_CAP_TEXT_INPUT: u8 = 0x04;
/// [`Hello::client_caps`] bit: the client renders the host cursor LOCALLY
/// (design/remote-desktop-sweep.md M2). It consumes [`CursorShape`](super::control::CursorShape)
/// control messages (RGBA bitmap + hotspot, cached by serial) and per-frame
/// [`CursorState`](super::datagram::CursorState) `0xD0` datagrams (position/visibility), and
/// draws the pointer itself — so the host must STOP compositing the cursor into the video
/// (`SessionPlan.cursor_blend = false`) or the user sees it twice. Active only when the host
/// answers with [`HOST_CAP_CURSOR`] (capable-and-agreed, the 444/clipboard precedent); toward
/// an older or incapable host nothing changes.
pub const CLIENT_CAP_CURSOR: u8 = 0x01;
/// [`Welcome::host_caps`] bit: the host CAN forward the cursor out-of-band (it captures cursor
/// metadata separately from the frame — the Linux portal `SPA_META_Cursor` path; NOT gamescope,
/// whose capture carries no cursor, and NOT Windows yet, where DWM composites into the IDD
/// frame). Set only when the client asked via [`CLIENT_CAP_CURSOR`]; when both bits agree the
/// host stops blending and ships [`CursorShape`](super::control::CursorShape) +
/// [`CursorState`](super::datagram::CursorState) instead. `0x08` — `0x04` is
/// [`HOST_CAP_TEXT_INPUT`], `0x01`/`0x02` are gamepad-state / clipboard.
pub const HOST_CAP_CURSOR: u8 = 0x08;
/// [`Welcome::host_caps`] bit: the host injects full-fidelity stylus input — it routes
/// [`PenBatch`](super::pen::PenBatch) `0xCC/0x05` datagrams (pressure, tilt, azimuth, barrel
/// roll, hover, eraser, barrel buttons) through the [`PenTracker`](super::pen::PenTracker)
/// into a virtual tablet device (design/pen-tablet-input.md). A capable client (Apple Pencil,
/// Android stylus) then splits pen contacts out of its finger/touch path and sends pen
/// batches; absent the bit it keeps folding the pen into touch/pointer like today, and
/// [`NativeClient::send_pen`](crate::client::NativeClient::send_pen) refuses to send. The
/// wire ships ahead of the backend (P0): no host sets this bit until the P1 injector lands —
/// which is exactly why the gate exists. `0x10` — `0x08` is [`HOST_CAP_CURSOR`], `0x04` is
/// [`HOST_CAP_TEXT_INPUT`], `0x01`/`0x02` are gamepad-state / clipboard.
pub const HOST_CAP_PEN: u8 = 0x10;
/// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software**
/// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST
/// advertise this.
-173
View File
@@ -783,146 +783,11 @@ impl ClipFetchHdr {
}
}
// --- Cursor channel (design/remote-desktop-sweep.md M2) --------------------------------------
// The host cursor, forwarded out-of-band so the CLIENT draws it as a real OS cursor (the
// Parsec/RDP model) instead of paying the video round-trip. Shape (rare, needs reliability)
// rides here on the control stream; per-frame position/visibility rides the lossy `0xD0`
// datagram plane ([`super::datagram::CursorState`]). Active only when the client's
// [`CLIENT_CAP_CURSOR`](super::caps::CLIENT_CAP_CURSOR) met the host's
// [`HOST_CAP_CURSOR`](super::caps::HOST_CAP_CURSOR) — the host stops compositing then.
// ---------------------------------------------------------------------------------------------
/// Type byte of [`CursorShape`] (host → client): the pointer's bitmap + hotspot changed.
pub const MSG_CURSOR_SHAPE: u8 = 0x50;
/// Type byte of [`CursorRenderMode`] (client → host): who renders the pointer right now.
pub const MSG_CURSOR_RENDER: u8 = 0x51;
/// Per-side pixel cap for a forwarded cursor bitmap. The control-stream frame is length-prefixed
/// with a `u16`, so a whole message must fit 65535 bytes — 128×128 RGBA (65536 B) already
/// overshoots before the 17-byte header. 120² (57.6 KiB + header) fits with headroom and covers
/// real cursors (typically ≤ 64 px, ≤ 96 px at HiDPI scale); the HOST downscales anything
/// larger before forwarding, so the cap is invisible to clients.
pub const CURSOR_SHAPE_MAX_SIDE: u16 = 120;
/// `host → client` ([`MSG_CURSOR_SHAPE`]): one cursor shape, sent when the pointer's bitmap
/// changes (never per-frame — [`super::datagram::CursorState`] carries the motion). The client
/// caches shapes by `serial` and re-installs a cached one without any bitmap crossing again
/// (the RDP pointer-cache idea for free: re-showing a known serial is a 14-byte
/// [`super::datagram::CursorState`], not a resend).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CursorShape {
/// Bitmap identity — bumped by the host's capture layer only on shape change; position
/// moves keep the serial stable. [`super::datagram::CursorState::serial`] references it.
pub serial: u32,
/// Bitmap dimensions in pixels, `1..=`[`CURSOR_SHAPE_MAX_SIDE`] each.
pub w: u16,
pub h: u16,
/// Hotspot (the pixel that IS the pointer position), within `w`×`h`.
pub hot_x: u16,
pub hot_y: u16,
/// Straight-alpha RGBA8, exactly `w * h * 4` bytes, no padding.
pub rgba: Vec<u8>,
}
impl CursorShape {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] serial[5..9] w[9..11] h[11..13] hot_x[13..15] hot_y[15..17] rgba…
let mut b = Vec::with_capacity(17 + self.rgba.len());
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CURSOR_SHAPE);
b.extend_from_slice(&self.serial.to_le_bytes());
b.extend_from_slice(&self.w.to_le_bytes());
b.extend_from_slice(&self.h.to_le_bytes());
b.extend_from_slice(&self.hot_x.to_le_bytes());
b.extend_from_slice(&self.hot_y.to_le_bytes());
b.extend_from_slice(&self.rgba);
b
}
pub fn decode(b: &[u8]) -> Result<CursorShape> {
if b.len() < 17 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CURSOR_SHAPE {
return Err(PunktfunkError::InvalidArg("bad CursorShape"));
}
let u16at = |o: usize| u16::from_le_bytes([b[o], b[o + 1]]);
let (w, h) = (u16at(9), u16at(11));
if w == 0 || h == 0 || w > CURSOR_SHAPE_MAX_SIDE || h > CURSOR_SHAPE_MAX_SIDE {
return Err(PunktfunkError::InvalidArg("bad CursorShape dims"));
}
if b.len() != 17 + (w as usize) * (h as usize) * 4 {
return Err(PunktfunkError::InvalidArg("bad CursorShape len"));
}
Ok(CursorShape {
serial: u32::from_le_bytes(b[5..9].try_into().unwrap()),
w,
h,
hot_x: u16at(13),
hot_y: u16at(15),
rgba: b[17..].to_vec(),
})
}
}
/// `client → host` ([`MSG_CURSOR_RENDER`]): who renders the pointer, switched live by the
/// client's mouse-model flip (⌃⌥⇧M — design/remote-desktop-sweep.md §8). `client_draws: true`
/// = the DESKTOP model: the host EXCLUDES the pointer from the video and forwards
/// shape/state ([`CursorShape`]/`0xD0`) for the client's local OS cursor. `false` = the
/// CAPTURE model: the host COMPOSITES the pointer into the video exactly as a
/// channel-less session would (DWM / encoder blend — full fidelity incl. XOR inversion)
/// and the forwarder goes quiet. Sessions that negotiated the cursor cap start in
/// `client_draws: true` (the pre-message behavior) until told otherwise; the message is
/// idempotent and latest-wins.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct CursorRenderMode {
pub client_draws: bool,
}
impl CursorRenderMode {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] client_draws[5]
let mut b = Vec::with_capacity(6);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CURSOR_RENDER);
b.push(self.client_draws as u8);
b
}
pub fn decode(b: &[u8]) -> Result<CursorRenderMode> {
if b.len() != 6 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CURSOR_RENDER {
return Err(PunktfunkError::InvalidArg("bad CursorRenderMode"));
}
Ok(CursorRenderMode {
client_draws: b[5] != 0,
})
}
}
#[cfg(test)]
mod tests {
use crate::config::Mode;
use crate::quic::*;
#[test]
fn cursor_render_mode_roundtrip() {
for client_draws in [true, false] {
let m = CursorRenderMode { client_draws };
assert_eq!(CursorRenderMode::decode(&m.encode()).unwrap(), m);
}
// Type byte separates it from the shape message (and vice versa).
assert!(CursorRenderMode::decode(
&CursorShape {
serial: 1,
w: 1,
h: 1,
hot_x: 0,
hot_y: 0,
rgba: vec![0; 4]
}
.encode()
)
.is_err());
}
#[test]
fn reconfigure_roundtrip() {
let rq = Reconfigure {
@@ -1282,42 +1147,4 @@ mod tests {
assert!(ClipFetchHdr::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
assert!(ClipFetchHdr::decode(&bytes[..bytes.len() - 1]).is_err());
}
#[test]
fn cursor_shape_roundtrip() {
let s = CursorShape {
serial: 7,
w: 2,
h: 3,
hot_x: 1,
hot_y: 2,
rgba: (0..2 * 3 * 4).map(|i| i as u8).collect(),
};
assert_eq!(CursorShape::decode(&s.encode()).unwrap(), s);
// Max-side shape still fits the u16 control frame with headroom.
let side = CURSOR_SHAPE_MAX_SIDE;
let big = CursorShape {
serial: u32::MAX,
w: side,
h: side,
hot_x: side - 1,
hot_y: 0,
rgba: vec![0xAB; side as usize * side as usize * 4],
};
let bytes = big.encode();
assert!(bytes.len() <= u16::MAX as usize, "must fit a control frame");
assert_eq!(CursorShape::decode(&bytes).unwrap(), big);
// Rejections: zero / oversize dims, and a length that disagrees with them.
let mut zero = s.encode();
zero[9] = 0;
zero[10] = 0;
assert!(CursorShape::decode(&zero).is_err());
let mut oversize = s.encode();
oversize[9..11].copy_from_slice(&(CURSOR_SHAPE_MAX_SIDE + 1).to_le_bytes());
assert!(CursorShape::decode(&oversize).is_err());
let mut short = s.encode();
short.pop();
assert!(CursorShape::decode(&short).is_err());
// Distinct from the neighboring vocabulary.
assert!(ClipState::decode(&s.encode()).is_err());
}
}

Some files were not shown because too many files have changed in this diff Show More