feat(host/steam): shippable usbip/vhci_hcd virtual Deck + client leave-shortcuts
apple / screenshots (push) Has been cancelled
android / android (push) Has been cancelled
apple / swift (push) Has been cancelled
audit / cargo-audit (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
ci / rust (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
flatpak / build-publish (push) Has been cancelled
release / apple (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
windows-host / package (push) Has been cancelled
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Has been cancelled
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Has been cancelled
windows / build (aarch64-pc-windows-msvc) (push) Has been cancelled
windows / build (x86_64-pc-windows-msvc) (push) Has been cancelled

Steam Deck pass-through (design/steam-deck-passthrough-plan.md), code-complete +
all CI checks green on Linux + adversarially reviewed; on-glass validation pending:

- usbip/`vhci_hcd` virtual Deck transport (inject/linux/steam_usbip.rs) for
  non-SteamOS hosts (Bazzite/generic) — presents a real interface-2 USB Deck so
  Steam Input promotes it. In-process vhci attach (loopback OP_REQ_IMPORT handshake
  → sysfs attach) with a bounded `usbip`-CLI fallback; detach on drop.
- Backed by a vendored, libusb-free trim of the `usbip` crate
  (crates/punktfunk-host/vendor/usbip-sim, MIT + NOTICE; host/cdc/hid + rusb/nusb
  removed; interrupt-IN paced by bInterval).
- Selection ladder raw_gadget (SteamOS fast-path) → usbip (universal) → UHID,
  with PUNKTFUNK_STEAM_USBIP / PUNKTFUNK_USBIP_ATTACH knobs.
- Shared Deck descriptors + the 0x83/0xAE feature contract + a Steam-accepted
  serial consolidated into steam_proto.rs; the raw_gadget backend reuses them.
- Linux client leave-shortcuts: Ctrl+Alt+Shift+D + holding the escape chord
  (L1+R1+Start+Select) >=1.5s end the session (short press still exits
  fullscreen); the chord state resets across sessions.

Also bundles in-progress work already staged in the tree:
- host(kwin): xdg-output logical-geometry mapping so the KWin fake_input backend
  places absolute coordinates correctly under display scaling.
- docs: design/README index entries + design/controller-only-mode.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-29 19:17:00 +00:00
parent 831b37b4b7
commit 580b1ea7a7
26 changed files with 3292 additions and 145 deletions
+299
View File
@@ -0,0 +1,299 @@
# Controller-only mode (Deck / desktop as a remote gamepad)
> **Status:** **DESIGN — not yet implemented.** Locked decisions (2026-06-29): build the
> **full-fidelity** path directly (no plain-Xbox interim), capture this as a doc before code.
> This is the **session-shape** complement to `design/steam-controller-deck-support.md` (which is
> the **input-fidelity** work: capturing the Deck's paddles/trackpads/gyro and injecting a virtual
> `hid-steam`/DualSense device). Controller-only mode reuses that capture + inject pipeline
> verbatim; it only adds a negotiated "no video / no audio" session shape so the Deck can be a
> wireless gamepad for a PC **without** the wasteful return video stream.
## 1. Goal + the use case
Let a punktfunk client (a Steam Deck, but also any desktop with a controller) connect to a
punktfunk host and forward **only controller input** — no video stream, no audio stream. The host
user watches their **own** monitor; the Deck is just a wireless, full-fidelity gamepad. Rumble /
lightbar / adaptive-trigger / HID feedback still flows back on the existing side planes.
Concretely this turns the Deck into:
- a **couch controller** for a PC wired to a TV — gyro aim, both trackpads, the 4 back grips, with
**lower** latency than streaming (no encode/decode round-trip at all), and no bandwidth wasted on
a video feed you aren't looking at;
- one of **several** Decks/pads driving one shared-screen PC for local co-op (rides the multi-pad
work already in flight);
- a way to use the Deck's superior input surface (trackpads + gyro) on a game running on a
beefier host.
**Non-goal (for v1):** forwarding a real keyboard/mouse. The Deck's trackpads/gyro are carried as
*gamepad* fidelity (DualSense/Steam touchpad + motion planes), which is display-independent (§5).
A genuine keyboard/mouse forward rides the libei/portal pointer path, which is *not*
display-independent — deferred to a follow-on (§9).
## 2. Does SteamOS already do this? — No, and that's the opening
Researched 2026-06-29 (web). Summary of why this is worth building:
- **Officially**, Valve's only endorsed "Deck as a controller for another PC" path is **Steam
Remote Play / Steam Link in reverse** — but it **always streams the game's video to the Deck**
even though you're looking at the PC's monitor. A dedicated controller-only mode has been
requested since **July 2022** (Steam community thread) and the matching `ValveSoftware/SteamOS`
issue **#1623 is "Closed as not planned."** SteamOS 3.8 (Jun 2026) and the new standalone Steam
Controller (2026 hardware) did **not** add it.
- **Community/DIY** splits three ways, all low-popularity (25129★), several stale or "currently
broken": USB-C HID gadget (**GadgetDeck** — wired-only, BIOS Dual-Role toggle, no gyro/trackpad,
unmaintained), Bluetooth HID (**steamdeck-bt-controller-emulator** — pairing/recognition pain + BT
latency), and network (**Deckpad** → commercial *paid* VirtualHere USB-over-IP, "currently
broken"; **swicd-remote-gamepad** → ViGEm, Windows-only, experimental).
- **The ceiling every existing tool hits:** none cleanly carries the Deck's **gyro + dual trackpads
+ the 4 back buttons (L4/L5/R4/R5)** to the remote host, because Steam's emulated Xbox pad
(`28DE:11FF`) hides them.
punktfunk is uniquely positioned: it already has a low-latency QUIC input back-channel, host-side
virtual Xbox-360 / DualSense / DS4 (and, in flight, `hid-steam`) pad injection with rumble/lightbar/
adaptive-trigger feedback, and SDL3 capture. Controller-only mode + the
`steam-controller-deck-support.md` capture work together deliver exactly the gap nobody else fills.
Sources (load-bearing): SteamOS issue #1623 (closed not-planned)
<https://github.com/ValveSoftware/SteamOS/issues/1623>; Steam community controller-only-mode request
<https://steamcommunity.com/app/1675200/discussions/2/3466100515592011642/>; Valve FAQ
<https://help.steampowered.com/en/faqs/view/0689-74B8-92AC-10F2>; GadgetDeck
<https://github.com/Frederic98/GadgetDeck>; Deckpad <https://github.com/HelloThisIsFlo/Deckpad>;
Steam Deck HID deep-dive <https://blogs.gnome.org/alicem/2024/10/24/steam-deck-hid-and-libmanette-adventures/>.
## 3. The key synergy: controller-only mode *dissolves* the Game-Mode capture wall
`steam-controller-deck-support.md` §6 / Wall A: on the Deck, SDL3's HIDAPI driver can open the raw
`28DE:1205` and expose paddles + both trackpads + gyro as a first-class SDL gamepad — **but in Deck
Game Mode, Steam Input grabs the device exclusively** and re-presents it as the gutted `28DE:11FF`
virtual XInput pad, so the rich controls silently vanish. The only escape there is the
disable-Steam-Input-per-title UX.
**Controller-only mode's natural launch context avoids that wall entirely.** The use case is "the
Deck is a controller, no game runs on the Deck" → it runs as a **desktop-mode / standalone app**,
where Steam Input is **not** managing the internal pad, so SDL3 binds `28DE:1205` and gets full
fidelity with no UX gymnastics. So the two features are mutually reinforcing: controller-only mode
is the very scenario in which full-fidelity Deck capture "just works."
The capture-side rule is therefore the same one §6 documents, and the client **must verify at
runtime it opened `28DE:1205` (HIDAPI GUID ending `6800`), not `28DE:11FF`** — if it only sees
`11FF`, Steam owns the pad and gyro/trackpad/grips are unavailable; surface that to the user.
## 4. Architecture — input plane is already decoupled from video
A punktfunk/1 native session already runs **two independent transports** (verified in-tree):
- a **QUIC** control connection that carries the `Hello`/`Welcome`/`Start` handshake **and every
side plane** as datagrams demuxed by first byte: input `0xC8`, rich input `0xCC` (DualSense/Deck
touchpad + motion), mic uplink `0xCB`, audio `0xC9`, rumble `0xCA`, HID-out `0xCD`;
- a **raw-UDP data plane** (`Session`, FEC + AES-GCM) that carries **only** the video AUs.
**Input never touches the UDP data plane** — it rides QUIC datagrams. So an input-only session is
"run the QUIC handshake + side planes, never bind/open the UDP data plane." The honest work is
making **both** ends *skip the data plane* and, on the host, *not spin up a virtual display +
encoder* (a desktop PC the operator is watching has no reason to allocate a headless virtual output
or burn an NVENC slot).
Critically: **gamepads are system-global kernel devices, not tied to any virtual output.** The
per-session `PadBackend` (`punktfunk1.rs:1396`) creates an Xbox-360 pad on `/dev/uinput`, or a
DualSense/DS4/Steam pad on `/dev/uhid` — all visible to Steam/Proton/every game on the host's real
seat with **zero** display involvement. Rumble/HID feedback (`0xCA`/`0xCD`) and DualSense/Deck
touchpad+motion (`0xCC` rich input, `apply_rich` at `:1467`/`:1606`) flow on the same per-session
input thread, also display-independent. So a controller-only session needs **no virtual display, no
compositor, no portal grant, no encoder** — just the input thread + the pad backend.
`clients/probe --input-test` already proves the shape: it connects, streams scripted gamepad
datagrams the host injects into a real pad, and never decodes video.
## 5. Protocol / ABI change — one `session_flags` byte (additive, fwd-compatible)
Reuse the **exact** trailing-byte back-compat discipline `Hello`/`Welcome` already apply to
`compositor`/`gamepad`/`video_caps`/`audio_channels` (`quic.rs:655-882`). Add a **session-flags**
byte as the new last trailing field on both.
```
quic.rs (core):
pub const SESSION_INPUT_ONLY: u8 = 0x01; // bit 0 of session_flags
Hello { …, audio_channels: u8, session_flags: u8 } // new trailing byte after audio_channels
Welcome { …, audio_channels: u8, session_flags: u8 } // echoes the resolved shape (offset 66)
```
### 5.1 Encode (placeholder discipline)
`Hello::encode` already emits `video_caps` / `audio_channels` only when non-default, emitting
upstream placeholders so each lands at a deterministic offset. Extend the same logic:
```
need_placeholders = video_caps != 0 || audio_channels != 2 || session_flags != 0;
// video_caps emitted when video_caps != 0 || audio_channels != 2 || session_flags != 0
// audio_channels emitted when audio_channels != 2 || session_flags != 0
// session_flags emitted when session_flags != 0 (one more trailing byte, after audio_channels)
```
`Hello::decode` reads it one past `audio_channels` (i.e. `video_caps_off + 2`), defaulting to `0`
(no flags) when absent → an older peer requests an ordinary video session, byte-identical wire.
`Welcome` is simpler (fixed-position trailing bytes): append `session_flags` at **offset 66**,
`b.get(66).copied().unwrap_or(0)`.
> **Why a flags byte, not an overloaded `Mode{0,0,0}` sentinel:** a flag is explicit and leaves
> room for future session-shape bits (e.g. audio-only, input+audio). `Mode` stays strictly a
> display mode. (Open question 7.1 — confirm with the user, but this is the recommendation.)
### 5.2 ABI
- New `connect_ex` rung in the existing ladder (the `connect_exN` precedent) that takes a
`session_flags` (or a bool `input_only`) and stores it on `NativeClient`; legacy `connect_ex*`
stay byte-for-byte. Regenerate `include/punktfunk_core.h` (CI fails on drift).
- New constant `PUNKTFUNK_SESSION_INPUT_ONLY = 0x01`.
- `next_au` / `next_frame` / `next_audio` simply return `NoFrame`/`Closed` in an input-only
connection; `send_input` / `send_rich_input` / `next_rumble` / `next_hidout` are unchanged — they
are already the full input-only surface.
## 6. Host changes — `serve_session` branch (`punktfunk1.rs:508`)
Branch on `hello.session_flags & SESSION_INPUT_ONLY`. When set:
| Step | Today (video session) | Input-only |
|---|---|---|
| `--max-concurrent` permit (`:640` `_permit`) | acquired (NVENC slot) | **skip** — input-only must not consume a GPU slot (§7.4) |
| `validate_dimensions` (`:654`) | required | **skip** |
| `resolve_compositor` (`:668`) | required (Virtual source) | **skip** — no virtual display |
| bit-depth / chroma / HDR / 444 probes, bitrate clamp | run | **skip** |
| `Welcome` (`:794`) | real mode + udp_port + caps | **sentinel** mode `{0,0,0}` + `udp_port=0` + `session_flags=INPUT_ONLY`; still carries the resolved `gamepad` backend |
| `Start::decode` (`:845`) | read client udp port | read + **ignore** (harmless no-op) |
| `input_thread` spawn (`:980`) | spawn | **spawn (unchanged)** — this is the whole point |
| client→host datagram demux (`:989`) | spawn | **spawn (unchanged)**`0xC8`/`0xCC`/`0xCB` in, `0xCA`/`0xCD` out |
| `audio_thread` (`:1032`, already gated on `source==Virtual`) | spawn | **skip** (add `&& !input_only`) |
| `virtual_stream(SessionContext{…})` (`:1155`) — the only place a display+encoder open and the UDP socket binds | run | **replace** with `await stop / conn.closed()` — no UDP bind happens (the bind lives inside that block), no display, no encoder |
| teardown (`:1190+`) | releases input thread + pads + held keys | **unchanged** — already correct |
Net host change is mostly **guards and deletions in one function**; no hot-path, FEC, crypto, or
injector changes. `cfg`-clean on Windows (no compositor/`virtual_stream` there either; the XUSB/UMDF
pads are likewise system-global, SendInput is irrelevant in a gamepad-only mode).
### 6.1 Pad backend / fidelity (reuses `steam-controller-deck-support.md` verbatim)
The full-fidelity path is **entirely** the in-flight Deck/DualSense inject work — controller-only
mode adds nothing here, it just runs it without video:
- Deck → host resolves the **Steam `hid-steam`** backend (Linux UHID) when available so Steam Input
re-emits `28DE:11FF` with the user's bindings + correct glyphs; trackpads → `RichInput::TouchpadEx`
(`0xCC 0x03`), gyro/accel → `RichInput::Motion` (`0xCC 0x02`), back grips → `BTN_PADDLE1..4`
(`0xC8` bits). See that doc §4–§7.
- Where a real Steam pad is unavailable, the **DualSense remap fallback** (`steam_remap.rs`) folds
Steam-only inputs into a virtual DualSense (gyro→motion, right pad→touchpad, grips→configured
fallback) so nothing is silently dropped.
- `GamepadPref` resolution policy is that doc's §7 — **unchanged**; the Welcome echoes the real
resolved backend (honest downgrade).
## 7. Client changes
### 7.1 Core connector (`client.rs::worker_main:714`)
Thread an `input_only` flag in. When set: do the full `Hello`/`Welcome` handshake and spawn the
input/mic/rich/ctrl/datagram-demux tasks (so `send_input`/`send_rich_input` + `next_rumble`/
`next_hidout` keep working), but **skip** the UDP port reservation + `Start`-derived
`UdpTransport::connect` + `Session` + the data-plane pump (`:810-849,1041`). `next_frame`/
`next_audio` return `Closed`/`NoFrame`.
### 7.2 Deck / Linux client (`clients/linux`) — a "Use as controller" path
Add a connect path that opens the connection **input-only** and runs **only** the app-lifetime
`GamepadService` (`clients/linux/src/gamepad.rs`) — it already `attach`es the connector, forwards
pads via `send_input` (`:161`), DualSense/Deck touchpad+motion via `send_rich_input`, and drains
`next_rumble` (`:566`)/`next_hidout` (`:583`). **Do not** run `session.rs`'s video/audio pump
(`:135-200`) — no decoder, no video window, no PipeWire player. UI is a minimal "Connected as
controller — <backend> · <host>" status surface (battery, latency, a Disconnect button), no video
widget.
On the Deck specifically: set `SDL_JOYSTICK_HIDAPI_STEAMDECK`/`HIDAPI_STEAM` before `sdl3::init()`,
resolve `GamepadPref::SteamDeck` from VID `0x28DE`, and **assert the opened device is `28DE:1205`
(HIDAPI GUID `…6800`), not `11FF`** — if `11FF`, show "Steam is managing this controller; full
gyro/trackpad/grip fidelity needs desktop mode / Steam Input off." (Capture mechanics =
`steam-controller-deck-support.md` §6.)
### 7.3 Other clients
`clients/windows` gets the same input-only connect + SDL `GamepadService`-only path (XUSB/UMDF pads
are system-global; no SwapChainPanel/decode). Apple/Android: parity is optional and lower-priority —
the connector ABI already exposes everything; a "controller-only" UI mode is a small add once the
core flag lands. Scope per the user's roadmap, not blocking.
### 7.4 Session lifetime / accounting
An input-only session is long-lived (until disconnect), like a video session, but must **not**
count against `--max-concurrent` (it holds no NVENC) and should be **exempt** from any `--seconds`
duration cap. The mgmt API / web console should list it distinctly (it is **not** a "stream" — no
fps/bitrate/mode), and `--max-sessions` accounting should likely treat it as its own class
(open question 8.x). mDNS advertisement is unchanged (the host already advertises native service;
input-only is a per-session negotiation, not a service variant).
## 8. Security / trust — unchanged
A controller-only session is a **full punktfunk/1 session**: same SPAKE2 PIN pairing / TOFU /
`--require-pairing` gate, same QUIC client-auth + pinned-fingerprint trust. The *only* difference is
the absent data plane. No new attack surface — if anything less (no UDP socket, no FEC reassembler,
no decoder fed attacker bytes). The host still requires `/dev/uinput` (+ `/dev/uhid` for DualSense/
Steam) writable — the documented `input` group + `60-punktfunk.rules` setup.
## 9. Milestones
- **M1 — protocol + ABI:** `session_flags` byte + `SESSION_INPUT_ONLY` on `Hello`/`Welcome` with
round-trip + old-peer-default unit tests; new `connect_ex` rung; regenerate the C header.
- **M2 — host branch:** `serve_session` input-only path (skip display/encoder/audio/permit, keep
input thread + pads), `cfg`-clean on Windows. Loopback test: handshake completes, **no UDP data
plane binds**, gamepad events still inject into a real uinput pad.
- **M3 — Deck/Linux client:** "Use as controller" connect + `GamepadService`-only run; `28DE:1205`
vs `11FF` runtime check + user messaging.
- **M4 — full fidelity on-glass:** with `steam-controller-deck-support.md`'s Deck capture + inject,
validate Deck (desktop mode) → host: paddles + both trackpads + gyro reach the host pad, Steam
Input re-emits with bindings/glyphs, rumble returns. Glass-to-glass input latency vs a wired pad.
- **M5 — Windows client parity + mgmt/web "controller session" surfacing.**
- **M6 (deferred) — keyboard/mouse forward** over the libei/portal path (needs the active-session
RemoteDesktop grant; not display-independent).
## 10. Risks / open questions
**Open questions (decide with the user):**
1. **`session_flags` byte vs `Mode{0,0,0}` sentinel** for "no video" — recommend the explicit flags
byte (room for future shapes). *(Recommended; confirm.)*
2. Should an input-only session be **exempt** from `--max-concurrent` and `--seconds`? (Recommend
yes — it holds no GPU.)
3. Should mgmt/web track controller-only sessions as a distinct class (no fps/bitrate/mode), and
should `--max-sessions` count them?
4. Deck default backend: **Steam `hid-steam`** (best — Steam Input bindings/glyphs) vs **DualSense**
(works off-Steam too). Tie to `steam-controller-deck-support.md`'s resolution policy.
**Risks:**
- **Capture wall (inherited):** full fidelity requires SDL to bind `28DE:1205`; if the Deck is in
Game Mode / Steam owns the pad, it degrades to `11FF` (sticks/buttons only). Mitigated by the
desktop-mode use case + runtime check + user messaging (§3, §7.2).
- **Host `/dev/uinput`(+`/dev/uhid`) perms** — a normal desktop PC operator must do the one-time
input-group/udev setup for the virtual pad to appear (already documented).
- **Handshake assumes a data plane:** `Start{client_udp_port}` + non-zero `udp_port` in `Welcome`
must become harmless no-ops (send 0 / ignore) without shifting the legacy wire — the trailing-byte
placeholder discipline is fragile; **add round-trip + old-peer tests** (M1).
- **Control-channel messages** (LossReport/Reconfigure/Probe/ClockProbe) assume a data plane; the
host (`:881`) + client (`:935`) control tasks must tolerate an input-only session with none —
mostly already fine since those are reactive.
- **Windows parity:** verify the input-only branch compiles `cfg`-clean where there is no
compositor/`virtual_stream`.
## 11. Validation plan
**Loopback (no hardware):**
- `quic.rs`: `session_flags` encode/decode round-trip; old-peer (no flags byte) → ordinary session;
flags coexist with non-default `video_caps`/`audio_channels` (placeholder offsets hold).
- Host: an input-only synthetic host+client asserting (a) handshake completes, (b) **no UDP socket
binds**, (c) no display/encoder opens, (d) scripted `0xC8`/`0xCC` events inject into a real pad,
(e) `0xCA`/`0xCD` feedback returns. Extend the `clients/probe` / `test-loopback.sh` harness.
**On-box / on-glass (with `steam-controller-deck-support.md` landed):**
- Deck (desktop mode) → this host, controller-only: confirm `28DE:1205` opens (not `11FF`); paddles
+ both trackpads + gyro reach the host pad; Steam Input on the host re-emits with bindings/glyphs;
rumble round-trips; measure input-only latency vs a wired pad and vs the streaming path.
- Confirm the host opens **no** virtual output / encoder (logs) and the session does **not** consume
a `--max-concurrent` slot (run alongside N video sessions).