punktfunk/design/controller-only-mode.md

# Controller-only mode (Deck / desktop as a remote gamepad)

> **Status:** **DESIGN — not yet implemented.** Locked decisions (2026-06-29): build the
> **full-fidelity** path directly (no plain-Xbox interim), capture this as a doc before code.
> This is the **session-shape** complement to `design/steam-controller-deck-support.md` (which is
> the **input-fidelity** work: capturing the Deck's paddles/trackpads/gyro and injecting a virtual
> `hid-steam`/DualSense device). Controller-only mode reuses that capture + inject pipeline
> verbatim; it only adds a negotiated "no video / no audio" session shape so the Deck can be a
> wireless gamepad for a PC **without** the wasteful return video stream.

## 1. Goal + the use case

Let a punktfunk client (a Steam Deck, but also any desktop with a controller) connect to a
punktfunk host and forward **only controller input** — no video stream, no audio stream. The host
user watches their **own** monitor; the Deck is just a wireless, full-fidelity gamepad. Rumble /
lightbar / adaptive-trigger / HID feedback still flows back on the existing side planes.

Concretely this turns the Deck into:

- a **couch controller** for a PC wired to a TV — gyro aim, both trackpads, the 4 back grips, with
  **lower** latency than streaming (no encode/decode round-trip at all), and no bandwidth wasted on
  a video feed you aren't looking at;
- one of **several** Decks/pads driving one shared-screen PC for local co-op (rides the multi-pad
  work already in flight);
- a way to use the Deck's superior input surface (trackpads + gyro) on a game running on a
  beefier host.

**Non-goal (for v1):** forwarding a real keyboard/mouse. The Deck's trackpads/gyro are carried as
*gamepad* fidelity (DualSense/Steam touchpad + motion planes), which is display-independent (§5).
A genuine keyboard/mouse forward rides the libei/portal pointer path, which is *not*
display-independent — deferred to a follow-on (§9).

## 2. Does SteamOS already do this? — No, and that's the opening

Researched 2026-06-29 (web). Summary of why this is worth building:

- **Officially**, Valve's only endorsed "Deck as a controller for another PC" path is **Steam
  Remote Play / Steam Link in reverse** — but it **always streams the game's video to the Deck**
  even though you're looking at the PC's monitor. A dedicated controller-only mode has been
  requested since **July 2022** (Steam community thread) and the matching `ValveSoftware/SteamOS`
  issue **#1623 is "Closed as not planned."** SteamOS 3.8 (Jun 2026) and the new standalone Steam
  Controller (2026 hardware) did **not** add it.
- **Community/DIY** splits three ways, all low-popularity (25–129★), several stale or "currently
  broken": USB-C HID gadget (**GadgetDeck** — wired-only, BIOS Dual-Role toggle, no gyro/trackpad,
  unmaintained), Bluetooth HID (**steamdeck-bt-controller-emulator** — pairing/recognition pain + BT
  latency), and network (**Deckpad** → commercial *paid* VirtualHere USB-over-IP, "currently
  broken"; **swicd-remote-gamepad** → ViGEm, Windows-only, experimental).
- **The ceiling every existing tool hits:** none cleanly carries the Deck's **gyro + dual trackpads
  + the 4 back buttons (L4/L5/R4/R5)** to the remote host, because Steam's emulated Xbox pad
  (`28DE:11FF`) hides them.

punktfunk is uniquely positioned: it already has a low-latency QUIC input back-channel, host-side
virtual Xbox-360 / DualSense / DS4 (and, in flight, `hid-steam`) pad injection with rumble/lightbar/
adaptive-trigger feedback, and SDL3 capture. Controller-only mode + the
`steam-controller-deck-support.md` capture work together deliver exactly the gap nobody else fills.

Sources (load-bearing): SteamOS issue #1623 (closed not-planned)
<https://github.com/ValveSoftware/SteamOS/issues/1623>; Steam community controller-only-mode request
<https://steamcommunity.com/app/1675200/discussions/2/3466100515592011642/>; Valve FAQ
<https://help.steampowered.com/en/faqs/view/0689-74B8-92AC-10F2>; GadgetDeck
<https://github.com/Frederic98/GadgetDeck>; Deckpad <https://github.com/HelloThisIsFlo/Deckpad>;
Steam Deck HID deep-dive <https://blogs.gnome.org/alicem/2024/10/24/steam-deck-hid-and-libmanette-adventures/>.

## 3. The key synergy: controller-only mode *dissolves* the Game-Mode capture wall

`steam-controller-deck-support.md` §6 / Wall A: on the Deck, SDL3's HIDAPI driver can open the raw
`28DE:1205` and expose paddles + both trackpads + gyro as a first-class SDL gamepad — **but in Deck
Game Mode, Steam Input grabs the device exclusively** and re-presents it as the gutted `28DE:11FF`
virtual XInput pad, so the rich controls silently vanish. The only escape there is the
disable-Steam-Input-per-title UX.

**Controller-only mode's natural launch context avoids that wall entirely.** The use case is "the
Deck is a controller, no game runs on the Deck" → it runs as a **desktop-mode / standalone app**,
where Steam Input is **not** managing the internal pad, so SDL3 binds `28DE:1205` and gets full
fidelity with no UX gymnastics. So the two features are mutually reinforcing: controller-only mode
is the very scenario in which full-fidelity Deck capture "just works."

The capture-side rule is therefore the same one §6 documents, and the client **must verify at
runtime it opened `28DE:1205` (HIDAPI GUID ending `6800`), not `28DE:11FF`** — if it only sees
`11FF`, Steam owns the pad and gyro/trackpad/grips are unavailable; surface that to the user.

## 4. Architecture — input plane is already decoupled from video

A punktfunk/1 native session already runs **two independent transports** (verified in-tree):

- a **QUIC** control connection that carries the `Hello`/`Welcome`/`Start` handshake **and every
  side plane** as datagrams demuxed by first byte: input `0xC8`, rich input `0xCC` (DualSense/Deck
  touchpad + motion), mic uplink `0xCB`, audio `0xC9`, rumble `0xCA`, HID-out `0xCD`;
- a **raw-UDP data plane** (`Session`, FEC + AES-GCM) that carries **only** the video AUs.

**Input never touches the UDP data plane** — it rides QUIC datagrams. So an input-only session is
"run the QUIC handshake + side planes, never bind/open the UDP data plane." The honest work is
making **both** ends *skip the data plane* and, on the host, *not spin up a virtual display +
encoder* (a desktop PC the operator is watching has no reason to allocate a headless virtual output
or burn an NVENC slot).

Critically: **gamepads are system-global kernel devices, not tied to any virtual output.** The
per-session `PadBackend` (`punktfunk1.rs:1396`) creates an Xbox-360 pad on `/dev/uinput`, or a
DualSense/DS4/Steam pad on `/dev/uhid` — all visible to Steam/Proton/every game on the host's real
seat with **zero** display involvement. Rumble/HID feedback (`0xCA`/`0xCD`) and DualSense/Deck
touchpad+motion (`0xCC` rich input, `apply_rich` at `:1467`/`:1606`) flow on the same per-session
input thread, also display-independent. So a controller-only session needs **no virtual display, no
compositor, no portal grant, no encoder** — just the input thread + the pad backend.

`clients/probe --input-test` already proves the shape: it connects, streams scripted gamepad
datagrams the host injects into a real pad, and never decodes video.

## 5. Protocol / ABI change — one `session_flags` byte (additive, fwd-compatible)

Reuse the **exact** trailing-byte back-compat discipline `Hello`/`Welcome` already apply to
`compositor`/`gamepad`/`video_caps`/`audio_channels` (`quic.rs:655-882`). Add a **session-flags**
byte as the new last trailing field on both.

```
quic.rs (core):
  pub const SESSION_INPUT_ONLY: u8 = 0x01;   // bit 0 of session_flags

  Hello   { …, audio_channels: u8, session_flags: u8 }   // new trailing byte after audio_channels
  Welcome { …, audio_channels: u8, session_flags: u8 }   // echoes the resolved shape (offset 66)
```

### 5.1 Encode (placeholder discipline)

`Hello::encode` already emits `video_caps` / `audio_channels` only when non-default, emitting
upstream placeholders so each lands at a deterministic offset. Extend the same logic:

```
need_placeholders = video_caps != 0 || audio_channels != 2 || session_flags != 0;
// video_caps emitted when video_caps != 0 || audio_channels != 2 || session_flags != 0
// audio_channels emitted when audio_channels != 2 || session_flags != 0
// session_flags emitted when session_flags != 0   (one more trailing byte, after audio_channels)
```

`Hello::decode` reads it one past `audio_channels` (i.e. `video_caps_off + 2`), defaulting to `0`
(no flags) when absent → an older peer requests an ordinary video session, byte-identical wire.
`Welcome` is simpler (fixed-position trailing bytes): append `session_flags` at **offset 66**,
`b.get(66).copied().unwrap_or(0)`.

> **Why a flags byte, not an overloaded `Mode{0,0,0}` sentinel:** a flag is explicit and leaves
> room for future session-shape bits (e.g. audio-only, input+audio). `Mode` stays strictly a
> display mode. (Open question 7.1 — confirm with the user, but this is the recommendation.)

### 5.2 ABI

- New `connect_ex` rung in the existing ladder (the `connect_exN` precedent) that takes a
  `session_flags` (or a bool `input_only`) and stores it on `NativeClient`; legacy `connect_ex*`
  stay byte-for-byte. Regenerate `include/punktfunk_core.h` (CI fails on drift).
- New constant `PUNKTFUNK_SESSION_INPUT_ONLY = 0x01`.
- `next_au` / `next_frame` / `next_audio` simply return `NoFrame`/`Closed` in an input-only
  connection; `send_input` / `send_rich_input` / `next_rumble` / `next_hidout` are unchanged — they
  are already the full input-only surface.

## 6. Host changes — `serve_session` branch (`punktfunk1.rs:508`)

Branch on `hello.session_flags & SESSION_INPUT_ONLY`. When set:

| Step | Today (video session) | Input-only |
|---|---|---|
| `--max-concurrent` permit (`:640` `_permit`) | acquired (NVENC slot) | **skip** — input-only must not consume a GPU slot (§7.4) |
| `validate_dimensions` (`:654`) | required | **skip** |
| `resolve_compositor` (`:668`) | required (Virtual source) | **skip** — no virtual display |
| bit-depth / chroma / HDR / 444 probes, bitrate clamp | run | **skip** |
| `Welcome` (`:794`) | real mode + udp_port + caps | **sentinel** mode `{0,0,0}` + `udp_port=0` + `session_flags=INPUT_ONLY`; still carries the resolved `gamepad` backend |
| `Start::decode` (`:845`) | read client udp port | read + **ignore** (harmless no-op) |
| `input_thread` spawn (`:980`) | spawn | **spawn (unchanged)** — this is the whole point |
| client→host datagram demux (`:989`) | spawn | **spawn (unchanged)** — `0xC8`/`0xCC`/`0xCB` in, `0xCA`/`0xCD` out |
| `audio_thread` (`:1032`, already gated on `source==Virtual`) | spawn | **skip** (add `&& !input_only`) |
| `virtual_stream(SessionContext{…})` (`:1155`) — the only place a display+encoder open and the UDP socket binds | run | **replace** with `await stop / conn.closed()` — no UDP bind happens (the bind lives inside that block), no display, no encoder |
| teardown (`:1190+`) | releases input thread + pads + held keys | **unchanged** — already correct |

Net host change is mostly **guards and deletions in one function**; no hot-path, FEC, crypto, or
injector changes. `cfg`-clean on Windows (no compositor/`virtual_stream` there either; the XUSB/UMDF
pads are likewise system-global, SendInput is irrelevant in a gamepad-only mode).

### 6.1 Pad backend / fidelity (reuses `steam-controller-deck-support.md` verbatim)

The full-fidelity path is **entirely** the in-flight Deck/DualSense inject work — controller-only
mode adds nothing here, it just runs it without video:

- Deck → host resolves the **Steam `hid-steam`** backend (Linux UHID) when available so Steam Input
  re-emits `28DE:11FF` with the user's bindings + correct glyphs; trackpads → `RichInput::TouchpadEx`
  (`0xCC 0x03`), gyro/accel → `RichInput::Motion` (`0xCC 0x02`), back grips → `BTN_PADDLE1..4`
  (`0xC8` bits). See that doc §4–§7.
- Where a real Steam pad is unavailable, the **DualSense remap fallback** (`steam_remap.rs`) folds
  Steam-only inputs into a virtual DualSense (gyro→motion, right pad→touchpad, grips→configured
  fallback) so nothing is silently dropped.
- `GamepadPref` resolution policy is that doc's §7 — **unchanged**; the Welcome echoes the real
  resolved backend (honest downgrade).

## 7. Client changes

### 7.1 Core connector (`client.rs::worker_main:714`)

Thread an `input_only` flag in. When set: do the full `Hello`/`Welcome` handshake and spawn the
input/mic/rich/ctrl/datagram-demux tasks (so `send_input`/`send_rich_input` + `next_rumble`/
`next_hidout` keep working), but **skip** the UDP port reservation + `Start`-derived
`UdpTransport::connect` + `Session` + the data-plane pump (`:810-849,1041`). `next_frame`/
`next_audio` return `Closed`/`NoFrame`.

### 7.2 Deck / Linux client (`clients/linux`) — a "Use as controller" path

Add a connect path that opens the connection **input-only** and runs **only** the app-lifetime
`GamepadService` (`clients/linux/src/gamepad.rs`) — it already `attach`es the connector, forwards
pads via `send_input` (`:161`), DualSense/Deck touchpad+motion via `send_rich_input`, and drains
`next_rumble` (`:566`)/`next_hidout` (`:583`). **Do not** run `session.rs`'s video/audio pump
(`:135-200`) — no decoder, no video window, no PipeWire player. UI is a minimal "Connected as
controller — <backend> · <host>" status surface (battery, latency, a Disconnect button), no video
widget.

On the Deck specifically: set `SDL_JOYSTICK_HIDAPI_STEAMDECK`/`HIDAPI_STEAM` before `sdl3::init()`,
resolve `GamepadPref::SteamDeck` from VID `0x28DE`, and **assert the opened device is `28DE:1205`
(HIDAPI GUID `…6800`), not `11FF`** — if `11FF`, show "Steam is managing this controller; full
gyro/trackpad/grip fidelity needs desktop mode / Steam Input off." (Capture mechanics =
`steam-controller-deck-support.md` §6.)

### 7.3 Other clients

`clients/windows` gets the same input-only connect + SDL `GamepadService`-only path (XUSB/UMDF pads
are system-global; no SwapChainPanel/decode). Apple/Android: parity is optional and lower-priority —
the connector ABI already exposes everything; a "controller-only" UI mode is a small add once the
core flag lands. Scope per the user's roadmap, not blocking.

### 7.4 Session lifetime / accounting

An input-only session is long-lived (until disconnect), like a video session, but must **not**
count against `--max-concurrent` (it holds no NVENC) and should be **exempt** from any `--seconds`
duration cap. The mgmt API / web console should list it distinctly (it is **not** a "stream" — no
fps/bitrate/mode), and `--max-sessions` accounting should likely treat it as its own class
(open question 8.x). mDNS advertisement is unchanged (the host already advertises native service;
input-only is a per-session negotiation, not a service variant).

## 8. Security / trust — unchanged

A controller-only session is a **full punktfunk/1 session**: same SPAKE2 PIN pairing / TOFU /
`--require-pairing` gate, same QUIC client-auth + pinned-fingerprint trust. The *only* difference is
the absent data plane. No new attack surface — if anything less (no UDP socket, no FEC reassembler,
no decoder fed attacker bytes). The host still requires `/dev/uinput` (+ `/dev/uhid` for DualSense/
Steam) writable — the documented `input` group + `60-punktfunk.rules` setup.

## 9. Milestones

- **M1 — protocol + ABI:** `session_flags` byte + `SESSION_INPUT_ONLY` on `Hello`/`Welcome` with
  round-trip + old-peer-default unit tests; new `connect_ex` rung; regenerate the C header.
- **M2 — host branch:** `serve_session` input-only path (skip display/encoder/audio/permit, keep
  input thread + pads), `cfg`-clean on Windows. Loopback test: handshake completes, **no UDP data
  plane binds**, gamepad events still inject into a real uinput pad.
- **M3 — Deck/Linux client:** "Use as controller" connect + `GamepadService`-only run; `28DE:1205`
  vs `11FF` runtime check + user messaging.
- **M4 — full fidelity on-glass:** with `steam-controller-deck-support.md`'s Deck capture + inject,
  validate Deck (desktop mode) → host: paddles + both trackpads + gyro reach the host pad, Steam
  Input re-emits with bindings/glyphs, rumble returns. Glass-to-glass input latency vs a wired pad.
- **M5 — Windows client parity + mgmt/web "controller session" surfacing.**
- **M6 (deferred) — keyboard/mouse forward** over the libei/portal path (needs the active-session
  RemoteDesktop grant; not display-independent).

## 10. Risks / open questions

**Open questions (decide with the user):**

1. **`session_flags` byte vs `Mode{0,0,0}` sentinel** for "no video" — recommend the explicit flags
   byte (room for future shapes). *(Recommended; confirm.)*
2. Should an input-only session be **exempt** from `--max-concurrent` and `--seconds`? (Recommend
   yes — it holds no GPU.)
3. Should mgmt/web track controller-only sessions as a distinct class (no fps/bitrate/mode), and
   should `--max-sessions` count them?
4. Deck default backend: **Steam `hid-steam`** (best — Steam Input bindings/glyphs) vs **DualSense**
   (works off-Steam too). Tie to `steam-controller-deck-support.md`'s resolution policy.

**Risks:**

- **Capture wall (inherited):** full fidelity requires SDL to bind `28DE:1205`; if the Deck is in
  Game Mode / Steam owns the pad, it degrades to `11FF` (sticks/buttons only). Mitigated by the
  desktop-mode use case + runtime check + user messaging (§3, §7.2).
- **Host `/dev/uinput`(+`/dev/uhid`) perms** — a normal desktop PC operator must do the one-time
  input-group/udev setup for the virtual pad to appear (already documented).
- **Handshake assumes a data plane:** `Start{client_udp_port}` + non-zero `udp_port` in `Welcome`
  must become harmless no-ops (send 0 / ignore) without shifting the legacy wire — the trailing-byte
  placeholder discipline is fragile; **add round-trip + old-peer tests** (M1).
- **Control-channel messages** (LossReport/Reconfigure/Probe/ClockProbe) assume a data plane; the
  host (`:881`) + client (`:935`) control tasks must tolerate an input-only session with none —
  mostly already fine since those are reactive.
- **Windows parity:** verify the input-only branch compiles `cfg`-clean where there is no
  compositor/`virtual_stream`.

## 11. Validation plan

**Loopback (no hardware):**
- `quic.rs`: `session_flags` encode/decode round-trip; old-peer (no flags byte) → ordinary session;
  flags coexist with non-default `video_caps`/`audio_channels` (placeholder offsets hold).
- Host: an input-only synthetic host+client asserting (a) handshake completes, (b) **no UDP socket
  binds**, (c) no display/encoder opens, (d) scripted `0xC8`/`0xCC` events inject into a real pad,
  (e) `0xCA`/`0xCD` feedback returns. Extend the `clients/probe` / `test-loopback.sh` harness.

**On-box / on-glass (with `steam-controller-deck-support.md` landed):**
- Deck (desktop mode) → this host, controller-only: confirm `28DE:1205` opens (not `11FF`); paddles
  + both trackpads + gyro reach the host pad; Steam Input on the host re-emits with bindings/glyphs;
  rumble round-trips; measure input-only latency vs a wired pad and vs the streaming path.
- Confirm the host opens **no** virtual output / encoder (logs) and the session does **not** consume
  a `--max-concurrent` slot (run alongside N video sessions).