punktfunk/design/controller-only-mode.md

Controller-only mode (Deck / desktop as a remote gamepad)

Status: DESIGN — not yet implemented. Locked decisions (2026-06-29): build the full-fidelity path directly (no plain-Xbox interim), capture this as a doc before code. This is the session-shape complement to design/steam-controller-deck-support.md (which is the input-fidelity work: capturing the Deck's paddles/trackpads/gyro and injecting a virtual hid-steam/DualSense device). Controller-only mode reuses that capture + inject pipeline verbatim; it only adds a negotiated "no video / no audio" session shape so the Deck can be a wireless gamepad for a PC without the wasteful return video stream.

1. Goal + the use case

Let a punktfunk client (a Steam Deck, but also any desktop with a controller) connect to a punktfunk host and forward only controller input — no video stream, no audio stream. The host user watches their own monitor; the Deck is just a wireless, full-fidelity gamepad. Rumble / lightbar / adaptive-trigger / HID feedback still flows back on the existing side planes.

Concretely this turns the Deck into:

• a couch controller for a PC wired to a TV — gyro aim, both trackpads, the 4 back grips, with lower latency than streaming (no encode/decode round-trip at all), and no bandwidth wasted on a video feed you aren't looking at;
• one of several Decks/pads driving one shared-screen PC for local co-op (rides the multi-pad work already in flight);
• a way to use the Deck's superior input surface (trackpads + gyro) on a game running on a beefier host.

Non-goal (for v1): forwarding a real keyboard/mouse. The Deck's trackpads/gyro are carried as gamepad fidelity (DualSense/Steam touchpad + motion planes), which is display-independent (§5). A genuine keyboard/mouse forward rides the libei/portal pointer path, which is not display-independent — deferred to a follow-on (§9).

2. Does SteamOS already do this? — No, and that's the opening

Researched 2026-06-29 (web). Summary of why this is worth building:

• Officially, Valve's only endorsed "Deck as a controller for another PC" path is Steam Remote Play / Steam Link in reverse — but it always streams the game's video to the Deck even though you're looking at the PC's monitor. A dedicated controller-only mode has been requested since July 2022 (Steam community thread) and the matching ValveSoftware/SteamOS issue #1623 is "Closed as not planned." SteamOS 3.8 (Jun 2026) and the new standalone Steam Controller (2026 hardware) did not add it.
• Community/DIY splits three ways, all low-popularity (25–129★), several stale or "currently broken": USB-C HID gadget (GadgetDeck — wired-only, BIOS Dual-Role toggle, no gyro/trackpad, unmaintained), Bluetooth HID (steamdeck-bt-controller-emulator — pairing/recognition pain + BT latency), and network (Deckpad → commercial paid VirtualHere USB-over-IP, "currently broken"; swicd-remote-gamepad → ViGEm, Windows-only, experimental).
• The ceiling every existing tool hits: none cleanly carries the Deck's **gyro + dual trackpads
  • the 4 back buttons (L4/L5/R4/R5)** to the remote host, because Steam's emulated Xbox pad (28DE:11FF) hides them.

punktfunk is uniquely positioned: it already has a low-latency QUIC input back-channel, host-side virtual Xbox-360 / DualSense / DS4 (and, in flight, hid-steam) pad injection with rumble/lightbar/ adaptive-trigger feedback, and SDL3 capture. Controller-only mode + the steam-controller-deck-support.md capture work together deliver exactly the gap nobody else fills.

Sources (load-bearing): SteamOS issue #1623 (closed not-planned) https://github.com/ValveSoftware/SteamOS/issues/1623; Steam community controller-only-mode request https://steamcommunity.com/app/1675200/discussions/2/3466100515592011642/; Valve FAQ https://help.steampowered.com/en/faqs/view/0689-74B8-92AC-10F2; GadgetDeck https://github.com/Frederic98/GadgetDeck; Deckpad https://github.com/HelloThisIsFlo/Deckpad; Steam Deck HID deep-dive https://blogs.gnome.org/alicem/2024/10/24/steam-deck-hid-and-libmanette-adventures/.

3. The key synergy: controller-only mode dissolves the Game-Mode capture wall

steam-controller-deck-support.md §6 / Wall A: on the Deck, SDL3's HIDAPI driver can open the raw 28DE:1205 and expose paddles + both trackpads + gyro as a first-class SDL gamepad — but in Deck Game Mode, Steam Input grabs the device exclusively and re-presents it as the gutted 28DE:11FF virtual XInput pad, so the rich controls silently vanish. The only escape there is the disable-Steam-Input-per-title UX.

Controller-only mode's natural launch context avoids that wall entirely. The use case is "the Deck is a controller, no game runs on the Deck" → it runs as a desktop-mode / standalone app, where Steam Input is not managing the internal pad, so SDL3 binds 28DE:1205 and gets full fidelity with no UX gymnastics. So the two features are mutually reinforcing: controller-only mode is the very scenario in which full-fidelity Deck capture "just works."

The capture-side rule is therefore the same one §6 documents, and the client must verify at runtime it opened 28DE:1205 (HIDAPI GUID ending 6800), not 28DE:11FF — if it only sees 11FF, Steam owns the pad and gyro/trackpad/grips are unavailable; surface that to the user.

4. Architecture — input plane is already decoupled from video

A punktfunk/1 native session already runs two independent transports (verified in-tree):

• a QUIC control connection that carries the Hello/Welcome/Start handshake and every side plane as datagrams demuxed by first byte: input 0xC8, rich input 0xCC (DualSense/Deck touchpad + motion), mic uplink 0xCB, audio 0xC9, rumble 0xCA, HID-out 0xCD;
• a raw-UDP data plane (Session, FEC + AES-GCM) that carries only the video AUs.

Input never touches the UDP data plane — it rides QUIC datagrams. So an input-only session is "run the QUIC handshake + side planes, never bind/open the UDP data plane." The honest work is making both ends skip the data plane and, on the host, not spin up a virtual display + encoder (a desktop PC the operator is watching has no reason to allocate a headless virtual output or burn an NVENC slot).

Critically: gamepads are system-global kernel devices, not tied to any virtual output. The per-session PadBackend (punktfunk1.rs:1396) creates an Xbox-360 pad on /dev/uinput, or a DualSense/DS4/Steam pad on /dev/uhid — all visible to Steam/Proton/every game on the host's real seat with zero display involvement. Rumble/HID feedback (0xCA/0xCD) and DualSense/Deck touchpad+motion (0xCC rich input, apply_rich at :1467/:1606) flow on the same per-session input thread, also display-independent. So a controller-only session needs no virtual display, no compositor, no portal grant, no encoder — just the input thread + the pad backend.

clients/probe --input-test already proves the shape: it connects, streams scripted gamepad datagrams the host injects into a real pad, and never decodes video.

5. Protocol / ABI change — one session_flags byte (additive, fwd-compatible)

Reuse the exact trailing-byte back-compat discipline Hello/Welcome already apply to compositor/gamepad/video_caps/audio_channels (quic.rs:655-882). Add a session-flags byte as the new last trailing field on both.

quic.rs (core):
  pub const SESSION_INPUT_ONLY: u8 = 0x01;   // bit 0 of session_flags

  Hello   { …, audio_channels: u8, session_flags: u8 }   // new trailing byte after audio_channels
  Welcome { …, audio_channels: u8, session_flags: u8 }   // echoes the resolved shape (offset 66)

5.1 Encode (placeholder discipline)

Hello::encode already emits video_caps / audio_channels only when non-default, emitting upstream placeholders so each lands at a deterministic offset. Extend the same logic:

need_placeholders = video_caps != 0 || audio_channels != 2 || session_flags != 0;
// video_caps emitted when video_caps != 0 || audio_channels != 2 || session_flags != 0
// audio_channels emitted when audio_channels != 2 || session_flags != 0
// session_flags emitted when session_flags != 0   (one more trailing byte, after audio_channels)

Hello::decode reads it one past audio_channels (i.e. video_caps_off + 2), defaulting to 0 (no flags) when absent → an older peer requests an ordinary video session, byte-identical wire. Welcome is simpler (fixed-position trailing bytes): append session_flags at offset 66, b.get(66).copied().unwrap_or(0).

Why a flags byte, not an overloaded Mode{0,0,0} sentinel: a flag is explicit and leaves room for future session-shape bits (e.g. audio-only, input+audio). Mode stays strictly a display mode. (Open question 7.1 — confirm with the user, but this is the recommendation.)

5.2 ABI

• New connect_ex rung in the existing ladder (the connect_exN precedent) that takes a session_flags (or a bool input_only) and stores it on NativeClient; legacy connect_ex* stay byte-for-byte. Regenerate include/punktfunk_core.h (CI fails on drift).
• New constant PUNKTFUNK_SESSION_INPUT_ONLY = 0x01.
• next_au / next_frame / next_audio simply return NoFrame/Closed in an input-only connection; send_input / send_rich_input / next_rumble / next_hidout are unchanged — they are already the full input-only surface.

6. Host changes — serve_session branch (punktfunk1.rs:508)

Branch on hello.session_flags & SESSION_INPUT_ONLY. When set:

Step	Today (video session)	Input-only
--max-concurrent permit (:640 _permit)	acquired (NVENC slot)	skip — input-only must not consume a GPU slot (§7.4)
validate_dimensions (:654)	required	skip
resolve_compositor (:668)	required (Virtual source)	skip — no virtual display
bit-depth / chroma / HDR / 444 probes, bitrate clamp	run	skip
Welcome (:794)	real mode + udp_port + caps	sentinel mode {0,0,0} + udp_port=0 + session_flags=INPUT_ONLY; still carries the resolved gamepad backend
Start::decode (:845)	read client udp port	read + ignore (harmless no-op)
input_thread spawn (:980)	spawn	spawn (unchanged) — this is the whole point
client→host datagram demux (:989)	spawn	spawn (unchanged) — 0xC8/0xCC/0xCB in, 0xCA/0xCD out
audio_thread (:1032, already gated on source==Virtual)	spawn	skip (add && !input_only)
virtual_stream(SessionContext{…}) (:1155) — the only place a display+encoder open and the UDP socket binds	run	replace with await stop / conn.closed() — no UDP bind happens (the bind lives inside that block), no display, no encoder
teardown (:1190+)	releases input thread + pads + held keys	unchanged — already correct

Net host change is mostly guards and deletions in one function; no hot-path, FEC, crypto, or injector changes. cfg-clean on Windows (no compositor/virtual_stream there either; the XUSB/UMDF pads are likewise system-global, SendInput is irrelevant in a gamepad-only mode).

6.1 Pad backend / fidelity (reuses steam-controller-deck-support.md verbatim)

The full-fidelity path is entirely the in-flight Deck/DualSense inject work — controller-only mode adds nothing here, it just runs it without video:

• Deck → host resolves the Steam hid-steam backend (Linux UHID) when available so Steam Input re-emits 28DE:11FF with the user's bindings + correct glyphs; trackpads → RichInput::TouchpadEx (0xCC 0x03), gyro/accel → RichInput::Motion (0xCC 0x02), back grips → BTN_PADDLE1..4 (0xC8 bits). See that doc §4–§7.
• Where a real Steam pad is unavailable, the DualSense remap fallback (steam_remap.rs) folds Steam-only inputs into a virtual DualSense (gyro→motion, right pad→touchpad, grips→configured fallback) so nothing is silently dropped.
• GamepadPref resolution policy is that doc's §7 — unchanged; the Welcome echoes the real resolved backend (honest downgrade).

7. Client changes

7.1 Core connector (client.rs::worker_main:714)

Thread an input_only flag in. When set: do the full Hello/Welcome handshake and spawn the input/mic/rich/ctrl/datagram-demux tasks (so send_input/send_rich_input + next_rumble/ next_hidout keep working), but skip the UDP port reservation + Start-derived UdpTransport::connect + Session + the data-plane pump (:810-849,1041). next_frame/ next_audio return Closed/NoFrame.

7.2 Deck / Linux client (clients/linux) — a "Use as controller" path

Add a connect path that opens the connection input-only and runs only the app-lifetime GamepadService (clients/linux/src/gamepad.rs) — it already attaches the connector, forwards pads via send_input (:161), DualSense/Deck touchpad+motion via send_rich_input, and drains next_rumble (:566)/next_hidout (:583). Do not run session.rs's video/audio pump (:135-200) — no decoder, no video window, no PipeWire player. UI is a minimal "Connected as controller — · " status surface (battery, latency, a Disconnect button), no video widget.

On the Deck specifically: set SDL_JOYSTICK_HIDAPI_STEAMDECK/HIDAPI_STEAM before sdl3::init(), resolve GamepadPref::SteamDeck from VID 0x28DE, and assert the opened device is 28DE:1205 (HIDAPI GUID …6800), not 11FF — if 11FF, show "Steam is managing this controller; full gyro/trackpad/grip fidelity needs desktop mode / Steam Input off." (Capture mechanics = steam-controller-deck-support.md §6.)

7.3 Other clients

clients/windows gets the same input-only connect + SDL GamepadService-only path (XUSB/UMDF pads are system-global; no SwapChainPanel/decode). Apple/Android: parity is optional and lower-priority — the connector ABI already exposes everything; a "controller-only" UI mode is a small add once the core flag lands. Scope per the user's roadmap, not blocking.

7.4 Session lifetime / accounting

An input-only session is long-lived (until disconnect), like a video session, but must not count against --max-concurrent (it holds no NVENC) and should be exempt from any --seconds duration cap. The mgmt API / web console should list it distinctly (it is not a "stream" — no fps/bitrate/mode), and --max-sessions accounting should likely treat it as its own class (open question 8.x). mDNS advertisement is unchanged (the host already advertises native service; input-only is a per-session negotiation, not a service variant).

8. Security / trust — unchanged

A controller-only session is a full punktfunk/1 session: same SPAKE2 PIN pairing / TOFU / --require-pairing gate, same QUIC client-auth + pinned-fingerprint trust. The only difference is the absent data plane. No new attack surface — if anything less (no UDP socket, no FEC reassembler, no decoder fed attacker bytes). The host still requires /dev/uinput (+ /dev/uhid for DualSense/ Steam) writable — the documented input group + 60-punktfunk.rules setup.

9. Milestones

• M1 — protocol + ABI: session_flags byte + SESSION_INPUT_ONLY on Hello/Welcome with round-trip + old-peer-default unit tests; new connect_ex rung; regenerate the C header.
• M2 — host branch: serve_session input-only path (skip display/encoder/audio/permit, keep input thread + pads), cfg-clean on Windows. Loopback test: handshake completes, no UDP data plane binds, gamepad events still inject into a real uinput pad.
• M3 — Deck/Linux client: "Use as controller" connect + GamepadService-only run; 28DE:1205 vs 11FF runtime check + user messaging.
• M4 — full fidelity on-glass: with steam-controller-deck-support.md's Deck capture + inject, validate Deck (desktop mode) → host: paddles + both trackpads + gyro reach the host pad, Steam Input re-emits with bindings/glyphs, rumble returns. Glass-to-glass input latency vs a wired pad.
• M5 — Windows client parity + mgmt/web "controller session" surfacing.
• M6 (deferred) — keyboard/mouse forward over the libei/portal path (needs the active-session RemoteDesktop grant; not display-independent).

10. Risks / open questions

Open questions (decide with the user):

1. session_flags byte vs Mode{0,0,0} sentinel for "no video" — recommend the explicit flags byte (room for future shapes). (Recommended; confirm.)
2. Should an input-only session be exempt from --max-concurrent and --seconds? (Recommend yes — it holds no GPU.)
3. Should mgmt/web track controller-only sessions as a distinct class (no fps/bitrate/mode), and should --max-sessions count them?
4. Deck default backend: Steam hid-steam (best — Steam Input bindings/glyphs) vs DualSense (works off-Steam too). Tie to steam-controller-deck-support.md's resolution policy.

Risks:

• Capture wall (inherited): full fidelity requires SDL to bind 28DE:1205; if the Deck is in Game Mode / Steam owns the pad, it degrades to 11FF (sticks/buttons only). Mitigated by the desktop-mode use case + runtime check + user messaging (§3, §7.2).
• Host /dev/uinput(+/dev/uhid) perms — a normal desktop PC operator must do the one-time input-group/udev setup for the virtual pad to appear (already documented).
• Handshake assumes a data plane: Start{client_udp_port} + non-zero udp_port in Welcome must become harmless no-ops (send 0 / ignore) without shifting the legacy wire — the trailing-byte placeholder discipline is fragile; add round-trip + old-peer tests (M1).
• Control-channel messages (LossReport/Reconfigure/Probe/ClockProbe) assume a data plane; the host (:881) + client (:935) control tasks must tolerate an input-only session with none — mostly already fine since those are reactive.
• Windows parity: verify the input-only branch compiles cfg-clean where there is no compositor/virtual_stream.

11. Validation plan

Loopback (no hardware):

• quic.rs: session_flags encode/decode round-trip; old-peer (no flags byte) → ordinary session; flags coexist with non-default video_caps/audio_channels (placeholder offsets hold).
• Host: an input-only synthetic host+client asserting (a) handshake completes, (b) no UDP socket binds, (c) no display/encoder opens, (d) scripted 0xC8/0xCC events inject into a real pad, (e) 0xCA/0xCD feedback returns. Extend the clients/probe / test-loopback.sh harness.

On-box / on-glass (with steam-controller-deck-support.md landed):

• Deck (desktop mode) → this host, controller-only: confirm 28DE:1205 opens (not 11FF); paddles
  • both trackpads + gyro reach the host pad; Steam Input on the host re-emits with bindings/glyphs; rumble round-trips; measure input-only latency vs a wired pad and vs the streaming path.
• Confirm the host opens no virtual output / encoder (logs) and the session does not consume a --max-concurrent slot (run alongside N video sessions).