feat: M4 stage 1 — the SwiftUI client is real: compiles, tested, first light on glass
ci / rust (push) Has been cancelled

The clients/apple scaffold is now a working macOS client, validated live against this
repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC +
AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60,
mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector
(~3.7k events injected in one session).

LumenKit:
- LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as
  integers while the enum constants import as a distinct Swift type — bridge by rawValue);
  close() is now safe from any thread (a close flag + pumpLock held across the blocking
  poll enforce the C contract "never close with a next_au in flight"; flag prevents
  lock-starvation by back-to-back polls).
- StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate
  on the next in-band parameter sets when the layer fails, no stale enqueue after restart.
- InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away),
  pressed-state tracking with release-all on focus loss and stop() (nothing sticks down
  host-side), global-singleton ownership guard (GC has one handler slot per process),
  X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs.
- LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD,
  LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs.
- Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded
  HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels);
  test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of
  c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN).

Host/build fixes that fell out:
- The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching
  are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule.
- Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the
  ei/wl axes, but GameStream's horizontal convention is positive = right
  (moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also
  un-inverts real Moonlight clients.
- AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's
  policy, instead of leaking them into the preceding NAL.
- build-xcframework.sh: deployment targets pinned to the package floor + an otool guard —
  cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship
  too-new minos objects.

Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified):
14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged
here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 14:38:01 +02:00
parent 520d7342dd
commit bf8a974e8b
23 changed files with 1212 additions and 180 deletions
+143 -21
View File
@@ -4,26 +4,50 @@
// injector expects for relative motion. GCKeyboard gives HID keycodes which we map to the
// Windows VK space the host's vk_to_evdev table consumes (same space Moonlight uses).
// Gamepads (GCController) come later the host's uinput pads already speak the
// GamepadButton/GamepadAxis event kinds.
// GamepadButton/GamepadAxis event kinds, but m3's injector path doesn't route them yet.
//
// SCAFFOLD: written on the Linux host, not yet compiled against Xcode. The VK map covers
// the common keys; extend alongside lumen-host/src/inject.rs::vk_to_evdev.
// The wire carries integer deltas; GC hands us Floats. We accumulate the fractional
// remainder per axis so slow, sub-pixel motion isn't truncated away.
//
// GC only delivers while the app is active, so anything held when focus leaves would
// stick down on the host forever we track pressed keys/buttons and release them all on
// didResignActive and on stop(). All GC handlers and notifications fire on the main
// queue (the framework default), so the mutable state here needs no locking.
//
// GCMouse.current/GCKeyboard.coalesced are process-global singletons with one handler
// slot each: only one InputCapture can be live per process. `activeCapture` tracks
// ownership so a stale capture's stop() can't clobber a newer one's handlers.
#if os(macOS)
import AppKit
import Foundation
import GameController
import LumenCore
public final class InputCapture {
private static weak var activeCapture: InputCapture?
private let connection: LumenConnection
private var observers: [NSObjectProtocol] = []
private var mice: [GCMouse] = []
private var keyboards: [GCKeyboard] = []
// Main-queue-only state (see header comment).
private var residualX: Float = 0
private var residualY: Float = 0
private var residualScrollX: Float = 0
private var residualScrollY: Float = 0
private var pressedVKs: Set<UInt32> = []
private var pressedButtons: Set<UInt32> = []
public init(connection: LumenConnection) {
self.connection = connection
}
/// Begin forwarding the current (and future) mouse/keyboard to the host.
/// Begin forwarding the current (and future) mouse/keyboard to the host. Steals the
/// global GC handler slots from any previous capture (one live capture per process).
public func start() {
Self.activeCapture = self
if let mouse = GCMouse.current { attach(mouse: mouse) }
if let keyboard = GCKeyboard.coalesced { attach(keyboard: keyboard) }
observers.append(NotificationCenter.default.addObserver(
@@ -36,44 +60,130 @@ public final class InputCapture {
) { [weak self] n in
if let k = n.object as? GCKeyboard { self?.attach(keyboard: k) }
})
// Focus loss: GC stops delivering, so release everything still held host-side.
observers.append(NotificationCenter.default.addObserver(
forName: NSApplication.didResignActiveNotification, object: nil, queue: .main
) { [weak self] _ in
self?.releaseAll()
})
}
public func stop() {
releaseAll()
observers.forEach(NotificationCenter.default.removeObserver(_:))
observers.removeAll()
// Don't clobber the handlers if a newer capture has taken the global devices.
if Self.activeCapture === self || Self.activeCapture == nil {
for mouse in mice {
guard let input = mouse.mouseInput else { continue }
input.mouseMovedHandler = nil
input.leftButton.pressedChangedHandler = nil
input.rightButton?.pressedChangedHandler = nil
input.middleButton?.pressedChangedHandler = nil
input.auxiliaryButtons?.forEach { $0.pressedChangedHandler = nil }
input.scroll.valueChangedHandler = nil
}
for keyboard in keyboards {
keyboard.keyboardInput?.keyChangedHandler = nil
}
Self.activeCapture = nil
}
mice.removeAll()
keyboards.removeAll()
}
deinit { stop() }
/// Send release events for everything currently held, and drop the motion residuals.
private func releaseAll() {
for vk in pressedVKs {
connection.send(.key(vk, down: false))
}
for button in pressedButtons {
connection.send(.mouseButton(button, down: false))
}
pressedVKs.removeAll()
pressedButtons.removeAll()
residualX = 0
residualY = 0
residualScrollX = 0
residualScrollY = 0
}
private func sendButton(_ button: UInt32, pressed: Bool) {
if pressed {
pressedButtons.insert(button)
} else {
pressedButtons.remove(button)
}
connection.send(.mouseButton(button, down: pressed))
}
private func attach(mouse: GCMouse) {
guard let input = mouse.mouseInput else { return }
let conn = connection
input.mouseMovedHandler = { _, dx, dy in
guard let input = mouse.mouseInput,
!mice.contains(where: { $0 === mouse }) // re-delivered on wake attach once
else { return }
mice.append(mouse)
input.mouseMovedHandler = { [weak self] _, dx, dy in
guard let self else { return }
// GC gives +y up; the host expects screen-space (+y down).
conn.send(.mouseMove(dx: Int32(dx), dy: Int32(-dy)))
let fx = dx + self.residualX
let fy = -dy + self.residualY
let ix = fx.rounded(.towardZero)
let iy = fy.rounded(.towardZero)
self.residualX = fx - ix
self.residualY = fy - iy
if ix != 0 || iy != 0 {
self.connection.send(.mouseMove(dx: Int32(ix), dy: Int32(iy)))
}
}
input.leftButton.pressedChangedHandler = { _, _, pressed in
conn.send(.mouseButton(1, down: pressed))
input.leftButton.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(1, pressed: pressed)
}
input.rightButton?.pressedChangedHandler = { _, _, pressed in
conn.send(.mouseButton(3, down: pressed))
input.rightButton?.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(3, pressed: pressed)
}
input.middleButton?.pressedChangedHandler = { _, _, pressed in
conn.send(.mouseButton(2, down: pressed))
input.middleButton?.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(2, pressed: pressed)
}
input.scroll.valueChangedHandler = { _, _, dy in
if dy != 0 { conn.send(.scroll(Int32(dy * 120))) }
// First two side buttons GameStream X1/X2.
if let aux = input.auxiliaryButtons {
for (i, button) in aux.prefix(2).enumerated() {
button.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(UInt32(4 + i), pressed: pressed)
}
}
}
input.scroll.valueChangedHandler = { [weak self] _, x, y in
guard let self else { return }
// WHEEL_DELTA(120) per notch; positive = up / right (Moonlight's convention).
let fy = y * 120 + self.residualScrollY
let fx = x * 120 + self.residualScrollX
let iy = fy.rounded(.towardZero)
let ix = fx.rounded(.towardZero)
self.residualScrollY = fy - iy
self.residualScrollX = fx - ix
if iy != 0 { self.connection.send(.scroll(Int32(iy))) }
if ix != 0 { self.connection.send(.scroll(Int32(ix), horizontal: true)) }
}
}
private func attach(keyboard: GCKeyboard) {
let conn = connection
keyboard.keyboardInput?.keyChangedHandler = { _, _, keyCode, pressed in
if let vk = Self.hidToVK[keyCode.rawValue] {
conn.send(.key(vk, down: pressed))
guard !keyboards.contains(where: { $0 === keyboard }) else { return }
keyboards.append(keyboard)
keyboard.keyboardInput?.keyChangedHandler = { [weak self] _, _, keyCode, pressed in
guard let self, let vk = Self.hidToVK[keyCode.rawValue] else { return }
if pressed {
self.pressedVKs.insert(vk)
} else {
self.pressedVKs.remove(vk)
}
self.connection.send(.key(vk, down: pressed))
}
}
/// HID usage (GCKeyCode raw) Windows VK (the host maps VK evdev).
/// HID usage (GCKeyCode raw) Windows VK (the host maps VK evdev; every VK emitted
/// here exists in lumen-host/src/inject.rs::vk_to_evdev extend the two together).
static let hidToVK: [Int: UInt32] = {
var m: [Int: UInt32] = [:]
// az: HID 0x04..0x1D VK 'A'..'Z'.
@@ -90,11 +200,23 @@ public final class InputCapture {
m[0x2F] = 0xDB; m[0x30] = 0xDD; m[0x31] = 0xDC // [ ] backslash
m[0x33] = 0xBA; m[0x34] = 0xDE; m[0x35] = 0xC0 // ; ' `
m[0x36] = 0xBC; m[0x37] = 0xBE; m[0x38] = 0xBF // , . /
m[0x39] = 0x14 // caps lock
// F1..F12: HID 0x3A..0x45 VK 0x70..0x7B.
for i in 0..<12 { m[0x3A + i] = UInt32(0x70 + i) }
m[0x46] = 0x2C; m[0x47] = 0x91; m[0x48] = 0x13 // printscreen scrolllock pause
m[0x4F] = 0x27; m[0x50] = 0x25; m[0x51] = 0x28; m[0x52] = 0x26 // arrows R L D U
m[0x49] = 0x2D; m[0x4A] = 0x24; m[0x4B] = 0x21 // insert home pageup
m[0x4C] = 0x2E; m[0x4D] = 0x23; m[0x4E] = 0x22 // delete end pagedown
// Keypad: NumLock, / * - +, Enter, 1..9, 0, decimal. KP Enter goes as
// VK_SEPARATOR (0x6C) this host maps it to KEY_KPENTER (Windows itself would
// send VK_RETURN+extended, which vk_to_evdev can't distinguish).
m[0x53] = 0x90
m[0x54] = 0x6F; m[0x55] = 0x6A; m[0x56] = 0x6D; m[0x57] = 0x6B
m[0x58] = 0x6C
for i in 0..<9 { m[0x59 + i] = UInt32(0x61 + i) }
m[0x62] = 0x60; m[0x63] = 0x6E
m[0x64] = 0xE2 // ISO 102nd key (<> next to left shift on ISO layouts)
m[0x65] = 0x5D // menu/application
m[0xE0] = 0xA2; m[0xE1] = 0xA0; m[0xE2] = 0xA4; m[0xE3] = 0x5B // Lctrl Lshift Lalt Lcmd
m[0xE4] = 0xA3; m[0xE5] = 0xA1; m[0xE6] = 0xA5; m[0xE7] = 0x5C // Rctrl Rshift Ralt Rcmd
return m