feat: M4 stage 1 — the SwiftUI client is real: compiles, tested, first light on glass
ci / rust (push) Has been cancelled

The clients/apple scaffold is now a working macOS client, validated live against this
repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC +
AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60,
mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector
(~3.7k events injected in one session).

LumenKit:
- LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as
  integers while the enum constants import as a distinct Swift type — bridge by rawValue);
  close() is now safe from any thread (a close flag + pumpLock held across the blocking
  poll enforce the C contract "never close with a next_au in flight"; flag prevents
  lock-starvation by back-to-back polls).
- StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate
  on the next in-band parameter sets when the layer fails, no stale enqueue after restart.
- InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away),
  pressed-state tracking with release-all on focus loss and stop() (nothing sticks down
  host-side), global-singleton ownership guard (GC has one handler slot per process),
  X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs.
- LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD,
  LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs.
- Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded
  HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels);
  test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of
  c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN).

Host/build fixes that fell out:
- The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching
  are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule.
- Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the
  ei/wl axes, but GameStream's horizontal convention is positive = right
  (moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also
  un-inverts real Moonlight clients.
- AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's
  policy, instead of leaking them into the preceding NAL.
- build-xcframework.sh: deployment targets pinned to the package floor + an otool guard —
  cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship
  too-new minos objects.

Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified):
14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged
here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 14:38:01 +02:00
parent 520d7342dd
commit bf8a974e8b
23 changed files with 1212 additions and 180 deletions
@@ -1,8 +1,9 @@
// Swift wrapper around the lumen-core C ABI's lumen/1 connection API.
//
// Threading contract (mirrors the C header): one LumenConnection is used from a single
// pump thread for nextAU(); nextAudio() may run on its own (single) audio thread;
// sendInput() is enqueue-only and safe alongside both. The pointers inside an AU/audio
// Threading contract (mirrors the C header): one LumenConnection is pumped from a single
// video thread via nextAU(); nextAudio()/nextRumble() may each run on their own (single)
// drain thread the core keeps per-plane borrow slots, so the planes never alias;
// send() is enqueue-only and safe alongside all of them. The pointers inside an AU/audio
// packet are only valid until the next call of the same kind, so we copy into Data here
// the copies are small and keep the Swift side memory-safe.
//
@@ -10,12 +11,22 @@
// `hostFingerprint` reports what a trust-on-first-use connect observed persist it, e.g.
// in UserDefaults keyed by host, and pin it from then on).
//
// SCAFFOLD: written on the Linux host, not yet compiled against Xcode expect to fix
// trivial issues on first build (see README.md "Handoff").
// close() is safe from any thread: it flags the pullers to exit at their next poll
// boundary, then takes the per-plane locks (each held across its blocking C poll), so the
// handle is never freed under an in-flight call the C contract ("never close with a
// next_au/next_audio call in flight") is enforced here rather than left to callers. After
// close, the pull methods throw `.closed` and the threads unwind on their own.
import Foundation
import LumenCore
// cbindgen's C17-compatible header spells the typedefs as plain integers
// (`typedef int32_t LumenStatus`, `typedef uint8_t LumenInputKind`) while the enum
// constants import as a distinct same-named Swift type bridge by raw value once here.
private let statusOK: Int32 = LUMEN_STATUS_OK.rawValue
private let statusNoFrame: Int32 = LUMEN_STATUS_NO_FRAME.rawValue
private let statusClosed: Int32 = LUMEN_STATUS_CLOSED.rawValue
/// One reassembled, FEC-recovered, decrypted access unit (Annex-B HEVC from the host).
public struct AccessUnit: Sendable {
public let data: Data
@@ -39,10 +50,22 @@ public enum LumenClientError: Error {
/// unpinned when the caller asked for verification would be a silent trust downgrade.
case invalidPin
case closed
case status(Int32)
}
public final class LumenConnection {
private var handle: OpaquePointer?
/// Set by close() before it contends for the plane locks: the pullers see it at their
/// next poll boundary and exit, so close() can't be starved by back-to-back polls
/// (NSLock is not fair).
private var closeRequested = false
/// Serializes send()/close() against each other and guards `handle`/`closeRequested`.
private let abiLock = NSLock()
/// Held across the blocking next_au call; close() takes it (same plane-lock abiLock
/// order as the pullers) so it can never free the handle under an in-flight poll.
private let pumpLock = NSLock()
/// Same role for the audio/rumble drain thread (its own plane in the core).
private let audioLock = NSLock()
/// Negotiated session mode (host-confirmed).
public private(set) var width: UInt32 = 0
@@ -86,87 +109,141 @@ public final class LumenConnection {
self.refreshHz = hz
}
/// Pull the next access unit; nil on timeout, throws once the session is closed.
/// Pull the next access unit; nil on timeout, throws `.closed` once the session ended.
/// Call from a single pump thread.
public func nextAU(timeoutMs: UInt32 = 100) throws -> AccessUnit? {
pumpLock.lock()
defer { pumpLock.unlock() }
guard let h = liveHandle() else { throw LumenClientError.closed }
var frame = LumenFrame()
switch lumen_connection_next_au(handle, &frame, timeoutMs) {
case LUMEN_STATUS_OK:
let data = Data(bytes: frame.data, count: frame.len) // copy: ptr valid only until next call
let rc = lumen_connection_next_au(h, &frame, timeoutMs)
switch rc {
case statusOK:
guard let base = frame.data, frame.len > 0 else { return nil }
let data = Data(bytes: base, count: Int(frame.len)) // copy: ptr valid only until next call
return AccessUnit(
data: data, ptsNs: frame.pts_ns,
frameIndex: frame.frame_index, flags: frame.flags)
case LUMEN_STATUS_NO_FRAME:
case statusNoFrame:
return nil
case LUMEN_STATUS_CLOSED:
case statusClosed:
throw LumenClientError.closed
default:
throw LumenClientError.closed
throw LumenClientError.status(rc)
}
}
/// Pull the next Opus audio packet; nil on timeout, throws once the session is closed.
/// Drain from a dedicated audio thread packets arrive every 5 ms (320 ms buffered).
/// Pull the next Opus audio packet; nil on timeout, throws `.closed` once the session
/// ended. Drain from a dedicated audio thread packets arrive every 5 ms (the core
/// buffers 320 ms and drops the newest when the puller lags).
public func nextAudio(timeoutMs: UInt32 = 100) throws -> AudioPacket? {
audioLock.lock()
defer { audioLock.unlock() }
guard let h = liveHandle() else { throw LumenClientError.closed }
var pkt = LumenAudioPacket()
switch lumen_connection_next_audio(handle, &pkt, timeoutMs) {
case LUMEN_STATUS_OK:
let data = Data(bytes: pkt.data, count: pkt.len) // copy: ptr valid only until next call
let rc = lumen_connection_next_audio(h, &pkt, timeoutMs)
switch rc {
case statusOK:
guard let base = pkt.data, pkt.len > 0 else { return nil }
let data = Data(bytes: base, count: Int(pkt.len)) // copy: ptr valid only until next call
return AudioPacket(data: data, ptsNs: pkt.pts_ns, seq: pkt.seq)
case LUMEN_STATUS_NO_FRAME:
case statusNoFrame:
return nil
default:
case statusClosed:
throw LumenClientError.closed
default:
throw LumenClientError.status(rc)
}
}
/// Pull the next force-feedback update for the GCController haptics engine:
/// `(pad, lowFrequency, highFrequency)` with 0...0xFFFF amplitudes, (0, 0) = stop.
public func nextRumble(timeoutMs: UInt32 = 100) throws -> (pad: UInt16, low: UInt16, high: UInt16)? {
/// Shares the audio drain thread's plane (call from that thread).
public func nextRumble(timeoutMs: UInt32 = 0) throws -> (pad: UInt16, low: UInt16, high: UInt16)? {
audioLock.lock()
defer { audioLock.unlock() }
guard let h = liveHandle() else { throw LumenClientError.closed }
var pad: UInt16 = 0, low: UInt16 = 0, high: UInt16 = 0
switch lumen_connection_next_rumble(handle, &pad, &low, &high, timeoutMs) {
case LUMEN_STATUS_OK:
let rc = lumen_connection_next_rumble(h, &pad, &low, &high, timeoutMs)
switch rc {
case statusOK:
return (pad, low, high)
case LUMEN_STATUS_NO_FRAME:
case statusNoFrame:
return nil
default:
case statusClosed:
throw LumenClientError.closed
default:
throw LumenClientError.status(rc)
}
}
/// Send one input event (delivered to the host as a QUIC datagram).
/// Send one input event (delivered to the host as a QUIC datagram). Thread-safe;
/// silently dropped after close.
public func send(_ event: LumenInputEvent) {
var ev = event
_ = lumen_connection_send_input(handle, &ev)
abiLock.lock()
defer { abiLock.unlock() }
guard let h = handle, !closeRequested else { return }
_ = lumen_connection_send_input(h, &ev)
}
/// Close the connection and free the handle. Safe from any thread, idempotent; waits
/// for in-flight pulls ( their timeouts) before tearing down.
public func close() {
if let h = handle {
lumen_connection_close(h)
handle = nil
abiLock.lock()
closeRequested = true
abiLock.unlock()
pumpLock.lock() // pullers exit at their next poll boundary, releasing these
audioLock.lock()
abiLock.lock()
let h = handle
handle = nil
abiLock.unlock()
audioLock.unlock()
pumpLock.unlock()
if let h {
lumen_connection_close(h) // joins the connection's internal Rust threads
}
}
deinit { close() }
/// Snapshot the handle unless close is pending (callers hold their plane lock).
private func liveHandle() -> OpaquePointer? {
abiLock.lock()
defer { abiLock.unlock() }
return closeRequested ? nil : handle
}
}
// Convenience constructors for the wire input events (field semantics match
// lumen_core::input::InputEvent; see lumen_core.h).
public extension LumenInputEvent {
private static func make(
_ kind: UInt32, code: UInt32, x: Int32, y: Int32, flags: UInt32 = 0
) -> LumenInputEvent {
LumenInputEvent(kind: UInt8(kind), _pad: (0, 0, 0), code: code, x: x, y: y, flags: flags)
}
static func mouseMove(dx: Int32, dy: Int32) -> LumenInputEvent {
LumenInputEvent(kind: LUMEN_INPUT_KIND_MOUSE_MOVE, _pad: (0, 0, 0), code: 0, x: dx, y: dy, flags: 0)
make(LUMEN_INPUT_KIND_MOUSE_MOVE.rawValue, code: 0, x: dx, y: dy)
}
/// GameStream button ids: 1=left 2=middle 3=right 4=X1 5=X2 (host maps to evdev BTN_*).
static func mouseButton(_ button: UInt32, down: Bool) -> LumenInputEvent {
LumenInputEvent(
kind: down ? LUMEN_INPUT_KIND_MOUSE_BUTTON_DOWN : LUMEN_INPUT_KIND_MOUSE_BUTTON_UP,
_pad: (0, 0, 0), code: button, x: 0, y: 0, flags: 0)
make(
(down ? LUMEN_INPUT_KIND_MOUSE_BUTTON_DOWN : LUMEN_INPUT_KIND_MOUSE_BUTTON_UP).rawValue,
code: button, x: 0, y: 0)
}
/// `vk` is a Windows virtual-key code (the host's vk_to_evdev table consumes these).
static func key(_ vk: UInt32, down: Bool) -> LumenInputEvent {
LumenInputEvent(
kind: down ? LUMEN_INPUT_KIND_KEY_DOWN : LUMEN_INPUT_KIND_KEY_UP,
_pad: (0, 0, 0), code: vk, x: 0, y: 0, flags: 0)
make((down ? LUMEN_INPUT_KIND_KEY_DOWN : LUMEN_INPUT_KIND_KEY_UP).rawValue, code: vk, x: 0, y: 0)
}
static func scroll(_ delta: Int32) -> LumenInputEvent {
LumenInputEvent(kind: LUMEN_INPUT_KIND_MOUSE_SCROLL, _pad: (0, 0, 0), code: 0, x: delta, y: 0, flags: 0)
/// WHEEL_DELTA(120)-scaled; positive = up (vertical) / right (horizontal) the
/// convention Moonlight/SDL use; the host maps onto the ei/wl axes.
static func scroll(_ delta: Int32, horizontal: Bool = false) -> LumenInputEvent {
make(LUMEN_INPUT_KIND_MOUSE_SCROLL.rawValue, code: horizontal ? 1 : 0, x: delta, y: 0)
}
// Gamepad (wire contract in lumen_core::input::gamepad): one transition per event,
@@ -175,16 +252,14 @@ public extension LumenInputEvent {
/// `button` is a GameStream buttonFlags bit (A=0x1000 B=0x2000 X=0x4000 Y=0x8000,
/// dpad=0x1/2/4/8, start=0x10 back=0x20 LS=0x40 RS=0x80 LB=0x100 RB=0x200 guide=0x400).
static func gamepadButton(_ button: UInt32, down: Bool, pad: UInt32 = 0) -> LumenInputEvent {
LumenInputEvent(
kind: LUMEN_INPUT_KIND_GAMEPAD_BUTTON,
_pad: (0, 0, 0), code: button, x: down ? 1 : 0, y: 0, flags: pad)
make(
LUMEN_INPUT_KIND_GAMEPAD_BUTTON.rawValue,
code: button, x: down ? 1 : 0, y: 0, flags: pad)
}
/// Axis ids: 0=LSX 1=LSY 2=RSX 3=RSY (32768...32767, XInput convention: +y = UP
/// `GCControllerDirectionPad.yAxis` already matches, no flip), 4=LT 5=RT (0...255).
static func gamepadAxis(_ axis: UInt32, value: Int32, pad: UInt32 = 0) -> LumenInputEvent {
LumenInputEvent(
kind: LUMEN_INPUT_KIND_GAMEPAD_AXIS,
_pad: (0, 0, 0), code: axis, x: value, y: 0, flags: pad)
make(LUMEN_INPUT_KIND_GAMEPAD_AXIS.rawValue, code: axis, x: value, y: 0, flags: pad)
}
}