feat: M4 stage 1 — the SwiftUI client is real: compiles, tested, first light on glass
ci / rust (push) Has been cancelled

The clients/apple scaffold is now a working macOS client, validated live against this
repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC +
AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60,
mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector
(~3.7k events injected in one session).

LumenKit:
- LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as
  integers while the enum constants import as a distinct Swift type — bridge by rawValue);
  close() is now safe from any thread (a close flag + pumpLock held across the blocking
  poll enforce the C contract "never close with a next_au in flight"; flag prevents
  lock-starvation by back-to-back polls).
- StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate
  on the next in-band parameter sets when the layer fails, no stale enqueue after restart.
- InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away),
  pressed-state tracking with release-all on focus loss and stop() (nothing sticks down
  host-side), global-singleton ownership guard (GC has one handler slot per process),
  X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs.
- LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD,
  LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs.
- Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded
  HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels);
  test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of
  c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN).

Host/build fixes that fell out:
- The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching
  are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule.
- Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the
  ei/wl axes, but GameStream's horizontal convention is positive = right
  (moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also
  un-inverts real Moonlight clients.
- AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's
  policy, instead of leaking them into the preceding NAL.
- build-xcframework.sh: deployment targets pinned to the package floor + an otool guard —
  cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship
  too-new minos objects.

Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified):
14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged
here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 14:38:01 +02:00
parent 520d7342dd
commit bf8a974e8b
23 changed files with 1212 additions and 180 deletions
+4
View File
@@ -7,3 +7,7 @@
/tools/*/target
node_modules/
dist/
# Swift package build artifacts + the locally-built xcframework (rebuild via scripts/build-xcframework.sh)
clients/apple/.build/
clients/apple/LumenCore.xcframework/
clients/apple/.swiftpm/
+13 -8
View File
@@ -44,11 +44,16 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc
## What's left
1. **M4 — client decode + present**: the SwiftUI client is scaffolded and handed off —
the lumen/1 connector is in the C ABI (`lumen_connect` & co., ABI-roundtrip-tested) with
an xcframework build script + LumenKit Swift package; **see
[`clients/apple/README.md`](clients/apple/README.md) for the Mac-side pickup**. Then
glass-to-glass numbers via `tools/latency-probe` (scaffold). The Linux reference client
1. **M4 — client decode + present: macOS stage 1 done, first light achieved
(2026-06-10).** LumenKit compiles and is tested on macOS (AnnexB → VideoToolbox →
`AVSampleBufferDisplayLayer`, GCMouse/GCKeyboard capture, `LumenClient` app shell);
validated live Mac ↔ this box at 720p60 — vkcube on glass, input injected via gamescope
EIS. Tests: `swift test` in `clients/apple` (unit + real-codec round trip),
`test-loopback.sh` (Swift client vs synthetic m3-host on loopback — runs on macOS),
`RemoteFirstLightTests` (full pipeline over the LAN). See
[`clients/apple/README.md`](clients/apple/README.md). Next: stage 2 presenter
(`VTDecompressionSession` + `CAMetalLayer` frame pacing), glass-to-glass numbers via
`tools/latency-probe` (scaffold), iOS variant. The Linux reference client
(`lumen-client-rs`) gets VAAPI + wgpu on the same connector later.
2. **Sub-frame pipelining**: overlap encode and transmit within a frame. Requires a direct
NVENC SDK wrapper (libavcodec only emits whole AUs) — the next big latency lever (~24 ms
@@ -61,9 +66,9 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc
HDR/10-bit/AV1 negotiation, surround audio, reconnect-at-new-mode robustness.
5. **Native clients** (`clients/{apple,android}` scaffolds) consuming `lumen_core.h`.
6. **This box, one-time setup still pending**: `sudo cp scripts/60-lumen.rules
/etc/udev/rules.d/` + user into `input` group (gamepads); `sudo ninja -C
/tmp/gamescope-src/build install` (the fixed gamescope 3.16.22 — until then use
`PATH=/tmp/gamescope-src/build/src:$PATH`); `apt install gnome-shell` (Mutter validation).
/etc/udev/rules.d/` + user into `input` group (gamepads); `apt install gnome-shell`
(Mutter validation). Done since last update: gamescope 3.16.22 is installed at
`/usr/local/bin` — the `PATH=/tmp/gamescope-src/...` override is no longer needed.
## Build / test / run
+1 -1
View File
@@ -17,7 +17,7 @@ negotiated extension. See [`docs/implementation-plan.md`](docs/implementation-pl
| M2 — P1 host → stock Moonlight | 🟡 capture+encode landed in M0; pairing/RTSP/vdisplay pending |
| M3 — measurement harness | 🟡 `tools/loss-harness` runs; `latency-probe` scaffolded |
| M4 — P2 transport + Rust client | 🟡 GF(2¹⁶) core done; `lumen-client-rs` scaffolded |
| M5 — Apple client | ⬜ scaffolded (`clients/apple`) |
| M5 — Apple client | 🟡 macOS first light: HEVC on glass + input over `lumen/1` (`clients/apple`) |
`lumen-core` is complete and verified: it builds and its full test suite (FEC recovery,
loopback round-trip under loss, property tests, and a **C ABI harness**) passes on
+5 -1
View File
@@ -8,7 +8,8 @@ let package = Package(
name: "LumenKit",
platforms: [.macOS(.v14), .iOS(.v17)],
products: [
.library(name: "LumenKit", targets: ["LumenKit"])
.library(name: "LumenKit", targets: ["LumenKit"]),
.executable(name: "LumenClient", targets: ["LumenClient"]),
],
targets: [
.binaryTarget(name: "LumenCore", path: "LumenCore.xcframework"),
@@ -22,5 +23,8 @@ let package = Package(
.linkedLibrary("resolv"),
]
),
// Development app shell (swift run LumenClient): connect form stream + input.
.executableTarget(name: "LumenClient", dependencies: ["LumenKit"]),
.testTarget(name: "LumenKitTests", dependencies: ["LumenKit"]),
]
)
+86 -76
View File
@@ -1,102 +1,112 @@
# lumen Apple client (SwiftUI) — handoff
# lumen Apple client (SwiftUI)
The native macOS/iOS client for **`lumen/1`** (the post-GameStream protocol). All
networking/protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM,
input datagrams — lives in the shared Rust core and is **done and tested**; this package
is the Swift shell: decode (VideoToolbox), present (SwiftUI), input capture.
input datagrams, Opus audio, cert pinning — lives in the shared Rust core (statically
linked as `LumenCore.xcframework`); this package is the Swift shell: decode
(VideoToolbox), present (SwiftUI), input capture.
## What exists (built + tested on the Linux host)
## Status — first light achieved (2026-06-10)
- **The connector**: `lumen_core::client::NativeClient` (Rust) exposed over the C ABI as
`lumen_connect` / `lumen_connection_next_au` / `lumen_connection_next_audio` /
`lumen_connection_next_rumble` / `lumen_connection_send_input` / `lumen_connection_mode`
/ `lumen_connection_close` (see `include/lumen_core.h`, guarded by `LUMEN_FEATURE_QUIC`).
**End-to-end tested through the C ABI** against an in-process host
(`crates/lumen-host/src/m3.rs::tests::c_abi_connection_roundtrip` — three sequential
sessions: TOFU, pinned reconnect, wrong-pin rejection).
- **The host to test against**: `lumen-host m3-host --source virtual --seconds 60` on the
Linux box — a **persistent listener** (sessions back to back, reconnect at will during
development; `--max-sessions N` to bound it). It creates a native virtual output at
whatever mode the client requests and streams HEVC + desktop **Opus audio**;
`LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube` for moving content.
- **This package (SCAFFOLD — written on Linux, never compiled in Xcode)**:
- `LumenConnection.swift` — Swift wrapper over the C ABI (AUs/audio copied into `Data`;
certificate pinning + TOFU fingerprint via `pinSHA256:`/`hostFingerprint`).
Validated live, Mac ↔ Linux box over the LAN: gamescope virtual output → NVENC HEVC →
`lumen/1` (GF(2¹⁶) FEC + AES-GCM over UDP, QUIC control) → VideoToolbox →
`AVSampleBufferDisplayLayer` on glass at 1280×720@60, with mouse/keyboard flowing back as
QUIC datagrams into the host's gamescope EIS injector (thousands of events injected during
the session). Headless variant of the same proof: `RemoteFirstLightTests` decoded 60/60
received AUs spanning 983 ms of host capture clock.
The connector underneath (`lumen_core::client::NativeClient` over the C ABI) carries the
full session: video AUs, **Opus audio** (`nextAudio()`), **rumble** (`nextRumble()`),
input incl. gamepads, and **cert pinning + TOFU** (`pinSHA256:`/`hostFingerprint`) — see
`m3.rs::tests::c_abi_connection_roundtrip` (three sequential sessions: TOFU, pinned
reconnect, wrong-pin rejection). The host (`lumen-host m3-host`) is a persistent listener:
reconnect at will during development.
What's here, all compiled and tested on macOS (Xcode 26.5 / Swift 6.3):
- **`LumenKit`** (library)
- `LumenConnection.swift` — wrapper over the C ABI. AUs/audio are copied into `Data`
(the C pointer is only valid until the next call of the same kind). `close()` is safe
from any thread: per-plane locks enforce the C contract ("never close with a
`next_au`/`next_audio` in flight") instead of leaving it to callers. Pinning + TOFU
via `pinSHA256:`/`hostFingerprint`.
- `AnnexB.swift` — in-band VPS/SPS/PPS → `CMVideoFormatDescription`; Annex-B → AVCC
`CMSampleBuffer` with `DisplayImmediately` set.
- `StreamView.swift` — SwiftUI `NSViewRepresentable` over `AVSampleBufferDisplayLayer`
(stage-1 presenter: the layer hardware-decodes compressed HEVC itself).
- `InputCapture.swift``GCMouse` raw deltas + `GCKeyboard` HID→VK mapping →
`lumen_connection_send_input`.
(stage-1 presenter: the layer hardware-decodes compressed HEVC itself). One pump
thread per view, token-cancelled so reconnects can't double-pump.
- `InputCapture.swift``GCMouse` raw deltas + `GCKeyboard` HID→VK mapping (the host's
`vk_to_evdev` consumes Windows VKs), with fractional-delta accumulation so sub-pixel
motion isn't truncated away. Buttons use GameStream ids (1=left … 5=X2); scroll is
WHEEL_DELTA(120)-scaled.
- **`LumenClient`** (development app shell): connect form → stream + input, fps/Mb-s HUD.
(Audio playback and gamepad capture are not wired into the app yet — the connector
surface is there; see notes 56.)
- **Tests** (`swift test`): byte-level Annex-B units; a real-codec round trip
(VTCompressionSession-encoded HEVC rebuilt as the host's wire shape → `AnnexB`
VTDecompressionSession → pixels); loopback integration against a real local host
(`test-loopback.sh`); the remote first-light test above.
## Build steps (on the Mac)
## Build / run / test (on a Mac)
```sh
rustup target add aarch64-apple-darwin x86_64-apple-darwin
bash scripts/build-xcframework.sh # → clients/apple/LumenCore.xcframework
open clients/apple/Package.swift # or add the package to an Xcode app project
cd clients/apple
swift build && swift test # loopback/remote tests self-skip without a host
swift run LumenClient # the app; or open Package.swift in Xcode
bash test-loopback.sh # full loopback proof: builds lumen-host
# (synthetic source — runs on macOS), streams
# byte-verified frames into the Swift client
# against the real host (Linux box, see CLAUDE.md "Running on this box") — m3-host is a
# persistent listener, reconnect at will:
# LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 \
# cargo run -rp lumen-host -- m3-host --source virtual --seconds 60
LUMEN_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests # headless
LUMEN_AUTOCONNECT=<box-ip> LUMEN_MODE=1280x720x60 swift run LumenClient # on glass
```
Minimal app around it:
## Notes for whoever picks this up next
```swift
@main struct LumenApp: App {
var body: some Scene { WindowGroup { ContentView() } }
}
struct ContentView: View {
@State private var conn: LumenConnection?
var body: some View {
if let conn {
StreamView(connection: conn)
.onAppear { InputCapture(connection: conn).start() }
} else {
Button("Connect") {
conn = try? LumenConnection(
host: "192.168.1.70", width: 2560, height: 1440, refreshHz: 120)
}
}
}
}
```
## Handoff — what the next agent needs to know
1. **Expect small compile fixes.** Every Swift file is flagged SCAFFOLD: API-checked from
documentation, never run through Xcode. Likely friction: the imported C enum spellings
(`LUMEN_STATUS_OK` etc. — cbindgen emits `QualifiedScreamingSnakeCase`), `LumenFrame()`
zero-init, `_pad` tuple shape on `LumenInputEvent`.
2. **ABI contract** (matches `lumen_core.h` docs): `next_au`'s pointer is valid only until
the *next* call on that handle (we copy to `Data` immediately); one pump thread per
connection, plus optionally one *separate* audio thread for `next_audio` (independent
borrow slots); `send_input` is enqueue-only and thread-safe alongside both; `close`
joins the Rust threads — never call it with a `next_au`/`next_audio` call in flight.
3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band,
and recovery keyframes re-send them — so "wait for the first format description, refresh
it on every IDR" (already what `StreamView` does) is sufficient; there is no out-of-band
extradata, ever.
4. **First-light test**: Linux box runs
`PATH=/tmp/gamescope-src/build/src:$PATH LUMEN_COMPOSITOR=gamescope \
LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 cargo run -rp lumen-host -- m3-host
--source virtual --seconds 120`; Mac connects with the app. Success = the spinning
vkcube on glass. Then mouse/keys should appear inside the gamescope session (verify
with `LUMEN_GAMESCOPE_APP=xev` and the box-side log `/tmp/lumen-gamescope.log`).
5. **Stage 2 (after first light)**: replace `AVSampleBufferDisplayLayer` with explicit
`VTDecompressionSession` + `CAMetalLayer` for frame-pacing control (ProMotion/120 Hz),
and add glass-to-glass measurement (`tools/latency-probe` is the scaffold; the host
already stamps `pts_ns` with its capture wall clock — across machines you'll need a
clock-offset estimate from the QUIC RTT, or the probe's visual timestamp loop).
6. **Audio**: `nextAudio()` yields raw Opus packets (48 kHz stereo, one 5 ms frame each,
1. **cbindgen import quirk** (the predicted "small compile fixes", now fixed): the
C17-compatible header spells `LumenStatus`/`LumenInputKind` as integer typedefs while
the enum *constants* import into Swift as a distinct same-named type — bridge with
`.rawValue` (see the top of `LumenConnection.swift`). Don't fight the generated header.
2. **ABI contract**: one video pump thread per connection, plus optionally one *separate*
audio drain thread for `nextAudio()`/`nextRumble()` (the core keeps per-plane borrow
slots, so the planes never alias); `send()` is enqueue-only and safe alongside all of
them. The wrapper's per-plane locks make `close()` safe from anywhere (it waits out
in-flight polls, ≤ their timeouts).
3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band
and recovery keyframes re-send them — "refresh the format description on every IDR"
(what `StreamView` does) is sufficient; there is no out-of-band extradata, ever.
4. **Stage 2 (next)**: explicit `VTDecompressionSession` + `CAMetalLayer` for frame-pacing
control (ProMotion/120 Hz), glass-to-glass measurement via `tools/latency-probe` (the
host stamps `pts_ns` with its capture wall clock; across machines you need a clock
offset estimate from the QUIC RTT).
5. **Audio**: `nextAudio()` yields raw Opus packets (48 kHz stereo, one 5 ms frame each,
sequence-numbered). Decode with libopus or `AVAudioConverter`/`kAudioFormatOpus` into an
`AVAudioEngine` source node; conceal gaps (drop/dup) rather than blocking — the Rust
side buffers 320 ms and drops the newest packet when the puller lags. Wall-clock
`ptsNs` shares the host clock with video AUs for A/V sync.
7. **Gamepads**: `GCController``.gamepadButton(...)`/`.gamepadAxis(...)` events (wire
`ptsNs` shares the host clock with video AUs for A/V sync. Wiring this into
`LumenClient` is the next app-side task.
6. **Gamepads**: `GCController``.gamepadButton(...)`/`.gamepadAxis(...)` events (wire
contract documented on the constructors; the host accumulates them into a virtual
Xbox 360 pad). Poll `nextRumble()` and feed `GCDeviceHaptics` for force feedback.
8. **Trust**: connect once with `pinSHA256: nil` (TOFU), persist `hostFingerprint` keyed
Client-side capture isn't in `InputCapture` yet.
7. **Trust**: connect once with `pinSHA256: nil` (TOFU), persist `hostFingerprint` keyed
by host, pass it on every later connect — a mismatch throws `.connectFailed`. The host
logs its fingerprint at startup ("clients pin this fingerprint") for out-of-band
verification UX; a PIN-style pairing ceremony is a later lumen-core task.
verification UX; a PIN-style pairing ceremony is a later lumen-core task. `LumenClient`
doesn't persist fingerprints yet — add it alongside the "add host" UX.
8. **Input capture caveats** (stage 1): GC handlers only fire while the app has focus —
on focus loss `InputCapture` auto-releases everything still held (keys + buttons) so
nothing sticks down host-side. Local shortcuts (⌘-anything) still also reach the host;
a capture toggle is a small follow-up. One live capture per process (the GC mouse/
keyboard singletons have a single handler slot — ownership is tracked so a stale
capture's stop() can't clobber a newer one).
9. **iOS**: same package (`BUILD_IOS=1` for the xcframework slice); `StreamView` needs the
`UIViewRepresentable` twin and touch→input mapping.
@@ -0,0 +1,122 @@
// Connect form live stream. Stage-1 UX: pick host + mode, see frames, type/aim.
import AppKit
import LumenKit
import SwiftUI
struct ContentView: View {
@StateObject private var model = SessionModel()
@AppStorage("lumen.host") private var host = "192.168.1.70"
@AppStorage("lumen.port") private var port = 9777
@AppStorage("lumen.width") private var width = 1920
@AppStorage("lumen.height") private var height = 1080
@AppStorage("lumen.hz") private var hz = 60
var body: some View {
Group {
if let conn = model.connection {
stream(conn)
} else {
connectForm
}
}
.onAppear { autoConnectIfAsked() }
.onDisappear { model.disconnect() } // window closed mid-session (Cmd+N spawns more)
}
/// Development hook: LUMEN_AUTOCONNECT=host[:port] connects immediately at the saved
/// (or LUMEN_MODE=WxHxHz) mode lets scripts drive first-light runs. (IPv4/hostname
/// only; an IPv6 literal would need bracket parsing.)
private func autoConnectIfAsked() {
guard let target = ProcessInfo.processInfo.environment["LUMEN_AUTOCONNECT"],
!target.isEmpty, model.connection == nil, !model.connecting
else { return }
let parts = target.split(separator: ":")
host = String(parts[0])
if parts.count == 2, let p = Int(parts[1]) { port = p }
if let mode = ProcessInfo.processInfo.environment["LUMEN_MODE"] {
let dims = mode.split(separator: "x").compactMap { Int($0) }
if dims.count == 3 {
width = dims[0]
height = dims[1]
hz = dims[2]
}
}
model.connect(
host: host, port: UInt16(clamping: port),
width: UInt32(clamping: width), height: UInt32(clamping: height),
hz: UInt32(clamping: hz))
}
private func stream(_ conn: LumenConnection) -> some View {
StreamView(
connection: conn,
onFrame: { [meter = model.meter] au in meter.note(byteCount: au.data.count) },
onSessionEnd: { [weak model] in
Task { @MainActor in model?.sessionEnded() }
}
)
.overlay(alignment: .topTrailing) { hud(conn) }
.frame(minWidth: 640, minHeight: 360)
.background(Color.black)
}
private func hud(_ conn: LumenConnection) -> some View {
VStack(alignment: .trailing, spacing: 4) {
Text("\(conn.width)×\(conn.height)@\(conn.refreshHz) \(model.fps) fps \(model.mbps, specifier: "%.1f") Mb/s")
.font(.system(.caption, design: .monospaced))
Button("Disconnect") { model.disconnect() }
.font(.caption)
}
.padding(8)
.background(.black.opacity(0.5), in: RoundedRectangle(cornerRadius: 6))
.foregroundStyle(.white)
.padding(10)
}
private var connectForm: some View {
VStack(spacing: 14) {
Text("lumen").font(.largeTitle.weight(.semibold))
Form {
TextField("Host", text: $host)
TextField("Port", value: $port, format: .number.grouping(.never))
HStack {
TextField("Width", value: $width, format: .number.grouping(.never))
Text("×")
TextField("Height", value: $height, format: .number.grouping(.never))
Text("@")
TextField("Hz", value: $hz, format: .number.grouping(.never))
}
Button("Use this display's mode") { fillFromMainScreen() }
.buttonStyle(.link)
}
.frame(width: 340)
if let error = model.errorMessage {
Text(error)
.font(.caption)
.foregroundStyle(.red)
.frame(width: 340)
}
Button(model.connecting ? "Connecting…" : "Connect") {
model.connect(
host: host, port: UInt16(clamping: port),
width: UInt32(clamping: width), height: UInt32(clamping: height),
hz: UInt32(clamping: hz))
}
.keyboardShortcut(.defaultAction)
.disabled(model.connecting || host.isEmpty)
}
.padding(28)
.frame(minWidth: 420, minHeight: 320)
}
private func fillFromMainScreen() {
guard let screen = NSScreen.main else { return }
let scale = screen.backingScaleFactor
width = Int(screen.frame.width * scale)
height = Int(screen.frame.height * scale)
hz = screen.maximumFramesPerSecond
}
}
@@ -0,0 +1,29 @@
// LumenClient development app shell around LumenKit (swift run LumenClient).
// Connect form StreamView (AVSampleBufferDisplayLayer HEVC) + InputCapture.
import AppKit
import SwiftUI
@main
struct LumenClientApp: App {
@NSApplicationDelegateAdaptor(AppDelegate.self) private var appDelegate
var body: some Scene {
WindowGroup("lumen") {
ContentView()
}
}
}
final class AppDelegate: NSObject, NSApplicationDelegate {
func applicationDidFinishLaunching(_ notification: Notification) {
// `swift run` launches an unbundled binary; promote it to a regular app so the
// window fronts and receives keyboard/mouse focus (GameController needs focus).
NSApp.setActivationPolicy(.regular)
NSApp.activate(ignoringOtherApps: true)
}
func applicationShouldTerminateAfterLastWindowClosed(_ sender: NSApplication) -> Bool {
true
}
}
@@ -0,0 +1,115 @@
// Session state for the app shell: owns the connection, the input capture, and the
// pump-thread main-actor stats relay.
import Foundation
import LumenKit
import SwiftUI
/// Pump-thread-side frame counters; a 1 Hz main-actor timer drains them into @Published
/// values. NSLock instead of an actor the writer is the (non-async) pump thread.
final class FrameMeter: @unchecked Sendable {
private let lock = NSLock()
private var frames = 0
private var bytes = 0
private var totalFrames = 0
func note(byteCount: Int) {
lock.lock()
frames += 1
bytes += byteCount
totalFrames += 1
lock.unlock()
}
/// Returns and resets the per-interval counters (the running total stays).
func drain() -> (frames: Int, bytes: Int, total: Int) {
lock.lock()
defer {
frames = 0
bytes = 0
lock.unlock()
}
return (frames, bytes, totalFrames)
}
}
@MainActor
final class SessionModel: ObservableObject {
@Published var connection: LumenConnection?
@Published var connecting = false
@Published var errorMessage: String?
@Published var fps = 0
@Published var mbps = 0.0
@Published var totalFrames = 0
let meter = FrameMeter()
private var inputCapture: InputCapture?
private var statsTimer: Timer?
func connect(host: String, port: UInt16, width: UInt32, height: UInt32, hz: UInt32) {
guard !connecting else { return }
connecting = true
errorMessage = nil
Task.detached(priority: .userInitiated) {
// LumenConnection.init blocks on the QUIC handshake keep it off the main actor.
let result = Result { try LumenConnection(
host: host, port: port, width: width, height: height, refreshHz: hz) }
await MainActor.run { [weak self] in
guard let self else { return }
self.connecting = false
switch result {
case .success(let conn):
self.connection = conn
self.startInput(conn)
self.startStatsTimer()
case .failure:
self.errorMessage = "Connection failed — is the host running? " +
"(lumen-host m3-host on \(host):\(port))"
}
}
}
}
func disconnect() {
inputCapture?.stop()
inputCapture = nil
statsTimer?.invalidate()
statsTimer = nil
if let conn = connection {
// close() waits out an in-flight poll (100 ms) and joins the Rust worker
// threads keep that off the main actor.
Task.detached { conn.close() }
}
connection = nil
fps = 0
mbps = 0
}
/// Called (via the main actor) when the pump hits end-of-session.
func sessionEnded() {
guard connection != nil else { return }
disconnect()
errorMessage = "Session ended by host."
}
private func startInput(_ conn: LumenConnection) {
let capture = InputCapture(connection: conn)
capture.start()
inputCapture = capture
}
private func startStatsTimer() {
let timer = Timer(timeInterval: 1.0, repeats: true) { [weak self] _ in
guard let self else { return }
Task { @MainActor in
let (frames, bytes, total) = self.meter.drain()
self.fps = frames
self.mbps = Double(bytes) * 8 / 1_000_000
self.totalFrames = total
}
}
// .common so the HUD keeps updating during window drags / menu tracking.
RunLoop.main.add(timer, forMode: .common)
statsTimer = timer
}
}
+8 -2
View File
@@ -12,6 +12,9 @@ import Foundation
public enum AnnexB {
/// Split an Annex-B stream into NAL units (start codes 00 00 01 / 00 00 00 01 stripped).
/// All zeros immediately preceding a start code are dropped: they're either the
/// 4-byte-code prefix or `trailing_zero_8bits` padding, never NAL payload (emulation
/// prevention keeps 00 00 0x out of conforming NAL bytes) same policy as ffmpeg.
public static func nalUnits(in data: Data) -> [Data] {
var nals: [Data] = []
let bytes = [UInt8](data)
@@ -19,8 +22,11 @@ public enum AnnexB {
var start = -1
while i + 2 < bytes.count {
if bytes[i] == 0, bytes[i + 1] == 0, bytes[i + 2] == 1 {
let codeStart = (i > 0 && bytes[i - 1] == 0) ? i - 1 : i
if start >= 0 {
var codeStart = i
while codeStart > 0, bytes[codeStart - 1] == 0 {
codeStart -= 1
}
if start >= 0, start < codeStart {
nals.append(Data(bytes[start..<codeStart]))
}
start = i + 3
+143 -21
View File
@@ -4,26 +4,50 @@
// injector expects for relative motion. GCKeyboard gives HID keycodes which we map to the
// Windows VK space the host's vk_to_evdev table consumes (same space Moonlight uses).
// Gamepads (GCController) come later the host's uinput pads already speak the
// GamepadButton/GamepadAxis event kinds.
// GamepadButton/GamepadAxis event kinds, but m3's injector path doesn't route them yet.
//
// SCAFFOLD: written on the Linux host, not yet compiled against Xcode. The VK map covers
// the common keys; extend alongside lumen-host/src/inject.rs::vk_to_evdev.
// The wire carries integer deltas; GC hands us Floats. We accumulate the fractional
// remainder per axis so slow, sub-pixel motion isn't truncated away.
//
// GC only delivers while the app is active, so anything held when focus leaves would
// stick down on the host forever we track pressed keys/buttons and release them all on
// didResignActive and on stop(). All GC handlers and notifications fire on the main
// queue (the framework default), so the mutable state here needs no locking.
//
// GCMouse.current/GCKeyboard.coalesced are process-global singletons with one handler
// slot each: only one InputCapture can be live per process. `activeCapture` tracks
// ownership so a stale capture's stop() can't clobber a newer one's handlers.
#if os(macOS)
import AppKit
import Foundation
import GameController
import LumenCore
public final class InputCapture {
private static weak var activeCapture: InputCapture?
private let connection: LumenConnection
private var observers: [NSObjectProtocol] = []
private var mice: [GCMouse] = []
private var keyboards: [GCKeyboard] = []
// Main-queue-only state (see header comment).
private var residualX: Float = 0
private var residualY: Float = 0
private var residualScrollX: Float = 0
private var residualScrollY: Float = 0
private var pressedVKs: Set<UInt32> = []
private var pressedButtons: Set<UInt32> = []
public init(connection: LumenConnection) {
self.connection = connection
}
/// Begin forwarding the current (and future) mouse/keyboard to the host.
/// Begin forwarding the current (and future) mouse/keyboard to the host. Steals the
/// global GC handler slots from any previous capture (one live capture per process).
public func start() {
Self.activeCapture = self
if let mouse = GCMouse.current { attach(mouse: mouse) }
if let keyboard = GCKeyboard.coalesced { attach(keyboard: keyboard) }
observers.append(NotificationCenter.default.addObserver(
@@ -36,44 +60,130 @@ public final class InputCapture {
) { [weak self] n in
if let k = n.object as? GCKeyboard { self?.attach(keyboard: k) }
})
// Focus loss: GC stops delivering, so release everything still held host-side.
observers.append(NotificationCenter.default.addObserver(
forName: NSApplication.didResignActiveNotification, object: nil, queue: .main
) { [weak self] _ in
self?.releaseAll()
})
}
public func stop() {
releaseAll()
observers.forEach(NotificationCenter.default.removeObserver(_:))
observers.removeAll()
// Don't clobber the handlers if a newer capture has taken the global devices.
if Self.activeCapture === self || Self.activeCapture == nil {
for mouse in mice {
guard let input = mouse.mouseInput else { continue }
input.mouseMovedHandler = nil
input.leftButton.pressedChangedHandler = nil
input.rightButton?.pressedChangedHandler = nil
input.middleButton?.pressedChangedHandler = nil
input.auxiliaryButtons?.forEach { $0.pressedChangedHandler = nil }
input.scroll.valueChangedHandler = nil
}
for keyboard in keyboards {
keyboard.keyboardInput?.keyChangedHandler = nil
}
Self.activeCapture = nil
}
mice.removeAll()
keyboards.removeAll()
}
deinit { stop() }
/// Send release events for everything currently held, and drop the motion residuals.
private func releaseAll() {
for vk in pressedVKs {
connection.send(.key(vk, down: false))
}
for button in pressedButtons {
connection.send(.mouseButton(button, down: false))
}
pressedVKs.removeAll()
pressedButtons.removeAll()
residualX = 0
residualY = 0
residualScrollX = 0
residualScrollY = 0
}
private func sendButton(_ button: UInt32, pressed: Bool) {
if pressed {
pressedButtons.insert(button)
} else {
pressedButtons.remove(button)
}
connection.send(.mouseButton(button, down: pressed))
}
private func attach(mouse: GCMouse) {
guard let input = mouse.mouseInput else { return }
let conn = connection
input.mouseMovedHandler = { _, dx, dy in
guard let input = mouse.mouseInput,
!mice.contains(where: { $0 === mouse }) // re-delivered on wake attach once
else { return }
mice.append(mouse)
input.mouseMovedHandler = { [weak self] _, dx, dy in
guard let self else { return }
// GC gives +y up; the host expects screen-space (+y down).
conn.send(.mouseMove(dx: Int32(dx), dy: Int32(-dy)))
let fx = dx + self.residualX
let fy = -dy + self.residualY
let ix = fx.rounded(.towardZero)
let iy = fy.rounded(.towardZero)
self.residualX = fx - ix
self.residualY = fy - iy
if ix != 0 || iy != 0 {
self.connection.send(.mouseMove(dx: Int32(ix), dy: Int32(iy)))
}
}
input.leftButton.pressedChangedHandler = { _, _, pressed in
conn.send(.mouseButton(1, down: pressed))
input.leftButton.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(1, pressed: pressed)
}
input.rightButton?.pressedChangedHandler = { _, _, pressed in
conn.send(.mouseButton(3, down: pressed))
input.rightButton?.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(3, pressed: pressed)
}
input.middleButton?.pressedChangedHandler = { _, _, pressed in
conn.send(.mouseButton(2, down: pressed))
input.middleButton?.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(2, pressed: pressed)
}
input.scroll.valueChangedHandler = { _, _, dy in
if dy != 0 { conn.send(.scroll(Int32(dy * 120))) }
// First two side buttons GameStream X1/X2.
if let aux = input.auxiliaryButtons {
for (i, button) in aux.prefix(2).enumerated() {
button.pressedChangedHandler = { [weak self] _, _, pressed in
self?.sendButton(UInt32(4 + i), pressed: pressed)
}
}
}
input.scroll.valueChangedHandler = { [weak self] _, x, y in
guard let self else { return }
// WHEEL_DELTA(120) per notch; positive = up / right (Moonlight's convention).
let fy = y * 120 + self.residualScrollY
let fx = x * 120 + self.residualScrollX
let iy = fy.rounded(.towardZero)
let ix = fx.rounded(.towardZero)
self.residualScrollY = fy - iy
self.residualScrollX = fx - ix
if iy != 0 { self.connection.send(.scroll(Int32(iy))) }
if ix != 0 { self.connection.send(.scroll(Int32(ix), horizontal: true)) }
}
}
private func attach(keyboard: GCKeyboard) {
let conn = connection
keyboard.keyboardInput?.keyChangedHandler = { _, _, keyCode, pressed in
if let vk = Self.hidToVK[keyCode.rawValue] {
conn.send(.key(vk, down: pressed))
guard !keyboards.contains(where: { $0 === keyboard }) else { return }
keyboards.append(keyboard)
keyboard.keyboardInput?.keyChangedHandler = { [weak self] _, _, keyCode, pressed in
guard let self, let vk = Self.hidToVK[keyCode.rawValue] else { return }
if pressed {
self.pressedVKs.insert(vk)
} else {
self.pressedVKs.remove(vk)
}
self.connection.send(.key(vk, down: pressed))
}
}
/// HID usage (GCKeyCode raw) Windows VK (the host maps VK evdev).
/// HID usage (GCKeyCode raw) Windows VK (the host maps VK evdev; every VK emitted
/// here exists in lumen-host/src/inject.rs::vk_to_evdev extend the two together).
static let hidToVK: [Int: UInt32] = {
var m: [Int: UInt32] = [:]
// az: HID 0x04..0x1D VK 'A'..'Z'.
@@ -90,11 +200,23 @@ public final class InputCapture {
m[0x2F] = 0xDB; m[0x30] = 0xDD; m[0x31] = 0xDC // [ ] backslash
m[0x33] = 0xBA; m[0x34] = 0xDE; m[0x35] = 0xC0 // ; ' `
m[0x36] = 0xBC; m[0x37] = 0xBE; m[0x38] = 0xBF // , . /
m[0x39] = 0x14 // caps lock
// F1..F12: HID 0x3A..0x45 VK 0x70..0x7B.
for i in 0..<12 { m[0x3A + i] = UInt32(0x70 + i) }
m[0x46] = 0x2C; m[0x47] = 0x91; m[0x48] = 0x13 // printscreen scrolllock pause
m[0x4F] = 0x27; m[0x50] = 0x25; m[0x51] = 0x28; m[0x52] = 0x26 // arrows R L D U
m[0x49] = 0x2D; m[0x4A] = 0x24; m[0x4B] = 0x21 // insert home pageup
m[0x4C] = 0x2E; m[0x4D] = 0x23; m[0x4E] = 0x22 // delete end pagedown
// Keypad: NumLock, / * - +, Enter, 1..9, 0, decimal. KP Enter goes as
// VK_SEPARATOR (0x6C) this host maps it to KEY_KPENTER (Windows itself would
// send VK_RETURN+extended, which vk_to_evdev can't distinguish).
m[0x53] = 0x90
m[0x54] = 0x6F; m[0x55] = 0x6A; m[0x56] = 0x6D; m[0x57] = 0x6B
m[0x58] = 0x6C
for i in 0..<9 { m[0x59 + i] = UInt32(0x61 + i) }
m[0x62] = 0x60; m[0x63] = 0x6E
m[0x64] = 0xE2 // ISO 102nd key (<> next to left shift on ISO layouts)
m[0x65] = 0x5D // menu/application
m[0xE0] = 0xA2; m[0xE1] = 0xA0; m[0xE2] = 0xA4; m[0xE3] = 0x5B // Lctrl Lshift Lalt Lcmd
m[0xE4] = 0xA3; m[0xE5] = 0xA1; m[0xE6] = 0xA5; m[0xE7] = 0x5C // Rctrl Rshift Ralt Rcmd
return m
@@ -1,8 +1,9 @@
// Swift wrapper around the lumen-core C ABI's lumen/1 connection API.
//
// Threading contract (mirrors the C header): one LumenConnection is used from a single
// pump thread for nextAU(); nextAudio() may run on its own (single) audio thread;
// sendInput() is enqueue-only and safe alongside both. The pointers inside an AU/audio
// Threading contract (mirrors the C header): one LumenConnection is pumped from a single
// video thread via nextAU(); nextAudio()/nextRumble() may each run on their own (single)
// drain thread the core keeps per-plane borrow slots, so the planes never alias;
// send() is enqueue-only and safe alongside all of them. The pointers inside an AU/audio
// packet are only valid until the next call of the same kind, so we copy into Data here
// the copies are small and keep the Swift side memory-safe.
//
@@ -10,12 +11,22 @@
// `hostFingerprint` reports what a trust-on-first-use connect observed persist it, e.g.
// in UserDefaults keyed by host, and pin it from then on).
//
// SCAFFOLD: written on the Linux host, not yet compiled against Xcode expect to fix
// trivial issues on first build (see README.md "Handoff").
// close() is safe from any thread: it flags the pullers to exit at their next poll
// boundary, then takes the per-plane locks (each held across its blocking C poll), so the
// handle is never freed under an in-flight call the C contract ("never close with a
// next_au/next_audio call in flight") is enforced here rather than left to callers. After
// close, the pull methods throw `.closed` and the threads unwind on their own.
import Foundation
import LumenCore
// cbindgen's C17-compatible header spells the typedefs as plain integers
// (`typedef int32_t LumenStatus`, `typedef uint8_t LumenInputKind`) while the enum
// constants import as a distinct same-named Swift type bridge by raw value once here.
private let statusOK: Int32 = LUMEN_STATUS_OK.rawValue
private let statusNoFrame: Int32 = LUMEN_STATUS_NO_FRAME.rawValue
private let statusClosed: Int32 = LUMEN_STATUS_CLOSED.rawValue
/// One reassembled, FEC-recovered, decrypted access unit (Annex-B HEVC from the host).
public struct AccessUnit: Sendable {
public let data: Data
@@ -39,10 +50,22 @@ public enum LumenClientError: Error {
/// unpinned when the caller asked for verification would be a silent trust downgrade.
case invalidPin
case closed
case status(Int32)
}
public final class LumenConnection {
private var handle: OpaquePointer?
/// Set by close() before it contends for the plane locks: the pullers see it at their
/// next poll boundary and exit, so close() can't be starved by back-to-back polls
/// (NSLock is not fair).
private var closeRequested = false
/// Serializes send()/close() against each other and guards `handle`/`closeRequested`.
private let abiLock = NSLock()
/// Held across the blocking next_au call; close() takes it (same plane-lock abiLock
/// order as the pullers) so it can never free the handle under an in-flight poll.
private let pumpLock = NSLock()
/// Same role for the audio/rumble drain thread (its own plane in the core).
private let audioLock = NSLock()
/// Negotiated session mode (host-confirmed).
public private(set) var width: UInt32 = 0
@@ -86,87 +109,141 @@ public final class LumenConnection {
self.refreshHz = hz
}
/// Pull the next access unit; nil on timeout, throws once the session is closed.
/// Pull the next access unit; nil on timeout, throws `.closed` once the session ended.
/// Call from a single pump thread.
public func nextAU(timeoutMs: UInt32 = 100) throws -> AccessUnit? {
pumpLock.lock()
defer { pumpLock.unlock() }
guard let h = liveHandle() else { throw LumenClientError.closed }
var frame = LumenFrame()
switch lumen_connection_next_au(handle, &frame, timeoutMs) {
case LUMEN_STATUS_OK:
let data = Data(bytes: frame.data, count: frame.len) // copy: ptr valid only until next call
let rc = lumen_connection_next_au(h, &frame, timeoutMs)
switch rc {
case statusOK:
guard let base = frame.data, frame.len > 0 else { return nil }
let data = Data(bytes: base, count: Int(frame.len)) // copy: ptr valid only until next call
return AccessUnit(
data: data, ptsNs: frame.pts_ns,
frameIndex: frame.frame_index, flags: frame.flags)
case LUMEN_STATUS_NO_FRAME:
case statusNoFrame:
return nil
case LUMEN_STATUS_CLOSED:
case statusClosed:
throw LumenClientError.closed
default:
throw LumenClientError.closed
throw LumenClientError.status(rc)
}
}
/// Pull the next Opus audio packet; nil on timeout, throws once the session is closed.
/// Drain from a dedicated audio thread packets arrive every 5 ms (320 ms buffered).
/// Pull the next Opus audio packet; nil on timeout, throws `.closed` once the session
/// ended. Drain from a dedicated audio thread packets arrive every 5 ms (the core
/// buffers 320 ms and drops the newest when the puller lags).
public func nextAudio(timeoutMs: UInt32 = 100) throws -> AudioPacket? {
audioLock.lock()
defer { audioLock.unlock() }
guard let h = liveHandle() else { throw LumenClientError.closed }
var pkt = LumenAudioPacket()
switch lumen_connection_next_audio(handle, &pkt, timeoutMs) {
case LUMEN_STATUS_OK:
let data = Data(bytes: pkt.data, count: pkt.len) // copy: ptr valid only until next call
let rc = lumen_connection_next_audio(h, &pkt, timeoutMs)
switch rc {
case statusOK:
guard let base = pkt.data, pkt.len > 0 else { return nil }
let data = Data(bytes: base, count: Int(pkt.len)) // copy: ptr valid only until next call
return AudioPacket(data: data, ptsNs: pkt.pts_ns, seq: pkt.seq)
case LUMEN_STATUS_NO_FRAME:
case statusNoFrame:
return nil
default:
case statusClosed:
throw LumenClientError.closed
default:
throw LumenClientError.status(rc)
}
}
/// Pull the next force-feedback update for the GCController haptics engine:
/// `(pad, lowFrequency, highFrequency)` with 0...0xFFFF amplitudes, (0, 0) = stop.
public func nextRumble(timeoutMs: UInt32 = 100) throws -> (pad: UInt16, low: UInt16, high: UInt16)? {
/// Shares the audio drain thread's plane (call from that thread).
public func nextRumble(timeoutMs: UInt32 = 0) throws -> (pad: UInt16, low: UInt16, high: UInt16)? {
audioLock.lock()
defer { audioLock.unlock() }
guard let h = liveHandle() else { throw LumenClientError.closed }
var pad: UInt16 = 0, low: UInt16 = 0, high: UInt16 = 0
switch lumen_connection_next_rumble(handle, &pad, &low, &high, timeoutMs) {
case LUMEN_STATUS_OK:
let rc = lumen_connection_next_rumble(h, &pad, &low, &high, timeoutMs)
switch rc {
case statusOK:
return (pad, low, high)
case LUMEN_STATUS_NO_FRAME:
case statusNoFrame:
return nil
default:
case statusClosed:
throw LumenClientError.closed
default:
throw LumenClientError.status(rc)
}
}
/// Send one input event (delivered to the host as a QUIC datagram).
/// Send one input event (delivered to the host as a QUIC datagram). Thread-safe;
/// silently dropped after close.
public func send(_ event: LumenInputEvent) {
var ev = event
_ = lumen_connection_send_input(handle, &ev)
abiLock.lock()
defer { abiLock.unlock() }
guard let h = handle, !closeRequested else { return }
_ = lumen_connection_send_input(h, &ev)
}
/// Close the connection and free the handle. Safe from any thread, idempotent; waits
/// for in-flight pulls ( their timeouts) before tearing down.
public func close() {
if let h = handle {
lumen_connection_close(h)
handle = nil
abiLock.lock()
closeRequested = true
abiLock.unlock()
pumpLock.lock() // pullers exit at their next poll boundary, releasing these
audioLock.lock()
abiLock.lock()
let h = handle
handle = nil
abiLock.unlock()
audioLock.unlock()
pumpLock.unlock()
if let h {
lumen_connection_close(h) // joins the connection's internal Rust threads
}
}
deinit { close() }
/// Snapshot the handle unless close is pending (callers hold their plane lock).
private func liveHandle() -> OpaquePointer? {
abiLock.lock()
defer { abiLock.unlock() }
return closeRequested ? nil : handle
}
}
// Convenience constructors for the wire input events (field semantics match
// lumen_core::input::InputEvent; see lumen_core.h).
public extension LumenInputEvent {
private static func make(
_ kind: UInt32, code: UInt32, x: Int32, y: Int32, flags: UInt32 = 0
) -> LumenInputEvent {
LumenInputEvent(kind: UInt8(kind), _pad: (0, 0, 0), code: code, x: x, y: y, flags: flags)
}
static func mouseMove(dx: Int32, dy: Int32) -> LumenInputEvent {
LumenInputEvent(kind: LUMEN_INPUT_KIND_MOUSE_MOVE, _pad: (0, 0, 0), code: 0, x: dx, y: dy, flags: 0)
make(LUMEN_INPUT_KIND_MOUSE_MOVE.rawValue, code: 0, x: dx, y: dy)
}
/// GameStream button ids: 1=left 2=middle 3=right 4=X1 5=X2 (host maps to evdev BTN_*).
static func mouseButton(_ button: UInt32, down: Bool) -> LumenInputEvent {
LumenInputEvent(
kind: down ? LUMEN_INPUT_KIND_MOUSE_BUTTON_DOWN : LUMEN_INPUT_KIND_MOUSE_BUTTON_UP,
_pad: (0, 0, 0), code: button, x: 0, y: 0, flags: 0)
make(
(down ? LUMEN_INPUT_KIND_MOUSE_BUTTON_DOWN : LUMEN_INPUT_KIND_MOUSE_BUTTON_UP).rawValue,
code: button, x: 0, y: 0)
}
/// `vk` is a Windows virtual-key code (the host's vk_to_evdev table consumes these).
static func key(_ vk: UInt32, down: Bool) -> LumenInputEvent {
LumenInputEvent(
kind: down ? LUMEN_INPUT_KIND_KEY_DOWN : LUMEN_INPUT_KIND_KEY_UP,
_pad: (0, 0, 0), code: vk, x: 0, y: 0, flags: 0)
make((down ? LUMEN_INPUT_KIND_KEY_DOWN : LUMEN_INPUT_KIND_KEY_UP).rawValue, code: vk, x: 0, y: 0)
}
static func scroll(_ delta: Int32) -> LumenInputEvent {
LumenInputEvent(kind: LUMEN_INPUT_KIND_MOUSE_SCROLL, _pad: (0, 0, 0), code: 0, x: delta, y: 0, flags: 0)
/// WHEEL_DELTA(120)-scaled; positive = up (vertical) / right (horizontal) the
/// convention Moonlight/SDL use; the host maps onto the ei/wl axes.
static func scroll(_ delta: Int32, horizontal: Bool = false) -> LumenInputEvent {
make(LUMEN_INPUT_KIND_MOUSE_SCROLL.rawValue, code: horizontal ? 1 : 0, x: delta, y: 0)
}
// Gamepad (wire contract in lumen_core::input::gamepad): one transition per event,
@@ -175,16 +252,14 @@ public extension LumenInputEvent {
/// `button` is a GameStream buttonFlags bit (A=0x1000 B=0x2000 X=0x4000 Y=0x8000,
/// dpad=0x1/2/4/8, start=0x10 back=0x20 LS=0x40 RS=0x80 LB=0x100 RB=0x200 guide=0x400).
static func gamepadButton(_ button: UInt32, down: Bool, pad: UInt32 = 0) -> LumenInputEvent {
LumenInputEvent(
kind: LUMEN_INPUT_KIND_GAMEPAD_BUTTON,
_pad: (0, 0, 0), code: button, x: down ? 1 : 0, y: 0, flags: pad)
make(
LUMEN_INPUT_KIND_GAMEPAD_BUTTON.rawValue,
code: button, x: down ? 1 : 0, y: 0, flags: pad)
}
/// Axis ids: 0=LSX 1=LSY 2=RSX 3=RSY (32768...32767, XInput convention: +y = UP
/// `GCControllerDirectionPad.yAxis` already matches, no flip), 4=LT 5=RT (0...255).
static func gamepadAxis(_ axis: UInt32, value: Int32, pad: UInt32 = 0) -> LumenInputEvent {
LumenInputEvent(
kind: LUMEN_INPUT_KIND_GAMEPAD_AXIS,
_pad: (0, 0, 0), code: axis, x: value, y: 0, flags: pad)
make(LUMEN_INPUT_KIND_GAMEPAD_AXIS.rawValue, code: axis, x: value, y: 0, flags: pad)
}
}
+80 -20
View File
@@ -5,8 +5,8 @@
// zero-copy on Apple silicon. Stage 2 (explicit VTDecompressionSession + CAMetalLayer)
// replaces this when we start tuning frame pacing / measuring glass-to-glass.
//
// SCAFFOLD: written on the Linux host, not yet compiled against Xcode. macOS-first
// (NSViewRepresentable); the iOS variant is the same layer under UIViewRepresentable.
// macOS-first (NSViewRepresentable); the iOS variant is the same layer under
// UIViewRepresentable.
#if os(macOS)
import AVFoundation
@@ -14,70 +14,130 @@ import SwiftUI
public struct StreamView: NSViewRepresentable {
private let connection: LumenConnection
private let onFrame: (@Sendable (AccessUnit) -> Void)?
private let onSessionEnd: (@Sendable () -> Void)?
public init(connection: LumenConnection) {
/// `onFrame`/`onSessionEnd` fire on the pump thread hop to the main actor for UI.
public init(
connection: LumenConnection,
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
onSessionEnd: (@Sendable () -> Void)? = nil
) {
self.connection = connection
self.onFrame = onFrame
self.onSessionEnd = onSessionEnd
}
public func makeNSView(context: Context) -> StreamLayerView {
let view = StreamLayerView()
view.start(connection: connection)
view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
return view
}
public func updateNSView(_ view: StreamLayerView, context: Context) {}
public func updateNSView(_ view: StreamLayerView, context: Context) {
// SwiftUI reuses the NSView across state changes repoint the pump only when the
// connection identity actually changed.
if view.connection !== connection {
view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
}
}
public static func dismantleNSView(_ view: StreamLayerView, coordinator: ()) {
view.stop()
}
}
public final class StreamLayerView: NSView {
/// Cancellation handle owned by exactly one pump thread a restart hands the old pump
/// its own token, so it can never be revived by a newer start().
private final class PumpToken: @unchecked Sendable {
private let lock = NSLock()
private var live = true
var isLive: Bool {
lock.lock()
defer { lock.unlock() }
return live
}
func cancel() {
lock.lock()
live = false
lock.unlock()
}
}
private let displayLayer = AVSampleBufferDisplayLayer()
private var pump: Thread?
private var running = false
private var token: PumpToken?
public private(set) var connection: LumenConnection?
public override init(frame: NSRect) {
super.init(frame: frame)
wantsLayer = true
displayLayer.videoGravity = .resizeAspect
layer = displayLayer
layer = displayLayer // layer-hosting: assign before wantsLayer
wantsLayer = true
}
public required init?(coder: NSCoder) { fatalError("not used") }
/// Pump thread: pull AUs from the connection, wrap, enqueue. The first IDR yields the
/// format description; non-IDR AUs before it are dropped (the host opens with an IDR).
public func start(connection: LumenConnection) {
guard !running else { return }
running = true
public func start(
connection: LumenConnection,
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
onSessionEnd: (@Sendable () -> Void)? = nil
) {
stop()
let token = PumpToken()
self.token = token
self.connection = connection
let layer = displayLayer
let thread = Thread { [weak self] in
layer.flush() // drop any frames a previous connection left queued
let thread = Thread {
var format: CMVideoFormatDescription?
while self?.running == true {
while token.isLive {
do {
guard let au = try connection.nextAU(timeoutMs: 100) else { continue }
onFrame?(au)
if let f = AnnexB.formatDescription(fromIDR: au.data) {
format = f // refreshed on every IDR (mode changes included)
}
guard let f = format,
let sample = AnnexB.sampleBuffer(au: au, format: f)
else { continue }
if layer.status == .failed {
// Decode wedged: flush and re-gate on the next in-band parameter
// sets resuming with a delta frame can't recover. (A
// request-IDR channel on lumen/1 is a host-side TODO; with the
// host's infinite GOP this may otherwise stay black until the
// next recovery keyframe.)
layer.flush()
format = AnnexB.formatDescription(fromIDR: au.data)
}
guard let f = format,
let sample = AnnexB.sampleBuffer(au: au, format: f),
token.isLive // don't enqueue a stale frame after a restart
else { continue }
layer.enqueue(sample)
} catch {
if token.isLive {
onSessionEnd?()
}
break // session closed
}
}
}
thread.name = "lumen-pump"
thread.qualityOfService = .userInteractive
pump = thread
thread.start()
}
/// Stop pumping ( one poll timeout). Does not close the connection that stays with
/// whoever owns it (LumenConnection.close() is safe alongside a draining pump).
public func stop() {
running = false
token?.cancel()
token = nil
connection = nil
}
deinit { running = false }
deinit {
token?.cancel()
}
}
#endif
@@ -0,0 +1,79 @@
// Unit tests for the Annex-B AVCC plumbing (pure byte-level; no codec involved
// VideoToolboxRoundTripTests covers the real-bitstream path).
import XCTest
@testable import LumenKit
final class AnnexBTests: XCTestCase {
/// NAL with the given HEVC type in bits 1..6 of the first header byte.
private func nal(type: UInt8, payload: [UInt8]) -> Data {
Data([type << 1, 0x01] + payload)
}
private let start4: [UInt8] = [0, 0, 0, 1]
private let start3: [UInt8] = [0, 0, 1]
func testSplitMixedStartCodes() {
let a = nal(type: 32, payload: [0xAA])
let b = nal(type: 33, payload: [0xBB, 0xBC])
let c = nal(type: 19, payload: [0xCC, 0xCD, 0xCE])
var au = Data(start4)
au.append(a)
au.append(contentsOf: start3)
au.append(b)
au.append(contentsOf: start4)
au.append(c)
let nals = AnnexB.nalUnits(in: au)
XCTAssertEqual(nals, [a, b, c])
XCTAssertEqual(nals.map(AnnexB.hevcNalType), [32, 33, 19])
}
func testSplitSingleNalNoTrailingCode() {
let v = nal(type: 34, payload: [1, 2, 3])
let au = Data(start3) + v
XCTAssertEqual(AnnexB.nalUnits(in: au), [v])
}
func testSplitEmptyAndGarbage() {
XCTAssertEqual(AnnexB.nalUnits(in: Data()), [])
// No start code at all no NALs.
XCTAssertEqual(AnnexB.nalUnits(in: Data([9, 8, 7, 6])), [])
}
func testSplitDropsTrailingZeroPadding() {
// trailing_zero_8bits between NALs (and >2 zeros forming a long separator) must
// not leak into the preceding NAL.
let a = nal(type: 33, payload: [0xAA])
let b = nal(type: 19, payload: [0xBB])
var au = Data(start4)
au.append(a)
au.append(contentsOf: [0, 0, 0, 0, 0, 1]) // padding + start code
au.append(b)
XCTAssertEqual(AnnexB.nalUnits(in: au), [a, b])
}
func testAvccDropsParameterSetsAndPrefixesLengths() {
let vps = nal(type: 32, payload: [0xAA])
let sps = nal(type: 33, payload: [0xBB])
let pps = nal(type: 34, payload: [0xCC])
let idr = nal(type: 19, payload: [0xDD, 0xDE, 0xDF, 0xE0])
var au = Data()
for n in [vps, sps, pps, idr] {
au.append(contentsOf: start4)
au.append(n)
}
let avcc = AnnexB.avcc(from: au)
// Only the IDR survives: 4-byte BE length, then the NAL bytes.
var expected = Data([0, 0, 0, UInt8(idr.count)])
expected.append(idr)
XCTAssertEqual(avcc, expected)
}
func testFormatDescriptionNilWithoutParameterSets() {
let idr = nal(type: 19, payload: [0xDD])
let au = Data(start4) + idr
XCTAssertNil(AnnexB.formatDescription(fromIDR: au))
}
}
@@ -0,0 +1,65 @@
// Integration: the Swift wrapper against a real lumen/1 host over QUIC + UDP on loopback
// the Swift twin of lumen-host's m3.rs::c_abi_connection_roundtrip, this time through the
// statically linked xcframework. Driven by clients/apple/test-loopback.sh, which builds and
// starts `lumen-host m3-host --source synthetic` and sets LUMEN_LOOPBACK_PORT.
import XCTest
@testable import LumenKit
final class LoopbackIntegrationTests: XCTestCase {
func testSyntheticStreamRoundTrip() throws {
guard let portStr = ProcessInfo.processInfo.environment["LUMEN_LOOPBACK_PORT"],
let port = UInt16(portStr)
else {
throw XCTSkip("needs a running m3-host — use clients/apple/test-loopback.sh")
}
let conn = try LumenConnection(
host: "127.0.0.1", port: port, width: 1280, height: 720, refreshHz: 60)
XCTAssertEqual(conn.width, 1280)
XCTAssertEqual(conn.height, 720)
XCTAssertEqual(conn.refreshHz, 60)
// Pull 25 synthetic frames and byte-verify the documented pattern:
// u32 LE frame index, then data[i] = (idx as u8) &+ (i as u8).
var got = 0
var lastIndex: UInt32 = 0
let deadline = Date().addingTimeInterval(30)
while got < 25 {
XCTAssertLessThan(Date(), deadline, "timed out after \(got) frames")
guard let au = try conn.nextAU(timeoutMs: 2000) else { continue }
let idx = au.data.prefix(4).reversed().reduce(UInt32(0)) { ($0 << 8) | UInt32($1) }
for (i, byte) in au.data.enumerated().dropFirst(4) {
let expected = UInt8(truncatingIfNeeded: idx) &+ UInt8(truncatingIfNeeded: i)
if byte != expected {
XCTFail("frame \(idx) corrupt at offset \(i)")
break
}
}
XCTAssertGreaterThan(au.ptsNs, 0)
lastIndex = idx
got += 1
}
XCTAssertGreaterThanOrEqual(lastIndex, 24)
// Input goes the other way (enqueue-only; the host logs the count on close).
conn.send(.mouseMove(dx: 1, dy: 2))
conn.send(.key(0x41, down: true))
conn.send(.key(0x41, down: false))
conn.close()
XCTAssertThrowsError(try conn.nextAU(timeoutMs: 10)) { error in
guard case LumenClientError.closed = error else {
return XCTFail("expected .closed, got \(error)")
}
}
}
func testConnectFailureThrows() {
// Nothing listens on this port; connect must fail within its timeout, not hang.
XCTAssertThrowsError(
try LumenConnection(
host: "127.0.0.1", port: 9, width: 640, height: 480, refreshHz: 30,
timeoutMs: 2000))
}
}
@@ -0,0 +1,80 @@
// First light, headless: the full client pipeline against a REAL remote host QUIC
// handshake over the LAN, NVENC HEVC AUs through FEC + AES-GCM, AnnexB conversion, and a
// real VTDecompressionSession turning them into pixels. Everything the GUI does except
// putting the layer on glass.
//
// Run (host side, on the Linux box):
// LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 \
// lumen-host m3-host --source virtual --seconds 120
// Then here:
// LUMEN_REMOTE_HOST=192.168.1.70 swift test --filter RemoteFirstLightTests
import CoreMedia
import VideoToolbox
import XCTest
@testable import LumenKit
final class RemoteFirstLightTests: XCTestCase {
func testRemoteStreamDecodesToPixels() throws {
guard let host = ProcessInfo.processInfo.environment["LUMEN_REMOTE_HOST"] else {
throw XCTSkip("set LUMEN_REMOTE_HOST (and start m3-host --source virtual there)")
}
let width: UInt32 = 1280
let height: UInt32 = 720
let conn = try LumenConnection(
host: host, width: width, height: height, refreshHz: 60)
defer { conn.close() }
XCTAssertEqual(conn.width, width)
XCTAssertEqual(conn.height, height)
var format: CMVideoFormatDescription?
var decoder: VTDecompressionSession?
defer { decoder.map { VTDecompressionSessionInvalidate($0) } }
var received = 0
var decoded = 0
var firstPtsNs: UInt64 = 0
var lastPtsNs: UInt64 = 0
let deadline = Date().addingTimeInterval(30)
while decoded < 60, Date() < deadline {
guard let au = try conn.nextAU(timeoutMs: 2000) else { continue }
received += 1
if firstPtsNs == 0 { firstPtsNs = au.ptsNs }
lastPtsNs = au.ptsNs
if let f = AnnexB.formatDescription(fromIDR: au.data) {
format = f
if decoder == nil {
let dims = CMVideoFormatDescriptionGetDimensions(f)
XCTAssertEqual(UInt32(dims.width), width)
XCTAssertEqual(UInt32(dims.height), height)
var session: VTDecompressionSession?
XCTAssertEqual(
VTDecompressionSessionCreate(
allocator: nil, formatDescription: f, decoderSpecification: nil,
imageBufferAttributes: nil, outputCallback: nil,
decompressionSessionOut: &session),
noErr)
decoder = session
}
}
guard let f = format, let dec = decoder,
let sample = AnnexB.sampleBuffer(au: au, format: f)
else { continue }
var gotPixels = false
VTDecompressionSessionDecodeFrame(
dec, sampleBuffer: sample, flags: [], infoFlagsOut: nil
) { status, _, imageBuffer, _, _ in
gotPixels = status == noErr && imageBuffer != nil
}
if gotPixels { decoded += 1 }
}
XCTAssertGreaterThanOrEqual(decoded, 60, "decoded \(decoded)/\(received) received AUs")
// The host stamps pts with its capture wall clock 60 frames should span ~1 s.
let spanMs = Double(lastPtsNs &- firstPtsNs) / 1_000_000
print("first light: \(decoded) frames decoded, \(received) received, pts span \(Int(spanMs)) ms")
}
}
@@ -0,0 +1,176 @@
// Real-bitstream proof of the decode-prep path: VTCompressionSession encodes HEVC, we
// rebuild the host's wire shape (Annex-B AU with in-band VPS/SPS/PPS exactly what
// lumen-host emits on every IDR), run it through AnnexB, and hand the result to a real
// VTDecompressionSession. Pixels out = the whole client decode path is sound.
import AVFoundation
import CoreMedia
import VideoToolbox
import XCTest
@testable import LumenKit
final class VideoToolboxRoundTripTests: XCTestCase {
private let width = 320
private let height = 240
func testEncodeAnnexBDecodeRoundTrip() throws {
let (formatDesc, avccSample) = try encodeOneHEVCKeyframe()
// Rebuild the host's wire format: Annex-B AU, parameter sets in-band before the VCL.
let annexB = try annexBAU(formatDesc: formatDesc, avccSample: avccSample)
// 1) Parameter-set extraction format description.
let rebuilt = try XCTUnwrap(
AnnexB.formatDescription(fromIDR: annexB),
"in-band VPS/SPS/PPS should yield a format description")
let dims = CMVideoFormatDescriptionGetDimensions(rebuilt)
XCTAssertEqual(Int(dims.width), width)
XCTAssertEqual(Int(dims.height), height)
// 2) Annex-B AVCC re-pack must reproduce the encoder's own sample bytes.
XCTAssertEqual(AnnexB.avcc(from: annexB), avccSample)
// 3) Sample buffer real decoder pixels.
let au = AccessUnit(data: annexB, ptsNs: 1_000_000, frameIndex: 0, flags: 0)
let sample = try XCTUnwrap(AnnexB.sampleBuffer(au: au, format: rebuilt))
var session: VTDecompressionSession?
XCTAssertEqual(
VTDecompressionSessionCreate(
allocator: nil, formatDescription: rebuilt, decoderSpecification: nil,
imageBufferAttributes: nil, outputCallback: nil,
decompressionSessionOut: &session),
noErr)
let decoder = try XCTUnwrap(session)
defer { VTDecompressionSessionInvalidate(decoder) }
var decoded: CVImageBuffer?
var decodeStatus: OSStatus = -1
// No async flag the handler runs before DecodeFrame returns.
VTDecompressionSessionDecodeFrame(
decoder, sampleBuffer: sample, flags: [], infoFlagsOut: nil
) { status, _, imageBuffer, _, _ in
decodeStatus = status
decoded = imageBuffer
}
XCTAssertEqual(decodeStatus, noErr)
let pixels = try XCTUnwrap(decoded) // CVImageBuffer and CVPixelBuffer are the same CF type
XCTAssertEqual(CVPixelBufferGetWidth(pixels), width)
XCTAssertEqual(CVPixelBufferGetHeight(pixels), height)
}
// MARK: - encode helpers
/// One forced-IDR HEVC frame; returns its format description and raw AVCC sample bytes.
private func encodeOneHEVCKeyframe() throws -> (CMVideoFormatDescription, Data) {
var session: VTCompressionSession?
let rc = VTCompressionSessionCreate(
allocator: nil, width: Int32(width), height: Int32(height),
codecType: kCMVideoCodecType_HEVC, encoderSpecification: nil,
imageBufferAttributes: nil, compressedDataAllocator: nil,
outputCallback: nil, refcon: nil, compressionSessionOut: &session)
guard rc == noErr, let encoder = session else {
throw XCTSkip("no HEVC encoder available (\(rc))")
}
defer { VTCompressionSessionInvalidate(encoder) }
VTSessionSetProperty(encoder, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue)
VTSessionSetProperty(
encoder, key: kVTCompressionPropertyKey_AllowFrameReordering, value: kCFBooleanFalse)
let lock = NSLock()
var output: CMSampleBuffer?
let done = expectation(description: "encoded")
VTCompressionSessionEncodeFrame(
encoder, imageBuffer: try gradientPixelBuffer(),
presentationTimeStamp: CMTime(value: 0, timescale: 30),
duration: CMTime(value: 1, timescale: 30),
frameProperties: [kVTEncodeFrameOptionKey_ForceKeyFrame: kCFBooleanTrue] as CFDictionary,
infoFlagsOut: nil
) { status, _, sample in
XCTAssertEqual(status, noErr)
lock.lock()
output = sample
lock.unlock()
done.fulfill()
}
VTCompressionSessionCompleteFrames(encoder, untilPresentationTimeStamp: .invalid)
wait(for: [done], timeout: 10)
lock.lock()
defer { lock.unlock() }
let sample = try XCTUnwrap(output)
let desc = try XCTUnwrap(CMSampleBufferGetFormatDescription(sample))
let block = try XCTUnwrap(CMSampleBufferGetDataBuffer(sample))
var bytes = Data(count: CMBlockBufferGetDataLength(block))
try bytes.withUnsafeMutableBytes { raw in
let rc = CMBlockBufferCopyDataBytes(
block, atOffset: 0, dataLength: raw.count,
destination: raw.baseAddress!)
if rc != noErr { throw NSError(domain: "CMBlockBuffer", code: Int(rc)) }
}
return (desc, bytes)
}
/// The host's wire shape: 4-byte start codes, VPS/SPS/PPS in-band, then the VCL NALs.
private func annexBAU(formatDesc: CMVideoFormatDescription, avccSample: Data) throws -> Data {
var au = Data()
var psCount = 0
var nalHeaderLen: Int32 = 0
XCTAssertEqual(
CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(
formatDesc, parameterSetIndex: 0, parameterSetPointerOut: nil,
parameterSetSizeOut: nil, parameterSetCountOut: &psCount,
nalUnitHeaderLengthOut: &nalHeaderLen),
noErr)
XCTAssertEqual(nalHeaderLen, 4, "AnnexB.avcc assumes 4-byte NAL length prefixes")
for i in 0..<psCount {
var ptr: UnsafePointer<UInt8>?
var size = 0
XCTAssertEqual(
CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(
formatDesc, parameterSetIndex: i, parameterSetPointerOut: &ptr,
parameterSetSizeOut: &size, parameterSetCountOut: nil,
nalUnitHeaderLengthOut: nil),
noErr)
au.append(contentsOf: [0, 0, 0, 1])
au.append(Data(bytes: try XCTUnwrap(ptr), count: size))
}
// AVCC sample (4-byte BE length per NAL) start codes.
var i = avccSample.startIndex
while i + 4 <= avccSample.endIndex {
let len = avccSample[i..<i + 4].reduce(0) { ($0 << 8) | Int($1) }
let body = avccSample.index(i, offsetBy: 4)
guard let end = avccSample.index(body, offsetBy: len, limitedBy: avccSample.endIndex)
else { break }
au.append(contentsOf: [0, 0, 0, 1])
au.append(avccSample[body..<end])
i = end
}
return au
}
private func gradientPixelBuffer() throws -> CVPixelBuffer {
var pb: CVPixelBuffer?
let attrs = [kCVPixelBufferIOSurfacePropertiesKey: [:]] as CFDictionary
XCTAssertEqual(
CVPixelBufferCreate(nil, width, height, kCVPixelFormatType_32BGRA, attrs, &pb),
kCVReturnSuccess)
let buf = try XCTUnwrap(pb)
CVPixelBufferLockBaseAddress(buf, [])
defer { CVPixelBufferUnlockBaseAddress(buf, []) }
let base = try XCTUnwrap(CVPixelBufferGetBaseAddress(buf))
let stride = CVPixelBufferGetBytesPerRow(buf)
for y in 0..<height {
let row = base.advanced(by: y * stride).assumingMemoryBound(to: UInt8.self)
for x in 0..<width {
row[x * 4 + 0] = UInt8(x & 0xFF) // B
row[x * 4 + 1] = UInt8(y & 0xFF) // G
row[x * 4 + 2] = UInt8((x ^ y) & 0xFF) // R
row[x * 4 + 3] = 0xFF
}
}
return buf
}
}
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Loopback integration: a real lumen/1 host (synthetic source — pure protocol, runs fine on
# macOS) on 127.0.0.1, then the Swift integration tests against it through the xcframework.
# The m3 host serves exactly one session and exits; the trap is just for failure paths.
set -euo pipefail
cd "$(dirname "$0")/../.."
PORT="${LUMEN_LOOPBACK_PORT:-19778}"
cargo build --release -p lumen-host
target/release/lumen-host m3-host --port "$PORT" --source synthetic --frames 300 &
HOST_PID=$!
trap 'kill "$HOST_PID" 2>/dev/null || true' EXIT
sleep 1
cd clients/apple
LUMEN_LOOPBACK_PORT="$PORT" swift test --filter LoopbackIntegrationTests
+19
View File
@@ -7,7 +7,26 @@
//! the media streams follow (see the M2 task list / plan).
pub mod apps;
#[cfg(target_os = "linux")]
mod audio;
/// Stub — the audio plane needs Linux (PipeWire capture + libopus); this keeps non-Linux
/// dev builds compiling (crate doc: "the crate compiles everywhere"). Reports failure the
/// same way the real stream thread does: by clearing `running`.
#[cfg(not(target_os = "linux"))]
mod audio {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
pub fn start(
running: Arc<AtomicBool>,
_gcm_key: [u8; 16],
_rikeyid: i32,
_audio_cap: Arc<Mutex<Option<Box<dyn crate::audio::AudioCapturer>>>>,
) {
tracing::error!("GameStream audio requires Linux (PipeWire + libopus)");
running.store(false, Ordering::SeqCst);
}
}
pub(crate) mod cert;
mod control;
mod crypto;
@@ -144,6 +144,7 @@ type PacketBatch = Vec<Vec<u8>>;
/// Send `pkts` with as few syscalls as possible (`sendmmsg`, up to 64 per call). The socket is
/// connected, so no per-message address. Returns an error on the first send failure.
#[cfg(target_os = "linux")]
fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec<u8>]) -> std::io::Result<()> {
use std::os::fd::AsRawFd;
const CHUNK: usize = 64;
@@ -179,6 +180,16 @@ fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec<u8>]) -> std::io::Result<()> {
Ok(())
}
/// Portable fallback (non-Linux dev builds — GameStream hosting never ships there): one
/// syscall per packet.
#[cfg(not(target_os = "linux"))]
fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec<u8>]) -> std::io::Result<()> {
for p in pkts {
sock.send(p)?;
}
Ok(())
}
/// Dedicated send thread: one [`PacketBatch`] per frame arrives on `rx`; its packets go out in
/// `sendmmsg` chunks, paced so the frame's data spreads over ~3/4 of the frame interval
/// (microburst shaping at chunk granularity — a real link drops line-rate bursts; the encode
+5 -3
View File
@@ -370,10 +370,12 @@ impl EiState {
InputKind::MouseScroll => match slot.interface::<ei::Scroll>() {
Some(s) => {
// GameStream sends WHEEL_DELTA(120)-scaled deltas in `x`; ei scroll_discrete
// uses the same 120-per-detent unit. Positive GameStream = up/left, which is
// negative on the ei axis (matches wl_pointer).
// uses the same 120-per-detent unit. Positive GameStream = up (vertical),
// which is negative on the ei axis, but = RIGHT (horizontal), which is
// already positive there (moonlight-qt/Sunshine pass horizontal through
// unnegated) — only the vertical axis flips.
if ev.code == SCROLL_HORIZONTAL {
s.scroll_discrete(-ev.x, 0);
s.scroll_discrete(ev.x, 0);
} else {
s.scroll_discrete(0, -ev.x);
}
+9 -2
View File
@@ -226,10 +226,17 @@ impl InputInjector for WlrootsInjector {
wl_pointer::Axis::VerticalScroll
};
// GameStream sends WHEEL_DELTA(120)-scaled units; a notch ≈ 15px. Positive
// GameStream = scroll up, which is negative on the Wayland axis.
// GameStream = up (vertical), negative on the Wayland axis; but = RIGHT
// (horizontal), already positive there (moonlight-qt/Sunshine pass
// horizontal through unnegated) — only the vertical axis flips.
let notches = event.x as f64 / 120.0;
let sign = if event.code == SCROLL_HORIZONTAL {
1.0
} else {
-1.0
};
self.pointer.axis_source(wl_pointer::AxisSource::Wheel);
self.pointer.axis(t, axis, -notches * 15.0);
self.pointer.axis(t, axis, sign * notches * 15.0);
self.pointer.frame();
}
InputKind::KeyDown | InputKind::KeyUp => {
+10
View File
@@ -448,6 +448,7 @@ fn input_thread(rx: std::sync::mpsc::Receiver<InputEvent>, conn: quinn::Connecti
/// The audio thread: desktop capture → Opus (48 kHz stereo, 5 ms, CBR — same tuning as the
/// GameStream path) → `AUDIO_MAGIC` datagrams. QUIC already encrypts; no extra layer.
/// The capturer comes from (and returns to) the persistent slot — see [`AudioCapSlot`].
#[cfg(target_os = "linux")]
fn audio_thread(conn: quinn::Connection, stop: Arc<AtomicBool>, audio_cap: AudioCapSlot) {
use crate::audio::{CHANNELS, SAMPLE_RATE};
const FRAME_MS: usize = 5;
@@ -519,6 +520,15 @@ fn audio_thread(conn: quinn::Connection, stop: Arc<AtomicBool>, audio_cap: Audio
}
}
/// Stub — lumen/1 audio needs Linux (PipeWire capture + libopus); non-Linux dev builds
/// run sessions without it, same as when the capturer fails to open.
#[cfg(not(target_os = "linux"))]
fn audio_thread(_conn: quinn::Connection, _stop: Arc<AtomicBool>, _audio_cap: AudioCapSlot) {
tracing::warn!(
"lumen/1 audio requires Linux (PipeWire + libopus) — session continues without it"
);
}
fn synthetic_stream(session: &mut Session, frames: u32, stop: &AtomicBool) -> Result<()> {
let interval = std::time::Duration::from_millis(1000 / 60);
for idx in 0..frames {
+16 -2
View File
@@ -13,11 +13,13 @@ cd "$(dirname "$0")/.."
TARGETS_MAC=(aarch64-apple-darwin x86_64-apple-darwin)
BUILD_IOS="${BUILD_IOS:-0}" # BUILD_IOS=1 adds an iOS slice (requires the ios target installed)
# Deployment targets must match Package.swift's platforms, or every consumer link emits
# "object file was built for newer macOS version" warnings.
for t in "${TARGETS_MAC[@]}"; do
cargo build --release -p lumen-core --features quic --target "$t"
MACOSX_DEPLOYMENT_TARGET=14.0 cargo build --release -p lumen-core --features quic --target "$t"
done
if [[ "$BUILD_IOS" == "1" ]]; then
cargo build --release -p lumen-core --features quic --target aarch64-apple-ios
IPHONEOS_DEPLOYMENT_TARGET=17.0 cargo build --release -p lumen-core --features quic --target aarch64-apple-ios
fi
STAGE="$(mktemp -d)"
@@ -49,6 +51,18 @@ if [[ "$BUILD_IOS" == "1" ]]; then
ARGS+=(-library target/aarch64-apple-ios/release/liblumen_core.a -headers "$STAGE/include")
fi
# Cargo does NOT fingerprint MACOSX_DEPLOYMENT_TARGET — units cached from a build without
# it keep their old minos forever. Refuse to ship anything newer than the package floor
# (objects BELOW it, e.g. rustup's precompiled std at 11.0, are fine and unavoidable).
for obj in "$STAGE"/macos/liblumen_core.a; do
bad=$(otool -l "$obj" 2>/dev/null | awk '/minos/ {print $2}' | sort -uV | awk -F. '$1 > 14' | head -1)
if [[ -n "$bad" ]]; then
echo "ERROR: $obj contains objects built for macOS $bad (> 14.0)." >&2
echo "Stale cache — rm -rf target/{aarch64,x86_64}-apple-darwin and rebuild." >&2
exit 1
fi
done
rm -rf clients/apple/LumenCore.xcframework
xcodebuild -create-xcframework "${ARGS[@]}" -output clients/apple/LumenCore.xcframework
echo "OK: clients/apple/LumenCore.xcframework"