From bf8a974e8b6ee0d6f81fe0b3de94b4306064d054 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Wed, 10 Jun 2026 14:38:01 +0200 Subject: [PATCH] =?UTF-8?q?feat:=20M4=20stage=201=20=E2=80=94=20the=20Swif?= =?UTF-8?q?tUI=20client=20is=20real:=20compiles,=20tested,=20first=20light?= =?UTF-8?q?=20on=20glass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The clients/apple scaffold is now a working macOS client, validated live against this repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC + AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60, mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector (~3.7k events injected in one session). LumenKit: - LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as integers while the enum constants import as a distinct Swift type — bridge by rawValue); close() is now safe from any thread (a close flag + pumpLock held across the blocking poll enforce the C contract "never close with a next_au in flight"; flag prevents lock-starvation by back-to-back polls). - StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate on the next in-band parameter sets when the layer fails, no stale enqueue after restart. - InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away), pressed-state tracking with release-all on focus loss and stop() (nothing sticks down host-side), global-singleton ownership guard (GC has one handler slot per process), X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs. - LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD, LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs. - Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels); test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN). Host/build fixes that fell out: - The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule. - Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the ei/wl axes, but GameStream's horizontal convention is positive = right (moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also un-inverts real Moonlight clients. - AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's policy, instead of leaking them into the preceding NAL. - build-xcframework.sh: deployment targets pinned to the package floor + an otool guard — cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship too-new minos objects. Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified): 14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots). Co-Authored-By: Claude Fable 5 --- .gitignore | 4 + CLAUDE.md | 21 ++- README.md | 2 +- clients/apple/Package.swift | 6 +- clients/apple/README.md | 162 ++++++++-------- .../Sources/LumenClient/ContentView.swift | 122 ++++++++++++ .../Sources/LumenClient/LumenClientApp.swift | 29 +++ .../Sources/LumenClient/SessionModel.swift | 115 ++++++++++++ clients/apple/Sources/LumenKit/AnnexB.swift | 10 +- .../apple/Sources/LumenKit/InputCapture.swift | 164 +++++++++++++--- .../Sources/LumenKit/LumenConnection.swift | 163 +++++++++++----- .../apple/Sources/LumenKit/StreamView.swift | 100 ++++++++-- .../Tests/LumenKitTests/AnnexBTests.swift | 79 ++++++++ .../LoopbackIntegrationTests.swift | 65 +++++++ .../LumenKitTests/RemoteFirstLightTests.swift | 80 ++++++++ .../VideoToolboxRoundTripTests.swift | 176 ++++++++++++++++++ clients/apple/test-loopback.sh | 17 ++ crates/lumen-host/src/gamestream/mod.rs | 19 ++ crates/lumen-host/src/gamestream/stream.rs | 11 ++ crates/lumen-host/src/inject/libei.rs | 8 +- crates/lumen-host/src/inject/wlr.rs | 11 +- crates/lumen-host/src/m3.rs | 10 + scripts/build-xcframework.sh | 18 +- 23 files changed, 1212 insertions(+), 180 deletions(-) create mode 100644 clients/apple/Sources/LumenClient/ContentView.swift create mode 100644 clients/apple/Sources/LumenClient/LumenClientApp.swift create mode 100644 clients/apple/Sources/LumenClient/SessionModel.swift create mode 100644 clients/apple/Tests/LumenKitTests/AnnexBTests.swift create mode 100644 clients/apple/Tests/LumenKitTests/LoopbackIntegrationTests.swift create mode 100644 clients/apple/Tests/LumenKitTests/RemoteFirstLightTests.swift create mode 100644 clients/apple/Tests/LumenKitTests/VideoToolboxRoundTripTests.swift create mode 100755 clients/apple/test-loopback.sh diff --git a/.gitignore b/.gitignore index 9822220..5a465ed 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,7 @@ /tools/*/target node_modules/ dist/ +# Swift package build artifacts + the locally-built xcframework (rebuild via scripts/build-xcframework.sh) +clients/apple/.build/ +clients/apple/LumenCore.xcframework/ +clients/apple/.swiftpm/ diff --git a/CLAUDE.md b/CLAUDE.md index 800e3b3..ab14728 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,11 +44,16 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc ## What's left -1. **M4 — client decode + present**: the SwiftUI client is scaffolded and handed off — - the lumen/1 connector is in the C ABI (`lumen_connect` & co., ABI-roundtrip-tested) with - an xcframework build script + LumenKit Swift package; **see - [`clients/apple/README.md`](clients/apple/README.md) for the Mac-side pickup**. Then - glass-to-glass numbers via `tools/latency-probe` (scaffold). The Linux reference client +1. **M4 — client decode + present: macOS stage 1 done, first light achieved + (2026-06-10).** LumenKit compiles and is tested on macOS (AnnexB → VideoToolbox → + `AVSampleBufferDisplayLayer`, GCMouse/GCKeyboard capture, `LumenClient` app shell); + validated live Mac ↔ this box at 720p60 — vkcube on glass, input injected via gamescope + EIS. Tests: `swift test` in `clients/apple` (unit + real-codec round trip), + `test-loopback.sh` (Swift client vs synthetic m3-host on loopback — runs on macOS), + `RemoteFirstLightTests` (full pipeline over the LAN). See + [`clients/apple/README.md`](clients/apple/README.md). Next: stage 2 presenter + (`VTDecompressionSession` + `CAMetalLayer` frame pacing), glass-to-glass numbers via + `tools/latency-probe` (scaffold), iOS variant. The Linux reference client (`lumen-client-rs`) gets VAAPI + wgpu on the same connector later. 2. **Sub-frame pipelining**: overlap encode and transmit within a frame. Requires a direct NVENC SDK wrapper (libavcodec only emits whole AUs) — the next big latency lever (~2–4 ms @@ -61,9 +66,9 @@ Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protoc HDR/10-bit/AV1 negotiation, surround audio, reconnect-at-new-mode robustness. 5. **Native clients** (`clients/{apple,android}` scaffolds) consuming `lumen_core.h`. 6. **This box, one-time setup still pending**: `sudo cp scripts/60-lumen.rules - /etc/udev/rules.d/` + user into `input` group (gamepads); `sudo ninja -C - /tmp/gamescope-src/build install` (the fixed gamescope ≥ 3.16.22 — until then use - `PATH=/tmp/gamescope-src/build/src:$PATH`); `apt install gnome-shell` (Mutter validation). + /etc/udev/rules.d/` + user into `input` group (gamepads); `apt install gnome-shell` + (Mutter validation). Done since last update: gamescope 3.16.22 is installed at + `/usr/local/bin` — the `PATH=/tmp/gamescope-src/...` override is no longer needed. ## Build / test / run diff --git a/README.md b/README.md index 362b3a9..df39fc3 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ negotiated extension. See [`docs/implementation-plan.md`](docs/implementation-pl | M2 — P1 host → stock Moonlight | 🟡 capture+encode landed in M0; pairing/RTSP/vdisplay pending | | M3 — measurement harness | 🟡 `tools/loss-harness` runs; `latency-probe` scaffolded | | M4 — P2 transport + Rust client | 🟡 GF(2¹⁶) core done; `lumen-client-rs` scaffolded | -| M5 — Apple client | ⬜ scaffolded (`clients/apple`) | +| M5 — Apple client | 🟡 macOS first light: HEVC on glass + input over `lumen/1` (`clients/apple`) | `lumen-core` is complete and verified: it builds and its full test suite (FEC recovery, loopback round-trip under loss, property tests, and a **C ABI harness**) passes on diff --git a/clients/apple/Package.swift b/clients/apple/Package.swift index b2beb43..3fa0316 100644 --- a/clients/apple/Package.swift +++ b/clients/apple/Package.swift @@ -8,7 +8,8 @@ let package = Package( name: "LumenKit", platforms: [.macOS(.v14), .iOS(.v17)], products: [ - .library(name: "LumenKit", targets: ["LumenKit"]) + .library(name: "LumenKit", targets: ["LumenKit"]), + .executable(name: "LumenClient", targets: ["LumenClient"]), ], targets: [ .binaryTarget(name: "LumenCore", path: "LumenCore.xcframework"), @@ -22,5 +23,8 @@ let package = Package( .linkedLibrary("resolv"), ] ), + // Development app shell (swift run LumenClient): connect form → stream + input. + .executableTarget(name: "LumenClient", dependencies: ["LumenKit"]), + .testTarget(name: "LumenKitTests", dependencies: ["LumenKit"]), ] ) diff --git a/clients/apple/README.md b/clients/apple/README.md index 77eb995..3325d78 100644 --- a/clients/apple/README.md +++ b/clients/apple/README.md @@ -1,102 +1,112 @@ -# lumen Apple client (SwiftUI) — handoff +# lumen Apple client (SwiftUI) The native macOS/iOS client for **`lumen/1`** (the post-GameStream protocol). All networking/protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM, -input datagrams — lives in the shared Rust core and is **done and tested**; this package -is the Swift shell: decode (VideoToolbox), present (SwiftUI), input capture. +input datagrams, Opus audio, cert pinning — lives in the shared Rust core (statically +linked as `LumenCore.xcframework`); this package is the Swift shell: decode +(VideoToolbox), present (SwiftUI), input capture. -## What exists (built + tested on the Linux host) +## Status — first light achieved (2026-06-10) -- **The connector**: `lumen_core::client::NativeClient` (Rust) exposed over the C ABI as - `lumen_connect` / `lumen_connection_next_au` / `lumen_connection_next_audio` / - `lumen_connection_next_rumble` / `lumen_connection_send_input` / `lumen_connection_mode` - / `lumen_connection_close` (see `include/lumen_core.h`, guarded by `LUMEN_FEATURE_QUIC`). - **End-to-end tested through the C ABI** against an in-process host - (`crates/lumen-host/src/m3.rs::tests::c_abi_connection_roundtrip` — three sequential - sessions: TOFU, pinned reconnect, wrong-pin rejection). -- **The host to test against**: `lumen-host m3-host --source virtual --seconds 60` on the - Linux box — a **persistent listener** (sessions back to back, reconnect at will during - development; `--max-sessions N` to bound it). It creates a native virtual output at - whatever mode the client requests and streams HEVC + desktop **Opus audio**; - `LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube` for moving content. -- **This package (SCAFFOLD — written on Linux, never compiled in Xcode)**: - - `LumenConnection.swift` — Swift wrapper over the C ABI (AUs/audio copied into `Data`; - certificate pinning + TOFU fingerprint via `pinSHA256:`/`hostFingerprint`). +Validated live, Mac ↔ Linux box over the LAN: gamescope virtual output → NVENC HEVC → +`lumen/1` (GF(2¹⁶) FEC + AES-GCM over UDP, QUIC control) → VideoToolbox → +`AVSampleBufferDisplayLayer` on glass at 1280×720@60, with mouse/keyboard flowing back as +QUIC datagrams into the host's gamescope EIS injector (thousands of events injected during +the session). Headless variant of the same proof: `RemoteFirstLightTests` decoded 60/60 +received AUs spanning 983 ms of host capture clock. + +The connector underneath (`lumen_core::client::NativeClient` over the C ABI) carries the +full session: video AUs, **Opus audio** (`nextAudio()`), **rumble** (`nextRumble()`), +input incl. gamepads, and **cert pinning + TOFU** (`pinSHA256:`/`hostFingerprint`) — see +`m3.rs::tests::c_abi_connection_roundtrip` (three sequential sessions: TOFU, pinned +reconnect, wrong-pin rejection). The host (`lumen-host m3-host`) is a persistent listener: +reconnect at will during development. + +What's here, all compiled and tested on macOS (Xcode 26.5 / Swift 6.3): + +- **`LumenKit`** (library) + - `LumenConnection.swift` — wrapper over the C ABI. AUs/audio are copied into `Data` + (the C pointer is only valid until the next call of the same kind). `close()` is safe + from any thread: per-plane locks enforce the C contract ("never close with a + `next_au`/`next_audio` in flight") instead of leaving it to callers. Pinning + TOFU + via `pinSHA256:`/`hostFingerprint`. - `AnnexB.swift` — in-band VPS/SPS/PPS → `CMVideoFormatDescription`; Annex-B → AVCC `CMSampleBuffer` with `DisplayImmediately` set. - `StreamView.swift` — SwiftUI `NSViewRepresentable` over `AVSampleBufferDisplayLayer` - (stage-1 presenter: the layer hardware-decodes compressed HEVC itself). - - `InputCapture.swift` — `GCMouse` raw deltas + `GCKeyboard` HID→VK mapping → - `lumen_connection_send_input`. + (stage-1 presenter: the layer hardware-decodes compressed HEVC itself). One pump + thread per view, token-cancelled so reconnects can't double-pump. + - `InputCapture.swift` — `GCMouse` raw deltas + `GCKeyboard` HID→VK mapping (the host's + `vk_to_evdev` consumes Windows VKs), with fractional-delta accumulation so sub-pixel + motion isn't truncated away. Buttons use GameStream ids (1=left … 5=X2); scroll is + WHEEL_DELTA(120)-scaled. +- **`LumenClient`** (development app shell): connect form → stream + input, fps/Mb-s HUD. + (Audio playback and gamepad capture are not wired into the app yet — the connector + surface is there; see notes 5–6.) +- **Tests** (`swift test`): byte-level Annex-B units; a real-codec round trip + (VTCompressionSession-encoded HEVC rebuilt as the host's wire shape → `AnnexB` → + VTDecompressionSession → pixels); loopback integration against a real local host + (`test-loopback.sh`); the remote first-light test above. -## Build steps (on the Mac) +## Build / run / test (on a Mac) ```sh rustup target add aarch64-apple-darwin x86_64-apple-darwin bash scripts/build-xcframework.sh # → clients/apple/LumenCore.xcframework -open clients/apple/Package.swift # or add the package to an Xcode app project +cd clients/apple +swift build && swift test # loopback/remote tests self-skip without a host +swift run LumenClient # the app; or open Package.swift in Xcode + +bash test-loopback.sh # full loopback proof: builds lumen-host + # (synthetic source — runs on macOS), streams + # byte-verified frames into the Swift client + +# against the real host (Linux box, see CLAUDE.md "Running on this box") — m3-host is a +# persistent listener, reconnect at will: +# LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 \ +# cargo run -rp lumen-host -- m3-host --source virtual --seconds 60 +LUMEN_REMOTE_HOST= swift test --filter RemoteFirstLightTests # headless +LUMEN_AUTOCONNECT= LUMEN_MODE=1280x720x60 swift run LumenClient # on glass ``` -Minimal app around it: +## Notes for whoever picks this up next -```swift -@main struct LumenApp: App { - var body: some Scene { WindowGroup { ContentView() } } -} -struct ContentView: View { - @State private var conn: LumenConnection? - var body: some View { - if let conn { - StreamView(connection: conn) - .onAppear { InputCapture(connection: conn).start() } - } else { - Button("Connect") { - conn = try? LumenConnection( - host: "192.168.1.70", width: 2560, height: 1440, refreshHz: 120) - } - } - } -} -``` - -## Handoff — what the next agent needs to know - -1. **Expect small compile fixes.** Every Swift file is flagged SCAFFOLD: API-checked from - documentation, never run through Xcode. Likely friction: the imported C enum spellings - (`LUMEN_STATUS_OK` etc. — cbindgen emits `QualifiedScreamingSnakeCase`), `LumenFrame()` - zero-init, `_pad` tuple shape on `LumenInputEvent`. -2. **ABI contract** (matches `lumen_core.h` docs): `next_au`'s pointer is valid only until - the *next* call on that handle (we copy to `Data` immediately); one pump thread per - connection, plus optionally one *separate* audio thread for `next_audio` (independent - borrow slots); `send_input` is enqueue-only and thread-safe alongside both; `close` - joins the Rust threads — never call it with a `next_au`/`next_audio` call in flight. -3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band, - and recovery keyframes re-send them — so "wait for the first format description, refresh - it on every IDR" (already what `StreamView` does) is sufficient; there is no out-of-band - extradata, ever. -4. **First-light test**: Linux box runs - `PATH=/tmp/gamescope-src/build/src:$PATH LUMEN_COMPOSITOR=gamescope \ - LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 cargo run -rp lumen-host -- m3-host - --source virtual --seconds 120`; Mac connects with the app. Success = the spinning - vkcube on glass. Then mouse/keys should appear inside the gamescope session (verify - with `LUMEN_GAMESCOPE_APP=xev` and the box-side log `/tmp/lumen-gamescope.log`). -5. **Stage 2 (after first light)**: replace `AVSampleBufferDisplayLayer` with explicit - `VTDecompressionSession` + `CAMetalLayer` for frame-pacing control (ProMotion/120 Hz), - and add glass-to-glass measurement (`tools/latency-probe` is the scaffold; the host - already stamps `pts_ns` with its capture wall clock — across machines you'll need a - clock-offset estimate from the QUIC RTT, or the probe's visual timestamp loop). -6. **Audio**: `nextAudio()` yields raw Opus packets (48 kHz stereo, one 5 ms frame each, +1. **cbindgen import quirk** (the predicted "small compile fixes", now fixed): the + C17-compatible header spells `LumenStatus`/`LumenInputKind` as integer typedefs while + the enum *constants* import into Swift as a distinct same-named type — bridge with + `.rawValue` (see the top of `LumenConnection.swift`). Don't fight the generated header. +2. **ABI contract**: one video pump thread per connection, plus optionally one *separate* + audio drain thread for `nextAudio()`/`nextRumble()` (the core keeps per-plane borrow + slots, so the planes never alias); `send()` is enqueue-only and safe alongside all of + them. The wrapper's per-plane locks make `close()` safe from anywhere (it waits out + in-flight polls, ≤ their timeouts). +3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band + and recovery keyframes re-send them — "refresh the format description on every IDR" + (what `StreamView` does) is sufficient; there is no out-of-band extradata, ever. +4. **Stage 2 (next)**: explicit `VTDecompressionSession` + `CAMetalLayer` for frame-pacing + control (ProMotion/120 Hz), glass-to-glass measurement via `tools/latency-probe` (the + host stamps `pts_ns` with its capture wall clock; across machines you need a clock + offset estimate from the QUIC RTT). +5. **Audio**: `nextAudio()` yields raw Opus packets (48 kHz stereo, one 5 ms frame each, sequence-numbered). Decode with libopus or `AVAudioConverter`/`kAudioFormatOpus` into an `AVAudioEngine` source node; conceal gaps (drop/dup) rather than blocking — the Rust side buffers 320 ms and drops the newest packet when the puller lags. Wall-clock - `ptsNs` shares the host clock with video AUs for A/V sync. -7. **Gamepads**: `GCController` → `.gamepadButton(...)`/`.gamepadAxis(...)` events (wire + `ptsNs` shares the host clock with video AUs for A/V sync. Wiring this into + `LumenClient` is the next app-side task. +6. **Gamepads**: `GCController` → `.gamepadButton(...)`/`.gamepadAxis(...)` events (wire contract documented on the constructors; the host accumulates them into a virtual Xbox 360 pad). Poll `nextRumble()` and feed `GCDeviceHaptics` for force feedback. -8. **Trust**: connect once with `pinSHA256: nil` (TOFU), persist `hostFingerprint` keyed + Client-side capture isn't in `InputCapture` yet. +7. **Trust**: connect once with `pinSHA256: nil` (TOFU), persist `hostFingerprint` keyed by host, pass it on every later connect — a mismatch throws `.connectFailed`. The host logs its fingerprint at startup ("clients pin this fingerprint") for out-of-band - verification UX; a PIN-style pairing ceremony is a later lumen-core task. + verification UX; a PIN-style pairing ceremony is a later lumen-core task. `LumenClient` + doesn't persist fingerprints yet — add it alongside the "add host" UX. +8. **Input capture caveats** (stage 1): GC handlers only fire while the app has focus — + on focus loss `InputCapture` auto-releases everything still held (keys + buttons) so + nothing sticks down host-side. Local shortcuts (⌘-anything) still also reach the host; + a capture toggle is a small follow-up. One live capture per process (the GC mouse/ + keyboard singletons have a single handler slot — ownership is tracked so a stale + capture's stop() can't clobber a newer one). 9. **iOS**: same package (`BUILD_IOS=1` for the xcframework slice); `StreamView` needs the `UIViewRepresentable` twin and touch→input mapping. diff --git a/clients/apple/Sources/LumenClient/ContentView.swift b/clients/apple/Sources/LumenClient/ContentView.swift new file mode 100644 index 0000000..58e2670 --- /dev/null +++ b/clients/apple/Sources/LumenClient/ContentView.swift @@ -0,0 +1,122 @@ +// Connect form ⇄ live stream. Stage-1 UX: pick host + mode, see frames, type/aim. + +import AppKit +import LumenKit +import SwiftUI + +struct ContentView: View { + @StateObject private var model = SessionModel() + @AppStorage("lumen.host") private var host = "192.168.1.70" + @AppStorage("lumen.port") private var port = 9777 + @AppStorage("lumen.width") private var width = 1920 + @AppStorage("lumen.height") private var height = 1080 + @AppStorage("lumen.hz") private var hz = 60 + + var body: some View { + Group { + if let conn = model.connection { + stream(conn) + } else { + connectForm + } + } + .onAppear { autoConnectIfAsked() } + .onDisappear { model.disconnect() } // window closed mid-session (Cmd+N spawns more) + } + + /// Development hook: LUMEN_AUTOCONNECT=host[:port] connects immediately at the saved + /// (or LUMEN_MODE=WxHxHz) mode — lets scripts drive first-light runs. (IPv4/hostname + /// only; an IPv6 literal would need bracket parsing.) + private func autoConnectIfAsked() { + guard let target = ProcessInfo.processInfo.environment["LUMEN_AUTOCONNECT"], + !target.isEmpty, model.connection == nil, !model.connecting + else { return } + let parts = target.split(separator: ":") + host = String(parts[0]) + if parts.count == 2, let p = Int(parts[1]) { port = p } + if let mode = ProcessInfo.processInfo.environment["LUMEN_MODE"] { + let dims = mode.split(separator: "x").compactMap { Int($0) } + if dims.count == 3 { + width = dims[0] + height = dims[1] + hz = dims[2] + } + } + model.connect( + host: host, port: UInt16(clamping: port), + width: UInt32(clamping: width), height: UInt32(clamping: height), + hz: UInt32(clamping: hz)) + } + + private func stream(_ conn: LumenConnection) -> some View { + StreamView( + connection: conn, + onFrame: { [meter = model.meter] au in meter.note(byteCount: au.data.count) }, + onSessionEnd: { [weak model] in + Task { @MainActor in model?.sessionEnded() } + } + ) + .overlay(alignment: .topTrailing) { hud(conn) } + .frame(minWidth: 640, minHeight: 360) + .background(Color.black) + } + + private func hud(_ conn: LumenConnection) -> some View { + VStack(alignment: .trailing, spacing: 4) { + Text("\(conn.width)×\(conn.height)@\(conn.refreshHz) \(model.fps) fps \(model.mbps, specifier: "%.1f") Mb/s") + .font(.system(.caption, design: .monospaced)) + Button("Disconnect") { model.disconnect() } + .font(.caption) + } + .padding(8) + .background(.black.opacity(0.5), in: RoundedRectangle(cornerRadius: 6)) + .foregroundStyle(.white) + .padding(10) + } + + private var connectForm: some View { + VStack(spacing: 14) { + Text("lumen").font(.largeTitle.weight(.semibold)) + Form { + TextField("Host", text: $host) + TextField("Port", value: $port, format: .number.grouping(.never)) + HStack { + TextField("Width", value: $width, format: .number.grouping(.never)) + Text("×") + TextField("Height", value: $height, format: .number.grouping(.never)) + Text("@") + TextField("Hz", value: $hz, format: .number.grouping(.never)) + } + Button("Use this display's mode") { fillFromMainScreen() } + .buttonStyle(.link) + } + .frame(width: 340) + + if let error = model.errorMessage { + Text(error) + .font(.caption) + .foregroundStyle(.red) + .frame(width: 340) + } + + Button(model.connecting ? "Connecting…" : "Connect") { + model.connect( + host: host, port: UInt16(clamping: port), + width: UInt32(clamping: width), height: UInt32(clamping: height), + hz: UInt32(clamping: hz)) + } + .keyboardShortcut(.defaultAction) + .disabled(model.connecting || host.isEmpty) + } + .padding(28) + .frame(minWidth: 420, minHeight: 320) + } + + private func fillFromMainScreen() { + guard let screen = NSScreen.main else { return } + let scale = screen.backingScaleFactor + width = Int(screen.frame.width * scale) + height = Int(screen.frame.height * scale) + hz = screen.maximumFramesPerSecond + } +} diff --git a/clients/apple/Sources/LumenClient/LumenClientApp.swift b/clients/apple/Sources/LumenClient/LumenClientApp.swift new file mode 100644 index 0000000..42f2481 --- /dev/null +++ b/clients/apple/Sources/LumenClient/LumenClientApp.swift @@ -0,0 +1,29 @@ +// LumenClient — development app shell around LumenKit (swift run LumenClient). +// Connect form → StreamView (AVSampleBufferDisplayLayer HEVC) + InputCapture. + +import AppKit +import SwiftUI + +@main +struct LumenClientApp: App { + @NSApplicationDelegateAdaptor(AppDelegate.self) private var appDelegate + + var body: some Scene { + WindowGroup("lumen") { + ContentView() + } + } +} + +final class AppDelegate: NSObject, NSApplicationDelegate { + func applicationDidFinishLaunching(_ notification: Notification) { + // `swift run` launches an unbundled binary; promote it to a regular app so the + // window fronts and receives keyboard/mouse focus (GameController needs focus). + NSApp.setActivationPolicy(.regular) + NSApp.activate(ignoringOtherApps: true) + } + + func applicationShouldTerminateAfterLastWindowClosed(_ sender: NSApplication) -> Bool { + true + } +} diff --git a/clients/apple/Sources/LumenClient/SessionModel.swift b/clients/apple/Sources/LumenClient/SessionModel.swift new file mode 100644 index 0000000..598fe62 --- /dev/null +++ b/clients/apple/Sources/LumenClient/SessionModel.swift @@ -0,0 +1,115 @@ +// Session state for the app shell: owns the connection, the input capture, and the +// pump-thread → main-actor stats relay. + +import Foundation +import LumenKit +import SwiftUI + +/// Pump-thread-side frame counters; a 1 Hz main-actor timer drains them into @Published +/// values. NSLock instead of an actor — the writer is the (non-async) pump thread. +final class FrameMeter: @unchecked Sendable { + private let lock = NSLock() + private var frames = 0 + private var bytes = 0 + private var totalFrames = 0 + + func note(byteCount: Int) { + lock.lock() + frames += 1 + bytes += byteCount + totalFrames += 1 + lock.unlock() + } + + /// Returns and resets the per-interval counters (the running total stays). + func drain() -> (frames: Int, bytes: Int, total: Int) { + lock.lock() + defer { + frames = 0 + bytes = 0 + lock.unlock() + } + return (frames, bytes, totalFrames) + } +} + +@MainActor +final class SessionModel: ObservableObject { + @Published var connection: LumenConnection? + @Published var connecting = false + @Published var errorMessage: String? + @Published var fps = 0 + @Published var mbps = 0.0 + @Published var totalFrames = 0 + + let meter = FrameMeter() + private var inputCapture: InputCapture? + private var statsTimer: Timer? + + func connect(host: String, port: UInt16, width: UInt32, height: UInt32, hz: UInt32) { + guard !connecting else { return } + connecting = true + errorMessage = nil + Task.detached(priority: .userInitiated) { + // LumenConnection.init blocks on the QUIC handshake — keep it off the main actor. + let result = Result { try LumenConnection( + host: host, port: port, width: width, height: height, refreshHz: hz) } + await MainActor.run { [weak self] in + guard let self else { return } + self.connecting = false + switch result { + case .success(let conn): + self.connection = conn + self.startInput(conn) + self.startStatsTimer() + case .failure: + self.errorMessage = "Connection failed — is the host running? " + + "(lumen-host m3-host on \(host):\(port))" + } + } + } + } + + func disconnect() { + inputCapture?.stop() + inputCapture = nil + statsTimer?.invalidate() + statsTimer = nil + if let conn = connection { + // close() waits out an in-flight poll (≤100 ms) and joins the Rust worker + // threads — keep that off the main actor. + Task.detached { conn.close() } + } + connection = nil + fps = 0 + mbps = 0 + } + + /// Called (via the main actor) when the pump hits end-of-session. + func sessionEnded() { + guard connection != nil else { return } + disconnect() + errorMessage = "Session ended by host." + } + + private func startInput(_ conn: LumenConnection) { + let capture = InputCapture(connection: conn) + capture.start() + inputCapture = capture + } + + private func startStatsTimer() { + let timer = Timer(timeInterval: 1.0, repeats: true) { [weak self] _ in + guard let self else { return } + Task { @MainActor in + let (frames, bytes, total) = self.meter.drain() + self.fps = frames + self.mbps = Double(bytes) * 8 / 1_000_000 + self.totalFrames = total + } + } + // .common so the HUD keeps updating during window drags / menu tracking. + RunLoop.main.add(timer, forMode: .common) + statsTimer = timer + } +} diff --git a/clients/apple/Sources/LumenKit/AnnexB.swift b/clients/apple/Sources/LumenKit/AnnexB.swift index 49608b3..00a5414 100644 --- a/clients/apple/Sources/LumenKit/AnnexB.swift +++ b/clients/apple/Sources/LumenKit/AnnexB.swift @@ -12,6 +12,9 @@ import Foundation public enum AnnexB { /// Split an Annex-B stream into NAL units (start codes 00 00 01 / 00 00 00 01 stripped). + /// All zeros immediately preceding a start code are dropped: they're either the + /// 4-byte-code prefix or `trailing_zero_8bits` padding, never NAL payload (emulation + /// prevention keeps 00 00 0x out of conforming NAL bytes) — same policy as ffmpeg. public static func nalUnits(in data: Data) -> [Data] { var nals: [Data] = [] let bytes = [UInt8](data) @@ -19,8 +22,11 @@ public enum AnnexB { var start = -1 while i + 2 < bytes.count { if bytes[i] == 0, bytes[i + 1] == 0, bytes[i + 2] == 1 { - let codeStart = (i > 0 && bytes[i - 1] == 0) ? i - 1 : i - if start >= 0 { + var codeStart = i + while codeStart > 0, bytes[codeStart - 1] == 0 { + codeStart -= 1 + } + if start >= 0, start < codeStart { nals.append(Data(bytes[start.. = [] + private var pressedButtons: Set = [] public init(connection: LumenConnection) { self.connection = connection } - /// Begin forwarding the current (and future) mouse/keyboard to the host. + /// Begin forwarding the current (and future) mouse/keyboard to the host. Steals the + /// global GC handler slots from any previous capture (one live capture per process). public func start() { + Self.activeCapture = self if let mouse = GCMouse.current { attach(mouse: mouse) } if let keyboard = GCKeyboard.coalesced { attach(keyboard: keyboard) } observers.append(NotificationCenter.default.addObserver( @@ -36,44 +60,130 @@ public final class InputCapture { ) { [weak self] n in if let k = n.object as? GCKeyboard { self?.attach(keyboard: k) } }) + // Focus loss: GC stops delivering, so release everything still held host-side. + observers.append(NotificationCenter.default.addObserver( + forName: NSApplication.didResignActiveNotification, object: nil, queue: .main + ) { [weak self] _ in + self?.releaseAll() + }) } public func stop() { + releaseAll() observers.forEach(NotificationCenter.default.removeObserver(_:)) observers.removeAll() + // Don't clobber the handlers if a newer capture has taken the global devices. + if Self.activeCapture === self || Self.activeCapture == nil { + for mouse in mice { + guard let input = mouse.mouseInput else { continue } + input.mouseMovedHandler = nil + input.leftButton.pressedChangedHandler = nil + input.rightButton?.pressedChangedHandler = nil + input.middleButton?.pressedChangedHandler = nil + input.auxiliaryButtons?.forEach { $0.pressedChangedHandler = nil } + input.scroll.valueChangedHandler = nil + } + for keyboard in keyboards { + keyboard.keyboardInput?.keyChangedHandler = nil + } + Self.activeCapture = nil + } + mice.removeAll() + keyboards.removeAll() + } + + deinit { stop() } + + /// Send release events for everything currently held, and drop the motion residuals. + private func releaseAll() { + for vk in pressedVKs { + connection.send(.key(vk, down: false)) + } + for button in pressedButtons { + connection.send(.mouseButton(button, down: false)) + } + pressedVKs.removeAll() + pressedButtons.removeAll() + residualX = 0 + residualY = 0 + residualScrollX = 0 + residualScrollY = 0 + } + + private func sendButton(_ button: UInt32, pressed: Bool) { + if pressed { + pressedButtons.insert(button) + } else { + pressedButtons.remove(button) + } + connection.send(.mouseButton(button, down: pressed)) } private func attach(mouse: GCMouse) { - guard let input = mouse.mouseInput else { return } - let conn = connection - input.mouseMovedHandler = { _, dx, dy in + guard let input = mouse.mouseInput, + !mice.contains(where: { $0 === mouse }) // re-delivered on wake — attach once + else { return } + mice.append(mouse) + input.mouseMovedHandler = { [weak self] _, dx, dy in + guard let self else { return } // GC gives +y up; the host expects screen-space (+y down). - conn.send(.mouseMove(dx: Int32(dx), dy: Int32(-dy))) + let fx = dx + self.residualX + let fy = -dy + self.residualY + let ix = fx.rounded(.towardZero) + let iy = fy.rounded(.towardZero) + self.residualX = fx - ix + self.residualY = fy - iy + if ix != 0 || iy != 0 { + self.connection.send(.mouseMove(dx: Int32(ix), dy: Int32(iy))) + } } - input.leftButton.pressedChangedHandler = { _, _, pressed in - conn.send(.mouseButton(1, down: pressed)) + input.leftButton.pressedChangedHandler = { [weak self] _, _, pressed in + self?.sendButton(1, pressed: pressed) } - input.rightButton?.pressedChangedHandler = { _, _, pressed in - conn.send(.mouseButton(3, down: pressed)) + input.rightButton?.pressedChangedHandler = { [weak self] _, _, pressed in + self?.sendButton(3, pressed: pressed) } - input.middleButton?.pressedChangedHandler = { _, _, pressed in - conn.send(.mouseButton(2, down: pressed)) + input.middleButton?.pressedChangedHandler = { [weak self] _, _, pressed in + self?.sendButton(2, pressed: pressed) } - input.scroll.valueChangedHandler = { _, _, dy in - if dy != 0 { conn.send(.scroll(Int32(dy * 120))) } + // First two side buttons → GameStream X1/X2. + if let aux = input.auxiliaryButtons { + for (i, button) in aux.prefix(2).enumerated() { + button.pressedChangedHandler = { [weak self] _, _, pressed in + self?.sendButton(UInt32(4 + i), pressed: pressed) + } + } + } + input.scroll.valueChangedHandler = { [weak self] _, x, y in + guard let self else { return } + // WHEEL_DELTA(120) per notch; positive = up / right (Moonlight's convention). + let fy = y * 120 + self.residualScrollY + let fx = x * 120 + self.residualScrollX + let iy = fy.rounded(.towardZero) + let ix = fx.rounded(.towardZero) + self.residualScrollY = fy - iy + self.residualScrollX = fx - ix + if iy != 0 { self.connection.send(.scroll(Int32(iy))) } + if ix != 0 { self.connection.send(.scroll(Int32(ix), horizontal: true)) } } } private func attach(keyboard: GCKeyboard) { - let conn = connection - keyboard.keyboardInput?.keyChangedHandler = { _, _, keyCode, pressed in - if let vk = Self.hidToVK[keyCode.rawValue] { - conn.send(.key(vk, down: pressed)) + guard !keyboards.contains(where: { $0 === keyboard }) else { return } + keyboards.append(keyboard) + keyboard.keyboardInput?.keyChangedHandler = { [weak self] _, _, keyCode, pressed in + guard let self, let vk = Self.hidToVK[keyCode.rawValue] else { return } + if pressed { + self.pressedVKs.insert(vk) + } else { + self.pressedVKs.remove(vk) } + self.connection.send(.key(vk, down: pressed)) } } - /// HID usage (GCKeyCode raw) → Windows VK (the host maps VK → evdev). + /// HID usage (GCKeyCode raw) → Windows VK (the host maps VK → evdev; every VK emitted + /// here exists in lumen-host/src/inject.rs::vk_to_evdev — extend the two together). static let hidToVK: [Int: UInt32] = { var m: [Int: UInt32] = [:] // a–z: HID 0x04..0x1D → VK 'A'..'Z'. @@ -90,11 +200,23 @@ public final class InputCapture { m[0x2F] = 0xDB; m[0x30] = 0xDD; m[0x31] = 0xDC // [ ] backslash m[0x33] = 0xBA; m[0x34] = 0xDE; m[0x35] = 0xC0 // ; ' ` m[0x36] = 0xBC; m[0x37] = 0xBE; m[0x38] = 0xBF // , . / + m[0x39] = 0x14 // caps lock // F1..F12: HID 0x3A..0x45 → VK 0x70..0x7B. for i in 0..<12 { m[0x3A + i] = UInt32(0x70 + i) } + m[0x46] = 0x2C; m[0x47] = 0x91; m[0x48] = 0x13 // printscreen scrolllock pause m[0x4F] = 0x27; m[0x50] = 0x25; m[0x51] = 0x28; m[0x52] = 0x26 // arrows R L D U m[0x49] = 0x2D; m[0x4A] = 0x24; m[0x4B] = 0x21 // insert home pageup m[0x4C] = 0x2E; m[0x4D] = 0x23; m[0x4E] = 0x22 // delete end pagedown + // Keypad: NumLock, / * - +, Enter, 1..9, 0, decimal. KP Enter goes as + // VK_SEPARATOR (0x6C) — this host maps it to KEY_KPENTER (Windows itself would + // send VK_RETURN+extended, which vk_to_evdev can't distinguish). + m[0x53] = 0x90 + m[0x54] = 0x6F; m[0x55] = 0x6A; m[0x56] = 0x6D; m[0x57] = 0x6B + m[0x58] = 0x6C + for i in 0..<9 { m[0x59 + i] = UInt32(0x61 + i) } + m[0x62] = 0x60; m[0x63] = 0x6E + m[0x64] = 0xE2 // ISO 102nd key (<> next to left shift on ISO layouts) + m[0x65] = 0x5D // menu/application m[0xE0] = 0xA2; m[0xE1] = 0xA0; m[0xE2] = 0xA4; m[0xE3] = 0x5B // Lctrl Lshift Lalt Lcmd m[0xE4] = 0xA3; m[0xE5] = 0xA1; m[0xE6] = 0xA5; m[0xE7] = 0x5C // Rctrl Rshift Ralt Rcmd return m diff --git a/clients/apple/Sources/LumenKit/LumenConnection.swift b/clients/apple/Sources/LumenKit/LumenConnection.swift index 0f5cea4..7dc6086 100644 --- a/clients/apple/Sources/LumenKit/LumenConnection.swift +++ b/clients/apple/Sources/LumenKit/LumenConnection.swift @@ -1,8 +1,9 @@ // Swift wrapper around the lumen-core C ABI's lumen/1 connection API. // -// Threading contract (mirrors the C header): one LumenConnection is used from a single -// pump thread for nextAU(); nextAudio() may run on its own (single) audio thread; -// sendInput() is enqueue-only and safe alongside both. The pointers inside an AU/audio +// Threading contract (mirrors the C header): one LumenConnection is pumped from a single +// video thread via nextAU(); nextAudio()/nextRumble() may each run on their own (single) +// drain thread — the core keeps per-plane borrow slots, so the planes never alias; +// send() is enqueue-only and safe alongside all of them. The pointers inside an AU/audio // packet are only valid until the next call of the same kind, so we copy into Data here — // the copies are small and keep the Swift side memory-safe. // @@ -10,12 +11,22 @@ // `hostFingerprint` reports what a trust-on-first-use connect observed — persist it, e.g. // in UserDefaults keyed by host, and pin it from then on). // -// SCAFFOLD: written on the Linux host, not yet compiled against Xcode — expect to fix -// trivial issues on first build (see README.md "Handoff"). +// close() is safe from any thread: it flags the pullers to exit at their next poll +// boundary, then takes the per-plane locks (each held across its blocking C poll), so the +// handle is never freed under an in-flight call — the C contract ("never close with a +// next_au/next_audio call in flight") is enforced here rather than left to callers. After +// close, the pull methods throw `.closed` and the threads unwind on their own. import Foundation import LumenCore +// cbindgen's C17-compatible header spells the typedefs as plain integers +// (`typedef int32_t LumenStatus`, `typedef uint8_t LumenInputKind`) while the enum +// constants import as a distinct same-named Swift type — bridge by raw value once here. +private let statusOK: Int32 = LUMEN_STATUS_OK.rawValue +private let statusNoFrame: Int32 = LUMEN_STATUS_NO_FRAME.rawValue +private let statusClosed: Int32 = LUMEN_STATUS_CLOSED.rawValue + /// One reassembled, FEC-recovered, decrypted access unit (Annex-B HEVC from the host). public struct AccessUnit: Sendable { public let data: Data @@ -39,10 +50,22 @@ public enum LumenClientError: Error { /// unpinned when the caller asked for verification would be a silent trust downgrade. case invalidPin case closed + case status(Int32) } public final class LumenConnection { private var handle: OpaquePointer? + /// Set by close() before it contends for the plane locks: the pullers see it at their + /// next poll boundary and exit, so close() can't be starved by back-to-back polls + /// (NSLock is not fair). + private var closeRequested = false + /// Serializes send()/close() against each other and guards `handle`/`closeRequested`. + private let abiLock = NSLock() + /// Held across the blocking next_au call; close() takes it (same plane-lock → abiLock + /// order as the pullers) so it can never free the handle under an in-flight poll. + private let pumpLock = NSLock() + /// Same role for the audio/rumble drain thread (its own plane in the core). + private let audioLock = NSLock() /// Negotiated session mode (host-confirmed). public private(set) var width: UInt32 = 0 @@ -86,87 +109,141 @@ public final class LumenConnection { self.refreshHz = hz } - /// Pull the next access unit; nil on timeout, throws once the session is closed. + /// Pull the next access unit; nil on timeout, throws `.closed` once the session ended. + /// Call from a single pump thread. public func nextAU(timeoutMs: UInt32 = 100) throws -> AccessUnit? { + pumpLock.lock() + defer { pumpLock.unlock() } + guard let h = liveHandle() else { throw LumenClientError.closed } + var frame = LumenFrame() - switch lumen_connection_next_au(handle, &frame, timeoutMs) { - case LUMEN_STATUS_OK: - let data = Data(bytes: frame.data, count: frame.len) // copy: ptr valid only until next call + let rc = lumen_connection_next_au(h, &frame, timeoutMs) + switch rc { + case statusOK: + guard let base = frame.data, frame.len > 0 else { return nil } + let data = Data(bytes: base, count: Int(frame.len)) // copy: ptr valid only until next call return AccessUnit( data: data, ptsNs: frame.pts_ns, frameIndex: frame.frame_index, flags: frame.flags) - case LUMEN_STATUS_NO_FRAME: + case statusNoFrame: return nil - case LUMEN_STATUS_CLOSED: + case statusClosed: throw LumenClientError.closed default: - throw LumenClientError.closed + throw LumenClientError.status(rc) } } - /// Pull the next Opus audio packet; nil on timeout, throws once the session is closed. - /// Drain from a dedicated audio thread — packets arrive every 5 ms (320 ms buffered). + /// Pull the next Opus audio packet; nil on timeout, throws `.closed` once the session + /// ended. Drain from a dedicated audio thread — packets arrive every 5 ms (the core + /// buffers 320 ms and drops the newest when the puller lags). public func nextAudio(timeoutMs: UInt32 = 100) throws -> AudioPacket? { + audioLock.lock() + defer { audioLock.unlock() } + guard let h = liveHandle() else { throw LumenClientError.closed } + var pkt = LumenAudioPacket() - switch lumen_connection_next_audio(handle, &pkt, timeoutMs) { - case LUMEN_STATUS_OK: - let data = Data(bytes: pkt.data, count: pkt.len) // copy: ptr valid only until next call + let rc = lumen_connection_next_audio(h, &pkt, timeoutMs) + switch rc { + case statusOK: + guard let base = pkt.data, pkt.len > 0 else { return nil } + let data = Data(bytes: base, count: Int(pkt.len)) // copy: ptr valid only until next call return AudioPacket(data: data, ptsNs: pkt.pts_ns, seq: pkt.seq) - case LUMEN_STATUS_NO_FRAME: + case statusNoFrame: return nil - default: + case statusClosed: throw LumenClientError.closed + default: + throw LumenClientError.status(rc) } } /// Pull the next force-feedback update for the GCController haptics engine: /// `(pad, lowFrequency, highFrequency)` with 0...0xFFFF amplitudes, (0, 0) = stop. - public func nextRumble(timeoutMs: UInt32 = 100) throws -> (pad: UInt16, low: UInt16, high: UInt16)? { + /// Shares the audio drain thread's plane (call from that thread). + public func nextRumble(timeoutMs: UInt32 = 0) throws -> (pad: UInt16, low: UInt16, high: UInt16)? { + audioLock.lock() + defer { audioLock.unlock() } + guard let h = liveHandle() else { throw LumenClientError.closed } + var pad: UInt16 = 0, low: UInt16 = 0, high: UInt16 = 0 - switch lumen_connection_next_rumble(handle, &pad, &low, &high, timeoutMs) { - case LUMEN_STATUS_OK: + let rc = lumen_connection_next_rumble(h, &pad, &low, &high, timeoutMs) + switch rc { + case statusOK: return (pad, low, high) - case LUMEN_STATUS_NO_FRAME: + case statusNoFrame: return nil - default: + case statusClosed: throw LumenClientError.closed + default: + throw LumenClientError.status(rc) } } - /// Send one input event (delivered to the host as a QUIC datagram). + /// Send one input event (delivered to the host as a QUIC datagram). Thread-safe; + /// silently dropped after close. public func send(_ event: LumenInputEvent) { var ev = event - _ = lumen_connection_send_input(handle, &ev) + abiLock.lock() + defer { abiLock.unlock() } + guard let h = handle, !closeRequested else { return } + _ = lumen_connection_send_input(h, &ev) } + /// Close the connection and free the handle. Safe from any thread, idempotent; waits + /// for in-flight pulls (≤ their timeouts) before tearing down. public func close() { - if let h = handle { - lumen_connection_close(h) - handle = nil + abiLock.lock() + closeRequested = true + abiLock.unlock() + pumpLock.lock() // pullers exit at their next poll boundary, releasing these + audioLock.lock() + abiLock.lock() + let h = handle + handle = nil + abiLock.unlock() + audioLock.unlock() + pumpLock.unlock() + if let h { + lumen_connection_close(h) // joins the connection's internal Rust threads } } deinit { close() } + + /// Snapshot the handle unless close is pending (callers hold their plane lock). + private func liveHandle() -> OpaquePointer? { + abiLock.lock() + defer { abiLock.unlock() } + return closeRequested ? nil : handle + } } // Convenience constructors for the wire input events (field semantics match // lumen_core::input::InputEvent; see lumen_core.h). public extension LumenInputEvent { + private static func make( + _ kind: UInt32, code: UInt32, x: Int32, y: Int32, flags: UInt32 = 0 + ) -> LumenInputEvent { + LumenInputEvent(kind: UInt8(kind), _pad: (0, 0, 0), code: code, x: x, y: y, flags: flags) + } static func mouseMove(dx: Int32, dy: Int32) -> LumenInputEvent { - LumenInputEvent(kind: LUMEN_INPUT_KIND_MOUSE_MOVE, _pad: (0, 0, 0), code: 0, x: dx, y: dy, flags: 0) + make(LUMEN_INPUT_KIND_MOUSE_MOVE.rawValue, code: 0, x: dx, y: dy) } + /// GameStream button ids: 1=left 2=middle 3=right 4=X1 5=X2 (host maps to evdev BTN_*). static func mouseButton(_ button: UInt32, down: Bool) -> LumenInputEvent { - LumenInputEvent( - kind: down ? LUMEN_INPUT_KIND_MOUSE_BUTTON_DOWN : LUMEN_INPUT_KIND_MOUSE_BUTTON_UP, - _pad: (0, 0, 0), code: button, x: 0, y: 0, flags: 0) + make( + (down ? LUMEN_INPUT_KIND_MOUSE_BUTTON_DOWN : LUMEN_INPUT_KIND_MOUSE_BUTTON_UP).rawValue, + code: button, x: 0, y: 0) } + /// `vk` is a Windows virtual-key code (the host's vk_to_evdev table consumes these). static func key(_ vk: UInt32, down: Bool) -> LumenInputEvent { - LumenInputEvent( - kind: down ? LUMEN_INPUT_KIND_KEY_DOWN : LUMEN_INPUT_KIND_KEY_UP, - _pad: (0, 0, 0), code: vk, x: 0, y: 0, flags: 0) + make((down ? LUMEN_INPUT_KIND_KEY_DOWN : LUMEN_INPUT_KIND_KEY_UP).rawValue, code: vk, x: 0, y: 0) } - static func scroll(_ delta: Int32) -> LumenInputEvent { - LumenInputEvent(kind: LUMEN_INPUT_KIND_MOUSE_SCROLL, _pad: (0, 0, 0), code: 0, x: delta, y: 0, flags: 0) + /// WHEEL_DELTA(120)-scaled; positive = up (vertical) / right (horizontal) — the + /// convention Moonlight/SDL use; the host maps onto the ei/wl axes. + static func scroll(_ delta: Int32, horizontal: Bool = false) -> LumenInputEvent { + make(LUMEN_INPUT_KIND_MOUSE_SCROLL.rawValue, code: horizontal ? 1 : 0, x: delta, y: 0) } // Gamepad (wire contract in lumen_core::input::gamepad): one transition per event, @@ -175,16 +252,14 @@ public extension LumenInputEvent { /// `button` is a GameStream buttonFlags bit (A=0x1000 B=0x2000 X=0x4000 Y=0x8000, /// dpad=0x1/2/4/8, start=0x10 back=0x20 LS=0x40 RS=0x80 LB=0x100 RB=0x200 guide=0x400). static func gamepadButton(_ button: UInt32, down: Bool, pad: UInt32 = 0) -> LumenInputEvent { - LumenInputEvent( - kind: LUMEN_INPUT_KIND_GAMEPAD_BUTTON, - _pad: (0, 0, 0), code: button, x: down ? 1 : 0, y: 0, flags: pad) + make( + LUMEN_INPUT_KIND_GAMEPAD_BUTTON.rawValue, + code: button, x: down ? 1 : 0, y: 0, flags: pad) } /// Axis ids: 0=LSX 1=LSY 2=RSX 3=RSY (−32768...32767, XInput convention: +y = UP — /// `GCControllerDirectionPad.yAxis` already matches, no flip), 4=LT 5=RT (0...255). static func gamepadAxis(_ axis: UInt32, value: Int32, pad: UInt32 = 0) -> LumenInputEvent { - LumenInputEvent( - kind: LUMEN_INPUT_KIND_GAMEPAD_AXIS, - _pad: (0, 0, 0), code: axis, x: value, y: 0, flags: pad) + make(LUMEN_INPUT_KIND_GAMEPAD_AXIS.rawValue, code: axis, x: value, y: 0, flags: pad) } } diff --git a/clients/apple/Sources/LumenKit/StreamView.swift b/clients/apple/Sources/LumenKit/StreamView.swift index db90193..40bb87f 100644 --- a/clients/apple/Sources/LumenKit/StreamView.swift +++ b/clients/apple/Sources/LumenKit/StreamView.swift @@ -5,8 +5,8 @@ // zero-copy on Apple silicon. Stage 2 (explicit VTDecompressionSession + CAMetalLayer) // replaces this when we start tuning frame pacing / measuring glass-to-glass. // -// SCAFFOLD: written on the Linux host, not yet compiled against Xcode. macOS-first -// (NSViewRepresentable); the iOS variant is the same layer under UIViewRepresentable. +// macOS-first (NSViewRepresentable); the iOS variant is the same layer under +// UIViewRepresentable. #if os(macOS) import AVFoundation @@ -14,70 +14,130 @@ import SwiftUI public struct StreamView: NSViewRepresentable { private let connection: LumenConnection + private let onFrame: (@Sendable (AccessUnit) -> Void)? + private let onSessionEnd: (@Sendable () -> Void)? - public init(connection: LumenConnection) { + /// `onFrame`/`onSessionEnd` fire on the pump thread — hop to the main actor for UI. + public init( + connection: LumenConnection, + onFrame: (@Sendable (AccessUnit) -> Void)? = nil, + onSessionEnd: (@Sendable () -> Void)? = nil + ) { self.connection = connection + self.onFrame = onFrame + self.onSessionEnd = onSessionEnd } public func makeNSView(context: Context) -> StreamLayerView { let view = StreamLayerView() - view.start(connection: connection) + view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd) return view } - public func updateNSView(_ view: StreamLayerView, context: Context) {} + public func updateNSView(_ view: StreamLayerView, context: Context) { + // SwiftUI reuses the NSView across state changes — repoint the pump only when the + // connection identity actually changed. + if view.connection !== connection { + view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd) + } + } + + public static func dismantleNSView(_ view: StreamLayerView, coordinator: ()) { + view.stop() + } } public final class StreamLayerView: NSView { + /// Cancellation handle owned by exactly one pump thread — a restart hands the old pump + /// its own token, so it can never be revived by a newer start(). + private final class PumpToken: @unchecked Sendable { + private let lock = NSLock() + private var live = true + var isLive: Bool { + lock.lock() + defer { lock.unlock() } + return live + } + func cancel() { + lock.lock() + live = false + lock.unlock() + } + } + private let displayLayer = AVSampleBufferDisplayLayer() - private var pump: Thread? - private var running = false + private var token: PumpToken? + public private(set) var connection: LumenConnection? public override init(frame: NSRect) { super.init(frame: frame) - wantsLayer = true displayLayer.videoGravity = .resizeAspect - layer = displayLayer + layer = displayLayer // layer-hosting: assign before wantsLayer + wantsLayer = true } public required init?(coder: NSCoder) { fatalError("not used") } /// Pump thread: pull AUs from the connection, wrap, enqueue. The first IDR yields the /// format description; non-IDR AUs before it are dropped (the host opens with an IDR). - public func start(connection: LumenConnection) { - guard !running else { return } - running = true + public func start( + connection: LumenConnection, + onFrame: (@Sendable (AccessUnit) -> Void)? = nil, + onSessionEnd: (@Sendable () -> Void)? = nil + ) { + stop() + let token = PumpToken() + self.token = token + self.connection = connection let layer = displayLayer - let thread = Thread { [weak self] in + layer.flush() // drop any frames a previous connection left queued + + let thread = Thread { var format: CMVideoFormatDescription? - while self?.running == true { + while token.isLive { do { guard let au = try connection.nextAU(timeoutMs: 100) else { continue } + onFrame?(au) if let f = AnnexB.formatDescription(fromIDR: au.data) { format = f // refreshed on every IDR (mode changes included) } - guard let f = format, - let sample = AnnexB.sampleBuffer(au: au, format: f) - else { continue } if layer.status == .failed { + // Decode wedged: flush and re-gate on the next in-band parameter + // sets — resuming with a delta frame can't recover. (A + // request-IDR channel on lumen/1 is a host-side TODO; with the + // host's infinite GOP this may otherwise stay black until the + // next recovery keyframe.) layer.flush() + format = AnnexB.formatDescription(fromIDR: au.data) } + guard let f = format, + let sample = AnnexB.sampleBuffer(au: au, format: f), + token.isLive // don't enqueue a stale frame after a restart + else { continue } layer.enqueue(sample) } catch { + if token.isLive { + onSessionEnd?() + } break // session closed } } } thread.name = "lumen-pump" thread.qualityOfService = .userInteractive - pump = thread thread.start() } + /// Stop pumping (≤ one poll timeout). Does not close the connection — that stays with + /// whoever owns it (LumenConnection.close() is safe alongside a draining pump). public func stop() { - running = false + token?.cancel() + token = nil + connection = nil } - deinit { running = false } + deinit { + token?.cancel() + } } #endif diff --git a/clients/apple/Tests/LumenKitTests/AnnexBTests.swift b/clients/apple/Tests/LumenKitTests/AnnexBTests.swift new file mode 100644 index 0000000..e4a32ed --- /dev/null +++ b/clients/apple/Tests/LumenKitTests/AnnexBTests.swift @@ -0,0 +1,79 @@ +// Unit tests for the Annex-B ⇄ AVCC plumbing (pure byte-level; no codec involved — +// VideoToolboxRoundTripTests covers the real-bitstream path). + +import XCTest +@testable import LumenKit + +final class AnnexBTests: XCTestCase { + /// NAL with the given HEVC type in bits 1..6 of the first header byte. + private func nal(type: UInt8, payload: [UInt8]) -> Data { + Data([type << 1, 0x01] + payload) + } + + private let start4: [UInt8] = [0, 0, 0, 1] + private let start3: [UInt8] = [0, 0, 1] + + func testSplitMixedStartCodes() { + let a = nal(type: 32, payload: [0xAA]) + let b = nal(type: 33, payload: [0xBB, 0xBC]) + let c = nal(type: 19, payload: [0xCC, 0xCD, 0xCE]) + var au = Data(start4) + au.append(a) + au.append(contentsOf: start3) + au.append(b) + au.append(contentsOf: start4) + au.append(c) + + let nals = AnnexB.nalUnits(in: au) + XCTAssertEqual(nals, [a, b, c]) + XCTAssertEqual(nals.map(AnnexB.hevcNalType), [32, 33, 19]) + } + + func testSplitSingleNalNoTrailingCode() { + let v = nal(type: 34, payload: [1, 2, 3]) + let au = Data(start3) + v + XCTAssertEqual(AnnexB.nalUnits(in: au), [v]) + } + + func testSplitEmptyAndGarbage() { + XCTAssertEqual(AnnexB.nalUnits(in: Data()), []) + // No start code at all → no NALs. + XCTAssertEqual(AnnexB.nalUnits(in: Data([9, 8, 7, 6])), []) + } + + func testSplitDropsTrailingZeroPadding() { + // trailing_zero_8bits between NALs (and >2 zeros forming a long separator) must + // not leak into the preceding NAL. + let a = nal(type: 33, payload: [0xAA]) + let b = nal(type: 19, payload: [0xBB]) + var au = Data(start4) + au.append(a) + au.append(contentsOf: [0, 0, 0, 0, 0, 1]) // padding + start code + au.append(b) + XCTAssertEqual(AnnexB.nalUnits(in: au), [a, b]) + } + + func testAvccDropsParameterSetsAndPrefixesLengths() { + let vps = nal(type: 32, payload: [0xAA]) + let sps = nal(type: 33, payload: [0xBB]) + let pps = nal(type: 34, payload: [0xCC]) + let idr = nal(type: 19, payload: [0xDD, 0xDE, 0xDF, 0xE0]) + var au = Data() + for n in [vps, sps, pps, idr] { + au.append(contentsOf: start4) + au.append(n) + } + + let avcc = AnnexB.avcc(from: au) + // Only the IDR survives: 4-byte BE length, then the NAL bytes. + var expected = Data([0, 0, 0, UInt8(idr.count)]) + expected.append(idr) + XCTAssertEqual(avcc, expected) + } + + func testFormatDescriptionNilWithoutParameterSets() { + let idr = nal(type: 19, payload: [0xDD]) + let au = Data(start4) + idr + XCTAssertNil(AnnexB.formatDescription(fromIDR: au)) + } +} diff --git a/clients/apple/Tests/LumenKitTests/LoopbackIntegrationTests.swift b/clients/apple/Tests/LumenKitTests/LoopbackIntegrationTests.swift new file mode 100644 index 0000000..d7a0b13 --- /dev/null +++ b/clients/apple/Tests/LumenKitTests/LoopbackIntegrationTests.swift @@ -0,0 +1,65 @@ +// Integration: the Swift wrapper against a real lumen/1 host over QUIC + UDP on loopback — +// the Swift twin of lumen-host's m3.rs::c_abi_connection_roundtrip, this time through the +// statically linked xcframework. Driven by clients/apple/test-loopback.sh, which builds and +// starts `lumen-host m3-host --source synthetic` and sets LUMEN_LOOPBACK_PORT. + +import XCTest +@testable import LumenKit + +final class LoopbackIntegrationTests: XCTestCase { + func testSyntheticStreamRoundTrip() throws { + guard let portStr = ProcessInfo.processInfo.environment["LUMEN_LOOPBACK_PORT"], + let port = UInt16(portStr) + else { + throw XCTSkip("needs a running m3-host — use clients/apple/test-loopback.sh") + } + + let conn = try LumenConnection( + host: "127.0.0.1", port: port, width: 1280, height: 720, refreshHz: 60) + XCTAssertEqual(conn.width, 1280) + XCTAssertEqual(conn.height, 720) + XCTAssertEqual(conn.refreshHz, 60) + + // Pull 25 synthetic frames and byte-verify the documented pattern: + // u32 LE frame index, then data[i] = (idx as u8) &+ (i as u8). + var got = 0 + var lastIndex: UInt32 = 0 + let deadline = Date().addingTimeInterval(30) + while got < 25 { + XCTAssertLessThan(Date(), deadline, "timed out after \(got) frames") + guard let au = try conn.nextAU(timeoutMs: 2000) else { continue } + let idx = au.data.prefix(4).reversed().reduce(UInt32(0)) { ($0 << 8) | UInt32($1) } + for (i, byte) in au.data.enumerated().dropFirst(4) { + let expected = UInt8(truncatingIfNeeded: idx) &+ UInt8(truncatingIfNeeded: i) + if byte != expected { + XCTFail("frame \(idx) corrupt at offset \(i)") + break + } + } + XCTAssertGreaterThan(au.ptsNs, 0) + lastIndex = idx + got += 1 + } + XCTAssertGreaterThanOrEqual(lastIndex, 24) + + // Input goes the other way (enqueue-only; the host logs the count on close). + conn.send(.mouseMove(dx: 1, dy: 2)) + conn.send(.key(0x41, down: true)) + conn.send(.key(0x41, down: false)) + + conn.close() + XCTAssertThrowsError(try conn.nextAU(timeoutMs: 10)) { error in + guard case LumenClientError.closed = error else { + return XCTFail("expected .closed, got \(error)") + } + } + } + + func testConnectFailureThrows() { + // Nothing listens on this port; connect must fail within its timeout, not hang. + XCTAssertThrowsError( + try LumenConnection( + host: "127.0.0.1", port: 9, width: 640, height: 480, refreshHz: 30, + timeoutMs: 2000)) + } +} diff --git a/clients/apple/Tests/LumenKitTests/RemoteFirstLightTests.swift b/clients/apple/Tests/LumenKitTests/RemoteFirstLightTests.swift new file mode 100644 index 0000000..afd094e --- /dev/null +++ b/clients/apple/Tests/LumenKitTests/RemoteFirstLightTests.swift @@ -0,0 +1,80 @@ +// First light, headless: the full client pipeline against a REAL remote host — QUIC +// handshake over the LAN, NVENC HEVC AUs through FEC + AES-GCM, AnnexB conversion, and a +// real VTDecompressionSession turning them into pixels. Everything the GUI does except +// putting the layer on glass. +// +// Run (host side, on the Linux box): +// LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 \ +// lumen-host m3-host --source virtual --seconds 120 +// Then here: +// LUMEN_REMOTE_HOST=192.168.1.70 swift test --filter RemoteFirstLightTests + +import CoreMedia +import VideoToolbox +import XCTest +@testable import LumenKit + +final class RemoteFirstLightTests: XCTestCase { + func testRemoteStreamDecodesToPixels() throws { + guard let host = ProcessInfo.processInfo.environment["LUMEN_REMOTE_HOST"] else { + throw XCTSkip("set LUMEN_REMOTE_HOST (and start m3-host --source virtual there)") + } + let width: UInt32 = 1280 + let height: UInt32 = 720 + + let conn = try LumenConnection( + host: host, width: width, height: height, refreshHz: 60) + defer { conn.close() } + XCTAssertEqual(conn.width, width) + XCTAssertEqual(conn.height, height) + + var format: CMVideoFormatDescription? + var decoder: VTDecompressionSession? + defer { decoder.map { VTDecompressionSessionInvalidate($0) } } + var received = 0 + var decoded = 0 + var firstPtsNs: UInt64 = 0 + var lastPtsNs: UInt64 = 0 + let deadline = Date().addingTimeInterval(30) + + while decoded < 60, Date() < deadline { + guard let au = try conn.nextAU(timeoutMs: 2000) else { continue } + received += 1 + if firstPtsNs == 0 { firstPtsNs = au.ptsNs } + lastPtsNs = au.ptsNs + + if let f = AnnexB.formatDescription(fromIDR: au.data) { + format = f + if decoder == nil { + let dims = CMVideoFormatDescriptionGetDimensions(f) + XCTAssertEqual(UInt32(dims.width), width) + XCTAssertEqual(UInt32(dims.height), height) + var session: VTDecompressionSession? + XCTAssertEqual( + VTDecompressionSessionCreate( + allocator: nil, formatDescription: f, decoderSpecification: nil, + imageBufferAttributes: nil, outputCallback: nil, + decompressionSessionOut: &session), + noErr) + decoder = session + } + } + guard let f = format, let dec = decoder, + let sample = AnnexB.sampleBuffer(au: au, format: f) + else { continue } + + var gotPixels = false + VTDecompressionSessionDecodeFrame( + dec, sampleBuffer: sample, flags: [], infoFlagsOut: nil + ) { status, _, imageBuffer, _, _ in + gotPixels = status == noErr && imageBuffer != nil + } + if gotPixels { decoded += 1 } + } + + XCTAssertGreaterThanOrEqual(decoded, 60, "decoded \(decoded)/\(received) received AUs") + // The host stamps pts with its capture wall clock — 60 frames should span ~1 s. + let spanMs = Double(lastPtsNs &- firstPtsNs) / 1_000_000 + print("first light: \(decoded) frames decoded, \(received) received, pts span \(Int(spanMs)) ms") + } +} diff --git a/clients/apple/Tests/LumenKitTests/VideoToolboxRoundTripTests.swift b/clients/apple/Tests/LumenKitTests/VideoToolboxRoundTripTests.swift new file mode 100644 index 0000000..e9ab429 --- /dev/null +++ b/clients/apple/Tests/LumenKitTests/VideoToolboxRoundTripTests.swift @@ -0,0 +1,176 @@ +// Real-bitstream proof of the decode-prep path: VTCompressionSession encodes HEVC, we +// rebuild the host's wire shape (Annex-B AU with in-band VPS/SPS/PPS — exactly what +// lumen-host emits on every IDR), run it through AnnexB, and hand the result to a real +// VTDecompressionSession. Pixels out = the whole client decode path is sound. + +import AVFoundation +import CoreMedia +import VideoToolbox +import XCTest +@testable import LumenKit + +final class VideoToolboxRoundTripTests: XCTestCase { + private let width = 320 + private let height = 240 + + func testEncodeAnnexBDecodeRoundTrip() throws { + let (formatDesc, avccSample) = try encodeOneHEVCKeyframe() + + // Rebuild the host's wire format: Annex-B AU, parameter sets in-band before the VCL. + let annexB = try annexBAU(formatDesc: formatDesc, avccSample: avccSample) + + // 1) Parameter-set extraction → format description. + let rebuilt = try XCTUnwrap( + AnnexB.formatDescription(fromIDR: annexB), + "in-band VPS/SPS/PPS should yield a format description") + let dims = CMVideoFormatDescriptionGetDimensions(rebuilt) + XCTAssertEqual(Int(dims.width), width) + XCTAssertEqual(Int(dims.height), height) + + // 2) Annex-B → AVCC re-pack must reproduce the encoder's own sample bytes. + XCTAssertEqual(AnnexB.avcc(from: annexB), avccSample) + + // 3) Sample buffer → real decoder → pixels. + let au = AccessUnit(data: annexB, ptsNs: 1_000_000, frameIndex: 0, flags: 0) + let sample = try XCTUnwrap(AnnexB.sampleBuffer(au: au, format: rebuilt)) + + var session: VTDecompressionSession? + XCTAssertEqual( + VTDecompressionSessionCreate( + allocator: nil, formatDescription: rebuilt, decoderSpecification: nil, + imageBufferAttributes: nil, outputCallback: nil, + decompressionSessionOut: &session), + noErr) + let decoder = try XCTUnwrap(session) + defer { VTDecompressionSessionInvalidate(decoder) } + + var decoded: CVImageBuffer? + var decodeStatus: OSStatus = -1 + // No async flag → the handler runs before DecodeFrame returns. + VTDecompressionSessionDecodeFrame( + decoder, sampleBuffer: sample, flags: [], infoFlagsOut: nil + ) { status, _, imageBuffer, _, _ in + decodeStatus = status + decoded = imageBuffer + } + XCTAssertEqual(decodeStatus, noErr) + let pixels = try XCTUnwrap(decoded) // CVImageBuffer and CVPixelBuffer are the same CF type + XCTAssertEqual(CVPixelBufferGetWidth(pixels), width) + XCTAssertEqual(CVPixelBufferGetHeight(pixels), height) + } + + // MARK: - encode helpers + + /// One forced-IDR HEVC frame; returns its format description and raw AVCC sample bytes. + private func encodeOneHEVCKeyframe() throws -> (CMVideoFormatDescription, Data) { + var session: VTCompressionSession? + let rc = VTCompressionSessionCreate( + allocator: nil, width: Int32(width), height: Int32(height), + codecType: kCMVideoCodecType_HEVC, encoderSpecification: nil, + imageBufferAttributes: nil, compressedDataAllocator: nil, + outputCallback: nil, refcon: nil, compressionSessionOut: &session) + guard rc == noErr, let encoder = session else { + throw XCTSkip("no HEVC encoder available (\(rc))") + } + defer { VTCompressionSessionInvalidate(encoder) } + VTSessionSetProperty(encoder, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue) + VTSessionSetProperty( + encoder, key: kVTCompressionPropertyKey_AllowFrameReordering, value: kCFBooleanFalse) + + let lock = NSLock() + var output: CMSampleBuffer? + let done = expectation(description: "encoded") + VTCompressionSessionEncodeFrame( + encoder, imageBuffer: try gradientPixelBuffer(), + presentationTimeStamp: CMTime(value: 0, timescale: 30), + duration: CMTime(value: 1, timescale: 30), + frameProperties: [kVTEncodeFrameOptionKey_ForceKeyFrame: kCFBooleanTrue] as CFDictionary, + infoFlagsOut: nil + ) { status, _, sample in + XCTAssertEqual(status, noErr) + lock.lock() + output = sample + lock.unlock() + done.fulfill() + } + VTCompressionSessionCompleteFrames(encoder, untilPresentationTimeStamp: .invalid) + wait(for: [done], timeout: 10) + + lock.lock() + defer { lock.unlock() } + let sample = try XCTUnwrap(output) + let desc = try XCTUnwrap(CMSampleBufferGetFormatDescription(sample)) + let block = try XCTUnwrap(CMSampleBufferGetDataBuffer(sample)) + var bytes = Data(count: CMBlockBufferGetDataLength(block)) + try bytes.withUnsafeMutableBytes { raw in + let rc = CMBlockBufferCopyDataBytes( + block, atOffset: 0, dataLength: raw.count, + destination: raw.baseAddress!) + if rc != noErr { throw NSError(domain: "CMBlockBuffer", code: Int(rc)) } + } + return (desc, bytes) + } + + /// The host's wire shape: 4-byte start codes, VPS/SPS/PPS in-band, then the VCL NALs. + private func annexBAU(formatDesc: CMVideoFormatDescription, avccSample: Data) throws -> Data { + var au = Data() + + var psCount = 0 + var nalHeaderLen: Int32 = 0 + XCTAssertEqual( + CMVideoFormatDescriptionGetHEVCParameterSetAtIndex( + formatDesc, parameterSetIndex: 0, parameterSetPointerOut: nil, + parameterSetSizeOut: nil, parameterSetCountOut: &psCount, + nalUnitHeaderLengthOut: &nalHeaderLen), + noErr) + XCTAssertEqual(nalHeaderLen, 4, "AnnexB.avcc assumes 4-byte NAL length prefixes") + for i in 0..? + var size = 0 + XCTAssertEqual( + CMVideoFormatDescriptionGetHEVCParameterSetAtIndex( + formatDesc, parameterSetIndex: i, parameterSetPointerOut: &ptr, + parameterSetSizeOut: &size, parameterSetCountOut: nil, + nalUnitHeaderLengthOut: nil), + noErr) + au.append(contentsOf: [0, 0, 0, 1]) + au.append(Data(bytes: try XCTUnwrap(ptr), count: size)) + } + + // AVCC sample (4-byte BE length per NAL) → start codes. + var i = avccSample.startIndex + while i + 4 <= avccSample.endIndex { + let len = avccSample[i.. CVPixelBuffer { + var pb: CVPixelBuffer? + let attrs = [kCVPixelBufferIOSurfacePropertiesKey: [:]] as CFDictionary + XCTAssertEqual( + CVPixelBufferCreate(nil, width, height, kCVPixelFormatType_32BGRA, attrs, &pb), + kCVReturnSuccess) + let buf = try XCTUnwrap(pb) + CVPixelBufferLockBaseAddress(buf, []) + defer { CVPixelBufferUnlockBaseAddress(buf, []) } + let base = try XCTUnwrap(CVPixelBufferGetBaseAddress(buf)) + let stride = CVPixelBufferGetBytesPerRow(buf) + for y in 0../dev/null || true' EXIT +sleep 1 + +cd clients/apple +LUMEN_LOOPBACK_PORT="$PORT" swift test --filter LoopbackIntegrationTests diff --git a/crates/lumen-host/src/gamestream/mod.rs b/crates/lumen-host/src/gamestream/mod.rs index f1cb75a..b630bdf 100644 --- a/crates/lumen-host/src/gamestream/mod.rs +++ b/crates/lumen-host/src/gamestream/mod.rs @@ -7,7 +7,26 @@ //! the media streams follow (see the M2 task list / plan). pub mod apps; +#[cfg(target_os = "linux")] mod audio; +/// Stub — the audio plane needs Linux (PipeWire capture + libopus); this keeps non-Linux +/// dev builds compiling (crate doc: "the crate compiles everywhere"). Reports failure the +/// same way the real stream thread does: by clearing `running`. +#[cfg(not(target_os = "linux"))] +mod audio { + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::{Arc, Mutex}; + + pub fn start( + running: Arc, + _gcm_key: [u8; 16], + _rikeyid: i32, + _audio_cap: Arc>>>, + ) { + tracing::error!("GameStream audio requires Linux (PipeWire + libopus)"); + running.store(false, Ordering::SeqCst); + } +} pub(crate) mod cert; mod control; mod crypto; diff --git a/crates/lumen-host/src/gamestream/stream.rs b/crates/lumen-host/src/gamestream/stream.rs index a30660f..239f022 100644 --- a/crates/lumen-host/src/gamestream/stream.rs +++ b/crates/lumen-host/src/gamestream/stream.rs @@ -144,6 +144,7 @@ type PacketBatch = Vec>; /// Send `pkts` with as few syscalls as possible (`sendmmsg`, up to 64 per call). The socket is /// connected, so no per-message address. Returns an error on the first send failure. +#[cfg(target_os = "linux")] fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec]) -> std::io::Result<()> { use std::os::fd::AsRawFd; const CHUNK: usize = 64; @@ -179,6 +180,16 @@ fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec]) -> std::io::Result<()> { Ok(()) } +/// Portable fallback (non-Linux dev builds — GameStream hosting never ships there): one +/// syscall per packet. +#[cfg(not(target_os = "linux"))] +fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec]) -> std::io::Result<()> { + for p in pkts { + sock.send(p)?; + } + Ok(()) +} + /// Dedicated send thread: one [`PacketBatch`] per frame arrives on `rx`; its packets go out in /// `sendmmsg` chunks, paced so the frame's data spreads over ~3/4 of the frame interval /// (microburst shaping at chunk granularity — a real link drops line-rate bursts; the encode diff --git a/crates/lumen-host/src/inject/libei.rs b/crates/lumen-host/src/inject/libei.rs index 4b0b69d..fd315ab 100644 --- a/crates/lumen-host/src/inject/libei.rs +++ b/crates/lumen-host/src/inject/libei.rs @@ -370,10 +370,12 @@ impl EiState { InputKind::MouseScroll => match slot.interface::() { Some(s) => { // GameStream sends WHEEL_DELTA(120)-scaled deltas in `x`; ei scroll_discrete - // uses the same 120-per-detent unit. Positive GameStream = up/left, which is - // negative on the ei axis (matches wl_pointer). + // uses the same 120-per-detent unit. Positive GameStream = up (vertical), + // which is negative on the ei axis, but = RIGHT (horizontal), which is + // already positive there (moonlight-qt/Sunshine pass horizontal through + // unnegated) — only the vertical axis flips. if ev.code == SCROLL_HORIZONTAL { - s.scroll_discrete(-ev.x, 0); + s.scroll_discrete(ev.x, 0); } else { s.scroll_discrete(0, -ev.x); } diff --git a/crates/lumen-host/src/inject/wlr.rs b/crates/lumen-host/src/inject/wlr.rs index 8700ded..96c3666 100644 --- a/crates/lumen-host/src/inject/wlr.rs +++ b/crates/lumen-host/src/inject/wlr.rs @@ -226,10 +226,17 @@ impl InputInjector for WlrootsInjector { wl_pointer::Axis::VerticalScroll }; // GameStream sends WHEEL_DELTA(120)-scaled units; a notch ≈ 15px. Positive - // GameStream = scroll up, which is negative on the Wayland axis. + // GameStream = up (vertical), negative on the Wayland axis; but = RIGHT + // (horizontal), already positive there (moonlight-qt/Sunshine pass + // horizontal through unnegated) — only the vertical axis flips. let notches = event.x as f64 / 120.0; + let sign = if event.code == SCROLL_HORIZONTAL { + 1.0 + } else { + -1.0 + }; self.pointer.axis_source(wl_pointer::AxisSource::Wheel); - self.pointer.axis(t, axis, -notches * 15.0); + self.pointer.axis(t, axis, sign * notches * 15.0); self.pointer.frame(); } InputKind::KeyDown | InputKind::KeyUp => { diff --git a/crates/lumen-host/src/m3.rs b/crates/lumen-host/src/m3.rs index 252e1a4..b35e350 100644 --- a/crates/lumen-host/src/m3.rs +++ b/crates/lumen-host/src/m3.rs @@ -448,6 +448,7 @@ fn input_thread(rx: std::sync::mpsc::Receiver, conn: quinn::Connecti /// The audio thread: desktop capture → Opus (48 kHz stereo, 5 ms, CBR — same tuning as the /// GameStream path) → `AUDIO_MAGIC` datagrams. QUIC already encrypts; no extra layer. /// The capturer comes from (and returns to) the persistent slot — see [`AudioCapSlot`]. +#[cfg(target_os = "linux")] fn audio_thread(conn: quinn::Connection, stop: Arc, audio_cap: AudioCapSlot) { use crate::audio::{CHANNELS, SAMPLE_RATE}; const FRAME_MS: usize = 5; @@ -519,6 +520,15 @@ fn audio_thread(conn: quinn::Connection, stop: Arc, audio_cap: Audio } } +/// Stub — lumen/1 audio needs Linux (PipeWire capture + libopus); non-Linux dev builds +/// run sessions without it, same as when the capturer fails to open. +#[cfg(not(target_os = "linux"))] +fn audio_thread(_conn: quinn::Connection, _stop: Arc, _audio_cap: AudioCapSlot) { + tracing::warn!( + "lumen/1 audio requires Linux (PipeWire + libopus) — session continues without it" + ); +} + fn synthetic_stream(session: &mut Session, frames: u32, stop: &AtomicBool) -> Result<()> { let interval = std::time::Duration::from_millis(1000 / 60); for idx in 0..frames { diff --git a/scripts/build-xcframework.sh b/scripts/build-xcframework.sh index 9330daf..fef7eb0 100644 --- a/scripts/build-xcframework.sh +++ b/scripts/build-xcframework.sh @@ -13,11 +13,13 @@ cd "$(dirname "$0")/.." TARGETS_MAC=(aarch64-apple-darwin x86_64-apple-darwin) BUILD_IOS="${BUILD_IOS:-0}" # BUILD_IOS=1 adds an iOS slice (requires the ios target installed) +# Deployment targets must match Package.swift's platforms, or every consumer link emits +# "object file was built for newer macOS version" warnings. for t in "${TARGETS_MAC[@]}"; do - cargo build --release -p lumen-core --features quic --target "$t" + MACOSX_DEPLOYMENT_TARGET=14.0 cargo build --release -p lumen-core --features quic --target "$t" done if [[ "$BUILD_IOS" == "1" ]]; then - cargo build --release -p lumen-core --features quic --target aarch64-apple-ios + IPHONEOS_DEPLOYMENT_TARGET=17.0 cargo build --release -p lumen-core --features quic --target aarch64-apple-ios fi STAGE="$(mktemp -d)" @@ -49,6 +51,18 @@ if [[ "$BUILD_IOS" == "1" ]]; then ARGS+=(-library target/aarch64-apple-ios/release/liblumen_core.a -headers "$STAGE/include") fi +# Cargo does NOT fingerprint MACOSX_DEPLOYMENT_TARGET — units cached from a build without +# it keep their old minos forever. Refuse to ship anything newer than the package floor +# (objects BELOW it, e.g. rustup's precompiled std at 11.0, are fine and unavoidable). +for obj in "$STAGE"/macos/liblumen_core.a; do + bad=$(otool -l "$obj" 2>/dev/null | awk '/minos/ {print $2}' | sort -uV | awk -F. '$1 > 14' | head -1) + if [[ -n "$bad" ]]; then + echo "ERROR: $obj contains objects built for macOS $bad (> 14.0)." >&2 + echo "Stale cache — rm -rf target/{aarch64,x86_64}-apple-darwin and rebuild." >&2 + exit 1 + fi +done + rm -rf clients/apple/LumenCore.xcframework xcodebuild -create-xcframework "${ARGS[@]}" -output clients/apple/LumenCore.xcframework echo "OK: clients/apple/LumenCore.xcframework"