feat: M4 stage 1 — the SwiftUI client is real: compiles, tested, first light on glass
ci / rust (push) Has been cancelled
ci / rust (push) Has been cancelled
The clients/apple scaffold is now a working macOS client, validated live against this repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC + AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60, mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector (~3.7k events injected in one session). LumenKit: - LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as integers while the enum constants import as a distinct Swift type — bridge by rawValue); close() is now safe from any thread (a close flag + pumpLock held across the blocking poll enforce the C contract "never close with a next_au in flight"; flag prevents lock-starvation by back-to-back polls). - StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate on the next in-band parameter sets when the layer fails, no stale enqueue after restart. - InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away), pressed-state tracking with release-all on focus loss and stop() (nothing sticks down host-side), global-singleton ownership guard (GC has one handler slot per process), X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs. - LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD, LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs. - Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels); test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN). Host/build fixes that fell out: - The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule. - Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the ei/wl axes, but GameStream's horizontal convention is positive = right (moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also un-inverts real Moonlight clients. - AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's policy, instead of leaking them into the preceding NAL. - build-xcframework.sh: deployment targets pinned to the package floor + an otool guard — cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship too-new minos objects. Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified): 14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+86
-76
@@ -1,102 +1,112 @@
|
||||
# lumen Apple client (SwiftUI) — handoff
|
||||
# lumen Apple client (SwiftUI)
|
||||
|
||||
The native macOS/iOS client for **`lumen/1`** (the post-GameStream protocol). All
|
||||
networking/protocol work — QUIC control plane, UDP data plane, GF(2¹⁶) FEC, AES-GCM,
|
||||
input datagrams — lives in the shared Rust core and is **done and tested**; this package
|
||||
is the Swift shell: decode (VideoToolbox), present (SwiftUI), input capture.
|
||||
input datagrams, Opus audio, cert pinning — lives in the shared Rust core (statically
|
||||
linked as `LumenCore.xcframework`); this package is the Swift shell: decode
|
||||
(VideoToolbox), present (SwiftUI), input capture.
|
||||
|
||||
## What exists (built + tested on the Linux host)
|
||||
## Status — first light achieved (2026-06-10)
|
||||
|
||||
- **The connector**: `lumen_core::client::NativeClient` (Rust) exposed over the C ABI as
|
||||
`lumen_connect` / `lumen_connection_next_au` / `lumen_connection_next_audio` /
|
||||
`lumen_connection_next_rumble` / `lumen_connection_send_input` / `lumen_connection_mode`
|
||||
/ `lumen_connection_close` (see `include/lumen_core.h`, guarded by `LUMEN_FEATURE_QUIC`).
|
||||
**End-to-end tested through the C ABI** against an in-process host
|
||||
(`crates/lumen-host/src/m3.rs::tests::c_abi_connection_roundtrip` — three sequential
|
||||
sessions: TOFU, pinned reconnect, wrong-pin rejection).
|
||||
- **The host to test against**: `lumen-host m3-host --source virtual --seconds 60` on the
|
||||
Linux box — a **persistent listener** (sessions back to back, reconnect at will during
|
||||
development; `--max-sessions N` to bound it). It creates a native virtual output at
|
||||
whatever mode the client requests and streams HEVC + desktop **Opus audio**;
|
||||
`LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube` for moving content.
|
||||
- **This package (SCAFFOLD — written on Linux, never compiled in Xcode)**:
|
||||
- `LumenConnection.swift` — Swift wrapper over the C ABI (AUs/audio copied into `Data`;
|
||||
certificate pinning + TOFU fingerprint via `pinSHA256:`/`hostFingerprint`).
|
||||
Validated live, Mac ↔ Linux box over the LAN: gamescope virtual output → NVENC HEVC →
|
||||
`lumen/1` (GF(2¹⁶) FEC + AES-GCM over UDP, QUIC control) → VideoToolbox →
|
||||
`AVSampleBufferDisplayLayer` on glass at 1280×720@60, with mouse/keyboard flowing back as
|
||||
QUIC datagrams into the host's gamescope EIS injector (thousands of events injected during
|
||||
the session). Headless variant of the same proof: `RemoteFirstLightTests` decoded 60/60
|
||||
received AUs spanning 983 ms of host capture clock.
|
||||
|
||||
The connector underneath (`lumen_core::client::NativeClient` over the C ABI) carries the
|
||||
full session: video AUs, **Opus audio** (`nextAudio()`), **rumble** (`nextRumble()`),
|
||||
input incl. gamepads, and **cert pinning + TOFU** (`pinSHA256:`/`hostFingerprint`) — see
|
||||
`m3.rs::tests::c_abi_connection_roundtrip` (three sequential sessions: TOFU, pinned
|
||||
reconnect, wrong-pin rejection). The host (`lumen-host m3-host`) is a persistent listener:
|
||||
reconnect at will during development.
|
||||
|
||||
What's here, all compiled and tested on macOS (Xcode 26.5 / Swift 6.3):
|
||||
|
||||
- **`LumenKit`** (library)
|
||||
- `LumenConnection.swift` — wrapper over the C ABI. AUs/audio are copied into `Data`
|
||||
(the C pointer is only valid until the next call of the same kind). `close()` is safe
|
||||
from any thread: per-plane locks enforce the C contract ("never close with a
|
||||
`next_au`/`next_audio` in flight") instead of leaving it to callers. Pinning + TOFU
|
||||
via `pinSHA256:`/`hostFingerprint`.
|
||||
- `AnnexB.swift` — in-band VPS/SPS/PPS → `CMVideoFormatDescription`; Annex-B → AVCC
|
||||
`CMSampleBuffer` with `DisplayImmediately` set.
|
||||
- `StreamView.swift` — SwiftUI `NSViewRepresentable` over `AVSampleBufferDisplayLayer`
|
||||
(stage-1 presenter: the layer hardware-decodes compressed HEVC itself).
|
||||
- `InputCapture.swift` — `GCMouse` raw deltas + `GCKeyboard` HID→VK mapping →
|
||||
`lumen_connection_send_input`.
|
||||
(stage-1 presenter: the layer hardware-decodes compressed HEVC itself). One pump
|
||||
thread per view, token-cancelled so reconnects can't double-pump.
|
||||
- `InputCapture.swift` — `GCMouse` raw deltas + `GCKeyboard` HID→VK mapping (the host's
|
||||
`vk_to_evdev` consumes Windows VKs), with fractional-delta accumulation so sub-pixel
|
||||
motion isn't truncated away. Buttons use GameStream ids (1=left … 5=X2); scroll is
|
||||
WHEEL_DELTA(120)-scaled.
|
||||
- **`LumenClient`** (development app shell): connect form → stream + input, fps/Mb-s HUD.
|
||||
(Audio playback and gamepad capture are not wired into the app yet — the connector
|
||||
surface is there; see notes 5–6.)
|
||||
- **Tests** (`swift test`): byte-level Annex-B units; a real-codec round trip
|
||||
(VTCompressionSession-encoded HEVC rebuilt as the host's wire shape → `AnnexB` →
|
||||
VTDecompressionSession → pixels); loopback integration against a real local host
|
||||
(`test-loopback.sh`); the remote first-light test above.
|
||||
|
||||
## Build steps (on the Mac)
|
||||
## Build / run / test (on a Mac)
|
||||
|
||||
```sh
|
||||
rustup target add aarch64-apple-darwin x86_64-apple-darwin
|
||||
bash scripts/build-xcframework.sh # → clients/apple/LumenCore.xcframework
|
||||
open clients/apple/Package.swift # or add the package to an Xcode app project
|
||||
cd clients/apple
|
||||
swift build && swift test # loopback/remote tests self-skip without a host
|
||||
swift run LumenClient # the app; or open Package.swift in Xcode
|
||||
|
||||
bash test-loopback.sh # full loopback proof: builds lumen-host
|
||||
# (synthetic source — runs on macOS), streams
|
||||
# byte-verified frames into the Swift client
|
||||
|
||||
# against the real host (Linux box, see CLAUDE.md "Running on this box") — m3-host is a
|
||||
# persistent listener, reconnect at will:
|
||||
# LUMEN_COMPOSITOR=gamescope LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 \
|
||||
# cargo run -rp lumen-host -- m3-host --source virtual --seconds 60
|
||||
LUMEN_REMOTE_HOST=<box-ip> swift test --filter RemoteFirstLightTests # headless
|
||||
LUMEN_AUTOCONNECT=<box-ip> LUMEN_MODE=1280x720x60 swift run LumenClient # on glass
|
||||
```
|
||||
|
||||
Minimal app around it:
|
||||
## Notes for whoever picks this up next
|
||||
|
||||
```swift
|
||||
@main struct LumenApp: App {
|
||||
var body: some Scene { WindowGroup { ContentView() } }
|
||||
}
|
||||
struct ContentView: View {
|
||||
@State private var conn: LumenConnection?
|
||||
var body: some View {
|
||||
if let conn {
|
||||
StreamView(connection: conn)
|
||||
.onAppear { InputCapture(connection: conn).start() }
|
||||
} else {
|
||||
Button("Connect") {
|
||||
conn = try? LumenConnection(
|
||||
host: "192.168.1.70", width: 2560, height: 1440, refreshHz: 120)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Handoff — what the next agent needs to know
|
||||
|
||||
1. **Expect small compile fixes.** Every Swift file is flagged SCAFFOLD: API-checked from
|
||||
documentation, never run through Xcode. Likely friction: the imported C enum spellings
|
||||
(`LUMEN_STATUS_OK` etc. — cbindgen emits `QualifiedScreamingSnakeCase`), `LumenFrame()`
|
||||
zero-init, `_pad` tuple shape on `LumenInputEvent`.
|
||||
2. **ABI contract** (matches `lumen_core.h` docs): `next_au`'s pointer is valid only until
|
||||
the *next* call on that handle (we copy to `Data` immediately); one pump thread per
|
||||
connection, plus optionally one *separate* audio thread for `next_audio` (independent
|
||||
borrow slots); `send_input` is enqueue-only and thread-safe alongside both; `close`
|
||||
joins the Rust threads — never call it with a `next_au`/`next_audio` call in flight.
|
||||
3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band,
|
||||
and recovery keyframes re-send them — so "wait for the first format description, refresh
|
||||
it on every IDR" (already what `StreamView` does) is sufficient; there is no out-of-band
|
||||
extradata, ever.
|
||||
4. **First-light test**: Linux box runs
|
||||
`PATH=/tmp/gamescope-src/build/src:$PATH LUMEN_COMPOSITOR=gamescope \
|
||||
LUMEN_GAMESCOPE_APP=vkcube LUMEN_ZEROCOPY=1 cargo run -rp lumen-host -- m3-host
|
||||
--source virtual --seconds 120`; Mac connects with the app. Success = the spinning
|
||||
vkcube on glass. Then mouse/keys should appear inside the gamescope session (verify
|
||||
with `LUMEN_GAMESCOPE_APP=xev` and the box-side log `/tmp/lumen-gamescope.log`).
|
||||
5. **Stage 2 (after first light)**: replace `AVSampleBufferDisplayLayer` with explicit
|
||||
`VTDecompressionSession` + `CAMetalLayer` for frame-pacing control (ProMotion/120 Hz),
|
||||
and add glass-to-glass measurement (`tools/latency-probe` is the scaffold; the host
|
||||
already stamps `pts_ns` with its capture wall clock — across machines you'll need a
|
||||
clock-offset estimate from the QUIC RTT, or the probe's visual timestamp loop).
|
||||
6. **Audio**: `nextAudio()` yields raw Opus packets (48 kHz stereo, one 5 ms frame each,
|
||||
1. **cbindgen import quirk** (the predicted "small compile fixes", now fixed): the
|
||||
C17-compatible header spells `LumenStatus`/`LumenInputKind` as integer typedefs while
|
||||
the enum *constants* import into Swift as a distinct same-named type — bridge with
|
||||
`.rawValue` (see the top of `LumenConnection.swift`). Don't fight the generated header.
|
||||
2. **ABI contract**: one video pump thread per connection, plus optionally one *separate*
|
||||
audio drain thread for `nextAudio()`/`nextRumble()` (the core keeps per-plane borrow
|
||||
slots, so the planes never alias); `send()` is enqueue-only and safe alongside all of
|
||||
them. The wrapper's per-plane locks make `close()` safe from anywhere (it waits out
|
||||
in-flight polls, ≤ their timeouts).
|
||||
3. **Decode flow**: the host opens every stream with an IDR carrying VPS/SPS/PPS in-band
|
||||
and recovery keyframes re-send them — "refresh the format description on every IDR"
|
||||
(what `StreamView` does) is sufficient; there is no out-of-band extradata, ever.
|
||||
4. **Stage 2 (next)**: explicit `VTDecompressionSession` + `CAMetalLayer` for frame-pacing
|
||||
control (ProMotion/120 Hz), glass-to-glass measurement via `tools/latency-probe` (the
|
||||
host stamps `pts_ns` with its capture wall clock; across machines you need a clock
|
||||
offset estimate from the QUIC RTT).
|
||||
5. **Audio**: `nextAudio()` yields raw Opus packets (48 kHz stereo, one 5 ms frame each,
|
||||
sequence-numbered). Decode with libopus or `AVAudioConverter`/`kAudioFormatOpus` into an
|
||||
`AVAudioEngine` source node; conceal gaps (drop/dup) rather than blocking — the Rust
|
||||
side buffers 320 ms and drops the newest packet when the puller lags. Wall-clock
|
||||
`ptsNs` shares the host clock with video AUs for A/V sync.
|
||||
7. **Gamepads**: `GCController` → `.gamepadButton(...)`/`.gamepadAxis(...)` events (wire
|
||||
`ptsNs` shares the host clock with video AUs for A/V sync. Wiring this into
|
||||
`LumenClient` is the next app-side task.
|
||||
6. **Gamepads**: `GCController` → `.gamepadButton(...)`/`.gamepadAxis(...)` events (wire
|
||||
contract documented on the constructors; the host accumulates them into a virtual
|
||||
Xbox 360 pad). Poll `nextRumble()` and feed `GCDeviceHaptics` for force feedback.
|
||||
8. **Trust**: connect once with `pinSHA256: nil` (TOFU), persist `hostFingerprint` keyed
|
||||
Client-side capture isn't in `InputCapture` yet.
|
||||
7. **Trust**: connect once with `pinSHA256: nil` (TOFU), persist `hostFingerprint` keyed
|
||||
by host, pass it on every later connect — a mismatch throws `.connectFailed`. The host
|
||||
logs its fingerprint at startup ("clients pin this fingerprint") for out-of-band
|
||||
verification UX; a PIN-style pairing ceremony is a later lumen-core task.
|
||||
verification UX; a PIN-style pairing ceremony is a later lumen-core task. `LumenClient`
|
||||
doesn't persist fingerprints yet — add it alongside the "add host" UX.
|
||||
8. **Input capture caveats** (stage 1): GC handlers only fire while the app has focus —
|
||||
on focus loss `InputCapture` auto-releases everything still held (keys + buttons) so
|
||||
nothing sticks down host-side. Local shortcuts (⌘-anything) still also reach the host;
|
||||
a capture toggle is a small follow-up. One live capture per process (the GC mouse/
|
||||
keyboard singletons have a single handler slot — ownership is tracked so a stale
|
||||
capture's stop() can't clobber a newer one).
|
||||
9. **iOS**: same package (`BUILD_IOS=1` for the xcframework slice); `StreamView` needs the
|
||||
`UIViewRepresentable` twin and touch→input mapping.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user