b6f4164454ff90d94be9c0bdb3ab7137c67ecb2c
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 enricobuehler
|
0755c823a5 |
feat: mic passthrough — client microphone → host virtual PipeWire source
ci / rust (push) Has been cancelled
The inverse of the host→client audio path: the client's mic, Opus-encoded, rides a new 0xCB QUIC datagram to the host, which decodes it into a virtual PipeWire Audio/Source its apps can record from (voice chat, etc.). Protocol (punktfunk-core): - MIC_MAGIC 0xCB + encode/decode_mic_datagram (mirror of the 0xC9 audio datagram). - NativeClient::send_mic(seq, pts_ns, opus) over a new outbound channel + worker task (mirror of send_input); C ABI punktfunk_connection_send_mic for native clients. Host: - audio::VirtualMic + PwMicSource: a PipeWire output stream tagged media.class= Audio/Source (Direction::Output) — a recordable microphone node, fed decoded PCM. - MicService: host-lifetime owner of the source + Opus decoder (mirror of InjectorService / the audio capturer slot); lazily opened, persists across sessions, self-heals. The per-session datagram reader now demuxes 0xCB→mic / 0xC8→input over a single read_datagram loop (two loops would race). - Adaptive jitter buffer in the producer: primes to ~3 consumer quanta before emitting, so the 5 ms push / N ms pull clock skew never underruns — without it ~58% of output was silence; with it, glitch-free across consumer quanta. Client: punktfunk-client-rs --mic-test streams a synthetic 440 Hz Opus tone as the mic uplink (opus dep added) for end-to-end validation without a real microphone. Validated live on headless KWin: client tone → host source → pw-record shows the punktfunk-mic Audio/Source node, 440 Hz dominant (Goertzel power 20.7 vs <0.001 elsewhere), RMS 0.179 ≈ the ideal 0.177, 0.3–0.4% silence at both 256 ms and 10 ms consumer quanta. Tests +1 (mic datagram roundtrip); workspace green, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
ff4fe197be |
fix(punktfunk/1): adversarial-review fixes — SPAKE2 pairing, renegotiation hardening, +more
ci / rust (push) Has been cancelled
Triaged the multi-agent review of the renegotiation + pairing + Sway + AV1/surround batch
(1 critical, 11 major/minor confirmed). Fixes:
CRITICAL — PIN pairing was offline-brute-forceable. The HMAC-of-PIN proof let an active
MITM who terminates the TOFU ceremony recover the 4-digit PIN by offline dictionary search
(all other inputs observable) and forge a correctly-bound proof. Replaced with **SPAKE2**
(balanced PAKE, `spake2` crate) + key-confirmation MACs, binding both cert fingerprints as
the SPAKE2 identities: an attacker gets exactly ONE online guess, no offline search, and
mismatched cert views (a real MITM) never reach a shared key. Also reworked the UX to an
"arming PIN" — one PIN per arming window shown at host startup (the SPAKE2 client needs the
PIN to build its first message, so it can't be minted per-connection). Validated live:
wrong PIN rejected in 0.1s, right PIN pairs + persists + the paired identity streams.
Pairing hardening: `--allow-pairing`/`--require-pairing` must arm pairing (default rejects
unsolicited ceremonies); per-host cooldown bounds online guessing; the client flushes its
CONNECTION_CLOSE so a refused ceremony can't wedge the sequential host for the full timeout;
atomic (temp+rename) paired-store writes.
Protocol: control/pairing messages use a distinct CTL_MAGIC (PKFc) — fully disjoint from
the positional Hello namespace (a future abi_version can't be misparsed as a control
message); all typed decodes are length-exact. ABI_VERSION → 2 (punktfunk_connect signature
gained the identity params; header regenerated).
Renegotiation: drain the reconfig channel to the NEWEST mode (one rebuild, not one per
stale step); validate refresh_hz; build the new pipeline BEFORE dropping the old so a
rebuild failure keeps the session on its current mode instead of killing it.
GameStream: packetDuration snaps to {5,10} (an in-between value isn't a legal Opus frame
size and would kill audio). Sway: chooser file moved to $XDG_RUNTIME_DIR (was a fixed
world-writable /tmp path — DoS / capture-misdirection by another local user).
Swift: fixed two compile breakers in the new pairing/identity APIs (Int32 status .rawValue,
UInt cap cast). New SPAKE2 + namespace-disjointness + pairing-roundtrip unit tests; the
in-process pairing test now also exercises the arming PIN + cooldown. 114 tests green,
clippy -D warnings clean (both feature sets), fmt, C-ABI harness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
429bd1e6ac |
Merge branch 'worktree-agent-a6fe98c40d55fd284' into m1-lumen-core
# Conflicts: # CLAUDE.md |
||
|
enricobuehler
|
4d26ac5c85 |
feat: punktfunk/1 — mid-stream mode renegotiation + PIN pairing ceremony
Renegotiation (no reconnect on resize): the handshake bi-stream stays open; the client
sends Reconfigure{mode} (typed post-handshake message), the host validates + acks
Reconfigured and rebuilds capture/encoder/virtual output at the new mode while the data
plane (keys, ports, FEC) runs untouched — the first new-mode AU is an IDR with in-band
parameter sets. NativeClient::request_mode / punktfunk_connection_request_mode; mode()
reflects the active mode. Validated live on KWin: one continuous stream, 225 frames
@1280x720 then 395 @1920x1080, ~90 ms pipeline rebuild (ffprobe shows both resolutions).
PIN pairing (mutual trust, kills TOFU MITM): clients get persistent self-signed
identities presented via QUIC client auth (generate_identity / client auth offered but
optional server-side — legacy clients still connect). Ceremony on the control stream:
PairRequest{name} → host shows a 4-digit PIN (log) + PairChallenge{salt} → client proves
with HMAC-SHA256(PIN‖salt, client_fp‖host_fp) — binding both certs means a MITM can't
forward a proof, single attempt per PIN, constant-time compare → PairResult; host
persists the fingerprint (~/.config/punktfunk/punktfunk1-paired.json), client pins the
host's. m3-host --require-pairing gates sessions on the paired set.
NativeClient::pair + punktfunk_pair/punktfunk_generate_identity in the ABI; reference
client: --pair PIN --name LABEL + auto-generated persistent identity, --remode for live
renegotiation testing. Swift wrapper: ClientIdentity/generateIdentity()/pair(),
requestMode()/currentMode(); README handoff updated.
Tested: reconfigure/pairing wire roundtrips, C-ABI mode switch ack, full in-process
ceremony (wrong PIN → Crypto, anonymous-vs-gate rejection, success → pinned session);
live wrong-PIN ceremony against the serving host (PIN logged, proof rejected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
3cc3c02b42 |
feat(gamestream): AV1 negotiation + 5.1/7.1 surround audio
Codec negotiation (M2 polish):
- ServerCodecModeSupport now advertises what we encode: H264|HEVC|AV1_MAIN8
= 65793 (flags verified against moonlight-common-c Limelight.h). The old
placeholder 3843 wrongly claimed HEVC Main10 + 4:4:4 and no AV1. Main10
bits stay off on purpose: Moonlight ties 10-bit to HDR, and capture is
8-bit SDR BGRx with no HDR metadata path (av1_nvenc -highbitdepth was
validated working for later).
- RTSP ANNOUNCE: bitStreamFormat 0/1/2 -> H264/HEVC/AV1 (already plumbed to
av1_nvenc; validated e2e via `m0 --codec av1` + ffprobe av01), and a
dynamicRangeMode!=0 request now logs + falls back to 8-bit SDR.
Surround audio (M2 polish):
- ANNOUNCE x-nv-audio.surround.{numChannels,AudioQuality} +
x-nv-aqos.packetDuration -> per-session AudioParams; DESCRIBE advertises
all six Opus configs (normal before HQ per channel count). Normal-quality
mappings are pre-rotated for the client's GFE-order LFE swap
(RtspConnection.c, verified verbatim) so its derived decoder mapping
equals our encoder mapping — including 7.1, where Sunshine's rotate only
covers [3,6) and scrambles LFE/SL/SR.
- 5.1/7.1 encode via libopus multistream (audiopus_sys, the sys layer the
opus crate already links) with Sunshine's layouts/bitrates, RAII wrapper;
the live-validated stereo wire is byte-identical (plain Opus, no FEC).
- Surround sessions add Sunshine-style RS(4,2) audio FEC (packetType 127 +
AUDIO_FEC_HEADER, the OpenFEC parity matrix both ends hardcode, nanors
gemm semantics verified from nanors/rs.c).
- PipeWire capture generalized to the negotiated channel count with explicit
FL FR FC LFE RL RR [SL SR] positions; missing sink channels are zero-
filled by the channel-mixer. PwAudioCapturer now tears down cleanly on
Drop (pipewire channel -> loop quit), so a channel-count change can
reopen without leaking a capture stream.
Tests: serverinfo mask, RTSP codec/audio param parsing, DESCRIBE contents,
surround-params strings + client-swap round trip, FEC parity self-recovery
and packet layout, real-codec 5.1 channel-identity round trip, and an
ignored live test (ran green against a 6ch null sink monitor).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
bfd64ce871 |
rename: lumen → punktfunk, everywhere
ci / rust (push) Has been cancelled
Full project rename, decided 2026-06-10: - Crates/binaries: punktfunk-core / punktfunk-host / punktfunk-client-rs. - C ABI: punktfunk_* symbols, Punktfunk* types, include/punktfunk_core.h, PUNKTFUNK_FEATURE_QUIC guard (header regenerated; cbindgen renames updated, incl. PUNKTFUNK_BTN_*/PUNKTFUNK_AXIS_* wire constants). - Protocol: punktfunk/1 — control-plane magic LMN1 → PKF1, nonce salt lmn1 → pkf1. WIRE BREAK: clients must be rebuilt from this revision. - Env knobs: PUNKTFUNK_VIDEO_SOURCE / PUNKTFUNK_COMPOSITOR / PUNKTFUNK_ZEROCOPY / …. - Host config dir: ~/.config/punktfunk (the box's dir was migrated in place — the persistent identity is unchanged, pinned fingerprints stay valid). - Swift package: PunktfunkKit + PunktfunkCore.xcframework + PunktfunkConnection (Sources/PunktfunkClient app + tests renamed with it); build-xcframework.sh updated. - scripts/: 60-punktfunk.rules, punktfunk-host.service; OpenAPI doc regenerated. Also: scripts/headless/run-headless-kde.sh — full headless Plasma bringup. Root cause of "desktop but no apps/settings" over the stream: plasmashell launched without XDG_MENU_PREFIX=plasma-, so the launcher resolved a nonexistent applications.menu and rendered an empty menu. The script sets the complete KDE session env (menu prefix, KDE_FULL_SESSION, session version) and rebuilds ksycoca before starting plasmashell. Gate: 97/97 tests, clippy -D warnings (both feature sets), fmt, C-ABI harness PASS, zero lumen references left outside .git. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
520d7342dd |
feat: M3 — full lumen/1 session planes: audio, gamepads+rumble, pinned trust, persistent listener
ci / rust (push) Has been cancelled
m3-host is now a real host, not a one-shot demo. Everything validated live on this box (two back-to-back sessions, pinned + TOFU, ~200 audio pkts/s, p50 0.84 ms at 720p60). lumen-core: - quic.rs: QUIC-datagram side planes demuxed by first byte — Opus audio 0xC9 ([magic][u32 seq][u64 pts_ns][opus], host→client) and rumble 0xCA ([magic][pad][low][high]). - Trust: endpoint::server_with_identity (persistent PEM identity) and endpoint::client_pinned — SHA-256 cert-fingerprint pinning with TOFU (observed fingerprint reported back for persisting). The verifier checks the TLS 1.3 CertificateVerify signature for real (an MITM replaying the host's public cert without its key is rejected; cert pinning alone would not prove key possession). - client.rs: NativeClient gains pin + host_fingerprint, audio/rumble receivers (next_audio / next_rumble); pull methods take &self so the C ABI's per-plane threads never alias a &mut (per-plane mutexed borrow slots in abi.rs). - abi.rs: lumen_connect(pin_sha256, observed_sha256_out) + lumen_connection_next_audio / next_rumble. input.rs: documented gamepad wire contract (GameStream buttonFlags bits, XInput axis conventions, +y = up) — exported as LUMEN_BTN_*/LUMEN_AXIS_* (bare BTN_* collides with <linux/input-event-codes.h> at different values). lumen-host (m3): - Persistent accept loop: sessions back to back on one endpoint (--max-sessions, 0 = forever); per-session failures log and the loop keeps serving; 10 s handshake deadline so a silent client can't wedge the sequential accept queue; teardown on every exit path (stop flag → conn.close → join audio+input threads). - Audio plane: desktop PipeWire capture → Opus 48 kHz stereo 5 ms CBR → datagrams; ONE capturer reused across sessions via an AudioCapSlot (PipeWire streams have no cheap teardown — per-session opens would leak a thread + core connection + live node each). - Gamepad routing: incremental GamepadButton/GamepadAxis datagrams accumulate into per-pad state feeding the uinput xpad manager; force feedback returns as rumble datagrams, with current state re-sent every 500 ms (idempotent-state healing for the lossy channel). QUIC endpoint serves the persistent ~/.config/lumen identity and logs the pinnable fingerprint. lumen-client-rs: --pin (malformed values abort — never silently downgrade to TOFU), TOFU fingerprint logging, audio/rumble datagram counters, gamepad events in --input-test. clients/apple: scaffold synced — pinSHA256/hostFingerprint (wrong-size pin throws, fail-closed), nextAudio/nextRumble, gamepad event constructors; README handoff updated (persistent listener, audio decode notes, trust UX). Adversarially reviewed (5-dimension multi-agent pass over the diff, 2-skeptic verification): fixed the MITM signature-check gap, a Y-axis contract inversion, header macro collisions, ABI aliasing UB, the PipeWire per-session leak, the missing handshake deadline, fail-open pin parsing, and teardown-on-error paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
68f2b19cca |
Merge main (management REST API) into m1-lumen-core
ci / rust (push) Has been cancelled
Resolutions: serve() keeps main's AppState::new() with our persisted-pairing load folded into it; main.rs keeps both the m3 and mgmt modules; mgmt's test LaunchSessions gain the new appid field; Cargo.lock re-resolved. Full gate green (92 tests, clippy, fmt). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
de3123038f |
feat: M3 seed — the lumen/1 native protocol: QUIC control plane + reference client (Phase 5)
The first end-to-end run of lumen's own protocol, past the GameStream compatibility layer. - lumen-core/src/quic.rs (behind the `quic` feature): the lumen/1 handshake — Hello/Welcome/ Start as length-prefixed LE binary on one QUIC bi-stream. Welcome carries the COMPLETE data-plane Config: mode, FEC scheme incl. GF(2^16) Leopard (inexpressible in GameStream), shard sizing, AES-GCM key + per-direction salt, data UDP port. Plus quinn endpoint helpers (self-signed server; accepts-any client — pinning lands with the trust model) and framed async IO. Round-trip unit-tested. - lumen-host m3-host: serves one lumen/1 session — QUIC handshake, then a NATIVE thread (no async on the frame path — design invariant) streams deterministic 64KB test frames through the hardened M1 Session over UdpTransport. - lumen-client-rs: from scaffold to working reference client — connects, negotiates, brings up the client Session over UDP, reassembles + FEC-recovers + byte-verifies every frame. VALIDATED END-TO-END on localhost: 300/300 frames verified, 0 mismatches, through QUIC-negotiated GF(2^16) FEC + AES-GCM over real UDP sockets. M4 (decode+present) builds on this exact client skeleton. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
826da9968e |
feat: M2 — Vulkan bridge: TRUE zero-copy for gamescope's LINEAR dmabufs (Phase 3)
The missing zero-copy path is closed. NVIDIA's EGL won't sample LINEAR and the CUDA driver rejects raw dmabuf fds — but Vulkan imports dmabufs (VK_EXT_external_memory_dma_buf) and exports OPAQUE_FD memory that CUDA officially imports. zerocopy/vulkan.rs (ash): dmabuf fd → VkBuffer (import cached per fd) → vkCmdCopyBuffer (GPU) → exportable VkBuffer → vkGetMemoryFdKHR(OPAQUE_FD) → cuImportExternalMemory → CUdeviceptr The exportable buffer + CUDA mapping are per-resolution; per frame it's one GPU buffer copy (fence-waited) + one pitched CUDA copy into the encoder's pool. No CPU touches pixels. EglImporter::import_linear now routes through the bridge (lazy init; any failure still falls back to the CPU mmap path). cuda::ExternalDmabuf gained import_owned_fd for the Vulkan-exported fd. Validated live: gamescope 720p120 → "Vulkan→CUDA exportable staging buffer ready size=3686400" (exactly 1280*720*4), full-rate 122.7 fps, decoded frame pixel-correct (vkcube). KWin's tiled EGL path regression-tested intact. NV12 negotiation dropped — moot now that BGRx is fully zero-copy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
c8f9032dec |
feat: M2 — harden gamescope capture path (blocked on gamescope ≥3.16.22 upstream fix)
Deep investigation (gdb + daemon traces) proved the gamescope capture stall is a gamescope
3.16.20 bug, not ours: it calls pw_loop_iterate() without pw_loop_enter()/leave(), and under
PipeWire 1.6's loop locking its main thread permanently holds the loop mutex — the pw thread
deadlocks, gamescope never acks the daemon's port_set_param(Format), and the link parks in
"negotiating" silently. Stock gst pipewiresrc fails identically. Fixed upstream by gamescope
commit e3ed1ea7 ("pipewire: Fix pipewire loop locking", pipewire#5148); first release 3.16.22.
Ubuntu 26.04 ships 3.16.20 (built ten days before the fix) — patch/upgrade required.
Consumer-side improvements from the investigation (all verified correct vs gamescope's pods,
and needed once the producer is fixed):
- discover the node from gamescope's own "stream available on node ID: N" log line (its
node.name appears on two objects; the advertised id is authoritative); pw-dump fallback
- CPU path accepts mappable dmabufs: Buffers param now offers MemPtr|MemFd|DmaBuf (gamescope
counter-offers exactly DmaBuf when its modifier pod wins, never MemPtr), mmap the fd
ourselves when MAP_BUFFERS didn't (Vulkan-exported dmabufs aren't flagged mappable), and
treat chunk.size==0 as the computed span
- warn_once on every silent frame-drop path in the process callback
- node.dont-reconnect on our capture streams: an orphaned stream re-targeted by wireplumber
onto a fresh node wedges it — and a stuck link head-blocks the daemon's shared work queue,
stalling ALL new link negotiation system-wide (this poisoned whole test sessions)
- LUMEN_GAMESCOPE_NODE (attach to an existing gamescope) + LUMEN_PW_FIXED_POD (negotiation
bisection) debug knobs
KWin path regression-tested (zero-copy intact). gamescope end-to-end validation pending the
patched gamescope build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
a339a0466e |
feat: M2 — management REST API with OpenAPI doc (control-pane groundwork)
A versioned control-plane REST API (/api/v1) on its own port (default 127.0.0.1:47990) serving host info, runtime status, paired-client management, the pairing PIN flow, and session control (stop / force-IDR). The OpenAPI 3.1 document is generated from the handlers by utoipa, served live at /api/v1/openapi.json (+ Scalar docs at /api/docs), printable via `lumen-host openapi`, and checked in at docs/api/openapi.json for client codegen — a test fails if it drifts, mirroring the cbindgen header rule. Auth: optional bearer token (--mgmt-token / LUMEN_MGMT_TOKEN), enforced on everything but /health, and mandatory for non-loopback binds. PinGate gains a waiter count so the API can report pin_pending; logs moved to stderr so stdout stays machine-readable. Supersedes the web.rs stub. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
669d40ae21 |
build: migrate to ffmpeg-next 8 (FFmpeg 8.x / libavcodec 62)
Ubuntu 26.04 ships FFmpeg 8.0 (libavcodec 62); bump ffmpeg-next 7.1 -> 8.1 to bind it as the intended pairing. No source changes needed — the encode API surface we use (avcodec_send_frame, hwframe contexts, AV_PIX_FMT_CUDA, av_log) is stable across 7->8. Workspace builds + all tests green; clippy/fmt clean. Refresh the 7.x doc references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
7d08e43c16 |
feat: M2 — KWin virtual-output backend behind a VirtualDisplay trait (native client resolution)
Honor the client's requested resolution by rendering a compositor virtual output at
exactly that size — native, headless, no scaling. There is no cross-compositor Wayland
protocol for this, so it's a per-compositor backend behind the (previously stubbed)
VirtualDisplay trait.
- vdisplay.rs: VirtualDisplay::create(mode) now returns a live VirtualOutput
{ node_id, remote_fd: Option<OwnedFd>, keepalive } with RAII teardown (drop releases
the output) instead of an inert OutputHandle + explicit destroy. Add compositor
detect() (LUMEN_COMPOSITOR / XDG_CURRENT_DESKTOP).
- vdisplay/kwin.rs: the KWin backend — the zkde_screencast_unstable_v1 stream_virtual_output
client (vendored protocol XML + wayland-scanner codegen). Creates a WxH output, returns
its PipeWire node (default daemon, remote_fd=None); a keepalive thread holds the Wayland
connection until dropped. (Moved here from capture/kwin.rs — it's a vdisplay backend, not
capture.)
- capture: generalize the PipeWire consumer to Option<OwnedFd> (portal remote vs. default
daemon) and add capture_virtual_output(vout), compositor-agnostic, owning the keepalive.
- gamestream/stream.rs: LUMEN_VIDEO_SOURCE=virtual creates a virtual display sized to the
client's cfg and captures it (self-contained, not pooled — a reconnect at a new
resolution gets a fresh output).
- m0: --source kwin-virtual goes through the trait.
Verified end-to-end against the running headless KWin: the request reaches the compositor
and is handled cleanly. Native creation needs a backend implementing createVirtualOutput —
the DRM backend, or the VirtualBackend since KWin 6.5.6; on this box's --virtual 6.4.5 it
returns "Could not find output" (expected; validates after the KWin upgrade). wlroots/Mutter
backends are the next ones to land on the same seam.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
16a00563a8 |
feat: M2 zero-copy foundation — EGL→CUDA import + NVENC CUDA-frame path
Scaffolding for dmabuf zero-copy (plan §9), opt-in via LUMEN_ZEROCOPY:
- src/zerocopy/{cuda,egl}.rs: hand-rolled CUDA Driver-API FFI (no Rust crate
exposes the EGL-interop calls / CUeglFrame) with a shared process-wide
CUcontext + pitched device buffers; an EGL importer (GBM platform on the
NVIDIA render node) that turns a dmabuf into an EGLImage, registers it with
CUDA, and copies it device-to-device into an owned buffer. `zerocopy-probe`
subcommand validates the FFI/linking/GPU access — confirmed on the box
(driver 595, EGL_EXT_image_dma_buf_import + modifiers).
- CapturedFrame gains a FramePayload enum (Cpu(Vec<u8>) | Cuda(DeviceBuffer));
the encoder branches: CPU keeps the expand+upload path, CUDA wraps the device
buffer in an AV_PIX_FMT_CUDA frame fed straight to hevc_nvenc (sharing our
CUcontext via a hand-declared AVCUDADeviceContext, since ffmpeg-sys doesn't
bind hwcontext_cuda.h). open_video/the encoder take a `cuda` flag derived from
the first frame's payload.
The capture-side dmabuf negotiation (which produces the Cuda frames) is the
next step; the CPU path is unchanged and remains the default + fallback. Builds
clean, clippy clean, tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
03a6a67354 |
feat: M2 P1.7 — libei input backend (portable to KWin/GNOME)
Add a second input-injection backend that works on compositors implementing
the org.freedesktop.portal.RemoteDesktop interface (KWin, GNOME/Mutter), where
the wlroots virtual-input protocols are absent. Uses ashpd 0.13 to open a
RemoteDesktop session + EIS fd and reis 0.6.1 to drive it as an EI sender:
bind pointer/keyboard/scroll/button capabilities and, per device,
start_emulating → emit → frame. Runs on a dedicated thread with its own tokio
runtime (the portal session + EIS connection must stay alive and the event
stream must be polled continuously); open() returns immediately so a slow or
denied portal can never freeze the ENet control thread, with events enqueued
over an unbounded channel until devices resume.
Backend now auto-selects per session (inject::default_backend): wlr on Sway,
libei on KDE/GNOME; LUMEN_INPUT_BACKEND overrides. Refactor inject.rs into the
inject/{wlr,libei}.rs layout matching the capture/encode convention. Keyboard
codes are evdev (the same space our VK→evdev table produces) and the compositor
supplies the keymap, so no keymap upload and no modifier serialization — pressing
the modifier keys Moonlight sends is enough.
Add a `lumen-host input-test` subcommand that injects a scripted mouse+keyboard
pattern through the session backend, so input injection can be validated without
a Moonlight client.
Live-validated on headless KWin (Plasma 6.4): mouse motion, left click, and the
'A' key inject correctly and are delivered to the focused client.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
72f8c05aa3 |
feat: M2 P1.5 (FEC) — nanors-exact Reed-Solomon recovery for the video stream
Moonlight now reconstructs lost video shards from our parity (verified live: under induced packet loss the picture recovers cleanly instead of failing with "network connection too bad"; 0% added loss in normal operation). The decisive finding: Moonlight's nanors uses a CAUCHY generator matrix (M[j][i] = inv[(m+i)^j], GF(2^8) poly 0x1d), while reed-solomon-erasure is Vandermonde — so its parity was NOT Moonlight-decodable, despite the old gf8.rs comment claiming equivalence. lumen-core: - Swap the GF(2^8) backend from reed-solomon-erasure to a vendored fec-rs (vendor/fec-rs, BSD-2), which builds the byte-identical Cauchy matrix. Pure Rust, no FFI — keeps the "one core" hot path. This makes both lumen's own protocol and the GameStream parity nanors-compatible. - Lock it with a regression test against real nanors vectors (k=4,m=2 [10,20,30,40] -> parity [136,0]) + an independent matrix-derived cross-check + an erase/recover round-trip. Existing FEC/loopback tests stay green, so lumen's own protocol is unaffected. lumen-host video.rs: - Generate m = ceil(k*pct/100) parity shards per FEC block via Gf8Coder; stamp fecInfo with the recomputed wire pct (100*m/k) so the client derives the same count; cap per-block data to 255*100/(100+pct) so k+m <= 255. - CRITICAL byte-exactness: RS runs over the whole `blocksize` shard (Moonlight decodes packetSize+16 bytes from the datagram start and PACKET_RECOVERY_FAILUREs on a bad reconstructed `flags` byte). So the NV header fields RS must reproduce (streamPacketIndex/frameIndex/flags/multiFec*) are written into data shards BEFORE encode, and only the transport fields (RTP header/seq/timestamp + fecInfo) are stamped AFTER — leaving the flags byte RS-covered. Matches Sunshine stream.cpp. Unit-tested incl. flags recovery. - fec_percentage wired from stream.rs (Sunshine default 20, LUMEN_FEC_PCT override; 0 = data-only). LUMEN_VIDEO_DROP injects loss to test recovery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
278a6330de |
feat: M2 P1.6 — audio (Opus + AES-CBC) and steady-rate video pacing
A stock Moonlight client now gets video + full input + AUDIO from the from-scratch GameStream host (verified live end-to-end on a macOS client). Audio (audio.rs, audio/linux.rs, gamestream/audio.rs): - Capture the default PipeWire sink's monitor (system output) as interleaved f32 stereo @ 48kHz via stream.capture.sink, on its own thread. - Opus-encode 5ms/240-sample stereo frames (RESTRICTED_LOWDELAY, CBR) and send as GameStream RTP audio: 12-byte BE RTP_PACKET (packetType 97, seq+1/pkt, timestamp += packetDuration, ssrc 0) on UDP 48000, after learning the client endpoint from its port-learning ping. - Encrypt the Opus payload with AES-128-CBC (PKCS7), key = launch rikey, IV = BE32(rikeyid + seq) in [0..4]. Like the control stream, modern Moonlight always decrypts audio regardless of the negotiated flags — plaintext makes it log "Failed to decrypt audio packet" and play silence (diagnosed from the client log). RTP header stays in the clear. Scheme cross-checked against Sunshine stream.cpp/crypto.cpp + moonlight AudioStream.c. - Pace each frame to its 5ms slot (PipeWire delivers ~1024-frame buffers) to avoid bursts the client's jitter buffer hears as glitches. LUMEN_AUDIO_GAIN applies optional linear gain for quiet sources. - DESCRIBE SDP advertises the stereo Opus config (a=fmtp:97 surround-params). Video (stream.rs): pace at a steady ≤60fps, re-encoding the last captured frame when the compositor produces none. wlroots only emits on damage, so a static or slow-updating desktop previously starved the client into a "network too slow" abort; an unchanged frame costs a near-empty P-frame. Adds a non-blocking Capturer::try_latest (portal drains to the freshest queued frame). Misc: serialize pipewire init across the video + audio capture threads (pwinit.rs, std::sync::Once) to avoid a concurrent pw_init race. Deps: opus, cbc; libopus-dev in bootstrap-ubuntu.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
4c2c41acba |
feat: M2 P1.4 — control-stream decryption + input injection (mouse/keyboard live)
A stock Moonlight client can now drive the headless Sway desktop: mouse
movement, buttons, scroll, and keyboard all inject through the streamed
session (verified live end-to-end — typing, clicking, window management).
Control stream (gamestream/control.rs):
- Moonlight encrypts the ENet control stream with AES-128-GCM even though we
negotiate no media encryption (it detects our Sunshine `state` and turns it
on). Decrypt per-packet under the /launch `rikey`.
- The exact GCM scheme is auto-detected on the first authenticating packet
(nonce construction × key byte-order × tag position × AAD) since GCM gives no
partial credit. Our client uses the legacy 16-byte nonce (`iv[0]=seq&0xff`)
because we advertise no encryption; the 12-byte SS_ENC_CONTROL_V2 nonce is
also supported. Key/IV/tag layout cross-checked against Sunshine stream.cpp +
crypto.cpp and moonlight-common-c ControlStream.c.
Input decode (gamestream/input.rs):
- Decrypted control messages (`[u16 type][u16 len][NV_INPUT packet]`, type
0x0206) decode into lumen_core::input::InputEvent: relative/abs mouse, buttons,
vert/horiz scroll, keyboard down/up. Struct layout from moonlight Input.h
(size BE, magic LE, body BE; keyCode LE masked to the low-byte VK), dispatch
per Sunshine input.cpp (Gen5+). Unit-tested against real captured bytes.
Injection (inject.rs):
- WlrootsInjector: connects to Sway as a Wayland client and injects via the
wlroots virtual-pointer + virtual-keyboard protocols (uinput is invisible to a
compositor running WLR_LIBINPUT_NO_DEVICES=1). Uploads an evdev/US xkb keymap,
tracks modifier state, and maps Windows VK → Linux evdev (full table).
Deps: aes-gcm, wayland-client, wayland-protocols-{wlr,misc}, xkbcommon (+
libxkbcommon-dev in bootstrap-ubuntu.sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
enricobuehler
|
de60650ed3 |
feat(m2): live video to stock Moonlight — ENet control + video data plane
A stock Moonlight client now decodes H.265 from the lumen host end-to-end (verified at 5120×1440@120 on RTX 5070 Ti): - control.rs: ENet control host on UDP 47999 (rusty_enet). Moonlight starts the control stream before video (STAGE_CONTROL_STREAM_START precedes _VIDEO_), so it must be up first — this was the blocker behind the earlier "error 35". - stream.rs: video data plane — on RTSP PLAY, learn the client endpoint from its ping, NVENC-encode at the negotiated mode, packetize (GameStream RTP/NV/FEC), send over UDP 47998; stops when the client disconnects. - rtsp.rs: ANNOUNCE → StreamConfig (resolution/fps/packetSize/bitrate/codec), PLAY starts the stream, TEARDOWN stops it; PairStatus=1 over the mutual-TLS port. P1.3 uses a synthetic test pattern + data-shards-only FEC (clean-LAN). Next: real portal desktop capture, input injection (decode control → uinput), nanors-exact FEC, encryption, audio. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
ab6dda2e5f |
feat: M0 capture→encode pipeline + M2 GameStream host (pairing, RTSP, video)
M0 (lumen-host) — verified on NVIDIA RTX 5070 Ti / Ubuntu 25.10: headless wlroots → xdg ScreenCast portal → PipeWire → NVENC HEVC → playable file, with each access unit round-tripped through a lumen_core host↔client Session (FEC + packetize + reassemble), 0 mismatches. - capture.rs: SyntheticCapturer + portal capture (ashpd 0.13 + pipewire 0.9), format-aware - encode/linux.rs: NVENC via ffmpeg-next 7 (BGRx/RGB → rgb0, no host-side swscale) - m0.rs: capture→encode→file + lumen-core loopback verification M2 P1 (lumen-host gamestream/) — a stock Moonlight client pairs + launches, verified live: - mDNS _nvstream._tcp + nvhttp /serverinfo (HTTP 47989, mutual-TLS HTTPS 47984) - 4-phase pairing: PIN→AES-128-ECB / SHA-256 / RSA-PKCS1v15 / X.509, custom rustls ClientCertVerifier for the mutual-TLS pairchallenge - /applist, /launch (rikey/rikeyid/mode), hand-rolled RTSP (OPTIONS/DESCRIBE/SETUP×3/ ANNOUNCE/PLAY, one-request-per-TCP-connection per moonlight-common-c's read-to-EOF) - video.rs: GameStream RTP + NV_VIDEO_PACKET wire packetizer, data-shards-only (0% FEC, clean-LAN), unit-tested (single/multi-block) Docs: docs/m2-plan.md (phased plan) + docs/research/ (ground-truth protocol spec). Bootstrap/setup updated for the verified path (libnvidia-gl, render/video groups, GPU EGL, pipewire 0.9). Workspace clippy-clean, tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
enricobuehler
|
a913042367 |
feat: M1 lumen-core (FEC/crypto/packet/session + C ABI) and workspace scaffold
Ground-up low-latency streaming stack per docs/implementation-plan.md. M1 is
complete and tested; Linux host backends are cfg-gated stubs to be filled in on
real hardware (M0/M2).
lumen-core (built + tested on macOS/aarch64 — 21 tests):
- fec: ErasureCoder over GF(2^8) (reed-solomon-erasure, Moonlight-compatible)
and GF(2^16) Leopard-RS (reed-solomon-simd, the >1 Gbps wall-breaker); proptested
- packet: zero-copy #[repr(C)] framing, multi-block, FEC-aware reassembly
- crypto: AES-128-GCM with per-direction nonce salts + sequence-as-AAD
- session: host submit / client poll hot paths + input; loopback & UDP transports
- abi: opaque handles, versioned LumenConfig, panic guards; cbindgen-generated header
- acceptance: Rust loopback+proptest and a C harness that links the staticlib
Scaffold (compiles green on all platforms): lumen-host (vdisplay/capture/encode/
inject/web/pipeline seams under cfg(linux)), lumen-client-rs, tools/{loss-harness,
latency-probe}, Apple/Android client stubs, Gitea CI, docs.
Hardened against a multi-agent adversarial review (13 verified findings fixed,
regression-tested): reassembler memory-DoS bounds + block-consistency validation,
GCM nonce-reuse direction separation, ABI struct_size guard + range checks, FEC
shard-length guards, shard_payload datagram bound, key zeroization + Debug redaction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|