Headless KDE startup was a chain of timing-sensitive handoffs gated by a blind `sleep 2`,
the dominant source of black screens. Phase-1 fixes:
- New `punktfunk-host probe-compositor` subcommand: exits 0 iff the detected compositor is
up AND ready to create a virtual output now. KWin gets a real check (connect + registry
roundtrip + the privileged zkde_screencast global must be advertised — what the backend
needs); gamescope/Mutter/wlroots create on demand so the probe just confirms Linux.
(vdisplay::probe dispatcher + kwin::probe; reuses kwin.rs's existing roundtrip path.)
- run-headless-kde.sh: replace `sleep 2` with an active readiness wait (poll probe-compositor
until ready, 30s deadline, and bail with kwin's log if kwin_wayland exits during init).
Move the portal restart to AFTER readiness, and precede it with `systemctl --user
import-environment` + `dbus-update-activation-environment` (the missing env import — the
Sway script does this; without it a restarted portal inherits a stale/empty WAYLAND_DISPLAY,
which is the "streams but eats no input/audio" failure). kwin's stderr → a log file.
Validated: probe-compositor exits 0 "Kwin ready" against the live session, exit 1 with a
clear diagnostic when the compositor is absent. 114 tests green, clippy/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Research-grounded sequence + per-goal approach/effort. Decisions: start with KDE startup
reliability; Bazzite via COPR RPM then bootc image; commit to full UHID DualSense.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multi-agent security review of 9856c04 (4 dimensions, 2-skeptic verification):
- CRITICAL functional+security: the session cookie inherited h3's Secure=true default;
browsers DROP Secure cookies over plain http://, so login silently failed on a LAN HTTP
client (worked only on localhost, a secure context — which is why the live test passed).
Now set the cookie attributes explicitly: HttpOnly + SameSite=Lax + Path=/, and Secure
only when PUNKTFUNK_UI_SECURE=1 (behind TLS). Verified: Set-Cookie no longer has Secure.
- Gate bypass: isPublicPath allowlisted any path ending in .json/.css/.png/etc., so
/api/v1/openapi.json (served unauthenticated on the mgmt side too) leaked the whole API
schema through the token-injecting proxy. Now /api is ALWAYS gated and the generic
extension allowlist is gone (client assets are all under /assets/, still allowlisted).
Verified: /api/v1/openapi.json and /api/v1/status.json → 401.
- Session lifetime: added maxAge (7d) — bounds a stolen cookie (cookie Max-Age + iron seal
TTL); previously never expired.
- Open redirect: the post-login `next` accepted protocol-relative `//evil.com`. Hardened
client + added safeNextPath() (same-origin path only).
Re-validated end to end: login assets public (200), /api/openapi.json gated (401), authed
/api/v1/status (200), unauth /→302. tsc + build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single-user, LAN-reachable-but-gated. The web server is a backend-for-frontend:
- Login: POST /_auth/login {password} checks PUNKTFUNK_UI_PASSWORD (constant-time) and
sets a SEALED session cookie (h3 useSession / AES-GCM). server/middleware/auth.ts gates
every request — pages 302 → /login, /api → 401 — and FAILS CLOSED (503) when
PUNKTFUNK_UI_PASSWORD is unset, so a misconfigured LAN-exposed server admits no one.
- The management API stays loopback-only + token (never LAN-exposed). The proxy
(server/routes/api/[...].ts) injects PUNKTFUNK_MGMT_TOKEN server-side and drops the
browser's cookie before forwarding — the token never reaches the browser, which only
holds the session cookie.
Nitro doesn't auto-scan a server/ dir, so the Nitro plugin gets an explicit scanDirs to
pick up middleware + routes. Client: removed the localStorage token (server injects it);
the fetcher bounces to /login on 401; new /login page (bare, no shell); Settings drops the
token field and gains a Sign-out button; en/de strings.
Validated live end to end: unauth /→302, /api→401; wrong pw→401; right pw→200+cookie;
authed /api/v1/status→200 (proxied, mgmt token injected — the host required it); logout→
session cleared→401. tsc + build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The earlier "render the shell with a custom script" was a hack. The real issues were a
version matrix and a missing server target:
- TanStack Start's start-plugin-core peer-requires Vite >= 7; on Vite 6 the build's
prerender/post-build buildApp plugin hook silently doesn't run (Vite 6 lets a
config-level builder.buildApp suppress plugin buildApp hooks; Vite 7 runs both). Pinned
Vite ^7 + @vitejs/plugin-react ^5 (v5 ↔ Vite 7; v6 needs Vite 8 / vite/internal).
- Added @tanstack/nitro-v2-vite-plugin with the `bun` preset — the server/deploy target.
`bun run build` → .output/ (bun-runnable server + .output/public). `bun run start` =
`bun run .output/server/index.mjs`.
- Full SSR instead of SPA mode: SPA-shell prerender points its preview server at the old
dist/server/server.js path that Nitro relocates, breaking the build. The Nitro server
renders the shell per request; React Query fetches client-side after hydration.
- Nitro routeRules proxy /api/** → PUNKTFUNK_MGMT_URL (default 127.0.0.1:47990), so the
browser stays same-origin (bearer token rides along, no CORS).
Toolchain is now Bun (package manager + runtime): bun.lock replaces pnpm-lock.yaml;
scripts/prepare/start use bun. Validated live: bun build → .output, bun server SSR-renders
the console on :3000 and proxies the API (health/host return through it). tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TanStack Start's dev server requires a React Refresh plugin; without it `/@react-refresh`
404s, the client entry 500s, and nothing hydrates (blank screen — the production build was
unaffected since rollup handles JSX there). Pinned to the v4 line: plugin-react 6 imports
`vite/internal` (Vite 7 only) and we're on Vite 6. Must sit after tanstackStart() in the
plugin list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Browser UI for the host's management REST API (mgmt.rs / docs/api/openapi.json).
Stack, exactly as specified:
- TanStack Start (Vite, SPA mode) — file-based routes, SSR shell + client hydration.
- React Query via orval codegen from the checked-in OpenAPI spec: a custom fetch mutator
(src/api/fetcher.ts) centralizes the base URL, the bearer token (Settings → localStorage),
JSON, and a throwing ApiError; the query client skips retries on 4xx. orval returns the
response body directly (includeHttpResponseReturnType:false) so a query's `.data` is the
typed payload; GET→useQuery, POST/DELETE→useMutation by method.
- shadcn/ui on Tailwind v4 (CSS-first tokens, dark-first) — button/card/badge/input/label/
table/skeleton primitives hand-authored from the canonical source.
- Paraglide i18n (en + de) with a reactive useLocale() hook and a language switcher.
Pages: dashboard (live status — video/audio/session/stream, stop-session + request-IDR,
2s polling), host (identity/codecs/ports), clients (paired list + unpair), pairing (PIN
submit, polls pin_pending), settings (API token + language).
Dev server proxies /api → 127.0.0.1:47990 (same-origin, no CORS; PUNKTFUNK_MGMT_URL to
override). Generated code (orval client, paraglide runtime, routeTree) is gitignored and
reproduced by `pnpm codegen` (prepare/pre* scripts). Validated live against `serve`: API
shapes match, dev proxy works, SSR shell renders the localized nav, build + tsc green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triaged the multi-agent review of the renegotiation + pairing + Sway + AV1/surround batch
(1 critical, 11 major/minor confirmed). Fixes:
CRITICAL — PIN pairing was offline-brute-forceable. The HMAC-of-PIN proof let an active
MITM who terminates the TOFU ceremony recover the 4-digit PIN by offline dictionary search
(all other inputs observable) and forge a correctly-bound proof. Replaced with **SPAKE2**
(balanced PAKE, `spake2` crate) + key-confirmation MACs, binding both cert fingerprints as
the SPAKE2 identities: an attacker gets exactly ONE online guess, no offline search, and
mismatched cert views (a real MITM) never reach a shared key. Also reworked the UX to an
"arming PIN" — one PIN per arming window shown at host startup (the SPAKE2 client needs the
PIN to build its first message, so it can't be minted per-connection). Validated live:
wrong PIN rejected in 0.1s, right PIN pairs + persists + the paired identity streams.
Pairing hardening: `--allow-pairing`/`--require-pairing` must arm pairing (default rejects
unsolicited ceremonies); per-host cooldown bounds online guessing; the client flushes its
CONNECTION_CLOSE so a refused ceremony can't wedge the sequential host for the full timeout;
atomic (temp+rename) paired-store writes.
Protocol: control/pairing messages use a distinct CTL_MAGIC (PKFc) — fully disjoint from
the positional Hello namespace (a future abi_version can't be misparsed as a control
message); all typed decodes are length-exact. ABI_VERSION → 2 (punktfunk_connect signature
gained the identity params; header regenerated).
Renegotiation: drain the reconfig channel to the NEWEST mode (one rebuild, not one per
stale step); validate refresh_hz; build the new pipeline BEFORE dropping the old so a
rebuild failure keeps the session on its current mode instead of killing it.
GameStream: packetDuration snaps to {5,10} (an in-between value isn't a legal Opus frame
size and would kill audio). Sway: chooser file moved to $XDG_RUNTIME_DIR (was a fixed
world-writable /tmp path — DoS / capture-misdirection by another local user).
Swift: fixed two compile breakers in the new pairing/identity APIs (Int32 status .rawValue,
UInt cap cast). New SPAKE2 + namespace-disjointness + pairing-roundtrip unit tests; the
in-process pairing test now also exercises the arming PIN + cooldown. 114 tests green,
clippy -D warnings clean (both feature sets), fmt, C-ABI harness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renegotiation (no reconnect on resize): the handshake bi-stream stays open; the client
sends Reconfigure{mode} (typed post-handshake message), the host validates + acks
Reconfigured and rebuilds capture/encoder/virtual output at the new mode while the data
plane (keys, ports, FEC) runs untouched — the first new-mode AU is an IDR with in-band
parameter sets. NativeClient::request_mode / punktfunk_connection_request_mode; mode()
reflects the active mode. Validated live on KWin: one continuous stream, 225 frames
@1280x720 then 395 @1920x1080, ~90 ms pipeline rebuild (ffprobe shows both resolutions).
PIN pairing (mutual trust, kills TOFU MITM): clients get persistent self-signed
identities presented via QUIC client auth (generate_identity / client auth offered but
optional server-side — legacy clients still connect). Ceremony on the control stream:
PairRequest{name} → host shows a 4-digit PIN (log) + PairChallenge{salt} → client proves
with HMAC-SHA256(PIN‖salt, client_fp‖host_fp) — binding both certs means a MITM can't
forward a proof, single attempt per PIN, constant-time compare → PairResult; host
persists the fingerprint (~/.config/punktfunk/punktfunk1-paired.json), client pins the
host's. m3-host --require-pairing gates sessions on the paired set.
NativeClient::pair + punktfunk_pair/punktfunk_generate_identity in the ABI; reference
client: --pair PIN --name LABEL + auto-generated persistent identity, --remode for live
renegotiation testing. Swift wrapper: ClientIdentity/generateIdentity()/pair(),
requestMode()/currentMode(); README handoff updated.
Tested: reconfigure/pairing wire roundtrips, C-ABI mode switch ack, full in-process
ceremony (wrong PIN → Crypto, anonymous-vs-gate rejection, success → pinned session);
live wrong-PIN ceremony against the serving host (PIN logged, proof rejected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codec negotiation (M2 polish):
- ServerCodecModeSupport now advertises what we encode: H264|HEVC|AV1_MAIN8
= 65793 (flags verified against moonlight-common-c Limelight.h). The old
placeholder 3843 wrongly claimed HEVC Main10 + 4:4:4 and no AV1. Main10
bits stay off on purpose: Moonlight ties 10-bit to HDR, and capture is
8-bit SDR BGRx with no HDR metadata path (av1_nvenc -highbitdepth was
validated working for later).
- RTSP ANNOUNCE: bitStreamFormat 0/1/2 -> H264/HEVC/AV1 (already plumbed to
av1_nvenc; validated e2e via `m0 --codec av1` + ffprobe av01), and a
dynamicRangeMode!=0 request now logs + falls back to 8-bit SDR.
Surround audio (M2 polish):
- ANNOUNCE x-nv-audio.surround.{numChannels,AudioQuality} +
x-nv-aqos.packetDuration -> per-session AudioParams; DESCRIBE advertises
all six Opus configs (normal before HQ per channel count). Normal-quality
mappings are pre-rotated for the client's GFE-order LFE swap
(RtspConnection.c, verified verbatim) so its derived decoder mapping
equals our encoder mapping — including 7.1, where Sunshine's rotate only
covers [3,6) and scrambles LFE/SL/SR.
- 5.1/7.1 encode via libopus multistream (audiopus_sys, the sys layer the
opus crate already links) with Sunshine's layouts/bitrates, RAII wrapper;
the live-validated stereo wire is byte-identical (plain Opus, no FEC).
- Surround sessions add Sunshine-style RS(4,2) audio FEC (packetType 127 +
AUDIO_FEC_HEADER, the OpenFEC parity matrix both ends hardcode, nanors
gemm semantics verified from nanors/rs.c).
- PipeWire capture generalized to the negotiated channel count with explicit
FL FR FC LFE RL RR [SL SR] positions; missing sink channels are zero-
filled by the channel-mixer. PwAudioCapturer now tears down cleanly on
Drop (pipewire channel -> loop quit), so a channel-count change can
reopen without leaking a capture stream.
Tests: serverinfo mask, RTSP codec/audio param parsing, DESCRIBE contents,
surround-params strings + client-swap round trip, FEC parity self-recovery
and packet layout, real-codec 5.1 channel-identity round trip, and an
ignored live test (ran green against a 6ch null sink monitor).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fourth VirtualDisplay backend: `swaymsg create_output` adds a HEADLESS-N
output (name found by diffing get_outputs), `output <NAME> mode --custom
WxH@HzHz` sets the client's exact mode (and the refresh clock a fresh headless
output needs to produce frames at all), and the PipeWire node comes from the
ScreenCast portal. Headless output selection is non-interactive via
xdg-desktop-portal-wlr's chooser hook: a managed config (chooser_type=simple,
chooser_cmd cats /tmp/punktfunk-xdpw-output; portal try-restarted when the
config changes) plus a per-session `Monitor: <NAME>` written to that file.
Teardown is RAII: drop ends the portal thread (zbus connection drop ends the
cast) then `swaymsg output <NAME> unplug`. swaymsg commands go after `--` so
tokens like `--custom` reach sway instead of swaymsg's getopt.
Validated live on headless sway 1.11 (gles2-on-NVIDIA, xdpw 0.8.1), zero-copy
dmabuf→CUDA on both runs: 720p60 257 frames p50 0.77 ms, 1080p60 480/480
frames p50 1.18 ms, output unplugged with the session both times. The
checked-in xdpw.config sample now matches the managed config (the old
chooser_type=none/HEADLESS-1 form would pin capture to the wrong output).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The awaiting-trust and streaming phases rendered StreamView in different switch
branches, so confirming trust dismantled and recreated the NSView — the fresh pump had
already missed the opening IDR (infinite GOP: no other keyframe ever comes) and decoded
nothing. One session branch now hosts a single StreamView; the trust card is an overlay
on the blurred stream and only the capturesCursor flag flips on confirmation.
Verified live against the box (gamescope+vkcube at 720p60, 11.7 Mb/s on glass). Note for
host runs: without PUNKTFUNK_COMPOSITOR=gamescope + PUNKTFUNK_GAMESCOPE_APP, m3-host
auto-picks KWin and streams its (black, empty) session — looks identical to a client
bug but isn't one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The portal processes bind to the compositor that existed when they started; after a kwin
restart the stale instances point at a dead socket and RemoteDesktop/EIS input injection
times out ("EIS setup timed out"). Hit live: a fresh session streamed fine but ate no
mouse/keyboard until the portals were restarted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The app grows from a dev connect form into a real client shell:
- Home is a grid of saved hosts (UserDefaults-persisted; context menu: Remove / Forget
Identity), "+" in the toolbar opens the add-host sheet, the stream mode moved into
Settings (⌘, / gear) — native resolution stays the only mode, no scaling.
- Trust is now explicit: the protocol always supported certificate pinning, but the app
passed no pin and discarded the observed fingerprint — silently trusting any host.
First connect now shows the host's SHA-256 fingerprint (compare with the "clients pin
this fingerprint" line in the host log) over the live-but-blurred stream; the stream
must pump immediately (the opening IDR is the only guaranteed one), so StreamView gains
a capturesCursor switch to keep the cursor free while the prompt needs clicking, and
input capture starts only after confirmation. Trusting pins the fingerprint per host;
a changed host identity then refuses to connect.
- PUNKTFUNK_AUTOCONNECT keeps working (auto-trusts, doesn't touch the saved hosts).
Host→client authorization (pairing PIN) remains a punktfunk-core roadmap item — the host
still accepts any client that can reach its port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
punktfunk_Logo.icon (Icon Composer 2.0) in App/, ASSETCATALOG_COMPILER_APPICON_NAME set.
Compiles with Xcode 27 beta's actool; Xcode 26.5's actool crashes on EVERY .icon file
(known regression, Apple FB20183399, expo/expo#46121) — build with the beta (or 26.4.1)
until a 26.x fix lands. The icon itself is fine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
clients/apple was a bare Swift package — fine for swift run/test, but app icons, a real
bundle (Info.plist, signing identity, TCC), and the normal Xcode build/run flow need an
app target. Punktfunk.xcodeproj (synchronized-folder format) wraps the SAME sources as
the CLI dev shell (Sources/PunktfunkClient) plus App/Assets.xcassets, and links
PunktfunkKit from the local package — no source duplication, both flows stay green:
swift build / swift test / swift run PunktfunkClient, and xcodebuild -scheme Punktfunk.
The asset catalog ships an empty AppIcon slot ready for the Icon Composer .icon
(drag in + set as App Icon + drop the placeholder; see README — including the actool
crash observed with the current icon bundle). Package tests on ⌘U need one GUI step
(Edit Scheme → Test → +); a hand-written package-test scheme reference doesn't resolve
headlessly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The host renders its own cursor from our raw deltas, so the local macOS cursor both
stays visible and drifts away from the remote one — and it can wander out of the window,
where a click focuses another app. While the stream has focus, do what Moonlight does:
warp the cursor mid-view, disconnect it from mouse movement
(CGAssociateMouseAndMouseCursorPosition(false) — GCMouse still delivers raw HID deltas),
and hide it. Released on app deactivation (Cmd+Tab is the escape hatch), view teardown,
and disconnect; re-captured when the stream regains focus. The HUD's Disconnect gains ⌘D
since a hidden, frozen cursor can't click it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full project rename, decided 2026-06-10:
- Crates/binaries: punktfunk-core / punktfunk-host / punktfunk-client-rs.
- C ABI: punktfunk_* symbols, Punktfunk* types, include/punktfunk_core.h,
PUNKTFUNK_FEATURE_QUIC guard (header regenerated; cbindgen renames updated, incl.
PUNKTFUNK_BTN_*/PUNKTFUNK_AXIS_* wire constants).
- Protocol: punktfunk/1 — control-plane magic LMN1 → PKF1, nonce salt lmn1 → pkf1.
WIRE BREAK: clients must be rebuilt from this revision.
- Env knobs: PUNKTFUNK_VIDEO_SOURCE / PUNKTFUNK_COMPOSITOR / PUNKTFUNK_ZEROCOPY / ….
- Host config dir: ~/.config/punktfunk (the box's dir was migrated in place — the
persistent identity is unchanged, pinned fingerprints stay valid).
- Swift package: PunktfunkKit + PunktfunkCore.xcframework + PunktfunkConnection
(Sources/PunktfunkClient app + tests renamed with it); build-xcframework.sh updated.
- scripts/: 60-punktfunk.rules, punktfunk-host.service; OpenAPI doc regenerated.
Also: scripts/headless/run-headless-kde.sh — full headless Plasma bringup. Root cause of
"desktop but no apps/settings" over the stream: plasmashell launched without
XDG_MENU_PREFIX=plasma-, so the launcher resolved a nonexistent applications.menu and
rendered an empty menu. The script sets the complete KDE session env (menu prefix,
KDE_FULL_SESSION, session version) and rebuilds ksycoca before starting plasmashell.
Gate: 97/97 tests, clippy -D warnings (both feature sets), fmt, C-ABI harness PASS,
zero lumen references left outside .git.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Mutter backend validated on headless GNOME Shell 50.1: RecordVirtual virtual monitor at
the client's mode, 360/360 frames @1080p60, p50 1.28 ms, tiled-dmabuf zero-copy. All
three VirtualDisplay backends (KWin/gamescope/Mutter) are now proven live.
- gamescope 3.16.22 system install validated (720/720 @720p120 via /usr/local/bin, the
PATH override is retired from the docs).
- Gamepads validated live over lumen/1: client datagrams -> uinput "Microsoft X-Box 360
pad" created (event9/js1), destroyed with the session (udev rule + input group active).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clients/apple scaffold is now a working macOS client, validated live against this
repo's host across the LAN: gamescope virtual output → NVENC HEVC → lumen/1 (GF(2¹⁶) FEC +
AES-GCM over UDP, QUIC control) → VideoToolbox → AVSampleBufferDisplayLayer at 720p60,
mouse/keyboard flowing back as QUIC datagrams into the host's gamescope EIS injector
(~3.7k events injected in one session).
LumenKit:
- LumenConnection: the predicted cbindgen compile fixes (C17 header spells the typedefs as
integers while the enum constants import as a distinct Swift type — bridge by rawValue);
close() is now safe from any thread (a close flag + pumpLock held across the blocking
poll enforce the C contract "never close with a next_au in flight"; flag prevents
lock-starvation by back-to-back polls).
- StreamView: per-pump cancellation token (reconnects can't double-pump), flush + re-gate
on the next in-band parameter sets when the layer fails, no stale enqueue after restart.
- InputCapture: fractional-delta accumulation (sub-pixel motion isn't truncated away),
pressed-state tracking with release-all on focus loss and stop() (nothing sticks down
host-side), global-singleton ownership guard (GC has one handler slot per process),
X1/X2 buttons, horizontal scroll, full keypad/CapsLock/ISO-102nd/PrintScreen/Menu VKs.
- LumenClient app shell (swift run LumenClient): connect form, fps/Mb-s HUD,
LUMEN_AUTOCONNECT/LUMEN_MODE for scripted first-light runs.
- Tests: Annex-B byte-level units; real-codec round trip (VTCompressionSession-encoded
HEVC rebuilt as the host's wire shape → AnnexB → VTDecompressionSession → pixels);
test-loopback.sh (Swift client vs a real local m3-host over loopback — the Swift twin of
c_abi_connection_roundtrip); RemoteFirstLightTests (full pipeline over the LAN).
Host/build fixes that fell out:
- The workspace builds on non-Linux again: gamestream audio (opus) and sendmmsg batching
are now platform-gated with stubs/fallback, per the crate's "compiles everywhere" rule.
- Horizontal scroll was inverted end-to-end: the injectors negated BOTH axes onto the
ei/wl axes, but GameStream's horizontal convention is positive = right
(moonlight-qt/Sunshine pass it through unnegated) — only vertical flips now. This also
un-inverts real Moonlight clients.
- AnnexB drops all zeros preceding a start code (trailing_zero_8bits padding), ffmpeg's
policy, instead of leaking them into the preceding NAL.
- build-xcframework.sh: deployment targets pinned to the package floor + an otool guard —
cargo does not fingerprint MACOSX_DEPLOYMENT_TARGET, so warm caches can silently ship
too-new minos objects.
Adversarially reviewed (5-dimension multi-agent pass, every finding refutation-verified):
14 confirmed findings, all fixed above; the send-while-polling core-contract gap flagged
here is closed by the lumen/1 session-planes work (&self pulls + per-plane borrow slots).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
m3-host is now a real host, not a one-shot demo. Everything validated live on this box
(two back-to-back sessions, pinned + TOFU, ~200 audio pkts/s, p50 0.84 ms at 720p60).
lumen-core:
- quic.rs: QUIC-datagram side planes demuxed by first byte — Opus audio 0xC9
([magic][u32 seq][u64 pts_ns][opus], host→client) and rumble 0xCA ([magic][pad][low][high]).
- Trust: endpoint::server_with_identity (persistent PEM identity) and
endpoint::client_pinned — SHA-256 cert-fingerprint pinning with TOFU (observed
fingerprint reported back for persisting). The verifier checks the TLS 1.3
CertificateVerify signature for real (an MITM replaying the host's public cert without
its key is rejected; cert pinning alone would not prove key possession).
- client.rs: NativeClient gains pin + host_fingerprint, audio/rumble receivers
(next_audio / next_rumble); pull methods take &self so the C ABI's per-plane threads
never alias a &mut (per-plane mutexed borrow slots in abi.rs).
- abi.rs: lumen_connect(pin_sha256, observed_sha256_out) + lumen_connection_next_audio /
next_rumble. input.rs: documented gamepad wire contract (GameStream buttonFlags bits,
XInput axis conventions, +y = up) — exported as LUMEN_BTN_*/LUMEN_AXIS_* (bare BTN_*
collides with <linux/input-event-codes.h> at different values).
lumen-host (m3):
- Persistent accept loop: sessions back to back on one endpoint (--max-sessions, 0 =
forever); per-session failures log and the loop keeps serving; 10 s handshake deadline
so a silent client can't wedge the sequential accept queue; teardown on every exit path
(stop flag → conn.close → join audio+input threads).
- Audio plane: desktop PipeWire capture → Opus 48 kHz stereo 5 ms CBR → datagrams; ONE
capturer reused across sessions via an AudioCapSlot (PipeWire streams have no cheap
teardown — per-session opens would leak a thread + core connection + live node each).
- Gamepad routing: incremental GamepadButton/GamepadAxis datagrams accumulate into
per-pad state feeding the uinput xpad manager; force feedback returns as rumble
datagrams, with current state re-sent every 500 ms (idempotent-state healing for the
lossy channel). QUIC endpoint serves the persistent ~/.config/lumen identity and logs
the pinnable fingerprint.
lumen-client-rs: --pin (malformed values abort — never silently downgrade to TOFU),
TOFU fingerprint logging, audio/rumble datagram counters, gamepad events in --input-test.
clients/apple: scaffold synced — pinSHA256/hostFingerprint (wrong-size pin throws,
fail-closed), nextAudio/nextRumble, gamepad event constructors; README handoff updated
(persistent listener, audio decode notes, trust UX).
Adversarially reviewed (5-dimension multi-agent pass over the diff, 2-skeptic
verification): fixed the MITM signature-check gap, a Y-axis contract inversion, header
macro collisions, ABI aliasing UB, the PipeWire per-session leak, the missing handshake
deadline, fail-open pin parsing, and teardown-on-error paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared-core architecture pays off: platform clients now link ONE Rust library that
does the entire lumen/1 protocol, and only add decode/present/input on top.
lumen-core:
- client.rs (quic feature): NativeClient — QUIC handshake + UDP data plane + input
datagrams on internal threads; embedder surface = connect / next_frame / send_input.
- abi.rs: lumen_connect / lumen_connection_next_au (borrow-until-next-call, matching
lumen_client_poll_frame semantics) / lumen_connection_send_input / lumen_connection_mode /
lumen_connection_close. Guarded in the generated header by LUMEN_FEATURE_QUIC (cbindgen
[defines] mapping), so the checked-in header is stable across feature sets.
- error.rs: append-only LumenStatus additions Timeout (-9) and Closed (-10).
- TESTED end-to-end through the C ABI: in-process lumen/1 host, lumen_connect pulls 25
byte-verified frames, sends input, closes (m3.rs::c_abi_connection_roundtrip).
Apple client (clients/apple — SCAFFOLD, written on Linux, first Xcode build pending):
- scripts/build-xcframework.sh: cargo per Apple target → universal staticlib + header
(LUMEN_FEATURE_QUIC pre-defined) + modulemap → LumenCore.xcframework.
- Package.swift (LumenKit) + Swift sources: LumenConnection (ABI wrapper), AnnexB
(in-band VPS/SPS/PPS → CMVideoFormatDescription, Annex-B → AVCC CMSampleBuffers with
DisplayImmediately), StreamView (SwiftUI over AVSampleBufferDisplayLayer — stage-1
presenter that hardware-decodes compressed HEVC itself), InputCapture (GCMouse raw
deltas + GCKeyboard HID→VK).
- README.md is the full handoff for the next (Mac-side) agent: build steps, ABI contract,
first-light test recipe against the Linux host, stage-2 (VT+Metal pacing) plan, and the
known host-side gaps (single-session m3-host, no lumen/1 audio yet, gamepad kinds not
yet routed in m3's injector, seed-stage trust).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolutions: serve() keeps main's AppState::new() with our persisted-pairing load folded
into it; main.rs keeps both the m3 and mgmt modules; mgmt's test LaunchSessions gain the
new appid field; Cargo.lock re-resolved. Full gate green (92 tests, clippy, fmt).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native protocol now does the real thing, end to end:
- Hello carries the client's requested mode; the host creates a NATIVE virtual output at
exactly that size/refresh (same vdisplay backends as the GameStream path) and streams
NVENC HEVC through the M1 Session (GF(2^16) Leopard FEC + AES-GCM, QUIC-negotiated).
- Input rides QUIC DATAGRAMS — encrypted, congestion-managed, no ENet retransmission
spikes — decoded into lumen_core InputEvents and fed to the session's input injector.
- Frames are stamped with the capture wall clock; the reference client computes per-frame
capture→reassembled latency percentiles and writes a playable .h265.
- m3-host gains --source synthetic|virtual + --seconds; the client gains --mode WxHxFPS,
--out, --input-test (scripted mouse/keyboard datagrams).
VALIDATED live (gamescope session, xev nested): client requested 1280x720@120 → host
created gamescope at that mode → 1680/1680 frames over 14s, zero loss, valid HEVC;
pipeline latency p50 0.83ms / p95 1.2ms / p99 1.3ms (capture→encode→FEC→crypto→UDP→
reassembled, same-host clock); 176 input datagrams sent → injector (GamescopeEi) → 164
X events observed inside the nested session.
Known follow-on: slice-level sub-frame pipelining needs the NVENC SDK directly (libavcodec
emits whole AUs only) — the next big latency lever.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first end-to-end run of lumen's own protocol, past the GameStream compatibility layer.
- lumen-core/src/quic.rs (behind the `quic` feature): the lumen/1 handshake — Hello/Welcome/
Start as length-prefixed LE binary on one QUIC bi-stream. Welcome carries the COMPLETE
data-plane Config: mode, FEC scheme incl. GF(2^16) Leopard (inexpressible in GameStream),
shard sizing, AES-GCM key + per-direction salt, data UDP port. Plus quinn endpoint helpers
(self-signed server; accepts-any client — pinning lands with the trust model) and framed
async IO. Round-trip unit-tested.
- lumen-host m3-host: serves one lumen/1 session — QUIC handshake, then a NATIVE thread
(no async on the frame path — design invariant) streams deterministic 64KB test frames
through the hardened M1 Session over UdpTransport.
- lumen-client-rs: from scaffold to working reference client — connects, negotiates, brings
up the client Session over UDP, reassembles + FEC-recovers + byte-verifies every frame.
VALIDATED END-TO-END on localhost: 300/300 frames verified, 0 mismatches, through
QUIC-negotiated GF(2^16) FEC + AES-GCM over real UDP sockets. M4 (decode+present) builds on
this exact client skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- gamestream/apps.rs: an app catalog (loaded from ~/.config/lumen/apps.json, with defaults:
Desktop + gamescope entries when gamescope/steam/vkcube are installed). /applist renders
it; /launch?appid=N selects the entry; RTSP PLAY resolves it and the stream honors the
app's compositor + nested command — so a Moonlight client picks "Steam" and gets a
gamescope session at its native resolution, or "Desktop" for the KWin/GNOME desktop.
- Persistent pairing: the paired-client cert allow-list now survives restarts
(~/.config/lumen/paired.json), saved on each successful pairing, loaded at boot.
- Quit semantics: /cancel now actually stops the media threads (streaming/audio flags),
tearing down the per-session virtual output / gamescope process via the capturer's RAII.
- scripts/lumen-host.service (systemd user unit) + scripts/host.env.example (config file
consumed by it) — the host runs as a managed service instead of an SSH shell.
Smoke-tested: serve boots, /applist serves the catalog (Desktop + vkcube gamescope entry
auto-detected on this box). GNOME backend validation still pending gnome-shell install;
wlroots vdisplay backend deliberately deferred (not in the priority compositor trio).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The missing zero-copy path is closed. NVIDIA's EGL won't sample LINEAR and the CUDA driver
rejects raw dmabuf fds — but Vulkan imports dmabufs (VK_EXT_external_memory_dma_buf) and
exports OPAQUE_FD memory that CUDA officially imports. zerocopy/vulkan.rs (ash):
dmabuf fd → VkBuffer (import cached per fd) → vkCmdCopyBuffer (GPU) →
exportable VkBuffer → vkGetMemoryFdKHR(OPAQUE_FD) → cuImportExternalMemory → CUdeviceptr
The exportable buffer + CUDA mapping are per-resolution; per frame it's one GPU buffer copy
(fence-waited) + one pitched CUDA copy into the encoder's pool. No CPU touches pixels.
EglImporter::import_linear now routes through the bridge (lazy init; any failure still falls
back to the CPU mmap path). cuda::ExternalDmabuf gained import_owned_fd for the
Vulkan-exported fd.
Validated live: gamescope 720p120 → "Vulkan→CUDA exportable staging buffer ready
size=3686400" (exactly 1280*720*4), full-rate 122.7 fps, decoded frame pixel-correct
(vkcube). KWin's tiled EGL path regression-tested intact. NV12 negotiation dropped — moot
now that BGRx is fully zero-copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stream_body no longer sends: each frame's packet batch goes over a depth-2 bounded queue to a
dedicated send thread, so a send spike can never stall capture/encode (a full queue drops the
NEWEST batch — FEC/RFI covers the client — rather than ever blocking). The sender ships packets
with sendmmsg (≤64/syscall: ~375 syscalls/s instead of ~24k at 5K@240) in 16-packet chunks
paced across ~3/4 of the frame interval — microburst shaping for real links without per-packet
sleep jitter. Client-gone detection moved to the sender (clears `running`); the LUMEN_VIDEO_DROP
FEC test knob moved with the send path. Loopback-tested: batches arrive complete and
byte-identical through the paced sendmmsg path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full controller path for the SteamOS-like session, mirroring Sunshine byte-for-byte
(wire formats verified against moonlight-common-c + Sunshine source; ioctl numbers and
struct layouts verified by compiling against this box's <linux/uinput.h>, locked in with
const asserts):
- gamestream/gamepad.rs: decode MULTI_CONTROLLER (magic 0x0C, mixed BE-size/LE-body) incl.
the Sunshine buttonFlags2 extension (paddles/touchpad/Misc — our appversion already
advertises Sunshine, so clients send it) and CONTROLLER_ARRIVAL (0x55000004); build the
0x010B rumble plaintext (with the mandatory 4-byte filler). Unit-tested.
- inject/gamepad.rs: VirtualPad clones the kernel xpad identity ("Microsoft X-Box 360 pad",
045e:028e, exact button/axis codes + absinfo) so SDL/Steam/Proton match their built-in
mapping with zero config. GamepadManager creates/destroys pads from activeGamepadMask
(hotplug), emits button transitions + axes (+Y-up → evdev +Y-down negation, D-pad as
HAT0X/Y) per frame. Rumble: non-blocking FF pump answers UI_BEGIN/END_FF_UPLOAD/ERASE
(games block in EVIOCSFF until answered), tracks effects with replay expiry + FF_GAIN,
mixes to (low, high) motor levels, dedups.
- control.rs: channel_limit 8 → 0x30 — Moonlight sends gamepad input on ENet channel
0x10+n, so the old limit silently discarded ALL controller input. Gamepad events route to
the manager; rumble is sealed with the client's detected GCM scheme direction-flipped
(V2 marker 'H?', own seq counter) and sent on the control peer every service tick.
- scripts/60-lumen.rules: udev rule (Sunshine-style) granting the input group /dev/uinput.
Live validation needs the udev rule installed (root-only /dev/uinput on this box) + a
Moonlight client with a controller; everything else is gated and unit/static-checked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Third compositor on the VirtualDisplay seam, via Mutter's direct D-Bus APIs (the
gnome-remote-desktop headless model, no portal grant): RemoteDesktop.CreateSession →
ScreenCast.CreateSession({remote-desktop-session-id}) → Session.RecordVirtual (creates a
virtual monitor) → Start → PipeWireStreamAdded(node_id). A keepalive thread owns the zbus
connection (sessions die with it — RAII teardown); select with LUMEN_COMPOSITOR=mutter or
XDG_CURRENT_DESKTOP=GNOME.
Mutter sizes its virtual monitor FROM the PipeWire format negotiation, so VirtualOutput
gains preferred_mode (w, h, refresh_hz), threaded into the consumer's format pods as the
default size/framerate. KWin/gamescope set it too (their outputs are already exact-size;
the preference just confirms it) — both regression-tested intact.
Compile/clippy/test clean; live validation needs gnome-shell installed (then
`gnome-shell --headless` + m0 --source kwin-virtual with LUMEN_COMPOSITOR=mutter).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gamescope only offers LINEAR dmabufs, which the EGL/GL interop path can't handle (NVIDIA's
EGL lists no LINEAR modifier for sampling). Attempt a direct CUDA external-memory import
(cuImportExternalMemory OPAQUE_FD, cached per buffer fd, one DtoD copy per frame into the
pooled buffer): the FFI + plumbing are in place, and LINEAR(0) is now advertised alongside
the tiled EGL modifiers (tiled first, so KWin still prefers it — regression-tested).
Empirically the 595 desktop driver rejects raw dmabuf fds as OPAQUE_FD (CUDA_ERROR_UNKNOWN),
matching the documented limitation — true LINEAR GPU import needs a Vulkan interop bridge
(import dmabuf via VK_EXT_external_memory_dma_buf, GPU-copy into an exportable allocation,
hand that to CUDA), noted as future work. So the importer now degrades instead of dying:
on GPU-import failure it logs once, disables itself, and falls through to the CPU mmap path.
Validated: gamescope + LUMEN_ZEROCOPY=1 runs full-rate (122.9 fps @720p120, valid HEVC) via
the fallback; KWin keeps real zero-copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gamescope runs its own EIS server and exports the socket to its children as LIBEI_SOCKET —
no portal involved. The gamescope backend now launches the nested app through a tiny shell
wrapper that relays that value to /tmp/lumen-gamescope-ei; the libei injector gains an
EiSource enum (Portal | SocketPathFile) and connects a UnixStream directly to gamescope's
socket (polling until the app has started), then runs the identical reis sender flow.
Backend::GamescopeEi is auto-selected when LUMEN_COMPOSITOR=gamescope
(LUMEN_INPUT_BACKEND=gamescope overrides).
Validated end-to-end: input-test against a headless gamescope running xev — 129
MotionNotify/KeyPress/ButtonPress events delivered into the nested X app ("Gamescope
Virtual Input" device bound, sender handshake + emulation working).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deep investigation (gdb + daemon traces) proved the gamescope capture stall is a gamescope
3.16.20 bug, not ours: it calls pw_loop_iterate() without pw_loop_enter()/leave(), and under
PipeWire 1.6's loop locking its main thread permanently holds the loop mutex — the pw thread
deadlocks, gamescope never acks the daemon's port_set_param(Format), and the link parks in
"negotiating" silently. Stock gst pipewiresrc fails identically. Fixed upstream by gamescope
commit e3ed1ea7 ("pipewire: Fix pipewire loop locking", pipewire#5148); first release 3.16.22.
Ubuntu 26.04 ships 3.16.20 (built ten days before the fix) — patch/upgrade required.
Consumer-side improvements from the investigation (all verified correct vs gamescope's pods,
and needed once the producer is fixed):
- discover the node from gamescope's own "stream available on node ID: N" log line (its
node.name appears on two objects; the advertised id is authoritative); pw-dump fallback
- CPU path accepts mappable dmabufs: Buffers param now offers MemPtr|MemFd|DmaBuf (gamescope
counter-offers exactly DmaBuf when its modifier pod wins, never MemPtr), mmap the fd
ourselves when MAP_BUFFERS didn't (Vulkan-exported dmabufs aren't flagged mappable), and
treat chunk.size==0 as the computed span
- warn_once on every silent frame-drop path in the process callback
- node.dont-reconnect on our capture streams: an orphaned stream re-targeted by wireplumber
onto a fresh node wedges it — and a stuck link head-blocks the daemon's shared work queue,
stalling ALL new link negotiation system-wide (this poisoned whole test sessions)
- LUMEN_GAMESCOPE_NODE (attach to an existing gamescope) + LUMEN_PW_FIXED_POD (negotiation
bisection) debug knobs
KWin path regression-tested (zero-copy intact). gamescope end-to-end validation pending the
patched gamescope build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Five confirmed findings from a 46-agent review panel:
- Empty --mgmt-token no longer satisfies the non-loopback token gate
(critical: 'Bearer ' with an empty token authenticated; parse_serve now
bails on blank tokens and mgmt::run treats blank as none)
- axum's built-in body rejections (400/415/422) now wear the documented
ApiError envelope via an ApiJson extractor, and the spec documents them
- GET /health carries security([{}]) in the spec, matching the server's
auth exemption
- unpairClient's description no longer claims revocation the TLS layer
doesn't enforce yet (gamestream/tls.rs accepts any cert — known gap)
- CLAUDE.md/README.md no longer reference the deleted web.rs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A versioned control-plane REST API (/api/v1) on its own port (default
127.0.0.1:47990) serving host info, runtime status, paired-client
management, the pairing PIN flow, and session control (stop / force-IDR).
The OpenAPI 3.1 document is generated from the handlers by utoipa, served
live at /api/v1/openapi.json (+ Scalar docs at /api/docs), printable via
`lumen-host openapi`, and checked in at docs/api/openapi.json for client
codegen — a test fails if it drifts, mirroring the cbindgen header rule.
Auth: optional bearer token (--mgmt-token / LUMEN_MGMT_TOKEN), enforced on
everything but /health, and mandatory for non-loopback binds. PinGate
gains a waiter count so the API can report pin_pending; logs moved to
stderr so stdout stays machine-readable. Supersedes the web.rs stub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Third compositor on the VirtualDisplay seam. gamescope's model differs from KWin/Mutter: it's
not a runtime protocol but a micro-compositor we spawn — `gamescope --backend headless -W -H -r
-- <app>` — which composites at the client's size AND refresh natively (so no separate
refresh step), runs the app nested, and exports a built-in PipeWire node named "gamescope".
The backend spawns it, discovers that node via pw-dump, and returns a VirtualOutput whose
keepalive owns the process (drop = kill = teardown). App via LUMEN_GAMESCOPE_APP. Select with
LUMEN_COMPOSITOR=gamescope; m0's virtual source now honors LUMEN_COMPOSITOR so any backend is
testable without a client. Input (gamescope's libei/EIS socket) is a follow-up.
Builds/clippy/fmt clean. Needs gamescope installed to validate; headless capture on the
proprietary NVIDIA driver is plausible-by-architecture but unproven — validate empirically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KWin creates virtual outputs at a hardcoded 60 Hz and zkde stream_virtual_output has no
refresh argument, so the *source* composited at 60 Hz even when the client asked for 120/240
(confirmed live: stream paced a stable 240 fps but only ~60 unique frames/s). KWin 6.6+ allows
custom modes on virtual outputs, so after creating the output we install + select a mode at the
client's refresh, before capture connects PipeWire. First cut shells out to kscreen-doctor
(output is "Virtual-<name>"); the in-process kde_output_management_v2 client is a follow-up.
Best-effort — failure leaves the source at 60 Hz (stream still works). Verified the mode is
applied (Virtual-lumen -> 1280x720@120). Empirically de-risked that this headless QEMU VM's
software vsync accepts >60 Hz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GB203 has two NVENC engines. A single HEVC p1 session tops out ~1 Gpix/s, so 5120x1440@240
(1.77 Gpix/s) is encoder-bound on one engine; split-frame encode runs it across both (~1.8x,
latency-neutral, output is standard HEVC the client decodes normally). NVENC's AUTO split
won't engage below ~2112px height, so force split_encode_mode=2 when the pixel rate exceeds
~1 Gpix/s (HEVC/AV1 only — not H.264). Below that (e.g. 5K@120) stay single-engine to avoid
the ~2% BD-rate cost. Override with LUMEN_SPLIT_ENCODE. Verified: engages at 240, not at 120.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The zero-copy import did real per-frame GPU churn that capped high-fps throughput: a fresh
~29MB cuMemAllocPitch + cuMemFree, a cuGraphicsGLRegisterImage/unregister, and a map of the
*same* persistent blit texture — every frame. Two fixes:
- BufferPool: a recycled free-list of pitched device buffers per resolution. DeviceBuffer
returns its allocation to the pool on drop (after the encoder synchronized) instead of
freeing — kills the per-frame 29MB alloc/free that took the device allocator lock and
serialized against the GPU.
- RegisteredTexture: register the (reused) GL_RGBA8 blit destination with CUDA ONCE when the
GlBlit is built; each frame only maps → copies the array → unmaps, instead of
registering/unregistering every frame.
This is the "zero-copy should be overhead-free" cleanup. Verified the import still produces
correct frames; the remaining per-frame cuCtxSynchronize pair (shared-context coupling) is
the next step (CUDA stream + events). lumen-host builds, clippy/fmt/tests clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
At 5120x1440 the stream froze on a ~2s cadence. Two compounding causes (confirmed by a
profiling pass + adversarial review):
1. Periodic IDR every 2s (set_gop(fps*2)). A keyframe at 5K is ~20-40x a P-frame — a
recurring multi-millisecond encode+packetize+send spike. Fix: infinite GOP (gop_size=-1),
one IDR at stream start, P-frames only; forced-idr makes a client recovery request (RFI via
request_keyframe) emit an IDR on demand — the Moonlight/Sunshine low-latency model.
2. Two pacing timers summing on the capture/encode thread: a per-packet thread::sleep pacer
(spread a frame's packets across a whole frame interval) PLUS a backstop sleep on top, so
every frame cost 1-2x the interval and the big IDR blew through it (the 2->120 oscillation).
Fix: delete both; send at line rate and drive cadence from a single absolute deadline.
(Proper microburst pacing belongs on a dedicated send thread — a follow-up.)
Also: honor the client's fps (pacing clamp 60->240) and add an env-gated (LUMEN_PERF)
per-stage timing log (enc/pkt/send µs + unique-vs-reencoded frames + max packet burst) for
diagnosing the remaining throughput ceiling. Verified live: freeze gone at 5120x1440.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ubuntu 26.04 ships FFmpeg 8.0 (libavcodec 62); bump ffmpeg-next 7.1 -> 8.1 to bind it
as the intended pairing. No source changes needed — the encode API surface we use
(avcodec_send_frame, hwframe contexts, AV_PIX_FMT_CUDA, av_log) is stable across 7->8.
Workspace builds + all tests green; clippy/fmt clean. Refresh the 7.x doc references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Honor the client's requested resolution by rendering a compositor virtual output at
exactly that size — native, headless, no scaling. There is no cross-compositor Wayland
protocol for this, so it's a per-compositor backend behind the (previously stubbed)
VirtualDisplay trait.
- vdisplay.rs: VirtualDisplay::create(mode) now returns a live VirtualOutput
{ node_id, remote_fd: Option<OwnedFd>, keepalive } with RAII teardown (drop releases
the output) instead of an inert OutputHandle + explicit destroy. Add compositor
detect() (LUMEN_COMPOSITOR / XDG_CURRENT_DESKTOP).
- vdisplay/kwin.rs: the KWin backend — the zkde_screencast_unstable_v1 stream_virtual_output
client (vendored protocol XML + wayland-scanner codegen). Creates a WxH output, returns
its PipeWire node (default daemon, remote_fd=None); a keepalive thread holds the Wayland
connection until dropped. (Moved here from capture/kwin.rs — it's a vdisplay backend, not
capture.)
- capture: generalize the PipeWire consumer to Option<OwnedFd> (portal remote vs. default
daemon) and add capture_virtual_output(vout), compositor-agnostic, owning the keepalive.
- gamestream/stream.rs: LUMEN_VIDEO_SOURCE=virtual creates a virtual display sized to the
client's cfg and captures it (self-contained, not pooled — a reconnect at a new
resolution gets a fresh output).
- m0: --source kwin-virtual goes through the trait.
Verified end-to-end against the running headless KWin: the request reaches the compositor
and is handled cleanly. Native creation needs a backend implementing createVirtualOutput —
the DRM backend, or the VirtualBackend since KWin 6.5.6; on this box's --virtual 6.4.5 it
returns "Could not find output" (expected; validates after the KWin upgrade). wlroots/Mutter
backends are the next ones to land on the same seam.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clients pick the resolution via mode=WxHxFPS / RTSP clientViewportWd-Ht, so the
host must bound attacker/typo-controlled dimensions before allocating buffers or
opening NVENC. Add encode::validate_dimensions: reject zero, odd, and over-limit
modes (H.264 ≤ 4096px/side; HEVC/AV1 ≤ 8192) with a clear message instead of a
buffer-math overflow or an opaque NVENC open failure. Gate both the stream path
(before any allocation) and open_video (also covers m0). Unit-tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PipeWire dmabuf now reaches NVENC with no CPU touch. Verified live against
headless KWin: a tiled BGRx dmabuf is imported and encoded to a pixel-correct
H.265 stream (decoded frame matches the captured desktop — no tiling artifacts,
no colour swap). The CPU-copy path stays the default and the runtime fallback.
Capture side (zerocopy::egl): desktop NVIDIA can't register a dmabuf EGLImage
with CUDA directly (cuGraphicsEGLRegisterImage is Tegra-only; cuGraphicsGLRegisterImage
rejects EGLImage-backed textures), so we follow OBS/Sunshine — bind the EGLImage
to a GL texture, render it through a fullscreen-triangle shader into an immutable
GL_RGBA8 texture (de-tiling + .bgra swizzle to the BGRx the encoder wants), then
register that texture with CUDA and copy it device-to-device into an owned buffer
so the dmabuf returns to the compositor immediately.
Encode side (encode/linux::submit_cuda): take a *pooled* CUDA surface via
av_hwframe_get_buffer and device→device-copy our imported buffer into it, instead
of wrapping our own pointer in a bare AVFrame. A bare frame is rejected with
EINVAL (NVENC ignores frames with null buf[0]; the encode path's av_frame_ref
needs a refcounted buffer), and a fresh device pointer every frame would thrash
NVENC's bounded resource-registration cache — the pool recycles a small set.
Also: gate FFmpeg AV_LOG_DEBUG behind LUMEN_FFMPEG_DEBUG for diagnosing
hw-frame rejects, and refresh the now-accurate module docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the capture side of zero-copy (LUMEN_ZEROCOPY=1):
- EGL importer now opens the headless EGLDisplay on the NVIDIA EGL device
(EGL_PLATFORM_DEVICE_EXT) and queries its importable DRM modifiers
(eglQueryDmaBufModifiersEXT).
- The PipeWire stream advertises a BGRx dmabuf format with those modifiers as a
mandatory enum Choice + a dmabuf-only Buffers param; the compositor fixates an
importable tiled modifier. param_changed reads the negotiated modifier; the
process callback imports the dmabuf (eglCreateImage with explicit LO/HI
modifier) and would copy it into a CUDA buffer for the encoder.
Validated against headless KWin (Plasma 6.4): negotiation succeeds (13 NVIDIA
modifiers advertised, KWin fixates one, stream reaches Streaming with a real
tiled dmabuf) and `eglCreateImage` succeeds. The remaining blocker is
`cuGraphicsEGLRegisterImage` returning CUDA_ERROR_INVALID_VALUE on the
dmabuf-imported EGLImage — the likely fix is to bind the EGLImage to a GL
texture (glEGLImageTargetTexture2DOES) and register that via
cuGraphicsGLRegisterImage (OBS/Sunshine's path), which needs a GL context.
The CPU-copy path stays the default and is unaffected (regression-checked: real
KWin capture → HEVC). LUMEN_ZEROCOPY is opt-in/experimental until the CUDA
registration lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scaffolding for dmabuf zero-copy (plan §9), opt-in via LUMEN_ZEROCOPY:
- src/zerocopy/{cuda,egl}.rs: hand-rolled CUDA Driver-API FFI (no Rust crate
exposes the EGL-interop calls / CUeglFrame) with a shared process-wide
CUcontext + pitched device buffers; an EGL importer (GBM platform on the
NVIDIA render node) that turns a dmabuf into an EGLImage, registers it with
CUDA, and copies it device-to-device into an owned buffer. `zerocopy-probe`
subcommand validates the FFI/linking/GPU access — confirmed on the box
(driver 595, EGL_EXT_image_dma_buf_import + modifiers).
- CapturedFrame gains a FramePayload enum (Cpu(Vec<u8>) | Cuda(DeviceBuffer));
the encoder branches: CPU keeps the expand+upload path, CUDA wraps the device
buffer in an AV_PIX_FMT_CUDA frame fed straight to hevc_nvenc (sharing our
CUcontext via a hand-declared AVCUDADeviceContext, since ffmpeg-sys doesn't
bind hwcontext_cuda.h). open_video/the encoder take a `cuda` flag derived from
the first frame's payload.
The capture-side dmabuf negotiation (which produces the Cuda frames) is the
next step; the CPU path is unchanged and remains the default + fallback. Builds
clean, clippy clean, tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The M2 teardown work added an `active` gate to the PipeWire capture callback
(idle by default so reconnects stay cheap, with the stream path calling
set_active(true) on PLAY). The `m0` subcommand was never updated, so its portal
capturer stayed inactive and the callback dropped every frame — `m0 --source
portal` failed with "no PipeWire frame within 10s" on every compositor. Call
set_active(true) before the capture loop.
Validated on headless KWin (Plasma 6.4) via the RemoteDesktop-anchored ScreenCast
session: real desktop frames flow (shm BGRx 1920x1080) and encode to valid H.265.
(Also folds in a rustfmt reflow of the input-test log line.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On RemoteDesktop-capable desktops (KWin, GNOME), select the ScreenCast source on
a session created via the RemoteDesktop portal and start it through RemoteDesktop,
so a single grant — pre-authorized headlessly via the `kde-authorized` permission,
exactly like the libei input path — also covers screen capture. Standalone
ScreenCast has no such bypass and would raise an un-clickable dialog on a headless
box. wlroots/Sway has no RemoteDesktop portal, so it keeps the plain ScreenCast
session; the choice keys off inject::default_backend(). The PipeWire consumer is
unchanged — the anchored session yields the same fd + node id.
Validated on headless KWin (Plasma 6.4): the portal grants the session with no
dialog and PipeWire negotiates the format (1920x1080 BGRx, Streaming). Frame
delivery on KWin still pends dmabuf import — KWin hands GPU dmabuf buffers and the
M0 consumer is CPU-copy/shm only (plan §9, zero-copy) — so it's the next step;
the CPU-copy path remains correct on wlroots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehler