punktfunk/docs/linux-setup.md

Linux host setup — NVIDIA GPU VM (M0/M2)

How to bring up the build environment for the lumen Linux host on an NVIDIA-GPU Ubuntu VM and run the M0 capture→encode spike. lumen-core already builds and is tested cross-platform; this is about the platform backends in crates/lumen-host.

Target Ubuntu 24.04 (noble): Sway 1.9, FFmpeg 6.1.1, xdg-desktop-portal 1.18. 22.04 (jammy) ships Sway 1.7 / FFmpeg 4.4 — too old for this path; build from source or upgrade. Package names/versions below were verified against the live Ubuntu archive.

1. Bootstrap

git clone git@git.unom.io:unom/lumen.git && cd lumen && git checkout m1-lumen-core
bash scripts/bootstrap-ubuntu.sh

It verifies the (already-installed) NVIDIA + NVENC stack, installs the Rust toolchain (rustup) and the build/runtime deps (PipeWire, xdg-desktop-portal + the wlroots backend, Sway, Wayland/DRM/EGL/GBM/VA dev libs, capture tools), gates the FFmpeg -dev headers so it can't clobber your custom NVENC FFmpeg, and drops headless-Sway + portal config templates into ~/.config (only if absent). It does not reboot or edit GRUB.

After it runs, sanity-check the core on Linux:

cargo test --workspace        # 21 tests; same suite that's green on macOS

2. NVIDIA prerequisites (one-time, may need a reboot)

Wayland on NVIDIA requires KMS modeset. The bootstrap checks it; if it isn't Y:

echo 'options nvidia-drm modeset=1 fbdev=1' | sudo tee /etc/modprobe.d/nvidia-drm.conf
sudo update-initramfs -u && sudo reboot
cat /sys/module/nvidia_drm/parameters/modeset   # must print Y after reboot

• Driver ≥ 535 is the floor for headless wlroots (EGL/dmabuf); 550+ recommended.
• A headless VM GPU exposes no DRM connectors — that's expected. We don't use the DRM backend; WLR_BACKENDS=headless renders to an offscreen GBM/EGL surface and creates a virtual HEADLESS-1 output. Use the render node /dev/dri/renderD128.
• NVENC in a VM: full PCI passthrough = bare-metal NVENC, no license. vGPU needs a valid license (vWS) or NVENC runs degraded — the bootstrap's smoke-encode tells you if it actually works. Consumer GeForce cards also cap concurrent NVENC sessions (~8); datacenter/RTX-pro are effectively unlimited — relevant once we serve many clients.

3. Bring up the headless compositor + prove capture→NVENC

# shell 1 — start headless Sway (prints WAYLAND_DISPLAY, default wayland-1)
bash scripts/headless/run-headless-sway.sh

# shell 2 — same user
export XDG_RUNTIME_DIR=/run/user/$(id -u) WAYLAND_DISPLAY=wayland-1
swaymsg -t get_outputs                       # confirm HEADLESS-1
swaymsg output HEADLESS-1 resolution 2560x1440@60Hz   # set your client size
bash scripts/headless/capture-smoke-test.sh  # wf-recorder (wlr-screencopy) -> hevc_nvenc
ffprobe /tmp/lumen-headless-test.mkv         # confirm a real H.265 stream

wf-recorder uses wlr-screencopy directly (no portal/D-Bus) — the fastest way to de-risk the GPU encode path. Note: screencopy encodes straight to a file and cannot feed PipeWire; the real integration uses the ScreenCast portal (see M0).

The wlroots-on-NVIDIA env workarounds (WLR_RENDERER=gles2, WLR_NO_HARDWARE_CURSORS=1, GBM_BACKEND=nvidia-drm, sway --unsupported-gpu, …) live in scripts/headless/env.sh — source it before launching anything Wayland.

4. M0 proper — wire it into lumen-core

Goal (plan §8): headless output → PipeWire ScreenCast → NVENC → a playable file, then feed the encoded access units into a lumen_core::Session (host role). The module seams exist in crates/lumen-host/src/{vdisplay,capture,encode,inject,pipeline}.rs.

Crate choices, verified current:

• Capture (portal path): ashpd 0.13 with the screencast + pipewire features → ScreenCast::create_session → select_sources (Monitor) → start → pipe_wire_node_id() + open_pipe_wire_remote(); pull frames with pipewire 0.10. (crates.io's "newest" field shows 0.9 for ashpd — ignore it, pin 0.13.) Set XDG_CURRENT_DESKTOP=sway so the wlr portal backend is chosen.
• Encode: ffmpeg-next 7.x (binds the system FFmpeg 6.1.1 via pkg-config; needs clang/libclang). Select the encoder by name — encoder::find_by_name("hevc_nvenc"), not by codec id (that's the SW encoder). Low-latency opts: preset=p1, tune=ull, rc=cbr, bf=0, delay=0, large g. If your FFmpeg is in a non-standard prefix, export FFMPEG_DIR=/that/prefix.
• Zero-copy is the hard part. There's no direct dmabuf→CUDA import in FFmpeg. Start with the CPU-copy fallback (download frame → hwupload_cuda → hevc_nvenc) to get an end-to-end stream, then chase true dmabuf zero-copy. The plan flags this (§9) and the capture module already has a cpu_bytes fallback field.
• Input (M2): reis (pure-Rust libei — no native libei needed) with input-linux/uinput as the universal fallback.

Then continue toward M2: serverinfo/RTSP/pairing enough for a stock Moonlight client to connect, a KWin virtual output created on connect, input via reis/uinput — the shippable milestone.

Troubleshooting

Symptom	Fix
Sway aborts on NVIDIA	add --unsupported-gpu (the helper scripts do)
not a KMS device / no connectors	expected on a headless VM GPU — use WLR_BACKENDS=headless, not the DRM backend
Sway won't start at all	WLR_RENDERER_ALLOW_SOFTWARE=1 WLR_RENDERER=pixman to prove the pipeline, then fix EGL
ScreenCast portal finds no output	ensure xdg-desktop-portal-wlr is running in the same session, XDG_CURRENT_DESKTOP=sway, and ~/.config/xdg-desktop-portal-wlr/config has output_name=HEADLESS-1
Cannot load libnvidia-encode.so.1	NVENC runtime lib missing (driver) or unlicensed vGPU
cargo build can't find FFmpeg	export FFMPEG_DIR=$(pkg-config --variable=prefix libavcodec) or point PKG_CONFIG_PATH at the custom build
bindgen: libclang not found	export LIBCLANG_PATH=$(llvm-config --libdir)