NixOS users compiled the whole workspace because we published no binaries #313

Merged

enricobuehler merged 2 commits from worktree-nix-binary-cache into main 2026-08-18 21:15:56 +00:00

Conversation 0 Commits 2 Files Changed 7

+512 -35

2 Commits

Author	SHA1	Message	Date
enricobuehler	dd097d1ef2	chore(nix): record disk headroom after the publish build too ... This job is now the heaviest on the fleet — a full workspace build plus gamescope fills the store with tens of GB, and this fleet ran a runner out of disk on 2026-08-06. The pre-existing Environment step reads df before any of that happens, which is the less useful of the two moments.	2026-08-18 22:43:47 +02:00
enricobuehler	66249710b9	feat(nix): publish a binary cache so NixOS users stop compiling the workspace ... Every other channel ships prebuilt binaries; Nix was the exception — `nix build` meant the whole Rust workspace *and* a gamescope build from source, roughly an hour, and `host.gamescopeHdr` defaults true so that compositor build is on the critical path of every `services.punktfunk.host.enable = true`. nix.yml grows a third tier: on a push to main it builds the Rust packages plus gamescope, signs them, and publishes to https://nix.unom.io. No new trigger is needed for releases — a release bumps the workspace version in Cargo.toml, which is already in the path filter. Gitea cannot host this: it has 23 package registry types and none is Nix, and the protocol wants fixed anonymous paths at a URL root (/nix-cache-info, /<hash>.narinfo, /nar/…) that /api/packages/{owner}/generic/… cannot express. The RustFS at storage.unom.io would work mechanically — nix speaks s3://…?endpoint= and the sccache credentials already exist — but it is a local box on the home uplink with no CDN, so every user download would compete with CI, and S3 answers 403 for a missing key unless the bucket policy grants anonymous ListBucket. Nix treats anything other than 404 as a hard error rather than a cache miss, so that would break users' builds for packages the cache never held. So it goes on unom-1 beside the flatpak repo, as a caddy:2-alpine container serving a static tree — which is all a binary cache is. Three decisions worth keeping: * Only punktfunk's own store paths are published. The rest of a runtime closure is stock nixpkgs, already on cache.nixos.org behind a real CDN; mirroring it would spend disk and home-to-cloud bandwidth to serve a worse copy. That is ~300 MB per publish instead of several GB. The step asserts every built output is matched by the name filter, so a future pname change fails the build rather than silently dropping the most expensive package from the cache. * NARs upload before narinfos, and rsync runs without --delete. A narinfo whose NAR has not landed is a hard download failure for whoever fetches it in that window; a NAR nothing points at is merely invisible. * prune.sh from the first publish, not after the box fills. The flatpak repo next door reached 3.84 GB publishing this same way with no sweep, on a box that has run out of disk before. It ages out narinfos, then sweeps unreferenced NARs — that order is the correctness argument, and it carries a self-check. Verified locally: the Caddyfile serves a fixture cache with 200s on hits, 404 on misses (the assertion the whole design rests on), and immutable cache headers; prune.sh passes its self-test on Debian including empty and all-stale caches; both workflows parse; the new run blocks pass shellcheck and dash -n. `set -eu` rather than `-euo pipefail` — dash dies on the latter — with the two pipelines whose left side must be able to fail rewritten as redirects. Docs: README gains the substituter snippet, a maintainer runbook, and a warning that inputs.punktfunk.inputs.nixpkgs.follows disables the cache entirely (every store path changes, so every package rebuilds). The install guide gains the same in short form. The public key is a fill-in until the first publish prints it — see the setup steps in packaging/nix/README.md.	2026-08-18 22:38:17 +02:00