NixOS users compiled the whole workspace because we published no binaries #313

Merged

enricobuehler merged 2 commits from worktree-nix-binary-cache into main 2026-08-18 21:15:56 +00:00

Conversation 0 Commits 2 Files Changed 7

+512 -35

enricobuehler commented 2026-08-18 20:38:48 +00:00

Owner

Copy Link

Copy Source

Every other channel ships prebuilt binaries; Nix was the exception. nix build meant the whole Rust workspace and a gamescope build from source — roughly an hour — and since host.gamescopeHdr defaults true, that compositor build is on the critical path of every services.punktfunk.host.enable = true.

nix.yml grows a third tier: on a push to main it builds the Rust packages plus gamescope, signs them, and publishes to https://nix.unom.io. No new trigger is needed for releases — a release bumps the workspace version in Cargo.toml, which is already in the path filter.

Why not Gitea, and why not storage.unom.io

• Gitea has 23 package registry types and none is Nix. Not a missing label: the protocol needs fixed anonymous paths at a URL root (/nix-cache-info, /<hash>.narinfo, /nar/…), which /api/packages/{owner}/generic/{name}/{version}/{file} cannot express.
• The RustFS at storage.unom.io would work mechanically — nix speaks s3://…?endpoint= and the sccache credentials already exist — but it is a local box on the home uplink with no CDN, so every user download would compete with CI. It also answers 403 for a missing key unless the bucket policy grants anonymous ListBucket, and nix treats anything other than 404 as a hard error rather than a cache miss. That would break users' builds for packages the cache never held.

So it lands on unom-1 beside the flatpak repo, as a caddy:2-alpine container serving a static tree — which is all a binary cache is.

Three decisions worth reviewing

• Only punktfunk's own store paths are published. The rest of a runtime closure is stock nixpkgs, already on cache.nixos.org behind a real CDN; mirroring it would spend disk and home-to-cloud bandwidth to serve a worse copy. ~300 MB per publish instead of several GB. The step asserts every built output is matched by the name filter, so a future pname change fails the build rather than silently dropping the most expensive package from the cache.
• NARs upload before narinfos, and rsync runs without --delete. A narinfo whose NAR has not landed is a hard download failure for whoever fetches it in that window; a NAR nothing points at is merely invisible.
• prune.sh from the first publish, not after the box fills. The flatpak repo next door reached 3.84 GB publishing this same way with no sweep, on a box that has run out of disk before. It ages out narinfos, then sweeps unreferenced NARs — that order is the correctness argument — and it carries a self-check.

Verified locally

• The Caddyfile serves a fixture cache: 200 on hits, 404 on misses (the assertion the whole design rests on), immutable cache headers, text/plain on narinfos.
• prune.sh passes its self-test on Debian, including empty and all-stale caches, and the shared-NAR case.
• Both workflows parse; the new run blocks pass shellcheck (POSIX) and dash -n. set -eu rather than -euo pipefail — dash dies on the latter — with the two pipelines whose left side must be able to fail rewritten as redirects.

Not verified: an actual nix copy / end-to-end publish, which needs a Nix box and the deploy secrets.

Before this does anything

Setup is documented in packaging/nix/README.md and is deliberately ordered — service first, secret last, so main stays green throughout:

1. Edge proxy nix.unom.io → 192.168.50.50:3250
2. Add 3250 to caddy_target_ports in unom/infra + terraform apply
3. DNS for nix.unom.io
4. Dispatch deploy-services.yml
5. nix key generate-secret --key-name punktfunk-cache-1 → repo secret NIX_CACHE_SIGNING_KEY
6. Push to main touching the flake; the publish step prints the public key

⚠ The public key is a <fill-in> in the docs until step 6 prints it. Both packaging/nix/README.md and docs-site/content/docs/install.md need that one value pasted in before the docs are useful to anyone.

Docs also gained a warning that inputs.punktfunk.inputs.nixpkgs.follows = "nixpkgs" disables the cache entirely — different inputs, different store paths, so every package rebuilds. That line is currently suggested in the README as an unqualified nicety.

Every other channel ships prebuilt binaries; Nix was the exception. `nix build` meant the whole Rust workspace **and** a gamescope build from source — roughly an hour — and since `host.gamescopeHdr` defaults true, that compositor build is on the critical path of every `services.punktfunk.host.enable = true`. `nix.yml` grows a third tier: on a push to `main` it builds the Rust packages plus gamescope, signs them, and publishes to `https://nix.unom.io`. No new trigger is needed for releases — a release bumps the workspace version in `Cargo.toml`, which is already in the path filter. ## Why not Gitea, and why not storage.unom.io - **Gitea** has 23 package registry types and none is Nix. Not a missing label: the protocol needs fixed anonymous paths at a URL *root* (`/nix-cache-info`, `/<hash>.narinfo`, `/nar/…`), which `/api/packages/{owner}/generic/{name}/{version}/{file}` cannot express. - **The RustFS at `storage.unom.io`** would work mechanically — nix speaks `s3://…?endpoint=` and the sccache credentials already exist — but it is a local box on the home uplink with no CDN, so every user download would compete with CI. It also answers **403** for a missing key unless the bucket policy grants anonymous `ListBucket`, and nix treats anything other than **404** as a hard error rather than a cache miss. That would break users' builds for packages the cache never held. So it lands on unom-1 beside the flatpak repo, as a `caddy:2-alpine` container serving a static tree — which is all a binary cache is. ## Three decisions worth reviewing - **Only punktfunk's own store paths are published.** The rest of a runtime closure is stock nixpkgs, already on `cache.nixos.org` behind a real CDN; mirroring it would spend disk and home-to-cloud bandwidth to serve a worse copy. ~300 MB per publish instead of several GB. The step asserts every built output is matched by the name filter, so a future `pname` change fails the build rather than silently dropping the most expensive package from the cache. - **NARs upload before narinfos, and rsync runs without `--delete`.** A narinfo whose NAR has not landed is a hard download failure for whoever fetches it in that window; a NAR nothing points at is merely invisible. - **`prune.sh` from the first publish, not after the box fills.** The flatpak repo next door reached 3.84 GB publishing this same way with no sweep, on a box that has run out of disk before. It ages out narinfos, *then* sweeps unreferenced NARs — that order is the correctness argument — and it carries a self-check. ## Verified locally - The Caddyfile serves a fixture cache: 200 on hits, **404 on misses** (the assertion the whole design rests on), immutable cache headers, `text/plain` on narinfos. - `prune.sh` passes its self-test on Debian, including empty and all-stale caches, and the shared-NAR case. - Both workflows parse; the new run blocks pass shellcheck (POSIX) and `dash -n`. `set -eu` rather than `-euo pipefail` — dash dies on the latter — with the two pipelines whose left side must be able to fail rewritten as redirects. Not verified: an actual `nix copy` / end-to-end publish, which needs a Nix box and the deploy secrets. ## Before this does anything Setup is documented in `packaging/nix/README.md` and is deliberately ordered — service first, secret last, so `main` stays green throughout: 1. Edge proxy `nix.unom.io → 192.168.50.50:3250` 2. Add `3250` to `caddy_target_ports` in `unom/infra` + terraform apply 3. DNS for `nix.unom.io` 4. Dispatch `deploy-services.yml` 5. `nix key generate-secret --key-name punktfunk-cache-1` → repo secret `NIX_CACHE_SIGNING_KEY` 6. Push to `main` touching the flake; the publish step prints the **public** key ⚠ **The public key is a `<fill-in>` in the docs until step 6 prints it.** Both `packaging/nix/README.md` and `docs-site/content/docs/install.md` need that one value pasted in before the docs are useful to anyone. Docs also gained a warning that `inputs.punktfunk.inputs.nixpkgs.follows = "nixpkgs"` disables the cache entirely — different inputs, different store paths, so every package rebuilds. That line is currently suggested in the README as an unqualified nicety.

enricobuehler added 1 commit 2026-08-18 20:38:49 +00:00

feat(nix): publish a binary cache so NixOS users stop compiling the workspace ... 66249710b9

Every other channel ships prebuilt binaries; Nix was the exception — `nix build`
meant the whole Rust workspace *and* a gamescope build from source, roughly an
hour, and `host.gamescopeHdr` defaults true so that compositor build is on the
critical path of every `services.punktfunk.host.enable = true`.

nix.yml grows a third tier: on a push to main it builds the Rust packages plus
gamescope, signs them, and publishes to https://nix.unom.io. No new trigger is
needed for releases — a release bumps the workspace version in Cargo.toml, which
is already in the path filter.

Gitea cannot host this: it has 23 package registry types and none is Nix, and the
protocol wants fixed anonymous paths at a URL root (/nix-cache-info,
/<hash>.narinfo, /nar/…) that /api/packages/{owner}/generic/… cannot express.
The RustFS at storage.unom.io would work mechanically — nix speaks
s3://…?endpoint= and the sccache credentials already exist — but it is a local
box on the home uplink with no CDN, so every user download would compete with CI,
and S3 answers 403 for a missing key unless the bucket policy grants anonymous
ListBucket. Nix treats anything other than 404 as a hard error rather than a
cache miss, so that would break users' builds for packages the cache never held.
So it goes on unom-1 beside the flatpak repo, as a caddy:2-alpine container
serving a static tree — which is all a binary cache is.

Three decisions worth keeping:

* Only punktfunk's own store paths are published. The rest of a runtime closure
  is stock nixpkgs, already on cache.nixos.org behind a real CDN; mirroring it
  would spend disk and home-to-cloud bandwidth to serve a worse copy. That is
  ~300 MB per publish instead of several GB. The step asserts every built output
  is matched by the name filter, so a future pname change fails the build rather
  than silently dropping the most expensive package from the cache.
* NARs upload before narinfos, and rsync runs without --delete. A narinfo whose
  NAR has not landed is a hard download failure for whoever fetches it in that
  window; a NAR nothing points at is merely invisible.
* prune.sh from the first publish, not after the box fills. The flatpak repo next
  door reached 3.84 GB publishing this same way with no sweep, on a box that has
  run out of disk before. It ages out narinfos, then sweeps unreferenced NARs —
  that order is the correctness argument, and it carries a self-check.

Verified locally: the Caddyfile serves a fixture cache with 200s on hits, 404 on
misses (the assertion the whole design rests on), and immutable cache headers;
prune.sh passes its self-test on Debian including empty and all-stale caches;
both workflows parse; the new run blocks pass shellcheck and dash -n. `set -eu`
rather than `-euo pipefail` — dash dies on the latter — with the two pipelines
whose left side must be able to fail rewritten as redirects.

Docs: README gains the substituter snippet, a maintainer runbook, and a warning
that inputs.punktfunk.inputs.nixpkgs.follows disables the cache entirely (every
store path changes, so every package rebuilds). The install guide gains the same
in short form.

The public key is a fill-in until the first publish prints it — see the setup
steps in packaging/nix/README.md.

enricobuehler added 1 commit 2026-08-18 20:45:41 +00:00

chore(nix): record disk headroom after the publish build too ... dd097d1ef2

This job is now the heaviest on the fleet — a full workspace build plus
gamescope fills the store with tens of GB, and this fleet ran a runner out of
disk on 2026-08-06. The pre-existing Environment step reads df before any of
that happens, which is the less useful of the two moments.

enricobuehler merged commit 4e03dcc280 into main 2026-08-18 21:15:56 +00:00

enricobuehler deleted branch worktree-nix-binary-cache 2026-08-18 21:16:00 +00:00

enricobuehler referenced this pull request 2026-08-18 21:27:01 +00:00

The Nix cache's setup steps pointed at a home-lab proxy that no longer exists #316

enricobuehler referenced this pull request 2026-08-18 21:50:04 +00:00

The cache signing key is installed, and its DNS was never a dashboard click #318

enricobuehler referenced this issue from a commit 2026-08-18 22:18:30 +00:00

release: 0.31.0 — version bump, notes, CHANGELOG, Play notes

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX HitFrostbite enricobuehler (Enrico Bühler) kszlim luxus nannku neoney sassycrown

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#313