From e78906fb06f0567e9c821f4bda4da785ff7c6948 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Thu, 6 Aug 2026 19:31:16 +0200 Subject: [PATCH] fix(ci/nix): the flake job could never start its container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nix gate landed in fb707b49 on `nixos/nix:latest`, and it has never run a single step. That image carries nix and essentially nothing else — including no `/bin/sleep` — and Gitea's act_runner starts every job container with `entrypoint=["/bin/sleep", "10800"]`: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/sleep": stat /bin/sleep: no such file or directory The expensive part is how it reports: with the container dead, every step is marked `cancelled` rather than `failed`, which reads exactly like a run that was superseded by a newer push. Run 15907 looked skipped, not broken. Switch to `node:22-bookworm` and install Nix in a step: * full Debian, so the entrypoint exists and coreutils are present; * a real node, so `actions/checkout` works with no pre-checkout install dance (the reason flatpak.yml's fedora job installs node before its checkout); * audit.yml already pulls this image on this fleet, so it is known to resolve. Nix comes from the Determinate installer with `--init none` — the container mode: no systemd, no daemon. That distribution is also what the hand-verification Nix box (.21) runs, so CI and it stay on the same Nix. MEASURED in a real `node:22-bookworm` container rather than assumed, since the last version of this file shipped on an untested assumption and cost a red run: * `/bin/sleep` present — the entrypoint failure is gone; * the installer completes and `nix --version` runs from the absolute path; * `nix build` of a trivial derivation SUCCEEDS, `nix store ping` reports `Store URL: local, Trusted: 1`, and flakes are enabled. That last one is the trap this change also pins. `--init none` runs no daemon, but the installer still writes a profile script exporting `NIX_REMOTE=daemon`; anything sourcing it (any `-l` login shell) then dies on "cannot connect to socket at '/nix/var/nix/daemon-socket/socket'". It is why the installer's own self-test fails, harmlessly, in the middle of this step's log. The steps here never source that profile — they invoke `$NIX` by absolute path — but `NIX_REMOTE` is now pinned empty at job level so a later step cannot reintroduce it. Also records `df -h` before the build: this fleet ran a runner out of disk today (ci.yml's `web` job died with "no space left on device" mid-`bun install`), a Nix build is the heaviest thing that would run here, and a future failure should be attributable at a glance rather than guessed. Still unproven, and only the first green run can settle it: whether `nix flake check` evaluates this flake cleanly under CI, and whether the runner has the disk to build punktfunk-web. Both now produce a real diagnostic instead of a container that never started. --- .gitea/workflows/nix.yml | 77 +++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 21 deletions(-) diff --git a/.gitea/workflows/nix.yml b/.gitea/workflows/nix.yml index b9d6a41f..5693d755 100644 --- a/.gitea/workflows/nix.yml +++ b/.gitea/workflows/nix.yml @@ -71,52 +71,86 @@ jobs: flake: runs-on: ubuntu-24.04 container: - # Official Nix image (Docker Hub, like this fleet's other WAN images: oven/bun:1, - # fedora:43, node:22-bookworm). It ships nix and little else. - image: nixos/nix:latest + # NOT nixos/nix. That image contains nix and essentially nothing else — in particular no + # /bin/sleep, and Gitea's act_runner starts every job container with + # `entrypoint=["/bin/sleep","10800"]`. The container therefore never starts: + # failed to create shim task: OCI runtime create failed: unable to start container + # process: exec: "/bin/sleep": stat /bin/sleep: no such file or directory + # and — the part that makes this expensive to debug — every step is then reported as + # `cancelled` rather than failed, which reads exactly like a superseded run. + # + # node:22-bookworm instead: a full Debian with coreutils (so the entrypoint exists) and a + # real node (so actions/checkout works with no pre-checkout install dance), and audit.yml + # already pulls it on this fleet, so it is proven to resolve here. Nix is installed below. + image: node:22-bookworm timeout-minutes: 90 env: - # The image defaults to stable Nix with the experimental features off; the flake needs both. - # Set at job level so every step — including the `nix profile install` below — sees it. + # The flake needs both experimental features. Also baked into the installer's --extra-conf + # below; this covers any step that shells out before that config is read. NIX_CONFIG: "experimental-features = nix-command flakes" + # Absolute path rather than $GITHUB_PATH: one less runner behaviour to assume. + NIX: /nix/var/nix/profiles/default/bin/nix + # `--init none` installs Nix with NO daemon running, but the installer still writes a profile + # script that exports NIX_REMOTE=daemon. Anything that sources it (any `-l` login shell) then + # dies on `cannot connect to socket at '/nix/var/nix/daemon-socket/socket'` — which is exactly + # how the installer's own self-test fails during this step, harmlessly, and would be a + # confusing first thing to read in the log. The steps below never source that profile, but pin + # the empty value so a future step cannot reintroduce it. Empty = talk to the local store + # directly, which works because the job runs as root (MEASURED: "Store URL: local, Trusted: 1", + # and a real `nix build` of a trivial derivation succeeds). + NIX_REMOTE: "" steps: - # actions/checkout is a JS action and needs node; a plain `run:` step executes through the - # container shell, so this must come BEFORE the checkout (same ordering, and the same - # reason, as flatpak.yml's fedora job). - - name: node + git for the JS actions - run: nix profile install nixpkgs#nodejs nixpkgs#git - - uses: actions/checkout@v4 + # The Determinate installer needs curl + xz; git so nix can read the flake from the checkout. + # (node:22-bookworm is the full image and already has all three — this is belt-and-braces + # against a future slim-image swap, and costs one cached apt call.) + - name: Installer prerequisites + run: apt-get update && apt-get install -y --no-install-recommends ca-certificates curl xz-utils git + + # `--init none` is the container mode: no systemd, no daemon. Running as root, nix then talks + # to the store directly. Determinate Nix is also what the Nix box (.21) runs, so CI and the + # hand-verification box stay on the same distribution. + - name: Install Nix + run: | + curl -fsSL https://install.determinate.systems/nix -o /tmp/nix-installer.sh + sh /tmp/nix-installer.sh install linux --init none --no-confirm \ + --extra-conf "experimental-features = nix-command flakes" + "$NIX" --version + # Nix reads the flake through libgit2 and refuses a checkout owned by another uid # ("detected dubious ownership"), which is the normal case for a container job. - name: Trust the checkout run: git config --global --add safe.directory "$PWD" - # First-run diagnostics — cheap, and the difference between "the gate found a real problem" - # and "the runner had no disk" is otherwise a guess. + # Diagnostics. This fleet ran a runner out of disk on 2026-08-06 (the ci.yml `web` job died + # with "no space left on device" mid-`bun install`), and a Nix build is the heaviest thing + # here — so record the headroom, or a future failure is a guess. - name: Environment - run: | - nix --version - df -h /nix /tmp || true + run: df -h / /nix /tmp || true # Evaluates + instantiates every flake output without building any of it. - name: nix flake check (eval only) - run: nix flake check --no-build --show-trace + run: | + "$NIX" flake check --no-build --show-trace # The bun packages, built for real. This is the leg that would have caught the stale # web/bun.nix end to end: the derivation's offline `bun install` runs against a store cache # built strictly from bun.nix, so a lockfile that cache does not cover fails here. + # Path-filtered, so it runs only when the packaging or a lockfile actually moves. If it ever + # starts going red on runner disk rather than on real defects, demote it to the dispatch + # opt-in below rather than leaving an infra-red gate on the board. - name: Build the bun packages - run: nix build --print-build-logs .#punktfunk-web .#punktfunk-scripting + run: | + "$NIX" build --print-build-logs .#punktfunk-web .#punktfunk-scripting # Both launchers exec pkgs.bun from the store; confirm they were produced and are real entry # points rather than dangling wrappers. - name: Smoke the built launchers run: | set -eu - web=$(nix path-info .#punktfunk-web) - scripting=$(nix path-info .#punktfunk-scripting) + web=$("$NIX" path-info .#punktfunk-web) + scripting=$("$NIX" path-info .#punktfunk-scripting) test -x "$web/bin/punktfunk-web-server" || { echo "no punktfunk-web-server in $web" >&2; exit 1; } test -x "$scripting/bin/punktfunk-scripting" || { echo "no punktfunk-scripting in $scripting" >&2; exit 1; } # The console must be the bun bundle, not a node one — the same assertion packages.nix @@ -129,4 +163,5 @@ jobs: # `github.event.inputs.*` (string) rather than `inputs.*` — the portable spelling. - name: Build the Rust packages (dispatch opt-in) if: ${{ github.event.inputs.build-rust == 'true' }} - run: nix build --print-build-logs .#punktfunk-host .#punktfunk-client + run: | + "$NIX" build --print-build-logs .#punktfunk-host .#punktfunk-client