fix(ci/nix): the flake job could never start its container #77

Merged

enricobuehler merged 1 commits from worktree-nix-flake-container-fix into main 2026-08-06 17:45:36 +00:00

Conversation 0 Commits 1 Files Changed 1

+56 -21

enricobuehler commented 2026-08-06 17:31:39 +00:00

Owner

Copy Link

Copy Source

It never ran a step

The nix gate landed in fb707b49 on nixos/nix:latest. That image carries nix and essentially nothing else — including no /bin/sleep — and Gitea's act_runner starts every job container with entrypoint=["/bin/sleep", "10800"]:

failed to create shim task: OCI runtime create failed: runc create failed:
unable to start container process: exec: "/bin/sleep": stat /bin/sleep: no such file or directory

The expensive part is how it reports. With the container dead, every step is marked cancelled, not failed — which reads exactly like a run superseded by a newer push. Run 15907 looked skipped, not broken.

For the record, the other half of that PR is fine: the bun-nix drift gate passed in real CI (run 15906, job 34626) in 66 s. That is the always-on, load-bearing gate and it is unaffected by this.

The fix

node:22-bookworm, with Nix installed in a step:

• full Debian, so the entrypoint exists and coreutils are present;
• a real node, so actions/checkout works with no pre-checkout install dance (the reason flatpak.yml's fedora job installs node before its checkout);
• audit.yml already pulls this image on this fleet, so it is known to resolve here.

Nix comes from the Determinate installer with --init none — the container mode: no systemd, no daemon. That is also the distribution the hand-verification Nix box (.21) runs, so CI and it stay on the same Nix.

Measured, not assumed

The previous version shipped on an untested assumption and cost a red run, so this one was exercised in a real node:22-bookworm container first:

check	result
/bin/sleep present	✅ the entrypoint failure is gone
Determinate installer completes	✅ nix (Determinate Nix 3.21.9) 2.34.8
nix --version from the absolute path	✅
nix build of a trivial derivation	✅ builds
nix store ping	✅ Store URL: local, Trusted: 1
flakes enabled	✅

The daemon trap this also pins

--init none runs no daemon, but the installer still writes a profile script exporting NIX_REMOTE=daemon. Anything that sources it — any -l login shell — then dies on cannot connect to socket at '/nix/var/nix/daemon-socket/socket'. That is precisely why the installer's own self-test fails, harmlessly, in the middle of this step's log, and it would be a confusing first thing to read.

The steps here never source that profile (they invoke $NIX by absolute path), but NIX_REMOTE is now pinned empty at job level so a later step cannot reintroduce it. Empty = talk to the local store directly, which works because the job runs as root — measured above.

Also

Records df -h before the build. This fleet ran a runner out of disk today — ci.yml's web job died with no space left on device mid-bun install on home-runner-2 — and a Nix build is the heaviest thing that would run here. A future failure should be attributable at a glance rather than guessed.

Still unproven

Only the first green run can settle these, and I have flagged them rather than claim otherwise:

• whether nix flake check evaluates this flake cleanly under CI (it eval-fetches the pinned windows-rs git rev through crane);
• whether the runner has the disk to build punktfunk-web.

Both now produce a real diagnostic instead of a container that never started. If the build leg proves to go red on disk rather than on real defects, demote it to the workflow_dispatch opt-in rather than leaving an infra-red gate on the board — there is a comment in the file saying so.

## It never ran a step The nix gate landed in `fb707b49` on `nixos/nix:latest`. That image carries nix and essentially nothing else — including **no `/bin/sleep`** — and Gitea's act_runner starts every job container with `entrypoint=["/bin/sleep", "10800"]`: ``` failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/sleep": stat /bin/sleep: no such file or directory ``` The expensive part is how it reports. With the container dead, **every step is marked `cancelled`, not `failed`** — which reads exactly like a run superseded by a newer push. [Run 15907](https://git.unom.io/unom/punktfunk/actions/runs/15907) looked skipped, not broken. For the record, the other half of that PR is fine: the `bun-nix` drift gate **passed in real CI** ([run 15906, job 34626](https://git.unom.io/unom/punktfunk/actions/runs/15906)) in 66 s. That is the always-on, load-bearing gate and it is unaffected by this. ## The fix `node:22-bookworm`, with Nix installed in a step: - full Debian, so the entrypoint exists and coreutils are present; - a real node, so `actions/checkout` works with no pre-checkout install dance (the reason `flatpak.yml`'s fedora job installs node before its checkout); - `audit.yml` already pulls this image on this fleet, so it is known to resolve here. Nix comes from the Determinate installer with `--init none` — the container mode: no systemd, no daemon. That is also the distribution the hand-verification Nix box (`.21`) runs, so CI and it stay on the same Nix. ## Measured, not assumed The previous version shipped on an untested assumption and cost a red run, so this one was exercised in a real `node:22-bookworm` container first: | check | result | |---|---| | `/bin/sleep` present | ✅ the entrypoint failure is gone | | Determinate installer completes | ✅ `nix (Determinate Nix 3.21.9) 2.34.8` | | `nix --version` from the absolute path | ✅ | | `nix build` of a trivial derivation | ✅ builds | | `nix store ping` | ✅ `Store URL: local, Trusted: 1` | | flakes enabled | ✅ | ## The daemon trap this also pins `--init none` runs **no daemon**, but the installer still writes a profile script exporting `NIX_REMOTE=daemon`. Anything that sources it — any `-l` login shell — then dies on `cannot connect to socket at '/nix/var/nix/daemon-socket/socket'`. That is precisely why the installer's *own self-test* fails, harmlessly, in the middle of this step's log, and it would be a confusing first thing to read. The steps here never source that profile (they invoke `$NIX` by absolute path), but `NIX_REMOTE` is now pinned empty at job level so a later step cannot reintroduce it. Empty = talk to the local store directly, which works because the job runs as root — measured above. ## Also Records `df -h` before the build. This fleet **ran a runner out of disk today** — `ci.yml`'s `web` job died with `no space left on device` mid-`bun install` on `home-runner-2` — and a Nix build is the heaviest thing that would run here. A future failure should be attributable at a glance rather than guessed. ## Still unproven Only the first green run can settle these, and I have flagged them rather than claim otherwise: - whether `nix flake check` evaluates this flake cleanly under CI (it eval-fetches the pinned windows-rs git rev through crane); - whether the runner has the disk to build `punktfunk-web`. Both now produce a **real diagnostic** instead of a container that never started. If the build leg proves to go red on disk rather than on real defects, demote it to the `workflow_dispatch` opt-in rather than leaving an infra-red gate on the board — there is a comment in the file saying so.

enricobuehler added 1 commit 2026-08-06 17:31:40 +00:00

fix(ci/nix): the flake job could never start its container ... e78906fb06

The nix gate landed in fb707b49 on `nixos/nix:latest`, and it has never run a
single step. That image carries nix and essentially nothing else — including no
`/bin/sleep` — and Gitea's act_runner starts every job container with
`entrypoint=["/bin/sleep", "10800"]`:

  failed to create shim task: OCI runtime create failed: runc create failed:
  unable to start container process: exec: "/bin/sleep":
  stat /bin/sleep: no such file or directory

The expensive part is how it reports: with the container dead, every step is
marked `cancelled` rather than `failed`, which reads exactly like a run that was
superseded by a newer push. Run 15907 looked skipped, not broken.

Switch to `node:22-bookworm` and install Nix in a step:

  * full Debian, so the entrypoint exists and coreutils are present;
  * a real node, so `actions/checkout` works with no pre-checkout install dance
    (the reason flatpak.yml's fedora job installs node before its checkout);
  * audit.yml already pulls this image on this fleet, so it is known to resolve.

Nix comes from the Determinate installer with `--init none` — the container mode:
no systemd, no daemon. That distribution is also what the hand-verification Nix
box (.21) runs, so CI and it stay on the same Nix.

MEASURED in a real `node:22-bookworm` container rather than assumed, since the
last version of this file shipped on an untested assumption and cost a red run:

  * `/bin/sleep` present — the entrypoint failure is gone;
  * the installer completes and `nix --version` runs from the absolute path;
  * `nix build` of a trivial derivation SUCCEEDS, `nix store ping` reports
    `Store URL: local, Trusted: 1`, and flakes are enabled.

That last one is the trap this change also pins. `--init none` runs no daemon,
but the installer still writes a profile script exporting `NIX_REMOTE=daemon`;
anything sourcing it (any `-l` login shell) then dies on "cannot connect to
socket at '/nix/var/nix/daemon-socket/socket'". It is why the installer's own
self-test fails, harmlessly, in the middle of this step's log. The steps here
never source that profile — they invoke `$NIX` by absolute path — but `NIX_REMOTE`
is now pinned empty at job level so a later step cannot reintroduce it.

Also records `df -h` before the build: this fleet ran a runner out of disk today
(ci.yml's `web` job died with "no space left on device" mid-`bun install`), a Nix
build is the heaviest thing that would run here, and a future failure should be
attributable at a glance rather than guessed.

Still unproven, and only the first green run can settle it: whether `nix flake
check` evaluates this flake cleanly under CI, and whether the runner has the disk
to build punktfunk-web. Both now produce a real diagnostic instead of a container
that never started.

enricobuehler merged commit 214e846984 into main 2026-08-06 17:45:36 +00:00

enricobuehler deleted branch worktree-nix-flake-container-fix 2026-08-06 17:45:37 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX enricobuehler (Enrico Bühler) kszlim nannku neoney sassycrown

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#77