The bun OOM is a 7 GiB cgroup cap, not a big build — measure it in the log #341
Assignees
FernandoMX
HitFrostbite
enricobuehler (Enrico Bühler)
kszlim
luxus
nannku
neoney
sassycrown
Clear assignees
FernandoMX
HitFrostbite
enricobuehler (Enrico Bühler)
kszlim
luxus
nannku
neoney
sassycrown
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: unom/punktfunk#341
Reference in New Issue
Block a user
The nix job's recurring red —
punktfunk-web> bun run buildSIGKILLed, exit 137 — has now cost more runs than every real bug this week, and it sits before the publish tier, so it also stops the binary cache filling.It is not a memory-hungry build. The job container is capped at 7 GiB.
free(1)and/proc/meminforeport the host inside a container, so every previous look at this showed 125 GiB total / ~48 GiB free and made the SIGKILL look impossible. The cap was invisible. The nitro build peaks right at that 7 GiB boundary, which is exactly why it fails intermittently and why identical commits pass and fail.What this PR does
Adds the memory, cpu and cgroup limit readout to the
Environmentstep. That's it.dfalone can never explain an exit 137, and this one number is what turned a day of guessing into a diagnosis.What it deliberately does not do
An earlier commit on this branch added
--max-jobs 1, on the theory thatpunktfunk-webandpunktfunk-scriptingbundling concurrently exhausted the box. Reverted — the theory is disproven: run 19444 failed identically with it, and a single bun build alone approaches the cap. Leaving it would have meant a confident, wrongMEASUREDcomment in the file, and serialising the publish tier would cost real wall-clock on six derivations including an hour of Rust.Also ruled out: bun is 1.3.13 in both the old and new nixpkgs, so #328's lock bump did not change the bundler. This predates that bump.
The actual fix is outside this repo
The cap is set on the CI box's act_runner (
container.options/ docker--memory), which isn't inunom/punktfunkorunom/infra. Raising it for this fleet is the fix — 7 GiB is simply too small for the console's SSR build.If raising it isn't attractive, the repo-side lever worth trying is
bun --smol run buildforpunktfunk-web(smaller heap, more aggressive GC). I have not measured that, and I'm not going to guess again — the number above should drive whatever comes next.The nix job's recurring red is not a broken bundle, it is memory: punktfunk-web> error: script "build" was terminated by signal SIGKILL (Forced quit) Reason: builder failed with exit code 137 MEASURED 2026-08-19: five runs across main and two branches died exactly there, while other runs on the SAME commits passed — the signature of pressure, not of a defect. nix builds punktfunk-web and punktfunk-scripting concurrently, and each one runs a memory-hungry bundler; that pairing is what the runner cannot afford. `--max-jobs 1` caps how many DERIVATIONS build at once, not the cores inside one, so two bundles that each pin a core simply run in sequence. Applied to the publish tier too, which builds six derivations including those same two. This step sits BEFORE the publish tier, so every one of those failures also stopped the binary cache from filling — it has blocked more runs today than every real bug combined. Also prints memory in the Environment step. An exit 137 cannot be explained by the disk numbers that step used to gather on their own.Reverts the --max-jobs 1 from the previous commit. The hypothesis it encoded — that punktfunk-web and punktfunk-scripting bundling concurrently exhausted the box — is DISPROVEN: run 19444 failed identically with it, and the Environment step it added shows why the premise was wrong: Mem: 125Gi total, 48Gi available Swap: 8.0Gi nproc: 16 overlay 172G, 108G avail Nothing was scarce. Serialising also would have cost real wall-clock on the publish tier, which builds six derivations including an hour of Rust, so it does not stay on an unproven basis. Also ruled out: bun is 1.3.13 in BOTH the old and new nixpkgs, so the lock bump did not change the bundler. What remains is what those numbers cannot see. Inside a container /proc/meminfo and free(1) report the HOST, so a cgroup memory cap is invisible to them — and a cap is the only explanation left for a SIGKILL with 48Gi apparently free. So read it directly: cgroup v2 memory.max, falling back to v1 memory.limit_in_bytes. No fix in this commit, deliberately. Two guesses have already cost a day of red runs; the next change should follow the number.