fix(ci/runner): grow the runner disk, and simplify the burst condition

The capacity lever, taken: home-runner-1's LXC rootfs went 123 G -> 175 G (`pct resize 116 rootfs +50G` on home-node-1, online, no downtime, thin pool had ~607 G spare). Free space went 77 G -> 124 G, which is the part that actually gives three concurrent Rust builds room; the 10-minute prune is now a backstop rather than the only thing standing between a push-storm and ENOSPC. MIN_FREE_GB stays at 45. It is deliberately an absolute floor, not a percentage: what three concurrent target/ dirs need does not change when the disk is resized, but a percentage threshold silently does — 80% meant ~25 G free before and ~35 G now. That is exactly why the percentage alone was the wrong instrument. Also replaces the multi-line `{ …; } || { …; }` burst condition with two flat tests into a flag. Same semantics, and shellcheck parses it — the brace-group form across a line break did not survive an edit to the comment above it. Truth-tabled: quiet at 25%/124G, fires on either signal alone, and stays quiet when df returns nothing rather than treating an empty reading as pressure. Deployed to home-runner-1; deployed md5 matches the repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:19:07 +02:00
parent 7fc387bcbf
commit 5f29db3975
2 changed files with 15 additions and 7 deletions
@@ -41,13 +41,20 @@ fi

 # 3) Burst guard: a push-storm fills the disk WITHIN one interval — three concurrent Rust builds,
 #    each with a multi-GB target/, on top of a ~40 GB containerd image baseline. Trigger on a free
-#    -space FLOOR as well as a percentage: the percentage alone is the wrong instrument here,
-#    because 80% of 123 G still leaves only ~25 G, which three jobs can swallow before the next
-#    poll. In-use images are protected by the daemon, so this cannot pull the rug from a live job.
+#    -space FLOOR as well as a percentage: the percentage is the wrong instrument on its own, since
+#    what matters is absolute headroom for three concurrent target/ dirs, not a ratio — and the
+#    ratio moves whenever the disk is resized (it went 123 G -> 175 G on 2026-07-29) while the
+#    headroom three jobs need does not. In-use images are protected by the daemon, so a burst clear
+#    cannot pull the rug from a live job.
 PCT=$(df --output=pcent / | tr -dc '0-9')
 FREE_GB=$(df --output=avail -BG / | tr -dc '0-9')
-if { [ -n "$PCT" ] && [ "$PCT" -ge "$BURST_PCT" ]; } ||
-   { [ -n "$FREE_GB" ] && [ "$FREE_GB" -lt "$MIN_FREE_GB" ]; }; then
+# Two flat tests into a flag rather than one multi-line `{ …; } || { …; }` condition: the brace-group
+# form is easy to get subtly wrong across a line break, and this reads as what it is — either signal
+# alone is enough. Empty values (a df that failed) simply leave the flag unset, i.e. no clear.
+BURST=0
+[ -n "$PCT" ] && [ "$PCT" -ge "$BURST_PCT" ] && BURST=1
+[ -n "$FREE_GB" ] && [ "$FREE_GB" -lt "$MIN_FREE_GB" ] && BURST=1
+if [ "$BURST" = 1 ]; then
  echo "disk ${PCT}% used, ${FREE_GB}G free (thresholds ${BURST_PCT}% / ${MIN_FREE_GB}G) — burst clear"
  docker image prune -af   || true
  docker builder prune -af || true
@@ -4,8 +4,9 @@
 # build failed with `as: BFD assertion fail` — assembler noise for "can't write, No space left on
 # device" — and by the time anyone looked, the disk was back to 37% used.)
 #
-# Ten minutes is mitigation, not a fix: the real levers are runner capacity (three replicas sharing
-# one 123 G disk) or a bigger disk. RandomizedDelaySec stays low so the guard is prompt.
+# Ten minutes is a backstop, not the fix. The fix was capacity: the LXC rootfs went 123 G -> 175 G
+# on 2026-07-29 (`pct resize 116 rootfs +50G` on home-node-1), which is what actually gives three
+# concurrent builds room. RandomizedDelaySec stays low so the guard is prompt.
 # Persistent=true catches up after downtime. Install: see the header of docker-prune.service.

 [Unit]