arch has been failing with `as: BFD (GNU Binutils) 2.47 assertion fail`, which
reads like a toolchain regression and is not one. The line above it is `can't
write 10 bytes to section .text._ZN6Vulkan...: 'No space left on device'` — the
assembler handling ENOSPC badly. Both recent arch failures are the runner filling
its disk, and by the time anyone looks, df reports 37% used.
Three things were wrong with the hygiene that was supposed to prevent this.
The cache cap and the burst-clear were dead code. They looked up the runner as
`docker ps -f name=gitea-runner-runner`, which matches zero containers now that
the replicas are `gitea-runner-fleet-runner-N-1`, so $RUNNER was always empty and
both branches were skipped. The store also moved: the fleet runs a standalone
cache-server bind-mounting a HOST directory, so no docker exec is needed at all.
The routine prune reclaimed nothing. `--filter until=6h` on a runner that rebuilds
its CI images every push means every image is younger than the window — measured:
0B reclaimed while docker system df reported 22.76 GB reclaimable. Now until=2h.
The burst guard never fired. It polled every 30 minutes for >=80% used, but three
concurrent Rust builds fill the disk and drain it again well inside that window,
so the poll kept landing on a healthy df. Now every 10 minutes, and it triggers on
a free-space FLOOR too — 80% of 123 G still leaves only ~25 G, which three jobs
swallow before the next poll.
Deployed to home-runner-1 and exercised: shellcheck clean, timer active on the new
interval, one run reclaimed 551 MB. Honest limit: this improves the odds, it does
not fix the cause. The remaining 22 GB of idle images are the fedora-rpm bases the
next run wants back, so there is no free headroom to reclaim — three replicas
building this workspace share one 123 G disk. The lever is capacity (grow the LXC)
or concurrency (drop to two replicas), and that is a judgement call, not a script.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enricobuehlerandClaude Opus 5
7fc387bcbf