fix(ci/runner): the disk guard had been dead code, and polled too slowly to matter
arch has been failing with `as: BFD (GNU Binutils) 2.47 assertion fail`, which reads like a toolchain regression and is not one. The line above it is `can't write 10 bytes to section .text._ZN6Vulkan...: 'No space left on device'` — the assembler handling ENOSPC badly. Both recent arch failures are the runner filling its disk, and by the time anyone looks, df reports 37% used. Three things were wrong with the hygiene that was supposed to prevent this. The cache cap and the burst-clear were dead code. They looked up the runner as `docker ps -f name=gitea-runner-runner`, which matches zero containers now that the replicas are `gitea-runner-fleet-runner-N-1`, so $RUNNER was always empty and both branches were skipped. The store also moved: the fleet runs a standalone cache-server bind-mounting a HOST directory, so no docker exec is needed at all. The routine prune reclaimed nothing. `--filter until=6h` on a runner that rebuilds its CI images every push means every image is younger than the window — measured: 0B reclaimed while docker system df reported 22.76 GB reclaimable. Now until=2h. The burst guard never fired. It polled every 30 minutes for >=80% used, but three concurrent Rust builds fill the disk and drain it again well inside that window, so the poll kept landing on a healthy df. Now every 10 minutes, and it triggers on a free-space FLOOR too — 80% of 123 G still leaves only ~25 G, which three jobs swallow before the next poll. Deployed to home-runner-1 and exercised: shellcheck clean, timer active on the new interval, one run reclaimed 551 MB. Honest limit: this improves the odds, it does not fix the cause. The remaining 22 GB of idle images are the fedora-rpm bases the next run wants back, so there is no free headroom to reclaim — three replicas building this workspace share one 123 G disk. The lever is capacity (grow the LXC) or concurrency (drop to two replicas), and that is a judgement call, not a script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,13 +1,19 @@
|
||||
# Runs docker-prune.service every 30 min. The runner is hammered with build bursts that can refill
|
||||
# the disk fast (and the actcache cap needs to react well within an hour), so 30 min beats hourly.
|
||||
# Runs docker-prune.service every 10 min. It was every 30, and that is how the burst guard never
|
||||
# fired once: three concurrent Rust builds fill the disk and drain it again well inside a 30-minute
|
||||
# window, so the poll kept landing on a healthy `df` while jobs died of ENOSPC in between. (An arch
|
||||
# build failed with `as: BFD assertion fail` — assembler noise for "can't write, No space left on
|
||||
# device" — and by the time anyone looked, the disk was back to 37% used.)
|
||||
#
|
||||
# Ten minutes is mitigation, not a fix: the real levers are runner capacity (three replicas sharing
|
||||
# one 123 G disk) or a bigger disk. RandomizedDelaySec stays low so the guard is prompt.
|
||||
# Persistent=true catches up after downtime. Install: see the header of docker-prune.service.
|
||||
|
||||
[Unit]
|
||||
Description=Run docker-prune every 30 min (CI runner disk hygiene + actcache cap + burst guard)
|
||||
Description=Run docker-prune every 10 min (CI runner disk hygiene + cache cap + burst guard)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*:0/30
|
||||
RandomizedDelaySec=120
|
||||
OnCalendar=*:0/10
|
||||
RandomizedDelaySec=30
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
|
||||
Reference in New Issue
Block a user