forked from unom/punktfunk
enricobuehlerandClaude Opus 5
ab6d7727d3
Measured over six hours after the last change: zero burst clears fired, while deb still died of ENOSPC between polls. Both 30 min and 10 min lost the same race — three concurrent Rust builds fill the disk and drain it again inside the interval, so every poll landed on a healthy df and the guard concluded all was well. Two changes, both aimed at that gap rather than at the symptom. The interval goes to 2 min: the script is a few docker calls and no-ops in about a second, so sampling five times more often costs nothing worth counting. And MIN_FREE_GB goes 45 -> 60, because the clear only reclaims idle images (~18 G measured) while three jobs can eat the remainder inside one interval — a guard that waits for "tight" has already lost. It has to act while there is still room to act in. This narrows the window; it does not close it. With a 43 G image baseline on a 172 G disk, three concurrent heavy jobs are working inside ~129 G and can exceed it. Sub-interval spikes are a polling problem, and the honest fixes are fewer concurrent replicas or more disk. Deployed to home-runner-1; deployed md5 matches the repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
25 lines
1.2 KiB
SYSTEMD
25 lines
1.2 KiB
SYSTEMD
# Runs docker-prune.service every 2 min. It was 30, then 10, and BOTH still lost the race: measured
|
|
# over six hours on 2026-07-29 the burst guard fired zero times while deb died of ENOSPC between
|
|
# polls, because three concurrent Rust builds fill the disk and drain it again inside the interval.
|
|
# (The arch symptom is `as: BFD assertion fail` — assembler noise for "can't write, No space left on
|
|
# device" — and by the time anyone looks, df is back under 40% used.)
|
|
#
|
|
# Every 2 min is affordable: the script is a handful of docker calls and no-ops in about a second
|
|
# when there is nothing to reclaim.
|
|
#
|
|
# Ten minutes is a backstop, not the fix. The fix was capacity: the LXC rootfs went 123 G -> 175 G
|
|
# on 2026-07-29 (`pct resize 116 rootfs +50G` on home-node-1), which is what actually gives three
|
|
# concurrent builds room. RandomizedDelaySec stays low so the guard is prompt.
|
|
# Persistent=true catches up after downtime. Install: see the header of docker-prune.service.
|
|
|
|
[Unit]
|
|
Description=Run docker-prune every 2 min (CI runner disk hygiene + cache cap + burst guard)
|
|
|
|
[Timer]
|
|
OnCalendar=*:0/2
|
|
RandomizedDelaySec=10
|
|
Persistent=true
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|