sassycrown/punktfunk
1
0
Fork 0
forked from unom/punktfunk
Code Pull Requests Activity

fix(ci/runner): the disk guard has to fire before the disk is tight, not after

Browse Source 
Measured over six hours after the last change: zero burst clears fired, while deb
still died of ENOSPC between polls. Both 30 min and 10 min lost the same race —
three concurrent Rust builds fill the disk and drain it again inside the interval,
so every poll landed on a healthy df and the guard concluded all was well.

Two changes, both aimed at that gap rather than at the symptom. The interval goes
to 2 min: the script is a few docker calls and no-ops in about a second, so
sampling five times more often costs nothing worth counting. And MIN_FREE_GB goes
45 -> 60, because the clear only reclaims idle images (~18 G measured) while three
jobs can eat the remainder inside one interval — a guard that waits for "tight"
has already lost. It has to act while there is still room to act in.

This narrows the window; it does not close it. With a 43 G image baseline on a
172 G disk, three concurrent heavy jobs are working inside ~129 G and can exceed
it. Sub-interval spikes are a polling problem, and the honest fixes are fewer
concurrent replicas or more disk.

Deployed to home-runner-1; deployed md5 matches the repo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
enricobuehler
2026-07-29 18:58:32 +02:00
co-authored by Claude Opus 5
parent 7fc2775fad
commit ab6d7727d3
2 changed files with 18 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/ci/docker-prune.timer
+11 -8
View File
@@ -1,8 +1,11 @@
# Runs docker-prune.service every 10 min. It was every 30, and that is how the burst guard never
# fired once: three concurrent Rust builds fill the disk and drain it again well inside a 30-minute
# window, so the poll kept landing on a healthy `df` while jobs died of ENOSPC in between. (An arch
# build failed with `as: BFD assertion fail` — assembler noise for "can't write, No space left on
# device" — and by the time anyone looked, the disk was back to 37% used.)
# Runs docker-prune.service every 2 min. It was 30, then 10, and BOTH still lost the race: measured
# over six hours on 2026-07-29 the burst guard fired zero times while deb died of ENOSPC between
# polls, because three concurrent Rust builds fill the disk and drain it again inside the interval.
# (The arch symptom is `as: BFD assertion fail` — assembler noise for "can't write, No space left on
# device" — and by the time anyone looks, df is back under 40% used.)
#
# Every 2 min is affordable: the script is a handful of docker calls and no-ops in about a second
# when there is nothing to reclaim.
#
# Ten minutes is a backstop, not the fix. The fix was capacity: the LXC rootfs went 123 G -> 175 G
# on 2026-07-29 (`pct resize 116 rootfs +50G` on home-node-1), which is what actually gives three
@@ -10,11 +13,11 @@
# Persistent=true catches up after downtime. Install: see the header of docker-prune.service.
[Unit]
Description=Run docker-prune every 10 min (CI runner disk hygiene + cache cap + burst guard)
Description=Run docker-prune every 2 min (CI runner disk hygiene + cache cap + burst guard)
[Timer]
OnCalendar=*:0/10
RandomizedDelaySec=30
OnCalendar=*:0/2
RandomizedDelaySec=10
Persistent=true
[Install]