fix(ci/runner): the disk guard has to fire before the disk is tight, not after
ci / web (push) Successful in 57s
ci / docs-site (push) Successful in 2m23s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 21s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 14s
docker / build-push (ci, ci/rust-ci-noble.Dockerfile, punktfunk-rust-ci-noble) (push) Successful in 15s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 19s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 57s
ci / bench (push) Successful in 6m45s
ci / rust-arm64 (push) Successful in 10m32s
ci / rust (push) Failing after 10m39s
decky / build-publish (push) Successful in 38s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 22s
docker / build-push-arm64cross (push) Successful in 23s
android / android (push) Successful in 12m39s
docker / deploy-docs (push) Successful in 1m18s
apple / swift (push) Successful in 4m56s
arch / build-publish (push) Successful in 20m44s
apple / screenshots (push) Successful in 22m0s
deb / build-publish-host (push) Successful in 10m47s
deb / build-publish-client-arm64 (push) Successful in 7m53s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Failing after 7m27s
deb / build-publish (push) Failing after 6m6s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Canceled after 11m40s

Measured over six hours after the last change: zero burst clears fired, while deb
still died of ENOSPC between polls. Both 30 min and 10 min lost the same race —
three concurrent Rust builds fill the disk and drain it again inside the interval,
so every poll landed on a healthy df and the guard concluded all was well.

Two changes, both aimed at that gap rather than at the symptom. The interval goes
to 2 min: the script is a few docker calls and no-ops in about a second, so
sampling five times more often costs nothing worth counting. And MIN_FREE_GB goes
45 -> 60, because the clear only reclaims idle images (~18 G measured) while three
jobs can eat the remainder inside one interval — a guard that waits for "tight"
has already lost. It has to act while there is still room to act in.

This narrows the window; it does not close it. With a 43 G image baseline on a
172 G disk, three concurrent heavy jobs are working inside ~129 G and can exceed
it. Sub-interval spikes are a polling problem, and the honest fixes are fewer
concurrent replicas or more disk.

Deployed to home-runner-1; deployed md5 matches the repo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-29 18:58:32 +02:00
co-authored by Claude Opus 5
parent 7fc2775fad
commit ab6d7727d3
2 changed files with 18 additions and 9 deletions
+11 -8
View File
@@ -1,8 +1,11 @@
# Runs docker-prune.service every 10 min. It was every 30, and that is how the burst guard never
# fired once: three concurrent Rust builds fill the disk and drain it again well inside a 30-minute
# window, so the poll kept landing on a healthy `df` while jobs died of ENOSPC in between. (An arch
# build failed with `as: BFD assertion fail` — assembler noise for "can't write, No space left on
# device" — and by the time anyone looked, the disk was back to 37% used.)
# Runs docker-prune.service every 2 min. It was 30, then 10, and BOTH still lost the race: measured
# over six hours on 2026-07-29 the burst guard fired zero times while deb died of ENOSPC between
# polls, because three concurrent Rust builds fill the disk and drain it again inside the interval.
# (The arch symptom is `as: BFD assertion fail` — assembler noise for "can't write, No space left on
# device" — and by the time anyone looks, df is back under 40% used.)
#
# Every 2 min is affordable: the script is a handful of docker calls and no-ops in about a second
# when there is nothing to reclaim.
#
# Ten minutes is a backstop, not the fix. The fix was capacity: the LXC rootfs went 123 G -> 175 G
# on 2026-07-29 (`pct resize 116 rootfs +50G` on home-node-1), which is what actually gives three
@@ -10,11 +13,11 @@
# Persistent=true catches up after downtime. Install: see the header of docker-prune.service.
[Unit]
Description=Run docker-prune every 10 min (CI runner disk hygiene + cache cap + burst guard)
Description=Run docker-prune every 2 min (CI runner disk hygiene + cache cap + burst guard)
[Timer]
OnCalendar=*:0/10
RandomizedDelaySec=30
OnCalendar=*:0/2
RandomizedDelaySec=10
Persistent=true
[Install]