fix(ci/runner): the disk guard has to fire before the disk is tight, not after

Measured over six hours after the last change: zero burst clears fired, while deb still died of ENOSPC between polls. Both 30 min and 10 min lost the same race — three concurrent Rust builds fill the disk and drain it again inside the interval, so every poll landed on a healthy df and the guard concluded all was well. Two changes, both aimed at that gap rather than at the symptom. The interval goes to 2 min: the script is a few docker calls and no-ops in about a second, so sampling five times more often costs nothing worth counting. And MIN_FREE_GB goes 45 -> 60, because the clear only reclaims idle images (~18 G measured) while three jobs can eat the remainder inside one interval — a guard that waits for "tight" has already lost. It has to act while there is still room to act in. This narrows the window; it does not close it. With a 43 G image baseline on a 172 G disk, three concurrent heavy jobs are working inside ~129 G and can exceed it. Sub-interval spikes are a polling problem, and the honest fixes are fewer concurrent replicas or more disk. Deployed to home-runner-1; deployed md5 matches the repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 18:58:32 +02:00
parent 7fc2775fad
commit ab6d7727d3
2 changed files with 18 additions and 9 deletions
@@ -19,7 +19,13 @@ export PATH=/usr/bin:/bin:/usr/local/bin:$PATH
 CACHE_DIR=${CACHE_DIR:-/home/runner/gitea-runner-fleet/cache}
 CAP_MB=${CAP_MB:-20000}                 # clear the cache store once it exceeds ~20 GB
 BURST_PCT=${BURST_PCT:-80}              # full clear once the disk is this % full
-MIN_FREE_GB=${MIN_FREE_GB:-45}          # ...or this little is left, whichever trips first
+MIN_FREE_GB=${MIN_FREE_GB:-60}          # ...or this little is left, whichever trips first.
+                                        # 60, not 45: this has to fire BEFORE the disk is
+                                        # actually tight, because the clear only reclaims idle
+                                        # images (~18 G) while three concurrent jobs can eat
+                                        # the remainder inside one poll interval. Measured
+                                        # 2026-07-29: zero burst clears fired in six hours
+                                        # while deb still died of ENOSPC between polls.

 # 1) Routine: trim aged images / build cache / stopped containers. sha-<commit> tags aren't
 #    dangling, so -a is required. until=2h, not 6h: on a busy day every image is younger than six