From 7fc387bcbf95c533c6e2993a5291b41b22d1e9ff Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Wed, 29 Jul 2026 13:13:24 +0200 Subject: [PATCH] fix(ci/runner): the disk guard had been dead code, and polled too slowly to matter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit arch has been failing with `as: BFD (GNU Binutils) 2.47 assertion fail`, which reads like a toolchain regression and is not one. The line above it is `can't write 10 bytes to section .text._ZN6Vulkan...: 'No space left on device'` — the assembler handling ENOSPC badly. Both recent arch failures are the runner filling its disk, and by the time anyone looks, df reports 37% used. Three things were wrong with the hygiene that was supposed to prevent this. The cache cap and the burst-clear were dead code. They looked up the runner as `docker ps -f name=gitea-runner-runner`, which matches zero containers now that the replicas are `gitea-runner-fleet-runner-N-1`, so $RUNNER was always empty and both branches were skipped. The store also moved: the fleet runs a standalone cache-server bind-mounting a HOST directory, so no docker exec is needed at all. The routine prune reclaimed nothing. `--filter until=6h` on a runner that rebuilds its CI images every push means every image is younger than the window — measured: 0B reclaimed while docker system df reported 22.76 GB reclaimable. Now until=2h. The burst guard never fired. It polled every 30 minutes for >=80% used, but three concurrent Rust builds fill the disk and drain it again well inside that window, so the poll kept landing on a healthy df. Now every 10 minutes, and it triggers on a free-space FLOOR too — 80% of 123 G still leaves only ~25 G, which three jobs swallow before the next poll. Deployed to home-runner-1 and exercised: shellcheck clean, timer active on the new interval, one run reclaimed 551 MB. Honest limit: this improves the odds, it does not fix the cause. The remaining 22 GB of idle images are the fedora-rpm bases the next run wants back, so there is no free headroom to reclaim — three replicas building this workspace share one 123 G disk. The lever is capacity (grow the LXC) or concurrency (drop to two replicas), and that is a judgement call, not a script. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/ci/docker-prune.sh | 52 ++++++++++++++++++++++------------- scripts/ci/docker-prune.timer | 16 +++++++---- 2 files changed, 44 insertions(+), 24 deletions(-) diff --git a/scripts/ci/docker-prune.sh b/scripts/ci/docker-prune.sh index 05b80aaf..4207b22f 100644 --- a/scripts/ci/docker-prune.sh +++ b/scripts/ci/docker-prune.sh @@ -9,34 +9,48 @@ set -u export PATH=/usr/bin:/bin:/usr/local/bin:$PATH -RUNNER=$(docker ps -q -f name=gitea-runner-runner | head -1) -ACTCACHE=/root/.cache/actcache/cache # path INSIDE the runner container (HOME=/root there) -CAP_MB=20000 # clear the actcache once its blob dir exceeds ~20 GB -BURST_PCT=80 # full clear once the disk is this % full +# The cache-server's blob store is a HOST directory: the fleet runs a standalone cache-server +# service (compose.yml) that bind-mounts this path to /data, and every replica points at it over +# HTTP (`external_server`). It used to live inside a runner container's writable layer, which is +# why this reached in with `docker exec` — that is no longer where it is, and the container name it +# looked for (`gitea-runner-runner`) does not exist either now that the replicas are named +# `gitea-runner-fleet-runner-N-1`. Both halves silently did nothing: the filter matched zero +# containers, so the cap and the burst-clear below were dead code. A plain host path needs neither. +CACHE_DIR=${CACHE_DIR:-/home/runner/gitea-runner-fleet/cache} +CAP_MB=${CAP_MB:-20000} # clear the cache store once it exceeds ~20 GB +BURST_PCT=${BURST_PCT:-80} # full clear once the disk is this % full +MIN_FREE_GB=${MIN_FREE_GB:-45} # ...or this little is left, whichever trips first # 1) Routine: trim aged images / build cache / stopped containers. sha- tags aren't -# dangling, so -a is required; until=6h keeps very recent ones for quick re-runs. -docker image prune -af --filter until=6h || true -docker builder prune -af --filter until=6h || true -docker buildx prune -af --filter until=6h || true -docker container prune -f --filter until=6h || true +# dangling, so -a is required. until=2h, not 6h: on a busy day every image is younger than six +# hours, so the filter matched nothing and a run reclaimed 0B while `docker system df` was +# reporting 20+ GB reclaimable. Two hours still protects a re-run of the push being worked on. +docker image prune -af --filter until=2h || true +docker builder prune -af --filter until=2h || true +docker buildx prune -af --filter until=2h || true +docker container prune -f --filter until=2h || true -# 2) Cap the act_runner cache server store (the real disk filler). Clearing the blobs is safe — -# act_runner repopulates it and cache keys are content-hashed, so this only drops stale entries. -if [ -n "$RUNNER" ]; then - SZ=$(docker exec "$RUNNER" du -sm "$ACTCACHE" 2>/dev/null | cut -f1) +# 2) Cap the cache-server store. Clearing the blobs is safe — act_runner repopulates it and cache +# keys are content-hashed, so this only drops stale entries. +if [ -d "$CACHE_DIR" ]; then + SZ=$(du -sm "$CACHE_DIR" 2>/dev/null | cut -f1) if [ -n "${SZ:-}" ] && [ "$SZ" -ge "$CAP_MB" ]; then - docker exec "$RUNNER" sh -c "rm -rf $ACTCACHE/*" && echo "actcache cleared (was ${SZ} MB)" + rm -rf "${CACHE_DIR:?}"/* && echo "cache-server store cleared (was ${SZ} MB)" fi fi -# 3) Burst guard: a push-storm can fill the disk within one interval. Once >=BURST_PCT% full, prune -# ALL idle images/cache AND clear the actcache, regardless of age. In-use images are protected. +# 3) Burst guard: a push-storm fills the disk WITHIN one interval — three concurrent Rust builds, +# each with a multi-GB target/, on top of a ~40 GB containerd image baseline. Trigger on a free +# -space FLOOR as well as a percentage: the percentage alone is the wrong instrument here, +# because 80% of 123 G still leaves only ~25 G, which three jobs can swallow before the next +# poll. In-use images are protected by the daemon, so this cannot pull the rug from a live job. PCT=$(df --output=pcent / | tr -dc '0-9') -if [ -n "$PCT" ] && [ "$PCT" -ge "$BURST_PCT" ]; then - echo "disk ${PCT}% >= ${BURST_PCT}% — burst clear" +FREE_GB=$(df --output=avail -BG / | tr -dc '0-9') +if { [ -n "$PCT" ] && [ "$PCT" -ge "$BURST_PCT" ]; } || + { [ -n "$FREE_GB" ] && [ "$FREE_GB" -lt "$MIN_FREE_GB" ]; }; then + echo "disk ${PCT}% used, ${FREE_GB}G free (thresholds ${BURST_PCT}% / ${MIN_FREE_GB}G) — burst clear" docker image prune -af || true docker builder prune -af || true docker buildx prune -af || true - [ -n "$RUNNER" ] && docker exec "$RUNNER" sh -c "rm -rf $ACTCACHE/*" || true + [ -d "$CACHE_DIR" ] && rm -rf "${CACHE_DIR:?}"/* || true fi diff --git a/scripts/ci/docker-prune.timer b/scripts/ci/docker-prune.timer index dacb7842..ef78378b 100644 --- a/scripts/ci/docker-prune.timer +++ b/scripts/ci/docker-prune.timer @@ -1,13 +1,19 @@ -# Runs docker-prune.service every 30 min. The runner is hammered with build bursts that can refill -# the disk fast (and the actcache cap needs to react well within an hour), so 30 min beats hourly. +# Runs docker-prune.service every 10 min. It was every 30, and that is how the burst guard never +# fired once: three concurrent Rust builds fill the disk and drain it again well inside a 30-minute +# window, so the poll kept landing on a healthy `df` while jobs died of ENOSPC in between. (An arch +# build failed with `as: BFD assertion fail` — assembler noise for "can't write, No space left on +# device" — and by the time anyone looked, the disk was back to 37% used.) +# +# Ten minutes is mitigation, not a fix: the real levers are runner capacity (three replicas sharing +# one 123 G disk) or a bigger disk. RandomizedDelaySec stays low so the guard is prompt. # Persistent=true catches up after downtime. Install: see the header of docker-prune.service. [Unit] -Description=Run docker-prune every 30 min (CI runner disk hygiene + actcache cap + burst guard) +Description=Run docker-prune every 10 min (CI runner disk hygiene + cache cap + burst guard) [Timer] -OnCalendar=*:0/30 -RandomizedDelaySec=120 +OnCalendar=*:0/10 +RandomizedDelaySec=30 Persistent=true [Install]