enricobuehler
3074b30988
apple / swift (push) Successful in 53s
android / android (push) Successful in 10m42s
ci / web (push) Successful in 27s
ci / docs-site (push) Successful in 28s
ci / rust (push) Successful in 11m39s
ci / bench (push) Successful in 4m43s
deb / build-publish (push) Successful in 3m7s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
decky / build-publish (push) Successful in 12s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 7m23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 7m24s
docker / deploy-docs (push) Failing after 8s
The hourly docker-prune could never reclaim the real disk filler: the act_runner cache server's blob store (cache.dir:"" -> /root/.cache/actcache/cache) lives in the long-running runner container's WRITABLE LAYER, which docker prune can't see. It grew to ~66 GB and filled the 125 GB disk on its own. - New docker-prune.sh holds the logic (inline ExecStart= broke under systemd's own $-expansion, which emptied $SZ/$(...) before sh ran them — silently no-oping the burst guard). The unit now just calls the script. - Caps the actcache: clears the blobs once they exceed ~20 GB (act_runner repopulates; keys are content-hashed, so only stale entries drop). - Burst guard lowered 85%->80% and now also clears the actcache. - Timer hourly -> every 30 min; image/cache `until` 12h -> 6h. Live: cleared 66 GB on home-runner-1 (93% -> 20%), deployed + verified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
43 lines
2.3 KiB
Bash
43 lines
2.3 KiB
Bash
#!/usr/bin/env bash
|
|
# CI runner disk hygiene — invoked by docker-prune.service (every 30 min). Lives in a real script
|
|
# rather than inline ExecStart= lines because systemd does its OWN $-expansion on ExecStart and
|
|
# empties shell vars / $(...) before /bin/sh sees them (silently breaking the logic under `|| true`).
|
|
#
|
|
# See docker-prune.service for the full why. The headline: the act_runner cache server's blob store
|
|
# lives INSIDE the long-running runner container's writable layer, where `docker prune` can't reach
|
|
# it — left alone it grows to tens of GB and fills the disk on its own.
|
|
set -u
|
|
export PATH=/usr/bin:/bin:/usr/local/bin:$PATH
|
|
|
|
RUNNER=$(docker ps -q -f name=gitea-runner-runner | head -1)
|
|
ACTCACHE=/root/.cache/actcache/cache # path INSIDE the runner container (HOME=/root there)
|
|
CAP_MB=20000 # clear the actcache once its blob dir exceeds ~20 GB
|
|
BURST_PCT=80 # full clear once the disk is this % full
|
|
|
|
# 1) Routine: trim aged images / build cache / stopped containers. sha-<commit> tags aren't
|
|
# dangling, so -a is required; until=6h keeps very recent ones for quick re-runs.
|
|
docker image prune -af --filter until=6h || true
|
|
docker builder prune -af --filter until=6h || true
|
|
docker buildx prune -af --filter until=6h || true
|
|
docker container prune -f --filter until=6h || true
|
|
|
|
# 2) Cap the act_runner cache server store (the real disk filler). Clearing the blobs is safe —
|
|
# act_runner repopulates it and cache keys are content-hashed, so this only drops stale entries.
|
|
if [ -n "$RUNNER" ]; then
|
|
SZ=$(docker exec "$RUNNER" du -sm "$ACTCACHE" 2>/dev/null | cut -f1)
|
|
if [ -n "${SZ:-}" ] && [ "$SZ" -ge "$CAP_MB" ]; then
|
|
docker exec "$RUNNER" sh -c "rm -rf $ACTCACHE/*" && echo "actcache cleared (was ${SZ} MB)"
|
|
fi
|
|
fi
|
|
|
|
# 3) Burst guard: a push-storm can fill the disk within one interval. Once >=BURST_PCT% full, prune
|
|
# ALL idle images/cache AND clear the actcache, regardless of age. In-use images are protected.
|
|
PCT=$(df --output=pcent / | tr -dc '0-9')
|
|
if [ -n "$PCT" ] && [ "$PCT" -ge "$BURST_PCT" ]; then
|
|
echo "disk ${PCT}% >= ${BURST_PCT}% — burst clear"
|
|
docker image prune -af || true
|
|
docker builder prune -af || true
|
|
docker buildx prune -af || true
|
|
[ -n "$RUNNER" ] && docker exec "$RUNNER" sh -c "rm -rf $ACTCACHE/*" || true
|
|
fi
|