Measured over six hours after the last change: zero burst clears fired, while deb
still died of ENOSPC between polls. Both 30 min and 10 min lost the same race —
three concurrent Rust builds fill the disk and drain it again inside the interval,
so every poll landed on a healthy df and the guard concluded all was well.
Two changes, both aimed at that gap rather than at the symptom. The interval goes
to 2 min: the script is a few docker calls and no-ops in about a second, so
sampling five times more often costs nothing worth counting. And MIN_FREE_GB goes
45 -> 60, because the clear only reclaims idle images (~18 G measured) while three
jobs can eat the remainder inside one interval — a guard that waits for "tight"
has already lost. It has to act while there is still room to act in.
This narrows the window; it does not close it. With a 43 G image baseline on a
172 G disk, three concurrent heavy jobs are working inside ~129 G and can exceed
it. Sub-interval spikes are a polling problem, and the honest fixes are fewer
concurrent replicas or more disk.
Deployed to home-runner-1; deployed md5 matches the repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The capacity lever, taken: home-runner-1's LXC rootfs went 123 G -> 175 G
(`pct resize 116 rootfs +50G` on home-node-1, online, no downtime, thin pool had
~607 G spare). Free space went 77 G -> 124 G, which is the part that actually
gives three concurrent Rust builds room; the 10-minute prune is now a backstop
rather than the only thing standing between a push-storm and ENOSPC.
MIN_FREE_GB stays at 45. It is deliberately an absolute floor, not a percentage:
what three concurrent target/ dirs need does not change when the disk is resized,
but a percentage threshold silently does — 80% meant ~25 G free before and ~35 G
now. That is exactly why the percentage alone was the wrong instrument.
Also replaces the multi-line `{ …; } || { …; }` burst condition with two flat
tests into a flag. Same semantics, and shellcheck parses it — the brace-group
form across a line break did not survive an edit to the comment above it.
Truth-tabled: quiet at 25%/124G, fires on either signal alone, and stays quiet
when df returns nothing rather than treating an empty reading as pressure.
Deployed to home-runner-1; deployed md5 matches the repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
arch has been failing with `as: BFD (GNU Binutils) 2.47 assertion fail`, which
reads like a toolchain regression and is not one. The line above it is `can't
write 10 bytes to section .text._ZN6Vulkan...: 'No space left on device'` — the
assembler handling ENOSPC badly. Both recent arch failures are the runner filling
its disk, and by the time anyone looks, df reports 37% used.
Three things were wrong with the hygiene that was supposed to prevent this.
The cache cap and the burst-clear were dead code. They looked up the runner as
`docker ps -f name=gitea-runner-runner`, which matches zero containers now that
the replicas are `gitea-runner-fleet-runner-N-1`, so $RUNNER was always empty and
both branches were skipped. The store also moved: the fleet runs a standalone
cache-server bind-mounting a HOST directory, so no docker exec is needed at all.
The routine prune reclaimed nothing. `--filter until=6h` on a runner that rebuilds
its CI images every push means every image is younger than the window — measured:
0B reclaimed while docker system df reported 22.76 GB reclaimable. Now until=2h.
The burst guard never fired. It polled every 30 minutes for >=80% used, but three
concurrent Rust builds fill the disk and drain it again well inside that window,
so the poll kept landing on a healthy df. Now every 10 minutes, and it triggers on
a free-space FLOOR too — 80% of 123 G still leaves only ~25 G, which three jobs
swallow before the next poll.
Deployed to home-runner-1 and exercised: shellcheck clean, timer active on the new
interval, one run reclaimed 551 MB. Honest limit: this improves the odds, it does
not fix the cause. The remaining 22 GB of idle images are the fedora-rpm bases the
next run wants back, so there is no free headroom to reclaim — three replicas
building this workspace share one 123 G disk. The lever is capacity (grow the LXC)
or concurrency (drop to two replicas), and that is a judgement call, not a script.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly docker-prune could never reclaim the real disk filler: the act_runner
cache server's blob store (cache.dir:"" -> /root/.cache/actcache/cache) lives in
the long-running runner container's WRITABLE LAYER, which docker prune can't see.
It grew to ~66 GB and filled the 125 GB disk on its own.
- New docker-prune.sh holds the logic (inline ExecStart= broke under systemd's
own $-expansion, which emptied $SZ/$(...) before sh ran them — silently no-oping
the burst guard). The unit now just calls the script.
- Caps the actcache: clears the blobs once they exceed ~20 GB (act_runner
repopulates; keys are content-hashed, so only stale entries drop).
- Burst guard lowered 85%->80% and now also clears the actcache.
- Timer hourly -> every 30 min; image/cache `until` 12h -> 6h.
Live: cleared 66 GB on home-runner-1 (93% -> 20%), deployed + verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This session's push storm refilled the runner to 100% WITHIN the prune timer's 24h window
(it only trims >24h), so a build hit ENOSPC and actions/cache saved a truncated target/ ->
`error[E0463]: can't find crate for shlex` in ci.yml's clippy. Two fixes:
- Bump cargo-target-v2- -> v3- in ci.yml + deb.yml so the poisoned tarball is bypassed (a
suffix bump can't — restore-keys falls back to the old prefix; same as the v1->v2 fix).
- Harden scripts/ci/docker-prune: run HOURLY (was 6h) with a burst guard — if the disk is
still >85% after the normal until=12h trim, prune ALL idle images + build cache (in-use
protected). A fast push-burst can fill 99 GB inside any time window, so the disk-pressure
trigger, not the age filter, is the real backstop. Applied live on home-runner-1 (reclaimed
95%->66%) and checked in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-hosted runner filled its disk (95%, builds failing on ENOSPC): every CI
push builds a sha-<commit>-tagged Docker image per pipeline, and since those tags
are never dangling a plain `docker image prune` skips them — they piled up to 589
images / ~85 GB plus 18 GB of build cache. Two parts:
- scripts/ci/docker-prune.{service,timer}: a host-level systemd timer (every 6h,
Persistent) that prunes images/build-cache/containers older than 24h — in-use
images stay protected. Checked in (the runner is hand-provisioned and shared
across orgs) and already installed live; reclaimed 89 GB -> 39 GB (95% -> 42%).
- ci.yml / deb.yml: bump the `cargo-target-<rustc>-*` cache key to `-v2-`. The
disk-full build let actions/cache save a truncated target/ (a dep's .rmeta went
missing -> "error[E0463]: can't find crate for pem_rfc7468" while compiling der).
A suffix bump is useless here — restore-keys would fall back to the poisoned
prefix — so the prefix is versioned to force one clean rebuild. cargo-home is
untouched (sources were intact; the failure was a missing build artifact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enricobuehlerandClaude Opus 5