5 Commits

Include renames

Author	SHA1	Message	Date
enricobuehlerandClaude Opus 5	5f29db3975	fix(ci/runner): grow the runner disk, and simplify the burst condition ... The capacity lever, taken: home-runner-1's LXC rootfs went 123 G -> 175 G (`pct resize 116 rootfs +50G` on home-node-1, online, no downtime, thin pool had ~607 G spare). Free space went 77 G -> 124 G, which is the part that actually gives three concurrent Rust builds room; the 10-minute prune is now a backstop rather than the only thing standing between a push-storm and ENOSPC. MIN_FREE_GB stays at 45. It is deliberately an absolute floor, not a percentage: what three concurrent target/ dirs need does not change when the disk is resized, but a percentage threshold silently does — 80% meant ~25 G free before and ~35 G now. That is exactly why the percentage alone was the wrong instrument. Also replaces the multi-line `{ …; } || { …; }` burst condition with two flat tests into a flag. Same semantics, and shellcheck parses it — the brace-group form across a line break did not survive an edit to the comment above it. Truth-tabled: quiet at 25%/124G, fires on either signal alone, and stays quiet when df returns nothing rather than treating an empty reading as pressure. Deployed to home-runner-1; deployed md5 matches the repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 13:19:07 +02:00
enricobuehlerandClaude Opus 5	7fc387bcbf	fix(ci/runner): the disk guard had been dead code, and polled too slowly to matter ... arch has been failing with `as: BFD (GNU Binutils) 2.47 assertion fail`, which reads like a toolchain regression and is not one. The line above it is `can't write 10 bytes to section .text._ZN6Vulkan...: 'No space left on device'` — the assembler handling ENOSPC badly. Both recent arch failures are the runner filling its disk, and by the time anyone looks, df reports 37% used. Three things were wrong with the hygiene that was supposed to prevent this. The cache cap and the burst-clear were dead code. They looked up the runner as `docker ps -f name=gitea-runner-runner`, which matches zero containers now that the replicas are `gitea-runner-fleet-runner-N-1`, so $RUNNER was always empty and both branches were skipped. The store also moved: the fleet runs a standalone cache-server bind-mounting a HOST directory, so no docker exec is needed at all. The routine prune reclaimed nothing. `--filter until=6h` on a runner that rebuilds its CI images every push means every image is younger than the window — measured: 0B reclaimed while docker system df reported 22.76 GB reclaimable. Now until=2h. The burst guard never fired. It polled every 30 minutes for >=80% used, but three concurrent Rust builds fill the disk and drain it again well inside that window, so the poll kept landing on a healthy df. Now every 10 minutes, and it triggers on a free-space FLOOR too — 80% of 123 G still leaves only ~25 G, which three jobs swallow before the next poll. Deployed to home-runner-1 and exercised: shellcheck clean, timer active on the new interval, one run reclaimed 551 MB. Honest limit: this improves the odds, it does not fix the cause. The remaining 22 GB of idle images are the fedora-rpm bases the next run wants back, so there is no free headroom to reclaim — three replicas building this workspace share one 123 G disk. The lever is capacity (grow the LXC) or concurrency (drop to two replicas), and that is a judgement call, not a script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 13:13:24 +02:00
enricobuehlerandClaude Opus 4.8	693eedd314	ci(runner): cap the act_runner cache + 30-min prune (fix recurring disk-full) ... The hourly docker-prune could never reclaim the real disk filler: the act_runner cache server's blob store (cache.dir:"" -> /root/.cache/actcache/cache) lives in the long-running runner container's WRITABLE LAYER, which docker prune can't see. It grew to ~66 GB and filled the 125 GB disk on its own. - New docker-prune.sh holds the logic (inline ExecStart= broke under systemd's own $-expansion, which emptied $SZ/$(...) before sh ran them — silently no-oping the burst guard). The unit now just calls the script. - Caps the actcache: clears the blobs once they exceed ~20 GB (act_runner repopulates; keys are content-hashed, so only stale entries drop). - Burst guard lowered 85%->80% and now also clears the actcache. - Timer hourly -> every 30 min; image/cache `until` 12h -> 6h. Live: cleared 66 GB on home-runner-1 (93% -> 20%), deployed + verified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-19 07:53:08 +00:00
enricobuehlerandClaude Opus 4.8	d0f6c02349	ci: bust the re-poisoned cargo cache (v3) + burst-guard the runner prune ... This session's push storm refilled the runner to 100% WITHIN the prune timer's 24h window (it only trims >24h), so a build hit ENOSPC and actions/cache saved a truncated target/ -> `error[E0463]: can't find crate for shlex` in ci.yml's clippy. Two fixes: - Bump cargo-target-v2- -> v3- in ci.yml + deb.yml so the poisoned tarball is bypassed (a suffix bump can't — restore-keys falls back to the old prefix; same as the v1->v2 fix). - Harden scripts/ci/docker-prune: run HOURLY (was 6h) with a burst guard — if the disk is still >85% after the normal until=12h trim, prune ALL idle images + build cache (in-use protected). A fast push-burst can fill 99 GB inside any time window, so the disk-pressure trigger, not the age filter, is the real backstop. Applied live on home-runner-1 (reclaimed 95%->66%) and checked in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 14:25:40 +00:00
enricobuehlerandClaude Opus 4.8	84fff71a6d	ci: bound runner disk + bust the disk-full-corrupted cargo target cache ... The self-hosted runner filled its disk (95%, builds failing on ENOSPC): every CI push builds a sha-<commit>-tagged Docker image per pipeline, and since those tags are never dangling a plain `docker image prune` skips them — they piled up to 589 images / ~85 GB plus 18 GB of build cache. Two parts: - scripts/ci/docker-prune.{service,timer}: a host-level systemd timer (every 6h, Persistent) that prunes images/build-cache/containers older than 24h — in-use images stay protected. Checked in (the runner is hand-provisioned and shared across orgs) and already installed live; reclaimed 89 GB -> 39 GB (95% -> 42%). - ci.yml / deb.yml: bump the `cargo-target-<rustc>-*` cache key to `-v2-`. The disk-full build let actions/cache save a truncated target/ (a dep's .rmeta went missing -> "error[E0463]: can't find crate for pem_rfc7468" while compiling der). A suffix bump is useless here — restore-keys would fall back to the poisoned prefix — so the prefix is versioned to force one clean rebuild. cargo-home is untouched (sources were intact; the failure was a missing build artifact). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 09:10:56 +00:00