unom/punktfunk
6
0
Fork 1
Code Issues 3 Pull Requests 3 Actions 1 Packages Projects Releases 45 Wiki Activity

fix(ci): runner hygiene stops eating its own jobs #84

Merged
enricobuehler merged 3 commits from worktree-ci-runner-hygiene into main 2026-08-07 08:37:44 +00:00
Conversation 0 Commits 3 Files Changed 6
+131 -11
enricobuehler commented 2026-08-07 08:11:41 +00:00
Owner
Copy Link
Copy Source

Three fixes out of the 2026-08-07 runner investigation ("most runs fail with no space left"):

1. The 2-minute image prune raced in-flight pulls (scripts/ci/docker-prune.sh)

docker image prune -af --filter until=2h keys on image creation time, so any CI base image (built days ago) with no live container was deleted on every tick — including one a job had just pulled whose container didn't exist yet. Three failures that morning each coincided with a prune run to the second (07:36:29, 07:38:10, 07:34:01): failed to create container: No such image: 192.168.1.58:5010/punktfunk-*-ci:latest, every step cancelled. It also re-pulled 4–7 GB of idle base images every few minutes.

The routine tick now retires only per-SHA app tags older than 2 h (their creation time is the local build time, so the age gate is exact) and then prunes dangling layers only — neither can touch a tagged, just-pulled image. The blanket -a prune survives solely in the near-ENOSPC burst guard.

Also checks in docker-reclaim.{sh,service,timer} — the hourly leak reclaimer that was hand-installed on home-runner-1 only. home-runner-2 went without it and re-accumulated 176 leaked job volumes (~60 GB) until jobs died of ENOSPC on 08-06/08-07 (runs 15986, 16046, 16050). Both hosts now install identical files from the repo.

2. Cache-hit builders jobs failed on a login they never use (.gitea/workflows/docker.yml)

The LAN-registry docker login only serves Push (Reconcile/Tag-for-release use curl -u), but ran unguarded — a hit=true leg on a host with a misconfigured daemon failed at login with nothing to push (run 16013, f44 leg). Now gated like Build/Push.

3. windows-host red on main since 81039581 (crates/punktfunk-host/src/windows/install.rs)

undocumented_unsafe_blocks (deny) flagged three blocks: the SAFETY comment sat outside the is_privileged closure so IsValidSid/EqualSid inside read as undocumented, and from_raw_parts shared a comment that only covered the GetLengthSid line. Comments-only change; the placements were shape-verified against the lint locally, and the windows-host leg on this PR is the real proof.

Deployed alongside (host ops, not in this diff): fixed prune script installed on runner-1, prune+reclaim units installed on runner-2, runner-2 daemon.json :5011 insecure-registry fix.

Three fixes out of the 2026-08-07 runner investigation ("most runs fail with no space left"): **1. The 2-minute image prune raced in-flight pulls** (`scripts/ci/docker-prune.sh`) `docker image prune -af --filter until=2h` keys on image *creation* time, so any CI base image (built days ago) with no live container was deleted on every tick — including one a job had just pulled whose container didn't exist yet. Three failures that morning each coincided with a prune run **to the second** (07:36:29, 07:38:10, 07:34:01): `failed to create container: No such image: 192.168.1.58:5010/punktfunk-*-ci:latest`, every step `cancelled`. It also re-pulled 4–7 GB of idle base images every few minutes. The routine tick now retires only per-SHA app tags older than 2 h (their creation time is the local build time, so the age gate is exact) and then prunes dangling layers only — neither can touch a tagged, just-pulled image. The blanket `-a` prune survives solely in the near-ENOSPC burst guard. Also checks in `docker-reclaim.{sh,service,timer}` — the hourly leak reclaimer that was hand-installed on home-runner-1 only. home-runner-2 went without it and re-accumulated 176 leaked job volumes (~60 GB) until jobs died of ENOSPC on 08-06/08-07 (runs 15986, 16046, 16050). Both hosts now install identical files from the repo. **2. Cache-hit builders jobs failed on a login they never use** (`.gitea/workflows/docker.yml`) The LAN-registry `docker login` only serves `Push` (Reconcile/Tag-for-release use `curl -u`), but ran unguarded — a `hit=true` leg on a host with a misconfigured daemon failed at login with nothing to push (run 16013, f44 leg). Now gated like Build/Push. **3. windows-host red on main since `81039581`** (`crates/punktfunk-host/src/windows/install.rs`) `undocumented_unsafe_blocks` (deny) flagged three blocks: the SAFETY comment sat outside the `is_privileged` closure so `IsValidSid`/`EqualSid` inside read as undocumented, and `from_raw_parts` shared a comment that only covered the `GetLengthSid` line. Comments-only change; the placements were shape-verified against the lint locally, and the windows-host leg on this PR is the real proof. Deployed alongside (host ops, not in this diff): fixed prune script installed on runner-1, prune+reclaim units installed on runner-2, runner-2 daemon.json `:5011` insecure-registry fix.
enricobuehler added 3 commits 2026-08-07 08:11:41 +00:00
fix(ci/prune): the 2-minute image prune stops deleting images mid-pull c3cdee9bf5 
docker image prune -af --filter until=2h keyed on image CREATION time, so a
base image built days ago that merely had no container at that instant was
"aged" — including one a job had just pulled and not yet created. Measured
2026-08-07: three job failures, each coinciding with a prune tick to the
second ("No such image: …punktfunk-rust-ci:latest", every step cancelled),
plus a 4-7 GB re-pull of every idle base image within minutes.

The routine tick now retires only what this host actually accretes — per-SHA
app tags older than 2h (their creation time IS the local build time) — then
sweeps dangling layers, which cannot touch a tagged image. The blanket -a
prune survives only in the near-ENOSPC burst guard, where one re-pull beats
every concurrent job dying.

docker-reclaim.{sh,service,timer} are the hourly leak reclaimer that so far
lived hand-installed on home-runner-1 only; home-runner-2 went without it and
re-accumulated 176 leaked volumes (~60 GB) until jobs died of ENOSPC on
2026-08-06/07. Checked in so both hosts install the same files from here.
fix(ci/docker): a cache-hit builders job stops failing on a login it never uses 39cfb7234c 
The LAN-registry docker login only serves the Push step (Reconcile and
Tag-for-release authenticate via curl -u), but it ran unguarded — so a
hit=true leg landing on a host with a misconfigured docker daemon failed at
login with nothing to push (run 16044/16013 f44 leg). Gate it like Build/Push.
fix(host/windows): the staging-dir SID checks document their unsafe blocks
ci / web (pull_request) Successful in 1m9s
Details
ci / docs-site (pull_request) Successful in 1m13s
Details
apple / swift (pull_request) Successful in 1m34s
Details
ci / bun-nix (pull_request) Successful in 22s
Details
apple / screenshots (pull_request) Skipped
Details
ci / rust-arm64 (pull_request) Successful in 4m4s
Details
android / android (pull_request) Successful in 4m28s
Details
ci / rust (pull_request) Successful in 9m12s
Details
138a1f1b2f 
clippy's undocumented_unsafe_blocks (deny) flagged the three blocks that
81039581 introduced: the SAFETY comment sat outside the closure, so
IsValidSid/EqualSid inside it read as undocumented, and from_raw_parts
shared a comment that only covered the GetLengthSid line above it. Windows
host clippy is the only leg that lints this cfg(windows) code, red since.
enricobuehler merged commit 3608de25ed into main 2026-08-07 08:37:44 +00:00
enricobuehler deleted branch worktree-ci-runner-hygiene 2026-08-07 08:37:45 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
FernandoMX enricobuehler (Enrico Bühler) kszlim nannku neoney sassycrown
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unom/punktfunk#84