forked from unom/punktfunk
enricobuehlerandClaude Fable 5
3d6c06bb06
The flatpak run dies most pushes at `flatpak remote-add` with an instant "[6] Could not resolve hostname" (runs 8755/8702/8689), 25ms after dnf pulled 329 packages fine — and deploy-docs separately hits "dial tcp: i/o timeout" to unom-1 (8716/8679). The busy runner drops UDP DNS + TCP dials under parallel-job load; tools with built-in retries (dnf) ride it out, single-shot fetches killed the whole 2h build slot. * scripts/ci/retry.sh: linear-backoff wrapper (5 tries, attempt*10s); stdout passes through untouched so $(…) capture works. * Tooling: retry the flathub remote-add. * Build split: a retried prefetch phase (--install-deps-only, then --download-only for every crate in cargo-sources.json) warms the state dir; the long compile then runs with no network left to flake on. * Seed step: probe (retried) whether the server repo exists — ONLY first publish may continue fresh. The old blanket `rsync || warn` swallowed transient failures too, producing the single-branch summary that clobbers the other channel (the exact bug the step's comment documents). * Deploy: retry the idempotent ssh/rsync calls to unom-1. * raw.githubusercontent curl: --retry 5 --retry-all-errors. docker.yml's deploy-docs has the same disease but sits on drone-ssh; left for a follow-up. Real root cause is runner-host networking (conntrack / upstream DNS under burst load) — retries make the jobs indifferent to it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
28 lines
1.3 KiB
Bash
28 lines
1.3 KiB
Bash
# shellcheck shell=bash
|
|
# Run a command, retrying with linear backoff: bash scripts/ci/retry.sh <attempts> <cmd> [args…]
|
|
#
|
|
# WHY THIS EXISTS: the runner box executes many jobs in parallel and its network drops
|
|
# packets under that load — observed as instant "[6] Could not resolve hostname" from
|
|
# flatpak/curl (dropped UDP DNS) and "dial tcp: i/o timeout" from the deploy steps
|
|
# (dropped TCP SYNs to unom-1), each seconds after other network calls in the same job
|
|
# succeeded. Tools with built-in mirror retries (dnf) ride it out; single-shot fetches
|
|
# (flatpak remote-add, rsync, ssh) killed the whole heavy flatpak build on one dropped
|
|
# packet. Wrap every single-shot network command in CI with this instead.
|
|
#
|
|
# Backoff is attempt*10s (10/20/30/…), so 5 attempts spread over ~1.5 min — long enough
|
|
# to outlive a load burst, short enough not to eat the job timeout. The wrapped command's
|
|
# stdout passes through untouched (safe for $(…) capture); retry chatter goes to stderr.
|
|
set -u
|
|
|
|
attempts="$1"; shift
|
|
rc=1
|
|
for i in $(seq 1 "$attempts"); do
|
|
"$@" && exit 0
|
|
rc=$?
|
|
[ "$i" -eq "$attempts" ] && break
|
|
echo "::warning::attempt $i/$attempts failed (rc=$rc): $* — retrying in $((i * 10))s" >&2
|
|
sleep $((i * 10))
|
|
done
|
|
echo "::error::all $attempts attempts failed (rc=$rc): $*" >&2
|
|
exit "$rc"
|