Files
punktfunk/scripts/ci/retry.sh
T
enricobuehler 3d6c06bb06
ci / web (push) Successful in 52s
apple / swift (push) Successful in 1m13s
ci / docs-site (push) Successful in 1m8s
decky / build-publish (push) Successful in 18s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 6s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
ci / bench (push) Successful in 6m13s
windows-host / package (push) Successful in 7m50s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 2m0s
docker / deploy-docs (push) Successful in 1m16s
arch / build-publish (push) Successful in 11m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 2m19s
release / apple (push) Successful in 11m7s
android / android (push) Successful in 13m1s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m2s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m18s
deb / build-publish (push) Successful in 14m52s
apple / screenshots (push) Successful in 5m41s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m13s
ci / rust (push) Successful in 22m12s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m18s
flatpak / build-publish (push) Successful in 5m18s
fix(ci): flatpak job rides out the runner's network flake — retry every single-shot fetch
The flatpak run dies most pushes at `flatpak remote-add` with an instant
"[6] Could not resolve hostname" (runs 8755/8702/8689), 25ms after dnf pulled
329 packages fine — and deploy-docs separately hits "dial tcp: i/o timeout"
to unom-1 (8716/8679). The busy runner drops UDP DNS + TCP dials under
parallel-job load; tools with built-in retries (dnf) ride it out, single-shot
fetches killed the whole 2h build slot.

* scripts/ci/retry.sh: linear-backoff wrapper (5 tries, attempt*10s); stdout
  passes through untouched so $(…) capture works.
* Tooling: retry the flathub remote-add.
* Build split: a retried prefetch phase (--install-deps-only, then
  --download-only for every crate in cargo-sources.json) warms the state dir;
  the long compile then runs with no network left to flake on.
* Seed step: probe (retried) whether the server repo exists — ONLY first
  publish may continue fresh. The old blanket `rsync || warn` swallowed
  transient failures too, producing the single-branch summary that clobbers
  the other channel (the exact bug the step's comment documents).
* Deploy: retry the idempotent ssh/rsync calls to unom-1.
* raw.githubusercontent curl: --retry 5 --retry-all-errors.

docker.yml's deploy-docs has the same disease but sits on drone-ssh; left
for a follow-up. Real root cause is runner-host networking (conntrack /
upstream DNS under burst load) — retries make the jobs indifferent to it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 10:20:40 +02:00

28 lines
1.3 KiB
Bash

# shellcheck shell=bash
# Run a command, retrying with linear backoff: bash scripts/ci/retry.sh <attempts> <cmd> [args…]
#
# WHY THIS EXISTS: the runner box executes many jobs in parallel and its network drops
# packets under that load — observed as instant "[6] Could not resolve hostname" from
# flatpak/curl (dropped UDP DNS) and "dial tcp: i/o timeout" from the deploy steps
# (dropped TCP SYNs to unom-1), each seconds after other network calls in the same job
# succeeded. Tools with built-in mirror retries (dnf) ride it out; single-shot fetches
# (flatpak remote-add, rsync, ssh) killed the whole heavy flatpak build on one dropped
# packet. Wrap every single-shot network command in CI with this instead.
#
# Backoff is attempt*10s (10/20/30/…), so 5 attempts spread over ~1.5 min — long enough
# to outlive a load burst, short enough not to eat the job timeout. The wrapped command's
# stdout passes through untouched (safe for $(…) capture); retry chatter goes to stderr.
set -u
attempts="$1"; shift
rc=1
for i in $(seq 1 "$attempts"); do
"$@" && exit 0
rc=$?
[ "$i" -eq "$attempts" ] && break
echo "::warning::attempt $i/$attempts failed (rc=$rc): $* — retrying in $((i * 10))s" >&2
sleep $((i * 10))
done
echo "::error::all $attempts attempts failed (rc=$rc): $*" >&2
exit "$rc"