3d6c06bb06
ci / web (push) Successful in 52s
apple / swift (push) Successful in 1m13s
ci / docs-site (push) Successful in 1m8s
decky / build-publish (push) Successful in 18s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 6s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
ci / bench (push) Successful in 6m13s
windows-host / package (push) Successful in 7m50s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 2m0s
docker / deploy-docs (push) Successful in 1m16s
arch / build-publish (push) Successful in 11m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 2m19s
release / apple (push) Successful in 11m7s
android / android (push) Successful in 13m1s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m2s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m18s
deb / build-publish (push) Successful in 14m52s
apple / screenshots (push) Successful in 5m41s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m13s
ci / rust (push) Successful in 22m12s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m18s
flatpak / build-publish (push) Successful in 5m18s
The flatpak run dies most pushes at `flatpak remote-add` with an instant "[6] Could not resolve hostname" (runs 8755/8702/8689), 25ms after dnf pulled 329 packages fine — and deploy-docs separately hits "dial tcp: i/o timeout" to unom-1 (8716/8679). The busy runner drops UDP DNS + TCP dials under parallel-job load; tools with built-in retries (dnf) ride it out, single-shot fetches killed the whole 2h build slot. * scripts/ci/retry.sh: linear-backoff wrapper (5 tries, attempt*10s); stdout passes through untouched so $(…) capture works. * Tooling: retry the flathub remote-add. * Build split: a retried prefetch phase (--install-deps-only, then --download-only for every crate in cargo-sources.json) warms the state dir; the long compile then runs with no network left to flake on. * Seed step: probe (retried) whether the server repo exists — ONLY first publish may continue fresh. The old blanket `rsync || warn` swallowed transient failures too, producing the single-branch summary that clobbers the other channel (the exact bug the step's comment documents). * Deploy: retry the idempotent ssh/rsync calls to unom-1. * raw.githubusercontent curl: --retry 5 --retry-all-errors. docker.yml's deploy-docs has the same disease but sits on drone-ssh; left for a follow-up. Real root cause is runner-host networking (conntrack / upstream DNS under burst load) — retries make the jobs indifferent to it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
28 lines
1.3 KiB
Bash
28 lines
1.3 KiB
Bash
# shellcheck shell=bash
|
|
# Run a command, retrying with linear backoff: bash scripts/ci/retry.sh <attempts> <cmd> [args…]
|
|
#
|
|
# WHY THIS EXISTS: the runner box executes many jobs in parallel and its network drops
|
|
# packets under that load — observed as instant "[6] Could not resolve hostname" from
|
|
# flatpak/curl (dropped UDP DNS) and "dial tcp: i/o timeout" from the deploy steps
|
|
# (dropped TCP SYNs to unom-1), each seconds after other network calls in the same job
|
|
# succeeded. Tools with built-in mirror retries (dnf) ride it out; single-shot fetches
|
|
# (flatpak remote-add, rsync, ssh) killed the whole heavy flatpak build on one dropped
|
|
# packet. Wrap every single-shot network command in CI with this instead.
|
|
#
|
|
# Backoff is attempt*10s (10/20/30/…), so 5 attempts spread over ~1.5 min — long enough
|
|
# to outlive a load burst, short enough not to eat the job timeout. The wrapped command's
|
|
# stdout passes through untouched (safe for $(…) capture); retry chatter goes to stderr.
|
|
set -u
|
|
|
|
attempts="$1"; shift
|
|
rc=1
|
|
for i in $(seq 1 "$attempts"); do
|
|
"$@" && exit 0
|
|
rc=$?
|
|
[ "$i" -eq "$attempts" ] && break
|
|
echo "::warning::attempt $i/$attempts failed (rc=$rc): $* — retrying in $((i * 10))s" >&2
|
|
sleep $((i * 10))
|
|
done
|
|
echo "::error::all $attempts attempts failed (rc=$rc): $*" >&2
|
|
exit "$rc"
|