forked from unom/punktfunk
enricobuehler
6f4613e146
scripts/ci/retry.sh already wraps every single-shot network call in CI, for the
reason documented there: the runner box runs many jobs in parallel and its
network sheds packets under that load. One of the largest fetches in this
workspace was never wrappable that way - skia-bindings pulls ~19 MB of prebuilt
Skia per target from INSIDE its build script, with a bare 'curl -sS -f -L' and
no retry (build_support/binary_cache/utils.rs).
Measured on main 2026-08-22, android job:
DOWNLOAD AND INSTALL FAILED: curl error code: "18"
curl stderr: "curl: (18) end of response with 17054400 bytes missing"
2 MB of 19,057,024 arrived before git.unom.io closed the connection; the same
asset pulls fine from a dev box. skia-bindings then swallowed it - its
try_prepare_download falls through to STARTING A FULL BUILD, a from-source Skia
build the CI containers carry no deps for - so the job surfaced as
'Clippy (Android target) failed' with a Gradle stack trace and the real cause
1,800 lines above it.
* A retrying curl shim first on PATH is the only lever that reaches inside a
build script, and the cheapest correct one: skia-bindings already passes
'-C -' and caches the part-file under OUT_DIR/.cache, so a retry CONTINUES
the truncated transfer rather than restarting it. --retry-all-errors is
load-bearing: a truncated transfer is not an HTTP status, so plain --retry
would let error 18 through.
* Wired into android.yml and both ci.yml rust jobs - pf-console-ui pulls
skia-safe too, so ci/rust downloads Skia on any target-cache miss.
* The rule android.yml's env block states in prose ('Every ABI's log must show
DOWNLOAD AND INSTALL SUCCEEDED') is now a gate that fails the job on
STARTING A FULL BUILD, so a dropped prebuilt can never masquerade as a lint
failure again.
63 lines
3.1 KiB
Bash
Executable File
63 lines
3.1 KiB
Bash
Executable File
#!/bin/sh
|
|
# Put a retrying `curl` first on PATH for the rest of the job.
|
|
#
|
|
# WHY THIS EXISTS: `scripts/ci/retry.sh` already wraps every single-shot network command in CI,
|
|
# for the reason documented there — the runner box runs many jobs in parallel and its network
|
|
# drops packets under that load. But one of the biggest fetches in this workspace is NOT ours to
|
|
# wrap: skia-bindings downloads ~19 MB of prebuilt Skia per target from inside its build script,
|
|
# with a bare `curl -sS -f -L` and no retry at all (build_support/binary_cache/utils.rs).
|
|
#
|
|
# When that transfer truncates the job does not fail with a network error. skia-bindings'
|
|
# `try_prepare_download` swallows it, prints `DOWNLOAD AND INSTALL FAILED`, and falls through to
|
|
# `STARTING A FULL BUILD` — a from-source Skia build that the CI containers carry no deps for.
|
|
# What the operator sees is a Gradle stack trace under "Clippy (Android target)" with the real
|
|
# cause 1,800 lines up. Measured on main 2026-08-22:
|
|
#
|
|
# DOWNLOAD AND INSTALL FAILED: curl error code: "18"
|
|
# curl stderr: "curl: (18) end of response with 17054400 bytes missing"
|
|
#
|
|
# (19,057,024 bytes on the wire; it got 2 MB before git.unom.io closed the connection. The same
|
|
# asset pulls fine from a dev box, so this is the load-shedding retry.sh was written for.)
|
|
#
|
|
# A shim is the only lever that reaches inside a build script. It is also the cheapest correct
|
|
# one: skia-bindings already passes `-C -` (resume) and caches the part-file under
|
|
# OUT_DIR/.cache, so a retry CONTINUES the truncated transfer instead of restarting it.
|
|
#
|
|
# Applies to every curl in the job, which is what we want — the workspace's other build-script
|
|
# fetches are single-shot too.
|
|
#
|
|
# POSIX sh on purpose: Gitea's act_runner executes a step's `run:` under `sh -e` (dash) inside
|
|
# the Linux job containers — see the shader-gate note in ci.yml for what assuming bash cost.
|
|
#
|
|
# Usage: sh scripts/ci/install-retrying-curl.sh
|
|
set -e
|
|
|
|
# Resolve the REAL curl before the shim is on PATH, and bake the absolute path into the shim —
|
|
# a shim that re-resolves `curl` by name would exec itself.
|
|
real_curl=$(command -v curl || true)
|
|
if [ -z "$real_curl" ]; then
|
|
echo "::warning::no curl on PATH — skipping the retrying-curl shim"
|
|
exit 0
|
|
fi
|
|
|
|
# RUNNER_TEMP (not /usr/local/bin): the job containers run as root but the macOS runner is a
|
|
# persistent host where a system dir is neither writable nor ours to litter.
|
|
shim_dir="${RUNNER_TEMP:-/tmp}/pf-retrying-curl"
|
|
mkdir -p "$shim_dir"
|
|
|
|
# --retry-all-errors is what makes this cover error 18: a truncated transfer is a *transfer*
|
|
# failure, not an HTTP status, so plain --retry (which only retries transient HTTP codes and
|
|
# connection errors) would let it through. Needs curl >= 7.71; the CI images are well past it.
|
|
cat > "$shim_dir/curl" <<EOF
|
|
#!/bin/sh
|
|
exec $real_curl --retry 5 --retry-delay 3 --retry-all-errors "\$@"
|
|
EOF
|
|
chmod +x "$shim_dir/curl"
|
|
|
|
if [ -n "${GITHUB_PATH:-}" ]; then
|
|
echo "$shim_dir" >> "$GITHUB_PATH"
|
|
echo "retrying curl installed: $shim_dir/curl -> $real_curl"
|
|
else
|
|
echo "::warning::GITHUB_PATH unset — shim written to $shim_dir but not on PATH"
|
|
fi
|