A dropped skia download read as a lint failure, and had no retry to survive on #383

Merged
enricobuehler merged 1 commits from ci-skia-download-retry into main 2026-08-23 08:25:01 +00:00
3 changed files with 98 additions and 1 deletions
+24 -1
View File
@@ -171,6 +171,13 @@ jobs:
- name: Rust Android targets (no-op unless the toolchain pin outran the image)
run: rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android
# Must precede every cargo step below: skia-bindings' ~19 MB prebuilt download runs inside
# a build script with no retry, and a truncated transfer here does not surface as a network
# error — it silently becomes a from-source Skia build that dies in the container. See the
# script for the measured failure.
- name: curl with retries (skia-bindings' prebuilt fetch has none)
run: sh scripts/ci/install-retrying-curl.sh
# Same key namespace as ci.yml/deb.yml ON PURPOSE: identical Cargo.lock, identical
# CARGO_HOME layout (/usr/local/cargo), so the registry/git downloads dedupe with
# the rest of the fleet in the central cache. target/ is deliberately NOT cached
@@ -209,9 +216,25 @@ jobs:
# The task lints arm64-v8a AND armeabi-v7a, and reuses the build task's exact cargo-ndk
# environment — see the long note on `registerCargoNdkClippy` in kit/build.gradle.kts for why
# both pointer widths are load-bearing and why the environment must not be duplicated here.
# The `STARTING A FULL BUILD` check turns the manual rule in this workflow's `env:` block
# ("Every ABI's log must show DOWNLOAD AND INSTALL SUCCEEDED") into something that fails the
# job by itself. Without it a missed prebuilt reads as a Gradle stack trace with the real
# cause ~1,800 lines up — which is exactly how 2026-08-22 spent a week looking like a lint
# failure. This is the first cargo step in the job, so it catches the drop earliest.
#
# No pipefail: the runner is dash. Capture, then decide.
- name: Clippy (Android target, deny warnings)
working-directory: clients/android
run: ./gradlew :kit:cargoNdkClippy --stacktrace
run: |
set -e
rc=0
./gradlew :kit:cargoNdkClippy --stacktrace > /tmp/android-clippy.log 2>&1 || rc=$?
cat /tmp/android-clippy.log
if grep -q "STARTING A FULL BUILD" /tmp/android-clippy.log; then
echo "::error::skia-bindings did not get its prebuilt archive and started building Skia from source — the download was dropped (see DOWNLOAD AND INSTALL FAILED above). This is a fetch failure, not a lint failure."
exit 1
fi
exit $rc
# The kit's JVM unit tests — the pure parsers, migrations and feedback policies. They were
# running nowhere: this workflow only assembled, and android-screenshots.yml runs the :app
+12
View File
@@ -107,6 +107,12 @@ jobs:
# registry/git are download caches, target/ the incremental build. The target key
# carries the rustc version — resolved via `rustc --version` (below) rather than parsed
# from rust-toolchain.toml, so a pin bump there invalidates stale incremental state too.
# `pf-console-ui` pulls skia-safe, so a target-cache miss makes this job download a prebuilt
# Skia from the same no-retry build-script fetch that took the android job out on
# 2026-08-22, over the same load-shedding runner network. Cheap insurance; see the script.
- name: curl with retries (skia-bindings' prebuilt fetch has none)
run: sh scripts/ci/install-retrying-curl.sh
- name: Cache keys
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
- uses: actions/cache@v4
@@ -270,6 +276,12 @@ jobs:
- name: sccache (no-op once the image bakes it)
run: sh scripts/ci/ensure-sccache.sh
# `pf-console-ui` pulls skia-safe, so a target-cache miss makes this job download a prebuilt
# Skia from the same no-retry build-script fetch that took the android job out on
# 2026-08-22, over the same load-shedding runner network. Cheap insurance; see the script.
- name: curl with retries (skia-bindings' prebuilt fetch has none)
run: sh scripts/ci/install-retrying-curl.sh
- name: Cache keys
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
- uses: actions/cache@v4
+62
View File
@@ -0,0 +1,62 @@
#!/bin/sh
# Put a retrying `curl` first on PATH for the rest of the job.
#
# WHY THIS EXISTS: `scripts/ci/retry.sh` already wraps every single-shot network command in CI,
# for the reason documented there — the runner box runs many jobs in parallel and its network
# drops packets under that load. But one of the biggest fetches in this workspace is NOT ours to
# wrap: skia-bindings downloads ~19 MB of prebuilt Skia per target from inside its build script,
# with a bare `curl -sS -f -L` and no retry at all (build_support/binary_cache/utils.rs).
#
# When that transfer truncates the job does not fail with a network error. skia-bindings'
# `try_prepare_download` swallows it, prints `DOWNLOAD AND INSTALL FAILED`, and falls through to
# `STARTING A FULL BUILD` — a from-source Skia build that the CI containers carry no deps for.
# What the operator sees is a Gradle stack trace under "Clippy (Android target)" with the real
# cause 1,800 lines up. Measured on main 2026-08-22:
#
# DOWNLOAD AND INSTALL FAILED: curl error code: "18"
# curl stderr: "curl: (18) end of response with 17054400 bytes missing"
#
# (19,057,024 bytes on the wire; it got 2 MB before git.unom.io closed the connection. The same
# asset pulls fine from a dev box, so this is the load-shedding retry.sh was written for.)
#
# A shim is the only lever that reaches inside a build script. It is also the cheapest correct
# one: skia-bindings already passes `-C -` (resume) and caches the part-file under
# OUT_DIR/.cache, so a retry CONTINUES the truncated transfer instead of restarting it.
#
# Applies to every curl in the job, which is what we want — the workspace's other build-script
# fetches are single-shot too.
#
# POSIX sh on purpose: Gitea's act_runner executes a step's `run:` under `sh -e` (dash) inside
# the Linux job containers — see the shader-gate note in ci.yml for what assuming bash cost.
#
# Usage: sh scripts/ci/install-retrying-curl.sh
set -e
# Resolve the REAL curl before the shim is on PATH, and bake the absolute path into the shim —
# a shim that re-resolves `curl` by name would exec itself.
real_curl=$(command -v curl || true)
if [ -z "$real_curl" ]; then
echo "::warning::no curl on PATH — skipping the retrying-curl shim"
exit 0
fi
# RUNNER_TEMP (not /usr/local/bin): the job containers run as root but the macOS runner is a
# persistent host where a system dir is neither writable nor ours to litter.
shim_dir="${RUNNER_TEMP:-/tmp}/pf-retrying-curl"
mkdir -p "$shim_dir"
# --retry-all-errors is what makes this cover error 18: a truncated transfer is a *transfer*
# failure, not an HTTP status, so plain --retry (which only retries transient HTTP codes and
# connection errors) would let it through. Needs curl >= 7.71; the CI images are well past it.
cat > "$shim_dir/curl" <<EOF
#!/bin/sh
exec $real_curl --retry 5 --retry-delay 3 --retry-all-errors "\$@"
EOF
chmod +x "$shim_dir/curl"
if [ -n "${GITHUB_PATH:-}" ]; then
echo "$shim_dir" >> "$GITHUB_PATH"
echo "retrying curl installed: $shim_dir/curl -> $real_curl"
else
echo "::warning::GITHUB_PATH unset — shim written to $shim_dir but not on PATH"
fi