A dropped skia download read as a lint failure, and had no retry to survive on #383
@@ -171,6 +171,13 @@ jobs:
|
||||
- name: Rust Android targets (no-op unless the toolchain pin outran the image)
|
||||
run: rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android
|
||||
|
||||
# Must precede every cargo step below: skia-bindings' ~19 MB prebuilt download runs inside
|
||||
# a build script with no retry, and a truncated transfer here does not surface as a network
|
||||
# error — it silently becomes a from-source Skia build that dies in the container. See the
|
||||
# script for the measured failure.
|
||||
- name: curl with retries (skia-bindings' prebuilt fetch has none)
|
||||
run: sh scripts/ci/install-retrying-curl.sh
|
||||
|
||||
# Same key namespace as ci.yml/deb.yml ON PURPOSE: identical Cargo.lock, identical
|
||||
# CARGO_HOME layout (/usr/local/cargo), so the registry/git downloads dedupe with
|
||||
# the rest of the fleet in the central cache. target/ is deliberately NOT cached
|
||||
@@ -209,9 +216,25 @@ jobs:
|
||||
# The task lints arm64-v8a AND armeabi-v7a, and reuses the build task's exact cargo-ndk
|
||||
# environment — see the long note on `registerCargoNdkClippy` in kit/build.gradle.kts for why
|
||||
# both pointer widths are load-bearing and why the environment must not be duplicated here.
|
||||
# The `STARTING A FULL BUILD` check turns the manual rule in this workflow's `env:` block
|
||||
# ("Every ABI's log must show DOWNLOAD AND INSTALL SUCCEEDED") into something that fails the
|
||||
# job by itself. Without it a missed prebuilt reads as a Gradle stack trace with the real
|
||||
# cause ~1,800 lines up — which is exactly how 2026-08-22 spent a week looking like a lint
|
||||
# failure. This is the first cargo step in the job, so it catches the drop earliest.
|
||||
#
|
||||
# No pipefail: the runner is dash. Capture, then decide.
|
||||
- name: Clippy (Android target, deny warnings)
|
||||
working-directory: clients/android
|
||||
run: ./gradlew :kit:cargoNdkClippy --stacktrace
|
||||
run: |
|
||||
set -e
|
||||
rc=0
|
||||
./gradlew :kit:cargoNdkClippy --stacktrace > /tmp/android-clippy.log 2>&1 || rc=$?
|
||||
cat /tmp/android-clippy.log
|
||||
if grep -q "STARTING A FULL BUILD" /tmp/android-clippy.log; then
|
||||
echo "::error::skia-bindings did not get its prebuilt archive and started building Skia from source — the download was dropped (see DOWNLOAD AND INSTALL FAILED above). This is a fetch failure, not a lint failure."
|
||||
exit 1
|
||||
fi
|
||||
exit $rc
|
||||
|
||||
# The kit's JVM unit tests — the pure parsers, migrations and feedback policies. They were
|
||||
# running nowhere: this workflow only assembled, and android-screenshots.yml runs the :app
|
||||
|
||||
@@ -107,6 +107,12 @@ jobs:
|
||||
# registry/git are download caches, target/ the incremental build. The target key
|
||||
# carries the rustc version — resolved via `rustc --version` (below) rather than parsed
|
||||
# from rust-toolchain.toml, so a pin bump there invalidates stale incremental state too.
|
||||
# `pf-console-ui` pulls skia-safe, so a target-cache miss makes this job download a prebuilt
|
||||
# Skia from the same no-retry build-script fetch that took the android job out on
|
||||
# 2026-08-22, over the same load-shedding runner network. Cheap insurance; see the script.
|
||||
- name: curl with retries (skia-bindings' prebuilt fetch has none)
|
||||
run: sh scripts/ci/install-retrying-curl.sh
|
||||
|
||||
- name: Cache keys
|
||||
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
|
||||
- uses: actions/cache@v4
|
||||
@@ -270,6 +276,12 @@ jobs:
|
||||
- name: sccache (no-op once the image bakes it)
|
||||
run: sh scripts/ci/ensure-sccache.sh
|
||||
|
||||
# `pf-console-ui` pulls skia-safe, so a target-cache miss makes this job download a prebuilt
|
||||
# Skia from the same no-retry build-script fetch that took the android job out on
|
||||
# 2026-08-22, over the same load-shedding runner network. Cheap insurance; see the script.
|
||||
- name: curl with retries (skia-bindings' prebuilt fetch has none)
|
||||
run: sh scripts/ci/install-retrying-curl.sh
|
||||
|
||||
- name: Cache keys
|
||||
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
|
||||
- uses: actions/cache@v4
|
||||
|
||||
Executable
+62
@@ -0,0 +1,62 @@
|
||||
#!/bin/sh
|
||||
# Put a retrying `curl` first on PATH for the rest of the job.
|
||||
#
|
||||
# WHY THIS EXISTS: `scripts/ci/retry.sh` already wraps every single-shot network command in CI,
|
||||
# for the reason documented there — the runner box runs many jobs in parallel and its network
|
||||
# drops packets under that load. But one of the biggest fetches in this workspace is NOT ours to
|
||||
# wrap: skia-bindings downloads ~19 MB of prebuilt Skia per target from inside its build script,
|
||||
# with a bare `curl -sS -f -L` and no retry at all (build_support/binary_cache/utils.rs).
|
||||
#
|
||||
# When that transfer truncates the job does not fail with a network error. skia-bindings'
|
||||
# `try_prepare_download` swallows it, prints `DOWNLOAD AND INSTALL FAILED`, and falls through to
|
||||
# `STARTING A FULL BUILD` — a from-source Skia build that the CI containers carry no deps for.
|
||||
# What the operator sees is a Gradle stack trace under "Clippy (Android target)" with the real
|
||||
# cause 1,800 lines up. Measured on main 2026-08-22:
|
||||
#
|
||||
# DOWNLOAD AND INSTALL FAILED: curl error code: "18"
|
||||
# curl stderr: "curl: (18) end of response with 17054400 bytes missing"
|
||||
#
|
||||
# (19,057,024 bytes on the wire; it got 2 MB before git.unom.io closed the connection. The same
|
||||
# asset pulls fine from a dev box, so this is the load-shedding retry.sh was written for.)
|
||||
#
|
||||
# A shim is the only lever that reaches inside a build script. It is also the cheapest correct
|
||||
# one: skia-bindings already passes `-C -` (resume) and caches the part-file under
|
||||
# OUT_DIR/.cache, so a retry CONTINUES the truncated transfer instead of restarting it.
|
||||
#
|
||||
# Applies to every curl in the job, which is what we want — the workspace's other build-script
|
||||
# fetches are single-shot too.
|
||||
#
|
||||
# POSIX sh on purpose: Gitea's act_runner executes a step's `run:` under `sh -e` (dash) inside
|
||||
# the Linux job containers — see the shader-gate note in ci.yml for what assuming bash cost.
|
||||
#
|
||||
# Usage: sh scripts/ci/install-retrying-curl.sh
|
||||
set -e
|
||||
|
||||
# Resolve the REAL curl before the shim is on PATH, and bake the absolute path into the shim —
|
||||
# a shim that re-resolves `curl` by name would exec itself.
|
||||
real_curl=$(command -v curl || true)
|
||||
if [ -z "$real_curl" ]; then
|
||||
echo "::warning::no curl on PATH — skipping the retrying-curl shim"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# RUNNER_TEMP (not /usr/local/bin): the job containers run as root but the macOS runner is a
|
||||
# persistent host where a system dir is neither writable nor ours to litter.
|
||||
shim_dir="${RUNNER_TEMP:-/tmp}/pf-retrying-curl"
|
||||
mkdir -p "$shim_dir"
|
||||
|
||||
# --retry-all-errors is what makes this cover error 18: a truncated transfer is a *transfer*
|
||||
# failure, not an HTTP status, so plain --retry (which only retries transient HTTP codes and
|
||||
# connection errors) would let it through. Needs curl >= 7.71; the CI images are well past it.
|
||||
cat > "$shim_dir/curl" <<EOF
|
||||
#!/bin/sh
|
||||
exec $real_curl --retry 5 --retry-delay 3 --retry-all-errors "\$@"
|
||||
EOF
|
||||
chmod +x "$shim_dir/curl"
|
||||
|
||||
if [ -n "${GITHUB_PATH:-}" ]; then
|
||||
echo "$shim_dir" >> "$GITHUB_PATH"
|
||||
echo "retrying curl installed: $shim_dir/curl -> $real_curl"
|
||||
else
|
||||
echo "::warning::GITHUB_PATH unset — shim written to $shim_dir but not on PATH"
|
||||
fi
|
||||
Reference in New Issue
Block a user