From 6f4613e146efa205ec013afc4e927444e8270b9e Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Sun, 23 Aug 2026 10:00:21 +0200 Subject: [PATCH] fix(ci): a dropped skia download read as a lint failure, and had no retry to survive on scripts/ci/retry.sh already wraps every single-shot network call in CI, for the reason documented there: the runner box runs many jobs in parallel and its network sheds packets under that load. One of the largest fetches in this workspace was never wrappable that way - skia-bindings pulls ~19 MB of prebuilt Skia per target from INSIDE its build script, with a bare 'curl -sS -f -L' and no retry (build_support/binary_cache/utils.rs). Measured on main 2026-08-22, android job: DOWNLOAD AND INSTALL FAILED: curl error code: "18" curl stderr: "curl: (18) end of response with 17054400 bytes missing" 2 MB of 19,057,024 arrived before git.unom.io closed the connection; the same asset pulls fine from a dev box. skia-bindings then swallowed it - its try_prepare_download falls through to STARTING A FULL BUILD, a from-source Skia build the CI containers carry no deps for - so the job surfaced as 'Clippy (Android target) failed' with a Gradle stack trace and the real cause 1,800 lines above it. * A retrying curl shim first on PATH is the only lever that reaches inside a build script, and the cheapest correct one: skia-bindings already passes '-C -' and caches the part-file under OUT_DIR/.cache, so a retry CONTINUES the truncated transfer rather than restarting it. --retry-all-errors is load-bearing: a truncated transfer is not an HTTP status, so plain --retry would let error 18 through. * Wired into android.yml and both ci.yml rust jobs - pf-console-ui pulls skia-safe too, so ci/rust downloads Skia on any target-cache miss. * The rule android.yml's env block states in prose ('Every ABI's log must show DOWNLOAD AND INSTALL SUCCEEDED') is now a gate that fails the job on STARTING A FULL BUILD, so a dropped prebuilt can never masquerade as a lint failure again. --- .gitea/workflows/android.yml | 25 +++++++++++- .gitea/workflows/ci.yml | 12 ++++++ scripts/ci/install-retrying-curl.sh | 62 +++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 1 deletion(-) create mode 100755 scripts/ci/install-retrying-curl.sh diff --git a/.gitea/workflows/android.yml b/.gitea/workflows/android.yml index d1e1fd184..aae0b3b2e 100644 --- a/.gitea/workflows/android.yml +++ b/.gitea/workflows/android.yml @@ -171,6 +171,13 @@ jobs: - name: Rust Android targets (no-op unless the toolchain pin outran the image) run: rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android + # Must precede every cargo step below: skia-bindings' ~19 MB prebuilt download runs inside + # a build script with no retry, and a truncated transfer here does not surface as a network + # error — it silently becomes a from-source Skia build that dies in the container. See the + # script for the measured failure. + - name: curl with retries (skia-bindings' prebuilt fetch has none) + run: sh scripts/ci/install-retrying-curl.sh + # Same key namespace as ci.yml/deb.yml ON PURPOSE: identical Cargo.lock, identical # CARGO_HOME layout (/usr/local/cargo), so the registry/git downloads dedupe with # the rest of the fleet in the central cache. target/ is deliberately NOT cached @@ -209,9 +216,25 @@ jobs: # The task lints arm64-v8a AND armeabi-v7a, and reuses the build task's exact cargo-ndk # environment — see the long note on `registerCargoNdkClippy` in kit/build.gradle.kts for why # both pointer widths are load-bearing and why the environment must not be duplicated here. + # The `STARTING A FULL BUILD` check turns the manual rule in this workflow's `env:` block + # ("Every ABI's log must show DOWNLOAD AND INSTALL SUCCEEDED") into something that fails the + # job by itself. Without it a missed prebuilt reads as a Gradle stack trace with the real + # cause ~1,800 lines up — which is exactly how 2026-08-22 spent a week looking like a lint + # failure. This is the first cargo step in the job, so it catches the drop earliest. + # + # No pipefail: the runner is dash. Capture, then decide. - name: Clippy (Android target, deny warnings) working-directory: clients/android - run: ./gradlew :kit:cargoNdkClippy --stacktrace + run: | + set -e + rc=0 + ./gradlew :kit:cargoNdkClippy --stacktrace > /tmp/android-clippy.log 2>&1 || rc=$? + cat /tmp/android-clippy.log + if grep -q "STARTING A FULL BUILD" /tmp/android-clippy.log; then + echo "::error::skia-bindings did not get its prebuilt archive and started building Skia from source — the download was dropped (see DOWNLOAD AND INSTALL FAILED above). This is a fetch failure, not a lint failure." + exit 1 + fi + exit $rc # The kit's JVM unit tests — the pure parsers, migrations and feedback policies. They were # running nowhere: this workflow only assembled, and android-screenshots.yml runs the :app diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 5368f3285..3ee64f2f1 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -107,6 +107,12 @@ jobs: # registry/git are download caches, target/ the incremental build. The target key # carries the rustc version — resolved via `rustc --version` (below) rather than parsed # from rust-toolchain.toml, so a pin bump there invalidates stale incremental state too. + # `pf-console-ui` pulls skia-safe, so a target-cache miss makes this job download a prebuilt + # Skia from the same no-retry build-script fetch that took the android job out on + # 2026-08-22, over the same load-shedding runner network. Cheap insurance; see the script. + - name: curl with retries (skia-bindings' prebuilt fetch has none) + run: sh scripts/ci/install-retrying-curl.sh + - name: Cache keys run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV" - uses: actions/cache@v4 @@ -270,6 +276,12 @@ jobs: - name: sccache (no-op once the image bakes it) run: sh scripts/ci/ensure-sccache.sh + # `pf-console-ui` pulls skia-safe, so a target-cache miss makes this job download a prebuilt + # Skia from the same no-retry build-script fetch that took the android job out on + # 2026-08-22, over the same load-shedding runner network. Cheap insurance; see the script. + - name: curl with retries (skia-bindings' prebuilt fetch has none) + run: sh scripts/ci/install-retrying-curl.sh + - name: Cache keys run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV" - uses: actions/cache@v4 diff --git a/scripts/ci/install-retrying-curl.sh b/scripts/ci/install-retrying-curl.sh new file mode 100755 index 000000000..7edb46ec6 --- /dev/null +++ b/scripts/ci/install-retrying-curl.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# Put a retrying `curl` first on PATH for the rest of the job. +# +# WHY THIS EXISTS: `scripts/ci/retry.sh` already wraps every single-shot network command in CI, +# for the reason documented there — the runner box runs many jobs in parallel and its network +# drops packets under that load. But one of the biggest fetches in this workspace is NOT ours to +# wrap: skia-bindings downloads ~19 MB of prebuilt Skia per target from inside its build script, +# with a bare `curl -sS -f -L` and no retry at all (build_support/binary_cache/utils.rs). +# +# When that transfer truncates the job does not fail with a network error. skia-bindings' +# `try_prepare_download` swallows it, prints `DOWNLOAD AND INSTALL FAILED`, and falls through to +# `STARTING A FULL BUILD` — a from-source Skia build that the CI containers carry no deps for. +# What the operator sees is a Gradle stack trace under "Clippy (Android target)" with the real +# cause 1,800 lines up. Measured on main 2026-08-22: +# +# DOWNLOAD AND INSTALL FAILED: curl error code: "18" +# curl stderr: "curl: (18) end of response with 17054400 bytes missing" +# +# (19,057,024 bytes on the wire; it got 2 MB before git.unom.io closed the connection. The same +# asset pulls fine from a dev box, so this is the load-shedding retry.sh was written for.) +# +# A shim is the only lever that reaches inside a build script. It is also the cheapest correct +# one: skia-bindings already passes `-C -` (resume) and caches the part-file under +# OUT_DIR/.cache, so a retry CONTINUES the truncated transfer instead of restarting it. +# +# Applies to every curl in the job, which is what we want — the workspace's other build-script +# fetches are single-shot too. +# +# POSIX sh on purpose: Gitea's act_runner executes a step's `run:` under `sh -e` (dash) inside +# the Linux job containers — see the shader-gate note in ci.yml for what assuming bash cost. +# +# Usage: sh scripts/ci/install-retrying-curl.sh +set -e + +# Resolve the REAL curl before the shim is on PATH, and bake the absolute path into the shim — +# a shim that re-resolves `curl` by name would exec itself. +real_curl=$(command -v curl || true) +if [ -z "$real_curl" ]; then + echo "::warning::no curl on PATH — skipping the retrying-curl shim" + exit 0 +fi + +# RUNNER_TEMP (not /usr/local/bin): the job containers run as root but the macOS runner is a +# persistent host where a system dir is neither writable nor ours to litter. +shim_dir="${RUNNER_TEMP:-/tmp}/pf-retrying-curl" +mkdir -p "$shim_dir" + +# --retry-all-errors is what makes this cover error 18: a truncated transfer is a *transfer* +# failure, not an HTTP status, so plain --retry (which only retries transient HTTP codes and +# connection errors) would let it through. Needs curl >= 7.71; the CI images are well past it. +cat > "$shim_dir/curl" <> "$GITHUB_PATH" + echo "retrying curl installed: $shim_dir/curl -> $real_curl" +else + echo "::warning::GITHUB_PATH unset — shim written to $shim_dir but not on PATH" +fi -- 2.54.0