A dropped skia download read as a lint failure, and had no retry to survive on #383

Merged

enricobuehler merged 1 commits from ci-skia-download-retry into main 2026-08-23 08:25:01 +00:00

Conversation 0 Commits 1 Files Changed 3

+98 -1

enricobuehler commented 2026-08-23 08:01:02 +00:00

Owner

Copy Link

Copy Source

scripts/ci/retry.sh already wraps every single-shot network call in CI, for the reason documented there — the runner box runs many jobs in parallel and its network sheds packets under that load. One of the largest fetches in this workspace was never wrappable that way: skia-bindings pulls ~19 MB of prebuilt Skia per target from inside its build script, with a bare curl -sS -f -L and no retry (build_support/binary_cache/utils.rs).

What it actually looked like

Measured on main, android job, 2026-08-22:

DOWNLOAD AND INSTALL FAILED: curl error code: "18"
curl stderr: "curl: (18) end of response with 17054400 bytes missing"

2 MB of 19,057,024 arrived before git.unom.io closed the connection. The same asset pulls fine from a dev box (http=200 size=19057024 in 4.7 s), so this is exactly the load-shedding retry.sh exists for.

skia-bindings then swallowed it. try_prepare_download treats a failed download as "no prebuilt available" and falls through to STARTING A FULL BUILD — a from-source Skia build the CI containers carry no deps for. So the job surfaced as "Clippy (Android target, deny warnings) failed" with a Gradle stack trace, and the real cause ~1,800 lines above it. That is why this has looked like a Rust lint problem for a while.

The fix

• A retrying curl shim first on PATH is the only lever that reaches inside a build script, and the cheapest correct one: skia-bindings already passes -C - (resume) and caches the part-file under OUT_DIR/.cache, so a retry continues the truncated transfer rather than restarting it.
  • --retry-all-errors is load-bearing: a truncated transfer is a transfer failure, not an HTTP status, so plain --retry would let error 18 straight through.
  • Shim lives in RUNNER_TEMP, not a system dir — the Linux job containers run as root but the macOS runner is a persistent host.
  • The real curl is resolved and baked in as an absolute path before the shim is on PATH, so it cannot exec itself.
• Wired into android.yml and both ci.yml rust jobs — pf-console-ui depends on skia-safe, so ci / rust downloads Skia on any target/ cache miss and is exposed to the same drop. (android caches no target/ at all, which is why it hit first and hardest.)
• The prose rule is now a gate. android.yml's env: block already said "🛑 skia-bindings never fails when no archive matches — it silently builds Skia from source. Every ABI's log must show DOWNLOAD AND INSTALL SUCCEEDED." That was a manual check nobody performs on a green run. The clippy step now fails the job on STARTING A FULL BUILD with a message naming the fetch, so a dropped prebuilt can never masquerade as a lint failure again.

Verification

The shim was run and exercised against the real asset, using skia-bindings' exact argument style:

$ PATH="$shim_dir:$PATH" curl -sS -f -L -o /dev/null -w '%{http_code} %{size_download}' <skia asset>
200 19057024

sh -n clean (POSIX sh — the runner is dash, per the shader-gate note in ci.yml).

Not fixed here

deb.yml's build-publish / build-publish-gamescope jobs also fail on main, but with every step including actions/checkout@v4 marked failed — that is a runner/container problem, not a workspace one, and wants a separate look at the fleet.

`scripts/ci/retry.sh` already wraps every single-shot network call in CI, for the reason documented there — the runner box runs many jobs in parallel and its network sheds packets under that load. One of the largest fetches in this workspace was never wrappable that way: **skia-bindings pulls ~19 MB of prebuilt Skia per target from inside its build script**, with a bare `curl -sS -f -L` and no retry (`build_support/binary_cache/utils.rs`). ## What it actually looked like Measured on `main`, android job, 2026-08-22: ``` DOWNLOAD AND INSTALL FAILED: curl error code: "18" curl stderr: "curl: (18) end of response with 17054400 bytes missing" ``` 2 MB of 19,057,024 arrived before git.unom.io closed the connection. The same asset pulls fine from a dev box (`http=200 size=19057024` in 4.7 s), so this is exactly the load-shedding `retry.sh` exists for. skia-bindings then **swallowed it**. `try_prepare_download` treats a failed download as "no prebuilt available" and falls through to `STARTING A FULL BUILD` — a from-source Skia build the CI containers carry no deps for. So the job surfaced as **"Clippy (Android target, deny warnings) failed"** with a Gradle stack trace, and the real cause ~1,800 lines above it. That is why this has looked like a Rust lint problem for a while. ## The fix * **A retrying `curl` shim first on PATH** is the only lever that reaches inside a build script, and the cheapest correct one: skia-bindings already passes `-C -` (resume) and caches the part-file under `OUT_DIR/.cache`, so a retry *continues* the truncated transfer rather than restarting it. * `--retry-all-errors` is load-bearing: a truncated transfer is a transfer failure, not an HTTP status, so plain `--retry` would let error 18 straight through. * Shim lives in `RUNNER_TEMP`, not a system dir — the Linux job containers run as root but the macOS runner is a persistent host. * The real `curl` is resolved and baked in as an absolute path before the shim is on PATH, so it cannot exec itself. * **Wired into `android.yml` and both `ci.yml` rust jobs** — `pf-console-ui` depends on `skia-safe`, so `ci / rust` downloads Skia on any `target/` cache miss and is exposed to the same drop. (android caches no `target/` at all, which is why it hit first and hardest.) * **The prose rule is now a gate.** `android.yml`'s `env:` block already said "🛑 skia-bindings never fails when no archive matches — it silently builds Skia from source. Every ABI's log must show `DOWNLOAD AND INSTALL SUCCEEDED`." That was a manual check nobody performs on a green run. The clippy step now fails the job on `STARTING A FULL BUILD` with a message naming the fetch, so a dropped prebuilt can never masquerade as a lint failure again. ## Verification The shim was run and exercised against the real asset, using skia-bindings' exact argument style: ``` $ PATH="$shim_dir:$PATH" curl -sS -f -L -o /dev/null -w '%{http_code} %{size_download}' <skia asset> 200 19057024 ``` `sh -n` clean (POSIX sh — the runner is dash, per the shader-gate note in `ci.yml`). ## Not fixed here `deb.yml`'s `build-publish` / `build-publish-gamescope` jobs also fail on main, but with **every step including `actions/checkout@v4` marked failed** — that is a runner/container problem, not a workspace one, and wants a separate look at the fleet.

enricobuehler added 1 commit 2026-08-23 08:01:04 +00:00

fix(ci): a dropped skia download read as a lint failure, and had no retry to survive on ... 6f4613e146

scripts/ci/retry.sh already wraps every single-shot network call in CI, for the
reason documented there: the runner box runs many jobs in parallel and its
network sheds packets under that load. One of the largest fetches in this
workspace was never wrappable that way - skia-bindings pulls ~19 MB of prebuilt
Skia per target from INSIDE its build script, with a bare 'curl -sS -f -L' and
no retry (build_support/binary_cache/utils.rs).

Measured on main 2026-08-22, android job:

  DOWNLOAD AND INSTALL FAILED: curl error code: "18"
  curl stderr: "curl: (18) end of response with 17054400 bytes missing"

2 MB of 19,057,024 arrived before git.unom.io closed the connection; the same
asset pulls fine from a dev box. skia-bindings then swallowed it - its
try_prepare_download falls through to STARTING A FULL BUILD, a from-source Skia
build the CI containers carry no deps for - so the job surfaced as
'Clippy (Android target) failed' with a Gradle stack trace and the real cause
1,800 lines above it.

* A retrying curl shim first on PATH is the only lever that reaches inside a
  build script, and the cheapest correct one: skia-bindings already passes
  '-C -' and caches the part-file under OUT_DIR/.cache, so a retry CONTINUES
  the truncated transfer rather than restarting it. --retry-all-errors is
  load-bearing: a truncated transfer is not an HTTP status, so plain --retry
  would let error 18 through.
* Wired into android.yml and both ci.yml rust jobs - pf-console-ui pulls
  skia-safe too, so ci/rust downloads Skia on any target-cache miss.
* The rule android.yml's env block states in prose ('Every ABI's log must show
  DOWNLOAD AND INSTALL SUCCEEDED') is now a gate that fails the job on
  STARTING A FULL BUILD, so a dropped prebuilt can never masquerade as a lint
  failure again.

enricobuehler merged commit 89eb031cd6 into main 2026-08-23 08:25:01 +00:00

enricobuehler referenced this issue from a commit 2026-08-23 10:01:50 +00:00

release: 0.31.3 — version bump, notes, CHANGELOG, Play notes

enricobuehler referenced this pull request 2026-08-23 10:02:22 +00:00

0.31.3 — the launch that dropped, the refresh a TV never output, and the probe that choked the link #386

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX FlipperDan HitFrostbite Seenko bluelightspecial enricobuehler (Enrico Bühler) kszlim luxus nannku neoney sonar

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#383