The flatpak build's "offline" step was never offline — and 15 of its 21 minutes were setup #279
Assignees
FernandoMX
FlipperDan
HitFrostbite
enricobuehler (Enrico Bühler)
kszlim
luxus
nannku
neoney
sassycrown
Clear assignees
FernandoMX
FlipperDan
HitFrostbite
enricobuehler (Enrico Bühler)
kszlim
luxus
nannku
neoney
sassycrown
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: unom/punktfunk#279
Reference in New Issue
Block a user
Three commits: one fixes the recurring failure, two cut the runtime.
1. The 503 failures —
5b2f795bRun 18870 died 5 min into "Build the flatpak (offline — deps + sources prefetched above)" with the prefetch step green:
That step was never actually offline. flatpak-builder re-runs the download phase as part of every build —
builder-main.ccallsbuilder_manifest_download()unconditionally (only--disable-downloadskips it) and passesupdate_vcs = !opt_disable_updates. With updates on (the default),builder_git_mirror_repo()always does a livegit ls-remote+git fetchfor every git source and every submodule, even ones pinned to an immutable commit sha. So the long compile carried 24 unretried fetches in front of it —retry.shonly wraps the prefetch.--disable-updatesmakesbuilder_git_mirror_repo()short-circuit ongit cat-file -e <commit>against the warm mirror and return before any network call (builder-git.c,if (already_exists && !update)). It cannot change what gets built: everytype: gitsource in the manifest is commit-pinned, so "don't update" is a semantic no-op. Anything genuinely missing still downloads, so a cold state dir self-heals.Three of those repos — wlroots, libliftoff, libdisplay-info — are never even built (
enable_gamescope=false); flatpak-builder mirrors them only because it clones submodules by default.2 & 3. The 21 minutes —
6e467344,7e151609Measured per-step on run 18855 (green, 21m23s):
dnf install nodejsOnly 6m10s is the actual compile.
New
ci/flatpak-ci.Dockerfile— flatpak.yml was the last workflow still running a raw distro image and installing its toolchain per run. Content-keyed on the LAN registry like every other builder. The Flathub runtimes are baked too, which is the bigger half; a package-only image would leave the single largest step in place.Baking them turned out to need no build-time privileges, which is not obvious. Verified in a plain
docker runcontainer wherebwrapwas proven broken first (bwrap: No permissions to creating new namespace):flatpak install --userstill exited 0 andflatpak info --userresolved the ref. flatpak's post-deploy triggers are the only part that wants bwrap, and they're best-effort. The image asserts each ref deployed rather than trusting the install's exit status — an image that merely looks warm would push 1.5 GB back onto every run, reading as "flatpak got slow again" instead of a broken image.The seed step isn't really a build-time problem, it just presents as one. It mirrored the entire published OSTree repo down over ssh, and since the local repo starts empty there's nothing to diff against, so it shipped all of it every run:
3.84 GB and 180s off a Hetzner box to publish a ~28 MB commit — and the repo grows by that 28 MB every build, since the upload runs without
--delete(deliberate) and nothing ever pruned it. What summary regeneration actually needs is each channel's tip, so it now pulls exactly that over HTTP with--depth=0 --mirror. The current channel's tip earns its place twice: it's the new commit's parent, so--generate-static-deltasstill emits the from-parent delta that keepsflatpak updateincremental.The invariant the step exists to protect is preserved and slightly tightened — it enumerates every app ref the published summary advertises and fails on any it cannot seed. Only a real 404 continues with a fresh repo; treating a flaky link as "nothing published yet" is exactly how a single-branch summary comes to clobber the other channel (the "No such ref" bug in its own comment). It also no longer needs a deploy secret.
Server-side growth is bounded by a guarded
ostree prune --refs-only --keep-younger-than='30 days ago'after the upload — unreachable commits only (superseded canaries), nothing recent, so a client mid-pull or a box a few builds behind still resolves everything. Never allowed to fail the deploy. Worth doing on its own merits: unom-1 has hit ENOSPC before.Expected result
~21 min → ~9 min, with the 6 min compile as the floor. Making that incremental is a different problem — no sccache is reachable from inside an offline flatpak sandbox, and flatpak-builder wipes the module build dir. Not attempted.
⚠ Bootstrap — expect one red flatpak run on the merge
The LAN registry must hold
punktfunk-flatpak-ci:latestbefore flatpak.yml can run. This merge firesdocker.ymlandflatpak.ymltogether, so the flatpak job can lose the race and fail on the image pull. Re-run it once docker.yml is green. Nothing self-heals it — the container never starts to run a fallback, and adnf installfallback would only paper over a stale image. Same rule as every other LAN builder (see docker.yml's header).Verification status
Validated locally: both workflow YAMLs parse, every
run:block passesbash -n, the Dockerfile passes buildkit--check, all 14 package names resolve in Fedora 43, and theflatpak install-without-bwrap behaviour was tested directly. Not validated by a real CI run — the timings above are measured, but the ~9 min projection is arithmetic on them. The next flatpak run is the proof.Run 18870 died 5 minutes into "Build the flatpak (offline — deps + sources prefetched above)" with the prefetch step green: Downloading sources Fetching full git repo https://github.com/ValveSoftware/gamescope.git Fetching git repo https://gitlab.freedesktop.org/wlroots/wlroots.git, ref refs/tags/0.19.3 Fetching git repo https://gitlab.freedesktop.org/emersion/libliftoff.git, ref refs/tags/v0.5.0 Fetching full git repo https://github.com/Joshua-Ashton/vkroots Fetching git repo https://gitlab.freedesktop.org/emersion/libdisplay-info, ref refs/tags/0.3.0 error: RPC failed; HTTP 503 curl 22 The requested URL returned error: 503 fatal: expected 'acknowledgments' Failed to download sources: module gamescope-wsi-layer: Child process exited with code 128 That step was never actually offline. flatpak-builder runs the download phase again as part of every build — builder-main.c calls builder_manifest_download() unconditionally (only --disable-download skips it) and passes `update_vcs = !opt_disable_updates`. With updates on (the default), builder_git_mirror_repo() always does a live `git ls-remote` + `git fetch` for every git source AND every submodule, even ones pinned to an immutable commit sha. So the long compile step carried five unretried network fetches in front of it, and gitlab.freedesktop.org returning 503 took the whole job down — outside retry.sh, which only wraps the prefetch. Pass --disable-updates ("only download missing sources, never update to latest vcs version") to the build. builder_git_mirror_repo() then short-circuits on `git cat-file -e <commit>` against the warm mirror and returns before any network call. It cannot change what gets built: every `type: git` source in the manifest is commit-pinned (gamescope, glm, stb — plus gamescope's submodules, pinned by their gitlinks), so "don't update" is a semantic no-op. Anything genuinely missing still downloads, so a cold state dir self-heals. Same flag on the prefetch's --download-only, where it is already retried: it makes a restored .flatpak-builder cache actually save network instead of re-fetching pins that cannot have moved. That is ~2.5 min per run of wlroots/libliftoff/vkroots/libdisplay-info — submodules this manifest never even builds, since the module sets enable_gamescope=false; flatpak-builder mirrors them only because it clones submodules by default. build-flatpak.sh gets the same flag so local/Deck builds match CI.flatpak.yml is the last workflow still running a raw distro image and installing its tools per run. Measured on run 18855, a green 21m23s build whose actual cargo compile is 6m10s: 63 s dnf -y install nodejs 240 s Tooling — 330 packages 168 s actions/cache restore of ~/.local/share/flatpak (1.5 GB) i.e. ~7.8 min of setup in front of a 6 min build. Every other heavy workflow already solved this the same way (rust-ci, fedora-rpm, android-ci, arch-ci...): a content-keyed image on the LAN registry, rebuilt only when ci/ changes. The runtimes are the interesting half — a package-only image would leave the single biggest step in place — and baking them turned out to be free of the privilege problem it looks like it should have. VERIFIED in a plain `docker run` container where bwrap was proven broken first ("bwrap: No permissions to creating new namespace"): `flatpak install --user` still exited 0 and `flatpak info --user` resolved the ref. flatpak's post-deploy triggers are the only part that wants bwrap and they are best-effort, so this image needs none of the --privileged the consuming job needs for flatpak-builder's real sandbox. The image asserts each ref deployed rather than trusting the install's exit status: an image that merely LOOKS warm would push 1.5 GB back onto every run, where it reads as "flatpak got slow again" rather than as a broken image. Related refs (GL.default{,-extra}, codecs-extra, Locale) are kept deliberately — they are what --install-deps-only would otherwise pull per run, so a --no-related image would look smaller and cost more. This moves the same content the runtime cache already held from the cache server to the registry, where Docker keeps it on the runner's disk instead of re-extracting it every run. ⚠ The runtime pins now live in two places: ci/flatpak-ci.Dockerfile's ARGs and the manifest's runtime-version. Drift is not fatal (the prefetch downloads whatever is missing, retried) but silently costs the win, so bump them together.Two independent costs in the same job, both measured on run 18855 (21m23s green, of which the cargo compile is 6m10s). 1. The toolchain. Swap fedora:43 for punktfunk-flatpak-ci and the two dnf steps (303 s) plus the 1.5 GB runtime cache restore (168 s) become image layers. The Tooling step stays as an ASSERTION rather than the usual no-op install guard: the only failure it can actually see is a :latest that lags a ci/ change, and dnf-ing on top of that would hide the drift while spending the time the image exists to save. The runtime cache step is deleted outright — with the runtimes baked it is not merely redundant but harmful, since restoring it would spend 168 s overwriting the baked installation with an older copy of itself. 2. The seed. This is not really a build-time problem, it just presents as one. The step mirrored the entire published OSTree repo down from unom-1 over ssh, and because the local repo starts empty there is nothing to diff against, so it transferred all of it, every run — rsync said so itself: received 3,835,169,389 bytes … total size is 3,845,084,524 speedup is 1.00 3.84 GB and 180 s off a Hetzner box to publish a ~28 MB commit, and the repo grows by that 28 MB every build: the upload runs without --delete (deliberate) and nothing ever pruned it. What the summary regeneration actually needs is each channel's TIP, so pull exactly that over HTTP with --depth=0 --mirror. The current channel's tip earns its place twice — it is also the new commit's parent, so --generate-static-deltas still emits the from-parent delta that makes `flatpak update` incremental. The invariant this step exists to protect is preserved and, if anything, tightened: it enumerates every app ref the published summary advertises and fails on any it cannot seed. Only a real 404 may continue with a fresh repo — treating a flaky link as "nothing published yet" is exactly how a single-branch summary comes to clobber the other channel, the "No such ref" bug the step's comment already documents. It also no longer needs a deploy secret, since it reads the public repo. Server-side growth is bounded by an `ostree prune --refs-only --keep-younger-than='30 days ago'` after the upload: it touches only commits no ref points at (superseded canaries) and spares anything recent, so a client mid-pull or a box a few builds behind still resolves every object it asks for. Guarded on ostree existing there and never allowed to fail the deploy — by that point the bundle and repo are published, and a full disk is a slower problem than a red release. Expected: ~21 min -> ~9 min, with the 6 min compile as the floor. Making that incremental is a different problem — no sccache is reachable from inside an offline flatpak sandbox, and flatpak-builder wipes the module build dir. ⚠ BOOTSTRAP: the LAN registry must hold punktfunk-flatpak-ci:latest before this runs. The merge that lands this fires docker.yml and flatpak.yml together, so this job can lose the race and fail on the image pull; re-run it once docker.yml is green. Noted in the workflow — nothing self-heals it, as the container never starts to run a fallback.