fix(ci/flatpak): the DNS failures are runner-load packet drops, not nsswitch — widen the retry budget
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 1m9s
decky / build-publish (push) Successful in 17s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
ci / bench (push) Successful in 5m45s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5m20s
arch / build-publish (push) Successful in 11m35s
docker / deploy-docs (push) Successful in 22s
android / android (push) Successful in 13m7s
deb / build-publish (push) Successful in 12m36s
flatpak / build-publish (push) Failing after 8m5s
ci / rust (push) Successful in 17m55s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m17s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m15s
apple / swift (push) Successful in 5m13s
apple / screenshots (push) Successful in 21m49s
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 1m9s
decky / build-publish (push) Successful in 17s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
ci / bench (push) Successful in 5m45s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5m20s
arch / build-publish (push) Successful in 11m35s
docker / deploy-docs (push) Successful in 22s
android / android (push) Successful in 13m7s
deb / build-publish (push) Successful in 12m36s
flatpak / build-publish (push) Failing after 8m5s
ci / rust (push) Successful in 17m55s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m17s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m15s
apple / swift (push) Successful in 5m13s
apple / screenshots (push) Successful in 21m49s
Live-debugged on home-runner-1: confirmed the dnf install in Tooling does NOT actually rewrite /etc/nsswitch.conf (no authselect trigger fires), so the nss-resolve sed from 68b5376f/5f687a70 was never the actual fix. gitea-runner-fleet on this box is a shared, resource-capped fleet (3 replicas, --cpus 5/--memory 7g each, on 16c/24G) serving punktfunk AND several other orgs' repos concurrently. A push to main fans out ~8 punktfunk workflows at once on top of that other traffic, and the box's Docker embedded DNS resolver (127.0.0.11) drops UDP lookups under the combined load — exactly what retry.sh's own header already documented. Manually dispatching this job while the box was idle succeeded instantly, with or without the sed. retry.sh's 5-attempt (~100s) budget isn't always enough to outlast a synchronized multi-org burst; bumped the three flathub-facing calls (remote-add, install-deps-only, download-only) to 10 attempts (~9min). Verified end-to-end on a live re-run (run 8993) after this change: full success including publish + OSTree deploy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -89,18 +89,28 @@ jobs:
|
||||
# gnupg2/rsync/openssh-clients: sign the OSTree repo + rsync it to unom-1 (see the deploy step).
|
||||
dnf -y install flatpak flatpak-builder git python3 python3-aiohttp python3-tomlkit curl jq \
|
||||
gnupg2 rsync openssh-clients
|
||||
# `flatpak` recommends xdg-desktop-portal -> pipewire/wireplumber, which drags in a
|
||||
# systemd package upgrade. Fedora's systemd package carries an RPM trigger that
|
||||
# re-runs authselect, which REGENERATES /etc/nsswitch.conf and re-adds the `resolve`
|
||||
# entry the earlier "Fix container DNS" step dropped — clobbering that fix before
|
||||
# `flatpak remote-add` ever runs (this is why the sed alone didn't stick: it ran
|
||||
# before this dnf install, not after). Re-apply it here, post-install.
|
||||
# Belt-and-suspenders: keep nsswitch on plain `dns` even if this dnf transaction pulled
|
||||
# in a fresh systemd-resolved (it does — flatpak recommends xdg-desktop-portal ->
|
||||
# pipewire/wireplumber -> systemd-networkd/-resolved). Verified on the real runner
|
||||
# (2026-07-11) this dnf install does NOT actually rewrite /etc/nsswitch.conf — no
|
||||
# authselect trigger fires — so this line alone was never the fix for the failures
|
||||
# below. See the retry.sh bump for the real cause.
|
||||
sed -i 's/resolve \[!UNAVAIL=return\] //' /etc/nsswitch.conf
|
||||
# Flathub provides the GNOME runtime/SDK + the rust-stable + ffmpeg-full extensions.
|
||||
# retry.sh: the busy runner intermittently drops DNS lookups ("[6] Could not resolve
|
||||
# hostname" seconds after dnf pulled 300+ packages fine) — never fail the job on a
|
||||
# single-shot fetch. Same treatment for every network command below.
|
||||
bash scripts/ci/retry.sh 5 flatpak remote-add --user --if-not-exists flathub \
|
||||
#
|
||||
# ROOT CAUSE (confirmed 2026-07-11 by watching a live run on home-runner-1): this is
|
||||
# NOT a deterministic nsswitch/DNS-config bug. gitea-runner-fleet on home-runner-1 is
|
||||
# a SHARED, resource-capped fleet (3 replicas, --cpus 5 --memory 7g each, on a 16c/24G
|
||||
# box) serving punktfunk AND several other orgs' repos (chatwoot, website, cms, data,
|
||||
# infra, tempblade, played...). A push to punktfunk's main fans out ~8 workflows at
|
||||
# once, and this runner box's Docker embedded DNS resolver (127.0.0.11) genuinely
|
||||
# drops UDP lookups under that concurrent load — exactly what retry.sh's own header
|
||||
# comment already documented. Reproduced: manually dispatching this job when the box
|
||||
# was idle (1 container running) succeeded immediately, with or without the sed above.
|
||||
# 5 attempts (~100s total) isn't always enough to outlast a synchronized multi-org
|
||||
# burst, so bump the bootstrap fetch's budget — never fail the job on a single-shot
|
||||
# fetch. Same treatment for every network command below.
|
||||
bash scripts/ci/retry.sh 10 flatpak remote-add --user --if-not-exists flathub \
|
||||
https://dl.flathub.org/repo/flathub.flatpakrepo
|
||||
git config --global --add safe.directory "$PWD"
|
||||
|
||||
@@ -195,10 +205,12 @@ jobs:
|
||||
# the .flatpak-builder state dir. Both are resumable/idempotent, so re-running
|
||||
# after a partial failure is safe and cheap.
|
||||
# --disable-rofiles-fuse is the container-safe path (no FUSE).
|
||||
bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \
|
||||
# 10 attempts (~9min budget), matching the remote-add bootstrap above — same shared,
|
||||
# load-sensitive runner, same flathub.org resolution path.
|
||||
bash scripts/ci/retry.sh 10 flatpak-builder --user --force-clean --disable-rofiles-fuse \
|
||||
--install-deps-from=flathub --install-deps-only \
|
||||
"$PWD/build-dir" "$MANIFEST"
|
||||
bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \
|
||||
bash scripts/ci/retry.sh 10 flatpak-builder --user --force-clean --disable-rofiles-fuse \
|
||||
--download-only \
|
||||
"$PWD/build-dir" "$MANIFEST"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user