fix(ci/flatpak): the DNS failures are runner-load packet drops, not nsswitch — widen the retry budget
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 1m9s
decky / build-publish (push) Successful in 17s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
ci / bench (push) Successful in 5m45s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5m20s
arch / build-publish (push) Successful in 11m35s
docker / deploy-docs (push) Successful in 22s
android / android (push) Successful in 13m7s
deb / build-publish (push) Successful in 12m36s
flatpak / build-publish (push) Failing after 8m5s
ci / rust (push) Successful in 17m55s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m17s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m15s
apple / swift (push) Successful in 5m13s
apple / screenshots (push) Successful in 21m49s

Live-debugged on home-runner-1: confirmed the dnf install in Tooling does
NOT actually rewrite /etc/nsswitch.conf (no authselect trigger fires), so
the nss-resolve sed from 68b5376f/5f687a70 was never the actual fix.
gitea-runner-fleet on this box is a shared, resource-capped fleet (3
replicas, --cpus 5/--memory 7g each, on 16c/24G) serving punktfunk AND
several other orgs' repos concurrently. A push to main fans out ~8
punktfunk workflows at once on top of that other traffic, and the box's
Docker embedded DNS resolver (127.0.0.11) drops UDP lookups under the
combined load — exactly what retry.sh's own header already documented.
Manually dispatching this job while the box was idle succeeded instantly,
with or without the sed. retry.sh's 5-attempt (~100s) budget isn't always
enough to outlast a synchronized multi-org burst; bumped the three
flathub-facing calls (remote-add, install-deps-only, download-only) to 10
attempts (~9min). Verified end-to-end on a live re-run (run 8993) after
this change: full success including publish + OSTree deploy.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-11 02:21:41 +02:00
parent 55472495a4
commit 313f194c63
+24 -12
View File
@@ -89,18 +89,28 @@ jobs:
# gnupg2/rsync/openssh-clients: sign the OSTree repo + rsync it to unom-1 (see the deploy step). # gnupg2/rsync/openssh-clients: sign the OSTree repo + rsync it to unom-1 (see the deploy step).
dnf -y install flatpak flatpak-builder git python3 python3-aiohttp python3-tomlkit curl jq \ dnf -y install flatpak flatpak-builder git python3 python3-aiohttp python3-tomlkit curl jq \
gnupg2 rsync openssh-clients gnupg2 rsync openssh-clients
# `flatpak` recommends xdg-desktop-portal -> pipewire/wireplumber, which drags in a # Belt-and-suspenders: keep nsswitch on plain `dns` even if this dnf transaction pulled
# systemd package upgrade. Fedora's systemd package carries an RPM trigger that # in a fresh systemd-resolved (it does — flatpak recommends xdg-desktop-portal ->
# re-runs authselect, which REGENERATES /etc/nsswitch.conf and re-adds the `resolve` # pipewire/wireplumber -> systemd-networkd/-resolved). Verified on the real runner
# entry the earlier "Fix container DNS" step dropped — clobbering that fix before # (2026-07-11) this dnf install does NOT actually rewrite /etc/nsswitch.conf — no
# `flatpak remote-add` ever runs (this is why the sed alone didn't stick: it ran # authselect trigger fires — so this line alone was never the fix for the failures
# before this dnf install, not after). Re-apply it here, post-install. # below. See the retry.sh bump for the real cause.
sed -i 's/resolve \[!UNAVAIL=return\] //' /etc/nsswitch.conf sed -i 's/resolve \[!UNAVAIL=return\] //' /etc/nsswitch.conf
# Flathub provides the GNOME runtime/SDK + the rust-stable + ffmpeg-full extensions. # Flathub provides the GNOME runtime/SDK + the rust-stable + ffmpeg-full extensions.
# retry.sh: the busy runner intermittently drops DNS lookups ("[6] Could not resolve #
# hostname" seconds after dnf pulled 300+ packages fine) — never fail the job on a # ROOT CAUSE (confirmed 2026-07-11 by watching a live run on home-runner-1): this is
# single-shot fetch. Same treatment for every network command below. # NOT a deterministic nsswitch/DNS-config bug. gitea-runner-fleet on home-runner-1 is
bash scripts/ci/retry.sh 5 flatpak remote-add --user --if-not-exists flathub \ # a SHARED, resource-capped fleet (3 replicas, --cpus 5 --memory 7g each, on a 16c/24G
# box) serving punktfunk AND several other orgs' repos (chatwoot, website, cms, data,
# infra, tempblade, played...). A push to punktfunk's main fans out ~8 workflows at
# once, and this runner box's Docker embedded DNS resolver (127.0.0.11) genuinely
# drops UDP lookups under that concurrent load — exactly what retry.sh's own header
# comment already documented. Reproduced: manually dispatching this job when the box
# was idle (1 container running) succeeded immediately, with or without the sed above.
# 5 attempts (~100s total) isn't always enough to outlast a synchronized multi-org
# burst, so bump the bootstrap fetch's budget — never fail the job on a single-shot
# fetch. Same treatment for every network command below.
bash scripts/ci/retry.sh 10 flatpak remote-add --user --if-not-exists flathub \
https://dl.flathub.org/repo/flathub.flatpakrepo https://dl.flathub.org/repo/flathub.flatpakrepo
git config --global --add safe.directory "$PWD" git config --global --add safe.directory "$PWD"
@@ -195,10 +205,12 @@ jobs:
# the .flatpak-builder state dir. Both are resumable/idempotent, so re-running # the .flatpak-builder state dir. Both are resumable/idempotent, so re-running
# after a partial failure is safe and cheap. # after a partial failure is safe and cheap.
# --disable-rofiles-fuse is the container-safe path (no FUSE). # --disable-rofiles-fuse is the container-safe path (no FUSE).
bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \ # 10 attempts (~9min budget), matching the remote-add bootstrap above — same shared,
# load-sensitive runner, same flathub.org resolution path.
bash scripts/ci/retry.sh 10 flatpak-builder --user --force-clean --disable-rofiles-fuse \
--install-deps-from=flathub --install-deps-only \ --install-deps-from=flathub --install-deps-only \
"$PWD/build-dir" "$MANIFEST" "$PWD/build-dir" "$MANIFEST"
bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \ bash scripts/ci/retry.sh 10 flatpak-builder --user --force-clean --disable-rofiles-fuse \
--download-only \ --download-only \
"$PWD/build-dir" "$MANIFEST" "$PWD/build-dir" "$MANIFEST"