From 313f194c637f9cb9eb7bd26d1d659ac4af017b88 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Sat, 11 Jul 2026 02:21:41 +0200 Subject: [PATCH] =?UTF-8?q?fix(ci/flatpak):=20the=20DNS=20failures=20are?= =?UTF-8?q?=20runner-load=20packet=20drops,=20not=20nsswitch=20=E2=80=94?= =?UTF-8?q?=20widen=20the=20retry=20budget?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live-debugged on home-runner-1: confirmed the dnf install in Tooling does NOT actually rewrite /etc/nsswitch.conf (no authselect trigger fires), so the nss-resolve sed from 68b5376f/5f687a70 was never the actual fix. gitea-runner-fleet on this box is a shared, resource-capped fleet (3 replicas, --cpus 5/--memory 7g each, on 16c/24G) serving punktfunk AND several other orgs' repos concurrently. A push to main fans out ~8 punktfunk workflows at once on top of that other traffic, and the box's Docker embedded DNS resolver (127.0.0.11) drops UDP lookups under the combined load — exactly what retry.sh's own header already documented. Manually dispatching this job while the box was idle succeeded instantly, with or without the sed. retry.sh's 5-attempt (~100s) budget isn't always enough to outlast a synchronized multi-org burst; bumped the three flathub-facing calls (remote-add, install-deps-only, download-only) to 10 attempts (~9min). Verified end-to-end on a live re-run (run 8993) after this change: full success including publish + OSTree deploy. Co-Authored-By: Claude Sonnet 5 --- .gitea/workflows/flatpak.yml | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/.gitea/workflows/flatpak.yml b/.gitea/workflows/flatpak.yml index f5cc0869..d25879c6 100644 --- a/.gitea/workflows/flatpak.yml +++ b/.gitea/workflows/flatpak.yml @@ -89,18 +89,28 @@ jobs: # gnupg2/rsync/openssh-clients: sign the OSTree repo + rsync it to unom-1 (see the deploy step). dnf -y install flatpak flatpak-builder git python3 python3-aiohttp python3-tomlkit curl jq \ gnupg2 rsync openssh-clients - # `flatpak` recommends xdg-desktop-portal -> pipewire/wireplumber, which drags in a - # systemd package upgrade. Fedora's systemd package carries an RPM trigger that - # re-runs authselect, which REGENERATES /etc/nsswitch.conf and re-adds the `resolve` - # entry the earlier "Fix container DNS" step dropped — clobbering that fix before - # `flatpak remote-add` ever runs (this is why the sed alone didn't stick: it ran - # before this dnf install, not after). Re-apply it here, post-install. + # Belt-and-suspenders: keep nsswitch on plain `dns` even if this dnf transaction pulled + # in a fresh systemd-resolved (it does — flatpak recommends xdg-desktop-portal -> + # pipewire/wireplumber -> systemd-networkd/-resolved). Verified on the real runner + # (2026-07-11) this dnf install does NOT actually rewrite /etc/nsswitch.conf — no + # authselect trigger fires — so this line alone was never the fix for the failures + # below. See the retry.sh bump for the real cause. sed -i 's/resolve \[!UNAVAIL=return\] //' /etc/nsswitch.conf # Flathub provides the GNOME runtime/SDK + the rust-stable + ffmpeg-full extensions. - # retry.sh: the busy runner intermittently drops DNS lookups ("[6] Could not resolve - # hostname" seconds after dnf pulled 300+ packages fine) — never fail the job on a - # single-shot fetch. Same treatment for every network command below. - bash scripts/ci/retry.sh 5 flatpak remote-add --user --if-not-exists flathub \ + # + # ROOT CAUSE (confirmed 2026-07-11 by watching a live run on home-runner-1): this is + # NOT a deterministic nsswitch/DNS-config bug. gitea-runner-fleet on home-runner-1 is + # a SHARED, resource-capped fleet (3 replicas, --cpus 5 --memory 7g each, on a 16c/24G + # box) serving punktfunk AND several other orgs' repos (chatwoot, website, cms, data, + # infra, tempblade, played...). A push to punktfunk's main fans out ~8 workflows at + # once, and this runner box's Docker embedded DNS resolver (127.0.0.11) genuinely + # drops UDP lookups under that concurrent load — exactly what retry.sh's own header + # comment already documented. Reproduced: manually dispatching this job when the box + # was idle (1 container running) succeeded immediately, with or without the sed above. + # 5 attempts (~100s total) isn't always enough to outlast a synchronized multi-org + # burst, so bump the bootstrap fetch's budget — never fail the job on a single-shot + # fetch. Same treatment for every network command below. + bash scripts/ci/retry.sh 10 flatpak remote-add --user --if-not-exists flathub \ https://dl.flathub.org/repo/flathub.flatpakrepo git config --global --add safe.directory "$PWD" @@ -195,10 +205,12 @@ jobs: # the .flatpak-builder state dir. Both are resumable/idempotent, so re-running # after a partial failure is safe and cheap. # --disable-rofiles-fuse is the container-safe path (no FUSE). - bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \ + # 10 attempts (~9min budget), matching the remote-add bootstrap above — same shared, + # load-sensitive runner, same flathub.org resolution path. + bash scripts/ci/retry.sh 10 flatpak-builder --user --force-clean --disable-rofiles-fuse \ --install-deps-from=flathub --install-deps-only \ "$PWD/build-dir" "$MANIFEST" - bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \ + bash scripts/ci/retry.sh 10 flatpak-builder --user --force-clean --disable-rofiles-fuse \ --download-only \ "$PWD/build-dir" "$MANIFEST"