From 3d6c06bb0673117bf83b49250ae9118bd50f76b8 Mon Sep 17 00:00:00 2001 From: enricobuehler Date: Fri, 10 Jul 2026 10:20:40 +0200 Subject: [PATCH] =?UTF-8?q?fix(ci):=20flatpak=20job=20rides=20out=20the=20?= =?UTF-8?q?runner's=20network=20flake=20=E2=80=94=20retry=20every=20single?= =?UTF-8?q?-shot=20fetch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The flatpak run dies most pushes at `flatpak remote-add` with an instant "[6] Could not resolve hostname" (runs 8755/8702/8689), 25ms after dnf pulled 329 packages fine — and deploy-docs separately hits "dial tcp: i/o timeout" to unom-1 (8716/8679). The busy runner drops UDP DNS + TCP dials under parallel-job load; tools with built-in retries (dnf) ride it out, single-shot fetches killed the whole 2h build slot. * scripts/ci/retry.sh: linear-backoff wrapper (5 tries, attempt*10s); stdout passes through untouched so $(…) capture works. * Tooling: retry the flathub remote-add. * Build split: a retried prefetch phase (--install-deps-only, then --download-only for every crate in cargo-sources.json) warms the state dir; the long compile then runs with no network left to flake on. * Seed step: probe (retried) whether the server repo exists — ONLY first publish may continue fresh. The old blanket `rsync || warn` swallowed transient failures too, producing the single-branch summary that clobbers the other channel (the exact bug the step's comment documents). * Deploy: retry the idempotent ssh/rsync calls to unom-1. * raw.githubusercontent curl: --retry 5 --retry-all-errors. docker.yml's deploy-docs has the same disease but sits on drone-ssh; left for a follow-up. Real root cause is runner-host networking (conntrack / upstream DNS under burst load) — retries make the jobs indifferent to it. Co-Authored-By: Claude Fable 5 --- .gitea/workflows/flatpak.yml | 62 ++++++++++++++++++++++++++---------- scripts/ci/retry.sh | 27 ++++++++++++++++ 2 files changed, 73 insertions(+), 16 deletions(-) create mode 100644 scripts/ci/retry.sh diff --git a/.gitea/workflows/flatpak.yml b/.gitea/workflows/flatpak.yml index ec8cce6b..42c620ce 100644 --- a/.gitea/workflows/flatpak.yml +++ b/.gitea/workflows/flatpak.yml @@ -73,7 +73,10 @@ jobs: dnf -y install flatpak flatpak-builder git python3 python3-aiohttp python3-tomlkit curl jq \ gnupg2 rsync openssh-clients # Flathub provides the GNOME runtime/SDK + the rust-stable + ffmpeg-full extensions. - flatpak remote-add --user --if-not-exists flathub \ + # retry.sh: the busy runner intermittently drops DNS lookups ("[6] Could not resolve + # hostname" seconds after dnf pulled 300+ packages fine) — never fail the job on a + # single-shot fetch. Same treatment for every network command below. + bash scripts/ci/retry.sh 5 flatpak remote-add --user --if-not-exists flathub \ https://dl.flathub.org/repo/flathub.flatpakrepo git config --global --add safe.directory "$PWD" @@ -108,7 +111,7 @@ jobs: # device" (see packaging/flatpak/prune-windows-lock.py). The committed Cargo.lock is # untouched; cargo --offline only needs sources for the crates it compiles. run: | - curl -fsSL -o /tmp/flatpak-cargo-generator.py \ + curl -fsSL --retry 5 --retry-all-errors --retry-delay 5 -o /tmp/flatpak-cargo-generator.py \ https://raw.githubusercontent.com/flatpak/flatpak-builder-tools/master/cargo/flatpak-cargo-generator.py python3 packaging/flatpak/prune-windows-lock.py Cargo.lock /tmp/Cargo.flatpak.lock python3 /tmp/flatpak-cargo-generator.py /tmp/Cargo.flatpak.lock \ @@ -142,18 +145,43 @@ jobs: DEST="${DEPLOY_USER}@${DEPLOY_HOST}" mkdir -p "$PWD/repo" # Pull the currently-published repo (all channels' objects + refs) into the repo the build - # will extend. No --delete: the local repo starts empty, so this only ADDS. A missing - # server repo (very first publish) is fine — we continue with a fresh repo. - rsync -az --info=stats1 -e "$SSH" "$DEST:$DEPLOY_DIR/site/repo/" "$PWD/repo/" \ - || echo "::warning::no published repo to seed (first publish?) — continuing fresh" + # will extend. No --delete: the local repo starts empty, so this only ADDS. + # Probe first (retried) whether a published repo exists at all: ONLY that case may + # continue with a fresh repo. A transient network failure must FAIL the job instead — + # a blanket `rsync || continue` here is exactly how a flaky link produces the + # single-branch summary that clobbers the other channel (the bug described above). + PRESENT=$(bash scripts/ci/retry.sh 5 $SSH "$DEST" \ + "[ -d $DEPLOY_DIR/site/repo/refs ] && echo present || echo absent") + if [ "$PRESENT" = present ]; then + bash scripts/ci/retry.sh 5 rsync -az --info=stats1 -e "$SSH" \ + "$DEST:$DEPLOY_DIR/site/repo/" "$PWD/repo/" + else + echo "::warning::no published repo on the server (first publish) — continuing fresh" + fi echo "seeded refs:"; ls "$PWD/repo/refs/heads/app/$APP_ID/x86_64/" 2>/dev/null || echo " (none)" - - name: Build the flatpak (install deps from Flathub, offline build) + - name: Prefetch deps + sources (retried — the network phase, split off the build) run: | - # --install-deps-from=flathub pulls everything the manifest declares: the GNOME 50 - # runtime/SDK + the rust-stable (//25.08, rustc 1.96) and llvm20 SDK extensions, plus - # the runtime's auto codecs-extra (HEVC libavcodec). --disable-rofiles-fuse is the - # container-safe path (no FUSE). + # All of the job's heavy network I/O happens HERE, retried, so a dropped DNS lookup + # or TCP dial costs a backoff-retry instead of the whole (long) compile: + # 1) --install-deps-only pulls everything the manifest declares from Flathub: the + # GNOME 50 runtime/SDK + the rust-stable (//25.08, rustc 1.96) and llvm20 SDK + # extensions, plus the runtime's auto codecs-extra (HEVC libavcodec). + # 2) --download-only fetches every source (all crates in cargo-sources.json) into + # the .flatpak-builder state dir. Both are resumable/idempotent, so re-running + # after a partial failure is safe and cheap. + # --disable-rofiles-fuse is the container-safe path (no FUSE). + bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \ + --install-deps-from=flathub --install-deps-only \ + "$PWD/build-dir" "$MANIFEST" + bash scripts/ci/retry.sh 5 flatpak-builder --user --force-clean --disable-rofiles-fuse \ + --download-only \ + "$PWD/build-dir" "$MANIFEST" + + - name: Build the flatpak (offline — deps + sources prefetched above) + run: | + # Everything is already local (state dir warmed by the prefetch step), so this long + # step needs no network; --install-deps-from stays as a no-op safety net. # --default-branch=$FLATPAK_BRANCH pins the ref to app/io.unom.Punktfunk/x86_64/ # (canary or stable) so the matching hosted .flatpakref resolves deterministically # (manifest sets no branch). @@ -272,11 +300,13 @@ jobs: printf '%s\n' "$DEPLOY_SSH_KEY" > ~/.ssh/deploy; chmod 600 ~/.ssh/deploy SSH="ssh -i $HOME/.ssh/deploy -p ${DEPLOY_PORT:-22} -o StrictHostKeyChecking=accept-new" DEST="${DEPLOY_USER}@${DEPLOY_HOST}" - $SSH "$DEST" "mkdir -p ~/$DEPLOY_DIR/site/repo" - rsync -az --info=stats1 -e "$SSH" repo/ "$DEST:$DEPLOY_DIR/site/repo/" - rsync -az -e "$SSH" site/unom.flatpakrepo "site/${APP_ID}.flatpakref" "site/${APP_ID}.Canary.flatpakref" site/index.html "$DEST:$DEPLOY_DIR/site/" - rsync -az -e "$SSH" packaging/flatpak/server/compose.production.yml packaging/flatpak/server/Caddyfile "$DEST:$DEPLOY_DIR/" - $SSH "$DEST" "cd ~/$DEPLOY_DIR && docker compose -f compose.production.yml up -d" + # All idempotent — retried because the runner's link to unom-1 drops TCP dials under + # load (the same flake that hits docker.yml's deploy-docs with "dial tcp: i/o timeout"). + bash scripts/ci/retry.sh 5 $SSH "$DEST" "mkdir -p ~/$DEPLOY_DIR/site/repo" + bash scripts/ci/retry.sh 5 rsync -az --info=stats1 -e "$SSH" repo/ "$DEST:$DEPLOY_DIR/site/repo/" + bash scripts/ci/retry.sh 5 rsync -az -e "$SSH" site/unom.flatpakrepo "site/${APP_ID}.flatpakref" "site/${APP_ID}.Canary.flatpakref" site/index.html "$DEST:$DEPLOY_DIR/site/" + bash scripts/ci/retry.sh 5 rsync -az -e "$SSH" packaging/flatpak/server/compose.production.yml packaging/flatpak/server/Caddyfile "$DEST:$DEPLOY_DIR/" + bash scripts/ci/retry.sh 5 $SSH "$DEST" "cd ~/$DEPLOY_DIR && docker compose -f compose.production.yml up -d" echo "deployed → $REPO_URL/${APP_ID}.flatpakref" - name: Attach bundle to the Gitea release (stable tags only) diff --git a/scripts/ci/retry.sh b/scripts/ci/retry.sh new file mode 100644 index 00000000..31f046ba --- /dev/null +++ b/scripts/ci/retry.sh @@ -0,0 +1,27 @@ +# shellcheck shell=bash +# Run a command, retrying with linear backoff: bash scripts/ci/retry.sh [args…] +# +# WHY THIS EXISTS: the runner box executes many jobs in parallel and its network drops +# packets under that load — observed as instant "[6] Could not resolve hostname" from +# flatpak/curl (dropped UDP DNS) and "dial tcp: i/o timeout" from the deploy steps +# (dropped TCP SYNs to unom-1), each seconds after other network calls in the same job +# succeeded. Tools with built-in mirror retries (dnf) ride it out; single-shot fetches +# (flatpak remote-add, rsync, ssh) killed the whole heavy flatpak build on one dropped +# packet. Wrap every single-shot network command in CI with this instead. +# +# Backoff is attempt*10s (10/20/30/…), so 5 attempts spread over ~1.5 min — long enough +# to outlive a load burst, short enough not to eat the job timeout. The wrapped command's +# stdout passes through untouched (safe for $(…) capture); retry chatter goes to stderr. +set -u + +attempts="$1"; shift +rc=1 +for i in $(seq 1 "$attempts"); do + "$@" && exit 0 + rc=$? + [ "$i" -eq "$attempts" ] && break + echo "::warning::attempt $i/$attempts failed (rc=$rc): $* — retrying in $((i * 10))s" >&2 + sleep $((i * 10)) +done +echo "::error::all $attempts attempts failed (rc=$rc): $*" >&2 +exit "$rc"