ci: cache the C/C++ half, link with mold, and split the debug/release target caches

Three independent reasons Rust CI stayed slow despite sccache, fixed together because
they share the same measurement.

1. sccache only ever covered RUSTC. Every C/C++ dependency in the tree — aws-lc-sys,
   openh264-sys2's vendored C++, the CMake-built libopus behind audiopus_sys — was
   compiled from scratch on every job of every workflow. CMAKE_{C,CXX}_COMPILER_LAUNCHER
   plus CC_/CXX_x86_64_unknown_linux_gnu route both build-script styles (cc-rs and
   cmake-rs) through the same shared cache.

   The CC_* vars are JOB-scoped in ci.yml and deb.yml, never workflow-scoped: the
   arm64 cross image sets its own CC_x86_64_unknown_linux_gnu=pf-host-cc, the wrapper
   that keeps ffmpeg-sys-next's host probe off the arm64 include dirs. Overwriting it
   would surface as a header mismatch rather than as a CI config error.

2. Linking is cacheable by nothing, and these jobs relink the host, client, session,
   cli, worker and tray on every run — twice per push for rpm (f43 + f44). The four
   Linux builder images now install mold and carry a $CARGO_HOME/config.toml that uses
   it for x86_64. aarch64 is deliberately left alone (cross driver, already-fast legs).
   Each image asserts `mold --version` in its build, so an image can never ship the
   flag without the linker: docker.yml goes red and :latest stays on the last good one.

3. THE EXPENSIVE ONE. ci.yml (debug) and deb.yml (release) named a byte-identical
   target-cache key, under a comment claiming the release build reused ci.yml's
   artifacts. It never could. actions/cache is first-saver-wins on an exact key and
   ci.yml is the faster job, so the shared key always held a debug-only target/ — and,
   worse, deb.yml could then never save its own, because the key was taken. Every
   canary .deb has been a from-scratch release build for as long as both keys existed.
   Same collision on the arm64 pair, and a third participant in
   linux-client-screenshots.yml. Split into -debug-/-release- key families; that job
   reads deb's tree via restore-keys but keeps its own exact key so it can never win
   the save race and replace a full tree with its single-crate one.

Also: one scripts/ci/ensure-sccache.sh replaces ten copy-pasted bootstrap blocks that
had already drifted into two dialects (GNU tar --wildcards vs bsdtar), every Rust job
now ends with --show-stats so a cache regression is visible instead of just "CI got
slower", and deb.yml's web install joins every other CI install on --ignore-scripts.

No behaviour change to any artifact: same compilers, same flags, same outputs.
This commit is contained in:
2026-08-13 11:57:13 +02:00
parent 124cb66324
commit f1dc6c9f94
14 changed files with 368 additions and 94 deletions
+9
View File
@@ -63,6 +63,15 @@ env:
SCCACHE_REGION: home-central
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_SECRET_ACCESS_KEY }}
# The C/C++ half of the cache. The one that pays here is the CMake-built vendored libopus
# (audiopus_sys), which kit/build.gradle.kts drives through cargo-ndk once per ABI — three
# from-scratch libopus builds per run until now. The per-ABI compilers come from the NDK via
# cargo-ndk's own CC_<android-triple> vars, which this does not touch; CC_x86_64_unknown_linux_gnu
# covers only the HOST build scripts and proc macros.
CMAKE_C_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
# sccache and incremental compilation are mutually exclusive; CI wants the shared
# cache, dev boxes keep incremental.
CARGO_INCREMENTAL: "0"
+6 -20
View File
@@ -85,17 +85,10 @@ jobs:
dirname "$RUSTUP" >> "$GITHUB_PATH"
"$RUSTUP" target add aarch64-apple-darwin x86_64-apple-darwin
# Shared compile cache. ~/.local/bin is on the runner daemon's PATH; GITHUB_PATH is
# belt-and-braces. bsdtar (macOS) globs by default — no --wildcards.
# Shared compile cache. The script handles the macOS side (user-prefix install +
# GITHUB_PATH, bsdtar globbing) — see scripts/ci/ensure-sccache.sh.
- name: sccache (self-healing install)
run: |
if ! command -v sccache >/dev/null; then
mkdir -p "$HOME/.local/bin"
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-aarch64-apple-darwin.tar.gz \
| tar -xz --strip-components=1 -C "$HOME/.local/bin" '*/sccache'
fi
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
sccache --version
run: sh scripts/ci/ensure-sccache.sh
# `punktfunk-core` now decodes Opus in-core for the Apple client (surround), pulling
# `audiopus_sys`, which builds a vendored static libopus via CMake when pkg-config can't find a
@@ -155,17 +148,10 @@ jobs:
"$RUSTUP" target add aarch64-apple-darwin x86_64-apple-darwin \
aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios
# Shared compile cache. ~/.local/bin is on the runner daemon's PATH; GITHUB_PATH is
# belt-and-braces. bsdtar (macOS) globs by default — no --wildcards.
# Shared compile cache. The script handles the macOS side (user-prefix install +
# GITHUB_PATH, bsdtar globbing) — see scripts/ci/ensure-sccache.sh.
- name: sccache (self-healing install)
run: |
if ! command -v sccache >/dev/null; then
mkdir -p "$HOME/.local/bin"
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-aarch64-apple-darwin.tar.gz \
| tar -xz --strip-components=1 -C "$HOME/.local/bin" '*/sccache'
fi
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
sccache --version
run: sh scripts/ci/ensure-sccache.sh
# See the swift job: audiopus_sys (via the in-core Opus decode) builds vendored libopus with CMake.
- name: CMake (for the vendored libopus audiopus_sys builds)
+19
View File
@@ -81,6 +81,14 @@ env:
SCCACHE_REGION: home-central
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_SECRET_ACCESS_KEY }}
# The C/C++ half of the cache (aws-lc-sys, vendored libopus, openh264's C++). Safe at workflow
# level — no cross-compiling job here (see ci.yml's `rust` job for that trap). NOTE these ALSO
# have to be named in the makepkg step's `sudo -u builder env …` list: sudo's env_reset drops
# everything not listed, which is why the sccache vars are already spelled out there.
CMAKE_C_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
CARGO_INCREMENTAL: "0"
jobs:
@@ -252,8 +260,19 @@ jobs:
SCCACHE_BUCKET="$SCCACHE_BUCKET" SCCACHE_ENDPOINT="$SCCACHE_ENDPOINT" \
SCCACHE_REGION="$SCCACHE_REGION" \
AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
CMAKE_C_COMPILER_LAUNCHER="$CMAKE_C_COMPILER_LAUNCHER" \
CMAKE_CXX_COMPILER_LAUNCHER="$CMAKE_CXX_COMPILER_LAUNCHER" \
CC_x86_64_unknown_linux_gnu="$CC_x86_64_unknown_linux_gnu" \
CXX_x86_64_unknown_linux_gnu="$CXX_x86_64_unknown_linux_gnu" \
makepkg -f -d --holdver
ls -lh "$GITHUB_WORKSPACE/dist"
# Visibility only. The stats have to be read as the SAME user that ran the compiles —
# sccache keeps its stats in a per-user server process, so a root `--show-stats` here
# would report an idle server and zero everything.
sudo -u builder env SCCACHE_BUCKET="$SCCACHE_BUCKET" SCCACHE_ENDPOINT="$SCCACHE_ENDPOINT" \
SCCACHE_REGION="$SCCACHE_REGION" \
AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
sccache --show-stats || true
# The host must ship a VERSIONED libav soname dep, and nothing else in this pipeline proves
# it. packaging/arch/PKGBUILD lists bare `libavcodec.so` etc. and relies on makepkg rewriting
+2 -7
View File
@@ -38,14 +38,9 @@ jobs:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
- name: Prep
run: |
+56 -18
View File
@@ -23,6 +23,12 @@ on:
# Shared compile cache: sccache -> RustFS S3 (storage.unom.io, LAN-pinned via ci-core's
# unbound). Keys include compiler hash + target + flags, so cross-OS/arch entries can
# never collide; every Rust job on every host feeds and reads one warm cache.
#
# RUSTC_WRAPPER covers RUST compilations and nothing else. The C/C++ half of this workspace —
# aws-lc-sys, openh264-sys2's vendored C++, the CMake-built libopus behind audiopus_sys, pyrowave —
# was paid in full on every run until the CMAKE_*_COMPILER_LAUNCHER / CC_* wiring below existed.
# Linking is the third phase and is cacheable by nothing: that one is addressed in the builder
# images with mold (ci/cargo-config-mold.toml).
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: unom-ci-sccache
@@ -30,6 +36,11 @@ env:
SCCACHE_REGION: home-central
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_SECRET_ACCESS_KEY }}
# Route CMake-driven C/C++ through the same cache. Safe at workflow level: it names no
# triple, and cmake-rs overrides it per-invocation with a `-D` flag when cc-rs reports a
# wrapper, so the two can never double-wrap into `sccache sccache cc`.
CMAKE_C_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
# sccache and incremental compilation are mutually exclusive; CI wants the shared
# cache, dev boxes keep incremental.
CARGO_INCREMENTAL: "0"
@@ -40,18 +51,25 @@ jobs:
container:
image: 192.168.1.58:5010/punktfunk-rust-ci:latest
timeout-minutes: 90
env:
# cc-rs recognises `sccache` as a compiler wrapper when it leads CC/CXX, and cmake-rs then
# forwards it as -DCMAKE_C_COMPILER_LAUNCHER, so this covers both build-script styles.
#
# ⚠ JOB-scoped, NOT workflow-scoped, and it must stay that way: the `rust-arm64` job below
# runs in the cross image, which sets CC_x86_64_unknown_linux_gnu=/usr/local/bin/pf-host-cc
# (ci/rust-ci-arm64cross.Dockerfile) — a wrapper that strips arm64 include dirs off
# HOST-targeted compiles so ffmpeg-sys-next's probe resolves against the amd64 headers.
# Setting this at workflow level would silently overwrite that wrapper and break the cross
# build in a way that looks like a header mismatch, not a CI config error.
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
steps:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
# punktfunk-client-linux link deps. Also baked into rust-ci.Dockerfile — but ci.yml
# runs against the image from the PREVIOUS push (docker.yml bootstrap note), so this
@@ -105,8 +123,17 @@ jobs:
# out of disk mid-build and actions/cache saved a truncated target/ (a dep's .rmeta
# went missing -> E0463 "can't find crate"). A suffix bump wouldn't help — restore-keys
# would fall back to the poisoned prefix — so the prefix itself is versioned.
key: cargo-target-v3-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-v3-${{ env.rustc }}-
#
# `-debug-`: THIS JOB BUILDS DEBUG. deb.yml builds RELEASE and used to share this exact
# key, with a comment claiming the release build "reuses ci.yml's clean artifacts" — it
# never could. actions/cache is first-saver-wins on an exact key and this job is the
# faster of the two, so what landed under the shared key was always a debug-only target/
# (target/debug, no target/release). deb.yml restored a tree containing nothing it could
# use and, because the key was already taken, never got to save its own — so every
# release build re-linked from scratch, forever. Splitting the profiles into separate key
# families is the fix; do not merge them again, however tempting the dedupe looks.
key: cargo-target-debug-v3-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-debug-v3-${{ env.rustc }}-
- name: Format
run: cargo fmt --all --check
@@ -220,14 +247,9 @@ jobs:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
- name: Cache keys
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
@@ -242,8 +264,16 @@ jobs:
with:
path: target
# Its OWN prefix: aarch64 artifacts must never share the amd64 jobs' target cache.
key: cargo-target-arm64-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-arm64-v1-${{ env.rustc }}-
#
# `-debug-` in the name is load-bearing. This job builds DEBUG (clippy + a
# `cargo build`), while deb.yml's arm64 leg builds RELEASE into the same
# target/aarch64-unknown-linux-gnu tree. They used to share this exact key, and
# actions/cache is first-saver-wins on an exact key: this job finishes in ~1.5 min and
# saved a debug-only tree, so the .deb leg's release artifacts were NEVER persisted and
# it re-linked everything from sccache on every run. Same disease as the amd64 pair —
# see the note on deb.yml's `cargo-target-release-v1-` key.
key: cargo-target-arm64-debug-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-arm64-debug-v1-${{ env.rustc }}-
- name: Clippy for aarch64 (deny warnings)
run: |
@@ -259,6 +289,14 @@ jobs:
cargo build --release --target aarch64-unknown-linux-gnu --locked \
-p punktfunk-client-session --no-default-features
# Visibility only — but the ONLY way to tell a warm cache from a cold one after the fact.
# Every Rust job in this repo ends with this line for that reason; a hit rate that quietly
# collapses (a toolchain bump, a flag change, an S3 outage) is otherwise invisible and just
# looks like "CI got slower".
- name: sccache stats (visibility only)
if: always()
run: sccache --show-stats
web:
runs-on: ubuntu-24.04
container:
+78 -32
View File
@@ -70,6 +70,11 @@ env:
SCCACHE_REGION: home-central
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_SECRET_ACCESS_KEY }}
# CMake-driven C/C++ through the same cache (aws-lc-sys, the vendored libopus). Workflow level
# is safe — it names no triple; the CC_*/CXX_* half is per-job below, because the arm64 leg's
# image needs its own CC_x86_64 wrapper. See ci.yml's `rust` job for the full note.
CMAKE_C_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
# sccache and incremental compilation are mutually exclusive; CI wants the shared
# cache, dev boxes keep incremental.
CARGO_INCREMENTAL: "0"
@@ -80,18 +85,18 @@ jobs:
container:
image: 192.168.1.58:5010/punktfunk-rust-ci:latest
timeout-minutes: 90
env:
# ⚠ Job-scoped, never workflow-scoped: build-publish-client-arm64 runs in the cross image,
# whose own CC_x86_64_unknown_linux_gnu=pf-host-cc must survive. See ci.yml's `rust` job.
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
steps:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
- name: Version + channel
# vX.Y.Z tag -> X.Y.Z, published to the `stable` apt distribution (a real release).
@@ -129,7 +134,9 @@ jobs:
apt-get install -y --no-install-recommends dpkg-dev python3 \
libgtk-4-dev libadwaita-1-dev libsdl3-dev
# Share ci.yml's cache keys so the release build reuses its registry + target artifacts.
# The cargo-home (registry/git) cache IS shared with ci.yml — those are pure downloads,
# profile-independent, and deduping them across the fleet is the whole point. The target
# cache is NOT; see below.
- name: Cache keys
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
- uses: actions/cache@v4
@@ -142,10 +149,20 @@ jobs:
- uses: actions/cache@v4
with:
path: target
# -v3-: bypass a target cache poisoned by a disk-full build (see ci.yml). Shares the
# key with ci.yml so the release build reuses its clean artifacts.
key: cargo-target-v3-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-v3-${{ env.rustc }}-
# -v3-: bypass a target cache poisoned by a disk-full build (see ci.yml).
#
# `-release-`, and NOT ci.yml's key. This step used to read
# `cargo-target-v3-<rustc>-<lock>` — byte-identical to ci.yml's — under a comment saying
# the release build "reuses its clean artifacts". It never did, and could not: ci.yml
# builds DEBUG, this job builds RELEASE, and actions/cache is first-saver-wins on an
# exact key. ci.yml's `rust` job finishes in ~6 min against this job's ~7-8, so ci.yml
# always won the save, this job always restored a target/ with an empty target/release,
# and — the expensive half — its own release artifacts were then never persisted,
# because the key it would have saved under was already taken. Every canary .deb has
# therefore been a from-scratch release build (sccache-assisted, but every link and
# every build script re-run) for as long as both keys have existed.
key: cargo-target-release-v3-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-release-v3-${{ env.rustc }}-
- name: Build release clients
env:
@@ -176,7 +193,12 @@ jobs:
}
export PATH="$HOME/.bun/bin:$PATH"
cd web
bun install --frozen-lockfile
# --ignore-scripts, like every other web install in CI (ci.yml, web-screenshots.yml,
# windows-host.yml, the SDK installs). This was the ONE site still running lifecycle
# scripts, and web's `postinstall` is `bun2nix -o bun.nix` — a Nix codegen step this job
# neither consumes nor commits, whose only effect here is to make the install depend on
# bun2nix resolving. `build` re-runs its own `prebuild` codegen regardless.
bun install --frozen-lockfile --ignore-scripts
bun run build
if ! grep -q 'Bun\.serve' .output/server/index.mjs; then
echo "ERROR: web build is not a bun bundle — need the 'bun' preset + custom entry"; exit 1
@@ -198,6 +220,13 @@ jobs:
# The plugin/script runner (bun-bundled Effect SDK) — same vendored-bun mechanics.
VERSION="$VERSION" BUN_BIN="$(command -v bun || true)" bash packaging/debian/build-scripting-deb.sh
# Visibility only. With the target cache now actually saving release artifacts (see the
# cache key note above), this is how a regression in that arrangement becomes visible:
# a run that suddenly reports thousands of misses is a cache that stopped restoring.
- name: sccache stats (visibility only)
if: always()
run: sccache --show-stats
- name: Publish to the Gitea apt registry
env:
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
@@ -242,18 +271,17 @@ jobs:
container:
image: 192.168.1.58:5010/punktfunk-rust-ci-noble:latest
timeout-minutes: 90
env:
# ⚠ Job-scoped — see build-publish above and ci.yml's `rust` job.
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
steps:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
- name: Version + channel
run: |
@@ -289,9 +317,12 @@ jobs:
with:
path: target
# Own key: this target dir is built against 24.04's glibc/toolchain and must NOT share
# ci.yml's 26.04 target cache (mixing would poison both).
key: cargo-target-noble-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-noble-v1-${{ env.rustc }}-
# ci.yml's 26.04 target cache (mixing would poison both). It is also the only job using
# this prefix, so — unlike the amd64/arm64 pairs above — it has always been able to save
# and restore its own release artifacts. `-release-` is spelled out anyway so the whole
# file reads consistently and nobody "unifies" it back into a shared key later.
key: cargo-target-noble-release-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-noble-release-v1-${{ env.rustc }}-
- name: Build release host
env:
@@ -326,6 +357,12 @@ jobs:
run: |
VERSION="$VERSION" BUNDLE_FFMPEG=1 bash packaging/debian/build-deb.sh
# Visibility only — placed here, right after the last compile, rather than at the end of the
# job: the gamescope gate below must stay the final step (see its own note).
- name: sccache stats (visibility only)
if: always()
run: sccache --show-stats
# Read the capability matrix out of the BUILT .deb before it is published. dpkg carries no
# capability metadata — the postinst applies them — so this reads the postinst that will
# actually run on a user's box, plus the payload. 0.26.0-1 granted the host cap_sys_nice=ep
@@ -476,14 +513,14 @@ jobs:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
#
# NOTE this job deliberately sets no CC_x86_64_unknown_linux_gnu: the cross image already
# points it at /usr/local/bin/pf-host-cc, which is what keeps ffmpeg-sys-next's HOST probe
# from picking up arm64 include dirs. The target-side compiles go through
# CC_aarch64_unknown_linux_gnu (also set by the image) and are not sccache-wrapped.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
# Byte-identical to build-publish's version step (pf-version.sh is deterministic per
# commit), so the arm64 package always shares the amd64 version line.
@@ -520,8 +557,13 @@ jobs:
path: target
# Its OWN key — these are aarch64 artifacts under target/aarch64-unknown-linux-gnu/
# and must never share the amd64 jobs' target cache.
key: cargo-target-arm64-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-arm64-v1-${{ env.rustc }}-
#
# `-release-`: this key was previously identical to ci.yml's `rust-arm64` key, which
# builds DEBUG (clippy) and finishes in ~1.5 min against this job's ~5. Exactly the
# amd64 collision described on the release key above — ci.yml won every save, this job
# restored a tree with no release artifacts and could never persist its own.
key: cargo-target-arm64-release-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-arm64-release-v1-${{ env.rustc }}-
- name: Build the arm64 client .deb
env:
@@ -534,6 +576,10 @@ jobs:
readelf -h target/aarch64-unknown-linux-gnu/release/punktfunk-session \
| grep -q AArch64 || { echo "ERROR: session binary is not AArch64"; exit 1; }
- name: sccache stats (visibility only)
if: always()
run: sccache --show-stats
- name: Publish to the Gitea apt registry
env:
TOKEN: ${{ secrets.REGISTRY_TOKEN }}
+24 -10
View File
@@ -29,6 +29,12 @@ env:
SCCACHE_REGION: home-central
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_SECRET_ACCESS_KEY }}
# The C/C++ half of the cache — same wiring as ci.yml/deb.yml. Safe at workflow level: no
# cross-compiling job here.
CMAKE_C_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
# sccache and incremental compilation are mutually exclusive; CI wants the shared
# cache, dev boxes keep incremental.
CARGO_INCREMENTAL: "0"
@@ -45,14 +51,9 @@ jobs:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
# Client link deps (baked into the image; kept here so the job is green across image
# rebuilds — a no-op once present) PLUS the headless-render extras: a virtual X server,
@@ -67,7 +68,8 @@ jobs:
libgl1-mesa-dri mesa-vulkan-drivers \
adwaita-icon-theme fonts-cantarell fonts-dejavu-core
# Reuse the workspace cargo caches (same keys as ci.yml/deb.yml).
# Reuse the workspace cargo caches. The cargo-home (download) cache is shared verbatim —
# it is profile-independent.
- name: Cache keys
run: echo "rustc=$(rustc --version | cut -d' ' -f2)" >> "$GITHUB_ENV"
- uses: actions/cache@v4
@@ -80,8 +82,20 @@ jobs:
- uses: actions/cache@v4
with:
path: target
key: cargo-target-v3-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: cargo-target-v3-${{ env.rustc }}-
# This job builds RELEASE (see the build step) in the same image and target layout as
# deb.yml's `build-publish`, so it wants THAT tree — it used to name ci.yml's key, which
# holds a debug build and gave it nothing. (Third participant in the collision documented
# on ci.yml's `cargo-target-debug-v3-` key.)
#
# Its OWN exact key with deb's prefix as a FALLBACK restore-key, deliberately: both
# workflows run on a v* tag, and an exact-key match would make them race for the single
# save slot — this job builds one crate, so if it won that race it would replace deb's
# full release tree with a nearly empty one for the rest of the lockfile's life. This way
# it always READS the warm tree and never blocks the job that fills it.
key: cargo-target-shots-release-v1-${{ env.rustc }}-${{ hashFiles('Cargo.lock') }}
restore-keys: |
cargo-target-shots-release-v1-${{ env.rustc }}-
cargo-target-release-v3-${{ env.rustc }}-
- name: Build client
run: cargo build --release -p punktfunk-client-linux --locked
+16 -7
View File
@@ -51,6 +51,14 @@ env:
SCCACHE_REGION: home-central
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_SECRET_ACCESS_KEY }}
# The C/C++ half of the cache (aws-lc-sys, the vendored libopus, openh264's C++). Safe at
# workflow level here: unlike ci.yml/deb.yml this workflow has no cross-compiling job whose
# image sets its own CC_x86_64_unknown_linux_gnu. See ci.yml's `rust` job for that trap.
# This matters twice per push — the f43 and f44 legs are the two longest jobs in the fleet.
CMAKE_C_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CC_x86_64_unknown_linux_gnu: sccache cc
CXX_x86_64_unknown_linux_gnu: sccache c++
# sccache and incremental compilation are mutually exclusive; CI wants the shared
# cache, dev boxes keep incremental.
CARGO_INCREMENTAL: "0"
@@ -80,14 +88,9 @@ jobs:
- uses: actions/checkout@v4
# Shared compile cache (sccache -> RustFS S3 over the LAN). Baked into the builder
# images; this fetch keeps the job green while the running :latest predates the bake.
# images; this heals the job while the running :latest predates the bake.
- name: sccache (no-op once the image bakes it)
run: |
command -v sccache >/dev/null 2>&1 || {
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache'
}
sccache --version
run: sh scripts/ci/ensure-sccache.sh
# rpmbuild + git archive need the checkout trusted; cache the crates download.
# The client link deps are also baked into the fedora-rpm image, but this job runs
@@ -151,6 +154,12 @@ jobs:
# Recommends both). Both need bun (ensured in Prep).
run: PF_VERSION="$PF_VERSION" PF_RELEASE="$PF_RELEASE" PF_WITH_WEB=1 PF_WITH_SCRIPTING=1 bash packaging/rpm/build-rpm.sh
# Visibility only — the two RPM legs are the longest jobs in the fleet, so a cache
# regression here is the most expensive one to leave undetected.
- name: sccache stats (visibility only)
if: always()
run: sccache --show-stats
# Signs with packages@unom.io (org secret) and self-verifies before publish. On a v* tag a
# missing key FAILS the build rather than publishing unsigned RPMs into a gpgcheck=1 repo.
- name: Sign RPMs
+18
View File
@@ -48,6 +48,11 @@ RUN pacman -Syu --noconfirm --needed \
hwdata luajit seatd sdl2-compat vulkan-icd-loader \
xcb-util-errors xcb-util-wm xorg-xwayland \
meson glm wayland-protocols benchmark libxcursor \
# mold: link-phase accelerator (sccache cannot cache linking). makepkg links the release
# host, client, worker and tray on every arch.yml run. Wired via cargo-config-mold.toml
# below. It does NOT affect the gamescope companion leg — that is meson + its own linker,
# and its `-static-libstdc++` link is untouched.
mold \
&& pacman -Scc --noconfirm
# bun builds the punktfunk-web console + the punktfunk-scripting runner AND is vendored
@@ -64,3 +69,16 @@ ARG SCCACHE_VERSION=0.10.0
RUN curl -fsSL "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache' \
&& sccache --version
# CARGO_HOME is declared here only so this image agrees with what arch.yml already sets at job
# level (and so `cargo` finds the config below when the image is used by hand). The workflow still
# passes CARGO_HOME explicitly across the `sudo -u builder env …` boundary, which strips ambient
# env — that is why the C/C++ sccache wiring has to be re-exported there by name while THIS file,
# being a file, crosses the boundary for free.
ENV CARGO_HOME=/usr/local/cargo
RUN mkdir -p /usr/local/cargo && chmod -R a+w /usr/local/cargo
# Link x86_64 with mold — see cargo-config-mold.toml's header for the rustflags traps, and
# rust-ci.Dockerfile for why the `mold --version` assertion sits next to the COPY.
COPY cargo-config-mold.toml /usr/local/cargo/config.toml
RUN mold --version && test -r /usr/local/cargo/config.toml
+42
View File
@@ -0,0 +1,42 @@
# Installed as $CARGO_HOME/config.toml in every Linux CI builder image (ci/*.Dockerfile).
#
# WHAT: link the x86_64 Linux targets with mold instead of GNU ld. Linking is the one phase of a
# Rust build that sccache CANNOT cache — every job relinks punktfunk-host, punktfunk-client-linux,
# punktfunk-client-session, punktfunk-cli, pf-update and punktfunk-encode-worker from scratch on
# every run, and the packaging legs (deb/rpm/arch) do it for release binaries with full debug info.
# mold is the only lever that touches that phase.
#
# ⚠ THE TRAP THIS FILE HAS TO STAY CLEAR OF — read before editing, and before adding rustflags
# anywhere else in this repo:
#
# 1. A `RUSTFLAGS` ENVIRONMENT VARIABLE OVERRIDES CONFIG RUSTFLAGS ENTIRELY. It does not merge
# and it does not append. Any job that sets RUSTFLAGS silently loses mold here (it still
# builds — just with the default linker), and, far worse, would lose the aarch64
# `--cfg aes_armv8` / `--cfg polyval_armv8` flags from the workspace's own .cargo/config.toml,
# which are worth a measured ~3x on the decrypt path. audit.yml's miri gf8 step is the one
# place in the repo that sets RUSTFLAGS, and its comment already carries this warning; keep it
# that way. Never "simplify" this file into a RUSTFLAGS export.
#
# 2. CONFIG FILES MERGE PER KEY, HIGHEST-PRECEDENCE FILE WINS — they do not concatenate. The
# workspace's .cargo/config.toml outranks this one ($CARGO_HOME is the LOWEST precedence).
# Today that is harmless because the two files touch DISJOINT keys: the workspace file defines
# only `target.'cfg(target_arch = "aarch64")'.rustflags`, this one only
# `target.x86_64-unknown-linux-gnu.rustflags`, and cargo JOINS a matching cfg-spec table with
# the triple table rather than picking one. But the moment someone adds an x86_64 rustflags
# entry to the workspace .cargo/config.toml, IT WINS and mold silently stops being used here.
# If that ever happens, move the link-arg into that file instead of duplicating it.
#
# 3. aarch64 IS DELIBERATELY NOT WIRED. The cross image links with aarch64-linux-gnu-gcc against a
# multiarch sysroot (ci/rust-ci-arm64cross.Dockerfile); pointing that driver at mold is a
# separate thing to prove, and those legs are already the fast ones (~1.5 min of clippy, ~5 min
# for the arm64 .deb). Add it only with a measurement, and in a commit of its own.
#
# Requires GCC >= 12.1 (or clang) for `-fuse-ld=mold`; every base here ships far newer. mold itself
# is installed in the same Dockerfile layer that copies this file, so an image can never carry the
# flag without the linker — see the `mold --version` assertion there.
#
# NOTE this affects the HOST-targeted compiles of build scripts and proc macros too (they are
# x86_64-unknown-linux-gnu), which is exactly what we want: those link constantly and are pure
# overhead.
[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=mold"]
+11
View File
@@ -22,6 +22,12 @@ RUN dnf -y install \
rpm-build rpmdevtools systemd-rpm-macros git tar gzip nodejs unzip \
# build toolchain + bindgen
gcc gcc-c++ clang clang-devel cmake nasm pkgconf-pkg-config curl ca-certificates \
# mold: link-phase accelerator (sccache cannot cache linking). This image links the release
# host, client, worker and tray on every rpm.yml run, TWICE per push (f43 + f44). Wired via
# cargo-config-mold.toml below. Note the linker DRIVER is unchanged — still gcc, so Fedora's
# default `-Wl,--build-id` still reaches the link and rpmbuild's debuginfo extraction (which
# hard-requires a build-id) behaves exactly as before; mold implements --build-id natively.
mold \
# ffmpeg (NVENC), capture/audio/display link deps
ffmpeg-devel pipewire-devel wayland-devel libxkbcommon-devel opus-devel \
mesa-libGL-devel mesa-libgbm-devel \
@@ -76,3 +82,8 @@ ARG SCCACHE_VERSION=0.10.0
RUN curl -fsSL "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache' \
&& sccache --version
# Link x86_64 with mold — see cargo-config-mold.toml's header for the rustflags traps, and
# rust-ci.Dockerfile for why the `mold --version` assertion sits next to the COPY.
COPY cargo-config-mold.toml /usr/local/cargo/config.toml
RUN mold --version && test -r /usr/local/cargo/config.toml
+10
View File
@@ -26,6 +26,9 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
# toolchain + bindgen; nodejs runs the JS actions (checkout/cache); unzip for the rustup installer's deps
build-essential clang libclang-dev pkg-config cmake git curl ca-certificates nodejs unzip \
# mold: link-phase accelerator (sccache cannot cache linking). This image links the release
# host + encode worker on every deb.yml run. Wired via cargo-config-mold.toml below.
mold \
# .deb assembly: dpkg-shlibdeps/dpkg-deb; patchelf repoints the binary's rpath at the bundled FFmpeg
dpkg-dev patchelf \
# FFmpeg 8 build deps: nasm (asm), VAAPI (libva/libdrm) so the built libav* keep the AMD/Intel
@@ -99,3 +102,10 @@ ARG SCCACHE_VERSION=0.10.0
RUN curl -fsSL "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache' \
&& sccache --version
# Link x86_64 with mold — see cargo-config-mold.toml's header for the rustflags traps, and
# rust-ci.Dockerfile for why the `mold --version` assertion sits next to the COPY.
# ⚠ This does NOT touch the from-source FFmpeg built above: that is a plain ./configure && make in
# an earlier layer, linked by GNU ld exactly as before. Only cargo's links move to mold.
COPY cargo-config-mold.toml /usr/local/cargo/config.toml
RUN mold --version && test -r /usr/local/cargo/config.toml
+12
View File
@@ -13,6 +13,9 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
# toolchain + bindgen; nodejs runs the JS actions (checkout/cache); unzip is for the bun installer
build-essential clang libclang-dev pkg-config cmake git curl ca-certificates nodejs unzip \
# mold: the link-phase accelerator. Linking is the one thing sccache cannot cache, and this
# image relinks the whole workspace on every job. Wired via cargo-config-mold.toml below.
mold \
# ffmpeg-next 9, built against whatever libav* 26.04 ships (FFmpeg 8 / libavcodec 62 today).
# The crate major is a CEILING — ffmpeg-sys-next 9 spans libavcodec 56..63 — so this image does
# not need to move in lockstep with Arch's FFmpeg 9; it just links what the distro has.
@@ -61,3 +64,12 @@ ARG SCCACHE_VERSION=0.10.0
RUN curl -fsSL "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
| tar -xz --wildcards --strip-components=1 -C /usr/local/bin '*/sccache' \
&& sccache --version
# Link x86_64 with mold (see the file's own header for the rustflags-precedence traps).
#
# The assertion is the point: an image carrying the flag but NOT the linker would fail every cargo
# invocation in every consuming job, which is a catastrophic way to find out that a base image
# renamed the package. `mold --version` fails the docker build instead, so nothing is pushed and
# `:latest` keeps pointing at the previous working image — consumers never see it.
COPY cargo-config-mold.toml /usr/local/cargo/config.toml
RUN mold --version && test -r /usr/local/cargo/config.toml
+65
View File
@@ -0,0 +1,65 @@
#!/bin/sh
# Ensure `sccache` is on PATH, self-healing on a runner/image that does not already carry it.
#
# WHY THIS EXISTS: this block was copy-pasted into ten jobs across six workflows (ci.yml x2,
# deb.yml x3, rpm.yml, bench.yml, apple.yml x2, the Apple release leg), in two dialects that had
# already drifted apart — the Linux copies pass `--wildcards` to GNU tar, the macOS copies must NOT
# (bsdtar globs by default and rejects the flag). One copy per platform, here, so a version bump or
# a mirror change is one edit rather than ten.
#
# The builder images (ci/*.Dockerfile) BAKE sccache, so on Linux this is a no-op in the normal case;
# it stays because a job runs against the image from the PREVIOUS push (docker.yml's bootstrap lag),
# and the macOS runner is a persistent host with no image at all.
#
# POSIX sh on purpose: Gitea's act_runner executes a step's `run:` under `sh -e` (dash) inside the
# Linux job containers — no bashisms, no process substitution (see the shader-gate note in ci.yml
# for what that cost the last time someone assumed bash).
#
# Usage: sh scripts/ci/ensure-sccache.sh
set -e
# Keep in step with the ARG SCCACHE_VERSION in ci/*.Dockerfile — the images bake this same version,
# and a job that heals to a DIFFERENT one would quietly split the shared cache's key universe in two
# (sccache's cache keys are not versioned across incompatible releases).
SCCACHE_VERSION="${SCCACHE_VERSION:-0.10.0}"
if command -v sccache >/dev/null 2>&1; then
sccache --version
exit 0
fi
BASE="https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}"
case "$(uname -s)" in
Darwin)
# The macOS runner is a LaunchAgent in the user's Aqua session, not root — install into the
# user prefix. ~/.local/bin is already on the runner daemon's PATH; GITHUB_PATH is
# belt-and-braces for the steps that follow in THIS job.
DEST="$HOME/.local/bin"
mkdir -p "$DEST"
case "$(uname -m)" in
arm64|aarch64) ARCH=aarch64-apple-darwin ;;
*) ARCH=x86_64-apple-darwin ;;
esac
# bsdtar globs by default and does not accept --wildcards.
curl -fsSL "$BASE/sccache-v${SCCACHE_VERSION}-${ARCH}.tar.gz" \
| tar -xz --strip-components=1 -C "$DEST" '*/sccache'
chmod 0755 "$DEST/sccache"
PATH="$DEST:$PATH"
export PATH
if [ -n "${GITHUB_PATH:-}" ]; then
echo "$DEST" >> "$GITHUB_PATH"
fi
;;
*)
# Linux job containers run as root; /usr/local/bin is on PATH already, so no GITHUB_PATH
# dance is needed. The musl build is static — one binary serves the Ubuntu, Fedora and Arch
# images alike.
DEST=/usr/local/bin
curl -fsSL "$BASE/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
| tar -xz --wildcards --strip-components=1 -C "$DEST" '*/sccache'
chmod 0755 "$DEST/sccache"
;;
esac
sccache --version