ci: cache the C/C++ half, link with mold, fix the debug/release cache collision, consolidate the Apple and Windows-client workflows #191
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: unom/punktfunk#191
Reference in New Issue
Block a user
Four commits. The first three are pure speed with no artifact change; the fourth is organisational and fixes one real ordering bug.
Measured baseline for one
crates/**push to main (2026-08-13, run981f32b8): ~115 job-minutes across 8 workflows, last finisher ~30 min after the push.1.
f1dc6c9f— sccache for the C/C++ half, mold for linking, and the cache-key collisionThree reasons Rust CI stayed slow despite sccache:
sccache only ever covered rustc. Every C/C++ dependency — aws-lc-sys, openh264-sys2's vendored C++, the CMake-built libopus behind audiopus_sys — was compiled from scratch on every job of every workflow.
CMAKE_{C,CXX}_COMPILER_LAUNCHERplusCC_/CXX_x86_64_unknown_linux_gnuroute both build-script styles (cc-rs and cmake-rs) through the same shared cache.Linking is cacheable by nothing, and these jobs relink host, client, session, cli, worker and tray every run — twice per push for rpm (f43 + f44). The four Linux builder images now install mold and carry a
$CARGO_HOME/config.tomlusing it for x86_64. aarch64 deliberately untouched (cross driver, already-fast legs). Each image assertsmold --versionin its own build, so it can never ship the flag without the linker: docker.yml goes red and:lateststays on the last good image.The expensive one. ci.yml (debug) and deb.yml (release) named a byte-identical target-cache key, under a comment claiming the release build reused ci.yml's artifacts. It never could: actions/cache is first-saver-wins on an exact key and ci.yml is the faster job, so the shared key always held a debug-only
target/— and deb.yml could then never save its own, because the key was taken. Every canary .deb has been a from-scratch release build for as long as both keys existed. Same collision on the arm64 pair, plus a third participant found while implementing:linux-client-screenshots.ymlbuilds--releaseunder ci.yml's key. Split into-debug-/-release-families; the screenshots job reads deb's tree viarestore-keysbut keeps its own exact key so it can never win the save race and replace a full tree with its single-crate one.Also: one
scripts/ci/ensure-sccache.shreplaces ten copy-pasted bootstrap blocks that had drifted into two dialects (GNU tar--wildcardsvs bsdtar); every Rust job ends with--show-stats; deb.yml's web install joins every other CI install on--ignore-scripts.2.
3f7fbf10— the web console built once per push, not six timesIt's a pure function of
web/andsdk/, and it was being built by ci.yml, deb, both RPM legs, arch and the docker app image at ~2.5 min each. windows-host.yml has cached it this way for a while; this extends the same arrangement, sharing one key family. The bun version is in the key, so the three builder images share while they agree and stop sharing when they diverge, rather than one image's bun silently producing the bundle another ships.Each path needed a different hand-off — rpm needed the most care:
build-rpm.shpackages agit archivetarball andweb/.outputis gitignored (so a workspace bundle is invisible to rpmbuild), and the spec's own build lands in rpmbuild's%{_topdir}, which is mktemp'd and removed on EXIT (so a console built there is gone before the cache's post step, and the cache would never populate — a miss that quietly rebuilt forever). The workflow builds it and hands it over through a new optionalpf_prebuilt_webmacro; undefined (plain rpmbuild, COPR) takes the original path unchanged.Every path asserts the bundle exists and carries the
Bun.servemarker, on cache hits too — the packaging scripts' build-if-missing behaviour is right for a local build and would otherwise turn a broken restore into a silent rebuild or a console-less package. windows-host.yml shipped 0.22.1 and 0.22.2 with no console for exactly that reason.3.
0bfc7fe9— release.yml → apple.yml, windows.yml + windows-msix.yml → windows-client.ymlrelease.ymlwas Apple-only distribution under a name that described neither its contents nor how releases actually work here. The name was the smaller problem: Gitea has no cross-workflowneeds, so nothing sequenced it against apple.yml's tests — a canary main push uploaded iOS, macOS and tvOS builds to TestFlight even whenswift testhad just failed on the same commit. It is nowneeds: swift, which is only expressible in one file.The two files'
paths:had also drifted (crates/**vscrates/punktfunk-core/**). The merged filter takes the narrow one, because that is the correct one: everything on that runner is built from punktfunk-core via build-xcframework.sh, and punktfunk-core's only path dependency is its own vendored fec-rs. The header carries the one-line check and says to widen it if that stops being true. Net effect: pushes touching host-side crates no longer occupy the shared mac mini at all.The Windows pair built the same three crates four times per client push on one runner (debug x64 + arm64 for lint/test, release x64 + arm64 for packaging). windows-host.yml already records why a second debug dep tree on that machine is a liability rather than a cost — it re-runs openh264-sys2's vendored C++ through cc-rs's cl.exe fan-out and tips the runner into C1069, which is disk exhaustion wearing a compiler error's clothes. One release build per arch now, with clippy/fmt/test against it.
The rename is safe, and this is worth recording because the GitHub instinct is wrong here:
github.run_numberis repo-wide in Gitea, not per-workflow — consecutive runs of different workflows get consecutive numbers (verified against the API: android 13226, apple 13227, arch 13228, ci 13229, deb 13230). The canary MSIX version<minor>.<run>.0and Apple'sCURRENT_PROJECT_VERSIONkeep climbing across a rename. On GitHub the same rename would reset both to 1, sorting every canary below the published ones and getting the TestFlight uploads rejected.25 workflows, down from 27; every
name:now matches its filename.Dropped: extracting the version stamp into a
pf-buildinfocrateThe plan called for it, on the theory that
PUNKTFUNK_BUILD_VERSIONchanging every canary run recompiles punktfunk-host and clients/linux for nothing. Tested with a two-crate probe before writing any code: changing the stamp rebuilt the tiny crate and its dependent, because cargo invalidates dependents of any rebuilt crate. The refactor would have moved the recompile, not removed it. No code written.What is not verified
Static checks only: all 25 workflows parse, job graphs resolve, shell/PKGBUILD/spec syntax is clean, no dangling cross-references. Not verified: that
moldis packaged under that name on all four bases, that the sccache C/C++ wrapping engages inside those images, and the rpm macro end-to-end.Two operational notes:
pull_requesttrigger, so the builder images are rebuilt after merge, not during PR CI. The mold assertion runs there. If a package name is wrong, docker.yml goes red and consumers keep working on the previous:latest— that failure is contained by design.Canary cadence for every workflow is unchanged; flatpak stays per-push so Deck canaries keep tracking main within the hour.
Three independent reasons Rust CI stayed slow despite sccache, fixed together because they share the same measurement. 1. sccache only ever covered RUSTC. Every C/C++ dependency in the tree — aws-lc-sys, openh264-sys2's vendored C++, the CMake-built libopus behind audiopus_sys — was compiled from scratch on every job of every workflow. CMAKE_{C,CXX}_COMPILER_LAUNCHER plus CC_/CXX_x86_64_unknown_linux_gnu route both build-script styles (cc-rs and cmake-rs) through the same shared cache. The CC_* vars are JOB-scoped in ci.yml and deb.yml, never workflow-scoped: the arm64 cross image sets its own CC_x86_64_unknown_linux_gnu=pf-host-cc, the wrapper that keeps ffmpeg-sys-next's host probe off the arm64 include dirs. Overwriting it would surface as a header mismatch rather than as a CI config error. 2. Linking is cacheable by nothing, and these jobs relink the host, client, session, cli, worker and tray on every run — twice per push for rpm (f43 + f44). The four Linux builder images now install mold and carry a $CARGO_HOME/config.toml that uses it for x86_64. aarch64 is deliberately left alone (cross driver, already-fast legs). Each image asserts `mold --version` in its build, so an image can never ship the flag without the linker: docker.yml goes red and :latest stays on the last good one. 3. THE EXPENSIVE ONE. ci.yml (debug) and deb.yml (release) named a byte-identical target-cache key, under a comment claiming the release build reused ci.yml's artifacts. It never could. actions/cache is first-saver-wins on an exact key and ci.yml is the faster job, so the shared key always held a debug-only target/ — and, worse, deb.yml could then never save its own, because the key was taken. Every canary .deb has been a from-scratch release build for as long as both keys existed. Same collision on the arm64 pair, and a third participant in linux-client-screenshots.yml. Split into -debug-/-release- key families; that job reads deb's tree via restore-keys but keeps its own exact key so it can never win the save race and replace a full tree with its single-crate one. Also: one scripts/ci/ensure-sccache.sh replaces ten copy-pasted bootstrap blocks that had already drifted into two dialects (GNU tar --wildcards vs bsdtar), every Rust job now ends with --show-stats so a cache regression is visible instead of just "CI got slower", and deb.yml's web install joins every other CI install on --ignore-scripts. No behaviour change to any artifact: same compilers, same flags, same outputs.The Nitro console bundle is a pure function of web/ and sdk/, and it was being built six times on every push: ci.yml, deb, both RPM legs (f43 + f44), arch, and the docker app image, at roughly 2.5 min each. windows-host.yml has cached it on exactly this shape for a while — this extends the same arrangement to the Linux packaging legs, sharing one key family so whichever job builds it first warms the others. The bun version is part of the key. Each builder image runs the bun.sh installer at image-build time, so rust-ci, fedora-rpm and arch-ci can drift apart; keying on it means they share while they agree and simply stop sharing when they do not, rather than one image's bun silently producing the bundle another image ships. Each packaging path needed a different hand-off: * deb — build-web-deb.sh already builds only if web/.output is missing, so the restore alone is enough; the workflow's build+smoke step is now gated on the miss. * arch — makepkg builds with PF_SRCDIR pointing at the workspace, so a restored bundle is already where it needs to be. PKGBUILD gains the same build-if-missing guard the deb script has. * rpm — neither direction works by default. build-rpm.sh packages a `git archive` tarball and web/.output is gitignored, so a bundle in the workspace is invisible to rpmbuild; and the spec's own build lands in rpmbuild's %{_topdir}, which build-rpm.sh mktemps and removes on EXIT, so a console built there is gone before the cache's post step and the cache would never populate — every run a miss that quietly rebuilt. So the workflow builds it, and hands it over by absolute path through a new optional `pf_prebuilt_web` macro. Undefined (plain rpmbuild, COPR) takes the original build path. Every path asserts the bundle exists and carries the Bun.serve marker, on cache hits too. A cache is one more place a wrong artifact can come from, and the packaging scripts' build-if-missing behaviour — correct for a local build — would otherwise turn a broken restore into either a silent rebuild or, with the build step skipped, a package with no console in it. That is not hypothetical: windows-host.yml shipped 0.22.1 and 0.22.2 with no console because an unset path variable was handled by a single Write-Host, which is why its equivalent step throws.