heads
heads copied to clipboard
Revise CircleCI caching strategies
Context:
- Clean build without cache on nix docker image: build time 1h6 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/7e6f69f6-7568-4edb-b3cf-a5c59fdf94ef
- Build with cache on nix docker image (rebuild to reuse cache): build time 1h8 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b
Cache speeding up builds : fail.
- Time to download cache layer: 20 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45679
- Attaching a workspace prior of build : 2-3 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45682/parallel-runs/0/steps/0-101
- Persisting build to next board being dependent: 5 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45682/parallel-runs/0/steps/0-108
- Save cache (useless since same cache on rebuild) : 13 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45708
Basically we loose around 45 minutes, some of which blocks other boards from building.
Some thoughts:
- We should build muslcross alone on circleci and chache and reuse that.
- For the rest i'm not sure I see the point of using caching anymore for to save CPU time from CI at the cost of more bandwidth and waiting time prior of starting builds in most use case
- On nix, we could put musl-cross-make as first step for both x86 and ppc64 under docker image and modify Makefile to check if provided per nix and if the case use it
- advance on prior step and use nix coreboot-buildstacks
After that, optimizations would come from moving all modules into nix expressions. But I'm not sure we want to go there yet. Seems to close contributions to nix developers which I do not categorize myself in still.
As confirmed under #1967
As said under #1666, we need insights on best next strategies to save cache/pass workspace caches under CircleCI per above links. Short version:
* No cache https://app.circleci.com/pipelines/github/tlaurion/heads/3315/workflows/9bc8a009-68c8-4547-9232-3ba3612ab0fb * restore cache step (prep_env) : 1m16s : no cache downloaded nor applied * musl-cross-make built from source: 16m36s * **total pipeline time: 1h24m (most of this time in compute time : no cache reused)** * First layer cache reusal (msul-cross-make) https://app.circleci.com/pipelines/github/tlaurion/heads/3314/workflows/e588480d-d13a-49e0-a1b6-78fed839b70b * restore cache step (prep_env) : 9m2s * musl-cross-make cache reused step time: 11m43s : https://app.circleci.com/pipelines/github/tlaurion/heads/3314/workflows/e588480d-d13a-49e0-a1b6-78fed839b70b/jobs/67601 * build time is still of 5m44s, rest is workspace download from prep_env step (attaching workspace step of 1m20s) and workspace creation (persisting to workspace step of 3m40s * **total build time : 1h24m29s (most of this time in computer time still for board builds but less cpu used because musl reused for ppc64/x86)** * third layer cache reusal (all built artifacts downloaded to minimize CPU build time) https://app.circleci.com/pipelines/github/tlaurion/heads/3315/workflows/c66c8669-327a-42d9-822a-47b2dc87241e * restore cache step (prep_env) : 16m9s * **t480-hotp-maximized : 18m30s here vs 42m6s: but only one board cache is used by arch right now** * total build time : 1h43m30s : we basically try to be more gentle for CircleCI CPU used time but do not economize much here and our board build time is less efficient than if we were not reusing any cache * To be efficient, we would need to: * Have musl-cross-make built into docker image for architectures we support instead of rebuilding from source, and/or * Combine workspace cache layers per arch, then have save_Cache depend on that layer so we have the same gains of the single cahced board build layer we currently have but for all boards (this is currently 50% less build time whe board cache layer is used for talos-2 and t480, which layer is reused only for board depending on it, otherwise twice time spent in building from source all coreboot and modules