hipcheck
hipcheck copied to clipboard
mitre
Speed up Docker image build
It's very slow (roughly 40 minutes): https://github.com/mitre/hipcheck/actions/runs/9617044102
Should look into how to speed this up. It's not a task we run frequently, but this still seems excessive.
The free tier github workflow runners aren't that beefy, so that's one issue. The other one is that yall aren't doing any caching for your docker layers both for downloading dependencies and actually compiling all the stuff (which to my understanding has been a significant pain point for rust for forever). I don't know if you want to necessarily introduce caching into your docker builds - we don't for the various SAF apps, but we're also ok with the usually <10min it takes to build Heimdall.
Yeah that all sounds right. The current state is with nothing done to make builds in Docker better.
One thing to be aware of in the area of improving the interaction between Rust builds and Docker image caching is cargo-chef. This is a custom Cargo subcommand designed to make Rust builds play nicely with Docker image layer caches by separating out a "build the dependencies" step from a "build the application" step in the Docker file.
Docker caches images layer-by-layer, and you can maximize the benefit of this caching by essentially ordering layers such that the layers which change more frequently are later in the files than the layers that change less frequently. Here, we are assuming that our own application code changes more frequently than our dependencies, and so we can benefit from caching dependencies separately to avoid expensive dependency rebuilds.
In our case at the moment, I don't know that this would help us substantially as we're only doing Docker builds when we release new versions (at minimum every 4 weeks), and we should generally expect at least some dependencies may change in that span.
However, we may make Docker builds more frequent in the future, and the cargo chef logic certainly shouldn't hurt build times much, so it may be useful anyway.
@cstepanian I think the first step here will be to work on some benchmarking in CI for this performance, ideally with some sort of historic tracking. This also ties into tracking we want to do for runtime performance. You, me, and @vcfxb should sync about these overlapping efforts.
I did some docker caching for rust in some previous open source work and the biggest improvement for me was building the whole project, stripping out any binaries and artifacts generated by my code (leaving ones from dependencies), saving the docker image, and then rebuilding again (this time just artifacts from the current crate). Rust doesn't play well with stuff like that by default, but it can be done.
Sounds similar to what cargo chef does! https://github.com/LukeMathWalker/cargo-chef