DEMO: `docker buildx bake`

[morganchristiansson]

Hello again!

While doing arm64 docker builds with docker buildx build I came across the docker buildx bake command.

It concurrently builds all images from a single command / actions runner and uses resources more efficently. The GH action runner is 2-core cpu with 7gb ram and doesn't have any trouble. (And can still use multiple GH runners for different groups).

Buildx also aims to provide support for high-level build concepts that go beyond invoking a single build command. We want to support building all the images in your application together and let the users define project specific reusable build flows that can then be easily invoked by anyone.

BuildKit efficiently handles multiple concurrent build requests and de-duplicating work. The build commands can be combined with general-purpose command runners (for example, make). However, these tools generally invoke builds in sequence and therefore cannot leverage the full potential of BuildKit parallelization, or combine BuildKit’s output for the user. For this use case, we have added a command called docker buildx bake.

So... I thought it's potentially a good fit instead of the GH matrix. And if building another platform image would double the number of builds, there's 100s of them...

I created PR to test.

• I created scripts/generate-docker-bake-helm.sh to generate docker-bake.hcl to build all helm images.
• Removes matrix strategy from .github/workflows/build.yaml
• I updated .github/workflows/build.yaml to run docker buildx bake --load --set *.cache-from=type=gha --set=*.cache-to=type=gha,mode=max and updated the test stages to loop each ansible version biuld for VERSION in $ANSIBLE_VERSION ; do xxx ; done in bash to make test ANSIBLE=$VERSION.

Also added --cache-to=type=gha option to build, which is new in buildkit I think. But it's too small to stay warm for next build at 10gb LRU size from test runs. Not sure you actually want cached runs tho..? It's helpful when making changes..

I was imagining a generated docker-bake.hcl per .github/workflows/build{,-kops,-helm}. It could be used for everything: GH runners and local dev etc.

It runs pretty fast compared to separate runner for every ansible version. I thought buildx bake is especially interesting here because there's so many images to build...

For example: docker buildx bake -f docker-bake.hcl -f docker-bake-kops.hcl -f docker-bake-helm.hcl [TARGET...] and can list image tags as targets.

Have a look at the actions output and lmk what you think? buildkit has a bit different output and bake runs stuff in parallel so its a bit messy now compared to having them output separate per worker..

[cytopia]

@morganchristiansson thanks for all the effort so far and raising this topic! :+1: I've now also started playing around with arm64 builds. Currently in the phase of finding a very generic solution that fits all my docker images. It will take some more time and energy, but will report back and then also try to combine best efforts from your solution into one single generic one.

[morganchristiansson]

Just seen this.

I'm maintaining a small fork at https://github.com/morganchristiansson/docker-ansible/ with only a few issues left for merging.

Some niggling issues are aws-iam-authenticator and openshift-client arm64 binaries but they are solvable. helm-diff recently unblocked with arm64 binary.

Also building arm64 with qemu uses lots of github action cpu time. There's just so many jobs queued up when adding arch=arm64 to matrix builds. docker buildx bake could potentially speed up and simplify building all these images.

Another option arm64 native build runners in circleci/travisci - not using qemu will speed up arm64 builds a lot. docker build cache also helps a lot.

Sorry for late reply I'm happy to refresh my PR or carve it up into smaller PRs.

[morganchristiansson]

I also had some partial success in having openshift-client in amd64 docker images only https://github.com/morganchristiansson/docker-ansible/pull/6

I don't think anyone expects openshift-client in arm64 docker image? So this would unblock openshift-client binary.