Arm64v8

[morganchristiansson]

As discussed in #73 for your review and suggestions.

As mentioned there's some outstanding tasks:

• ~~kubectl~~ aws-iam-authenticator uses aws eks vended version arm64 binaries. Should maybe not use eks vended binaries and this disabled the existing auto-update of patch versions.
• I removed oc as quick fix. There is arm64 binaries available at alternative sources need to investigate further.
• kops arm64 binaries are only released from kops 1.19 (latest is 1.20)
• the arm64 builds are pretty slow using qemu. TravisCI for example can run builds natively without qemu arm64 cpu emulator.
• the -amd64 and -arm64v8 images are not yet joined into the final multi-arch manifest image at the end of build.
• The number of github actions is excessive at 350+ and the arm64 especially takes long to complete.

Also I removed the retry logic because I don't want the failures to be retried, and it slows down feedback on failures while fixing errors. Unfortunately this is cluttering up the diff. I could remove it from the PR for review perhaps. Probably the retry could be extracted to a bash script instead.

[morganchristiansson]

I simplified the retry logic in a separate PR and rebased my changes on top of that. These changed can be viewed without the noise from removing retries here https://github.com/morganchristiansson/docker-ansible/pull/2/files

[morganchristiansson]

Also this PR uses github container registry for build cache. You may need to enable it to use it: https://docs.github.com/en/packages/working-with-a-github-packages-registry/enabling-improved-container-support-with-the-container-registry

There's also a docker registry that works largely the same I believe and doesn't need to be turned on. However docs said: The GitHub Packages Docker registry is superseded by the Container registry, which offers improved container support

You an even use multiple --cache-from, so --cache-from=type,registry,ref=ghcr.io/morganchristiansson/docker-ansible:${TAG} would use warm build cache on first run of arm64 builds in cytopia/docker-ansible.

[morganchristiansson]

Failed on make login USERNAME=${{ secrets.DOCKERHUB_USERNAME }} PASSWORD=${{ secrets.DOCKERHUB_PASSWORD }}

It's using same secrets as before.. The same branch is building fine with GH actions in my fork https://github.com/morganchristiansson/docker-ansible/pull/2

Btw I'm happy to complete this work with a final join step to create the multi-arch manifest tags... Let me know what you think about arm64 builds running in this repo/elsewhere...

[morganchristiansson]

Oh it will try to push the tags in my docker hub... docker buildx build --push is used.. Can remove docker buildx build --push flag.. ~~Some newer image builders write layers directly to/from registry without using local docker daemon.~~ In this case it's using local registry and should be able to push later as original behaviour. https://docs.docker.com/engine/reference/commandline/buildx_build/#output

[morganchristiansson]

I removed the --push it should use buildcache and for the builder image IMAGE_CACHE = ghcr.io/$(GITHUB_ACTOR)/docker-ansible.

Which needs to be enabled https://docs.github.com/en/packages/working-with-a-github-packages-registry/enabling-improved-container-support-with-the-container-registry

(it said the old github docker registry way was deprecated, I didn't realise ghcr that they're recommending as current is a toggle under feature previews).

[morganchristiansson]

Please enable github container registry in feature previews per my link above and re-run the actions. I've prepared the PR to use warm build cache from ghcr.io/morganchristiansson/docker-ansible (which is also viewable here https://github.com/users/morganchristiansson/packages/container/package/docker-ansible ). So all build should run fast!

(We will need an option to build without build cache also, atm it's always cached, which made dev&test work easier).

[cytopia]

I already had this enabled:

In this page I do not see any other settings.

[morganchristiansson]

The PR looks much better! Can actually read thru it now :+1:

The GH actions should have worked then... per the docs for gh container registry and buildx build I'm doing

docker buildx create --use
echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
IMAGE_CACHE = ghcr.io/$(GITHUB_ACTOR)/docker-ansible
docker buildx build \
		--cache-from type=registry,ref=$(IMAGE_CACHE):cache-builder-$(PLATFORM) \
		--cache-to type=registry,ref=$(IMAGE_CACHE):cache-builder-$(PLATFORM),mode=max

[morganchristiansson]

It will take a while for actions to run. This PR will be the 365 checks at the back of the queue

if my warm cache works it should run quite quick tho. (can supply multiple --cache-from)

IMAGE_CACHE_WARMUP = ghcr.io/morganchristiansson/docker-ansible
	--cache-from type=registry,ref=$(IMAGE_CACHE_WARMUP):cache-$(ANSIBLE)-$(FLAVOUR)-$(PLATFORM) \

[morganchristiansson]

I missed enabling buildcache from ghcr.io/morganchristiansson on builder image, which felt like one of the slowest images to build and it runs in every action so thought it'd be worth pushing. However my push triggered 8 extra runs. You can cancel all but top 4 here to run faster.. https://github.com/cytopia/docker-ansible/actions?query=branch%3Aarm64v8

The reason I switched from docker build to docker buildx late in my commits is cause it was slow and using ghcr.io for build cache makes things fast, is already authed in github actions, and without enabling inline cache in the published docker hub images which I guess is a plus.

docker build only supports inline cache.

[morganchristiansson]

After almost a whole day all builds have finished.. I managed to exclude kops 1.18 and earlier from arm64 so got no failures in build-kops now. (I left comment above/inline)

[morganchristiansson]

I believe outstanding issues are:

• aws-iam-authenticator needs new source for arm64
• oc I'm struggling to find arm64 binary for..
• Add uncached runs for upgrading dependencies
  • Tho I wonder if we could just move the get-latest-version-of-dependency that supplies --build-arg DEPENDENCY_VERSION which would play nice with build cache. Updated base image should also trigger rebuilds.
• Also maybe some process to delete old images from ghcr.io.. I don't think there's any auto-delete and it would be impolite to have github store cache image layers forever. I saw some gcr-lifecycle-policy tool that can be used to delete old images.
• Join script to create multi-arch image manifests.. Maybe should work out uncached runs and everything first..

Also the arm64 builds are running very fast with build cache.. Maybe using docker buildx build --platform linux/amd64,linux/arm64 to build both amd64&arm64 images at the same time is not a bad idea now that builds are running smoothly. It would simplify the changes, and eliminate the join step. Tho would need to run all tests twice for amd64/arm64 arch which will add complexity.

[morganchristiansson]

Also add arm64 to nightly builds with/without build cache

[morganchristiansson]

I'm not sure there's a way to do COPY openshift-client conditionally in Dockerfile. We may need a separate Dockerfile for arm64?

Or if Dockerfile-aws-oc was split out to separate image tag from Dockerfile-awsk8s .. but that will remove oc from existing images... Maybe a arm64 specific Dockerfile-awsk8s-arm64 ?? Tho this will rule out using docker buildx build --platform x,y as it can only use 1 Dockerfile for both platforms...

[morganchristiansson]

Could we split out openshift client to a new image tag?

[morganchristiansson]

I decided I will build arm64 only images in my fork. So I simply removed openshift and old kops that don't support arm64 and not build those tags.

I am creating multi-arch manifest using cytopia/ansible for amd64 images in my fork, no need to rebuild the amd64.

docker manifest create \
  $(IMAGE):$(TAG) \
  --amend $(IMAGE):$(TAG)-$(PLATFORM) \
  --amend cytopia/ansible:$(TAG)

multi-arch tags for all supported should be available at https://hub.docker.com/r/morganchristiansson/ansible/tags

It seems cytopia/ansible is stable where removing/splitting openshift client cannot be done. My fork does not have this constraint. And as a bonus the openshift client is retained in amd64 by using cytopia/ansible images amd64 platform

Happy to stay aligned - I will run fork to build multi-arch arm64 tags for now.

I pushed to master @ https://github.com/morganchristiansson/docker-ansible/

[morganchristiansson]

Latest aws-iam-authenticator now has arm64 release binaries https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/