community

community copied to clipboard

As discussed in #111 , we can use kind to test controllers architectural compatibility for arm and x86

EKS Arm support is GA now: announcement

Hey there, sorry I've gotten sidetracked with other projects. Is using kind to test arch compat still desirable or would y'all rather use the now-GA EKS functionality? Happy to help build out the automation either way.

I'd be for all in on Graviton2 @scottmalkie since we can then use this as a datapoint as well ;)

Any updates here?

@smuralee has been working on it, I am going to get with him early next week to help out however I can.

@smuralee has been working on it

thanks for confirming, that was my hunch but was unsure since this ticket said you alone ;)

I am going to get with him early next week to help out however I can.

super, super!

• Kind does not have arm64 image
• Built the base image using docker build -t ack/kind-base:latest . from go/src/sigs.k8s.io/kind/images/base/
• docker image inspect ack/kind-base returns "Architecture": "arm64", "Os": "linux"
• Fails to run the kind build node-image --base-image=ack/kind-base:latest fails with below error

Step 1/16 : FROM k8s.gcr.io/build-image/kube-cross:v1.15.0-1
v1.15.0-1: Pulling from build-image/kube-cross
no matching manifest for linux/arm64/v8 in the manifest list entries

Ran into the same issue, @smuralee - I build kind itself just fine on arm64 but building the node-image errors out on that same kube-cross line. I added to the existing Arm support issue here: https://github.com/kubernetes-sigs/kind/issues/166#issuecomment-692239707

The base image is now multi-arch by default including arm64 shipping in kind 0.9 (aim to release today), but the node image on top of that is a bit more involved at the moment. It's feasible to build from arm64 host with kind build node-image --type=bazel AIUI, but I don't have access / time for this myself currently to confirm.

In Q4 I'd like to make this easier, but I'm not making any promises yet.

@BenTheElder If there is anything we can do to assist, please let us know. I will test/document the bazel build process and report back.

[ssm-user bin]$ ./kind build node-image --type=bazel
Starting to build Kubernetes
Loading:
Loading: 0 packages loaded
DEBUG: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: rbe_default not using checked in configs; Bazel version 3.5.0 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0":["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"]}
Analyzing: 4 targets (0 packages loaded, 0 targets configured)
INFO: Repository go-runner-linux-arm64 instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule container_pull defined at:
  /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/io_bazel_rules_docker/container/pull.bzl:228:33: in <toplevel>
INFO: Repository debian-iptables-arm64 instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule container_pull defined at:
  /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/io_bazel_rules_docker/container/pull.bzl:228:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'debian-iptables-arm64':
   Traceback (most recent call last):
        File "/home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/io_bazel_rules_docker/container/pull.bzl", line 171, column 13, in _impl
                fail("Pull command failed: %s (%s)" % (result.stderr, " ".join([str(a) for a in args])))
Error in fail: Pull command failed: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: cannot execute binary file
 (/home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded -directory /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/debian-iptables-arm64/image -os linux -os-version  -os-features  -architecture arm64 -variant  -features  -name k8s.gcr.io/build-image/debian-iptables@sha256:a83cf1d501ad33f5aa93e2719baa59b054939b8a819c3997f915a6acfaa8e31a)
ERROR: /home/ssm-user/go/src/k8s.io/kubernetes/build/BUILD:61:22: //build:kube-proxy-internal depends on @debian-iptables-arm64//image:image in repository @debian-iptables-arm64 which failed to fetch. no such package '@debian-iptables-arm64//image': Pull command failed: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: cannot execute binary file
 (/home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded -directory /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/debian-iptables-arm64/image -os linux -os-version  -os-features  -architecture arm64 -variant  -features  -name k8s.gcr.io/build-image/debian-iptables@sha256:a83cf1d501ad33f5aa93e2719baa59b054939b8a819c3997f915a6acfaa8e31a)
ERROR: Analysis of target '//build:docker-artifacts' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.414s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (72 packages loaded, 142 targets configured)
FAILED: Build did NOT complete successfully (72 packages loaded, 142 targets configured)
Failed to build Kubernetes: command "bazel build //cmd/kubeadm:kubeadm //cmd/kubectl:kubectl //cmd/kubelet:kubelet //build:docker-artifacts" failed with error: exit status 1
ERROR: error building node image: failed to build kubernetes: command "bazel build //cmd/kubeadm:kubeadm //cmd/kubectl:kubectl //cmd/kubelet:kubelet //build:docker-artifacts" failed with error: exit status 1
Command Output: Loading:
Loading: 0 packages loaded
DEBUG: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: rbe_default not using checked in configs; Bazel version 3.5.0 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0":["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"]}
Analyzing: 4 targets (0 packages loaded, 0 targets configured)
INFO: Repository go-runner-linux-arm64 instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule container_pull defined at:
  /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/io_bazel_rules_docker/container/pull.bzl:228:33: in <toplevel>
INFO: Repository debian-iptables-arm64 instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule container_pull defined at:
  /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/io_bazel_rules_docker/container/pull.bzl:228:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'debian-iptables-arm64':
   Traceback (most recent call last):
        File "/home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/io_bazel_rules_docker/container/pull.bzl", line 171, column 13, in _impl
                fail("Pull command failed: %s (%s)" % (result.stderr, " ".join([str(a) for a in args])))
Error in fail: Pull command failed: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: cannot execute binary file
 (/home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded -directory /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/debian-iptables-arm64/image -os linux -os-version  -os-features  -architecture arm64 -variant  -features  -name k8s.gcr.io/build-image/debian-iptables@sha256:a83cf1d501ad33f5aa93e2719baa59b054939b8a819c3997f915a6acfaa8e31a)
ERROR: /home/ssm-user/go/src/k8s.io/kubernetes/build/BUILD:61:22: //build:kube-proxy-internal depends on @debian-iptables-arm64//image:image in repository @debian-iptables-arm64 which failed to fetch. no such package '@debian-iptables-arm64//image': Pull command failed: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded: cannot execute binary file
 (/home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/go_puller_linux/file/downloaded -directory /home/ssm-user/.cache/bazel/_bazel_ssm-user/0637ad9b317d911f1a295ea0292a0956/external/debian-iptables-arm64/image -os linux -os-version  -os-features  -architecture arm64 -variant  -features  -name k8s.gcr.io/build-image/debian-iptables@sha256:a83cf1d501ad33f5aa93e2719baa59b054939b8a819c3997f915a6acfaa8e31a)
ERROR: Analysis of target '//build:docker-artifacts' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.414s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (72 packages loaded, 142 targets configured)
FAILED: Build did NOT complete successfully (72 packages loaded, 142 targets configured)
[ssm-user bin]$

@BenTheElder Building with bazel fails as well, I am using a newer bazel version but will also try with the pinned one. If the right answer is "just work with upstream to get multi-arch support there, then build 'normally'", that's fine too. Thanks again, kind is awesome.

I think long term that's the right, most powerful answer, xref: https://github.com/kubernetes/kubernetes/issues/88553 there's some discussion of removing it.

I'd like pre-built multi-arch images in kind eventually, but constraints on the rather non-standard node image build are going to make that a bit of a mess, and even if we landed that you'd be stuck with whatever versions we'd pre-published.

Getting the make build to work on not-amd64 will be a win for others beyond kind as well. I reached out in #sig-release (kubernetes slack) and received this pointer to the upstream tracking for that: https://github.com/kubernetes/release/blob/0816bc88c00d682b8a05fe374fe52c7c764d05d8/images/build/cross/Makefile#L27-L31

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

/remove-lifecycle stale

FTR https://github.com/aws-controllers-k8s/community/issues/138#issuecomment-693690826 happened, KIND has supported ARM64 OOTB for some time now. As I understand it the fixes we've written or merged into Kubernetes from other threads also now mean that building on arm64 should work fine and we've landed a KEP to dedupe the build systems and focus on the one.

Thanks @BenTheElder! We will look into building and testing non-amd64 builds of ACK controllers soon :) Cheers!

/lifecycle frozen

Any updates on this?

@Preen we have on our roadmap the construction and testing of our controller images for different architectures. I would like to get to this before the end of this year, but unfortunately cannot give any firm timelines.

cc @SantyAMZN

HI there, trying to follow the conversation and linked issues but they all appear merged or closed. Is there an update that can be shared regarding aarch64 support for controllers?

Any progress on this?

Hello, Any news regarding this topic?

Hi @nunodomingues-td @asonnleitner @thegoldenmule - We have a contributor (@kahirokunn) that worked on this feature and have few PRs open against the build pipelines and the code-generation. I'm willing to merge those and deploy everything to our pipesline before the end of the year.

For others watching this issue code-generator was updated and keyspaces was the first project to release a multi arch image. Now I'm just waiting for the projects I use to be updated/released with arm64 support. Thanks @kahirokunn !

Hi @kylekluever - we're trying to rollout multi-namespace-watch mode to all the controllers, which will be delivered with ARM support as well. ETA as soon as we figure out the RBAC stuff for multi-namespace (expecting end of this week) :)

All controllers received an ARM release yesterday (except for memorydb) - the catalogs are also updated https://gallery.ecr.aws/aws-controllers-k8s . Many thanks to @kahirokunn for his contributions in this area :tada:

cc @kylekluever @nunodomingues-td @asonnleitner @thegoldenmule @Preen @scottmalkie @garrusnapp @jaredtbates @renatomjr @kahirokunn @kylekluever