[FEATURE]: ARM64 Support

[NapalmCodes]

Feature Request

The ability to run on ARM architectures like those found on IoT devices and hobbyist platforms like Raspberry PI.

Background / Motivation

Learning how garden keeps the feedback loop nice and short, and the fact that I recently spun up a k8s cluster to learn with on Raspberry Pis.

What should the user be able to do?

Run the features provided on other platforms on ARM.

Why do they want to do this? What problem does it solve?

To use the product on other architectures. It solves the problem of long feedback loops while in the midst of developing on these kinds of clusters.

Suggested Implementation(s)

The user experience would be the same as the x86/x64 version today just for ARM64. Implementation would involve building for this architecture.

How important is this feature for you/your team?

Crucial (Garden is unusable for us without it) High (Not having this feature makes using Garden painful) Medium (Using Garden without it is mildly inconvenient) Low (Nice to have someday)

[eysi09]

Hi @napalm684 and thanks for the feature request! This would indeed be nice but I'm not entirely ensure if there are any gotchas for running on ARM architecture. We use Zeit/pkg for packaging Garden and that supports different architectures.

However, we currently don't have any means of testing ARM64 builds but happy to except a PR if you want to take a stab at it and can get it working on your end :)

[edvald]

Another thing to consider is that all the tools that we depend on need to also support ARM. I think most or even all of them do, but we'd need to make sure, and appropriately handle cases where they don't.

[gueux]

Hi, i've tried to init garden for k3s on arm (raspberry pi 4) with the next result:

╰─➤  garden --env=local-pi plugins kubernetes cluster-init
✔ providers                 → Preparing environment... → Done
   ✔ kubernetes                → Configuring... → Ready
Initializing/updating cluster-wide services for local-pi environment ⚙️

✔ nfs-provisioner           → Syncing module sources (12 files)... → Done (took 0 sec)
✔ nfs-provisioner           → Building version v-fb1a1ba60f... → Done (took 0.2 sec)
✖ nfs-provisioner           → Deploying version v-fb1a1ba60f...
✔ build-sync                → Syncing module sources (6 files)... → Done (took 0.1 sec)
✔ registry-proxy            → Syncing module sources (5 files)... → Done (took 0.1 sec)
✔ docker-daemon             → Syncing module sources (6 files)... → Done (took 0.1 sec)
✔ build-sync                → Building version v-d7edebe99a... → Done (took 0.3 sec)
✔ docker-registry           → Building version v-21782affbe... → Done (took 2.8 sec)
✔ registry-proxy            → Building version v-da9a9c6749... → Done (took 0.3 sec)
✖ build-sync                → Deploying version v-d7edebe99a...
✔ docker-daemon             → Building version v-8c27a58ecb... → Done (took 0.2 sec)
✖ registry-proxy            → Deploying version v-da9a9c6749...
   ℹ registry-proxy            → Waiting for resources to be ready...
✖ docker-registry           → Deploying version v-21782affbe...
Cleaning up old resources...

Done!

╰─➤  KUBECONFIG=~/.kube/config.pi kubectl -n garden-system get pods                                                                                                                                                                       1 ↵
NAME                                         READY   STATUS             RESTARTS   AGE
garden-docker-registry-6cf456ff4f-96pxh      0/1     Pending            0          22h
garden-build-sync-7644c95976-8flvd           0/1     Pending            0          22h
garden-build-sync-7644c95976-wk4kh           0/1     Pending            0          22h
garden-build-sync-7644c95976-bpv2s           0/1     Pending            0          22h
garden-nfs-provisioner-v2-5b44cb79db-zl24t   0/1     Pending            0          14m
garden-registry-proxy-twpl4                  0/1     CrashLoopBackOff   95         22h

It looks like docker-registry docker image is not suitable for ARM:

╰─➤  KUBECONFIG=~/.kube/config.pi kubectl -n garden-system logs garden-registry-proxy-twpl4
standard_init_linux.go:211: exec user process caused "exec format error"

Also there are some issues with storage class provisioner:

  Warning  ProvisioningFailed    15h (x1900 over 22h)  persistentvolume-controller  storageclass.storage.k8s.io "garden-system-nfs-v2" not found
  Normal   ExternalProvisioning  20m (x17 over 24m)    persistentvolume-controller  waiting for a volume to be created, either by external provisioner "cluster.local/garden-nfs-provisioner-v2" or manually created by system administrator
  Warning  ProvisioningFailed    118s (x68 over 26m)   persistentvolume-controller  storageclass.storage.k8s.io "garden-system-nfs-v2" not found

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 60 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[altitudems]

I was testing garden.io for deployment onto and edge cluster using arm64-based nodes and ran into the same issue.

Adding support for arm64 would be fantastic for everything from dev clusters, edge clusters, and AWS Arm64 EC2 instances as part of an EKS cluster.

[altitudems]

For further reference, I get a 'exec format error' error when trying to init the cluster. For me it bombed trying to deploy a garden-nfs-provisioner-v2 pod:

✖ nfs-provisioner           → Deploying version v-4f6500bdde...
      
Failed deploying service 'nfs-provisioner' (from module 'nfs-provisioner'). Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error deploying nfs-provisioner: BackOff - Back-off restarting failed container

━━━ Events ━━━
Deployment garden-nfs-provisioner-v2: ScalingReplicaSet - Scaled up replica set garden-nfs-provisioner-v2-65fd68d5f6 to 1
Deployment garden-nfs-provisioner-v2: ScalingReplicaSet - Scaled up replica set garden-nfs-provisioner-v2-65fd68d5f6 to 1
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: FailedScheduling - 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: FailedScheduling - 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: Scheduled - Successfully assigned garden-system/garden-nfs-provisioner-v2-65fd68d5f6-7ncb5 to cluster-node-3
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-23364261-610b-49a8-a8ba-ab3afee20962" 
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: Pulled - Container image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2" already present on machine
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: Created - Created container nfs-server-provisioner
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: Started - Started container nfs-server-provisioner
Pod garden-nfs-provisioner-v2-65fd68d5f6-7ncb5: BackOff - Back-off restarting failed container

━━━ Pod logs ━━━
<Showing last 30 lines per pod in this Deployment. Run the following command for complete logs>
$ kubectl -n garden-system --context=prod logs deployment/garden-nfs-provisioner-v2

****** garden-nfs-provisioner-v2-65fd68d5f6-7ncb5 ******
------ nfs-server-provisioner ------standard_init_linux.go:219: exec user process caused: exec format error

[highb]

@edvald Could we get the stale label removed from this issue? I'm seeing it after trying to run cluster-init on my RPi lan server. I'm assuming that there is probably a docker image somewhere in the cluster init process that needs to be switched to the ARM64 version, yeah?

[highb]

If someone like myself wanted to dive in and try to figure out how to resolve the issue, where would I start? I'm assuming https://github.com/garden-io/garden/tree/master/core/src/plugins/kubernetes is where all the code lives?

~~edit: Looks like the nfs-server-provisioner has been moved multiple times by the k8s project maintainers and now lives here https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner . It appears that they do build arm64 images so the work might be as "easy" as updating the helm chart to their new version and passing in the right arch to it. That's assuming that nfs-server-provisioner is the only thing that would need to be swapped, though. :sweat_smile:~~

~~edit 2: This all leads me to believe that the Garden k8s provider should probably have an arch config which defaults to amd64 but could be set to arm64 or aarch64/aarch32. Is that what this template string is for? https://github.com/garden-io/garden/blob/master/docs/reference/template-strings/providers.md#localarch~~

edit 3: Sorry for the stream of consciousness here, I'm using this comment like a lab notebook. If I had actually looked at the error I was having, it appears that envoy is the container that has the wrong arch now.

━━━ Events ━━━
DaemonSet garden-registry-proxy: SuccessfulCreate - Created pod: garden-registry-proxy-2xbtc
DaemonSet garden-registry-proxy: SuccessfulCreate - Created pod: garden-registry-proxy-55xhz
DaemonSet garden-registry-proxy: SuccessfulCreate - Created pod: garden-registry-proxy-dtg4h
Pod garden-registry-proxy-dtg4h: Scheduled - Successfully assigned garden-system/garden-registry-proxy-dtg4h to bwolf2
Pod garden-registry-proxy-dtg4h: Pulling - Pulling image "envoyproxy/envoy-alpine:v1.11.0"
Pod garden-registry-proxy-dtg4h: Pulled - Successfully pulled image "envoyproxy/envoy-alpine:v1.11.0" in 6.970859847s
Pod garden-registry-proxy-dtg4h: Created - Created container registry-proxy
Pod garden-registry-proxy-dtg4h: Started - Started container registry-proxy
Pod garden-registry-proxy-dtg4h: Pulled - Container image "envoyproxy/envoy-alpine:v1.11.0" already present on machine
Pod garden-registry-proxy-dtg4h: BackOff - Back-off restarting failed container

━━━ Pod logs ━━━
<Showing last 30 lines per pod in this DaemonSet. Run the following command for complete logs>
$ kubectl -n garden-system --context=bwolf-lan logs daemonset/garden-registry-proxy

****** garden-registry-proxy-dtg4h ******
------ registry-proxy ------exec /usr/local/bin/envoy: exec format error


See .garden/error.log for detailed error message
Time: 0h:00m:49s      

[edvald]

This is actually not stale at all, we have been working behind the scenes on this. Reopening.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[stefreak]

@TimBeyer @vvagaytsev One aspect of "ARM64 support" that is still missing, is support for ARM64 Kubernetes worker nodes and multi-arch container builds. For this we should open a new GitHub issue, wdyt?

[stefreak]

It might also be worth it to open issues for missing support for arm64 in the homebrew installer and missing arm64 builds of the official Garden Docker containers.