[FEATURE]: ARM64 support for the garden CLI

[MFAshby]

Feature Request

Hi! I would love to be able to use garden from my low power ARM64 based laptop.

Background / Motivation

I work on a fairly heavyweight project, a web service backed by multiple docker containers running java apps, but I'd love to be able to work from a low power machine without being tied to a large workstation all day.

What should the user be able to do?

Install and use garden CLI on ARM64 hardware, connected to a cluster of any architecture (most likely x86_64)

Why do they want to do this? What problem does it solve?

See motivation above.

Suggested Implementation(s)

I have tried to simply remove the check in src/core/garden.ts and building according to the instructions in CONTRIBUTING.md. This results in a usable garden executable being built. When I try to initialize a remote cluster to use with garden however, I get an error

Sample garden.yml, just the basic example. k8s cluster is running on my x86_64 workstation, and kubectl is connected correctly.

kind: Project
name: phr
environments:
- name: dev
  providers:
  - name: kubernetes
    context: k3d-k3s-default
defaultEnvironment: dev

Sample error

[martin@martin-pinebook phr]$ garden --env=dev plugins kubernetes cluster-init
Plugins

Initializing/updating cluster-wide services for dev environment

ℹ providers                 → Getting status...

Failed resolving provider container. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command docker doesn't have a spec for this platform/architecture (linux-arm64)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed resolving one or more providers:
- container

See error.log for detailed error message
 providers                 → Getting status...

How important is this feature for you/your team?

🌹 It’s a nice to have, but nice things are nice 🙂

[MFAshby]

I took a quick look, it seems garden downloads binaries for some external tools like kubectl, docker, etc, so finding arm64 binaries for those should do it. Made a start on that here https://github.com/garden-io/garden/compare/master...MFAshby:arm64-cli

One problem might be hadolint: it doesn't supply an arm64 build although someone is trying to fix that: https://github.com/hadolint/hadolint/issues/411

[thsig]

Hi @MFAshby—thanks for flagging this! I think finding arm64 binaries for the external tools should do the trick.

I wonder if it would make sense to conditionally skip loading/requiring the hadolint plugin on arm64 until they have an arm64 build available? @edvald

[altitudems]

I just added a comment on #1547.

I'm guessing this wasn't high priority, got stale and closed. Totally get it, but maybe we keep this on the radar?

Support for arm64 cluster-init and multiarch image building would be a great thing, esp. since arm64 is growing momentum :)

• Arm64 dev workstations, edge nodes, raspberry pi clusters, etc.
• Google, Amazon, Microsoft, Apple, Nvidia, and Intel are all investing in Arm
• G.A. Arm64 EC2 for EKS

[altitudems]

@MFAshby If you start a pull request, I can try and help.

[qeternity]

Would be great to see darwin arm64 as well given the Apple Silicon push. Happy to take the reigns on that if nobody is looking at it.

[edvald]

Agreed, it would be good to get Mac ARM64 sorted as well, but we may be blocked on that front for the moment because of mandatory code signing issues that collide with how we're currently doing our binary distribution. We have some thoughts on how we might proceed on that front but it'd take a bit more doing and I'd expect to to be a fairly deep rabbit hole for you @qeternity :) Have you found any issues with using the x64 build on the M1 Macs? We've not seen any problems reported on that front yet, but I'm always looking out for those.

[qeternity]

Understood, having done some macos notarization in the past, I am perfectly content to let this be someone else's problem 😂

As for the x64 version, it runs fine but I notice substantially more disk usage (like an absurd amount) when I run Rosetta 2 apps so I am trying to get usage down to zero (granted, this is an Apple problem, not a garden problem).

[edvald]

That's interesting. Can you elaborate on the disk usage a bit? How/where does that manifest and how much are we talking?

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[remotezygote]

Where are things on this? I think a few folks like me now have these M1 Max machines in hand - would love to know if there's a workaround of some sort...

[remotezygote]

To update, using the previous attempt's branch, I am able to get everything to build except the dashboard, and it appears to be only one piece: node-sass - no version before 6.0.1 will build on arm64. But when you just upgrade that:

$ react-scripts build                                                                                                                   ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  Creating an optimized production build...                                                                                               ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  Browserslist: caniuse-lite is outdated. Please run:                                                                                     ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  npx browserslist@latest --update-db                                                                                                     ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  Why you should do it regularly:                                                                                                         ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  https://github.com/browserslist/browserslist#browsers-data-updating                                                                     ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  Failed to compile.                                                                                                     ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄                                                                                                                         ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  ./src/styles/padding-margin-mixin.scss                                                                                 ┄┄
dashboard ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄  Error: Node Sass version 6.0.1 is incompatible with ^4.0.0.                                                            ┄┄
error Command failed with exit code 1.                                                                                 ┄┄
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.                                   ┄┄

build script in package @garden-io/dashboard failed with code 1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[remotezygote]

For more context, I did have to make some changes (commit here: https://github.com/remotezygote/garden/commit/f078573a6bba3f2c5e4ad7bbb7aa3f83a95734db), and had to do some stuff to get things to yarn successfully:

brew install nodejs yarn python cmake
export npm_config_python=/usr/bin/python2.7

If you change the version of node-sass in dashboard/package.json, the yarn install will succeed, but the dashboard will not build (everything else will).

And make sure you're using node 16 or less - lots of cert issues with 17 still.

[remotezygote]

Worth noting for others (I did not know this, maybe I missed it somewhere): you have to install Rosetta for garden to work on an M1 Mac:

/usr/sbin/softwareupdate --install-rosetta --agree-to-license

[eysi09]

Thanks for the update @remotezygote, this is super useful.

Does it work, now that you've installed Rosetta?

[remotezygote]

It does! Seems to work fine. But at the same time, doesn't feel like it's far off at all from working under arm...

[edvald]

Just a quick update, we are on a path to do this. A little tricky because it involves several updates to 3rd party tools we use, but we'll get there soon.

[qeternity]

It does! Seems to work fine. But at the same time, doesn't feel like it's far off at all from working under arm...

This is the least of your concerns. There are a half-dozen undocumented quirks that you will spend hours discovering (eg. you have to run a specific version of cert-manager because garden uses specific alpha manifests, this is not documented anywhere).

I'm very appreciative of all the hard work the garden team do, but would highly advise any team to steer clear. Minor updates often break something. Documentation is wildly inaccurate and out of date. I'm not sure if enterprise clients have better experiences. But ARM compatibility is the least of your worries.

[edvald]

I'm very sorry to hear about your experience @qeternity, truly. We do our best, and I do thankfully know that most of our users and customers have a far better experience, but reading this does break my heart. Last thing we'd want is developers burning hours fighting issues (Kubernetes itself is a steep enough hill to climb) and seems we've put too many rakes in your path to step on. I can only apologize for that, and assure you we take it seriously.

Just on the one specific example you mentioned, we've unfortunately neglected the cert-manager integration, and part of the learning is to avoid adding features we can't maintain properly.

For anyone else reading, please do not hesitate to talk to us, on our community channel (#garden on the Kubernetes Slack) or post issues here. Those are valuable contributions. Garden is an ambitious project, wrangling a tough problem, and we make a point of being within reach and as helpful as we can. Our team is growing rapidly this year, we're constantly improving ourselves and our work, and given the opportunity we'll always make the time to support users.

Sorry again @qeternity, all the best going forward.

[stefreak]

I just ran into this problem after brew install garden-cli. Installing rosetta fixed the garden CLI. My suggestions if the real solution is still too far away:

• Maybe the brew recipe could make sure to automatically install rosetta, or at least tell the user to do so
• the error message could be improved (currently zsh: bad CPU type in executable: garden, maybe a platform independent wrapper script could detect M1 and ask the user to install rosetta instead)

[pepov]

fyi I've created an asdf plugin and hardcoded amd64 in it for now, adding a comment here so that I remember to fix it when this gets resolved: https://github.com/pepov/asdf-garden

[vvagaytsev]

Closing as a duplicate of #1547.