ARM64/Multi-Arch Image

[cawfeecoder]

Please describe your use case / problem. I want to be able to deploy Ambassador on one of my ARM64 nodes.

Describe the solution you'd like I'd like to see officially released Ambassador ARM64 images

[richarddli]

We'd be happy to support you on this, but adding another platform is a non-trivial undertaking, e.g., our continuous integration systems need to support ARM64. Would you be willing to sponsor this work?

[samgranieri]

I would definitely like to run Ambassador on my raspberry pi 4 cluster and am willing to help out. I use ambassador at work and would like to use it at home too.

[samgranieri]

There's some progress building envoy for arm in https://github.com/envoyproxy/envoy/issues/1861

[kflynn]

To comment further:

• Ambassador itself shouldn't be a big problem. There are various points where we hardcode things like linux_amd64 for Go architecture, but those are probably relatively simple to manage.
• Envoy is a different story, though... if you want to try the instructions at https://stevesloka.com/compile-envoy-on-raspberry-pi4/ we'd be interested in hearing about how it goes...

[al45tair]

@kflynn Those instructions basically work. I had to do a little bit more (and I didn't disable things like Steve Sloka did), and there was one tiny "bug" in the Envoy code (I say "bug" in quotes, because it's actually just a warning because of a '\x00' <= c expression where c turns out, for whatever reason, to be unsigned). Also, the envoy binary you get at the end of the build process is… rather large. It's a good idea to strip it, as that brings it down from a couple of GBs to 38MB or so.

Finally, there is a new gotcha in that Ubuntu 20 is out, so it's easy to do what I did and build with the wrong Ubuntu version, which means I need to re-do my Envoy build because the glibc versions don't match what the Docker container had installed.

I'll give it another go tomorrow, if I have time.

[al45tair]

@kflynn It turns out that a much easier approach is to do clone Envoy's sources and run, on an ARM system, ci/run_envoy_docker.sh 'ci/do_ci.sh bazel.release.server_only', then (in my case) docker build -f ci/Dockerfile-envoy -t al45tair/envoy-arm64 ..

I haven't tried that on a Raspberry Pi (it might fail because of trying to build too many things simultaneously, especially if it either makes the Pi too hot or runs it out of RAM); I have access to a big-iron ARM system, so I did the build there.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[al45tair]

This is still something that is worth having. Further, with Apple shifting its machines to ARM and Amazon offering Graviton and Graviton2 instances, I think there's a good argument for having ARM64 versions of most infrastructure images (like Ambassador).

[shaneutt]

We're nearly there it would seem with Envoy on ARM. This PR was merged since this issue last updated:

https://github.com/envoyproxy/envoy/pull/11813

And the dev image is now a multi-arch manifest image with ARM64 support:

https://hub.docker.com/r/envoyproxy/envoy-dev/tags

I didn't see anything in the release notes that made it seem particularly official, but it's practically complete.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaneutt]

I still have a personal and professional interest in ARM64 support for Ambassador, would someone closer to the project than me be willing to write up a short post or doc that enumerates the work that needs to be done to complete this?

[benwalker]

Agree. Arm devices are very effective way to build clusters and HA in s/w.

[kflynn]

So, off the top of my head (and therefore likely incomplete!), here's what I think needs to happen next to demonstrate viability:

• We need a proof-of-concept build of Envoy (from https://github.com/datawire/envoy) for ARM64. Per @shaneutt's comment from last September, this might be easy? This will have to be taken all the way to having a Docker image pushed somewhere accessible to the Ambassador build.

• We need a proof-of-concept build of Ambassador (from this repo) for ARM64. This is also likely to be pretty straightforward if done on an ARM64 Linux box. (It "should" also be possible to get Ambassador to cross-compile the Golang bits on anything else, but that's not currently likely to be simpler than just using Linux/ARM64.)

• We will need to see the Envoy test suite pass on the PoC Envoy build. (This may be a bigger problem than compiling Envoy: the tests need a lot of disk and a fair amount of RAM. When we run the Envoy tests on x86, we use a GCE node with something like 600GiB of RAM so that we can keep test results on a RAMdisk, to get the run time below "many hours".)

• We will need to see the full Ambassador CI test suite pass on the PoC Ambassador build. Note that this doesn't require running it in CI, but I'm deliberately including the full matrix of configurations that CI tests – e.g. testing with and without AMBASSADOR_FAST_RECONFIGURE and AMBASSADOR_LEGACY_MODE, and testing that the generated Envoy configuration from an ARM build exactly matches the one from an x86 build as well as testing that the generated configuration makes Envoy on ARM do the same thing as Envoy on x86.

To go from a PoC to production:

• We will need to sort out CI support for ARM64.

• We will need to sort out exactly how to distribute the ARM code – I personally don't think a multi-arch image is the way to go, given how big Ambassador's image is just with x86 support. 🙁 This isn't likely to be a big deal, but it needs some thought.

• We will need to sort out how to include ARM builds in the release process.

The actual work of getting from PoC to production will pretty much need to be done by Ambassador Labs, but getting to the PoC is frankly the longer pole here. Most of the PoC steps above are likely not hard, but they are very definitely time-consuming, and they require beefier ARM hardware than I have lying around[1]. So while we'd be delighted to support anyone who wants to do the work to get to the PoC, it'll be challenging to find the bandwidth to do it in-house in the near term. 🙁

[1] I'm pretty sure that the beefiest ARM machine I have lying around is a PocketBeagle, which probably can't compile Ambassador and definitely can't compile Envoy... 😂

[cawfeecoder]

On the ARM64 machine front, look at Packet. They rent out Caviuum (?) based ARM processor machines. AWS also rents out machines with their Gravitron2 (ARM64) processors.

On Fri, Jan 22, 2021 at 10:48 AM Flynn [email protected] wrote:

So, off the top of my head (and therefore likely incomplete!), here's what I think needs to happen next to demonstrate viability:

We need a proof-of-concept build of Envoy (from https://github.com/datawire/envoy) for ARM64. Per @shaneutt https://github.com/shaneutt's comment from last September, this might be easy? This will have to be taken all the way to having a Docker image pushed somewhere accessible to the Ambassador build.

We need a proof-of-concept build of Ambassador (from this repo) for ARM64. This is also likely to be pretty straightforward if done on an ARM64 Linux box. (It "should" also be possible to get Ambassador to cross-compile the Golang bits on anything else, but that's not currently likely to be simpler than just using Linux/ARM64.)

We will need to see the Envoy test suite pass on the PoC Envoy build. (This may be a bigger problem than compiling Envoy: the tests need a lot of disk and a fair amount of RAM. When we run the Envoy tests on x86, we use a GCE node with something like 600GiB of RAM so that we can keep test results on a RAMdisk, to get the run time below "many hours".)

We will need to see the full Ambassador CI test suite pass on the PoC Ambassador build. Note that this doesn't require running it in CI, but I'm deliberately including the full matrix of configurations that CI tests – e.g. testing with and without AMBASSADOR_FAST_RECONFIGURE and AMBASSADOR_LEGACY_MODE, and testing that the generated Envoy configuration from an ARM build exactly matches the one from an x86 build as well as testing that the generated configuration makes Envoy on ARM do the same thing as Envoy on x86.

To go from a PoC to production:

We will need to sort out CI support for ARM64.

We will need to sort out exactly how to distribute the ARM code – I personally don't think a multi-arch image is the way to go, given how big Ambassador's image is just with x86 support. 🙁 This isn't likely to be a big deal, but it needs some thought.

We will need to sort out how to include ARM builds in the release process.

The actual work of getting from PoC to production will pretty much need to be done by Ambassador Labs, but getting to the PoC is frankly the longer pole here. Most of the PoC steps above are likely not hard, but they are very definitely time-consuming, and they require beefier ARM hardware than I have lying around[1]. So while we'd be delighted to support anyone who wants to do the work to get to the PoC, it'll be challenging to find the bandwidth to do it in-house in the near term. 🙁

[1] I'm pretty sure that the beefiest ARM machine I have lying around is a PocketBeagle https://beagleboard.org/pocket, which probably can't compile Ambassador and definitely can't compile Envoy... 😂

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/datawire/ambassador/issues/2121#issuecomment-765498875, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRK6QPYNYV5RVIJ7G466BDS3GM47ANCNFSM4J4DR2EQ .

-- Regards, Nicholas Frush Founder ghostbox C: 443-974-0421

[al45tair]

• We need a proof-of-concept build of Envoy (from https://github.com/datawire/envoy) for ARM64. Per @shaneutt's comment from last September, this might be easy? This will have to be taken all the way to having a Docker image pushed somewhere accessible to the Ambassador build.

Envoy is easy enough to build, though you'll want a reasonably fast ARM64 box to do it on. Building it on a Raspberry Pi is quite tricky (it might work with the new 8GB Pi 4); I gave up on that and used a ThunderX2 box I have access to at work to do it (the large core counts on these are very useful for this). An alternative might be to use MacStadium to rent some time on an M1 Mac, which would only have 8 cores (versus 56 for the ThunderX2 box I was using) but might be cheaper than buying time on a higher core count box (e.g. one of AWS's Gravitron2 instances, or someone's random Cavium box) and those 8 cores are all faster than the Cavium cores. If you're using the Mac approach, you'll need the beta version of Docker Desktop in order to natively run ARM64 Linux, but that's fine.

Another option worth considering might be to run the build in Travis, which has ARM64 Linux support (via AWS, I believe). I don't know what the specs on the systems they'll let you spin up are like, but they do have the advantage of having a CI system ready to go as well.

• We need a proof-of-concept build of Ambassador (from this repo) for ARM64. This is also likely to be pretty straightforward if done on an ARM64 Linux box. (It "should" also be possible to get Ambassador to cross-compile the Golang bits on anything else, but that's not currently likely to be simpler than just using Linux/ARM64.)
• We will need to see the Envoy test suite pass on the PoC Envoy build. (This may be a bigger problem than compiling Envoy: the tests need a lot of disk and a fair amount of RAM. When we run the Envoy tests on x86, we use a GCE node with something like 600GiB of RAM so that we can keep test results on a RAMdisk, to get the run time below "many hours".)

This is conceivably somewhere where the M1 Macs might shine, as they have fast SSD storage. I don't know how fast the storage is on available higher core count boxes or on Travis, but realistically I'd expect most of the available options to be decent, at least.

• We will need to sort out CI support for ARM64.
• We will need to sort out exactly how to distribute the ARM code – I personally don't think a multi-arch image is the way to go, given how big Ambassador's image is just with x86 support. 🙁 This isn't likely to be a big deal, but it needs some thought.

If you're talking about multi-arch Docker images, you don't actually ever download all the architectures — indeed, the images themselves are essentially separate (it's just that there's a manifest that tells Docker which to use). So I don't think this is as big an issue as you might imagine.

[fspaniol]

Hello there! Are there any ETA on this? Cheers :)

[ryankurte]

hey just dropping past because i didn't see it mentioned in the thread, you may be able to use docker's buildx to deal with cross building the container images, so you shouldn't actually need target-specific infrastructure at least for the container building part of this.

[christianhuening]

@khussey see you removed this from a cycle. Is that kinda like a decision to not add ARM64 support at all?

[the-wondersmith]

@cawfeecoder @samgranieri @al45tair @christianhuening Please check out issue #4184.

[Alpenbelieve]

So, off the top of my head (and therefore likely incomplete!), here's what I think needs to happen next to demonstrate viability:

• We need a proof-of-concept build of Envoy (from https://github.com/datawire/envoy) for ARM64. Per @shaneutt's comment from last September, this might be easy? This will have to be taken all the way to having a Docker image pushed somewhere accessible to the Ambassador build.
• We need a proof-of-concept build of Ambassador (from this repo) for ARM64. This is also likely to be pretty straightforward if done on an ARM64 Linux box. (It "should" also be possible to get Ambassador to cross-compile the Golang bits on anything else, but that's not currently likely to be simpler than just using Linux/ARM64.)
• We will need to see the Envoy test suite pass on the PoC Envoy build. (This may be a bigger problem than compiling Envoy: the tests need a lot of disk and a fair amount of RAM. When we run the Envoy tests on x86, we use a GCE node with something like 600GiB of RAM so that we can keep test results on a RAMdisk, to get the run time below "many hours".)
• We will need to see the full Ambassador CI test suite pass on the PoC Ambassador build. Note that this doesn't require running it in CI, but I'm deliberately including the full matrix of configurations that CI tests – e.g. testing with and without AMBASSADOR_FAST_RECONFIGURE and AMBASSADOR_LEGACY_MODE, and testing that the generated Envoy configuration from an ARM build exactly matches the one from an x86 build as well as testing that the generated configuration makes Envoy on ARM do the same thing as Envoy on x86.

To go from a PoC to production:

• We will need to sort out CI support for ARM64.
• We will need to sort out exactly how to distribute the ARM code – I personally don't think a multi-arch image is the way to go, given how big Ambassador's image is just with x86 support. 🙁 This isn't likely to be a big deal, but it needs some thought.
• We will need to sort out how to include ARM builds in the release process.

The actual work of getting from PoC to production will pretty much need to be done by Ambassador Labs, but getting to the PoC is frankly the longer pole here. Most of the PoC steps above are likely not hard, but they are very definitely time-consuming, and they require beefier ARM hardware than I have lying around[1]. So while we'd be delighted to support anyone who wants to do the work to get to the PoC, it'll be challenging to find the bandwidth to do it in-house in the near term. 🙁

[1] I'm pretty sure that the beefiest ARM machine I have lying around is a PocketBeagle, which probably can't compile Ambassador and definitely can't compile Envoy... 😂

Hi :) , I'm curious about the current status of building am ARM image, it's now Nov 2022, are there any progress or any future plans ? Thanks 😊

[V1ct0rHM]

I'm migrating all my company infrastructure to ARM64 and we are facing this issue now, the only tool we use that doesn't have an arm image is emissary ingress, so sad

[gecube]

https://github.com/emissary-ingress/emissary/issues/4184#issuecomment-1769504882

[notwedtm]

How is this issue 5 years old?!

[kflynn]

Wow, that is pretty terrifying. Anyone who'd like to test the unofficial ARM64 image I've built, please ping me on the CNCF Slack -- I'm (at)flynn there. 🙂

[christianhuening]

What's needed to get that image upstream released? Or is that still part of the discussion above from 3y ago to add all the steps to the official pipeline and to have a beefy ARM machine?