Unable to build / deploy on ARM

[zwan2]

trafficstars

 - deployment/adservice: container server terminated with exit code 1
    - pod/adservice-5b65c5ff6-zzfkl: container server terminated with exit code 1
      > [adservice-5b65c5ff6-zzfkl server] Error occurred during initialization of VM
      > [adservice-5b65c5ff6-zzfkl server] Could not find agent library /opt/cprof/profiler_java_agent.so in absolute path, with error: /opt/cprof/profiler_java_agent.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)
 - deployment/adservice failed. Error: container server terminated with exit code 1.

[xtineskim]

Hello! Thanks for bringing this up!! At first glance, it seems like it might be a minikube issue, could you specify your environment (os, minikube version, anything else that may be relevant)? Thanks!

[zwan2]

• m1 mac Big Sur 11.6x
• minikube:latest using docker

It may be resolved when x86 condition

[jxlwqq]

Does not support deployment it under arm64 architecture, such as Apple chip's Mac, Raspberry Pi. @zwan2

Are there any plans to support arm64? @ckim328

[NimJay]

tl;dr - We will not be (consciously) supporting the deployment of our microservice sample images on ARM anytime soon. I say "consciously" because you might still be able to build the Docker images locally (and target ARM) and get them to deploy on ARM machines.

---

Hi @zwan2 and @jxlwqq! Thank you for raising this issue! 😁

We've actually been talking about ARM support a lot recently. This issue is becoming a growing concern — especially given the popularity of the M1 chip.

Currently, the Docker images (for each microservice) that we host (e.g., gcr.io/google-samples/microservices-demo/checkoutservice:v0.3.1) use AMD64 architecture.

> docker image inspect gcr.io/google-samples/microservices-demo/checkoutservice:v0.3.1 | grep Arch
"Architecture": "amd64",

And, unfortunately, we do not intend to create sample images that use ARM architecture anytime soon — mainly because Google Kubernetes Engine does not currently support ARM-based image deployments.

@zwan2 In your Kubernetes deployments, are you using the images hosted at gcr.io/google-samples/microservices-demo/...? If so, have you tried building the images locally (and targeting ARM) and using those locally-built Docker images in minikube? I can't guarantee that this works (because I am not sure if the base images used in each of microservices' Dockerfiles support ARM). But it's worth trying. :)

[bourgeoisor]

Not fully supported yet, but note that you can alternatively use docker buildx from your ARM machine to build amd64 images: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

We're currently investigating the use of ARM chips by developers to build our sample applications.

[xtineskim]

cooling down out of our oslo, still investigations being done for the use of ARM

[NimJay]

@zwan2, @jxlwqq

GKE (Google Kubernetes Engine) now supports the ability to run containerized workloads using the Arm architecture. :) These resources might be of interest to you:

1. YouTube - Run your Arm workloads on GKE!
2. Google Cloud Doc - Arm workloads on GKE
3. Google Cloud Blog - Run your Arm workloads on Google Kubernetes Engine with Tau T2A VMs

I have not tested your specific user journey myself (i.e., I have not run Arm-based container images on GKE, built on Mac's M1 chips). But I hope this info helps.

[minherz]

@zwan2 , @jxlwqq would you mind to try the tutorial for deploying Online Boutique sample application described in the official documentation? Please, let us know if it does not work. Deploying the application does not require building containers.

[smazzone]

just checking if there are plans to build and publish arm64 images in the default repo for people who are willing to run the demo locally (i.e. M1 and docker desktop or similar). Thanks again.

[mathieu-benoit]

One more evidence we need to support this: https://github.com/GoogleCloudPlatform/microservices-demo/issues/810

[mathieu-benoit]

Another evidence here: https://github.com/GoogleCloudPlatform/microservices-demo/issues/832

[chenhy97]

need the ARM architecture supports!

[GregCKrause]

Not fully supported yet, but note that you can alternatively use docker buildx from your ARM machine to build amd64 images: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

We're currently investigating the use of ARM chips by developers to build our sample applications.

Following up on @bourgeoisor's response.

To use this with skaffold, set build.local.useBuildkit to true in skaffold.yaml:

build:
  ...
  local:
    useBuildkit: true

[smazzone]

@GregCKrause I have been running the demo on arm64 for a couple of weeks. I have also used buildx to build images on both architectures. But, changing skaffold.yaml was not enough. I had to slightly re-adapt Dockerfile as health probes are hardcoded for amd64 architecture see example here

RUN GRPC_HEALTH_PROBE_VERSION=v0.4.14 && \ wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \ chmod +x /bin/grpc_health_probe

I hope this helps. Stefano

[minherz]

I will keep this issue open for now and mark it as a "good first issue" for someone who wants to submit a kustomize or Helm configuration that will build and deploy on ARM platform.

[NimJay]

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023.

More specifically:

• A user should be able to run Online Boutique in development mode (skaffold dev), on an ARM-based device.
• A user should be able to deploy Online Boutique, as is, on an ARM-based Kubernetes cluster. In this sentence, “as is” means that the user does not have to build Online Boutique’s container images themselves. Official ARM-supporting container images should be published by Online Boutique.

We hope to achieve this by the end of this year (2023).

Thank you, @smazzone and @GregCKrause for the tips you've shared — those will be helpful.

[odidev]

I have made changes in the Dockerfiles to build the images for arm64 platform and now skaffold is running successfully and pods are in running state.

I have used my docker images that I built for arm64 in kubernetes-manifests.yaml and deployed on minikube cluster. Now all the pods are running successfully but when applying the same kubernetes-mainfests.yaml file in GKE cluster the pods are in pending state for arm64.

I have also built the images using buildx and the pods are running for amd64 on GKE cluster but for arm64 it is still in pending state. Can you please share your feedback regarding the same?

[NimJay]

Heads-up:

A comment above mentioned:

I had to slightly re-adapt Dockerfile as health probes are hardcoded for amd64 architecture see example here

This additional step (of swapping the grpc-health-probe binaries deployed next to Online Boutique's gRPC microservices) is no longer necessary, since the merge of https://github.com/GoogleCloudPlatform/microservices-demo/pull/1837. In other word, the grpc-health-probe binary is no longer used.