Envoy Proxy Docker Image for v1.23.0 is broken for arm64

[mmeier86]

Overview of the Issue

I'm using Consul Connect with Nomad. One of my Nomad nodes is a Raspberry Pi 4, meaning it is an arm64 machine. Today, I started getting the following error when Nomad tries to start an envoy proxy for Nomad jobs running on the Pi 4 node:

/docker-entrypoint.sh: 29: exec: su-exec: Exec format error

That same error can be reproduced on the Pi itself by executing docker run envoyproxy/envoy:v1.23.0. The problem is already known on the Envoy side in envoyproxy/envoy#22261.

I have already confirmed that the 1.23.1 image works again.

I am creating the issue here instead of in the Nomad repository because as far as I understand, by default Nomad asks Consul which image version should be used if the user hasn't configured one themself.

Reproduction Steps

Try to launch a Nomad job using consul connect on an arm64 node, using envoy version 1.23.0.

Consul info for both Client and Server

Both Consul Server and Client are v1.13.1.

Operating system and Environment details

The client is a Raspberry Pi 4 with arm64 architecture.

Log Fragments

See description above.

[ViViDboarder]

Because I encountered this, my workaround was to explicitly set a working version in my Nomad client config as described here: https://learn.hashicorp.com/tutorials/nomad/consul-service-mesh#alternative-architectures

The proper fix would probably be for Consul to either remove 1.23.0 or replace it with 1.23.1.

[david-yu]

Hi @ViViDboarder and @mmeier86 we are aware that those container images are not working on arm64 since the image tags actually reference x86 images. I would recommend to upgrade to 1.23.1 and we're looking to update their on the docs on our side.

[blake]

@ViViDboarder PR #14573 updates Consul to return Envoy 1.23.1 in the supported proxy version list. Once this PR is merge and available in a release, it should resolve the issue you're experiencing.

In the mean time, as David suggested, a workaround is to configure Nomad to use the newer Envoy image version by either specifying the version in the job spec, or in the Nomad client's metadata.

# Override in Nomad job spec
job "<job name>" {

  group "<group>" {
    network {
      mode = "bridge"
    }

    service {
      ...
      connect {
        sidecar_service {}

        sidecar_task {
          config {
            image = "envoyproxy/envoy:1.23.1"
          }
        }
      }
    }
  }
}

# Override in Nomad client config
client {
  meta = {
    "connect.sidecar_image" = "envoyproxy/envoy:v1.23.1"
}

[mmeier86]

I've just upgraded my cluster to Consul v1.13.2 and can confirm that Envoy for aarch64 hosts now chooses a working image again.

Closing this ticket. Thanks for the release and PR!