Error when kubernetes cluster is running on arm64 processor

[ssleptsov]

What were you trying to do?

Swap development

What did you expect to happen?

Swap development

What happened instead?

Crash. My development machine is MacOS, but kubernetes cluster is running on arm64 processor, so in the pod, I'm getting such error also. How I can compile and run it arm64?

Message: standard_init_linux.go:178: exec user process caused "exec format error"

Automatically included information

Command line: ['/usr/local/bin/telepresence', '--swap-deployment', 'aos-auth', '--namespace=aos-auth'] Version: 0.92-3-gc40ed30 Python version: 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] kubectl version: Client Version: v1.9.3 // Server Version: v1.11.2 oc version: (error: [Errno 2] No such file or directory: 'oc': 'oc') OS: Darwin cyberss 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 Traceback:

Traceback (most recent call last):
  File "/usr/local/bin/telepresence/telepresence/cli.py", line 129, in crash_reporting
    yield
  File "/usr/local/bin/telepresence/telepresence/main.py", line 61, in main
    remote_info = start_proxy(runner)
  File "/usr/local/bin/telepresence/telepresence/proxy/__init__.py", line 90, in start_proxy
    run_id=run_id,
  File "/usr/local/bin/telepresence/telepresence/proxy/remote.py", line 205, in get_remote_info
    wait_for_pod(runner, remote_info)
  File "/usr/local/bin/telepresence/telepresence/proxy/remote.py", line 136, in wait_for_pod
    "Pod isn't starting or can't be found: {}".format(pod["status"])
RuntimeError: Pod isn't starting or can't be found: {'conditions': [{'lastProbeTime': None, 'lastTransitionTime': '2018-08-24T10:12:15Z', 'status': 'True', 'type': 'Initialized'}, {'lastProbeTime': None, 'lastTransitionTime': '2018-08-24T10:12:15Z', 'message': 'containers with unready status: [aos-auth]', 'reason': 'ContainersNotReady', 'status': 'False', 'type': 'Ready'}, {'lastProbeTime': None, 'lastTransitionTime': None, 'message': 'containers with unready status: [aos-auth]', 'reason': 'ContainersNotReady', 'status': 'False', 'type': 'ContainersReady'}, {'lastProbeTime': None, 'lastTransitionTime': '2018-08-24T10:12:15Z', 'status': 'True', 'type': 'PodScheduled'}], 'containerStatuses': [{'containerID': 'docker://863473f4a587364bbd6162c1e147c75ec0a1d077a16f32464895cd1c73b7bd48', 'image': 'datawire/telepresence-k8s:0.92', 'imageID': 'docker-pullable://datawire/telepresence-k8s@sha256:0fe11f7d0b2a28d681a36ef090e0519475245b7a22fdf66d3174c54365c14730', 'lastState': {'terminated': {'containerID': 'docker://863473f4a587364bbd6162c1e147c75ec0a1d077a16f32464895cd1c73b7bd48', 'exitCode': 1, 'finishedAt': '2018-08-24T10:13:58Z', 'message': 'standard_init_linux.go:178: exec user process caused "exec format error"\n', 'reason': 'Error', 'startedAt': '2018-08-24T10:13:58Z'}}, 'name': 'aos-auth', 'ready': False, 'restartCount': 4, 'state': {'waiting': {'message': 'Back-off 1m20s restarting failed container=aos-auth pod=aos-auth-545e19e6d90740b58e9277148c0a023d-7b785c5cf9-g5g6b_aos-auth(2d18d56b-a786-11e8-97d7-5a58c176161f)', 'reason': 'CrashLoopBackOff'}}}], 'hostIP': '192.168.1.169', 'phase': 'Running', 'podIP': '10.244.0.70', 'qosClass': 'BestEffort', 'startTime': '2018-08-24T10:12:15Z'}

Logs:

s-auth get pod aos-auth-545e19e6d90740b58e9277148c0a023d-7b785c5cf9-g5g6b -o json
 125.2 TEL | [272] Capturing: kubectl --context kubernetes-admin@kubernetes --namespace aos-auth get pod aos-auth-545e19e6d90740b58e9277148c0a023d-7b785c5cf9-g5g6b -o json
 125.6 TEL | [273] Capturing: kubectl --context kubernetes-admin@kubernetes --namespace aos-auth get pod aos-auth-545e19e6d90740b58e9277148c0a023d-7b785c5cf9-g5g6b -o json
 126.1 TEL | [274] Capturing: kubectl --context kubernetes-admin@kubernetes --namespace aos-auth get pod aos-auth-545e19e6d90740b58e9277148c0a023d-7b785c5cf9-g5g6b -o json
 126.6 TEL | [275] Capturing: kubectl --context kubernetes-admin@kubernetes --namespace aos-auth get pod aos-auth-545e19e6d90740b58e9277148c0a023d-7b785c5cf9-g5g6b -o json
 127.1 TEL | [276] Capturing: kubectl --context kubernetes-admin@kubernetes 

[ark3]

You can build ARM64 images on your hardware. The steps are a little complicated; we need to document this better. Here's an edited copy-paste from my shell session, where I build images, push them to my own Docker registry (ark3), and then run Telepresence using those images. You'll want to build and push the images from an ARM64 machine.

Clone the source tree and try something like this.

$ make version
0.92-3-gc40ed30
$ export TELEPRESENCE_VERSION=0.92-3-gc40ed30
$ export TELEPRESENCE_REGISTRY=ark3  # Use your Docker registry, e.g., ssleptov
$ 
$ # Build and push the images
$ ./build --registry $TELEPRESENCE_REGISTRY --no-tests --build-and-push --version-suffix ""
[...]
653b27b59784: Layer already exists 
8dfad2055603: Layer already exists 
0.92-3-gc40ed30: digest: sha256:98adf076b55bac5e48ceb3a3a3d1f02330e91dc206b0deb413b53b6e0024b09e size: 1575
$ 
$ # Run Tel, use the images
$ # (depends on env vars from above)
$ telepresence --run curl -sk https://kubernetes.default/api                                                           
T: Starting proxy with method 'vpn-tcp', which has the following limitations: All processes are affected, only one     
[...]
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "35.184.12.241"
    }
  ]
}T: Exit cleanup in progress
$ 

Use your own Docker registry name in place of ark3 above, of course. Also note the empty string passed to the --version-suffix option to the build script; if you don't include that, the script will attach an OS and timestamp to the version.

When you run Telepresence, it will use the values in TELEPRESENCE_REGISTRY and TELEPRESENCE_VERSION to pick the image it launches in your cluster. This will allow you to test and ultimately use your images. Just be aware of those environment variables; Telepresence does not report to the user that it is using alternate images except in the log file. (Perhaps this should be more verbose and explicit?)

The developer experience around Telepresence needs some work. We would appreciate your feedback on all of this. Thank you!

[ssleptsov]

Great, Thanks! I will try this and write a feedback. Thanks for the great tool!

[ark3]

For future reference: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/

[ssleptsov]

@ark3 Thanks, and I confirm that your script helps me compile and successfully run it at arm64 device.

[richarddli]

@ssleptsov Thanks! Do you mind adding docs on how to do this on https://github.com/telepresenceio/telepresence/blob/master/docs/reference/developing.md? (Or maybe some other place if you think it would make more sense). We need to revamp the doc structure, but I don't want to lose the info.

[dkrizic]

The perfect solution would be:

• telepresence has a Docker image with multi arch support for amd64 and arm64
• telepresence has a option with a node selector for force running the pod on a selected node

[dklotz]

Since I just ran into the dreaded exec user process caused "exec format error" (from the logs of the pod, this always shows it's trying to run a binary with the wrong arch) when I wanted to try telepresence on my local RPi cluster (installed with k3s), I can wholeheartedly second the suggestion of building a multiarch image for telepresence (amd64, arm64 and arm if possible). It would be really awesome if this just worked out of the box on arm.

[donnyyung]

I'm not sure if this is still an issue with Telepresence 2, so we should test this and find out. Until then leaving this issue open.

[jason-edstrom]

I'm not sure if this is still an issue with Telepresence 2, so we should test this and find out. Until then leaving this issue open.

I'm currently attempting this install with a raspberry pi K3s cluster and the traffic manager currently crashes in my setup. Comes back with a CrashLoopback error.

[XiNiHa]

I tried to build an image for ARM64 by myself (since the targeting PR is somewhat outdated) but didn't get it working. (my knowledge about Docker and all the build system stuff is kinda limited) Can we expect any support here? Since the traffic binary itself looks working on Linux ARM64 without any problem, I guess it'll not require that much work to add support for this. The previous PR was stuck on the issue related to ko publish and I believe that's not the case anymore since I don't see any usage of ko anymore.

[thallgren]

We support arm64 since 2.11.1

[alaminopu]

@thallgren I don't see the arm64 image here - https://hub.docker.com/r/datawire/tel2. What am I missing?

[thallgren]

@alaminopu we currently only support arm64 with our pro-version, which can be found here: https://hub.docker.com/r/datawire/ambassador-telepresence-agent.