Docker Compose examples do not work on Apple M1 Hardware

[josephwoodward]

Describe the bug

When running the Docker Compose examples on Apple's M1 ARM hardware the synthetic load generator container starts but hangs (I'll update this with the error message seen in the logs later).

Looking at the GitHub repository for the image, it appears that it has been archived and is not actively maintained any more so there'll be no plans to support ARM based processors.

To Reproduce Steps to reproduce the behavior:

1. Start one of the Docker Compose examples that uses the synthetic-load-generator image on an Apple M1 machine

Expected behavior Docker Compose starts as expected and load is generated

Environment: Laptop

Additional Context

[kvrhdn]

Yeah, I have the same issue unfortunately. The synthetic load generator we use has been archived and its images are a couple of years old by now. I noticed older Docker images (1+ year) will often not work properly on M1.

We should fix this by replacing the Omnition/synthetic-load-generator by something that is somewhat more actively maintained and works on all common platforms. An alternative could be Jaeger's hotrod, but this requires some manual action to generate traces.

There is no proper workaround. You can build the image Docker image from source (if you have a Java toolchain), but the tags on Docker hub don't match with any tags in the repository. Unfortunately the json format changed somewhere so this will not work unless you change the input json as well. I haven't been able to find the exact commit matching with the version our examples use.

[josephwoodward]

Going to close this as it's not really an actionable issue other than to build it yourself as @kvrhdn suggests 👍

[kvrhdn]

I'm going to reopen, this is something we should fix.

[sidharthv96]

This is how I was able to get HotROD working. It does require a manual click in the UI, but I suppose that can be automated using a curl command if required.

@kvrhdn if that fix is ok, I can raise a PR.

diff --git a/example/docker-compose/local/docker-compose.yaml b/example/docker-compose/local/docker-compose.yaml
index f1d4203..3ad9d06 100644
--- a/example/docker-compose/local/docker-compose.yaml
+++ b/example/docker-compose/local/docker-compose.yaml
@@ -8,6 +8,7 @@ services:
       - ./tempo-local.yaml:/etc/tempo.yaml
       - ./tempo-data:/tmp/tempo
     ports:
+      - "6831"   # jaeger thrift compact
       - "14268"  # jaeger ingest
       - "3200"   # tempo
       - "55680"  # otlp grpc
@@ -15,12 +16,13 @@ services:
       - "9411"   # zipkin
 
   synthetic-load-generator:
-    image: omnition/synthetic-load-generator:1.0.25
-    volumes:
-      - ../shared/load-generator.json:/etc/load-generator.json
+    image: jaegertracing/example-hotrod:latest
+    command: [ "all" ]
     environment:
-      - TOPOLOGY_FILE=/etc/load-generator.json
-      - JAEGER_COLLECTOR_URL=http://tempo:14268
+      - JAEGER_AGENT_HOST=tempo
+      - JAEGER_AGENT_PORT=6831
+    ports: 
+      - "8080:8080"
     depends_on:
       - tempo

[urashidmalik]

Helm chart deployed two containers tempo and tempo query

Helm Chart Version: 0.14.2 Helm Chart Source: https://grafana.github.io/helm-charts

Tempo Seems to startup fine

level=info ts=2022-05-06T22:40:39.390339635Z caller=tempodb.go:459 msg="polling enabled" interval=5m0s concurrency=50
level=info ts=2022-05-06T22:40:39.390546802Z caller=ring.go:281 msg="ring doesn't exist in KV store yet"
level=info ts=2022-05-06T22:40:39.390572761Z caller=client.go:254 msg="value is nil" key=collectors/ring index=1
level=info ts=2022-05-06T22:40:39.390587177Z caller=module_service.go:64 msg=initialising module=distributor
level=info ts=2022-05-06T22:40:39.390580594Z caller=compactor.go:156 msg="enabling compaction"
level=warn ts=2022-05-06T22:40:39.390583219Z caller=rescan_blocks.go:24 msg="failed to open search wal directory" err="open /var/tempo/wal/search: no such file or directory"
level=info ts=2022-05-06T22:40:39.390606469Z caller=ingester.go:353 msg="wal replay complete"
level=info ts=2022-05-06T22:40:39.390626677Z caller=ingester.go:367 msg="reloading local blocks" tenants=0
level=info ts=2022-05-06T22:40:39.390663344Z caller=lifecycler.go:540 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2022-05-06T22:40:39.390696761Z caller=lifecycler.go:569 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2022-05-06T22:40:39.390778553Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
level=info ts=2022-05-06T22:40:39.390822553Z caller=module_service.go:64 msg=initialising module=querier
level=info ts=2022-05-06T22:40:39.390601261Z caller=tempodb.go:434 msg="compaction and retention enabled."
level=info ts=2022-05-06T22:40:39.390893761Z caller=worker.go:193 msg="adding connection" addr=127.0.0.1:9095
level=info ts=2022-05-06T22:40:39.391148887Z caller=app.go:270 msg="Tempo started"

Tempo-Query is failing with errors related ARM64/8 image missing.

standard_init_linux.go:228: exec user process caused: exec format error

Tempo-Query

---

Tempo

[urashidmalik]

Seems its fixed but the default Image for tempo-query is old in Helm Charts. tempo-query:1.4.1 Tag seems to be working on Apple M1 Pro

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.