kafka-docker
kafka-docker copied to clipboard
wurstmeister
Support for ARM64
Hi all,
With better ARM64 devices being released in increased speed, it is getting easier and easier to setup a local cluster for testing purposes (and even production of smaller scale cases).
The current image derives from openjdk:8u212-jre-alpine which is available in the ARM64 architecture, so it should be fairly easy (I assume) to release an image for that architecture too. The Zookeeper image (also from wurstmeister) does this.
I have tried many docker images of kafka on arm (which all have 0 documentation) but so far couldn't make any of them work on my swarm cluster. I am running Kali 64bit OS on a cluster of RaspberryPi4 4gbs. Would be happy to receive any pointers from you.
For anyone else who might need this: I cloned the repo on my OS (Kali, 64bit ARM) and built it there. Worked out of the box. So I think only a multi-arch release is missing in this repo.
the only problem i see is, the installation of glibc which is not aarch64 i guess.
Can confirm it's working without problems! I was able to get it running on a raspberry pi and had to build the image there, thank you.
@yalopov I see this ticket is still opened, can you share where can I get the official build support for AMR?
As far as I know, wurstmeister/kafka-docker is not generating arm builds, I had to build it myself on a raspberry pi (any aarch64 environment will do) and published it to dockerhub (nyadesu/kafka-docker)
@yalopov thanks for your help, so in order to build kafka for ARM, we just need to run docker build on any aarch64 env?
@yalopov thanks for your help, so in order to build kafka for ARM, we just need to run
docker buildon any aarch64 env?
Yes! In fact what some other projects are doing to handle this is to setup an aarch64 env with QEMU and building it there
@yalopov, thank you very much, the image you provide works for me too!
Wondering why wurstmeister/kafka-docker does not internally support ARM64 image by using a multi-arch parent images and multi-arch docker builds, using docker buildx. This solution would be more maintainable, because every version of this image will become available on all possible architectures.
So, I've tried to clone and build it with
docker build . or docker build --platform linux/arm64 . or docker build --platform linux/arm64/v8 .
which resulted in
Step 11/14 : RUN apk add --no-cache bash curl jq docker && chmod a+x /tmp/*.sh && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKA_HOME} && rm /tmp/* && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk && rm glibc-${GLIBC_VERSION}.apk ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested ---> Running in 38ccc2ea820d standard_init_linux.go:228: exec user process caused: exec format error
every single time.
Is there anything I did wrong?
the glibc version (https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk) is still amd64 i guess you must build it yourself.
It looks as though an attempt was made to create arm64 images but those images are still running x64 JRE. I am referring to this commit: https://github.com/wurstmeister/kafka-docker/commit/c4e8f17fc9bbfe713b31bd9969babbb59523147b
docker run wurstmeister/kafka:2.13-2.8.1 arch
Unable to find image 'wurstmeister/kafka:2.13-2.8.1' locally
2.13-2.8.1: Pulling from wurstmeister/kafka
Digest: sha256:4916aa312512d255a6d82bed2dc5fbee29df717fd9efbdfd673fc81c6ce03a5f
Status: Downloaded newer image for wurstmeister/kafka:2.13-2.8.1
x86_64
Is there any developments on this? It's a bit annoying that based on the Docker Hub ARM64 is supported but those images are still have AMD64 binaries. The Dockerfile contains a step which fetches an AMD64 only glibc APK. I think until that is there then the image won't really usable on ARM64 which is sad because this blocks our migration to AWS Graviton instances.
I also had problems running the docker image on raspberry pi. Also building the Dockerfile was not possible. I have changed following in the Dockerfile locally and build it on raspberry Pi 4B:
Changed version to 18 -> supports ARM FROM azul/zulu-openjdk-alpine:18
Changed glibc_version to 2.35 ARG glibc_version=2.35-r0
I don't know if there are arising any new issues, but with the new build i finally got my project running.
https://hub.docker.com/r/ricktauss/kafka
Hello everyone, check if this helps https://github.com/arm64-compat/confluent-platform/pkgs/container/confluentinc%2Fcp-kafka
These are not wurstmeister/kafka-docker images but confluent ones. I usually run arm64-compat project to provide compatibility for the popular images which doesn't currently have an official arm64 support.
I would try to port wurstmeister/kafka-docker images if there is enough support, meanwhile I have been using confluent images on my local and it is working perfectly.
These images are built on arm64 machine and not using qemu simulations, so won't be having any downside in performance on local.
Having wurstmeister/kafka-docker would be great. I can help with them and test. The confluent images don't support everything that these images do. For example https://github.com/confluentinc/kafka-images/issues/70 is really nice for us as we manage kafka topics outside of our applications.
I also had problems running the docker image on raspberry pi. Also building the Dockerfile was not possible. I have changed following in the Dockerfile locally and build it on raspberry Pi 4B:
Changed version to 18 -> supports ARM FROM azul/zulu-openjdk-alpine:18
Changed glibc_version to 2.35 ARG glibc_version=2.35-r0
I don't know if there are arising any new issues, but with the new build i finally got my project running.
https://hub.docker.com/r/ricktauss/kafka
this glibc_"version".apk support aarch64/amd64 at same time? @ricktauss
update: It seems like glibc_"version".apk does not support aarch64, https://github.com/SonarSource/docker-sonarqube/issues/526 we can delete glibc related code because kafka doesn't need it to run
