pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

ARM based docker image

Open vkuznecovas opened this issue 3 years ago • 29 comments

Is your enhancement request related to a problem? Please describe. I'd like to see an ARM based docker image for pulsar. The problem is that under new macs the pulsar containers have to be run under emulation which makes them horribly slow.

Describe the solution you'd like ARM based docker image should be available in the apachepulsar/pulsar repo.

Describe alternatives you've considered Building a native image has not proved trivial, so ideally someone with more knowledge of the project would be able to create one.

vkuznecovas avatar Nov 23 '21 13:11 vkuznecovas

+1. This would be a good feature for end users. My only concern as a contributor is that it will increase build times. That shouldn't stop this feature, though.

Ideally, we'll be able to follow the instructions here: https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/. I haven't done a mutli-arch build using maven before (we build our docker images using maven), so that is one nuance.

michaeljmarshall avatar Nov 24 '21 05:11 michaeljmarshall

I'm glad to work on this if there is nobody already working on it? I've recently created a build job to build amd64/arm64 Docker images for Apache SkyWalking and can apply the experience here, perhaps.

kezhenxu94 avatar Dec 01 '21 13:12 kezhenxu94

Thanks @kezhenxu94, I have assigned the issue to you. Looking forward to your PRs.

codelipenghui avatar Dec 02 '21 13:12 codelipenghui

I am also interested in support for ARM docker images, especially for testing locally on Apple Silicon based machines

krissetto avatar Dec 07 '21 14:12 krissetto

I am also interested in support for ARM docker images, especially for testing locally on Apple Silicon based machines

Thank you @krissetto !! Will let you know when I have a docker image for testing on ARM!

kezhenxu94 avatar Dec 07 '21 14:12 kezhenxu94

+1!

lorthirk avatar Dec 09 '21 11:12 lorthirk

+1

jibbolo avatar Dec 09 '21 14:12 jibbolo

Hi all in this thread, I just pushed a multi-arch image to my own repo, anyone interested is welcome to pull and test it in your environment. https://hub.docker.com/repository/docker/kezhenxu94/pulsar/tags?page=1&ordering=last_updated

To test, just replace the docker image with kezhenxu94/pulsar, example:

docker run -it \                                                                                                                  
  -p 6650:6650 \
  -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  kezhenxu94/pulsar \
  bin/pulsar standalone

Note this is not official and not well tested now, I'll test, polish and then open a PR so everyone can review on the build process.

Any feedback are also welcome and appreciated 🙇

kezhenxu94 avatar Dec 14 '21 07:12 kezhenxu94

Hi all in this thread, I just pushed a multi-arch image to my own repo, anyone interested is welcome to pull and test it in your environment. https://hub.docker.com/repository/docker/kezhenxu94/pulsar/tags?page=1&ordering=last_updated

To test, just replace the docker image with kezhenxu94/pulsar, example:

docker run -it \                                                                                                                  
  -p 6650:6650 \
  -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  kezhenxu94/pulsar \
  bin/pulsar standalone

Note this is not official and not well tested now, I'll test, polish and then open a PR so everyone can review on the build process.

Any feedback are also welcome and appreciated 🙇

I've tested it out, seems to work great. Thanks!

vkuznecovas avatar Dec 15 '21 09:12 vkuznecovas

I have also started testing it on an M1 device, and it seems to work 😄 I haven't gone very deep into it yet though, so i'll be back with feedback if I encounter any specific issues with it. Thanks for the build! Ping when you have a PR too since i'm curious

krissetto avatar Dec 15 '21 10:12 krissetto

I have also started testing it on an M1 device, and it seems to work 😄 I haven't gone very deep into it yet though, so i'll be back with feedback if I encounter any specific issues with it. Thanks for the build! Ping when you have a PR too since i'm curious

Sure, will keep you all in this thread posted. Thanks all for testing. 🙇

kezhenxu94 avatar Dec 15 '21 11:12 kezhenxu94

I can also validate that this appears to work on M1 Pro Max based machine. Would love to see the PR so we can get an official image; this works for local development in the meantime (and eliminates yet another tie to an Intel based machine!). Thanks so much!

tdrozdowski avatar Dec 29 '21 17:12 tdrozdowski

Heads up: The last (hard) step is to build the pulsar client with multiple architectures. Everything else is ready. I have some ideas to build pulsar client in multi architectures but need some time to verify, which may be also ready in 2 weeks (due to public holidays and my personal stuffs)

kezhenxu94 avatar Dec 29 '21 17:12 kezhenxu94

@kezhenxu94 thanks for working on this, our company relies on pulsar and a recent move to Mac M1 has led us to run into a ton of problems with local development. Would it be possible to share the Dockerfile so that we could temporarily build arm supported images for older versions of pulsar? If not, do you have a timeline so we can be ready to start switching out our images?

mgagliardo91 avatar Jan 27 '22 14:01 mgagliardo91

@kezhenxu94 thanks for working on this, our company relies on pulsar and a recent move to Mac M1 has led us to run into a ton of problems with local development. Would it be possible to share the Dockerfile so that we could temporarily build arm supported images for older versions of pulsar? If not, do you have a timeline so we can be ready to start switching out our images?

I can draft a PR of what I've done tomorrow (in my time zone) so that you can try it out. Stay tune please.

kezhenxu94 avatar Jan 27 '22 15:01 kezhenxu94

Hi, I've just drafted a PR https://github.com/apache/pulsar/pull/14005

kezhenxu94 avatar Jan 28 '22 09:01 kezhenxu94

Are there any updates on this issue? I see the draft PR hasn't changed since last month's revision

krissetto avatar Feb 21 '22 15:02 krissetto

The issue had no activity for 30 days, mark with Stale label.

github-actions[bot] avatar Mar 24 '22 02:03 github-actions[bot]

For those of you who are stuck at this issue, I was able to create ARM docker images on my M1 Mac by the following simple change. The test dockers from the following builds passed the shade integration test on my laptop.

# docker builds
mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

# shade test run
./build/run_integration_group.sh SHADE
hsohn@HeesungSohns-MacBook-Pro pulsar % git diff
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index 27fef14fb7..201ca40115 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -76,8 +76,12 @@ RUN pip3 install pyyaml==5.4.1
 # 4. /pulsar - hadoop writes to this directory
 RUN mkdir /pulsar && chmod g+w /pulsar

-ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
-RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
+RUN java_home=$(ls -d1 /usr/lib/jvm/java-17-openjdk-*) ; \
+    ln -s "$java_home" /usr/lib/jvm/java
+
+ENV JAVA_HOME /usr/lib/jvm/java
+RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java/conf/security/java.security
+
 ADD target/python-client/ /pulsar/pulsar-client

 ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE
diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
index 048b3da7cb..d4aa4d2eb1 100644
--- a/docker/pulsar/pom.xml
+++ b/docker/pulsar/pom.xml
@@ -75,7 +75,7 @@
                   <executable>${project.basedir}/../../pulsar-client-cpp/docker/build-wheels.sh</executable>
                   <arguments>
                     <!-- build python 3.8 -->
-                    <argument>3.8 cp38-cp38 manylinux2014 x86_64</argument>
+                    <argument>3.8 cp38-cp38 manylinux2014 aarch64</argument>
                   </arguments>
                 </configuration>
               </execution>
diff --git a/docker/pulsar/scripts/install-pulsar-client.sh b/docker/pulsar/scripts/install-pulsar-client.sh
index 56702b10db..bf81ccad97 100755
--- a/docker/pulsar/scripts/install-pulsar-client.sh
+++ b/docker/pulsar/scripts/install-pulsar-client.sh
@@ -23,3 +23,9 @@ set -x
 PYTHON_MAJOR_MINOR=$(python3 -V | sed -E 's/.* ([[:digit:]]+)\.([[:digit:]]+).*/\1\2/')
 WHEEL_FILE=$(ls /pulsar/pulsar-client | grep "cp${PYTHON_MAJOR_MINOR}")
 pip3 install /pulsar/pulsar-client/${WHEEL_FILE}[all]
+
+# The following `update build-essential python3-dev` are added due to grpcio `cc` not found error
+# TODO: remove these lines if the error got resolved.
+# WARNING: currently grpcio build takes long.
+apt-get update
+apt-get -y install build-essential python3-dev
diff --git a/tests/docker-images/java-test-image/Dockerfile b/tests/docker-images/java-test-image/Dockerfile
index b7f12b7596..99f626f1f6 100644
--- a/tests/docker-images/java-test-image/Dockerfile
+++ b/tests/docker-images/java-test-image/Dockerfile
@@ -40,8 +40,10 @@ RUN sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirr
      && apt-get -y dist-upgrade \
      && apt-get -y install openjdk-17-jdk-headless

-ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
-RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
+RUN java_home=$(ls -d1 /usr/lib/jvm/java-17-openjdk-*) ; \
+    ln -s "$java_home" /usr/lib/jvm/java
+ENV JAVA_HOME /usr/lib/jvm/java
+RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java/conf/security/java.security

 # /pulsar/bin/watch-znode.py requires python3-kazoo
 # /pulsar/bin/pulsar-managed-ledger-admin requires python3-protobuf
diff --git a/tests/docker-images/latest-version-image/Dockerfile b/tests/docker-images/latest-version-image/Dockerfile
index f5c6726831..d6adb5cfa8 100644
--- a/tests/docker-images/latest-version-image/Dockerfile
+++ b/tests/docker-images/latest-version-image/Dockerfile
@@ -29,8 +29,8 @@ RUN apt-get install -y procps curl git build-essential

 ENV GOLANG_VERSION 1.15.8

-RUN curl -sSL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz \
-               | tar -C /usr/local -xz
+RUN export arch=$(uname -m | sed -r 's/aarch64/arm64/g' |  awk '!/arm64/{$0="amd64"}1'); \
+curl -sSL https://golang.org/dl/go$GOLANG_VERSION.linux-$arch.tar.gz \

 # RUN wget https://dl.google.com/go/go1.13.3.linux-amd64.tar.gz && tar -xvf go1.13.3.linux-amd64.tar.gz && mv go /usr/local
 # RUN export GOROOT=/usr/local/go && export GOPATH=$HOME/go && export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
(END)

heesung-sn avatar Apr 20 '22 03:04 heesung-sn

Hi All, I'm interested to learn if I can support this effort.

rhenwood-arm avatar Apr 26 '22 13:04 rhenwood-arm

Hi All, I'm interested to learn if I can support this effort.

PRs are welcome!

dave2wave avatar Apr 26 '22 17:04 dave2wave

We discussed the requirements for an Arm Docker image during our Pulsar Community meeting yesterday. The main dependency that hasn't been discussed here is our dependency on moving to Apache Bookkeeper version 4.15.0. That work is here: https://github.com/apache/pulsar/pull/15142. Also, the end goal should be to update our release procedure so that the release produces a multi-arch docker image. These are my notes from the meeting:

Michael: what needs to be done for an Arm docker image (https://github.com/apache/pulsar/issues/12944)? Matteo: need to get the go runtime for functions (that is easy). There are going to be conflicts in the Dockerfile because of removing Python 2. Python wheel files already have the script to build the Arm and the Intell files. We just need the Linux Intel and Linux Arm. Ideal state is that a release will have both images. Matteo doesn’t think CI should worry about this, but releases should. The build would specify the target arch. We’ll want to make sure that the release manager’s process is easy. Michael: is there anything about bookkeeper that should be optimized? Matteo: there are Linux Intel optimizations for the JNI. Therefore, it won’t be as fast on Arm. Bookkeeper would need to pack multiple shared libraries and cross compile for different target arch. Rocks DB does this. It makes the binary larger. That could be done in Bookkeeper. That isn’t a docker problem, though. Michael: do we need bookkeeper 4.15 for an Arm based docker image? Matteo: yes, and that is because we need it for the right Rocks DB version.

Here is a link to the mailing list notes: https://lists.apache.org/thread/kpddnlm85yj9r3y16xol1v4778k1w5dy.

michaeljmarshall avatar Apr 29 '22 15:04 michaeljmarshall

Looks like a related PR has been closed, but ARM image has not yet been published. I would appreciate any update on this

grishick avatar Jul 08 '22 23:07 grishick

Now that https://github.com/apache/pulsar/pull/15142 is merged, there are no technical blockers for this work to continue on in the Pulsar project. I'll follow up on this one to try to push this through for the 2.11 release.

michaeljmarshall avatar Jul 09 '22 06:07 michaeljmarshall

New mailing list thread: https://lists.apache.org/thread/52bncdx2j4j982m9xob85sjmsr23tgzt

michaeljmarshall avatar Jul 09 '22 07:07 michaeljmarshall

This can be related to #16652 .

Also, when I build on Apple M1 (ARM64), I find that pulsar-client-cpp/docker/manylinux1/Dockerfile binds to x86_64 and thus the build will fail with:

ERROR: pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl is not a supported wheel on this platform.

tisonkun avatar Aug 18 '22 08:08 tisonkun

Trying to workaround with:

$ pulsar-client-cpp/docker/build-wheels.sh 3.8 cp38-cp38 manylinux2014 aarch64
$ mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DskipBuildPythonClient=true -DskipSourceReleaseAssembly=true

We should integrate this snippet into build scripts, though.

FAILED with Python client depends on grpcio = 1.27.2, which doesn't support aarch64. A newer version can help, see also https://github.com/grpc/grpc/pull/25418 and https://github.com/grpc/grpc/issues/21283.

cc @BewareMyPower @shibd @RobertIndie

After a closer look, it seems the failure is not directly on arch:

File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic
cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'],
File "/usr/lib/python3.8/subprocess.py", line 858, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cc'
Build output:
[INFO] --- docker-maven-plugin:0.40.2:build (default) @ pulsar-docker-image ---
[INFO] Building tar: /Users/chenzili/Brittani/pulsar/docker/pulsar/target/docker/apachepulsar/pulsar/tmp/docker-build.tar
[INFO] DOCKER> [apachepulsar/pulsar:latest]: Created docker-build.tar in 2 seconds 
[INFO] DOCKER> Step 1/28 : FROM busybox as pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> a5ab4ab35b15
[INFO] DOCKER> Step 2/28 : ARG PULSAR_TARBALL
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> f654625fc5f5
[INFO] DOCKER> Step 3/28 : ADD ${PULSAR_TARBALL} /
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 3a10a7f0f465
[INFO] DOCKER> Step 4/28 : RUN mv /apache-pulsar-* /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 121b72edf26c
[INFO] DOCKER> Step 5/28 : COPY scripts/apply-config-from-env.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 29977ea123f6
[INFO] DOCKER> Step 6/28 : COPY scripts/apply-config-from-env-with-prefix.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> f3465cf9514c
[INFO] DOCKER> Step 7/28 : COPY scripts/gen-yml-from-env.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> f33d63ce245d
[INFO] DOCKER> Step 8/28 : COPY scripts/generate-zookeeper-config.sh /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> b824025f88bc
[INFO] DOCKER> Step 9/28 : COPY scripts/pulsar-zookeeper-ruok.sh /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> be232a3180e8
[INFO] DOCKER> Step 10/28 : COPY scripts/watch-znode.py /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> e1208c4fae8b
[INFO] DOCKER> Step 11/28 : COPY scripts/install-pulsar-client.sh /pulsar/bin
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 1de314e3f901
[INFO] DOCKER> Step 12/28 : RUN for SUBDIRECTORY in conf data download logs; do      [ -d /pulsar/$SUBDIRECTORY ] || mkdir /pulsar/$SUBDIRECTORY;      chmod -R g+w /pulsar/$SUBDIRECTORY;      done
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 1482c4534ac8
[INFO] DOCKER> Step 13/28 : RUN chmod g+w /pulsar/lib/presto
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> e7bf9d6530fc
[INFO] DOCKER> Step 14/28 : FROM ubuntu:20.04
[INFO] DOCKER> 
[INFO] DOCKER> ---> eb574ad4e57c
[INFO] DOCKER> Step 15/28 : ARG DEBIAN_FRONTEND=noninteractive
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 480737f2b5cc
[INFO] DOCKER> Step 16/28 : ARG UBUNTU_MIRROR=mirror://mirrors.ubuntu.com/mirrors.txt
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> b93da317a8e6
[INFO] DOCKER> Step 17/28 : RUN sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirrors.ubuntu.com/mirrors.txt}|g" /etc/apt/sources.list      && echo 'Acquire::http::Timeout "30";\nAcquire::ftp::Timeout "30";\nAcquire::Retries "3";' > /etc/apt/apt.conf.d/99timeout_and_retries      && apt-get update      && apt-get -y dist-upgrade      && apt-get -y install --no-install-recommends openjdk-17-jdk-headless netcat dnsutils less procps iputils-ping                  python3 python3-kazoo python3-pip                  curl ca-certificates      && apt-get -y --purge autoremove      && apt-get autoclean      && apt-get clean      && rm -rf /var/lib/apt/lists/*
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 953397f00737
[INFO] DOCKER> Step 18/28 : RUN pip3 install pyyaml==5.4.1
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 0d5d392d05bb
[INFO] DOCKER> Step 19/28 : RUN mkdir /pulsar && chmod g+w /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 31f147428bde
[INFO] DOCKER> Step 20/28 : ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> dba415d71afe
[INFO] DOCKER> Step 21/28 : RUN mkdir -p /usr/lib/jvm/java-17-openjdk-amd64/conf/security
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> d2cf7fef934e
[INFO] DOCKER> Step 22/28 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
[INFO] DOCKER> 
[INFO] DOCKER> ---> Using cache
[INFO] DOCKER> ---> 83eaa4f453f4
[INFO] DOCKER> Step 23/28 : ADD target/python-client/ /pulsar/pulsar-client
[INFO] DOCKER> 
[INFO] DOCKER> ---> cf055b5ede4f
[INFO] DOCKER> Step 24/28 : ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in 2d90766ce2e2
[INFO] DOCKER> Removing intermediate container 2d90766ce2e2
[INFO] DOCKER> ---> 951081b9ab71
[INFO] DOCKER> Step 25/28 : COPY --from=pulsar /pulsar /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> b9e5fd50bebc
[INFO] DOCKER> Step 26/28 : WORKDIR /pulsar
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in 4a02a04acbe7
[INFO] DOCKER> Removing intermediate container 4a02a04acbe7
[INFO] DOCKER> ---> 01c16f5856d5
[INFO] DOCKER> Step 27/28 : RUN /pulsar/bin/install-pulsar-client.sh
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in 0e4ed0ece4bb
[INFO] DOCKER> [91m++ python3 -V

[INFO] DOCKER> [91m++ sed -E 's/.* ([[:digit:]]+).([[:digit:]]+).*/\1\2/'

[INFO] DOCKER> [91m+ PYTHON_MAJOR_MINOR=38

[INFO] DOCKER> [91m++ ls /pulsar/pulsar-client

[INFO] DOCKER> [91m++ grep cp38

[INFO] DOCKER> [91m+ WHEEL_FILE=pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

  • pip3 install '/pulsar/pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl[all]'

[INFO] DOCKER> Processing ./pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl [INFO] DOCKER> Collecting certifi [INFO] DOCKER> Downloading certifi-2022.6.15-py3-none-any.whl (160 kB) [INFO] DOCKER> Collecting apache-bookkeeper-client>=4.9.2; extra == "all" [INFO] DOCKER> Downloading apache_bookkeeper_client-4.15.0-py2.py3-none-any.whl (70 kB) [INFO] DOCKER> Collecting prometheus-client; extra == "all" [INFO] DOCKER> Downloading prometheus_client-0.14.1-py3-none-any.whl (59 kB) [INFO] DOCKER> Collecting ratelimit; extra == "all" [INFO] DOCKER> Downloading ratelimit-2.2.1.tar.gz (5.3 kB) [INFO] DOCKER> Collecting protobuf<=3.20.*,>=3.6.1; extra == "all" [INFO] DOCKER> Downloading protobuf-3.19.4-cp38-cp38-manylinux2014_aarch64.whl (913 kB) [INFO] DOCKER> Collecting grpcio<1.28,>=1.8.2; extra == "all" [INFO] DOCKER> Downloading grpcio-1.27.2.tar.gz (16.9 MB) [INFO] DOCKER> [91m ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-od2wltzl/grpcio/setup.py'"'"'; file='"'"'/tmp/pip-install-od2wltzl/grpcio/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-od2wltzl/grpcio/pip-egg-info cwd: /tmp/pip-install-od2wltzl/grpcio/ Complete output (13 lines): /tmp/pip-install-od2wltzl/grpcio/src/python/grpcio/commands.py:102: SyntaxWarning: "is not" with a literal. Did you mean "!="? if exit_code is not 0: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 191, in if check_linker_need_libatomic(): File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'], File "/usr/lib/python3.8/subprocess.py", line 858, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'cc' ----------------------------------------

[INFO] DOCKER> [91mERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[INFO] DOCKER> Removing intermediate container 0e4ed0ece4bb [ERROR] DOCKER> Unable to build image [apachepulsar/pulsar] : "The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1" ["The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1" ]

tisonkun avatar Aug 18 '22 10:08 tisonkun

It seems all the issues are included in https://github.com/apache/pulsar/issues/12944#issuecomment-1103444708. I'll try it out.

tisonkun avatar Aug 19 '22 08:08 tisonkun

With #17733 we should be able to build ARM-based docker image manually. Integrating it with automation or CI is another topic.

tisonkun avatar Sep 20 '22 09:09 tisonkun

if you want to run pulsar standalone without docker you can simply add this to conf/standalone.conf, this will disable RocksDB

ledgerStorageClass=org.apache.bookkeeper.bookie.SortedLedgerStorage

eolivelli avatar Feb 01 '23 17:02 eolivelli