librdkafka
librdkafka copied to clipboard
confluentinc
Runtime missing for Apple Silicon (M1)
Description
The NuGet package for .NET projects doesn't include the ARM runtime for macOS (osx-arm64) and therefore it doesn't work.
https://nuget.info/packages/librdkafka.redist/1.7.0
How to reproduce
Runtime is missing
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
- [x] librdkafka version (release number or git tag):
1.7.0 - [x] Apache Kafka version:
<REPLACE with e.g., 0.10.2.3> - [x] librdkafka client configuration:
<REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..> - [x] Operating system:
macOS 11.4, Apple M1 - [ ] Provide logs (with
debug=..as necessary) from librdkafka - [ ] Provide broker log excerpts
- [x] Critical issue
We're currently transitioning CI systems, so M1 builds will unfortunately miss the upcoming 1.9 release, but will be included in the next release after that.
No update. We're probably moving to Semaphore CI, but it too lacks M1 workers, so we'll see.
@edenhill Regardless of CI, looks like tons of people are having this issue including core team members. Any reason this can't be fixed regardless of CI status? I think most of the community would help test this in the mean time. Even call it alpha support on a stable release if you have to.
We use CIs to verify builds&tests, as well as create build artifacts, as they provide an isolated, dependable, audited, and integrated (with Github, etc) environment. So until there's a viable CI that provides M1 workers, there isn't much we can do. But we're actively working on getting such support on Semaphore to unblock you all.
I get that, one could also commit prebuilt from a signed commit that could be trusted and audited for the short term, not ideal... also why not use this https://github.com/actions/virtual-environments/issues/2187 with self hosted runner?
Hi, I've just noticed this thread... I'm curious how significant the M1 hardware is - and if a general AArch64 Linux platform would help fill a gap?
M1 is significant because Apple is pushing it quite hard, and apparently it's now hard for developers to get new Intel-based Macs even though so many things are not yet compatible with M1.
linux-aarch64 would be perfect for us because the M1 users would actually be running this under Docker, on a Linux VM.
M1 is significant because Apple is pushing it quite hard, and apparently it's now hard for developers to get new Intel-based Macs even though so many things are not yet compatible with M1.
Can confirm this impacts a lot of development teams.
M1 support is available as of https://github.com/edenhill/librdkafka/releases/tag/v1.9.1
M1 support is available OOTB in 1.9.1 for OSX. We don't yet bundle a linux-aarch64 librdkafka build. It seems there is sufficient demand though, so we probably will in the future. Confluent.Kafka should work with a custom linux-aarch64 librdkafka build.
Gotcha. We're trying to release Docker images for Intel and M1 macbooks to run. Running the code (tests) locally has been working for a while, but the ARM containers fail to find librdkafka.so.1
M1 support is available OOTB in 1.9.1 for OSX. We don't yet bundle a linux-aarch64 librdkafka build. It seems there is sufficient demand though, so we probably will in the future.
Any ETA for this? @mhowlett
@mhowlett What needs to be done to provide support for linux-aarch64 builds?
@jonchiu Any plans/roadmap on adding linux-aarch64 builds? Let me know if I can be of any help.
@edenhill I am not familiar with the codebase. Not sure if this is a minor or major issue.
Just checking if there are any plans to take it up soon.
With ChatGPT help and after a bit of trial and error, I managed to build linux-aarch64 wheels using the below Dockerfile
FROM ubuntu:22.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update && apt install -y wget git curl g++ make postgresql-client \
nano less shared-mime-info openjdk-17-jre-headless \
libpq-dev vim tzdata python3 python3-dev
RUN apt install -y python3-pip
RUN python3 -m pip install setuptools
WORKDIR /
RUN git clone https://github.com/confluentinc/confluent-kafka-python
WORKDIR confluent-kafka-python
COPY . /app
WORKDIR /app
RUN ./configure --arch=aarch64 --prefix=/usr
RUN make
RUN make install
WORKDIR /confluent-kafka-python
RUN python3 setup.py install