M1 Mac - confluent-kafka librdkafka error message when building Docker image

[yusufosman]

Description

I'm getting the below error whenever I try to pip install confluent-kafka within my Dockerfile and I build that Dockerfile using python:3.10-slim-bullseye as my base image.

error: #error "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"

Please note that I have already installed librdkafka-dev as part of my Dockerfile, and this has been working fine for months until recently, but now it seems to be complaining that I don't have librdkafka and I'm not sure why.

How to reproduce

Dockerfile

FROM public.ecr.aws/docker/library/python:3.10-slim-bullseye

# hadolint ignore=DL3008
RUN apt-get update && apt-get install --no-install-recommends -y \
  gcc \
  librdkafka-dev \
  build-essential \
  && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir --upgrade pip==22.1.2 && pip install confluent-kafka

run docker build - < Dockerfile against the above Docker file and you will see the error I provided.

The only way I can successfully build the image is via the following command: docker build --platform linux/amd64 - < Dockerfile

[mhowlett]

it seems like the librdkafka version installed via librdkafka-dev is less than 1.9.0.

does it work without installing librdkafka-dev? (i'm unsure what architecture will be required given your setup, but think there is a binary wheel that might work)

[yusufosman]

it seems like the librdkafka version installed via librdkafka-dev is less than 1.9.0.

does it work without installing librdkafka-dev? (i'm unsure what architecture will be required given your setup, but think there is a binary wheel that might work)

No unfortunately not, I need to have librdkafka-dev in order for for the build to work. If I don't use librdkafka-dev then I get different errors related to how it can't find certain header files.

[pontusstjerna]

I ran into exactly the same problem and for now I build from source with these few lines:

RUN apt update && apt -y install software-properties-common gcc
RUN git clone https://github.com/edenhill/librdkafka
RUN cd librdkafka && ./configure && make && make install && ldconfig

Works for me, although its a temporary fix until they update to 1.9.0

[tfrokt]

Thanks @pontusstjerna that finally worked for me.

I have a similar issue installing it in Docker FROM ubuntu:22.04 on my M1 host.

Tried to add debian/ubuntu repository:

> wget -qO - https://packages.confluent.io/deb/7.2/archive.key | gpg --dearmor | tee /etc/apt/trusted.gpg.d/confluent.gpg > /dev/null
> add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/7.2 stable main"
> add-apt-repository "deb https://packages.confluent.io/clients/deb focal main"  # jammy release doesn't exist
> apt udpate
> apt install confluent-platform

with the following outcome:

confluent-platform : Depends: confluent-cli (= 7.2.1-1) but it is not installable
                      Recommends: confluent-server (= 7.2.1-1) but it is not going to be installed or
                                  kafka

alright, let's install kafka then:

> apt install -y kafka

Package kafka is a virtual package provided by:
  confluent-server 7.2.1-1
  confluent-kafka 7.2.1-1
You should explicitly select one to install.

E: Package 'kafka' has no installation candidate

:(

apt install confluent-kafka

works.

> apt install confluent-platform

The following packages have unmet dependencies:
 confluent-platform : Depends: confluent-cli (= 7.2.1-1) but it is not installable
E: Unable to correct problems, you have held broken packages.

https://docs.confluent.io/confluent-cli/current/install.html

> curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest

confluentinc/cli crit platform linux/arm64 is not supported.  Please contact Confluent support if you believe this is a mistake.

Alright, I am giving up.

Thanks again @pontusstjerna for the solution that worked.

[tmbo]

I am running into this issue as well trying to build an ubuntu based container on an M1 host. I can confirm that following the instructions to build from source results in a librdkafka-dev < 1.9.0

$ apt install -y librdkafka-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  librdkafka++1 librdkafka1
The following NEW packages will be installed:
  librdkafka++1 librdkafka-dev librdkafka1
0 upgraded, 3 newly installed, 0 to remove and 8 not upgraded.
Need to get 1313 kB of archives.
After this operation, 6261 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main arm64 librdkafka1 arm64 1.6.0-1 [515 kB]
Get:2 http://deb.debian.org/debian bullseye/main arm64 librdkafka++1 arm64 1.6.0-1 [51.5 kB]
Get:3 http://deb.debian.org/debian bullseye/main arm64 librdkafka-dev arm64 1.6.0-1 [746 kB]
Fetched 1313 kB in 1s (2574 kB/s)      
Selecting previously unselected package librdkafka1:arm64.
(Reading database ... 32324 files and directories currently installed.)
Preparing to unpack .../librdkafka1_1.6.0-1_arm64.deb ...
Unpacking librdkafka1:arm64 (1.6.0-1) ...
Selecting previously unselected package librdkafka++1:arm64.
Preparing to unpack .../librdkafka++1_1.6.0-1_arm64.deb ...
Unpacking librdkafka++1:arm64 (1.6.0-1) ...
Selecting previously unselected package librdkafka-dev:arm64.
Preparing to unpack .../librdkafka-dev_1.6.0-1_arm64.deb ...
Unpacking librdkafka-dev:arm64 (1.6.0-1) ...
Setting up librdkafka1:arm64 (1.6.0-1) ...
Setting up librdkafka++1:arm64 (1.6.0-1) ...
Setting up librdkafka-dev:arm64 (1.6.0-1) ...
Processing triggers for libc-bin (2.31-13+deb11u4) ...

[mhowlett]

using #1439 to track this.

you could consider getting the 1.9.x source from github, rather than install with a package.

[anmollp]

I had a similar issue:

In my specific case, the confluent-kafka version needed was >= 2.0.2, hence the version in command. Since I am using a python slim buster, that's why wget, gcc, g++ and make, otherwise rest should work.

This worked for me:

RUN apt-get update && apt-get install -y wget gcc g++ make
RUN wget https://github.com/edenhill/librdkafka/archive/v2.0.2.tar.gz && tar xvzf v2.0.2.tar.gz
RUN cd librdkafka-2.0.2/ && ./configure && make && make install && ldconfig

[harshit98]

@anmollp - the solution which you've shared worked for me, thanks! 👍🏼