Unable to build docker image with librdkafka and confluent-kafka

[sngit1234]

trafficstars

Description

I am using 3.8-alpine image to create docker container and trying to install librdkafka and confluent-kafka libraries but getting error while installing confluent-kafka error: command 'gcc' failed with exit status 1

#16 6.183 gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.8 -c /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.8/tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.o #16 6.183 In file included from /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:17: #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.h:66:2: error: #error "confluent-kafka-python requires librdkafka v2.1.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html" #16 6.183 66 | #error "confluent-kafka-python requires librdkafka v2.1.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html" #16 6.183 | ^~~~~ #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.h:386:30: error: unknown type name 'rd_kafka_Node_t' #16 6.183 386 | PyObject *c_Node_to_py(const rd_kafka_Node_t *c_node); #16 6.183 | ^~~~~~~~~~~~~~~ #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c: In function 'Message_new0': #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:799:30: warning: implicit declaration of function 'rd_kafka_message_leader_epoch'; did you mean 'rd_kafka_message_headers'? [-Wimplicit-function-declaration] #16 6.183 799 | self->leader_epoch = rd_kafka_message_leader_epoch(rkm); #16 6.183 | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #16 6.183 | rd_kafka_message_headers #16 6.183 In file included from /usr/local/include/python3.8/Python.h:106, #16 6.183 from /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.h:18: #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c: In function 'c_parts_to_py': #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1149:41: warning: implicit declaration of function 'rd_kafka_topic_partition_get_leader_epoch'; did you mean 'TopicPartition_get_leader_epoch'? [-Wimplicit-function-declaration] #16 6.183 1149 | rd_kafka_topic_partition_get_leader_epoch(rktpar), #16 6.183 | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #16 6.183 /usr/local/include/python3.8/listobject.h:73:74: note: in definition of macro 'PyList_SET_ITEM' #16 6.183 73 | #define PyList_SET_ITEM(op, i, v) (((PyListObject *)(op))->ob_item[i] = (v)) #16 6.183 | ^ #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c: In function 'py_to_c_parts': #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1193:17: warning: implicit declaration of function 'rd_kafka_topic_partition_set_leader_epoch'; did you mean 'TopicPartition_get_leader_epoch'? [-Wimplicit-function-declaration] #16 6.183 1193 | rd_kafka_topic_partition_set_leader_epoch(rktpar, #16 6.183 | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #16 6.183 | TopicPartition_get_leader_epoch #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c: At top level: #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1450:30: error: unknown type name 'rd_kafka_Node_t' #16 6.183 1450 | PyObject *c_Node_to_py(const rd_kafka_Node_t *c_node) { #16 6.183 | ^~~~~~~~~~~~~~~ #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c: In function 'c_Node_to_py': #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1464:41: warning: implicit declaration of function 'rd_kafka_Node_id'; did you mean 'rd_kafka_commit'? [-Wimplicit-function-declaration] #16 6.183 1464 | cfl_PyDict_SetInt(kwargs, "id", rd_kafka_Node_id(c_node)); #16 6.183 | ^~~~~~~~~~~~~~~~ #16 6.183 | rd_kafka_commit #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1465:43: warning: implicit declaration of function 'rd_kafka_Node_port'; did you mean 'rd_kafka_queue_poll'? [-Wimplicit-function-declaration] #16 6.183 1465 | cfl_PyDict_SetInt(kwargs, "port", rd_kafka_Node_port(c_node)); #16 6.183 | ^~~~~~~~~~~~~~~~~~ #16 6.183 | rd_kafka_queue_poll #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1466:46: warning: implicit declaration of function 'rd_kafka_Node_host'; did you mean 'rd_kafka_conf_get'? [-Wimplicit-function-declaration] #16 6.183 1466 | cfl_PyDict_SetString(kwargs, "host", rd_kafka_Node_host(c_node)); #16 6.183 | ^~~~~~~~~~~~~~~~~~ #16 6.183 | rd_kafka_conf_get #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:1466:46: warning: passing argument 3 of 'cfl_PyDict_SetString' makes pointer from integer without a cast [-Wint-conversion] #16 6.183 1466 | cfl_PyDict_SetString(kwargs, "host", rd_kafka_Node_host(c_node)); #16 6.183 | ^~~~~~~~~~~~~~~~~~~~~~~~~~ #16 6.183 | | #16 6.183 | int #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.h:328:74: note: expected 'const char *' but argument is of type 'int' #16 6.183 328 | void cfl_PyDict_SetString (PyObject *dict, const char *name, const char *val); #16 6.183 | ~~~~~~~~~~~~^~~ #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c: In function 'set_sasl_credentials': #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:2673:17: warning: implicit declaration of function 'rd_kafka_sasl_set_credentials'; did you mean 'rd_kafka_conf_set_events'? [-Wimplicit-function-declaration] #16 6.183 2673 | error = rd_kafka_sasl_set_credentials(self->rk, username, password); #16 6.183 | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #16 6.183 | rd_kafka_conf_set_events #16 6.183 /tmp/pip-install-f8ntwuj3/confluent-kafka_2b91591bccf34627b47c212a83bca576/src/confluent_kafka/src/confluent_kafka.c:2673:15: warning: assignment to 'rd_kafka_error_t *' {aka 'struct rd_kafka_error_s *'} from 'int' makes pointer from integer without a cast [-Wint-conversion] #16 6.183 2673 | error = rd_kafka_sasl_set_credentials(self->rk, username, password); #16 6.183 | ^ #16 6.183 error: command 'gcc' failed with exit status 1 #16 6.183 [end of output] #16 6.183 #16 6.183 note: This error originates from a subprocess, and is likely not a problem with pip. #16 6.186 ERROR: Failed building wheel for confluent-kafka #16 6.188 Running setup.py clean for confluent-kafka #16 6.665 Failed to build confluent-kafka #16 6.666 ERROR: Could not build wheels for confluent-kafka, which is required to install pyproject.toml-based projects #16 ERROR: executor failed running [/bin/sh -c pip3 install --no-binary :all: confluent-kafka]: exit code: 1

How to reproduce

FROM python:3.8-alpine

RUN mkdir /opt/program RUN mkdir /opt/program/src

RUN ls -l /opt/program/src

COPY . /opt/program/src

RUN ls -l /opt/program/src

WORKDIR /opt/program/src

RUN apk add --no-cache alpine-sdk git py-pip py-cffi krb5

change to edge alpine branch to install librdkafka package

RUN sed -i -e 's/v3.4/edge/g' /etc/apk/repositories
&& apk upgrade --update-cache --available
&& apk add --no-cache librdkafka-dev python3-dev

RUN pip install --upgrade pip RUN pip install -r requirements.txt RUN pip3 install --no-binary :all: confluent-kafka

CMD ["python3","test.py"]

Checklist

Please provide the following information:

• [ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• [ ] Apache Kafka broker version:
• [ ] Client configuration: {...}
• [ ] Operating system: 3.8-alpine
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[pranavrth]

The error shows that librdkafka is not installed before installing confluent-kafka. You need to install that first. Though, I can see that instructions for installing librdkafka. If it is not installed on default location, you need to point to the library with environment variables as described here.

[sivankumar86]

Have you tried this ?

FROM python:3.9.11-alpine3.15 

# upgrade pip to the latest version
RUN apk --no-cache upgrade \
    && pip install --upgrade pip \
    && apk --no-cache add tzdata build-base gcc libc-dev g++ make git bash

RUN git clone https://github.com/edenhill/librdkafka.git && cd librdkafka \
    && git checkout tags/v2.2.0 && ./configure --clean \
    && ./configure --prefix /usr/local \
    && make && make install
RUN pip install confluent-kafka==2.2.0

[lcarcamo1526]

@sngit1234 The following image already has all the dependencies.

[GuillaumeCisco]

@sngit1234 The following image already has all the dependencies.

Is it possible to know how this image has been built? I'd like to do the same for python 3.10, currently, it works for python 3.11

[lcarcamo1526]

@sngit1234 The following image already has all the dependencies.

Is it possible to know how this image has been built? I'd like to do the same for python 3.10, currently, it works for python 3.11

The following repo has the Dockerfile, hope it works!

[GuillaumeCisco]

I ended up using not an alpine but a slim-bookworm, because I also install pandas, and it took forever on an alpine... Curiously, not using an alpine made the way to install librdkafka unecessary...

Thanks for the answer by the way, I really appreciate.

[nhaq-confluent]

@sngit1234 based on the discussions above, is your issue resolved? If so I can mark this closed

[sngit1234]

@sngit1234 based on the discussions above, is your issue resolved? If so I can mark this closed

Yes, this issue can be closed.

[pranavrth]

Closing.