confluent-kafka-python icon indicating copy to clipboard operation
confluent-kafka-python copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

Errors while installing latest version

Open orlagetty opened this issue 1 year ago • 15 comments

Description

Errors while installing latest version (2.5.0).

/mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka/src/confluent_kafka.h:18:10: fatal error: Python.h: No such file or directory
     #include <Python.h>

confluent-kafka version: 2.5.0 OS: Amazon Linux 2023

How to reproduce

Executing command: sudo python3 -m pip install confluent-kafka

Error output

Collecting confluent-kafka
  Using cached confluent-kafka-2.5.0.tar.gz (136 kB)
Using legacy 'setup.py install' for confluent-kafka, since package 'wheel' is not installed.
Installing collected packages: confluent-kafka
    Running setup.py install for confluent-kafka ... error
    ERROR: Command errored out with exit status 1:
     command: /bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/tmp/pip-install-blxssofn/confluent-kafka/setup.py'"'"'; __file__='"'"'/mnt/tmp/pip-install-blxssofn/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /mnt/tmp/pip-record-ntd03130/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/confluent-kafka
         cwd: /mnt/tmp/pip-install-blxssofn/confluent-kafka/
    Complete output (67 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/confluent_kafka
    copying src/confluent_kafka/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka
    copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-x86_64-3.7/confluent_kafka
    copying src/confluent_kafka/error.py -> build/lib.linux-x86_64-3.7/confluent_kafka
    copying src/confluent_kafka/serializing_producer.py -> build/lib.linux-x86_64-3.7/confluent_kafka
    creating build/lib.linux-x86_64-3.7/confluent_kafka/_model
    copying src/confluent_kafka/_model/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/_model
    creating build/lib.linux-x86_64-3.7/confluent_kafka/_util
    copying src/confluent_kafka/_util/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/_util
    copying src/confluent_kafka/_util/conversion_util.py -> build/lib.linux-x86_64-3.7/confluent_kafka/_util
    copying src/confluent_kafka/_util/validation_util.py -> build/lib.linux-x86_64-3.7/confluent_kafka/_util
    creating build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_acl.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_cluster.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_config.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_group.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_listoffsets.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_metadata.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_records.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_resource.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_scram.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    copying src/confluent_kafka/admin/_topic.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
    creating build/lib.linux-x86_64-3.7/confluent_kafka/avro
    copying src/confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
    copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
    copying src/confluent_kafka/avro/error.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
    copying src/confluent_kafka/avro/load.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
    creating build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
    creating build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/avro.py -> build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/error.py -> build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.linux-x86_64-3.7/confluent_kafka/schema_registry
    creating build/lib.linux-x86_64-3.7/confluent_kafka/serialization
    copying src/confluent_kafka/serialization/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/serialization
    creating build/lib.linux-x86_64-3.7/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro/serializer
    running build_ext
    building 'confluent_kafka.cimpl' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/mnt
    creating build/temp.linux-x86_64-3.7/mnt/tmp
    creating build/temp.linux-x86_64-3.7/mnt/tmp/pip-install-blxssofn
    creating build/temp.linux-x86_64-3.7/mnt/tmp/pip-install-blxssofn/confluent-kafka
    creating build/temp.linux-x86_64-3.7/mnt/tmp/pip-install-blxssofn/confluent-kafka/src
    creating build/temp.linux-x86_64-3.7/mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka
    creating build/temp.linux-x86_64-3.7/mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka/src
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
-fPIC -I/usr/include/python3.7m -c /mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.7/mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka/src/confluent_kafka.o
    In file included from /mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka/src/confluent_kafka.c:17:0:
    /mnt/tmp/pip-install-blxssofn/confluent-kafka/src/confluent_kafka/src/confluent_kafka.h:18:10: fatal error: Python.h: No such file or directory
     #include <Python.h>
              ^~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/tmp/pip-install-blxssofn/confluent-kafka/setup.py'"'"'; __file__='"'"'/mnt/tmp/pip-install-blxssofn/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /mnt/tmp/pip-record-ntd03130/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/confluent-kafka Check the logs for full command output.

Checklist

Please provide the following information:

  • [ x] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
  • [ ] Apache Kafka broker version:
  • [ ] Client configuration: {...}
  • [ x] Operating system:
  • [x ] Provide client logs (with 'debug': '..' as necessary)
  • [ ] Provide broker log excerpts
  • [ ] Critical issue

orlagetty avatar Jul 17 '24 10:07 orlagetty

Installing the following new libs fixed it for me image

TheVexy avatar Jul 17 '24 12:07 TheVexy

This is not able to get the wheel and installing from the source which is not expected for most of the cases.

Which python and pip version are you using?

pranavrth avatar Jul 18 '24 06:07 pranavrth

Python 3.7

orlagetty avatar Jul 18 '24 07:07 orlagetty

Please upgrade pip first and try installing conflurnt-kafka again.

python -m pip install --upgrade pip

pranavrth avatar Jul 22 '24 09:07 pranavrth

We have a bootstrap script that always runs when deploying the scripts and part of it is sudo python3 -m pip install --upgrade pip

So I don't think it's that.

orlagetty avatar Jul 22 '24 10:07 orlagetty

From logs I could see one more thing that wheel package is not installed. Please install wheel package first and then try to install confluent-kafka. Generally this is the problem with older pip versions. Just for understanding, please provide exact version of python and pip as well using python3 -V and python3 -m pip -V respectively.

Using legacy 'setup.py install' for confluent-kafka, since package 'wheel' is not installed.

pranavrth avatar Jul 22 '24 14:07 pranavrth

image

orlagetty avatar Jul 22 '24 16:07 orlagetty

Can you please install wheel first? I am not facing this issue with amazonlinux or 3.7.16 in my local.

python3 -m pip install wheel python3 -m pip install confluent-kafka

If this doesn't work, please provide output of python3 -m pip install -vvv confluent-kafka.

pranavrth avatar Jul 23 '24 07:07 pranavrth

I can somewhat confirm this bug.

On an up-to-date Ubuntu 20.04 system with pip installed via apt (which leads to pip version 20.0.2) installing confluent-kafka fails with an error on wheels:

grafik

First Upgrading pip via itself, and then trying to install the newest version of confluent-kafka, resolves the issue:

grafik

As another hint: Installing version 2.4.0 of confluent-kafka in the original environment with pip version 20.0.2 works fine:

grafik

So, in conclusion some aspects of version 2.5.0 of confluent-kafka seem to break installation on older pip versions.

airballking avatar Jul 23 '24 12:07 airballking

We have moved building manylinux wheels to manylinux_2_28 in this release. This might have caused something with old pip. It would be really helpful if you can provide verbose logs while installing using python3 -m pip install -vvv confluent-kafka. I am not able to replicate this issue in my local.

For the meantime, please fix the issue by upgrading the pip version to latest.

pranavrth avatar Jul 26 '24 10:07 pranavrth

@pranavrth my suggestion to reproducing the issue on any Linux machine with docker:

<your-user>@<your-machine>:~$ sudo docker image pull ubuntu:20.04
<your-user>@<your-machine>:~$ sudo docker run -i -t ubuntu:20.04 /bin/bash
root@<your-container>:/# apt update && apt -y upgrade
root@<your-container>:/# apt install python3-pip -y
root@<your-container>:/# python3 -m pip install confluent-kafka

When I run this with the requested verbose logs, I receive the following:

Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-tgkztrlm
Created temporary directory: /tmp/pip-req-tracker-8ttwmeni
Initialized build tracking at /tmp/pip-req-tracker-8ttwmeni
Created build tracker: /tmp/pip-req-tracker-8ttwmeni
Entered build tracker: /tmp/pip-req-tracker-8ttwmeni
Created temporary directory: /tmp/pip-install-4yx5yvi2
1 location(s) to search for versions of confluent-kafka:
* https://pypi.org/simple/confluent-kafka/
Fetching project page and analyzing links: https://pypi.org/simple/confluent-kafka/
Getting page https://pypi.org/simple/confluent-kafka/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/confluent-kafka/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/confluent-kafka/ HTTP/1.1" 200 67557
Updating cache with response from "https://pypi.org/simple/confluent-kafka/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/b5/0a/2c4053a5ad38b71e74f606c7987513818f009e344cbc1cdadda723f0d251/confluent-kafka-0.9.1.1.tar.gz#sha256=339a3f58fe19a32affdae76bfc8a4dc67d9e30aa4ff41e69bc8a1d8420e55145 (from https://pypi.org/simple/confluent-kafka/), version: 0.9.1.1
  Found link https://files.pythonhosted.org/packages/2b/48/63278a8b857805f5459caa06f1bc881c5022e2c1c9e9e9751310d986b73e/confluent-kafka-0.9.1.2.tar.gz#sha256=1b75613c9a5f35f02b8205fe248b956c83a7654ba179ef48d3306c99d37179ed (from https://pypi.org/simple/confluent-kafka/), version: 0.9.1.2
  Found link https://files.pythonhosted.org/packages/92/23/78acb04c6660f32fbb8f789252e202898c0b3e0cea2ec4ac0913b7970188/confluent-kafka-0.9.2.tar.gz#sha256=cb366b2e66e880bf458218173f1c4062d7f7c7ae871bd5571f2c166b2f392720 (from https://pypi.org/simple/confluent-kafka/), version: 0.9.2
  Found link https://files.pythonhosted.org/packages/bb/aa/4183a3ea74fe75a05a76a4ee97edd897f03a2bd2e25c76ff8c437717fb24/confluent-kafka-0.9.4.tar.gz#sha256=931bcc3512245ae003c86aa860fb62bbfb4036763a3c2908c50150e412bf0aed (from https://pypi.org/simple/confluent-kafka/), version: 0.9.4
  Found link https://files.pythonhosted.org/packages/96/36/516a2b7f592376968296d10de10a20f1ce411e5ff24a86b1a23d5a5f4042/confluent-kafka-0.11.0.tar.gz#sha256=4c34bfe8f823ee3777d93820ec6578365d2bde3cd1302cbd0e44c86b68643667 (from https://pypi.org/simple/confluent-kafka/), version: 0.11.0
  Found link https://files.pythonhosted.org/packages/21/84/68b1dcae3d849d2ff07ea08ba1913c6699105ac64c01c50da69937111812/confluent-kafka-0.11.4.tar.gz#sha256=8cf480199685127c9692b0bf1e15eac82e71ae34b7967a016ab31a318741abb1 (from https://pypi.org/simple/confluent-kafka/), version: 0.11.4
  Skipping link: none of the wheel's tags match: cp27-cp27m-macosx_10_6_intel: https://files.pythonhosted.org/packages/b6/6a/3919a5bacc000746384ab6fb05a23021a4769cd14409a2faa475adcc3c83/confluent_kafka-0.11.4-cp27-cp27m-macosx_10_6_intel.whl#sha256=774fc81908cbda66f415a5b0a0c6140a9c9d00186248c2b4c7bb497f856b2033 (from https://pypi.org/simple/confluent-kafka/)


... <a long list of skipped links that like the ones above and below here> ...


  Skipping link: none of the wheel's tags match: cp39-cp39-win_amd64: https://files.pythonhosted.org/packages/e1/72/8ca0813b7055af7f471540ffb5b19091d7a17d9df7c7e6a9bd81c41f2152/confluent_kafka-2.5.0-cp39-cp39-win_amd64.whl#sha256=4bda1b5fa87cb993bcd964d271a76cc11cafa2455de02ab5eff6efd9e688d55e (from https://pypi.org/simple/confluent-kafka/)
Given no hashes to check 48 links for project 'confluent-kafka': discarding no candidates
Using version 2.5.0 (newest of versions: 0.9.1.1, 0.9.1.2, 0.9.2, 0.9.4, 0.11.0, 0.11.4, 0.11.5, 0.11.6, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.6.0, 1.6.1, 1.7.0, 1.8.2, 1.9.0, 1.9.2, 2.0.2, 2.1.0, 2.1.1, 2.2.0, 2.3.0, 2.4.0, 2.5.0)
Collecting confluent-kafka
  Created temporary directory: /tmp/pip-unpack-ouisr477
  Looking up "https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz HTTP/1.1" 200 136857
  Downloading confluent-kafka-2.5.0.tar.gz (136 kB)
     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f| 133 kB 3.9 MB/s eta 0:00:01  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz"
  Caching due to etag
     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 136 kB 3.9 MB/s 
  Added confluent-kafka from https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz#sha256=551cabaade717bb56ec13eb860ce439bedbcf1c97f4a4aa26957572ed1bfa74f to build tracker '/tmp/pip-req-tracker-8ttwmeni'
    Running setup.py (path:/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py) egg_info for package confluent-kafka
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info
    writing /tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/requires.txt
    writing top-level names to /tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/top_level.txt
    writing manifest file '/tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'test-requirements.txt'
    writing manifest file '/tmp/pip-install-4yx5yvi2/confluent-kafka/pip-egg-info/confluent_kafka.egg-info/SOURCES.txt'
  Source in /tmp/pip-install-4yx5yvi2/confluent-kafka has version 2.5.0, which satisfies requirement confluent-kafka from https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz#sha256=551cabaade717bb56ec13eb860ce439bedbcf1c97f4a4aa26957572ed1bfa74f
  Removed confluent-kafka from https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz#sha256=551cabaade717bb56ec13eb860ce439bedbcf1c97f4a4aa26957572ed1bfa74f from build tracker '/tmp/pip-req-tracker-8ttwmeni'
Building wheels for collected packages: confluent-kafka
  Created temporary directory: /tmp/pip-wheel-3u0mm7bw
  Building wheel for confluent-kafka (setup.py) ...   Destination directory: /tmp/pip-wheel-3u0mm7bw
  Running command /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-3u0mm7bw
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/confluent_kafka
  copying src/confluent_kafka/error.py -> build/lib.linux-x86_64-3.8/confluent_kafka
  copying src/confluent_kafka/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka
  copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-x86_64-3.8/confluent_kafka
  copying src/confluent_kafka/serializing_producer.py -> build/lib.linux-x86_64-3.8/confluent_kafka
  creating build/lib.linux-x86_64-3.8/confluent_kafka/serialization
  copying src/confluent_kafka/serialization/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/serialization
  creating build/lib.linux-x86_64-3.8/confluent_kafka/_model
  copying src/confluent_kafka/_model/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_model
  creating build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_group.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_records.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_config.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_cluster.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_listoffsets.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_topic.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_metadata.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_resource.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_acl.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  copying src/confluent_kafka/admin/_scram.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
  creating build/lib.linux-x86_64-3.8/confluent_kafka/avro
  copying src/confluent_kafka/avro/error.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
  copying src/confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
  copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
  copying src/confluent_kafka/avro/load.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
  creating build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
  creating build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/error.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/avro.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
  creating build/lib.linux-x86_64-3.8/confluent_kafka/_util
  copying src/confluent_kafka/_util/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_util
  copying src/confluent_kafka/_util/conversion_util.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_util
  copying src/confluent_kafka/_util/validation_util.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_util
  creating build/lib.linux-x86_64-3.8/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/tmp
  creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2
  creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka
  creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src
  creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka
  creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c /tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.o
  In file included from /tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.c:17:
  /tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
     23 | #include <librdkafka/rdkafka.h>
        |          ^~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
error
  ERROR: Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
  Running command /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
  running clean
  removing 'build/temp.linux-x86_64-3.8' (and everything under it)
  removing 'build/lib.linux-x86_64-3.8' (and everything under it)
  'build/bdist.linux-x86_64' does not exist -- can't clean it
  'build/scripts-3.8' does not exist -- can't clean it
  removing 'build'
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
  Created temporary directory: /tmp/pip-record-amssofoy
    Running command /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-amssofoy/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/confluent-kafka
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/confluent_kafka
    copying src/confluent_kafka/error.py -> build/lib.linux-x86_64-3.8/confluent_kafka
    copying src/confluent_kafka/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka
    copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-x86_64-3.8/confluent_kafka
    copying src/confluent_kafka/serializing_producer.py -> build/lib.linux-x86_64-3.8/confluent_kafka
    creating build/lib.linux-x86_64-3.8/confluent_kafka/serialization
    copying src/confluent_kafka/serialization/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/serialization
    creating build/lib.linux-x86_64-3.8/confluent_kafka/_model
    copying src/confluent_kafka/_model/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_model
    creating build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_group.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_records.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_config.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_cluster.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_listoffsets.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_topic.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_metadata.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_resource.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_acl.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    copying src/confluent_kafka/admin/_scram.py -> build/lib.linux-x86_64-3.8/confluent_kafka/admin
    creating build/lib.linux-x86_64-3.8/confluent_kafka/avro
    copying src/confluent_kafka/avro/error.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
    copying src/confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
    copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
    copying src/confluent_kafka/avro/load.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro
    creating build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-3.8/confluent_kafka/kafkatest
    creating build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/error.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/avro.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.linux-x86_64-3.8/confluent_kafka/schema_registry
    creating build/lib.linux-x86_64-3.8/confluent_kafka/_util
    copying src/confluent_kafka/_util/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_util
    copying src/confluent_kafka/_util/conversion_util.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_util
    copying src/confluent_kafka/_util/validation_util.py -> build/lib.linux-x86_64-3.8/confluent_kafka/_util
    creating build/lib.linux-x86_64-3.8/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-3.8/confluent_kafka/avro/serializer
    running build_ext
    building 'confluent_kafka.cimpl' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/tmp
    creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2
    creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka
    creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src
    creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka
    creating build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c /tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.8/tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.o
    In file included from /tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.c:17:
    /tmp/pip-install-4yx5yvi2/confluent-kafka/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
       23 | #include <librdkafka/rdkafka.h>
          |          ^~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    Running setup.py install for confluent-kafka ... error
Cleaning up...
  Removing source in /tmp/pip-install-4yx5yvi2/confluent-kafka
Removed build tracker: '/tmp/pip-req-tracker-8ttwmeni'
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-amssofoy/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/confluent-kafka Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 421, in run
    installed = install_given_reqs(
  File "/usr/lib/python3/dist-packages/pip/_internal/req/__init__.py", line 67, in install_given_reqs
    requirement.install(
  File "/usr/lib/python3/dist-packages/pip/_internal/req/req_install.py", line 820, in install
    install_legacy(
  File "/usr/lib/python3/dist-packages/pip/_internal/operations/install/legacy.py", line 70, in install
    runner(
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4yx5yvi2/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-amssofoy/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/confluent-kafka Check the logs for full command output.

airballking avatar Aug 07 '24 13:08 airballking

I am also facing similar issue. Recreating the issue:

docker run -v ${pwd}:/var/task "amazon/aws-sam-cli-build-image-python3.9" /bin/sh -c "
    pip install --upgrade pip &&
    python -m pip install -vvv confluent-kafka; exit"

Output as requested by @pranavrth

Requirement already satisfied: pip in /var/lang/lib/python3.9/site-packages (23.0.1)
Collecting pip
  Downloading pip-24.2-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 14.8 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-24.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Using pip 24.2 from /var/lang/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Disabling truststore because Python version isn't 3.10+
Created temporary directory: /tmp/pip-build-tracker-vwcy5rg0
Initialized build tracking at /tmp/pip-build-tracker-vwcy5rg0
Created build tracker: /tmp/pip-build-tracker-vwcy5rg0
Entered build tracker: /tmp/pip-build-tracker-vwcy5rg0
Created temporary directory: /tmp/pip-install-7g7fjlcr
Created temporary directory: /tmp/pip-ephem-wheel-cache-f4j1e54f
1 location(s) to search for versions of confluent-kafka:
* https://pypi.org/simple/confluent-kafka/
Fetching project page and analyzing links: https://pypi.org/simple/confluent-kafka/
Getting page https://pypi.org/simple/confluent-kafka/
Found index url https://pypi.org/simple/
Looking up "https://pypi.org/simple/confluent-kafka/" in the cache
Request header has "max_age" as 0, cache bypassed
No cache entry available
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/confluent-kafka/ HTTP/1.1" 200 77169
Updating cache with response from "https://pypi.org/simple/confluent-kafka/"
etag object cached for 1209600 seconds
Caching due to etag
Fetched page https://pypi.org/simple/confluent-kafka/ as application/vnd.pypi.simple.v1+json
  Found link https://files.pythonhosted.org/packages/b5/0a/2c4053a5ad38b71e74f606c7987513818f009e344cbc1cdadda723f0d251/confluent-kafka-0.9.1.1.tar.gz (from https://pypi.org/simple/confluent-kafka/), version: 0.9.1.1
  Found link https://files.pythonhosted.org/packages/2b/48/63278a8b857805f5459caa06f1bc881c5022e2c1c9e9e9751310d986b73e/confluent-kafka-0.9.1.2.tar.gz (from https://pypi.org/simple/confluent-kafka/), version: 0.9.1.2
  Found link https://files.pythonhosted.org/packages/92/23/78acb04c6660f32fbb8f789252e202898c0b3e0cea2ec4ac0913b7970188/confluent-kafka-0.9.2.tar.gz (from https://pypi.org/simple/confluent-kafka/), version: 0.9.2
  Found link https://files.pythonhosted.org/packages/bb/aa/4183a3ea74fe75a05a76a4ee97edd897f03a2bd2e25c76ff8c437717fb24/confluent-kafka-0.9.4.tar.gz (from https://pypi.org/simple/confluent-kafka/), version: 0.9.4
  Found link https://files.pythonhosted.org/packages/96/36/516a2b7f592376968296d10de10a20f1ce411e5ff24a86b1a23d5a5f4042/confluent-kafka-0.11.0.tar.gz (from https://pypi.org/simple/confluent-kafka/), version: 0.11.0
  Found link https://files.pythonhosted.org/packages/21/84/68b1dcae3d849d2ff07ea08ba1913c6699105ac64c01c50da69937111812/confluent-kafka-0.11.4.tar.gz (from https://pypi.org/simple/confluent-kafka/), version: 0.11.4
  ............
  ............
  ............
Skipping link: not a file: https://pypi.org/simple/confluent-kafka/
Given no hashes to check 42 links for project 'confluent-kafka': discarding no candidates
Collecting confluent-kafka
  Created temporary directory: /tmp/pip-unpack-qsm80orp
  Looking up "https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz" in the cache
  No cache entry available
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz HTTP/1.1" 200 136857
  Downloading confluent-kafka-2.5.0.tar.gz (136 kB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz"
  etag object cached for 1209600 seconds
  Caching due to etag
  Added confluent-kafka from https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz to build tracker '/tmp/pip-build-tracker-vwcy5rg0'
  Running setup.py (path:/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/setup.py) egg_info for package confluent-kafka
  Created temporary directory: /tmp/pip-pip-egg-info-xd486t19
  Preparing metadata (setup.py): started
  Running command python setup.py egg_info
  running egg_info
  creating /tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info
  writing /tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/requires.txt
  writing top-level names to /tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'test-requirements.txt'
  adding license file 'LICENSE.txt'
  writing manifest file '/tmp/pip-pip-egg-info-xd486t19/confluent_kafka.egg-info/SOURCES.txt'
  Preparing metadata (setup.py): finished with status 'done'
  Source in /tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b has version 2.5.0, which satisfies requirement confluent-kafka from https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz
  Removed confluent-kafka from https://files.pythonhosted.org/packages/94/51/2e8f4048edbcde0346047dcd5228985ddcd0ef4d518f878321c07cc1b39c/confluent-kafka-2.5.0.tar.gz from build tracker '/tmp/pip-build-tracker-vwcy5rg0'
Created temporary directory: /tmp/pip-unpack-z2g2vjx1
Building wheels for collected packages: confluent-kafka
  Created temporary directory: /tmp/pip-wheel-rb_60q97
  Building wheel for confluent-kafka (setup.py): started
  Destination directory: /tmp/pip-wheel-rb_60q97
  Running command python setup.py bdist_wheel
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/confluent_kafka
  copying src/confluent_kafka/error.py -> build/lib.linux-x86_64-3.9/confluent_kafka
  copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-x86_64-3.9/confluent_kafka
  ...............
  ...............
  ...............
  copying src/confluent_kafka/serialization/__init__.py -> build/lib.linux-x86_64-3.9/confluent_kafka/serialization
  creating build/lib.linux-x86_64-3.9/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-3.9/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-3.9/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.linux-x86_64-3.9
  creating build/temp.linux-x86_64-3.9/tmp
  creating build/temp.linux-x86_64-3.9/tmp/pip-install-7g7fjlcr
  creating build/temp.linux-x86_64-3.9/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b
  creating build/temp.linux-x86_64-3.9/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src
  creating build/temp.linux-x86_64-3.9/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src/confluent_kafka
  creating build/temp.linux-x86_64-3.9/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src/confluent_kafka/src
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.9 -c /tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src/confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-3.9/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src/confluent_kafka/src/Admin.o       
  In file included from /tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src/confluent_kafka/src/Admin.c:17:0:
  /tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
   #include <librdkafka/rdkafka.h>
            ^~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /var/lang/bin/python -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize

  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)

  __file__ = %r
  sys.argv[0] = __file__

  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"

  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /tmp/pip-wheel-rb_60q97
  cwd: /tmp/pip-install-7g7fjlcr/confluent-kafka_9a3c7c6a7dbb4764b3a4d26658ee059b/
  Building wheel for confluent-kafka (setup.py): finished with status 'error'
  ERROR: Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
  Running command python setup.py clean
  running clean
  removing 'build/temp.linux-x86_64-3.9' (and everything under it)
  removing 'build/lib.linux-x86_64-3.9' (and everything under it)
  'build/bdist.linux-x86_64' does not exist -- can't clean it
  'build/scripts-3.9' does not exist -- can't clean it
  removing 'build'
Failed to build confluent-kafka
Disabling truststore because Python version isn't 3.10+
Remote version of pip: 24.2
Local version of pip:  24.2
Was pip installed by pip? True
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (confluent-kafka)
Exception information:
Traceback (most recent call last):
  File "/var/lang/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
  File "/var/lang/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
  File "/var/lang/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/var/lang/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 434, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: ERROR: Failed to build installable wheels for some pyproject.toml based projects (confluent-kafka)
Removed build tracker: '/tmp/pip-build-tracker-vwcy5rg0'

wholesale-error avatar Aug 09 '24 12:08 wholesale-error

I have understood the issue here. We recently changed the manylinux wheel generation to use manylinux_2_28 distro and hence the tags that are present in the newer version is includes manylinux_2_28. Some of the older version of python and pip combination doesn't include this tag and hence the installation falls back to the installation from source distribution which is not advised during the normal installation as we provide the wheels. I will check on how to resolve this issue for the upcoming releases. Maybe using some older manylinux distribution should work. Will check this.

To resolve the above issues I think there are a few options here:

  • Install latest version of Python and pip that you can use.
  • Install pip using with the official way using get-pip.py

If it is still using source distribution and failing then you can troubleshoot using the following guide:

Issue 1 getting error on #include <Python.h> like the original logs of this post provided by @orlagetty.

     #include <Python.h>
              ^~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1

Cause - This error is caused because you don't have devel package installed for python. Solution Install python3-devel/python3-dev package before trying to reinstall. Refer: this answer for more help.

Issue 2 Getting error on #include <librdkafka/rdkafka.h>

   #include <librdkafka/rdkafka.h>
            ^~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  error: subprocess-exited-with-error

Cause Installation from source distribution requires librdkafka to be installed before installing confluent_kafka. Solution Installing librdkafka should fix this issue. Please refer installation doc for librdkafka

Please let me know if you have any concern.

pranavrth avatar Aug 14 '24 10:08 pranavrth

Hello!

I installed librdkafka in my machine and then restart the opened terminal and it works.

Python 3.13.0 pip 24.2 confluent-kafka==2.6.0

solucionesvaltech avatar Nov 18 '24 15:11 solucionesvaltech

To add to this thread, I'm facing installation errors on windows. I get the same error as above (librd) is missing. I'm running python 3.13.0 and pip 24.3.1. Wheels is installed. Anyone else facing this issue?

AndreMCCarvalho avatar Nov 27 '24 15:11 AndreMCCarvalho

That's very surprising that you see this on 3.13 with pip 24.3.1 -- I know this is an old issue but can you @AndreMCCarvalho give more details on the error and machine setup if you still are having problems? I'm going to close the issue as the original problem of old pip/python version is resolved by upgrading, but happy to engage if there's something further problematic with certain configurations.

MSeal avatar Jul 23 '25 22:07 MSeal