Problems to install with pip into Docker container

[jjmmolina]

Description

I'm using Python 3.10.2 and I have this message when I try to install into my venv environment running pip install confluent-kafka

 In file included from /private/var/folders/td/w1gcl5210h74qmvykhkskw2w0000gn/T/pip-install-o7ox0olt/confluent-kafka_a282f0a8ff244cb0bfc2ae5297a17742/src/confluent_kafka/src/Admin.c:17:
      /private/var/folders/td/w1gcl5210h74qmvykhkskw2w0000gn/T/pip-install-o7ox0olt/confluent-kafka_a282f0a8ff244cb0bfc2ae5297a17742/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
      #include <librdkafka/rdkafka.h>
               ^~~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.

How to reproduce

Run pip install confluent-kafka using Python 3.10.2

Checklist

Please provide the following information:

• [x] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): Trying to install confluent-kafka-1.9.0
• [x] Apache Kafka broker version: Using docker confluentinc/cp-kafka:7.0.1
• [x] Client configuration: {...}
• [x] Operating system: MacOs Monterey
• [x] Provide client logs (with 'debug': '..' as necessary)
• [x] Provide broker log excerpts
• [x] Critical issue

[mayteio]

+1 here, I am getting the same error when installing with PDM https://github.com/pdm-project/pdm/issues/1286:

ERRORS:
add confluent-kafka failed:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, 
in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pdm/installers/synchronizers.py", line 217, in install_candidate
    self.manager.install(can)
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pdm/installers/manager.py", line 39, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pdm/models/candidates.py", line 353, in build
    builder.build(build_dir, metadata_directory=self._metadata_dir)
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pdm/builders/wheel.py", line 28, in build
    filename = self._hook.build_wheel(out_dir, config_settings, metadata_directory)
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pep517/wrappers.py", line 208, in build_wheel
    return self._call_hook('build_wheel', {
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pep517/wrappers.py", line 322, in _call_hook
    self._subprocess_runner(
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pdm/builders/base.py", line 248, in subprocess_runner
    return log_subprocessor(cmd, cwd, extra_environ=env)
  File "/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pdm/builders/base.py", line 87, in log_subprocessor
    raise BuildError(
pdm.exceptions.BuildError: Call command ['/opt/homebrew/Cellar/pdm/2.1.0/libexec/bin/python3.10', 
'/opt/homebrew/Cellar/pdm/2.1.0/libexec/lib/python3.10/site-packages/pep517/in_process/_in_process.py', 'build_wheel', 
'/var/folders/2s/4jbl0g9n7951t49j6jkfpz700000gn/T/tmp35g7rdmi'] return non-zero status(1). Make sure the package is PEP 517-compliant, or you can
add `--no-isolation` to the command.

[povilasb]

Manually hinting where to find the librdkafka helped me:

$ brew ls librdkafka
/opt/homebrew/Cellar/librdkafka/1.9.2/include/librdkafka/ (3 files)
/opt/homebrew/Cellar/librdkafka/1.9.2/lib/librdkafka++.1.dylib
/opt/homebrew/Cellar/librdkafka/1.9.2/lib/librdkafka.1.dylib
/opt/homebrew/Cellar/librdkafka/1.9.2/lib/pkgconfig/ (4 files)
/opt/homebrew/Cellar/librdkafka/1.9.2/lib/ (5 other files)
/opt/homebrew/Cellar/librdkafka/1.9.2/share/doc/ (7 files)

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.9.2/include/
$ pip install confluent-kafka

[jjmmolina]

It's not useful for me if I want to use confluent-Kafka in a project with more teammates, deploy to no local environments, etc.

[povilasb]

Yes, would be ideal of we had more wheel packages at https://pypi.org/project/confluent-kafka/

I personally need a Python 3.10 universal MacOS build.

[mhowlett]

@jjmmolina are you on M1? if not what platform?

[mhowlett]

closing under the assumption this will be resolved with #1439

[jjmmolina]

@mhowlett yes, Apple M1 Pro. I'll check the update #1439