Add bullet point for Windows as well

[slominskir]

Windows is conspicuously missing from list of OSes with build/install notes

[slominskir]

I noticed there are a lot of projects that wrap librdkafka - it might be better to simply replace the build/install librdkafka section of all these README.md files to simply link back to https://github.com/edenhill/librdkafka#installation to avoid duplicate notes.

Likely incomplete list of projects using librdkafka: https://github.com/topics/librdkafka

The individual project install/build sections could have language specific stuff though still. I haven't got the Python lib fully working yet on Windows so I'm not even sure what all is required (it works great on Linux in Docker containers - but trying to develop / contribute code via an IDE running on Windows is proving tricky - Developer notes are mum about things like python -m pip install --editable . for example, which I assume is needed, but haven't quite gotten to work yet).

[edenhill]

We provide prebuilt binary wheels for Windows that contain librdkafka and its dependencies, and for people that want to build their own they are in for a rough ride due to the random nature of windows/py toolchains. That is to say; I'm not sure there's much value in adding the librdkafka.redist link to the README, it may just confuse users thinking they need to install librdkafka separately.

[slominskir]

I guess Windows wheels only works with Python 3.8 at the moment? Or maybe I need to re-install a few things? I'm on Python 3.9. Working through the Developer notes for example attempting to run the Unit tests doesn't work fully - I can run individual tests (that likely don't actually require librdkafka - I assume integration tests do that), but run all doesn't work (least of which because syntax is invalid on Windows - pytest -s -v tests/test_*.py assumes UNIX shell?). I also haven't figured out how to make a change in the source and have the unit test detect it - sys.path always using site-packages 1.6.1 install of lib. If I modify PYTHONPATH then the changes in the source presumably are picked up but whole thing explodes due to not finding C libs.

[edenhill]

There are Py 3.9 windows wheels now (1.7.0)

[slominskir]

Might want to update the project contributing guide for Windows users to explicitly indicate they must use Python 3.6, 3.7, 3.8, or 3.9 but not 3.10 wheels as it don't exist yet.

Also, I never have figured out how to use develop mode (pip install -e) with this project as it appears to uninstall the wheel version replacing it with the source version (good for developing), but I think that means the pre-compiled objects for librdkafka are now nowhere to be found leading to the dreaded ImportError: DLL load failed while importing cimpl: The specified module could not be found. when you run the unit tests. Unit tests work (first one anyways) with wheel installed. Is there a way to install binary librdkafka from wheel into local source dir cloned from github instead of into site-packages? I'm not familiar with interface between Python and C code.

[edenhill]

@slominskir So it used to be easier, I tended to do python3 setup.py develop and it would work out of the box.. But nowadays I can't get that to work anylonger, so I typically just do python3 -m pip install . from the top-level directory. A bit slower, but makes the thing work.

[slominskir]

@edenhill Couldn't get pip install . to work for me. It just ran the legacy build (no wheel), but still unit tests didn't work. Experimenting some more it looks like the following might work though:

(.venv) pip install C:\Users\ryans\Downloads\confluent_kafka-1.8.2-cp39-cp39-win_amd64.whl -t ./src --upgrade
(.venv) pip install -e .
(.venv) pytest -s -v tests/test_Producer.py

from a venv pinning Python 3.9 and with the current directory as the confluent-kafka-python git project it actually did install the librdkafka binaries into the source (via -t flag) as the unit tests passed! Unfortunately it also modifies a few source files too (git diff) due to latest git repo having some changes over the wheel, plus a few file ending changes (likely fixable with .gitattributes ), plus .c files added. Might be better to just use -t /tmp and then manaully copy .pyd into src/confluent_kafka and entire confluent_kafka.libs dir into src. Or might be easier to just pull out my travel Macbook (UNIX) laptop and punt on Windows development with this project!

[slominskir]

A better solution is probably to update setup.py to conditionally download pre-complied extension instead of running C++ compile.

It could copy pre-compiled extension into the src directory such that develop mode works (or perhaps update PYTHONPATH / sys.path to point to pre-compiled extensions). Might be able to just rely on pip to do the download and choose the right wheel and target -t temp dir.

Discussed here: https://discuss.python.org/t/building-extension-modules-the-2020-way/5950.

Ideally we can avoid requiring contributors from having to install C++ and can avoid the error: Microsoft Visual C++ 14.0 or greater is required.

Alternatively, adding one more layer of indirection by moving the extension to a separate Python module (and separate PyPi package) and including this new module/package as a dependency would also work. Basically, we need to make it easier to contribute to this project.

[cla-assistant[bot]]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}