Distribute Python `sdist` through PyPI

[diegoferigo]

I'm very glad to see that Python support https://github.com/artivis/manif/issues/187 has been introduced recently in https://github.com/artivis/manif/pull/201 (previous https://github.com/artivis/manif/pull/179).

I'm opening this issue to discuss / propose the possibility to support distributing the project through PyPI. For those that are not very familiar with the process, there are few incremental steps that could be done, each of them extending more and more the support. To begin with, I'd point interested readers to https://packaging.python.org/tutorials/packaging-projects/ and Python Packaging User Guide.

The main background to understand this issue is grasping the differences between a source distribution (aka sdist) and a binary distribution (aka wheel or bdist_wheel).

As first step, it would be nice distributing manifpy in the form of a source distribution. If the project would introduce the support of pypa/setuptools_scm, the source distribution would be no more no less than a compressed version of the git repository. When users execute pip install manif, it would download the sdist and run setup.py. The process is quite straightforward.

Of course, it is clear to see that in this case, and as documented in https://github.com/artivis/manif/pull/201, setup.py expects to find some dependency in the system (eigen, cmake, pybind). Here come PEP517 and PEP518 to help us. In few words, during the pip install manif process, they create an isolated Python environment (e.g. in /tmp) and install the build dependencies before building the sdist downloaded from PyPI. In this way, cmake and pybind can be automatically installed, and users do not have to manually install them in their system. AFAIK, eigen is not in PyPI, therefore for the moment it needs to be found in the system. Alternatively, a simple FetchContent logic could be added to the repo.

A very useful tool to implement without much effort all what I explained above is diegoferigo/cmake-build-extension, that acts as a bridge between CMake-based projects and setup.py. Disclaimer: I'm the author of the package. Beyond the official documentation, a MWE that also supports PEP600 wheels and implements an automatic CI/CD pipeline is diegoferigo/gazebo-yarp-synchronizer.

Right now manif only installs the shared library generated by pybind11 in the site-package folder. cmake-build-extension enables creating a real Python package (with an __init__.py) and also installing the exported CMake project (headers, libraries, cmake files), so that other packages could import and consume it. The same approach is implemented in the pybind11 package itself, beyond other ones.

@artivis I liked to ask if you'd be interested in implementing something similar in manif. If needed, I can provide support and guidance.

Further information can be found in the downstream https://github.com/dic-iit/bipedal-locomotion-framework/pull/296.

[artivis]

Hi @diegoferigo,

Thanks for the detailed break down of releasing in PyPi. To immediately answer this ticket, yes a PyPI release is on my todo list and I have briefly looked into it at the time of developing #201 (mainly this tutorial). However there are a couple open PRs that I'd like to land before issuing a new tag and releasing to PyPI. As this is a first for me, I'll need time to get my head around the whole process and its implications. I may very well come back to you for guidance indeed. I will also want to automatize the release from CI so that I don't have to go back to all those links each and every time. To that end your cmake-build-extension project may come handy, I'll have to see.

so that other packages could import and consume it.

You mean other C++ project right? I had not think of that but I guess it makes sense.

[diegoferigo]

Thanks for the detailed break down of releasing in PyPi. To immediately answer this ticket, yes a PyPI release is on my todo list and I have briefly looked into it at the time of developing #201 (mainly this tutorial). However there are a couple open PRs that I'd like to land before issuing a new tag and releasing to PyPI. As this is a first for me, I'll need time to get my head around the whole process and its implications.

Great! No problem, do as many attempts as you prefer. One advice I can give you is to always use the testing index test.pypi.org instead of the official one during your early experiment. In fact, if you create a new package in the official index and push by mistake a wrong version, you are not allowed to override it. You can delete it, but then you have lost the usage of that version if needed in the future. (I personally did this mistake in the beginning). As the blog post you linked reports, manually uploading with twine is the most simple start, that could be automatize with CI in the future.

I may very well come back to you for guidance indeed. I will also want to automatize the release from CI so that I don't have to go back to all those links each and every time. To that end your cmake-build-extension project may come handy, I'll have to see.

Feel free to ping me at any point :wink:

so that other packages could import and consume it.

You mean other C++ project right? I had not think of that but I guess it makes sense.

Yes, it took a while also to me to figure that out. PyPI, in theory, could be used also to distribute non-python resources. If wheels are distributed properly (and I mean that C++ projects include the entire install tree, including headers + libraries + exported targets), by pointing CMAKE_INSTALL_PREFIX to the site-package folder, other packages can import the targets and consume them. This is what cmake-build-extension does under the hood with its cmake_depends_on option. To make a practical example, bipedal-locomotion-framework needs to find manif installed, and this approach comes handy because during the process used by pip that I explained in my previous comment (creating an isolated environment), manif targets can be found in the isolated environment and blf can be built without installing manif system-wide.

Another project I found using a similar approach is CasADI. You can install it and inspect its folder inside site-packages.

The case of manif and pybind11 is easier because they are header-only libs, the compiler does not play a big role. However, I want to point out again that projects distributing in this way their binaries must use the new PEP600 standard. Otherwise, manylinux1 and manylinux2014 will use a too old gcc version that does not have C++11 ABI and bad things can happen (https://github.com/robotology/idyntree/issues/776).

[diegoferigo]

For future reference, I've found another explanation of the inclusion of headers / libs / targets etc inside the wheel in pybind's documentation.