pypa
Missing library files at repair step & wheel configuration through pyproject
Description
First of all, thank you for providing cibw to the community!
Missing library files at repair step
I'm trying to provide a pre-built linux only wheel for OpenBabel. I'm taking inspiration from https://github.com/njzjz/openbabel-wheel which is unfortunately not maintained anymore.
The basic setup is that the repository I build only has a CMakeLists.txt that will be used by cibw/scikit-build-core to build the wheel:
CMakeLists.txt
cmake_minimum_required(VERSION 3.11)
project(ObabelPythonDistributions CXX)
# download OpenBabel to ${OBABEL_SOURCE_ROOT}
include(FetchContent)
FetchContent_Declare(openbabel
GIT_REPOSITORY https://github.com/openbabel/openbabel
GIT_TAG ${OBABEL_VERSION}
)
FetchContent_GetProperties(openbabel)
if(NOT openbabel_POPULATED)
FetchContent_Populate(openbabel)
add_subdirectory(${openbabel_SOURCE_DIR} ${openbabel_BINARY_DIR})
install(
DIRECTORY ${openbabel_SOURCE_DIR}/scripts/python/openbabel
DESTINATION ${CMAKE_INSTALL_PREFIX}/openbabel
)
endif()
My pyproject defines the version and build flags I want to use:
pyproject.toml
[tool.scikit-build]
minimum-version = "0.3.0"
[tool.scikit-build.cmake.define]
OBABEL_VERSION = "f495cc6"
CMAKE_BUILD_TYPE = "Release"
WITH_INCHI = "ON"
PYTHON_BINDINGS = "ON"
RUN_SWIG = "ON"
[tool.cibuildwheel]
archs = ["x86_64"]
build = ["cp311-manylinux2014-x86_64"]
test-command = [
'python -c "from openbabel import openbabel"',
"obabel -:C -oxyz --gen3d",
]
[tool.cibuildwheel.linux]
before-all = [
"yum install -y zlib-devel-1.2.7 eigen3-devel-3.3.7 libxml2-devel-2.9.1",
]
and I use Github actions to build everything.
The issue I have is that the wheel builds fine:
truncated install logs
2024-08-23T08:19:42.0620747Z *** Installing project into wheel...
2024-08-23T08:19:42.1426718Z -- Install configuration: "Release"
2024-08-23T08:19:42.1427901Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel
2024-08-23T08:19:42.1437400Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel/pybel.py
...
2024-08-23T08:19:42.2599648Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0
2024-08-23T08:19:42.2607617Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7
2024-08-23T08:19:42.2614782Z -- Set non-toolchain portion of runtime path of "/tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0" to "/tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/lib"
2024-08-23T08:19:42.2621652Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so
...
2024-08-23T08:19:42.4828912Z *** Making wheel...
2024-08-23T08:19:45.5498324Z *** Created aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl
2024-08-23T08:19:45.5688505Z Building wheel for aqemia-openbabel (pyproject.toml): finished with status 'done'
2024-08-23T08:19:45.5786590Z Created wheel for aqemia-openbabel: filename=aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl size=11102455 sha256=363d1d05dad22183022cd9e4f7c9f8a62d912bc44394f8835c03ef171b461551
2024-08-23T08:19:45.5788988Z Stored in directory: /root/.cache/pip/wheels/21/14/79/baf708a1463c156ec37e5504ed91f66de9ca8b1bc844f71781
2024-08-23T08:19:45.5824955Z Successfully built aqemia-openbabel
2024-08-23T08:19:45.5940993Z Removed build tracker: '/tmp/pip-build-tracker-q2u7ftms'
2024-08-23T08:19:45.6790642Z + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
2024-08-23T08:19:45.7065003Z + rm -rf /tmp/cibuildwheel/repaired_wheel
2024-08-23T08:19:45.7091788Z + mkdir -p /tmp/cibuildwheel/repaired_wheel
2024-08-23T08:19:45.7124575Z ##[endgroup]
however the auditwheel repair fails as it cannot find libraries since they only live in the wheel file
logs
2024-08-23T08:19:45.7126351Z ##[group]Repairing wheel...
2024-08-23T08:19:45.7126696Z
2024-08-23T08:19:45.7130107Z + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'
2024-08-23T08:19:46.0501385Z INFO:auditwheel.main_repair:Repairing aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl
2024-08-23T08:20:06.1838904Z Traceback (most recent call last):
2024-08-23T08:20:06.1870673Z File "/usr/local/bin/auditwheel", line 8, in <module>
2024-08-23T08:20:06.1871685Z sys.exit(main())
2024-08-23T08:20:06.1873139Z File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main.py", line 54, in main
2024-08-23T08:20:06.1874430Z rval = args.func(args, p)
2024-08-23T08:20:06.1875837Z File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main_repair.py", line 173, in execute
2024-08-23T08:20:06.1877130Z out_wheel = repair_wheel(
2024-08-23T08:20:06.1878534Z File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/repair.py", line 78, in repair_wheel
2024-08-23T08:20:06.1879819Z raise ValueError(
2024-08-23T08:20:06.1880749Z ValueError: Cannot repair wheel, because required library "libopenbabel.so.7" could not be located
2024-08-23T08:20:06.5140592Z ##[endgroup]
2024-08-23T08:20:06.5147019Z [31m✕ [0m20.80s
2024-08-23T08:20:06.5184462Z ##[error]Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'] failed with code 1.
I believe I must be missing something obvious here but I don't see how to fix the problem except by using a before-all action that basically builds the project. This doubles the build time which is not great. Would there be other options that I missed?
Wheel configuration through pyproject
Also I'm struggling defining the wheels I want to build. I'm only interested in building cp311-manylinux2014-x86_64.
When I set my pyproject.toml (which is what I understand as the preferred way) to
[tool.cibuildwheel]
archs = ["x86_64"]
build = ["cp311-manylinux2014-x86_64"]
platform = ["linux"]
it seems that cibuildwheel does't pick the configuration and the action run fails with
cibuildwheel: No build identifiers selected: BuildSelector(build_config='cp311-manylinux2014-x86_64', skip_config='', requires_python=<SpecifierSet('<3.12,>=3.11')>, prerelease_pythons=False, free_threaded_support=False)
So I end up defining CIBW_ARCHS, CIWB_BUILD and CIWB_PLATFORM which I would rather not. Is there some precedence rule that I misunderstood?
Thanks a lot for any help!
Build log
No response
CI config
No response
The build identifiers don’t include the manylinux docker image variant (like 2014). build = ["cp311-manylinux_x86_64"]
See the table in the docs for a full list of build identifiers, they don't include the manylinux variant as Henry says.
I miss this update. How to specify now the manylinux image to use?
You can choose the image you want to use with the manylinux-ARCH-image options. It's always been this way 🙃
Do you have a link to the repo? I'd start by running pipx run build --wheel and seeing what is actually in the wheel.
Thanks a lot for your answers! Sorry, I did not pay attention to the fact that the variant was not part of the build parameter. This matter is all resolved. I assumed that part of the log (build_config='cp311-manylinux2014-x86_64') was actually confirming that this wasn't an issue in providing the right values
For the "double build", here is a repo reproducing my setup: https://github.com/marchelbling/wheels/. Note that I actually realized that the Python dev dependencies were not picked by cmake and the python extension was not built. I updated the CMakeLists.txt but it seems that I still have issues at link time.
After re-reading the scikit-build-core documentation:
- https://scikit-build-core.readthedocs.io/en/latest/cmakelists.html#finding-python
- https://scikit-build-core.readthedocs.io/en/latest/faqs.html#finding-python
I'm starting to think that I will not manage to build OpenBabel using the manylinux image as I understand it does not include Python libraries. Am I correct?
Wheels are not supposed to "embed" Python. Python is already present when are you loading a wheel, so you don't need to link to it, and it could even lock you to the patch release you linked against. So yes, the Python libs are removed when building the manylinux image. See https://github.com/pypa/manylinux/pull/1185 for some context.
Thank you @henryiii that really helped me! It would have been awesome that the PR was merged but it was not too difficult to apply the PR diff on manylinux main branch.
Now my wheel finally builds cleanly except for the "double" build though. Note that I found another project that seems to have the same issue: https://github.com/spglib/spglib/blob/d9c572d1986b0eccc4ed5f18c8d827729a320313/pyproject.toml#L111-L112. Please let me know if you would have a chance to look at what I could be doing wrong.
I actually came across https://github.com/pypa/auditwheel/issues/285.
I'm not sure I fully understand the conclusion from the thread. I'm really not sure what the "proper" way to fix this. I stopped building twice and I'm now doing a manual repair where I unzip the wheel and set LD_LIBRARY_PATH before running auditwheel repair but that doesn't seem "right".