cmake-python-distributions
cmake-python-distributions copied to clipboard
scikit-build
3.28.1: why is build own copy of cmake? 🤔
build output
+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
running egg_info
creating src/cmake.egg-info
writing src/cmake.egg-info/PKG-INFO
writing dependency_links to src/cmake.egg-info/dependency_links.txt
writing entry points to src/cmake.egg-info/entry_points.txt
writing requirements to src/cmake.egg-info/requires.txt
writing top-level names to src/cmake.egg-info/top_level.txt
writing manifest file 'src/cmake.egg-info/SOURCES.txt'
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
WARNING setuptools_scm._file_finders.git git archive detected - fallback to listing all files
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory '_skbuild'
warning: no previously-included files matching '*' found under directory 'CMake-src'
adding license file 'LICENSE_Apache_20'
adding license file 'LICENSE_BSD_3'
adding license file 'AUTHORS.rst'
writing manifest file 'src/cmake.egg-info/SOURCES.txt'
* Building wheel...
--------------------------------------------------------------------------------
-- Trying 'Ninja' generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- The C compiler identification is GNU 14.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 14.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying 'Ninja' generator - success
--------------------------------------------------------------------------------
Configuring Project
Working directory:
/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build
Command:
/usr/bin/cmake /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3 -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-install/src/cmake/data -DPYTHON_VERSION_STRING:STRING=3.8.18 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/usr/lib/python3.8/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:PATH=/usr/lib64/libpython3.8.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.8 -DCMAKE_BUILD_TYPE:STRING=Release
Not searching for unused variables given on the command line.
-- The CXX compiler identification is GNU 14.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- ***************************************************
-- Build CMake from source: ON
-- ***************************************************
CMake Warning (dev) at /usr/share/cmake/Modules/ExternalProject.cmake:3195 (message):
The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
not set. The policy's OLD behavior will be used. When using a URL
download, the timestamps of extracted files should preferably be that of
the time of extraction, otherwise code that depends on the extracted
contents might not be rebuilt if the URL changes. The OLD behavior
preserves the timestamps from the archive instead, but this is usually not
what you want. Update your project to the NEW behavior or specify the
DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
robustness issue.
Call Stack (most recent call first):
/usr/share/cmake/Modules/ExternalProject.cmake:4418 (_ep_add_download_command)
CMakeLists.txt:138 (ExternalProject_add)
This warning is for project developers. Use -Wno-dev to suppress it.
-- SuperBuild - CMakeProject-src-download
-- SuperBuild - CMakeProject-src-download - URL: https://github.com/Kitware/CMake/releases/download/v3.28.3/cmake-3.28.3.tar.gz
-- SuperBuild - CMakeProject-src-download - CMakeProject_SOURCE_DIR: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/CMake-src
-- Looking for __GLIBC__
-- Looking for __GLIBC__ - found
-- Looking for __GLIBC_MINOR__
-- Looking for __GLIBC_MINOR__ - found
-- SuperBuild - CMakeProject-build
-- SuperBuild - CMakeProject-build - CMakeProject_BINARY_DIR: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build
-- SuperBuild - CMakePythonDistributions
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build
[1/27] Performing download step (download, verify and extract) for 'CMakeProject-src-download'
-- Downloading...
dst='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.28.3.tar.gz'
timeout='none'
inactivity timeout='none'
-- Using src='https://github.com/Kitware/CMake/releases/download/v3.28.3/cmake-3.28.3.tar.gz'
-- [download 0% complete]
-- [download 1% complete]
-- [download 2% complete]
-- [download 3% complete]
-- [download 4% complete]
-- [download 5% complete]
-- [download 6% complete]
-- [download 7% complete]
-- [download 8% complete]
-- [download 9% complete]
-- [download 10% complete]
-- [download 11% complete]
-- [download 12% complete]
-- [download 13% complete]
-- [download 14% complete]
-- [download 15% complete]
-- [download 16% complete]
-- [download 17% complete]
-- [download 18% complete]
-- [download 19% complete]
-- [download 20% complete]
-- [download 21% complete]
-- [download 22% complete]
-- [download 23% complete]
-- [download 24% complete]
-- [download 25% complete]
-- [download 26% complete]
-- [download 27% complete]
-- [download 28% complete]
-- [download 29% complete]
-- [download 30% complete]
-- [download 31% complete]
-- [download 32% complete]
-- [download 33% complete]
-- [download 34% complete]
-- [download 35% complete]
-- [download 36% complete]
-- [download 37% complete]
-- [download 38% complete]
-- [download 39% complete]
-- [download 40% complete]
-- [download 41% complete]
-- [download 42% complete]
-- [download 43% complete]
-- [download 44% complete]
-- [download 45% complete]
-- [download 46% complete]
-- [download 47% complete]
-- [download 48% complete]
-- [download 49% complete]
-- [download 50% complete]
-- [download 51% complete]
-- [download 52% complete]
-- [download 53% complete]
-- [download 54% complete]
-- [download 55% complete]
-- [download 56% complete]
-- [download 57% complete]
-- [download 58% complete]
-- [download 59% complete]
-- [download 60% complete]
-- [download 61% complete]
-- [download 62% complete]
-- [download 63% complete]
-- [download 64% complete]
-- [download 65% complete]
-- [download 66% complete]
-- [download 67% complete]
-- [download 68% complete]
-- [download 69% complete]
-- [download 70% complete]
-- [download 71% complete]
-- [download 72% complete]
-- [download 73% complete]
-- [download 74% complete]
-- [download 75% complete]
-- [download 76% complete]
-- [download 77% complete]
-- [download 78% complete]
-- [download 79% complete]
-- [download 80% complete]
-- [download 81% complete]
-- [download 82% complete]
-- [download 83% complete]
-- [download 84% complete]
-- [download 85% complete]
-- [download 86% complete]
-- [download 87% complete]
-- [download 88% complete]
-- [download 89% complete]
-- [download 90% complete]
-- [download 91% complete]
-- [download 92% complete]
-- [download 93% complete]
-- [download 95% complete]
-- [download 96% complete]
-- [download 98% complete]
-- [download 99% complete]
-- [download 100% complete]
-- verifying file...
file='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.28.3.tar.gz'
-- Downloading... done
-- extracting...
src='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.28.3.tar.gz'
dst='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/CMake-src'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[.and so on.]
Why it is not checked cmake command availability add is added only wrapper on system cmake command which checks only version of the system cmake? 🤔
It does not make ANY sense.
Wheels can not have any logic in them. They simply are unzipped to the correct directories. And what are the chances the system will have exactly the same version of cmake that you are installing? Or would you expect pip install cmake==3.28.3 to be happy with 3.26.1 or whatever is on your system?
I think what you want is a smart build backend that only adds cmake if the command is either not present or not of sufficient version - which is exactly what scikit-build-core does!
Wheels can not have any logic in them. They simply are unzipped to the correct directories. And what are the chances the system will have exactly the same version of cmake that you are installing? Or would you expect
pip install cmake==3.28.3to be happy with3.26.1or whatever is on your system?
I'm expecting that this module would be just wrapper checking cmake executable version according to its version and that's it ..
I think what you want is a smart build backend that only adds
cmakeif the command is either not present or not of sufficient version - which is exactly what scikit-build-core does!
I don't see at all building own copy of cmake it it provides only interface to use cmake.
The same pkgconfig python module (https://github.com/matze/pkgconfig/) is not building own copy of the pkgconfig.
It provides only python interface to use it.
The pkgconfig module is not a PyPI copy of pkg-config, it's a wrapper around a command, just like scikit-build-core is a wrapper around the cmake command providing a custom Python interface. If you don't have pkg-config pre-installed, then pkgconfig module raises an error. The version of pkgconfig (Python) has nothing to do with the version of peg-config you've pre-installed.
The entire point of the cmake module is to distribute cmake. It does not wrap the command (other than providing a very, very simple and not quite perfect cmake runner wrapper to make it easier to get the paths right).
cmake executable version according to its version
The chances of it exactly matching are really small. If you have 3.28.1 and the cmake module is 3.28.2, then it would still have to build, since you requested 3.28.2. We don't release every patch release, so some system CMake's wouldn't be possible to request a "dummy" for at all. And what do you do if cmake is present but ctest is missing?
What's worse, the builder can't tell if you are installing or making a wheel (modern packaging - there used to be a setup.py install but it's replaced by always going through a wheel). If you are preparing the wheels for offline use (such as in test suites - scikit-build-core does exactly this in it's tests, HPC/grid/cloud computing, etc), then how do you know that you can't just make the "dummy" cmake package if you happen to have the matching CMake version present? Would your CI just randomly break if you upgrade CMake? Also, if you get a universal wheel, you'd expect a universal CMake even if you only have x86 or ARM installed.
Python packaging is not compatible with this sort of design. A package should not build different wheels based on its environment. A user expects to get the version of CMake they asked for. And you can't tell what a user actually asked for! cmake > 3.15 will resolve to cmake-3.28.3.tar.gz, and there's no way to then backtrack and tell that the system CMake (say, 3.17) is actually fine according to what the user asked. And once you've installed it, there's no way to re-evalutate it if someone then askes for cmake > 3.21, the system things that 3.28.3 is installed even if it's a dummy package for 3.17!
There's a very simple and easy solution for building package with CMake: Only ask for the Python package if you need a copy of cmake! As I've said, scikit-build-core does this, and there's an example in scikit-build's docs on how to do it yourself, it's less LoC than most setuptools customizations used to be. meson-python does this with ninja and patchelf too, which are also binary redistributions on PyPI.
This was discussed for ninja-python-distributions multiple times with the same conclusion: fix the builders to only ask for what they need instead of making lying packages that don't include what they are supposed to include based on the system they were built on.
By the way, 95% of the time you should get a binary; nothing will build. We provide wheels for every system we can. It only builds if you make it build (as shown above) or if you are on a system like Pyodide / FreeBSD / Android / iOS, where we can't provide wheels. Again, the fix is easy - only request it in get_requires_for_build_wheel() if cmake is not present or not in the version range you need. scikit-build-core 0.8+ even lets you specify a version range or disallow specific versions (any valid Python version specifiers allowed).
By the way, 95% of the time you should get a binary; nothing will build. We provide wheels for every system we can. It only builds if you make it build (as shown above) or if you are on a system like Pyodide / FreeBSD / Android / iOS, where we can't provide wheels. Again, the fix is easy - only request it in get_requires_for_build_wheel() if cmake is not present or not in the version range you need. scikit-build-core 0.8+ even lets you specify a version range or disallow specific versions (any valid Python version specifiers allowed).
Waste of time. cmake is now peaty stable and is available as REGULAR and NATIVE package on all those systems. New cmake releases are only providing some minor cleanups/bugfixes and release this module ne version only because new cmake version has been released is nothing more than another waste of time. Look on last this module releases diffs and you will find that actually NOTHING has changed in actual cmake interface. All changes are related to downloading new versions of source code.
I catalog changes in cmake on this page: https://cliutils.gitlab.io/modern-cmake/chapters/intro/newcmake.html
And the versions available on various systems on this page: https://cliutils.gitlab.io/modern-cmake/chapters/intro/installing.html
And there are massive, huge changes in each version. 3.28 added support for C++20 modules. So if I want to use those, I have to set cmake>=3.28. The first version to properly support FindPython was 3.18.2 (with important changes as recently as 3.26, like support for Stable ABI), which you'll find is not available on even the newest LTS Ubuntu. And the next version coming out (RC1 came out a couple of days ago) will finally have the ability to make the test target depend on ALL!
If you want to make a "cmake-if-needed" SDist only package, go ahead. There's nothing stopping you. But we can't break everyone (including most scikit-build-core users!) who need a modern CMake and request it (scikit-build-core requests it only if actually needed, responsibly!) by not shipping cmake in a cmake package. No one is asking ruff, clang-format, or any of the other built CLI tools to ignore distributing ruff or clang-format binaries if there is already a matching copy on the system (for the long list of reasons above, it would be a terrible idea).
This is not the right solution to the problem. The correct solution is to fix those packages to only request cmake if the system doesn't provide one. The mechanism for doing exactly that was standardized in 2016.
As I said on the ninja discussions, I'd also be okay with an opt-in way to make a dummy package, but we are about to move the build backend, so it would be best to wait on that. I also want to change the way the binary is provided, which will happen after the move but before any dummy option.
Also, 3.27 removed (sort-of) the old FindPythonLibs/FindPythonInterp. So the CMake version really matters to Python users.
And there are massive, huge changes in each version. 3.28 added support for C++20 modules. So if I want to use those, I have to set cmake>=3.28. The first version to properly support FindPython was 3.18.2 (with important changes as recently as 3.26, like support for Stable ABI), which you'll find is not available on even the newest LTS Ubuntu. And the next version coming out (RC1 came out a couple of days ago) will finally have the ability to make the test target depend on ALL!
Please show me ONE python module which depends on some cmake features added after first cmake 3.0. I have already packaged as rpm packages
[tkloczko@pers-jacek SPECS]$ ls python-*spec -1| wc -l
1245
Amongst those modules there ONLY SIX
[tkloczko@pers-jacek SPECS]$ grep "BuildRequires:.*cmake$" python-*
python-gcovr.spec:BuildRequires: cmake
python-pybind11.spec:BuildRequires: cmake
python-rapidfuzz.spec:BuildRequires: cmake
python-scikit-build-core.spec:BuildRequires: cmake
python-scikit-build.spec:BuildRequires: cmake
python-ssh-python.spec:BuildRequires: cmake
which are using cmake and none are using your module. NONE of those modules depends on any cmake features added after first cmake 3.x release.
cmake 3.18.2 has been released almost FOUR years ago. It was the time when still python 2.x was in use, And look on stats. ~1.245k python modules and less than HALF percent uses now cmake.
Please show me ONE python module which depends on some cmake features added after first cmake 3.0. I have already packaged as rpm packages
A little more than one example: All our C++ projects with Python bindings (e.g., https://github.com/cda-tum/mqt-core) require CMake>=3.19 and might, at some point, jump to 3.24 for better FetchContent support.
Why do you want to use FetchContent if cmake already is installed in the system image? 🤔
Why do you want to use
FetchContentif cmake already is installed in the system image? 🤔
We do not use FetchContent to get CMake, but other project dependencies. CMake is just provided via the mechanisms in scikit-build-core described above by Henry.
We do not use FetchContent to get CMake, but other project dependencies.
Name of those modules.
None of those six modules which I've mentioned so far which are using cmake are using FetchContent.
Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network?
How do you recommend to perform build of your module in such conditions? 🤔
Name of those modules. None of those six modules which I've mentioned so far which are using cmake are using
FetchContent.
For example, googletest or nlohmann_josn to name the most prominent ones.
Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network? How do you recommend to perform build of your module in such conditions? 🤔
By making use of the modern find_package integration of FetchContent that landed in 3.24 (which is manually back ported to 3.19 in our projects). See https://cmake.org/cmake/help/latest/module/FetchContent.html#integrating-with-find-package
Name of those modules. None of those six modules which I've mentioned so far which are using cmake are using
FetchContent.For example,
googletestornlohmann_josnto name the most prominent ones.
https://pypi.org/project/gtest/ is no longer maintained. Homepage shows 404.
https://github.com/nlohmann/json/ it is not python module.
Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network? How do you recommend to perform build of your module in such conditions? 🤔
By making use of the modern
find_packageintegration ofFetchContentthat landed in 3.24 (which is manually back ported to 3.19 in our projects). See https://cmake.org/cmake/help/latest/module/FetchContent.html#integrating-with-find-package
I've not been asking for what is FetchContent and with what it is integrated.
Sorry but please read again my question.
https://pypi.org/project/gtest/ is no longer maintained. Homepage shows 404.
https://github.com/nlohmann/json/ it is not python module.
Yeah. And I never claimed that. These are dependencies of our C++ code that is then exposed as a Python module via pybind11 and distributed to PyPI. So, although these are dependencies of our C++ code, they are also dependencies of the resulting Python module. And building that Python module requires CMake>=3.19.
I've not been asking for what is
FetchContentand with what it is integrated. Sorry but please read again my question.
... 🙃
Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network?
Yes.
How do you recommend to perform build of your module in such conditions? 🤔
By having the required dependencies installed on the system before the build. FetchContent will directly pick up the installed version without ever accessing the public network.
How do you recommend to perform build of your module in such conditions? 🤔
By having the required dependencies installed on the system before the build.
FetchContentwill directly pick up the installed version without ever accessing the public network.
So WHY you don't require have installed cmake before build your module? 🤔
How do you recommend to perform build of your module in such conditions? 🤔
By having the required dependencies installed on the system before the build.
FetchContentwill directly pick up the installed version without ever accessing the public network.So WHY you don't require have installed cmake before build your module? 🤔
That is not a question I am here for to answer. You were asking for examples of Python modules that depend on modern CMake features and that is what I tried to show you.
That is not a question I am here for to answer. You were asking for examples of Python modules that depend on modern CMake features and that is what I tried to show you.
Because you provided bogus exampled it allowed me BACK to main question which is: why your module cannot just use system installed cmake? 🤔
Because you provided bogus exampled it allowed me BACK to main question which is: why your module cannot just use system installed cmake? 🤔
It does use the system installed cmake if the corresponding version is new enough. That is precisely what scikit-build-core does for you and what Henry alluded to in the very first reply to your initial issue.
OK I give up .. I've not been asking about what scikit-build-core ids doing form me. You are not even trying to answer on my questions. All what you doing is trying to AVOID straight answers.
Thanks everyone for commenting by sharing perspectives, details, links and resources, I learnt a lot reviewing these :pray:
I think what you want is a smart build backend that only adds cmake if the command is either not present or not of sufficient version - which is exactly what scikit-build-core does!
Once we are done integrating https://github.com/scikit-build/cmake-python-distributions/pull/312, it seems like the particular issue reported by @kloczek would be addressed.
The entire point of the cmake module is to distribute cmake
@kloczek To help refocus, is there a particular package you are tying to distribute as Python wheels ?
@kloczek To help refocus, is there a particular package you are tying to distribute as Python wheels ?
Yes .. https://github.com/AcademySoftwareFoundation/OpenTimelineIO/ build currently depends on cmake module.
I thing that easier would be just patch that build procedure removing cmake from build dependencies replacing that by checking is cmake/ctest commands are already installed because cmake python module does not provide anything useful than paths to those executables.
re: OpenTimelineIO
Great. Do you have already a branch to update the project to use scikit-build-core backend ? I see that it currently builds the wheels using the legacy setup.py file.
That project requires CMake 3.18.2+: https://github.com/AcademySoftwareFoundation/OpenTimelineIO/blob/d22935e0033510bd834abbeb42e95b95ea94fab6/CMakeLists.txt#L3
It also uses the cmake command (not import cmake): https://github.com/AcademySoftwareFoundation/OpenTimelineIO/blob/d22935e0033510bd834abbeb42e95b95ea94fab6/setup.py#L157
So all you need to do is build without isolation (which I would recommend when packaging for a distribution anyway, isolated == needs internet) and avoid validating build dependencies (which is an extra flag with pip). This is what many conda packages do.
So all you need to do is build without isolation (which I would recommend when packaging for a distribution anyway, isolated == needs internet) and avoid validating build dependencies (which is an extra flag with pip). This is what many conda packages do.
Seems you did not red what I've already wrote that some build systems are intentionally cut off from access to the public network.
conda is not part of any python PEP specification.
Currently all modules should be PEP571 compliant.
To move forward with this, I suggest you modernize the OpenTimelineIO project to remove the use of setup.py and then adopt the scikit-build-core backend.
Let us know if you need help moving forward with this.
@jcfr, I believe @kloczek is trying to make an RPM for OpenTimelineIO, and is not otherwise associated with the project. @JeanChristopheMorinPerso is.
An "isolated build" in PEP 517 terms is one that is isolated from the current Python environment by making a temporary Python environment and downloading dependencies from the network! I know it's the reverse of what you might think of as "isolated", but isolated PEP 517 builds == access to network. So you want to turn off isolation so that you can use your current, prebuilt build environment (it's still a PEP 517 build). All third party distributions should be disabling build isolation, that's only for users that have access to the internet and are getting the package directly from PyPI and do not have a build recipe.
Conda-forge is a distro - it has a strong Python focus, but it's just a redistributor; I was using that as an example of where this problem is already solved by using the technique I'm outlining here. Add cmake (the RPM) to the dependencies, add the --no-build-isolation flag when building, and you should be fine, no need to worry about this package (a cmake redistribution for PyPI) at all.
If there's a problem doing this, let me know, I can help.
@jcfr, I believe @kloczek is trying to make an RPM for OpenTimelineIO, and is not otherwise associated with the project. @JeanChristopheMorinPerso is.
Correct.
An "isolated build" in PEP 517 terms is one that is isolated from the current Python environment by making a temporary Python environment and downloading dependencies from the network! I know it's the reverse of what you might think of as "isolated", but isolated PEP 517 builds == access to network. So you want to turn off isolation so that you can use your current, prebuilt build environment (it's still a PEP 517 build). All third party distributions should be disabling build isolation, that's only for users that have access to the internet and are getting the package directly from PyPI and do not have a build recipe.
As I'm building my own packages I'm not interested to test what I'm packaging against random modules downloaded as .whl archives bu ONLY against what I have already packaged (as I wrote I have now almost ~1.25k such packaged modules) to test that what I've already packaged is OK. This is why I';m using obligatory rule to execute rpm %check in which usually is pytest execution (currently 99% those packages are tested that way .. pytest can handle unittest based test suites as well)
Using obligatory test suite execution on packaging automatically proves with very high probability that what is packages works together as well.
[tkloczko@pers-jacek SPECS]$ ls -1 python-*spec | wc -l; grep ^%pytest python-*spec|wc -l
1246
1234
So currently only 12 of those packages are not using pytest.
Using pytest is possible to test modules even if it do not have any test units.
Because each package build is performed in dedicated build env spawned only to build single package in which are installed only build dependencies using tox, nox or virtualenv does not make any sense as well.
Conda-forge is a distro - it has a strong Python focus, but it's just a redistributor; I was using that as an example of where this problem is already solved by using the technique I'm outlining here.
Which additionally on packaging stuff as rpm packages adds only tons of additional build dependencies.
Add cmake (the RPM) to the dependencies, add the
--no-build-isolationflag when building, and you should be fine, no need to worry about this package (a cmake redistribution for PyPI) at all.
Nope .. that is not true because cmake is listed as pep517 build dependency and example build (executed as python3 -sBm build -w --no-isolation) will instantly exit non-zero exit code if cmake python module is not present in build env.
I hope that above provides full context of what I'm doing (why, using what kind assumptions, and what kind of methodology). In that exact context I'm not interested to have two copies of cmake in whole packages distribution.
Add cmake (the RPM) to the dependencies, add the --no-build-isolation flag when building, and you should be fine, no need to worry about this package (a cmake redistribution for PyPI) at all.
Nope .. that is not true because cmake is listed as pep517 build dependency and example build (executed as python3 -sBm build -w --no-isolation) will instantly exit non-zero exit code if cmake python module is not present in build env.
What Henry suggests here is to add cmake and setuptools as a build dependency in your RPM spec file, the same way you would do for a C++ library that requires cmake to be built, and use python -m build --no-isolation or pip install . --no-build-isolation. On top of that, the sdist on PyPI contains all the code necessary (all submodules are already there), so you can build it without network access.
Also, I'll add that you don't need to use the python building tools to compile OTIO (including the python bindings). OTIO was developed in an industry where C++ is heavily used and where Python gets embedded into C++ apps. You can simply call cmake -DOTIO_CXX_INSTALL=ON -DOTIO_PYTHON_INSTALL=ON and it will build the bindings and install the python modules for you. It's a supported use case.
Yes .. https://github.com/AcademySoftwareFoundation/OpenTimelineIO/ build currently depends on cmake module. I thing that easier would be just patch that build procedure removing cmake from build dependencies replacing that by checking is cmake/ctest commands are already installed because cmake python module does not provide anything useful than paths to those executables.
@kloczek Having cmake in the build-system.requires section does provide a lot of value for a lot of users. It allows our users to build the project themselves if they want (for example by calling pip install opentimelineio --no-binary=:all: and not have to worry about having cmake or which version of cmake they need. This was a big pain point before.
If you have any OTIO related questions, please feel free to open an issue with the project at https://github.com/AcademySoftwareFoundation/OpenTimelineIO/issues.
Nope .. that is not true because cmake is listed as pep517 build dependency and example build (executed as
python3 -sBm build -w --no-isolation) will instantly exit non-zero exit code if cmake python module is not present in build env.
Build requires the -x (or --skip-dependency-check) option. That's literally why that option exists, for use in exactly this situation. I'd use -wnx if you like short options. Third party packagers are not expected to have the Python redistributions for build dependencies like cmake, ninja, etc, and are expected to add this.
What Henry suggests here is to add
cmakeandsetuptoolsas a build dependency in your RPM spec file, the same way you would do for a C++ library that requires cmake to be built, and usepython -m build --no-isolationorpip install . --no-build-isolation. On top of that, the sdist on PyPI contains all the code necessary (all submodules are already there), so you can build it without network access.
Here are my spec files for cmake and OpenTimelineIO modules which did not changed since I've started this discussion.
Summary: CMake is an open-source, cross-platform family of tools designed to build, test and package software
Name: python-cmake
Version: 3.28.3
Release: 2%{?dist}
License: Apache-2.0 (https://spdx.org/licenses/Apache-2.0.html)
URL: https://pypi.org/project/cmake/
VCS: https://github.com/scikit-build/cmake-python-distributions/
Source: %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
Patch: %{name}-man3_level.patch
BuildRequires: cmake
BuildRequires: gcc-c++
BuildRequires: openssl-devel
BuildRequires: python3dist(build)
BuildRequires: python3dist(installer)
BuildRequires: python3dist(setuptools-scm)
BuildRequires: python3dist(scikit-build)
BuildRequires: python3dist(sphinx)
BuildRequires: python3dist(wheel)
# ChcekRequires:
BuildRequires: python3dist(pytest)
Requires: cmake
%description
CMake is used to control the software compilation process using simple platform
and compiler independent configuration files, and generate native makefiles and
workspaces that can be used in the compiler environment of your choice.
%prep
%autosetup -p1 -n cmake-python-distributions-%{version}
%build
%pyproject_wheel
%sphinx_build_man
%install
%pyproject_install
%__install -Dm644 build/sphinx/man/*.3 -t %{buildroot}%{_mandir}/man3
%check
%pytest
%files
%{_mandir}/man3/*
%{python3_sitearch}/cmake
%{python3_sitearch}/cmake-*.*-info
Summary: Editorial interchange format and API
Name: python-OpenTimelineIO
Version: 0.15
Release: 2%{?dist}
License: Apache-2.0 (https://spdx.org/licenses/Apache-2.0.html)
URL: https://pypi.org/project/OpenTimelineIO/
VCS: https://github.com/AcademySoftwareFoundation/OpenTimelineIO/
Source: %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
Patch: %{name}-man3_level.patch
BuildRequires: gcc-c++
BuildRequires: python3dist(build)
BuildRequires: python3dist(cmake)
BuildRequires: python3dist(installer)
BuildRequires: python3dist(setuptools)
BuildRequires: python3dist(sphinx)
BuildRequires: python3dist(wheel)
# CheckRequires:
#BuildRequires: python3dist(pyaaf2) >= 1.4
BuildRequires: python3dist(pytest)
%description
Editorial interchange format and API
%prep
%autosetup -p1 -n OpenTimelineIO-%{version}
%build
%pyproject_wheel
%sphinx_build_man
%install
%pyproject_install
%__install -Dm644 build/sphinx/man/*.3 -t %{buildroot}%{_mandir}/man3
%check
%pytest
%files
%{_mandir}/man3/*
%{python3_sitelib}/*
As you see al your suggestions are already on place.
@kloczek Having cmake in the build-system.requires section does provide a lot of value for a lot of users. It allows our users to build the project themselves if they want (for example by calling pip install opentimelineio --no-binary=:all: and not have to worry about having cmake or which version of cmake they need. This was a big pain point before.
So you want to me to shoot in my own foot to obey PEP517 build time dependencies checks to be surprised in the future by on next upgrade to be surprised that rpm build procedure did not bark that those dependencies has been changed? 🤔 Thank you for .. a "advice" ..