3.9.0: swig/python python module is not ready for PEP517 build procedure

[kloczek]

What is the bug?

cmake uses setup.py build_ext which is deprecated. I've tested as well disable python bindings in cmake and than build modules using pep517 based build procedure and it failed with

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
error: Multiple top-level packages discovered in a flat-layout: ['data', 'osgeo'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel

Steps to reproduce the issue

build with enabled python bindings with latest setuptools or disable python bindings in cmake and build module using pep517 based build procedure in swig/python.

Versions and provenance

gdal 3.9.0 with setuptools 69.4.0.

Additional context

No response

[rouault]

Can you give more details on the components that are installed on your system?

I can't reproduce with:

$ ~/install-python-3.12.0/bin/python3 -m venv myvenv_py312
$ source myvenv_py312/bin/activate
$ python3 -m pip install setuptools "numpy>=2.0.0rc1" wheel build
$ pip list
Package         Version
--------------- --------
build           1.2.1
numpy           2.0.0rc2
packaging       24.0
pip             23.2.1
pyproject_hooks 1.1.0
setuptools      69.5.1
wheel           0.43.0

$ python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
Using numpy 2.0.0rc2
running egg_info
writing gdal-utils/GDAL.egg-info/PKG-INFO
writing dependency_links to gdal-utils/GDAL.egg-info/dependency_links.txt
writing entry points to gdal-utils/GDAL.egg-info/entry_points.txt
writing requirements to gdal-utils/GDAL.egg-info/requires.txt
writing top-level names to gdal-utils/GDAL.egg-info/top_level.txt
reading manifest file 'gdal-utils/GDAL.egg-info/SOURCES.txt'
writing manifest file 'gdal-utils/GDAL.egg-info/SOURCES.txt'
* Building wheel...
Using numpy 2.0.0rc2
running bdist_wheel
running build
running build_py
[... snip ... ]
Successfully built GDAL-3.9.0-cp312-cp312-linux_x86_64.whl

[kloczek]

Here is list of python modules installed in build env

Package                       Version
----------------------------- -----------
alabaster                     0.7.16
attrs                         23.2.0
Babel                         2.15.0
breathe                       4.35.0
build                         1.2.1
charset-normalizer            3.3.2
defusedxml                    0.7.1
docutils                      0.20.1
exceptiongroup                1.1.3
filelock                      3.14.0
idna                          3.7
imagesize                     1.4.1
importlib_metadata            7.1.0
iniconfig                     2.0.0
installer                     0.7.0
Jinja2                        3.1.4
jsonschema                    4.20.0
jsonschema-specifications     2023.12.1
lxml                          5.2.2
MarkupSafe                    2.1.5
numpy                         1.26.4
packaging                     24.0
pluggy                        1.4.0
Pygments                      2.18.0
pyproject_hooks               1.0.0
pytest                        8.1.1
python-dateutil               2.9.0.post0
referencing                   0.32.0
requests                      2.31.0
rpds-py                       0.18.1
setuptools                    69.4.0
snowballstemmer               2.2.0
Sphinx                        7.3.7
sphinx_rtd_theme              2.0.0
sphinxcontrib-applehelp       1.0.8
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.0.5
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.7
sphinxcontrib-serializinghtml 1.1.10
tokenize_rt                   5.2.0
tomli                         2.0.1
urllib3                       2.2.1
wheel                         0.43.0
zipp                          3.18.2

Are you building module from gdal source tree? (from swig/python)

[kloczek]

And again .. gdal cmake is using deprecated setup.py build_ext.

[rouault]

Are you building module from gdal source tree? (from swig/python)

yes

[kloczek]

I have bo idea how did you get no errors when in pyproject.toml are missing entries listed in error message which I've copied.

[rouault]

@kloczek How can I reproduce your environment? Ideally, running a sequence of commands in a Docker image with the base distribution you use.

[kloczek]

You don't need any docker. Just run the command which currently failing in git source tree.

[rouault]

ok, I finally managed to reproduce your error message if I run "python3 -sBm build -w --no-isolation" from the "swig/python" subdirectory of the source tree, which is fully expected. This must be run from the swig/python subdirectory of the bulid directory, because setup.py is a generated file, and you must also generate the SWIG generated files with the "python_generated_files" target

From the build directory, you can do the following:

cmake .. -DDBUILD_PYTHON_BINDINGS=OFF
make -j$(nproc)
cmake .. -DDBUILD_PYTHON_BINDINGS=ON
make python_generated_files
python3 -sBm build -w --no-isolation