PEP 632: Proposed deprectation of `distutils`

[FRidh]

There is a PEP proposed for deprecating distutils. Meson uses distutils in the python module for, if I am correct, building extension modules, so this PEP may be of interest to you.

[lazka]

distutils is only used to decide if extensions should link against libpython to follow the platform defaults (Debian patched to not link, default was to link). With Python 3.8 upstream Python default changed to not link to libpython by default, so if distutils ever gets removed, meson can just default to not linking always: https://docs.python.org/3/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

[eli-schwartz]

Note that in #9288 we added an additional reliance on distutils (in the same exact introspection command that already imports distutils). In addition to detecting whether to link against libpython, we now also use distutils to detect whether sysconfig is completely crippled by the vendor and unable to tell the truth about install paths.

Handling libpython linking could be done by a try/except that reacts to an ImportError by saying "clearly this must be sys.version_info >= (3, 8) so the answer is don't link".

Handling debian paths cannot be resolved other than by waiting on debian to fix things.

As we get closer to the final removal of distutils in 3.12 we will need to both consider that try/except for libpython, and keep an eye on the debian situation.

[tacaswell]

distutils has now been pulled out of the main branch of CPython and meson is failing to detect it as a valid Python:

$ meson --version
0.64.0
$ python --version
Python 3.12.0a1+

adapted from numpy's meson branch:

project(
  'test',
  'c',
  version: '0',
  license: 'BSD-3',
  meson_version: '>= 0.64.0',
)

cc = meson.get_compiler('c')

# https://mesonbuild.com/Python-module.html
py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency()

$ meson setup build --prefix=/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages
The Meson build system
Version: 0.64.0
Source dir: /home/tcaswell/source/p/numpy/numpy
Build dir: /home/tcaswell/source/p/numpy/numpy/build
Build type: native build
Project name: test
Project version: 0
C compiler for the host machine: ccache cc (gcc 12.2.0 "cc (GCC) 12.2.0")
C linker for the host machine: cc ld.bfd 2.39.0
Host machine cpu family: x86_64
Host machine cpu: x86_64

meson.build:13:0: ERROR: <PythonExternalProgram 'python3' -> ['/home/tcaswell/.virtualenvs/bleeding/bin/python3']> is not a valid python or it is missing distutils

A full log can be found at /home/tcaswell/source/p/numpy/numpy/build/meson-logs/meson-log.txt

[CAM-Gerlach]

FYI, there's been a migration among packaging tools to use the standalone, modern sysconfig stdlib module over distutils' equivalent, which mesonpy's @FFY00 has led the charge on. Perhaps he has some insightful comments on that here?

[eli-schwartz]

We already use sysconfig for most things. There is:

• one thing that sysconfig doesn't currently provide -- I've been meaning to open a CPython ticket to have it added (I mentioned it in passing in a meson-python ticket somewhere) which I will try doing soon
• one thing that sysconfig does provide, but Debian has frustratingly broken, which I need to investigate whether it works now.

I'll try to see if I can get news on both counts in the near future. Because one of them is totally dependent on Debian's timeframe and they already know about the problem but have been flatly ignoring it for years, I've been letting the general issue slide, hoping that if I check in on it closer to the cutoff point it will finally resolve itself. But I didn't think about people using the alphas for 3.12 a year before the cutoff point. :)

[CAM-Gerlach]

I see, thanks. In that case, I'd suggest opening a CPython issue for the former as soon as possible (and the latter, if there's anything needed on the upstream end), to allow plenty of time before the beta freeze in less than 6 months for discussion, implementation, review and merging, and preferably allow you, distros and others to test with it.

[tacaswell]

The suggestion of installing a setuptools 60+ worked for getting meson to work with py312.

[dnicolodi]

It seems that recent Debian patches sysconfig too now https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/patches/sysconfig-debian-schemes.diff I'll have a look at when this was introduced. If it is old enough (how old?) we can probably rely on it. Otherwise we will need to make the checks conditional. I'll see what can be done.

Edit: it looks like the sysconfig patching goes way back to Python 3.7 https://salsa.debian.org/cpython-team/python3/-/blob/python3.7/debian/patches/sysconfig-debian-schemes.diff I think we can probably safely rely on it. I'll prepare a PR and see what the CI tests think about it.

[dnicolodi]

It would be great to have some input from the Debian Python maintainers on this.

[dnicolodi]

Probably worth checking https://www.debian.org/doc/packaging-manuals/python-policy/

[dnicolodi]

I've done some more archeology. The sysconfig patch is in the repository for the Debian's Python package since a long time, but it is actually installed only starting with Python 3.10, in Debian testing. The only thing that can be done is to check the Python version ad fall back to distutils for Python versions earlier than 3.10.

The other distutils is used for is this: https://github.com/mesonbuild/meson/blob/47d1a55e780f928c9c1613220d6267de530e64b4/mesonbuild/modules/python.py#L366-L370 for which there is no replacement AFAICT. @eli-schwartz Do you want me to open an issue with CPython for this?

[FFY00]

I've done some more archeology. The sysconfig patch is in the repository for the Debian's Python package since a long time, but it is actually installed only starting with Python 3.10, in Debian testing. The only thing that can be done is to check the Python version ad fall back to distutils for Python versions earlier than 3.10.

See https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/ if you want some more archeology :stuck_out_tongue:

Do you want me to open an issue with CPython for this?

https://github.com/python/cpython/issues/88611 is the proposed fix.

[dnicolodi]

IIUC, the linked issue is about where to find the headers, not about whether to link or not with libpython. Aren't these separate issues?

[FFY00]

Yes, sorry, I misread it.

[FFY00]

The reason Debian patches Python in this way is https://github.com/python/cpython/issues/65735. I am trying to understand if it is still relevant, or if we can just revert to looking at sysconfig.get_config_var('Py_ENABLE_SHARED').

[eli-schwartz]

Well, yes.

For Linux, it depends on the python version, and on older versions of python, it also depends on whether there's a Debian patch.

There's other rules for other platforms.

[FFY00]

But older platforms will have distutils available to do the check, we need to figure out if this is still relevant on newer versions, and if so, figure out a way to support it without distutils.

[dnicolodi]

@FFY00 In https://github.com/python/cpython/issues/88611 the conclusion is that Python itself should not bother defining the include path. This means that whatever solution will be identified will be in one of the PyPA packages. Which makes it unsuitable for consumption by Meson because of the no-external-dependencies policy. What's the plan B?

[dnicolodi]

@FFY00 I have the impression that your analysis in https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/ is not complete for recent Python distributed by Debian. Debian patches Python 3.10 to also update the paths returned by sysconfig.

[eli-schwartz]

I see, thanks. In that case, I'd suggest opening a CPython issue for the former as soon as possible (and the latter, if there's anything needed on the upstream end), to allow plenty of time before the beta freeze in less than 6 months for discussion, implementation, review and merging, and preferably allow you, distros and others to test with it.

I finally got around to this (plus a bit more research into all the moving parts): https://github.com/python/cpython/issues/99942

And in #11133 I have a draft PR that fully removes distutils, but may need a CPython fix. It actually turns out that there's a sysconfig variable that almost gives us what we need for link_libpython.

Debian should simply work without fuss after a version gate, mid-3.10.x

[eli-schwartz]

I finally got around to this (plus a bit more research into all the moving parts): python/cpython#99942

I have two PRs linked to that issue, and it would be great to have confirmation that I'm on the right track here (obviously I think I am). Also, I think that maybe to handle Android correctly, we need to drop the pkg-config detection method? :thinking: This isn't tested in CI, but Cygwin is. Cygwin works okay, because Cygwin patches it to be correct.

Assuming that CPython upstream agrees with me, I think we can handle this reasonably okay on our end.

[CAM-Gerlach]

I've pinged about it to the core dev team.

[danyeaw]

I just tested doing a meson build with Python 3.12.0b3. python = python.find_installation('python3') fails because it doesn't detect this version of Python as a valid one without distutils being there.

[dnicolodi]

@danyeaw, currently you need to install setuptools to make meson work with Python 3.12 or later: setuptools provides distutils.

[danyeaw]

@dnicolodi Correct, but meson finds the system Python, not necessarily the activated virtualenv. This may require users to install setuptools system wide, which isn't great.

[dnicolodi]

I know, and this is way this issue is open and there is a draft pull request removing the dependency on distutils. Installing setuptools is the current way to get things working, the alternative is to do not use Meson with Python 3.12. Note also that you can point Meson to any Python environment you like, including a venv.

[danyeaw]

How can I help, are we still waiting on CPython to do something?

[FFY00]

No changes from CPython's side are required to fix this, and nothing significant will happen there until 3.14 at least anyway.