Status of packaging tools

[rgommers]

• [x] packaging: 24.0 has full support
• [x] Build frontends:
  • [x] pip: in 24.1 (released 21 June)- https://github.com/pypa/pip/pull/12300
  • [x] build: probably nothing to do, since it depends on latest packaging
• [x] Build backends:
  • [x] meson-python: done modulo adding CI job, see meson-python#499
  • [x] scikit-build-core: https://github.com/scikit-build/scikit-build-core/pull/741
  • [x] setuptools: 69.5.0 upgraded the vendored packaging library to 24.0
• [x] manylinux: https://github.com/pypa/manylinux/pull/1564
• [x] cibuildwheel: https://github.com/pypa/cibuildwheel/issues/1657 (no macOS installer yet, waiting for 3.13.0b2)
• [x] wheel publishing tools (probably all fine, will be ticked off once they're known to work):
  • [x] auditwheel
  • [x] delvewheel
  • [x] delocate
  • [x] repair-wheel

[henryiii]

• Scikit-build-core done in https://github.com/scikit-build/scikit-build-core/pull/741
• manylinux done
• Cibuildwheel done except no macOS yet, waiting on beta 2 for official installers

[rgommers]

Thanks @henryiii! Table updated.

[henryiii]

Looks like both meson and CMake don't support Windows + free-threading yet, they don't know to look for the library with the "t" for linking. I think I can fix this in scikit-build-core if I can tell it the correct library from sysconfig info.

[rgommers]

Yeah, working on the same thing for Meson. I think adding the t in python313t.lib based on:

bool(sysconfig.get_config_var('Py_GIL_DISABLED'))

should do the job. Plus the debug flavor of it (there's no limited API variant).

[henryiii]

Windows free-threaded fixed in scikit-build-core 0.9.5.

[henryiii]

FYI, sysconfig's LIBRARY is correct as far as the "t" goes but is .dll instead of .lib.

[henryiii]

Free-threading including macOS with 3.13.0b2 fully supported in cibuildwheel 2.19.0.

[rgommers]

The final release of Pip 24.1 is out 🎉. That was the biggest missing piece here, we're in a pretty happy place now.

[rgommers]

The Windows support in Meson by fixed in https://github.com/mesonbuild/meson/pull/13338. @henryiii there is a second issue on Windows beyond libpython, namely that pyconfig.h is missing the Py_GIL_DISABLED define (discussed in https://discuss.python.org/t/windows-installer-freethreading-and-building-extension-modules/54391/2). I only lightly browsed your CMake/scikit-build-core PRs, so maybe I missed it - but it wasn't clear to me that that fix was included.

EDIT: xref https://gitlab.kitware.com/cmake/cmake/-/issues/26016 for CMake Windows issue.

[minrk]

Is it appropriate to add tracking of pyO3 and cffi here in build backends?

• pyO3 tracking issue: https://github.com/PyO3/pyo3/issues/4265
• cffi: https://github.com/python-cffi/cffi/issues/40

I was testing Jupyter today, and of the extension dependencies for JupyterLab, these appear to both be limiting factors. Most extensions are working fine, though:

• pyzmq works without modification when built with the cython prerelease
• argon-cffi-bindings uses cffi which unconditionally sets the unsupported py_limited_api=True (I got this to build with https://github.com/python-cffi/cffi/pull/104)
• jsonschema uses rpds-py, which uses pyO3. rpds builds, but crashes at import

so I can run a nogil kernel, but not the UI yet.

[rgommers]

Hey @minrk, thanks for the input! I'd say that these two aren't packaging tools so this isn't quite the right issue - and they're both important enough to deserve their own tracking issue.

PyO3 working will take a while, work has just started and it's quite complex. @ngoldbaum perhaps you could open a separate tracking issue for this?

cffi hasn't moved since December I think (aside from the PR you just opened), and it's the first time it has come up here. Maybe you want to open a new issue for it on this repo?

[ngoldbaum]

I opened https://github.com/Quansight-Labs/free-threaded-compatibility/issues/53 for PyO3, thanks for the reminder.

[minrk]

Thanks, I somehow overlooked that pybind11 is not in this list, which is what I was thinking of as a peer for these. Opened #54 for cffi.

[henryiii]

but it wasn't clear to me that that fix was included.

FYI, this was fixed in CMake 3.30.3 a month or two ago.

[rgommers]

Great, thanks @henryiii!

[rgommers]

Windows wheels are also starting to materialize and that all looks good, so I ticked off delvewheel. I'm sure there are some corner case bugs lurking somewhere, but overall we're in good shape here.

[rgommers]

I think this is in pretty good shape, everything has been working well for about half a year. The only two packaging tools I'm aware of that still need support are hatch and mamba. Those are tracked in https://py-free-threading.github.io/tracking/, so I'll close this in favor of that tracker.