[BUG]: latest pybind11 (2.11.1) version not supporting numpy 2.0

[rgommers]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.11.1

Problem description

Hi pybind11 devs, I'd like to request a new release to support numpy 2.0. The first release candidate for numpy 2.0 is expected in early February. The underlying issue for 2.0 support was addressed ~3 months ago in gh-4857.

Right now, numpy is trying to warn users testing a nightly or main that there is an issue when digging into private internals of numpy, and the message specifically mentions pybind11 not being compatible (most probably root cause):

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0.dev0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled against NumPy 2.0.

If you are a user of the module, the easiest solution will be to
either downgrade NumPy or update the failing module (if available).

NOTE: When testing against pre-release versions of NumPy 2.0
or building nightly wheels for it, it is necessary to ensure
the NumPy pre-release is used at build time.
The main way to ensure this is using no build isolation
and installing dependencies manually with NumPy.
For cibuildwheel for example, this may be achieved by using
the flag to pip:
    CIBW_BUILD_FRONTEND: pip; args: --no-build-isolation
installing NumPy with:
    pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
in the `CIBW_BEFORE_BUILD` step.  Please compare with the
solutions e.g. in astropy or matplotlib for how to make this
conditional for nightly wheel builds using expressions.
If you do not worry about using pre-releases of all
dependencies, you can also use `--pre --extra-index-url` in the
build frontend (instead of build isolation).
This will become unnecessary as soon as NumPy 2.0 is released.

If your dependencies have the issue, check whether they
have nightly wheels build against NumPy 2.0.

pybind11 note: You may see this message if using pybind11,
this is not problematic at pre-release time
it indicates the need for a new pybind11 release.

After that message, the offending code may yield an exception (happens for, e.g., Matplotlib) or silently continue like nothing else happened (happens for, e.g., Dask). The latter is especially confusing - some dependency of Dask is using pybind11, but it's not even possible to see which one (dask itself is pure Python).

Once numpy 2.0.0rc1 appears on PyPI, a lot more users are going to see the above message and be confused, or come to this issue tracker. The frequency will go up both because a lot of packages use --pre in their CI and because the numpy team will strongly encourage users to test that RC. A new release containing gh-4857 soon would be very helpful.

EDIT: additional reason: other libraries need to release new versions with wheels that work with numpy 2.0 - and if there's no new pybind11 release by then, it looks like they have to use pybind11 for their next releases. That's really undesirable.

Reproducible example code

This issue can be reproduced with for example a new environment in which one installs the latest `matplotlib` release from PyPI plus a `numpy` nightly:


$ mamba create -n pybind11-check python=3.11
$ mamba activate pybind11-check
$ pip install matplotlib
$ pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy -U
$ python -c "import matplotlib


A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0.dev0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled against NumPy 2.0.
...
pybind11 note: You may see this message if using pybind11,
this is not problematic at pre-release time
it indicates the need for a new pybind11 release.
...


As soon as numpy `2.0.0rc1` appears on PyPI, the reproducer will be simpler: `pip install numpy matplotlib --pre` is all it would take.

Is this a regression? Put the last known working version here if it is.

Not a regression

[rgommers]

Thanks for having a look @Skylion007. It's not entirely clear what "help wanted" means here - a non-maintainer can't really help with doing a new release I suspect. I did test the master branch of this repo with SciPy a while back, and that all looked good.

[seberg]

I will note that there is one serious change that pybind11 will notice and will have to act on. And that is that we really need to change the PyArray_Descr struct for NumPy 2. It has some issues which make future extension unnecessary hard or impossible, which would be a shame with new user DTypes having much more possibilities now.

This isn't a particularly difficult change of course, but it will require a version dependent use of a different struct to look up some of the members.

(Direct access of this struct isn't very common in downstream outside pybind11 since you would rather use macros like PyArray_ITEMSIZE(), but pybind11 doesn't use that API.)

[henryiii]

Any ETA on the RC? That would make it easier to test with.

I really wish our NumPy support was pulled out into it's own module, that would have allowed a potential alternative module for NumPy 2 using the API instead of having to work with the integrated one, but it is what it is.

[seberg]

I am working on the last steps for the most interesting changes and I think I am making decent progress. I Will ping you when I have a PR that includes the main breakage (should in 1-2 days, also because then I am away for a while after that).

We will need to see how this affects Scipy. I hope it just doesn't matter in practice and you can basically fix it with some leisure during the rc period.

[seberg]

Sorry that it took a bit, there was a bit of a back and forth and I was on a long planned vacation. In either case, this would be the change: https://github.com/numpy/numpy/pull/25943. The most important point is that .itemsize() needs to check for the NumPy version. I think that isn't particularly hard and hopefully will create a PR tomorrow to get started.

The important thing is that it would be great to get this, and just as important the import location change, into a new release soon (and if it is a --pre one). While many users will probably not notice the changes, some may (I added a few hacks in SciPy to make it work).

[sasidhara-kashyap0903]

Can I contribute to this issue?

[seberg]

I have created gh-5050 with the necessary fixes. Some users might be unhappy (if they use structured dtypes, which I doubt many will, but you never know) so we might have to follow up eventually with adding helpers for them.

Otherwise, I think it should be fine, I slightly hoping that there is no major reason not to stick with the current pybind11 release for those few who cannot adopt NumPy 2 quickly.

[rgommers]

Fixed by gh-5050, which is included in the just-released pybind11 2.12.0 release. Thanks everyone!