Add support for Numpy 2.x

[aMarcireau]

The changes are based on recommendations from https://numpy.org/devdocs/numpy_2_0_migration_guide.html#c-api-changes.

The most visible user-facing change is the addition of two feature flags (numpy-1 and numpy-2). By default, both features are enabled and the code is compiled with support for both ABI versions (with runtime checks to select the right function offsets). Functions that are only available in numpy 1 or 2 are not exposed in this case. Disabling default features (for instance numpy = {version = "0.21.0", default-features = false, features = ["numpy-1"]}) exposes version-specific functions and fields but the library will panic if the runtime numpy version does not match.

I have not done much testing, this should be tried on different code bases (ideally ones that use low-level field access) before merging.

This currently uses std::sync::OnceLock to cache the runtime version. I realised too late that this is not compatible with the Minimum Supported Rust Version (it was introduced in 1.70.0). Using pyo3::sync::GILOnceCell isn't straightforward since py is not always available in functions that need to check the version to pick an implementation.

[aMarcireau]

@adamreichold I just pushed a new version. Here's a brief overview of breaking changes.

• The struct numpy::npyffi::PyArray_Descr now only contains public fields with the same position in numpy 1 and numpy 2. The remaining fields (elsize, alignment, metadata, subarray, names, fields, and c_metadata) may be accessed with the functions PyDataType_$field, where $field is ELSIZE, ALIGNMENT and so on.
• The following functions of the struct numpy::PyArrayDescr and the trait numpy::PyArrayDescrMethods now have an extra parameter py: Python<'py> in first position: itemsize, alignment, flags, ndim, has_object, is_aligned_struct, has_subarray, and has_fields.
• The type of dtype flags is now u64 to support numpy 2. It was c_char before . Negative values were technically possible but the numpy 1 to numpy 2 back port suggests to simply cast flag values to unsigned (C) chars, which is what we do here (in the accessor function).

[Icxolu]

• The following functions of the struct numpy::PyArrayDescr and the trait numpy::PyArrayDescrMethods now have an extra parameter py: Python<'py> in first position: itemsize, alignment, flags, ndim, has_object, is_aligned_struct, has_subarray, and has_fields.

I don't think it is neccessary to change the API here. Since these are Python types we already have the proof that the GIL is held. You can just get the token via self.py(). For the trait you probably have to remove the default implementation and move them into the impl on Bound to get access to the token.

[aMarcireau]

@Icxolu Good point! I made the changes that your suggested.

@adamreichold Let me know if you would like me to squash the commits that modified src/dtype, since I ended up rolling back many of the edits.

[adamreichold]

Sorry for not getting to this yet. We are in crunch mode at $DAYJOB until next week. Will look into as soon as I can.

[aMarcireau]

Thanks @stinodego. I replicated your changes in this PR.

[adamreichold]

The dtype::tests::test_dtype_names unit test appears to fail in the CI running NumPy 2. More generally, we might want to have at least one test matrix entry that explicitly builds against NumPy 1 to keep ourselves honest in supporting both major versions.

(I am not what the macOS issue about installing Python is but I would be surprised if it were actually related to this PR. @davidhewitt Is this something that already affected PyO3 and which we could shamelessly copy here?)

[adamreichold]

(I am not what the macOS issue about installing Python is but I would be surprised if it were actually related to this PR. @davidhewitt Is this something that already affected PyO3 and which we could shamelessly copy here?)

Going to tackle this separately in #436...

[adamreichold]

@aMarcireau I merged the necessary fixes so that the CI passes on the main branch now. You should rebase and then make sure to relax the various numpy<2 requirements I added when you add one CI job to explicitly test the older NumPy version. (I would suggest doing this on a Linux runner.)

[davidhewitt]

Continued by #442. Thank you all for the efforts in this PR and its follow-up!