NumPy 2.0 development status & announcements

[rgommers]

trafficstars

The purpose of this issue is to serve as a brief "umbrella issue" which (a) links out to some key design proposals and other places where design changes and guidance for the 2.0 release are described, and (b) everyone who is interested can subscribe to in order to get updates from maintainers around the 2.0 release (not everyone may want to subscribe to the mailing list). This issue will remain pinned for high visibility.

The tentative release date for the first release candidate of NumPy 2.0 is around 1 Mar 2024, and the final release 6-8 weeks later.

EDIT: this issue is for announcements, please do not start technical discussions here

Design changes / proposals

• NumPy 2.0 Project Board: https://github.com/orgs/numpy/projects/9

• Main NumPy Enhancement Proposals for 2.0:

  • NEP 53 - Evolving the NumPy C-API for NumPy 2.0
  • NEP 52 - Python API cleanup for NumPy 2.0
  • NEP 50 - Promotion rules for Python scalars

• Slides from the NumPy 2.0 Developer Meeting in April'23: https://github.com/numpy/archive/tree/main/2.0_developer_meeting

Key guidance for users and downstream package authors

1. Please review the 2.0 migration guide. And for a more detailed overview of changes, please see the 2.0.0 release notes
2. If you rely on the NumPy C API (e.g. via direct use in C/C++, or via Cython code that uses NumPy), please add a numpy<2.0 requirement in your package's dependency metadata (for releases only, not on your main dev branch). Rationale: the NumPy C ABI will change in 2.0, so any compiled extension modules that rely on NumPy are likely to break, they need to be recompiled.
3. If you rely on a large API surface from NumPy's Python API, also consider adding the same numpy<2.0 requirement to your metadata. Rationale: we will do a significant cleanup (see NEP 52), so unless you only use modern/recommended functions and object, your code is likely to require at least some adjustments.
4. Consider cleaning up your code. E.g. remove from numpy import *, or importing any private modules like numpy.core. See https://github.com/numpy/numpy/blob/main/numpy/tests/test_public_api.py#L114-L126 for what we consider public/private. If it's not in the NumPy docs or in the list of public modules there, don't use it!
5. Plan to do a release of your own packages which depend on numpy shortly after the first NumPy 2.0 release candidate is released (probably in Dec 2023). Rationale: at that point, you can release packages that will work with both 2.0 and 1.X, and hence your own end users will not be seeing much/any disruption (you want pip install mypacackage to continue working on the day NumPy 2.0 is released).
6. Consider testing against NumPy nightlies in your own CI. We publish those at https://anaconda.org/scientific-python-nightly-wheels/numpy, and have documented that as a stable location at https://numpy.org/devdocs/dev/depending_on_numpy.html. Rationale: this will detect potential issues in your code so you can fix them well ahead of the NumPy 2.0 release.

Compatibility status of downstream packages

See https://github.com/numpy/numpy/issues/26191

[seberg]

@andyfaff in an attempt to prep for a next step where we do meaningful ABI breaks (changing the default integer may be that already). Could we get SciPy to upload nightlies that are build against the NumPy nightlies? They don't have to pass/run the tests with it, but would have to use it at build time to get newer headers.

The next in line might be matplotlib, but SciPy doing it would go a long way I suspect.

Example changes:

• @mtsokol might want to move the import, it would be nice to avoid the warning downstream (although probably not vital).
• Opaquify the PyArray_Descr struct. This requires version sensitive macros for accessing the fields (We can probably retain most fields position, but I don't think all).
• The int64 on windows adds a way to get the default integer (new, version sensitive macro). You will want to use it with an #ifdef, but if it is used anywhere the right thing would be to compile with new NumPy.

[andyfaff]

I don't know what the scipy plan is to handle that transition. i.e. when do we need to make a release that only builds against numpy2.0 and newer. I'd have to look into how to do it, I wouldn't want to overly complicate the existing wheel build (too many conditionals make it harder to handle). I wonder if making another workflow file would be easier?

@rgommers, could we handle making all nightlies against numpy 2.0, or do we need to keep building against already released numpy?

[rgommers]

It depends on whether those scipy nightlies will then be ABI-compatible with both numpy 2.0-dev and with 1.25.x. That is the plan for the final 2.0 release, but will it be the case during the development cycle @seberg?

[seberg]

Yes, of course they will be ABI compliant to both. I am not really planning on a compat package anymore anyway, I suspect we have few enough things that a few header-only functions are sufficient (and far easier to manage).

[andyfaff]

e.g. at the moment the scipy pyproject.toml file will want to build against numpy==1.22.4 for Python 3.10 wheels, that's what the last scipy would've been built against. If we start making wheels against numpy>=2.0 then I think, but am not sure, that the numpy versionw in pyproject.toml would need to be bumped to the oldest ABI compatible version (1.25.2?).

I know how to amend the build process, but am not sure how to amend the metadata (oldest-supported-numpy as well?) so that we don't get a raft of complaints from people using too old a version of numpy.

[seberg]

You don't need oldest-supported-numpy anymore (since 1.25.x) unless you need it for things beyond being compatible with older NumPy versions.

[rgommers]

Great, then in the nightlies we can just use an unpinned numpy build requirement, and install numpy from the scientific-python nightlies bucket, that should work fine.

[rgommers]

@andyfaff that'd be a change that is fine to make in SciPy's main branch. There are already notes in pyproject.toml about this topic.

This is a pretty detailed discussion for this tracking issue. How about we leave it here at "SciPy nightlies will switch to building against NumPy nightlies soon" and then circle back here once that works to recommend other projects do the same?

[jakirkham]

Following up on the discussion above, are there now SciPy nightlies being built with NumPy 2? If so, how would one get them? If not, do we have a sense of when they might be available?

[andyfaff]

@jakirkham yes, the scipy nightlies are being built against numpy>=2.0.0.dev0. They are available from https://anaconda.org/scientific-python-nightly-wheels/scipy/files.

[jakirkham]

Thanks Andrew! 🙏

[EwoutH]

Plan to do a release of your own packages which depend on numpy shortly after the first NumPy 2.0 release candidate is released (probably in Dec 2023).

Would it be useful to do a bit more extensive pre-release schedule than with a regular (minor) NumPy release? Maybe alpha builds every 2 or 3 weeks starting soon, then a month or two ahead every week a beta, and then the last week release candidates.

Since it's such a large release it would be nice being able to test extensively against pre-releases, and having a bit more time for that. It would also engage other communities and maintainers which could offer valuable perspectives before the final release is made.

[andyfaff]

There are regular nightly builds which should aid in prerelease testing

[EwoutH]

There are regular nightly builds which should aid in prerelease testing

Agreed, I think there is additional value for "formal" pre-releases, which are published on GitHub and PyPI, can be installed with pip install numpy --pre, and feedback can be delivered on. It gives a concrete and easy targets to install and test against.

[rgommers]

We don't want to do that, certainly not alpha's - and probably no beta's either, only RCs. A lot of things are in flux, may be reverted, etc. We're in the middle of ABI changes, and having any pre-release up on PyPI before the ABI changes are complete may be quite disruptive, both immediately and because pip may cache wheels with an intermediate state of the ABI.

Until we're ready for wider testing by anyone who may have --pre jobs, interested folks should test against https://anaconda.org/scientific-python-nightly-wheels/numpy.

[seberg]

Upper version pins in requires biting things again... The recommendation makes sense for releases (especially if the releases are regular), but it is no good for development versions.

There are two plausible views, I can see:

• Only release versions should have an upper version pin (unless you know its broken). Dev branches should not (for many packages it may be better to risk releasing without than vice versa).
• Use numpy<=2.0.0.dev0 specifically allowing trying to use our nightlies.

I am tempted to adjust the recommendation to numpy<=2.0.0.dev0 as a safe and workable path for packages who feel more comfortable adding an upper version pin to ensure that official release versions have one.

[rgommers]

An update on timeline: there are ~6 topics on the roadmap that still require time and are important to get in for 2.0. At the community meeting two days ago we discussed that we'll likely need another 2 months or so to get those all done and are therefore targeting a first release candidate around the end of the year. The period from RC to final release is at least 6 weeks as usual, but it's possible that that will stretch out more - that depends on how many issues get reported.

[thomasjpfan]

I agree with adjusting the pinning recommendation from https://github.com/numpy/numpy/issues/24300#issuecomment-1770748369. In https://github.com/scikit-learn/scikit-learn/pull/27899, scikit-learn needed to explicitly install NumPy 2.0 from the nightly index because pandas had numpy<2.0 on the dev branch.

[rgommers]

numpy<=2.0.0.dev0 in released versions of downstream packages does not make sense to me. We have already broken ABI and API pretty substantially, so if a package builds against 1.2X and then pick up the latest 2.0.0.dev0 that will not work.

Only release versions should have an upper version pin (unless you know its broken). . ... because pandas had numpy<2.0 on the dev branch.

Agreed. It seems like this needs a "for releases only" addition - adding now. That was clearly intended, one should never pin anything on their main branch unless there is a known source-level incompatibility.

[braingram]

Thanks all for your hard work on this. Should

The tentative release date for NumPy 2.0 is end of December 2023.

be updated based on https://github.com/numpy/numpy/issues/24300#issuecomment-1783511140 to state that the first RC will be end of Dec with the release expected in early Feb 2024?

[rgommers]

Thanks @braingram, updated now.

[rgommers]

Hi all, one more update on timing. We just reviewed all the remaining work items that are deemed important for NumPy 2.0 in the community meeting - they're all in progress or close to the finish line and we're reasonably confident that in one month from now they'll all be done and we can have the first release candidate at the end of January. So this moves the timeline by 1 month.

[jakevdp]

Hi @rgommers – I'm having trouble squaring the advice above:

at that point, you can release packages that will work with both 2.0 and 1.X, and hence your own end users will not be seeing much/any disruption (you want pip install mypacackage to continue working on the day NumPy 2.0 is released).

... with the general documentation on ABI compatibility (from https://numpy.org/doc/1.21/user/depending_on_numpy.html#build-time-dependency)

Because the NumPy ABI is only forward compatible, you must build your own binaries (wheels or other package formats) against the lowest NumPy version that you support (or an even older version).

My experience so far is that if packages are built against 1.X, they don't work with numpy nightly wheels because of a hard-coded ABI version check. But if I build against 2.X, the Build-time dependency docs quoted above suggest that we shouldn't expect the result to be compatible with numpy 1.X at runtime.

How should downstream package authors plan to build wheels for release to ensure compatibility with both NumPy 2.0 and NumPy 1.X?

[rgommers]

My experience so far is that if packages are built against 1.X, they don't work with numpy nightly wheels because of a hard-coded ABI version check.

Yes, this will not work indeed.

But if I build against 2.X, the Build-time dependency docs quoted above suggest that we shouldn't expect the result to be compatible with numpy 1.X at runtime.

(could you update the link? it goes to something private)

The summary here is:

• This has always been true, however there was a change in NumPy 1.25 where the exported C API is no longer that for the current NumPy version but a much older one (1.19), so things got better compat wise.
• In addition, 2.0 has a compat shim that bridges the ABI differences, in order to make wheels built against 2.0 also work with 1.X by default.

How should downstream package authors plan to build wheels for release to ensure compatibility with both NumPy 2.0 and NumPy 1.X?

Build against 2.0 after the ABI-stable 2.0.0rc1 arrives, and declare a runtime dependency of >=1.2x.y (any x.y here should work).

[jakevdp]

Link updated, sorry about that

[rgommers]

https://numpy.org/doc/1.26/dev/depending_on_numpy.html#build-time-dependency should have more info on the "exported C API version" topic already than those 1.21 docs.

The 2.0 compat thing isn't anywhere yet except for in this issue I think. We should add it to that same "for downstream distributors" page, and then link it from https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

[jakevdp]

Ah, thanks. I just found the 1.21 docs via google search; I hadn't seen the 1.26 update.

[bashtage]

Is there any documentation about how to publish NumPy 2 wheels for downstream packages. As I understand things, packages might need 2 versions - 1 for NumPy <= 1.99 and a second for NumPy >= 2, for each version/ABI supported on PyPI. Will this be possible?

Edit: Just read some I missed above, and it sounds like any build against NumPy 2 will also work on NumPy >= 1.20.x, assuming that other API changes have been accounted for (e.g., the removal of issubsctype which has no Deprecation/FutureWarning).

FWIW the removal of issubsctype will affect some less maintained packages, e.g., patsy < 0.5.5.

[rgommers]

it sounds like any build against NumPy 2 will also work on NumPy >= 1.20.x,

Indeed - and it has to be that way (it's not possible to upload two wheels where the only difference is the numpy version used during the build). No docs yet though, we should fix that before RC1 comes out.

[seberg]

It's been documented since 1.25: https://numpy.org/doc/stable/dev/depending_on_numpy.html#build-time-dependency Maybe you think it should be clarified, otherwise it should just be mentioned again in the release notes wherever it says that you must compile with 2.0.