Numpy binary incompatibility warning

[hassec]

To create a minimal reproducer (I used Python 3.11.9):

python -m venv test_nc4
source test_nc4/bin/activate
pip install netcdf4
python -c "import numpy;import warnings;warnings.filterwarnings('error');import netCDF4"

This will raise an error with: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

This isn't numpy 2.0 specific, pip install netcdf4 "numpy<2.0" will raise the same warning.

To the best of my understanding this means that the numpy that was used during the netcdf4 build is incompatible with the numpy version I'm using.

I know that this is usually just a warning, but this means I've got to manually ignore these in my pytest suite.

It's also hard to gauge for a user whether this warning is problematic. Thus I'm wondering if this can be fixed on the netcdf4 side?

[mhogan-nwra]

On my Windows and Linux platforms, this is not a warning but an actual ValueError

>>> import netCDF4
Traceback (most recent call last):
  File "C:\Users\mhogan\AppData\Local\pypoetry\Cache\virtualenvs\apex-n-5SkYZI-py3.9\lib\site-packages\IPython\core\interactiveshell.py", line 3550, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-9588a3d4fb24>", line 1, in <module>
    import netCDF4
  File "C:\Users\mhogan\AppData\Local\Programs\PyCharm Professional\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\mhogan\AppData\Local\pypoetry\Cache\virtualenvs\apex-n-5SkYZI-py3.9\lib\site-packages\netCDF4\__init__.py", line 31, in <module>
    from ._netCDF4 import *
  File "C:\Users\mhogan\AppData\Local\Programs\PyCharm Professional\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "src\\netCDF4\\_netCDF4.pyx", line 1, in init netCDF4._netCDF4
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
>>> np.__version__
Out[3]: '2.0.2'

The error disappears when I downgrade to NumPy 1.26.4

Platform specifics

Windows 11.0 Memory: 8192M Cores: 20 CPython version: 3.9.13 netCDF4 version 1.6.5

EDIT: It appears my real issue is addressed in Issue 1338. According to the discussion, the netCDF4 version built with NumPy 2 is 1.7.0. That version PyPI is yanked! I tried to update the 1.7.1.post1, but that broke my tests! So I guess I am stuck for a little while

[ChrisBarker-NOAA]

I've been seeing this warning a lot -- unfortunately, I can't tell where it's coming from. I use netCDF4 a lot, but I also use all kinds of other stuff that builds against numpy, so I'm not sure it's netCDF4's fault.

But I use conda-forge for everything, so in that case, something is getting built and pinned wriong in conda-forge.

Which is odd, as conda forge does have numpy version pinning.

But PyPi packages are more of a problem here.

Hmm -- now that I think about it, I've had those working before numpy2, so ???

Anyway, maybe not useful but one more data point.

[djhoese]

As another data point, I only see this when installing netcdf4-python 1.6.x. Installing 1.7.x seems to all be built with numpy 2.x so backwards compatibility makes it work with numpy 1.x (I think). My unrelated problem now is that installing packages from PyPI results in not being able to use compression plugins (ex. import hdf5plugin has no effect).

[ocefpaf]

@hassec what is the Python, netcdf4, and numpy versions in your vitualenv? Note that, it may be fine, regardless of the warning.

@ChrisBarker-NOAA you should not see those in conda-forge. If you do, please report in the netcdf4-feedstock.

@djhoese can you please open a new issue for the plugin problem? It seems unrelated to the discussions here.

[djhoese]

I should have noted I only had issues with PyPI packages. Letting conda install the newest compatible conda-forge packages worked as expected.

[hassec]

@ocefpaf originally used python 3.11.9 and tested with the latest numpy at that time and the latest pre 2.0 and both triggered the warning.

I can still reproduce it with a fresh env. Now using 3.12.6 and uv:

❯ uv pip install netcdf4
Resolved 4 packages in 393ms
Installed 4 packages in 57ms
 + certifi==2024.12.14
 + cftime==1.6.4.post1
 + netcdf4==1.7.2
 + numpy==2.2.0

❯ python -c "import numpy;import warnings;warnings.filterwarnings('error');import netCDF4"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/tmp/netcdf_test/.venv/lib/python3.12/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
  File "src/netCDF4/_netCDF4.pyx", line 1, in init netCDF4._netCDF4
RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

[ocefpaf]

@hassec I was able to reproduce that warning, but the test suit passed when installing that wheel locally.

We did build those wheels with numpy>=2.0.0rc1 for python >=3.9, and it does break if you downgrade numpy, indicating that the ABI is compatible with 2.0. My guess is that the warning can be ignored.

You can see from the logs that, for python 3.12, the numpy used was 2.1.2.

2024-10-22T17:49:10.7521563Z Collecting numpy (from netCDF4==1.7.2) 2024-10-22T17:49:10.7534725Z Using cached numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)

Maybe something change from 2.1 to 2.2 that trigger that warning? Sill, they should be compatible.

[hassec]

I can reproduce with numpy 2.1.2 as well:

... they should be compatible

That's fine for me for a one-off usecase, but this is exactly the kind of uncertainty that I don't want to put on a user, so I've been avoiding using this library for now, as these warnings make users uneasy.

[ocefpaf]

If warnings make your users uneasy, it will be hard for them to use any Software.

BTW, numpy is the one silencing those warnings:

https://github.com/numpy/numpy/blob/main/numpy/init.py#L320-L323

A lot of the history behind that can be read in https://github.com/numpy/numpy/issues/11788

TL;DR I don't see what we can do here to solve this. It is unlikely a problem with netCDF4.

Your options are:

1. Use the conda package, my guess is that the Cython used there does't trigger these;
2. Pay someone to build your own wheels in your own controlled environment;
3. Accept that the warnings is just a harmless warning and use the wheels that the community built.

[ChrisBarker-NOAA]

@ChrisBarker-NOAA you should not see those in conda-forge. If you do, please report in the netcdf4-feedstock.

I was getting those a lot in the numpy-2 transition, not so much anymore. And we have not only netCDF4, but a bunch of other code that's compiled against numpy, so I was never sure exactly what was triggering it.

So: nothing to be done here -- it/when it pops up again, I'll try to take a closer look.

[hassec]

BTW, numpy is the one silencing those warnings:

https://github.com/numpy/numpy/blob/main/numpy/init.py#L320-L323

Thanks, I was not aware of this. I'd argue then that the problem is actually the user code that was doing warnings.filterwarnings('error') and thus unknowingly overwriting the numpy ignores.

Let's close this, I agree that it doesn't seem to be a netcdf issue.

[veenstrajelmer]

BTW, numpy is the one silencing those warnings: https://github.com/numpy/numpy/blob/main/numpy/init.py#L320-L323

Here is a permalink so people can still find it in the future (and also today the line numbers have changed, but I assume you meant these lines): https://github.com/numpy/numpy/blob/46268caa719db95ab2198bcd619f0e3120d7c52e/numpy/init.py#L704-L707