xcdat
xcdat copied to clipboard
xCDAT
[Bug]: xESMF related issue "AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead."
What happened?
xESMF is conflicting with numpy>=2.0, which is breaking the GitHub Actions build (here).
I opened an issue in the xESMF repo here: https://github.com/pangeo-data/xESMF/issues/370
What did you expect to happen? Are there are possible answers you came across?
Related code in xESMF: https://github.com/pangeo-data/xESMF/blob/530b804c28a9b4f64dd360384897c4c2b34ab8c3/xesmf/smm.py#L229
- [x] Update conda envs to constrain
numpy <2.0 - [x] Update conda-forge feedstock recipe with same constraint (link)
Minimal Complete Verifiable Example (MVCE)
No response
Relevant log output
=================================== FAILURES ===================================
________________________ TestXESMFRegridder.test_regrid ________________________
self = <tests.test_regrid.TestXESMFRegridder object at 0x7f78dd5a7190>
def test_regrid(self):
ds = self.ds.copy()
regridder = xesmf.XESMFRegridder(ds, self.new_grid, "bilinear")
> output = regridder.horizontal("ts", ds)
tests/test_regrid.py:734:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xcdat/regridder/xesmf.py:158: in horizontal
regridder = xe.Regridder(
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/frontend.py:928: in __init__
super().__init__(
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/frontend.py:385: in __init__
self.weights = add_nans_to_weights(self.weights)
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/smm.py:229: in add_nans_to_weights
m.data[krow] = [np.NaN] if m.data[krow] == [] else m.data[krow]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
attr = 'NaN'
def __getattr__(attr):
# Warn for expired attributes
import warnings
if attr == "linalg":
import numpy.linalg as linalg
return linalg
elif attr == "fft":
import numpy.fft as fft
return fft
elif attr == "dtypes":
import numpy.dtypes as dtypes
return dtypes
elif attr == "random":
import numpy.random as random
return random
elif attr == "polynomial":
import numpy.polynomial as polynomial
return polynomial
elif attr == "ma":
import numpy.ma as ma
return ma
elif attr == "ctypeslib":
import numpy.ctypeslib as ctypeslib
return ctypeslib
elif attr == "exceptions":
import numpy.exceptions as exceptions
return exceptions
elif attr == "testing":
import numpy.testing as testing
return testing
elif attr == "matlib":
import numpy.matlib as matlib
return matlib
elif attr == "f2py":
import numpy.f2py as f2py
return f2py
elif attr == "typing":
import numpy.typing as typing
return typing
elif attr == "rec":
import numpy.rec as rec
return rec
elif attr == "char":
import numpy.char as char
return char
elif attr == "array_api":
raise AttributeError("`numpy.array_api` is not available from "
"numpy 2.0 onwards")
elif attr == "core":
import numpy.core as core
return core
elif attr == "strings":
import numpy.strings as strings
return strings
elif attr == "distutils":
if 'distutils' in __numpy_submodules__:
import numpy.distutils as distutils
return distutils
else:
raise AttributeError("`numpy.distutils` is not available from "
"Python 3.12 onwards")
if attr in __future_scalars__:
# And future warnings for those that will change, but also give
# the AttributeError
warnings.warn(
f"In the future `np.{attr}` will be defined as the "
"corresponding NumPy scalar.", FutureWarning, stacklevel=2)
if attr in __former_attrs__:
raise AttributeError(__former_attrs__[attr])
if attr in __expired_attributes__:
> raise AttributeError(
f"`np.{attr}` was removed in the NumPy 2.0 release. "
f"{__expired_attributes__[attr]}"
)
E AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead.
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/numpy/__init__.py:397: AttributeError
___________________ TestXESMFRegridder.test_preserve_bounds ____________________
self = <tests.test_regrid.TestXESMFRegridder object at 0x7f78dd3af7d0>
def test_preserve_bounds(self):
ds = fixtures.generate_dataset(
decode_times=True, cf_compliant=False, has_bounds=True
)
ds = ds.drop_vars(["lat_bnds", "lon_bnds"])
regridder = xesmf.XESMFRegridder(ds, self.new_grid, method="bilinear")
> output = regridder.horizontal("ts", ds)
tests/test_regrid.py:793:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xcdat/regridder/xesmf.py:158: in horizontal
regridder = xe.Regridder(
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/frontend.py:928: in __init__
super().__init__(
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/frontend.py:385: in __init__
self.weights = add_nans_to_weights(self.weights)
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/smm.py:229: in add_nans_to_weights
m.data[krow] = [np.NaN] if m.data[krow] == [] else m.data[krow]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
attr = 'NaN'
def __getattr__(attr):
# Warn for expired attributes
import warnings
if attr == "linalg":
import numpy.linalg as linalg
return linalg
elif attr == "fft":
import numpy.fft as fft
return fft
elif attr == "dtypes":
import numpy.dtypes as dtypes
return dtypes
elif attr == "random":
import numpy.random as random
return random
elif attr == "polynomial":
import numpy.polynomial as polynomial
return polynomial
elif attr == "ma":
import numpy.ma as ma
return ma
elif attr == "ctypeslib":
import numpy.ctypeslib as ctypeslib
return ctypeslib
elif attr == "exceptions":
import numpy.exceptions as exceptions
return exceptions
elif attr == "testing":
import numpy.testing as testing
return testing
elif attr == "matlib":
import numpy.matlib as matlib
return matlib
elif attr == "f2py":
import numpy.f2py as f2py
return f2py
elif attr == "typing":
import numpy.typing as typing
return typing
elif attr == "rec":
import numpy.rec as rec
return rec
elif attr == "char":
import numpy.char as char
return char
elif attr == "array_api":
raise AttributeError("`numpy.array_api` is not available from "
"numpy 2.0 onwards")
elif attr == "core":
import numpy.core as core
return core
elif attr == "strings":
import numpy.strings as strings
return strings
elif attr == "distutils":
if 'distutils' in __numpy_submodules__:
import numpy.distutils as distutils
return distutils
else:
raise AttributeError("`numpy.distutils` is not available from "
"Python 3.12 onwards")
if attr in __future_scalars__:
# And future warnings for those that will change, but also give
# the AttributeError
warnings.warn(
f"In the future `np.{attr}` will be defined as the "
"corresponding NumPy scalar.", FutureWarning, stacklevel=2)
if attr in __former_attrs__:
raise AttributeError(__former_attrs__[attr])
if attr in __expired_attributes__:
> raise AttributeError(
f"`np.{attr}` was removed in the NumPy 2.0 release. "
f"{__expired_attributes__[attr]}"
)
Warning: yaksa: 10 leaked handle pool objects
E AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead.
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/numpy/__init__.py:397: AttributeError
_________________________ TestAccessor.test_horizontal _________________________
self = <tests.test_regrid.TestAccessor object at 0x7f78dd3f4ed0>
def test_horizontal(self):
output_grid = grid.create_gaussian_grid(32)
> output_data = self.horizontal_ds.regridder.horizontal(
"ts", output_grid, tool="xesmf", method="bilinear"
)
tests/test_regrid.py:1161:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xcdat/regridder/accessor.py:205: in horizontal
output_ds = regridder.horizontal(data_var, self._ds)
xcdat/regridder/xesmf.py:158: in horizontal
regridder = xe.Regridder(
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/frontend.py:928: in __init__
super().__init__(
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/frontend.py:385: in __init__
self.weights = add_nans_to_weights(self.weights)
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/xesmf/smm.py:229: in add_nans_to_weights
m.data[krow] = [np.NaN] if m.data[krow] == [] else m.data[krow]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
attr = 'NaN'
def __getattr__(attr):
# Warn for expired attributes
import warnings
if attr == "linalg":
import numpy.linalg as linalg
return linalg
elif attr == "fft":
import numpy.fft as fft
return fft
elif attr == "dtypes":
import numpy.dtypes as dtypes
return dtypes
elif attr == "random":
import numpy.random as random
return random
elif attr == "polynomial":
import numpy.polynomial as polynomial
return polynomial
elif attr == "ma":
import numpy.ma as ma
return ma
elif attr == "ctypeslib":
import numpy.ctypeslib as ctypeslib
return ctypeslib
elif attr == "exceptions":
import numpy.exceptions as exceptions
return exceptions
elif attr == "testing":
import numpy.testing as testing
return testing
elif attr == "matlib":
import numpy.matlib as matlib
return matlib
elif attr == "f2py":
import numpy.f2py as f2py
return f2py
elif attr == "typing":
import numpy.typing as typing
return typing
elif attr == "rec":
import numpy.rec as rec
return rec
elif attr == "char":
import numpy.char as char
return char
elif attr == "array_api":
raise AttributeError("`numpy.array_api` is not available from "
"numpy 2.0 onwards")
elif attr == "core":
import numpy.core as core
return core
elif attr == "strings":
import numpy.strings as strings
return strings
elif attr == "distutils":
if 'distutils' in __numpy_submodules__:
import numpy.distutils as distutils
return distutils
else:
raise AttributeError("`numpy.distutils` is not available from "
"Python 3.12 onwards")
if attr in __future_scalars__:
# And future warnings for those that will change, but also give
# the AttributeError
warnings.warn(
f"In the future `np.{attr}` will be defined as the "
"corresponding NumPy scalar.", FutureWarning, stacklevel=2)
if attr in __former_attrs__:
raise AttributeError(__former_attrs__[attr])
if attr in __expired_attributes__:
> raise AttributeError(
f"`np.{attr}` was removed in the NumPy 2.0 release. "
f"{__expired_attributes__[attr]}"
)
E AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead.
/usr/share/miniconda3/envs/xcdat_ci/lib/python3.11/site-packages/numpy/__init__.py:397: AttributeError
Anything else we need to know?
No response
Environment
Latest xCDAT, numpy>=2.0 and xesmf=0.8.5
This issue was recently fixed in xESMF by https://github.com/pangeo-data/xESMF/pull/373. We need to wait for a new release of xesmf for this to be carried over, although we still might want to constrain numpy <2.0.0 for at least another year maybe?
Wow, existing xcdat users having to wait an extra year to get a major numpy update seems quite drastic
Worse, it might prevent potential new xcdat users to install xcdat if they try to install it and conda warns them that they have to downgrade numpy (and possibly other packages)
[UPDATE]
I have experimented a bit, and updated two test environments on two different servers. I don't know how dependencies and priorities work, but I ended up in both cases with numpy 2.0.0 and xcdat 0.7.0, and conda tells me All requested packages already installed when I run conda update -n my_env --all again
It's only when I ask conda to update only xcdat that conda warns me about a numpy downgrade. So it seems that numpy users are safe by default, and that the only risk is getting stuck with an old version of xcdat
$ conda update -n my_env xcdat
Channels:
- conda-forge
- defaults
Platform: linux-64
[...]
The following packages will be UPDATED:
xcdat 0.7.0-pyhd8ed1ab_0 --> 0.7.1-pyhd8ed1ab_0
The following packages will be DOWNGRADED:
numpy 2.0.0-py312h22e1c76_0 --> 1.26.4-py312heda63a1_0
Proceed ([y]/n)? n
Wow, existing
xcdatusers having to wait an extra year to get a majornumpyupdate seems quite drastic
I wonder if a year was sufficient because numpy 2.0 is a major release with many breaking changes (link). It usually takes time for software to become stable after a major release and other packages that depend on that software need to migrate over.
In our case, it looks like xESMF might be the only xCDAT dependency that breaks with numpy 2.0 which would mean we don't need to a year to remove the constraint.
Turns out that one of our post-doc reported today that regridding in xesmf (0.8.6) was not working anymore, and it worked again when I downgraded numpy from 2.0.0 to 1.26.4. This was not an np.NaN error
An intern had the same problem last Friday. I have also downgraded his numpy and asked him if this fixed the problem