regression in cftime on s390

[amckinstry]

What happened?

This is on Debian, which supports the S390/s390X archs from IBM.

For the 2022.06.0 release: ======================================================================================== short test summary info ======================================================================================== FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[365_day] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-14 16:00:00 FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[360_day] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-14 16:00:00 FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[julian] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-14 16:00:00 FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[all_leap] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-14 16:00:00 FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[366_day] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-14 16:00:00 FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[gregorian] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-14 16:00:00 FAILED xarray/tests/test_accessor_dt.py::test_calendar_cftime_2D[proleptic_gregorian] - pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -809793270280017-04-1

Details are tracked here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004870

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Debian Sid cftime 1.6.1 pandas 1.3.5+dfsg python 3.10

This is in a build environment to test any possible fixes or debug.

[amckinstry]

INSTALLED VERSIONS

commit: None python: 3.10.5 (main, Jun 8 2022, 09:26:22) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.10.0-16-s390x machine: s390x processor: byteorder: big LC_ALL: None LANG: None LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.7 libnetcdf: 4.9.0

xarray: 0.16.1 pandas: 1.3.5 numpy: 1.21.5 scipy: 1.8.1 netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.2 h5py: 3.7.0 Nio: None zarr: 2.12.0 cftime: 1.6.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.0 cfgrib: 0.9.10.1 iris: None bottleneck: 1.3.2 dask: 2022.02.0+dfsg distributed: None matplotlib: 3.5.2 cartopy: 0.20.3 seaborn: 0.11.2 numbagg: None fsspec: 2022.5.0 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 59.6.0 pip3: None conda: None pytest: 7.1.2 IPython: 7.31.1 sphinx: 4.5.0

[spencerkclark]

Apologies for taking a while to look into this. I have not been able to set up an environment to reproduce these test failures, which makes it tricky. It seems like the tests are failing in the setup step, where a DataArray of some random times is generated:

data = xr.DataArray(np.random.randint(1, 1000000, size=(4, 5)).astype("<M8[h]"), dims=("x", "y"))

In principle the NumPy code should not generate any times larger than 1,000,000 hours since 1970-01-01, i.e. 2084-01-29T16:00:00, which in theory should be should be representable with a nanosecond-precision pandas Timestamp.

Trying to narrow things down, I guess my first question would be: does the following fail in this environment? Is this maybe a pandas issue?

>>> import numpy as np; import pandas as pd
>>> pd.Timestamp(np.int64(1000000).astype("<M8[h]"))
Timestamp('2084-01-29 16:00:00')

I think these tests could be simplified some to remove the randomness, but that's probably a separate issue.

[amckinstry]

The code snippet succeeds on s390x in the environment.

[spencerkclark]

Thanks for trying that. Maybe it has to do with casting to a pd.Series. Could you maybe also try:

>>> import numpy as np; import pandas as pd
>>> pd.Series(pd.Timestamp(np.int64(1000000).astype("<M8[h]")))
0   2084-01-29 16:00:00
dtype: datetime64[ns]

[headtr1ck]

Is it not possible to have a CI run with ARM and S390x architectures? Maybe that would be a good improvement.

[amckinstry]

@spencerkclark That works on s390x too (same output).

we have CI: https://ci.debian.net/packages/p/python-xarray/unstable/s390x/ https://ci.debian.net/packages/p/python-xarray/unstable/i386/

Its working ok on Arm, but failing i386 (32-bi) and s390x (64-bit).

[spencerkclark]

Thanks @amckinstry. I guess my last try to produce a pandas minimal example might be:

>>> import numpy as np; import pandas as pd
>>> pd.Series(np.array([np.int64(1000000).astype("<M8[h]")]))
0   2084-01-29 16:00:00
dtype: datetime64[ns]

or potentially more simply:

>>> import numpy as np; import pandas as pd
>>> pd.Series(np.int64(1000000).astype("<M8[h]"))
0   2084-01-29 16:00:00
dtype: datetime64[ns]

Somewhere something is going wrong in converting a non-nanosecond-precision datetime value to a nanosecond-precision one (maybe the cast to a pd.Timestamp in my earlier example was short-circuiting this).

I think #6988 should likely work around this issue at least on the xarray side, since it passes datetime64[ns] values into the DataArray constructor immediately. It also seems like the function where the error occurs (ensure_datetime64ns) was recently eliminated in favor of an updated implementation in pandas, so I wonder if this will be an issue there going forward.

[amckinstry]

Both code fragments work on s390x as described. This is for pandas 1.4.3 and xarray 2022.06.0. The commit in #6988 solves the issue.

[spencerkclark]

Interesting. Thanks for checking that #6988 indeed solves this. I went ahead and merged it, but when I get a chance I’ll keep trying to track down the root cause of this issue.

[spencerkclark]

I was able to reproduce this issue in a Docker container using the s390x Debian image. After a little experimentation I narrowed it down to the following minimal example:

>>> import numpy as np; import pandas as pd
>>> np.__version__
'1.23.3'
>>> pd.__version__
'1.4.4'
>>> pd.Series(np.array([1]).astype("<M8[h]"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/series.py", line 451, in __init__
    data = sanitize_array(data, index, dtype, copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/construction.py", line 570, in sanitize_array
    subarr = _try_cast(data, dtype, copy, raise_cast_failure)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/construction.py", line 729, in _try_cast
    return sanitize_to_nanoseconds(arr, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1717, in sanitize_to_nanoseconds
    values = conversion.ensure_datetime64ns(values)
  File "pandas/_libs/tslibs/conversion.pyx", line 257, in pandas._libs.tslibs.conversion.ensure_datetime64ns
  File "pandas/_libs/tslibs/np_datetime.pyx", line 120, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 8220291319602-05-05 16:00:00

This confirms it is an upstream issue. Interestingly if we use the native byte order (big-endian on this architecture) for the dtype, this example works:

>>> pd.Series(np.array([1]).astype("M8[h]"))
0   1970-01-01 01:00:00
dtype: datetime64[ns]

or more explicitly

>>> pd.Series(np.array([1]).astype(">M8[h]"))
0   1970-01-01 01:00:00
dtype: datetime64[ns]

It appears the inverse of this issue (big-endian dtype leading to a failure on a little-endian system) came up once in pandas: https://github.com/pandas-dev/pandas/issues/29684. @amckinstry I'm not sure what it will take to fix this issue in pandas, but you are welcome to open an issue there. They may also have a difficult time reproducing and testing this, however (https://github.com/pandas-dev/pandas/pull/30976#issuecomment-573989082).