mapply icon indicating copy to clipboard operation
mapply copied to clipboard

Support pandas 3.0

Open dxdc opened this issue 1 year ago • 5 comments

The use of array_split from NumPy is now reporting as deprecated.

https://github.com/ddelange/mapply/blob/d15007b8508cf3e0a1e53ff428db6501233d88e2/src/mapply/mapply.py#L151

With the latest numpy, it gives warnings, i.e.

'Series.swapaxes' is deprecated and will be removed in a future version:FutureWarning
'DataFrame.swapaxes' is deprecated and will be removed in a future version:FutureWarning

Some more details here: https://github.com/numpy/numpy/issues/23217 and https://github.com/numpy/numpy/issues/24889, in particular this comment: https://github.com/numpy/numpy/issues/24889#issuecomment-1895503977 for a proposed resolution.

The explanation here is that np.array_split somewhat magically works on a pandas DataFrame because the implementation of that function under the hood only uses features that happen to work the same on an array of dataframe. But, one of those is the np.swapaxes function, which when called on a DataFrame will call the swapaxes method of that DataFrame. However, this method on the DataFrame is deprecated (for a DataFrame, which is 2D it does nothing different than transpose), and so that means that once this method is removed from pandas (probably in pandas 3.0), calling np.array_split on a DataFrame will also stop working.

It's unclear if pandas (or numpy) may address this at some point?

dxdc avatar Feb 22 '24 19:02 dxdc

many thanks for the report!

ddelange avatar Feb 22 '24 20:02 ddelange

I did some digging: https://github.com/numpy/numpy/issues/24889#issuecomment-1975076215

I will keep this issue open until mapply + pandas v3 compatibility is confirmed, but I think no further action is required here.

ddelange avatar Mar 03 '24 07:03 ddelange

nice research @ddelange! is there a way to hide the warnings in the short term? e.g., something like this:

import warnings

# Filter out specific deprecation warnings
warnings.filterwarnings("ignore", message=".*Series.swapaxes is deprecated and will be removed in a future version.*")
warnings.filterwarnings("ignore", message=".*DataFrame.swapaxes is deprecated and will be removed in a future version.*")

dxdc avatar Mar 03 '24 21:03 dxdc

Released 0.1.25

github-actions[bot] avatar Mar 04 '24 15:03 github-actions[bot]

quick update: tried make test with pandas 3.0

pip install -U --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pandas

and all tests (series, dataframe, groupby) will start failing due to various reasons. see for instance https://github.com/numpy/numpy/issues/24889#issuecomment-1981503361

ddelange avatar Mar 06 '24 18:03 ddelange