pandas-dev
BUG: Quantile function only works when Categorical contains NA-values.
Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import numpy as np
#case 1
cat1 = pd.Series([np.nan, 'a', 'b', 'c', 'd', 'd']).astype(
pd.CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True))
# next line works because if-block in quantile.py (line 193)
# contains cast from float to int (result = result.astype(values.dtype, copy=False))
cat1.quantile(0.5)
#case 2
cat2 = pd.Series(['a', 'b', 'c', 'd', 'd']).astype(
pd.CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True))
# next line raises AssertionError because else-block in quantile.py (line 215)
# does NOT contain cast from float to int (result = result.astype(values.dtype, copy=False))
cat2.quantile(0.5)
Issue Description
The first case works as expected and outputs 'c'.
The second case doesn't work as expected.
I have traced this back to the lines of code 193-225 in quantile.py of Pandas 1.5.3.
When the Series contains NA-values, an if-block will be executed because mask.any() returns true (line 193-214) and this will result in an extra cast of a float value to int.: result = result.astype(values.dtype, copy=False)
In the other case, the else-block (lines 215-225) is executed, and this block doesn't contain the same cast from float to int.
In both cases, though, the underlying result of the quantile computation is correct (i.e.: 2.0 --> 'c').
Expected Behavior
The quantile function should function in all cases where the quantile value can be computed.
In the case where the quantile(0.5) function can't be computed, the same AssertionError is raised, but this not very informative. A more informative error would be warranted.
#case with extra 'a' --> has no Q2 value. s3 = pd.Series([np.nan, 'a', 'a', 'b', 'c', 'd', 'd']).astype( pd.CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True)) #next line raises sames AssertionError as in s2, but this is not very informative. cat3.quantile(0.5)
Tested solution
Change line 216 in quantile.py
return np.percentile(
values,
qs,
axis=1,
# error: No overload variant of "percentile" matches argument types
# "ndarray[Any, Any]", "ndarray[Any, dtype[floating[_64Bit]]]",
# "int", "Dict[str, str]" [call-overload]
**{np_percentile_argname: interpolation}, # type: ignore[call-overload]`
with this:
result = np.percentile(
values,
qs,
axis=1,
# error: No overload variant of "percentile" matches argument types
# "ndarray[Any, Any]", "ndarray[Any, dtype[floating[_64Bit]]]",
# "int", "Dict[str, str]" [call-overload]
**{np_percentile_argname: interpolation}, # type: ignore[call-overload]
return result.astype(values.dtype, copy=False)
Installed Versions
pyxlsb : None s3fs : None scipy : 1.10.1 snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None tzdata : None
Thank you for opening an issue @DekwoKybon. What version are you using? I run your example on my machine and it works correctly on both cases (with and without NA values). But I'm using the dev version.
Thanks @DekwoKybon... actually it's failing on the dev version too. I had a messy jupyter file.. sorry. But it works on an older version: 1.3.4., in case it's super urgent you have this case working.
Would you want to do a PR? you did all the tracking already! (no pressure :-))
@DekwoKybon Trying out your solution causes test failures, as they expect a float but your solution changes it to ints. I feel like its breaking the behaviour of the quantile function (please let me know if im wrong!)