pandas icon indicating copy to clipboard operation
pandas copied to clipboard

BUG: `isin` propagates nulls for DataFrame but not Series

Open brandon-b-miller opened this issue 3 years ago • 2 comments

Pandas version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of pandas.

  • [X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# dataframe with an Int64 column and a missing value
df = pd.DataFrame(
    {
        'a': pd.Series([1,2,pd.NA], dtype='Int64')
    }
)

values = pd.Series([1,2,3], dtype='Int64')

print(df.isin(values)) # [True, True, <NA>]

# equivalent series
sr = pd.Series([1,2,pd.NA], dtype='Int64')

print(sr.isin(values)) # [True, True, False]

Issue Description

The null propagation behavior for isin differs depending on if the calling object is a Series or a DataFrame, should they be the same? e.g. should the series isin call return a null in the last position?

Expected Behavior

I would expect to get a null at index 2 in both cases.

Installed Versions

INSTALLED VERSIONS

commit : 66e3805b8cabe977f40c05259cc3fcf7ead5687d python : 3.8.12.final.0 python-bits : 64 OS : Linux OS-release : 4.15.0-76-generic Version : 86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.3.5 numpy : 1.21.5 pytz : 2021.3 dateutil : 2.8.2 pip : 22.0.3 setuptools : 59.8.0 Cython : 0.29.27 pytest : 7.0.0 hypothesis : 6.36.1 sphinx : 4.4.0 blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.0.3 IPython : 8.0.1 pandas_datareader: None bs4 : 4.10.0 bottleneck : None fsspec : 2022.01.0 fastparquet : None gcsfs : None matplotlib : 3.5.1 numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 6.0.1 pyxlsb : None s3fs : None scipy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None numba : 0.55.1

brandon-b-miller avatar Mar 11 '22 16:03 brandon-b-miller

This was discussed in https://github.com/pandas-dev/pandas/pull/38379#issuecomment-757510449

But no definite solution I think

phofl avatar Mar 11 '22 16:03 phofl

cc @jorisvandenbossche

Thoughts here?

If we don't want NA to propagate, then the Series case is wrong

phofl avatar Aug 05 '22 21:08 phofl