BUG: DataFrame.replace fails to replace value when column contains pd.NA

[pschwientek]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame({'A': [0, 1, 2]})
print(df)
#    A
# 0  0
# 1  1
# 2  2
df['A'].replace(to_replace=2, value=99, inplace=True)
print(df)
#     A
# 0   0
# 1   1
# 2  99
df.at[0, 'A'] = pd.NA
df['A'].replace(to_replace=1, value=100, inplace=True)
print(df)
#       A
# 0  <NA>
# 1     1  <-- should be 100
# 2    99

Issue Description

Pandas replace function does not seem to work on a column if the column contains at least one pd.NA value

Expected Behavior

replace function should work even if pd.NA values are in the column

Installed Versions

INSTALLED VERSIONS

commit : 66e3805b8cabe977f40c05259cc3fcf7ead5687d python : 3.10.0.final.0 python-bits : 64 OS : Linux OS-release : 5.16.19-76051619-generic Version : #202204081339~1649696161~20.04~091f44b~dev-Ubuntu SMP PREEMPT Tu machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.3.5 numpy : 1.21.2 pytz : 2021.3 dateutil : 2.8.2 pip : 21.2.4 setuptools : 58.0.4 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : 3.0.3 lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.0.2 IPython : 7.29.0 pandas_datareader: None bs4 : None bottleneck : None fsspec : None fastparquet : None gcsfs : None matplotlib : 3.5.1 numexpr : None odfpy : None openpyxl : 3.0.9 pandas_gbq : None pyarrow : None pyxlsb : None s3fs : None scipy : 1.8.0 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None numba : None

[phofl]

Hi, thanks for your report.

Did you check on 1.4.2 and main? Because this works on both for me.

[phofl]

might need tests

[johnmantios]

Hi @phofl. I tried replicating the issue in 1.4.3 on my local machine and I got the following error:

df["A"].replace(to_replace=1, value=100, inplace=True)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/series.py", line 4960, in replace
    return super().replace(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/generic.py", line 6747, in replace
    new_data = self._mgr.replace(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 441, in replace
    return self.apply(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 683, in replace
    mask = missing.mask_missing(values, to_replace)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/missing.py", line 98, in mask_missing
    new_mask = new_mask.to_numpy(dtype=bool, na_value=False)
AttributeError: 'bool' object has no attribute 'to_numpy'

What versions do you use in your own environment? I'd be curious to know since you say it works fine for you. Mine are:

INSTALLED VERSIONS

commit : e8093ba372f9adfe79439d90fe74b0b5b6dea9d6 python : 3.10.5.final.0 python-bits : 64 OS : Darwin OS-release : 17.7.0 Version : Darwin Kernel Version 17.7.0: Fri Oct 30 13:34:27 PDT 2020; root:xnu-4570.71.82.8~1/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : el_GR.UTF-8 LOCALE : el_GR.UTF-8

pandas : 1.4.3 numpy : 1.23.0 pytz : 2022.1 dateutil : 2.8.2 setuptools : 58.1.0 pip : 22.0.4 Cython : 0.29.30 pytest : 7.1.2 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : None pandas_datareader: None bs4 : None bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None markupsafe : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None

[phofl]

Hmm good point, does not work on 1.4.2, but on main

[MikiPWata]

Hi there, I'm interested in working on this issue(this will be my first contribution). I'm guessing I'm supposed to write some unit tests related to this bug?

[NaveenKaidbettu]

Hi @phofl,

I have similar error as as @johnmantios , the issue seem to be coming even from 1.4.3 as well. The issue is being thrown on line 100 here of ..\pandas\core\missing.py. There are no accessible definition for to_numpy based on any of the imports.

[jcbedoyam]

I tried solving this issue by overloading the comparation operator __eq__ for the NA class since comparisons return again <NA> instead of a boolean. However this fails the arithmetics comparison tests. Is there a reason why the comparison for this class is implemented this way or can I rewrite the comparison tests to fit the new comparison functionality?

[phofl]

NA = NA is again NA, this happens on purpose

[pspiagicw]

Is anybody working on this ? Can I investigate ?

[Shadimrad]

NA = NA is again NA, this happens on purpose

Just out of curiosity may I ask why? @phofl

[phofl]

https://en.m.wikipedia.org/wiki/Three-valued_logic

Kleene logic

[Shadimrad]

Thank you so much!

On Mon, 8 Aug 2022, 1:36 pm Patrick Hoefler, @.***> wrote:

https://en.m.wikipedia.org/wiki/Three-valued_logic

Kleene logic

— Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/47480#issuecomment-1208585016, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWI7Q4YTNTVMBUCE6XAYK2LVYFVTZANCNFSM5ZSRIGKA . You are receiving this because you commented.Message ID: @.***>

[phofl]

Works on main

df = pd.DataFrame({'A': [0, 1, 2]}, dtype="Int64")
df.at[0, 'A'] = pd.NA
df['A'].replace(to_replace=1, value=100, inplace=True)

returns

      A
0  <NA>
1   100
2     2

[yuanx749]

I would like to take this. @phofl The bug exists if not specifying dtype in df = pd.DataFrame({"A": [pd.NA, 1, 2]}).

[AkshayJain1995]

Hi @phofl , If the issue still exists can I pick this up? Please assign this to me. Thanks!

[Lahiry]

I just did use your reproducible example and it just works fine for me. I guess you can mark this issue as closed as this is probably fixed by now. But please let me know if you continue having the issue, because I´m interested in helping!

[phofl]

We try to add tests if something was fixed without getting closed

[vs-araujo]

take

[vs-araujo]

Tested using the Docker image and the bug still exists. It seems to happen only when pd.Series has dtype object. If it contains NAs but has dtype declared as Float64 or Int64 it runs just fine.

From what I found, the problem is at the function mask_missing, at .\pandas\core\missing.py (as @NaveenKaidbettu stated). On line:

new_mask = arr == x

In this situation arr has type numpy.ndarray and x is int. The evaluation is expected to return a BooleanArray, but is returning a single bool instead and raising the exeception @johnmantios posted above. Still don't know why.

Should I send pull request with failing tests?