pandas icon indicating copy to clipboard operation
pandas copied to clipboard

BUG: identity checking NA in map incorrect

Open droussea2001 opened this issue 1 year ago • 8 comments

  • [x] closes #57390
  • [x] Tests added and passed if fixing a bug or adding a new feature
  • [ ] All code checks passed.
  • [N.A.] Added type annotations to new arguments/methods/functions.
  • [x] Added an entry in the latest doc/source/whatsnew/v3.0.0.rst file if fixing a bug or adding a new feature.

In bug #57390 Identity checking NA in map is incorrect because pd.NA are coerced to np.nan when evaluating a UDF.

This PR proposes to correct this problem in: A) Avoiding cast in object for map input B) Avoiding cast in object for map output

That's why test_map and test_map_na_action_ignore were modified in this way (we expect in this modified tests to keep pd.NA after a map)

droussea2001 avatar Apr 23 '24 20:04 droussea2001

@mroeschke: Hello, hope you're doing well, would it be possible to merge this PR or should I propose another approach ? Thanks in advance

droussea2001 avatar May 06 '24 09:05 droussea2001

@mroeschke: for information I put this PR in Draft because I try to avoid a cast in object before calling lib.map_infer in map_array

droussea2001 avatar May 20 '24 22:05 droussea2001

pre-commit.ci autofix

droussea2001 avatar May 27 '24 11:05 droussea2001

For information, the idea in this new PR version is to be able to take into account into lib.map_infer:

  • BooleanArray, FloatingArray and IntegerArray values without casting them into object before
  • pd.NA value as valid value for a UDF

That's why I add a mask parameter to lib.map_infer.

droussea2001 avatar May 27 '24 11:05 droussea2001

@WillAyd : Hello Will, I would have a question: the check "Docstring validation, typing, and other manual pre-commit hooks" is in the state "cancelled" in the CI just after the following line:

[90/152] Compiling C object pandas/_libs/join.cpython-310-x86_64-linux-gnu.so.p/meson-generated_pandas__libs_join.pyx.c.o Error: The operation was canceled.

I don't see any error (I try to rebuild it locally from scratch) and the other checks seems to be ok. Would you have an idea about what causing this problem ? Thanks in advance

droussea2001 avatar May 29 '24 14:05 droussea2001

I agree that looks strange - just restarted the job. Let's see what happens

WillAyd avatar May 29 '24 14:05 WillAyd

The error is clearer now, I will investigate. :-) Thanks a lot !

droussea2001 avatar May 29 '24 17:05 droussea2001

pre-commit.ci autofix

droussea2001 avatar Jul 25 '24 20:07 droussea2001