ENH: Processing of .mask() for pd.NA #56844 #58730
Conversation
This commit addresses an issue where using pandas.NA in conjunction with the mask() method resulted in unexpected behavior. The problem arose when comparing a Series containing pandas.NA with a condition within mask(), causing inconsistencies in the output. This fix ensures that pandas.NA behaves consistently with False in logical operations within mask().
Co-authored-by: Xiao Yuan <[email protected]>
This commit addresses an issue where using pandas.NA in conjunction with the mask() method resulted in unexpected behavior. The problem arose when comparing a Series containing pandas.NA with a condition within mask(), causing inconsistencies in the output. This fix ensures that pandas.NA behaves consistently with False in logical operations within mask().
Co-authored-by: Xiao Yuan <[email protected]>
| @@ -9999,6 +9999,9 @@ def mask( | |||
| cond = common.apply_if_callable(cond, self) | |||
| other = common.apply_if_callable(other, self) | |||
|
|
|||
| if isinstance(cond, (ABCDataFrame, ABCSeries)): | |||
There was a problem hiding this comment.
Rather than a fillna like this do you see where the logic is breaking down that causes .mask to treat pd.NA the same was as True? We shouldn't have to mess around with pd.NA equality semantics for this fix
There was a problem hiding this comment.
For indexing this goes through check_array_indexer which essentially also converts NA to False (but lower in the stack):
pandas/pandas/core/indexers/utils.py
Lines 532 to 534 in 2aa155a
mroeschke
added
the
Missing-data
|
@phofl @jorisvandenbossche this seems related to part of the ice cream agreement. Are we ready to move forward with that part? I’d expect to do the indexing bits all at once |
|
@jbrockmendel I think we can simply treat this as a bug fix of the current implementation. We do handle the nullable boolean dtype specifically in indexing already (considering NA as False, a behaviour which has been discussed in the past already, #31591), so fixing that for |
Co-authored-by: William Ayd <[email protected]>
|
|
||
| def test_mask_with_NA(): | ||
| df = DataFrame({"A": [0, 1, 2]}) | ||
| cond = Series([True, False, pd.NA], dtype=pd.BooleanDtype()) |
There was a problem hiding this comment.
To pass the tests, I think dtype=BooleanDtype() should be used here, and from pandas import BooleanDtype at the top of the file, just like other dtypes.
|
Hi, |
|
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
|
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This commit addresses an issue where using NA in conjunction with the mask() method that resulted in unexpected behavior. The problem arose when comparing a Series containing NA as a condition within mask(), causing inconsistencies in the output. This fix ensures that NA behaves consistently as False in logical operations within mask().