ENH: nullables use Kleene logic for any/all reductions #41970
Conversation
mzeitlin11
added
NA - MaskedArrays
|
|
||
| def any(self, *, skipna: bool = True, **kwargs): | ||
| """ | ||
| Return whether any element is truthy. |
| ): | ||
| ser = Series(data, dtype="boolean") | ||
| ser = Series(data, dtype="boolean").astype(dtype) |
There was a problem hiding this comment.
why is this astyped and not the dtype passed to the Series constructor?
There was a problem hiding this comment.
Have changed the parameterization to avoid needing to do this
The reason was that something like Series([True, pd.NA, True], dtype="Int64") raises. Though using just pure booleans doesn't raise, eg
Series([True, True, True], dtype="Int64") is fine. Does this inconsistency make sense, or should I open an issue?
There was a problem hiding this comment.
Does this inconsistency make sense, or should I open an issue?
pd.Series([True, np.nan, True], dtype="Int64") works fine, but pd.Series([True, None, True], dtype="Int64") also raises TypeError: object cannot be converted to an IntegerDtype
There does seam to appear to be inconsistency here.
|
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this. |
| ([False, pd.NA, False], [[pd.NA, False], [False, False]]), | ||
| ([True, pd.NA, True], [[True, pd.NA], [True, True]]), | ||
| ([True, pd.NA, False], [[True, False], [True, False]]), | ||
| ([0, 0, 0], [[0, 0], [0, 0]]), |
There was a problem hiding this comment.
why do these return ints rather than True/False?
There was a problem hiding this comment.
Thanks, that was a mistake - it actually returns bools, test just happened to work because the reductions return a scalar and 1 == True, etc
|
lgtm. @jorisvandenbossche if you can give a quick glance. |
|
thanks @mzeitlin11 |
|
Thanks @mzeitlin11 ! |
This gives consistency for nullable dtype treatment across
.any/.alland the groupby versions which already used Kleene logic. Diff looks big, but this is just movingBooleanArray.anyandBooleanArray.alltoBaseMaskedArray