API: setting np.nan into Series[bool]

[jbrockmendel]

ser = pd.Series([True, False, True, False])
ser[0] = np.nan

>>> ser
0    NaN
1    0.0
2    1.0
3    0.0
dtype: float64

It would be more consistent (and would simplify the code) if this cast to object and did not convert True/False to 1.0/0.0. ATM we have 2 tests that such a change would break.

[jorisvandenbossche]

Isn't this actually consistent with how other numpy dtypes work? We generally upcast on setitem-like operations, and for boolean this means upcasting to float (like integers also get upcasted to float)?

[jbrockmendel]

I think it is consistent with how numpy works, but bool-casting is one of the cases where we often behave differently from numpy. Ideally we would consistently use find_common_type:

>>> find_common_type([np.float64, np.bool_])
dtype('O')

[jreback]

i agree with @jbrockmendel here

this is something we should do correctly

numpy does this because it for backwards compatibility not because it's the right thing to do

we want to preserve dtypes whenever possible

bools are a wart when converted to float (ideally we would actually just convert to Boolean though)

[jorisvandenbossche]

Where is this find_common_type used? Since it are not extension dtypes, for example concat also doesn't follow that:

In [17]: pd.concat([pd.Series([True, False]), pd.Series([0.1, 2])])
Out[17]: 
0    1.0
1    0.0
0    0.1
1    2.0
dtype: float64

[jorisvandenbossche]

bools are a wart when converted to float

Bools are just as much a wart when converted to object dtype, though.
There is no such thing as "correct" IMO, just choices with trade-offs (and for sure, we should discuss those trade-offs and make a choice, but converting to object dtype being "correct" is not an argument IMO)

[jbrockmendel]

Where is this find_common_type used?

If we were to change this behavior, the change would look like #38709, which uses coerce_to_target_dtype in place of the custom logic currently in Block.setitem. coerce_to_target_dtype now uses find_common_type.

for example concat also doesn't follow that:

I haven't looked at concat all that closely, but based on the example you gave id like that to be object too.

[jreback]

what if we convert directly to Boolean here? that is the long term type we want anyhow.

[jbrockmendel]

what if we convert directly to Boolean here? that is the long term type we want anyhow.

long-term that'll be nice, but its still opt-in. Plus that comes with surprising behavior changes (copy/view semantics, 1D/2D shenanigans)

[jreback]

what if we convert directly to Boolean here? that is the long term type we want anyhow.

long-term that'll be nice, but its still opt-in. Plus that comes with surprising behavior changes (copy/view semantics, 1D/2D shenanigans)

hmm I actually think changing to nullable int would totally be fine here (sure its a breaking change), but so is what you are proposing (convert to object)

[jbrockmendel]

nullable int

-1 while it is still opt-in

[jbrockmendel]

-1 while it is still opt-in

or more specifically, until we have a BooleanArray-backed index, so we can keep the Series/Index behavior consistent