BUG: fillna overwriting other missing vals

[mzeitlin11]

• closes BUG: Series.fillna is altering None values of unspecified columns #40498
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[jreback]

i am not sure i buy that this is a bug at all.

In [149]: series.isna()                                                                                                                                               
Out[149]: 
1    False
2     True
3     True
4    False
dtype: bool

Trying to use mixed null values in an object series is just not supported.

[mzeitlin11]

i am not sure i buy that this is a bug at all.

In [149]: series.isna()                                                                                                                                               
Out[149]: 
1    False
2     True
3     True
4    False
dtype: bool

Trying to use mixed null values in an object series is just not supported.

I think the confusion from #40498 is not that isna(None) = True, but that the value at index 2 is modified even though the argument {'3':0} should ensure that only index 3 is modified.

[jreback]

i am not sure i buy that this is a bug at all.

In [149]: series.isna()                                                                                                                                               
Out[149]: 
1    False
2     True
3     True
4    False
dtype: bool

Trying to use mixed null values in an object series is just not supported.

I think the confusion from #40498 is not that isna(None) = True, but that the value at index 2 is modified even though the argument {'3':0} should ensure that only index 3 is modified.

I am ok with that, but your patch is not structured that way. it shouldn't be testing for nulls

[mzeitlin11]

Sorry, can you please clarify what you mean by that?

What the patch should be doing is fixing the following: for the series ser = Series({'1': 1,'2': None,'3': np.nan, '4': 1}), when the fill argument '{'3':0}' hits the line I changed, it previously became [NaN, NaN, 0, NaN].

When this hits ObjectBlock.fillna, mask is determined from isna(ser), giving [False, True, True, False]. So then when putmask_inplace is called with this mask, None is replaced with np.nan since the mask is True at the position.

To avoid having to change the behavior of lower level fillna functionality, this patch changes the fill argument to instead become [NaN, None, 0, NaN], so that None just ends up being replaced by itself.

[jreback]

i think you are better off calling .align here. What you have on L6531 is too specific, fragile and trying to fix something after the fact.

[mzeitlin11]

Yeah I see what you mean. Not seeing a clear solution with .align either - will have to have some sort of ugly isna check at some point I think to support this. Will close for now and reopen if I can find a better way to do this