BUG: Float64 gets mixed with np.nan and cause sum to be nan

[ChiQiao]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
a = pd.Series([0, np.nan], dtype="float")
b = pd.Series([0, 1], dtype="Int64")
(a * b).sum()

Issue Description

According to https://pandas.pydata.org/docs/dev/user_guide/missing_data.html#calculations-with-missing-data, missing data should be treated as 0 when calculating the sum, while the code above yields np.nan.

When constructing a Series with dtype="Float64", np.nan is converted to pd.NA. In the code above, the type of a * b is inferred as Float64, but np.nan persists.

Expected Behavior

returns 0

Installed Versions

INSTALLED VERSIONS

commit : bb1f651
python : 3.9.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19043
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

pandas : 1.4.0
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
pip : 22.0.2
setuptools : 59.8.0
Cython : None
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : 4.7.1
html5lib : None
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.30.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
fastparquet : 0.8.0
fsspec : 2021.11.1
gcsfs : None
matplotlib : 3.5.1
numba : None
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 3.0.0
pyreadstat : None
pyxlsb : None
s3fs : 0.4.2
scipy : 1.7.3
sqlalchemy : 1.4.31
tables : 3.7.0
tabulate : 0.8.9
xarray : 0.21.1
xlrd : 2.0.1
xlwt : None
zstandard : None

[attack68]

On the main branch I can't verify this, I get:

import pandas as pd
a = pd.Series([0, np.nan], dtype="Float64")
b = pd.Series([0, 1], dtype="Int64")
(a * b).sum()
0.0

[rhshadrach]

@attack68 - "float" in line 2, not "Float64". With that change, I also get nan which looks incorrect to me.

[attack68]

But isn't that the point of this issue. OP is reporting that with either "float" or "Float64" he gets "nan", but that he is expecting to get 0 only in the case of "Float64", which seems to be as described in the docs he links?
And I'm suggesting that on main this all works as he expects.
Maybe I misunderstand.

[rhshadrach]

OP is reporting that with either "float" or "Float64" he gets "nan"

I don't think that's the case. I believe OP is saying that doing [float] * [Int64] the dtype gets coerced to Float64, but containing the value np.nan (rather than coercing to the expected pd.NA). With this, doing the sum then gives the value np.nan rather than the expected 0.

which seems to be as described in the docs he links?

The docs say When summing data, NA (missing) values will be treated as zero. So with the dtype being float in your line 2 above, shouldn't the result here be 0?

[jorisvandenbossche]

I think this is essentially the same issues as #42630 (but for a different operation). When mixing a nullable and non-nullable column in element-wise operations such as multiplication, we should ensure to convert the non-nullable column to a nullable dtype, so np.nan gets converted to pd.NA, and the element-wise operation (and also subsequent sum) proceeds as expected.

[dimitra-karadima]

Hello all! I would like to try to tackle this issue. I am new to contributing to pandas so if you have any suggestions or a high-level specification in mind please let me know!

[attack68]

Hello all! I would like to try to tackle this issue. I am new to contributing to pandas so if you have any suggestions or a high-level specification in mind please let me know!

this is quite a complicated issue. i would suggest you find an issue labelled "good first issue" or "contributions welcome". that will help you explore the code and learn the PR and approvals process without complications.

[dimitra-karadima]

Ok @attack68 thank you!