BUG: inconsistent result when groupby then sum values that contain inf

[jerrytim]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd

>>> df = pd.DataFrame({'a': ['hello', 'hello', 'world', 'world'], 'b': [np.inf, 10, np.nan, 10]})
>>> gb = df.groupby('a')
>>> gb.sum()
          b
a          
hello   NaN
world  10.0
>>> gb.get_group('hello').b.sum()
inf

Problem description

while directly doing groupby.sum, the group with inf will return np.nan, but if we get the group and do sum, it returns np.inf

Expected Output

groupby.sum should also return np.inf if the values contian np.inf

Output of pd.show_versions()

INSTALLED VERSIONS

commit : c7f7443
python : 3.7.6.final.0
python-bits : 64
OS : Darwin
OS-release : 18.2.0
Version : Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
pandas : 1.3.1
numpy : 1.21.1
pytz : 2021.1
dateutil : 2.8.2
pip : 21.1.2
setuptools : 57.0.0
Cython : None
pytest : 4.5.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : 0.7.11.None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.26.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : None
scipy : 1.1.0
sqlalchemy : 1.3.24
tables : 3.5.2
tabulate : 0.7.7
xarray : None
xlrd : None
xlwt : None
numba : None

[hec10r]

Small comment that may help:
When using version 1.1.3 I am getting inf as expected, but when using master I am getting the same results as you.

[AbhirathAnupamJoshi]

take

[simonjayhawkins]

Thanks @jerrytim for the report.

When using version 1.1.3 I am getting inf as expected, but when using master I am getting the same results as you.

also worked in 1.2.5

first bad commit: [5cc24c2] ENH: Use Kahan summation to calculate groupby.sum() (#38903)

cc @phofl

[mzeitlin11]

This regression pops up anywhere we use kahan summation, for example also

In [4]: ser = pd.Series([np.inf, 1, 1])

In [5]: ser.rolling(3).sum()
Out[5]:
0   NaN
1   NaN
2   NaN
dtype: float64

In [6]: ser.groupby([1, 1, 1]).cumsum()
Out[6]:
0    inf
1    NaN
2    NaN
dtype: float64

These NaNs pop up because we end computing inf - inf = NaN. Could be fixed by special casing inf so we never end up with compensation = NaNinf, might hurt perf though

[simonjayhawkins]

Could be fixed by special casing inf so we never end up with compensation = NaNinf, might hurt perf though

perf is secondary to correctness.

we also have #39622 open which is reporting a performance degradation.

from #39622 (comment)

maybe we can make Kahan summation opt-in?

so that could also be an option.

[phofl]

Actually rolling returned nan even before we implemented Kahan there. So maybe opening up a separate issue @mzeitlin11 ?

[simonjayhawkins]

changing milestone to 1.3.5

[thomasbridi]

please fix this before bringing The Wrath of K(a)han upon us

[phofl]

A pr adressing this is welcome

[simonjayhawkins]

removing from 1.3.x milestone, xref #44526 (comment)

[rhshadrach]

Closed by #53623