BUG: wrong result in rolling mean\sum (possibly related with cython implementation)

[auderson]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here
df = pd.Series(np.random.random(20), dtype='float64')

df[:10] *= 1e6
df[-10:] = 0
df
Out[124]: 
0     235266.408832
1     979278.480747
2     242108.917259
3     513980.266425
4      44316.957433
5     408655.731842
6     490360.015982
7     855399.526047
8     321975.011810
9     831082.388210
10         0.000000
11         0.000000
12         0.000000
13         0.000000
14         0.000000
15         0.000000
16         0.000000
17         0.000000
18         0.000000
19         0.000000
dtype: float64

df.rolling(6).mean()
Out[126]: 
0              NaN
1              NaN
2              NaN
3              NaN
4              NaN
5     2.682618e+05
6     2.723245e+05
7     2.937347e+05
8     3.589816e+05
9     2.737199e+05
10    2.429527e+05
11    2.411146e+05
12    1.258042e+05
13    9.726441e+04
14    1.032524e+04
15    2.546585e-11
16    2.546585e-11
17    2.546585e-11
18    2.546585e-11
19    2.546585e-11
dtype: float64

df.rolling(6).sum()
Out[127]: 
0              NaN
1              NaN
2              NaN
3              NaN
4              NaN
5     1.609571e+06
6     1.633947e+06
7     1.762408e+06
8     2.153889e+06
9     1.642319e+06
10    1.457716e+06
11    1.446688e+06
12    7.548251e+05
13    5.835865e+05
14    6.195146e+04
15    1.527951e-10
16    1.527951e-10
17    1.527951e-10
18    1.527951e-10
19    1.527951e-10
dtype: float64

Problem description

if the head is substantially larger than the 0s in the tail, then rolling mean\sum will not generate correct results. I roughly looked into the source code and guess maybe it's related to cython "rolling_mean"

Expected Output

0 in row 15 to 20, not 2.546585e-11

Output of pd.show_versions()

INSTALLED VERSIONS

commit : db08276
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-112-generic
Version : #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.1.3
numpy : 1.19.2
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.1
setuptools : 51.0.0.post20201207
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.1
html5lib : None
pymysql : 0.9.2
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: 0.9.0
bs4 : None
bottleneck : None
fsspec : 0.8.4
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.4
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.51.2

[jreback]

try on master / 1.2.0rc0
we recently changed to use kahan summation which should fix this

[mroeschke]

I'm getting similar results on master and thanks for the report.

This behavior is similar to #24019 and #37051 and I think our Kahan summation implementation doesn't handle large to small value transitions.

Since this is a known issue, I'm going to close as a duplicate

[phofl]

I am getting

0               NaN
1               NaN
2               NaN
3               NaN
4               NaN
5     663436.774348
6     744481.486713
7     651646.553540
8     734838.516497
9     583653.271318
10    477420.570783
11    335957.922536
12    188207.055765
13    166502.059749
14     14815.242535
15         0.000000
16         0.000000
17         0.000000
18         0.000000
19         0.000000
dtype: float64
0              NaN
1              NaN
2              NaN
3              NaN
4              NaN
5     3.980621e+06
6     4.466889e+06
7     3.909879e+06
8     4.409031e+06
9     3.501920e+06
10    2.864523e+06
11    2.015748e+06
12    1.129242e+06
13    9.990124e+05
14    8.889146e+04
15    0.000000e+00
16    0.000000e+00
17    0.000000e+00
18    0.000000e+00
19    0.000000e+00
dtype: float64

Process finished with exit code 0

on master. Kahan summation can handle the transition, floating point precision for something like 2/3 is a problem.