Inconsistent results between rolling.corr and corr

[evanzd]

Code Sample, a copy-pastable example if possible

import pandas as pd
x = pd.Series([0.125, 0.625, 0.5, 0.8125, 0.1875])
y = pd.Series([0.2, 0.2, 0.2, 0.2, 0.2])
print('rolling.corr:', x.rolling(5).corr(y).iloc[-1])
print('corr:', x.corr(y))

Output:

rolling.corr: -inf
corr: nan

Problem description

For constant vector (e.g., [0.2, 0.2, 0.2, 0.2, 0.2]), corr should be nan but not inf.

Expected Output

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit : None
python : 3.6.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 0.25.1
numpy : 1.16.5
pytz : 2019.3
dateutil : 2.8.0
pip : 19.2.3
setuptools : 41.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.8.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[mmitalidis]

It seems that this happens because of arithmetic on floating point numbers and the different implementations of x.corr(y) and x.rolling(5).corr(y)

Specifically, x.corr(y) calls:

pandas/pandas/core/nanops.py

Line 1253 in 825f811

return np.corrcoef(a, b)[0, 1]

which works as expected.

However, x.rolling(5).corr(y) will calculate the correlation at:

pandas/pandas/core/window/rolling.py

Line 1636 in 1effb56

return a.cov(b, **kwargs) / (a.std(**kwargs) * b.std(**kwargs))

where a = x.rolling(5) and b = y.rolling(5).

x.rolling(5).cov(y) is calculated at:

pandas/pandas/core/window/rolling.py

Line 1503 in 1effb56

return (mean(X * Y) - mean(X) * mean(Y)) * bias_adj

which returns a very small number (but not 0.0).

Thus, the correlation is not a (0. / 0.) operation and you don't get nan.

To reproduce the non-zero covariance result:

import numpy as np

x = np.array([0.125, 0.625, 0.5, 0.8125, 0.1875])
y = np.array([0.2, 0.2, 0.2, 0.2, 0.2])

print(np.mean(x*y))
# 0.09
print(np.mean(x) * np.mean(y))
# 0.09000000000000001
print(np.mean(x * y) - np.mean(x) * np.mean(y))
# -1.3877787807814457e-17

or similarly with pandas:

import pandas as pd

x = pd.Series([0.125, 0.625, 0.5, 0.8125, 0.1875])
y = pd.Series([0.2, 0.2, 0.2, 0.2, 0.2])

print(x.rolling(5).cov(y).iloc[-1])
# -1.734723475976807e-17

[simonjayhawkins]

closing as duplicate of #24019