BUG: rolling() function does not work with Float64 columns with missing values

[janlugt]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

Simple series with floats, dtype float64:

>>> df = pd.Series([0.0, 1.0, 2.0, 3.0, 4.0])
>>> df
0    0.0
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64

diff() introduces a NaN in the first position:

>>> df.diff()
0    NaN
1    1.0
2    1.0
3    1.0
4    1.0
dtype: float64

This works as expected:

>>> df.diff().rolling(2).sum()
0    NaN
1    NaN
2    2.0
3    2.0
4    2.0

We can cast to a Float64:

>>> df.astype('Float64')
0    0.0
1    1.0
2    2.0
3    3.0
4    4.0
dtype: Float64

diff() still works fine, but gives us a pd.NA instead of a NaN:

>>> df.astype('Float64').diff()
0    <NA>
1     1.0
2     1.0
3     1.0
4     1.0
dtype: Float64

Now, when we call rolling(), everything comes tumbling down:

>>> df.astype('Float64').diff().rolling(2).sum()
Traceback (most recent call last):
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 321, in _prep_values
    values = ensure_float64(values)
  File "pandas/_libs/algos_common_helper.pxi", line 45, in pandas._libs.algos.ensure_float64
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/arrays/masked.py", line 335, in __array__
    return self.to_numpy(dtype=dtype)
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/arrays/masked.py", line 292, in to_numpy
    raise ValueError(
ValueError: cannot convert to 'float64'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 402, in _apply_series
    values = self._prep_values(obj._values)
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 323, in _prep_values
    raise TypeError(f"cannot handle this type -> {values.dtype}") from err
TypeError: cannot handle this type -> Float64

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 1723, in sum
    return super().sum(*args, engine=engine, engine_kwargs=engine_kwargs, **kwargs)
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 1233, in sum
    return self._apply(window_func, name="sum", **kwargs)
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 539, in _apply
    return self._apply_blockwise(homogeneous_func, name)
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 417, in _apply_blockwise
    return self._apply_series(homogeneous_func, name)
  File "/home/janlugt/repositories/streaming_anomaly_detection/.venv/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 404, in _apply_series
    raise DataError("No numeric types to aggregate") from err
pandas.core.base.DataError: No numeric types to aggregate

Problem description

My expectation would be that whether we call rolling on a float64 or a Float64 series, the output would be the same. The promise of pd.NA is that it can be used consistently across data types, and make the use of specific NaN types such as np.nan or NaT unnecessary, which is clearly not the case here.

Expected Output

>>> df.astype('Float64').diff().rolling(2).sum()
0    <NA>
1    <NA>
2    2.0
3    2.0
4    2.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 5f648bf
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.16.3-microsoft-standard-WSL2
Version : #1 SMP Fri Apr 2 22:23:49 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.2
numpy : 1.21.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.2.4
setuptools : 57.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[simonjayhawkins]

Thanks @janlugt for the report.

on master, this no longer raises but does not give the expected dtype. The result is

0    NaN
1    NaN
2    2.0
3    2.0
4    2.0
dtype: float64

[jreback]

@mroeschke made an issue about this as this is a breaking change

[simonjayhawkins]

Float64 is still experimental? we can fix this?

[mroeschke]

Yes this repurposed issue, #11446, is about having rolling operations return the same dtype as the caller (if possible).

The docstrings and this windowing section document that these windowing operations always return np.float64
#43030
https://pandas.pydata.org/docs/user_guide/window.html#overview

Since this overlaps with #11446, going to close in favor of that issue, but definitely possible to support Float64 in the near future (idk if it would be an "enhancement" or a "breaking change" in this case)

[janlugt]

Thanks for looking into this. I'll test my code with 1.3.3 when it comes out later this month (according to https://pandas.pydata.org/pandas-docs/dev/whatsnew/v1.3.3.html).

[simonjayhawkins]

@janlugt I didn't test with 1.3.x, #43381 (comment) was with master, so may not work with 1.3.3 unless the commit that fixed was backported.

[drtoche]

I'm coming to this issue with very little expertise (coming from R and Julia): created a variable by dividing one Int64 by another and got a Float64, then tried to apply the rolling() function and hit the error. It's strange that such a simple step would uncover an issue going back to 2015. My workaround is to convert all the variables to float64:

def pandas_debug(df, cols):
    """pandas.rolling() function bugs with Float64, so converting to float64."""
    for col in cols:
        df[col] = df[col].astype(float)
    return df

[simonjayhawkins]

@drtoche

fixed by #43174, 1.4.0 release candidate expected next month with release in early January. Please feel free to evaluate the release candidate when available or checkout the nightly builds to confirm this fixes the issue.

https://anaconda.org/scipy-wheels-nightly/pandas/files

[drtoche]

Great, thank you so much Simon!

[olofahlen]

Hi,

I think this bug still exists (in pandas 1.5.1) but for .ewm. See example below

>>> s = pd.Series(range(100), dtype=pd.Float64Dtype())
>>> s.iloc[98] = pd.NA
>>> s.ewm(halflife=5).cov()
Traceback (most recent call last):
  File "C:\Users\oahlen\.conda\envs\experiments\lib\site-packages\pandas\core\window\rolling.py", line 404, in _prep_values
    values = ensure_float64(values)
  File "pandas\_libs\algos_common_helper.pxi", line 44, in pandas._libs.algos.ensure_float64
  File "C:\Users\oahlen\.conda\envs\experiments\lib\site-packages\pandas\core\series.py", line 893, in __array__
    return np.asarray(self._values, dtype)
  File "C:\Users\oahlen\.conda\envs\experiments\lib\site-packages\pandas\core\arrays\masked.py", line 490, in __array__
    return self.to_numpy(dtype=dtype)
  File "C:\Users\oahlen\.conda\envs\experiments\lib\site-packages\pandas\core\arrays\masked.py", line 412, in to_numpy
    raise ValueError(
ValueError: cannot convert to 'float64'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.