BUG: Rolling .apply() with method='table' ignores min_periods

[kasparthommen]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]], columns=['A', 'B'])
print(df)

# prints this:
#    A   B
# 0  1   2
# 1  3   4
# 2  5   6
# 3  7   8
# 4  9  10

df.rolling(3, min_periods=3).apply(lambda x: bool(print(x, '\n')))

# prints this, so as expected, only batches of 3 are being evaluated
# 0    1.0
# 1    3.0
# 2    5.0
# dtype: float64 
# 
# 1    3.0
# 2    5.0
# 3    7.0
# dtype: float64 
# 
# 2    5.0
# 3    7.0
# 4    9.0
# dtype: float64 
# 
# 0    2.0
# 1    4.0
# 2    6.0
# dtype: float64 
# 
# 1    4.0
# 2    6.0
# 3    8.0
# dtype: float64 
# 
# 2     6.0
# 3     8.0
# 4    10.0
# dtype: float64 
# 
# Out[33]: 
#      A    B
# 0  NaN  NaN
# 1  NaN  NaN
# 2  0.0  0.0
# 3  0.0  0.0
# 4  0.0  0.0

df.rolling(3, method='table', min_periods=3).apply(lambda x: bool(print(x, '\n')), raw=True, engine='numba')

# prints this, where we see that the first two batches of sizes 1 and 2 are also being evaluated
# [[1. 2.]]   <-- batch of size 1, illegal
# 
# [[1. 2.]    <-- batch of size 2, illegal
#  [3. 4.]] 
# 
# [[1. 2.]
#  [3. 4.]
#  [5. 6.]] 
# 
# [[3. 4.]
#  [5. 6.]
#  [7. 8.]] 
# 
# [[ 5.  6.]
#  [ 7.  8.]
#  [ 9. 10.]] 
# Out[34]: 
#      A    B
# 0  NaN  NaN
# 1  NaN  NaN
# 2  0.0  0.0
# 3  0.0  0.0
# 4  0.0  0.0

Issue Description

See example above, which violates the contract imposed by the min_periods parameter.

Expected Behavior

The min_periods parameter should be respected.

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.12.3.final.0
python-bits : 64
OS : Windows
OS-release : 11
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : German_Switzerland.utf8
pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
setuptools : 69.5.1
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.2.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.24.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 16.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report - further investigations and PRs to fix are welcome!

[rob-sil]

I put up a quick fix, but there's a larger issue with how min_periods is used with the indexers that choose the windows for rolling operations. Specifically, indexers have a method get_window_bounds that takes in the min_periods argument, but none of the implementations use it.

If an indexer ever used min_periods, it would probably cause the rolling operation to crash. Usually, these operations validate the output of get_window_bounds and raise if the number of windows returned doesn't match the number of rows in the data.

Since the indexers ignore min_periods, most rolling methods manually check min_periods elsewhere. In the case of this bug, min_periods was checked after applying the function, so the output correctly had two rows of NA, but incorrectly called the function on smaller windows. (I think there's one other case where min_periods isn't properly applied. I'll check and open a separate bug report if so.)

In my opinion, either indexers shouldn't take in min_periods at all, or they should respect it and only return window indexes that have at least min_periods periods.