BUG: Series rolling .var with window>14, center=True, and any win_type crashes python with no error message

[joseortiz3]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

"""
Crash occurs often when:
    window >= 15, 
    center=True,
    win_type='triang' (or any win_type AFAIK),
    usually in the first few trials.

Crash does not result in an error message, python just quits.

Crash does not occur when:
    window <= 14
    win_type is not specified
    center=False
    .var() is changed to .mean()
"""

x = pd.Series(0)

N = 10

for i in range(N):
    print('Starting trial %s out of %s...' % (i+1,N), end='')
    result = x.rolling(window=16, center=True, win_type='triang').var()
    print('success')

Issue Description

This script, if executed, causes python to simply stop (as if calling quit()) ~3/4ths of the time, usually when i > 1.

Crash seems to occur often when:

• window >= 15,
• center=True,
• win_type='triang' (or any win_type AFAIK),

Crash does not seem to occur when:

• window <= 14
• win_type is not specified
• center=False
• .var() is changed to .mean()

Example of repeatedly running the script:

>>python module1.py
Starting trial 1 out of 10...success
Starting trial 2 out of 10...success
Starting trial 3 out of 10...success
Starting trial 4 out of 10...success

>>python module1.py
Starting trial 1 out of 10...success
Starting trial 2 out of 10...success
Starting trial 3 out of 10...success

>>python module1.py
Starting trial 1 out of 10...success
Starting trial 2 out of 10...success
Starting trial 3 out of 10...success
Starting trial 4 out of 10...success
Starting trial 5 out of 10...success
Starting trial 6 out of 10...success
Starting trial 7 out of 10...success
Starting trial 8 out of 10...success
Starting trial 9 out of 10...success
Starting trial 10 out of 10...success

Expected Behavior

The script should run reliably, or at least deterministically.

An error should appear if there is an issue, but instead python appears to just quit()/crash without any message.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.10.4.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.4.2
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
pip : 22.0.4
setuptools : 58.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[mroeschke]

Thanks for the report. I can't seem to replicate this on the main branch on macOS with scipy 1.8.0; maybe it's windows specific? Would you be able to confirm that this happens on another platform?

[joseortiz3]

@mroeschke I'm hoping someone can test whether this happens on Linux. As a sanity check, I ran the script on another windows machine with the same result. Could very well be a windows problem.

[mroeschke]

Actually since I was running this in ipython, after exiting I get

python(1183,0x118a2adc0) malloc: *** error for object 0x7ff8000000000000: pointer being freed was not allocated
python(1183,0x118a2adc0) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort      ipython

Which is probably related to why your program crashes.

[joseortiz3]

Is that output suggesting there is a memory issue? Would love to learn how someone debugs something like this.

[mroeschke]

Yeah, I think this change resolves the issue

--- a/pandas/_libs/window/aggregations.pyx
+++ b/pandas/_libs/window/aggregations.pyx
@@ -1592,7 +1592,7 @@ def roll_weighted_var(const float64_t[:] values, const float64_t[:] weights,

     with nogil:

-        for i in range(win_n):
+        for i in range(min(win_n, n)):
             add_weighted_var(values[i], weights[i], &t,
                              &sum_w, &mean, &nobs)

in your example, win_n > n therefore the loop will try to access values[i] where i > n which can lead to a segfault.

Would you like to make a PR if you could also verify the change on your end?

[joseortiz3]

I can make a PR in the morning. Verifying the change is not as easy as just editing aggregations.pyx in my site-packages, right? I need to make that change in my fork of pandas and build pandas from source to test it? Thanks for the help.

[mroeschke]

I need to make that change in my fork of pandas and build pandas from source to test it?

Correct, https://pandas.pydata.org/pandas-docs/stable/development/contributing_environment.html#creating-a-python-environment