Intermittent issue with rolling_min function - Calculation "blows up"

[DiSchi123]

I noticed a strange behavior that happens intermittently and only happens on my laptop. Not reproducible on my Windows 8 Desktop which concerns me.

Run the following:

from pandas import Series, DataFrame
import pandas as pd
import numpy as np


#inputdata
windows=np.array([5,10,15,60,120,300,0,0,0,0] )  #parameters
dates=pd.date_range('1/1/2000 15:00:00', periods =100000, freq='1H')
hitemps=-50+np.round(np.random.rand(100000),3)*100  # simulated  hourly high temperature    readings
lowtemps=hitemps-2*np.round(np.random.rand(100000),3)  # low temps
d = {'HiTemp': hitemps, 'LoTemp' : lowtemps}

When running the following about 10 to 20 times, about 1 out of ~10 times the columns get populated with weird data. With my actual data set (too long to attach) it happens more often, like 1 in 4 times. Run the below code repeatedly. Sample problem and correct output attached in xls (further below):

df=DataFrame(data=d, index=dates)

# create new df columns of forward looking moves down, with varying time frames (window sizes). use window size in column name
for i, windowsize in enumerate(windows):   
    if windowsize==0:
        continue
    else:
        colname_df=str('Fw'+str(windowsize)+'Dwn')
        df[colname_df]=df.HiTemp-(pd.rolling_min(df.LoTemp,  window=windowsize).shift(-windowsize))  
df

rolling_min bug report outputs.xlsx

This was captured on a 2014 MacBook Air running Windows 10 via Bootcamp (see above comment, problem not happening on HP Windows Desktop).
Installed versions result:
pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.17.1
nose: 1.3.7
pip: 7.1.2
setuptools: 19.1.1
Cython: 0.23.4
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 4.0.1
sphinx: 1.3.1
patsy: 0.4.0
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.4
matplotlib: 1.5.0
openpyxl: 2.2.6
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.7
lxml: 3.4.4
bs4: 4.4.1
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.9
pymysql: None
psycopg2: None
Jinja2: None

[jreback]

this is might be related to using numexpr=2.4.4, try upgrading to 2.4.6 and see if you can repro

see #12023

[jreback]

btw, what you are doing in that loop is highly inefficient.

much better to:

l = []
for i, windowsize in enumerate(windows):   
    if  colname_df=str('Fw'+str(windowsize)+'Dwn')
        s = df.HiTemp-(pd.rolling_min(df.LoTemp,  window=windowsize).shift(-windowsize))
        s.name = colname_df
        l.append(l)
df = pd.concat(l,axis=1)

[DiSchi123]

Thanks! I suspected my loop is not ideal although I improved it quite a bit up to this. I couldn't find anything on custom named columns.

Will get back once I know more re version..

[DiSchi123]

Right on the money! I upgraded numexpr to 2.4.6 and the issue is gone. My Windows PC is at 2.3.1. of numexpr, I suppose the problem occured in between somewhere.

[DiSchi123]

Problem resolved - issue was numexpr 2.4.4. Upgraded to 2.4.6 and problem gone.

[jreback]

gr8! thanks.

yeh have seen that a few times. I think its a bug (only) on windows with numexpr 2.4.4