Skip to content

Rolling window and np.nanmean outputs only NaN #14517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ngenain opened this issue Oct 27, 2016 · 2 comments
Closed

Rolling window and np.nanmean outputs only NaN #14517

ngenain opened this issue Oct 27, 2016 · 2 comments
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode Usage Question

Comments

@ngenain
Copy link

ngenain commented Oct 27, 2016

Hi,

I am having an issue with the rolling apply method combinated with numpy's nanmean method. Here is my isssue:

import pandas as pd
import numpy as np

>>> df = pd.DataFrame({'A': [np.nan, np.nan, np.nan, 5, np.nan, np.nan]})
>>> df
     A
0  NaN
1  NaN
2  NaN
3  5.0
4  NaN
5  NaN

>>> df.rolling(3).apply(np.nanmean)
    A
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN

Expected Output

After index 3 I was expecting to see 5 in every line, how come I got only Nan?

Thanks for your help!

Output of pd.show_versions()

# Paste the output here ## INSTALLED VERSIONS

commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-100-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.19.0
nose: 1.3.7
pip: 8.1.2
setuptools: 24.0.3
Cython: None
numpy: 1.11.2
scipy: 0.18.0
statsmodels: 0.6.1
xarray: None
IPython: 5.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: None
pandas_datareader: None

@jreback jreback added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Usage Question labels Oct 27, 2016
@jreback jreback added this to the No action milestone Oct 27, 2016
@jreback
Copy link
Contributor

jreback commented Oct 27, 2016

you need to supply min_periods, which defaults to the window size.

Note that using a numpy function directly with .apply is much slower (some are mapped directly to the pandas impl, e.g. np.mean); I suppose np.nan* should be though they only exist in later versions of pandas.

In [1]: df = pd.DataFrame({'A': [np.nan, np.nan, np.nan, 5, np.nan, np.nan]})

In [2]: df.rolling(3).apply(np.nanmean)
Out[2]: 
    A
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN

In [3]: df.rolling(3).mean()
Out[3]: 
    A
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN

In [4]: df.rolling(3, min_periods=1).apply(np.nanmean)
Out[4]: 
     A
0  NaN
1  NaN
2  NaN
3  5.0
4  5.0
5  5.0

In [5]: df.rolling(3, min_periods=1).mean()
Out[5]: 
     A
0  NaN
1  NaN
2  NaN
3  5.0
4  5.0
5  5.0

@ngenain
Copy link
Author

ngenain commented Oct 27, 2016

Thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode Usage Question
Projects
None yet
Development

No branches or pull requests

2 participants