BUG: asfreq / pct_change strange behavior

[shura-v]

In the first case, is it a bug (those NaNs at the end) or a feature? I just don't get the reason behind this behavior:

In[5]: s = Series(range(10), date_range('2014', periods=10, freq='H'))
In[6]: s
Out[6]: 
2014-05-31 00:00:00    0
2014-05-31 01:00:00    1
2014-05-31 02:00:00    2
2014-05-31 03:00:00    3
2014-05-31 04:00:00    4
2014-05-31 05:00:00    5
2014-05-31 06:00:00    6
2014-05-31 07:00:00    7
2014-05-31 08:00:00    8
2014-05-31 09:00:00    9
Freq: H, dtype: int64
In[7]: s.pct_change(periods=1, freq='5H')
Out[7]: 
2014-05-31 00:00:00         NaN
2014-05-31 01:00:00         NaN
2014-05-31 02:00:00         NaN
2014-05-31 03:00:00         NaN
2014-05-31 04:00:00         NaN
2014-05-31 05:00:00         inf
2014-05-31 06:00:00    5.000000
2014-05-31 07:00:00    2.500000
2014-05-31 08:00:00    1.666667
2014-05-31 09:00:00    1.250000
2014-05-31 10:00:00         NaN
2014-05-31 11:00:00         NaN
2014-05-31 12:00:00         NaN
2014-05-31 13:00:00         NaN
2014-05-31 14:00:00         NaN
dtype: float64

but this seems ok:

In[8]: s.pct_change(periods=5)
Out[8]: 
2014-05-31 00:00:00         NaN
2014-05-31 01:00:00         NaN
2014-05-31 02:00:00         NaN
2014-05-31 03:00:00         NaN
2014-05-31 04:00:00         NaN
2014-05-31 05:00:00         inf
2014-05-31 06:00:00    5.000000
2014-05-31 07:00:00    2.500000
2014-05-31 08:00:00    1.666667
2014-05-31 09:00:00    1.250000
Freq: H, dtype: float64

[jreback]

So pct_change is just s divided by its 5H shift (slightly more complicated as it handles various fill methods). So this, while I agree looks a bit odd, seems correct. That said I could also see that it should reindex to the original series

In [6]: s
Out[6]: 
2014-06-01 00:00:00    0
2014-06-01 01:00:00    1
2014-06-01 02:00:00    2
2014-06-01 03:00:00    3
2014-06-01 04:00:00    4
2014-06-01 05:00:00    5
2014-06-01 06:00:00    6
2014-06-01 07:00:00    7
2014-06-01 08:00:00    8
2014-06-01 09:00:00    9
Freq: H, dtype: int64

In [7]: s.shift(freq='5H')
Out[7]: 
2014-06-01 05:00:00    0
2014-06-01 06:00:00    1
2014-06-01 07:00:00    2
2014-06-01 08:00:00    3
2014-06-01 09:00:00    4
2014-06-01 10:00:00    5
2014-06-01 11:00:00    6
2014-06-01 12:00:00    7
2014-06-01 13:00:00    8
2014-06-01 14:00:00    9
dtype: int64

Proposed

In [10]: s.div(s.shift(freq='5H')).reindex_like(s)
Out[10]: 
2014-06-01 00:00:00         NaN
2014-06-01 01:00:00         NaN
2014-06-01 02:00:00         NaN
2014-06-01 03:00:00         NaN
2014-06-01 04:00:00         NaN
2014-06-01 05:00:00         inf
2014-06-01 06:00:00    6.000000
2014-06-01 07:00:00    3.500000
2014-06-01 08:00:00    2.666667
2014-06-01 09:00:00    2.250000
Freq: H, dtype: float64

[jreback]

want to do a pr to fix this?

[jreback]

@shura-v how's this coming?

[jreback]

@shura-v ?

[shura-v]

I made pull request on June 2nd:
Here is my commit: https://github.com/shura-v/pandas/commit/eb250b6cead505bad3cf67e838f975b6148cdeae

[jreback]

needs some tests

[minggli]

Hi, will attempt to work on this issue. reverting.

[minggli]

@jreback

since pandas.core.generic.NDFrame.pct_change essentially calls s.shift method to work out pct change, I've looked into how different outputs were produced with difference choices of params.

s.shift(periods=5, freq=None) uses the underlying _data block manager to shift values without touching the index.

s.shift(freq='5H') calls s.tshift and shift index, hence when calculating pct_change (unshifted divided by shifted frame), the resulting frame will have longer index when shifting with frequencies than when shifting with periods, because the latter scenario, index remains unchanged.

So that explains the difference of observations as per earlier discussion. raising PR. reverting.