ENH: allow .rolling / .expanding as groupby methods

[jreback]

closes #12738
closes #12486
closes #12363

• more tests (other methods)

- [ ] doc section in groupby will do later

In [3]: pd.options.display.max_rows=10

In [4]:    df = pd.DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8,
                      'B': np.arange(40)})

In [5]: df
Out[5]: 
    A   B
0   1   0
1   1   1
2   1   2
3   1   3
4   1   4
.. ..  ..
35  3  35
36  3  36
37  3  37
38  3  38
39  3  39

[40 rows x 2 columns]

In [6]:    df.groupby('A').apply(lambda x: x.rolling(4).B.mean())
Out[6]: 
A    
1  0      NaN
   1      NaN
   2      NaN
   3      1.5
   4      2.5
         ... 
3  35    33.5
   36    34.5
   37    35.5
   38    36.5
   39    37.5
Name: B, dtype: float64

In [7]:    df.groupby('A').rolling(4).B.mean()
Out[7]: 
A    
1  0      NaN
   1      NaN
   2      NaN
   3      1.5
   4      2.5
         ... 
3  35    33.5
   36    34.5
   37    35.5
   38    36.5
   39    37.5
Name: B, dtype: float64
In [9]:    df.index = pd.date_range('20130101',freq='s',periods=40)

In [10]:    df
Out[10]: 
                     A   B
2013-01-01 00:00:00  1   0
2013-01-01 00:00:01  1   1
2013-01-01 00:00:02  1   2
2013-01-01 00:00:03  1   3
2013-01-01 00:00:04  1   4
...                 ..  ..
2013-01-01 00:00:35  3  35
2013-01-01 00:00:36  3  36
2013-01-01 00:00:37  3  37
2013-01-01 00:00:38  3  38
2013-01-01 00:00:39  3  39

[40 rows x 2 columns]

In [11]:    df.groupby('A').apply(lambda x: x.resample('4s').mean())
Out[11]: 
                         A     B
A                               
1 2013-01-01 00:00:00  1.0   1.5
  2013-01-01 00:00:04  1.0   5.5
  2013-01-01 00:00:08  1.0   9.5
  2013-01-01 00:00:12  1.0  13.5
  2013-01-01 00:00:16  1.0  17.5
2 2013-01-01 00:00:20  2.0  21.5
  2013-01-01 00:00:24  2.0  25.5
  2013-01-01 00:00:28  2.0  29.5
3 2013-01-01 00:00:32  3.0  33.5
  2013-01-01 00:00:36  3.0  37.5

In [12]:    df.groupby('A').resample('4s').mean()
Out[12]: 
                         A     B
A                               
1 2013-01-01 00:00:00  1.0   1.5
  2013-01-01 00:00:04  1.0   5.5
  2013-01-01 00:00:08  1.0   9.5
  2013-01-01 00:00:12  1.0  13.5
  2013-01-01 00:00:16  1.0  17.5
2 2013-01-01 00:00:20  2.0  21.5
  2013-01-01 00:00:24  2.0  25.5
  2013-01-01 00:00:28  2.0  29.5
3 2013-01-01 00:00:32  3.0  33.5
  2013-01-01 00:00:36  3.0  37.5

[jreback]

cc @lminer

[jreback]

@sinhrks this fixed a couple of tests in groupby below, but I don't know if we have a related issue. any idea?

related (but did not fix) #12363

[sinhrks]

Not sure, I can find no open issue.

[jreback]

ok, this is ready. comments.

@shoyer @jorisvandenbossche @TomAugspurger @sinhrks

[jorisvandenbossche]

Is the example at the top still up to date?
I am wondering a bit if the grouper should be in the result or not?

[jorisvandenbossche]

Also, how does the result looks like if you have a non-sorted grouper?
Currently with apply it sorts the groups (but also includes the grouper as the index)

[jreback]

@jorisvandenbossche all updated. Had to work thru some issues. But much more fully tested now. This basically replicates what .apply is doing with the new syntax by actually using apply. In theory we could have a more efficient impl, but will leave that for later.

Also this PR cleans up a bunch of cases where the name is not returned for groupbys. So much more consistency now.

[jreback]

this is a massive hack using. well hack is the wrong word. Its a way of informing the groupby that we want to force the multiindex construction path which is normally taken only when things are mutated. its not an external visible kw.

[jreback]

@jorisvandenbossche if you'd have a look

[jorisvandenbossche]

Will take a look later today!

[jorisvandenbossche]

"These return another object where you operate" -> this is not really a clear sentence. What do exactly want to say?

[jreback]

updated

[TomAugspurger]

@jreback one more lost index name, when you group a Series by another (using the original dataframe)

In [27]: df['B'].groupby(df.A).rolling(4).mean()
Out[27]:
A  idx
1  0       NaN
   1       NaN
   2       NaN
   3       1.5
   4       2.5
          ...
3  35     33.5
   36     34.5
   37     35.5
   38     36.5
   39     37.5
dtype: float64

Let me know if you want to fix that here, otherwise I'll open an issue.

[TomAugspurger]

And I'm (correctly) seeing a MultiIndex for the df.groupby('A').resample('3s').mean()

[jorisvandenbossche]

Will try again (I just fetched the head of this PR, didn't rebase myself as I thought you already did that)

[jorisvandenbossche]

@TomAugspurger It's with the rolling case I do not see a MultiIndex (df.groupby('A').rolling(3).mean()), resample did work for me as well.

@jreback Tested it again (fetched this PR, rebased on latest master myself, rebuild pandas to be certain), and still getting the same result as I showed in #12743 (comment) (using python 2.7, numpy 1.10.1, Windows)

[jreback]

@jorisvandenbossche hmm, ok I did on 2.7/1.10.4 and it doesn't look right. odd. (on windows)

[jreback]

ok, this has been failing on windows on my branch : https://ci.appveyor.com/project/jreback/pandas/build/1.0.2077/job/qfrsv7p46f6ns6xg

[jreback]

ok, this is not a 2.7/numpy issue, but something specific to windows.

[jreback]

ok pushed a commit to fix. This comes back to some of the 'figuring what the shape of the return' is logic. IOW its a heuristic depending on whether something is mutated (this is the standard case). But when we are doing a chained window/resample we need to force this.

[jreback]

pushed a new version, had some flake/formatting issues on windows.

[jreback]

@TomAugspurger I picked up that last name fix (was a bug in the concat step).

[jreback]

@TomAugspurger @jorisvandenbossche any more comments.

[TomAugspurger]

Nothing else from me 👍

[jreback]

ok will merge shortly unless @jorisvandenbossche has any further comments.

[jorisvandenbossche]

Tested on master, and now indeed the inconsistency is solved! Thanks

[ibigquant]

@jreback, thanks.

I want to do grouby then rolling then corr on two columns, code below takes 18s+ on 1M rows:

df.groupby('name', as_index=False, sort=False, group_keys=False).apply(
        lambda x: x['_a'].rolling(d).corr(other=x['_b'], pairwise=True))

Is there a faster way to do this? i tried code but it does not work (got exceptions):

_g = df.groupby('name', as_index=False, sort=False, group_keys=False)
_g['_a'].rolling(5).corr(other=_g['_b'], pairwise=False)

Another question: how to ignore the group key (just keep the original index) for below code:

_g['_a'].rolling(d).min()