PERF: Use Indexers to implement groupby rolling

[mroeschke]

• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff

Currently, grouby.rolling is implemented essentially as groupby.apply(lambda x: x.rolling()) which can be potentially slow.

This PR implements groupby.rolling by calculating bounds with a GroupbyRollingIndxer and using the rolling aggregations in cython to compute the results.

[mroeschke]

Here are preliminary benchmarks. The performance so far is fairly similar. I suspect that the fact that I have to reconstruct the resulting index is killing the performance

With this benchmark:

In [1]: n = 1000
   ...: df = pd.DataFrame({"A": [str(i) for i in range(n)] * 10, "B": list(range(n)) * 10})
   ...: g = df.groupby("A").rolling(window=2)
   ...: %timeit g.sum()
1.5 s ± 5.77 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <-- PR
1.52 s ± 49 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <-- master

[jreback]

try with a longer window as well
also profile this should be way faster

[mroeschke]

I was accidentally still dispatching to the old implementation in this PR, here are the performance results with that removed

-- PR
In [2]: %timeit g.sum()
61.2 ms ± 4.61 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit g.sum()
71.9 ms ± 10.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: %timeit g.sum()
57.9 ms ± 714 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)


-- master
In [2]: %timeit g.sum()
1.42 s ± 9.08 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit g.sum()
1.44 s ± 26.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit g.sum()
1.48 s ± 73.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[jreback]

can you enhance the benchamarks for this (if we don't have?)

also if you can add a whatsnew note.

[pep8speaks]

Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-05-21 04:47:41 UTC

[mroeschke]

@jreback ready for another look

[jreback]

can you add a timebased one as well

[jreback]

do we have a way of hitting the api here?

[jreback]

maybe better to call super()._create_blocks(obj) (e.g. add obj as an optional arg that defaults to self._selected_obj)

[mroeschke]

Final benchmarks:

# rolling over 1000 groups
In [1]: In [1]: n = 1000
   ...:    ...: df = pd.DataFrame({"A": [str(i) for i in range(n)] * 10, "B": list(range(n)) * 10})
   ...:    ...: g = df.groupby("A").rolling(window=2)

-- master
In [2]: %timeit g.sum()
1.38 s ± 10.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit g.sum()
1.4 s ± 13.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit g.sum()
1.37 s ± 6.29 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

--- PR
In [2]: %timeit g.sum()
63.1 ms ± 396 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit g.sum()
70.1 ms ± 8.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: %timeit g.sum()
63.7 ms ± 471 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

[jreback]

looks good, some doc comment requests. ping on green.

[jreback]

can you add some comments here on what you are doing

[jreback]

if you can type: Union[Series, DataFrame] (I think we have an annoation for that).

[jreback]

if you can add some commments here on what you are doing

[jreback]

if you can type

[mroeschke]

@jreback Ping all green

[jreback]

thanks @mroeschke very nice!

[ddofer]

Does this affect also "Expanding"?

[mroeschke]

No this currently doesn't apply to expanding. PR's to make it apply to expanding welcome!