ENH add cumcount groupby method

[hayd]

closes #4646

Happy to hear ways to make this faster, but I think this is a good convenience function as is.

Update: made significantly faster...

[hayd]

As an aside, I was sure you used to do be able to do g.value_counts()... guess I was mistaken.

[jtratner]

@hayd that's an open issue and being re-enabled. You could reincorporate it here if necessary to make this PR works.

[jtratner]

mind removing this blank line if you're editing this?

[hayd]

I added it in specifically, pep8 says it should be there, right?

[jtratner]

okay, no idea.

[hayd]

I really there ought to be tests for dupe index too. In which case this implementation fails (!)

@jreback Is this a wider bug in apply alignment / or is there a cheeky way to do this without a sort:

In [52]: df = pd.DataFrame([['a'], ['a'], ['a'], ['b'], ['b'], ['a']], columns=['A'], index=[0] * 6)

In [53]: g = df.groupby('A')

In [54]: pd.concat(pd.DataFrame(np.arange(len(v)), v) for v in g.indices.values()).sort_index().set_index(df.index)
Out[54]: 
   0
0  0
0  1
0  2
0  0
0  1
0  3

In [55]: g.apply(lambda x: pd.Series(np.arange(len(x)), x.index))  # not correct 
Out[55]: 
0    0
0    1
0    2
0    3
0    0
0    1
dtype: int64

[hayd]

Refactored to work with dupes, also faster:

In [1]: import pandas as pd; import numpy as np

In [2]: df = pd.DataFrame(np.random.randint(0, 2, 1000), columns=['A'], index=[0] * 1000); g = df.groupby('A')

In [3]: %timeit g.apply(lambda x: pd.Series(np.arange(len(x)), x.index))
1000 loops, best of 3: 786 µs per loop

In [4]: %timeit pd.concat(pd.DataFrame(np.arange(len(v)), v) for v in g.indices.values()).sort_index().set_index(df.index)
1000 loops, best of 3: 568 µs per loop

In [5]: %timeit g.cumcount()
10000 loops, best of 3: 41.6 µs per loop


In [8]: df = pd.DataFrame(np.random.randint(0, 100, 1000), columns=['A'], index=[0] * 1000); g = df.groupby('A')

In [9]: %timeit g.apply(lambda x: pd.Series(np.arange(len(x)), x.index))
100 loops, best of 3: 7.35 ms per loop

In [10]: %timeit pd.concat(pd.DataFrame(np.arange(len(v)), v) for v in g.indices.values()).sort_index().set_index(df.index)
100 loops, best of 3: 10.7 ms per loop

In [11]: %timeit g.cumcount()
1000 loops, best of 3: 215 µs per loop

[hayd]

@jreback / @cpcloud too late for 0.13 ?

[jorisvandenbossche]

There has to be a blank line between "Essentially" and the code to render it as code I think (in html docstring).

[hayd]

Thanks, fixed :)

[jreback]

@hayd I think this is ok fine for 0.13. maybe a doc mention in groupby.rst (could be a separate PR)

[jtratner]

clever way to do it :)

[jreback]

I think you need to specify dtype here (to avoid issues on 32-bit plat), e.g. np.arange(len(v),dtype='int64')

[hayd]

Updated those to int64, good spot.

I'm not sure what/where to add in the groupby.rst (there is nothing about cumsum and friends either)... "Other useful features" ?

[jreback]

maybe create another section after Flexible apply, which apply examples?

[hayd]

Just appended a small mention of it at the end of groupby.rst. ...cumsum etc. are covered in dispatched (though could do more in the future). :S

[hayd]

This is just horrible

[hayd]

maybe "To see the order in which each row appears within its group, use the
cumcount method"

[hayd]

going to go ahead an merge this :)