API: way to exclude the grouped column with apply

[jreback]

http://stackoverflow.com/questions/23709811/best-way-to-sum-group-value-counts-in-pandas/23712433?noredirect=1#comment36441762_23712433
any of these look palatable?

df.groupby('c',as_index='exclude').apply(....)
df.groupby('c')['~c'].apply(...)
df.groupby('c',filter='c')
df.groupby('c',filter=True)

rather than doing a negated selection (df.columns-['c'])

df = pd.DataFrame.from_dict({'b1':['aa','ac','ac','ad'],\
                             'b2':['bb','bc','ad','cd'],\
                             'b3':['cc','cd','cc','ae'],\
                             'c' :['a','b','a','b']})

df.groupby('c')[df.columns - ['c']].apply(lambda x: x.unstack().value_counts())
Out[92]: 
c    
a  cc    2
   bb    1
   ad    1
   ac    1
   aa    1
b  cd    2
   ad    1
   ae    1
   ac    1
   bc    1
dtype: int64

[jbrockmendel]

@rhshadrach some of what you're working on with selected_obj vs obj_with_exclusions might be related?

[rhshadrach]

Throughout most of groupby, the grouping column is not included when computing the result. groupby(...).apply is the one exception I know of (maybe there are others?), and here, we first try it with the groupings and if that fails, we try it without the groupings. This makes apply inconsistent with both itself and other methods.

I think we should always excluding the groupings, and plan to put up a deprecation of this as part of 2.x.

[topper-123]

The example in OP doesn't work any more (even adjusted for the - list issue):

In [14]: df = pd.DataFrame.from_dict({'b1':['aa','ac','ac','ad'],\
   ...:                              'b2':['bb','bc','ad','cd'],\
   ...:                              'b3':['cc','cd','cc','ae'],\
   ...:                              'c' :['a','b','a','b']})
   ...:
   ...: df.groupby('c')[df.columns.drop('c')].apply(lambda x: x.unstack().value_counts())
TypeError: Series.name must be a hashable type

This must be a bug?

For reference, I got it to work like this:

In [15]: df[df.columns.drop('c')].groupby(df['c']).apply(lambda x: x.unstack().value_counts())
Out[15]:
c
a  cc    2
   aa    1
   ac    1
   bb    1
   ad    1
b  cd    2
   ac    1
   ad    1
   bc    1
   ae    1
dtype: int64

Which is not a nice API and the first example should work IMO.

[topper-123]

I think it look reasonable to deprecate keeping the grouping keys in the groupby.apply to keep the method similar to the other groupby methods. I'd say in 99,5 % if the time, that is what users would want/expect.

Keeping the deprecation process reasonable can be tricky (avoid warnings all over is a concern). How would we deprecate this?

[rhshadrach]

I have a branch to deprecate that I think is all set to go. It looks quite noisy if you look at our tests, but would only warn the user if:

• apply with the groupings is successful
• There is actually difference between _selected_obj and _obj_with_exclusions

To avoid the warning, users can exclude the groupings, e.g.

df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]})
gb = df.groupby('a')

# Raises warning
gb.apply(len)

# No warning
gb[['b']].apply(len)

[rhshadrach]

Reopening since #52477 was reverted.

[rhshadrach]

xref: #52477 (comment)

As the deprecation was determined to be noisy, I plan to reintroduce the linked PR once 2.1 is released with an option (mode.groupings_in_apply) to allow users to adopt future behavior and silence the warning. It will default to True; setting to False gives future behavior. The option will then be deprecated in 3.x.

@phofl - does this sound reasonable?

[madelavar12]

I feel like I must be misunderstanding the fix for this issue because I am now running into the opposite of this issue. I am wanting to use the groupby functionality to be able to apply functions to specific groups of data and return the entire dataframe. The current functionality provides that but sounds like it is being deprecated in pandas 3.0. Is there going to be a way to include the groupby column without explicitly subsetting all of the columns?

[rhshadrach]

@madelavar12 - could you open a new issue and provide an example of the computation you wish to perform.