Fixes Issues#44132, #40148, #29033, #22275, #18869: groupby

[ikramersh]

• closes BUG: groupby.sum removed columns in case sequential calls of several groupby.sum #44132
• closes BUG: groupby drops string dtype #40148
• closes Inconsistent behavior between df.sum() and groupby(col).agg('sum') on lists #29033
• closes Decimal fields dropped in group by with more than one column #22275
• closes Groupby doesn't call aggregation on empty groups #18869
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[pep8speaks]

Hello @ikramersh! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-11-10 04:53:05 UTC

[mroeschke]

Of note: it appears you are modifying a lot of existing tests / docs which is a sign that this fix is probably not the right approach. Usually a fix should only address the example in the associated issue.

[ikramersh]

Of note: it appears you are modifying a lot of existing tests / docs which is a sign that this fix is probably not the right approach. Usually a fix should only address the example in the associated issue.

I agree and would like some confirmation that the fix is desirable. It is a minute core change with a broad impact. To me it opens up a lot of opportunity because it's more pure Python where anything that can be summed is allowed, for example strings, tupples, lists, dictionaries etc are all included in results now. Previously strings for example were automatically excluded and not expected by the existing tests as results.

Please can I get some direction as to the desirability of this approach?

[jreback]

-1 on this change

it is unexpected from a user pov as all operations know ally to numeric columns

this can easily be done in nested columns with an .apply

[ikramersh]

-1 on this change

it is unexpected from a user pov as all operations know ally to numeric columns

this can easily be done in nested columns with an .apply

That makes sense too but the fix also makes use of the numeric_only=True argument more applicable now as well as the existing warnings that non numeric columns should be filtered out prior to calling numeric operations.

Happy to abort just explaining what I have discovered on this journey.

[ikramersh]

The reported issue #44132 shows that the inclusion of not only numeric but all summable types in the result of .sum() is already part of dataframe behaviour. The complaint is that behaviour is not carried through after calling groupby() which is what this fix attempts to resolve.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[jbrockmendel]

I think we already have a deprecation path in place for numeric_only=None, can close this.

[jreback]

closing as stale and #44142 (comment) comment