DOC: Clarify how groupby forms groups for mixed-value columns

[gabuzi]

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.groupby.html#pandas.DataFrame.groupby
https://pandas.pydata.org/docs/dev/user_guide/groupby.html#

Documentation problem

In my broad view of things, groupby can work on any (combination of) hashables, but there can be some unintuitive situations that should be mentioned in the docs.

E.g., the following should and does work (work as in there is no exception, but the results are not necessarily as expected):

>>> df = pd.DataFrame({'a': [False, 1, True, 'string'], 'b': [1, 2, 3, 4]})
        a  b
0   False  1
1       1  2
2    True  3
3  string  4

# let's groupby on 'a':
>>> df.groupby('a').describe()
           b
       count mean       std  min   25%  50%   75%  max
a
False    1.0  1.0       NaN  1.0  1.00  1.0  1.00  1.0
1        2.0  2.5  0.707107  2.0  2.25  2.5  2.75  3.0
string   1.0  4.0       NaN  4.0  4.00  4.0  4.00  4.0

We can see that the value 1 and True were grouped together. From a Python background, that is not unexpected, as 1 == True evaluates as True, i.e. 1 and True are equal and thus in the same group.

While this data layout may be questionable, it certainly happens in reality. I find myself often using (abusing?) groupby() on multiple cols to identify unique combinations of values, and in this case, I would not want 1 to be considered equal to True (or 0 equal to False and whichever other 'convenience' conversions Python does).

Suggested fix for documentation

I think the docs could benefit from mentioning this pitfall. The suggested workaround that I see would be to first convert the columns to strings before the groupby:

>>> df.groupby(df['a'].apply(str)).describe()
           b
       count mean std  min  25%  50%  75%  max
a
1        1.0  2.0 NaN  2.0  2.0  2.0  2.0  2.0
False    1.0  1.0 NaN  1.0  1.0  1.0  1.0  1.0
True     1.0  3.0 NaN  3.0  3.0  3.0  3.0  3.0
string   1.0  4.0 NaN  4.0  4.0  4.0  4.0  4.0

[rhshadrach]

Thanks for the report, I think a line in the notes section of the docstring on DataFrame.groupby and Series.groupby would be appropriate, something like:

The implementation of groupby is hash-based, meaning in particular that objects that compare as equal will be considered as the same group. An exception to this is that pandas has special handling of NA values: any NA values will be collapsed to a single group, regardless of how they compare. See the user guide for more details.

An example in the user guide and your suggested workaround could then be linked to in the last sentence.

Would you be interested in submitting a PR?

[gabuzi]

Thanks for the suggestion. I'd be happy to open a PR on this. Should I insert that here at current L97?

pandas/pandas/core/shared_docs.py

Lines 95 to 99 in bdc79c1

	used to group large amounts of data and compute operations on these
	groups.
	Parameters
	----------

Should I add an example?

I just have another follow-up question: When you say hash-based, I assume this means that it always uses __hash__()? __eq__() is never considered? Should this be mentioned? Maybe in the linked user guide? The implication is that a simple a == b test during debugging as I did above is not actually the right thing to look at (although proper code should maintain a.__eq__(b) == True => a.__hash__() == b.__hash__(), but bugs do happen...).

I also just realized that the string workaround obviously has its own drawbacks, as it would now group the string '1' and the int 1 as equals, but I guess we don't have to provide a general solution.

[rhshadrach]

Should I insert that here at current L97?

I think this should go in the notes section which is below See Also; L190 here.

When you say hash-based, I assume this means that it always uses hash()? eq() is never considered? Should this be mentioned?

No - hash is used to produce a non-necessarily unique but often distinct number; __eq__ is then used in the case of conflicts. This is why a requirement for writing a hash function is that two objects that are equal must have equal hashes.

I also just realized that the string workaround obviously has its own drawbacks, as it would now group the string '1' and the int 1 as equals, but I guess we don't have to provide a general solution.

That's a good point - this makes me quite hesitant to mention the workaround.