BUG: groupby agg fails silently with mixed dtypes

[debnathshoham]

• closes BUG: groupby aggregation methods produce empty DataFrame with mixed DataTypes #43209
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[simonjayhawkins]

is the expected also a mixed dtype dataframe?

[debnathshoham]

here the expected would be either entirely Int64 or int64

[simonjayhawkins]

is this test representative of the reported issue?

[debnathshoham]

yes, it's the exact same example (mixed int one)

[jreback]

cc @rhshadrach @jbrockmendel for a look

[jreback]

ok 2 comments @debnathshoham pls ping when reasolved & green

[debnathshoham]

@jreback added comments to both places + Green

[simonjayhawkins]

Thanks @debnathshoham I'm not sure that this should be backported as I don't understand the reasoning for the change in behavior. see #43213 (comment)

simplified code sample

import pandas as pd

print(pd.__version__)
df = pd.DataFrame([[1, 2, 3, 4, 5, 6]] * 3)
df.columns = pd.MultiIndex.from_product([["a", "b"], ["i", "j", "k"]])
for col in df.columns:
    if col[-1] == "j":
        df[col] = df[col].astype("Int64")
print("mixed int:")
result = df.groupby(level=1, axis=1).sum()
print(result)
print(result.dtypes)

on 1.2.5

1.2.5
mixed int:
   i    j  k
0  5  7.0  9
1  5  7.0  9
2  5  7.0  9
i      int64
j    float64
k      int64
dtype: object

this PR

1.4.0.dev0+537.g8d1bfb1cb9
mixed int:
   i  j  k
0  5  7  9
1  5  7  9
2  5  7  9
i    Int64
j    Int64
k    Int64
dtype: object

Can you explain why the dtypes of columns i and k are now Int64

[debnathshoham]

sorry, I didn't follow it in the first comment.

Actually in master, I don't think dtype of the output is being maintained in any of these funcs
'count', 'sum', 'std', 'var', 'sem', 'mean', 'median', 'prod', 'min', 'max'

Is the expectation that the output will have the original dtypes, i.e. int64, Int64, int64 ?

[simonjayhawkins]

Is the expectation that the output will have the original dtypes, i.e. int64, Int64, int64 ?

for the code sample, that would be reasonable, although returning float64 for the Int64 columns would probably also be acceptable as that was the return type before the regression.

How does the changes in PR relate to the changes in the PR that caused the regression #41706? (or to put it a different way, which change in #41706 caused the regression and how does this PR rectify that?)

[debnathshoham]

Before #41706, the actual calculation was happening in agg_general in result = self.aggregate(lambda x: npfunc(x, axis=self.axis)).
Honestly, I didn't try to undo what that patch did, I went on to fix the inconsistency in the result coming post that patch.

[rhshadrach]

Agreed @debnathshoham - on master, the dtypes in @simonjayhawkins' example come out at all float. What you have here is certainly an improvement over that, but it doesn't fully fix the regression. It's unclear to me if this patch is a good step in fully resolving the regression, or if fully solving it would make this patch obsolete. Will need to investigate more.

[jbrockmendel]

Doesn't the PR that changes DataFrame.transpose fix this? That seems like a much better option than reverting #41706, which fixed a bunch of bugs.

Again shout it from the rooftops: this is caused by the lack of 2D EAs.

[debnathshoham]

yes, but that was casting the the result df to the common_dtype (which wasn't matching 1.2.5 behaviour)

[jreback]

can you show what these test functions return in 1.2.5 vs what this PR gives. and show any difference in a comment.

[debnathshoham]

added comments in the tests.
To summarise, for df with mixed dtypes like below.
For sum - Columns with Int64 was cast to float64 in 1.2.5, whereas in this PR it would cast to int64.
For std - ValueError : Length mismatch in 1.2.5. In this PR they come as float64.
Fot var - DataError : No numeric types to aggregate in 1.2.5. In this PR they come as float64

In [32]:     df = DataFrame(
    ...:         np.arange(12).reshape(3, 4),
    ...:         index=Index([0, 1, 0], name="y"),
    ...:         columns=Index([10, 20, 10, 20], name="x"),
    ...:         dtype="int64",
    ...:     ).astype({10: "Int64"})

In [33]: df.dtypes
Out[33]: 
x
10    Int64
20    int64
10    Int64
20    int64
dtype: object

[debnathshoham]

@jreback @simonjayhawkins
wondering if you got a chance to look into this?
xref - #43501 (I think this is related)

[rhshadrach]

Minor request on the tests; otherwise this looks great to me. The code change is small and makes sense, and the dtypes are improved over 1.2.5.

[rhshadrach]

Can you remove the comments here - appreciate the extra detail, but they shouldn't be part of a test; surfacing this in a comment on github directly would be great (and you already have another comment that does something similar - thanks for that as well).

[jreback]

request & pls merge master ping on green.

[jreback]

move these to pandas/tests/groupby/aggregate/test_aggreagte.py (or maybe test_cython) see where similar are.

[debnathshoham]

@jreback Green

[jreback]

thanks @debnathshoham

[jreback]

@meeseeksdev backport 1.3.x

[lumberbot-app]

Something went wrong ... Please have a look at my logs.

[jbrockmendel]

im a little late here, but could potentially use _get_data_to_aggregate here