BUG: df.groupby().resample()[[cols]] without key columns raise KeyError

[GYHHAHA]

• closes BUG: df.groupby().resample()[[cols]] without key columns raise KeyError #47362
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v1.5.0.rst file if fixing a bug or adding a new feature.

[pep8speaks]

Hello @GYHHAHA! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-07-06 05:46:26 UTC

[GYHHAHA]

The reason why the current result only returns one column is _cython_agg_general blocks the non-numeric dtype when calling the aggregate from resample.

pandas/pandas/core/resample.py

Line 1266 in a9a496c

result = obj.groupby(self.grouper, axis=self.axis).aggregate(how, **kwargs)

pandas/pandas/core/groupby/groupby.py

Line 1662 in a9a496c

result = self._cython_agg_general(

I think we should allow the aggregation on datetime-like columns when corresponding key(s) is/are included, but this will change the current groupby behavior (and I'm not sure which one is desired). Also this may be related to #47177 , not clear about the logic.

>>> df = pd.DataFrame(
    data={
        "date": pd.date_range(start="2016-01-01", periods=8),
        "group": [0, 0, 0, 0, 1, 1, 1, 1],
    }
)
>>> df.groupby("group")[["date", "group"]].mean()
# original
       group
group
0        0.0
1        1.0
# changed
                     date group
group
0     2016-01-02 12:00:00   0.0
1     2016-01-06 12:00:00   0.0

cc @mroeschke @rhshadrach

[GYHHAHA]

I find a simple solution is keep the resample exclusions empty:

pandas/pandas/core/resample.py

Line 169 in 9612375

self.exclusions = frozenset([self.groupby.key])

But this fails the following test:

pandas/pandas/tests/resample/test_resampler_grouper.py

Line 446 in 9612375

def test_resample_groupby_agg_object_dtype_all_nan(consolidate):

I personally think the test is not reasonable since we do want to get all the valid aggreation column results, thus the expect should have shape of (6, 4) instead of (6, 3). And as mentioned in the above pr, the default behavior will be changed in the future. A little confused. Discussion needed.

[GYHHAHA]

Since now we have DEPR the numeric_only on agg for resample and groupby, we should also DEPR the agg on groupby.resample agg function, which is currently not. And also the numeric_only keyword is still not work.

>>>df=pd.DataFrame({"a":[0,0,1,1], "b":pd.date_range("20200101", "20200104")})
>>>df.groupby("a").resample("D", on="b")[["a", "b"]].mean()
# should DEPR but not
                a
a b
0 2020-01-01  0.0
  2020-01-02  0.0
1 2020-01-03  1.0
  2020-01-04  1.0
>>>df.groupby("a").resample("D", on="b")[["a", "b"]].mean(numeric_only=False)
# should include "b" column agg result but not
                a
a b
0 2020-01-01  0.0
  2020-01-02  0.0
1 2020-01-03  1.0
  2020-01-04  1.0

[jreback]

@GYHHAHA #47605 (comment) is fine to do but another PR

[GYHHAHA]

For the current BUG, I think the fix is enough. I can not add the test for ["val", "date"] for this one since the result will be influenced by latter DEPR pr. @jreback

[mroeschke]

The current fix looks reasonable cc @rhshadrach

[rhshadrach]

lgtm

[mroeschke]

Thanks @GYHHAHA