BUG: pd.Grouper cannot be reused in some cases

[mimakaev]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import numpy as np 
import pandas as pd 
np.random.seed(1)
# Create one dataframe 
dr = pd.date_range("2021-01-01", "2021-08-30")
ind = np.random.randint(0,5,len(dr))
something = np.random.random(len(ind))
df = pd.DataFrame({"date":dr, "ind":ind, "something":something})

# Define what to group on 
GROUP_ON = ["ind", pd.Grouper(key="date", freq="2W")]
# Copy used if you uncomment lines below 
GROUP_ON_COPY = ["ind", pd.Grouper(key="date", freq="2W")]

# Comment either of the two lines below, and it works 
df.groupby(GROUP_ON).mean().reset_index().groupby(GROUP_ON).mean()
df.groupby(GROUP_ON).mean().reset_index()

# Use a different groupby in the second case, and it works 
#df.groupby(GROUP_ON).mean().reset_index().groupby(GROUP_ON).sum()
#df.groupby(GROUP_ON_COPY).mean().reset_index()

# Bonus points: do only one groupby first, and it works 
# df.groupby(GROUP_ON).mean().reset_index()
# df.groupby(GROUP_ON).mean().reset_index()

Problem description

In some cases it seems that pd.Grouper cannot be reused. I was not able to figure out exactly why, or if the output can sometimes corrupt. It leads to very painful debugging.

Tested with master branch, and with python 3.8 and 3.9

Expected Output

pd.Grouper should be able to be used any number of times, or should raise explicit error upon any reuse.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.4.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.0-19041-Microsoft
Version : #488-Microsoft Mon Sep 01 13:43:00 PST 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.2
setuptools : 57.0.0
Cython : 0.29.23
pytest : 6.2.4
hypothesis : None
sphinx : 4.0.2
blosc : None
feather : None
xlsxwriter : 1.4.3
lxml.etree : 4.6.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.24.1
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : 2021.06.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : None

[mzeitlin11]

Thanks for reporting this @mimakaev, I see this on current master as well. Agree that either of your alternatives would be much better - would be good to know if the lack of reusability is a bug or just implicit by design. Worth mentioning there has been discussion about deprecating this (#41297) in favor of alternatives mentioned in that issue (which may serve as a workaround for your use case).

[NowanIlfideme]

It turns out this bug was the root cause of a completely different issue, "different shapes don't align" within some internal codes when grouping by datetime. It seems to be caching these codes for performance reasons, not expecting to be reused in a later context. It's likely that this even silently caused improper data after grouping!