API: should SeriesGroupby.apply overwrite result name?

[MarcoGorelli]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

def func(ser):
    return ser.rename('foo')

print(df.groupby('a')['b'].apply(lambda ser: ser.rename('foo')))

Issue Description

I get

a   
1  0    4
2  1    5
3  2    6
Name: b, dtype: int64

Expected Behavior

I would've expected

a   
1  0    4
2  1    5
3  2    6
Name: foo, dtype: int64

Looks like this is set very explicitly though:

pandas/pandas/core/groupby/generic.py

Line 404 in 2f751ad

result.name = self.obj.name

Is this right?

Trying to do something similar with DataFrameGroupby.apply gives

>>> df.groupby('a')[['b']].apply(lambda ser: ser.rename(columns=lambda _: 'foo'))
     foo
a       
1 0    4
2 1    5
3 2    6

which is more like what I was expecting

Related

#46369

---

@rhshadrach thoughts here?

Installed Versions

INSTALLED VERSIONS

commit : 3b09765
python : 3.8.15.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.0.0.dev0+759.g3b09765bc9
numpy : 1.23.5
pytz : 2022.6
dateutil : 2.8.2
setuptools : 65.5.1
pip : 22.3.1
Cython : 0.29.32
pytest : 7.2.0
hypothesis : 6.58.0
sphinx : 4.5.0
blosc : None
feather : None
xlsxwriter : 3.0.3
lxml.etree : 4.9.1
html5lib : 1.1
pymysql : 1.0.2
psycopg2 : 2.9.3
jinja2 : 3.0.3
IPython : 8.6.0
pandas_datareader: 0.10.0
bs4 : 4.11.1
bottleneck : 1.3.5
brotli :
fastparquet : 2022.11.0
fsspec : 2021.11.0
gcsfs : 2021.11.0
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.3
odfpy : None
openpyxl : 3.0.10
pandas_gbq : 0.17.9
pyarrow : 9.0.0
pyreadstat : 1.2.0
pyxlsb : 1.0.10
s3fs : 2021.11.0
scipy : 1.9.3
snappy :
sqlalchemy : 1.4.44
tables : 3.7.0
tabulate : 0.9.0
xarray : 2022.11.0
xlrd : 2.0.1
zstandard : 0.19.0
tzdata : None
qtpy : None
pyqt5 : None
None

[rhshadrach]

Agreed a user could find the result in the example above is unexpected. However I wonder if there are use cases where the name gets set naturally via pandas operations within the UDF but the current behavior of returning the name of the column is desired. For example, doing

df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]})
gb = df.groupby('a')['b']
result = gb.cumsum()

result has the name 'b' and this is the same if one instead does gb.apply(lambda x: x.cumsum()). This is complicated by the fact that the name of the Series passed to the UDF are the group values (1 and 2 in the above example).

Thinking about various return cases for name - the group value, None, and all others, along with the possibility of different groups returning different cases (e.g for one group the result is named None, another is named the group value, and a third a different name altogether), it seems to me trying to do anything more advanced is trying to guess at what the user might want. Perhaps we're likely to get more cases right, but at the expense of having a more complex description of what the resulting name is.

In situations like this, my preference would be to have a simple and easy to predict result - e.g. always the name of the Series. However I'm certainly willing to entertain other suggestions, and I'm still thinking about the difference in behavior between DataFrameGroupBy and SeriesGroupBy as highlighted in the OP.

[MarcoGorelli]

Thanks for sharing your thoughts!

In situations like this, my preference would be to have a simple and easy to predict result - e.g. always the name of the Series.

Yeah, if it's intentional then this sounds like a nice rule. To explain how I came across it, it was in the context of #49912 - would you be OK with setting

    # https://github.com/pandas-dev/pandas/issues/49909
    expected = expected.rename("count")

in the test that applies value_counts both as df.groupby(col1)[col2].value_counts() and df.groupby(col1)[col2].apply(Series.value_counts)? If we accept the above rule, then it'd be expected that the results would differ (and IMO that'd be OK)

[rhshadrach]

would you be OK with... If we accept the above rule, then it'd be expected that the results would differ (and IMO that'd be OK)

Yes - most certainly. In general, we don't expect DataFrameGroupBy.apply(lambda x: x.method()) to agree with DataFrameGroupBy.method(). The former has to operate with the UDF as a black box whereas the latter can take the operation itself into account in order to produce better output.

[MarcoGorelli]

makes sense, thanks! closing then