Behavior change of 1.4.rc0 when group by apply returns a copy of the passed data frame

[aberres]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

df = pd.DataFrame([[1,2,3,4]], columns=["A", "B", "C", "D"])

grouped = df.groupby(['A', 'B'], as_index= True)[["C", "D"]]

combined = grouped.apply(lambda x: x.copy())

combined.reset_index().columns
# Pandas 1.3: Index(['A', 'B', 'level_2', 'C', 'D'], dtype='object')
# Pandas 1.4: Index(['index', 'C', 'D'], dtype='object')

Issue Description

While testing our code base against 1.4rc0 I stumbled upon a behavior change of Pandas 1.4.

Formerly when apply was called on a grouper it seems as if a returned multi-index was honored in the resulting data frame.
With 1.4 the index is dropped.

No idea if this is an intentional change or not. Wanted to let you know to be on the safe side.
also did not check why or code does what it does (the method called in apply is pretty complex, not just a copy). Looks a bit fishy...

Expected Behavior

Unchanged behavior.

Installed Versions

Replace this line with the output of pd.show_versions()

[aberres]

It seems my analysis is not really correct.

What is passed to the apply func seems to be unchanged. It is just that formerly returning a copy created a frame with a multi-index while this does not happen anymore?

The behavior when just returning what is passed (grouped.apply(lambda x: x)) seems to be unchanged. So yes, aligning the behavior makes sense I guess.

[jreback]

cc @rhshadrach if u could have a quick look

[simonjayhawkins]

While testing our code base against 1.4rc0 I stumbled upon a behavior change of Pandas 1.4.

first bad commit: [d037ff6] REF: remove libreduction.apply_frame_axis0 (#42992)

cc @jbrockmendel

[rhshadrach]

@aberres - thanks for the report! The change here was not intentional, in that it was an behavior change that was not noticed until after it was merged and without a deprecation notice. However, it does move pandas in the right direction of consistency, and would be very hard to revert, so I am +1 on moving forward with the behavior in 1.4 as-is.

apply was meant to be flexible, handling aggregations, transforms, and everything in between. The result is different when apply detects an operation was a transform. Previously, in 1.3, pandas used two different definitions of whether something was a transform (i.e. whether the index was "mutated") - one code path used "is" whereas another code path used "equals" when determining whether the input index is the same as the output index. The PR highlighted by @simonjayhawkins removed this inconsistency.

Being able to control what happens with the group keys in apply (regardless of whether pandas detects a transform) would be resolved by #34998.

[simonjayhawkins]

Thanks @rhshadrach should we close this or leave open until #34998 is merged and move off 1.4 milestone?

[simonjayhawkins]

The change here was not intentional, in that it was an behavior change that was not noticed until after it was merged and without a deprecation notice.

Thinking some more, it's not too late to add something to the release notes about the breaking change.

[rhshadrach]

That sounds good @simonjayhawkins - I should be able to get a PR up by the end of today.

[rhshadrach]

Closing this as the remark about allowing user control of the group keys being resolved by #34998 is duplicative of e.g. #34809.