Skip to content

BUG: groupby behavior changed since 1.4.x #46893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
Yikun opened this issue Apr 29, 2022 · 5 comments
Closed
3 tasks done

BUG: groupby behavior changed since 1.4.x #46893

Yikun opened this issue Apr 29, 2022 · 5 comments
Labels
Apply Apply, Aggregate, Transform, Map Bug Duplicate Report Duplicate issue or pull request Groupby

Comments

@Yikun
Copy link
Contributor

Yikun commented Apr 29, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# After 1.4.x
>>> import pandas as pd
>>> pdf = pd.DataFrame({"timestamp": [0.0], "car_id": ["A"]})
>>> pdf.groupby("car_id").apply(lambda _: pd.DataFrame({"column": [0.0]})).index
Int64Index([0], dtype='int64')
>>> pdf.groupby("car_id").apply(lambda _: pd.DataFrame({"column": [0.0]}))
   column
0     0.0

Issue Description

groupby behavior changed since 1.4.x

Expected Behavior

# Before 1.4.x
>>> import pandas as pd
>>> pdf = pd.DataFrame({"timestamp": [0.0], "car_id": ["A"]})
>>> pdf.groupby("car_id").apply(lambda _: pd.DataFrame({"column": [0.0]})).index
MultiIndex([('A', 0)],
           names=['car_id', None])
>>> pdf.groupby("car_id").apply(lambda _: pd.DataFrame({"column": [0.0]}))
          column
car_id
A      0     0.0

Installed Versions

1.4.x

@Yikun Yikun added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 29, 2022
@Yikun
Copy link
Contributor Author

Yikun commented Apr 29, 2022

First bad commit: d037ff6

cc @jbrockmendel

@twoertwein
Copy link
Member

Potentially a duplicate of #45315

@jbrockmendel
Copy link
Member

The bisected commit removed apply_frame_axis0, which was a fastpath that behaved subtly differently from the non-fastpath in ways that we were never really able to consistently debug. The non-fastpath (new and current) behavior is the canonically correct behavior.

@rhshadrach
Copy link
Member

In 1.5, you'll be able to specify the group_keys argument to groupby, and apply will adhere to it.

@rhshadrach rhshadrach added Groupby Apply Apply, Aggregate, Transform, Map Closing Candidate May be closeable, needs more eyeballs and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 30, 2022
@simonjayhawkins
Copy link
Member

Potentially a duplicate of #45315

closing as duplicate and resolved by calling out this breaking change in the 1.4 release notes https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.4.0.html#groupby-apply-consistent-transform-detection

@simonjayhawkins simonjayhawkins added Duplicate Report Duplicate issue or pull request and removed Closing Candidate May be closeable, needs more eyeballs labels May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Apply Apply, Aggregate, Transform, Map Bug Duplicate Report Duplicate issue or pull request Groupby
Projects
None yet
Development

No branches or pull requests

5 participants