Skip to content

PERF: Improve performance when accessing GroupBy.groups #53088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 5, 2023
Merged

PERF: Improve performance when accessing GroupBy.groups #53088

merged 3 commits into from
May 5, 2023

Conversation

phofl
Copy link
Member

@phofl phofl commented May 4, 2023

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Tests added and passed if fixing a bug or adding a new feature
  • All code checks passed.
  • Added type annotations to new arguments/methods/functions.
  • Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
df = DataFrame(np.random.randint(1, 100, (5_000_000, 5)))
df.groupby([0, 1]).groups

# main
%timeit df.groupby([0, 1]).groups
1.68 s ± 7.52 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# pr
%timeit df.groupby([0, 1]).groups
504 ms ± 9.24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

@phofl phofl added Groupby Performance Memory or execution speed performance labels May 4, 2023
jbrockmendel
jbrockmendel reviewed May 4, 2023
View reviewed changes
pandas/core/groupby/ops.py Outdated
@@ -47,6 +47,7 @@
ensure_uint64,
is_1d_only_ea_dtype,
)
from pandas.core.dtypes.inference import is_array_like
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use a specific isinstance check instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, we can go the other way round

@mroeschke mroeschke added this to the 2.1 milestone May 5, 2023
@mroeschke mroeschke merged commit b0140bf into pandas-dev:main May 5, 2023
@mroeschke
Copy link
Member

Thanks @phofl

@phofl phofl deleted the perf_groupby_groups branch May 5, 2023 18:00
topper-123 pushed a commit to topper-123/pandas that referenced this pull request May 7, 2023
@phofl @topper-123
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
1a7bb85 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
topper-123 pushed a commit to topper-123/pandas that referenced this pull request May 7, 2023
@phofl @topper-123
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
0764b43 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
topper-123 pushed a commit to topper-123/pandas that referenced this pull request May 9, 2023
@phofl @topper-123
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
8ed1091 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
Rylie-W pushed a commit to Rylie-W/pandas that referenced this pull request May 19, 2023
@phofl
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
4752a21 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
topper-123 pushed a commit to topper-123/pandas that referenced this pull request May 27, 2023
@phofl @topper-123
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
cb7fbe9 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
topper-123 pushed a commit to topper-123/pandas that referenced this pull request May 27, 2023
@phofl @topper-123
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
22f5238 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
Daquisu pushed a commit to Daquisu/pandas that referenced this pull request Jul 8, 2023
@phofl
PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…
b950e19 
…3088)

* PERF: Improve performance when accessing GroupBy.groups

* Update v2.1.0.rst

* Fix
@taytzehao taytzehao mentioned this pull request Aug 14, 2023
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbrockmendel jbrockmendel jbrockmendel left review comments

@mroeschke mroeschke mroeschke approved these changes

Assignees
No one assigned
Labels
Groupby Performance Memory or execution speed performance
Projects
None yet
Milestone
2.1
Development

Successfully merging this pull request may close these issues.
3 participants
@phofl @mroeschke @jbrockmendel