Skip to content

PERF: MultiIndex.argsort / MultiIndex.sort_values #48406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 9, 2022

Conversation

lukemanley
Copy link
Member

@lukemanley lukemanley commented Sep 6, 2022

Perf improvement in MultiIndex.argsort and MultiIndex.sort_values for the vast majority of cases (where MultiIndex.argsort is called without additional passthrough numpy parameters).

Also, an xfail was removed because np.lexsort does not complain about complex dtype arrays whereas np.argsort will complain about an object array of tuples containing complex and non-complex dtypes.

ASV added:

       before           after         ratio
     [fa211d47]       [b2fea756]
     <main>           <perf-multiindex-argsort>
-      81.0±0.9ms       27.9±0.7ms     0.34  multiindex_object.SortValues.time_sort_values('Int64')
-        96.9±2ms       2.00±0.2ms     0.02  multiindex_object.SortValues.time_sort_values('int64')

@lukemanley lukemanley added Performance Memory or execution speed performance MultiIndex labels Sep 6, 2022
Copy link
Member

@phofl phofl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

Could you add a Benchmark for Nullable dtypes too?

@lukemanley
Copy link
Member Author

Could you add a Benchmark for Nullable dtypes too?

Added a benchmark for nullable Int64 and updated the timings in the summary. However, the nullable index used in the asv does not contain any pd.NA values as numpy will raise if it encounters pd.NA. That is true for numpy.argsort (with a list of tuples) and numpy.lexsort so this isn't new behavior. Can handling pd.NA be an enhancement/bug fix for a separate PR?

Copy link
Member

@phofl phofl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mroeschke mroeschke merged commit 769c242 into pandas-dev:main Sep 9, 2022
@mroeschke mroeschke added this to the 1.6 milestone Sep 9, 2022
@mroeschke
Copy link
Member

Thanks @lukemanley

@lukemanley lukemanley deleted the perf-multiindex-argsort branch September 10, 2022 00:49
@mroeschke mroeschke modified the milestones: 1.6, 2.0 Oct 13, 2022
noatamir pushed a commit to noatamir/pandas that referenced this pull request Nov 9, 2022
* perf improvement in MultiIndex.argsort and MultiIndex.sort_values

* whatsnew

* add asv for Int64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MultiIndex Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants