Skip to content

PERF: merge on sorted MultiIndex #48504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 19, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions asv_bench/benchmarks/join_merge.py
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,21 @@ def time_multiby(self, direction, tolerance):
)


class MergeMultiIndex:
def setup(self):
n = 100_000
mi1 = MultiIndex.from_arrays([np.arange(0, n)] * 2)
mi2 = MultiIndex.from_arrays([np.arange(1, n + 1)] * 2)
self.df1 = DataFrame({"col1": 1}, index=mi1)
self.df2 = DataFrame({"col2": 2}, index=mi2)

def time_merge_sorted_multiindex(self):
# copy to avoid MultiIndex._values caching
df1 = self.df1.copy()
df2 = self.df2.copy()
merge(df1, df2, how="left", left_index=True, right_index=True)


class Align:
def setup(self):
size = 5 * 10**5
Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.6.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ Performance improvements
- Performance improvement in :meth:`.GroupBy.mean` and :meth:`.GroupBy.var` for extension array dtypes (:issue:`37493`)
- Performance improvement for :meth:`Series.value_counts` with nullable dtype (:issue:`48338`)
- Performance improvement for :class:`Series` constructor passing integer numpy array with nullable dtype (:issue:`48338`)
- Performance improvement in :func:`merge` and :meth:`DataFrame.join` when joining on a sorted :class:`MultiIndex` (:issue:`48504`)
- Performance improvement for :meth:`MultiIndex.unique` (:issue:`48335`)
-

Expand Down
6 changes: 2 additions & 4 deletions pandas/core/indexes/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -4699,13 +4699,11 @@ def join(
else:
return self._join_non_unique(other, how=how)
elif (
# GH48504: exclude MultiIndex to avoid going through MultiIndex._values
self.is_monotonic_increasing
and other.is_monotonic_increasing
and self._can_use_libjoin
and (
not isinstance(self, ABCMultiIndex)
or not any(is_categorical_dtype(dtype) for dtype in self.dtypes)
)
and not isinstance(self, ABCMultiIndex)
and not is_categorical_dtype(self.dtype)
):
# Categorical is monotonic if data are ordered as categories, but join can
Expand Down