-
-
Notifications
You must be signed in to change notification settings - Fork 18.4k
PERF: tighten _should_compare for MultiIndex #42231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -5345,6 +5345,16 @@ def _get_indexer_non_comparable( | |||||
""" | ||||||
if method is not None: | ||||||
other = unpack_nested_dtype(target) | ||||||
if self._is_multi ^ other._is_multi: | ||||||
kind = other.dtype.type if self._is_multi else self.dtype.type | ||||||
raise TypeError( | ||||||
f"'<' not supported between instances of {kind} and 'tuple'" | ||||||
) | ||||||
elif self._is_multi and other._is_multi: | ||||||
assert self.nlevels != other.nlevels | ||||||
# Python allows comparison between tuples of different lengths, | ||||||
# but for our purposes such a comparison is not meaningful. | ||||||
raise TypeError("'<' not supported between tuples of different lengths") | ||||||
raise TypeError(f"Cannot compare dtypes {self.dtype} and {other.dtype}") | ||||||
|
||||||
no_matches = -1 * np.ones(target.shape, dtype=np.intp) | ||||||
|
@@ -5474,6 +5484,14 @@ def _should_compare(self, other: Index) -> bool: | |||||
|
||||||
other = unpack_nested_dtype(other) | ||||||
dtype = other.dtype | ||||||
if other._is_multi: | ||||||
if not self._is_multi: | ||||||
# other contains only tuples so unless we are object-dtype, | ||||||
# there can never be any matches | ||||||
return self._is_comparable_dtype(dtype) | ||||||
return self.nlevels == other.nlevels | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the change that is breaking MultiIndex broadcasting. If one has 3 levels and the other has 2, then this is False. Previously these were comparable and so would be compared and expanded. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thanks. do you know what the calling method is in the problematic case? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Walking back, previous is pandas/pandas/core/indexes/base.py Line 3481 in ddd90b0
then pandas/pandas/core/indexes/base.py Line 3887 in ddd90b0
Here There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The big change is driven by the return difference of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK, i think ive got a handle on whats going on here. The long-term fix will be in MultiIndex.get_indexer, but for now this should just be reverted. |
||||||
# TODO: we can get more specific requiring levels are comparable? | ||||||
|
||||||
return self._is_comparable_dtype(dtype) or is_object_dtype(dtype) | ||||||
|
||||||
def _is_comparable_dtype(self, dtype: DtypeObj) -> bool: | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do tests hit this? e.g. as you didn't change anything
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of the affected cases was not tested; just added a test for that case.