PERF: Fixed cut regression, improve Categorical #34952
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This has two changes to address a performance regression in cut / qcut. 1. Avoid an unnecessary `set` conversion in cut. 2. Aviod a costly conversion to object in the Categoriacl constructor for dtypes we don't have a hashtable for. ```python In [2]: idx = pd.interval_range(0, 1, periods=10000) In [3]: %timeit pd.Categorical(idx, idx) ``` ``` 10.4 ms ± 351 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 256 ms ± 5.85 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) 53.2 µs ± 1.26 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) ``` And for the qcut ASV ``` 58.5 ms ± 3.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 134 ms ± 9.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 29.8 ms ± 1.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) ``` Closes pandas-dev#33921
TomAugspurger
commented
Jun 23, 2020
pandas/core/reshape/tile.py
Outdated
| @@ -425,6 +426,7 @@ def _bins_to_cuts( | |||
| labels = _format_labels( | |||
| bins, precision, right=right, include_lowest=include_lowest, dtype=dtype | |||
| ) | |||
| labels_are_unique = True | |||
There was a problem hiding this comment.
Looking again, it may not be safe to assume that _format_labels returns a unique IntervalIndex like I thought. Might revert this for now.
TomAugspurger
added
Categorical
jreback
requested changes
Jun 23, 2020
| @@ -2611,6 +2611,11 @@ def _get_codes_for_values(values, categories): | |||
| values = ensure_object(values) | |||
| categories = ensure_object(categories) | |||
|
|
|||
| if isinstance(categories, ABCIndexClass): | |||
There was a problem hiding this comment.
hmm shouldn't this be in the caller to this routine?
There was a problem hiding this comment.
I think it's best here. We call it in ~3 places, and I think each would have to repeat this check & the object casting stuff.
pandas/core/reshape/tile.py
Outdated
| categories=labels if len(set(labels)) == len(labels) else None, | ||
| ordered=ordered, | ||
| ) | ||
| if labels_are_unique: |
There was a problem hiding this comment.
Sorry... I thought I reverted this. Forgot to push.
But, the set(labels) stuff is somewhat expensive (don't have the exact numbers right now). There may be cases where we know that we have unique labels and can avoid the check.
jreback
approved these changes
Jun 24, 2020
|
thanks @TomAugspurger |
fangchenli
pushed a commit
to fangchenli/pandas
that referenced
this pull request
Jun 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This has two changes to address a performance regression in
cut / qcut.
1. Avoid an unnecessary(reverted for now)setconversion in cut.2. Aviod a costly conversion to object in the Categoriacl constructor
for dtypes we don't have a hashtable for.
And for the qcut ASV
Closes #33921