-
-
Notifications
You must be signed in to change notification settings - Fork 18.5k
PERF/REGR: isin slowdown for masked type #42714
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phofl
reviewed
Jul 25, 2021
jreback
reviewed
Jul 26, 2021
doc/source/whatsnew/v1.3.2.rst
Outdated
@@ -14,6 +14,7 @@ including other versions of pandas. | |||
|
|||
Fixed regressions | |||
~~~~~~~~~~~~~~~~~ | |||
- Performance regression in :meth:`DataFrame.isin` and :meth:`Series.isin` for nullable data types (:issue:`42708`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you change to this PR number
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed
thanks @mzeitlin11 |
@meeseeksdev backport 1.3.x |
meeseeksmachine
pushed a commit
to meeseeksmachine/pandas
that referenced
this pull request
Jul 26, 2021
Something went wrong ... Please have a look at my logs. |
simonjayhawkins
pushed a commit
that referenced
this pull request
Jul 26, 2021
Co-authored-by: Matthew Zeitlin <[email protected]>
debnathshoham
pushed a commit
to debnathshoham/pandas
that referenced
this pull request
Jul 27, 2021
CGe0516
pushed a commit
to CGe0516/pandas
that referenced
this pull request
Jul 29, 2021
feefladder
pushed a commit
to feefladder/pandas
that referenced
this pull request
Sep 7, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
isin
isin method
NA - MaskedArrays
Related to pd.NA and nullable extension arrays
Performance
Memory or execution speed performance
Regression
Functionality that used to work in a prior pandas version
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
xref #42708
Fixes
isin
slowdowns mentioned in https://simonjayhawkins.github.io/fantastic-dollop/#regressions?sort=3&dir=desc&branch=1.3.xBenchmarks:
Not sure if we do this anywhere else, but based on this slowdown seems like
empty/zeros_like(extensionarr)
should be avoided (this call has extra overhead through dispatching, but also triggers anastype
somewhere, which is the main perf difference).Also as a sidenote seems like
empty_like/zeros_like
should be avoided in general in favor ofzeros/empty
, see https://stackoverflow.com/questions/27464039/why-the-performance-difference-between-numpy-zeros-and-numpy-zeros-likeBut
zeros_like
->zeros
makes a much smaller difference thanzeros_like(self)
->zeros_like(self._data)