-
-
Notifications
You must be signed in to change notification settings - Fork 18.4k
PERF: Use generator expression for Blocks.replace_list #50778
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
pandas/core/internals/blocks.py
Outdated
else: | ||
# GH#38086 faster if we know we dont need to check for regex | ||
masks = [missing.mask_missing(values, s[0]) for s in pairs] | ||
masks = ( | ||
extract_bool_array(missing.mask_missing(values, s[0])) for s in pairs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a chance of an API change here (not necessarily a bad one) with something like
src_list = [1, 2, 3]
dest_list = [2, 3, 4]
values = np.array([1, 2, 3])
IIUC in the status quo we end up with [2, 3, 4]
but with this we'd end up with all-4s?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, looks like this would be invalid for inplace=True. Good catch!
I'll also add a test case, since it looks like this isn't caught by CI.
Might address #6697? |
Only partially. This optimization only works for inplace=False. We'd need to write our own replace function, to avoid generating the masks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs rebase, otherwise LGTM
@meeseeksdev backport 2.x |
@meeseeksdev backport 2.0.x |
…ks.replace_list
…r Blocks.replace_list) (#51714) Backport PR #50778: PERF: Use generator expression for Blocks.replace_list Co-authored-by: Thomas Li <[email protected]>
doc/source/whatsnew/vX.X.X.rst
file if fixing a bug or adding a new feature.inplace=True is not covered by benchmarks but should be unchanged.