PERF: Increase threashold for using binary search in IndexEngine #54746
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
l3robot
commented
Aug 25, 2023
| @@ -375,7 +375,7 @@ cdef class IndexEngine: | |||
| # map each starget to its position in the index | |||
| if ( | |||
| stargets and | |||
| len(stargets) < 5 and | |||
| len(stargets) < (n / (2 * n.bit_length())) and | |||
There was a problem hiding this comment.
bit_lenght is just a fast way of computing the ceiling of log2.
mroeschke
reviewed
Aug 25, 2023
doc/source/whatsnew/v2.2.0.rst
Outdated
| @@ -105,7 +105,7 @@ Deprecations | |||
|
|
|||
| Performance improvements | |||
| ~~~~~~~~~~~~~~~~~~~~~~~~ | |||
| - | |||
| - Increase the threshold to use binary search instead of a linear search in IndexEngine :issue:`54550` | |||
There was a problem hiding this comment.
Suggested change
| - Increase the threshold to use binary search instead of a linear search in IndexEngine :issue:`54550` | |
| - Performance improvement when indexing with more than 4 keys (:issue:`54550`) |
There was a problem hiding this comment.
Nice, thanks a lot for the review.
done!
mroeschke
added
Indexing
l3robot
changed the title
mroeschke
approved these changes
Aug 26, 2023
|
Thanks @l3robot |
Glad to contribute! |
mroeschke
pushed a commit
to mroeschke/pandas
that referenced
this pull request
Sep 11, 2023
…das-dev#54746) * Increase threashold for using binary search in IndexEngine * Add an entry to the latest whatsnew * Improve entry in the lastest whatsnew --------- Co-authored-by: Patrick Hoefler <[email protected]>
3 tasks
This was referenced Mar 20, 2024
Closed
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
.locwith big non-unique index dataframe #54550 (Replace xxxx with the GitHub issue number)doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.Explanation
So after a bit of research for #54550, I found that the source of the behavior is coming from PR #22826 at this line and particularly this comment here.
In the comment, @rtlee9 makes an excellent explanation for the reason behind the threshold. The default behavior is a linear search (
O(n)) in the pandas index and #22826 added a binary search optimization for monotonically increasing index which is inO(m*log(n))wheremis the number of target indexes.Proposed Solution
I'm proposing we increase the limit, which was set conservatively to 5. We could at least increase it to n / (2 * log2[n]) which is derivable from m*log2[n]<n and dividing per 2 for keeping a certain margin. I'll open a PR for that.
New results of benchmarking + homemade script
Here is a screenshot of
asv continuous -f 1.1 -E virtualenv upstream/main HEAD -b ^indexing.*Index*output (done on Macbook Air m1, inside the docker container provided by pandas):Here is also a new output for the script use to demonstrate the problem in #54700:
Thanks in advance for the review!