BUG: MultiIndex.get_indexer not matching nan values #49442
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phofl
added
Indexing
mroeschke
reviewed
Nov 1, 2022
|
Nice. Does this close all the issues linked in #48877? Are any of those tests worth adding? |
|
It should, and yes. I'd like to merge the tests of your pr after this is through if you are ok with that |
mroeschke
reviewed
Nov 2, 2022
mroeschke
reviewed
Nov 2, 2022
Co-authored-by: Matthew Roeschke <[email protected]>
jbrockmendel
reviewed
Nov 2, 2022
pandas/_libs/index.pyx
Outdated
| # with positive integers (-1 for NaN becomes 1). This enables us to | ||
| # differentiate between values that are missing in other and matching | ||
| # NaNs. We will set values that are not found to 0 later: | ||
| codes = (np.array( |
There was a problem hiding this comment.
can this be split onto multiple lines
jbrockmendel
reviewed
Nov 2, 2022
| @@ -36,6 +36,8 @@ from pandas._libs.missing cimport ( | |||
| is_matching_na, | |||
| ) | |||
|
|
|||
| multiindex_nulls_shift = 2 | |||
mroeschke
approved these changes
Nov 4, 2022
|
Thanks @phofl |
10 tasks
phofl
added a commit
to phofl/pandas
that referenced
this pull request
Nov 9, 2022
* BUG: MultiIndex.get_indexer not matching nan values * Fix tests * Use variable for shifts * Fix mypy * Add gh ref * Update pandas/_libs/index.pyx Co-authored-by: Matthew Roeschke <[email protected]> * Adress review Co-authored-by: Matthew Roeschke <[email protected]>
noatamir
pushed a commit
to noatamir/pandas
that referenced
this pull request
Nov 9, 2022
* BUG: MultiIndex.get_indexer not matching nan values * Fix tests * Use variable for shifts * Fix mypy * Add gh ref * Update pandas/_libs/index.pyx Co-authored-by: Matthew Roeschke <[email protected]> * Adress review Co-authored-by: Matthew Roeschke <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This was bugging me for quite some time.
The previous solution did not provide a way to differentiate between matching missing values (e.g. NaN and NaN) and values from other that were missing in self. With shifting NaNs to 1 and values from other that are not in self to 0, we can calculate the correct result.