Skip to content

Revert BUG-24212 fix usage of Index.take in pd.merge #24904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 24, 2019
Merged

Revert BUG-24212 fix usage of Index.take in pd.merge #24904

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
634030b
Revert BUG-24212 fix usage of Index.take in pd.merge
TomAugspurger Jan 24, 2019
fedcec2
test 0.23.4 output
TomAugspurger Jan 24, 2019
9c45869
added note about buggy test
TomAugspurger Jan 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion doc/source/whatsnew/v0.24.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1826,7 +1826,6 @@ Reshaping
- Bug in :func:`DataFrame.unstack` where a ``ValueError`` was raised when unstacking timezone aware values (:issue:`18338`)
- Bug in :func:`DataFrame.stack` where timezone aware values were converted to timezone naive values (:issue:`19420`)
- Bug in :func:`merge_asof` where a ``TypeError`` was raised when ``by_col`` were timezone aware values (:issue:`21184`)
- Bug in :func:`merge` when merging by index name would sometimes result in an incorrectly numbered index (:issue:`24212`)
- Bug showing an incorrect shape when throwing error during ``DataFrame`` construction. (:issue:`20742`)

.. _whatsnew_0240.bug_fixes.sparse:
Expand Down
41 changes: 2 additions & 39 deletions pandas/core/reshape/merge.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -757,19 +757,13 @@ def _get_join_info(self):

if self.right_index:
if len(self.left) > 0:
join_index = self._create_join_index(self.left.index,
self.right.index,
left_indexer,
how='right')
join_index = self.left.index.take(left_indexer)
else:
join_index = self.right.index.take(right_indexer)
left_indexer = np.array([-1] * len(join_index))
elif self.left_index:
if len(self.right) > 0:
join_index = self._create_join_index(self.right.index,
self.left.index,
right_indexer,
how='left')
join_index = self.right.index.take(right_indexer)
else:
join_index = self.left.index.take(left_indexer)
right_indexer = np.array([-1] * len(join_index))
Expand All @@ -780,37 +774,6 @@ def _get_join_info(self):
join_index = join_index.astype(object)
return join_index, left_indexer, right_indexer

def _create_join_index(self, index, other_index, indexer, how='left'):
"""
Create a join index by rearranging one index to match another

Parameters
----------
index: Index being rearranged
other_index: Index used to supply values not found in index
indexer: how to rearrange index
how: replacement is only necessary if indexer based on other_index

Returns
-------
join_index
"""
join_index = index.take(indexer)
if (self.how in (how, 'outer') and
not isinstance(other_index, MultiIndex)):
# if final index requires values in other_index but not target
# index, indexer may hold missing (-1) values, causing Index.take
# to take the final value in target index
mask = indexer == -1
if np.any(mask):
# if values missing (-1) from target index,
# take from other_index instead
join_list = join_index.to_numpy()
join_list[mask] = other_index.to_numpy()[mask]
join_index = Index(join_list, dtype=join_index.dtype,
name=join_index.name)
return join_index

def _get_merge_keys(self):
"""
Note: has side effects (copy/delete key columns)
Expand Down
15 changes: 15 additions & 0 deletions pandas/tests/reshape/merge/test_merge.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -940,6 +940,7 @@ def test_merge_two_empty_df_no_division_error(self):
merge(a, a, on=('a', 'b'))

@pytest.mark.parametrize('how', ['left', 'outer'])
@pytest.mark.xfail(reason="GH-24897")
def test_merge_on_index_with_more_values(self, how):
# GH 24212
# pd.merge gets [-1, -1, 0, 1] as right_indexer, ensure that -1 is
Expand All @@ -959,6 +960,20 @@ def test_merge_on_index_with_more_values(self, how):
expected.set_index('a', drop=False, inplace=True)
assert_frame_equal(result, expected)

def test_merge_right_index_right(self):
# https://github.com/pandas-dev/pandas/issues/24897
left = pd.DataFrame({'a': [1, 2, 3], 'key': [0, 1, 1]})
right = pd.DataFrame({'b': [1, 2, 3]})

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can others double-check this expected output? This is what 0.23.4 / this PR gives. I'm a bit confused about the duplicate 2 in the index.

In [5]: expected
Out[5]:
     a  key  b
0  1.0    0  1
1  2.0    1  2
2  3.0    1  2
2  NaN    2  3

I may have expected something like

In [11]: pd.merge(left, right.rename_axis('key').reset_index(), on='key', how='right').set_index('key', drop=False)
Out[11]:
       a  key  b
key
0    1.0    0  1
1    2.0    1  2
1    3.0    1  2
2    NaN    2  3

where the 1 is duplicated. Or am I missing something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I was first looking at the 'b' column (where the duplicate 2 is of course normal), but the index :)

I assume this is because with how='right and right_index=True, we use the index of the right for the index of the final dataframe (so the 'key' column corresponds to the index in the result).

Not sure this is fully correct though. I don't know if we have somewhere a specification of what the resulting index of a merge should be.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the 'key' column corresponds to the index in the result

and again looking wrongly, because what I describe is your expected result, not the actual ...

Yes, so that certainly looks wrong

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

I may just add a comment that this test should probably fail as well, and open a new issue about it, since this is the behavior on 0.23.4.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, a comment would be good

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm this looks vaguely similar to #17257

expected = pd.DataFrame({'a': [1, 2, 3, None],
'key': [0, 1, 1, 2],
'b': [1, 2, 2, 3]},
columns=['a', 'key', 'b'],
index=[0, 1, 2, 2])
result = left.merge(right, left_on='key', right_index=True,
how='right')
tm.assert_frame_equal(result, expected)


def _check_merge(x, y):
for how in ['inner', 'left', 'outer']:
Expand Down