BUG: join on column with index

[debnathshoham]

• closes BUG: Left join on index and column gives incorrect output #28243
• closes BUG? merging on column of empty frame with index of right frame #15692
• closes Right/outer merge behaviour on left column and right index is unexpected #17257

(below can be closed if agree on expected result)
xref #28220
xref #33232

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Lot's of existing tests were changed, i think there are multiple reported bugs on this.. I might be missing a few others as well.

The changed tests could be broadly classified as:

1. The result contains value in that was not present in the original column.
2. The index of the merged df was different. According to the docs, when merged on a index it should pass on.
3. The data type of the expected df was object instead of say datetime, or someother
4. In merge_asof I think since it defaults to left join, it should inherit the index from left df (these are working fine, there was some trouble with local build initially)

One particular test, which I am little puzzled about the expected output is the below. Joining on index on one df, and partial index and partial column in the other df. I was not able to figure out, so skipped for the time

pandas/pandas/tests/reshape/merge/test_merge_index_as_string.py

Lines 166 to 189 in f9ff379

	@pytest.mark.parametrize("left_index", ["inner", ["inner", "outer"]])
	def test_join_indexes_and_columns_on(df1, df2, left_index, join_type):
	# Construct left_df
	left_df = df1.set_index(left_index)
	# Construct right_df
	right_df = df2.set_index(["outer", "inner"])
	# Result
	expected = (
	left_df.reset_index()
	.join(
	right_df, on=["outer", "inner"], how=join_type, lsuffix="_x", rsuffix="_y"
	)
	.set_index(left_index)
	)
	# Perform join
	result = left_df.join(
	right_df, on=["outer", "inner"], how=join_type, lsuffix="_x", rsuffix="_y"
	)
	tm.assert_frame_equal(result, expected, check_like=True)

[WillAyd]

Cool thanks for giving this a shot. Not sure about the changed tests - so you think all of these exhibit buggy behavior?

[WillAyd]

Might be misreading this but what is the point of this loop? Only ever continues?

[debnathshoham]

this would hit continue, only when it's merging on a col and an index and index already contains correct values

[WillAyd]

Can you revert this?

[debnathshoham]

this would fail.. since to_join is being joined on the index.. the final df would contain that index

this behaviour is mentioned in the docs for merge.. but since join also uses most of the same stuff.. I think they should be consistent?!

[WillAyd]

Hmm not sure I follow - what does this produce now? Why does the monotonicity of the index change?

[debnathshoham]

Previously joined inherited the index from data df. After the change it will inherit from tojoin.

My reason for this to be expected:

• In merge, with column on one side and index on the other, the result inherits the index from the data frame merged on the index (edit: this behaviour is as per doc)
• Since join uses merge, I think their output should be consistent with each other

[debnathshoham]

Most of them the tests are. For the rest I'm not very sure about the expected behaviour.

[WillAyd]

Do all three of these need to be fixed together? I think it would be easier to move forward if we can fix one thing at a time - I'm not sure all these changes are desired

[WillAyd]

Hmm not sure I follow - what does this produce now? Why does the monotonicity of the index change?

[debnathshoham]

All the three bugs are kind of similar.
You're joining / merging two dfs using a column and an index. These are just specific cases I believe.

EDIT: Even the two xref bugs are similar. But I am not very sure about the expected dfs from those.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.