BUG: Cartesian product in outer merge with duplicated rows

[antonchasnyk]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
a = pd.DataFrame({'timestamp': ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-03'],
                      'A': [1, 2, 3, 3]}).set_index('timestamp')
b = pd.DataFrame({'timestamp': ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-03'],
                      'B': [1, 2, 3, 3]}).set_index('timestamp')
df = reduce(lambda l, r: pd.merge(l, r, left_index=True, right_index=True, how="outer"), [a, b])
print(df)

Issue Description

If the left and right data frame has duplicated rows then these rows will produce cartesian multiplicated rows in the result.

            A  B
timestamp       
2000-01-01  1  1
2000-01-02  2  2
2000-01-03  3  3
2000-01-03  3  3
2000-01-03  3  3 <---
2000-01-03  3  3 <---

this could potentially produce a huge data frame.

Expected Behavior

            A  B
timestamp       
2000-01-01  1  1
2000-01-02  2  2
2000-01-03  3  3
2000-01-03  3  3

Installed Versions

INSTALLED VERSIONS

commit : 06d2301
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-107-generic
Version : #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : ru_RU.UTF-8
LOCALE : ru_RU.UTF-8

pandas : 1.4.1
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
pip : 20.0.2
setuptools : 44.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.4.35
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[GYHHAHA]

What's the problem here? Since we have 2 rows for 2000-01-03 in a and 2 rows for 2000-01-03 in b, then we will indeed produce 2*2 rows for 2000-01-03 in the final result.

[attack68]

What's the problem here? Since we have 2 rows for 2000-01-03 in a and 2 rows for 2000-01-03 in b, then we will indeed produce 2*2 rows for 2000-01-03 in the final result.

Agreed, the input dataframes are specifically coded with duplicate values on the index so the output will also have duplicates.

[mroeschke]

Thanks for the question but this is the expected behavior, you can first drop duplicate index labels or groupby + aggregate in first in order for the outer merge result to have unique labels

[antonchasnyk]

Maybe you can throw a warning in this case? Because merging multicolumn dataframes can consume a lot of RAM. In my case, a few duplicated rows consumed 8Gb RAM for dataframe with the expected result shape (40150, 715).

[attack68]

A warning is not required, you are expected to check if the index contains duplicates, and drop if required, see @mroeschke. Note that not all use cases are like yours.