Skip to content

DOC: warning section on memory overflow when joining/merging dataframes on index with duplicate keys #14788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Dec 11, 2016
Merged

DOC: warning section on memory overflow when joining/merging dataframes on index with duplicate keys #14788

Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
93a63a5
add doc regarding #14736
xgdgsc Dec 2, 2016
795a369
remove blank line
xgdgsc Dec 2, 2016
06d4430
change index to keys , fix spaces and format
xgdgsc Dec 3, 2016
fb021fb
fix format
xgdgsc Dec 3, 2016
403b099
add example , change warning position
xgdgsc Dec 11, 2016
ee9fc60
fix format
xgdgsc Dec 11, 2016
7581ce0
change fig name, change sentence
xgdgsc Dec 11, 2016
48e5322
remove blank line
xgdgsc Dec 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions doc/source/merging.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -817,6 +817,12 @@ equivalent:
left.join(right, on=key_or_keys)
pd.merge(left, right, left_on=key_or_keys, right_index=True,
how='left', sort=False)

.. warning::

Joining on keys with duplicate values when joining large dataframes would cause severe memory overflow, sometimes freezes the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not very useful. I think better is a simple statement that joining / merging on duplicate keys can cause a returned frame that is the multiplication of the row dimensions. Please show a small example as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is strange to include example in a warning section. Is there any example in existing warning sections? Why do you think an example is necessary?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xgdgsc well, reading what you just did, it is completely non-obvious what you mean. an example is worth 1000 words. And when I mean example, I mean about 2 lines of code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

In [20]: left = pd.DataFrame({'A' : [1,2], 'B' : [1, 2]})

In [21]: left
Out[21]: 
   A  B
0  1  1
1  2  2

In [22]: right = pd.DataFrame({'A' : [4,5,6], 'B': [2,2,2]})

In [23]: right
Out[23]: 
   A  B
0  4  2
1  5  2
2  6  2

In [24]: pd.merge(left, right, on='B', how='outer')
Out[24]: 
   A_x  B  A_y
0    1  1  NaN
1    2  2  4.0
2    2  2  5.0
3    2  2  6.0

In [25]: left_dups = pd.DataFrame({'A' : [1,2], 'B' : [2, 2]})

In [26]: left_dups
Out[26]: 
   A  B
0  1  2
1  2  2

In [27]: pd.merge(left_dups, right, on='B', how='outer')
Out[27]: 
   A_x  B  A_y
0    1  2    4
1    1  2    5
2    1  2    6
3    2  2    4
4    2  2    5
5    2  2    6

I would actually show this example and put it in a sub-section with a nice warning. The idea is that having duplicates on BOTH inputs blows up the result.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah current description reads like a pandas problem, though it is actually an usage issue.

computer and user has to hard reboot, which can be dangerous for unsaved work. Please make sure no duplicate values in keys before
joining.

Obviously you can choose whichever form you find more convenient. For
many-to-one joins (where one of the DataFrame's is already indexed by the join
Expand Down