Skip to content

ENH: try to preserve the dtype on combine_first for the case where the two DataFrame objects have the same columns #39051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jan 15, 2021
Merged

ENH: try to preserve the dtype on combine_first for the case where the two DataFrame objects have the same columns #39051

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
0c1d126
ENH: add argument to preserve dtypes of common columns in combine_first
danielhrisca Jan 11, 2021
1a5fe0f
fix black code style
danielhrisca Jan 11, 2021
24f6ffc
fix misspelled word in docstring
danielhrisca Jan 11, 2021
d0f9ed3
update tests and remove preserve_dtypes argument from combine_first
danielhrisca Jan 11, 2021
198eaa4
fix isort and flake8 errors
danielhrisca Jan 11, 2021
f209590
updates according to erview
danielhrisca Jan 12, 2021
5e252d0
update whatsnew with example code
danielhrisca Jan 12, 2021
7c67e3c
wrong header in documentation
danielhrisca Jan 12, 2021
47d0911
fix black code style
danielhrisca Jan 12, 2021
1b5691c
update whatsnew with example code as requested in the review
danielhrisca Jan 13, 2021
ba49f9c
remove redundant check
danielhrisca Jan 13, 2021
a2d4e38
further fix and polish the whatsnew entry
danielhrisca Jan 13, 2021
f937928
fix single letter variable names
danielhrisca Jan 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions doc/source/whatsnew/v1.3.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,41 @@ Notable bug fixes
These are bug fixes that might have notable behavior changes.


Preserve dtypes in :meth:`~pandas.DataFrame.combine_first`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:meth:`~pandas.DataFrame.combine_first` will now preserve dtypes (:issue:`7509`)

*pandas 1.2.x*

.. code-block:: ipython

In [1]: df1 = pd.DataFrame({"A": [1, 2, 3], "B": [1, 2, 3]}, index=[0, 1, 2])
df2 = pd.DataFrame({"B": [4, 5, 6], "C": [1, 2, 3]}, index=[2, 3, 4])
combined = df1.combine_first(df2)
(combined, "---------------", combined.dtypes)
Out[2]:
( A B C
0 1.0 1.0 NaN
1 2.0 2.0 NaN
2 3.0 3.0 1.0
3 NaN 5.0 2.0
4 NaN 6.0 3.0,
'---------------',
A float64
B float64
C float64
dtype: object)

*pandas 1.3.0*

.. ipython:: python

df1 = pd.DataFrame({"A": [1, 2, 3], "B": [1, 2, 3]}, index=[0, 1, 2])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can do the construction in an ioythok block before both of these as it's the same

only show the combined (and dtypes) in the before / after

df2 = pd.DataFrame({"B": [4, 5, 6], "C": [1, 2, 3]}, index=[2, 3, 4])
combined = df1.combine_first(df2)
(combined, "---------------", combined.dtypes)


.. _whatsnew_130.api_breaking.deps:

Expand Down
17 changes: 16 additions & 1 deletion pandas/core/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6482,7 +6482,22 @@ def combiner(x, y):

return expressions.where(mask, y_values, x_values)

return self.combine(other, combiner, overwrite=False)
combined = self.combine(other, combiner, overwrite=False)

dtypes = {
col: find_common_type([self.dtypes[col], other.dtypes[col]])
for col in self.columns.intersection(other.columns)
if not is_dtype_equal(combined.dtypes[col], self.dtypes[col])
and not is_dtype_equal(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do u have this complicated condition?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first if check for columns that are candidates for preserving the dtype, and the second if avoids extra computation if the common dtype is the same as the combined dtype

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok no need for that, as astype will handle this if you pass copy=False, though i actually don't think this really matters, i would just remove the redudant check

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

combined.dtypes[col],
find_common_type([self.dtypes[col], other.dtypes[col]]),
)
}

if dtypes:
combined = combined.astype(dtypes)

return combined

def update(
self,
Expand Down
83 changes: 63 additions & 20 deletions pandas/tests/frame/methods/test_combine_first.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
import numpy as np
import pytest

from pandas.core.dtypes.cast import find_common_type, is_dtype_equal

import pandas as pd
from pandas import DataFrame, Index, MultiIndex, Series
import pandas._testing as tm
Expand All @@ -18,9 +20,7 @@ def test_combine_first_mixed(self):
b = Series(range(2), index=range(5, 7))
g = DataFrame({"A": a, "B": b})

exp = DataFrame(
{"A": list("abab"), "B": [0.0, 1.0, 0.0, 1.0]}, index=[0, 1, 5, 6]
)
exp = DataFrame({"A": list("abab"), "B": [0, 1, 0, 1]}, index=[0, 1, 5, 6])
combined = f.combine_first(g)
tm.assert_frame_equal(combined, exp)

Expand Down Expand Up @@ -144,7 +144,7 @@ def test_combine_first_return_obj_type_with_bools(self):
)
df2 = DataFrame([[-42.6, np.nan, True], [-5.0, 1.6, False]], index=[1, 2])

expected = Series([True, True, False], name=2, dtype=object)
expected = Series([True, True, False], name=2, dtype=bool)

result_12 = df1.combine_first(df2)[2]
tm.assert_series_equal(result_12, expected)
Expand All @@ -157,22 +157,22 @@ def test_combine_first_return_obj_type_with_bools(self):
(
(
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
[None, None, None],
[pd.NaT, pd.NaT, pd.NaT],
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
),
(
[None, None, None],
[pd.NaT, pd.NaT, pd.NaT],
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
),
(
[datetime(2000, 1, 2), None, None],
[datetime(2000, 1, 2), pd.NaT, pd.NaT],
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
[datetime(2000, 1, 2), datetime(2000, 1, 2), datetime(2000, 1, 3)],
),
(
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
[datetime(2000, 1, 2), None, None],
[datetime(2000, 1, 2), pd.NaT, pd.NaT],
[datetime(2000, 1, 1), datetime(2000, 1, 2), datetime(2000, 1, 3)],
),
),
Expand All @@ -196,13 +196,13 @@ def test_combine_first_align_nan(self):

res = dfa.combine_first(dfb)
exp = DataFrame(
{"a": [pd.Timestamp("2011-01-01"), pd.NaT], "b": [2.0, 5.0]},
{"a": [pd.Timestamp("2011-01-01"), pd.NaT], "b": [2, 5]},
columns=["a", "b"],
)
tm.assert_frame_equal(res, exp)
assert res["a"].dtype == "datetime64[ns]"
# ToDo: this must be int64
assert res["b"].dtype == "float64"
assert res["b"].dtype == "int64"

res = dfa.iloc[:0].combine_first(dfb)
exp = DataFrame({"a": [np.nan, np.nan], "b": [4, 5]}, columns=["a", "b"])
Expand All @@ -219,14 +219,12 @@ def test_combine_first_timezone(self):
columns=["UTCdatetime", "abc"],
data=data1,
index=pd.date_range("20140627", periods=1),
dtype="object",
)
data2 = pd.to_datetime("20121212 12:12").tz_localize("UTC")
df2 = DataFrame(
columns=["UTCdatetime", "xyz"],
data=data2,
index=pd.date_range("20140628", periods=1),
dtype="object",
)
res = df2[["UTCdatetime"]].combine_first(df1)
exp = DataFrame(
Expand All @@ -239,13 +237,10 @@ def test_combine_first_timezone(self):
},
columns=["UTCdatetime", "abc"],
index=pd.date_range("20140627", periods=2, freq="D"),
dtype="object",
)
assert res["UTCdatetime"].dtype == "datetime64[ns, UTC]"
assert res["abc"].dtype == "datetime64[ns, UTC]"
# Need to cast all to "obejct" because combine_first does not retain dtypes:
# GH Issue 7509
res = res.astype("object")

tm.assert_frame_equal(res, exp)

# see gh-10567
Expand Down Expand Up @@ -360,12 +355,11 @@ def test_combine_first_int(self):
df2 = DataFrame({"a": [1, 4]}, dtype="int64")

result_12 = df1.combine_first(df2)
expected_12 = DataFrame({"a": [0, 1, 3, 5]}, dtype="float64")
expected_12 = DataFrame({"a": [0, 1, 3, 5]})
tm.assert_frame_equal(result_12, expected_12)

result_21 = df2.combine_first(df1)
expected_21 = DataFrame({"a": [1, 4, 3, 5]}, dtype="float64")

expected_21 = DataFrame({"a": [1, 4, 3, 5]})
tm.assert_frame_equal(result_21, expected_21)

@pytest.mark.parametrize("val", [1, 1.0])
Expand Down Expand Up @@ -404,11 +398,38 @@ def test_combine_first_string_dtype_only_na(self):
def test_combine_first_timestamp_bug(scalar1, scalar2, nulls_fixture):
# GH28481
na_value = nulls_fixture

frame = DataFrame([[na_value, na_value]], columns=["a", "b"])
other = DataFrame([[scalar1, scalar2]], columns=["b", "c"])

common_dtype = find_common_type([frame.dtypes["b"], other.dtypes["b"]])

if is_dtype_equal(common_dtype, "object") or frame.dtypes["b"] == other.dtypes["b"]:
val = scalar1
else:
val = na_value

result = frame.combine_first(other)
expected = DataFrame([[na_value, scalar1, scalar2]], columns=["a", "b", "c"])

expected = DataFrame([[na_value, val, scalar2]], columns=["a", "b", "c"])

expected["b"] = expected["b"].astype(common_dtype)

tm.assert_frame_equal(result, expected)


def test_combine_first_timestamp_bug_NaT():
# GH28481
frame = DataFrame([[pd.NaT, pd.NaT]], columns=["a", "b"])
other = DataFrame(
[[datetime(2020, 1, 1), datetime(2020, 1, 2)]], columns=["b", "c"]
)

result = frame.combine_first(other)
expected = DataFrame(
[[pd.NaT, datetime(2020, 1, 1), datetime(2020, 1, 2)]], columns=["a", "b", "c"]
)

tm.assert_frame_equal(result, expected)


Expand Down Expand Up @@ -439,3 +460,25 @@ def test_combine_first_with_nan_multiindex():
index=mi_expected,
)
tm.assert_frame_equal(res, expected)


def test_combine_preserve_dtypes():
# GH7509
a = Series(["a", "b"], index=range(2))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the issue number as a comment

b = Series(range(2), index=range(2))
f = DataFrame({"A": a, "B": b})

c = Series(["a", "b"], index=range(5, 7))
b = Series(range(-1, 1), index=range(5, 7))
g = DataFrame({"B": b, "C": c})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: can you avoid 1-letter variable names? makes it harder to grep for things


exp = DataFrame(
{
"A": ["a", "b", np.nan, np.nan],
"B": [0, 1, -1, 0],
"C": [np.nan, np.nan, "a", "b"],
},
index=[0, 1, 5, 6],
)
combined = f.combine_first(g)
tm.assert_frame_equal(combined, exp)