Skip to content

BUG/PERF: Series.combine_first converting int64 to float64 #51777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v2.1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ Groupby/resample/rolling

Reshaping
^^^^^^^^^
-
- Bug in :meth:`Series.combine_first` converting ``int64`` dtype to ``float64`` (:issue:`51764`)
-

Sparse
Expand Down
10 changes: 9 additions & 1 deletion pandas/core/series.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,13 @@
from pandas.core.dtypes.cast import (
LossySetitemError,
convert_dtypes,
find_common_type,
maybe_box_native,
maybe_cast_pointwise_result,
)
from pandas.core.dtypes.common import (
is_dict_like,
is_dtype_equal,
is_extension_array_dtype,
is_integer,
is_iterator,
Expand Down Expand Up @@ -3272,7 +3274,13 @@ def combine_first(self, other) -> Series:
if this.dtype.kind == "M" and other.dtype.kind != "M":
other = to_datetime(other)

return this.where(notna(this), other)
combined = this.where(notna(this), other)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this won't break anything, but for the series case there might be a proper fix. You could set a compatible fill_value for the reindex ops. Only disadvantage is, that you'll have to compute notna(this) somehow


if not is_dtype_equal(combined.dtype, self.dtype):
dtype = find_common_type([self.dtype, other.dtype])
combined = combined.astype(dtype, copy=False)

return combined
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a finalise call with self, otherwise we will loose metadata.

This might changed result ordering? Can we do a reindex in the end?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, thanks


def update(self, other: Series | Sequence | Mapping) -> None:
"""
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/series/methods/test_combine_first.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,11 @@ def test_combine_first_timezone_series_with_empty_series(self):
s2 = Series(index=time_index)
result = s1.combine_first(s2)
tm.assert_series_equal(result, s1)

def test_combine_first_preserves_dtype(self):
# GH51764
s1 = Series([4, 5])
s2 = Series([6, 7, 8])
result = s1.combine_first(s2)
expected = Series([4, 5, 8])
tm.assert_series_equal(result, expected)