Skip to content

BUG: Series[object].fillna ignoring downcast='infer' #45062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 27, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v1.4.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -732,6 +732,8 @@ Missing
- Bug in :meth:`Series.interpolate` and :meth:`DataFrame.interpolate` with ``inplace=True`` not writing to the underlying array(s) in-place (:issue:`44749`)
- Bug in :meth:`Index.fillna` incorrectly returning an un-filled :class:`Index` when NA values are present and ``downcast`` argument is specified. This now raises ``NotImplementedError`` instead; do not pass ``downcast`` argument (:issue:`44873`)
- Bug in :meth:`DataFrame.dropna` changing :class:`Index` even if no entries were dropped (:issue:`41965`)
- Bug in :meth:`Series.fillna` with an object-dtype incorrectly ignoring ``downcast="infer"`` (:issue:`44241`)
-

MultiIndex
^^^^^^^^^^
Expand Down
5 changes: 5 additions & 0 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -6399,6 +6399,11 @@ def fillna(
else:
if self.ndim == 1:
if isinstance(value, (dict, ABCSeries)):
if not len(value):
# test_fillna_nonscalar
if inplace:
return None
return self.copy()
value = create_series_with_explicit_dtype(
value, dtype_if_empty=object
)
Expand Down
8 changes: 4 additions & 4 deletions pandas/core/internals/blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -518,11 +518,11 @@ def split_and_operate(self, func, *args, **kwargs) -> list[Block]:
def _maybe_downcast(self, blocks: list[Block], downcast=None) -> list[Block]:

if self.dtype == _dtype_obj:
# TODO: why is behavior different for object dtype?
if downcast is not None:
return blocks

# GH#44241 We downcast regardless of the argument;
# respecting 'downcast=None' may be worthwhile at some point,
# but ATM it breaks too much existing code.
# split and convert the blocks

return extend_blocks(
[blk.convert(datetime=True, numeric=False) for blk in blocks]
)
Expand Down
36 changes: 36 additions & 0 deletions pandas/tests/series/methods/test_fillna.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,42 @@ def test_fillna_downcast(self):
expected = Series([1, 0])
tm.assert_series_equal(result, expected)

def test_fillna_downcast_infer_objects_to_numeric(self):
# GH#44241 if we have object-dtype, 'downcast="infer"' should
# _actually_ infer

arr = np.arange(5).astype(object)
arr[3] = np.nan

ser = Series(arr)

res = ser.fillna(3, downcast="infer")
expected = Series(np.arange(5), dtype=np.int64)
tm.assert_series_equal(res, expected)

res = ser.ffill(downcast="infer")
expected = Series([0, 1, 2, 2, 4], dtype=np.int64)
tm.assert_series_equal(res, expected)

res = ser.bfill(downcast="infer")
expected = Series([0, 1, 2, 4, 4], dtype=np.int64)
tm.assert_series_equal(res, expected)

# with a non-round float present, we will downcast to float64
ser[2] = 2.5

expected = Series([0, 1, 2.5, 3, 4], dtype=np.float64)
res = ser.fillna(3, downcast="infer")
tm.assert_series_equal(res, expected)

res = ser.ffill(downcast="infer")
expected = Series([0, 1, 2.5, 2.5, 4], dtype=np.float64)
tm.assert_series_equal(res, expected)

res = ser.bfill(downcast="infer")
expected = Series([0, 1, 2.5, 4, 4], dtype=np.float64)
tm.assert_series_equal(res, expected)

def test_timedelta_fillna(self, frame_or_series):
# GH#3371
ser = Series(
Expand Down