Skip to content

BUG: fillna overwriting other missing vals #40509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

BUG: fillna overwriting other missing vals #40509

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5ab2cdf
BUG: fillna other missing vals
mzeitlin11 Mar 19, 2021
7041106
Add whatsnew
mzeitlin11 Mar 19, 2021
3e86934
precommit fixup
mzeitlin11 Mar 19, 2021
39893a7
Fix whitespace
mzeitlin11 Mar 19, 2021
d365360
Fix whitespace
mzeitlin11 Mar 19, 2021
7dc693b
Merge remote-tracking branch 'origin/master' into bug/fillna_missing_…
mzeitlin11 Mar 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,7 @@ Missing

- Bug in :class:`Grouper` now correctly propagates ``dropna`` argument and :meth:`DataFrameGroupBy.transform` now correctly handles missing values for ``dropna=True`` (:issue:`35612`)
- Bug in :func:`isna`, and :meth:`Series.isna`, :meth:`Index.isna`, :meth:`DataFrame.isna` (and the corresponding ``notna`` functions) not recognizing ``Decimal("NaN")`` objects (:issue:`39409`)
- Bug in :meth:`Series.fillna` and :meth:`DataFrame.fillna` overwriting missing values in indices skipped by the ``value`` argument (:issue:`40498`)
-

MultiIndex
Expand Down
5 changes: 5 additions & 0 deletions pandas/core/generic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6524,6 +6524,11 @@ def fillna(
value, dtype_if_empty=object
)
value = value.reindex(self.index, copy=False)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you are better off calling .align here. What you have on L6531 is too specific, fragile and trying to fix something after the fact.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I see what you mean. Not seeing a clear solution with .align either - will have to have some sort of ugly isna check at some point I think to support this. Will close for now and reopen if I can find a better way to do this

# GH-40498: Indices to not apply fillna to are marked with NaN,
# but that will still cause other missing values to be replaced
# with NaN (which is problematic if those aren't NaN)
value.loc[self.isna() & value.isna()] = self
value = value._values
elif not is_list_like(value):
pass
Expand Down
13 changes: 13 additions & 0 deletions pandas/tests/frame/methods/test_fillna.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,19 @@ def test_fillna_tzaware_different_column(self):
)
tm.assert_frame_equal(result, expected)

def test_other_missing_vals_not_modified(
self, unique_nulls_fixture, unique_nulls_fixture2
):
# GH-40498
missing_val1, missing_val2 = unique_nulls_fixture, unique_nulls_fixture2
df = DataFrame(
{"A": [1, missing_val1, missing_val2], "B": [2, missing_val1, missing_val2]}
)
filler = {"A": {1: 0}, "B": {2: 0}}
result = df.fillna(filler)
expected = DataFrame({"A": [1, 0, missing_val2], "B": [2, missing_val1, 0]})
tm.assert_frame_equal(result, expected)

def test_na_actions_categorical(self):

cat = Categorical([1, 2, 3, np.nan], categories=[1, 2, 3])
Expand Down
10 changes: 10 additions & 0 deletions pandas/tests/series/methods/test_fillna.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -620,6 +620,16 @@ def test_fillna_numeric_inplace(self):
expected = x.fillna(value=0)
tm.assert_series_equal(y, expected)

def test_fillna_does_not_modify_other_missing_vals(
self, unique_nulls_fixture, unique_nulls_fixture2
):
# GH-40498
missing_val1, missing_val2 = unique_nulls_fixture, unique_nulls_fixture2
ser = Series([1, missing_val1, missing_val2, ""])
result = ser.fillna({2: 0})
expected = Series([1, missing_val1, 0, ""])
tm.assert_series_equal(result, expected)

# ---------------------------------------------------------------
# CategoricalDtype

Expand Down