Skip to content

BUG: Fix for fillna ignoring axis=1 parameter (issues #17399 #17409) #28441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
3 changes: 2 additions & 1 deletion pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -6274,7 +6274,8 @@ def fillna(
)

new_data = self._data.fillna(
value=value, limit=limit, inplace=inplace, downcast=downcast
value=value, axis=axis, limit=limit, inplace=inplace,
downcast=downcast
)

elif isinstance(value, (dict, ABCSeries)):
Expand Down
17 changes: 7 additions & 10 deletions pandas/core/internals/blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,7 @@ def apply(self, func, **kwargs):

return result

def fillna(self, value, limit=None, inplace=False, downcast=None):
def fillna(self, value, axis=0, limit=None, inplace=False, downcast=None):
Copy link
Member

@gfyoung gfyoung Sep 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As is, this is a breaking API change. To be safe, you would need to put axis at the end.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add an annotation for axis? Should be importable from pandas._typing

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As is, this is a breaking API change. To be safe, you would need to put axis at the end.

I have just done this, thanks.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add an annotation for axis? Should be importable from pandas._typing

Sorry, I'm not sure what that means. Could you direct me to an example or docs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea so a variable annotation. You'd essentially do:

from pandas._typing import Axis

def fillna(..., axis: Axis = 0):

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks. I have pushed this.

""" fillna on the block with the value. If we fail, then convert to
ObjectBlock and try again
"""
Expand All @@ -398,13 +398,10 @@ def fillna(self, value, limit=None, inplace=False, downcast=None):
raise ValueError("Limit must be an integer")
if limit < 1:
raise ValueError("Limit must be greater than 0")
mask[mask.cumsum(self.ndim - 1) > limit] = False
mask[mask.cumsum(int(axis == 0 and self.ndim > 1)) > limit] = False

if not self._can_hold_na:
if inplace:
return self
else:
return self.copy()
return self if inplace else self.copy()

if self._can_hold_element(value):
# equivalent: _try_coerce_args(value) would not raise
Expand Down Expand Up @@ -1842,7 +1839,7 @@ def concat_same_type(self, to_concat, placement=None):
placement = placement or slice(0, len(values), 1)
return self.make_block_same_class(values, ndim=self.ndim, placement=placement)

def fillna(self, value, limit=None, inplace=False, downcast=None):
def fillna(self, value, axis=0, limit=None, inplace=False, downcast=None):
values = self.values if inplace else self.values.copy()
values = values.fillna(value=value, limit=limit)
return [
Expand Down Expand Up @@ -2406,15 +2403,15 @@ def concat_same_type(self, to_concat, placement=None):
return ObjectBlock(values, ndim=self.ndim, placement=placement)
return super().concat_same_type(to_concat, placement)

def fillna(self, value, limit=None, inplace=False, downcast=None):
def fillna(self, value, axis=0, limit=None, inplace=False, downcast=None):
# We support filling a DatetimeTZ with a `value` whose timezone
# is different by coercing to object.
if self._can_hold_element(value):
return super().fillna(value, limit, inplace, downcast)
return super().fillna(value, axis, limit, inplace, downcast)

# different timezones, or a non-tz
return self.astype(object).fillna(
value, limit=limit, inplace=inplace, downcast=downcast
value, axis=axis, limit=limit, inplace=inplace, downcast=downcast
)

def setitem(self, indexer, value):
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/frame/test_missing.py
Original file line number Diff line number Diff line change
Expand Up @@ -671,6 +671,14 @@ def test_fillna_columns(self):
expected = df.astype(float).fillna(method="ffill", axis=1)
assert_frame_equal(result, expected)

def test_fillna_columns(self):
df = DataFrame(np.random.randn(10, 4))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add GH 17399 as a comment here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also generally for test you can simplify by just constructing a DataFrame with literal values instead of random data - should make smaller and help readability

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the comment (and properly fixed the test function name). I'm afraid I found this bug on the first day of my learning to use pandas, so I'm not quite there yet to write a custom test. Rather, I used/modified the existing one from test_fillna_columns()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm this test works on master for me without these changes - is this capturing the changed behavior?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorvis can you respond here? Need to make sure we're actually fixing a real issue.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to make a more direct test case. This is the demo that made me find it:

https://imgur.com/a/3ZrXqSv

After that I found an existing PR which had grown stale, and @jreback asked me to update it, so I thought I'd give it a try.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated it now to reflect a test similar to that Jupyter image above.

df.iloc[1:4, 1:4] = nan
expected = df.copy()
expected.iloc[1:4, 1:4] = 0
result = df.fillna(value=0, axis=1)
tm.assert_frame_equal(result, expected)

def test_fillna_invalid_method(self, float_frame):
with pytest.raises(ValueError, match="ffil"):
float_frame.fillna(method="ffil")
Expand Down