Skip to content

GH #14499 Panel.ffill ignores axis parameter and fill along axis=1 #14528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.20.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -300,3 +300,4 @@ Bug Fixes
- Bug in ``Series.unique()`` in which unsigned 64-bit integers were causing overflow (:issue:`14721`)
- Require at least 0.23 version of cython to avoid problems with character encodings (:issue:`14699`)
- Bug in converting object elements of array-like objects to unsigned 64-bit integers (:issue:`4471`)
- Bug in ``pd.Panel.ffill`` where ``axis`` was not propagated (:issue:`14499`)
53 changes: 34 additions & 19 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -3262,26 +3262,21 @@ def fillna(self, value=None, method=None, axis=None, inplace=False,
'you passed a "{0}"'.format(type(value).__name__))
self._consolidate_inplace()

# set the default here, so functions examining the signaure
# set the default here, so functions examining the signature
# can detect if something was set (e.g. in groupby) (GH9221)
if axis is None:
axis = 0
if self.ndim == 3:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

axis = self._stat_axis_number give the correct default

axis = 1
else:
axis = 0

axis = self._get_axis_number(axis)
method = missing.clean_fill_method(method)

from pandas import DataFrame
if value is None:
if method is None:
raise ValueError('must specify a fill method or value')
if self._is_mixed_type and axis == 1:
if inplace:
raise NotImplementedError()
result = self.T.fillna(method=method, limit=limit).T

# need to downcast here because of all of the transposes
result._data = result._data.downcast()

return result

# > 3d
if self.ndim > 3:
Expand All @@ -3290,21 +3285,41 @@ def fillna(self, value=None, method=None, axis=None, inplace=False,

# 3d
elif self.ndim == 3:

# fill in 2d chunks
result = dict([(col, s.fillna(method=method, value=value))
for col, s in self.iteritems()])
if axis == 0:
frame = self.swapaxes(0, 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question (not too familiar with Panels): swapping the axes here can potentially cause dtypes to change? (ints upcast to float, or numeric upcast to object when you have mixed dtypes)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let me check

axis2d = 0
else:
frame = self
axis2d = axis - 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

axis == 2?

result = dict([(col, s.fillna(method=method,
value=value,
axis=axis2d))
for col, s in frame.iteritems()])
new_obj = self._constructor.\
from_dict(result).__finalize__(self)
if axis == 0:
new_obj = new_obj.swapaxes(0, 1)
new_data = new_obj._data

else:
# 2d or less
method = missing.clean_fill_method(method)
new_data = self._data.interpolate(method=method, axis=axis,
limit=limit, inplace=inplace,
coerce=True,
downcast=downcast)
if self._is_mixed_type and axis == 1:
if inplace:
raise NotImplementedError()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a message to the NotImplementedError

result = self.T.fillna(method=method, limit=limit).T

# need to downcast here because of all of the transposes
result._data = result._data.downcast()

return result
else:
method = missing.clean_fill_method(method)
new_data = self._data.interpolate(method=method, axis=axis,
limit=limit,
inplace=inplace,
coerce=True,
downcast=downcast)
else:
if method is not None:
raise ValueError('cannot specify both a fill method and value')
Expand Down
44 changes: 44 additions & 0 deletions pandas/tests/test_panel.py
Original file line number Diff line number Diff line change
Expand Up @@ -1519,6 +1519,50 @@ def test_ffill_bfill(self):
assert_panel_equal(self.panel.bfill(),
self.panel.fillna(method='bfill'))

def test_ffill_bfill_axis(self):
# GH 14499
a = Panel({
'a': np.arange(15, dtype=float).reshape((5, 3)),
'b': np.arange(15, 30, dtype=float).reshape((5, 3))
})
f0 = a.copy()
f1 = a.copy()
f2 = a.copy()
b0 = a.copy()
b1 = a.copy()
b2 = a.copy()
a['a'].loc[1, 1] = np.nan
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set using full .loc, not chained indexing, e.g. f0.loc['a', 1, 1] = np.nan

f0['a'].loc[1, 1] = np.nan
f1['a'].loc[1, 1] = 1
f2['a'].loc[1, 1] = 3
b0['a'].loc[1, 1] = 19.0
b1['a'].loc[1, 1] = 7
b2['a'].loc[1, 1] = 5

# method='ffill'
# axis=0
assert_panel_equal(a.ffill(axis=0), f0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make these expected before each statement, e.g.

result = a.ffill(axis=0)
expected = ......
assert_panel_equal(result, expected)


# method='ffill'
# axis=1
assert_panel_equal(a.ffill(axis=1), f1)

# method='ffill'
# axis=2
assert_panel_equal(a.ffill(axis=2), f2)

# method='bfill'
# axis=1
assert_panel_equal(a.bfill(axis=0), b0)

# method='bfill'
# axis=1
assert_panel_equal(a.bfill(axis=1), b1)

# method='bfill'
# axis=2
assert_panel_equal(a.bfill(axis=2), b2)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally also tests with a panel that has a non-numeric type as well (IOW, add a string to the above example); this should be a separate set of tests.

def test_truncate_fillna_bug(self):
# #1823
result = self.panel.truncate(before=None, after=None, axis='items')
Expand Down