Skip to content

BUG: Fix issue with apply on empty DataFrame #28213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Sep 20, 2019
Merged

BUG: Fix issue with apply on empty DataFrame #28213

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
444209d
Use try / except on empty Series
Aug 29, 2019
b749cca
Fix PEP8
Aug 29, 2019
d9b620b
Remove try / except
Aug 29, 2019
a82479a
Only apply on non-empty agg_axis
Aug 29, 2019
aa4b33b
Add tests
Aug 29, 2019
61f432f
Move condition
Aug 29, 2019
cb68153
Don't unpack args twice
Aug 30, 2019
f4c7a97
Add nunique test
Aug 30, 2019
633a6b8
Add to release notes
Aug 30, 2019
1993c7c
Parametrize tests
Aug 30, 2019
a0c09eb
Revert "Parametrize tests"
Aug 31, 2019
51de6ef
Parametrize tests
Aug 30, 2019
91ef657
Use getattr
Aug 31, 2019
4a9ea0b
Add nunique test
Aug 31, 2019
42b2209
Set index
Aug 31, 2019
de0e1dd
Move note under reshaping
Sep 2, 2019
6b98060
Merge branch 'master' into apply-fix
Sep 8, 2019
9d6ae49
Use should_reduce
Sep 8, 2019
2067c1a
Fix merge
Sep 9, 2019
9442a04
Narrow exception catching
Sep 9, 2019
20a3199
Merge branch 'master' into apply-fix
Sep 11, 2019
f3077aa
Revert "Narrow exception catching"
Sep 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Performance improvements
Bug fixes
~~~~~~~~~

- Bug in :meth:`DataFrame.apply` that caused incorrect output with empty :class:`DataFrame` (:issue:`28202`, :issue:`21959`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is specific to np.* and .nunique()? if so can you be more specific

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd apply to any reduction whose output isn't empty when the input is, should I say something to that effect?


Categorical
^^^^^^^^^^^
Expand Down
11 changes: 7 additions & 4 deletions pandas/core/apply.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,16 +204,19 @@ def apply_empty_result(self):
from pandas import Series

if not reduce:

EMPTY_SERIES = Series([])
try:
r = self.f(EMPTY_SERIES, *self.args, **self.kwds)
r = self.f(Series([]))
reduce = not isinstance(r, Series)
except Exception:
pass

if reduce:
return self.obj._constructor_sliced(np.nan, index=self.agg_axis)
if len(self.agg_axis):
r = self.f(Series([]))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pass args & kwargs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already are trying to reduce above (line 208), why are you calling the function again? does this hit the Except?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that in case reduce was already True at line 206, so that the try block wouldn't have been executed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am still puzzled why you can not pass args/kwargs

Copy link
Member Author

@dsaxton dsaxton Sep 13, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what's happening is the function self.f is getting curried around here:

pandas/pandas/core/apply.py

Line 109 in cb68153

if (kwds or args) and not isinstance(func, (np.ufunc, str)):
so when we pass the arguments again we get an error (this is from the df.nunique() test):

>           r = self.f(Series([]), *self.args, **self.kwds)
E           TypeError: f() got an unexpected keyword argument 'dropna'

because at that point f only takes a single argument. I could imagine there could end up being a problem if this currying doesn't happen, so there's probably a hidden corner case that just isn't covered by the existing tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was confused by this to but I think this makes sense. I suppose this was hitting the except before due to a TypeError for wrong number of arguments?

else:
r = np.nan

return self.obj._constructor_sliced(r, index=self.agg_axis)
else:
return self.obj.copy()

Expand Down
9 changes: 9 additions & 0 deletions pandas/tests/frame/test_apply.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,15 @@ def test_apply_with_reduce_empty(self):
# Ensure that x.append hasn't been called
assert x == []

@pytest.mark.parametrize("func", ["sum", "prod", "any", "all"])
def test_apply_funcs_over_empty(self, func):
# GH 28213
df = DataFrame(columns=["a", "b", "c"])

result = df.apply(eval(func))
expected = operator.methodcaller(func)(df)
assert_series_equal(result, expected)

def test_apply_deprecate_reduce(self):
empty_frame = DataFrame()

Expand Down