BUG: fix df.where(cond) when cond is empty

[pajachiet]

• when cond is empty, cond.dtypes are objects, which raised
  ValueError: Boolean array expected for the condition, not object
• closes #xxxxx
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[gfyoung]

@pajachiet : Thanks for the PR! Can you add a test and a whatsnew entry in 0.24.0.txt as well?

From a higher level, this seems like a corner case to me here: why would you want to pass in an empty object for cond for .where ?

[jreback]

how is this not an error?

if you don’t have a conf then where doesn’t make sure

[gfyoung]

how is this not an error?

@jreback : That's what I was wondering too myself. Perhaps a better error message is in order?

[codecov]

Codecov Report

Merging #21947 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #21947      +/-   ##
==========================================
+ Coverage   92.25%   92.25%   +<.01%     
==========================================
  Files         161      161              
  Lines       51169    51170       +1     
==========================================
+ Hits        47207    47208       +1     
  Misses       3962     3962

Flag	Coverage Δ
#multiple	90.64% <100%> (ø)	⬆️
#single	42.28% <0%> (-0.01%)	⬇️

Impacted Files	Coverage Δ
pandas/core/generic.py	96.81% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c6366f5...f930315. Read the comment docs.

[pajachiet]

Hello

I wanted to write a test, but I did not find where were the unit tests for the 'where' function.

This is indeed a corner case, when both df and cond are empty. Example:

import pandas as pd
data = [1]
df = pd.DataFrame(columns=['a'], data=data)
cond = df.applymap(lambda x: x > 0)
df.where(cond)  # Ok

data = []
df = pd.DataFrame(columns=['a'], data=data)
cond = df.applymap(lambda x: x > 0)
df.where(cond)  # Error

This kind of corner case do happen :

• when you do unit testing
• when your dataframe comes from real 'empty' data.

I had the same kind of corner case with df.duplicated in previous version of pandas.

[jreback]

yeah certainly could have a better error message (e.g. you can detect the .empty), but this should just be an error.

[jreback]

the df empty is ok, but the cond cannot be empty (unless the df itself is empty).

[pajachiet]

Yes, cond can be empty if and only if df is empty.

But it is a bug to raise an error when they are both empty, no ?

As the bug is on cond dtype checking, I thought the easiest way to solve it was to avoid dtypes checking on empty dataframe (as it does not makes sense anyway).

One could also explicitly check for df.empty, but this would be weird as it already works when cond is empty with correct dtypes :

data = []
df = pd.DataFrame(columns=['a'], data=data)
cond = df > 0 
df.where(cond)  # Ok
cond.dtypes  # bool

[jreback]

needs a test

add in pandas/tests/frame/test_indexing (you may need one for series, though I don't know if this path is hit, in pandas/tests/series/tests_indexing)

[pep8speaks]

Hello @pajachiet! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on August 09, 2018 at 11:45 Hours UTC

[pajachiet]

Hello,
I added a simple test as requested.
I changed the commit two times because of pep8 mistake, then trying to fix the build, but it is still failing for a reason I do not understand (it does not seem related to the change).
Could you help fixing it ?

[jreback]

pls add a whatsnew entry in v0.24.0 / bug fixes / reshaping

[jreback]

use this PR number as the issue number (as I don't think there is an issue about this)

[jreback]

closing as stale. if you'd like to continue, pls ping.

[pajachiet]

I believe the bug in the CI was not caused by changes in this PR.

This PR fixed a simple but real bug. I managed to fix it in another way for my client, so I only did this PR if it can improve pandas, especially in those edges cases that can be really annoying.

I added a test, a whatsnew entry and answered your comments.
Could you please manage to merge it ?

Otherwise keep it close, it's too much an investment for such a minor improvment.

[gfyoung]

@pajachiet : Do you want to finish this up? I think it requires a rebase, that's all.

[jreback]

Otherwise keep it close, it's too much an investment for such a minor improvment.

well if it’s a bug then it will probably come up again

it’s always helpful to improve things

[gfyoung]

Let's take this for another spin.

[gfyoung]

@jreback : Is there a reason why we can't attempt to merge this for 0.24.0 ? I addressed the last comment (regarding the issue reference). It's also rebased and green. Did you have any other comments to add?

[gfyoung]

Just saw your email regarding 0.24.0. That answers my question above.

[jreback]

minor comment.

[jreback]

thanks!

[pajachiet]

Many thanks ! :)