BUG: groupby.agg coercing booleans

[enfeizhan]

Code Sample, a copy-pastable example if possible

import pandas as pd

dat = pd.DataFrame({'a': [1, 1, 2, 2], 'b': [0, 1, 2, 3], 'c': [None, None, 1, 1]})

gp = dat.groupby('a')

case_one = gp['b'].aggregate(lambda x: (x != 0).all())

case_two = gp['c'].aggregate(lambda x: x.isnull().all())

Problem description

By comparing case_one and case_two, it seems that aggregate function turns boolean values sometimes to integers 1 or 0.

For case_one we get,

a
1    False
2     True
Name: b, dtype: bool

However for case_two we get

a
1    1.0
2    0.0
Name: c, dtype: float64

Expected Output

For case_two we should get

a
1    True
2     False
Name: b, dtype: bool

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-53-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_AU.UTF-8 LOCALE: en_AU.UTF-8

pandas: 0.19.1
nose: 1.3.7
pip: 8.1.2
setuptools: 23.0.0
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: None
xarray: None
IPython: 4.2.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

This same issue appears in a different environment as well.

INSTALLED VERSIONS ------------------ commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-53-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_AU.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 9.0.1
setuptools: 23.0.0
Cython: None
numpy: 1.11.2
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.0.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.3
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[jreback]

this is a duplicate of this: #14423

.groupby tries to cast back to the original types which is generally the correct result. Since you are performaning an arbitrary opaque function (this is by-definition what a lambada is), it is not clear what the correct result should be.

I suppose we could be a little softer in the casting w.r.t. boolean results. (but of course someone will have an issue with that :)

[jreback]

this is independent of #14423

looks like _try_cast needs to respect boolean

[mroeschke]

This looks fixed. Could use a test.

In [12]: case_two
Out[12]:
a
1     True
2    False
Name: c, dtype: bool

In [13]: pd.__version__
Out[13]: '0.25.0.dev0+50.gf75a220ff'

[TrigonaMinima]

@mroeschke is this issue really fixed? In the #21240 issue referenced just above, @jreback says this issue is the root cause of that issue. And when I try to run the code part from that issue, the problem still seems to be there.

[mroeschke]

#21240 isn't fixed but this specific case is fixed