Series.any() and .all() don't return bool values if dtype=object

[jdavies1618]

Code Sample, a copy-pastable example if possible

s = pd.Series(index=range(5), data=['a', 'b', 'c', 'd', 'e'], dtype=object)
print [s.any(), s.all()]

Expected Output

[True, True]

Actual output

['a', 'e']

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Darwin
OS-release: 15.0.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: None
nose: 1.3.7
pip: 8.1.1
setuptools: None
Cython: None
numpy: 1.10.1
scipy: 0.16.1
statsmodels: None
IPython: 4.0.0
sphinx: 1.3.3
patsy: None
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.11
pymysql: None
psycopg2: None
Jinja2: None

[jreback]

what you are doing doesn't make any sense but it is correct.

http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.any.html

I guess this should return a boolean. But to be honest I don't know what numpy is doing.

In [2]: np.array(['a','b','c'],dtype='object').any()
Out[2]: 'a'

cc @charris
cc @njsmith

[charris]

Hmm..., yes, it is correct

In [1]: 'a' or 'b'
Out[1]: 'a'

I suppose we could call bool on the result for the logical operators.

In [2]: bool('a' or 'b')
Out[2]: True

[jreback]

yep, looks like this is the case for dtype=object in general

In [4]: np.array([1,2,3],dtype=object).any()
Out[4]: 1

[jreback]

ok, I think we will add a bool() if its a scalar on the pandas side for compat.

@jdavies1618 want to do a pull-request?

[jdavies1618]

Will do - thanks for the quick feedback!

[artheist]

This is weird, it looks like there are some discrepancies in the test suite for nanall and nanany.
(test_anaylitics.py file in series test suite)

• In test_all_any, these three lines seems to assert the reported bug is the desired behaviour
  # Alternative types, with implicit 'object' dtype.
  s = Series(['abc', True])
  self.assertEqual('abc', s.any()) # 'abc' || True => 'abc'
• Also, in the following test test_all_any_params, the first lines which test for NaN are not coherent
  This line seems to state that NaN is considered true
  self.assertTrue(s1.all(skipna=False)) # nan && True => True
  but this line contradicts if:
  self.assertTrue(np.isnan(s2.any(skipna=False))) # nan || False => nan

In my opinion, it would be relevant (and more coherent) to have the same behavior as python any and all buitlin methods, such that:
any(['abc', True]) -> True
all(['abc', False]) -> True
any([float('NaN'), True]) -> True
all([float('NaN'), True]) - True
any([float('NaN'), False]) -> True
all([float('NaN'), False]) - False

If this is the case, I have a commit that fixes it, but I then would have to change the test suite also.

[jreback]

so what are you proposing for changes?

[jdavies1618]

Here's a pass I took:

--- a/pandas/tests/series/test_analytics.py
+++ b/pandas/tests/series/test_analytics.py
@@ -548,16 +548,19 @@ class TestSeriesAnalytics(TestData, tm.TestCase):

          self.assertTrue(bool_series.any())
          # Alternative types, with implicit 'object' dtype.
 +        # Changed for gh-12863
          s = Series(['abc', True])
 -        self.assertEqual('abc', s.any())  # 'abc' || True => 'abc'
 +        self.assertEqual(True, s.any())  # 'abc' || True => 'abc'

      def test_all_any_params(self):
          # Check skipna, with implicit 'object' dtype.
 +        # Changed for gh-12863
          s1 = Series([np.nan, True])
          s2 = Series([np.nan, False])
 -        self.assertTrue(s1.all(skipna=False))  # nan && True => True
 +        self.assertTrue(s1.all(skipna=False))  # bool(nan && True) => True
          self.assertTrue(s1.all(skipna=True))
 -        self.assertTrue(np.isnan(s2.any(skipna=False)))  # nan || False => nan
 +        # Below: bool(nan || False) => False
 +        self.assertFalse(np.isnan(s2.any(skipna=False)))
          self.assertFalse(s2.any(skipna=True))

[artheist]

There are more complete tests to be done for proper coverage of all possible situation.
I just made a pull request.

[artheist]

Sorry, I was too fast, there are still errors in tests, sorry

[toobaz]

I guess this should return a boolean. But to be honest I don't know what numpy is doing.

Notice this is considered a bug and is (or was) worked on in numpy:
numpy/numpy#4352
numpy/numpy#5267

So that's probably where effort would better be directed.

[jreback]

So that's probably where effort would better be directed.

good luck with that. And even if a PR was actually proposed / accepted, then this bug would linger in pandas even longer. Much better to do a fix here.

[ShaharNaveh]

take