Categorical equality with NaN behaves unexpectedly

[ageorgou]

Code Sample, a copy-pastable example if possible

import pandas as pd

cat_type = pd.CategoricalDtype(categories=["a", "b", "c"])
s1_cat = pd.Series(["a", "b", "c"], dtype=cat_type)
s2_cat = pd.Series([np.nan, "a", "b"], dtype=cat_type)
s1_cat != s2_cat  # expected all True

Output:

0    False
1     True
2     True
dtype: bool

Problem description

I would expect anything to compare != to NaN.

Comparing the categorical series with == works as expected:

s1_cat == s2_cat  # expect all False

Output:

0    False
1    False
2    False
dtype: bool

Element-wise comparison seems to work fine:

for left, right in zip(s1_cat, s2_cat):
    print(left != right)  # expect all True

Output:

True
True
True

This also works as expected with other data types, e.g.:

s1_int = pd.Series([1, 2, 3])
s2_int = pd.Series([np.nan, 1, 2])
s1_int != s2_int  # expect all True

gives

0    True
1    True
2    True
dtype: bool

Apologies if this is a duplicate; I have found various issues about categorical types and missing values, but not about comparing them in this way.

Expected Output

s1_cat != s2_cat  # expected all True

0    True
1    True
2    True
dtype: bool

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Darwin
OS-release : 16.7.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 0.25.1
numpy : 1.17.2
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.2
setuptools : 41.0.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[TomAugspurger]

cc @jorisvandenbossche (gets into the NA vs. NaN discussion as part of #28095).

[WillAyd]

This is standard behavior per IEE 7544

https://stackoverflow.com/questions/1565164/what-is-the-rationale-for-all-comparisons-returning-false-for-ieee754-nan-values

So I think nothing to be done directly on this issue. If you have feedback on generally how missing values should be handled I would suggest rolling into linked issue above

[jorisvandenbossche]

@WillAyd as noted in the issue, for other dtypes it does equate to True, as well for scalars:

In [9]: np.nan != np.nan  
Out[9]: True

[jorisvandenbossche]

To make the difference in behaviour between categorical and non-categorical very explicit:

In [14]: pd.Series(['a', 'b', np.nan]) != pd.Series(['b', 'a', 'a'])
Out[14]: 
0    True
1    True
2    True
dtype: bool

In [15]: pd.Series(['a', 'b', np.nan], dtype='category') != pd.Series(['b', 'a', 'a'], dtype='category') 
Out[15]: 
0     True
1     True
2    False
dtype: bool

I suppose this has to do with np.nan being represented in categorical as -1, and this might not be special cased.

[quale1]

Just to further emphasize the problems this causes,

In [3]: x = pd.Series(['a', 'b', np.nan], dtype='category')

In [4]: y = pd.Series(['b', 'a', 'a'], dtype='category')

In [5]: x[2] != y[2]
Out[5]: True

In [6]: (x != y)[2]
Out[6]: False

Result 5 makes sense, but 6 doesn't. It's impossible to successfully reason about a Pandas program when x[2] != y[2] and (x != y)[2] give different results.

[simonjayhawkins]

looks to be fixed on master (@jbrockmendel maybe #36237?)

>>> pd.__version__
'1.2.0.dev0+378.g8df0218a4b'
>>> pd.Series(['a', 'b', np.nan], dtype='category') != pd.Series(['b', 'a', 'a'], dtype='category')
0    True
1    True
2    True
dtype: bool
>>>

[junjunjunk]

take