BUG: Compare returns wrong results for nullable int type

[zazeng]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd

df1 = pd.DataFrame({'a': [4, 4, 4], 'b': [1, 2, 3]}).astype({'a': 'float64', 'b': 'float64'})

# Nullable int 'Int64' gives wrong results. 'int64' works just fine.
df2 = pd.DataFrame({'a': [1, 2, 3], 'b': [1, 2, 3]}).astype({'a': 'Int64', 'b': 'float64'})

df1.compare(df2, keep_shape=True)

Issue Description

The reproducible example returns

	('a', 'self')	('a', 'other')	('b', 'self')	('b', 'other')
0	nan	1	nan	nan
1	nan	2	nan	nan
2	nan	3	nan	nan

Expected Behavior

We expect the reproducible example to return

	('a', 'self')	('a', 'other')	('b', 'self')	('b', 'other')
0	4	1	nan	nan
1	4	2	nan	nan
2	4	3	nan	nan

Installed Versions

INSTALLED VERSIONS

commit : 73c6825
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.11.0-37-generic
Version : #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_SG.UTF-8
LOCALE : en_SG.UTF-8

pandas : 1.3.3
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 21.1.1
setuptools : 56.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.1
lxml.etree : 4.6.3
html5lib : None
pymysql : 0.8.1
psycopg2 : None
jinja2 : 2.11.3
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.7.1
sqlalchemy : 1.3.24
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[gabrieldi95]

I believe the problem is that on pandas.core.generic.where:
mask = ~((self == other) | (self.isna() & other.isna()))
returns

0  True  False
1  True  False
2  True  False

which is right, but mask.dtypes returns:

a    boolean
b       bool

But if you change both columns to 'bool', as in mask.astype({'a': 'bool', 'b': 'bool'})
It returns the expected behavior above.

I can work on a solution later, if that's ok.

[mzeitlin11]

Thanks for looking into a fix @gabrieldi95! Going straight to bool might cause issues when NA is present because converting NA to bool will raise. Having the mask contain a boolean column at this point might be fine - maybe can fix at a later place? Or at least make sure when testing to include some examples including missing values.

[gabrieldi95]

Thank you for the feedback @mzeitlin11, I'll try to work on it today.

[gabrieldi95]

take

[gabrieldi95]

@mzeitlin11 I found out what the problem is:

If you have a DataFrame with a dtype='boolean', using .values results in a ndarray with dtype=dtype('O'), in opposite to a value with dtype='bool', which results in a ndarray with dtype=dtype('bool').

This happens on pandas.core.internals.managers.py, line 318:
kwargs[k] = obj.iloc[:, b.mgr_locs.indexer]._values

Further on the execution, on pandas.core.internals.blocks.py, line 1211:
icond, noop = validate_putmask(values, ~cond)

~cond returns array([[-2], [-2], [-2]], dtype=object) if dtype=dtype('O'), and array([[False], [False], [False]]) if dtype=dtype('bool').

I'm not sure what's the best solution though...

Forcing cond to dtype('bool') fixes this problem but if you change the exemple to:

import pandas as pd

df1 = pd.DataFrame({'a': [4, 4, 4], 'b': [1, 2, 3]}).astype({'a': 'float64', 'b': 'float64'})

# Nullable int 'Int64' gives wrong results. 'int64' works just fine.
df2 = pd.DataFrame({'a': [1, 2, None], 'b': [1, 2, 3]}).astype({'a': 'Int64', 'b': 'float64'})

df1.compare(df2, keep_shape=True)

It returns

     a          b      
  self other self other
0  4.0     1  NaN   NaN
1  4.0     2  NaN   NaN
2  NaN  <NA>  NaN   NaN

Which is wrong right?

I think we would either have to negate cond in another way, or find out why .values results in a ndarray with dtype=dtype('O'). What do you think?

[jbrockmendel]

If you have a DataFrame with a dtype='boolean', using .values results in a ndarray with dtype=dtype('O')

shout it from the rooftops: this would be solved by 2D EAs.