BUG: __array_ufunc__ with for functions with side-effects depends on _data

[mlondschien]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

IIUC for pandas extension arrays, for each location where _mask is True, the value of _data should be irrelevant. However this is not the case for __array_ufunc__:

In [1]: import pandas as pd
   ...: import numpy as np
   ...: 
   ...: arr = pd.array([True, False, pd.NA], dtype="boolean")
   ...: arr
Out[1]: 
<BooleanArray>
[True, False, <NA>]
Length: 3, dtype: boolean

In [2]: y = np.ones(3, bool)
   ...: out = np.ones(3, bool)
   ...: 
   ...: np.logical_and(arr, y, out=out)
   ...: out
Out[2]: array([ True, False, False])

In [3]: arr._data[2] = True  # This does not change arr since arr._mask[2] = True
   ...: 
   ...: np.logical_and(arr, y, out=out)
   ...: out
Out[3]: array([ True, False,  True])

Relevant code:

pandas/pandas/core/arrays/boolean.py

Lines 366 to 390 in 17ecb56

	mask = np.zeros(len(self), dtype=bool)
	inputs2 = []
	for x in inputs:
	if isinstance(x, BooleanArray):
	mask |= x._mask
	inputs2.append(x._data)
	else:
	inputs2.append(x)
	def reconstruct(x):
	# we don't worry about scalar `x` here, since we
	# raise for reduce up above.
	if is_bool_dtype(x.dtype):
	m = mask.copy()
	return BooleanArray(x, m)
	else:
	x[mask] = np.nan
	return x
	result = getattr(ufunc, method)(*inputs2, **kwargs)
	if isinstance(result, tuple):
	tuple(reconstruct(x) for x in result)
	else:
	return reconstruct(result)

Here we are calling

result = np.logical_and(arr._data, y, out=out)
return BooleanArray(result, arr._mask)

which adjusts out in-place, ignoring the mask and depending on arr._data[2].

This issue shows how this might be an issue:
numpy/numpy#19374

Output of pd.show_versions()

In [4]: pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-74-generic
Version : #83-Ubuntu SMP Sat May 8 02:35:39 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.21.0
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 49.6.0.post20210108
Cython : None
pytest : 6.2.3
hypothesis : None
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.23.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.04.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 4.0.1
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : None

[xhochy]

Another important aspect is that we currently return an object-typed numpy array if we we don't specify out but rather let it return an ExtensionArray and then convert to numpy:

out = np.logical_and(arr, y)
out

# <BooleanArray>
# [True, False, <NA>]
# Length: 3, dtype: boolean

out.to_numpy()

# array([True, False, <NA>], dtype=object)

That sounds like reasonable behaviour to me.

As it is hard to tell which boolean value should be used for missings, I would raise an error in the case that you have inputs with missings but the out-dtype doesn't support it.