BUG: Dividing Int64 and float64 results in a mix of null-types

[tobiasbrodd]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np

data = {
    "A": [15, np.nan, 5, 4],
    "B": [15, 5, np.nan, 4],
}
df = pd.DataFrame(data)
df = df.astype({"A": "Int64", "B": "float64"})
df["C"] = df["A"] / df["B"]
print(df)
print(df.dtypes)
print(df.dropna(subset=["C"]))

This example outputs:

      A     B     C
0    15  15.0   1.0
1  <NA>   5.0  <NA>
2     5   NaN   NaN
3     4   4.0   1.0

A      Int64
B    float64
C    Float64
dtype: object

    A     B    C
0  15  15.0  1.0
2   5   NaN  NaN
3   4   4.0  1.0

Problem description

Dividing an Int64 column by a float64 column results in a Float64 column that can contain both <NA> and NaN values. However, only <NA> values will be removed by the dropna function. I would expect only <NA> values in the new column and I would also expect the dropna function to remove all NA variants (both <NA> and NaN).

Expected Output

The row with index 2 should be dropped.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.9.4.final.0
python-bits : 64
OS : Darwin
OS-release : 20.5.0
Version : Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 1.3.0
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.3
setuptools : 49.2.1
Cython : 0.29.23
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.3
IPython : 7.23.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : 2021.06.0
fastparquet : 0.6.3
gcsfs : None
matplotlib : 3.4.1
numexpr : None
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : 4.0.1
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : 1.4.17
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
numba : None

[simonjayhawkins]

Thanks @tobiasbrodd for the report. cc @jorisvandenbossche

[sudhanshu2]

Can I take this?

[sudhanshu2]

take

[sudhanshu2]

@simonjayhawkins a potential solution for fixing this bug is _isna_array to treat all arrays of numeric type as same for the purpose of null-types. So essentially check for all possible numeric null-types (such as NaN and <NA>) in all numeric arrays. Do you think this is a viable solution? I assume this will need some documentation update.

[jbrockmendel]

Blocked by #32265

[jorisvandenbossche]

This doesn't necessarily need to be blocked by #32265, I think. Currently, for constructors or casting, when a non-nullable array is converted to a nullable one, we convert NaN to NA:

>>> np_arr = np.array([1, np.nan], dtype="float64")
>>> pd.array(np_arr, dtype="Float64")
<FloatingArray>
[1.0, <NA>]
Length: 2, dtype: Float64

If we follow that logic, I think also for an operation that involves a nullable and non-nullable array which results in a nullable array, I think it is up to the nullable array to properly cast the non-nullable array in its ops implementation. So for example in:

>>> pd.Series([0, 1], dtype="Float64") / pd.Series(np_arr)
0    0.0
1    NaN
dtype: Float64

I think it is the responsibility of FloatArray.__div__(self, other) to properly cast the other operand, so the above would be consistent with doing this cast manually:

>>> pd.Series([0, 1], dtype="Float64") / pd.Series(np_arr).astype("Float64")
0     0.0
1    <NA>
dtype: Float64

(and if at some point we decide to stop converting NaN to NA in the cast from np.float64 to pd.Float64Dtype, then the behaviour here can also follow that. But on the short term I think we can already make it internally consistent)

[jbrockmendel]

when a non-nullable array is converted to a nullable one, we convert NaN to NA [...] But on the short term I think we can already make it internally consistent

AFAICT making everything internally consistent with the constructor behavior is (close to?) equivalent to deciding on #32265 that pd.NA precludes NaN.

[jorisvandenbossche]

I don't think that is necessarily related. Whether we allow np.nan to be present or not in the nullable columns, we would regard np.nan in input in both cases as missing (by default). So I also think that arithmetic operations should do that (by default).

If you adapt the example slightly to add some 0's:

data = {
    "A": [15, np.nan, 5, 4, 0],
    "B": [15, 5, np.nan, 4, 0],
}
df = pd.DataFrame(data)
df = df.astype({"A": "Int64", "B": "float64"})
df["C"] = df["A"] / df["B"]

>>> df
      A     B     C
0    15  15.0   1.0
1  <NA>   5.0  <NA>
2     5   NaN   NaN
3     4   4.0   1.0
4     0   0.0   NaN

The above is on master, while if we would regard the NaNs in column "B" as missing, but still allow NaNs to be created in operations, the result would be:

>>> df
      A     B     C
0    15  15.0   1.0
1  <NA>   5.0  <NA>
2     5   NaN  <NA>
3     4   4.0   1.0
4     0   0.0   NaN

If we would decide to never have NaNs in a nullable columns (and thus convert NaN to NA in the result of operations that can create NaNs), the result would be:

>>> df
      A     B     C
0    15  15.0   1.0
1  <NA>   5.0  <NA>
2     5   NaN  <NA>
3     4   4.0   1.0
4     0   0.0  <NA>

But so in both cases (as I have showed it here), the third row of the last column is NA, and not NaN (as on the main branch).

[jbrockmendel]

Whether we allow np.nan to be present or not in the nullable columns, we would regard np.nan in input in both cases as missing (by default)

I think I understand (and re-reading your earlier comment, could have communicated better earlier).

I think also for an operation that involves a nullable and non-nullable array which results in a nullable array, I think it is up to the nullable array to properly cast the non-nullable array in its ops implementation

IIUC the decision that "cast[ing] the non-nullable array" is the "proper" constitutes deciding that in the affected cases, we don't distinguish between np.nan and pd.NA. Though I now see that it isn't equivalent to a decision on #32265, it would only pin down a subset of cases in there.

AFAICT there are two reasonable-ish things that IntegerArray.__truediv__ could do when operating with a ndarray[float64]:

def __truediv__1(self, other):
    [omitting handling for non-ndarray cases]

    other = pd.array(other)
    other_vals = other._data
    other_mask = other._mask

    res_mask = self._mask | other_mask
    res_values = self._data / other_vals
    return FloatingArray(res_values, res_mask)


def __truediv__2(self, other):
    [omitting handling for non-ndarray cases]

    res_values = self._data / other
    res_mask = self._mask.copy()
    return FloatingArray(res_values, res_mask)

Suppose we decide in #32265 to treat np.nan and pd.NA as semantically distinct. In that case __truediv__2 correctly retains nan entries while __truediv__1 changes them. On the flip side if we decide they are not distinct, then __truediv__1 behavior is more correct.