BUG: to_numpy_dtype_inference returns None when array is non-numeric and dtype isn't given

[derinwalters]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

mport numpy as np
from pandas.core.arrays._utils import to_numpy_dtype_inference

# Issue 1: ArrayLike type of np.ndarray doesn't have atrribute .dtype.numpy_dtype
try:
    to_numpy_dtype_inference(np.array([1,2,3]), None, np.nan, False)
except Exception as e:
    print(e)

# Issue 2: If dtype isn't given and array is not a numeric dtype, returned dtype is None 
print(to_numpy_dtype_inference(np.array(['b','b','c']), None, np.nan, False))

Issue Description

Two places in utility method 'to_numpy_dtype_inference' have issues. The first is that the input array arg "arr" can be of type[ndarray] but ndarray doesn't have an attribute ".dtype.numpy_dtype". The second is that non-numeric array inputs without a given dtype will return None instead of np.dtype(np.object_)

def to_numpy_dtype_inference(
    arr: ArrayLike, dtype: npt.DTypeLike | None, na_value, hasna: bool
) -> tuple[npt.DTypeLike, Any]:
    if dtype is None and is_numeric_dtype(arr.dtype):
        dtype_given = False
        if hasna:
            if arr.dtype.kind == "b":
                dtype = np.dtype(np.object_)
            else:
                if arr.dtype.kind in "iu":
                    dtype = np.dtype(np.float64)
                else:
                    dtype = arr.dtype.numpy_dtype  # type: ignore[union-attr] # <-- Issue 1
                if na_value is lib.no_default:
                    na_value = np.nan
        else:
            dtype = arr.dtype.numpy_dtype  # type: ignore[union-attr]
    elif dtype is not None:
        dtype = np.dtype(dtype)
        dtype_given = True
    else:
        dtype_given = True # <-- Issue 2 [this should instead be "dtype = np.dtype(np.object_)"]

    if na_value is lib.no_default:
        na_value = arr.dtype.na_value

    if not dtype_given and hasna:
        try:
            np_can_hold_element(dtype, na_value)  # type: ignore[arg-type]
        except LossySetitemError:
            dtype = np.dtype(np.object_)
    return dtype, na_value

Expected Behavior

Issue 1: This case doesn't apply to me so I don't have any expectation other than Pandas shouldn't throw an exception.
Issue 2: return np.dtype(np.object_) when array is non-numeric and dtype is not provided

Installed Versions

INSTALLED VERSIONS

commit : f538741
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 23.2.0
Version : Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.0
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 69.0.3
pip : 23.3.2
Cython : None
pytest : 7.4.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 14.0.2
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.11.4
sqlalchemy : None
tables : None
tabulate : None
xarray : 2024.1.0
xlrd : None
zstandard : 0.22.0
tzdata : 2023.4
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report. This method is not part of the public API and should only be used internally by pandas.

cc @phofl

[phofl]

Agree with @rhshadrach, please don’t use this method

[phofl]

Additionally, this is an private file, we don't guarantee anything from version to version for private files

[derinwalters]

For what it's worth, I don't directly call this method. I am extending the BaseMaskedArray for a custom dtype and the to_numpy() method calls it. I understand BaseMaskedArray also isn't likely part of the Public API (would be nice) as opposed to ExtensionArray. Thanks!

[phofl]

Can you explain a little bit more how you ran into issues with the dtype then? Since this is only used from within to_numpy, this should still cast to object for you?

Subclassing BaseMaskedArray should be fine, but it's not exactly public, that's correct.

[derinwalters]

Minimum reproducible example, using publicly available pd.StringDtype() to create a masked array extension. My specific use-case is a masked array representing a nullable MongoDB ObjectId type. Pandas 2.1.4 works fine but fails on 2.2.0 (looks like the masked array to_numpy method was overhauled).

import numpy as np
import pandas as pd
from pandas.core.arrays.masked import BaseMaskedArray

class MyStringArray(BaseMaskedArray):
    dtype = pd.StringDtype()
    _dtype_cls = pd.StringDtype
    _internal_fill_value = pd.NA

a = MyStringArray(values=np.array(['a', 'b', 'c']), mask=np.array([False, True, False]))
# <MyStringArray>
# [a, <NA>, c]
# Length: 3, dtype: string

# Works fine...
a.to_numpy(dtype=np.object_)
# ['a' <NA> 'c']

# Exception due to to_numpy_dtype_inference returning None...
a.to_numpy()
#Traceback (most recent call last):
#  File ".../lib/python3.10/site-packages/pandas/core/arrays/masked.py", line 508, in to_numpy
#    raise ValueError(ValueError: cannot convert to 'None'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.

[phofl]

Thx, I'll address this (I am on vacation this week, but a fix will make it into 2.2.1)

Edit: For future reference: A reproducer like this transports your intent more clearly than referring to internal function, we return None on purpose since we use the function in other places as well were we need to distinguish between we can infer something or we can't)