ENH: Add na_value argument to DataFrame.to_numpy

[dsaxton]

• closes ENH: Add na_value to DataFrame.to_numpy #33820
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[simonjayhawkins]

Thanks @dsaxton maybe some tests for error messages and mixed dataframes

[simonjayhawkins]

>>> df = pd.DataFrame({"A": [1, 2, None]})
>>> df.to_numpy(na_value=pd.NA)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas\pandas\core\frame.py", line 1349, in to_numpy
    result[isna(result)] = na_value
TypeError: float() argument must be a string or a number, not 'NAType'
>>>

do we want a more user friendly message, or perhaps validate the fill_value.

[jorisvandenbossche]

That will be difficult in general, I think (since it depends on the behaviour of numpy on what it accepts as value to set to the array).
For this specific case, pd.NA is basically only possible for object dtype (so that could be checked specifically. Although it is also not the typical use case for to_numpy, since you typically want to get rid of pd.NA, since that doesn't work nicely with numpy).

Now, for an extension array, you already get a slightly better error message:

In [18]: pd.array([1, 2, None]).to_numpy(float, na_value=pd.NA)   
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-3c4196341619> in <module>
----> 1 pd.array([1, 2, None]).to_numpy(float, na_value=pd.NA)

~/scipy/pandas/pandas/core/arrays/masked.py in to_numpy(self, dtype, copy, na_value)
    145             ):
    146                 raise ValueError(
--> 147                     f"cannot convert to '{dtype}'-dtype NumPy array "
    148                     "with missing values. Specify an appropriate 'na_value' "
    149                     "for this dtype."

ValueError: cannot convert to '<class 'float'>'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.

[jorisvandenbossche]

That will be difficult in general, I think (since it depends on the behaviour of numpy on what it accepts as value to set to the array).
For this specific case, pd.NA is basically only possible for object dtype (so that could be checked specifically. Although it is also not the typical use case for to_numpy, since you typically want to get rid of pd.NA, since that doesn't work nicely with numpy).

Now, for an extension array, you already get a slightly better error message:

In [18]: pd.array([1, 2, None]).to_numpy(float, na_value=pd.NA)   
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-3c4196341619> in <module>
----> 1 pd.array([1, 2, None]).to_numpy(float, na_value=pd.NA)

~/scipy/pandas/pandas/core/arrays/masked.py in to_numpy(self, dtype, copy, na_value)
    145             ):
    146                 raise ValueError(
--> 147                     f"cannot convert to '{dtype}'-dtype NumPy array "
    148                     "with missing values. Specify an appropriate 'na_value' "
    149                     "for this dtype."

ValueError: cannot convert to '<class 'float'>'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.

[jorisvandenbossche]

So I think you need to deal with this deeper into the internals, as I mentioned before. It is there that the custom pandas logic to decide on the resulting dtype is done (which is indeed different from numpy's result dtype)

[jorisvandenbossche]

Check BlockManager.as_array -> this also already handles the case of a single Block (to keep the no copy behaviour there).

I think we can expand that method to support na_value

[dsaxton]

This had to change because now the output array is a view, but only on the transpose of itself (which is a copy of the original data)

[jreback]

why is this needed?

[jorisvandenbossche]

It's to avoid making a second copy in certain cases where we can detect this (eg with multiple dtypes (going through _interleave), the constructed array is never the original values, so an additional copy is not needed

[jreback]

then i would just make this

copy = copy or na_value is not lib.no_default

and add an explanation line

[jreback]

you might be able to remove the block aboev (is_datetimetz) as that is an extension block already

[jorisvandenbossche]

Indeed, the DatetimeArray.to_numpy behaviour seems correct for the default (having object dtype with timestamps instead of datetime64)

[dsaxton]

Nice 👍

[jreback]

thanks @dsaxton