WIP: NumPyBackedExtensionArray

Adds a NumPyBckedExtensionArray, a thin wrapper around ndarray implementing
the EA interface.

We use this to ensure that `Series.array -> ExtensionArray`, rather than
a `Union[ndarray, ExtensionArray]`.

Author: TomAugspurger

doc/source/basics.rst

@@ -71,8 +71,9 @@ the **array** property
    s.array
    s.index.array
 
-Depending on the data type (see :ref:`basics.dtypes`), :attr:`~Series.array`
-be either a NumPy array or an :ref:`ExtensionArray <extending.extension-type>`.
+
+:attr:`~Series.array` will always be an ``ExtensionArray``.
+
 If you know you need a NumPy array, use :meth:`~Series.to_numpy`
 or :meth:`numpy.asarray`.
 
@@ -81,8 +82,7 @@ or :meth:`numpy.asarray`.
    s.to_numpy()
    np.asarray(s)
 
-For Series and Indexes backed by NumPy arrays (like we have here), this will
-be the same as :attr:`~Series.array`. When the Series or Index is backed by
+When the Series or Index is backed by
 a :class:`~pandas.api.extension.ExtensionArray`, :meth:`~Series.to_numpy`
 may involve copying data and coercing values.
 
@@ -115,8 +115,8 @@ drawbacks:
 
 1. When your Series contains an :ref:`extension type <extending.extension-type>`, it's
    unclear whether :attr:`Series.values` returns a NumPy array or the extension array.
-   :attr:`Series.array` will always return the actual array backing the Series,
-   while :meth:`Series.to_numpy` will always return a NumPy array.
+   :attr:`Series.array` will never require copying data, while :meth:`Series.to_numpy`
+   will always return a NumPy array (potentially at the cost of copying / coercing values).
 2. When your DataFrame contains a mixture of data types, :attr:`DataFrame.values` may
    involve copying data and coercing values to a common dtype, a relatively expensive
    operation. :meth:`DataFrame.to_numpy`, being a method, makes it clearer that the

doc/source/dsintro.rst

@@ -146,11 +146,15 @@ If you need the actual array backing a ``Series``, use :attr:`Series.array`.
 
    s.array
 
-Again, this is often a NumPy array, but may instead be a
-:class:`~pandas.api.extensions.ExtensionArray`. See :ref:`basics.dtypes` for more.
 Accessing the array can be useful when you need to do some operation without the
 index (to disable :ref:`automatic alignment <dsintro.alignment>`, for example).
 
+:attr:`Series.array` will always be an :class:`~pandas.api.extensions.ExtensionArray`.
+Briefly, and ExtensionArray is a thin wrapper around one or more *real* arrays like a
+:class:`numpy.ndarray`. Pandas knows how to take an ``ExtensionArray`` and
+store it in a ``Series`` or a column of a ``DataFrame``.
+See :ref:`basics.dtypes` for more.
+
 While Series is ndarray-like, if you need an *actual* ndarray, then use
 :meth:`Series.to_numpy`.
 

doc/source/whatsnew/v0.24.0.rst

@@ -65,8 +65,9 @@ If you need an actual NumPy array, use :meth:`Series.to_numpy` or :meth:`Index.t
    idx.to_numpy()
    pd.Series(idx).to_numpy()
 
-For Series and Indexes backed by normal NumPy arrays, this will be the same thing (and the same
-as ``.values``).
+For Series and Indexes backed by normal NumPy arrays, :attr:`Series.array` will return a
+new :class:`NumPyBackedExtensionArray`, which is a thin (no-copy) wrapper around a
+:class:`numpy.ndarray`.
 
 .. ipython:: python
 
@@ -75,7 +76,7 @@ as ``.values``).
    ser.to_numpy()
 
 We haven't removed or deprecated :attr:`Series.values` or :attr:`DataFrame.values`, but we
-recommend and using ``.array`` or ``.to_numpy()`` instead.
+highly recommend and using ``.array`` or ``.to_numpy()`` instead.
 
 See :ref:`basics.dtypes` and :ref:`dsintro.attrs` for more.
 

pandas/core/arrays/__init__.py

@@ -9,3 +9,4 @@
 from .integer import (  # noqa
     IntegerArray, integer_array)
 from .sparse import SparseArray  # noqa
+from .numpy_ import NumPyExtensionArray, NumPyExtensionDtype  # noqa

pandas/core/arrays/categorical.py

@@ -16,11 +16,11 @@
 from pandas.core.dtypes.cast import (
     coerce_indexer_dtype, maybe_infer_to_datetimelike)
 from pandas.core.dtypes.common import (
-    ensure_int64, ensure_object, ensure_platform_int, is_categorical,
-    is_categorical_dtype, is_datetime64_dtype, is_datetimelike, is_dict_like,
-    is_dtype_equal, is_extension_array_dtype, is_float_dtype, is_integer_dtype,
-    is_iterator, is_list_like, is_object_dtype, is_scalar, is_sequence,
-    is_timedelta64_dtype)
+    ensure_int64, ensure_object, ensure_platform_int, extract_array,
+    is_categorical, is_categorical_dtype, is_datetime64_dtype, is_datetimelike,
+    is_dict_like, is_dtype_equal, is_extension_array_dtype, is_float_dtype,
+    is_integer_dtype, is_iterator, is_list_like, is_object_dtype, is_scalar,
+    is_sequence, is_timedelta64_dtype)
 from pandas.core.dtypes.dtypes import CategoricalDtype
 from pandas.core.dtypes.generic import (
     ABCCategoricalIndex, ABCIndexClass, ABCSeries)
@@ -2078,8 +2078,7 @@ def __setitem__(self, key, value):
             `Categorical` does not have the same categories
         """
 
-        if isinstance(value, (ABCIndexClass, ABCSeries)):
-            value = value.array
+        value = extract_array(value)
 
         # require identical categories set
         if isinstance(value, Categorical):

pandas/core/arrays/numpy_.py

@@ -0,0 +1,271 @@
+import numpy as np
+
+from pandas._libs import lib
+
+from pandas.core.dtypes.common import extract_array
+from pandas.core.dtypes.dtypes import ExtensionDtype
+from pandas.core.dtypes.generic import ABCIndexClass, ABCSeries
+from pandas.core.dtypes.inference import is_list_like
+
+from pandas import compat
+from pandas.core import nanops
+
+from .base import ExtensionArray, ExtensionOpsMixin
+
+
+class NumPyExtensionDtype(ExtensionDtype):
+    _metadata = ('_dtype',)
+
+    def __init__(self, dtype):
+        assert isinstance(dtype, np.dtype)
+        self._dtype = dtype
+        self._name = dtype.name
+        self._type = dtype.type
+
+    @property
+    def name(self):
+        return self._name
+
+    @property
+    def type(self):
+        return self._type
+
+    @property
+    def _is_numeric(self):
+        # TODO: find numeric types
+        return True
+
+    @property
+    def _is_boolean(self):
+        return self.kind == 'b'  # object?
+
+    @classmethod
+    def construct_from_string(cls, string):
+        return cls(np.dtype(string))
+
+    def construct_array_type(cls):
+        return NumPyExtensionArray
+
+    @property
+    def kind(self):
+        return self._dtype.kind
+
+    @property
+    def itemsize(self):
+        return self._dtype.itemsize
+
+
+class NumPyExtensionArray(ExtensionArray, ExtensionOpsMixin):
+    __array_priority__ = 1000
+
+    def __init__(self, values):
+        if isinstance(values, type(self)):
+            values = values._ndarray
+        assert isinstance(values, np.ndarray)
+        assert values.ndim == 1
+
+        self._ndarray = values
+        self._dtype = NumPyExtensionDtype(values.dtype)
+
+    @classmethod
+    def _from_sequence(cls, scalars, dtype=None, copy=False):
+        #
+        # if isinstance(dtype, NumpyDtype):
+        #     dtype = dtype._dtype
+        # we deliberately ignore dtype to not deal with casting issues.
+
+        result = np.asarray(scalars)
+        if copy and result is scalars:
+            result = result.copy()
+        return cls(result)
+
+    @classmethod
+    def _from_factorized(cls, values, original):
+        return cls(values)
+
+    @classmethod
+    def _concat_same_type(cls, to_concat):
+        return cls(np.concatenate(to_concat))
+
+    @property
+    def dtype(self):
+        return self._dtype
+
+    def __array__(self, dtype=None):
+        return np.asarray(self._ndarray, dtype=dtype)
+
+    def __getitem__(self, item):
+        if isinstance(item, type(self)):
+            item = item._ndarray
+
+        result = self._ndarray[item]
+        if not lib.is_scalar(result):
+            result = type(self)(result)
+        return result
+
+    def __setitem__(self, key, value):
+        value = extract_array(value)
+
+        if not lib.is_scalar(key) and is_list_like(key):
+            key = np.asarray(key)
+            if not len(key):
+                # early return to avoid casting unnecessarily.
+                return
+
+        if not lib.is_scalar(value):
+            value = np.asarray(value)
+
+        values = self._ndarray
+        t = np.result_type(value, values)
+        if t != self._ndarray.dtype:
+            values = values.astype(t, casting='safe')
+            values[key] = value
+            self._dtype = NumPyExtensionDtype(t)
+            self._ndarray = values
+        else:
+            self._ndarray[key] = value
+
+    def __len__(self):
+        return len(self._ndarray)
+
+    @property
+    def nbytes(self):
+        return self._ndarray.nbytes
+
+    def isna(self):
+        from pandas import isna
+
+        return isna(self._ndarray)
+
+    def fillna(self, value=None, method=None, limit=None):
+        from pandas.api.types import is_array_like
+        from pandas.util._validators import validate_fillna_kwargs
+        from pandas.core.missing import pad_1d, backfill_1d
+
+        # TODO: really need to implement `_values_for_fillna`.
+        value, method = validate_fillna_kwargs(value, method)
+
+        mask = self.isna()
+
+        if is_array_like(value):
+            if len(value) != len(self):
+                raise ValueError("Length of 'value' does not match. Got ({}) "
+                                 " expected {}".format(len(value), len(self)))
+            value = value[mask]
+
+        if mask.any():
+            if method is not None:
+                func = pad_1d if method == 'pad' else backfill_1d
+                new_values = func(self._ndarray, limit=limit,
+                                  mask=mask)
+                new_values = self._from_sequence(new_values, dtype=self.dtype)
+            else:
+                # fill with value
+                new_values = self.copy()
+                new_values[mask] = value
+        else:
+            new_values = self.copy()
+        return new_values
+
+    def take(self, indices, allow_fill=False, fill_value=None):
+        from pandas.core.algorithms import take
+
+        result = take(self._ndarray, indices, allow_fill=allow_fill,
+                      fill_value=fill_value)
+        return type(self)(result)
+
+    def copy(self, deep=False):
+        return type(self)(self._ndarray.copy())
+
+    def _values_for_argsort(self):
+        return self._ndarray
+
+    def _values_for_factorize(self):
+        return self._ndarray, -1
+
+    def unique(self):
+        from pandas import unique
+
+        return type(self)(unique(self._ndarray))
+
+    def _reduce(self, name, skipna=True, **kwargs):
+        meth = getattr(self, name, None)
+        if meth is None:
+            # raise from the parent
+            super(ExtensionArray, self).__reduce__(
+                name=name, skipna=skipna, **kwargs
+            )
+
+        return meth(skipna=skipna, **kwargs)
+
+    def min(self, skipna=True):
+        return nanops.nanmin(self._ndarray, skipna=skipna)
+
+    def max(self, skipna=True):
+        return nanops.nanmax(self._ndarray, skipna=skipna)
+
+    def any(self, skipna=True):
+        return nanops.nanany(self._ndarray, skipna=skipna)
+
+    def all(self, skipna=True):
+        return nanops.nanall(self._ndarray, skipna=skipna)
+
+    def sum(self, skipna=True, min_count=0):
+        return nanops.nansum(self._ndarray, skipna=skipna,
+                             min_count=min_count)
+
+    def mean(self, skipna=True):
+        return nanops.nanmean(self._ndarray, skipna=skipna)
+
+    def median(self, skipna=True):
+        return nanops.nanmedian(self._ndarray, skipna=skipna)
+
+    def prod(self, min_count=0, skipna=True):
+        return nanops.nanprod(self._ndarray, min_count=min_count,
+                              skipna=skipna)
+
+    def std(self, skipna=True, ddof=1):
+        return nanops.nanstd(self._ndarray, skipna=skipna, ddof=ddof)
+
+    def var(self, skipna=True, ddof=1):
+        return nanops.nanvar(self._ndarray, skipna=skipna, ddof=ddof)
+
+    def kurt(self, skipna=True):
+        return nanops.nankurt(self._ndarray, skipna=skipna)
+
+    def skew(self, skipna=True):
+        return nanops.nanskew(self._ndarray, skipna=skipna)
+
+    def sem(self, skipna=True):
+        return nanops.nansem(self._ndarray, skipna=skipna)
+
+    def __invert__(self):
+        return type(self)(~self._ndarray)
+
+    @classmethod
+    def _create_arithmetic_method(cls, op):
+        def arithmetic_method(self, other):
+            if isinstance(other, (ABCIndexClass, ABCSeries)):
+                return NotImplemented
+
+            elif isinstance(other, cls):
+                other = other._ndarray
+
+            with np.errstate(all="ignore"):
+                result = op(self._ndarray, other)
+
+            if op is divmod:
+                a, b = result
+                return cls(a), cls(b)
+
+            return cls(result)
+
+        return compat.set_function_name(arithmetic_method,
+                                        "__{}__".format(op.__name__),
+                                        cls)
+
+    _create_comparison_method = _create_arithmetic_method
+
+
+NumPyExtensionArray._add_arithmetic_ops()
+NumPyExtensionArray._add_comparison_ops()