Skip to content

TYP: Typing changes for ExtensionArray.astype #41251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Sep 6, 2021
Merged

TYP: Typing changes for ExtensionArray.astype #41251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
dda21d1
TYP: Typing changes for ExtensionArray.astype
Dr-Irv May 1, 2021
da626a0
replace casts with type-ignore and update docs on astype
Dr-Irv May 10, 2021
e2f27f7
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv May 23, 2021
ed106fc
Change NpDtype
Dr-Irv May 23, 2021
7de25ae
Merge branch 'master' into type_astype
Dr-Irv May 31, 2021
d9ef38f
doc changes. Use np.dtype(object)
Dr-Irv May 31, 2021
1807fd2
merge with master
Dr-Irv Jun 1, 2021
e683efe
fix indentation in docstring
Dr-Irv Jun 1, 2021
ffee0e5
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Jul 3, 2021
4815ede
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Jul 4, 2021
a61e7ae
add ExtensionDType argument
Dr-Irv Jul 5, 2021
80d780e
Use npt.DTypeLike
Dr-Irv Jul 5, 2021
5ce17cb
quotes around npt.DTypeLike
Dr-Irv Jul 5, 2021
5d74047
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Jul 6, 2021
1f19758
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Jul 8, 2021
a6d7ebe
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Jul 28, 2021
84777e6
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Jul 29, 2021
6a135f2
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Aug 3, 2021
9282e8e
Merge remote-tracking branch 'upstream/master' into type_astype
Dr-Irv Sep 6, 2021
640f4bc
merge searchsorted and astype changes
Dr-Irv Sep 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions pandas/_typing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,10 +126,9 @@
]

# dtypes
NpDtype = Union[str, np.dtype]
Dtype = Union[
"ExtensionDtype", NpDtype, type_t[Union[str, float, int, complex, bool, object]]
]
NpDtype = Union[str, np.dtype, type_t[Union[str, float, int, complex, bool, object]]]
Dtype = Union["ExtensionDtype", NpDtype]
Comment on lines +129 to +130
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this PR need to go in before #41203?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they should be independent. Either can go first, and once one goes in, I can do the mypy check on the other.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to use NpDtype = npt.DTypeLike to be consistent between numpy and pandas?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to use NpDtype = npt.DTypeLike to be consistent between numpy and pandas?

That's a big change, as it affects a lot more than just the astype() signature. The types are different:

>>> import numpy.typing as npt
>>> from pandas._typing import NpDtype
>>> NpDtype
typing.Union[str, numpy.dtype, typing.Type[typing.Union[str, float, int, complex, bool, object]]]
>>> npt.DTypeLike
typing.Union[numpy.dtype, NoneType, type, numpy.typing._dtype_like._SupportsDType[numpy.dtype], str, typing.Tuple[typing.Any, int], typing.Tuple[typing.Any, typing.Union[typing.SupportsIndex, typing.Sequence[typing.SupportsIndex]]], typing.List[typing.Any], numpy.typing._dtype_like._DTypeDict, typing.Tuple[typing.Any, typing.Any]]

AstypeArg = Union["ExtensionDtype", "npt.DTypeLike"]
# DtypeArg specifies all allowable dtypes in a functions its dtype argument
DtypeArg = Union[Dtype, Dict[Hashable, Dtype]]
DtypeObj = Union[np.dtype, "ExtensionDtype"]
Expand Down
32 changes: 27 additions & 5 deletions pandas/core/arrays/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,15 @@
Sequence,
TypeVar,
cast,
overload,
)

import numpy as np

from pandas._libs import lib
from pandas._typing import (
ArrayLike,
AstypeArg,
Dtype,
FillnaOptions,
PositionalIndexer,
Expand Down Expand Up @@ -520,9 +522,21 @@ def nbytes(self) -> int:
# Additional Methods
# ------------------------------------------------------------------------

def astype(self, dtype, copy=True):
@overload
def astype(self, dtype: npt.DTypeLike, copy: bool = ...) -> np.ndarray:
...

@overload
def astype(self, dtype: ExtensionDtype, copy: bool = ...) -> ExtensionArray:
...

@overload
def astype(self, dtype: AstypeArg, copy: bool = ...) -> ArrayLike:
...

def astype(self, dtype: AstypeArg, copy: bool = True) -> ArrayLike:
"""
Cast to a NumPy array with 'dtype'.
Cast to a NumPy array or ExtensionArray with 'dtype'.

Parameters
----------
Expand All @@ -535,8 +549,10 @@ def astype(self, dtype, copy=True):

Returns
-------
array : ndarray
NumPy ndarray with 'dtype' for its dtype.
array : np.ndarray or ExtensionArray
An ExtensionArray if dtype is StringDtype,
or same as that of underlying array.
Otherwise a NumPy ndarray with 'dtype' for its dtype.
"""
from pandas.core.arrays.string_ import StringDtype

Expand All @@ -552,7 +568,11 @@ def astype(self, dtype, copy=True):
# allow conversion to StringArrays
return dtype.construct_array_type()._from_sequence(self, copy=False)

return np.array(self, dtype=dtype, copy=copy)
# error: Argument "dtype" to "array" has incompatible type
# "Union[ExtensionDtype, dtype[Any]]"; expected "Union[dtype[Any], None, type,
# _SupportsDType, str, Union[Tuple[Any, int], Tuple[Any, Union[int,
# Sequence[int]]], List[Any], _DTypeDict, Tuple[Any, Any]]]"
return np.array(self, dtype=dtype, copy=copy) # type: ignore[arg-type]

def isna(self) -> np.ndarray | ExtensionArraySupportsAnyAll:
"""
Expand Down Expand Up @@ -863,6 +883,8 @@ def searchsorted(
# 2. Values between the values in the `data_for_sorting` fixture
# 3. Missing values.
arr = self.astype(object)
if isinstance(value, ExtensionArray):
value = value.astype(object)
return arr.searchsorted(value, side=side, sorter=sorter)

def equals(self, other: object) -> bool:
Expand Down
23 changes: 21 additions & 2 deletions pandas/core/arrays/boolean.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
from __future__ import annotations

import numbers
from typing import TYPE_CHECKING
from typing import (
TYPE_CHECKING,
overload,
)
import warnings

import numpy as np
Expand All @@ -12,7 +15,9 @@
)
from pandas._typing import (
ArrayLike,
AstypeArg,
Dtype,
npt,
type_t,
)
from pandas.compat.numpy import function as nv
Expand All @@ -33,6 +38,7 @@
from pandas.core.dtypes.missing import isna

from pandas.core import ops
from pandas.core.arrays import ExtensionArray
from pandas.core.arrays.masked import (
BaseMaskedArray,
BaseMaskedDtype,
Expand Down Expand Up @@ -392,7 +398,20 @@ def reconstruct(x):
def _coerce_to_array(self, value) -> tuple[np.ndarray, np.ndarray]:
return coerce_to_array(value)

def astype(self, dtype, copy: bool = True) -> ArrayLike:
@overload
def astype(self, dtype: npt.DTypeLike, copy: bool = ...) -> np.ndarray:
...

@overload
def astype(self, dtype: ExtensionDtype, copy: bool = ...) -> ExtensionArray:
...

@overload
def astype(self, dtype: AstypeArg, copy: bool = ...) -> ArrayLike:
...

def astype(self, dtype: AstypeArg, copy: bool = True) -> ArrayLike:

"""
Cast to a NumPy array or ExtensionArray with 'dtype'.

Expand Down
22 changes: 16 additions & 6 deletions pandas/core/arrays/categorical.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
TypeVar,
Union,
cast,
overload,
)
from warnings import (
catch_warnings,
Expand All @@ -32,6 +33,7 @@
from pandas._libs.lib import no_default
from pandas._typing import (
ArrayLike,
AstypeArg,
Dtype,
NpDtype,
Ordered,
Expand Down Expand Up @@ -482,7 +484,19 @@ def _constructor(self) -> type[Categorical]:
def _from_sequence(cls, scalars, *, dtype: Dtype | None = None, copy=False):
return Categorical(scalars, dtype=dtype, copy=copy)

def astype(self, dtype: Dtype, copy: bool = True) -> ArrayLike:
@overload
def astype(self, dtype: npt.DTypeLike, copy: bool = ...) -> np.ndarray:
...

@overload
def astype(self, dtype: ExtensionDtype, copy: bool = ...) -> ExtensionArray:
...

@overload
def astype(self, dtype: AstypeArg, copy: bool = ...) -> ArrayLike:
...

def astype(self, dtype: AstypeArg, copy: bool = True) -> ArrayLike:
"""
Coerce this type to another dtype

Expand Down Expand Up @@ -2458,11 +2472,7 @@ def _str_get_dummies(self, sep="|"):
# sep may not be in categories. Just bail on this.
from pandas.core.arrays import PandasArray

# error: Argument 1 to "PandasArray" has incompatible type
# "ExtensionArray"; expected "Union[ndarray, PandasArray]"
return PandasArray(self.astype(str))._str_get_dummies( # type: ignore[arg-type]
sep
)
return PandasArray(self.astype(str))._str_get_dummies(sep)


# The Series.cat accessor
Expand Down
18 changes: 17 additions & 1 deletion pandas/core/arrays/floating.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

from typing import overload
import warnings

import numpy as np
Expand All @@ -10,7 +11,9 @@
)
from pandas._typing import (
ArrayLike,
AstypeArg,
DtypeObj,
npt,
)
from pandas.compat.numpy import function as nv
from pandas.util._decorators import cache_readonly
Expand All @@ -31,6 +34,7 @@
)
from pandas.core.dtypes.missing import isna

from pandas.core.arrays import ExtensionArray
from pandas.core.arrays.numeric import (
NumericArray,
NumericDtype,
Expand Down Expand Up @@ -271,7 +275,19 @@ def _from_sequence_of_strings(
def _coerce_to_array(self, value) -> tuple[np.ndarray, np.ndarray]:
return coerce_to_array(value, dtype=self.dtype)

def astype(self, dtype, copy: bool = True) -> ArrayLike:
@overload
def astype(self, dtype: npt.DTypeLike, copy: bool = ...) -> np.ndarray:
...

@overload
def astype(self, dtype: ExtensionDtype, copy: bool = ...) -> ExtensionArray:
...

@overload
def astype(self, dtype: AstypeArg, copy: bool = ...) -> ArrayLike:
...

def astype(self, dtype: AstypeArg, copy: bool = True) -> ArrayLike:
"""
Cast to a NumPy array or ExtensionArray with 'dtype'.

Expand Down
18 changes: 17 additions & 1 deletion pandas/core/arrays/integer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

from typing import overload
import warnings

import numpy as np
Expand All @@ -11,8 +12,10 @@
)
from pandas._typing import (
ArrayLike,
AstypeArg,
Dtype,
DtypeObj,
npt,
)
from pandas.compat.numpy import function as nv
from pandas.util._decorators import cache_readonly
Expand All @@ -33,6 +36,7 @@
)
from pandas.core.dtypes.missing import isna

from pandas.core.arrays import ExtensionArray
from pandas.core.arrays.masked import (
BaseMaskedArray,
BaseMaskedDtype,
Expand Down Expand Up @@ -333,7 +337,19 @@ def _from_sequence_of_strings(
def _coerce_to_array(self, value) -> tuple[np.ndarray, np.ndarray]:
return coerce_to_array(value, dtype=self.dtype)

def astype(self, dtype, copy: bool = True) -> ArrayLike:
@overload
def astype(self, dtype: npt.DTypeLike, copy: bool = ...) -> np.ndarray:
...

@overload
def astype(self, dtype: ExtensionDtype, copy: bool = ...) -> ExtensionArray:
...

@overload
def astype(self, dtype: AstypeArg, copy: bool = ...) -> ArrayLike:
...

def astype(self, dtype: AstypeArg, copy: bool = True) -> ArrayLike:
"""
Cast to a NumPy array or ExtensionArray with 'dtype'.

Expand Down
22 changes: 17 additions & 5 deletions pandas/core/arrays/masked.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
Any,
Sequence,
TypeVar,
overload,
)

import numpy as np
Expand All @@ -15,10 +16,11 @@
)
from pandas._typing import (
ArrayLike,
Dtype,
AstypeArg,
NpDtype,
PositionalIndexer,
Scalar,
npt,
type_t,
)
from pandas.errors import AbstractMethodError
Expand Down Expand Up @@ -282,9 +284,7 @@ def to_numpy( # type: ignore[override]
if na_value is lib.no_default:
na_value = libmissing.NA
if dtype is None:
# error: Incompatible types in assignment (expression has type
# "Type[object]", variable has type "Union[str, dtype[Any], None]")
dtype = object # type: ignore[assignment]
dtype = object
if self._hasna:
if (
not is_object_dtype(dtype)
Expand All @@ -303,7 +303,19 @@ def to_numpy( # type: ignore[override]
data = self._data.astype(dtype, copy=copy)
return data

def astype(self, dtype: Dtype, copy: bool = True) -> ArrayLike:
@overload
def astype(self, dtype: npt.DTypeLike, copy: bool = ...) -> np.ndarray:
...

@overload
def astype(self, dtype: ExtensionDtype, copy: bool = ...) -> ExtensionArray:
...

@overload
def astype(self, dtype: AstypeArg, copy: bool = ...) -> ArrayLike:
...

def astype(self, dtype: AstypeArg, copy: bool = True) -> ArrayLike:
dtype = pandas_dtype(dtype)

if is_dtype_equal(dtype, self.dtype):
Expand Down
4 changes: 1 addition & 3 deletions pandas/core/arrays/period.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,9 +351,7 @@ def freq(self) -> BaseOffset:
def __array__(self, dtype: NpDtype | None = None) -> np.ndarray:
if dtype == "i8":
return self.asi8
# error: Non-overlapping equality check (left operand type: "Optional[Union[str,
# dtype[Any]]]", right operand type: "Type[bool]")
elif dtype == bool: # type: ignore[comparison-overlap]
elif dtype == bool:
return ~self._isnan

# This will raise TypeError for non-object dtypes
Expand Down
7 changes: 3 additions & 4 deletions pandas/core/arrays/sparse/array.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
from pandas._libs.tslibs import NaT
from pandas._typing import (
ArrayLike,
AstypeArg,
Dtype,
NpDtype,
Scalar,
Expand Down Expand Up @@ -527,9 +528,7 @@ def __array__(self, dtype: NpDtype | None = None) -> np.ndarray:
try:
dtype = np.result_type(self.sp_values.dtype, type(fill_value))
except TypeError:
# error: Incompatible types in assignment (expression has type
# "Type[object]", variable has type "Union[str, dtype[Any], None]")
dtype = object # type: ignore[assignment]
dtype = object

out = np.full(self.shape, fill_value, dtype=dtype)
out[self.sp_index.to_int_index().indices] = self.sp_values
Expand Down Expand Up @@ -1072,7 +1071,7 @@ def _concat_same_type(

return cls(data, sparse_index=sp_index, fill_value=fill_value)

def astype(self, dtype: Dtype | None = None, copy=True):
def astype(self, dtype: AstypeArg | None = None, copy=True):
"""
Change the dtype of a SparseArray.

Expand Down
7 changes: 1 addition & 6 deletions pandas/core/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,12 +232,7 @@ def asarray_tuplesafe(values, dtype: NpDtype | None = None) -> np.ndarray:
# expected "ndarray")
return values._values # type: ignore[return-value]

# error: Non-overlapping container check (element type: "Union[str, dtype[Any],
# None]", container item type: "type")
if isinstance(values, list) and dtype in [ # type: ignore[comparison-overlap]
np.object_,
object,
]:
if isinstance(values, list) and dtype in [np.object_, object]:
return construct_1d_object_array_from_listlike(values)

result = np.asarray(values, dtype=dtype)
Expand Down
Loading