TYP: Typing changes for ExtensionArray.astype #41251

Dr-Irv · 2021-05-01T19:37:45Z

Minimal (hopefully) typing changes for ExtensionArray.astype .

jbrockmendel · 2021-05-01T19:51:20Z

pandas/core/arrays/base.py

@@ -511,7 +513,15 @@ def nbytes(self) -> int:
    # Additional Methods
    # ------------------------------------------------------------------------

-    def astype(self, dtype, copy=True):
+    @overload
+    def astype(self, dtype: NpDtype, copy: bool = True) -> np.ndarray:


dt64/td64 generally go to DTA/TDA

Don't think so. When you pass a numpy dtype, you get a np.ndarray:

>>> a=pd.array([np.datetime64("2021-03-15"), np.datetime64("2021-04-05")]) >>> a <DatetimeArray> ['2021-03-15 00:00:00', '2021-04-05 00:00:00'] Length: 2, dtype: datetime64[ns] >>> b=a.astype(np.datetime64) >>> b array(['2021-03-15T00:00:00.000000000', '2021-04-05T00:00:00.000000000'], dtype='datetime64[ns]') >>> type(b) <class 'numpy.ndarray'>

pandas/core/arrays/base.py

Dr-Irv · 2021-05-16T14:18:24Z

@simonjayhawkins can you review, please?

simonjayhawkins · 2021-05-22T17:04:02Z

This is now failing mypy with master merged. can you merge master and fixup.

pandas/core/groupby/groupby.py

pandas/core/internals/blocks.py

pandas/core/arrays/base.py

pandas/_testing/asserters.py

simonjayhawkins · 2021-05-22T17:41:47Z

pandas/core/arrays/base.py

+        ...
+
+    @overload
+    def astype(self, dtype: Dtype, copy: bool = True) -> ArrayLike:


I think we could add an overload for ExtensionDtype and this overload could be str for the union return type if type_t[Union[str, float, int, complex, bool, object] is moved as suggested?

the accepted strings for numpy is already known? so we should eventually be able to remove the union return type.

pandas/core/arrays/base.py

Dr-Irv · 2021-05-30T17:49:19Z

@simonjayhawkins pushed changes a week ago, so looking forward to a new review.

pandas/core/arrays/base.py

pandas/core/construction.py

jbrockmendel · 2021-05-30T21:13:46Z

pandas/core/dtypes/dtypes.py

@@ -1291,7 +1290,7 @@ class PandasDtype(ExtensionDtype):

    _metadata = ("_dtype",)

-    def __init__(self, dtype: NpDtype | PandasDtype | None):
+    def __init__(self, dtype: str | np.dtype | PandasDtype | None):


why this change? what prevents me from passing int?

Fails mypy for some reason. Might have to do with the typing of how the function np.dtype is typed by numpy

PandasDtype(int) returns PandasDtype('int64')

pls let the question-asker do the "mark as resolved"

does npt.DTypeLike work?

(from @Dr-Irv): Yes, it does. In next commit.

simonjayhawkins

This branch is 260 commits behind master and if merged would cause a CI failure

pandas/core/arrays/period.py:346: error: unused "type: ignore" comment

simonjayhawkins

Thanks @Dr-Irv

is it worth posting the results of a test file with reveal_types for this?

simonjayhawkins · 2021-07-04T15:34:23Z

pandas/_testing/asserters.py

-                np.sum((left._values != right._values).astype(int)) * 100.0 / len(left)
-            )
+            thesum = np.sum((left._values != right._values).astype(int))
+            diff = thesum * 100.0 / len(left)


why is this changed?

Must have had to change it in a previous commit with an earlier version of numpy. Reverted in next commit.

simonjayhawkins · 2021-07-04T15:49:09Z

pandas/_typing.py

+NpDtype = Union[str, np.dtype, type_t[Union[str, float, int, complex, bool, object]]]
+Dtype = Union["ExtensionDtype", NpDtype]


does this PR need to go in before #41203?

No, they should be independent. Either can go first, and once one goes in, I can do the mypy check on the other.

Would it make sense to use NpDtype = npt.DTypeLike to be consistent between numpy and pandas?

Would it make sense to use NpDtype = npt.DTypeLike to be consistent between numpy and pandas?

That's a big change, as it affects a lot more than just the astype() signature. The types are different:

>>> import numpy.typing as npt >>> from pandas._typing import NpDtype >>> NpDtype typing.Union[str, numpy.dtype, typing.Type[typing.Union[str, float, int, complex, bool, object]]] >>> npt.DTypeLike typing.Union[numpy.dtype, NoneType, type, numpy.typing._dtype_like._SupportsDType[numpy.dtype], str, typing.Tuple[typing.Any, int], typing.Tuple[typing.Any, typing.Union[typing.SupportsIndex, typing.Sequence[typing.SupportsIndex]]], typing.List[typing.Any], numpy.typing._dtype_like._DTypeDict, typing.Tuple[typing.Any, typing.Any]]

simonjayhawkins · 2021-07-04T15:49:52Z

pandas/core/arrays/boolean.py

@@ -392,7 +396,16 @@ def reconstruct(x):
    def _coerce_to_array(self, value) -> tuple[np.ndarray, np.ndarray]:
        return coerce_to_array(value)

-    def astype(self, dtype, copy: bool = True) -> ArrayLike:
+    @overload
+    def astype(self, dtype: NpDtype, copy: bool = True) -> np.ndarray:


Suggested change

def astype(self, dtype: NpDtype, copy: bool = True) -> np.ndarray:

def astype(self, dtype: NpDtype, copy: bool = ...) -> np.ndarray:

and elsewhere

fixed in next commit

simonjayhawkins · 2021-07-04T15:50:47Z

pandas/core/arrays/base.py

@@ -515,9 +517,17 @@ def nbytes(self) -> int:
    # Additional Methods
    # ------------------------------------------------------------------------

-    def astype(self, dtype, copy=True):
+    @overload
+    def astype(self, dtype: NpDtype, copy: bool = ...) -> np.ndarray:


could this use npt.DtypeLike?

could this use npt.DtypeLike?

No. The following code is fine with mypy

import numpy.typing as npt class Foo: pass def myfun(x: npt.DTypeLike): pass myfun(type(Foo)) myfun("abcd") myfun(str) myfun(object) myfun("str")

That's because npt.DTypeLike accepts any type or any str. We want to separate out the ExtensionDtype that are passed as strings.

Never mind. Was able to get this to work by making astype() accept npt.DTypeLike. Created a new type AstypeArg to mean ExtensionDtype | npt.DTypeLike so that I don't have to change lots of other parts where NpDtype is used.

simonjayhawkins · 2021-07-04T15:51:36Z

pandas/core/arrays/base.py

+        ...
+
+    @overload
+    def astype(self, dtype: Dtype, copy: bool = ...) -> ArrayLike:


if we pass a EA dtype object, we get a union return type?

Good point. Fixed in next commit

simonjayhawkins · 2021-07-04T15:57:53Z

pandas/core/dtypes/dtypes.py

@@ -1291,7 +1290,7 @@ class PandasDtype(ExtensionDtype):

    _metadata = ("_dtype",)

-    def __init__(self, dtype: NpDtype | PandasDtype | None):
+    def __init__(self, dtype: str | np.dtype | PandasDtype | None):


does npt.DTypeLike work?

(from @Dr-Irv): Yes, it does. In next commit.

Dr-Irv · 2021-07-04T21:59:02Z

Thanks @Dr-Irv

is it worth posting the results of a test file with reveal_types for this?

Can you point me to an example?

Dr-Irv · 2021-07-14T18:51:45Z

@simonjayhawkins ping...

Dr-Irv · 2021-07-26T11:36:57Z

pinging @simonjayhawkins

jreback · 2021-07-28T01:43:47Z

@Dr-Irv thanks for the patience. can you rebase

@simonjayhawkins if you can have a look

Dr-Irv · 2021-08-03T16:16:16Z

weekly ping @simonjayhawkins

Dr-Irv · 2021-08-23T01:46:42Z

another ping @simonjayhawkins

Dr-Irv · 2021-09-06T15:45:51Z

biweekly ping @simonjayhawkins

jreback · 2021-09-06T15:47:52Z

@Dr-Irv ok if you can merge master and make sure still passing (i think it was last) time, will merge these.

Dr-Irv · 2021-09-06T17:04:12Z

@jreback all green except for a Windows timeout on one of the tests

jreback · 2021-09-06T17:10:57Z

can you rebase again just to be sure (as merged the other)

cc @twoertwein

pandas/core/arrays/integer.py

Dr-Irv · 2021-09-06T19:47:07Z

@jreback all green after merge with the delete/searchsorted changes

jreback · 2021-09-06T19:48:35Z

thanks @Dr-Irv

TYP: Typing changes for ExtensionArray.astype

dda21d1

Dr-Irv requested a review from simonjayhawkins May 1, 2021 19:37

jbrockmendel reviewed May 1, 2021

View reviewed changes

Dr-Irv mentioned this pull request May 2, 2021

TYP: Update type hints for ExtensionArray and ExtensionDtype #39501

Closed

simonjayhawkins added ExtensionArray Extending pandas with custom dtypes or arrays. Typing type annotations, mypy/pyright type checking labels May 2, 2021

simonjayhawkins reviewed May 10, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 10, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 10, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

replace casts with type-ignore and update docs on astype

da626a0

Dr-Irv requested a review from simonjayhawkins May 10, 2021 12:17

simonjayhawkins reviewed May 22, 2021

View reviewed changes

pandas/core/groupby/groupby.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 22, 2021

View reviewed changes

pandas/core/internals/blocks.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 22, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 22, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 22, 2021

View reviewed changes

pandas/_testing/asserters.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed May 22, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

simonjayhawkins added this to the 1.3 milestone May 22, 2021

Dr-Irv added 2 commits May 23, 2021 11:28

Merge remote-tracking branch 'upstream/master' into type_astype

e2f27f7

Change NpDtype

ed106fc

simonjayhawkins removed this from the 1.3 milestone May 24, 2021

Dr-Irv mentioned this pull request May 25, 2021

REF: standardize astype in EA subclasses #41652

Closed

jbrockmendel reviewed May 30, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

jbrockmendel reviewed May 30, 2021

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

jbrockmendel reviewed May 30, 2021

View reviewed changes

pandas/core/construction.py Outdated Show resolved Hide resolved

jbrockmendel reviewed May 30, 2021

View reviewed changes

simonjayhawkins requested changes Jul 4, 2021

View reviewed changes

simonjayhawkins reviewed Jul 4, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into type_astype

4815ede

Dr-Irv added 5 commits July 5, 2021 13:44

add ExtensionDType argument

a61e7ae

Use npt.DTypeLike

80d780e

quotes around npt.DTypeLike

5ce17cb

Merge remote-tracking branch 'upstream/master' into type_astype

5d74047

Merge remote-tracking branch 'upstream/master' into type_astype

1f19758

Dr-Irv added 3 commits July 28, 2021 08:14

Merge remote-tracking branch 'upstream/master' into type_astype

a6d7ebe

Merge remote-tracking branch 'upstream/master' into type_astype

84777e6

Merge remote-tracking branch 'upstream/master' into type_astype

6a135f2

Merge remote-tracking branch 'upstream/master' into type_astype

9282e8e

jreback added this to the 1.4 milestone Sep 6, 2021

merge searchsorted and astype changes

640f4bc

twoertwein reviewed Sep 6, 2021

View reviewed changes

pandas/core/arrays/integer.py Show resolved Hide resolved

jreback merged commit 40274ac into pandas-dev:master Sep 6, 2021

Dr-Irv deleted the type_astype branch September 6, 2021 19:48

feefladder pushed a commit to feefladder/pandas that referenced this pull request Sep 7, 2021

TYP: Typing changes for ExtensionArray.astype (pandas-dev#41251)

f8c1e59

		NpDtype = Union[str, np.dtype, type_t[Union[str, float, int, complex, bool, object]]]
		Dtype = Union["ExtensionDtype", NpDtype]

	def astype(self, dtype: NpDtype, copy: bool = True) -> np.ndarray:
	def astype(self, dtype: NpDtype, copy: bool = ...) -> np.ndarray:

TYP: Typing changes for ExtensionArray.astype #41251

TYP: Typing changes for ExtensionArray.astype #41251

Conversation

Dr-Irv commented May 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dr-Irv commented May 16, 2021

simonjayhawkins commented May 22, 2021

Choose a reason for hiding this comment

Dr-Irv commented May 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins Jul 4, 2021 • edited by Dr-Irv Loading

Choose a reason for hiding this comment

simonjayhawkins left a comment

Choose a reason for hiding this comment

simonjayhawkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins Jul 4, 2021 • edited by Dr-Irv Loading

Choose a reason for hiding this comment

Dr-Irv commented Jul 4, 2021

Dr-Irv commented Jul 14, 2021

Dr-Irv commented Jul 26, 2021

jreback commented Jul 28, 2021

Dr-Irv commented Aug 3, 2021

Dr-Irv commented Aug 23, 2021

Dr-Irv commented Sep 6, 2021

jreback commented Sep 6, 2021

Dr-Irv commented Sep 6, 2021

jreback commented Sep 6, 2021

Dr-Irv commented Sep 6, 2021

jreback commented Sep 6, 2021

simonjayhawkins Jul 4, 2021 •

edited by Dr-Irv

Loading

simonjayhawkins Jul 4, 2021 •

edited by Dr-Irv

Loading