Skip to content

Remove pandas.core.index.datetimelike from MyPy Blacklist #26280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jun 25, 2019
Merged

Remove pandas.core.index.datetimelike from MyPy Blacklist #26280

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a3a1605
Add type annotation and remove blacklist item
makbigc May 4, 2019
9c60e73
isort datetimelike.py
makbigc May 8, 2019
717bb83
Add comment for the need of type: ignore
makbigc May 8, 2019
4b3af23
merge
makbigc May 30, 2019
29dbd92
Add ignore comment
makbigc May 30, 2019
213af4a
Add annotation DatetimeLikeArray to _data
makbigc Jun 10, 2019
a5f7d7b
resolve conflict
makbigc Jun 10, 2019
b7f5a9c
Import EA directly
makbigc Jun 11, 2019
2bd7542
Removed DatetimeLikeArray TypeVar for now
WillAyd Jun 24, 2019
d9ab7b0
Removed DatetimeLikeArray annotations
WillAyd Jun 24, 2019
dea0cb1
isort fixup
WillAyd Jun 25, 2019
b061ac1
Update _typing.py
WillAyd Jun 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions mypy.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,4 @@ ignore_missing_imports=True
follow_imports=silent

[mypy-pandas.conftest,pandas.tests.*]
ignore_errors=True

[mypy-pandas.core.indexes.datetimelike]
ignore_errors=True
ignore_errors=True
5 changes: 5 additions & 0 deletions pandas/_typing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,17 @@
from pandas.core.dtypes.generic import (
ABCExtensionArray, ABCIndexClass, ABCSeries, ABCSparseSeries)

from pandas.core.arrays import DatetimeArray, PeriodArray, TimedeltaArray

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don’t try to import these use a quoted version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following is the import error in the failed tests:
ImportError: cannot import name '_INT64_DTYPE'

The name of the module _typing.py is started with underscore. In PEP 8 (https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles ), it states

_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose names start with an underscore.

Should we remove underscore from the name _typing? Or something else?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can’t import these names as they would cause circular imports
and don’t need to just use ‘DatetimeArray’ and such

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_INT64_DTYPE is just an alias, i.e., _INT64_DTYPE = np.dtype(np.int64). The import of numpy and the definition of _INT64_DTYPE are in the same file pandas/core/dtype/common.py.

I did a little experiment. Open a file called 'typing.py' (without underscore prefix) in the same directory as '_typing.py'. One can import DatetimeArray in typing.py.

Importing PeriodArray and TImedeltaArray also run into the same ImportError. Without them, one cannot define the mypy object TypeVar.
DatetimeLikeArray = TypeVar('DatetimeLikeArray', DatetimeArray, PeriodArray, TimedeltaArray).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WillAyd Would you tell me your opinion? Should we change the filename of _typing.py?

AnyArrayLike = TypeVar('AnyArrayLike',
ABCExtensionArray,
ABCIndexClass,
ABCSeries,
ABCSparseSeries,
np.ndarray)
DatetimeLikeArray = TypeVar('DatetimeLikeArray', DatetimeArray,
PeriodArray, TimedeltaArray)

ArrayLike = TypeVar('ArrayLike', ABCExtensionArray, np.ndarray)
DatetimeLikeScalar = TypeVar('DatetimeLikeScalar', Period, Timestamp,
Timedelta)
Expand Down
3 changes: 2 additions & 1 deletion pandas/core/algorithms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@
from pandas.core.dtypes.missing import isna, na_value_for_dtype

from pandas.core import common as com
from type import Dict

_shared_docs = {}
_shared_docs = {} # type: Dict[str, str]


# --------------- #
Expand Down
4 changes: 2 additions & 2 deletions pandas/core/arrays/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
without warning.
"""
import operator
from typing import Any, Callable, Optional, Sequence, Tuple, Union
from typing import Any, Callable, Dict, Optional, Sequence, Tuple, Union

import numpy as np

Expand All @@ -26,7 +26,7 @@

_not_implemented_message = "{} does not implement {}."

_extension_array_shared_docs = dict()
_extension_array_shared_docs = dict() # type: Dict[str, str]


class ExtensionArray:
Expand Down
27 changes: 16 additions & 11 deletions pandas/core/indexes/datetimelike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
Base and utility classes for tseries type pandas objects.
"""
import operator
from typing import Set
import warnings

import numpy as np
Expand All @@ -17,6 +18,7 @@
is_period_dtype, is_scalar)
from pandas.core.dtypes.generic import ABCIndex, ABCIndexClass, ABCSeries

from pandas._typing import DatetimeLikeArray
from pandas.core import algorithms, ops
from pandas.core.accessor import PandasDelegate
from pandas.core.arrays import ExtensionOpsMixin
Expand Down Expand Up @@ -57,19 +59,22 @@ class DatetimeIndexOpsMixin(ExtensionOpsMixin):
"""
common ops mixin to support a unified interface datetimelike Index
"""
_data = None
_data = None # type: DatetimeLikeArray

# DatetimeLikeArrayMixin assumes subclasses are mutable, so these are
# properties there. They can be made into cache_readonly for Index
# subclasses bc they are immutable
inferred_freq = cache_readonly(DatetimeLikeArrayMixin.inferred_freq.fget)
_isnan = cache_readonly(DatetimeLikeArrayMixin._isnan.fget)
hasnans = cache_readonly(DatetimeLikeArrayMixin._hasnans.fget)
inferred_freq = cache_readonly(
DatetimeLikeArrayMixin.inferred_freq.fget) # type: ignore
_isnan = cache_readonly(DatetimeLikeArrayMixin._isnan.fget) # type: ignore
hasnans = cache_readonly(
DatetimeLikeArrayMixin._hasnans.fget) # type: ignore
_hasnans = hasnans # for index / array -agnostic code
_resolution = cache_readonly(DatetimeLikeArrayMixin._resolution.fget)
resolution = cache_readonly(DatetimeLikeArrayMixin.resolution.fget)
_resolution = cache_readonly(
DatetimeLikeArrayMixin._resolution.fget) # type: ignore
resolution = cache_readonly(
DatetimeLikeArrayMixin.resolution.fget) # type: ignore

_box_values = ea_passthrough(DatetimeLikeArrayMixin._box_values)
_maybe_mask_results = ea_passthrough(
DatetimeLikeArrayMixin._maybe_mask_results)
__iter__ = ea_passthrough(DatetimeLikeArrayMixin.__iter__)
Expand Down Expand Up @@ -130,11 +135,11 @@ def _ndarray_values(self):
# Abstract data attributes

@property
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mypy has the following error for the annotated value function:

pandas/core/indexes/datetimelike.py:139: error: "None" has no attribute "_data"

I add annotation # type: DatetimeArray to _data. But mypy still has the same error.

pandas/pandas/core/indexes/datetimes.py

Line 327 in 8154efb

result._data = dtarr

I guess that mypy mixed up the instance attribute _data with the class attributes _data which is set None in the beginning of DatetimeIndexOpsMixin.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @WillAyd

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry missed this ping. This could be a similar conversation to https://github.com/pandas-dev/pandas/pull/26518/files#r287573810

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a better solution is to provide the appropriate type to _data. Can you create a DatetimeLikeArray in pandas._typing which should be something like DatetimeLikeArray = TypeVar('DatetimeLikeArray', DatetimeArray, PeriodArray, TimedeltaArray) and assign that type to _data in the class? I think that should resolve.

cc @jbrockmendel in case he has other insights on the types here

def values(self) -> np.ndarray:
def values(self):
# Note: PeriodArray overrides this to return an ndarray of objects.
return self._data._data

@property
@property # type: ignore # https://github.com/python/mypy/issues/1362
@Appender(DatetimeLikeArrayMixin.asi8.__doc__)
def asi8(self):
return self._data.asi8
Expand Down Expand Up @@ -761,9 +766,9 @@ class DatetimelikeDelegateMixin(PandasDelegate):
boxed in an index, after being returned from the array
"""
# raw_methods : dispatch methods that shouldn't be boxed in an Index
_raw_methods = set()
_raw_methods = set() # type: Set[str]
# raw_properties : dispatch properties that shouldn't be boxed in an Index
_raw_properties = set()
_raw_properties = set() # type: Set[str]
name = None
_data = None

Expand Down