REF: share _simple_new

[jbrockmendel]

@simonjayhawkins two questions: 1) is there a nice way to annotate the values arg (maybe a Protocol thing?) and 2) is the use of cls.__annotations__["_data"] an egregious antipattern?

[jreback]

lgtm

[simonjayhawkins]

2. is the use of cls.__annotations__["_data"] an egregious antipattern?

to begin with a quote from Fluent Python, p8

"The first thing to know about special methods is that they are meant to be called by the Python interpreter, and not by you"

using type annotations in a runtime context is not something we've discussed.

I think in general it could be fragile. we can only use with certain types so cannot now change cls._data for the subclasses without potentially breaking the code. So we wouldn't, for example, be able to makes Indexes generic, e.g. Categorical[str] if we used this pattern in other Indexes.

>>> from pandas.core.indexes.datetimelike import DatetimeIndexOpsMixin
>>>
>>> DatetimeIndexOpsMixin.__annotations__
{'_data': typing.Union[pandas.core.arrays.datetimes.DatetimeArray, pandas.core.a
rrays.timedeltas.TimedeltaArray, pandas.core.arrays.period.PeriodArray], 'freq':
 typing.Union[pandas._libs.tslibs.offsets.BaseOffset, NoneType], 'freqstr': typi
ng.Union[str, NoneType], '_resolution_obj': <enum 'Resolution'>, '_bool_ops': ty
ping.List[str], '_field_ops': typing.List[str]}
>>>
>>> isinstance(42, DatetimeIndexOpsMixin.__annotations__["_data"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\anaconda3\envs\pandas-dev\lib\typing.py", line 769, in __
instancecheck__
    return self.__subclasscheck__(type(obj))
  File "C:\Users\simon\anaconda3\envs\pandas-dev\lib\typing.py", line 777, in __
subclasscheck__
    raise TypeError("Subscripted generics cannot be used with"
TypeError: Subscripted generics cannot be used with class and instance checks
>>>

assuming that the assert has been added for runtime debugging and not to appease mypy:

can you not add something akin to construct_array_type from ExtensionDtype as a class attribute to Index to store the backing array type?

1. is there a nice way to annotate the values arg (maybe a Protocol thing?)

is the reason type annotations for values has not been added related to the above?

[jbrockmendel]

can you not add something akin to construct_array_type from ExtensionDtype as a class attribute to Index to store the backing array type?

will do

is there a nice way to annotate the values arg (maybe a Protocol thing?)

is the reason type annotations for values has not been added related to the above?

I can add an annotation with Union[DatetimeArray, TimedeltaArray, PeriodArray], was wondering if there is a way to make it isolate just the one of those three (like the annotations in the current code with three separate simple_new implementations)

[simonjayhawkins]

is there a nice way to annotate the values arg (maybe a Protocol thing?)

is the reason type annotations for values has not been added related to the above?

I can add an annotation with Union[DatetimeArray, TimedeltaArray, PeriodArray], was wondering if there is a way to make it isolate just the one of those three (like the annotations in the current code with three separate simple_new implementations)

right. so we want to tie the type of values, a parameter of a class method, to the type of self._data, a class attribute (and maybe a construct_array_type like attribute)? I think this is where we would use Generic. the backing array type would be the type parameter.

so something like (completely untested!)

(ArrayLikeT is the same as ArrayLike but we want to change that to a Union, and we want a typevar here)

ArrayLikeT = TypeVar("ArrayLikeT", "ExtensionArray", np.ndarray)

class Index(IndexOpsMixin, PandasObject, Generic(ArrayLikeT)):

    _data : ArrayLikeT
    _construct_array_type: Type[ArrayLikeT]

    def _simple_new(cls, values: ArrayLikeT, name: Label = None):
         ...

[simonjayhawkins]

Thanks @jbrockmendel lgtm pending green

[simonjayhawkins]

because this is the base class it should be ArrayLike? (or we can leave this fix for #36092)

[jbrockmendel]

no, its specifically np.ndarray

[simonjayhawkins]

hmm, I guess you want to make Index np.ndarray specific and have ExtensionIndex handle EA backed indexes but in that case some other type annotations in base would need to be changed e,g, _data: Union[ExtensionArray, np.ndarray] so that methods in the base class that currently handle EA can be simplified and overridden.

In #37898, im getting error: Unsupported left operand type for - ("ExtensionArray") in the base class since def _values(self) -> Union[ExtensionArray, np.ndarray] so atm the base class is not EA agnostic.

[jbrockmendel]

updated to get revert this annotation

[simonjayhawkins]

can now just use Hashable, see #37134

[jbrockmendel]

OK, will edit on subsequent pass

[jbrockmendel]

updated + green