ENH: Add dtype-support for pandas' type-hinting

[erezinman]

It would be great if you could annotate a series, for example with -

def process_series(ser: pd.Series[int, str]) -> pd.Series[int, int]:

to indicate that the function should accept a string-valued series with an integer index, and output
int-valued series with an integer index.
It could be even better if you would build on that such that the series member's types could be inferred using TypeVars (for example, pd.Series[int,str].iloc would accept an integer and return a string), but that's not necessary - just a bonus or a later milestone.

To do that, you could add a new module (maybe "pandas.typing"?) that would contain these type-hints and would require minimal integration (if any) into the pandas' infrastructure. There's a similar package for "numpy" that's external to it called nptyping that could be used as a reference.

[simonjayhawkins]

Thanks @erezinman for the report. Adding this would certainly be easier with stubs, but we have not yet reached a decision on stubs, xref #28142.

To do this with type annotations in the code, we would need to use typing.Generic

Thanks for the link. I suspect we would follow NumPy conventions for the type parameters. https://github.com/numpy/numpy-stubs

This might be as simple as writing np.ndarray[np.float64], but will need a decision about appropriate syntax for shape typing to ensure that this is forwards compatible with typing shapes.

In pandas we would also need to allow Series to be backed by pandas Extension Arrays for the values and index, so I suspect the type parameters would be numpy/pandas extension dtypes and not Python types

so maybe something like pd.Series[pd.Int64Dtype(),pd.StringDtype()].

This would be cumbersome so having pd.Series[int, str]) represent the same thing would certainly be welcome from a user perspective.

Further investigation and PRs welcome.

[simonjayhawkins]

fixed in pandas-dev/pandas-stubs@ba7aa5f

[erezinman]

Hi, @simonjayhawkins
Could you please elaborate how this commit solves the issue? I looked at the referred commit and couldn't see how.
Thanks.

[simonjayhawkins]

IIUC Series and Index are now generic wrt dtype in https://github.com/pandas-dev/pandas-stubs

@Dr-Irv is this documented anywhere how to use these?

For DataFrame (not explicitly mentioned in this issue), there is an open issue pandas-dev/pandas-stubs#295

[Dr-Irv]

IIUC Series and Index are now generic wrt dtype in https://github.com/pandas-dev/pandas-stubs

Right now, only Series is generic. We have open issues to investigate making Index generic.

@Dr-Irv is this documented anywhere how to use these?

What we have written up is here: https://github.com/pandas-dev/pandas-stubs/blob/main/docs/philosophy.md#use-of-generic-types

Note that if you want to specify an annotation like def f(x: Series[int]) -> None, you probably have to surround the type declaration with quotes, i.e. def f(x: "Series[int]") -> None because at runtime, the generic declaration is unknown.

It's worth mentioning that the pandera project supports the generic types at both a typing level and at runtime. See https://pandera.readthedocs.io/en/stable/schema_models.html . I haven't used this, but was made aware of it by the pandera authors when they reported some issues with pandas-stubs