API: Add pandas.api.typing

[rhshadrach]

• closes API: Add pandas.api.typing #48577 (Replace xxxx with the Github issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[rhshadrach]

Marking this as a draft until the discussion in the corresponding issue is resolved.

[phofl]

Can you merge main?

[rhshadrach]

@phofl - done.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[rhshadrach]

@simonjayhawkins - friendly ping

[simonjayhawkins]

Thanks @rhshadrach lgtm pending green. and conflicts to resolve

I think this could be coupled to #27522, so as a follow-up or here if others think that the coupling is strong (e.g. need for api namespace)

[rhshadrach]

Thanks @simonjayhawkins for the review

I think this could be coupled to #27522, so as a follow-up or here if others think that the coupling is strong (e.g. need for api namespace)

The issue with the Styler is still a blocker here, and I plan to address it in a followup PR. Last I looked, it would require a bit of refactoring. Because of this and the amount of changes deprecating pandas.core would take, I'm quite resistant to including such work here.

[simonjayhawkins]

I'm quite resistant to including such work here.

sure. if we deprecate core it would enforce the usage of pandas[.api].typing but I guess changing the api docs for the public classes currently in core is a step in the right direction.

[rhshadrach]

@simonjayhawkins - friendly ping

[rhshadrach]

@Dr-Irv - friendly ping.

[Dr-Irv]

@rhshadrach I may be being a bit pedantic here, but now the docs don't correspond to what happens at runtime, and maybe you should add tests for this.

Consider this:

>>> import pandas as pd
>>> df=pd.DataFrame({"x":[i for i in range(10)]}, index=pd.date_range("230101", periods=10))
>>> df
            x
2001-01-23  0
2001-01-24  1
2001-01-25  2
2001-01-26  3
2001-01-27  4
2001-01-28  5
2001-01-29  6
2001-01-30  7
2001-01-31  8
2001-02-01  9
>>> df.resample("2d")
<pandas.core.resample.DatetimeIndexResampler object at 0x000002425662CF40>
>>> type(df.rolling(2))
<class 'pandas.core.window.rolling.Rolling'>

So the docs are saying that df.resample() is returning a pandas.api.typing.Resampler, but that's not the user experience. The docs say that df.rolling() returns a pandas.api.typing.Rolling, but that's not the user experience. I think the same is true for the other types as well.

Now it is probably the case that the type pandas.api.typing.Resampler is the same as pandas.core.resample.DatetimeIndex.Resampler and that the type pandas.api.typingRolling is the same as pandas.core.window.rolling.Rolling, but shouldn't we have tests for each of these types you created to make sure that the types we document that we return do correspond to the actual types returned?

I'm probably suggesting this because when we change things in pandas-stubs to correspond to this change, we will end up creating those tests there (in a slightly different form)

[rhshadrach]

Thanks @Dr-Irv - it's an interesting point. But is there something that can be done here? For example, within DataFrame.groupby replacing

from pandas.core.groupby.generic import DataFrameGroupBy

with

from pandas.api.typing import DataFrameGroupBy

I still see

print(type(pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).groupby('a')))
# <class 'pandas.core.groupby.generic.DataFrameGroupBy'>

[Dr-Irv]

it's an interesting point. But is there something that can be done here?

I don't think we want to through the effort of changing the returned classes, but I think you can create a test like this:

gb = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).groupby('a')
assert isinstance(gb, pd.api.typing.DataFrameGroupBy)

That would at least test that you've correctly mapped the types in the typing module with the actual types.

[mroeschke]

Just commenting that in a follow up PR it might make sense for these objects to have their own API Reference page?

[rhshadrach]

Yea - agreed. I can open an issue to track.

[rhshadrach]

That would at least test that you've correctly mapped the types in the typing module with the actual types.

But there is no mapping here, yea? We import the object and then include said object in __all__. If there was a mapping, I would agree we should test it. As it stands, I think the only thing this would test is if there were two e.g. DataFrameGroupBy, that we've imported the right one. But if there were indeed two different DataFrameGroupBy objects, we'd have bigger problems I think 😆

I don't think such a test is warranted, and I'll also note we don't have tests like this for any existing pandas.api objects.

[Dr-Irv]

But there is no mapping here, yea? We import the object and then include said object in __all__. If there was a mapping, I would agree we should test it. As it stands, I think the only thing this would test is if there were two e.g. DataFrameGroupBy, that we've imported the right one. But if there were indeed two different DataFrameGroupBy objects, we'd have bigger problems I think 😆

I don't think such a test is warranted, and I'll also note we don't have tests like this for any existing pandas.api objects.

Maybe "mapping" is the wrong word.

I'm thinking of the following. Suppose that someone removed one of the types from pandas/api/typing/__init__.py or that you imported a class in that file, but forgot to include it in __all__. The test would capture these kind of errors.

Just because we don't have tests for the other pandas.api objects, doesn't mean we can't start now....

[mroeschke]

Generally LGTM

[rhshadrach]

I'm thinking of the following. Suppose that someone removed one of the types from pandas/api/typing/__init__.py or that you imported a class in that file, but forgot to include it in __all__. The test would capture these kind of errors.

Ah, I was under the false impression that this was being done. Agreed we should be testing this! I've added the ones for api.typing and plan to do the others in a followup (I've opened #52688 to track).

[Dr-Irv]

Thanks @rhshadrach . Let me know if I should do the merge, or if someone else should do it.

[mroeschke]

Thanks @rhshadrach