TYP: use __all__ to signal public API to type checkers

[twoertwein]

• closes TYP: Redundant imports to define the public API #43664
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[jreback]

cc @simonjayhawkins

[simonjayhawkins]

LGTM ex @jreback comment. This is a precursor to #43672?

[twoertwein]

This is a precursor to #43672?

No, but once we add a py.typed file we could use pyright to make sure that a list of intended to be public functions/attributes are picked up as public by type checkers.

[twoertwein]

Is there a reason why api, testing, and arrays was not imported with "from pandas import"? I changed it to from pandas import to include them in __all__.

[jreback]

these should be equivalent (ex the name space visiblity).

note that we may not be fully testing this (see in pandas/tests/api/test_api.py)

[twoertwein]

I added a test for __all__ and found in that process a few un-exported modules.

[twoertwein]

assert actual == expected would work, but it doesn't create a nice log in pytest.

[twoertwein]

I think the modules I added in the last commit are not meant to be public (compat, core, io, ...)? @simonjayhawkins

If the above modules are not meant to be public, there probably needs to be a discussion which additional classes should be exported in pandas/__init__.py. I think it would currently not be possible for a user to do

def mean(grouper: pd.core.groupby.DataFrameGroupBy) -> pd.DataFrame:
    return group.mean()

without using "private" APIs (pd.core.groupby.DataFrameGroupBy).

[jreback]

none of those are documented as public

we have an issue about deprecating and making these private but hasn't gotten done

don't add them explicitly to any exports

[jreback]

some of these apis should be exported in pd.api as well (eg DataframeGroupby for example)

[twoertwein]

So the only public "top-level sub-packages" are: "api", "arrays", "options", "test", "testing"? They are all imported in pandas/__init__.py.

[twoertwein]

Reading through the documentation, the following should be listed as public in pandas/__init__.py:
pandas.errors: just defines warnings/errors
pandas.plotting: has itself __all__ to define what is public
pandas/testing.py: has itself __all__ to define what is public
pandas/io: contains some public classes but is missing an __all__
pandas/tseries: contains some public classes but is missing an __all__

[twoertwein]

I was trying to expose only meant-to-be-public sub-modules (public = listed in the documentation) in pandas/io/__init__.py and pandas/tseries/__init__.py by importing them and adding them to __all__ (and then repeating this for their sub-modules that do not yet have an __all__). Unfortunately, this leads to circular imports :(

Here is an example how I would pandas/io/__init__.py to look like

# import modules that have public classes/functions
from pandas.io import (
    formats,
    json,
    stata,
)

# and mark only those modules as public
__all__ = ["formats", "json", "stata"]

I think many people assume that everything that doesn't start with an underscore is public. __all__ could be a gentle nudge to remind people what is public. For example, after renaming get_filepath_buffer to _get_filepath_buffer in #37639 it is clear that at least two projects (PEtab-dev/PEtab#493 and fphammerle/freesurfer-stats#15) assumed it to be public even though it is not listed in the documentation.

[jreback]

can you rebase

[jreback]

https://github.com/pandas-dev/pandas/pull/43695/checks?check_run_id=3734385197

hmm maybe need to be more explict in the docs

[twoertwein]

https://github.com/pandas-dev/pandas/pull/43695/checks?check_run_id=3734385197

from pandas import * causes the deprecated models to be imported as they are listed in __all__.

I think there are two options:

1. don't use from pandas import *
2. do not add the deprecated modules to __all__

Which one is preferred?

[jreback]

i don't think we should add the deprecated modules to all

[jreback]

thanks @twoertwein very nice!