CI/TYP: run stubtest

[twoertwein]

Stubtest enforces that pyi files do not get out of sync with the implementation.

stubtest and mypy seems to have a slight discrepancy (had to change annotations so that they both error in the same cases).

stubtest needs pandas dev to be installed. This might not be something people want to do locally: if pandas dev is not installed: scripts/run_stubtest.py will 'succeed' but print a warning (the CI has pandas dev installed).

Might want to wait for the CI using mypy 0.971 (#47806 waiting for conda).

@hauntsaninja: The allow list acts more or less like ignore comments, is there an option to error on unused allow list entries?

Closes #47760

[hauntsaninja]

Awesome, this looks pretty good to me.

I took a second look at the output of python -m mypy.stubtest pandas --mypy-config-file pyproject.toml --concise (i.e. don't just run on the pyi files), since ideally that should work. It looks like there are three problems:
a) First, the mypy build doesn't work out of the box. I'll look into why this differs from regular mypy invocation. But once you fix that:
b) There are a lot of overloads where things are typed as keyword only because of deprecate_nonkeyword_arguments, but aren't "actually" keyword only. I assume at some point the deprecations will turn into removals.
c) A lot of repeated errors from the way stubtest currently handles aliases. This will be improved in the next release of mypy.

[twoertwein]

I took a second look at the output of python -m mypy.stubtest pandas --mypy-config-file pyproject.toml --concise (i.e. don't just run on the pyi files)

What would be the added benefit of doing this? Shouldn't mypy find all those issues already?

b) There are a lot of overloads where things are typed as keyword only because of deprecate_nonkeyword_arguments, but aren't "actually" keyword only. I assume at some point the deprecations will turn into removals.

The idea is that the type annotation is a kind of an added "deprecation" warning when people use a type checker while using deprecated behavior (and probably more importantly it let's us move a bit more quickly with the annotation process). I expect that the implementation will be keyword-only after the 1.5 release (running subtest on all files will be helpful to make all of those changes).

[hauntsaninja]

What would be the added benefit of doing this? Shouldn't mypy find all those issues already?

Yes, sorry. My post was more notes to myself than anything immediately helpful for this PR. I was just thinking through things that would make stubtest more user friendly, so fewer people have to write wrapper scripts around it :-)

[twoertwein]

I assume this PR doesn't need to wait for mypy 0.971 (seems to already work)

[twoertwein]

I was just thinking through things that would make stubtest more user friendly, so fewer people have to write wrapper scripts around it :-)

Two suggestions:

• if stubtest is not meant as a mypy++ (does what mypy does and more), it might be cool to ignore any errors in function bodies as stubtest will not check function internals (might even speed stubtest up).
• if stubtest is meant as mypy++, it might be nice to document that running mypy in addition to stubtest is redundant (assuming stubtest is run on all files not just on files that have a pyi file)

[mroeschke]

LGTM just a merge conflict

[twoertwein]

Could go before or after #47806. In either case, the other PR should be rebased before merging.

[hauntsaninja]

stubtest is not intended as a "mypy++". Made this clearer in the documentation here: python/mypy#13293

Seeing if we can skip some work / ignore errors that block the actual stubtest logic from running is a good suggestion, I will see what improvements can be made there :-)

[mroeschke]

Thanks @twoertwein