CI: run pyright with mypy

[twoertwein]

The github action running the pre-commit checks does not install all necessary python packages for type checking. This triggered a few unnecessary reportOverlappingOverload by pyright because numpy was not installed.

Keep pyright in the pre-commit but disable it on the CI (SKIP=pyright). Instead, run pyright together with mypy in code_checks.sh.

[jreback]

thanks @twoertwein

[attack68]

@twoertwein

I have merged master and done the following:

$ pip install -r requirement-dev.txt
$ python setup.py build_ext -j 4
$ pre-commit install
$ pre-commit autoupdate
$ pre-commit run --all-files pyright

I get the following error:

Failed
- hook id: pyright
- exit code: 1

No configuration file found.
pyproject.toml file found at /Users/Darbyshire/pandas.
Loading pyproject.toml file at /Users/Darbyshire/pandas/pyproject.toml
Assuming Python platform Darwin
stubPath /Users/Darbyshire/pandas/typings is not a valid directory.
Searching for source files
Found 308 source files
/Users/Darbyshire/pandas/pandas/core/base.py
  /Users/Darbyshire/pandas/pandas/core/base.py:1251:9 - error: Overload 2 for "searchsorted" will never be used because its parameters overlap overload 1 (reportOverlappingOverload)

Am I missing something.. not familiar with pyright.

[twoertwein]

Which numpy version do you have installed? I think I got a similar error when I used an older version of numpy.

[twoertwein]

I assume that npt.ArrayLike evaluates to Any with your numpy, which then let's pyright (correctly) complain that the second overload will never be reached.

Do you mind adding a reveal_type(npt.ArrayLike) and then running pyright.

[twoertwein]

If you can confirm that upgrading numpy>1.21.0 fixes this issue, we should bump the numpy version in requirement-dev.txt

[twoertwein]

When downgrading numpy to 1.20.0 pyright cannot determine the type anymore:
info: Type of "npt._ScalarLike_co" is "Unknown"

I will create a PR to bump the version of numpy

[attack68]

I re installed my env, but have the same error with numpy 1.20.3. Trying to upgrade now..

[twoertwein]

I re installed my env, but have the same error with numpy 1.20.3. Trying to upgrade now..

Sorry about that, numpy>1.21.0 or locally adding reportOverlappingOverload = false to pyproject.toml should be workaround.

[attack68]

I upgraded to numpy 1.21.2, and incidentally got the error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
numba 0.54.0 requires numpy<1.21,>=1.17, but you have numpy 1.21.2 which is incompatible.

If running pre-commit run --all-files pyright then I get PASSED.

If using git hooks and committing from my IDE (pycharm) I get the following error (and I did add reportOverlappingOverload = false to pyproject.toml so I don't know why its still reporting it: weird anyway late here so I take another look tomorrow)

Failed - hook id: pyright - exit code: 1 No configuration file found. pyproject.toml file found at /Users/Darbyshire/pandas. 
Loading pyproject.toml file at /Users/Darbyshire/pandas/pyproject.toml Assuming Python platform Darwin stubPath 
/Users/Darbyshire/pandas/typings is not a valid directory. Searching for source files Found 308 source files 
/Users/Darbyshire/pandas/pandas/core/base.py /Users/Darbyshire/pandas/pandas/core/base.py:1251:9 - error: Overload 2 for
 "searchsorted" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/common.py /Users/Darbyshire/pandas/pandas/core/common.py:398:5 - error: 
Overload 2 for "random_state" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/frame.py /Users/Darbyshire/pandas/pandas/core/frame.py:1370:9 - error: Overload 1
 for "dot" overlaps overload 2 and returns an incompatible type (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/array_algos/take.py /Users/Darbyshire/pandas/pandas/core/array_algos/take.py:44:5 -
 error: Overload 2 for "take_nd" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/arrays/_mixins.py /Users/Darbyshire/pandas/pandas/core/arrays/_mixins.py:216:9 - 
error: Overload 2 for "__getitem__" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) 
/Users/Darbyshire/pandas/pandas/core/arrays/base.py /Users/Darbyshire/pandas/pandas/core/arrays/base.py:309:9 - error:
 Overload 2 for "__getitem__" will never be used because its parameters overlap overload 1 (reportOverlappingOverload)

with

Package                           Version
--------------------------------- -----------
aiobotocore                       1.4.1
aiohttp                           3.7.4.post0
aioitertools                      0.8.0
alabaster                         0.7.12
appdirs                           1.4.4
appnope                           0.1.2
argon2-cffi                       21.1.0
asv                               0.4.2
async-timeout                     3.0.1
attrs                             21.2.0
Babel                             2.9.1
backcall                          0.2.0
backports.entry-points-selectable 1.1.0
beautifulsoup4                    4.10.0
black                             21.5b2
bleach                            4.1.0
blosc                             1.10.4
boto3                             1.17.106
botocore                          1.20.106
Bottleneck                        1.3.2
cachetools                        4.2.3
certifi                           2021.5.30
cffi                              1.14.6
cfgv                              3.3.1
cftime                            1.5.1
chardet                           4.0.0
charset-normalizer                2.0.6
click                             8.0.1
cloudpickle                       2.0.0
coverage                          5.5
cpplint                           1.5.5
cramjam                           2.4.0
cryptography                      3.4.8
cycler                            0.10.0
Cython                            0.29.24
dask                              2021.9.1
debugpy                           1.4.3
decorator                         5.1.0
defusedxml                        0.7.1
distlib                           0.3.3
docutils                          0.17.1
entrypoints                       0.3
et-xmlfile                        1.1.0
execnet                           1.9.0
fastparquet                       0.7.1
feedparser                        6.0.8
filelock                          3.1.0
flake8                            3.9.2
flake8-bugbear                    21.3.2
flake8-comprehensions             3.1.0
Flask                             2.0.1
fsspec                            2021.5.0
gcsfs                             2021.5.0
gitdb                             4.0.7
GitPython                         3.1.24
google-auth                       2.2.1
google-auth-oauthlib              0.4.6
greenlet                          1.1.2
html5lib                          1.1
hypothesis                        6.23.1
identify                          2.2.15
idna                              3.2
imagesize                         1.2.0
iniconfig                         1.1.1
ipykernel                         6.4.1
ipython                           7.28.0
ipython-genutils                  0.2.0
ipywidgets                        7.6.5
isort                             5.9.3
itsdangerous                      2.0.1
jedi                              0.18.0
Jinja2                            3.0.1
jmespath                          0.10.0
jsonschema                        3.2.0
jupyter-client                    7.0.5
jupyter-core                      4.8.1
jupyterlab-pygments               0.1.2
jupyterlab-widgets                1.0.2
kiwisolver                        1.3.2
llvmlite                          0.37.0
locket                            0.2.1
lxml                              4.6.3
Markdown                          3.3.4
MarkupSafe                        2.0.1
matplotlib                        3.4.3
matplotlib-inline                 0.1.3
mccabe                            0.6.1
mistune                           0.8.4
more-itertools                    8.10.0
moto                              2.2.8
multidict                         5.1.0
mypy                              0.910
mypy-extensions                   0.4.3
natsort                           7.1.1
nbclient                          0.5.4
nbconvert                         6.2.0
nbformat                          5.1.3
nbsphinx                          0.8.7
nest-asyncio                      1.5.1
nodeenv                           1.6.0
notebook                          6.4.4
numba                             0.54.0
numexpr                           2.7.3
numpy                             1.21.2
numpydoc                          1.1.0
oauthlib                          3.1.1
odfpy                             1.4.1
openpyxl                          3.0.9
packaging                         21.0
pandas                            1.3.3
pandas-dev-flaker                 0.2.0
pandoc                            1.1.0
pandocfilters                     1.5.0
parso                             0.8.2
partd                             1.2.0
pathspec                          0.9.0
patsy                             0.5.2
pep517                            0.11.0
pexpect                           4.8.0
pickleshare                       0.7.5
Pillow                            8.3.2
pip                               21.2.4
pip-tools                         6.3.0
platformdirs                      2.4.0
pluggy                            1.0.0
plumbum                           1.7.0
ply                               3.11
pre-commit                        2.15.0
prometheus-client                 0.11.0
prompt-toolkit                    3.0.20
ptyprocess                        0.7.0
py                                1.10.0
pyarrow                           5.0.0
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycodestyle                       2.7.0
pycparser                         2.20
pydata-sphinx-theme               0.6.3
pyflakes                          2.3.1
Pygments                          2.10.0
pyparsing                         2.4.7
pyreadstat                        1.1.2
pyrsistent                        0.18.0
pytest                            6.2.5
pytest-asyncio                    0.15.1
pytest-cov                        2.12.1
pytest-forked                     1.3.0
pytest-instafail                  0.4.2
pytest-xdist                      2.4.0
python-dateutil                   2.8.2
python-snappy                     0.6.0
pytz                              2021.1
pyupgrade                         2.29.0
PyYAML                            5.4.1
pyzmq                             22.3.0
regex                             2021.9.24
requests                          2.26.0
requests-oauthlib                 1.3.0
responses                         0.14.0
rsa                               4.7.2
s3fs                              2021.5.0
s3transfer                        0.4.2
scipy                             1.7.1
seaborn                           0.11.2
Send2Trash                        1.8.0
setuptools                        57.4.0
sgmllib3k                         1.0.0
six                               1.16.0
smmap                             4.0.0
snowballstemmer                   2.1.0
sortedcontainers                  2.4.0
soupsieve                         2.2.1
Sphinx                            4.2.0
sphinx-panels                     0.6.0
sphinxcontrib-applehelp           1.0.2
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
SQLAlchemy                        1.4.25
statsmodels                       0.12.2
tables                            3.6.1
tabulate                          0.8.9
terminado                         0.12.1
testpath                          0.5.0
thrift                            0.15.0
tokenize-rt                       4.1.0
toml                              0.10.2
tomli                             1.2.1
toolz                             0.11.1
tornado                           6.1
traitlets                         5.1.0
types-PyMySQL                     1.0.2
types-python-dateutil             2.8.0
types-pytz                        2021.1.2
types-setuptools                  57.4.0
typing-extensions                 3.10.0.2
urllib3                           1.26.7
virtualenv                        20.8.1
wcwidth                           0.2.5
webencodings                      0.5.1
Werkzeug                          2.0.1
wheel                             0.37.0
widgetsnbextension                3.5.1
wrapt                             1.12.1
xarray                            0.18.2
xlrd                              2.0.1
XlsxWriter                        3.0.1
xlwt                              1.3.0
xmltodict                         0.12.0
yarl                              1.6.3

[twoertwein]

I'm sorry for this frustrating experience.

Based on your output pyright reads the pyproejct.toml file so it should ignore reportOverlappingOverload errors if you have reportOverlappingOverload = false in the pyright section.

You could run pyright only during the manual stage by adding stages: [manual] to the pyright hook in .pre-commit-config.yaml

[attack68]

I'm sorry for this frustrating experience.

Don't worry about it. I'm learning and these things happen. If I set stages to manual as described then I get the following issue (I think because I am not including the changes to the pre-commit file in the final commit) (but I see you have a PR that will do this anyway)

07:23	Commit failed with error
			0 file committed, 1 file failed to commit: add version added
			[ERROR] Your pre-commit configuration is unstaged.
			`git add .pre-commit-config.yaml` to fix this.

[simonjayhawkins]

numba 0.54.0 requires numpy<1.21,>=1.17, but you have numpy 1.21.2 which is incompatible.

this is strange since numba 0.53.1 is ok with numpy 1.21.2 (with conda)