CI: dev requirements version inconsistencies

[attack68]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

..

Issue Description

Having difficulty replicating the ci checks locally so trying to install a new environment. My package resolver doesn't work. Trying to debug I have:

>>> pip-compile requirements-dev.in  (converted from .txt)
There are incompatible versions in the resolved dependencies:
  aiobotocore<2.0.0 (from -r requirements-dev2.in (line 77))
  aiobotocore~=2.0.1 (from s3fs==2021.11.1->-r requirements-dev2.in (line 76))

OK so I lower my version of s3fs to s3fs<2021.11.1

>>> pip-compile requirements-dev.in  (converted from .txt)
There are incompatible versions in the resolved dependencies:
  botocore>=1.11 (from -r requirements-dev2.in (line 40))
  botocore<1.24.0,>=1.23.27 (from boto3==1.20.27->-r requirements-dev2.in (line 39))
  botocore<1.20.107,>=1.20.106 (from aiobotocore==1.4.2->-r requirements-dev2.in (line 77))
  botocore>=1.12.201 (from moto==2.3.0->-r requirements-dev2.in (line 42))

OK so now I lower boto3 to boto3<1.17.1:

>>> pip-compile ...
There are incompatible versions in the resolved dependencies:
  fsspec>=0.7.4 (from -r requirements-dev2.in (line 78))
  fsspec (from fastparquet==0.7.2->-r requirements-dev2.in (line 72))
  fsspec>=0.6.0 (from dask==2021.12.0->-r requirements-dev2.in (line 31))
  fsspec==2021.11.0 (from s3fs==2021.11.0->-r requirements-dev2.in (line 76))
  fsspec==2021.11.1 (from gcsfs==2021.11.1->-r requirements-dev2.in (line 79))

And finally I lower gcsfs to gcsfs==2021.11.0.

And it resolves!

I wonder if it is worth using a package compiler to avoid these issues now that many packages are in the development requirements and haven't been specifically pinned? Then at least everyone can be on the same environment?

I also have a suspicion that numpy==1.20.3 is not sufficient to run the mypy typing checks since I get a npt error. With 1.21 installed I don't get this error.

#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile requirements-dev2.in
#
aiobotocore==1.4.2
    # via
    #   -r requirements-dev2.in
    #   s3fs
aiohttp==3.8.1
    # via
    #   aiobotocore
    #   gcsfs
    #   s3fs
aioitertools==0.8.0
    # via aiobotocore
aiosignal==1.2.0
    # via aiohttp
alabaster==0.7.12
    # via sphinx
appdirs==1.4.4
    # via black
appnope==0.1.2
    # via
    #   ipykernel
    #   ipython
argon2-cffi==21.3.0
    # via notebook
argon2-cffi-bindings==21.2.0
    # via argon2-cffi
asv==0.4.2
    # via -r requirements-dev2.in
async-timeout==4.0.2
    # via aiohttp
attrs==21.4.0
    # via
    #   aiohttp
    #   flake8-bugbear
    #   hypothesis
    #   jsonschema
    #   pytest
babel==2.9.1
    # via sphinx
backcall==0.2.0
    # via ipython
beautifulsoup4==4.10.0
    # via -r requirements-dev2.in
black==21.5b2
    # via -r requirements-dev2.in
bleach==4.1.0
    # via nbconvert
blosc==1.10.6
    # via -r requirements-dev2.in
boto3==1.17.0
    # via
    #   -r requirements-dev2.in
    #   moto
botocore==1.20.106
    # via
    #   -r requirements-dev2.in
    #   aiobotocore
    #   boto3
    #   moto
    #   s3transfer
bottleneck==1.3.2
    # via -r requirements-dev2.in
cachetools==4.2.4
    # via google-auth
certifi==2021.10.8
    # via requests
cffi==1.15.0
    # via
    #   argon2-cffi-bindings
    #   cryptography
cfgv==3.3.1
    # via pre-commit
cftime==1.5.1.1
    # via -r requirements-dev2.in
charset-normalizer==2.0.9
    # via
    #   aiohttp
    #   requests
click==8.0.3
    # via
    #   black
    #   flask
cloudpickle==2.0.0
    # via
    #   -r requirements-dev2.in
    #   dask
coverage[toml]==6.2
    # via pytest-cov
cpplint==1.5.5
    # via -r requirements-dev2.in
cramjam==2.5.0
    # via fastparquet
cryptography==36.0.1
    # via moto
cycler==0.11.0
    # via matplotlib
cython==0.29.26
    # via -r requirements-dev2.in
dask==2021.12.0
    # via -r requirements-dev2.in
debugpy==1.5.1
    # via ipykernel
decorator==5.1.0
    # via
    #   gcsfs
    #   ipython
defusedxml==0.7.1
    # via
    #   nbconvert
    #   odfpy
distlib==0.3.4
    # via virtualenv
docutils==0.17.1
    # via
    #   nbsphinx
    #   sphinx
    #   sphinx-panels
entrypoints==0.3
    # via
    #   jupyter-client
    #   nbconvert
et-xmlfile==1.1.0
    # via openpyxl
execnet==1.9.0
    # via pytest-xdist
fastparquet==0.7.2
    # via -r requirements-dev2.in
feedparser==6.0.8
    # via -r requirements-dev2.in
filelock==3.4.2
    # via virtualenv
flake8==4.0.1
    # via
    #   -r requirements-dev2.in
    #   flake8-bugbear
    #   flake8-comprehensions
    #   pandas-dev-flaker
flake8-bugbear==21.3.2
    # via -r requirements-dev2.in
flake8-comprehensions==3.7.0
    # via -r requirements-dev2.in
flask==2.0.2
    # via -r requirements-dev2.in
fonttools==4.28.5
    # via matplotlib
frozenlist==1.2.0
    # via
    #   aiohttp
    #   aiosignal
fsspec==2021.11.0
    # via
    #   -r requirements-dev2.in
    #   dask
    #   fastparquet
    #   gcsfs
    #   s3fs
gcsfs==2021.11.0
    # via -r requirements-dev2.in
gitdb==4.0.9
    # via
    #   -r requirements-dev2.in
    #   gitpython
gitpython==3.1.24
    # via -r requirements-dev2.in
google-api-core==2.3.2
    # via
    #   google-cloud-core
    #   google-cloud-storage
google-auth==2.3.3
    # via
    #   gcsfs
    #   google-api-core
    #   google-auth-oauthlib
    #   google-cloud-core
    #   google-cloud-storage
google-auth-oauthlib==0.4.6
    # via gcsfs
google-cloud-core==2.2.1
    # via google-cloud-storage
google-cloud-storage==1.43.0
    # via gcsfs
google-crc32c==1.3.0
    # via google-resumable-media
google-resumable-media==2.1.0
    # via google-cloud-storage
googleapis-common-protos==1.54.0
    # via google-api-core
greenlet==1.1.2
    # via sqlalchemy
html5lib==1.1
    # via -r requirements-dev2.in
hypothesis==6.34.1
    # via -r requirements-dev2.in
identify==2.4.1
    # via pre-commit
idna==3.3
    # via
    #   requests
    #   yarl
imagesize==1.3.0
    # via sphinx
importlib-metadata==4.10.0
    # via markdown
iniconfig==1.1.1
    # via pytest
ipykernel==6.6.0
    # via
    #   -r requirements-dev2.in
    #   ipywidgets
    #   notebook
ipython==7.30.1
    # via
    #   -r requirements-dev2.in
    #   ipykernel
    #   ipywidgets
ipython-genutils==0.2.0
    # via
    #   ipywidgets
    #   nbformat
    #   notebook
ipywidgets==7.6.5
    # via -r requirements-dev2.in
isort==5.10.1
    # via -r requirements-dev2.in
itsdangerous==2.0.1
    # via flask
jedi==0.18.1
    # via ipython
jinja2==3.0.3
    # via
    #   -r requirements-dev2.in
    #   flask
    #   moto
    #   nbconvert
    #   nbsphinx
    #   notebook
    #   numpydoc
    #   sphinx
jmespath==0.10.0
    # via
    #   boto3
    #   botocore
jsonschema==4.3.3
    # via nbformat
jupyter-client==7.1.0
    # via
    #   ipykernel
    #   nbclient
    #   notebook
jupyter-core==4.9.1
    # via
    #   jupyter-client
    #   nbconvert
    #   nbformat
    #   notebook
jupyterlab-pygments==0.1.2
    # via nbconvert
jupyterlab-widgets==1.0.2
    # via ipywidgets
kiwisolver==1.3.2
    # via matplotlib
llvmlite==0.37.0
    # via numba
locket==0.2.1
    # via partd
lxml==4.7.1
    # via -r requirements-dev2.in
markdown==3.3.6
    # via -r requirements-dev2.in
markupsafe==2.0.1
    # via
    #   jinja2
    #   moto
matplotlib==3.5.1
    # via
    #   -r requirements-dev2.in
    #   seaborn
matplotlib-inline==0.1.3
    # via
    #   ipykernel
    #   ipython
mccabe==0.6.1
    # via flake8
mistune==0.8.4
    # via nbconvert
moto==2.3.0
    # via -r requirements-dev2.in
multidict==5.2.0
    # via
    #   aiohttp
    #   yarl
mypy==0.930
    # via -r requirements-dev2.in
mypy-extensions==0.4.3
    # via
    #   black
    #   mypy
natsort==8.0.2
    # via -r requirements-dev2.in
nbclient==0.5.9
    # via nbconvert
nbconvert==6.4.0
    # via
    #   -r requirements-dev2.in
    #   nbsphinx
    #   notebook
nbformat==5.1.3
    # via
    #   -r requirements-dev2.in
    #   ipywidgets
    #   nbclient
    #   nbconvert
    #   nbsphinx
    #   notebook
nbsphinx==0.8.8
    # via -r requirements-dev2.in
nest-asyncio==1.5.4
    # via
    #   jupyter-client
    #   nbclient
    #   notebook
nodeenv==1.6.0
    # via pre-commit
notebook==6.4.6
    # via
    #   -r requirements-dev2.in
    #   widgetsnbextension
numba==0.54.1
    # via -r requirements-dev2.in
numexpr==2.8.1
    # via
    #   -r requirements-dev2.in
    #   tables
numpy==1.20.3
    # via
    #   -r requirements-dev2.in
    #   bottleneck
    #   cftime
    #   fastparquet
    #   matplotlib
    #   numba
    #   numexpr
    #   pandas
    #   patsy
    #   pyarrow
    #   scipy
    #   seaborn
    #   statsmodels
    #   tables
    #   xarray
numpydoc==1.1.0
    # via -r requirements-dev2.in
oauthlib==3.1.1
    # via requests-oauthlib
odfpy==1.4.1
    # via -r requirements-dev2.in
openpyxl==3.0.9
    # via -r requirements-dev2.in
packaging==21.3
    # via
    #   bleach
    #   dask
    #   matplotlib
    #   numexpr
    #   pytest
    #   sphinx
    #   tables
pandas==1.3.5
    # via
    #   fastparquet
    #   pyreadstat
    #   seaborn
    #   statsmodels
    #   xarray
pandas-dev-flaker==0.2.0
    # via -r requirements-dev2.in
pandoc==2.0.1
    # via -r requirements-dev2.in
pandocfilters==1.5.0
    # via nbconvert
parso==0.8.3
    # via jedi
partd==1.2.0
    # via
    #   -r requirements-dev2.in
    #   dask
pathspec==0.9.0
    # via black
patsy==0.5.2
    # via statsmodels
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
pillow==9.0.0
    # via matplotlib
platformdirs==2.4.1
    # via virtualenv
pluggy==1.0.0
    # via pytest
plumbum==1.7.2
    # via pandoc
ply==3.11
    # via pandoc
pre-commit==2.16.0
    # via -r requirements-dev2.in
prometheus-client==0.12.0
    # via notebook
prompt-toolkit==3.0.24
    # via ipython
protobuf==3.19.1
    # via
    #   google-api-core
    #   google-cloud-storage
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
py==1.11.0
    # via
    #   pytest
    #   pytest-forked
pyarrow==6.0.1
    # via -r requirements-dev2.in
pyasn1==0.4.8
    # via
    #   pyasn1-modules
    #   rsa
pyasn1-modules==0.2.8
    # via google-auth
pycodestyle==2.8.0
    # via
    #   -r requirements-dev2.in
    #   flake8
pycparser==2.21
    # via cffi
pyflakes==2.4.0
    # via flake8
pygments==2.11.1
    # via
    #   ipython
    #   jupyterlab-pygments
    #   nbconvert
    #   sphinx
pyparsing==3.0.6
    # via
    #   matplotlib
    #   packaging
pyreadstat==1.1.4
    # via -r requirements-dev2.in
pyrsistent==0.18.0
    # via jsonschema
pytest==6.2.5
    # via
    #   -r requirements-dev2.in
    #   pytest-asyncio
    #   pytest-cov
    #   pytest-cython
    #   pytest-forked
    #   pytest-instafail
    #   pytest-xdist
pytest-asyncio==0.16.0
    # via -r requirements-dev2.in
pytest-cov==3.0.0
    # via -r requirements-dev2.in
pytest-cython==0.1.1.post0
    # via -r requirements-dev2.in
pytest-forked==1.4.0
    # via pytest-xdist
pytest-instafail==0.4.2
    # via -r requirements-dev2.in
pytest-xdist==2.5.0
    # via -r requirements-dev2.in
python-dateutil==2.8.2
    # via
    #   -r requirements-dev2.in
    #   botocore
    #   jupyter-client
    #   matplotlib
    #   moto
    #   pandas
python-snappy==0.6.0
    # via -r requirements-dev2.in
pytz==2021.3
    # via
    #   -r requirements-dev2.in
    #   babel
    #   moto
    #   pandas
pyupgrade==2.31.0
    # via -r requirements-dev2.in
pyyaml==6.0
    # via
    #   -r requirements-dev2.in
    #   dask
    #   pre-commit
pyzmq==22.3.0
    # via
    #   jupyter-client
    #   notebook
regex==2021.11.10
    # via black
requests==2.27.0
    # via
    #   -r requirements-dev2.in
    #   gcsfs
    #   google-api-core
    #   google-cloud-storage
    #   moto
    #   requests-oauthlib
    #   responses
    #   sphinx
requests-oauthlib==1.3.0
    # via google-auth-oauthlib
responses==0.16.0
    # via moto
rsa==4.8
    # via google-auth
s3fs==2021.11.0
    # via -r requirements-dev2.in
s3transfer==0.3.7
    # via boto3
scipy==1.7.3
    # via
    #   -r requirements-dev2.in
    #   seaborn
    #   statsmodels
seaborn==0.11.2
    # via -r requirements-dev2.in
send2trash==1.8.0
    # via notebook
sgmllib3k==1.0.0
    # via feedparser
six==1.16.0
    # via
    #   asv
    #   bleach
    #   google-auth
    #   google-cloud-storage
    #   html5lib
    #   patsy
    #   python-dateutil
    #   responses
    #   thrift
    #   virtualenv
smmap==5.0.0
    # via gitdb
snowballstemmer==2.2.0
    # via sphinx
sortedcontainers==2.4.0
    # via hypothesis
soupsieve==2.3.1
    # via beautifulsoup4
sphinx==4.3.2
    # via
    #   -r requirements-dev2.in
    #   nbsphinx
    #   numpydoc
    #   sphinx-panels
sphinx-panels==0.6.0
    # via -r requirements-dev2.in
sphinxcontrib-applehelp==1.0.2
    # via sphinx
sphinxcontrib-devhelp==1.0.2
    # via sphinx
sphinxcontrib-htmlhelp==2.0.0
    # via sphinx
sphinxcontrib-jsmath==1.0.1
    # via sphinx
sphinxcontrib-qthelp==1.0.3
    # via sphinx
sphinxcontrib-serializinghtml==1.1.5
    # via sphinx
sqlalchemy==1.4.29
    # via -r requirements-dev2.in
statsmodels==0.13.1
    # via -r requirements-dev2.in
tables==3.7.0
    # via -r requirements-dev2.in
tabulate==0.8.9
    # via -r requirements-dev2.in
terminado==0.12.1
    # via notebook
testpath==0.5.0
    # via nbconvert
thrift==0.15.0
    # via fastparquet
tokenize-rt==4.2.1
    # via pyupgrade
toml==0.10.2
    # via
    #   black
    #   pre-commit
    #   pytest
tomli==2.0.0
    # via
    #   coverage
    #   mypy
toolz==0.11.2
    # via
    #   -r requirements-dev2.in
    #   dask
    #   partd
tornado==6.1
    # via
    #   ipykernel
    #   jupyter-client
    #   notebook
    #   terminado
traitlets==5.1.1
    # via
    #   ipykernel
    #   ipython
    #   ipywidgets
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
    #   nbsphinx
    #   notebook
types-pymysql==1.0.8
    # via -r requirements-dev2.in
types-python-dateutil==2.8.4
    # via -r requirements-dev2.in
types-pytz==2021.3.3
    # via -r requirements-dev2.in
types-setuptools==57.4.5
    # via -r requirements-dev2.in
typing-extensions==4.0.1
    # via
    #   gitpython
    #   mypy
urllib3==1.26.7
    # via
    #   botocore
    #   requests
    #   responses
virtualenv==20.13.0
    # via pre-commit
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via
    #   bleach
    #   html5lib
werkzeug==2.0.2
    # via
    #   flask
    #   moto
widgetsnbextension==3.5.2
    # via ipywidgets
wrapt==1.13.3
    # via aiobotocore
xarray==0.18.2
    # via -r requirements-dev2.in
xlrd==2.0.1
    # via -r requirements-dev2.in
xlsxwriter==3.0.2
    # via -r requirements-dev2.in
xlwt==1.3.0
    # via -r requirements-dev2.in
xmltodict==0.12.0
    # via moto
yarl==1.7.2
    # via aiohttp
zipp==3.7.0
    # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools

Expected Behavior

packages are correctly resolved and valid.

Installed Versions

Replace this line with the output of pd.show_versions()

[mroeschke]

I wonder if it is worth using a package compiler to avoid these issues now that many packages are in the development requirements and haven't been specifically pinned? Then at least everyone can be on the same environment?

Would be good to start using conda lock files for local development (and maybe even CI builds) to enforce more reproducibility. Additionally having a GHA workflow to validating conda lock files are up to date & valid would be good.

https://github.com/conda-incubator/conda-lock

[twoertwein]

When parsing requirements-dev.txt with poetry it fails with this "nicely" generated sentence.:

And because boto3 (1.20.27) depends on botocore (>=1.23.27,<1.24.0)
and no versions of boto3 match >1.20.27,<2.0.0, aiobotocore (<2.0.0) is incompatible with boto3 (>=1.20.27,<2.0.0).
So, because pandas depends on both boto3 (^1.20.27) and aiobotocore (<2.0.0), version solving failed.

After adding

[tool.poetry]
name = "pandas"
description = "pandas"
authors = ["authors"]
version = "1.3.5"
license = "BSD-3-Clause"

[tool.poetry.dependencies]
python = "^3.8,<3.10"

to pyproject.toml and running

poetry add $(grep -v "#" requirements-dev.txt)

[attack68]

I think the biggest inconsistency here is actually that the most recent release of numba==0.54.1 requires numpy<1.21.0, but the mypy checks reveal many typing errors if not run with numpy>=1.21.p

[twoertwein]

This tracker for numba 0.55rc1 suggests that 0.55 will work with numpy==1.21

edit: but I guess we will soon need numpy>=1.22.

[iasoon]

On current master we do indeed need numpy>=1.22 (I think == 1.22.3) to pass the mypy check, which is currently incompatible with the latest version of numba (0.55.1).