Skip to content

DEV speed up pip install -r requirements-dev.txt by nearly 2000% #50295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 16, 2022
Merged

DEV speed up pip install -r requirements-dev.txt by nearly 2000% #50295

merged 1 commit into from
Dec 16, 2022

Conversation

MarcoGorelli
Copy link
Member

@MarcoGorelli MarcoGorelli commented Dec 16, 2022 •
edited
Loading

it looks like it may not be necessary to list boto3 and botocore, and they get pulled in anyway by s3fs and moto

EDIT:

yup, this works! and we can stop listing even more unpinned dependencies!

  • botocore and boto3 are pulled in by s3fs
  • beautifulsoup4 is pulled in by nbsphinx
  • coverage is pulled in by pytest-cov
  • fsspec is pulled in by s3fs
  • gitdb is pulled in by GitPython
  • ipykernel is pulled in by ipywidgets
  • ipython is pulled in by ipywidgets
  • jinja2 is pulled in by moto
  • nbformat is pulled in by nbsphinx
  • psutil is pulled in by ipywidgets
  • pyyaml is pulled in by dask
  • requests is pulled in by gcsfs
  • sphinx is pulled in by nbsphinx

By removing them, pip install -r requirements-dev.txt goes from taking over 54 minutes, to just under 3 minutes!

EDIT2: only removing boto3 and botocore. Timing is still under 3 minutes 🥳

@MarcoGorelli MarcoGorelli changed the title wip WIP try speeding up pip install -r requirements-dev.txt Dec 16, 2022
@MarcoGorelli MarcoGorelli marked this pull request as ready for review December 16, 2022 13:55
@MarcoGorelli MarcoGorelli changed the title WIP try speeding up pip install -r requirements-dev.txt DEV speed up pip install -r requirements-dev.txt by 2100% Dec 16, 2022
@MarcoGorelli MarcoGorelli changed the title DEV speed up pip install -r requirements-dev.txt by 2100% DEV speed up pip install -r requirements-dev.txt by nearly 2000% Dec 16, 2022
@MarcoGorelli MarcoGorelli added the Dependencies Required and optional dependencies label Dec 16, 2022
@jorisvandenbossche
Copy link
Member

I am slightly -1 on this. I think it is clearer that any package that we use directly is listed in the requirements (which eg was not the case for aiobotocore). This is more explicit and also more robust (in case some other package changes its dependencies).

It is pip's "fault" that it cannot handle this properly. Of course, given that this makes the pip install so unusable, we can be pragmatic and help that on our side. But then I would personally limit it to removing the biggest culprits (eg just removing boto3/botocore already resolves most of it?), and keep being explicit for the others.

@MarcoGorelli
Copy link
Member Author

But then I would personally limit it to removing the biggest culprits (eg just removing boto3/botocore already resolves most of it?)

OK, sure, as long as we can do that because pip installation is currently unusable

@mroeschke
Copy link
Member

Agree with @jorisvandenbossche's reasoning. Would be good to keep an issue open regarding adding back boto3 and botocore if pip can more quickly resolve this in the future. Maybe also adding lower pins on all our dependencies would help the solver?

@MarcoGorelli MarcoGorelli mentioned this pull request Dec 16, 2022
Closed
@MarcoGorelli
Copy link
Member Author

Agree with @jorisvandenbossche's reasoning. Would be good to keep an issue open regarding adding back boto3 and botocore if pip can more quickly resolve this in the future. Maybe also adding lower pins on all our dependencies would help the solver?

sure, done #50300

@MarcoGorelli MarcoGorelli merged commit 0986922 into pandas-dev:main Dec 16, 2022
@MarcoGorelli MarcoGorelli added this to the 2.0 milestone Dec 16, 2022
phofl pushed a commit to phofl/pandas that referenced this pull request Dec 17, 2022
@MarcoGorelli @phofl
DEV speed up pip install -r requirements-dev.txt by nearly 2000% (p…
2e29028 
…andas-dev#50295)

dont explicitly list boto3 and botocore

Co-authored-by: MarcoGorelli <>
@phofl
Copy link
Member

phofl commented Dec 19, 2022

Can we backport? Otherwise the docker change might take forever

@anmyachev anmyachev mentioned this pull request Aug 22, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mroeschke mroeschke mroeschke approved these changes

@rhshadrach rhshadrach Awaiting requested review from rhshadrach

@jorisvandenbossche jorisvandenbossche Awaiting requested review from jorisvandenbossche

Assignees
No one assigned
Labels
Dependencies Required and optional dependencies
Projects
None yet
Milestone
2.0
Development

Successfully merging this pull request may close these issues.

DEV pip install -r requirements-dev.txt takes...54 minutes?
4 participants
@MarcoGorelli @jorisvandenbossche @mroeschke @phofl