Skip to content

CI: Bump timeouts Python dev builds #48973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Oct 10, 2022
Merged

CI: Bump timeouts Python dev builds #48973

merged 6 commits into from
Oct 10, 2022

Conversation

mroeschke
Copy link
Member

@mroeschke mroeschke marked this pull request as draft October 6, 2022 17:15
@mroeschke mroeschke force-pushed the debug/ci/dev_builds branch from 390abd5 to 504a480 Compare October 6, 2022 19:03
@jbrockmendel
Copy link
Member

I just ran the tests (--skip-slow --skip-db) from April 1 and they took 19:30, compared to 30:06 from yesterday. Of that half hour, 8 minutes is in tests/extension/test_arrow.py. Not sure what to do with this, just making sure its written down.

@mroeschke
Copy link
Member Author

(FWIW: The Python dev builds don't run the optional deps)

Each test in extensions/test_arrow.py is essentially parameterized over 37 pyarrow dtypes (plus want to test list and struct support in the future to defer nested data support to pyarrow). There may be areas where skips per dtypes is possible, but otherwise the tests are very thorough (and long) because of this.

@lithomas1
Copy link
Member

I think the main issue is the build step. Building pandas shouldn't take 25 minutes like it does on the macOS job.

@mroeschke mroeschke changed the title DEBUG: CI timeouts Python dev builds CI: Bump timeouts Python dev builds Oct 7, 2022
@mroeschke mroeschke added the CI Continuous Integration label Oct 7, 2022
@mroeschke
Copy link
Member Author

At least this gives us a buffer (other builds are at 120)

@lithomas1 any quick suggestions how we might speed things up or should that be addressed in a followup?

@lithomas1
Copy link
Member

Try changing this line https://github.com/pandas-dev/pandas/pull/48973/files#diff-f6f33d5fc5cfd739f0d6efd63ea9dff886abe35cdfff5672e9a863ec8da8e72cR78 to
python -m pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy

Numpy should be uploading Python 3.11 wheels so, we'll shave off ~5 minutes for sure from not compiling numpy. We might get a little more speed-up in the tests too from this.

@mroeschke mroeschke marked this pull request as ready for review October 7, 2022 21:18
@lithomas1 lithomas1 added this to the 1.5.1 milestone Oct 7, 2022
jarrodmillman
jarrodmillman reviewed Oct 8, 2022
View reviewed changes
.github/workflows/python-dev.yml
@@ -75,7 +75,7 @@ jobs:
run: |
python --version
python -m pip install --upgrade pip setuptools wheel
python -m pip install git+https://github.com/numpy/numpy.git
python -m pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
python -m pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
python -m pip install numpy==1.23.3

They have wheels for mac, windows, and linux:
https://pypi.org/project/numpy/1.23.3/#files

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, it might be worth it to stay with numpy main, until Python 3.11 is out, just in case compat issues arise between numpy and Python 3.11.

lithomas1
lithomas1 approved these changes Oct 10, 2022
View reviewed changes
Copy link
Member

@lithomas1 lithomas1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you open a follow up issue so we can make a plan to get these timeouts back down?

@mroeschke mroeschke merged commit 5f0ac04 into pandas-dev:main Oct 10, 2022
@mroeschke mroeschke deleted the debug/ci/dev_builds branch October 10, 2022 16:10
@meeseeksmachine meeseeksmachine mentioned this pull request Oct 10, 2022
Merged
meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Oct 10, 2022
@mroeschke @meeseeksmachine
Backport PR pandas-dev#48973: CI: Bump timeouts Python dev builds
6498319
lithomas1 pushed a commit that referenced this pull request Oct 10, 2022
@meeseeksmachine @mroeschke
Backport PR #48973 on branch 1.5.x (CI: Bump timeouts Python dev buil…
20ee5ec 
…ds) (#49028)

Backport PR #48973: CI: Bump timeouts Python dev builds

Co-authored-by: Matthew Roeschke <[email protected]>
noatamir pushed a commit to noatamir/pandas that referenced this pull request Nov 9, 2022
@mroeschke @noatamir
CI: Bump timeouts Python dev builds (pandas-dev#48973)
a7f0fb5 
* Bump timeout

* add --no-index

* install numpy nightlies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jarrodmillman jarrodmillman jarrodmillman left review comments

@lithomas1 lithomas1 lithomas1 approved these changes

Assignees
No one assigned
Labels
CI Continuous Integration
Projects
None yet
Milestone
1.5.1
Development

Successfully merging this pull request may close these issues.
4 participants
@mroeschke @jbrockmendel @lithomas1 @jarrodmillman