ENH/CI: Add Cache for Azure pipelines - posix #44029

trallard · 2021-10-14T10:59:00Z

closes #ENH/CI: Cache packages installed from conda #42580
tests added / passed
Ensure all linting tests pass, see here for how to run them
whatsnew entry

Trying to leverage the use of cache on Azure pipelines - currently WIP will make some investigations

trallard · 2021-10-25T18:50:03Z

Hey folks it seems I have most of this figured, some notes

MacOs is working now as expected (see Azure pipeline)
Windows - some nagging issues with paths are still present but can work on that
Up to discussion: it would be best to explore the use of https://github.com/conda-incubator/conda-lock since the dependencies are not explicitly pinned in environment.yaml

trallard · 2021-10-26T10:31:43Z

Update: The MacOS pipelines are still working as expected, however since yesterday I noticed that the Windows pipelines are getting stuck in conda-update and eventually timeout (see here)

I might for now try and do some debugging and see if I can pinpoint to the root cause, but in the meantime a review to the rest of the PR would be grand

lithomas1 · 2021-10-30T00:14:33Z

Thanks for the PR. Do you know how much time this saves?

Update: The MacOS pipelines are still working as expected, however since yesterday I noticed that the Windows pipelines are getting stuck in conda-update and eventually timeout (see here)

I might for now try and do some debugging and see if I can pinpoint to the root cause, but in the meantime a review to the rest of the PR would be grand

If it's easier for you, you can update the posix pipelines first and do the Windows pipelines in a follow up.

Also, I noticed that as of the latest commit conda update is not being performed. We probably want to make sure that the any old packages in the cache are updated.

ci/azure/posix.yml

Co-authored-by: Thomas Li <[email protected]>

trallard · 2021-11-01T17:51:22Z

Hey @lithomas1 thanks for having a look - I have gone ahead and split the posix and Windows cache parts as suggested.

As is the posix cache seems to be saving ~5 mins per job

jreback · 2021-11-14T03:10:15Z

can you merge master

trallard · 2021-11-15T20:26:01Z

Sorted!

lithomas1

The Azure build is failing unfortunately. I left a suggestion that might fix it, and some other feedback.

lithomas1 · 2021-11-15T21:32:36Z

ci/azure/posix.yml

+      condition: eq(variables.CONDA_CACHE_RESTORED, 'false')
+
+    - script: |
+        conda update -q -y -n pandas-dev --file $ENV_FILE


Suggested change

conda update -q -y -n pandas-dev --file $ENV_FILE

conda env update -q -y -n pandas-dev --file $ENV_FILE

lithomas1 · 2021-11-15T21:35:54Z

ci/azure/posix.yml

+        python -m pip install --no-build-isolation -e .
+        echo "conda list"
+        conda list
+      displayName: 'Build extensions and install pandas'


Is there a reason you are not using ci/setup_env.sh? Note that that script is also used by other builds on Github Actions, so if you are changing that file, you should gate your changes behind an env variable to avoid it affecting other builds.

jreback · 2021-11-28T20:54:38Z

does this help in build time? can you show before / after

jbrockmendel · 2021-12-27T00:25:03Z

@trallard can you merge master and address comments

trallard · 2022-01-04T16:27:27Z

Sorry folks I took some time off at the end of year - will merge master and add the details requested today

jbrockmendel · 2022-02-02T04:19:00Z

@trallard what's the status here?

lithomas1

Sorry for the late review and thanks for updating the PR. Can you merge main again?

lithomas1 · 2022-02-05T18:21:44Z

ci/azure/posix.yml

+        source activate pandas-dev
+        conda list pandas
+        python setup.py build_ext -q -j2
+        python -m pip install --no-deps -U pip wheel setuptools


I think setuptools will need to be pinned to <60 to pass CI.

mroeschke · 2022-03-30T01:22:29Z

Thanks for the PR, but it appears to have gone stale. Additionally, there is some discussion about moving aware from Azure pipelines as well #46351. Closing, but would be happy to take similar changes to our GHA builds!

trallard changed the title ~~Trallard/ci cache~~ BLD: Add Cache for Azure pipelines Oct 14, 2021

trallard changed the title ~~BLD: Add Cache for Azure pipelines~~ ENH/CI: Add Cache for Azure pipelines Oct 14, 2021

trallard marked this pull request as draft October 14, 2021 11:56

jreback added the CI Continuous Integration label Oct 21, 2021

jreback mentioned this pull request Oct 21, 2021

CI: Add Cache task on Windows #43966

Closed

4 tasks

trallard added 4 commits October 25, 2021 15:39

BLD: Add conda cache to azure ci

39fa0b0

BLD: Add conda cache to Windows azure CI

2198ca8

BLD: Trivial change to test cache

bba2735

ENH: Add conda cache - windows and posix

7dc9abe

trallard force-pushed the trallard/CI-cache branch from 5c4bf62 to 7dc9abe Compare October 25, 2021 15:23

trallard added 5 commits October 25, 2021 16:27

ENH: Should be a path not a string

9de1b58

ENH: Move pandas build - should be its own task

8950f42

ENH: Ensure the cache path exists - Windows

7371150

ENH: cache miniconda too

8d93bc8

ENH: Ensure cache dir exists - Windows

58e8df5

trallard added 4 commits October 25, 2021 20:22

ENH: Force make cache dir

027684a

ENH: Revert to original setup_env.sh

834f824

ENH: use bash for conda update

df307fa

ENH: Use Miniconda dir in Windows

76ba935

trallard marked this pull request as ready for review October 26, 2021 11:43

trallard added 5 commits October 28, 2021 16:46

Trivial change to trigger pipeline

f13efbe

Revert to base CI

77cd415

ENH: Add cache again

56dc350

👷 Add cache hit

2480556

🐛 Fix Cache path

9a9f1db

trallard added 2 commits October 30, 2021 11:17

Trivial change to trigger pipelines

8325e03

Print conda info

464dddf

trallard added 3 commits October 30, 2021 11:38

Add conda update to posix

1277c6a

🐛 Fix imcomplte script

578ec73

Fix update command

03692a2

lithomas1 reviewed Oct 30, 2021

View reviewed changes

ci/azure/posix.yml Outdated Show resolved Hide resolved

trallard and others added 2 commits November 1, 2021 17:48

Update ci/azure/posix.yml

a26a896

Co-authored-by: Thomas Li <[email protected]>

Revert Windows ci

4df7ede

trallard changed the title ~~ENH/CI: Add Cache for Azure pipelines~~ ENH/CI: Add Cache for Azure pipelines - posix Nov 1, 2021

Merge remote-tracking branch 'upstream/master' into trallard/CI-cache

f6dd43f

lithomas1 requested changes Nov 15, 2021

View reviewed changes

amrutha-mukkatira mentioned this pull request Jan 11, 2022

ENH/CI: Cache packages installed from conda #42580

Closed

trallard added 3 commits January 18, 2022 13:14

Merge remote-tracking branch 'origin/main' into trallard/CI-cache

ce89fde

Fix env update

dfcd0d9

Remove illegal flag

bff1bee

lithomas1 reviewed Feb 5, 2022

View reviewed changes

mroeschke closed this Mar 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH/CI: Add Cache for Azure pipelines - posix #44029

ENH/CI: Add Cache for Azure pipelines - posix #44029

trallard commented Oct 14, 2021 •

edited

Loading

trallard commented Oct 25, 2021 •

edited

Loading

trallard commented Oct 26, 2021

lithomas1 commented Oct 30, 2021

trallard commented Nov 1, 2021

jreback commented Nov 14, 2021

trallard commented Nov 15, 2021

lithomas1 left a comment

lithomas1 Nov 15, 2021

lithomas1 Nov 15, 2021

jreback commented Nov 28, 2021

jbrockmendel commented Dec 27, 2021

trallard commented Jan 4, 2022

jbrockmendel commented Feb 2, 2022

lithomas1 left a comment

lithomas1 Feb 5, 2022

mroeschke commented Mar 30, 2022

	conda update -q -y -n pandas-dev --file $ENV_FILE
	conda env update -q -y -n pandas-dev --file $ENV_FILE

ENH/CI: Add Cache for Azure pipelines - posix #44029

ENH/CI: Add Cache for Azure pipelines - posix #44029

Conversation

trallard commented Oct 14, 2021 • edited Loading

trallard commented Oct 25, 2021 • edited Loading

trallard commented Oct 26, 2021

lithomas1 commented Oct 30, 2021

trallard commented Nov 1, 2021

jreback commented Nov 14, 2021

trallard commented Nov 15, 2021

lithomas1 left a comment

Choose a reason for hiding this comment

lithomas1 Nov 15, 2021

Choose a reason for hiding this comment

lithomas1 Nov 15, 2021

Choose a reason for hiding this comment

jreback commented Nov 28, 2021

jbrockmendel commented Dec 27, 2021

trallard commented Jan 4, 2022

jbrockmendel commented Feb 2, 2022

lithomas1 left a comment

Choose a reason for hiding this comment

lithomas1 Feb 5, 2022

Choose a reason for hiding this comment

mroeschke commented Mar 30, 2022

trallard commented Oct 14, 2021 •

edited

Loading

trallard commented Oct 25, 2021 •

edited

Loading