Skip to content

ENH/CI: Add Cache for Azure pipelines - posix #44029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 29 commits into from
Closed

ENH/CI: Add Cache for Azure pipelines - posix #44029

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
39fa0b0
BLD: Add conda cache to azure ci
trallard Oct 14, 2021
2198ca8
BLD: Add conda cache to Windows azure CI
trallard Oct 14, 2021
bba2735
BLD: Trivial change to test cache
trallard Oct 14, 2021
7dc9abe
ENH: Add conda cache - windows and posix
trallard Oct 25, 2021
9de1b58
ENH: Should be a path not a string
trallard Oct 25, 2021
8950f42
ENH: Move pandas build - should be its own task
trallard Oct 25, 2021
7371150
ENH: Ensure the cache path exists - Windows
trallard Oct 25, 2021
8d93bc8
ENH: cache miniconda too
trallard Oct 25, 2021
58e8df5
ENH: Ensure cache dir exists - Windows
trallard Oct 25, 2021
027684a
ENH: Force make cache dir
trallard Oct 25, 2021
834f824
ENH: Revert to original setup_env.sh
trallard Oct 25, 2021
df307fa
ENH: use bash for conda update
trallard Oct 25, 2021
76ba935
ENH: Use Miniconda dir in Windows
trallard Oct 26, 2021
f13efbe
Trivial change to trigger pipeline
trallard Oct 28, 2021
77cd415
Revert to base CI
trallard Oct 28, 2021
56dc350
ENH: Add cache again
trallard Oct 28, 2021
2480556
:construction_worker: Add cache hit
trallard Oct 28, 2021
9a9f1db
:bug: Fix Cache path
trallard Oct 28, 2021
8325e03
Trivial change to trigger pipelines
trallard Oct 30, 2021
464dddf
Print conda info
trallard Oct 30, 2021
1277c6a
Add conda update to posix
trallard Oct 30, 2021
578ec73
:bug: Fix imcomplte script
trallard Oct 30, 2021
03692a2
Fix update command
trallard Oct 30, 2021
a26a896
Update ci/azure/posix.yml
trallard Nov 1, 2021
4df7ede
Revert Windows ci
trallard Nov 1, 2021
f6dd43f
Merge remote-tracking branch 'upstream/master' into trallard/CI-cache
trallard Nov 15, 2021
ce89fde
Merge remote-tracking branch 'origin/main' into trallard/CI-cache
trallard Jan 18, 2022
dfcd0d9
Fix env update
trallard Jan 18, 2022
bff1bee
Remove illegal flag
trallard Jan 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion ci/azure/posix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,33 @@ jobs:
- script: echo '##vso[task.prependpath]$(HOME)/miniconda3/bin'
displayName: 'Set conda path'

- task: Cache@2
displayName: Use cached Anaconda environment
inputs:
key: 'conda | "$(Agent.OS)" | $(ENV_FILE)'
path: $(HOME)/miniconda3/
restoreKeys: |
python | "$(Agent.OS)"
python
cacheHitVar: CONDA_CACHE_RESTORED

- script: ci/setup_env.sh
displayName: 'Setup environment and build pandas'
displayName: 'Setup environment'
condition: eq(variables.CONDA_CACHE_RESTORED, 'false')

- script: |
conda env update -q -n pandas-dev --file $ENV_FILE
displayName: 'Update packages in pandas-dev'

- script: |
source activate pandas-dev
conda list pandas
python setup.py build_ext -q -j2
python -m pip install --no-deps -U pip wheel setuptools
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think setuptools will need to be pinned to <60 to pass CI.

python -m pip install --no-build-isolation -e .
echo "conda list"
conda list
displayName: 'Build extensions and install pandas'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason you are not using ci/setup_env.sh? Note that that script is also used by other builds on Github Actions, so if you are changing that file, you should gate your changes behind an env variable to avoid it affecting other builds.


- script: |
source activate pandas-dev
Expand Down
8 changes: 3 additions & 5 deletions ci/setup_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ if [[ "$(uname)" == "Linux" && -n "$LC_ALL" ]]; then
echo
fi


echo "Install Miniconda"
DEFAULT_CONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-latest"
if [[ "$(uname -m)" == 'aarch64' ]]; then
Expand All @@ -26,8 +25,8 @@ elif [[ "$(uname)" == 'Linux' ]]; then
elif [[ "$(uname)" == 'Darwin' ]]; then
CONDA_URL="$DEFAULT_CONDA_URL-MacOSX-x86_64.sh"
else
echo "OS $(uname) not supported"
exit 1
echo "OS $(uname) not supported"
exit 1
fi
echo "Downloading $CONDA_URL"
wget -q $CONDA_URL -O miniconda.sh
Expand All @@ -46,7 +45,7 @@ echo
echo "update conda"
conda config --set ssl_verify false
conda config --set quiet true --set always_yes true --set changeps1 false
conda install pip conda # create conda to create a historical artifact for pip & setuptools
conda install pip conda # create conda to create a historical artifact for pip & setuptools
conda update -n base conda
conda install -y -c conda-forge mamba

Expand All @@ -66,7 +65,6 @@ echo
echo "mamba env create -q --file=${ENV_FILE}"
time mamba env create -q --file="${ENV_FILE}"


if [[ "$BITS32" == "yes" ]]; then
# activate 32-bit compiler
export CONDA_BUILD=1
Expand Down