Skip to content

CI: Refactor sdist workflow and add setup-python action #46538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

CI: Refactor sdist workflow and add setup-python action #46538

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
266b06c
Use setup-python action
jonashaag Mar 29, 2022
83481aa
Move pip reqs to file
jonashaag Apr 4, 2022
c522db3
fix
jonashaag Apr 4, 2022
180e2b8
review feedback
jonashaag Apr 5, 2022
8ef99ce
Merge branch 'main' into refactor-ci-sdist
jonashaag May 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .github/actions/setup-python/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: Setup Python and install requirements
inputs:
python-version:
required: true
architecture:
default: x64
runs:
using: composite
steps:
# TODO: GH#44980 https://github.com/pypa/setuptools/issues/2941
- name: Create temporary requirements.txt
run: |
# Drop cache at least once per month
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why would we cache this at all?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it saves a couple of minutes but we can also add it later.

month_today="$(date '+%Y-%m')"
cat > requirements.txt <<EOF
# $month_today

# Python deps
pip
setuptools<60.0.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm don't love that these deps (pandas + test) are also specified in yaml files as well (aware over multiple yaml files as well). Any idea on how to combine them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before the refactoring they were part of an inline pip install ....

We might be able to auto-generate them from a minimal environment.yml file but not sure if that's better than having a little duplication.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. Could this be maintained in a separate file and modified here with the date to invalidate the cache?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to do it, but we'd still need to add the current timestamp (or something derived from it, like current month) to the file because otherwise the cache will only be invalidated when we change the file, which we don't do on a regular basis.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB: For setup-python a stale cache just means slower installs, not older versions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah makes sense; this block can create a copy of the environment file and add a date. I think it's just easier to manage/know that our dependencies are in files and not as ad hoc like this.

wheel

# Pandas deps
# GH 39416
numpy
cython
python-dateutil
pytz

# Test deps
git+https://github.com/nedbat/coveragepy.git
hypothesis
pytest>=6.2.5
pytest-xdist
pytest-cov
pytest-asyncio>=0.17
EOF
shell: bash -el {0}

- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: ${{ inputs.python-version }}
architecture: ${{ inputs.architecture }}
cache: pip

- name: Fix $PATH on macOS
run: |
# On macOS, the Python version we installed above is too late in $PATH
# to be effective if using "bash -l" (which we need for code that works
# with Conda envs).
cat >> ~/.bash_profile <<EOF
export PATH="\$(echo "\$PATH" | grep -Eio '[^:]+hostedtoolcache/python[^:]+bin'):\$PATH" \
EOF
shell: bash -el {0}
if: ${{ runner.os == 'macOS' }}

- name: Install dependencies
run: |
cat requirements.txt
# TODO https://github.com/numpy/numpy/issues/21196
bits32=$(python -c 'import sys; print(int(sys.maxsize > 2**32))')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm what is this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is required for 32 bit builds but we can add that back later

NPY_DISABLE_SVML=$bits32 pip install -r requirements.txt
pip list
shell: bash -el {0}
10 changes: 6 additions & 4 deletions .github/workflows/sdist.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ on:
- "doc/**"

jobs:
build:
if: ${{ github.event.label.name == 'Build' || contains(github.event.pull_request.labels.*.name, 'Build') || github.event_name == 'push'}}
sdist:
#if: ${{ github.event.label.name == 'Build' || contains(github.event.pull_request.labels.*.name, 'Build') || github.event_name == 'push'}}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to undo this once GHA is green.

runs-on: ubuntu-latest
timeout-minutes: 60
defaults:
Expand All @@ -26,6 +26,8 @@ jobs:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10"]
name: sdist Python ${{ matrix.python-version }}

concurrency:
# https://github.community/t/concurrecy-not-work-for-push/183068/7
group: ${{ github.event_name == 'push' && github.run_number || github.ref }}-${{matrix.python-version}}-sdist
Expand All @@ -36,8 +38,8 @@ jobs:
with:
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v3
- name: Set up Python ${{ matrix.python-version }} and install dependencies
uses: ./.github/actions/setup-python
with:
python-version: ${{ matrix.python-version }}

Expand Down