Skip to content

CI Builds: Conditional builds with path filtering #9787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .circleci/config-conditional-builds.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
version: 2.1

orbs:
codecov: codecov/[email protected]

jobs:
tests:
docker:
- image: 'cimg/python:3.10'
environment:
TOX_POSARGS: ''
- image: 'docker.elastic.co/elasticsearch/elasticsearch:7.14.0'
name: search
environment:
discovery.type: single-node
ES_JAVA_OPTS: -Xms750m -Xmx750m
steps:
- when:
condition: build-code
steps:
- checkout
- run: git submodule sync
- run: git submodule update --init
- run: pip install --user 'tox<5'
- run: tox -e py310
- codecov/upload

tests-embedapi:
docker:
- image: 'cimg/python:3.10'
steps:
- when:
condition: build-code
steps:
- checkout
- run: git submodule sync
- run: git submodule update --init
- run: pip install --user 'tox<4'
- run: tox -c tox.embedapi.ini

checks:
docker:
- image: 'cimg/python:3.10'
environment:
NODE_VERSION: 10.17.0
# This could also be done more conditionally like the former tests
steps:
- checkout
- run: git submodule sync
- run: git submodule update --init
- run: git fetch origin main # needed for comparisson in pre-commit
- run: git branch -f --track main origin/main # needed for comparisson in pre-commit
- run: pip install --user 'tox<5'
- run: tox -e migrations
- run: tox -e pre-commit
- run: tox -e lint
- run: scripts/circle/install_node.sh
- run:
name: Add node to the path
command: |
echo 'export PATH=~/.nvm/versions/node/v${NODE_VERSION}/bin:$PATH' >> $BASH_ENV
source $BASH_ENV
- run: tox -e eslint

workflows:
version: 2
test:
jobs:
- checks
- tests
- tests-embedapi:
requires:
- checks
- tests
74 changes: 13 additions & 61 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,67 +1,19 @@
version: 2.1

orbs:
codecov: codecov/[email protected]

jobs:
tests:
docker:
- image: 'cimg/python:3.10'
environment:
TOX_POSARGS: ''
- image: 'docker.elastic.co/elasticsearch/elasticsearch:7.14.0'
name: search
environment:
discovery.type: single-node
ES_JAVA_OPTS: -Xms750m -Xmx750m
steps:
- checkout
- run: git submodule sync
- run: git submodule update --init
- run: pip install --user 'tox<5'
- run: tox -e py310
- codecov/upload
setup: true

tests-embedapi:
docker:
- image: 'cimg/python:3.10'
steps:
- checkout
- run: git submodule sync
- run: git submodule update --init
- run: pip install --user 'tox<4'
- run: tox -c tox.embedapi.ini

checks:
docker:
- image: 'cimg/python:3.10'
environment:
NODE_VERSION: 10.17.0
steps:
- checkout
- run: git submodule sync
- run: git submodule update --init
- run: git fetch origin main # needed for comparisson in pre-commit
- run: git branch -f --track main origin/main # needed for comparisson in pre-commit
- run: pip install --user 'tox<5'
- run: tox -e pre-commit
- run: tox -e migrations
- run: tox -e lint
- run: scripts/circle/install_node.sh
- run:
name: Add node to the path
command: |
echo 'export PATH=~/.nvm/versions/node/v${NODE_VERSION}/bin:$PATH' >> $BASH_ENV
source $BASH_ENV
- run: tox -e eslint
orbs:
path-filtering: circleci/[email protected]

workflows:
version: 2
test:
generate-config:
jobs:
- checks
- tests
- tests-embedapi:
requires:
- checks
- tests
- path-filtering/filter:
base-revision: main
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we target branches different from main, there might be some false positives. I've opened up a feature request: CircleCI-Public/path-filtering-orb#65

config-path: .circleci/config-conditional-builds.yml
mapping: |
requirements/.* build-code true
readthedocs/.* build-code true
pytest.* build-code true
tox.* build-code true
package.* build-code true
Copy link
Contributor Author

@benjaoming benjaoming Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are essential bits to adjust. We want to run this stuff whenever something essential is changed.

2 considerations that we might have for later:

  1. Breaking up checks and tests pipelines and give them separate path filters.
  2. Branch rules: I think we should also consider an alternative pipeline that always runs on the main branch and sets build-code to true regardless of the path. This requires adding a separate pipeline that runs on main (and configuring the other pipeline to NOT run on main), calling the continuation Orb.