Skip to content

refactor(ci): fix fetch-depth and add some caching #5563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 19, 2022
Merged

refactor(ci): fix fetch-depth and add some caching #5563

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64bdc80
refactor: rename ci -> build.yaml
jsjoeio Sep 16, 2022
6f0ff23
feat: add build.yaml
jsjoeio Sep 16, 2022
733dac4
feat: add node caching to platform jobs
jsjoeio Sep 16, 2022
79d0ea2
trigger ci
jsjoeio Sep 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 40 additions & 28 deletions .github/workflows/ci.yaml → .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ concurrency:
# Note: if: success() is used in several jobs -
# this ensures that it only executes if all previous jobs succeeded.

# if: steps.cache-yarn.outputs.cache-hit != 'true'
# if: steps.cache-node-modules.outputs.cache-hit != 'true'
# will skip running `yarn install` if it successfully fetched from cache

jobs:
Expand All @@ -29,9 +29,6 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0
submodules: true

- name: Install Node.js v16
uses: actions/setup-node@v3
Expand All @@ -40,9 +37,14 @@ jobs:

- name: Install helm
uses: azure/[email protected]
with:
token: ${{ secrets.GITHUB_TOKEN }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes warning from helm. Token needed for v3 and later.


- name: Install helm kubeval plugin
run: helm plugin install https://github.com/instrumenta/helm-kubeval
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this for our lint.sh to actually lint the Helm chart.


- name: Fetch dependencies from cache
id: cache-yarn
id: cache-node-modules
uses: actions/cache@v3
with:
path: "**/node_modules"
Expand All @@ -51,8 +53,8 @@ jobs:
yarn-build-

- name: Install dependencies
if: steps.cache-yarn.outputs.cache-hit != 'true'
run: yarn --frozen-lockfile
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: SKIP_SUBMODULE_DEPS=1 yarn --frozen-lockfile

- name: Run yarn fmt
run: yarn fmt
Expand All @@ -73,11 +75,13 @@ jobs:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0
submodules: true

- name: Install quilt
run: sudo apt update && sudo apt install quilt
uses: awalsh128/cache-apt-pkgs-action@latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat action!

with:
packages: quilt
version: 1.0

- name: Patch Code
run: quilt push -a
Expand All @@ -88,7 +92,7 @@ jobs:
node-version: "16"

- name: Fetch dependencies from cache
id: cache-yarn
id: cache-node-modules
uses: actions/cache@v3
with:
path: "**/node_modules"
Expand All @@ -97,7 +101,7 @@ jobs:
yarn-build-

- name: Install dependencies
if: steps.cache-yarn.outputs.cache-hit != 'true'
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: yarn --frozen-lockfile

- name: Build code-server
Expand Down Expand Up @@ -171,8 +175,6 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Download artifact
uses: actions/download-artifact@v3
Expand Down Expand Up @@ -224,8 +226,6 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install Node.js v16
uses: actions/setup-node@v3
Expand Down Expand Up @@ -262,8 +262,18 @@ jobs:
- name: Build standalone release
run: source scl_source enable devtoolset-9 && yarn release:standalone

- name: Fetch dependencies from cache
id: cache-node-modules
uses: actions/cache@v3
with:
path: "**/node_modules"
key: yarn-build-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
yarn-build-
Comment on lines +265 to +272
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we could add this block to the package-linux-cross step as well.

I think we might want to add platform keys so we cache separately for different platforms. The reason being is that I vaguely recall seeing native modules not rebuild if they already exist even if they are for the wrong platform.

So something like:

          key: yarn-build-linux-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            yarn-build-linux-

Also the yarn release:standalone script runs npm install and it would be nice to cache those modules as well but we could handle that in another PR. But broadly speaking I think we could remove the standalone script and put those steps straight into the yaml instead that way we can gate the install behind the cache hit. I also think we could just do it straight in release instead of copying everything to a release-standalone directory first.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively we could remove just the npm install line and put that in the yaml. Or make yarn release:standalone take an argument for whether to install.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good thinking! I'm going to add this to my todo list since I didn't get to it here. I'm thinking we do the approach you suggested and move the platform steps out of build.yaml so maybe I can tackle this with that!


- name: Install test dependencies
run: SKIP_SUBMODULE_DEPS=1 yarn install
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: SKIP_SUBMODULE_DEPS=1 yarn --frozen-lockfile

- name: Run integration tests on standalone release
run: yarn test:integration
Expand Down Expand Up @@ -320,8 +330,6 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install Node.js v16
uses: actions/setup-node@v3
Expand Down Expand Up @@ -373,8 +381,6 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install Node.js v16
uses: actions/setup-node@v3
Expand All @@ -398,7 +404,17 @@ jobs:
- name: Build standalone release
run: yarn release:standalone

- name: Fetch dependencies from cache
id: cache-node-modules
uses: actions/cache@v3
with:
path: "**/node_modules"
key: yarn-build-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
yarn-build-
Comment on lines +407 to +414
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as previous except yarn-build-macos-.


- name: Install test dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: SKIP_SUBMODULE_DEPS=1 yarn install

- name: Run integration tests on standalone release
Expand All @@ -425,16 +441,14 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install Node.js v16
uses: actions/setup-node@v3
with:
node-version: "16"

- name: Fetch dependencies from cache
id: cache-yarn
id: cache-node-modules
uses: actions/cache@v3
with:
path: "**/node_modules"
Expand All @@ -455,7 +469,7 @@ jobs:
mv code-server*-linux-amd64 code-server-linux-amd64

- name: Install dependencies
if: steps.cache-yarn.outputs.cache-hit != 'true'
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: SKIP_SUBMODULE_DEPS=1 yarn --frozen-lockfile

- name: Install Playwright OS dependencies
Expand Down Expand Up @@ -488,16 +502,14 @@ jobs:
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install Node.js v16
uses: actions/setup-node@v3
with:
node-version: "16"

- name: Fetch dependencies from cache
id: cache-yarn
id: cache-node-modules
uses: actions/cache@v3
with:
path: "**/node_modules"
Expand All @@ -518,7 +530,7 @@ jobs:
mv code-server*-linux-amd64 code-server-linux-amd64

- name: Install dependencies
if: steps.cache-yarn.outputs.cache-hit != 'true'
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: SKIP_SUBMODULE_DEPS=1 yarn --frozen-lockfile

- name: Install Playwright OS dependencies
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
id: download
with:
branch: release/v${{ steps.version.outputs.version }}
workflow: ci.yaml
workflow: build.yaml
workflow_conclusion: completed
name: "npm-package"
path: release-npm-package
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
id: download
with:
branch: ${{ github.ref }}
workflow: ci.yaml
workflow: build.yaml
workflow_conclusion: completed
check_artifacts: true
name: release-packages
Expand Down
4 changes: 2 additions & 2 deletions docs/SECURITY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ We use the following tools to help us stay on top of vulnerability mitigation.
- [trivy](https://github.com/aquasecurity/trivy)
- Comprehensive vulnerability scanner that runs on PRs into the default
branch and scans both our container image and repository code (see
`trivy-scan-repo` and `trivy-scan-image` jobs in `ci.yaml`)
`trivy-scan-repo` and `trivy-scan-image` jobs in `build.yaml`)
- [`audit-ci`](https://github.com/IBM/audit-ci)
- Audits npm and Yarn dependencies in CI (see `Audit for vulnerabilities` step
in `ci.yaml`) on PRs into the default branch and fails CI if moderate or
in `build.yaml`) on PRs into the default branch and fails CI if moderate or
higher vulnerabilities (see the `audit.sh` script) are present.

## Supported Versions
Expand Down