[Auth CI] Log warning if chrome version has changed #7872
Conversation
|
Size Report 1Affected ProductsNo changes between base commit (2e32eeb) and merge commit (444c058).Test Logs |
Size Analysis Report 1Affected ProductsNo changes between base commit (2e32eeb) and merge commit (444c058).Test Logs |
.github/workflows/test-all.yml
Outdated
| @@ -39,6 +40,11 @@ jobs: | |||
| - name: install Chrome stable | |||
| run: | | |||
| npx @puppeteer/browsers install chrome@stable | |||
| chromeVersionString=$(ls chrome) | |||
| if [ "$CHROME_VALIDATED_VERSION" != "$chromeVersionString" ]; then | |||
There was a problem hiding this comment.
Will this stand out in the workflow in any way if the tests fail? I'm guessing contributors will look at the failed step further down if tests fail, is there anything to direct them to look at this one?
| @@ -25,6 +25,7 @@ env: | |||
| # the beahvior to use the new URLs. | |||
| CHROMEDRIVER_CDNURL: https://googlechromelabs.github.io/ | |||
| CHROMEDRIVER_CDNBINARIESURL: https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/ | |||
| CHROME_VALIDATED_VERSION: linux-120.0.6099.71 | |||
There was a problem hiding this comment.
Is this possible that this line can be automatically updated?
There was a problem hiding this comment.
Not easily, unless you have any ideas!
DellaBitta
requested review from
lisajian,
prameshj and
sam-gc
as code owners
Discussion
The GitHub Action CI installs the latest version of Chrome. There have been occurrences in the past when the Auth tests suddenly, and unexpectedly, started to fail, and after hours of investigation it was determined that the root cause of the problem was a silent upgrade of the Chrome version that we tested against.
While pinning the chrome version would keep the tests from suddenly failing, this solution has some drawbacks. 1: we're not testing against the latest version of chrome. 2: it will take someone to remember to update the version of chrome periodically, and this might fall to the wayside over time.
As an alternative, this solution will cause the CI workflow to produce an annotation warning if the chrome version has suddenly changed. If tests fail, the engineer investigating the tests can see clearly that the chrome version has changed, and perhaps this is what caused the test to fail.
There will be some burden of support. An engineer will have to update an environment variable in the workflow which stores the last valid version of chrome that we tested against. But forgetting to update this value is preferable to forgetting to update the version of chrome (if we were to pin it), so this seems like the better course of action overall.
Testing
CI run with the annotation from an intentionally mangled chrome version name to force the warning:
API Changes
N/A