Shard gl jasmine tests #2933
Shard gl jasmine tests #2933
Conversation
.circleci/test.sh
Outdated
| @@ -34,7 +35,12 @@ case $1 in | |||
| ;; | |||
|
|
|||
| jasmine2) | |||
| retry npm run test-jasmine -- --tags=gl --skip-tags=noCI,flaky | |||
| SHARDS=($(node $ROOT/tasks/shard_jasmine_tests.js --tag=gl)) | |||
There was a problem hiding this comment.
node tasks/shard_jasmine_tests.js --tag=gl outputs:
gl2d_plot_interact_test.js
gl3d_plot_interact_test.js
parcoords_test.js
gl2d_click_test.js
splom_test.js,cartesian_test.js,gl2d_scatterplot_contour_test.js,gl2d_pointcloud_test.js,gl_plot_interact_basic_test.js
streamtube_test.js,gl2d_date_axis_render_test.js,cone_test.js
which is then converted into a bash array (that's the outer ()) and then looped below.
| glob(path.join(pathToJasmineTests, '*.js'), function(err, files) { | ||
| if(err) throw err; | ||
|
|
||
| var file2cnt = {}; |
There was a problem hiding this comment.
to note, with --limit=20 and --tag=gl, this fills up to:
{ 'cartesian_test.js': 1,
'cone_test.js': 5,
'gl_plot_interact_basic_test.js': 2,
'gl2d_click_test.js': 20,
'gl2d_date_axis_render_test.js': 4,
'gl2d_plot_interact_test.js': 28,
'gl2d_pointcloud_test.js': 2,
'gl2d_scatterplot_contour_test.js': 2,
'gl3d_plot_interact_test.js': 26,
'parcoords_test.js': 30,
'splom_test.js': 10,
'streamtube_test.js': 10 }
| Plotly.purge(gd); | ||
| destroyGraphDiv(); | ||
| setTimeout(done, 1000); |
There was a problem hiding this comment.
this appears to reduce the number of intermittent failures (and the number of retry attempts) a little.
|
Nice! 💃 Non-blocking: This leaves |
|
Another (non-blocking) thought to speed up |
That's a great idea. I'll push another commit this morning. I was also looking at ways to make the retry not waste time bundling on every attempt. I couldn't find anything in user land, but this could be a nice karma plugin. Bundling our test code takes about ~10s, over multiple retries we can easily spend 1 minute just bundling code. |
- which makes karma exits upon first test failure
- this might help us debug intermittent failures in our tests
| @@ -235,6 +241,8 @@ func.defaultConfig = { | |||
| suppressPassed: true, | |||
| suppressSkipped: false, | |||
| showSpecTiming: false, | |||
| // use 'karma-fail-fast-reporter' to fail fast w/o conflicting | |||
| // with other karma plugins | |||
| failFast: false | |||
There was a problem hiding this comment.
So this failFast shouldn't ever be used?
There was a problem hiding this comment.
Setting it to true won't break anything, but it won't actually fail fast.
There was a problem hiding this comment.
More on this topic: to make karma-spec-reporter fail fast, one needs to add it to the plugins list: https://github.com/mlex/karma-spec-reporter#configuration
When doing so, it conflicts with karma-browserify.
There might be a way, to list them (order dependent?) so that they don't conflict, but using karma-fail-fast-reporter was easy enough.
| n=$[$n+1] | ||
| done | ||
|
|
||
| if [ $n -eq $MAX_AUTO_RETRY ]; then | ||
| log "one last time, w/o failing fast" | ||
| "$@" && n=0 |
There was a problem hiding this comment.
man, shell scripts are ugly... that said, very nicely done 👍
There was a problem hiding this comment.
That was the cleanest solution I could think of 😑
- this is a relic from the circular-dependency era, - we use to have to do this in order to run test per-suite to setup the registry correctly. - this should speed bundling times in single-suite runs
|
💃 is still dancing. |
resolves #2922 (hopefully)
This PR adds a sharding script that splits our jasmine test suites into groups of a given number of
itblocks. We use it here to shard jasmine tests with tag@gl, but the same logic could eventually be used in other jasmine tests.To make the sharding logic simple and predictable, we enforce the following jasmine test tag rules:
@gl,@noCI,@noCIdepand@flakyare the only allowed tags@gland@flakycan't be set indescribeblocks, only initblocksitblock can't have both a@gland a@flakytag (i.e. all@glare flaky and are in runned in retry loop on ci, setting both tags is redundant)Along the way, commit 7712507 made
gl2d_plot_interact_test.jsandgl2d_click_test.jsmore robust and bdf468e moved the@flakytest run fromjasmine)tojasmine2)on CI. Things appear to be working fine: many 📗 runs are occuring in succession on CI.cc @alexcjohnson hopefully this will be enough to not have to worry about intermittent failures for a while.