Skip to content

Flaky image tests #4321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 30, 2019
Merged

Flaky image tests #4321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
da53969
add --skip-flaky and --just-flaky CLI opts for `text-image`
etpinard Oct 30, 2019
0d11c5c
add `treemap_textposition` to list of flaky mocks
etpinard Oct 30, 2019
245b38f
add skip-flaky to image) run, run --just-flaky in image2)
etpinard Oct 30, 2019
11bc625
add `@flaky` to 'hover effects for titles that used to be blank' test
etpinard Oct 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .circleci/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,12 @@ case $1 in

image)
SUITE=$(find $ROOT/test/image/mocks/ -type f -printf "%f\n" | circleci tests split)
npm run test-image -- $SUITE --filter || EXIT_STATE=$?
npm run test-image -- $SUITE --filter --skip-flaky || EXIT_STATE=$?
exit $EXIT_STATE
;;

image2)
retry npm run test-image -- --just-flaky
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't even need to run this in a retry loop, but I thought might as well do this.

Let me know if you think 🔪 that retry would be a better idea.

npm run test-export || EXIT_STATE=$?
exit $EXIT_STATE
;;
Expand Down
77 changes: 44 additions & 33 deletions test/image/compare_pixels_test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,13 @@ var QUEUE_WAIT = 10;
* Run all gl3d image test in queue:
*
* npm run test-image -- gl3d_* --queue
*
*
*/

var argv = minimist(process.argv.slice(2), {boolean: ['queue', 'filter' ]});
var isInQueue = argv.queue;
var filter = argv.filter;
var argv = minimist(process.argv.slice(2), {
boolean: ['queue', 'filter', 'skip-flaky', 'just-flaky']
});

var allMock = false;
// If no pattern is provided, all mocks are compared
Expand Down Expand Up @@ -80,40 +82,39 @@ allMockList = allMockList.filter(unique);

// filter out untestable mocks if no pattern is specified (ie. we're testing all mocks)
// or if flag '--filter' is provided
if(allMock || filter) {
console.log('');
if(allMock || argv.filter) {
console.log('Filtering out untestable mocks:');
allMockList = allMockList.filter(untestableFilter);
console.log('\n');
}

sortGl2dMockList(allMockList);

// main
if(isInQueue) {
runInQueue(allMockList);
} else {
runInBatch(allMockList);
// Test cases:
// - font-wishlist
// - all mapbox
// don't behave consistently from run-to-run and/or
// machine-to-machine; skip over them for now.
allMockList = allMockList.filter(function(mockName) {
var cond = !(
mockName === 'font-wishlist' ||
mockName.indexOf('mapbox_') !== -1
);
if(!cond) console.log(' -', mockName);
return cond;
});
}

/* Test cases:
*
* - font-wishlist
* - all mapbox
*
* don't behave consistently from run-to-run and/or
* machine-to-machine; skip over them for now.
*
*/
function untestableFilter(mockName) {
var cond =
!(
mockName === 'font-wishlist' ||
mockName.indexOf('mapbox_') !== -1
);

if(!cond) console.log(' -', mockName);
var FLAKY_LIST = [
'treemap_textposition'
];

return cond;
console.log('');
if(argv['skip-flaky']) {
allMockList = allMockList.filter(function(mockName) {
var cond = FLAKY_LIST.indexOf(mockName) === -1;
if(!cond) console.log('Skipping flaky mock', mockName);
return cond;
});
} else if(argv['just-flaky']) {
allMockList = allMockList.filter(function(mockName) {
return FLAKY_LIST.indexOf(mockName) !== -1;
});
}

/* gl2d pointcloud and other non-regl gl2d mock(s)
Expand Down Expand Up @@ -277,3 +278,13 @@ function comparePixels(mockName, cb) {
.pipe(saveImageStream)
.on('close', checkImage);
}

sortGl2dMockList(allMockList);
console.log('');

// main
if(argv.queue) {
runInQueue(allMockList);
} else {
runInBatch(allMockList);
}
2 changes: 1 addition & 1 deletion test/jasmine/tests/titles_test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1141,7 +1141,7 @@ describe('Editable titles', function() {
.then(done);
});

it('has no hover effects for titles that used to be blank', function(done) {
it('@flaky has no hover effects for titles that used to be blank', function(done) {
Plotly.plot(gd, data, {
xaxis: {title: {text: ''}},
yaxis: {title: {text: ''}},
Expand Down