Major refactor of the image test scripts

tasks/baseline.sh

@@ -14,7 +14,7 @@ CMD=(
     "cp -f test/image/index.html ../server_app/index.html &&"
     "supervisorctl restart nw1 &&"
     "wget --server-response --spider --tries=10 --retry-connrefused http://localhost:9010/ping &&"
-    "node test/image/make_baseline.js $1"
+    "node test/image/make_baseline.js $@"
 )
 
 docker exec -i $CONTAINER_NAME /bin/bash -c "${CMD[*]}"

tasks/pretest.js

@@ -36,3 +36,11 @@ var setPlotConfig = [
 fs.writeFile(constants.pathToSetPlotConfig, setPlotConfig, function(err) {
     if(err) throw err;
 });
+
+// make artifact folders for image tests
+if(!fs.existsSync(constants.pathToTestImagesDiff)) {
+    fs.mkdirSync(constants.pathToTestImagesDiff);
+}
+if(!fs.existsSync(constants.pathToTestImages)) {
+    fs.mkdirSync(constants.pathToTestImages);
+}

tasks/test_export.sh

@@ -14,7 +14,7 @@ CMD=(
     "cp -f test/image/index.html ../server_app/index.html &&"
     "supervisorctl restart nw1 && "
     "wget --server-response --spider --tries=10 --retry-connrefused http://localhost:9010/ping &&"
-    "node test/image/export_test.js $1"
+    "node test/image/export_test.js $@"
 )
 
 docker exec -i $CONTAINER_NAME /bin/bash -c "${CMD[*]}"

tasks/test_image.sh

@@ -14,7 +14,7 @@ CMD=(
     "cp -f test/image/index.html ../server_app/index.html &&"
     "supervisorctl restart nw1 && "
     "wget --server-response --spider --tries=10 --retry-connrefused http://localhost:9010/ping &&"
-    "node test/image/compare_pixels_test.js $1"
+    "node test/image/compare_pixels_test.js $@"
 )
 
 docker exec -i $CONTAINER_NAME /bin/bash -c "${CMD[*]}"

test/image/README.md

@@ -26,7 +26,7 @@ the above evaluates the output of `docker-machine env default`.
 
 ### Step 1: Run the testing container
 
-Plotly.js uses `docker-compose` to ease the creation/stopping/deletion of the testing docker container.
+We use `docker-compose` to ease the creation/stopping/deletion of the testing docker container.
 
 Inside your `plotly.js` directory, run
 
@@ -40,25 +40,86 @@ as listed on [hub.docker.com](https://hub.docker.com/r/plotly/testbed/tags/) and
 
 ### Step 2: Run the image tests
 
-Inside your `plotly.js` directory, run
+The image testing docker container allows plotly.js developers to ([A](#a-run-image-comparison-tests) run image
+comparison tests, ([B](#b-run-image-export-tests) run image export tests and ([C](#c-generate-or-update-existing-baseline-image)) generate baseline
+images.
+
+**IMPORTANT:** the image tests scripts do **not** bundle the source files before
+running the image tests. We recommend running `npm run watch` or `npm start` in
+a separate tab to ensure that the most up-to-date code is used.
+
+##### A: Run image comparison tests
+
+Image comparison tests take in plotly.js mock json files (found in
+[`test/image/mocks`][mocks]), generate test png images (saved in
+`build/test_images/` - which is git-ignored) and compare them pixel-by-pixel to
+their corresponding baseline images (found in
+[`test/image/baselines`][baselines]) using
+[`GraphicsMagick`](https://github.com/aheckmann/gm).
+
+To run the image comparison tests, in your `plotly.js` directory:
 
 ```bash
 npm run test-image
 ```
 
-if some tests fail, compare their outputs using `npm run start-image_viewer`.
+which runs all image comparison tests in batch. If some tests fail, compare their outputs
+by booting up the test image viewer using `npm run start-image_viewer`.
 
-**IMPORTANT:** `npm run test-image` does **not** bundle the source files before running the image tests. We recommend runnnig `npm run watch` or `npm run start-test_dashboard` in a separate tab to ensure that the most up-to-date code is tested. 
+As an alternative to running all image comparison tests at once, you can provide
+a [glob](https://github.com/isaacs/node-glob) as argument to target one or multiple test mocks found in
+[`test/image/mocks`][mocks].
+For example,
 
-### Step 2b: Make a new or update an existing baseline image
+```bash
+# Run one test (e.g. the 'contour_nolines' test):
+$ npm run test-image -- contour_nolines
 
-Inside your `plotly.js` directory, run
+# Run all gl3d image test in batch:
+$ npm run test-image -- gl3d_*
+```
+
+Developers on weak hardware might encounter batch timeout issue. These are most
+common when generated WebGL-based graphs. In this case, running the image
+comparison tests in queue (i.e. with no concurrency) is recommended:
 
 ```bash
-npm run baseline -- mock.json
+# Run all gl3d image test in queue:
+$ npm run test-image -- gl3d_* --queue
+```
+
+##### B: Run image export tests
+
+Image export tests check that image export works for formats other than png.
+
+To run the image export tests, in your `plotly.js` directory:
+
+```bash
+npm run test-export
+
+# or
+npm run test-export -- <glob>
+```
+
+##### C: Generate or update existing baseline image
+
+To generate a new baseline image, add a new mock file in
+[`test/image/mocks`][mocks]. Note that mock file needs to be a valid JSON and
+have both a "data" and a `"layout"` field. Then, in your plotly.js directory,
+run:
+
+```bash
+npm run baseline -- <name-of-mock>
+```
+
+which generates a baseline png image in [`test/image/baselines`][baselines].
+
+To update existing baseline image(s), run
+
+```bash
+npm run basline -- <glob-of-mocks-to-update>
 ```
 
-where `mock.json` is the name of a `{"data": [], "layout": {}}` json file found in [`test/image/mocks/`](https://github.com/plotly/plotly.js/tree/master/test/image/mocks). The `"data"` and `"layout"` field are passed to `Plotly.plot` to produce an image saved in [`test/image/baslines`](https://github.com/plotly/plotly.js/tree/master/test/image/baselines).
 
 ### Step 3: Stop your testing container
 
@@ -74,7 +135,7 @@ Mac and Windows user should also kill their docker-machine (named `default`) onc
 docker-machine kill default
 ```
 
-### Docker tricks 
+### Docker tricks
 
 ##### SSH into docker container
 
@@ -145,3 +206,6 @@ docker-machine kill default
 ```
 
 For more comprehensive information about docker, please refer to the [docker docs](http://docs.docker.com/).
+
+[mocks]: https://github.com/plotly/plotly.js/tree/master/test/image/mocks
+[baselines]: https://github.com/plotly/plotly.js/tree/master/test/image/baselines

test/image/assets/get_image_paths.js

@@ -0,0 +1,29 @@
+var path = require('path');
+var constants = require('../../../tasks/util/constants');
+
+var DEFAULT_FORMAT = 'png';
+
+
+/**
+ * Return paths to baseline, test-image and diff images for a given mock name.
+ *
+ * @param {string} mockName
+ * @param {string} format
+ * @return {object}
+ *      baseline
+ *      test
+ *      diff
+ */
+module.exports = function getImagePaths(mockName, format) {
+    format = format || DEFAULT_FORMAT;
+
+    return {
+        baseline: join(constants.pathToTestImageBaselines, mockName, format),
+        test: join(constants.pathToTestImages, mockName, format),
+        diff: join(constants.pathToTestImagesDiff, mockName, format)
+    };
+};
+
+function join(basePath, mockName, format) {
+    return path.join(basePath, mockName) + '.' + format;
+}

test/image/assets/get_image_request_options.js

@@ -0,0 +1,32 @@
+var path = require('path');
+var constants = require('../../../tasks/util/constants');
+
+var DEFAULT_URL = 'http://localhost:9010/';
+var DEFAULT_FORMAT = 'png';
+var DEFAULT_SCALE = 1;
+
+/**
+ *  Return the image server request options for a given mock (and specs)
+ *
+ *  @param {object} specs
+ *      mockName : name of json mock to plot
+ *      format (optional): format of generated image
+ *      scale (optional): scale of generated image
+ *      url (optional): URL of image server
+ */
+module.exports = function getRequestOpts(specs) {
+    var pathToMock = path.join(constants.pathToTestImageMocks, specs.mockName) + '.json';
+    var figure = require(pathToMock);
+
+    var body = {
+        figure: figure,
+        format: specs.format || DEFAULT_FORMAT,
+        scale: specs.scale || DEFAULT_SCALE
+    };
+
+    return {
+        method: 'POST',
+        url: specs.url || DEFAULT_URL,
+        body: JSON.stringify(body)
+    };
+};

test/image/assets/get_mock_list.js

@@ -0,0 +1,29 @@
+var path = require('path');
+var glob = require('glob');
+
+var constants = require('../../../tasks/util/constants');
+
+
+/**
+ *  Return array of mock name corresponding to input glob pattern
+ *
+ *  @param {string} pattern
+ *  @return {array}
+ */
+module.exports = function getMocks(pattern) {
+    // defaults to 'all'
+    pattern = pattern || '*';
+
+    // defaults to '.json' ext is none is provided
+    if(path.extname(pattern) === '') pattern += '.json';
+
+    var patternFull = constants.pathToTestImageMocks + '/' + pattern;
+    var matches = glob.sync(patternFull);
+
+    // return only the mock name (not a full path, no ext)
+    var mockNames = matches.map(function(match) {
+        return path.basename(match).split('.')[0];
+    });
+
+    return mockNames;
+};