Image tests using orca

[etpinard]

Supersedes #1972

In brief, this PR brings:

• electron based image testing (bye bye nw.js)
• no http requests are made, the runner app reads the JSON mocks directly from your file system, that means no need to boot a server, ping the server, wait until it responds before each test run
• a pure node-js image comparison routine using Mapbox's pixelmatch and pngjs. That means bye-bye gm and graphicsmagick
• much faster test runs. On my machine, npm run test-image runs in under 1:30 minute (N.B. that's over 400 mocks folks)
• an official way to run and or debug the image tests locally outside of docker using a LOCAL_ELECTRON environment variable

Now, a few things have changed since my first attempt back in the fall of 2017:

• the image-exporter project has been renamed orca 🐳
• we figured out how to workaround the hangs-up in docker (ref Work around Chrome SHM issue (causes hang) orca#50)

plotly.js/tasks/docker.js

Lines 26 to 31 in 93009aa

	// set shared memory size as a workaround
	// - https://github.com/plotly/orca/pull/50
	// - https://bugs.chromium.org/p/chromium/issues/detail?id=736452
	// - https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md#tips
	'--shm-size=2g',
	'--volume /dev/shm:/dev/shm',

• our test commands appear to work (i.e. not hang) on CircleCI.

A few things that are still not perfect:

• npm run test-image doesn't behave consistently locally (i.e. on my laptop) vs on CircleCI. That is, we currently can't use the same baselines to make npm run test-image pass locally and on CI. Perhaps, we could solve this by reducing the pixel-comparison threshold. Or maybe something is wrong (e.g. a race condition) in the pixel-comparison script.
• The Droid font isn't looking great:

cc @alexcjohnson @dy @nicolaskruchten

[etpinard]

Even with a 0.1 pixel-comparison threshold, I can't get npm run test-image to pass locally and on CI with the same baselines.

Can anyone try on their machine:

• pull down this branch
• npm i
• npm run docker -- pull
• npm run docker -- run
• npm run test-image

[alexcjohnson]

I'm really not happy about the idea of anything less than perfect matching. There have been quite a few cases of very small changes in the image output being the only evidence of a real bug. I'm open to discussing it though.

[etpinard]

Something is really off on CI. It can't even generate the same pixels run to tun. We get

off fcdb9b9 which has ci-generated baselines.

[etpinard]

Writing down some observations I made during my last few attempts:

---

The main question we're trying to answer is: why is CI generating different pixels than locally?

I first tried to isolate the problem further by bringing back gm as our image diff tools (as opposed to pixelmatch, like on this branch). This is in commits f7c388d and b2ed37d using orca branch plotly/orca@single-entry...graphicsmagick. Results were positive. Tests were passing on CI and locally using the same baselines with a tolerance of 0.001. Tests did not pass when lowering the tolerance to 0.0001 as even less so when lowering it our current value on master of 1e-6.

For those interested in why gm and pixelmatch don't generate the same diff, see the pixelmatch algo here. For pixelmatch, one bad pixel is enough to create a diff whereas gm (as far as I know) uses statistics on the whole set of pixels (see docs).

Then, I tried tweaking the xvfb-run -screen and the docker run -shm-size values to no avail. This has no impact on the generated images.

Then, I tried to find out if a tolerance of 0.001 is enough to catch bugs. See my experiment in commit 2f0df5e where I tweaked a few mocks slightly. The results were pretty clear, a tolerance of 0.001 isn't enough. Only two out of eight wrong mocks generated diffs.

---

So, we still don't know why the same docker container generates different images locally vs on ci. The proposed workaround (of switching back gm with a 0.001 tolerance) isn't feasible.

So what's next?

• Dig into how CircleCI implements docker images?
• Maybe the "base" docker image has a few libs installed that can modify images are generated?
• Notice that the old nw.js-based imagetest container installed a few more libs than we currently do for orca. Maybe that could be the answer?

[etpinard]

Notice that the old nw.js-based imagetest container installed a few more libs than we currently do for orca. Maybe that could be the answer?

This doesn't appear to be the answer using plotly/orca@single-entry...graphicsmagick

---

Given up on this until next week. 😡

In the meantime, if someone could try this branch out (as described in #2615 (comment)), this would help me a lot. Thanks!

[alexcjohnson]

in 265a882 I changed id --user to id -u (and added a tighter check on the docker run -> docker start fallback so I could see the error message with id), now npm run docker -- run works on my mac. But I still didn't get npm run test-image to run. It failed with:

Running image comparison test using build/plotly.js from 2018-5-8 15:39:37 

A JavaScript error occurred in the main process
Uncaught Exception:
Error: Failed to get 'appData' path
    at Object.<anonymous> (/var/www/image-exporter/node_modules/electron/dist/resources/electron.asar/browser/init.js:149:39)
    at Object.<anonymous> (/var/www/image-exporter/node_modules/electron/dist/resources/electron.asar/browser/init.js:173:3)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:167:16)
    at bootstrap_node.js:589:3

Which people seem to indicate is a permissions issue, but AFAICT id -u is doing the right thing...

[etpinard]

@alexcjohnson what if you try

npm run docker -- run
docker exec -ti orcabed /bin/bash

# ... stepping inside docker container

cd /var/www/image-exporter/plotly.js
CIRCLECI=1 npm run test-image

[alexcjohnson]

what if you try ...

same error Failed to get 'appData' path

[etpinard]

Small update: I tried bumping the electron version (to v2.0.2), but to no avail we still get different pixels to render on CI vs my laptop.

[etpinard]

Next stop (unless someone has a better idea), try travis ci: https://docs.travis-ci.com/user/docker/

[etpinard]

Time to close this thing.

We're still a long way off from using orca for plotly.js image tests, but with @antoinerg 's 💪 work in #3237 we should be able to find a way to get this done in a not-so-distance future.