feat: Upgrade Nightwatch to v1.2 and update bundled config and generated tests (vuejs#4541)

* feat: Upgraded Nightwatch to version 1.2; updated distributed config; added new cli flags: --headless, --parallel...; added support for running chromedriver and geckodriver standalone; added unit tests

* docs: updated nightwatch plugin readme

* feat: add chromedriver and geckodriver as peer dependencies

* updated readme and driver depedencies

* updated optional dependencies

* fixed failing tests

* updated generated tests

* updated tests

Author: beatfactor

packages/@vue/cli-plugin-e2e-nightwatch/README.md

@@ -6,32 +6,138 @@
 
 - **`vue-cli-service test:e2e`**
 
-  run e2e tests with [NightwatchJS](http://nightwatchjs.org).
+  Run end-to-end tests with [Nightwatch.js](https://nightwatchjs.org).
 
   Options:
 
   ```
-  --url        run e2e tests against given url instead of auto-starting dev server
-  --config     use custom nightwatch config file (overrides internals)
-  -e, --env    specify comma-delimited browser envs to run in (default: chrome)
-  -t, --test   specify a test to run by name
-  -f, --filter glob to filter tests by filename
+  --url                 run the tests against given url instead of auto-starting dev server
+  --config              use custom nightwatch config file (overrides internals)
+  --headless            use chrome or firefox in headless mode
+  --parallel            enable parallel mode via test workers (only available in chromedriver)
+  --use-selenium        use Selenium standalone server instead of chromedriver or geckodriver
+  -e, --env             specify comma-delimited browser envs to run in (default: chrome)
+  -t, --test            specify a test to run by name
+  -f, --filter          glob to filter tests by filename
   ```
 
-  > Note: this plugin currently uses Nightwatch v0.9.x. We are waiting for Nightwatch 1.0 to stabilize before upgrading.
+  Additionally, all [Nightwatch CLI options](https://nightwatchjs.org/guide/running-tests/#command-line-options) are also supported. 
+  E.g.: `--verbose`, `--retries` etc.
+  
 
-  Additionally, [all Nightwatch CLI options are also supported](https://nightwatchjs.org/guide#command-line-options).
+## Project Structure
 
-## Configuration
+The following structure will be generated when installing this plugin. There are examples for most testing concepts in Nightwatch available.    
+
+```
+tests/e2e/
+  ├── custom-assertions/
+  |   └── elementCount.js
+  ├── custom-commands/
+  |   ├── customExecute.js
+  |   ├── openHomepage.js
+  |   └── openHomepageClass.js
+  ├── page-objects/
+  |   └── homepage.js
+  ├── specs/
+  |   ├── test.js
+  |   └── test-with-pageobjects.js
+  └── globals.js
+```
 
-We've pre-configured Nightwatch to run with Chrome by default. If you wish to run e2e tests in additional browsers, you will need to add a `nightwatch.config.js` or `nightwatch.json` in your project root to configure additional browsers. The config will be merged into the [internal Nightwatch config](https://github.com/vuejs/vue-cli/blob/dev/packages/%40vue/cli-plugin-e2e-nightwatch/nightwatch.config.js).
+#### `specs`
+The main location where tests are located. Can contain sub-folders which can be targeted during the run using the `--group` argument. [More info](https://nightwatchjs.org/guide/running-tests/#test-groups).
 
-Alternatively, you can completely replace the internal config with a custom config file using the `--config` option.
+#### `custom-assertions`
+Files located here are loaded automatically by Nightwatch and placed onto the `.assert` and `.verify` api namespaces to extend the Nightwatch built-in assertions. See [writing custom assertions](https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-assertions) for details.
+
+#### `custom-commands`
+Files located here are loaded automatically by Nightwatch and placed onto the main `browser` api object to extend the built-in Nightwatch commands. See [writing custom commands](https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands) for details.
 
-Consult Nightwatch docs for [configuration options](http://nightwatchjs.org/gettingstarted#settings-file) and how to [setup browser drivers](http://nightwatchjs.org/gettingstarted#browser-drivers-setup).
+#### `page objects`
+Working with page objects is a popular methodology in end-to-end UI testing. Files placed in this folder are automatically loaded onto the `.page` api namespace, with the name of the file being the name of the page object. See [working with page objects](https://nightwatchjs.org/guide/working-with-page-objects/) section for details.  
+
+#### `globals.js`
+The external globals file which can hold global properties or hooks. See [test globals](https://nightwatchjs.org/gettingstarted/configuration/#test-globals) section.
 
 ## Installing in an Already Created Project
 
 ``` sh
 vue add e2e-nightwatch
 ```
+
+## Configuration
+
+We've pre-configured Nightwatch to run with Chrome by default. Firefox is also available via `--env firefox`. If you wish to run end-to-end tests in additional browsers (e.g. Safari, Microsoft Edge), you will need to add a `nightwatch.conf.js` or `nightwatch.json` in your project root to configure additional browsers. The config will be merged into the [internal Nightwatch config](https://github.com/vuejs/vue-cli/blob/dev/packages/%40vue/cli-plugin-e2e-nightwatch/nightwatch.config.js).
+
+Alternatively, you can completely replace the internal config with a custom config file using the `--config` option.
+
+Consult Nightwatch docs for [configuration options](https://nightwatchjs.org/gettingstarted/configuration/) and how to [setup browser drivers](http://nightwatchjs.org/gettingstarted#browser-drivers-setup).
+
+## Running Tests
+
+By default, all tests inside the `specs` folder will be run using Chrome. If you'd like to run end-to-end tests against Chrome (or Firefox) in headless mode, simply pass the `--headless` argument.
+
+```sh
+$ vue-cli-service test:e2e
+```
+
+**Running a single test**
+
+To run a single test supply the filename path. E.g.:
+
+```sh
+$ vue-cli-service test:e2e tests/e2e/specs/test.js
+```
+
+**Skip Dev server auto-start**
+
+If the development server is already running and you want to skip starting it automatically, pass the `--url` argument:
+
+```sh
+$ vue-cli-service test:e2e --url http://localhost:8080/
+```
+
+**Running in Firefox**
+
+Support for running tests in Firefox is also available by default. Simply run the following (optionally add `--headless` to run Firefox in headless mode):
+
+```sh
+$ vue-cli-service test:e2e --env firefox [--headless]
+```
+
+**Running in Firefox and Chrome simultaneously**
+
+You can also run the tests simultaneously in both browsers by supplying both test environments separated by a comma (",") - no spaces.
+
+```sh
+$ vue-cli-service test:e2e --env firefox,chrome [--headless]
+```
+
+**Running Tests in Parallel**
+
+For a significantly faster test run, you can enable parallel test running when there are several test suites. Concurrency is performed at the file level and is distributed automatically per available CPU core.  
+
+For now, this is only available in Chromedriver. Read more about [parallel running](https://nightwatchjs.org/guide/running-tests/#parallel-running) in the Nightwatch docs.
+
+```sh
+$ vue-cli-service test:e2e --parallel
+```
+
+**Running with Selenium**
+
+Since `v4`, the Selenium standalone server is not included anymore in this plugin and in most cases running with Selenium is not required since Nightwatch v1.0. 
+
+It is still possible to use the Selenium server, by following these steps:
+
+__1.__ Install `selenium-server` NPM package:
+
+  ```sh
+  $ npm install selenium-server --save-dev
+  ```
+  
+__2.__ Run with `--use-selenium` cli argument:
+
+  ```sh
+  $ vue-cli-service test:e2e --use-selenium
+  ```

packages/@vue/cli-plugin-e2e-nightwatch/__tests__/lib/globals-gecko.js

@@ -0,0 +1,10 @@
+/**
+ * This file is copied during the firefox test inside the project folder and used to inspect the results
+ */
+const fs = require('fs')
+
+module.exports = {
+  reporter (results, cb) {
+    fs.writeFile('test_results_gecko.json', JSON.stringify(results), cb)
+  }
+}

packages/@vue/cli-plugin-e2e-nightwatch/__tests__/lib/globals-generated.js

@@ -0,0 +1,14 @@
+/**
+ * This file is copied during the test inside the project folder and used to inpsect the results
+ */
+const fs = require('fs')
+
+module.exports = {
+  afterEach (browser, cb) {
+    fs.writeFile('test_settings.json', JSON.stringify(browser.options), cb)
+  },
+
+  reporter (results, cb) {
+    fs.writeFile('test_results.json', JSON.stringify(results), cb)
+  }
+}

packages/@vue/cli-plugin-e2e-nightwatch/__tests__/lib/nightwatch.conf.js

@@ -0,0 +1,6 @@
+/**
+ * This file is copied during the test inside the project folder
+ */
+module.exports = {
+  globals_path: './tests/e2e/globals-gecko.js'
+}

packages/@vue/cli-plugin-e2e-nightwatch/__tests__/nightwatchPlugin.spec.js

@@ -1,13 +1,100 @@
-jest.setTimeout(40000)
+jest.setTimeout(process.env.APPVEYOR ? 300000 : 120000)
 
+const fs = require('fs-extra')
+const path = require('path')
 const create = require('@vue/cli-test-utils/createTestProject')
 
-test('should work', async () => {
-  const project = await create('e2e-nightwatch', {
-    plugins: {
-      '@vue/cli-plugin-babel': {},
-      '@vue/cli-plugin-e2e-nightwatch': {}
+describe('nightwatch e2e plugin', () => {
+  let project
+
+  beforeAll(async () => {
+    project = await create('e2e-nightwatch', {
+      plugins: {
+        '@vue/cli-plugin-babel': {},
+        '@vue/cli-plugin-e2e-nightwatch': {}
+      }
+    })
+
+    await fs.copy(path.join(__dirname, './lib/globals-generated.js'),
+      path.join(project.dir, 'tests/e2e/globals-generated.js'))
+
+    const config = {
+      globals_path: './tests/e2e/globals-generated.js'
     }
+    await project.write('nightwatch.json', JSON.stringify(config))
+  })
+
+  test('should run all tests successfully', async () => {
+    await project.run(`vue-cli-service test:e2e --headless`)
+    let results = await project.read('test_results.json')
+    results = JSON.parse(results)
+    expect(Object.keys(results.modules)).toEqual([
+      'test-with-pageobjects',
+      'test'
+    ])
+  })
+
+  test('should run single test with custom nightwatch.json', async () => {
+    await project.run(`vue-cli-service test:e2e --headless -t tests/e2e/specs/test.js`)
+    let results = await project.read('test_results.json')
+    results = JSON.parse(results)
+    expect(Object.keys(results.modules)).toEqual([
+      'test'
+    ])
+  })
+
+  test('should run single test with custom nightwatch.json and selenium server', async () => {
+    await project.run(`vue-cli-service test:e2e --headless --with-selenium -t tests/e2e/specs/test.js`)
+    let results = await project.read('test_results.json')
+    results = JSON.parse(results)
+
+    let testSettings = await project.read('test_settings.json')
+    testSettings = JSON.parse(testSettings)
+
+    expect(testSettings).toHaveProperty('selenium')
+    expect(testSettings.selenium.start_process).toStrictEqual(true)
+    expect(testSettings.selenium).toHaveProperty('cli_args')
+    expect(Object.keys(results.modules)).toEqual([
+      'test'
+    ])
+  })
+
+  test('should run tests in parallel', async () => {
+    await project.run(`vue-cli-service test:e2e --headless --parallel`)
+    let results = await project.read('test_results.json')
+    results = JSON.parse(results)
+
+    let testSettings = await project.read('test_settings.json')
+    testSettings = JSON.parse(testSettings)
+
+    expect(testSettings.parallel_mode).toStrictEqual(true)
+    expect(testSettings.test_workers).toStrictEqual(true)
+
+    expect(Object.keys(results.modules).sort()).toEqual([
+      'test', 'test-with-pageobjects'
+    ])
+  })
+
+  // This test requires Firefox to be installed
+  const testFn = process.env.APPVEYOR ? test.skip : test
+  testFn('should run single test with custom nightwatch.conf.js in firefox', async () => {
+    // nightwatch.conf.js take priority over nightwatch.json
+    const copyConfig = fs.copy(path.join(__dirname, './lib/nightwatch.conf.js'),
+      path.join(project.dir, 'nightwatch.conf.js'))
+
+    const copyGlobals = fs.copy(path.join(__dirname, './lib/globals-gecko.js'),
+      path.join(project.dir, 'tests/e2e/globals-gecko.js'))
+
+    await Promise.all([copyConfig, copyGlobals])
+
+    await project.run(`vue-cli-service test:e2e --headless --env firefox -t tests/e2e/specs/test.js`)
+    let results = await project.read('test_results_gecko.json')
+    results = JSON.parse(results)
+
+    expect(Object.keys(results.modules)).toEqual([
+      'test'
+    ])
+    expect(results.modules.test).toHaveProperty('reportPrefix')
+    expect(results.modules.test.reportPrefix).toMatch(/^FIREFOX_.+/)
   })
-  await project.run(`vue-cli-service test:e2e`)
 })

packages/@vue/cli-plugin-e2e-nightwatch/generator/index.js

@@ -8,7 +8,8 @@ module.exports = api => {
       'test:e2e': 'vue-cli-service test:e2e'
     },
     devDependencies: {
-      chromedriver: '^74.0.0'
+      chromedriver: '^76.0.1',
+      geckodriver: '^1.16.2'
     }
   })
 }

packages/@vue/cli-plugin-e2e-nightwatch/generator/template/tests/e2e/custom-assertions/elementCount.js

@@ -1,13 +1,27 @@
-// A custom Nightwatch assertion.
-// The assertion name is the filename.
-// Example usage:
-//
-//   browser.assert.elementCount(selector, count)
-//
-// For more information on custom assertions see:
-// http://nightwatchjs.org/guide#writing-custom-assertions
+/**
+ * A custom Nightwatch assertion. The assertion name is the filename.
+ *
+ * Example usage:
+ *   browser.assert.elementCount(selector, count)
+ *
+ * For more information on custom assertions see:
+ *   https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-assertions
+ *
+ *
+ * @param {string|object} selectorOrObject
+ * @param {number} count
+ */
+
+exports.assertion = function elementCount (selectorOrObject, count) {
+  let selector;
+
+  // when called from a page object element or section
+  if (typeof selectorOrObject == 'object' && selectorOrObject.selector) {
+    selector = selectorOrObject.selector
+  } else {
+    selector = selectorOrObject
+  }
 
-exports.assertion = function elementCount (selector, count) {
   this.message = `Testing if element <${selector}> has count: ${count}`
   this.expected = count
   this.pass = val => val === count

packages/@vue/cli-plugin-e2e-nightwatch/generator/template/tests/e2e/custom-commands/customExecute.js

@@ -0,0 +1,37 @@
+/**
+ * A very basic Nightwatch custom command. The command name is the filename and the
+ *  exported "command" function is the command.
+ *
+ * Example usage:
+ *   browser.customExecute(function() {
+ *     console.log('Hello from the browser window')
+ *   });
+ *
+ * For more information on writing custom commands see:
+ *   https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands
+ *
+ * @param {*} data
+ */
+exports.command = function(data) {
+  // Other Nightwatch commands are available via "this"
+
+  // .execute() inject a snippet of JavaScript into the page for execution.
+  //  the executed script is assumed to be synchronous.
+  //
+  // See https://nightwatchjs.org/api/execute.html for more info.
+  //
+  this.execute(
+    // The function argument is converted to a string and sent to the browser
+    function(argData) {return argData;},
+
+    // The arguments for the function to be sent to the browser are specified in this array
+    [data],
+
+    function(result) {
+      // The "result" object contains the result from the what we have sent back from the browser window
+      console.log('custom execute result:', result.value)
+    }
+  );
+
+  return this;
+};

packages/@vue/cli-plugin-e2e-nightwatch/generator/template/tests/e2e/custom-commands/openHomepage.js

@@ -0,0 +1,22 @@
+/**
+ * A basic Nightwatch custom command which demonstrates usage of ES6 async/await instead of using callbacks.
+ *  The command name is the filename and the exported "command" function is the command.
+ *
+ * Example usage:
+ *   browser.openHomepage();
+ *
+ * For more information on writing custom commands see:
+ *   https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands
+ *
+ */
+module.exports = {
+  command: async function () {
+    // Other Nightwatch commands are available via "this"
+    // .init() simply calls .url() command with the value of the "launch_url" setting
+    this.init();
+    this.waitForElementVisible('#app');
+
+    const result = await this.elements('css selector', '#app ul');
+    this.assert.strictEqual(result.value.length, 3);
+  }
+};