breaking: rename consistent-output rule to require-test-output

Changes the default rule behavior to always require a test `output` assertion. The old default  behavior of `consistent` (only require `output` assertions if some tests have it) is still provided as an available option.

* [As of ESLint 7](https://eslint.org/docs/user-guide/migrating-to-7.0.0#additional-validation-added-to-the-ruletester-class), the `output` property is required for test cases that provide an autofix
* And even when test cases don't autofix, it's still best practice to assert that there's no autofix with `output: null`

Author: bmish

README.md

@@ -47,7 +47,6 @@ Then configure the rules you want to use under the rules section.
 <!-- __BEGIN AUTOGENERATED TABLE__ -->
 Name | ✔️ | 🛠 | 💡 | Description
 ----- | ----- | ----- | ----- | -----
-[consistent-output](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/consistent-output.md) |  |  |  | enforce consistent use of output assertions in rule tests
 [fixer-return](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/fixer-return.md) | ✔️ |  |  | require fixer function to return a fix
 [meta-property-ordering](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/meta-property-ordering.md) |  | 🛠 |  | enforce the order of meta properties
 [no-deprecated-context-methods](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/no-deprecated-context-methods.md) |  | 🛠 |  | disallow usage of deprecated methods on rule context objects
@@ -68,6 +67,7 @@ Name | ✔️ | 🛠 | 💡 | Description
 [require-meta-has-suggestions](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-meta-has-suggestions.md) |  |  |  | require suggestable rules to implement a `meta.hasSuggestions` property
 [require-meta-schema](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-meta-schema.md) |  | 🛠 |  | require rules to implement a meta.schema property
 [require-meta-type](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-meta-type.md) |  |  |  | require rules to implement a meta.type property
+[require-test-output](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-test-output.md) |  |  |  | enforce use of `output` assertions in rule tests
 [test-case-property-ordering](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/test-case-property-ordering.md) |  | 🛠 |  | require the properties of a test case to be placed in a consistent order
 [test-case-shorthand-strings](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/test-case-shorthand-strings.md) |  | 🛠 |  | enforce consistent usage of shorthand strings for test cases with no options
 <!-- __END AUTOGENERATED TABLE__ -->

docs/rules/consistent-output.md renamed to docs/rules/require-test-output.md

@@ -1,19 +1,19 @@
-# Enforce consistent use of output assertions in rule tests (consistent-output)
+# Enforce use of `output` assertions in rule tests (require-test-output)
 
-When writing tests for a fixable rule with `RuleTester`, you can assert the autofix output of your test cases. However, it can be easy to forget to assert the output of a particular test case.
+When writing tests for a fixable rule with `RuleTester`, you can assert the autofix `output` of your test cases.
 
 [As of ESLint 7](https://eslint.org/docs/user-guide/migrating-to-7.0.0#additional-validation-added-to-the-ruletester-class), test cases that trigger an autofix are required to provide the `output` property.
 
-Even test that do not trigger an autofix can benefit from asserting that they have no autofix using `output: null`.
+However, it can be easy to forget to assert the output of a particular test case, leading to the autofix being untested. And even tests that do not trigger an autofix can benefit from asserting that they have no autofix using `output: null`.
 
 ## Rule Details
 
-This rule aims to ensure that if any invalid test cases have output assertions, then all test cases have output assertions.
+This rule ensures all invalid test cases have `output` assertions.
 
 Examples of **incorrect** code for this rule:
 
 ```js
-/* eslint eslint-plugin/consistent-output: error */
+/* eslint eslint-plugin/require-test-output: error */
 
 new RuleTester().run('example-rule', rule, {
   valid: [],
@@ -34,7 +34,7 @@ new RuleTester().run('example-rule', rule, {
 Examples of **correct** code for this rule:
 
 ```js
-/* eslint eslint-plugin/consistent-output: error */
+/* eslint eslint-plugin/require-test-output: error */
 
 new RuleTester().run('example-rule', rule, {
   valid: [],
@@ -60,14 +60,9 @@ new RuleTester().run('example-rule', rule, {
 
 ## Options
 
-This rule takes an optional string enum option with one of the following values:
+This rule takes an optional object containing:
 
-* `"consistent"` - (default) if any invalid test cases have output assertions, then all invalid test cases must have output assertions
-* `"always"` - always require invalid test cases to have output assertions
-
-## When Not To Use It
-
-If you're not writing fixable rules, or you want to write test cases without output assertions, do not enable this rule.
+* `boolean` -- `consistent` -- only require `output` assertions when some test cases have them (default `false`)
 
 ## Further Reading
 

lib/rules/consistent-output.js renamed to lib/rules/require-test-output.js

@@ -1,5 +1,5 @@
 /**
- * @fileoverview Enforce consistent use of output assertions in rule tests
+ * @fileoverview Enforce use of `output` assertions in rule tests
  * @author Teddy Katz
  */
 
@@ -15,24 +15,34 @@ module.exports = {
   meta: {
     type: 'suggestion',
     docs: {
-      description: 'enforce consistent use of output assertions in rule tests',
+      description: 'enforce use of `output` assertions in rule tests',
       category: 'Tests',
       recommended: false,
     },
     fixable: null, // or "code" or "whitespace"
     schema: [
       {
-        type: 'string',
-        enum: ['always', 'consistent'],
+        type: 'object',
+        properties: {
+          consistent: {
+            type: 'boolean',
+            default: false,
+          },
+        },
+        additionalProperties: false,
       },
     ],
+    messages: {
+      missingOutput: 'This test case should have an `output` assertion.',
+    },
   },
 
   create (context) {
     // ----------------------------------------------------------------------
     // Public
     // ----------------------------------------------------------------------
-    const always = context.options[0] && context.options[0] === 'always';
+    const consistent = context.options[0] && context.options[0].consistent;
+    const always = !consistent;
 
     return {
       Program (ast) {
@@ -48,7 +58,7 @@ module.exports = {
             casesWithoutOutput.forEach(testCase => {
               context.report({
                 node: testCase,
-                message: 'This test case should have an output assertion.',
+                messageId: 'missingOutput',
               });
             });
           }

tests/lib/rules/consistent-output.js

@@ -0,0 +1,117 @@
+/**
+ * @fileoverview Enforce use of `output` assertions in rule tests
+ * @author Teddy Katz
+ */
+
+'use strict';
+
+// ------------------------------------------------------------------------------
+// Requirements
+// ------------------------------------------------------------------------------
+
+const rule = require('../../../lib/rules/require-test-output');
+const RuleTester = require('eslint').RuleTester;
+
+const ERROR = { messageId: 'missingOutput', type: 'ObjectExpression' };
+
+// ------------------------------------------------------------------------------
+// Tests
+// ------------------------------------------------------------------------------
+
+const ruleTester = new RuleTester();
+ruleTester.run('require-test-output', rule, {
+  valid: [
+    {
+      // Explicit option of `consistent` (no output in any tests).
+      code: `
+        new RuleTester().run('foo', bar, {
+          valid: [],
+          invalid: [
+            { code: 'foo', errors: ['bar'] },
+            { code: 'baz', errors: ['qux'] }
+          ]
+        });
+      `,
+      options: [{ consistent: true }],
+    },
+    {
+      // Explicit option of `consistent` (output in all tests).
+      code: `
+        new RuleTester().run('foo', bar, {
+          valid: [],
+          invalid: [
+            { code: 'foo', output: 'baz', errors: ['bar'] },
+            { code: 'foo', output: 'qux', errors: ['bar'] },
+          ]
+        });
+      `,
+      options: [{ consistent: true }],
+    },
+    {
+      // Explicitly turning off the `consistent` option (results in "always" behavior).
+      code: `
+        new RuleTester().run('foo', bar, {
+          valid: [],
+          invalid: [
+            { code: 'foo', output: 'baz', errors: ['bar'] },
+            { code: 'foo', output: 'qux', errors: ['bar'] },
+            { code: 'foo', output: null, errors: ['bar'] },
+          ]
+        });
+      `,
+      options: [{ consistent: false }],
+    },
+    // When defaulting to the "always" behavior.
+    `
+      new RuleTester().run('foo', bar, {
+        valid: [],
+        invalid: [
+          { code: 'foo', output: 'baz', errors: ['bar'] },
+        ]
+      });
+    `,
+  ],
+
+  invalid: [
+    {
+      // Explicit option of `consistent`.
+      code: `
+        new RuleTester().run('foo', bar, {
+          valid: [],
+          invalid: [
+            { code: 'foo', output: 'baz', errors: ['bar'] },
+            { code: 'foo', errors: ['bar'] },
+            { code: 'foo bar', errors: ['bar'] },
+          ]
+        });
+      `,
+      options: [{ consistent: true }],
+      errors: [ERROR, ERROR],
+    },
+    {
+      // Explicitly turning off the `consistent` option (results in "always" behavior).
+      code: `
+        new RuleTester().run('foo', bar, {
+          valid: [],
+          invalid: [
+            { code: 'foo', errors: ['bar'] },
+          ]
+        });
+      `,
+      options: [{ consistent: false }],
+      errors: [ERROR],
+    },
+    {
+      // When defaulting to the "always" behavior.
+      code: `
+        new RuleTester().run('foo', bar, {
+          valid: [],
+          invalid: [
+            { code: 'foo', errors: ['bar'] },
+          ]
+        });
+      `,
+      errors: [ERROR],
+    },
+  ],
+});