breaking: rename consistent-output rule to require-test-output

Changes the default rule behavior to always require a test `output` assertion. The old default  behavior of `consistent` (only require `output` assertions if some tests have it) is still provided as an available option.

* [As of ESLint 7](https://eslint.org/docs/user-guide/migrating-to-7.0.0#additional-validation-added-to-the-ruletester-class), the `output` property is required for test cases that provide an autofix
* And even when test cases don't autofix, it's still best practice to assert that there's no autofix with `output: null`

Author: bmish

README.md

@@ -47,7 +47,6 @@ Then configure the rules you want to use under the rules section.
 <!-- __BEGIN AUTOGENERATED TABLE__ -->
 Name | ✔️ | 🛠 | 💡 | Description
 ----- | ----- | ----- | ----- | -----
-[consistent-output](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/consistent-output.md) |  |  |  | enforce consistent use of output assertions in rule tests
 [fixer-return](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/fixer-return.md) | ✔️ |  |  | require fixer function to return a fix
 [meta-property-ordering](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/meta-property-ordering.md) |  | 🛠 |  | enforce the order of meta properties
 [no-deprecated-context-methods](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/no-deprecated-context-methods.md) |  | 🛠 |  | disallow usage of deprecated methods on rule context objects
@@ -68,6 +67,7 @@ Name | ✔️ | 🛠 | 💡 | Description
 [require-meta-has-suggestions](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-meta-has-suggestions.md) |  |  |  | require suggestable rules to implement a `meta.hasSuggestions` property
 [require-meta-schema](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-meta-schema.md) |  | 🛠 |  | require rules to implement a meta.schema property
 [require-meta-type](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-meta-type.md) |  |  |  | require rules to implement a meta.type property
+[require-test-output](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/require-test-output.md) |  |  |  | enforce use of `output` assertions in rule tests
 [test-case-property-ordering](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/test-case-property-ordering.md) |  | 🛠 |  | require the properties of a test case to be placed in a consistent order
 [test-case-shorthand-strings](https://github.com/not-an-aardvark/eslint-plugin-eslint-plugin/blob/master/docs/rules/test-case-shorthand-strings.md) |  | 🛠 |  | enforce consistent usage of shorthand strings for test cases with no options
 <!-- __END AUTOGENERATED TABLE__ -->

docs/rules/consistent-output.md renamed to docs/rules/require-test-output.md

@@ -1,19 +1,19 @@
-# Enforce consistent use of output assertions in rule tests (consistent-output)
+# Enforce use of `output` assertions in rule tests (require-test-output)
 
-When writing tests for a fixable rule with `RuleTester`, you can assert the autofix output of your test cases. However, it can be easy to forget to assert the output of a particular test case.
+When writing tests for a fixable rule with `RuleTester`, you can assert the autofix `output` of your test cases.
 
 [As of ESLint 7](https://eslint.org/docs/user-guide/migrating-to-7.0.0#additional-validation-added-to-the-ruletester-class), test cases that trigger an autofix are required to provide the `output` property.
 
-Even test that do not trigger an autofix can benefit from asserting that they have no autofix using `output: null`.
+However, it can be easy to forget to assert the output of a particular test case, leading to the autofix being untested. And even tests that do not trigger an autofix can benefit from asserting that they have no autofix using `output: null`.
 
 ## Rule Details
 
-This rule aims to ensure that if any invalid test cases have output assertions, then all test cases have output assertions.
+This rule ensures all invalid test cases have `output` assertions.
 
 Examples of **incorrect** code for this rule:
 
 ```js
-/* eslint eslint-plugin/consistent-output: error */
+/* eslint eslint-plugin/require-test-output: error */
 
 new RuleTester().run('example-rule', rule, {
   valid: [],
@@ -34,7 +34,7 @@ new RuleTester().run('example-rule', rule, {
 Examples of **correct** code for this rule:
 
 ```js
-/* eslint eslint-plugin/consistent-output: error */
+/* eslint eslint-plugin/require-test-output: error */
 
 new RuleTester().run('example-rule', rule, {
   valid: [],
@@ -60,14 +60,9 @@ new RuleTester().run('example-rule', rule, {
 
 ## Options
 
-This rule takes an optional string enum option with one of the following values:
+This rule takes an optional object containing:
 
-* `"consistent"` - (default) if any invalid test cases have output assertions, then all invalid test cases must have output assertions
-* `"always"` - always require invalid test cases to have output assertions
-
-## When Not To Use It
-
-If you're not writing fixable rules, or you want to write test cases without output assertions, do not enable this rule.
+* `boolean` -- `consistent` -- only require `output` assertions when some test cases have them (default `false`)
 
 ## Further Reading
 

lib/rules/consistent-output.js renamed to lib/rules/require-test-output.js

@@ -1,5 +1,5 @@
 /**
- * @fileoverview Enforce consistent use of output assertions in rule tests
+ * @fileoverview Enforce use of `output` assertions in rule tests
  * @author Teddy Katz
  */
 
@@ -15,24 +15,34 @@ module.exports = {
   meta: {
     type: 'suggestion',
     docs: {
-      description: 'enforce consistent use of output assertions in rule tests',
+      description: 'enforce use of `output` assertions in rule tests',
       category: 'Tests',
       recommended: false,
     },
     fixable: null, // or "code" or "whitespace"
     schema: [
       {
-        type: 'string',
-        enum: ['always', 'consistent'],
+        type: 'object',
+        properties: {
+          consistent: {
+            type: 'boolean',
+            default: false,
+          },
+        },
+        additionalProperties: false,
       },
     ],
+    messages: {
+      missingOutput: 'This test case should have an `output` assertion.',
+    },
   },
 
   create (context) {
     // ----------------------------------------------------------------------
     // Public
     // ----------------------------------------------------------------------
-    const always = context.options[0] && context.options[0] === 'always';
+    const consistent = context.options[0] && context.options[0].consistent;
+    const always = !consistent;
 
     return {
       Program (ast) {
@@ -48,7 +58,7 @@ module.exports = {
             casesWithoutOutput.forEach(testCase => {
               context.report({
                 node: testCase,
-                message: 'This test case should have an output assertion.',
+                messageId: 'missingOutput',
               });
             });
           }

tests/lib/rules/consistent-output.js

@@ -237,6 +237,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 24 }],
     },
     {
@@ -252,6 +253,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'ArrowFunctionExpression', line: 5, endLine: 5, column: 34, endColumn: 36 }],
     },
     {
@@ -267,6 +269,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'ArrowFunctionExpression', line: 5, endLine: 5, column: 34, endColumn: 36 }],
     },
     {
@@ -280,6 +283,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'ArrowFunctionExpression', line: 5, endLine: 5, column: 32, endColumn: 34 }],
     },
     {
@@ -295,6 +299,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 25 }],
     },
     {
@@ -310,6 +315,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 25 }],
     },
     {
@@ -325,6 +331,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 26 }],
     },
     {
@@ -340,6 +347,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 26 }],
     },
     {
@@ -356,6 +364,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 26 }],
     },
     {
@@ -371,6 +380,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 26 }],
     },
     {
@@ -386,6 +396,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 26 }],
     },
     {
@@ -400,6 +411,7 @@ ruleTester.run('fixer-return', rule, {
             }
         };
         `,
+      output: null,
       errors: [{ messageId: 'missingFix', type: 'FunctionExpression', line: 5, column: 26 }],
     },
   ],

tests/lib/rules/fixer-return.js

@@ -142,6 +142,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           }
         };
       `,
+      output: null,
       errors: [error('bar')],
     },
     {
@@ -156,6 +157,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           }
         };
       `,
+      output: null,
       errors: [error('bar')],
     },
     {
@@ -170,6 +172,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           }
         };
       `,
+      output: null,
       errors: [error('hasOwnProperty')],
     },
     {
@@ -178,6 +181,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           context.report(node, 'foo {{bar}}', { baz: 'qux' });
         };
       `,
+      output: null,
       errors: [error('bar')],
     },
     {
@@ -188,6 +192,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           context.report(node, MESSAGE, { baz: 'qux' });
         };
       `,
+      output: null,
       errors: [error('bar', 'Identifier')],
     },
     {
@@ -196,6 +201,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           context.report(node, { line: 1, column: 3 }, 'foo {{bar}}', { baz: 'baz' });
         };
       `,
+      output: null,
       errors: [error('bar')],
     },
     {
@@ -210,6 +216,7 @@ ruleTester.run('no-missing-placeholders', rule, {
           }
         };
       `,
+      output: null,
       errors: [error('bar')],
     },
   ],