Rule severity vs diagnostic severity

[rjmholt]

From #1610.

One consideration that comes up when authoring rules is:

• Rules can have a "severity"
• The individual diagnostics can also have a "severity"

When we configure things, this creates the issue of:

• Does a configured severity refer to a filter on rules or diagnostics?
• If it's rules, how do rules perform broad analysis on a topic area and possibly output diagnostics of different severity?
  • Should rules designate their severity as the highest severity they may emit? If so, how do we enforce that statically?
• If it's diagnostics, how do we ensure rules are run efficiently? If we can't statically determine ahead of time which rules will emit diagnostics of the right severity, are we forced to always run all the rules?

We could potentially offer both configurations, but I personally think this is an unhelpful level of granularity to offer to users, since we should work to make the concept of Severity a simple and monolithic one.

My current view on this is:

• Configured severity refers to rules to run
• Rules should be marked with the highest severity they can emit
• We should run all such rules and then filter out any diagnostics of unwanted severities

But that's just while I'm thinking on it now, and I'd like to open this question up for discussion

[fflaten]

The difference between rule and diagnostic severity are currently not very clear for new users - maybe it's just me. I can see the benefit of being able to output a record with a lower level inside a rule, e.g. Information-event for minor/potential future problems, however it makes things complicated.

Given if(..problem found..) { Write-Diagnostic -Severity Warning } elseif (..something minor..) { Write-Diagnostic -Severity Information }, then the rule would have a rule-level of Warning using your proposal. How would this actually play out combined with a Severity-filter?

Invoke-ScriptAnalyzer -Severity Warning
- Rule has level warning -> will run.
- Minor problem found, information record created -> record is not warning, removed by filter.

Invoke-ScriptAnalyzer -Severity Information
- Rule has level warning, higher than severity-setting -> will not run

In both scenarios, the rule has lost value by not returning the Information-event which is unexpected/confusing.

Could Rule-severity be an array of all possible diagnostic-levels used in the rule?
The author could specify it in the proposed rule-attribute. I guess it could be parsed from calls to the propsed Write-Diagnostic-function in the rule while loading/caching rulemodules, unless the possibility of someone splatting the parameters to Write-Diagnostics is hard to detect. Manually declaring it in the attribute wouldn't take much time for the author.

Then the user would only have to keep track of one Severity-type.

• Get-ScriptAnalyzerRule would show all possible severity-levels per rule which makes it natural for the user to specify one or more when executing a PSSA run.
• Rules will only run when the ruleseverity-array contains one or more of the user-provided severity-levels for the run
• Output/records will be filtered using the same user-provided severity-setting (evaluated in Write-Diagnostic?)
• Attempts to create diagnostic records using a level not defined in attribute would throw (only show warning for interactive usage of `Invoke-ScriptAnalyzer?) to make it easier to detect the bug/mismatch.