Severity-filter works unexpected with custom script-rules

[fflaten]

Using the -Severity <level> or similar setting in PSScriptAnalyzerSettings.psd1 doesn't work as expected with custom rules.

Steps to reproduce

AnalyzerRules.psm1

function Measure-AlwaysFail {
    [CmdletBinding()]
    [OutputType([Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord[]])]
    Param
    (
        [Parameter(Mandatory = $true)]
        [ValidateNotNullOrEmpty()]
        [System.Management.Automation.Language.CommandAst]
        $CommandAst
    )

    Process {
        $commandName = $CommandAst.GetCommandName()

        $results = @()
        try {
            $result = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord]@{
                'Message'              = "Always fails on commands."
                'Extent'               = $CommandAst.Extent
                'RuleName'             = $PSCmdlet.MyInvocation.InvocationName
                'Severity'             = 'Information'
                'RuleSuppressionID'    = $commandName
            }
            $results += $result
            return $results
        }
        catch {
            $PSCmdlet.ThrowTerminatingError($PSItem)
        }
    }
}

Expected behavior

PS > $commonSettings = @{
    CustomRulePath = '.\AnalyzerRules.psm1'
    ScriptDefinition = ({ Write-Host "Hello world" }.ToString())
}
PS > Invoke-ScriptAnalyzer @commonSettings

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
AnalyzerRules\Measure-AlwaysFail    Information             1     Always fails on commands.

PS > Invoke-ScriptAnalyzer @commonSettings -Severity Information

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
AnalyzerRules\Measure-AlwaysFail    Information             1     Always fails on commands.

PS > Invoke-ScriptAnalyzer @commonSettings -Severity Warning
PS > Invoke-ScriptAnalyzer @commonSettings -Severity Error
PS > Invoke-ScriptAnalyzer @commonSettings -Severity Warning, Error
PS > 

Actual behavior

PS > $commonSettings = @{
    CustomRulePath = '.\AnalyzerRules.psm1'
    ScriptDefinition = ({ Write-Host "Hello world" }.ToString())
}
PS > Invoke-ScriptAnalyzer @commonSettings

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
AnalyzerRules\Measure-AlwaysFail    Information             1     Always fails on commands.


PS > Invoke-ScriptAnalyzer @commonSettings -Severity Information
PS > Invoke-ScriptAnalyzer @commonSettings -Severity Warning

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
AnalyzerRules\Measure-AlwaysFail    Information             1     Always fails on commands.

PS > Invoke-ScriptAnalyzer @commonSettings -Severity Error
PS > Invoke-ScriptAnalyzer @commonSettings -Severity Warning, Error

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
AnalyzerRules\Measure-AlwaysFail    Information             1     Always fails on commands.

PS > 

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      7.0.3
PSEdition                      Core
GitCommitId                    7.0.3
OS                             Microsoft Windows 10.0.19041
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0
> (Get-Module -ListAvailable PSScriptAnalyzer).Version | ForEach-Object { $_.ToString() }

1.19.1

Same result in PowerShell 5.1

[rjmholt]

This occurs because the -Severity switch applies to rule severities, rather than diagnostic severities.

PSSA looks at the rule and decides based on the set severity whether to run it (rather than running it and then filtering out the diagnostics):

PSScriptAnalyzer/Engine/ScriptAnalyzer.cs

Lines 1948 to 1978 in d945a41

	bool IsRuleAllowed(IRule rule)
	{
	IEnumerable<uint> allowedSeverities = GetAllowedSeveritiesInInt();
	bool includeRegexMatch = false;
	bool excludeRegexMatch = false;
	foreach (Regex include in includeRegexList)
	{
	if (include.IsMatch(rule.GetName()))
	{
	includeRegexMatch = true;
	break;
	}
	}
	foreach (Regex exclude in excludeRegexList)
	{
	if (exclude.IsMatch(rule.GetName()))
	{
	excludeRegexMatch = true;
	break;
	}
	}
	bool helpRule = String.Equals(rule.GetName(), "PSUseUTF8EncodingForHelpFile", StringComparison.OrdinalIgnoreCase);
	bool includeSeverity = IsSeverityAllowed(allowedSeverities, rule);
	return (includeRule == null || includeRegexMatch)
	&& (excludeRule == null || !excludeRegexMatch)
	&& IsSeverityAllowed(allowedSeverities, rule)
	&& IsRuleEnabled(rule);
	}

There's no way to designate an external rule as having any severity other than Warning right now:

PSScriptAnalyzer/Engine/Generic/ExternalRule.cs

Lines 55 to 59 in d945a41

	//Set the community rule level as warning as the current implementation does not require user to specify rule severity when defining their functions in PS scripts
	public RuleSeverity GetSeverity()
	{
	return RuleSeverity.Warning;
	}

I have an open proposal for a rework of custom rules in PSSA in #1551 that you might like to look at.

[fflaten]

Thanks. I guess -ExcludeRule is the way to go for now, at least until PSSA2 is ready. Hopefully this can be more intuitive then.