ScriptCallDepthException thrown during Pester-tests only in PS5.1 Integrated Console

[fflaten]

Prerequisites

• I have written a descriptive issue title.
• I have searched all issues to ensure it has not already been reported.
• I have read the troubleshooting guide.
• I am sure this issue is with the extension itself and does not reproduce in a standalone PowerShell instance.
• I have verified that I am using the latest version of Visual Studio Code and the PowerShell extension.
• If this is a security issue, I have read the security issue reporting guidance.

Summary

Powershell throws ScriptCallDepthException when invoking Pester-tests. This was reported as an issue in Pester-repo, but I'm only able to reproduce it using vscode-powershell integrated console with PS5.1

Running in PS 5.1 standalone or integrated console with PS7 works fine.
Not sure how to troubleshoot this further on our end. Tips?

PowerShell Version

=====> PowerShell Integrated Console v2022.5.1 <=====

PS C:\Users\Frode\Git\Pester> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.22000.613
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.22000.613
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Visual Studio Code Version

code --version
1.67.0
57fd6d0195bb9b9d1b49f6da5db789060795de47
x64

Extension Version

code --list-extensions --show-versions | Select-String powershell

[email protected]
[email protected]
[email protected]

(Preview-extension is disabled.)

Steps to Reproduce

See pester/Pester#2174 for repro.

• Running the repro using a file + F5 makes it fail mid-run as issue mentions.
• Running 0..10 | ForEach-Object { Invoke-Pester 'C:\Samples\demoIssue2174\Pester.Tests.ps1' -Output Detailed } in Integrated Console work the first time, but once you run the loop again it fails on first.

Visuals

Broken in Integrated Console using F5:

Broken in Integrated Console using console:

Working in PS7 Integrated Console with F5:

Working in PS5.1 terminal:

Logs

No response

[fflaten]

Using a tip from a previous Pester-issue it looks like PS5.1 Integrated Console suffers from the same limited call depth as was seen with jobs in the linked comment.

PS5.1 standalone:

function f ($i=0) { write-host $i; f (++$i) }
f

...
4941
4942
The script failed due to call depth overflow.

PS5.1 Integrated console:

function f ($i=0) { write-host $i; f (++$i) }
f

...
228
229
An error occurred while running the REPL loop:
System.Management.Automation.ScriptCallDepthException: The script failed due to call depth overflow

PS7.2 Integrated console:

function f ($i=0) { write-host $i; f (++$i) }
f

...
1281
1282
An error occurred while running the REPL loop:
System.Management.Automation.ScriptCallDepthException: The script failed due to call depth overflow.

Knowing this I'm also able to reproduce the issue in PS 5.1 standalone also using jobs.

Start-Job { Import-Module C:\Users\Frode\Git\Pester\bin\Pester.psd1; 0..20 | ForEach-Object { Invoke-Pester 'C:\Users\Frode\Git\Pester\Samples\demoIssue2174\Pester.Tests.ps1' -Output Detailed } } | Receive-Job -Wait

Any way to improve PS5.1 integrated console in vscode-powershell so it works as PS7.2 integrated console?
Or is this a PS 5.1 issue which will never be fixed?

[andyleejordan]

Any way to improve PS5.1 integrated console in vscode-powershell so it works as PS7.2 integrated console? Or is this a PS 5.1 issue which will never be fixed?

You hit the nail on the head, it is exceedingly unlikely to be fixed in PS 5.1, and there's nothing we can do in the integrated console to workaround it. We're launching whatever PowerShell executable we're pointed out (extant bugs and all).

Thanks for investigating and figuring this out! Making my job easy 🙏

[andyleejordan]

I see that the number of recursive calls successfully made between PS 5.1 standalone and in the IC are quite different, and it does seem to be job related, but I think I'll see if I can investigate any further.

[nohwnd]

So maybe it is how the session is started? I don't see this problem in (as it was said above) in normal console, and I don't see it when I run using script using the PowerShell object. And since this is PS5 if we come up with a fix that uses reflection we can be pretty safe it won't change in the future.

So is this behavior common to any powershell 5 code running in separate runspace? This is the entrypoint used in the console process, https://github.com/PowerShell/PowerShellEditorServices/blob/master/src/PowerShellEditorServices.Hosting/Commands/StartEditorServicesCommand.cs

[nohwnd]

The issue is that the thread that is hosting the session is started with default stack size, which gives only 512KB of stack, rather than 10MB of stack that we would normally get in console session. See the PR above.

Interestingly setting the value that Windows PowerShell uses internally (at least based on debugging it) gives us way more recursion calls than in console host (1700 in host, vs some 5000 in the integrated console).

[nohwnd]

Anyways I figured this out by running [Environment]::StackTrace to see where I am running. Saw that the execution happens in a thread. So I looked at the name, which is [Threading.Thread]::CurrentThread -> PSES Pipeline Execution Thread, which is started here: https://github.com/PowerShell/PowerShellEditorServices/blob/dd870ec1887c3cd69f18459d3e226067ab4f2098/src/PowerShellEditorServices/Services/PowerShell/Host/PsesInternalHost.cs#L115-L118

So there I tried to experiment with the values provided to max stack size, which gave me less calls, so I was below the default, and I just needed to find the correct value to specify. Arbitrary value like 10 times the default seemed okay. But I wanted to know what the real thing does. I did not find the source, but in decompiled code it does similar thing to this, so I just used that:

https://github.com/PowerShell/PowerShell/blob/44098e771a059c06f1619eac8f97deeb118d127e/src/System.Management.Automation/engine/hostifaces/LocalPipeline.cs#L23-L29

[fflaten]

Wow. Awesome work 👏

Hopefully this is configurable for jobs/remoting in some way too as a workaround for some other issues.

[nohwnd]

You have to set the value Before starting the thread (AFAIK), so we would have to be very lucky to have that value set from a non-readonly property, or a field, so we can re-write it on runtime. Maybe there is some override that the native api takes into account but I could not find any.

Seems like modern powershell has the same issue, https://github.com/PowerShell/PowerShell/blob/a6d04226c365d811d271340174e8453cbb53b26c/src/System.Management.Automation/engine/remoting/common/RemoteSessionNamedPipe.cs#L689

It is just less visible because .NET default stack size is bigger.

[nohwnd]

Finding this was a bit easier. I just attached to the main process and put breakpoint into Process.Start, then walked up the call stack to see where the process is started, and put BP on the next line. This puts me in place to have the child process running, but not yet connected to the parent.

Then I attached to the newly spawned process and put BP into Thread.Start again, to see where it called from and with which parameters.

[andyleejordan]

I agree, this was awesome work. Thanks so much for investigating and fixing it. We'll get a new release out ASAP.

[andyleejordan]

@fflaten and @nohwnd this is now out in preview! Thanks so much for the fix, sorry it took so long to get out.