Add side-by-side support to pwsh.exe

[KirkMunro]

No description provided.

[iSazonov]

• To make WindowsUpdate work we need to have one main/default version installed in well-known folder. (It was in previous discussion.)
• We need to define/register a default version is started by pwsh without Version parameter.
• To enumerate installed versions we need a version registration method.

[KirkMunro]

I was under the assumption that the latest stable version would always be the default version. That is the only way to allow updates to continue to notify for new versions that are not installed. Otherwise if you had an older default, would Windows Update prompt you to upgrade to a newer stable version that you have installed?

For version registration, I get why you suggest it is needed, but it adds some complexity. We could do registration like dotnet core does (I have no idea how they do it on xplat platforms but copying their example that is already in place should be easy). Or, we could instead do the following:

1. Only allow additional version installs in folders where pwsh is already installed (i.e. there can only be one folder with pwsh, with each version installed being in a versioned subfolder).
2. Set up an environment variable with the base folder or just pull it from $PSHome in the default version that is installed, and place new side-by-side versions there.

On the downside, if pwsh is installed in a system path, end users may not be able to install additional side-by-side versions in personal folders. But that's an upside as well, because it means that admins can control what is installed more easily. Thoughts?

[iSazonov]

I was under the assumption that the latest stable version would always be the default version.

Can we add this explicitly? Although I doubt it will work for WindowsUpdate - as I remember it requires the same folder.

For version registration, I get why you suggest it is needed, but it adds some complexity. We could do registration like dotnet core does

I guess Core uses directory structure:

 dir "C:\Program Files\dotnet\sdk\"


    Directory: C:\Program Files\dotnet\sdk


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       17.04.2019     13:55                2.1.202
d-----       29.03.2019     19:08                2.1.504
d-----       17.04.2019     13:49                2.1.505
d-----       24.05.2019     16:07                2.1.507
d-----       24.05.2019     17:17                2.1.700
d-----       24.05.2019     17:38                2.2.300
d-----       20.05.2019      9:22                3.0.100-preview5-011568
d-----       24.05.2019     17:41                NuGetFallbackFolder

On the downside, if pwsh is installed in a system path, end users may not be able to install additional side-by-side versions in personal folders.

We could have well-known system and user folders Admins could control versions by policy (like GPO, we have RFC for implementing policies).

[KirkMunro]

Of course. I added two additional consideration sections for these. Those will need to be flushed out as I get additional feedback.

[Jaykul]

I guess Core uses directory structure:

Core isn't supported by Windows Update.

Assuming that 7.0 is going to be LTS (I think that's impossible, since .NET 3.0.0 won't be LTS), should it not go in a folder that allows other versions of 7 to be installed around it?

[KirkMunro]

Assuming that 7.0 is going to be LTS (I think that's impossible, since .NET 3.0.0 won't be LTS), should it not go in a folder that allows other versions of 7 to be installed around it?

From https://dotnet.microsoft.com/platform/support/policy/dotnet-core:

.NET release cadence

We will publish new major releases of .NET Core on a regular cadence, enabling developers, the community and businesses to plan their roadmaps. Beginning with .NET Core 3.1, these releases will happen every November and every other release will be LTS.

So your point is that this won't happen until .NET Core 3.1? Yet from the PowerShell Team blog announcing PowerShell 7, they state:

In PowerShell 7, we will align more closely with the .NET Core support lifecycle, enabling PowerShell 7 to have both LTS (Long Term Servicing) and non-LTS releases.

That's a little confusing.

That aside, yes, I think it should install automatically to a system/all users folder or a current user folder rather than be installable anywhere like PowerShell Core can today.

[KirkMunro]

One of the key motivators that I had behind this RFC was to reduce the drag. It is way too common for the community to push for downlevel solutions that are implemented in modules when we need core features in the language for current and future versions. In the past products took a hard dependency on a specific version of PowerShell (current documentation even states that the SCOM 2019 Management Console and Agent require PowerShell 2.0 or 3.0), and those dependencies slow down adoption of newer versions of PowerShell.

@Jaykul pointed out in a side conversation that we have PowerShell Standard Libraries that can be used with by module authors to create modules that support many different versions of PowerShell (from Windows PowerShell 3.0 up to PowerShell Core 6.2.1, or from Windows PowerShell 5.1 up to PowerShell Core 6.2.1), and that resolves this problem.

If PowerShell Standard indeed resolves this problem such that machines don't need to be unnecessarily locked to a specific LTS version of PowerShell, then there isn't really much need for SxS support in pwsh.exe and this PR can be closed.

[SteveL-MSFT]

Is there a real need to have patch versions side-by-side? I would think the common scenario is 6.2, 7.0, 7.1.

[rkeithhill]

Indeed. And if PowerShell releases truly follow semver rules for versioning, I would expect 7.1 to be completely back compat with 7.0. So I'd only expect side-by-side support for major versions e.g. 5, 6, 7, 8, etc. Oh, looking ahead, you still need side-by-side support for previews as well but that is already handled today for major versions.

[joeyaiello]

We're explicitly not using semver for the version of PowerShell itself. Modules should use semver, but PS versions are just intended to signal relative size of feature sets, among other things.

[SteveL-MSFT]

I think it should be explicitly stated that this will only look in the expected installed locations and not search the entire filesystem. This means that custom deployments of the .zip/.tgz won't be found unless they are expanded in the same locations.

[bwright86]

Maybe use a file inventory that each version of PowerShell updates during install/uninstall. This method could read the file and list the versions there. Maybe with some validation that the source truly exists. Or detect Choco and list versions that way.

[essentialexch]

if Windows is included here, the registry should be examined, IMO.

[SteveL-MSFT]

There are some implications we need to think about here. On Unix systems, we can leverage exec so that there is a single pwsh process. On Windows, if you run this from cmd or PowerShell, the assumption is you want to use the current console window. So that means that pwsh will start the newest one (as that's likely the one in PATH), which will then start a child process with the requested version. So you'll end up with two pwsh processes. There might be a way for pwsh.exe to host the requested pwsh.dll, but then you might have issues with pwsh.exe being compiled against a different version of .NET Core. Need to think through this or just accept there's two processes.

[rkeithhill]

Within PowerShell at least, couldn't it intercept this request to start the native app called pwsh with a specific -Version param/arg, lookup that version and start it, passing it the rest of parameters except for -Version? I'm not a fan of special-case code like this but it seems doable.

[SteveL-MSFT]

It could be special cased in PowerShell itself, but not sure if it's worth it. Also, it seems that this is most useful for scripts using #Requires -Version X to automatically run under those versions which isn't called out in this RFC.

[bwright86]

The “PowerShell.exe -Version x.x -File “C:\Path\To\script.ps1” could work here as well. It would be nice if a warning could be displayed, if that version isn’t installed and include what version was used instead.

[bwright86]

Also, I think it would be good to spawn a new child process, if calling a different “pwsh” from the console. This is expected behavior in Windows and *nix world when requesting a different shell. Unless a very good cleanup job was done of the current process, using some .dll swapping could get messy.

[essentialexch]

I think there should be an error if the specified version isn't available, not using something else.

[iSazonov]

What if the version is not installed? Will we fail or run upper version?

[SteveL-MSFT]

It seems that if a specific version is requested and not found, it should not run and would return an error. If 6.2 is requested, then 6.2.2 would run, if available. It probably doesn't make sense to support any preview releases with this feature.

[SteveL-MSFT]

Windows Update requires that the path to the binaries don't change. This means that we don't have even a complete version as the folder name hence it's just 6 and not 6.1 or 6.2. This means that this may limit the usefulness to just major versions.

[essentialexch]

s/some/many/

[essentialexch]

Does this RFC include Windows? Or is it MacOS/Linux only? Where does WSL fit?

The examples are using a lot of third-party (by which I mean "non-Windows" and "non-PowerShell") commands. I find myself quite confused by this. I know what "apt get" is, but no clue what "brew" or "choco" or etc. do.

[SteveL-MSFT]

brew is apt for macOS and choco is apt for Windows (3rd party so not available by default). This RFC needs to work with non-Windows.

[essentialexch]

I think that Get-InstalledPSVersion should definitely include Windows PowerShell, but I don't think the rest is really necessary given the binaries are different names. But it's not an important point.

[essentialexch]

Who is your audience?

I'm an admin scripter. And while I know what Docker/containers are, I certainly don't use them, nor do I know how (I have done a Windows container lab, but not using Docker). Lots of desktops/laptops don't have virtualization turned on, nor have a virtualizer installed nor will your average user know how to do so. But their admins may have very good reasons to need multiple versions of PS.

[joeyaiello]

@PowerShell/powershell-committee had a long discussion on this one. While we understand the behavior you're proposing, we're not sure that the scenario is widespread.

Windows PowerShell was updated much less frequently than PowerShell Core, and had more sweeping changes from version-to-version. Plus, Microsoft products were often tied to specific Windows PowerShell versions, and so side-by-side (SxS) became more necessary. To address this, we made SxS a high priority in PowerShell Core, allowing you to manage multiple installs. This was primarily enabled by .NET Core and the fact that we can ship app-local copies of the runtime with each version of PowerShell Core. For those that absolutely need SxS, we haven't heard that symlinks/xcopies don't meet their needs.

There are a few other complexities here that also haven't been addressed:

• Both apt-get and yum do not support side-by-side packages. With packages like Python, they have to fork the namespace out (e.g. python2 and python3), and that's not something we're okay signing up to maintain right now. brew does support SxS versions (a la Nuget), but it's the exception there, and we're not sure we can depend long-term on the version path magic they do today.
• We're still working through whether the 7 stable MSI should upgrade the 6 stable MSI, or if it should be SxS. There are complexities around the PATH that need to be addressed before we can make that call (like whether we append/prepend today, what we should do going forward, etc.) We'll file an issue for this one to discuss.

Therefore, we're rejecting this one blocked on:

• finding out whether the needs of SxS are not actually met today by xcopy-able versions and symlinks
• a reconciliation of whether and how PowerShells would be "installed" (as opposed to xcopy-ed) SxS (how can we know where to detect them if we don't know where they live yet?)

Our current recommendation until then is to use your own file path and symlinks.

Ideally we should do something like venv or rbenv where the user can configure which pwsh on a box they want to use, and scope their PSModulePath to some workspace/project-specific folder. This doesn't need to initially be done in PowerShell itself, so if the community sees a need, they should start experimenting in that space.

[iSazonov]

Both apt-get and yum do not support side-by-side packages.

This means that what we discussed before the 6 version release remains valid - we are doomed to have one main (system-wide) version. Updating versions of other products is the task of the manufacturers of these products.

how can we know where to detect them if we don't know where they live yet?

Updating versions installed side-by-side is another task and we already have a RFC for detecting new versions at startup.

So we could have some update channels - (1) Passive (Windows Update/WSUS) that is good for main, LTS versions and enterprises, (2) Active (at startup) that is good for side-by-side, personal versions (stable, preview, daily - that may be in PowerShell config file (ignored for main installation)).

[eedwards-sk]

Ideally we should do something like venv or rbenv where the user can configure which pwsh on a box they want to use, and scope their PSModulePath to some workspace/project-specific folder. This doesn't need to initially be done in PowerShell itself, so if the community sees a need, they should start experimenting in that space.

I'm moving to powershell in nix scripting specifically to get away from the mess that is rbenv / pyenv and friends.

If powershell itself wants to handle the version selection, I would expect each major version to be installable as an explicit version (pwsh6, pwsh7), as well as some kind of "powershell library entrypoint" that provides pwsh which is responsible for invoking the appropriate underlying version.

Then when writing scripts, one could explicitly target a required version (pwsh7), or target it more like a 'library' (pwsh --version 7). The former requires system admins to either explicitly install that script's needed version, ensuring it's used, or to symlink pwsh7 to another version (say pwsh8). The latter allows the script author to target a minimum version, but allow the system to run it on any newer version.

I see both patterns in the wild. Sometimes you want to ensure your script runs exactly as expected with that explicit version, forcing the admins to configure the system for it, and putting the burden on them if they choose to break it with newer versions.

Other times it's a library or generic tool that the author wants to support multiple versions, and allow the system to provide any minimum version, and thus the burden is on the script author to fix when it breaks with newer versions.

Not sure if that's helpful / relevant, but imo the more "magic" we put in the way of version selection, the more risk of needing workarounds to workaround the magic when it doesn't work as we expect.