[Regression] PS7.3 - Weird network issues inside Session

[phdavis]

Prerequisites

• [X] Write a descriptive title.
• [X] Make sure you are able to repro it on the latest released version
• [X] Search the existing issues.
• [X] Refer to the FAQ.
• [X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

When inside a session to a remote machine with PS7.3+ and attempt to invoke some commands that rely on the network they seem to fail. Not all network commands fail, such as Test-NetConnection, while others such as Invoke-WebRequest, Connect-AzAccount, etc do. Those that do fail seem to reference the same failure condition 'Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0' when I can get enough output from them.

• Reverting the remote machine back to 7.2.6/7.2.7 resolves the problem and the same code runs without issue.
• If ConfigurationName is set to anything lower than 7.3.0 things begin working as well.
• All systems have good internal network access and internet access.
• Account/s used for testing are all local admins.
• The same code works locally on the machines with 7.3.0. This appears to only be occurring within sessions.

Create a session or invoke a command directly on a remote machine with 7.3+ installed

$TestSession = New-PSSession -Name 'TestSession' -ComputerName 'PS73Machine.tld.com' -ConfigurationName PowerShell.7.3.0 -Credential $Creds

Setup your test and invoke the commands. Some commands work, such as Invoke-RestMethod while others fail.

$TestParams = @{
    Session = $TestSession
    ScriptBlock = {
        try {
            Invoke-WebRequest -Uri 'https://www.powershellgallery.com/api/v2' # Fails
            Find-Module -Name Az -Repository PSGallery # Fails
        } catch {
            $_.Exception
        }
    }
}

Invoke-Command @TestParams

Expected behavior

Commands such as Find-Module with session should return modules matching "Az" from PSGallery.

Actual behavior

Commands such as Find-Module, Invoke-WebRequest, etc within session fail. For those that return an error they reference -

Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0, Culture=neutral The system cannot find the file specified.

Error details

Invoke-WebRequest example -

Exception                     : System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
                                File name: 'System.Net.Quic, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
                                at System.Net.Http.HttpConnectionPool..ctor(HttpConnectionPoolManager poolManager, HttpConnectionKind kind, String host, Int32 port, String sslHostName, Uri proxyUri)
                                at System.Net.Http.HttpConnectionPoolManager.SendAsyncCore(HttpRequestMessage request, Uri proxyUri, Boolean async, Boolean doRequestAuth, Boolean isProxyConnect, CancellationToken cancellationToken)
                                at System.Net.Http.HttpConnectionPoolManager.SendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
                                at System.Net.Http.HttpConnectionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
                                at System.Net.Http.HttpMessageHandlerStage.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
                                at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
                                at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
                                at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
                                at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(HttpClient client, HttpRequestMessage request, Boolean keepAuthorization)
                                at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
                                at System.Management.Automation.CommandProcessor.ProcessRecord()
FullyQualifiedErrorId         : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
InvocationInfo                : System.Management.Automation.InvocationInfo
ErrorCategory_Activity        : Invoke-WebRequest
ErrorCategory_Reason          : FileNotFoundException
ErrorCategory_Message         : NotSpecified: (:) [Invoke-WebRequest], FileNotFoundException
ErrorDetails_ScriptStackTrace : at <ScriptBlock>, <No file>: line 3

Environment data

Test Source -
Name                           Value
----                           -----
PSVersion                      7.3.0
PSEdition                      Core
GitCommitId                    7.3.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Test Destination -
Name                           Value
----                           -----
PSVersion                      7.3.0
PSEdition                      Core
GitCommitId                    7.3.0
OS                             Microsoft Windows 10.0.20348
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[iSazonov]

/cc @SteveL-MSFT @PaulHigin

[PaulHigin]

It looks like dotNet assembly path probing logic has changed for v7.3, and is preventing the dependent System.Net.Quic.dll binary from loading. When I try to load it directly, it fails from looking in the wrong place (system drive, instead of $PSHOME).

[remote]: PS> Add-Type -AssemblyName 'System.Net.Quic' -PassThru

Error: Cannot find path 'C:\System.Net.Quic.dll' because it does not exist.

However, I am able to get the probing path corrected by running other commands:

[remote]: PS> Find-module -name az -repository psgallery
[remote]: PS> Add-Type -AssemblyName 'System.Net.Quic' -PassThru

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
False    False    Interop                                  System.Object
...

[remote]: PS> Invoke-WebRequest -Uri 'https://www.powershellgallery.com/api/v2'

StatusCode        : 200
StatusDescription : OK
...

@daxian-dbw I suspect this problem is happening on remote sessions because they are hosted as WinRM plugins, and the dotNet default assembly probing location is not correct (should be $PSHOME). For some reason running other cmdlets (Find-Module) fixes the probing path so that the assembly can be loaded. Are you aware of any changes in dotNet v7.3 that would cause this, and is there anything we can do to ensure the probing path for remote sessions starts at $PSHOME?

[daxian-dbw]

This is because the Trusted Platform Assemblies used in our WinRM plugin was not updated. The fix was submitted: https://github.com/PowerShell/PowerShell-Native/pull/83

[dmpe]

Hi, does this require a new patch release for 7.3? Or is there any workaround when PS 7.3 is already installed?

[iSazonov]

@dmpe The fix will be in 7.3.1 (Jan 23).

[daxian-dbw]

A correction, the fix will be in 7.3.2, which will be released in January.

[kasini3000]

Does “System.Net.Quic.dll” rely on libmsquic.so on the Linux platform?

[eeskildsen]

I'm running 7.3.2 locally and remotely and getting a similar error, which was marked as a duplicate of this issue in https://github.com/PowerShell/PowerShell/issues/18671:

Enter-PSSession: Connecting to remote server 108.61.242.20 failed with the following error message : Could not load file or assembly 'System.Security.Cryptography, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. For more information, see the about_Remote_Troubleshooting Help topic.

Local $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.3.2
PSEdition                      Core
GitCommitId                    7.3.2
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Remote $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.3.2
PSEdition                      Core
GitCommitId                    7.3.2
OS                             Microsoft Windows 10.0.14393
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Edit: Corrected remote $PSVersionTable. Looks like I originally pasted the same one for local and remote by mistake.

[phdavis]

Can confirm - I am still having the same issue with remote system on 7.3.2. I will be rolling back again.

(Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.)

[daxian-dbw]

I have verified that this was fixed with the v7.3.2 release of PowerShell. @eeskildsen and @phdavis, please double check the PSSessionConfiguration you use when creating the remote session. You may be using the default configuration that doesn't use v7.3.2 version of PowerShell.

The following screenshot shows the results from running Invoek-WebRequest on remote sessions targeting v7.3.1 and v7.3.2:

[phdavis]

@daxian-dbw my apologies. I must not have caught that when I was testing. I can confirm forcing it to use 7.3.2 specifically works. I'm not sure which version it was trying to use previously since I thought I had removed the old configurations.

[Oscaruzzo]

This issue should NOT be closed. The update from 7.2.x to 7.3.x still doesn't fix the configuration 'Powershell.7' which is left dangling and pointing to a non existent 7.2. Forcing the connection to 'Powershell.7.3.8' is not acceptable beacuse any script will stop working when the server side is update to 7.3.9.

[DotNetSparky]

This issue should NOT be closed. The update from 7.2.x to 7.3.x still doesn't fix the configuration 'Powershell.7' which is left dangling and pointing to a non existent 7.2. Forcing the connection to 'Powershell.7.3.8' is not acceptable beacuse any script will stop working when the server side is update to 7.3.9.

Try deleting all existing configurations and then run Enable-PSRemoting again -- I've had to do this once per affected machine and the problem hasn't returned.

Get-PSSessionConfiguration | Unregister-PSSessionConfiguration
Enable-PSRemoting

[Oscaruzzo]

This issue should NOT be closed. The update from 7.2.x to 7.3.x still doesn't fix the configuration 'Powershell.7' which is left dangling and pointing to a non existent 7.2. Forcing the connection to 'Powershell.7.3.8' is not acceptable beacuse any script will stop working when the server side is update to 7.3.9.

Try deleting all existing configurations and then run Enable-PSRemoting again -- I've had to do this once per affected machine and the problem hasn't returned.

Get-PSSessionConfiguration | Unregister-PSSessionConfiguration
Enable-PSRemoting

Thank you, I know this workaround (and it works). But if it (still) needs a workaround, it's a bug, and it's not closed ;)