Same kind of issue: Posh-SSH with Cisco Switch #308

[apagnott]

Hi,

I'm creating a powershell script based on posh-ssh module to apply a configuration to a liste of cisco switches..

I run into the same problem as in Posh-SSH with Cisco Switch #308... I get sometimes not everytime...

New-SSHSession -ComputerName $device -Port $remoteport -Credential $creds -AcceptKey:$true -KeepAliveInterval 1 -Verbose -ErrorAction Stop -ConnectionTimeout $conntimeout -OperationTimeout $opertimeout

I've tried with different values for $opertimeout and $connTimeout... but same issue... I try at the same time tu connect with putty and it works with it... How could I troubleshoot this ?

VERBOSE: Using SSH Username and Password authentication for connection.

Session operation has timed out

Thanks in advance for your help

[darkoperator]

Anything on the logs on the cisco? teminal monitor I think is the command to see the debug info

Sent from my iPhone

On Nov 12, 2020, at 7:49 AM, apagnott [email protected] wrote:

 Hi,

I'm creating a powershell script based on posh-ssh module to apply a configuration to a liste of cisco switches..

I run into the same problem as in Posh-SSH with Cisco Switch #308... I get sometimes not everytime...

New-SSHSession -ComputerName $device -Port $remoteport -Credential $creds -AcceptKey:$true -KeepAliveInterval 1 -Verbose -ErrorAction Stop -ConnectionTimeout $conntimeout -OperationTimeout $opertimeout

I've tried with different values for $opertimeout and $connTimeout... but same issue... I try at the same time tu connect with putty and it works with it... How could I troubleshoot this ?

VERBOSE: Using SSH Username and Password authentication for connection.

Session operation has timed out

Thanks in advance for your help

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[apagnott]

I've check the logs > sh logs .... but I dont see anything regarding the connexion attempt... maybe I have to check somewhere else on the switch... do you have an idea ? The strange thing is that for the same switch sometimes it works sometimes not.... how can I troubleshoot on powershell or .net side what coud go wrong ?

[darkoperator]

It has been over a decade since I worked on Cisco gear but back in the ios 11 and 12 days debugging info was printed to the console

Sent from my iPhone

On Nov 12, 2020, at 9:24 AM, apagnott [email protected] wrote:

 I've check the logs > sh logs .... but I dont see anything regarding the connexion attempt... maybe I have to check somewhere eslse on the switch... do you have an idea ? The strange thing is that for the same switch sometimes it works sometimes not....

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[apagnott]

How could I debug the powershell module ? Do you have a debug version?

[darkoperator]

What I meant was on the Cisco Switch with a serial cable connected configure it with a high debug level and to print debug to the console and try to connect

On Nov 12, 2020, at 4:44 PM, apagnott [email protected] wrote:

How could I debug the powershell module ? Do you have a debug version?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darkoperator/Posh-SSH/issues/354#issuecomment-726331838, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7IHXPKMPASURIFL4QAE3SPRCLPANCNFSM4TTF2EFA.

[apagnott]

I can definitely try that route and see if a clue of the problem can be identified on the switch, but...the issue is the same on other switches and as I said sometimes it works and sometimes I get the session operation time out exception...not consistent...my feeling is that the problem is on the pc running the powershell command...what can look at on the PC in order to investigate? Could I test the underlying interface and see if the error returned gives me more clues?

[apagnott]

Reading through other opened issues I found this one... that could be the same I'm facing

Cisco Router frequent New-SSHSession : Session operation has timed out #340

[apagnott]

I wondering if I have to use ssh.client that is referenced here... I didn't and it worked so maybe the built in ssh.client library is already in .net framework 4. Could you help on this ? HOw can force the posh-module to use this https://www.nuget.org/packages/ssh.net/ ?

[reybanger]

@apagnott How do you run your script? via ISE or from powershell command line? I have recently noticed that the behavior is different considering how I do run my script. From the ISE I always face strange errors. Would that be the case for you?

[apagnott]

I use ISE...I m gonna try to run my script with powershell.exe

Thanks for trying to help me .... Do you know how to use ssh.net and install it? I did try without it...it seems to be included in .net framework...or I dont understand how this ssh.net is used by posh....

Le Fri, 20 Nov 2020 07:58:53 -0800, reybanger a écrit :

@apagnott How do you run your script? via ISE or from powershell command line? I have recently noticed that the behavior is different considering how I do run my script. From the ISE I always face strange errors. Would that be the case for you?

-- Sent using Dekko from my Ubuntu device

[darkoperator]

SSH.Net http://ssh.net/ is just a library, you can download the code from Github and compile it in Visual Studio for the framework you need. I’m using the latest version with some additional fixes from PRs that are in the GitHub.

On Nov 20, 2020, at 1:30 PM, apagnott [email protected] wrote:

I use ISE...I m gonna try to run my script with powershell.exe

Thanks for trying to help me .... Do you know how to use ssh.net and install it? I did try without it...it seems to be included in .net framework...or I dont understand how this ssh.net is used by posh....

Le Fri, 20 Nov 2020 07:58:53 -0800, reybanger a écrit :

@apagnott How do you run your script? via ISE or from powershell command line? I have recently noticed that the behavior is different considering how I do run my script. From the ISE I always face strange errors. Would that be the case for you?

-- Sent using Dekko from my Ubuntu device — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darkoperator/Posh-SSH/issues/354#issuecomment-731302060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7IHU3VR4O3TASHYKGAELSQ2RUZANCNFSM4TTF2EFA.

[apagnott]

Ok I thought posh-ssh is using it....so it is not ? the module seems to work without it...

To get back to my issue .... I m not the only one facing it? Could it be related to the .net framework insralled on my box? Is operation timeout parameters really implemented in posh?

Le Fri, 20 Nov 2020 09:40:29 -0800, Carlos Perez a écrit :

SSH.Net http://ssh.net/ is just a library, you can download the code from Github and compile it in Visual Studio for the framework you need. I’m using the latest version with some additional fixes from PRs that are in the GitHub.

On Nov 20, 2020, at 1:30 PM, apagnott [email protected] wrote:

I use ISE...I m gonna try to run my script with powershell.exe

Thanks for trying to help me .... Do you know how to use ssh.net and install it? I did try without it...it seems to be included in .net framework...or I dont understand how this ssh.net is used by posh....

Le Fri, 20 Nov 2020 07:58:53 -0800, reybanger a écrit :

@apagnott How do you run your script? via ISE or from powershell command line? I have recently noticed that the behavior is different considering how I do run my script. From the ISE I always face strange errors. Would that be the case for you?

-- Sent using Dekko from my Ubuntu device — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darkoperator/Posh-SSH/issues/354#issuecomment-731302060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7IHU3VR4O3TASHYKGAELSQ2RUZANCNFSM4TTF2EFA.

-- Sent using Dekko from my Ubuntu device

[darkoperator]

I use the library, just not the old one in the nugget repo. It is outdated and has some bugs

On Nov 20, 2020, at 2:30 PM, apagnott [email protected] wrote:

Ok I thought posh-ssh is using it....so it is not ? the module seems to work without it...

To get back to my issue .... I m not the only one facing it? Could it be related to the .net framework insralled on my box? Is operation timeout parameters really implemented in posh?

Le Fri, 20 Nov 2020 09:40:29 -0800, Carlos Perez a écrit :

SSH.Net http://ssh.net/ is just a library, you can download the code from Github and compile it in Visual Studio for the framework you need. I’m using the latest version with some additional fixes from PRs that are in the GitHub.

On Nov 20, 2020, at 1:30 PM, apagnott [email protected] wrote:

I use ISE...I m gonna try to run my script with powershell.exe

Thanks for trying to help me .... Do you know how to use ssh.net and install it? I did try without it...it seems to be included in .net framework...or I dont understand how this ssh.net is used by posh....

Le Fri, 20 Nov 2020 07:58:53 -0800, reybanger a écrit :

@apagnott How do you run your script? via ISE or from powershell command line? I have recently noticed that the behavior is different considering how I do run my script. From the ISE I always face strange errors. Would that be the case for you?

-- Sent using Dekko from my Ubuntu device — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darkoperator/Posh-SSH/issues/354#issuecomment-731302060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7IHU3VR4O3TASHYKGAELSQ2RUZANCNFSM4TTF2EFA.

-- Sent using Dekko from my Ubuntu device — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darkoperator/Posh-SSH/issues/354#issuecomment-731338402, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7IHR2C7INFOHWMFDF7QLSQ2YWFANCNFSM4TTF2EFA.

[apagnott]

How could I invetigate on client (module) side.... if the remote ssh host is waiting for anything ? Maybe the issue is due to a bug in this ssh.net library... I would like to check that can you help me with a command I could use ? Moreover, when I have my issue, I try later and it works..... so it's like a number of subsequent calls to different remote host is blocked... or something like that

[Genfilf]

Hi. I'm facing same issue while working with cisco gear via posh-ssh. Connection works in about 1/10 times. Wireshark shows that in success case server (cisco switch) starts key exchange init; In failure cases there is no key exchange at all. This explains timeout errors. Seems like it's Cisco problem, but I'm not sure. I'm not even sure that this comment would be helpful.

[darkoperator]

Sadly not much I can do since I do not own or have the skill to modify the library to work with Cisco version of SSH :(

Sent from my iPhone

On Nov 28, 2020, at 7:33 PM, Genfilf [email protected] wrote:

 Hi. I'm facing same issue while working with cisco gear via posh-ssh. Connection works in about 1/10 times. Wireshark shows that in success case server (cisco switch) starts key exchange init; In failure cases there is no key exchange at all. This explains timeout errors. Seems like it's Cisco problem, but I'm not sure. I'm not even sure that this comment would be helpful.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[Genfilf]

Thanks for reply. I see some issues in SSH.NET repo that might be related, so it might be a trouble with the library that used in module, or the Cisco SSH itself (but I'm not sure since I'm not much into programming).

Have a nice day!

[Patrick92x]

I've got the same issue. However, i might have some additional information that could be of interest.

The module used to work just fine in this scenario:

• Server: Windows Server 2008 R2
• Posh-SSH: version 2.2

Obviously we've migrated the server because it's end-of-life. Our new setup has the same issue as described (only 1 in 10 connections work):

• Server: Windows Server 2019
• Posh-SSH: version 2.3 (since the 2.2 version doesn't work on server 2019)

Maybe someone with the right skills can look at the things that have changed? Since i don't know where to begin.

[darkoperator]

On it was mostly me renaming the library so it stopped conflicting with MS including it for some weird storage migration component they included.

On Dec 11, 2020, at 10:26 AM, Patrick [email protected] wrote:

I've got the same issue. However, i might have some additional information that could be of interest.

The module used to work just fine in this scenario:

Server: Windows Server 2008 R2 Posh-SSH: version 2.2 Obviously we've migrated the server because it's end-of-life. Our new setup has the same issue as described (only 1 in 10 connections work):

Server: Windows Server 2019 Posh-SSH: version 2.3 (since the 2.2 version doesn't work on server 2019) Maybe someone with the right skills can look at the things that have changed? Since i don't know where to begin.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darkoperator/Posh-SSH/issues/354#issuecomment-743222181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7IHSYYVSSV5BULKHJCXTSUITZBANCNFSM4TTF2EFA.

[Patrick92x]

@darkoperator Thanks for the reply. After a day of testing during the holiday I've found the issue.

I've tested multiple versions of Posh-SSH on Windows Server 2012 (since all versions work over there because it doesn't conflict). The conclusion:

• Version 2.2 works fine
• Version 2.3 gives a timeout to some devices at random (even if its the same device)

Issue:

• Posh-SSH version 2.2 uses the "Renci.SshNet.dll" version 2016.1.0-beta2
• Posh-SSH version 2.3 uses the "Renci.SshNet.dll" (renamed to Renci.SshNetPS.dll) version 2016.1.0

Workaround:

• When i replace the Renci.SshNetPS.dll with the 2016.1.0-beta2 version the random timeouts don't happen.

I've also done some other debugging on the enddevice, and with wireshark. No real differences in the negotiation as far as i can tell. I guess the only thing left to do is submit to the SSH Repo.

P.S. It was nice to meet you at PSConf 2019 ;)

[darkoperator]

Interesting I know there where some changes in version information around OpenSSH needing clients to report version info. Wonder if Cisco devices since they forked and modified for their devices OpenSSH have a problem with that.

[Patrick92x]

It sure is weird since it's so inconsistent.

The only additional info i have is the following:

• both versions show simular traffic within wireshark
• when the connection fails it never started the key-exchange (thus the timeout)

So the question is... Why does it sometimes start the key exchange, and in other times is does not...

Unfortunately the debug on the Cisco side for SSH is just like in #340 so i have no clue.

[darkoperator]

Please test the alpha release to see if issue is resolved https://github.com/darkoperator/Posh-SSH/releases/tag/3.0.0-Alpha

This is a major release with breaking changes and it is cross platform for the latest version of powershell.

[thewizard1002]

I still have the same issue with 3.0.0. The only way I can get it to work still is when i replace the Renci.SshNetPS.dll with the 2016.1.0-beta2 version

[darkoperator]

Sadly it is an issue with he library and not the PowerShell module. I opened a Issue in their project. So far no reply from the library author.

[apagnott]

can you tell me the exact operation you are doing for this workaround?

Do you know the project for this library) page so that I could also join to mention / document the isdie

Le Thu, 25 Feb 2021 07:11:56 -0800, thewizard1002 a écrit :

I still have the same issue with 3.0.0. The only way I can get it to work still is when i replace the Renci.SshNetPS.dll with the 2016.1.0-beta2 version

-- Sent using Dekko from my Ubuntu device

[Patrick92x]

can you tell me the exact operation you are doing for this workaround?

See the workaround details I've described in #340

[darkoperator]

Version 3.1.2 includes the latest version of SSH.NET, which should address this.