Unable to backup Cisco NCS-540 (Cisco IOS XR)

[michaelarnauts]

This is the same issue as https://github.com/netfishers-onl/Netshot/issues/250

Hopefully, I can provide more information to troubleshoot this.

Device information: Cisco NCS-540 Cisco IOS XR Software, Version 7.9.2

Telnet used to work fine, but we've disabled telnet access, and SSH doesn't seem to work here.

Job log output from the GUI is:

[INFO] Snapshot task for device bru4-pe1 (xxx.xxx.xxx.xxx).
[INFO] Trying SSH to xxx.xxx.xxx.xxx:0 using credentials DEVICESPECIFIC-a5a04b57-fb45-47c4-91ce-500b9cabf781.
[WARN] Unable to open an SSH socket to xxx.xxx.xxx.xxx:0: Session.connect: java.net.SocketTimeoutException: Read timed out
[INFO] Auto-trying Telnet with credentials TELNET | configbackup.
[WARN] Unable to open a Telnet socket to xxx.xxx.xxx.xxx:0.
[ERROR] Error while taking the snapshot: Couldn't open either SSH or Telnet socket with the device.

Debug log is empty (0 bytes).

Netshot log output is:

2024-08-22 14:11:25,923 WARN  [NetshotRunnerScheduler_Worker-63] TaskJob: Running the task 23444975 of type onl.netfishers.netshot.work.tasks.TakeSnapshotTask
2024-08-22 14:11:25,927 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: Connecting to xxx.xxx.xxx.xxx port 22
2024-08-22 14:11:25,929 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: Connection established
2024-08-22 14:11:26,152 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: Remote version string: SSH-2.0-Cisco-2.0
2024-08-22 14:11:26,152 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: Local version string: SSH-2.0-JSCH_0.2.16
2024-08-22 14:11:26,152 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: CheckCiphers: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2024-08-22 14:11:26,153 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: CheckMacs: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512
2024-08-22 14:11:26,153 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: CheckKexes: diffie-hellman-group14-sha1,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: CheckSignatures: rsa-sha2-256,rsa-sha2-512,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-ed448
2024-08-22 14:11:26,263 DEBUG [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server_host_key proposal before known_host reordering is: rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss,ssh-ed25519
2024-08-22 14:11:26,263 DEBUG [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server_host_key proposal after known_host reordering is: rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss,ssh-ed25519
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: SSH_MSG_KEXINIT sent
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: SSH_MSG_KEXINIT received
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: KEX algorithms: ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group14-sha1,curve25519-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,[email protected]
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: host key algorithms: ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ssh-dss,rsa-sha2-512,rsa-sha2-256,ssh-rsa
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: ciphers c2s: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: ciphers s2c: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: MACs c2s: hmac-sha2-512,hmac-sha2-256,hmac-sha1
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: MACs s2c: hmac-sha2-512,hmac-sha2-256,hmac-sha1
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: compression c2s: none
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: compression s2c: none
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: languages c2s:
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: server proposal: languages s2c:
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: KEX algorithms: diffie-hellman-group18-sha512,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1,ext-info-c,[email protected]
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: host key algorithms: rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss,ssh-ed25519
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: ciphers c2s: [email protected],aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,[email protected],aes256-ctr,aes256-cbc
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: ciphers s2c: [email protected],aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,[email protected],aes256-ctr,aes256-cbc
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: MACs c2s: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-96,hmac-md5-96
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: MACs s2c: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-96,hmac-md5-96
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: compression c2s: none
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: compression s2c: none
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: languages c2s:
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: client proposal: languages s2c:
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: kex: algorithm: diffie-hellman-group16-sha512
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: kex: host key algorithm: rsa-sha2-256
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
2024-08-22 14:11:26,263 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
2024-08-22 14:11:26,274 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: SSH_MSG_KEXDH_INIT sent
2024-08-22 14:11:26,274 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: expecting SSH_MSG_KEXDH_REPLY
2024-08-22 14:11:36,282 INFO  [NetshotRunnerScheduler_Worker-63] Ssh$JschLogger: Disconnecting from xxx.xxx.xxx.xxx port 22
2024-08-22 14:11:36,282 WARN  [NetshotRunnerScheduler_Worker-63] CliScript: Unable to open an SSH connection to xxx.xxx.xxx.xxx:0.
2024-08-22 14:11:36,289 WARN  [NetshotRunnerScheduler_Worker-63] CliScript: Unable to open a Telnet connection to xxx.xxx.xxx.xxx:0.
2024-08-22 14:11:36,290 ERROR [NetshotRunnerScheduler_Worker-63] TakeSnapshotTask: Task 23444975. Error while taking the snapshot.
2024-08-22 14:11:36,296 WARN  [NetshotRunnerScheduler_Worker-63] TaskJob: End of task 23444975.
2024-08-22 14:11:37,302 WARN  [NetshotRunnerScheduler_Worker-63] TaskJob: Running the task 23444980 of type onl.netfishers.netshot.work.tasks.RunDiagnosticsTask

[michaelarnauts]

It seems older versions of IOS XR don't have this issue:

• Works: 6.8.x, 7.1.x, 7.2.x, 7.3.x
• Doesn't work: 7.6.x, 7.7.x, 7.9.x

Ofcourse, in a terminal, just ssh'ing works fine.

[SCadilhac]

Hello, can you try to increase the SSH connection timeout in Netshot config?

e.g.

netshot.cli.ssh.connectiontimeout = 30000

It seems that with newer KEX algorithms, NCS540 take quite some time to generate their crypto material.

[SIRANYAN]

Hello

I have the same issue (ssh fail connection) with :

Device : Cisco C9200 L-48P-4X version 17.6.7 I have installed netshot version 0.19.4 on Rocky Linux 9.4

Can someone help please? (sorry for my english)

Thank you

[SCadilhac]

Have you tried the suggestion with connectiontimeout?

[michaelarnauts]

It seems that updating the connection timeouts did fix the issue, although I'm not sure why.

When connecting normally with ssh, it definitely doesn't take 5 seconds (the DEFAULT_CONNECTION_TIMEOUT value) to connect.

Our current settings:

# Connection settings
netshot.cli.telnet.connectiontimeout = 10000
netshot.cli.ssh.connectiontimeout = 30000

[SCadilhac]

As you can see in the logs you provided, the selected key exchange algorithm was diffie-hellman-group16-sha512, so you you can try it and compare the connection delay:

ssh -o KexAlgorithms=diffie-hellman-group16-sha512 xxx.xxx.xxx.xxx