Internal Server Error on ssh to juniper router.

[sipvoip]

Getting a Internal Server Error on ssh to router. I am able to ssh manually with key.

Bug Description

Looks like SSH lib is not sshing into router.

Expected behavior

ssh or at least a ssh error, running tcpdump I don't even see an attempt.

Steps to Reproduce

Any query.

Local Configurations

routers:
  - name: router1 - Ashburn, VA
    address: 10.0.0.2
    network: 
      name: production
      display_name: AS398334
    credential:
      username: hyperglass
      key: /root/.ssh/id_rsa
    port: 22
    nos: juniper_junos
    vrfs:
      - name: global
        default: true
        ipv4:
          source_address: 204.89.189.2
        ipv6:
          source_address: 2602:fc11::2

Logs

[DEBUG] 20220328 11:33:26 | hyperglass.execution.drivers._construct:27 | __init__ → Constructing bgp_route query for '204.89.189.0/24'
[DEBUG] 20220328 11:33:26 | hyperglass.execution.drivers._construct:111 | queries → Constructed query: ['show route protocol bgp table inet.0 204.89.189.0/24 best detail | display xml']
[DEBUG] 20220328 11:33:26 | hyperglass.execution.drivers.ssh_scrapli:83 | collect → Connecting directly to router1 - Ashburn, VA
[INFO] 20220328 11:33:26 | scrapli.driver.base.base_driver:897 | _pre_open_closing_log → opening connection to '10.0.0.2' on port '22'
[CRITICAL] 20220328 11:33:55 | hyperglass.exceptions:40 | __init__ → [DANGER] Error connecting to router1 - Ashburn, VA: Request timed out.
Exception in callback Loop._read_from_self
handle: <Handle Loop._read_from_self>
Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 73, in uvloop.loop.Handle._run
  File "uvloop/loop.pyx", line 359, in uvloop.loop.Loop._read_from_self
  File "uvloop/loop.pyx", line 364, in uvloop.loop.Loop._invoke_signals
  File "uvloop/loop.pyx", line 339, in uvloop.loop.Loop._ceval_process_signals
  File "/usr/local/lib/python3.6/site-packages/hyperglass/execution/main.py", line 39, in handler
    raise DeviceTimeout(**exc_args)
hyperglass.exceptions.DeviceTimeout: Error connecting to router1 - Ashburn, VA: Request timed out.

Possible Solution

Environment

Server

[root@lookingglass hyperglass]# hyperglass system-info ❕ Please copy & paste this table in your bug report:

Metric	Value
hyperglass Version	1.0.4
hyperglass Path	/etc/hyperglass
Python Version	3.6.8
Node Version	14.19.1
Platform Info	Linux-5.4.17-2136.302.7.2.1.el8uek.x86_64-x86_64-with-oracle-8.5
CPU Info	Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
Logical Cores	2
Physical Cores	2
Processor Speed	3.0GHz
Total Memory	3.82 GB
Memory Utilization	18.3%
Total Disk Space	29.83 GB
Disk Utilization	15.1%

[dd1245]

I have exactly the same error when trying to use Juniper devices.

handle: <Handle Loop._read_from_self> Traceback (most recent call last): File "uvloop/cbhandles.pyx", line 73, in uvloop.loop.Handle._run File "uvloop/loop.pyx", line 359, in uvloop.loop.Loop._read_from_self File "uvloop/loop.pyx", line 364, in uvloop.loop.Loop._invoke_signals File "uvloop/loop.pyx", line 339, in uvloop.loop.Loop._ceval_process_signals File "/home/dan_kennedy/.local/lib/python3.6/site-packages/hyperglass/execution/main.py", line 39, in handler raise DeviceTimeout(**exc_args)

[dd1245]

Update: If I switch the Juniper device to a netmiko driver if works fine.

[sipvoip]

Great, how do I change the driver?

<> nathan stratton

On Wed, Mar 30, 2022 at 12:35 PM dd1245 @.***> wrote:

Update: If I switch the Juniper device to a netmiko driver if works fine.

— Reply to this email directly, view it on GitHub https://github.com/thatmattlove/hyperglass/issues/196#issuecomment-1083367427, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJRIEUONS6V64DIGEABAXLVCR7EPANCNFSM5R3RXEHQ . You are receiving this because you authored the thread.Message ID: @.***>

[dd1245]

in the devices.yaml under the device...

nos: juniper
driver: netmiko

[budiwijaya]

I'm also having this issue, with my installation:

- nos: bird
- ssh key
- driver: scrapli

Tried tcpdump both side, no attempt recorded.

However, if I switch from ssh-key to password, tcpdump recorded an attempt. But no successful to login.

On the bird side, /var/log/auth.log shows:

Apr 13 00:02:10 rsiixa sshd[7734]: Connection closed by 203.x.x.x port 53484 [preauth]
Apr 13 00:02:11 rsiixa sshd[7736]: Connection closed by 203.x.x.x port 53488 [preauth]
Apr 13 00:02:13 rsiixa sshd[7738]: Connection closed by 203.x.x.x port 53492 [preauth]

Googling found that preauth error was because the connection was attempted to use ssh-key?

-- Solved by downgrade asynchssh to 2.8.1 Reference: https://github.com/thatmattlove/hyperglass/issues/190

[andrewyager]

The correct fix seems to actually be to update scrapli

pip install -U scrapli==2022.1.30.post1

Once done, you can use the current version of asyncssh and this will work with key based authentication.

[gullio23]

hello have problem with show route with juniper mx, i have followed hints here to solve but no luck. here is the output:

[DEBUG] 20220522 19:30:24 | hyperglass.execution.drivers._construct:24 | __init__ → Constructing bgp_route query for '8.8.8.0/24' [DEBUG] 20220522 19:30:24 | hyperglass.execution.drivers._construct:111 | queries → Constructed query: ['show route protocol bgp table inet.0 8.8.8.0/24 best detail | display xml'] [DEBUG] 20220522 19:30:24 | hyperglass.execution.drivers.ssh_netmiko:57 | collect → Connecting directly to MX [ERROR] 20220522 19:30:46 | hyperglass.execution.drivers.ssh_netmiko:108 | collect → Timed-out reading channel, data not available. [ERROR] 20220522 19:30:46 | hyperglass.execution.drivers.ssh_netmiko:108 | collect → Timed-out reading channel, data not available. [CRITICAL] 20220522 19:30:46 | hyperglass.exceptions:40 | __init__ → [DANGER] Error connecting to MX: Request timed out. [CRITICAL] 20220522 19:30:46 | hyperglass.exceptions:40 | __init__ → [DANGER] Error connecting to MX: Request timed out. [CRITICAL] 20220522 19:30:53 | hyperglass.exceptions:40 | __init__ → [DANGER] Error connecting to MX: Request timed out. [CRITICAL] 20220522 19:30:53 | hyperglass.exceptions:40 | __init__ → [DANGER] Error connecting to MX: Request timed out. Exception in callback Loop._read_from_self handle: <Handle Loop._read_from_self> Traceback (most recent call last): File "uvloop/cbhandles.pyx", line 73, in uvloop.loop.Handle._run File "uvloop/loop.pyx", line 359, in uvloop.loop.Loop._read_from_self File "uvloop/loop.pyx", line 364, in uvloop.loop.Loop._invoke_signals File "uvloop/loop.pyx", line 339, in uvloop.loop.Loop._ceval_process_signals File "/usr/local/lib/python3.8/dist-packages/hyperglass/execution/main.py", line 39, in handler raise DeviceTimeout(**exc_args) hyperglass.exceptions.DeviceTimeout: Error connecting to MX: Request timed out. here the conf:

`routers:

• name: MX address: 10.10.10.1 network: name: production display_name: ASxxxx credential: username: xx password: key: /sshkey/id_rsa port: x.x.x.x nos: juniper_junos driver: netmiko vrfs:
  • name: global default: true ipv4: source_address: x.x.x.x `

traceroute and ping working as expected then i suppose that authentication works.

please advice if there is something that need to be done

[sipvoip]

Found it! Thanks.

<> nathan stratton

On Wed, Mar 30, 2022 at 12:41 PM Nathan Stratton @.***> wrote:

Great, how do I change the driver?

<> nathan stratton

On Wed, Mar 30, 2022 at 12:35 PM dd1245 @.***> wrote:

Update: If I switch the Juniper device to a netmiko driver if works fine.

— Reply to this email directly, view it on GitHub https://github.com/thatmattlove/hyperglass/issues/196#issuecomment-1083367427, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJRIEUONS6V64DIGEABAXLVCR7EPANCNFSM5R3RXEHQ . You are receiving this because you authored the thread.Message ID: @.***>