Fresh installation not working any with nos BIRD

[michaelwingender]

Bug Description

When I set up a new hyper glass instance I not able to setup a device using nos bird. I have used a configuration that was working before but I had to setup a new machine. I am running hyperglass as a system user and I am using a virtualenv. Installation is working fine, I can see the web GUI but when I try to submit a query I am getting "Something went wrong message" after 1 second. On the console I can see that there should be connection attempt "[INFO] 20220225 14:44:40 | scrapli.driver.base.base_driver:897 | _pre_open_closing_log → opening connection to '127.0.0.1' on port '22'" but I do not recognize the attempt anywhere. Even tcpdump on loopback interface is not showing any incoming packets. BIRD is running on the same host, so loopback should be fine, but also the address from NIC is not working.

Expected behavior

At least a connection attempt to the server should be seen.

Steps to Reproduce

Install according to the documentation, pip3 install hyperglass should be done in an virtualenv

Local Configurations

devices.yaml
routers:
  - name: New York City, NY
    address: 127.0.0.1
    network:
      name: production
      display_name: AS65000
    credential:
      username: hyperglass
      password: hyper
    port: 22
    nos: bird
    vrfs:
      - name: global
        default: true
        ipv4:
          source_address: 192.168.42.194
        ipv6: null


hyperglass.yaml
debug: false
developer_mode: false
org_name: Beloved Hyperglass User
site_title: hyperglass
site_description: "{org_name} Network Looking Glass"
site_keywords: [hyperglass, looking glass, routing, bgp]
request_timeout: 30
listen_address: "0.0.0.0"
listen_port: 8001

Logs

The error message on the console is being seen after the timeout. The error message in the web GUI just pops up after one second.

[INFO] 20220225 14:44:40 | scrapli.driver.base.base_driver:897 | _pre_open_closing_log → opening connection to '127.0.0.1' on port '22'
[CRITICAL] 20220225 14:45:09 | hyperglass.exceptions:40 | __init__ → [DANGER] Error connecting to New York City, NY: Request timed out.
Exception in callback Loop._read_from_self
handle: <Handle Loop._read_from_self>
Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 73, in uvloop.loop.Handle._run
  File "uvloop/loop.pyx", line 359, in uvloop.loop.Loop._read_from_self
  File "uvloop/loop.pyx", line 364, in uvloop.loop.Loop._invoke_signals
  File "uvloop/loop.pyx", line 339, in uvloop.loop.Loop._ceval_process_signals
  File "/home/hyperglass/testing/lib/python3.9/site-packages/hyperglass/execution/main.py", line 39, in handler
    raise DeviceTimeout(**exc_args)
hyperglass.exceptions.DeviceTimeout: Error connecting to New York City, NY: Request timed out.

I have tested fresh CentOS7 and fresh Debian Bullseye. Both show the same behavior.

[michaelwingender]

Seems I have found the issue. If you install hyperglass currently it will install asyncssh 2.9.0. This seems not to work in this case. I have tried "pip3 remove asyncssh" and "pip3 install -Iv asyncssh==2.7.0" and with this version it is working again.

[jorqensen]

Can confirm downgrading to asyncssh 2.7 fixed the issues we've been facing for 3 days now.

Thanks @michaelwingender :+1:

[netravnen]

https://github.com/thatmattlove/hyperglass/blob/c52a6f609843177671d38bcad59b8bd658f46b64/pyproject.toml#L53

@thatmattlove possible the dependency for asyncssh can be updated to reflect asyncssh 2.9.0 should not be used ? (or a not about it can be included in the installation docs)

[netravnen]

setup a device using nos bird

Tried the same setup using frr 7.5.x (ubuntu 20.04 lts default package version). Same issue, as both bird and frr use the scrapli driver with 1.0.x releases

[budiwijaya]

Downgrade to asynchssh 2.8.1 also works.

[Mr-Sheep]

downgrade works for password authentication, private key does not work with asyncssh v2.7.0:

error message

File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/errors.py", line 159, in __call__
  await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/cors.py", line 86, in __call__
  await self.simple_response(scope, receive, send, request_headers=headers)
File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/cors.py", line 142, in simple_response
  await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/dist-packages/starlette/exceptions.py", line 82, in __call__
  raise exc from None
File "/usr/local/lib/python3.9/dist-packages/starlette/exceptions.py", line 71, in __call__
  await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 566, in __call__
  await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 227, in handle
  await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 41, in app
  response = await func(request)
File "/usr/local/lib/python3.9/dist-packages/fastapi/routing.py", line 201, in app
  raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/dist-packages/fastapi/routing.py", line 148, in run_endpoint_function
  return await dependant.call(**values)
File "/usr/local/lib/python3.9/dist-packages/hyperglass/api/routes.py", line 130, in query
  cache_output = await execute(query_data)
File "/usr/local/lib/python3.9/dist-packages/hyperglass/execution/main.py", line 74, in execute
  response = await driver.collect()
File "/usr/local/lib/python3.9/dist-packages/hyperglass/execution/drivers/ssh_scrapli.py", line 119, in collect
  async with driver as connection:
File "/usr/local/lib/python3.9/dist-packages/scrapli/driver/base/async_driver.py", line 40, in __aenter__
  await self.open()
File "/usr/local/lib/python3.9/dist-packages/scrapli/driver/base/async_driver.py", line 82, in open
  await self.transport.open()
File "/usr/local/lib/python3.9/dist-packages/scrapli/transport/plugins/asyncssh/transport.py", line 114, in open
  self.session = await asyncio.wait_for(
File "/usr/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
  return fut.result()
File "/usr/lib/python3.9/asyncio/tasks.py", line 690, in _wrap_awaitable
  return (yield from awaitable.__await__())
File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 7682, in connect
  new_options = SSHClientConnectionOptions(options, config=config, host=host,
File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 6181, in __init__
  super().__init__(options=options, last_config=last_config, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/misc.py", line 350, in __init__
  self.prepare(**self.kwargs)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 7036, in prepare
  load_keypairs(cast(KeyPairListArg, client_keys), passphrase,
File "/usr/local/lib/python3.9/dist-packages/asyncssh/public_key.py", line 3442, in load_keypairs
  read_private_key_and_certs(key_to_load, passphrase)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/public_key.py", line 3256, in read_private_key_and_certs
  key, cert = import_private_key_and_certs(read_file(filename), passphrase)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/public_key.py", line 3129, in import_private_key_and_certs
  key, end = _decode_private(data, passphrase)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/public_key.py", line 2734, in _decode_private
  key = _decode_pem_private(pem_name, headers, data, passphrase)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/public_key.py", line 2645, in _decode_pem_private
  return _decode_openssh_private(data, passphrase)
File "/usr/local/lib/python3.9/dist-packages/asyncssh/public_key.py", line 2482, in _decode_openssh_private
  raise KeyImportError('Passphrase must be specified to import '

[typicalGtaTG]

Did anyone manage to get key authentication to work?