netbox-metatype-importer
netbox-metatype-importer copied to clipboard
Onemind-Services-LLC
[Bug]: Connection reset by peer when clicking op "Load" on the Device Imports screen
NetBox plugin version
v0.3.0
NetBox version
Current: v3.7.1
But same issue for versions v3.7.0, v3.6.0, v3.5.3, ...
Steps to Reproduce
Centos Stream 8 The server has a system wide proxy set : In /etc/environment:
http_proxy=http://httppxxxx.srv.xxx.com:8080
https_proxy=http://httppxxxx.srv.xxx.com:8080
no_proxy=127.0.0.1,localhost
in Netbox Config:
HTTP_PROXIES = {
'http': 'http://httppxxxx.srv.xxx.com:8080',
'https': 'http://httppxxxx.srv.xxx.com:8080'
}
'netbox_metatype_importer': {
'github_token': '[mygithubtoken]'
In Netbox, with the plugin installed, clicking op "Load" on the Device Imports screen
Expected Behavior
the list of device types would load
Observed Behavior
Error message:
Server Error There was a problem with your request. Please contact an administrator.
The complete exception is provided below:
<class 'requests.exceptions.ConnectionError'>
('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Python version: 3.11.5 NetBox version: 3.7.0
This does not look like an issue with the plugin unless you can provide detailed steps to reproduce it. Please recheck your deployment and make sure that the repo that you are trying to access is allowed via your proxy.
Could you give detailed information about your setup? Because I'm unable to reproduce this with a simple setup connecting to GitHub via a proxy.
This is the setup:
A virtual server CentOS Stream 8 Python v3.11, NetBox v3.5.7, Plugin v0.1.0 The server is located behind a firewall and a proxy. The proxy requires authentication (basic authentication seems to work)
I've set the global environment variables via /etc/environment:
proxy=http://user:[email protected]:8080
PROXY=http://user:[email protected]:8080
https_proxy=http://user:[email protected]:8080
http_proxy=http://user:[email protected]:8080
no_proxy=127.0.0.1,localhost
HTTPS_PROXY=http://user:[email protected]:8080
These settings work for wget, git, curl, dnf, the update script of netbox
Is this info helping?
Hi,
I updated the initial info in the topic start:
- I now can use a systemwide proxy which does not need authentication
- updated netbox version and plugin version
- added the related configuration items
Testing:
curl https://raw.githubusercontent.com/netbox-community/devicetype-library/master/device-types/Cisco/2951-ISR.yaml
---
manufacturer: Cisco
model: 2951 ISR
part_number: CISCO2951/K9
slug: cisco-2951-isr
u_height: 2
is_full_depth: true
console-ports:
- name: con 0
type: rj-45
power-ports:
- name: PS-1
type: iec-60320-c14
allocated_draw: 70
maximum_draw: 340
interfaces:
- name: GigabitEthernet0/0
type: 1000base-t
- name: GigabitEthernet0/1
type: 1000base-t
- name: GigabitEthernet0/2
type: 1000base-t
So I can access github through the proxy
I also was able to turn on netbox logging, giving more insights:
`Internal Server Error: /plugins/netbox_metatype_importer/meta-device-types/load/
Traceback (most recent call last):
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
conn.connect()
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 642, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/ssl.py", line 1108, in _create
self.do_handshake()
File "/usr/lib64/python3.11/ssl.py", line 1379, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
conn.connect()
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 642, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/ssl.py", line 1108, in _create
self.do_handshake()
File "/usr/lib64/python3.11/ssl.py", line 1379, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/netbox/venv/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/django/views/generic/base.py", line 104, in view
return self.dispatch(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/netbox/utilities/views.py", line 53, in dispatch
return super().dispatch(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
return handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/netbox_metatype_importer/views.py", line 66, in post
models = gh_api.get_tree()
^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/netbox_metatype_importer/gql.py", line 84, in get_tree
data = self.get_query(query)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/netbox_metatype_importer/gql.py", line 66, in get_query
response = self.session.post(url=self.url, json={'query': query})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Not Found: /favicon.ico
`
I can do actions like:
- upgrading Netbox
- upgrading or adding plugins
- wget/curl a url
So for these actions, the proxy setup I have is working
Can you check if you can access https://api.github.com/graphql using CURL? We are using GitHub APIs rather than raw objects
It looks like that is working:
# curl --request GET --url "https://api.github.com/graphql" --header "Authorization: Bearer [MY GITHUB TOKEN]" --header "X-GitHub-Api-Version: 2022-11-28" > curltest.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2469k 100 2469k 0 0 82870 0 0:00:30 0:00:30 --:--:-- 413k
# cat curltest.txt | more
{"data":{"__schema":{"queryType":{"name":"Query"},"mutationType":{"name":"Mutation"},"subscriptionType":null,"types":[{"kind":"INPUT_OBJECT","name":"AbortQueuedMigra
tionsInput","description":"Autogenerated input type of AbortQueuedMigrations","fields":null,"inputFields":[{"name":"ownerId","description":"The ID of the organizatio
n that is running the migrations.","type":{"kind":"NON_NULL","name":null,"ofType":{"kind":"SCALAR","name":"ID","ofType":null}},"defaultValue":null},{"name":"clientMu
tationId","description":"A unique identifier for the client performing the mutation.","type":{"kind":"SCALAR","name":"String","ofType":null},"defaultValue":null}],"i
nterfaces":null,"enumValues":null,"possibleTypes":null},{"kind":"OBJECT","name":"AbortQueuedMigrationsPayload","description":"Autogenerated return type of AbortQueue
dMigrations","fields":[{"name":"clientMutationId","description":"A unique identifier for the client performing the mutation.","args":[],"type":{"kind":"SCALAR","name
":"String","ofType":null},"isDeprecated":false,"deprecationReason":null},
-------------- and so on, it's a long datastream
I have the same issue as well. Proxy in netbox works on all other connectivity except this plugin.
Internal Server Error: /plugins/netbox_metatype_importer/meta-module-types/load/
ConnectionError at /plugins/netbox_metatype_importer/meta-module-types/load/ HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa8b194bb50>: Failed to establish a new connection: [Errno 110] Connection timed out'))
curl --request GET --url "https://api.github.com/graphql" --header "Authorization: Bearer [MY GITHUB TOKEN]" --header "X-GitHub-Api-Version: 2022-11-28" > curltest.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2483k 100 2483k 0 0 734k 0 0:00:03 0:00:03 --:--:-- 734k
we've tried with different proxies, with and without authentication, etc.... so far we've not found a way to resolve this issue
I was able to confirm, that the plugin when it tries to update doesn't honor the proxy settings. on my connection it was trying to go to 140.82.112.6 port 443 directly without using the proxy.
Upgraded the the latest version v0.6.0 today, the proxy issue is still there.
Checking again if anyone has a solution for the plugin not honoring the proxy settings?
With a bit of persistence and a lot of trial and error, I came to the following which temporary solves the issue:
Although we've defined the proxy servers in the netbox config
# HTTP proxies NetBox should use when sending outbound HTTP requests (e.g. for webhooks).
HTTP_PROXIES = {
'http': 'HTTP://OURPROXYSERVER:PORT',
'https': 'HTTP://OURPROXYSERVER:PORT'
}
The plugin does not seem to honor/use those, so I thought, why not try to force the plugin to use them Have a look at the code between
# start custom code to try and fix the proxy issue
# end custom code to try and fix the proxy issue
Editing the gl.py file in the active environment:
vi /opt/netbox/venv/lib/python3.12/site-packages/netbox_metatype_importer/graphql/gql.py
I'm not a plugin developer, and I didn't find yet how to import the above config variable into the plugin; I tried with:
def __init__(self, url='https://api.github.com/graphql', token=None, owner=None, repo=None, branch=None, path=None):
self.session = requests.session()
# start custom code to try and fix the proxy issue
# Explicitly enable usage of the OS environment proxy settings
self.session.trust_env = True
# the above does not seem to work, so we'll explicitly update the session proxies from the environment variables.
self.session.proxies.update(HTTP_PROXIES)
# end custom code to try and fix the proxy issue
self.session.headers.update({'Authorization': f'token {token}'})
self.path = path
self.url = url
self.token = token
self.owner = owner
self.repo = repo
self.branch = branch
That didn't work, however, I got this far:
def __init__(self, url='https://api.github.com/graphql', token=None, owner=None, repo=None, branch=None, path=None):
self.session = requests.session()
# start custom code to try and fix the proxy issue
# Explicitly enable usage of the OS environment proxy settings
self.session.trust_env = True
# the above does not seem to work, so we'll explicitly update the session proxies from the environment variables.
proxies = {}
proxies["http"] = 'HTTP://OURPROXYSERVER:PORT'
proxies["https"] = 'HTTP://OURPROXYSERVER:PORT'
self.session.proxies.update(proxies)
# end custom code to try and fix the proxy issue
self.session.headers.update({'Authorization': f'token {token}'})
self.path = path
self.url = url
self.token = token
self.owner = owner
self.repo = repo
self.branch = branch
At this moment, I can do the load, and I was able to succesfuly import a device.
@giffordjim : not sure if this helps you?
@abhi1693 : does the above help to fix the bug in the plugin?
a step closer to an official solution?
first:
from django.conf import settings
at the top
and then
def __init__(self, url='https://api.github.com/graphql', token=None, owner=None, repo=None, branch=None, path=None):
self.session = requests.session()
# added custom code to try and fix the proxy issue
# Explicitly enable usage of environment proxy settings
self.session.trust_env = True
# the above does not seem to work, so we'll explicitly update the session proxies from the environment variables.
self.session.proxies.update(settings.HTTP_PROXIES)
# end of custom code
self.session.headers.update({'Authorization': f'token {token}'})
self.path = path
self.url = url
self.token = token
self.owner = owner
self.repo = repo
self.branch = branch
seems to work so far