ElasticHQ wont connect to cluster; multiple errors

[allen-olivas]

General information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu Server 18.04
• ElasticHQ Version: 3.5.12
• Elasticsearch Version: 7.6.1
• Python version (ignore is using docker image): 3.6.9
• Browser Vendor and Version (if applicable):

Issue Description

Sadly I cannot get this to run. It looks very cool very promising but I can't get a connection to my cluster.

I've received a number of errors, most often is the RecursionError: Maximum recursion depth exceeded. Most likely the issues i'm having are due to my own ineptitude with this stuff.

I'm using the following command and options: python3 application.py --enable-ssl true --ca-certs rootCA.pem --verify_certs false

I access the site just fine but when i try to connect using https://<serverIP>:9200 or https://:<PW>@<serverIP>:9200 i never get a connection and end up with the Maximum recursion error. Honestly I think it may have to do with my SSL selfsigned certs. I did use the option to not verify. It looks like there are issues with the client-cert-credentials in the stack trace but i have no idea how or where i state the path to my client certs.

Any help getting this up and running would be greatly appreciated!

Source Code / Logs

python3 application.py --enable-ssl true --ca-certs rootCA.pem --verify_certs false config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/usr/share/elasticsearch/plugins/elasticsearch-HQ/settings.json,/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/settings.json,/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/config/settings.json loading config /usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/config/logger.json 2020-04-09 15:26:26,793 INFO engineio server.init:140 Server initialized for eventlet. /usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq-key.pem /usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq.pem 2020-04-09 15:26:30,438 INFO elastichq ConnectionService.create_connection:60 Verify: False 2020-04-09 15:26:30,439 INFO elastichq ConnectionService.create_connection:61 Cert File: rootCA.pem 2020-04-09 15:26:30,439 INFO elastichq ConnectionService.create_connection:76 Basic Auth is True 2020-04-09 15:26:30,439 INFO elastichq ConnectionService.create_connection:78 SSL enabled 2020-04-09 15:26:30,439 INFO elastichq ConnectionService.create_connection:80 Verify Certs is False --- Logging error --- Traceback (most recent call last): File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/service/ConnectionService.py", line 83, in create_connection cert=client_cert_credentials) File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 852, in validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 314, in connect cert_reqs=resolve_cert_reqs(self.cert_reqs), File "/usr/lib/python3/dist-packages/urllib3/util/ssl.py", line 269, in create_urllib3_context context.options |= options File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) [Previous line repeated 307 more times] RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/logging/init.py", line 994, in emit msg = self.format(record) File "/usr/lib/python3.6/logging/init.py", line 840, in format return fmt.format(record) File "/usr/lib/python3.6/logging/init.py", line 577, in format record.message = record.getMessage() File "/usr/lib/python3.6/logging/init.py", line 338, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 818, in process_request proto.init(conn_state, self) File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 357, in init self.handle() File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 390, in handle self.handle_one_request() File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 466, in handle_one_request self.handle_one_response() File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 566, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.6/dist-packages/flask_socketio/init.py", line 43, in call start_response) File "/usr/local/lib/python3.6/dist-packages/engineio/middleware.py", line 67, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/usr/local/lib/python3.6/dist-packages/flask_restful/init.py", line 480, in wrapper resp = resource(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/flask/views.py", line 89, in view return self.dispatch_request(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/flask_restful/init.py", line 595, in dispatch_request resp = meth(*args, **kwargs) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/common/exceptions.py", line 29, in _request_wrapper return functor(*args, **kwargs) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/api/clusters.py", line 129, in post client_cert=client_cert) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/service/ConnectionService.py", line 173, in create_connection LOG.error(message, ex) Message: 'Unable to create connection to: https://<serverIP>:9200' Arguments: (RecursionError('maximum recursion depth exceeded',),) --- Logging error --- Traceback (most recent call last): File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/service/ConnectionService.py", line 83, in create_connection cert=client_cert_credentials) File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 852, in validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 314, in connect cert_reqs=resolve_cert_reqs(self.cert_reqs), File "/usr/lib/python3/dist-packages/urllib3/util/ssl.py", line 269, in create_urllib3_context context.options |= options File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) [Previous line repeated 307 more times] RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/logging/handlers.py", line 71, in emit if self.shouldRollover(record): File "/usr/lib/python3.6/logging/handlers.py", line 187, in shouldRollover msg = "%s\n" % self.format(record) File "/usr/lib/python3.6/logging/init.py", line 840, in format return fmt.format(record) File "/usr/lib/python3.6/logging/init.py", line 577, in format record.message = record.getMessage() File "/usr/lib/python3.6/logging/init.py", line 338, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 818, in process_request proto.init(conn_state, self) File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 357, in init self.handle() File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 390, in handle self.handle_one_request() File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 466, in handle_one_request self.handle_one_response() File "/usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py", line 566, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.6/dist-packages/flask_socketio/init.py", line 43, in call start_response) File "/usr/local/lib/python3.6/dist-packages/engineio/middleware.py", line 67, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/usr/local/lib/python3.6/dist-packages/flask_restful/init.py", line 480, in wrapper resp = resource(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/flask/views.py", line 89, in view return self.dispatch_request(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/flask_restful/init.py", line 595, in dispatch_request resp = meth(*args, **kwargs) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/common/exceptions.py", line 29, in _request_wrapper return functor(*args, **kwargs) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/api/clusters.py", line 129, in post client_cert=client_cert) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/service/ConnectionService.py", line 173, in create_connection LOG.error(message, ex) Message: 'Unable to create connection to: https://<serverIP>:9200' Arguments: (RecursionError('maximum recursion depth exceeded',),) 2020-04-09 15:26:30,455 ERROR elastichq exceptions._request_wrapper:37 Oops! Something bad happened. Traceback (most recent call last): File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/common/exceptions.py", line 29, in _request_wrapper return functor(*args, **kwargs) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/api/clusters.py", line 129, in post client_cert=client_cert) File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/service/ConnectionService.py", line 174, in create_connection raise ex File "/usr/share/elasticsearch/plugins/elasticsearch-HQ/elastichq/service/ConnectionService.py", line 83, in create_connection cert=client_cert_credentials) File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 852, in validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 314, in connect cert_reqs=resolve_cert_reqs(self.cert_reqs), File "/usr/lib/python3/dist-packages/urllib3/util/ssl.py", line 269, in create_urllib3_context context.options |= options File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) File "/usr/lib/python3.6/ssl.py", line 465, in options super(SSLContext, SSLContext).options.set(self, value) [Previous line repeated 307 more times] RecursionError: maximum recursion depth exceeded