Error connecting to _one_ AWS ES Service

[vnagendra]

General information

We are having odd errors connecting to 1 (out of 3) AWS ES Services. As far as we can tell there is no difference (apart from the size of the clusters). They are all setup with the same module in Terraform, so no difference in security groups, etc.,

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): container.

• ElasticHQ Version: 3.5.0

• Elasticsearch Version: 6.5 (on all of them)

• Python version: Default - whatever is in the container

• Browser Vendor and Version (if applicable): Chrome

Issue Description

We have 3 ES clusters. I have very slightly modified the URLs to protect the identity of the cluster 😃. The number of characters have been preserved exactly (since it's so weird, I wanted to preserve as much as I could)

https://vpc-elasticsearch64-ws6taaeqctns4sldi5j37kkk3y.us-west-2.es.amazonaws.com
https://vpc-elasticsearch64-percolator-xlxgvaz6utabcde3pilxnaouui.us-west-2.es.amazonaws.com
https://vpc-elasticsearch6-event-zwnlsw5so76hyc7s6wugj7cknq.us-west-2.es.amazonaws.com

The last 2, work just fine. It connects etc., The first one -- no dice. There is no difference we think, and it is not obvious to me as to what could be causing the issue below

Source Code / Logs

--- Logging error ---
Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 71, in create_connection
response = requests.get(scheme + "://" + ip + ":" + port, timeout=REQUEST_TIMEOUT)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 513, in getaddrinfo
qname, addrs = _getaddrinfo_lookup(host, family, flags)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 496, in _getaddrinfo_lookup
answer = resolve(host, family, False)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 424, in resolve
return _proxy.query(name, rdtype, raise_on_no_answer=raises)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 382, in query
return end()
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 361, in end
raise result[1]
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 342, in step
a = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/resolver.py", line 916, in query
source_port=source_port)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 807, in tcp
r = dns.message.from_wire(wire, keyring=q.keyring, request_mac=q.mac)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/message.py", line 820, in from_wire
ignore_trailing)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/message.py", line 601, in __init__
self.wire = dns.wiredata.maybe_wrap(wire)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/wiredata.py", line 103, in maybe_wrap
raise ValueError("unhandled type %s" % type(wire))
ValueError: unhandled type <class 'bytearray'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/__init__.py", line 994, in emit
msg = self.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format
return fmt.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 577, in format
record.message = record.getMessage()
File "/usr/local/lib/python3.6/logging/__init__.py", line 338, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/local/lib/python3.6/site-packages/eventlet/greenthread.py", line 218, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 116, in handle
super(EventletWorker, self).handle(listener, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/__init__.py", line 43, in __call__
start_response)
File "/usr/local/lib/python3.6/site-packages/engineio/middleware.py", line 67, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
return functor(*args, **kwargs)
File "/src/elastichq/api/clusters.py", line 121, in post
enable_ssl=enable_ssl, ca_certs=ca_certs)
File "/src/elastichq/service/ConnectionService.py", line 117, in create_connection
LOG.error(message, ex)
Message: 'Unable to create connection to: https://vpc-elasticsearch64-ws6taaeqctns4sldi5j37kkk3y.us-west-2.es.amazonaws.com:443'
Arguments: (ValueError("unhandled type <class 'bytearray'>",),)
--- Logging error ---
Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 71, in create_connection
response = requests.get(scheme + "://" + ip + ":" + port, timeout=REQUEST_TIMEOUT)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 513, in getaddrinfo
qname, addrs = _getaddrinfo_lookup(host, family, flags)
2019-07-03 15:41:08,391 ERROR elastichq exceptions._request_wrapper:37 Oops! Something bad happened.
Traceback (most recent call last):
File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
return functor(*args, **kwargs)
File "/src/elastichq/api/clusters.py", line 121, in post
enable_ssl=enable_ssl, ca_certs=ca_certs)
File "/src/elastichq/service/ConnectionService.py", line 118, in create_connection
raise ex
File "/src/elastichq/service/ConnectionService.py", line 71, in create_connection
response = requests.get(scheme + "://" + ip + ":" + port, timeout=REQUEST_TIMEOUT)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 513, in getaddrinfo
qname, addrs = _getaddrinfo_lookup(host, family, flags)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 496, in _getaddrinfo_lookup
answer = resolve(host, family, False)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 424, in resolve
return _proxy.query(name, rdtype, raise_on_no_answer=raises)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 382, in query
return end()
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 361, in end
raise result[1]
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 342, in step
a = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/resolver.py", line 916, in query
source_port=source_port)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/greendns.py", line 807, in tcp
r = dns.message.from_wire(wire, keyring=q.keyring, request_mac=q.mac)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/message.py", line 820, in from_wire
ignore_trailing)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/message.py", line 601, in __init__
self.wire = dns.wiredata.maybe_wrap(wire)
File "/usr/local/lib/python3.6/site-packages/eventlet/support/dns/wiredata.py", line 103, in maybe_wrap
raise ValueError("unhandled type %s" % type(wire))
ValueError: unhandled type <class 'bytearray'>

Any suggestions on what I can try?