elasticsearch-HQ
elasticsearch-HQ copied to clipboard
ElasticHQ
JS error on Metrics UI with Nginx reverse proxy setup
General information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux (Amazon Linux release 2 (Karoo))
- ElasticHQ Version: 3.5.0
- Elasticsearch Version: 6.5.4
- Python version: 3.7.1
- Browser Vendor and Version (if applicable): Google Chrome 71.0.3578.98
Issue Description
We are trying to set up a Nginx reverse proxy for ElasticHQ. Nginx and ElasticHQ is deployed on an Amazon EC2 instance and the Elasticsearch cluster to be monitored is deployed separate EC2 instances. HTTP accesses to ElasticHQ are restricted by Basic authentication.
Our nginx configuration is here:
server {
...
location /elastichq/ {
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
rewrite ^/elastichq/(.*) /$1 break;
proxy_pass http://localhost:5000;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
}
location /socket.io/ {
#rewrite ^/socket.io/(.*) /$1 break;
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
error_page 405 =200 $uri;
}
...
}
ElasticHQ works fine and we can connect it through the nginx proxy, except for Metrics UI. When we access to the Metrics UI, the graphs are not shown and an error message is continually printed in the Chrome's developer tools console.
Error: <path> attribute d: Expected number, "MNaN,190 NaN,0".
The Metrics UI works fine when I run ElasticHQ on the local machine, so I think this is caused by nginx or network setup. I found a similar issue #340, but the problem discussed in the issue seems slightly different with ours.
Are there any workarounds or solutions for this? I welcome any advice from users having similar setups to ours.
Thanks again for maintaining this excellent tool!
I'm having the same issue. It looks like the websocket connection is upgrading okay, but no frames are sent.
I think this being caused by the lack of data. Looks like it's trying to get the data point under the mouse cursor, but there's no data there. After clicking the Metrics button the errors don't start logging until the cursor is moved over a graph.
The problem arises when the clusterNodesController instantiates the SocketIO object. It's including any extra path components in the URL, and SocketIO is including this in the message data. Presumably the UI isn't expecting this extra path component, and discarding the data. Changing the argument value to the origin got the data flowing. Not an elegant solution, but now I understand what's happening.
diff -Naur elasticsearch-HQ.orig/ui/src/containers/cluster-nodes/cluster-nodes.controller.js elasticsearch-HQ/ui/src/containers/cluster-nodes/cluster-nodes.controller.js
--- elasticsearch-HQ.orig/ui/src/containers/cluster-nodes/cluster-nodes.controller.js 2019-02-01 14:56:49.000000000 -0600
+++ elasticsearch-HQ/ui/src/containers/cluster-nodes/cluster-nodes.controller.js 2019-02-01 14:49:43.000000000 -0600
@@ -27,7 +27,7 @@
baseUrl = baseUrl += `/ws`;
// console.log('---- baseUrl: ', baseUrl)
- this.socket = io(baseUrl);
+ this.socket = io(location.origin + '/ws');
this.socket.on('connect', () => {
this.connected = true;
this.socket.emit('join', {"room_name": this.clusterName + "::nodes"});
Interesting. This may be tricky to add as a pull request, as it will affect non-proxy users. I can talk to @pcasa and see if he can add a context switch in case we get blank data. We will have to set things up locally with nginx as a reverse proxy. @djlambert do you have a sample nginx config, we can use to test with?
@royrusso I had a feeling that might be the case, but had to throw in the towel for the day. Was starting to have flashbacks of battles with SocketIO and Flask-SocketIO in my own projects.
I think this is the relevant nginx.conf:
http {
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
location /elastichq {
proxy_read_timeout 90;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Script-Name /elastichq;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://elastichq:5000;
}
}
}
I'm also wrapping the WSGI application to handle the URI instead of using nginx to rewrite, to avoid this exact issue :) I didn't consider WebSocket though, or SocketIO anyways. It might work okay with native HTML5 WebSocket.
from flask import Flask
from application import application
class ReverseProxied(object):
def __init__(self, app: Flask):
self.app = app
def __call__(self, environ, start_response):
script_name = environ.get('HTTP_X_SCRIPT_NAME', '')
if script_name:
environ['SCRIPT_NAME'] = script_name
path_info = environ['PATH_INFO']
if path_info.startswith(script_name):
environ['PATH_INFO'] = path_info[len(script_name):]
return self.app(environ, start_response)
app = ReverseProxied(application)
In my use case I'm creating a containerized management server, with each application in its own path (i.e http://server/elastichq, http://server/pgadmin, etc.). nginx is proxying to the applications running under gunicorn.
@djlambert can you post what the output is for window.location object that is logged to the console? Need to know what the output is for href & origin.
---- window.location Location {replace: ƒ, assign: ƒ, href: "https://192.168.42.168/elastichq/#!/clusters/mail-services", ancestorOrigins: DOMStringList, origin: "https://192.168.42.168", …}
ancestorOrigins: DOMStringList {length: 0}
assign: ƒ ()
hash: "#!/clusters/mail-services/nodes"
host: "192.168.42.168"
hostname: "192.168.42.168"
href: "https://192.168.42.168/elastichq/#!/clusters/mail-services/nodes"
origin: "https://192.168.42.168"
pathname: "/elastichq/"
port: ""
protocol: "https:"
reload: ƒ reload()
replace: ƒ ()
search: ""
toString: ƒ toString()
valueOf: ƒ valueOf()
Symbol(Symbol.toPrimitive): undefined
__proto__: Location
Believe this is a configuration issue due to proxy, but I'm no NGINX / APACHE expert.
What happens if you add the following to your nginx.conf above the location /elastichq { line?
NOTE: you might have to undo your ReverseProxied in the flask app.
location /elastichq/ws {
proxy_pass http://elastichq:5000/ws;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Script-Name /elastichq;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_redirect default;
}
If I add that to the nginx config and remove the patch and the ReversedProxied, I'm not able to bring up elastichq at all. The ui tries to load the bundles from the site root.
GET https://192.168.253.40/static/app.bundle.css?110e62edca51cd300982 net::ERR_ABORTED 404 (Not Found)
GET https://192.168.253.40/static/commons.110e62edca51cd300982.js?110e62edca51cd300982 net::ERR_ABORTED 404 (Not Found)
If I add the ReversedProxied back I am able to load the ui, but the SocketIO library attempts to connect to https://192.168.253.40/socket.io/?EIO=3&transport=polling&t=MZ3-nvT.
If I change the nginx config to proxy `/socket.io'
location /socket.io {
proxy_pass http://elastichq:5000/socket.io;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_redirect default;
}
Then socket.io does make the connection and upgrade to websocket, but no data is displayed.
engineio logs the websocket messages:
engineio socket.receive:52 e344e10938b04235a41244a119b2405e: Received packet MESSAGE data 0/elastichq/ws,
engineio socket.send:91 e344e10938b04235a41244a119b2405e: Sending packet MESSAGE data 0/elastichq/ws
I think the problem is the message data is being send with 0/elastichq/ws, but the ui is looking for 0/ws
@djlambert sent you an email.
can you replace this.socket = io(location.origin + '/ws');
with
const path = window.location.pathname === "/" ? "/socket.io" : `${window.location.pathname}/socket.io`
this.socket = io(baseUrl, { path: path });
And report the new errors?
Hello
I just wanted to say I am currently experiencing the same problem with "/elastichq" prefix. I followed your discussion, config looks the same (except for nginx rewrite in mine), checked out feature/455 but it didn't change anything, metrics not showing. The only meaningful console error I can see is numerous Error: <path> attribute d: Expected number, "MNaN,190 NaN,0".
My application.log says:
2019-03-22 17:46:59,600 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet OPEN data {'sid': 'e44e4b27172e4c468d09b0ca30d87cca', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
2019-03-22 17:46:59,602 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet MESSAGE data 0
2019-03-22 17:46:59,769 INFO engineio socket.receive:52 e44e4b27172e4c468d09b0ca30d87cca: Received packet MESSAGE data 0/elastichq/ws,
2019-03-22 17:46:59,769 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet MESSAGE data 0/elastichq/ws
2019-03-22 17:46:59,776 INFO engineio socket.handle_get_request:101 e44e4b27172e4c468d09b0ca30d87cca: Received request to upgrade to websocket
2019-03-22 17:46:59,789 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet NOOP data None
2019-03-22 17:46:59,799 INFO engineio socket.receive:52 e44e4b27172e4c468d09b0ca30d87cca: Received packet MESSAGE data 2/elastichq/ws,["join",{"room_name":"xxx-log::nodes"}]
2019-03-22 17:46:59,800 INFO socketio server._handle_event:453 received event "join" from e44e4b27172e4c468d09b0ca30d87cca [/elastichq/ws]
2019-03-22 17:46:59,814 INFO engineio socket._websocket_handler:203 e44e4b27172e4c468d09b0ca30d87cca: Upgrade to websocket successful
2019-03-22 17:47:24,771 INFO engineio socket.receive:52 e44e4b27172e4c468d09b0ca30d87cca: Received packet PING data None
2019-03-22 17:47:24,773 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet PONG data None
2019-03-22 17:47:50,024 INFO engineio socket.receive:52 e44e4b27172e4c468d09b0ca30d87cca: Received packet PING data None
2019-03-22 17:47:50,024 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet PONG data None
2019-03-22 17:48:16,037 INFO engineio socket.receive:52 e44e4b27172e4c468d09b0ca30d87cca: Received packet PING data None
2019-03-22 17:48:16,039 INFO engineio socket.send:91 e44e4b27172e4c468d09b0ca30d87cca: Sending packet PONG data None
window.location:
ancestorOrigins: DOMStringList {length: 0}
assign: ƒ ()
hash: "#!/clusters/xxx-log/nodes"
host: "logger.xxx.pl"
hostname: "logger.xxx.pl"
href: "https://logger.xxx.pl/elastichq/#!/clusters/xxx-log/nodes"
origin: "https://logger.xxx.pl"
pathname: "/elastichq/"
port: ""
protocol: "https:"
reload: ƒ reload()
replace: ƒ ()
search: ""
toString: ƒ toString()
valueOf: ƒ valueOf()
Symbol(Symbol.toPrimitive): undefined
__proto__: Location
Moving to 3.6.0. Depends on having #485 working, so we can test nginx + HQ and close all related issues with reverse proxies.
@royrusso just to comment on this one. We've implemented your great tool on our prod stack but are battling with a couple of security concerns. Nginx gives us IP restriction as well as basic HTTP authentication so this issue would be good to be resolved. We are just going to open the firewall port to restricted IP addresses but there is still a low threat vector in that you could snoop and use the query functionality to access Live data and potentially Personally Identifiable Information. Another way around that is to make the query functionality configurable so we can just turn it off. Will leave it to the experts whether any of this is feasible or you have any alternative ideas
