elastalert-kibana-plugin icon indicating copy to clipboard operation
elastalert-kibana-plugin copied to clipboard

Published 20 hours ago • verified publisher icon bitsensor

Client request error: connect ECONNREFUSED 127.0.0.1:3030

Open yangcaixing opened this issue 6 years ago • 26 comments

Hi all, I encountered an error after install the elastalert-kibana-plugin as below, I have pre-installed the ElastAlert Server with this command "pip install "elastalert>=0.2.0b", but I don't know if need some configuration for the elastalert server, I don't use docker to setup the ELK and elastalert, so if anyone have the same issue as me, I'm stuck in it, looking forward your help, thanks in advance. elastalert

yangcaixing avatar Jun 06 '19 09:06 yangcaixing

and after I installed the elastalert server with docker image, it also display the bellow error, I also add configuration"elastalert-kibana-plugin.serverHost: 127.0.0.1, elastalert-kibana-plugin.serverPort: 9200" to the /etc/kibana/kibana.yml, if anyone can help me ,thanks a lot. elastalert-server

yangcaixing avatar Jun 06 '19 10:06 yangcaixing

Do you have any logs?

I got a similar error but that was because my container image wasn't able to start properly due to trendmicro dsagent.exe having some issue.

robincher avatar Jun 08 '19 13:06 robincher

Do you have any logs?

I got a similar error but that was because my container image wasn't able to start properly due to trendmicro dsagent.exe having some issue.

I don't find the docker logs, but it seems the docker container run up and immediately closed, so kibana cannot connect port 3033, it will display the error"502-Bad Gateway", and now I still not find the solution.

yangcaixing avatar Jun 12 '19 06:06 yangcaixing

Hi,

Are you able to see some system logs for docker? Probably it will show the reason why the container wasn't able to start properly.

robincher avatar Jun 21 '19 07:06 robincher

Hi,

Are you able to see some system logs for docker? Probably it will show the reason why the container wasn't able to start properly.

Hi Rob, I got bellow error, could you please help check ,thanks.

@bitsensor/[email protected] start /opt/elastalert-server sh ./scripts/start.sh

03:00:02.566Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. 03:00:02.568Z INFO elastalert-server: Config: Proceeding to look for normal config file. 03:00:02.570Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config. 03:00:02.601Z INFO elastalert-server: Router: Listening for GET request on /. 03:00:02.601Z INFO elastalert-server: Router: Listening for GET request on /status. 03:00:02.602Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action. 03:00:02.602Z INFO elastalert-server: Router: Listening for GET request on /status/errors. 03:00:02.602Z INFO elastalert-server: Router: Listening for GET request on /rules. 03:00:02.603Z INFO elastalert-server: Router: Listening for GET request on /rules/:id. 03:00:02.603Z INFO elastalert-server: Router: Listening for POST request on /rules/:id. 03:00:02.603Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id. 03:00:02.603Z INFO elastalert-server: Router: Listening for GET request on /templates. 03:00:02.603Z INFO elastalert-server: Router: Listening for GET request on /templates/:id. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /templates/:id. 03:00:02.604Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /test. 03:00:02.604Z INFO elastalert-server: Router: Listening for GET request on /config. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /config. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /download. 03:00:02.604Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type. 03:00:02.604Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /search/:index. 03:00:02.607Z INFO elastalert-server: ProcessController: Starting ElastAlert 03:00:02.607Z INFO elastalert-server: ProcessController: Creating index 03:00:14.583Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/elastalert/elastalert/create_index.py", line 275, in main() File "/opt/elastalert/elastalert/create_index.py", line 127, in main esversion = es.info()["version"]["number"] File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped return func(*args, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/init.py", line 259, in info return self.transport.perform_request("GET", "/", params=params) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request timeout=timeout, File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 127, in perform_request raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb5f5ba110>: Failed to establish a new connection: [Errno 111] Connection refused',))) caused by: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb5f5ba110>: Failed to establish a new connection: [Errno 111] Connection refused',)))

03:00:14.584Z ERROR elastalert-server: ProcessController: Index create exited with code 1 03:00:14.584Z WARN elastalert-server: ProcessController: ElastAlert will start but might not be able to save its data! 03:00:14.585Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none] 03:00:14.597Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 49) 03:00:14.598Z INFO elastalert-server: Server: Server listening on port 3030 03:00:14.599Z INFO elastalert-server: Server: Websocket listening on port 3333 03:00:14.599Z INFO elastalert-server: Server: Server started 03:00:15.431Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.002s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe8653fde90>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:16.431Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.001s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c150>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:19.438Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.003s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c1d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:26.447Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.002s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c110>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:26.448Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/elastalert/elastalert/elastalert.py", line 1929, in

03:00:26.448Z ERROR elastalert-server: ProcessController: sys.exit(main(sys.argv[1:])) File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

03:00:26.449Z ERROR elastalert-server: ProcessController: client.start() File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

03:00:26.449Z ERROR elastalert-server: ProcessController: self.run_all_rules() File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

03:00:26.450Z ERROR elastalert-server: ProcessController: self.send_pending_alerts() File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

03:00:26.450Z ERROR elastalert-server: ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit) File "/opt/elastalert/elastalert/elastalert.py", line 1517, in find_recent_pending_alerts

03:00:26.451Z ERROR elastalert-server: ProcessController: if self.is_atleastfive(): File "/opt/elastalert/elastalert/elastalert.py", line 159, in is_atleastfive

03:00:26.451Z ERROR elastalert-server: ProcessController: return int(self.es_version.split(".")[0]) >= 5 File "/opt/elastalert/elastalert/elastalert.py", line 155, in es_version

03:00:26.451Z ERROR elastalert-server: ProcessController: self._es_version = self.get_version() File "/opt/elastalert/elastalert/elastalert.py", line 149, in get_version info = self.writeback_es.info() File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped

03:00:26.451Z ERROR elastalert-server: ProcessController: return func(*args, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/init.py", line 259, in info

03:00:26.451Z ERROR elastalert-server: ProcessController: return self.transport.perform_request("GET", "/", params=params) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request

03:00:26.451Z ERROR elastalert-server: ProcessController: timeout=timeout, File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 127, in perform_request

03:00:26.451Z ERROR elastalert-server: ProcessController: raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError 03:00:26.451Z ERROR elastalert-server: ProcessController: : ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c110>: Failed to establish a new connection: [Errno 111] Connection refused',))) caused by: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c110>: Failed to establish a new connection: [Errno 111] Connection refused',)))

03:00:26.490Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1 03:00:26.490Z INFO elastalert-server: Server: Stopping server 03:00:26.490Z INFO elastalert-server: ProcessController: ElastAlert is not running 03:00:26.490Z INFO elastalert-server: Server: Server stopped. Bye!

yangcaixing avatar Jul 02 '19 03:07 yangcaixing

It is trying to connect to elasticsearch at localhost:9200. Where is your elsaticsearch hosted? You might need to update the config file to the exact elasticsearch hostname.

robincher avatar Jul 02 '19 14:07 robincher

It is trying to connect to elasticsearch at localhost:9200. Where is your elsaticsearch hosted? You might need to update the config file to the exact elasticsearch hostname.

Hi rob,
My elasticsearch host is localhost, same host with docker, the config file default also localhost, so I don't know why elastialert can not connect elasticsearch port 9200.

yangcaixing avatar Jul 03 '19 02:07 yangcaixing

Is your elasticsearch up and running?

curl -vvv http://localhost:9200

Do you have any anti-virus or other programs using port 9200?

robincher avatar Jul 03 '19 02:07 robincher

of course, it's up . My ELK is v.7.0.1, and in elasticseach.yml ,network.host must be set to 127.0.0.1 or localhost, otherwise elasticseach.service will also failed. I don't know if it have any impact to the issue.

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 127.0.0.1

Set a custom port for HTTP:

http.port: 9200 the docker will try to connect elastiseach 9200 about 35 seconds, then failed. root@elk:~/elastalert# docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4480117ab0a5 bitsensor/elastalert:latest "npm start" 38 seconds ago Up 34 seconds 0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp elastalert root@elk:~/elastalert# docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4480117ab0a5 bitsensor/elastalert:latest "npm start" 39 seconds ago Exited (0) 2 seconds ago elastalert

root@elk:~/elastalert# curl -vvv http://localhost:9200

  • Rebuilt URL to: http://localhost:9200/
  • Trying ::1...
  • TCP_NODELAY set
  • connect to ::1 port 9200 failed: Connection refused
  • Trying 127.0.0.1...
  • TCP_NODELAY set
  • Connected to localhost (127.0.0.1) port 9200 (#0)

GET / HTTP/1.1 Host: localhost:9200 User-Agent: curl/7.58.0 Accept: /

< HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 495 < { "name" : "elk", "cluster_name" : "elasticsearch", "cluster_uuid" : "f0ewVq1kR8CEC57OB07yAQ", "version" : { "number" : "7.0.1", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "e4efcb5", "build_date" : "2019-04-29T12:56:03.145736Z", "build_snapshot" : false, "lucene_version" : "8.0.0", "minimum_wire_compatibility_version" : "6.7.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

  • Connection #0 to host localhost left intact

yangcaixing avatar Jul 03 '19 02:07 yangcaixing

Can pase your elasticsearch.yml thanks.

robincher avatar Jul 03 '19 13:07 robincher

here is my elasticsearch.yml:

elasticsearch.txt

yangcaixing avatar Jul 04 '19 05:07 yangcaixing

Have you tried setting this in elasticsearch.yml?

network.host: 0.0.0.0

robincher avatar Jul 04 '19 06:07 robincher

yes, I tried, but for elk v7.0.1, if setup network.host:0.0.0.0, elasticsearch.service will always failed, must setup 127.0.0.1 or localhost , you can have a try.

yangcaixing avatar Jul 04 '19 06:07 yangcaixing

+1 docker container does not see the elastic running. [Errno 111] Connection refused

akamzin avatar Aug 09 '19 19:08 akamzin

This is happening for me as well.

AcidRobot avatar Oct 16 '19 21:10 AcidRobot

+1 same thing

vvvprabhakar avatar Nov 27 '19 07:11 vvvprabhakar

pass the docker ports -p 3030:3030 -p 3000:3000

vvvprabhakar avatar Dec 05 '19 14:12 vvvprabhakar

@vvvprabhakar has that solved your problem? And shall i mount these 2 ports in kibana docker?

sprksh avatar Dec 28 '19 19:12 sprksh

same issue here

santosh-kore avatar Feb 07 '20 00:02 santosh-kore

any updates here?

santosh-kore avatar Feb 07 '20 00:02 santosh-kore

It's giving me the same error. My setup is outlined below ELK containers on 7.5.2 Elastalert container on 0.2.1 Elastalert-kibana-plugin 7.5.0 (tweaked package.json to point to 7.5.2) Elastalert.yaml and config.json pointed to my server private ip (same host) and connection seems successful as it returns elastic version 7.5.2 in logs

23:05:33.328Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. 23:05:33.329Z INFO elastalert-server: Config: Proceeding to look for normal config file. 23:05:33.330Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config. 23:05:33.337Z INFO elastalert-server: Router: Listening for GET request on /. 23:05:33.337Z INFO elastalert-server: Router: Listening for GET request on /status. 23:05:33.337Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action. 23:05:33.338Z INFO elastalert-server: Router: Listening for GET request on /status/errors. 23:05:33.338Z INFO elastalert-server: Router: Listening for GET request on /rules. 23:05:33.339Z INFO elastalert-server: Router: Listening for GET request on /rules/:id. 23:05:33.339Z INFO elastalert-server: Router: Listening for POST request on /rules/:id. 23:05:33.339Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for GET request on /templates. 23:05:33.340Z INFO elastalert-server: Router: Listening for GET request on /templates/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /templates/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /test. 23:05:33.340Z INFO elastalert-server: Router: Listening for GET request on /config. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /config. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /download. 23:05:33.341Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type. 23:05:33.341Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index. 23:05:33.341Z INFO elastalert-server: Router: Listening for POST request on /search/:index. 23:05:33.344Z INFO elastalert-server: ProcessController: Starting ElastAlert 23:05:33.344Z INFO elastalert-server: ProcessController: Creating index 23:05:34.129Z INFO elastalert-server: ProcessController: Elastic Version: 7.5.2 Reading Elastic 6 index mappings: Reading index mapping 'es_mappings/6/silence.json' Reading index mapping 'es_mappings/6/elastalert_status.json' Reading index mapping 'es_mappings/6/elastalert.json' Reading index mapping 'es_mappings/6/past_elastalert.json' Reading index mapping 'es_mappings/6/elastalert_error.json' Index elastalert_status already exists. Skipping index creation.

23:05:34.130Z INFO elastalert-server: ProcessController: Index create exited with code 0 23:05:34.131Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none] 23:05:34.151Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 37) 23:05:34.154Z INFO elastalert-server: Server: Server listening on port 3030 23:05:34.157Z INFO elastalert-server: Server: Websocket listening on port 3333 23:05:34.157Z INFO elastalert-server: Server: Server started

Able to curl on localhost:3030 {"name":"elastalert-server","port":3030,"version":"3.0.0-beta.0"}

Any insights here?

kk-stateless avatar Feb 10 '20 23:02 kk-stateless

Update - I edited Kibana.yaml inside the container with the following lines and restarted it, it worked! elastalert-kibana-plugin.serverHost: 10.x.x.x elastalert-kibana-plugin.serverPort: 3030

10.x.x.x is my host ip.

kk-stateless avatar Feb 11 '20 19:02 kk-stateless

worked with setup suggested by @kk-stateless ...Thanks....

santosh-kore avatar Feb 12 '20 04:02 santosh-kore

@kk-stateless @santosh-kore I'm having the same issue. Can you please let me know how exactly you modified the kibana.yaml inside the container? Can't seem to find the file in my elastalert docker container. Unless I'm missing something... most likely because my docker-foo is very lame :-( Much appreciated.

bcisse avatar Feb 24 '20 21:02 bcisse

Kibana.yml inside the container is present under /usr/share/kibana/config

I'd recommend mapping Kibana.yml from the host to /usr/share/kibana/config/Kibana.yml inside the container so you can make changes from the host and don't have to exec into the container

kk-stateless avatar Feb 24 '20 21:02 kk-stateless

@kk-stateless thx a bunch for the tip. Did as suggested and it is working now!

bcisse avatar Feb 24 '20 22:02 bcisse