Errors in elastalert-server

[mukeshchouhan]

trafficstars

I am getting below error in ElastAlert. I am using latest ElastAlert Image My Elastic search Version is 6.2.3 I have around 48 Rules Configured in ElastAlert

07:06:02.208Z ERROR elastalert-server:
    ProcessController:  WARNING:elasticsearch:POST http://escoordinator.*******.com:9200/elastalert_status_status/elastalert_status [status:429 request:0.007s]

07:06:02.218Z ERROR elastalert-server:
    ProcessController:  ERROR:root:Error writing alert info to Elasticsearch: TransportError(429, u'es_rejected_execution_exception', u'rejected execution of org.elasticsearch.transport.TransportService$7@61f5f91d on EsThreadPoolExecutor[name = server.********.com/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1f0b4b9a[Running, pool size = 56, active threads = 56, queued tasks = 294, completed tasks = 2310423249]]')
    Traceback (most recent call last):
      File "/opt/elastalert/elastalert/elastalert.py", line 1518, in writeback
        doc_type=doc_type, body=body)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/__init__.py", line 319, in index
        _make_path(index, doc_type, id), params=params, body=body)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/transport.py", line 318, in perform_request
        status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/base.py", line 125, in _raise_error
        raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
    TransportError: TransportError(429, u'es_rejected_execution_exception', u'rejected execution of org.elasticsearch.transport.TransportService$7@61f5f91d on EsThreadPoolExecutor[name = server.********.com/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1f0b4b9a[Running, pool size = 56, active threads = 56, queued tasks = 294, completed tasks = 2310423249]]')

07:06:02.219Z ERROR elastalert-server:
    ProcessController:  WARNING:root:Querying from 2018-12-27 06:59 UTC to 2018-12-27 07:06 UTC took longer than 0:00:05!

07:06:02.980Z ERROR elastalert-server:
    ProcessController:  WARNING:root:Querying from 2018-12-27 06:59 UTC to 2018-12-27 07:06 UTC took longer than 0:00:05!

[martijnrondeel]

Does this issue still occur?

[mukeshchouhan]

@martijnrondeel Yes. Alerts seems to be working fine. But still I see these error continuously.

[martijnrondeel]

It seems it wants to write the status to elastalert_status_status, can you confirm that index exists in your ES instance?

[mukeshchouhan]

Yes. It Exists.

It got fixed for some days when I modified the settings es_send_get_body_as from GET to POST in elastalert.yaml file. But it is coming up again.

[iwasnobody]

@mukeshchouhan Did you fix it?

[trixprod]

Same error here.

[Sreevani871]

Any update on this error. Facing a similar issue.

[mukeshchouhan]

Can you try settings es_send_get_body_as from GET to POST in elastalert.yaml file

[Sreevani871]

Can you try settings es_send_get_body_as from GET to POST in elastalert.yaml file

Updated es_send_get_body_as: POST. Still the same error is coming. Error screenshot