centreon-plugins icon indicating copy to clipboard operation
centreon-plugins copied to clipboard

Extend ELK monitoring for Logstash and Kibana

Open joschi99 opened this issue 3 years ago • 14 comments

It should be interessed to extend the monitoring of elasticsearch to monitor trough elastic api:

  • Logstash nodes (CPU, memory, uptime, config reloads, events received, events emmited, events latency, ...) - filter for nodes
  • Logstash pipelines (no. of pipelines, events emitted rate, no of nodes) - filter for nodes & pipelines
  • Kibana (health status, Alerts, memory, max. response time, avg load, uptime) - filter for Kibana instance

Let me know if you need some sample datas

joschi99 avatar Jun 04 '21 05:06 joschi99

If you can provide curl commands and results, it will be great!

garnier-quentin avatar Jun 04 '21 07:06 garnier-quentin

Have done some research. Logstash provides a dedicated api for monitoring. There is a full description with curls and examples here: https://www.elastic.co/guide/en/logstash/7.13/node-stats-api.html

This should allow the check if logstash is working very well, uptime, memory and cpu usage and pipelines

For pipeline (https://www.elastic.co/guide/en/logstash/7.13/node-stats-api.html#pipeline-stats) is should be important to know if pipeline is runinng (I think otherwise should not present on output), and the events emitted rate, reload errors, queue type (memory or persited)

image

image

But I have to check how to reach the api from elasticsearch server,because the local logstash API will responde only to localhost

joschi99 avatar Jun 04 '21 09:06 joschi99

It should be possible to connect with ssh and execute the curls locally? It should probably the most simply way

joschi99 avatar Jun 04 '21 09:06 joschi99

for logstash is also important to monitor open files usage. If open_file_descriptors >= max_file_descriptors the process will stop

curl -XGET 'localhost:9600/_node/stats/process?pretty' { "host" : "loa-sr-ls01", "version" : "7.10.2", "http_address" : "127.0.0.1:9600", "id" : "8caa0e0c-a4e4-4617-b518-2220dfb432e1", "name" : "loa-sr-ls01", "ephemeral_id" : "38fb9e94-8819-407a-9924-c9f01a8523c8", "status" : "green", "snapshot" : false, "pipeline" : { "workers" : 4, "batch_size" : 125, "batch_delay" : 50 }, "monitoring" : { "hosts" : [ "https://loa-sr-es01:9200" ], "username" : "logstash_system" }, "process" : { "open_file_descriptors" : 754, "peak_open_file_descriptors" : 817, "max_file_descriptors" : 16384, "mem" : { "total_virtual_in_bytes" : 7640616960 }, "cpu" : { "total_in_millis" : 2052510, "percent" : 8, "load_average" : { "1m" : 0.68, "5m" : 1.75, "15m" : 3.06 } } } }

joschi99 avatar Jun 05 '21 08:06 joschi99

@joschi99

Doing a SSH to execute locally a curl seems not very straightforward. It's not possible to access the API stats endpoints remotely ?

Sims24 avatar Jul 27 '21 14:07 Sims24

Hi @Sims24, have done some further investigation. Yes, it should be possible to configure Logstash that we can reach the API from remote, so all requests described here (https://www.elastic.co/guide/en/logstash/7.13/monitoring-logstash.html) can be done without SSH.

joschi99 avatar Jul 30 '21 06:07 joschi99

Thanks @joschi99

Last question, is there any authentication process/method to reach the API from a remote host? Or is it public?

Regards,

Sims24 avatar Aug 03 '21 13:08 Sims24

There is no authentication, this is the reason why the endpoint normally is exposed only to 127.0.0.1 and not to a public interface.

joschi99 avatar Aug 03 '21 13:08 joschi99

Ok that's a problem. We won't want to ask people to open potential security holes to monitor their Logstash.

We will do stg like: https://github.com/centreon/centreon-plugins/blob/2fd7bbb451f3dbf44b193d3566649bd0c2ed8c3e/apps/bluemind/local/mode/core.pm

It's officially in our internal backlog. Thanks

Sims24 avatar Aug 03 '21 13:08 Sims24

There are only 2 options at the moment that Elastic offers to resolve this problem:

  1. monitor logstash locally using 127.0.0.1 (through ssh?)
  2. monitor logstash centrally through X-Pack configuration, where logstash send's his metrics and status to Elasticsearch

The second option has authentication from Elasticsearch, but requires configuration on Logstash, needs almost X-Pack basic, but could be possible. What do you think about them?

joschi99 avatar Aug 03 '21 13:08 joschi99

The first one sounds like the less intrusive and with the better trade-off.

In this kind of situation, I think monitoring individually each component is more reliable than relying on additional configuration on third-party API.

Sims24 avatar Aug 03 '21 13:08 Sims24

I agree with you. But to avoid a security problem could we execute the curl commands locally through SSH or maybe yo know a better approach?

joschi99 avatar Aug 04 '21 06:08 joschi99

Yes, that is what we will do ;) The initial approach you proposed is the best one.

Sims24 avatar Aug 04 '21 10:08 Sims24

Hi @Sims24, do you have some news about this integration?

joschi99 avatar Jan 24 '22 19:01 joschi99