Stopped working with logstash 6

[AHeinlein]

Since upgrading to ELK 6, I now get this from this plugin:

/usr/local/lib/nagios/plugins/check_logstash.rb:195:in `rescue in get': pipeline.events.out (CheckLogstash::Result::InvalidField)
	from /usr/local/lib/nagios/plugins/check_logstash.rb:187:in `get'
	from /usr/local/lib/nagios/plugins/check_logstash.rb:183:in `get'
	from /usr/local/lib/nagios/plugins/check_logstash.rb:388:in `inflight_events_health'
	from /usr/local/lib/nagios/plugins/check_logstash.rb:342:in `health'
	from /usr/local/lib/nagios/plugins/check_logstash.rb:79:in `run'
	from /usr/local/lib/nagios/plugins/check_logstash.rb:435:in `<main>'

[uu]

It just moved in space. Find & replace pipeline.* with pipelines.main.* as a workaround.

pipeline.events.out -> pipelines.main.events.out

This sensor definitely needs some logstash version checking.

[jorgemarti]

Same as uu, if Logstash version is 6.X, any pipeline.* reference must be changed to pipelines.main.*

[rlueckl]

"change pipeline.* to pipelines.main.*" is the worst thing you can do!

Logstash 6 has introduced multiple pipelines with a good reason. Users can define their own pipelines and they can have as many as they want. Adding "main" is only good for users using default config and bad for everyone using more pipelines or who give name to their pipelines.

The check should be able to automatically find all pipelines and iterate over them or at least have an option to select the desired pipeline(s) to be checked.

I haven't written Ruby in my life, but I'll try to implement this.

[rlueckl]

I've updated the check, so it works now with Logstash 6. It'll automatically select all configured pipelines and check them. PerfData is also generated separately for each pipeline.

Examples:

Logstash 5 or older (same behavior as before):

OK - Logstash seems to be doing fine. | process.cpu.percent=23%;;;0;100 jvm.mem.heap_used_percent=65%;70;80;0;100 jvm.threads.count=317;;;0; process.open_file_descriptors=1512;13855;15485;0;16384 pipeline.events.out=4268964215c;;;0; inflight_events=871738;;;0;
OK: CPU usage in percent: 23
OK: Config reload syntax check
OK: Inflight events: 871738
OK: Heap usage at 65.00% (8442019848 out of 12797673472 bytes in use)
OK: Open file descriptors at 9.23%. (1512 out of 16384 file descriptors are open)

Logstash 6 or newer:

OK - Logstash seems to be doing fine. | process.cpu.percent=0%;;;0;100 jvm.mem.heap_used_percent=3%;70;80;0;100 jvm.threads.count=187;;;0; process.open_file_descriptors=349;13855;15485;0;16384 pipelines.pipeline-u01.events.out=614895c;;;0; inflight_events_pipeline-u01=0;;;0; pipelines.pipeline-u02.events.out=251307c;;;0; inflight_events_pipeline-u02=0;;;0;
OK: CPU usage in percent: 0
OK: Config reload syntax check: pipeline-u01: OK; pipeline-u02: OK;
OK: Inflight events: pipeline-u01: 0; pipeline-u02: 0;
OK: Heap usage at 3.00% (1034482056 out of 29951393792 bytes in use)
OK: Open file descriptors at 2.13%. (349 out of 16384 file descriptors are open)

I think that's "good enough" for now. A nice-to-have feature would be to be able to select the desired pipelines via command line parameter (something like --pipelines=all or --pipelines=this,and,that).

[widhalmt]

Thank you very, very, very much for this pull request. And I'm very, very, very, sorry I was too busy to even have a look at this. (Might have to do with the Icinga 2 book, though). The PR works and I published the merge as the new tag. I'll have a closer look into this to get it configureable like you suggested.

[akqopensystems]

Hi, is there any new status on this issue? I've checked with version 0.7.3 / LS 6.8.1 and multiple pipelines and still get the error:

./check_logstash -H localhost -p 9600 --heap-usage-threshold-warn 70 --heap-usage-threshold-crit 90
./check_logstash:207:in `rescue in get': in (CheckLogstash::Result::InvalidField)
        from ./check_logstash:199:in `get'
        from ./check_logstash:202:in `get'
        from ./check_logstash:202:in `get'
        from ./check_logstash:202:in `get'
        from ./check_logstash:195:in `get'
        from ./check_logstash:438:in `block in inflight_events_health'
        from ./check_logstash:436:in `each'
        from ./check_logstash:436:in `inflight_events_health'
        from ./check_logstash:384:in `health'
        from ./check_logstash:91:in `run'
        from ./check_logstash:555:in `<main>'

curl http://localhost:9600/
{"host":"xxx.yyy.de","version":"6.8.1","http_address":"0.0.0.0:9600","id":"7a738c7e-463e-40a5-9852-5441a24581bb","name":"xxx","build_date":"2019-06-18T14:11:07+00:00","build_sha":"e99c3a5386ec3b64789dc003fc99e0a86ede0f0f","build_snapshot":false}

[widhalmt]

Hi @akqopensystems : This is different from what I experienced. On my systems it works as expected. The issue is just open because I wanted it to be configurable.

I'll test it again on other systems.

[akqopensystems]

Hi,

we've debugged our problem further: If there are configured, but non-running, pipelines, there is no "event in" value returned by the LS API (even though"event out" is present with 0). So it isn't directly related to the original problem. Yes, our pipeline setup is somewhat non-standard.

[widhalmt]

That makes sense. The initial problem was that Elastic introduced a new pipeline for monitoring which lacked some of the values we were searching for.

Seems like the problem exists with every non-running pipeline. I'll have a look but it might take some time to tackle this. Maybe we should have a talk about your Logstash setup aside from this issue. :)

[martialblog]

Fixed in https://github.com/NETWAYS/check_logstash/pull/23