Events metrics appear to be inverted

[sdanbury]

I am using the following logstash config with filebeat as the only input and loggly as the only output:

The input block is provided by the incubator/helm Chart, which uses this exporter Docker image.

The output block I use is the following, overriding using my own values.yaml file.

output {
      loggly {
        proto => "https"
        host => "logs-01.loggly.com"
        key => REDACTED
      }
}

When using the community logstash grafana dashboard, I get the following results:

The underlying queries are:

Left graph:

sum(rate(logstash_node_plugin_events_in_total[$interval])) by (plugin)

Right graph:

sum(rate(logstash_node_plugin_events_out_total[$interval])) by (plugin)

It appears that they are inverted, meaning that I am seeing large input numbers for loggly, and large output numbers for beats. Very possible I am misunderstanding what "Events In" and "Events Out" means in relation to plugins, but if that is the case, the metric names are very confusing.

[phspagiari]

+1. I added some ids to debug my logstash flow and AFAIK it's inverted.

• sum(logstash_node_plugin_events_in_total{plugin_id=~"input.*"}) results 0
• sum(logstash_node_plugin_events_out_total{plugin_id=~"output.*"}) results 0
• sum(logstash_node_plugin_events_in_total{plugin_id=~"output.*"}) results the output number.
• sum(logstash_node_plugin_events_out_total{plugin_id=~"input.*"}) results the input number.

[tkornai]

I think this is more about that plugins.inputs.events.in is not an existing metric in the pipeline stats, only plugins.inputs.events.out: https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats So having this metric populated doesn't make sense: https://github.com/BonnierNews/logstash_exporter/blob/d0391964df9c3ba49a605c0c7d98399b0e54e586/collector/nodestats_collector.go#L539-L547