prometheus_output_monitor not updating `num_errors`

[tscolari]

I'm trying to understand if my expectation of the value num_errors is wrong, or if I'm doing something wrong in the setup.

My intention was to use num_errors from my elasticsearch_dynamic output plugin to account for rejections when pushing to es. I've modified the elasticsearch_dynamic code so it correctly emits an error event using router.emit_error_event when it gets a 400 response back. But it seems that the counter metric for num_errors of this plugin is constantly 0.

Am I wrong in assuming that router.emit_error_event would increase it? Otherwise, if I'm sure router.emit_error_event is getting called properly, any suggestion of what kind of misconfiguration could lead to this?

[etiennejournet]

Same problem here, my fluentd shows:

error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]

But fluentd_output_status_num_errors stays null.

[tscolari]

@etiennejournet I've done some investigation on this, and it seems that fluentd_output_status_num_errors is not intended to work like that.

If your fluentd-elasticsearch-plugin has a connection error (or raise an exception), only then fluentd_output_status_num_errors will be increased.

To count 400 responses from elastic search you need to setup a <label @ERROR> to collect them. e.g.:

<label @ERROR>
   # Count error events for kubernetes.containers entries
   <match **>
     @type prometheus
     <metric>
       name fluentd_log_errors
       type counter
       desc The total number of errors from kubernetes containers output
     </metric>
   </match>
 </label>

Then that counter (fluentd_log_errors) will be incremented. Also note that this only works if you're using @type elasticsearch, the @type elasticsearch_dynamic doesn't support this error emitting.