How an emit count is being calculated

[preeti-16]

.Sample log files I tried to parse small log file (having 100logs) - 1 emit count got increased I tried to parse big log file (having 200logs)- suddenly emit got increased with large number

we were trying to find a metrics/ratio/or formula to identify failure of fluentd destinations such as (opensearch,kafka). While browsing we found flush error ratio can be used which is an error count to an emit count. But we found emit count drastically increased whereas an error increased by just 1. and hence ratio was way low to raise an alarm.

Below are the metrics that we used to calculate flush error ratio:

1. fluentd_output_status_emit_count -> To calculate emit count
2. fluentd_output_status_num_errors -> To calculate no of errors

Below are the configuration for output destination

For kafka2 plugin <match> @type kafka2 brokers kf-train-headless:9092 default_topic your_topic output_data_type json @log_level debug <buffer> @type file chunk_limit_size 8MB path /var/log/fluent/is-fluentd-buffer/kafka.logging.all total_limit_size 1024m flush_interval 3s retry_max_times 0 </buffer> <format> @type json </format> </match>

For opensearch
<match> @type opensearch @log_level info include_tag_key true host indexsearch port 9200 logstash_format true logstash_prefix log-${tag[2]} reload_connections false reconnect_on_error true reload_on_failure true request_timeout 10s <buffer> @type file chunk_limit_size 8MB path /var/log/fluent/is-fluentd-buffer/os.logging.all total_limit_size 1024m retry_max_times 0 </buffer> </match>

I tried to limit out the no of retries to 0 so to check does an error count reports to an every emit happened since output destination (kafka, opensearch) is unreachable.

So Can you tell how emit count is calculated.

[preeti-16]

Hi, Can you please check the above query asap and tell how emit count is calculated.