Number of incoming records by input plugin

[fpytloun]

Is it possible to have metric showing number of records by input plugin?

Seems it's only possible to have custom filter counting eg. by tag:

    <filter **>
      @type prometheus
      <metric>
        name fluentd_input_status_num_records_total
        type counter
        desc The total number of incoming records
        <labels>
          tag ${tag}
          hostname ${hostname}
        </labels>
      </metric>
    </filter>

I would like the same thing but per input plugin.

One solution might be to let input plugins append field with plugin name into processed record. But that would require patching input plugins, can't find any native way yet.

[fpytloun]

Forward plugin has these parameters:

      source_address_key forward_source_address
      source_hostname_key forward_source_hostname

so it's possible to use record_transformer like this:

    <filter **>
      @type record_transformer
      enable_ruby true
      auto_typecast true
      <record>
        input ${record['sqs_source_url'] != nil ? "sqs" : (record['forward_source_address'] != nil ? "forward" : nil)}
        types input:string
      </record>
    </filter>

and prometheus filter:

    <filter **>
      @type prometheus
      <metric>
        name fluentd_input_plugin_num_records_total
        type counter
        desc The total number of records by input plugin
        <labels>
          input ${input}
          hostname ${hostname}
        </labels>
      </metric>
    </filter>

[kazegusuri]

It's a little hard to summarize something per input plugin. As you mention, it can be achieved by populating those information into messages and consuming by prometheus plugin. But I recommend to add a common label to filter that watches the target input plugin so that prometheus summarizes them by query.