Measuring fluentd incoming data volumes

[galactus009]

We have fluentd setup as secure forwarder. we want to find out way how to measure the amount of data in bytes or bytes/sec we are getting. I see only input counter to count number of records only. Please suggest how do we can get the objective.

<source>
  @type forward
  @id forward
  port 24225
  bind 0.0.0.0

  <security>
    self_hostname myhost
    shared_key XXXXXXXXXXXXXXXXXX
  </security>
  <transport tls>
    version TLSv1_2
    ca_path   /fluentd/etc/ssl/certs/ca_private.crt
    cert_path  /fluentd/etc/ssl/certs/XXXX.crt
    private_key_path  /fluentd/etc/ssl/private/.key
    keepalive 3600
    client_cert_auth true
  </transport>
</source>

[galactus009]

any suggestions ?

[dgonzalezruiz]

You can create a new field on your records, such as:

<record>
  log_size ${record['log'].length}
</record>

using record_modifier (or any similar alternative)

Then, using the prometheus plugin, you can setup a counter metric that uses log_size for counter instrumentation (i.e. the byte length of each record will be summed to the labeled counters).

Note that you can remove this field before output plugin execution, so that it's only used for instrumentation and not stored on your storage backend.

On prometheus side, you can work with rate() of this counter to get the logging throughput indicator you were looking for. You could also divide this bandwidth by the total number of record, to get an average log message size for a specific service.

[timown]

thanks @dgonzalezruiz, this worked great

[dkulchinsky]

came across this as I was looking into something similar, I ended up with a slight different approach:

<record>
  log_size ${record.to_json.length}
</record>

this casts the record object into a json, which then allows me to capture the length the entire record payload (not just the log key)