Consider buffering, aggregation, and configurable Flush interval

[StevenACoffman]

The default highest-resolution retention period for graphite is 10 seconds, but it is configurable in graphite.

If you emit a metric from this fluent-plugin-graphite any faster than every 10 seconds, multiple values for the same metric may reach Graphite at any given 10-second timespan, and only the last value would take hold and be persisted - so your data would immediately be partially lost.

To fix that, simply ensure your flush interval is at least as long as the highest-resolution retention. See more here

[cosmo0920]

Hi, I guess that this issue can fix by convert to buffered output plugin from current implementation which is called non-buffered output plugin. Does anyone want to send patch to convert buffered output plugin?

[cosmo0920]

Ah, wait. Simply converting buffered output plugin does not solve this issue. Using https://github.com/tagomoris/fluent-plugin-flowcounter to record aggregation before using fluent-plugin-graphite solves your use case for aggregation?

[StevenACoffman]

Yes, that is is what I ended up doing, but then my only option was to pick 1 minute unit, so my flush interval is courser than 10 seconds. Metrics take longer to arrive, and there are a few other minor issues.

[cosmo0920]

BufferedOutput will set flush interval to 60 seconds by default. And you can set arbitrary interval greater than 10 seconds in config with flush_interval parameter when this plugin converted to buffered output plugin. https://github.com/studio3104/fluent-plugin-graphite/pull/11/files#diff-b1fd7b4c3f191881845209220c5a6e02R55 ensures at least as long as the highest-resolution retention.