Higher performance Fluent Bit Plugin available?

[pontiphex]

Hello!

I noticed on https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit that there's a higher performance plugin written in C available (https://docs.fluentbit.io/manual/pipeline/outputs/firehose). Is it also true that the parallel output plugin for Data Streams (https://docs.fluentbit.io/manual/pipeline/outputs/kinesis) is higher performance than the Go plugin?

Only asking because I saw the notice on that repo but not this one and am curious if that will affect how we use the AWS for fluent bit image.

Thanks!

[ashraf133]

Hello I haven't mesured performance but the GO plugin offers additionals features like : compression, aggregation ....

[PettitWesley]

Yes, the C/core plugin is higher performance and should have much lower resource usage, especially lower memory usage: https://docs.fluentbit.io/manual/pipeline/outputs/kinesis

But as noted, this repo (the go plugin) has some features which are not yet in the high performance version. So you have to make a decision based on which is more important for your use case.

[seasuresh]

@PettitWesley Is there a plan to incorporate the "Match," "Compression," and "Retry Limit" features from the existing plugin into the new "C" plugin? We are interested in using the new plugin, but are unable to do so because these features are currently absent.

[PettitWesley]

@seasuresh Match and Retry_Limit are basic Fluent Bit config options that work for all outputs because they are managed by the core engine. Compression support for kinesis_streams C plugin is a feature request.