fluent-bit
fluent-bit copied to clipboard
fluent
in_kafka: add parser support and improve performance
in_kafka: add parser support and improve performance
When processing records, the in_kafka plugin lacks parser ability, This commit added support to parse payload. It has been tested with Kafka.
When processing the record, the in_kafka plugin currently will commit every single message poll from kafka. which is good in normal cases. But with this default behavior, the performance is strictly limited. This commit has added auto_commit as an option when the performance matters more. It has been tested with Kafka. There is a sample benchmark as follows.
| topic_size | topic_offset | time_cost | |
|---|---|---|---|
| before | 251 MB | 506701 | 1m30s |
| after | 251 MB | 506701 | 10s |
Signed-off-by: luoyy8 [email protected]
