fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Support Redis output

Open krzwiatrzyk opened this issue 11 months ago • 2 comments

Is your feature request related to a problem? Please describe.

We use stream based topology for our logs, similar like described in Vector documentation:

https://vector.dev/docs/setup/deployment/topologies/#stream-based

where in our case, currently:

  • Filebeat is a log collector
  • Redis (x N) is a stream storage
  • Vector is ETL to transform and forward
  • and Elastic is meant as storage

To test replacing Filebeat with Fluentbit, we need support in Fluentbit to push data to Redis compatible storage (using "compatible" term here as it would work with Dragonfly or Redis forks as well).

Describe the solution you'd like Support Redis as output in Fluentbit. It should support configuring many output addresses (x N) and some load balancing to those targets (i.e via timestamp modulo).

Describe alternatives you've considered None, evaluating Fluentbit as replacement for Filebeat.

Additional context Submitting this issue as a result of discussion at Kubecon 2024 Paris.

krzwiatrzyk avatar Mar 23 '24 06:03 krzwiatrzyk