Add output plugins for popular cloud object storage (s3, gcs, azurestorage)

[nnoto]

Is your feature request related to a problem? Please describe. Fluentd offers output plugins for popular cloud object storage such as s3, gcs, and azurestorage. It would be nice to have similar functionality for Fluent-bit.

Describe the solution you'd like An output plugin for each of the following:

• AWS S3 - Send logs to an S3 bucket
• Google GCS - Send logs to a GCS bucket
• Azure storage - Send logs to an Azure blob

Describe alternatives you've considered Currently the easiest way to send logs from Fluent-bit to object store is to use the forwarding output plugin for Fluent-bit to forward the logs to a central Fluentd instance, and then use the Fluentd output plugins to send the logs to object store buckets/blobs. Alternatively, we can use the HTTP output plugin for Fluent-bit to send the logs to our own central service that writes them to object storage. Neither solutions are ideal as they require additional resources (Fluentd introduces a Ruby dependency, etc).

[StevenACoffman]

By the way, the second option you are suggesting is what we use and our code is here: https://github.com/StevenACoffman/logsink/tree/master/go-s3-logsink

We deploy it as a sidecar with fluent-bit for low latency, but it could easily be centralized.

[flixr]

See also #1004

[chestercheng]

I'm also created a output plugin for Azure blob: https://github.com/chestercheng/fluent-bit-go-azblob

Anyone wants to try it out?

[Z4ck404]

Are there any intentions to have official gcs output plugin ?

We tried with the HMAC keys + s3 output plugin => gcs but this has some limitations as mentioned in this discussion https://github.com/fluent/fluent-bit/discussions/6636#discussioncomment-8390168

[Lord-Y]

@Z4ck404 I strongly advise you to use fluent-bit with S3 plugin for GCS. I use it in production in end november until beginning of January and it wasn't working well at ALL!!!! So much memory issues and files not uploaded in GCS without any obvious reasons. Fluent-bit alone = OK Fluent-bit + S3 plugin for GCS = MESS You can check my issue here during my basic setup https://github.com/fluent/fluent-bit/issues/8238

In the end we switched to https://vector.dev/ . I'm not working at datadog and as no link with them but I can say that they did a great job and it's free to use. After my setup in production of fluent-bit + S3 plugin, I didn't report the memory issues as we switch to vector. It's been already a month and we are trully happy about vector with 0 issues!

I hope that will help you.

[Z4ck404]

Thanks a lot for your helpful reply @Lord-Y I can confirm the instability of the gcs output workaround with the s3 output plugin. Will give vector.dev a try.