[AWS for Fluent Bit] [FireLens]: AWS for Fluent Bit upgrades fluent-bit version to 2.X

[yeya24]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Update AWS for Fluent Bit's fluent bit dependency from 1.9.10 to 2.X. https://github.com/aws/aws-for-fluent-bit Fluent Bit 2.X contains a lot of new improvements since version 1.9.10. For us, the most important feature is those new V2 APIs exposed from Fluent Bit https://docs.fluentbit.io/manual/administration/monitoring#http-server.

For example, /api/v2/metrics/prometheus API exposes more useful Prometheus metrics than the V1 /api/v1/metrics/prometheus API, which are crucial for our monitoring and alerting purposes.

Which service(s) is this request for? Fargate, ECS, FireLens, EKS using Fluent Bit can use this too

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We need some metrics exposed from the V2 API so that we can understand the state of fluent bit and do proper alerting on it. With the current V1 API, some important metrics are not exposed, for example ingestion paused metrics.

Are you currently working around this issue? There is no workaround right now.

[jlizen]

Specific features I am interested in:

2.x generally:

• the yaml configuration definition (I haven't adopted in 1.9 as it is only experimental there)
• the improved storage metrics
• the new ring buffer in-memory buffering strategy and improved memory metrics
• the ability to pass fluent bit metrics into input plugins directly rather than parsing prometheus over localhost

and then 2.2.x specifically:

• improved filter chaining performance
• ingestion-paused metric for input plugins
• improvements to cloudwatch log output plugin error logs

[msobkowiak-olx]

For us - we moved from Docker Engine to containerd. 1.9 has quite some defects for CRI parser - and we had to use our own (which also broke for example multiline support).