[fluent-bit] Liveness/Readiness probe fails when running on AWS EKS 1.20 with Bottlerocket OS

[z0rc]

Cross reference to https://github.com/fluent/fluent-bit/issues/3521

Pods deployed by this chart fail liveness probe when running on AWS EKS 1.20 with Bottlerocket OS. As far I was able to understand, fluent-bit's http_server stops listening almost immediately after pod start, which leads to failed readiness/liveness probes. This happens only on nodes with Bottlerocket OS, regular nodes with Amazon Linux 2 run fluent-bit pods just fine.

To reproduce:

• Run AWS EKS 1.20 with Bottlerocket nodes https://docs.aws.amazon.com/eks/latest/userguide/launch-node-bottlerocket.html
• Deploy fluent-bit helm chart with default values, but dummy input and null output and empty filters
• Observe how fluent-bit pods go into CrashLoopBackOff