Fluent-bit Connection Retry Issue

[ShahzaibAhmedKhan31]

Issue Title

Fluent-bit continuously retries connecting to Elasticsearch with occasional connection reset errors

Issue Description

When deploying Fluent-bit using the provided configuration via Helm chart, logs are periodically forwarded to Elasticsearch. However, there seems to be an issue where Fluent-bit encounters connection errors, specifically "Connection reset by peer" and "Broken pipe," during the process of flushing chunks to Elasticsearch. Sometimes it is able to reconnect, but at other times, despite periodic retries, the issue persists intermittently, leading to a constant retrying state without successful log forwarding.

Fluent bit Configuration

serviceMonitor:
  enabled: true
config:
  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

    [PARSER]
        Name multiline
        Format regex
        Regex /(?Dec \d+ \d+\:\d+\:\d+)(?.*)/
        Time_Key  time
        Time_Format %b %d %H:%M:%S
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines Off
        Multiline on
        Parser_Firstline multiline
        storage.type filesystem
        DB /var/log/flb_kubernetes_log.db
    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
        storage.type filesystem
        DB /var/log/sys_kubernetes_log.db        
  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Logstash_Format On
        Logstash_Prefix applic
        Include_Tag_Key true
        Tag_Key tags
        Trace_Output On
        Trace_Error On
        Host elasticsearch
        HTTP_User elastic
        HTTP_Passwd tpstps
        tls  Off
        tls.verify  Off
        tls.debug  4
        Suppress_Type_Name  On
        cloud_id **********
        cloud_auth ********
        Retry_Limit 20
        Replace_Dots  On
    [OUTPUT]
        Name es
        Match host.*
        Logstash_Format On
        Logstash_Prefix node
        Host elasticsearch
        HTTP_User elastic
        HTTP_Passwd tpstps
        tls  Off
        tls.verify  Off
        tls.debug  4
        Suppress_Type_Name  On
        cloud_id **********
        cloud_auth ********
        Retry_Limit 20
        Replace_Dots  On
  service: |
    [SERVICE]
        Grace                     60
        Flush                     15
        Log_Level                 info
        Daemon                    Off
        storage.path              /var/log/flb-storage/
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On
        storage.delete_irrecoverable_chunks On
        storage.metrics On
        storage.max_chunks_up 128

Expected Behavior

Fluent-bit should establish a stable connection to Elasticsearch without encountering frequent connection reset or broken pipe errors. Log forwarding should occur consistent.

Steps to Reproduce

1. Deploy Fluent-bit using the provided configuration via helm chart.
2. Observe the Fluent-bit logs and Elasticsearch connection status.
3. Monitor the intermittent connection errors and constant retrying behavior.

Additional Information

• Fluent-bit version: latest fluent bit helm chart
• Elasticsearch version: v 7.17 (Using Managed Elastic Search)

Following are the log snippets that I am experiencing in my case

When it retries connection and is successful:

When it retries connection and is unsuccessful:

Any assistance with this issue would be highly appreciated, as I need to address it before deploying to production.

[edsiper]

hi @ShahzaibAhmedKhan31 , which Fluent Bit version are you using ? (note that I am noticing that the remote endpoint is closing the connection).

are you in the last v2.2.x ?

[ShahzaibAhmedKhan31]

Hello @edsiper, thank you for your response. I am using Fluent Bit version 2.2.0. During the time when Fluent Bit was retrying its connection, I tried to insert data through Postman using the Elasticsearch API and was successful in doing so. Therefore, I believe Elasticsearch was up and running normally at that time. (Note that I have installed Fluent Bit through the Helm chart)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[smallc2009]

@ShahzaibAhmedKhan31 have you resolved it? I have the same issue right now.