TCP connection failed: hostip:12201 (Connection refused), [output:gelf:gelf.2] no upstream connections available

[balimidi24]

Bug Report

Describe the bug

I am trying to get Kubernetes logs to Graylog. I Setup MongoDB, ElasticSearch, Graylog Server. Then to Kubernetes Logs to Graylog i am trying to deploy fluent-bit. I am configuring GELF output like this

inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_graylog.db
        Parser            docker
        Docker_Mode       On
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Key               log

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Merge_Log_Key       log_processed
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off		
        Annotations         Off
        Labels              On

    [FILTER]
        Name nest
        Match *
        Operation lift
        Nested_under kubernetes
    
    # [FILTER]
    #     Name record_modifier
    #     Match *
    #     Remove_key annotations
    #     Remove_key labels

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Host elasticsearch-master
        Port 9200
        Logstash_Format On
        Retry_Limit Off
        Replace_Dots On

    [OUTPUT]
        Name es
        Match host.*
        Host elasticsearch-master
        Port 9200
        Logstash_Format On
        Logstash_Prefix node
        Retry_Limit Off
        Replace_Dots On
    
    [OUTPUT]
        Name gelf
        Match *
        Host graylog.example.com 
        Port 12201
        Mode tcp
        Gelf_Short_Message_Key short_message
    
    [OUTPUT]
        Name syslog
        Match *
        Host graylog.example.com
        Port 541
        Mode udp
        Syslog_Format rfc5424
        Syslog_Maxsize 2048
        Syslog_Severity_Key  severity
        Syslog_Facility_Key  facility
        Syslog_Sd_Key sd
        Syslog_Message_key   message

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log
   
    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}

[2022/08/01 13:51:32] [error] [net] TCP connection failed: 10.0.0.0:12201 (Connection refused)
[2022/08/01 13:51:32] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/01 13:51:32] [ warn] [engine] failed to flush chunk '1-1659361891.239578990.flb', retry in 8 seconds: task_id=3, input=tail.0 > output=gelf.2 (out_id=2)
[2022/08/01 13:51:33] [error] [net] TCP connection failed: 10.0.0.0:12201 (Connection refused)
[2022/08/01 13:51:33] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/01 13:51:33] [ warn] [engine] chunk '1-1659361882.240079432.flb' cannot be retried: task_id=0, input=tail.0 > output=gelf.2
[2022/08/01 13:51:33] [error] [net] TCP connection failed: 10.0.0.0::12201 (Connection refused)
[2022/08/01 13:51:33] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/01 13:51:33] [ warn] [engine] failed to flush chunk '1-1659361892.239400072.flb', retry in 11 seconds: task_id=4, input=tail.0 > output=gelf.2 (out_id=2)
[2022/08/01 13:51:34] [error] [net] TCP connection failed: 10.0.0.0::12201 (Connection refused)
[2022/08/01 13:51:34] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/01 13:51:34] [error] [net] TCP connection failed: 10.0.0.0::12201 (Connection refused)
[2022/08/01 13:51:34] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/01 13:51:34] [ warn] [engine] chunk '1-1659361885.159984900.flb' cannot be retried: task_id=10, input=tail.0 > output=gelf.2
[2022/08/01 13:51:34] [error] [net] TCP connection failed: 10.0.0.0:12201 (Connection refused)

• Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

• Version used: Latest
• Configuration: Using Helm Chart
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes AWS EKS, 1.22
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[patrick-stephens]

Is that the right IP address from the perspective of the Fluent Bit pod? It looks like it is failing to connect.

[balimidi24]

In Host given Graylog host IP address. Then I tried to like this. I am using Graylog hostname graylog.example.com and port 12201

[2022/08/02 06:30:27] [error] [upstream] connection #140 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:30:27] [error] [upstream] connection #137 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:30:27] [error] [upstream] connection #159 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:30:27] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:30:27] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:30:27] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:30:49] [ warn] [engine] chunk '1-1659421818.793592277.flb' cannot be retried: task_id=67, input=tail.0 > output=gelf.2
[2022/08/02 06:30:49] [ warn] [engine] chunk '1-1659421819.738373884.flb' cannot be retried: task_id=11, input=tail.0 > output=gelf.2
[2022/08/02 06:30:49] [ warn] [engine] chunk '1-1659421819.994731435.flb' cannot be retried: task_id=69, input=systemd.1 > output=gelf.2

[balimidi24]

Is that the right IP address from the perspective of the Fluent Bit pod? It looks like it is failing to connect.

I updated my query.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[jin-ahn]

@balimidi24 did you find a resolution to this?