fluent-bit
fluent-bit copied to clipboard
Splunk HEC Forwarder - getaddrinfo Error - Domain name not found
Bug Report
Describe the bug amazon/aws-for-fluent-bit:2.21.5 deployed in eks 1.22 cluster fails to resolve splunkcollector domain name.
Manual event trigger from the fluent-bit container was successful.
bash-4.2# curl -k -H "Authorization: Splunk <removed>" https://splunk<removed>.net/services/collector/event -d '{"event":"hello splunk2 world"}'
{"text":"Success","code":0}
To Reproduce configmap/fluent-bit-config
[FILTER]
Name modify
Match application.*
Set index <removed>_dev_log
Set source eks
Set sourcetype <removed>_logs
Set host ${HOST_NAME}
[OUTPUT]
name splunk
match application.*
host https://splunk<removed>.net/services/collector
splunk_token <removed>
tls on
tls.verify off
[net] getaddrinfo(host='https://splunkcollector<removed>.net/services/collector', err=4):
- Steps to reproduce the problem: not sure if this is eks issue. But steps include deploy an eks 1.22 cluster with amazon linux worker instances. Deploy aws-for-fluentbit:2.21.5 chart, and configure fluentbit configmaps with splunk filter and output..
Expected behavior container logs forwarded to splunk index; searchable in splunk.
Your Environment aws eks 1.22 cluster, amazon/aws-for-fluent-bit:2.21.5
Additional context Since manual event trigger is successful, is this an issue with the host resolve logic with/without Authentication Splunk HEC token? Notice fluentd has an fullUrl option. was able to use fluentd to forward logs, but ideally would like fluent-bit to work..
try out:
- removing
https://
and/services/collector
from thehost
parameter
Thanks @edsiper.
Using the host name timeout at upstream. pod hits the alb for splunk collector on 8088, and gets a timeout. Can we do something equivalent to the full_Url, to get around the host, port and path?
Here is the output for 443 and 8088 within the fluentbit container -
bash-4.2# curl -k -H "Authorization: Splunk <removed>" https://splunk<removed>.net:8088/services/collector/event -d '{"event":"hello splunk2 world"}'
^C
bash-4.2# curl -k -H "Authorization: Splunk <removed>" https://splunk<removed>.net:443/services/collector/event -d '{"event":"hello splunk2 world"}'
{"text":"Success","code":0}
[OUTPUT]
name splunk
match application.*
host splunk<removed>.net
splunk_token <removed>
tls on
tls.verify off
[2022/09/04 20:41:43] [error] [upstream] connection #222 to splunk<removed>.net:8088 timed out after 10 seconds
[2022/09/04 20:41:43] [error] [upstream] connection #225 to splunk<removed>.net:8088 timed out after 10 seconds
[2022/09/04 20:41:47] [error] [upstream] connection #223 to splunk<removed>.net:8088 timed out after 10 seconds
@edsiper any thoughts on this approach - enabling full_URL for Splunk OUTPUT ?
Was able to utilize host and port to stream logs to splunk.. Thank you..