fluent-bit fluent-bit 3.0.3 upgrade has broken splunk output plugin when event_sourcetype

Bug Report

Describe the bug fluent-bit 3.0.3 upgrade has broken Splunk output plugin when event_sourcetype_key attribute is specified. It sends a wrong Authorization header to Splunk causing it to respond with an HTTP 401 error. It now sends Authorization: hec_token which should be Authorization: Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4 where db496524-e7e6-4ae9-b3f0-2287d8e65cd4 is a HEC token.

[2024/05/01 23:11:43] [ warn] [output:splunk:splunk.0] http_status=401:
{"text":"Invalid authorization","code":3}

To Reproduce

Steps to reproduce the problem: On Linux

Create sample log.

cat <<EOF > sample.log
log line 1
log line 2
log line 3
EOF

Run nc in a different terminal as dummy Splunk server.

echo -e 'HTTP/1.1 200 OK\r\n' | nc -l -p 8081

Start fluent-bit 3.0.3 and wait for [http_client] broken connection to 127.0.0.1:8081 ? error. As we are using nc the broken connection is expected.

docker run -ti -v $PWD/sample.log:/sample.log \
--network="host" fluent/fluent-bit:3.0.3 /fluent-bit/bin/fluent-bit \
-i tail -p path=/sample.log -p read_from_head=true \
-o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p 'event_sourcetype_key=sourcetype' \
-f 1

View the request sent to nc.

POST /services/collector/event HTTP/1.1
Host: 127.0.0.1:8081
Content-Length: 242
User-Agent: Fluent-Bit
Authorization: hec_token
Connection: keep-alive

{"time":1714604689.81654,"sourcetype":"sourcetype","event":{"log":"log line 1"}}{"time":1714604689.816549,"sourcetype":"sourcetype","event":{"log":"log line 2"}}{"time":1714604689.816551,"sourcetype":"sourcetype","event":{"log":"log line 3"}}

Expected behavior

Running the same steps with fluent-bit 3.0.2 docker image sends correct authorization header.

docker run -ti -v $PWD/sample.log:/sample.log \
--network="host" fluent/fluent-bit:3.0.2 /fluent-bit/bin/fluent-bit \
-i tail -p path=/sample.log -p read_from_head=true \
-o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p 'event_sourcetype_key=sourcetype' \
-f 1

POST /services/collector/event HTTP/1.1
Host: 127.0.0.1:8081
Content-Length: 243
User-Agent: Fluent-Bit
Authorization: Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4
Connection: keep-alive

{"time":1714604726.505695,"sourcetype":"sourcetype","event":{"log":"log line 1"}}{"time":1714604726.505703,"sourcetype":"sourcetype","event":{"log":"log line 2"}}{"time":1714604726.505704,"sourcetype":"sourcetype","event":{"log":"log line 3"}}

Your Environment

Version used: 3.0.3
Configuration:

[INPUT]
    Name             tail
    Path             /sample.log
    Read_From_Head   On
[OUTPUT]
    Name                 splunk
    Match                *
    Port                 8081
    Splunk_Token         db496524-e7e6-4ae9-b3f0-2287d8e65cd4
    event_sourcetype_key sourcetype

Environment name and version (e.g. Kubernetes? What version?): EKS 1.27
Server type and version: NA
Operating System and version: Amazon Linux 2023
Filters and plugins: tail, splunk

May 01 '24 23:05 mannbiher

Facing a very similar issue here. After some digging I found https://github.com/fluent/fluent-bit/pull/8738 which might have caused this issue. Although I am not sure how to use the newly introduced event format correctly.