in_splunk is not replying with the expected ok message

[lecaros]

Bug Report

Describe the bug The in_splunk plugin used to reply with {"text":"Success","code":0} upon successful delivery of data. Now it no longer send this response.

To Reproduce

Use the following config.

splunk.yml

pipeline:
  inputs:
    - name: splunk
      port: 9883
      host: 0.0.0.0
  outputs:
    - name: stdout
      match: '*'

• Steps to reproduce the problem:

Run Fluent Bit 3.0.3

docker run --rm -it -p 9883:9883 -v $(pwd)/splunk.yml:/fluent-bit/etc/fluent-bit.yml fluent/fluent-bit:3.0.3 -c /fluent-bit/etc/fluent-bit.yml

Send a request with curl.

➜  ~ curl -vvv \
  --url http://localhost:9883/services/collector \
  --header 'Authorization: Splunk secret-token' \
  --header 'Content-Type: application/json' \
  --data '{"User":"Admin","password":"my_secret_password","Event":"Some text in the event"}'
* Host localhost:9883 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9883...
* Connected to localhost (::1) port 9883
> POST /services/collector HTTP/1.1
> Host: localhost:9883
> User-Agent: curl/8.6.0
> Accept: */*
> Authorization: Splunk secret-token
> Content-Type: application/json
> Content-Length: 81
>
< HTTP/1.1 200 OK
< content-type: application/json
* no chunk, no close, no size. Assume close to signal end
<
* Closing connection
{"text":"Success","code":0}%

We can see the expected response {"text":"Success","code":0} in the last line.

We can also see the expected log in Fluent Bit.

docker run --rm -it -p 9883:9883 -v /Users/lecaros/workspaces/fluent/fluent-bit/splunk.yml:/fluent-bit/etc/fluent-bit.yml fluent/fluent-bit:3.0.3 -c /fluent-bit/etc/fluent-bit.yml
Fluent Bit v3.0.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/06/25 21:25:50] [ info] [fluent bit] version=3.0.3, commit=3529bbb132, pid=1
[2024/06/25 21:25:50] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/06/25 21:25:50] [ info] [cmetrics] version=0.9.0
[2024/06/25 21:25:50] [ info] [ctraces ] version=0.5.1
[2024/06/25 21:25:50] [ info] [input:splunk:splunk.0] initializing
[2024/06/25 21:25:50] [ info] [input:splunk:splunk.0] storage_strategy='memory' (memory only)
[2024/06/25 21:25:50] [ info] [sp] stream processor started
[2024/06/25 21:25:50] [ info] [output:stdout:stdout.0] worker #0 started
[0] splunk.0: [[1719350754.003082510, {"hec_token"=>"Splunk secret-token"}], {"User"=>"Admin", "password"=>"my_secret_password", "Event"=>"Some text in the event"}]

Now run Fluent Bit 3.0.4 or the latest from master.

docker run --rm -it -p 9883:9883 -v /Users/lecaros/workspaces/fluent/fluent-bit/splunk.yml:/fluent-bit/etc/fluent-bit.yml fluent/fluent-bit:3.0.4 -c /fluent-bit/etc/fluent-bit.yml

Then send the same curl request:

➜  ~ curl -vvv \
  --url http://localhost:9883/services/collector \
  --header 'Authorization: Splunk secret-token' \
  --header 'Content-Type: application/json' \
  --data '{"User":"Admin","password":"my_secret_password","Event":"Some text in the event"}'
* Host localhost:9883 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9883...
* Connected to localhost (::1) port 9883
> POST /services/collector HTTP/1.1
> Host: localhost:9883
> User-Agent: curl/8.6.0
> Accept: */*
> Authorization: Splunk secret-token
> Content-Type: application/json
> Content-Length: 81
>
< HTTP/1.1 200 OK
< content-type: application/json
* no chunk, no close, no size. Assume close to signal end
<

The expected body ({"text":"Success","code":0}) is not present. Also, the * Closing connection is not present.

Expected behavior

To have the expected body ({"text":"Success","code":0}) in the response.

Your Environment

• Version used: 3.0.3 without issues. 3.0.4 or build from master have the issue.
• Filters and plugins: in_splunk

Additional context

Workaround: Add the header Connection: close to curl.

curl -vvv \
  --url http://localhost:9883/services/collector \
  --header 'Authorization: Splunk secret-token' \
  --header 'Content-Type: application/json' \
  --header 'Connection: close'  \
  --data '{"User":"Admin","password":"my_secret_password","Event":"Some text in the event"}'

This commit seems to be the culprit. I tried a version from master without this change, and it works. I'm not saying the solution is to remove the keepalive, but we need to ensure the body is sent over the connection without the need to close it.