in_http: add a new option add_request_header

[nokute78]

Fixes #5386.

This patch is to add a new option add_request_header to append a HTTP request header key/val to record.

name	description	default
add_request_header	Specify a regexp to append a request header to record.

Known issue

Currently, we need to treat a header as lower case string since request header is replaced by lower case in monkey.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [X] Example configuration file for the change
• [X] Debug log output from testing the change

• [X] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• [ ] Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

Configuration

[INPUT]
    Name http
    Add_Request_Header hoge

[OUTPUT]
    Name stdout

Debug / Valgrind log

curl command:

curl -d $'{"k":"v"}' -H "A_HOGE_HEADER: test" -H "content-type: application/json" http://127.0.0.1:9880

"a_hoge_header"=>"test" is appended.

[0] http.0: [1660694745.463866298, {"k"=>"v", "a_hoge_header"=>"test"}]

$ valgrind --leak-check=full bin/fluent-bit -c 5386/a.conf 
==164167== Memcheck, a memory error detector
==164167== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==164167== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==164167== Command: bin/fluent-bit -c 5386/a.conf
==164167== 
Fluent Bit v2.0.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/08/17 09:05:43] [ info] [fluent bit] version=2.0.0, commit=62431ad0ae, pid=164167
[2022/08/17 09:05:43] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/08/17 09:05:43] [ info] [cmetrics] version=0.3.5
[2022/08/17 09:05:43] [ info] [input:http:http.0] listening on 0.0.0.0:9880
[2022/08/17 09:05:43] [ info] [sp] stream processor started
[2022/08/17 09:05:43] [ info] [output:stdout:stdout.0] worker #0 started
[0] http.0: [1660694745.463866298, {"k"=>"v", "a_hoge_header"=>"test"}]
^C[2022/08/17 09:05:50] [engine] caught signal (SIGINT)
[2022/08/17 09:05:50] [ warn] [engine] service will shutdown in max 5 seconds
[2022/08/17 09:05:50] [ info] [engine] service has stopped (0 pending tasks)
[2022/08/17 09:05:50] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/08/17 09:05:50] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==164167== 
==164167== HEAP SUMMARY:
==164167==     in use at exit: 106,442 bytes in 3,650 blocks
==164167==   total heap usage: 5,644 allocs, 1,994 frees, 1,359,071 bytes allocated
==164167== 
==164167== LEAK SUMMARY:
==164167==    definitely lost: 0 bytes in 0 blocks
==164167==    indirectly lost: 0 bytes in 0 blocks
==164167==      possibly lost: 0 bytes in 0 blocks
==164167==    still reachable: 106,442 bytes in 3,650 blocks
==164167==         suppressed: 0 bytes in 0 blocks
==164167== Reachable blocks (those to which a pointer was found) are not shown.
==164167== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==164167== 
==164167== For lists of detected and suppressed errors, rerun with: -s
==164167== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[ryn9]

@nokute78 .. any chance you are also looking at add_query_params as mentioned here? https://github.com/fluent/fluent-bit/issues/5386

[nokute78]

@ryn9 I think this patch supports add_query_params .

Send http requests.

taka@taka-VirtualBox:~$ curl -d $'{"k":"v"}' -H "HTTP_TEST: query" -H "content-type: application/json" http://127.0.0.1:9880
taka@taka-VirtualBox:~$ curl -d $'{"k":"v"}' -H "QUERY_TEST: query" -H "content-type: application/json" http://127.0.0.1:9880

$ bin/fluent-bit -c issues/5904/a.conf 
Fluent Bit v2.0.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/09/10 07:10:22] [ info] [fluent bit] version=2.0.0, commit=8cb84b17d2, pid=14205
[2022/09/10 07:10:22] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/09/10 07:10:22] [ info] [cmetrics] version=0.3.5
[2022/09/10 07:10:22] [ info] [input:http:http.0] listening on 0.0.0.0:9880
[2022/09/10 07:10:22] [ info] [sp] stream processor started
[2022/09/10 07:10:22] [ info] [output:stdout:stdout.0] worker #0 started
[0] http.0: [1662761425.606426774, {"k"=>"v", "http_test"=>"query"}]
[0] http.0: [1662761430.901608808, {"k"=>"v", "query_test"=>"query"}]
^C[2022/09/10 07:10:32] [engine] caught signal (SIGINT)
[2022/09/10 07:10:32] [ warn] [engine] service will shutdown in max 5 seconds
[2022/09/10 07:10:32] [ info] [engine] service has stopped (0 pending tasks)
[2022/09/10 07:10:32] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/09/10 07:10:32] [ info] [output:stdout:stdout.0] thread worker #0 stopped

Configuration:

[INPUT]
    Name http
    Add_Request_Header ^http_
    Add_Request_Header ^query_


[OUTPUT]
    Name stdout

[nokute78]

I rebased to fix conflict.

[nokute78]

Currently, we need to treat a header as lower case string since request header is replaced by lower case in monkey.

Note: https://github.com/fluent/fluent-bit/blob/v1.9.8/lib/monkey/mk_server/mk_http_parser.c#L337-L340

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[agup006]

@edsiper something we could look at for 2.0.7?

[ryn9]

@ryn9 I think this patch supports add_query_params .

.....

@nokute78 unfortunately the example your provide does not look to test query parameter - it still was testing headers.

In this example curl -d $'{"k":"v"}' -H "content-type: application/json" http://127.0.0.1:9880/path1/path2?pkey1=pvalue1&key2=pvalue2

something like add_query_params would have the capability of extracting: pkey1=pvalue1 pkey2=pvalue2

I would, though, happily take headers now with support for parameters hopefully soon thereafter :)

ref: to my previous asks on the matter https://github.com/fluent/fluent-bit/issues/5386

[ryn9]

@nokute78 any chance you will be working on this in the near future?

[leonardo-albertovich]

I don't think we should merge this feature at the moment, while I do agree that the functionality is useful I think we need to wait for the upcoming event record format refactor to be merged and then adapt this.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.