Authorization: ApiKey support (for HTTP and ES output plugins)

[ronaldpetty]

I do not see a way to use an ApiKey for authorization for either http or elasticsearch output plugins.

For example, if I have es running on 9200 locally, I would like to forward logs

# es
docker container run cr.fluentbit.io/fluent/fluent-bit:3.0.2 -i cpu -t cpu -o es://host.docker.internal:9200/twitter/tweet  -m "*" -p tls=off -p tls.verify=off -p 'HTTP_Header=Authorization: ApiKey QTBSNi1JNEI1SmIzenRxbXFyRjg6VVJOeHRfRmNTRnliMHV0SndPUW5ydW=='

# http to es
docker container run cr.fluentbit.io/fluent/fluent-bit:3.0.2 -i cpu -t cpu -o http://host.docker.internal:9200/twitter/tweet  -m "*" -p tls=on -p tls.verify=off -p "Header=Authorization: ApiKey QTBSNi1JNEI1SmIzenRxbXFyRjg6VVJOeHRfRmNTRnliMHV0SndPUW5ydW=="

Use of ApiKey I believe is pretty standard in web tooling for auth. I see support for HTTP basic, so suspect this is very close to working already.

[patrick-stephens]

Do you mean like this? https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#fluent-bit--elastic-cloud

Or just specify the HTTP header directly: https://docs.fluentbit.io/manual/pipeline/outputs/http

name	Description
header_tag	Specify an optional HTTP header field for the original message tag.
header	Add a HTTP header key/value pair. Multiple headers can be set.

HTTP headers can be specified generically, i.e. any header you like so not just API keys but any other random API specific detail (API key in this case is just one example: it might be called token or anything else as well).

[ronaldpetty]

Thank you for the help @patrick-stephens , for the "fluent-bit--elastic-cloud" I was thinking that only applied to elastic-cloud and not a Docker install. I did try the second option with http. However, I think I am doing something incorrect.

In my example above, I tried but suspect quoting was messing it up. What I tried here, fails in the same way but I think its closer.

docker container run cr.fluentbit.io/fluent/fluent-bit:3.0.2 -i cpu -t cpu -o http://host.docker.internal:9200/twitter/tweet  -m "*" -p tls=on -p tls.verify=off -p header="Authorization: ApiKey QTBSNi1JNEI1SmIzenRxbXFyRjg6VVJOeHRfRmNTRnliMHV0SndPUW5ydW=="

This is the error I see in either attempt.

[2024/04/23 13:59:55] [error] [output:http:http.0] host.docker.internal:9200, HTTP status=401
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/twitter/tweet]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/twitter/tweet]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024/04/23 13:59:55] [error] [engine] chunk '1-1713880794.671760650.flb' cannot be retried: task_id=3, input=cpu.0 > output=http.0

I have to review, but so far I am unclear on capitalization of properties (docs show lower, example show upper). Any advice on -p format for headers appreciated (fwiw, doing CLI style -- if I can -- because we are demoing things and minimize use of files / mounts for visual simplicity).

[ronaldpetty]

I was able to confirm headers going through with http plugin if I used a configuration file, but not with -p option.

When using es plugin, HTTP_User / HTTP_Passwd worked but I see no way for ApiKey. Will keep looking.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.