Fluentbit HTTP output - send one log per http request

[yorondevops]

Bug Report

Describe the bug

I need to forward logs via HTTP in JSON format to an HTTP endpoint that can only accept 1 log per POST, but the HTTP output plugin sends log lines in batches when using the JSON format, seemingly with no option to send 1 log line per request.

To Reproduce Use the following configuration:

[SERVICE]
    flush 2
    daemon off
    log_level debug
    
[INPUT]
    name tail
    path c:\something\*.log
    refresh_interval 2

[OUTPUT]
    name http
    match *
    host myhost
    port 1234
    uri /something/logs
    header content-type application/json
    format json

I have tried the json and json_stream formats from the plugin's docs, but both batch the messages (json incapsulates them in an array and json_stream separates them with commas)

Expected behavior

I expect/want fluentbit to send a single log per HTTP POST. Screenshots

Your Environment

• Version used: td-agent-bit-1.6.3
• Configuration: listed above
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version: Windows Server
• Filters and plugins: Using the tail input plugin and HTTP output plgin

Additional context

[a-thaler]

I'm having a similar problem. I want to create logs in my multiple applications in a well-defined json format whenever an audit relevant event happens. Then I want to use fluent-bit to filter the relevant audit logs, transform them into the right format and send them to the appropriate service for the audit logs. Unfortunately, this proprietary service has a simple REST API only and does not provide any bulk format.

I would not call it a bug of the http plugin but a missing feature. A possible solution could be to add a new format type like "json_single" which is sending one line per request. I would like to contribute that feature together with my colleagues.

@edsiper what do you think about that feature? Does it sound meaningful and you would accept such a contribution? Any hints on where to look in the sources in regards to switching the retry logic from a chunk-based level to a log entry based one? I saw the same json format types in the stdout plugin, that probably should be adjusted as well to be consistent?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[erichorwath]

push

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[a-thaler]

In fluentd it looks like that you can specify the buffer per output plugin and with that you can set the chunk_limit_records to 1 potentially solving the problem. Is there anything similar planned for fluent-bit?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[apratina]

Can we get some update on this? If there is a way to do this today? Otherwise can we consider this as a feature request. It would provide good flexibility to integrate with custom http server.

[apratina]

push

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[a-thaler]

unstale

[juzy]

I have the same problem, did you solve it?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[jacobnollette]

Did we ever figure this out?

[DMaxter]

I am also facing a similar issue

Is there an option to prevent grouping the logs?

[yingchen0706v]

Same issue here.

[baonq-me]

Same issue here

[agup006]

Happy to reopen this, though we don’t have someone scoped to this asap. Maybe we could have this under a webhook plugin type option?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[DMaxter]

Any updates here?

[obitoquilt]

Any updates here?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.