Redirect fluentbit logs to data pipeline

[marwansalem]

I'd like to be be able to monitor fluentbit logs to find out what state it is in.

I know that by default the logs are sent to stderr.

I can make the logs go to a logs file using some argument to the [SERVICE] section. I can then make fluentbit watch that path and use it with an [INPUT] plugin.

I would love to skip all that and just redirect fluentbit logs to the output directly (or it can go through the data pipeline like filters)

Why? I am using fluentbit as a sidecar container, I don't want to have fluentbit write the logs on a volume that I may not have access to. It could be a waste having fluentbit write its logs to disk, then read them again to pass them through the data pipeline.

If there is an input plugin that can capture logs from STDERR or STDOUTPUT that would be great too

[patrick-stephens]

A feature request then similar to the Fluent Bit metrics plugin which handles Fluent Bit metrics to forward to whatever output but for the logs?

[patrick-stephens]

I guess the main issue would be if there is a problem with your end point, how would you ever know as no logs would be sent to it? I think the best option would be to keep local logs but add support for forwarding them as well potentially - although you would have to be careful of log "storms" when the endpoint is down as I say, you don't want it logging in a recursive loop it cannot write to the endpoint. This would be some extra complexity though.

The current approach as you say is just to tail the log file that Fluent Bit produces which keeps it simple. I'm not sure this is a huge waste of resources for the complexity trade off but is an interesting idea.

[marwansalem]

The main reason I want to capture fluentbit logs is to be able to debug, and troubleshoot, and keep track of the history.

The metric endpoint will give me some insight on what is happening at the given moment. I might need to find out what happened yesterday.

I am also actually building something based on the throttle plugin logs to detect if my log traffic is heavy.

I appreciate your input @patrick-stephens

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.