How to configure multiple parsers

[changdongkai]

For logs in different formats, I want to parse them according to different regular expressions. If there is no matching log, what should I do

[patrick-stephens]

It would help to have an example.

You can specify a different parser per tail input, if a parser is not matched then it will just not be applied and a default log entry will be created (this is configurable). Data is not lost. See a previous blog I did on this: https://www.couchbase.com/blog/fluent-bit-tips-tricks-log-forwarding-couchbase/#parsing as well as the docs: https://docs.fluentbit.io/manual/pipeline/inputs/tail

Key
When a message is unstructured (no parser applied), it's appended as a string under the key name log. This option allows to define an alternative name for that key.

You can also have parser filters downstream and use these to attempt to parse in different formats: https://docs.fluentbit.io/manual/pipeline/filters/parser

It all really depends on specifics, e.g. you may want a basic cri parser to handle multiline from kubernetes logs that then feeds into a log-specific set of filters. There are quite a few discussions around all this in the Slack channel as well.

[block011]

Is there a reason why we can't have conditions on parser filters similar to modify?

Something like this:

[FILTER]
    Name    parser
    Match   kube.*
    Parser my_parser
    Condition Key_Value_Equals $kubernetes['labels']['app'] myApp

This would allow users to implement multiple parsers on a single input stream.

[changdongkai]

I want to ask you a new question, how to configure fluent bit to not collect log content at service startup by default

[patrick-stephens]

@changdongkai I would recommend discussions or the Slack channel for general questions.

@block011 your feature request sounds like a good idea to me, I'd raise a separate issue on it explicitly so it is not lost (double check there is not one already).

[changdongkai]

Thank you for answering my question again. I have a new question for you, for example, there are many historical pod before today, no configuration "annotations: fluentbit. IO/exclude: "true"", can collect all the log content, but it may not be practical to redeploy these services, but I still want to exclude some logs from the historical service implementation without making configuration changes to the historical service. How should this be implemented, or is this not supported.Looking forward to your reply.

[changdongkai]

@patrick-stephens Thank you for answering my question again. I have a new question for you, for example, there are many historical pod before today, no configuration "annotations: fluentbit. IO/exclude: "true"", can collect all the log content, but it may not be practical to redeploy these services, but I still want to exclude some logs from the historical service implementation without making configuration changes to the historical service. How should this be implemented, or is this not supported.Looking forward to your reply.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.