Behaviour undocumented when multiple filters defined

[ppar]

Fluent-bit's documentation does not answer these questions:

• How does fluent-bit behave when the configuraition contains multiple [FILTER] definitions?
• Assuming filters are chained, in what order does are they executed?

There are various examples around where multiple [FILTER] blocks are used in a fluent-bit configuration. However, combing through the general documentation for 1.7 / 1.8:

• https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_filter
• https://docs.fluentbit.io/manual/concepts/key-concepts#filtering
• https://docs.fluentbit.io/manual/concepts/data-pipeline/filter

as well as individual filters' docs:

• https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#workflow-of-tail-kubernetes-filter

... I haven't been able to find any reference to the filters' processing order. It's not even explicitly mentioned that multiple [FILTER] blocks are allowed. One could assume they're executed in the order they appear in the config file, but that seems to me a big assumption, especially since the docs hint at some integration between the tail & systemd input modules and the kubernetes filter (see the last link above).

The only reference to this I've found is in https://github.com/fluent/fluent-bit/issues/1800 , and that issue has been "waiting-for-user" since 2019.

[fujimotos]

How does fluent-bit behave when the configuraition contains multiple [FILTER] definitions?

Fluent Bit will chain these filters plugins.

Assuming filters are chained, in what order does are they executed?

Applied in the order of definitions.

Try the following configuration if you want to check how it works. You will see that these three filters are applied in the order of "top to bottom".

[INPUT]
  Name dummy
  tag test.log

[FILTER]
  Name modify
  Set key1 added
  Match *

[FILTER]
  Name stdout
  Match *

[FILTER]
  Name modify
  Set key2 added
  Match *

[OUTPUT]
  Name stdout

[agup006]

Let's add that explicitly in the filter section in a few sentences

[agup006]

Target for 1.7 / 1.8 / master

[mahima145]

I am Trying to use multiple grep FILTER on one INPUT. I want to take docker logs from muliple container along with loglevel that is mentioned. I am using it on the same Tag that I have created in INPUT section. I am not getting any output. If I keep one Filter section than it's working. Please help me with what's wrong in this configuration.

Config: [INPUT] Name systemd Tag agent_logs Path /log/journal Systemd_Filter _SYSTEMD_UNIT=docker.service [FILTER] name grep match agent* regex CONTAINER_NAME proxy* [FILTER] name grep match agent* regex MESSAGE dbug [FILTER] name grep match agent* regex CONTAINER_NAME proxy2* [FILTER] name grep match agent* regex MESSAGE info

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[elenacbc]

Hi @mahima145 sorry for the late response. This last issue is better suited for slack, stack overflow or fluent-bit repo.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.