feature request: Add support for fluent-bit yaml configuration file mode

[cw-Guo]

Is your feature request related to a problem? Please describe.

do we have a plan to add the operator support for the yaml configuration file mode for fluent bit? it will be nice to resolve back pressure (possibly) if the load is extremely high.

Describe the solution you'd like

Add support for fluent-bit yaml configuration file mode

Additional context

No response

[benjaminhuo]

@cw-Guo it's good to support the yaml configuration file, are you able to work on this? It'll be great if you can do this

[cw-Guo]

I will try to figure out how to support two modes at the same time

[benjaminhuo]

I will try to figure out how to support two modes at the same time

That'll be great!

[cw-Guo]

It has been for a while since my initial feature request. A very simple proposal is as follows.

Goal:

1. we need to support the new yaml file and the old configure file at the same time
2. processor pattern need to be supported as this is the main reason why I want to use yaml config file

I am assuming that the difference between yaml config file and the classic config file is only the config format

Implementation instead of create all the plugins from beginning again to support the yaml file, we can instead change the format to yaml when generating the config files.

we can add a few new functions to generate yaml file when rendering.

Some related files are as follows

1. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusterfilter_types.go#L107-L150
2. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusterfluentbitconfig_types.go#L222-L302
3. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusterinput_types.go#L106-L145
4. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusteroutput_types.go#L139-L186
5. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/params/kvs.go#L53-L67

We just need to create similar functions like the above ones to render the yaml config file.

A new variable can be added to FluentBitConfigSpec to declare the wanted config file format.

And a few more changes are also needed in the cmd/fluent-watcher/fluentbit/main.go to differentiate the config file format (different config file path).

parser files

The YAML configuration file does not support the following sections yet: Parsers

As is stated in the fluentbit official document, parser files are not support in yaml configuration, so that we can just attach the parser files in the classic format and inject it via command line option -R https://docs.fluentbit.io/manual/administration/configuring-fluent-bit

processors

a few processors were added in the latest fluent-bit. By adding a new interface type filed into the input plugin is enough to support them, without creating a new kind processor. (reference)

@benjaminhuo Can you please review this proposal?

[benjaminhuo]

It has been for a while since my initial feature request. A very simple proposal is as follows.

Goal:

1. we need to support the new yaml file and the old configure file at the same time
2. processor pattern need to be supported as this is the main reason why I want to use yaml config file

I am assuming that the difference between yaml config file and the classic config file is only the config format

Implementation instead of create all the plugins from beginning again to support the yaml file, we can instead change the format to yaml when generating the config files.

we can add a few new functions to generate yaml file when rendering.

Some related files are as follows

1. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusterfilter_types.go#L107-L150
2. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusterfluentbitconfig_types.go#L222-L302
3. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusterinput_types.go#L106-L145
4. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/clusteroutput_types.go#L139-L186
5. https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/params/kvs.go#L53-L67

We just need to create similar functions like the above ones to render the yaml config file.

A new variable can be added to FluentBitConfigSpec to declare the wanted config file format.

And a few more changes are also needed in the cmd/fluent-watcher/fluentbit/main.go to differentiate the config file format (different config file path).

parser files

The YAML configuration file does not support the following sections yet: Parsers

As is stated in the fluentbit official document, parser files are not support in yaml configuration, so that we can just attach the parser files in the classic format and inject it via command line option -R https://docs.fluentbit.io/manual/administration/configuring-fluent-bit

processors

a few processors were added in the latest fluent-bit. By adding a new interface type filed into the input plugin is enough to support them, without creating a new kind processor. (reference)

@benjaminhuo Can you please review this proposal?

@wanjunlei @adiforluls @wenchajun @Gentleelephant pls help to review the proposal

[adiforluls]

Proposal looks okay to me.

An edge case for you when implementing: There's an option to supply raw fluent-bit configuration for plugins that haven't been implemented as a type in the operator. What happens when users select yaml format config but supply custom plugin config in the old format? https://github.com/fluent/fluent-operator/blob/master/docs/best-practice/custom-plugin.md