Allow control the order of Output Plugin

[sugaf1204]

Is your feature request related to a problem? Please describe.

In the Fluentd, I can't completely control the order of output plugin.

The order of plugins cannot be controlled between them that have different values for the tag field of output plugin. ~~For example, when using rewrite_tag_filter, the problem arises because we cannot control the order of customPlugin that matches the tag before rewritten and s3 plugin that matches it after rewritten.~~

apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: example
spec: 
  outputs: 
  - tag: "k8s.log.*"
     customPlugin:
       config: |
         <match k8s.log.*>
           @type rewrite_tag_filter
           <rule>
             key $.type
             pattern ^(.+)$
             tag $1.log
           </rule>
         </match>
  - tag: "A.log"
     s3:
     ......
  - tag: "B.log" 
     forward:
     ......
  - tag: "*.log" 
     elasticsearch:
     ......
  - tag: "k8s.**" # catch all
     stdout:
     ......

The order in which these are written out to config is undifined.

Describe the solution you'd like

In the current implementation, within a group of plugins with the same tag field value, the order of them can probably be controlled by metadata.name of CR. (grouping by tag field value: https://github.com/fluent/fluent-operator/blob/v2.7.0/apis/fluentd/v1alpha1/helper.go#L296) (sorted by metadata.name: https://github.com/fluent/fluent-operator/blob/v2.7.0/apis/fluentd/v1alpha1/helper.go#L124)

I can solve this problem by writing everything in single customPlugin, but it is not good because it raises the possibility of managing a very huge single config in the future. I want to control the order of all plugins according to the order of the outputs array defined in one CR, not just the metadata.name.

Additional context

No response

[benjaminhuo]

@sugaf1204 https://github.com/fluent/fluent-operator/pull/1106 pls take a look if this PR from @antrema fixed the problem you mentioned

[sugaf1204]

Sorry, I just misunderstood the behavior of rewrite_tag_filter.

I don't know how to determine the order of the rewrite_tag_filter plugin and the plugins that catch the tags that did not match any of the others.

Is there way to resolve it?

[SvenThies]

@sugaf1204, the configuration will be built always in the same way, as #1106 describes. The execution of filters is then also always the same. So the order is controlled by your configuration.

Is this still relevant? Can we close this issue?