help request: app.conf size and label_router control

[kaiohenricunha]

Describe the issue

I have deployed a multi-tenant solution leveraging fluentbit and fluentd according to this documentation.

So far, I have just 3 tenants and 1 Fluentbit ClusterFilter.

But Fluentd's app.conf is already looking enormous:

 kubectl exec -it fluentd-0 -n fluent-system -- cat /fluentd/etc/app.conf

<source>
  @type  forward
  bind  0.0.0.0
  port  24224
</source>
<match **>
  @id  main
  @type  label_router
  <route>
    @label  @b129def99623e1778c83fa647cbb2c60
    <match>
    </match>
  </route>
  <route>
    @label  @db4b58ede44bea85d919a0030f9faec4
    <match>
      namespaces  XXXXX
    </match>
  </route>
  <route>
    @label  @c9ce9b26357ba0a190e4d01fbf7ef506
    <match>
      labels  XXXX
      namespaces  XXXX
    </match>
  </route>
</match>
<label @b129def99623e1778c83fa647cbb2c60>
  <filter **>
    @type dedot
    de_dot_separator _
    de_dot_nested true
  </filter>
  <match **>
    @type opensearch
    host "XXXXXX.es.amazonaws.com"
    port 443
    logstash_format  true
    logstash_prefix logs-XXXX
    scheme https
    log_os_400_reason true
    @log_level info
    <endpoint>
      url "https://XXXX.es.amazonaws.com"
      region "XXXX"
      assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
      assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
    </endpoint>
  </match>
</label>
<label @db4b58ede44bea85d919a0030f9faec4>
  <filter **>
    @type dedot
    de_dot_separator _
    de_dot_nested true
  </filter>
  <match **>
    @type opensearch
    host "XXXXX.es.amazonaws.com"
    port 443
    logstash_format  true
    logstash_prefix logs-XXXX
    scheme https
    log_os_400_reason true
    @log_level info
    <endpoint>
      url "https://XXXX.es.amazonaws.com"
      region "XXXX"
      assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
      assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
    </endpoint>
  </match>
</label>
<label @c9ce9b26357ba0a190e4d01fbf7ef506>
  <filter **>
    @type dedot
    de_dot_separator _
    de_dot_nested true
  </filter>
  <match **>
    @type opensearch
    host "XXXX.es.amazonaws.com"
    port 443
    logstash_format  true
    logstash_prefix logs-XXXX
    scheme https
    log_os_400_reason true
    @log_level info
    <endpoint>
      url "https://XXXX.es.amazonaws.com"
      region "XXXX"
      assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
      assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
    </endpoint>
  </match>

1. Is there any limitation to the app.conf size in fluentd?

and

2. Do we have any kind of control over the label_router?

How did you install fluent operator?

Via Helm chart.

Additional context

No response

[wenchajun]

There is no limit to the size of app.conf in fluentd, so perhaps this is a point that could be optimized.

[zhu733756]

I wonder if the configuration could be splited by some comands like include xxx.conf.

There is no limit to the size of app.conf in fluentd, so perhaps this is a point that could be optimized.