Helm Chart: allow advanced/custom configuation of output plugins

[hostalp]

Is your feature request related to a problem? Please describe.

I use fluentd with the elsticsearch output plugin, installed from Helm chart. The related part of my Helm chart values.yaml is as follows:

fluentd:
  ...
  output:
    es:
      enable: true
      host: ...
      port: 9200
      logstashPrefix: k8s_logs_sb
      buffer:
        enable: true
        type: file
        path: /buffers/es

Now I'm experiencing some performance issues with ES calls failing due to timeout (read timeout reached), therefore I need to perform some configuration tuning.

This is however not so easy with your Helm chart as it doesn't allow anything beyoned the above to be configured by default (and many parameters aren't even supported in the fluentd ClusterOutput or Output resource so it isn't possible to specify them in custom resources either).

Describe the solution you'd like

Would it be possible to add at least some custom output plugin configuration section where one would directly specify the plugin parameters (in this case those documented at https://github.com/uken/fluent-plugin-elasticsearch#configuration) that would be then added to the generated configuration?

This would be useful for both fluentd and fluentbit output plugins and for the custom resources Output and ClusterOutput as well.

Describe alternatives you've considered

I (rather temporarily) added the request_timeout parameter to the config via editing the fluentd-config secret and restarting the fluentd statefulset, but that isn't quite straightforward (one needs to decode tbe base64 encoded config, modify it, reencode and save it).

Additional context

No response

[wenchajun]

Okay, thank you for your contribution. This is a very good suggestion and we will consider these issues with reference to the documentation you gave us.

[hostalp]

I'm also encountering the issue https://github.com/uken/fluent-plugin-elasticsearch/issues/785 which could be remedied with setting suppress_type_name true but again it's currently impossible to specify it via Helm chart values.

As there's quite a lot of possible settings, it would be probably easier to add some "custom" configuration section instead of directly supporting each setting supported by the particular plugin.

[tanmx]

@hostalp I have the same problem

[benjaminhuo]

@hostalp @tanmx, we already have the custom plugin feature that might solve your problem in a different way, see https://github.com/fluent/fluent-operator/blob/master/docs/best-practice/custom-plugin.md . But what you mentioned is a custom section feature for existing strong-typed plugins like es and we'd like to see a proposal and a following PR from the community.

[tanmx]

@benjaminhuo thanks a lot, customplugin olve my problem

[kaiohenricunha]

@tanmx could you please show how you used it in addition to your previous fluentd configuration?

I'm trying to add the following configuration to my fluentd opensearch output:

  <endpoint>
    url https://my-domain-endpoint.us-west-2.es.amazonaws.com
    region "#{ENV['AWS_REGION']}"
    assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
    assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
  </endpoint>

By applying this:

apiVersion: fluentd.fluent.io/v1alpha1
kind: Fluentd
metadata:
  name: fluentd
  namespace: fluent-system
  labels:
    app.kubernetes.io/name: fluentd
spec:
  globalInputs:
    - forward:
        bind: 0.0.0.0
        port: 24224
  replicas: 1
  image: kubesphere/fluentd:v1.15.3
  resources:
    limits:
      cpu: 500m
      memory: 500Mi
    requests:
      cpu: 100m
      memory: 128Mi
  fluentdCfgSelector:
    matchLabels:
      config.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  labels:
    config.fluentd.fluent.io/enabled: "true"
  name: fluentd-config
spec:
  clusterFilterSelector:
    matchLabels:
      filter.fluentd.fluent.io/enabled: "true"
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"
  watchedNamespaces:
    - default
    - fluent-system
    - observability-system
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-os
  labels:
    output.fluentd.fluent.io/scope: "cluster"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
    - customPlugin:
        config: |
          <match **>
            @type opensearch
            host my-domain-endpoint.us-west-2.es.amazonaws.com
            port 443
            logstash_format  true
            logstash_prefix  logs
            scheme https
            <endpoint>
              url https://my-domain-endpoint.us-west-2.es.amazonaws.com
              region "#{ENV['AWS_REGION']}"
              assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
              assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
            </endpoint>
          </match>

But when I apply it throws the following error:

error: error validating "fluentd.yaml": error validating data: ValidationError(ClusterOutput.spec.outputs[0]): unknown field "customPlugin" in io.fluent.fluentd.v1alpha1.ClusterOutput.spec.outputs; if you choose to ignore these errors, turn validation off with --validate=false