Kafka Output Plugin - how to use secrets for rdkafka.sasl.username, rdkafka.sasl.password

[corrantho]

Describe the issue

Hi,

I'm currently working with the Kafka Output Plugin and I would like to configure authentication credentials with a secret.

The following works:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: kafka
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "fluentbit-only"
spec:
  match: .*
  kafka:
    brokers: broker1:9094,broker2:9094,broker3:9094,broker4:9094
    topics: my.topic
    rdkafka:
      security.protocol: sasl_ssl
      acks: "0"
      sasl.mechanism: PLAIN
      sasl.username: myusername
      sasl.password: mypassword

and I would like to end with something like this

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: kafka
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "fluentbit-only"
spec:
  match: .*
  kafka:
    brokers: broker1:9094,broker2:9094,broker3:9094,broker4:9094
    topics: my.topic
    rdkafka:
      security.protocol: sasl_ssl
      acks: "0"
      sasl.mechanism: PLAIN
      sasl.username:
        valueFrom:
          secretKeyRef:
            key: username
            name: my-kafka-secret
      sasl.password:
        valueFrom:
          secretKeyRef:
            key: password
            name: my-kafka-secret
      enable.ssl.certificate.verification: "false"
    format: json_lines

However, when checking the output plugin code, we have the following

https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/output/kafka_types.go#L36

Rdkafka map[string]string `json:"rdkafka,omitempty"`

Which seems incompatible with what I'm trying to achieve...

Would you have any idea/suggestion on how to do this or do we have to adapt the output plugin to support this?

Thanks in advance.

Regards, Anthony

How did you install fluent operator?

N/A

Additional context

N/A

[wenchajun]

Can we perhaps bring this item up separately, like this https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/output/elasticsearch_types.go#L52 ?

[corrantho]

Sounds a good idea. Should we keep the rdkafka item to handle unmanaged properties and bring up only special cases such as the username, password and tls secrets?

[wenchajun]

I think it is better to deal with special cases only, what do you think? @benjaminhuo @wanjunlei @corrantho