@username and @password are not present as config_param

[Gaurav-Puri101187]

Describe the bug

out_rdkafka2.rb is using the username and password to set values for rdkafka, sasl.username and sasl.password as we can see set rdkafka's username and password but there is no config_param to pass the same from fluent configuration. Which also makes it difficult to substitute these params with the environment variables.

A workaround for this is something like rdkafka_options "#{'{ "security.protocol": "sasl_ssl", "sasl.mechanisms": "PLAIN", "sasl.username": "$ConnectionString", "sasl.password": "%s" }' % [ENV['FLUENTD_SECRET']]}"

A proper solution would be to include username and password as config_param which allows to easily pass these values separately.

To Reproduce

 <match **>
              @type rdkafka2

              #general settings
              brokers "#{ENV['EVENT_HUB_BROKERS']}"
              default_topic "#{ENV['EVENT_HUB_TOPIC']}"

              # timeout settings check again for eventhub
              required_acks -1
              ack_timeout 30

               # sasl settings
               username "#{ENV['username']}" -- value not being used
               password "#{ENV['pass']}" -- value not bein used
               

              <format>
                   @type 'json'
              </format>
..........

Expected behavior

username and password values specified should be used, so that no need to do

rdkafka_options "#{'{ "security.protocol": "sasl_ssl", "sasl.mechanisms": "PLAIN", "sasl.username": "$ConnectionString", "sasl.password": "%s" }' % [ENV['FLUENTD_SECRET']]}"

Your Environment

- Fluentd version: 1.17.1
- fluent-plugin-kafka version: 0.19.3
- Operating system: Linux

Your Configuration

 <match **>
              @type rdkafka2

              #general settings
              brokers "#{ENV['EVENT_HUB_BROKERS']}"
              default_topic "#{ENV['EVENT_HUB_TOPIC']}"

              # timeout settings check again for eventhub
              required_acks -1
              ack_timeout 30

               # sasl settings
               username "#{ENV['username']}" -- value not being used
               password "#{ENV['pass']}" -- value not bein used
               

              <format>
                   @type 'json'
              </format>
..........

Your Error Log

Config params not getting used

Additional context

No response

[RamHaridas]

Hi Gaurav,

Just to get a clarification about one thing. Have you tried replacing ENV references with actual literals? And if yes, did that work? Were you able to send messages to the broker?