druid icon indicating copy to clipboard operation
druid copied to clipboard

Add Kafka SASL / SCRAM configuration to Apache Kafka Ingestion

Open techdocsmith opened this issue 4 years ago • 3 comments

If Kafka is configured with SASL / SCRAM, then Druid, which is the Kafka consumer, should pass the SASL / SCRAM credentials in the consumerProperties block of ioConfig section of the Kafka supervisor spec:

"ioConfig": {
      "topic": "some_topic",
      "consumerProperties": {
         "bootstrap.servers": "host1:9092,host2:9092,host3:9092",
         "sasl.mechanism": "SCRAM-SHA-256",
         "security.protocol": "SASL_SSL",
         "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"some_userID\" password=\"some_password\";"
      }
      "taskCount": 1,
      "replicas": 1,
      "taskDuration": "PT1H"
}

cc: @druid-matt , @petermarshallio

Might be a good issue for a community contributor.

techdocsmith avatar Nov 10 '20 21:11 techdocsmith

It's not supported by now?

layoaster avatar Oct 11 '22 15:10 layoaster

The issue here I think is to make some improvement on the doc. You can pass necessary properties required by the SASL in the consumerProperties as above

FrankChen021 avatar Oct 12 '22 02:10 FrankChen021

indeed it seems it's related to the lack of documentation.

layoaster avatar Oct 13 '22 10:10 layoaster