quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Add sasl authentication support for kafka sources

Open GentleGhostCoder opened this issue 2 years ago • 5 comments

Description

Related to https://github.com/quickwit-oss/quickwit/issues/2067

Changes:

  • Add "libsasl2-dev" to build deb-requirements in Dockerfile (https://github.com/quickwit-oss/quickwit/blob/main/Dockerfile)
  • Add sasl to rdkafka features in quickwit-indexing Cargo.toml (https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-indexing/Cargo.toml)

How was this PR tested?

  • Builded with docker
  • Run a Pod on k8s with a Kafka-Cluster (e.g. bitnami helm chart deployment) with sasl-authentication
  • Run a Pod on k8s with the builded Image
  • Configured a kafka source configuration and with a linked index

Example source config:

#
      # Kafka source config file.
      #
      source_id: kafka-source
      source_type: kafka
      params:
        topic: kafka-crawl
        client_params:
          bootstrap.servers: "kafka-0.kafka-headless.kafka.svc.cluster.local:9092"
          group.id: quickwit
          security.protocol: SASL_PLAINTEXT
          sasl.mechanism: SCRAM-SHA-256
          sasl.username: *****
          sasl.password: *****

GentleGhostCoder avatar Oct 06 '22 12:10 GentleGhostCoder

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Oct 06 '22 12:10 CLAassistant

Thank you!

guilload avatar Oct 06 '22 13:10 guilload

Ok, apparently somewhere there is still missing an "apt install libsasl2-dev"

GentleGhostCoder avatar Oct 06 '22 13:10 GentleGhostCoder

CI is breaking because our builder image is missing the dependency. Let me update it.

guilload avatar Oct 06 '22 13:10 guilload

@semmjon, CI is now passing. So I'll let you rebase and merge this PR.

guilload avatar Oct 07 '22 12:10 guilload