quickwit
quickwit copied to clipboard
Add sasl authentication support for kafka sources
Description
Related to https://github.com/quickwit-oss/quickwit/issues/2067
Changes:
- Add "libsasl2-dev" to build deb-requirements in Dockerfile (https://github.com/quickwit-oss/quickwit/blob/main/Dockerfile)
- Add sasl to rdkafka features in quickwit-indexing Cargo.toml (https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-indexing/Cargo.toml)
How was this PR tested?
- Builded with docker
- Run a Pod on k8s with a Kafka-Cluster (e.g. bitnami helm chart deployment) with sasl-authentication
- Run a Pod on k8s with the builded Image
- Configured a kafka source configuration and with a linked index
Example source config:
#
# Kafka source config file.
#
source_id: kafka-source
source_type: kafka
params:
topic: kafka-crawl
client_params:
bootstrap.servers: "kafka-0.kafka-headless.kafka.svc.cluster.local:9092"
group.id: quickwit
security.protocol: SASL_PLAINTEXT
sasl.mechanism: SCRAM-SHA-256
sasl.username: *****
sasl.password: *****
Thank you!
Ok, apparently somewhere there is still missing an "apt install libsasl2-dev"
CI is breaking because our builder image is missing the dependency. Let me update it.
@semmjon, CI is now passing. So I'll let you rebase and merge this PR.