AWS authentication for the kafka source and sink

[binarylogic]

I did some light reading regarding Amazon's MSK service authentication strategy:

https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam.html

It appears we'll need to support the basic AWS authentication strategy for both the kafka source and sink. We accomplished this in the elasticsearch sink by adding an auth strategy:

[sinks.es]
  type = "elasticsearch"
  auth.strategy = "aws"

I would like to do the same here.

[wanis-fahmy]

Hello,

any plans if aws auth strategy would be available? currently not able to connect to MSK that is publicly accessible with IAM support.

Cheers

[raytung]

My understanding is that this depends on librdkafka's support. Here's a couple of relevant links:

• https://github.com/confluentinc/librdkafka/discussions/3385
• https://github.com/confluentinc/librdkafka/issues/3402

[xcrf]

AWS has recently released support for the SASL_OAUTHBEARER mechanism, which means it may no longer be necessary for any changes in librdkafka to enable support for IAM authentication to MSK. Announcement and docs.

[vainkop]

Any updates? I was trying to use vector to send logs to AWS MSK & use ingestion pipeline to get them to AWS OpenSearch implementing this but can't do it since to use ingestion pipeline AWS MSK must have IAM enabled :(

[kalavt]

bumping up, any plan to support IAM auth? SASL/OAUTHBEARER is already supported in librdkafka

[pront]

bumping up, any plan to support IAM auth? SASL/OAUTHBEARER is already supported in librdkafka

Hello, unfortunately this is not prioritized at the moment. But, we are always happy to review community contributions.