Enable support for aws eks rbac and aws iam msk

[creed123]

This pull request aims to:

1. Enable kafkadrop ui running as a pod in an aws eks cluster to connect to iam authenicated aws msk. It makes use of iam roles scoped to a service-account to generate temporary credentials to connect to an iam authenticated msk. More on this can be found here: https://docs.amazonaws.cn/en_us/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html https://aws.amazon.com/blogs/big-data/securing-apache-kafka-is-easy-and-familiar-with-iam-access-control-for-amazon-msk/

Additional environment variables need to be specified:
--KAFKA_IAM_ENABLED=true --KAFKA_SASL_MECHANISM=AWS_MSK_IAM --KAFKA_SECURITY_PROTOCOL=SASL_SSL --KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;' --KAFKA_IS_SECURED=true

[fabioformosa]

What are the differences respect to this other PR to address the same target?https://github.com/obsidiandynamics/kafdrop/pull/275

Have you pushed the docker image of your forked project in dockerhub?

[fabioformosa]

Using your PR, I get

Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config 
at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110)
at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:134) 
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:740)

any idea?

[creed123]

Using your PR, I get

Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config 
at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110)
at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:134) 
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:740)

any idea?

@fabioformosa We need to specify a semi colon at the end of the --KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;' Can check after making the change?

[creed123]

What are the differences respect to this other PR to address the same target?#275

Have you pushed the docker image of your forked project in dockerhub?

This MR aims to use the role scoped to a service account for a pod in an eks cluster if such a role exists. #275 by default would use the role assigned to an ec2 instance.

[fabioformosa]

@fabioformosa We need to specify a semi colon at the end of the --KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;' Can check after making the change?

@creed123 Do you mean maybe software.amazon.msk.auth.iam.IAMLoginModule required; ?

Yes, I've tried. It solves but now I get:

ERROR 1 [| kafdrop-admin] o.a.k.c.NetworkClient: [AdminClient clientId=kafdrop-admin] Connection to node 
(b-2.xxx.eu-south-1.amazonaws.com/xxxx:9098) failed authentication due to: Access denied
WARN 1 [| kafdrop-admin] o.a.k.c.a.i.AdminMetadataManager : [AdminClient clientId=kafdrop-admin] Metadata update
 failed due to authentication error org.apache.kafka.common.errors.SaslAuthenticationException: Access denied

My eks automatically created a IAM Role, I gave fullAdminAccess to this Role to try to solve. Same error.

In your view, what can it be the cause?

[creed123]

@fabioformosa We need to specify a semi colon at the end of the --KAFKA_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule;' Can check after making the change?

@creed123 Do you mean maybe software.amazon.msk.auth.iam.IAMLoginModule required; ?

Yes, I've tried. It solves but now I get:

ERROR 1 [| kafdrop-admin] o.a.k.c.NetworkClient: [AdminClient clientId=kafdrop-admin] Connection to node 
(b-2.xxx.eu-south-1.amazonaws.com/xxxx:9098) failed authentication due to: Access denied
WARN 1 [| kafdrop-admin] o.a.k.c.a.i.AdminMetadataManager : [AdminClient clientId=kafdrop-admin] Metadata update
 failed due to authentication error org.apache.kafka.common.errors.SaslAuthenticationException: Access denied

My eks automatically created a IAM Role, I gave fullAdminAccess to this Role to try to solve. Same error.

In your view, what can it be the cause?

@fabioformosa Can you check if the role attached to your ec2 instance has the correct permissions?

[yortch]

In case it helps, I verified this works from OpenShift (Kubernetes) cluster using IRSA (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). The only additional change I had to make was to provide AWS_REGION as an environment variable for AWS STS (Security Token Service) to work. I do believe this could work just with the pom.xml changes added to this pull request. For convenience I published my image to quay.io: https://quay.io/yortch/kafdrop and this is how I built it:

git clone https://github.com/creed123/kafdrop.git
cd kafdrop
mvn clean package
mvn assembly:single docker:build
docker run -d --rm obsidiandynamics/kafdrop:3.28.0-SNAPSHOT
docker login quay.io
#provide quay.io credentials
docker ps -l 
#get container ID and replace it below
docker commit <container_id> quay.io/yortch/kafdrop:3.28.0-SNAPSHOT
docker push quay.io/yortch/kafdrop:3.28.0-SNAPSHOT
#subsequently made quay.io repository public

[tooptoop4]

gentle ping

[mfinger-incontact]

Gentle ping X2

[kirkokada]

gentle ping x3

[davideicardi]

I will be happy to merge this PR, but someone need to resolve the conflicts and review it.

[mfinger-incontact]

I will be happy to merge this PR, but someone need to resolve the conflicts and review it. <

I'm happy to help out with the conflicts, but I guess I need to become a contributor over here.

[davideicardi]

I will be happy to merge this PR, but someone need to resolve the conflicts and review it. <

I'm happy to help out with the conflicts, but I guess I need to become a contributor over here.

@mfinger-incontact I think it is better to just fork the repository (or this branch) and apply the same changes, resolve conflicts then create a new PullRequest

[aberenshtein]

@mfinger-incontact gentle ping