redpanda-data
AWS IAM authentication doesn't work properly with IRSA on Kubernetes
Based on this comment https://github.com/redpanda-data/console/issues/275#issuecomment-2100002274
We're using Redpanda Console as a k8s deployment on EKS deployed using the official Helm chart, with IRSA authentication. Everything works well after initial deployment, but after exactly 1 hour, we start getting SASL_AUTHENTICATION_FAILED errors.
By default, the IRSA token has duration of 1h and I guess the problem is that there is no token refresh implemented. Looking at the code here, it looks like that is indeed the case - token is fetched at startup and reused for all subsequent requests. That works until the token expires, after which all requests fail as unauthenticated.
Here is the config we use for Redpanda Console and for the service account:
console:
config:
kafka:
brokers:
- b-1.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
- b-2.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
- b-3.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
sasl:
enabled: true
mechanism: AWS_MSK_IAM
tls:
enabled: true
Here is the config we use for service account:
serviceAccount:
create: true
automountServiceAccountToken: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/something
And here is the error log after one hour, once the token expires:
{"level":"error","ts":"2024-05-02T08:43:12.242Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"2","err":"[55396e20-7c33-4065-b9bd-fb7413070759]: Access denied: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
Kafka Console release
- Helm chart:
console-0.7.26 - Redpanda console version:
v2.4.6
Mention @mislavmandaricaxilis
