mimir icon indicating copy to clipboard operation
mimir copied to clipboard

Add support for EKS Pod Identity

Open ripa1993 opened this issue 1 year ago • 1 comments

Is your feature request related to a problem? Please describe.

AWS has introduced EKS Pod Identity as a new way for authenticating Kubernetes Pods against AWS APIs.

Describe the solution you'd like

This requires upgrading the Go AWS SDK v2 to at least 1.23.0, currently 1.16.0 is being used by dependency thanos-io/objstore

Describe alternatives you've considered

Using IRSA (iam roles for service accounts) for Mimir, but I would prefer to stick with EKS Pod Identity for uniformity with other tools running in Kubernetes

Additional context

Add any other context or screenshots about the feature request here.

Linked issue: https://github.com/thanos-io/objstore/issues/131

ripa1993 avatar Aug 22 '24 10:08 ripa1993

Thank you for the issue. As stated, the bump first needs to happen in thanos-io/objstore. Will there be any benefits doing that change?

armandgrillet avatar Oct 25 '24 14:10 armandgrillet

Thanos appears to support EKS Pod Identities as of this PR: https://github.com/thanos-io/thanos/pull/7335

They mentioned it would be available in v0.36.0 here: https://github.com/thanos-io/thanos/discussions/7156#discussioncomment-9668145

An individual noted that it required aws_sdk_auth=true here: https://github.com/thanos-io/objstore/issues/131

skpaz avatar Dec 06 '24 18:12 skpaz

@skpaz The comment on https://github.com/thanos-io/objstore/issues/131 says that it is working in the current version (at least for that user) but aws_sdk_auth must be false.

In Mimir, it defaults to false, and can be set by the native_aws_auth_enabled config -common.storage.s3.native-aws-auth-enabled.

Loki is also just just using the thanos-io/objstore on a recent version but only with github.com/aws/aws-sdk-go-v2 v1.16.0 dependency . Do we have information on what it looks like when it does work in Loki but does not work in Mimir?

francoposa avatar Dec 06 '24 18:12 francoposa

@francoposa I have a GEL cluster up w/ Pod Identities in AWS and a complete write-up of the steps taken, Helm values.yaml, etc.

Want to touch base next week? I'm not sure what you need, but if I can provide it, I'd be happy to help.

skpaz avatar Dec 06 '24 22:12 skpaz

Any updates about this? We already do grafana loki with eks pod identity and it would be great if we could do the same for mimir

sreddahi avatar Apr 30 '25 13:04 sreddahi

We need this!

wanpdsantos avatar May 02 '25 16:05 wanpdsantos

Does anybody have a working configuration to share?

ruzickap avatar Aug 29 '25 12:08 ruzickap

I just tested this and as I thought it just works. Given that the AWS SDK used is recent enough to detect the injected credentials by the Pod Identity Agent. I believe Mimir is using the Minio libs which supports this.

So you just have to configure Pod Identity as described in the AWS docs

https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

Initially I saw the Access Denied errors and after configuring Pod Identory for the mimir Service Account I got this in the logs (needs a pod restart)

ts=2025-09-03T07:28:43.447256559Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"

vincentvdk avatar Sep 03 '25 07:09 vincentvdk

You are right... I guess this issue can be closed...

ruzickap avatar Sep 03 '25 13:09 ruzickap