Feature: AWS STS to support AssumeRoleWithWebIdentity / OIDC when accessing S3 buckets

[frittentheke]

Is your feature request related to a problem? Please describe. Enterprises tend to use their own, existing authentication system to federate / manage access to AWS resources. Ceph RADOSGW or MinIO, support this was well:

• MinIO - https://min.io/docs/minio/linux/operations/external-iam/configure-openid-external-identity-management.html
• Ceph RADOSGW - https://docs.ceph.com/en/quincy/radosgw/STS

AWS' capabilities also extend into using those external identities within roles and access policies, see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc_user-id.html, similar things can be done with straight bucket policies.

Having support for OIDC / WebIdentities would allow for more flexibility integrating Lok in such an environments.

Describe the solution you'd like Currently only "regular" and static EC2 credentials seem to be supported, looking at https://github.com/grafana/loki/blob/0f139e2cb35daa9e56f8e0bbdaca472dbf802201/pkg/storage/chunk/client/aws/s3_storage_client.go#L65

I was wondering if your code and your config could allow for OIDC credentials to be used and the other related settings? AWS Go SDK used by Loki can do this out of the box: https://docs.aws.amazon.com/sdk-for-go/api/service/sts/#STS.AssumeRoleWithWebIdentity, also the newer v2 of the SDK does have the feature and code examples ready to be used for S3 clients: https://aws.github.io/aws-sdk-go-v2/docs/migrating/#assumerolewithwebidentity

Describe alternatives you've considered Certainly some proxy could be used to mask the OIDC / STS / AssumeRoleWithWebidentitiy could be used to make Loki work in the decribed environment. But with AWS SDK already in use, a native solution is much more lightweight and less error-prone.

Additional context While instructions on using OIDC roles via annotations from within an AWS EKS cluster was documented in https://github.com/grafana/loki/commit/691b8be6c74da1c55d17ebf8d0b004b3072b5a76, this does not really require Loki to do anything special. This feature request is about allowing OIDC / AssumeRoleWithWebidentity to be used explicitly within the S3 client of Loki.

[frittentheke]

@jeschkies @pabloajz while your terraform code works for AWS and K8s which integrates this type of authentication quite nicely via the role annotations. But if there is not K8s orchestrating the assumption of the role for a service and all of the rest of the auth lifecycle it would be nice if Loki could make use of STS directly.

But thinking about the Loki side of your documented setup again .. maybe this is already covered by the magic of the AWS SDK and the simple use of an AWS_WEB_IDENTITY_TOKEN_FILE ENV var to make it happen?

[jeschkies]

I'd have to look in more detail but it seems we need to configure and use NewWebIdentityRoleProvider if this is not done already.

[frittentheke]

Thanks @jeschkies!

[frittentheke]

@jeschkies May I kindly ask you for an update on this one?

[jeschkies]

@frittentheke I'm sorry. I've moved on to other projects. I vaguely remember that there was some work on the bucket clients as well. I can only suggest to try to propose a change in a PR.