STSSessionToken to support IAM Role Sessions

[Dietatko]

Describe the bug The STSSessionToken fails to create session credentials from long-term credentials with error: unable to get authorization token: AccessDenied: Cannot call GetSessionToken with session credentials\n\tstatus code: 403 while assuming role.

To Reproduce

1. Use ESO 0.13.0 (oci.external-secrets.io/external-secrets/external-secrets:v0.13.0) on k3s v1.31.4 on-premises (no EC2 or EKS).
2. Create secret with long-term IAM credentials:

apiVersion: v1
kind: Secret
metadata:
  name: iam-cluster-credentials
type: Opaque
data:
  accessKey: ...
  secretAccessKey: ...

3. Test the long-term crdentials with aws-cli: AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... aws sts assume-role --role-arn "arn:aws:iam::1234567890:role/my-role" --role-session-name "test"

-> Produces a new session credentials 4. Create STSSessionToken generator:

apiVersion: generators.external-secrets.io/v1alpha1
kind: STSSessionToken
metadata:
  name: sts-gen
spec:
  auth:
    secretRef:
      accessKeyIDSecretRef:
        key: accessKey
        name: iam-cluster-credentials
      secretAccessKeySecretRef:
        key: secretAccessKey
        name: iam-cluster-credentials
  region: eu-west-1
  requestParameters:
    sessionDuration: 3600
  role: arn:aws:iam::1234567890:role/my-role

5. Create ExternalSecret to generate the session token:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sts
spec:
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: STSSessionToken
          name: sts-gen
  refreshInterval: 1h
  target:
    name: sts

Expected behavior A new secret named sts is created with session credentials.

Screenshots ESO logs:

{"level":"debug","ts":"2025-05-08T10:28:42.614Z","logger":"provider.aws","msg":"using credentials from secretRef"}
{"level":"info","ts":"2025-05-08T10:28:42.714Z","logger":"provider.aws","msg":"using aws session","region":"eu-west-1","credentials":{}}
{"level":"debug","ts":"2025-05-08T10:28:43.412Z","logger":"events","msg":"error processing spec.dataFrom[0].sourceRef.generatorRef, err: error using generator: unable to get authorization token: AccessDenied: Cannot call GetSessionToken with session credentials\n\tstatus code: 403, request id: a47ce8db-d230-482d-b323-fddafb97cf49","type":"Warning","object":{"kind":"ExternalSecret","namespace":"memento","name":"local-sts","uid":"6655e0c6-3a21-4d5b-893c-618970f199cc","apiVersion":"external-secrets.io/v1beta1","resourceVersion":"12282826"},"reason":"UpdateFailed"}
{"level":"error","ts":"2025-05-08T10:28:43.502Z","msg":"Reconciler error","controller":"externalsecret","controllerGroup":"external-secrets.io","controllerKind":"ExternalSecret","ExternalSecret":{"name":"local-sts","namespace":"memento"},"namespace":"memento","name":"local-sts","reconcileID":"6b21a3ed-9722-46b1-a1ff-38f41ef8b615","error":"error processing spec.dataFrom[0].sourceRef.generatorRef, err: error using generator: unable to get authorization token: AccessDenied: Cannot call GetSessionToken with session credentials\n\tstatus code: 403, request id: a47ce8db-d230-482d-b323-fddafb97cf49","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}

Additional context

• I am using the same credentials and role with ECRAuthorizationToken generator successfully.
• The trick from #4679 does not help. Not sure this is related as this is not EC2 instance.
• I just found out it works when no role is requested. A session credentials are correctly generated. Am I using the role parameter correctly?

[gusfcarvalho]

The problem, as you stated in the issue, is that you are not using the role parameter correctly. When you set it up, ESO will try to AssumeRole before doing anything agaisnt AWS (in this case, generating the token).

You are probably just missing to give the correct sts:AssumeRole permissions to your AWS IAM key.

This is not a bug :)

[Dietatko]

The problem, as you stated in the issue, is that you are not using the role parameter correctly. When you set it up, ESO will try to AssumeRole before doing anything agaisnt AWS (in this case, generating the token).

What do you mean by "ESO will try to AssumeRole before doing anything agaisnt AWS". I though assuming role is doing something against AWS. Does ESO first assumes role (and thus gets session creds) and then it uses those to get session token? I am confused what exactly is the parameter role good for?

What I am trying to achieve is that ESO is the one having access to IAM key. And it creates a short-term session credentials with role used by the application. Is there another approach to do achieve that?

You are probably just missing to give the correct sts:AssumeRole permissions to your AWS IAM key.

The same key can successfully assume role from aws-cli. And also successfully used by ECRAuthorizationToken with the same role. So I believe it has the permissions needed.

[gusfcarvalho]

did you read the docs before creating this issue? 😄

In there we state clearly:

You can attach a role to the pod using [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), [kiam](https://github.com/uswitch/kiam) or [kube2iam](https://github.com/jtblin/kube2iam). When no other authentication method is configured in the Kind=Secretstore this role is used to make all API calls against AWS Secrets Manager or SSM Parameter Store.

Based on the Pod's identity you can do a sts:assumeRole before fetching the secrets to limit access to certain keys in your provider. This is optional.

role is not a flag meant for your use case, thus, not a bug.

If you want, we can change the title / description for this to be a feature request instead.

[gusfcarvalho]

Actually, sorry about that. Even though our docs state so, our code should support assumeRole for your use case. checking something here

[Dietatko]

You can attach a role to the pod using IRSA, kiam or kube2iam. When no other authentication method is configured in the Kind=Secretstore this role is used to make all API calls against AWS Secrets Manager or SSM Parameter Store.

Based on the Pod's identity you can do a sts:assumeRole before fetching the secrets to limit access to certain keys in your provider. This is optional.

I understand this part and it has a lot of sense for AWS Secrets Manager or SSM Parameter Store (i.e. kind: SecretStore ). First get a role-limited session token and then use it to make the actual call to the target AWS service.

But I do not see the point regarding the STSSessionToken generator where the pure purpose is to get the session token. This just gets a role-limited session token and then use it to request session token? It seems to me the first step is all the generator should do here and return that token.

[gusfcarvalho]

yeah, I took a glance to the code base, your 403 error happens after ESO successfully logs in and assumes the role you've set. So this goes back to a permission problem as to why ESO cannot get a STS token with that role.

If it were enable to create the session, it would have errored a different error message: unable to create aws session

You should check your audit trail to see what's AWS returning for your 403. We identify the user agent as external-secrets if that helps.

Does not look like an external-secrets issue still....

[Dietatko]

So this goes back to a permission problem as to why ESO cannot get a STS token with that role.

Because the role-assumed session token cannot create another session token. It is a session token already. The GetSessionToken operation must be called by using the long-term AWS security credentials of an IAM user..

Once the role-assumed session token is created this is the target token already. It should not request GetSessionToken again.

[gusfcarvalho]

😄 if you want to do that, it basically means you're not giving a role token to your app. You're giving the long lived credentials token, which basically means any app can just change the role field and impersonate anything else.

This just points to incompatibility (which is why probably this is not documented in docs, even though the code path allows it to)

[gusfcarvalho]

This is, for sure, out of scope of STS token generator from the codebase, and I've already spent too much of my free time trying to help you!

If you do find anything on the source code (i couldn't) that indicates this is a bug, post it here and I'll update the tags to reflect that.

Meanwhile someone from the community might have a better expertise on AWS to tell you what's happening.

Code-wise, there is nothing wrong I could see.

Also pinging @Skarlso as he contributed this generator. Maybe he knows something 🤷

[Dietatko]

Thank you for looking into this @gusfcarvalho! Much appreciated...

Unfortunately the role parameter is unusable with the current implementation IMHO. And as there exists such a parameter in the generator, I still see this as a bug. The generator can be used to create short-term credentials but it is pity we cannot assume a role at the same time.

I see the problem in the code but I am not sure how to fix it without breaking the rest as the auth code is shared. Essentially the generator shall call AssumeRole (which is a special GetSessionToken) or GetSessionToken API. But not both:

• https://github.com/external-secrets/external-secrets/blob/d8f598aff86736eb9e4275c13f2b335f5f6b6ff3/pkg/provider/aws/auth/auth.go#L199C2-L202C3
• https://github.com/external-secrets/external-secrets/blob/d8f598aff86736eb9e4275c13f2b335f5f6b6ff3/pkg/generator/sts/sts.go#L84

The other complication is that the request parameters are ignored while assuming role.

[gusfcarvalho]

I see the problem in the code but I am not sure how to fix it without breaking the rest as the auth code is shared. Essentially the generator shall call AssumeRole (which is a special GetSessionToken) or GetSessionToken API. But not both

STS Generator == GetSessionToken. I don't think we should use it to provide a AssumeRole instead at all. As I said, its not like we do an API call and the new credential is bound to the role - this is just client sided config (at least from what I see in our codebase) - which means it is unsafe and untrustworthy to believe people will just "not" change the roles after getting the root creds.

We should probably remove the role from the STS Generator

Edit: given that i don't know aws that much and maybe the aws.sess() is somehow creating a new token in the end (though it only provides a client 🤷 ), what you can do to have the feature you want is change the sts logic to call iamAuth with no role, then you check for roles after that and do a proper switch between getSessionToken and AssumeRole, adding the needed request parameters.

[Dietatko]

As I said, its not like we do an API call and the new credential is bound to the role - this is just client sided config (at least from what I see in our codebase) - which means it is unsafe and untrustworthy to believe people will just "not" change the roles after getting the root creds.

This is not just client side configuration of the SDK. This is the line eventually calling AWS AssumeRole API and requesting new temporary credentials bound to a role by wrapping the credentials provider. https://github.com/external-secrets/external-secrets/blob/d8f598aff86736eb9e4275c13f2b335f5f6b6ff3/pkg/provider/aws/auth/auth.go#L201

Those are brand new credentials. Not just the original one with a "flag" of role. These are temporary snd you cannot use them to get back to the original long-term credentials. These new credentials are immediately set to sess.Config to use for future calls to AWS. Those will be used also to make calls to the Secret Manager in case of usage in SecretStore or ECR in case of ECRAuthorizationToken generator.

Those session credentials cannot be used to request another credentials or assume another role though. That is exactly why the subsequent GetSessionToken fails as it is not possible to request new session token using previous session token.

Removing the role parameter is an option to remove the confusion. But it would be significant functionality hit as assuming a role is one of 2 major use-cases for using the STS generator. As docs mention there are 2 options:

1. Using long term credentials to get session token. And assuming a role is the major use case as if I have long-term credentials it brings little benefits to just request session token with the same permissions. So this option is practically useless. But assuming role is the crucial security model of AWS. This is the main usage of this option.
2. Using JWT token to request session token.

I tried to propose some exact changes to the code but unfortunately I am not experienced in Go enough to make reliable changes to the code.

[gusfcarvalho]

Those are brand new credentials. Not just the original one with a "flag" of role. These are temporary snd you cannot use them to get back to the original long-term credentials. These new credentials are immediately set to sess.Config to use for future calls to AWS. Those will be used also to make calls to the Secret Manager in case of usage in SecretStore or ECR in case of ECRAuthorizationToken generator.

Those session credentials cannot be used to request another credentials or assume another role though. That is exactly why the subsequent GetSessionToken fails as it is not possible to request new session token using previous session token.

Removing the role parameter is an option to remove the confusion. But it would be significant functionality hit as assuming a role is one of 2 major use-cases for using the STS generator. As docs mention there are 2 options:

Using long term credentials to get session token. And assuming a role is the major use case as if I have long-term credentials it brings little benefits to just request session token with the same permissions. So this option is practically useless. But assuming role is the crucial security model of AWS. This is the main usage of this option. Using JWT token to request session token. I tried to propose some exact changes to the code but unfortunately I am not experienced in Go enough to make reliable changes to the code.

Thanks for the explanation. In this case, we should do what I suggested. 😄 . Feel free to push a PR over, we'll review it and propose changes until we deem it reliable 😃.

[gusfcarvalho]

updated to a feature request, as the bug fix would be 'stop supporting roles on the Generator manifes spec'.

[Skarlso]

Sorry, I missed this ping. Going to read through this, but I generally agree with Gustavo usually. :D

[Skarlso]

This is basically a call to https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html.

It's reusing the authentication method from ecr, so you're right that the role should be removed from here.

[Skarlso]

This gives me an idea... I need to add an MFA token generator Generator. :)

Edit: I did this :D

[Skarlso]

Actually, @Dietatko are you aware that GetSessionToken cannot be called with temporary credentials but only and access key + and id? ( GetSessionToken is what STSSessionToken is using to generate a session token )

[gusfcarvalho]

Yeah, the idea is to use a GetSessionToken without AssumingRoles (using AWS Auth) and then Assume Roles using a different method to allow the assumed-role STS to be provided. IMO it makes sense

[Dietatko]

Actually, @Dietatko are you aware that GetSessionToken cannot be called with temporary credentials but only and access key + and id? ( GetSessionToken is what STSSessionToken is using to generate a session token )

Exactly. That is the root cause of the problem.

The ECR generator is calling the ECR endpoint. To do so it creates a new session here: https://github.com/external-secrets/external-secrets/blob/d8f598aff86736eb9e4275c13f2b335f5f6b6ff3/pkg/generator/ecr/ecr.go#L72 And there are two options. Either:

1. I am not assuming role and the session credentials are simply the original credentials (https://github.com/external-secrets/external-secrets/blob/307c0a8468967cd862a0cfd6eb14ca3c5e86c88e/pkg/provider/aws/auth/auth.go#L200C3-L200C37). Important is that no request to AWS IAM was done yet in this case.
2. I am assuming role and the session credentials are filled by requesting the temporary credentials here: https://github.com/external-secrets/external-secrets/blob/307c0a8468967cd862a0cfd6eb14ca3c5e86c88e/pkg/provider/aws/auth/auth.go#L208C71-L209C3. These token is with role assumed already.

For the ECR generator this is working well as the target API is ECR. In the second case I have the token (with role assumed) already. In the first case the token is requested and used with ECR.

Now the STS is using the same logic but the target API is STS itself. And that is the problem as:

1. If I am not assuming role, this works well as I can call the GetSessionToken with the original credentials. This is the first call to STS.
2. If I am assuming role, this is the problem as the code requested the credentials and received the token with role assumed already (same code as before). And as the target API is STS the code then tries to call the GetSessionToken API with those temporary credentials, which is not allowed as you mentioned. The interesting part is that the credentials used for the failing GetSessionToken request are actually the credentials the generator shall return. The expected credentials are existing already. The problem is that the current code is trying to use them to request yet another token.

[Skarlso]

So, basically, don't share the auth logic with ECR? I mean, basically, just don't fill the JWT auth part?

[gusfcarvalho]

So, basically, don't share the auth logic with ECR? I mean, basically, just don't fill the JWT auth part?

yup, and for roles, also use a different approach so that we can AssumeRole independently, I guess

[Dietatko]

Another option would be to unify the "with role" and "without role" paths and simply always fetch a temporary token here: https://github.com/external-secrets/external-secrets/blob/307c0a8468967cd862a0cfd6eb14ca3c5e86c88e/pkg/provider/aws/auth/auth.go#L208C71-L209C3

Whether the token has role assumed or not, it can be used with ECR generator. So no difference there. And in case of STS generator, this would effectively become a null operation in code as the token is already fetched by session authentication and that token can be returned.

This is sometimes even recommended from security point of view to always use the temporary token rather then original credentials. But I never understood why is it more secure as you anyway have the long tern credentials in the beginning. So the exposure seems same to me.