Allow pods to use other roles

[IuryAlves]

What would you like to be added:

Allow pods to use other roles

Why is this needed:

Currently, to use other roles in pods I need to mount the token volume manually.

The code [1], does not add the env variables if they are already present, which is good, but it does not mount the volume [2]

• [1]
• [2]

It would be nice if the volume is mounted even when the env vars are present.

This Feature Request is based on the discussion from #20

[micahhausler]

Thanks for opening this issue. Before coming up with a change, I'd like a little better definition around what the expected behavior would be:

1. Would you want two containers in the same pod to use different IAM roles? There would be no real security restriction between container a and container b using different roles, as they would get the same service account identity and could have the capability to assume both roles, they just wouldn't under standard configuration.
2. Would you want certain containers to not have the volume or env var (and thus not be able to use the role)?
3. Would you want support for specifying a different role than the Service Account specifies? (I think this is the main ask)
4. Should the UX be a pod level annotation for an override?
5. Should the UX be a container env var specifying AWS_ROLE_ARN?

My personal preference (click for spoiler)

I tend to think 1 is outside the scope for this webhook. If you want to do something that fancy, you could add the volumes and env var yourself. The security nuance is easily lost, and I'd like to avoid confusion by not supporting it in the webhook.

The second feature is a legit, least privilege use case that could be its own separate implementation. I'm not sure what the UX would be, I don't love the idea of encoding excluded container names in an annotation. I'm open to suggestions.

The third feature is my preferred UX: it is easy to look up, requires very little change for the user, and is easy to find.

The last UX option is my least preferred, as it requires a user to do a lot more editing.

cc @errordeveloper @jasonrichardsmith @jaypipes

[IuryAlves]

I think I can tell a little bit more about the background for this issue.

We offer Jenkins a a service for teams and teams can define custom agents where they can run their jobs. Each agent is basically a pod in the team namespace.

We want to offer to teams the ability to use IAM roles in their pods.

now, answering your questions:

Would you want two containers in the same pod to use different IAM roles? There would be no real security restriction between container a and container b using different roles, as they would get the same service account identity and could have the capability to assume both roles, they just wouldn't under standard configuration.

I only need different roles in the POD level. We don't need to specify different roles for each container.

Would you want certain containers to not have the volume or env var (and thus not be able to use the role)?

By default, if a team does not specify a Role to use in the pod it should get the one provided by the Service Account. We are fine with having a default role configured in the Service Account. There is no need to change that.

Would you want support for specifying a different role than the Service Account specifies? (I think this is the main ask)

Yes

Should the UX be a pod level annotation for an override?

I think that the AWS_ROLE_ARN env var works best for our case.

A team might need to change the AWS_ROLE_ARN when a Jenkins job is running (when deploying to another environment, for example).

This is fairly easy to do if the role is set using an env var.

@micahhausler

[IuryAlves]

The last UX option is my least preferred, as it requires a user to do a lot more editing.

The way I see it, the only change for the user would be defining another env var. I might be missing something ?

[jasonrichardsmith]

@IuryAlves Yes the env variable is the only thing that needs to change to assume a new role. This is not a functionality of EKS or IAM but the SDK itself. The Service Account is the Identity The trust Policy is binding that identity to a role The role gives permissions You can have multiple roles trust the same Service Account identity The SDK (depending on env variable) determines the assumed role

I hope that clears it up

[micahhausler]

@IuryAlves thanks for the input! If adding an env var is preferable to an annotation, I'm fine with supporting that instead.

Since you currently have a workaround (mounting the token volume manually), I'd like to keep this issue open for 3 weeks and gather broader input and use cases.

[IuryAlves]

Thanks for the update @micahhausler . I appreciate your help.

[gseshagirirao]

Hi @micahhausler , We have a requirement to attach two roles to the same service account, Does the current annotation allows us to do it? If yes, would you please provide us an example?

[IuryAlves]

@gseshagirirao

I don't think this is currently possible, but you can set AWS_ROLE_ARN to another role. For example:

AWS_ROLE_ARN=arn:aws:iam::AWS-ACCOUNT:role/ANOTHER-ROLE-NAME

[gseshagirirao]

@IuryAlves , let me explain the requirement in detail.. The two containers in same pod need to talk to different services in different AWS accounts.. also the EKS cluster is running in another account. Is it possible to annotate SA with multiple roles like comma seperated ?

[IuryAlves]

Hi @gseshagirirao

thanks for the explanation.

Is it possible to annotate SA with multiple roles like comma separated ?

I don't think that is possible. I am also not sure how would the webhook know which role belongs to container 1 and container 2.

There is an alternative though (which I haven't tested), but it should be possible to set different AWS_ROLE_ARN env vars for each container.

[jasonrichardsmith]

I just wanted to clarify. This webhook just automates abilities already inherent in Kubernetes. You don't need to use the webhook.

You should be able to do this. Without annotating the ServiceAccount.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  namespace: default
spec:
  serviceAccountName: my-serviceaccount
  containers:
  - name: container-one
    image: container-image:version
    env:
    - name: AWS_ROLE_ARN
      value: "arn:aws:iam::111122223333:role/account1-role"
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
    volumeMounts:
    - mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/"
      name: aws-token
  - name: container-two
    image: container-image:version
    env:
    - name: AWS_ROLE_ARN
      value: "arn:aws:iam::444455556666:role/account2-role"
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
    volumeMounts:
    - mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/"
      name: aws-token
  volumes:
  - name: aws-token
    projected:
      sources:
      - serviceAccountToken:
          audience: "sts.amazonaws.com"
          expirationSeconds: 86400
          path: token

But you have to add the EKS OIDC provider to both accounts AND add the service account identity to each trust policy.

[IuryAlves]

Thanks @jasonrichardsmith.

This is exactly what I wanted to express.

[gseshagirirao]

@IuryAlves @jasonrichardsmith Thank you so much for your help. I tested the option of setting env variables for both containers in same pod talking to different aws account roles, it is working.

We have requirement to make a single container in a POD connect to two different aws account roles is it possible to implement it?

[jasonrichardsmith]

Please look a little closer at the example. I am specifying two different roles in two different accounts.

[gseshagirirao]

@jasonrichardsmith Thanks again for quick reply. Yes, the above example is for two containers in same pod.. how about one container connecting to two different roles?

[IuryAlves]

You can switch roles on the fly. Let's say your container runs the following sh script:

export AWS_ROLE_ARN=arn:aws:iam::111122223333:role/account1-role

# do something in account 111122223333

# then if you set it to another role, for example

export AWS_ROLE_ARN=arn:aws:iam::444455556666:role/account2-role

# do something in another account

[micahhausler]

@gseshagirirao

We have requirement to make a single container in a POD connect to two different aws account roles is it possible to implement it?

For any single process (or container) to use multiple roles, you'll have to manage that in the application code. This is totally possible, but the Environment Variables for the AWS SDKs only help you out if there is a single role. Since that can't be configured by environment variable, I'd consider that use case outside the scope of this project

[AntonAleksandrov13]

Is there any progress on this issue? I really would like this to happen! @micahhausler