Permission Denied when using IAM Roles for Service Accounts (IRSA) with cross account

[jzhangflir]

Hi, I am getting permission denied when pushing images cross account using assume role with IRSA. Not sure where i did wrong, please help! Here are my configurations

~/.aws/config

[profile default]
role_arn = arn:aws:iam::{accountA_id}:role/{Role_nameA}
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token
region = us-east-1

[profile int-profile]
role_arn = arn:aws:iam::{accountB_id}:role/{Role_nameB}
source_profile = default
region = us-east-1

I have tried pushing the docker image to repo in accountB, while setting the AWS_PROFILE=int-profile, and here is the error message.

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "{accountB}.dkr.ecr.us-east-1.amazonaws.com/{Repo}": POST https://{accountB}.dkr.ecr.us-east-1.amazonaws.com/v2/{Repo}/blobs/uploads/: DENIED: User: arn:aws:sts::{accountA_id}:assumed-role/{Role_nameA}/{session} is not authorized to perform: ecr:InitiateLayerUpload on resource: arn:aws:ecr:us-east-1:{AccountB_id}:{Repo}

I have tested pushing the image to Repository in {accountA_id} and it is working, which confirm that it was able to use the Service account role,

I have also tested the same profile using aws ecr command with int-profile to login, and i was able to push image to {accountB_id}

[TBBle]

If you're using the latest release of the helper (0.4.0), this may be #232 .

Try setting the env-var AWS_SDK_LOAD_CONFIG to true, to make source_profile work.

[wzooff]

Have faced with same issue. But helper version: 0.6.3 and previously mentioned variables should be true already

According to https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/#specifying-profiles

If you specify credentials in environment variables, the SDK always uses those credentials, no matter which profile you specify.

When you are using IRSA, k8s automatically sets two variables AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. So it means that you'll be always under AWS_ROLE_ARN for cred helper.

Fix for me - unset these variables

unset AWS_WEB_IDENTITY_TOKEN_FILE
unset AWS_ROLE_ARN

[TBBle]

Yeah, that makes sense, and also matches the original poster's error message.

If you're using ~/.aws/config as the original poster showed, then you can safely unset those env-vars, as the default config is doing the same thing.

I don't see any suggestion in the docs that the env-vars can be disabled by a Pod annotation, but the project README gives this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  namespace: default
  annotations:
    # optional: A comma-separated list of initContainers and container names
    #   to skip adding volumes and environment variables
    eks.amazonaws.com/skip-containers: "init-first,sidecar"

except that will skip adding the volumes too, so doesn't solve this use-case, but is handy if you need some containers in the Pod to not use the IRSA credentials, but use other credentials, e.g. burned into the image. (Although note https://github.com/aws/amazon-eks-pod-identity-webhook/issues/93 suggests there might be an issue with this)

On the other hand, looking at the source that adds the env-vars, if you define both AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE on your container, they won't be redefined by the web hook; I'm not sure if you can define them to "" or nil, and hence cause them to not be added by the webhook, but also not processed by SDK applications.

Anyway, if you need support for adding the volume with the IRSA token, but not setting the env-cars, maybe file a bug against https://github.com/aws/amazon-eks-pod-identity-webhook as that is where it would be implemented, if such a feature request doesn't already exist.

[mihir83in]

I have spent good 3 days troubleshooting this exact issue, I can confirm setting the variables AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN does resolve the issue although in a hacky way.

There seems to be a thread about this https://github.com/aws/amazon-eks-pod-identity-webhook/issues/32 here

[JohnYoungers]

If you specify credentials in environment variables, the SDK always uses those credentials, no matter which profile you specify.

When you are using IRSA, k8s automatically sets two variables AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. So it means that you'll be always under AWS_ROLE_ARN for cred helper.

Fix for me - unset these variables

unset AWS_WEB_IDENTITY_TOKEN_FILE
unset AWS_ROLE_ARN

Thank you! This resolved our issue as well (our pods include aws config files where the default profile should be attempting to assume a role in another account)