Plugin uses instance role instead of pod role

[ghost]

Jenkins and plugins versions report

I've noticed that the plugin is using the IAM role of the EC2 instance where is running instead of use the IAM role of the pod (https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/).

All env variables looks good:

jenkins@jenkins-test-0:/$ env | grep -i aws
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1
AWS_ROLE_ARN=arn:aws:iam::1234567:role/test-jenkins-role
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
jenkins@jenkins-test-0:/$

But when Jenkins wants to get credentials it's not using the right role:

Jan 13, 2022 11:23:11 AM WARNING io.jenkins.plugins.credentials.secretsmanager.AwsCredentialsProvider getCredentials

Could not list credentials in Secrets Manager: message=[User: arn:aws:sts::1234567:assumed-role/eks-NodeInstanceRole-QWERTY/i-0c8134712934 is not authorized to perform: secretsmanager:ListSecrets because no identity-based policy allows the secretsmanager:ListSecrets action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: f3177c8b-e8a8-4cd8-8a0a-37bb0c5de346; Proxy: null)]

I've tried configuring the env variables manually in the pod and creating the .aws/credentials file but the result it's always the same. Finally I confirmed the problem adding privileges to get secrets to the instance IAM role and everything worked without issues.

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins: 2.303.3 OS: Linux - 5.4.91-41.139.amzn2.x86_64

ace-editor:1.1 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 aws-credentials:1.33 aws-java-sdk:1.12.131-302.vbef9650c6521 aws-java-sdk-cloudformation:1.12.131-302.vbef9650c6521 aws-java-sdk-codebuild:1.12.131-302.vbef9650c6521 aws-java-sdk-ec2:1.12.131-302.vbef9650c6521 aws-java-sdk-ecr:1.12.131-302.vbef9650c6521 aws-java-sdk-ecs:1.12.131-302.vbef9650c6521 aws-java-sdk-elasticbeanstalk:1.12.131-302.vbef9650c6521 aws-java-sdk-iam:1.12.131-302.vbef9650c6521 aws-java-sdk-logs:1.12.131-302.vbef9650c6521 aws-java-sdk-minimal:1.12.131-302.vbef9650c6521 aws-java-sdk-ssm:1.12.131-302.vbef9650c6521 aws-secrets-manager-credentials-provider:0.5.6 aws-secrets-manager-secret-source:0.0.1 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-4 bouncycastle-api:2.25 branch-api:2.7.0 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-folder:6.17 command-launcher:1.2 configuration-as-code:1.54 credentials:2.6.1 credentials-binding:1.27.1 display-url-api:2.3.5 durable-task:493.v195aefbb0ff2 echarts-api:5.2.2-2 font-awesome-api:5.15.4-5 git:4.10.0 git-client:3.11.0 git-server:1.10 handlebars:3.0.8 jackson2-api:2.13.1-246.va8a9f3eaf46a javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0 jdk-tool:1.0 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.53 kubernetes:1.30.11 kubernetes-client-api:5.11.2-182.v0f1cf4c5904e kubernetes-credentials:0.9.0 lockable-resources:2.13 mailer:408.vd726a_1130320 matrix-project:1.20 metrics:4.0.2.8.1 momentjs:1.1.1 pipeline-build-step:2.15 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:427.va6441fa17010 pipeline-milestone-step:1.3.2 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.20 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.20 plain-credentials:1.7 plugin-util-api:2.12.0 popper-api:1.16.1-2 popper2-api:2.11.2-1 scm-api:595.vd5a_df5eb_0e39 script-security:1131.v8b_b_5eda_c328e snakeyaml-api:1.29.1 ssh-credentials:1.19 sshd:3.1.0 structs:308.v852b473a2b8c thycotic-secret-server:1.0.0 trilead-api:1.0.13 variant:1.4 workflow-aggregator:2.6 workflow-api:1108.v57edf648f5d4 workflow-basic-steps:2.24 workflow-cps:2648.va9433432b33c workflow-cps-global-lib:552.vd9cc05b8a2e1 workflow-durable-task-step:1112.vda00e6febcc1 workflow-job:1145.v7f2433caa07f workflow-multibranch:706.vd43c65dec013 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:813.vb_d7c3d2984a_0

Reproduction steps

Steps to reproduce:

1. Deploy Jenkins in kubernetes with grained iam roles
2. Install aws-secrets-manager-credentials-provider-plugin
3. Try to use it

Expected Results

Jenkins will get Jenkins credentials from AWS Secret Manager using the pod IAM role.

Actual Results

Jenkins uses the instance IAM role to get secrets.

Anything else?

No response

[chriskilding]

The plugin instantiates a standard instance of the AWS Java SDK client; the only override to its behaviour is to change the EndpointConfiguration if you've set this in Jenkins config.

By default the SDK's authentication strategy is DefaultAWSCredentialsProviderChain:

AWS credentials provider chain that looks for credentials in this order:

• Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)
• Java System Properties - aws.accessKeyId and aws.secretKey
• Web Identity Token credentials from the environment or container
• Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
• Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
• Instance profile credentials delivered through the Amazon EC2 metadata service

I would recommend running the AWS CLI in your container environment to do some Secrets Manager calls - with all the same env vars as you've given Jenkins - and see which IAM role it uses.

[ghost]

AWS CLI works fine, so based in what you said, I think the problem is related to: https://github.com/aws/aws-sdk-java/issues/2136. As far as I see the sdk version used by the plugin is 1.12.131 so it should be ok but it's necessary to add aws-java-sdk-sts to the pom file. I've never tried to build a plugin but I will try to test it.

[chriskilding]

Nice detective work :) If adding aws-java-sdk-sts to the pom fixes it for you, then I'll be sure to get that included

[parmou]

@nahuelcassinarijamf did it work as expected after adding aws-java-sdk-sts to the pom?

[ghost]

@parveshmourya sorry, super delayed answer. Nope, it didn't work but maybe it was I'm not experienced with Jenkins plugins. Finally we decided to follow a different method to gather secrets from AWS Secret Manager and I didn't have time to back to this. I didn't test the new version but if it's working I think we can close this issue.