Using approle for Vault secret source in JCasC results in 403

[imalobster]

Jenkins and plugins versions report

Environment

Jenkins: 2.416 OS: Linux - 5.15.46-Unraid Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 bootstrap5-api:5.3.0-1 bouncycastle-api:2.29 branch-api:2.1122.v09cb_8ea_8a_724 build-timeout:1.31 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloudbees-folder:6.815.v0dd5a_cb_40e0e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.10.0-68.v0d0b_c439292b_ config-file-provider:952.va_544a_6234b_46 configuration-as-code:1670.v564dc8b_982d0 credentials:1271.v54b_1c2c6388a_ credentials-binding:631.v861c06d062b_4 display-url-api:2.3.8 durable-task:513.vc48a_a_075a_d93 echarts-api:5.4.0-5 email-ext:2.100 font-awesome-api:6.4.0-2 git:5.2.0 git-client:4.4.0 git-parameter:0.9.19 github:1.37.2 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1732.v3f1889a_c475b_ gradle:2.8.2 hashicorp-vault-plugin:360.v0a_1c04cf807d htmlpublisher:1.32 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jjwt-api:0.11.5-77.v646c772fddb_0 jquery3-api:3.7.0-1 junit:1217.v4297208a_a_b_ce ldap:682.v7b_544c9d1512 mailer:463.vedf8358e006b_ matrix-auth:3.1.10 matrix-project:802.v8013b_40c7edc mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ okhttp-api:4.11.0-157.v6852a_a_fa_ec11 pam-auth:1.10 pipeline-build-step:505.v5f0844d8d126 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:671.v07c339c842e8 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 pipeline-stage-view:2.33 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.3.0 resource-disposer:0.23 scm-api:676.v886669a_199a_a_ script-security:1264.vecf66020eb_7d snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.916.vd17b_43357ce4 sshd:3.312.v1c601b_c83b_0e structs:324.va_f5d6774f3a_d timestamper:1.26 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1251.vd4889a_b_0a_065 workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3731.ve4b_5b_857b_a_d3 workflow-durable-task-step:1278.v94b_dc2b_50c6f workflow-job:1316.vd2290d3341a_f workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:848.v5a_383b_d14921 ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Unraid v6.10.3 Jenkins running as container built from jenkins/jenkins:latest

Reproduction steps

1. Create (or use existing) AppRole in Vault instance that can read a KV secret for testing (in my case using KV1 engine)
2. Write a JCasC file that relies on values to be substituted by a secret source (see attached jenkins,yml for example)
3. Write a Dockerfile to build custom Jenkins container image that includes Hashicorp Vault Plugin (see attached Dockerfile for example)
4. Include the following ENV values in the Dockerfile so that the Hashicorp Vault secret source is used: ENV CASC_VAULT_PATHS <path_to_kv_secret> ENV CASC_VAULT_ENGINE_VERSION 1 ENV CASC_VAULT_APPROLE <approle_role_id>' 'ENV CASC_VAULT_APPROLE_SECRET <approle_secret_id> ENV CASC_VAULT_URL <vault_url>
5. Build the container image
6. Run the new container image
7. Observe Jenkins start up log Dockerfile.txt jenkins.yml.txt

Expected Results

JCasC uses the CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET environment variables to successfully authenticate to HC Vault and retrieve the key-value secret pairs at the path(s) specified. If successful, Jenkins start up logs do not report any details.

Actual Results

2023-08-08 10:18:14.416+0000 [id=35] WARNING c.d.j.v.j.s.VaultSecretSource#init: Could not authenticate with vault client com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403 Response body: {"errors":["permission denied"]}

    at com.bettercloud.vault.api.Auth.loginByAppRole(Auth.java:524)
    at com.datapipe.jenkins.vault.jcasc.secrets.VaultAppRoleAuthenticator.authenticate(VaultAppRoleAuthenticator.java:25)
    at com.datapipe.jenkins.vault.jcasc.secrets.VaultSecretSource.init(VaultSecretSource.java:253)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:778)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:652)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:314)
    at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:306)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1166)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

2023-08-08 10:18:14.497+0000 [id=35] WARNING i.j.p.c.SecretSourceResolver$UnresolvedLookup#lookup: Configuration import: Found unresolved variable 'PAT'. Will default to empty string

Anything else?

If I exec into the running Jenkins container and manually call Vault to generate a token using the CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET environment variables, I am able to successfully read the secrets at CASC_VAULT_PATHS. Therefore it doesn't seem to be blocked by the AppRole permissions to secrets at this path nor blocked by any bound CIDRs.

Additionally, if I generate a token using the AppRole role-id and secret-id values, and instead use this generated token in my Dockerfile via CASC_VAULT_TOKEN (instead of CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET), then it works as expected.