Access denied to Vault Secrets at 'path/to/secret'

[johncblandii]

I have tried numerous things to get this working and it simply doesn't. I verified the user does have access to the secret and can list it.

I'm guessing I'm simply doing something wrong.

Failing pipeline:

def secrets = [
    [path: 'path/to/dev']
]

pipeline {
   agent any

   stages {
      stage('vault') {
         steps {
            // inside this block your credentials will be available as env variables
            withVault([vaultSecrets: secrets]) {
                sh 'env'
            }
         }
      }
   }
}

This results in Access denied to Vault Secrets at 'path/to/dev'

I tried wrapping withVault as well:

def secrets = [
    [path: 'path/to/dev']
]

pipeline {
   agent any

   stages {
      stage('vault') {
         steps {
            withCredentials([[$class: 'VaultTokenCredentialBinding', credentialsId: 'jenkins_token', vaultAddr: 'https://vault.url.here']]) {
                // values will be masked
                sh 'echo TOKEN=$VAULT_TOKEN'
                sh 'echo ADDR=$VAULT_ADDR'
                
                withVault([configuration: [vaultUrl: VAULT_ADDR, vaultCredentialId: 'jenkins_token', engineVersion: 2], vaultSecrets: secrets]) {
                    sh 'env'
                }
            }
         }
      }
   }
}

No matter what I do...it fails:

Masking supported pattern matches of $VAULT_ADDR or $VAULT_TOKEN or $VAULT_NAMESPACE
[Pipeline] {
[Pipeline] sh
+ echo 'TOKEN=****'
TOKEN=****
[Pipeline] sh
+ echo 'ADDR=****'
ADDR=****
[Pipeline] wrap
Access denied to Vault Secrets at 'path/to/dev'

[johncblandii]

A helpful piece would be to "validate credentials" option in the settings so we can test there to make sure the creds are good.

[jetersen]

@johncblandii currently there is no support for getting all secrets on the path. You have to mention the vaultKey, which will become an env, optionally you can also pass an envVar to rename the variable inside the pipeline.

Please see the example in the readme.

Make sure your using the correct engine version.

[johncblandii]

I followed the exact examples from the README. withVault simply doesn't work for any configs I try.

def secrets = [
    [
        path: 'path/to/dev', 
        engineVersion: 2, 
        secretValues: [
            [envVar: 'application_name', vaultKey: 'application_name']
        ]
    ]
]

def configuration = [
    vaultUrl: 'https://my.vault.app',
    vaultCredentialId: 'admin-cred',
    engineVersion: 2
]

pipeline {
   agent any

   stages {
      stage('vault') {
         steps {
            withVault([configuration: configuration, vaultSecrets: secrets]) {
                echo "$application_name"
            }
         }
      }
   }
}

What am I missing here? I've tried to wrap it in withCredentials, change values, use an admin token, app role, app role token, and many combinations of config options. The engineVersion is right and I've matched the README examples.

[johncblandii]

BTW, is there a debug mode or option here to get more into the requests being sent, their response, etc?

[jetersen]

Are you sure you are using engineVersion 2? Try setting it to 1

[johncblandii]

Positive.

Any debug options here? I'm not even sure if the requests are going out at this point.

[jetersen]

debugging is to clone the repo and use mvn hpi:run from intellij that will attach a debugger

[johncblandii]

👀 I'm not sure where to go from here. mvn won't help me debug this on our Jenkins cluster. 🤷‍♂

Any other suggestions?

[jetersen]

mvn hpi:run will spin up a local Jenkins master where you can debug the plugin against your vault instance.

in any case what credential are you using? What rules/policy have you applied to the credentials?

[johncblandii]

Yeah...trying to debug the pipeline within our system, though.

I tried an approle, admin token (mine), and a token from an approle login. The policies are from admin to the following ci role I've tweaked a few times to test this out:

path "app/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "app/dev/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/create" {
  capabilities = ["update"]
}

path "auth/token/lookup-self" {
    capabilities = ["read"]
}

[jetersen]

any IP restrictions? Certificate errors?

[johncblandii]

I can't tell if there are any issues, tbh. I can't track the request from this side. I did turn on Skip SSL verification, though. It didn't help.

Let me check Cloudflare stats. Vault and Jenkins are in the same k8s cluster so I'll try a few other things as well. It definitely shouldn't be IP restricted.

[jetersen]

I assume you tried already but in case you did not. You can add a logger: com.datapipe.jenkins.vault https://wiki.jenkins.io/display/JENKINS/Logging

[johncblandii]

It doesn't really give any logs after doing that.

I do thank you for your time. I know it is hard to do this blindly and you're doing your best for free so thank you for that...seriously.

I'll keep tinkering/debugging and see what I can find out. I did change it to the local k8s service url and received a different sort of error.

Vault responded with 400 error code.
Vault responded with errors: missing client token

It makes me wonder if there is a version issue between vault 1.3.0 and the version you're expecting. It also makes me wonder if something is up with Cloudflare potentially blocking the request somehow since I don't get the same error with the external URL.

[johncblandii]

I moved back to trying approle and I at least get a stacktrace to see it could be the approle that is the issue. Digging into those settings to ensure I have those bits right.

com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
at com.bettercloud.vault.api.Auth.loginByAppRole(Auth.java:437)
at com.datapipe.jenkins.vault.credentials.VaultAppRoleCredential.getToken(VaultAppRoleCredential.java:54)
Caused: com.datapipe.jenkins.vault.exception.VaultPluginException: could not log in into vault
at com.datapipe.jenkins.vault.credentials.VaultAppRoleCredential.getToken(VaultAppRoleCredential.java:57)
at com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredential.authorizeWithVault(AbstractVaultTokenCredential.java:20)
at com.datapipe.jenkins.vault.VaultAccessor.init(VaultAccessor.java:39)
at com.datapipe.jenkins.vault.VaultBuildWrapper.provideEnvironmentVariablesFromVault(VaultBuildWrapper.java:146)
at com.datapipe.jenkins.vault.VaultBuildWrapper.setUp(VaultBuildWrapper.java:95)
at org.jenkinsci.plugins.workflow.steps.CoreWrapperStep$Execution2.doStart(CoreWrapperStep.java:97)
at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

[johncblandii]

Ahhhhhhh...so the 403 is a misnomer. I went back to trying my token and it worked after I changed my path to: app%2Fdev/secret.

Basically, our engines are segmented by stage (dev, qa, etc). That doesn't need the / escaped when using the CLI, but when using with this plugin it seems required.

approle isn't working for me, though. I can sign in with the token I gen from the following CLI commands:

ROLE_ID=$(vault read -field=role_id auth/approle/role/jenkins/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/jenkins/secret-id)
vault write -field=token auth/approle/login role_id=$ROLE_ID secret_id=$SECRET_ID

[johncblandii]

Ok...we're golden. Much obliged for all of the support here.

[jetersen]

Great not sure why you need to encode the URL 🤔

[johncblandii]

Me either, but I noticed it in the actual Vault URL so that gave me a hint.

[hchakroun]

Take a look to this: https://github.com/jenkinsci/hashicorp-vault-plugin/issues/209

[tiadobatima]

Ahhhhhhh...so the 403 is a misnomer. I went back to trying my token and it worked after I changed my path to: app%2Fdev/secret

Another option is use prefixPath in the configuration. @johncblandii 's example above:

def secrets = [
    [
        path: 'path/to/secret', 
        engineVersion: 2, 
        secretValues: [
            [envVar: 'application_name', vaultKey: 'application_name']
        ]
    ]
]

def configuration = [
    vaultUrl: 'https://my.vault.app',
    vaultCredentialId: 'admin-cred',
    engineVersion: 2,
    prefixPath: "app/dev/secret"
]

The above would lookup the secret in app/dev/secret/path/to/secret. An yes, as @johncblandii mentioned, escaping slashes in the namespace part of the path only also works, eg:

def secrets = [
    [
        path: 'app%2Fdev/secret/'path/to/secret', 
        engineVersion: 2, 
        secretValues: [
            [envVar: 'application_name', vaultKey: 'application_name']
        ]
    ]
]

def configuration = [
    vaultUrl: 'https://my.vault.app',
    vaultCredentialId: 'admin-cred',
    engineVersion: 2,
]

IMO, this is a bug that needs fixing, since I imagine most people would just try to access the secret via the full path, instead of using prefixPath 🤷

[tiadobatima]

I think I might have found the problem... I'm not a java guy so I could be totally off here:

1. LogicalResponse() calls vault.logical().read(path) here: https://github.com/jenkinsci/hashicorp-vault-plugin/blob/master/src/main/java/com/datapipe/jenkins/vault/VaultAccessor.java#L117
2. vault.logical().read(path) calls adjustPathForReadOrWrite() here: https://github.com/BetterCloud/vault-java-driver/blob/master/src/main/java/com/bettercloud/vault/api/Logical.java#L87
3. Without prefixPath, adjustPathForReadOrWrite(), places the data/ modifier for the REST call in the wrong place here: https://github.com/BetterCloud/vault-java-driver/blob/master/src/main/java/com/bettercloud/vault/api/LogicalUtilities.java#L72
4. That's because when prefixPath is empty, VaultConfig.prefixPathDepth defaults to 1 here: https://github.com/BetterCloud/vault-java-driver/blob/900ffe9a47dced88484588b315803210e17b349a/src/main/java/com/bettercloud/vault/VaultConfig.java#L39

For example, if we have the full path myspace/secrets/path/to/secret:

1. Without the prefix, this is the final URI /myspace/data/secrets/path/to/secret
2. Without the prefix, if we replace the first slash with %2F, we trick the API, by generating this URI: /myspace%2Fsecrets/data/path/to/secret. The API seems to translate %2F back to a slash.
3. With the prefix (myspace/secrets), LogicalUtilities.addQualifierToPath() is able to calculate exactly where to place the modifier, and the correct URI is generated: /myspace/secrets/data/path/to/secret

I feel this might not be an easy fix because of how v2 API was designed, by requiring the data/ modifier right in the middle of the URL, between the namespace and the path, so without a prefix, it's hard to guess where the modifier should be placed.

This should at least be thoroughly documented.

[tiadobatima]

Now that we have a potential lead, should we reopen this issue or create a new one? 🤔 I feel reopening is better because all the context about the issue is already here... @johncblandii for vis

[amousa1968]

Hello everyone, I am performing POC automation CI/CD Jenkins read secrets from hashi vault version 1.12.3 installed on a aws ubuntu server. Both Jenkins and vault on the same server. Created approle on Jenkins global credentials and tested the role_id and secret_id from the command line following all for the steps works fine, not vault running in a dev mode using following vault server -/etc/vault/config.json. I am logged in on the Linux server with my userid aymousa as said all vault commands runs no problems. Jenkins job runs local user admin and it's able to connect to vault url but getting error Access denied to Vault Secrets. I've added vaultURL to the pipeline using sh 'export vaultURL......... got both errors Access denied to Vault Secrets and sh command seems Jenkins is redirecting http to https://127.0.0.1.

Does anyone knows what's happening? Please advice.

[davama]

This thread has been really helpful.

I too see data being placed in the wrong location when jenkins does the API call.

I have two secrets from two different paths:

_path1/jenkins - this one works because the api call results in /v1/_path1/data/jenkins _path2/bla/secrets - this fails because the api call results in /v1/_path2/data/bla/secrets

the config:

def secrets = [
    [
        path: '_path1/jenkins',
        engineVersion: 2,
        secretValues: [
            [envVar: 'key', vaultKey: 'key'],
            [envVar: 'password', vaultKey: 'password']
        ]
    ],
    [
        path: '_path2/bla/secrets',
        engineVersion: 2, secretValues: [
            [envVar: 'gitea_token', vaultKey: 'gitea_api_token']
        ]
    ]
]
def configuration = [                
    vaultUrl: env.VAULT_URL,
    vaultCredentialId: 'jenkins_vault_role',
    engineVersion: 2
]
 withVault([configuration: configuration, vaultSecrets: secrets]) {
     sh  '''
     echo showing value of key: $key
     echo showing value of passpord: $password
      echo showing value of gitea_token: $gitea_token
      '''
}

If i add prefixPath: "_path2/bla" and change my second path to just path: 'secrets', then the API call is correct /v1/_path2/bla/data/secrets but then my first path is wrong....

My 0.0.2: if the user simply says path: 'my/path/to/data/secret' and the plugin no longer guesses where to insert data, this would easier i think.