Kubernetes authentication not working

[emas80]

Hi, I have followed the documentation (even if it does not seem up to date) to use the Kubernetes authentication but I am having issues.

I have a simple job free style that is meant only to authenticate to Vault and expose one single key as env var.

I get

FATAL: could not log in into vault
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
Response body: error code: 1010
	at com.bettercloud.vault.api.Auth.loginByJwt(Auth.java:1045)
	at com.datapipe.jenkins.vault.credentials.VaultKubernetesCredential.getToken(VaultKubernetesCredential.java:60)
Caused: com.datapipe.jenkins.vault.exception.VaultPluginException: could not log in into vault
	at com.datapipe.jenkins.vault.credentials.VaultKubernetesCredential.getToken(VaultKubernetesCredential.java:63)
	at com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredential.authorizeWithVault(AbstractVaultTokenCredential.java:20)
	at com.datapipe.jenkins.vault.VaultAccessor.init(VaultAccessor.java:39)
	at com.datapipe.jenkins.vault.VaultBuildWrapper.provideEnvironmentVariablesFromVault(VaultBuildWrapper.java:154)
	at com.datapipe.jenkins.vault.VaultBuildWrapper.setUp(VaultBuildWrapper.java:98)
	at jenkins.tasks.SimpleBuildWrapper.setUp(SimpleBuildWrapper.java:146)
	at hudson.model.Build$BuildExecution.doRun(Build.java:157)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1880)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:428)
Finished: FAILURE

and I have no idea what I am doing wrong.

I have a kubernetes service account named "jenkins", and I have created a policy named "jenkins" with the role "jenkins". I use the k8s authentication for Vault, so the role has the service account "jenkins" and the namespace "jenkins" configured

  vault write auth/${CLUSTER_NAME_REGION}/role/jenkins \
    bound_service_account_names="jenkins" \
    bound_service_account_namespaces="jenkins" \
    policies="jenkins" \
    ttl="1h"

Please consider that if I test by running a pod with the "jenkins" service account inside of the "jenkins" namespace and they I try and login to Vault everything works as expected and I am able to get a token with the "jenkins" policy. This is the base setup of the entire infrastructure we have, and all of our pods are able to correctly authenticate with Vault and retrieve the secrets.

When I have created the credentials, I have put the value of CLUSTER_NAME_REGION as a "Mount Path", I think that part is correct. The role is also correct, as it's simply "jenkins".

Our Jenkins runs on Kubernetes, and both the master and any of the pods are run under the "jenkins" service account inside of the "jenkins" namespace.

Vault is version 1.2.4.

Could anyone tell me what am I doing wrong?

[emas80]

I've also tested with the approle instead of the Kubernetes authentication, and it still does not work, I have the same problem. Unless the latest version of the plugin is somehow broken, I must be doing something wrong.

I am using the version of the plugin 3.6.0 and the latest version of Jenkins 2.235.1, and all the Jenkins plugins are up to date at the time I am writing.

[jetersen]

have you applied the JWT to the file system on the Jenkins master? https://github.com/jenkinsci/hashicorp-vault-plugin/blob/eed39e30f830a5313eb08288f39bceb435578445/src/main/java/com/datapipe/jenkins/vault/credentials/VaultKubernetesCredential.java#L21

[emas80]

Hi @jetersen , sorry I don't understand what you mean with "apply JWT".

I've done some more tests, I have created a free style job which does a simple bash step:

VAULT_ADDR=XXX
KUBERNETES_AUTH_PATH=YYY
VAULT_LOGIN_ROLE=jenkins
KUBE_SA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
VAULT_KV_NAME=ZZZ
key=my-secret

VAULT_TOKEN=$(curl -sS --request POST \
        ${VAULT_ADDR}/v1/auth/${KUBERNETES_AUTH_PATH}/login \
        -H "Content-Type: application/json" \
        -d '{"role":"'"${VAULT_LOGIN_ROLE}"'","jwt":"'"${KUBE_SA_TOKEN}"'"}' | \
        jq -r 'if .errors then . else .auth.client_token end')

I run this shell on the jenkins master (which is a kuberneted pod) and on a jenkins slave pod (which a of course is a kubernetes pod) and everything works as expected, I can obtain a token which I can use to retrieve my-secret. The two pods run with the Jenkins kubernetes SA.

I then have changed that to run on a jenkins slave which is a GCP instance running inside of GCP under the jenkins GCP service account - this uses the GCP authentication

VAULT_ADDR=XXX
KUBERNETES_AUTH_PATH=YYY
VAULT_LOGIN_ROLE=jenkins
VAULT_KV_NAME=ZZZ
KEY=my-secret

export GCE_TOKEN=$(curl --header "Metadata-Flavor: Google" \
			--get --data-urlencode "audience=http://vault/jenkins" \
            --data-urlencode "format=full" \
            "http://metadata/computeMetadata/v1/instance/service-accounts/default/identity")
export VAULT_TOKEN=$(curl --request POST \
	--data '{"jwt": "'"$GCE_TOKEN"'", "role": "jenkins"}' \
    $VAULT_ADDR/v1/auth/gcp/login | jq -r .auth.client_token)

and also this script works perfectly, I can get a token which I can use to obtain my-secret.

I am pretty sure then the Vault setup (roles and authentication, policies) is correct in the two cases - kubernetes and GCP, and I can run a bash script to retrieve the vault token on the pod and on the VM.

I assume the plugin would do pretty much the same things in the two scenarios. Unfortunately the stacktrace does not help to identify what can be a mistake in configuring the Jenkins credentials or maybe a bug of the version 3.6.0 (which has been released only 2 weeks ago)