[BUG] http 403 on injected init container

[airkewld]

Note: Make sure to check out known issues (https://akv2k8s.io/troubleshooting/known-issues/) before submitting

Components and versions Select which component(s) the bug relates to with [X].

[ ] Controller, version: x.x.x (docker image tag) [x] Env-Injector (webhook), version: 1.2.3 (docker image tag) [ ] Other

Describe the bug Injected pods getting http 403 I0602 20:17:15.481562 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/ ││ job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate" status="403 Forbidden" statusCode=403 ││ E0602 20:17:15.481676 1 main.go:261] "failed to get credentials" err="failed to validate credentials, got http status code 403" failedT ││ imes=3

To Reproduce Steps to reproduce the behavior: Specific to our env since we are using custom code, but in short:

• deployed via helm 3 with controller enabled: false and replicas:4
• injected pod has two containers (one of which is an init with akv secrets)
• init container fails to auth with akv2k8s endpoint
• webhook pod has the following logs: E0602 20:17:06.251002 1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="job-simulate-migrate-up-mvl8l" namespace="p1"

Expected behavior All containers inside of pod should be able to pull secrets from kv.

Logs App logs

I0602 20:17:06.119144       1 version.go:31] "version info" version="" commit="a7b2d04" buildDate="2021-03-11T07:33:36Z" component="vaultenv"
I0602 20:17:06.119305       1 main.go:176] "azure key vault env injector initializing"
I0602 20:17:06.122732       1 main.go:245] "found original container command" cmd="/cli" args=[/cli]
I0602 20:17:06.122880       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate"
I0602 20:17:06.312144       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate" status="403 Forbidden" statusCode=403
I0602 20:17:06.312327       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate"
I0602 20:17:06.411822       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate" status="403 Forbidden" statusCode=403
I0602 20:17:09.412185       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate"
I0602 20:17:09.459415       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate" status="403 Forbidden" statusCode=403
I0602 20:17:15.459709       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate"
I0602 20:17:15.481562       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/p1/job-simulate-migrate-up-mvl8l?secret=akv2k8s-job-simulate-migrate" status="403 Forbidden" statusCode=403
E0602 20:17:15.481676       1 main.go:261] "failed to get credentials" err="failed to validate credentials, got http status code 403" failedTimes=3

webhook logs

I0602 20:16:59.195107       1 main.go:143] "found pod to mutate" pod="p1/"
I0602 20:16:59.195142       1 pod.go:285] "creating client certificate to use with auth service" p1/="(MISSING)"
I0602 20:16:59.318527       1 pod.go:292] "mutate init-containers" p1/="(MISSING)"
I0602 20:16:59.318562       1 pod.go:116] "found container to mutate" container="p1/dbautoscale"
I0602 20:16:59.318572       1 pod.go:119] "checking for env vars to inject" container="p1/dbautoscale"
I0602 20:16:59.318587       1 pod.go:122] "found env var to inject" env="simulate-db-conn@azurekeyvault" container="p1/dbautoscale"
E0602 20:16:59.529341       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I0602 20:17:00.401561       1 pod.go:149] "found container arguments to use for env-injector" cmd="/cli" container="p1/dbautoscale"
I0602 20:17:00.905116       1 pod.go:298] "mutate containers" p1/="(MISSING)"
I0602 20:17:00.905150       1 pod.go:116] "found container to mutate" container="p1/migrate-up"
I0602 20:17:00.905161       1 pod.go:119] "checking for env vars to inject" container="p1/migrate-up"
I0602 20:17:00.905169       1 pod.go:139] found no env vars to injectcontainerp1/migrate-up
I0602 20:17:00.905204       1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="p1/"
E0602 20:17:06.251002       1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="job-simulate-migrate-up-mvl8l" namespace="p1"
E0602 20:17:06.362161       1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="job-simulate-migrate-up-mvl8l" namespace="p1"
E0602 20:17:09.458781       1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="job-simulate-migrate-up-mvl8l" namespace="p1"
E0602 20:17:15.481078       1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="job-simulate-migrate-up-mvl8l" namespace="p1"

[airkewld]

bump

[Haavare]

Hi, what is your configuration, specifically keyVaultAuth?

[airkewld]

Hello, please see below:

args:
            - "--cloudconfig=/etc/kubernetes/azure.json"
            - "--version=1.2.3"
            - "--versionenvimage=1.2.2"
            - "--v=2"
            - "--logging-format=text"
          env:
          # - name: AZURE_TENANT_ID
          #   value: "tenant-id@azurekeyvault"
          # - name: AZURE_CLIENT_ID
          #   value: "clientd-id@azurekeyvault"
          # - name: AZURE_CLIENT_SECRET
          #   value:  "client-secret@azurekeyvault"
          - name: HTTP_PORT
            value: "8080"
          - name: HTTP_PORT_EXTERNAL
            value: "80"
          - name: TLS_PORT
            value: "8443"
          - name: TLS_PORT_EXTERNAL
            value: "443"
          - name: MTLS_PORT
            value: "9443"
          - name: MTLS_PORT_EXTERNAL
            value: "9443"
          - name: MTLS_PORT
            value: "9443"
          - name: RUNNING_INSIDE_AZURE_AKS
            value: "true"
          - name: TLS_CERT_DIR
            value: /var/serving-cert
          - name: CA_CERT_DIR
            value: /var/ca-cert
          - name: ENV_INJECTOR_EXEC_DIR
            value: /azure-keyvault/
          - name: WEBHOOK_AUTH_SERVICE
            value: akv2k8s-envinjector
          - name: AUTH_TYPE
            value: azureCloudConfig
          - name: USE_AUTH_SERVICE
            value: "true"
          - name: AZUREKEYVAULT_ENV_IMAGE
            value: INJECTED_ENV_IMAGE
          - name: DOCKER_IMAGE_INSPECTION_TIMEOUT
            value: "20"
          - name: METRICS_ENABLED
            value: "false"

[kristeey]

Do you use some kind of proxy server? could this be related https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/issues/221#issuecomment-905028654?

[howardjones]

I think I am seeing this with 1.3.1 too. Other applications using akv2k8s are working fine - just this initContainer is failing. No proxy in use.

In the controller log:

2022-07-13T16:03:57.651556345Z E0713 16:03:57.651463       1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="k8s-init-test" namespace="hj-testing"

In the initContainer log:

    I0714 07:24:36.500070       1 version.go:31] "version info" version="" commit="11d9861" buildDate="2022-04-18T15:59:28Z" component="vaultenv"


    I0714 07:24:45.644307       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/hj-testing/k8s-init-test?secret=akv2k8s-k8s-init-test"
    E0714 07:24:45.677662       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/hj-testing/k8s-init-test?secret=akv2k8s-k8s-init-test" status="403 Forbidden" statusCode=403
    E0714 07:24:45.677734       1 main.go:269] "failed to get credentials" err="failed to validate credentials, got http status code 403" failedTimes=3

[tspearconquest]

Voting to close based on lack of response from OP