[BUG] Failure sending request - context deadline exceeded

[dan-pulse]

Components and versions Select which component(s) the bug relates to with [X].

[ ] Controller, version: x.x.x (docker image tag) [ x] Env-Injector (webhook), version: 1.2.0 (docker image tag) [ ] Other

Describe the bug When deploying a job the env secrets do not get injected into the environment, this works fine when we use helm to dpeloy our services such as deployments/replica sets.

To Reproduce 1.19 AKS cluster deploy a pod with secrets from azure key vualt

Expected behavior Secrets injected into the pod created by the job

Logs If applicable, add logs to help explain your problem.

I0716 14:25:46.637685       1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ddcbb1fd-n642n?secret=akv2k8s-cms-database-provisioner"
I0716 14:25:46.638159       1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/q22-lab/cms-database-provisioner-ddcbb1fd-n642n"
I0716 14:25:46.674863       1 authentication.go:167] "successfully received oauth token"
E0716 14:26:16.711075       1 main.go:313] "failed to read secret from azure key vault" err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" azurekeyvaultsecret="q22-lab/content-repo-password"

Env settings in pod created from the job:

    Environment:
      MYSQL_PWD:                             mysql-password@azurekeyvault
      MYSQL_HOST:                            <name deleted>
      MYSQL_USERNAME:                        <name deleted>
      content_repo_username:                 content_repo_admin
      content_repo_password:                 content-repo-password@azurekeyvault
      cms_api_username:                      cms_api_admin
      cms_api_password:                      cms-api-password@azurekeyvault
      user_app_username:                     user-app_admin
      user_app_password:                     user-app-password@azurekeyvault

Azurekeyvaultsecret exists:

kubectl get azurekeyvaultsecret | grep content-repo
content-repo-password   q22lab-northeurope-kv   content-repo-password                           27h

Additional context The same secrets are pulled into a pod created from a deployment.

[kristeey]

The err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" error is saying that the Get Secret request times out when talking to AKV. This could be due to something blocking outgoing requests from the cluster, or miss-configuration of the AzureKeyVaultSecret object.

From experience, this error occurs if you have miss-configured the key vault name in the AzureKeyVaultSecret object, hence the request will time out since the vault does not exist.

Ensure that the following is correct

spec:
  vault:
    name: <KEY VAULT NAME>

Similar issue: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/issues/127

[dan-pulse]

I'm not sure that the other issue is the same as this, the weird behaviour we see for this same secret (content-repo-password):

• Job created via a helm deployment - the pod that is created cannot get the secret
• deployment created via helm deployment - the pod that is created can get the secret

Everything is in the same namespace so I wouldn't have thought there is anything different as both create pods it's just the one that is created via a job does not work for injecting the secret.

[kristeey]

Could you provide the manifest for the Job, Deployment and AzureKeyVaultSecret (content-repo-password)?

[dan-pulse]

akv2k8s.txt Have uploaded a text file with manifests, as a slight update as I'm just revisiting this for the first time in a while, these are the errors we now see. From the pod that needs the secret:

time="2021-07-28T12:23:46Z" level=info msg="received new Vault token" addr= app=vault-env path=kubernetes role=default
time="2021-07-28T12:23:46Z" level=info msg="initial Vault token arrived" app=vault-env
time="2021-07-28T12:23:46Z" level=info msg="renewed Vault token" app=vault-env ttl=1h0m0s
time="2021-07-28T12:23:46Z" level=info msg="spawning process: [/azure-keyvault/azure-keyvault-env /bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}]" app=vault-env
I0728 12:23:46.977665       1 version.go:31] "version info" version="" commit="92f953b" buildDate="2021-02-24T09:08:55Z" component="vaultenv"
I0728 12:23:46.977732       1 main.go:174] "azure key vault env injector initializing"
I0728 12:23:46.977894       1 main.go:224] "found original container command" cmd="/bin/sh" args=[/bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}]
I0728 12:23:46.977932       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-tnt9s?secret=akv2k8s-cms-database-provisioner"
I0728 12:23:46.992522       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-tnt9s?secret=akv2k8s-cms-database-provisioner" status="403 Forbidden" statusCode=403

From the injector pod:

E0728 12:23:37.053198       1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="cms-database-provisioner-ac4c5842-tnt9s" namespace="q22-lab"

[kristeey]

I just did a small test in a basic AKS with kubernetes version v1.19.11 following the steps:

1. Create an AKV with name akv2k8s-test-keyvault and add an access policy with the AKS service principal. Create a new secret my-test-secret in that key vault.
2. Install Akv2k8s helm chart in akv2k8s namepace: helm upgrade --install akv2k8s spv-charts/akv2k8s --namespace akv2k8s
3. Apply an test AzureKeyVaultSecret in the akv-test namespace (that has the label azure-key-vault-env-injection: enabled

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-sync-akv-test-namespace
  namespace: akv-test
spec:
  vault:
    name: akv2k8s-test-keyvault # name of key vault
    object:
      name: my-test-secret # name of the akv secret
      type: secret

4. Apply a simple Job in the akv-test namespace:

apiVersion: batch/v1
kind: Job
metadata:
  name: secret-injection-job
  namespace: akv-test
spec:
  template:
    spec:
      containers:
      - name: akv2k8s-env-test
        image: spvest/akv2k8s-env-test:2.0.1 # Test image for injection
        args: ["TEST_SECRET"]
        env:
        - name: TEST_SECRET
          value: secret-sync-akv-test-namespace@azurekeyvault
      restartPolicy: Never

And this works. Output of secret-injection-job pod:

I0728 12:27:19.944877       1 version.go:31] "version info" version="" commit="a7b2d04" buildDate="2021-03-11T07:33:36Z" component="vaultenv"
I0728 12:27:19.944930       1 main.go:176] "azure key vault env injector initializing"
I0728 12:27:19.945129       1 main.go:245] "found original container command" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh TEST_SECRET]
I0728 12:27:19.945173       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/akv-test/secret-injection-job-sfjvc?secret=akv2k8s-secret-injection"
I0728 12:27:20.024202       1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/akv-test/secret-injection-job-sfjvc?secret=akv2k8s-secret-injection"
I0728 12:27:20.024790       1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/akv-test/secret-injection-job-sfjvc"
I0728 12:27:20.423365       1 authentication.go:167] "successfully received oauth token"
I0728 12:27:20.997455       1 main.go:342] "secret injected into env var" azurekeyvaultsecret="akv-test/secret-sync-akv-test-namespace" env="TEST_SECRET"
I0728 12:27:20.997523       1 main.go:348] "starting process with secrets in env vars" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh TEST_SECRET]
value-of-my-test-secret
waiting forever...

[dan-pulse]

yep so we see the same successful output in a pod created by a deployment:

I0728 12:41:24.572982       1 version.go:31] "version info" version="" commit="92f953b" buildDate="2021-02-24T09:08:55Z" component="vaultenv"
I0728 12:41:24.573121       1 main.go:174] "azure key vault env injector initializing"
I0728 12:41:24.573309       1 main.go:224] "found original container command" cmd="/usr/local/openjdk-11/bin/java" args=[java -cp /app/resources:/app/classes:/app/libs/* com.pulselive.cms.Conte
ntRepositoryApplication]
I0728 12:41:24.573364       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/content-repo-6b9cb8ccc5-
scsmg?secret=akv2k8s-content-repo"
I0728 12:41:24.668631       1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/content-repo-6b9cb8ccc5-scsmg?secret=akv2k8s-conte
nt-repo"
I0728 12:41:24.669145       1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/q22-lab/content-repo-6b9cb8ccc5-scsmg"
I0728 12:41:24.810412       1 authentication.go:167] "successfully received oauth token"
I0728 12:41:25.109050       1 main.go:321] "secret injected into env var" azurekeyvaultsecret="q22-lab/content-repo-password" env="DATABASE_PASSWORD"
I0728 12:41:25.109118       1 main.go:327] "starting process with secrets in env vars" cmd="/usr/local/openjdk-11/bin/java" args=[java -cp /app/resources:/app/classes:/app/libs/* com.pulselive.
cms.ContentRepositoryApplication]

But with the pod created with the job we see the errors, I'll keep investigating to see if I can see anything else that may be going wrong.

[kristeey]

You can also use the global.logLevel=debug value in the helm chart for more detailed logs from the env injector.

[dan-pulse]

so ran with debug logging, we get this on the env-injector pod logs:

2021/07/28 15:12:55 [DEBUG] reviewing request 0c397fba-77c8-40a2-9dbb-61271654b09f, named: q22-lab/
I0728 15:12:55.373416       1 main.go:143] "found pod to mutate" pod="q22-lab/"
I0728 15:12:55.373435       1 pod.go:285] "creating client certificate to use with auth service" q22-lab/="(MISSING)"
I0728 15:12:55.373465       1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I0728 15:12:55.373607       1 clientCert.go:32] "parse certificate"
I0728 15:12:55.373675       1 clientCert.go:38] "generating client key"
I0728 15:12:55.398839       1 clientCert.go:44] "generating serial number"
I0728 15:12:55.398860       1 clientCert.go:66] "crating x509 certificate"
I0728 15:12:55.400491       1 pod.go:292] "mutate init-containers" q22-lab/="(MISSING)"
I0728 15:12:55.400503       1 pod.go:298] "mutate containers" q22-lab/="(MISSING)"
I0728 15:12:55.400511       1 pod.go:116] "found container to mutate" container="q22-lab/database-provisioner"
I0728 15:12:55.400517       1 pod.go:119] "checking for env vars to inject" container="q22-lab/database-provisioner"
I0728 15:12:55.400527       1 pod.go:122] "found env var to inject" env="mysql-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400538       1 pod.go:122] "found env var to inject" env="content-repo-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400550       1 pod.go:122] "found env var to inject" env="cms-api-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400562       1 pod.go:122] "found env var to inject" env="user-app-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400571       1 pod.go:122] "found env var to inject" env="content-metadata-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400580       1 registry.go:30] "getting container command for container" container="q22-lab/database-provisioner"
I0728 15:12:55.400588       1 registry.go:51] "found cmd override in kubernetes for container, no need to inspect docker image configuration" image="q22acr.azurecr.io/registry/hub/docker/com/library/mysql:5.7" container="q22-lab/database-provisioner"
I0728 15:12:55.400609       1 pod.go:149] "found container arguments to use for env-injector" cmd="/bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}" container="q22-lab/database-provisioner"
I0728 15:12:55.553729       1 pod.go:265] "signed arguments to prevent override" container="q22-lab/database-provisioner"
I0728 15:12:55.553795       1 pod.go:272] "public signing key for argument verification" key="-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv8Q7PXPiPrOsr9ox1maB\njuWPxl0/P+GLyHslE9gWtFO4BirhrGlOdIQWj27WrjxkNR6CDXb94ea6y/i1gbD7\n2EJg01YMs5odFbyG01F2JWBFhEDmvmf2g70EcRJ2ppPIk3aG+Njeo2sesqvz/v0q\nAWoNh9T+ovd9gB6uvVwDsrfaock1cAgO8j5pOdUAsBpsEszbibZxSf0sBK8lG7Ku\nBly2MLrFAlatqWJykdERsN5XUdAJGPs5XnG5o28aSzicxRi/6mG9x+4SYn3/RsFe\nEebr8nUaguJssTNWsv/yGaETMlaR4fR3Jr7zA3RDpCbpq4T5QFRfoYQTeHpsSzSv\n7wIDAQAB\n-----END RSA PUBLIC KEY-----\n" container="q22-lab/database-provisioner"
I0728 15:12:55.553811       1 pod.go:159] "full exec path" path="/azure-keyvault/azure-keyvault-env" container="q22-lab/database-provisioner"
I0728 15:12:55.553822       1 pod.go:170] "mounting volume" volume="azure-keyvault-env" path="/azure-keyvault/" container="q22-lab/database-provisioner"
I0728 15:12:55.602203       1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="q22-lab/"
2021/07/28 15:12:55 [DEBUG] json patch for request 0c397fba-77c8-40a2-9dbb-61271654b09f: [{"op":"add","path":"/spec/volumes/2","value":{"name":"azure-keyvault-env","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/3","value":{"name":"akv2k8s-client-cert","secret":{"secretName":"akv2k8s-cms-database-provisioner","defaultMode":420}}},{"op":"add","path":"/spec/initContainers","value":[{"name":"copy-azurekeyvault-env","image":"spvest/azure-keyvault-env:1.2.0","command":["sh","-c","cp /usr/local/bin/azure-keyvault-env /azure-keyvault/"],"resources":{},"volumeMounts":[{"name":"azure-keyvault-env","mountPath":"/azure-keyvault/"}],"imagePullPolicy":"IfNotPresent"}]},{"op":"remove","path":"/spec/containers/0/command/2"},{"op":"remove","path":"/spec/containers/0/command/1"},{"op":"replace","path":"/spec/containers/0/command/0","value":"/azure-keyvault/azure-keyvault-env"},{"op":"add","path":"/spec/containers/0/args/0","value":"-c"},{"op":"add","path":"/spec/containers/0/args/0","value":"-ex"},{"op":"add","path":"/spec/containers/0/args/0","value":"/bin/sh"},{"op":"add","path":"/spec/containers/0/env/11","value":{"name":"ENV_INJECTOR_ARGS_SIGNATURE","value":"S1NZK3ZseFNJQS90aFZSbmhhd1RZd3F0SzBjaWJNMUJOQStnbVRUOUIvTlFqSXRreVp6a0pPTFd3cnl1Z2orRGppQjF2Kzh6K2d5VUR2Z0E2T1pKNVJKc1d2VEFQczh4OVl4THMzMkdyQVNGM1lNU2plMUt1Rm5Gck5yY3AvUjc0d2tBTmtqa1QrTU5jOGRlb0lXcXVpaFExakhxMXdUQ3A4RGF4Y2F0UG5XdkZselBFUnVWTDNPTzhpVFo0UnJhMlp3OFRNdVlwU3RYSHdvSko4eEROaGtkWkI5bUJkeVBBNlI2OFErZjMzQTZTdVNUbmIxZFZDZHlMV1ZHc21MaGpJTEdSWTRJUnVOY1NrRnUyalNuS1d3bExVZ21RbVNYMWhuUTAxVlkxdzdHUlhMMnBHcW1qSktFR0d2TzBUYnFoTTN2U050ZW8xU2VDSE1FZisyUUpBPT0="}},{"op":"add","path":"/spec/containers/0/env/12","value":{"name":"ENV_INJECTOR_ARGS_KEY","value":"LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdjhRN1BYUGlQck9zcjlveDFtYUIKanVXUHhsMC9QK0dMeUhzbEU5Z1d0Rk80QmlyaHJHbE9kSVFXajI3V3JqeGtOUjZDRFhiOTRlYTZ5L2kxZ2JENwoyRUpnMDFZTXM1b2RGYnlHMDFGMkpXQkZoRURtdm1mMmc3MEVjUkoycHBQSWszYUcrTmplbzJzZXNxdnovdjBxCkFXb05oOVQrb3ZkOWdCNnV2VndEc3JmYW9jazFjQWdPOGo1cE9kVUFzQnBzRXN6YmliWnhTZjBzQks4bEc3S3UKQmx5Mk1MckZBbGF0cVdKeWtkRVJzTjVYVWRBSkdQczVYbkc1bzI4YVN6aWN4UmkvNm1HOXgrNFNZbjMvUnNGZQpFZWJyOG5VYWd1SnNzVE5Xc3YveUdhRVRNbGFSNGZSM0pyN3pBM1JEcENicHE0VDVRRlJmb1lRVGVIcHNTelN2Cjd3SURBUUFCCi0tLS0tRU5EIFJTQSBQVUJMSUMgS0VZLS0tLS0K"}},{"op":"add","path":"/spec/containers/0/env/13","value":{"name":"ENV_INJECTOR_USE_AUTH_SERVICE","value":"true"}},{"op":"add","path":"/spec/containers/0/env/14","value":{"name":"ENV_INJECTOR_CLIENT_CERT_DIR","value":"/var/client-cert/"}},{"op":"add","path":"/spec/containers/0/env/15","value":{"name":"ENV_INJECTOR_POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}},{"op":"add","path":"/spec/containers/0/env/16","value":{"name":"ENV_INJECTOR_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}},{"op":"add","path":"/spec/containers/0/env/17","value":{"name":"ENV_INJECTOR_AUTH_SERVICE","value":"https://akv2k8s-envinjector.akv2k8s.svc:9443"}},{"op":"add","path":"/spec/containers/0/env/18","value":{"name":"ENV_INJECTOR_AUTH_SERVICE_VALIDATION","value":"http://akv2k8s-envinjector.akv2k8s.svc:80"}},{"op":"add","path":"/spec/containers/0/env/19","value":{"name":"ENV_INJECTOR_AUTH_SERVICE_SECRET","value":"akv2k8s-cms-database-provisioner"}},{"op":"add","path":"/spec/containers/0/volumeMounts/2","value":{"name":"azure-keyvault-env","readOnly":true,"mountPath":"/azure-keyvault/"}},{"op":"add","path":"/spec/containers/0/volumeMounts/3","value":{"name":"akv2k8s-client-cert","readOnly":true,"mountPath":"/var/client-cert/"}}]
I0728 15:12:57.199618       1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I0728 15:12:57.199762       1 clientCert.go:32] "parse certificate"
I0728 15:12:57.199822       1 clientCert.go:38] "generating client key"
I0728 15:12:57.210168       1 clientCert.go:44] "generating serial number"
I0728 15:12:57.210185       1 clientCert.go:66] "crating x509 certificate"
I0728 15:12:57.244995       1 auth.go:154] "served oauth token" pod="cms-database-provisioner-ac4c5842-72mqm" namespace="q22-lab"

And then the logs for the pod that fails to get the secet:

I0728 15:13:28.224939       1 version.go:31] "version info" version="" commit="92f953b" buildDate="2021-02-24T09:08:55Z" component="vaultenv"
I0728 15:13:28.225056       1 main.go:174] "azure key vault env injector initializing"
I0728 15:13:28.225190       1 main.go:224] "found original container command" cmd="/bin/sh" args=[/bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}]
I0728 15:13:28.225227       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-72mqm?secret=akv2k8s-cms-database-provisioner"
I0728 15:13:28.292589       1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-72mqm?secret=akv2k8s-cms-database-provisioner"
I0728 15:13:28.292897       1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/q22-lab/cms-database-provisioner-ac4c5842-72mqm"
I0728 15:13:28.321809       1 authentication.go:167] "successfully received oauth token"
E0728 15:13:58.346957       1 main.go:313] "failed to read secret from azure key vault" err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" azurekeyvaultsecret="q22-lab/content-repo-password"

As a note it seems to fail to get a different azurekeyvaultsecret each time as there are multiple but I don't think this is related to the issue.

[theseion]

Any update on this? I'm struggling with an issue that looks very much like this. The problem appears only on some pods, all pods them are managed through ArgoCD.

I just realized: the only pods that fail are those with init containers. The interesting thing is that I had tried using init containers with an example deployment and that definitely works. Here's the log of the injector:

I0916 15:10:46.580501       1 registry.go:30] "getting container command for container" container="forecast-customer-str/keycloak"
I0916 15:10:46.580691       1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="<redacted>" container="forecast-customer-str/keycloak"
I0916 15:10:46.580728       1 registry.go:89] "found image in cache" image="<redacted>"
I0916 15:10:46.580747       1 pod.go:149] "found container arguments to use for env-injector" cmd="<redacted>" container="forecast-customer-str/keycloak"
I0916 15:10:46.594597       1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.594761       1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.594927       1 acr.go:73] using managed identity for acr credentials
I0916 15:10:46.595072       1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.595168       1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.607838       1 acr.go:155] "discovering auth redirects" url="<redacted>"
I0916 15:10:46.609376       1 acr.go:161] exchanging an acr refresh_token
I0916 15:10:46.618957       1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.619106       1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.619287       1 acr.go:73] using managed identity for acr credentials
I0916 15:10:46.619435       1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.619530       1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.630360       1 acr.go:155] "discovering auth redirects" url="<redacted>"
I0916 15:10:46.631934       1 acr.go:161] exchanging an acr refresh_token
I0916 15:10:46.688449       1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.688705       1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.690011       1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.690169       1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.768506       1 pod.go:265] "signed arguments to prevent override" container="forecast-customer-str/keycloak"
I0916 15:10:46.768594       1 pod.go:272] "public signing key for argument verification" key="-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6hjv8Ty0eNtl5CniWU+/\nl/4dycJEHjNbSxLdg8nnVN+IgTE8oLcrj3H4+57ZCbuEAzs8dEZsq3Td5Hof13tP\nPPDj6rjaj4H9MsFdXbVu2ZmyO1W3IqrHcKTO8cF9n/TMEN6FveTErmtzUQIAcroP\n4gtIdi0isit2Ku1Hd31mXGCP7KpPyS6951xW3gknfQ9H7in0CJiw3G+I22GS4f+F\nnjy7fWbN1yWh7bkZU3zxQm+HbH8GG81dnw8aYC8ahbdxLbG/KzpNA8TQ2Ps8Zww2\n0MI460KirfqM+FFZ0nvHRN/R2EIH3okiuA/GRj9g2A4THsGS8R9qqUxIBKMFoygf\ngQIDAQAB\n-----END RSA PUBLIC KEY-----\n" container="forecast-customer-str/keycloak"
I0916 15:10:46.768618       1 pod.go:159] "full exec path" path="/azure-keyvault/azure-keyvault-env" container="forecast-customer-str/keycloak"
I0916 15:10:46.768641       1 pod.go:170] "mounting volume" volume="azure-keyvault-env" path="/azure-keyvault/" container="forecast-customer-str/keycloak"
I0916 15:10:46.777980       1 round_trippers.go:454] POST https://10.0.0.1:443/api/v1/namespaces/forecast-customer-str/secrets 409 Conflict in 8 milliseconds
I0916 15:10:46.782746       1 round_trippers.go:454] PUT https://10.0.0.1:443/api/v1/namespaces/forecast-customer-str/secrets/akv2k8s-keycloak 200 OK in 4 milliseconds
I0916 15:10:46.783273       1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="forecast-customer-str/"
2

[theseion]

My issue is probably unrelated. I've opened a separate issue: #254

[paulissoft]

Please see my comment here: bug #127.

TL;DR There are no certificates installed so you may run an init container first (kalaksi/ca-certificates) to populate /etc/ssl/certs. Then you attach that directory to your container and you should be good.