Adding CA certificate into Coder template failing build (k8s)

[SamStenton]

Using the devcontainer Coder template I'm seeing the following issue when mounting my CA certificates:

error: temp remount: temp remount: bind mount /ca-certs/ca-file => /.envbuilder/mnt/ca-certs/ca-file: permission denied

Adding certificate directory to ENV:

locals {
    ...
    "SSL_CERT_DIR" : "/ca-certs"
     //or
    "SSL_CERT_FILE": "/ca-certs/ca-file"
}

resource "kubernetes_deployment" "main" {
  ...

  spec {
    template {
      spec {
          ...
          volume_mount {
            name       = "ca-file"
            mount_path = "/ca-certs/ca-file"
          }
        }

        volume {
          name = "ca-file"
          secret {
            secret_name = "coder-tls"
          }
       ...
    }
}

Just trying to have the container built trust my private PKI services.

[mason3263]

I believe you need to tell Envbuilder to ignore any directories you manually mount. Use the ENVBUILDER_IGNORE_PATHS variable to do this. In your case:

ENVBUILDER_IGNORE_PATHS=/ca-certs

Also only certificates found using either SSL_CERT_DIR or SSL_CERT_FILE will be trusted so if you only provide your self-signed certificate Envbuilder won't trust any of the "standard" root certificate authorities.

[SamStenton]

Thanks! Is there a better way to do this that will add the cert to the trust store instead of overwrite? I assume building our own images...

[mason3263]

You can add the custom certificate to the hosts root certificate store and then mount the folder that your host uses into the container. At least this is the best solution that I have come up with.

For example, on Ubuntu, if you added your self-signed certificate to /usr/local/share/ca-certificates then ran update-ca-certificates your self-signed certificate should be added to /etc/ssl/certs along with all the other root authorities. You could then mount /etc/ssl/certs into your container and specify that as your root certificate directory by setting SSL_CERT_DIR=/etc/ssl/certs, then all of the other root authorities will also be trusted. Other distributions have different methods of adding self-signed certificates but they should work on the same principle.

[Kariton]

i struggle here a bit: added ENVBUILDER_IGNORE_PATHS, volume and volume_mount

locals {
  [...]
  envbuilder_env = {
    [...]
    "ENVBUILDER_IGNORE_PATHS" : "/usr/local/share/ca-certificates/"
  }
}

resource "kubernetes_deployment" "main" {
  [...]
  spec {
    [...]
    template {
      [...]
      spec {
        [...]
        container {
          [...]
          volume_mount {
            mount_path = "/usr/local/share/ca-certificates/gitlab-certificate.crt"
            name       = "gitlab-certificate"
            read_only  = true
          }
        }
        [...]
        volume {
          name = "gitlab-certificate"
          secret {
            secret_name = "gitlab-self-signed-certificate"
          }
        [...]
        }
      }
    }
  }
}

encounter:

skip mount /usr/local/share/ca-certificates/gitlab-certificate.crt under ignored prefix /usr/local/share/ca-certificates temp remount /var/run/secrets/kubernetes.io/serviceaccount error: temp remount: temp remount: bind mount /var/run/secrets/kubernetes.io/serviceaccount => /.envbuilder/mnt/var/run/secrets/kubernetes.io/serviceaccount: operation not permitted

adding /var/run/secrets/kubernetes.io/serviceaccount to ENVBUILDER_IGNORE_PATHS getting:

error: build with kaniko: do build: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/namespace: read-only file system

changing it to /var/run/secrets/kubernetes.io did not change anything.

[Kariton]

locals {
  [...]
  envbuilder_env = {
    [...]
    "ENVBUILDER_INSECURE" : "true"
  }
}

helped for now. would appreciate some help. to not run --insecure maybe we can get this also added to the docs