support install https proxy certificate

[shiying2012]

What would you like to be added:

The office we work in requires setting the https_proxy proxy parameter and providing a self-signed CA certificate to access the Internet. For example, for ubuntu, we would put the CA certificate in the /usr/local/share/ca-certificates/ directory and run update-ca-certificates.

We expect to implement this by modifying the configuration file and mounting the certificate on the host into the container instead of building the node image ourselves.

Why is this needed:

The office requires access to the Internet via an HTTPS proxy, and KinD's node image requires an additional certificate to be installed. Very inconvenient.

[BenTheElder]

We expect to implement this by modifying the configuration file and mounting the certificate on the host into the container instead of building the node image ourselves.

https://kind.sigs.k8s.io/docs/user/configuration/#extra-mounts

Then docker exec to run update-ca-ceerificates

[pataluc]

Then docker exec to run update-ca-ceerificates

I find it not very satisfying having to exec in the container to run udpate-ca-certificates... It seems to me that it would be possible to run the update-ca-certificates from the entrypoint script to apply whatever cert would have been mounted in the container, no?

At the moment I chose to inherit the kindest/node image like this:

FROM kindest/node:v1.25.0

ADD certs/ac.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

[BenTheElder]

I find it not very satisfying having to exec in the container to run udpate-ca-certificates...

understandable

It seems to me that it would be possible to run the update-ca-certificates from the entrypoint script to apply whatever cert would have been mounted in the container, no?

We'd have to know where it would be mounted. In which case, I'm not sure this is the right model either.

At the moment I chose to inherit the kindest/node image like this:

This is a pretty good work-around IMHO, my only addition would be to do something like FROM kindest/node:v1.25.2@sha256: 9be91e9e9cdf116809841fc77ebdb8845443c4c72fe5218f3ae9eb57fdb4bace (pin to the image from the release you are using, this one is from v0.16.0)

If you're aware of any similar tools (VM tools, minikube, microk8s etc.) that have support for this I'd love to see what established UX looks like.

If this were more standardized I'd love to look at automatically importing the cert root.

[pataluc]

We'd have to know where it would be mounted. In which case, I'm not sure this is the right model either.

As far as I understand, update-ca-certificates (on Ubuntu but probably all Debian-like distrib) reads this directory: /usr/local/share/ca-certificates (cf. https://manpages.ubuntu.com/manpages/xenial/man8/update-ca-certificates.8.html)

Furthermore  all  certificates  with  a  .crt  extension  found below /usr/local/share/ca-
       certificates are also included as implicitly trusted.

So it wouldn't be "to apply whatever cert would have been mounted in the container" but "to apply whatever certs would have been mounted on /usr/local/share/ca-certificates in the container"

If you're aware of any similar tools (VM tools, minikube, microk8s etc.) that have support for this I'd love to see what established UX looks like.

I don't have any particular example, no...

;)

[siredmar]

@BenTheElder i can give you an example for the UX perspective how talos does it.

You can add a config-patch for the talosctl command. This is an example where the content of a file gets appended to the valid ca-certificates.

machine:
  files:
    - content: |
        -----BEGIN CERTIFICATE-----
        NEEDSTOBEREPLACED
        -----END CERTIFICATE-----        
      permissions: 0644
      path: /etc/ssl/certs/ca-certificates
      op: append

[BenTheElder]

Thanks.

So it wouldn't be "to apply whatever cert would have been mounted in the container" but "to apply whatever certs would have been mounted on /usr/local/share/ca-certificates in the container"

so kind doesn't commit to never switching distros for the node image (we have before), only that the node images work with kind itself and provide what it needs to run kubernetes.

I think a less fragile version of this might be: "If users mount certs to a well-known kind-specific location, kind will ensure they're added to the node's system cert store, and we'll document this location".

then we can check in the entrypoint for any files in our well-known reserved directory and handle them appropriately.

[BenTheElder]

The downside I see is that is an environment detail leaking into config. Even in the talos example this makes the config less portable.

Maybe we should support an environment variable similar to HTTPS_PROXY but telling kind "please copy in and enable these cert files" ... of course remote dockerd may make that confusing, and some users may not have individual certs handy, only a bundle (not to mention platforms like docker desktop on Mac / windows). That's true with the config approach as well though ...

[awoimbee]

If you're aware of any similar tools (VM tools, minikube, microk8s etc.) that have support for this I'd love to see what established UX looks like.

With minikube I use the --embed-certs option (docs).

---

I need this feature because in the cluster I have an OIDC provider and OIDC clients, the clients needs to connect to the provider via HTTPS, so they needs to trust the self-signed cert I give to my ingress. I use a different cert (generated on the spot) each time.

EDIT to respond to the comment bellow without polluting the discussion:

But these clients shouldn't be running in the node host? Are they?

When I say "client", I mean things like "Sign in with Oauth" in grafana, verifying a JWT (requires static auth server URL), ...

[BenTheElder]

I need this feature because in the cluster I have an OIDC provider and OIDC clients, the clients needs to connect to the provider via HTTPS, so they needs to trust the self-signed cert I give to my ingress. I use a different cert (generated on the spot) each time.

But these clients shouldn't be running in the node host? Are they?

If you want certs on your own host, or in your pods, that shouldn't be related to this issue.

Running your own applications directly in the node container is not generally supported / portable.

Where adding certs is expected to be relevant is when HTTPS_PROXY is MITM connections node-wide and e.g. containerd on the node needs to trust the cert.

There are a few options for that currently, like https://kind.sigs.k8s.io/docs/user/private-registries/#use-a-certificate

[BenTheElder]

Thanks for the pointer -- Supporting a special certs dir under the user's config home in a kind specific directory makes sense.

I'd suggest that it should probably be controlled with an env and maybe even opt-out, used on kind create cluster, similar to how HTTPS_PROXY doesn't need to be in individual kind configurations/test pipelines instead of having a flag.

[jackivanov]

I found that mounting extra volumes directly to /etc/ssl/certs/ doesn't require update-ca-certificates

  - role: control-plane
    image: kindest/node:v1.27.11
    extraMounts:
      - hostPath: local-dev/xxx.pem
        containerPath: /etc/ssl/certs/xxx.pem

[pataluc]

Indeed I've been doing the same thing for a while... On my my part I've used the containerPath /etc/ssl/ca-certificates.crt so I think I'm overriding the whole AC, but in my environment it is not a problem as every image pull is done from an enterprise Artifactory...

[bitemarcz]

I'm stuck on this problem at the moment. Still not sure what I'm missing. I commented on another closed issue for the same problem. I''m using a .crt file but it appears to not be working if I move the crt to the ca-certificates path. Wondering if I should try moving it to the /etc/ssl/certs path instead.

I also noticed the containers running my worker nodes don't have the mount path added to my kind cluster config file. Wondering if I should just auto populate it for all my worker nodes. I think I had it working earlier somehow with a single node but quickly found out I didn't have enough nodes.