(golang/v4) Generate local webhooks for `make run`

[damsien]

trafficstars

What do you want to happen?

Description of the needs

As a k8s operator developer, I would like to test my webhooks by using make run.

The "classic way" to quickly test the manager logic is to execute make run, create some resources, check the logs at the same time, CTRL+C, change the code, make run, etc... However, as for today, we cannot test the webhook server logic by executing make run since the the manager is launched within the shell and not as a Pod in a container. Therefore, the MutatingWebhookConfiguration & ValidatingWebhookConfiguration objects cannot request the Service attached to the manager's Pod (since there is neither Service nor Pod).

Is this really needed?

Today, the only way to test the webhook server logic is either to write tests in the my_webhook_test.go file or by deploying the operator logic as a Pod in a cluster by using make deploy. The problem of these two methods is that it takes some time to be run whereas make run is fast (because it is meant to be fast to run) and allow us to test our operator the "classic way".

Also, as we can we see in this issue #400, even if it has been closed, people still have a discussion about an implementation of this feature.

Solution proposal

My solution is based on this issue comment https://github.com/kubernetes-sigs/kubebuilder/issues/400#issuecomment-1185296118.

1. Update the Makefile

Set the LOCALHOST_BRIDGE variable. This is the docker0 bridge address that allows any container to make a request the loopback address of the host.

• On MacOS, you can reach the host from inside your docker containers by using the hostname host.docker.internal
• On Linux, you can reach the host from inside your docker containers by using the IP Address 172.17.0.1 (this might change depending on your docker network configuration, but by default this is the one).

Execute a script to generate the caBundle & the server certificate (make gen-local-certs ?). This script can be located in /hack?

2. Certificate generation script

This script will first generate a CA bundle & certificate using the docker0 bridge host as SAN

[alt_names]
DNS.1 = $LOCALHOST_BRIDGE
IP.1 = $LOCALHOST_BRIDGE

The ca.crt & tls.crt will be located int the /tmp/k8s-webhook-server/serving-certs directory since controller-runtime will look at it first. Therefore, we do not have to modify our main.go to specify a temporary custom cert dir.

https://github.com/kubernetes-sigs/controller-runtime/blob/2484715d39199d4e02628c79e9be7c6f76fae28b/pkg/webhook/server.go#L143-L145

And then, it will generate a local webhook manifest on the flight based on the config/webhook/manifests.yaml. Thus, make manifests must be executed before (the same way as we use make deploy). The generation will basically copy paste the content of the manifests.yaml and update the clientConfig part.

Convert

clientConfig:
  service:
    name: webhook-service
    namespace: system
    path: /validate-my-domain-apiversion-my-kind

into

clientConfig:
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0....
  url: https://$LOCALHOST_BRIDGE/validate-my-domain-apiversion-my-kind

This part of the script removes the clientConfig.service specification and place url instead. Also, it will inject the generated CA bundle.

3. Apply the webhook

Finally, we have to apply the local webhook manifest on the cluster.

TL;DR

Create a script that generate a certificate and inject it in the webhook object. Then, the webhook make the request toward the docker bridge. The manager that runs within the shell (so it listen on the loopback address) will receive the request an can process the webhook request.

Do no hesitate to ask me questions since I already implemented this mechanism in the operator I am working on.

Extra Labels

No response