Certwatcher only detects cert changes on leader

[hrak]

trafficstars

What broke? What's expected?

On a controller generated with kubebuilder 4.5.2, we are using secure metrics and a deployment with 2 replicas, and noticed that only the leader notices & acts on cert changes, any non-leaders will not reload the cert.

In main.go the metrics certwatcher is added to the manager as a runnable using mgr.Add. It seems like this runnable is acting like a LeaderElectionRunnable and thus only operates on the leader.

logs from the leader:

2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "CHMOD         \"/tmp/k8s-metrics-server/metrics-certs/tls.key\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "REMOVE        \"/tmp/k8s-metrics-server/metrics-certs/tls.key\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "CHMOD         \"/tmp/k8s-metrics-server/metrics-certs/tls.crt\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "REMOVE        \"\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate

logs from a non leader (instantly starts logging bad certificate after rotation, can only be solved by restarting the pod):

I0425 13:16:25.693120       1 leaderelection.go:250] attempting to acquire leader lease eco-system/etcd-cluster-operator-controller-leader-election-helper...
2025/04/30 15:50:42 http: TLS handshake error from 192.168.2.129:45812: remote error: tls: bad certificate

We were expecting non-leader pods to reload the cert too.

Reproducing this issue

No response

KubeBuilder (CLI) Version

kubebuilder 4.5.2

PROJECT version

3

Plugin versions

go.kubebuilder.io/v4

Other versions

No response

Extra Labels

No response