adyanth
What is webhook-server-cert ?
Hey Adyanth,
I’ve been using the chart successfully up to version 0.12.0.
While upgrading to 0.13.1, I’m encountering an issue related to no "webhook-server-cert" secret. Could you help clarify what this certificate is and how to obtain or configure it?
I did check the documentation at docs/migrations/operator/v0.13.md, but the instructions were a bit confusing for me.
Thanks, Rahul Sharma
I did check the documentation at docs/migrations/operator/v0.13.md, but the instructions were a bit confusing for me.
Could you expand on what was confusing so that the docs could be improved?
When i install the operator
❯ kubectl apply -k 'https://github.com/adyanth/cloudflare-operator.git//config/default?ref=v0.13.1'
Warning: resource namespaces/cloudflare-operator-system is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/cloudflare-operator-system configured
Warning: resource customresourcedefinitions/accesstunnels.networking.cfargotunnel.com is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/accesstunnels.networking.cfargotunnel.com configured
Warning: resource customresourcedefinitions/clustertunnels.networking.cfargotunnel.com is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
Warning: Detected changes to resource clustertunnels.networking.cfargotunnel.com which is currently being deleted.
customresourcedefinition.apiextensions.k8s.io/clustertunnels.networking.cfargotunnel.com configured
Warning: resource customresourcedefinitions/tunnelbindings.networking.cfargotunnel.com is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
Warning: Detected changes to resource tunnelbindings.networking.cfargotunnel.com which is currently being deleted.
customresourcedefinition.apiextensions.k8s.io/tunnelbindings.networking.cfargotunnel.com configured
Warning: resource customresourcedefinitions/tunnels.networking.cfargotunnel.com is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/tunnels.networking.cfargotunnel.com configured
serviceaccount/cloudflare-operator-controller-manager unchanged
role.rbac.authorization.k8s.io/cloudflare-operator-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-accesstunnel-editor-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-accesstunnel-viewer-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-clustertunnel-admin-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-clustertunnel-editor-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-clustertunnel-viewer-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-manager-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-metrics-auth-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-tunnel-admin-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-tunnel-editor-role unchanged
clusterrole.rbac.authorization.k8s.io/cloudflare-operator-tunnel-viewer-role unchanged
rolebinding.rbac.authorization.k8s.io/cloudflare-operator-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cloudflare-operator-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cloudflare-operator-metrics-auth-rolebinding unchanged
service/cloudflare-operator-controller-manager-metrics-service created
service/cloudflare-operator-webhook-service created
deployment.apps/cloudflare-operator-controller-manager created
certificate.cert-manager.io/cloudflare-operator-metrics-certs unchanged
certificate.cert-manager.io/cloudflare-operator-serving-cert unchanged
issuer.cert-manager.io/cloudflare-operator-selfsigned-issuer unchanged
It Deploy below resources.
Every 5.0s: kubectl get all JJML7K3Q2T: 18:32:58
in 0.294s (0)
NAME READY STATUS RESTARTS AGE
pod/cloudflare-operator-controller-manager-7c547db6d9-q79dg 0/1 ContainerCreating 0 21s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cloudflare-operator-controller-manager-metrics-service ClusterIP 10.109.204.61 <none> 8443/TCP 80s
service/cloudflare-operator-webhook-service ClusterIP 10.102.84.150 <none> 443/TCP 80s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cloudflare-operator-controller-manager 1/1 1 1 80s
NAME DESIRED CURRENT READY AGE
replicaset.apps/cloudflare-operator-controller-manager-7c547db6d9 1 1 0 21s
replicaset.apps/cloudflare-operator-controller-manager-86cfc77f66 1 1 1 80s
The Pod Show below error
2025/06/10 13:03:38 http: TLS handshake error from 10.244.0.155:39834: remote error: tls: bad certificate
2025/06/10 13:03:39 http: TLS handshake error from 10.244.0.155:39844: remote error: tls: bad certificate
2025/06/10 13:03:40 http: TLS handshake error from 10.244.0.155:39856: remote error: tls: bad certificate
E0610 13:03:40.053514 1 reflector.go:200] "Failed to watch" err="failed to list *v1alpha2.ClusterTunnel: conversion webhook for networking.cfargotunnel.com/v1alpha1, Kind=ClusterTunnel failed: Post \"https://cloudflare-operator-webhook-service.cloudflare-operator-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match cloudflare-operator-webhook-service.cloudflare-operator-system.svc" logger="UnhandledError" reflector="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:285" type="*v1alpha2.ClusterTunnel"
2025/06/10 13:03:40 http: TLS handshake error from 10.244.0.155:39872: remote error: tls: bad certificate
2025/06/10 13:03:41 http: TLS handshake error from 10.244.0.155:39886: remote error: tls: bad certificate
2025/06/10 13:03:42 http: TLS handshake error from 10.244.0.155:39892: remote error: tls: bad certificate
E0610 13:03:42.656785 1 reflector.go:200] "Failed to watch" err="failed to list *v1alpha2.ClusterTunnel: conversion webhook for networking.cfargotunnel.com/v1alpha1, Kind=ClusterTunnel failed: Post \"https://cloudflare-operator-webhook-service.cloudflare-operator-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match cloudflare-operator-webhook-service.cloudflare-operator-system.svc" logger="UnhandledError" reflector="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:285" type="*v1alpha2.ClusterTunnel"
2025/06/10 13:03:42 http: TLS handshake error from 10.244.0.155:39894: remote error: tls: bad certificate
2025/06/10 13:03:43 http: TLS handshake error from 10.244.0.155:39906: remote error: tls: bad certificate
2025/06/10 13:03:44 http: TLS handshake error from 10.244.0.155:39922: remote error: tls: bad certificate
2025/06/10 13:03:45 http: TLS handshake error from 10.244.0.155:39936: remote error: tls: bad certificate
Could you post the outputs of kubectl -n cloudflare-operator-system get certificate and kubectl -n cloudflare-operator-system get secret please?
❯ kubectl -n cloudflare-operator-system get certificate
NAME READY SECRET AGE
cloudflare-operator-metrics-certs True metrics-server-cert 5h8m
cloudflare-operator-serving-cert True webhook-server-cert 5h8m
❯ kubectl -n cloudflare-operator-system get secret
NAME TYPE DATA AGE
cert-manager-webhook-ca Opaque 3 4h12m
metrics-server-cert kubernetes.io/tls 3 4h11m
webhook-server-cert kubernetes.io/tls 2 6m5s
okay, i'll try to replicate this issue and get back to you. On a fresh install, there isn't really much to do other than install cert manager followed by kubectl apply.
Do i have to install cert manager separately, after 'kubectl apply -k 'https://github.com/adyanth/cloudflare-operator.git//config/default?ref=v0.13.1'?, or the apply does that for us ?
On a Fresh Install i get below error in while creating pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30s default-scheduler Successfully assigned cloudflare-operator-system/cloudflare-operator-controller-manager-86cfc77f66-hrxpg to i1-1806-talos-worker01
Warning FailedMount 15s (x6 over 31s) kubelet MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found
cert-manager >= v1.0 needs to be installed to get certificates for the webhook server
Needs to be installed outside this operator.
After Cert Manager installation secret 'secret/webhook-server-cert' getting created, perhaps pod never come healthy because bad certificate error.
E0610 14:32:08.400088 1 reflector.go:200] "Failed to watch" err="failed to list *v1alpha2.ClusterTunnel: conversion webhook for networking.cfargotunnel.com/v1alpha1, Kind=ClusterTunnel failed: Post \"https://cloudflare-operator-webhook-service.cloudflare-operator-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority" logger="UnhandledError" reflector="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:285" type="*v1alpha2.ClusterTunnel"
2025/06/10 14:32:09 http: TLS handshake error from 10.244.0.155:57284: remote error: tls: bad certificate
I believe the error was caused by a stale clustertunnels.networking.cfargotunnel.com CRD from an earlier deployment (v0.12 or similar).
After uninstalling and reinstalling the operator—including the CRDs—I performed a fresh installation with the clustertunnels resource, and the issue is now resolved.
Optional suggestion for improvements:
Cert‑manager creates three pods:
❯ k get all -n cert-manager
NAME READY STATUS RESTARTS AGE
pod/cert-manager-788d58b76f-mp2pp 1/1 Running 0 15h
pod/cert-manager-cainjector-5f6f659459-md6mk 1/1 Running 0 15h
pod/cert-manager-webhook-75d4c8db8b-4fwbw 1/1 Running 0 15h
These are only needed during certificate issuance and renewal. Once the certificate is in place, most users don’t need all these components running—but disabling or removing them requires careful coordination to avoid breaking cert‑manager’s automated functionality.