k8ssandra
Certificate Request failure due to missing secret in cert-manager
Hi,
We have observed that there are certificate requests failing for the past 12 days due to a missing secret.
Below is the error message from the event log: Failed to create CertificateRequest: certificaterequests.cert-manager.io "k8ssandra-operator-serving-cert-1" already exists.
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
k8ssandra-operator-cass-operator-serving-cert-1 True False k8ssandra-operator-cass-operator-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager 70d
k8ssandra-operator-serving-cert-1 True False k8ssandra-operator-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager 70d
The specific error for k8ssandra-operator-serving-cert-1 is as follows: Referenced secret k8ssandra-operator/k8ssandra-operator-serving-cert-gj7jj not found: secrets "k8ssandra-operator-serving-cert-gj7jj" not found.
Interestingly, there is a k8ssandra-operator-serving-cert-jnrk4 (created 12 days ago, coinciding with the start of the error).
Do you have any suggestions on how to resolve this issue?
Thank you.
-
Helm charts version info cert-manager cert-manager 2 2024-01-13 12:50:00.362899 +0100 CET deployed cert-manager-v1.13.3 v1.13.3
k8ssandra-operator k8ssandra-operator 3 2024-03-23 18:19:49.737575 +0100 CET deployed k8ssandra-operator-1.13.0 1.13.0
prometheus-grafana k8ssandra-operator 1 2024-01-16 12:13:12.095415 +0100 CET deployed kube-prometheus-stack-55.8.2 v0.70.0 -
Helm charts user-supplied values N.A
-
Kubernetes version information: Client Version: v1.29.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.8-gke.1067004
-
Kubernetes cluster kind: K8s on GKE
Are there some additional tools or stuff installed to the cluster? I'm not able to even replicate the naming of the Secret objects:
➜ cass-operator git:(master) ✗ kubectl get certificate
NAME READY SECRET AGE
k8ssandra-cass-operator-serving-cert True k8ssandra-cass-operator-webhook-server-cert 44s
k8ssandra-k8ssandra-operator-serving-cert True k8ssandra-k8ssandra-operator-webhook-server-cert 44s
➜ cass-operator git:(master) ✗
➜ cass-operator git:(master) ✗ kubectl get certificaterequest
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
k8ssandra-cass-operator-serving-cert-1 True True k8ssandra-cass-operator-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager 4m7s
k8ssandra-k8ssandra-operator-serving-cert-1 True True k8ssandra-k8ssandra-operator-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager 4m7s
➜ cass-operator git:(master) ✗
➜ cass-operator git:(master) ✗ kubectl get secret
NAME TYPE DATA AGE
k8ssandra-cass-operator-webhook-server-cert kubernetes.io/tls 3 48s
k8ssandra-k8ssandra-operator-token kubernetes.io/service-account-token 3 48s
k8ssandra-k8ssandra-operator-webhook-server-cert kubernetes.io/tls 3 48s
sh.helm.release.v1.k8ssandra.v1 helm.sh/release.v1 1 48s
➜ cass-operator git:(master) ✗
➜ cass-operator git:(master) ✗ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
k8ssandra k8ssandra-operator 1 2024-03-25 17:27:55.707636 +0200 EET deployed k8ssandra-operator-1.13.0 1.13.0
➜ cass-operator git:(master) ✗
This cluster is purely dedicated to K8ssandra. I have installed a prometheus operator in the same namespace as recommended here https://docs.k8ssandra.io/tasks/monitor/prometheus-grafana/ . Everything was going fine since many months.
kubectl get secret -n k8ssandra-operator | grep k8ssandra in zsh at 16:38:15
k8ssandra-operator-cass-operator-serving-cert-xrp2v Opaque 1 14d
k8ssandra-operator-cass-operator-webhook-server-cert kubernetes.io/tls 3 72d
k8ssandra-operator-serving-cert-jnrk4 Opaque 1 14d
k8ssandra-operator-token kubernetes.io/service-account-token 3 57d
k8ssandra-operator-webhook-server-cert kubernetes.io/tls 3 72d
k8ssandra-reaper-ui Opaque 2 72d
prod-k8ssandra-medusa-key Opaque 1 72d
sh.helm.release.v1.k8ssandra-operator.v1 helm.sh/release.v1 1 72d
sh.helm.release.v1.k8ssandra-operator.v2 helm.sh/release.v1 1 57d
sh.helm.release.v1.k8ssandra-operator.v3 helm.sh/release.v1 1 46h
kubectl get certificaterequest -n k8ssandra-operator in zsh at 16:39:58
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
k8ssandra-operator-cass-operator-serving-cert-1 True False k8ssandra-operator-cass-operator-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager 72d
k8ssandra-operator-serving-cert-1 True False k8ssandra-operator-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager 72d
kubectl get certificate -n k8ssandra-operator in zsh at 16:40:01
NAME READY SECRET AGE
k8ssandra-operator-cass-operator-serving-cert True k8ssandra-operator-cass-operator-webhook-server-cert 72d
k8ssandra-operator-serving-cert True k8ssandra-operator-webhook-server-cert 72d
helm list -A in zsh at 16:40:51
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cert-manager cert-manager 2 2024-01-13 12:50:00.362899 +0100 CET deployed cert-manager-v1.13.3 v1.13.3
k8ssandra-operator k8ssandra-operator 3 2024-03-23 18:19:49.737575 +0100 CET deployed k8ssandra-operator-1.13.0 1.13.0
prometheus-grafana k8ssandra-operator 1 2024-01-16 12:13:12.095415 +0100 CET deployed kube-prometheus-stack-55.8.2 v0.70.0