Error with Cert Manager webhook

[charliemaiors]

Hello thanks for the wonderful job with the step issuer. I've deployed a Kubernetes vanilla cluster version 1.32 and the latest version of cert-manager. Issuing a new certificate for Minio operator:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: minio-operator-ca-certificate
  namespace: minio-operator
spec:
  isCA: true
  commonName: operator
  secretName: operator-ca-tls
  duration: 8760h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: step-issuer
    kind: StepClusterIssuer
    group: certmanager.step.sm

Everything is fine on the step ca server side, but when the issuer pod tries to reconcile with the cert-manager obtains the following error:

"level":"error","ts":"2025-03-21T09:53:39Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"minio-operator-ca-certificate-1","namespace":"minio-operator"},"namespace":"minio-operator","name":"minio-operator-ca-certificate-1","reconcileID":"e3647103-0119-4137-aad7-159dc751f7eb","error":"Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}

I don't know how to debug it.

[hslatman]

Hi @charliemaiors, from a quick glance it looks like the error is coming from controller-runtime itself, and could indicate connectivity issues. Have you tried the troubleshooting steps at https://cert-manager.io/docs/troubleshooting/webhook/#error-context-deadline-exceeded?

[charliemaiors]

Hi @hslatman, thanks for your answer. I tried the suggested solutions and work either from the port forward on my local machine either from a Pod in the same namespace of the step-issuer. I don't know if there are any missing configuration from step-issuer side. These are my values:

# Default values for step-issuer.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: cr.step.sm/smallstep/step-issuer
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

kubeRBACproxy:
  image:
    repository: gcr.io/kubebuilder/kube-rbac-proxy
    pullPolicy: IfNotPresent
    tag: v0.15.0
  # security context for container
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    # seccompProfile:
    #   type: RuntimeDefault

# List of secret keys used to pull images from private registries.
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

deployment:
  # Configure arguments to pass to the step issuer
  args:
    enableLeaderElection: true
    disableApprovalCheck: false
  terminationGracePeriodSeconds: 10

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 50Mi

service:
  type: ClusterIP
  port: 8443
  targetPorts: https
  controlPlane: controller-manager
  scrape: true
  scrapePort: 8080

# Security Context for the pod
podSecurityContext: {}
  # fsGroup: 2000

# security context for container
securityContext:
  runAsUser: 1000
  runAsGroup: 1000
  # seccompProfile:
  #   type: RuntimeDefault

serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

# mandatory values to generate stepIssuer resource
# please follow the https://github.com/smallstep/step-issuer#getting-started to setup step-ca and get step-issuer values
stepIssuer:
  create: false
  caUrl: ""
  caBundle: ""
  provisioner:
    name: ""
    kid: ""
    passwordRef:
      name: ""
      key: ""

# mandatory values to generate stepClusterIssuer resource
# please follow the https://github.com/smallstep/step-issuer#getting-started to setup step-ca and get step-issuer values
stepClusterIssuer:
  create: true
  caUrl: REDACTED
  caBundle: REDACTED
  provisioner:
    name: REDACTED
    kid: REDACTED
    passwordRef:
      name: REDACTED
      namespace: "step-issuer"
      key: "password"

# For the cert-manager approver
certManager:
  serviceAccount:
    name: cert-manager
    namespace: cert-manager

# Experimental support for the TLS-over-TLS tunnel in the step-ca client.
# Do not enable this unless you know what to use.
tunnel:
  enabled: false
  crt: "/opt/tunnel/tls.crt"
  key: "/opt/tunnel/tls.key"
  host: "" # host:port
  root: "/opt/tunnel/ca.crt"
  tlsCertsRef:
    secret:
      secretName: ""

# Configure extra volumes and volume mounts as YAML expressions.
volumes: false
volumeMounts: false

# nodeSelector contains the node labels for pod assignment.
nodeSelector: {}

# tolerations contains the toleration labels for pod assignment.
tolerations: []

# affinity contains the affinity settings for pod assignment.
affinity: {}

[charliemaiors]

Fixed, was a problem on the cert-manager side with kubeadm. The webhook validator must be created using hostNetwork: true in the helm chart.

[hslatman]

Great to hear, @charliemaiors! I'll close the issue now, but feel free to reopen if necessary.