Default certificate does not exist

[Roxyrob]

ref: Default certificate does not exist #601

On haproxy ingress controller v1.10.10 I get this error in logs:

ERROR global.go:272 default certificate: annotation 'ssl-certificate': secret 'namespace/namespace-default-cert' does not exist namespace is custom namespace (not default).

I cannot use this ingress controller in https. If I manually create requested certificate that controller will remove it.

Sometimes ingress controller seems to deploy and auto create default certificates, with the same ingress configuration.

I know I can create custom one, but I need to know how haproxy ingress controller check and choose to create or not its default certificate to support tls without custom one ?

[Roxyrob]

As written in the referenced haproxy ingress controller issue, Until this issue is identified and resolved I found a WORKAROUND that seems stable enough.

Following this link: Haproxy document SSL / TLS

1. Manually create a certificate

openssl req -x509 -newkey rsa:2048 -keyout example.key -out example.crt -days 365 -nodes -subj "/C=US/ST=Ohio/L=Columbus/O=MyCompany/CN=example.com"

3. Manually create a secret

kubectl create secret tls -n {namespace} haproxytempcert --cert="example.crt" --key="example.key"

4. Temporary Change haproxy ingress ConfigMap

     kubectl edit configmap -n {namespace} {configmap_name}

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: ...
         namespace: {namespace}
       data:
	 ...
         ssl-certificate: "{namespace}/haproxytempcert"    
    

Doing this "sometimes" seems "to wakeup" ingress controller that create default cert using haproxytempcert. Default certificate now exists and contains my Self Signed certificate.

If ingress controller DOES NOT wake-up, leave the haproxytempcert secret and ConfigMap annotation (ssl-certificate) to be used by haproxy ingress controller.

If Yes now you can delete temporary secrets and ConfigMap annotation:

4. Delete "ssl-certificate" from ConfigMap

     kubectl edit configmap -n {namespace} {configmap_name}

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: ...
         namespace: {namespace}
       data:
         ...
    

5. Delete secret

kubectl delete secret -n {namespace} haproxytempcert

Default certificate survive and now contains default haproxy Self-Signed certificate. What one expect from start.

Better to know why and where this happen.

[dkorunic]

@Roxyrob Can you try with the following custom settings, where controller.defaultTLSSecret.secretNamespace refers to your custom namespace?

controller:
  defaultTLSSecret:
    enabled: true
    secretNamespace: namespace

[Roxyrob]

@dkorunic, was the first try.

Indeed deployment manifest has "--default-ssl-certificate" correctly set but no default certificate was created anyway:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ...
  labels:
    app.kubernetes.io/instance: ingress-haproxy-internal
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.10.10
    helm.sh/chart: kubernetes-ingress-1.35.5
  name: ingress-haproxy-internal
  namespace: ingress-haproxy-internal
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: ingress-haproxy-internal
      app.kubernetes.io/name: kubernetes-ingress
    ...
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: ingress-haproxy-internal
        app.kubernetes.io/name: kubernetes-ingress
    spec:
      containers:
        - args:
            - >-
              --default-ssl-certificate=ingress-haproxy-internal/ingress-haproxy-internal-default-cert
            - '--configmap=ingress-haproxy-internal/ingress-haproxy-internal'
            ...

When deployment do not create default certificate the only way is the WORKAROUND above.

[dkorunic]

I haven't been able to reproduce your issue, it works for me without any issues. Configuration override used:

controller:
  defaultTLSSecret:
    enabled: true
    secretNamespace: testspace

Helm command to install:

$ helm install testissue216 --create-namespace --namespace testspace -f myvalues.yaml ./kubernetes-ingress

Verify secrets:

$ kubectl get secrets --namespace testspace                                       NAME                                           TYPE                 DATA   AGE
sh.helm.release.v1.testissue216.v1             helm.sh/release.v1   1      30s
testissue216-kubernetes-ingress-default-cert   kubernetes.io/tls    2      29s

$ kubectl describe secret testissue216-kubernetes-ingress-default-cert --namespace testspace
Name:         testissue216-kubernetes-ingress-default-cert
Namespace:    testspace
Labels:       app.kubernetes.io/instance=testissue216
             app.kubernetes.io/managed-by=Helm
             app.kubernetes.io/name=kubernetes-ingress
             app.kubernetes.io/version=1.10.10
             helm.sh/chart=kubernetes-ingress-1.36.0
Annotations:  helm.sh/hook: pre-install
             helm.sh/hook-delete-policy: before-hook-creation

Type:  kubernetes.io/tls

Data
====
tls.crt:  1159 bytes
tls.key:  1679 bytes

Check IC logs:

$ kubectl logs testissue216-kubernetes-ingress-d86cf687d-jkwjn --namespace testspace| grep certificate
2024/01/16 16:04:54 Default ssl certificate: testspace/testissue216-kubernetes-ingress-default-cert

Everything looks correct. To me it looks like you haven't matched secretNamespace and namespace while installing your Ingress Controller instance. Sadly due to how Helm renders values from values.yaml we cannot match that automatically, it has to be specified by hand.

[dkorunic]

In fact, I was wrong in thinking that we need to match controller.defaultTLSSecret.secretNamespace, it contains '{{ include "kubernetes-ingress.namespace" . }}' which will correctly render into string with {{ tpl .Values.controller.defaultTLSSecret.secretNamespace . }}.

As shown below it works as expected, no visible issues whatsoever. Required ingress controller service arguments are enabled, default SSL certificate is being created and it gets exported to the filesystem as needed:

$ helm install testissue216 --create-namespace --namespace testspace2 -f myvalues.yaml ./kubernetes-ingress

$ kubectl get secrets --namespace testspace2
NAME                                           TYPE                 DATA   AGE
sh.helm.release.v1.testissue216.v1             helm.sh/release.v1   1      2m54s
testissue216-kubernetes-ingress-default-cert   kubernetes.io/tls    2      2m54s

$ kubectl describe secret testissue216-kubernetes-ingress-default-cert --namespace testspace2
Name:         testissue216-kubernetes-ingress-default-cert
Namespace:    testspace2
Labels:       app.kubernetes.io/instance=testissue216
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kubernetes-ingress
              app.kubernetes.io/version=1.10.10
              helm.sh/chart=kubernetes-ingress-1.36.0
Annotations:  helm.sh/hook: pre-install
              helm.sh/hook-delete-policy: before-hook-creation

Type:  kubernetes.io/tls

Data
====
tls.crt:  1164 bytes
tls.key:  1679 bytes

$ kubectl logs testissue216-kubernetes-ingress-b556d758d-pfd4g --namespace testspace2| grep ssl
2024/01/17 12:33:09 Default ssl certificate: testspace2/testissue216-kubernetes-ingress-default-cert

$ kubectl exec -it testissue216-kubernetes-ingress-b556d758d-pfd4g --namespace testspace2 -- ls -al /etc/haproxy/certs/frontend/0_testspace2_testissue216-kubernetes-ingress-default-cert.pem
-rw-r--r--    1 haproxy  haproxy       2843 Jan 17 12:33 /etc/haproxy/certs/frontend/0_testspace2_testissue216-kubernetes-ingress-default-cert.pem

[dkorunic]

Closing the ticket as no-op, nothing to fix here.

[Roxyrob]

Ok. I'll go deep on this, probably misconfiguration or issue on ArgoCD part creating secret resource .yaml from parameters.

[dkorunic]

@Roxyrob We have been doing quite a few ArgoCD-related workarounds lately (https://github.com/haproxytech/helm-charts/issues/214, https://github.com/haproxytech/helm-charts/issues/211 and possibly more), so it is quite possible.

[rcazzatoApk]

This issue is related to this ArgoCD behavior/bug interpreting Helm Hooks:

Argo hook not running on auto sync only on manual sync #9830

As explained here:

"automated sync doesn't trigger synchooks when it's a self-heal action"

As soon as ArgoCD will correct to look at "initial deployment" as "self-heal action", Job creating CRDs and Self-Signed certificate (Jobs annotated with helm hooks) will be run correctly in PreSync and GitOps with ArgoCD with syncPolicy.automated will work correctly.

This is a common issue for every deployment using Helm Hooks for environment preparation and actually the WORKAROUND is to start a Manual Sync after deplopyment (this can be annoying as haproxy in internal version to reach ArgoCD WebUI will not be immediately ready after deployment - it is needed to temporary use kubectl port-forward or argocd app sync cli).

[dkorunic]

Let us keep this issue open for now and I'll keep tracking https://github.com/argoproj/argo-cd/issues/9830.

[dkorunic]

ArgoCD doesn't seem to be moving in any helpful direction in regards to this issue.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.