invalid memory address or nil pointer in operator-controller-manager when providing http certs

[bmaguireibm]

Hi, thanks for the great operator. I believe I've hit a bug when trying to provide my own certificates for the external http api. Below are the details of the error, any help is greatly appreciated.

Kubernetes version: v1.26.6 opensearch-operator version: 2.4.0 platform: AKS

Expected behaviour: I was trying to provide a TSL certificate for the HTTP API. The secret is generated by vault secret operator, but ultimately this produces a Kubernetes tls secret in PEM format with tls.key, tls.crt. I also provide a separate secret for ca.crt. Both secrets are generated in the same namespace and appear to be valid PEM formatted certs with the correct keys. I expect the cluster to be created using this cert for it's http api at 9200.

Actual behaviour: The operator-controller-manager goes into crash loop back off with the following error in the logs.

{"level":"info","ts":"2023-11-23T15:35:37.377Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-cluster","namespace":"middleware"},"namespace":"middleware","name":"test-cluster","reconcileID":"124ef20c-40a0-4fa3-8695-5a667cda86ab","interface":"transport"}
{"level":"info","ts":"2023-11-23T15:35:37.388Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-cluster","namespace":"middleware"},"namespace":"middleware","name":"test-cluster","reconcileID":"124ef20c-40a0-4fa3-8695-5a667cda86ab"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1321275]

goroutine 328 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x18c5d60, 0x2aee8c0})
        /usr/local/go/src/runtime/panic.go:884 +0x212
opensearch.opster.io/pkg/tls.(*implCertValidater).IsSignedByCA(0xc000fd69b0, {0x1d9a8a0?, 0xc002450690?})
        /workspace/pkg/tls/pki.go:265 +0x35
opensearch.opster.io/pkg/reconcilers.(*TLSReconciler).shouldCreateAdminCert(0xc0001cf080, {0x1d9a8a0, 0xc002450690})
        /workspace/pkg/reconcilers/tls.go:211 +0x23d
opensearch.opster.io/pkg/reconcilers.(*TLSReconciler).createAdminSecret(0xc0001cf080, {0x1d9a8a0, 0xc002450690})
        /workspace/pkg/reconcilers/tls.go:224 +0x45
opensearch.opster.io/pkg/reconcilers.(*TLSReconciler).handleAdminCertificate(0xc0001cf080)
        /workspace/pkg/reconcilers/tls.go:122 +0x6a
opensearch.opster.io/pkg/reconcilers.(*TLSReconciler).Reconcile(0xc0001cf080)
        /workspace/pkg/reconcilers/tls.go:83 +0x89
opensearch.opster.io/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning(0xc00003e690, {0x1d99898, 0xc0008b03c0})
        /workspace/controllers/opensearchController.go:321 +0x74b
opensearch.opster.io/controllers.(*OpenSearchClusterReconciler).Reconcile(0xc00003e690, {0x1d99898, 0xc0008b03c0}, {{{0xc0007b4066, 0xa}, {0xc0001bb188, 0x17}}})
        /workspace/controllers/opensearchController.go:142 +0x768
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d99898?, {0x1d99898?, 0xc0008b03c0?}, {{{0xc0007b4066?, 0x1829e20?}, {0xc0001bb188?, 0x10?}}})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000538640, {0x1d997f0, 0xc00052e740}, {0x1942280?, 0xc000152420?})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314 +0x3a5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000538640, {0x1d997f0, 0xc00052e740})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

My cluster config is as follows:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: test-cluster
spec:
  security:
    config:
    tls:
       http:
          generate: false
          secret:
            name: opensearch-certs
          caSecret:
            name: ca-secret
       transport:
          generate: true
          perNode: true
  general:
    httpPort: 9200
    serviceName: test-cluster
    version: 2.3.0
    pluginsList: ["repository-s3"]
    drainDataNodes: true
    setVMMaxMapCount: true
  dashboards:
    tls:
      enable: true
      generate: true
    version: 2.3.0
    enable: true
    replicas: 1
    diskSize: "10Gi"
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: masters
      replicas: 3
      resources:
         requests:
            memory: 4Gi
            cpu: 1000m
         limits:
            memory: 4Gi
            cpu: 1000m
      roles:
        - "data"
        - "cluster_manager"
```