feat: Allow disabling the secret existence verification (elasticsearchRefs)

[syepes]

When trying to use the k8s SecretProvider feature to load the values from the external Key Vault, the ECK Operator is per-verifying the existence / presence of the elasticsearchRefs secrets before deploying the logstash StatefulSets. This prevents the deployment of the pods and by consequence the creation of the necessary secrets as they are only created at the pod deployment.

Proposal

It would be great to have a configuration option that allows the operator to do the deployment without verifying the presence of the elasticsearchRefs secretNames objects, this would allow the pods to get deployed and the SecretProvider would attach and create the necessary secrets (elasticsearchRefs)

Note: This works when using the SecretProvider just for the standard envFrom, but not for the elasticsearchRefs as there is some kind of per-verification mechanism for this kind of object.

Environment

• ECK version: 2.13.0

• Kubernetes information: AKS / 1.29.4

• Resource definition:

Note: This is a simplified example to illustrate the usage and configuration of the SecretProvider

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: keyvault
spec:
  provider: azure
  secretObjects:
    - secretName: secrets-ext-es
      type: Opaque
      data:
        - objectName: MON-URL
          key: url
        - objectName: MON-USR
          key: username
        - objectName: MON-PWD
          key: password
    - secretName: secrets-agt
      type: Opaque
      data:
        - objectName: XPACK-MANAGEMENT-ELASTICSEARCH-API-KEY
          key: XPACK_MANAGEMENT_ELASTICSEARCH_API_KEY
  parameters:
    tenantId: XXX
    userAssignedIdentityID: XXX
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    keyvaultName: keyvault-lab
    objects: |
      array:
        - |
          objectName: XPACK-MANAGEMENT-ELASTICSEARCH-API-KEY
          objectType: secret
        - |
          objectName: MON-URL
          objectType: secret
        - |
          objectName: MON-USR
          objectType: secret
        - |
          objectName: MON-PWD
          objectType: secret
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash
spec:
  count: 1
  version: 8.14.3
  elasticsearchRefs:
    - clusterName: ext-es
      secretName: secrets-ext-es
  monitoring:
    metrics:
      elasticsearchRefs:
        - secretName: secrets-ext-es
    logs:
      elasticsearchRefs:
        - secretName: secrets-ext-es
  podTemplate:
    spec:
      terminationGracePeriodSeconds: 300
      containers:
        - name: logstash
          envFrom:
            - secretRef:
                name: secrets-agt
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/sstore"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: "secrets-store.csi.k8s.io"
            readOnly: true
            volumeAttributes:
              secretProviderClass: keyvault

• Logs:

{"log.level":"debug","@timestamp":"2024-07-17T22:29:12.751Z","log.logger":"manager.eck-operator.events","message":"Association status changed from [] to [lab-dev/secrets-ext-es: Pending]","service.version":"2.13.0+8896afe1","service.type":"eck","ecs.version":"1.4.0","type":"Normal","object":{"kind":"Logstash","namespace":"lab-dev","name":"logstash","uid":"4d271d64-317b-41d4-9d0d-4d02e01b1628","apiVersion":"logstash.k8s.elastic.co/v1alpha1","resourceVersion":"2068825"},"reason":"AssociationStatusChange"}