Better way to set certificate_authorities for metricbeat / filebeat containers

[sratz]

Proposal

Consider the following situation:

• Certificate provided via cert-manager using non-public CA
• The resulting certificate secret will not contain a ca.crt.
• For Kibana to be able to connect to ES, this C must be added manually according to https://github.com/elastic/cloud-on-k8s/issues/1494#issuecomment-519003310, which is rather straight-forward:

  apiVersion: kibana.k8s.elastic.co/v1
  kind: Kibana
  metadata:
    name: logsearch
    namespace: logsearch-prod
  spec:
    version: 8.6.2
    count: 1
    elasticsearchRef:
      name: logsearch
+   config:
+     elasticsearch.ssl.certificateAuthorities: /mnt/rootcas/ca.crt
+   podTemplate:
+     spec:
+       volumes:
+       - name: rootcas
+         secret:
+           secretName: rootcas
+       containers:
+       - name: kibana
+         volumeMounts:
+         - name: rootcas
+           mountPath: /mnt/rootcas
    http:
      tls:
        certificate:
          secretName: elk-http-tls
        selfSignedCertificate:
          disabled: true

• But now consider a simple (self-)monitoring setup:

  monitoring:
    metrics:
      elasticsearchRefs:
      - name: logsearch
    logs:
      elasticsearchRefs:
      - name: logsearch

The config.elasticsearch.ssl.certificateAuthorities option does not apply to the meticbeat and filebeat configuration.

The only way to get this set is:

  apiVersion: kibana.k8s.elastic.co/v1
  kind: Kibana
  metadata:
    name: logsearch
    namespace: logsearch-prod
  spec:
    version: 8.6.2
    count: 1
    elasticsearchRef:
      name: logsearch
    config:
      elasticsearch.ssl.certificateAuthorities: /mnt/rootcas/ca.crt
    podTemplate:
      spec:
        volumes:
        - name: rootcas
          secret:
            secretName: rootcas
        containers:
        - name: kibana
          volumeMounts:
          - name: rootcas
            mountPath: /mnt/rootcas
+       - name: filebeat
+         args:
+           - '-c'
+           - /etc/filebeat-config/filebeat.yml
+           - '-e'
+           - '-E'
+           - 'output.elasticsearch.ssl.certificate_authorities=["/mnt/rootcas/ca.crt"]'
+           - '-E'
+           - 'setup.kibana.ssl.certificate_authorities=["/mnt/rootcas/ca.crt"]'
+         volumeMounts:
+         - name: rootcas
+           mountPath: /mnt/rootcas
+       - name: metricbeat
+         args:
+           - '-c'
+           - /etc/metricbeat-config/metricbeat.yml
+           - '-e'
+           - '-E'
+           - 'output.elasticsearch.ssl.certificate_authorities=["/mnt/rootcas/ca.crt"]'
+           - '-E'
+           - 'metricbeat.modules.0.ssl.certificate_authorities=["/mnt/rootcas/ca.crt"]'
+         volumeMounts:
+         - name: rootcas
+           mountPath: /mnt/rootcas
    http:
      tls:
        certificate:
          secretName: elk-http-tls
        selfSignedCertificate:
          disabled: true
    monitoring:
      metrics:
        elasticsearchRefs:
        - name: logsearch
      logs:
        elasticsearchRefs:
        - name: logsearch

Overwriting args like this is quite error-prone. Alternatively, the whole filebeat.yml / elasticbeat.yml would have to be provided.

It would be better if this could be properly set via a dedicated configuration option just like config.elasticsearch.ssl.certificateAuthorities.

Environment

• ECK version:

  2.7.0

[pebrc]

One thing we could potentially do is add section similar to what we did recently for transport:

  http:
    tls:
      certificateAuthorities:
        configMapName: trust

and then have the operator ensure that:

1. the trusted CA certificates are propagated to the monitoring Beats
2. the trusted CA certificates are propagated across associations e.g. Kibana