Add mTLS Support Via ECK Operator

[DanielChanJA]

Description Currently going through the process of setting up mTLS between Kibana -> ElasticSearch, and I discovered a few quirks.

1. If you set elasticsearchRef in Kibana, ECK's operator sets up a service account token and injects it into the manifest, and ignores any mTLS setup that you might have.
2. If elasticsearchRef is null'd, there's some validation going on that prevents the manifest from getting applied (despite the connection settings being set).

I've been setting up mTLS in Kibana using these values set in config,

          elasticsearch.ssl.verificationMode: full
          elasticsearch.ssl.certificate: /usr/share/kibana/config/kibana-tls-certs/tls.crt
          elasticsearch.ssl.key: /usr/share/kibana/config/kibana-tls-certs/tls.key
          elasticsearch.ssl.certificateAuthorities:
            - /usr/share/kibana/config/kibana-tls-certs/ca.crt

Expectation When specifying the elasticsearchRef, we should be able to specify connection values, if we want to use mTLS using the PKI realm or using a service account token. I would also expect that we would be able to configure WHO the certificate authority is for the deployment and for each of the services.

Also needs better documentation for this, I think reading the codebase cloud-on-k8s is a much better approach than reading the docs on elastic.co if I'm being completely honest.

Our end goal is that mTLS is setup everywhere (without the use of a service mesh).

[greicefaustino]

Hello Daniel,

ECK doesn't natively support mTLS.

mTLS can be configured using a 3rd party service mesh solution as described at: https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/service-meshes

I created a kb to specifically state this: https://support.elastic.co/knowledge/e0368e80

and raised an enhancement request: https://github.com/elastic/cloud-on-k8s/issues/8907

[pebrc]

Cross-posting from the original issue here. But I think mTLS already works today. At least for Kibana and ES. I have not tested other stack components like Agent or Beats. What does not work is to make client certs required because the operator itself needs to talk to Elasticsearch still. But we could look into provisioning client certs for the operator as part of this issue.

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 9.2.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
     # for simplicity we use the Kibana server certificate as a client certificate in this demo
    elasticsearch.ssl.certificate: /mnt/elastic-internal/http-certs/tls.crt
    elasticsearch.ssl.key: /mnt/elastic-internal/http-certs/tls.key
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 9.2.0
  nodeSets:
  - name: default
    count: 3
    config:
      xpack.security.http.ssl.certificate_authorities: 
      # this is the CA generated for Elasticsearch by ECK(I had trouble merging the settings so I am doing this explicitly here)
      - /usr/share/elasticsearch/config/http-certs/ca.crt
      # this is the CA mounted below for the Kibana client certificate
      - /usr/share/elasticsearch/config/kibana-ca/ca.crt
      xpack.security.http.ssl.client_authentication: "optional"
      # this is a demo we do not use memory mapped files here 
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          volumeMounts:
          - name: kb-ca
            mountPath: /usr/share/elasticsearch/config/kibana-ca
        volumes:
        - name: kb-ca
          secret:
            # we use the HTTP certificate of the Kibana server as a client certificate in this demo
            # this assumes that both are deployed in the same namespace
            secretName: kibana-kb-http-certs-public

ECK could automate the mTLS setup between stack apps:

• copy/mount the necessary certificate material
• add the additional configuraiton parameters to Kibana ECK could also introduce mTLS in connections between the operator and ES
• issue a client certificate for the operator
• make all ES clusters trust it
• configure the internal client to present the certificate

if we want to use mTLS using the PKI realm or using a service account token

The auth realm in use by Elasticsearch relies on mTLS being set up. But you can run mTLS with service account tokens or mTLS with PKI based auth.