[BUG] [opensearch-dashboards] "certificate signature failure" with self signed certs

[iamtanmay]

Describe the bug

Error:

"tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: certificate signature failure"} "tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}

My dashboards chart. Host:

opensearchHosts: "https://opensearch-cluster-master.opensearch.svc.cluster.local:9200"

Config:

`config: opensearch_dashboards.yml: |

 server:
 
     name: dashboards
     
     host: "{{.Values.serverHost}}"
     
     ssl:
     
         enabled: true
         
         key: /usr/share/opensearch-dashboards/config/certs/client-key.pem
         
         certificate: /usr/share/opensearch-dashboards/config/certs/client.pem
         
 opensearch:
 
    ssl:
    
        verificationMode: certificate
        
        key: /usr/share/opensearch-dashboards/config/certs/node1-key.pem
        
        certificate: /usr/share/opensearch-dashboards/config/certs/node1.pem
        
        certificateAuthorities: /usr/share/opensearch-dashboards/config/certs/root-ca.pem`

Using Ingress annotation to force HTTPS to get access to backend, makes backend work, but breaks dashboard. Without it, dashboard works, but backend is inaccessible via browser/curl

annotations: nginx.ingress.kubernetes.io/backend-protocol: HTTPS

Self signed client certs are generated as follows. I use same node cert for all 3 backend nodes. All certs and node, client, admin keys are mounted in nodes and dashboard containers:

`# Root CA

export SUBJ="/C=CZ/ST=PRG/L=PRG/O=DHL/OU=OSS/CN=opensearch-backend.apps-odfe.datalake-dev.dhl.com" export DAYS=365

openssl genrsa -out root-ca-key.pem 2048 openssl req -new -x509 -sha256 -key root-ca-key.pem -subj $SUBJ -out root-ca.pem -days $DAYS

Admin

export SUBJ="/C=CZ/ST=PRG/L=PRG/O=DHL/OU=OSS/CN=A" openssl genrsa -out admin-key-temp.pem 2048 openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem openssl req -new -key admin-key.pem -subj $SUBJ -out admin.csr openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days $DAYS

Node

export SUBJ="/C=CZ/ST=PRG/L=PRG/O=DHL/OU=OSS/CN=opensearch-cluster-master.opensearch.svc.cluster.local" openssl genrsa -out node1-key-temp.pem 2048 openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem openssl req -new -key node1-key.pem -subj $SUBJ -out node1.csr echo 'subjectAltName=DNS:node1.dns.a-record' > node1.ext openssl x509 -req -in node1.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node1.pem -days $DAYS -extfile node1.ext

Client

export SUBJ="/C=CZ/ST=PRG/L=PRG/O=DHL/OU=OSS/CN=opensearch-dashboards.opensearch.svc.cluster.local" openssl genrsa -out client-key-temp.pem 2048 openssl pkcs8 -inform PEM -outform PEM -in client-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out client-key.pem openssl req -new -key client-key.pem -subj $SUBJ -out client.csr echo 'subjectAltName=DNS:client.dns.a-record' > client.ext openssl x509 -req -in client.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out client.pem -days $DAYS -extfile client.ext`

I tried to verify if the signature of the client.pem is invalid, but got ok:

openssl verify -verbose -CAfile root-ca.pem client.pem client.pem: OK

To Reproduce Steps to reproduce the behavior:

1. Generate self signed certs as above

2. Mount certs to 'opensearch' namespace

3. helm deploy default opensearch and dashboard charts ver 2.3.0 to kubernetes cluster with a simple Ingress for both in same namespace and the above mentione HTTPS forcing annotation

4. Wait for all containers to be ready via checking the logs

5. I am able to see the default response of backend in browser and via curl, but dashboards gives 502 bad gateway, looking in container shows the error "certificate signature failure". Removing the HTTPS forcing annotation in Ingress gives dashboards access but I am unable to reach backend

Expected behavior Dashboards should be accessible as normal

Host/Environment (please complete the following information):

• Helm Version: 3.9.0
• Kubernetes Version: 1.21

[iamtanmay]

Just to add, there is nothing in the node container logs, so it looks like the dashboard service couldn't even reach it

[rishabh6788]

@prudhvigodithi @peterzhuamazon can you please take a look at this?

[peterzhuamazon]

This seems more of an issue on security plugin. Let us know if we can move this issue to their repo. Thanks.

[el33tneel]

@iamtanmay hey were you able to solve this? I am getting the same error and I have similar configuration