AWS access Kibana dashboard from my browser (TLS)

[tcstapas]

Since operator makes Kubernete cluster private, no way I can access from outside. I understand port forward is a solution but it can't be permanent solution. That's why I have added ngnix ingress load balancer with type Load balancer so that it can create AWS load balancer. As certificate (es-certs-example-es-cluster) already created by operator, I have used the same in ingress yaml file
But unfortunately it's not working - it's giving me following error -

W0603 12:28:02.515655 7 backend_ssl.go:46] error obtaining PEM from secret operator/es-certs-example-es-cluster: no keypair or CA cert could be found in operator/es-certs-example-es-cluster

Here is my ingress.yaml file configuration -

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: my-ingress namespace: operator annotations: ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: "nginx" nginx.org/ssl-services: kibana-example-es-cluster spec: tls:

• hosts:
  • es1.tcs-tronc.com secretName: es-certs-example-es-cluster rules:
• host: es1.tcs-tronc.com http: paths:
  • path: / backend: serviceName: kibana-example-es-cluster servicePort: 80

[gianrubio]

if you're using nginx-ingress you should add the secure-backend annotation to avoid this issue

nginx.ingress.kubernetes.io/secure-backends: "true"

[tcstapas]

Thanks for the reply. I have tried the above step but no luck. I am facing the same problem.

Here is the error message in the ngnix controller pod

W0605 10:05:03.204634 7 backend_ssl.go:46] error obtaining PEM from secret operator/es-certs-example-es-cluster: no keypair or CA cert could be found in operator/es-certs-example-es-cluster

For the error message, it looks like it's looking for key pair for CA cert but I don't find anything apart from CA.pem within the secret which is generated by operator.

Please let me know if I am missing anything.

[tcstapas]

Additional info:- I have used existing secret which is generated by operator.

Here are few lines from ngnix controller log which is created in kube-system namespace

I0605 10:43:06.921522 7 launch.go:108] Watching for ingress class: nginx I0605 10:43:06.922320 7 launch.go:262] Creating API server client for https://100.64.0.1:443 I0605 10:43:06.930494 7 launch.go:124] validated kube-system/default-http-backend as the default backend I0605 10:43:06.935434 7 controller.go:1190] starting Ingress controller I0605 10:43:06.937326 7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"operator", Name:"my-ingress", UID:"0f8b5327-68a4-11e8-86bc-0e8dab227a9c", APIVersion:"extensions", ResourceVersion:"8483", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress operator/my-ingress I0605 10:43:07.035831 7 leaderelection.go:203] attempting to acquire leader lease... W0605 10:43:07.035864 7 backend_ssl.go:46] error obtaining PEM from secret operator/es-certs-example-es-cluster: no keypair or CA cert could be found in operator/es-certs-example-es-cluster I0605 10:43:07.511479 7 controller.go:1052] ssl certificate "operator/es-certs-example-es-cluster" does not exist in local store I0605 10:43:07.511996 7 controller.go:428] backend reload required I0605 10:43:07.512082 7 metrics.go:34] changing prometheus collector from to default I0605 10:43:07.572568 7 controller.go:438] ingress backend successfully reloaded... I0605 10:43:10.264590 7 controller.go:1052] ssl certificate "operator/es-certs-example-es-cluster" does not exist in local store I0605 10:43:13.597893 7 controller.go:1052] ssl certificate "operator/es-certs-example-es-cluster" does not exist in local store

sh-4.2$ kubectl describe secret es-certs-example-es-cluster -n operator Name: es-certs-example-es-cluster Namespace: operator Labels: Annotations:

Type: Opaque

Data

cerebro.pem: 1619 bytes kibana-key.pem: 1679 bytes kibana.pem: 1619 bytes node-key.pem: 1675 bytes node-keystore.jks: 3498 bytes node.pem: 1619 bytes ca-key.pem: 1675 bytes ca.pem: 1367 bytes truststore.jks: 1032 bytes cerebro-key.pem: 1679 bytes

[gianrubio]

The certificate generated by the operator will not work on nginx, I'd recommend you to generate a valid one using cert-manager or skip it

[tcstapas]

Are you saying to useSSL: false . FYI - I have tested with useSSL it did not work for me. Can we generate ssl using openssl (key and cert and secret) or are you saying to replace the operator generated secret es-certs-example-es-cluster? If yes can you please send some steps to do that? Right now I am stuck and not able to proceed.

[tcstapas]

use-ssl: false unable to create pods for kibana and cerebro-example-es-cluster. It's throwing me following error MountVolume.SetUp failed for volume "es-certs-example-es-cluster" : secrets "es-certs-example-es-cluster" not found

[tcstapas]

I found that you pushed one bug fix for use-ssl: false couple of days back, I took the master branch instead of documented tag and it's working now. Thanks for fixing the defect. If you can guide us how to generate the SSL for this project this will be really helpful for production scenario

[gianrubio]

Are you saying to useSSL: false . FYI - I have tested with useSSL it did not work for me. Can we generate ssl using openssl (key and cert and secret) or are you saying to replace the operator generated secret es-certs-example-es-cluster? If yes can you please send some steps to do that? Right now I am stuck and not able to proceed.

No, I’m not saying to disable ssl, I’m saying to use a valid tls certifate on nginx ingress ( like letsencrypt certificate) and use the secure-backend annotation in the ingress

[tcstapas]

I was trying with self signed certificate first. I hope that will work.. I was trying with following steps - openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -sha256 -subj "/CN=kibana.example.com" -days 1024 -out ca.crt kubectl create secret tls example-com-tls --key=ca.key --cert=ca.crt -n operator

I used this tls in my ingress load balancer.. It's not working. It's giving me 404 error. Am I missing anything?

[gianrubio]

I used this tls in my ingress load balancer.. It's not working. It's giving me 404 error. Am I missing anything?

404 means your ingress is not well configured, please see the ingress logs

[tcstapas]

I fixed the issue. I used useSSL=false and generate one certificate and it worked for me. Thanks! I have another question. How can I restore S3 snapshot to PV (generated by operator)? or how can we migrate existing ELK to this framework, my question is primarily on migrating EBS volume.. I am not sure this is right forum to ask this, in case not please let us know so that I can close this issue and put the question in different thread..

[fabriziofortino]

@tcstapas The error MountVolume.SetUp failed for volume "es-certs-example-es-cluster" : secrets "es-certs-example-es-cluster" not found with useSSL: false has been fixed with #212

[tcstapas]

Thanks! Can you please explain the restore of volume process using this operator. If restore one volume from S3 with same level will it work? - I just checking how we are going to restore AWS EBS if it's destroyed.. Also how can we migrate from traditional elastic search volume to this cluster?