Sending logs with Beats using SSL -> Rest

[fnzv]

Hello, I'm trying to ship some logs directly from Beats to an Elasticsearch Cluster with Rest plugin. I have used the basic configuration with SSL enabled:

http.type: ssl_netty4
readonlyrest:
    enable: true
    ssl:
      enable: true
      keystore_file: "/elasticsearch/plugins/readonlyrest/keystore.jks"
      keystore_pass: readonlyrest
      key_pass: readonlyrest

    response_if_req_forbidden: Forbidden

    access_control_rules:
..
..

(Copy Pasted from your Doc the ACL rules)

But from the shipper logs i see: 2017-04-01T16:49:08Z ERR Connecting error publishing events (retrying): Get https://myserverip:9200: read tcp mylocalip:59926-> myserverip:9200: read: connection reset by peer

Do i need to set some flag to allow unverified SSL connections to ES or there is no support for direct SSL connections to Rest via Beats?

I want to use direct SSL connections to ES to avoid using a Logstash in the middle

[sscarduzio]

Do i need to set some flag to allow unverified SSL connections to ES or there is no support for direct SSL connections to Rest via Beats?

Probably, if your cert if self signed.

[fnzv]

Do you know how can i set this on ES?

[ld57]

Hi,

here how I did :

configuration of metricbeat - elasticsearch section

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  output.elasticsearch:
  username: metricbeat
  password: hereyourpasswordformetricbeat
  protocol: https
  hosts: ["xx.xx.xx.xx:9x00"]
  worker: 1
  index: "log_metricbeat-%{+yyyy.MM}"
  template.enabled: false
  template.versions.2x.enabled: false
  ssl.enabled: true
  ssl.certificate_authorities: ["./certs/your-rootca_cert.pem"]
  ssl.certificate: "./certs/your_srv_cert.pem"
  ssl.key: "./certs/your_srv_key.pem"

of course, if you do not use ssl, disable it. On my side, I use a ssl wrapper in front of elasticsearch , if you use embedded ssl from readonlyrest, I did not try yet to implement it (using an old version)

now elasticsearch readonlyrest config here just the block

    - name: "metricbeat can write and create its own indices"
      auth_key_sha1: fd2e44724a234234454324253094080986e8fda
      type: allow
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["metricbeat-*",  "log_metricbeat*"]

[sscarduzio]

@ld57 thanks for sharing this, I'm going to copy paste your comment into official documentation. I hope you don't mind!

I also took the liberty to remove the <no-index> as nowadays ReadonlyREST does not need it anymore (it skips checking for indices if the request does not involve indices).

[fnzv]

Hi, I solved disabling SSL and encrypting TCP traffic via SSH. You can bind the remote host on the shipper with a no shell user via: ssh -L9200:127.0.0.1:9200 -N noshelluser@remotehost After you created the user with no shell and added the ssh key on the host Then you can configure filebeat to send logs to localhost:9200 without SSL and the logs will be shipped to remote ES via an SSH tunnel

You can mark this issue closed

[sscarduzio]

what if ssh dies? who restarts it for you?

[ld57]

@sscarduzio no problem, It is a part of that I did not post yet, and all my work regarding guide :) Regarding quotes, I still use 1.13.2 since i did not take time to go to 1.14.0. I ll go only to es 2.4.4 for now because of some code limitation.

@fnzv , you really should go for an another way, by using the internal netty ssl, or else go to wrapper solution like stunnel, but avoid using ssh, as not a handled service. It will cost no time to use internal ssl, and a bit more if you wrap with stunnel, but costless regarding issue troubleshooting or resolution.

[fnzv]

@ld57 I know, it was just for testing purposes binding the socket via ssh on a screen session. If i want it running as a service i will use autossh , systemd or another way to keep the ssh session on.

[ld57]

okay, it was ust a recommendation in case if you would use that in production environment