SELKS
SELKS copied to clipboard
Published
20 hours ago •
StamusNetworks
StamusNetworks
Securing Elasticsearch (X-Pack enabled)
Hi,
Is there an "easy way" with SELKS to secure Elasticsearch node to star receiving data securely from beats like Winlogbeat? By "easy way" I mean an easier way, other than the long way with many steps described in Elasticsearch official documentation. I wonder if SELKS has an easy way to enable secure data transmission either if secure connection comes enabled by default using Nginx (or can be setup) or any other simple way (easy).
Now that Elasticsearch released XPACK for the free distribution, it would be wonderful to have SELKS deployed with secure data transmission enabled by default, and then give user the option to disable it if needed.
Thank you
I have the same question. I can enable xpack but after this scirius cannot connect to elastic and get statistics. Is it possible to use authentication? I didn't find it in settings.py
You can ship encrypted logs to SELKS via filebeat - that is no problem. So you should basically set up the remote sender to use filebeat with ssl cert encryption to ship to SELKS.
Was that what you had in mind?
I have a different goal. I want to set up monitoring for Elastic through Zabbix and HTTP checks. It works but I need secure connection to Elastic. For free it is possible using native realm. So it's nessesary to set up authentication between scirius, kibana, elastic, evebox, etc....
So, can I use a cert that SELKS suite has already or do I need to create my own cert? Is there a way to setup Winlogbeat with SSL cert instead of Filebeat? I would like to keep using Winlogbeat index patterns, but need to secure the communication between server and clients.
I don't want to break Scirius or anything else in the SELKS suite by messing too much with xpack. So I would like to know what can be modified and how, to keep SELKS running, but having the transmission secured as well.
I think you can create your own cert and use it with filebeat. I am not familiar with Winlogbeat to be honest. I dont think you will be breaking anything by adding more logs.
OK. I will try both then. My only concern is that could break any SELKS internal communication if I enable xpack and start messing with the security options. I tried to upgrade ELK once, from 6.8.8 to 7, by just following ELK steps and I got SELKS broken.
I'll try to be very careful.
Thanks @pevma
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 192.168.0.179,127.0.0.1,localhost
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12
To my mind nothing wrong in yml. Do i understand correctly that after enabling xpack you can not set passwords for default accounts with elasticsearch-setup-passwords? What's in logs after starting elasticsearch with enabled xpack?
That is correct. I cannot connect to ES to set the passwords due to a connection issue.
This is the console output:
user1@testserver:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Connection failure to: https://192.168.0.179:9200/_xpack/security/_authenticate?pretty failed: Connection refused (Connection refused)
ERROR: Failed to connect to elasticsearch at https://192.168.0.179:9200/_xpack/security/_authenticate?pretty. Is the URL correct and elasticsearch running?
user1@testserver:~$
This link contains the ES log output, which is big: https://pastebin.com/H302an8H
I believe this could be a Java issue, but I will let the experts to determine the root cause ;)
Maybe Nginx is doing some redirection causing https://192.168.0.179:9200 to be unreachable
I have elastic published through nginx too.But i published it on port 9201 with SSL and nginx redirects all requests to 9200. So you can stop nginx and try again. What about ES it would be better you share clean log after the start of elasticsearch. Could you stop ES, rename default file log, just start it and share the newly created log?
ManuelFFF [email protected] 17 апреля 2020 г. 23:22:01 написал:
Maybe Nginx is doing some redirection causing https://192.168.0.179:9200 to be unreachable
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
I am good with the default SELKS settings and I would like to continue using Nginx as until now. In fact I would like to try avoiding the need to modify anything in the SELKS configuration, if possible, but if inevitable, I need to know exactly what changes I should or can make.
This is the clean log output:
user1@testserver:~$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2020-04-17T16:53:32,486][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [74.3gb], net total_space [88gb], types [ext4]
[2020-04-17T16:53:32,489][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] heap size [4.9gb], compressed ordinary object pointers [true]
[2020-04-17T16:53:32,630][INFO ][o.e.n.Node ] [z_C3Jta] node name derived from node ID [z_C3JtakT4e0YWZwI5z1Sw]; set [node.name] to override
[2020-04-17T16:53:32,631][INFO ][o.e.n.Node ] [z_C3Jta] version[6.8.8], pid[4853], build[default/deb/2f4c224/2020-03-18T23:22:18.622755Z], OS[Linux/4.19.0-8-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]
[2020-04-17T16:53:32,631][INFO ][o.e.n.Node ] [z_C3Jta] JVM arguments [-Xms5g, -Xmx5g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2237996795266627361, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2020-04-17T16:53:33,787][ERROR][o.e.b.Bootstrap ] [z_C3Jta] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.8.jar:6.8.8]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
... 15 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
... 15 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_242]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_242]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_242]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
... 15 more
[2020-04-17T16:53:33,818][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [z_C3Jta] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.8.jar:6.8.8]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_242]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_242]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_242]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
user1@testserver:~$
Your ES has errors with xpack and to my mind doesn't work as expected. ERROR][o.e.b.Bootstrap ] [z_C3Jta] Exception java.lang.IllegalStateException: failed to load plugin class Further you can find Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory And further Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12 Maybe user elasticsearch can not read file with certificates? Try to disable all xpack setting except just enabling xpack. Start ES, check for the errors and set passwords. I have done it on my productions with the same 6.8.8 and it works. But i reverted all back because of the reasons explained here earlier.
Cert was generated and stored in /etc/elasticsearch. I did not see any step regarding the need to change any permission for the cert file, but I guess I could try that too.
user1@testserver:~$ sudo systemctl stop elasticsearch
user1@testserver:~$ sudo mv /var/log/elasticsearch/elasticsearch.log /var/log/elasticsearch/elasticsearch-2020-04-17-2.log
user1@testserver:~$ sudo systemctl start elasticsearch
user1@testserver:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Connection failure to: http://192.168.0.179:9200/_xpack/security/_authenticate?pretty failed: Connection refused (Connection refused)
ERROR: Failed to connect to elasticsearch at http://192.168.0.179:9200/_xpack/security/_authenticate?pretty. Is the URL correct and elasticsearch running?
user1@testserver:~$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2020-04-17T17:28:45,196][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [74.2gb], net total_space [88gb], types [ext4]
[2020-04-17T17:28:45,199][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] heap size [4.9gb], compressed ordinary object pointers [true]
[2020-04-17T17:28:45,310][INFO ][o.e.n.Node ] [z_C3Jta] node name derived from node ID [z_C3JtakT4e0YWZwI5z1Sw]; set [node.name] to override
[2020-04-17T17:28:45,310][INFO ][o.e.n.Node ] [z_C3Jta] version[6.8.8], pid[8194], build[default/deb/2f4c224/2020-03-18T23:22:18.622755Z], OS[Linux/4.19.0-8-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]
[2020-04-17T17:28:45,310][INFO ][o.e.n.Node ] [z_C3Jta] JVM arguments [-Xms5g, -Xmx5g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-182354423605169464, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2020-04-17T17:28:48,400][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [aggs-matrix-stats]
[2020-04-17T17:28:48,400][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [analysis-common]
[2020-04-17T17:28:48,400][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [ingest-common]
[2020-04-17T17:28:48,400][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [ingest-geoip]
[2020-04-17T17:28:48,400][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [ingest-user-agent]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [lang-expression]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [lang-mustache]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [lang-painless]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [mapper-extras]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [parent-join]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [percolator]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [rank-eval]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [reindex]
[2020-04-17T17:28:48,401][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [repository-url]
[2020-04-17T17:28:48,402][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [transport-netty4]
[2020-04-17T17:28:48,402][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [tribe]
[2020-04-17T17:28:48,402][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-ccr]
[2020-04-17T17:28:48,402][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-core]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-deprecation]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-graph]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-ilm]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-logstash]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-ml]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-monitoring]
[2020-04-17T17:28:48,403][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-rollup]
[2020-04-17T17:28:48,404][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-security]
[2020-04-17T17:28:48,404][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-sql]
[2020-04-17T17:28:48,404][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-upgrade]
[2020-04-17T17:28:48,404][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-watcher]
[2020-04-17T17:28:48,404][INFO ][o.e.p.PluginsService ] [z_C3Jta] no plugins loaded
[2020-04-17T17:28:54,308][INFO ][o.e.x.s.a.s.FileRolesStore] [z_C3Jta] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-04-17T17:28:56,109][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [z_C3Jta] [controller/8255] [Main.cc@109] controller (64 bit): Version 6.8.8 (Build 398321ce7838fd) Copyright (c) 2020 Elasticsearch BV
[2020-04-17T17:28:57,201][DEBUG][o.e.a.ActionModule ] [z_C3Jta] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-04-17T17:28:58,017][INFO ][o.e.d.DiscoveryModule ] [z_C3Jta] using discovery type [zen] and host providers [settings]
[2020-04-17T17:28:59,205][INFO ][o.e.n.Node ] [z_C3Jta] initialized
[2020-04-17T17:28:59,206][INFO ][o.e.n.Node ] [z_C3Jta] starting ...
[2020-04-17T17:28:59,389][INFO ][o.e.t.TransportService ] [z_C3Jta] publish_address {192.168.0.179:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}, {192.168.0.179:9300}
[2020-04-17T17:28:59,485][INFO ][o.e.b.BootstrapChecks ] [z_C3Jta] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-04-17T17:28:59,530][ERROR][o.e.b.Bootstrap ] [z_C3Jta] node validation exception
[1] bootstrap checks failed
[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2020-04-17T17:28:59,538][INFO ][o.e.n.Node ] [z_C3Jta] stopping ...
[2020-04-17T17:28:59,595][INFO ][o.e.n.Node ] [z_C3Jta] stopped
[2020-04-17T17:28:59,595][INFO ][o.e.n.Node ] [z_C3Jta] closing ...
[2020-04-17T17:28:59,619][INFO ][o.e.n.Node ] [z_C3Jta] closed
user1@testserver:~$
I think I have to enable at least a 2nd config line:
[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
Yes, my settings were different, i just wanted to enable native authentication and i enabled another setting of xpack in section authentication. https://www.elastic.co/guide/en/elasticsearch/reference/current/native-realm.html
But what concerns your wishes you have enabled section transport, so it's another topic. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/bootstrap-checks.html I think you have to notice cluster that it has only one node in production. Otherwise it thinks that it's impossible to run alone by design.
Hi,
Before I try the native realm settings or the bootstrap check settings, I just to be sure we are in the same page:
1- I am using SELKS 5.0 (Debian 10 upgrade applied via SELKS script) with the default configuration (only changed loopback address to access ELK from other server or PC) 2- I only have one ELK node
My goals are: 1- Secure internal ELK communication 2- Secure foreign connections (be able to use https) coming from different beats (PCs and servers out of the ELK network) to ES, Kibana and Logstash 3- Create different users with different level access, so each one can see only specific dashboards
I don't have any other specific settings in mind as I'm just starting to dabble in the ELK world and I am setting up my first node the best I can. I thought that with the default SELKS configuration, I was going to be able to reach my three basic objectives without major complications. I don't even know if I need to have enabled or not the section transport.
With that in mind and assuming I have just installed SELKS 5 from scratch, what modifications do I need to make in the configuration, to reach my goals?
I have many other questions related to this topic only, but I don't want to overload you with too many questions.
Any advice is welcome ... Thank you
Could it be ELK is having difficulties to find/interact with the included Java version when enable XPACK? I just noticed that many of the errors exposed here (https://github.com/StamusNetworks/SELKS/issues/216#issuecomment-615459608) are Java related.
user1@testserver:~$ sudo journalctl -f -u elasticsearch
-- Logs begin at Mon 2020-04-20 01:54:44 EDT. --
Apr 20 10:49:12 testserver systemd[1]: Started Elasticsearch.
Apr 20 10:49:12 testserver elasticsearch[35525]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Apr 20 10:49:24 testserver systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Apr 20 10:49:24 testserver systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Dont think that could be it. There is a nginx config in place and Scirius is controlling the authorization - think that might be the reason
Oh, I see. That's what I meant by not "crash" any of the specific settings that SELKS has, when I wanted to modify something in ELK. So what steps should I take to reach my 3 simple goals, once the installation of SELKS 5 is complete?
Tried giving ES user permissions to read the certificate file, but this did not resolved the issue.
Changed from this:
-rw------- 1 root elasticsearch 3443 Apr 17 14:04 elastic-certificates.p12
-rw-rw---- 1 root elasticsearch 199 Apr 9 09:53 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch 3359 Apr 21 12:54 elasticsearch.yml
To this:
-rw-rw---- 1 root elasticsearch 3443 Apr 17 14:04 elastic-certificates.p12
-rw-rw---- 1 root elasticsearch 199 Apr 9 09:53 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch 3359 Apr 21 12:54 elasticsearch.yml
Console output;
user1@testserver:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Connection failure to: https://192.168.0.179:9200/_xpack/security/_authenticate?pretty failed: Connection refused (Connection refused)
ERROR: Failed to connect to elasticsearch at https://192.168.0.179:9200/_xpack/security/_authenticate?pretty. Is the URL correct and elasticsearch running?
Connection fails due to ES service failed to start, and service failed to start as soon I enable xpack in ES config and attempt to restart the service (logs output at the end of the post).
Again, this is is a fresh SELKS 5 installation, then upgraded as described in https://github.com/StamusNetworks/SELKS/wiki/SELKS-5.0-Buster-upgrade. I have not added any content yet nor enabled any other feature nor modified the configuration. I only replaced 127.0.0.1 or localhost by an IP to access ELK from the local network.
I am stuck on the start and can't find what could be causing this issue when I have not modified anything on SELKS yet.
Could this be a bug affecting XPACK on this SELKS distribution?
I am available to assist with tests or try solutions.
Thanks
Log output:
user1@testserver:~$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2020-04-21T00:00:01,371][INFO ][o.e.c.m.MetaDataIndexTemplateService] [z_C3Jta] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2020-04-21T00:33:44,908][INFO ][o.e.c.m.MetaDataCreateIndexService] [z_C3Jta] [sessions2-200421] creating index, cause [auto(bulk api)], templates [sessions2_template], shards [1]/[0], mappings [session]
[2020-04-21T00:33:46,344][INFO ][o.e.c.r.a.AllocationService] [z_C3Jta] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[sessions2-200421][0]] ...]).
[2020-04-21T01:00:00,006][INFO ][o.e.x.m.e.l.LocalExporter] [z_C3Jta] cleaning up [4] old indices
[2020-04-21T01:00:00,127][INFO ][o.e.c.m.MetaDataDeleteIndexService] [z_C3Jta] [.monitoring-kibana-6-2020.04.13/P4BxV-gYRsOkN33j5FaCxg] deleting index
[2020-04-21T01:00:00,128][INFO ][o.e.c.m.MetaDataDeleteIndexService] [z_C3Jta] [.monitoring-kibana-6-2020.04.14/VIB1LOl4R_KI8PelNQ3vIw] deleting index
[2020-04-21T01:00:00,128][INFO ][o.e.c.m.MetaDataDeleteIndexService] [z_C3Jta] [.monitoring-es-6-2020.04.13/1W7ARJAkTOmlOhQ0tKM5WQ] deleting index
[2020-04-21T01:00:00,128][INFO ][o.e.c.m.MetaDataDeleteIndexService] [z_C3Jta] [.monitoring-es-6-2020.04.14/RJVKZVDBRiuQwJJNOT1onw] deleting index
[2020-04-21T01:30:00,004][INFO ][o.e.x.m.MlDailyMaintenanceService] [z_C3Jta] triggering scheduled [ML] maintenance tasks
[2020-04-21T01:30:00,005][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [z_C3Jta] Deleting expired data
[2020-04-21T01:30:00,275][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [z_C3Jta] Completed deletion of expired ML data
[2020-04-21T01:30:00,275][INFO ][o.e.x.m.MlDailyMaintenanceService] [z_C3Jta] Successfully completed [ML] maintenance tasks
[2020-04-21T09:16:21,543][INFO ][o.e.n.Node ] [z_C3Jta] stopping ...
[2020-04-21T09:16:21,591][INFO ][o.e.x.w.WatcherService ] [z_C3Jta] stopping watch service, reason [shutdown initiated]
[2020-04-21T09:16:22,477][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [z_C3Jta] [controller/4921] [Main.cc@148] Ml controller exiting
[2020-04-21T09:16:22,598][INFO ][o.e.x.m.p.NativeController] [z_C3Jta] Native controller process has stopped - no new native processes can be started
[2020-04-21T09:16:26,625][INFO ][o.e.n.Node ] [z_C3Jta] stopped
[2020-04-21T09:16:26,625][INFO ][o.e.n.Node ] [z_C3Jta] closing ...
[2020-04-21T09:16:26,693][INFO ][o.e.n.Node ] [z_C3Jta] closed
[2020-04-21T09:16:51,457][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [73.6gb], net total_space [88gb], types [ext4]
[2020-04-21T09:16:51,474][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] heap size [4.9gb], compressed ordinary object pointers [true]
[2020-04-21T09:16:59,379][INFO ][o.e.n.Node ] [z_C3Jta] node name derived from node ID [z_C3JtakT4e0YWZwI5z1Sw]; set [node.name] to override
[2020-04-21T09:16:59,380][INFO ][o.e.n.Node ] [z_C3Jta] version[6.8.8], pid[29668], build[default/deb/2f4c224/2020-03-18T23:22:18.622755Z], OS[Linux/4.19.0-8-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]
[2020-04-21T09:16:59,380][INFO ][o.e.n.Node ] [z_C3Jta] JVM arguments [-Xms5g, -Xmx5g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2389892655683333288, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [aggs-matrix-stats]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [analysis-common]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [ingest-common]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [ingest-geoip]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [ingest-user-agent]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [lang-expression]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [lang-mustache]
[2020-04-21T09:17:17,368][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [lang-painless]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [mapper-extras]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [parent-join]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [percolator]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [rank-eval]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [reindex]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [repository-url]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [transport-netty4]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [tribe]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-ccr]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-core]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-deprecation]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-graph]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-ilm]
[2020-04-21T09:17:17,369][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-logstash]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-ml]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-monitoring]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-rollup]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-security]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-sql]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-upgrade]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] loaded module [x-pack-watcher]
[2020-04-21T09:17:17,370][INFO ][o.e.p.PluginsService ] [z_C3Jta] no plugins loaded
[2020-04-21T09:17:34,400][INFO ][o.e.x.s.a.s.FileRolesStore] [z_C3Jta] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-04-21T09:17:37,517][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [z_C3Jta] [controller/29823] [Main.cc@109] controller (64 bit): Version 6.8.8 (Build 398321ce7838fd) Copyright (c) 2020 Elasticsearch BV
[2020-04-21T09:17:38,862][DEBUG][o.e.a.ActionModule ] [z_C3Jta] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-04-21T09:17:40,569][INFO ][o.e.d.DiscoveryModule ] [z_C3Jta] using discovery type [zen] and host providers [settings]
[2020-04-21T09:17:42,239][INFO ][o.e.n.Node ] [z_C3Jta] initialized
[2020-04-21T09:17:42,240][INFO ][o.e.n.Node ] [z_C3Jta] starting ...
[2020-04-21T09:17:43,710][INFO ][o.e.t.TransportService ] [z_C3Jta] publish_address {192.168.0.179:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}, {192.168.0.179:9300}
[2020-04-21T09:17:43,843][INFO ][o.e.b.BootstrapChecks ] [z_C3Jta] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-04-21T09:17:47,121][INFO ][o.e.c.s.MasterService ] [z_C3Jta] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {z_C3Jta}{z_C3JtakT4e0YWZwI5z1Sw}{HWKUxH7YRomrXFgZbA2SEg}{192.168.0.179}{192.168.0.179:9300}{ml.machine_memory=10468331520, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2020-04-21T09:17:47,127][INFO ][o.e.c.s.ClusterApplierService] [z_C3Jta] new_master {z_C3Jta}{z_C3JtakT4e0YWZwI5z1Sw}{HWKUxH7YRomrXFgZbA2SEg}{192.168.0.179}{192.168.0.179:9300}{ml.machine_memory=10468331520, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {z_C3Jta}{z_C3JtakT4e0YWZwI5z1Sw}{HWKUxH7YRomrXFgZbA2SEg}{192.168.0.179}{192.168.0.179:9300}{ml.machine_memory=10468331520, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2020-04-21T09:17:47,401][INFO ][o.e.h.n.Netty4HttpServerTransport] [z_C3Jta] publish_address {192.168.0.179:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}, {192.168.0.179:9200}
[2020-04-21T09:17:47,401][INFO ][o.e.n.Node ] [z_C3Jta] started
[2020-04-21T09:17:48,593][INFO ][o.e.c.s.ClusterSettings ] [z_C3Jta] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-04-21T09:17:58,123][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [z_C3Jta] Failed to clear cache for realms [[]]
[2020-04-21T09:17:58,215][INFO ][o.e.l.LicenseService ] [z_C3Jta] license [30386688-598d-4f99-b93e-a9ed2b4f5fdb] mode [basic] - valid
[2020-04-21T09:17:58,247][INFO ][o.e.g.GatewayService ] [z_C3Jta] recovered [39] indices into cluster_state
[2020-04-21T09:19:35,394][INFO ][o.e.c.r.a.AllocationService] [z_C3Jta] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_task_manager][0], [logstash-2020.04.09][0], [.kibana_1][0]] ...]).
[2020-04-21T09:19:51,946][INFO ][o.e.c.m.MetaDataIndexTemplateService] [z_C3Jta] adding template [.management-beats] for index patterns [.management-beats]
[2020-04-21T09:19:57,666][INFO ][o.e.c.m.MetaDataIndexTemplateService] [z_C3Jta] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2020-04-21T12:54:29,645][INFO ][o.e.n.Node ] [z_C3Jta] stopping ...
[2020-04-21T12:54:29,685][INFO ][o.e.x.w.WatcherService ] [z_C3Jta] stopping watch service, reason [shutdown initiated]
[2020-04-21T12:54:29,972][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [z_C3Jta] [controller/29823] [Main.cc@148] Ml controller exiting
[2020-04-21T12:54:29,972][INFO ][o.e.x.m.p.NativeController] [z_C3Jta] Native controller process has stopped - no new native processes can be started
[2020-04-21T12:54:34,257][INFO ][o.e.n.Node ] [z_C3Jta] stopped
[2020-04-21T12:54:34,257][INFO ][o.e.n.Node ] [z_C3Jta] closing ...
[2020-04-21T12:54:34,281][INFO ][o.e.n.Node ] [z_C3Jta] closed
[2020-04-21T12:54:53,509][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [73.6gb], net total_space [88gb], types [ext4]
[2020-04-21T12:54:53,513][INFO ][o.e.e.NodeEnvironment ] [z_C3Jta] heap size [4.9gb], compressed ordinary object pointers [true]
[2020-04-21T12:54:53,801][INFO ][o.e.n.Node ] [z_C3Jta] node name derived from node ID [z_C3JtakT4e0YWZwI5z1Sw]; set [node.name] to override
[2020-04-21T12:54:53,802][INFO ][o.e.n.Node ] [z_C3Jta] version[6.8.8], pid[49216], build[default/deb/2f4c224/2020-03-18T23:22:18.622755Z], OS[Linux/4.19.0-8-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]
[2020-04-21T12:54:53,802][INFO ][o.e.n.Node ] [z_C3Jta] JVM arguments [-Xms5g, -Xmx5g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-262103612002416489, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2020-04-21T12:54:55,202][ERROR][o.e.b.Bootstrap ] [z_C3Jta] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.8.jar:6.8.8]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
... 15 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
... 15 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_242]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_242]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_242]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
... 15 more
[2020-04-21T12:54:55,215][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [z_C3Jta] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.8.jar:6.8.8]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_242]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_242]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_242]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_242]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_242]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.8.jar:6.8.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.8.jar:6.8.8]
... 6 more
Hi, as I read before you are new to ES. So here is not a good place to find a solution how to change settings for customized integration of ES, evebox, moloch, etc.... To my mind you should better install pure ES, then integrate it with winlogbeat,then enable security features. After this you dive deeper into ES and understand how to realize it in SELKS.
Hi, I'm not a developer but just a user. I wrote you earlier
But what concerns your wishes you have enabled section transport, so it's another topic. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/bootstrap-checks.html I think you have to notice cluster that it has only one node in production. Otherwise it thinks that it's impossible to run alone by design.
So correct your YML and ES will start
@ManuelFFF what are the permission to the /etc/elasticsearch directory, elasticsearch user may be unable to enter it.
On the create password command, check on which address ES is running (ss -nltp) then do add '-u URL' to initial command.
Regarding connection from Scirius to Elasticsearch once xpack is installed and activated, there may be difficulty here.
@VN1977 - what were the settings you adjusted in your yaml ? same as suggested on the link or others as well ?
Oh now I see @VN1977 . I have misinterpreted your words and intentions. I am very sorry for the confusion.
Regarding your suggestion. Actually I have not "activated" anything after the installation of SELKS 5. If section transport appears enabled, it is because it was like that from the beginning.
I just tried to enable the xpack basic package "xpack.security.enabled", but when restarting the ES service I get an error saying I also have to enable the transport related packages
"[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false] "
Maybe that's why you tell me that I have the transport section activated. Not that I wanted to, it's that the software requests it.
Anyway, I appreciate the advice and I'm going to read the article you suggest.
Thank you
Regarding connection from Scirius to Elasticsearch once xpack is installed and activated, there may be difficulty here.
If you activate authentication in xpack, SELKS definitely will be broken. Scirius doesn't have info from Elasticsearch and all graphs are empty. The same situation with page Hunt. @pevma, maybe it's possible to adjust something in scirius to provide authentication to ES?
@VN1977 - what were the settings you adjusted in your yaml ? same as suggested on the link or others as well ?
I just enabled xpack.security.authc.realms as wrote here https://www.elastic.co/guide/en/elasticsearch/reference/6.8/configuring-native-realm.html After this I created passwords for all users like elastic, kibana, beats, etc.. Changed settings for kibana, logstash and everything worked except scirius. Somehow I need to tell scirius login and password to connect to ES.