Warning and error logs found in wazuh-indexer in demo environment

[juliamagan]

Wazuh version	Install type	Platform
4.3.0-rc7	Wazuh Indexer	Demo environment

Some error logs were found in https://github.com/wazuh/wazuh-qa/issues/2819#issuecomment-1108504090 and we created #1489. These logs continue to appear, but in addition, these new warnings have been found:

systemctl status wazuh-indexer -l:

may 04 09:18:10 ip-10-0-2-125.us-west-1.compute.internal systemd-entrypoint[16738]: WARNING: An illegal reflective access operation has occurred
may 04 09:18:10 ip-10-0-2-125.us-west-1.compute.internal systemd-entrypoint[16738]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
may 04 09:18:10 ip-10-0-2-125.us-west-1.compute.internal systemd-entrypoint[16738]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
may 04 09:18:10 ip-10-0-2-125.us-west-1.compute.internal systemd-entrypoint[16738]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
may 04 09:18:10 ip-10-0-2-125.us-west-1.compute.internal systemd-entrypoint[16738]: WARNING: All illegal access operations will be denied in a future release

These logs have been found on all machines with wazuh-indexer (bootstrap, master b, master c, and dashboard).

egrep -i "ERROR|WARNING" /var/log/wazuh-indexer/wazuh.log:

• Bootstrap:

[2022-05-04T08:40:50,436][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-3] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 is not enabled or supported in server context
[2022-05-04T09:17:59,309][INFO ][o.o.n.Node               ] [node-3] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms3948m, -Xmx3948m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5513364696324844172, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=2069889024, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-05-04T09:18:09,281][ERROR][o.o.s.a.s.SinkProvider   ] [node-3] Default endpoint could not be created, auditlog will not work properly.

• Master B:

[2022-05-04T09:16:44,967][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-2] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 is not enabled or supported in server context
[2022-05-04T09:23:50,650][INFO ][o.o.n.Node               ] [node-2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms3948m, -Xmx3948m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-4567178945924237329, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=2069889024, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-05-04T09:24:00,307][ERROR][o.o.s.a.s.SinkProvider   ] [node-2] Default endpoint could not be created, auditlog will not work properly.

• Master C:

[2022-05-04T08:51:28,825][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 is not enabled or supported in server context
[2022-05-04T09:26:33,991][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms3948m, -Xmx3948m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-15123537735382070843, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=2069889024, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-05-04T09:26:44,108][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

• Dashboard:

[2022-05-04T09:31:41,457][INFO ][o.o.n.Node               ] [node-7] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2560m, -Xmx2560m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14437330058389193133, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=1342177280, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2022-05-04T09:31:52,361][ERROR][o.o.s.a.s.SinkProvider   ] [node-7] Default endpoint could not be created, auditlog will not work properly.

[c-bordon]

I was researching this error message, it is because the SSLv3 protocol is considered insecure, so it is directly disabled in the JDK. https://www.oracle.com/java/technologies/javase/instructions-to-mitigate-the-ssl-v30-vulnerability.html

Regarding the Indexer configuration, we have the same default configuration as opensearch, that is, we do not set the protocol to use to be SSLv3, so in the future, we could investigate the forced use of TLS and apply it in our configuration as default. https://discuss.elastic.co/t/no-cipher-suites-in-common-sslv3-not-enabled-or-not-supported/33475/6

I could not find the logs again or generate this SSLv3 event again, so probably at some point the nodes try to use this protocol and since it is blocked, it continues with another, I could not verify this, but the application continues to work for which I do not consider it critical and I think that the investigation can be advanced in the next release

[teddytpc1]

These options have been added and tested to resolve this error:

[2022-06-15T09:56:04,970][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-3] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f616c696173657320485454502f312e310d0a486f73743a2031332e35322e3136302e3231393a393230300d0a557365722d4167656e743a204d6f7a696c6c612f352e30207a677261622f302e780d0a4163636570743a202a2f2a0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a

It seems to be resolved. The logs will be checked again tomorrow.

[alberpilot]

We need to modify the SG disallowing the public 9200 access and deploy/run the test over a deployment VM.

[teddytpc1]

Some tasks to copy the python tests scripts were created. The use cases tests playbook is being modified to be executed from the First indexer node. The demo test environment will be running until everything is tested.

[teddytpc1]

The copy task were improved to reduce the execution time. The pytest tests were updated to use the Indexer private IP for the curl command. Some additional packages installation and pip modules were added. A new environment deployment is being created in order to test everything.

[teddytpc1]

Finally the previous changes were reverted and a new branch (1511-test-fix) was created to change the SG and the IP used to run the curl commands for the tests. It was tested with this pipeline.

[jmv74211]

I am reopening this because this error keeps popping up in each wazuh-indexer:

[2022-10-07T11:25:09,098][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

See https://github.com/wazuh/wazuh/issues/15099

[teddytpc1]

I have been analyzing this error message:

• It only occurs only in the first start of the cluster (in the first 2 minutes). After that, the error does not appear again.
• I reviewed the logs to check if there were any preceding related errors but did not find any.
• I could not find a solution. This is the most accurate (but insufficient) answer I have found.
• It also happens with Wazuh indexer 4.4.0.
• It seems to be harmless. The cluster is working properly.
• It does not seem to be related to the Demo environment configuration. I have found that it was reported in a manual test here.

[teddytpc1]

A test enabling audit logs will be made in order to determine if the error is related to that configuration: https://opensearch.org/docs/latest/security-plugin/audit-logs/index/

[teddytpc1]

Test

I performed the test:

1. Restarted Wazuh indexer to check if the error appears on startup (it does): [2022-11-08T18:56:21,349][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly.
2. Modified the opensearch.yml configuration to add the plugins.security.audit.type: internal_opensearch line.
3. Restarted Wazuh indexer again and the error was not generated.
4. Removed the added line.
5. Restarted Wazuh indexer and the error was generated again.

[2022-11-08T18:56:21,349][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2022-11-08T18:59:08,786][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

Conclusion

The error is related to the plugins.security.audit.type configuration. As far as I concern, the error should be a warning instead.

[teddytpc1]

We might open an issue in the Opensearch repository. I will change the status to on hold meanwhile.

[alberpilot]

As @teddytpc1 commented, when using the audit default configuration, i.e. internal_opensearch, there are no errors. Removing this configuration drives to receive the error message due to the following opensearch code:

		// create default sink
		defaultSink = this.createSink(DEFAULTSINK_NAME, settings.get(ConfigConstants.SECURITY_AUDIT_TYPE_DEFAULT), settings, ConfigConstants.SECURITY_AUDIT_CONFIG_DEFAULT);
		if (defaultSink == null) {
			log.error("Default endpoint could not be created, auditlog will not work properly.");
			return;
		}

https://github.com/opensearch-project/security/blob/f431ec2201e1466b7c12528347a1f54cf64387c9/src/main/java/org/opensearch/security/auditlog/sink/SinkProvider.java#L61-L66

If it's not received a configuration, it shows the error. IMO, this message should be INFO or, at least, WARNING, because audit default sink can't be created because no configuration is defined, and there is no error in the component.

Thanks, @teddytpc1 for researching, I proceed to close this issue. I leave it in your hands (optative) to open an issue in Opensearch requesting the change of message type.