[Alerts index pattern] 4002 - Could not retrieve templates from Elasticsearch due to Response Error

[jesse-zhangh]

deployment envs: K8S version: v4.3.10 When the K8S nodes changed(add a new node and delete an old node), the wauzh have an error: INFO: Index pattern id in cookie: yes [wazuh-alerts-] INFO: Getting list of valid index patterns... INFO: Valid index patterns found: 1 INFO: Found default index pattern with title [wazuh-alerts-]: yes INFO: Checking the app default pattern exists: id [wazuh-alerts-]... INFO: Default pattern with id [wazuh-alerts-] exists: yes ACTION: Default pattern id [wazuh-alerts-] set as default index pattern INFO: Checking the index pattern id [wazuh-alerts-] exists... INFO: Index pattern id exists [wazuh-alerts-]: yes INFO: Index pattern id in cookie: yes [wazuh-alerts-] INFO: Checking if the index pattern id [wazuh-alerts-] exists... INFO: Index pattern id [wazuh-alerts-] found: yes title [wazuh-alerts-] INFO: Checking if exists a template compatible with the index pattern title [wazuh-alerts-] ERROR: 4002 - Could not retrieve templates from Elasticsearch due to Response Error

[Alerts index pattern] 4002 - Could not retrieve templates from Elasticsearch due to Response Error

Check1: curl -k -u uuuu:pppp https://10.x.x.x:9200/_cluster/health?pretty { "cluster_name" : "wazuh", "status" : "red", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "discovered_master" : true, "active_primary_shards" : 11, "active_shards" : 22, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 91.66666666666666 }

Check2: curl -k -u uuuu:pppp https://10.x.x.x:9200/_cat/indices OpenSearch Security not initialized.

Check3: curl -k -u uuuu:pppp https://10.x.x.x:9200/_cat/indices {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:monitor/settings/get] and User [name=kibanaserver, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:monitor/settings/get] and User [name=kibanaserver, backend_roles=[], requestedTenant=null]"},"status":403}

note: sometimes it shows check2 result, sometimes check3.

try solve1: curl https://raw.githubusercontent.com/wazuh/wazuh/v4.3.10/extensions/elasticsearch/7.x/wazuh-template.json | curl -X PUT "https://localhost:9200/_template/wazuh" -H 'Content-Type: application/json' -d @- -u <elasticsearch_user>:<elasticsearch_password> -k

Still not working fine.

How can I fix this exception？ thanks

[jesse-zhangh]

some logs in /var/log/wazuh-indexer/wazuh.log of pod wazuh-indexer-0: [2022-12-29T07:40:23,473][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:23,474][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:23,475][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:23,476][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:25,974][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:25,975][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:25,976][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:25,977][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:27,194][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:28,474][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:28,475][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:28,477][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:28,478][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:30,973][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:30,974][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:30,975][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:30,976][ERROR][o.o.s.a.BackendRegistry ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin) [2022-12-29T07:40:30,995][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [wazuh-indexer-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security) org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:202) ~[opensearch-1.2.4.jar:1.2.4] at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:188) ~[opensearch-1.2.4.jar:1.2.4] at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:76) ~[opensearch-1.2.4.jar:1.2.4] at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.2.4.jar:1.2.4] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.2.4.0.jar:1.2.4.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.2.4.0.jar:1.2.4.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:554) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.2.4.0.jar:1.2.4.0] at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.2.4.0.jar:1.2.4.0] at java.lang.Thread.run(Thread.java:832) [?:?] [2022-12-29T07:40:32,303][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] master not discovered or elected yet, an election requires at least 2 nodes with ids from [oR297Sl7RwOK_i9UXvPJiw, qOp4UeJkSba5wIbTG7nobw, jZqcVHfBRC-9TQPOKvzfjA], have discovered [{wazuh-indexer-0}{oR297Sl7RwOK_i9UXvPJiw}{1bB3X9StQVWGCg1vvqIARQ}{10.244.7.5}{10.244.7.5:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{oR297Sl7RwOK_i9UXvPJiw}{1bB3X9StQVWGCg1vvqIARQ}{10.244.7.5}{10.244.7.5:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 1, last-accepted version 130 in term 1

[vcerenu]

Hello @jesse-zhangh

According to the error that you are presenting, the node has apparently lost the previous data since when restarting a pod container, it should mount all the information that the previous one used, but in your case it seems that it is lost and a deployment is generated from scratch.

Can you verify what retention policy you have configured in the PVCs associated with the Wazuh indexer pods and if in case of being in retain, is the same PVC being mounted again or if a new one is created?

Also please send a describe of the Wazuh indexer pods to know what configuration they currently have.

[mourya-satyam-888]

@vcerenu I am having the issue that when I navigate to /usr/share/wazuh-indexer/data folder, no data is being saved there hence when I restart the pod it erases all the existing indices in opensearch, I am using wazuh indexer version 4.4.4.

can you help with this using this default configs only nothing changed from the repo