[BUG] After upgrade to 4.9.0-1 (debian12) stuck with "Response Error" in log and "Wazuh dashboard server is not ready yet"

[DaLynxx]

Describe the bug

A clear and concise description of what the bug is.

To Reproduce I upgraded from 4.8 (4.8.2 I believe), following https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html

Steps to reproduce the behavior:

1. Upgrade to 4.9.0.
2. Try to reach the dashboard login screen via browser
3. Web page shows "Wazuh dashboard server is not ready yet"

Expected behavior

1. Dashboard login screen when browsing to the server.

OpenSearch Version

wazuh-indexer/now 4.9.0-1 amd64 [installed,local] wazuh-manager/now 4.9.0-1 amd64 [installed,local]

Dashboards Version

wazuh-dashboard/now 4.9.0-1 amd64 [installed,local]

Plugins

Please list all plugins currently enabled.

Not sure how I find out. I think I run more or less "out of the box" installation.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: debian 12, Linux 6.8.12-1-pve (LXC container on proxmox)
• Browser and version: Firefox 130.0

Additional context

I have updated before. Started with a 4.7.x version, got through a couple of 4.8.x steps. All worked well previously following the instructions.

wazuh-manager and wazuh-indexer and wazuh-dashboard all show "active (running)" when asking systemctl status .

However, wazuh-dashboard fills the "log" (journalctl) with

Sep 05 21:35:47 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:47Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:49 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:49Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:52 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:52Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:54 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:54Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:57 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:57Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}

Restarting wazuh-dashboard generates the following log-flow

Sep 05 21:37:14 wazuh systemd[1]: Started wazuh-dashboard.service - wazuh-dashboard.
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"dataSourceManagement\" has been disabled since the following >
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"applicationConfig\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"cspHandler\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"dataSource\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"visTypeXy\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-system"],"pid":3109,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageColl>
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","savedobjects-service"],"pid":3109,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch>
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["error","savedobjects-service"],"pid":3109,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 05 21:37:19 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:19Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:21 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:21Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:24 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:24Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}

Curl to the dashboard does not respond either.

Content of opensearch_dashboards.yml

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://127.0.0.1:9200
opensearch.ssl.verificationMode: certificate
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home
opensearch_security.cookie.secure: true

netstat

root@wazuh:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:1515            0.0.0.0:*               LISTEN      603/wazuh-authd
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      814/wazuh-remoted
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3491/node
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      418/master
tcp        0      0 0.0.0.0:55000           0.0.0.0:*               LISTEN      555/python3
tcp6       0      0 :::22                   :::*                    LISTEN      1/init
tcp6       0      0 ::1:25                  :::*                    LISTEN      418/master
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      164/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      164/java
udp        0      0 0.0.0.0:68              0.0.0.0:*                           89/dhclient

[DaLynxx]

Hm, just added the netstat output. Weird. Why is tcp6 listed for the 127.0.0.1:9200 and 9300

[DaLynxx]

9200 and 9300 is tied to the indexer rather than the dashboard, or?

[kslintubs]

Did the update too on Ubuntu 24.04.1 LTS , same error message

[kullarkert]

I'm also having exact same issue and same messages.

I have Ubuntu 22.04.4 LTS and did apt-get upgrade

Some log files

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

{"date":"2024-09-05T12:00:19.865Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.881Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.895Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.910Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

[2024-09-06T07:19:59,574][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,576][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37318}], closing connection
[2024-09-06T07:19:59,581][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,583][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37326}], closing connection
[2024-09-06T07:19:59,588][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,590][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37340}], closing connection
[2024-09-06T07:20:02,069][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,072][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37352}], closing connection
[2024-09-06T07:20:02,077][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,078][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37362}], closing connection
[2024-09-06T07:20:02,085][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,087][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37372}], closing connection
[2024-09-06T07:20:02,092][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,094][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37380}], closing connection
[2024-09-06T07:20:02,499][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

[BooopLJ]

I also have the same problem after updating to version 4.9.0 on Ubuntu 22.04.4 LTS

[kullarkert]

This command did the trick. Got answer from slack channel.

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<Secr3tP4ssw*rd>'

In all services in one server deployment, changing the password should solve the issue.

[tkolaski]

in wazuh-cluster.log I had entries:

Authentication finally failed for kibanaserver from 127.0.0.1:51086

I found:

https://groups.google.com/g/wazuh/c/rdCF0MBR6oU

It helped

[BooopLJ]

@kullarkert thank you for suggesting a solution, but I still have this error. The server was reset after the password was changed. opensearch-dashboards[5609]: {"type":"log","@timestamp":"2024-09-06T08:03:00Z","tags":["error","opensearch","data"],"pid":5609,"message":"[ResponseError]: Response Error"}

[PeterKnotek]

Hi

have problem after update

service wazuh-dashboard status
● wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-09-06 17:07:15 CEST; 13min ago
   Main PID: 14160 (node)
      Tasks: 11 (limit: 9371)
     Memory: 166.8M
        CPU: 18.727s
     CGroup: /system.slice/wazuh-dashboard.service
             └─14160 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml

Sep 06 17:20:04 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:04Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:06 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:06Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:09 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:09Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:11 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:11Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:14 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:14Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:16 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:16Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:19 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:19Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:21 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:21Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:24 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:24Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:26 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:26Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}

Can help me

Thanks

[asteriscos]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.
2. Backup your current keystore: cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak
3. Backup the keystore from the new location: cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak
4. Move the keystore to the correct location: mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore
5. Restart the Wazuh dashboard.

[borkedporcupine]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.

2. Backup your current keystore:
   `cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak`

3. Backup the keystore from the new location:
   `cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak`

4. Move the keystore to the correct location:
   `mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore`

5. Restart the Wazuh dashboard.

I can confirm that I was able to upgrade to 4.9.0 now without issue. Thanks for the fix.

[stefardi]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs. To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm Wazuh 4.x repositories are available again. Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.

2. Backup your current keystore:
   `cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak`

3. Backup the keystore from the new location:
   `cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak`

4. Move the keystore to the correct location:
   `mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore`

5. Restart the Wazuh dashboard.

I can confirm that I was able to upgrade to 4.9.0 now without issue. Thanks for the fix.

this is working for me. Thank you.

[genseirin]

No luck here. The error appeared after upgrading wazuh-dashboard from 4.8.2-1 to 4.9.0-2 and wazuh-indexer and wazuh-manager from 4.8.2-1 to 4.9.0-1. I stopped and started the three services in the recommended order and tried wazuh-passwords-tool.sh (https://github.com/wazuh/wazuh-dashboard/issues/292#issuecomment-2333424200), which, if I understand it correctly, should also sync the password at all other locations (being an all-in-one installation).

[asteriscos]

@genseirin can you please provide the output of these commands:

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn" curl -k -u '<USER>:<PASSWORD>' https://127.0.0.1:9200/_cluster/health?pretty

lsof -i -P -n | grep LISTEN | grep wazuh-indexer

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

ls -la /usr/share/wazuh-dashboard/config/
ls -la /etc/wazuh-dashboard/

cat /etc/default/wazuh-dashboard

[Dara-cy]

Hi @asteriscos I have a same issue can you help me out

journalctl -u wazuh-indexer | grep -iE "err|warn"

Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 10:23:39 WAZUH systemd-entrypoint[1201]: WARNING: COMPAT locale provider will be removed in a future release Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: COMPAT locale provider will be removed in a future release Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u '<USER>:<PASSWORD>' https://127.0.0.1:9200/_cluster/health?pretty

{ "cluster_name" : "wazuh-cluster", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 500, "active_shards" : 500, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 35, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 93.45794392523365

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 13 11:58:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T04:58:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:00:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:00:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:02:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:02:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:04:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:04:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:06:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:06:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:08:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:08:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:10:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:10:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:12:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:12:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:14:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:14:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:16:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:16:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:18:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:18:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:20:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:20:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:22:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:22:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:24:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:24:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:26:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:26:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:28:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:28:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:30:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:30:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:32:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:32:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:34:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:34:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}

[root@DCPAWAZUH ossec]# ls -la /usr/share/wazuh-dashboard/config/ total 12 drwxr-x---. 2 wazuh-dashboard wazuh-dashboard 95 Sep 13 11:52 . drwxr-x---. 9 wazuh-dashboard wazuh-dashboard 191 Sep 12 13:35 .. -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:25 node.options -rw-r-----. 1 root root 634 Sep 13 11:52 opensearch_dashboards.yml -rw-r-----. 1 root root 634 Sep 13 10:36 opensearch_dashboards.yml.bk

ls -la /etc/wazuh-dashboard/

drwxr-x---. 3 wazuh-dashboard wazuh-dashboard 4096 Sep 13 10:37 . drwxr-xr-x. 115 root root 8192 Sep 13 10:23 .. dr-x------. 2 wazuh-dashboard wazuh-dashboard 111 Apr 25 16:49 certs -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:28 node.options -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 254 Sep 13 11:37 opensearch_dashboards.keystore -rw-r-----. 1 root root 254 Sep 12 17:18 opensearch_dashboards.keystore.bak -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 635 Sep 13 11:49 opensearch_dashboards.yml -rw-r-----. 1 root root 635 May 28 16:36 opensearch_dashboards.yml.bk2 -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 634 Sep 7 03:28 opensearch_dashboards.yml.rpmnew

[root@DCPAWAZUH ossec]# cat /etc/default/wazuh-dashboard user="wazuh-dashboard" group="wazuh-dashboard" chroot="/" chdir="/" nice="" KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

[genseirin]

@asteriscos Thank you!

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"

...
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 07:29:41 ns3103184 systemd-entrypoint[14041]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u 'kibana:<corresponding pw>' https://127.0.0.1:9200/_cluster/health?pretty

no output

lsof -i -P -n | grep LISTEN | grep wazuh-indexer

java      14041               wazuh-indexer  615u  IPv6 531705783      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      14041               wazuh-indexer  617u  IPv6 531710409      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["error","opensearch","data"],"pid":19513,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["error","opensearch","data"],"pid":20813,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","savedobjects-service"],"pid":13304,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 13 07:29:25 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:25Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:27 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:27Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:30 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:30Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:32 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:32Z","tags":["warning","savedobjects-service"],"pid":13304,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["warning","savedobjects-service"],"pid":15518,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 13 07:30:43 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:43Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:45 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:45Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:48 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:48Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
(keeps repeating)

ls -la /usr/share/wazuh-dashboard/config/

drwxr-x---  2 wazuh-dashboard wazuh-dashboard 4096 Sep 11 13:59 .
drwxr-x--- 10 wazuh-dashboard wazuh-dashboard 4096 Sep 11 14:02 ..
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml

ls -la /etc/wazuh-dashboard/

drwxr-x---   3 wazuh-dashboard wazuh-dashboard 4096 Sep 12 07:53 .
drwxr-xr-x 119 root            root            4096 Sep 10 06:32 ..
dr-x------   2 wazuh-dashboard wazuh-dashboard 4096 Mar 18 10:37 certs
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 13 07:29 opensearch_dashboards.keystore
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 11 14:43 opensearch_dashboards.keystore.bak
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  697 Jun 14 06:30 opensearch_dashboards.yml
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml.dpkg-dist

cat /etc/default/wazuh-dashboard

user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

[asteriscos]

@asteriscos Thank you!

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"

...
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 07:29:41 ns3103184 systemd-entrypoint[14041]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u 'kibana:<corresponding pw>' https://127.0.0.1:9200/_cluster/health?pretty

no output

lsof -i -P -n | grep LISTEN | grep wazuh-indexer

java      14041               wazuh-indexer  615u  IPv6 531705783      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      14041               wazuh-indexer  617u  IPv6 531710409      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["error","opensearch","data"],"pid":19513,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["error","opensearch","data"],"pid":20813,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","savedobjects-service"],"pid":13304,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 13 07:29:25 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:25Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:27 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:27Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:30 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:30Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:32 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:32Z","tags":["warning","savedobjects-service"],"pid":13304,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["warning","savedobjects-service"],"pid":15518,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 13 07:30:43 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:43Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:45 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:45Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:48 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:48Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
(keeps repeating)

ls -la /usr/share/wazuh-dashboard/config/

drwxr-x---  2 wazuh-dashboard wazuh-dashboard 4096 Sep 11 13:59 .
drwxr-x--- 10 wazuh-dashboard wazuh-dashboard 4096 Sep 11 14:02 ..
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml

ls -la /etc/wazuh-dashboard/

drwxr-x---   3 wazuh-dashboard wazuh-dashboard 4096 Sep 12 07:53 .
drwxr-xr-x 119 root            root            4096 Sep 10 06:32 ..
dr-x------   2 wazuh-dashboard wazuh-dashboard 4096 Mar 18 10:37 certs
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 13 07:29 opensearch_dashboards.keystore
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 11 14:43 opensearch_dashboards.keystore.bak
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  697 Jun 14 06:30 opensearch_dashboards.yml
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml.dpkg-dist

cat /etc/default/wazuh-dashboard

user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@genseirin I see a couple of problems in the logs:

Credentials

If the curl to https://127.0.0.1:9200/_cluster/health?pretty doesn't have an output, it usually means a credentials problem. Did you try with the admin user?

If you don't remember the credentials you can reset the password using this tool: Download the tool curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.9/wazuh-passwords-tool.sh

Execute it bash wazuh-passwords-tool.sh -a

https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

kibana_3 index

I can see in the logs you have an issue with .kibana_3 index, so I suggest to refresh the index and restart Wazuh dashboard afterward. curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 systemctl restart wazuh-dashboard

[asteriscos]

Hi @asteriscos I have a same issue can you help me out

journalctl -u wazuh-indexer | grep -iE "err|warn"

Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 10:23:39 WAZUH systemd-entrypoint[1201]: WARNING: COMPAT locale provider will be removed in a future release Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: COMPAT locale provider will be removed in a future release Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u ':' https://127.0.0.1:9200/_cluster/health?pretty

{ "cluster_name" : "wazuh-cluster", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 500, "active_shards" : 500, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 35, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 93.45794392523365

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 13 11:58:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T04:58:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:00:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:00:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:02:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:02:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:04:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:04:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:06:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:06:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:08:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:08:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:10:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:10:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:12:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:12:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:14:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:14:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:16:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:16:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:18:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:18:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:20:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:20:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:22:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:22:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:24:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:24:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:26:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:26:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:28:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:28:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:30:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:30:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:32:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:32:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:34:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:34:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}

[root@DCPAWAZUH ossec]# ls -la /usr/share/wazuh-dashboard/config/ total 12 drwxr-x---. 2 wazuh-dashboard wazuh-dashboard 95 Sep 13 11:52 . drwxr-x---. 9 wazuh-dashboard wazuh-dashboard 191 Sep 12 13:35 .. -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:25 node.options -rw-r-----. 1 root root 634 Sep 13 11:52 opensearch_dashboards.yml -rw-r-----. 1 root root 634 Sep 13 10:36 opensearch_dashboards.yml.bk

ls -la /etc/wazuh-dashboard/

drwxr-x---. 3 wazuh-dashboard wazuh-dashboard 4096 Sep 13 10:37 . drwxr-xr-x. 115 root root 8192 Sep 13 10:23 .. dr-x------. 2 wazuh-dashboard wazuh-dashboard 111 Apr 25 16:49 certs -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:28 node.options -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 254 Sep 13 11:37 opensearch_dashboards.keystore -rw-r-----. 1 root root 254 Sep 12 17:18 opensearch_dashboards.keystore.bak -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 635 Sep 13 11:49 opensearch_dashboards.yml -rw-r-----. 1 root root 635 May 28 16:36 opensearch_dashboards.yml.bk2 -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 634 Sep 7 03:28 opensearch_dashboards.yml.rpmnew

[root@DCPAWAZUH ossec]# cat /etc/default/wazuh-dashboard user="wazuh-dashboard" group="wazuh-dashboard" chroot="/" chdir="/" nice="" KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@Dara-cy what type of deployment do you have? is it an all-in-one installation or a distributed one? I see Wazuh Dashboard trying to reach Wazuh indexer, but the error doesn't seem to be related to credentials. Can you confirm Wazuh dashboard can reach the address of Wazuh Indexer?

[metalcated]

Was having these issues too, resetting all passwords and updating nodes did the trick for all but one warning message which I am investigating outside of this thread. Thanks!

[uguronduc]

@asteriscos Thank you!

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"

...
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 07:29:41 ns3103184 systemd-entrypoint[14041]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u 'kibana:<corresponding pw>' https://127.0.0.1:9200/_cluster/health?pretty no output lsof -i -P -n | grep LISTEN | grep wazuh-indexer

java      14041               wazuh-indexer  615u  IPv6 531705783      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      14041               wazuh-indexer  617u  IPv6 531710409      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["error","opensearch","data"],"pid":19513,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["error","opensearch","data"],"pid":20813,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","savedobjects-service"],"pid":13304,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 13 07:29:25 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:25Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:27 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:27Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:30 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:30Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:32 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:32Z","tags":["warning","savedobjects-service"],"pid":13304,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["warning","savedobjects-service"],"pid":15518,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 13 07:30:43 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:43Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:45 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:45Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:48 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:48Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
(keeps repeating)

ls -la /usr/share/wazuh-dashboard/config/

drwxr-x---  2 wazuh-dashboard wazuh-dashboard 4096 Sep 11 13:59 .
drwxr-x--- 10 wazuh-dashboard wazuh-dashboard 4096 Sep 11 14:02 ..
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml

ls -la /etc/wazuh-dashboard/

drwxr-x---   3 wazuh-dashboard wazuh-dashboard 4096 Sep 12 07:53 .
drwxr-xr-x 119 root            root            4096 Sep 10 06:32 ..
dr-x------   2 wazuh-dashboard wazuh-dashboard 4096 Mar 18 10:37 certs
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 13 07:29 opensearch_dashboards.keystore
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 11 14:43 opensearch_dashboards.keystore.bak
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  697 Jun 14 06:30 opensearch_dashboards.yml
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml.dpkg-dist

cat /etc/default/wazuh-dashboard

user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@genseirin I see a couple of problems in the logs:

Credentials

If the curl to https://127.0.0.1:9200/_cluster/health?pretty doesn't have an output, it usually means a credentials problem. Did you try with the admin user?

If you don't remember the credentials you can reset the password using this tool: Download the tool curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.9/wazuh-passwords-tool.sh

Execute it bash wazuh-passwords-tool.sh -a

https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

kibana_3 index

I can see in the logs you have an issue with .kibana_3 index, so I suggest to refresh the index and restart Wazuh dashboard afterward. curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 systemctl restart wazuh-dashboard

Thanks for solution. I solved with curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3` systemctl restart wazuh-dashboard

[Dara-cy]

Hi @asteriscos

Actually it is an all in one installation, but have 2 ip interface that 10.84.54.51 ( for dashboard) and 10.84.55.5 ( for indexer and rsyslog server). It's running well before i upgrade. Now agent, rsyslog working and dashboard is not ready yet.

Dashboard config

server.host: 0.0.0.0 server.port: 443 #opensearch.hosts: https://10.84.54.51:9200 opensearch.hosts: https://127.0.0.1:9200 #opensearch.hosts: https://10.84.55.5:9200

opensearch.ssl.verificationMode: certificate #opensearch.username: #opensearch.password: opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"] opensearch_security.multitenancy.enabled: false opensearch_security.readonly_mode.roles: ["kibana_read_only"] server.ssl.enabled: true server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem" server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem" opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"] uiSettings.overrides.defaultRoute: /app/wz-home

Curl check

curl -k -u 'admin:admin' https://127.0.1:9200/_cluster/health?pretty { "cluster_name" : "wazuh-cluster", "status" : "yellow", "timed_out" : false,V "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 505, "active_shards" : 505, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 34, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 93.69202226345084 }

wazuh-dashboard status

wazuh-dashboard.service - wazuh-dashboard Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled) Active: active (running) since Sun 2024-09-15 16:17:41 +07; 10min ago Main PID: 96251 (node) Tasks: 11 (limit: 407812) Memory: 176.2M CPU: 6.996s CGroup: /system.slice/wazuh-dashboard.service └─96251 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist

journalctl -u wazuh-dashboard | grep -iE "err|warn"

(restart dashboard and index service)

Sep 15 16:16:28 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:28Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:30 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:30Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:33 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:33Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:35 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:35Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:38 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:38Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:40 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:40Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:43 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:43Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:45 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:45Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:48 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:48Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:50 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:50Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"} Sep 15 16:16:53 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:53Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"} Sep 15 16:16:53 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:53Z","tags":["warning","savedobjects-service"],"pid":91828,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "} Sep 15 16:16:56 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:56Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"} Sep 15 16:16:58 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:58Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"} Sep 15 16:17:01 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:01Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"} Sep 15 16:17:03 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:03Z","tags":["error","opensearch","data"],"pid":91828,"message":"[resource_already_exists_exception]: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"} Sep 15 16:17:03 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:03Z","tags":["warning","savedobjects-service"],"pid":91828,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"} Sep 15 16:17:03 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:03Z","tags":["warning","savedobjects-service"],"pid":91828,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."} Sep 15 16:17:44 DCPAWAZUH01 opensearch-dashboards[96251]: {"type":"log","@timestamp":"2024-09-15T09:17:44Z","tags":["error","opensearch","data"],"pid":96251,"message":"[resource_already_exists_exception]: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"} Sep 15 16:17:44 DCPAWAZUH01 opensearch-dashboards[96251]: {"type":"log","@timestamp":"2024-09-15T09:17:44Z","tags":["warning","savedobjects-service"],"pid":96251,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"} Sep 15 16:17:44 DCPAWAZUH01 opensearch-dashboards[96251]: {"type":"log","@timestamp":"2024-09-15T09:17:44Z","tags":["warning","savedobjects-service"],"pid":96251,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}

Thank you again help.

[Dara-cy]

Hi @asteriscos

Run this fix my dashboard

''' curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 '''

But wazuh agent has connection error, I will check version or reinstall agent and notify everyone soon

Many thanks

[genseirin]

@genseirin I see a couple of problems in the logs:

Credentials

If the curl to https://127.0.0.1:9200/_cluster/health?pretty doesn't have an output, it usually means a credentials problem. Did you try with the admin user?

If you don't remember the credentials you can reset the password using this tool: Download the tool curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.9/wazuh-passwords-tool.sh

Execute it bash wazuh-passwords-tool.sh -a

https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

kibana_3 index

I can see in the logs you have an issue with .kibana_3 index, so I suggest to refresh the index and restart Wazuh dashboard afterward. curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 systemctl restart wazuh-dashboard

Your solution was very helpful, thanks a lot!

[BooopLJ]

Hi @asteriscos thank you for your solution. It worked for me:

"1. Install the new Wazuh dashboard revision 2 packages. 2. Backup your current keystore: cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak 3. Backup the keystore from the new location: cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak 4. Move the keystore to the correct location: mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore 5. curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 6. systemctl restart wazuh-dashboard"

Have a nice day :)

[MAH69IK]

Hi, I'm facing this problem too, but I'm not able to solve it yet. I updated Wazuh from 4.7.5 to 4.8.2 and then to 4.9.0. Now I am getting the error “Wazuh dashboard server is not ready yet”.

Here is what I tried:

• I don't have the file /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore, so the advice from @asteriscos didn't work (probably because I put 4.9.0-2 right away, without 4.9.0-1).
• I tried the command $ sudo /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add opensearch.password specifying the password from the file /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml - didn't help.
• I tried the command $ sudo /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver - that didn't help either.
• I restarted both wazuh-indexer and wazuh-dashboard after both commands.

Here is the information about my system:

Wazuh indexer

$ sudo journalctl --since today -u wazuh-indexer | grep -iE "err|warn"
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager will be removed in a future release
сен 17 14:38:22 wazuh systemd-entrypoint[3180279]: WARNING: COMPAT locale provider will be removed in a future release
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager will be removed in a future release
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager will be removed in a future release
сен 17 15:11:33 wazuh systemd-entrypoint[3181519]: WARNING: COMPAT locale provider will be removed in a future release
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager will be removed in a future release

$ curl -k -u admin:... https://localhost:9200/_cat/health?pretty
1726579263 13:21:03 wazuh-cluster green 1 1 true 447 447 0 0 0 0 - 100.0%

$ sudo lsof -i -nP | grep LISTEN | grep wazuh-indexer
java      3181519   wazuh-indexer  576u  IPv6 3837239206      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      3181519   wazuh-indexer  578u  IPv6 3837239246      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

$ sudo journalctl --since today -u wazuh-dashboard | grep -iE "err|warn"
сен 17 15:15:15 wazuh opensearch-dashboards[3181832]: {"type":"log","@timestamp":"2024-09-17T12:15:15Z","tags":["error","opensearch","data"],"pid":3181832,"message":"[TimeoutError]: Request timed out"}
сен 17 15:15:15 wazuh opensearch-dashboards[3181832]: {"type":"log","@timestamp":"2024-09-17T12:15:15Z","tags":["error","savedobjects-service"],"pid":3181832,"message":"Unable to retrieve version information from OpenSearch nodes."}
сен 17 15:17:15 wazuh opensearch-dashboards[3181832]: {"type":"log","@timestamp":"2024-09-17T12:17:15Z","tags":["error","opensearch","data"],"pid":3181832,"message":"[TimeoutError]: Request timed out"}

(then the last line is repeated)

$ sudo ls -lhFAv /usr/share/wazuh-dashboard/config/
total 8.0K
-rw-r----- 1 wazuh-dashboard wazuh-dashboard 312 мая  5  2023 node.options
-rw-r----- 1 wazuh-dashboard wazuh-dashboard 634 мая  5  2023 opensearch_dashboards.yml

$ sudo ls -lhFAv /etc/wazuh-dashboard/
total 20K
dr-x------ 2 wazuh-dashboard wazuh-dashboard 4.0K июн 17  2022 certs/
-rw-r----- 1 wazuh-dashboard wazuh-dashboard  312 мая  5  2023 node.options
-rw-r--r-- 1 wazuh-dashboard wazuh-dashboard  254 сен 17 15:04 opensearch_dashboards.keystore
-rw-r----- 1 wazuh-dashboard wazuh-dashboard  634 сен 16 17:16 opensearch_dashboards.yml
-rw-r----- 1 wazuh-dashboard wazuh-dashboard  713 авг  4  2023 opensearch_dashboards.yml.dpkg-old

$ sudo cat /etc/default/wazuh-dashboard
user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

$ sudo systemctl restart wazuh-dashboard.service && sudo journalctl --since "$(date +"%F %T")" -fu wazuh-dashboard.service
-- Journal begins at Fri 2024-08-30 18:45:40 EEST. --
сен 17 16:39:32 wazuh systemd[1]: Stopping wazuh-dashboard...
сен 17 16:39:32 wazuh opensearch-dashboards[3183161]: {"type":"log","@timestamp":"2024-09-17T13:39:32Z","tags":["info","plugins-system"],"pid":3183161,"message":"Stopping all plugins."}
сен 17 16:39:32 wazuh opensearch-dashboards[3183161]: {"type":"log","@timestamp":"2024-09-17T13:39:32Z","tags":["info","savedobjects-service"],"pid":3183161,"message":"Starting saved objects migrations"}
сен 17 16:39:32 wazuh systemd[1]: wazuh-dashboard.service: Succeeded.
сен 17 16:39:32 wazuh systemd[1]: Stopped wazuh-dashboard.
сен 17 16:39:32 wazuh systemd[1]: wazuh-dashboard.service: Consumed 9.527s CPU time.
сен 17 16:39:32 wazuh systemd[1]: Started wazuh-dashboard.
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"applicationConfig\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"cspHandler\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"dataSource\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"visTypeXy\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-system"],"pid":3183190,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,embeddable,legacyExport,expressions,data,savedObjects,home,apmOss,reportsDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,visTypeMarkdown,visTypeTable,visBuilder,visAugmenter,alertingDashboards,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,indexManagementDashboards,notificationsDashboards,management,indexPatternManagement,advancedSettings,console,dataExplorer,bfetch,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,securityDashboards,wazuhCore,wazuhCheckUpdates,wazuh]"}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:41 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:41Z","tags":["info","savedobjects-service"],"pid":3183190,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
сен 17 16:41:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:41:41Z","tags":["error","opensearch","data"],"pid":3183190,"message":"[TimeoutError]: Request timed out"}
сен 17 16:41:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:41:41Z","tags":["error","savedobjects-service"],"pid":3183190,"message":"Unable to retrieve version information from OpenSearch nodes."}
сен 17 16:43:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:43:41Z","tags":["error","opensearch","data"],"pid":3183190,"message":"[TimeoutError]: Request timed out"}

[gdiazlo]

@MAH69IK your problems seems to indicate a communication problem between dahsboard and indexer.

On your message you said you changed the password in the keystore to be like the one stored in wazuh.yml but those files are not related, and do not require having the same password at all.

To fix your deployment, I would follow the changing password documentation from https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

[BooopLJ]

Hi @asteriscos thank you for your solution. It worked for me:

"1. Install the new Wazuh dashboard revision 2 packages. 2. Backup your current keystore: cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak 3. Backup the keystore from the new location: cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak 4. Move the keystore to the correct location: mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore 5. curl -k -XDELETE -u : https://:9200/.kibana_3 6. systemctl restart wazuh-dashboard"

Have a nice day :)

Hi, after resetting the server, the problem with the message returned "Wazuh dashboard server is not ready yet". Do you have any idea where the problem is?

My Output: curl -k -u 'user:pass' https://IP:9200/_cluster/health?pretty { "cluster_name" : "wazuh-indexer-cluster", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 999, "active_shards" : 999, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

sudo cat /etc/default/wazuh-dashboard user="wazuh-dashboard" group="wazuh-dashboard" chroot="/" chdir="/" nice="" KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

sudo systemctl restart wazuh-dashboard.service && sudo journalctl --since "$(date +"%F %T")" -fu wazuh-dashboard.service wrz 26 12:16:49 serversiem systemd[1]: wazuh-dashboard.service: Deactivated successfully. wrz 26 12:16:49 serversiem systemd[1]: Stopped wazuh-dashboard. wrz 26 12:16:49 serversiem systemd[1]: wazuh-dashboard.service: Consumed 27.778s CPU time. wrz 26 12:16:49 serversiem systemd[1]: Started wazuh-dashboard. wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "dataSourceManagement" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"} wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "applicationConfig" is disabled."} wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "cspHandler" is disabled."} wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "dataSource" is disabled."} wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "visTypeXy" is disabled."} wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-system"],"pid":33310,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,savedObjects,home,apmOss,reportsDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,visTypeTable,visTypeMarkdown,visBuilder,visAugmenter,alertingDashboards,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,indexManagementDashboards,notificationsDashboards,management,indexPatternManagement,advancedSettings,console,dataExplorer,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,securityDashboards,wazuhCore,wazuhCheckUpdates,wazuh,bfetch]"} wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:02Z","tags":["info","savedobjects-service"],"pid":33310,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."} wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["info","savedobjects-service"],"pid":33310,"message":"Starting saved objects migrations"} wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["info","savedobjects-service"],"pid":33310,"message":"Creating index .kibana_1."} wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["error","opensearch","data"],"pid":33310,"message":"[resource_already_exists_exception]: index [.kibana_1/9mbl3wdpQmyx7ZmEhdSW9w] already exists"} wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["warning","savedobjects-service"],"pid":33310,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_1/9mbl3wdpQmyx7ZmEhdSW9w] already exists"} wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["warning","savedobjects-service"],"pid":33310,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting OpenSearchDashboards."}

sudo systemctl status wazuh-indexer wrz 26 11:28:10 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) wrz 26 11:28:10 serversiem systemd-entrypoint[20515]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch wrz 26 11:28:10 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager will be removed in a future release wrz 26 11:28:11 serversiem systemd-entrypoint[20515]: wrz 26, 2024 11:28:11 AM sun.util.locale.provider.LocaleProviderAdapter wrz 26 11:28:11 serversiem systemd-entrypoint[20515]: WARNING: COMPAT locale provider will be removed in a future release wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: A terminally deprecated method in java.lang.System has been called wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager will be removed in a future release wrz 26 11:28:30 serversiem systemd[1]: Started wazuh-indexer.

sudo lsof -i -nP | grep LISTEN | grep wazuh-indexer java 20515 wazuh-indexer 607u IPv6 230815 0t0 TCP 192.168.254.6:9300 (LISTEN) java 20515 wazuh-indexer 609u IPv6 230823 0t0 TCP 192.168.254.6:9200 (LISTEN)

[kwiha]

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

where are these packages?

[satheeshkv72]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.
2. Backup your current keystore: cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak
3. Backup the keystore from the new location: cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak
4. Move the keystore to the correct location: mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore
5. Restart the Wazuh dashboard.

Hi @asteriscos I'm facing the same issue and running with dashboard version 4.9.2-1. Can I follow these steps for resolution?