[BUG] OpenSearch Dashboards server is not ready yet

[stcoats]

I am following the instructions at https://opensearch.org/docs/latest/quickstart/, but can't get the Dashboards component working.

I installed Docker, Docker compose, and the plugins. I downloaded the docker compose yaml file from Opensearch. I ran docker compose.

Opensearch is up and running, but the dashboard cannot be opened. The message on http://localhost:5601/ is "OpenSearch Dashboards server is not ready yet".

The relevant log lines are:

{"type":"log","@timestamp":"2023-07-25T09:56:38Z","tags":["error","opensearch","data"],"pid":1,"message":"[resource_already_exists_exception]: index [.kibana_1/SVgoRGdSRBaQXa-adE98Sg] already exists"}
{"type":"log","@timestamp":"2023-07-25T09:56:38Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_1/SVgoRGdSRBaQXa-adE98Sg] already exists"}
{"type":"log","@timestamp":"2023-07-25T09:56:38Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting OpenSearchDashboards."}

I have tried the approaches mentioned here and in StackOverflow or elsewhere:

curl -XDELETE http://localhost:9200/.kibana_1 and related commands have no effect.

Some have suggested restarting kibana, which confuses me: kibana is not installed, as far as I know.

Do I need to go into the docker container and delete or change something in there? If so, how do I do that?

If it's relevant, I had Elasticsearch installed on this machine, a VM, a few years ago. I removed it with yum. Maybe there are still some settings hiding deep in my machine. Where do I look for them and delete them?

OpenSearch Version 2.8.0, latest docker-compose version

Dashboards Version 2.8.0, latest docker-compose version

Plugins Whatever the docker yaml installed

Screenshots

• OS: Centos 7
• Browser and version Chrome/Firefox

Many thanks for any troubleshooting tips you can provide.

[kavilla]

Hello @stcoats,

Thanks for opening. Yes it seems like you see that the application has not started. Within OpenSearch Dashboards we utilize a system index (.kibana), which stores your visualizations, dashboards, index patterns, etc. So therefore I would recommend watching out for deleting the the index .kibana_# if you have any the previous mentioned items.

In this situation there can be a couple issues, there could be a greater version of the index available within your environment for example .kibana_2. Which would prevent the application from starting successfully if your alias for the system index is pointing to .kibana_1. Would you be able to do the following:

curl localhost:9200/_cat/aliases/*.kibana*?v

curl localhost:9200/_cat/indices/*.kibana*?v

Then let me know the results on if you have multiple .kibana_# indices and which index does the alias point to?

[stcoats]

Hi @kavilla,

Thanks for your response. The output of curl localhost:9200/_cat/aliases/*.kibana*?v is alias index filter routing.index routing.search is_write_index

The output of curl localhost:9200/_cat/indices/*.kibana*?v is health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

[kavilla]

Looks like the index no longer exists. Are you able to restart OpenSearch Dashboards and it successfully start up?

[stcoats]

I'm not sure how to restart OpenSearch Dashboards independently of the docker container.

I'm using docker compose up -d, as per the instructions in the "Quickstart". When I restart now, curl localhost:9200/_cat/indices/*.kibana*?v shows

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size red open .kibana_1 2LzPmsm1S5SZO9Xz0tHGJw 1 1

If I delete that with curl -XDELETE http://localhost:9200/.kibana_1, how do I restart just the Dashboard?

[kavilla]

So when OpenSearch Dashboards starts up it will attempt to recreate the .kibana index and migrate any docs from the previous index. If it doesn't exist it will recreate it. After restarting the container and with the new index created are you still experiencing issues? Or did you already deleted the index?

[stcoats]

Yes, stopping the container then restarting it just produces the same "OpenSearch Dashboards server is not ready yet" message at http://localhost:5601/, whether or not I delete the .kibana index.

[kavilla]

@stcoats, do you mind telling me if there is any logs available.

For example:

docker-compose logs opensearch-dashboards

[stcoats]

@kavilla, if the container is running, the output of docker compose logs opensearch-dashboards is

opensearch-dashboards  | Disabling OpenSearch Security Dashboards Plugin
opensearch-dashboards  | v16.20.0
opensearch-dashboards  | Removing securityDashboards...
opensearch-dashboards  | Plugin removal complete
opensearch-dashboards  | v16.20.0

I disabled security in the yaml file for testing purposes.

[pmc-a]

@stcoats @kavilla were you able to get this resolved? Running into the same issue when attempting to spin the dashboards container up.

I can see that the .kibana index exists within the OpenSearch cluster, but still hit the following:

2024-01-05 11:29:21 {"type":"log","@timestamp":"2024-01-05T11:29:21Z","tags":["error","opensearch","data"],"pid":1,"message":"[resource_already_exists_exception]: index [.kibana_1/FZ3Zn_AARMKCExe0YyULMQ] already exists"}
2024-01-05 11:29:21 {"type":"log","@timestamp":"2024-01-05T11:29:21Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_1/FZ3Zn_AARMKCExe0YyULMQ] already exists"}
2024-01-05 11:29:21 {"type":"log","@timestamp":"2024-01-05T11:29:21Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting OpenSearchDashboards."}

If I delete the index and restart the dashboards container, it begins creating the index but appears to timeout:

2024-01-05 11:35:58 {"type":"log","@timestamp":"2024-01-05T11:35:58Z","tags":["info","savedobjects-service"],"pid":1,"message":"Creating index .kibana_1."}
2024-01-05 11:36:28 {"type":"log","@timestamp":"2024-01-05T11:36:28Z","tags":["error","opensearch","data"],"pid":1,"message":"[TimeoutError]: Request timed out"}
2024-01-05 11:36:28 {"type":"log","@timestamp":"2024-01-05T11:36:28Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: Request timed out"}

But it actually does create the index:

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   .kibana_1 W74hZoWwToiTobVABGX8hA   1   1

And then the dashboards container just throws the same initial errors/logs again saying it already exists, another instance is migrating etc.

Any help would be greatly appreciated!

[stcoats]

@pmc-a no, I ended up using a different technology for my application. Sorry I can't help!

[navjotjsingh]

Is there any fix or solution? I am facing the same issue with OpenSearch and OpenSearch Dashboards 2.12

[quiet-ranger]

By looking at the logs, it appears that opensearch-dashboards cannot connect to the cluster. Could this be because of the introduction of a mandatory admin password? Anyway, I am speculating only, I have the same problem and I am also wondering if there was any progress on this.

You can see the logs (once everything is supposedly up and running) with:

$ docker-compose logs opensearch-dashboards

Here are the log entries that could be relevant:

opensearch-dashboards    | [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:25Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:25Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.19.0.2:9200"}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:25Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:28Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.19.0.2:9200"}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:30Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.19.0.2:9200"}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:33Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.19.0.2:9200"}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:35Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.19.0.2:9200"}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:38Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.19.0.2:9200"}
opensearch-dashboards    | {"type":"log","@timestamp":"2024-03-21T20:57:40Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: getaddrinfo ENOTFOUND opensearch-node1"}

[alien-keric]

i have been experiencing the same problem for the past three days so far i haven't get any solution about this

I'm not sure how to restart OpenSearch Dashboards independently of the docker container.

I'm using docker compose up -d, as per the instructions in the "Quickstart". When I restart now, curl localhost:9200/_cat/indices/*.kibana*?v shows

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size red open .kibana_1 2LzPmsm1S5SZO9Xz0tHGJw 1 1

If I delete that with curl -XDELETE http://localhost:9200/.kibana_1, how do I restart just the Dashboard?

did u manage to find a solution?. because i have been experiencig the same problem for past three days, when i try to configure opensearch i get the same

[TobiasKrimmer71]

I am experiencing the same issue sporadically when upgrading the minor version of opensearch. In my case, the root cause is that the alias ".kibana" still points to the old ".kibana_x" index. It seems that during upgrade the dashboards wants to clone the kibana index but then fails or aborts. So the index is there, empty and orphaned.

To check if you have the same issue, just do

curl -X GET http://localhost:9200/_cat/aliases/.kibana* --> Note down to which kibana index the alias points curl -X GET http://localhost:9200/_cat/indices/.kibana* --> Check whether there is a kibana index with higher number

It can be resolved by deleting the latest kibana_x index and restarting opensearch-dashboards. curl -X DELETE http://localhost:9200/.kibana_x (where x is the number you found above)

[JakeIn2l]

Hello,

We decided to go with a different product because of this issue. We encountered it on a fresh install following the getting started documents. Hope you folks can fix it!

[thautwarm]

When using docker-compose to start open search, you might get blocked silently if your OPENSEARCH_INITIAL_ADMIN_PASSWORD is not valid:

opensearch-node1  | Password <xxxxxxx> failed validation: "Password does not match validation regex". Please re-try with a minimum 8 character password and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character that is strong. Password strength can be tested here: https://lowe.github.io/tryzxcvbn

[vanshdhar999]

Is there any update on this issue? really want to use the opensearch-dashboard

[mustaphaAmzil]

I faced the same issue and managed to resolve it by making a few changes to the docker-compose.yml file:

1. Configured OpenSearch to run as a single-node instance.

2. Added the following environment variables to the OpenSearch Dashboards service:

   OPENSEARCH_USERNAME='admin'

   OPENSEARCH_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}

3. Ensured proper startup order by adding depends_on so that OpenSearch Dashboards waits for OpenSearch to be ready.

Here's a snippet of the relevant configuration:

services:
  opensearch:
    image: opensearchproject/opensearch:latest
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    restart: unless-stopped
    ports:
      - 9200:9200
      - 9600:9600
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    restart: unless-stopped
    ports:
      - 5600:5600
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
      OPENSEARCH_USERNAME: 'admin'
      OPENSEARCH_PASSWORD: ${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    depends_on:
      - opensearch
    networks:
      - opensearch-net

volumes:
  opensearch-data:

networks:
  opensearch-net:

Hope this helps others facing the same issue!

[mustaphaAmzil]

When using docker-compose to start open search, you might get blocked silently if your OPENSEARCH_INITIAL_ADMIN_PASSWORD is not valid:

opensearch-node1  | Password <xxxxxxx> failed validation: "Password does not match validation regex". Please re-try with a minimum 8 character password and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character that is strong. Password strength can be tested here: https://lowe.github.io/tryzxcvbn

What worked for me to handle this, was generating a password that meets the zxcvbn strength requirements. You can try using a tool like qit.tools Password Generator—it might help you create a strong enough password.