[BUG] Documented docker-compose.yml doesn't work

[hemna]

Using the sample docker-compose.yml file here: https://opensearch.org/docs/latest/opensearch/install/docker#sample-docker-compose-file results in a failed startup.

└─> docker --version
Docker version 20.10.14, build a224086

My docker-compose, which is copy/pasted from above documentation url

└─> cat docker-compose.yml
version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:1.3.1
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:1.3.1
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:1.3.0
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

I get several errors during docker-compose up

opensearch-node1         | [2022-04-25T13:35:27,342][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node1] PerformanceAnalyzer Enabled: false
opensearch-node1         | [2022-04-25T13:35:27,380][INFO ][o.o.n.Node               ] [opensearch-node1] initialized
opensearch-node1         | [2022-04-25T13:35:27,380][INFO ][o.o.n.Node               ] [opensearch-node1] starting ...
opensearch-node1         | [2022-04-25T13:35:27,564][INFO ][o.o.t.TransportService   ] [opensearch-node1] publish_address {172.28.0.3:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node1         | [2022-04-25T13:35:27,739][INFO ][o.o.b.BootstrapChecks    ] [opensearch-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node1         | ERROR: [1] bootstrap checks failed
opensearch-node1         | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
opensearch-node1         | ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/opensearch-cluster.log
opensearch-node1         | [2022-04-25T13:35:27,748][INFO ][o.o.n.Node               ] [opensearch-node1] stopping ...
opensearch-node2         | [2022-04-25T13:35:27,752][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node2] PerformanceAnalyzer Enabled: false
opensearch-node2         | [2022-04-25T13:35:27,790][INFO ][o.o.n.Node               ] [opensearch-node2] initialized
opensearch-node2         | [2022-04-25T13:35:27,790][INFO ][o.o.n.Node               ] [opensearch-node2] starting ...
opensearch-node1         | [2022-04-25T13:35:27,798][INFO ][o.o.n.Node               ] [opensearch-node1] stopped
opensearch-node1         | [2022-04-25T13:35:27,798][INFO ][o.o.n.Node               ] [opensearch-node1] closing ...
opensearch-node1         | [2022-04-25T13:35:27,819][INFO ][o.o.n.Node               ] [opensearch-node1] closed
opensearch-node2         | [2022-04-25T13:35:27,986][INFO ][o.o.t.TransportService   ] [opensearch-node2] publish_address {172.28.0.2:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node1         | Killing performance analyzer process 103
opensearch-node2         | [2022-04-25T13:35:28,156][INFO ][o.o.b.BootstrapChecks    ] [opensearch-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node2         | ERROR: [1] bootstrap checks failed
opensearch-node2         | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
opensearch-node2         | ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/opensearch-cluster.log
opensearch-node2         | [2022-04-25T13:35:28,165][INFO ][o.o.n.Node               ] [opensearch-node2] stopping ...
opensearch-node2         | [2022-04-25T13:35:28,200][INFO ][o.o.n.Node               ] [opensearch-node2] stopped
opensearch-node2         | [2022-04-25T13:35:28,200][INFO ][o.o.n.Node               ] [opensearch-node2] closing ...
opensearch-node2         | [2022-04-25T13:35:28,219][INFO ][o.o.n.Node               ] [opensearch-node2] closed
opensearch-node2         | Killing performance analyzer process 103
opensearch-node1         | OpenSearch exited with code 78
opensearch-node1         | Performance analyzer exited with code 143
opensearch-dashboards    | {"type":"log","@timestamp":"2022-04-25T13:35:28Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: getaddrinfo ENOTFOUND opensearch-node1 opensearch-node1:9200"}
opensearch-node2         | OpenSearch exited with code 78
opensearch-node2         | Performance analyzer exited with code 143
opensearch-node1 exited with code 0
opensearch-node2 exited with code 0
opensearch-dashboards    | {"type":"log","@timestamp":"2022-04-25T13:35:31Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: getaddrinfo ENOTFOUND opensearch-node1 opensearch-node1:9200"}

[elvince]

It is linked to : https://opensearch.org/docs/latest/opensearch/install/important-settings/

Look here, you will have your solution on how to setup the vm.max_map_count https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-prod-prerequisites

Might be interesting to update the OpenSearch website to have those command for all environnent

[kartg]

As @elvince has noted, the issue stems from the configured max_map_count value, as indicated by the logs:

ERROR: [1] bootstrap checks failed
opensearch-node1         | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

We do call this out on the Opensearch Docker/Download page witih a link to the docs page on how to update these settings.

@hemna can you confirm if this resolves your issue? If so, what could we improve on the OpenSearch website that would've made it easier to identify and resolve this?

[dblock]

Note that I just ran into the same thing in https://github.com/opensearch-project/opensearch-rs/issues/45#issuecomment-1111193478. Can we turn this error into a warning?

[elvince]

@kartg I think it be great if the doc contains all the different setup per environment - like the link I posted. Otherwise, I'm not sure how to improve the search as I found the issue with the "max_map_count" terms in the web site search bar.

[kartg]

Note that I just ran into the same thing in https://github.com/opensearch-project/opensearch-rs/issues/45#issuecomment-1111193478. Can we turn this error into a warning?

@dblock we're currently logging the error as a warning but this doesn't fail bootstrap/startup.

The next step up from that would be to set alwaysEnforce for MaxMapCountCheck - this would result in bootstrap checks failing by throwing a NodeValidationException

[dblock]

@kartg But is that the issue with the node? @elvince did it fix your problem?

[elvince]

@kartg But is that the issue with the node? @elvince did it fix your problem?

I have no issue, I just helped to solve an issue.

[andrross]

The current behavior is that if you are bound to the loopback address and using a single node cluster, then this is logged as warning but the server can start up. Otherwise, it is being assumed that the server is intended for production use and this is logged as an error and the server is prevented from starting. The error message is pretty explicit about what the problem is, and searching vm.max_map_count on opensearch.org will get you to the important settings page.

@dblock Do you think we should change this behavior?

cc @kartg

[kartg]

cc @hemna to confirm if adjusting the max_map_count value fixed their issue

[dblock]

@dblock Do you think we should change this behavior?

If users report that their installation doesn't work and changing this setting makes it work, then yes. Let's see what @hemna says and whether we have other reports like this one?

[dzeroc]

Hi,

I'm experiencing that documented docker-compose.yml doesn't work, but in a different way. May I open another issue?

I'm on Ubuntu 22.04 LTS with:

Docker Engine - Community - Version: 20.10.15 Docker Compose version v2.5.0

$ cat /proc/sys/vm/max_map_count 262144

I followed this and then this. Running "docker-compose up" with https://opensearch.org/samples/docker-compose.yml it always returns:

opensearch-node1       | [2022-05-11T15:09:35,744][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-node1, opensearch-node2] to bootstrap a cluster: have discovered [{opensearch-node1}{HAXYh2n_TnSM-FiYgnZQDA}{sCEi7BXzSjy9AVRmUbfM5A}{172.24.0.4}{172.24.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.24.0.3:9300] from hosts providers and [{opensearch-node1}{HAXYh2n_TnSM-FiYgnZQDA}{sCEi7BXzSjy9AVRmUbfM5A}{172.24.0.4}{172.24.0.4:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node2       | [2022-05-11T15:09:45,308][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-node1, opensearch-node2] to bootstrap a cluster: have discovered [{opensearch-node2}{n6ZaxN9zRtir4noIuiPkMg}{_2Q8WoCLSd2gEiC1dPRkLw}{172.24.0.3}{172.24.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.24.0.4:9300] from hosts providers and [{opensearch-node2}{n6ZaxN9zRtir4noIuiPkMg}{_2Q8WoCLSd2gEiC1dPRkLw}{172.24.0.3}{172.24.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

I changed docker image tag from latest to 1.3.2 and 1.3.0 with same results. Have i missed something? Thanks in advance

[dblock]

@dzeroc Do open a new issue please.

[maxmanus96]

Hi everyone, I also face the same issue, I just commented node2 because I want to have one node only but also use the docker-compose file, received the same error 78 on node1 which is 2023-03-18 16:39:26 ERROR: [1] bootstrap checks failed 2023-03-18 16:39:26 [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144], besides that opensearch-dashboard up and running, looking for your suggestions, best regards,

Additionally, when I run single node with the command docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" --name opensearch-node -d opensearchproject/opensearch:latest, works fine but ofcourse no visuals,

[kid1412621]

this might be helpful. https://stackoverflow.com/questions/42111566/elasticsearch-in-windows-docker-image-vm-max-map-count

[gaiksaya]

Looks like the documentation is updated Increase the number of memory maps available to OpenSearch. for docker https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/

@hemna Is your issue resolved? Please feel free to re-open the issue in case. Thanks!