[BUG] Docker Opensearch 2.1.0 Exited 137 OOM after Running Full Sweep

[randytan]

Describe the bug During a run of Docker Opensearch version 2.1.0 via docker-compose way, the Docker Opensearch container killed herself during full sweep execution.

We have provisioned 32GB RAM (50% of the total RAM) to the Opensearch container. Mem lock -1 and ulimit 65535.

version: '3.5'
services:
  opensearch-node1:
    image: dockerregistry:5000/opensearch:2.1.0
    container_name: "cls${CLS_VERSION}-opensearch-node1"
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms32G -Xmx32G" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
     . . .

To Reproduce Steps to reproduce the behavior:

1. Download the Opensearch 2.1.0 from Docker Hub
2. Adjust the Docker compose file to use version 2.1.0
3. Run docker-compose up and wait for Opensearch running full sweep
4. See error

Expected behavior Running full sweep does not killed the Docker container.

Plugins N/A (default)

Screenshots

[2022-08-01T08:04:23,684][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:09:23,684][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:14:23,684][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:19:23,685][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:24:23,685][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:24:23,871][INFO ][o.o.a.t.CronTransportAction] [b7f93d7d22f7] Start running AD hourly cron.
[2022-08-01T08:24:23,872][INFO ][o.o.a.t.ADTaskManager    ] [b7f93d7d22f7] Start to maintain running historical tasks
[2022-08-01T08:24:23,872][INFO ][o.o.a.c.HourlyCron       ] [b7f93d7d22f7] Hourly maintenance succeeds
[2022-08-01T08:29:23,685][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:34:23,686][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:39:23,686][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:44:23,686][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:49:23,687][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:54:23,687][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T08:59:23,687][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:04:23,687][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:09:23,688][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:14:23,688][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:19:23,688][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:24:23,688][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:24:23,872][INFO ][o.o.a.t.CronTransportAction] [b7f93d7d22f7] Start running AD hourly cron.
[2022-08-01T09:24:23,872][INFO ][o.o.a.t.ADTaskManager    ] [b7f93d7d22f7] Start to maintain running historical tasks
[2022-08-01T09:24:23,872][INFO ][o.o.a.c.HourlyCron       ] [b7f93d7d22f7] Hourly maintenance succeeds
[2022-08-01T09:29:23,689][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:34:23,689][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:39:23,689][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:44:23,690][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
[2022-08-01T09:49:23,895][INFO ][o.o.j.s.JobSweeper       ] [b7f93d7d22f7] Running full sweep
Killing opensearch process 33
OpenSearch exited with code 137
Performance analyzer exited with code 137
./opensearch-docker-entrypoint.sh: line 116:    34 Killed                  $OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli > $OPENSEARCH_HOME/logs/performance-analyzer.log 2>&1

Host/Environment (please complete the following information):

• OS: CentOS Linux release 7.9.2009 (Core)
• Version: 7.9.2009

Additional context We are running Opensearch container with Opensearch Dashboard. Dashboard is ok without any issue.