[BUG] Higher memory consumption when running with JDK 21

[kroepke]

Describe the bug

We've noticed that some of our integration tests started failing with circuit breaker issues after updating the test container to 2.12.0. The test in question tries to optimize the size of large bulk indexing requests, but the failure isn't related to the functionality but to overall heap behavior.

Essentially, we are seeing the same issue as described in #12454

Additional testing to isolate the issue leads us to believe that the behavior change is due to the switch to JDK 21. Please see https://github.com/kroepke/opensearch-jdk21-memory for a smaller test case. It uses bulk indexing mainly to achieve memory pressure.

Flight recorder data yielded nothing obvious except a higher number of class version errors related to Painless components. It's unclear whether this is related or a red herring.

Related component

Other

To Reproduce

Please see https://github.com/kroepke/opensearch-jdk21-memory for a reproduction scenario.

Expected behavior

Switching the default JDK should not massively impact performance.

Additional Details

No response

[bernd]

We added a JUnit test in addition to the Docker Compose script test that emulates our bulk indexing handling.

https://github.com/kroepke/opensearch-jdk21-memory/commit/7fb693571dbea66f4850bc9e1a86f841125299ba

[bernd]

This issue in Elasticsearch seems to be related: https://github.com/elastic/elasticsearch/issues/99592

[andrross]

[Triage - attendees 1 2 3 4 5 6] @bernd Thanks for filing and the detailed reproduction.

[zakisaad]

This is a real problem for sure on AWS managed OpenSearch (when running 2.13), more context can be seen here: https://github.com/opensearch-project/OpenSearch/issues/12454#issuecomment-2174752960.