[BUG] OpenSearch gets stuck opening indices since homebrew OpenJDK update

[unilynx]

Describe the bug

First - I know opensearch currently isn't supported on M3s and homebrew is not an official release, but it's still really convenient to have and I hope I can get some pointers here.

since homebrew updated its openjdk version (to 21.0.1) - https://github.com/Homebrew/homebrew-core/commit/1c027971bffc50813dec26940d6633b9762f785e - opensearch has been broken for me and a few colleagues when you're in the 100+ index range.

when starting opensearch it will only open part of the existing indices (the rest stays red). how many indices go green differs per startup. often just 24 out of 229, but also eg. 43 or 142. setting -Elogger.level=DEBUG seems to open more (but still not all) indices after restart, so I suspect a locking/timing issue.

Indices that stay in red all debug-log messages like this at startup

• throttling allocation
• found 1 allocation candidates of
  • (the above two messages repeat a few times)
• allocating
• found 1 allocation candidates
• allocating

and then the messages stop for that index. I notice that a log line with allocating appears twice which is suspect because indices that do make it to green log this:

• throttling allocation
• found 1 allocation candidates of
  • (the above two messages repeat a few times)
• allocating
• creating index
• creating Index ..shards [1]/[0] - reason [CREATE_INDEX]

just one 'allocating' and after that the index opens and makes it to green.

downgrading openjdk fixes it and allows me to open all indices, so the data itself doesn't appear to have been corrupted. I can reproduce this on muiltiple machines and wiping the data dirs completely doesn't make a difference.

Related component

Search

To Reproduce

Please see above

Expected behavior

All indices should go green like they do when I downgrade openjdk

Additional Details

Plugins analysis-icu

Host/Environment (please complete the following information):

• OS: Sonoma 14.2.1 (23C71)
• Hardware: Macbook M3 Pro