Andrew Ross

I'm looking into this now

I have cherry picked the [modified integration test](https://github.com/itiyamas/OpenSearch-1/commit/1267b4360185f99b07d6672753656f8fc1d00117) into the latest on the main branch and have verified that this bug is fairly reliably reproducible. It usually takes me between...

Thanks @itiyama! Yes, the approach you've describe is exactly what I'm looking into.

@itiyama Please take a look at my [draft PR](https://github.com/opensearch-project/OpenSearch/pull/4133). I still need to make the test deterministic, and some of the various sleeps are still in place. The approach I...

> @saratvemulapalli Would you please check or provide the logs upgrade from 2.0.1 to 3.0.0 without security plugin? I don't see the related logs in https://ci.opensearch.org/logs/ci/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/gradle_check_6087.log. Did I miss anything?...

@itiyama @Bukhtawar I've added a test in CorruptedFileIT that deterministically fails without this fix and passes with it.

Also thinking out loud, I do kind of like option 2 as a good guarantee that we'll never reuse a segment number no matter what weird distributed system edge case...

Another failure here: https://build.ci.opensearch.org/job/gradle-check/977/ ``` "root_cause" : [ { "type" : "snapshot_exception", "reason" : "[repository_permanent:snapshot-one/B67YkZ99R9uEZ5BpNEeGJA] failed to update snapshot in repository", "stack_trace" : "SnapshotException[[repository_permanent:snapshot-one/B67YkZ99R9uEZ5BpNEeGJA] failed to update snapshot in repository];...

> Hi， "Ultrawarm has built everywhere", what's its meaning? Much of the implementation of Ultrawarm will be applicable here, so that code will be ported over where appropriate. > Will...

Quick update on the progress here: Phase 1 has been [released as an experimental feature in OpenSearch 2.3](https://opensearch.org/blog/releases/2022/09/opensearch-2-3-is-ready-for-download/)! We welcome any and all feedback as we work towards finalizing this...