Andrew Ross

@peternied Thanks! This looks promising. I did some basic testing and created an issue to discuss specifically enabling this: opensearch-project/opensearch-build-libraries#420

[Triage - attendees [1](https://github.com/andrross) [2](https://github.com/reta) [3](https://github.com/lukas-vlcek/) [4](https://github.com/Mahima081) [5](https://github.com/dbwiddis) [6](https://github.com/srikanthpadakanti)] @jed326 Thanks for filing and providing all the details on the reproduction. Looking forward to seeing a fix here.

> it looks to me the proposal is more about having a side store for raw data that is being ingested into the index? @reta @Bukhtawar There indeed seems to...

Another failure: https://github.com/opensearch-project/OpenSearch/pull/10388#issuecomment-1747944196

@rishabh6788 Looks very promising, I'm excited to see a dashboard with this information!

https://github.com/opensearch-project/OpenSearch/pull/10519#issuecomment-1754410824

I believe I have traced this back to the commit that introduced the flakiness: 9119b6dc20ea11d95a399c68505f1d858b78e30e (#9105) The following command will reliably reproduce the failure for me: ``` ./gradlew ':server:internalClusterTest' --tests...

Adding `Storage:Remote` label as it appears PR #9105 introduced this flakiness

I can get this to fail every time with the following seed: ``` ./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.DeleteSnapshotIT.testDeleteShallowCopySnapshot" -Dtests.seed=4CD3155D4F1C1A9F ``` ``` java.lang.AssertionError: java.lang.IllegalArgumentException: Provided Lock Name metadata__9223372036854775806__9223372036854775803__9223372036854775790__9223372036854775800___Hf3Dbw2QQagfGLlVBOUrg__9223370340398865071__1___ZxZ4Wh89SXyEPmSYAHrIrQ.lock is not Valid. at...

@peterzhuamazon Does the double quote approach work? If not, we could add snake case environment variables in core if that make things easier.