Scott Dugas

Related: https://github.com/FoundationDB/fdb-record-layer/issues/3146 the synchronized block can run into issues because it's not actually sufficient.

This has been resolved in #3702 but I didn't realize there was an existing issue.

ok, I created a reasonable reproduction at: https://github.com/FoundationDB/fdb-record-layer/pull/2823/files About half the time, it fails with timeouts just for the reads of `SystemKeyspace.METADATA_VERSION_KEY` and `Batch GRV request rate limit exceeded` for...

#2431 set a timeout for non-slow tests, but it only applies if you are skipping slow tests, by doing this we can always enforce it, so we won't be surprised...

`LuceneOptimizedPointsFormatTest.testAddIndexes` fails with: ``` org.junit.ComparisonFailure: expected: but was: at __randomizedtesting.SeedInfo.seed([57071E4365FA2B0E:E87D1748C5B83B39]:0) at org.junit.Assert.assertEquals(Assert.java:117) at org.junit.Assert.assertEquals(Assert.java:146) at com.apple.foundationdb.record.lucene.codec.TestFDBDirectory.openInput(TestFDBDirectory.java:166) ```

`LuceneOptimizedPointsFormatTest.testOneDimEqual` fails with: ``` com.apple.foundationdb.record.RecordCoreException: java.util.concurrent.ExecutionException: com.apple.foundationdb.record.RecordCoreException: File Reference missing for open IndexInput at __randomizedtesting.SeedInfo.seed([8987B24FB10B0D0B:B01E0ABA78438491]:0) ``` Similarly `LuceneOptimizedPointsFormatTest.testOneDimEqual` fails with: ``` com.apple.foundationdb.record.RecordCoreException: java.util.concurrent.ExecutionException: com.apple.foundationdb.record.RecordCoreException: File Reference missing for open IndexInput...

Another failure observed in the nightly build: https://github.com/FoundationDB/fdb-record-layer/actions/runs/14259839749 ``` ExplainPlanVisitorTest > doNotEvaluateOutsideOfLimit[seed=-7276650170682111482] com.apple.foundationdb.record.RecordCoreException: no default implementation at app//com.apple.foundationdb.record.query.plan.explain.ExplainPlanVisitor.visitDefault(ExplainPlanVisitor.java:667) at app//com.apple.foundationdb.record.query.plan.explain.ExplainPlanVisitor.visitDefault(ExplainPlanVisitor.java:138) at app//com.apple.foundationdb.record.query.plan.plans.RecordQueryPlanVisitor.visit(RecordQueryPlanVisitor.java:188) at app//com.apple.foundationdb.record.query.plan.explain.ExplainPlanVisitor.visit(ExplainPlanVisitor.java:656) at app//com.apple.foundationdb.record.query.plan.explain.ExplainPlanVisitor.visitAndJoin(ExplainPlanVisitor.java:194) at app//com.apple.foundationdb.record.query.plan.explain.ExplainPlanVisitor.visitUnorderedUnionPlan(ExplainPlanVisitor.java:616) at app//com.apple.foundationdb.record.query.plan.explain.ExplainPlanVisitor.visitUnorderedUnionPlan(ExplainPlanVisitor.java:138)...

Another failure on the nightly build: https://github.com/FoundationDB/fdb-record-layer/actions/runs/15038681872 ``` ExplainPlanVisitorTest > doNotEvaluateOutsideOfLimit[seed=-5890824399832672985] com.apple.foundationdb.record.RecordCoreException: no default implementation at app//com.apple.foundationdb.record.query.plan.cascades.explain.ExplainPlanVisitor.visitDefault(ExplainPlanVisitor.java:676) at app//com.apple.foundationdb.record.query.plan.cascades.explain.ExplainPlanVisitor.visitDefault(ExplainPlanVisitor.java:146) at app//com.apple.foundationdb.record.query.plan.plans.RecordQueryPlanVisitor.visit(RecordQueryPlanVisitor.java:188) at app//com.apple.foundationdb.record.query.plan.cascades.explain.ExplainPlanVisitor.visit(ExplainPlanVisitor.java:665) at app//com.apple.foundationdb.record.query.plan.cascades.explain.ExplainPlanVisitor.visitAndJoin(ExplainPlanVisitor.java:203) at app//com.apple.foundationdb.record.query.plan.cascades.explain.ExplainPlanVisitor.visitUnorderedUnionPlan(ExplainPlanVisitor.java:625) at app//com.apple.foundationdb.record.query.plan.cascades.explain.ExplainPlanVisitor.visitUnorderedUnionPlan(ExplainPlanVisitor.java:146) at...

After adding the timeout, it seems clear that one test will get a timeout, and then all remaining tests will either get an asyncToSync timeout at the first store open...

This was removed in https://github.com/FoundationDB/fdb-record-layer/pull/3448/files#diff-737b24ee2e542fa2d6bd98ea9f9d0ad7b51262c6b7f0fb60b4738b9f5079a6e8