Batch GRV Rate Limit Exceeded is not always thrown

[ScottDugas]

The tests noted in https://github.com/FoundationDB/fdb-record-layer/issues/2813 will occasionally run forever due to this code: https://github.com/FoundationDB/fdb-record-layer/blob/200ac05041a1af712f621a27b4c5c37f9eab001c/fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/storestate/FDBRecordStoreStateCacheEntry.java#L97-L100

Where it is combining two futures. The first one: recordStore.loadRecordStoreStateAsync is doing a regular read. The second one is doing a snapshot get of SystemKeyspace.METADATA_VERSION_KEY.

The first future fails with Batch GRV request rate limit exceeded (code 1051). The second future never completes.

I have tried to reproduce this in a more isolated environment, but it is proving tricky to get it to reliably start failing with Batch GRV request rate limit exceeded.