Validate object store Garbage Collection policies

[refset]

Errors like Object 'tables/xt$txs/meta/l02-rc-p1-b31de0.arrow' doesn't exist may be observed when nodes disagree about exactly when trie files become garbage, and when such files can be safely evicted from object storage.

The default policy is to keep a minimum of 10 block files AND 24 hours of block file history (i.e. potentially much more than 10 blocks), with all the associated live trie files - per https://github.com/xtdb/xtdb/blob/c5cf87163ee0ee23a8f610aaf3c34e3c056d6048/core/src/main/kotlin/xtdb/api/GarbageCollectorConfig.kt#L11-L12

This policy may not be being applied correctly - potentially indicated by the error above (or it may have another cause) - further investigation is required.

In the meantime, it is safest to run XTDB with object store versioning enabled and disable the GC by adding the following to the config.yaml:

    garbageCollector:
      enabled: false

Internal Slack thread