ledger-tool: Get shreds from BigTable blocks
Problem
There is often a desire to examine/replay/etc blocks. If the blocks are very recent, they can often be pulled from an actively running node. Otherwise, the blocks must be pulled down from the warehouse node archives. These archives are uploaded on a per-epoch basis so they are quite large (100's of GB's). Even with a good download speed and capable machine, it can take several hours before having access to the block. And, 100's of GB's must be downloaded and expanded even if access to only a single block is desired.
Summary of Changes
With the addition of Entry data to BigTable, blocks, in the form that solana-validator and solana-ledger-tool operate with, can be recreated from BigTable. This change add a new BigTable command that does just that; fetch BigTable block data, parse it, and then insert the block as shreds into a local Blockstore.
Several important callouts:
- Shreds for some slot S will not have valid shred signatures; instead, shreds will signed with a dummy keypair. This does not inhibit the utility of these shreds by other solana-ledger-tool commands.
- Entry PoH data does not go back to genesis in BigTable. While this
data could be extracted and uploaded from existing rocksdb archives,
I'm not sure if that work is planned. So, this change adds a flag that
generates mock PoH entries for such blocks. These blocks are replayable
by passing
--skip-poh-verifytosolana-ledger-toolcommands.