Support reconstruction of historical states from snapshots

[quickchase]

Description

Lighthouse has a flag (--reconstruct-historic-states) this flag allows the node to reconstruct all the historical states at any point in time, including after a snapshot sync. This allows for the node to be up and operational quickly while, in the background, recreating the historical states that allows it to return old data (i.e. archive node).

[ajsutton]

We'll need a way to specify the genesis state (or some state to start generating from). If checkpoint sync is used our anchor state will be from close to chain tip. For built-in network definitions we can provide that genesis state automatically (same as the default initial state we would have used). Will need a custom CLI arg for custom networks.

Then create a new service to regenerate historic states:

1. Get the genesis state (use same mechanism as we do for loading --initial-state)
2. Check the genesis state root matches the state root from block 0
3. Iterate through finalized blocks in the database (forwards), applying each block to the state with spec.replayValidatedBlock
4. After each block, check if it's been more than --data-storage-archive-frequency slots since the last state was stored, if so store this new state.

Note that storing the state probably means adding a new method to StorageUpdateChannel and Database and it will eventually call KvStoreFinalizedDao.FinalizedUpdater.addFinalizedState

Once we have the basics working, some follow up improvements:

1. Hook the new service in so Teku starts it after the HistoricalBlockSyncService completes downloading all historic blocks. HistoricalBlockSyncService is probably the right place to hook this in since it knows when it's done.
2. Store the finalized state roots as well. We can probably reuse StateRootRecorder which is used when importing blocks during a normal sync.
3. Handle restarts while part way through reconstructing historic states.
   • Suggest that each time we record a state, we also update a variable with the last state generated
   • When starting up if the variable is set, we can start from that last state rather than from genesis
   • Need to set the variable to a special value to indicate we've completed regeneration

[ajsutton]

Also in terms of where to put the new service, it probably makes sense to put it in a package alongside HIstoricalBlockFetcher in :beacon:sync.

[courtneyeh]

TODO: Implement doStop() method in ReconstructHistoricalStatesService class properly

[courtneyeh]

TODO: Also need to record the finalized state roots

[courtneyeh]

TODO: Add testing for service

[rolfyone]

closed too early, lets check our feature definition of done...

[ajsutton]

Docs written, options unhidden. All done. :tada: