espresso-sequencer icon indicating copy to clipboard operation
espresso-sequencer copied to clipboard

Fix fee catchup after restart

Open jbearer opened this issue 1 year ago • 1 comments

This PR reproduces and will fix an issue discovered in staging, in which, after a staggered restart of all nodes, the network has forgotten the fee state for all accounts except those which actively produced blocks during the restart. Since the state is needed for undecided blocks, it cannot directly be obtained from persisted merklized state, which is only available for decided blocks.

This PR:

  • Adds a restart-style test reproducing the issue
  • Adds a new catchup endpoint to fetch multiple accounts at once, for performance
  • Modifies SQL-based fee account catchup to reconstruct fee state by searching back to the last decided state and replaying headers
  • Caches results of successful reconstruction in HotShot's validated state map
  • Decouples storage from API layer by creating CatchupStorage trait, distinct from CatchupDataSource. This allows us to get more information when querying storage, such as the leaf corresponding to fetched account state, which in turn ensures we are always able to cache fetched state in the HotShot view map
  • Adds a background task to fetch missing quorum proposals that are parents of proposals we have. This ensures that every node will eventually have every proposal in a chain between valid proposals and the anchor, which is required for replay to work
  • UPDATES HOTSHOT TO A NON-RELEASE TAG -- let's figure out what to do about that before merging

Key places to review

jbearer avatar Oct 10 '24 21:10 jbearer