forest icon indicating copy to clipboard operation
forest copied to clipboard

Re-write batch validator to download tipsets in parallel.

Open lemmih opened this issue 1 year ago • 0 comments

Issue summary

The sync_messages_check_state function downloads and validates tipset messages one at a time. It would be significantly more efficient to download messages in a background thread such that the validator never waits on IO.

Downloading messages is done by the chain_exchange_messages function which downloads the messages of a tipset + N of its parents (who have a lower epoch number). Right now, we process tipsets in chronological order (ie from low epoch to high epoch) which makes it difficult to download a batch of tipset messages. The code should be refactored such that batches of tipset messages are downloaded and fed to the validator. For example, if we're validating tipsets from epoch 100 to 200, we would download messages for epoch 110 + 9 parents (epochs 109,108,...,100), validate them in chronological order (100, 101,...,110), and repeat until we're done.

Other information and links

lemmih avatar Sep 13 '22 12:09 lemmih