Syncing strategy

[wemeetagain]

Current syncing code is brittle and doesn't work well. The reason is that it has had features bolted on, and the sync strategy has not been updated as needed.

Below is a proposal for how to organize our syncing.

A first step is to list out some different situations the node may find itself in.

• The node has synced to a checkpoint in the past. We see peer statuses with a finalized chain ahead of our own.
• The node has synced to the latest finalized checkpoint. We see peer statuses with heads we haven't seen.
• The node has synced to the latest finalized checkpoint. The network has been experiencing a long period of non-finality and we see peer statuses with heads we haven't seen.
• The node has synced to head. We receive a gossip block with a parent we haven't seen.
• The node has synced to head. We receive a gossip block but haven't received all blobs / columns for that block.
• The node has synced to head. We receive a gossip blob/column but haven't received all blobs / columns or the block.

Currently, we have three modes of syncing (organized in two classes) to tackle these scenarios:

• Finalized range sync: This syncing mode organizes peers by status, uses the by_range requests to forward sync to a finalized checkpoint
• Unfinalized range sync: This syncing mode organizes peers by status, uses the by_range requests to forward sync.
• Unknown block sync: This syncing mode tracks unknown block roots and block inputs unavailable data and builds chains backwards and retrieves all blobs/columns.

This current strategy has several deficiencies:

• Unfinalized range sync has a race condition that cannot be avoided. Between statusing a peer and retrieving a batch of blocks, the peer's head may have changed. We cannot be sure which chain of blocks the peer will be sending us. Matters are made worse when considering blobs and columns. Syncing becomes unpredictable.
• Unknown block sync cannot handle long periods of non-finality. Because the syncing mechanism caches entire blocks + blobs/columns in memory, a strict cap must be enacted to avoid OOMing. Thus, we cannot rely on unknown block sync to handle periods of non-finality longer than this cap. We have to fall back to the unpredictable unfinalized range sync

We can overcome these deficiencies in the following way:

• Unfinalized range sync must use by_root requests to forward sync. This will ensure predictable syncing by always knowing which chain we are syncing. This implies that the header chain is already known in advance.
• Unknown block sync must decouple caching blocks + data from crafting a chain of headers. This will allow us to build long chains of headers that can happen during non-finality and not be limited by an artificially-low cap.

To a path forward:

• We can keep finalized range sync as it is, no problems.
• We need a new unfinalized forward sync that is fed a chain of headers (slot, blockRoot), downloads batches of block inputs, feeds into the block processor. (It can use a block input cache to avoid re-downloading previously-seen stuff)
• We need a new unknown block sync that is fed unknown blocks / block-roots and finds parents backwards until it intersects with our chain or is irrelevant. It can't cache the entire block + blobs/columns, it must only handle simplified block headers.

Integration:

• When we get a block or blob that is not fully available (but its parent is known), the slot+root can be fed to unfinalized forward sync.
• When we get a block or blob where the parent is not known, the header can be fed to the unknown block sync
• The block input cache could still maintain block inputs that are not fully available, or when the parent is not known.

[wemeetagain]

There may also be a benefit for unfinalized forwards syncing by_range (at least blocks by_range followed by data by_root).

There's a tradeoff that exists for the unknown block sync, whereby you get a complete view of the block dag of your peers, but at the cost of not progressing your own chain until linking with your known chain.

This tradeoff can be offset by unfinalized forwards syncing by_range. Where you don't know reliably which chain you're syncing to, but you can start progressing your own chain immediately.

[philknows]

As mentioned during standup, @nazarhussain will help create a flowchart/wire diagram of how we flow through the syncing process based on this issue and what was discussed on https://github.com/ChainSafe/lodestar/discussions/8214, so assigning him here for now.