Implement set reconciliation based sync (syncv2)

[ivan4th]

Motivation

Existing sync is causing a lot of resource usage, including memory consumption and network traffic, and can't replace gossip which can also be unreliable at times.

Description

This implements the new sync protocol as described here. The algorithm is based on Range-Based Set Reconciliation paper by Aljoscha Meyer and also loosely based on SREP: Out-Of-Band Sync of Transaction Pools for Large-Scale Blockchains paper by Novak Boškov, Sevval Simsek, Ari Trachtenberg, and David Starobinski.

Test Plan

Add systests that verify new sync. Verify on testnet, then on mainnet.

TODO

• [ ] Explain motivation or link existing issue(s)
• [X] Sync implementation based on red-black monoid tree
• [X] P2P wire protocol for sync
• [x] Implement SREP-like multi-peer sync
• [ ] Make code more efficient by getting rid of excess monoid abstraction
• [ ] Implement syncv2 for ATXs. This may include adding an untrusted LayerID field to ATXs to make sync more efficient
• [ ] Implement SQLite-based "pre-balanced" monoid tree for old data to reduce memory consumption
• [ ] Harden the protocol by enforcing communication limits
• [ ] Consider replacing red-black monoid tree with a BTree so it can also be used for atxsdata
• [ ] Implement sync for other object types
• [ ] Test changes and document test plan
• [ ] Update documentation as needed
• [ ] Update changelog as needed

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 81.49957% with 644 lines in your changes missing coverage. Please review.

Project coverage is 81.7%. Comparing base (4ab0d09) to head (1d7f9a7). Report is 15 commits behind head on develop.

Files with missing lines	Patch %	Lines
sync2/hashsync/rangesync.go	67.8%	101 Missing and 55 partials :warning:
sync2/dbsync/fptree.go	83.8%	81 Missing and 28 partials :warning:
sync2/hashsync/sync_tree.go	82.6%	61 Missing and 20 partials :warning:
sync2/dbsync/dbitemstore.go	74.2%	30 Missing and 23 partials :warning:
sync2/hashsync/handler.go	80.3%	23 Missing and 20 partials :warning:
sync2/dbsync/dbiter.go	80.8%	27 Missing and 13 partials :warning:
sync2/hashsync/split_sync.go	78.8%	18 Missing and 8 partials :warning:
sync2/hashsync/sync_tree_store.go	72.8%	16 Missing and 9 partials :warning:
sync2/dbsync/syncedtable.go	92.0%	9 Missing and 6 partials :warning:
sync2/hashsync/log.go	59.4%	15 Missing :warning:
... and 11 more

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5769     +/-   ##
=========================================
- Coverage     81.7%   81.7%   -0.1%     
=========================================
  Files          312     334     +22     
  Lines        34625   38076   +3451     
=========================================
+ Hits         28306   31127   +2821     
- Misses        4480    4908    +428     
- Partials      1839    2041    +202     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[mathcrypto]

Did we agree already on adding the LayerID field to ATXs? @poszu @ivan4th. ATXs should only include what's need to be there

[ivan4th]

@mathcrypto the discussion was not finished yet, but I restarted it https://github.com/spacemeshos/go-spacemesh/pull/5785#discussion_r1590916018

[poszu]

@ivan4th This PR is reaching 14k LoC. Do you think you could start factoring out changes that are already done and could be reviewed to separate PRs?

[ivan4th]

@poszu that's the plan. Currently debugging ATX sync and this may cause a few more changes in rangesync, but other than that, rangesync together with types subpackages are close to being finished and soon I will split them off into a separate PR