Backup Sync

[fboucquez]

Currently, every time a new node is created, it needs to synchronize from scratch. This takes hours/days. It may be possible to have a preloaded rockdb and/or mongo database in the cloud (CDN, tardigrade, or s3 for example) where symbol bootstrap can download it, cache it, and unzip it locally in the different nodes.

This s3 zip must be kept up to date. NGL should be pushing new versions of the zip every week or so from one of the trusted running nodes.

Bootstrap will check and update the zip file locally when creating new nodes.

I've done a POC and it seems working fine when both rockdb and mongodb are provided. What's required:

1. Set up the cloud infrastructure to maintain and upload the zip file
2. Create the node(s) that updates the zip file in the cloud with a cron job.
3. Validate with core than nothing node-specific is pushed inside the zip (like logs, harvesting data, certificates, or keys).
4. Exclude the mongo db from the zip. The broker should be able to upgrade an empty mongo db from a preloaded rockdb
5. Add a --backupSync flag that pulls the data from the cloud and copies it into the different new nodes' data folders.
6. Cache and verify downloaded zip files so they are only pulled when necessary, maybe some kind of s3 sync (so only new bytes are downloaded, not the whole zip)

[fboucquez]

Renamed from fast sync to backup sync

[fboucquez]

Update. I've tested:

1 - Unzip a mongo db to be used as-is. Not approved by the core team as it may not work in all the cases. 2 - The importer tool. It's currently too to show a big improvement over a regular sync.

Next option would be mongo dump and recovery. It's not trivial as the mongo needs to be running in order to perform these operations.

We are getting close to mainnet launch and the benefits of this feature will not be visible in the early stages of mainnet. I'm downgrading it to P3 to be worked after launch