FgForrest
Replica bootstrap
When new replica starts it needs to get up to date. We expect that the exporting facility of the evitaDB in a cluster environment will be configured to S3 endpoint and that the regular backups will be available for all replicas from this S3 endpoint.
New replica will fetch engineState from the leader and initiates new engine CDC stream from the leader to capture new changes.
For each catalog in the state it checks latest available SNAPSHOT backup in the S3 service and its catalogVersion.
For each catalog it fetches the current catalogVersion and WAL size delta from the leader. If the delta is within tolerated range it downloads the SNAPSHOT from the S3 otherwise it asks random replica to create up-to-date SNAPSHOT backup on the fly.
It restores the catalog from the backup. It reads missing WAL operations from random replica and replays them on internal SNAPSHOT until it reaches catalogVersion visible in the initial leader metadata fetch.
It opens CDC stream to leader and fetches all missing changes until local catalogVersion is equal to the catalogVersion of the leader (or within tolerated interval).
After that is makes replica available in the read-only endpoint slice.
