QN checkpoints proof-of-concept

[ondratra]

Many times we had a situation when Atlas or Pioneer developer made a blockchain transaction that caused a crash in the processor (due to a bug). While the transaction was related only to one feature developers were testing, and they could theoretically work on other things, the whole processor was stuck. Then they wished the transaction never happened because then it blocked all of the other work they could be doing if the processor was still running.

Let's create a new feature for QN that will create checkpoints by periodically saving database states for the indexer and processor (for example, every 1000 blocks) and will enable state rollback to one of these backups. This feature will be available in development environments only.

For processor and indexer, we can improve Hydra to save and load the db via pg_dump and pgsql commands (they are part of the Postgress package in most Linux distros). Then we will likely need to create a new small service with GUI that will show a list of available checkpoints and buttons for executing rollback (and "rollforward").

Also, there will be a need to save the Joystream node state as well so it can be rolled back to the same block as the processor/indexer.

This is one of topics discussed in https://github.com/Joystream/joystream/issues/3708.