Make trxdb-loader resume from the last processed block on restarts (instead of always using the start-block)

[fschoell]

It would be helpful to have the trxdb-loader resume from the last processed block in case it needed a restart (due to KV or object store failures for example) instead of using the start-block (when in batch mode).

Procedure could be like this:

• On shutdown store the last successfully loaded block on disk if block-number < stop-block
• On startup check if there's such a file. Replace the start-block with the stored block number if that stored block number is between start-block and end-block. Otherwise delete that file and use the start_block from the config file.

Another possibility would be to look up the highest block_number from the KV store that is between the start-block and stop-block (if that's possible).

[sduchesneau]

I'm OK with proposed procedure (the last successfully loaded block could even be flushed every 100 blocks or whatever.

Another possibility would be to look up the highest block_number from the KV store that is between the start-block and stop-block (if that's possible). <-- this particular case is what we want to avoid, it goes against expected behavior of batch-mode. We sometimes use batch mode to cover ranges of blocks that were already written or partially written, etc. but a kind of local checkmark file is fine. Should not be hard to code, PRs welcome if you want to get your hands dirty !

things to watch out for:

1. save progress mark on fully written irreversible block only
2. use a flag for the location of that file, with the feature disabled if the file does not exist
3. make sure that the value is between start and stop block when loading the file on startup, else hard fail is probably the least surprising behavior.

[matthewdarwin]

Also the merged-filter should do the same.