Investigate solutions for erroneous blocks in historical retrieval

[Cherrett]

In Cosmos and Substrate Node monitoring, we are retrieving historical/archive blocks to be used for slashing alerting. Currently, if a block retrieval fails during this process, the process will stop and this might happen again in the next historical retrieval round since we will attempt to retrieve the erroneous block again. This block will be skipped once it is outside of the _max_catchup_blocks threshold.

Hence, we must find a solution for erroneous blocks during historical block retrieval. One solution might be to check for errors and not stop the process. Instead we retry fetching the erroneous block for x amount of times. If this fails, we skip the block and continue from the next one. We can also retrieve the next block to check whether the block is erroneous or its an error from the source used.