services/horizon: State verifier can become stuck of connectivity issues

[bartekn]

What version are you using?

2.13.0

What did you do?

State verifier in Horizon use ingest.CheckpointChangeReader to stream ledger entry changes from buckets in history archive. While it's created with ctx that can cancel connections (ex. on shut down), there are no ctxs used when requesting certain bucket files. Because of this it's not possible to set a timeout that will kill a connection when it's stuck.

However, just setting a context with timeout to a connection is not a good solution because state processing is done while the data is downloaded. Instead, we should probably set a timeout on processing a single ledger entry.