firefly-ethconnect icon indicating copy to clipboard operation
firefly-ethconnect copied to clipboard

Increase checkpoint block number when there are no events, to minimize cost on restart

Open peterbroadhurst opened this issue 4 years ago • 0 comments

Currently we only increase the block high-watermark on a subscription to an event stream, when we have successfully processed a batch: https://github.com/kaleido-io/ethconnect/blob/ad40de61370ec2b99a8cfc79cdf72e5dcac3d914/internal/kldevents/logprocessor.go#L83

This means that if no events arrive on a subscription for a long time (millions of blocks) we never increase the blockHWM, and write a new checkpoint. So on restart (of ethconnect or the node), we drive a high load on the node as we recreate the filter at a very old block number.

Instead we need a system to safely move the blockHWM forwards, at regular intervals, so it's never more than a few hundred blocks behind the head. This will mean periodically in the polling cycle, asking the node for the current block height ahead of the call to eth_getFilterChanges and if no results are found, then setting the blockHWM to a safe number (such as 100 blocks behind the head, if that's higher than zero, and higher than the previous blockHWM).

fyi @vdamle @panghalamit

peterbroadhurst avatar Jan 21 '21 15:01 peterbroadhurst