firefly-ethconnect
firefly-ethconnect copied to clipboard
Increase checkpoint block number when there are no events, to minimize cost on restart
Currently we only increase the block high-watermark on a subscription to an event stream, when we have successfully processed a batch: https://github.com/kaleido-io/ethconnect/blob/ad40de61370ec2b99a8cfc79cdf72e5dcac3d914/internal/kldevents/logprocessor.go#L83
This means that if no events arrive on a subscription for a long time (millions of blocks) we never increase the blockHWM, and write a new checkpoint. So on restart (of ethconnect or the node), we drive a high load on the node as we recreate the filter at a very old block number.
Instead we need a system to safely move the blockHWM forwards, at regular intervals, so it's never more than a few hundred blocks behind the head. This will mean periodically in the polling cycle, asking the node for the current block height ahead of the call to eth_getFilterChanges
and if no results are found, then setting the blockHWM to a safe number (such as 100 blocks behind the head, if that's higher than zero, and higher than the previous blockHWM).
fyi @vdamle @panghalamit