hyperlane-monorepo
hyperlane-monorepo copied to clipboard
Implement a backward & forward-backward version of a watermarked cursor
Problem
See the first part of https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 for a little background on WatermarkContractSync / the RateLimitedContractSyncCursor, which we use for indexing IGP payments and message deliveries.
At the moment, we've only implemented RateLimitedContractSyncCursor as a forward-only cursor. This makes sense as the "rate limiting" part is only something that kicks in when we're close to the tip. However this means that we end up indexing IGP payments & message deliveries from the beginning of time moving forward.
Solution
We should be able to index IGP payments and message deliveries in a "forward backward" manner, just like we're able to do with message indexing. This way we can spin up a relayer or scraper and immediately start getting useful data from the tip.
I imagine it'd work something like:
- assuming https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 is done first and is implemented only for block-range indexing, implement a backward cursor that indexes from the tip backward with sliding block windows. We'd want to persist some information so that upon starting up, we only index blocks that haven't been previously indexed. Maybe this is already persisted with the existing watermark, where if the persisted watermark is < the tip, we backward index to that watermark, and we forward index from the tip?
- create a forward-backward cursor that includes this new backward one and the refactored version of the forward RateLimitedContractSyncCursor
Noting as @daniel-savu had mentioned that after https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 this is no longer super important because we're less bottlenecked on IGP indexing. Gonna de-prio accordingly