graph-node icon indicating copy to clipboard operation
graph-node copied to clipboard

Ingest historical block hashes

Open leoyvens opened this issue 3 years ago • 5 comments

An issue affecting the network, which sends queries by block hash, is that graph node will fail if it cannot find the block hash in its block cache. The block ingestor will ingest blocks within the reorg threshold, but holes can happen and nothing is done to fill them.

Since historical queries mean that all block hashes are potentially necessary, a way to solve this is to run a backfill ingestor job that ingests all historical hashes.

In terms of implementation, we'd want to be able to have only the block hash and number rather than the whole header, which is not valid in the current block cache structure. Also we'd probably want to leverage the firehose and not need a json-rpc implementation of the backfill.

leoyvens avatar May 02 '22 15:05 leoyvens

This would also address #1467

lutter avatar May 02 '22 23:05 lutter

I think this is supported for Firehose Blockstreams @maoueh ?

azf20 avatar May 03 '22 14:05 azf20

After #3738 we'll also need to ingest the timestamps to continue supporting the _meta field.

leoyvens avatar Aug 04 '22 12:08 leoyvens

We could alternatively lazily fetch blocks when there is a query (similar to #4768 )

azf20 avatar Aug 07 '23 14:08 azf20