graph-node icon indicating copy to clipboard operation
graph-node copied to clipboard

Add runtime support for offchain data sources & templates

Open Theodus opened this issue 2 years ago • 1 comments

This implements most of the runtime support for handling offchain data sources. Unfortunately there's a lot of boilerplate here, but I think it's best for a first pass since my initial approach at something more cohesive became a massive time sink.

Theodus avatar Jul 29 '22 20:07 Theodus

Great stuff!

One thing I hadn't previously considered is handling reverts. In the DB we get that for free, but we need to also revert the in-memory state. It might be that this https://github.com/graphprotocol/graph-node/blob/c75e0d7570d33b6669cd52c989b476d190a24013/core/src/subgraph/instance.rs#L200 just happens to be sufficent, but do we additionally need any revert handling in the offchain monitor?

leoyvens avatar Aug 08 '22 12:08 leoyvens

This PR is now sufficient to to merge for internal testing, after review and integration cluster of course.

There is a severe though infrequent known bug, which is that if a file data source is created on block B1, but the files are only found on a later block B2 and B2 is reverted, then the entities will be gone forever. The intended fix is to write the entities for file data sources on the creation block rather than the current block, this way they will only be reverted if the data source also is. This will be the first follow up to this PR in case it isn't done in time for merging.

leoyvens avatar Aug 22 '22 15:08 leoyvens

Integration cluster shows no divergences, merging!

leoyvens avatar Aug 25 '22 11:08 leoyvens