Getting receipt block mismatch

[jurgenwerk]

Hi! We just upgraded our graph node to 0.27.0 and deployed a new subgraph using the receipt: true feature in some of the handlers. The sync works for a while but then it stops syncing at block 21373261 (Sokol network).

This is from the logs:

Aug 05 09:17:05.140 INFO Applying 22 entity operation(s), block_hash: 0x01b3fa8fb115d18ee9f78e4acdbc58cf52e17cb863a617c7497f2d2df718b8c1, block_number: 21373178, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: SubgraphInstanceManager
Aug 05 09:17:05.386 INFO Done processing trigger, gas_used: 26159270796, data_source: CardToken, handler: handleTransfer, total_ms: 239, transaction: 0x4577…b6a8, address: 0xb236…f7ee, signature: Transfer(indexed address,indexed address,uint256), sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: SubgraphInstanceManager
Aug 05 09:17:05.409 INFO Applying 9 entity operation(s), block_hash: 0x832ce6b0ac64b6163e8390377938f1e92aefe4395e92c95a8d3e9076b3085a30, block_number: 21373179, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: SubgraphInstanceManager
Aug 05 09:17:27.198 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148399, current_block_head: 27148398, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:17:47.066 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148400, current_block_head: 27148399, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:17:53.194 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148401, current_block_head: 27148400, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:18:17.280 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148402, current_block_head: 27148401, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:18:41.414 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148403, current_block_head: 27148402, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:19:04.899 INFO Scanning blocks [21373252, 21373252], range_size: 1, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: BlockStream
Aug 05 09:19:05.025 INFO Scanning blocks [21373253, 21373262], range_size: 10, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: BlockStream
Aug 05 09:19:05.181 INFO Scanning blocks [21373263, 21373362], range_size: 100, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: BlockStream
Aug 05 09:19:05.304 INFO Scanning blocks [21373363, 21374362], range_size: 1000, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: BlockStream
Aug 05 09:19:05.359 INFO Done processing trigger, gas_used: 20783062011, data_source: DaiToken, handler: handleTransfer, total_ms: 213, transaction: 0x89d4…afe5, address: 0xfedc…30d1, signature: Transfer(indexed address,indexed address,uint256), sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: SubgraphInstanceManager
Aug 05 09:19:05.363 INFO Applying 6 entity operation(s), block_hash: 0xa4159a345ebee9d14ca8777b609c215475c6613e7ace3169cfdad5f03badb769, block_number: 21373261, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: SubgraphInstanceManager
Aug 05 09:19:05.420 INFO receipt block mismatch, tx_hash: 0xb06d…a1d5, block_hash: 0x0219…2ed0, receipt_block_hash: 0x1e76…298d, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: BlockStream
Aug 05 09:19:08.832 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148404, current_block_head: 27148403, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:19:33.160 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148405, current_block_head: 27148404, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:19:56.090 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 27148406, current_block_head: 27148405, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:20:21.553 INFO receipt block mismatch, tx_hash: 0x12a7…7460, block_hash: 0x3622…36e6, receipt_block_hash: 0x238d…c067, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:20:21.564 INFO receipt block mismatch, tx_hash: 0xda6d…f8ca, block_hash: 0x3622…36e6, receipt_block_hash: 0x238d…c067, provider: poa-sokol-rpc-0, component: BlockIngestor
Aug 05 09:20:21.572 INFO receipt block mismatch, tx_hash: 0xc7c5…fb6d, block_hash: 0x3622…36e6, receipt_block_hash: 0x238d…c067, provider: poa-sokol-rpc-0, component: BlockIngestor
...

Any ideas? What does this error mean?

[leoyvens]

It means that when requesting receipts for a recent block, the json-rpc server returned a receipt with a block hash that did not match the block we were originally requesting the receipt for. This can happen in normal operation when a reorg happens, but if the graph node cannot make progress then that means it is getting inconsistent data from json-rpc.

[jurgenwerk]

@leoyvens thank you but I'm still struggling to understand why is this happening. We're using a stable and reliable RPC node which works great, but after upgrading the graph node from 0.26.0 to 0.27.0 and specifying receipt: true for a couple of event handlers (to get the gasUsed), we start getting this error and the syncing stops.

[azf20]

hi @jurgenwerk - can you confirm the block where indexing stops? In the logs I see

Aug 05 09:19:05.420 INFO receipt block mismatch, tx_hash: 0xb06d…a1d5, block_hash: 0x0219…2ed0, receipt_block_hash: 0x1e76…298d, sgd: 1, subgraph_id: QmTD59KigFww9aCNHDDvwMh8g6HxLfv4hZMFzFahdi7HFb, component: BlockStream

But when I look at the block you mention (21373261), that block is successfully processed, so it seems to be encountering the error at the next block with a trigger - and our logs don't give the whole tx_hash or block_hash. Could you run the Graph Node with GRAPH_LOG: trace and see if there is more information?

[gzliudan]

Try to set envirment:

export ETHEREUM_ANCESTOR_COUNT=1
export ETHEREUM_REORG_THRESHOLD=1

[leoyvens]

@gzliudan ETHEREUM_ANCESTOR_COUNT has actually been removed, just setting ETHEREUM_REORG_THRESHOLD is enough.

[gzliudan]

@leoyvens Yes, ETHEREUM_ANCESTOR_COUNT has been removed. Thank you very much.

[github-actions[bot]]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[jurgenwerk]

Closing because we abandoned the upgrade to 0.27 and I am unsure what went wrong.