Invalid ancestor re-org test reports wrong invalid transaction

[Rjected]

Describe the bug

The Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync hive test currently fails due to an invalid block. However, it is not the correct invalid block.

The test sets up a valid chain (of length 15) using forkchoiceUpdated / getPayload / newPayload. It then creates an invalid chain in the background, with an invalid block. In this case the invalid block is block 8 (P8').

The simulation fails because we report SYNCING (as we download blocks one by one) until we eventually report the wrong block as INVALID:

INFO (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Secondary Node has correct block
>> (d2d82b78) {"jsonrpc":"2.0","id":47,"method":"engine_newPayloadV1","params":[{"parentHash":"0xfaf8c0fa94e9f2bdef86fb3774a185fe2ad53a35b144eaf6ebccc6ab7d980939","feeRecipient":"0x0000000000000000000000000000000000000000","stateRoot":"0x96dd2f9b51e6e62e5eb0a61989f84b14f9a5b4143d6e44f48a0784448bdb16f7","receiptsRoot":"0x5d721667f3f41c6e8a5e630b2c5f259855d2365513295c4fd0c5d05498ce6489","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","prevRandao":"0xf0c12e098ed6a02bfe6a6f34b9f0fbf6a37b765b7a9d3f817341f88c25973e0a","blockNumber":"0xf","gasLimit":"0x3043ed","gasUsed":"0xa860","timestamp":"0x1243","extraData":"0x01","baseFeePerGas":"0x8545e76","blockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","transactions":["0x02f86c0709843b9aca008506fc23ac00830124f89400000000000000000000000000000000000003160180c080a0bb476cdbc61448c8f3df3a60b239ece15aecef4b3e0e2ea83bb826e14d014286a056697ae094e913705310289f44afa0503f911444a5e766fc43cfafb3791eb3da"]}]}
<< (d2d82b78) {"jsonrpc":"2.0","result":{"status":"SYNCING","latestValidHash":null,"validationError":null},"id":47}
INFO (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Response from main client: {SYNCING <nil> <nil>}

... repeats

>> (d2d82b78) {"jsonrpc":"2.0","id":53,"method":"engine_newPayloadV1","params":[{"parentHash":"0xfaf8c0fa94e9f2bdef86fb3774a185fe2ad53a35b144eaf6ebccc6ab7d980939","feeRecipient":"0x0000000000000000000000000000000000000000","stateRoot":"0x96dd2f9b51e6e62e5eb0a61989f84b14f9a5b4143d6e44f48a0784448bdb16f7","receiptsRoot":"0x5d721667f3f41c6e8a5e630b2c5f259855d2365513295c4fd0c5d05498ce6489","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","prevRandao":"0xf0c12e098ed6a02bfe6a6f34b9f0fbf6a37b765b7a9d3f817341f88c25973e0a","blockNumber":"0xf","gasLimit":"0x3043ed","gasUsed":"0xa860","timestamp":"0x1243","extraData":"0x01","baseFeePerGas":"0x8545e76","blockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","transactions":["0x02f86c0709843b9aca008506fc23ac00830124f89400000000000000000000000000000000000003160180c080a0bb476cdbc61448c8f3df3a60b239ece15aecef4b3e0e2ea83bb826e14d014286a056697ae094e913705310289f44afa0503f911444a5e766fc43cfafb3791eb3da"]}]}
<< (d2d82b78) {"jsonrpc":"2.0","result":{"status":"SYNCING","latestValidHash":null,"validationError":null},"id":53}
INFO (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Response from main client: {SYNCING <nil> <nil>}
>> (d2d82b78) {"jsonrpc":"2.0","id":54,"method":"engine_forkchoiceUpdatedV1","params":[{"headBlockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","safeBlockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","finalizedBlockHash":"0x0000000000000000000000000000000000000000000000000000000000000000"},null]}
<< (d2d82b78) {"jsonrpc":"2.0","result":{"payloadStatus":{"status":"INVALID","latestValidHash":"0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5","validationError":"links to previously rejected block"},"payloadId":null},"id":54}
INFO (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Response from main client fcu: {INVALID 0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5 0x4005f51cd0}

... repeats as simulation waits

>> (d2d82b78) {"jsonrpc":"2.0","id":71,"method":"engine_newPayloadV1","params":[{"parentHash":"0xfaf8c0fa94e9f2bdef86fb3774a185fe2ad53a35b144eaf6ebccc6ab7d980939","feeRecipient":"0x0000000000000000000000000000000000000000","stateRoot":"0x96dd2f9b51e6e62e5eb0a61989f84b14f9a5b4143d6e44f48a0784448bdb16f7","receiptsRoot":"0x5d721667f3f41c6e8a5e630b2c5f259855d2365513295c4fd0c5d05498ce6489","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","prevRandao":"0xf0c12e098ed6a02bfe6a6f34b9f0fbf6a37b765b7a9d3f817341f88c25973e0a","blockNumber":"0xf","gasLimit":"0x3043ed","gasUsed":"0xa860","timestamp":"0x1243","extraData":"0x01","baseFeePerGas":"0x8545e76","blockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","transactions":["0x02f86c0709843b9aca008506fc23ac00830124f89400000000000000000000000000000000000003160180c080a0bb476cdbc61448c8f3df3a60b239ece15aecef4b3e0e2ea83bb826e14d014286a056697ae094e913705310289f44afa0503f911444a5e766fc43cfafb3791eb3da"]}]}
<< (d2d82b78) {"jsonrpc":"2.0","result":{"status":"SYNCING","latestValidHash":null,"validationError":null},"id":71}
INFO (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Response from main client: {SYNCING <nil> <nil>}
>> (d2d82b78) {"jsonrpc":"2.0","id":72,"method":"engine_forkchoiceUpdatedV1","params":[{"headBlockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","safeBlockHash":"0xb2efad206aa9c68d7f93720bf8ef8c1f1aeb2de6e95ae0a370d1cbd0e8047fe4","finalizedBlockHash":"0x0000000000000000000000000000000000000000000000000000000000000000"},null]}
<< (d2d82b78) {"jsonrpc":"2.0","result":{"payloadStatus":{"status":"INVALID","latestValidHash":"0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5","validationError":"links to previously rejected block"},"payloadId":null},"id":72}
INFO (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Response from main client fcu: {INVALID 0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5 0x400680cb10}
FAIL (Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot, Invalid P8', Reveal using sync): Timeout waiting for main client to detect invalid chain

Reth reports that block 6, which the first valid block in the sidechain, and before the invalid block for that sidechain, is invalid:

2023-06-07T05:53:27.014018Z TRACE consensus::engine: start downloading full block. hash=0xce88b40d4b60053eb485e2143bbcd36e87f30d144192b5b8a617bbd1c5f0bde9
2023-06-07T05:53:27.015846Z TRACE consensus::engine: Downloaded full block hash=0xce88b40d4b60053eb485e2143bbcd36e87f30d144192b5b8a617bbd1c5f0bde9 number=8
2023-06-07T05:53:27.015921Z TRACE blockchain_tree: Inserting block hash=0xce88b40d4b60053eb485e2143bbcd36e87f30d144192b5b8a617bbd1c5f0bde9 number=8 parent_hash=0x07cecf1eb7bfee3a485f605e00bcafef25062bb74af3eb192d3296d57dfe3061
2023-06-07T05:53:27.015954Z  INFO try_insert_validated_block{block=(8, 0xce88b40d4b60053eb485e2143bbcd36e87f30d144192b5b8a617bbd1c5f0bde9)}: blockchain_tree: return=Ok(Disconnected { missing_parent: (7, 0x07cecf1eb7bfee3a485f605e00bcafef25062bb74af3eb192d3296d57dfe3061) })
2023-06-07T05:53:27.015961Z TRACE consensus::engine: start downloading full block. hash=0x07cecf1eb7bfee3a485f605e00bcafef25062bb74af3eb192d3296d57dfe3061
2023-06-07T05:53:27.025847Z TRACE consensus::engine: Downloaded full block hash=0x07cecf1eb7bfee3a485f605e00bcafef25062bb74af3eb192d3296d57dfe3061 number=7
2023-06-07T05:53:27.025915Z TRACE blockchain_tree: Inserting block hash=0x07cecf1eb7bfee3a485f605e00bcafef25062bb74af3eb192d3296d57dfe3061 number=7 parent_hash=0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb
2023-06-07T05:53:27.025974Z  INFO try_insert_validated_block{block=(7, 0x07cecf1eb7bfee3a485f605e00bcafef25062bb74af3eb192d3296d57dfe3061)}: blockchain_tree: return=Ok(Disconnected { missing_parent: (6, 0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb) })
2023-06-07T05:53:27.025992Z TRACE consensus::engine: start downloading full block. hash=0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb
2023-06-07T05:53:27.032844Z TRACE consensus::engine: Downloaded full block hash=0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb number=6
2023-06-07T05:53:27.032908Z TRACE blockchain_tree: Inserting block hash=0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb number=6 parent_hash=0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5
2023-06-07T05:53:27.032948Z DEBUG try_insert_validated_block{block=(6, 0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb)}:try_append_canonical_chain: blockchain_tree: Appending block to canonical chain head=0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb parent=(5, 0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5)
2023-06-07T05:53:27.033010Z TRACE try_insert_validated_block{block=(6, 0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb)}:try_append_canonical_chain: blockchain_tree: Executing sidechain block canonical_fork=(5, 0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5)
2023-06-07T05:53:27.033056Z  INFO try_insert_validated_block{block=(6, 0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb)}: blockchain_tree: return=Err(InsertBlockError { error: Execution(Validation(EVM { hash: 0x68578d8e615f14e5ff8babeaa4116e28263f1ddab4d67b7dc3d4502f18c8c6f9, message: "Transaction(LackOfFundForGasLimit { gas_limit: 0x0000000000000000000000000000000000000000000000000007fe5cf2bea001_U256, balance: 0x0000000000000000000000000000000000000000000000000000000000000000_U256 })" })), hash: 0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb, number: 6, parent_hash: 0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5, num_txs: 1, .. })

The error is LackOfFundForGasLimit:

2023-06-07T05:53:27.033069Z DEBUG consensus::engine: Failed to insert downloaded block err=InsertBlockError { error: Execution(Validation(EVM { hash: 0x68578d8e615f14e5ff8babeaa4116e28263f1ddab4d67b7dc3d4502f18c8c6f9, message: "Transaction(LackOfFundForGasLimit { gas_limit: 0x0000000000000000000000000000000000000000000000000007fe5cf2bea001_U256, balance: 0x0000000000000000000000000000000000000000000000000000000000000000_U256 })" })), hash: 0x60e9824348b6e06eeca12b55ed25332a9dc3a3024e64b486a7839755e796cefb, number: 6, parent_hash: 0x5a949798b3aaa04604a28146a9a0ada452b4507e7d9884ad00844249395ca2d5, num_txs: 1, .. }

Steps to reproduce

Run the hive test (it may help to add some traces to clients/reth/reth.sh):

./hive --client reth --sim "ethereum/engine" --sim.limit "engine-api/Invalid Ancestor Chain Re-Org, Invalid ReceiptsRoot"

Node logs

No response

Platform(s)

No response

What version/commit are you on?

No response

Code of Conduct

• [X] I agree to follow the Code of Conduct