reth icon indicating copy to clipboard operation
reth copied to clipboard

Published 20 hours ago • verified publisher icon paradigmxyz

Bad block with header hash(number: 17526352)

Open jsvisa opened this issue 2 years ago • 5 comments

Describe the bug

syncing with error: Bad block with header hash

Steps to reproduce

run with docker-compose, and meets the error as following

Node logs

WARN consensus::engine: Bad block with header hash: 0x36affed4e3c12d1c2f8cc086fc5492e75706a97b0df08d68cf3214a1b8fc15a4, invalid ancestor: Header { parent_hash: 0x6f4d1c0e39b3c67fb39d3f35417db6e309a745620168c02c4a66adcfac76b4af, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x2015fc753573262885784c11853c04bc52d09dbc, state_root: 0x990fe3c7a3069bbe90f3e7111d789e64b4b27043786197d409d980ae8bb8df5a, transactions_root: 0x03111edef9a83fe8b5ac70b59fb8ff976035a62661c4bb8fe722f21e067180ec, receipts_root: 0x59918c198d067cf09d5a11dc8910ab2f8f43aec0d3a0cc8ece6e02ffcb19e91a, withdrawals_root: Some(0xf6e09ee47909a1badeb6646c183364d78e17e3ceb9c5e0c584ae8fd124c6dcc6), logs_bloom: 0x002100000000000200202000800440000000000000000020000100c020001000004400000a00010d0000020810000480020120088801200000800000802c006000000008001000080860c00800900020000060800044080004400083802020001000402042180001000400000201284001100000014004000000001040080400084004630040000104420082020010228000000101010008000010c0000000000268004400102000802000000000008024008040000000080080000060000004002408028800000000001000000300104044020004020092000010520030e0000010240000000884000220000114028085001032000000400000500004080410, difficulty: 0x0000000000000000000000000000000000000000000000000000000000000000_U256, number: 17526352, gas_limit: 30000000, gas_used: 29951943, timestamp: 1687330067, mix_hash: 0x064705ae0f5896751b7a199d3b594242bf795ab3dcbfe04c7b54ba9ef5f4321e, nonce: 0, base_fee_per_gas: Some(12786906578), extra_data: Bytes(0x) }



### Platform(s)

Linux (x86)

### What version/commit are you on?

ghcr.io/paradigmxyz/reth:v0.1.0-alpha.1

### What database version are you on?

1

### If you've built Reth from source, provide the full command you used

_No response_

### Code of Conduct

- [X] I agree to follow the Code of Conduct

jsvisa avatar Jun 21 '23 09:06 jsvisa

Hi @jsvisa! Did syncing completely stop after this? It would be great as well to have some more reth logs, as well as CL client logs from when this occurred

Rjected avatar Jun 21 '23 17:06 Rjected

I had basically the same thing happen, here's some reth logs: reth.txt

CL was Lighthouse, and the only interesting Lighthouse log looks like this:

ERRO Error fetching block for peer           error: InconsistentPayloadReconstructed { slot: Slot(6598529), exec_block_hash: 0x654c2a482899fc2e84bf00f5288a7168727945e7d7947d94be56bbdfe635ffb0, canonical_transactions_root: 0xdb17c2156fa474cf7bb4b57aec7bbff997f340e3df7fc0604eda1856d50181c7, reconstructed_transactions_root: 0xcf7cc18b8270e69d55a3a5ad727095bf1a3ad4e2d1b974e34c33cc9485cb33cf }, block_root: 0xb9c938192fb1c51ab68e022caa757c09f0efa9c126570c0388a60c401ce18ee7

quickchase avatar Jun 21 '23 17:06 quickchase

Here is the reth logs:

reth     | 2023-06-22T00:30:31.434281Z  INFO net: Session established remote_addr=136.54.127.131:30304 client_version=erigon/v2.43.0/darwin-arm64/go1.20.2 peer_id=0xab525782629ef7d4b13fcfca8bf6104a3c9a93ecdd419086a89e9a3f6807f97a6b882e0db13be2040c692a49ea7c1d113937a09ca752f899c89fc952ae69abbb total_active=66
reth     | 2023-06-22T00:30:33.557463Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xa5f2c087da9a78cf7f8e3b4af56fb530052b72158b9504ef11b27110e872e5e7 safe_block_hash=0xcda277e70fa0ac251afc0f1ca0e1b9dc3b21dc8ce2007c4e6c92f4a631e7ece1 finalized_block_hash=0x7dc46efcdccbba7f6363f273ef662aa739ef931747a49dc76c5df782b5dd4339 status=Syncing
reth     | 2023-06-22T00:30:33.796825Z  WARN consensus::engine: Bad block with header hash: 0x36affed4e3c12d1c2f8cc086fc5492e75706a97b0df08d68cf3214a1b8fc15a4, invalid ancestor: Header { parent_hash: 0x6f4d1c0e39b3c67fb39d3f35417db6e309a745620168c02c4a66adcfac76b4af, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x2015fc753573262885784c11853c04bc52d09dbc, state_root: 0x990fe3c7a3069bbe90f3e7111d789e64b4b27043786197d409d980ae8bb8df5a, transactions_root: 0x03111edef9a83fe8b5ac70b59fb8ff976035a62661c4bb8fe722f21e067180ec, receipts_root: 0x59918c198d067cf09d5a11dc8910ab2f8f43aec0d3a0cc8ece6e02ffcb19e91a, withdrawals_root: Some(0xf6e09ee47909a1badeb6646c183364d78e17e3ceb9c5e0c584ae8fd124c6dcc6), logs_bloom: 0x002100000000000200202000800440000000000000000020000100c020001000004400000a00010d0000020810000480020120088801200000800000802c006000000008001000080860c00800900020000060800044080004400083802020001000402042180001000400000201284001100000014004000000001040080400084004630040000104420082020010228000000101010008000010c0000000000268004400102000802000000000008024008040000000080080000060000004002408028800000000001000000300104044020004020092000010520030e0000010240000000884000220000114028085001032000000400000500004080410, difficulty: 0x0000000000000000000000000000000000000000000000000000000000000000_U256, number: 17526352, gas_limit: 30000000, gas_used: 29951943, timestamp: 1687330067, mix_hash: 0x064705ae0f5896751b7a199d3b594242bf795ab3dcbfe04c7b54ba9ef5f4321e, nonce: 0, base_fee_per_gas: Some(12786906578), extra_data: Bytes(0x) }
reth     | 2023-06-22T00:30:35.071019Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x6f4d1c0e39b3c67fb39d3f35417db6e309a745620168c02c4a66adcfac76b4af safe_block_hash=0xcda277e70fa0ac251afc0f1ca0e1b9dc3b21dc8ce2007c4e6c92f4a631e7ece1 finalized_block_hash=0x7dc46efcdccbba7f6363f273ef662aa739ef931747a49dc76c5df782b5dd4339 status=Syncing
reth     | 2023-06-22T00:30:54.442075Z  INFO reth::cli: Status connected_peers=66 stage=Execution checkpoint=29.1% eta=1day 20h 41m
reth     | 2023-06-22T00:31:06.951503Z  INFO net: Session established remote_addr=23.88.142.81:30404 client_version=Nethermind/v1.19.2+42ba08b7/linux-x64/dotnet7.0.5 peer_id=0x4334e3924b81da43c79e17c18f66efcba795bfd7c70da9f30ae73814524828906cbb8cb34c43949630426fa045a3888ca262cbe335fe1207dd7a109565eb29db total_active=67
reth     | 2023-06-22T00:31:23.955654Z  INFO net: Session established remote_addr=46.38.243.67:30303 client_version=besu/v23.4.1/linux-x86_64/openjdk-java-17 peer_id=0x9b2ee2fb6be69458c981a53ae450bb3b8511d246c48fb688045cdb6aff187fed78bafc621b484170210fde4850c91457d9a2e22d227b0dafb527cc3c058acbd7 total_active=68
reth     | 2023-06-22T00:31:24.441808Z  INFO reth::cli: Status connected_peers=67 stage=Execution checkpoint=29.1% eta=1day 20h 40m 30s
reth     | 2023-06-22T00:31:41.347211Z  INFO net: Session established remote_addr=82.119.96.78:30303 client_version=Nethermind/v1.19.2+42ba08b7/linux-x64/dotnet7.0.5 peer_id=0x8305017076ab5dc5a7062afdfcae399336875f7443888f9de8db6d46bb9ce5c68a80f256adb48824407ddcc6a626e5d63a9bedec34c96c073894f84a28cb0066 total_active=67
reth     | 2023-06-22T00:31:54.441577Z  INFO reth::cli: Status connected_peers=67 stage=Execution checkpoint=29.1% eta=1day 20h 40m
reth     | 2023-06-22T00:32:21.684546Z  INFO net: Session established remote_addr=171.97.0.168:30404 client_version=Nethermind/v1.19.2+42ba08b7/linux-x64/dotnet7.0.5 peer_id=0xc3aa43412103c29203e36106458c601c7f1e30431836e06d0f9f1aa624d8b5bd5cda6207521b65d295acd78d90cb784ad191f1da7a0490870644379e741d4b78 total_active=67
reth     | 2023-06-22T00:32:22.437097Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xa5f2c087da9a78cf7f8e3b4af56fb530052b72158b9504ef11b27110e872e5e7 safe_block_hash=0xcda277e70fa0ac251afc0f1ca0e1b9dc3b21dc8ce2007c4e6c92f4a631e7ece1 finalized_block_hash=0x7dc46efcdccbba7f6363f273ef662aa739ef931747a49dc76c5df782b5dd4339 status=Syncing
reth     | 2023-06-22T00:32:23.039241Z  WARN consensus::engine: Bad block with header hash: 0x36affed4e3c12d1c2f8cc086fc5492e75706a97b0df08d68cf3214a1b8fc15a4, invalid ancestor: Header { parent_hash: 0x6f4d1c0e39b3c67fb39d3f35417db6e309a745620168c02c4a66adcfac76b4af, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x2015fc753573262885784c11853c04bc52d09dbc, state_root: 0x990fe3c7a3069bbe90f3e7111d789e64b4b27043786197d409d980ae8bb8df5a, transactions_root: 0x03111edef9a83fe8b5ac70b59fb8ff976035a62661c4bb8fe722f21e067180ec, receipts_root: 0x59918c198d067cf09d5a11dc8910ab2f8f43aec0d3a0cc8ece6e02ffcb19e91a, withdrawals_root: Some(0xf6e09ee47909a1badeb6646c183364d78e17e3ceb9c5e0c584ae8fd124c6dcc6), logs_bloom: 0x002100000000000200202000800440000000000000000020000100c020001000004400000a00010d0000020810000480020120088801200000800000802c006000000008001000080860c00800900020000060800044080004400083802020001000402042180001000400000201284001100000014004000000001040080400084004630040000104420082020010228000000101010008000010c0000000000268004400102000802000000000008024008040000000080080000060000004002408028800000000001000000300104044020004020092000010520030e0000010240000000884000220000114028085001032000000400000500004080410, difficulty: 0x0000000000000000000000000000000000000000000000000000000000000000_U256, number: 17526352, gas_limit: 30000000, gas_used: 29951943, timestamp: 1687330067, mix_hash: 0x064705ae0f5896751b7a199d3b594242bf795ab3dcbfe04c7b54ba9ef5f4321e, nonce: 0, base_fee_per_gas: Some(12786906578), extra_data: Bytes(0x) }
reth     | 2023-06-22T00:32:23.139295Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x6f4d1c0e39b3c67fb39d3f35417db6e309a745620168c02c4a66adcfac76b4af safe_block_hash=0xcda277e70fa0ac251afc0f1ca0e1b9dc3b21dc8ce2007c4e6c92f4a631e7ece1 finalized_block_hash=0x7dc46efcdccbba7f6363f273ef662aa739ef931747a49dc76c5df782b5dd4339 status=Syncing
reth     | 2023-06-22T00:32:24.441884Z  INFO reth::cli: Status connected_peers=67 stage=Execution checkpoint=29.1% eta=1day 20h 39m 30s

And here is the CL(prysm-v4.0.6) logs:

prysm_1  | time="2023-06-22 00:33:30" level=info msg="Processing block 0xe5a319bb... 6709184/6714165 - estimated time remaining 4m50s" blocksPerSecond=17.1 peers=43 prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:30" level=info msg="Processing block 0xb574cacc... 6709216/6714165 - estimated time remaining 4m23s" blocksPerSecond=18.8 peers=43 prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:30" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:30" level=info msg="Processing block 0x7e42c838... 6709248/6714165 - estimated time remaining 4m1s" blocksPerSecond=20.4 peers=43 prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:30" level=info msg="Processing block 0x8776b266... 6709280/6714165 - estimated time remaining 3m42s" blocksPerSecond=21.9 peers=43 prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:30" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:33" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x7dc46efcdccb headPayloadBlockHash=0x6f4d1c0e39b3 headSlot=6708836 prefix=blockchain
prysm_1  | time="2023-06-22 00:33:33" level=error msg="Could not find common ancestor root" error="unknown common ancestor" prefix=blockchain
prysm_1  | time="2023-06-22 00:33:33" level=warning msg="could not determine node weight" prefix=blockchain root=0xa971f4226b9f6b5b69d3b445f9d43442a7f55f7207d6c4799c7d2f4afa2b4035
prysm_1  | time="2023-06-22 00:33:33" level=info msg="Chain reorg occurred" commonAncestorRoot=0x0000000000000000000000000000000000000000000000000000000000000000 depth=6708837 distance=13417673 newRoot=0x5645a9a02630fb5edde3d8bb273697e13968e33db7273f424a02b979dc1996f5 newSlot=6708836 newWeight=621600000000000 oldRoot=0xa971f4226b9f6b5b69d3b445f9d43442a7f55f7207d6c4799c7d2f4afa2b4035 oldSlot=6708837 oldWeight=0 prefix=blockchain
prysm_1  | time="2023-06-22 00:33:44" level=info msg="Processing block 0x7bdc079d... 6708864/6714166 - estimated time remaining 3m45s" blocksPerSecond=23.5 peers=40 prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:44" level=info msg="Processing block 0xd1c8b455... 6708896/6714166 - estimated time remaining 3m30s" blocksPerSecond=25.1 peers=40 prefix=initial-sync
prysm_1  | time="2023-06-22 00:33:44" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync

And the reth's execution is continus growing:

image

Maybe this is caused by the CL's out-of-syncing issue?(the prysm's dataset is only 200MB)

jsvisa avatar Jun 22 '23 00:06 jsvisa

I had basically the same thing happen, here's some reth logs: reth.txt

CL was Lighthouse, and the only interesting Lighthouse log looks like this:

ERRO Error fetching block for peer           error: InconsistentPayloadReconstructed { slot: Slot(6598529), exec_block_hash: 0x654c2a482899fc2e84bf00f5288a7168727945e7d7947d94be56bbdfe635ffb0, canonical_transactions_root: 0xdb17c2156fa474cf7bb4b57aec7bbff997f340e3df7fc0604eda1856d50181c7, reconstructed_transactions_root: 0xcf7cc18b8270e69d55a3a5ad727095bf1a3ad4e2d1b974e34c33cc9485cb33cf }, block_root: 0xb9c938192fb1c51ab68e022caa757c09f0efa9c126570c0388a60c401ce18ee7

hey @quickchase! It looks like these logs were converted to WARN on the published tag for alpha: https://github.com/paradigmxyz/reth/blob/96081c753e293108c7b9a7b37fd5792af0543b03/crates/blockchain-tree/src/blockchain_tree.rs#L891

and it looks like this is a separate error message entirely, do you mind first pulling the latest commit or image possible, and filing a separate issue if you run into this again? thanks!

Rjected avatar Jun 22 '23 02:06 Rjected

My commit hash should be: b2451931bdb948b5f45f8753be5cf4230cf652e6

I'll see if I can open a new issue tomorrow, I was told to post here by @gakonst

quickchase avatar Jun 22 '23 02:06 quickchase

The error disappeared after full synced.

jsvisa avatar Jun 27 '23 15:06 jsvisa