go-ethereum
go-ethereum copied to clipboard
ethereum
BAD BLOCK error, cannot sync
System information
Geth version: 1.10.23 Prysm version: 3.0.0 OS & Version: Ubuntu 22.04.1 Commit hash : -
Expected behavior
sync normally
Actual behaviour
BAD BLOCK errors
Steps to reproduce the behaviour
I got many error logs saying "BAD BLOCK" and "beacon sync not yet started" today, it seemed it cannot sync anymore. Before this it is fully synced with the mainnet.
my start commands is
./geth --http --http.api eth,net,engine,admin --datadir ~/geth/data --authrpc.jwtsecret ~/jwt.hex --http.addr 192.168.1.101
Backtrace
ERROR[09-06|10:07:14.694] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"
WARN [09-06|10:07:14.694] Synchronisation failed, dropping peer peer=9802982d232f9e810d7253693444ea4a9482e26cd9737957922c8b7f5c6c55b3 err="retrieved hash chain is invalid: could not apply tx 254 [0x7f1874884305034878b0938dfa3e8aebd719abbbc173af9672ee0cfa90065013]: nonce too high: address 0x524540a2D2E07c7dCf1EA6e0Bc82DCebC30a3656, tx: 5 state: 0"
WARN [09-06|10:08:14.695] Synchronisation failed, dropping peer peer=9802982d232f9e810d7253693444ea4a9482e26cd9737957922c8b7f5c6c55b3 err=timeout
INFO [09-06|10:08:18.832] Downloader queue stats receiptTasks=0 blockTasks=0 itemSize=80.83KiB throttle=3243
INFO [09-06|10:08:19.693] Importing heavy sidechain segment blocks=983 start=15,476,126 end=15,477,108
ERROR[09-06|10:08:19.783] Unhandled trie error: missing trie node d91ac33fd2279f87f63926e7f8ad632730f0dfde5079c4888d36bd331e721eeb (path 080e0a030a0408) <nil>
INFO [09-06|10:08:19.785] Skip duplicated bad block number=15,476,126 hash=bc169d..f89114
ERROR[09-06|10:08:19.785]
########## BAD BLOCK #########
Chain config: Chain ID: 1 (mainnet)
Consensus: Beacon (proof-of-stake), merging from Ethash (proof-of-work)
Pre-Merge hard forks:
- Homestead: 1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
- DAO Fork: 1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
- Tangerine Whistle (EIP 150): 2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
- Spurious Dragon/1 (EIP 155): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
- Spurious Dragon/2 (EIP 158): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
- Byzantium: 4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
- Constantinople: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
- Petersburg: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
- Istanbul: 9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
- Muir Glacier: 9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
- Berlin: 12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
- London: 12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
- Arrow Glacier: 13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
- Gray Glacier: 15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
Merge configured:
- Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
- Network known to be merged: false
- Total terminal difficulty: 58750000000000000000000
- Merge netsplit block: <nil>
Number: 15476126
Hash: 0xbc169d7a58047cd65bf9a7cd78d6a254b011bec285e94d66b8f45d3717f89114
Error: could not apply tx 254 [0x7f1874884305034878b0938dfa3e8aebd719abbbc173af9672ee0cfa90065013]: nonce too high: address 0x524540a2D2E07c7dCf1EA6e0Bc82DCebC30a3656, tx: 5 state: 0
##############################
ERROR[09-06|10:08:19.788] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"
WARN [09-06|10:08:19.788] Synchronisation failed, dropping peer peer=53c808bde92acb604a5b4af85a58769aa36316d2bc203626eae38a3c26bb96b0 err="retrieved hash chain is invalid: could not apply tx 254 [0x7f1874884305034878b0938dfa3e8aebd719abbbc173af9672ee0cfa90065013]: nonce too high: address 0x524540a2D2E07c7dCf1EA6e0Bc82DCebC30a3656, tx: 5 state: 0"
ERROR[09-06|10:08:19.783] Unhandled trie error: missing trie node d91ac33fd2279f87f63926e7f8ad632730f0dfde5079c4888d36bd331e721eeb (path 080e0a030a0408)
Looks like your database is fried. Most likely the 1.10.22 issue. Easiest way to fix is to remove the state and resync, just do ./geth removedb, you can keep the ancients, just remove the chaindata
but I am new to geth, never used 1.10.22 before. why 1.10.22 issue will affect me?
Could you provide some logs from earlier on, around when that block was first encountered and the error was produced? Also, if you have it, the first ~100 lines of the startup.
Could you provide some logs from earlier on, around when that block was first encountered and the error was produced? Also, if you have it, the first ~100 lines of the startup.
Sorry, as I know geth is not saving logs by default. and I just follow the quick start guide, so I haven't it. Below are my first 100 lines when I restart geth
INFO [09-06|10:53:35.814] Starting Geth on Ethereum mainnet...
INFO [09-06|10:53:35.814] Bumping default cache on mainnet provided=1024 updated=4096
INFO [09-06|10:53:35.816] Maximum peer count ETH=50 LES=0 total=50
INFO [09-06|10:53:35.816] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [09-06|10:53:35.818] Sanitizing cache to Go's GC limits provided=4096 updated=3316
INFO [09-06|10:53:35.819] Set global gas cap cap=50,000,000
INFO [09-06|10:53:35.820] Allocated trie memory caches clean=497.00MiB dirty=829.00MiB
INFO [09-06|10:53:35.820] Allocated cache and file handles database=/home/asuka/geth/data/geth/chaindata cache=1.62GiB handles=50000
INFO [09-06|10:53:36.719] Opened ancient database database=/home/asuka/geth/data/geth/chaindata/ancient/chain readonly=false
INFO [09-06|10:53:36.722]
INFO [09-06|10:53:36.722] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [09-06|10:53:36.722] Chain ID: 1 (mainnet)
INFO [09-06|10:53:36.722] Consensus: Beacon (proof-of-stake), merging from Ethash (proof-of-work)
INFO [09-06|10:53:36.722]
INFO [09-06|10:53:36.722] Pre-Merge hard forks:
INFO [09-06|10:53:36.722] - Homestead: 1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [09-06|10:53:36.722] - DAO Fork: 1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [09-06|10:53:36.722] - Tangerine Whistle (EIP 150): 2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [09-06|10:53:36.722] - Spurious Dragon/1 (EIP 155): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [09-06|10:53:36.723] - Spurious Dragon/2 (EIP 158): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [09-06|10:53:36.723] - Byzantium: 4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [09-06|10:53:36.723] - Constantinople: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [09-06|10:53:36.723] - Petersburg: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [09-06|10:53:36.723] - Istanbul: 9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [09-06|10:53:36.723] - Muir Glacier: 9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [09-06|10:53:36.723] - Berlin: 12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [09-06|10:53:36.723] - London: 12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [09-06|10:53:36.723] - Arrow Glacier: 13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [09-06|10:53:36.723] - Gray Glacier: 15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [09-06|10:53:36.723]
INFO [09-06|10:53:36.723] Merge configured:
INFO [09-06|10:53:36.723] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [09-06|10:53:36.723] - Network known to be merged: false
INFO [09-06|10:53:36.723] - Total terminal difficulty: 58750000000000000000000
INFO [09-06|10:53:36.723] - Merge netsplit block: <nil>
INFO [09-06|10:53:36.723] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [09-06|10:53:36.723]
INFO [09-06|10:53:37.050] Disk storage enabled for ethash caches dir=/home/asuka/geth/data/geth/ethash count=3
INFO [09-06|10:53:37.050] Disk storage enabled for ethash DAGs dir=/home/asuka/.ethash count=2
INFO [09-06|10:53:37.050] Initialising Ethereum protocol network=1 dbversion=8
INFO [09-06|10:53:37.235] Loaded most recent local header number=15,476,125 hash=8d9556..9d73b9 td=58,000,486,640,375,988,633,957 age=1d5h44s
INFO [09-06|10:53:37.235] Loaded most recent local full block number=15,476,125 hash=8d9556..9d73b9 td=58,000,486,640,375,988,633,957 age=1d5h44s
INFO [09-06|10:53:37.236] Loaded most recent local fast block number=15,476,125 hash=8d9556..9d73b9 td=58,000,486,640,375,988,633,957 age=1d5h44s
INFO [09-06|10:53:37.236] Loaded last fast-sync pivot marker number=15,461,213
WARN [09-06|10:53:37.236] Enabling snapshot recovery chainhead=15,476,125 diskbase=15,478,127
WARN [09-06|10:53:37.244] Loaded snapshot journal diskroot=202918..8576d9 diffs=unmatched
WARN [09-06|10:53:37.244] Snapshot is not continuous with chain snaproot=202918..8576d9 chainroot=f03a4f..9efbcb
INFO [09-06|10:53:37.244] Loaded local transaction journal transactions=0 dropped=0
INFO [09-06|10:53:37.244] Regenerated local transaction journal transactions=0 accounts=0
WARN [09-06|10:53:37.244] Switch sync mode from snap sync to full sync
WARN [09-06|10:53:37.244] Chain pre-merge, sync via PoW (ensure beacon client is ready)
INFO [09-06|10:53:37.244] Gasprice oracle is ignoring threshold set threshold=2
WARN [09-06|10:53:37.244] Unclean shutdown detected booted=2022-09-04T04:58:17+0000 age=2d5h55m
WARN [09-06|10:53:37.244] Unclean shutdown detected booted=2022-09-05T14:03:57+0000 age=20h49m40s
WARN [09-06|10:53:37.244] Engine API enabled protocol=eth
INFO [09-06|10:53:37.245] Starting peer-to-peer node instance=Geth/v1.10.23-stable-d901d853/linux-amd64/go1.18.5
INFO [09-06|10:53:37.255] New local node record seq=1,662,041,897,684 id=cfcce68fd0196271 ip=127.0.0.1 udp=30303 tcp=30303
INFO [09-06|10:53:37.256] Started P2P networking self=enode://688e713339375261742a59c04465379649124b54113acb7b0c21313f670e95fa79ad1bd8018061f943249772d1094e864043169e502a6715011001eec38c9236@127.0.0.1:30303
INFO [09-06|10:53:37.256] IPC endpoint opened url=/home/asuka/geth/data/geth.ipc
INFO [09-06|10:53:37.256] Loaded JWT secret file path=/home/asuka/jwt.hex crc32=0xca4fe9b5
INFO [09-06|10:53:37.257] HTTP server started endpoint=192.168.1.101:8545 auth=false prefix= cors= vhosts=localhost
INFO [09-06|10:53:37.257] WebSocket enabled url=ws://127.0.0.1:8551
INFO [09-06|10:53:37.257] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost
INFO [09-06|10:53:41.468] New local node record seq=1,662,041,897,685 id=cfcce68fd0196271 ip=222.172.146.196 udp=30303 tcp=30303
INFO [09-06|10:53:47.635] Looking for peers peercount=1 tried=17 static=0
INFO [09-06|10:53:57.808] Looking for peers peercount=1 tried=27 static=0
INFO [09-06|10:54:07.849] Looking for peers peercount=3 tried=30 static=0
INFO [09-06|10:54:17.920] Looking for peers peercount=3 tried=24 static=0
WARN [09-06|10:54:18.071] Snapshot extension registration failed peer=e9cccd6e err="peer connected on snap without compatible eth support"
WARN [09-06|10:54:20.805] Snapshot extension registration failed peer=f343148b err="peer connected on snap without compatible eth support"
INFO [09-06|10:54:27.921] Looking for peers peercount=2 tried=29 static=0
Notable:
WARN [09-06|10:53:37.236] Enabling snapshot recovery chainhead=15,476,125 diskbase=15,478,127
WARN [09-06|10:53:37.244] Loaded snapshot journal diskroot=202918..8576d9 diffs=unmatched
WARN [09-06|10:53:37.244] Snapshot is not continuous with chain snaproot=202918..8576d9 chainroot=f03a4f..9efbcb
And chainhead is lower than diskbase, which looks odd to me.
@holiman It's already another restart, and I think the chainhead is already rewound to the historic block and it's expected that diskbase is higher.
ERROR[09-06|10:08:19.783] Unhandled trie error: missing trie node d91ac33fd2279f87f63926e7f8ad632730f0dfde5079c4888d36bd331e721eeb (path 080e0a030a0408) <nil>
nonce too high: address 0x524540a2D2E07c7dCf1EA6e0Bc82DCebC30a3656, tx: 5 state: 0
looks like your database is fired, one trie node is missing so the account nonce is interpreted as 0(empty account).
Ok, I will clear the data. Can I still keep the ancients? If so how to do it?
Can I still keep the ancients? If so how to do it?
Yes. geth --datadir ~/geth/data removedb, then first answer y and then n when it asks you if it should delete the ancients
Can I still keep the ancients? If so how to do it?
Yes.
geth --datadir ~/geth/data removedb, then first answeryand thennwhen it asks you if it should delete the ancients
Got it, thank you.
I'm having a similar issue. Is the cause the same?
https://gist.github.com/ardevd/6c2906db9606e51bba6ec38a10eae790
For the others with the same issue: If geth --datadir ~/geth/data removedb didn't help you, check your time settings and configure server time sync.
