go-ethereum
go-ethereum copied to clipboard
ethereum
Failed to commit trie from trie database err="read /ethereum/.ethereum/geth/chaindata/102907544.ldb: input/output error"
System information
Geth version: 1.10.23-stable
OS & Version: CentOS Linux release 8.3.2011
Commit hash : d901d85377c2c2f05f09f423c7d739c0feecd90a
Expected behaviour
Continued syncing
Actual behaviour
Always CRIT after restart
CRIT [09-06|19:14:34.310] Failed to delete frozen canonical blocks err="read /ethereum/.ethereum/geth/chaindata/102907544.ldb: input/output error"
Steps to reproduce the behaviour
nohup ./geth --syncmode=full --gcmode=archive --http --http.api=admin,debug,web3,eth,net,txpool --ws --ws.api=admin,debug,web3,eth,net,txpool --maxpeers=1000 --cache=51200 --datadir=/ethereum/.ethereum &
Backtrace
100 lines of head log
INFO [09-06|18:55:50.178] Starting Geth on Ethereum mainnet...
INFO [09-06|18:55:50.180] Maximum peer count ETH=1000 LES=0 total=1000
INFO [09-06|18:55:50.181] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [09-06|18:55:50.195] Disable transaction unindexing for archive node
WARN [09-06|18:55:50.196] Sanitizing cache to Go's GC limits provided=51200 updated=21300
INFO [09-06|18:55:50.198] Enabling recording of key preimages since archive mode is used
INFO [09-06|18:55:50.198] Set global gas cap cap=50,000,000
INFO [09-06|18:55:50.200] Allocated trie memory caches clean=6.24GiB dirty=0.00B
INFO [09-06|18:55:50.200] Allocated cache and file handles database=/ethereum/.ethereum/geth/chaindata cache=10.40GiB handles=131,072
INFO [09-06|18:57:32.301] Found legacy ancient chain path location=/ethereum/.ethereum/geth/chaindata/ancient
INFO [09-06|18:57:32.306] Opened ancient database database=/ethereum/.ethereum/geth/chaindata/ancient readonly=false
INFO [09-06|18:57:32.370]
INFO [09-06|18:57:32.370] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [09-06|18:57:32.370] Chain ID: 1 (mainnet)
INFO [09-06|18:57:32.370] Consensus: Beacon (proof-of-stake), merging from Ethash (proof-of-work)
INFO [09-06|18:57:32.370]
INFO [09-06|18:57:32.370] Pre-Merge hard forks:
INFO [09-06|18:57:32.370] - Homestead: 1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [09-06|18:57:32.370] - DAO Fork: 1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [09-06|18:57:32.370] - Tangerine Whistle (EIP 150): 2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [09-06|18:57:32.370] - Spurious Dragon/1 (EIP 155): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [09-06|18:57:32.370] - Spurious Dragon/2 (EIP 158): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [09-06|18:57:32.370] - Byzantium: 4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [09-06|18:57:32.370] - Constantinople: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [09-06|18:57:32.370] - Petersburg: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [09-06|18:57:32.370] - Istanbul: 9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [09-06|18:57:32.370] - Muir Glacier: 9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [09-06|18:57:32.370] - Berlin: 12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [09-06|18:57:32.370] - London: 12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [09-06|18:57:32.370] - Arrow Glacier: 13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [09-06|18:57:32.370] - Gray Glacier: 15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [09-06|18:57:32.370]
INFO [09-06|18:57:32.370] Merge configured:
INFO [09-06|18:57:32.370] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [09-06|18:57:32.371] - Network known to be merged: false
INFO [09-06|18:57:32.371] - Total terminal difficulty: 58750000000000000000000
INFO [09-06|18:57:32.371] - Merge netsplit block: <nil>
INFO [09-06|18:57:32.371] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [09-06|18:57:32.371]
INFO [09-06|18:58:20.189] Disk storage enabled for ethash caches dir=/ethereum/.ethereum/geth/ethash count=3
INFO [09-06|18:58:20.189] Disk storage enabled for ethash DAGs dir=/home/fengfeng/.ethash count=2
INFO [09-06|18:58:20.191] Initialising Ethereum protocol network=1 dbversion=8
INFO [09-06|18:58:26.060] Loaded most recent local header number=15,481,225 hash=ba6ea6..9bf6fe td=58,061,932,683,795,506,239,105 age=9h20m20s
INFO [09-06|18:58:26.060] Loaded most recent local full block number=15,481,225 hash=ba6ea6..9bf6fe td=58,061,932,683,795,506,239,105 age=9h20m20s
INFO [09-06|18:58:26.060] Loaded most recent local fast block number=15,481,225 hash=ba6ea6..9bf6fe td=58,061,932,683,795,506,239,105 age=9h20m20s
WARN [09-06|18:58:26.081] Loaded snapshot journal diskroot=b80ea3..612087 diffs=unmatched
WARN [09-06|18:58:26.081] Failed to load snapshot, regenerating err="head doesn't match snapshot: have 0xb80ea3eb3cb0b468ad6ee9a21e641dd72ee10390d9856bccb56813c2d7612087, want 0xfc655212b85c82a49e4e8ed1f23f029fcc197a5f7113b2dd15a787c4c755eb8b"
INFO [09-06|18:58:26.081] Rebuilding state snapshot
INFO [09-06|18:58:26.082] Resuming state snapshot generation root=fc6552..55eb8b accounts=0 slots=0 storage=0.00B dangling=0 elapsed="888.053µs"
INFO [09-06|18:58:26.082] Loaded local transaction journal transactions=0 dropped=0
INFO [09-06|18:58:26.082] Regenerated local transaction journal transactions=0 accounts=0
WARN [09-06|18:58:26.083] Chain pre-merge, sync via PoW (ensure beacon client is ready)
INFO [09-06|18:58:26.083] Gasprice oracle is ignoring threshold set threshold=2
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2021-06-22T17:48:47+0800 age=1y2mo3w
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2021-12-13T11:36:40+0800 age=8mo3w6d
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2021-12-23T09:53:37+0800 age=8mo2w3d
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-01-07T18:25:34+0800 age=8mo2d32m
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-03-23T15:46:18+0800 age=5mo2w3d
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-04-06T11:16:40+0800 age=5mo3d7h
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-05-30T10:27:58+0800 age=3mo1w2d
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-06-27T15:02:15+0800 age=2mo1w4d
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-08-01T05:45:55+0800 age=1mo6d13h
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-08-29T11:30:52+0800 age=1w1d7h
WARN [09-06|18:58:26.083] Unclean shutdown detected booted=2022-09-06T09:34:23+0800 age=9h24m3s
WARN [09-06|18:58:26.083] Engine API enabled protocol=eth
INFO [09-06|18:58:26.084] Starting peer-to-peer node instance=Geth/v1.10.23-stable-d901d853/linux-amd64/go1.18.5
INFO [09-06|18:58:26.331] New local node record seq=319 id=30ac3a1347f20730 ip=127.0.0.1 udp=30303 tcp=30303
INFO [09-06|18:58:26.331] Started P2P networking self=enode://a30921dcd7014ffdbdb4583dde28f81c6fa8abf193b4ce6f456982c2e8259f197c9a679fcfc95077e58bf39b66c81c8860eefa025cf4d372d54e1a8ad40d88a9@127.0.0.1:30303
INFO [09-06|18:58:26.332] IPC endpoint opened url=/ethereum/.ethereum/geth.ipc
INFO [09-06|18:58:26.333] Loaded JWT secret file path=/ethereum/.ethereum/geth/jwtsecret crc32=0xef6b53a4
INFO [09-06|18:58:26.333] HTTP server started endpoint=[::]:8545 auth=false prefix= cors= vhosts=localhost
INFO [09-06|18:58:26.333] WebSocket enabled url=ws://[::]:8546
INFO [09-06|18:58:26.334] WebSocket enabled url=ws://127.0.0.1:8551
INFO [09-06|18:58:26.334] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost
INFO [09-06|18:58:34.084] Generating state snapshot root=fc6552..55eb8b at=00348a..de3136 accounts=144,771 slots=278,374 storage=27.65MiB dangling=0 elapsed=8.003s eta=2h46m15.284s
INFO [09-06|18:58:36.502] Looking for peers peercount=0 tried=14 static=0
INFO [09-06|18:58:38.464] Block synchronisation started
WARN [09-06|18:58:38.465] Synchronisation failed, retrying err="peer is unknown or unhealthy"
INFO [09-06|18:58:42.093] Generating state snapshot root=fc6552..55eb8b in=0042a5..807326 at=b9dd8a..55e49d accounts=183,699 slots=953,682 storage=77.20MiB dangling=0 elapsed=16.012s eta=4h22m9.975s
INFO [09-06|18:58:46.503] Looking for peers peercount=0 tried=28 static=0
INFO [09-06|18:58:50.804] Generating state snapshot root=fc6552..55eb8b in=00696a..11a37d at=a5ce19..f90d72 accounts=290,802 slots=1,461,142 storage=118.48MiB dangling=0 elapsed=24.722s eta=4h15m45.077s
INFO [09-06|18:58:56.507] Looking for peers peercount=0 tried=27 static=0
INFO [09-06|18:58:58.807] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=055524..e12813 accounts=395,607 slots=1,842,520 storage=153.17MiB dangling=0 elapsed=32.726s eta=4h8m44.278s
WARN [09-06|18:59:01.120] Merge is configured, but no beacon client seen. Please ensure you have one available before the transition arrives! eta=1w2d5h
INFO [09-06|18:59:06.625] Looking for peers peercount=0 tried=29 static=0
INFO [09-06|18:59:06.837] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=2543c4..cdfb28 accounts=395,607 slots=2,614,616 storage=221.18MiB dangling=0 elapsed=40.755s eta=5h9m45.69s
INFO [09-06|18:59:14.879] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=478025..34bb01 accounts=395,607 slots=3,440,984 storage=293.98MiB dangling=0 elapsed=48.798s eta=6h10m53.487s
INFO [09-06|18:59:16.627] Looking for peers peercount=1 tried=35 static=0
INFO [09-06|18:59:22.883] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=71e5d9..bfb0c7 accounts=395,607 slots=4,463,960 storage=384.11MiB dangling=0 elapsed=56.801s eta=7h11m43.042s
INFO [09-06|18:59:26.716] Looking for peers peercount=1 tried=33 static=0
INFO [09-06|18:59:30.885] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=9c42a9..a424b2 accounts=395,607 slots=5,486,936 storage=474.24MiB dangling=0 elapsed=1m4.804s eta=8h12m32.598s
INFO [09-06|18:59:36.825] Looking for peers peercount=0 tried=25 static=0
INFO [09-06|18:59:38.889] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=c70ec5..9343e5 accounts=395,607 slots=6,520,152 storage=565.28MiB dangling=0 elapsed=1m12.807s eta=9h13m22.153s
INFO [09-06|18:59:46.893] Generating state snapshot root=fc6552..55eb8b in=008f65..483efb at=f22ba9..7f0be2 accounts=395,607 slots=7,558,489 storage=656.77MiB dangling=0 elapsed=1m20.812s eta=10h14m12.62s
INFO [09-06|18:59:47.047] Looking for peers peercount=1 tried=29 static=0
INFO [09-06|18:59:48.882] Downloader queue stats receiptTasks=0 blockTasks=375 itemSize=5.77KiB throttle=8192
INFO [09-06|18:59:48.903] Importing sidechain segment start=15,481,226 end=15,481,227
INFO [09-06|18:59:50.454] Imported new chain segment blocks=2 txs=537 mgas=35.273 elapsed=1.551s mgasps=22.742 number=15,481,227 hash=95d8e7..78db68 age=9h21m14s dirty=0.00B
INFO [09-06|18:59:54.894] Generating state snapshot root=fc6552..55eb8b at=00ad6d..2b7df8 accounts=478,729 slots=8,117,422 storage=706.69MiB dangling=0 elapsed=1m28.812s eta=9h17m52.252s
INFO [09-06|18:59:56.155] Aborting state snapshot generation root=fc6552..55eb8b at=00b85c..7bcc40 accounts=508,809 slots=8,165,373 storage=711.65MiB dangling=0 elapsed=1m30.073s eta=8h52m9.029s
INFO [09-06|18:59:56.155] Resuming state snapshot generation root=24eaf7..2a4b7a at=00b85c..7bcc40 accounts=508,809 slots=8,165,373 storage=711.65MiB dangling=0 elapsed=1m30.073s eta=8h52m9.029s
INFO [09-06|18:59:56.564] Aborting state snapshot generation root=24eaf7..2a4b7a at=00bbb1..da6c73 accounts=518,154 slots=8,177,270 storage=712.99MiB dangling=0 elapsed=1m30.482s eta=8h45m2.869s
INFO [09-06|18:59:56.564] Resuming state snapshot generation root=182177..ffd94f at=00bbb1..da6c73 accounts=518,154 slots=8,177,270 storage=712.99MiB dangling=0 elapsed=1m3
100 lines of tail log
INFO [09-06|19:14:05.785] Aborting state snapshot generation root=56289b..65d778 at=0f767b..1dd5a9 accounts=10,935,116 slots=57,924,603 storage=4.71GiB dangling=0 elapsed=15m39.704s eta=4h3m37.903s
INFO [09-06|19:14:05.786] Resuming state snapshot generation root=3e9635..141874 at=0f767b..1dd5a9 accounts=10,935,116 slots=57,924,603 storage=4.71GiB dangling=0 elapsed=15m39.704s eta=4h3m37.903s
INFO [09-06|19:14:06.029] Aborting state snapshot generation root=3e9635..141874 at=0f7769..1ec1e2 accounts=10,937,677 slots=57,925,734 storage=4.71GiB dangling=0 elapsed=15m39.947s eta=4h3m38.022s
INFO [09-06|19:14:06.033] Resuming state snapshot generation root=135aa3..98cd44 at=0f7769..1ec1e2 accounts=10,937,677 slots=57,925,734 storage=4.71GiB dangling=0 elapsed=15m39.952s eta=4h3m38.099s
INFO [09-06|19:14:06.076] Imported new chain segment blocks=13 txs=2671 mgas=209.806 elapsed=8.171s mgasps=25.675 number=15,482,882 hash=6edee5..53b6ac age=3h10m37s dirty=0.00B
INFO [09-06|19:14:06.107] Aborting state snapshot generation root=135aa3..98cd44 at=0f77a4..3e2cf9 accounts=10,938,318 slots=57,925,870 storage=4.71GiB dangling=0 elapsed=15m40.025s eta=4h3m38.335s
INFO [09-06|19:14:06.108] Resuming state snapshot generation root=fc21ad..364ce0 at=0f77a4..3e2cf9 accounts=10,938,318 slots=57,925,870 storage=4.71GiB dangling=0 elapsed=15m40.027s eta=4h3m38.366s
INFO [09-06|19:14:06.163] Aborting state snapshot generation root=fc21ad..364ce0 at=0f77c6..3757f3 accounts=10,938,703 slots=57,925,982 storage=4.71GiB dangling=0 elapsed=15m40.081s eta=4h3m38.684s
INFO [09-06|19:14:06.163] Resuming state snapshot generation root=0e6872..0ba33c at=0f77c6..3757f3 accounts=10,938,703 slots=57,925,982 storage=4.71GiB dangling=0 elapsed=15m40.082s eta=4h3m38.699s
INFO [09-06|19:14:06.846] Aborting state snapshot generation root=0e6872..0ba33c at=0f7c25..2fc1d0 accounts=10,950,740 slots=57,940,028 storage=4.72GiB dangling=0 elapsed=15m40.765s eta=4h3m32.144s
INFO [09-06|19:14:06.847] Resuming state snapshot generation root=b75bdc..ce5a98 at=0f7c25..2fc1d0 accounts=10,950,740 slots=57,940,028 storage=4.72GiB dangling=0 elapsed=15m40.765s eta=4h3m32.144s
INFO [09-06|19:14:07.133] Aborting state snapshot generation root=b75bdc..ce5a98 in=0f7e66..196c35 at=045c47..5dd3ee accounts=10,956,987 slots=57,945,111 storage=4.72GiB dangling=0 elapsed=15m41.051s eta=4h3m27.758s
INFO [09-06|19:14:07.133] Resuming state snapshot generation root=587eee..253b6b in=0f7e66..196c35 at=045c47..5dd3ee accounts=10,956,987 slots=57,945,111 storage=4.72GiB dangling=0 elapsed=15m41.052s eta=4h3m27.773s
INFO [09-06|19:14:07.262] Aborting state snapshot generation root=587eee..253b6b in=0f7e66..196c35 at=3ca9fb..9dcd77 accounts=10,956,987 slots=57,958,424 storage=4.72GiB dangling=0 elapsed=15m41.181s eta=4h3m29.776s
INFO [09-06|19:14:07.264] Resuming state snapshot generation root=d6e47a..a28efd in=0f7e66..196c35 at=3ca9fb..9dcd77 accounts=10,956,987 slots=57,958,424 storage=4.72GiB dangling=0 elapsed=15m41.183s eta=4h3m29.807s
INFO [09-06|19:14:08.121] Aborting state snapshot generation root=d6e47a..a28efd at=0f80c4..cfa339 accounts=10,963,519 slots=58,032,366 storage=4.72GiB dangling=0 elapsed=15m42.039s eta=4h3m33.809s
INFO [09-06|19:14:08.121] Resuming state snapshot generation root=93a7a1..d83261 at=0f80c4..cfa339 accounts=10,963,519 slots=58,032,366 storage=4.72GiB dangling=0 elapsed=15m42.039s eta=4h3m33.809s
INFO [09-06|19:14:09.057] Aborting state snapshot generation root=93a7a1..d83261 in=0f86e8..b3ae47 at=10b7c0..e4ac9a accounts=10,980,481 slots=58,078,436 storage=4.73GiB dangling=0 elapsed=15m42.975s eta=4h3m24.272s
INFO [09-06|19:14:09.057] Resuming state snapshot generation root=84c6ba..f26297 in=0f86e8..b3ae47 at=10b7c0..e4ac9a accounts=10,980,481 slots=58,078,436 storage=4.73GiB dangling=0 elapsed=15m42.976s eta=4h3m24.287s
INFO [09-06|19:14:09.841] Aborting state snapshot generation root=84c6ba..f26297 at=0f8b11..c09b45 accounts=10,992,133 slots=58,112,785 storage=4.73GiB dangling=0 elapsed=15m43.760s eta=4h3m20.152s
INFO [09-06|19:14:09.841] Resuming state snapshot generation root=fe7cab..b25406 at=0f8b11..c09b45 accounts=10,992,133 slots=58,112,785 storage=4.73GiB dangling=0 elapsed=15m43.760s eta=4h3m20.152s
INFO [09-06|19:14:10.006] Aborting state snapshot generation root=fe7cab..b25406 at=0f8c74..5120c2 accounts=10,995,975 slots=58,116,590 storage=4.73GiB dangling=0 elapsed=15m43.924s eta=4h3m17.272s
INFO [09-06|19:14:10.006] Resuming state snapshot generation root=a1ce5c..74ebc7 at=0f8c74..5120c2 accounts=10,995,975 slots=58,116,590 storage=4.73GiB dangling=0 elapsed=15m43.925s eta=4h3m17.288s
INFO [09-06|19:14:10.888] Aborting state snapshot generation root=a1ce5c..74ebc7 at=0f93f1..a92441 accounts=11,016,843 slots=58,143,641 storage=4.73GiB dangling=0 elapsed=15m44.806s eta=4h3m1.704s
INFO [09-06|19:14:10.888] Resuming state snapshot generation root=6b2ed1..f28e1f at=0f93f1..a92441 accounts=11,016,843 slots=58,143,641 storage=4.73GiB dangling=0 elapsed=15m44.806s eta=4h3m1.704s
INFO [09-06|19:14:11.168] Aborting state snapshot generation root=6b2ed1..f28e1f in=0f9621..488dc8 at=09e12c..037fa2 accounts=11,022,854 slots=58,151,028 storage=4.73GiB dangling=0 elapsed=15m45.086s eta=4h2m57.51s
INFO [09-06|19:14:11.168] Resuming state snapshot generation root=ae0822..acdaef in=0f9621..488dc8 at=09e12c..037fa2 accounts=11,022,854 slots=58,151,028 storage=4.73GiB dangling=0 elapsed=15m45.087s eta=4h2m57.525s
INFO [09-06|19:14:11.814] Aborting state snapshot generation root=ae0822..acdaef in=0f9621..488dc8 at=b16f4f..adc6e3 accounts=11,022,854 slots=58,202,229 storage=4.74GiB dangling=0 elapsed=15m45.732s eta=4h3m7.474s
INFO [09-06|19:14:11.814] Resuming state snapshot generation root=ee7e52..49a856 in=0f9621..488dc8 at=b16f4f..adc6e3 accounts=11,022,854 slots=58,202,229 storage=4.74GiB dangling=0 elapsed=15m45.733s eta=4h3m7.489s
INFO [09-06|19:14:11.915] Aborting state snapshot generation root=ee7e52..49a856 in=0f9621..488dc8 at=cc7123..1d1971 accounts=11,022,854 slots=58,210,422 storage=4.74GiB dangling=0 elapsed=15m45.833s eta=4h3m9.032s
INFO [09-06|19:14:11.915] Resuming state snapshot generation root=8426e7..3767dc in=0f9621..488dc8 at=cc7123..1d1971 accounts=11,022,854 slots=58,210,422 storage=4.74GiB dangling=0 elapsed=15m45.834s eta=4h3m9.047s
INFO [09-06|19:14:12.174] Aborting state snapshot generation root=8426e7..3767dc in=0f9722..faf90d at=290dec..f3e563 accounts=11,025,617 slots=58,228,053 storage=4.74GiB dangling=0 elapsed=15m46.093s eta=4h3m9.138s
INFO [09-06|19:14:12.175] Resuming state snapshot generation root=ee6d98..c6e741 in=0f9722..faf90d at=290dec..f3e563 accounts=11,025,617 slots=58,228,053 storage=4.74GiB dangling=0 elapsed=15m46.094s eta=4h3m9.154s
INFO [09-06|19:14:12.303] Aborting state snapshot generation root=ee6d98..c6e741 at=0f9859..38ea0b accounts=11,028,946 slots=58,229,033 storage=4.74GiB dangling=0 elapsed=15m46.221s eta=4h3m6.393s
INFO [09-06|19:14:12.304] Resuming state snapshot generation root=badb57..8b5ac9 at=0f9859..38ea0b accounts=11,028,946 slots=58,229,033 storage=4.74GiB dangling=0 elapsed=15m46.222s eta=4h3m6.408s
INFO [09-06|19:14:12.554] Aborting state snapshot generation root=badb57..8b5ac9 at=0f9aa2..0a6230 accounts=11,035,348 slots=58,233,501 storage=4.74GiB dangling=0 elapsed=15m46.472s eta=4h3m1.367s
INFO [09-06|19:14:12.554] Resuming state snapshot generation root=be25d3..a7c404 at=0f9aa2..0a6230 accounts=11,035,348 slots=58,233,501 storage=4.74GiB dangling=0 elapsed=15m46.473s eta=4h3m1.382s
INFO [09-06|19:14:13.152] Aborting state snapshot generation root=be25d3..a7c404 at=0fa0a0..688c14 accounts=11,051,862 slots=58,240,956 storage=4.74GiB dangling=0 elapsed=15m47.070s eta=4h2m47.311s
INFO [09-06|19:14:13.152] Resuming state snapshot generation root=f932b7..0435b0 at=0fa0a0..688c14 accounts=11,051,862 slots=58,240,956 storage=4.74GiB dangling=0 elapsed=15m47.070s eta=4h2m47.311s
INFO [09-06|19:14:13.791] Aborting state snapshot generation root=f932b7..0435b0 at=0fa5d1..cd79e8 accounts=11,066,103 slots=58,252,850 storage=4.74GiB dangling=0 elapsed=15m47.710s eta=4h2m37.038s
INFO [09-06|19:14:13.792] Resuming state snapshot generation root=04f59f..74459a at=0fa5d1..cd79e8 accounts=11,066,103 slots=58,252,850 storage=4.74GiB dangling=0 elapsed=15m47.710s eta=4h2m37.038s
INFO [09-06|19:14:13.899] Aborting state snapshot generation root=04f59f..74459a at=0fa68b..fcd6d3 accounts=11,068,024 slots=58,255,395 storage=4.74GiB dangling=0 elapsed=15m47.818s eta=4h2m35.885s
INFO [09-06|19:14:13.900] Resuming state snapshot generation root=a427d6..48eb44 at=0fa68b..fcd6d3 accounts=11,068,024 slots=58,255,395 storage=4.74GiB dangling=0 elapsed=15m47.818s eta=4h2m35.885s
INFO [09-06|19:14:14.666] Aborting state snapshot generation root=a427d6..48eb44 at=0fae27..cefb60 accounts=11,088,765 slots=58,264,809 storage=4.75GiB dangling=0 elapsed=15m48.585s eta=4h2m18.24s
INFO [09-06|19:14:14.667] Resuming state snapshot generation root=440c5a..daf3a8 at=0fae27..cefb60 accounts=11,088,765 slots=58,264,809 storage=4.75GiB dangling=0 elapsed=15m48.585s eta=4h2m18.24s
INFO [09-06|19:14:14.827] Imported new chain segment blocks=20 txs=2930 mgas=295.525 elapsed=8.750s mgasps=33.771 number=15,482,902 hash=be5e5a..39e6b6 age=3h8m1s dirty=0.00B
INFO [09-06|19:14:15.571] Aborting state snapshot generation root=440c5a..daf3a8 at=0fb308..6a5e3b accounts=11,102,463 slots=58,312,798 storage=4.75GiB dangling=0 elapsed=15m49.489s eta=4h2m13.285s
INFO [09-06|19:14:15.571] Resuming state snapshot generation root=01ec58..8e01f0 at=0fb308..6a5e3b accounts=11,102,463 slots=58,312,798 storage=4.75GiB dangling=0 elapsed=15m49.490s eta=4h2m13.3s
INFO [09-06|19:14:15.837] Aborting state snapshot generation root=01ec58..8e01f0 in=0fb3ce..166cb2 at=7bb450..83a41b accounts=11,104,511 slots=58,344,319 storage=4.75GiB dangling=0 elapsed=15m49.756s eta=4h2m14.383s
INFO [09-06|19:14:15.838] Resuming state snapshot generation root=9e5ffe..d940b3 in=0fb3ce..166cb2 at=7bb450..83a41b accounts=11,104,511 slots=58,344,319 storage=4.75GiB dangling=0 elapsed=15m49.756s eta=4h2m14.383s
INFO [09-06|19:14:16.605] Aborting state snapshot generation root=9e5ffe..d940b3 at=0fb65b..289d33 accounts=11,111,527 slots=58,394,018 storage=4.76GiB dangling=0 elapsed=15m50.523s eta=4h2m16.299s
INFO [09-06|19:14:16.606] Resuming state snapshot generation root=1007a9..cbc4d1 at=0fb65b..289d33 accounts=11,111,527 slots=58,394,018 storage=4.76GiB dangling=0 elapsed=15m50.524s eta=4h2m16.314s
INFO [09-06|19:14:17.844] Aborting state snapshot generation root=1007a9..cbc4d1 at=0fc0aa..9ab0ee accounts=11,140,203 slots=58,428,781 storage=4.76GiB dangling=0 elapsed=15m51.763s eta=4h1m55.619s
INFO [09-06|19:14:17.845] Resuming state snapshot generation root=36c6ec..57962c at=0fc0aa..9ab0ee accounts=11,140,203 slots=58,428,781 storage=4.76GiB dangling=0 elapsed=15m51.763s eta=4h1m55.619s
INFO [09-06|19:14:18.045] Aborting state snapshot generation root=36c6ec..57962c at=0fc279..fe6021 accounts=11,145,198 slots=58,430,661 storage=4.76GiB dangling=0 elapsed=15m51.963s eta=4h1m51.745s
INFO [09-06|19:14:18.046] Resuming state snapshot generation root=0769d9..25d567 at=0fc279..fe6021 accounts=11,145,198 slots=58,430,661 storage=4.76GiB dangling=0 elapsed=15m51.964s eta=4h1m51.76s
INFO [09-06|19:14:18.969] Aborting state snapshot generation root=0769d9..25d567 at=0fc6e3..d6eeed accounts=11,157,536 slots=58,507,078 storage=4.77GiB dangling=0 elapsed=15m52.887s eta=4h1m48.906s
INFO [09-06|19:14:18.970] Resuming state snapshot generation root=e148f1..6592d1 at=0fc6e3..d6eeed accounts=11,157,536 slots=58,507,078 storage=4.77GiB dangling=0 elapsed=15m52.888s eta=4h1m48.921s
INFO [09-06|19:14:19.918] Aborting state snapshot generation root=e148f1..6592d1 at=0fcf3e..9e56c6 accounts=11,180,323 slots=58,535,233 storage=4.77GiB dangling=0 elapsed=15m53.836s eta=4h1m31.398s
INFO [09-06|19:14:19.918] Resuming state snapshot generation root=7d91ac..a4d17f at=0fcf3e..9e56c6 accounts=11,180,323 slots=58,535,233 storage=4.77GiB dangling=0 elapsed=15m53.837s eta=4h1m31.413s
INFO [09-06|19:14:20.699] Aborting state snapshot generation root=7d91ac..a4d17f in=0fd2fb..6e0b62 at=da7542..4f1ba1 accounts=11,190,525 slots=58,580,292 storage=4.77GiB dangling=0 elapsed=15m54.618s eta=4h1m29.017s
INFO [09-06|19:14:20.700] Resuming state snapshot generation root=64825d..ae8693 in=0fd2fb..6e0b62 at=da7542..4f1ba1 accounts=11,190,525 slots=58,580,292 storage=4.77GiB dangling=0 elapsed=15m54.619s eta=4h1m29.032s
INFO [09-06|19:14:21.061] Aborting state snapshot generation root=64825d..ae8693 at=0fd446..2f9aae accounts=11,194,111 slots=58,605,141 storage=4.77GiB dangling=0 elapsed=15m54.979s eta=4h1m29.562s
INFO [09-06|19:14:21.061] Resuming state snapshot generation root=64fc8a..a660f6 at=0fd446..2f9aae accounts=11,194,111 slots=58,605,141 storage=4.77GiB dangling=0 elapsed=15m54.980s eta=4h1m29.577s
INFO [09-06|19:14:21.231] Aborting state snapshot generation root=64fc8a..a660f6 at=0fd5b7..16e026 accounts=11,198,208 slots=58,609,590 storage=4.78GiB dangling=0 elapsed=15m55.149s eta=4h1m26.65s
INFO [09-06|19:14:21.231] Resuming state snapshot generation root=d283ce..e6efca at=0fd5b7..16e026 accounts=11,198,208 slots=58,609,590 storage=4.78GiB dangling=0 elapsed=15m55.150s eta=4h1m26.665s
INFO [09-06|19:14:21.903] Aborting state snapshot generation root=d283ce..e6efca in=0fdada..aebdbe at=290dec..f3e563 accounts=11,212,482 slots=58,615,867 storage=4.78GiB dangling=0 elapsed=15m55.822s eta=4h1m17.306s
INFO [09-06|19:14:21.904] Resuming state snapshot generation root=0b4c9b..2680eb in=0fdada..aebdbe at=290dec..f3e563 accounts=11,212,482 slots=58,615,867 storage=4.78GiB dangling=0 elapsed=15m55.822s eta=4h1m17.306s
INFO [09-06|19:14:22.616] Aborting state snapshot generation root=0b4c9b..2680eb in=0fddd3..2a81e2 at=290dec..f3e563 accounts=11,220,804 slots=58,663,773 storage=4.78GiB dangling=0 elapsed=15m56.535s eta=4h1m16.81s
INFO [09-06|19:14:22.618] Resuming state snapshot generation root=30c16f..7895de in=0fddd3..2a81e2 at=290dec..f3e563 accounts=11,220,804 slots=58,663,773 storage=4.78GiB dangling=0 elapsed=15m56.536s eta=4h1m16.825s
INFO [09-06|19:14:22.747] Aborting state snapshot generation root=30c16f..7895de at=0fdeb1..934471 accounts=11,223,237 slots=58,666,731 storage=4.78GiB dangling=0 elapsed=15m56.665s eta=4h1m15.483s
INFO [09-06|19:14:22.747] Resuming state snapshot generation root=c4a2d7..bc16f9 at=0fdeb1..934471 accounts=11,223,237 slots=58,666,731 storage=4.78GiB dangling=0 elapsed=15m56.666s eta=4h1m15.498s
INFO [09-06|19:14:22.941] Aborting state snapshot generation root=c4a2d7..bc16f9 at=0fe026..eb2b1e accounts=11,227,334 slots=58,678,265 storage=4.78GiB dangling=0 elapsed=15m56.859s eta=4h1m12.884s
INFO [09-06|19:14:22.942] Resuming state snapshot generation root=2aef38..92f589 at=0fe026..eb2b1e accounts=11,227,334 slots=58,678,265 storage=4.78GiB dangling=0 elapsed=15m56.860s eta=4h1m12.9s
ERROR[09-06|19:14:22.965] Failed to commit trie from trie database err="read /ethereum/.ethereum/geth/chaindata/102907544.ldb: input/output error"
ERROR[09-06|19:14:22.966] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"
WARN [09-06|19:14:22.975] Synchronisation failed, dropping peer peer=3fe6f503fc7ff1e3ea2fcfa30d7f2ae5adb61f478e43e5f42165c562e5d54926 err="action from bad peer ignored: header request failed: timeout"
WARN [09-06|19:14:29.889] Snapshot extension registration failed peer=925ab3ac err="peer connected on snap without compatible eth support"
INFO [09-06|19:14:30.948] Generating state snapshot root=2aef38..92f589 in=1019a1..0c370e at=02555f..aea511 accounts=11,386,896 slots=59,008,393 storage=4.81GiB dangling=0 elapsed=16m4.866s eta=3h59m37.004s
CRIT [09-06|19:14:34.310] Failed to delete frozen canonical blocks err="read /ethereum/.ethereum/geth/chaindata/102907544.ldb: input/output error"
disk usage
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 227M 31G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/cl-root 70G 7.5G 63G 11% /
/dev/sda1 1014M 322M 693M 32% /boot
/dev/mapper/cl-home 792G 25G 767G 4% /home
/dev/sdc1 19T 12T 6.0T 66% /ethereum
tmpfs 6.3G 16K 6.3G 1% /run/user/42
tmpfs 6.3G 0 6.3G 0% /run/user/1002
tmpfs 6.3G 4.0K 6.3G 1% /run/user/1001
CRIT [09-06|19:14:34.310] Failed to delete frozen canonical blocks err="read /ethereum/.ethereum/geth/chaindata/102907544.ldb: input/output error"
Did you unplug your external device? It's an error thrown by Leveldb and we can do nothing here.
CRIT [09-06|19:14:34.310] Failed to delete frozen canonical blocks err="read /ethereum/.ethereum/geth/chaindata/102907544.ldb: input/output error"Did you unplug your external device? It's an error thrown by Leveldb and we can do nothing here.
As far as I know, no one has touched the server recently.
Yeah, it's some error from OS, we can't do too much here. Maybe you can try to restart your node, re-mount your device, or whatever. Theoretically Geth can recover from it.
Yeah, it's some error from OS, we can't do too much here. Maybe you can try to restart your node, re-mount your device, or whatever. Theoretically Geth can recover from it.
Can I try to call debug_setHead to fix it? I don't want to resync block data because my node is an archive node.
Can you try restart first? If possible please post some logs here.
And you can definitely try to use debug_setHead and rewind your chain before this "buggy block". But I will still recommend to restart first.
Can you try restart first? If possible please post some logs here.
And you can definitely try to use
debug_setHeadand rewind your chain before this "buggy block". But I will still recommend to restart first.
I have restart geth, will update the log here later.
Can you try restart first? If possible please post some logs here.
And you can definitely try to use
debug_setHeadand rewind your chain before this "buggy block". But I will still recommend to restart first.
After restarting the node, it is still CRIT.
geth.log is the complete running log
Not much we can do here. The database internally is fried. If you can confirm that mounts and permissions are correct, then the only fault I can imagine is a drive error. You should probably check what smartctl or other tools say about the health of the SSD.
Not much we can do here. The database internally is fried. If you can confirm that mounts and permissions are correct, then the only fault I can imagine is a drive error. You should probably check what
smartctlor other tools say about the health of the SSD.
I'm pretty sure the permissions are correct, I've had the archive node running smoothly for over a year now, and just the other day my archive node had this problem without warning.
Seems like this issue is stale now, have you checked the smart values on your ssd?
