erigon
erigon copied to clipboard
Published
20 hours ago •
erigontech
erigontech
Erigon halted
System information
Erigon version: git_branch=devel git_tag=v2021.10.03-1907-ga9bde213d-dirty
OS & Version: Linux
Commit hash : a9bde213d9537bdae31e2a4a7427acf45be8481d
Network: mainnet
Erigon suddenly halted and could not recover. Restarting resulted in the same error and logs. Reverting to 2c1d617b41d626642a7cba5d5d2c495a516cfd31 fixed the issue.
[INFO] [09-23|00:34:01.420] Commit cycle in=1.835843ms
[INFO] [09-23|00:34:01.420] RPC Daemon notified of new headers from=15590987 to=15590991 header sending=6.212µs log sending=140ns
[INFO] [09-23|00:34:01.420] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-23|00:34:01.549] [2/16 Headers] New payload: need to download parent height=15590993 hash=0xe5df6381f2c1d86b82b6b4ee64ab6e156640346d033266facd86f86dba6703d9 parentHash=0x15928dfb8a58015ff88aaf8c45fc4062eb52a1ce1a8fc4374b058d721c82120e
[INFO] [09-23|00:34:01.549] [2/16 Headers] Downloading PoS headers... height=15590992 hash=0x15928dfb8a58015ff88aaf8c45fc4062eb52a1ce1a8fc4374b058d721c82120e requestId=1119
[WARN] [09-23|00:34:01.567] Could not validate block err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:253 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:657 collector.go:276 collector.go:144 stage_headers.go:670 stage_headers.go:469 stage_headers.go:202 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]"
[WARN] [09-23|00:34:01.576] Removing beacon request due to err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:253 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:657 collector.go:276 collector.go:144 stage_headers.go:670 stage_headers.go:469 stage_headers.go:202 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]" requestId=1119
[INFO] [09-23|00:34:01.576] Timings (slower than 50ms) Headers=156ms
[INFO] [09-23|00:34:01.576] Tables PlainState=74.3GB AccountChangeSet=163.9GB StorageChangeSet=242.3GB BlockTransaction=160.6GB TransactionLog=451.2GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-23|00:34:01.576] Commit cycle in=4.438µs
[INFO] [09-23|00:34:01.576] RPC Daemon notified of new headers from=15590987 to=15590991 header sending=7.765µs log sending=150ns
[INFO] [09-23|00:34:01.576] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-23|00:34:01.577] Commit cycle in=5.651µs
[INFO] [09-23|00:34:01.577] RPC Daemon notified of new headers from=15590987 to=15590991 header sending=12.805µs log sending=200ns
[INFO] [09-23|00:34:01.577] [2/16 Headers] Waiting for Consensus Layer...
[WARN] [09-23|00:34:01.664] [NewPayload] Previously known bad block hash=0xacab5c2e2c89d38eca25a6151a20f0e17d5d802d2c492a5a1a2a8ce6526ce8ce parentHash=0xe5df6381f2c1d86b82b6b4ee64ab6e156640346d033266facd86f86dba6703d9
[WARN] [09-23|00:34:01.892] [NewPayload] Previously known bad block hash=0x27c3b41e4f5d4a1dfe3191f8e2f1378914470575b1784c8506560e2c38bd38de parentHash=0xacab5c2e2c89d38eca25a6151a20f0e17d5d802d2c492a5a1a2a8ce6526ce8ce
[WARN] [09-23|00:34:02.049] [NewPayload] Previously known bad block hash=0x512358749152b02b4c6893b8c1efb7e6493d05482796e07c1a5ceed60d824f96 parentHash=0x27c3b41e4f5d4a1dfe3191f8e2f1378914470575b1784c8506560e2c38bd38de
[WARN] [09-23|00:34:02.225] [NewPayload] Previously known bad block hash=0x087cb674f969f64344c29964530ee904d4f548dca0038bbabaf01654712c37ca parentHash=0x512358749152b02b4c6893b8c1efb7e6493d05482796e07c1a5ceed60d824f96
[WARN] [09-23|00:34:02.356] [ForkChoiceUpdated] Previously known bad block hash=0x087cb674f969f64344c29964530ee904d4f548dca0038bbabaf01654712c37ca
[WARN] [09-23|00:34:02.519] [NewPayload] Previously known bad block hash=0x26a681cb3c8f43a46a4c032a8b9a6837d0b83e87b887c20ad9c9a44b9ad03eaf parentHash=0x087cb674f969f64344c29964530ee904d4f548dca0038bbabaf01654712c37ca
[WARN] [09-23|00:34:02.708] [ForkChoiceUpdated] Previously known bad block hash=0x26a681cb3c8f43a46a4c032a8b9a6837d0b83e87b887c20ad9c9a44b9ad03eaf
[WARN] [09-23|00:34:12.323] [NewPayload] Previously known bad block hash=0xbb2a2db7ba05f55d0baa586b083a91d318f4f787ac9702ecf4028897cf08fd6f parentHash=0x26a681cb3c8f43a46a4c032a8b9a6837d0b83e87b887c20ad9c9a44b9ad03eaf
[WARN] [09-23|00:34:12.429] [ForkChoiceUpdated] Previously known bad block hash=0xbb2a2db7ba05f55d0baa586b083a91d318f4f787ac9702ecf4028897cf08fd6f
[WARN] [09-23|00:34:24.640] [NewPayload] Previously known bad block hash=0xc321c414678e76d15ac9661dbd397b71d82ffb6cc60fcf7ca3b58ee3d8d57db3 parentHash=0xbb2a2db7ba05f55d0baa586b083a91d318f4f787ac9702ecf4028897cf08fd6f
[WARN] [09-23|00:34:24.816] [ForkChoiceUpdated] Previously known bad block hash=0xc321c414678e76d15ac9661dbd397b71d82ffb6cc60fcf7ca3b58ee3d8d57db3
[INFO] [09-23|00:34:26.475] [p2p] GoodPeers eth66=4
[INFO] [09-23|00:34:26.837] [txpool] stat block=15590991 pending=384 baseFee=30000 queued=30000 alloc=2.4GB sys=3.8GB
[WARN] [09-23|00:34:37.385] [NewPayload] Previously known bad block hash=0x63141ed8da942e446fbab6ac8befa2fda061f4af5c725f54dc2abea8633c764d parentHash=0xc321c414678e76d15ac9661dbd397b71d82ffb6cc60fcf7ca3b58ee3d8d57db3
[WARN] [09-23|00:34:37.488] [ForkChoiceUpdated] Previously known bad block hash=0x63141ed8da942e446fbab6ac8befa2fda061f4af5c725f54dc2abea8633c764d
[WARN] [09-23|00:34:46.513] [ForkChoiceUpdated] Previously known bad block hash=0x63141ed8da942e446fbab6ac8befa2fda061f4af5c725f54dc2abea8633c764d
[WARN] [09-23|00:34:48.647] [NewPayload] Previously known bad block hash=0xc134cf8896d1b6c4e7e59574088f4d3698b46057fb90a59ec6d087ed681d7042 parentHash=0x63141ed8da942e446fbab6ac8befa2fda061f4af5c725f54dc2abea8633c764d
[WARN] [09-23|00:34:49.231] [ForkChoiceUpdated] Previously known bad block hash=0xc134cf8896d1b6c4e7e59574088f4d3698b46057fb90a59ec6d087ed681d7042
[WARN] [09-23|00:35:00.810] [NewPayload] Previously known bad block hash=0x6216e5b100210ba42d3b1b812f7397bf8c3af32cf2998b4da53b4f291e36ef27 parentHash=0xc134cf8896d1b6c4e7e59574088f4d3698b46057fb90a59ec6d087ed681d7042
[WARN] [09-23|00:35:00.975] [ForkChoiceUpdated] Previously known bad block hash=0x6216e5b100210ba42d3b1b812f7397bf8c3af32cf2998b4da53b4f291e36ef27
[WARN] [09-23|00:35:12.564] [NewPayload] Previously known bad block hash=0xf01177831bea2376a2fdd2b5e59c66b61b54a618d5477be2ec88407c62700cd3 parentHash=0x6216e5b100210ba42d3b1b812f7397bf8c3af32cf2998b4da53b4f291e36ef27
[WARN] [09-23|00:35:12.739] [ForkChoiceUpdated] Previously known bad block hash=0xf01177831bea2376a2fdd2b5e59c66b61b54a618d5477be2ec88407c62700cd3
[WARN] [09-23|00:35:23.690] [NewPayload] Previously known bad block hash=0x37ce7f00b3e611e1d056c2c217c30fb9d3566e85f2f38ea6191be4959aed1c53 parentHash=0xf01177831bea2376a2fdd2b5e59c66b61b54a618d5477be2ec88407c62700cd3
[WARN] [09-23|00:35:23.754] [ForkChoiceUpdated] Previously known bad block hash=0x37ce7f00b3e611e1d056c2c217c30fb9d3566e85f2f38ea6191be4959aed1c53
[INFO] [09-23|00:35:27.478] [txpool] stat block=15590991 pending=827 baseFee=30000 queued=30000 alloc=2.3GB sys=3.8GB
[INFO] [09-23|00:36:26.475] [p2p] GoodPeers eth66=3
[INFO] [09-23|00:36:26.838] [txpool] stat block=15590991 pending=877 baseFee=30000 queued=30000 alloc=2.3GB sys=3.8GB
[WARN] [09-23|00:37:18.621] [NewPayload] Previously known bad block hash=0xb481bdd8d42fecdb61c82b703f4836c58cadce719cd1b18a83b0a89085af590a parentHash=0x37ce7f00b3e611e1d056c2c217c30fb9d3566e85f2f38ea6191be4959aed1c53
[WARN] [09-23|00:37:19.039] [NewPayload] Previously known bad block hash=0x16a7b8c00bd93dd1beee7d8aead00eb33de2fff55c505c3ff5a252063d44f2b7 parentHash=0xb481bdd8d42fecdb61c82b703f4836c58cadce719cd1b18a83b0a89085af590a
[WARN] [09-23|00:37:19.259] [NewPayload] Previously known bad block hash=0xca22c5431c179ca67be50d6add8169a8bd166569368cb71a7dec6bc4b8185eff parentHash=0x16a7b8c00bd93dd1beee7d8aead00eb33de2fff55c505c3ff5a252063d44f2b7
[WARN] [09-23|00:37:19.417] [NewPayload] Previously known bad block hash=0x3ed6ce262d18ab91d0aa6570a3b209979ff136fafbf133c7ef6e4baf31b8e430 parentHash=0xca22c5431c179ca67be50d6add8169a8bd166569368cb71a7dec6bc4b8185eff
[WARN] [09-23|00:37:19.784] [NewPayload] Previously known bad block hash=0x4faf456fadb0104e4bddb485ba5bb577ae70b20a99eca1e1e0152e1a767f629b parentHash=0x3ed6ce262d18ab91d0aa6570a3b209979ff136fafbf133c7ef6e4baf31b8e430
[WARN] [09-23|00:37:20.083] [NewPayload] Previously known bad block hash=0x26677bbbd4f3c69a8738855b936303accc155005560cde45fbdcd43f547eb542 parentHash=0x4faf456fadb0104e4bddb485ba5bb577ae70b20a99eca1e1e0152e1a767f629b
[WARN] [09-23|00:37:20.277] [NewPayload] Previously known bad block hash=0xc31e2b19df044ccf8357a8025de5518232ce1a40b8443a3d57995f87c074358d parentHash=0x26677bbbd4f3c69a8738855b936303accc155005560cde45fbdcd43f547eb542
[WARN] [09-23|00:37:20.502] [ForkChoiceUpdated] Previously known bad block hash=0x26677bbbd4f3c69a8738855b936303accc155005560cde45fbdcd43f547eb542
[WARN] [09-23|00:37:20.570] [NewPayload] Previously known bad block hash=0x7ade8e67a1305ad10c23d8910822b9d670aafaa3b35d3f96ec3b307a590a44b2 parentHash=0xc31e2b19df044ccf8357a8025de5518232ce1a40b8443a3d57995f87c074358d
[WARN] [09-23|00:37:20.849] [NewPayload] Previously known bad block hash=0x3efd1706402978fb25e46d3c6d0eb357c4fc920e8bf1addd3a5265e27d771b47 parentHash=0x7ade8e67a1305ad10c23d8910822b9d670aafaa3b35d3f96ec3b307a590a44b2
[WARN] [09-23|00:37:20.953] [ForkChoiceUpdated] Previously known bad block hash=0x3efd1706402978fb25e46d3c6d0eb357c4fc920e8bf1addd3a5265e27d771b47
[WARN] [09-23|00:37:24.102] [NewPayload] Previously known bad block hash=0x5afb44e0ac451c2f9335f9666f5834f13222fd8eddc208d2af4f041b814d5e49 parentHash=0x3efd1706402978fb25e46d3c6d0eb357c4fc920e8bf1addd3a5265e27d771b47
[WARN] [09-23|00:37:24.373] [ForkChoiceUpdated] Previously known bad block hash=0x5afb44e0ac451c2f9335f9666f5834f13222fd8eddc208d2af4f041b814d5e49
[INFO] [09-23|00:37:26.829] [txpool] stat block=15590991 pending=1393 baseFee=30000 queued=30000 alloc=2.2GB sys=3.8GB
[WARN] [09-23|00:37:36.413] [NewPayload] Previously known bad block hash=0xca777ed633ba6351610f4af847cbe4e0461142da46c8cd8f8ca6e838f84aad24 parentHash=0x5afb44e0ac451c2f9335f9666f5834f13222fd8eddc208d2af4f041b814d5e49
[WARN] [09-23|00:37:36.555] [ForkChoiceUpdated] Previously known bad block hash=0xca777ed633ba6351610f4af847cbe4e0461142da46c8cd8f8ca6e838f84aad24
[WARN] [09-23|00:37:48.050] [NewPayload] Previously known bad block hash=0x132821323757c4a036ff3756ec2651367d8c6901518ab633d5b84449aa285e97 parentHash=0xca777ed633ba6351610f4af847cbe4e0461142da46c8cd8f8ca6e838f84aad24
[WARN] [09-23|00:37:48.321] [ForkChoiceUpdated] Previously known bad block hash=0x132821323757c4a036ff3756ec2651367d8c6901518ab633d5b84449aa285e97
[WARN] [09-23|00:38:00.335] [NewPayload] Previously known bad block hash=0x6b574517644811d8a38ce5bd9f6752e81038d4ea36aabcdb36e70ed9238fd9a7 parentHash=0x132821323757c4a036ff3756ec2651367d8c6901518ab633d5b84449aa285e97
[WARN] [09-23|00:38:00.568] [ForkChoiceUpdated] Previously known bad block hash=0x6b574517644811d8a38ce5bd9f6752e81038d4ea36aabcdb36e70ed9238fd9a7
[WARN] [09-23|00:38:13.250] [NewPayload] Previously known bad block hash=0x0422e6cb6467bac5894ffaf0fbd7d6bb0832396a3be63a038d830e1a2f5d9b40 parentHash=0x6b574517644811d8a38ce5bd9f6752e81038d4ea36aabcdb36e70ed9238fd9a7
[WARN] [09-23|00:38:13.429] [ForkChoiceUpdated] Previously known bad block hash=0x0422e6cb6467bac5894ffaf0fbd7d6bb0832396a3be63a038d830e1a2f5d9b40
[WARN] [09-23|00:38:23.742] [NewPayload] Previously known bad block hash=0x4564d1df85ae2237a0e92e6495feff0cc617f008db39602048c373cc2f04426e parentHash=0x0422e6cb6467bac5894ffaf0fbd7d6bb0832396a3be63a038d830e1a2f5d9b40
[WARN] [09-23|00:38:23.856] [ForkChoiceUpdated] Previously known bad block hash=0x4564d1df85ae2237a0e92e6495feff0cc617f008db39602048c373cc2f04426e
[INFO] [09-23|00:38:26.475] [p2p] GoodPeers eth66=2
[INFO] [09-23|00:38:26.837] [txpool] stat block=15590991 pending=1908 baseFee=30000 queued=30000 alloc=2.3GB sys=3.8GB
[WARN] [09-23|00:38:36.440] [NewPayload] Previously known bad block hash=0x1d615767f30e4522ee555ffe7b952dbe94382c70cb40f6c844e2566738fa9a08 parentHash=0x4564d1df85ae2237a0e92e6495feff0cc617f008db39602048c373cc2f04426e
[WARN] [09-23|00:38:36.668] [ForkChoiceUpdated] Previously known bad block hash=0x1d615767f30e4522ee555ffe7b952dbe94382c70cb40f6c844e2566738fa9a08
[WARN] [09-23|00:38:47.532] [NewPayload] Previously known bad block hash=0x6ad1102b8ecf4b2135adf6ad4063a57ad0ec9dabff8b558039b1bdb6eb9c425a parentHash=0x1d615767f30e4522ee555ffe7b952dbe94382c70cb40f6c844e2566738fa9a08
[WARN] [09-23|00:38:47.627] [ForkChoiceUpdated] Previously known bad block hash=0x6ad1102b8ecf4b2135adf6ad4063a57ad0ec9dabff8b558039b1bdb6eb9c425a
[WARN] [09-23|00:38:59.727] [NewPayload] Previously known bad block hash=0xddd5ce601b78ab9a6f43ccb992f17e7e779f2ac2c3e74fb3eb2e4494630e777a parentHash=0x6ad1102b8ecf4b2135adf6ad4063a57ad0ec9dabff8b558039b1bdb6eb9c425a
[WARN] [09-23|00:38:59.947] [ForkChoiceUpdated] Previously known bad block hash=0xddd5ce601b78ab9a6f43ccb992f17e7e779f2ac2c3e74fb3eb2e4494630e777a
[WARN] [09-23|00:39:11.996] [NewPayload] Previously known bad block hash=0xc4d5e81e45edd6523a1814187f90bd4823460bb1c1260c0e7bd3fac1511264e4 parentHash=0xddd5ce601b78ab9a6f43ccb992f17e7e779f2ac2c3e74fb3eb2e4494630e777a
[WARN] [09-23|00:39:12.087] [ForkChoiceUpdated] Previously known bad block hash=0xc4d5e81e45edd6523a1814187f90bd4823460bb1c1260c0e7bd3fac1511264e4
[WARN] [09-23|00:39:24.609] [NewPayload] Previously known bad block hash=0x987363255f9a49efda9de10c8ed35f53ae9539870e2b2e211fb0ce034471d812 parentHash=0xc4d5e81e45edd6523a1814187f90bd4823460bb1c1260c0e7bd3fac1511264e4
[WARN] [09-23|00:39:24.814] [ForkChoiceUpdated] Previously known bad block hash=0x987363255f9a49efda9de10c8ed35f53ae9539870e2b2e211fb0ce034471d812
[INFO] [09-23|00:39:26.829] [txpool] stat block=15590991 pending=2288 baseFee=30000 queued=30000 alloc=2.2GB sys=3.8GB
[WARN] [09-23|00:39:35.502] [NewPayload] Previously known bad block hash=0x9bb9d25da6a2f473b28796ed25511dbc27857edf943928fadadc04011ea9f057 parentHash=0x987363255f9a49efda9de10c8ed35f53ae9539870e2b2e211fb0ce034471d812
run ./build/bin/integration state_stage --unwind=500 --datadir=your/datadir --chain=your-chain that might be a work around to your problem
run
./build/bin/integration state_stage --unwind=500 --datadir=your/datadir --chain=your-chainthat might be a work around to your problem
ok, so that's weird, I updated erigon and started it again, to get the error so I can run the command, and it looks like its running fine for now. I will update if the issue persists
looks like it's broken again after a few hours running normally.
[INFO] [09-24|04:02:18.397] [5/7 Execution] Completed on block=15599202
[INFO] [09-24|04:02:30.847] Timings (slower than 50ms) Execution=5.606s IntermediateHashes=12.419s
[INFO] [09-24|04:02:30.847] Tables PlainState=96.0KB AccountChangeSet=32.0KB StorageChangeSet=72.0KB BlockTransaction=228.0KB TransactionLog=316.0KB FreeList=0B
[INFO] [09-24|04:02:30.848] Timings (slower than 50ms) Headers=31.115s
[INFO] [09-24|04:02:30.848] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.3GB TransactionLog=451.9GB FreeList=10.7MB ReclaimableSpace=10.7GB
[INFO] [09-24|04:02:31.876] Commit cycle in=1.027869989s
[INFO] [09-24|04:02:31.876] RPC Daemon notified of new headers from=15599201 to=15599202 header sending=10.55µs log sending=251ns
[INFO] [09-24|04:02:31.876] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-24|04:02:40.055] [txpool] stat block=15599202 pending=522 baseFee=30000 queued=30000 alloc=2.3GB sys=3.2GB
[INFO] [09-24|04:02:44.910] [2/16 Headers] New payload: need to download parent height=15599204 hash=0xa3e5288f549ae58ed7c8781636761337ff6853d2d4e4e99a1d489559cd0caaa6 parentHash=0x3b8958735da4789cf298f4ac9a40a36d217c20fe904df51b34b6dcf0f46e5118
[INFO] [09-24|04:02:44.910] [2/16 Headers] Downloading PoS headers... height=15599203 hash=0x3b8958735da4789cf298f4ac9a40a36d217c20fe904df51b34b6dcf0f46e5118 requestId=1255
[INFO] [09-24|04:02:45.022] Timings (slower than 50ms) Headers=13.145s
[INFO] [09-24|04:02:45.022] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.3GB TransactionLog=451.9GB FreeList=10.7MB ReclaimableSpace=10.7GB
[INFO] [09-24|04:02:45.022] Commit cycle in=33.703µs
[INFO] [09-24|04:02:45.022] RPC Daemon notified of new headers from=15599201 to=15599202 header sending=11.823µs log sending=270ns
[WARN] [09-24|04:02:45.369] Could not validate block err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:658 collector.go:276 collector.go:144 stage_headers.go:671 stage_headers.go:718 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]"
[WARN] [09-24|04:02:45.375] Removing beacon request due to err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:658 collector.go:276 collector.go:144 stage_headers.go:671 stage_headers.go:718 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]" requestId=1255
[INFO] [09-24|04:02:45.375] Timings (slower than 50ms) Headers=352ms
[INFO] [09-24|04:02:45.375] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.3GB TransactionLog=451.9GB FreeList=10.7MB ReclaimableSpace=10.7GB
[INFO] [09-24|04:02:45.375] Commit cycle in=4.358µs
[INFO] [09-24|04:02:45.375] RPC Daemon notified of new headers from=15599201 to=15599202 header sending=5.781µs log sending=170ns
[INFO] [09-24|04:02:45.375] [2/16 Headers] Waiting for Consensus Layer...
[WARN] [09-24|04:02:49.902] [NewPayload] Previously known bad block hash=0xabf986ecf818ee6aaf748b4fae817606acb7c399dfc7418abe10d6ee820fd200 parentHash=0xa3e5288f549ae58ed7c8781636761337ff6853d2d4e4e99a1d489559cd0caaa6
[WARN] [09-24|04:02:51.549] [ForkChoiceUpdated] Previously known bad block hash=0xabf986ecf818ee6aaf748b4fae817606acb7c399dfc7418abe10d6ee820fd200
[WARN] [09-24|04:03:01.075] [NewPayload] Previously known bad block hash=0x6c2370474194ecd3924cbb34bb1b71ca0fcbb52d3347cae8885ca138ea8cbd8a parentHash=0xabf986ecf818ee6aaf748b4fae817606acb7c399dfc7418abe10d6ee820fd200
[WARN] [09-24|04:03:02.297] [ForkChoiceUpdated] Previously known bad block hash=0x6c2370474194ecd3924cbb34bb1b71ca0fcbb52d3347cae8885ca138ea8cbd8a
[WARN] [09-24|04:03:11.884] [NewPayload] Previously known bad block hash=0x593bda5522781bd1eef1c8d7d9bdae19dcce594120d18df5ad70ff1caaf36355 parentHash=0x6c2370474194ecd3924cbb34bb1b71ca0fcbb52d3347cae8885ca138ea8cbd8a
[WARN] [09-24|04:03:12.106] [ForkChoiceUpdated] Previously known bad block hash=0x593bda5522781bd1eef1c8d7d9bdae19dcce594120d18df5ad70ff1caaf36355
[WARN] [09-24|04:03:24.695] [NewPayload] Previously known bad block hash=0x0e2edfc551fe1909e01590d6c72bd8dc1f5d0c5580d0b9d40bebe14051cf0aeb parentHash=0x593bda5522781bd1eef1c8d7d9bdae19dcce594120d18df5ad70ff1caaf36355
[WARN] [09-24|04:03:25.652] [ForkChoiceUpdated] Previously known bad block hash=0x0e2edfc551fe1909e01590d6c72bd8dc1f5d0c5580d0b9d40bebe14051cf0aeb
[WARN] [09-24|04:03:37.542] [NewPayload] Previously known bad block hash=0x1c37f66f0a920699e6876b1e5653aad51503c15c67733af2c65a535ee03f1c09 parentHash=0x0e2edfc551fe1909e01590d6c72bd8dc1f5d0c5580d0b9d40bebe14051cf0aeb
[WARN] [09-24|04:03:37.786] [ForkChoiceUpdated] Previously known bad block hash=0x1c37f66f0a920699e6876b1e5653aad51503c15c67733af2c65a535ee03f1c09
[INFO] [09-24|04:03:39.650] [p2p] GoodPeers eth66=14
[INFO] [09-24|04:03:40.005] [txpool] stat block=15599202 pending=1134 baseFee=30000 queued=30000 alloc=2.5GB sys=3.2GB
[WARN] [09-24|04:03:48.423] [NewPayload] Previously known bad block hash=0x01071abcd93a038904de186f7b69d17583194d52f9aeb45123426cb7e77df72a parentHash=0x1c37f66f0a920699e6876b1e5653aad51503c15c67733af2c65a535ee03f1c09
[WARN] [09-24|04:03:48.968] [ForkChoiceUpdated] Previously known bad block hash=0x01071abcd93a038904de186f7b69d17583194d52f9aeb45123426cb7e77df72a
[WARN] [09-24|04:04:01.112] [NewPayload] Previously known bad block hash=0x20d3ce5dbb34b1d91c44eb6a80ed6fb878f1b04a131cef9adb4a81482ce547ef parentHash=0x01071abcd93a038904de186f7b69d17583194d52f9aeb45123426cb7e77df72a
[INFO] [09-24|04:04:01.233] [2/16 Headers] Fork choice: need to download header with hash 20d3ce5dbb34b1d91c44eb6a80ed6fb878f1b04a131cef9adb4a81482ce547ef
[INFO] [09-24|04:04:01.233] [2/16 Headers] Downloading PoS headers... height=0 hash=0x20d3ce5dbb34b1d91c44eb6a80ed6fb878f1b04a131cef9adb4a81482ce547ef requestId=1256
[INFO] [09-24|04:04:01.233] [2/16 Headers] DONE in=1m15.857221323s
[INFO] [09-24|04:04:01.233] Timings (slower than 50ms) Headers=1m15.857s
[INFO] [09-24|04:04:01.233] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.3GB TransactionLog=451.9GB FreeList=10.7MB ReclaimableSpace=10.7GB
[INFO] [09-24|04:04:01.233] Commit cycle in=5.901µs
[INFO] [09-24|04:04:01.233] RPC Daemon notified of new headers from=15599201 to=15599202 header sending=6.963µs log sending=180ns
[WARN] [09-24|04:04:19.277] [NewPayload] Previously known bad block hash=0xce1c2bb1a0c77012ae13e2974906f7f301e74d196759f9701a4806e9647a3bf7 parentHash=0x20d3ce5dbb34b1d91c44eb6a80ed6fb878f1b04a131cef9adb4a81482ce547ef
[WARN] [09-24|04:04:20.414] Could not validate block err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:658 collector.go:276 collector.go:144 stage_headers.go:671 stage_headers.go:718 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]"
[WARN] [09-24|04:04:20.418] Removing beacon request due to err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:658 collector.go:276 collector.go:144 stage_headers.go:671 stage_headers.go:718 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]" requestId=1256
[INFO] [09-24|04:04:20.418] Timings (slower than 50ms) Headers=19.184s
[INFO] [09-24|04:04:20.418] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.3GB TransactionLog=451.9GB FreeList=10.7MB ReclaimableSpace=10.7GB
[INFO] [09-24|04:04:20.418] Commit cycle in=9.979µs
[INFO] [09-24|04:04:20.418] RPC Daemon notified of new headers from=15599201 to=15599202 header sending=14.146µs log sending=290ns
[INFO] [09-24|04:04:20.418] [2/16 Headers] Waiting for Consensus Layer...
[WARN] [09-24|04:04:24.503] [NewPayload] Previously known bad block hash=0x2eb9a0163a0f22d042839612526074fc354a9356303b71b35c6a818c434aa96b parentHash=0xce1c2bb1a0c77012ae13e2974906f7f301e74d196759f9701a4806e9647a3bf7
[WARN] [09-24|04:04:25.119] [ForkChoiceUpdated] Previously known bad block hash=0x2eb9a0163a0f22d042839612526074fc354a9356303b71b35c6a818c434aa96b
[WARN] [09-24|04:04:35.952] [NewPayload] Previously known bad block hash=0x0c2a9f9c036158f22eb105e1ade624a85af5717b8ca28e32936d22bfeca6fba0 parentHash=0x2eb9a0163a0f22d042839612526074fc354a9356303b71b35c6a818c434aa96b
[WARN] [09-24|04:04:36.096] [ForkChoiceUpdated] Previously known bad block hash=0x0c2a9f9c036158f22eb105e1ade624a85af5717b8ca28e32936d22bfeca6fba0
[INFO] [09-24|04:04:40.398] [txpool] stat block=15599202 pending=1689 baseFee=30000 queued=30000 alloc=2.3GB sys=3.2GB
[WARN] [09-24|04:04:47.575] [NewPayload] Previously known bad block hash=0xd0b2314ce3d3611c0f7a34d0da224049ceabe64c8d83844b4db20b4e5122b03d parentHash=0x0c2a9f9c036158f22eb105e1ade624a85af5717b8ca28e32936d22bfeca6fba0
[WARN] [09-24|04:04:47.709] [ForkChoiceUpdated] Previously known bad block hash=0xd0b2314ce3d3611c0f7a34d0da224049ceabe64c8d83844b4db20b4e5122b03d
[WARN] [09-24|04:05:00.141] [NewPayload] Previously known bad block hash=0x57b43d1d9a6ed23be07aec647d76c3b6589571f3dc2c0a5d2e4c04a521ed4718 parentHash=0xd0b2314ce3d3611c0f7a34d0da224049ceabe64c8d83844b4db20b4e5122b03d
[WARN] [09-24|04:05:01.748] [ForkChoiceUpdated] Previously known bad block hash=0x57b43d1d9a6ed23be07aec647d76c3b6589571f3dc2c0a5d2e4c04a521ed4718
[WARN] [09-24|04:05:14.071] [NewPayload] Previously known bad block hash=0x61cc4e065655b4060d267efef704c1de3e026d5d725ae1851f605f7445b8d748 parentHash=0x57b43d1d9a6ed23be07aec647d76c3b6589571f3dc2c0a5d2e4c04a521ed4718
[WARN] [09-24|04:05:18.791] [ForkChoiceUpdated] Previously known bad block hash=0x61cc4e065655b4060d267efef704c1de3e026d5d725ae1851f605f7445b8d748
[WARN] [09-24|04:05:24.514] [NewPayload] Previously known bad block hash=0xacd847c2b57baf8dbfc98581bf9070f00d7283928a7fbce510539c93f5f9054d parentHash=0x61cc4e065655b4060d267efef704c1de3e026d5d725ae1851f605f7445b8d748
[WARN] [09-24|04:05:24.629] [ForkChoiceUpdated] Previously known bad block hash=0xacd847c2b57baf8dbfc98581bf9070f00d7283928a7fbce510539c93f5f9054d
[WARN] [09-24|04:05:36.531] [NewPayload] Previously known bad block hash=0xe3b7596170fbde4c276d613ebbf94d9ec40678b14edff5f9d5dfdbf83fca7669 parentHash=0xacd847c2b57baf8dbfc98581bf9070f00d7283928a7fbce510539c93f5f9054d
[WARN] [09-24|04:05:36.688] [ForkChoiceUpdated] Previously known bad block hash=0xe3b7596170fbde4c276d613ebbf94d9ec40678b14edff5f9d5dfdbf83fca7669
[INFO] [09-24|04:05:39.651] [p2p] GoodPeers eth66=14
[INFO] [09-24|04:05:40.798] [txpool] stat block=15599202 pending=2242 baseFee=30000 queued=30000 alloc=2.4GB sys=3.2GB
[WARN] [09-24|04:05:49.503] [NewPayload] Previously known bad block hash=0x0bce8d17fbda27ad4a52728e8e663c393b8093129d7d01b60adb35806d50897b parentHash=0xe3b7596170fbde4c276d613ebbf94d9ec40678b14edff5f9d5dfdbf83fca7669
[WARN] [09-24|04:05:50.273] [ForkChoiceUpdated] Previously known bad block hash=0x0bce8d17fbda27ad4a52728e8e663c393b8093129d7d01b60adb35806d50897b
[WARN] [09-24|04:05:58.516] [ForkChoiceUpdated] Previously known bad block hash=0x0bce8d17fbda27ad4a52728e8e663c393b8093129d7d01b60adb35806d50897b
[WARN] [09-24|04:06:01.908] [NewPayload] Previously known bad block hash=0xef4f512b71d551897bbe72bc9a7573125b88c114bb63189a0585934d7942df74 parentHash=0x0bce8d17fbda27ad4a52728e8e663c393b8093129d7d01b60adb35806d50897b
[WARN] [09-24|04:06:06.155] [ForkChoiceUpdated] Previously known bad block hash=0xef4f512b71d551897bbe72bc9a7573125b88c114bb63189a0585934d7942df74
[WARN] [09-24|04:06:12.282] [NewPayload] Previously known bad block hash=0xeb8db468648782fd233c6083de2197901cb030a1d0a10ddafd214ee1c6c69c56 parentHash=0xef4f512b71d551897bbe72bc9a7573125b88c114bb63189a0585934d7942df74
[WARN] [09-24|04:06:24.869] [ForkChoiceUpdated] Previously known bad block hash=0xeb8db468648782fd233c6083de2197901cb030a1d0a10ddafd214ee1c6c69c56
[WARN] [09-24|04:06:26.044] [NewPayload] Previously known bad block hash=0x4752b38d7115b32c3e1e9d1d5427cdee76a742626fd708ef1da712c14483fc41 parentHash=0xeb8db468648782fd233c6083de2197901cb030a1d0a10ddafd214ee1c6c69c56
[WARN] [09-24|04:06:31.282] [ForkChoiceUpdated] Previously known bad block hash=0x4752b38d7115b32c3e1e9d1d5427cdee76a742626fd708ef1da712c14483fc41
[WARN] [09-24|04:06:37.390] [NewPayload] Previously known bad block hash=0x70acf5f6a927bde397eb5e13021df8e4b971931aacf48d903e587da45478a4b7 parentHash=0x4752b38d7115b32c3e1e9d1d5427cdee76a742626fd708ef1da712c14483fc41
[INFO] [09-24|04:06:40.611] [txpool] stat block=15599202 pending=2767 baseFee=30000 queued=30000 alloc=2.2GB sys=3.2GB
[WARN] [09-24|04:06:43.738] [ForkChoiceUpdated] Previously known bad block hash=0x70acf5f6a927bde397eb5e13021df8e4b971931aacf48d903e587da45478a4b7
[WARN] [09-24|04:06:49.113] [NewPayload] Previously known bad block hash=0x98e9c9c21881009bb4e7146df7d530b8e87df48310d2096d375d6cae9955375f parentHash=0x70acf5f6a927bde397eb5e13021df8e4b971931aacf48d903e587da45478a4b7
[WARN] [09-24|04:06:59.192] [NewPayload] Previously known bad block hash=0x98e9c9c21881009bb4e7146df7d530b8e87df48310d2096d375d6cae9955375f
[WARN] [09-24|04:06:59.315] [ForkChoiceUpdated] Previously known bad block hash=0x98e9c9c21881009bb4e7146df7d530b8e87df48310d2096d375d6cae9955375f
[WARN] [09-24|04:07:01.085] [NewPayload] Previously known bad block hash=0xda65b093210e8ee787e8bd027e9afd44a7e047f12fa9a5814749f63294e0c655 parentHash=0x98e9c9c21881009bb4e7146df7d530b8e87df48310d2096d375d6cae9955375f
[WARN] [09-24|04:07:01.328] [ForkChoiceUpdated] Previously known bad block hash=0xda65b093210e8ee787e8bd027e9afd44a7e047f12fa9a5814749f63294e0c655
[WARN] [09-24|04:07:13.042] [NewPayload] Previously known bad block hash=0x5ea2cae5d8c45b4c3940b672973060bafd5253eec10650eaaa390a2378d076ae parentHash=0xda65b093210e8ee787e8bd027e9afd44a7e047f12fa9a5814749f63294e0c655
[WARN] [09-24|04:07:13.240] [ForkChoiceUpdated] Previously known bad block hash=0x5ea2cae5d8c45b4c3940b672973060bafd5253eec10650eaaa390a2378d076ae
[WARN] [09-24|04:07:24.692] [NewPayload] Previously known bad block hash=0x7f7167a79bc810e9475b8dbd583d1e9efb2f652a56b2416006988f2e064a1911 parentHash=0x5ea2cae5d8c45b4c3940b672973060bafd5253eec10650eaaa390a2378d076ae
[WARN] [09-24|04:07:24.898] [ForkChoiceUpdated] Previously known bad block hash=0x7f7167a79bc810e9475b8dbd583d1e9efb2f652a56b2416006988f2e064a1911
[WARN] [09-24|04:07:37.007] [NewPayload] Previously known bad block hash=0xcb0ace6b374421293511c9e075721a05dd3fdc79f396715c13002dbcd6fba923 parentHash=0x7f7167a79bc810e9475b8dbd583d1e9efb2f652a56b2416006988f2e064a1911
[WARN] [09-24|04:07:37.249] [ForkChoiceUpdated] Previously known bad block hash=0xcb0ace6b374421293511c9e075721a05dd3fdc79f396715c13002dbcd6fba923
[INFO] [09-24|04:07:39.651] [p2p] GoodPeers eth66=14
[INFO] [09-24|04:07:40.003] [txpool] stat block=15599202 pending=3871 baseFee=30000 queued=30000 alloc=2.5GB sys=3.2GB
[WARN] [09-24|04:07:48.583] [NewPayload] Previously known bad block hash=0xf19ea57922d702f44ee2d39b6bf8ad36052094025d9eedff5741b4c68b017797 parentHash=0xcb0ace6b374421293511c9e075721a05dd3fdc79f396715c13002dbcd6fba923
[WARN] [09-24|04:07:48.807] [ForkChoiceUpdated] Previously known bad block hash=0xf19ea57922d702f44ee2d39b6bf8ad36052094025d9eedff5741b4c68b017797
[WARN] [09-24|04:07:59.659] [NewPayload] Previously known bad block hash=0x8a9821b2a260a97f65a47809fc0219749b39fa24adfd06a4b178ae085458e3a5 parentHash=0xf19ea57922d702f44ee2d39b6bf8ad36052094025d9eedff5741b4c68b017797
[WARN] [09-24|04:07:59.762] [ForkChoiceUpdated] Previously known bad block hash=0x8a9821b2a260a97f65a47809fc0219749b39fa24adfd06a4b178ae085458e3a5
[WARN] [09-24|04:08:12.384] [NewPayload] Previously known bad block hash=0x65e082267a9d84fccdecdfcca108e947a260db13956adf0fa78d6b5901f8eca7 parentHash=0x8a9821b2a260a97f65a47809fc0219749b39fa24adfd06a4b178ae085458e3a5
[WARN] [09-24|04:08:12.619] [ForkChoiceUpdated] Previously known bad block hash=0x65e082267a9d84fccdecdfcca108e947a260db13956adf0fa78d6b5901f8eca7
[WARN] [09-24|04:08:23.984] [NewPayload] Previously known bad block hash=0xbe5877de7c9177f788b34bef1587744ec6ee7f8f0eff71495c955c7600822163 parentHash=0x65e082267a9d84fccdecdfcca108e947a260db13956adf0fa78d6b5901f8eca7
[WARN] [09-24|04:08:24.173] [ForkChoiceUpdated] Previously known bad block hash=0xbe5877de7c9177f788b34bef1587744ec6ee7f8f0eff71495c955c7600822163
[WARN] [09-24|04:08:36.382] [NewPayload] Previously known bad block hash=0x7bdcbecd228534e22b5190d02da8ba7f079f3d9945601a92df59d42c3ca1fe72 parentHash=0xbe5877de7c9177f788b34bef1587744ec6ee7f8f0eff71495c955c7600822163
I run the command above, where it found the bad block and unwinded, but trying to run with the latest version from that point results to the same error.
[INFO] [09-24|06:32:23.974] [txpool] Started
[INFO] [09-24|06:32:25.015] [Snapshots] Stat blocks=14000k indices=14000k alloc=2.3GB sys=2.4GB
[INFO] [09-24|06:32:25.016] [7/16 Execution] Blocks execution from=15599292 to=15599792
[INFO] [09-24|06:32:45.065] [7/16 Execution] Executed blocks number=15599548 blk/s=12.8 tx/s=1955.1 Mgas/s=192.0 gasState=0.01 estimated duration=39.573s batch=6.7MB alloc=2.8GB sys=3.2GB
[INFO] [09-24|06:32:52.043] new subscription to newHeaders established
[INFO] [09-24|06:32:52.043] New txs subscriber joined
[INFO] [09-24|06:33:02.130] [7/16 Execution] Completed on block=15599792
[INFO] [09-24|06:33:02.130] [8/16 HashState] Promoting plain state from=15599292 to=15599792
[INFO] [09-24|06:33:02.130] [8/16 HashState] Incremental promotion from=15599292 to=15599792 codes=true csbucket=AccountChangeSet
[INFO] [09-24|06:33:05.103] [8/16 HashState] Incremental promotion from=15599292 to=15599792 codes=false csbucket=AccountChangeSet
[INFO] [09-24|06:33:06.037] [8/16 HashState] Incremental promotion from=15599292 to=15599792 codes=false csbucket=StorageChangeSet
[INFO] [09-24|06:33:09.513] [9/16 IntermediateHashes] Generating intermediate hashes from=15599292 to=15599792
[WARN] [09-24|06:33:14.990] Served conn=127.0.0.1:38182 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=82.555688ms err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-24|06:33:17.517] [txpool] stat block=15599792 pending=924 baseFee=30000 queued=22446 alloc=3.3GB sys=4.6GB
[INFO] [09-24|06:33:51.575] [13/16 LogIndex] processing from=15599293 to=15599792
[INFO] [09-24|06:34:05.859] Timings (slower than 50ms) Snapshots=7.488s Execution=37.113s HashState=7.381s IntermediateHashes=17.239s CallTraces=3.692s AccountHistoryIndex=8.752s StorageHistoryIndex=12.379s LogIndex=6.504s TxLookup=7.757s
[INFO] [09-24|06:34:05.862] RPC Daemon notified of new headers from=15599292 to=15599875 header sending=3.033496ms log sending=211ns
[INFO] [09-24|06:34:05.863] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-24|06:34:11.770] [2/16 Headers] New payload: need to download parent height=15599959 hash=0xb2a9e5f4f5db93ddd49b63b46f73c94534858568eb8936af8a93d6ffb1d8673d parentHash=0x3835001c41c6a0c63a7757daa99185dba9a40fb596a2e373364b6580ba817f93
[INFO] [09-24|06:34:11.770] [2/16 Headers] Downloading PoS headers... height=15599958 hash=0x3835001c41c6a0c63a7757daa99185dba9a40fb596a2e373364b6580ba817f93 requestId=1
[INFO] [09-24|06:34:11.772] Timings (slower than 50ms) Headers=5.908s
[INFO] [09-24|06:34:11.772] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.4GB TransactionLog=451.9GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-24|06:34:11.772] Commit cycle in=8.075µs
[INFO] [09-24|06:34:11.772] RPC Daemon notified of new headers from=15599792 to=15599875 header sending=94.328µs log sending=221ns
[INFO] [09-24|06:34:14.729] PoS headers verified and saved requestId=1 fork head=0x23a2f2e2cf934cd0b059d32a8d9a2663e90dc0c37c36cd7d1b0aba04aec29cee
[INFO] [09-24|06:34:14.730] Timings (slower than 50ms) Headers=2.957s
[INFO] [09-24|06:34:14.730] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.4GB TransactionLog=451.9GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-24|06:34:14.738] Commit cycle in=7.764133ms
[INFO] [09-24|06:34:14.738] RPC Daemon notified of new headers from=15599792 to=15599958 header sending=178.328µs log sending=300ns
[INFO] [09-24|06:34:14.738] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-24|06:34:14.750] Commit cycle in=4.060789ms
[INFO] [09-24|06:34:14.750] RPC Daemon notified of new headers from=15599792 to=15599959 header sending=171.084µs log sending=190ns
[INFO] [09-24|06:34:14.750] [2/16 Headers] Waiting for Consensus Layer...
[WARN] [09-24|06:34:14.910] Served conn=127.0.0.1:37302 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=59.703µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-24|06:34:16.492] [p2p] GoodPeers eth66=2
[INFO] [09-24|06:34:17.524] [txpool] stat block=15599792 pending=1183 baseFee=30000 queued=22829 alloc=3.7GB sys=5.0GB
[INFO] [09-24|06:34:24.363] Timings (slower than 50ms) Headers=9.612s
[INFO] [09-24|06:34:24.363] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.4GB TransactionLog=451.9GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-24|06:34:24.367] Commit cycle in=3.897332ms
[INFO] [09-24|06:34:24.367] RPC Daemon notified of new headers from=15599792 to=15599960 header sending=189.909µs log sending=331ns
[INFO] [09-24|06:34:24.367] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-24|06:34:24.519] [2/16 Headers] Fork choice re-org headerNumber=15599960 forkingPoint=15599792
[INFO] [09-24|06:34:24.519] UnwindTo block=15599792 bad_block_hash=0x0000000000000000000000000000000000000000000000000000000000000000
[INFO] [09-24|06:34:24.519] [2/16 Headers] Unsettled forkchoice after unwind height=15599960 forkchoice="&{HeadBlockHash:0xb820b81355c7647537947ff84160a90a56e3a2436256dbb71c530022c8eafdb8 SafeBlockHash:0xc52f592f927e3f1de62ca59efb24a79ceb89f0dcdb42fb9647c9b9ce67787ad9 FinalizedBlockHash:0x5a8761e1bbc359b28ee213819073e758fe06b7972d32af146c4ab934774627c3}"
[INFO] [09-24|06:34:24.534] [5/16 Bodies] Processing bodies... from=15599792 to=15599960
[INFO] [09-24|06:34:27.908] [5/16 Bodies] Writing block bodies block_num=15599823 remaining=137 alloc=3.8GB sys=5.0GB
[INFO] [09-24|06:34:30.335] [5/16 Bodies] Writing block bodies block_num=15599854 remaining=106 alloc=3.9GB sys=5.0GB
[INFO] [09-24|06:34:31.813] [5/16 Bodies] Writing block bodies block_num=15599884 remaining=76 alloc=3.9GB sys=5.0GB
[INFO] [09-24|06:34:34.157] [5/16 Bodies] Writing block bodies block_num=15599907 remaining=53 alloc=4.0GB sys=5.0GB
[INFO] [09-24|06:34:35.950] [5/16 Bodies] Writing block bodies block_num=15599960 remaining=0 alloc=2.1GB sys=5.0GB
[INFO] [09-24|06:34:36.116] [5/16 Bodies] Processed highest=15599960
[INFO] [09-24|06:34:36.116] [6/16 Senders] Started from=15599792 to=15599960
[INFO] [09-24|06:34:36.344] [7/16 Execution] Blocks execution from=15599792 to=15599960
[INFO] [09-24|06:34:48.835] [7/16 Execution] Completed on block=15599960
[INFO] [09-24|06:34:48.835] [8/16 HashState] Promoting plain state from=15599792 to=15599960
[INFO] [09-24|06:34:48.835] [8/16 HashState] Incremental promotion from=15599792 to=15599960 codes=true csbucket=AccountChangeSet
[INFO] [09-24|06:34:49.554] [8/16 HashState] Incremental promotion from=15599792 to=15599960 codes=false csbucket=AccountChangeSet
[INFO] [09-24|06:34:51.187] [8/16 HashState] Incremental promotion from=15599792 to=15599960 codes=false csbucket=StorageChangeSet
[INFO] [09-24|06:34:53.688] [9/16 IntermediateHashes] Generating intermediate hashes from=15599792 to=15599960
[INFO] [09-24|06:35:01.786] [10/16 CallTraces] Pruned call trace intermediate table from=15509792 to=15509959
[INFO] [09-24|06:35:10.614] [13/16 LogIndex] processing from=15599793 to=15599960
[WARN] [09-24|06:35:14.911] Served conn=127.0.0.1:46698 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=62.118µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-24|06:35:16.331] Timings (slower than 50ms) Headers=152ms Bodies=11.581s Senders=228ms Execution=12.49s HashState=4.853s IntermediateHashes=8.026s CallTraces=3.459s AccountHistoryIndex=2.485s StorageHistoryIndex=2.954s LogIndex=2.684s TxLookup=3.032s
[INFO] [09-24|06:35:16.331] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.4GB TransactionLog=452.0GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-24|06:35:17.516] [txpool] stat block=15599792 pending=1514 baseFee=30000 queued=23052 alloc=2.6GB sys=5.1GB
[INFO] [09-24|06:35:19.512] Commit cycle in=3.180769548s
[INFO] [09-24|06:35:19.512] RPC Daemon notified of new headers from=15599792 to=15599960 header sending=136.699µs log sending=160ns
[INFO] [09-24|06:35:19.513] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-24|06:35:24.516] [2/16 Headers] New payload: need to download parent height=15599965 hash=0x0c937a52eb88e989c7257c16f38a5eb8a206fa0cbfe14eeb7e173a15143e4aa4 parentHash=0x6292c7882694129abd4645d8a0dc4e04a56798eddbd8c6be903283f81e248aa8
[INFO] [09-24|06:35:24.516] [2/16 Headers] Downloading PoS headers... height=15599964 hash=0x6292c7882694129abd4645d8a0dc4e04a56798eddbd8c6be903283f81e248aa8 requestId=4
[INFO] [09-24|06:35:24.622] Timings (slower than 50ms) Headers=5.108s
[INFO] [09-24|06:35:24.622] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.4GB TransactionLog=452.0GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-24|06:35:24.622] Commit cycle in=26.019µs
[INFO] [09-24|06:35:24.622] RPC Daemon notified of new headers from=15599960 to=15599961 header sending=9.658µs log sending=320ns
[WARN] [09-24|06:35:25.806] Could not validate block err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:658 collector.go:276 collector.go:144 stage_headers.go:671 stage_headers.go:718 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]"
[WARN] [09-24|06:35:25.811] Removing beacon request due to err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:658 collector.go:276 collector.go:144 stage_headers.go:671 stage_headers.go:718 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]" requestId=4
[INFO] [09-24|06:35:25.811] Timings (slower than 50ms) Headers=1.189s
[INFO] [09-24|06:35:25.811] Tables PlainState=74.4GB AccountChangeSet=164.0GB StorageChangeSet=242.6GB BlockTransaction=161.4GB TransactionLog=452.0GB FreeList=10.5MB ReclaimableSpace=10.5GB
[INFO] [09-24|06:35:25.811] Commit cycle in=4.058µs
[INFO] [09-24|06:35:25.811] RPC Daemon notified of new headers from=15599960 to=15599961 header sending=5.19µs log sending=190ns
[INFO] [09-24|06:35:25.811] [2/16 Headers] Waiting for Consensus Layer...
[WARN] [09-24|06:35:36.785] [NewPayload] Previously known bad block hash=0x0d2866e41243b2c1b379a715bf0973f350bb4b8cf88eec5f9fd879e9ae16af54 parentHash=0x0c937a52eb88e989c7257c16f38a5eb8a206fa0cbfe14eeb7e173a15143e4aa4
[WARN] [09-24|06:35:36.997] [ForkChoiceUpdated] Previously known bad block hash=0x0d2866e41243b2c1b379a715bf0973f350bb4b8cf88eec5f9fd879e9ae16af54
[WARN] [09-24|06:35:48.024] [NewPayload] Previously known bad block hash=0x4fe2a7a69fc617836622408c207dd545625906a8ea495a17adf1080848c32c9b parentHash=0x0d2866e41243b2c1b379a715bf0973f350bb4b8cf88eec5f9fd879e9ae16af54
[WARN] [09-24|06:35:48.157] [ForkChoiceUpdated] Previously known bad block hash=0x4fe2a7a69fc617836622408c207dd545625906a8ea495a17adf1080848c32c9b
[WARN] [09-24|06:36:00.502] [NewPayload] Previously known bad block hash=0x695e22bad78a2577542adedced6fecbd0511732aeab6813079238d60b2f78162 parentHash=0x4fe2a7a69fc617836622408c207dd545625906a8ea495a17adf1080848c32c9b
[WARN] [09-24|06:36:00.782] [ForkChoiceUpdated] Previously known bad block hash=0x695e22bad78a2577542adedced6fecbd0511732aeab6813079238d60b2f78162
[WARN] [09-24|06:36:11.718] [NewPayload] Previously known bad block hash=0x454bce7acd1947f118d88f7ac2c35912aa8f6dde53dd44bf4f439b0415f07a19 parentHash=0x695e22bad78a2577542adedced6fecbd0511732aeab6813079238d60b2f78162
[WARN] [09-24|06:36:11.835] [ForkChoiceUpdated] Previously known bad block hash=0x454bce7acd1947f118d88f7ac2c35912aa8f6dde53dd44bf4f439b0415f07a19
[WARN] [09-24|06:36:14.913] Served conn=127.0.0.1:60680 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=63.901µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-24|06:36:16.492] [p2p] GoodPeers eth66=3
[INFO] [09-24|06:36:17.522] [txpool] stat block=15599960 pending=1520 baseFee=30000 queued=21349 alloc=2.8GB sys=5.1GB
running the older version I mentioned above seems to be working fine for now, although I'm getting a lot of these messages:
[WARN] [09-24|04:44:14.937] Served conn=127.0.0.1:59478 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=65.023µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
I had the same problem (Could not validate block err="Not implemented [...]) with fb0f811f19c41d2f005fc6173e4af1e6f01f6319. Reverting to v2022.09.03 fixed it.
I tried using #5510's commit but I ended up just getting a bunch of "[NewPayload] Previously known bad block hash=0x88e729e7bda2a8fd56b14ee32db574f2b2fb56670e3ef5b3bbf9f823e426d7be" in a never ending loop, stuck at block 15612204.
I tried using #5510's commit but I ended up just getting a bunch of "[NewPayload] Previously known bad block hash=0x88e729e7bda2a8fd56b14ee32db574f2b2fb56670e3ef5b3bbf9f823e426d7be" in a never ending loop, stuck at block 15612204.
You need to unwind
I don't know how to do that. I tried just restarting Erigon but it was still stuck with the "Previously known bad block" message. On the other hand, reverting to 2022.09.03 fixed it. So there's clearly still a regression in the #5510 branch.
(oh, I see you gave instructions on how to rewind above, but it wasn't needed with 2022.09.03)
(oh, I see you gave instructions on how to rewind above, but it wasn't needed with 2022.09.03)
Yes, you need to unwind so erigon doesn't count them as bad blocks anymore maybe restarting erigon will work but I don't remember well. Anyhow unwinding ~10 blocks should do the trick
(oh, I see you gave instructions on how to rewind above, but it wasn't needed with 2022.09.03)
Yes, you need to unwind so erigon doesn't count them as bad blocks anymore maybe restarting erigon will work but I don't remember well. Anyhow unwinding ~10 blocks should do the trick
If I update to the latest version and restart, and then unwind but without having the error pop up yet would that work? or unwinding has to be done only after getting the error?
(oh, I see you gave instructions on how to rewind above, but it wasn't needed with 2022.09.03)
Yes, you need to unwind so erigon doesn't count them as bad blocks anymore maybe restarting erigon will work but I don't remember well. Anyhow unwinding ~10 blocks should do the trick
If I update to the latest version and restart, and then unwind but without having the error pop up yet would that work? or unwinding has to be done only after getting the error?
Unwind if you were getting the problem bad block hash after the upgrade and restarting.
(oh, I see you gave instructions on how to rewind above, but it wasn't needed with 2022.09.03)
Yes, you need to unwind so erigon doesn't count them as bad blocks anymore maybe restarting erigon will work but I don't remember well. Anyhow unwinding ~10 blocks should do the trick
If I update to the latest version and restart, and then unwind but without having the error pop up yet would that work? or unwinding has to be done only after getting the error?
Unwind if you were getting the problem bad block hash after the upgrade and restarting.
Unfortunately it looks like its not working.
Erigon running https://github.com/ledgerwatch/erigon/pull/5510 (before unwinding):
[INFO] [09-26|05:09:14.818] [1/16 Snapshots] Fetching torrent files metadata
[INFO] [09-26|05:09:20.609] [txpool] Started
[INFO] [09-26|05:09:21.947] [Snapshots] Stat blocks=14000k indices=14000k alloc=2.3GB sys=2.4GB
[INFO] [09-26|05:09:21.949] [7/16 Execution] Blocks execution from=15613791 to=15613811
[WARN] [09-26|05:09:22.293] Served conn=127.0.0.1:48876 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=224.265µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-26|05:09:24.852] [7/16 Execution] Completed on block=15613811
[INFO] [09-26|05:09:24.852] [8/16 HashState] Promoting plain state from=15613791 to=15613811
[INFO] [09-26|05:09:24.852] [8/16 HashState] Incremental promotion from=15613791 to=15613811 codes=true csbucket=AccountChangeSet
[INFO] [09-26|05:09:25.286] [8/16 HashState] Incremental promotion from=15613791 to=15613811 codes=false csbucket=AccountChangeSet
[INFO] [09-26|05:09:25.409] [8/16 HashState] Incremental promotion from=15613791 to=15613811 codes=false csbucket=StorageChangeSet
[INFO] [09-26|05:09:25.726] [9/16 IntermediateHashes] Generating intermediate hashes from=15613791 to=15613811
[INFO] [09-26|05:09:27.787] Timings (slower than 50ms) Snapshots=7.132s Execution=2.903s HashState=866ms IntermediateHashes=1.341s CallTraces=173ms AccountHistoryIndex=107ms StorageHistoryIndex=176ms LogIndex=182ms TxLookup=83ms
[INFO] [09-26|05:09:27.787] RPC Daemon notified of new headers from=15613791 to=15613811 header sending=24.086µs log sending=170ns
[INFO] [09-26|05:09:27.787] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-26|05:09:36.556] [2/16 Headers] New payload: need to download parent height=15613875 hash=0x41ae068b098ad703b5d608f98cb29f0e5c077443c12f48dfe4419c1733195e19 parentHash=0x351c514f32ea28bcb70787f61752a797f234df409b80d4cabfe4545cbbc087bb
[INFO] [09-26|05:09:36.556] [2/16 Headers] Downloading PoS headers... height=15613874 hash=0x351c514f32ea28bcb70787f61752a797f234df409b80d4cabfe4545cbbc087bb requestId=1
[INFO] [09-26|05:09:36.556] Timings (slower than 50ms) Headers=8.769s
[INFO] [09-26|05:09:36.557] Tables PlainState=74.6GB AccountChangeSet=164.3GB StorageChangeSet=243.0GB BlockTransaction=162.5GB TransactionLog=453.1GB FreeList=9.7MB ReclaimableSpace=9.7GB
[INFO] [09-26|05:09:36.557] Commit cycle in=11.742µs
[INFO] [09-26|05:09:36.557] RPC Daemon notified of new headers from=15613811 to=15613812 header sending=7.885µs log sending=411ns
[WARN] [09-26|05:09:39.131] Could not validate block err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:657 collector.go:276 collector.go:144 stage_headers.go:670 stage_headers.go:717 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]"
[WARN] [09-26|05:09:39.136] Removing beacon request due to err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:657 collector.go:276 collector.go:144 stage_headers.go:670 stage_headers.go:717 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]" requestId=1
[INFO] [09-26|05:09:39.136] Timings (slower than 50ms) Headers=2.579s
[INFO] [09-26|05:09:39.136] Tables PlainState=74.6GB AccountChangeSet=164.3GB StorageChangeSet=243.0GB BlockTransaction=162.5GB TransactionLog=453.1GB FreeList=9.7MB ReclaimableSpace=9.7GB
[INFO] [09-26|05:09:39.136] Commit cycle in=7.504µs
[INFO] [09-26|05:09:39.136] RPC Daemon notified of new headers from=15613811 to=15613812 header sending=6.342µs log sending=341ns
[INFO] [09-26|05:09:39.136] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-26|05:09:43.560] New txs subscriber joined
[INFO] [09-26|05:09:43.567] new subscription to newHeaders established
[WARN] [09-26|05:09:48.062] [NewPayload] Previously known bad block hash=0x8c9c5d449a1018d94e28382ae2bc86c03004096e9296c7350660c7d2db302081 parentHash=0x41ae068b098ad703b5d608f98cb29f0e5c077443c12f48dfe4419c1733195e19
[WARN] [09-26|05:09:48.155] [ForkChoiceUpdated] Previously known bad block hash=0x8c9c5d449a1018d94e28382ae2bc86c03004096e9296c7350660c7d2db302081
[WARN] [09-26|05:09:58.522] [ForkChoiceUpdated] Previously known bad block hash=0x8c9c5d449a1018d94e28382ae2bc86c03004096e9296c7350660c7d2db302081
[WARN] [09-26|05:10:01.174] [NewPayload] Previously known bad block hash=0x137c7927cd032fcdf31f336854b359aa7017f67e313b9c0172c7fdd2ec0b6dcf parentHash=0x8c9c5d449a1018d94e28382ae2bc86c03004096e9296c7350660c7d2db302081
[WARN] [09-26|05:10:02.064] [ForkChoiceUpdated] Previously known bad block hash=0x137c7927cd032fcdf31f336854b359aa7017f67e313b9c0172c7fdd2ec0b6dcf
[WARN] [09-26|05:10:12.461] [NewPayload] Previously known bad block hash=0x871040dd5d3ed004439235d2ae97d34e9563f6b714aa9b32aa876c452f366695 parentHash=0x137c7927cd032fcdf31f336854b359aa7017f67e313b9c0172c7fdd2ec0b6dcf
[WARN] [09-26|05:10:12.685] [ForkChoiceUpdated] Previously known bad block hash=0x871040dd5d3ed004439235d2ae97d34e9563f6b714aa9b32aa876c452f366695
[INFO] [09-26|05:10:14.808] [txpool] stat block=15613811 pending=10000 baseFee=0 queued=28489 alloc=2.0GB sys=2.9GB
[WARN] [09-26|05:10:22.293] Served conn=127.0.0.1:38714 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=72.598µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[WARN] [09-26|05:10:23.852] [NewPayload] Previously known bad block hash=0x053a2ed4a4d7d2caf5e2643614c715f2e6a29cf1e202a93acd7c8d19b780b8a1 parentHash=0x871040dd5d3ed004439235d2ae97d34e9563f6b714aa9b32aa876c452f366695
[WARN] [09-26|05:10:23.943] [ForkChoiceUpdated] Previously known bad block hash=0x053a2ed4a4d7d2caf5e2643614c715f2e6a29cf1e202a93acd7c8d19b780b8a1
[WARN] [09-26|05:10:36.510] [NewPayload] Previously known bad block hash=0xb225435b73924856c1d55d07318dcc6821132052469da050c92b06b72cc69fe9 parentHash=0x053a2ed4a4d7d2caf5e2643614c715f2e6a29cf1e202a93acd7c8d19b780b8a1
[WARN] [09-26|05:10:36.810] [ForkChoiceUpdated] Previously known bad block hash=0xb225435b73924856c1d55d07318dcc6821132052469da050c92b06b72cc69fe9
unwinding 500 blocks:
INFO[09-26|05:11:45.076] Initialised chain configuration config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Terminal Total Difficulty: <nil>, Engine: ethash}"
INFO[09-26|05:11:57.324] trie account integrity key=020d0e030b00
INFO[09-26|05:12:07.325] trie account integrity key=050c030400
INFO[09-26|05:12:17.323] trie account integrity key=080a070e
INFO[09-26|05:12:27.325] trie account integrity key=0b08090700
INFO[09-26|05:12:37.325] trie account integrity key=0e06090800
INFO[09-26|05:12:47.325] trie storage integrity key=02b79774b3c1e64b8fcc557b1272c35f70d7a334a9fd14a672af9c5d39097fe70000000000000001000d09
INFO[09-26|05:12:57.322] trie storage integrity key=0db49b2fec5a0061f299c36d85fad9cc2a212f60dd12da06f9575758821681900000000000000001010f00
INFO[09-26|05:13:07.325] trie storage integrity key=137fad469e042d96bab5e8eb1e8635682e1fc5dc63491ced2ce4abd0d7ebb37b00000000000000010400
INFO[09-26|05:13:17.323] trie storage integrity key=1ad835ddf35c24036e1cc313e32da923979a82f6cb904b88a0bbb110b66cda1700000000000000010500
INFO[09-26|05:13:27.325] trie storage integrity key=26bcf90625b83c12354b4fa6909459b03dedd1f6c25f1d8e449cb35dfff4303a000000000000000100
INFO[09-26|05:13:37.325] trie storage integrity key=322e908feb0df7fbdd99b77b2a245096ee5c12bec695e93177ef74951087df650000000000000001010d00
INFO[09-26|05:13:47.325] trie storage integrity key=3e6f212888faf365156e5962988c217db6b85c41e09deaf5c03ef9c819aabece000000000000000100
INFO[09-26|05:13:57.323] trie storage integrity key=490ee429867dd3e7dcf8d1e2c9959f845bf5ad5b49c2bfdd1309402f2dda9d20000000000000000100
INFO[09-26|05:14:07.325] trie storage integrity key=4ffa622476dd15d4e2d2154c62b80d469ddd470443d165a4f103e7c26a4525c00000000000000001070d00
INFO[09-26|05:14:17.323] trie storage integrity key=5d5409697818f282a90a8b0f9f3fc74d04cb4c43c90741b5b955dba454b4503700000000000000010800
INFO[09-26|05:14:27.324] trie storage integrity key=6917989b98d7fa020e807b588f05fa0c4b55fdc45d429f384bee7cf53f0bfda200000000000000010a00
INFO[09-26|05:14:37.323] trie storage integrity key=71e9e01e9e60e18fac4e84478f7e1accc7a209e4d0ccc988d6afe8bbdeee663e0000000000000001020e00
INFO[09-26|05:14:47.325] trie storage integrity key=7b62c8f361fa9a10f2f8db0ec19abeecd6a5d0954f9ce77b7272a01416e2e03900000000000000010100
INFO[09-26|05:14:57.324] trie storage integrity key=8679e8eda65bd257638cf8cf09b8238888947cc3c0bea2aa2cc3f1c4ac7a3002000000000000000109010c00
INFO[09-26|05:15:07.325] trie storage integrity key=8f21cbab2122844bab193637836f157f375930076d93c62439e5eb521e89531c00000000000000010200
INFO[09-26|05:15:17.323] trie storage integrity key=9ba8d10ddc872580ed3146e0b639b103935d2a6c3ab3c4d5a4aebe058aebbe070000000000000001080c0800
INFO[09-26|05:15:27.323] trie storage integrity key=a220237f15a972695a71baa47a769d806ef2e20a9ec17105c3866c57b72086db000000000000000106
INFO[09-26|05:15:37.324] trie storage integrity key=aad3a5d2a7f82e424cf82ff9fe581e405b0bd0eb8a273161e73e14cf5eba1ece000000000000000108020207
INFO[09-26|05:15:47.322] trie storage integrity key=b1caacaec12ad118a966be44d5179449ab5258510b7659dc7eb24d3a9645e89b00000000000000010a00
INFO[09-26|05:15:57.325] trie storage integrity key=bf5fd985a465e1ea99059d01f042b415ab24db831264b882f246eac49aedcd340000000000000001070d00
INFO[09-26|05:16:07.323] trie storage integrity key=cca195e30f6f6cf2132f200a9bd5f18ed64faf147231293fa04f05297f960953000000000000000103060908
INFO[09-26|05:16:17.323] trie storage integrity key=d7c760e579a8e7be6c42ceb96af1f09aca58e94d7e9ddcf1adaac662140aaa0800000000000000010d0800
INFO[09-26|05:16:27.325] trie storage integrity key=e510597a893494dfc2e0760dd30f2d8b05151af00790f23639052db63eb178cf000000000000000100
INFO[09-26|05:16:37.325] trie storage integrity key=f02ea35ca750d6478d33e19c95e6ad53ae88d568953e70be84e9427db92c834400000000000000010c
INFO[09-26|05:16:47.322] trie storage integrity key=fbd1592b616670c3c966b8ed9942aecdb2b25ab10837f7d1c8d1e66c10f243a800000000000000010905030001
INFO[09-26|05:16:57.324] trie storage integrity key=ff8d2f5906268c29014ad4d7eb5b13ff3c6eed2a5cb71a32c29b51a0c2705bc5000000000000000100
INFO[09-26|05:16:57.906] UnwindTo block=15613311 bad_block_hash=0x0000000000000000000000000000000000000000000000000000000000000000
INFO[09-26|05:17:57.304] [10/16 CallTraces] Unwind from=15613811 to=15613311
INFO[09-26|05:18:23.538] [8/16 HashState] Unwinding started from=15613811 to=15613311 storage=false codes=true
INFO[09-26|05:18:26.278] [8/16 HashState] Unwinding started from=15613811 to=15613311 storage=false codes=false
INFO[09-26|05:18:38.214] [8/16 HashState] Unwinding started from=15613811 to=15613311 storage=true codes=false
INFO[09-26|05:19:00.977] [9/16 IntermediateHashes] Unwinding of trie hashes from=15613811 to=15613311 csbucket=AccountChangeSet
INFO[09-26|05:19:06.849] [9/16 IntermediateHashes] Unwinding of trie hashes from=15613811 to=15613311 csbucket=StorageChangeSet
INFO[09-26|05:19:39.688] [9/16 IntermediateHashes] Trie root hash=0xfebb4771ce5665c440bd687e63eb0710a9c13458d384eb80c392d08eb8f168b0
INFO[09-26|05:20:05.976] [9/16 IntermediateHashes] Unwind done in=1m5.005958193s
INFO[09-26|05:20:05.986] [6/16 Execution] Unwind Execution from=15613811 to=15613311
INFO[09-26|05:20:23.423] Timings (slower than 50ms) Unwind TxLookup=10.89s Unwind LogIndex=14.99s Unwind StorageHistoryIndex=18.894s Unwind AccountHistoryIndex=14.62s Unwind CallTraces=26.226s Unwind HashState=37.439s Unwind IntermediateHashes=1m5.005s Unwind Execution=17.435s
INFO[09-26|05:20:23.423] Tables freelist=9.7Mb PlainState=74.6Gb AccountChangeSet=164.3Gb StorageChangeSet=243.0Gb BlockTransaction=162.5Gb TransactionLog=453.0Gb
INFO[09-26|05:20:33.426] trie account integrity key=0301010700
INFO[09-26|05:20:43.425] trie account integrity key=06010e0f00
INFO[09-26|05:20:53.426] trie account integrity key=0904040600
INFO[09-26|05:21:03.426] trie account integrity key=0c06060e0f0d
INFO[09-26|05:21:13.423] trie account integrity key=0f080d0f0601
INFO[09-26|05:21:23.427] trie storage integrity key=0981bb62d3a453ecb3d0546d74a8f65f903cd08d6145f1b5f20de62b39f41c1b0000000000000001070a00
INFO[09-26|05:21:33.426] trie storage integrity key=0f27a427a06c148ea9a08f515db9ba0cdcfc7260580a2e512385db59487acd140000000000000001030b00
INFO[09-26|05:21:43.426] trie storage integrity key=15e0cc6b076b1bc80f88aeb79196865eb14834fa12c0395cba0849276155cd33000000000000000100
INFO[09-26|05:21:53.426] trie storage integrity key=217a2960d7545d6a27f2b2679a1146ed57d4910f978c0c690b9ea18cd8dd8dd100000000000000010f0a00
INFO[09-26|05:22:03.426] trie storage integrity key=2d3153a55430bd0cbddeaa7e1cf1616b688c6071e285bc5b726d918fac6d4d7e00000000000000010600
INFO[09-26|05:22:13.426] trie storage integrity key=3a55e670852b2b985cd8c252fddae2acb4f798d0fecdcb1e2da53726332eb5590000000000000001030000
INFO[09-26|05:22:23.427] trie storage integrity key=475eb5b9905a00dbc4198ce5e20965fa51618eec8924a0e09429109db9d2067f000000000000000100
INFO[09-26|05:22:33.423] trie storage integrity key=4e6f19bdf5c8f4aa5e46a382f33fbed981bd7d93c25885c5ecdabfdc102f8b6700000000000000010a0f0602030d
INFO[09-26|05:22:43.426] trie storage integrity key=58eb9d452c1ce2e53e1a11dbccf18dbec587327f0877dda7635caf202378f513000000000000000100
INFO[09-26|05:22:53.426] trie storage integrity key=6607f50cc1b6f613129603962b50a0eabfafbf375ff7fcaef220871ce9cfc52500000000000000010900
INFO[09-26|05:23:03.426] trie storage integrity key=7006375ac2fae9614f6258206c1e86e5112c551b246dd6efb4783aa4b1b62d750000000000000001030f0a00
INFO[09-26|05:23:13.426] trie storage integrity key=7a261c3eff71369847f3e043518d339190854ad32dec02767fdcc40a48596997000000000000000100
INFO[09-26|05:23:23.426] trie storage integrity key=8529d15c388a38e67904920127d61ba5d60e91749766c93679f6f2cff96ef57e00000000000000010e0c00
INFO[09-26|05:23:33.426] trie storage integrity key=8d72d2cf0fcf989c0231daf115c4afbd4bae7b8191a8444796f5a660b7618283000000000000000100
INFO[09-26|05:23:43.427] trie storage integrity key=9b44f9cb7992ac44101740361d95817192e4f081628ef8dfe437c351a84f3e1f0000000000000001020f00
INFO[09-26|05:23:53.426] trie storage integrity key=9f13f88230a70de90ed5fa41ba35a5fb78bc55d11cc9406f17d314fb67047ac700000000000000010f0c0300
INFO[09-26|05:24:03.427] trie storage integrity key=a7273be56697277ab914acabb63e12eb97a50318552a915e475f57d5c4b8899800000000000000010400
INFO[09-26|05:24:13.426] trie storage integrity key=aeaf07b87dc2703648c1f01ca9230bb0ee01f69b67a5c9b0d46990d09689ce8a00000000000000010a0600
INFO[09-26|05:24:23.426] trie storage integrity key=baf01928b5933b00277721188ff9a085a6e3e0f97e627ac3946da61936c6be8f000000000000000100
INFO[09-26|05:24:33.427] trie storage integrity key=c4bdba36c15ad8c14f39a4ae4d3f864c7ccb645bb68ebaaaa868bcd63460894c000000000000000100
INFO[09-26|05:24:43.427] trie storage integrity key=d09832ca53e8a0e441d4dc5101efa6a123b85a3a2a219f1a5fd625a5d66144900000000000000001090900
INFO[09-26|05:24:53.426] trie storage integrity key=dadfdd2aa01ebd96d7888f09d31bedacd0611c537950a6e57eb8d049706da1d3000000000000000100
INFO[09-26|05:25:03.426] trie storage integrity key=e6711e2e67fe259fdc0edee04f3c810dc65e824096a81c25c982bc374cce2d130000000000000001
INFO[09-26|05:25:13.427] trie storage integrity key=efd3b82f5132bc8b27df93bb69c11f5228245fe547bc009fd8d296518615dc76000000000000000100
INFO[09-26|05:25:23.427] trie storage integrity key=fbd1592b616670c3c966b8ed9942aecdb2b25ab10837f7d1c8d1e66c10f243a8000000000000000106060600
INFO[09-26|05:25:33.428] trie storage integrity key=fe35eb2a62be983e71c0fd83e4eb62a1ec45bdd6601ebd77546946363f985130000000000000000100
Erigon running https://github.com/ledgerwatch/erigon/pull/5510 (after unwinding):
[INFO] [09-26|05:26:36.823] [1/16 Snapshots] Fetching torrent files metadata
[INFO] [09-26|05:26:41.774] [txpool] Started
[INFO] [09-26|05:26:43.446] [Snapshots] Stat blocks=14000k indices=14000k alloc=2.2GB sys=2.3GB
[INFO] [09-26|05:26:43.448] [7/16 Execution] Blocks execution from=15613311 to=15613811
[INFO] [09-26|05:27:03.517] [7/16 Execution] Executed blocks number=15613600 blk/s=14.4 tx/s=2138.1 Mgas/s=214.6 gasState=0.01 estimated duration=34.806s batch=7.7MB alloc=2.4GB sys=3.7GB
[INFO] [09-26|05:27:17.213] [7/16 Execution] Completed on block=15613811
[INFO] [09-26|05:27:17.214] [8/16 HashState] Promoting plain state from=15613311 to=15613811
[INFO] [09-26|05:27:17.214] [8/16 HashState] Incremental promotion from=15613311 to=15613811 codes=true csbucket=AccountChangeSet
[INFO] [09-26|05:27:19.931] [8/16 HashState] Incremental promotion from=15613311 to=15613811 codes=false csbucket=AccountChangeSet
[INFO] [09-26|05:27:21.138] [8/16 HashState] Incremental promotion from=15613311 to=15613811 codes=false csbucket=StorageChangeSet
[WARN] [09-26|05:27:22.336] Served conn=127.0.0.1:48154 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=284.259µs err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-26|05:27:25.950] [9/16 IntermediateHashes] Generating intermediate hashes from=15613311 to=15613811
[INFO] [09-26|05:27:36.836] [txpool] stat block=15613811 pending=9999 baseFee=0 queued=28587 alloc=2.8GB sys=4.6GB
[INFO] [09-26|05:27:56.413] [13/16 LogIndex] processing from=15613312 to=15613811
[INFO] [09-26|05:28:14.440] Timings (slower than 50ms) Snapshots=6.626s Execution=33.765s HashState=8.728s IntermediateHashes=16.353s CallTraces=1.968s AccountHistoryIndex=2.364s StorageHistoryIndex=9.784s LogIndex=7.693s TxLookup=10.319s
[INFO] [09-26|05:28:14.447] RPC Daemon notified of new headers from=15613311 to=15613811 header sending=6.404543ms log sending=281ns
[INFO] [09-26|05:28:14.448] [2/16 Headers] Waiting for Consensus Layer...
[WARN] [09-26|05:28:22.345] Served conn=127.0.0.1:35862 method=engine_exchangeTransitionConfigurationV1 reqid=1 t=7.399319ms err="the execution layer doesn't have a terminal total difficulty. expected: 0xc70d808a128d7380000"
[INFO] [09-26|05:28:24.359] [2/16 Headers] New payload: need to download parent height=15613969 hash=0x50f70e20b2de2fb222e447e993521f422957746ddfec6c5ccab817b453be9747 parentHash=0xd99c297d047729aff21059db9c1da488e4117b687a015bc2ae319df4cdf27de7
[INFO] [09-26|05:28:24.359] [2/16 Headers] Downloading PoS headers... height=15613968 hash=0xd99c297d047729aff21059db9c1da488e4117b687a015bc2ae319df4cdf27de7 requestId=1
[INFO] [09-26|05:28:24.360] Timings (slower than 50ms) Headers=9.911s
[INFO] [09-26|05:28:24.360] Tables PlainState=74.6GB AccountChangeSet=164.3GB StorageChangeSet=243.0GB BlockTransaction=162.5GB TransactionLog=453.1GB FreeList=9.6MB ReclaimableSpace=9.6GB
[INFO] [09-26|05:28:24.360] Commit cycle in=5.079µs
[INFO] [09-26|05:28:24.361] RPC Daemon notified of new headers from=15613811 to=15613812 header sending=5.039µs log sending=310ns
[WARN] [09-26|05:28:28.701] Could not validate block err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:657 collector.go:276 collector.go:144 stage_headers.go:670 stage_headers.go:717 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]"
[WARN] [09-26|05:28:28.707] Removing beacon request due to err="Not implemented, trace: [stageloop.go:262 panic.go:884 memory_mutation.go:277 stage_bodies.go:138 default_stages.go:260 sync.go:352 sync.go:255 stageloop.go:355 backend.go:333 fork_validator.go:258 fork_validator.go:216 stage_headers.go:657 collector.go:276 collector.go:144 stage_headers.go:670 stage_headers.go:717 stage_headers.go:175 stage_headers.go:142 default_stages.go:36 sync.go:352 sync.go:255 stageloop.go:169 stageloop.go:90 asm_amd64.s:1594]" requestId=1
[INFO] [09-26|05:28:28.707] Timings (slower than 50ms) Headers=4.346s
[INFO] [09-26|05:28:28.707] Tables PlainState=74.6GB AccountChangeSet=164.3GB StorageChangeSet=243.0GB BlockTransaction=162.5GB TransactionLog=453.1GB FreeList=9.6MB ReclaimableSpace=9.6GB
[INFO] [09-26|05:28:28.707] Commit cycle in=4.469µs
[INFO] [09-26|05:28:28.707] RPC Daemon notified of new headers from=15613811 to=15613812 header sending=4.298µs log sending=310ns
[INFO] [09-26|05:28:28.707] [2/16 Headers] Waiting for Consensus Layer...
[INFO] [09-26|05:28:35.756] [p2p] GoodPeers eth66=2
[WARN] [09-26|05:28:35.885] [NewPayload] Previously known bad block hash=0x37a1f6179bf1d025d9a52835cb0adc49f37fffd4901f0f7b8297dd5ee4a15ca9 parentHash=0x50f70e20b2de2fb222e447e993521f422957746ddfec6c5ccab817b453be9747
[WARN] [09-26|05:28:36.022] [ForkChoiceUpdated] Previously known bad block hash=0x37a1f6179bf1d025d9a52835cb0adc49f37fffd4901f0f7b8297dd5ee4a15ca9
[INFO] [09-26|05:28:36.816] [txpool] stat block=15613811 pending=10000 baseFee=0 queued=28844 alloc=3.0GB sys=4.8GB
[WARN] [09-26|05:28:48.821] [NewPayload] Previously known bad block hash=0xcd617b17e8bce0346e274739572c12310188dd592e11af4fae31e102516c3a76 parentHash=0x37a1f6179bf1d025d9a52835cb0adc49f37fffd4901f0f7b8297dd5ee4a15ca9
[WARN] [09-26|05:28:49.037] [ForkChoiceUpdated] Previously known bad block hash=0xcd617b17e8bce0346e274739572c12310188dd592e11af4fae31e102516c3a76
[WARN] [09-26|05:29:00.593] [NewPayload] Previously known bad block hash=0x55fafc40fc1237486d3b5552d9ea8d0b513fcca0200866766714b4a1a6dd3fbd parentHash=0xcd617b17e8bce0346e274739572c12310188dd592e11af4fae31e102516c3a76
[WARN] [09-26|05:29:00.677] [ForkChoiceUpdated] Previously known bad block hash=0x55fafc40fc1237486d3b5552d9ea8d0b513fcca0200866766714b4a1a6dd3fbd
[WARN] [09-26|05:29:10.516] [ForkChoiceUpdated] Previously known bad block hash=0x55fafc40fc1237486d3b5552d9ea8d0b513fcca0200866766714b4a1a6dd3fbd
Note: I probably run an old version of integration since I didn't build it this time. Probably it doesn't matter but let me know if it does.
Yes, you need to unwind so erigon doesn't count them as bad blocks anymore maybe restarting erigon will work but I don't remember well. Anyhow unwinding ~10 blocks should do the trick
As I've already stated, I did not need to manually unwind when using 2022.09.03. So a new bug must have been introduced (and not yet fixed) between 2022.09.03 and #5510.
Note: I probably run an old version of
integrationsince I didn't build it this time. Probably it doesn't matter but let me know if it does.
I tried once more with everything in the newest version, and things are looking good for now... Fingers crossed, I will update if the issue persists.
Edit: the node has been running fine for a few days now, so I think the issue is fixed! Thanks a lot
I've run unwind but it keeps halting.
[INFO] [10-08|02:15:47.567] Commit cycle in=11.73884ms
[INFO] [10-08|02:15:47.567] RPC Daemon notified of new headers from=15692870 to=15692931 header sending=76.941µs log sending=340ns
[INFO] [10-08|02:15:47.567] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:15:47.580] [5/7 Execution] Completed on block=15692932
[INFO] [10-08|02:15:47.602] Commit cycle in=9.672726ms
[INFO] [10-08|02:15:47.602] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=58.32µs log sending=230ns
[INFO] [10-08|02:15:47.602] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:15:47.776] [5/7 Execution] Completed on block=15692933
[WARN] [10-08|02:15:47.961] Could not validate block err="[7/7 IntermediateHashes] 7/7 IntermediateHashes: put: k=ab14d68802a763f7db875346d03fbf86f137de55814b191c069e721f47474733000000000000000101030a0407, table: TrieStorage, err: mdbx_cursor_put: MDBX_MAP_FULL: Environment mapsize limit reached"
[WARN] [10-08|02:15:47.984] Validation failed for header hash=0xf9a6d5d63d6725f82b5141df5fed1ea12ef3a9b9a1b16a75e8f5954ae2e60eea height=15692933 err="[7/7 IntermediateHashes] 7/7 IntermediateHashes: put: k=ab14d68802a763f7db875346d03fbf86f137de55814b191c069e721f47474733000000000000000101030a0407, table: TrieStorage, err: mdbx_cursor_put: MDBX_MAP_FULL: Environment mapsize limit reached"
[INFO] [10-08|02:15:47.984] Timings (slower than 50ms) Headers=381ms
[INFO] [10-08|02:15:47.984] Tables PlainState=75.5GB AccountChangeSet=1.5GB StorageChangeSet=2.7GB BlockTransaction=21.8GB TransactionLog=7.1GB FreeList=187.2MB ReclaimableSpace=187.2GB
[INFO] [10-08|02:15:47.984] Commit cycle in=15.281µs
[INFO] [10-08|02:15:47.984] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=57.68µs log sending=220ns
[INFO] [10-08|02:15:47.984] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:15:49.772] [txpool] stat block=15692933 pending=10000 baseFee=30000 queued=30000 alloc=3.0GB sys=4.7GB
[INFO] [10-08|02:16:29.194] Timings (slower than 50ms) Headers=41.208s
[INFO] [10-08|02:16:29.194] Tables PlainState=75.5GB AccountChangeSet=1.5GB StorageChangeSet=2.7GB BlockTransaction=21.8GB TransactionLog=7.1GB FreeList=187.2MB ReclaimableSpace=187.2GB
[INFO] [10-08|02:16:29.194] Commit cycle in=10.99µs
[INFO] [10-08|02:16:29.194] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=82.151µs log sending=330ns
[INFO] [10-08|02:16:29.194] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.202] Commit cycle in=7.52µs
[INFO] [10-08|02:16:29.202] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=54.771µs log sending=170ns
[INFO] [10-08|02:16:29.202] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.214] Commit cycle in=10.051µs
[INFO] [10-08|02:16:29.214] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=84.891µs log sending=240ns
[INFO] [10-08|02:16:29.214] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.227] Commit cycle in=9.311µs
[INFO] [10-08|02:16:29.227] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=81.43µs log sending=220ns
[INFO] [10-08|02:16:29.227] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.236] Commit cycle in=7.81µs
[INFO] [10-08|02:16:29.236] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=73.461µs log sending=160ns
[INFO] [10-08|02:16:29.236] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.249] Commit cycle in=8.36µs
[INFO] [10-08|02:16:29.249] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=75.62µs log sending=210ns
[INFO] [10-08|02:16:29.249] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.252] Commit cycle in=7.67µs
[INFO] [10-08|02:16:29.252] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=70.31µs log sending=200ns
[INFO] [10-08|02:16:29.253] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.271] Commit cycle in=7.84µs
[INFO] [10-08|02:16:29.271] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=65.111µs log sending=210ns
[INFO] [10-08|02:16:29.271] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.283] Commit cycle in=8.84µs
[INFO] [10-08|02:16:29.283] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=63.07µs log sending=210ns
[INFO] [10-08|02:16:29.283] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.287] Commit cycle in=7.71µs
[INFO] [10-08|02:16:29.287] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=64.26µs log sending=180ns
[INFO] [10-08|02:16:29.287] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.298] Commit cycle in=6.57µs
[INFO] [10-08|02:16:29.298] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=51.4µs log sending=150ns
[INFO] [10-08|02:16:29.298] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.306] Commit cycle in=9.521µs
[INFO] [10-08|02:16:29.306] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=75.08µs log sending=200ns
[INFO] [10-08|02:16:29.306] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.321] Commit cycle in=8.39µs
[INFO] [10-08|02:16:29.322] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=65.23µs log sending=190ns
[INFO] [10-08|02:16:29.322] [1/16 Headers] Waiting for Beacon Chain...
[INFO] [10-08|02:16:29.334] Commit cycle in=9.64µs
[INFO] [10-08|02:16:29.335] RPC Daemon notified of new headers from=15692870 to=15692932 header sending=75.681µs log sending=210ns
[INFO] [10-08|02:16:29.335] [1/16 Headers] Waiting for Beacon Chain...
I got the same problem on v2.29.0. I think it was caused by an unstable DSL connection, and it couldn't recover
Edit: rewinding the blocks as described above resolved the issue, but it's concerning that erigon couldn't recover by itself
I have a similar issue, however, Erigon (v2.30.0) is haulted with this error log (not really an "error" per say): [2/16 Headers] Waiting for Consensus Layer...
I am running a Consensus Layer (Lighthouse) and I figure this message is just because the Beacon Chain is not yet synced that Erigon is stopped. Any thoughts on this from more informed uses?
This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been stalled for 7 days with no activity.