prysm
prysm copied to clipboard
prysmaticlabs
Merged testnet doesn't sync back after being offline for a few days
🐞 Bug Report
Description
My issue is similar to this, but the error messages are not quite the same: https://github.com/prysmaticlabs/prysm/issues/11279
I have a Goeli/Prater testnet running for some time. I don't usually keep it up 100% of the time, I usually just stop geth and prysm just so that 1) it doesn't use much of my Internet 2) I can test how it syncs back after being down.
I've done this up and down cycles for a while, the setup merged and moved to PoS when that happened, I kept it up for a couple of days, then stopped it. I started everything again after a week or so, but now it doesn't seem to be syncing back again like it used to.
Has this worked before in a previous version?
Yes, although I don't think it's version related, more like a pre/post merge thing.
🔬 Minimal Reproduction
- Have a Geth/Prysm beacon pair running fine, merged using PoS
- Stop the services and let a few days go by
- Start everything back up, now it's not syncing
🔥 Error
This is what's been logging for a few hours now:
beacon-testnet_1 | time="2022-08-22 15:29:54" level=info msg="Processing block batch of size 41 starting from 0xde547ff2... 3639936/3722849 - estimated time remaining 11h14m5s" blocksPerSecond=2.0 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:57" level=warning msg="Batch is not processed" error="could not process block in batch: could not retrieve block roots: invalid end slot and start slot provided" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:57" level=info msg="Processing block batch of size 41 starting from 0x760fad54... 3639999/3722849 - estimated time remaining 5h36m47s" blocksPerSecond=4.1 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="could not process block in batch: could not fill in missing blocks to forkchoice: not descendant of finalized checkpoint" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=info msg="Processing block batch of size 41 starting from 0x9a57fc36... 3640063/3722849 - estimated time remaining 3h44m21s" blocksPerSecond=6.2 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xe55bf00010af74b59978b0b66e6317297ef5922cbeb67f6ec3ce9a8f14a1e08f (in processBatchedBlocks, slot=3640063)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=info msg="Processing block batch of size 44 starting from 0x85aba678... 3640129/3722849 - estimated time remaining 2h45m6s" blocksPerSecond=8.3 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x72271da8634dcb0839757352bda0bd911e54c2a129a47e33e2201c5c73d28ebf (in processBatchedBlocks, slot=3640129)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=info msg="Processing block batch of size 41 starting from 0x722fee7b... 3640191/3722849 - estimated time remaining 2h12m27s" blocksPerSecond=10.4 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8140fc050dd491e32a300f8bc29f6e23e439b0a6770359871e939a337150a300 (in processBatchedBlocks, slot=3640191)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=info msg="Processing block batch of size 24 starting from 0x968a6d54... 3640255/3722849 - estimated time remaining 1h58m40s" blocksPerSecond=11.6 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x10ddc824797a7190d8bc208eae7c27340ef8900c5839e6d50d942a7c84b06a35 (in processBatchedBlocks, slot=3640255)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=info msg="Processing block batch of size 44 starting from 0x6ef9d899... 3640321/3722849 - estimated time remaining 1h39m40s" blocksPerSecond=13.8 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xad991e44a3ff970e193c78454920fd503d278e233810729ee9a59ed089557be3 (in processBatchedBlocks, slot=3640321)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=info msg="Processing block batch of size 47 starting from 0x3948fa5f... 3640386/3722849 - estimated time remaining 1h25m6s" blocksPerSecond=16.1 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:58" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x0f1870f8fa53698dffa0b8189b77dcedb26f9a5c21d75d2928447ba82233204e (in processBatchedBlocks, slot=3640386)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:29:59" level=warning msg="Unable to cache headers for execution client votes" error="(1661105248 > 1660188288): provided time is later than the current eth1 head" prefix=powchain
beacon-testnet_1 | time="2022-08-22 15:30:14" level=warning msg="Unable to cache headers for execution client votes" error="(1661105248 > 1660188288): provided time is later than the current eth1 head" prefix=powchain
beacon-testnet_1 | time="2022-08-22 15:30:20" level=info msg="Processing block batch of size 41 starting from 0xde547ff2... 3639936/3722851 - estimated time remaining 11h14m6s" blocksPerSecond=2.0 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:24" level=warning msg="Batch is not processed" error="could not process block in batch: could not retrieve block roots: invalid end slot and start slot provided" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:24" level=info msg="Processing block batch of size 41 starting from 0x760fad54... 3639999/3722852 - estimated time remaining 5h36m48s" blocksPerSecond=4.1 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="could not process block in batch: could not fill in missing blocks to forkchoice: not descendant of finalized checkpoint" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=info msg="Processing block batch of size 41 starting from 0x9a57fc36... 3640063/3722852 - estimated time remaining 3h44m21s" blocksPerSecond=6.2 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xe55bf00010af74b59978b0b66e6317297ef5922cbeb67f6ec3ce9a8f14a1e08f (in processBatchedBlocks, slot=3640063)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=info msg="Processing block batch of size 44 starting from 0x85aba678... 3640129/3722852 - estimated time remaining 2h45m6s" blocksPerSecond=8.3 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x72271da8634dcb0839757352bda0bd911e54c2a129a47e33e2201c5c73d28ebf (in processBatchedBlocks, slot=3640129)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=info msg="Processing block batch of size 41 starting from 0x722fee7b... 3640191/3722852 - estimated time remaining 2h12m28s" blocksPerSecond=10.4 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8140fc050dd491e32a300f8bc29f6e23e439b0a6770359871e939a337150a300 (in processBatchedBlocks, slot=3640191)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=info msg="Processing block batch of size 43 starting from 0x968a6d54... 3640255/3722852 - estimated time remaining 1h49m41s" blocksPerSecond=12.6 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x10ddc824797a7190d8bc208eae7c27340ef8900c5839e6d50d942a7c84b06a35 (in processBatchedBlocks, slot=3640255)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=info msg="Processing block batch of size 44 starting from 0x6ef9d899... 3640321/3722852 - estimated time remaining 1h33m15s" blocksPerSecond=14.8 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xad991e44a3ff970e193c78454920fd503d278e233810729ee9a59ed089557be3 (in processBatchedBlocks, slot=3640321)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=info msg="Processing block batch of size 47 starting from 0x3948fa5f... 3640386/3722852 - estimated time remaining 1h20m22s" blocksPerSecond=17.1 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x0f1870f8fa53698dffa0b8189b77dcedb26f9a5c21d75d2928447ba82233204e (in processBatchedBlocks, slot=3640386)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-22 15:30:29" level=warning msg="Unable to cache headers for execution client votes" error="(1661105248 > 1660188288): provided time is later than the current eth1 head" prefix=powchain
🌍 Your Environment
Operating System:
Ubuntu 22.04.1 LTS
What version of Prysm are you running? (Which release)
2.1.4
Anything else relevant (validator index / public key)?
Let me know I can provide more info to diagnose, this worries me a bit for the mainnet setup.
Is just starting the services up again enough to have both EL/CL sync up like it used to happen, should that be done in an order? both together? I already tried restarting everything, multiple times, in different order to no avail.
How would something like this get fixed in case it was DB corruption now that EL/CL work together? Can I start prysm from scratch? should I do prysm and geth both from the beginning? Here's what geth is logging in case it helps:
geth-testnet_1 | WARN [08-22|15:34:46.283] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:34:46.284] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:34:46.288] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:34:46.304] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:34:46.315] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:34:46.328] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:34:46.335] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.073] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.075] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.078] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.088] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.091] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.097] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.111] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.111] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.118] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.132] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.134] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.137] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.141] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.144] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.157] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.157] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.162] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.178] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.189] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.201] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:03.208] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.053] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.056] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.059] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.069] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.073] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.077] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.089] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.090] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.098] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.113] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.116] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.119] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.122] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.126] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.139] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.140] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.144] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.160] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.171] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.183] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:23.190] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.032] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.034] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.038] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.049] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.052] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.057] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.069] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.069] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.076] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.092] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.096] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.100] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.103] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.107] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.119] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.120] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.124] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.140] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.150] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.162] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:35:39.170] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:09.632] Previously seen beacon client is offline. Please ensure it is operational to follow the chain!
geth-testnet_1 | WARN [08-22|15:36:20.679] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.681] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.684] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.695] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.698] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.703] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.715] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.715] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.722] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.738] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.742] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.746] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.749] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.753] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.766] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.766] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.773] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.789] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.800] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.813] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:20.820] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.668] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.670] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.673] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.684] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.687] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.692] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.704] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.704] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.711] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.725] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.728] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.731] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.734] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.738] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.750] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.751] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.755] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.772] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.783] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.795] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:33.802] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.388] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.390] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.393] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.403] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.407] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.412] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.424] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.424] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.431] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.445] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.447] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.450] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.453] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.457] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.470] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.471] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.475] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.491] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.502] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.514] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:36:47.521] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.838] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.840] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.843] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.853] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.856] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.861] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.873] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.873] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.880] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.894] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.896] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.899] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.903] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.906] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.919] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.920] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.924] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.940] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.951] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.964] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:01.971] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.002] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.004] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.008] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.018] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.022] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.027] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.039] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.039] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.047] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.061] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.063] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.066] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.069] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.073] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.086] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.086] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.091] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.109] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.120] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.132] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:26.140] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.247] Ignoring already known beacon payload number=7,383,108 hash=3640d5..9a2bcb age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.249] Ignoring already known beacon payload number=7,383,109 hash=f69b83..b79c46 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.252] Ignoring already known beacon payload number=7,383,110 hash=66256a..f96820 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.263] Ignoring already known beacon payload number=7,383,111 hash=d383ca..134bf4 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.266] Ignoring already known beacon payload number=7,383,112 hash=d6a293..90acc9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.271] Ignoring already known beacon payload number=7,383,113 hash=77c97f..18501e age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.283] Ignoring already known beacon payload number=7,383,114 hash=9bb975..fe55ce age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.284] Ignoring already known beacon payload number=7,383,115 hash=7fe9dd..982401 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.291] Ignoring already known beacon payload number=7,383,116 hash=bfd093..1a6a0c age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.305] Ignoring already known beacon payload number=7,383,117 hash=a0e3dc..280634 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.307] Ignoring already known beacon payload number=7,383,118 hash=7137a1..ab4991 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.310] Ignoring already known beacon payload number=7,383,119 hash=554999..f479f1 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.314] Ignoring already known beacon payload number=7,383,120 hash=e56265..42c853 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.317] Ignoring already known beacon payload number=7,383,121 hash=0af61b..0a3e36 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.330] Ignoring already known beacon payload number=7,383,122 hash=0dd2e5..0ad224 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.331] Ignoring already known beacon payload number=7,383,123 hash=53625c..dd186b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.335] Ignoring already known beacon payload number=7,383,124 hash=189005..906c9b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.352] Ignoring already known beacon payload number=7,383,125 hash=ace8ed..4b9ff9 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.363] Ignoring already known beacon payload number=7,383,126 hash=3bf2af..3f89c7 age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.374] Ignoring already known beacon payload number=7,383,127 hash=86136d..a7335b age=1w4d12h
geth-testnet_1 | WARN [08-22|15:37:54.382] Ignoring already known beacon payload number=7,383,128 hash=00e189..004295 age=1w4d12h
Following up on that other issue, here's my finalized state, to help diagnosing
curl -H "Accept: application/octet-stream" http://localhost:3500/eth/v1/debug/beacon/states/finalized -o finalized.ssz
https://share.ipfs.io/#/bafybeihudylnjqzj6kxe7ffbydal3rivs43qzelettyanjfp55ndlbmdky
Hey @pablote , your issue seems different from the other issue. Do you have all your logs during this period ? Ex: before offline period, during offline period, after back online period
No sorry, I don't keep logs of before the offline period, it was just running fine then. And after starting things back again it's logging what I posted continously.
No worries, it would be helpful if you had the logs of when it first started going wrong. As usually it would give an indication of what is wrong with the node. But in any case, we can continue investigating this on our end.
Maybe I didn't explain correctly the scenario. There's no point in time, and in the logs, in which Prysm beacon goes from working well to stop syncing with these messages. Prysm was working well, it was stopped for a week, and then started again. It's when starting it again that it can't seem to sync back. I'm attaching logs from service startup:
beacon-testnet_1 | time="2022-08-23 13:25:25" level=warning msg="Running on the Prater Testnet" prefix=flags
beacon-testnet_1 | time="2022-08-23 13:25:25" level=warning msg="Enabled feature flag" enable-vectorized-htr="Enables new go sha256 library which utilizes optimized routines for merkle trees" prefix=flags
beacon-testnet_1 | time="2022-08-23 13:25:25" level=warning msg="Enabled feature flag" enable-forkchoice-doubly-linked-tree="Enables new forkchoice store structure that uses doubly linked trees" prefix=flags
beacon-testnet_1 | time="2022-08-23 13:25:25" level=warning msg="Enabled feature flag" enable-only-blinded-beacon-blocks="Enables storing only blinded beacon blocks in the database without full execution layer transactions" prefix=flags
beacon-testnet_1 | time="2022-08-23 13:25:25" level=info msg="Checking DB" database-path="/data/beaconchaindata" prefix=node
beacon-testnet_1 | time="2022-08-23 13:25:25" level=info msg="Opening Bolt DB at /data/beaconchaindata/beaconchain.db" prefix=db
beacon-testnet_1 | time="2022-08-23 13:25:26" level=info msg="Deposit contract: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b" prefix=node
beacon-testnet_1 | time="2022-08-23 13:25:39" level=info msg="Running node with peer id of 16Uiu2HAmGJ7Va4qc3wrDYnYh2yKJFNAy9hSYF6ZvuFouKB4A8VK2 " prefix=p2p
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="--weak-subjectivity-checkpoint not provided. Prysm recommends providing a weak subjectivity checkpointfor nodes synced from genesis, or manual verification of block and state roots for checkpoint sync nodes." prefix=blockchain
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Waiting for state to be initialized" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="gRPC server listening on port" address="0.0.0.0:4000" prefix=rpc
beacon-testnet_1 | time="2022-08-23 13:25:42" level=warning msg="You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=rpc
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Starting beacon node" prefix=node version="Prysm/v2.1.4/4e225fc667d8df6f8254803d5ae3122be08fd003. Built at: 2022-08-10 13:55:21+00:00"
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Enabled debug gRPC endpoints" prefix=rpc
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Blockchain data already exists in DB, initializing..." prefix=blockchain
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Starting initial chain sync..." prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=0
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Starting API middleware" prefix=gateway
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Starting gRPC gateway" address="127.0.0.1:3500" prefix=gateway
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Started discovery v5" ENR="enr:-MK4QC_kJHyxxt4DdtBokbTfiOZDcvBzOBd0SyM2MpVgjQ0qd0d2Q_wdpqiMHVSxnUXAviF8hqtMBvA03J2VhXPcmG2GAYLK4Vkoh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDCzjqoAgAQIP__________gmlkgnY0gmlwhKwSAAKJc2VjcDI1NmsxoQM2FaCIMcBs__x6wa05PQrbRESECerFbGD2tRbeAXLFDYhzeW5jbmV0cwCDdGNwgjLIg3VkcIIu4A" prefix=p2p
beacon-testnet_1 | time="2022-08-23 13:25:42" level=info msg="Node started p2p server" multiAddr="/ip4/172.18.0.2/tcp/13000/p2p/16Uiu2HAmGJ7Va4qc3wrDYnYh2yKJFNAy9hSYF6ZvuFouKB4A8VK2" prefix=p2p
beacon-testnet_1 | time="2022-08-23 13:25:47" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=2
beacon-testnet_1 | time="2022-08-23 13:25:54" level=info msg="gRPC client connected to beacon node" addr="172.18.0.3:58850" prefix=rpc
beacon-testnet_1 | time="2022-08-23 13:25:56" level=info msg="Processing block batch of size 20 starting from 0x0d956550... 3639905/3729429 - estimated time remaining 24h52m4s" blocksPerSecond=1.0 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:25:57" level=info msg="Connected to new endpoint: http://geth-testnet:8551" prefix=powchain
beacon-testnet_1 | time="2022-08-23 13:26:15" level=info msg="Processing block batch of size 42 starting from 0xde547ff2... 3639936/3729431 - estimated time remaining 8h1m9s" blocksPerSecond=3.1 peers=6 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:21" level=warning msg="Batch is not processed" error="could not process block in batch: could not retrieve block roots: invalid end slot and start slot provided" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:21" level=info msg="Processing block batch of size 41 starting from 0x49191c11... 3640000/3729431 - estimated time remaining 5h59m9s" blocksPerSecond=4.2 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:23" level=warning msg="Batch is not processed" error="could not process block in batch: could not fill in missing blocks to forkchoice: not descendant of finalized checkpoint" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:23" level=info msg="Processing block batch of size 40 starting from 0xef7f74bb... 3640064/3729431 - estimated time remaining 4h2m11s" blocksPerSecond=6.2 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:23" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x9a57fc36bd6e404edec70aa1d1b63bd34b1862048a5395645c6f48cdb0e36d2c (in processBatchedBlocks, slot=3640064)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:23" level=info msg="Processing block batch of size 45 starting from 0x85aba678... 3640129/3729431 - estimated time remaining 2h57m11s" blocksPerSecond=8.4 peers=5 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:23" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x72271da8634dcb0839757352bda0bd911e54c2a129a47e33e2201c5c73d28ebf (in processBatchedBlocks, slot=3640129)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=warning msg="Batch is not processed" error="no good blocks in batch" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=info msg="Processing block batch of size 41 starting from 0x5be5f5da... 3640194/3729432 - estimated time remaining 2h22m19s" blocksPerSecond=10.4 peers=4 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x722fee7bf9b57d775511097c5b8a70fa3158590a154f38b2f6cff81fd6e7e15e (in processBatchedBlocks, slot=3640194)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=info msg="Processing block batch of size 42 starting from 0x92634f9b... 3640256/3729432 - estimated time remaining 1h58m25s" blocksPerSecond=12.6 peers=4 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x968a6d54c81c1149eea877e134678c6d7724a2cb154af522c9b214bff6acf5ae (in processBatchedBlocks, slot=3640256)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=info msg="Processing block batch of size 44 starting from 0x6ef9d899... 3640321/3729432 - estimated time remaining 1h40m41s" blocksPerSecond=14.8 peers=4 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:25" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xad991e44a3ff970e193c78454920fd503d278e233810729ee9a59ed089557be3 (in processBatchedBlocks, slot=3640321)" prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:28" level=warning msg="Unable to cache headers for execution client votes" error="(1661203552 > 1660188288): provided time is later than the current eth1 head" prefix=powchain
beacon-testnet_1 | time="2022-08-23 13:26:28" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=2
beacon-testnet_1 | time="2022-08-23 13:26:29" level=info msg="Processing block batch of size 41 starting from 0xde547ff2... 3639936/3729432 - estimated time remaining 1h28m47s" blocksPerSecond=16.8 peers=4 prefix=initial-sync
beacon-testnet_1 | time="2022-08-23 13:26:33" level=warning msg="Batch is not processed" error="could not process block in batch: could not retrieve block roots: invalid end slot and start slot provided" prefix=initial-sync
Ok this is much more helpful @pablote , the logs you provided give a better indication of the issue