berachain
v1.1.2 Beacond falls behind chain head frequently and requires restart to sync to tip
Hi,
Version: v1.1.2
I have a weird bug...
There is a case where beacond falls behind chain head and requires restart to sync to tip. It will still get the right blocks (hashes and state root OK) but just too slowly. It's distance to chain head will then increase over time. Upon restart beacond it will sync up quickly again and work great for up to 24hours.
Anyone else getting this?
Sounds peering related. Try the following:
Delete the addressbook.json file in your beacond home directory
$BEACON_HOME_DIR/config/addrbook.json
Then kill your beacon node, then execution client and restart both.
Sounds peering related. Try the following:
Delete the addressbook.json file in your beacond home directory
$BEACON_HOME_DIR/config/addrbook.json Then kill your beacon node, then execution client and restart both.
On restart, without deleting anything, it will find peers and sync to tip extremely fast. It will then function flawlessly for up to 24 hours before the issue re-emerges. Still think this is peering?
It's likely still peering but perhaps a different issue to the one my solution solves. Can you dump your beacond and execution logs next time it gets slow. Also the execution client and version you're running. Hard to do much more without further information
It's likely still peering but perhaps a different issue to the one my solution solves. Can you dump your beacond and execution logs next time it gets slow. Also the execution client and version you're running. Hard to do much more without further information
This time it took 2 days to crash. Here are the logs:
Mind you block 1850538 is here about 28 minutes old. Block 1850540 is 10 seconds later "Finalized block module=state height=1850540". It keeps to be going at its own pace. Timeouts?
2025-03-03T14:34:08+01:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xf1fa4d70b1638c630533011bcdbdd3050a7f0c6232e8998c0af5e16364079d3e safe_block_hash=0x349296c785e9a5acc0c09bb37584e24c94358e2b21869e3084331c07bd2e9014 finalized_block_hash=0x349296c785e9a5acc0c09bb37584e24c94358e2b21869e3084331c07bd2e9014 with_attributes=false 2025-03-03T14:34:08+01:00 INFO Committed state module=state height=1850538 block_app_hash=77E3E50CD0B60C498EEFADCED4894854F45467981402D25C30AFFFF9B8EEA235 2025-03-03T14:34:08+01:00 INFO Timed out module=consensus dur=69.002859 height=1850539 round=0 step=RoundStepNewHeight 2025-03-03T14:34:10+01:00 INFO Timed out module=consensus dur=2000 height=1850539 round=0 step=RoundStepPropose 2025-03-03T14:34:16+01:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil height=1850539 commit_round=0 module=consensus proposal= commit=15CB1F4DDB6A709251A5323BE87A0BD599CB6591007FBB5632DC71733CEC24F1 2025-03-03T14:34:17+01:00 INFO Received complete proposal block module=consensus height=1850539 hash=15CB1F4DDB6A709251A5323BE87A0BD599CB6591007FBB5632DC71733CEC24F1 2025-03-03T14:34:17+01:00 INFO Finalizing commit of block module=consensus height=1850539 hash=15CB1F4DDB6A709251A5323BE87A0BD599CB6591007FBB5632DC71733CEC24F1 root=A85AC4508B411D8C95044EA22D249C9182BD22D51204D0225A716EF69EE93326 num_txs=2 2025-03-03T14:34:17+01:00 INFO processExecutionPayload service=state-processor consensus height=1850539 payload height=1850539 payload timestamp=1741007318 consensus timestamp=1741007319 skip payload verification=false 2025-03-03T14:34:17+01:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xd22c22e1cf5c92922e6d2da9117866116ee20dc59b7a53c5fd0aa5a0ce1e254f payload_parent_block_hash=0xf1fa4d70b1638c630533011bcdbdd3050a7f0c6232e8998c0af5e16364079d3e is_optimistic=true 2025-03-03T14:34:17+01:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-03-03T14:34:17+01:00 INFO Finalized block module=state height=1850539 num_txs_res=2 num_val_updates=0 block_app_hash=2D6D65BCFF9AC08B37A83A0ABB87BF22D9DCCEF3215C53D6FD5CBE0BE21222DE syncing_to_height=1850539 2025-03-03T14:34:17+01:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xd22c22e1cf5c92922e6d2da9117866116ee20dc59b7a53c5fd0aa5a0ce1e254f safe_block_hash=0xf1fa4d70b1638c630533011bcdbdd3050a7f0c6232e8998c0af5e16364079d3e finalized_block_hash=0xf1fa4d70b1638c630533011bcdbdd3050a7f0c6232e8998c0af5e16364079d3e with_attributes=false 2025-03-03T14:34:17+01:00 INFO Committed state module=state height=1850539 block_app_hash=A85AC4508B411D8C95044EA22D249C9182BD22D51204D0225A716EF69EE93326 2025-03-03T14:34:17+01:00 INFO Timed out module=consensus dur=-156.122072 height=1850540 round=0 step=RoundStepNewHeight 2025-03-03T14:34:19+01:00 INFO Timed out module=consensus dur=2000 height=1850540 round=0 step=RoundStepPropose 2025-03-03T14:34:25+01:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=1850540 commit_round=0 proposal= commit=6DE67CBD15311E305E1C3191C689B8DDCAD8988B3FE63A31967B174B8DB8C557 2025-03-03T14:34:26+01:00 INFO Received complete proposal block module=consensus height=1850540 hash=6DE67CBD15311E305E1C3191C689B8DDCAD8988B3FE63A31967B174B8DB8C557 2025-03-03T14:34:26+01:00 INFO Finalizing commit of block module=consensus height=1850540 hash=6DE67CBD15311E305E1C3191C689B8DDCAD8988B3FE63A31967B174B8DB8C557 root=2D6D65BCFF9AC08B37A83A0ABB87BF22D9DCCEF3215C53D6FD5CBE0BE21222DE num_txs=2 2025-03-03T14:34:26+01:00 INFO processExecutionPayload service=state-processor consensus height=1850540 payload height=1850540 payload timestamp=1741007320 consensus timestamp=1741007321 skip payload verification=false 2025-03-03T14:34:26+01:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xec3586fa551f77287f12563c4d73e42279c9719267725f14e8a003041620c4f0 payload_parent_block_hash=0xd22c22e1cf5c92922e6d2da9117866116ee20dc59b7a53c5fd0aa5a0ce1e254f is_optimistic=true 2025-03-03T14:34:26+01:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-03-03T14:34:26+01:00 INFO Finalized block module=state height=1850540 num_txs_res=2 num_val_updates=0 block_app_hash=458123596A3751673DF965207745ECF8AC2E8FA001E52141F3BC7FE37101CCBD syncing_to_height=1850540 2025-03-03T14:34:26+01:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xec3586fa551f77287f12563c4d73e42279c9719267725f14e8a003041620c4f0 safe_block_hash=0xd22c22e1cf5c92922e6d2da9117866116ee20dc59b7a53c5fd0aa5a0ce1e254f finalized_block_hash=0xd22c22e1cf5c92922e6d2da9117866116ee20dc59b7a53c5fd0aa5a0ce1e254f with_attributes=false 2025-03-03T14:34:26+01:00 INFO Committed state module=state height=1850540 block_app_hash=2D6D65BCFF9AC08B37A83A0ABB87BF22D9DCCEF3215C53D6FD5CBE0BE21222DE 2025-03-03T14:34:26+01:00 INFO Timed out module=consensus dur=-612.575248 height=1850541 round=0 step=RoundStepNewHeight
2025-03-03T14:34:31+01:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=1850541 commit_round=0 proposal= commit=A1D16A4CCE6FCA5E3F2C92A20BC18CA16B879C79C5BD681D04466657A256C998 2025-03-03T14:34:31+01:00 INFO Received complete proposal block module=consensus height=1850541 hash=A1D16A4CCE6FCA5E3F2C92A20BC18CA16B879C79C5BD681D04466657A256C998 2025-03-03T14:34:31+01:00 INFO Finalizing commit of block module=consensus height=1850541 hash=A1D16A4CCE6FCA5E3F2C92A20BC18CA16B879C79C5BD681D04466657A256C998 root=458123596A3751673DF965207745ECF8AC2E8FA001E52141F3BC7FE37101CCBD num_txs=2 2025-03-03T14:34:31+01:00 INFO processExecutionPayload service=state-processor consensus height=1850541 payload height=1850541 payload timestamp=1741007322 consensus timestamp=1741007322 skip payload verification=false 2025-03-03T14:34:31+01:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0x3e0abfb672afb5fe4e8003209cd11d9bb112ed26a660ae32cc5cbb85d30121e4 payload_parent_block_hash=0xec3586fa551f77287f12563c4d73e42279c9719267725f14e8a003041620c4f0 is_optimistic=true 2025-03-03T14:34:31+01:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-03-03T14:34:31+01:00 INFO Finalized block module=state height=1850541 num_txs_res=2 num_val_updates=0 block_app_hash=4C4CB96079857CB9335F1257D7377D5D527EDB0ADB555BA5760E6699E6E356A8 syncing_to_height=1850541 2025-03-03T14:34:31+01:00 INFO Forkchoice updated service=execution-engine head_block_hash=0x3e0abfb672afb5fe4e8003209cd11d9bb112ed26a660ae32cc5cbb85d30121e4 safe_block_hash=0xec3586fa551f77287f12563c4d73e42279c9719267725f14e8a003041620c4f0 finalized_block_hash=0xec3586fa551f77287f12563c4d73e42279c9719267725f14e8a003041620c4f0 with_attributes=false 2025-03-03T14:34:31+01:00 INFO Committed state module=state height=1850541 block_app_hash=458123596A3751673DF965207745ECF8AC2E8FA001E52141F3BC7FE37101CCBD 2025-03-03T14:34:31+01:00 INFO Timed out module=consensus dur=-3.845773 height=1850542 round=0 step=RoundStepNewHeight 2025-03-03T14:34:33+01:00 INFO Timed out module=consensus dur=2000 height=1850542 round=0 step=RoundStepPropose
It's likely still peering but perhaps a different issue to the one my solution solves. Can you dump your beacond and execution logs next time it gets slow. Also the execution client and version you're running. Hard to do much more without further information
Happened twice tonight... This needs a high priority fix imo
I will gladly pay 500$ for a rapid fix to this problem.. Its causing me trading losses
Hey @Ethics3606 thanks for your update:
- we're looking at this as our top priority
- no need to pay anything.
A couple of follow ups:
- can you clarify what EL client are you using? My guess is Reth but want to confirm
- Any log from the past night would be greatly greatly appreciated
Hey @Ethics3606 thanks for your update:
- we're looking at this as our top priority
- no need to pay anything.
A couple of follow ups:
- can you clarify what EL client are you using? My guess is Reth but want to confirm
- Any log from the past night would be greatly greatly appreciated
Great! Can confirm it's Reth. I will try to collect more logs and paste them here as it happends
Looking forward to it. The reason I ask if EL is Reth is because we are seeing an unusually high latency over some rpc calls if compared with Geth. Since v.1.1.3, those rpc calls are blocking, so that may explain why the node falls behind. However you are seeing the issues with v.1.1.2 correct? I mean you have not upgraded to v.1.1.3 in the meantime?
Looking forward to it. The reason I ask if EL is Reth is because we are seeing an unusually high latency over some rpc calls if compared with Geth. Since v.1.1.3, those rpc calls are blocking, so that may explain why the node falls behind. However you are seeing the issues with v.1.1.2 correct? I mean you have not upgraded to v.1.1.3 in the meantime?
I've been weary of further interruptions, so still running 1.1.2. But I can spin up a 2nd node with v1.1.3 with geth instead and see if it improves
Well if anything, v.1.1.3 keeps CL and EL in lockstep, so any latency issue is likely to be amplified wrt 1.1.2 (which has other issues, namely a deposit mismatch issue from time to time). I would suggest, keep you current versions and provide logs when the issue happens.
Looking forward to it. The reason I ask if EL is Reth is because we are seeing an unusually high latency over some rpc calls if compared with Geth. Since v.1.1.3, those rpc calls are blocking, so that may explain why the node falls behind. However you are seeing the issues with v.1.1.2 correct? I mean you have not upgraded to v.1.1.3 in the meantime?
Which Geth implementation/version does Berachain use? Genesis import is not working on most recent Geth commit: Fatal: Failed to write genesis block: invalid chain configuration: missing entry for fork "cancun" in blobSchedule
edit: Works with Geth 1.14.5. Its syncing from start now
@abi87 I have the same issue on 1.1.3, everything works again after restarting the beacon client. I will be able to provide some logs for you later when it stucks again.
Thanks @donbeave, just to confirm, are you running Reth/Geth/what else?
I'm using reth. You can check my configuration for reth container here: https://github.com/ChainArgos/blockchain-nodes/blob/f6fe9f57b164ef7967b76d88025dd34a4f00be5c/docker-reth/start.sh#L35
I have the same problem.After restarting the node, you can catch up, but eventually you will continue to fall behind, and this will happen repeatedly.
I use ssd disk ubuntu24.04
https://github.com/paradigmxyz/reth/releases/tag/v1.2.0
reth node --config=/xxx/xxx/xxx/reth/reth.toml --authrpc.jwtsecret=/xxx/xxx/xxx/jwt.hex --chain=/xxx/xxx/conf/reth/genesis.json --datadir=/xxx/xxx/xxx/reth --port=30303 --http --http.addr=0.0.0.0 --http.api=eth,net,web3,txpool,debug --http.port=8545 --http.corsdomain=* --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --authrpc.addr=0.0.0.0 --authrpc.port=8551 --metrics=0.0.0.0:6060 --engine.persistence-threshold=0 --engine.memory-block-buffer-target=0 --bootnodes=enode://0c5a4a3c0e81fce2974e4d317d88df783731183d534325e32e0fdf8f4b119d7889fa254d3a38890606ec300d744e2aa9c87099a4a032f5c94efe53f3fcdfecfe@34.64.176.79:30303,enode://b6a3137d3a36ef37c4d31843775a9dc293f41bcbde33b6309c80b1771b6634827cd188285136a57474427bd8845adc2f6fe2e0b106bd58d14795b08910b9c326@34.64.181.70:30303,enode://0b6633300614bc2b9749aee0cace7a091ec5348762aee7b1d195f7616d03a9409019d9bef336624bab72e0d069cd4cf0b0de6fbbf53f04f6b6e4c5b39c6bdca6@34.64.39.31:30303,enode://552b001abebb5805fcd734ad367cd05d9078d18f23ec598d7165460fadcfc51116ad95c418f7ea9a141aa8cbc496c8bea3322b67a5de0d3380f11aab1a797513@34.64.183.158:30303,enode://5b037f66099d5ded86eb7e1619f6d06ceb15609e8cc345ced22a4772b06178004e1490a3cd32fd1222789de4c6e4021c2d648a3d750f6d5323e64b771bbd8de7@34.87.142.180:30303,enode://846db253c53753d3ea1197aec296306dc84c25f3afdf142b65cb0fe0f984de55072daa3bbf05a9aea046a38a2292403137b6eafefd5646fcf62120b74e3b898d@34.142.170.110:30303,enode://64b7f6ee9bcd942ad4949c70f2077627f078a057dfd930e6e904e12643d8952f5ae87c91e24559765393f244a72c9d5c011d7d5176e59191d38f315db85a20f5@34.126.161.16:30303,enode://cf4d19bfb8ec507427ec882bac0bac85a0c8c9ddaa0ec91b773bb614e5e09d107cd9fbe323b96f62f31c493f8f42cc5495c18b87c08560c5dea1dfd25256dcf6@35.247.162.2:30303,enode://bb7e44178543431feac8f0ee3827056b7b84d8235b802a8bdbbcd4939dab7f7dd2579ff577a38b002bb0139792af67abd2dd5c9f4f85b8da6e914fa76dca82bc@35.198.150.35:30303,enode://8fef1f5df45e7b31be00a21e1da5665d5a5f5bf4c379086b843f03eade941bdd157f08c95b31880c492577edb9a9b185df7191eaebf54ab06d5bd683b289f3af@34.107.7.241:30303,enode://ce9c87cfe089f6811d26c96913fa3ec10b938d9017fc6246684c74a33679ee34ceca9447180fb509e37bf2b706c2877a82085d34bfd83b5b520ee1288b0fc32f@35.198.109.49:30303,enode://713657eb6a53feadcbc47e634ad557326a51eb6818a3e19a00a8111492f50a666ccbf2f5d334d247ecf941e68d242ef5c3b812b63c44d381ef11f79c2cdb45c7@34.141.15.100:30303,enode://d071fa740e063ce1bb9cdc2b7937baeff6dc4000f91588d730a731c38a6ff0d4015814812c160fab8695e46f74b9b618735368ea2f16db4d785f16d29b3fb7b0@35.203.2.210:30303,enode://ffc452fe451a2e5f89fe634744aea334d92dcd30d881b76209d2db7dbf4b7ee047e7c69a5bb1633764d987a7441d9c4bc57ccdbfd6442a2f860bf953bc89a9b9@34.152.50.224:30303,enode://da94328302a1d1422209d1916744e90b6095a48b2340dcec39b22002c098bb4d58a880dab98eb26edf03fa4705d1b62f99a8c5c14e6666e4726b6d3066d8a4d7@34.95.61.106:30303,enode://19c7671a4844699b481e81a5bcfe7bafc7fefa953c16ebbe1951b1046371e73839e9058de6b7d3c934318fe7e7233dde3621c1c1018eb8b294ea3d4516147150@35.203.82.137:30303 --trusted-peers=enode://0c5a4a3c0e81fce2974e4d317d88df783731183d534325e32e0fdf8f4b119d7889fa254d3a38890606ec300d744e2aa9c87099a4a032f5c94efe53f3fcdfecfe@34.22.104.177:30303,enode://b6a3137d3a36ef37c4d31843775a9dc293f41bcbde33b6309c80b1771b6634827cd188285136a57474427bd8845adc2f6fe2e0b106bd58d14795b08910b9c326@34.64.247.85:30303,enode://0b6633300614bc2b9749aee0cace7a091ec5348762aee7b1d195f7616d03a9409019d9bef336624bab72e0d069cd4cf0b0de6fbbf53f04f6b6e4c5b39c6bdca6@34.22.73.21:30303,enode://552b001abebb5805fcd734ad367cd05d9078d18f23ec598d7165460fadcfc51116ad95c418f7ea9a141aa8cbc496c8bea3322b67a5de0d3380f11aab1a797513@34.64.37.55:30303,enode://5b037f66099d5ded86eb7e1619f6d06ceb15609e8cc345ced22a4772b06178004e1490a3cd32fd1222789de4c6e4021c2d648a3d750f6d5323e64b771bbd8de7@35.247.182.34:30303,enode://846db253c53753d3ea1197aec296306dc84c25f3afdf142b65cb0fe0f984de55072daa3bbf05a9aea046a38a2292403137b6eafefd5646fcf62120b74e3b898d@34.87.9.231:30303,enode://64b7f6ee9bcd942ad4949c70f2077627f078a057dfd930e6e904e12643d8952f5ae87c91e24559765393f244a72c9d5c011d7d5176e59191d38f315db85a20f5@34.126.78.49:30303,enode://cf4d19bfb8ec507427ec882bac0bac85a0c8c9ddaa0ec91b773bb614e5e09d107cd9fbe323b96f62f31c493f8f42cc5495c18b87c08560c5dea1dfd25256dcf6@35.240.200.36:30303,enode://bb7e44178543431feac8f0ee3827056b7b84d8235b802a8bdbbcd4939dab7f7dd2579ff577a38b002bb0139792af67abd2dd5c9f4f85b8da6e914fa76dca82bc@34.40.14.50:30303,enode://8fef1f5df45e7b31be00a21e1da5665d5a5f5bf4c379086b843f03eade941bdd157f08c95b31880c492577edb9a9b185df7191eaebf54ab06d5bd683b289f3af@35.246.168.217:30303,enode://ce9c87cfe089f6811d26c96913fa3ec10b938d9017fc6246684c74a33679ee34ceca9447180fb509e37bf2b706c2877a82085d34bfd83b5b520ee1288b0fc32f@34.40.28.159:30303,enode://713657eb6a53feadcbc47e634ad557326a51eb6818a3e19a00a8111492f50a666ccbf2f5d334d247ecf941e68d242ef5c3b812b63c44d381ef11f79c2cdb45c7@35.234.82.236:30303,enode://d071fa740e063ce1bb9cdc2b7937baeff6dc4000f91588d730a731c38a6ff0d4015814812c160fab8695e46f74b9b618735368ea2f16db4d785f16d29b3fb7b0@35.203.86.197:30303,enode://ffc452fe451a2e5f89fe634744aea334d92dcd30d881b76209d2db7dbf4b7ee047e7c69a5bb1633764d987a7441d9c4bc57ccdbfd6442a2f860bf953bc89a9b9@34.118.187.161:30303,enode://da94328302a1d1422209d1916744e90b6095a48b2340dcec39b22002c098bb4d58a880dab98eb26edf03fa4705d1b62f99a8c5c14e6666e4726b6d3066d8a4d7@34.95.30.190:30303,enode://19c7671a4844699b481e81a5bcfe7bafc7fefa953c16ebbe1951b1046371e73839e9058de6b7d3c934318fe7e7233dde3621c1c1018eb8b294ea3d4516147150@34.47.60.196:30303
https://github.com/berachain/beacon-kit/releases/tag/v1.1.3 /xxx/xxx/xxx/beacond start --home=/xxx/xxx/xxx/beacond seeds = "[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656"
beacon log: 2025-03-12T10:42:08+08:00 INFO Timed out module=consensus dur=2000 height=2200035 round=0 step=RoundStepPropose 2025-03-12T10:42:15+08:00 ERRR Failed to send commit to peer module=consensus peer=Peer{MConn{65.108.108.29:54656} 8c2682602556cafa90c1e73ce4013f0fe43c6a5b out} height=2191262 commit={"height":2191262,"round":0,"block_id":{"hash":"6B691647C8FDD1BAEC2E795A5E2E2B5E357DF05B30983FB691E3202C46E47D6B","parts":{"total":1,"hash":"D086116B6FD2728160E038009999E59DEE7094FEC4547872B6785B6E57F03DB2"}},"signatures":[{"block_id_flag":4,"validator_address":"09C4AB26B20EF50B371EAF4CB006D6D5B72A53B9","timestamp":"0001-01-01T00:00:00Z","signature":"rj952WZ890oJWR+pEXvkym3+5VVxvEGbFIPqSC7THk0WZ5s1qDb3DfhIqPVTC5CUBYM9qhG0Jj9QsdQWekPbtzERDDBxNIFzL5vE54gXt3rmHrjQ/3nx8cNTMiN4shn8"},{"block_id_flag":5,"validator_address":"0EB600A5EB0DCCD405B3C71953C727975E39EDE4","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"17348776DE5BC1F4BE6F1DB84042DAC57D71C890","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"27219ACFC8E974C0DB5E137CC42E8427553802FC","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"5951C4349AB792BFB3A63956512663CC3B733D6E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"76D5AF272836142878B608C2125DCE729190EF5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8CAED645FC96CDA80F0217B018B2129481050AD9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"9DB42D740B0D0C6A08543D679BF50CED94CA8E3C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CB1DCA2CBFD59BDE8B61C9FC95EEB65D0EDED2B3","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D00F2981BCE83818D2EECFF87F85000DBDD0AF08","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"310D164F10AAE6E85D6CD32CC64DDC2F48F5919B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8B7419E9E076797E9F1C2160218DEE06B8B4478D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CAF0EAD038B1A2F9E8E2762999535BD85380C64E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D13490E5D3BE85D637544484AEBB970C47BA9844","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"DD20845E62583B038100A652D5AA8611D9D4026E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7DF743C6CE84A99660FB91239E53754416943671","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6B3D9D5DFE82E0BF52D6045DF17F16560A027717","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EA9508CAB5DA1954EBE6CFD1425CF0E2D6EC4B0A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EACF2006E60A59979E5850B71702DDB354A2048B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"901FB60F9AF7935456867CFC8355B70FA226A069","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E37FB6E5CFCB23F8AA412A79D793C6A04F54747D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B29EA710A83B6D518A5566AC8AB90C2979CACC16","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EFB1A37176B9D0252B10D7F56851E68CF096B5CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C0B3BFA566E1CBB93BF8B358F29C8F7B10120987","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8C611647259F4440DF572119AFFA58148FFAE0D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1ED255BA54A031E4ACE64B4AC9B0CD999AF1AB71","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6D4649CD64A83E777CE89E0B268654E56DAC1F96","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"06BC43DC34E9F52CFBBDAD93F9FA743BF5278E7F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2C1B7662D046B2FCAF685C804037D3FC728E9E80","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4C79631C9E3BACA9D266C8FB3AD6A5304431D055","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"616EABCEC6FB0DB1AC54C5461D952BFB46B45819","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6DE14F6BCFECCE0F3F86936B6DB580258E542047","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8F7E52CBD156EDD9317042F67AE7ABD1D64C9E19","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B08ADE9A00DFC0A0EACB84BF579B6B6C0C32A645","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C52C7CDD8E64EF9581473E2AE286A96FB6A444A9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0E1982A8083CF41A0B0CDF3CC1A42F5540FC211A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"AC26D89FB38D59464EE04CF66A44F24CE64DA7D4","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0634E825DCD51BF722CCF02086899CDE1E862E5A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19395121376B12A00F8C6AD5FE8C3AD9A77CBFC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19D48D2EBF5A4D5C31228C47828BDF6BFDD7B77A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1D4DCFB35BE65B5E09C4270F0B4E3329F967B4E2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2FBB6C9798950CDE7647CB68602498874336999F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"39CCAF0EB8D31A9592ABA51B6705ABBBFC38086C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"3AECCB026FAEA18F43E79FE116CB1B94E768075C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"41462F6C19492B6FF4F5FAF7D6ED16E1264D1AAE","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"45C2BA0FF19BCB262FA5713E993FBA4408258BC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4D21CFCB799E0C9B66CE566F6AB68BCCCBC75FDA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4F5F37B9DB46C29C48C0B2231BEAD3E14F04CD1A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"64703E2FCEA0339EDE427510BE1642F4E53FC4D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"65697A8BAA1891DC3D466DFEDBE51EAC264936B7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7299ADB3E975B1A7B461F45C4CCA6AFA39DBC9A7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8A93B739376EBA41191E593847B1C844E66F29CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"A49685C59E968B2B097B0CB09E8D08D6B7FCD69E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C9D2DCB223D4353076E6A0E7E82DD99E42980CCA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CCB476A8C39D26D81428AB9FBD848F9B17E1BCE0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D94347C994432FFC131B1FA33A61159E5FAB7D5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E1E054EDC9AEBA25B10965C927AB35D173080DC9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F1D993F20EFE45757E62F9AEB6BB14B15DBF2AE2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F314C84E265C6CF2D834F0CAAD99900CF6857FB2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"FDCD7DBB810E66664F7ACA241E05834DDB79A5E0","timestamp":"0001-01-01T00:00:00Z","signature":null}]} 2025-03-12T10:42:16+08:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=2200035 commit_round=0 proposal= commit=053AC5BBB68EA47EF8240A8EC2E35C34D02D37EAF96EC0B5D4C4A10C14294FB9 2025-03-12T10:42:19+08:00 INFO Received complete proposal block module=consensus height=2200035 hash=053AC5BBB68EA47EF8240A8EC2E35C34D02D37EAF96EC0B5D4C4A10C14294FB9 2025-03-12T10:42:19+08:00 INFO Finalizing commit of block height=2200035 module=consensus hash=053AC5BBB68EA47EF8240A8EC2E35C34D02D37EAF96EC0B5D4C4A10C14294FB9 root=3C234EFCBB801C2BD3B0EA12245069D0699A4A88F118F0FF1502516EE34FD0F4 num_txs=2 2025-03-12T10:42:19+08:00 INFO processExecutionPayload service=state-processor consensus height=2200035 payload height=2200035 payload timestamp=1741704185 consensus timestamp=1741704186 verify payload=true 2025-03-12T10:42:19+08:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0x6420a863e6488404022cc0e7a4afcde7cc79931637abaf0d962c73cbcec271ae payload_parent_block_hash=0x9977f2cb809d49bf61b3a8207819973d782965e825adb51cd82ef4df3e309d9a 2025-03-12T10:42:19+08:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-03-12T10:42:19+08:00 INFO Forkchoice updated service=execution-engine head_block_hash=0x6420a863e6488404022cc0e7a4afcde7cc79931637abaf0d962c73cbcec271ae safe_block_hash=0x9977f2cb809d49bf61b3a8207819973d782965e825adb51cd82ef4df3e309d9a finalized_block_hash=0x9977f2cb809d49bf61b3a8207819973d782965e825adb51cd82ef4df3e309d9a with_attributes=false 2025-03-12T10:42:19+08:00 INFO Finalized block module=state height=2200035 num_txs_res=2 num_val_updates=0 block_app_hash=3E22B6166D08377D3A16126A7B853368DA6F6BB9FC8D53EEAF15148247890869 syncing_to_height=2200035 2025-03-12T10:42:19+08:00 INFO Committed state module=state height=2200035 block_app_hash=3C234EFCBB801C2BD3B0EA12245069D0699A4A88F118F0FF1502516EE34FD0F4 2025-03-12T10:42:19+08:00 INFO indexed block events module=txindex height=2200035 2025-03-12T10:42:19+08:00 INFO Timed out module=consensus dur=-2510.715511 height=2200036 round=0 step=RoundStepNewHeight 2025-03-12T10:42:21+08:00 INFO Timed out module=consensus dur=2000 height=2200036 round=0 step=RoundStepPropose 2025-03-12T10:42:26+08:00 ERRR Failed to send commit to peer module=consensus peer=Peer{MConn{65.108.108.29:54656} 8c2682602556cafa90c1e73ce4013f0fe43c6a5b out} height=2191264 commit={"height":2191264,"round":0,"block_id":{"hash":"B03BF4F8C2993123BA81BC4F6C497747BDE130A7D591F658AED44402BF6B893C","parts":{"total":1,"hash":"9AE3849A24C6B1D2CFCE64FABD0CBB2C9480980A4FC4E88DC8C45DACAB165032"}},"signatures":[{"block_id_flag":4,"validator_address":"09C4AB26B20EF50B371EAF4CB006D6D5B72A53B9","timestamp":"0001-01-01T00:00:00Z","signature":"iPKG6R+VCGdksukgKd/XywsT5VkFuPa+eFmhrATVxUPEuxAaKR+y3A9JeIt5RLzZGEL0pFheLNZMlCTH9cRFeu4zhsuheVY8Vv9p8jYbX/bc3vO177eT+gRPMTTgBpt/"},{"block_id_flag":5,"validator_address":"0EB600A5EB0DCCD405B3C71953C727975E39EDE4","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"17348776DE5BC1F4BE6F1DB84042DAC57D71C890","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"27219ACFC8E974C0DB5E137CC42E8427553802FC","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"5951C4349AB792BFB3A63956512663CC3B733D6E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"76D5AF272836142878B608C2125DCE729190EF5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8CAED645FC96CDA80F0217B018B2129481050AD9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"9DB42D740B0D0C6A08543D679BF50CED94CA8E3C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CB1DCA2CBFD59BDE8B61C9FC95EEB65D0EDED2B3","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D00F2981BCE83818D2EECFF87F85000DBDD0AF08","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"310D164F10AAE6E85D6CD32CC64DDC2F48F5919B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8B7419E9E076797E9F1C2160218DEE06B8B4478D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CAF0EAD038B1A2F9E8E2762999535BD85380C64E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D13490E5D3BE85D637544484AEBB970C47BA9844","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"DD20845E62583B038100A652D5AA8611D9D4026E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7DF743C6CE84A99660FB91239E53754416943671","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6B3D9D5DFE82E0BF52D6045DF17F16560A027717","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EA9508CAB5DA1954EBE6CFD1425CF0E2D6EC4B0A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EACF2006E60A59979E5850B71702DDB354A2048B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"901FB60F9AF7935456867CFC8355B70FA226A069","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E37FB6E5CFCB23F8AA412A79D793C6A04F54747D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B29EA710A83B6D518A5566AC8AB90C2979CACC16","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EFB1A37176B9D0252B10D7F56851E68CF096B5CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C0B3BFA566E1CBB93BF8B358F29C8F7B10120987","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8C611647259F4440DF572119AFFA58148FFAE0D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1ED255BA54A031E4ACE64B4AC9B0CD999AF1AB71","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6D4649CD64A83E777CE89E0B268654E56DAC1F96","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"06BC43DC34E9F52CFBBDAD93F9FA743BF5278E7F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2C1B7662D046B2FCAF685C804037D3FC728E9E80","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4C79631C9E3BACA9D266C8FB3AD6A5304431D055","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"616EABCEC6FB0DB1AC54C5461D952BFB46B45819","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6DE14F6BCFECCE0F3F86936B6DB580258E542047","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8F7E52CBD156EDD9317042F67AE7ABD1D64C9E19","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B08ADE9A00DFC0A0EACB84BF579B6B6C0C32A645","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C52C7CDD8E64EF9581473E2AE286A96FB6A444A9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0E1982A8083CF41A0B0CDF3CC1A42F5540FC211A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"AC26D89FB38D59464EE04CF66A44F24CE64DA7D4","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0634E825DCD51BF722CCF02086899CDE1E862E5A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19395121376B12A00F8C6AD5FE8C3AD9A77CBFC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19D48D2EBF5A4D5C31228C47828BDF6BFDD7B77A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1D4DCFB35BE65B5E09C4270F0B4E3329F967B4E2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2FBB6C9798950CDE7647CB68602498874336999F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"39CCAF0EB8D31A9592ABA51B6705ABBBFC38086C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"3AECCB026FAEA18F43E79FE116CB1B94E768075C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"41462F6C19492B6FF4F5FAF7D6ED16E1264D1AAE","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"45C2BA0FF19BCB262FA5713E993FBA4408258BC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4D21CFCB799E0C9B66CE566F6AB68BCCCBC75FDA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4F5F37B9DB46C29C48C0B2231BEAD3E14F04CD1A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"64703E2FCEA0339EDE427510BE1642F4E53FC4D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"65697A8BAA1891DC3D466DFEDBE51EAC264936B7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7299ADB3E975B1A7B461F45C4CCA6AFA39DBC9A7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8A93B739376EBA41191E593847B1C844E66F29CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"A49685C59E968B2B097B0CB09E8D08D6B7FCD69E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C9D2DCB223D4353076E6A0E7E82DD99E42980CCA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CCB476A8C39D26D81428AB9FBD848F9B17E1BCE0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D94347C994432FFC131B1FA33A61159E5FAB7D5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E1E054EDC9AEBA25B10965C927AB35D173080DC9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F1D993F20EFE45757E62F9AEB6BB14B15DBF2AE2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F314C84E265C6CF2D834F0CAAD99900CF6857FB2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"FDCD7DBB810E66664F7ACA241E05834DDB79A5E0","timestamp":"0001-01-01T00:00:00Z","signature":null}]} 2025-03-12T10:42:28+08:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil height=2200036 commit_round=0 module=consensus proposal= commit=BF6CA4CFD4AA19AE8B6D4E9BE49D5F09CFE725F756E177DCEB242F629656685E 2025-03-12T10:42:28+08:00 INFO Received complete proposal block module=consensus height=2200036 hash=BF6CA4CFD4AA19AE8B6D4E9BE49D5F09CFE725F756E177DCEB242F629656685E 2025-03-12T10:42:28+08:00 INFO Finalizing commit of block module=consensus height=2200036 hash=BF6CA4CFD4AA19AE8B6D4E9BE49D5F09CFE725F756E177DCEB242F629656685E root=3E22B6166D08377D3A16126A7B853368DA6F6BB9FC8D53EEAF15148247890869 num_txs=2 2025-03-12T10:42:28+08:00 INFO processExecutionPayload service=state-processor consensus height=2200036 payload height=2200036 payload timestamp=1741704187 consensus timestamp=1741704188 verify payload=true 2025-03-12T10:42:28+08:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xdcda602486e521e765fb07bbd5f85f4e8905669f1b055d86dcdcecaecfb31603 payload_parent_block_hash=0x6420a863e6488404022cc0e7a4afcde7cc79931637abaf0d962c73cbcec271ae 2025-03-12T10:42:28+08:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-03-12T10:42:28+08:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xdcda602486e521e765fb07bbd5f85f4e8905669f1b055d86dcdcecaecfb31603 safe_block_hash=0x6420a863e6488404022cc0e7a4afcde7cc79931637abaf0d962c73cbcec271ae finalized_block_hash=0x6420a863e6488404022cc0e7a4afcde7cc79931637abaf0d962c73cbcec271ae with_attributes=false 2025-03-12T10:42:28+08:00 INFO Finalized block module=state height=2200036 num_txs_res=2 num_val_updates=0 block_app_hash=B80F69828FAED46243EEC832C0FE85864EA870880BC8292A41E1A78A7A2DFCA4 syncing_to_height=2200036 2025-03-12T10:42:28+08:00 INFO Committed state module=state height=2200036 block_app_hash=3E22B6166D08377D3A16126A7B853368DA6F6BB9FC8D53EEAF15148247890869 2025-03-12T10:42:28+08:00 INFO indexed block events module=txindex height=2200036 2025-03-12T10:42:28+08:00 INFO Timed out module=consensus dur=-63.744490 height=2200037 round=0 step=RoundStepNewHeight 2025-03-12T10:42:30+08:00 INFO Timed out module=consensus dur=2000 height=2200037 round=0 step=RoundStepPropose 2025-03-12T10:42:31+08:00 INFO Ensure peers module=pex numOutPeers=46 numInPeers=0 numDialing=0 numToDial=14 2025-03-12T10:42:34+08:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=2200037 commit_round=0 proposal= commit=163E6FAFB0E3F287BF6E67D71BD26155714F60D956FE41A577D714AD3B748CC6 2025-03-12T10:42:35+08:00 INFO Received complete proposal block module=consensus height=2200037 hash=163E6FAFB0E3F287BF6E67D71BD26155714F60D956FE41A577D714AD3B748CC6 2025-03-12T10:42:35+08:00 INFO Finalizing commit of block module=consensus height=2200037 hash=163E6FAFB0E3F287BF6E67D71BD26155714F60D956FE41A577D714AD3B748CC6 root=B80F69828FAED46243EEC832C0FE85864EA870880BC8292A41E1A78A7A2DFCA4 num_txs=2 2025-03-12T10:42:35+08:00 INFO processExecutionPayload service=state-processor consensus height=2200037 payload height=2200037 payload timestamp=1741704189 consensus timestamp=1741704190 verify payload=true 2025-03-12T10:42:35+08:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xe9d8452ba390b92ab21722132d37121f3040789b10d7debd5ee72680ed7efd88 payload_parent_block_hash=0xdcda602486e521e765fb07bbd5f85f4e8905669f1b055d86dcdcecaecfb31603 2025-03-12T10:42:35+08:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-03-12T10:42:35+08:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xe9d8452ba390b92ab21722132d37121f3040789b10d7debd5ee72680ed7efd88 safe_block_hash=0xdcda602486e521e765fb07bbd5f85f4e8905669f1b055d86dcdcecaecfb31603 finalized_block_hash=0xdcda602486e521e765fb07bbd5f85f4e8905669f1b055d86dcdcecaecfb31603 with_attributes=false 2025-03-12T10:42:35+08:00 INFO Finalized block module=state height=2200037 num_txs_res=2 num_val_updates=0 block_app_hash=0B05203555156972F0774C21D94A7496DED8C5A4C16F905DC1A5A59EC386BCD9 syncing_to_height=2200037 2025-03-12T10:42:35+08:00 INFO Committed state module=state height=2200037 block_app_hash=B80F69828FAED46243EEC832C0FE85864EA870880BC8292A41E1A78A7A2DFCA4 2025-03-12T10:42:35+08:00 INFO indexed block events module=txindex height=2200037 2025-03-12T10:42:35+08:00 INFO Timed out module=consensus dur=9.308109 height=2200038 round=0 step=RoundStepNewHeight 2025-03-12T10:42:37+08:00 INFO Timed out module=consensus dur=2000 height=2200038 round=0 step=RoundStepPropose 2025-03-12T10:42:43+08:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=2200038 commit_round=0 proposal= commit=95ACC1C406F7A81547E37115F186A0C61DFCA07C437F11B4CDAF9F5253B6743A
Well if anything, v.1.1.3 keeps CL and EL in lockstep, so any latency issue is likely to be amplified wrt 1.1.2 (which has other issues, namely a deposit mismatch issue from time to time). I would suggest, keep you current versions and provide logs when the issue happens.
Hi, here is an update.
I spun up a 2nd node using Geth 1.14.5 and Beacond v1.1.3. It is also falling behind the chain head (as reth with beacond v1.1.2), although less frequently. Still a major issue that requires a fix.
Any progress on your end? If you have tips for suggested code changes that you don't yet want to risk put into main, I can try it
Trying 1.1.4 with Reth and geth
1.1.4 falling behind every hour.... much worse. reverting to 1.1.3
@donbeave
I see reth 1.3.4 is specified in your repository.
our advice remains to stay on 1.1.x
https://docs.berachain.com/nodes/evm-execution
Reth > 1.2.0 is appallingly slow, it spends a lot of time maxing out the CPU.
@Ethics3606 what is the computer specifications? Is it dedicated hardware or a cloud instance? What does vmstat 5 output show? Which version of reth are you using?
You might get quicker engagement on this subject on discord.gg/berachain #node-support which is where I can be usually found.
I had this exact same issue. Long story short, downgrading my reth from 1.1.5 to 1.1.4 resolved the issue of falling behind. Becond version is 1.1.4.
To note, this issue was also present for beacond 1.1.4 and 1.1.3 both combined with reth 1.1.5. My node would start falling behind instantly once syncing completed. I'm at 24+ hours without a single issue.
I had this exact same issue. Long story short, downgrading my reth from 1.1.5 to 1.1.4 resolved the issue of falling behind. Becond version is 1.1.4.
To note, this issue was also present for beacond 1.1.4 and 1.1.3 both combined with reth 1.1.5. My node would start falling behind instantly once syncing completed. I'm at 24+ hours without a single issue.
Well I've done 24h+ as well with 1.1.5. Actually just survived the entire weekend. Keep us posted
Here are some logs from today. Had 4 desync episodes today on reth. 0 with geth.
The errors from beacond seem to be about timeouts:
","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4D21CFCB799E0C9B66CE566F6AB68BCCCBC75FDA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4F5F37B9DB46C29C48C0B2231BEAD3E14F04CD1A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"64703E2FCEA0339EDE427510BE1642F4E53FC4D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"65697A8BAA1891DC3D466DFEDBE51EAC264936B7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7299ADB3E975B1A7B461F45C4CCA6AFA39DBC9A7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8A93B739376EBA41191E593847B1C844E66F29CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"A49685C59E968B2B097B0CB09E8D08D6B7FCD69E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C9D2DCB223D4353076E6A0E7E82DD99E42980CCA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CCB476A8C39D26D81428AB9FBD848F9B17E1BCE0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D94347C994432FFC131B1FA33A61159E5FAB7D5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E1E054EDC9AEBA25B10965C927AB35D173080DC9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F1D993F20EFE45757E62F9AEB6BB14B15DBF2AE2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F314C84E265C6CF2D834F0CAAD99900CF6857FB2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"FDCD7DBB810E66664F7ACA241E05834DDB79A5E0","timestamp":"0001-01-01T00:00:00Z","signature":null}]} 2025-04-04T15:16:32+02:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil height=3254385 commit_round=0 module=consensus proposal= commit=D0B1E1F3AC75B65AA5A7E94238B8C344B4A6B8B5AA7562CEF8BA6101E8031ED3 2025-04-04T15:16:32+02:00 INFO Received complete proposal block module=consensus height=3254385 hash=D0B1E1F3AC75B65AA5A7E94238B8C344B4A6B8B5AA7562CEF8BA6101E8031ED3 2025-04-04T15:16:32+02:00 INFO Finalizing commit of block module=consensus height=3254385 hash=D0B1E1F3AC75B65AA5A7E94238B8C344B4A6B8B5AA7562CEF8BA6101E8031ED3 root=F9EE5BBB566C1934A61A9BD285935BCF4CA28C1E06CE2955C146D34CEDCC7D89 num_txs=2 2025-04-04T15:16:32+02:00 INFO processExecutionPayload service=state-processor consensus height=3254385 payload height=3254385 payload timestamp=1743770996 consensus timestamp=1743770997 verify payload=true 2025-04-04T15:16:32+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0x7f783b1b63c48703049e613f09ecdc0de2b817510bf3ecb4d9aca472666a26b0 payload_parent_block_hash=0xb623ff02ea115afe0f4b1fb2161d121ac1bac3db4d141e85b18adf80f3fbb831 2025-04-04T15:16:32+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T15:16:32+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0x7f783b1b63c48703049e613f09ecdc0de2b817510bf3ecb4d9aca472666a26b0 safe_block_hash=0xb623ff02ea115afe0f4b1fb2161d121ac1bac3db4d141e85b18adf80f3fbb831 finalized_block_hash=0xb623ff02ea115afe0f4b1fb2161d121ac1bac3db4d141e85b18adf80f3fbb831 with_attributes=false 2025-04-04T15:16:32+02:00 INFO Finalized block module=state height=3254385 num_txs_res=2 num_val_updates=0 block_app_hash=FD9D47C7D8859969846E75574D26D7F6AD4DC3F1E544451F6E6864530DD22EB7 syncing_to_height=3254385 2025-04-04T15:16:32+02:00 INFO Committed state module=state height=3254385 block_app_hash=F9EE5BBB566C1934A61A9BD285935BCF4CA28C1E06CE2955C146D34CEDCC7D89 2025-04-04T15:16:32+02:00 INFO Timed out module=consensus dur=384.832099 height=3254386 round=0 step=RoundStepNewHeight 2025-04-04T15:16:34+02:00 INFO Timed out module=consensus dur=2000 height=3254386 round=0 step=RoundStepPropose 2025-04-04T15:16:37+02:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=3254386 commit_round=0 proposal= commit=BDF0DD50EA44E20B3C47EE72F0148B55496A1AF8A609DB57FD3FC877C7B99B5E 2025-04-04T15:16:37+02:00 INFO Received complete proposal block module=consensus height=3254386 hash=BDF0DD50EA44E20B3C47EE72F0148B55496A1AF8A609DB57FD3FC877C7B99B5E 2025-04-04T15:16:37+02:00 INFO Finalizing commit of block height=3254386 module=consensus hash=BDF0DD50EA44E20B3C47EE72F0148B55496A1AF8A609DB57FD3FC877C7B99B5E root=FD9D47C7D8859969846E75574D26D7F6AD4DC3F1E544451F6E6864530DD22EB7 num_txs=2 2025-04-04T15:16:37+02:00 INFO processExecutionPayload service=state-processor consensus height=3254386 payload height=3254386 payload timestamp=1743770998 consensus timestamp=1743770999 verify payload=true 2025-04-04T15:16:37+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xe4c6da7b55193dd58b8475115e1e34498ab2d1a382b37769ec8dacc40a6cf7b8 payload_parent_block_hash=0x7f783b1b63c48703049e613f09ecdc0de2b817510bf3ecb4d9aca472666a26b0 2025-04-04T15:16:37+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T15:16:37+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xe4c6da7b55193dd58b8475115e1e34498ab2d1a382b37769ec8dacc40a6cf7b8 safe_block_hash=0x7f783b1b63c48703049e613f09ecdc0de2b817510bf3ecb4d9aca472666a26b0 finalized_block_hash=0x7f783b1b63c48703049e613f09ecdc0de2b817510bf3ecb4d9aca472666a26b0 with_attributes=false 2025-04-04T15:16:37+02:00 INFO Finalized block module=state height=3254386 num_txs_res=2 num_val_updates=0 block_app_hash=44760F97502AD73F8504F5EB8C41EF7C8AD747C9D62A4665D463F52199DB7000 syncing_to_height=3254386 2025-04-04T15:16:37+02:00 INFO Committed state module=state height=3254386 block_app_hash=FD9D47C7D8859969846E75574D26D7F6AD4DC3F1E544451F6E6864530DD22EB7 2025-04-04T15:16:37+02:00 INFO Timed out module=consensus dur=87.823371 height=3254387 round=0 step=RoundStepNewHeight
y payload=true 2025-04-04T15:16:26+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xb623ff02ea115afe0f4b1fb2161d121ac1bac3db4d141e85b18adf80f3fbb831 payload_parent_block_hash=0x8d90a76744a5ff5622ee5a0d731dde3c448184d8bf986b43c14460cab0d91ce2 2025-04-04T15:16:26+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T15:16:26+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xb623ff02ea115afe0f4b1fb2161d121ac1bac3db4d141e85b18adf80f3fbb831 safe_block_hash=0x8d90a76744a5ff5622ee5a0d731dde3c448184d8bf986b43c14460cab0d91ce2 finalized_block_hash=0x8d90a76744a5ff5622ee5a0d731dde3c448184d8bf986b43c14460cab0d91ce2 with_attributes=false 2025-04-04T15:16:26+02:00 INFO Finalized block module=state height=3254384 num_txs_res=2 num_val_updates=0 block_app_hash=F9EE5BBB566C1934A61A9BD285935BCF4CA28C1E06CE2955C146D34CEDCC7D89 syncing_to_height=3254384 2025-04-04T15:16:26+02:00 INFO Committed state module=state height=3254384 block_app_hash=F89817C37CA2E22376A1F0168A27FAB830CB7EFE83C5539884E720A9B697E06B 2025-04-04T15:16:26+02:00 INFO Timed out module=consensus dur=238.905413 height=3254385 round=0 step=RoundStepNewHeight 2025-04-04T15:16:28+02:00 INFO Timed out module=consensus dur=2000 height=3254385 round=0 step=RoundStepPropose 2025-04-04T15:16:30+02:00 ERRR Failed to send commit to peer module=consensus peer=Peer{MConn{69.10.45.50:16883} d562454a31adca4f9f8c8d02c4073104dc8e8c1c in} height=3227188 commit={"height":3227188,"round":0,"block_id":{"hash":"3C70656ADA911ECA9AC43987279D4EEEBD8B92B3A1AD80EAE6FCBA38A246FCA5","parts":{"total":1,"hash":"9AAFE477C94037CABCB1EB96B20611F177AB98FEC0649DB48A265DA231CBEF2 6"}},"signatures":[{"block_id_flag":4,"validator_address":"09C4AB26B20EF50B371EAF4CB006D6D5B72A53B9","timestamp":"0001-01-01T00:00:00Z","signature":"hqfXFxY9YPAq+m2ue8luXj8tIbBI/TYY+eOejQ07CHOINyc1raxQPEFFF7/g7ZrKCydRc2W16Ta5/slzGsaJi3UV9JpaOS3ke/1Ex3Xsrxdb6l5bWqr5S3SHfP4oAJb7"},{"block_id_flag":5,"validator_address":"0EB600A5EB0DCCD405B3C71953C727975E39EDE4","timestamp":"0001-01-0 1T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"17348776DE5BC1F4BE6F1DB84042DAC57D71C890","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"valida tor_address":"27219ACFC8E974C0DB5E137CC42E8427553802FC","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"5951C4349AB792BFB3A63956512663CC3B733D6E"," timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"76D5AF272836142878B608C2125DCE729190EF5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"blo ck_id_flag":5,"validator_address":"8CAED645FC96CDA80F0217B018B2129481050AD9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"9DB42D740B0D0C6A08543D679BF50CED94CA8E3C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CB1DCA2CBFD59BDE8B61C9FC95EEB65D0EDED2B3","timestamp":"0001-01-01T00:00:00Z","s ignature":null},{"block_id_flag":5,"validator_address":"D00F2981BCE83818D2EECFF87F85000DBDD0AF08","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D D20845E62583B038100A652D5AA8611D9D4026E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"310D164F10AAE6E85D6CD32CC64DDC2F48F5919B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8B7419E9E076797E9F1C2160218DEE06B8B4478D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5," validator_address":"CAF0EAD038B1A2F9E8E2762999535BD85380C64E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D13490E5D3BE85D637544484AEBB970C47BA9844","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"5E1F812F4D7607E3E844D01862D32E9C1A956CB0","timestamp":"0001-01-01T00:00:00Z","signature":null} ,{"block_id_flag":5,"validator_address":"7DF743C6CE84A99660FB91239E53754416943671","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6B3D9D5DFE82E0BF52D6045DF17F16560A027717","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C0B3BFA566E1CBB93BF8B358F29C8F7B10120987","timestamp":"0001-01-01T00:00:0 0Z","signature":null},{"block_id_flag":5,"validator_address":"EA9508CAB5DA1954EBE6CFD1425CF0E2D6EC4B0A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":1,"validator_addre ss":"","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"901FB60F9AF7935456867CFC8355B70FA226A069","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E37FB6E5CFCB23F8AA412A79D793C6A04F54747D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8C611647259F44 40DF572119AFFA58148FFAE0D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6D4649CD64A83E777CE89E0B268654E56DAC1F96","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B29EA710A83B6D518A5566AC8AB90C2979CACC16","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_add ress":"EFB1A37176B9D0252B10D7F56851E68CF096B5CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"90106A700F58A4BF7EBE5C083BDBBAF8A8761E5B","timesta mp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C52C7CDD8E64EF9581473E2AE286A96FB6A444A9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_f lag":5,"validator_address":"1ED255BA54A031E4ACE64B4AC9B0CD999AF1AB71","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"616EABCEC6FB0DB1AC54C5461D952 BFB46B45819","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"06BC43DC34E9F52CFBBDAD93F9FA743BF5278E7F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2C1B7662D046B2FCAF685C804037D3FC728E9E80","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4C79631C 9E3BACA9D266C8FB3AD6A5304431D055","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6DE14F6BCFECCE0F3F86936B6DB580258E542047","timestamp":"0001-01-01 T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8F7E52CBD156EDD9317042F67AE7ABD1D64C9E19","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B08ADE9A00DFC0A0EACB84BF579B6B6C0C32A645","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"AC26D89FB38D59464EE04CF66A44F24CE64DA7D4","t imestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0E1982A8083CF41A0B0CDF3CC1A42F5540FC211A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0634E825DCD51BF722CCF02086899CDE1E862E5A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19395121376B12A00F8C6AD 5FE8C3AD9A77CBFC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19D48D2EBF5A4D5C31228C47828BDF6BFDD7B77A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1D4DCFB35BE65B5E09C4270F0B4E3329F967B4E2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2F BB6C9798950CDE7647CB68602498874336999F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"39CCAF0EB8D31A9592ABA51B6705ABBBFC38086C","timestamp":"0001 -01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"3AECCB026FAEA18F43E79FE116CB1B94E768075C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"41462F6C19492B6FF4F5FAF7D6ED16E1264D1AAE","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"45C2BA0FF19BCB262FA5713E993FBA4408258B C1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4D21CFCB799E0C9B66CE566F6AB68BCCCBC75FDA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4F5F37B9DB46C29C48C0B2231BEAD3E14F04CD1A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"64703E2FCEA0339ED E427510BE1642F4E53FC4D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"65697A8BAA1891DC3D466DFEDBE51EAC264936B7","timestamp":"0001-01-01T00:00:00 Z","signature":null},{"block_id_flag":5,"validator_address":"7299ADB3E975B1A7B461F45C4CCA6AFA39DBC9A7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_addres s":"8A93B739376EBA41191E593847B1C844E66F29CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"A49685C59E968B2B097B0CB09E8D08D6B7FCD69E","timestamp" :"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C9D2DCB223D4353076E6A0E7E82DD99E42980CCA","timestamp":"0001-01-01T00:0
ng_to_height=3254386 2025-04-04T15:16:37+02:00 INFO Committed state module=state height=3254386 block_app_hash=FD9D47C7D8859969846E75574D26D7F6AD4DC3F1E544451F6E6864530DD22EB7 2025-04-04T15:16:37+02:00 INFO Timed out module=consensus dur=87.823371 height=3254387 round=0 step=RoundStepNewHeight 2025-04-04T15:16:39+02:00 INFO Timed out module=consensus dur=2000 height=3254387 round=0 step=RoundStepPropose 2025-04-04T15:16:43+02:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=3254387 commit_round=0 proposal= commit=1B995F6CC70E270E1D10C19F54FE76BE3F3184771C4AE5C7F76824A73D31B5CB 2025-04-04T15:16:43+02:00 INFO Received complete proposal block module=consensus height=3254387 hash=1B995F6CC70E270E1D10C19F54FE76BE3F3184771C4AE5C7F76824A73D31B5CB 2025-04-04T15:16:43+02:00 INFO Finalizing commit of block height=3254387 module=consensus hash=1B995F6CC70E270E1D10C19F54FE76BE3F3184771C4AE5C7F76824A73D31B5CB root=44760F97502AD73F8504F5EB8C41EF7C8AD747C9D62A4665D463F52199DB7000 num_txs=2 2025-04-04T15:16:43+02:00 INFO processExecutionPayload service=state-processor consensus height=3254387 payload height=3254387 payload timestamp=1743771000 consensus timestamp=1743771001 verify payload=true 2025-04-04T15:16:43+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0x89ab41138c253c0519fe0c48d5bfa2d734ca71c0f03d716000a98ef481ac74d5 payload_parent_block_hash=0xe4c6da7b55193dd58b8475115e1e34498ab2d1a382b37769ec8dacc40a6cf7b8 2025-04-04T15:16:43+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T15:16:43+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0x89ab41138c253c0519fe0c48d5bfa2d734ca71c0f03d716000a98ef481ac74d5 safe_block_hash=0xe4c6da7b55193dd58b8475115e1e34498ab2d1a382b37769ec8dacc40a6cf7b8 finalized_block_hash=0xe4c6da7b55193dd58b8475115e1e34498ab2d1a382b37769ec8dacc40a6cf7b8 with_attributes=false 2025-04-04T15:16:43+02:00 INFO Finalized block module=state height=3254387 num_txs_res=2 num_val_updates=0 block_app_hash=E34F83A66A8787FF6D285C5539FBE62177175334891A7223E03923D6E76F528C syncing_to_height=3254387 2025-04-04T15:16:43+02:00 INFO Committed state module=state height=3254387 block_app_hash=44760F97502AD73F8504F5EB8C41EF7C8AD747C9D62A4665D463F52199DB7000 2025-04-04T15:16:44+02:00 INFO Timed out module=consensus dur=202.842424 height=3254388 round=0 step=RoundStepNewHeight ^[[5~2025-04-04T15:16:46+02:00 INFO Timed out module=consensus dur=2000 height=3254388 round=0 step=RoundStepPropose 2025-04-04T15:16:49+02:00 INFO Saving AddrBook to file module=p2p book=/root/berachain-data/config/addrbook.json size=1002 2025-04-04T15:16:49+02:00 INFO Ensure peers module=pex numOutPeers=41 numInPeers=120 numDialing=0 numToDial=19 2025-04-04T15:16:50+02:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=3254388 commit_round=0 proposal= commit=A8225F1B18F27AF6A295BD71549560B883FAC656C81F99671B82A3B1593E86C0 2025-04-04T15:16:50+02:00 INFO Received complete proposal block module=consensus height=3254388 hash=A8225F1B18F27AF6A295BD71549560B883FAC656C81F99671B82A3B1593E86C0 2025-04-04T15:16:50+02:00 INFO Finalizing commit of block height=3254388 module=consensus hash=A8225F1B18F27AF6A295BD71549560B883FAC656C81F99671B82A3B1593E86C0 root=E34F83A66A8787FF6D285C5539FBE62177175334891A7223E03923D6E76F528C num_txs=2 2025-04-04T15:16:50+02:00 INFO processExecutionPayload service=state-processor consensus height=3254388 payload height=3254388 payload timestamp=1743771002 consensus timestamp=1743771003 verify payload=true 2025-04-04T15:16:50+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0xff5136d7b2129962374829be317c779c41e406b60cfc62460ab5deea83f0dc1b payload_parent_block_hash=0x89ab41138c253c0519fe0c48d5bfa2d734ca71c0f03d716000a98ef481ac74d5 2025-04-04T15:16:50+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T15:16:50+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0xff5136d7b2129962374829be317c779c41e406b60cfc62460ab5deea83f0dc1b safe_block_hash=0x89ab41138c253c0519fe0c48d5bfa2d734ca71c0f03d716000a98ef481ac74d5 finalized_block_hash=0x89ab41138c253c0519fe0c48d5bfa2d734ca71c0f03d716000a98ef481ac74d5 with_attributes=false 2025-04-04T15:16:50+02:00 INFO Finalized block module=state height=3254388 num_txs_res=2 num_val_updates=0 block_app_hash=40F88F9352CCD769EFECE13D0C65B2E79A52DD3B5D0030D882B6692EA867B6B0 syncing_to_height=3254388 2025-04-04T15:16:50+02:00 INFO Committed state module=state height=3254388 block_app_hash=E34F83A66A8787FF6D285C5539FBE62177175334891A7223E03923D6E76F528C 2025-04-04T15:16:50+02:00 INFO Timed out module=consensus dur=275.691117 height=3254389 round=0 step=RoundStepNewHeight 2025-04-04T15:16:52+02:00 INFO Timed out module=consensus dur=2000 height=3254389 round=0 step=RoundStepPropose 2025-04-04T15:16:55+02:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=3254389 commit_round=0 proposal= commit=12ECEDDE6ECB10FA39983AC0B96EA1DCF41FAE77E1F2F565FCAAD81638B44257 2025-04-04T15:16:55+02:00 INFO Received complete proposal block module=consensus height=3254389 hash=12ECEDDE6ECB10FA39983AC0B96EA1DCF41FAE77E1F2F565FCAAD81638B44257 2025-04-04T15:16:55+02:00 INFO Finalizing commit of block module=consensus height=3254389 hash=12ECEDDE6ECB10FA39983AC0B96EA1DCF41FAE77E1F2F565FCAAD81638B44257 root=40F88F9352CCD769EFECE13D0C65B2E79A52DD3B5D0030D882B6692EA867B6B0 num_txs=2 2025-04-04T15:16:55+02:00 INFO processExecutionPayload service=state-processor consensus height=3254389 payload height=3254389 payload timestamp=1743771004 consensus timestamp=1743771005 verify payload=true 2025-04-04T15:16:56+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0x5aafec2041ea56166dc8d598feb104b07626cd3a67184e39bbb09e41b4f036c5 payload_parent_block_hash=0xff5136d7b2129962374829be317c779c41e406b60cfc62460ab5deea83f0dc1b 2025-04-04T15:16:56+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T15:16:56+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0x5aafec2041ea56166dc8d598feb104b07626cd3a67184e39bbb09e41b4f036c5 safe_block_hash=0xff5136d7b2129962374829be317c779c41e406b60cfc62460ab5deea83f0dc1b finalized_block_hash=0xff5136d7b2129962374829be317c779c41e406b60cfc62460ab5deea83f0dc1b with_attributes=false 2025-04-04T15:16:56+02:00 INFO Finalized block module=state height=3254389 num_txs_res=2 num_val_updates=0 block_app_hash=77A86A65C60ABEF2ED09C9D8EE3B01D87ED895AB37FCE578048D1C6B89C77F1A syncing_to_height=3254389 2025-04-04T15:16:56+02:00 INFO Committed state module=state height=3254389 block_app_hash=40F88F9352CCD769EFECE13D0C65B2E79A52DD3B5D0030D882B6692EA867B6B0 2025-04-04T15:16:56+02:00 INFO Timed out module=consensus dur=163.239712 height=3254390 round=0 step=RoundStepNewHeight 2025-04-04T15:16:58+02:00 INFO Timed out module=consensus dur=2000 height=3254390 round=0 step=RoundStepPropose
5-04-04T22:07:25+02:00 INFO Timed out module=consensus dur=89.515303 height=3266485 round=0 step=RoundStepNewHeight 2025-04-04T22:07:27+02:00 ERRR Failed to send commit to peer module=consensus peer=Peer{MConn{199.254.199.48:50330} 4d2cf34078c4638737c5df7b518bae6292f66c28 in} height=3257465 commit={"height":3257465,"round":0,"block_id":{"hash":"5DDC6333D86857E24E92F88A495625D712D05D669B444A88022543A81D28D810","parts":{"total":1,"hash":"4541F589518ED63C33868F22A1134C628D261B760512CB634E02EF90F555D5B1"}},"signatures":[{"block_id_flag":4,"validator_address":"09C4AB26B20EF50B371EAF4CB006D6D5B72A53B9","timestamp":"0001-01-01T00:00:00Z","signature":"ks8rQELn/Tw4eeZD7TQVuew9F2szrpigVNjNLsyEnHdsG9nKn35YsdwHo1Ix0rEHBE/JKnSnzDj23bXPaNodfcPNhOaUzKGFkdbsCeJIPQbypQWinB0a/jwNa3/Js+jZ"},{"block_id_flag":5,"validator_address":"0EB600A5EB0DCCD405B3C71953C727975E39EDE4","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"17348776DE5BC1F4BE6F1DB84042DAC57D71C890","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"27219ACFC8E974C0DB5E137CC42E8427553802FC","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"5951C4349AB792BFB3A63956512663CC3B733D6E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"76D5AF272836142878B608C2125DCE729190EF5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8CAED645FC96CDA80F0217B018B2129481050AD9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"9DB42D740B0D0C6A08543D679BF50CED94CA8E3C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CB1DCA2CBFD59BDE8B61C9FC95EEB65D0EDED2B3","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D00F2981BCE83818D2EECFF87F85000DBDD0AF08","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"DD20845E62583B038100A652D5AA8611D9D4026E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"310D164F10AAE6E85D6CD32CC64DDC2F48F5919B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8B7419E9E076797E9F1C2160218DEE06B8B4478D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CAF0EAD038B1A2F9E8E2762999535BD85380C64E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D13490E5D3BE85D637544484AEBB970C47BA9844","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"5E1F812F4D7607E3E844D01862D32E9C1A956CB0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7DF743C6CE84A99660FB91239E53754416943671","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6B3D9D5DFE82E0BF52D6045DF17F16560A027717","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C0B3BFA566E1CBB93BF8B358F29C8F7B10120987","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EA9508CAB5DA1954EBE6CFD1425CF0E2D6EC4B0A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EACF2006E60A59979E5850B71702DDB354A2048B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"901FB60F9AF7935456867CFC8355B70FA226A069","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E37FB6E5CFCB23F8AA412A79D793C6A04F54747D","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8C611647259F4440DF572119AFFA58148FFAE0D0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6D4649CD64A83E777CE89E0B268654E56DAC1F96","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B29EA710A83B6D518A5566AC8AB90C2979CACC16","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"EFB1A37176B9D0252B10D7F56851E68CF096B5CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"90106A700F58A4BF7EBE5C083BDBBAF8A8761E5B","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C52C7CDD8E64EF9581473E2AE286A96FB6A444A9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1ED255BA54A031E4ACE64B4AC9B0CD999AF1AB71","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"616EABCEC6FB0DB1AC54C5461D952BFB46B45819","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"06BC43DC34E9F52CFBBDAD93F9FA743BF5278E7F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2C1B7662D046B2FCAF685C804037D3FC728E9E80","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4C79631C9E3BACA9D266C8FB3AD6A5304431D055","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"6DE14F6BCFECCE0F3F86936B6DB580258E542047","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8F7E52CBD156EDD9317042F67AE7ABD1D64C9E19","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"B08ADE9A00DFC0A0EACB84BF579B6B6C0C32A645","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"AC26D89FB38D59464EE04CF66A44F24CE64DA7D4","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0E1982A8083CF41A0B0CDF3CC1A42F5540FC211A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"0634E825DCD51BF722CCF02086899CDE1E862E5A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"19395121376B12A00F8C6AD5FE8C3AD9A77CBFC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":1,"validator_address":"","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"1D4DCFB35BE65B5E09C4270F0B4E3329F967B4E2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"2FBB6C9798950CDE7647CB68602498874336999F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"39CCAF0EB8D31A9592ABA51B6705ABBBFC38086C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"3AECCB026FAEA18F43E79FE116CB1B94E768075C","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"41462F6C19492B6FF4F5FAF7D6ED16E1264D1AAE","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"45C2BA0FF19BCB262FA5713E993FBA4408258BC1","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4D21CFCB799E0C9B66CE566F6AB68BCCCBC75FDA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"4F5F37B9DB46C29C48C0B2231BEAD3E14F04CD1A","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":1,"validator_address":"","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"65697A8BAA1891DC3D466DFEDBE51EAC264936B7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"7299ADB3E975B1A7B461F45C4CCA6AFA39DBC9A7","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"8A93B739376EBA41191E593847B1C844E66F29CA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"A49685C59E968B2B097B0CB09E8D08D6B7FCD69E","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"C9D2DCB223D4353076E6A0E7E82DD99E42980CCA","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"CCB476A8C39D26D81428AB9FBD848F9B17E1BCE0","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"D94347C994432FFC131B1FA33A61159E5FAB7D5F","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"E1E054EDC9AEBA25B10965C927AB35D173080DC9","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F1D993F20EFE45757E62F9AEB6BB14B15DBF2AE2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"F314C84E265C6CF2D834F0CAAD99900CF6857FB2","timestamp":"0001-01-01T00:00:00Z","signature":null},{"block_id_flag":5,"validator_address":"FDCD7DBB810E66664F7ACA241E05834DDB79A5E0","timestamp":"0001-01-01T00:00:00Z","signature":null}]} 2025-04-04T22:07:27+02:00 INFO Timed out module=consensus dur=2000 height=3266485 round=0 step=RoundStepPropose
2025-04-04T22:07:27+02:00 INFO Timed out module=consensus dur=2000 height=3266485 round=0 step=RoundStepPropose 2025-04-04T22:07:32+02:00 INFO Commit is for a block we do not know about; set ProposalBlock=nil height=3266485 commit_round=0 module=consensus proposal= commit=7ED55DDCD6F383DE19903DB1BE8C5DFC22A9DC6F313AF53FE3ADA9D00ACD3597 2025-04-04T22:07:32+02:00 INFO Received complete proposal block module=consensus height=3266485 hash=7ED55DDCD6F383DE19903DB1BE8C5DFC22A9DC6F313AF53FE3ADA9D00ACD3597 2025-04-04T22:07:32+02:00 INFO Finalizing commit of block module=consensus height=3266485 hash=7ED55DDCD6F383DE19903DB1BE8C5DFC22A9DC6F313AF53FE3ADA9D00ACD3597 root=F9D657EF80A634A5B48F5FFD98240E8BD8DD466563744609C19B957E023FB940 num_txs=2 2025-04-04T22:07:32+02:00 INFO processExecutionPayload service=state-processor consensus height=3266485 payload height=3266485 payload timestamp=1743794398 consensus timestamp=1743794399 verify payload=true 2025-04-04T22:07:32+02:00 INFO Inserted new payload into execution chain service=execution-engine payload_block_hash=0x6ed9ada159870693747ceb418e743809bf8c190e323e5ace75a802cae0e0280b payload_parent_block_hash=0x70951cbac670f78d99a18b6badca7e311f0baaaae8cb001d4862f7d7ce0de9f4 2025-04-04T22:07:33+02:00 INFO Processed withdrawals service=state-processor num_withdrawals=1 evm_inflation=5.75 2025-04-04T22:07:33+02:00 INFO Forkchoice updated service=execution-engine head_block_hash=0x6ed9ada159870693747ceb418e743809bf8c190e323e5ace75a802cae0e0280b safe_block_hash=0x70951cbac670f78d99a18b6badca7e311f0baaaae8cb001d4862f7d7ce0de9f4 finalized_block_hash=0x70951cbac670f78d99a18b6badca7e311f0baaaae8cb001d4862f7d7ce0de9f4 with_attributes=false 2025-04-04T22:07:33+02:00 INFO Finalized block module=state height=3266485 num_txs_res=2 num_val_updates=0 block_app_hash=287B8ECAA537072B669D0E62838ACEE0F41198C7AF36D4D9BCDFD7C9A69724ED syncing_to_height=3266485 2025-04-04T22:07:33+02:00 INFO Committed state module=state height=3266485 block_app_hash=F9D657EF80A634A5B48F5FFD98240E8BD8DD466563744609C19B957E023FB940 2025-04-04T22:07:33+02:00 INFO Timed out module=consensus dur=132.946819 height=3266486 round=0 step=RoundStepNewHeight 2025-04-04T22:07:35+02:00 INFO Timed out module=consensus dur=2000 height=3266486 round=0 step=RoundStepPropose
I had this exact same issue. Long story short, downgrading my reth from 1.1.5 to 1.1.4 resolved the issue of falling behind. Becond version is 1.1.4.
To note, this issue was also present for beacond 1.1.4 and 1.1.3 both combined with reth 1.1.5. My node would start falling behind instantly once syncing completed. I'm at 24+ hours without a single issue.
Any update here? Will try reth 1.1.4 myself now
Hello @Ethics3606,
I recommend reducing the total number of connections your node sustains. BeaconKit v.1.1.4 introduces warning to nudge your config in the right direction.
Recommendation is to have max_num_outbound_peers and max_num_inbound_peers set to 10 and 40 respectively.
Also increase the rpc_timeout config to 2 seconds (which also v.1.1.4 does automatically)
Same issue with beacon kit v1.1.4 and geth v1.14.13
Geth falls behind after several hours of working
Restart helps, but not for long
Same issue with
beacon kit v1.1.4andgeth v1.14.13Geth falls behind after several hours of working Restart helps, but not for long
Hello. @e1ghtball, can you clarify how many connections the node has? We recommend limiting them to max 50 as per https://github.com/berachain/beacon-kit/issues/2553#issuecomment-2781422796
Hello. @e1ghtball, can you clarify how many connections the node has? We recommend limiting them to max 50 as per #2553 (comment)
thanks for a quick reply @abi87 !
node started from official quickstart script, here is config.toml:
max_num_inbound_peers = 40
max_num_outbound_peers = 10
The only significant difference from default settings is that i set EL_ARCHIVE_NODE=true in env.sh
Can you tell me how to check number of current connections please?
Hello. @e1ghtball, can you clarify how many connections the node has? We recommend limiting them to max 50 as per #2553 (comment)
thanks for a quick reply @abi87 ! node started from official quickstart script, here is
config.toml:max_num_inbound_peers = 40 max_num_outbound_peers = 10The only significant difference from default settings is that i set
EL_ARCHIVE_NODE=truein env.shCan you tell me how to check number of current connections please?
Hello @e1ghtball,
so those limits should be strict unless you configured the node as seed, which I don't think you have.
To check peers at consensus layer level, you can look at curl http://localhost:26657/net_info RPC and look for peers