bor icon indicating copy to clipboard operation
bor copied to clipboard

Bor upgrade version 1.4.0 is out of sync

Open okogie opened this issue 1 year ago • 23 comments

  • Bor Version: v1.4.0

  • Description The bor log keeps outputting Whitelisting milestone deferred. When bor is restarted, it will catch up with the latest block and then get stuck at Whitelisting milestone deferred.

  • Logs Image20240916171647

okogie avatar Sep 16 '24 09:09 okogie

we found this issue too in amoy testnet

wangjiangw avatar Sep 17 '24 14:09 wangjiangw

we found this issue too in amoy testnet

Is there any solution?

okogie avatar Sep 18 '24 01:09 okogie

Hi, Good morning.

Iam a node operator, am also facing the same issue in one of my polygon node. have you been able to a find a solution for this?

Thanks.

ep 19 00:50:15 polygon-sentry-01 bor[706971]: INFO [09-19|00:50:15.856] Commit Interrupt. Pre-committing the current block block=61,984,255 Sep 19 00:50:15 polygon-sentry-01 bor[706971]: WARN [09-19|00:50:15.948] Tx Level Interrupt hash=000000..000000 Sep 19 00:50:19 polygon-sentry-01 bor[706971]: INFO [09-19|00:50:19.914] Whitelisting milestone deferred err="chain out of sync" Sep 19 00:50:31 polygon-sentry-01 bor[706971]: INFO [09-19|00:50:31.915] Whitelisting milestone deferred err="chain out of sync" Sep 19 00:50:43 polygon-sentry-01 bor[706971]: INFO [09-19|00:50:43.915] Whitelisting milestone deferred err="chain out of sync" Sep 19 00:50:55 polygon-sentry-01 bor[706971]: INFO [09-19|00:50:55.915] Whitelisting milestone deferred err="chain out of sync" Sep 19 00:51:07 polygon-sentry-01 bor[706971]: INFO [09-19|00:51:07.916] Whitelisting milestone deferred err="chain out of sync" Sep 19 00:51:19 polygon-sentry-01 bor[706971]: INFO [09-19|00:51:19.915] Whitelisting milestone deferred err="chain out of sync"

AmardeepSambaru avatar Sep 19 '24 05:09 AmardeepSambaru

We're also seeing exactly the same behavior as described, we've just upgraded to v1.4.1 and that hasn't made any difference.

adamf-sendblocks avatar Sep 22 '24 09:09 adamf-sendblocks

Yes, I updated to v1.4.1 version but same errors. it worked fine for 30 minutes to an hour, but then I started encountering the error mentioned above.

AmardeepSambaru avatar Sep 22 '24 15:09 AmardeepSambaru

I upgraded two nodes in total, and each node had the same problem. The first node automatically recovered after running for a few days, but the other node still had the same problem

okogie avatar Sep 23 '24 02:09 okogie

ut the other node still had the same problem

our node have been normal after a few hours, and in that time i have tried to remove the node key info, but i don't know if it makes the node normally.

wangjiangw avatar Sep 23 '24 08:09 wangjiangw

Hi,

Can you please follow the steps mentioned below. it will work. i tried the same way.

  1. Increase the maxpeer to 400.
  2. Add "enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303" under bootnodes param.
  3. Add all the enodes mentioned in the bootnodes to static-nodes and trusted-nodes.

After making the changes, please restart the node and wait for sometime for the node to catch. Make sure you are on latest bor version v1.4.1.

like below

[p2p] maxpeers = 400 port = 30303 # maxpendpeers = 50 # bind = "0.0.0.0" # nodiscover = false # nat = "any" # netrestrict = "" # nodekey = "" # nodekeyhex = "" # txarrivalwait = "500ms" [p2p.discovery] # v4disc = true # v5disc = false bootnodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] # bootnodesv4 = [] # bootnodesv5 = [] static-nodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] trusted-nodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] # Recommended setting with sentry <> validator architecture # dns = []

Thanks.

AmardeepSambaru avatar Sep 24 '24 05:09 AmardeepSambaru

你好,

您能否按照下面提到的步骤操作?它会起作用。我尝试了同样的方法。

  1. 将 maxpeer 增加到 400。
  2. 在bootnodes参数下添加“enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303”。
  3. 将 bootnodes 中提到的所有 enodes 添加到 static-nodes 和 Trusted-nodes 中。

进行更改后,请重新启动节点并等待一段时间以便节点捕获。确保您使用的是最新版 bor v1.4.1。

像下面这样

[p2p] maxpeers = 400 port = 30303 # maxpendpeers = 50 # bind = "0.0.0.0" # nodiscover = false # nat = "any" # netrestrict = "" # nodekey = "" # nodekeyhex = "" # txarrivalwait = "500ms" [p2p.discovery] # v4disc = true # v5disc = false bootnodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", “enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303”, “enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303”] #bootnodesv4 = [] #bootnodesv5 = [] 静态节点= ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", “enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303”, “enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303”] 受信任节点 = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", “enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303”, "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] # 使用 sentry 的推荐设置 <> 验证器架构 # dns = []

谢谢。

I do this and it's still stuck INFO [09-24|09:30:14.589] Whitelisting milestone deferred err="chain out of sync"

After one night, the block automatically caught up

yuanwj210 avatar Sep 24 '24 09:09 yuanwj210

Hi,

Can you please follow the steps mentioned below. it will work. i tried the same way.

1. Increase the maxpeer to 400.

2. Add "enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303" under bootnodes param.

3. Add all the enodes mentioned in the bootnodes to static-nodes and trusted-nodes.

After making the changes, please restart the node and wait for sometime for the node to catch. Make sure you are on latest bor version v1.4.1.

like below

[p2p] maxpeers = 400 port = 30303 # maxpendpeers = 50 # bind = "0.0.0.0" # nodiscover = false # nat = "any" # netrestrict = "" # nodekey = "" # nodekeyhex = "" # txarrivalwait = "500ms" [p2p.discovery] # v4disc = true # v5disc = false bootnodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] # bootnodesv4 = [] # bootnodesv5 = [] static-nodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] trusted-nodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"] # Recommended setting with sentry <> validator architecture # dns = []

Thanks.

Hello,

I was having exact same issues when trying to sync from snapshot. Thought there might be an issue with the snapshot, pruned all bor data. Delete bor. Reinstalled bor v1.4.1. Started syncing from the scratch. Nothing at all.

Cannot connect to a single peer. Getting following error

Sep 25 07:19:05 polygon-node bor[517422]: TRACE[09-25|07:19:05.911] Failed RLPx handshake addr=148.251.110.179:30313 conn=dyndial err="read tcp 10.10.0.175:51994->148.251.110.179:30313: read: connection reset by peer"

eth.syncing false admin.peers [] eth.blockNumber 0

xass1on avatar Sep 25 '24 07:09 xass1on

After one night, the block automatically caught up

@yuanwj210 please elaborate: do you mean that you had the same syncing issue as described, and that eventually it went away by itself?

adamf-sendblocks avatar Sep 25 '24 07:09 adamf-sendblocks

faces the same issue as well with Bor nodes on version v1.4.1. It seems to resolved after some time but keeps happening making the node go in and out of sync. Any fixes / cause of the issue yet?

Brindrajsinh-Chauhan avatar Sep 25 '24 16:09 Brindrajsinh-Chauhan

faces the same issue as well with Bor nodes on version v1.4.1. It seems to resolved after some time but keeps happening making the node go in and out of sync. Any fixes / cause of the issue yet?

facing exactly the same issue.

derostenlebt avatar Sep 26 '24 10:09 derostenlebt

faces the same issue as well with Bor nodes on version v1.4.1. It seems to resolved after some time but keeps happening making the node go in and out of sync. Any fixes / cause of the issue yet?

facing exactly the same issue.

+1

0x1un avatar Oct 02 '24 06:10 0x1un

the same issue

andreyvg avatar Oct 07 '24 11:10 andreyvg

the same issue bot 1.4.1

maximdogonov avatar Oct 10 '24 06:10 maximdogonov

This seems to be because of the lack of peers. What is the peer count in the node?

For Mainnet, can you try using this config option to get connected to more peers?

[p2p]
    maxpeers = 200
    # Discovery settings for finding other nodes
    [p2p.discovery]
        dns = ["enrtree://AKUEZKN7PSKVNR65FZDHECMKOJQSGPARGTPPBI7WS2VUL4EGR6XPC@pos.polygon-peers.io"] 

ssandeep avatar Oct 10 '24 10:10 ssandeep

Thank you @ssandeep. This dns setting has solved my issue with peers. Before applying this I had about 1-8 peers.

andreyvg avatar Oct 10 '24 11:10 andreyvg

It works like a charm. Make sure to have the list uncommented [p2p.discovery] ;)

Creamers158 avatar Oct 10 '24 11:10 Creamers158

This seems to be because of the lack of peers. What is the peer count in the node?

For Mainnet, can you try using this config option to get connected to more peers?

[p2p]
    maxpeers = 200
    # Discovery settings for finding other nodes
    [p2p.discovery]
        dns = ["enrtree://AKUEZKN7PSKVNR65FZDHECMKOJQSGPARGTPPBI7WS2VUL4EGR6XPC@pos.polygon-peers.io"] 

After adding DNS, I still can't keep up, and I get stuck every time I keep up Whitelisting checkpoint deferred err="chain out of sync"

yuanwj210 avatar Oct 14 '24 01:10 yuanwj210

@yuanwj210 - did you restart your node after updating the config file? Also how many peers your node is connected with?

pratikspatil024 avatar Oct 17 '24 05:10 pratikspatil024

@yuanwj210 - did you restart your node after updating the config file? Also how many peers your node is connected with? @pratikspatil024 After each reboot, it catches up with the latest block and stops moving, and again,Here is my configuration as follows

[p2p.discovery]
        # v5disc = false
bootnodes = ["enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303","enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303"]

static-nodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"]

trusted-nodes = ["enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303", "enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303", "enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"]

dns = ["enrtree://AKUEZKN7PSKVNR65FZDHECMKOJQSGPARGTPPBI7WS2VUL4EGR6XPC@pos.polygon-peers.io"]

How do I check how many peers are connected?

Every time the node catches up with the latest block, it reports the following error, which can not be repaired? It is very inexplicable INFO [10-18|02:09:40.152] Whitelisting milestone deferred err="chain out of sync"

yuanwj210 avatar Oct 18 '24 02:10 yuanwj210

Hey @yuanwj210 you can find out the number of peers using admin.peers.length in the bor ipc. Also please try upgrading your node to the new release.

Raneet10 avatar Oct 24 '24 04:10 Raneet10

Hey @yuanwj210 you can find out the number of peers using admin.peers.length in the bor ipc. Also please try upgrading your node to the new release.

It has been resolved with an updated version

yuanwj210 avatar Oct 29 '24 01:10 yuanwj210

Hey @yuanwj210 you can find out the number of peers using admin.peers.length in the bor ipc. Also please try upgrading your node to the new release.

It has been resolved with an updated version

What version did you update to? I'm already on 1.4.1 and no luck at all

xass1on avatar Oct 30 '24 04:10 xass1on

@xass1on - can you upgrade to v1.5.2?

pratikspatil024 avatar Oct 30 '24 15:10 pratikspatil024

@pratikspatil024 Updated to v1.5.2 and it is syncing at the moment.

xass1on avatar Oct 31 '24 02:10 xass1on