erigon
erigon copied to clipboard
Published
20 hours ago •
erigontech
erigontech
Erigon Polygon mainnet don't synchronize
System information
Erigon version: erigon version 2022.09.1-alpha-d0a0d1df
OS & Version: Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Steps to reproduce the behaviour
Run fully Polygon mainnet synchronized archive node.
Backtrace
Few days ago stop synchronize
[INFO] [09-09|07:52:40.128] Queue sizes anchors=1 links=83678 persisted=0
[INFO] [09-09|07:52:40.128] {32795679-32879335 links=83678 (32795679-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879335)} => 0f2aa7b682ca66beca871475dc2203e373702a6117a95c879fcd7af364239735, anchorQueue.idx=0, next retry in 4.064391442s
[INFO] [09-09|07:53:00.126] Req/resp stats req=15 reqMin=32795474 reqMax=32795677 skel=16 skelMin=32798504 skelMax=33093224 resp=44 respMin=32795476 respMax=32879345 dups=639
[INFO] [09-09|07:53:00.129] Queue sizes anchors=1 links=83700 persisted=0
[INFO] [09-09|07:53:00.129] {32795667-32879345 links=83700 (32795667-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879345)} => fd8a661845da0249e72781d1d321e3ff3be2025eafc9356ebfd889bb7e1ccb4b, anchorQueue.idx=0, next retry in 4.506130301s
[INFO] [09-09|07:53:20.127] Req/resp stats req=28 reqMin=32795447 reqMax=32795665 skel=15 skelMin=32798504 skelMax=33093224 resp=59 respMin=32795449 respMax=32879355 dups=749
[INFO] [09-09|07:53:20.128] Queue sizes anchors=1 links=83737 persisted=0
[INFO] [09-09|07:53:20.128] {32795640-32879355 links=83737 (32795640-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879355)} => a65cc06e0a49f3e0d3b9523638fc45b38c5633b8bfae2b3f4b2fad72950f2091, anchorQueue.idx=0, next retry in 4.783409629s
[INFO] [09-09|07:53:27.576] [p2p] GoodPeers eth66=33
[INFO] [09-09|07:53:29.636] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=3.0GB sys=9.0GB
[INFO] [09-09|07:53:40.126] Req/resp stats req=23 reqMin=32795426 reqMax=32795638 skel=15 skelMin=32798504 skelMax=33093224 resp=47 respMin=32795428 respMax=32879363 dups=743
[INFO] [09-09|07:53:40.129] Queue sizes anchors=1 links=83766 persisted=0
[INFO] [09-09|07:53:40.129] {32795619-32879363 links=83766 (32795619-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879363)} => d42b8e48ddf02c4cb8dc401e229f598c9dedd094e56d3529faf2e622976be431, anchorQueue.idx=0, next retry in 4.925656634s
[INFO] [09-09|07:54:00.126] Req/resp stats req=9 reqMin=32795419 reqMax=32795617 skel=16 skelMin=32798504 skelMax=33093224 resp=43 respMin=32795421 respMax=32879373 dups=696
[INFO] [09-09|07:54:00.129] Queue sizes anchors=1 links=83783 persisted=0
[INFO] [09-09|07:54:00.129] {32795612-32879373 links=83783 (32795612-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879373)} => 2f8f64e86cf0b317c15e50bef7242a3ba168e2bf743d0d8c9ed4d2be7a9b3f9a, anchorQueue.idx=0, next retry in -602.000874ms
[INFO] [09-09|07:54:20.127] Req/resp stats req=26 reqMin=32795397 reqMax=32795611 skel=16 skelMin=32798504 skelMax=33093224 resp=55 respMin=32795399 respMax=32879383 dups=693
[INFO] [09-09|07:54:20.129] Queue sizes anchors=1 links=83815 persisted=0
[INFO] [09-09|07:54:20.129] {32795590-32879383 links=83815 (32795590-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879383)} => 014ece2c1daae5ac5a9ab9fc596f22d98267c8c42388b061c24708208111ea58, anchorQueue.idx=0, next retry in 4.899841691s
[INFO] [09-09|07:54:29.637] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=3.2GB sys=9.0GB
[INFO] [09-09|07:54:40.126] Req/resp stats req=19 reqMin=32795381 reqMax=32795588 skel=16 skelMin=32798504 skelMax=33093224 resp=53 respMin=32795383 respMax=32879393 dups=802
[INFO] [09-09|07:54:40.129] Queue sizes anchors=1 links=83841 persisted=0
[INFO] [09-09|07:54:40.129] {32795574-32879393 links=83841 (32795574-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879393)} => 37520e96e8ac2fecebb312578131e643f70241efe86c4217ae38d022efe336e5, anchorQueue.idx=0, next retry in 4.91400581s
[INFO] [09-09|07:55:00.126] Req/resp stats req=22 reqMin=32795361 reqMax=32795572 skel=15 skelMin=32798504 skelMax=33093224 resp=50 respMin=32795363 respMax=32879403 dups=638
[INFO] [09-09|07:55:00.129] Queue sizes anchors=1 links=83872 persisted=0
[INFO] [09-09|07:55:00.129] {32795554-32879403 links=83872 (32795554-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879397-32879403)} => fc8cf48e35903ba8c092f8629b14c1dc4a9846b91cd464974ccfd7605d726bb1, anchorQueue.idx=0, next retry in 2.9820826s
[INFO] [09-09|07:55:20.126] Req/resp stats req=6 reqMin=32795358 reqMax=32795553 skel=15 skelMin=32798504 skelMax=33093224 resp=34 respMin=32795360 respMax=32879413 dups=745
[INFO] [09-09|07:55:20.130] Queue sizes anchors=1 links=83885 persisted=0
[INFO] [09-09|07:55:20.130] {32795551-32879413 links=83885 (32795551-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32879397-32879413)} => 794edbc20320e2a6dddaf067bdba58e50a3a4cc8ee8ae5e1f1abaf98d841c099, anchorQueue.idx=0, next retry in 1.898040139s
[INFO] [09-09|07:55:27.575] [p2p] GoodPeers eth66=33
[INFO] [09-09|07:55:29.637] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=2.9GB sys=9.0GB
We run the same issue on full node (Effective prune_flags="--prune.h.older=20000 --prune.r.older=20000 --prune.t.older=20000 --prune.c.older=20000" snapshot_flags="--snapshots=true" history.v2=false) when upgrading from 07.04 to 09.01. Upgrade took a few minutes of downtime and now it can never catch up, even devel branch build doesn't help as mentioned in https://github.com/ledgerwatch/erigon/issues/5272.
Here are some logs right after start:
[INFO] [09-09|07:19:29.643] Started P2P networking version=66 self=enode://4027c742ce1f134db93a386af78c09a0bb33f75ca2fbf0e032539b83a50eb7f1635f7e375a9449e067ad3a232029e4703c7581f54bca811193557441987c1da9@127.0.0.1:30308 name=erigon/v2022.09.1-alpha-d0a0d1df/linux-amd64/go1.19
[INFO] [09-09|07:19:29.655] [Snapshots] Fetching torrent files metadata
[INFO] [09-09|07:19:33.172] [txpool] Started
[INFO] [09-09|07:19:36.880] [Snapshots] Stat blocks=32707k indices=32707k alloc=2.7GB sys=2.9GB
[INFO] [09-09|07:19:56.881] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=1392076
[INFO] [09-09|07:20:16.881] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=2698344
[INFO] [09-09|07:20:29.654] [txpool] stat block=32796968 pending=10000 baseFee=177 queued=30000 alloc=3.3GB sys=5.9GB
[INFO] [09-09|07:20:41.837] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=3728270
[INFO] [09-09|07:20:56.880] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=4877208
[INFO] [09-09|07:21:16.880] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=6450648
[INFO] [09-09|07:21:27.575] [p2p] GoodPeers eth66=14
[INFO] [09-09|07:21:29.675] [txpool] stat block=32796968 pending=10000 baseFee=177 queued=30000 alloc=5.4GB sys=5.9GB
[INFO] [09-09|07:21:37.605] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=7456541
...
[INFO] [09-09|07:27:56.883] [1/16 Snapshots] Writing total difficulty index for snapshots block_num=31753365
[INFO] [09-09|07:28:29.637] [txpool] stat block=32796968 pending=10000 baseFee=177 queued=30000 alloc=3.7GB sys=6.1GB
[INFO] [09-09|07:28:49.772] [Snapshots] ETL [2/2] Loading into=HeaderNumber current key=454766eb... alloc=4.5GB sys=6.1GB
[INFO] [09-09|07:29:19.772] [Snapshots] ETL [2/2] Loading into=HeaderNumber current key=89e1424a... alloc=3.0GB sys=6.1GB
[INFO] [09-09|07:29:27.575] [p2p] GoodPeers eth66=32
[INFO] [09-09|07:29:29.969] [txpool] stat block=32796968 pending=10000 baseFee=177 queued=30000 alloc=3.4GB sys=6.1GB
[INFO] [09-09|07:29:49.772] [Snapshots] ETL [2/2] Loading into=HeaderNumber current key=ce095f26... alloc=4.2GB sys=6.1GB
[INFO] [09-09|07:30:13.136] [1/16 Snapshots] DONE in=10m43.489438047s
[INFO] [09-09|07:30:13.136] [2/16 Headers] Waiting for headers... from=32796968
[INFO] [09-09|07:30:29.636] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=3.3GB sys=6.1GB
[INFO] [09-09|07:30:33.137] [2/16 Headers] Inserting headers progress=32816035 queue=1
[INFO] [09-09|07:30:53.137] [2/16 Headers] Inserting headers progress=32852064 queue=1
[INFO] [09-09|07:31:13.138] [2/16 Headers] Inserting headers progress=32878689 queue=1
[INFO] [09-09|07:31:13.565] [2/16 Headers] Processed highest inserted=32878712 age=1s
[INFO] [09-09|07:31:13.565] [2/16 Headers] DONE in=1m0.429066589s
[INFO] [09-09|07:31:15.861] [5/16 Bodies] Processing bodies... from=32706999 to=32878712
[INFO] [09-09|07:31:27.575] [p2p] GoodPeers eth66=32
[INFO] [09-09|07:31:29.636] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=3.1GB sys=6.1GB
[INFO] [09-09|07:32:29.637] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=3.5GB sys=8.0GB
[INFO] [09-09|07:33:27.576] [p2p] GoodPeers eth66=33
[INFO] [09-09|07:33:29.659] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=6.4GB sys=8.3GB
[INFO] [09-09|07:33:35.874] [5/16 Bodies] Wrote block bodies block_num=32797337 delivery/sec=35.9MB wasted/sec=3.1MB alloc=4.3GB sys=8.5GB
[INFO] [09-09|07:33:55.862] [5/16 Bodies] Wrote block bodies block_num=32797337 delivery/sec=16.8MB wasted/sec=621.4KB alloc=5.9GB sys=8.5GB
...
[INFO] [09-09|07:36:55.863] [5/16 Bodies] Wrote block bodies block_num=32858454 delivery/sec=237.4KB wasted/sec=0B alloc=4.6GB sys=9.0GB
[INFO] [09-09|07:37:14.614] [5/16 Bodies] Processed highest=32878712
[INFO] [09-09|07:37:14.614] [5/16 Bodies] DONE in=5m58.753535602s
[INFO] [09-09|07:37:14.618] [6/16 Senders] Started from=32706999 to=32878712
[INFO] [09-09|07:37:27.575] [p2p] GoodPeers eth66=32
[INFO] [09-09|07:37:29.637] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=4.5GB sys=9.0GB
[INFO] [09-09|07:37:44.618] [6/16 Senders] Recovery block_number=32873653
[INFO] [09-09|07:37:52.562] [7/16 Execution] Blocks execution from=32796968 to=32878712
[WARN] [09-09|07:37:52.729] [7/16 Execution] Execution failed block=32796969 hash=0x703df002f3c443e4a89abd2ae3495c2e9fc03fb245970616bef848c8b81ffd14 err="mismatched receipt headers for block 32796969 (0x27cc3bafd77c5f0d25974e498fc93a35c8b3212282f7878280fff41fad6be8be != 0x71ec06343ab71883799c6e2c5e6bd9afdcbc71a2e044eca188e1a206d05a1c23)"
[INFO] [09-09|07:37:52.729] UnwindTo block=32796968 bad_block_hash=0x703df002f3c443e4a89abd2ae3495c2e9fc03fb245970616bef848c8b81ffd14
[INFO] [09-09|07:37:52.729] [7/16 Execution] Completed on block=32796968
[INFO] [09-09|07:38:00.126] [2/16 Headers] Waiting for headers... from=32796968
[INFO] [09-09|07:38:29.639] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=5.5GB sys=9.0GB
[INFO] [09-09|07:39:27.575] [p2p] GoodPeers eth66=32
[INFO] [09-09|07:39:29.636] [txpool] stat block=32796968 pending=10000 baseFee=179 queued=30000 alloc=3.1GB sys=9.0GB
[INFO] [09-09|07:39:40.127] Req/resp stats req=110 reqMin=32796366 reqMax=32796666 skel=17 skelMin=32798504 skelMax=33093224 resp=146 respMin=32796368 respMax=32878957 dups=909
[INFO] [09-09|07:39:40.129] Queue sizes anchors=1 links=82420 persisted=0
[INFO] [09-09|07:39:40.129] {32796559-32878957 links=82420 (32796559-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32878957)} => fd0a4b4603ee30effc9b329ab329cd0e4667a331dc9f67557f91c5ff46ebcb66, anchorQueue.idx=0, next retry in 3.100359694s
[INFO] [09-09|07:40:00.126] Req/resp stats req=71 reqMin=32796297 reqMax=32796558 skel=15 skelMin=32798504 skelMax=33093224 resp=103 respMin=32796299 respMax=32878967 dups=802
[INFO] [09-09|07:40:00.129] Queue sizes anchors=1 links=82499 persisted=0
[INFO] [09-09|07:40:00.129] {32796490-32878967 links=82499 (32796490-32878828-32878848-32878850-32878852-32878853-32878856-32878857-32878858-32878871-32878877-32878878-32878881-32878882-32878885-32878891-32878903-32878905-32878908-32878909-32878910-32878911-32878967)} => 5eff06c457300b0f6e69aa56b16e7d5f83357265cdba21c51b41ec12fa7f83d4, anchorQueue.idx=0, next retry in 2.278128752s
[INFO] [09-09|07:40:20.127] Req/resp stats req=81 reqMin=32796218 reqMax=32796489 skel=17 skelMin=32798504 skelMax=33093224 resp=110 respMin=32796220 respMax=32878975 dups=851
[INFO] [09-09|07:40:20.130] Queue sizes anchors=1 links=82586 persisted=0
Any help would be highly appreciated!
I tried this from the official snapshot and then from scratch. Both of them eventually get stuck on something similar
INFO[09-12|20:54:26.244] [txpool] stat block=1066783 pending=0 baseFee=0 queued=10833 alloc=9.3GB sys=16.0GB
INFO[09-12|20:54:43.510] Req/resp stats req=10803 reqMin=2062019 reqMax=2072855 skel=20 skelMin=2135103 skelMax=2429823 resp=10843 respMin=2062078 respMax=33026082 dups=658984
INFO[09-12|20:54:43.551] Queue sizes anchors=1 links=520217 persisted=65536
INFO[09-12|20:54:43.551] { 2062212-2428479 links=366268 (2062212-2428479)} => 114cbfdc963d91b109bd59b55ee8b25dc4fa7802bd0f0414da7771e311c1941b, anchorQueue.idx=0, next retry in 4.999296934s
INFO[09-12|20:55:03.508] Req/resp stats req=10668 reqMin=2051512 reqMax=2062210 skel=20 skelMin=2135103 skelMax=2429823 resp=10745 respMin=2051561 respMax=33026092 dups=669600
INFO[09-12|20:55:03.532] Queue sizes anchors=1 links=530734 persisted=65536
INFO[09-12|20:55:03.532] { 2051705-2428479 links=376775 (2051705-2428479)} => 1f8ac41acb12987136f1513c81ab97e1882f95e9380433563a679fc3ddbbbdcc, anchorQueue.idx=0, next retry in 4.997715513s````
@SwaroopH try this repo https://github.com/maticnetwork/erigon.git
@masayil yup, just started using that with a new snapshot. Will report once it's syncs up.
This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
For anyone else looking at the issue, the above repo did sync. I've run into issues on few nodes but we do have a working Erigon archive node fully synced to Matic Mainnet.
This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.
This issue was closed because it has been stalled for 7 days with no activity.